[petsc-dev] SuperLU link error on BGP FEN

[Aron Ahmadia]

Just a point of information on this.  I recently re-configured PETSc
(no external packages) on a local disk here on Shaheen and it took 3
minutes:

xxx=========================================================================xxx
   Configure stage complete. Now build PETSc libraries with:
   make PETSC_DIR=/tmp/petsc-3.1-p5 PETSC_ARCH=linux-gnu-c-debug all
xxx=========================================================================xxx
./configure_shaheen.py  84.36s user 85.01s system 89% cpu 3:08.26 total

At some point we might want to figure out why GPFS is so much slower.

A

On Thu, Dec 2, 2010 at 11:34 PM, John R. Cary <cary at txcorp.com> wrote:
> On 12/2/2010 1:21 PM, Barry Smith wrote:
>>
>> On Dec 2, 2010, at 1:59 PM, John R. Cary wrote:
>>
>>> Anyway, given the pokeyness of the FENs on surveyor, it should be some
>>> time before midnight!
>>
>>   This is completely unacceptable! PETSc takes less than 10 minutes to
>> configure and compile on my $2000 laptop (and probably half as long on
>> Satish's $1000 laptop). On a $100 million machine it takes one half a day?
>>
>>   That is frankly disgusting and indicates extreme problems with the
>> configuration of that machine. Why has this problem not been fixed years
>> ago?
>>
>
> Well, I may be exaggerating a bit :-), but I started configuring at:
>
> -rwxrw-r--  1 cary users     924 Dec  2 13:55
> surveyor.alcf.anl.gov-petsc-ser.sh
>
> and it is not done at
>
> Thu Dec  2 14:33:00 CST 2010
>
> This is just the petsc part, not all the packages, so it really does take a
> long time,
> I find.
>
> I do find that the XL compilers are both slow and buggy.  I have submitted
> bug
> reports but they seem to disappear into a black hole at IBM.  For example, I
> submitted alcf-support
> ticket #32756 on August 11, 2009, complaining that the mpi.mod in
> /bgsys/drivers/V1R3M0_460_2008-081112P/ppc/comm/default/include/mpi.mod
> was built with gfortran and incompatible with XLF, even though it is
> called by the XLF compiler wrapper.  The guys at ALCF contacted IBM about
> this,
> but here it is, over a year later, and the same problem is still present.
>  It seems
> like IBM is just not responsive.
>
> OTOH, both Ray and Vitali have been very helpful, without their help I might
> not have found my way around several compiler bugs and other issues.
>
> .....John
>
>
>
>
>
>
>
>>> John
>>>
>>>> satish
>>>>
>>>> On Thu, 2 Dec 2010, Matthew Knepley wrote:
>>>>
>>>>> We always always always always need configure.log.
>>>>>
>>>>>    Matt
>>>>>
>>>>> On Thu, Dec 2, 2010 at 8:41 AM, John R. Cary<cary at txcorp.com>   wrote:
>>>>>
>>>>>> I am trying to link facets on a FEN of surveyor.alcf.anl.gov.  It ends
>>>>>> with the error,
>>>>>>
>>>>>>
>>>>>> /gpfs/home/projects/facets/surveyor/contrib-xlc-9.0/petsc-3.1-p4-ser/lib/libsuperlu_4.0.a(dldperm.o):
>>>>>> In function `dldperm':
>>>>>>
>>>>>> /gpfs/home/cary/facetspkgs/builds/petsc-3.1-p4/ser/externalpackages/SuperLU_4.0/SRC/dldperm.c:127:
>>>>>> undefined reference to `mc64id_'
>>>>>>
>>>>>> /gpfs/home/cary/facetspkgs/builds/petsc-3.1-p4/ser/externalpackages/SuperLU_4.0/SRC/dldperm.c:134:
>>>>>> undefined reference to `mc64ad_'
>>>>>>
>>>>>> which indicates that the SuperLU compiled with PETSc did
>>>>>> not get the fortran underscoring flag correct (which should
>>>>>> be no underscore with xlf).  nm shows
>>>>>>
>>>>>> login1.surveyor$ nm
>>>>>>
>>>>>> /gpfs/home/projects/facets/surveyor/contrib-xlc-9.0/petsc-3.1-p4-ser/lib/libsuperlu_4.0.a
>>>>>> | grep mc64
>>>>>>                U mc64ad_
>>>>>>                U mc64id_
>>>>>> mc64ad.o:
>>>>>> 0000000000000018 D mc64ad
>>>>>> 0000000000000030 D mc64bd
>>>>>> 0000000000000048 D mc64dd
>>>>>> 0000000000000060 D mc64ed
>>>>>> 0000000000000078 D mc64fd
>>>>>> 0000000000000000 D mc64id
>>>>>> 00000000000000c0 D mc64qd
>>>>>> 0000000000000090 D mc64rd
>>>>>> 00000000000000a8 D mc64sd
>>>>>> 00000000000000d8 D mc64ud
>>>>>> 00000000000000f0 D mc64wd
>>>>>>                U mc64ad_
>>>>>>                U mc64id_
>>>>>>
>>>>>> that the underscored symbol is being called, but the
>>>>>> underscore-free symbol is what was defined.
>>>>>>
>>>>>> My PETSc configure line was
>>>>>>
>>>>>> #!/bin/sh
>>>>>>
>>>>>> /gpfs/home/cary/facetspkgs/builds-surveyor-xlc/facetspkgs/petsc-3.1-p4/ser/configure
>>>>>> \
>>>>>>
>>>>>> --prefix=/gpfs/home/projects/facets/surveyor/contrib-xlc-9.0/petsc-3.1-p4-ser
>>>>>> \
>>>>>> --with-mpi=0 \
>>>>>> --with-debugging=0 \
>>>>>> --with-x=0 \
>>>>>> --with-cc='xlc_r' \
>>>>>> --with-cxx='xlC_r' \
>>>>>> --with-fc='xlf_r' \
>>>>>> --COPTFLAGS='-O2 -g' \
>>>>>> --download-superlu \
>>>>>>
>>>>>> --with-lapack-lib=/home/projects/facets/intrepid/contrib/lapack-ser/lib/liblapack.a
>>>>>> \
>>>>>>
>>>>>> --with-blas-lib=/home/projects/facets/intrepid/contrib/lapack-ser/lib/libblas.a
>>>>>> \
>>>>>>
>>>>>> PETSC_DIR=/gpfs/home/cary/facetspkgs/builds-surveyor-xlc/facetspkgs/petsc-3.1-p4/ser
>>>>>> \
>>>>>> PETSC_ARCH=facets-ser \
>>>>>> --CFLAGS='-q64 -qlanglvl=redefmac' \
>>>>>> --CXXFLAGS='-q64 -qlanglvl=redefmac' \
>>>>>> --FFLAGS='-q64 -qextname=flush' \
>>>>>> --with-shared=1
>>>>>>
>>>>>> so I think that PETSc had enough info to figure out the underscoring.
>>>>>> Perhaps this is a bug.
>>>>>>
>>>>>> But regardless, is there a workaround?
>>>>>>
>>>>>> Thx....John
>>>>>>
>>>>>
>>>>>
>>
>
>