[Darshan-users] darshan3.1.5 issue on Cray XC40 cle6.up05
Phil Carns
carns at mcs.anl.gov
Mon Feb 26 11:17:14 CST 2018
Actually, it turns out that we don't have a up05 system to test on :)
Can you show us the output of the failing ftn command if you add the
-craype-verbose command line option? It might also be helpful to see
the contents of the darshan-runtime.pc and module file.
We'll try building scalapack on some up04 systems to see if we either
see the same problem or can tell what the environment difference is if not.
thanks,
-Phil
On 02/26/2018 10:18 AM, Phil Carns wrote:
> Hi Bilel,
>
> Thanks for the bug report. We can try to reproduce here and confirm.
>
> In the mean time, can you tell me if the test_scalapack.f90 code is
> using MPI or not?
>
> thanks,
> -Phil
>
> On 02/24/2018 02:25 PM, Bilel Hadri wrote:
>>
>> Dear Darshan colleagues,
>>
>> I recently installed darshan 3.1.5 on Shaheen, Cray XC40, we recently
>> upgraded the OS to CLE6.up05 and using the 17.12 PrgEnv.
>>
>> Compiling Scalapack with Cray Libsci fails with the error shown below
>> with all programming environment. Similar issue was observed for
>> other codes, like a simple petsc code.
>>
>> After digging, it seems that it is related to darshan3.1.5 version
>> recently installed on Shaheen. When unloading darshan, the
>> compilation works fine with no issue.
>>
>> The error is not appearing when using darshan 3.1.4.
>>
>> ftn -o exe test_scalapack.f90
>>
>> /opt/cray/dmapp/default/lib64/libdmapp.a(dmapp_internal.o): In
>> function `_dmappi_is_pure_dmapp_job':
>>
>> /home/abuild/rpmbuild/BUILD/cray-dmapp-7.1.1/src/dmapp_internal.c:1401:
>> undefined reference to `__wrap_MPI_Init'
>>
>> /opt/cray/pe/libsci/17.12.1/CRAY/8.6/x86_64/lib/libsci_cray_mpi_mp.a(blacs_exit_.o):
>> In function `blacs_exit_':
>>
>> /b/worker/csml-libsci-sles/build/mp/scalapack/BLACS/SRC/blacs_exit_.c:42:
>> undefined reference to `__wrap_MPI_Finalize'
>>
>> /opt/cray/pe/libsci/17.12.1/CRAY/8.6/x86_64/lib/libsci_cray_mpi_mp.a(blacs_pinfo_.o):
>> In function `blacs_pinfo_':
>>
>> /b/worker/csml-libsci-sles/build/mp/scalapack/BLACS/SRC/blacs_pinfo_.c:18:
>> undefined reference to `__wrap_MPI_Init'
>>
>> /opt/cray/pe/libsci/17.12.1/CRAY/8.6/x86_64/lib/libsci_cray_mpi_mp.a(blacs_pinfo_.oo):
>> In function `Cblacs_pinfo':
>>
>> /b/worker/csml-libsci-sles/build/mp/scalapack/BLACS/SRC/blacs_pinfo_.c:18:
>> undefined reference to `__wrap_MPI_Init'
>>
>> /opt/cray/pe/cce/8.6.5/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
>> link errors found, deleting executable `exe'
>>
>> /usr/bin/sha1sum: exe: No such file or directory
>>
>> hadrib at cdl1:~> ll /usr/bin/sha1
>>
>> sha1pass sha1sum
>>
>> hadrib at cdl1:~> ll /usr/bin/sha1sum
>>
>> -rwxr-xr-x 1 root root 43912 Aug 6 2016 /usr/bin/sha1sum
>>
>> hadrib at cdl1:~> which sha1sum
>>
>> /usr/bin/sha1sum
>>
>> hadrib at cdl1:~> module list
>>
>> Currently Loaded Modulefiles:
>>
>> 1) modules/3.2.10.6 9) pmi/5.0.13 17) atp/2.1.1
>>
>> 2) eproxy/2.0.22-6.0.5.0_2.1__g1ebe45c.ari 10)
>> dmapp/7.1.1-6.0.5.0_49.8__g1125556.ari 18) perftools-base/7.0.0
>>
>> 3) cce/8.6.5 11)
>> gni-headers/5.0.12-6.0.5.0_2.15__g2ef1ebc.ari 19) PrgEnv-cray/6.0.4
>>
>> 4) craype-network-aries 12)
>> xpmem/2.2.4-6.0.5.0_4.8__g35d5e73.ari 20) cray-mpich/7.7.0
>>
>> 5) craype/2.5.13 13) job/2.2.2-6.0.5.0_8.47__g3c644b5.ari
>> 21) slurm/slurm
>>
>> 6) cray-libsci/17.12.1 14)
>> dvs/2.7_2.2.52-6.0.5.2_17.6__g5170dea 22) craype-haswell
>>
>> 7) udreg/2.3.2-6.0.5.0_13.12__ga14955a.ari 15)
>> alps/6.5.28-6.0.5.0_18.6__g13a91b6.ari 23) texlive/2017
>>
>> 8) ugni/6.0.14-6.0.5.0_16.9__g19583bb.ari 16)
>> rca/2.2.16-6.0.5.0_15.34__g5e09e6d.ari 24) darshan/3.1.5
>>
>> hadrib at cdl1:~> module swap PrgEnv-cray/6.0.4 PrgEnv-intel
>>
>> hadrib at cdl1:~> ftn -o exe_i test_scalapack.f90
>>
>> /opt/cray/pe/libsci/17.12.1/INTEL/16.0/x86_64/lib/libsci_intel_mpi.a(blacs_exit_.o):
>> In function `blacs_exit_':
>>
>> blacs_exit_.c:(.text+0xe9): undefined reference to `__wrap_MPI_Finalize'
>>
>> /opt/cray/pe/libsci/17.12.1/INTEL/16.0/x86_64/lib/libsci_intel_mpi.a(blacs_pinfo_.o):
>> In function `blacs_pinfo_':
>>
>> blacs_pinfo_.c:(.text+0x9b): undefined reference to `__wrap_MPI_Init'
>>
>> /opt/cray/pe/libsci/17.12.1/INTEL/16.0/x86_64/lib/libsci_intel_mpi.a(blacs_pinfo_.oo):
>> In function `Cblacs_pinfo':
>>
>> blacs_pinfo_.c:(.text+0x9b): undefined reference to `__wrap_MPI_Init'
>>
>> hadrib at cdl1:~>
>>
>> hadrib at cdl1:~>
>>
>> hadrib at cdl1:~> module swap PrgEnv-intel/6.0.4 PrgEnv-gnu
>>
>> PrgEnv-gnu PrgEnv-gnu/6.0.4
>>
>> hadrib at cdl1:~> module swap PrgEnv-intel/6.0.4 PrgEnv-gnu
>>
>> hadrib at cdl1:~> ftn -o exe_i test_scalapack.f90
>>
>> /opt/cray/pe/libsci/17.12.1/GNU/6.1/x86_64/lib/libsci_gnu_61_mpi.a(blacs_exit_.o):
>> In function `blacs_exit_':
>>
>> blacs_exit_.c:(.text+0xdb): undefined reference to `__wrap_MPI_Finalize'
>>
>> /opt/cray/pe/libsci/17.12.1/GNU/6.1/x86_64/lib/libsci_gnu_61_mpi.a(blacs_pinfo_.o):
>> In function `blacs_pinfo_':
>>
>> blacs_pinfo_.c:(.text+0xb3): undefined reference to `__wrap_MPI_Init'
>>
>> /opt/cray/pe/libsci/17.12.1/GNU/6.1/x86_64/lib/libsci_gnu_61_mpi.a(blacs_pinfo_.oo):
>> In function `Cblacs_pinfo':
>>
>> blacs_pinfo_.c:(.text+0xb3): undefined reference to `__wrap_MPI_Init'
>>
>> =====
>>
>> implicit none
>>
>> integer :: n, nb ! problem size and block size
>>
>> integer :: myunit ! local output unit number
>>
>> integer :: myArows, myAcols ! size of local subset of global array
>>
>> integer :: i,j, igrid,jgrid, iproc,jproc, myi,myj, p
>>
>> real*8, dimension(:,:), allocatable :: myA,myB,myC
>>
>> integer :: numroc ! blacs routine
>>
>> integer :: me, procs, icontxt, prow, pcol, myrow, mycol ! blacs data
>>
>> integer :: info ! scalapack return value
>>
>> integer, dimension(9) :: ides_a, ides_b, ides_c ! scalapack array desc
>>
>> open(unit=1,file="ABCp.dat",status="old",form="formatted")
>>
>> read(1,*)prow
>>
>> read(1,*)pcol
>>
>> read(1,*)n
>>
>> read(1,*)nb
>>
>> close(1)
>>
>> if (((n/nb) < prow) .or. ((n/nb) < pcol)) then
>>
>> print *,"Problem size too small for processor set!"
>>
>> stop 100
>>
>> endif
>>
>> call blacs_pinfo (me,procs)
>>
>> call blacs_get (0, 0, icontxt)
>>
>> call blacs_gridinit(icontxt, 'R', prow, pcol)
>>
>> call blacs_gridinfo(icontxt, prow, pcol, myrow, mycol)
>>
>> myunit = 10+me
>>
>> write(myunit,*)"--------"
>>
>> write(myunit,*)"Output for processor ",me," to unit ",myunit
>>
>> write(myunit,*)"Proc ",me,": myrow, mycol in p-array is ", &
>>
>> myrow, mycol
>>
>> myArows = numroc(n, nb, myrow, 0, prow)
>>
>> myAcols = numroc(n, nb, mycol, 0, pcol)
>>
>> write(myunit,*)"Size of global array is ",n," x ",n
>>
>> write(myunit,*)"Size of block is ",nb," x ",nb
>>
>> write(myunit,*)"Size of local array is ",myArows," x ",myAcols
>>
>> allocate(myA(myArows,myAcols))
>>
>> allocate(myB(myArows,myAcols))
>>
>> allocate(myC(myArows,myAcols))
>>
>> do i=1,n
>>
>> call g2l(i,n,prow,nb,iproc,myi)
>>
>> if (myrow==iproc) then
>>
>> do j=1,n
>>
>> call g2l(j,n,pcol,nb,jproc,myj)
>>
>> if (mycol==jproc) then
>>
>> myA(myi,myj) = real(i+j)
>>
>> myB(myi,myj) = real(i-j)
>>
>> myC(myi,myj) = 0.d0
>>
>> endif
>>
>> enddo
>>
>> endif
>>
>> enddo
>>
>> ! Prepare array descriptors for ScaLAPACK
>>
>> ides_a(1) = 1 ! descriptor type
>>
>> ides_a(2) = icontxt ! blacs context
>>
>> ides_a(3) = n ! global number of rows
>>
>> ides_a(4) = n ! global number of columns
>>
>> ides_a(5) = nb ! row block size
>>
>> ides_a(6) = nb ! column block size
>>
>> ides_a(7) = 0 ! initial process row
>>
>> ides_a(8) = 0 ! initial process column
>>
>> ides_a(9) = myArows ! leading dimension of local array
>>
>> do i=1,9
>>
>> ides_b(i) = ides_a(i)
>>
>> ides_c(i) = ides_a(i)
>>
>> enddo
>>
>> ! Call ScaLAPACK library routine
>>
>> call pdgemm('T','T',n,n,n,1.0d0, myA,1,1,ides_a, &
>>
>> myB,1,1,ides_b,0.d0, &
>>
>> myC,1,1,ides_c )
>>
>> ! Print results
>>
>> call g2l(n,n,prow,nb,iproc,myi)
>>
>> call g2l(n,n,pcol,nb,jproc,myj)
>>
>> if ((myrow==iproc) .and. (mycol==jproc)) &
>>
>> write(*,*) 'c(',n,n,')=',myC(myi,myj)
>>
>> ! Deallocate the local arrays
>>
>> deallocate(myA, myB, myC)
>>
>> ! End blacs for processors that are used
>>
>> call blacs_gridexit(icontxt)
>>
>> call blacs_exit(0)
>>
>> end
>>
>> ! convert global index to local index in block-cyclic distribution
>>
>> subroutine g2l(i,n,np,nb,p,il)
>>
>> implicit none
>>
>> integer :: i ! global array index, input
>>
>> integer :: n ! global array dimension, input
>>
>> integer :: np ! processor array dimension, input
>>
>> integer :: nb ! block size, input
>>
>> integer :: p ! processor array index, output
>>
>> integer :: il ! local array index, output
>>
>> integer :: im1
>>
>> im1 = i-1
>>
>> p = mod((im1/nb),np)
>>
>> il = (im1/(np*nb))*nb + mod(im1,nb) + 1
>>
>> return
>>
>> end
>>
>> ! convert local index to global index in block-cyclic distribution
>>
>> subroutine l2g(il,p,n,np,nb,i)
>>
>> implicit none
>>
>> integer :: il ! local array index, input
>>
>> integer :: p ! processor array index, input
>>
>> integer :: n ! global array dimension, input
>>
>> integer :: np ! processor array dimension, input
>>
>> integer :: nb ! block size, input
>>
>> integer :: i ! global array index, output
>>
>> integer :: ilm1
>>
>> ilm1 = il-1
>>
>> i = (((ilm1/nb) * np) + p)*nb + mod(ilm1,nb) + 1
>>
>> return
>>
>> end
>>
>> -------
>>
>> Bilel Hadri, PhD
>>
>> Computational Scientist
>>
>> KAUST Supercomputing Lab
>>
>> Al Khawarizmi Bldg. (1) Office 126
>>
>> 4700 King Abdullah University of Science and Technology
>>
>> Thuwal 23955-6900
>>
>> Kingdom of Saudi Arabia
>>
>> Office Phone: +966 12 808 0654
>>
>> Cell Phone: + 966 544 700 893
>>
>>
>> ------------------------------------------------------------------------
>> This message and its contents including attachments are intended
>> solely for the original recipient. If you are not the intended
>> recipient or have received this message in error, please notify me
>> immediately and delete this message from your computer system. Any
>> unauthorized use or distribution is prohibited. Please consider the
>> environment before printing this email.
>>
>>
>> _______________________________________________
>> Darshan-users mailing list
>> Darshan-users at lists.mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
>
>
>
>
> _______________________________________________
> Darshan-users mailing list
> Darshan-users at lists.mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/darshan-users/attachments/20180226/c56c38fa/attachment-0001.html>
More information about the Darshan-users
mailing list