[Darshan-users] compiling statically with intel toolchain

[Shane Snyder]

In my experience, not getting a log file in cases where Darshan appears 
to be linked in correctly (as evidenced by darshan symbols from 'nm') is 
caused by Darshan not actually intercepting MPI_Init/Finalize.

A method we've used in the past to confirm this is by getting the linker 
to trace the MPI_Init symbol to see where it is actually defined. You 
can do this by adding this flag to your link command:

-Wl,-yMPI_Init

We'd expect to see something like the following in the output:

/usr/common/software/darshan/3.0.0-pre3/lib/libdarshan.a(darshan-core-init-finalize.o): 
definition of MPI_Init

Maybe that gives us some sort of indication of what's going awry?

Thanks,
--Shane

On 05/23/2016 02:46 PM, Salmon, Rene wrote:
> Hi Phil,
>
> Thanks for the info that was helpful.  I did manage to link in darshan and MPI statically I had to put in the “lib.a” files by hand as the “darshan-config –pre/post” scripts where still trying to link in darshan dynamically.
> Here is the link line I used that seem to generate a binary.
>
>
> mpiifort  -DLinux -Dx86_64 -DINTEL -DCompLevel16 -DX86_64 -D__x86_64__ -static-intel -static_mpi -DOpenMP -openmp -DSWAP -O3 -g -o /hpcdata/salmr0/mspain/usp2hdf5/dds/bin/Linux/3.0/x86_64/ddswriter.revUnversioned /hpc/tstapps/intel/x86_64/ics-2016.update.1/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib/libmpifort.a -L/hpc/tstapps/SLES/3.0/x86_64/darshan/3.0.0/lib /hpc/tstapps/SLES/3.0/x86_64/darshan/3.0.0/lib/libdarshan.a -lz -Wl,@/hpc/tstapps/SLES/3.0/x86_64/darshan/3.0.0/share/ld-opts/darshan-base-ld-opts svn_version.o PartitionLayout_mod.o ddswriter.o -L/tstapps/asi/src/dds/lib/Linux/3.0/x86_64/Intel-16.0.1.150MPI5.1.2MKL11.3.1 -ldds_r3 -lgio -lfhost -L/hpc/tstapps/SLES/3.0/x86_64/darshan/3.0.0/lib -Wl,--start-group /hpc/tstapps/SLES/3.0/x86_64/darshan/3.0.0/lib/libdarshan.a /hpc/tstapps/SLES/3.0/x86_64/darshan/3.0.0/lib/libdarshan-stubs.a -Wl,--end-group -lz
>
>
>
> Here is what the binary look like:
>
> hpcbuild02(salmr0)1181:ldd bin/Linux/3.0/x86_64/ddswriter
> 	linux-vdso.so.1 =>  (0x00007ffe791a7000)
> 	libz.so.1 => /lib64/libz.so.1 (0x00002aafaab7c000)
> 	libdl.so.2 => /lib64/libdl.so.2 (0x00002aafaad92000)
> 	librt.so.1 => /lib64/librt.so.1 (0x00002aafaaf96000)
> 	libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aafab1a0000)
> 	libm.so.6 => /lib64/libm.so.6 (0x00002aafab3bd000)
> 	libc.so.6 => /lib64/libc.so.6 (0x00002aafab636000)
> 	/lib64/ld-linux-x86-64.so.2 (0x00002aafaa95a000)
> 	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002aafab9b3000)
> hpcbuild02(salmr0)1182:nm bin/Linux/3.0/x86_64/ddswriter|grep darshan
> 0000000000482e90 T darshan_clean_file_path
> 0000000000483210 t darshan_common_val_compare
> 00000000004830c0 T darshan_common_val_counter
> 00000000004831a0 t darshan_common_val_walker
> 000000000101e860 b darshan_core
> 0000000000463f50 t darshan_core_cleanup
> 00000000004655e0 T darshan_core_initialize
> 0000000000fcfa80 d darshan_core_mutex
> 0000000000464720 T darshan_core_register_module
> 0000000000464b80 T darshan_core_register_record
> 0000000000461c10 T darshan_core_shutdown
> 0000000000464840 T darshan_core_unregister_module
> 0000000000465310 T darshan_core_unregister_record
> 00000000004655b0 T darshan_core_wtime
> 0000000000463e40 t darshan_deflate_buffer..0
> 00000000004642b0 t darshan_get_shared_records
> 00000000004806a0 T darshan_hash
> 0000000000fcfab0 d darshan_mem_alignment
> 0000000000fcfaec d darshan_mem_alignment
> 0000000000b64848 r darshan_module_names
> 0000000000b6487c r darshan_module_versions
> 0000000000479ae0 T darshan_mpiio_shutdown_bench_setup
> 0000000000fcfa20 D darshan_path_exclusions
> 0000000000470250 T darshan_posix_shutdown_bench_setup
> 0000000000461750 T darshan_shutdown_bench
> 0000000000483230 T darshan_variance_reduce
> 0000000000483170 T darshan_walk_common_vals
>
>
>
> The problem I am having now is that I don’t get a darshan log file when I run my executable. I configured darshan with this configure flag “--with-log-path-by-env=DARSHANLOGPATH”.  I set that environment variable in my shell to some directory and then run my executable.  The executable runs fine but no darshan log file gets generated.
>
> Any ideas?   Thanks in advanced.
>
> Rene
>
>
>
>
>
>
> On 5/20/16, 7:33 AM, "darshan-users-bounces at lists.mcs.anl.gov on behalf of Phil Carns" <darshan-users-bounces at lists.mcs.anl.gov on behalf of carns at mcs.anl.gov> wrote:
>
>> Hi Rene,
>>
>> Does mpiifort support the "-show" option (or something similar) to
>> display the underlying link command that it is using?
>>
>> I can describe the manual steps needed to add Darshan support just to
>> see if you can get it working, and then we can maybe circle back to why
>> the Darshan-generated script is triggering dynamic linking.  In general,
>> there are two sets of extra link flags that Darshan needs to add to the
>> link command.  Some should appear before the user-specified libraries in
>> the true link command, some should appear after them.  On my test box
>> they happen to look like this:
>>
>> pcarns at carns-x1:~/working/install/bin$ darshan-config --pre-ld-flags
>> -L/home/pcarns/working/install/lib -ldarshan -lz
>> -Wl,@/home/pcarns/working/install/share/ld-opts/darshan-base-ld-opts
>>
>> pcarns at carns-x1:~/working/install/bin$ darshan-config --post-ld-flags
>> -L/home/pcarns/working/install/lib -Wl,--start-group -ldarshan
>> -ldarshan-stubs -Wl,--end-group -lz -lrt -lpthread
>>
>> On most MPI implementations there is also one more library that you have
>> to add for MPI function wrapping to work.  Darshan-config doesn't add
>> that because it is dependent on the MPI implementation. On most versions
>> of MPI this is either -lmpifort or -lfmpich.  I'm not sure which one
>> Intel MPI uses now.
>>
>> You might be able to do all of that external to mpiifort, but I'm not
>> sure.  If so, then your link command would look something like this
>> (possibly swapping out -lmpifort for -lfmpich or omitting it entirely):
>>
>> mpiifort -DLinux -Dx86_64 -DINTEL -DCompLevel16 -DX86_64 -D__x86_64__ -static-intel -static_mpi -DOpenMP -openmp -DSWAP -O3 -g -o ./ddswriter.revUnversioned -lmpifort `darshan-config --pre-ld-flags` PartitionLayout_mod.o ddswriter.o -L/tstapps/asi/src/dds/lib/Linux/3.0/x86_64/Intel-16.0.1.150MPI5.1.2MKL11.3.1 -ldds_r3 -lgio -lfhost `darshan-config --post-ld-flags`
>>
>>
>> If that doesn't work then you'll need to find out what the underlying
>> link command looks like (via -show or similar) and insert it there.
>>
>> thanks,
>> -Phil
>>
>
> _______________________________________________
> Darshan-users mailing list
> Darshan-users at lists.mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users