[Darshan-users] darshan 3.3.0 issues

[Harms, Kevin]

Thomas,

  1. Not sure why the Intel MPI is tripping up on the configure check. I'm assuming it is MPI3 based. Can you send us the config.log output from that one? Maybe we can see why the check fails.

  2. The partial log indicates the log is incorrect, so those parser errors are expected. I don't know why the finalize hangs. Was this a Fortran hello world example? I'm not familiar with ParaStationMPI but since it is based on MPICH, it should work. Can you send the test code and how you built it? We can try it on a system here.

  3. Autoperf can't be disabled at runtime yet. We have a broader plan to add the ability to enable/disable modules during runtime, but not available yet. We have tested AutoPerf with CrayMPI, MPICH3.3 and OpenMPI. The systems we tested on were generic Linux laptop, Cray XC-40 and Nvidia DGX A100. As far as what can be done with APMPI data, we have some python analysis script based on pydarshan.

https://xgitlab.cels.anl.gov/AutoPerf/autoperf/-/blob/master/apmpi/util/apmpi-analysis.py

  The counters are also output by darshan-parser. We are still in the process of building more analysis based on this work.

kevin

________________________________________
From: Darshan-users <darshan-users-bounces at lists.mcs.anl.gov> on behalf of Thomas Breuer <t.breuer at fz-juelich.de>
Sent: Wednesday, May 19, 2021 6:52 AM
To: darshan-users at lists.mcs.anl.gov
Subject: [Darshan-users] darshan 3.3.0 issues

Dear Darshan Team,

I have installed the latest darshan version (3.3.0) for different MPIs on our HPC JUWELS (https://apps.fz-juelich.de/jsc/hps/juwels/configuration.html) and would like to report two issues:

1. Intel (19.1.3.304) Compiler with IntelMPI/2019.8.254:
- Configure Step fails for the new APMPI feature:
cd darshan-runtime; ./configure --prefix=/path/to/darshan-runtime/3.3.0-iimpi-2020-APMPI --with-mem-align=8 --with-log-path-by-env=DARSHAN_LOG_P
ATH  --with-jobid-env=SLURM_JOBID CC=mpicc --enable-hdf5-mod=$EBROOTHDF5 --enable-apmpi-mod --enable-apmpi-coll-sync
- Error msg: configure: error: APMPI module requires MPI version 3+
- without the new APMPI Options the configure steps ends successfully:
cd darshan-runtime; ./configure --prefix=/p/software/juwels/stages/Devel-2020/software/darshan-runtime/3.3.0-iimpi-2020 --with-mem-align=8 --with-log-path-by-env=DARSHAN_LOG_PATH  --with-jobid-env=SLURM_JOBID CC=mpicc --enable-hdf5-mod=$EBROOTHDF5


2. GCC/9.3.0 Compiler with ParaStationMPI/5.4.7-1 (based on MPICH 3.3.2) (https://github.com/ParaStation/psmpi/):
- darshan-runtime configured with --enable-apmpi-mod --enable-apmpi-coll-sync
- For a simple helloworld code (MPI + OMP) the application seems to be hanging in the MPI_FINALIZE call.
- if I open the *.darshan_partial file with `darshan-parser`, then the following output is printed:
Error: incompatible darshan file.
Error: expected version 3.21, but got
Error: failed to read darshan log file header.
- There are no issues without APMPI.

3. Further questions:
- Is it possible to switch on/off APMPI during runtime?
- Are there any examples available that demonstrate the additional value that can be achieved by using the new AutoPerf feature?
- Can you confirm that APMPI works on none-Cray systems ?

Best regards,
Thomas

--
Thomas Breuer

Division Application Support             Forschungszentrum Jülich GmbH
Jülich Supercomputing Centre (JSC)       Wilhelm-Johnen-Straße
http://www.fz-juelich.de/ias/jsc         52425 Jülich (Germany)
Phone: +49 2461 61-96742 (currently not available via phone)
Email: t.breuer at fz-juelich.de<mailto:t.breuer at fz-juelich.de>

-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Frauke Melchior
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------