<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body>
<p>Hi Shane,</p>
<p><br>
</p>
<p>I can confirm that the master branch fixes the reported configure
issue with Intel.</p>
<p><br>
</p>
<p>Thanks for looking into ParaStationMPI as well.</p>
<p><br>
</p>
<p>I would have to look into the script again to check what I have
extracted in addition to the statistics that you already show in
the PDF. I have definitely added a table of all files (especially
the individual files) with a few values from the binary log per
file. <br>
</p>
<p><br>
</p>
<p>Best regards,</p>
<p>Thomas</p>
<p><br>
</p>
<div class="moz-cite-prefix">Am 22.05.2021 um 00:21 schrieb Snyder,
Shane:<br>
</div>
<blockquote type="cite"
cite="mid:DM8PR09MB7174D34FE90CDC443149CD12FE299@DM8PR09MB7174.namprd09.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Hi Thomas,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
For the first issue you mentioned related to Darshan configure
failures for your Intel compiler when building with APMPI
support, I think this is due to a bug in our configure scripts.
It turns out Intel compilers produce warnings rather than errors
for one of our tests, which was leading to unexpected results
that cause APMPI not to be built. We do most of our testing with
GNU compilers, and just didn't catch this unexpected behavior.
It should be fixed now -- if you're interested, you could try
building directly from our main branch in GitHub (<a
href="https://github.com/darshan-hpc/darshan" id="LPlnk"
moz-do-not-send="true">https://github.com/darshan-hpc/darshan</a>)
to confirm, but we will be sure to include this fix in our next
release.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
I will try to reproduce the hang that you reported with
ParaStationMPI and get back to you soon.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
You are correct that APMPI data is not yet included in our PDF
reports, unfortunately. You can only obtain the data as raw text
using darshan-parser, or you can analyze it manually using
PyDarshan (<a href="https://pypi.org/project/darshan/"
id="LPlnk" moz-do-not-send="true">https://pypi.org/project/darshan/</a>).
We are in the process of redesigning our analysis tools to use
PyDarshan, and will try to make sure they include information on
all instrumentation modules, including APMPI, so hopefully this
is more useful in the future. Please let us know if you think
there's some information missing from these reports that you
would like to see, and we can think about how to include it.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Thanks!</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
--Shane<br>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt"
face="Calibri, sans-serif" color="#000000"><b>From:</b>
Darshan-users <a class="moz-txt-link-rfc2396E" href="mailto:darshan-users-bounces@lists.mcs.anl.gov"><darshan-users-bounces@lists.mcs.anl.gov></a>
on behalf of Thomas Breuer <a class="moz-txt-link-rfc2396E" href="mailto:t.breuer@fz-juelich.de"><t.breuer@fz-juelich.de></a><br>
<b>Sent:</b> Wednesday, May 19, 2021 10:33 AM<br>
<b>To:</b> Harms, Kevin <a class="moz-txt-link-rfc2396E" href="mailto:harms@alcf.anl.gov"><harms@alcf.anl.gov></a>;
<a class="moz-txt-link-abbreviated" href="mailto:darshan-users@lists.mcs.anl.gov">darshan-users@lists.mcs.anl.gov</a>
<a class="moz-txt-link-rfc2396E" href="mailto:darshan-users@lists.mcs.anl.gov"><darshan-users@lists.mcs.anl.gov></a><br>
<b>Subject:</b> Re: [Darshan-users] darshan 3.3.0 issues</font>
<div> </div>
</div>
<div>
<p>Hi Kevin,</p>
<p>thanks for the quick reply!<br>
</p>
<p>1. the IntelMPI version I have mentioned is based on MPICH
3.3. I have attached the log (iimpi_error_log.txt).<br>
</p>
<p>2. I have attached the code as well (which just writes a
couple of lines to stdout) (hello_world.c).<br>
Compilation command: <i>mpicxx -fopenmp hello_world.c -o
hello_world.exe<br>
</i>Hint: With OpenMPI/4.1.0rc2 the configuration with APMPI
and execution works. The PDF report seems to be properly
created.</p>
<p>3. If I interpret that correctly the data collected by APMPI
are not shown yet in the PDF report?<i><br>
</i>FYI: A couple of years ago I have written a python script
that extracts the data from the binary log file with
darshan-parser to get the raw data which you use to create the
PDF report. I was able to reproduce the statistics shown in
the PDF and have added a few more tables which helped us to
get a deeper understanding of applications IO at that time.
Since I have not touched this script for a long time it might
not work anymore. That's why it also interest for me to have a
look at what pydarshan is offering.<br>
</p>
<p>Thomas<i><br>
</i></p>
<div class="x_moz-cite-prefix">Am 19.05.2021 um 16:46 schrieb
Harms, Kevin:<br>
</div>
<blockquote type="cite">
<pre class="x_moz-quote-pre">Thomas,
1. Not sure why the Intel MPI is tripping up on the configure check. I'm assuming it is MPI3 based. Can you send us the config.log output from that one? Maybe we can see why the check fails.
2. The partial log indicates the log is incorrect, so those parser errors are expected. I don't know why the finalize hangs. Was this a Fortran hello world example? I'm not familiar with ParaStationMPI but since it is based on MPICH, it should work. Can you send the test code and how you built it? We can try it on a system here.
3. Autoperf can't be disabled at runtime yet. We have a broader plan to add the ability to enable/disable modules during runtime, but not available yet. We have tested AutoPerf with CrayMPI, MPICH3.3 and OpenMPI. The systems we tested on were generic Linux laptop, Cray XC-40 and Nvidia DGX A100. As far as what can be done with APMPI data, we have some python analysis script based on pydarshan.
<a class="x_moz-txt-link-freetext" href="https://xgitlab.cels.anl.gov/AutoPerf/autoperf/-/blob/master/apmpi/util/apmpi-analysis.py" moz-do-not-send="true">https://xgitlab.cels.anl.gov/AutoPerf/autoperf/-/blob/master/apmpi/util/apmpi-analysis.py</a>
The counters are also output by darshan-parser. We are still in the process of building more analysis based on this work.
kevin
________________________________________
From: Darshan-users <a class="x_moz-txt-link-rfc2396E" href="mailto:darshan-users-bounces@lists.mcs.anl.gov" moz-do-not-send="true"><darshan-users-bounces@lists.mcs.anl.gov></a> on behalf of Thomas Breuer <a class="x_moz-txt-link-rfc2396E" href="mailto:t.breuer@fz-juelich.de" moz-do-not-send="true"><t.breuer@fz-juelich.de></a>
Sent: Wednesday, May 19, 2021 6:52 AM
To: <a class="x_moz-txt-link-abbreviated" href="mailto:darshan-users@lists.mcs.anl.gov" moz-do-not-send="true">darshan-users@lists.mcs.anl.gov</a>
Subject: [Darshan-users] darshan 3.3.0 issues
Dear Darshan Team,
I have installed the latest darshan version (3.3.0) for different MPIs on our HPC JUWELS (<a class="x_moz-txt-link-freetext" href="https://apps.fz-juelich.de/jsc/hps/juwels/configuration.html" moz-do-not-send="true">https://apps.fz-juelich.de/jsc/hps/juwels/configuration.html</a>) and would like to report two issues:
1. Intel (19.1.3.304) Compiler with IntelMPI/2019.8.254:
- Configure Step fails for the new APMPI feature:
cd darshan-runtime; ./configure --prefix=/path/to/darshan-runtime/3.3.0-iimpi-2020-APMPI --with-mem-align=8 --with-log-path-by-env=DARSHAN_LOG_P
ATH --with-jobid-env=SLURM_JOBID CC=mpicc --enable-hdf5-mod=$EBROOTHDF5 --enable-apmpi-mod --enable-apmpi-coll-sync
- Error msg: configure: error: APMPI module requires MPI version 3+
- without the new APMPI Options the configure steps ends successfully:
cd darshan-runtime; ./configure --prefix=/p/software/juwels/stages/Devel-2020/software/darshan-runtime/3.3.0-iimpi-2020 --with-mem-align=8 --with-log-path-by-env=DARSHAN_LOG_PATH --with-jobid-env=SLURM_JOBID CC=mpicc --enable-hdf5-mod=$EBROOTHDF5
2. GCC/9.3.0 Compiler with ParaStationMPI/5.4.7-1 (based on MPICH 3.3.2) (<a class="x_moz-txt-link-freetext" href="https://github.com/ParaStation/psmpi/" moz-do-not-send="true">https://github.com/ParaStation/psmpi/</a>):
- darshan-runtime configured with --enable-apmpi-mod --enable-apmpi-coll-sync
- For a simple helloworld code (MPI + OMP) the application seems to be hanging in the MPI_FINALIZE call.
- if I open the *.darshan_partial file with `darshan-parser`, then the following output is printed:
Error: incompatible darshan file.
Error: expected version 3.21, but got
Error: failed to read darshan log file header.
- There are no issues without APMPI.
3. Further questions:
- Is it possible to switch on/off APMPI during runtime?
- Are there any examples available that demonstrate the additional value that can be achieved by using the new AutoPerf feature?
- Can you confirm that APMPI works on none-Cray systems ?
Best regards,
Thomas
--
Thomas Breuer
Division Application Support Forschungszentrum Jülich GmbH
Jülich Supercomputing Centre (JSC) Wilhelm-Johnen-Straße
<a class="x_moz-txt-link-freetext" href="http://www.fz-juelich.de/ias/jsc" moz-do-not-send="true">http://www.fz-juelich.de/ias/jsc</a> 52425 Jülich (Germany)
Phone: +49 2461 61-96742 (currently not available via phone)
Email: <a class="x_moz-txt-link-abbreviated" href="mailto:t.breuer@fz-juelich.de" moz-do-not-send="true">t.breuer@fz-juelich.de</a><a class="x_moz-txt-link-rfc2396E" href="mailto:t.breuer@fz-juelich.de" moz-do-not-send="true"><mailto:t.breuer@fz-juelich.de></a>
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Frauke Melchior
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
</pre>
</blockquote>
<pre class="x_moz-signature" cols="72">--
Thomas Breuer
Division Application Support Forschungszentrum Jülich GmbH
Jülich Supercomputing Centre (JSC) Wilhelm-Johnen-Straße
<a class="x_moz-txt-link-freetext" href="http://www.fz-juelich.de/ias/jsc" moz-do-not-send="true">http://www.fz-juelich.de/ias/jsc</a> 52425 Jülich (Germany)
Phone: +49 2461 61-96742 (currently not available via phone)
Email: <a class="x_moz-txt-link-abbreviated" href="mailto:t.breuer@fz-juelich.de" moz-do-not-send="true">t.breuer@fz-juelich.de</a>
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Frauke Melchior
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------</pre>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Thomas Breuer
Division Application Support Forschungszentrum Jülich GmbH
Jülich Supercomputing Centre (JSC) Wilhelm-Johnen-Straße
<a class="moz-txt-link-freetext" href="http://www.fz-juelich.de/ias/jsc">http://www.fz-juelich.de/ias/jsc</a> 52425 Jülich (Germany)
Phone: +49 2461 61-96742 (currently not available via phone)
Email: <a class="moz-txt-link-abbreviated" href="mailto:t.breuer@fz-juelich.de">t.breuer@fz-juelich.de</a>
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Frauke Melchior
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------</pre>
</body>
</html>