<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Hi Niels,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
I apologize for the delayed response. We don't have a comprehensive solution for merging logs, but there are a few things you can try:</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<ul style="margin-top: 0px; margin-bottom: 0px; list-style-type: disc;" data-editing-info="{"applyListStyleFromLevel":false,"unorderedStyleType":1}">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<div class="elementToProof">The latest Darshan release includes some new command line utilities to aid in summarizing data for a large number of log files. This does
<b>not </b>produce a report in the same form as a Darshan summary, but rather tables that can summarize and rank logs or files in various ways. It may provide the information you are looking for, or at least help you to focus on the most important logs. You
can find information about these tools at <a href="https://www.mcs.anl.gov/research/projects/darshan/docs/pydarshan/usage.html#other-darshan-cli-tools" id="OWA4e597730-46eb-6586-cc40-e4d0d970eb81" class="OWAAutoLink">
https://www.mcs.anl.gov/research/projects/darshan/docs/pydarshan/usage.html#other-darshan-cli-tools</a>.</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<div>If you want a single summary report (as you would get from a single log) you might be able to use the "darshan-merge" command line tool to join multiple logs into a single log. The darshan-merge utility was designed for a different use case, though (one
in which a job terminates abruptly and we want to join fragments of a single log back together), and it depending on the nature of the logs it won't necessarily behave as expected for joining truly independent logs. If it works, though, the resulting unified
log can be analyzed using the usual summary tool.</div>
</li></ul>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Thank you for pointing out this use case though; we have received questions about this before. It would be a great feature to add to Darshan, ideally by generalizing the darshan-merge utility to make sure that it handles more cases.</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
thanks,</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
-Phil</div>
<div id="appendonsend" style="color: inherit;"></div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr style="display: inline-block; width: 98%;">
<div dir="ltr" id="divRplyFwdMsg" style="color: inherit;"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> Darshan-users <darshan-users-bounces@lists.mcs.anl.gov> on behalf of OGER Niels <niels.oger@meteo.fr><br>
<b>Sent:</b> Friday, May 23, 2025 1:56 PM<br>
<b>To:</b> darshan-users@lists.mcs.anl.gov <darshan-users@lists.mcs.anl.gov><br>
<b>Subject:</b> [Darshan-users] Summary report of several darshan logfiles of a non-MPI job ?</span>
<div> </div>
</div>
<div style="visibility: visible !important; opacity: 1 !important; max-width: none !important; max-height: none !important; display: block !important; text-align: left !important; margin: 16px 0px !important; padding: 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: rgb(208, 216, 220); border-top: 4px solid rgb(144, 164, 174) !important; color: inherit;" id="x_pfptBannernqkx6kl" dir="ltr">
<div style="visibility: visible !important; opacity: 1 !important; background-color: rgb(208, 216, 220); max-height: none !important; float: left !important; display: block !important; margin: 0px 0px 1px !important; max-width: 600px !important; color: inherit;" id="x_pfptBannernqkx6kl">
<div style="opacity: 1 !important; max-width: none !important; max-height: none !important; display: block !important; visibility: visible !important; background-color: rgb(208, 216, 220); color: rgb(0, 0, 0); font-family: "Arial", sans-serif !important; font-weight: bold !important; font-size: 14px !important; line-height: 18px !important;" id="x_pfptBannernqkx6kl">
This Message Is From an External Sender</div>
<div style="opacity: 1 !important; max-width: none !important; max-height: none !important; display: block !important; visibility: visible !important; background-color: rgb(208, 216, 220); color: rgb(0, 0, 0); font-weight: normal; font-family: "Arial", sans-serif !important; font-size: 12px !important; line-height: 18px !important; margin-top: 2px !important;" id="x_pfptBannernqkx6kl">
This message came from outside your organization.</div>
</div>
<div style="line-height: 0; display: block; font-size: 0.01px;"> </div>
</div>
<div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">
Hello,<br>
<br>
</div>
<div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">
I'm trying to use Darshan for the first time to get the I/O profil of several kinds of jobs (Fortran and Python).</div>
<div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">
For the Fortran jobs I have no issue, I get 1 darshan logfile per execution and I can build a report with this command python -m darshan summary "log.darshan".</div>
<div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">
For the Python jobs, I use export DARSHAN_ENABLE_NONMPI=1 before running the job.</div>
<div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">
My Python script uses multiprocessing to read several files in parallel. I get one darshan logfile for each sub-process at the end of the execution.</div>
<div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">
I have tried to use " python -m darshan summary " to get a report merging all the logfiles but did not manage to get the right syntax. I am wondering if what I want to do is even possible with Pydarshan.</div>
<div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">
I tried using the "--include_names" argument but it seems to be a filter on the files accessed by the job and not on the log files themselves.</div>
<div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">
Can someone confirm me if it's possible to 'merge' several logfiles in one report or not ?</div>
<div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">
best regards,</div>
<div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">
Niels</div>
<div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">
--</div>
<div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">
----- Météo-France -----<br>
OGER NIELS<br>
DSI/D - Chef de projet Calcul Intensif<br>
niels.oger@meteo.fr<br>
Fixe : +33 561078198</div>
</body>
</html>