<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Razvan,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Just wanted to clarify the note you asked about in your initial email, but I think all Python programs are viewed as dynamically-linked. I.e.:
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
shane@shane-x1-carbon ~ $ ldd `which python`
<div>linux-vdso.so.1 (0x00007ffcadf26000)</div>
<div>libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa3756eb000)</div>
<div>libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa3756c8000)</div>
<div>libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa3756c2000)</div>
<div>libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fa3756bd000)</div>
<div>libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fa3756a1000)</div>
<div>libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa375552000)</div>
<div>/lib64/ld-linux-x86-64.so.2 (0x00007fa375caa000)</div>
<br>
So, I think all of these Python frameworks should be the same in this regard and should work with Dashan non-MPI instrumentation. That caveat in the documentation probably applies better to C applications that have been statically-linked, in which case the
Darshan library has no hooks to interpose itself between the application and I/O libraries.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
--Shane<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Darshan-users <darshan-users-bounces@lists.mcs.anl.gov> on behalf of Razvan Stefanescu <razvan.stefanescu@spire.com><br>
<b>Sent:</b> Sunday, October 18, 2020 4:33 PM<br>
<b>To:</b> Sandra A. Mendez <smendez.fi.unju@gmail.com><br>
<b>Cc:</b> darshan-users@lists.mcs.anl.gov <darshan-users@lists.mcs.anl.gov><br>
<b>Subject:</b> Re: [Darshan-users] profiling DASK application using darshan</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hello Sandra,
<div><br>
</div>
<div>This is really helpful. Thank you so much!</div>
<div><br>
</div>
<div>Razvan</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Sun, Oct 18, 2020 at 1:01 PM Sandra A. Mendez <<a href="mailto:smendez.fi.unju@gmail.com">smendez.fi.unju@gmail.com</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">Hi Razvan,
<div>My python programs were non-MPI for this reason I set up the <span style="color:rgb(85,85,85); letter-spacing:0.2px; white-space:nowrap"><b>DARSHAN_ENABLE_NONMPI</b> variable.</span></div>
<div>For a simple python program in my local machine, I executed as follows:</div>
<blockquote style="margin:0px 0px 0px 40px; border:none; padding:0px">
<div><font face="monospace">export LD_PRELOAD=$PATH_INSTALL/darshan/3.2.1-master/lib64/libdarshan.so</font></div>
<div><font face="monospace">export <b>DARSHAN_ENABLE_NONMPI=1</b></font></div>
<div><font face="monospace">python images-cla.py</font></div>
</blockquote>
<div><br>
</div>
<div>Another example in a cluster (tensorflow benchmark):</div>
<div># I use my Darshan installation so I need to set up LD_PRELOAD in the job submission.</div>
<blockquote style="margin:0px 0px 0px 40px; border:none; padding:0px">
<div><font face="monospace">export LD_PRELOAD=$PATH_INSTALL/darshan-runtime/3.2.1-master/lib/libdarshan.so</font></div>
<div><b><font face="monospace">export DARSHAN_ENABLE_NONMPI=1</font></b></div>
<div><font face="monospace">horovodrun -np $SLURM_NTASKS $HOSTS_FLAG --network-interface ib0 --gloo
<b>python3.7 scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py</b> --model resnet101 --batch_size 64 --variable_update horovod</font></div>
</blockquote>
<div><br>
</div>
<div>Thanks,</div>
<div>Sandra.-</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Sun, 18 Oct 2020 at 20:30, Razvan Stefanescu <<a href="mailto:razvan.stefanescu@spire.com" target="_blank">razvan.stefanescu@spire.com</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">Hello Sandra,
<div><br>
</div>
<div>What command did you actually use for tracing your python code? Did you use something like</div>
<div><br>
</div>
<div>mpiexec -n 4 -env LD_PRELOAD $PATH/libdarshan.so python script.py ?<br>
</div>
<div><br>
</div>
<div>Thank you,</div>
<div><br>
</div>
<div>Razvan</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Sun, Oct 18, 2020 at 7:30 AM Sandra A. Mendez <<a href="mailto:smendez.fi.unju@gmail.com" target="_blank">smendez.fi.unju@gmail.com</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">Hi Razvan,
<div>I have traced a python program by using Darshan. You only need to set up the DARSHAN_ENABLE_NONMPI variable as follows:</div>
<blockquote style="margin:0px 0px 0px 40px; border:none; padding:0px">
<div>export DARSHAN_ENABLE_NONMPI=1</div>
</blockquote>
<div>Before running your application. Only a comment, the maximum number of files to trace by Darshan is 1024 files per parallel task. In case of python applications, Darshan traces all the files opened not only the application files (but also the files opened
by the python libraries). So take care if your application overcomes that limit.</div>
<div>Thanks,</div>
<div>Sandra.-<br>
<div><br>
</div>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Sun, 18 Oct 2020 at 15:09, Razvan Stefanescu <<a href="mailto:razvan.stefanescu@spire.com" target="_blank">razvan.stefanescu@spire.com</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">Hello All,
<div><br>
</div>
<div>Following the documentation note saying that Darshan instrumentation of non-MPI applications is only possible with dynamically-linked applications, it seems I have to convert the multiprocessing and multithreading DASK code to a dynamically-linked code.
I know some folks were able to profile Spark and TensorFlow codes with Darshan, so I wonder if you could provide some suggestions about creating such dynamically-linked code.</div>
<div><br>
</div>
<div>Thank you,</div>
<div><br>
</div>
<div>Razvan</div>
<div>
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr" style="font-size:12.8px"><b style="font-size:12.8px"><font color="#000000">RAZVAN STEFANESCU </font></b><br>
</div>
<div dir="ltr"><font color="#000000"><span style="font-size:12.8px">Head of Statistics and Machine Learning Branch </span><br>
</font></div>
<div dir="ltr" style="font-size:12.8px">
<div dir="ltr">
<div style="font-family:Helvetica; font-size:14px; color:rgb(136,136,136)"><small>
<p style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:12.8px; margin-top:0px; margin-bottom:0px">
<span style="color:rgb(0,0,0); font-size:12.8px">Senior Data Assimilation and Data Scientist</span><b style="font-size:12.8px"><br>
</b></p>
<div style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:12.8px">
</div>
<p style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:12.8px; margin-top:0px; margin-bottom:0px">
<b style="font-size:12.8px"><font color="#cc0000">Spire Global, Inc.</font></b><br>
</p>
<p style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:12.8px; margin-top:0px; margin-bottom:0px">
<span style="font-size:12.8px">1050 Walnut Street, Suite 402, Boulder, CO 80302 USA</span></p>
<p style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:13px; margin-top:0px; margin-bottom:0px">
<span style="background-color:rgba(255,255,255,0)"><font color="#0b5394" style="font-size:12.8px"><a href="tel:7206432245" target="_blank" style="color:rgb(17,85,204)">+1-720-</a><u>643-2231</u></font></span></p>
</small></div>
<div style="font-family:Helvetica; font-size:14px; color:rgb(136,136,136)"><small><a href="tel:4178491948" target="_blank" style="color:rgb(17,85,204); font-family:arial,sans-serif; font-size:13px">+1-850-443-1718</a></small></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Darshan-users mailing list<br>
<a href="mailto:Darshan-users@lists.mcs.anl.gov" target="_blank">Darshan-users@lists.mcs.anl.gov</a><br>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users" rel="noreferrer" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/darshan-users</a><br>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr" style="font-size:12.8px"><b style="font-size:12.8px"><font color="#000000">RAZVAN STEFANESCU </font></b><br>
</div>
<div dir="ltr"><font color="#000000"><span style="font-size:12.8px">Head of Statistics and Machine Learning Branch </span><br>
</font></div>
<div dir="ltr" style="font-size:12.8px">
<div dir="ltr">
<div style="font-family:Helvetica; font-size:14px; color:rgb(136,136,136)"><small>
<p style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:12.8px; margin-top:0px; margin-bottom:0px">
<span style="color:rgb(0,0,0); font-size:12.8px">Senior Data Assimilation and Data Scientist</span><b style="font-size:12.8px"><br>
</b></p>
<div style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:12.8px">
</div>
<p style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:12.8px; margin-top:0px; margin-bottom:0px">
<b style="font-size:12.8px"><font color="#cc0000">Spire Global, Inc.</font></b><br>
</p>
<p style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:12.8px; margin-top:0px; margin-bottom:0px">
<span style="font-size:12.8px">1050 Walnut Street, Suite 402, Boulder, CO 80302 USA</span></p>
<p style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:13px; margin-top:0px; margin-bottom:0px">
<span style="background-color:rgba(255,255,255,0)"><font color="#0b5394" style="font-size:12.8px"><a href="tel:7206432245" target="_blank" style="color:rgb(17,85,204)">+1-720-</a><u>643-2231</u></font></span></p>
</small></div>
<div style="font-family:Helvetica; font-size:14px; color:rgb(136,136,136)"><small><a href="tel:4178491948" target="_blank" style="color:rgb(17,85,204); font-family:arial,sans-serif; font-size:13px">+1-850-443-1718</a></small></div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr" style="font-size:12.8px"><b style="font-size:12.8px"><font color="#000000">RAZVAN STEFANESCU </font></b><br>
</div>
<div dir="ltr"><font color="#000000"><span style="font-size:12.8px">Head of Statistics and Machine Learning Branch </span><br>
</font></div>
<div dir="ltr" style="font-size:12.8px">
<div dir="ltr">
<div style="font-family:Helvetica; font-size:14px; color:rgb(136,136,136)"><small>
<p style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:12.8px; margin-top:0px; margin-bottom:0px">
<span style="color:rgb(0,0,0); font-size:12.8px">Senior Data Assimilation and Data Scientist</span><b style="font-size:12.8px"><br>
</b></p>
<div style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:12.8px">
</div>
<p style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:12.8px; margin-top:0px; margin-bottom:0px">
<b style="font-size:12.8px"><font color="#cc0000">Spire Global, Inc.</font></b><br>
</p>
<p style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:12.8px; margin-top:0px; margin-bottom:0px">
<span style="font-size:12.8px">1050 Walnut Street, Suite 402, Boulder, CO 80302 USA</span></p>
<p style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:13px; margin-top:0px; margin-bottom:0px">
<span style="background-color:rgba(255,255,255,0)"><font color="#0b5394" style="font-size:12.8px"><a href="tel:7206432245" target="_blank" style="color:rgb(17,85,204)">+1-720-</a><u>643-2231</u></font></span></p>
</small></div>
<div style="font-family:Helvetica; font-size:14px; color:rgb(136,136,136)"><small><a href="tel:4178491948" target="_blank" style="color:rgb(17,85,204); font-family:arial,sans-serif; font-size:13px">+1-850-443-1718</a></small></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>