<div dir="ltr">Hello Shane,<div><br></div><div>This is much appreciated. Since Darshin works with both Spark or Tensorflow I should have success with DASK too. For sure I will keep you updated on my progress.</div><div><br></div><div>Thank you,</div><div><br></div><div>Razvan </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Oct 14, 2020 at 10:13 AM Snyder, Shane <<a href="mailto:ssnyder@mcs.anl.gov">ssnyder@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
You will also have to set the DARSHAN_ENABLE_NONMPI environment variable (i.e., export DARSHAN_ENABLE_NONMPI=1) to ensure logs are generated in the case of non-MPI applications. As long as that variable is set, you'll get Darshan logs for basically any process
running from your shell, so make sure you only set it when you really want instrumentation.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Please let us know how it goes. I've had success getting Darshan working with Spark, and I know other users have had success using with TensorFlow, so hopefully Dask isn't much of a problem.<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
--Shane<br>
</div>
<div id="gmail-m_1805068164046823524appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_1805068164046823524divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Darshan-users <<a href="mailto:darshan-users-bounces@lists.mcs.anl.gov" target="_blank">darshan-users-bounces@lists.mcs.anl.gov</a>> on behalf of Harms, Kevin <<a href="mailto:harms@alcf.anl.gov" target="_blank">harms@alcf.anl.gov</a>><br>
<b>Sent:</b> Tuesday, October 13, 2020 6:53 PM<br>
<b>To:</b> Razvan Stefanescu <<a href="mailto:razvan.stefanescu@spire.com" target="_blank">razvan.stefanescu@spire.com</a>>; <a href="mailto:darshan-users@lists.mcs.anl.gov" target="_blank">darshan-users@lists.mcs.anl.gov</a> <<a href="mailto:darshan-users@lists.mcs.anl.gov" target="_blank">darshan-users@lists.mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [Darshan-users] DASK and darshan</font>
<div> </div>
</div>
<div><font size="2"><span style="font-size:11pt">
<div>Razvan,<br>
<br>
It's be a couple of years since i used Dask, but you should be able to profile by setting LD_PRELOAD=/path/to/darshan/install/lib/libdarshan.so.<br>
Please note that Dask will start a process for each python instance so you will have a darshan log for each process. A few more caveats, if you're using remote processes, I'm not sure if the Dask servers will pass environment variables or not. I think Dask
has other non-distributed schedulers that might work differently if running on a single node.<br>
<br>
Also note, by default Darshan assumes MPI and depends on it. If you're applications aren't using MPI, then make sure you configure darshan with the switch --without-mpi.<br>
<br>
kevin<br>
<br>
________________________________________<br>
From: Darshan-users <<a href="mailto:darshan-users-bounces@lists.mcs.anl.gov" target="_blank">darshan-users-bounces@lists.mcs.anl.gov</a>> on behalf of Razvan Stefanescu <<a href="mailto:razvan.stefanescu@spire.com" target="_blank">razvan.stefanescu@spire.com</a>><br>
Sent: Tuesday, October 13, 2020 4:41 PM<br>
To: <a href="mailto:darshan-users@lists.mcs.anl.gov" target="_blank">darshan-users@lists.mcs.anl.gov</a><br>
Subject: [Darshan-users] DASK and darshan<br>
<br>
Hello All,<br>
<br>
We need to profile some DASK parallel I/O application, and I wonder if you can provide an example on how to use darshan to perform such analysis. Your help is much appreciated.<br>
<br>
Thank you,<br>
<br>
Razvan<br>
<br>
--<br>
RAZVAN STEFANESCU<br>
Head of Statistics and Machine Learning Branch<br>
<br>
Senior Data Assimilation and Data Scientist<br>
<br>
Spire Global, Inc.<br>
<br>
1050 Walnut Street, Suite 402, Boulder, CO 80302 USA<br>
<br>
+1-720-<tel:7206432245>643-2231<br>
<br>
+1-850-443-1718<tel:4178491948><br>
_______________________________________________<br>
Darshan-users mailing list<br>
<a href="mailto:Darshan-users@lists.mcs.anl.gov" target="_blank">Darshan-users@lists.mcs.anl.gov</a><br>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/darshan-users</a><br>
</div>
</span></font></div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr" style="font-size:12.8px"><b style="font-size:12.8px"><font color="#000000">RAZVAN STEFANESCU </font></b><br></div><div dir="ltr"><font color="#000000"><span style="font-size:12.8px">Head of Statistics and Machine Learning Branch </span><br></font></div><div dir="ltr" style="font-size:12.8px"><div dir="ltr"><div style="font-family:Helvetica;font-size:14px;color:rgb(136,136,136)"><small><p style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;margin-top:0px;margin-bottom:0px"><span style="color:rgb(0,0,0);font-size:12.8px">Senior Data Assimilation and Data Scientist</span><b style="font-size:12.8px"><br></b></p><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px"></div><p style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;margin-top:0px;margin-bottom:0px"><b style="font-size:12.8px"><font color="#cc0000">Spire Global, Inc.</font></b><br></p><p style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;margin-top:0px;margin-bottom:0px"><span style="font-size:12.8px">1050 Walnut Street, Suite 402, Boulder, CO 80302 USA</span></p><p style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px;margin-top:0px;margin-bottom:0px"><span style="background-color:rgba(255,255,255,0)"><font color="#0b5394" style="font-size:12.8px"><a href="tel:7206432245" style="color:rgb(17,85,204)" target="_blank">+1-720-</a><u>643-2231</u></font></span></p></small></div><div style="font-family:Helvetica;font-size:14px;color:rgb(136,136,136)"><small><a href="tel:4178491948" style="color:rgb(17,85,204);font-family:arial,sans-serif;font-size:13px" target="_blank">+1-850-443-1718</a></small></div></div></div></div></div></div></div>