<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
You will also have to set the DARSHAN_ENABLE_NONMPI environment variable (i.e., export DARSHAN_ENABLE_NONMPI=1) to ensure logs are generated in the case of non-MPI applications. As long as that variable is set, you'll get Darshan logs for basically any process
running from your shell, so make sure you only set it when you really want instrumentation.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Please let us know how it goes. I've had success getting Darshan working with Spark, and I know other users have had success using with TensorFlow, so hopefully Dask isn't much of a problem.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
--Shane<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Darshan-users <darshan-users-bounces@lists.mcs.anl.gov> on behalf of Harms, Kevin <harms@alcf.anl.gov><br>
<b>Sent:</b> Tuesday, October 13, 2020 6:53 PM<br>
<b>To:</b> Razvan Stefanescu <razvan.stefanescu@spire.com>; darshan-users@lists.mcs.anl.gov <darshan-users@lists.mcs.anl.gov><br>
<b>Subject:</b> Re: [Darshan-users] DASK and darshan</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Razvan,<br>
<br>
It's be a couple of years since i used Dask, but you should be able to profile by setting LD_PRELOAD=/path/to/darshan/install/lib/libdarshan.so.<br>
Please note that Dask will start a process for each python instance so you will have a darshan log for each process. A few more caveats, if you're using remote processes, I'm not sure if the Dask servers will pass environment variables or not. I think Dask
has other non-distributed schedulers that might work differently if running on a single node.<br>
<br>
Also note, by default Darshan assumes MPI and depends on it. If you're applications aren't using MPI, then make sure you configure darshan with the switch --without-mpi.<br>
<br>
kevin<br>
<br>
________________________________________<br>
From: Darshan-users <darshan-users-bounces@lists.mcs.anl.gov> on behalf of Razvan Stefanescu <razvan.stefanescu@spire.com><br>
Sent: Tuesday, October 13, 2020 4:41 PM<br>
To: darshan-users@lists.mcs.anl.gov<br>
Subject: [Darshan-users] DASK and darshan<br>
<br>
Hello All,<br>
<br>
We need to profile some DASK parallel I/O application, and I wonder if you can provide an example on how to use darshan to perform such analysis. Your help is much appreciated.<br>
<br>
Thank you,<br>
<br>
Razvan<br>
<br>
--<br>
RAZVAN STEFANESCU<br>
Head of Statistics and Machine Learning Branch<br>
<br>
Senior Data Assimilation and Data Scientist<br>
<br>
Spire Global, Inc.<br>
<br>
1050 Walnut Street, Suite 402, Boulder, CO 80302 USA<br>
<br>
+1-720-<tel:7206432245>643-2231<br>
<br>
+1-850-443-1718<tel:4178491948><br>
_______________________________________________<br>
Darshan-users mailing list<br>
Darshan-users@lists.mcs.anl.gov<br>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users">https://lists.mcs.anl.gov/mailman/listinfo/darshan-users</a><br>
</div>
</span></font></div>
</body>
</html>