[Darshan-users] DASK and darshan

[Harms, Kevin]

Razvan,

  It's be a couple of years since i used Dask, but you should be able to profile by setting LD_PRELOAD=/path/to/darshan/install/lib/libdarshan.so.
  Please note that Dask will start a process for each python instance so you will have a darshan log for each process. A few more caveats, if you're using remote processes, I'm not sure if the Dask servers will pass environment variables or not. I think Dask has other non-distributed schedulers that might work differently if running on a single node.

  Also note, by default Darshan assumes MPI and depends on it. If you're applications aren't using MPI, then make sure you configure darshan with the switch --without-mpi.

kevin

________________________________________
From: Darshan-users <darshan-users-bounces at lists.mcs.anl.gov> on behalf of Razvan Stefanescu <razvan.stefanescu at spire.com>
Sent: Tuesday, October 13, 2020 4:41 PM
To: darshan-users at lists.mcs.anl.gov
Subject: [Darshan-users] DASK and darshan

Hello All,

We need to profile some DASK parallel I/O application, and I wonder if you can provide an example on how to use darshan to perform such analysis. Your help is much appreciated.

Thank you,

Razvan

--
RAZVAN STEFANESCU
Head of Statistics and Machine Learning Branch

Senior Data Assimilation and Data Scientist

Spire Global, Inc.

1050 Walnut Street, Suite 402, Boulder, CO 80302 USA

+1-720-<tel:7206432245>643-2231

+1-850-443-1718<tel:4178491948>