[Darshan-users] DASK and darshan

[Razvan Stefanescu]

Hello Kevin,

Very nice feedback. I really appreciate it. Will let you know how it goes.

Razvan

On Tue, Oct 13, 2020 at 5:53 PM Harms, Kevin <harms at alcf.anl.gov> wrote:

> Razvan,
>
>   It's be a couple of years since i used Dask, but you should be able to
> profile by setting LD_PRELOAD=/path/to/darshan/install/lib/libdarshan.so.
>   Please note that Dask will start a process for each python instance so
> you will have a darshan log for each process. A few more caveats, if you're
> using remote processes, I'm not sure if the Dask servers will pass
> environment variables or not. I think Dask has other non-distributed
> schedulers that might work differently if running on a single node.
>
>   Also note, by default Darshan assumes MPI and depends on it. If you're
> applications aren't using MPI, then make sure you configure darshan with
> the switch --without-mpi.
>
> kevin
>
> ________________________________________
> From: Darshan-users <darshan-users-bounces at lists.mcs.anl.gov> on behalf
> of Razvan Stefanescu <razvan.stefanescu at spire.com>
> Sent: Tuesday, October 13, 2020 4:41 PM
> To: darshan-users at lists.mcs.anl.gov
> Subject: [Darshan-users] DASK and darshan
>
> Hello All,
>
> We need to profile some DASK parallel I/O application, and I wonder if you
> can provide an example on how to use darshan to perform such analysis. Your
> help is much appreciated.
>
> Thank you,
>
> Razvan
>
> --
> RAZVAN STEFANESCU
> Head of Statistics and Machine Learning Branch
>
> Senior Data Assimilation and Data Scientist
>
> Spire Global, Inc.
>
> 1050 Walnut Street, Suite 402, Boulder, CO 80302 USA
>
> +1-720-<tel:7206432245>643-2231
>
> +1-850-443-1718<tel:4178491948>
>


-- 
*RAZVAN STEFANESCU *
Head of Statistics and Machine Learning Branch

Senior Data Assimilation and Data Scientist

*Spire Global, Inc.*

1050 Walnut Street, Suite 402, Boulder, CO 80302 USA

+1-720- <7206432245>*643-2231*
+1-850-443-1718 <4178491948>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/darshan-users/attachments/20201014/81a9f3e1/attachment.html>