[Darshan-users] DASK and darshan

[Razvan Stefanescu]

Hello Shane,

This is much appreciated. Since Darshin works with both Spark or Tensorflow
I should have success with DASK too. For sure I will keep you updated on my
progress.

Thank you,

Razvan

On Wed, Oct 14, 2020 at 10:13 AM Snyder, Shane <ssnyder at mcs.anl.gov> wrote:

> You will also have to set the DARSHAN_ENABLE_NONMPI environment variable
> (i.e., export DARSHAN_ENABLE_NONMPI=1) to ensure logs are generated in the
> case of non-MPI applications. As long as that variable is set, you'll get
> Darshan logs for basically any process running from your shell, so make
> sure you only set it when you really want instrumentation.
>
> Please let us know how it goes. I've had success getting Darshan working
> with Spark, and I know other users have had success using with TensorFlow,
> so hopefully Dask isn't much of a problem.
>
> --Shane
> ------------------------------
> *From:* Darshan-users <darshan-users-bounces at lists.mcs.anl.gov> on behalf
> of Harms, Kevin <harms at alcf.anl.gov>
> *Sent:* Tuesday, October 13, 2020 6:53 PM
> *To:* Razvan Stefanescu <razvan.stefanescu at spire.com>;
> darshan-users at lists.mcs.anl.gov <darshan-users at lists.mcs.anl.gov>
> *Subject:* Re: [Darshan-users] DASK and darshan
>
> Razvan,
>
>   It's be a couple of years since i used Dask, but you should be able to
> profile by setting LD_PRELOAD=/path/to/darshan/install/lib/libdarshan.so.
>   Please note that Dask will start a process for each python instance so
> you will have a darshan log for each process. A few more caveats, if you're
> using remote processes, I'm not sure if the Dask servers will pass
> environment variables or not. I think Dask has other non-distributed
> schedulers that might work differently if running on a single node.
>
>   Also note, by default Darshan assumes MPI and depends on it. If you're
> applications aren't using MPI, then make sure you configure darshan with
> the switch --without-mpi.
>
> kevin
>
> ________________________________________
> From: Darshan-users <darshan-users-bounces at lists.mcs.anl.gov> on behalf
> of Razvan Stefanescu <razvan.stefanescu at spire.com>
> Sent: Tuesday, October 13, 2020 4:41 PM
> To: darshan-users at lists.mcs.anl.gov
> Subject: [Darshan-users] DASK and darshan
>
> Hello All,
>
> We need to profile some DASK parallel I/O application, and I wonder if you
> can provide an example on how to use darshan to perform such analysis. Your
> help is much appreciated.
>
> Thank you,
>
> Razvan
>
> --
> RAZVAN STEFANESCU
> Head of Statistics and Machine Learning Branch
>
> Senior Data Assimilation and Data Scientist
>
> Spire Global, Inc.
>
> 1050 Walnut Street, Suite 402, Boulder, CO 80302 USA
>
> +1-720-<tel:7206432245>643-2231
>
> +1-850-443-1718<tel:4178491948>
> _______________________________________________
> Darshan-users mailing list
> Darshan-users at lists.mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
>


-- 
*RAZVAN STEFANESCU *
Head of Statistics and Machine Learning Branch

Senior Data Assimilation and Data Scientist

*Spire Global, Inc.*

1050 Walnut Street, Suite 402, Boulder, CO 80302 USA

+1-720- <7206432245>*643-2231*
+1-850-443-1718 <4178491948>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/darshan-users/attachments/20201014/7598609c/attachment.html>