[Darshan-users] Darshan not creating trace file and not complaining

[Wadud Miah]

Hi Shane,

Yes, you are right that I am trying to profile a Fortran code. Thanks for letting me know about the pre-release. I will wait for it once it is released as I don’t have an urgent need for it yet. Thanks for the quick resolution.

Cheers,
Wadud.

From: Darshan-users [mailto:darshan-users-bounces at lists.mcs.anl.gov] On Behalf Of Shane Snyder
Sent: 31 October 2017 16:36
To: darshan-users at lists.mcs.anl.gov
Subject: Re: [Darshan-users] Darshan not creating trace file and not complaining

Hi Wadud,

It looks like you are trying to run one of the NAS benchmark codes, which I believe are Fortran programs, right?

This is due to a known "issue" with OpenMPI2 where the Fortran bindings are calling PMPI_ routines directly, bypassing Darshan's use of the MPI profiling interface (PMPI). Basically, Darshan is unable to intercept any MPI calls, including MPI_Init/Finalize which is where Darshan initializes its data structures and does the generation of the log file, respectively. No error is generated for the same reason: Darshan never initializes itself and doesn't think it should be instrumenting anything. If interested in more details, check out this github issue: https://github.com/open-mpi/ompi/issues/3954

Ignoring the philosophical question of whether the "issue" here lies with the MPI implementation or tools that use the MPI profiling interface (this issue affects all PMPI tools, not just Darshan), we've actually spent some time developing a workaround for this issue recently, so you are in luck!

I have not gone through with making a pre-release version of Darshan with this workaround publicly available, though I really have been meaning to. I will definitely make sure to do that today, and will send an update to this mailing list when its ready so you (and others) are aware. So, you have a couple of options:

1.) Wait for that pre-release to come out later this afternoon, or
2.) Check out the master branch of the Darshan git repo (https://xgitlab.cels.anl.gov/darshan/darshan)

Either should resolve the issue, but be sure to let us know if not.

Thanks,
--Shane
On 10/31/2017 10:51 AM, Wadud Miah wrote:

Hi everyone,



I built Darshan 3.1.4 with OpenMPI 2.0.2 and gcc 7.2.1 using the following configure flags:



./configure --with-mem-align=8 --with-log-path-by-env=DARSHAN_LOGPATH --with-jobid-env=NONE CC=mpicc --prefix=/usr/local/darshan-3.1.4-ompi



I have also attached the config.log file. When I run my application with LD_PRELOAD, Darshan does not create any trace file and neither does it complain:


[miahw at bedale bin]$ export DARSHAN_LOGPATH=.
[miahw at bedale bin]$ LD_PRELOAD=/usr/local/darshan-3.1.4-ompi/lib/libdarshan.so mpirun -n 4 ./bt.A.4.mpi_io_full
[ ... lots of application output ... ]
[miahw at bedale bin]$ ls
bt.A.4  bt.A.4.mpi_io_full  btio.full.out  inputbt.data

As you can see, there is no trace file in the current working directory. Any help will be greatly appreciated.

Thanks in advance,
Wadud.



---------------------------------------

Dr. Wadud Miah

Numerical Algorithms Group






Disclaimer

The Numerical Algorithms Group Ltd is a company registered in England and Wales with company number 1249803. The registered office is: Wilkinson House, Jordan Hill Road, Oxford OX2 8DR, United Kingdom.

This e-mail has been scanned for all viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business.




_______________________________________________

Darshan-users mailing list

Darshan-users at lists.mcs.anl.gov<mailto:Darshan-users at lists.mcs.anl.gov>

https://lists.mcs.anl.gov/mailman/listinfo/darshan-users

Disclaimer

The Numerical Algorithms Group Ltd is a company registered in England and Wales with company number 1249803. The registered office is:
Wilkinson House, Jordan Hill Road, Oxford OX2 8DR, United Kingdom.

This e-mail has been scanned for all viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/darshan-users/attachments/20171031/7c73d0d7/attachment.html>