[Darshan-users] Darshan unable to write log files

[Gunter, David O]

I’m a new user to Darshan and I’m hitting a serious problem. Darshan always fails with an “unable to open log file” error message.

I have tried to use a system-installed build of 2.3.0 and I’ve recently downloaded both 2.3.0 and 2.3.1-pre2 and built the runtime myself.  Nothing appears to work. Here are the details.

My application is built with gcc 4.4.7 and openmpi-1.6.5. The application does strict MPI I/O. It was built dynamically so I’m using the LD_PRELOAD method to get info.

I used the same compiler and Open-MPI version to build Darshan. Here is the Darshan runtime config line:

 ./configure --with-mem-align=8 --with-log-path-by-env=DARSHAN_LOGPATH --with-jobid-env=SLURM_JOBID CC=mpicc

It configures, builds and installs without error.

My app writes to a Panasas file system, /scratch/dog/test_prob and I have set
DARSHAN_LOGPATH to /scratch/dog/darshan_logs/

The permissions on the directory are good. My mpi job runs to completion and then I get the error message.

$ mpirun -n 16 ./higrad_driver_noCBE ./params.16pe.in 

Total time for simulation = 69.468311
HiGrad simulation is complete!!
Shutting down MPI environment!
darshan library warning: unable to open log file /scratch/dog/darshan_logs/dog_higrad_driver_noCBE_id501442_2-26-49816-14771330631875741401.darshan_partial: MPI_ERR_IO: input/output error
darshan library warning: unable to write log file /scratch/dog/darshan_logs/dog_higrad_driver_noCBE_id501442_2-26-49816-14771330631875741401.darshan_partial

--
David Gunter
HPC-5: Applications Readiness Team