[Darshan-users] trouble running pyDarshan on laptop

[Robinson, John-Paul]

Just a follow-up to share that the PyPi install worked well.  I had some trouble just updating the darshan module alone but when I created a clean virtual environment and installed with --no-cache-dir it worked and resolved the errors I was seeing with the darshan.DarshanReport().

Thanks.
________________________________
From: Robinson, John-Paul <jpr at uab.edu>
Sent: Tuesday, July 16, 2024 12:40 PM
To: Snyder, Shane <ssnyder at mcs.anl.gov>; darshan-users at lists.mcs.anl.gov <darshan-users at lists.mcs.anl.gov>
Subject: Re: trouble running pyDarshan on laptop

Thanks for the feedback.

I did do the git submodules init and update and see the .h file in the autoperf submodules dir.

I'll try the pypi approach and let you know how it goes.

Thanks.
________________________________
From: Snyder, Shane <ssnyder at mcs.anl.gov>
Sent: Tuesday, July 16, 2024 11:08:21 AM
To: Robinson, John-Paul <jpr at uab.edu>; darshan-users at lists.mcs.anl.gov <darshan-users at lists.mcs.anl.gov>
Subject: Re: trouble running pyDarshan on laptop

You don't often get email from ssnyder at mcs.anl.gov. Learn why this is important<https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!Z5Z5HGdplTxIh1EHh1lGh7mMlEPwfYGCcxpII2u6TaL1TR1r5IUgi2U0oZe807DOgKoN2ICM0Lthk06nZM3tsaQ$ >
Hi,

The AutoPerf modules are actually provided as git submodules in Darshan, so there's an additional step you'll need to run from the repo: git submodule update --init​

FWIW, unless you're modifying PyDarshan code or running on a system that we don't provide wheels for, I'd recommend just installing PyDarshan straight from PyPI rather than trying to install it from source (i.e., just pip install darshan​ in your venv). There are some strange quirks with things like the AutoPerf modules as you've found, but we make sure all of that stuff is properly built inside of the wheels we provide.

Thanks,
--Shane
________________________________
From: Darshan-users <darshan-users-bounces at lists.mcs.anl.gov> on behalf of Robinson, John-Paul <jpr at uab.edu>
Sent: Tuesday, July 16, 2024 10:04 AM
To: darshan-users at lists.mcs.anl.gov <darshan-users at lists.mcs.anl.gov>
Subject: [Darshan-users] trouble running pyDarshan on laptop

Hi I'm trying to use pyDardshan on my laptop to analyze logs from Polaris. I've successfully built and installed darshan-util and the pydarshan from a cloned repo. ./prepare. sh cd darshan-util/ ./configure --prefix ~/. local --enable-pydarshan
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd
Hi

I'm trying to use pyDardshan on my laptop to analyze logs from Polaris.   I've successfully built and installed darshan-util and the pydarshan from a cloned repo.

./prepare.sh
cd darshan-util/
./configure --prefix ~/.local --enable-pydarshan
make
make install

Followed by installing the module into my virtual environment.

cd pydarshon
pip install .

When I  import darshan and then try to run "report = darshan.DarshanReport(filename, read_all=True)" where filename is one of my Polaris darshan logs I get an error message about APMPI (one of the modules used to generate the darshan log).


File ~/projects/presto/venv/lib/python3.10/site-packages/darshan/report.py:753, in DarshanReport.mod_read_all_apmpi_records(self, mod, dtype, warnings)
    749     self.counters[mod] = {}
    751 # fetch records
    752 # fetch header record
--> 753 rec = backend.log_get_apmpi_record(self.log, mod, "HEADER", dtype=dtype)
    754 while rec != None:
    755     self.records[mod].append(rec)

AttributeError: module 'darshan.backend.cffi_backend' has no attribute 'log_get_apmpi_record'

I assume this is because the module wasn't included during the darshan-util build.

I'm trying to build this module into darshan-util and have added the autoperf submodule to the darshan repo.  When I add --enable-apmpi-mod to the configure step above i get an error that it can't find the header.

checking for GNU default python prefix... ${prefix}
checking for GNU default python exec_prefix... ${exec_prefix}
checking for python script directory (pythondir)... ${PYTHON_PREFIX}/lib/python3.10/site-packages
checking for python extension module directory (pyexecdir)... ${PYTHON_EXEC_PREFIX}/lib/python3.10/site-packages
checking for /home/jpr/projects/presto/darshan/darshan-util/../modules/autoperf/apmpi/darshan-apmpi-log-format.h... no
configure: error: The autoperf MPI module is not present

Looking at the config.log, it seems the test is just trying to confirm the include of the header using the same path, which does exist on my laptop.

I'm a bit stuck on how to proceed. Any hints appreciated.

Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/darshan-users/attachments/20240716/1fc285c3/attachment-0001.html>