[Darshan-users] Recording more than 1024 files

[Snyder, Shane]

We do have a work-in-progress PR on our repo that is supposed to help workaround this limit: https://github.com/darshan-hpc/darshan/pull/405

That PR is generally about modifying Darshan to accept different configuration options from users at runtime (currently only using a config file, but ultimately there will be corresponding env variables to control things), including an ability to control how many records are pre-allocated for each module (via MAX_RECORDS configuration setting). I think things work well enough on that branch that you could try requesting more records for the POSIX module, or whatever modules you're interested in, and see how things work. The PR has some details on how to do this.

As Phil mentions, we hope to have this functionality cleaned up for our next release. Hopefully that's something we can have ready by the end of the year or soon thereafter. In the meantime, please let us know if you have any questions about using that branch or if you have feedback on anything.

Thanks,
--Shane

________________________________
From: Darshan-users <darshan-users-bounces at lists.mcs.anl.gov> on behalf of Phil Carns <carns at mcs.anl.gov>
Sent: Monday, November 1, 2021 10:47 AM
To: darshan-users at lists.mcs.anl.gov <darshan-users at lists.mcs.anl.gov>
Subject: Re: [Darshan-users] Recording more than 1024 files


Hi George,


Unfortunately it is hardcoded for the moment (see DARSHAN_DEF_MOD_REC_COUNT) regardless of how much total memory in bytes you allow each module to use.  We are working to have that tunable in an upcoming release.


Another option, depending on your use case, may be to configure Darshan to exclude some directories so that it doesn't use up entries on files that you aren't interested in.  There is some info about that option in this mailing list post:


https://lists.mcs.anl.gov/pipermail/darshan-users/2020-July/000636.html


thanks,

-Phil


On 10/29/21 7:09 PM, George Papadimitriou wrote:
Dear darshan devs,

I ran into a limit when using Darshan 3.3.1, where darshan-runtime records only the first 1024 files.
Is there a way to build Darshan and increase this limit?
I couldn't find something in the documentation. I increased the module memory, but it didn't affect this.

Thanks,
George



_______________________________________________
Darshan-users mailing list
Darshan-users at lists.mcs.anl.gov<mailto:Darshan-users at lists.mcs.anl.gov>
https://lists.mcs.anl.gov/mailman/listinfo/darshan-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/darshan-users/attachments/20211101/40fc58ef/attachment-0001.html>