[Darshan-users] my pnetcdf changes

[Rob Latham]

On Mon, Jul 19, 2010 at 01:56:59PM -0400, Phil Carns wrote:
> Maybe that's over-engineering, though :)  

Yeah, it probably is...

> We need to think about how
> likely we are to use this for other APIs.  

Well, darshan does provide a working platform for "linker
wrapper"-type profiling.  I had started a very simple-minded library
using the same linker wrapper tricks (no log file, just printfs) but
had some kind of trouble getting it to do the right thing with pnetcdf
symbols.  Instead of figuring things out I branched darshan and had
working stuff -- not just printfs, but a modified lightweight log file
and a parser that understood the new fields -- in about a day.

I'm sure if I started augmenting another library or the entire pnetcdf
library I'd find some design issue since darshan was not meant to be a
general logging platform, but I'm pretty excited about how easy it was
to extend: maybe it can be a general logging platform without much
more work.

> In the short term picture
> there isn't that much reason not to just keep adding netcdf
> counters.  It's not likely to make much difference in the output
> file size (gzip is great at compressing zeroed out counters), so its
> mainly a question of if there is some maintenance value in splitting
> up some things...

The most obvious maintenance issue is that adding to the log file
requires modifications to at least the enums, the human readable tags,
the file format parser, and the statistics collection calls.  I like
very much the trick of sizing enums with a final THIS_MANY_ELEMENTS
member, though.  that's slick.
 
> I'm cc'ing Jason too, because he has done some work in hooking
> darshan up to zoidfs.  Maybe that's another example where it might
> be helpful to be able to add in wrappers and counters without
> perturbing the base file format?

zoidfs reminds me of other end-to-end logging efforts.  Would this
plugin approach work better if each component makes a separate log
file?  It would primarily address when the logs come from a client
application, the I/O forwarder, and a server.  Well, maybe not the
server, unless you have a way to make darshan start/stop/dump logging
on command.

> A side issue (but related) is that I imagine its hard to maintain
> extra counters outside of trunk when Kevin and I keep inserting new
> base counters here and there in the main arrays.

I just noticed the new entries.  Here's hoping that 'svn merge'
doesn't totally crap out on me.... Well, that wasn't so bad.  I
imagine I should merge with trunk pretty often if I'm ever going to
commit this branch back to trunk.

==rob

-- 
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA