[Darshan-users] my pnetcdf changes

[Phil Carns]

On 07/19/2010 02:51 PM, Rob Latham wrote:
> On Mon, Jul 19, 2010 at 01:56:59PM -0400, Phil Carns wrote:
>    
>> Maybe that's over-engineering, though :)
>>      
> Yeah, it probably is...
>
>    
>> We need to think about how
>> likely we are to use this for other APIs.
>>      
> Well, darshan does provide a working platform for "linker
> wrapper"-type profiling.  I had started a very simple-minded library
> using the same linker wrapper tricks (no log file, just printfs) but
> had some kind of trouble getting it to do the right thing with pnetcdf
> symbols.  Instead of figuring things out I branched darshan and had
> working stuff -- not just printfs, but a modified lightweight log file
> and a parser that understood the new fields -- in about a day.
>
> I'm sure if I started augmenting another library or the entire pnetcdf
> library I'd find some design issue since darshan was not meant to be a
> general logging platform, but I'm pretty excited about how easy it was
> to extend: maybe it can be a general logging platform without much
> more work.
>
>    
>> In the short term picture
>> there isn't that much reason not to just keep adding netcdf
>> counters.  It's not likely to make much difference in the output
>> file size (gzip is great at compressing zeroed out counters), so its
>> mainly a question of if there is some maintenance value in splitting
>> up some things...
>>      
> The most obvious maintenance issue is that adding to the log file
> requires modifications to at least the enums, the human readable tags,
> the file format parser, and the statistics collection calls.  I like
> very much the trick of sizing enums with a final THIS_MANY_ELEMENTS
> member, though.  that's slick.
>
>    
>> I'm cc'ing Jason too, because he has done some work in hooking
>> darshan up to zoidfs.  Maybe that's another example where it might
>> be helpful to be able to add in wrappers and counters without
>> perturbing the base file format?
>>      
> zoidfs reminds me of other end-to-end logging efforts.  Would this
> plugin approach work better if each component makes a separate log
> file?  It would primarily address when the logs come from a client
> application, the I/O forwarder, and a server.  Well, maybe not the
> server, unless you have a way to make darshan start/stop/dump logging
> on command.
>
>    
>> A side issue (but related) is that I imagine its hard to maintain
>> extra counters outside of trunk when Kevin and I keep inserting new
>> base counters here and there in the main arrays.
>>      
> I just noticed the new entries.  Here's hoping that 'svn merge'
> doesn't totally crap out on me.... Well, that wasn't so bad.  I
> imagine I should merge with trunk pretty often if I'm ever going to
> commit this branch back to trunk.
>    

I can't make up my mind.  Maybe the thing to do is to just write the 
netcdf counters to their own file like you are saying.  You could put 
the counters in a separate array so that they don't interfere with the 
standard counters / file format.

If you go that route we can still always come back later and re-factor 
it with a generic mechanism to tack information onto the main file.

-Phil