[Darshan-users] guidance on using --with-mem-align

[David Shrader]

Hello Phil,

Thank you for the information! Having an idea what Darshan uses that 
parameter for is very helpful. We'll see how far we get with setting it 
to 8!

Thanks,
David

On 03/05/2013 03:37 PM, Phil Carns wrote:
> On 03/05/2013 01:04 PM, David Shrader wrote:
>> Hello All,
>>
>> I am new to Darshan and am working on making it available on our 
>> clusters. I am having some trouble figuring out what to give the 
>> --with-mem-align configure parameter. I have read through the 
>> documentation where I see 8 and 16 used. Is this referring only to 
>> the minimum read and write size of physical memory on a node (not 
>> disk, but RAM)? Are there any resources for trying to figure out what 
>> this parameter should be set to for a particular system? My Google 
>> searches have come up with some research topics and programs for 
>> exposing the differences in timing for aligned vs non-aligned writes, 
>> but I'm not sure how to apply this information to what Darshan is 
>> looking for. For example, the recipe for IBM Blue Gene/Q sets it to 
>> 16 partly because "that is the optimal alignment for the network 
>> transport used between compute nodes and I/O nodes in the system." 
>> This statement makes it seem like not only is physical memory 
>> important, but also network transport.
>>
>> Does anyone have any resources or guidance on how to think about this 
>> parameter that they can point me to?
>>
>> Any and all help is appreciated!
>> David
>>
>
> Hi David,
>
> Thanks for trying out Darshan on your clusters, and let us know how it 
> turns out!
>
> Unfortunately there aren't really any great guidelines for setting 
> that parameter.  All it does is allow Darshan to increment a counter 
> (CP_MEM_NOT_ALIGNED) any time a read or write occurs to a memory 
> buffer that doesn't start on an address that is aligned with the value 
> that you specify there.  We made it a configure argument instead of 
> automatically detecting the alignment at run time just in case, as you 
> speculated, there is some other factor affecting memory alignment 
> performance beyond the word size.
>
> We added it mainly out of concern over that 16 byte network transport 
> alignment issue on the Q, but we ended up never really learning 
> anything useful from the memory alignment counters in our logs. 
> Assuming the clusters are 64 bit, I would suggest just setting it to 
> 8.  That way you could at least find out if applications are reading 
> or writing out of non-word-aligned buffers if you want to look at that 
> later.
>
> thanks,
> -Phil
>
>
> _______________________________________________
> Darshan-users mailing list
> Darshan-users at lists.mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users