[Darshan-users] guidance on using --with-mem-align

[Phil Carns]

On 03/05/2013 01:04 PM, David Shrader wrote:
> Hello All,
>
> I am new to Darshan and am working on making it available on our 
> clusters. I am having some trouble figuring out what to give the 
> --with-mem-align configure parameter. I have read through the 
> documentation where I see 8 and 16 used. Is this referring only to the 
> minimum read and write size of physical memory on a node (not disk, 
> but RAM)? Are there any resources for trying to figure out what this 
> parameter should be set to for a particular system? My Google searches 
> have come up with some research topics and programs for exposing the 
> differences in timing for aligned vs non-aligned writes, but I'm not 
> sure how to apply this information to what Darshan is looking for. For 
> example, the recipe for IBM Blue Gene/Q sets it to 16 partly because 
> "that is the optimal alignment for the network transport used between 
> compute nodes and I/O nodes in the system." This statement makes it 
> seem like not only is physical memory important, but also network 
> transport.
>
> Does anyone have any resources or guidance on how to think about this 
> parameter that they can point me to?
>
> Any and all help is appreciated!
> David
>

Hi David,

Thanks for trying out Darshan on your clusters, and let us know how it 
turns out!

Unfortunately there aren't really any great guidelines for setting that 
parameter.  All it does is allow Darshan to increment a counter 
(CP_MEM_NOT_ALIGNED) any time a read or write occurs to a memory buffer 
that doesn't start on an address that is aligned with the value that you 
specify there.  We made it a configure argument instead of automatically 
detecting the alignment at run time just in case, as you speculated, 
there is some other factor affecting memory alignment performance beyond 
the word size.

We added it mainly out of concern over that 16 byte network transport 
alignment issue on the Q, but we ended up never really learning anything 
useful from the memory alignment counters in our logs. Assuming the 
clusters are 64 bit, I would suggest just setting it to 8.  That way you 
could at least find out if applications are reading or writing out of 
non-word-aligned buffers if you want to look at that later.

thanks,
-Phil