[mpich-discuss] Performance issue when writing large files with MPI-IO/ROMIO and OrangeFS (PVFS)

[Rob Latham]

On Wed, Mar 14, 2012 at 06:18:34PM +0100, Louis-Claude Canon wrote:
> I am seeing significant variability and low performance when writing
> large files, whereas it is stable with small ones (the threshold
> seems to be around 100 MB) with one server and one client.

> IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 10000000 -b 10000000 
># 10 MB
> Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev
> write          92.06      88.43       91.61      1.06

> IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 200000000 -b 200000000 
> # 200 MB
> write          94.67      35.72       68.07     26.17

Writes are funny operations.  They may or may not get cached.  What
might be happening is that once you've exceeded a certain size, the
PVFS servers will start exhausting cache and have to actually write to
disk.

Doesn't IOR report a bandwith that also includes open/close times?  If
so, I suspect you will get more consistent values that way.

> I am not sure if it is related to OrangeFS or ROMIO. When I use
> pvfs2-cp with a file of size 2GB, the bandwidth is correct and
> stable, which suggests that it comes from ROMIO:
> dd if=/dev/zero of=/tmp/test bs=1000000 count=2000
> pvfs2-cp -t /tmp/test /mnt/pvfs2/test
> Wrote 2000000000 bytes in 19.527506 seconds. 97.674973 MB/seconds
> 
> On the other hand when I enable TroveSyncData or when I put
> TroveMethod to directio with OrangeFS, the variability disappears.

This setting further makes me suspect the server-size VFS cache: both
those settings (in different ways) make the VFS cache irrelevant.

==rob

-- 
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA