[mpich-discuss] Performance issue when writing large files with MPI-IO/ROMIO and OrangeFS (PVFS)

[Louis-Claude Canon]

Dear,

I am seeing significant variability and low performance when writing 
large files, whereas it is stable with small ones (the threshold seems 
to be around 100 MB) with one server and one client.
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 10000000 -b 10000000 # 
10 MB
Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev
write          92.06      88.43       91.61      1.06
IOR -a MPIIO -i 10 -o pvfs2:/mnt/pvfs2/iortest -t 200000000 -b 200000000 
# 200 MB
write          94.67      35.72       68.07     26.17

I am not sure if it is related to OrangeFS or ROMIO. When I use pvfs2-cp 
with a file of size 2GB, the bandwidth is correct and stable, which 
suggests that it comes from ROMIO:
dd if=/dev/zero of=/tmp/test bs=1000000 count=2000
pvfs2-cp -t /tmp/test /mnt/pvfs2/test
Wrote 2000000000 bytes in 19.527506 seconds. 97.674973 MB/seconds

On the other hand when I enable TroveSyncData or when I put TroveMethod 
to directio with OrangeFS, the variability disappears.

I used the following settings with OrangeFS (default settings): 
TroveSyncData no, TroveMethod alt-io. The IO scheduler is cfq and the 
versions are: OrangeFS 2.8.5, MPICH 1.4.1p1, IOR 2.10.3, kernel 
2.6.32-5-amd64 (debian squeeze), ext3 file system and Gigabit Ethernet. 
I can provide the configuration file and additional measures if needed.

Regards,

Louis-Claude Canon