[MPICH2-dev] problem with "read-modife-write" in ADIOI_NFS_WriteStrided() function on IPF SLES9 systems

[Ryzhykh, Alexey]

Hi all,

I got the following failure on SLES9 IPF system for one node and 2 processes:

#------------------------------------------------------------------------------------------

# Benchmarking P_Write_Indv 

# #processes = 2 

#------------------------------------------------------------------------------------------

#

#    MODE: NON-AGGREGATE 

#

       #bytes #repetitions  t_min[usec]  t_max[usec]  t_avg[usec]   Mbytes/sec      defects

            0           10        51.19        54.10        52.64         0.00         0.00

            1           10     12967.40     14173.91     13570.65         0.00         0.00

            2           10     13039.40     14255.71     13647.56         0.00         0.00

            4           10     12352.80     13607.41     12980.10         0.00         0.00

            8           10     18121.31     19392.51     18756.91         0.00         0.00

           16           10     18214.11     19414.11     18814.11         0.00         0.00

           32           10     18416.62     19636.30     19026.46         0.00         0.00

           64           10     18043.02     19242.60     18642.81         0.00         0.00

          128           10     18240.69     19533.30     18887.00         0.01         0.00

          256           10     18082.09     19281.48     18681.79         0.01         0.00

          512           10     18573.50     19767.50     19170.50         0.02         0.00

2: read fail fd=10 offset=2048 bufflen=1024 buf=0x20000000049dc010

 read error:: Input/output error

MPI error  497025056 occurred

 7!: error code=497025056

 30!: file write fail 497025056

Other I/O error , error stack:

(unknown)(): Other I/O error

 

I investigated this failure and found that it appears due to call of read () function inside function ADIOI_NFS_WriteStrided() for "read-modify-write", file romio/adio/ad_nfs/ad_nfs_write.c.

Below is the piece of code showing such read() operations:

       /* initial read for the read-modify-write */

        writebuf_off = offset;

        writebuf = (char *) ADIOI_Malloc(max_bufsize);

        writebuf_len = (int)(ADIOI_MIN(max_bufsize,end_offset-writebuf_off+1));

        if (!(fd->atomicity)) ADIOI_WRITE_LOCK(fd, writebuf_off, SEEK_SET, writebuf_len);

        lseek(fd->fd_sys, writebuf_off, SEEK_SET); \

        err = read(fd->fd_sys, writebuf, writebuf_len);

 

This read  sometimes  fails with errno equal to EIO

Description from man:

EIO    I/O error. This will happen for example when the process is in a

              background  process  group,  tries  to read from its controlling

              tty, and either it is ignoring or blocking SIGTTIN or  its  pro-

              cess  group is orphaned.  It may also occur when there is a low-

              level I/O error while reading from a disk or tape.

 

I think this kernel or NFS specific issue because there are no any causes for this error.

And all work fine on other Linux.  

I suggest the following fix for this problem is to replace all "read-modify-write"

        lseek(fd->fd_sys, writebuf_off, SEEK_SET); \

        err = read(fd->fd_sys, writebuf, writebuf_len);

by 

      {               

            int i;      

            for( i=0; i<I_MPI_NREAD_ATTEMPTS; i++) {            

                lseek(fd->fd_sys, writebuf_off, SEEK_SET);      

                err = read(fd->fd_sys, writebuf, writebuf_len); 

                if( (err == 0 ) || (errno != EIO)) break;       

            }           \

        }

 

Attached is the patch for file romio/adio/ad_nfs/ad_nfs_write.c.

 

 

With best regards,

--

Alexey Ryzhykh

Intel Sarov,

Russia

 

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.mcs.anl.gov/mailman/private/mpich2-dev/attachments/20051202/75d1a0d5/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ad_nfs_write.c.diff
Type: application/octet-stream
Size: 1024 bytes
Desc: ad_nfs_write.c.diff
URL: <https://lists.mcs.anl.gov/mailman/private/mpich2-dev/attachments/20051202/75d1a0d5/attachment.obj>