pnetcdf and mvapich2 2.2

[Rob Latham]

On 02/03/2017 05:21 PM, Wei-keng Liao wrote:
> Hi, Mark
>
> The Lustre driver in mvapich2 appears to append O_CREAT to the open mode,
> (line 50, in file src/mpi/romio/adio/ad_lustre/ad_lustre_open.c), even
> if the file is open for read-only. This is the root cause of one of the
> error messages you are seeing:
>    "expect error code NC_ENOENT but got NC_ENOTNC"
>
> Attached is a small MPI program to verify such error. Could you please
> give it a try on your mvapich2 build and Lustre?
> Compile command:
>     mpicc test_open_no_such_file.c -o test_open_no_such_file
> Run command:
>     mpiexec -n 1 ./test_open_no_such_file /lustre/path/non-exist-file
>
> If it is indeed an internal issue of mvapich, I can file a bug report to them.
> thanks

This bug affected both Panasas and Lustre.  I wanted to confirm that I 
fixed this in ROMIO's latest:

$ ./test_open_no_such_file  /mnt/lustre/robl/no_such_file
non-exiting file "/mnt/lustre/robl/no_such_file"
MPI error string: File does not exist, error stack:
ADIOI_LUSTRE_OPEN(42): File /mnt/lustre/robl/no_such_file does not exist
Error class = MPI_ERR_NO_SUCH_FILE
[robl at centos6 ~]$ ./test_open_no_such_file  no_such_file
non-exiting file "no_such_file"
MPI error string: File does not exist, error stack:
ADIOI_UFS_OPEN(42): File no_such_file does not exist
Error class = MPI_ERR_NO_SUCH_FILE

If you report this to mvapich, you might want to point them to this 
not-so-recent change: http://git.mpich.org/mpich.git/commit/92f1c69f0de87f93

It might have taken a few commits for me to get this 100% correct.  I 
hope they just take latest ROMIO/MPICH.

==rob


>
> Wei-keng
>
> On Feb 3, 2017, at 12:25 PM, Wei-keng Liao wrote:
>
>> Hi, Mark
>>
>> For running "make check" on Lustre, could you please set the environment
>> variable PNETCDF_HINTS to "nc_header_align_size=512;nc_var_align_size=1"
>> and run "make check" again? I think it should pass make check. Do let me
>> know. These errors only occur for file systems whose striping size is
>> larger than 1. So, ext4 is not affected. I am working on a fix for that
>> test program. Please note this is a bug in the test program. the PnetCDF
>> library itself is intact.
>>
>> When running "make check", I suggest not to set the environment variable
>> PNETCDF_VERBOSE_DEBUG_MODE, as many error checks are designed on
>> purpose. Those debugging messages can easily mask the true errors. That
>> environment variable is designed for testing one program at a time.
>>
>> As for the errors from mvapich2, I do not have access to a machine with
>> infiniband and thus could not give it a try. However, the errors look like
>> a similar issue that has been discovered in OpenMPI recently: fail to
>> return the correct MPI error codes. I will look into the mvapich2 source
>> codes to confirm.
>>
>> Thanks for trying various compilers and reporting the problem !
>>
>> Wei-keng
>>
>