write out about 4GB file with pnetcdf

[Lie-Quan Lee]

Dear parallel-netcdf developers,

I work on a large-scale finite-element based simulation and use
parallel-netcdf for checkpointing internal status of a simulation.
Recently we are doing simulations with about 500 million degrees of
freedom. This means I will need to write about at least 500 million
doubles into one file during a checkpointing operation. And I got the
following error:


============
Fatal error in MPI_Type_create_subarray: Invalid argument, error stack:
MPI_Type_create_subarray(331): MPI_Type_create_subarray(ndims=1,
array_of_sizes=0x2355e880, array_of_subsizes=0x2325b7a0,
array_of_starts=0x2359b330, order=56, MPI_BYTE, newtype=0x7ffffffdbe54)
failed
MPI_Type_create_subarray(109): Argument array_of_subsizes has value
976288 but must be within [0,-308901744]
=============

I wrote a testing program to just write 500 million doubles using
parallel-netcdf (see attachment) and did some debugging, It seems that
the variable shape is out of this limit at line 1190 in
parallel-netcdf-1.0.2/src/lib/mpinetcdf.c because its type is int, not
size_t or long. In the parallel netcdf desgin, the varp->shape is size_t
type. However, MPI standard requires the second parameter of the
function MPI_Type_create_subarray is an int. Is there any workaround on
this problem? Or am I missing something there? 

Thanks very much.

Lie-Quan Lee, Ph.D.,
Stanford Linear Accelerator Center



-------------- next part --------------
A non-text attachment was scrubbed...
Name: pncdftest.C
Type: text/x-c++src
Size: 2132 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/parallel-netcdf/attachments/20080523/5d072d2a/attachment.cc>