write out about 4GB file with pnetcdf

[Wei-keng Liao]

Hi, Lie-Quan,

Pnetcdf supports CDF-2 file format to allow the file size bigger than 4GB. 
However, the array size is still limited to 2G elements and 4 GB size. 
This is due to the CDF-2 format using signed 4-byte integers to store 
array dimensionalities in the file header.

I can see you are defining a one-dimensional array, which can easily reach 
this limitation. One way to avoid it is to use multi-dimensional arrays, 
but I don't know if it is feasible for your program.


Wei-keng


On Fri, 23 May 2008, Lie-Quan Lee wrote:

> Dear parallel-netcdf developers,
> 
> I work on a large-scale finite-element based simulation and use
> parallel-netcdf for checkpointing internal status of a simulation.
> Recently we are doing simulations with about 500 million degrees of
> freedom. This means I will need to write about at least 500 million
> doubles into one file during a checkpointing operation. And I got the
> following error:
> 
> 
> ============
> Fatal error in MPI_Type_create_subarray: Invalid argument, error stack:
> MPI_Type_create_subarray(331): MPI_Type_create_subarray(ndims=1,
> array_of_sizes=0x2355e880, array_of_subsizes=0x2325b7a0,
> array_of_starts=0x2359b330, order=56, MPI_BYTE, newtype=0x7ffffffdbe54)
> failed
> MPI_Type_create_subarray(109): Argument array_of_subsizes has value
> 976288 but must be within [0,-308901744]
> =============
> 
> I wrote a testing program to just write 500 million doubles using
> parallel-netcdf (see attachment) and did some debugging, It seems that
> the variable shape is out of this limit at line 1190 in
> parallel-netcdf-1.0.2/src/lib/mpinetcdf.c because its type is int, not
> size_t or long. In the parallel netcdf desgin, the varp->shape is size_t
> type. However, MPI standard requires the second parameter of the
> function MPI_Type_create_subarray is an int. Is there any workaround on
> this problem? Or am I missing something there? 
> 
> Thanks very much.
> 
> Lie-Quan Lee, Ph.D.,
> Stanford Linear Accelerator Center
> 
> 
> 
>