write out about 4GB file with pnetcdf

[Lie-Quan Lee]

I was googling the topic and aslo found the limitation of CDF-2 format.
Thanks for the suggestions. Ideally I would like to store them in just a
one-dimensional array because conceptually it is just one eigenvector.
Practically speaking, I could break it into several pieces when it is
larger than the CDF-2 limit. 

best regards,
Lie-Quan 

On Fri, 2008-05-23 at 14:58 -0500, Wei-keng Liao wrote:
> Hi, Lie-Quan,
> 
> Pnetcdf supports CDF-2 file format to allow the file size bigger than 4GB. 
> However, the array size is still limited to 2G elements and 4 GB size. 
> This is due to the CDF-2 format using signed 4-byte integers to store 
> array dimensionalities in the file header.
> 
> I can see you are defining a one-dimensional array, which can easily reach 
> this limitation. One way to avoid it is to use multi-dimensional arrays, 
> but I don't know if it is feasible for your program.
> 
> 
> Wei-keng
> 
> 
> On Fri, 23 May 2008, Lie-Quan Lee wrote:
> 
> > Dear parallel-netcdf developers,
> > 
> > I work on a large-scale finite-element based simulation and use
> > parallel-netcdf for checkpointing internal status of a simulation.
> > Recently we are doing simulations with about 500 million degrees of
> > freedom. This means I will need to write about at least 500 million
> > doubles into one file during a checkpointing operation. And I got the
> > following error:
> > 
> > 
> > ============
> > Fatal error in MPI_Type_create_subarray: Invalid argument, error stack:
> > MPI_Type_create_subarray(331): MPI_Type_create_subarray(ndims=1,
> > array_of_sizes=0x2355e880, array_of_subsizes=0x2325b7a0,
> > array_of_starts=0x2359b330, order=56, MPI_BYTE, newtype=0x7ffffffdbe54)
> > failed
> > MPI_Type_create_subarray(109): Argument array_of_subsizes has value
> > 976288 but must be within [0,-308901744]
> > =============
> > 
> > I wrote a testing program to just write 500 million doubles using
> > parallel-netcdf (see attachment) and did some debugging, It seems that
> > the variable shape is out of this limit at line 1190 in
> > parallel-netcdf-1.0.2/src/lib/mpinetcdf.c because its type is int, not
> > size_t or long. In the parallel netcdf desgin, the varp->shape is size_t
> > type. However, MPI standard requires the second parameter of the
> > function MPI_Type_create_subarray is an int. Is there any workaround on
> > this problem? Or am I missing something there? 
> > 
> > Thanks very much.
> > 
> > Lie-Quan Lee, Ph.D.,
> > Stanford Linear Accelerator Center
> > 
> > 
> > 
> > 
>