netcdf file size limits (was RE: parallel netcdf ...)

[Rob Ross]

Jianwei,

Quite right.  I feel like we need to come up with some solution to this
problem, even if it means that we have an optional configure-time flag
that breaks with the original netCDF standard.  We would need to print
some nasty warning at build time indicating that this wouldn't be
compatible with existing netCDF applications or something :(.  But I would
guess that some users are really going to find the file limitation
unacceptable, and would rather lose support from original netCDF than have
to split up files.

We should interact with Russ on this and see if he has any ideas on how we 
could most cleanly extend the netCDF header to support larger files, or if 
they have plans to add this sort of support into the existing netCDF 
library.  It would be nice to have a plan that included serial netCDF if 
possible.

All this said, we should save this until we've got these Fortran binding 
issues worked out and some test applications in place!

Rob

On Tue, 29 Jul 2003, Jianwei Li wrote:

> Right.
> But the > 4G filesize is not really solved neither in netCDF3.5 nor
> in PnetCDF0.8.6, because this drawback is implanted in the netCDF
> file format itself, which makes use of byte-based offsets & sizes
> in the file header.
> 
> Jianwei
> 
> On Mon, 28 Jul 2003, Rob Ross wrote:
> 
> > Ok, if that's true then the first call should be hvector() as well.
> > Although given the MPI_BYTE type it won't make a lot of difference except
> > in cases where the offset is bigger than an int (and MPI_Aint is bigger
> > than an int).