Writing large files from pnetcdf on IBM

[John Michalakes]

Hi Robert,

If I just remove the NF_64BIT_OFFSET flag and try to write the file out larger
than 2GB, I get the same corruption from od -x as before.

If I remove the NF_64BIT_OFFSET flag but also remove some fields from the output
so that the file size is less than 2GB, I get:

0000000  4344 4601 0000 0001 0000 000a 0000 0008
0000020  0000 0004 5469 6d65 0000 0000 0000 000a
0000040  4461 7465 5374 724c 656e 0000 0000 0013

In other words, it works okay (as one would expect, since the file isn't too
large any more).

If I restore the NF_64BIT_OFFSET flag but write out the file with all its
fields, so it's > 2GB, I get:

bluevista:/ptmp/michalak/run_hemi 305 > od -x wrfrst_d02_2005-04-02_18_00_00
0000000  0000 0000 0000 0000 0000 0000 0000 0000
*
(all zeros)

If I keep the NF_64BIT_OFFSET flag and write the file that is < 2GB, I get:

0000000  4344 4601 0000 0001 0000 000a 0000 0008
0000020  0000 0004 5469 6d65 0000 0000 0000 000a
0000040  4461 7465 5374 724c 656e 0000 0000 0013

Which is correct, but it's still CDF1, even though the flag is set in the
NFMPI_CREATE.

BTW, if I use serial NetCDF to write the >2GB file with NF_64BIT_OFFSET:

0000000  4344 4602 0000 0000 0000 000a 0000 0008
0000020  0000 0004 5469 6d65 0000 0000 0000 000a
0000040  4461 7465 5374 724c 656e 0000 0000 0013

but it is *very* slow.

Without NF_64BIT_OFFSET, serial NetCDF fails trying to open the file.

John












> -----Original Message-----
> From: Robert Latham [mailto:robl at mcs.anl.gov]
> Sent: Tuesday, November 28, 2006 10:19 AM
> To: John Michalakes
> Cc: parallel-netcdf at mcs.anl.gov
> Subject: Re: Writing large files from pnetcdf on IBM
>
>
> On Tue, Nov 28, 2006 at 09:45:32AM -0700, John Michalakes wrote:
>
> > bluevista:/ptmp/michalak/run_hemi 157 > od -x
> wrfrst_d02_2005-04-02_18_00_00 |
> > head
> > 0000000  4801 6000 0000 0001 0000 0000 0000 0000
> > 0000020  0000 0000 0000 0000 0000 0000 0000 0000
>
> yeah, that file is definitely scrambled somehow
>
> > (For comparison, with a file known to work:
> >
> > bluevista:/ptmp/michalak/run_hemi 158 > od -x
> wrfrst_d01_2005-04-03_00_00_00 |
> > head
> > 0000000  4344 4601 0000 0001 0000 000a 0000 0009
>
> So you can see the 'CDF1' bytes (43 44 46 01) in this file.
>
> > /* The number of bytes in a off_t */
> > #define SIZEOF_OFF_T 8
> >
> > /* The number of bytes in an MPI_Offset */
> > #define SIZEOF_MPI_OFFSET 8
>
> Maybe we have a problem in the way we implement the CDF-2 file format
> on 64 bit systems?   Can you re-run this program without the
> NF_64BIT_OFFSET flag and report what happens?  the CDF-1 file format
> can be bigger than 2 gb, as long as the variables start somewhere in
> the first 2 gb.
>
> ==rob
>
> --
> Rob Latham
> Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
> Argonne National Lab, IL USA                 B29D F333 664A 4280 315B