[petsc-users] PetscViewer with 64bit
Jed Brown
jed at jedbrown.org
Thu Feb 16 23:39:30 CST 2023
Okay, this works now. I'm pretty sure I tested this long ago with connectivity using Int64 and found that didn't work. That may have been ancient history, but I'm hesitant to revamp to match PetscInt. If doing that, it would require changing the signature of DMPlexGetVTKConnectivity to use PetscInt instead of PetscVTKInt. I'm already underwater and don't have the stamina to test it, but this MR will get you going for problems in which individual parts don't have more than 2B dofs.
https://gitlab.com/petsc/petsc/-/merge_requests/6081/diffs?commit_id=27ba695b8b62ee2bef0e5776c33883276a2a1735
Mike Michell <mi.mike1021 at gmail.com> writes:
> Jed,
>
> It does not work for me even with the reproducer case with the small 2D
> square mesh. Can you run the case that I sent and open the created
> "sol.vtu" file with paraview?
>
> Thanks,
>
>
>> Thanks, Dave.
>>
>> Mike, can you test that this branch works with your large problems? I
>> tested that .vtu works in parallel for small problems, where works = loads
>> correctly in Paraview and VisIt.
>>
>> https://gitlab.com/petsc/petsc/-/merge_requests/6081
>>
>> Dave May <dave.mayhem23 at gmail.com> writes:
>>
>> > On Tue 14. Feb 2023 at 21:27, Jed Brown <jed at jedbrown.org> wrote:
>> >
>> >> Dave May <dave.mayhem23 at gmail.com> writes:
>> >>
>> >> > On Tue 14. Feb 2023 at 17:17, Jed Brown <jed at jedbrown.org> wrote:
>> >> >
>> >> >> Can you share a reproducer? I think I recall the format requiring
>> >> certain
>> >> >> things to be Int32.
>> >> >
>> >> >
>> >> > By default, the byte offset used with the appended data format is
>> >> UInt32. I
>> >> > believe that’s where the sizeof(int) is coming from. This default is
>> >> > annoying as it limits the total size of your appended data to be < 3
>> GB.
>> >> > That said, in the opening of the paraview file you can add this
>> attribute
>> >> >
>> >> > header_type="UInt64"
>> >>
>> >> You mean in the header of the .vtu?
>> >
>> >
>> > Yeah, within the open VTKFile tag.
>> > Like this
>> > < VTKFile type=“xxx”, byte_order="LittleEndian" header_type="UInt64" >
>> >
>> > Do you happen to have an example or pointers to docs describing this
>> >> feature?
>> >
>> >
>> > Example yes - will send it to you tomorrow. Docs… not really. Only stuff
>> > like this
>> >
>> >
>> https://kitware.github.io/paraview-docs/latest/python/paraview.simple.XMLPStructuredGridWriter.html
>> >
>> >
>> >
>> https://kitware.github.io/paraview-docs/v5.8.0/python/paraview.simple.XMLMultiBlockDataWriter.html
>> >
>> > All the writers seem to support it.
>> >
>> >
>> > Can we always do this?
>> >
>> >
>> > Yep!
>> >
>> >
>> > It isn't mentioned in these:
>> >>
>> >> https://vtk.org/wp-content/uploads/2015/04/file-formats.pdf (PDF was
>> >> created in 2003)
>> >>
>> >>
>> https://kitware.github.io/vtk-examples/site/VTKFileFormats/#xml-file-formats
>> >>
>> >
>> > Yes I know. I’ve tied myself in knots for years because the of the
>> > assumption that the offset had to be an int.
>> >
>> > Credit for the discovery goes to Carsten Uphoff. He showed me this.
>> >
>> > Cheers,
>> > Dave
>> >
>> >
>> >
>> >> > then the size of the offset is now UInt64 and now large files can be
>> >> > finally written.
>> >> >
>> >> >
>> >> > Cheers,
>> >> > Dave
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >>
>> >> >> Mike Michell <mi.mike1021 at gmail.com> writes:
>> >> >>
>> >> >> > Thanks for the note.
>> >> >> > I understood that PETSc calculates the offsets for me through
>> >> "boffset"
>> >> >> > variable in plexvtu.c file. Please correct me if it is wrong.
>> >> >> >
>> >> >> > If plexvtu.c has a bug, it could be around "write file header"
>> part in
>> >> >> > which the boffset is also computed. Is this correct? I am not using
>> >> >> complex
>> >> >> > number.
>> >> >> > There are several mixed parts among "Int32, UInt8, PetscInt_FMT,
>> >> >> > PetscInt64_FMT" in writing the header.
>> >> >> >
>> >> >> > Which combination of those flags is correct for 64bit indices? I am
>> >> gonna
>> >> >> > modify plexvtu.c file with "#if defined(PETSC_USE_64BIT_INDICES)"
>> >> >> > statement, but I do not know what is the correct form of the header
>> >> flag
>> >> >> > for 64bit indices.
>> >> >> >
>> >> >> > It is also confusing to me:
>> >> >> > boffset += gpiece[r].ncells * sizeof(PetscInt) + sizeof(int);
>> >> >> > How is sizeof(PetscInt) different from sizeof(int)?
>> >> >> >
>> >> >> > Thanks,
>> >> >> > Mike
>> >> >> >
>> >> >> >
>> >> >> >> On Tue, Feb 14, 2023 at 11:45 AM Mike Michell <
>> mi.mike1021 at gmail.com
>> >> >
>> >> >> >> wrote:
>> >> >> >>
>> >> >> >>> I was trying to modify the header flags from "Int32" to "Int64",
>> but
>> >> >> the
>> >> >> >>> problem was not resolved. Could I get any additional comments?
>> >> >> >>>
>> >> >> >>
>> >> >> >> The calculated offsets are not correct I think.
>> >> >> >>
>> >> >> >> Matt
>> >> >> >>
>> >> >> >>
>> >> >> >>> Thanks,
>> >> >> >>> Mike
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>> Thanks for the comments.
>> >> >> >>>> To be precise on the question, the entire part of the header of
>> the
>> >> >> .vtu
>> >> >> >>>> file is attached:
>> >> >> >>>>
>> >> >> >>>> <?xml version="1.0"?>
>> >> >> >>>> <VTKFile type="UnstructuredGrid" version="0.1"
>> >> >> byte_order="LittleEndian">
>> >> >> >>>> <UnstructuredGrid>
>> >> >> >>>> <Piece NumberOfPoints="4872" NumberOfCells="4000">
>> >> >> >>>> <Points>
>> >> >> >>>> <DataArray type="Float64" Name="Position"
>> >> >> NumberOfComponents="3"
>> >> >> >>>> format="appended" offset="0" />
>> >> >> >>>> </Points>
>> >> >> >>>> <Cells>
>> >> >> >>>> <DataArray type="Int32" Name="connectivity"
>> >> >> >>>> NumberOfComponents="1" format="appended" offset="116932" />
>> >> >> >>>> <DataArray type="Int32" Name="offsets"
>> >> >> >>>> NumberOfComponents="1" format="appended" offset="372936" />
>> >> >> >>>> <DataArray type="UInt8" Name="types"
>> >> >> >>>> NumberOfComponents="1" format="appended" offset="404940" />
>> >> >> >>>> </Cells>
>> >> >> >>>> <CellData>
>> >> >> >>>> <DataArray type="Int32" Name="Rank"
>> NumberOfComponents="1"
>> >> >> >>>> format="appended" offset="408944" />
>> >> >> >>>> </CellData>
>> >> >> >>>> <PointData>
>> >> >> >>>> <DataArray type="Float64"
>> Name="Vec_0x37c89c0_4Field_0.0"
>> >> >> >>>> NumberOfComponents="1" format="appended" offset="424948" />
>> >> >> >>>> </PointData>
>> >> >> >>>> </Piece>
>> >> >> >>>> <Piece NumberOfPoints="4872" NumberOfCells="4000">
>> >> >> >>>> <Points>
>> >> >> >>>> <DataArray type="Float64" Name="Position"
>> >> >> NumberOfComponents="3"
>> >> >> >>>> format="appended" offset="463928" />
>> >> >> >>>> </Points>
>> >> >> >>>> <Cells>
>> >> >> >>>> <DataArray type="Int32" Name="connectivity"
>> >> >> >>>> NumberOfComponents="1" format="appended" offset="580860" />
>> >> >> >>>> <DataArray type="Int32" Name="offsets"
>> >> >> >>>> NumberOfComponents="1" format="appended" offset="836864" />
>> >> >> >>>> <DataArray type="UInt8" Name="types"
>> >> >> >>>> NumberOfComponents="1" format="appended" offset="868868" />
>> >> >> >>>> </Cells>
>> >> >> >>>> <CellData>
>> >> >> >>>> <DataArray type="Int32" Name="Rank"
>> NumberOfComponents="1"
>> >> >> >>>> format="appended" offset="872872" />
>> >> >> >>>> </CellData>
>> >> >> >>>> <PointData>
>> >> >> >>>> <DataArray type="Float64"
>> Name="Vec_0x37c89c0_4Field_0.0"
>> >> >> >>>> NumberOfComponents="1" format="appended" offset="888876" />
>> >> >> >>>> </PointData>
>> >> >> >>>> </Piece>
>> >> >> >>>> </UnstructuredGrid>
>> >> >> >>>> <AppendedData encoding="raw">
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>> Thanks,
>> >> >> >>>> Mike
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>>> On Sun, Feb 12, 2023 at 6:15 PM Mike Michell <
>> >> mi.mike1021 at gmail.com>
>> >> >> >>>>> wrote:
>> >> >> >>>>>
>> >> >> >>>>>> Dear PETSc team,
>> >> >> >>>>>>
>> >> >> >>>>>> I am a user of PETSc with Fortran. My code uses DMPlex to
>> handle
>> >> dm
>> >> >> >>>>>> object. To print out output variable and mesh connectivity, I
>> use
>> >> >> VecView()
>> >> >> >>>>>> by defining PetscSection on that dm and borrow a vector. The
>> type
>> >> >> of the
>> >> >> >>>>>> viewer is set to PETSCVIEWERVTK.
>> >> >> >>>>>>
>> >> >> >>>>>> With 32bit indices, the above work flow has no issue.
>> However, if
>> >> >> >>>>>> PETSc is configured with 64bit indices, my output .vtu file
>> has
>> >> an
>> >> >> error if
>> >> >> >>>>>> I open the file with visualization tools, such as Paraview or
>> >> >> Tecplot,
>> >> >> >>>>>> saying that:
>> >> >> >>>>>> "Cannot read cell connectivity from Cells in piece 0 because
>> the
>> >> >> >>>>>> "offsets" array is not monotonically increasing or starts
>> with a
>> >> >> value
>> >> >> >>>>>> other than 0."
>> >> >> >>>>>>
>> >> >> >>>>>> If I open the .vtu file from terminal, I can see such a line:
>> >> >> >>>>>> ...
>> >> >> >>>>>> <DataArray type="Int32" Name="connectivity"
>> >> NumberOfComponents="1"
>> >> >> >>>>>> format="appended" offset="580860" />
>> >> >> >>>>>> ...
>> >> >> >>>>>>
>> >> >> >>>>>> I expected "DataArray type="Int64", since the PETSc has 64bit
>> >> >> indices.
>> >> >> >>>>>> Could I get recommendations that I need to check to resolve
>> the
>> >> >> issue?
>> >> >> >>>>>>
>> >> >> >>>>>
>> >> >> >>>>> This is probably a bug. We will look at it.
>> >> >> >>>>>
>> >> >> >>>>> Jed, I saw that Int32 is hardcoded in plexvtu.c, but
>> >> sizeof(PetscInt)
>> >> >> >>>>> is used to calculate the offset, which looks inconsistent. Can
>> you
>> >> >> take a
>> >> >> >>>>> look?
>> >> >> >>>>>
>> >> >> >>>>> Thanks,
>> >> >> >>>>>
>> >> >> >>>>> Matt
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>>> Thanks,
>> >> >> >>>>>> Mike
>> >> >> >>>>>>
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> --
>> >> >> >>>>> What most experimenters take for granted before they begin
>> their
>> >> >> >>>>> experiments is infinitely more interesting than any results to
>> >> which
>> >> >> their
>> >> >> >>>>> experiments lead.
>> >> >> >>>>> -- Norbert Wiener
>> >> >> >>>>>
>> >> >> >>>>> https://www.cse.buffalo.edu/~knepley/
>> >> >> >>>>> <http://www.cse.buffalo.edu/~knepley/>
>> >> >> >>>>>
>> >> >> >>>>
>> >> >> >>
>> >> >> >> --
>> >> >> >> What most experimenters take for granted before they begin their
>> >> >> >> experiments is infinitely more interesting than any results to
>> which
>> >> >> their
>> >> >> >> experiments lead.
>> >> >> >> -- Norbert Wiener
>> >> >> >>
>> >> >> >> https://www.cse.buffalo.edu/~knepley/
>> >> >> >> <http://www.cse.buffalo.edu/~knepley/>
>> >> >> >>
>> >> >>
>> >>
>>
More information about the petsc-users
mailing list