[petsc-users] Scaling of the Petsc Binary Viewer

[Thibault Bridel-Bertomeu]

Dear all,

I have been having issues with large Vec (based on DMPLex) and massive MPI
I/O  ... it looks like the data that is written by the Petsc Binary Viewer
is gibberish for large meshes split on a high number of processes. For
instance, I am using a mesh that has around 50 million cells, split on 1024
processors.
The computation seems to run fine, the timestep computed from the data
makes sense so I think internally everything is fine. But when I look at
the solution (one example attached) it's noise - at this point it should
show a bow shock developing on the left near the step.
The piece of code I use is below for the output :

                call DMGetOutputSequenceNumber(dm, save_seqnum,
save_seqval, ierr); CHKERRA(ierr)
                call DMSetOutputSequenceNumber(dm, -1, 0.d0, ierr);
CHKERRA(ierr)
                write(filename,'(A,I8.8,A)') "restart_", stepnum, ".bin"
                call PetscViewerCreate(PETSC_COMM_WORLD, binViewer, ierr);
CHKERRA(ierr)
                call PetscViewerSetType(binViewer, PETSCVIEWERBINARY,
ierr); CHKERRA(ierr)
                call PetscViewerFileSetMode(binViewer, FILE_MODE_WRITE,
ierr); CHKERRA(ierr);
                call PetscViewerBinarySetUseMPIIO(binViewer, PETSC_TRUE,
ierr); CHKERRA(ierr);
                call PetscViewerFileSetName(binViewer, trim(filename),
ierr); CHKERRA(ierr)
                call VecView(X, binViewer, ierr); CHKERRA(ierr)
                call PetscViewerDestroy(binViewer, ierr); CHKERRA(ierr)
                call DMSetOutputSequenceNumber(dm, save_seqnum,
save_seqval, ierr); CHKERRA(ierr)

I do not think there is anything wrong with it but of course I would be
happy to hear your feedback.
Nonetheless my question was : how far have you tested the binary mpi i/o of
a Vec ? Does it make some sense that for a 50 million cell mesh split on
1024 processes, it could somehow fail ?
Or is it my python drawing method that is completely incapable of handling
this dataset ? (paraview displays the same thing though so I'm not sure ...)

Thank you very much for your advice and help !!!

Thibault
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20210707/31b45dea/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: restart_00001000.png
Type: image/png
Size: 20597 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20210707/31b45dea/attachment-0001.png>