<div dir="ltr"><div dir="ltr">On Wed, May 28, 2025 at 12:38 PM 程奔 via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:</div><div class="gmail_quote gmail_quote_container"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div>
<div>
<span style="font-size:14px"></span>
<div style="font-size:14px">
Hello,
</div>
<div style="font-size:14px">
<br>
Recently I create unstructure mesh from Gmsh and its mesh format is msh file. However the mesh file contain around 100 million nodes, so when I use <strong><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/DMPlex/DMPlexCreateFromFile/__;!!G_uCfscf7eWS!b8tabJOQMBPGdL853VyP1yu-DrBuHaOQDcvf368RW41TmzhfdTRagLyrwBnZBeZ6OM3g6neOJBuBS2RCNnBwe35XChWNH5S3bg$" style="color:rgb(0,0,0)" target="_blank">DMPlexCreateFromFile</a> </strong>
</div>
<div style="font-size:14px">
it only perform on a single CPU process thus out of memory.
</div>
<div style="font-size:14px">
<br>
</div>
<div>
I have sent email to report this situation to your guys and you told me that I can use h5 mesh file. Thanks for your advise so I try to generate h5 file.
</div>
<div>
<br>
</div>
<div>
In order to load the very large msh, I first use a single <span>computer </span>node(64 CPU <span>process</span>) that with large CPU memory. I just use <span style="color:rgb(229,51,51);text-decoration:underline">one</span><span style="text-decoration:none"><span style="color:rgb(229,51,51)"> </span></span>CPU process to perform the code to generate h5 mesh file:
</div>
<div style="font-size:14px">
<p>
<br>
</p>
<p>
-----------------------------------------------------------------------------------------------
</p>
<p style="text-indent:4em">
//Set up or input the mesh
</p>
<p style="text-indent:4em">
<span style="text-indent:4em">i</span><span style="text-indent:4em">err = DMPlexCreateFromFile(PETSC_COMM_WORLD, "mesh.msh","hjtest" ,PETSC_TRUE, &dm_nodes);CHKERRQ(ierr);</span>
</p>
<p>
<span><br>
</span><span><span style="white-space:pre-wrap"> </span>//Distribute the mesh into different processers</span><br>
<span><span style="white-space:pre-wrap"> </span>DM dm_nodes_Dist; //the dm object using to contain the distributed mesh temporarily </span><br>
<span><span style="white-space:pre-wrap"> </span>PetscPartitioner part; //for distributing the mesh</span><br>
<span><span style="white-space:pre-wrap"> </span>ierr = DMPlexGetPartitioner(dm_nodes, &part);CHKERRQ(ierr); //Get the partitioner of the mesh we just created</span><br>
<span><span style="white-space:pre-wrap"> </span>ierr = PetscPartitionerSetType(part,PETSCPARTITIONERPARMETIS);CHKERRQ(ierr); //Set the partitioner from the commond line option</span><br>
<span><span style="white-space:pre-wrap"> </span>ierr = DMPlexDistribute(dm_nodes,0,NULL,&dm_nodes_Dist); CHKERRQ(ierr); //distribute the mesh into different processers</span><br>
<span><span style="white-space:pre-wrap"> </span>if (dm_nodes_Dist) {DMDestroy(&dm_nodes); dm_nodes = dm_nodes_Dist;} //delete the origin mesh and use the distributed one</span></p></div></div></div></div></blockquote><div>This distribution is unnecessary.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div><div style="font-size:14px"><p><br>
<span><span style="white-space:pre-wrap"> </span>PetscViewerHDF5Open(PETSC_COMM_WORLD, "mesh.h5", FILE_MODE_WRITE, &viewer);</span><br>
<span><span style="white-space:pre-wrap"> </span>DMView(dm_nodes, viewer);</span><br>
<span><span style="white-space:pre-wrap"> </span>PetscViewerDestroy(&viewer);</span></p></div></div></div></div></blockquote><div>It would be easier to use</div><div><br></div><div> PetscCall(DMViewFromOptions(dm_nodes, NULL, "-dm_view"));</div><div><br></div><div>so that we could easily customize things. Here you will need to write the file in an updated format</div><div><br></div><div> -dm_view hdf5:mesh.h5</div><div> -dm_plex_view_hdf5_storage_version 3.1.0</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div><div style="font-size:14px"><p><br>
<span><span style="white-space:pre-wrap"> </span>DMDestroy(&dm_nodes);</span><br><span>-----------------------------------------------------------------------------------------------</span></p><p><span><br></span></p><p><span>I get the large h5 mesh, then I want to use another computer(equiped with 20 <span>computer </span><span>node</span> 1280 CPU process that have not so large <span>CPU memory</span>) to perform the parallel computation. So I perform the code that use <span>20 </span><span>computer </span><span>node</span><span> 1280 CPU process</span>:</span></p><p><span><br></span></p><p><span>*************************************************************************************************</span></p><p style="text-indent:2em"><span style="text-indent:2em"></span><span style="text-indent:2em"><span style="white-space:pre-wrap"> </span>ierr = PetscViewerHDF5Open(PETSC_COMM_WORLD, "mesh.h5", FILE_MODE_READ, &viewer); CHKERRQ(ierr);</span></p><span><span style="white-space:pre-wrap"> </span>ierr = PetscViewerSetFormat(viewer, PETSC_VIEWER_HDF5); CHKERRQ(ierr); </span><br> <br><span><br></span><br><span><span style="white-space:pre-wrap"> </span>ierr = DMPlexCreate(PETSC_COMM_WORLD, &dm_nodes); CHKERRQ(ierr);</span><br><span><span style="white-space:pre-wrap"> </span>ierr = DMSetType(dm_nodes, DMPLEX); CHKERRQ(ierr);</span><br><span><br></span><br><span><br></span><br><span><span style="white-space:pre-wrap"> </span>PetscPartitioner part;</span><br><span><span style="white-space:pre-wrap"> </span>ierr = DMPlexGetPartitioner(dm_nodes, &part); CHKERRQ(ierr);</span><br><span><span style="white-space:pre-wrap"> </span>ierr = PetscPartitionerSetType(part, PETSCPARTITIONERPARMETIS); CHKERRQ(ierr);</span><br><span><br></span><br><span><span style="white-space:pre-wrap"> </span>ierr = DMLoad(dm_nodes, viewer); CHKERRQ(ierr);</span><br><span><span style="white-space:pre-wrap"> </span>ierr = PetscViewerDestroy(&viewer); CHKERRQ(ierr);</span><br><span><span style="white-space:pre-wrap"></span></span></div></div></div></div></blockquote><div><br></div><div>I think it is simpler to just use</div><div><br></div><div> PetscCall(DMCreate(comm, &dm));</div><div> PetscCall(DMSetType(dm, DMPLEX));</div><div> PetscCall(DMSetFromOptions(dm));</div><div><br></div><div>and then use</div><div><br></div><div> -dm_plex_filename mesh.h5</div><div><br></div><div>to load it.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div><div style="font-size:14px"><span><span style="white-space:pre-wrap"> </span>PetscPrintf(PETSC_COMM_WORLD,"# Loaded mesh from HDF5\n");</span></div><div style="font-size:14px"><span><br></span><span>*************************************************************************************************</span><br><p><br>
</p><p><br></p><p>I don't know if in this way, the PETSc can load h5 mesh file <span>parallelly</span> so that it will not <span>only perform on a single CPU process thus out of memory. I try several times but it seems that it always load the h5 file on the one CPU process, I don't know if it have some mistakes or something.</span></p><p><span><br></span></p><p><span>And I use the version of PETSc is 3.21.6 and the configure is attached, I do download the hdf5.</span></p><p><br></p><p>So I write this email to ask for the help.</p><p><br>
</p>Looking forward to your reply!</div><div style="font-size:14px"><br></div><div style="font-size:14px"><span>sinserely,</span><br><span>Cheng.</span><br></div><div style="font-size:14px"><br></div><span style="font-size:14px"></span><br>
<span style="font-size:16px"></span>
</div></div></div></blockquote></div><div><br clear="all"></div><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YotTbyod-NJW43FaXi_Oc9Cqlb1kMpzfZv43sF-b6wDFrCR7USjxhBX_7BAwH2hQ-QuGQXxSBrkUJ6enKDLL$" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>