<style>p {margin:0 0 14px 0}.default-font-1748450206900 {font-size: 14px;font-family: 宋体, arial, Verdana, sans-serif}</style><div class="default-font-1748450206900"><style>p {margin:0 0 14px 0}.default-font-1747286136793 {font-size: 14px;font-family: 宋体, arial, Verdana, sans-serif}</style>
<div class="default-font-1747286136793">
<style>p {margin:0 0 14px 0}.default-font-1699420820908 {font-size: 14px;font-family: 宋体, arial, Verdana, sans-serif}</style>
<div class="default-font-1699420820908">
<span style="font-size:14px;text-wrap:nowrap;text-wrap-style:initial;"></span>
<div style="font-size:14px;text-wrap-mode:wrap;">
Hello,
</div>
<div style="font-size:14px;text-wrap-mode:wrap;">
<br>
Recently I create unstructure mesh from Gmsh and its mesh format is msh file. However the mesh file contain around 100 million nodes, so when I use <strong><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/DMPlex/DMPlexCreateFromFile/__;!!G_uCfscf7eWS!b8tabJOQMBPGdL853VyP1yu-DrBuHaOQDcvf368RW41TmzhfdTRagLyrwBnZBeZ6OM3g6neOJBuBS2RCNnBwe35XChWNH5S3bg$" target="_blank" style="color:#000000;">DMPlexCreateFromFile</a> </strong>
</div>
<div style="font-size:14px;text-wrap-mode:wrap;">
it only perform on a single CPU process thus out of memory.
</div>
<div style="font-size:14px;text-wrap-mode:wrap;">
<br>
</div>
<div style="text-wrap-mode:wrap;">
I have sent email to report this situation to your guys and you told me that I can use h5 mesh file. Thanks for your advise so I try to generate h5 file.
</div>
<div style="text-wrap-mode:wrap;">
<br>
</div>
<div style="text-wrap-mode:wrap;">
In order to load the very large msh, I first use a single <span style="text-wrap-mode:wrap;">computer </span>node(64 CPU <span style="text-wrap-mode:wrap;">process</span>) that with large CPU memory. I just use <span style="color:#E53333;text-decoration:underline;">one</span><span style="text-decoration:none;"><span style="color:#E53333;"> </span></span>CPU process to perform the code to generate h5 mesh file:
</div>
<div style="font-size:14px;text-wrap-mode:wrap;">
<p>
<br>
</p>
<p>
-----------------------------------------------------------------------------------------------
</p>
<p style="text-indent:4em;">
//Set up or input the mesh
</p>
<p style="text-indent:4em;">
<span style="text-indent:4em;text-wrap-mode:nowrap;">i</span><span style="text-indent:4em;text-wrap-mode:nowrap;">err = DMPlexCreateFromFile(PETSC_COMM_WORLD, "mesh.msh","hjtest" ,PETSC_TRUE, &dm_nodes);CHKERRQ(ierr);</span>
</p>
<p>
<span style="text-wrap-mode:nowrap;"><br>
</span><span style="text-wrap-mode:nowrap;"><span style="white-space:pre"> </span>//Distribute the mesh into different processers</span><br>
<span style="text-wrap-mode:nowrap;"><span style="white-space:pre"> </span>DM dm_nodes_Dist; //the dm object using to contain the distributed mesh temporarily </span><br>
<span style="text-wrap-mode:nowrap;"><span style="white-space:pre"> </span>PetscPartitioner part; //for distributing the mesh</span><br>
<span style="text-wrap-mode:nowrap;"><span style="white-space:pre"> </span>ierr = DMPlexGetPartitioner(dm_nodes, &part);CHKERRQ(ierr); //Get the partitioner of the mesh we just created</span><br>
<span style="text-wrap-mode:nowrap;"><span style="white-space:pre"> </span>ierr = PetscPartitionerSetType(part,PETSCPARTITIONERPARMETIS);CHKERRQ(ierr); //Set the partitioner from the commond line option</span><br>
<span style="text-wrap-mode:nowrap;"><span style="white-space:pre"> </span>ierr = DMPlexDistribute(dm_nodes,0,NULL,&dm_nodes_Dist); CHKERRQ(ierr); //distribute the mesh into different processers</span><br>
<span style="text-wrap-mode:nowrap;"><span style="white-space:pre"> </span>if (dm_nodes_Dist) {DMDestroy(&dm_nodes); dm_nodes = dm_nodes_Dist;} //delete the origin mesh and use the distributed one</span><br>
<br>
<span style="text-wrap-mode:nowrap;"><br>
</span><br>
<span style="text-wrap-mode:nowrap;"><span style="white-space:pre"> </span>PetscViewerHDF5Open(PETSC_COMM_WORLD, "mesh.h5", FILE_MODE_WRITE, &viewer);</span><br>
<span style="text-wrap-mode:nowrap;"><span style="white-space:pre"> </span>DMView(dm_nodes, viewer);</span><br>
<span style="text-wrap-mode:nowrap;"><span style="white-space:pre"> </span>PetscViewerDestroy(&viewer);</span><br>
<span style="text-wrap-mode:nowrap;"><br>
</span><br>
<span style="text-wrap-mode:nowrap;"><span style="white-space:pre"> </span>DMDestroy(&dm_nodes);</span><br><span style="text-wrap-mode:wrap;">-----------------------------------------------------------------------------------------------</span></p><p><span style="text-wrap-mode:wrap;"><br></span></p><p><span style="text-wrap-mode:wrap;">I get the large h5 mesh, then I want to use another computer(equiped with 20 <span style="text-wrap-mode:wrap;">computer </span><span style="text-wrap-mode:wrap;">node</span> 1280 CPU process that have not so large <span style="text-wrap-mode:wrap;">CPU memory</span>) to perform the parallel computation. So I perform the code that use <span style="text-wrap-mode:wrap;">20 </span><span style="text-wrap-mode:wrap;">computer </span><span style="text-wrap-mode:wrap;">node</span><span style="text-wrap-mode:wrap;"> 1280 CPU process</span>:</span></p><p><span style="text-wrap-mode:wrap;"><br></span></p><p><span style="text-wrap-mode:wrap;">*************************************************************************************************</span></p><p style="text-indent:2em;"><span style="text-wrap-mode:nowrap;text-indent:2em;"></span><span style="text-wrap-mode:nowrap;text-indent:2em;"><span style="white-space:pre"> </span>ierr = PetscViewerHDF5Open(PETSC_COMM_WORLD, "mesh.h5", FILE_MODE_READ, &viewer); CHKERRQ(ierr);</span></p><span style="text-wrap-mode:nowrap;"><span style="white-space:pre"> </span>ierr = PetscViewerSetFormat(viewer, PETSC_VIEWER_HDF5); CHKERRQ(ierr); </span><br> <br><span style="text-wrap-mode:nowrap;"><br></span><br><span style="text-wrap-mode:nowrap;"><span style="white-space:pre"> </span>ierr = DMPlexCreate(PETSC_COMM_WORLD, &dm_nodes); CHKERRQ(ierr);</span><br><span style="text-wrap-mode:nowrap;"><span style="white-space:pre"> </span>ierr = DMSetType(dm_nodes, DMPLEX); CHKERRQ(ierr);</span><br><span style="text-wrap-mode:nowrap;"><br></span><br><span style="text-wrap-mode:nowrap;"><br></span><br><span style="text-wrap-mode:nowrap;"><span style="white-space:pre"> </span>PetscPartitioner part;</span><br><span style="text-wrap-mode:nowrap;"><span style="white-space:pre"> </span>ierr = DMPlexGetPartitioner(dm_nodes, &part); CHKERRQ(ierr);</span><br><span style="text-wrap-mode:nowrap;"><span style="white-space:pre"> </span>ierr = PetscPartitionerSetType(part, PETSCPARTITIONERPARMETIS); CHKERRQ(ierr);</span><br><span style="text-wrap-mode:nowrap;"><br></span><br><span style="text-wrap-mode:nowrap;"><span style="white-space:pre"> </span>ierr = DMLoad(dm_nodes, viewer); CHKERRQ(ierr);</span><br><span style="text-wrap-mode:nowrap;"><span style="white-space:pre"> </span>ierr = PetscViewerDestroy(&viewer); CHKERRQ(ierr);</span><br><span style="text-wrap-mode:nowrap;"><span style="white-space:pre"> </span>PetscPrintf(PETSC_COMM_WORLD,"# Loaded mesh from HDF5\n");</span></div><div style="font-size: 14px;"><span style="text-wrap-mode: nowrap;"><br></span><span style="text-wrap-mode: wrap;">*************************************************************************************************</span><br><p style="text-wrap-mode: wrap;"><br>
</p><p style="text-wrap-mode: wrap;"><br></p><p style="text-wrap-mode: wrap;">I don't know if in this way, the PETSc can load h5 mesh file <span style="text-wrap-mode:nowrap;">parallelly</span> so that it will not <span style="text-wrap-mode:wrap;">only perform on a single CPU process thus out of memory. I try several times but it seems that it always load the h5 file on the one CPU process, I don't know if it have some mistakes or something.</span></p><p style="text-wrap-mode: wrap;"><span style="text-wrap-mode:wrap;"><br></span></p><p style="text-wrap-mode: wrap;"><span style="text-wrap-mode:wrap;">And I use the version of PETSc is 3.21.6 and the configure is attached, I do download the hdf5.</span></p><p style="text-wrap-mode: wrap;"><br></p><p style="text-wrap-mode: wrap;">So I write this email to ask for the help.</p><p style="text-wrap-mode: wrap;"><br>
</p>Looking forward to your reply!</div><div style="font-size: 14px;"><br></div><div style="font-size: 14px;"><span style="text-wrap-style:initial;">sinserely,</span><br style="text-wrap-mode:wrap;"><span style="text-wrap-style:initial;">Cheng.</span><br></div><div style="font-size: 14px;"><br></div><span style="font-size:14px;text-wrap:nowrap;text-wrap-style:initial;"></span><br>
<span style="font-size:16px;"></span>
</div></div></div>