<div dir="ltr"><div dir="ltr">On Wed, Nov 24, 2021 at 5:45 PM Bhargav Subramanya <<a href="mailto:bhargav.subramanya@kaust.edu.sa">bhargav.subramanya@kaust.edu.sa</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Wed, Nov 24, 2021 at 8:59 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Wed, Nov 24, 2021 at 12:27 PM Bhargav Subramanya <<a href="mailto:bhargav.subramanya@kaust.edu.sa" target="_blank">bhargav.subramanya@kaust.edu.sa</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Dear Matt and Mark,<div><br></div><div>Thank you very much for your reply. Your inputs are very useful to me.</div><div><br></div></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Nov 22, 2021 at 9:38 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Mon, Nov 22, 2021 at 12:10 PM Bhargav Subramanya <<a href="mailto:bhargav.subramanya@kaust.edu.sa" target="_blank">bhargav.subramanya@kaust.edu.sa</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Dear All,<div><br></div><div>I have a prismatic mesh generated by extruding a base icosahedron spherical surface mesh. The mesh is distributed on 2 processes after extrusion. The dof for my problem are defined only on the simplices of the spherical layers.</div><div><br></div><div>1. For a given spherical layer, I want to get points, which are basically simplices, lying on that layer only. DMPlexGetHeightStratum returns the points on all the spherical Layers. I can probably use DMPlexFindVertices (based on the radius of the spherical layer) and DMPlexGetSimplexOrBoxCells. Could you suggest if there is a better way to retrieve the points (which are simplices) on a given spherical layer in the extruded mesh?</div></div></blockquote><div><br></div><div>DMPlexGetHeightStratum() refers to height in the Hasse Diagram, which is a DAG, not in the mesh. For example, height 0 points are the cells, height 1 are the faces, etc.<br></div><div><br></div><div>I believe the default numbering for extrusion (in the main branch), is that all vertices produced from a given vertex be in order. This would mean that if v were the vertex point number, then</div><div><br></div><div> (v - vStart) % Nl == l</div><div><br></div><div>where Nl is the number of layers and l is the layer of that vertex. It is also the same for triangles. So if you want to segregate each shell, after extrusion, make a label that gives triangles this marker, and then use DMPlexLabelComplete(). Then after refinement you will still have the shells labeled correctly.</div><div><br></div><div>I would be happy to help you make an example that does this. It seems cool.</div></div></div></blockquote><div> </div><div>----- Sure, I am interested to make an example of it. And, I really appreciate your help.<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>2. I am able to visualize the entire mesh using dm_view. Is there a way to get the mesh file for the local dm from a specific process? </div></div></blockquote><div><br></div><div>You can use -dm_partition_view which outputs a field with the process number. Then use Clip inside Paraview and clip to the process number you want,</div><div>or just view that field so each process has a different color.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>3. One of my previous emails might have got lost as I mistakenly attached a large mesh file and sent it. So I am repeating the question here. DMPlexExtrude gives the following error after distributing the base 2D spherical mesh. Both refinement or/and extrusion after dm distribution does not work for me. In fact, I tried with src/dm/impls/plex/tutorials/ex10.c also. Although the mesh is distributed after the extrusion in ex10.c (which is working fine for me), I tried to distribute before extrusion, and I get the following error. Could you please suggest where I might be making a mistake?</div></div></blockquote><div><br></div><div>So you want to distribute the mesh before extruding. For that small example (using the main branch), I run</div><div><br></div><div> mpiexec -n 3 ./meshtest -dm_plex_shape sphere -dm_refine_pre 2 -dm_distribute -dm_partition_view -dm_view hdf5:mesh.h5 -dm_extrude 3</div></div></div></blockquote><div><br></div><div>----- I pulled the code from the main branch and configured my petsc again. I am now finally able to reproduce the mesh that you have attached. I am able to do parallel mesh refinement, followed by extrusion, and use the following code to check that. However, I still have one problem. Attached are the extruded mesh files. </div><div><br></div><div>1. Mesh doesn't seem to be refined properly near interfaces between the processes. I then used -init_dm_distribute_overlap of 1 and that fixed the problem. I hope this is the way to do that. </div></div></div></blockquote><div><br></div><div>Hmm, something is wrong here. We should track this down and fix it. I do not see this. Could you tell me how to reproduce the problem?</div></div></div></blockquote><div><br></div><div>----- I mainly wanted to check if the mesh was distributed or not before the extrusion. I do this by specifying different prefix options as shown in the code below. I think using different prefix options is probably causing the problem.</div><div><br></div><div> #include <petsc.h></div><br>int main(int argc, char **argv)<br>{<br> DM dm;<br> PetscBool distributed;<br> PetscErrorCode ierr;<br><br><br> ierr = PetscInitialize(&argc, &argv, NULL, NULL);if (ierr) return ierr;<br> ierr = DMCreate(PETSC_COMM_WORLD, &dm);CHKERRQ(ierr);<br> ierr = DMSetType(dm, DMPLEX);CHKERRQ(ierr);<br> ierr = PetscObjectSetOptionsPrefix((PetscObject) dm, "init_");CHKERRQ(ierr);<br> ierr = DMSetFromOptions(dm);CHKERRQ(ierr);<br><br> ierr = DMPlexIsDistributed(dm, &distributed);<br> ierr = PetscSynchronizedPrintf(PETSC_COMM_WORLD, "%d \n", distributed); CHKERRQ(ierr);<br> ierr = PetscSynchronizedFlush(PETSC_COMM_WORLD,PETSC_STDOUT);<br><br> ierr = PetscObjectSetOptionsPrefix((PetscObject) dm, NULL);CHKERRQ(ierr);<br> ierr = DMSetFromOptions(dm);CHKERRQ(ierr);<br> ierr = DMViewFromOptions(dm, NULL, "-dm_view");CHKERRQ(ierr);<br> ierr = DMDestroy(&dm);CHKERRQ(ierr);<br> ierr = PetscFinalize();<br> return ierr;</div><div class="gmail_quote">}</div><div class="gmail_quote"><br></div><div class="gmail_quote">The command line options are: mpiexec -n 3 ./cavity_flow.out -init_dm_plex_shape sphere -init_dm_plex_sphere_radius 1.0 -init_dm_refine_pre 2 -init_dm_distribute -dm_refine 2 -dm_extrude 3 -dm_partition_view -dm_view hdf5:mesh.h5 </div></div></blockquote><div><br></div><div>Okay, I run exactly this code and command on 'main', and it does work for me:</div><div><br></div><div>master *:~/Downloads/tmp/hpc1$ /PETSc3/petsc/apple/bin/mpiexec -n 3 ./meshtest -init_dm_plex_shape sphere -init_dm_plex_sphere_radius 1.0 -init_dm_refine_pre 2 -init_dm_distribute -init_dm_view -dm_refine 2 -dm_extrude 3 -dm_partition_view_no -dm_view -malloc_debug 0<br>DM Object: sphere (init_) 3 MPI processes<br> type: plex<br>sphere in 2 dimensions:<br> 0-cells: 66 66 67<br> 1-cells: 171 173 172<br> 2-cells: 106 108 106<br>Labels:<br> depth: 3 strata with value/size (0 (66), 1 (171), 2 (106))<br> celltype: 3 strata with value/size (0 (66), 1 (171), 3 (106))<br>1 <br>1 <br>1 <br>DM Object: sphere 3 MPI processes<br> type: plex<br>sphere in 3 dimensions:<br> 0-cells: 3588 3636 3604<br> 1-cells: 13059 (2691) 13271 (2727) 13087 (2703)<br> 2-cells: 14560 (7776) 14820 (7908) 14572 (7788)<br> 3-cells: 5088 5184 5088<br>Labels:<br> celltype: 6 strata with value/size (5 (7776), 9 (5088), 0 (3588), 1 (10368), 3 (6784), 2 (2691))<br> depth: 4 strata with value/size (0 (3588), 1 (13059), 2 (14560), 3 (5088))<br>WARNING! There are options you set that were not used!<br>WARNING! could be spelling mistake, etc!<br>There is one unused database option. It is:<br>Option left: name:-dm_partition_view_no (no value)<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div>2. The parallel mesh refinement doesn't seem to take care of the spherical geometry (shown in the attached mesh file). Could you please suggest how to fix it?</div></div></div></blockquote><div><br></div><div>Okay, this is more subtle. Let me explain how things work inside.</div><div><br></div><div>1) When you refine, it inserts new cells/faces/edges/vertices. When we insert a new vertex, we ask what the coordinates should be. If no special information is there, we</div><div> just average the surrounding vertices.</div><div><br></div><div>2) Then we allow a second step that processed all coordinates in the mesh after refinement</div><div><br></div><div>For the sphere, we use 2) to make all points stick to the implicit surface (radius R). We cannot do this for the extruded mesh, so we turn it off.</div></div></div></blockquote><div><br></div><div>---- thanks for this explanation. </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div>What makes the most sense to me is to do</div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br></div><div>1) Serial mesh refinement to get enough cells to distribute evenly</div><div><br></div><div>2) Mesh distribution</div><div><br></div><div>3) Mesh refinement</div><div><br></div><div>4) Extrusion</div><div><br></div><div>This should preserve the geometry and scale well. Does that sound right?</div></div></div></blockquote><div> </div><div>-- Yes, this sounds right, and this must preserve the geometry. In fact, I was under the assumption that I was doing the exact same thing by using these options:</div><div>mpiexec -n 3 ./cavity_flow.out -init_dm_plex_shape sphere -init_dm_plex_sphere_radius 1.0 -init_dm_refine_pre 2 -init_dm_distribute -dm_refine 2 -dm_extrude 3 -dm_partition_view -dm_view hdf5:mesh.h5<br></div><div>Do you think these options are not doing the things in the order of 1,2,3 and 4 as you specified? If not, could you suggest the options that I need to use to get in the order of 1,2,3 and 4?</div></div></div></blockquote><div><br></div><div>Yes, I am getting the surface of the sphere with this.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div>command line options:</div><div>-init_dm_plex_shape sphere -init_dm_plex_sphere_radius 1.0 -init_dm_refine_pre 2 -init_dm_distribute -dm_refine 2 -dm_partition_view -dm_view hdf5:mesh.h5 -dm_extrude 3<br></div><div><br></div><div> ierr = DMCreate(PETSC_COMM_WORLD, &dm);CHKERRQ(ierr);<br>ierr = DMSetType(dm, DMPLEX);CHKERRQ(ierr);<br>ierr = PetscObjectSetOptionsPrefix((PetscObject) dm, "init_");CHKERRQ(ierr);<br>ierr = DMSetFromOptions(dm);CHKERRQ(ierr);<br><br>ierr = DMPlexIsDistributed(dm, &distributed);<br>ierr = PetscSynchronizedPrintf(PETSC_COMM_WORLD, "%d \n", distributed); CHKERRQ(ierr);<br>ierr = PetscSynchronizedFlush(PETSC_COMM_WORLD,PETSC_STDOUT);<br><br>ierr = PetscObjectSetOptionsPrefix((PetscObject) dm, NULL);CHKERRQ(ierr);<br>ierr = DMSetFromOptions(dm);CHKERRQ(ierr);<br>ierr = DMViewFromOptions(dm, NULL, "-dm_view");CHKERRQ(ierr);<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div></div><div>and I get the attached picture.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>[0]PETSC ERROR: Object is in wrong state<br>[0]PETSC ERROR: This DMPlex is distributed but its PointSF has no graph set<br>[0]PETSC ERROR: See <a href="https://urldefense.com/v3/__https://petsc.org/release/faq/__;!!Nmw4Hv0!hb2EaePjZqAA4Ns2bFzuHqtjvdZrySmfJcO-7YL5Twmqdh_n0Ym4CneeBI7x1Z5UiaOW2dk_COg$" target="_blank">https://petsc.org/release/faq/</a> for trouble shooting.<br>[0]PETSC ERROR: Petsc Release Version 3.16.1, unknown<br>[0]PETSC ERROR: ./cavity_flow.out on a arch-darwin-c-debug named kl-21859 by subrambm Mon Nov 22 19:47:14 2021<br>[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --download-superlu_dist --download-hypre --download-fiat --download-generator --download-triangle --download-tetgen --download-chaco --download-make -download-boost --download-cmake --download-ml --download-mumps=<a href="https://urldefense.com/v3/__https://bitbucket.org/petsc/pkg-mumps.git__;!!Nmw4Hv0!hb2EaePjZqAA4Ns2bFzuHqtjvdZrySmfJcO-7YL5Twmqdh_n0Ym4CneeBI7x1Z5UiaOW414fjd0$" target="_blank">https://bitbucket.org/petsc/pkg-mumps.git</a> --download-mumps-commit=v5.4.1-p1 --download-scalapack --download-ptscotch --download-hdf5 --force<br>[0]PETSC ERROR: #1 DMPlexCheckPointSF() at /Users/subrambm/petsc/src/dm/impls/plex/plex.c:8554<br>[0]PETSC ERROR: #2 DMPlexOrientInterface_Internal() at /Users/subrambm/petsc/src/dm/impls/plex/plexinterpolate.c:595<br>[0]PETSC ERROR: #3 DMPlexInterpolate() at /Users/subrambm/petsc/src/dm/impls/plex/plexinterpolate.c:1357<br>[0]PETSC ERROR: #4 DMPlexExtrude() at /Users/subrambm/petsc/src/dm/impls/plex/plexcreate.c:1543<br>[0]PETSC ERROR: #5 CreateMesh() at ex10.c:161<br>[0]PETSC ERROR: #6 main() at ex10.c:180<br>[0]PETSC ERROR: PETSc Option Table entries:<br>[0]PETSC ERROR: -dm_plex_extrude_layers 3<br>[0]PETSC ERROR: -dm_view vtk:mesh.vtk<br>[0]PETSC ERROR: -init_dm_plex_dim 2<br>[0]PETSC ERROR: -petscpartitioner_type simple<br>[0]PETSC ERROR: -srf_dm_distribute<br>[0]PETSC ERROR: -srf_dm_refine 0<br>[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint@mcs.anl.gov----------</div><div><br></div><div>Thanks,</div><div>Bhargav</div><div><br></div></div>
<br>
<div><hr></div><font face="Arial" size="1">This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.</font></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!Nmw4Hv0!hb2EaePjZqAA4Ns2bFzuHqtjvdZrySmfJcO-7YL5Twmqdh_n0Ym4CneeBI7x1Z5UiaOWdEg9at8$" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div></div>
<br>
<div><hr></div><font face="Arial" size="1">This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.</font></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!Nmw4Hv0!jtcvFDJFDmYH37N1wm6apnWSZeA9Tu6jsXTTbAqZ7w6gvNxKzNSfHoeVDIUfLDjsDO1vgciyMAw$" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div></div>
<br>
<div><hr></div><font face="Arial" size="1">This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.</font></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>