<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Thank you very much ! <div class=""><br class=""></div><div class="">I have succeed to distribute the mesh by this using this routine. </div><div class=""><br class=""></div><div class="">PetscDM dmdist = NULL; </div><div class="">PetscPartitioner part; </div><div class="">DMPlexGetPartitioner(dm_mesh, &part);</div><div class=""> PetscPartitionerSetType(part, PETSCPARTITIONERPARMETIS); </div><div class="">PetscPartitionerSetFromOptions(part); </div><div class="">DMPlexDistribute(dm_mesh, 0, NULL, &dmdist);</div><div class=""> if (dmdist)</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span> { </div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>DMDestroy(&dm_mesh); </div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>dm_mesh = dmdist; </div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>}</div><div class="">DMView(dm_mesh, PETSC_VIEWER_STDOUT_WORLD);</div><div class=""><br class=""></div><div class="">Th result is given as below:</div><div class=""><div class="">DM Object: Parallel Mesh 4 MPI processes</div><div class=""> type: plex</div><div class="">Parallel Mesh in 3 dimensions:</div><div class=""> 0-cells: 12 12 12 12</div><div class=""> 3-cells: 2 2 2 2</div><div class="">Labels:</div><div class=""> depth: 2 strata with value/size (0 (12), 1 (2))</div><div class=""> material-id: 1 strata with value/size (68 (2))</div></div><div class=""><br class=""></div><div class="">Thank you a lot for your help and have a good day,</div><div class=""><br class=""></div><div class="">Sami</div><div class=""><br class=""><div class="">
<meta charset="UTF-8" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">--</div><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Dr. Sami BEN ELHAJ SALAH<br class="">Ingénieur de Recherche (CNRS)<br class="">Institut Pprime - ISAE - ENSMA<br class="">Mobile: 06.62.51.26.74<br class=""><a href="mailto:sami.ben-elhaj-salah@ensma.fr" class="">Email: sami.ben-elhaj-salah@ensma.fr</a></div><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><a href="https://samiben91.github.io/samibenelhajsalah/index.html" class="">www.samibenelhajsalah.com</a><br class=""><br class=""><br class=""></div></div></div></div>
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">Le 31 mai 2022 à 18:20, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> a écrit :</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div dir="ltr" class="">On Tue, May 31, 2022 at 12:08 PM Sami BEN ELHAJ SALAH <<a href="mailto:sami.ben-elhaj-salah@ensma.fr" class="">sami.ben-elhaj-salah@ensma.fr</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="overflow-wrap: break-word;" class=""><div class=""><div class="">Hi Matthew,</div><div class=""><br class=""></div><div class=""><u class=""><b class="">Two question in this mail:</b></u></div><div class=""><u class=""><b class=""><br class=""></b></u></div><div class="">I tested my code<span class="Apple-converted-space"> </span><font color="#0433ff" class=""><b class="">with 2 processes</b></font>. I use gmsh mesh( example: 8 hexahedral elements, 27 nodes, and Global Size of my jacobian matrix = 81 ). Then I used DMView(dm_mesh, PETSC_VIEWER_STDOUT_WORLD) to visualize my DM and see if it is correctly distributed over processes, so I got this:</div><div class=""><br class=""></div><div class=""><font color="#ff2600" size="4" class=""><b class="">(***)</b></font></div><div class=""><b class="">DM Object: DM_0x3_0 2 MPI processes</b></div><div class=""><b class=""> <span class="Apple-converted-space"> </span>type: plex</b></div><div class=""><b class="">DM_0x3_0 in 3 dimensions:</b></div><div class=""><b class=""> <span class="Apple-converted-space"> </span>0-cells: 27 0</b></div><div class=""><b class=""> <span class="Apple-converted-space"> </span>3-cells: 8 0</b></div></div></div></blockquote><div class=""><br class=""></div><div class="">Notice here that your whole mesh is on process 0. You would probably call DMPlexDistribute() to rebalance it.</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="overflow-wrap: break-word;" class=""><div class=""><div class=""><b class="">Labels:</b></div><div class=""><b class=""> <span class="Apple-converted-space"> </span>material-id: 3 strata with value/size (66 (4), 67 (2), 68 (2))</b></div><div class=""><b class=""> <span class="Apple-converted-space"> </span>depth: 2 strata with value/size (0 (27), 1 (8))</b></div><div class=""><b class=""> <span class="Apple-converted-space"> </span>celltype: 2 strata with value/size (7 (8), 0 (27))</b></div><div class=""><br class=""></div><div class="">As you see, I created some material-id labels, so I found them over my 8 cells when using DMView. So it seems good to me. The SNES viewer is shown below:</div><div class=""><br class=""></div><div class="">SNES Object: 2 MPI processes</div><div class=""> <span class="Apple-converted-space"> </span>type: ksponly</div><div class=""> <span class="Apple-converted-space"> </span>SNES has not been set up so information may be incomplete</div><div class=""> <span class="Apple-converted-space"> </span>maximum iterations=50, maximum function evaluations=10000</div><div class=""> <span class="Apple-converted-space"> </span>tolerances: relative=1e-08, absolute=1e-50, solution=1e-08</div><div class=""> <span class="Apple-converted-space"> </span>total number of linear solver iterations=0</div><div class=""> <span class="Apple-converted-space"> </span>total number of function evaluations=0</div><div class=""> <span class="Apple-converted-space"> </span>norm schedule ALWAYS</div><div class="">SNES Object: 2 MPI processes</div><div class=""> <span class="Apple-converted-space"> </span>type: ksponly</div><div class=""> <span class="Apple-converted-space"> </span>SNES has not been set up so information may be incomplete</div><div class=""> <span class="Apple-converted-space"> </span>maximum iterations=50, maximum function evaluations=10000</div><div class=""> <span class="Apple-converted-space"> </span>tolerances: relative=1e-08, absolute=1e-50, solution=1e-08</div><div class=""> <span class="Apple-converted-space"> </span>total number of linear solver iterations=0</div><div class=""> <span class="Apple-converted-space"> </span>total number of function evaluations=0</div><div class=""> <span class="Apple-converted-space"> </span>norm schedule ALWAYS</div><div class=""> <span class="Apple-converted-space"> </span>SNESLineSearch Object: 2 MPI processes</div><div class=""> <span class="Apple-converted-space"> </span>type: bt</div><div class=""> <span class="Apple-converted-space"> </span>interpolation: cubic</div><div class=""> <span class="Apple-converted-space"> </span>alpha=1.000000e-04</div><div class=""> <span class="Apple-converted-space"> </span>SNESLineSearch Object: 2 MPI processes</div><div class=""> <span class="Apple-converted-space"> </span>type: bt</div><div class=""> <span class="Apple-converted-space"> </span>interpolation: cubic</div><div class=""> <span class="Apple-converted-space"> </span>alpha=1.000000e-04</div><div class=""> <span class="Apple-converted-space"> </span>maxstep=1.000000e+08, minlambda=1.000000e-12</div><div class=""> <span class="Apple-converted-space"> </span>tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08</div><div class=""> <span class="Apple-converted-space"> </span>maximum iterations=40</div><div class=""> <span class="Apple-converted-space"> </span>maxstep=1.000000e+08, minlambda=1.000000e-12</div><div class=""> <span class="Apple-converted-space"> </span>tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08</div><div class=""> <span class="Apple-converted-space"> </span>maximum iterations=40</div><div class=""> <span class="Apple-converted-space"> </span>KSP Object: 2 MPI processes</div><div class=""> <span class="Apple-converted-space"> </span>type: gmres</div><div class=""> <span class="Apple-converted-space"> </span>restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement</div><div class=""> <span class="Apple-converted-space"> </span>KSP Object: 2 MPI processes</div><div class=""> <span class="Apple-converted-space"> </span>type: gmres</div><div class=""> <span class="Apple-converted-space"> </span>restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement</div><div class=""> <span class="Apple-converted-space"> </span>happy breakdown tolerance 1e-30</div><div class=""> <span class="Apple-converted-space"> </span>maximum iterations=10000, initial guess is zero</div><div class=""> <span class="Apple-converted-space"> </span>tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</div><div class=""> <span class="Apple-converted-space"> </span>left preconditioning</div><div class=""> <span class="Apple-converted-space"> </span>using DEFAULT norm type for convergence test</div><div class=""> <span class="Apple-converted-space"> </span>PC Object: 2 MPI processes</div><div class=""> <span class="Apple-converted-space"> </span>happy breakdown tolerance 1e-30</div><div class=""> <span class="Apple-converted-space"> </span>maximum iterations=10000, initial guess is zero</div><div class=""> <span class="Apple-converted-space"> </span>tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</div><div class=""> <span class="Apple-converted-space"> </span>left preconditioning</div><div class=""> <span class="Apple-converted-space"> </span>using DEFAULT norm type for convergence test</div><div class=""> <span class="Apple-converted-space"> </span>PC Object: 2 MPI processes</div><div class=""> <span class="Apple-converted-space"> </span>type: bjacobi</div><div class=""> <span class="Apple-converted-space"> </span>PC has not been set up so information may be incomplete</div><div class=""> <span class="Apple-converted-space"> </span>number of blocks = -1</div><div class=""> <span class="Apple-converted-space"> </span>Local solve is same for all blocks, in the following KSP and PC objects:</div><div class=""> <span class="Apple-converted-space"> </span>linear system matrix = precond matrix:</div><div class=""> <span class="Apple-converted-space"> </span>type: bjacobi</div><div class=""> <span class="Apple-converted-space"> </span>PC has not been set up so information may be incomplete</div><div class=""> <span class="Apple-converted-space"> </span>number of blocks = -1</div><div class=""> <span class="Apple-converted-space"> </span>Local solve is same for all blocks, in the following KSP and PC objects:</div><div class=""> <span class="Apple-converted-space"> </span>linear system matrix = precond matrix:</div><div class=""> <span class="Apple-converted-space"> </span>Mat Object: 2 MPI processes</div><div class=""> <span class="Apple-converted-space"> </span>type: mpiaij</div><div class=""> <span class="Apple-converted-space"> </span>Mat Object: 2 MPI processes</div><div class=""> <span class="Apple-converted-space"> </span>type: mpiaij</div><div class=""> <span class="Apple-converted-space"> </span>rows=81, cols=81, bs=3</div><div class=""> <span class="Apple-converted-space"> </span>rows=81, cols=81, bs=3</div><div class=""> <span class="Apple-converted-space"> </span>total: nonzeros=3087, allocated nonzeros=3087</div><div class=""> <span class="Apple-converted-space"> </span>total number of mallocs used during MatSetValues calls=0</div><div class=""> <span class="Apple-converted-space"> </span>total: nonzeros=3087, allocated nonzeros=3087</div><div class=""> <span class="Apple-converted-space"> </span>total number of mallocs used during MatSetValues calls=0</div><div class=""> <span class="Apple-converted-space"> </span>using I-node (on process 0) routines: found 27 nodes, limit used is 5</div><div class=""> <span class="Apple-converted-space"> </span>not using I-node (on process 0) routines</div><div class=""><br class=""></div><div class=""><b class=""><font color="#ff2600" class="">Question 1:</font></b></div><div class=""><font color="#0433ff" class=""><b class="">B</b><span class=""><b class="">ased on the result given by DMView (see </b></span></font><b style="color: rgb(255, 38, 0);" class="">(***)</b><font color="#0433ff" class=""><b class=""> </b></font><b style="color: rgb(4, 51, 255);" class="">), </b><b style="color: rgb(4, 51, 255);" class="">I didn't understand if my mesh is correctly distributed </b><b style="color: rgb(4, 51, 255);" class="">? or my code is missing something ? because when I visualize the local and global sizes of my Jacobian matrix, I found</b></div><div class=""><br class=""></div><div class=""><b class="">PETSc::NonLinearSolver::INIT Size from Jac Matrix: M=81 m =0 //(M: global size, m: local size) this result is given by the proc 1</b></div><div class=""><b class="">and </b></div><div class=""><b class="">PETSc::NonLinearSolver::INIT Size from Jac Matrix: M=81 m =81 // </b><b class="">this result is given by the proc 2</b></div></div></div></blockquote><div class=""><br class=""></div><div class="">Yes, your mesh in only on process 0.</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="overflow-wrap: break-word;" class=""><div class=""><div class="">Let me give other information:</div><div class="">I create my jacobian matrix using:</div><div class="">PetscErrorCode err = DMCreateMatrix(dm_mesh, &_matrix);</div><div class=""><b class="">and I use the PetscSection to tell the DM to use this data layout.</b></div><div class=""><br class=""></div><div class="">In my code I wrote this line: </div><div class="">DMSetFromOptions(dm_mesh);</div><div class="">Then, to run my code I use</div><div class="">mpirun -np 2 /home/benelhasa/fox_petsc/build_test/bin/Debug/FoXtroT -snes_test_jacobian_view -snes_converged_reason -snes_monitor -ksp_monitor -ksp_xmonitor -dm_view<span class="Apple-converted-space"> </span><b class=""><font color="#ff2600" class="">-petscpartitioner_type parmetis -dm_distribute</font></b><span class="Apple-converted-space"> </span>-dm_refine 0 cub_8C3D8.fxt</div></div></div></blockquote><div class=""><br class=""></div><div class="">Something else is wrong in how this code is creating the mesh, because it is not distributing the mesh. Can you just use</div><div class=""><br class=""></div><div class=""> PetscCall(DMCreate(comm, dm));<br class=""> PetscCall(DMSetType(*dm, DMPLEX));<br class=""> PetscCall(DMSetFromOptions(*dm));<br class=""></div><div class=""><br class=""></div><div class="">as we do in the examples and then</div><div class=""><br class=""></div><div class=""> -dm_plex_filename <your gmsh file></div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="overflow-wrap: break-word;" class=""><div class=""><div class=""><b class=""><font color="#ff2600" class="">Question 2:</font></b></div><div class=""><b class=""><font color="#0433ff" class="">I have another question, just for the comprehension. I understand that DMSetFromOptions(dm_mesh) allows me to use the parameter<span class="Apple-converted-space"> </span></font><font color="#ff2600" class="">dm_distribute</font><font color="#0433ff" class=""><span class="Apple-converted-space"> </span>but I didn't understand how I can use -petscpartitioner_type parmetis argument (see this example that I used to help me (</font></b><span class=""><b class=""><a href="https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/plex/tutorials/ex14.c" target="_blank" class="">https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/plex/tutorials/ex14.c</a></b></span><b class=""><font color="#0433ff" class="">), may be when the DM uses a data layout either by PetscSection or PetscDS, then I can use automatically the -petscpartitioner_type parmetis/simple/scotch ?? (Can you tell me more about this topic please) </font></b></div></div></div></blockquote><div class=""><br class=""></div><div class="">I do not yet understand the question. The partitioner only applies to the topology, not to functions on the mesh.</div><div class=""><br class=""></div><div class=""> Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="overflow-wrap: break-word;" class=""><div class=""><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">Ps: My solution is converging when I use 1 process and I don’t have problem.</div><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><br class=""></div><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">--</div><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">Dr. Sami BEN ELHAJ SALAH<br class="">Ingénieur de Recherche (CNRS)<br class="">Institut Pprime - ISAE - ENSMA<br class="">Mobile: 06.62.51.26.74<br class=""><a href="mailto:sami.ben-elhaj-salah@ensma.fr" target="_blank" class="">Email: sami.ben-elhaj-salah@ensma.fr</a></div><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><a href="https://samiben91.github.io/samibenelhajsalah/index.html" target="_blank" class="">www.samibenelhajsalah.com</a><br class=""><br class=""><br class=""></div></div></div></div></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">Le 29 mai 2022 à 18:02, Sami BEN ELHAJ SALAH <<a href="mailto:sami.ben-elhaj-salah@ensma.fr" target="_blank" class="">sami.ben-elhaj-salah@ensma.fr</a>> a écrit :</div><br class=""><div class=""><div style="overflow-wrap: break-word;" class=""><div class=""><div class="">Hi Matthew,</div><div class="">Thank you for this example. It seems exactly what I am looking for.</div><div class="">Thank you again for your help and have a good day.</div><div class="">Sami,</div><div class=""><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">--</div><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">Dr. Sami BEN ELHAJ SALAH<br class="">Ingénieur de Recherche (CNRS)<br class="">Institut Pprime - ISAE - ENSMA<br class="">Mobile: 06.62.51.26.74<br class=""><a href="mailto:sami.ben-elhaj-salah@ensma.fr" target="_blank" class="">Email: sami.ben-elhaj-salah@ensma.fr</a></div><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><a href="https://samiben91.github.io/samibenelhajsalah/index.html" target="_blank" class="">www.samibenelhajsalah.com</a><br class=""><br class=""><br class=""></div></div></div></div></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">Le 28 mai 2022 à 20:20, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> a écrit :</div><br class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div dir="ltr" class="">On Sat, May 28, 2022 at 2:19 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div dir="ltr" class="">On Sat, May 28, 2022 at 1:35 PM Sami BEN ELHAJ SALAH <<a href="mailto:sami.ben-elhaj-salah@ensma.fr" target="_blank" class="">sami.ben-elhaj-salah@ensma.fr</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class=""><div class="">Hi Matthew,</div><div class=""><br class=""></div><div class="">Thank you for your response.</div><div class=""><br class=""></div><div class="">I don't have that. My DM object is not linked to PetscSection yet. I'll try that. </div><div class="">Is there any example that manages this case? (DMPlexDistribute & PetscSection) or any guideline will be helpful.</div><div class=""></div></div></blockquote><div class=""><br class=""></div><div class="">Here is an example where we create a section without DS.</div></div></div></blockquote><div class=""><br class=""></div><div class="">Forgot the link: <a href="https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/plex/tutorials/ex14.c" target="_blank" class="">https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/plex/tutorials/ex14.c</a></div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div class="gmail_quote"><div class=""> <span class=""> </span>THanks,</div><div class=""><br class=""></div><div class=""> <span class=""> </span>Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class=""><div class="">Thanks in advance,</div><div class=""><br class=""></div><div class="">Sami,</div><div class=""><br class=""></div><div class=""><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">--</div><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">Dr. Sami BEN ELHAJ SALAH<br class="">Ingénieur de Recherche (CNRS)<br class="">Institut Pprime - ISAE - ENSMA<br class="">Mobile: 06.62.51.26.74<br class=""><a href="mailto:sami.ben-elhaj-salah@ensma.fr" target="_blank" class="">Email: sami.ben-elhaj-salah@ensma.fr</a></div><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><a href="https://samiben91.github.io/samibenelhajsalah/index.html" target="_blank" class="">www.samibenelhajsalah.com</a><br class=""><br class=""><br class=""></div></div></div></div></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">Le 27 mai 2022 à 20:45, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> a écrit :</div><br class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">On Fri, May 27, 2022 at 9:42 AM Sami BEN ELHAJ SALAH <<a href="mailto:sami.ben-elhaj-salah@ensma.fr" target="_blank" class="">sami.ben-elhaj-salah@ensma.fr</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class=""><div class=""><div class="">Hello Isaac,</div><div class=""><br class=""></div><div class="">Thank you for your reply!</div><div class=""><br class=""></div><div class="">Let me confirm that when I use DMCreateMatrix() with the orig_dm, I got my jacobian_matrix. Also, I have succeeded to solve my system and my solution was converged<b class=""><span class=""> </span>by using one process</b>.</div><div class="">Let me give you some other information about my code. Currently, I am using my own discretization system and not the PetscDS object. Considering the nonlinear solver SNES, I use the following routines (like the basic snes usage):</div><div class="">- SNESSetFunction(snes,residual_vector,compute_residual ,(void*) _finite_element_formulation)</div><div class="">- SNESSetJacobian(snes, jacobian.matrix(), _jacobian_precond_matrix, compute_jacobian, (void*) _finite_element_formulation) </div><div class="">- SNESSolve(snes,NULL,x)</div><div class=""><br class=""></div><div class="">Regarding your last answer, I will try to reformulate my question as follows:</div><div class="">Using a distributed dm instead of the original dm (not distributed dm) and my own discretization system (not the PetscDS object),</div></div></div></blockquote><div class=""><br class=""></div><div class="">You do not have to use PetscDS, but the DM does need to a PetscSection in order to compute sizes and sparsity patterns. Do you have that?</div><div class=""><br class=""></div><div class=""> <span class=""> </span>Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class=""><div class=""><div class="">should I add something specific to get a distributed jacobian_matrix over processes?</div><div class="">I precise that I just replaced the orig_dm by a distributed mesh with the routine that I wrote in my first mail. So is it enough ?</div><div class=""> </div><div class="">Thank you and have a good day,</div><div class="">Sami,</div></div><div class=""><br class=""></div><br class=""><div class=""><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">--</div><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">Dr. Sami BEN ELHAJ SALAH<br class="">Ingénieur de Recherche (CNRS)<br class="">Institut Pprime - ISAE - ENSMA<br class="">Mobile: 06.62.51.26.74<br class=""><a href="mailto:sami.ben-elhaj-salah@ensma.fr" target="_blank" class="">Email: sami.ben-elhaj-salah@ensma.fr</a></div><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><a href="https://samiben91.github.io/samibenelhajsalah/index.html" target="_blank" class="">www.samibenelhajsalah.com</a><br class=""><br class=""><br class=""></div></div></div></div></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">Le 25 mai 2022 à 19:41, Toby Isaac <<a href="mailto:toby.isaac@gmail.com" target="_blank" class="">toby.isaac@gmail.com</a>> a écrit :</div><br class=""><div class=""><div class="">Hi Sami,<br class=""><br class="">Just to verify: if you call DMCreateMatrix() on orig_dm, do you get a<br class="">Jacobian matrix?<br class=""><br class="">The DMPlex must be told what kind of discretized fields you want a<br class="">matrix for and what equations you are discretizing. This is handled<br class="">by the PetscDS object. In snes/tutorials/ex59.c, see the code after<br class="">DMGetDS() for an example.<br class=""><br class="">- Toby<br class=""><br class="">On Wed, May 25, 2022 at 1:17 PM Sami BEN ELHAJ SALAH<br class=""><<a href="mailto:sami.ben-elhaj-salah@ensma.fr" target="_blank" class="">sami.ben-elhaj-salah@ensma.fr</a>> wrote:<br class=""><blockquote type="cite" class=""><br class="">Dear PETSc developer team,<br class=""><br class="">I m trying to create à jacobian_matrix from distributed DM. I have followed the two examples (snes/tests/ex2.c and ex56.c). So I wrote this routine:<br class=""><br class="">PetscDM orig_dm;<br class="">PetscDM dist_dm = NULL;<br class="">PetscPartitioner part;<br class="">DMPlexGetPartitioner(orig_dm, &part);<br class="">PetscPartitionerSetType(part, PETSCPARTITIONERPARMETIS);<br class="">DMPlexDistribute(orig_dm, 0, NULL, &dist_dm);<br class=""><br class="">PetscErrorCode err = DMCreateMatrix(dist_dm, &jacobian_matrix);<br class="">PetscInt M, N, m, n;<br class="">MatGetSize(jacobian_matrix, &M, &N);<br class="">MatGetLocalSize(jacobian_matrix, &m, &n);<br class=""><br class="">Then I run my code with 2 processes and I obtained this result:<br class="">Size from jacobian_matrix: M=0 m =0 (this result is the same in all processes).<br class=""><br class="">I didn't understand if I forgot something in my code to obtain the correct values for the local and global sizes for the jacobean_matrix? (I mean if my code is missing something to obtain a distributed mesh over processes ?)<br class=""><br class="">Thank you in advance for any help!<br class="">Sami,<br class=""><br class=""><br class="">--<br class="">Dr. Sami BEN ELHAJ SALAH<br class="">Ingénieur de Recherche (CNRS)<br class="">Institut Pprime - ISAE - ENSMA<br class="">Mobile: 06.62.51.26.74<br class=""><a href="mailto:sami.ben-elhaj-salah@ensma.fr" target="_blank" class="">Email: sami.ben-elhaj-salah@ensma.fr</a><br class=""><a href="http://www.samibenelhajsalah.com/" target="_blank" class="">www.samibenelhajsalah.com</a><br class=""><br class=""><br class=""><br class=""></blockquote></div></div></blockquote></div><br class=""></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>--<span class=""> </span><br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div></div></blockquote></div><br class=""></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>--<span class=""> </span><br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>--<span class=""> </span><br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a></div></div></div></div></div></div></div></div></div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>--<span class="Apple-converted-space"> </span><br class=""><div dir="ltr" class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a></div></div></div></div></div></div></div></div></div></blockquote></div><br class=""></div></body></html>