<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> Yes, you should partition the elements and redistribute them for optimal parallelism.<div><br></div><div> You can use the MatPartitioning object to partition the graph of the elements which will tell you what elements should be assigned to each MPI process. But then you need to move the element information to the correct process. At that point your code will remain pretty much as it is now.</div><div><br></div><div> Barry</div><div><br><div><br><blockquote type="cite"><div>On Mar 14, 2023, at 10:59 AM, Pantelis Moschopoulos <pmoschopoulos@outlook.com> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">Dear Dave,<span class="Apple-converted-space"> </span><br></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);"><br></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">Yes, I observe this in parallel runs. How I can change the parallel layout of the matrix? In my implementation, I read the mesh file, and the I split the domain where the first rank gets the first N elements, the second rank gets the next N elements etc. Should I use metis to distribute elements? Note that I use continuous finite elements, which means that some values will be cached in a temporary buffer.</div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);"><br></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">Thank you very much,<span class="Apple-converted-space"> </span><br></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">Pantelis<br></div><div id="appendonsend" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"></div><hr tabindex="-1" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 542.90625px;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;"></span><div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><font face="Calibri, sans-serif" style="font-size: 11pt;"><b>From:</b><span class="Apple-converted-space"> </span>Dave May <dave.mayhem23@gmail.com><br><b>Sent:</b><span class="Apple-converted-space"> </span>Tuesday, March 14, 2023 4:40 PM<br><b>To:</b><span class="Apple-converted-space"> </span>Pantelis Moschopoulos <pmoschopoulos@outlook.com><br><b>Cc:</b><span class="Apple-converted-space"> </span>petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] Memory Usage in Matrix Assembly.</font><div> </div></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div><br></div><div><br><div class="x_gmail_quote"><div dir="ltr" class="x_gmail_attr">On Tue 14. Mar 2023 at 07:15, Pantelis Moschopoulos <<a href="mailto:pmoschopoulos@outlook.com">pmoschopoulos@outlook.com</a>> wrote:<br></div><blockquote class="x_gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr"><div><a></a></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">Hi everyone,<span class="Apple-converted-space"> </span><br></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);"><br></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">I am a new Petsc user that incorporates Petsc for FEM in a Fortran code. <br></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">My question concerns the sudden increase of the memory that Petsc needs during the assembly of the jacobian matrix. After this point, memory is freed. It seems to me like Petsc performs memory allocations and the deallocations during assembly.<span class="Apple-converted-space"> </span><br></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">I have used the following commands with no success:<span class="Apple-converted-space"> </span><br></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">CALL MatSetOption(petsc_A, MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE,ier)<div>CALL MatSetOption(petsc_A, MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE,ier)</div>CALL MatSetOption(petsc_A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE,ier).</div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">CALL MatSetOption(petsc_A, MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE,ier)</div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);"><br></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">The structure of the matrix does not change during my simulation, just the values. I am expecting this behavior the first time that I create this matrix because the preallocation instructions that I use are not very accurate but this continues every time I assemble the matrix.</div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">What I am missing here?</div></div></blockquote><div dir="auto"><br></div><div dir="auto">I am guessing this observation is seen when you run a parallel job.</div><div dir="auto"><br></div><div dir="auto">MatSetValues() will cache values in a temporary memory buffer if the values are to be sent to a different MPI rank.<br></div><div dir="auto">Hence if the parallel layout of your matrix doesn’t closely match the layout of the DOFs on each mesh sub-domain, then a huge number of values can potentially be cached. After you call MatAssemblyBegin(), MatAssemblyEnd() this cache will be freed.<br></div><div dir="auto"><br></div><div dir="auto">Thanks,</div><div dir="auto">Dave</div><div dir="auto"><br></div><div dir="auto"><br></div><blockquote class="x_gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr"><div dir="auto" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);"></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);"><br></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">Thank you very much,<span class="Apple-converted-space"> </span><br></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">Pantelis<span class="Apple-converted-space"> </span></div></div></blockquote></div></div></div></div></blockquote></div><br></div></body></html>