<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="SV" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Thanks for the comment! I actually solved the problem this morning. I had simply made a mistake, partially
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">(as you suggested) related to the construction of edofF. So, combining DMDA and DMStag in the way
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">outlined in the sketch below seems to work perfectly fine<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Patrick Sanan <patrick.sanan@gmail.com>
<br>
<b>Sent:</b> den 25 april 2022 08:48<br>
<b>To:</b> Carl-Johan Thore <carl-johan.thore@liu.se><br>
<b>Cc:</b> Barry Smith <bsmith@petsc.dev>; petsc-users@mcs.anl.gov<br>
<b>Subject:</b> Re: [petsc-users] Combining DMDA and DMStag<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">In that case, from your original message, it seems like the issue might be in the
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> </span>// populate edofF, edofT <o:p>
</o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Or, perhaps, it's simply due to the fact that the numbering/ordering depends on the number of ranks. This is illustrated in this image in the manual:
<a href="https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Fdocs%2Fmanual%2Fvec%2F%23fig-daao&data=05%7C01%7Ccarl-johan.thore%40liu.se%7Cef98948377bc4771eab108da2687906f%7C913f18ec7f264c5fa816784fe9a58edd%7C0%7C0%7C637864660932072744%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=smjpJ77ZiE10xeEP2OkfD5FYgbt2xKMviPWD%2FmmJEtM%3D&reserved=0">
https://petsc.org/release/docs/manual/vec/#fig-daao</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">This is for DMDA but DMStag does something similar. There is a "natural" ordering, which is the ordering you'd get with a single rank. There's also "PETSc ordering" which depends on the number of ranks - in this case each rank's portion
of the vector is a contiguous range. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">If this might be the explanation for your problem, note that for DMDA there are already some helper functions which can map "PETSc" to "natural" ordering. While these obviously require a lot of communication between ranks and so you'd want
to avoid them in production runs, these are useful in situations where you want to directly compare vectors on different numbers of ranks for diagnostic or I/O purposes.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">E.g. see this man page:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Fdocs%2Fmanualpages%2FDMDA%2FDMDANaturalToGlobalBegin.html&data=05%7C01%7Ccarl-johan.thore%40liu.se%7Cef98948377bc4771eab108da2687906f%7C913f18ec7f264c5fa816784fe9a58edd%7C0%7C0%7C637864660932072744%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dCPc2l6Uzg0FGSRKZyIC20OiFv60UjgT2%2BQ8JTRjrPk%3D&reserved=0">https://petsc.org/release/docs/manualpages/DMDA/DMDANaturalToGlobalBegin.html</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I haven't implemented the equivalent for DMStag, but if it's needed I can look at adding it.<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Am Fr., 22. Apr. 2022 um 17:28 Uhr schrieb Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se">carl-johan.thore@liu.se</a>>:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal">Thanks for the quick replies!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Using dmstag for the temperature, or even for both fields is a possibility of course. Preferably however I'd like to keep the temp. code as it is and be able to quickly switch between different methods for the flow, based on dmda, dmstag
and so on, so I'll try a bit more with the dmda-dmstag approach first. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="gmail-m_-8400698669009399192divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Patrick Sanan <<a href="mailto:patrick.sanan@gmail.com" target="_blank">patrick.sanan@gmail.com</a>><br>
<b>Sent:</b> 22 April 2022 17:04<br>
<b>To:</b> Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>><br>
<b>Cc:</b> Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" target="_blank">carl-johan.thore@liu.se</a>>;
<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Combining DMDA and DMStag</span> <o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Am Fr., 22. Apr. 2022 um 16:04 Uhr schrieb Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>>:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"> We would need more details as to exactly what goes wrong to determine any kind of fix; my guess would be that the layout of the velocity vectors and temperature vectors is slightly different since the DMDA uses nelx+1 while the stag
uses nelx and may split things up slightly differently in parallel. You could try very small problems with say 2 or 4 ranks and put known values into the vectors and look at the ghost point update locations and exact locations in the local and global vectors
to make sure everything is where you expect it to be. <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> There is also the possibility of using a DMStag for the temperature instead of DMDA since DMDA provides essentially a subset of the functionality of DMDA to get a more consistent layout of the unknowns in the two vectors.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><br>
This was going to be my first suggestion as well - one way to ensure compatibility would be to use DMStag for everything. E.g. you could create your temperature DMStag with only vertex/corner (dof30 degrees of freedom, and then create one or more a "compatible"
DMStags (same elements on each MPI rank) for your other fields, using DMStagCreateCompatibleDMStag() .<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"> Finally, one could use a single DMStag with all the unknowns but treat subvectors of the unknowns differently in your discretization and solve process. So assign velocity values (I guess) to the cell faces and temperature to the cell
vertices, this will give consistent parallel decomposition of values but you will have to manage the fact that the unknowns are interlaced into a single vector so your solver portions of the code may need to "pull" out appropriate subvectors.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> Barry<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Apr 22, 2022, at 9:45 AM, Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" target="_blank">carl-johan.thore@liu.se</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Hi!</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">I'm working on a convection-diffusion heat transfer problem. The temperature</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">is discretized using standard Q1 elements and a DMDA. The flow is modelled</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">using a stabilized Q1-Q0 method for which DMStag seemed like a good choice. The codes for the temperature</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">and flow work fine separately (both in serial and parallel), but when combined and running</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">in parallel, a problem sometimes arises in the assembly of the thermal system matrix.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Here’s a rough sketch of the combined code:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">// Create dmda for thermal problem and dmstag for flow problem</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">DMDACreate3d(PETSC_COMM_WORLD, bx, by, bz, DMDA_STENCIL_BOX, nelx+1, nely+1, nelz+1, PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> 1, stencilWidth, 0, 0, 0, &dmthermal); </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">…</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">// A bit of code to adjust Lx,Ly,Lz so that dmthermal and dmflow are compatible in the sense of having the same</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">// local elements</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">…</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">DMStagCreate3d(PETSC_COMM_WORLD, bx, by, bz, nelx, nely, nelz, md,nd,pd,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="text-indent:65.2pt"><span lang="EN-US">3,0,0,0,DMSTAG_STENCIL_BOX,stencilWidth,Lx,Ly,Lz,&dmflow); </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">PetscInt edofT[8]; // 8-noded element with 1 temp DOF per node</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">DMStagStencil edofF[24]; // 8 nodes with 3 velocity DOFs each</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">// Assemble thermal system matrix K</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">for (PetscInt e=0 ...) // Loop over local elements</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">{</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="text-indent:65.2pt"><span lang="EN-US">// Populate edofF, edofT</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> // Get element velocities in ue from local velocity vector uloc</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> DMStagVecGetValuesStencil(dmflow,uloc,24,edof,ue);</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> ...</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> Ke = Ke_diffusion + Ke_convection(ue)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> ...</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> MatSetValuesLocal(K, 8, edofT, 8, edofT, Ke, ADD_VALUES);</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">}</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">This always works fine in serial, but depending on the mesh and the number of ranks,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">we don't always get the correct values in the element velocity vector ue. I suspect</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">this has something to do with the ordering of the elements and/or the DOFs, because</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">the elements in the global velocity vector are always the same but their order may change</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">(judging from the output of VecView at least).</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Is it possible to ensure compatibility between the dm:s, or find some kind of mapping</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">between them, so that something along the lines of the code above always works?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Kind regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Carl-Johan<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>