<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> We would need more details as to exactly what goes wrong to determine any kind of fix; my guess would be that the layout of the velocity vectors and temperature vectors is slightly different since the DMDA uses nelx+1 while the stag uses nelx and may split things up slightly differently in parallel. You could try very small problems with say 2 or 4 ranks and put known values into the vectors and look at the ghost point update locations and exact locations in the local and global vectors to make sure everything is where you expect it to be.<div class=""><br class=""></div><div class=""> There is also the possibility of using a DMStag for the temperature instead of DMDA since DMDA provides essentially a subset of the functionality of DMDA to get a more consistent layout of the unknowns in the two vectors.</div><div class=""><br class=""></div><div class=""> Finally, one could use a single DMStag with all the unknowns but treat subvectors of the unknowns differently in your discretization and solve process. So assign velocity values (I guess) to the cell faces and temperature to the cell vertices, this will give consistent parallel decomposition of values but you will have to manage the fact that the unknowns are interlaced into a single vector so your solver portions of the code may need to "pull" out appropriate subvectors.</div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Apr 22, 2022, at 9:45 AM, Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" class="">carl-johan.thore@liu.se</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">Hi!<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">I'm working on a convection-diffusion heat transfer problem. The temperature<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">is discretized using standard Q1 elements and a DMDA. The flow is modelled<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">using a stabilized Q1-Q0 method for which DMStag seemed like a good choice. The codes for the temperature<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">and flow work fine separately (both in serial and parallel), but when combined and running<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">in parallel, a problem sometimes arises in the assembly of the thermal system matrix.<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">Here’s a rough sketch of the combined code:<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">// Create dmda for thermal problem and dmstag for flow problem<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">DMDACreate3d(PETSC_COMM_WORLD, bx, by, bz, DMDA_STENCIL_BOX, nelx+1, nely+1, nelz+1, PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE,<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class=""> 1, stencilWidth, 0, 0, 0, &dmthermal); <o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">…<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">// A bit of code to adjust Lx,Ly,Lz so that dmthermal and dmflow are compatible in the sense of having the same<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">// local elements<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">…<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">DMStagCreate3d(PETSC_COMM_WORLD, bx, by, bz, nelx, nely, nelz, md,nd,pd,<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 65.2pt;" class=""><span lang="EN-US" class="">3,0,0,0,DMSTAG_STENCIL_BOX,stencilWidth,Lx,Ly,Lz,&dmflow); <o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class=""> <o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">PetscInt edofT[8]; // 8-noded element with 1 temp DOF per node<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">DMStagStencil edofF[24]; // 8 nodes with 3 velocity DOFs each<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">// Assemble thermal system matrix K<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">for (PetscInt e=0 ...) // Loop over local elements<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">{<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 65.2pt;" class=""><span lang="EN-US" class="">// Populate edofF, edofT<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class=""> // Get element velocities in ue from local velocity vector uloc<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class=""> DMStagVecGetValuesStencil(dmflow,uloc,24,edof,ue);<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class=""> ...<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class=""> Ke = Ke_diffusion + Ke_convection(ue)<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class=""> ...<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class=""> MatSetValuesLocal(K, 8, edofT, 8, edofT, Ke, ADD_VALUES);<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">}<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">This always works fine in serial, but depending on the mesh and the number of ranks,<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">we don't always get the correct values in the element velocity vector ue. I suspect<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">this has something to do with the ordering of the elements and/or the DOFs, because<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">the elements in the global velocity vector are always the same but their order may change<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">(judging from the output of VecView at least).<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">Is it possible to ensure compatibility between the dm:s, or find some kind of mapping<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class="">between them, so that something along the lines of the code above always works?<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-US" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="">Kind regards,<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="">Carl-Johan</span></div></div></div></blockquote></div><br class=""></div></body></html>