<div dir="ltr"><div class="gmail-aju" style="display:flex;float:none;height:80px;padding:0px 16px;min-width:40px;font-family:"Google Sans",Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:medium"><div class="gmail-aCi" style=""><img id="gmail-:ms_35-e" name=":ms" src="https://lh3.googleusercontent.com/a/AAcHTtf6Y22wOaEkZkvNtqhFjJ83I4I8px6KALMxapJJuBl0=s80-p-mo" class="gmail-ajn" aria-hidden="true" style="width: 40px; height: 40px; border-radius: 50%; display: block; background-color: rgb(204, 204, 204);"></div></div><div class="gmail-gs" style="margin:0px;padding:0px 0px 20px;width:732.667px;font-family:"Google Sans",Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:medium"><div class="gmail-gE gmail-iv gmail-gt" style="font-size:0.875rem;padding:20px 0px 0px"><table cellpadding="0" class="gmail-cf gmail-gJ" style="border-collapse:collapse;margin-top:0px;width:auto;font-size:0.875rem;display:block"><tbody style="display:block"><tr class="gmail-acZ" style="display:flex;height:auto"><td class="gmail-gF gmail-gK" style="padding:0px;vertical-align:top;width:540.115px;line-height:20px;display:block;max-height:20px"><table cellpadding="0" class="gmail-cf gmail-ix" style="border-collapse:collapse;table-layout:fixed;width:540.115px"><tbody><tr><td class="gmail-c2" style="display:flex"><h3 class="gmail-iw" style="overflow:hidden;font-size:0.75rem;font-weight:inherit;margin:inherit;text-overflow:ellipsis;color:rgb(95,99,104);line-height:20px"><span class="gmail-qu" role="gridcell" tabindex="-1"><span name="袁煕" class="gmail-gD" style="color:rgb(31,31,31);font-size:0.875rem;font-weight:bold;display:inline;vertical-align:top;line-height:20px"><span style="vertical-align:top">袁煕</span></span> <span class="gmail-cfXrwd"></span><span class="gmail-go" style="vertical-align:top;color:rgb(94,94,94)"><span aria-hidden="true"><</span><a href="mailto:yuanxi@advancesoft.jp">yuanxi@advancesoft.jp</a><span aria-hidden="true">></span></span></span></h3></td></tr></tbody></table></td><td class="gmail-gH gmail-bAk" style="text-align:right;vertical-align:top;display:block;max-height:20px"><div class="gmail-gK" style="padding:0px;display:flex"><span id="gmail-:67q" class="gmail-g3" title="2023/07/24 0:50" alt="2023/07/24 0:50" role="gridcell" tabindex="-1" style="vertical-align:top;margin:0px;font-size:0.75rem;color:rgb(94,94,94);display:block;line-height:20px">0:50 (0 分前)</span><div class="gmail-zd gmail-bi4" aria-label="スターなし" tabindex="0" role="checkbox" aria-checked="false" style="display:inline-block;height:20px;margin-left:20px;outline:0px"><span class="gmail-T-KT" style="display:inline-flex;height:20px;text-align:center;width:20px;padding:0px;margin:0px;border:none;outline:none"></span></div></div></td><td class="gmail-gH" style="text-align:right;vertical-align:top;display:flex"></td><td class="gmail-gH gmail-acX gmail-bAm" rowspan="2" style="text-align:right;vertical-align:top;display:block;max-height:20px"><div class="gmail-T-I gmail-J-J5-Ji gmail-T-I-Js-IF gmail-aaq gmail-T-I-ax7 gmail-L3" role="button" tabindex="0" aria-label="返信" style="display:inline-flex;border-radius:2px 0px 0px 2px;font-size:0.875rem;text-align:center;margin:0px 0px 0px 20px;height:20px;line-height:18px;min-width:0px;outline:none;padding:0px;background:transparent;color:rgb(68,68,68);border:none"><img class="gmail-hB gmail-T-I-J3" role="button" src="https://mail.google.com/mail/u/0/images/cleardot.gif" alt="" style="background: url("//ssl.gstatic.com/ui/v1/icons/mail/gm3/2x/reply_baseline_nv700_20dp.png") 50% 50% / 20px no-repeat; height: 20px; margin: 0px; vertical-align: middle; width: 20px; opacity: 1; display: inline-block; padding: 0px; transition: opacity 0.15s cubic-bezier(0.4, 0, 0.2, 1) 0s;"></div><div id="gmail-:67f" class="gmail-T-I gmail-J-J5-Ji gmail-T-I-Js-Gs gmail-aap gmail-T-I-awG gmail-T-I-ax7 gmail-L3" role="button" tabindex="0" aria-expanded="false" aria-haspopup="true" aria-label="その他" style="display:inline-flex;border-radius:0px 2px 2px 0px;font-size:0.875rem;text-align:center;margin:0px 0px 0px 20px;height:20px;line-height:18px;min-width:0px;outline:none;padding:0px;background:transparent;color:rgb(68,68,68);border:none"><img class="gmail-hA gmail-T-I-J3" role="menu" src="https://mail.google.com/mail/u/0/images/cleardot.gif" alt="" style="background: url("//ssl.gstatic.com/ui/v1/icons/mail/gm3/2x/more_vert_baseline_nv700_20dp.png") 50% 50% / 20px no-repeat; height: 20px; width: 20px; margin: 0px; vertical-align: middle; opacity: 1; display: inline-block; padding: 0px; transition: opacity 0.15s cubic-bezier(0.4, 0, 0.2, 1) 0s;"></div></td></tr><tr class="gmail-acZ gmail-xD" style="display:flex;height:auto"><td colspan="3"><table cellpadding="0" class="gmail-cf gmail-adz" style="border-collapse:collapse;table-layout:fixed;width:732.667px"><tbody><tr><td class="gmail-ady" style="overflow:visible;text-overflow:ellipsis;display:flex;line-height:20px"><div class="gmail-iw gmail-ajw" style="overflow:hidden;max-width:92%;display:inline-block"><span class="gmail-hb" style="vertical-align:top;color:rgb(94,94,94);font-size:0.75rem;line-height:20px">To <span name="PETSc" class="gmail-g2" style="vertical-align:top">PETSc</span></span></div><div id="gmail-:67g" aria-haspopup="true" class="gmail-ajy" role="button" tabindex="0" aria-label="詳細を表示" style="display:inline-flex;margin-left:4px;vertical-align:top;border:none;outline:none"><img class="gmail-ajz" src="https://mail.google.com/mail/u/0/images/cleardot.gif" alt="" style="background: url("https://www.gstatic.com/images/icons/material/system_gm/2x/arrow_drop_down_black_20dp.png") 50% 50% / 20px no-repeat; cursor: pointer; padding: 0px; vertical-align: baseline; height: 20px; width: 20px; border: none; margin: 0px 0px 0px auto; right: 0px; top: 0px; display: flex; opacity: 0.71;"></div></td></tr></tbody></table></td></tr></tbody></table></div><div id="gmail-:67u"><div class="gmail-qQVYZb"></div><div class="gmail-utdU2e"></div><div class="gmail-lQs8Hd"></div><div class="gmail-btm"></div></div><div class="gmail-"><div class="gmail-aHl" style=""></div><div id="gmail-:67h" tabindex="-1"></div><div id="gmail-:67s" class="gmail-ii gmail-gt" style="direction:ltr;margin:8px 0px 0px;padding:0px;font-size:0.875rem"><div id="gmail-:67t" class="gmail-a3s gmail-aiL" style="font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:small;line-height:1.5;font-family:Arial,Helvetica,sans-serif;overflow:hidden"><div dir="ltr">Hi,<div><br></div><div>I used PETSc to assemble a FEM stiff matrix of an overlapped (overlap=2) DMPlex and used the MUMPS solver to solve it. But I got a different solution by using 1 CPU and MPI parallel computation. I am wondering if I missed some necessary step or setting during my implementation.</div><div><br></div><div>    My calling process like follows</div><div><br></div><div>1. Generate matrix pkij from DMPlex dm_mesh</div><div><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">```
      PetscCallA( DMCreateMatrix(dm_mesh, pkij, ierr) )<br>      PetscCallA( MatSetBlockSize(pkij, 1, ierr))<br>      if( overlap>0 ) call MatSetOption(pkij,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE,ierr)<br>      PetscCallA( MatSetFromOptions(pkij, ierr))<br><br>      PetscCallA(MatGetLocalSize(pkij, m, n, ierr));<br>      PetscCallA(MatGetSize(pkij, gm, gn, ierr));<br>      PetscCallA(MatGetBlockSize(pkij, bs, ierr));
```</pre><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">2. Do Matrix preallocation</pre></div><div><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">```<br>      PetscCallA(MatCreate(PETSC_COMM_WORLD, preallocator, ierr));<br>      PetscCallA(MatSetType(preallocator, MATPREALLOCATOR, ierr));<br>      PetscCallA(MatSetSizes(preallocator, m, n, gm, gn, ierr));<br>      PetscCallA(MatSetBlockSize(preallocator, bs, ierr));<br>      PetscCallA(MatSetUp(preallocator, ierr));</pre><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">      .......</pre><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">      PetscCallA( MatPreallocatorPreallocate(preallocator, PETSC_TRUE, pkij, ierr) )<br>      PetscCallA( MatDestroy(preallocator,ierr) )
```</pre><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">3. Assemble matrix pkij by calling MatSetValue of all overlapped elements.</pre><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">In my above implementation, I used MatSetOption(pkij,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE,ierr). Is that correct? Or even other options are needed?</pre><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">Thanks for your help.</pre><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">------</pre><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">X. Yuan, Ph.D. of solid mechanics</pre><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">Advancesoft, Japan</pre><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">---------</pre></div></div></div></div></div></div></div>