<div dir="ltr"><div dir="ltr">On Sun, Jul 23, 2023 at 11:54 AM 袁煕 <<a href="mailto:yuanxi@advancesoft.jp">yuanxi@advancesoft.jp</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="display:flex;float:none;height:80px;padding:0px 16px;min-width:40px;font-family:"Google Sans",Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:medium"><div><img id="m_-5633064360383637069gmail-:ms_35-e" name="m_-5633064360383637069_:ms" src="https://lh3.googleusercontent.com/a/AAcHTtf6Y22wOaEkZkvNtqhFjJ83I4I8px6KALMxapJJuBl0=s80-p-mo" aria-hidden="true" style="width: 40px; height: 40px; border-radius: 50%; display: block; background-color: rgb(204, 204, 204);"></div></div><div style="margin:0px;padding:0px 0px 20px;width:732.667px;font-family:"Google Sans",Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:medium"><div style="font-size:0.875rem;padding:20px 0px 0px"><table cellpadding="0" style="border-collapse:collapse;margin-top:0px;width:auto;font-size:0.875rem;display:block"><tbody style="display:block"><tr style="display:flex;height:auto"><td style="padding:0px;vertical-align:top;width:540.115px;line-height:20px;display:block;max-height:20px"><table cellpadding="0" style="border-collapse:collapse;table-layout:fixed;width:540.115px"><tbody><tr><td style="display:flex"><h3 style="overflow:hidden;font-size:0.75rem;font-weight:inherit;margin:inherit;text-overflow:ellipsis;color:rgb(95,99,104);line-height:20px"><span role="gridcell"><span name="袁煕" style="color:rgb(31,31,31);font-size:0.875rem;font-weight:bold;display:inline;vertical-align:top;line-height:20px"><span style="vertical-align:top">袁煕</span></span> <span></span><span style="vertical-align:top;color:rgb(94,94,94)"><span aria-hidden="true"><</span><a href="mailto:yuanxi@advancesoft.jp" target="_blank">yuanxi@advancesoft.jp</a><span aria-hidden="true">></span></span></span></h3></td></tr></tbody></table></td><td style="text-align:right;vertical-align:top;display:block;max-height:20px"><div style="padding:0px;display:flex"><span id="m_-5633064360383637069gmail-:67q" title="2023/07/24 0:50" alt="2023/07/24 0:50" role="gridcell" style="vertical-align:top;margin:0px;font-size:0.75rem;color:rgb(94,94,94);display:block;line-height:20px">0:50 (0 分前)</span><div aria-label="スターなし" role="checkbox" aria-checked="false" style="display:inline-block;height:20px;margin-left:20px;outline:0px"><span style="display:inline-flex;height:20px;text-align:center;width:20px;padding:0px;margin:0px;border:none;outline:none"></span></div></div></td><td style="text-align:right;vertical-align:top;display:flex"></td><td rowspan="2" style="text-align:right;vertical-align:top;display:block;max-height:20px"><div role="button" aria-label="返信" style="display:inline-flex;border-radius:2px 0px 0px 2px;font-size:0.875rem;text-align:center;margin:0px 0px 0px 20px;height:20px;line-height:18px;min-width:0px;outline:none;padding:0px;background:transparent;color:rgb(68,68,68);border:none"><img role="button" src="https://mail.google.com/mail/u/0/images/cleardot.gif" alt="" style="background: url("") 50% 50% / 20px no-repeat; height: 20px; margin: 0px; vertical-align: middle; width: 20px; opacity: 1; display: inline-block; padding: 0px;"></div><div id="m_-5633064360383637069gmail-:67f" role="button" aria-expanded="false" aria-haspopup="true" aria-label="その他" style="display:inline-flex;border-radius:0px 2px 2px 0px;font-size:0.875rem;text-align:center;margin:0px 0px 0px 20px;height:20px;line-height:18px;min-width:0px;outline:none;padding:0px;background:transparent;color:rgb(68,68,68);border:none"><img role="menu" src="https://mail.google.com/mail/u/0/images/cleardot.gif" alt="" style="background: url("") 50% 50% / 20px no-repeat; height: 20px; width: 20px; margin: 0px; vertical-align: middle; opacity: 1; display: inline-block; padding: 0px;"></div></td></tr><tr style="display:flex;height:auto"><td colspan="3"><table cellpadding="0" style="border-collapse:collapse;table-layout:fixed;width:732.667px"><tbody><tr><td style="overflow:visible;text-overflow:ellipsis;display:flex;line-height:20px"><div style="overflow:hidden;max-width:92%;display:inline-block"><span style="vertical-align:top;color:rgb(94,94,94);font-size:0.75rem;line-height:20px">To <span name="PETSc" style="vertical-align:top">PETSc</span></span></div><div id="m_-5633064360383637069gmail-:67g" aria-haspopup="true" role="button" aria-label="詳細を表示" style="display:inline-flex;margin-left:4px;vertical-align:top;border:none;outline:none"><img src="https://mail.google.com/mail/u/0/images/cleardot.gif" alt="" style="background: url("https://ci6.googleusercontent.com/proxy/BJuLi1pdSuXlptCkICPLIHLeVbm4czRchUvNtY2K9OONpuRXMIlSWZwTSOwJD16n5dn4krDIjJWUN5XKIFx1ZnYv8ar0aMA7TesprxAt5mw7ew1XuimFnJrdNN03XZsLCrPiL76GcaqpORS5=s0-d-e1-ft#https://www.gstatic.com/images/icons/material/system_gm/2x/arrow_drop_down_black_20dp.png") 50% 50% / 20px no-repeat; padding: 0px; vertical-align: baseline; height: 20px; width: 20px; border: none; margin: 0px 0px 0px auto; display: flex; opacity: 0.71;"></div></td></tr></tbody></table></td></tr></tbody></table></div><div id="m_-5633064360383637069gmail-:67u"><div></div><div></div><div></div><div></div></div><div><div></div><div id="m_-5633064360383637069gmail-:67h"></div><div id="m_-5633064360383637069gmail-:67s" style="direction:ltr;margin:8px 0px 0px;padding:0px;font-size:0.875rem"><div id="m_-5633064360383637069gmail-:67t" style="font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:small;line-height:1.5;font-family:Arial,Helvetica,sans-serif;overflow:hidden"><div dir="ltr">Hi,<div><br></div><div>I used PETSc to assemble a FEM stiff matrix of an overlapped (overlap=2) DMPlex and used the MUMPS solver to solve it. But I got a different solution by using 1 CPU and MPI parallel computation. I am wondering if I missed some necessary step or setting during my implementation.</div><div><br></div><div>    My calling process like follows</div><div><br></div><div>1. Generate matrix pkij from DMPlex dm_mesh</div><div><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">```
      PetscCallA( DMCreateMatrix(dm_mesh, pkij, ierr) )<br>      PetscCallA( MatSetBlockSize(pkij, 1, ierr))<br>      if( overlap>0 ) call MatSetOption(pkij,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE,ierr)<br>      PetscCallA( MatSetFromOptions(pkij, ierr))<br><br>      PetscCallA(MatGetLocalSize(pkij, m, n, ierr));<br>      PetscCallA(MatGetSize(pkij, gm, gn, ierr));<br>      PetscCallA(MatGetBlockSize(pkij, bs, ierr));
```</pre><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">2. Do Matrix preallocation</pre></div><div><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">```<br>      PetscCallA(MatCreate(PETSC_COMM_WORLD, preallocator, ierr));<br>      PetscCallA(MatSetType(preallocator, MATPREALLOCATOR, ierr));<br>      PetscCallA(MatSetSizes(preallocator, m, n, gm, gn, ierr));<br>      PetscCallA(MatSetBlockSize(preallocator, bs, ierr));<br>      PetscCallA(MatSetUp(preallocator, ierr));</pre><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">      .......</pre><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">      PetscCallA( MatPreallocatorPreallocate(preallocator, PETSC_TRUE, pkij, ierr) )<br>      PetscCallA( MatDestroy(preallocator,ierr) )
```</pre><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">3. Assemble matrix pkij by calling MatSetValue of all overlapped elements.</pre><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">In my above implementation, I used MatSetOption(pkij,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE,ierr). Is that correct? Or even other options are needed?</pre></div></div></div></div></div></div></div></blockquote><div><br></div><div>Do you want to ignore off-process entries? Usually in FEM we share part of  the boundary with other processes, and thus ignoring the off-process entries will not give the correct result.</div><div><br></div><div>  Thanks,</div><div><br></div><div>     Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="margin:0px;padding:0px 0px 20px;width:732.667px;font-family:"Google Sans",Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:medium"><div><div id="m_-5633064360383637069gmail-:67s" style="direction:ltr;margin:8px 0px 0px;padding:0px;font-size:0.875rem"><div id="m_-5633064360383637069gmail-:67t" style="font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:small;line-height:1.5;font-family:Arial,Helvetica,sans-serif;overflow:hidden"><div dir="ltr"><div><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px"> </pre></div></div></div></div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="margin:0px;padding:0px 0px 20px;width:732.667px;font-family:"Google Sans",Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:medium"><div><div id="m_-5633064360383637069gmail-:67s" style="direction:ltr;margin:8px 0px 0px;padding:0px;font-size:0.875rem"><div id="m_-5633064360383637069gmail-:67t" style="font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:small;line-height:1.5;font-family:Arial,Helvetica,sans-serif;overflow:hidden"><div dir="ltr"><div><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">Thanks for your help.</pre><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">------</pre><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">X. Yuan, Ph.D. of solid mechanics</pre><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">Advancesoft, Japan</pre><pre style="box-sizing:border-box;font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:13.6px;margin-top:0.5rem;margin-bottom:16px;padding:16px;overflow:hidden auto;line-height:1.45;color:rgb(31,35,40);border-radius:6px">---------</pre></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>