<div dir="ltr"><div dir="ltr">On Mon, Oct 16, 2023 at 9:22 AM erdemguer <<a href="mailto:erdemguer@proton.me">erdemguer@proton.me</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-family:Arial,sans-serif;font-size:14px">Thank you for your responses many times. Looks like I'm missing something, sorry for my confusion, but let's take processor 0 on your first output. cEndInterior: 16 and cEnd: 24.</div><div style="font-family:Arial,sans-serif;font-size:14px">I'm calculating jacobian for cell=14, dof=0 (row = 42) and cell=18, dof=2 (col = 56) have influence on it. (Cell 18 is on processor boundary)</div><div style="font-family:Arial,sans-serif;font-size:14px">Shouldn't I have to write values on the (42,56)?</div></blockquote><div><br></div><div>Imagine you are me getting this mail. When I mail you, I show you _exactly_ what I ran and which command line options I used. You do not. I provide you all the output. You do not. You can see that someone would only be guessing when replying to this email. Also note that you have two dofs per cell, so the cell numbers are not the row numbers for the Jacobian. Please send something reproducible when you want help on running.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-family:Arial,sans-serif;font-size:14px">Thanks,</div><div style="font-family:Arial,sans-serif;font-size:14px">Guer</div><div style="font-family:Arial,sans-serif;font-size:14px"><br></div>
<div style="font-family:Arial,sans-serif;font-size:14px">
<div>
</div>
<div>
Sent with <a href="https://proton.me/" rel="noopener noreferrer" target="_blank">Proton Mail</a> secure email.
</div>
</div>
<div style="font-family:Arial,sans-serif;font-size:14px"><br></div><div>
------- Original Message -------<br>
On Monday, October 16th, 2023 at 4:11 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br><br>
<blockquote type="cite">
<div dir="ltr"><div dir="ltr">On Mon, Oct 16, 2023 at 6:54 AM erdemguer <<a href="mailto:erdemguer@proton.me" rel="noreferrer nofollow noopener" target="_blank">erdemguer@proton.me</a>> wrote:<br></div><div class="gmail_quote"><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div style="font-family:Arial,sans-serif;font-size:14px">Hey again. </div><div style="font-family:Arial,sans-serif;font-size:14px"><br></div><div style="font-family:Arial,sans-serif;font-size:14px">This code outputs for example:</div><div style="font-family:Arial,sans-serif;font-size:14px"><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace"><br></span></div><div style="font-family:Arial,sans-serif;font-size:14px"><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">After Distribution Rank: 1, cStart: 0, cEndInterior: 14, cEnd: 24</span><br></div><div style="font-family:Arial,sans-serif;font-size:14px"><span><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">After Distribution Rank: 0, cStart: 0, cEndInterior: 13, cEnd: 27</span><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[0] m: 39 n: 39</span></div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[1] m: 42 n: 42</span><br></span></div><div><span style="line-height:normal"><br></span></div><div><span><span style="font-family:Arial,sans-serif;font-size:14px;line-height:normal;font-weight:400">Shouldn't it be 39 x 81 and 42 x 72 because of the overlapping cells on processor boundaries?</span></span></div></blockquote><div><br></div><div>Here is my output</div><div><br></div><div>master *:~/Downloads/tmp/Guer$ /PETSc3/petsc/apple/bin/mpiexec -n 2 ./ex1 -malloc_debug 0 -dm_refine 1<br>Before Distribution Rank: 1, cStart: 0, cEndInterior: 0, cEnd: 0<br>Before Distribution Rank: 0, cStart: 0, cEndInterior: 32, cEnd: 32<br>After Distribution Rank: 1, cStart: 0, cEndInterior: 16, cEnd: 24<br>After Distribution Rank: 0, cStart: 0, cEndInterior: 16, cEnd: 24<br>[0] m: 48 n: 48<br>[1] m: 48 n: 48<br></div><div><br></div><div>The mesh is 4x4 and also split into two triangles, so 32 triangles. Then we split it and have 8 overlap cells on each side. You can get quads using</div><div><br></div><div>master *:~/Downloads/tmp/Guer$ /PETSc3/petsc/apple/bin/mpiexec -n 2 ./ex1 -malloc_debug 0 -dm_plex_simplex 0 -dm_refine 1 -dm_view<br>Before Distribution Rank: 1, cStart: 0, cEndInterior: 0, cEnd: 0<br>Before Distribution Rank: 0, cStart: 0, cEndInterior: 16, cEnd: 16<br>After Distribution Rank: 1, cStart: 0, cEndInterior: 8, cEnd: 12<br>After Distribution Rank: 0, cStart: 0, cEndInterior: 8, cEnd: 12<br>[0] m: 24 n: 24<br>[1] m: 24 n: 24<br></div><div> </div><div>It is the same 4x4 mesh, but now with quads.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div><br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div><span><span style="font-family:Arial,sans-serif;font-size:14px;line-height:normal;font-weight:400">P.S. It looks like I should use PetscFV or something like that at the first place. At first I thought, "I will just use SNES, I will compute only residual and jacobian on cells so why do bother with PetscFV?" So </span></span></div><div style="font-family:Arial,sans-serif;font-size:14px"><br></div><div style="font-family:Arial,sans-serif;font-size:14px">Thanks,</div><div style="font-family:Arial,sans-serif;font-size:14px">E. </div>
<div style="font-family:Arial,sans-serif;font-size:14px">
<div>
</div>
<div>
Sent with <a rel="noreferrer nofollow noopener" href="https://proton.me/" target="_blank">Proton Mail</a> secure email.
</div>
</div>
<div style="font-family:Arial,sans-serif;font-size:14px"><br></div><div>
------- Original Message -------<br>
On Friday, October 13th, 2023 at 3:00 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" rel="noreferrer nofollow noopener" target="_blank">knepley@gmail.com</a>> wrote:<br><br>
<blockquote type="cite">
<div dir="ltr"><div dir="ltr">On Fri, Oct 13, 2023 at 7:26 AM erdemguer <<a rel="noreferrer nofollow noopener" href="mailto:erdemguer@proton.me" target="_blank">erdemguer@proton.me</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-family:Arial,sans-serif;font-size:14px">Hi, unfortunately it's me again. </div><div style="font-family:Arial,sans-serif;font-size:14px"><br></div><div style="font-family:Arial,sans-serif;font-size:14px">I have some weird troubles with creating matrix with DMPlex. Actually I might not need to create matrix explicitly, but SNESSolve crashes at there too. So, I updated the code you provided. When I tried to use DMCreateMatrix() at first, I got an error "<span>Unknown discretization type for field 0" at first I applied DMSetLocalSection() and this error is gone. But this time when I run the code with multiple processors, sometimes I got an output like:</span></div></blockquote><div><br></div><div>Some setup was out of order so the section size on proc1 was 0, and I was not good about checking this.</div><div>I have fixed it and attached.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-family:Arial,sans-serif;font-size:14px"><span><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">Before Distribution Rank: 0, cStart: 0, cEndInterior: 27, cEnd: 27</span><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">Before Distribution Rank: 1, cStart: 0, cEndInterior: 0, cEnd: 0</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[1] ghost cell 14</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[1] ghost cell 15</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[1] ghost cell 16</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[1] ghost cell 17</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[1] ghost cell 18</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[1] ghost cell 19</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[1] ghost cell 20</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[1] ghost cell 21</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[1] ghost cell 22</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">After Distribution Rank: 1, cStart: 0, cEndInterior: 14, cEnd: 23</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[0] ghost cell 13</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[0] ghost cell 14</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[0] ghost cell 15</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[0] ghost cell 16</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[0] ghost cell 17</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[0] ghost cell 18</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[0] ghost cell 19</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[0] ghost cell 20</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[0] ghost cell 21</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[0] ghost cell 22</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">[0] ghost cell 23</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">After Distribution Rank: 0, cStart: 0, cEndInterior: 13, cEnd: 24</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">Fatal error in internal_Waitall: Unknown error class, error stack:</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">internal_Waitall(82)......................: MPI_Waitall(count=1, array_of_requests=0xaaaaf5f72264, array_of_statuses=0x1) failed</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">MPIR_Waitall(1099)........................:</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">MPIR_Waitall_impl(1011)...................:</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">MPIR_Waitall_state(976)...................:</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">MPIDI_CH3i_Progress_wait(187).............: an error occurred while handling an event returned by MPIDI_CH3I_Sock_Wait()</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">MPIDI_CH3I_Progress_handle_sock_event(411):</span></div><div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">ReadMoreData(744).........................: ch3|sock|immedread 0xffff8851c5c0 0xaaaaf5e81cd0 0xaaaaf5e8a880</span></div><span style="font-size:9pt;line-height:normal;font-family:Menlo,Consolas,"Courier New",monospace">MPIDI_CH3I_Sock_readv(2553)...............: the supplied buffer contains invalid memory (set=0,sock=1,errno=14:Bad address)</span><br></span></div><div style="font-family:Arial,sans-serif;font-size:14px"><br></div><div style="font-family:Arial,sans-serif;font-size:14px">Sometimes the error message isn't appearing but for example I'm trying to print size of the matrix but it isn't working.</div><div style="font-family:Arial,sans-serif;font-size:14px">If necessary, my <span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal">Configure options --download-mpich --download-hwloc --download-pastix --download-hypre --download-ml --download-ctetgen --download-triangle --download-exodusii --download-netcdf --download-zlib --download-pnetcdf --download-ptscotch --download-hdf5 --with-cc=clang-16 --with-cxx=clang++-16 COPTFLAGS="-g -O2" CXXOPTFLAGS="-g -O2" FOPTFLAGS="-g -O2" --with-debugging=1</span></div><div style="font-family:Arial,sans-serif;font-size:14px"><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal"><br></span></div><div style="font-family:Arial,sans-serif;font-size:14px"><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal">Version: <span>Petsc Release Version 3.20.0</span></span></div><div style="font-family:Arial,sans-serif;font-size:14px"><br></div><div style="font-family:Arial,sans-serif;font-size:14px">Thank you,</div><div style="font-family:Arial,sans-serif;font-size:14px">Guer</div><div style="font-family:Arial,sans-serif;font-size:14px"><br></div>
<div style="font-family:Arial,sans-serif;font-size:14px">
<div>
</div>
<div>
Sent with <a href="https://proton.me/" rel="noreferrer nofollow noopener" target="_blank">Proton Mail</a> secure email.
</div>
</div>
<div style="font-family:Arial,sans-serif;font-size:14px"><br></div><div>
------- Original Message -------<br>
On Thursday, October 12th, 2023 at 12:59 AM, erdemguer <<a rel="noreferrer nofollow noopener" href="mailto:erdemguer@proton.me" target="_blank">erdemguer@proton.me</a>> wrote:<br><br>
<blockquote type="cite">
<div style="font-family:Arial,sans-serif;font-size:14px"><span style="text-decoration:none;background-color:rgb(255,255,255);display:inline">Thank you! That's exactly what I need.</span><br></div><div style="font-family:Arial,sans-serif;font-size:14px"><br></div>
<div style="font-family:Arial,sans-serif;font-size:14px">
<div>
</div>
<div>
Sent with <a rel="noreferrer nofollow noopener" href="https://proton.me/" target="_blank">Proton Mail</a> secure email.
</div>
</div>
<div style="font-family:Arial,sans-serif;font-size:14px"><br></div><div>
------- Original Message -------<br>
On Wednesday, October 11th, 2023 at 4:17 PM, Matthew Knepley <<a rel="noreferrer nofollow noopener" href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br><br>
<blockquote type="cite">
<div dir="ltr"><div dir="ltr">On Wed, Oct 11, 2023 at 4:42 AM erdemguer <<a rel="noreferrer nofollow noopener" href="mailto:erdemguer@proton.me" target="_blank">erdemguer@proton.me</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-family:Arial,sans-serif;font-size:14px">Hi again, </div><div style="font-family:Arial,sans-serif;font-size:14px"></div></blockquote><div><br></div><div>I see the problem. FV ghosts mean extra boundary cells added in FV methods using DMPlexCreateGhostCells() in order to impose boundary conditions. They are not the "ghost" cells for overlapping parallel decompositions. I have changed your code to give you what you want. It is attached.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-family:Arial,sans-serif;font-size:14px">Here is my code:</div><div style="font-family:Arial,sans-serif;font-size:14px"><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal">#include <petsc.h></span><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal">static char help[] = "dmplex";</span></div><div><br></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal">int main(int argc, char **argv)</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal">{</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscCall(PetscInitialize(&argc, &argv, NULL, help));</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> DM dm, dm_dist;</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscSection section;</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscInt cStart, cEndInterior, cEnd, rank;</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscInt nc[3] = {3, 3, 3};</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscReal upper[3] = {1, 1, 1};</span></div><div><br></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscCallMPI(MPI_Comm_rank(PETSC_COMM_WORLD, &rank));</span></div><div><br></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> DMPlexCreateBoxMesh(PETSC_COMM_WORLD, 3, PETSC_FALSE, nc, NULL, upper, NULL, PETSC_TRUE, &dm);</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> DMViewFromOptions(dm, NULL, "-dm1_view");</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscCall(DMSetFromOptions(dm));</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> DMViewFromOptions(dm, NULL, "-dm2_view");</span></div><div><br></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscCall(DMPlexGetDepthStratum(dm, 3, &cStart, &cEnd));</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> DMPlexComputeCellTypes(dm);</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscCall(DMPlexGetCellTypeStratum(dm, DM_POLYTOPE_INTERIOR_GHOST, &cEndInterior, NULL));</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscPrintf(PETSC_COMM_SELF, "Before Distribution Rank: %d, cStart: %d, cEndInterior: %d, cEnd: %d\n", rank, cStart,</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> cEndInterior, cEnd);</span></div><div><br></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscInt nField = 1, nDof = 3, field = 0;</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscCall(PetscSectionCreate(PETSC_COMM_WORLD, §ion));</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscSectionSetNumFields(section, nField);</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscCall(PetscSectionSetChart(section, cStart, cEnd));</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> for (PetscInt p = cStart; p < cEnd; p++)</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> {</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscCall(PetscSectionSetFieldDof(section, p, field, nDof));</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscCall(PetscSectionSetDof(section, p, nDof));</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> }</span></div><div><br></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscCall(PetscSectionSetUp(section));</span></div><div><br></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> DMSetLocalSection(dm, section);</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> DMViewFromOptions(dm, NULL, "-dm3_view");</span></div><div><br></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> DMSetAdjacency(dm, field, PETSC_TRUE, PETSC_TRUE);</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> DMViewFromOptions(dm, NULL, "-dm4_view");</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscCall(DMPlexDistribute(dm, 1, NULL, &dm_dist));</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> if (dm_dist)</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> {</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> DMDestroy(&dm);</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> dm = dm_dist;</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> }</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:10px;text-decoration:none;background-color:rgb(255,255,255);display:inline"> DMViewFromOptions(dm, NULL, "-dm5_view");</span><br></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscCall(DMPlexGetDepthStratum(dm, 3, &cStart, &cEnd));</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> DMPlexComputeCellTypes(dm);</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscCall(DMPlexGetCellTypeStratum(dm, DM_POLYTOPE_FV_GHOST, &cEndInterior, NULL));</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscPrintf(PETSC_COMM_SELF, "After Distribution Rank: %d, cStart: %d, cEndInterior: %d, cEnd: %d\n", rank, cStart,</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> cEndInterior, cEnd);</span></div><div><br></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> DMDestroy(&dm);</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal"> PetscCall(PetscFinalize());</span></div><span></span><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:7.5pt;line-height:normal">}</span></div><div style="font-family:Arial,sans-serif;font-size:14px"><br></div><div style="font-family:Arial,sans-serif;font-size:14px">This codes output is currently (on 2 processors) is:</div><div style="font-family:Arial,sans-serif;font-size:14px"><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal">Before Distribution Rank: 1, cStart: 0, cEndInterior: -1, cEnd: 14</span><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal">Before Distribution Rank: 0, cStart: 0, cEndInterior: -1, cEnd: 13</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal">After Distribution Rank: 0, cStart: 0, cEndInterior: -1, cEnd: 27</span></div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal">After Distribution Rank: 1, cStart: 0, cEndInterior: -1, cEnd: 24</span><br></div><div style="font-family:Arial,sans-serif;font-size:14px"><br></div><div style="font-family:Arial,sans-serif;font-size:14px">DMView outputs:</div><div style="font-family:Arial,sans-serif;font-size:14px">dm1_view (after creation):</div><div style="font-family:Arial,sans-serif;font-size:14px"><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal">DM Object: 2 MPI processes</span><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal"> type: plex</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal">DM_0x84000004_0 in 3 dimensions:</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal"> Number of 0-cells per rank: 64 0</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal"> Number of 1-cells per rank: 144 0</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal"> Number of 2-cells per rank: 108 0</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal"> Number of 3-cells per rank: 27 0</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal">Labels:</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal"> marker: 1 strata with value/size (1 (218))</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal"> Face Sets: 6 strata with value/size (6 (9), 5 (9), 3 (9), 4 (9), 1 (9), 2 (9))</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal"> depth: 4 strata with value/size (0 (64), 1 (144), 2 (108), 3 (27))</span></div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal"> celltype: 4 strata with value/size (7 (27), 0 (64), 4 (108), 1 (144))</span><br></div><div style="font-family:Arial,sans-serif;font-size:14px"><br></div><div style="font-family:Arial,sans-serif;font-size:14px">dm2_view (after setfromoptions):</div><div style="font-family:Arial,sans-serif;font-size:14px"><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal">DM Object: 2 MPI processes</span><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal"> type: plex</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal">DM_0x84000004_0 in 3 dimensions:</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal"> Number of 0-cells per rank: 40 46</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal"> Number of 1-cells per rank: 83 95</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal"> Number of 2-cells per rank: 57 64</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal"> Number of 3-cells per rank: 13 14</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal">Labels:</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal"> depth: 4 strata with value/size (0 (40), 1 (83), 2 (57), 3 (13))</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal"> marker: 1 strata with value/size (1 (109))</span></div><div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal"> Face Sets: 5 strata with value/size (1 (6), 2 (1), 3 (7), 5 (5), 6 (4))</span></div><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal"> celltype: 4 strata with value/size (0 (40), 1 (83), 4 (57), 7 (13))</span><br></div><div style="font-family:Arial,sans-serif;font-size:14px"><span style="font-family:Menlo,Consolas,"Courier New",monospace;font-size:9pt;line-height:normal"><br></span></div><div><span style="line-height:normal;font-family:Arial,sans-serif;font-size:14px;font-weight:400">dm3_view (after setting local section):</span></div><div><span style="line-height:normal;font-family:Arial,sans-serif;font-size:14px;font-weight:400"><span style="font-size:9pt;font-family:Menlo,Consolas,"Courier New",monospace">DM Object: 2 MPI processes</span><div><span style="font-size:9pt;font-family:Menlo,Consolas,"Courier New",monospace"> type: plex</span></div><div><span style="font-size:9pt;font-family:Menlo,Consolas,"Courier New",monospace">DM_0x84000004_0 in 3 dimensions:</span></div><div><span style="font-size:9pt;font-family:Menlo,Consolas,"Courier New",monospace"> Number of 0-cells per rank: 40 46</span></div><div><span style="font-size:9pt;font-family:Menlo,Consolas,"Courier New",monospace"> Number of 1-cells per rank: 83 95</span></div><div><span style="font-size:9pt;font-family:Menlo,Consolas,"Courier New",monospace"> Number of 2-cells per rank: 57 64</span></div><div><span style="font-size:9pt;font-family:Menlo,Consolas,"Courier New",monospace"> Number of 3-cells per rank: 13 14</span></div><div><span style="font-size:9pt;font-family:Menlo,Consolas,"Courier New",monospace">Labels:</span></div><div><span style="font-size:9pt;font-family:Menlo,Consolas,"Courier New",monospace"> depth: 4 strata with value/size (0 (40), 1 (83), 2 (57), 3 (13))</span></div><div><span style="font-size:9pt;font-family:Menlo,Consolas,"Courier New",monospace"> marker: 1 strata with value/size (1 (109))</span></div><div><span style="font-size:9pt;font-family:Menlo,Consolas,"Courier New",monospace"> Face Sets: 5 strata with value/size (1 (6), 2 (1), 3 (7), 5 (5), 6 (4))</span></div><div><span style="font-size:9pt;font-family:Menlo,Consolas,"Courier New",monospace"> celltype: 4 strata with value/size (0 (40), 1 (83), 4 (57), 7 (13))</span></div><div><span style="font-size:9pt;font-family:Menlo,Consolas,"Courier New",monospace">Field Field_0:</span></div><span style="font-size:9pt;font-family:Menlo,Consolas,"Courier New",monospace"> adjacency FEM</span><br></span></div><div><span style="line-height:normal;font-family:Arial,sans-serif;font-size:14px;font-weight:400"><span style="font-size:9pt;font-family:Menlo,Consolas,"Courier New",monospace"><br></span></span></div><div><span style="line-height:normal;font-family:Arial,sans-serif;font-size:14px;font-weight:400">dm4_view (after setting adjacency):</span></div><div><span style="line-height:normal;font-family:Arial,sans-serif;font-size:14px;font-weight:400"><span style="font-size:9pt;font-family:Menlo,Consolas,"Courier New",monospace"><span>DM Object: 2 MPI processes</span><div><span> type: plex</span></div><div><span>DM_0x84000004_0 in 3 dimensions:</span></div><div><span> Number of 0-cells per rank: 40 46</span></div><div><span> Number of 1-cells per rank: 83 95</span></div><div><span> Number of 2-cells per rank: 57 64</span></div><div><span> Number of 3-cells per rank: 13 14</span></div><div><span>Labels:</span></div><div><span> depth: 4 strata with value/size (0 (40), 1 (83), 2 (57), 3 (13))</span></div><div><span> marker: 1 strata with value/size (1 (109))</span></div><div><span> Face Sets: 5 strata with value/size (1 (6), 2 (1), 3 (7), 5 (5), 6 (4))</span></div><div><span> celltype: 4 strata with value/size (0 (40), 1 (83), 4 (57), 7 (13))</span></div><div><span>Field Field_0:</span></div><span> adjacency FVM++</span><br></span></span></div><div><span style="line-height:normal;font-family:Arial,sans-serif;font-size:14px;font-weight:400"><span style="font-size:9pt;font-family:Menlo,Consolas,"Courier New",monospace"><span><br></span></span></span></div><div><span style="line-height:normal;font-family:Arial,sans-serif;font-size:14px;font-weight:400">dm5_view (after distribution):</span></div><div><span style="line-height:normal;font-family:Arial,sans-serif;font-size:14px;font-weight:400"><span style="font-size:9pt;font-family:Menlo,Consolas,"Courier New",monospace"><span><span>DM Object: Parallel Mesh 2 MPI processes</span><div><span> type: plex</span></div><div><span>Parallel Mesh in 3 dimensions:</span></div><div><span> Number of 0-cells per rank: 64 60</span></div><div><span> Number of 1-cells per rank: 144 133</span></div><div><span> Number of 2-cells per rank: 108 98</span></div><div><span> Number of 3-cells per rank: 27 24</span></div><div><span>Labels:</span></div><div><span> depth: 4 strata with value/size (0 (64), 1 (144), 2 (108), 3 (27))</span></div><div><span> marker: 1 strata with value/size (1 (218))</span></div><div><span> Face Sets: 6 strata with value/size (1 (9), 2 (9), 3 (9), 4 (9), 5 (9), 6 (9))</span></div><div><span> celltype: 4 strata with value/size (0 (64), 1 (144), 4 (108), 7 (27))</span></div><div><span>Field Field_0:</span></div><span> adjacency FVM++</span><br></span></span></span></div><div style="font-family:Arial,sans-serif;font-size:14px"><br></div><div style="font-family:Arial,sans-serif;font-size:14px">Thanks,</div><div style="font-family:Arial,sans-serif;font-size:14px">Guer.</div>
<div style="font-family:Arial,sans-serif;font-size:14px">
<div>
</div>
<div>
Sent with <a href="https://proton.me/" rel="noreferrer nofollow noopener" target="_blank">Proton Mail</a> secure email.
</div>
</div>
<div style="font-family:Arial,sans-serif;font-size:14px"><br></div><div>
------- Original Message -------<br>
On Wednesday, October 11th, 2023 at 3:33 AM, Matthew Knepley <<a rel="noreferrer nofollow noopener" href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br><br>
<blockquote type="cite">
<div dir="ltr"><div dir="ltr">On Tue, Oct 10, 2023 at 7:01 PM erdemguer <<a href="mailto:erdemguer@proton.me" rel="noreferrer nofollow noopener" target="_blank">erdemguer@proton.me</a>> wrote:<br></div><div class="gmail_quote"><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div style="font-family:Arial,sans-serif;font-size:14px"><br></div><div style="font-family:Arial,sans-serif;font-size:14px">Hi, </div><div style="font-family:Arial,sans-serif;font-size:14px">Sorry for my late response. I tried with your suggestions and I think I made a progress. But I still got issues. Let me explain my latest mesh routine:</div><div style="font-family:Arial,sans-serif;font-size:14px"><br></div><div><ol><li style="font-family:Arial,sans-serif;font-size:14px;list-style-type:"1) ""><span><span>DMPlexCreateBoxMesh</span><br></span></li><li style="font-family:Arial,sans-serif;font-size:14px;list-style-type:"2) ""><span><span>DMSetFromOptions</span></span></li><li style="font-family:Arial,sans-serif;font-size:14px;list-style-type:"3) ""><span><span>PetscSectionCreate</span></span></li><li style="font-family:Arial,sans-serif;font-size:14px;list-style-type:"4) "">PetscSectionSetNumFields</li><li style="font-family:Arial,sans-serif;font-size:14px;list-style-type:"5) ""><span style="text-decoration:none;display:inline;background-color:rgb(255,255,255)">PetscSectionSetFieldDof</span><br></li><li style="font-family:Arial,sans-serif;font-size:14px;list-style-type:"6) ""><span style="text-decoration:none;display:inline;background-color:rgb(255,255,255)">PetscSectionSetDof</span><br></li><li style="font-family:Arial,sans-serif;font-size:14px;list-style-type:"7) "">PetscSectionSetUp</li><li style="font-family:Arial,sans-serif;font-size:14px;list-style-type:"8) "">DMSetLocalSection</li><li style="font-family:Arial,sans-serif;font-size:14px;list-style-type:"9) "">DMSetAdjacency</li><li style="font-family:Arial,sans-serif;font-size:14px;list-style-type:"10) "">DMPlexDistribute</li></ol><div style="font-family:Arial,sans-serif;font-size:14px;line-height:30px"></div><div style="font-family:Arial,sans-serif;font-size:14px;line-height:30px"><br></div></div><div style="line-height:30px"><font face="Arial, sans-serif">It's still not working but it's promising, if I call DMPlexGetDepthStratum for cells, I can see that after distribution processors have more cells.</font></div></blockquote><div><br></div><div>Please send the output of DMPlexView() for each incarnation of the mesh. What I do is put</div><div><br></div><div> DMViewFromOptions(dm, NULL, "-dm1_view")</div><div><br></div><div><br></div><div>with a different string after each call.</div><div> </div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div style="line-height:30px"><font face="Arial, sans-serif"> But I couldn't figure out how to decide where the ghost/processor boundary cells start.</font></div></blockquote><div><br></div><div>Please send the actual code because the above is not specific enough. For example, you will not have</div><div>"ghost cells" unless you partition with overlap. This is because by default cells are the partitioned quantity,</div><div>so each process gets a unique set.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div style="line-height:30px"><font face="Arial, sans-serif"> In older mails I saw there is a function <span style="text-decoration:none;display:inline;background-color:rgb(255,255,255)">DMPlexGetHybridBounds but I think that function is deprecated. I tried to use, </span><span style="text-decoration:none"><span>DMPlexGetCellTypeStratum</span><span> </span>as in ts/tutorials/ex11_sa.c but I'm getting -1 as cEndInterior before and after distribution. I tried it for <span>DM_POLYTOPE_FV_GHOST, </span><span>DM_POLYTOPE_INTERIOR_GHOST polytope types. I also tried calling DMPlexComputeCellTypes before DMPlexGetCellTypeStratum but nothing changed. I think I can calculate the ghost cell indices using cStart/cEnd before & after distribution but I think there is a better way I'm currently missing.</span></span></font></div><div style="line-height:30px"><br></div><div style="line-height:30px"><font face="Arial, sans-serif"><span style="text-decoration:none"><span>Thanks again,</span></span></font></div><div style="line-height:30px"><font face="Arial, sans-serif"><span style="text-decoration:none"><span>Guer.</span></span></font></div>
<div style="font-family:Arial,sans-serif;font-size:14px">
<div>
</div>
<div>
</div>
</div>
<div style="font-family:Arial,sans-serif;font-size:14px"><br></div><div>
------- Original Message -------<br>
On Thursday, September 28th, 2023 at 10:42 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" rel="noreferrer nofollow noopener" target="_blank">knepley@gmail.com</a>> wrote:<br><br>
<blockquote type="cite">
<div dir="ltr"><div dir="ltr">On Thu, Sep 28, 2023 at 3:38 PM erdemguer via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" rel="noreferrer nofollow noopener" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br></div><div class="gmail_quote"><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div style="font-family:Arial,sans-serif;font-size:14px"><div style="font-size:16px;white-space:pre-wrap;text-decoration:none;border:0px solid rgb(217,217,227);box-sizing:border-box;margin:20px 0px;font-family:Söhne,ui-sans-serif,system-ui,-apple-system,"Segoe UI",Roboto,Ubuntu,Cantarell,"Noto Sans",sans-serif,"Helvetica Neue",Arial,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";color:rgb(209,213,219)"><span></span></div><span style="font-family:"Trebuchet MS",sans-serif">Hi,</span><div><br></div><div><span style="font-family:"Trebuchet MS",sans-serif">I am currently using DMPlex in my code. It runs serially at the moment, but I'm interested in adding parallel options. Here is my workflow:</span></div><div><br></div><div><span style="font-family:"Trebuchet MS",sans-serif">Create a DMPlex mesh from GMSH.</span></div><div><span style="font-family:"Trebuchet MS",sans-serif">Reorder it with DMPlexPermute.</span></div><div><span style="font-family:"Trebuchet MS",sans-serif">Create necessary pre-processing arrays related to the mesh/problem.</span></div><div><span style="font-family:"Trebuchet MS",sans-serif">Create field(s) with multi-dofs.</span></div><div><span style="font-family:"Trebuchet MS",sans-serif">Create residual vectors.</span></div><div><span style="font-family:"Trebuchet MS",sans-serif">Define a function to calculate the residual for each cell and, use SNES.</span></div><div><span style="font-family:"Trebuchet MS",sans-serif">As you can see, I'm not using FV or FE structures (most examples do). Now, I'm trying to implement this in parallel using a similar approach. However, I'm struggling to understand how to create corresponding vectors and how to obtain index sets for each processor. Is there a tutorial or paper that covers this topic?</span></div></div></blockquote><div><br></div><div>The intention was that there is enough information in the manual to do this.</div><div><br></div><div>Using PetscFE/PetscFV is not required. However, I strongly encourage you to use PetscSection. Without this, it would be incredibly hard to do what you want. Once the DM has a Section, it can do things like automatically create vectors and matrices for you. It can redistribute them, subset them, etc. The Section describes how dofs are assigned to pieces of the mesh (mesh points). This is in the manual, and there are a few examples that do it by hand.</div><div><br></div><div>So I suggest changing your code to use PetscSection, and then letting us know if things still do not work.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div style="font-family:Arial,sans-serif;font-size:14px"><span style="font-family:"Trebuchet MS",sans-serif">Thank you.</span><br></div><div style="font-family:Arial,sans-serif;font-size:14px"><span style="font-family:"Trebuchet MS",sans-serif">Guer.</span></div><div style="font-family:Arial,sans-serif;font-size:14px"><br></div>
<div style="font-family:Arial,sans-serif;font-size:14px">
<div>
</div>
<div>
Sent with <a rel="noreferrer nofollow noopener" href="https://proton.me/" target="_blank">Proton Mail</a> secure email.
</div>
</div>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div class="gmail_signature" dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer nofollow noopener" target="_blank">https://www.cse.buffalo.edu/~knepley/</a></div><div><br></div></div></div></div></div></div></div></div>
</blockquote><br>
</div></blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div class="gmail_signature" dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer nofollow noopener" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote><br>
</div></blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a rel="noreferrer nofollow noopener" href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote><br>
</div>
</blockquote><br>
</div></blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a rel="noreferrer nofollow noopener" href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote><br>
</div></blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div class="gmail_signature" dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer nofollow noopener" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote><br>
</div></blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>