<div dir="auto">Hi Miguel,</div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue 1. Oct 2024 at 07:56, MIGUEL MOLINOS PEREZ <<a href="mailto:mmolinos@us.es">mmolinos@us.es</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
<div style="line-break:after-white-space">
Thank you Matt, it works!
<div><br>
</div>
<div>The implementation is straightforward:</div>
<div>- 1º Define the paddle regions using <span style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255);color:rgb(99,99,36)">DMGetLocalBoundingBox</span> with the background DMDA mesh as an auxiliary
mesh for the domain-partitioning. </div>
<div>
<div>- 2º Create an integer to count the local number of particles to be used as ghost particle for other processors (<span style="color:rgb(0,0,0)">N_ghost</span>). One particle can be counted more than one time. At the same
time, fill two arrays:</div>
<div><span style="white-space:pre-wrap"></span>- one with the index of the "main particle” (local particle),</div>
<div><span style="white-space:pre-wrap"></span>- and the other with the target rank of the
<span style="color:rgb(0,0,0)">"main particle”</span>.<br>
<div>- 3º Create the new particles using <span style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255);color:rgb(99,99,36)">DMSwarmAddNPoints</span></div>
<div>- 4º <font style="color:rgb(0,0,0)">Fill the new particles with the information of the “main particle” but set the internal variable </font><span style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255);color:rgb(69,69,69)">DMSwarmField_rank</span><span style="color:rgb(0,0,0)"> with
the target rank.</span></div>
<div><span style="color:rgb(0,0,0)">- 5º Call </span><span style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255);color:rgb(99,99,36)">DMSwarmMigrate</span><span style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255);color:rgb(94,94,94)">(</span><span style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255)"><font style="font-family:Menlo,Monaco,"Courier New",monospace;color:rgb(126,126,126)">*</font></span><span style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255);color:rgb(94,94,94)">,</span><span style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255);color:rgb(73,104,57)">PETSC_TRUE</span><span style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255);color:rgb(94,94,94)">)</span>.
Therefore, we send the ghost particles to the corresponding processors and we delete them from the original processor. </div>
<div>- 6º Do stuff… </div>
<div>- 7º Delete ghost particles. This is very easy, we just have to call <span style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255);color:rgb(99,99,36)">DMSwarmRemovePoint</span> N_ghost times.</div>
<div><br>
</div>
<div>I think this can be easily implemented as closed routine for the DMSwarm class.</div>
<div><br>
</div>
<div>The remaining question is: how to do the communication between the “original" particle and the ghost particles? For instance, if we update some particle variable (locally) inside of a SNES context, this same variable should be updated in the ghost particles
at the other processors. </div></div></div></div></blockquote><div dir="auto"><br></div><div dir="auto">I think what you are asking about is an operation similar to VecGhostUpdate{Begin,End}(). In the case of a DMSwarm I’m not sure how to define the InsertMode = ADD_VALUES? Some swarm fields do not make sense to be added. INSERT_VALUES is fine.</div><div dir="auto"><br></div><div dir="auto">One solution might be to have something like this</div><div dir="auto"><br></div><div dir="auto">DMSwarmCollectViewUpdateGhostOwners(DM dm, InsertMode mode, PetscInt nfields, const char *fieldNames[]);</div><div dir="auto"><br></div><div dir="auto">where one can specify the insert mode and the fields on which the insert mode will apply.</div><div dir="auto"><br></div><div dir="auto">Would something like that work?</div><div dir="auto"><br></div><div dir="auto">Cheers,</div><div dir="auto">Dave</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div style="line-break:after-white-space"><div><div><div>
<div><br>
</div>
<div>PS: <span style="color:rgb(0,0,0)">Hope this helps someone in the future :-)</span></div></div></div></div></div><div style="line-break:after-white-space"><div><div><div>
<div><span style="color:rgb(0,0,0)"><br>
</span></div>
<div><br>
<blockquote type="cite">
<div>On Sep 27, 2024, at 10:50 AM, MIGUEL MOLINOS PEREZ <<a href="mailto:mmolinos@us.es" target="_blank">mmolinos@us.es</a>> wrote:</div>
<br>
<div>
<div style="line-break:after-white-space">
Thank you Matt, let me give it try.
<div><br>
</div>
<div>Miguel<br id="m_-5540017801342568302lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On Sep 27, 2024, at 3:44 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div>
<br>
<div>
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="ltr" style="font-family:Helvetica">On Thu, Sep 26, 2024 at 7:18 PM MIGUEL MOLINOS PEREZ <<a href="mailto:mmolinos@us.es" target="_blank" style="font-family:Helvetica">mmolinos@us.es</a>> wrote:<br>
</div>
<div class="gmail_quote" style="font-family:Helvetica">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;font-family:Helvetica;border-left-color:rgb(204,204,204)">
<div style="font-family:Helvetica">I see, you mean:
<div style="font-family:Helvetica"><br>
<div style="font-family:Helvetica">Create the ghost particles at the local cell with the same properties as particle 1 <span style="font-family:Helvetica">(duplicate the original particle) but different value<span style="font-family:Helvetica"> </span></span>DMSwarmField_rank<span style="font-family:Helvetica">. Then, call </span>DMSwarmMigrate(*,PETSC_FALSE)
so we do the migration and delete the local copies of the particle 1. Right? </div>
</div>
</div>
</blockquote>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">Yep. I think it will work, from what I know about BASIC.</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"> Thanks,</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"> Matt</div>
<div style="font-family:Helvetica"> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;font-family:Helvetica;border-left-color:rgb(204,204,204)">
<div style="font-family:Helvetica">
<div style="font-family:Helvetica">
<div style="font-family:Helvetica">
<div style="font-family:Helvetica">Thanks,</div>
<div style="font-family:Helvetica">Miguel</div>
<div style="font-family:Helvetica">
<div style="font-family:Helvetica"><br>
<blockquote type="cite" style="font-family:Helvetica">
<div style="font-family:Helvetica">On Sep 26, 2024, at 11:09 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" style="font-family:Helvetica">knepley@gmail.com</a>> wrote:</div>
<br>
<div style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="ltr" style="font-family:Helvetica">On Thu, Sep 26, 2024 at 11:20 AM MIGUEL MOLINOS PEREZ <<a href="mailto:mmolinos@us.es" target="_blank" style="font-family:Helvetica">mmolinos@us.es</a>> wrote:<br>
</div>
<div class="gmail_quote" style="font-family:Helvetica">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;font-family:Helvetica;border-left-color:rgb(204,204,204)">
<div style="font-family:Helvetica">
<div style="font-family:Helvetica">Thank you Matt.</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">Okey, let me have a careful look to the <span style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255);color:rgb(99,99,36)">DMSwarmMigrate_Push_Basic</span> implementation to see if there
is some workaround.</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">The idea of adding new particles is interesting. However, in that case, we need to initialize the new <span style="font-family:Helvetica">(ghost)</span> particles using the fields of the “real” particle, right? This can be done using something like:</div>
<div style="font-family:Helvetica">
<pre id="m_-5540017801342568302m_8591528946925750581m_6734545526208535891codecell72" style="box-sizing:border-box;overflow:auto;clear:both;font-size:0.875em;margin-top:0px;margin-bottom:0px;border-radius:0.25rem;line-height:17.5px;padding:1rem;font-family:monospace"><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhostUpdateBegin/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEHvOdaqj$" style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)" target="_blank">VecGhostUpdateBegin</a></span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)">(</span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEJ7VDssN$" style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)" target="_blank">Vec</a></span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)"> </span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)">globalout</span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)">,</span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertMode/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEHXxW4d9$" style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)" target="_blank">InsertMode</a></span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)"> </span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VALUES/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEAOykCv4$" style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)" target="_blank">ADD_VALUES</a></span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)">,</span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)"> </span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/ScatterMode/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAENQNA5xS$" style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)" target="_blank">ScatterMode</a></span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)"> </span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER_REVERSE/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEAN6FYSr$" style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)" target="_blank">SCATTER_REVERSE</a></span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)">);</span>
<span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhostUpdateEnd/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEOdwyTUf$" style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)" target="_blank">VecGhostUpdateEnd</a></span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)">(</span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEJ7VDssN$" style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)" target="_blank">Vec</a></span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)"> </span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)">globalout</span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)">,</span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertMode/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEHXxW4d9$" style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)" target="_blank">InsertMode</a></span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)"> </span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VALUES/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEAOykCv4$" style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)" target="_blank">ADD_VALUES</a></span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)">,</span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)"> </span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/ScatterMode/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAENQNA5xS$" style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)" target="_blank">ScatterMode</a></span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)"> </span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER_REVERSE/__;!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEAN6FYSr$" style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)" target="_blank">SCATTER_REVERSE</a></span><span style="box-sizing:border-box;font-family:monospace;color:rgb(15,15,8)">);</span></pre>
</div>
<div style="font-family:Helvetica">for the particle fields (?). </div>
</div>
</blockquote>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">I think we can just copy from the local particle. For example, suppose I decide that particle 1 should go to rank 5, 12, and 27. Then</div>
<div style="font-family:Helvetica">I first set p1.rank = 5, then I add two new particles with the same values as particle 1, but with rank = 12 and 27. Then when I call migrate, it will move these three particles to the correct processes, and delete the original particles and the copies
from the local set.</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"> Thanks,</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"> Matt</div>
<div style="font-family:Helvetica"> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;font-family:Helvetica;border-left-color:rgb(204,204,204)">
<div style="font-family:Helvetica">
<div style="font-family:Helvetica">Thanks,</div>
<div style="font-family:Helvetica">Miguel</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"><br>
<blockquote type="cite" style="font-family:Helvetica">
<div style="font-family:Helvetica">On Sep 26, 2024, at 3:53 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" style="font-family:Helvetica">knepley@gmail.com</a>> wrote:</div>
<br>
<div style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="ltr" style="font-family:Helvetica">On Thu, Sep 26, 2024 at 6:31 AM MIGUEL MOLINOS PEREZ <<a href="mailto:mmolinos@us.es" target="_blank" style="font-family:Helvetica">mmolinos@us.es</a>> wrote:<br>
</div>
<div class="gmail_quote" style="font-family:Helvetica">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;font-family:Helvetica;border-left-color:rgb(204,204,204)">
<div style="font-family:Helvetica">
<div style="font-family:Helvetica">Hi Matt et al,</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">I’ve been working on the scheme that you proposed to create ghost particles (atoms in my case), and it works! With a couple of caveats: </div>
<div style="font-family:Helvetica">-1º In general the overlap particles will be migrate from their own rank to more than one neighbor rank, this is specially relevant for those located close to the corners. Therefore, you'll need to call DMSwarmMigrate several times (27 times for 3D cells),
during the migration process. </div>
<div style="font-family:Helvetica"></div>
</div>
</blockquote>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends the particle to all overlapping neighbors at once. It can't be that hard.</div>
<div style="font-family:Helvetica"> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;font-family:Helvetica;border-left-color:rgb(204,204,204)">
<div style="font-family:Helvetica">
<div style="font-family:Helvetica">-2º You need to set <span style="font-family:Helvetica;background-color:rgb(255,255,255)"><span style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;color:rgb(73,104,57)">DMSWARM_MIGRATE_BASIC.
</span><span style="white-space:pre-wrap;font-family:Helvetica"><font style="font-family:Helvetica;color:rgb(0,0,0)">Otherwise the proposed algorithm will not work at all!</font></span></span></div>
</div>
</blockquote>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">Oh, I should have thought of that. Sorry.</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">I can help code up that extension. Can you take a quick look at the BASIC code? Right now, we just use the rank attached to the particle</div>
<div style="font-family:Helvetica">to send it. We could have an arrays of ranks, but that seems crazy, and would blow up particle storage. How about just adding new particles</div>
<div style="font-family:Helvetica">with the other ranks right before migration?</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"> Thanks,</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"> Matt</div>
<div style="font-family:Helvetica"> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;font-family:Helvetica;border-left-color:rgb(204,204,204)">
<div style="font-family:Helvetica">
<div style="font-family:Helvetica">Hope this helps to other folks!</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">I have a follow-up question about periodic bcc on this context, should I open a new thread of keep posting here?</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">Thanks,</div>
Miguel<br id="m_-5540017801342568302m_8591528946925750581m_6734545526208535891m_-1382752044991804335lineBreakAtBeginningOfMessage">
<div style="font-family:Helvetica"><br>
<blockquote type="cite" style="font-family:Helvetica">
<div style="font-family:Helvetica">On Aug 7, 2024, at 4:22 AM, MIGUEL MOLINOS PEREZ <<a href="mailto:mmolinos@us.es" target="_blank" style="font-family:Helvetica">mmolinos@us.es</a>> wrote:</div>
<br>
<div style="font-family:Helvetica">
<div style="font-family:Helvetica">
<div style="font-family:Helvetica">
<div style="font-family:Helvetica">Thanks Matt, I think I'll start by making a small program as a proof of concept. Then, if it works I'll implement it in my code and I'll be happy to share it too :-)</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">Miguel</div>
<div style="font-family:Helvetica"><br>
<blockquote type="cite" style="font-family:Helvetica">
<div style="font-family:Helvetica">On Aug 4, 2024, at 3:30 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" style="font-family:Helvetica">knepley@gmail.com</a>> wrote:</div>
<br>
<div style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="ltr" style="font-family:Helvetica">On Fri, Aug 2, 2024 at 7:15 PM MIGUEL MOLINOS PEREZ <<a href="mailto:mmolinos@us.es" target="_blank" style="font-family:Helvetica">mmolinos@us.es</a>> wrote:<br>
</div>
<div class="gmail_quote" style="font-family:Helvetica">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;font-family:Helvetica;border-left-color:rgb(204,204,204)">
<div style="font-family:Helvetica">
<div style="font-family:Helvetica">
<div style="font-family:Helvetica">Thanks again Matt, that makes a lot more sense !!</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">Just to check that we are on the same page. You are saying: </div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">1. create a field define a field called "owner rank" for each particle. </div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable. </div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles. </div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again.</div>
</div>
</div>
</blockquote>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">I don't think we need this last step. We can just remove those ghost particles for the next step I think.</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"> Thanks,</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"> Matt</div>
<div style="font-family:Helvetica"> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;font-family:Helvetica;border-left-color:rgb(204,204,204)">
<div style="font-family:Helvetica">
<div style="font-family:Helvetica">
<div style="font-family:Helvetica">Thank you,</div>
<div style="font-family:Helvetica">Miguel</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">
<blockquote type="cite" style="font-family:Helvetica">
<div style="font-family:Helvetica">On Aug 2, 2024, at 5:33 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" style="font-family:Helvetica">knepley@gmail.com</a>> wrote:</div>
<br>
<div style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="ltr" style="font-family:Helvetica">On Fri, Aug 2, 2024 at 11:15 AM MIGUEL MOLINOS PEREZ <<a href="mailto:mmolinos@us.es" target="_blank" style="font-family:Helvetica">mmolinos@us.es</a>> wrote:<br>
</div>
<div class="gmail_quote" style="font-family:Helvetica">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;font-family:Helvetica;border-left-color:rgb(204,204,204)">
<div style="font-family:Helvetica">Thank you Matt for your time,
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">What you describe seems to me the ideal approach. </div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">
<blockquote type="cite" style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">
<div class="gmail_quote" style="font-family:Helvetica">
<div style="font-family:Helvetica">1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST</div>
</div>
</div>
</blockquote>
<div style="font-family:Helvetica">This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model?</div>
</div>
</div>
</blockquote>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles.
Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">1) Setup the non-overlapping particles</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">2) Identify the halo particles</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">3) Change the `rank`, but not the `owner rank`</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">4) Call DMSwarmMigrate()</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">Now we can identify ghost particles by the `owner rank`</div>
<div style="font-family:Helvetica"> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;font-family:Helvetica;border-left-color:rgb(204,204,204)">
<div style="font-family:Helvetica">
<div style="font-family:Helvetica">
<div style="font-family:Helvetica">If that so, how to do the communication between the ghost particles living in the rank i and their “real” counterpart in the rank j.</div>
</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">Algo, as an alternative, what about:</div>
<div style="font-family:Helvetica">1) Use an IS tag which contains,<span style="font-family:Helvetica"> </span><span style="font-family:Helvetica">for each rank,</span><span style="font-family:Helvetica"> a </span>list of the global index of the neighbors particles outside of the rank. </div>
<div style="font-family:Helvetica">2) Use <span style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255);color:rgb(75,81,1)">VecCreateGhost t</span>o create a new vector which contains extra local space for the ghost components
of the vector. </div>
<div style="font-family:Helvetica">3) Use <font style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255);color:rgb(78,73,0)">VecScatterCreate,
</font><span style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255);color:rgb(78,73,0)">VecScatterBegin, and
</span><span style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255);color:rgb(78,73,0)">VecScatterEnd
</span>to do the transference of data between a vector obtained with<span style="font-family:Helvetica"> </span><span style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255);color:rgb(99,99,36)"> DMSwarmCreateGlobalVectorFromField</span> </div>
<div style="font-family:Helvetica">4) Do necessary computations using the vectors created with <span style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255);color:rgb(75,81,1)">VecCreateGhost</span>.</div>
</div>
</blockquote>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">This is essentially what Migrate() does. I was trying to reuse the code.</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"> Thanks,</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"> Matt</div>
<div style="font-family:Helvetica"> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;font-family:Helvetica;border-left-color:rgb(204,204,204)">
<div style="font-family:Helvetica">
<div style="font-family:Helvetica">Thanks,</div>
<div style="font-family:Helvetica">Miguel</div>
<div style="font-family:Helvetica">
<div style="font-family:Helvetica"><br>
<blockquote type="cite" style="font-family:Helvetica">
<div style="font-family:Helvetica">On Aug 2, 2024, at 8:58 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" style="font-family:Helvetica">knepley@gmail.com</a>> wrote:</div>
<br>
<div style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">On Thu, Aug 1, 2024 at 4:40 PM MIGUEL MOLINOS PEREZ <<a href="mailto:mmolinos@us.es" target="_blank" style="font-family:Helvetica">mmolinos@us.es</a>> wrote:<br>
</div>
<div class="gmail_quote" style="font-family:Helvetica">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;font-family:Helvetica;border-left-color:rgb(204,204,204)">
<div style="font-family:Helvetica"><u style="font-family:Helvetica"></u><u style="font-family:Helvetica"></u>
<div dir="ltr" id="m_-5540017801342568302m_8591528946925750581m_6734545526208535891m_-1382752044991804335m_6339866382597766808m_5114540859675030278m_6853013589290146639pfptBanner8j38xab" style="display:block;text-align:left;margin:16px 0px;padding:8px 16px;border-radius:4px;min-width:200px;border-top-width:4px;border-top-style:solid;font-family:Helvetica;background-color:rgb(208,216,220);border-top-color:rgb(144,164,174)">
<div id="m_-5540017801342568302m_8591528946925750581m_6734545526208535891m_-1382752044991804335m_6339866382597766808m_5114540859675030278m_6853013589290146639pfptBanner8j38xab" style="float:left;display:block;margin:0px 0px 1px;max-width:600px;font-family:Helvetica">
<div id="m_-5540017801342568302m_8591528946925750581m_6734545526208535891m_-1382752044991804335m_6339866382597766808m_5114540859675030278m_6853013589290146639pfptBanner8j38xab" style="display:block;font-family:Arial,sans-serif;font-weight:bold;font-size:14px;line-height:18px;background-color:rgb(208,216,220)">
This Message Is From an External Sender<span style="font-family:Arial,sans-serif"> </span></div>
<div id="m_-5540017801342568302m_8591528946925750581m_6734545526208535891m_-1382752044991804335m_6339866382597766808m_5114540859675030278m_6853013589290146639pfptBanner8j38xab" style="font-weight:normal;display:block;font-family:Arial,sans-serif;font-size:12px;line-height:18px;margin-top:2px;background-color:rgb(208,216,220)">
This message came from outside your organization.<span style="font-family:Arial,sans-serif"> </span></div>
</div>
<div style="height:0px;clear:both;display:block;line-height:0;font-size:0.01px;font-family:Helvetica">
</div>
</div>
<u style="font-family:Helvetica"></u>
<div style="font-family:Helvetica">
<pre style="font-family:sans-serif;font-size:12px;white-space:pre-wrap">Dear all,
I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor.
This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something?
I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions.</pre>
</div>
</div>
</blockquote>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want.</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination</div>
<div style="font-family:Helvetica"> seems application specific, so I would leave this determination to the user right now. We do two things to these particles</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"> a) Mark chosen particles as OVERLAP</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"> b) Change rank to process we are sending to</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">4) Mark OVERLAP particles as GHOST when they arrive</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica">There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this</div>
<div style="font-family:Helvetica">in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified.</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"> THanks,</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"> Matt</div>
<div style="font-family:Helvetica"> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;font-family:Helvetica;border-left-color:rgb(204,204,204)">
<div style="font-family:Helvetica">
<div style="font-family:Helvetica">
<pre style="font-family:sans-serif;font-size:12px;white-space:pre-wrap">Thank you,
Miguel
</pre>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div style="font-family:Helvetica"><br>
</div>
<span class="gmail_signature_prefix" style="font-family:Helvetica">--<span style="font-family:Helvetica"> </span></span><br>
<div dir="ltr" class="gmail_signature" style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">
<div style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">
<div style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">
<div style="font-family:Helvetica">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEAifLp3_$" target="_blank" style="font-family:Helvetica">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div style="font-family:Helvetica"><br>
</div>
<span class="gmail_signature_prefix" style="font-family:Helvetica">--<span style="font-family:Helvetica"> </span></span><br>
<div dir="ltr" class="gmail_signature" style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">
<div style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">
<div style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">
<div style="font-family:Helvetica">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEAifLp3_$" target="_blank" style="font-family:Helvetica">https://www.cse.buffalo.edu/~knepley/</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div style="font-family:Helvetica"><br>
</div>
<span class="gmail_signature_prefix" style="font-family:Helvetica">--<span style="font-family:Helvetica"> </span></span><br>
<div dir="ltr" class="gmail_signature" style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">
<div style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">
<div style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">
<div style="font-family:Helvetica">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEAifLp3_$" target="_blank" style="font-family:Helvetica">https://www.cse.buffalo.edu/~knepley/</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br clear="all">
<div style="font-family:Helvetica"><br>
</div>
<span class="gmail_signature_prefix" style="font-family:Helvetica">--<span style="font-family:Helvetica"> </span></span><br>
<div dir="ltr" class="gmail_signature" style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">
<div style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">
<div style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">
<div style="font-family:Helvetica">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEAifLp3_$" target="_blank" style="font-family:Helvetica">https://www.cse.buffalo.edu/~knepley/</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br clear="all">
<div style="font-family:Helvetica"><br>
</div>
<span class="gmail_signature_prefix" style="font-family:Helvetica">--<span style="font-family:Helvetica"> </span></span><br>
<div dir="ltr" class="gmail_signature" style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">
<div style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">
<div style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">
<div style="font-family:Helvetica">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEAifLp3_$" target="_blank" style="font-family:Helvetica">https://www.cse.buffalo.edu/~knepley/</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div style="font-family:Helvetica"><br>
</div>
<span class="gmail_signature_prefix" style="font-family:Helvetica">--<span style="font-family:Helvetica"> </span></span><br>
<div dir="ltr" class="gmail_signature" style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">
<div style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">
<div style="font-family:Helvetica">
<div dir="ltr" style="font-family:Helvetica">
<div style="font-family:Helvetica">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div style="font-family:Helvetica"><br>
</div>
<div style="font-family:Helvetica"><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bZXXf6uG-_sIhXmwCjiE6lnmtykfj5x1NmdfSKglwmtTzvFaM04Mt9F1ZU8HPt4Uc5pSoVVeWA7KbLQNFFlAEAifLp3_$" target="_blank" style="font-family:Helvetica">https://www.cse.buffalo.edu/~knepley/</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</blockquote></div></div>