<div dir="ltr"><div dir="ltr">On Thu, Sep 26, 2024 at 11:20 AM MIGUEL MOLINOS PEREZ <<a href="mailto:mmolinos@us.es">mmolinos@us.es</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Thank you Matt.</div>
<div><br>
</div>
<div>Okey, let me have a careful look to the <span style="color:rgb(99,99,36);font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255)">DMSwarmMigrate_Push_Basic</span> implementation
to see if there is some workaround.</div>
<div><br>
</div>
<div>The idea of adding new particles is interesting. However, in that case, we need to initialize the new <span style="color:rgb(0,0,0)">(ghost)</span> particles using the fields of the “real” particle, right? This can be done
using something like:</div>
<div>
<pre id="m_6734545526208535891codecell72" style="box-sizing:border-box;overflow:auto;clear:both;font-size:0.875em;margin-top:0px;margin-bottom:0px;border-radius:0.25rem;line-height:17.5px;padding:1rem"><span style="box-sizing:border-box;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhostUpdateBegin/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgp0DqZlu$" style="box-sizing:border-box;color:rgb(15,15,8)" target="_blank">VecGhostUpdateBegin</a></span><span style="box-sizing:border-box;color:rgb(15,15,8)">(</span><span style="box-sizing:border-box;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgod4XH0M$" style="box-sizing:border-box;color:rgb(15,15,8)" target="_blank">Vec</a></span><span style="box-sizing:border-box;color:rgb(15,15,8)"> </span><span style="box-sizing:border-box;color:rgb(15,15,8)">globalout</span><span style="box-sizing:border-box;color:rgb(15,15,8)">,</span><span style="box-sizing:border-box;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertMode/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgo6Ycd6D$" style="box-sizing:border-box;color:rgb(15,15,8)" target="_blank">InsertMode</a></span><span style="box-sizing:border-box;color:rgb(15,15,8)"> </span><span style="box-sizing:border-box;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VALUES/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgqRE6BHz$" style="box-sizing:border-box;color:rgb(15,15,8)" target="_blank">ADD_VALUES</a></span><span style="box-sizing:border-box;color:rgb(15,15,8)">,</span><span style="box-sizing:border-box;color:rgb(15,15,8)"> </span><span style="box-sizing:border-box;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/ScatterMode/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNglNJk_18$" style="box-sizing:border-box;color:rgb(15,15,8)" target="_blank">ScatterMode</a></span><span style="box-sizing:border-box;color:rgb(15,15,8)"> </span><span style="box-sizing:border-box;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER_REVERSE/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgvURZcIx$" style="box-sizing:border-box;color:rgb(15,15,8)" target="_blank">SCATTER_REVERSE</a></span><span style="box-sizing:border-box;color:rgb(15,15,8)">);</span>
<span style="box-sizing:border-box;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhostUpdateEnd/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgnpOD3Mg$" style="box-sizing:border-box;color:rgb(15,15,8)" target="_blank">VecGhostUpdateEnd</a></span><span style="box-sizing:border-box;color:rgb(15,15,8)">(</span><span style="box-sizing:border-box;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgod4XH0M$" style="box-sizing:border-box;color:rgb(15,15,8)" target="_blank">Vec</a></span><span style="box-sizing:border-box;color:rgb(15,15,8)"> </span><span style="box-sizing:border-box;color:rgb(15,15,8)">globalout</span><span style="box-sizing:border-box;color:rgb(15,15,8)">,</span><span style="box-sizing:border-box;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertMode/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgo6Ycd6D$" style="box-sizing:border-box;color:rgb(15,15,8)" target="_blank">InsertMode</a></span><span style="box-sizing:border-box;color:rgb(15,15,8)"> </span><span style="box-sizing:border-box;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VALUES/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgqRE6BHz$" style="box-sizing:border-box;color:rgb(15,15,8)" target="_blank">ADD_VALUES</a></span><span style="box-sizing:border-box;color:rgb(15,15,8)">,</span><span style="box-sizing:border-box;color:rgb(15,15,8)"> </span><span style="box-sizing:border-box;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/ScatterMode/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNglNJk_18$" style="box-sizing:border-box;color:rgb(15,15,8)" target="_blank">ScatterMode</a></span><span style="box-sizing:border-box;color:rgb(15,15,8)"> </span><span style="box-sizing:border-box;color:rgb(15,15,8)"><a href="https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER_REVERSE/__;!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgvURZcIx$" style="box-sizing:border-box;color:rgb(15,15,8)" target="_blank">SCATTER_REVERSE</a></span><span style="box-sizing:border-box;color:rgb(15,15,8)">);</span></pre>
</div>
<div>for the particle fields (?). </div></div></blockquote><div><br></div><div>I think we can just copy from the local particle. For example, suppose I decide that particle 1 should go to rank 5, 12, and 27. Then</div><div>I first set p1.rank = 5, then I add two new particles with the same values as particle 1, but with rank = 12 and 27. Then when I call migrate, it will move these three particles to the correct processes, and delete the original particles and the copies from the local set.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div>Thanks,</div>
<div>Miguel</div>
<div><br>
</div>
<div><br>
<blockquote type="cite">
<div>On Sep 26, 2024, at 3:53 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div>
<br>
<div>
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="ltr">On Thu, Sep 26, 2024 at 6:31 AM MIGUEL MOLINOS PEREZ <<a href="mailto:mmolinos@us.es" target="_blank">mmolinos@us.es</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hi Matt et al,</div>
<div><br>
</div>
<div>I’ve been working on the scheme that you proposed to create ghost particles (atoms in my case), and it works! With a couple of caveats: </div>
<div>-1º In general the overlap particles will be migrate from their own rank to more than one neighbor rank, this is specially relevant for those located close to the corners. Therefore, you'll need to call DMSwarmMigrate several times (27 times for 3D cells),
during the migration process. </div>
<div></div>
</div>
</blockquote>
<div><br>
</div>
<div>That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends the particle to all overlapping neighbors at once. It can't be that hard.</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>-2º You need to set <span style="background-color:rgb(255,255,255)"><span style="color:rgb(73,104,57);font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap">DMSWARM_MIGRATE_BASIC.
</span><span style="white-space:pre-wrap"><font>Otherwise the proposed algorithm will not work at all!</font></span></span></div>
</div>
</blockquote>
<div><br>
</div>
<div>Oh, I should have thought of that. Sorry.</div>
<div><br>
</div>
<div>I can help code up that extension. Can you take a quick look at the BASIC code? Right now, we just use the rank attached to the particle</div>
<div>to send it. We could have an arrays of ranks, but that seems crazy, and would blow up particle storage. How about just adding new particles</div>
<div>with the other ranks right before migration?</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hope this helps to other folks!</div>
<div><br>
</div>
<div>I have a follow-up question about periodic bcc on this context, should I open a new thread of keep posting here?</div>
<div><br>
</div>
<div>Thanks,</div>
Miguel<br id="m_6734545526208535891m_-1382752044991804335lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On Aug 7, 2024, at 4:22 AM, MIGUEL MOLINOS PEREZ <<a href="mailto:mmolinos@us.es" target="_blank">mmolinos@us.es</a>> wrote:</div>
<br>
<div>
<div>
<div>
<div>Thanks Matt, I think I'll start by making a small program as a proof of concept. Then, if it works I'll implement it in my code and I'll be happy to share it too :-)</div>
<div><br>
</div>
<div>Miguel</div>
<div><br>
<blockquote type="cite">
<div>On Aug 4, 2024, at 3:30 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div>
<br>
<div>
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="ltr">On Fri, Aug 2, 2024 at 7:15 PM MIGUEL MOLINOS PEREZ <<a href="mailto:mmolinos@us.es" target="_blank">mmolinos@us.es</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>
<div>Thanks again Matt, that makes a lot more sense !!</div>
<div><br>
</div>
<div>Just to check that we are on the same page. You are saying: </div>
<div><br>
</div>
<div>1. create a field define a field called "owner rank" for each particle. </div>
<div><br>
</div>
<div>2. Identify the phantom particles and modify the internal variable defined by the DMSwarmField_rank variable. </div>
<div><br>
</div>
<div>3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the new local vector including the ghost particles. </div>
<div><br>
</div>
<div>4. Then, once the calculations are done, rename the DMSwarmField_rank variable using the "owner rank" variable and call DMSwarmMigrate(*,PETSC_FALSE) once again.</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I don't think we need this last step. We can just remove those ghost particles for the next step I think.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>
<div>Thank you,</div>
<div>Miguel</div>
<div><br>
</div>
<div><br>
</div>
<div>
<blockquote type="cite">
<div>On Aug 2, 2024, at 5:33 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div>
<br>
<div>
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="ltr">On Fri, Aug 2, 2024 at 11:15 AM MIGUEL MOLINOS PEREZ <<a href="mailto:mmolinos@us.es" target="_blank">mmolinos@us.es</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>Thank you Matt for your time,
<div><br>
</div>
<div>What you describe seems to me the ideal approach. </div>
<div><br>
</div>
<div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div>1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST</div>
</div>
</div>
</blockquote>
<div>This means, locally, I need to allocate Nlocal + ghost particles (duplicated) for my model?</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I would do it another way. I would allocate the particles with no overlap and set them up. Then I would identify the halo particles, mark them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as GHOST, then unmark the OVERLAP particles.
Shoot! That marking will not work since we cannot tell the difference between particles we received and particles we sent. Okay, instead of the `ghost` field we need an `owner rank` field. So then we</div>
<div><br>
</div>
<div>1) Setup the non-overlapping particles</div>
<div><br>
</div>
<div>2) Identify the halo particles</div>
<div><br>
</div>
<div>3) Change the `rank`, but not the `owner rank`</div>
<div><br>
</div>
<div>4) Call DMSwarmMigrate()</div>
<div><br>
</div>
<div>Now we can identify ghost particles by the `owner rank`</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>
<div>If that so, how to do the communication between the ghost particles living in the rank i and their “real” counterpart in the rank j.</div>
</div>
<div><br>
</div>
<div>Algo, as an alternative, what about:</div>
<div>1) Use an IS tag which contains,<span> </span><span>for each rank,</span><span> a </span>list of the global index of the neighbors particles outside of the rank. </div>
<div>2) Use <span style="color:rgb(75,81,1);font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255)">VecCreateGhost t</span>o create a new vector which contains extra local space for the ghost components
of the vector. </div>
<div>3) Use <font color="#4e4900" style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255)">VecScatterCreate,
</font><span style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255);color:rgb(78,73,0)">VecScatterBegin, and
</span><span style="font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255);color:rgb(78,73,0)">VecScatterEnd
</span>to do the transference of data between a vector obtained with<span> </span><span style="color:rgb(99,99,36);font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255)"> DMSwarmCreateGlobalVectorFromField</span> </div>
<div>4) Do necessary computations using the vectors created with <span style="color:rgb(75,81,1);font-family:Menlo,Monaco,"Courier New",monospace;white-space:pre-wrap;background-color:rgb(255,255,255)">VecCreateGhost</span>.</div>
</div>
</blockquote>
<div><br>
</div>
<div>This is essentially what Migrate() does. I was trying to reuse the code.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Thanks,</div>
<div>Miguel</div>
<div>
<div><br>
<blockquote type="cite">
<div>On Aug 2, 2024, at 8:58 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div>
<br>
<div>
<div dir="ltr">
<div dir="ltr">On Thu, Aug 1, 2024 at 4:40 PM MIGUEL MOLINOS PEREZ <<a href="mailto:mmolinos@us.es" target="_blank">mmolinos@us.es</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div><u></u><u></u>
<div dir="ltr" id="m_6734545526208535891m_-1382752044991804335m_6339866382597766808m_5114540859675030278m_6853013589290146639pfptBanner8j38xab" style="display:block;text-align:left;margin:16px 0px;padding:8px 16px;border-radius:4px;min-width:200px;background-color:rgb(208,216,220);border-top:4px solid rgb(144,164,174)">
<div id="m_6734545526208535891m_-1382752044991804335m_6339866382597766808m_5114540859675030278m_6853013589290146639pfptBanner8j38xab" style="float:left;display:block;margin:0px 0px 1px;max-width:600px">
<div id="m_6734545526208535891m_-1382752044991804335m_6339866382597766808m_5114540859675030278m_6853013589290146639pfptBanner8j38xab" style="display:block;background-color:rgb(208,216,220);font-family:Arial,sans-serif;font-weight:bold;font-size:14px;line-height:18px">
This Message Is From an External Sender<span> </span></div>
<div id="m_6734545526208535891m_-1382752044991804335m_6339866382597766808m_5114540859675030278m_6853013589290146639pfptBanner8j38xab" style="font-weight:normal;display:block;background-color:rgb(208,216,220);font-family:Arial,sans-serif;font-size:12px;line-height:18px;margin-top:2px">
This message came from outside your organization.<span> </span></div>
</div>
<div style="height:0px;clear:both;display:block;line-height:0;font-size:0.01px">
</div>
</div>
<u></u>
<div>
<pre style="font-family:sans-serif;font-size:12px;white-space:pre-wrap">Dear all,
I am implementing a Molecular Dynamics (MD) code using the DMSWARM interface. In the MD simulations we evaluate on each particle (atoms) some kind of scalar functional using data from the neighbouring atoms. My problem lies in the parallel implementation of the model, because sometimes, some of these neighbours lie on a different processor.
This is usually solved by using ghost particles. A similar approach (with nodes instead) is already implemented for other PETSc mesh structures like DMPlexConstructGhostCells. Unfortunately, I don't see this kind of constructs for DMSWARM. Am I missing something?
I this could be done by applying a buffer region by exploiting the background DMDA mesh that I already use to do domain decomposition. Then using the buffer region of each cell to locate the ghost particles and finally using VecCreateGhost. Is this feasible? Or is there an easier approach using other PETSc functions.</pre>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>This is feasible, but it would be good to develop a set of best practices, since we have been mainly focused on the case of non-redundant particles. Here is how I think I would do what you want.</div>
<div><br>
</div>
<div>1) Add a particle field 'ghost' that identifies ghost vs owned particles. I think it needs options OWNED, OVERLAP, and GHOST</div>
<div><br>
</div>
<div>2) At some interval identify particles that should be sent to other processes as ghosts. I would call these "overlap particles". The determination</div>
<div> seems application specific, so I would leave this determination to the user right now. We do two things to these particles</div>
<div><br>
</div>
<div> a) Mark chosen particles as OVERLAP</div>
<div><br>
</div>
<div> b) Change rank to process we are sending to</div>
<div><br>
</div>
<div>3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag</div>
<div><br>
</div>
<div>4) Mark OVERLAP particles as GHOST when they arrive</div>
<div><br>
</div>
<div>There is one problem in the above algorithm. It does not allow sending particles to multiple ranks. We would have to do this</div>
<div>in phases right now, or make a small adjustment to the interface allowing replication of particles when a set of ranks is specified.</div>
<div><br>
</div>
<div> THanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>
<pre style="font-family:sans-serif;font-size:12px;white-space:pre-wrap">Thank you,
Miguel
</pre>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
<span class="gmail_signature_prefix">--<span> </span></span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgjAy34J0$" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
<span class="gmail_signature_prefix">--<span> </span></span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgjAy34J0$" target="_blank">https://www.cse.buffalo.edu/~knepley/</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
<span class="gmail_signature_prefix">--<span> </span></span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgjAy34J0$" target="_blank">https://www.cse.buffalo.edu/~knepley/</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
<span class="gmail_signature_prefix">--<span> </span></span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgjAy34J0$" target="_blank">https://www.cse.buffalo.edu/~knepley/</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aikPEw2N9TJ1IX4nFiTcOEsUOqcmEW2AwEUjS3Pw-WhS71aoiH_YonU7jU5aHVRNC-CqA7OIsJSNgjAy34J0$" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>