<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="text-align: left; text-indent: 0px; margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hello Mark and Mathew, </div>
<div class="elementToProof" style="text-align: left; text-indent: 0px; margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="text-align: left; text-indent: 0px; margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Apologies for the delay in reply (I was gone for a vacation). Really appreciate the prompt response. </div>
<div class="elementToProof" style="text-align: left; text-indent: 0px; margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="text-align: left; text-indent: 0px; margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I am now planning to redo these tests with the load balancing suggestions you have provided.
<b>Would you suggest any load balancing options to use as default when dealing with unstructured meshes in general</b>? I use PETSc as an external linear solver for my software, where I supply a Poisson system discretised using 3D simplical elements and FEM
- which are solved using AMG. I observed bad weak scaling behaviour for my application for 20k DOF/rank, which prompted me to test something similar only in PETSc. </div>
<div class="elementToProof" style="text-align: left; text-indent: 0px; margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="text-align: left; text-indent: 0px; margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I choose ex12 instead of ex56 because it uses 3D FEM. I am not sure if I can make ex56 work for tetrahedrons out of the box. Maybe ex13 is more suited as Mark mentioned.</div>
<div class="elementToProof" style="text-align: left; text-indent: 0px; margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="text-align: left; text-indent: 0px; margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
On point 3,4 from Mathew:</div>
<div class="elementToProof" style="text-align: left; text-indent: 0px; margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The plot below is from the numbers extracted from the -log_view option for all the runs. I have attached a sample log file from my runs, and pasted a sample output in the email.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
------------------------------------------------------------------ PETSc Performance Summary: ------------------------------------------------------------------</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
------------------------------------------------------------------------------------------------------------------------</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Event Count Time (sec) Flop --- Global --- --- Stage ---- Total</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
------------------------------------------------------------------------------------------------------------------------</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
--- Event Stage 0: Main Stage</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
KSPSolve 2 1.0 1.4079e-01 1.0 2.14e+07 2.0 1.2e+03 1.1e+04 4.4e+01 2 4 26 16 17 2 4 26 16 18 875</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
SNESSolve 1 1.0 2.9310e+00 1.0 1.69e+08 1.1 1.7e+03 2.0e+04 6.1e+01 46 46 37 38 23 46 46 37 38 25 445</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
PCApply 23 1.0 1.2774e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
------------------------------------------------------------------------------------------------------------------------</div>
<div id="appendonsend"></div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks and Best,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Parv</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr style="display: inline-block; width: 98%;">
<div id="divRplyFwdMsg" dir="ltr"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> Mark Adams <mfadams@lbl.gov><br>
<b>Sent:</b> 31 October 2024 11:30<br>
<b>To:</b> Matthew Knepley <knepley@gmail.com><br>
<b>Cc:</b> Khurana, Parv <p.khurana22@imperial.ac.uk>; petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-users] Expected weak scaling behaviour for AMG libraries?</span>
<div> </div>
</div>
<table cellspacing="0" cellpadding="0" style="box-sizing: border-box; border-collapse: collapse; border-spacing: 0px;">
<tbody>
<tr>
<td style="border-width: initial; border-style: initial; border-color: rgb(240, 240, 240); background-color: rgb(255, 255, 204); padding: 0cm;">
<p style="margin: 0cm 0cm 0pt;"><span style="font-family: Arial, sans-serif; font-size: 13px;">This email from mfadams@lbl.gov originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender,
add them to your <a href="https://urldefense.us/v3/__https://spam.ic.ac.uk/SpamConsole/Senders.aspx__;!!G_uCfscf7eWS!f47A4_a_tRPU1XvaYrgWMAp2uGFajfIIWf6QG4FERzGhIyI7U-eiYao8U73sCFqUwb_u9HrBY8TMcMT4qnKeIzOyDrGDgvqU$" id="OWA372749d1-f8bb-6f3c-8a8f-b956db9b214b" class="OWAAutoLink" data-auth="NotApplicable" style="margin-top: 0px; margin-bottom: 0px;">
safe senders list</a> to disable email stamping for this address.</span></p>
</td>
</tr>
</tbody>
</table>
<p> </p>
<div style="direction: ltr;">As Matt said snes ex56 is better because it does a convergence test that refines the grid. You need/want these two parameters to have the same arg (eg, 2,2,1): -dm_plex_box_faces 2,2,1 -petscpartitioner_simple_process_grid 2,2,1.</div>
<div style="direction: ltr;">This will put one cell per process.</div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">Then you use: -max_conv_its N, to specify the N levels of refinement to do. It will run the 2,2,1 first then a 4,4,2, etc., N times.</div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">/src/snes/tests/ex13.c is designed for benchmarking and it uses '-petscpartitioner_simple_node_grid 1,1,1 [default]' to give you a two level partitioner. </div>
<div style="direction: ltr;">You need to have dm_plex_box_faces_i = petscpartitioner_simple_process_grid_i * petscpartitioner_simple_node_grid_i</div>
<div style="direction: ltr;">Again, you should put one cell per process (NP = product of dm_plex_box_faces args) and use -dm_refine N to get a single solve.</div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">Mark</div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;"><br>
</div>
<br>
<div style="direction: ltr;">On Wed, Oct 30, 2024 at 11:02 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" id="OWA26fe3e68-df41-5fb7-c2ca-970936ea1fae" class="OWAAutoLink">knepley@gmail.com</a>> wrote:</div>
<blockquote style="margin: 0px 0px 0px 0.8ex; padding-left: 1ex; border-left: 1px solid rgb(204, 204, 204);">
<div style="direction: ltr;">On Wed, Oct 30, 2024 at 4:13 PM Khurana, Parv <<a href="mailto:p.khurana22@imperial.ac.uk" id="OWA9fd17dbb-3609-f007-a0e2-81b5e5aa5405" class="OWAAutoLink">p.khurana22@imperial.ac.uk</a>> wrote:</div>
<blockquote style="margin: 0px 0px 0px 0.8ex; padding-left: 1ex; border-left: 1px solid rgb(204, 204, 204);">
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hello PETSc Community,</div>
<div style="direction: ltr; margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I am trying to understand the scaling behaviour of AMG methods in PETSc (Hypre for now) and how many DOFs/Rank are needed for a performant AMG solve.</div>
<div style="direction: ltr; margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I’m currently conducting weak scaling tests using <code>src/snes/tutorials/ex12.c</code> in 3D, applying Dirichlet BCs with FEM at P=1. The tests keep DOFs per processor constant while increasing the mesh size and processor count, specifically:</div>
<ul style="direction: ltr;">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); direction: ltr;">
<b>20000 and 80000 DOF/RANK</b> configurations.</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); direction: ltr;">
Running SNES twice, using GMRES with a tolerance of 1e-5 and preconditioning with Hypre-BoomerAMG.</li></ul>
<div style="margin-top: 1em; margin-bottom: 1em;"></div>
</blockquote>
<div style="direction: ltr;">A couple of quick points in order to make sure that there is no confusion:</div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">1) Partitioner type "simple" is for the CI. It is a very bad partition, and should not be used for timing. The default is ParMetis which should be good enough.</div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">2) You start out with 6^3 = 216 elements, distribute that, and then refine it. This will be _really_ bad load balance on all arrangement except the divisors of 216. You usually want to start out with something bigger at the later
stages. You can use -dm_refine_pre to refine before distribution.</div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">3) It is not clear you are using the timing for just the solver (SNESSolve). It could be that extraneous things are taking time. When asking questions like this, please always send the output of -log_view for timing, and at least
-ksp_monitor_true_residial for convergence.</div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">4) SNES ex56 is the example we use for GAMG scalability testing</div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;"> Thanks,</div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;"> Matt</div>
<blockquote style="margin: 0px 0px 0px 0.8ex; padding-left: 1ex; border-left: 1px solid rgb(204, 204, 204);">
<div style="direction: ltr; margin-top: 1em; margin-bottom: 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Unfortunately, parallel efficiency degrades noticeably with increased processor counts. Are there any insights or rules of thumb for using AMG more effectively? I have been looking at this issue for a while now and would love to engage in a further discussion.
Please find below the weak scaling results and the options I use to run the tests.</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<img id="x_m_-2677784089042558004m_1953988574953062207image_0" width="651" height="334" size="270405" contenttype="image/png" style="width: 651px; height: 334px;" data-outlook-trace="F:1|T:1" src="cid:ii_192e0800b4dcb971f161"></div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<b>#Run type</b></div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-run_type full</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-petscpartitioner_type simple</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<b>#Mesh settings</b></div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-dm_plex_dim 3</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-dm_plex_simplex 1</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-dm_refine 5 #Varied this</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-dm_plex_box_faces 6,6,6</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<b>#BCs and FEM space</b></div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-bc_type dirichlet</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-petscspace_degree 1</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<b>#Solver settings</b></div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-snes_max_it 2</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-ksp_type gmres</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-ksp_rtol 1.0e-5</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
#Same settings as what we use for LOR</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-pc_type hypre</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-pc_hypre_type boomeramg</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-pc_hypre_boomeramg_coarsen_type hmis</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-pc_hypre_boomeramg_relax_type_all symmetric-sor/jacobi</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-pc_hypre_boomeramg_strong_threshold 0.7</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-pc_hypre_boomeramg_interp_type ext+i</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-pc_hypre_boomeramg_P_max 2</div>
<div style="direction: ltr; margin: 0in; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-pc_hypre_boomeramg_truncfactor 0.3</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Best,</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Parv</div>
</blockquote>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">--</div>
<div style="direction: ltr;">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;"><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YXR-8qKioRYS0fNOHacYGkm6WaIuKge2zoTiW1n0vLsWQUBiyLM48cg58pRLtNm0QjVigIZYftn2x09fmjiN$" id="OWA66b07c16-4a36-ffb9-bc04-974e6e02bc12" class="OWAAutoLink" data-auth="NotApplicable">https://www.cse.buffalo.edu/~knepley/</a></div>
</blockquote>
</body>
</html>