<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, Apr 20, 2018 at 4:28 PM, Zou, Ling <span dir="ltr"><<a href="mailto:ling.zou@inl.gov" target="_blank">ling.zou@inl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Mat, is this book you recommended?<div><br></div><div><a href="https://www.amazon.com/Using-MPI-Programming-Message-Passing-Engineering/dp/0262527391/ref=pd_lpo_sbs_14_img_0?_encoding=UTF8&psc=1&refRID=EYCV0H0J5EQ9M0GDKWFT" target="_blank">https://www.amazon.com/Using-<wbr>MPI-Programming-Message-<wbr>Passing-Engineering/dp/<wbr>0262527391/ref=pd_lpo_sbs_14_<wbr>img_0?_encoding=UTF8&psc=1&<wbr>refRID=EYCV0H0J5EQ9M0GDKWFT</a></div></div></blockquote><div><br></div><div>Yep. I think its the best one, although Victor's book is also nice</div><div><br></div><div> <a href="http://pages.tacc.utexas.edu/~eijkhout/istc/istc.html">http://pages.tacc.utexas.edu/~eijkhout/istc/istc.html</a></div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thanks,</div><div><br></div><div>Ling</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 20, 2018 at 2:17 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Fri, Apr 20, 2018 at 4:07 PM, Klaus Burkart <span dir="ltr"><<a href="mailto:k_burkart@yahoo.com" target="_blank">k_burkart@yahoo.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:10px"><div><font size="3">Sorry, I don't understand you:<br></font><div><font size="3"><br></font><div><font size="3">The example says:<br></font><div><font size="3"><br><span>Step 1: Create the gobal matrix<br><br> MatCreate(...,&A); // no problem<br><br>Step 2: Make it a parallel matrix<br><br> MatSetType(A,MATMPIAIJ); // no problem<br><br>Step 3: Define the size of the global matrix and the number of rows per process IF this number is the same for all processes<br><br> MatSetSizes(A, N,n,N,N); In the example, I have the problem with n which is 3 or 2 depending on the process but I can only set n to 3 or 2 so it will be the wrong for at least one process<br></span></font></div></div></div></div></div></div></blockquote><div><br></div></span><div>1) Get the book "Using MPI" just like I suggested. It explains this part of parallel programming that you do not understand.</div><div><br></div><div>2) Suppose we have two processes P0 and P1. Here are the call made on both processes for a matrix with 5 rows split 3, 2:</div><div><br></div><div> P0: MatSetSizes(A, 3, 3, 5, 5);</div><div> P1: MatSetSizes(A, 2, 2, 5, 5);</div><div><br></div><div>See how different processes give different numbers to the routine? This is what SPMD programming is all about.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div><div class="gmail-m_-1399776624805439203h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:10px"><div><div><div><div><font size="3"><span>Step 4: Preallocate memory for the d_nnz and o_nnz values which are different for each row (for each n)<br><br> MatMPIAIJSetPreallocation(A,0,<wbr>d_nnz[n],0,o_nnz[n]); // How to do this repeatedly for all processes especially when using PETEC_DECIDE for n and m as shown in many examples in which case I don't even know the n per process<br><br>I have to retrieve the relevant values (content and structure) from an application and I absolutely don't understand how to enter the exact same matrix structure as shown in the example (only much larger) into PETSc using the retrieved data?<br><br>How would assign the d_nnz and o_nnz values of a global matrix on a 32 core workstation to the 32 processes using MatMPIAIJSetPreallocation(A,0,<wbr>d_nnz[n],0,o_nnz[n]);? (Which means in my case assigning 64 arrays with different content containing the d_nnz and o_nnz values for the 32 processes)<br><br>Would I use a for loop - but then how to get hold of the individual processes? Same for Step 3?<br><br>I don't get the underlying idea about how to create a parallel matrix structure in PETSc - don't get me wrong I understand the layout "simply" not how to enter more than one local matrix it into PETSc.<br><br>Klaus</span></font><br></div></div></div></div><div><br></div>
<div id="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yahoo_quoted_5033386504" class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yahoo_quoted">
<div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:13px;color:rgb(38,40,42)">
<div>
Am Freitag, 20. April 2018, 19:52:11 MESZ hat Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> Folgendes geschrieben:
</div>
<div><br></div>
<div><br></div>
<div><div id="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761"><div><div dir="ltr"><div class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761gmail_extra"><div class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761gmail_quote">On Fri, Apr 20, 2018 at 1:30 PM, Klaus Burkart <span dir="ltr"><<a rel="nofollow" shape="rect" href="mailto:k_burkart@yahoo.com" target="_blank">k_burkart@yahoo.com</a>></span> wrote:<br clear="none"><blockquote class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:10px"><div><div><font size="3">In my case N=M but n for process 0, 1, 2, 3,... no_processes-1 can be different from the nth process like in the example where the nth process=Proc2 and has only two rows while all other processes have three rows:</font></div></div></div></div></blockquote><div><br clear="none"></div><div>Yes.</div><div> </div><blockquote class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:10px"><div><div><div><div><font size="3">Example from the PETSc webpage mentioned before:<br clear="none"></font></div><font size="3"><span></span></font><pre> 1 2 0 | 0 3 0 | 0 4
Proc0 0 5 6 | 7 0 0 | 8 0
9 0 10 | 11 0 0 | 12 0
------------------------------ -------
13 0 14 | 15 16 17 | 0 0
Proc1 0 18 0 | 19 20 21 | 0 0
0 0 0 | 22 23 0 | 24 0
------------------------------ -------
Proc2 25 26 27 | 0 0 28 | 29 0
30 0 0 | 31 32 33 | 0 34</pre>and<span><font size="3"> I need to enter different values for d_nnz and o_nnz for each row somewhere too<br clear="none"></font><font size="3"><span></span></font></span><pre> proc0: d_nnz = [2,2,2] and o_nnz = [2,2,2]
proc1: d_nnz = [3,3,2] and o_nnz = [2,1,1]
proc2: d_nnz = [1,1] and o_nnz = [4,4]</pre></div></div></div></div></div></blockquote><div>Each process only sets d_nnz and o_nnz for its LOCAL rows. Thus, it is exactly as shown above. Proc1 sets values</div><div>only for rows 3-5.</div><div><br clear="none"></div><div> Thanks,</div><div><br clear="none"></div><div> Matt</div><div class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761yqt7169460321" id="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761yqtfd83747"><div> </div></div><blockquote class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:10px"><div class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761yqt7169460321" id="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761yqtfd03244"><div><div><div><div><div><font size="3">I simply can't identify the function(s) used to set the values for n, d_nnz and o_nnz for the individual local matrices allocated to all the processes if n isn't the same for all processes and d_nnz and o_nnz are different for each local matrix?<br clear="none"></font><div><div><br clear="none"><font size="3">Approach described on the PETSc webpage:</font><br clear="none"></div><span></span><pre><font size="3"> <a rel="nofollow" shape="rect" href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs.anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Mat_MatCreate.html-23MatCreate&d=DwMFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=kuHHom1yjd94zUrBWecnYg&m=N7Qj9HPi7_sDdfpeUPyFGoLWR95huadKQYfwt6HFZTg&s=fuaWHR03gK4mnNvFfFyicLPfKIaRobbPAKA7KvdbbQM&e=" target="_blank">MatCreate</a>(...,&A);
</font></pre><div><br clear="none"></div><div><br clear="none"></div><font size="3">
</font><pre><font size="3"> <a rel="nofollow" shape="rect" href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs.anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Mat_MatSetType.html-23MatSetType&d=DwMFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=kuHHom1yjd94zUrBWecnYg&m=N7Qj9HPi7_sDdfpeUPyFGoLWR95huadKQYfwt6HFZTg&s=Kh5W-9DyiSe82fN1OgujmDWs-IijI441g7ujXZGreiY&e=" target="_blank">MatSetType</a>(A,<a rel="nofollow" shape="rect" href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs.anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Mat_MATMPIAIJ.html-23MATMPIAIJ&d=DwMFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=kuHHom1yjd94zUrBWecnYg&m=N7Qj9HPi7_sDdfpeUPyFGoLWR95huadKQYfwt6HFZTg&s=CpJmovpIlWJ4se4dYFhIrakh2Hve3t62RiWMcpRtL54&e=" target="_blank">MATMPIAIJ</a>);
</font></pre><div><br clear="none"></div><div><br clear="none"></div><font size="3">
</font><pre><font size="3"> <a rel="nofollow" shape="rect" href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs.anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Mat_MatSetSizes.html-23MatSetSizes&d=DwMFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=kuHHom1yjd94zUrBWecnYg&m=N7Qj9HPi7_sDdfpeUPyFGoLWR95huadKQYfwt6HFZTg&s=rdrEy7Pg8fLeMoQveCRqUHzi3IpOfCjvBGB-gFL73Ws&e=" target="_blank">MatSetSizes</a>(A, m,n,M,N); // for the example above using this function would set the no. of rows for Proc2 to 3 but it's 2
</font></pre><div><br clear="none"></div><div><br clear="none"></div><div><br clear="none"></div><font size="3">
</font><pre><font size="3"> <a rel="nofollow" shape="rect" href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs.anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Mat_MatMPIAIJSetPreallocation.html-23MatMPIAIJSetPreallocation&d=DwMFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=kuHHom1yjd94zUrBWecnYg&m=N7Qj9HPi7_sDdfpeUPyFGoLWR95huadKQYfwt6HFZTg&s=RsveZcVp1senr6bpfVWoVoin2d28-0D1000viYN2aqA&e=" target="_blank">MatMPIAIJSetPreallocation</a>(A,.. .); // this function can be used to set values for ONE local matrix only</font></pre><div><br clear="none"></div><br clear="none"><font size="3">In addition to that I don't know which functions to use to preallocate memory for ALL local matrices when each of them has different values for d_nnz and o_nnz.<br clear="none"><br clear="none">I other words, what's the code for the 3 process example above? (entering the matrix structure and allocating memory)<br clear="none"></font><br clear="none"></div></div></div></div></div></div><font size="3">
</font><div><font size="3">Klaus<br clear="none"></font></div><div><br clear="none"></div>
</div><div class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761m_800689018898067999yahoo_quoted" id="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761m_800689018898067999yahoo_quoted_4478250076"><div class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761yqt7169460321" id="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761yqtfd65069">
</div><div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:13px;color:rgb(38,40,42)"><div class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761yqt7169460321" id="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761yqtfd54786">
<div>
Am Freitag, 20. April 2018, 17:13:26 MESZ hat Smith, Barry F. <<a rel="nofollow" shape="rect" href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> Folgendes geschrieben:
</div>
<div><br clear="none"></div>
<div><br clear="none"></div>
</div><div><div dir="ltr"><div class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761yqt7169460321" id="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761yqtfd60505"><br clear="none"> For square matrices almost always n is the same as m. On different processes m can be different. You get to decide what makes sense for each processes what its m should be.<br clear="none"><br clear="none"> Barry<br clear="none"><br clear="none"></div><div class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761m_800689018898067999yqt5043941089" id="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761m_800689018898067999yqtfd85913"><div class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761yqt7169460321" id="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761yqtfd56772"><br clear="none">> On Apr 20, 2018, at 10:05 AM, Klaus Burkart <<a rel="nofollow" shape="rect" href="mailto:k_burkart@yahoo.com" target="_blank">k_burkart@yahoo.com</a>> wrote:<br clear="none">> <br clear="none">> I think I understood the matrix structure for parallel computation with the rows, diagonal (d) and off-diagonal (o) structure, where I have problems is how to do the setup including memory allocation in PETSc:<br clear="none">> <br clear="none">> Lets assume, I use a 16 core workstation (=16 processes) and the number of nonzeros varies in each row for both d and o and the number of rows assigned to each process differs too - at least for the nth process.<br clear="none">> <br clear="none">> Looking at the manual and <a rel="nofollow" shape="rect" href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs.anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Mat_MatCreateAIJ.html-23MatCreateAIJ-2C&d=DwMFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=kuHHom1yjd94zUrBWecnYg&m=N7Qj9HPi7_sDdfpeUPyFGoLWR95huadKQYfwt6HFZTg&s=-_yf6zmL3d2iNa4ipS2vN3X9L0S7LxolfdXQI12_u7c&e=" target="_blank">http://www.mcs.anl.gov/petsc/ petsc-current/docs/ manualpages/Mat/MatCreateAIJ. html#MatCreateAIJ, </a>I don't understand how to enter a global matrix when n is NOT the same for each process as e.g. in MatSetSizes(A, m,n,M,N); n and m are integers, not arrays?<br clear="none">> <br clear="none">> MatCreate(...,&A);<br clear="none">> <br clear="none">> MatSetType(A,MATMPIAIJ);<br clear="none">> <br clear="none">> MatSetSizes(A, m,n,M,N); // seems to assume n and m are the same for each process which isn't even the case in the example on the page mentioned above?!<br clear="none">> <br clear="none">> MatMPIAIJSetPreallocation(A,.. .);<br clear="none">> <br clear="none">> <br clear="none">> How can I enter the parallel global-local matrix structure?<br clear="none">> <br clear="none">> How can the memory preallocation be done?<br clear="none">> <br clear="none">> Klaus<br clear="none">> <br clear="none">> Am Donnerstag, 19. April 2018, 01:47:59 MESZ hat Smith, Barry F. <<a rel="nofollow" shape="rect" href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> Folgendes geschrieben:<br clear="none">> <br clear="none">> <br clear="none">> <br clear="none">> <br clear="none">> > On Apr 18, 2018, at 4:42 PM, <a rel="nofollow" shape="rect" href="mailto:k_burkart@yahoo.com" target="_blank">k_burkart@yahoo.com</a> wrote:<br clear="none">> > <br clear="none">> > So, practically speaking, l should invent routines to decompose the matrix e.g. into a block matrix structure to be able to make real use of PETSc ie. be able to solve a linear system using more than one process/core?<br clear="none">> <br clear="none">> To really use PETSc efficiently/effectively you need to generate your matrix in parallel.<br clear="none">> <br clear="none">> Barry<br clear="none">> <br clear="none">> > <br clear="none">> > Klaus<br clear="none">> > <br clear="none">> > Von meinem Huawei-Mobiltelefon gesendet<br clear="none">> > <br clear="none">> > <br clear="none">> > -------- Originalnachricht --------<br clear="none">> > Betreff: Re: [petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import?<br clear="none">> > Von: "Smith, Barry F." <br clear="none">> > An: Klaus Burkart <br clear="none">> > Cc: PETSc Users List <br clear="none">> > <br clear="none">> > <br clear="none">> > <br clear="none">> > If you can only generate the nonzero allocation sequentially you can only solve sequentially which means your matrix is MATSEQAIJ and your vector is VECSEQ and your communicator is PETSC_COMM_SELF.<br clear="none">> > <br clear="none">> > If you pass and array for nnz, what you pass for nz is irrelevant, you might as well pass 0.<br clear="none">> > <br clear="none">> > Barry<br clear="none">> > <br clear="none">> > <br clear="none">> > > On Apr 18, 2018, at 10:48 AM, Klaus Burkart wrote:<br clear="none">> > > <br clear="none">> > > More questions about matrix and vector type selection for my application:<br clear="none">> > > <br clear="none">> > > My starting point is a huge sparse matrix which can be symmetric or asymmetric and a rhs vector. There's no defined local or block structure at all, just row and column indices and the values and an array style rhs vector together describing the entire linear system to be solved. With quite some effort, I should be able to create an array nnz[N] containing the number of nonzeros per row in the global matrix for memory allocation which would leave me with MatSeqAIJSetPreallocation(M, 0, nnz); as the only option for efficient memory allocation ie. a MATSEQAIJ matrix and VECSEQ. I assume here, that 0 indicates different numbers of nonzero values in each row, the exact number being stored in the nnz array. Regarding this detail but one example assume a constant number of nz per row so I am not sure whether I should write 0 or NULL for nz?<br clear="none">> > > <br clear="none">> > > I started with:<br clear="none">> > > <br clear="none">> > > MatCreate(PETSC_COMM_WORLD, &M);<br clear="none">> > > MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N);<br clear="none">> > > MatSetFromOptions(M);<br clear="none">> > > <br clear="none">> > > taken from a paper and assume, the latter would set the matrix type to MATSEQAIJ which might conflict with PETSC_COMM_WORLD. Maybe decompositioning took place at an earlier stage and the authors of the paper were able to retrieve the local data and structure. <br clear="none">> > > <br clear="none">> > > What type of matrix and vector should I use for my application e.g. MATSEQAIJ and VECSEQ to be able to use MatSeqAIJSetPreallocation(M, 0, nnz); for efficient memory allocation?<br clear="none">> > > </div><br clear="none">> > > In this case, where would the decompositioning / MPI process allocation take place?<br clear="none">> > <br clear="none"></div></div></div><span class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851HOEnZb"><font color="#888888">
</font></span></div><span class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851HOEnZb"><font color="#888888">
</font></span></div></div></div></blockquote></div><span class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851HOEnZb"><font color="#888888"><br clear="none"><br clear="all"><span class="gmail-HOEnZb"><font color="#888888"><div><br clear="none"></div>-- <br clear="none"><div class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br clear="none">-- Norbert Wiener</div><div><br clear="none"></div><div><a rel="nofollow" shape="rect" href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.caam.rice.edu_-7Emk51_&d=DwMFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=kuHHom1yjd94zUrBWecnYg&m=N7Qj9HPi7_sDdfpeUPyFGoLWR95huadKQYfwt6HFZTg&s=X1XTDkxAzSG3ajhCqaTWt8j4MtmpP7m1h8PAUX0xslA&e=" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><div class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761yqt7169460321" id="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761yqtfd70644"><br clear="none"></div></div></div></div></div></div><div class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761yqt7169460321" id="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851m_-9050146912808005398yiv4365262761yqtfd47085">
</div></font></span></font></span></div></div></div></div></div><span class="gmail-HOEnZb"><font color="#888888">
</font></span></div><span class="gmail-HOEnZb"><font color="#888888">
</font></span></div></div></div></blockquote></div></div></div><span class="gmail-HOEnZb"><font color="#888888"><div><div class="gmail-m_-1399776624805439203h5"><br><br clear="all"><div><br></div>-- <br><div class="gmail-m_-1399776624805439203m_-1359940501892032001m_751625284409407851gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.caam.rice.edu_-7Emk51_&d=DwMFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=kuHHom1yjd94zUrBWecnYg&m=N7Qj9HPi7_sDdfpeUPyFGoLWR95huadKQYfwt6HFZTg&s=X1XTDkxAzSG3ajhCqaTWt8j4MtmpP7m1h8PAUX0xslA&e=" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br></div></div></div></div></div>
</div></div></font></span></div></div>
</blockquote></div><br></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div>
</div></div>