<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style>body{font-family:Helvetica,Arial;font-size:13px}</style>
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">
<br>
</div>
<p class="airmail_on">On May 11, 2018 at 6:03:09 PM, Smith, Barry F. (<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>) wrote:</p>
<div>
<div>
<blockquote type="cite" class="clean_bq" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;">
<span>
<div>
<div></div>
<div><br>
Shidi,<span class="Apple-converted-space"> </span><br>
<br>
You stated:<span class="Apple-converted-space"> </span><br>
<br>
>> and create the matrix at the beginning of each iteration.<span class="Apple-converted-space"> </span><br>
<br>
We never anticipated your use case where you keep the same KSP and make a new matrix for each KSPSolve so we have bugs in PETSc that do not handle that case.<span class="Apple-converted-space"> </span></div>
</div>
</span></blockquote>
</div>
<p>We do this in TAO for bounded Newton algorithms where the constraints are handled with an active-set method. We have to construct a new reduced Hessian with a different size every time the active variable indexes change. </p>
<p>We don’t completely destroy the KSP solver and create a brand new one whenever this happens. We simply call KSPReset() followed by KSPSetOperators(). This preserves the solver type and other KSP options set in the beginning (e.g.: maximum iterations or various
tolerances), but permits the matrix (and the preconditioner) to be completely different objects from one KSPSolve() to another.</p>
<p>I don’t know if this behavior is universally supported for all KSP solvers, but it’s been working bug-free for us for STCG, NASH and GLTR solvers. I’ve also made it work with GMRES in a separate test. It may be worth it for Shidi to give it a try and see
if it works.</p>
<div>
<blockquote type="cite" class="clean_bq" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;">
<span>
<div>
<div><br>
<br>
Barry<span class="Apple-converted-space"> </span><br>
<br>
<br>
> On May 11, 2018, at 5:58 PM, Y. Shidi <ys453@cam.ac.uk> wrote:<span class="Apple-converted-space"> </span><br>
><span class="Apple-converted-space"> </span><br>
> Thank you for your reply.<span class="Apple-converted-space"> </span><br>
>> For now, if you give the same Mat object back, it will do what you<span class="Apple-converted-space"> </span><br>
>> expect.<span class="Apple-converted-space"> </span><br>
> Sorry, I am confused here.<span class="Apple-converted-space"> </span><br>
> The same Mat object, is it that I do not destroy the Mat at the<span class="Apple-converted-space"> </span><br>
> end of iteration?<span class="Apple-converted-space"> </span><br>
> Moreover, which function should I actually call to put the Mat<span class="Apple-converted-space"> </span><br>
> object back?<span class="Apple-converted-space"> </span><br>
> Sorry for being stupid on this.<span class="Apple-converted-space"> </span><br>
><span class="Apple-converted-space"> </span><br>
> Kind Regards,<span class="Apple-converted-space"> </span><br>
> Shidi<span class="Apple-converted-space"> </span><br>
><span class="Apple-converted-space"> </span><br>
><span class="Apple-converted-space"> </span><br>
><span class="Apple-converted-space"> </span><br>
> On 2018-05-11 18:10, Matthew Knepley wrote:<span class="Apple-converted-space"> </span><br>
>> On Fri, May 11, 2018 at 1:02 PM, Y. Shidi <ys453@cam.ac.uk> wrote:<span class="Apple-converted-space"> </span><br>
>>> Thank you very much for your reply, Barry.<span class="Apple-converted-space"> </span><br>
>>>> This is a bug in PETSc. Since you are providing a new matrix<span class="Apple-converted-space"> </span><br>
>>>> with<span class="Apple-converted-space"> </span><br>
>>>> the same "state" value as the previous matrix the PC code the<span class="Apple-converted-space"> </span><br>
>>>> following code<span class="Apple-converted-space"> </span><br>
>>> So what you mean is that every time I change the value in the<span class="Apple-converted-space"> </span><br>
>>> matrix,<span class="Apple-converted-space"> </span><br>
>>> the PETSc only determines if the nonzero pattern change but not the<span class="Apple-converted-space"> </span><br>
>>> values, and if it is unchanged neither of symbolic and numeric<span class="Apple-converted-space"> </span><br>
>>> happens.<span class="Apple-converted-space"> </span><br>
>> No, that is not what Barry is saying.<span class="Apple-converted-space"> </span><br>
>> PETSc looks at the matrix.<span class="Apple-converted-space"> </span><br>
>> If the structure has changed, it does symbolic and numeric<span class="Apple-converted-space"> </span><br>
>> factorization.<span class="Apple-converted-space"> </span><br>
>> If only values have changes, it does numeric factorization.<span class="Apple-converted-space"> </span><br>
>> HOWEVER, you gave it a new matrix with accidentally the same state<span class="Apple-converted-space"> </span><br>
>> marker,<span class="Apple-converted-space"> </span><br>
>> so it thought nothing had changed. We will fix this by also checking<span class="Apple-converted-space"> </span><br>
>> the pointer.<span class="Apple-converted-space"> </span><br>
>> For now, if you give the same Mat object back, it will do what you<span class="Apple-converted-space"> </span><br>
>> expect.<span class="Apple-converted-space"> </span><br>
>> Matt<span class="Apple-converted-space"> </span><br>
>>> I found the following code:<span class="Apple-converted-space"> </span><br>
>>> if (!pc->setupcalled) {<span class="Apple-converted-space"> </span><br>
>>> ierr = PetscInfo(pc,"Setting up PC for first<span class="Apple-converted-space"> </span><br>
>>> timen");CHKERRQ(ierr);<span class="Apple-converted-space"> </span><br>
>>> pc->flag = DIFFERENT_NONZERO_PATTERN;<span class="Apple-converted-space"> </span><br>
>>> } else if (matstate == pc->matstate) {<span class="Apple-converted-space"> </span><br>
>>> ierr = PetscInfo(pc,"Leaving PC with identical preconditioner<span class="Apple-converted-space"> </span><br>
>>> since operator is unchangedn");CHKERRQ(ierr);<span class="Apple-converted-space"> </span><br>
>>> PetscFunctionReturn(0);<span class="Apple-converted-space"> </span><br>
>>> } else {<span class="Apple-converted-space"> </span><br>
>>> if (matnonzerostate > pc->matnonzerostate) {<span class="Apple-converted-space"> </span><br>
>>> ierr = PetscInfo(pc,"Setting up PC with different nonzero<span class="Apple-converted-space"> </span><br>
>>> patternn");CHKERRQ(ierr);<span class="Apple-converted-space"> </span><br>
>>> pc->flag = DIFFERENT_NONZERO_PATTERN;<span class="Apple-converted-space"> </span><br>
>>> } else {<span class="Apple-converted-space"> </span><br>
>>> ierr = PetscInfo(pc,"Setting up PC with same nonzero<span class="Apple-converted-space"> </span><br>
>>> patternn");CHKERRQ(ierr);<span class="Apple-converted-space"> </span><br>
>>> pc->flag = SAME_NONZERO_PATTERN;<span class="Apple-converted-space"> </span><br>
>>> }<span class="Apple-converted-space"> </span><br>
>>> }<span class="Apple-converted-space"> </span><br>
>>> and I commend out "else if (matstate == pc->matstate){}", so it<span class="Apple-converted-space"> </span><br>
>>> will do "Setting up PC with same nonzero patternn"; and it seems<span class="Apple-converted-space"> </span><br>
>>> work in my case, only "MatFactorNumeric_MUMPS()" is calling in the<span class="Apple-converted-space"> </span><br>
>>> subsequent iterations. But I am not quite sure, need some more<span class="Apple-converted-space"> </span><br>
>>> tests.<span class="Apple-converted-space"> </span><br>
>>> Thank you very much for your help indeed.<span class="Apple-converted-space"> </span><br>
>>> Kind Regards,<span class="Apple-converted-space"> </span><br>
>>> Shidi<span class="Apple-converted-space"> </span><br>
>>> On 2018-05-11 16:13, Smith, Barry F. wrote:<span class="Apple-converted-space"> </span><br>
>>> On May 11, 2018, at 8:14 AM, Y. Shidi <ys453@cam.ac.uk> wrote:<span class="Apple-converted-space"> </span><br>
>>> Thank you for your reply.<span class="Apple-converted-space"> </span><br>
>>> How are you changing the matrix? Do you remember to assemble?<span class="Apple-converted-space"> </span><br>
>>> I use MatCreateMPIAIJWithArrays() to create the matrix,<span class="Apple-converted-space"> </span><br>
>>> and after that I call MatAssemblyBegin() and MatAssemblyEnd().<span class="Apple-converted-space"> </span><br>
>> If you use MatCreateMPIAIJWithArrays() you don't need to call<span class="Apple-converted-space"> </span><br>
>> MatAssemblyBegin() and MatAssemblyEnd().<span class="Apple-converted-space"> </span><br>
>>> But I actually destroy the matrix at the end of each iteration<span class="Apple-converted-space"> </span><br>
>>> and create the matrix at the beginning of each iteration.<span class="Apple-converted-space"> </span><br>
>> This is a bug in PETSc. Since you are providing a new matrix with<span class="Apple-converted-space"> </span><br>
>> the same "state" value as the previous matrix the PC code the<span class="Apple-converted-space"> </span><br>
>> following code<span class="Apple-converted-space"> </span><br>
>> kicks in:<span class="Apple-converted-space"> </span><br>
>> ierr =<span class="Apple-converted-space"> </span><br>
>> PetscObjectStateGet((PetscObject)pc->pmat,&matstate);CHKERRQ(ierr);<span class="Apple-converted-space"> </span><br>
>> ierr = MatGetNonzeroState(pc->pmat,&matnonzerostate);CHKERRQ(ierr);<span class="Apple-converted-space"> </span><br>
>> if (!pc->setupcalled) {<span class="Apple-converted-space"> </span><br>
>> ierr = PetscInfo(pc,"Setting up PC for first<span class="Apple-converted-space"> </span><br>
>> timen");CHKERRQ(ierr);<span class="Apple-converted-space"> </span><br>
>> pc->flag = DIFFERENT_NONZERO_PATTERN;<span class="Apple-converted-space"> </span><br>
>> } else if (matstate == pc->matstate) {<span class="Apple-converted-space"> </span><br>
>> ierr = PetscInfo(pc,"Leaving PC with identical preconditioner<span class="Apple-converted-space"> </span><br>
>> since operator is unchangedn");CHKERRQ(ierr);<span class="Apple-converted-space"> </span><br>
>> PetscFunctionReturn(0);<span class="Apple-converted-space"> </span><br>
>> and it returns without refactoring.<span class="Apple-converted-space"> </span><br>
>> We need an additional check that the matrix also remains the same.<span class="Apple-converted-space"> </span><br>
>> We will also need a test example that reproduces the problem to<span class="Apple-converted-space"> </span><br>
>> confirm that we have fixed it.<span class="Apple-converted-space"> </span><br>
>> Barry<span class="Apple-converted-space"> </span><br>
>>> Cheers,<span class="Apple-converted-space"> </span><br>
>>> Shidi<span class="Apple-converted-space"> </span><br>
>>> On 2018-05-11 12:59, Matthew Knepley wrote:<span class="Apple-converted-space"> </span><br>
>>> On Fri, May 11, 2018 at 7:14 AM, Y. Shidi <ys453@cam.ac.uk> wrote:<span class="Apple-converted-space"> </span><br>
>>> Dear Matt,<span class="Apple-converted-space"> </span><br>
>>> Thank you for your help last time.<span class="Apple-converted-space"> </span><br>
>>> I want to get more detail about the Petsc-MUMPS factorisation;<span class="Apple-converted-space"> </span><br>
>>> so I go to look the code "/src/mat/impls/aij/mpi/mumps/mumps.c".<span class="Apple-converted-space"> </span><br>
>>> And I found the following functions are quite important to<span class="Apple-converted-space"> </span><br>
>>> the question:<span class="Apple-converted-space"> </span><br>
>>> PetscErrorCode MatCholeskyFactorSymbolic_MUMPS(Mat F,Mat A,IS<span class="Apple-converted-space"> </span><br>
>>> r,const MatFactorInfo *info);<span class="Apple-converted-space"> </span><br>
>>> PetscErrorCode MatFactorNumeric_MUMPS(Mat F,Mat A,const<span class="Apple-converted-space"> </span><br>
>>> MatFactorInfo *info);<span class="Apple-converted-space"> </span><br>
>>> PetscErrorCode MatSolve_MUMPS(Mat A,Vec b,Vec x);<span class="Apple-converted-space"> </span><br>
>>> I print some sentence to trace when these functions are called.<span class="Apple-converted-space"> </span><br>
>>> Then I test my code; the values in the matrix is changing but the<span class="Apple-converted-space"> </span><br>
>>> structure stays the same. Below is the output.<span class="Apple-converted-space"> </span><br>
>>> We can see that at 0th step, all the symbolic, numeric and solve<span class="Apple-converted-space"> </span><br>
>>> are called; in the subsequent steps only the solve stage is called,<span class="Apple-converted-space"> </span><br>
>>> the numeric step is not called.<span class="Apple-converted-space"> </span><br>
>>> How are you changing the matrix? Do you remember to assemble?<span class="Apple-converted-space"> </span><br>
>>> Matt<span class="Apple-converted-space"> </span><br>
>>> Iteration 0 Step 0.0005 Time 0.0005<span class="Apple-converted-space"> </span><br>
>>> [INFO]: Direct Solver setup<span class="Apple-converted-space"> </span><br>
>>> MatCholeskyFactorSymbolic_MUMPS<span class="Apple-converted-space"> </span><br>
>>> finish MatCholeskyFactorSymbolic_MUMPS<span class="Apple-converted-space"> </span><br>
>>> MatFactorNumeric_MUMPS<span class="Apple-converted-space"> </span><br>
>>> finish MatFactorNumeric_MUMPS<span class="Apple-converted-space"> </span><br>
>>> MatSolve_MUMPS<span class="Apple-converted-space"> </span><br>
>>> Iteration 1 Step 0.0005 Time 0.0005<span class="Apple-converted-space"> </span><br>
>>> MatSolve_MUMPS<span class="Apple-converted-space"> </span><br>
>>> Iteration 2 Step 0.0005 Time 0.001<span class="Apple-converted-space"> </span><br>
>>> MatSolve_MUMPS<span class="Apple-converted-space"> </span><br>
>>> [INFO]: End of program!!!<span class="Apple-converted-space"> </span><br>
>>> I am wondering if there is any possibility to split the numeric<span class="Apple-converted-space"> </span><br>
>>> and solve stage (as you mentioned using KSPSolve).<span class="Apple-converted-space"> </span><br>
>>> Thank you very much indeed.<span class="Apple-converted-space"> </span><br>
>>> Kind Regards,<span class="Apple-converted-space"> </span><br>
>>> Shidi<span class="Apple-converted-space"> </span><br>
>>> On 2018-05-04 21:10, Y. Shidi wrote:<span class="Apple-converted-space"> </span><br>
>>> Thank you very much for your reply.<span class="Apple-converted-space"> </span><br>
>>> That is really clear.<span class="Apple-converted-space"> </span><br>
>>> Kind Regards,<span class="Apple-converted-space"> </span><br>
>>> Shidi<span class="Apple-converted-space"> </span><br>
>>> On 2018-05-04 21:05, Matthew Knepley wrote:<span class="Apple-converted-space"> </span><br>
>>> On Fri, May 4, 2018 at 3:54 PM, Y. Shidi <ys453@cam.ac.uk> wrote:<span class="Apple-converted-space"> </span><br>
>>> Dear Matt,<span class="Apple-converted-space"> </span><br>
>>> Thank you very much for your reply!<span class="Apple-converted-space"> </span><br>
>>> So what you mean is that I can just do the KSPSolve() every<span class="Apple-converted-space"> </span><br>
>>> iteration<span class="Apple-converted-space"> </span><br>
>>> once the MUMPS is set?<span class="Apple-converted-space"> </span><br>
>>> Yes.<span class="Apple-converted-space"> </span><br>
>>> That means inside the KSPSolve() the numerical factorization is<span class="Apple-converted-space"> </span><br>
>>> performed. If that is the case, it seems that the ksp object is<span class="Apple-converted-space"> </span><br>
>>> not changed when the values in the matrix are changed.<span class="Apple-converted-space"> </span><br>
>>> Yes.<span class="Apple-converted-space"> </span><br>
>>> Or do I need to call both KSPSetOperators() and KSPSolve()?<span class="Apple-converted-space"> </span><br>
>>> If you do SetOperators, it will redo the factorization. If you do<span class="Apple-converted-space"> </span><br>
>>> not,<span class="Apple-converted-space"> </span><br>
>>> it will look<span class="Apple-converted-space"> </span><br>
>>> at the Mat object, determine that the structure has not changed,<span class="Apple-converted-space"> </span><br>
>>> and<span class="Apple-converted-space"> </span><br>
>>> just redo<span class="Apple-converted-space"> </span><br>
>>> the numerical factorization.<span class="Apple-converted-space"> </span><br>
>>> Thanks,<span class="Apple-converted-space"> </span><br>
>>> Matt<span class="Apple-converted-space"> </span><br>
>>> On 2018-05-04 14:44, Matthew Knepley wrote:<span class="Apple-converted-space"> </span><br>
>>> On Fri, May 4, 2018 at 9:40 AM, Y. Shidi <ys453@cam.ac.uk> wrote:<span class="Apple-converted-space"> </span><br>
>>> Dear PETSc users,<span class="Apple-converted-space"> </span><br>
>>> I am currently using MUMPS to solve linear systems directly.<span class="Apple-converted-space"> </span><br>
>>> Generally, we use ICNTL(7) or ICNTL(29) to do the preprocessing<span class="Apple-converted-space"> </span><br>
>>> step and then solve the system.<span class="Apple-converted-space"> </span><br>
>>> In my code, the values in the matrix is changed in each iteration,<span class="Apple-converted-space"> </span><br>
>>> but the structure of the matrix stays the same, which means the<span class="Apple-converted-space"> </span><br>
>>> performance can be improved if symbolic factorisation is only<span class="Apple-converted-space"> </span><br>
>>> performed once. Hence, it is necessary to split the symbolic<span class="Apple-converted-space"> </span><br>
>>> and numeric factorisation. However, I cannot find a specific step<span class="Apple-converted-space"> </span><br>
>>> (control parameter) to perform the numeric factorisation.<span class="Apple-converted-space"> </span><br>
>>> I have used ICNTL(3) and ICNTL(4) to print the MUMPS information,<span class="Apple-converted-space"> </span><br>
>>> it seems that the symbolic and numeric factorisation always perform<span class="Apple-converted-space"> </span><br>
>>> together.<span class="Apple-converted-space"> </span><br>
>>> If you use KSPSolve instead, it will automatically preserve the<span class="Apple-converted-space"> </span><br>
>>> symbolic<span class="Apple-converted-space"> </span><br>
>>> factorization.<span class="Apple-converted-space"> </span><br>
>>> Thanks,<span class="Apple-converted-space"> </span><br>
>>> Matt<span class="Apple-converted-space"> </span><br>
>>> So I am wondering if anyone has an idea about it.<span class="Apple-converted-space"> </span><br>
>>> Below is how I set up MUMPS solver:<span class="Apple-converted-space"> </span><br>
>>> PC pc;<span class="Apple-converted-space"> </span><br>
>>> PetscBool flg_mumps, flg_mumps_ch;<span class="Apple-converted-space"> </span><br>
>>> flg_mumps = PETSC_FALSE;<span class="Apple-converted-space"> </span><br>
>>> flg_mumps_ch = PETSC_FALSE;<span class="Apple-converted-space"> </span><br>
>>> PetscOptionsGetBool(NULL, NULL, "-use_mumps_lu", &flg_mumps,<span class="Apple-converted-space"> </span><br>
>>> NULL);<span class="Apple-converted-space"> </span><br>
>>> PetscOptionsGetBool(NULL, NULL, "-use_mumps_ch", &flg_mumps_ch,<span class="Apple-converted-space"> </span><br>
>>> NULL);<span class="Apple-converted-space"> </span><br>
>>> if(flg_mumps ||flg_mumps_ch)<span class="Apple-converted-space"> </span><br>
>>> {<span class="Apple-converted-space"> </span><br>
>>> KSPSetType(_ksp, KSPPREONLY);<span class="Apple-converted-space"> </span><br>
>>> PetscInt ival,icntl;<span class="Apple-converted-space"> </span><br>
>>> PetscReal val;<span class="Apple-converted-space"> </span><br>
>>> KSPGetPC(_ksp, &pc);<span class="Apple-converted-space"> </span><br>
>>> /// Set preconditioner type<span class="Apple-converted-space"> </span><br>
>>> if(flg_mumps)<span class="Apple-converted-space"> </span><br>
>>> {<span class="Apple-converted-space"> </span><br>
>>> PCSetType(pc, PCLU);<span class="Apple-converted-space"> </span><br>
>>> }<span class="Apple-converted-space"> </span><br>
>>> else if(flg_mumps_ch)<span class="Apple-converted-space"> </span><br>
>>> {<span class="Apple-converted-space"> </span><br>
>>> MatSetOption(A, MAT_SPD, PETSC_TRUE);<span class="Apple-converted-space"> </span><br>
>>> PCSetType(pc, PCCHOLESKY);<span class="Apple-converted-space"> </span><br>
>>> }<span class="Apple-converted-space"> </span><br>
>>> PCFactorSetMatSolverPackage(pc, MATSOLVERMUMPS);<span class="Apple-converted-space"> </span><br>
>>> PCFactorSetUpMatSolverPackage(pc);<span class="Apple-converted-space"> </span><br>
>>> PCFactorGetMatrix(pc, &_F);<span class="Apple-converted-space"> </span><br>
>>> icntl = 7; ival = 0;<span class="Apple-converted-space"> </span><br>
>>> MatMumpsSetIcntl( _F, icntl, ival );<span class="Apple-converted-space"> </span><br>
>>> MatMumpsSetIcntl(_F, 3, 6);<span class="Apple-converted-space"> </span><br>
>>> MatMumpsSetIcntl(_F, 4, 2);<span class="Apple-converted-space"> </span><br>
>>> }<span class="Apple-converted-space"> </span><br>
>>> KSPSetUp(_ksp);<span class="Apple-converted-space"> </span><br>
>>> Kind Regards,<span class="Apple-converted-space"> </span><br>
>>> Shidi<span class="Apple-converted-space"> </span><br>
>>> --<span class="Apple-converted-space"> </span><br>
>>> What most experimenters take for granted before they begin their<span class="Apple-converted-space"> </span><br>
>>> experiments is infinitely more interesting than any results to<span class="Apple-converted-space"> </span><br>
>>> which<span class="Apple-converted-space"> </span><br>
>>> their experiments lead.<span class="Apple-converted-space"> </span><br>
>>> -- Norbert Wiener<span class="Apple-converted-space"> </span><br>
>>> https://www.cse.buffalo.edu/~knepley/ [1] [1] [1] [1]<span class="Apple-converted-space"> </span><br>
>>> Links:<span class="Apple-converted-space"> </span><br>
>>> ------<span class="Apple-converted-space"> </span><br>
>>> [1] http://www.caam.rice.edu/~mk51/ [2] [2] [2]<span class="Apple-converted-space"> </span><br>
>>> --<span class="Apple-converted-space"> </span><br>
>>> What most experimenters take for granted before they begin their<span class="Apple-converted-space"> </span><br>
>>> experiments is infinitely more interesting than any results to<span class="Apple-converted-space"> </span><br>
>>> which<span class="Apple-converted-space"> </span><br>
>>> their experiments lead.<span class="Apple-converted-space"> </span><br>
>>> -- Norbert Wiener<span class="Apple-converted-space"> </span><br>
>>> https://www.cse.buffalo.edu/~knepley/ [1] [1] [2]<span class="Apple-converted-space"> </span><br>
>>> Links:<span class="Apple-converted-space"> </span><br>
>>> ------<span class="Apple-converted-space"> </span><br>
>>> [1] https://www.cse.buffalo.edu/~knepley/ [1] [1]<span class="Apple-converted-space"> </span><br>
>>> [2] http://www.caam.rice.edu/~mk51/ [2] [2]<span class="Apple-converted-space"> </span><br>
>>> --<span class="Apple-converted-space"> </span><br>
>>> What most experimenters take for granted before they begin their<span class="Apple-converted-space"> </span><br>
>>> experiments is infinitely more interesting than any results to<span class="Apple-converted-space"> </span><br>
>>> which<span class="Apple-converted-space"> </span><br>
>>> their experiments lead.<span class="Apple-converted-space"> </span><br>
>>> -- Norbert Wiener<span class="Apple-converted-space"> </span><br>
>>> https://www.cse.buffalo.edu/~knepley/ [1] [2]<span class="Apple-converted-space"> </span><br>
>>> Links:<span class="Apple-converted-space"> </span><br>
>>> ------<span class="Apple-converted-space"> </span><br>
>>> [1] https://www.cse.buffalo.edu/~knepley/ [1]<span class="Apple-converted-space"> </span><br>
>>> [2] http://www.caam.rice.edu/~mk51/ [2]<span class="Apple-converted-space"> </span><br>
>> --<span class="Apple-converted-space"> </span><br>
>> What most experimenters take for granted before they begin their<span class="Apple-converted-space"> </span><br>
>> experiments is infinitely more interesting than any results to which<span class="Apple-converted-space"> </span><br>
>> their experiments lead.<span class="Apple-converted-space"> </span><br>
>> -- Norbert Wiener<span class="Apple-converted-space"> </span><br>
>> https://www.cse.buffalo.edu/~knepley/ [2]<span class="Apple-converted-space"> </span><br>
>> Links:<span class="Apple-converted-space"> </span><br>
>> ------<span class="Apple-converted-space"> </span><br>
>> [1] https://www.cse.buffalo.edu/~knepley/<span class="Apple-converted-space"> </span><br>
>> [2] http://www.caam.rice.edu/~mk51/ </div>
</div>
</span></blockquote>
</div>
</div>
<p><strong>Alp Dener</strong><br>
Postdoctoral Appointee<br>
Argonne National Laboratory<br>
Mathematics and Computer Science Division</p>
</body>
</html>