<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, Apr 18, 2018 at 5:52 PM, Sepideh Kavousi <span dir="ltr"><<a href="mailto:skavou1@lsu.edu" target="_blank">skavou1@lsu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_-5359782560104242096divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Mathew and Dave,</p>
<p style="margin-top:0;margin-bottom:0">Thank you so much it is working perfectly now.</p></div></div></blockquote><div><br></div><div>Excellent.</div><div><br></div><div>If you want your name to appear on the next PETSc release as a contributor, you</div><div>can make a PR with this change :)</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div id="m_-5359782560104242096divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Sepideh<br>
</p>
</div>
<hr style="display:inline-block;width:98%">
<div id="m_-5359782560104242096divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Dave May <<a href="mailto:dave.mayhem23@gmail.com" target="_blank">dave.mayhem23@gmail.com</a>><br>
<b>Sent:</b> Wednesday, April 18, 2018 3:13:33 PM<br>
<b>To:</b> Sepideh Kavousi<br>
<b>Cc:</b> Matthew Knepley; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
<b>Subject:</b> Re: [petsc-users] error running parallel on cluster</font>
<div> </div>
</div>
<div>
<div dir="ltr"><br>
<div class="m_-5359782560104242096x_gmail_extra"><br>
<div class="m_-5359782560104242096x_gmail_quote">On 18 April 2018 at 21:06, Sepideh Kavousi <span dir="ltr">
<<a href="mailto:skavou1@lsu.edu" target="_blank">skavou1@lsu.edu</a>></span> wrote:<br>
<blockquote class="m_-5359782560104242096x_gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_-5359782560104242096x_m_6365648478595197029divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif">
<p style="margin-top:0;margin-bottom:0">Mathew</p>
<p style="margin-top:0;margin-bottom:0">I added the lines and I still have the same issue. It may be a silly question but should I configure and install petsc again using this new lines added? or changing the line is enough? the highlighted lines are the lines
I modified. <br>
</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0"></p>
<div>PetscErrorCode ierr;<br>
DM dm;<br>
DMTS_DA *dmdats = (DMTS_DA*)ctx;<br>
DMDALocalInfo info;<br>
<span style="background-color:rgb(255,255,0)">Vec Xloc,Xdotloc</span>;<br>
void *x,*f,*xdot;<br>
<br>
PetscFunctionBegin;<br>
PetscValidHeaderSpecific(ts,TS<wbr>_CLASSID,1);<br>
PetscValidHeaderSpecific(X,VEC<wbr>_CLASSID,2);<br>
PetscValidHeaderSpecific(F,VEC<wbr>_CLASSID,3);<br>
if (!dmdats->ifunctionlocal) SETERRQ(PetscObjectComm((Petsc<wbr>Object)ts),PETSC_ERR_PLIB,"<wbr>Corrupt context");<br>
ierr = TSGetDM(ts,&dm);CHKERRQ(ierr);<span><br>
<span style="background-color:rgb(255,255,0)"> ierr = DMGetLocalVector(dm,&Xdotloc);<wbr>CHKERRQ(ierr);</span><br>
<span style="background-color:rgb(255,255,0)"> ierr = DMGlobalToLocalBegin(dm,Xdot,I<wbr>NSERT_VALUES,Xdotloc);CHKERRQ(<wbr>ierr);</span><br>
<span style="background-color:rgb(255,255,0)"> ierr = DMGlobalToLocalEnd(dm,Xdot,INS<wbr>ERT_VALUES,Xdotloc);CHKERRQ(<wbr>ierr);
</span><br>
</span> ierr = DMGetLocalVector(dm,&Xloc);CHK<wbr>ERRQ(ierr);<br>
ierr = DMGlobalToLocalBegin(dm,X,INSE<wbr>RT_VALUES,Xloc);CHKERRQ(ierr);<br>
ierr = DMGlobalToLocalEnd(dm,X,INSERT<wbr>_VALUES,Xloc);CHKERRQ(ierr);<br>
ierr = DMDAGetLocalInfo(dm,&info);CHK<wbr>ERRQ(ierr);<br>
ierr = DMDAVecGetArray(dm,Xloc,&x);CH<wbr>KERRQ(ierr);<br>
<span style="background-color:rgb(255,255,0)"> ierr = DMDAVecGetArray(dm,Xdotloc,&xd<wbr>ot);CHKERRQ(ierr);</span></div>
<p></p>
<p style="margin-top:0;margin-bottom:0"></p>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Don't forget to include these calls (in this order) after you are done with the Xdotloc vector</div>
<div><br>
</div>
<div>
<div>ierr = DMDAVecRestoreArray(dm,<wbr>Xdotloc,&xdot);CHKERRQ(ierr);</div>
</div>
<div>ierr = DMRestoreLocalVector(dm,&<wbr>Xdotloc);CHKERRQ(ierr);</div>
<div><br>
</div>
<div>Failure to do so will result in a memory leak.</div>
<div><br>
</div>
<div> </div>
<blockquote class="m_-5359782560104242096x_gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_-5359782560104242096x_m_6365648478595197029divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif">
<p style="margin-top:0;margin-bottom:0">Thanks,</p>
<p style="margin-top:0;margin-bottom:0">Sepideh<br>
</p>
</div>
<hr style="display:inline-block;width:98%">
<div id="m_-5359782560104242096x_m_6365648478595197029divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Sent:</b> Tuesday, April 17, 2018 5:59:12 PM
<div>
<div class="m_-5359782560104242096x_h5"><br>
<b>To:</b> Sepideh Kavousi<br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
<b>Subject:</b> Re: [petsc-users] error running parallel on cluster</div>
</div>
</font>
<div> </div>
</div>
<div>
<div class="m_-5359782560104242096x_h5">
<div>
<div dir="ltr">
<div class="m_-5359782560104242096x_m_6365648478595197029x_gmail_extra">
<div class="m_-5359782560104242096x_m_6365648478595197029x_gmail_quote">On Tue, Apr 17, 2018 at 3:07 PM, Sepideh Kavousi
<span dir="ltr"><<a href="mailto:skavou1@lsu.edu" target="_blank">skavou1@lsu.edu</a>></span> wrote:<br>
<blockquote class="m_-5359782560104242096x_m_6365648478595197029x_gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif">
<p style="margin-top:0;margin-bottom:0">The reason I can not use your method is that the input arguments of SetIFunctionLocal are the arrays of x,x_t instead of x,x_t vectors. In your method which was:</p>
<p style="margin-top:0;margin-bottom:0"><font size="3" face="Calibri,Helvetica,sans-serif,EmojiFont,Apple Color Emoji,Segoe UI Emoji,NotoColorEmoji,Segoe UI Symbol,Android Emoji,EmojiSymbols" color="black"><span id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407m_1367266365533787276divtagdefaultwrapper" style="font-size:12pt"></span></font></p>
<pre align="left" style="text-indent:0;text-align:justify;background-color:white;margin:0;padding:0"><font size="1" face="SFMono-Medium,SF Mono,Segoe UI Mono,Roboto Mono,Ubuntu Mono,Menlo,Courier,monospace" color="#172B4D"><span style="font-size:12px;background-color:white;font-weight:normal;text-transform:none">ierr <b>=</b> DMGetLocalVector(dm,<b>&</b>Xdotloc);<wbr>CHKERRQ(ierr);
<a name="m_-5359782560104242096_x_m_6365648478595197029_x_m_2201054418181576407_m_1367266365533787276_x_dmdats.c-70" id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407LPlnk347913"><font color="#0052CC"><span id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407m_1367266365533787276LPlnk399294"></span></font></a> ierr <b>=</b> DMGlobalToLocalBegin(dm,Xdot,I<wbr>NSERT_VALUES,Xdotloc);CHKERRQ(<wbr>ierr);
<a name="m_-5359782560104242096_x_m_6365648478595197029_x_m_2201054418181576407_m_1367266365533787276_x_dmdats.c-71" id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407LPlnk477188"><font color="#0052CC"><span id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407m_1367266365533787276LPlnk18444"></span></font></a> ierr <b>=</b> DMGlobalToLocalEnd(dm,Xdot,INS<wbr>ERT_VALUES,Xdotloc);CHKERRQ(ie<wbr>rr); <div align="left" style="text-indent:0;text-align:justify;background-color:white;margin:0;padding:0"><font size="1" face="SFMono-Medium,SF Mono,Segoe UI Mono,Roboto Mono,Ubuntu Mono,Menlo,Courier,monospace" color="#172B4D"><span style="font-size:12px;background-color:white"> ierr <b>=</b> DMDAVecGetArray(dm,Xdotloc,<b>&</b>xd<wbr>ot);CHKERRQ(ierr); </span></font></div></span></font></pre>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>You misunderstand my suggestion. I mean stick this code in right here in PETSc</div>
<div><br>
</div>
<div>
<div style="color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px;font-style:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)">
<a href="https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&fileviewer=file-view-default#dmdats.c-68" id="m_-5359782560104242096x_m_6365648478595197029x_gmail-m_2201054418181576407x_m_1367266365533787276LPlnk856698" style="color:rgb(17,85,204)" target="_blank">https://bitbucket.org/petsc/pe<wbr>tsc/annotate/be3efd428a942676a<wbr>0189b3273b3c582694ff011/src/ts<wbr>/utils/dmdats.c?at=master&file<wbr>viewer=file-view-default#dmdat<wbr>s.c-68</a><br>
</div>
<br class="m_-5359782560104242096x_m_6365648478595197029x_gmail-Apple-interchange-newline">
Then the X_t array you get in your local function will be ghosted.</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="m_-5359782560104242096x_m_6365648478595197029x_gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif">
<p style="margin-top:0;margin-bottom:0">I need to have the vector of Xdot, not the array. So I think I should use SetIFunction instead of SetIFunctionLocal.</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Sepideh<br>
</p>
</div>
<hr style="display:inline-block;width:98%">
<div id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Sent:</b> Tuesday, April 17, 2018 1:22:53 PM<br>
<b>To:</b> Sepideh Kavousi<br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
<b>Subject:</b> Re: [petsc-users] error running parallel on cluster</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_gmail_extra">
<div class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_gmail_quote">On Tue, Apr 17, 2018 at 1:50 PM, Sepideh Kavousi
<span dir="ltr"><<a href="mailto:skavou1@lsu.edu" target="_blank">skavou1@lsu.edu</a>></span> wrote:<br>
<blockquote class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<div id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0;margin-bottom:0">Mathew,</p>
I previously use <span>DMDATSSetIFunctionLocal(user.d<wbr>a,INSERT_VALUES,(DMDATSIFuncti<wbr>onLocal) FormFunction,&user) in my code. If I want to use your solution I can not use it because in the
<span>FormFunction definition I must use arrays, not vectors.So to solve this issue I followed two methods where none were able to solve it.
<br>
</span></span></div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span><span>1- in first method I decided to use <span>TSSetIFunction</span> instead of
<span>DMDATSSetIFunctionLocal</span><br>
</span></span></div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span><span><br>
</span></span></div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span><span>for this means first in the main function, I use <span>TSSetDM and my form function variables were as:</span></span></span></div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span><span><span><span><span>PetscErrorCode</span> FormFunction(TS ts,PetscScalar t,Vec Y,Vec Ydot,Vec F, struct VAR_STRUCT *user)</span> {</span></span></span></div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span><span><span>.</span></span></span></div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span><span><span>.</span></span></span></div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span><span><span>.</span></span></span></div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span><span><span>.<br>
</span></span></span></div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span><span><span>
<div> ierr = TSGetDM(ts,&dmda);CHKERRQ(ierr<wbr>);<br>
ierr= DMDAGetLocalInfo(dmda,&info2) ;CHKERRQ(ierr);<br>
<br>
ierr = DMGetLocalVector(dmda,&Ydot_lo<wbr>cal);CHKERRQ(ierr);<br>
ierr = DMGlobalToLocalBegin(dmda,Ydot<wbr>,INSERT_VALUES,Ydot_local);CHK<wbr>ERRQ(ierr);<br>
ierr = DMGlobalToLocalEnd(dmda,Ydot,I<wbr>NSERT_VALUES,Ydot_local);CHKER<wbr>RQ(ierr); </div>
.</span></span></span></div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span><span><span>.</span></span></span></div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span><span><span>.</span></span></span></div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span><span><span><br>
</span></span></span></div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span><span><span>}<br>
</span></span></span></div>
But still, it does not consider vectors y,ydot,f related to dmda (problem executing
<span>DMDAVecGetArray</span>)</div>
</div>
</blockquote>
<div><br>
</div>
<div>We cannot help you if you do not show full error messages.</div>
<div><br>
</div>
<div>Why not fix the code with SetIFunctionLocal(), as I said in my last email. I will fix PETSc proper in branch at the end of the week. I</div>
<div>have a proposal due tomorrow, so I cannot do it right now.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<div>2- in second method I decided to use <span>DMTSSetIFunction</span></div>
<div><span>but still, FormFunction is in form of <span>TSIFunction</span> where we do not define dm object and I think it does not understand dm and da are connected, although I have used
<span><span><span>TSSetDM in the main function. <br>
</span></span></span></span></div>
<div><span><span><span><span><br>
</span></span></span></span></div>
<div><span><span><span><span></span></span></span></span>Can you please help me what should I do?</div>
<div>Regards,</div>
<div>Sepideh<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<br>
<p></p>
</div>
<hr style="display:inline-block;width:98%">
<div id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Sent:</b> Monday, April 16, 2018 9:34:01 PM
<div>
<div class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_h5"><br>
<b>To:</b> Sepideh Kavousi<br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
<b>Subject:</b> Re: [petsc-users] error running parallel on cluster</div>
</div>
</font>
<div> </div>
</div>
<div>
<div class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_h5">
<div>
<div dir="ltr">
<div class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail_extra">
<div class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail_quote">
On Mon, Apr 16, 2018 at 10:22 PM, Sepideh Kavousi <span dir="ltr"><<a href="mailto:skavou1@lsu.edu" id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276LPlnk326764" target="_blank">skavou1@lsu.edu</a>></span> wrote:<br>
<blockquote class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0px;margin-bottom:0px">I run with valgrind with the following command:</p>
<p style="margin-top:0px;margin-bottom:0px"><span>/work/skavou1/petsc/debug/bin/<wbr>mpiexec -np 2 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p ./one.out -malloc off -ts_monitor -snes_fd_color -ts_max_snes_failures -1 -ts_type beuler
-pc_type bjacobi -snes_linesearch_type nleqerr -snes_type newtontr -ksp_gmres_restart 31 -sub_pc_type ilu -snes_monitor -snes_converged_reason -ksp_monitor</span></p>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Okay, I think this is coded wrong. The DMDA stuff ignores DMLocal and only chooses to ghost the input vector X and not X_t</div>
<div><br>
</div>
<div><a href="https://bitbucket.org/petsc/petsc/annotate/be3efd428a942676a0189b3273b3c582694ff011/src/ts/utils/dmdats.c?at=master&fileviewer=file-view-default#dmdats.c-68" id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276LPlnk856698" target="_blank">https://bitbucket.org/petsc/pe<wbr>tsc/annotate/be3efd428a942676a<wbr>0189b3273b3c582694ff011/src/ts<wbr>/utils/dmdats.c?at=master&file<wbr>viewer=file-view-default#dmdat<wbr>s.c-68</a><br>
</div>
<div><br>
</div>
<div>Unfortunately, this was done by Peter in 2012 and he is gone, so we can't blame him.</div>
<div>It is true that very few people use spatial derivatives of X_t, but it does make sense.</div>
<div>Right there, you want to add</div>
<div><br>
</div>
<div>
<pre style="margin:0px;padding:0px;font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Courier,monospace;font-size:12px;line-height:1.4;letter-spacing:0px;color:rgb(23,43,77);font-style:normal;font-weight:400;text-align:left;text-indent:0px;text-transform:none;word-spacing:0px;background-color:rgb(255,255,255)"><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n"> ierr</span> <span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-o" style="font-weight:700">=</span> <span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">DMGetLocalVector</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">(</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">dm</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">,</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-o" style="font-weight:700">&</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">Xdotloc</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">);</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n"><wbr>CHKERRQ</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">(</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">ierr</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">);</span>
<a name="m_-5359782560104242096_x_m_6365648478595197029_x_m_2201054418181576407_x_m_1367266365533787276_x_dmdats.c-70" id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276LPlnk399294" style="color:rgb(0,82,204);text-decoration:none"></a> <span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">ierr</span> <span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-o" style="font-weight:700">=</span> <span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">DMGlobalToLocalBegin</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">(</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">dm</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">,</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">Xdot</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">,</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">I<wbr>NSERT_VALUES</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">,</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">Xdotloc</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">);</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">CHKERRQ</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">(</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n"><wbr>ierr</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">);</span>
<a name="m_-5359782560104242096_x_m_6365648478595197029_x_m_2201054418181576407_x_m_1367266365533787276_x_dmdats.c-71" id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276LPlnk18444" style="color:rgb(0,82,204);text-decoration:none"></a> <span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">ierr</span> <span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-o" style="font-weight:700">=</span> <span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">DMGlobalToLocalEnd</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">(</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">dm</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">,</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">Xdot</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">,</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">INS<wbr>ERT_VALUES</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">,</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">Xdotloc</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">);</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">CHKERRQ</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">(</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">ie<wbr>rr</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">);</span> <pre style="margin:0px;padding:0px;font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Courier,monospace;font-size:12px;line-height:1.4;letter-spacing:0px;color:rgb(23,43,77);font-style:normal;font-weight:400;text-align:left;text-indent:0px;text-transform:none;word-spacing:0px;background-color:rgb(255,255,255)"> <span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">ierr</span> <span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-o" style="font-weight:700">=</span> <span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">DMDAVecGetArray</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">(</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">dm</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">,</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">Xdotloc</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">,</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-o" style="font-weight:700">&</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">xd<wbr>ot</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">);</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">CHKERRQ</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">(</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-n">ierr</span><span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-p">);</span>
</pre><br class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-Apple-interchange-newline"></pre>
and it should work.</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0px;margin-bottom:0px">the valgringoutput is as following:</p>
<p style="margin-top:0px;margin-bottom:0px"></p>
<div>==39396== Invalid read of size 8<br>
==39396== at 0x402679: FormFunction (one.c:212)<br>
==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79)<br>
==39396== by 0x61349BD: TSComputeIFunction (ts.c:830)<br>
==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649)<br>
==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569)<br>
==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203)<br>
==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105)<br>
==39396== by 0x60246C0: SNESSolve (snes.c:4312)<br>
==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176)<br>
==39396== by 0x6237CA8: TSStep_Theta (theta.c:216)<br>
==39396== by 0x6153380: TSStep (ts.c:3548)<br>
==39396== by 0x615559D: TSSolve (ts.c:3731)<br>
==39396== by 0x403C95: main (one.c:303)<br>
==39396== Address 0x83f5158 is 8 bytes before a block of size 7,200 alloc'd<br>
==39396== at 0x4A05588: memalign (vg_replace_malloc.c:727)<br>
==39396== by 0x4D87516: PetscMallocAlign (mal.c:42)<br>
==39396== by 0x4D88DE0: PetscMallocA (mal.c:397)<br>
==39396== by 0x50AB230: VecGetArray2d (rvector.c:2167)<br>
==39396== by 0x589437E: DMDAVecGetArray (dagetarray.c:73)<br>
==39396== by 0x611A8ED: TSComputeIFunction_DMDA (dmdats.c:74)<br>
==39396== by 0x61349BD: TSComputeIFunction (ts.c:830)<br>
==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649)<br>
==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569)<br>
==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203)<br>
==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105)<br>
==39396== by 0x60246C0: SNESSolve (snes.c:4312)<br>
==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176)<br>
==39396== by 0x6237CA8: TSStep_Theta (theta.c:216)<br>
==39396== by 0x6153380: TSStep (ts.c:3548)<br>
==39396== by 0x615559D: TSSolve (ts.c:3731)<br>
==39396== by 0x403C95: main (one.c:303)<br>
==39396== <br>
==39396== Invalid read of size 8<br>
==39396== at 0x402689: FormFunction (one.c:212)<br>
==39396== by 0x611AA5F: TSComputeIFunction_DMDA (dmdats.c:79)<br>
==39396== by 0x61349BD: TSComputeIFunction (ts.c:830)<br>
==39396== by 0x623DD1C: SNESTSFormFunction_Theta (theta.c:649)<br>
==39396== by 0x615E599: SNESTSFormFunction (ts.c:4569)<br>
==39396== by 0x600C84E: SNESComputeFunction (snes.c:2203)<br>
==39396== by 0x60A34B6: SNESSolve_NEWTONTR (tr.c:105)<br>
==39396== by 0x60246C0: SNESSolve (snes.c:4312)<br>
==39396== by 0x6237346: TSTheta_SNESSolve (theta.c:176)<br>
==39396== by 0x6237CA8: TSStep_Theta (theta.c:216)<br>
==39396== by 0x6153380: TSStep (ts.c:3548)<br>
==39396== by 0x615559D: TSSolve (ts.c:3731)<br>
==39396== by 0x403C95: main (one.c:303)<br>
==39396== Address 0x18 is not stack'd, malloc'd or (recently) free'd<br>
==39396== <br>
</div>
<p></p>
<p style="margin-top:0px;margin-bottom:0px">Thanks,</p>
<p style="margin-top:0px;margin-bottom:0px">Sepideh<br>
</p>
</div>
<hr style="display:inline-block;width:98%">
<div id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276LPlnk308356" target="_blank">knepley@gmail.com</a>><br>
<b>Sent:</b> Monday, April 16, 2018 8:44:43 PM<br>
<b>To:</b> Sepideh Kavousi<br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276LPlnk616573" target="_blank">
petsc-users@mcs.anl.gov</a><br>
<b>Subject:</b> Re: [petsc-users] error running parallel on cluster</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_gmail_extra">
<div class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_gmail_quote">
On Mon, Apr 16, 2018 at 9:39 PM, Sepideh Kavousi <span dir="ltr"><<a href="mailto:skavou1@lsu.edu" id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276LPlnk714949" target="_blank">skavou1@lsu.edu</a>></span> wrote:<br>
<blockquote class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_m_-1220196563703675155divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0px;margin-bottom:0px">this is not used forghost points . my code is :</p>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>1) You are using ghost points, since you use i-1 as an index</div>
<div><br>
</div>
<div>2) I was wrong. X_t should also be ghosted.</div>
<div><br>
</div>
<div>Run under valgrind.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_m_-1220196563703675155divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0px;margin-bottom:0px"></p>
<div> for (j=info2->ys;j<info2->ys+info2<wbr>->ym;j++){<br>
for (i=info2->xs;i<info2->xs+info2<wbr>->xm;i++){<br>
if (i==0) {aF[j][i].U=aY[j][i+1].U-aY[j]<wbr>[i].U ; aF[j][i].p=aY[j][i+1].p-aY[j][<wbr>i].p ;}<br>
else if (i==info2->mx-1) {aF[j][i].U=aY[j][i-1].U-aY[j]<wbr>[i].U ; aF[j][i].p=aY[j][i].p-aY[j][i-<wbr>1].p ;}<br>
else if (j==0) {aF[j][i].U=aY[j+1][i].U-aY[j]<wbr>[i].U; aF[j][i].p=aY[j+1][i].p-aY[j][<wbr>i].p ;}<br>
else if (j==info2->my-1) {aF[j][i].U=aY[j-1][i].U-aY[j]<wbr>[i].U ; aF[j][i].p=aY[j][i].p-aY[j-1][<wbr>i].p ;}<br>
else {<br>
//derivatives of U and p<br>
Px=(aY[j][i+1].p-aY[j][i-1].p<wbr>)/(2.0*user->hx);<br>
Py=(aY[j+1][i].p-aY[j-1][i].p<wbr>)/(2.0*user->hy);<br>
Pxx=((aY[j][i+1].p+aY[j][i-1]<wbr>.p-2.0*aY[j][i].p)/(hx2));<br>
Pxy=((aY[j+1][i+1].p+aY[j-1][<wbr>i-1].p-aY[j+1][i-1].p-aY[j-1][<wbr>i+1].p)/(4.0*hxy));<br>
Pyy=((aY[j+1][i].p+aY[j-1][i]<wbr>.p-2.0*aY[j][i].p)/hy2);<br>
Pxt=(aYdot[j][i+1].p-aYdot[j]<wbr>[i-1].p)/(2.0*user->hx);<br>
Pyt=(aYdot[j+1][i].p-aYdot[j-<wbr>1][i].p)/(2.0*user->hy);<br>
<br>
<br>
if (((Px*Px)+(Py*Py))>0.0 ) { //<br>
theta=PetscAtanReal(Py/Px);<br>
thetax=(Px*Pxy-Py*Pxx)/((Px*P<wbr>x)+(Py*Py));<br>
thetay=(Px*Pyy-Py*Pxy)/((Px*P<wbr>x)+(Py*Py));<br>
}<br>
else { <br>
theta=PETSC_PI*0.5;<br>
thetax=0.0;<br>
thetay=0.0;<br>
}<br>
e=(1.0+user->epsilon*cos(4.0*<wbr>theta));;<br>
ep=-(4.0*user->epsilon*sin(4.<wbr>0*theta));<br>
epp=-(4.0*user->epsilon*4.0*c<wbr>os(4.0*theta));<br>
<br>
<br>
<br>
Ux=(aY[j][i+1].U-aY[j][i-1].U<wbr>)/(2.0*user->hx);<br>
Uy=(aY[j+1][i].U-aY[j-1][i].U<wbr>)/(2.0*user->hy);<br>
Uxx=((aY[j][i+1].U+aY[j][i-1]<wbr>.U-2.0*aY[j][i].U)/(hx2));<br>
Uyy=((aY[j+1][i].U+aY[j-1][i]<wbr>.U-2.0*aY[j][i].U)/hy2);<br>
<br>
U1=user->D*user->tau_0/(user-<wbr>>W*user->W);<br>
U2= ((Px*Px+Py*Py)>0.0) ? user->a_p/sqrt(Px*Px+Py*Py) : 0.0 ;<br>
//U2=0.0;<br>
P1= (user->W*user->hy*j-user->Vp*u<wbr>ser->tau_0*t)/user->lT;<br>
<br>
aF[j][i].U=(U1*(-0.5*(Ux*Px+U<wbr>y*Py)+0.5*(1-aY[j][i].p)*(Uxx+<wbr>Uyy))+U2*(1+(1-user->k)*aY[j][<wbr>i].U)*(Pxt*Px+Pyt*Py+aYdot[j][<wbr>i].p*(Pxx+Pyy))+U2*(1-user->k)<wbr>*aYdot[j][i].p*(Ux*Px+Uy*Py)+0<wbr>.5*(1+(1-user->k)*aY[j][i].U)*<wbr>aYdot[j][i].p-0.5*(1+user->k-(<wbr>1-user->k)*aY[j][i].p)*aYdot[j<wbr>][i].U)*hx2;<br>
<br>
aF[j][i].p=( e*e*(Pxx+Pyy)+2.0*e*ep*(thetax<wbr>*Px+thetay*Py)+(e*epp+ep*ep)*(<wbr>thetay*Px-thetax*Py)+(aY[j][i]<wbr>.p-aY[j][i].p*aY[j][i].p*aY[j]<wbr>[i].p)-user->lambda*(aY[j][i].<wbr>p*aY[j][i].p-1)*(aY[j][i].p*aY<wbr>[j][i].p-1)*(aY[j][i].U+P1)-(1<wbr>-(1-user->k)*P1)*(1.0+4.0*user<wbr>->epsilon*cos(4.0*theta))*(1.0<wbr>+4.0*user->epsilon*cos(4.0*the<wbr>ta))*aYdot[j][i].p
)*hx2 ;<br>
<br>
}<br>
}</div>
<p></p>
<p style="margin-top:0px;margin-bottom:0px">Sepideh<br>
</p>
</div>
<hr style="display:inline-block;width:98%">
<div id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_m_-1220196563703675155divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276LPlnk760566" target="_blank">knepley@gmail.com</a>><br>
<b>Sent:</b> Monday, April 16, 2018 8:36:37 PM<br>
<b>To:</b> Sepideh Kavousi<br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276LPlnk630525" target="_blank">
petsc-users@mcs.anl.gov</a><br>
<b>Subject:</b> Re: [petsc-users] error running parallel on cluster</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_m_-1220196563703675155x_gmail_extra">
<div class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_m_-1220196563703675155x_gmail_quote">
On Mon, Apr 16, 2018 at 9:20 PM, Sepideh Kavousi <span dir="ltr"><<a href="mailto:skavou1@lsu.edu" id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276LPlnk863698" target="_blank">skavou1@lsu.edu</a>></span> wrote:<br>
<blockquote class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_m_-1220196563703675155x_gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_m_-1220196563703675155x_m_734736110172724093divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<div>Hello,</div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
I am solving a PDE where I have the spacial derivtive of the time derivative of a variable. In the
<span>DMDATSSetIFunctionLocal</span> function I defined d(dY/dt)/dx as:</div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span>(aYdot[j][i+1].p-aYdot[j][i-1]<wbr>.p)/(2.0*user->hx)</span></div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I do not think that you get ghosted Ydot in that function.</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_m_-1220196563703675155x_gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_m_-1220196563703675155x_m_734736110172724093divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span>on my workstation, it is working but on the cluster it gives an error. I can not run with debugger on the cluster but I have checked and the error is related to this part and I do not have any problem on cluster when running in series. Can you please
help me about it. <br>
</span></div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span><br>
</span></div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span>the error is:</span></div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span>
<div>0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range<br>
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
[0]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276LPlnk744568" target="_blank">
http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html#valgrind</a><br>
[0]PETSC ERROR: or try <a href="http://valgrind.org" id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276LPlnk992665" target="_blank">
http://valgrind.org</a> on GNU/linux and Apple Mac OS X to find memory corruption errors<br>
[0]PETSC ERROR: likely location of problem given in stack below<br>
[0]PETSC ERROR: --------------------- Stack Frames ------------------------------<wbr>------<br>
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>
[0]PETSC ERROR: INSTEAD the line number of the start of the function<br>
[0]PETSC ERROR: is given.<br>
[0]PETSC ERROR: [0] TSComputeIFunction_DMDA line 63 /ddnB/work/skavou1/petsc/src/t<wbr>s/utils/dmdats.c<br>
[0]PETSC ERROR: [0] TS user implicit function line 829 /ddnB/work/skavou1/petsc/src/t<wbr>s/interface/ts.c<br>
[0]PETSC ERROR: [0] TSComputeIFunction line 815 /ddnB/work/skavou1/petsc/src/t<wbr>s/interface/ts.c<br>
[0]PETSC ERROR: [0] SNESTSFormFunction_Theta line 640 /ddnB/work/skavou1/petsc/src/t<wbr>s/impls/implicit/theta/theta.c<br>
[0]PETSC ERROR: [0] SNESTSFormFunction line 4564 /ddnB/work/skavou1/petsc/src/t<wbr>s/interface/ts.c<br>
[0]PETSC ERROR: [0] SNES user function line 2202 /ddnB/work/skavou1/petsc/src/s<wbr>nes/interface/snes.c<br>
[0]PETSC ERROR: [0] SNESComputeFunction line 2187 /ddnB/work/skavou1/petsc/src/s<wbr>nes/interface/snes.c<br>
[0]PETSC ERROR: [0] SNESSolve_NEWTONTR line 90 /ddnB/work/skavou1/petsc/src/s<wbr>nes/impls/tr/tr.c<br>
[0]PETSC ERROR: [0] SNESSolve line 4203 /ddnB/work/skavou1/petsc/src/s<wbr>nes/interface/snes.c<br>
[0]PETSC ERROR: [0] TSTheta_SNESSolve line 175 /ddnB/work/skavou1/petsc/src/t<wbr>s/impls/implicit/theta/theta.c<br>
[0]PETSC ERROR: [0] TSStep_Theta line 191 /ddnB/work/skavou1/petsc/src/t<wbr>s/impls/implicit/theta/theta.c<br>
[0]PETSC ERROR: [0] TSStep line 3526 /ddnB/work/skavou1/petsc/src/t<wbr>s/interface/ts.c<br>
[0]PETSC ERROR: [0] TSSolve line 3668 /ddnB/work/skavou1/petsc/src/t<wbr>s/interface/ts.c<br>
[0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
[0]PETSC ERROR: Signal received<br>
[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276LPlnk423132" target="_blank">
http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.<br>
[0]PETSC ERROR: Petsc Release Version 3.9.0, unknown <br>
[0]PETSC ERROR: ./one.out on a debug named mike1 by skavou1 Mon Apr 16 20:11:44 2018<br>
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack<br>
[0]PETSC ERROR: #1 User provided function() line 0 in unknown file<br>
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0<br>
==29057== <br>
==29057== HEAP SUMMARY:<br>
==29057== in use at exit: 560,151 bytes in 383 blocks<br>
==29057== total heap usage: 8,734 allocs, 8,351 frees, 4,449,350 bytes allocated<br>
==29057== <br>
==29057== LEAK SUMMARY:<br>
==29057== definitely lost: 0 bytes in 0 blocks<br>
==29057== indirectly lost: 0 bytes in 0 blocks<br>
==29057== possibly lost: 0 bytes in 0 blocks<br>
==29057== still reachable: 560,151 bytes in 383 blocks<br>
==29057== suppressed: 0 bytes in 0 blocks<br>
==29057== Rerun with --leak-check=full to see details of leaked memory<br>
==29057== <br>
==29057== For counts of detected and suppressed errors, rerun with: -v<br>
==29057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6)<br>
<br>
</div>
<br>
</span></div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span><br>
</span></div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span><br>
</span></div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span>Thanks,</span></div>
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<span>Sepideh<span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_HOEnZb"><font color="#888888"><br>
</font></span></span></div>
<span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_HOEnZb"><font color="#888888"></font></span></div>
<span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_HOEnZb"><font color="#888888"></font></span></div>
<span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_HOEnZb"><font color="#888888"></font></span></blockquote>
<span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_HOEnZb"><font color="#888888"></font></span></div>
<span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_HOEnZb"><font color="#888888"><br>
<br clear="all">
<span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-HOEnZb"><font color="#888888">
<div><br>
</div>
-- <br>
<div class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_m_-1220196563703675155x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.caam.rice.edu/~mk51/" id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276LPlnk958456" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</font></span></font></span></div>
<span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-HOEnZb"><font color="#888888"></font></span></div>
<span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-HOEnZb"><font color="#888888"></font></span></div>
<span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-HOEnZb"><font color="#888888"></font></span></div>
<span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-HOEnZb"><font color="#888888"></font></span></blockquote>
<span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-HOEnZb"><font color="#888888"></font></span></div>
<span class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-HOEnZb"><font color="#888888"><br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail-m_8651246771025479649x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.caam.rice.edu/~mk51/" id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276LPlnk675356" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<span class="m_-5359782560104242096x_m_6365648478595197029x_HOEnZb"><font color="#888888">
<div><br>
</div>
-- <br>
<div class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.caam.rice.edu/~mk51/" id="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_m_1367266365533787276LPlnk440141" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
<span class="m_-5359782560104242096x_m_6365648478595197029x_HOEnZb"><font color="#888888"></font></span></div>
<span class="m_-5359782560104242096x_m_6365648478595197029x_HOEnZb"><font color="#888888"></font></span></div>
<span class="m_-5359782560104242096x_m_6365648478595197029x_HOEnZb"><font color="#888888"></font></span></div>
<span class="m_-5359782560104242096x_m_6365648478595197029x_HOEnZb"><font color="#888888"></font></span></div>
<span class="m_-5359782560104242096x_m_6365648478595197029x_HOEnZb"><font color="#888888"></font></span></div>
<span class="m_-5359782560104242096x_m_6365648478595197029x_HOEnZb"><font color="#888888"></font></span></div>
<span class="m_-5359782560104242096x_m_6365648478595197029x_HOEnZb"><font color="#888888"></font></span></blockquote>
<span class="m_-5359782560104242096x_m_6365648478595197029x_HOEnZb"><font color="#888888"></font></span></div>
<span class="m_-5359782560104242096x_m_6365648478595197029x_HOEnZb"><font color="#888888"><br>
<br clear="all"><span class="HOEnZb"><font color="#888888">
<div><br>
</div>
-- <br>
<div class="m_-5359782560104242096x_m_6365648478595197029x_m_2201054418181576407x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</font></span></font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="m_-5359782560104242096x_m_6365648478595197029x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div>
</div></div>