<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Alright, thank you.</p>
<p>Nidish<br>
</p>
<div class="moz-cite-prefix">On 8/9/20 1:43 PM, Matthew Knepley
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAMYG4Gmj=sS3QShCj88B3Fr6cDe+FZfDsKVYUUqxO9ptnO3hYg@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr">On Sun, Aug 9, 2020 at 2:37 PM Nidish <<a
href="mailto:nb25@rice.edu" moz-do-not-send="true">nb25@rice.edu</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p><br>
</p>
<div>On 8/9/20 1:45 AM, Barry Smith wrote:<br>
</div>
<blockquote type="cite">
<blockquote type="cite">
<pre>On Aug 9, 2020, at 1:14 AM, Nidish <a href="mailto:nb25@rice.edu" target="_blank" moz-do-not-send="true"><nb25@rice.edu></a> wrote:
Hello,
My question is related to solving a generic nonlinear algebraic system in parallel. I'm using the word generic to point to the fact that the system may or may not be from a finite element/mesh type object - it's just a residual function and a Jacobian function that I can provide.
I followed the serial example in $PETSC_DIR/src/snes/tutorials/ex1.c pretty much verbatim (albeit with a slightly different objective function whose size can be controlled), and have been trying to extend it to a parallel application (the given example is strictly serial).
When I run my code in serial everything works well and I get the solution I expect. However, even if I use just two cores, the code starts returning garbage. I had a feeling it has something to do with index ordering, but I want to get this code working without using a DM object. I'm attaching my code with this email.
TL;DR Version: What modifications must be made in "$PETSC_DIR/src/snes/tutorials/ex1.c" to make it work in Parallel? Is it possible to make minimal modifications to make it run in parallel too?
</pre>
</blockquote>
<pre> This is wrong:
VecGetSize(x, &N);
VecGetOwnershipRange(x, &is, &ie);
/* VecGetArrayRead(x, &xx); */
VecGetArray(x, &xx);
for (int i=is; i<ie; ++i)
You appear to be trying to mimic a DM example but without a DM.
VecGetArray() always returns an array with indices from 0 to VecGetLocalSize()
Thus you need to make your loops over local indices, not global indices.
In addition you are likely to need "ghost" values of the input vectors. If you are not using DM then you need to have code that determines what ghost values are needed and create a VecScatter to manage communicating those ghost values. Look for examples that use VecScatterBegin in the formfunction routine for how this may be done.
Barry
</pre>
</blockquote>
<p><font color="#0d29db">I wasn't accounting for the ghost
values! My compiler wasn't giving segfaults when I was
accessing the arrays returned by VecGetArray beyond
the allocation size. (the loop indices was me trying
to be smart, I was accessing xx by xx[i -is] and so
on). <br>
</font></p>
<p><font color="#0d29db">I've changed my implementation
with VecCreateGhost - the code's slightly longer but
it works, thank you for this!</font></p>
<p><font color="#0d29db"><br>
</font></p>
<p><font color="#0d29db">I also had another related
question about the storage of the arrays from
VecGetArray() calls. Do these functions allocate new
locations in memory (and can be expected to be slow)?
Even for the ghosted vectors I found distinct
addresses for the ghost values. I'm assuming the
addresses are distinct to accommodate for cluster
programming, but should a user be judicious in calls
to VecGetArray() ?<br>
</font></p>
</div>
</blockquote>
<div>No. For the vectors you are using, this is a no-copy
operation.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p><font color="#0d29db"> </font></p>
<p><font color="#0d29db">Thank you,<br>
Nidish</font><br>
</p>
<blockquote type="cite">
<blockquote type="cite">
<pre>Thank You,
Nidish
<sness.c>
</pre>
</blockquote>
</blockquote>
<div>-- <br>
Nidish</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before
they begin their experiments is infinitely more
interesting than any results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/"
target="_blank" moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div class="moz-signature">-- <br>
Nidish</div>
</body>
</html>