<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Mar 20, 2018 at 10:11 AM, Ali Berk Kahraman <span dir="ltr"><<a href="mailto:aliberkkahraman@yahoo.com" target="_blank">aliberkkahraman@yahoo.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>First of all, thank you for elaboration.</p>
<p>The problem is, I want to generate J myself, using FD, for the
following reason.<br></p></div></blockquote><div>Okay. This generation should happen entirely inside FormJacobian(), so you do not need MFFD. <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>
</p>
<p>I have a WORLD matrix M that I feed into F(u) using user defined
context pointer *ctx. And I call collective functions on that
matrix. <br></p></div></blockquote><div>That should be fine. To be very precise, when you create the SNES, you give it a comm. The FormFunction() and FormJacobian()</div><div>functions are collective on this comm. <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>
</p>
<p>I call a for loop inside CreateJacobian function to perturb the
F() entries one by one. But since I have collective calls on a
WORLD matrix M inside F, I have to do this virtually
sequentially(not "truly" sequential, but other processes must wait
while one is calling F()). I cannot have two different processes
perturb the u vector at different locations, this confuses the
collective calls on that matrix (or so I have observed, maybe
there was another bug in my code.).</p></div></blockquote><div>Okay, this is a "bug" in your FD algorithm. What collective calls are you making when doing the finite difference?</div><div><br></div><div>Note that if you really do need collective calls, then you are serializing the construction of this matrix, meaning no matter</div><div>how many procs you use, the sequence will still be as long as one proc. If, however, that is really what you want, then</div><div>you need to make sure that everyone calls that function for each perturbation. This means you need to tell everyone how</div><div>many calls they need to make (either sum(perturbations) or max(perturbations) depending on what you are doing). You</div><div>can use MPI_Allreduce() for this at the start of your function.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
<p>Here, what I want to do is to have a SELF copy of the u vector
and the input matrix M to F() on every process, so every process
can independently work the data. So, I need a way to copy the
WORLD matrix into SELF. My current idea is to write the WORLD
matrix as a binary file, and read it into a SELF matrix. <br></p></div></blockquote><div>That is really nonscalable. I cannot believe this is really what you want to do. However, you can do it using</div><div>VecScatterCreateToAll() and<br></div><div><br></div><div> <a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateSubMatrices.html">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateSubMatrices.html</a></div><div><br></div><div>However, I would write out the algorithm in more detail than you have explained it here. So far, I am not</div><div>convinced you should be doing this.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>
</p>
<p>I am aware that what I want to do should be done using coloring,
much memory and time effective. But, I am working on adaptive grid
finite differences so there is not an easy way to create coloring
on my grid. If there is a Petsc function that generates the
coloring for me, I can also try it. <br>
</p>
<p>My apologies, even the simple form got this long :)</p>
<br>
<div class="gmail-m_-5336419880793543424moz-cite-prefix">On 20-03-2018 16:47, Matthew Knepley
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Mar 20, 2018 at 9:31 AM, Ali
Berk Kahraman <span dir="ltr"><<a href="mailto:aliberkkahraman@yahoo.com" target="_blank">aliberkkahraman@yahoo.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Thank
your for the reply.<br>
<br>
I have tried it, but could not get it to work. For it to
work, I have to know at which vector entry the epsilon in
the FD approximation is to be added. It is an input for my
inexpensive function, and I could not find a way to pass
it in using MFFD context.<br>
</blockquote>
<div><br>
</div>
<div>MFFD has it in the context as current_u. There is no
accessor, but we can easily write one. However, that is
not how MFFD works. Here is the idea:</div>
<div><br>
</div>
<div> You give us a function F, your inexpensive function.
We get to call F however we want to call it, and produce
the action of the matrix. So for instance,</div>
<div> we could do this</div>
<div><br>
</div>
<div> F(x + h u) - F(x)</div>
<div> ---------------------</div>
<div> h ||u||</div>
<div><br>
</div>
<div> but we do not have to do that. You should not worry
about how we perturb the function, you just return the
value of the function when we call it.</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I have been trying to get it working with PETSC_COMM_WORLD
too with no luck. My guess is that my FormRHSFunction
becomes a collective function on WORLD, so I cannot
generate many separate copies each doing its own thing.
For this reason I am trying to move it to SELF, but I
cannot do that if I cannot find a way to represent my
WORLD matrix input fully on each process.<br>
</blockquote>
<div><br>
</div>
<div>I think things have gotten too confused here for us to
help. Lets step back and simplify. Can you tell us briefly
exactly what you want to accomplish? I will start by</div>
<div>explaining SNES. We are essentially doing Newton's
method. This means calculating a step du, based on an
initial solution u, using this equation</div>
<div><br>
</div>
<div> J(u) du = -F(u)</div>
<div><br>
</div>
<div>so we need a function to form the residual F, and maybe
the Jacobian J but we can also form it using calls to F.</div>
<div><br>
</div>
<div>It sounds like you want to do something inside F, but I
cannot figure out what yet.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Sorry I am stuck in the same question, but any ideas on
how to move a matrix from WORLD to SELF, similar to the
way it is possible with vectors using VecScatter?<br>
<br>
<br>
<br>
On 19-03-2018 21:08, Smith, Barry F. wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
On Mar 19, 2018, at 12:00 PM, Ali Berk Kahraman <<a href="mailto:aliberkkahraman@yahoo.com" target="_blank">aliberkkahraman@yahoo.com</a>>
wrote:<br>
<br>
Dear All,<br>
<br>
I have to create my Jacobian numerically using an FD
function I have written, but the problem is that the
function evaluation FormRHSFunction creates
PETSC_COMM_WORLD objects in it, and takes 1
PETSC_COMM_WORLD matrix as an input. So I cannot work
it on parallel (every processor calculating its own
portion of Jacobian), I tried but the machine gets
confused.<br>
</blockquote>
Huh. It is fine to have the function work on
PETSC_COMM_WORLD, and it can be in parallel.<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
My initial idea was to create the objects in
PETSC_COMM_SELF type within the RHS function, but the
input COMM_WORLD matrix standed where it was. So my
question is, is there a way to get a complete copy of
a distributed matrix on all the processors as seq
matrices?<br>
<br>
I cannot use the Petsc's version FD function because
the FormRHSFunction I call to create the Jacobian is
slightly different than the original FormRHSFunction.
The original function is too expensive, so I only
calculate the relevant parts of it in the Jacobian
function.<br>
</blockquote>
You can do this with MatCreateMFFD() and set your
inexpensive function as the function.<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Best Regards,<br>
<br>
Ali Berk Kahraman<br>
M.Sc. Student, Mechanical Engineering<br>
Bogazici Uni. Istanbul, Turkey<br>
<br>
</blockquote>
</blockquote>
<br>
</blockquote>
</div>
<br>
<br clear="all"><span class="gmail-HOEnZb"><font color="#888888">
<div><br>
</div>
-- <br>
<div class="gmail-m_-5336419880793543424gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before
they begin their experiments is infinitely more
interesting than any results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">https://www.cse.buffalo.edu/~<wbr>knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
</div>
</blockquote>
<br>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div>
</div></div>