<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Now I see my problem. What I want to do is to generate many rows
of Jacobian once. For this, the method that must be used is
coloring. If I am to try it without calling coloring, I have no
way but to give the all processes the whole data, but as you have
pointed out it is very very nonscalable. It instantly multiplies
the memory use with number of processors.</p>
<p>Thank you very much for your constructive replies.</p>
<p>Ali<br>
</p>
<br>
<div class="moz-cite-prefix">On 20-03-2018 17:30, Matthew Knepley
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAMYG4GmQwtjsEqKux9r0UvvqyEwX-gbYwFfpPAimZB35Mg8OuA@mail.gmail.com">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Mar 20, 2018 at 10:11 AM, Ali
Berk Kahraman <span dir="ltr"><<a
href="mailto:aliberkkahraman@yahoo.com" target="_blank"
moz-do-not-send="true">aliberkkahraman@yahoo.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>First of all, thank you for elaboration.</p>
<p>The problem is, I want to generate J myself, using
FD, for the following reason.<br>
</p>
</div>
</blockquote>
<div>Okay. This generation should happen entirely inside
FormJacobian(), so you do not need MFFD. <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> </p>
<p>I have a WORLD matrix M that I feed into F(u) using
user defined context pointer *ctx. And I call
collective functions on that matrix. <br>
</p>
</div>
</blockquote>
<div>That should be fine. To be very precise, when you
create the SNES, you give it a comm. The FormFunction()
and FormJacobian()</div>
<div>functions are collective on this comm. <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> </p>
<p>I call a for loop inside CreateJacobian function to
perturb the F() entries one by one. But since I have
collective calls on a WORLD matrix M inside F, I have
to do this virtually sequentially(not "truly"
sequential, but other processes must wait while one is
calling F()). I cannot have two different processes
perturb the u vector at different locations, this
confuses the collective calls on that matrix (or so I
have observed, maybe there was another bug in my
code.).</p>
</div>
</blockquote>
<div>Okay, this is a "bug" in your FD algorithm. What
collective calls are you making when doing the finite
difference?</div>
<div><br>
</div>
<div>Note that if you really do need collective calls, then
you are serializing the construction of this matrix,
meaning no matter</div>
<div>how many procs you use, the sequence will still be as
long as one proc. If, however, that is really what you
want, then</div>
<div>you need to make sure that everyone calls that function
for each perturbation. This means you need to tell
everyone how</div>
<div>many calls they need to make (either sum(perturbations)
or max(perturbations) depending on what you are doing).
You</div>
<div>can use MPI_Allreduce() for this at the start of your
function.</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Here, what I want to do is to have a SELF copy of the
u vector and the input matrix M to F() on every
process, so every process can independently work the
data. So, I need a way to copy the WORLD matrix into
SELF. My current idea is to write the WORLD matrix as
a binary file, and read it into a SELF matrix. <br>
</p>
</div>
</blockquote>
<div>That is really nonscalable. I cannot believe this is
really what you want to do. However, you can do it using</div>
<div>VecScatterCreateToAll() and<br>
</div>
<div><br>
</div>
<div> <a
href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateSubMatrices.html"
moz-do-not-send="true">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateSubMatrices.html</a></div>
<div><br>
</div>
<div>However, I would write out the algorithm in more detail
than you have explained it here. So far, I am not</div>
<div>convinced you should be doing this.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> </p>
<p>I am aware that what I want to do should be done
using coloring, much memory and time effective. But, I
am working on adaptive grid finite differences so
there is not an easy way to create coloring on my
grid. If there is a Petsc function that generates the
coloring for me, I can also try it. <br>
</p>
<p>My apologies, even the simple form got this long :)</p>
<br>
<div class="gmail-m_-5336419880793543424moz-cite-prefix">On
20-03-2018 16:47, Matthew Knepley wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Mar 20, 2018 at
9:31 AM, Ali Berk Kahraman <span dir="ltr"><<a
href="mailto:aliberkkahraman@yahoo.com"
target="_blank" moz-do-not-send="true">aliberkkahraman@yahoo.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">Thank your
for the reply.<br>
<br>
I have tried it, but could not get it to work.
For it to work, I have to know at which vector
entry the epsilon in the FD approximation is
to be added. It is an input for my inexpensive
function, and I could not find a way to pass
it in using MFFD context.<br>
</blockquote>
<div><br>
</div>
<div>MFFD has it in the context as current_u.
There is no accessor, but we can easily write
one. However, that is not how MFFD works. Here
is the idea:</div>
<div><br>
</div>
<div> You give us a function F, your
inexpensive function. We get to call F however
we want to call it, and produce the action of
the matrix. So for instance,</div>
<div> we could do this</div>
<div><br>
</div>
<div> F(x + h u) - F(x)</div>
<div> ---------------------</div>
<div> h ||u||</div>
<div><br>
</div>
<div> but we do not have to do that. You should
not worry about how we perturb the function,
you just return the value of the function when
we call it.</div>
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex"> I have
been trying to get it working with
PETSC_COMM_WORLD too with no luck. My guess is
that my FormRHSFunction becomes a collective
function on WORLD, so I cannot generate many
separate copies each doing its own thing. For
this reason I am trying to move it to SELF,
but I cannot do that if I cannot find a way to
represent my WORLD matrix input fully on each
process.<br>
</blockquote>
<div><br>
</div>
<div>I think things have gotten too confused
here for us to help. Lets step back and
simplify. Can you tell us briefly exactly what
you want to accomplish? I will start by</div>
<div>explaining SNES. We are essentially doing
Newton's method. This means calculating a step
du, based on an initial solution u, using this
equation</div>
<div><br>
</div>
<div> J(u) du = -F(u)</div>
<div><br>
</div>
<div>so we need a function to form the residual
F, and maybe the Jacobian J but we can also
form it using calls to F.</div>
<div><br>
</div>
<div>It sounds like you want to do something
inside F, but I cannot figure out what yet.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex"> Sorry I am
stuck in the same question, but any ideas on
how to move a matrix from WORLD to SELF,
similar to the way it is possible with vectors
using VecScatter?<br>
<br>
<br>
<br>
On 19-03-2018 21:08, Smith, Barry F. wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex"> <br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex"> On Mar
19, 2018, at 12:00 PM, Ali Berk Kahraman
<<a
href="mailto:aliberkkahraman@yahoo.com"
target="_blank" moz-do-not-send="true">aliberkkahraman@yahoo.com</a>>
wrote:<br>
<br>
Dear All,<br>
<br>
I have to create my Jacobian numerically
using an FD function I have written, but
the problem is that the function
evaluation FormRHSFunction creates
PETSC_COMM_WORLD objects in it, and takes
1 PETSC_COMM_WORLD matrix as an input. So
I cannot work it on parallel (every
processor calculating its own portion of
Jacobian), I tried but the machine gets
confused.<br>
</blockquote>
Huh. It is fine to have the function
work on PETSC_COMM_WORLD, and it can be in
parallel.<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex"> My
initial idea was to create the objects in
PETSC_COMM_SELF type within the RHS
function, but the input COMM_WORLD matrix
standed where it was. So my question is,
is there a way to get a complete copy of a
distributed matrix on all the processors
as seq matrices?<br>
<br>
I cannot use the Petsc's version FD
function because the FormRHSFunction I
call to create the Jacobian is slightly
different than the original
FormRHSFunction. The original function is
too expensive, so I only calculate the
relevant parts of it in the Jacobian
function.<br>
</blockquote>
You can do this with MatCreateMFFD() and
set your inexpensive function as the
function.<br>
<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex"> Best
Regards,<br>
<br>
Ali Berk Kahraman<br>
M.Sc. Student, Mechanical Engineering<br>
Bogazici Uni. Istanbul, Turkey<br>
<br>
</blockquote>
</blockquote>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<span class="gmail-HOEnZb"><font color="#888888">
<div><br>
</div>
-- <br>
<div
class="gmail-m_-5336419880793543424gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for
granted before they begin their
experiments is infinitely more
interesting than any results to
which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a
href="http://www.caam.rice.edu/%7Emk51/"
target="_blank"
moz-do-not-send="true">https://www.cse.buffalo.edu/~<wbr>knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before
they begin their experiments is infinitely more
interesting than any results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.caam.rice.edu/%7Emk51/"
target="_blank" moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>