<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Oct 17, 2017 at 6:08 AM, Michael Werner <span dir="ltr"><<a href="mailto:michael.werner@dlr.de" target="_blank">michael.werner@dlr.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
Because usally this code is called just once. It runs one multiple
processes, but there it's still always processing the whole domain.
I can't run it on only one subdomain. As I understand it now, when I
call it from PETSc, this call is issued once per process, so I would
end up running several contesting instances of the computation on
the whole domain.<br>
<br>
But maybe that's only because I haven't completly understood how MPI
really works in such cases...<br></div></blockquote><div><br></div><div>No, it makes one call in which all processes participate. So you would call your external CFD routine once from all processes, passing in the MPI communicator.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
Kind regards,<br>
Michael<br>
<br>
<div class="m_7608694679485040253moz-cite-prefix">Am 17.10.2017 um 11:50 schrieb Matthew
Knepley:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Oct 17, 2017 at 5:46 AM,
Michael Werner <span dir="ltr"><<a href="mailto:michael.werner@dlr.de" target="_blank">michael.werner@dlr.de</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> I'm not sure what you mean with
this question?<br>
The external CFD code, if that was what you referred to,
can be run in parallel.<br>
</div>
</blockquote>
<div><br>
</div>
<div>Then why is it a problem that "in a parallel case, this
call obviously gets called once per process"?</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<div class="m_7608694679485040253gmail-m_-6501336469052660476moz-cite-prefix">Am
17.10.2017 um 11:11 schrieb Matthew Knepley:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Oct 17, 2017 at
4:21 AM, Michael Werner <span dir="ltr"><<a href="mailto:michael.werner@dlr.de" target="_blank">michael.werner@dlr.de</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> That's something I'm
still struggling with. In the serial case, I
can simply extract the values from the
original grid, and since the ordering of the
Jacobian is the same there is no problem. In
the parallel case this is still a more or
less open question. That's why I thought
about reordering the Jacobian. As long as
the position of the individual IDs is the
same for both, I don't have to care about
their absolute position.<br>
<br>
I also wanted to thank you for your previous
answer, it seems that the application
ordering might be what I'm looking for.
However, in the meantime I stumbled about
another problem, that I have to solve first.
My new problem is, that I call the external
code within the shell matrix' multiply call.
But in a parallel case, this call obviously
gets called once per process. So right now
I'm trying to circumvent this, so it might
take a while before I'm able to come back to
the original problem...<br>
</div>
</blockquote>
<div><br>
</div>
<div>I am not understanding. Is your original
code parallel?</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> Kind regards,<br>
Michael<br>
<br>
<div class="m_7608694679485040253gmail-m_-6501336469052660476m_-5791166958946276202moz-cite-prefix">Am
16.10.2017 um 17:25 schrieb Praveen C:<br>
</div>
<blockquote type="cite">
<div dir="ltr">I am interested to learn
more about how this works. How are the
vectors created if the ids are not
contiguous in a partition ?
<div><br>
</div>
<div>Thanks</div>
<div>praveen</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Oct 16,
2017 at 2:02 PM, Stefano Zampini <span dir="ltr"><<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">
<div>
<div class="m_7608694679485040253gmail-m_-6501336469052660476m_-5791166958946276202h5">2017-10-16
10:26 GMT+03:00 Michael
Werner <span dir="ltr"><<a href="mailto:michael.werner@dlr.de" target="_blank">michael.werner@dlr.de</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello,<br>
<br>
I'm having trouble with
parallelizing a
matrix-free code with
PETSc. In this code, I
use an external CFD code
to provide the
matrix-vector product
for an iterative solver
in PETSc. To increase
convergence rate, I'm
using an explicitly
stored Jacobian matrix
to precondition the
solver. This works fine
for serial runs.
However, when I try to
use multiple processes,
I face the problem that
PETSc decomposes the
preconditioner matrix,
and probably also the
shell matrix, in a
different way than the
external CFD code
decomposes the grid.<br>
<br>
The Jacobian matrix is
built in a way, that its
rows and columns
correspond to the global
IDs of the individual
points in my CFD mesh<br>
<br>
The CFD code decomposes
the domain based on the
proximity of points to
each other, so that the
resulting subgrids are
coherent. However, since
its an unstructured
grid, those subgrids are
not necessarily made up
of points with
successive global IDs.
This is a problem, since
PETSc seems to partition
the matrix in coherent
slices.<br>
<br>
I'm not sure what the
best approach to this
problem might be. Is it
maybe possible to
exactly tell PETSc,
which rows/columns it
should assign to the
individual processes?<br>
<br>
</blockquote>
<div><br>
</div>
</div>
</div>
<div>If you are explicitly
setting the values in your
Jacobians via
MatSetValues(), you can
create a
ISLocalToGlobalMapping </div>
<div><br>
</div>
<div><a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/IS/ISLocalToGlobalMappingCreate.html" target="_blank">http://www.mcs.anl.gov/petsc/p<wbr>etsc-current/docs/manualpages/<wbr>IS/ISLocalToGlobalMappingCreat<wbr>e.html</a></div>
<div><br>
</div>
<div>that maps the numbering
you use for the Jacobians to
their counterpart in the CFD
ordering, then call
MatSetLocalToGlobalMapping
and then use
MatSetValuesLocal with the
same arguments you are
calling MatSetValues now.</div>
<div><br>
</div>
<div>Otherwise, you can play
with the application
ordering <a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/AO/index.html" target="_blank">http://www.mcs.anl.gov/petsc/p<wbr>etsc-current/docs/manualpages/<wbr>AO/index.html</a></div>
<span class="m_7608694679485040253gmail-m_-6501336469052660476m_-5791166958946276202HOEnZb"><font color="#888888">
<div> </div>
</font></span></div>
<span class="m_7608694679485040253gmail-m_-6501336469052660476m_-5791166958946276202HOEnZb"><font color="#888888"><br>
<br clear="all">
<span class="m_7608694679485040253gmail-m_-6501336469052660476HOEnZb"><font color="#888888">
<div><br>
</div>
-- <br>
<div class="m_7608694679485040253gmail-m_-6501336469052660476m_-5791166958946276202m_-5153143099617110787gmail_signature">Stefano</div>
</font></span></font></span></div>
<span class="m_7608694679485040253gmail-m_-6501336469052660476HOEnZb"><font color="#888888"> </font></span></div>
<span class="m_7608694679485040253gmail-m_-6501336469052660476HOEnZb"><font color="#888888"> </font></span></blockquote>
<span class="m_7608694679485040253gmail-m_-6501336469052660476HOEnZb"><font color="#888888"> </font></span></div>
<span class="m_7608694679485040253gmail-m_-6501336469052660476HOEnZb"><font color="#888888"> <br>
</font></span></div>
<span class="m_7608694679485040253gmail-m_-6501336469052660476HOEnZb"><font color="#888888"> </font></span></blockquote>
<span class="m_7608694679485040253gmail-m_-6501336469052660476HOEnZb"><font color="#888888"> <br>
<pre class="m_7608694679485040253gmail-m_-6501336469052660476m_-5791166958946276202moz-signature" cols="72">--
______________________________<wbr>______________________
Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR)
Institut für Aerodynamik und Strömungstechnik | <a href="https://maps.google.com/?q=Bunsenstr.+10+%7C+37073+G%C3%B6ttingen&entry=gmail&source=g" target="_blank">Bunsenstr. 10 | 37073 Göttingen</a>
Michael Werner
Telefon 0551 709-2627 | Telefax 0551 709-2811 | <a class="m_7608694679485040253gmail-m_-6501336469052660476m_-5791166958946276202moz-txt-link-abbreviated" href="mailto:Michael.Werner@dlr.de" target="_blank">Michael.Werner@dlr.de</a>
DLR.de
</pre>
</font></span></div>
</blockquote>
</div>
<br>
<br clear="all">
<span class="m_7608694679485040253gmail-HOEnZb"><font color="#888888">
<div><br>
</div>
-- <br>
<div class="m_7608694679485040253gmail-m_-6501336469052660476gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for
granted before they begin their
experiments is infinitely more
interesting than any results to
which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
<span class="m_7608694679485040253gmail-HOEnZb"><font color="#888888"> </font></span></div>
<span class="m_7608694679485040253gmail-HOEnZb"><font color="#888888"> </font></span></blockquote>
<span class="m_7608694679485040253gmail-HOEnZb"><font color="#888888"> <br>
<pre class="m_7608694679485040253gmail-m_-6501336469052660476moz-signature" cols="72">--
______________________________<wbr>______________________
Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR)
Institut für Aerodynamik und Strömungstechnik | <a href="https://maps.google.com/?q=Bunsenstr.+10+%7C+37073+G%C3%B6ttingen&entry=gmail&source=g" target="_blank">Bunsenstr. 10 | 37073 Göttingen</a>
Michael Werner
Telefon 0551 709-2627 | Telefax 0551 709-2811 | <a class="m_7608694679485040253gmail-m_-6501336469052660476moz-txt-link-abbreviated" href="mailto:Michael.Werner@dlr.de" target="_blank">Michael.Werner@dlr.de</a>
DLR.de
</pre>
</font></span></div>
</blockquote>
</div>
<br>
<br clear="all"><span class="HOEnZb"><font color="#888888">
<div><br>
</div>
-- <br>
<div class="m_7608694679485040253gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before
they begin their experiments is infinitely more
interesting than any results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">https://www.cse.buffalo.edu/~<wbr>knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
<pre class="m_7608694679485040253moz-signature" cols="72">--
______________________________<wbr>______________________
Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR)
Institut für Aerodynamik und Strömungstechnik | <a href="https://maps.google.com/?q=Bunsenstr.+10+%7C+37073+G%C3%B6ttingen&entry=gmail&source=g">Bunsenstr. 10 | 37073 Göttingen</a>
Michael Werner
Telefon 0551 709-2627 | Telefax 0551 709-2811 | <a class="m_7608694679485040253moz-txt-link-abbreviated" href="mailto:Michael.Werner@dlr.de" target="_blank">Michael.Werner@dlr.de</a>
DLR.de
</pre>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div>
</div></div>