<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Did you renumber your on-rank nodes after partitioning?<br>
<br>
T<br>
<br>
On 06/28/2012 08:59 AM, Thomas DE-SOZA wrote:
<blockquote
cite="mid:OF766534BE.497D903D-ONC1257A2B.0047F467-C1257A2B.004CD8B9@notes.edfgdf.fr"
type="cite">
<br>
<font size="2" face="sans-serif">Dear PETSc Users,</font>
<br>
<br>
<font size="2" face="sans-serif">We're experiencing performances
issues after having switched to fully distributed meshes in our
in-house code and would like your opinion on the matter.</font>
<br>
<br>
<font size="2" face="sans-serif">In the current version of our
structural mechanics FEA software (<a class="moz-txt-link-freetext" href="http://www.code-aster.org/">http://www.code-aster.org/</a>),
all MPI processes have knowledge of the whole matrix and
therefore can easily pass it to PETSc without the need for any
communication. In a nutshell, stash is empty after MatSetValues
and no mallocs occur during assembly.</font>
<br>
<font size="2" face="sans-serif">We're now building a distributed
version of the software with each process reading its own
subdomain in order to save memory. The mesh was partitioned with
Metis and as a first approach we built a simple partition of the
degrees of freedom based on the gradual subdomains. This
eliminates the need for Application Ordering but yields an
unbalanced decomposition in terms of rows. If we take an example
with 2 MPI processes : processor 0 will have more unknowns than
processor 1 and will receive entries lying on the interface
whereas processor 1 will have all entries locally.</font>
<br>
<br>
<font size="2" face="sans-serif">PETSc manual states that "It is
fine to generate some entries on the "wrong" process. Often this
can lead to cleaner, simpler, less buggy codes. One should never
make code overly complicated in order to generate all values
locally. Rather, one should organize the code in such a way that
most values are generated locally."</font>
<br>
<font size="2" face="sans-serif">Judging from the performance we
obtain on a simple cube with two processes, it seems we have
generated too much entries on the wrong process. Indeed our
distributed code runs slower than the current one. However the
stash does not seem to contain that much (650 000 over a total
of 50 000 000 nnz). We have attached the output obtained with
"-info" as well as the "-log_summary" profiling. Most of the
time is spent in the assembly and a lot of mallocs occur.</font>
<br>
<br>
<br>
<br>
<br>
<font size="2" face="sans-serif">What's your advice on this ? Is
working with ghost cells the only option ?</font>
<br>
<font size="2" face="sans-serif">We were wondering if we could
preallocate the stash for example to decrease the number of
mallocs.</font>
<br>
<br>
<font size="2" face="sans-serif">Regards,</font>
<br>
<br>
<font size="2" face="sans-serif">Thomas</font>
<p><br>
Ce message et toutes les pièces jointes (ci-après le 'Message')
sont établis à l'intention exclusive des destinataires et les
informations qui y figurent sont strictement confidentielles.
Toute utilisation de ce Message non conforme à sa destination,
toute diffusion ou toute publication totale ou partielle, est
interdite sauf autorisation expresse.</p>
<p>Si vous n'êtes pas le destinataire de ce Message, il vous est
interdit de le copier, de le faire suivre, de le divulguer ou
d'en utiliser tout ou partie. Si vous avez reçu ce Message par
erreur, merci de le supprimer de votre système, ainsi que toutes
ses copies, et de n'en garder aucune trace sur quelque support
que ce soit. Nous vous remercions également d'en avertir
immédiatement l'expéditeur par retour du message.</p>
<p>Il est impossible de garantir que les communications par
messagerie électronique arrivent en temps utile, sont sécurisées
ou dénuées de toute erreur ou virus.<br>
____________________________________________________</p>
<p>This message and any attachments (the 'Message') are intended
solely for the addressees. The information contained in this
Message is confidential. Any use of information contained in
this Message not in accord with its purpose, any dissemination
or disclosure, either whole or partial, is prohibited except
formal approval.</p>
<p>If you are not the addressee, you may not copy, forward,
disclose or use any part of it. If you have received this
message in error, please delete it and all copies from your
system and notify the sender immediately by return message.</p>
<p>E-mail communication cannot be guaranteed to be timely secure,
error or virus-free.</p>
</blockquote>
<br>
</body>
</html>