<div dir="ltr"><div dir="ltr">On Fri, Oct 18, 2019 at 5:53 PM Danyang Su <<a href="mailto:danyang.su@gmail.com">danyang.su@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi All,</p>
<p>I am now able to reproduce the partition problem using a
relatively small mesh (attached). The mesh consists of 9087 nodes,
15656 prism cells. There are 39 layers with 233 nodes for each
layer. I have tested the partition using PETSc as well as Gmsh
3.0.1.</p></div></blockquote><div>Great job finding a good test case. Can you send me that mesh?<br></div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
<p>Taking 4 partitions as an example, the partition from PETSc 3.9
and 3.10 are reasonable though not perfect, with total number of
ghost nodes / total number of nodes ratio 2754 / 9087.<br>
</p>
<p>The partition from PETSc 3.11, PETSc 3.12 and PETSc-dev look
weird, with total number of ghost nodes / total number of nodes:
12413 / 9087. The nodes are not well connected for the same
processor.<br>
</p>
<p>Note: the z axis is scaled by 25 for better visualization in
paraview.<br>
</p>
<p><img src="cid:16de1624d9c59c40e261" alt="" width="1164" height="563"></p>
<p><br>
</p>
<p>The partition from Gmsh-Metis is a bit different but still quite
similar to PETSc 3.9 and 3.10.<br>
</p>
<img src="cid:16de1624d9cee4baa0c2" alt="" width="1218" height="683"><br>
<p>Finally, the partition using Gmsh-Chaco Multilevel-KL algorithm
is the best one, with total number of ghost nodes / total number
of nodes: 741 / 9087 . For most of my simulation cases with much
larger meshes, PETSc 3.9 and 3.10 generate partition similar to
the one below, which work pretty well and the code can get very
good speedup. <br>
</p>
<p><img src="cid:16de1624d9cdb2922ee3" alt="" width="1188" height="668"></p>
<p>Thanks,<br>
</p>
<p>Danyang<br>
</p>
<div>On 2019-09-18 11:44 a.m., Danyang Su
wrote:<br>
</div>
<blockquote type="cite">
<br>
On 2019-09-18 10:56 a.m., Smith, Barry F. via petsc-users wrote:
<br>
<blockquote type="cite">
<br>
<blockquote type="cite">On Sep 18, 2019, at 12:25 PM, Mark Lohry
via petsc-users <a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><petsc-users@mcs.anl.gov></a> wrote:
<br>
<br>
Mark,
<br>
</blockquote>
Mark,
<br>
<br>
Good point. This has been a big headache forever
<br>
<br>
Note that this has been "fixed" in the master version of
PETSc and will be in its next release. If you use
--download-parmetis in the future it will use the same random
numbers on all machines and thus should produce the same
partitions on all machines.
<br>
<br>
I think that metis has aways used the same random
numbers and all machines and thus always produced the same
results.
<br>
<br>
Barry
<br>
</blockquote>
Good to know this. I will the same configuration that causes
strange partition problem to test the next version.
<br>
<br>
Thanks,
<br>
<br>
Danyang
<br>
<br>
<blockquote type="cite">
<br>
<br>
<blockquote type="cite">The machine, compiler and MPI version
should not matter.
<br>
<br>
I might have missed something earlier in the thread, but
parmetis has a dependency on the machine's glibc srand, and it
can (and does) create different partitions with different
srand versions. The same mesh on the same code on the same
process count can and will give different partitions (possibly
bad ones) on different machines.
<br>
<br>
On Tue, Sep 17, 2019 at 1:05 PM Mark Adams via petsc-users
<a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><petsc-users@mcs.anl.gov></a> wrote:
<br>
<br>
<br>
On Tue, Sep 17, 2019 at 12:53 PM Danyang Su
<a href="mailto:danyang.su@gmail.com" target="_blank"><danyang.su@gmail.com></a> wrote:
<br>
Hi Mark,
<br>
<br>
Thanks for your follow-up.
<br>
<br>
The unstructured grid code has been verified and there is no
problem in the results. The convergence rate is also good. The
3D mesh is not good, it is based on the original stratum which
I haven't refined, but good for initial test as it is relative
small and the results obtained from this mesh still makes
sense.
<br>
<br>
The 2D meshes are just for testing purpose as I want to
reproduce the partition problem on a cluster using PETSc3.11.3
and Intel2019. Unfortunately, I didn't find problem using this
example.
<br>
<br>
The code has no problem in using different PETSc versions
(PETSc V3.4 to V3.11)
<br>
<br>
OK, it is the same code. I thought I saw something about your
code changing.
<br>
<br>
Just to be clear, v3.11 never gives you good partitions. It is
not just a problem on this Intel cluster.
<br>
<br>
The machine, compiler and MPI version should not matter.
<br>
and MPI distribution (MPICH, OpenMPI, IntelMPI), except for
one simulation case (the mesh I attached) on a cluster with
PETSc3.11.3 and Intel2019u4 due to the very different
partition compared to PETSc3.9.3. Yet the simulation results
are the same except for the efficiency problem because the
strange partition results into much more communication (ghost
nodes).
<br>
<br>
I am still trying different compiler and mpi with PETSc3.11.3
on that cluster to trace the problem. Will get back to you
guys when there is update.
<br>
<br>
<br>
This is very strange. You might want to use 'git bisect'. You
set a good and a bad SHA1 (we can give you this for 3.9 and
3.11 and the exact commands). The git will go to a version in
the middle. You then reconfigure, remake, rebuild your code,
run your test. Git will ask you, as I recall, if the version
is good or bad. Once you get this workflow going it is not too
bad, depending on how hard this loop is of course.
<br>
Thanks,
<br>
<br>
danyang
<br>
<br>
</blockquote>
</blockquote>
</blockquote>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>