<div dir="ltr">Hi Matt,<div><br></div><div>I'm not sure if this is something related to the issue I filed <a href="https://gitlab.com/petsc/petsc/issues/423">https://gitlab.com/petsc/petsc/issues/423</a> where an unbalanced partitioning was observed even on a very tiny mesh, created in petsc/src/dm/impls/plex/examples/tutorials/ex7.c<br><br>Thanks,</div><div>Valeria</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Oct 18, 2019 at 6:21 PM <<a href="mailto:petsc-users-request@mcs.anl.gov">petsc-users-request@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Send petsc-users mailing list submissions to<br>
<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/petsc-users" rel="noreferrer" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/petsc-users</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:petsc-users-request@mcs.anl.gov" target="_blank">petsc-users-request@mcs.anl.gov</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:petsc-users-owner@mcs.anl.gov" target="_blank">petsc-users-owner@mcs.anl.gov</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of petsc-users digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: Strange Partition in PETSc 3.11 version on some<br>
computers (Matthew Knepley)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Fri, 18 Oct 2019 20:20:13 -0400<br>
From: Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
To: Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>><br>
Cc: "Smith, Barry F." <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>>, Mark Lohry<br>
<<a href="mailto:mlohry@gmail.com" target="_blank">mlohry@gmail.com</a>>, PETSc <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
Subject: Re: [petsc-users] Strange Partition in PETSc 3.11 version on<br>
some computers<br>
Message-ID:<br>
<CAMYG4Gk00hHAxspwm9CXPR8Nw+Tjh3QJN3rkwESRyc_dMkjG=<a href="mailto:A@mail.gmail.com" target="_blank">A@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
On Fri, Oct 18, 2019 at 5:53 PM Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>> wrote:<br>
<br>
> Hi All,<br>
><br>
> I am now able to reproduce the partition problem using a relatively small<br>
> mesh (attached). The mesh consists of 9087 nodes, 15656 prism cells. There<br>
> are 39 layers with 233 nodes for each layer. I have tested the partition<br>
> using PETSc as well as Gmsh 3.0.1.<br>
><br>
Great job finding a good test case. Can you send me that mesh?<br>
<br>
Thanks,<br>
<br>
Matt<br>
<br>
> Taking 4 partitions as an example, the partition from PETSc 3.9 and 3.10<br>
> are reasonable though not perfect, with total number of ghost nodes / total<br>
> number of nodes ratio 2754 / 9087.<br>
><br>
> The partition from PETSc 3.11, PETSc 3.12 and PETSc-dev look weird, with<br>
> total number of ghost nodes / total number of nodes: 12413 / 9087. The<br>
> nodes are not well connected for the same processor.<br>
><br>
> Note: the z axis is scaled by 25 for better visualization in paraview.<br>
><br>
><br>
> The partition from Gmsh-Metis is a bit different but still quite similar<br>
> to PETSc 3.9 and 3.10.<br>
><br>
> Finally, the partition using Gmsh-Chaco Multilevel-KL algorithm is the<br>
> best one, with total number of ghost nodes / total number of nodes: 741 /<br>
> 9087 . For most of my simulation cases with much larger meshes, PETSc 3.9<br>
> and 3.10 generate partition similar to the one below, which work pretty<br>
> well and the code can get very good speedup.<br>
><br>
> Thanks,<br>
><br>
> Danyang<br>
> On 2019-09-18 11:44 a.m., Danyang Su wrote:<br>
><br>
><br>
> On 2019-09-18 10:56 a.m., Smith, Barry F. via petsc-users wrote:<br>
><br>
><br>
> On Sep 18, 2019, at 12:25 PM, Mark Lohry via petsc-users<br>
> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
><br>
> Mark,<br>
><br>
><br>
> Mark,<br>
><br>
> Good point. This has been a big headache forever<br>
><br>
> Note that this has been "fixed" in the master version of PETSc and<br>
> will be in its next release. If you use --download-parmetis in the future<br>
> it will use the same random numbers on all machines and thus should produce<br>
> the same partitions on all machines.<br>
><br>
> I think that metis has aways used the same random numbers and all<br>
> machines and thus always produced the same results.<br>
><br>
> Barry<br>
><br>
> Good to know this. I will the same configuration that causes strange<br>
> partition problem to test the next version.<br>
><br>
> Thanks,<br>
><br>
> Danyang<br>
><br>
><br>
><br>
> The machine, compiler and MPI version should not matter.<br>
><br>
> I might have missed something earlier in the thread, but parmetis has a<br>
> dependency on the machine's glibc srand, and it can (and does) create<br>
> different partitions with different srand versions. The same mesh on the<br>
> same code on the same process count can and will give different partitions<br>
> (possibly bad ones) on different machines.<br>
><br>
> On Tue, Sep 17, 2019 at 1:05 PM Mark Adams via petsc-users<br>
> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
><br>
><br>
> On Tue, Sep 17, 2019 at 12:53 PM Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>><br>
> <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>> wrote:<br>
> Hi Mark,<br>
><br>
> Thanks for your follow-up.<br>
><br>
> The unstructured grid code has been verified and there is no problem in<br>
> the results. The convergence rate is also good. The 3D mesh is not good, it<br>
> is based on the original stratum which I haven't refined, but good for<br>
> initial test as it is relative small and the results obtained from this<br>
> mesh still makes sense.<br>
><br>
> The 2D meshes are just for testing purpose as I want to reproduce the<br>
> partition problem on a cluster using PETSc3.11.3 and Intel2019.<br>
> Unfortunately, I didn't find problem using this example.<br>
><br>
> The code has no problem in using different PETSc versions (PETSc V3.4 to<br>
> V3.11)<br>
><br>
> OK, it is the same code. I thought I saw something about your code<br>
> changing.<br>
><br>
> Just to be clear, v3.11 never gives you good partitions. It is not just a<br>
> problem on this Intel cluster.<br>
><br>
> The machine, compiler and MPI version should not matter.<br>
> and MPI distribution (MPICH, OpenMPI, IntelMPI), except for one<br>
> simulation case (the mesh I attached) on a cluster with PETSc3.11.3 and<br>
> Intel2019u4 due to the very different partition compared to PETSc3.9.3. Yet<br>
> the simulation results are the same except for the efficiency problem<br>
> because the strange partition results into much more communication (ghost<br>
> nodes).<br>
><br>
> I am still trying different compiler and mpi with PETSc3.11.3 on that<br>
> cluster to trace the problem. Will get back to you guys when there is<br>
> update.<br>
><br>
><br>
> This is very strange. You might want to use 'git bisect'. You set a good<br>
> and a bad SHA1 (we can give you this for 3.9 and 3.11 and the exact<br>
> commands). The git will go to a version in the middle. You then<br>
> reconfigure, remake, rebuild your code, run your test. Git will ask you, as<br>
> I recall, if the version is good or bad. Once you get this workflow going<br>
> it is not too bad, depending on how hard this loop is of course.<br>
> Thanks,<br>
><br>
> danyang<br>
><br>
><br>
<br>
-- <br>
What most experimenters take for granted before they begin their<br>
experiments is infinitely more interesting than any results to which their<br>
experiments lead.<br>
-- Norbert Wiener<br>
<br>
<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20191018/a51cd5a4/attachment.html" rel="noreferrer" target="_blank">http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20191018/a51cd5a4/attachment.html</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: basin-3d-dgr20000.png<br>
Type: image/png<br>
Size: 85113 bytes<br>
Desc: not available<br>
URL: <<a href="http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20191018/a51cd5a4/attachment.png" rel="noreferrer" target="_blank">http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20191018/a51cd5a4/attachment.png</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: gmsh-partition-metis.png<br>
Type: image/png<br>
Size: 61754 bytes<br>
Desc: not available<br>
URL: <<a href="http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20191018/a51cd5a4/attachment-0001.png" rel="noreferrer" target="_blank">http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20191018/a51cd5a4/attachment-0001.png</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: gmsh-partition-Chaco.png<br>
Type: image/png<br>
Size: 66392 bytes<br>
Desc: not available<br>
URL: <<a href="http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20191018/a51cd5a4/attachment-0002.png" rel="noreferrer" target="_blank">http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20191018/a51cd5a4/attachment-0002.png</a>><br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
petsc-users mailing list<br>
<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/petsc-users" rel="noreferrer" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/petsc-users</a><br>
<br>
<br>
------------------------------<br>
<br>
End of petsc-users Digest, Vol 130, Issue 75<br>
********************************************<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div><em><font color="#0000ce">Valeria Barra,</font></em></div>
<div><em><font color="#0000ce">Postdoctoral Research Associate,</font></em></div>
<div><em><font color="#0000ce">Department of Computer Science,</font></em></div>
<div><em><font color="#0000ce">University of Colorado at Boulder</font></em></div>
<div><a href="https://csel-web.cs.colorado.edu/~vaba3353" target="_blank">https://csel-web.cs.colorado.edu/~vaba3353</a><br></div></div></div></div></div></div>