<div dir="ltr"><div dir="ltr"><div>Hello Mark,</div><div><br></div><div><div>Yes, there are 5 KSP calls within a time-step (3 for the solution of momentum equation + 2 for the solution of pressure), this is the classical non iterative PISO by Issa ( the exact sequence of operations is : solve momentum implicitly, solve pressure-correction, momentum explicitly, pressure correction). The pressure correction equation ,which is something similar to a Poisson equation for incompressible flows, is the one that determines the overall performance in my code such as in the others. Usually, when the pressure is being solved for the second time, the solution is faster since there is a better input guess and, as in my case, the preconditioner is not recomputed again. </div><div><br></div><div>Have you got some advices for the multigrid configuration in this scenario, which are not the default one, in order to increase performances?</div><div><br></div><div>I do not know if this may impact drastically the performance, but I am running on a E4 workstation with 16 Intel's Xeon processors (2.3GH/12MB cache) and 128GB of RAM .</div><div><br></div><div>Thank you very much for your helpful comments,</div></div><div><br></div><div><br></div><div>Edoardo</div><div><div dir="ltr" class="gmail_signature"><div dir="ltr">------<div><br></div><div>Edoardo Alinovi, Ph.D.</div><div><br></div><div>DICCA, Scuola Politecnica<br>Universita' di Genova<br>1, via Montallegro<br>16145 Genova, Italy<br></div><div><br></div><div>email: <a href="mailto:edoardo.alinovi@dicca.unige.it" target="_blank">edoardo.alinovi@dicca.unige.it</a></div><div>Tel: +39 010 353 2540<br></div><div><br></div><div><br></div></div></div></div><br><br><div class="gmail_quote"><div dir="ltr">Il giorno mer 7 nov 2018 alle ore 17:59 Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">please respond to petsc-users.<div><br></div><div>You are doing 5 solves here in 14 seconds. You seem to be saying that the two pressure solves are taking all of this time. I don't know why the two solves are different.</div><div><br></div><div>You seem to be saying that OpenFOAM solves the problem in 10 seconds and PETSc solves it in 14 seconds. Is that correct? Hypre seems to be running fine.<br><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Nov 7, 2018 at 11:24 AM Edoardo alinovi <<a href="mailto:edoardo.alinovi@gmail.com" target="_blank">edoardo.alinovi@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thanks a lot Mark for your kind replay. The solver is mine and I use PETSc for the solution of momentum and pressure. The first is solved very fast by a standard bcgs + bjacobi, but the pressure is the source of all evils and, unfortunately, I am pretty sure that almost all the time within the time-step is needed by KSP to solve the pressure (see log attached). I have verified this also putting a couple of mpi_wtime around the kspsolve call. The pressure is solved 2 times (1 prediction + 1 correction), the prediction takes around 11s , the correction around 4s (here I am avoiding to recompute the preconditioner), all the rest of the code (flux assembling + mometum solution + others) around 1s. Openfoam does the same procedure with the same tolerance in 10s using its gamg version (50 it to converge). The number of iteration required to solve the pressure with hypre are 12. Gamg performs similarly to hypre in terms of speed, but with 50 iterations to converge. Am I missing something in the setup in your opinion?</div><div><br></div><div>thanks a lot,</div><div><br></div><div>Edo<br> </div><div><br><div><div><div dir="ltr" class="gmail-m_3433251665540188435m_4362404713328493493m_5919425827911290490gmail_signature"><div dir="ltr">------<div><br></div><div>Edoardo Alinovi, Ph.D.</div><div><br></div><div>DICCA, Scuola Politecnica<br>Universita' di Genova<br>1, via Montallegro<br>16145 Genova, Italy<br></div><div><br></div><div>email: <a href="mailto:edoardo.alinovi@dicca.unige.it" target="_blank">edoardo.alinovi@dicca.unige.it</a></div><div>Tel: +39 010 353 2540<br></div><div><br></div><div><br></div></div></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr">Il giorno mer 7 nov 2018 alle ore 16:50 Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">You can try -pc_type gamg, but hypre is a pretty good solver for the Laplacian. If hypre is just a little faster than LU on a 3D problem (that takes 10 seconds to solve) then AMG is not doing well. I would expect that AMG is taking a lot of iterations (eg, >> 10). You can check that with -ksp_monitor.<div><br></div><div>The PISO algorithm is a multistage algorithm with a pressure correction in it. It also has a solve for the velocity, from what I can tell. Are you building PISO yourself and using PETSc just for the pressure correction? Are you sure the time is spent in this solver? You can use -log_view to see performance numbers and look for KSPSolve to see how much time is spent in the PETSc solver.</div><div><br></div><div>Mark</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Nov 7, 2018 at 10:26 AM Zhang, Hong via petsc-maint <<a href="mailto:petsc-maint@mcs.anl.gov" target="_blank">petsc-maint@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr">Edoardo:<br>
</div>
<div>Forwarding your request to petsc-maint where you can get fast and expert advise. I do not have suggestion for your application, but someone in our team likely will make suggestion.</div>
<div>Hong</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">Hello Hong,
<div><br>
</div>
<div>Well, using -sub_pc_type lu it super slow. I am desperately triying to enhance performaces of my code (CFD, finite volume, PISO alghoritm), in particular I have a strong bottleneck in the solution of pressure correction equation which takes almost the
90% of computational time. Using multigrid as preconditoner (hypre with default options) is slighlty better, but comparing the results against the multigrid used in openFOAM, my code is losing 10s/iteration which a huge amount of time. Now, since that all
the time is employed by KSPSolve, I feel a bit powerless. Do you have any helpful advice? </div>
<div><br>
</div>
<div>Thank you very much! </div>
<div>------
<div>
<div dir="ltr" class="gmail-m_3433251665540188435m_4362404713328493493m_5919425827911290490m_-6650170222397477307m_-4486351366788598251m_-8531426731262477850gmail_signature">
<div dir="ltr">
<div><br>
</div>
<div>Edoardo Alinovi, Ph.D.</div>
<div><br>
</div>
<div>DICCA, Scuola Politecnica<br>
Universita' di Genova<br>
1, via Montallegro<br>
16145 Genova, Italy<br>
</div>
<div><br>
</div>
<div>email: <a href="mailto:edoardo.alinovi@dicca.unige.it" target="_blank">edoardo.alinovi@dicca.unige.it</a></div>
<div>Tel: +39 010 353 2540<br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">Il giorno mar 6 nov 2018 alle ore 17:15 Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>> ha scritto:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr">Edoardo:<br>
</div>
<div>Interesting. I thought it would not affect performance much. What happens if you use -sub_pc_type lu'?</div>
<div>Hong</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">Dear Hong and Matt,
<div><br>
</div>
<div>thank you for your kind replay. I have just tested your suggestions and applied " -sub_pc_type ilu -sub_pc_factor_mat_ordering_type nd/rcm" and, in both cases, I have found <span style="color:rgb(0,0,0)">a deterioration of performances with respect to
doing nothing (thus just putting default PCBJACOBI). Is it normal? However, I guess this is very problem dependent.</span></div>
<div>
<div>
<div dir="ltr" class="gmail-m_3433251665540188435m_4362404713328493493m_5919425827911290490m_-6650170222397477307m_-4486351366788598251m_-8531426731262477850m_5870894216385883731m_-5311843450511196384gmail_signature">
<div dir="ltr">------
<div><br>
</div>
<div>Edoardo Alinovi, Ph.D.</div>
<div><br>
</div>
<div>DICCA, Scuola Politecnica<br>
Universita' di Genova<br>
1, via Montallegro<br>
16145 Genova, Italy<br>
</div>
<div><br>
</div>
<div>email: <a href="mailto:edoardo.alinovi@dicca.unige.it" target="_blank">edoardo.alinovi@dicca.unige.it</a></div>
<div>Tel: +39 010 353 2540<br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">Il giorno mar 6 nov 2018 alle ore 16:04 Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>> ha scritto:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr">Edoardo:<br>
</div>
<div>You can test runtime option '-sub_pc_factor_mat_ordering_type' and use '-log_view' to get performance on different orderings, e.g.,petsc/src/ksp/ksp/examples/tutorials/ex2.c:</div>
<div>mpiexec -n 2 ./ex2 -ksp_view -sub_pc_type ilu -sub_pc_factor_mat_ordering_type nd<br>
</div>
<div><br>
</div>
<div>I do not think the ordering inside block for ilu would affect performance much. Let us know what you will get.</div>
<div>Hong</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div>
<div>Dear users, </div>
<div><br>
</div>
<div>I have a question about the correct use of the option "PCFactorSetMatOrderingType" in PETSc. </div>
<div><br>
</div>
<div>I am solving a problem with 2.5M of unknowns distributed along 16 processors and I am using the block jacobi preconditioner and MPIAIJ matrix format. I cannot figure out if the above option can be useful or not in decreasing the computational time. Any
suggestion or tips?</div>
<div><br>
</div>
<div>Thank you very much for the kind help </div>
</div>
<br clear="all">
<div>
<div dir="ltr" class="gmail-m_3433251665540188435m_4362404713328493493m_5919425827911290490m_-6650170222397477307m_-4486351366788598251m_-8531426731262477850m_5870894216385883731m_-5311843450511196384m_-4940824767034093214gmail-m_6398222335066066296gmail_signature">
<div dir="ltr">------
<div><br>
</div>
<div>Edoardo Alinovi, Ph.D.</div>
<div><br>
</div>
<div>DICCA, Scuola Politecnica<br>
Universita' di Genova<br>
1, via Montallegro<br>
16145 Genova, Italy<br>
</div>
<div><br>
</div>
<div>email: <a href="mailto:edoardo.alinovi@dicca.unige.it" target="_blank">edoardo.alinovi@dicca.unige.it</a></div>
<div>Tel: +39 010 353 2540<br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div></div></div>