<div dir="ltr">I am not that familiar with hypre's options. AMG is complicated and I barely keep my options straight.<div><br></div><div>OpenFOAM seems to have highly specialized solvers so being with 50% of them is decent.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Nov 8, 2018 at 12:03 PM Edoardo alinovi <<a href="mailto:edoardo.alinovi@gmail.com">edoardo.alinovi@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">Yes, it is like you are saying. This is mostly due to the time employed by ksp to solve the pressure equation. However, I have worked a lot on the problem and I have found out that the default configuration is far to be the optimal one, at least in this case. <div dir="auto"><br></div><div dir="auto">Actually my cpu time is decreased by more than twice with respect to the default configuration. I put here the changes, maybe they can be usefull to other users in the future:<div dir="auto"><div dir="auto"><br></div><div dir="auto">-pc_hypre_boomeramg_no_CF</div><div dir="auto"><span style="font-family:sans-serif">-pc_hypre_boomeramg_agg_nl 1</span><br></div><div dir="auto"><span style="font-family:sans-serif">pc_hypre_boomeramg_coarsen_type HMIS</span><span style="font-family:sans-serif"><br></span></div><div dir="auto"><span style="font-family:sans-serif">-pc_hypre_boomeramg_interp_type FF1</span></div><div dir="auto"><span style="font-family:sans-serif"><br></span></div><div dir="auto"><span style="font-family:sans-serif">The last two seem to be essential in enhancing performances. </span></div><div dir="auto"><span style="font-family:sans-serif"><br></span></div><div dir="auto"><font face="sans-serif">Do you know other configurations that worth to test? </font><span style="font-family:sans-serif">Also, I would like to know if a list of command options for hypre is available somewhere. Looking at hypre's doc there are a lot of options, but I do not exactly know which one is available in petsc and their name.</span></div><div dir="auto"><span style="font-family:sans-serif"><br></span></div><div dir="auto"><span style="font-family:sans-serif">Thank you very much for the kind support and sorry for the long emal.!</span></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">Il giorno gio 8 nov 2018, 17:32 Mark Adams <<a href="mailto:mfadams@lbl.gov" rel="noreferrer" target="_blank">mfadams@lbl.gov</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">To repeat: <div><br></div><div>You seem to be saying that OpenFOAM solves the problem in 10 seconds and PETSc solves it in 14 seconds. Is that correct?<br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Nov 8, 2018 at 3:42 AM Edoardo alinovi via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" rel="noreferrer noreferrer" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>Hello Mark,</div><div><br></div><div><div>Yes, there are 5 KSP calls within a time-step (3 for the solution of momentum equation + 2 for the solution of pressure), this is the classical non iterative PISO by Issa ( the exact sequence of operations is : solve momentum implicitly, solve pressure-correction, momentum explicitly, pressure correction). The pressure correction equation ,which is something similar to a Poisson equation for incompressible flows, is the one that determines the overall performance in my code such as in the others. Usually, when the pressure is being solved for the second time, the solution is faster since there is a better input guess and, as in my case, the preconditioner is not recomputed again. </div><div><br></div><div>Have you got some advices for the multigrid configuration in this scenario, which are not the default one, in order to increase performances?</div><div><br></div><div>I do not know if this may impact drastically the performance, but I am running on a E4 workstation with 16 Intel's Xeon processors (2.3GH/12MB cache) and 128GB of RAM .</div><div><br></div><div>Thank you very much for your helpful comments,</div></div><div><br></div><div><br></div><div>Edoardo</div><div><div dir="ltr" class="m_-4063899835513259760m_6431051228435980834m_-840276202190525904m_9132936605349306105gmail_signature"><div dir="ltr">------<div><br></div><div>Edoardo Alinovi, Ph.D.</div><div><br></div><div>DICCA, Scuola Politecnica<br>Universita' di Genova<br>1, via Montallegro<br>16145 Genova, Italy<br></div><div><br></div><div>email: <a href="mailto:edoardo.alinovi@dicca.unige.it" rel="noreferrer noreferrer" target="_blank">edoardo.alinovi@dicca.unige.it</a></div><div>Tel: +39 010 353 2540<br></div><div><br></div><div><br></div></div></div></div><br><br><div class="gmail_quote"><div dir="ltr">Il giorno mer 7 nov 2018 alle ore 17:59 Mark Adams <<a href="mailto:mfadams@lbl.gov" rel="noreferrer noreferrer" target="_blank">mfadams@lbl.gov</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">please respond to petsc-users.<div><br></div><div>You are doing 5 solves here in 14 seconds. You seem to be saying that the two pressure solves are taking all of this time. I don't know why the two solves are different.</div><div><br></div><div>You seem to be saying that OpenFOAM solves the problem in 10 seconds and PETSc solves it in 14 seconds. Is that correct? Hypre seems to be running fine.<br><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Nov 7, 2018 at 11:24 AM Edoardo alinovi <<a href="mailto:edoardo.alinovi@gmail.com" rel="noreferrer noreferrer" target="_blank">edoardo.alinovi@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thanks a lot Mark for your kind replay. The solver is mine and I use PETSc for the solution of momentum and pressure. The first is solved very fast by a standard bcgs + bjacobi, but the pressure is the source of all evils and, unfortunately, I am pretty sure that almost all the time within the time-step is needed by KSP to solve the pressure (see log attached). I have verified this also putting a couple of mpi_wtime around the kspsolve call. The pressure is solved 2 times (1 prediction + 1 correction), the prediction takes around 11s , the correction around 4s (here I am avoiding to recompute the preconditioner), all the rest of the code (flux assembling + mometum solution + others) around 1s. Openfoam does the same procedure with the same tolerance in 10s using its gamg version (50 it to converge). The number of iteration required to solve the pressure with hypre are 12. Gamg performs similarly to hypre in terms of speed, but with 50 iterations to converge. Am I missing something in the setup in your opinion?</div><div><br></div><div>thanks a lot,</div><div><br></div><div>Edo<br> </div><div><br><div><div><div dir="ltr" class="m_-4063899835513259760m_6431051228435980834m_-840276202190525904m_9132936605349306105gmail-m_3433251665540188435m_4362404713328493493m_5919425827911290490gmail_signature"><div dir="ltr">------<div><br></div><div>Edoardo Alinovi, Ph.D.</div><div><br></div><div>DICCA, Scuola Politecnica<br>Universita' di Genova<br>1, via Montallegro<br>16145 Genova, Italy<br></div><div><br></div><div>email: <a href="mailto:edoardo.alinovi@dicca.unige.it" rel="noreferrer noreferrer" target="_blank">edoardo.alinovi@dicca.unige.it</a></div><div>Tel: +39 010 353 2540<br></div><div><br></div><div><br></div></div></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr">Il giorno mer 7 nov 2018 alle ore 16:50 Mark Adams <<a href="mailto:mfadams@lbl.gov" rel="noreferrer noreferrer" target="_blank">mfadams@lbl.gov</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">You can try -pc_type gamg, but hypre is a pretty good solver for the Laplacian. If hypre is just a little faster than LU on a 3D problem (that takes 10 seconds to solve) then AMG is not doing well. I would expect that AMG is taking a lot of iterations (eg, >> 10). You can check that with -ksp_monitor.<div><br></div><div>The PISO algorithm is a multistage algorithm with a pressure correction in it. It also has a solve for the velocity, from what I can tell. Are you building PISO yourself and using PETSc just for the pressure correction? Are you sure the time is spent in this solver? You can use -log_view to see performance numbers and look for KSPSolve to see how much time is spent in the PETSc solver.</div><div><br></div><div>Mark</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Nov 7, 2018 at 10:26 AM Zhang, Hong via petsc-maint <<a href="mailto:petsc-maint@mcs.anl.gov" rel="noreferrer noreferrer" target="_blank">petsc-maint@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr">Edoardo:<br>
</div>
<div>Forwarding your request to petsc-maint where you can get fast and expert advise. I do not have suggestion for your application, but someone in our team likely will make suggestion.</div>
<div>Hong</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">Hello Hong,
<div><br>
</div>
<div>Well, using -sub_pc_type lu it super slow. I am desperately triying to enhance performaces of my code (CFD, finite volume, PISO alghoritm), in particular I have a strong bottleneck in the solution of pressure correction equation which takes almost the
90% of computational time. Using multigrid as preconditoner (hypre with default options) is slighlty better, but comparing the results against the multigrid used in openFOAM, my code is losing 10s/iteration which a huge amount of time. Now, since that all
the time is employed by KSPSolve, I feel a bit powerless. Do you have any helpful advice? </div>
<div><br>
</div>
<div>Thank you very much! </div>
<div>------
<div>
<div dir="ltr" class="m_-4063899835513259760m_6431051228435980834m_-840276202190525904m_9132936605349306105gmail-m_3433251665540188435m_4362404713328493493m_5919425827911290490m_-6650170222397477307m_-4486351366788598251m_-8531426731262477850gmail_signature">
<div dir="ltr">
<div><br>
</div>
<div>Edoardo Alinovi, Ph.D.</div>
<div><br>
</div>
<div>DICCA, Scuola Politecnica<br>
Universita' di Genova<br>
1, via Montallegro<br>
16145 Genova, Italy<br>
</div>
<div><br>
</div>
<div>email: <a href="mailto:edoardo.alinovi@dicca.unige.it" rel="noreferrer noreferrer" target="_blank">edoardo.alinovi@dicca.unige.it</a></div>
<div>Tel: +39 010 353 2540<br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">Il giorno mar 6 nov 2018 alle ore 17:15 Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" rel="noreferrer noreferrer" target="_blank">hzhang@mcs.anl.gov</a>> ha scritto:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr">Edoardo:<br>
</div>
<div>Interesting. I thought it would not affect performance much. What happens if you use -sub_pc_type lu'?</div>
<div>Hong</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">Dear Hong and Matt,
<div><br>
</div>
<div>thank you for your kind replay. I have just tested your suggestions and applied " -sub_pc_type ilu -sub_pc_factor_mat_ordering_type nd/rcm" and, in both cases, I have found <span style="color:rgb(0,0,0)">a deterioration of performances with respect to
doing nothing (thus just putting default PCBJACOBI). Is it normal? However, I guess this is very problem dependent.</span></div>
<div>
<div>
<div dir="ltr" class="m_-4063899835513259760m_6431051228435980834m_-840276202190525904m_9132936605349306105gmail-m_3433251665540188435m_4362404713328493493m_5919425827911290490m_-6650170222397477307m_-4486351366788598251m_-8531426731262477850m_5870894216385883731m_-5311843450511196384gmail_signature">
<div dir="ltr">------
<div><br>
</div>
<div>Edoardo Alinovi, Ph.D.</div>
<div><br>
</div>
<div>DICCA, Scuola Politecnica<br>
Universita' di Genova<br>
1, via Montallegro<br>
16145 Genova, Italy<br>
</div>
<div><br>
</div>
<div>email: <a href="mailto:edoardo.alinovi@dicca.unige.it" rel="noreferrer noreferrer" target="_blank">edoardo.alinovi@dicca.unige.it</a></div>
<div>Tel: +39 010 353 2540<br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">Il giorno mar 6 nov 2018 alle ore 16:04 Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" rel="noreferrer noreferrer" target="_blank">hzhang@mcs.anl.gov</a>> ha scritto:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr">Edoardo:<br>
</div>
<div>You can test runtime option '-sub_pc_factor_mat_ordering_type' and use '-log_view' to get performance on different orderings, e.g.,petsc/src/ksp/ksp/examples/tutorials/ex2.c:</div>
<div>mpiexec -n 2 ./ex2 -ksp_view -sub_pc_type ilu -sub_pc_factor_mat_ordering_type nd<br>
</div>
<div><br>
</div>
<div>I do not think the ordering inside block for ilu would affect performance much. Let us know what you will get.</div>
<div>Hong</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div>
<div>Dear users, </div>
<div><br>
</div>
<div>I have a question about the correct use of the option "PCFactorSetMatOrderingType" in PETSc. </div>
<div><br>
</div>
<div>I am solving a problem with 2.5M of unknowns distributed along 16 processors and I am using the block jacobi preconditioner and MPIAIJ matrix format. I cannot figure out if the above option can be useful or not in decreasing the computational time. Any
suggestion or tips?</div>
<div><br>
</div>
<div>Thank you very much for the kind help </div>
</div>
<br clear="all">
<div>
<div dir="ltr" class="m_-4063899835513259760m_6431051228435980834m_-840276202190525904m_9132936605349306105gmail-m_3433251665540188435m_4362404713328493493m_5919425827911290490m_-6650170222397477307m_-4486351366788598251m_-8531426731262477850m_5870894216385883731m_-5311843450511196384m_-4940824767034093214gmail-m_6398222335066066296gmail_signature">
<div dir="ltr">------
<div><br>
</div>
<div>Edoardo Alinovi, Ph.D.</div>
<div><br>
</div>
<div>DICCA, Scuola Politecnica<br>
Universita' di Genova<br>
1, via Montallegro<br>
16145 Genova, Italy<br>
</div>
<div><br>
</div>
<div>email: <a href="mailto:edoardo.alinovi@dicca.unige.it" rel="noreferrer noreferrer" target="_blank">edoardo.alinovi@dicca.unige.it</a></div>
<div>Tel: +39 010 353 2540<br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>