<div dir="ltr">I don't think redundant should make any difference. The log summary data here does not seem to be the 20 sec/solve data (slow non-redundant solves).<div><br></div><div>Mark</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 29, 2015 at 10:45 AM, Michele Rosso <span dir="ltr"><<a href="mailto:mrosso@uci.edu" target="_blank">mrosso@uci.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div>
Hi Barry,<br>
<br>
I tried what you suggested:<br>
<br>
1) 5 levels of MG + defaults at the coarse level (PCREDUNDANT)<br>
2) 5 levels of MG + 2 levels of MG via DMDAREPART + defaults at the coarse level (PCREDUNDANT)<br>
<br>
I attached ksp_view and log_summary for both cases.<br>
The use of PCREDUNDAND halves the time for case 1 ( from ~ 20 sec per solve to ~ 10 sec per solve ), while it seems not having much effect on case 2.<br>
Any thoughts on this?<br>
<br>
Thanks,<br>
Michele<div><div class="h5"><br>
<br>
<br>
On Sat, 2015-07-25 at 22:18 -0500, Barry Smith wrote:
<blockquote type="CITE">
<pre> This dmdarepart business, which I am guessing is running PCMG on smaller sets of processes with a DMDDA on that smaller set of processes for a coarse problem is a fine idea but you should keep in mind the rule of thumb that that parallel iterative (and even more direct) solvers don't do well we there is roughly 10,000 or fewer degrees of freedom per processor. So you should definitely not be using SuperLU_DIST in parallel to solve a problem with 1048 degrees of freedom on 128 processes, just use PCREDUNDANT and its default (sequential) LU. That should be faster.
Barry
<font color="#737373">> On Jul 25, 2015, at 10:09 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:</font>
<font color="#737373">> </font>
<font color="#737373">> </font>
<font color="#737373">> Don't use </font>
<font color="#737373">> </font>
<font color="#737373">> -mg_coarse_pc_factor_mat_solver_package superlu_dist</font>
<font color="#737373">> -mg_coarse_pc_type lu</font>
<font color="#737373">> </font>
<font color="#737373">> with 8000+ processes and 1 degree of freedom per process SuperLU_DIST will be terrible. Just leave the defaults for this and send the -log_summary</font>
<font color="#737373">> </font>
<font color="#737373">> Barry</font>
<font color="#737373">> </font>
<font color="#737373">>> On Jul 24, 2015, at 2:44 PM, Michele Rosso <<a href="mailto:mrosso@uci.edu" target="_blank">mrosso@uci.edu</a>> wrote:</font>
<font color="#737373">>> </font>
<font color="#737373">>> Barry,</font>
<font color="#737373">>> </font>
<font color="#737373">>> I attached ksp_view and log_summary for two different setups:</font>
<font color="#737373">>> </font>
<font color="#737373">>> 1) Plain MG on 5 levels + LU at the coarse level (files ending in mg5)</font>
<font color="#737373">>> 2) Plain MG on 5 levels + custom PC + LU at the coarse level (files ending in mg7)</font>
<font color="#737373">>> </font>
<font color="#737373">>> The custom PC works on a subset of processes, thus allowing to use two more levels of MG, for a total of 7.</font>
<font color="#737373">>> Case 1) is extremely slow ( ~ 20 sec per solve ) and converges in 21 iterations.</font>
<font color="#737373">>> Case 2) is way faster ( ~ 0.25 sec per solve ) and converges in 29 iterations.</font>
<font color="#737373">>> </font>
<font color="#737373">>> Thanks for your help!</font>
<font color="#737373">>> </font>
<font color="#737373">>> Michele</font>
<font color="#737373">>> </font>
<font color="#737373">>> </font>
<font color="#737373">>> On Fri, 2015-07-24 at 13:56 -0500, Barry Smith wrote:</font>
<font color="#737373">>>> The coarse problem for the PCMG (geometric multigrid) is </font>
<font color="#737373">>>> </font>
<font color="#737373">>>> Mat Object: 8192 MPI processes</font>
<font color="#737373">>>> type: mpiaij</font>
<font color="#737373">>>> rows=8192, cols=8192</font>
<font color="#737373">>>> </font>
<font color="#737373">>>> then it tries to solve it with algebraic multigrid on 8192 processes (which is completely insane). A lot of the time is spent in setting up the algebraic multigrid (not surprisingly).</font>
<font color="#737373">>>> </font>
<font color="#737373">>>> 8192 is kind of small to parallelize. Please run the same code but with the default coarse grid problem instead of PCGAMG and send us the -log_summary again</font>
<font color="#737373">>>> </font>
<font color="#737373">>>> Barry</font>
<font color="#737373">>>> </font>
<font color="#737373">>>> </font>
<font color="#737373">>>>> On Jul 24, 2015, at 1:35 PM, Michele Rosso <<a href="mailto:mrosso@uci.edu" target="_blank">mrosso@uci.edu</a>> wrote:</font>
<font color="#737373">>>>> </font>
<font color="#737373">>>>> Hi Mark and Barry,</font>
<font color="#737373">>>>> </font>
<font color="#737373">>>>> I am sorry for my late reply: it was a busy week!</font>
<font color="#737373">>>>> I run a test case for a larger problem with as many levels (i.e. 5) of MG I could and GAMG as PC at the coarse level. I attached the output of info ( after grep for "gmag"), ksp_view and log_summary.</font>
<font color="#737373">>>>> The solve takes about 2 seconds on 8192 cores, which is way too much. The number of iterations to convergence is 24.</font>
<font color="#737373">>>>> I hope there is a way to speed it up.</font>
<font color="#737373">>>>> </font>
<font color="#737373">>>>> Thanks,</font>
<font color="#737373">>>>> Michele</font>
<font color="#737373">>>>> </font>
<font color="#737373">>>>> </font>
<font color="#737373">>>>> On Fri, 2015-07-17 at 09:38 -0400, Mark Adams wrote:</font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> On Thu, Jul 16, 2015 at 8:18 PM, Michele Rosso <<a href="mailto:mrosso@uci.edu" target="_blank">mrosso@uci.edu</a>> wrote:</font>
<font color="#737373">>>>>> Barry,</font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> thank you very much for the detailed answer. I tried what you suggested and it works.</font>
<font color="#737373">>>>>> So far I tried on a small system but the final goal is to use it for very large runs. How does PCGAMG compares to PCMG as far as performances and scalability are concerned?</font>
<font color="#737373">>>>>> Also, could you help me to tune the GAMG part ( my current setup is in the attached ksp_view.txt file )? </font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> I am going to add this to the document today but you can run with -info. This is very noisy so you might want to do the next step at run time. Then grep on GAMG. This will be about 20 lines. Send that to us and we can go from there.</font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> Mark</font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> I also tried to use superlu_dist for the LU decomposition on mg_coarse_mg_sub_</font>
<font color="#737373">>>>>> -mg_coarse_mg_coarse_sub_pc_type lu</font>
<font color="#737373">>>>>> -mg_coarse_mg_coarse_sub_pc_factor_mat_solver_package superlu_dist</font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> but I got an error:</font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> ****** Error in MC64A/AD. INFO(1) = -2 </font>
<font color="#737373">>>>>> ****** Error in MC64A/AD. INFO(1) = -2</font>
<font color="#737373">>>>>> ****** Error in MC64A/AD. INFO(1) = -2</font>
<font color="#737373">>>>>> ****** Error in MC64A/AD. INFO(1) = -2</font>
<font color="#737373">>>>>> ****** Error in MC64A/AD. INFO(1) = -2</font>
<font color="#737373">>>>>> ****** Error in MC64A/AD. INFO(1) = -2</font>
<font color="#737373">>>>>> ****** Error in MC64A/AD. INFO(1) = -2</font>
<font color="#737373">>>>>> symbfact() error returns 0</font>
<font color="#737373">>>>>> symbfact() error returns 0</font>
<font color="#737373">>>>>> symbfact() error returns 0</font>
<font color="#737373">>>>>> symbfact() error returns 0</font>
<font color="#737373">>>>>> symbfact() error returns 0</font>
<font color="#737373">>>>>> symbfact() error returns 0</font>
<font color="#737373">>>>>> symbfact() error returns 0</font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> Thank you,</font>
<font color="#737373">>>>>> Michele</font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> On Thu, 2015-07-16 at 18:07 -0500, Barry Smith wrote:</font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>>> On Jul 16, 2015, at 5:42 PM, Michele Rosso <<a href="mailto:mrosso@uci.edu" target="_blank">mrosso@uci.edu</a>> wrote:</font>
<font color="#737373">>>>>>>> </font>
<font color="#737373">>>>>>>> Barry,</font>
<font color="#737373">>>>>>>> </font>
<font color="#737373">>>>>>>> thanks for your reply. So if I want it fixed, I will have to use the master branch, correct?</font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>> Yes, or edit mg.c and remove the offending lines of code (easy enough). </font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>>> </font>
<font color="#737373">>>>>>>> On a side note, what I am trying to achieve is to be able to use how many levels of MG I want, despite the limitation imposed by the local number of grid nodes.</font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>> I assume you are talking about with DMDA? There is no generic limitation for PETSc's multigrid, it is only with the way the DMDA code figures out the interpolation that causes a restriction.</font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>>> So far I am using a borrowed code that implements a PC that creates a sub communicator and perform MG on it.</font>
<font color="#737373">>>>>>>> While reading the documentation I found out that PCMGSetLevels takes in an optional array of communicators. How does this work?</font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>> It doesn't work. It was an idea that never got pursued.</font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>>> Can I can simply define my matrix and rhs on the fine grid as I would do normally ( I do not use kspsetoperators and kspsetrhs ) and KSP would take care of it by using the correct communicator for each level?</font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>> No.</font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>> You can use the PCMG geometric multigrid with DMDA for as many levels as it works and then use PCGAMG as the coarse grid solver. PCGAMG automatically uses fewer processes for the coarse level matrices and vectors. You could do this all from the command line without writing code. </font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>> For example if your code uses a DMDA and calls KSPSetDM() use for example -da_refine 3 -pc_type mg -pc_mg_galerkin -mg_coarse_pc_type gamg -ksp_view </font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>> Barry</font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>>> </font>
<font color="#737373">>>>>>>> Thanks,</font>
<font color="#737373">>>>>>>> Michele</font>
<font color="#737373">>>>>>>> </font>
<font color="#737373">>>>>>>> </font>
<font color="#737373">>>>>>>> </font>
<font color="#737373">>>>>>>> </font>
<font color="#737373">>>>>>>> On Thu, 2015-07-16 at 17:30 -0500, Barry Smith wrote:</font>
<font color="#737373">>>>>>>>> Michel,</font>
<font color="#737373">>>>>>>>> </font>
<font color="#737373">>>>>>>>> This is a very annoying feature that has been fixed in master </font>
<font color="#737373">>>>>>>>> <a href="http://www.mcs.anl.gov/petsc/developers/index.html" target="_blank">http://www.mcs.anl.gov/petsc/developers/index.html</a></font>
<font color="#737373">>>>>>>>> I would like to have changed it in maint but Jed would have a shit-fit :-) since it changes behavior.</font>
<font color="#737373">>>>>>>>> </font>
<font color="#737373">>>>>>>>> Barry</font>
<font color="#737373">>>>>>>>> </font>
<font color="#737373">>>>>>>>> </font>
<font color="#737373">>>>>>>>>> On Jul 16, 2015, at 4:53 PM, Michele Rosso <<a href="mailto:mrosso@uci.edu" target="_blank">mrosso@uci.edu</a>> wrote:</font>
<font color="#737373">>>>>>>>>> </font>
<font color="#737373">>>>>>>>>> Hi,</font>
<font color="#737373">>>>>>>>>> </font>
<font color="#737373">>>>>>>>>> I am performing a series of solves inside a loop. The matrix for each solve changes but not enough to justify a rebuilt of the PC at each solve.</font>
<font color="#737373">>>>>>>>>> Therefore I am using KSPSetReusePreconditioner to avoid rebuilding unless necessary. The solver is CG + MG with a custom PC at the coarse level.</font>
<font color="#737373">>>>>>>>>> If KSP is not updated each time, everything works as it is supposed to. </font>
<font color="#737373">>>>>>>>>> When instead I allow the default PETSc behavior, i.e. updating PC every time the matrix changes, the coarse level KSP , initially set to PREONLY, is changed into GMRES </font>
<font color="#737373">>>>>>>>>> after the first solve. I am not sure where the problem lies (my PC or PETSc), so I would like to have your opinion on this.</font>
<font color="#737373">>>>>>>>>> I attached the ksp_view for the 2 successive solve and the options stack.</font>
<font color="#737373">>>>>>>>>> </font>
<font color="#737373">>>>>>>>>> Thanks for your help,</font>
<font color="#737373">>>>>>>>>> Michel</font>
<font color="#737373">>>>>>>>>> </font>
<font color="#737373">>>>>>>>>> </font>
<font color="#737373">>>>>>>>>> </font>
<font color="#737373">>>>>>>>>> <ksp_view.txt><petsc_options.txt></font>
<font color="#737373">>>>>>>>> </font>
<font color="#737373">>>>>>>>> </font>
<font color="#737373">>>>>>>>> </font>
<font color="#737373">>>>>>>> </font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>>> </font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>>> </font>
<font color="#737373">>>>> </font>
<font color="#737373">>>>> <info.txt><ksp_view.txt><log_gamg.txt></font>
<font color="#737373">>>> </font>
<font color="#737373">>>> </font>
<font color="#737373">>>> </font>
<font color="#737373">>> </font>
<font color="#737373">>> <ksp_view_mg5.txt><ksp_view_mg7.txt><log_mg5.txt><log_mg7.txt></font>
<font color="#737373">> </font>
</pre>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br></div>