[petsc-users] on the performance of MPI PETSc

[Sun, Hui]

Thank you Barry, I will check on that. However, I tried something else, and now have more questions on the multigrid solver in PETSc. 

If I run ex34 using the command:
./ex34 -pc_type mg -pc_mg_type full -ksp_type fgmres -ksp_monitor_short -pc_mg_levels 3 -mg_coarse_pc_factor_shift_type nonzero

I get the following output:
  0 KSP Residual norm 0.0289149 
  1 KSP Residual norm 0.0186085 
  2 KSP Residual norm 0.000732811 
  3 KSP Residual norm 1.07394e-05 
  4 KSP Residual norm 1.01043e-06 
  5 KSP Residual norm 5.62122e-08 
Residual norm 5.62122e-08
Error norm 0.000200841
Error norm 5.18298e-05
Error norm 4.90288e-08
Time cost: 0.068264 14.4999 0.0826752

If I run ex34 using the command:
./ex34 -pc_type none -ksp_type gmres -ksp_monitor_short

I get the following output:
  0 KSP Residual norm 0.0289149 
  1 KSP Residual norm < 1.e-11
Residual norm 9.14804e-15
Error norm 0.00020064
Error norm 5.18301e-05
Error norm 4.90288e-08
Time cost: 1.60657 4.67008 0.0784049

>From the output, it seems that solving Poisson without a preconditioner is actually faster than using multigrid as a preconditioner. I think multigrid should do better than that. How should I look more into this issue? What are the options you would recommend me to try in order to raise up the efficiency of multigrid? 

Another question is: Petsc uses multigrid as a preconditioner, how do I specify the option so that it becomes a solver? Is it by doing:
./ex34 -pc_type mg -pc_mg_type full -ksp_type richardson -ksp_monitor_short -pc_mg_levels 3 -mg_coarse_pc_factor_shift_type nonzero

Best, 
Hui


________________________________________
From: Barry Smith [bsmith at mcs.anl.gov]
Sent: Thursday, June 04, 2015 11:56 AM
To: Sun, Hui
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] on the performance of MPI PETSc

> On Jun 4, 2015, at 1:24 PM, Sun, Hui <hus003 at ucsd.edu> wrote:
>
> Hello,
>
> I'm testing ex34.c under the examples of KSP. It's a multigrid 3D poisson solver.
>
> For 64^3 mesh, the time cost is 1s for 1 node with 12 cores; for 128^3 mesh, the time cost is 13s for 1 node with 12 cores, and the same for 2 nodes with 6 cores. For 256^3 mesh, I use 2 nodes with 12 cores, and time cost goes up to 726s. This doesn't seem right for I'm expecting O(N log(N)). I think it could be the memory bandwidth is not sufficient, and I need to do the bind-to-socket stuff.
>
> But I'm wondering what is the typical time cost for a 256^3 mesh, and then a 512^3 mesh? Please give me a rough idea. Thank you.

   There is no way we can answer that. What we can say is that given the numbers you have for 64 and 128 meshes, on an appropriate machine you should get much better numbers than 726 seconds for a 256 mesh. You need to first run streams on your machine, http://www.mcs.anl.gov/petsc/documentation/faq.html#computers check up on the binding business and if you still get poor results prepare a report on what you did and send it to us.

   Barry

>
> Best,
> Hui