[petsc-dev] [issue1595] Issues of limited number of MPI communicators when having many instances of hypre boomerAMG with Moose

[Yang, Ulrike Meier]

The sub-communicator on the coarsest grid is generally not of size 1.
You can avoid using this version with the sub communicator by setting the smoother/solver on the coarse grid to a different one using
HYPRE_BoomerAMGSetCycleRelaxtype( <solver>, relax_type, 3);
If you set relax_type to 19, it will use the old version for Gaussian elimination.
You could also set it to a smoother, e.g. 6, which would be 1 sweep of SGS.
Ulrike

-----Original Message-----
From: Smith, Barry F. <bsmith at mcs.anl.gov> 
Sent: Tuesday, April 3, 2018 12:56 PM
To: hypre-support <hypre-support at llnl.gov>
Cc: fande.kong at inl.gov; friedmud at gmail.com; Li, Ruipeng <li50 at llnl.gov>; Osei-Kuffuor, Daniel <oseikuffuor1 at llnl.gov>; petsc-dev at mcs.anl.gov; Schroder, Jacob B. <schroder2 at llnl.gov>; tzanio at llnl.gov; umyang at llnl.gov; Wang, Lu <wang84 at llnl.gov>
Subject: Re: [issue1595] Issues of limited number of MPI communicators when having many instances of hypre boomerAMG with Moose



> On Apr 3, 2018, at 1:46 PM, Rob Falgout hypre Tracker <hypre-support at llnl.gov> wrote:
> 
> 
> Rob Falgout <rfalgout at llnl.gov> added the comment:
> 
> Hi Barry,
> 
> It looks like the only time we call MPI_Comm_create is to build a communicator for the coarsest grid solve using Gaussian elimination.  There are probably alternatives that do not require creating a sub-communicator.

    When the sub communicator is of size 1 you can use MPI_COMM_SELF instead of creating a new communicator each time.

>  Ulrike or someone else more familiar with the code should comment.
> 
> I don't see a need to do a Comm_dup() before calling hypre.

   If I have 1000 hypre solvers on the same communicator how do I know that hypre won't send messages between the different solvers and hence gets messed up? In other words how do you handle tags to prevent conflicts between different matrices? What communicators do you actually do communication on and where do you get them? From the hypre matrix?



> 
> Hope this helps.
> 
> -Rob
> 
> ----------
> status: unread -> chatting
> 
> ____________________________________________
> hypre Issue Tracker <hypre-support at llnl.gov> 
> <http://cascb1.llnl.gov/hypre/issue1595>
> ____________________________________________