[petsc-users] Parallelize in the y direction

[Barry Smith]

   Trying to solve many "one-dimensional" problems each in parallel on different subset of ranks will be massive pain to do specifically. I recommend just forming a single matrix for all these systems and solving it with KSPSolve and block Jacobi preconditioning or even a parallel direct solver such as with -pc_type lu -pc_factor_mat_solver_type mumps

   Barry

Yes, this single system, in a certain ordering is block diagonal (each block being tridiagonal) so contains "independent" subsystems; what I suggest above essentially takes advantage of this structure to be reasonably efficient, yet trivial to code.

> On Aug 20, 2021, at 9:14 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Fri, Aug 20, 2021 at 7:53 AM Joauma Marichal <joauma.marichal at uclouvain.be <mailto:joauma.marichal at uclouvain.be>> wrote:
>  Dear Sir or Madam,
> 
> I am looking for advice regarding some of PETSc functionnalities. I am currently using PETSc to solve the Navier-Stokes equations on a 3D mesh decomposed over several processors.  However, until now, the processors are distributed along the x and z directions but not along the y one. Indeed, at some point in the algorithm, I must solve a tridiagonal system that depends only on y. Until now, I have therefore performed something like this:
> for(int k = cornp->zs, k<cornp->zs+cornp->zm; ++k){
>      for(int i = cornp->xs, i<cornp->xs+cornp->xm; ++i){
>            Create and solve a tridiagonal system for all the y coordinates (which are on the same process)
> }
> However, I would like to decompose my mesh in the y direction (as this should improve the code efficiency).
> I managed to do so by creating a system based on the 3D DM of all my case (so 1 system of size x*y*z). Unfortunately, this does not seem to be very efficient. 
> Do you have some advice on how to cut in the y direction while still being able to solve x*z systems of size y? Should I create 1D DMs?
> 
> 1) Are you using a 3D DMDA?
> 
> 2) Is the coupling much different in the x and z than in the y direction?
> 
>   Thanks,
> 
>      Matt
>  
> Thanks a lot for your help. 
> 
> Best regards, 
> 
> Joauma Marichal
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20210820/c3c965a0/attachment.html>