<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> Ok, fieldsplit, pretty much the same as what I said for block Jacobi. <div class=""><br class=""></div><div class=""><div class="">static PetscErrorCode PCApply_FieldSplit(PC pc,Vec x,Vec y)</div><div class="">{</div><div class=""> PC_FieldSplit *jac = (PC_FieldSplit*)pc->data;</div><div class=""> PetscErrorCode ierr;</div><div class=""> PC_FieldSplitLink ilink = jac->head;</div><div class=""> PetscInt cnt,bs;</div><div class=""><br class=""></div><div class=""> PetscFunctionBegin;</div><div class=""> if (jac->type == PC_COMPOSITE_ADDITIVE) {</div><div class=""> if (jac->defaultsplit) {</div><div class=""> ierr = VecGetBlockSize(x,&bs);CHKERRQ(ierr);</div><div class=""> if (jac->bs > 0 && bs != jac->bs) SETERRQ2(PetscObjectComm((PetscObject)pc),PETSC_ERR_ARG_WRONGSTATE,"Blocksize of x vector %D does not match fieldsplit blocksize %D",bs,jac->bs);</div><div class=""> ierr = VecGetBlockSize(y,&bs);CHKERRQ(ierr);</div><div class=""> if (jac->bs > 0 && bs != jac->bs) SETERRQ2(PetscObjectComm((PetscObject)pc),PETSC_ERR_ARG_WRONGSTATE,"Blocksize of y vector %D does not match fieldsplit blocksize %D",bs,jac->bs);</div><div class=""> ierr = VecStrideGatherAll(x,jac->x,INSERT_VALUES);CHKERRQ(ierr);</div><div class=""> while (ilink) {</div><div class=""> ierr = PetscLogEventBegin(ilink->event,ilink->ksp,ilink->x,ilink->y,NULL);CHKERRQ(ierr);</div><div class=""> ierr = KSPSolve(ilink->ksp,ilink->x,ilink->y);CHKERRQ(ierr);</div><div class=""><br class=""></div><div class="">you need these KSPSolve() to not block which if you are using SuperLU_DIST means each call to </div><div class=""><br class=""></div><div class=""><div class=""> PetscStackCall("SuperLU_DIST:PStatInit",PStatInit(&stat)); /* Initialize the statistics variables. */</div><div class="">#if defined(PETSC_USE_COMPLEX)</div><div class=""> PetscStackCall("SuperLU_DIST:pzgssvx",pzgssvx(&lu->options,&lu->A_sup,&lu->ScalePermstruct,(doublecomplex*)bptr,m,1,&lu->grid,&lu->LUstruct,&lu->SOLVEstruct,berr,&stat,&info));</div><div class="">#else</div><div class=""> PetscStackCall("SuperLU_DIST:pdgssvx",pdgssvx(&lu->options,&lu->A_sup,&lu->ScalePermstruct,bptr,m,1,&lu->grid,&lu->LUstruct,&lu->SOLVEstruct,berr,&stat,&info));</div><div class="">#endif</div></div><div class=""><br class=""></div><div class="">must be on a different stream and cannot block. Which means essentially the PStatInit and pdgssvx have to run completely on the GPU or have some complex CPU babysitting that allows them to be interlaced on the GPU while doing their CPU stuff. I doubt either thing is true and you are best off using the cuSparse Solvers which do run completely on the GPU in any stream you give them.</div><div class=""><br class=""></div><div class="">The factorization is even more of a nightmare, I think Sherry uses the GPU just to do blocks for her sequential factorization so I doubt very much it can handle multiple simultaneous factorizations. </div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""> ierr = KSPCheckSolve(ilink->ksp,pc,ilink->y);CHKERRQ(ierr);</div><div class=""> ierr = PetscLogEventEnd(ilink->event,ilink->ksp,ilink->x,ilink->y,NULL);CHKERRQ(ierr);</div><div class=""> ilink = ilink->next;</div><div class=""> }</div><div class=""><br class=""></div><div class=""><br class=""></div><div><br class=""><blockquote type="cite" class=""><div class="">On Jan 9, 2021, at 6:06 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jan 9, 2021 at 5:14 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br class="">
If it is non-overlapping do you mean block Jacobi with multiple blocks per MPI rank? (one block per rank is trivial and should work now). <br class=""></blockquote><div class=""><br class=""></div><div class="">Yes, only one MPI rank.</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br class="">
If you mean block Jacobi with multiple blocks per MPI rank you should start with the PCApply_BJacob_Multiblock(). It monkey's with pointers into the vector and then calls KSPSolve() for each block. So you just need a non-blocking KSPSolve(). What KSP and what PC do you want to use per block? </blockquote><div class=""><br class="">SuperLU</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">If you want to use LU then I think you proceed largely as I said a few days ago. All the routines in MatSolve_SeqAIJCUSparse can be made non-blocking as discussed with each one using its own stream (supplied with a hack or with approaches from Junchao and Jacob in the future possibly; but a hack is all that is needed for this trivial case.)<br class=""></blockquote><div class=""><br class=""></div><div class="">I'm not sure what goes into this hack. </div><div class=""><br class=""></div><div class="">I am using fieldsplit and I now see that I don't specify asm, just lu, so I guess PCApply_FieldSplit is the target</div><div class=""><br class=""></div><div class=""> KSP Object: 1 MPI processes<br class=""> type: preonly<br class=""> maximum iterations=10000, initial guess is zero<br class=""> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br class=""> left preconditioning<br class=""> using NONE norm type for convergence test<br class=""> PC Object: 1 MPI processes<br class=""> type: fieldsplit<br class=""> FieldSplit with ADDITIVE composition: total splits = 2<br class=""> Solver info for each split is in the following KSP objects:<br class=""> Split number 0 Defined by IS<br class=""> KSP Object: (fieldsplit_e_) 1 MPI processes<br class=""> type: preonly<br class=""> maximum iterations=10000, initial guess is zero<br class=""> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br class=""> left preconditioning<br class=""> using NONE norm type for convergence test<br class=""> PC Object: (fieldsplit_e_) 1 MPI processes<br class=""> type: lu<br class=""> out-of-place factorization<br class=""> tolerance for zero pivot 2.22045e-14<br class=""> matrix ordering: nd<br class=""> factor fill ratio given 5., needed 1.30805<br class=""> Factored matrix follows:<br class=""> Mat Object: 1 MPI processes<br class=""> type: seqaij<br class=""> rows=448, cols=448<br class=""> package used to perform factorization: petsc<br class=""> total: nonzeros=14038, allocated nonzeros=14038<br class=""> using I-node routines: found 175 nodes, limit used is 5<br class=""> linear system matrix = precond matrix:<br class=""> Mat Object: (fieldsplit_e_) 1 MPI processes<br class=""> type: seqaij<br class=""> rows=448, cols=448<br class=""> total: nonzeros=10732, allocated nonzeros=10732<br class=""> total number of mallocs used during MatSetValues calls=0<br class=""> using I-node routines: found 197 nodes, limit used is 5<br class=""> Split number 1 Defined by IS<br class=""> KSP Object: (fieldsplit_i1_) 1 MPI processes<br class=""></div><div class=""> ....</div><div class=""><br class=""></div></div></div>
</div></blockquote></div><br class=""></div></body></html>