<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Manav<div class=""><br class=""></div><div class="">Can you add a MPI_Barrier before</div><div class=""><span style="font-family: Menlo; font-size: 13px; background-color: rgb(255, 255, 255);" class=""><br class=""></span></div><div class=""><span style="font-family: Menlo; font-size: 13px; background-color: rgb(255, 255, 255);" class="">ierr = MatAssemblyBegin(aij->A,mode);CHKERRQ(ierr);</span><div><br class=""></div><div>Also, in order to assess where the issue is, we need to see the values (per rank) of </div><div><br class=""></div><div><span style="font-family: Menlo; font-size: 13px; background-color: rgb(255, 255, 255);" class="">((Mat_SeqAIJ*)aij->B->data)->nonew</span></div><div><span style="font-family: Menlo; font-size: 13px; background-color: rgb(255, 255, 255);" class="">mat->was_assembled</span></div><div><span style="font-family: Menlo; font-size: 13px; background-color: rgb(255, 255, 255);" class="">aij->donotstash</span></div><div><span style="font-family: Menlo; font-size: 13px; background-color: rgb(255, 255, 255);" class="">mat->nooffprocentries</span></div><div><br class=""></div><div>Another question: is this the first matrix assembly of the code?</div><div>If you change to pc_none, do you get the same issue?</div><div><br class=""><blockquote type="cite" class=""><div class="">On Aug 20, 2020, at 3:10 PM, Manav Bhatia <<a href="mailto:bhatiamanav@gmail.com" class="">bhatiamanav@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html; charset=us-ascii" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Aug 19, 2020, at 9:39 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Jed is more knowledgeable about the communication, but I have a simple question about the FEM method. Normally, the way</div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">we divide unknowns is that the only unknowns which might have entries computed off-process are those on the partition boundary.</div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">However, it sounds like you have a huge number of communicated values. Is it possible that the division of rows in your matrix does</div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">not match the division of the cells you compute element matrices for?</div></div></blockquote></div><div class=""><br class=""></div><div class="">I hope that is not the case. I am using libMesh to manage the mesh and creation of sparsity pattern, which uses Parmetis to create the partitions. </div><div class="">libMesh ensures that off-process entries are only at the partition boundary (unless an extra set of DoFs are marked for coupling. </div><div class=""><br class=""></div><div class="">I also printed and looked at the n_nz and n_oz values on each rank and it does not seem to raise any flags. </div><div class=""><br class=""></div><div class="">I will try to dig in a bit further to make sure everything checks out. </div><div class=""><br class=""></div><div class="">Looking at the screenshots I had shared yesterday, all processes are in this function: </div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; color: rgb(88, 208, 247); background-color: rgb(255, 255, 255);" class="">PetscErrorCode<span style="" class=""> </span>MatAssemblyEnd_MPIAIJ<span style="" class="">(Mat mat,MatAssemblyType mode)</span></div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">{</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> Mat_MPIAIJ *aij = (Mat_MPIAIJ*)mat->data;</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> Mat_SeqAIJ *a = (Mat_SeqAIJ*)aij->A->data;</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> PetscErrorCode ierr;</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> PetscMPIInt n;</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> PetscInt i,j,rstart,ncols,flg;</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> PetscInt *row,*col;</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> PetscBool other_disassembled;</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> PetscScalar *val;</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255); min-height: 15px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; color: rgb(154, 170, 186); background-color: rgb(255, 255, 255);" class=""><span style="" class=""> </span>/* do not use 'b = (Mat_SeqAIJ*)aij->B->data' as B can be reset in disassembly */</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255); min-height: 15px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> PetscFunctionBegin;</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> <span style="color: rgb(202, 6, 91);" class=""><b class="">if</b></span> (!aij->donotstash && !mat->nooffprocentries) {</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> <span style="color: rgb(202, 6, 91);" class=""><b class="">while</b></span> (<span style="color: rgb(174, 153, 45);" class="">1</span>) {</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> ierr = MatStashScatterGetMesg_Private(&mat->stash,&n,&row,&col,&val,&flg);CHKERRQ(ierr);</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> <span style="color: rgb(202, 6, 91);" class=""><b class="">if</b></span> (!flg) <span style="color: rgb(202, 6, 91);" class=""><b class="">break</b></span>;</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255); min-height: 15px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> <span style="color: rgb(202, 6, 91);" class=""><b class="">for</b></span> (i=<span style="color: rgb(174, 153, 45);" class="">0</span>; i<n; ) {</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; color: rgb(154, 170, 186); background-color: rgb(255, 255, 255);" class=""><span style="" class=""> </span>/* Now identify the consecutive vals belonging to the same row */</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> <span style="color: rgb(202, 6, 91);" class=""><b class="">for</b></span> (j=i,rstart=row[j]; j<n; j++) {</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> <span style="color: rgb(202, 6, 91);" class=""><b class="">if</b></span> (row[j] != rstart) <span style="color: rgb(202, 6, 91);" class=""><b class="">break</b></span>;</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> }</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> <span style="color: rgb(202, 6, 91);" class=""><b class="">if</b></span> (j < n) ncols = j-i;</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> <span style="color: rgb(202, 6, 91);" class=""><b class="">else</b></span> ncols = n-i;</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; color: rgb(154, 170, 186); background-color: rgb(255, 255, 255);" class=""><span style="" class=""> </span>/* Now assemble all these values with a single function call */</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> ierr = MatSetValues_MPIAIJ(mat,<span style="color: rgb(174, 153, 45);" class="">1</span>,row+i,ncols,col+i,val+i,mat->insertmode);CHKERRQ(ierr);</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255); min-height: 15px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> i = j;</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> }</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> }</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> ierr = MatStashScatterEnd_Private(&mat->stash);CHKERRQ(ierr);</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> }</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> ierr = MatAssemblyBegin(aij->A,mode);CHKERRQ(ierr);</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> ierr = MatAssemblyEnd(aij->A,mode);CHKERRQ(ierr);</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255); min-height: 15px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; color: rgb(154, 170, 186); background-color: rgb(255, 255, 255);" class=""><span style="" class=""> </span>/* determine if any processor has disassembled, if so we must</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; color: rgb(154, 170, 186); background-color: rgb(255, 255, 255);" class=""> also disassemble ourselfs, in order that we may reassemble. */</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> <span style="color: rgb(154, 170, 186);" class="">/*</span></div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; color: rgb(154, 170, 186); background-color: rgb(255, 255, 255);" class=""> if nonzero structure of submatrix B cannot change then we know that</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; color: rgb(154, 170, 186); background-color: rgb(255, 255, 255);" class=""> no processor disassembled thus we can skip this stuff</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; color: rgb(154, 170, 186); background-color: rgb(255, 255, 255);" class=""> */</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> <span style="color: rgb(202, 6, 91);" class=""><b class="">if</b></span> (!((Mat_SeqAIJ*)aij->B->data)->nonew) {</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> ierr = MPIU_Allreduce(&mat->was_assembled,&other_disassembled,<span style="color: rgb(174, 153, 45);" class="">1</span>,MPIU_BOOL,MPI_PROD,PetscObjectComm((PetscObject)mat));CHKERRQ(ierr);</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> <span style="color: rgb(202, 6, 91);" class=""><b class="">if</b></span> (mat->was_assembled && !other_disassembled) {</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> ierr = MatDisAssemble_MPIAIJ(mat);CHKERRQ(ierr);</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> }</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> }</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> <span style="color: rgb(202, 6, 91);" class=""><b class="">if</b></span> (!mat->was_assembled && mode == MAT_FINAL_ASSEMBLY) {</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> ierr = MatSetUpMultiply_MPIAIJ(mat);CHKERRQ(ierr);</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> }</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> ierr = MatSetOption(aij->B,MAT_USE_INODES,PETSC_FALSE);CHKERRQ(ierr);</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> ierr = MatAssemblyBegin(aij->B,mode);CHKERRQ(ierr);</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> ierr = MatAssemblyEnd(aij->B,mode);CHKERRQ(ierr);</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255); min-height: 15px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> ierr = PetscFree2(aij->rowvalues,aij->rowindices);CHKERRQ(ierr);</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255); min-height: 15px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> aij->rowvalues = <span style="color: rgb(174, 153, 45);" class="">0</span>;</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255); min-height: 15px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> ierr = VecDestroy(&aij->diag);CHKERRQ(ierr);</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> <span style="color: rgb(202, 6, 91);" class=""><b class="">if</b></span> (a->inode.size) mat->ops->multdiagonalblock = MatMultDiagonalBlock_MPIAIJ;</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255); min-height: 15px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; color: rgb(154, 170, 186); background-color: rgb(255, 255, 255);" class=""><span style="" class=""> </span>/* if no new nonzero locations are allowed in matrix then only set the matrix state the first time through */</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> <span style="color: rgb(202, 6, 91);" class=""><b class="">if</b></span> ((!mat->was_assembled && mode == MAT_FINAL_ASSEMBLY) || !((Mat_SeqAIJ*)(aij->A->data))->nonew) {</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> PetscObjectState state = aij->A->nonzerostate + aij->B->nonzerostate;</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> ierr = MPIU_Allreduce(&state,&mat->nonzerostate,<span style="color: rgb(174, 153, 45);" class="">1</span>,MPIU_INT64,MPI_SUM,PetscObjectComm((PetscObject)mat));CHKERRQ(ierr);</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> }</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> PetscFunctionReturn(<span style="color: rgb(174, 153, 45);" class="">0</span>);</div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">}</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""> I noticed that of the 8 MPI processes, 2 were stuck at </div><div style="margin: 0px; font-stretch: normal; font-size: 13px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> ierr = MatStashScatterGetMesg_Private(&mat->stash,&n,&row,&col,&val,&flg);CHKERRQ(ierr);</div><div class=""><span style="caret-color: rgb(0, 0, 0);" class=""><br class=""></span></div><div class=""><span style="caret-color: rgb(0, 0, 0);" class="">Other two were stuck at</span></div><div class=""><span style="caret-color: rgb(0, 0, 0);" class=""> </span><span style="font-family: Menlo; font-size: 13px; background-color: rgb(255, 255, 255);" class=""> </span><span style="font-family: Menlo; font-size: 13px; background-color: rgb(255, 255, 255);" class=""> </span><span style="font-family: Menlo; font-size: 13px; background-color: rgb(255, 255, 255);" class="">ierr = MatStashScatterEnd_Private(&mat->stash);CHKERRQ(ierr);</span></div><div class=""><span style="caret-color: rgb(0, 0, 0);" class=""><br class=""></span></div><div class=""><span style="caret-color: rgb(0, 0, 0);" class="">And remaining four were under</span></div><div class=""><span style="caret-color: rgb(0, 0, 0);" class=""> </span><span style="font-family: Menlo; font-size: 13px; background-color: rgb(255, 255, 255);" class=""> </span><span style="font-family: Menlo; font-size: 13px; background-color: rgb(255, 255, 255);" class=""> </span><span style="font-family: Menlo; font-size: 13px; background-color: rgb(255, 255, 255);" class="">ierr = MatSetUpMultiply_MPIAIJ(mat);CHKERRQ(ierr);</span></div><div class=""><span style="font-family: Menlo; font-size: 13px; background-color: rgb(255, 255, 255);" class=""><br class=""></span></div><div class=""><span style="caret-color: rgb(0, 0, 0);" class="">Is it expected for processes to be at different stages in this function? </span></div><div class=""><span style="caret-color: rgb(0, 0, 0);" class=""><br class=""></span></div><div class=""><span style="caret-color: rgb(0, 0, 0);" class="">-Manav</span></div><div class=""><span style="caret-color: rgb(0, 0, 0);" class=""><br class=""></span></div><div class=""><span style="caret-color: rgb(0, 0, 0);" class=""><br class=""></span></div></div></div></blockquote></div><br class=""></div></body></html>