<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Aug 20, 2020, at 8:31 AM, Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" class="">stefano.zampini@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Can you add a MPI_Barrier before</div><div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span class="" style="font-family: Menlo; font-size: 13px; background-color: rgb(255, 255, 255);"><br class=""></span></div><div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span class="" style="font-family: Menlo; font-size: 13px; background-color: rgb(255, 255, 255);">ierr = MatAssemblyBegin(aij->A,mode);CHKERRQ(ierr);</span><div class=""><br class=""></div></div></div></blockquote><div><br class=""></div><div>With a MPI_Barrier before this function call:</div><div>— three of the processes have already hit this barrier, </div><div>— the other 5 are inside MatStashScatterGetMesg_Private -> MatStashScatterGerMesg_BTS -> MPI_Waitsome(2 processes)/MPI_Waitall(3 processes)</div><div><br class=""></div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div class="">Also, in order to assess where the issue is, we need to see the values (per rank) of </div><div class=""><br class=""></div><div class=""><span class="" style="font-family: Menlo; font-size: 13px; background-color: rgb(255, 255, 255);">((Mat_SeqAIJ*)aij->B->data)->nonew</span></div><div class=""><span class="" style="font-family: Menlo; font-size: 13px; background-color: rgb(255, 255, 255);">mat->was_assembled</span></div><div class=""><span class="" style="font-family: Menlo; font-size: 13px; background-color: rgb(255, 255, 255);">aij->donotstash</span></div><div class=""><span class="" style="font-family: Menlo; font-size: 13px; background-color: rgb(255, 255, 255);">mat->nooffprocentries</span></div><div class=""><br class=""></div></div></div></blockquote><div><br class=""></div><div>I am working to get this information. </div><br class=""><blockquote type="cite" class=""><div class=""><div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div class="">Another question: is this the first matrix assembly of the code?</div></div></div></blockquote><div><br class=""></div><div>Yes, this is the first matrix assembly in the code. </div><br class=""><blockquote type="cite" class=""><div class=""><div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div class="">If you change to pc_none, do you get the same issue?</div></div></div></blockquote></div><div class=""><br class=""></div><div class="">Yes, with "-pc_type none” the code is stuck at the same spot. </div></body></html>