<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Sat, Jan 7, 2017 at 5:33 PM, Manuel Valera <span dir="ltr"><<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks Barry and Matt,<div><br></div><div>I was able to detect a bug that i just solved, as suggested the loop parameters weren't updated as it should, now it does and the program still freezes but now in the beginning of the loop... ?</div></div></blockquote><div><br></div><div>You have called a collective function from only one process. Stepping through on both processes in your run will find this easily.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Im attaching screen so you have an idea. Im thinking about it...</div><div><br></div><div>Thanks</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jan 7, 2017 at 3:21 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Sat, Jan 7, 2017 at 4:59 PM, Manuel Valera <span dir="ltr"><<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I would have to think and code a MWE for this problem before sending it since the model is much bigger than the petsc solver. Attached here is a screenshot of the debugger as barry taught me, is that the stack trace you need ?<div><br></div><div>the ucmsMain.f90:522 that shows is the call (from all processes) to the routine that updates the rhs vector (from root) and scatters it (from all processes). </div></div></blockquote><div><br></div></span><div>Yes, so one process is here and the other has moved on, so there is a mismatch in calls.</div><div><br></div><div>You could do what Barry suggests, but I think it would be better to just step through your main routine once (its slow going), and</div><div>see where the divergence happens.</div><div><br></div><div> Matt</div><div><div class="m_-8024679799604385859h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>This routine is itself inside a double loop that occurs in all processes but the only call from all processes to the solver is this one, the rest of the loop which involves correcting for velocities, pressure and temperature, all happens in root node.</div><div><br></div><div>Sorry for the convoluted program design, this is the first beta version of the model working on parallel and was the best i could come with, i suppose it makes more sense in serial,</div><div><br></div><div>Thanks <br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jan 7, 2017 at 2:24 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Sat, Jan 7, 2017 at 4:20 PM, Manuel Valera <span dir="ltr"><<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thank you Matthew, <div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jan 7, 2017 at 1:49 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Sat, Jan 7, 2017 at 3:32 PM, Manuel Valera <span dir="ltr"><<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Devs, hope you are having a great weekend,</div><div><br></div><div>I could finally parallelize my linear solver and implement it into the rest of the code in a way that only the linear system is solved in parallel, great news for my team, but there is a catch and is that i don't see any speedup in the linear system, i don't know if its the MPI in the cluster we are using, but im not sure on how to debug it,</div></div></blockquote><div><br></div></span><div>We need to see -log_view output for any performance question.</div><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>On the other hand and because of this issue i was trying to do -log_summary or -log_view and i noticed the program in this context hangs when is time of producing the log, if i debug this for 2 cores, process 0 exits normally but process 1 hangs in the vectorscatterbegin() with scatter_reverse way back in the code,</div></div></blockquote><div><br></div></span><div>You are calling a collective routine from only 1 process.</div><div> <br></div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div> Matt</div></div></div></div></blockquote><div><br></div><div>I am pretty confident this is not the case, </div></div></div></div></blockquote><div><br></div></span><div>This is still the simplest explanation. Can you send the stack trace for the 2 process run?</div><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>the callings to vecscattercreatetozero and vecscatterbegin are made in all processes, the program goes thru all of the iterations on the linear solver, writes output correctly and even closes all the petsc objects without complaining, the freeze occurs at the very end when the log is to be produced.</div></div></div></div></blockquote><div><br></div></span><div>If you can send us a code to run, we can likely find the error.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div><div class="m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Thanks,</div><div><br></div><div>Manuel</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117m_8553903996033108124h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div> and even after destroying all associated objects and calling petscfinalize(), so im really clueless on why is this, as it only happens for -log_* or -ksp_view options.</div><div><br></div><div>my -ksp_view shows this:</div><div><br></div><div> <span style="font-family:menlo;font-size:11px">KSP Object: 2 MPI processes</span></div>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> type: gcr</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> GCR: restart = 30 </span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> GCR: restarts performed = 20 </span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> maximum iterations=10000, initial guess is zero</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> tolerances: relative=1e-14, absolute=1e-50, divergence=10000.</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> right preconditioning</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> using UNPRECONDITIONED norm type for convergence test</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">PC Object: 2 MPI processes</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> type: bjacobi</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> block Jacobi: number of blocks = 2</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Local solve is same for all blocks, in the following KSP and PC objects:</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> KSP Object: (sub_) 1 MPI processes</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> type: preonly</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> maximum iterations=10000, initial guess is zero</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> left preconditioning</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> using NONE norm type for convergence test</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> PC Object: (sub_) 1 MPI processes</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> type: ilu</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> ILU: out-of-place factorization</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> 0 levels of fill</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> tolerance for zero pivot 2.22045e-14</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> matrix ordering: natural</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> factor fill ratio given 1., needed 1.</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Factored matrix follows:</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Mat Object: 1 MPI processes</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> type: seqaij</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> rows=100000, cols=100000</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> package used to perform factorization: petsc</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> total: nonzeros=1675180, allocated nonzeros=1675180</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> total number of mallocs used during MatSetValues calls =0</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> not using I-node routines</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> linear system matrix = precond matrix:</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Mat Object: 1 MPI processes</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> type: seqaij</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> rows=100000, cols=100000</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> total: nonzeros=1675180, allocated nonzeros=1675180</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> total number of mallocs used during MatSetValues calls =0</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> not using I-node routines</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> linear system matrix = precond matrix:</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> Mat Object: 2 MPI processes</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> type: mpiaij</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> rows=200000, cols=200000</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> total: nonzeros=3373340, allocated nonzeros=3373340</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> total number of mallocs used during MatSetValues calls =0</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> not using I-node (on process 0) routines</span></p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"><br></span></p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"><br></span></p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures">And i configured my PC object as:</span></p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"><br></span></p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(52,187,199)">call</span><span style="font-variant-ligatures:no-common-ligatures"> PCSetType(mg,PCHYPRE,ierr)</span></p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(52,187,199)">call</span><span style="font-variant-ligatures:no-common-ligatures"> PCHYPRESetType(mg,</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">'boomeramg'</span><span style="font-variant-ligatures:no-common-ligatures">,<wbr>ierr)</span></p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(52,187,199)">call</span><span style="font-variant-ligatures:no-common-ligatures"> PetscOptionsSetValue(PETSC_NUL<wbr>L_OBJECT,</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">'pc_hypre_boomeramg_n<wbr>odal_coarsen'</span><span style="font-variant-ligatures:no-common-ligatures">,</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">'1'</span><span style="font-variant-ligatures:no-common-ligatures">,ierr)</span></p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo"><span style="font-variant-ligatures:no-common-ligatures"> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(52,187,199)">call</span><span style="font-variant-ligatures:no-common-ligatures"> PetscOptionsSetValue(PETSC_NUL<wbr>L_OBJECT,</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">'pc_hypre_boomeramg_v<wbr>ec_interp_variant'</span><span style="font-variant-ligatures:no-common-ligatures">,</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">'1'</span><span style="font-variant-ligatures:no-common-ligatures">,ierr)</span></p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo">
</p><p style="margin:0px;font-size:11px;line-height:normal;font-family:menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p><div><br></div><div>What are your thoughts ?</div><div><br></div><div>Thanks,</div><div><br></div><div>Manuel </div><div><br></div><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 6, 2017 at 1:58 PM, Manuel Valera <span dir="ltr"><<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Awesome, that did it, thanks once again.<div><br></div></div><div class="m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117m_8553903996033108124m_-3129844120704230112m_8213395465890979450gmail-HOEnZb"><div class="m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117m_8553903996033108124m_-3129844120704230112m_8213395465890979450gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 6, 2017 at 1:53 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
Take the scatter out of the if () since everyone does it and get rid of the VecView().<br>
<br>
Does this work? If not where is it hanging?<br>
<span><br>
<br>
> On Jan 6, 2017, at 3:29 PM, Manuel Valera <<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>> wrote:<br>
><br>
> Thanks Dave,<br>
><br>
> I think is interesting it never gave an error on this, after adding the vecassembly calls it still shows the same behavior, without complaining, i did:<br>
><br>
> if(rankl==0)then<br>
><br>
> call VecSetValues(bp0,nbdp,ind,Rhs,<wbr>INSERT_VALUES,ierr)<br>
> call VecAssemblyBegin(bp0,ierr) ; call VecAssemblyEnd(bp0,ierr);<br>
> CHKERRQ(ierr)<br>
><br>
</span>endif<br>
<span class="m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117m_8553903996033108124m_-3129844120704230112m_8213395465890979450gmail-m_9152750964337008787im m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117m_8553903996033108124m_-3129844120704230112m_8213395465890979450gmail-m_9152750964337008787HOEnZb">><br>
><br>
> call VecScatterBegin(ctr,bp0,bp2,IN<wbr>SERT_VALUES,SCATTER_REVERSE,ie<wbr>rr)<br>
> call VecScatterEnd(ctr,bp0,bp2,INSE<wbr>RT_VALUES,SCATTER_REVERSE,ierr<wbr>)<br>
> print*,"done! "<br>
> CHKERRQ(ierr)<br>
><br>
><br>
</span><div class="m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117m_8553903996033108124m_-3129844120704230112m_8213395465890979450gmail-m_9152750964337008787HOEnZb"><div class="m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117m_8553903996033108124m_-3129844120704230112m_8213395465890979450gmail-m_9152750964337008787h5">> CHKERRQ(ierr)<br>
><br>
><br>
> Thanks.<br>
><br>
> On Fri, Jan 6, 2017 at 12:44 PM, Dave May <<a href="mailto:dave.mayhem23@gmail.com" target="_blank">dave.mayhem23@gmail.com</a>> wrote:<br>
><br>
><br>
> On 6 January 2017 at 20:24, Manuel Valera <<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>> wrote:<br>
> Great help Barry, i totally had overlooked that option (it is explicit in the vecscatterbegin call help page but not in vecscattercreatetozero, as i read later)<br>
><br>
> So i used that and it works partially, it scatters te values assigned in root but not the rest, if i call vecscatterbegin from outside root it hangs, the code currently look as this:<br>
><br>
> call VecScatterCreateToZero(bp2,ctr<wbr>,bp0,ierr); CHKERRQ(ierr)<br>
><br>
> call PetscObjectSetName(bp0, 'bp0:',ierr)<br>
><br>
> if(rankl==0)then<br>
><br>
> call VecSetValues(bp0,nbdp,ind,Rhs,<wbr>INSERT_VALUES,ierr)<br>
><br>
> call VecView(bp0,PETSC_VIEWER_STDOU<wbr>T_WORLD,ierr)<br>
><br>
><br>
> You need to call<br>
><br>
> VecAssemblyBegin(bp0);<br>
> VecAssemblyEnd(bp0);<br>
> after your last call to VecSetValues() before you can do any operations with bp0.<br>
><br>
> With your current code, the call to VecView should produce an error if you used the error checking macro CHKERRQ(ierr) (as should VecScatter{Begin,End}<br>
><br>
> Thanks,<br>
> Dave<br>
><br>
><br>
> call VecScatterBegin(ctr,bp0,bp2,IN<wbr>SERT_VALUES,SCATTER_REVERSE,ie<wbr>rr)<br>
> call VecScatterEnd(ctr,bp0,bp2,INSE<wbr>RT_VALUES,SCATTER_REVERSE,ierr<wbr>)<br>
> print*,"done! "<br>
> CHKERRQ(ierr)<br>
><br>
> endif<br>
><br>
> ! call VecScatterBegin(ctr,bp0,bp2,IN<wbr>SERT_VALUES,SCATTER_REVERSE,ie<wbr>rr)<br>
> ! call VecScatterEnd(ctr,bp0,bp2,INSE<wbr>RT_VALUES,SCATTER_REVERSE,ierr<wbr>)<br>
><br>
> call VecView(bp2,PETSC_VIEWER_STDOU<wbr>T_WORLD,ierr)<br>
><br>
> call PetscBarrier(PETSC_NULL_OBJECT<wbr>,ierr)<br>
><br>
> call exit()<br>
><br>
><br>
><br>
> And the output is: (with bp the right answer)<br>
><br>
> Vec Object:bp: 2 MPI processes<br>
> type: mpi<br>
> Process [0]<br>
> 1.<br>
> 2.<br>
> Process [1]<br>
> 4.<br>
> 3.<br>
> Vec Object:bp2: 2 MPI processes (before scatter)<br>
> type: mpi<br>
> Process [0]<br>
> 0.<br>
> 0.<br>
> Process [1]<br>
> 0.<br>
> 0.<br>
> Vec Object:bp0: 1 MPI processes<br>
> type: seq<br>
> 1.<br>
> 2.<br>
> 4.<br>
> 3.<br>
> done!<br>
> Vec Object:bp2: 2 MPI processes (after scatter)<br>
> type: mpi<br>
> Process [0]<br>
> 1.<br>
> 2.<br>
> Process [1]<br>
> 0.<br>
> 0.<br>
><br>
><br>
><br>
><br>
> Thanks inmensely for your help,<br>
><br>
> Manuel<br>
><br>
><br>
> On Thu, Jan 5, 2017 at 4:39 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>
><br>
> > On Jan 5, 2017, at 6:21 PM, Manuel Valera <<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>> wrote:<br>
> ><br>
> > Hello Devs is me again,<br>
> ><br>
> > I'm trying to distribute a vector to all called processes, the vector would be originally in root as a sequential vector and i would like to scatter it, what would the best call to do this ?<br>
> ><br>
> > I already know how to gather a distributed vector to root with VecScatterCreateToZero, this would be the inverse operation,<br>
><br>
> Use the same VecScatter object but with SCATTER_REVERSE, not you need to reverse the two vector arguments as well.<br>
><br>
><br>
> > i'm currently trying with VecScatterCreate() and as of now im doing the following:<br>
> ><br>
> ><br>
> > if(rank==0)then<br>
> ><br>
> ><br>
> > call VecCreate(PETSC_COMM_SELF,bp0,<wbr>ierr); CHKERRQ(ierr) !if i use WORLD<br>
> > !freezes in SetSizes<br>
> > call VecSetSizes(bp0,PETSC_DECIDE,n<wbr>bdp,ierr); CHKERRQ(ierr)<br>
> > call VecSetType(bp0,VECSEQ,ierr)<br>
> > call VecSetFromOptions(bp0,ierr); CHKERRQ(ierr)<br>
> ><br>
> ><br>
> > call VecSetValues(bp0,nbdp,ind,Rhs,<wbr>INSERT_VALUES,ierr)<br>
> ><br>
> > !call VecSet(bp0,5.0D0,ierr); CHKERRQ(ierr)<br>
> ><br>
> ><br>
> > call VecView(bp0,PETSC_VIEWER_STDOU<wbr>T_WORLD,ierr)<br>
> ><br>
> > call VecAssemblyBegin(bp0,ierr) ; call VecAssemblyEnd(bp0,ierr) !rhs<br>
> ><br>
> > do i=0,nbdp-1,1<br>
> > ind(i+1) = i<br>
> > enddo<br>
> ><br>
> > call ISCreateGeneral(PETSC_COMM_SEL<wbr>F,nbdp,ind,PETSC_COPY_VALUES,l<wbr>ocis,ierr)<br>
> ><br>
> > !call VecScatterCreate(bp0,PETSC_NUL<wbr>L_OBJECT,bp2,is,ctr,ierr) !if i use SELF<br>
> > !freezes here.<br>
> ><br>
> > call VecScatterCreate(bp0,locis,bp2<wbr>,PETSC_NULL_OBJECT,ctr,ierr)<br>
> ><br>
> > endif<br>
> ><br>
> > bp2 being the receptor MPI vector to scatter to<br>
> ><br>
> > But it freezes in VecScatterCreate when trying to use more than one processor, what would be a better approach ?<br>
> ><br>
> ><br>
> > Thanks once again,<br>
> ><br>
> > Manuel<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On Wed, Jan 4, 2017 at 3:30 PM, Manuel Valera <<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>> wrote:<br>
> > Thanks i had no idea how to debug and read those logs, that solved this issue at least (i was sending a message from root to everyone else, but trying to catch from everyone else including root)<br>
> ><br>
> > Until next time, many thanks,<br>
> ><br>
> > Manuel<br>
> ><br>
> > On Wed, Jan 4, 2017 at 3:23 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
> > On Wed, Jan 4, 2017 at 5:21 PM, Manuel Valera <<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>> wrote:<br>
> > I did a PetscBarrier just before calling the vicariate routine and im pretty sure im calling it from every processor, code looks like this:<br>
> ><br>
> > From the gdb trace.<br>
> ><br>
> > Proc 0: Is in some MPI routine you call yourself, line 113<br>
> ><br>
> > Proc 1: Is in VecCreate(), line 130<br>
> ><br>
> > You need to fix your communication code.<br>
> ><br>
> > Matt<br>
> ><br>
> > call PetscBarrier(PETSC_NULL_OBJECT<wbr>,ierr)<br>
> ><br>
> > print*,'entering POInit from',rank<br>
> > !call exit()<br>
> ><br>
> > call PetscObjsInit()<br>
> ><br>
> ><br>
> > And output gives:<br>
> ><br>
> > entering POInit from 0<br>
> > entering POInit from 1<br>
> > entering POInit from 2<br>
> > entering POInit from 3<br>
> ><br>
> ><br>
> > Still hangs in the same way,<br>
> ><br>
> > Thanks,<br>
> ><br>
> > Manuel<br>
> ><br>
> ><br>
> ><br>
> > On Wed, Jan 4, 2017 at 2:55 PM, Manuel Valera <<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>> wrote:<br>
> > Thanks for the answers !<br>
> ><br>
> > heres the screenshot of what i got from bt in gdb (great hint in how to debug in petsc, didn't know that)<br>
> ><br>
> > I don't really know what to look at here,<br>
> ><br>
> > Thanks,<br>
> ><br>
> > Manuel<br>
> ><br>
> > On Wed, Jan 4, 2017 at 2:39 PM, Dave May <<a href="mailto:dave.mayhem23@gmail.com" target="_blank">dave.mayhem23@gmail.com</a>> wrote:<br>
> > Are you certain ALL ranks in PETSC_COMM_WORLD call these function(s). These functions cannot be inside if statements like<br>
> > if (rank == 0){<br>
> > VecCreateMPI(...)<br>
> > }<br>
> ><br>
> ><br>
> > On Wed, 4 Jan 2017 at 23:34, Manuel Valera <<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>> wrote:<br>
> > Thanks Dave for the quick answer, appreciate it,<br>
> ><br>
> > I just tried that and it didn't make a difference, any other suggestions ?<br>
> ><br>
> > Thanks,<br>
> > Manuel<br>
> ><br>
> > On Wed, Jan 4, 2017 at 2:29 PM, Dave May <<a href="mailto:dave.mayhem23@gmail.com" target="_blank">dave.mayhem23@gmail.com</a>> wrote:<br>
> > You need to swap the order of your function calls.<br>
> > Call VecSetSizes() before VecSetType()<br>
> ><br>
> > Thanks,<br>
> > Dave<br>
> ><br>
> ><br>
> > On Wed, 4 Jan 2017 at 23:21, Manuel Valera <<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>> wrote:<br>
> > Hello all, happy new year,<br>
> ><br>
> > I'm working on parallelizing my code, it worked and provided some results when i just called more than one processor, but created artifacts because i didn't need one image of the whole program in each processor, conflicting with each other.<br>
> ><br>
> > Since the pressure solver is the main part i need in parallel im chosing mpi to run everything in root processor until its time to solve for pressure, at this point im trying to create a distributed vector using either<br>
> ><br>
> > call VecCreateMPI(PETSC_COMM_WORLD,<wbr>PETSC_DECIDE,nbdp,xp,ierr)<br>
> > or<br>
> > call VecCreate(PETSC_COMM_WORLD,xp,<wbr>ierr); CHKERRQ(ierr)<br>
> > call VecSetType(xp,VECMPI,ierr)<br>
> > call VecSetSizes(xp,PETSC_DECIDE,nb<wbr>dp,ierr); CHKERRQ(ierr)<br>
> ><br>
> ><br>
> > In both cases program hangs at this point, something it never happened on the naive way i described before. I've made sure the global size, nbdp, is the same in every processor. What can be wrong?<br>
> ><br>
> > Thanks for your kind help,<br>
> ><br>
> > Manuel.<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> > -- Norbert Wiener<br>
> ><br>
> ><br>
><br>
><br>
><br>
><br>
<br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>
</blockquote></div></div></div><div><div class="m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117m_8553903996033108124h5"><br><br clear="all"><span class="m_-8024679799604385859m_-3650215258836467221HOEnZb"><font color="#888888"><span class="m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117HOEnZb"><font color="#888888"><div><br></div>-- <br><div class="m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117m_8553903996033108124m_-3129844120704230112gmail_signature" data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</font></span></font></span></div></div></div></div><span class="m_-8024679799604385859m_-3650215258836467221HOEnZb"><font color="#888888">
</font></span></blockquote></div><span class="m_-8024679799604385859m_-3650215258836467221HOEnZb"><font color="#888888"><br></font></span></div></div><span class="m_-8024679799604385859m_-3650215258836467221HOEnZb"><font color="#888888">
</font></span></blockquote></div></div></div><span class="m_-8024679799604385859m_-3650215258836467221HOEnZb"><font color="#888888"><div><div class="m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091h5"><br><br clear="all"><span class="HOEnZb"><font color="#888888"><div><br></div>-- <br><div class="m_-8024679799604385859m_-3650215258836467221m_-9127560712157743668m_-836951528220087091m_1920829511659719117gmail_signature" data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</font></span></div></div></font></span></div></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><br></font></span></div></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote></div></div></div><span class="HOEnZb"><font color="#888888"><div><div class="m_-8024679799604385859h5"><br><br clear="all"><div><br></div>-- <br><div class="m_-8024679799604385859m_-3650215258836467221gmail_signature" data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div></font></span></div></div>
</blockquote></div><br></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>