<div dir="ltr">Derek,<div><br></div><div>can you run with --track-origins=yes? There are few possibilities for the uninitialized warning (candidates for the uninitialized errors are the arrays starts, indices, and the variables bs or n) in the below code, and this valgrind option will help. </div><div><br></div><div>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">PetscErrorCode VecScatterMemcpyPlanCreate_Index(PetscInt n,</span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(52,188,38)">const</span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"> PetscInt *starts,</span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(52,188,38)">const</span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"> PetscInt *indices,PetscInt bs,VecScatterMemcpyPlan *plan)</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">{</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>PetscErrorCode ierr;</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>PetscInt <span class="gmail-Apple-converted-space"> </span>i,j,k,my_copies,n_copies=</span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">0</span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">,step;</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>PetscBool<span class="gmail-Apple-converted-space"> </span>strided,has_strided;</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:16px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>PetscFunctionBegin;</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>ierr<span class="gmail-Apple-converted-space"> </span>= PetscMemzero(plan,</span><span class="gmail-s4" style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">sizeof</span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">(VecScatterMemcpyPlan));CHKERRQ(ierr);</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>plan->n = n;</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>ierr<span class="gmail-Apple-converted-space"> </span>= PetscMalloc2(n,&plan->optimized,n+</span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">1</span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">,&plan->copy_offsets);CHKERRQ(ierr);</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:16px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p3" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(82,48,225)"><span class="gmail-s5" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"><span class="gmail-Apple-converted-space"> </span></span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">/* check if each remote part of the scatter is made of copies, and count total_copies */</span></p>
<p class="gmail-p3" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(82,48,225)"><span class="gmail-s5" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"><span class="gmail-Apple-converted-space"> </span></span><span class="gmail-s4" style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">for</span><span class="gmail-s5" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"> (i=</span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">0</span><span class="gmail-s5" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">; i<n; i++) { </span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">/* for each target processor procs[i] */</span></p>
<p class="gmail-p3" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(82,48,225)"><span class="gmail-s5" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"><span class="gmail-Apple-converted-space"> </span>my_copies = </span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">1</span><span class="gmail-s5" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">; </span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">/* count num. of copies for procs[i] */</span></p>
<p class="gmail-p3" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(82,48,225)"><span class="gmail-s5" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"><span class="gmail-Apple-converted-space"> </span></span><span class="gmail-s4" style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">for</span><span class="gmail-s5" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"> (j=starts[i]; j<starts[i+</span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">1</span><span class="gmail-s5" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">]-</span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">1</span><span class="gmail-s5" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">; j++) { </span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">/* go through indices from the first to the second to last */</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span></span><span class="gmail-s4" style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">if</span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"> (indices[j]+bs != indices[j+</span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">1</span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">]) my_copies++;</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>}</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span></span><span class="gmail-s4" style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">if</span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"> (bs*(starts[i+</span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">1</span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">]-starts[i])*</span><span class="gmail-s4" style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">sizeof</span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">(PetscScalar)/my_copies >= </span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">256</span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">) { </span><span class="gmail-s6" style="font-variant-ligatures:no-common-ligatures;color:rgb(82,48,225)">/* worth using memcpy? */</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>plan->optimized[i] = PETSC_TRUE;</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>n_copies += my_copies;</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>} </span><span class="gmail-s4" style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">else</span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"> {</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>plan->optimized[i] = PETSC_FALSE;</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>}</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>}</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:16px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p3" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(82,48,225)"><span class="gmail-s5" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"><span class="gmail-Apple-converted-space"> </span></span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">/* do malloc with the recently known n_copies */</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space">-> THIS IS THE VAGRIND WARNING </span>ierr = PetscMalloc2(n_copies,&plan->copy_starts,n_copies,&plan->copy_lengths);CHKERRQ(ierr);</span></p></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno gio 21 mar 2019 alle ore 09:00 Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com">stefano.zampini@gmail.com</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Derek<div dir="auto"><br></div><div dir="auto">I have fixed the optimized plan few weeks ago</div><div dir="auto"><br></div><div dir="auto"><a href="https://bitbucket.org/petsc/petsc/commits/c3caad8634d376283f7053f3b388606b45b3122c" target="_blank">https://bitbucket.org/petsc/petsc/commits/c3caad8634d376283f7053f3b388606b45b3122c</a><br></div><div dir="auto"><br></div><div dir="auto">Maybe this will fix your problem too?</div><div dir="auto"><br></div><div dir="auto">Stefano</div><div dir="auto"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il Gio 21 Mar 2019, 04:21 Zhang, Junchao via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div dir="ltr">Hi, Derek,
<div> Try to apply this tiny (but dirty) patch on your version of PETSc to disable the VecScatterMemcpyPlan optimization to see if it helps. </div>
<div> Thanks.<br clear="all">
<div>
<div dir="ltr" class="gmail-m_-3688999174852170079m_4030954464987556008gmail_signature">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Mar 20, 2019 at 6:33 PM Junchao Zhang <<a href="mailto:jczhang@mcs.anl.gov" rel="noreferrer" target="_blank">jczhang@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Did you see the warning with small scale runs? Is it possible to provide a test code?</div>
<div>You mentioned "changing PETSc now would be pretty painful". Is it because it will affect your performance (but not your code)? If yes, could you try PETSc master and run you code with or without -vecscatter_type sf. I want to isolate the problem and
see if it is due to possible bugs in VecScatter.</div>
<div>If the above suggestion is not feasible, I will disable VecScatterMemcpy. It is an optimization I added. Sorry I did not have an option to turn off it because I thought it was always useful:) I will provide you a patch later to disable it. With that you
can run again to isolate possible bugs in VecScatterMemcpy.</div>
<div>Thanks.</div>
<div>
<div dir="ltr" class="gmail-m_-3688999174852170079m_4030954464987556008gmail-m_958850508585676410gmail_signature">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Mar 20, 2019 at 5:40 PM Derek Gaston via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" rel="noreferrer" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Trying to track down some memory corruption I'm seeing on larger scale runs (3.5B+ unknowns). Was able to run Valgrind on it... and I'm seeing quite a lot of uninitialized value errors coming from ghost updating. Here are some of the traces:</div>
<div><br>
</div>
<div>==87695== Conditional jump or move depends on uninitialised value(s)<br>
==87695== at 0x73236D3: PetscMallocAlign (mal.c:28)<br>
==87695== by 0x7323C70: PetscMallocA (mal.c:390)<br>
==87695== by 0x739048E: VecScatterMemcpyPlanCreate_Index (vscat.c:284)<br>
==87695== by 0x73A5D97: VecScatterMemcpyPlanCreate_PtoP (vpscat_mpi1.c:312)<br>
==64730== by 0x7393E8A: VecScatterSetUp_vectype_private (vscat.c:857)<br>
==64730== by 0x7395E5D: VecScatterSetUp_MPI1 (vpscat_mpi1.c:2543)<br>
==64730== by 0x73DDD39: VecScatterSetUp (vscatfce.c:212)<br>
==64730== by 0x73DCD73: VecScatterCreateWithData (vscreate.c:333)<br>
==64730== by 0x7444232: VecCreateGhostWithArray (pbvec.c:685)<br>
==64730== by 0x744490D: VecCreateGhost (pbvec.c:741)<br>
</div>
<div><br>
</div>
<div>==133582== Conditional jump or move depends on uninitialised value(s)<br>
==133582== at 0x4030384: memcpy@@GLIBC_2.14 (vg_replace_strmem.c:1034)<br>
==133582== by 0x739E4F9: PetscMemcpy (petscsys.h:1649)<br>
==133582== by 0x739E4F9: VecScatterMemcpyPlanExecute_Pack (vecscatterimpl.h:150)<br>
==133582== by 0x739E4F9: VecScatterBeginMPI1_1 (vpscat_mpi1.h:69)<br>
==133582== by 0x73DD964: VecScatterBegin (vscatfce.c:110)<br>
==133582== by 0x744E195: VecGhostUpdateBegin (commonmpvec.c:225)<br>
</div>
<div><br>
</div>
<div>This is from a Git checkout of PETSc... the hash I branched from is: 0e667e8fea4aa from December 23rd (updating would be really hard at this point as I've completed 90% of my dissertation with this version... and changing PETSc now would be pretty painful!).</div>
<div><br>
</div>
<div>Any ideas? Is it possible it's in my code? Is it possible that there are later PETSc commits that already fix this?</div>
<div><br>
</div>
<div>Thanks for any help,</div>
<div>Derek<br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">Stefano</div>