<div dir="ltr">Thanks a lot!<div>That clears things up.</div><div>I do have one more question though.</div><div>The behavior of DMDAVecGetArray is different then? From ex3.c (below), it doesn't seem like the raw array needs the local indices?</div><div><br></div><div>From ex3.c, </div><div>
<pre width="80" style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><a name="line82"> </a> <a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDAVecGetArray.html#DMDAVecGetArray">DMDAVecGetArray</a>(cda,global,&coors);</pre><pre width="80" style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">...</pre><pre width="80" style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><pre width="80" style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"> <a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDAGetCorners.html#DMDAGetCorners">DMDAGetCorners</a>(cda,&mstart,&nstart,&pstart,&m,&n,&p);</pre><pre width="80" style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">...</pre><pre width="80" style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><pre style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap"> for (i=mstart; i<mstart+m; i++) {
for (j=nstart; j<nstart+n; j++) {
for (k=pstart; k<pstart+p; k++) {
...
coors[k][j][i].x = ...</pre>
<br></pre>
<br></pre>
<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Apr 22, 2018 at 3:14 PM, Dave May <span dir="ltr"><<a href="mailto:dave.mayhem23@gmail.com" target="_blank">dave.mayhem23@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><br><div class="gmail_quote"><span class=""><div dir="auto">On Sun, 22 Apr 2018 at 20:13, Zin Lin <<a href="mailto:zinlin.zinlin@gmail.com" target="_blank">zinlin.zinlin@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>Hi <div>I am experiencing possible memory issues with VecGetArray when it is used under sub-communicators (when I split the PETSC_COMM_WORLD to multiple subcomms). The following is the minimal code. Basically, you can test that if you parallelize the vector to more than one processor under a subcomm, the array obtained from the VecGetArray call doesn't seem to be correct.</div><div>Please test it with </div><div><br></div><div>1) mpirun -np 1 ./example -ncomms 1</div><div>2)
<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">mpirun -np 2 ./example -ncomms 2</span> </div><div>3)
<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">mpirun -np 2 ./example -ncomms 1</span>
<br>4) <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">mpirun -np 4 ./example -ncomms 2</span><br><div class="m_2845576699813478239m_-6496824137315727320gmail_signature"><div><br></div><div>you will 1) and 2) work as expected while in 3) and 4) some entries of the array are assigned erroneous values.</div></div></div></div></blockquote><div dir="auto"><br></div></span><div dir="auto">First - your call to PetscPrintf contains a mistake. The second instance of i is missing.</div><div dir="auto"><br></div><div dir="auto">Second (more important), you cannot access the raw array obtained via VecGetArray() using global indices. You must use local indices.</div><div dir="auto">Change the access to _u[i-ns] and the code should work.</div><div dir="auto"><br></div><div dir="auto">Also, debugging based on what printf() tells you can be misleading. It's much better to use valgrind - see here </div><div dir="auto"><br></div><div dir="auto"><a href="https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" target="_blank">https://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>valgrind</a><br></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto">Thanks,</div><div dir="auto"> Dave</div><div><div class="h5"><div dir="auto"><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div class="m_2845576699813478239m_-6496824137315727320gmail_signature"><div></div><div><br></div><div>Any input will be appreciated.</div><div>Thanks</div><div>Zin</div><div><br></div><div>Minimal Code</div><div><br></div><div><div><font face="monospace, monospace">PetscErrorCode main(int argc, char **argv)</font></div><div><font face="monospace, monospace">{</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> MPI_Init(NULL, NULL);</font></div><div><font face="monospace, monospace"> PetscInitialize(&argc,&argv,<wbr>NULL,NULL);</font></div><div><font face="monospace, monospace"> int size;</font></div><div><font face="monospace, monospace"> MPI_Comm_size(MPI_COMM_WORLD, &size);</font></div><div><font face="monospace, monospace"> PetscPrintf(PETSC_COMM_WORLD,"<wbr>\tThe total number of processors is %d\n",size);</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> //check if the number of processors is divisible by the number of subcomms</font></div><div><font face="monospace, monospace"> int ncomms, np_per_comm;</font></div><div><font face="monospace, monospace"> PetscOptionsGetInt(NULL,"-<wbr>ncomms",&ncomms,NULL);</font></div><div><font face="monospace, monospace"> if(!(size%ncomms==0)) SETERRQ(PETSC_COMM_WORLD,1,"<wbr>The number of processes must be a multiple of ncomms so that it is divisible by the number of subcomms.");</font></div><div><font face="monospace, monospace"> np_per_comm=size/ncomms;</font></div><div><font face="monospace, monospace"> PetscPrintf(PETSC_COMM_WORLD,"<wbr>\tThe number of subcomms is %d.\n\tEach subcomm has %d processors.\n",ncomms,np_per_<wbr>comm);</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> //calculate the colour of each subcomm ( = rank of each processor / number of processors in each subcomm )</font></div><div><font face="monospace, monospace"> //note once calculated, the colour is fixed throughout the entire run</font></div><div><font face="monospace, monospace"> int rank;</font></div><div><font face="monospace, monospace"> MPI_Comm_rank(MPI_COMM_WORLD, &rank);</font></div><div><font face="monospace, monospace"> MPI_Comm subcomm;</font></div><div><font face="monospace, monospace"> int colour = rank/np_per_comm;</font></div><div><font face="monospace, monospace"> MPI_Comm_split(MPI_COMM_WORLD, colour, rank, &subcomm);</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> Vec u;</font></div><div><font face="monospace, monospace"> PetscScalar *_u;</font></div><div><font face="monospace, monospace"> int i,ns,ne;</font></div><div><font face="monospace, monospace"> PetscScalar tmp;</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> VecCreateMPI(subcomm,PETSC_<wbr>DECIDE,10,&u);</font></div><div><font face="monospace, monospace"> VecSet(u,1.0+PETSC_i*0);</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> VecGetArray(u,&_u);</font></div><div><font face="monospace, monospace"> VecGetOwnershipRange(u,&ns,&<wbr>ne);</font></div><div><font face="monospace, monospace"> for(i=ns;i<ne;i++){</font></div><div><font face="monospace, monospace"> VecGetValues(u,1,&i,&tmp);</font></div><div><font face="monospace, monospace"> PetscPrintf(PETSC_COMM_SELF,"<wbr>colour %d, u[%d]_array = %g + i * (%g), u[%d]_vec = %g + i * %g \n",colour,i,creal(_u[i]),<wbr>cimag(_u[i]),</font></div></div></div></div></div></blockquote><div dir="auto"><br></div></div></div><div dir="auto">Insert i here</div><div dir="auto"><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div class="m_2845576699813478239m_-6496824137315727320gmail_signature"><div><div><font face="monospace, monospace">creal(tmp),cimag(tmp));</font></div><div><font face="monospace, monospace"> }</font></div><div><font face="monospace, monospace"> VecRestoreArray(u,&_u);</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> PetscFinalize();</font></div><div><font face="monospace, monospace"> return 0;</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">}</font></div><br></div></div>
</div></div>
</blockquote></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Zin Lin<br><br></div></div>
</div>