On Tue, Apr 3, 2012 at 7:40 AM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I think you are mis-understanding the global ordering when using DA vectors in parallel. You might consider using VecSetValuesLocal() or VecSetValuesStencil() instead of VecSetValues(). Read up on the users manual and examples about PETSc global ordering.</blockquote>
<div><br></div><div>More explicitly, you do set the 5th value of the vector each time, but the layout is different when created with the DA</div><div>since it is splitting the grid somehow. When you call VecView() it reorders the Vec to match the grid before output.</div>
<div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="HOEnZb"><font color="#888888"><br>
Barry<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
On Apr 3, 2012, at 3:46 AM, khalid ashraf wrote:<br>
<br>
> Hi Matt, I ran the program on two processors. I am trying to enter value into a<br>
> vector associated with DA. I am using VecSetValues since I may need to assign nonlocal<br>
> values to the vector. However, I find that on 2 procs, a general parallel vector works but<br>
> the DAVector indexing is not working. Do you see any problem with the DA code ?<br>
><br>
> Thanks.<br>
><br>
> Khalid<br>
><br>
> THIS WORKS: 30 is written at index 5<br>
> -------------------------------<br>
> ppy=5; ppx=30;<br>
> VecCreate(PETSC_COMM_WORLD,&testVec);<br>
> VecSetFromOptions(testVec);<br>
> VecSetSizes(testVec,PETSC_DECIDE,384);<br>
><br>
> VecSetValues(testVec,1,&ppy,&ppx,INSERT_VALUES);<br>
> VecAssemblyBegin(testVec);<br>
> VecAssemblyEnd(testVec);<br>
> =====================================<br>
><br>
> THIS DOES NOT WORK: puts 30 at index 9 instead of 5 when run on two procs.<br>
> ---------------------------------<br>
> ppy=5; ppx=30;<br>
> ierr = DACreate3d(PETSC_COMM_WORLD,DA_YPERIODIC,DA_STENCIL_BOX,8,8,6,<br>
> PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL,<br>
> &appctx.da);CHKERRQ(ierr);<br>
> ierr = DACreateGlobalVector(da,&testVec);CHKERRQ(ierr)<br>
> VecSetValues(testVec,1,&ppy,&ppx,INSERT_VALUES);<br>
> VecAssemblyBegin(testVec);<br>
> VecAssemblyEnd(testVec);<br>
><br>
</div></div><div class="HOEnZb"><div class="h5">> From: Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>><br>
> To: khalid ashraf <<a href="mailto:khalid_eee@yahoo.com">khalid_eee@yahoo.com</a>><br>
> Cc: PETSc users list <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br>
> Sent: Monday, April 2, 2012 3:08 PM<br>
> Subject: Re: [petsc-users] transfer vector data diagonally on DA<br>
><br>
> On Mon, Apr 2, 2012 at 4:53 PM, khalid ashraf <<a href="mailto:khalid_eee@yahoo.com">khalid_eee@yahoo.com</a>> wrote:<br>
> Thanks Matt. It works now.<br>
><br>
> I have another problem. I am writing this program.<br>
><br>
> 1) Always test on 2 procs once<br>
><br>
> 2) I have no idea what you actually want to do. However, it really simple to just print out<br>
> the indices you are using in row[] and the values in bodyFx[] and check that VecSetValues()<br>
> is working as you expect.<br>
><br>
> Matt<br>
><br>
><br>
> for (k=zs; k<zs+zm; k++) {<br>
> for (j=ys; j<ys+ym; j++) {<br>
> for (i=xs; i<xs+xm; i++) {<br>
> if ( i!=(mx-1) || j!=my-1 || k!=mz-1)<br>
> {<br>
> bodyFx[0]=1;<br>
> bodyFx[1]=-0.3;<br>
> bodyFx[2]=-0.3;<br>
> bodyFx[3]=-0.3;<br>
> row[0]=k*mx*my+j*mx+i;<br>
> row[1]=k*mx*my+j*mx+i+1;<br>
> row[2]=k*mx*my+(j+1)*mx+i;<br>
> row[3]=(k+1)*mx*my+j*mx+i;<br>
> VecSetValues(fx_test,4,row,bodyFx,ADD_VALUES);<br>
> }<br>
> VecAssemblyBegin(fx_test);<br>
> VecAssemblyEnd(fx_test);<br>
><br>
> Output: Print fx_test<br>
><br>
> Here fx_test is a global vector. This program gives correct output on 1 proc. but on 4 processors,<br>
> the program runs but the outputs are wrong.<br>
><br>
> I also tried<br>
> for (k=zs; k<zs+zm; k++) {<br>
> for (j=ys; j<ys+ym; j++) {<br>
> for (i=xs; i<xs+xm; i++) {<br>
> if ( i!=(mx-1) || j!=my-1 || k!=mz-1)<br>
> {<br>
> bodyFx[0]=1;<br>
> bodyFx[1]=-0.3;<br>
> bodyFx[2]=-0.3;<br>
> bodyFx[3]=-0.3;<br>
> fx1_localptr[k][j][i+1]+=bodyFx[4];<br>
> fx1_localptr[k][j+1][i]+=bodyFx[3];<br>
> fx1_localptr[k+1][j][i]+=bodyFx[1];<br>
> fx1_localptr[k][j][i]+=bodyFx[0];<br>
> }<br>
> Here, fx1_localptr is an array from a local vector that is communicated from the global vector fx_test.<br>
> In this case as well, I get similar errors.<br>
><br>
> Thanks.<br>
> Khalid<br>
><br>
> Some final numbers of the vector with 8*8*6 grid, 4 proc,<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> 1.0000000000000009e-01<br>
> 1.0000000000000009e-01<br>
> 1.0000000000000009e-01<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> 0.0000000000000000e+00<br>
> 4.0000000000000002e-01<br>
> 1.0000000000000009e-01<br>
> 1.0000000000000009e-01<br>
> 1.0000000000000009e-01<br>
> 0.0000000000000000e+00<br>
> 0.0000000000000000e+00<br>
> 0.0000000000000000e+00<br>
> 0.0000000000000000e+00<br>
> 1.0000000000000009e-01<br>
> 1.0000000000000009e-01<br>
> 1.0000000000000009e-01<br>
> -2.9999999999999999e-01<br>
> 0.0000000000000000e+00<br>
> 0.0000000000000000e+00<br>
> 0.0000000000000000e+00<br>
><br>
><br>
> Some final numbers of the vector with 8*8*6 grid, 1 proc,<br>
> 0.0000000000000000e+00<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> 0.0000000000000000e+00<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> 0.0000000000000000e+00<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> -2.9999999999999999e-01<br>
> 0.0000000000000000e+00<br>
> 0.0000000000000000e+00<br>
> 0.0000000000000000e+00<br>
> 0.0000000000000000e+00<br>
> 0.0000000000000000e+00<br>
> 0.0000000000000000e+00<br>
> 0.0000000000000000e+00<br>
> 0.0000000000000000e+00<br>
> 0.0000000000000000e+00<br>
><br>
><br>
><br>
</div></div><div class="HOEnZb"><div class="h5">> From: Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>><br>
> To: khalid ashraf <<a href="mailto:khalid_eee@yahoo.com">khalid_eee@yahoo.com</a>>; PETSc users list <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br>
> Sent: Monday, April 2, 2012 5:41 AM<br>
> Subject: Re: [petsc-users] transfer vector data diagonally on DA<br>
><br>
> On Mon, Apr 2, 2012 at 3:36 AM, khalid ashraf <<a href="mailto:khalid_eee@yahoo.com">khalid_eee@yahoo.com</a>> wrote:<br>
> Hi Jed,<br>
><br>
> I am using petsc/3.1 and the include file "petscdmda.h" is not working. I am using the "petscda.h"<br>
> and DACreate3D.<br>
> unew_localptr[][][] is from a global vector using VecGetArray<br>
> u_localptr[][][] is from a local vector communicated from a global vector.<br>
> I tried<br>
> unew_localptr[k][j][i]=u_localptr[k-1][j-1][i-1]<br>
> but it gives Segmentation Violation error.<br>
><br>
> I would guess (because you provide almost no information about what you are doing), that<br>
> this is a domain on the edge. If you truly mean to have a periodic domain, you must set that in the<br>
> creation call. Then the local vector will also ghost regions outside the domain boundary.<br>
><br>
> Matt<br>
><br>
> I also tried unew_localptr[k][j][i]=u_localptr[k][j+1][i+1] which works but<br>
> unew_localptr[k][j][i]=u_localptr[k+1][j+1][i+1] or unew_localptr[k][j][i]=u_localptr[k][j-1][i-1]<br>
> does not work. I am running the program on4 processors.<br>
><br>
> Thanks.<br>
><br>
><br>
> ierr = DACreate3d(PETSC_COMM_WORLD,DA_YPERIODIC,DA_STENCIL_BOX,appctx.l,appctx.m,appctx.n,<br>
> PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL,<br>
> &appctx.da);CHKERRQ(ierr);<br>
> for (k=zs; k<zs+zm; k++) {<br>
> for (j=ys; j<ys+ym; j++) {<br>
> for (i=xs; i<xs+xm; i++) {<br>
><br>
> if(i!=0 || j!=0 || k!=0|| i!=mx-1 || j!=my-1 || k!=mz-1)<br>
><br>
> unew_localptr[k][j][i]=u_localptr[k+1][j+1][i+1];<br>
> }}}<br>
</div></div><div class="HOEnZb"><div class="h5">> From: Jed Brown <<a href="mailto:jedbrown@mcs.anl.gov">jedbrown@mcs.anl.gov</a>><br>
> To: khalid ashraf <<a href="mailto:khalid_eee@yahoo.com">khalid_eee@yahoo.com</a>>; PETSc users list <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br>
> Sent: Sunday, April 1, 2012 10:07 PM<br>
> Subject: Re: [petsc-users] transfer vector data diagonally on DA<br>
><br>
> On Sun, Apr 1, 2012 at 22:01, khalid ashraf <<a href="mailto:khalid_eee@yahoo.com">khalid_eee@yahoo.com</a>> wrote:<br>
> I want to transfer vector data diagonally in the DA grid like<br>
> for (k=zs; k<zs+zm; k++) {<br>
> for (j=ys; j<ys+ym; j++) {<br>
> for (i=xs; i<xs+xm; i++) {<br>
> if(i!=mx-1 || j!=my-1 || k!=mz-1){<br>
> u_new[k+1][j+1][i+1]=u[k][j][i];}<br>
> }}}<br>
><br>
> Could you please suggest the best way to do it minimizing interprocessor assignments.<br>
><br>
> Both are on the same DMDA?<br>
><br>
> Communicate U to Ulocal (DMGlobalToLocalBegin/End) using a BOX stencil with width at least 1, get the global array u_new[][][] from UGlobalNew and the local arrays u[][][] from Ulocal, then assign u_new[k][j][i] = u[k-1][j-1][i-1].<br>
><br>
><br>
><br>
><br>
><br>
> --<br>
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> -- Norbert Wiener<br>
><br>
><br>
><br>
><br>
><br>
> --<br>
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> -- Norbert Wiener<br>
><br>
><br>
<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>