On Tue, Apr 19, 2011 at 6:26 PM, khalid ashraf <span dir="ltr"><<a href="mailto:khalid_eee@yahoo.com">khalid_eee@yahoo.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div><div style="font-family:arial, helvetica, sans-serif;font-size:12pt"><div><span style="font-size:13px">>>How much of a difference?</span></div><div style="font-family:arial, helvetica, sans-serif;font-size:12pt">
<div style="font-family:arial, helvetica, sans-serif;font-size:10pt">With the applied XYZPeriodic, if I keep the follwoing lines </div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt"><br></div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">
if (i==0 || j==0 || k==0 || i==mx-1 || j==my-1 || k==mz-1){</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt"> v[0] = 2.0*(HxHydHz + HxHzdHy + HyHzdHx);</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">
ierr =
MatSetValuesStencil(B,1,&row,1,&row,v,INSERT_VALUES);CHKERRQ(ierr);</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt"> } else </div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">
<br></div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt"> Then the error between 1proc and 4 procs is only after 5th decimal point. However, if I comment out the </div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">
above lines then the results are completely different for 1 and 4 procs.</div></div></div></div></blockquote><div><br></div><div>Yes, without this the problem is rank deficient. I suspect that your 5th decimal place difference</div>
<div>comes from either a) different partitions b) different convergence (stop on a different iterate) or c)</div><div>parallel reordering (but it seems big).</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div><div style="font-family:arial, helvetica, sans-serif;font-size:12pt"><div style="font-family:arial, helvetica, sans-serif;font-size:12pt"><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">I am attaching the output of some last data points of a 10X10X8 grid.</div>
<div style="font-family:arial, helvetica, sans-serif;font-size:10pt">1 proc output:</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt"><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">
-4.40214</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">-4.39202</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">-4.38693</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">
-4.38547</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">-4.38687</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">-4.39047</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">
<br></div><div>4 proc output:</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">0.000188031</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">0.000169784</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">
0.000157229</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">0.000178713</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">0.000179637</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">
0.000188031</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">0.000169784</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">0.000157229</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">
0.000178713</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">0.000179637</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt"><br></div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">
I am attaching the faulty code here for your review.</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt"><br></div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">Thanks.</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">
<br></div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">Khalid</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt"><br></div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">
<div style="font-family:arial, helvetica, sans-serif;font-size:10pt">static char help[] = "Solves 3D Laplacian using multigrid.\n\n";</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt"><br>
</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">#include "petscda.h"</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">#include "petscksp.h"</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">
#include "petscdmmg.h"</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">#include "myHeaderfile.h"</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt"><br>
</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">extern PetscErrorCode ComputeMatrix(DMMG,Mat,Mat);</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">extern PetscErrorCode ComputeRHS(DMMG,Vec);</div>
<div style="font-family:arial, helvetica, sans-serif;font-size:10pt"><br></div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">#undef __FUNCT__</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">
#define __FUNCT__ "main"</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">int main(int argc,char **argv)</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">{</div>
<div style="font-family:arial, helvetica, sans-serif;font-size:10pt"> PetscErrorCode ierr;</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt"> DMMG *dmmg;</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">
PetscReal norm;</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt"> DA da;</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt"><br></div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">
PetscInitialize(&argc,&argv,(char *)0,help);</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt"><br></div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt"> ierr = DMMGCreate(PETSC_COMM_WORLD,1,PETSC_NULL,&dmmg);CHKERRQ(ierr);</div>
<div style="font-family:arial, helvetica, sans-serif;font-size:10pt"> ierr = DACreate3d(PETSC_COMM_WORLD,DA_XYZPERIODIC,DA_STENCIL_STAR,10,10,8,PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,0,0,0,&da);CHKERRQ(ierr);</div>
<div style="font-family:arial, helvetica, sans-serif;font-size:10pt"> ierr = DMMGSetDM(dmmg,(DM)da);CHKERRQ(ierr);</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt"> // ierr = DADestroy(da);CHKERRQ(ierr);</div>
<div style="font-family:arial, helvetica, sans-serif;font-size:10pt"><br></div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt"> ierr = DMMGSetKSP(dmmg,ComputeRHS,ComputeMatrix);CHKERRQ(ierr);</div><div style="font-family:arial, helvetica, sans-serif;font-size:10pt">
ierr =DMMGSetNullSpace(dmmg,PETSC_TRUE,0,PETSC_NULL);</div><div><br></div><div><div> ierr = DMMGSetUp(dmmg);CHKERRQ(ierr);</div><div> ierr = DMMGSolve(dmmg);CHKERRQ(ierr);</div><div><br></div><div> ierr = MatMult(DMMGGetJ(dmmg),DMMGGetx(dmmg),DMMGGetr(dmmg));CHKERRQ(ierr);</div>
<div> ierr = VecAXPY(DMMGGetr(dmmg),-1.0,DMMGGetRHS(dmmg));CHKERRQ(ierr);</div><div> ierr =
VecNorm(DMMGGetr(dmmg),NORM_2,&norm);CHKERRQ(ierr);</div><div> /* ierr = PetscPrintf(PETSC_COMM_WORLD,"Residual norm %G\n",norm);CHKERRQ(ierr); */</div><div> ierr=VecView_VTK(DMMGGetx(dmmg),"X",&appctx);</div>
<div><br></div><div> ierr = DMMGDestroy(dmmg);CHKERRQ(ierr);</div><div> ierr = PetscFinalize();CHKERRQ(ierr);</div><div><br></div><div> return 0;</div><div>}</div><div><br></div><div>#undef __FUNCT__</div><div>#define __FUNCT__ "ComputeRHS"</div>
<div>PetscErrorCode ComputeRHS(DMMG dmmg,Vec b)</div><div>{</div><div> PetscErrorCode ierr;</div><div> PetscInt mx,my,mz;</div><div> PetscScalar h;</div><div><br></div><div> PetscFunctionBegin;</div><div> ierr = DAGetInfo((DA)dmmg->dm,0,&mx,&my,&mz,0,0,0,0,0,0,0);CHKERRQ(ierr);</div>
<div> h =
10.0/((mx-1)*(my-1)*(mz-1));</div><div> ierr = VecSet(b,h);CHKERRQ(ierr);</div><div> PetscFunctionReturn(0);</div><div>}</div><div><br></div><div><div>#undef __FUNCT__</div><div>#define __FUNCT__ "ComputeMatrix"</div>
<div>PetscErrorCode ComputeMatrix(DMMG dmmg,Mat jac,Mat B)</div><div>{</div><div> DA da = (DA)dmmg->dm;</div><div> PetscErrorCode ierr;</div><div> PetscInt i,j,k,mx,my,mz,xm,ym,zm,xs,ys,zs;</div><div>
PetscScalar v[7],Hx,Hy,Hz,HxHydHz,HyHzdHx,HxHzdHy;</div><div> MatStencil row,col[7];</div><div><br></div><div> ierr = DAGetInfo(da,0,&mx,&my,&mz,0,0,0,0,0,0,0);CHKERRQ(ierr);</div><div> Hx = 1.0 / (PetscReal)(mx-1); Hy = 1.0 / (PetscReal)(my-1); Hz = 1.0 / (PetscReal)(mz-1);</div>
<div> HxHydHz = Hx*Hy/Hz; HxHzdHy = Hx*Hz/Hy; HyHzdHx
= Hy*Hz/Hx;</div><div> ierr = DAGetCorners(da,&xs,&ys,&zs,&xm,&ym,&zm);CHKERRQ(ierr);</div><div><br></div><div> for (k=zs; k<zs+zm; k++){</div><div> for (j=ys; j<ys+ym; j++){</div><div>
for(i=xs; i<xs+xm; i++){</div><div> row.i = i; row.j = j; row.k = k;</div><div> /* if (i==0 || j==0 || k==0 || i==mx-1 || j==my-1 || k==mz-1){</div><div> v[0] = 2.0*(HxHydHz + HxHzdHy + HyHzdHx);</div>
<div> ierr = MatSetValuesStencil(B,1,&row,1,&row,v,INSERT_VALUES);CHKERRQ(ierr);</div><div> } else */</div><div> {</div><div> v[0] = -HxHydHz;col[0].i = i; col[0].j = j; col[0].k = k-1;</div>
<div>
v[1] = -HxHzdHy;col[1].i = i; col[1].j = j-1; col[1].k = k;</div><div> v[2] = -HyHzdHx;col[2].i = i-1; col[2].j = j; col[2].k = k;</div><div> v[3] = 2.0*(HxHydHz + HxHzdHy + HyHzdHx);col[3].i = row.i; col[3].j = row.j; col[3].k = row.k;</div>
<div> v[4] = -HyHzdHx;col[4].i = i+1; col[4].j = j; col[4].k = k;</div><div> v[5] = -HxHzdHy;col[5].i = i; col[5].j = j+1; col[5].k = k;</div><div> v[6] = -HxHydHz;col[6].i = i; col[6].j = j; col[6].k = k+1;</div>
<div><div> ierr = MatSetValuesStencil(B,1,&row,7,col,v,INSERT_VALUES);CHKERRQ(ierr);</div><div> }</div><div> }</div><div> }</div><div> }</div><div> ierr =
MatAssemblyBegin(B,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);</div><div> ierr = MatAssemblyEnd(B,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);</div><div> return 0;</div><div>}</div><div><br></div></div></div></div></div></div></div><div>
</div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>