<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div>
<div>Yes, I tested the implementation with both MatZeroRowsIS() and MatZeroRowsColumnsIS(). But first, I will be more explicit about the problem I was set to solve: </div>
<div><br>
</div>
<div>We have a Dirichlet block of size (L,W,H) and centered (xc,yc,zc), which is much smaller than the model domain, and we set Vx = Vpush, Vy=0 within the block (Vz is let free for easier convergence). </div>
<div>As I said before, since the code does not have a monolithic matrix, but 4 submatrices (VV VP; PV PP), and the rhs has 2 sub vectors rhs=(f; g), my approach is to modify only (VV, VP, f) for the Dirichlet BC.</div>
<div><br>
</div>
<div>The way I tested the implementation:</div>
<div>1) Output (VV, VP, f, Dirichlet dofs) - unmodified (no Dirichlet BC)</div>
<div>2) Output (VV, VP, f, Dirichlet dofs) - a) modified with MatZeroRowsIS(), </div>
<div><span class="Apple-tab-span" style="white-space:pre"></span> - b) modified with MatZeroRowsColumnsIS() -> S_PETSc</div>
<div><span class="Apple-tab-span" style="white-space:pre"></span>Again, the only difference between a) and b) is:</div>
<div><span class="Apple-tab-span" style="white-space:pre"></span><span class="Apple-style-span" style="color: rgb(80, 143, 115); font-family: Monaco; font-size: 11px; ">//</span><span class="Apple-style-span" style="color: rgb(80, 143, 115); font-family: Monaco; font-size: 11px; "><span class="Apple-tab-span" style="white-space:pre">
</span></span><span class="Apple-style-span" style="color: rgb(80, 143, 115); font-family: Monaco; font-size: 11px; "><span style="text-decoration: underline">ierr</span></span><span class="Apple-style-span" style="color: rgb(80, 143, 115); font-family: Monaco; font-size: 11px; ">
= MatZeroRowsColumnsIS(VV_MAT,</span><span class="Apple-style-span" style="color: rgb(80, 143, 115); font-family: Monaco; font-size: 11px; "><span style="text-decoration: underline">isx</span></span><span class="Apple-style-span" style="color: rgb(80, 143, 115); font-family: Monaco; font-size: 11px; ">,v_vv,x_push,</span><span class="Apple-style-span" style="color: rgb(80, 143, 115); font-family: Monaco; font-size: 11px; "><span style="text-decoration: underline">rhs</span></span><span class="Apple-style-span" style="color: rgb(80, 143, 115); font-family: Monaco; font-size: 11px; ">);
CHKERRQ(</span><span class="Apple-style-span" style="color: rgb(80, 143, 115); font-family: Monaco; font-size: 11px; "><span style="text-decoration: underline">ierr</span></span><span class="Apple-style-span" style="color: rgb(80, 143, 115); font-family: Monaco; font-size: 11px; ">);</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; color: rgb(80, 143, 115); ">
//<span class="Apple-tab-span" style="white-space:pre"> </span><span style="text-decoration: underline">ierr</span> = MatZeroRowsColumnsIS(VV_MAT,<span style="text-decoration: underline">isy</span>,v_vv,x_push,<span style="text-decoration: underline">rhs</span>);
CHKERRQ(<span style="text-decoration: underline">ierr</span>);</div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; min-height: 15px; ">
<br>
</div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; ">
<span class="Apple-tab-span" style="white-space:pre"></span>ierr = <span style="color: #774191">
MatZeroRowsIS</span>(VV_MAT,isx,v_vv,x_push,rhs); <span style="text-decoration: underline">
CHKERRQ(ierr)</span>;</div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; ">
<span class="Apple-tab-span" style="white-space:pre"></span>ierr = <span style="color: #774191">
MatZeroRowsIS</span>(VV_MAT,isy,v_vv,x_push,rhs); <span style="text-decoration: underline">
CHKERRQ(ierr)</span>;</div>
<div>3) Read them in Matlab and perform the exact same operations on the unmodified matrices and f vector. -> S_Matlab</div>
<div>4) Compare S_PETSc with S_Matlab. If the implementation is correct, they should be equal (VV, VP, f).</div>
<div>5) Check for 1 cpu and 4 cpus.</div>
<div><br>
</div>
<div>Now to answer your questions:</div>
<div><br>
</div>
a,b,d) Yes, matrix modification is done correctly (check the spy diagrams below) in all cases: MatZeroRowsIS() and MatZeroRowsColumnsIS() on 1 and 4 cpus.</div>
<div><br>
</div>
<div>I should have said that in the piece of code above:</div>
<div> v_vv = 1.0;<br>
<div>
<div> v_vp = 0.0;</div>
<div>The vector x_push is a duplicate of rhs, with zero elements except the values for the Dirichlet dofs. </div>
</div>
<div><br>
</div>
<div>c) The rhs is a different matter. With MatZeroRows() there is no problem. The rhs is equivalent with the one in Matlab, sequential and parallel.</div>
<div> However, with MatZeroRowsColumns(), the residual contains nonzero elements, and in parallel the nonzero pattern is even bigger (1 cpu - 63, 4 cpu - 554). But if you look carefully, the values of the nonzero residuals are very small < +/- 1e-10.</div>
<div>So, I did a tolerance filter:</div>
<div><br>
</div>
<div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Courier; ">
tol = 1e-10;</div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Courier; ">
res = f_petsc - f_mod_matlab;</div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Courier; ">
<span style="color: #0044fc">for</span> i=1:length(res)</div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Courier; ">
<span style="color: #0044fc">if</span> abs(res(i))>0 & abs(res(i))<tol</div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Courier; ">
res(i)=0;</div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Courier; ">
<span style="color: #0044fc">end</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Courier; color: rgb(0, 68, 252); ">
end</div>
</div>
<div><br>
</div>
<div>and then the f_petsc and f_mod_matlab are equivalent on 1 and 4 cpus (figure 5). So it seems that MatZeroRowsColumnsIS() might give some nonzero residuals. </div>
<div><br>
</div>
<div>Calculating the norm2 of the residuals defined above in each case gives:</div>
<div>MatZeroRowsIS() 1cpu: norm(res,2) = <span class="Apple-style-span" style="font-size: 12px; "> 0</span></div>
<div>MatZeroRowsIS() 4cpu: norm(res,2) = <span class="Apple-style-span" style="font-size: 12px; "> 0</span></div>
<div><span class="Apple-style-span" style="font-size: 12px; "><span class="Apple-style-span" style="font-size: medium; ">MatZeroRowsColumnsIS() 1cpu: norm(res,2) = </span></span><span class="Apple-style-span" style="font-size: 12px; "> 1.6880e-10</span></div>
<div><span class="Apple-style-span" style="font-size: 12px; "><span class="Apple-style-span" style="font-size: medium; ">MatZeroRowsColumnsIS() 4cpu: norm(res,2) = </span></span><span class="Apple-style-span" style="font-size: 12px; "> 7.3786e+06</span></div>
<div><br>
</div>
<div>Since this is purely a problem of matrix and vector assembly/manipulation, I think the nonzero residuals of the rhs with MatZeroRowsColumnsIS() give the parallel artefacts that I showed last time.</div>
<div>If you need the raw data and the matlab scripts that I used for testing for your consideration, please let me know. </div>
<div><br>
</div>
<div>Thanks,</div>
<div>Adina</div>
<div><br>
</div>
<div>When performing the manual operations on the unmodified matrices and rhs vector in Matlab, I took into account: </div>
<div>- matlab indexing = petsc indexing +1;</div>
<div>- the vectors written to file for matlab (<span class="Apple-style-span" style="color: rgb(0, 52, 201); font-family: Monaco; font-size: 11px; ">PETSC_VIEWER_BINARY_MATLAB</span>) have the natural ordering, rather than the petsc ordering. On 1 cpu, they
are equivalent, but on 4 cpus, the Dirichlet BC indices had to be converted to natural indices in order to perform the correct operations on the rhs.</div>
<div><br>
</div>
</div>
<div><img height="904" width="1051" apple-width="yes" apple-height="yes" id="44af5250-04e7-4b32-89de-5a856410bc35" src="cid:8547B616-9585-4E29-BC0D-D86348480A10@Geo.Uni-Mainz.DE"></div>
<div><img height="904" width="1051" apple-width="yes" apple-height="yes" id="71633dce-1992-4da3-99dc-4fda66023cfd" src="cid:A982EFF3-D817-404A-BCD3-E74CE9CB9EE4@Geo.Uni-Mainz.DE"></div>
<div><img height="904" width="1051" apple-width="yes" apple-height="yes" id="90e66391-9371-4d7a-b4cc-24d8c397bd4f" src="cid:ED674059-4A40-445D-8D43-FE23792E4B9F@Geo.Uni-Mainz.DE"></div>
<div><img height="904" width="1051" apple-width="yes" apple-height="yes" id="445542cf-53a3-45b5-9763-a0289cc6c43c" src="cid:0C431827-2AA7-47EF-A158-4004D3C0D9CE@Geo.Uni-Mainz.DE"></div>
<div><img height="397" width="823" apple-width="yes" apple-height="yes" id="84e2092a-fb82-45c3-bda7-2f421eb0f28d" src="cid:1C25FB42-709A-4D7B-BE52-94879AB5FAC7@Geo.Uni-Mainz.DE"></div>
<br>
<div>
<div>On May 6, 2014, at 4:22 PM, Matthew Knepley wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, May 6, 2014 at 7:23 AM, Püsök, Adina-Erika <span dir="ltr">
<<a href="mailto:puesoek@uni-mainz.de" target="_blank">puesoek@uni-mainz.de</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">Hello!
<div><br>
</div>
<div>I was trying to implement some internal Dirichlet boundary conditions into an aij matrix of the form: A=( VV VP; PV PP ). The idea was to create an internal block (let's say Dirichlet block) that moves with constant velocity within the domain (i.e.
check all the dofs within the block and set the values accordingly to the desired motion).</div>
<div><br>
</div>
<div>Ideally, this means to zero the rows and columns in VV, VP, PV corresponding to the dirichlet dofs and modify the corresponding rhs values. However, since we have submatrices and not a monolithic matrix A, we can choose to modify only VV and PV matrices. </div>
<div>The global indices of the velocity points within the Dirichlet block are contained in the arrays rowid_array. </div>
<div><br>
</div>
<div>What I want to point out is that the function MatZeroRowsColumnsIS() seems to create parallel artefacts, compared to MatZeroRowsIS() when run on more than 1 processor. Moreover, the results on 1 cpu are identical. </div>
<div>See below the results of the test (the Dirichlet block is outlined in white) and the piece of the code involved where the 1) - 2) parts are the only difference. </div>
</div>
</blockquote>
<div><br>
</div>
<div>I am assuming that you are showing the result of solving the equations. It would be more useful, and presumably just as easy</div>
<div>to say:</div>
<div><br>
</div>
<div> a) Are the correct rows zeroed out?</div>
<div><br>
</div>
<div> b) Is the diagonal element correct?</div>
<div><br>
</div>
<div> c) Is the rhs value correct?</div>
<div><br>
</div>
<div> d) Are the columns zeroed correctly?</div>
<div><br>
</div>
<div>If we know where the problem is, its easier to fix. For example, if the rhs values are</div>
<div>correct and the rows are zeroed, then something is wrong with the solution procedure.</div>
<div>Since ZeroRows() works and ZeroRowsColumns() does not, this is a distinct possibility.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> <br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>Thanks,</div>
<div>Adina Pusok</div>
<div><br>
</div>
<div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco;color:rgb(80,143,115)">
<span style="color:#000000"><span style="white-space:pre-wrap"></span></span>// Create an IS required by MatZeroRows()</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco">
<span style="white-space:pre-wrap"></span>ierr = <span style="color:#774191">ISCreateGeneral</span>(PETSC_COMM_WORLD,numRowsx,rowidx_array,<span style="color:#0034c9">PETSC_COPY_VALUES</span>,&isx);
<span style="text-decoration:underline">CHKERRQ(ierr)</span>;</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco">
<span style="white-space:pre-wrap"></span>ierr = <span style="color:#774191">ISCreateGeneral</span>(PETSC_COMM_WORLD,numRowsy,rowidy_array,<span style="color:#0034c9">PETSC_COPY_VALUES</span>,&isy);
<span style="text-decoration:underline">CHKERRQ(ierr)</span>;</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco">
<span style="white-space:pre-wrap"></span>ierr = <span style="color:#774191">ISCreateGeneral</span>(PETSC_COMM_WORLD,numRowsz,rowidz_array,<span style="color:#0034c9">PETSC_COPY_VALUES</span>,&isz);
<span style="text-decoration:underline">CHKERRQ(ierr)</span>;</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco;min-height:15px">
<br>
</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco;color:rgb(80,143,115)">
<span style="color:#000000">1)<span style="white-space:pre-wrap"> </span></span>/*<span style="white-space:pre-wrap">
</span><span style="text-decoration:underline">ierr</span> = MatZeroRowsColumnsIS(VV_MAT,<span style="text-decoration:underline">isx</span>,v_vv,x_push,<span style="text-decoration:underline">rhs</span>); CHKERRQ(<span style="text-decoration:underline">ierr</span>);</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco;color:rgb(80,143,115)">
<span style="white-space:pre-wrap"></span><span style="text-decoration:underline">ierr</span> = MatZeroRowsColumnsIS(VV_MAT,<span style="text-decoration:underline">isy</span>,v_vv,x_push,<span style="text-decoration:underline">rhs</span>); CHKERRQ(<span style="text-decoration:underline">ierr</span>);</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco;color:rgb(80,143,115)">
<span style="white-space:pre-wrap"></span><span style="text-decoration:underline">ierr</span> = MatZeroRowsColumnsIS(VV_MAT,<span style="text-decoration:underline">isz</span>,v_vv,x_push,<span style="text-decoration:underline">rhs</span>); CHKERRQ(<span style="text-decoration:underline">ierr</span>);*/</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco;min-height:15px">
<br>
</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco">
2)<span style="white-space:pre-wrap"> </span>ierr = <span style="color:#774191">MatZeroRowsIS</span>(VV_MAT,isx,v_vv,x_push,rhs);
<span style="text-decoration:underline">CHKERRQ(ierr)</span>;</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco">
<span style="white-space:pre-wrap"></span>ierr = <span style="color:#774191">MatZeroRowsIS</span>(VV_MAT,isy,v_vv,x_push,rhs);
<span style="text-decoration:underline">CHKERRQ(ierr)</span>;</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco">
<span style="white-space:pre-wrap"></span>ierr = <span style="color:#774191">MatZeroRowsIS</span>(VV_MAT,isz,v_vv,x_push,rhs);
<span style="text-decoration:underline">CHKERRQ(ierr)</span>;</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco;min-height:15px">
<br>
</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco">
<span style="white-space:pre-wrap"></span>ierr = <span style="color:#774191">MatZeroRowsIS</span>(VP_MAT,isx,v_vp,<span style="text-decoration:underline">PETSC_NULL</span>,<span style="text-decoration:underline">PETSC_NULL</span>);
<span style="text-decoration:underline">CHKERRQ(ierr)</span>;</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco">
<span style="white-space:pre-wrap"></span>ierr = <span style="color:#774191">MatZeroRowsIS</span>(VP_MAT,isy,v_vp,<span style="text-decoration:underline">PETSC_NULL</span>,<span style="text-decoration:underline">PETSC_NULL</span>);
<span style="text-decoration:underline">CHKERRQ(ierr)</span>;</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco">
<span style="white-space:pre-wrap"></span>ierr = <span style="color:#774191">MatZeroRowsIS</span>(VP_MAT,isz,v_vp,<span style="text-decoration:underline">PETSC_NULL</span>,<span style="text-decoration:underline">PETSC_NULL</span>);
<span style="text-decoration:underline">CHKERRQ(ierr)</span>;</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco;min-height:15px">
<br>
</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco">
<span style="white-space:pre-wrap"></span>ierr = <span style="color:#774191">ISDestroy</span>(&isx);
<span style="text-decoration:underline">CHKERRQ(ierr)</span>;</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco">
<span style="white-space:pre-wrap"></span>ierr = <span style="color:#774191">ISDestroy</span>(&isy);
<span style="text-decoration:underline">CHKERRQ(ierr)</span>;</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco">
<span style="white-space:pre-wrap"></span>ierr = <span style="color:#774191">ISDestroy</span>(&isz);
<span style="text-decoration:underline">CHKERRQ(ierr)</span>;</div>
</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco">
<br>
</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco">
<br>
</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco">
Results (velocity) with <span style="font-family:Helvetica;font-size:medium">MatZeroRowsColumnsIS().</span></div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco">
<span style="font-family:Helvetica;font-size:medium">1cpu<span><r01_1cpu_rows_columns.png></span> </span>4cpu<span><r01_rows_columns.png></span></div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco">
<br>
</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco">
Results (velocity)<span style="font-family:Monaco;font-size:11px"> </span>with <span style="font-family:Helvetica;font-size:medium">MatZeroRowsIS():</span></div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font:normal normal normal 11px/normal Monaco">
<span style="font-family:Helvetica;font-size:medium">1cpu<span><r01_1cpu_rows.png></span> 4cpu</span><span><r01_rows.png></span></div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener </div>
</div>
</blockquote>
</div>
<br>
</body>
</html>