<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><br></div><br><div><div>On Jun 17, 2010, at 11:44 AM, Matthew Knepley wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">On Thu, Jun 17, 2010 at 11:27 AM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div style="word-wrap:break-word"><div><br></div> Matt,<div><br></div><div> Jed is not on petsc-maint. He is suppose to writing his thesis :-)</div></div></blockquote><div><br></div><div>Cool, I only asked because I knew he did stuff on MatGetSubMatrix and its interaction</div>
<div>with FS.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div style="word-wrap:break-word"><div> MatGetSubMatrices() was written for doing ASM, hence it only works in that limited way. </div>
</div></blockquote><div><br></div><div>Can't this be fixed by making these submatrices have blocksize one by default, specified from</div><div>FS? That would leave the ASM behavior intact. This moves MatGetSubmatrices() more in the</div>
<div>direction of MatLoad() where you can pass in constructed Mats if you want.</div></div></blockquote><div><br></div> Yes, eventually we should provide more general support for the format of the submatrices. It is just ALL the logic for MatGetSubMatrix.... for BAIJ matrices (which is pretty nasty complicate) is built around always get all of a block, so a good amount of complex code needs to be written to provide the general support; won't happen today.</div><div><br></div><div> Barry</div><div><br><blockquote type="cite"><div class="gmail_quote"><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div style="word-wrap:break-word"><div> This guy should just be using AIJ matrices, he get's no advantage from using BAIJ anyways.</div><div><br></div><font color="#888888"><div> Barry</div></font><div><div></div>
<div class="h5"><div><br><div><div>On Jun 17, 2010, at 9:47 AM, Matthew Knepley wrote:</div><br><blockquote type="cite">Barry and Jed,<br><br> I tried to replicate this, and the problem seems to be with<br>MatGetSubMatrix() and BAIJ<br>
matrices. It is assumed that the submatrix has the same block size, which<br>will not be<br>true for FieldSplit. How does this not screw up for the DA version?<br><br> Matt<br><br>On Thu, Jun 17, 2010 at 8:14 AM, Laurent Michel <<a href="mailto:laurent.michel@epfl.ch" target="_blank">laurent.michel@epfl.ch</a>>wrote:<br>
<br><blockquote type="cite">Hi,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Matthew Knepley wrote:<br></blockquote><blockquote type="cite"><blockquote type="cite">On Thu, Jun 17, 2010 at 5:43 AM, Laurent Michel<br>
</blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><<a href="mailto:laurent.michel@epfl.ch" target="_blank">laurent.michel@epfl.ch</a> <<a href="mailto:laurent.michel@epfl.ch" target="_blank">mailto:laurent.michel@epfl.ch</a>>> wrote:<br>
</blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> Hi again,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">
<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> one more thing I am a bit worried about. Following your advice, I am<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">
intending to use the fieldsplit preconditioner in my code. I have<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> browsed the web a bit and I have not been able to find much<br></blockquote>
</blockquote><blockquote type="cite"><blockquote type="cite"> information<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> about it. Even your documentation is not that clear to me, hence my<br>
</blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> question.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">
I am solving a Stokes problem in 3d (P1-P1 stabilized elements).<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> Originally, the matrix of the linear system is ordered in the<br></blockquote>
</blockquote><blockquote type="cite"><blockquote type="cite"> following<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> basis (for local elements):<br></blockquote></blockquote><blockquote type="cite">
<blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> {(phi0, 0, 0), ..., (phi3, 0, 0), (0, phi0, 0), ..., (0, phi3, 0),<br></blockquote></blockquote><blockquote type="cite">
(0,<br></blockquote><blockquote type="cite"><blockquote type="cite"> 0, phi0), ..., (0, 0, phi3), phi0, phi1, phi2, phi3}<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote>
<blockquote type="cite"><blockquote type="cite"> I am using the MATMPIBAIJ matrix format, and I have chosen a<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> matrix size<br></blockquote>
</blockquote><blockquote type="cite"><blockquote type="cite"> of 4, because I have 4 degrees of freedom (following your<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> documentation). Anyway, if I wanted to choose a block size of 16<br>
</blockquote></blockquote><blockquote type="cite">(what<br></blockquote><blockquote type="cite"><blockquote type="cite"> would've been a natural choice for me), it wouldn't work, because,<br></blockquote></blockquote>
<blockquote type="cite"><blockquote type="cite"> if I<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> have well understood, the number of rows must be divisible by the<br></blockquote></blockquote>
<blockquote type="cite"><blockquote type="cite"> block<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> size.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">
<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Let's start here. There is some misunderstanding. I cannot understand<br>
</blockquote></blockquote><blockquote type="cite"><blockquote type="cite">what you<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">mean by degree of freedom, or we might say "unknown". In P1-P1, on I am<br>
</blockquote></blockquote><blockquote type="cite"><blockquote type="cite">assuming a hex grid, you have 4 unknowns per vertex, and thus 16 unknowns<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">
per element. You would use a blocksize of 4 (indicating a vertex with<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">u,v,w,p).<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">
Then you could split apart the fields if you wanted.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> Matt<br></blockquote>
</blockquote><blockquote type="cite">Ok. My degrees of freedom are as you say, a 3d velocity (vx, vy, vz) and<br></blockquote><blockquote type="cite">a pressure (p). I have an unstructured mesh with<br></blockquote><blockquote type="cite">
<br></blockquote><blockquote type="cite">337983 elements<br></blockquote><blockquote type="cite">63686 nodes<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I have penalised the velocity where it was set to 0, so that I can use<br>
</blockquote><blockquote type="cite">MPIBAIJ. My system consists of a matrix of<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">254744 = 4*63686 rows<br></blockquote><blockquote type="cite">
14490112 non-zeros<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">The maximal number of neighbours of a node is 116.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">
Given that matrix, created with<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">MatCreateMPIBAIJ(4,<br></blockquote><blockquote type="cite"> PETSC_DECIDE,<br></blockquote><blockquote type="cite">
PETSC_DECIDE,<br></blockquote><blockquote type="cite"> 254744<br></blockquote><blockquote type="cite"> 254744<br></blockquote><blockquote type="cite"> 116,<br>
</blockquote><blockquote type="cite"> PETSC_DECIDE,<br></blockquote><blockquote type="cite"> &A);<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">if I now call my program with options (on any number of procs, but below<br>
</blockquote><blockquote type="cite">with 8 procs)<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">-pc_type fieldsplit -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3<br></blockquote>
<blockquote type="cite"><br></blockquote><blockquote type="cite">I get the error (when the script is calling KSPSolve()):<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">[7]PETSC ERROR: --------------------- Error Message<br>
</blockquote><blockquote type="cite">------------------------------------<br></blockquote><blockquote type="cite">[7]PETSC ERROR: Nonconforming object sizes!<br></blockquote><blockquote type="cite">[7]PETSC ERROR: Local column sizes 47760 do not add up to total number<br>
</blockquote><blockquote type="cite">of columns 63686!<br></blockquote><blockquote type="cite">[7]PETSC ERROR:<br></blockquote><blockquote type="cite">------------------------------------------------------------------------<br>
</blockquote><blockquote type="cite">[7]PETSC ERROR: Petsc Release Version 3.1.0, Patch 1, Thu Apr 8<br></blockquote><blockquote type="cite">14:16:50 CDT 2010<br></blockquote><blockquote type="cite">[7]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
</blockquote><blockquote type="cite">[7]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br></blockquote><blockquote type="cite">[7]PETSC ERROR: See docs/index.html for manual pages.<br></blockquote><blockquote type="cite">
[7]PETSC ERROR:<br></blockquote><blockquote type="cite">------------------------------------------------------------------------<br></blockquote><blockquote type="cite">[7]PETSC ERROR: stokes-5.3b01 on a linux-gnu named iacspc3 by lmichel<br>
</blockquote><blockquote type="cite">Thu Jun 17 15:05:51 2010<br></blockquote><blockquote type="cite">[7]PETSC ERROR: Libraries linked from<br></blockquote><blockquote type="cite">/data/progr/petsc-3.1-openmpi-1.4.2/lib<br>
</blockquote><blockquote type="cite">[7]PETSC ERROR: Configure run at Sat May 29 14:01:23 2010<br></blockquote><blockquote type="cite">[7]PETSC ERROR: Configure options<br></blockquote><blockquote type="cite">--prefix=/data/progr/petsc-3.1-openmpi-1.4.2<br>
</blockquote><blockquote type="cite">--PETSC_ARCH=linux-gnu-cxx-openmpi-1.4.2-opt --with-clanguage=C++<br></blockquote><blockquote type="cite">--with-debugging=0 --with-blas-lib=-lblas --with-lapack-lib=-llapack<br></blockquote>
<blockquote type="cite">--with-shared --with-mpi-dir=/data/progr/openmpi-1.4.2<br></blockquote><blockquote type="cite">--download-umfpack=1 --with-umfpack=1 --download-parmetis=1<br></blockquote><blockquote type="cite">--with-parmetis=1 --download-prometheus=1 --with-prometheus=1<br>
</blockquote><blockquote type="cite">[7]PETSC ERROR:<br></blockquote><blockquote type="cite">------------------------------------------------------------------------<br></blockquote><blockquote type="cite">[7]PETSC ERROR: MatGetSubMatrix_MPIBAIJ() line 1927 in<br>
</blockquote><blockquote type="cite">src/mat/impls/baij/mpi/mpibaij.c<br></blockquote><blockquote type="cite">[7]PETSC ERROR: MatGetSubMatrix_MPIBAIJ() line 1851 in<br></blockquote><blockquote type="cite">src/mat/impls/baij/mpi/mpibaij.c<br>
</blockquote><blockquote type="cite">[7]PETSC ERROR: MatGetSubMatrix() line 6425 in src/mat/interface/matrix.c<br></blockquote><blockquote type="cite">[7]PETSC ERROR: PCSetUp_FieldSplit() line 296 in<br></blockquote><blockquote type="cite">
src/ksp/pc/impls/fieldsplit/fieldsplit.c<br></blockquote><blockquote type="cite">[7]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c<br></blockquote><blockquote type="cite">[7]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c<br>
</blockquote><blockquote type="cite">[7]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">thanks for your help,<br></blockquote>
<blockquote type="cite"><br></blockquote><blockquote type="cite">L.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">--<br></blockquote><blockquote type="cite">Laurent Michel<br></blockquote>
<blockquote type="cite">PhD candidate<br></blockquote><blockquote type="cite">EPFL SB IACS ASN<br></blockquote><blockquote type="cite">MA C2 644<br></blockquote><blockquote type="cite">+41 21 693 42 46<br></blockquote><blockquote type="cite">
+41 77 433 38 94<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><br><br>-- <br>What most experimenters take for granted before they begin their experiments<br>
is infinitely more interesting than any results to which their experiments<br>lead.<br>-- Norbert Wiener<br><br>Barry and Jed,<div><br></div><div> I tried to replicate this, and the problem seems to be with MatGetSubMatrix() and BAIJ</div>
<div>matrices. It is assumed that the submatrix has the same block size, which will not be</div>
<div>true for FieldSplit. How does this not screw up for the DA version?</div><div><br></div><div> Matt<br><div><br><div class="gmail_quote">On Thu, Jun 17, 2010 at 8:14 AM, Laurent Michel <span dir="ltr"><<a href="mailto:laurent.michel@epfl.ch" target="_blank">laurent.michel@epfl.ch</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<div><br>
Matthew Knepley wrote:<br>
> On Thu, Jun 17, 2010 at 5:43 AM, Laurent Michel<br>
</div><div><div></div><div>> <<a href="mailto:laurent.michel@epfl.ch" target="_blank">laurent.michel@epfl.ch</a> <mailto:<a href="mailto:laurent.michel@epfl.ch" target="_blank">laurent.michel@epfl.ch</a>>> wrote:<br>
><br>
> Hi again,<br>
><br>
> one more thing I am a bit worried about. Following your advice, I am<br>
> intending to use the fieldsplit preconditioner in my code. I have<br>
> browsed the web a bit and I have not been able to find much<br>
> information<br>
> about it. Even your documentation is not that clear to me, hence my<br>
> question.<br>
><br>
> I am solving a Stokes problem in 3d (P1-P1 stabilized elements).<br>
> Originally, the matrix of the linear system is ordered in the<br>
> following<br>
> basis (for local elements):<br>
><br>
> {(phi0, 0, 0), ..., (phi3, 0, 0), (0, phi0, 0), ..., (0, phi3, 0), (0,<br>
> 0, phi0), ..., (0, 0, phi3), phi0, phi1, phi2, phi3}<br>
><br>
> I am using the MATMPIBAIJ matrix format, and I have chosen a<br>
> matrix size<br>
> of 4, because I have 4 degrees of freedom (following your<br>
> documentation). Anyway, if I wanted to choose a block size of 16 (what<br>
> would've been a natural choice for me), it wouldn't work, because,<br>
> if I<br>
> have well understood, the number of rows must be divisible by the<br>
> block<br>
> size.<br>
><br>
><br>
> Let's start here. There is some misunderstanding. I cannot understand<br>
> what you<br>
> mean by degree of freedom, or we might say "unknown". In P1-P1, on I am<br>
> assuming a hex grid, you have 4 unknowns per vertex, and thus 16 unknowns<br>
> per element. You would use a blocksize of 4 (indicating a vertex with<br>
> u,v,w,p).<br>
> Then you could split apart the fields if you wanted.<br>
><br>
> Matt<br>
</div></div>Ok. My degrees of freedom are as you say, a 3d velocity (vx, vy, vz) and<br>
a pressure (p). I have an unstructured mesh with<br>
<br>
337983 elements<br>
63686 nodes<br>
<br>
I have penalised the velocity where it was set to 0, so that I can use<br>
MPIBAIJ. My system consists of a matrix of<br>
<br>
254744 = 4*63686 rows<br>
14490112 non-zeros<br>
<br>
The maximal number of neighbours of a node is 116.<br>
<br>
Given that matrix, created with<br>
<br>
MatCreateMPIBAIJ(4,<br>
PETSC_DECIDE,<br>
PETSC_DECIDE,<br>
254744<br>
254744<br>
116,<br>
PETSC_DECIDE,<br>
&A);<br>
<br>
if I now call my program with options (on any number of procs, but below<br>
with 8 procs)<br>
<br>
-pc_type fieldsplit -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3<br>
<br>
I get the error (when the script is calling KSPSolve()):<br>
<br>
[7]PETSC ERROR: --------------------- Error Message<br>
------------------------------------<br>
[7]PETSC ERROR: Nonconforming object sizes!<br>
[7]PETSC ERROR: Local column sizes 47760 do not add up to total number<br>
of columns 63686!<br>
[7]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[7]PETSC ERROR: Petsc Release Version 3.1.0, Patch 1, Thu Apr 8<br>
14:16:50 CDT 2010<br>
[7]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
[7]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
[7]PETSC ERROR: See docs/index.html for manual pages.<br>
[7]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[7]PETSC ERROR: stokes-5.3b01 on a linux-gnu named iacspc3 by lmichel<br>
Thu Jun 17 15:05:51 2010<br>
[7]PETSC ERROR: Libraries linked from<br>
/data/progr/petsc-3.1-openmpi-1.4.2/lib<br>
[7]PETSC ERROR: Configure run at Sat May 29 14:01:23 2010<br>
[7]PETSC ERROR: Configure options<br>
--prefix=/data/progr/petsc-3.1-openmpi-1.4.2<br>
--PETSC_ARCH=linux-gnu-cxx-openmpi-1.4.2-opt --with-clanguage=C++<br>
--with-debugging=0 --with-blas-lib=-lblas --with-lapack-lib=-llapack<br>
--with-shared --with-mpi-dir=/data/progr/openmpi-1.4.2<br>
--download-umfpack=1 --with-umfpack=1 --download-parmetis=1<br>
--with-parmetis=1 --download-prometheus=1 --with-prometheus=1<br>
[7]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[7]PETSC ERROR: MatGetSubMatrix_MPIBAIJ() line 1927 in<br>
src/mat/impls/baij/mpi/mpibaij.c<br>
[7]PETSC ERROR: MatGetSubMatrix_MPIBAIJ() line 1851 in<br>
src/mat/impls/baij/mpi/mpibaij.c<br>
[7]PETSC ERROR: MatGetSubMatrix() line 6425 in src/mat/interface/matrix.c<br>
[7]PETSC ERROR: PCSetUp_FieldSplit() line 296 in<br>
src/ksp/pc/impls/fieldsplit/fieldsplit.c<br>
[7]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c<br>
[7]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c<br>
[7]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c<br>
<br>
thanks for your help,<br>
<div><div></div><div><br>
L.<br>
<br>
--<br>
Laurent Michel<br>
PhD candidate<br>
EPFL SB IACS ASN<br>
MA C2 644<br>
+41 21 693 42 46<br>
+41 77 433 38 94<br>
<br>
<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>
</div></div>
</blockquote></div><br></div></div></div></div></blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>
</blockquote></div><br></body></html>