On Thu, Jun 17, 2010 at 11:27 AM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div style="word-wrap:break-word"><div><br></div> Matt,<div><br></div><div> Jed is not on petsc-maint. He is suppose to writing his thesis :-)</div></div></blockquote><div><br></div><div>Cool, I only asked because I knew he did stuff on MatGetSubMatrix and its interaction</div>
<div>with FS.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div style="word-wrap:break-word"><div> MatGetSubMatrices() was written for doing ASM, hence it only works in that limited way. </div>
</div></blockquote><div><br></div><div>Can't this be fixed by making these submatrices have blocksize one by default, specified from</div><div>FS? That would leave the ASM behavior intact. This moves MatGetSubmatrices() more in the</div>
<div>direction of MatLoad() where you can pass in constructed Mats if you want.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div style="word-wrap:break-word"><div> This guy should just be using AIJ matrices, he get's no advantage from using BAIJ anyways.</div><div><br></div><font color="#888888"><div> Barry</div></font><div><div></div>
<div class="h5"><div><br><div><div>On Jun 17, 2010, at 9:47 AM, Matthew Knepley wrote:</div><br><blockquote type="cite">Barry and Jed,<br><br> I tried to replicate this, and the problem seems to be with<br>MatGetSubMatrix() and BAIJ<br>
matrices. It is assumed that the submatrix has the same block size, which<br>will not be<br>true for FieldSplit. How does this not screw up for the DA version?<br><br> Matt<br><br>On Thu, Jun 17, 2010 at 8:14 AM, Laurent Michel <<a href="mailto:laurent.michel@epfl.ch" target="_blank">laurent.michel@epfl.ch</a>>wrote:<br>
<br><blockquote type="cite">Hi,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Matthew Knepley wrote:<br></blockquote><blockquote type="cite"><blockquote type="cite">On Thu, Jun 17, 2010 at 5:43 AM, Laurent Michel<br>
</blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><<a href="mailto:laurent.michel@epfl.ch" target="_blank">laurent.michel@epfl.ch</a> <<a href="mailto:laurent.michel@epfl.ch" target="_blank">mailto:laurent.michel@epfl.ch</a>>> wrote:<br>
</blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> Hi again,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">
<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> one more thing I am a bit worried about. Following your advice, I am<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">
intending to use the fieldsplit preconditioner in my code. I have<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> browsed the web a bit and I have not been able to find much<br></blockquote>
</blockquote><blockquote type="cite"><blockquote type="cite"> information<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> about it. Even your documentation is not that clear to me, hence my<br>
</blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> question.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">
I am solving a Stokes problem in 3d (P1-P1 stabilized elements).<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> Originally, the matrix of the linear system is ordered in the<br></blockquote>
</blockquote><blockquote type="cite"><blockquote type="cite"> following<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> basis (for local elements):<br></blockquote></blockquote><blockquote type="cite">
<blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> {(phi0, 0, 0), ..., (phi3, 0, 0), (0, phi0, 0), ..., (0, phi3, 0),<br></blockquote></blockquote><blockquote type="cite">
(0,<br></blockquote><blockquote type="cite"><blockquote type="cite"> 0, phi0), ..., (0, 0, phi3), phi0, phi1, phi2, phi3}<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote>
<blockquote type="cite"><blockquote type="cite"> I am using the MATMPIBAIJ matrix format, and I have chosen a<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> matrix size<br></blockquote>
</blockquote><blockquote type="cite"><blockquote type="cite"> of 4, because I have 4 degrees of freedom (following your<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> documentation). Anyway, if I wanted to choose a block size of 16<br>
</blockquote></blockquote><blockquote type="cite">(what<br></blockquote><blockquote type="cite"><blockquote type="cite"> would've been a natural choice for me), it wouldn't work, because,<br></blockquote></blockquote>
<blockquote type="cite"><blockquote type="cite"> if I<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> have well understood, the number of rows must be divisible by the<br></blockquote></blockquote>
<blockquote type="cite"><blockquote type="cite"> block<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> size.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">
<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Let's start here. There is some misunderstanding. I cannot understand<br>
</blockquote></blockquote><blockquote type="cite"><blockquote type="cite">what you<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">mean by degree of freedom, or we might say "unknown". In P1-P1, on I am<br>
</blockquote></blockquote><blockquote type="cite"><blockquote type="cite">assuming a hex grid, you have 4 unknowns per vertex, and thus 16 unknowns<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">
per element. You would use a blocksize of 4 (indicating a vertex with<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">u,v,w,p).<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">
Then you could split apart the fields if you wanted.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> Matt<br></blockquote>
</blockquote><blockquote type="cite">Ok. My degrees of freedom are as you say, a 3d velocity (vx, vy, vz) and<br></blockquote><blockquote type="cite">a pressure (p). I have an unstructured mesh with<br></blockquote><blockquote type="cite">
<br></blockquote><blockquote type="cite">337983 elements<br></blockquote><blockquote type="cite">63686 nodes<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I have penalised the velocity where it was set to 0, so that I can use<br>
</blockquote><blockquote type="cite">MPIBAIJ. My system consists of a matrix of<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">254744 = 4*63686 rows<br></blockquote><blockquote type="cite">
14490112 non-zeros<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">The maximal number of neighbours of a node is 116.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">
Given that matrix, created with<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">MatCreateMPIBAIJ(4,<br></blockquote><blockquote type="cite"> PETSC_DECIDE,<br></blockquote><blockquote type="cite">
PETSC_DECIDE,<br></blockquote><blockquote type="cite"> 254744<br></blockquote><blockquote type="cite"> 254744<br></blockquote><blockquote type="cite"> 116,<br>
</blockquote><blockquote type="cite"> PETSC_DECIDE,<br></blockquote><blockquote type="cite"> &A);<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">if I now call my program with options (on any number of procs, but below<br>
</blockquote><blockquote type="cite">with 8 procs)<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">-pc_type fieldsplit -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3<br></blockquote>
<blockquote type="cite"><br></blockquote><blockquote type="cite">I get the error (when the script is calling KSPSolve()):<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">[7]PETSC ERROR: --------------------- Error Message<br>
</blockquote><blockquote type="cite">------------------------------------<br></blockquote><blockquote type="cite">[7]PETSC ERROR: Nonconforming object sizes!<br></blockquote><blockquote type="cite">[7]PETSC ERROR: Local column sizes 47760 do not add up to total number<br>
</blockquote><blockquote type="cite">of columns 63686!<br></blockquote><blockquote type="cite">[7]PETSC ERROR:<br></blockquote><blockquote type="cite">------------------------------------------------------------------------<br>
</blockquote><blockquote type="cite">[7]PETSC ERROR: Petsc Release Version 3.1.0, Patch 1, Thu Apr 8<br></blockquote><blockquote type="cite">14:16:50 CDT 2010<br></blockquote><blockquote type="cite">[7]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
</blockquote><blockquote type="cite">[7]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br></blockquote><blockquote type="cite">[7]PETSC ERROR: See docs/index.html for manual pages.<br></blockquote><blockquote type="cite">
[7]PETSC ERROR:<br></blockquote><blockquote type="cite">------------------------------------------------------------------------<br></blockquote><blockquote type="cite">[7]PETSC ERROR: stokes-5.3b01 on a linux-gnu named iacspc3 by lmichel<br>
</blockquote><blockquote type="cite">Thu Jun 17 15:05:51 2010<br></blockquote><blockquote type="cite">[7]PETSC ERROR: Libraries linked from<br></blockquote><blockquote type="cite">/data/progr/petsc-3.1-openmpi-1.4.2/lib<br>
</blockquote><blockquote type="cite">[7]PETSC ERROR: Configure run at Sat May 29 14:01:23 2010<br></blockquote><blockquote type="cite">[7]PETSC ERROR: Configure options<br></blockquote><blockquote type="cite">--prefix=/data/progr/petsc-3.1-openmpi-1.4.2<br>
</blockquote><blockquote type="cite">--PETSC_ARCH=linux-gnu-cxx-openmpi-1.4.2-opt --with-clanguage=C++<br></blockquote><blockquote type="cite">--with-debugging=0 --with-blas-lib=-lblas --with-lapack-lib=-llapack<br></blockquote>
<blockquote type="cite">--with-shared --with-mpi-dir=/data/progr/openmpi-1.4.2<br></blockquote><blockquote type="cite">--download-umfpack=1 --with-umfpack=1 --download-parmetis=1<br></blockquote><blockquote type="cite">--with-parmetis=1 --download-prometheus=1 --with-prometheus=1<br>
</blockquote><blockquote type="cite">[7]PETSC ERROR:<br></blockquote><blockquote type="cite">------------------------------------------------------------------------<br></blockquote><blockquote type="cite">[7]PETSC ERROR: MatGetSubMatrix_MPIBAIJ() line 1927 in<br>
</blockquote><blockquote type="cite">src/mat/impls/baij/mpi/mpibaij.c<br></blockquote><blockquote type="cite">[7]PETSC ERROR: MatGetSubMatrix_MPIBAIJ() line 1851 in<br></blockquote><blockquote type="cite">src/mat/impls/baij/mpi/mpibaij.c<br>
</blockquote><blockquote type="cite">[7]PETSC ERROR: MatGetSubMatrix() line 6425 in src/mat/interface/matrix.c<br></blockquote><blockquote type="cite">[7]PETSC ERROR: PCSetUp_FieldSplit() line 296 in<br></blockquote><blockquote type="cite">
src/ksp/pc/impls/fieldsplit/fieldsplit.c<br></blockquote><blockquote type="cite">[7]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c<br></blockquote><blockquote type="cite">[7]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c<br>
</blockquote><blockquote type="cite">[7]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">thanks for your help,<br></blockquote>
<blockquote type="cite"><br></blockquote><blockquote type="cite">L.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">--<br></blockquote><blockquote type="cite">Laurent Michel<br></blockquote>
<blockquote type="cite">PhD candidate<br></blockquote><blockquote type="cite">EPFL SB IACS ASN<br></blockquote><blockquote type="cite">MA C2 644<br></blockquote><blockquote type="cite">+41 21 693 42 46<br></blockquote><blockquote type="cite">
+41 77 433 38 94<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><br><br>-- <br>What most experimenters take for granted before they begin their experiments<br>
is infinitely more interesting than any results to which their experiments<br>lead.<br>-- Norbert Wiener<br><br>Barry and Jed,<div><br></div><div> I tried to replicate this, and the problem seems to be with MatGetSubMatrix() and BAIJ</div>
<div>matrices. It is assumed that the submatrix has the same block size, which will not be</div>
<div>true for FieldSplit. How does this not screw up for the DA version?</div><div><br></div><div> Matt<br><div><br><div class="gmail_quote">On Thu, Jun 17, 2010 at 8:14 AM, Laurent Michel <span dir="ltr"><<a href="mailto:laurent.michel@epfl.ch" target="_blank">laurent.michel@epfl.ch</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<div><br>
Matthew Knepley wrote:<br>
> On Thu, Jun 17, 2010 at 5:43 AM, Laurent Michel<br>
</div><div><div></div><div>> <<a href="mailto:laurent.michel@epfl.ch" target="_blank">laurent.michel@epfl.ch</a> <mailto:<a href="mailto:laurent.michel@epfl.ch" target="_blank">laurent.michel@epfl.ch</a>>> wrote:<br>
><br>
> Hi again,<br>
><br>
> one more thing I am a bit worried about. Following your advice, I am<br>
> intending to use the fieldsplit preconditioner in my code. I have<br>
> browsed the web a bit and I have not been able to find much<br>
> information<br>
> about it. Even your documentation is not that clear to me, hence my<br>
> question.<br>
><br>
> I am solving a Stokes problem in 3d (P1-P1 stabilized elements).<br>
> Originally, the matrix of the linear system is ordered in the<br>
> following<br>
> basis (for local elements):<br>
><br>
> {(phi0, 0, 0), ..., (phi3, 0, 0), (0, phi0, 0), ..., (0, phi3, 0), (0,<br>
> 0, phi0), ..., (0, 0, phi3), phi0, phi1, phi2, phi3}<br>
><br>
> I am using the MATMPIBAIJ matrix format, and I have chosen a<br>
> matrix size<br>
> of 4, because I have 4 degrees of freedom (following your<br>
> documentation). Anyway, if I wanted to choose a block size of 16 (what<br>
> would've been a natural choice for me), it wouldn't work, because,<br>
> if I<br>
> have well understood, the number of rows must be divisible by the<br>
> block<br>
> size.<br>
><br>
><br>
> Let's start here. There is some misunderstanding. I cannot understand<br>
> what you<br>
> mean by degree of freedom, or we might say "unknown". In P1-P1, on I am<br>
> assuming a hex grid, you have 4 unknowns per vertex, and thus 16 unknowns<br>
> per element. You would use a blocksize of 4 (indicating a vertex with<br>
> u,v,w,p).<br>
> Then you could split apart the fields if you wanted.<br>
><br>
> Matt<br>
</div></div>Ok. My degrees of freedom are as you say, a 3d velocity (vx, vy, vz) and<br>
a pressure (p). I have an unstructured mesh with<br>
<br>
337983 elements<br>
63686 nodes<br>
<br>
I have penalised the velocity where it was set to 0, so that I can use<br>
MPIBAIJ. My system consists of a matrix of<br>
<br>
254744 = 4*63686 rows<br>
14490112 non-zeros<br>
<br>
The maximal number of neighbours of a node is 116.<br>
<br>
Given that matrix, created with<br>
<br>
MatCreateMPIBAIJ(4,<br>
PETSC_DECIDE,<br>
PETSC_DECIDE,<br>
254744<br>
254744<br>
116,<br>
PETSC_DECIDE,<br>
&A);<br>
<br>
if I now call my program with options (on any number of procs, but below<br>
with 8 procs)<br>
<br>
-pc_type fieldsplit -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3<br>
<br>
I get the error (when the script is calling KSPSolve()):<br>
<br>
[7]PETSC ERROR: --------------------- Error Message<br>
------------------------------------<br>
[7]PETSC ERROR: Nonconforming object sizes!<br>
[7]PETSC ERROR: Local column sizes 47760 do not add up to total number<br>
of columns 63686!<br>
[7]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[7]PETSC ERROR: Petsc Release Version 3.1.0, Patch 1, Thu Apr 8<br>
14:16:50 CDT 2010<br>
[7]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
[7]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
[7]PETSC ERROR: See docs/index.html for manual pages.<br>
[7]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[7]PETSC ERROR: stokes-5.3b01 on a linux-gnu named iacspc3 by lmichel<br>
Thu Jun 17 15:05:51 2010<br>
[7]PETSC ERROR: Libraries linked from<br>
/data/progr/petsc-3.1-openmpi-1.4.2/lib<br>
[7]PETSC ERROR: Configure run at Sat May 29 14:01:23 2010<br>
[7]PETSC ERROR: Configure options<br>
--prefix=/data/progr/petsc-3.1-openmpi-1.4.2<br>
--PETSC_ARCH=linux-gnu-cxx-openmpi-1.4.2-opt --with-clanguage=C++<br>
--with-debugging=0 --with-blas-lib=-lblas --with-lapack-lib=-llapack<br>
--with-shared --with-mpi-dir=/data/progr/openmpi-1.4.2<br>
--download-umfpack=1 --with-umfpack=1 --download-parmetis=1<br>
--with-parmetis=1 --download-prometheus=1 --with-prometheus=1<br>
[7]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[7]PETSC ERROR: MatGetSubMatrix_MPIBAIJ() line 1927 in<br>
src/mat/impls/baij/mpi/mpibaij.c<br>
[7]PETSC ERROR: MatGetSubMatrix_MPIBAIJ() line 1851 in<br>
src/mat/impls/baij/mpi/mpibaij.c<br>
[7]PETSC ERROR: MatGetSubMatrix() line 6425 in src/mat/interface/matrix.c<br>
[7]PETSC ERROR: PCSetUp_FieldSplit() line 296 in<br>
src/ksp/pc/impls/fieldsplit/fieldsplit.c<br>
[7]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c<br>
[7]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c<br>
[7]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c<br>
<br>
thanks for your help,<br>
<div><div></div><div><br>
L.<br>
<br>
--<br>
Laurent Michel<br>
PhD candidate<br>
EPFL SB IACS ASN<br>
MA C2 644<br>
+41 21 693 42 46<br>
+41 77 433 38 94<br>
<br>
<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>
</div></div>
</blockquote></div><br></div></div></div></div></blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>