[petsc-users] Any suggestion for this kinds of matrix?

Fri Nov 11 05:12:07 CST 2011

Dear Jed,

The following is the result with the options you told me. The iteration has
converged, while converged at a wrong solution, compared to the result from
LU.

tutorials]$ time ./ex78 -Ain A_phi -rhs rhs_phi -solu solu_phi -noshift
-pc_type hypre -pc_hypre_type parasails -ksp_gmres_restart 600 -ksp_rtol
1e-7 -ksp_typ gmres -ksp_monitor_true_residual -ksp_converged_reason

 Read matrix in ascii format ...
m: 288399, n: 288399, nz: 4023176
 read A completed
 rowNumber[0] = 13
 rowNumber[1] = 13
 rowNumber[2] = 19
 read A is complete !

 Read rhs in ascii format ...

 Read exact solution in ascii format ...
  0 KSP preconditioned resid norm 1.311815748108e+00 true resid norm
3.838432566849e-03 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP preconditioned resid norm 5.507600629359e-01 true resid norm
1.878066463331e-03 ||r(i)||/||b|| 4.892795250727e-01
  2 KSP preconditioned resid norm 4.013507277599e-01 true resid norm
1.785692106749e-03 ||r(i)||/||b|| 4.652138797934e-01
  3 KSP preconditioned resid norm 3.186589484171e-01 true resid norm
1.939928539000e-03 ||r(i)||/||b|| 5.053960191341e-01
  4 KSP preconditioned resid norm 2.533816267053e-01 true resid norm
2.182716525094e-03 ||r(i)||/||b|| 5.686478756839e-01
  5 KSP preconditioned resid norm 1.956749847727e-01 true resid norm
2.436511231157e-03 ||r(i)||/||b|| 6.347672360328e-01
  6 KSP preconditioned resid norm 1.489551919079e-01 true resid norm
2.625495747280e-03 ||r(i)||/||b|| 6.840020507214e-01
  7 KSP preconditioned resid norm 1.129706022530e-01 true resid norm
2.726727688051e-03 ||r(i)||/||b|| 7.103753004809e-01
  8 KSP preconditioned resid norm 8.528153693722e-02 true resid norm
2.764872691683e-03 ||r(i)||/||b|| 7.203129515839e-01
  9 KSP preconditioned resid norm 6.419522418091e-02 true resid norm
2.765551151077e-03 ||r(i)||/||b|| 7.204897058665e-01
 10 KSP preconditioned resid norm 4.793073337207e-02 true resid norm
2.754803244091e-03 ||r(i)||/||b|| 7.176896288040e-01
 11 KSP preconditioned resid norm 3.590594904610e-02 true resid norm
2.738481920714e-03 ||r(i)||/||b|| 7.134375485361e-01
 12 KSP preconditioned resid norm 2.683482240096e-02 true resid norm
2.722031760807e-03 ||r(i)||/||b|| 7.091519033880e-01
 13 KSP preconditioned resid norm 2.001207136261e-02 true resid norm
2.709429246945e-03 ||r(i)||/||b|| 7.058686585627e-01
 14 KSP preconditioned resid norm 1.493908729876e-02 true resid norm
2.699791991674e-03 ||r(i)||/||b|| 7.033579318265e-01
 15 KSP preconditioned resid norm 1.111558666088e-02 true resid norm
2.692890954089e-03 ||r(i)||/||b|| 7.015600527536e-01
 16 KSP preconditioned resid norm 8.272119255509e-03 true resid norm
2.688069519102e-03 ||r(i)||/||b|| 7.003039580057e-01
 17 KSP preconditioned resid norm 6.143976425601e-03 true resid norm
2.684895702336e-03 ||r(i)||/||b|| 6.994771057136e-01
 18 KSP preconditioned resid norm 4.563685459707e-03 true resid norm
2.682859292253e-03 ||r(i)||/||b|| 6.989465740324e-01
 19 KSP preconditioned resid norm 3.394656398417e-03 true resid norm
2.681459888330e-03 ||r(i)||/||b|| 6.985819971124e-01
 20 KSP preconditioned resid norm 2.518916365228e-03 true resid norm
2.680609739005e-03 ||r(i)||/||b|| 6.983605136526e-01
 21 KSP preconditioned resid norm 1.872307188353e-03 true resid norm
2.680081243394e-03 ||r(i)||/||b|| 6.982228283859e-01
 22 KSP preconditioned resid norm 1.390334828536e-03 true resid norm
2.679742748684e-03 ||r(i)||/||b|| 6.981346427259e-01
 23 KSP preconditioned resid norm 1.034606694934e-03 true resid norm
2.679535108562e-03 ||r(i)||/||b|| 6.980805476966e-01
 24 KSP preconditioned resid norm 7.710134967260e-04 true resid norm
2.679397134577e-03 ||r(i)||/||b|| 6.980446022989e-01
 25 KSP preconditioned resid norm 5.725407938260e-04 true resid norm
2.679329449922e-03 ||r(i)||/||b|| 6.980269688889e-01
 26 KSP preconditioned resid norm 4.272990427118e-04 true resid norm
2.679284171589e-03 ||r(i)||/||b|| 6.980151728414e-01
 27 KSP preconditioned resid norm 3.181341598383e-04 true resid norm
2.679247206576e-03 ||r(i)||/||b|| 6.980055426050e-01
 28 KSP preconditioned resid norm 2.368729933003e-04 true resid norm
2.679233163958e-03 ||r(i)||/||b|| 6.980018841799e-01
 29 KSP preconditioned resid norm 1.766017339700e-04 true resid norm
2.679224053276e-03 ||r(i)||/||b|| 6.979995106376e-01
 30 KSP preconditioned resid norm 1.313377419946e-04 true resid norm
2.679217016981e-03 ||r(i)||/||b|| 6.979976775210e-01
 31 KSP preconditioned resid norm 9.789603459870e-05 true resid norm
2.679213290696e-03 ||r(i)||/||b|| 6.979967067380e-01
 32 KSP preconditioned resid norm 7.275708495896e-05 true resid norm
2.679210423371e-03 ||r(i)||/||b|| 6.979959597340e-01
 33 KSP preconditioned resid norm 5.412802491776e-05 true resid norm
2.679209810847e-03 ||r(i)||/||b|| 6.979958001573e-01
 34 KSP preconditioned resid norm 4.026672785271e-05 true resid norm
2.679209362635e-03 ||r(i)||/||b|| 6.979956833876e-01
 35 KSP preconditioned resid norm 2.990907253308e-05 true resid norm
2.679208426592e-03 ||r(i)||/||b|| 6.979954395269e-01
 36 KSP preconditioned resid norm 2.226822676398e-05 true resid norm
2.679208136434e-03 ||r(i)||/||b|| 6.979953639342e-01
 37 KSP preconditioned resid norm 1.654703590780e-05 true resid norm
2.679208182746e-03 ||r(i)||/||b|| 6.979953759996e-01
 38 KSP preconditioned resid norm 1.229268254949e-05 true resid norm
2.679208209944e-03 ||r(i)||/||b|| 6.979953830852e-01
 39 KSP preconditioned resid norm 9.149145951039e-06 true resid norm
2.679208050822e-03 ||r(i)||/||b|| 6.979953416303e-01
 40 KSP preconditioned resid norm 6.813825018110e-06 true resid norm
2.679207932900e-03 ||r(i)||/||b|| 6.979953109089e-01
 41 KSP preconditioned resid norm 5.075333494970e-06 true resid norm
2.679208029175e-03 ||r(i)||/||b|| 6.979953359907e-01
 42 KSP preconditioned resid norm 3.770609781438e-06 true resid norm
2.679208069198e-03 ||r(i)||/||b|| 6.979953464175e-01
 43 KSP preconditioned resid norm 2.808924777973e-06 true resid norm
2.679208000517e-03 ||r(i)||/||b|| 6.979953285246e-01
 44 KSP preconditioned resid norm 2.094599249993e-06 true resid norm
2.679207985642e-03 ||r(i)||/||b|| 6.979953246492e-01
 45 KSP preconditioned resid norm 1.559223301396e-06 true resid norm
2.679208018840e-03 ||r(i)||/||b|| 6.979953332981e-01
 46 KSP preconditioned resid norm 1.160309778657e-06 true resid norm
2.679208029228e-03 ||r(i)||/||b|| 6.979953360044e-01
 47 KSP preconditioned resid norm 8.638154916854e-07 true resid norm
2.679208013013e-03 ||r(i)||/||b|| 6.979953317800e-01
 48 KSP preconditioned resid norm 6.436084879799e-07 true resid norm
2.679208008459e-03 ||r(i)||/||b|| 6.979953305937e-01
 49 KSP preconditioned resid norm 4.797395939888e-07 true resid norm
2.679208018385e-03 ||r(i)||/||b|| 6.979953331796e-01
 50 KSP preconditioned resid norm 3.573839482305e-07 true resid norm
2.679208020910e-03 ||r(i)||/||b|| 6.979953338374e-01
 51 KSP preconditioned resid norm 2.662426448119e-07 true resid norm
2.679208017655e-03 ||r(i)||/||b|| 6.979953329896e-01
 52 KSP preconditioned resid norm 1.984893339085e-07 true resid norm
2.679208016597e-03 ||r(i)||/||b|| 6.979953327137e-01
 53 KSP preconditioned resid norm 1.484050273141e-07 true resid norm
2.679208018006e-03 ||r(i)||/||b|| 6.979953330809e-01
 54 KSP preconditioned resid norm 1.106994152625e-07 true resid norm
2.679208019541e-03 ||r(i)||/||b|| 6.979953334807e-01
Linear solve converged due to CONVERGED_RTOL iterations 54

 Accuracy of the soltuion on the solution from LU: | u -U_lu |_2 : 3321.15

 Iteration number is  :  54

 Accuracy of the soltuion: | b - A*u |_2 : 0.004681

real    0m15.393s
user    0m14.895s
sys    0m0.251s

> Message: 8
> > Date: Thu, 10 Nov 2011 14:10:14 +0000
> > From: Matthew Knepley <knepley at gmail.com>
> > Subject: Re: [petsc-users] Any suggestion for this kinds of matrix?
> > To: PETSc users list <petsc-users at mcs.anl.gov>
> > Message-ID:
> >        <
> CAMYG4GnxR2A+TcVzmsiOgD4wp4P+a_GRnPBg1YQKz7bJT5XiDw at mail.gmail.com
> > >
> > Content-Type: text/plain; charset="iso-8859-1"
> >
> > On Thu, Nov 10, 2011 at 1:48 PM, Bao Kai <paeanball at gmail.com> wrote:
> >
> > > Dear all,
> > >
> > > I have been trying with PETSC to solve the linear system from mixed
> > finite
> > > element method.
> > >
> > > The pattern of the matrix is as the following, but due to the irregular
> > > boundary involved, the matrix A is not strictly symmetric.
> > >
> > >    A     dt* C
> > >
> > >    C^T     0
> > >
> > >  As a  result of the matrix pattern, the diagonal entries of the
> > > bottom-right portion are all zero.
> > >
> > > I am just wondering if there are any suggestion of the type of the
> solver
> > > and preconditioner for this kinds of linear system?  Thank you very
> much.
> > >
> > > When I tried to solve the system with PETSC, I got the following
> > > information. (  PCType PCASM,  KSPType, KSPFGMRES )
> > >
> >
> > ILU is jsut not going to work for this type of matrix (a saddle point). I
> > suggest reading about PCFIELDSPLIT.
> >
> >   Matt
> >
> >
> > > [0]PETSC ERROR: --------------------- Error Message
> > > ------------------------------------
> > > [0]PETSC ERROR: Object is in wrong state!
> > > [0]PETSC ERROR: Matrix is missing diagonal entry 288398!
> > > [0]PETSC ERROR:
> > >
> ------------------------------------------------------------------------
> > > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 4, Sun Oct 23
> 12:23:18
> > > CDT 2011
> > > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > > [0]PETSC ERROR: See docs/index.html for manual pages.
> > > [0]PETSC ERROR:
> > >
> ------------------------------------------------------------------------
> > > [0]PETSC ERROR: Libraries linked from
> > > /home/baok/software/petsc-3.2-p4/arch-linux2-c-debug-withhypre/lib
> > > [0]PETSC ERROR: Configure run at Thu Nov 10 11:49:03 2011
> > > [0]PETSC ERROR: Configure options --download-hypre=yes
> > > [0]PETSC ERROR:
> > >
> ------------------------------------------------------------------------
> > > [0]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ_ilu0() line 1636 in
> > > /home/baok/software/petsc-3.2-p4/src/mat/impls/aij/seq/aijfact.c
> > > [0]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ() line 1740 in
> > > /home/baok/software/petsc-3.2-p4/src/mat/impls/aij/seq/aijfact.c
> > > [0]PETSC ERROR: MatILUFactorSymbolic() line 6092 in
> > > /home/baok/software/petsc-3.2-p4/src/mat/interface/matrix.c
> > > [0]PETSC ERROR: PCSetUp_ILU() line 216 in
> > > /home/baok/software/petsc-3.2-p4/src/ksp/pc/impls/factor/ilu/ilu.c
> > > [0]PETSC ERROR: PCSetUp() line 819 in
> > > /home/baok/software/petsc-3.2-p4/src/ksp/pc/interface/precon.c
> > > [0]PETSC ERROR: KSPSetUp() line 260 in
> > > /home/baok/software/petsc-3.2-p4/src/ksp/ksp/interface/itfunc.c
> > > [0]PETSC ERROR: PCSetUpOnBlocks_ASM() line 339 in
> > > /home/baok/software/petsc-3.2-p4/src/ksp/pc/impls/asm/asm.c
> > > [0]PETSC ERROR: PCSetUpOnBlocks() line 852 in
> > > /home/baok/software/petsc-3.2-p4/src/ksp/pc/interface/precon.c
> > > [0]PETSC ERROR: KSPSetUpOnBlocks() line 154 in
> > > /home/baok/software/petsc-3.2-p4/src/ksp/ksp/interface/itfunc.c
> > > [0]PETSC ERROR: KSPSolve() line 380 in
> > > /home/baok/software/petsc-3.2-p4/src/ksp/ksp/interface/itfunc.c
> > > [0]PETSC ERROR: main() line 261 in
> src/ksp/ksp/examples/tutorials/ex78.c
> > >
> > >
> > > Best Regards,
> > > Kai
> > >
> > >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> > experiments is infinitely more interesting than any results to which
> their
> > experiments lead.
> > -- Norbert Wiener
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> > URL: <
> >
> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20111110/2ea2b385/attachment.htm
> > >
> >
> > ------------------------------
> >
> > _______________________________________________
> > petsc-users mailing list
> > petsc-users at mcs.anl.gov
> > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users
> >
> >
> > End of petsc-users Digest, Vol 35, Issue 29
> > *******************************************
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20111110/da069f92/attachment-0001.htm
> >
>
> ------------------------------
>
> Message: 2
> Date: Thu, 10 Nov 2011 14:49:55 -0600
> From: Jed Brown <jedbrown at mcs.anl.gov>
> Subject: Re: [petsc-users] Any suggestion for this kinds of matrix?
> To: PETSc users list <petsc-users at mcs.anl.gov>
> Message-ID:
>        <CAM9tzSkW7PZaBLH_sXpxToYv6x3seupTQcY+sPj=fHc_ax-AjA at mail.gmail.com
> >
> Content-Type: text/plain; charset="utf-8"
>
> On Thu, Nov 10, 2011 at 14:45, Bao Kai <paeanball at gmail.com> wrote:
>
> > PCFIELDSPLIT seems a little more complex, I will try that.
> >
> > I tried some different preconditioners, only lu can get right results.
> >
> > With some pc, some wrong results can be obtained, such as the following
> > one.
> >
> > tutorials]$ time ./ex78 -Ain A_in -rhs rhs -solu solu -noshift -pc_type
> > hypre -pc_hypre_type parasails -ksp_gmres_restart 600 -ksp_rtol 1e-5
> > -ksp_typ gmres
> >
>
> Always run with -ksp_monitor_true_residual -ksp_converged_reason when
> checking whether a preconditioner is working.
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20111110/3d5e1de6/attachment-0001.htm
> >
>
> ------------------------------
>
> Message: 3
> Date: Thu, 10 Nov 2011 22:11:01 -0700
> From: Xiang Hao <haoxiang at yahoo.cn>
> Subject: [petsc-users] Problem with including petscdmda.h
> To: PETSc users list <petsc-users at mcs.anl.gov>
> Message-ID:
>        <CAL-Bv1oxfJhXrrxJxGz1+mEGbnP7fRREGHpqsyB+rk0jTfpFCw at mail.gmail.com
> >
> Content-Type: text/plain; charset="windows-1252"
>
> Hi,
>
> I have a question about petscdmda.h.
>
> I have a program using PETSc, which is running well. Now I just add a new
> line #include <petscdmda.h> in my code and get the following error. I don't
> understand what's going on here. Any help?
>
>
> -------------------------------------------------------------------------------------------------------------
> In file included from /home/sci/hao/software/PETSc/include/petscdmda.h:4:0,
>                 from
> /home/sci/hao/programming/C++/ITK/SolveAlpha/SolveAlpha.h:18,
>                 from
> /home/sci/hao/programming/C++/ITK/SolveAlpha/SolveAlpha.cxx:9:
> /home/sci/hao/software/PETSc/include/petscdm.h:27:8: error: ?PetscClassId?
> does not name a type
> /home/sci/hao/software/PETSc/include/petscdm.h:48:8: error: ?PetscBool?
> does not name a type
> /home/sci/hao/software/PETSc/include/petscdm.h:120:55: error: ?PetscBool?
> has not been declared
> /home/sci/hao/software/PETSc/include/petscdm.h:144:46: error: ?PetscBool?
> has not been declared
> /home/sci/hao/software/PETSc/include/petscdm.h:145:42: error: ?PetscBool?
> has not been declared
> /home/sci/hao/software/PETSc/include/petscdm.h:146:42: error: ?PetscBool?
> has not been declared
> In file included from /home/sci/hao/software/PETSc/include/petscdm.h:157:0,
>                 from /home/sci/hao/software/PETSc/include/petscdmda.h:4,
>                 from
> /home/sci/hao/programming/C++/ITK/SolveAlpha/SolveAlpha.h:18,
>                 from
> /home/sci/hao/programming/C++/ITK/SolveAlpha/SolveAlpha.cxx:9:
> /home/sci/hao/software/PETSc/include/petscbag.h:44:60: error: ?PetscBool?
> has not been declared
> In file included from /home/sci/hao/software/PETSc/include/petscdmda.h:5:0,
>                 from
> /home/sci/hao/programming/C++/ITK/SolveAlpha/SolveAlpha.h:18,
>                 from
> /home/sci/hao/programming/C++/ITK/SolveAlpha/SolveAlpha.cxx:9:
> /home/sci/hao/software/PETSc/include/petscpf.h:41:8: error: ?PetscClassId?
> does not name a type
> In file included from /home/sci/hao/software/PETSc/include/petscdmda.h:5:0,
>                 from
> /home/sci/hao/programming/C++/ITK/SolveAlpha/SolveAlpha.h:18,
>                 from
> /home/sci/hao/programming/C++/ITK/SolveAlpha/SolveAlpha.cxx:9:
> /home/sci/hao/software/PETSc/include/petscpf.h:52:8: error: ?PetscBool?
> does not name a type
>
> -------------------------------------------------------------------------------------------------------------------
>
>
> The head file of my program begin with the following. The read line is the
> one I just added.
>
> #include <fstream>
> #include <iostream>
> #include <itkImage.h>
> #include <itkImageFileReader.h>
> #include <itkImageFileWriter.h>
> #include <itkVectorImage.h>
> #include <itkRecursiveGaussianImageFilter.h>
> #include <petscksp.h>
> #include <petscdmda.h>
> #include <itkMedianImageFilter.h>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20111110/a9209e2e/attachment-0001.htm
> >
>
> ------------------------------
>
> Message: 4
> Date: Thu, 10 Nov 2011 23:17:27 -0600
> From: Jed Brown <jedbrown at mcs.anl.gov>
> Subject: Re: [petsc-users] Problem with including petscdmda.h
> To: PETSc users list <petsc-users at mcs.anl.gov>
> Message-ID:
>        <CAM9tzSkbrwNLoMCu4p_eL9syVxMXdm6jtmgvts6WwohniREMJw at mail.gmail.com
> >
> Content-Type: text/plain; charset="utf-8"
>
> On Thu, Nov 10, 2011 at 23:11, Xiang Hao <haoxiang at yahoo.cn> wrote:
>
> > I have a program using PETSc, which is running well. Now I just add a new
> > line #include <petscdmda.h> in my code and get the following error. I
> don't
> > understand what's going on here. Any help?
> >
> >
> >
> -------------------------------------------------------------------------------------------------------------
> > In file included from
> /home/sci/hao/software/PETSc/include/petscdmda.h:4:0,
> >                  from
> > /home/sci/hao/programming/C++/ITK/SolveAlpha/SolveAlpha.h:18,
> >                  from
> > /home/sci/hao/programming/C++/ITK/SolveAlpha/SolveAlpha.cxx:9:
> > /home/sci/hao/software/PETSc/include/petscdm.h:27:8: error:
> ?PetscClassId?
> > does not name a type
> > /home/sci/hao/software/PETSc/include/petscdm.h:48:8: error: ?PetscBool?
> > does not name a type
> >
>
> The most likely explanation is that a path to an old version of PETSc
> appears before this one in the header search paths, therefore it finds the
> file with a new name (petscdm.h) in the new directory, but finds all the
> supporting headers (which did not change names) in the old directory. Check
> the command line and environment.
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20111110/7c64d647/attachment-0001.htm
> >
>
> ------------------------------
>
> Message: 5
> Date: Fri, 11 Nov 2011 12:39:02 +0200
> From: Konstantinos Kontzialis <ckontzialis at lycos.com>
> Subject: [petsc-users] configuration problem
> To: petsc-users at mcs.anl.gov
> Message-ID: <4EBCFB46.8050509 at lycos.com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Dear all,
>
> I'm trying to configure petsc v3.2 with the following options:
>
> ./configure --with-debugging=1 --with-mpi-dir=/usr/lib64/mpich2/bin
> --with-shared-libraries
> --with-large-file-io=1
> --with-precision=__float128
> --with-blacs=1
> --download-blacs=yes
> --download-f-blas-lapack=yes
> --with-plapack=1
> --download-plapack=yes
> --with-scalapack=1
> --download-scalapack=yes
> --with-superlu=1
> --download-superlu=yes
>  --with-superlu_dist=1
> --download-superlu_dist=yes
>  --with-ml=1 --download-ml=yes
>  --with-umfpack=1
>  --download-umfpack=yes
>  --with-sundials=1
> --download-sundials=1
> --with-parmetis=1
>  --download-parmetis=1
> --with-hypre=1
> --download-hypre=1
>
>
> and I get:
>
>
>
> *******************************************************************************
>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log
> for details):
>
> -------------------------------------------------------------------------------
> Could not use downloaded f-blas-lapack?
>
> *******************************************************************************
>
> What should I do?
>
> Kostas
>
>
> ------------------------------
>
> _______________________________________________
> petsc-users mailing list
> petsc-users at mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/petsc-users
>
>
> End of petsc-users Digest, Vol 35, Issue 32
> *******************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20111111/f17fcdca/attachment-0001.htm>