From karpeev at mcs.anl.gov Mon Feb 4 12:30:39 2013 From: karpeev at mcs.anl.gov (Dmitry Karpeev) Date: Mon, 4 Feb 2013 12:30:39 -0600 Subject: [petsc-users] [Libmesh-users] Preconditioning of a Stokes problem In-Reply-To: References: Message-ID: Francesco, Are you using any pressure stabilization scheme? If not, the 11-block in your Jacobian A = [A00 A01; A10 A11] would typically be zero, and preconditioning it with jacobi wouldn't really work. If A11 = 0, you ought to use the fieldsplit of type Schur, for example (assuming petsc-3.3; prior versions use slightly different option names): -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type full The question is then how you want to solve the Schur complement system. Since the A11 block for you is zero, the easiest choice is to use S as the preconditioning matrix: -pc_fieldsplit_schur_precondition self Preconditioning options for the Schur solver have prefix -fieldsplit_1_. Since for Stokes the Schur complement is spectrally equivalent to the pressure mass matrix, you can hope that an unpreconditioned GMRES (default when using -pc_fieldsplit_schur_precondition self) will solve it. A better choice might be to use the Least Squares Commutator preconditioner -fieldsplit_1_pc_type lsc See http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCLSC.html and note the relevant references at the bottom of that manual page. You might also find this useful: http://www.icfd.rdg.ac.uk/ICFD25/Talks/AWathen.pdf PCLSC preconditions S = A10 inv(A00) A01 with inv(A10A01) A10 A00 A01 inv(A10A01) The "Laplacian" (a true Laplacian in the case of incompressible Stokes) can be preconditioned using AMG: -fieldsplit_1_lsc_pc_type ml (assuming you have ML built, e.g., with --download-ml) or -fieldsplit_1_lsc_pc_type gamg with some appropriate choice of gamg options: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMG.html To start with I would use -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type full -pc_fieldsplit_schur_precondition self -fieldsplit_1_pc_type lsc -fieldsplit_1_lsc_pc_type ml gamg may also be very good in place of ml, but will require more option setting. You might find src/ksp/ksp/examples/tutorials/ex42.c and ex43.c useful, albeit, possibly, a bit heavyweight for your needs. A few more comments below: On Mon, Feb 4, 2013 at 10:20 AM, John Peterson wrote: > On Thu, Dec 27, 2012 at 4:31 AM, Francesco Ballarin > wrote: > > Dear libMesh users, > > I am trying to solve a Stokes problem in a 3D domain, but I am currently > > struggling with the choice of the preconditioner. > > I have slightly modified systems_of_equations_ex1 to accommodate for the > > third velocity component and boundary conditions of the problem at hand > > (Dirichlet BCs on the inlet on the walls, enforced using a > > DirichletBoundary object, and homogeneous Neumann BC on the outflow) > > > > I am trying to use PETSc block preconditioners as shown, e.g., in > > http://acts.nersc.gov/events/Workshop2012/slides/PETSc.pdf > > section 4. I have tried most of the preconditioners discussed during the > > talk, but I am not satisfied because I get either PETSc errors or a > > performance that I hope can be increased. > > > > For example, with the following options: > > ******* > > PetscOptionsSetValue("-ksp_type", "gmres"); > > PetscOptionsSetValue("-ksp_right_pc", PETSC_NULL); > > PetscOptionsSetValue("-pc_type", "fieldsplit"); > > PetscOptionsSetValue("-pc_fieldsplit_detect_saddle_point", PETSC_NULL); > > PetscOptionsSetValue("-pc_fieldsplit_type", "multiplicative"); > > PetscOptionsSetValue("-fieldsplit_0_ksp_type", "preonly"); > > PetscOptionsSetValue("-fieldsplit_0_pc_type", "ml"); > > PetscOptionsSetValue("-fieldsplit_1_ksp_type", "preonly"); > > PetscOptionsSetValue("-fieldsplit_1_pc_type", "jacobi"); > You do not need to set options to PETSC_NULL, if you do not intend to use them. Also, how are you forming the splits? -pc_fieldsplit_detect_saddle_point is actually rather useful here. Further, there are programmatic ways to set most of these options, for example: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetType.html Even so, we typically recommend setting these on the command line whenever possible: this way you do not hardwire your solver choices and can quickly experiment with different solvers from the command-line without recompiling your code. > > ******* > > I obtain a reasonable solution but after thousands of gmres iterations. > Is > > there any way to improve that? > > > > Or, if I try to embed the previous block preconditioner in mg: > > ******* > > PetscOptionsSetValue("-pc_type", "mg"); > > PetscOptionsSetValue("-pc_mg_levels", "5"); > > PetscOptionsSetValue("-pc_mg_galerkin", PETSC_NULL); > > PetscOptionsSetValue("-ksp_type", "gmres"); > > PetscOptionsSetValue("-mg_levels_pc_type", "fieldsplit"); > > PetscOptionsSetValue("-mg_levels_pc_fieldsplit_detect_saddle_point", > > PETSC_NULL); > > PetscOptionsSetValue("-mg_levels_pc_fieldsplit_type", "multiplicative"); > > PetscOptionsSetValue("-mg_levels_fieldsplit_0_ksp_type", "preonly"); > > PetscOptionsSetValue("-mg_levels_fieldsplit_0_pc_type", "gamg"); > > PetscOptionsSetValue("-mg_levels_fieldsplit_1_ksp_type", "preonly"); > > PetscOptionsSetValue("-mg_levels_fieldsplit_1_pc_type", "jacobi"); > > ******* > > I get the following error: > Always send the full stack trace, when reporting errors. Finally, in the future you might want to post these questions to petsc-users at mcs.anl.gov Dmitry. > > ******* > > PETSC ERROR: Null argument, when expecting valid pointer! > > PETSC ERROR: Null Object: Parameter # 2! > > PETSC ERROR: MatPtAP() line 8237 in > petsc-3.3-p4/src/mat/interface/matrix.c > > ******* > > > > Do you have any suggestion? What are you currently using as > preconditioner > > for saddle point problems? > > Apologies, it appears that this email fell through the cracks > somewhere back around the holidays. > > Did you ever find a satisfactory answer to your question? > > Petsc developers may occasionally read this list, but for questions > that are this advanced you would probably have better luck mailing the > petsc developers list directly. > > And of course if you do find a good block preconditioning strategy for > systems_of_equations_ex1, please feel free to report it here! > > -- > John > > > ------------------------------------------------------------------------------ > Everyone hates slow websites. So do we. > Make your web apps faster with AppDynamics > Download AppDynamics Lite for free today: > http://p.sf.net/sfu/appdyn_d2d_jan > _______________________________________________ > Libmesh-users mailing list > Libmesh-users at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/libmesh-users > -- Dmitry Karpeev, Ph.D. Assistant Computational Mathematician Mathematics and Computer Science Argonne National Laboratory Argonne, Illinois, USA and Fellow Computation Institute University of Chicago 5735 S. Ellis Avenue Chicago, IL 60637 ----------------------- Phone: 630-252-1229 Fax: 630-252-5986 -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhalen at gmail.com Mon Feb 4 12:31:46 2013 From: gokhalen at gmail.com (Nachiket Gokhale) Date: Mon, 4 Feb 2013 13:31:46 -0500 Subject: [petsc-users] setting name for matlab objects Message-ID: I am outputting some matrices and vectors to matlab using the ascii viewer and setting format to matlab. However, the objects created in matlab have names like Mat_0x84000001_0 Vec_0x84000001_5 Is it possible for the user to set the name? It seems to be possible for Mathematica: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerMathematicaSetName.html -Nachiket From tisaac at ices.utexas.edu Mon Feb 4 12:45:53 2013 From: tisaac at ices.utexas.edu (Tobin Isaac) Date: Mon, 4 Feb 2013 12:45:53 -0600 Subject: [petsc-users] [Libmesh-users] Preconditioning of a Stokes problem In-Reply-To: References: Message-ID: <20130204184553.GA14979@ices.utexas.edu> On Mon, Feb 04, 2013 at 12:30:39PM -0600, Dmitry Karpeev wrote: > Francesco, > > Are you using any pressure stabilization scheme? If not, the 11-block in > your Jacobian > A = [A00 A01; A10 A11] would typically be zero, and preconditioning it with > jacobi wouldn't really work. > > If A11 = 0, you ought to use the fieldsplit of type Schur, for example > (assuming petsc-3.3; prior versions use slightly different option names): > -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type full > The question is then how you want to solve the Schur complement system. > Since the A11 block for you is zero, the easiest choice is to use S as the > preconditioning matrix: > -pc_fieldsplit_schur_precondition self > > Preconditioning options for the Schur solver have prefix -fieldsplit_1_. > Since for Stokes the Schur complement is spectrally equivalent to the > pressure mass matrix, you can > hope that an unpreconditioned GMRES (default when using > -pc_fieldsplit_schur_precondition self) will solve it. > > > A better choice might be to use the Least Squares Commutator preconditioner > -fieldsplit_1_pc_type lsc > See > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCLSC.html > and note the relevant references at the bottom of that manual page. > You might also find this useful: > http://www.icfd.rdg.ac.uk/ICFD25/Talks/AWathen.pdf > > PCLSC preconditions S = A10 inv(A00) A01 with inv(A10A01) A10 A00 A01 > inv(A10A01) > The "Laplacian" (a true Laplacian in the case of incompressible Stokes) can > be preconditioned > using AMG: > -fieldsplit_1_lsc_pc_type ml > (assuming you have ML built, e.g., with --download-ml) > or > -fieldsplit_1_lsc_pc_type gamg > with some appropriate choice of gamg options: > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMG.html > > To start with I would use -pc_fieldsplit_type schur > -pc_fieldsplit_schur_fact_type full -pc_fieldsplit_schur_precondition self > -fieldsplit_1_pc_type lsc -fieldsplit_1_lsc_pc_type ml > gamg may also be very good in place of ml, but will require more option > setting. > > > You might find src/ksp/ksp/examples/tutorials/ex42.c and ex43.c useful, > albeit, possibly, a bit heavyweight for your needs. Also, if your problem isn't stabilized then your vectors (hopefully) aren't layed out in strided form. A good example of how to setup a fieldsplit preconditioner in this more general setting is src/snes/examples/tutorials/ex70.c > > A few more comments below: > > On Mon, Feb 4, 2013 at 10:20 AM, John Peterson wrote: > > > On Thu, Dec 27, 2012 at 4:31 AM, Francesco Ballarin > > wrote: > > > Dear libMesh users, > > > I am trying to solve a Stokes problem in a 3D domain, but I am currently > > > struggling with the choice of the preconditioner. > > > I have slightly modified systems_of_equations_ex1 to accommodate for the > > > third velocity component and boundary conditions of the problem at hand > > > (Dirichlet BCs on the inlet on the walls, enforced using a > > > DirichletBoundary object, and homogeneous Neumann BC on the outflow) > > > > > > I am trying to use PETSc block preconditioners as shown, e.g., in > > > http://acts.nersc.gov/events/Workshop2012/slides/PETSc.pdf > > > section 4. I have tried most of the preconditioners discussed during the > > > talk, but I am not satisfied because I get either PETSc errors or a > > > performance that I hope can be increased. > > > > > > For example, with the following options: > > > ******* > > > PetscOptionsSetValue("-ksp_type", "gmres"); > > > PetscOptionsSetValue("-ksp_right_pc", PETSC_NULL); > > > PetscOptionsSetValue("-pc_type", "fieldsplit"); > > > PetscOptionsSetValue("-pc_fieldsplit_detect_saddle_point", PETSC_NULL); > > > PetscOptionsSetValue("-pc_fieldsplit_type", "multiplicative"); > > > PetscOptionsSetValue("-fieldsplit_0_ksp_type", "preonly"); > > > PetscOptionsSetValue("-fieldsplit_0_pc_type", "ml"); > > > PetscOptionsSetValue("-fieldsplit_1_ksp_type", "preonly"); > > > PetscOptionsSetValue("-fieldsplit_1_pc_type", "jacobi"); > > > > You do not need to set options to PETSC_NULL, if you do not intend to use > them. > Also, how are you forming the splits? > -pc_fieldsplit_detect_saddle_point is actually rather useful here. > > Further, there are programmatic ways to set most of these options, for > example: > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetType.html > > Even so, we typically recommend setting these on the command line whenever > possible: > this way you do not hardwire your solver choices and can quickly experiment > with different > solvers from the command-line without recompiling your code. > > > > > ******* > > > I obtain a reasonable solution but after thousands of gmres iterations. > > Is > > > there any way to improve that? > > > > > > Or, if I try to embed the previous block preconditioner in mg: > > > ******* > > > PetscOptionsSetValue("-pc_type", "mg"); > > > PetscOptionsSetValue("-pc_mg_levels", "5"); > > > PetscOptionsSetValue("-pc_mg_galerkin", PETSC_NULL); > > > PetscOptionsSetValue("-ksp_type", "gmres"); > > > PetscOptionsSetValue("-mg_levels_pc_type", "fieldsplit"); > > > PetscOptionsSetValue("-mg_levels_pc_fieldsplit_detect_saddle_point", > > > PETSC_NULL); > > > PetscOptionsSetValue("-mg_levels_pc_fieldsplit_type", "multiplicative"); > > > PetscOptionsSetValue("-mg_levels_fieldsplit_0_ksp_type", "preonly"); > > > PetscOptionsSetValue("-mg_levels_fieldsplit_0_pc_type", "gamg"); > > > PetscOptionsSetValue("-mg_levels_fieldsplit_1_ksp_type", "preonly"); > > > PetscOptionsSetValue("-mg_levels_fieldsplit_1_pc_type", "jacobi"); > > > ******* > > > I get the following error: > > > Always send the full stack trace, when reporting errors. > > Finally, in the future you might want to post these questions to > petsc-users at mcs.anl.gov > > Dmitry. > > > > ******* > > > PETSC ERROR: Null argument, when expecting valid pointer! > > > PETSC ERROR: Null Object: Parameter # 2! > > > PETSC ERROR: MatPtAP() line 8237 in > > petsc-3.3-p4/src/mat/interface/matrix.c > > > ******* > > > > > > Do you have any suggestion? What are you currently using as > > preconditioner > > > for saddle point problems? > > > > Apologies, it appears that this email fell through the cracks > > somewhere back around the holidays. > > > > Did you ever find a satisfactory answer to your question? > > > > Petsc developers may occasionally read this list, but for questions > > that are this advanced you would probably have better luck mailing the > > petsc developers list directly. > > > > And of course if you do find a good block preconditioning strategy for > > systems_of_equations_ex1, please feel free to report it here! > > > > -- > > John > > > > > > ------------------------------------------------------------------------------ > > Everyone hates slow websites. So do we. > > Make your web apps faster with AppDynamics > > Download AppDynamics Lite for free today: > > http://p.sf.net/sfu/appdyn_d2d_jan > > _______________________________________________ > > Libmesh-users mailing list > > Libmesh-users at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/libmesh-users > > > > > > -- > Dmitry Karpeev, Ph.D. > Assistant Computational Mathematician > Mathematics and Computer Science > Argonne National Laboratory > Argonne, Illinois, USA > and > Fellow > Computation Institute > University of Chicago > 5735 S. Ellis Avenue > Chicago, IL 60637 > ----------------------- > Phone: 630-252-1229 > Fax: 630-252-5986 From jedbrown at mcs.anl.gov Mon Feb 4 12:46:22 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 4 Feb 2013 12:46:22 -0600 Subject: [petsc-users] setting name for matlab objects In-Reply-To: References: Message-ID: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObjectSetName.html On Mon, Feb 4, 2013 at 12:31 PM, Nachiket Gokhale wrote: > I am outputting some matrices and vectors to matlab using the ascii > viewer and setting format to matlab. However, the objects created in > matlab have names like > > Mat_0x84000001_0 > Vec_0x84000001_5 > > Is it possible for the user to set the name? It seems to be possible > for Mathematica: > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerMathematicaSetName.html > > > -Nachiket > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Feb 4 13:01:48 2013 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 4 Feb 2013 14:01:48 -0500 Subject: [petsc-users] [Libmesh-users] Preconditioning of a Stokes problem In-Reply-To: <20130204184553.GA14979@ices.utexas.edu> References: <20130204184553.GA14979@ices.utexas.edu> Message-ID: On Mon, Feb 4, 2013 at 1:45 PM, Tobin Isaac wrote: > On Mon, Feb 04, 2013 at 12:30:39PM -0600, Dmitry Karpeev wrote: > > Francesco, > > > > Are you using any pressure stabilization scheme? If not, the 11-block in > > your Jacobian > > A = [A00 A01; A10 A11] would typically be zero, and preconditioning it > with > > jacobi wouldn't really work. > > > > If A11 = 0, you ought to use the fieldsplit of type Schur, for example > > (assuming petsc-3.3; prior versions use slightly different option names): > > -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type full > > The question is then how you want to solve the Schur complement system. > > Since the A11 block for you is zero, the easiest choice is to use S as > the > > preconditioning matrix: > > -pc_fieldsplit_schur_precondition self > > > > Preconditioning options for the Schur solver have prefix -fieldsplit_1_. > > Since for Stokes the Schur complement is spectrally equivalent to the > > pressure mass matrix, you can > > hope that an unpreconditioned GMRES (default when using > > -pc_fieldsplit_schur_precondition self) will solve it. > > > > > > A better choice might be to use the Least Squares Commutator > preconditioner > > -fieldsplit_1_pc_type lsc > > See > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCLSC.html > > and note the relevant references at the bottom of that manual page. > > You might also find this useful: > > http://www.icfd.rdg.ac.uk/ICFD25/Talks/AWathen.pdf > > > > PCLSC preconditions S = A10 inv(A00) A01 with inv(A10A01) A10 A00 A01 > > inv(A10A01) > > The "Laplacian" (a true Laplacian in the case of incompressible Stokes) > can > > be preconditioned > > using AMG: > > -fieldsplit_1_lsc_pc_type ml > > (assuming you have ML built, e.g., with --download-ml) > > or > > -fieldsplit_1_lsc_pc_type gamg > > with some appropriate choice of gamg options: > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMG.html > > > > To start with I would use -pc_fieldsplit_type schur > > -pc_fieldsplit_schur_fact_type full -pc_fieldsplit_schur_precondition > self > > -fieldsplit_1_pc_type lsc -fieldsplit_1_lsc_pc_type ml > > gamg may also be very good in place of ml, but will require more option > > setting. > > > > > > You might find src/ksp/ksp/examples/tutorials/ex42.c and ex43.c useful, > > albeit, possibly, a bit heavyweight for your needs. > > Also, if your problem isn't stabilized then your vectors (hopefully) > aren't layed out in strided form. A good example of how to setup a > fieldsplit preconditioner in this more general setting is > src/snes/examples/tutorials/ex70.c Note that this is also done for unstructured mesh Stokes in SNES ex62, and you can see all the tests here: https://bitbucket.org/petsc/petsc-dev/src/e273ef2c2517b1e5787fa19bd0738bf91d2229e2/config/builder.py?at=default#cl-301 Thanks, Matt > > > > A few more comments below: > > > > On Mon, Feb 4, 2013 at 10:20 AM, John Peterson > wrote: > > > > > On Thu, Dec 27, 2012 at 4:31 AM, Francesco Ballarin > > > wrote: > > > > Dear libMesh users, > > > > I am trying to solve a Stokes problem in a 3D domain, but I am > currently > > > > struggling with the choice of the preconditioner. > > > > I have slightly modified systems_of_equations_ex1 to accommodate for > the > > > > third velocity component and boundary conditions of the problem at > hand > > > > (Dirichlet BCs on the inlet on the walls, enforced using a > > > > DirichletBoundary object, and homogeneous Neumann BC on the outflow) > > > > > > > > I am trying to use PETSc block preconditioners as shown, e.g., in > > > > http://acts.nersc.gov/events/Workshop2012/slides/PETSc.pdf > > > > section 4. I have tried most of the preconditioners discussed during > the > > > > talk, but I am not satisfied because I get either PETSc errors or a > > > > performance that I hope can be increased. > > > > > > > > For example, with the following options: > > > > ******* > > > > PetscOptionsSetValue("-ksp_type", "gmres"); > > > > PetscOptionsSetValue("-ksp_right_pc", PETSC_NULL); > > > > PetscOptionsSetValue("-pc_type", "fieldsplit"); > > > > PetscOptionsSetValue("-pc_fieldsplit_detect_saddle_point", > PETSC_NULL); > > > > PetscOptionsSetValue("-pc_fieldsplit_type", "multiplicative"); > > > > PetscOptionsSetValue("-fieldsplit_0_ksp_type", "preonly"); > > > > PetscOptionsSetValue("-fieldsplit_0_pc_type", "ml"); > > > > PetscOptionsSetValue("-fieldsplit_1_ksp_type", "preonly"); > > > > PetscOptionsSetValue("-fieldsplit_1_pc_type", "jacobi"); > > > > > > > You do not need to set options to PETSC_NULL, if you do not intend to use > > them. > > Also, how are you forming the splits? > > -pc_fieldsplit_detect_saddle_point is actually rather useful here. > > > > Further, there are programmatic ways to set most of these options, for > > example: > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetType.html > > > > Even so, we typically recommend setting these on the command line > whenever > > possible: > > this way you do not hardwire your solver choices and can quickly > experiment > > with different > > solvers from the command-line without recompiling your code. > > > > > > > > ******* > > > > I obtain a reasonable solution but after thousands of gmres > iterations. > > > Is > > > > there any way to improve that? > > > > > > > > Or, if I try to embed the previous block preconditioner in mg: > > > > ******* > > > > PetscOptionsSetValue("-pc_type", "mg"); > > > > PetscOptionsSetValue("-pc_mg_levels", "5"); > > > > PetscOptionsSetValue("-pc_mg_galerkin", PETSC_NULL); > > > > PetscOptionsSetValue("-ksp_type", "gmres"); > > > > PetscOptionsSetValue("-mg_levels_pc_type", "fieldsplit"); > > > > PetscOptionsSetValue("-mg_levels_pc_fieldsplit_detect_saddle_point", > > > > PETSC_NULL); > > > > PetscOptionsSetValue("-mg_levels_pc_fieldsplit_type", > "multiplicative"); > > > > PetscOptionsSetValue("-mg_levels_fieldsplit_0_ksp_type", "preonly"); > > > > PetscOptionsSetValue("-mg_levels_fieldsplit_0_pc_type", "gamg"); > > > > PetscOptionsSetValue("-mg_levels_fieldsplit_1_ksp_type", "preonly"); > > > > PetscOptionsSetValue("-mg_levels_fieldsplit_1_pc_type", "jacobi"); > > > > ******* > > > > I get the following error: > > > > > Always send the full stack trace, when reporting errors. > > > > Finally, in the future you might want to post these questions to > > petsc-users at mcs.anl.gov > > > > Dmitry. > > > > > > ******* > > > > PETSC ERROR: Null argument, when expecting valid pointer! > > > > PETSC ERROR: Null Object: Parameter # 2! > > > > PETSC ERROR: MatPtAP() line 8237 in > > > petsc-3.3-p4/src/mat/interface/matrix.c > > > > ******* > > > > > > > > Do you have any suggestion? What are you currently using as > > > preconditioner > > > > for saddle point problems? > > > > > > Apologies, it appears that this email fell through the cracks > > > somewhere back around the holidays. > > > > > > Did you ever find a satisfactory answer to your question? > > > > > > Petsc developers may occasionally read this list, but for questions > > > that are this advanced you would probably have better luck mailing the > > > petsc developers list directly. > > > > > > And of course if you do find a good block preconditioning strategy for > > > systems_of_equations_ex1, please feel free to report it here! > > > > > > -- > > > John > > > > > > > > > > ------------------------------------------------------------------------------ > > > Everyone hates slow websites. So do we. > > > Make your web apps faster with AppDynamics > > > Download AppDynamics Lite for free today: > > > http://p.sf.net/sfu/appdyn_d2d_jan > > > _______________________________________________ > > > Libmesh-users mailing list > > > Libmesh-users at lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/libmesh-users > > > > > > > > > > > -- > > Dmitry Karpeev, Ph.D. > > Assistant Computational Mathematician > > Mathematics and Computer Science > > Argonne National Laboratory > > Argonne, Illinois, USA > > and > > Fellow > > Computation Institute > > University of Chicago > > 5735 S. Ellis Avenue > > Chicago, IL 60637 > > ----------------------- > > Phone: 630-252-1229 > > Fax: 630-252-5986 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From slivkaje at gmail.com Mon Feb 4 14:23:08 2013 From: slivkaje at gmail.com (Jelena Slivka) Date: Mon, 4 Feb 2013 15:23:08 -0500 Subject: [petsc-users] Solving A*X = B where A and B are matrices In-Reply-To: <8C7CCD12-F869-4FE8-9DEA-0BBA1283DAEC@mcs.anl.gov> References: <55DD94DF-150F-4917-AA26-C0680107E065@mcs.anl.gov> <8C7CCD12-F869-4FE8-9DEA-0BBA1283DAEC@mcs.anl.gov> Message-ID: Hello, I apologize for asking the very similar question again. I have a problem in which the greatest bottleneck is the Trace(A\B) function. Thus I am trying to run this function in parallel. If I understand correctly, MatMatSolve is a sequential operation, am I right? I have tried the following approach - I divide the matrix B by columns, so basically I solve a bunch of linear systems of type A\b where A is a matrix and b is a vector (column of matrix B). Since these A\b operations are independent, I would like to run them all in parallel. In my experiment I have: 1) loaded the whole matrix A on each process 2) loaded matrix B as shared on all processes 3) transposed matrix B On each separate process in parallel: - run Cholesky factorization (only for the first A\b solve) - Solve linear systems A\b_i for each row i of matrix B that is stored on that process - Send the partial trace result to the main process Please find attached the complete code. My question is about scalability of this solution. I see the improvement when running the code on 20 nodes as opposed to 10 nodes, but if I run the code on 40 nodes, my calculation time is even worse then calculation time on 20 nodes. Please find attached my log summary for 10 nodes, 20 nodes and 40 nodes. I should maybe point out that I am only interested in speeding up the Trace(A\B) calculation time (noted with "Trace calculations took: " in my output). Could you please help me figure out what is causing my performance to worsen when using more nodes? I am sorry if this is a very basic question, I am fairly new to PETSc. Grateful in advance, Jelena On Mon, Dec 3, 2012 at 2:21 PM, Barry Smith wrote: > > http://www.mcs.anl.gov/petsc/documentation/faq.html#computers > > On Dec 3, 2012, at 1:08 PM, Jelena Slivka wrote: > > > Thank you very much! > > However, I have another question. I have a cluster of 4 nodes and each > node has 6 cores. If I run my code using 6 cores on one node (using the > command "mpiexec -n 6") it is much faster than running it on just one > process (which is expected). However, if I try running the code on multiple > nodes (using "mpiexec -f machinefile -ppn 4", where machinefile is the file > which contains the node names), it runs much slower than on just one > process. This also happens with tutorial examples. I have checked the > number of iteration for KSP solver when spread on multiple processors and > it doesn't seem to be the problem. Do you have any suggestions on what am I > doing wrong? Are the commands I am using wrong? > > > > > > On Sat, Dec 1, 2012 at 6:03 PM, Barry Smith wrote: > > > > We recommend following the directions > http://www.mcs.anl.gov/petsc/documentation/faq.html#schurcomplement for > computing a Schur complement; just skip the unneeded step. MUMPS supports a > parallel Cholesky but you can also use a parallel LU with MUMPS, PaSTIX or > SuperLU_Dist and those will work fine also. With current software Cholesky > in parallel is not tons better than LU so generally not worth monkeying > with. > > > > Barry > > > > > > On Dec 1, 2012, at 12:05 PM, Jelena Slivka wrote: > > > > > Hello! > > > I am trying to solve A*X = B where A and B are matrices, and then find > trace of the resulting matrix X. My approach has been to partition matrix B > in column vectors bi and then solve each system A*xi = bi. Then, for all > vectors xi I would extract i-th element xi(i) and sum those elements in > order to get Trace(X). > > > Pseudo-code: > > > 1) load matrices A and B > > > 2) transpose matrix B (so that each right-hand side bi is in the row, > as operation MatGetColumnVector is slow) > > > 3) set up KSPSolve > > > 4) create vector diagonal (in which xi(i) elements will be stored) > > > 5) for each row i of matrix B owned by current process: > > > - create vector bi by extracting row i from matrix B > > > - apply KSPsolve to get xi > > > - insert value xi(i) in diagonal vector (only the process > which > > > holds the ith value of vector x(i) should do so) > > > 6) sum vector diagonal to get the trace. > > > However, my code (attached, along with the test case) runs fine on one > process, but hangs if started on multiple processes. Could you please help > me figure out what am I doing wrong? > > > Also, could you please tell me is it possible to use Cholesky > factorization when running on multiple processes (I see that I cannot use > it when I set the format of matrix A to MPIAIJ)? > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: log_10nodes.log Type: application/octet-stream Size: 11461 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: log_20nodes.log Type: application/octet-stream Size: 11463 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: log_40nodes.log Type: application/octet-stream Size: 11462 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ComputeTraceParallelKSP.c Type: text/x-csrc Size: 3720 bytes Desc: not available URL: From rupp at mcs.anl.gov Mon Feb 4 14:32:18 2013 From: rupp at mcs.anl.gov (Karl Rupp) Date: Mon, 04 Feb 2013 14:32:18 -0600 Subject: [petsc-users] Solving A*X = B where A and B are matrices In-Reply-To: References: <55DD94DF-150F-4917-AA26-C0680107E065@mcs.anl.gov> <8C7CCD12-F869-4FE8-9DEA-0BBA1283DAEC@mcs.anl.gov> Message-ID: <51101AD2.9040003@mcs.anl.gov> Hi Jelena, there are two things to note: a) From the logs: ########################################################## # # # WARNING!!! # # # # This code was compiled with a debugging option, # # To get timing results run ./configure # # using --with-debugging=no, the performance will # # be generally two or three times faster. # # # ########################################################## b) It seems like you are transposing the full matrix on every process. I doubt that this is what you really want, as it eats up about 45 percent of the total time in the case of 40 processes. You should instead load the entries into the rhs-vector just when you need it. Best regards, Karli On 02/04/2013 02:23 PM, Jelena Slivka wrote: > Hello, > I apologize for asking the very similar question again. I have a problem > in which the greatest bottleneck is the Trace(A\B) function. Thus I am > trying to run this function in parallel. If I understand correctly, > MatMatSolve is a sequential operation, am I right? > I have tried the following approach - I divide the matrix B by columns, > so basically I solve a bunch of linear systems of type A\b where A is a > matrix and b is a vector (column of matrix B). Since these A\b > operations are independent, I would like to run them all in parallel. > In my experiment I have: > > 1) loaded the whole matrix A on each process > 2) loaded matrix B as shared on all processes > 3) transposed matrix B > On each separate process in parallel: > - run Cholesky factorization (only for the first A\b solve) > - Solve linear systems A\b_i for each row i of matrix B that is > stored on that process > - Send the partial trace result to the main process > Please find attached the complete code. > > My question is about scalability of this solution. I see the improvement > when running the code on 20 nodes as opposed to 10 nodes, but if I run > the code on 40 nodes, my calculation time is even worse then calculation > time on 20 nodes. Please find attached my log summary for 10 nodes, 20 > nodes and 40 nodes. I should maybe point out that I am only interested > in speeding up the Trace(A\B) calculation time (noted with "Trace > calculations took: " in my output). > Could you please help me figure out what is causing my performance to > worsen when using more nodes? I am sorry if this is a very basic > question, I am fairly new to PETSc. > Grateful in advance, > Jelena > > > > > > > > > On Mon, Dec 3, 2012 at 2:21 PM, Barry Smith > wrote: > > > http://www.mcs.anl.gov/petsc/documentation/faq.html#computers > > On Dec 3, 2012, at 1:08 PM, Jelena Slivka > wrote: > > > Thank you very much! > > However, I have another question. I have a cluster of 4 nodes and > each node has 6 cores. If I run my code using 6 cores on one node > (using the command "mpiexec -n 6") it is much faster than running it > on just one process (which is expected). However, if I try running > the code on multiple nodes (using "mpiexec -f machinefile -ppn 4", > where machinefile is the file which contains the node names), it > runs much slower than on just one process. This also happens with > tutorial examples. I have checked the number of iteration for KSP > solver when spread on multiple processors and it doesn't seem to be > the problem. Do you have any suggestions on what am I doing wrong? > Are the commands I am using wrong? > > > > > > On Sat, Dec 1, 2012 at 6:03 PM, Barry Smith > wrote: > > > > We recommend following the directions > http://www.mcs.anl.gov/petsc/documentation/faq.html#schurcomplement > for computing a Schur complement; just skip the unneeded step. > MUMPS supports a parallel Cholesky but you can also use a parallel > LU with MUMPS, PaSTIX or SuperLU_Dist and those will work fine also. > With current software Cholesky in parallel is not tons better than > LU so generally not worth monkeying with. > > > > Barry > > > > > > On Dec 1, 2012, at 12:05 PM, Jelena Slivka > wrote: > > > > > Hello! > > > I am trying to solve A*X = B where A and B are matrices, and > then find trace of the resulting matrix X. My approach has been to > partition matrix B in column vectors bi and then solve each system > A*xi = bi. Then, for all vectors xi I would extract i-th element > xi(i) and sum those elements in order to get Trace(X). > > > Pseudo-code: > > > 1) load matrices A and B > > > 2) transpose matrix B (so that each right-hand side bi is in > the row, as operation MatGetColumnVector is slow) > > > 3) set up KSPSolve > > > 4) create vector diagonal (in which xi(i) elements will be stored) > > > 5) for each row i of matrix B owned by current process: > > > - create vector bi by extracting row i from matrix B > > > - apply KSPsolve to get xi > > > - insert value xi(i) in diagonal vector (only the > process which > > > holds the ith value of vector x(i) should do so) > > > 6) sum vector diagonal to get the trace. > > > However, my code (attached, along with the test case) runs fine > on one process, but hangs if started on multiple processes. Could > you please help me figure out what am I doing wrong? > > > Also, could you please tell me is it possible to use Cholesky > factorization when running on multiple processes (I see that I > cannot use it when I set the format of matrix A to MPIAIJ)? > > > > > > > > > > > > From wlowrie at uw.edu Mon Feb 4 16:01:27 2013 From: wlowrie at uw.edu (Weston Lowrie) Date: Mon, 4 Feb 2013 17:01:27 -0500 Subject: [petsc-users] petsc4py Vec().setSizes() Message-ID: Hi, I'm confused what the Vec().setSizes() routine is doing in petsc4py. Consider this example: #!/usr/bin/env python import sys,os from petsc4py import PETSc from numpy import * mpi_rank = PETSc.COMM_WORLD.getRank() mpi_size = PETSc.COMM_WORLD.getSize() sizes = zeros(4) sizes[0] = 35675 sizes[1] = 365510 sizes[2] = 365742 sizes[3] = 35443 X = PETSc.Vec().create(comm=PETSc.COMM_WORLD) X.setSizes(mpi_size*sizes[mpi_rank],bsize=1) X.setFromOptions() ilow,ihigh = X.getOwnershipRange() print "rank: ",mpi_rank,"low/high: ",ilow,ihigh Why is it that when setting the local sizes explicitly do I need to multiply by the mpi_size? My understanding is that when using this routine it is telling PETSc what the local processor core size should be. It seems to divide it by total number of processors cores. Thanks, Wes -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Feb 4 17:40:46 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 4 Feb 2013 17:40:46 -0600 Subject: [petsc-users] petsc4py Vec().setSizes() In-Reply-To: References: Message-ID: petsc4py is "too clever" in the sense that it tries to interpret many different kids of "sizes" arguments. You can always pass the pair (localsize, globalsize), but you can also pass only a global size (in which case the vector will be split apart). If you want to set only the local size, you should pass (localsize, None). Your example is invalid, with each process passing different global sizes. petsc-dev will now error if you do this. I changed your example to: X = PETSc.Vec().create(comm=PETSc.COMM_WORLD) X.setSizes((sizes[mpi_rank],PETSc.DECIDE),bsize=1) X.setFromOptions() ilow,ihigh = X.getOwnershipRange() PETSc.Sys.syncPrint("rank: ",mpi_rank,"low/high: ",ilow,ihigh) PETSc.Sys.syncFlush() and now get the output: rank: 0 low/high: 0 35675 rank: 1 low/high: 35675 401185 rank: 2 low/high: 401185 766927 rank: 3 low/high: 766927 802370 On Mon, Feb 4, 2013 at 4:01 PM, Weston Lowrie wrote: > Hi, > I'm confused what the Vec().setSizes() routine is doing in petsc4py. > Consider this example: > > #!/usr/bin/env python > import sys,os > from petsc4py import PETSc > from numpy import * > > mpi_rank = PETSc.COMM_WORLD.getRank() > mpi_size = PETSc.COMM_WORLD.getSize() > > sizes = zeros(4) > sizes[0] = 35675 > sizes[1] = 365510 > sizes[2] = 365742 > sizes[3] = 35443 > > X = PETSc.Vec().create(comm=PETSc.COMM_WORLD) > X.setSizes(mpi_size*sizes[mpi_rank],bsize=1) > X.setFromOptions() > ilow,ihigh = X.getOwnershipRange() > > print "rank: ",mpi_rank,"low/high: ",ilow,ihigh > > > > Why is it that when setting the local sizes explicitly do I need to > multiply by the mpi_size? My understanding is that when using this routine > it is telling PETSc what the local processor core size should be. It seems > to divide it by total number of processors cores. > > Thanks, > Wes > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wlowrie at uw.edu Tue Feb 5 07:59:38 2013 From: wlowrie at uw.edu (Weston Lowrie) Date: Tue, 5 Feb 2013 08:59:38 -0500 Subject: [petsc-users] petsc4py Vec().setSizes() In-Reply-To: References: Message-ID: That makes sense now. It was just creating 'mpi_size' (in my case 4) separate Vec objects and distributing them evenly among the processes. That is why it appeared to give the correct sizes. Good to know about the petsc4py "size" argument. It was unclear to me that it could take multiple values, but that makes sense to keep the API general and include as much of the functionality of the C++/Fortran APIs. I have been referencing this: http://packages.python.org/petsc4py/apiref/index.html where it shows: setSizes(self, size, bsize=None) Maybe there is a better reference somewhere, or I missed where it describes the size argument? Thanks, Wes On Mon, Feb 4, 2013 at 6:40 PM, Jed Brown wrote: > petsc4py is "too clever" in the sense that it tries to interpret many > different kids of "sizes" arguments. You can always pass the pair > (localsize, globalsize), but you can also pass only a global size (in which > case the vector will be split apart). If you want to set only the local > size, you should pass (localsize, None). > > Your example is invalid, with each process passing different global sizes. > petsc-dev will now error if you do this. > > I changed your example to: > > > X = PETSc.Vec().create(comm=PETSc.COMM_WORLD) > X.setSizes((sizes[mpi_rank],PETSc.DECIDE),bsize=1) > X.setFromOptions() > ilow,ihigh = X.getOwnershipRange() > > PETSc.Sys.syncPrint("rank: ",mpi_rank,"low/high: ",ilow,ihigh) > PETSc.Sys.syncFlush() > > and now get the output: > > rank: 0 low/high: 0 35675 > rank: 1 low/high: 35675 401185 > rank: 2 low/high: 401185 766927 > rank: 3 low/high: 766927 802370 > > > > On Mon, Feb 4, 2013 at 4:01 PM, Weston Lowrie wrote: > >> Hi, >> I'm confused what the Vec().setSizes() routine is doing in petsc4py. >> Consider this example: >> >> #!/usr/bin/env python >> import sys,os >> from petsc4py import PETSc >> from numpy import * >> >> mpi_rank = PETSc.COMM_WORLD.getRank() >> mpi_size = PETSc.COMM_WORLD.getSize() >> >> sizes = zeros(4) >> sizes[0] = 35675 >> sizes[1] = 365510 >> sizes[2] = 365742 >> sizes[3] = 35443 >> >> X = PETSc.Vec().create(comm=PETSc.COMM_WORLD) >> X.setSizes(mpi_size*sizes[mpi_rank],bsize=1) >> X.setFromOptions() >> ilow,ihigh = X.getOwnershipRange() >> >> print "rank: ",mpi_rank,"low/high: ",ilow,ihigh >> >> >> >> Why is it that when setting the local sizes explicitly do I need to >> multiply by the mpi_size? My understanding is that when using this routine >> it is telling PETSc what the local processor core size should be. It seems >> to divide it by total number of processors cores. >> >> Thanks, >> Wes >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Feb 5 08:09:46 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 5 Feb 2013 08:09:46 -0600 Subject: [petsc-users] petsc4py Vec().setSizes() In-Reply-To: References: Message-ID: On Tue, Feb 5, 2013 at 7:59 AM, Weston Lowrie wrote: > That makes sense now. It was just creating 'mpi_size' (in my case 4) > separate Vec objects and distributing them evenly among the processes. > That is why it appeared to give the correct sizes. > > Good to know about the petsc4py "size" argument. It was unclear to me > that it could take multiple values, but that makes sense to keep the API > general and include as much of the functionality of the C++/Fortran APIs. > > I have been referencing this: > http://packages.python.org/petsc4py/apiref/index.html > > where it shows: > setSizes(self, size, bsize=None) > > Maybe there is a better reference somewhere, or I missed where it > describes the size argument? > AFAIK, it's not documented. Most of petsc4py is a direct mapping from PETSc and I think we (Lisandro) have been hoping to find a way to automatically generate Python docs. For those features that are actually different (like the size arguments), it would be very useful to write doc strings. (They would show up in the apiref above.) I'm sure Lisandro would be happy to accept documentation patches. > > Thanks, > Wes > > > > On Mon, Feb 4, 2013 at 6:40 PM, Jed Brown wrote: > >> petsc4py is "too clever" in the sense that it tries to interpret many >> different kids of "sizes" arguments. You can always pass the pair >> (localsize, globalsize), but you can also pass only a global size (in which >> case the vector will be split apart). If you want to set only the local >> size, you should pass (localsize, None). >> >> Your example is invalid, with each process passing different global >> sizes. petsc-dev will now error if you do this. >> >> I changed your example to: >> >> >> X = PETSc.Vec().create(comm=PETSc.COMM_WORLD) >> X.setSizes((sizes[mpi_rank],PETSc.DECIDE),bsize=1) >> X.setFromOptions() >> ilow,ihigh = X.getOwnershipRange() >> >> PETSc.Sys.syncPrint("rank: ",mpi_rank,"low/high: ",ilow,ihigh) >> PETSc.Sys.syncFlush() >> >> and now get the output: >> >> rank: 0 low/high: 0 35675 >> rank: 1 low/high: 35675 401185 >> rank: 2 low/high: 401185 766927 >> rank: 3 low/high: 766927 802370 >> >> >> >> On Mon, Feb 4, 2013 at 4:01 PM, Weston Lowrie wrote: >> >>> Hi, >>> I'm confused what the Vec().setSizes() routine is doing in petsc4py. >>> Consider this example: >>> >>> #!/usr/bin/env python >>> import sys,os >>> from petsc4py import PETSc >>> from numpy import * >>> >>> mpi_rank = PETSc.COMM_WORLD.getRank() >>> mpi_size = PETSc.COMM_WORLD.getSize() >>> >>> sizes = zeros(4) >>> sizes[0] = 35675 >>> sizes[1] = 365510 >>> sizes[2] = 365742 >>> sizes[3] = 35443 >>> >>> X = PETSc.Vec().create(comm=PETSc.COMM_WORLD) >>> X.setSizes(mpi_size*sizes[mpi_rank],bsize=1) >>> X.setFromOptions() >>> ilow,ihigh = X.getOwnershipRange() >>> >>> print "rank: ",mpi_rank,"low/high: ",ilow,ihigh >>> >>> >>> >>> Why is it that when setting the local sizes explicitly do I need to >>> multiply by the mpi_size? My understanding is that when using this routine >>> it is telling PETSc what the local processor core size should be. It seems >>> to divide it by total number of processors cores. >>> >>> Thanks, >>> Wes >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From w_ang_temp at 163.com Tue Feb 5 13:27:57 2013 From: w_ang_temp at 163.com (w_ang_temp) Date: Wed, 6 Feb 2013 03:27:57 +0800 (CST) Subject: [petsc-users] The condition number Message-ID: <26210d64.8dd1.13cabd438e7.Coremail.w_ang_temp@163.com> Hello, I want to get the condition number of the matrix. I use the command: mpiexec -n 1 ./ex4f -ksp_type gmres -pc_type none -ksp_monitor_singular_value -ksp_gmres_restart 10000 The information is: 287 KSP Residual norm 2.709682524692e-05 % max 2.926321440259e+04 min 5.402536868919e-01 max/min 5.416569125321e+04 288 KSP Residual norm 2.519306234382e-05 % max 2.926321440259e+04 min 5.266603977255e-01 max/min 5.556372669936e+04 289 KSP Residual norm 2.378149039323e-05 % max 2.926321440259e+04 min 5.172748873301e-01 max/min 5.657188299559e+04 290 KSP Residual norm 2.287472821678e-05 % max 2.926321440260e+04 min 5.115884179685e-01 max/min 5.720069762095e+04 291 KSP Residual norm 2.228155716304e-05 % max 2.926321440260e+04 min 5.080217090257e-01 max/min 5.760229116728e+04 292 KSP Residual norm 2.192114571943e-05 % max 2.926321440260e+04 min 5.059182789026e-01 max/min 5.784178121826e+04 293 KSP Residual norm 2.167188877219e-05 % max 2.926321440260e+04 min 5.044948817455e-01 max/min 5.800497777371e+04 294 KSP Residual norm 2.148234628716e-05 % max 2.926321440260e+04 min 5.034325309589e-01 max/min 5.812738073732e+04 295 KSP Residual norm 2.132127329266e-05 % max 2.926321440260e+04 min 5.025455984087e-01 max/min 5.822996857452e+04 296 KSP Residual norm 2.113112645062e-05 % max 2.926321440260e+04 min 5.015166226585e-01 max/min 5.834944063764e+04 Linear solve converged due to CONVERGED_RTOL iterations 296 So is the condition number: max/min 5.834944063764e+04? The value of the last iteration? Thanks. Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Feb 5 13:40:16 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 5 Feb 2013 13:40:16 -0600 Subject: [petsc-users] The condition number In-Reply-To: <26210d64.8dd1.13cabd438e7.Coremail.w_ang_temp@163.com> References: <26210d64.8dd1.13cabd438e7.Coremail.w_ang_temp@163.com> Message-ID: <1729A593-2A74-4C76-8650-B0FF22050E20@mcs.anl.gov> With this type of condition number estimator there is no certainty that this is exactly correct, it may be a bit higher but likely this is a good estimate of the condition number. Barry On Feb 5, 2013, at 1:27 PM, w_ang_temp wrote: > Hello, > I want to get the condition number of the matrix. I use the command: > mpiexec -n 1 ./ex4f -ksp_type gmres -pc_type none -ksp_monitor_singular_value -ksp_gmres_restart 10000 > The information is: > 287 KSP Residual norm 2.709682524692e-05 % max 2.926321440259e+04 min 5.402536868919e-01 max/min 5.416569125321e+04 > 288 KSP Residual norm 2.519306234382e-05 % max 2.926321440259e+04 min 5.266603977255e-01 max/min 5.556372669936e+04 > 289 KSP Residual norm 2.378149039323e-05 % max 2.926321440259e+04 min 5.172748873301e-01 max/min 5.657188299559e+04 > 290 KSP Residual norm 2.287472821678e-05 % max 2.926321440260e+04 min 5.115884179685e-01 max/min 5.720069762095e+04 > 291 KSP Residual norm 2.228155716304e-05 % max 2.926321440260e+04 min 5.080217090257e-01 max/min 5.760229116728e+04 > 292 KSP Residual norm 2.192114571943e-05 % max 2.926321440260e+04 min 5.059182789026e-01 max/min 5.784178121826e+04 > 293 KSP Residual norm 2.167188877219e-05 % max 2.926321440260e+04 min 5.044948817455e-01 max/min 5.800497777371e+04 > 294 KSP Residual norm 2.148234628716e-05 % max 2.926321440260e+04 min 5.034325309589e-01 max/min 5.812738073732e+04 > 295 KSP Re sidual norm 2.132127329266e-05 % max 2.926321440260e+04 min 5.025455984087e-01 max/min 5.822996857452e+04 > 296 KSP Residual norm 2.113112645062e-05 % max 2.926321440260e+04 min 5.015166226585e-01 max/min 5.834944063764e+04 > Linear solve converged due to CONVERGED_RTOL iterations 296 > > So is the condition number: max/min 5.834944063764e+04? The value of the last iteration? > Thanks. > Jim > > From mark.adams at columbia.edu Wed Feb 6 15:41:03 2013 From: mark.adams at columbia.edu (Mark F. Adams) Date: Wed, 6 Feb 2013 16:41:03 -0500 Subject: [petsc-users] PETSC_VERSION_LT Message-ID: <9E9CD09D-3022-43C5-A9BA-7D1486EC8EB4@columbia.edu> I'm running with v3.3.00 and have this in my code: #ifdef PETSC_VERSION_LT(3,3,0) call MatCreateMPIAIJ( comm, PETSC_DECIDE, PETSC_DECIDE, nnodes, nnodes, & maxn1, PETSC_NULL_INTEGER, maxn1/2, PETSC_NULL_INTEGER, & AA, ierr ) #else call MatCreateAIJ( comm, PETSC_DECIDE, PETSC_DECIDE, nnodes, nnodes, & maxn1, PETSC_NULL_INTEGER, maxn1/2, PETSC_NULL_INTEGER, & AA, ierr ) #endif and I get errors like: /autofs/na1_home/adams/XGC1/XGC1_3/./module.F95:1189: undefined reference to `matcreatempiaij_' /autofs/na1_home/adams/XGC1/XGC1_3/./module.F95:1215: undefined reference to `matcreatempiaij_' /autofs/na1_home/adams/XGC1/XGC1_3/./module.F95:1235: undefined reference to `matcreatempiaij_' Am I not using this correctly? From jedbrown at mcs.anl.gov Wed Feb 6 18:49:24 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 6 Feb 2013 18:49:24 -0600 Subject: [petsc-users] PETSC_VERSION_LT In-Reply-To: <9E9CD09D-3022-43C5-A9BA-7D1486EC8EB4@columbia.edu> References: <9E9CD09D-3022-43C5-A9BA-7D1486EC8EB4@columbia.edu> Message-ID: On Wed, Feb 6, 2013 at 3:41 PM, Mark F. Adams wrote: > I'm running with v3.3.00 and have this in my code: > > #ifdef PETSC_VERSION_LT(3,3,0) > You need #if PETSC_VERSION_LT(3,3,0), not #ifdef (the macro is always defined, it just might be zero). Note that this macro was only put in just before petsc-3.3, so if you want to support versions older than 3.3, you'll need something like #if !defined(PETSC_VERSION_LT) # define PETSC_VERSION_LT(major,minor,subminor) ... #endif > call MatCreateMPIAIJ( comm, PETSC_DECIDE, PETSC_DECIDE, nnodes, > nnodes, & > maxn1, PETSC_NULL_INTEGER, maxn1/2, PETSC_NULL_INTEGER, & > AA, ierr ) > #else > call MatCreateAIJ( comm, PETSC_DECIDE, PETSC_DECIDE, nnodes, nnodes, & > maxn1, PETSC_NULL_INTEGER, maxn1/2, PETSC_NULL_INTEGER, & > AA, ierr ) > #endif > > > and I get errors like: > > /autofs/na1_home/adams/XGC1/XGC1_3/./module.F95:1189: undefined reference > to `matcreatempiaij_' > /autofs/na1_home/adams/XGC1/XGC1_3/./module.F95:1215: undefined reference > to `matcreatempiaij_' > /autofs/na1_home/adams/XGC1/XGC1_3/./module.F95:1235: undefined reference > to `matcreatempiaij_' > > > Am I not using this correctly? -------------- next part -------------- An HTML attachment was scrubbed... URL: From w_ang_temp at 163.com Wed Feb 6 23:11:27 2013 From: w_ang_temp at 163.com (w_ang_temp) Date: Thu, 7 Feb 2013 13:11:27 +0800 (CST) Subject: [petsc-users] Convergence is different with different processors Message-ID: <155b505e.2849.13cb310c848.Coremail.w_ang_temp@163.com> Hello, I use the same project, but I find that when different number of processors is choosed, the convergence is different. For example, when the processors are 4, it is divergent; when the processors are 8, it is convergent. So what is the reason? Thanks. Jim. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Feb 6 23:16:51 2013 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 7 Feb 2013 00:16:51 -0500 Subject: [petsc-users] Convergence is different with different processors In-Reply-To: <155b505e.2849.13cb310c848.Coremail.w_ang_temp@163.com> References: <155b505e.2849.13cb310c848.Coremail.w_ang_temp@163.com> Message-ID: On Thu, Feb 7, 2013 at 12:11 AM, w_ang_temp wrote: > Hello, > I use the same project, but I find that when different number of > processors is choosed, > the convergence is different. For example, when the processors are 4, it > is divergent; when > the processors are 8, it is convergent. > So what is the reason? > It is likely that your preconditioner changed. Matt > Thanks. > Jim. > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdettrick at gmail.com Wed Feb 6 23:28:27 2013 From: sdettrick at gmail.com (Sean Dettrick) Date: Wed, 6 Feb 2013 21:28:27 -0800 Subject: [petsc-users] PETSc on Blue Gene/Q ? Message-ID: <5BB2B923-DB1E-4036-8995-D90521F940EA@gmail.com> Dear Petsc Users, Does anyone have experience with running a PETSc app or benchmark on Blue Gene/Q? Does PETSc compile and run? Does it link to the IBM MASS libraries? How does a single Blue Gene CPU compare to a Xeon? Does your PETSc code scale well on BGQ? Thanks! Sean Dettrick From jedbrown at mcs.anl.gov Wed Feb 6 23:42:28 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 6 Feb 2013 23:42:28 -0600 Subject: [petsc-users] PETSc on Blue Gene/Q ? In-Reply-To: <5BB2B923-DB1E-4036-8995-D90521F940EA@gmail.com> References: <5BB2B923-DB1E-4036-8995-D90521F940EA@gmail.com> Message-ID: On Wed, Feb 6, 2013 at 11:28 PM, Sean Dettrick wrote: > Dear Petsc Users, > > Does anyone have experience with running a PETSc app or benchmark on Blue > Gene/Q? Does PETSc compile and run? We've been running PETSc on BG/Q for more than a year. > Does it link to the IBM MASS libraries? No, it's not relevant to PETSc library functionality, but applications are welcome to use it. > How does a single Blue Gene CPU compare to a Xeon? For PETSc sparse matrix-vector and vector kernels, memory bandwidth tells 90% of the story. Some applications have other parts of their code that do a lot of flops. Note that the vectorization primitives are different between BG and Intel. > Does your PETSc code scale well on BGQ? > BG/Q is probably the easiest architecture available for this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Sun Feb 10 06:10:38 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 10 Feb 2013 13:10:38 +0100 Subject: [petsc-users] Lower spectrum limit for Lanczos type eigensolvers Message-ID: <51178E3E.5080708@tudelft.nl> Dear all, I am trying to experiment with a block Lanczos eigensolver code of mine for symmetric systems. I have a practical question which I could not find a direct answer yet. For large sparse matrices, what should be the number of eigenvalues to be extracted with this Krylov subspace based solver? I am looking for a ratio or a rule of thumb like 'if you want k eigenvalues and your system size is n, then the ratio given by k/n should not exceed 0.01'. In this case '0.01' is just a coefficient, say r, any practical experience on choosing this coefficient r? I know Lanczos method is efficient on extracting extreme values over the spectrum but is there a, more or less, a practical limit like the one I tried to explain? Best regards, Umut From goncalo.albuquerque at insilicio.com Thu Feb 14 06:58:57 2013 From: goncalo.albuquerque at insilicio.com (=?ISO-8859-1?Q?Gon=E7alo_Albuquerque?=) Date: Thu, 14 Feb 2013 13:58:57 +0100 Subject: [petsc-users] Communication time with threading enabled. Message-ID: Dear All, I'm experimenting with PETSc hybrid MPI/OpenMP capabilities and I have run a rather simple test case (2D magnetostatic) using PETSc compiled with both OpenMPI and thread support (both PThreads and OpenMP) on a Ubuntu 12.04 system. I cannot figure out the results obtained when comparing runs made using the same number of MPI processes (2) and specifying either no threads (-threadcomm_type nothread) or 1 OpenMP thread (-threadcomm_type openmp -threadcomm_nthreads 1). I attached the logs of both runs. It seems that the communication time has literally exploded. A grep over the logs gives: No threading: Average time for MPI_Barrier(): 1.38283e-06 Average time for zero size MPI_Send(): 7.03335e-06 1 OpenMP thread: Average time for MPI_Barrier(): 0.00870218 Average time for zero size MPI_Send(): 0.00614798 The same things is occurring when running ksp ex5 (see attached logs). Any ideas as to what I'm missing? Many thanks in advance, Gon?alo -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: nothread.log Type: application/octet-stream Size: 9356 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: openmp_nthreads_1.log Type: application/octet-stream Size: 9375 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex5_nothread.log Type: application/octet-stream Size: 13199 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex5_openmp_nthreads_1.log Type: application/octet-stream Size: 13244 bytes Desc: not available URL: From abhyshr at mcs.anl.gov Thu Feb 14 10:32:02 2013 From: abhyshr at mcs.anl.gov (Shri) Date: Thu, 14 Feb 2013 10:32:02 -0600 Subject: [petsc-users] Communication time with threading enabled. In-Reply-To: References: Message-ID: This has something to do with OpenMPI. I cannot reproduce this issue with MPICH. Can you try switching to MPICH (--download-mpich). Shri On Feb 14, 2013, at 6:58 AM, Gon?alo Albuquerque wrote: > Dear All, > > I'm experimenting with PETSc hybrid MPI/OpenMP capabilities and I have run a rather simple test case (2D magnetostatic) using PETSc compiled with both OpenMPI and thread support (both PThreads and OpenMP) on a Ubuntu 12.04 system. I cannot figure out the results obtained when comparing runs made using the same number of MPI processes (2) and specifying either no threads (-threadcomm_type nothread) or 1 OpenMP thread (-threadcomm_type openmp -threadcomm_nthreads 1). I attached the logs of both runs. It seems that the communication time has literally exploded. A grep over the logs gives: > > No threading: > Average time for MPI_Barrier(): 1.38283e-06 > Average time for zero size MPI_Send(): 7.03335e-06 > > 1 OpenMP thread: > Average time for MPI_Barrier(): 0.00870218 > Average time for zero size MPI_Send(): 0.00614798 > > The same things is occurring when running ksp ex5 (see attached logs). > > Any ideas as to what I'm missing? > > Many thanks in advance, > > Gon?alo > From cchoudary at rnet-tech.com Thu Feb 14 12:22:26 2013 From: cchoudary at rnet-tech.com (Chekuri Choudary) Date: Thu, 14 Feb 2013 13:22:26 -0500 Subject: [petsc-users] Viewing local matrices Message-ID: <000001ce0ae0$3f23dcb0$bd6b9610$@rnet-tech.com> How can I view the local matrices on each processor. I am trying to understand how a DMDA matrix is being distributed among multiple processes. Thanks Chekuri -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Feb 14 12:32:15 2013 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 14 Feb 2013 13:32:15 -0500 Subject: [petsc-users] Viewing local matrices In-Reply-To: <000001ce0ae0$3f23dcb0$bd6b9610$@rnet-tech.com> References: <000001ce0ae0$3f23dcb0$bd6b9610$@rnet-tech.com> Message-ID: On Thu, Feb 14, 2013 at 1:22 PM, Chekuri Choudary wrote: > ** ** > > How can I view the local matrices on each processor. I am trying to > understand how a DMDA matrix is being distributed among multiple processes. > Rows are contiguous so viewing the global matrix shows you the local pieces. Matt > **** > > Thanks**** > > Chekuri**** > > ** ** > > ** ** > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenglun.wei at gmail.com Thu Feb 14 12:47:26 2013 From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei) Date: Thu, 14 Feb 2013 12:47:26 -0600 Subject: [petsc-users] Error when updating Message-ID: <511D313E.8060102@gmail.com> Dear all, I have a problem while updating the PETSC-dev. When I 'make' it, it says: libfast in: /work/zlwei/PETSc/petsc-dev/src/snes/f90-mod petscsnesmod.F:4.72: use petsckspdef 1 Fatal Error: Can't open module file 'petsckspdef.mod' for reading at (1): No such file or directory best, Alan From knepley at gmail.com Thu Feb 14 12:54:34 2013 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 14 Feb 2013 13:54:34 -0500 Subject: [petsc-users] Error when updating In-Reply-To: <511D313E.8060102@gmail.com> References: <511D313E.8060102@gmail.com> Message-ID: Always send make.log to petsc-maint. Matt On Thu, Feb 14, 2013 at 1:47 PM, Zhenglun (Alan) Wei wrote: > Dear all, > I have a problem while updating the PETSC-dev. When I 'make' it, it > says: > > libfast in: /work/zlwei/PETSc/petsc-dev/**src/snes/f90-mod > petscsnesmod.F:4.72: > > use petsckspdef > 1 > Fatal Error: Can't open module file 'petsckspdef.mod' for reading at (1): > No such file or directory > > best, > Alan > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Feb 14 13:00:16 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 14 Feb 2013 13:00:16 -0600 Subject: [petsc-users] Viewing local matrices In-Reply-To: References: <000001ce0ae0$3f23dcb0$bd6b9610$@rnet-tech.com> Message-ID: <77302C3A-8B42-451F-82E8-4259614AAA07@mcs.anl.gov> On Feb 14, 2013, at 12:32 PM, Matthew Knepley wrote: > On Thu, Feb 14, 2013 at 1:22 PM, Chekuri Choudary wrote: > > > How can I view the local matrices on each processor. I am trying to understand how a DMDA matrix is being distributed among multiple processes. > Consider 2d: the domain is split into rectangular subregions (depending on how many processors there are and how many grid points in the x and y directions). Each process is assigned one of these subregions. For any run you can use DMView(da,PETSC_VIEWER_STDOUT_(comm)); to see the decomposition. Now as always with PETSc vectors all the vector entries on the first process are numbered first, followed by the vector entries on the second, third etc. Similarly for matrices the rows associated with the vector entries on the first process are numbered first, then the rows associated with vector entries on the second etc. Thus the entries in the vector are not in the "natural ordering for the entire domain". But VecView() and MatView() for DMDA automatically reorder so that when you see the vectors and matrices they are in the natural ordering. You are over thinking things if you want to see the matrix in the "non-natural ordering", there is no reason to worry about that. Just use MatSetValuesLocal() or MatSetValuesStencil() to enter entries and the fact that PETSc internally uses this other non-natural ordering is not important in using it. What particular issue are you trying to resolve by "viewing the local matrices on each process"? Barry > > Rows are contiguous so viewing the global matrix shows you the local pieces. > > Matt > > > Thanks > > Chekuri > > > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From goncalo.albuquerque at insilicio.com Thu Feb 14 14:06:17 2013 From: goncalo.albuquerque at insilicio.com (=?ISO-8859-1?Q?Gon=E7alo_Albuquerque?=) Date: Thu, 14 Feb 2013 21:06:17 +0100 Subject: [petsc-users] Communication time with threading enabled. In-Reply-To: References: Message-ID: Dear Shri, Just tried with --download-mpich and all is well. Thanks. Confirmed that using --download-openmpi (instead of using Ubuntu's OpenMPI) produces the same slowdown in communications. Another visible symptom is that it takes a long time to print the log_summary data. Any hints as to why OpenMPI displays this behavior. Thank you again for your help. Regards, Gon?alo On Thu, Feb 14, 2013 at 5:32 PM, Shri wrote: > This has something to do with OpenMPI. I cannot reproduce this issue with > MPICH. Can you try switching to MPICH (--download-mpich). > > Shri > On Feb 14, 2013, at 6:58 AM, Gon?alo Albuquerque wrote: > > > Dear All, > > > > I'm experimenting with PETSc hybrid MPI/OpenMP capabilities and I have > run a rather simple test case (2D magnetostatic) using PETSc compiled with > both OpenMPI and thread support (both PThreads and OpenMP) on a Ubuntu > 12.04 system. I cannot figure out the results obtained when comparing runs > made using the same number of MPI processes (2) and specifying either no > threads (-threadcomm_type nothread) or 1 OpenMP thread (-threadcomm_type > openmp -threadcomm_nthreads 1). I attached the logs of both runs. It seems > that the communication time has literally exploded. A grep over the logs > gives: > > > > No threading: > > Average time for MPI_Barrier(): 1.38283e-06 > > Average time for zero size MPI_Send(): 7.03335e-06 > > > > 1 OpenMP thread: > > Average time for MPI_Barrier(): 0.00870218 > > Average time for zero size MPI_Send(): 0.00614798 > > > > The same things is occurring when running ksp ex5 (see attached logs). > > > > Any ideas as to what I'm missing? > > > > Many thanks in advance, > > > > Gon?alo > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lyh03259.aps at gmail.com Fri Feb 15 10:34:56 2013 From: lyh03259.aps at gmail.com (Yonghui) Date: Fri, 15 Feb 2013 10:34:56 -0600 Subject: [petsc-users] A question of VecSetValues Message-ID: <000901ce0b9a$654394c0$2fcabe40$@gmail.com> Dear PESTc users, I don't understand VecSetValues very well. I compiled ex2 and run with mpiexec -n 3. I got result like this: Vector Object: 3 MPI processes type: mpi Process [0] 4 Process [1] 4 4 Process [2] 4 3 2 Here is how I am comfused: RANK 0: do 100 i=0,0 call VecSetValues(x,1,i,1,ADD_VALUES,ierr) 100 continue RANK 1: do 100 i=0,1 call VecSetValues(x,1,i,1,ADD_VALUES,ierr) 100 continue RANK 2: do 100 i=0,2 call VecSetValues(x,1,i,1,ADD_VALUES,ierr) 100 continue I don't see any clue for getting the result above. Please show me how it works! Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Fri Feb 15 11:08:12 2013 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Fri, 15 Feb 2013 11:08:12 -0600 Subject: [petsc-users] A question of VecSetValues In-Reply-To: <000901ce0b9a$654394c0$2fcabe40$@gmail.com> References: <000901ce0b9a$654394c0$2fcabe40$@gmail.com> Message-ID: Yonghui : ... ierr = VecSet(x,one);CHKERRQ(ierr); // set x= [ 1, 1, 1, 1, 1, 1] RANK 0: do 100 i=0,N-1 (N=6) ^ call VecSetValues(x,1,i,1,ADD_VALUES,ierr) 100 continue // x = [2, 2, 2, 2, 2, 2] RANK 1: do 100 i=0,N-2 ^ call VecSetValues(x,1,i,1,ADD_VALUES,ierr) 100 continue // x = [3, 3, 3, 3, 3, 2] RANK 2: do 100 i=0,N-3 ^ call VecSetValues(x,1,i,1,ADD_VALUES,ierr) 100 continue // x = [4, 4, 4, 4, 3, 2] Hong > Dear PESTc users, > > > > I don?t understand VecSetValues very well. I compiled ex2 and run with > mpiexec ?n 3. I got result like this: > > > > Vector Object: 3 MPI processes > > type: mpi > > Process [0] > > 4 > > Process [1] > > 4 > > 4 > > Process [2] > > 4 > > 3 > > 2 > > > > Here is how I am comfused: > > RANK 0: > > do 100 i=0,0 > > call VecSetValues(x,1,i,1,ADD_VALUES,ierr) > > 100 continue > > > > RANK 1: > > do 100 i=0,1 > > call VecSetValues(x,1,i,1,ADD_VALUES,ierr) > > 100 continue > > > > RANK 2: > > do 100 i=0,2 > > call VecSetValues(x,1,i,1,ADD_VALUES,ierr) > > 100 continue > > > > I don?t see any clue for getting the result above. Please show me how it > works! Thanks. From mike.hui.zhang at hotmail.com Sat Feb 16 02:42:45 2013 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Sat, 16 Feb 2013 09:42:45 +0100 Subject: [petsc-users] No Matlab for complex numbers? Message-ID: When I configure --with-matlab=1, I got Cannot use Matlab with complex numbers it is not coded for this capability Why not supporting this capability? From mike.hui.zhang at hotmail.com Sat Feb 16 04:46:41 2013 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Sat, 16 Feb 2013 11:46:41 +0100 Subject: [petsc-users] VecDuplicate throwing error messages Message-ID: I updated petsc-dev just now with the last changeset: 26190:c5c1560f6126 I reconfigure and rebuild petsc. I modified src/sys/threadcomm/examples/tutorials/ex4.c to include a VecDuplicate like the following ediff file (see behind, ! indicating the modifications). Then I got error messages (using mpiexec -np 1) [0]PETSC ERROR: PetscCommGetThreadComm() line 111 in /Users/huizhang/Software/petsc-dev/src/sys/threadcomm/interface/threadcomm.c [0]PETSC ERROR: PetscThreadCommRunKernel0() line 792 in /Users/huizhang/Software/petsc-dev/src/sys/threadcomm/interface/threadcomm.c [0]PETSC ERROR: PetscThreadCommStackDestroy() line 197 in /Users/huizhang/Software/petsc-dev/src/sys/threadcomm/interface/threadcomm.c [0]PETSC ERROR: PetscThreadCommDestroy() line 242 in /Users/huizhang/Software/petsc-dev/src/sys/threadcomm/interface/threadcomm.c [0]PETSC ERROR: Petsc_DelThreadComm() line 71 in /Users/huizhang/Software/petsc-dev/src/sys/threadcomm/interface/dlregisthreadcomm.c [0]PETSC ERROR: PetscFinalize() line 1269 in /Users/huizhang/Software/petsc-dev/src/sys/objects/pinit.c Using more processes gives such error messages for each process (or thread?). I use scalar-tye complex number. I also got the same error messages from my application programs (still with scalar-type complex), in which I do not include petscthreadcomm.h. *** 12,18 **** { PetscErrorCode ierr; PetscScalar dot=0.0,v; ! Vec x,y; PetscInt N =8; PetscScalar one=1.0,two=2.0,alpha=2.0; --- 12,18 ---- { PetscErrorCode ierr; PetscScalar dot=0.0,v; ! Vec x,y,z; PetscInt N =8; PetscScalar one=1.0,two=2.0,alpha=2.0; *** 26,32 **** ierr = VecSetFromOptions(x);CHKERRQ(ierr); ierr = VecSet(x,one);CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"x = %lf\n",PetscRealPart(one));CHKERRQ(ierr); ! ierr = VecCreate(PETSC_COMM_WORLD,&y);CHKERRQ(ierr); ierr = VecSetSizes(y,PETSC_DECIDE,N);CHKERRQ(ierr); ierr = VecSetFromOptions(y);CHKERRQ(ierr); --- 26,32 ---- ierr = VecSetFromOptions(x);CHKERRQ(ierr); ierr = VecSet(x,one);CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"x = %lf\n",PetscRealPart(one));CHKERRQ(ierr); ! ierr = VecDuplicate(x,&z); CHKERRQ(z); ierr = VecCreate(PETSC_COMM_WORLD,&y);CHKERRQ(ierr); ierr = VecSetSizes(y,PETSC_DECIDE,N);CHKERRQ(ierr); ierr = VecSetFromOptions(y);CHKERRQ(ierr); From knepley at gmail.com Sat Feb 16 05:46:17 2013 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 16 Feb 2013 06:46:17 -0500 Subject: [petsc-users] VecDuplicate throwing error messages In-Reply-To: References: Message-ID: On Sat, Feb 16, 2013 at 5:46 AM, Hui Zhang wrote: > I updated petsc-dev just now with the last changeset: 26190:c5c1560f6126 > > I reconfigure and rebuild petsc. I modified > src/sys/threadcomm/examples/tutorials/ex4.c to include a VecDuplicate like > the following ediff file (see behind, ! indicating the modifications). > > Then I got error messages (using mpiexec -np 1) > > [0]PETSC ERROR: PetscCommGetThreadComm() line 111 in > /Users/huizhang/Software/petsc-dev/src/sys/threadcomm/interface/threadcomm.c > [0]PETSC ERROR: PetscThreadCommRunKernel0() line 792 in > /Users/huizhang/Software/petsc-dev/src/sys/threadcomm/interface/threadcomm.c > [0]PETSC ERROR: PetscThreadCommStackDestroy() line 197 in > /Users/huizhang/Software/petsc-dev/src/sys/threadcomm/interface/threadcomm.c > [0]PETSC ERROR: PetscThreadCommDestroy() line 242 in > /Users/huizhang/Software/petsc-dev/src/sys/threadcomm/interface/threadcomm.c > [0]PETSC ERROR: Petsc_DelThreadComm() line 71 in > /Users/huizhang/Software/petsc-dev/src/sys/threadcomm/interface/dlregisthreadcomm.c > [0]PETSC ERROR: PetscFinalize() line 1269 in > /Users/huizhang/Software/petsc-dev/src/sys/objects/pinit.c > You failed to VecDestroy the vector. Matt > Using more processes gives such error messages for each process (or > thread?). I use scalar-tye complex number. > > I also got the same error messages from my application programs (still > with scalar-type complex), in which I do not include petscthreadcomm.h. > > > *** 12,18 **** > { > PetscErrorCode ierr; > PetscScalar dot=0.0,v; > ! Vec x,y; > PetscInt N =8; > PetscScalar one=1.0,two=2.0,alpha=2.0; > > --- 12,18 ---- > { > PetscErrorCode ierr; > PetscScalar dot=0.0,v; > ! Vec x,y,z; > PetscInt N =8; > PetscScalar one=1.0,two=2.0,alpha=2.0; > > *** 26,32 **** > ierr = VecSetFromOptions(x);CHKERRQ(ierr); > ierr = VecSet(x,one);CHKERRQ(ierr); > ierr = PetscPrintf(PETSC_COMM_WORLD,"x = > %lf\n",PetscRealPart(one));CHKERRQ(ierr); > ! > ierr = VecCreate(PETSC_COMM_WORLD,&y);CHKERRQ(ierr); > ierr = VecSetSizes(y,PETSC_DECIDE,N);CHKERRQ(ierr); > ierr = VecSetFromOptions(y);CHKERRQ(ierr); > --- 26,32 ---- > ierr = VecSetFromOptions(x);CHKERRQ(ierr); > ierr = VecSet(x,one);CHKERRQ(ierr); > ierr = PetscPrintf(PETSC_COMM_WORLD,"x = > %lf\n",PetscRealPart(one));CHKERRQ(ierr); > ! ierr = VecDuplicate(x,&z); CHKERRQ(z); > ierr = VecCreate(PETSC_COMM_WORLD,&y);CHKERRQ(ierr); > ierr = VecSetSizes(y,PETSC_DECIDE,N);CHKERRQ(ierr); > ierr = VecSetFromOptions(y);CHKERRQ(ierr); > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike.hui.zhang at hotmail.com Sat Feb 16 05:55:06 2013 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Sat, 16 Feb 2013 12:55:06 +0100 Subject: [petsc-users] VecDuplicate throwing error messages In-Reply-To: References: Message-ID: Yes! Thank you! On Feb 16, 2013, at 12:46 PM, Matthew Knepley wrote: > On Sat, Feb 16, 2013 at 5:46 AM, Hui Zhang wrote: > I updated petsc-dev just now with the last changeset: 26190:c5c1560f6126 > > I reconfigure and rebuild petsc. I modified src/sys/threadcomm/examples/tutorials/ex4.c to include a VecDuplicate like the following ediff file (see behind, ! indicating the modifications). > > Then I got error messages (using mpiexec -np 1) > > [0]PETSC ERROR: PetscCommGetThreadComm() line 111 in /Users/huizhang/Software/petsc-dev/src/sys/threadcomm/interface/threadcomm.c > [0]PETSC ERROR: PetscThreadCommRunKernel0() line 792 in /Users/huizhang/Software/petsc-dev/src/sys/threadcomm/interface/threadcomm.c > [0]PETSC ERROR: PetscThreadCommStackDestroy() line 197 in /Users/huizhang/Software/petsc-dev/src/sys/threadcomm/interface/threadcomm.c > [0]PETSC ERROR: PetscThreadCommDestroy() line 242 in /Users/huizhang/Software/petsc-dev/src/sys/threadcomm/interface/threadcomm.c > [0]PETSC ERROR: Petsc_DelThreadComm() line 71 in /Users/huizhang/Software/petsc-dev/src/sys/threadcomm/interface/dlregisthreadcomm.c > [0]PETSC ERROR: PetscFinalize() line 1269 in /Users/huizhang/Software/petsc-dev/src/sys/objects/pinit.c > > You failed to VecDestroy the vector. > > Matt > > Using more processes gives such error messages for each process (or thread?). I use scalar-tye complex number. > > I also got the same error messages from my application programs (still with scalar-type complex), in which I do not include petscthreadcomm.h. > > > *** 12,18 **** > { > PetscErrorCode ierr; > PetscScalar dot=0.0,v; > ! Vec x,y; > PetscInt N =8; > PetscScalar one=1.0,two=2.0,alpha=2.0; > > --- 12,18 ---- > { > PetscErrorCode ierr; > PetscScalar dot=0.0,v; > ! Vec x,y,z; > PetscInt N =8; > PetscScalar one=1.0,two=2.0,alpha=2.0; > > *** 26,32 **** > ierr = VecSetFromOptions(x);CHKERRQ(ierr); > ierr = VecSet(x,one);CHKERRQ(ierr); > ierr = PetscPrintf(PETSC_COMM_WORLD,"x = %lf\n",PetscRealPart(one));CHKERRQ(ierr); > ! > ierr = VecCreate(PETSC_COMM_WORLD,&y);CHKERRQ(ierr); > ierr = VecSetSizes(y,PETSC_DECIDE,N);CHKERRQ(ierr); > ierr = VecSetFromOptions(y);CHKERRQ(ierr); > --- 26,32 ---- > ierr = VecSetFromOptions(x);CHKERRQ(ierr); > ierr = VecSet(x,one);CHKERRQ(ierr); > ierr = PetscPrintf(PETSC_COMM_WORLD,"x = %lf\n",PetscRealPart(one));CHKERRQ(ierr); > ! ierr = VecDuplicate(x,&z); CHKERRQ(z); > ierr = VecCreate(PETSC_COMM_WORLD,&y);CHKERRQ(ierr); > ierr = VecSetSizes(y,PETSC_DECIDE,N);CHKERRQ(ierr); > ierr = VecSetFromOptions(y);CHKERRQ(ierr); > > > > > > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From mike.hui.zhang at hotmail.com Sat Feb 16 08:11:54 2013 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Sat, 16 Feb 2013 15:11:54 +0100 Subject: [petsc-users] ((PetscObject)M)->comm Message-ID: Why? When I try to use ((PetscObject)M)->comm in my function as follows, PetscErrorCode SetupDirectSolver(KSP *ksp, Mat M) { KSPCreate(((PetscObject)M)->comm,ksp); } I got error: dereferencing pointer to incomplete type From knepley at gmail.com Sat Feb 16 08:25:51 2013 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 16 Feb 2013 09:25:51 -0500 Subject: [petsc-users] ((PetscObject)M)->comm In-Reply-To: References: Message-ID: On Sat, Feb 16, 2013 at 9:11 AM, Hui Zhang wrote: > Why? When I try to use ((PetscObject)M)->comm in my function as follows, > We now have PetscObjectComm((PetscObject) M) which is safer. Matt > PetscErrorCode SetupDirectSolver(KSP *ksp, Mat M) > { > > KSPCreate(((PetscObject)M)->comm,ksp); > > } > > I got > > error: dereferencing pointer to incomplete type > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike.hui.zhang at hotmail.com Sat Feb 16 08:30:42 2013 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Sat, 16 Feb 2013 15:30:42 +0100 Subject: [petsc-users] ((PetscObject)M)->comm In-Reply-To: References: Message-ID: It works! Thank you! On Feb 16, 2013, at 3:25 PM, Matthew Knepley wrote: > PetscObjectComm((PetscObject) M) From mike.hui.zhang at hotmail.com Sat Feb 16 08:40:29 2013 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Sat, 16 Feb 2013 15:40:29 +0100 Subject: [petsc-users] VecView in AO to a binary file? Message-ID: I have an AO that defines a global ordering. I want to write a Vec into a binary file in the global ordering defined by AO. How to do it? Thanks in advance! From jedbrown at mcs.anl.gov Sat Feb 16 08:53:18 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 16 Feb 2013 08:53:18 -0600 Subject: [petsc-users] No Matlab for complex numbers? In-Reply-To: References: Message-ID: MATLAB stores complex numbers in two separate arrays rather than in normal C and Fortran format as one array with struct {double real, imag;}. Thus the interface would need to make a lot of copies and require a significant amount of work to write. We would accept patches if someone wants to implement this in a maintainable way. On Sat, Feb 16, 2013 at 2:42 AM, Hui Zhang wrote: > When I configure --with-matlab=1, I got > > Cannot use Matlab with complex numbers it is not coded for this capability > > Why not supporting this capability? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Feb 16 08:58:35 2013 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 16 Feb 2013 09:58:35 -0500 Subject: [petsc-users] VecView in AO to a binary file? In-Reply-To: References: Message-ID: On Sat, Feb 16, 2013 at 9:40 AM, Hui Zhang wrote: > I have an AO that defines a global ordering. I want to write a Vec into a > binary file in the global ordering defined by AO. How to do it? Thanks in > advance! I think the easiest way is to map the IS rowStart...rowEnd using PetscToApplication, and then create a VecScatter with those two ISes. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From w_ang_temp at 163.com Sat Feb 16 11:36:35 2013 From: w_ang_temp at 163.com (w_ang_temp) Date: Sun, 17 Feb 2013 01:36:35 +0800 (CST) Subject: [petsc-users] Convergence is different with different processors In-Reply-To: References: <155b505e.2849.13cb310c848.Coremail.w_ang_temp@163.com> Message-ID: <541d511a.8a.13ce41434d3.Coremail.w_ang_temp@163.com> I use the two commands in the same project. The first is divergent while the second is convergent. nohup mpiexec -n 4 ./ex4f -ksp_type bcgs -pc_type bjacobi -ksp_rtol 1.0e-5 -ksp_converged_reason >out.txt & nohup mpiexec -n 8 ./ex4f -ksp_type bcgs -pc_type bjacobi -ksp_rtol 1.0e-5 -ksp_converged_reason >out.txt & So what is the reason? Thanks. Jim >> 2013-02-07 13:16:51?"Matthew Knepley" ??? >> On Thu, Feb 7, 2013 at 12:11 AM, w_ang_temp wrote: >> Hello, >> I use the same project, but I find that when different number of processors is choosed, >> the convergence is different. For example, when the processors are 4, it is divergent; when >> the processors are 8, it is convergent. >> So what is the reason? >It is likely that your preconditioner changed. > Matt >> Thanks. >> Jim. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Feb 16 11:57:46 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 16 Feb 2013 11:57:46 -0600 Subject: [petsc-users] Convergence is different with different processors In-Reply-To: <541d511a.8a.13ce41434d3.Coremail.w_ang_temp@163.com> References: <155b505e.2849.13cb310c848.Coremail.w_ang_temp@163.com> <541d511a.8a.13ce41434d3.Coremail.w_ang_temp@163.com> Message-ID: This is completely possible. The convergence of iterative methods is a complicated business. Even something seemingly simple like block Jacobi whose convergence depends not only on the number of blocks, but the "shapes" of the blocks and what part of the matrix is "discarded" in the preconditioner depending on the number and "shape" of the blocks. Decomposing the domain into 4 and 8 pieces results in different "shapes" of the blocks and different parts of the matrix being "discarded" in the preconditioner. Barry On Feb 16, 2013, at 11:36 AM, w_ang_temp wrote: > > > I use the two commands in the same project. The first is divergent while the second is convergent. > nohup mpiexec -n 4 ./ex4f -ksp_type bcgs -pc_type bjacobi -ksp_rtol 1.0e-5 -ksp_converged_reason >out.txt & > nohup mpiexec -n 8 ./ex4f -ksp_type bcgs -pc_type bjacobi -ksp_rtol 1.0e-5 -ksp_converged_reason >out.txt & > So what is the reason? > Thanks. Jim > > > > >> 2013-02-07 13:16:51?"Matthew Knepley" ??? > >> On Thu, Feb 7, 2013 at 12:11 AM, w_ang_temp wrote: > >> Hello, > >> I use the same project, but I find that when different number of processors is choosed, > >> the convergence is different. For example, when the processors are 4, it is divergent; when > >> the processors are 8, it is convergent. > >> So what is the reason? > > >It is likely that your preconditioner changed. > > > Matt > > >> Thanks. > >> Jim. > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > From mike.hui.zhang at hotmail.com Sat Feb 16 12:02:15 2013 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Sat, 16 Feb 2013 19:02:15 +0100 Subject: [petsc-users] Convergence is different with different processors In-Reply-To: <541d511a.8a.13ce41434d3.Coremail.w_ang_temp@163.com> References: <155b505e.2849.13cb310c848.Coremail.w_ang_temp@163.com> <541d511a.8a.13ce41434d3.Coremail.w_ang_temp@163.com> Message-ID: From my understanding of the bjacobi, it is a block Jacobi method with each block corresponding to a processor. What I can not understand is why you get convergence with eight blocks but divergence with four (fewer) blocks. Which directory is the program ex4f from? On Feb 16, 2013, at 6:36 PM, w_ang_temp wrote: > > > I use the two commands in the same project. The first is divergent while the second is convergent. > nohup mpiexec -n 4 ./ex4f -ksp_type bcgs -pc_type bjacobi -ksp_rtol 1.0e-5 -ksp_converged_reason >out.txt & > nohup mpiexec -n 8 ./ex4f -ksp_type bcgs -pc_type bjacobi -ksp_rtol 1.0e-5 -ksp_converged_reason >out.txt & > So what is the reason? > Thanks. Jim > > > > >> 2013-02-07 13:16:51?"Matthew Knepley" ??? > >> On Thu, Feb 7, 2013 at 12:11 AM, w_ang_temp wrote: > >> Hello, > >> I use the same project, but I find that when different number of processors is choosed, > >> the convergence is different. For example, when the processors are 4, it is divergent; when > >> the processors are 8, it is convergent. > >> So what is the reason? > > >It is likely that your preconditioner changed. > > > Matt > > >> Thanks. > >> Jim. > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > From w_ang_temp at 163.com Sat Feb 16 12:19:38 2013 From: w_ang_temp at 163.com (w_ang_temp) Date: Sun, 17 Feb 2013 02:19:38 +0800 (CST) Subject: [petsc-users] Convergence is different with different processors In-Reply-To: References: <155b505e.2849.13cb310c848.Coremail.w_ang_temp@163.com> <541d511a.8a.13ce41434d3.Coremail.w_ang_temp@163.com> Message-ID: <6f33ea4f.14a.13ce43ba062.Coremail.w_ang_temp@163.com> Please do not focus any attention on ex4f. I just rename my project name to ex4f in order to use the exsiting makefile because I do not know how use the makefile. At 2013-02-17 02:02:15,"Hui Zhang" wrote: >From my understanding of the bjacobi, it is a block Jacobi method with each block corresponding to a processor. >What I can not understand is why you get convergence with eight blocks but divergence with four (fewer) blocks. >Which directory is the program ex4f from? > > >On Feb 16, 2013, at 6:36 PM, w_ang_temp wrote: > >> >> >> I use the two commands in the same project. The first is divergent while the second is convergent. >> nohup mpiexec -n 4 ./ex4f -ksp_type bcgs -pc_type bjacobi -ksp_rtol 1.0e-5 -ksp_converged_reason >out.txt & >> nohup mpiexec -n 8 ./ex4f -ksp_type bcgs -pc_type bjacobi -ksp_rtol 1.0e-5 -ksp_converged_reason >out.txt & >> So what is the reason? >> Thanks. Jim >> >> >> >> >> 2013-02-07 13:16:51?"Matthew Knepley" ??? >> >> On Thu, Feb 7, 2013 at 12:11 AM, w_ang_temp wrote: >> >> Hello, >> >> I use the same project, but I find that when different number of processors is choosed, >> >> the convergence is different. For example, when the processors are 4, it is divergent; when >> >> the processors are 8, it is convergent. >> >> So what is the reason? >> >> >It is likely that your preconditioner changed. >> >> > Matt >> >> >> Thanks. >> >> Jim. >> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From w_ang_temp at 163.com Sat Feb 16 12:21:29 2013 From: w_ang_temp at 163.com (w_ang_temp) Date: Sun, 17 Feb 2013 02:21:29 +0800 (CST) Subject: [petsc-users] Convergence is different with different processors In-Reply-To: References: <155b505e.2849.13cb310c848.Coremail.w_ang_temp@163.com> <541d511a.8a.13ce41434d3.Coremail.w_ang_temp@163.com> Message-ID: <5c3759fc.155.13ce43d5297.Coremail.w_ang_temp@163.com> Hello, Barry Is it true that BJacobi preconditioner is only one type of ASM preconditioner(overlap=0), it is only one part of ASM? I find that sometimes the two have the same results while sometimes they are not. Is the reason that PETSc can internally get the better parameter when using ASM, and sometimes it use overlap=0 then ASM and BJacobi have the same results. If it is true, ASM is always better than BJacobi? Thanks. Jim At 2013-02-17 01:57:46,"Barry Smith" wrote: > > This is completely possible. The convergence of iterative methods is a complicated business. Even something seemingly simple like block Jacobi whose conver> gence depends not only on the number of blocks, but the "shapes" of the blocks and what part of the matrix is "discarded" in the preconditioner depending on the > number and "shape" of the blocks. Decomposing the domain into 4 and 8 pieces results in different "shapes" of the blocks and different parts of the matrix being "> discarded" in the preconditioner. > > Barry > >On Feb 16, 2013, at 11:36 AM, w_ang_temp wrote: > >> >> >> I use the two commands in the same project. The first is divergent while the second is convergent. >> nohup mpiexec -n 4 ./ex4f -ksp_type bcgs -pc_type bjacobi -ksp_rtol 1.0e-5 -ksp_converged_reason >out.txt & >> nohup mpiexec -n 8 ./ex4f -ksp_type bcgs -pc_type bjacobi -ksp_rtol 1.0e-5 -ksp_converged_reason >out.txt & >> So what is the reason? >> Thanks. Jim >> >> >> >> >> 2013-02-07 13:16:51?"Matthew Knepley" ??? >> >> On Thu, Feb 7, 2013 at 12:11 AM, w_ang_temp wrote: >> >> Hello, >> >> I use the same project, but I find that when different number of processors is choosed, >> >> the convergence is different. For example, when the processors are 4, it is divergent; when >> >> the processors are 8, it is convergent. >> >> So what is the reason? >> >> >It is likely that your preconditioner changed. >> >> > Matt >> >> >> Thanks. >> >> Jim. >> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Feb 16 12:26:53 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 16 Feb 2013 12:26:53 -0600 Subject: [petsc-users] Convergence is different with different processors In-Reply-To: <5c3759fc.155.13ce43d5297.Coremail.w_ang_temp@163.com> References: <155b505e.2849.13cb310c848.Coremail.w_ang_temp@163.com> <541d511a.8a.13ce41434d3.Coremail.w_ang_temp@163.com> <5c3759fc.155.13ce43d5297.Coremail.w_ang_temp@163.com> Message-ID: On Sat, Feb 16, 2013 at 12:21 PM, w_ang_temp wrote: > Is it true that BJacobi preconditioner is only one type of ASM > preconditioner(overlap=0), it is only one part of ASM? > I find that sometimes the two have the same results while sometimes > they are not. Is the reason that PETSc can > internally get the better parameter when using ASM, and sometimes it use > overlap=0 then ASM and BJacobi have the same > results. > If it is true, ASM is always better than BJacobi? > BJacobi is slightly more efficient to implement, but -pc_type asm -pc_asm_overlap 0 is block Jacobi. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Feb 16 16:50:21 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 16 Feb 2013 16:50:21 -0600 Subject: [petsc-users] Convergence is different with different processors In-Reply-To: <5c3759fc.155.13ce43d5297.Coremail.w_ang_temp@163.com> References: <155b505e.2849.13cb310c848.Coremail.w_ang_temp@163.com> <541d511a.8a.13ce41434d3.Coremail.w_ang_temp@163.com> <5c3759fc.155.13ce43d5297.Coremail.w_ang_temp@163.com> Message-ID: On Feb 16, 2013, at 12:21 PM, w_ang_temp wrote: > > > Hello, Barry > > Is it true that BJacobi preconditioner is only one type of ASM preconditioner(overlap=0), it is only one part of ASM? > I find that sometimes the two have the same results while sometimes they are not. Is the reason that PETSc can > internally get the better parameter when using ASM, and sometimes it use overlap=0 then ASM and BJacobi have the same > results. > If it is true, ASM is always better than BJacobi? By default PETSc's ASM uses an overlap of 1, if you set the overlap to 0 then it is the same algorithm as block Jacobi. Generally ASM with overlap > 0 converges better than block Jacobi but it might not on rare occasions. ASM with overlap > 1 may take more or less time than block Jacobi since it requires more time to set up and more time per iteration. So what is best in terms of time depends on the relative expense of the two methods and how much ASM improves the convergence over block Jacobi (if it does). There are entire books written on this subject and they barely scratch the surface in being able to predict for a given problem what approach is best. Barry > Thanks. Jim > > > > > At 2013-02-17 01:57:46,"Barry Smith" wrote: > > > > This is completely possible. The convergence of iterative methods is a complicated business. Even something seemingly simple like block Jacobi whose conver> gence depends not only on the number of blocks, but the "shapes" of the blocks and what part of the matrix is "discarded" in the preconditioner depending on the > number and "shape" of the blocks. Decomposing the domain into 4 and 8 pieces results in different "shapes" of the blocks and different parts of the matrix being "> discarded" in the preconditioner. > > > > Barry > > > >On Feb 16, 2013, at 11:36 AM, w_ang_temp wrote: > > > >> > >> > >> I use the two commands in the same project. The first is divergent while the second is convergent. > >> nohup mpiexec -n 4 ./ex4f -ksp_type bcgs -pc_type bjacobi -ksp_rtol 1.0e-5 -ksp_converged_reason >out.txt & > >> nohup mpiexec -n 8 ./ex4f -ksp_type bcgs -pc_type bjacobi -ksp_rtol 1.0e-5 -ksp_converged_reason >out.txt & > >> So what is the reason? > >> Thanks. Jim > >> > >> > >> > >> >> 2013-02-07 13:16:51?"Matthew Knepley" ??? > >> >> On Thu, Feb 7, 2013 at 12:11 AM, w_ang_temp wrote: > >> >> Hello, > >> >> I use the same project, but I find that when different number of processors is choosed, > >> >> the convergence is different. For example, when the processors are 4, it is divergent; when > >> >> the processors are 8, it is convergent. > >> >> So what is the reason? > >> > >> >It is likely that your preconditioner changed. > >> > >> > Matt > >> > >> >> Thanks. > >> >> Jim. > >> > >> > >> > >> > >> > >> -- > >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > >> -- Norbert Wiener > >> > >> > > > > > From mike.hui.zhang at hotmail.com Sun Feb 17 06:56:24 2013 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Sun, 17 Feb 2013 13:56:24 +0100 Subject: [petsc-users] MatGetLocalSubMatrix Message-ID: I want to use MatGetLocalSubMatrix for assembly purpose. PetscErrorCode MatGetLocalSubMatrix(Mat mat,IS isrow,IS iscol,Mat *submat) I want to get a submat shared by some (not all) of the processes that share the mat. Some questions. 1. Shall I create isrow and iscol shared by the processes that will share the submat? 2. The manual says the submat supports MatSetValuesLocal. Is the LocalToGlobalMapping already defined for submat, or I need to define it by myself? If it is already defined, is it derived from the ltog of the mat such that for this processor the local numbering of the submat are in the same order as the local numbering for the mat? From jedbrown at mcs.anl.gov Sun Feb 17 09:45:40 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 17 Feb 2013 09:45:40 -0600 Subject: [petsc-users] MatGetLocalSubMatrix In-Reply-To: References: Message-ID: Look at src/snes/examples/tutorials/ex28.c for an example usage, or src/mat/examples/tests/ex159.c for a more advanced/manual test that demonstrates recursive nesting. On Sun, Feb 17, 2013 at 6:56 AM, Hui Zhang wrote: > I want to use MatGetLocalSubMatrix for assembly purpose. > > PetscErrorCode MatGetLocalSubMatrix(Mat mat,IS isrow,IS iscol,Mat *submat) > > I want to get a submat shared by some (not all) of the processes that > share the mat. > > Some questions. > > 1. Shall I create isrow and iscol shared by the processes that will share > the submat? > The submat is not guaranteed to be collective (this depends on the matrix format), but it does always support MatSetValuesLocal(). The isrow and iscol are _local_ so they should use PETSC_COMM_SELF. > 2. The manual says the submat supports MatSetValuesLocal. Is the > LocalToGlobalMapping already defined for submat, or I need to define it by > myself? > It is induced by the LocalToGlobalMapping of the coupled (parent) matrix and the selected local indices. > > If it is already defined, is it derived from the ltog of the mat such > that for this processor the local numbering of the submat are in the same > order > > as the local numbering for the mat? > The local numbering is with respect to the submat. (This is necessary, otherwise there would be no way to apply this recursively and you would have to keep track of the index mapping. Keeping the submat indexing as private as possible was a key motivation for this interface.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike.hui.zhang at hotmail.com Sun Feb 17 10:15:21 2013 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Sun, 17 Feb 2013 17:15:21 +0100 Subject: [petsc-users] MatGetLocalSubMatrix In-Reply-To: References: Message-ID: thanks a lot! I seems to understand much better now! On Feb 17, 2013, at 4:45 PM, Jed Brown wrote: > Look at src/snes/examples/tutorials/ex28.c for an example usage, or src/mat/examples/tests/ex159.c for a more advanced/manual test that demonstrates recursive nesting. > > > On Sun, Feb 17, 2013 at 6:56 AM, Hui Zhang wrote: > I want to use MatGetLocalSubMatrix for assembly purpose. > > PetscErrorCode MatGetLocalSubMatrix(Mat mat,IS isrow,IS iscol,Mat *submat) > > I want to get a submat shared by some (not all) of the processes that share the mat. > > Some questions. > > 1. Shall I create isrow and iscol shared by the processes that will share the submat? > > The submat is not guaranteed to be collective (this depends on the matrix format), but it does always support MatSetValuesLocal(). The isrow and iscol are _local_ so they should use PETSC_COMM_SELF. > > 2. The manual says the submat supports MatSetValuesLocal. Is the LocalToGlobalMapping already defined for submat, or I need to define it by myself? > > It is induced by the LocalToGlobalMapping of the coupled (parent) matrix and the selected local indices. > > > If it is already defined, is it derived from the ltog of the mat such that for this processor the local numbering of the submat are in the same order > > as the local numbering for the mat? > > The local numbering is with respect to the submat. (This is necessary, otherwise there would be no way to apply this recursively and you would have to keep track of the index mapping. Keeping the submat indexing as private as possible was a key motivation for this interface.) From subramanya.g at gmail.com Mon Feb 18 16:55:55 2013 From: subramanya.g at gmail.com (Subramanya G) Date: Mon, 18 Feb 2013 17:55:55 -0500 Subject: [petsc-users] Compiling petsc-3.3 p6 on mountain lion Message-ID: HI, I am unable to do ./configure successfully on os x mountain lion. I get a a cannot run executables error. This is weird as I have petsc-3.3p5 successfully installed on the same computer. Has anybody faced similar problems? Does anybody have any work-arounds? Thanks, Subramanya G Sadasiva, Graduate Research Assistant, Hierarchical Design and Characterization Laboratory, School of Mechanical Engineering, Purdue University. "The art of structure is where to put the holes" Robert Le Ricolais, 1894-1977 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Feb 18 17:01:53 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 18 Feb 2013 17:01:53 -0600 Subject: [petsc-users] Compiling petsc-3.3 p6 on mountain lion In-Reply-To: References: Message-ID: Send configure.log to petsc-maint at mcs.anl.gov On Feb 18, 2013, at 4:55 PM, Subramanya G wrote: > HI, > I am unable to do ./configure successfully on os x mountain lion. I get a a cannot run executables error. This is weird as I have petsc-3.3p5 successfully installed on the same computer. Has anybody faced similar problems? Does anybody have any work-arounds? > Thanks, > > > Subramanya G Sadasiva, > > Graduate Research Assistant, > Hierarchical Design and Characterization Laboratory, > School of Mechanical Engineering, > Purdue University. > > "The art of structure is where to put the holes" > Robert Le Ricolais, 1894-1977 From m.guterres at gmail.com Tue Feb 19 10:36:35 2013 From: m.guterres at gmail.com (Marcelo Xavier Guterres) Date: Tue, 19 Feb 2013 13:36:35 -0300 Subject: [petsc-users] "error of the solution" of GMRES Message-ID: Esteemed colleagues, My problem is: Div ( Grad ( phi ) ) = 0 mesh: 0 < x < 1; 0 < y < 1; with boundary conditions: phi ( x , 0 ) = x; phi ( 0 ,y ) = x; phi ( x , 1 ) = x; phi ( 1 ,y ) = x; with serial processing. solved with ( KSPCG x PCJACOBI ) and (KSPGMRS x PCJACOBI). I have a question, why the "*norm error** **of the solution*" of GMRES is high, when the mesh is large. It is a problem in methods or truncation error? the results were: *m* *n* *m x n* *error ( KSPCG x PCJACOBI )* *error ( KSPGMRS x PCJACOBI ) * 3 3 9 1,92E-16 4,42E-16 4 4 16 2,08E-16 6,46E-16 5 5 25 4,41E-16 9,63E-16 6 6 36 8,77E-16 8,26E-16 7 7 49 2,37E-06 2,52E-06 8 8 64 1,17E-05 1,33E-05 9 9 81 9,32E-06 1,26E-05 10 10 100 7,33E-06 9,93E-06 20 20 400 4,22E-05 3,16E-04 30 30 900 1,06E-04 2,37E-02 40 40 1600 2,14E-04 3,77E-02 50 50 2500 3,37E-04 9,36E-03 100 100 10000 1,27E-03 6,53E-02 200 200 40000 4,32E-03 3,67E-01 300 300 90000 9,53E-02 1,02E+00 400 400 160000 1,68E-02 2,10E+00 500 500 250000 2,61E-02 3,66E+00 how can I improve the performance of GMRES? a good week for all. -- Ph.d student Marcelo Xavier Guterres Rio de Janeiro , Brazil. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Tue Feb 19 10:57:10 2013 From: mark.adams at columbia.edu (Mark F. Adams) Date: Tue, 19 Feb 2013 11:57:10 -0500 Subject: [petsc-users] "error of the solution" of GMRES In-Reply-To: References: Message-ID: <8BAD7F6D-CF57-4E60-A679-B7C02565593D@columbia.edu> CG is a better method than GMRES for symmetric positive definite problems. You want to use a better preconditioner. Try '-pc_type gamg -pc_gamg_agg_nsmooths 1' On Feb 19, 2013, at 11:36 AM, Marcelo Xavier Guterres wrote: > Esteemed colleagues, > > > > My problem is: > > Div ( Grad ( phi ) ) = 0 > > mesh: > > 0 < x < 1; > 0 < y < 1; > > with boundary conditions: > > phi ( x , 0 ) = x; > phi ( 0 ,y ) = x; > phi ( x , 1 ) = x; > phi ( 1 ,y ) = x; > > with serial processing. > > > solved with ( KSPCG x PCJACOBI ) and (KSPGMRS x PCJACOBI). I have a question, why the "norm error of the solution" of GMRES is high, when the mesh is large. It is a problem in methods or truncation error? > > > the results were: > > > m n m x n error ( KSPCG x PCJACOBI ) error ( KSPGMRS x PCJACOBI ) > 3 3 9 1,92E?16 4,42E?16 > > 4 4 16 2,08E?16 6,46E?16 > > 5 5 25 4,41E?16 9,63E?16 > > 6 6 36 8,77E?16 8,26E?16 > > 7 7 49 2,37E?06 2,52E?06 > > 8 8 64 1,17E?05 1,33E?05 > > 9 9 81 9,32E?06 1,26E?05 > > 10 10 100 7,33E?06 9,93E?06 > > 20 20 400 4,22E?05 3,16E?04 > > 30 30 900 1,06E?04 2,37E?02 > > 40 40 1600 2,14E?04 3,77E?02 > > 50 50 2500 3,37E?04 9,36E?03 > > 100 100 10000 1,27E?03 6,53E?02 > > 200 200 40000 4,32E?03 3,67E?01 > > 300 300 90000 9,53E?02 1,02E+00 > > 400 400 160000 1,68E?02 2,10E+00 > > 500 500 250000 2,61E?02 3,66E+00 > > > how can I improve the performance of GMRES? > > > a good week for all. > > > -- > Ph.d student Marcelo Xavier Guterres > Rio de Janeiro , Brazil. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhalen at gmail.com Tue Feb 19 11:05:04 2013 From: gokhalen at gmail.com (Nachiket Gokhale) Date: Tue, 19 Feb 2013 12:05:04 -0500 Subject: [petsc-users] Compiling petsc-3.3 p6 on mountain lion (Subramanya G) Message-ID: A few months ago I got this error with petsc-dev "************************************************ UNABLE to EXECUTE BINARIES for ./configure ------------------------------------------------------------------------------- Cannot run executables created with C. If this machine uses a batch system to submit jobs you will need to configure using ./configure with the additional option --with-batch. Otherwise there is problem with the compilers. Can you compile and run code with your C/C++ (and maybe Fortran) compilers?" If this is the error you got, then, to make along story short, this went way when I removed the symbol "+" from my PETSC_ARCH. -Nachiket From mark.adams at columbia.edu Tue Feb 19 12:00:01 2013 From: mark.adams at columbia.edu (Mark F. Adams) Date: Tue, 19 Feb 2013 13:00:01 -0500 Subject: [petsc-users] Question PETSc In-Reply-To: References: Message-ID: <8A3942C3-A0E1-4446-9A0F-0FEAA9622981@columbia.edu> On Feb 19, 2013, at 12:53 PM, Marcelo Xavier Guterres wrote: > Esteemed colleague, > > thanks for the previous answer. But still can not understand one aspect. > > > The GMRES behaves well up to a point, but then the quality of the final solution is bad, compared to analytical solution. > > > It is a problem in the method or truncation error? No, it is an approximate solver and it gets worse as the problem size increases. CG gets worse also. CG is just better for this problem. > > > m n m x n error ( KSPCG x PCJACOBI ) error ( KSPGMRS x PCJACOBI ) > 3 3 9 1,92E?16 4,42E?16 > > 4 4 16 2,08E?16 6,46E?16 > > 5 5 25 4,41E?16 9,63E?16 > > 6 6 36 8,77E?16 8,26E?16 > > 7 7 49 2,37E?06 2,52E?06 > > 8 8 64 1,17E?05 1,33E?05 > > 9 9 81 9,32E?06 1,26E?05 > > 10 10 100 7,33E?06 9,93E?06 > > 20 20 400 4,22E?05 3,16E?04 > > 30 30 900 1,06E?04 2,37E?02 > > 40 40 1600 2,14E?04 3,77E?02 > > 50 50 2500 3,37E?04 9,36E?03 > > 100 100 10000 1,27E?03 6,53E?02 > > 200 200 40000 4,32E?03 3,67E?01 > > 300 300 90000 9,53E?02 1,02E+00 > > 400 400 160000 1,68E?02 2,10E+00 > > 500 500 250000 2,61E?02 3,66E+00 > > > > > -- > Sauda??es Pampeanas > -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- > > Phd. student Marcelo -------------- next part -------------- An HTML attachment was scrubbed... URL: From irving at naml.us Tue Feb 19 17:39:58 2013 From: irving at naml.us (Geoffrey Irving) Date: Tue, 19 Feb 2013 15:39:58 -0800 Subject: [petsc-users] checking whether a ksp type wants definite matrices Message-ID: Currently I have a command line option that controls whether my force classes perform definiteness projection, which is necessary for CG and unnecessary for GMRES, etc. It would be nice to automatically turn on definiteness if I notice that the petsc ksp type is cg. Is CG the only KSP that requires definiteness? If not, is there a standard way of checking whether KSP object or type supports indefinite matrices? Thanks, Geoffrey From jedbrown at mcs.anl.gov Tue Feb 19 18:04:21 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 19 Feb 2013 18:04:21 -0600 Subject: [petsc-users] checking whether a ksp type wants definite matrices In-Reply-To: References: Message-ID: In addition to the CG family, MINRES requires an SPD preconditioner. There is not currently a way to query this, but it would also be helpful documentation for users. Right now, it's (sometimes) scattered around man pages. On Tue, Feb 19, 2013 at 5:39 PM, Geoffrey Irving wrote: > Currently I have a command line option that controls whether my force > classes perform definiteness projection, which is necessary for CG and > unnecessary for GMRES, etc. It would be nice to automatically turn on > definiteness if I notice that the petsc ksp type is cg. > > Is CG the only KSP that requires definiteness? If not, is there a > standard way of checking whether KSP object or type supports > indefinite matrices? > > Thanks, > Geoffrey > -------------- next part -------------- An HTML attachment was scrubbed... URL: From miyan at us.ibm.com Wed Feb 20 08:11:08 2013 From: miyan at us.ibm.com (Mi Yan) Date: Wed, 20 Feb 2013 09:11:08 -0500 Subject: [petsc-users] pass PETSC_NULL_INTEGER to dynamic array of Vec in Frotran90 Message-ID: xlf9014.1 reported error for the below FORTRAN 90 code: ****** MatNullSpace nullsp PetscErrorCode :: ierr call MatNullSpaceCreate( PETSC_COMM_WORLD, PETSC_TRUE, 0, PETSC_NULL_INTEGER, nullsp, ierr ); **** The error message is: Actual argument attributes do not match those specified by an accessible explicit interface. The error implies the real argument the attribute of "PETSC_NULL_INTEGER" does not match with those specified by the interface. The interface of MatNullSpaceCreate is declared in finclude/ftn-auto/petscmat.h90 , whose 4th dummy argument is declared as Vec vecs (*) In my code, PETSC_NULL_INTEGER is passed to an array of Vec. Is there anything wrong here? thanks, Mi -------------- next part -------------- An HTML attachment was scrubbed... URL: From Shuangshuang.Jin at pnnl.gov Wed Feb 20 12:19:31 2013 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Wed, 20 Feb 2013 10:19:31 -0800 Subject: [petsc-users] PETSc DAE solver Message-ID: <6778DE83AB681D49BFC2CD850610FEB1018FC699EAB7@EMAIL04.pnl.gov> Hi, I'm trying to use PETSc to solve a set of first-order differential and algebraic equations such as: x' = f(x,y) 0 = g(x,y) The resources I can find online so far are 1. PETSc Manual: Chapter 6 TS: Scalable ODE and DAE Solvers 2. petsc-3.3-p6/src/ts/examples/tutorials which still seems vague to me at this moment. Can anyone tell me if there's any more specific references I can refer to as a starting point to tackle this problem? Thanks, Shuangshuang -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhyshr at mcs.anl.gov Wed Feb 20 12:34:26 2013 From: abhyshr at mcs.anl.gov (Shri) Date: Wed, 20 Feb 2013 12:34:26 -0600 Subject: [petsc-users] PETSc DAE solver In-Reply-To: <6778DE83AB681D49BFC2CD850610FEB1018FC699EAB7@EMAIL04.pnl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FC699EAB7@EMAIL04.pnl.gov> Message-ID: <8F12B6A5-1944-47F7-80D0-F51454C755F2@mcs.anl.gov> Jin, If you are working on integrating power grid DAE equations then you should use petsc-dev instead as we've added an example for it. petsc-dev/src/ts/examples/tutorials/power_grid/stabiltiy_9bus/ex9bus.c. There are other examples in the power_grid directory. Shri On Feb 20, 2013, at 12:19 PM, Jin, Shuangshuang wrote: > Hi, I?m trying to use PETSc to solve a set of first-order differential and algebraic equations such as: > x' = f(x,y) > 0 = g(x,y) > > The resources I can find online so far are > PETSc Manual: Chapter 6 TS: Scalable ODE and DAE Solvers > petsc-3.3-p6/src/ts/examples/tutorials > which still seems vague to me at this moment. > > Can anyone tell me if there?s any more specific references I can refer to as a starting point to tackle this problem? > > Thanks, > Shuangshuang > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Shuangshuang.Jin at pnnl.gov Wed Feb 20 12:39:02 2013 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Wed, 20 Feb 2013 10:39:02 -0800 Subject: [petsc-users] PETSc DAE solver In-Reply-To: <8F12B6A5-1944-47F7-80D0-F51454C755F2@mcs.anl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FC699EAB7@EMAIL04.pnl.gov> <8F12B6A5-1944-47F7-80D0-F51454C755F2@mcs.anl.gov> Message-ID: <6778DE83AB681D49BFC2CD850610FEB1018FC699EAC7@EMAIL04.pnl.gov> Shri, thank you! That's very helpful information. I'm going to obtain the development version of PETSc now. Thanks, Shuangshuang From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Shri Sent: Wednesday, February 20, 2013 10:34 AM To: PETSc users list Subject: Re: [petsc-users] PETSc DAE solver Jin, If you are working on integrating power grid DAE equations then you should use petsc-dev instead as we've added an example for it. petsc-dev/src/ts/examples/tutorials/power_grid/stabiltiy_9bus/ex9bus.c. There are other examples in the power_grid directory. Shri On Feb 20, 2013, at 12:19 PM, Jin, Shuangshuang wrote: Hi, I'm trying to use PETSc to solve a set of first-order differential and algebraic equations such as: x' = f(x,y) 0 = g(x,y) The resources I can find online so far are 1. PETSc Manual: Chapter 6 TS: Scalable ODE and DAE Solvers 2. petsc-3.3-p6/src/ts/examples/tutorials which still seems vague to me at this moment. Can anyone tell me if there's any more specific references I can refer to as a starting point to tackle this problem? Thanks, Shuangshuang -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhyshr at mcs.anl.gov Wed Feb 20 12:48:56 2013 From: abhyshr at mcs.anl.gov (Shri) Date: Wed, 20 Feb 2013 12:48:56 -0600 Subject: [petsc-users] PETSc DAE solver In-Reply-To: <6778DE83AB681D49BFC2CD850610FEB1018FC699EAC7@EMAIL04.pnl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FC699EAB7@EMAIL04.pnl.gov> <8F12B6A5-1944-47F7-80D0-F51454C755F2@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FC699EAC7@EMAIL04.pnl.gov> Message-ID: <03A84ACD-D18D-4EC9-B868-72E2AE208A6C@mcs.anl.gov> http://www.mcs.anl.gov/petsc/developers/index.html On Feb 20, 2013, at 12:39 PM, Jin, Shuangshuang wrote: > Shri, thank you! That?s very helpful information. I?m going to obtain the development version of PETSc now. > > Thanks, > Shuangshuang > > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Shri > Sent: Wednesday, February 20, 2013 10:34 AM > To: PETSc users list > Subject: Re: [petsc-users] PETSc DAE solver > > Jin, > If you are working on integrating power grid DAE equations then you should use petsc-dev instead as we've added an example for it. petsc-dev/src/ts/examples/tutorials/power_grid/stabiltiy_9bus/ex9bus.c. There are other examples in the power_grid directory. > > Shri > > On Feb 20, 2013, at 12:19 PM, Jin, Shuangshuang wrote: > > > Hi, I?m trying to use PETSc to solve a set of first-order differential and algebraic equations such as: > x' = f(x,y) > 0 = g(x,y) > > The resources I can find online so far are > PETSc Manual: Chapter 6 TS: Scalable ODE and DAE Solvers > petsc-3.3-p6/src/ts/examples/tutorials > which still seems vague to me at this moment. > > Can anyone tell me if there?s any more specific references I can refer to as a starting point to tackle this problem? > > Thanks, > Shuangshuang > -------------- next part -------------- An HTML attachment was scrubbed... URL: From slivkaje at gmail.com Wed Feb 20 13:06:41 2013 From: slivkaje at gmail.com (Jelena Slivka) Date: Wed, 20 Feb 2013 14:06:41 -0500 Subject: [petsc-users] fminunc in PETSc Message-ID: Hello, I am trying to solve an unconstrained convex optimization problem and I am looking for a function similar to Matlab's fminunc to which I could provide a function that calculates the derivative of my objective. Is there a similar function in PETSc package? Grateful in advance, Jelena Slivka -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Feb 20 13:32:43 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 20 Feb 2013 13:32:43 -0600 Subject: [petsc-users] fminunc in PETSc In-Reply-To: References: Message-ID: <7BCE14D1-AA29-428A-A5CD-1E7F3EE3286A@mcs.anl.gov> http://www.mcs.anl.gov/research/projects/tao/ On Feb 20, 2013, at 1:06 PM, Jelena Slivka wrote: > Hello, > I am trying to solve an unconstrained convex optimization problem and I am looking for a function similar to Matlab's fminunc to which I could provide a function that calculates the derivative of my objective. Is there a similar function in PETSc package? > Grateful in advance, > Jelena Slivka From Shuangshuang.Jin at pnnl.gov Wed Feb 20 14:08:07 2013 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Wed, 20 Feb 2013 12:08:07 -0800 Subject: [petsc-users] PETSc DAE solver In-Reply-To: <03A84ACD-D18D-4EC9-B868-72E2AE208A6C@mcs.anl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FC699EAB7@EMAIL04.pnl.gov> <8F12B6A5-1944-47F7-80D0-F51454C755F2@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FC699EAC7@EMAIL04.pnl.gov> <03A84ACD-D18D-4EC9-B868-72E2AE208A6C@mcs.anl.gov> Message-ID: <6778DE83AB681D49BFC2CD850610FEB1018FC699EB2F@EMAIL04.pnl.gov> Hi, Shri, I was able to run the ex9bus.c code successfully with 'mpirun -n 1 ex9bus', but failed with 'mpirun -n 2 ex9bus' with the following message: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: No support for this operation for this object type! [0]PETSC ERROR: Only for sequential runs! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: fab2442df69c95ef24bf6631a38c7dc028d7ee34 HG Date: Tue Feb 19 16:16:15 2013 -0600 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ex9bus on a arch-linux2-c-debug named olympus.local by d3m956 Wed Feb 20 11:36:02 2013 [0]PETSC ERROR: Libraries linked from /pic/projects/ds/petsc-dev/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Wed Feb 20 11:01:15 2013 [0]PETSC ERROR: Configure options [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: main() line 807 in src/ts/examples/tutorials/power_grid/stability_9busex9bus.c Does that mean the parallelized version of DAE solver in PETSc is not available yet? Or I missed some PETSc options to enable the function? Another question is I also found two other examples in previous installed non-dev PETSc version: petsc-3.3-p6/src/ts/examples/tutorials/ex8.c and ex19.c. I felt that they are somewhat similar to my problem. Do you think those two are useful examples to me as well? However, it seems like they are uniprocessor examples only too. Please correct me if I understand it wrong. Thanks, Shuangshuang From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Shri Sent: Wednesday, February 20, 2013 10:49 AM To: PETSc users list Subject: Re: [petsc-users] PETSc DAE solver http://www.mcsanl.gov/petsc/developers/index.html On Feb 20, 2013, at 12:39 PM, Jin, Shuangshuang wrote: Shri, thank you! That's very helpful information. I'm going to obtain the development version of PETSc now. Thanks, Shuangshuang From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Shri Sent: Wednesday, February 20, 2013 10:34 AM To: PETSc users list Subject: Re: [petsc-users] PETSc DAE solver Jin, If you are working on integrating power grid DAE equations then you should use petsc-dev instead as we've added an example for it. petsc-dev/src/ts/examples/tutorials/power_grid/stabiltiy_9bus/ex9bus.c. There are other examples in the power_grid directory. Shri On Feb 20, 2013, at 12:19 PM, Jin, Shuangshuang wrote: Hi, I'm trying to use PETSc to solve a set of first-order differential and algebraic equations such as: x' = f(x,y) 0 = g(x,y) The resources I can find online so far are 1. PETSc Manual: Chapter 6 TS: Scalable ODE and DAE Solvers 2. petsc-3.3-p6/src/ts/examples/tutorials which still seems vague to me at this moment. Can anyone tell me if there's any more specific references I can refer to as a starting point to tackle this problem? Thanks, Shuangshuang -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Feb 20 14:22:14 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 20 Feb 2013 14:22:14 -0600 Subject: [petsc-users] PETSc DAE solver In-Reply-To: <6778DE83AB681D49BFC2CD850610FEB1018FC699EB2F@EMAIL04.pnl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FC699EAB7@EMAIL04.pnl.gov> <8F12B6A5-1944-47F7-80D0-F51454C755F2@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FC699EAC7@EMAIL04.pnl.gov> <03A84ACD-D18D-4EC9-B868-72E2AE208A6C@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FC699EB2F@EMAIL04.pnl.gov> Message-ID: <77E6C1BA-B0DA-4620-A7A3-5A8E824A216E@mcs.anl.gov> On Feb 20, 2013, at 2:08 PM, "Jin, Shuangshuang" wrote: > Hi, Shri, I was able to run the ex9bus.c code successfully with ?mpirun ?n 1 ex9bus?, but failed with ?mpirun ?n 2 ex9bus? with the following message: This particular example is not set up to run in parallel. The PETSc solvers are all parallel but to use the ODE integrators in parallel one must provide the function evaluation (and Jacobian optionally) also in parallel. For this example we did not provide the parallel function evaluation. Many of the examples in the ts/examples/tutorials are parallel as are in the subdirectory advection-diffusion-reaction Barry > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: Only for sequential runs! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Development HG revision: fab2442df69c95ef24bf6631a38c7dc028d7ee34 HG Date: Tue Feb 19 16:16:15 2013 -0600 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ex9bus on a arch-linux2-c-debug named olympus.local by d3m956 Wed Feb 20 11:36:02 2013 > [0]PETSC ERROR: Libraries linked from /pic/projects/ds/petsc-dev/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Wed Feb 20 11:01:15 2013 > [0]PETSC ERROR: Configure options > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: main() line 807 in src/ts/examples/tutorials/power_grid/stability_9busex9bus.c > > Does that mean the parallelized version of DAE solver in PETSc is not available yet? Or I missed some PETSc options to enable the function? > > Another question is I also found two other examples in previous installed non-dev PETSc version: > petsc-3.3-p6/src/ts/examples/tutorials/ex8.c and ex19.c. I felt that they are somewhat similar to my problem. Do you think those two are useful examples to me as well? However, it seems like they are uniprocessor examples only too. Please correct me if I understand it wrong. > > Thanks, > Shuangshuang > > > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Shri > Sent: Wednesday, February 20, 2013 10:49 AM > To: PETSc users list > Subject: Re: [petsc-users] PETSc DAE solver > > http://www.mcsanl.gov/petsc/developers/index.html > > On Feb 20, 2013, at 12:39 PM, Jin, Shuangshuang wrote: > > > Shri, thank you! That?s very helpful information. I?m going to obtain the development version of PETSc now. > > Thanks, > Shuangshuang > > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Shri > Sent: Wednesday, February 20, 2013 10:34 AM > To: PETSc users list > Subject: Re: [petsc-users] PETSc DAE solver > > Jin, > If you are working on integrating power grid DAE equations then you should use petsc-dev instead as we've added an example for it. petsc-dev/src/ts/examples/tutorials/power_grid/stabiltiy_9bus/ex9bus.c. There are other examples in the power_grid directory. > > Shri > > On Feb 20, 2013, at 12:19 PM, Jin, Shuangshuang wrote: > > > > Hi, I?m trying to use PETSc to solve a set of first-order differential and algebraic equations such as: > x' = f(x,y) > 0 = g(x,y) > > The resources I can find online so far are > ? PETSc Manual: Chapter 6 TS: Scalable ODE and DAE Solvers > ? petsc-3.3-p6/src/ts/examples/tutorials > which still seems vague to me at this moment. > > Can anyone tell me if there?s any more specific references I can refer to as a starting point to tackle this problem? > > Thanks, > Shuangshuang > > From abhyshr at mcs.anl.gov Wed Feb 20 14:39:02 2013 From: abhyshr at mcs.anl.gov (Shri) Date: Wed, 20 Feb 2013 14:39:02 -0600 Subject: [petsc-users] PETSc DAE solver In-Reply-To: <6778DE83AB681D49BFC2CD850610FEB1018FC699EB2F@EMAIL04.pnl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FC699EAB7@EMAIL04.pnl.gov> <8F12B6A5-1944-47F7-80D0-F51454C755F2@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FC699EAC7@EMAIL04.pnl.gov> <03A84ACD-D18D-4EC9-B868-72E2AE208A6C@mcs.anl.gov> <6778DE83AB681D49BFC2CD850610FEB1018FC699EB2F@EMAIL04.pnl.gov> Message-ID: <88612E31-1093-4588-8B70-B45FA27B705B@mcs.anl.gov> On Feb 20, 2013, at 2:08 PM, Jin, Shuangshuang wrote: > Hi, Shri, I was able to run the ex9bus.c code successfully with ?mpirun ?n 1 ex9bus?, but failed with ?mpirun ?n 2 ex9bus? with the following message: > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: Only for sequential runs! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Development HG revision: fab2442df69c95ef24bf6631a38c7dc028d7ee34 HG Date: Tue Feb 19 16:16:15 2013 -0600 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ex9bus on a arch-linux2-c-debug named olympus.local by d3m956 Wed Feb 20 11:36:02 2013 > [0]PETSC ERROR: Libraries linked from /pic/projects/ds/petsc-dev/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Wed Feb 20 11:01:15 2013 > [0]PETSC ERROR: Configure options > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: main() line 807 in src/ts/examples/tutorials/power_grid/stability_9busex9bus.c > > Does that mean the parallelized version of DAE solver in PETSc is not available yet? Or I missed some PETSc options to enable the function? This example is written for a single processor only, it cannot be run in parallel. > > Another question is I also found two other examples in previous installed non-dev PETSc version: > petsc-3.3-p6/src/ts/examples/tutorials/ex8.c and ex19.c. These examples are in petsc-dev too. ex8 is actually an ODE but it shows how to use the framework for implicit time-stepping solvers. ex19 is a DAE example. > I felt that they are somewhat similar to my problem. Its hard for me to tell what "my problem" means. What are you trying to solve? > Do you think those two are useful examples to me as well? However, it seems like they are uniprocessor examples only too. Yes these are all uniprocessor examples because they are really small. See ex17 for a parallel TS example. > Please correct me if I understand it wrong. > > Thanks, > Shuangshuang > > > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Shri > Sent: Wednesday, February 20, 2013 10:49 AM > To: PETSc users list > Subject: Re: [petsc-users] PETSc DAE solver > > http://www.mcsanl.gov/petsc/developers/index.html > > On Feb 20, 2013, at 12:39 PM, Jin, Shuangshuang wrote: > > > Shri, thank you! That?s very helpful information. I?m going to obtain the development version of PETSc now. > > Thanks, > Shuangshuang > > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Shri > Sent: Wednesday, February 20, 2013 10:34 AM > To: PETSc users list > Subject: Re: [petsc-users] PETSc DAE solver > > Jin, > If you are working on integrating power grid DAE equations then you should use petsc-dev instead as we've added an example for it. petsc-dev/src/ts/examples/tutorials/power_grid/stabiltiy_9bus/ex9bus.c. There are other examples in the power_grid directory. > > Shri > > On Feb 20, 2013, at 12:19 PM, Jin, Shuangshuang wrote: > > > > Hi, I?m trying to use PETSc to solve a set of first-order differential and algebraic equations such as: > x' = f(x,y) > 0 = g(x,y) > > The resources I can find online so far are > PETSc Manual: Chapter 6 TS: Scalable ODE and DAE Solvers > petsc-3.3-p6/src/ts/examples/tutorials > which still seems vague to me at this moment. > > Can anyone tell me if there?s any more specific references I can refer to as a starting point to tackle this problem? > > Thanks, > Shuangshuang > -------------- next part -------------- An HTML attachment was scrubbed... URL: From slivkaje at gmail.com Wed Feb 20 15:25:58 2013 From: slivkaje at gmail.com (Jelena Slivka) Date: Wed, 20 Feb 2013 16:25:58 -0500 Subject: [petsc-users] fminunc in PETSc In-Reply-To: <7BCE14D1-AA29-428A-A5CD-1E7F3EE3286A@mcs.anl.gov> References: <7BCE14D1-AA29-428A-A5CD-1E7F3EE3286A@mcs.anl.gov> Message-ID: Thank you very much. On Wed, Feb 20, 2013 at 2:32 PM, Barry Smith wrote: > > http://www.mcs.anl.gov/research/projects/tao/ > > > On Feb 20, 2013, at 1:06 PM, Jelena Slivka wrote: > > > Hello, > > I am trying to solve an unconstrained convex optimization problem and I > am looking for a function similar to Matlab's fminunc to which I could > provide a function that calculates the derivative of my objective. Is there > a similar function in PETSc package? > > Grateful in advance, > > Jelena Slivka > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knteran at sandia.gov Wed Feb 20 20:08:51 2013 From: knteran at sandia.gov (Teranishi, Keita) Date: Thu, 21 Feb 2013 02:08:51 +0000 Subject: [petsc-users] A quick question In-Reply-To: Message-ID: <6CEF41059DA2934FA3DCFEF07584EC61AEA513@EXMB09.srn.sandia.gov> Hi PETSc users. I just want to double-chcek if MatMultMPIAIJ requires matrix and vector to have the same data (row) partitioning. Otherwise, it throws PETSC_ERROR. Is it true? Thanks, Keita -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Feb 20 20:15:06 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 20 Feb 2013 20:15:06 -0600 Subject: [petsc-users] A quick question In-Reply-To: <6CEF41059DA2934FA3DCFEF07584EC61AEA513@EXMB09.srn.sandia.gov> References: <6CEF41059DA2934FA3DCFEF07584EC61AEA513@EXMB09.srn.sandia.gov> Message-ID: On Wed, Feb 20, 2013 at 8:08 PM, Teranishi, Keita wrote: > Hi PETSc users. > > I just want to double-chcek if MatMultMPIAIJ requires matrix and vector > to have the same data (row) partitioning. Otherwise, it throws > PETSC_ERROR. Is it true? > Yes, in A * x ---> y, the column-layout of A must match x and the row-layout must match y. We should raise an error in case of mismatch. -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Thu Feb 21 07:22:23 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Thu, 21 Feb 2013 14:22:23 +0100 Subject: [petsc-users] advice on a linear system solution for a matrix in blocks Message-ID: <51261F8F.1090509@tudelft.nl> Dear all, I have a linear system in blocks that I would like to solve. The system is written as [ A C^T ] b_1 = [ C B] b_2 where C matrix results from a coupling condition in fluid-structure interaction and rather sparse depending on the size of the coupling surfaces. A and B are also sparse symmetric matrices. I was wondering if I could find a solution to this problem without factorizing the complete operator matrix but only using, somehow, A and B and using some algebraic tricks and the sparsity of C. Any pointers/ideas are appreciated. Best regards, Umut From abhyshr at mcs.anl.gov Thu Feb 21 07:35:38 2013 From: abhyshr at mcs.anl.gov (Shri) Date: Thu, 21 Feb 2013 07:35:38 -0600 (CST) Subject: [petsc-users] advice on a linear system solution for a matrix in blocks In-Reply-To: <51261F8F.1090509@tudelft.nl> Message-ID: <291684369.5686930.1361453738161.JavaMail.root@mcs.anl.gov> PETSc's solver for block matrices PCFieldsplit http://www.mcs.anl.gov/petsc/petsc-3.3/docs/manualpages/PC/PCFIELDSPLIT.html#PCFIELDSPLIT is what you need. More information on fieldsplit is given in section 4.5 of the manual. ----- Original Message ----- > Dear all, > > I have a linear system in blocks that I would like to solve. > > The system is written as > > [ A C^T ] ? b_1 > ? ? ? ? ? ? ? ?= > [ C ? ? ? B] ? b_2 > > where C matrix results from a coupling condition in fluid-structure > interaction and rather sparse depending on the size of the coupling > surfaces. A and B are also sparse symmetric matrices. > > I was wondering if I could find a solution to this problem without > factorizing the complete operator matrix but only using, somehow, A > and > B and using some algebraic tricks and the sparsity of C. > > Any pointers/ideas are appreciated. > > Best regards, > > Umut > From balay at mcs.anl.gov Thu Feb 21 11:05:31 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 21 Feb 2013 11:05:31 -0600 (CST) Subject: [petsc-users] pass PETSC_NULL_INTEGER to dynamic array of Vec in Frotran90 In-Reply-To: References: Message-ID: For one - you might need PETSC_NULL_OBJECT. And even then - perhaps the compiler might complain [as it expects an array]. And then the fortran interface to MatNullSpaceCreate() doesn't have code to handle PETSC_NULL for this argument. [so it won't work anyway] So you are better off passing in a dummy Vec Vec :: dummyv(1) Also - its best to pass in 'variables' not cosntants like '0' to fortran subroutines. PetscInt :: zero zero = 0 Satish On Wed, 20 Feb 2013, Mi Yan wrote: > xlf9014.1 reported error for the below FORTRAN 90 code: > > ****** > MatNullSpace nullsp > PetscErrorCode :: ierr > call MatNullSpaceCreate( PETSC_COMM_WORLD, PETSC_TRUE, 0, > PETSC_NULL_INTEGER, nullsp, ierr ); > **** > The error message is: > Actual argument attributes do not match those specified by an accessible > explicit interface. > > The error implies the real argument the attribute of "PETSC_NULL_INTEGER" > does not match with those specified by the interface. The interface of > MatNullSpaceCreate is declared in finclude/ftn-auto/petscmat.h90 , whose > 4th dummy argument is declared as > Vec vecs (*) > > In my code, PETSC_NULL_INTEGER is passed to an array of Vec. Is there > anything wrong here? > > thanks, > Mi From irving at naml.us Thu Feb 21 13:06:03 2013 From: irving at naml.us (Geoffrey Irving) Date: Thu, 21 Feb 2013 11:06:03 -0800 Subject: [petsc-users] best way to have debug/without-debug builds side by side in petsc-dev Message-ID: What's the standard way of having debug and non-debug builds side by side in an instance of petsc-dev? If I do ./configure --with-shared-libraries=1 --PETSC_DIR=/Users/irving/otherlab/petsc --download-ml=yes --with-mpi-dir=/usr/local --with-debugging=0 it defaults to PETSC_ARCH=darwin10.6.0-cxx-debug, which is the same PETSC_ARCH as for --with-debugging=1. I can manually change PETSC_ARCH, but it seems odd to have to do this. Thanks, Geoffrey From balay at mcs.anl.gov Thu Feb 21 13:10:06 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 21 Feb 2013 13:10:06 -0600 (CST) Subject: [petsc-users] best way to have debug/without-debug builds side by side in petsc-dev In-Reply-To: References: Message-ID: On Thu, 21 Feb 2013, Geoffrey Irving wrote: > What's the standard way of having debug and non-debug builds side by > side in an instance of petsc-dev? If I do > > ./configure --with-shared-libraries=1 > --PETSC_DIR=/Users/irving/otherlab/petsc --download-ml=yes > --with-mpi-dir=/usr/local --with-debugging=0 > > it defaults to PETSC_ARCH=darwin10.6.0-cxx-debug, which is the same > PETSC_ARCH as for --with-debugging=1. I can manually change > PETSC_ARCH, but it seems odd to have to do this. Perhaps you have PETSC_ARCH set in env. If so - configure picks it up. Satish > > Thanks, > Geoffrey > From jedbrown at mcs.anl.gov Thu Feb 21 13:10:37 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 21 Feb 2013 13:10:37 -0600 Subject: [petsc-users] best way to have debug/without-debug builds side by side in petsc-dev In-Reply-To: References: Message-ID: On Thu, Feb 21, 2013 at 1:06 PM, Geoffrey Irving wrote: > What's the standard way of having debug and non-debug builds side by > side in an instance of petsc-dev? If I do > > ./configure --with-shared-libraries=1 > --PETSC_DIR=/Users/irving/otherlab/petsc --download-ml=yes > --with-mpi-dir=/usr/local --with-debugging=0 > > it defaults to PETSC_ARCH=darwin10.6.0-cxx-debug, which is the same > PETSC_ARCH as for --with-debugging=1. I can manually change > PETSC_ARCH, but it seems odd to have to do this. > I've always chosen my own PETSC_ARCHes (I have almost 50 of them at present). I normally do ./configure PETSC_ARCH=foo --whatever-options foo/conf/reconfigure-foo.py PETSC_ARCH=foo-opt --with-debugging=0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From irving at naml.us Thu Feb 21 13:13:33 2013 From: irving at naml.us (Geoffrey Irving) Date: Thu, 21 Feb 2013 11:13:33 -0800 Subject: [petsc-users] best way to have debug/without-debug builds side by side in petsc-dev In-Reply-To: References: Message-ID: Yep, that was the problem. Thanks, and apologies for the noise. Geoffrey On Thu, Feb 21, 2013 at 11:10 AM, Satish Balay wrote: > On Thu, 21 Feb 2013, Geoffrey Irving wrote: > >> What's the standard way of having debug and non-debug builds side by >> side in an instance of petsc-dev? If I do >> >> ./configure --with-shared-libraries=1 >> --PETSC_DIR=/Users/irving/otherlab/petsc --download-ml=yes >> --with-mpi-dir=/usr/local --with-debugging=0 >> >> it defaults to PETSC_ARCH=darwin10.6.0-cxx-debug, which is the same >> PETSC_ARCH as for --with-debugging=1. I can manually change >> PETSC_ARCH, but it seems odd to have to do this. > > Perhaps you have PETSC_ARCH set in env. If so - configure picks it up. > > Satish > >> >> Thanks, >> Geoffrey >> > From balay at mcs.anl.gov Thu Feb 21 13:25:44 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 21 Feb 2013 13:25:44 -0600 (CST) Subject: [petsc-users] best way to have debug/without-debug builds side by side in petsc-dev In-Reply-To: References: Message-ID: Generally its good to tell configure what to do [instead of it guessing]. In this case that would mean specifying PETSC_ARCH as an option to configure. Satish On Thu, 21 Feb 2013, Geoffrey Irving wrote: > Yep, that was the problem. Thanks, and apologies for the noise. > > Geoffrey > > On Thu, Feb 21, 2013 at 11:10 AM, Satish Balay wrote: > > On Thu, 21 Feb 2013, Geoffrey Irving wrote: > > > >> What's the standard way of having debug and non-debug builds side by > >> side in an instance of petsc-dev? If I do > >> > >> ./configure --with-shared-libraries=1 > >> --PETSC_DIR=/Users/irving/otherlab/petsc --download-ml=yes > >> --with-mpi-dir=/usr/local --with-debugging=0 > >> > >> it defaults to PETSC_ARCH=darwin10.6.0-cxx-debug, which is the same > >> PETSC_ARCH as for --with-debugging=1. I can manually change > >> PETSC_ARCH, but it seems odd to have to do this. > > > > Perhaps you have PETSC_ARCH set in env. If so - configure picks it up. > > > > Satish > > > >> > >> Thanks, > >> Geoffrey > >> > > > From mail2amneet at gmail.com Thu Feb 21 14:44:00 2013 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Thu, 21 Feb 2013 14:44:00 -0600 Subject: [petsc-users] MATNEST with shell matrices Message-ID: Hi All, I want to solve a system of equations which looks like A x + Bu = f1 C x + Du = f2 A, B, C, and D as shell operators and {x , u, f1, and f2} are non-native PETSc vectors. This is how I am thinking of solving it in PETSc 1) Define shell operators which define Matrix-vector multiplication routines {Ax,Bu, Cx, Du}. 2) Define a composite Matrix MAT_COMP as Mat mat_array[ ] = {A,B,C,D}; Mat MAT_COMP; MatCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL, 2, PETSC_NULL, mat_array, &MAT_COMP); 3) Define a composite Vector VEC_COMP as Vec vec_array[ ] = {f1,f2}; Vec VEC_COMP; VecCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL, vec_array, &VEC_COMP); 4) Create a shell matrix for composite matrix (MAT_COMP) which defines the action f1 = Ax + Bu; f2 = Cx + Du; i.e MATOP_MULT, MATOP_MULT_ADD, and MATOP_GET_VECS to duplicate vectors Does this sound like a reasonable approach? -- Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Feb 21 14:57:06 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 21 Feb 2013 14:57:06 -0600 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: Message-ID: What you describe will basically work, but it won't give you preconditioning. I would NOT recommend using MatNest or VecNest until you have a fully-implemented algorithm that you're happy with and that performs well with fieldsplit or custom preconditioning that will be amenable to your MatShells, AND for which you have systematically demonstrated that those operations that are comparatively expensive when using monolithic formats are actually bottlenecks for your application. If you write assembly using MatGetLocalSubMatrix [1] and MatSetValuesLocal, then adding in (optional) support for MatNest later will involve only a small amount of setup code and zero modifications to your assembly routines. Even then, I would not use VecNest. In fact, VecNest should pretty much never be used. It demonstrates a concept and may occasionally be useful, but that occurrence is so rare that you shouldn't even consider it for a new code. [1] http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html On Thu, Feb 21, 2013 at 2:44 PM, Amneet Bhalla wrote: > Hi All, > > I want to solve a system of equations which looks like > > A x + Bu = f1 > C x + Du = f2 > > A, B, C, and D as shell operators and {x , u, f1, and f2} are non-native > PETSc vectors. > > This is how I am thinking of solving it in PETSc > > 1) Define shell operators which define Matrix-vector multiplication > routines {Ax,Bu, Cx, Du}. > > 2) Define a composite Matrix MAT_COMP as > Mat mat_array[ ] = {A,B,C,D}; > Mat MAT_COMP; > MatCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL, 2, PETSC_NULL, > mat_array, &MAT_COMP); > > 3) Define a composite Vector VEC_COMP as > Vec vec_array[ ] = {f1,f2}; > Vec VEC_COMP; > VecCreateNest(PETSC_COMM_WORLD, 2, PETSC_NULL, vec_array, &VEC_COMP); > > 4) Create a shell matrix for composite matrix (MAT_COMP) which defines the > action > f1 = Ax + Bu; > f2 = Cx + Du; > > i.e MATOP_MULT, MATOP_MULT_ADD, and MATOP_GET_VECS to duplicate vectors > > > Does this sound like a reasonable approach? > > -- > Amneet > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail2amneet at gmail.com Thu Feb 21 15:20:49 2013 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Thu, 21 Feb 2013 15:20:49 -0600 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: Message-ID: On Thu, Feb 21, 2013 at 2:57 PM, Jed Brown wrote: > that you're happy with and that performs well with fieldsplit or custom > preconditioning that will be amenable to your MatShells Is fieldsplit an alternative approach to MatNest and VecNest? The problem is that I am working on nonnative PETSc data; so I am not sure which recent methods would support shell approach. All I want is to use PETSc's algorithm for solving multicomponent system of equation. I am not bothering about preconditioning yet, as I will be trying a custom preconditioner down the road. -- Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Feb 21 15:24:53 2013 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 21 Feb 2013 16:24:53 -0500 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: Message-ID: On Thu, Feb 21, 2013 at 4:20 PM, Amneet Bhalla wrote: > > On Thu, Feb 21, 2013 at 2:57 PM, Jed Brown wrote: > >> that you're happy with and that performs well with fieldsplit or custom >> preconditioning that will be amenable to your MatShells > > > Is fieldsplit an alternative approach to MatNest and VecNest? The problem > is that I am working on nonnative PETSc > data; so I am not sure which recent methods would support shell approach. > All I want is to use PETSc's algorithm > for solving multicomponent system of equation. I am not bothering about > preconditioning yet, as I will be trying > a custom preconditioner down the road. > MatNest is just an optimization matrix format. Get your stuff working with FieldSplit and then think about optimization. Matt > > -- > Amneet > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail2amneet at gmail.com Thu Feb 21 16:19:01 2013 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Thu, 21 Feb 2013 16:19:01 -0600 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: Message-ID: Can you guys comment on what example case would be best to start off for shell operators with FieldSplit? The examples I am looking into all start with creating native PETSc matrices and vectors. If possible could you write down steps that would be needed in setting such system... a) Create matrices using XXXX (?) b) Create vectors using YYYY .... On Thu, Feb 21, 2013 at 3:24 PM, Matthew Knepley wrote: > On Thu, Feb 21, 2013 at 4:20 PM, Amneet Bhalla wrote: > >> >> On Thu, Feb 21, 2013 at 2:57 PM, Jed Brown wrote: >> >>> that you're happy with and that performs well with fieldsplit or custom >>> preconditioning that will be amenable to your MatShells >> >> >> Is fieldsplit an alternative approach to MatNest and VecNest? The problem >> is that I am working on nonnative PETSc >> data; so I am not sure which recent methods would support shell approach. >> All I want is to use PETSc's algorithm >> for solving multicomponent system of equation. I am not bothering about >> preconditioning yet, as I will be trying >> a custom preconditioner down the road. >> > > MatNest is just an optimization matrix format. Get your stuff working with > FieldSplit and then think about optimization. > > Matt > > >> >> -- >> Amneet >> >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Feb 21 16:21:34 2013 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 21 Feb 2013 17:21:34 -0500 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: Message-ID: On Thu, Feb 21, 2013 at 5:19 PM, Amneet Bhalla wrote: > Can you guys comment on what example case would be best to start off for > shell > operators with FieldSplit? The examples I am looking into all start with > creating native > PETSc matrices and vectors. > > If possible could you write down steps that would be needed in setting > such system... > > a) Create matrices using XXXX (?) b) Create vectors using YYYY .... > 1) Solve your system with a standard KSP solver, like GMRES 2) Tell PCFIELDSPLIT about your fields using PCFieldSplitSetIS(), http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCFieldSplitSetIS.html Now you are ready. Matt > On Thu, Feb 21, 2013 at 3:24 PM, Matthew Knepley wrote: > >> On Thu, Feb 21, 2013 at 4:20 PM, Amneet Bhalla wrote: >> >>> >>> On Thu, Feb 21, 2013 at 2:57 PM, Jed Brown wrote: >>> >>>> that you're happy with and that performs well with fieldsplit or custom >>>> preconditioning that will be amenable to your MatShells >>> >>> >>> Is fieldsplit an alternative approach to MatNest and VecNest? The >>> problem is that I am working on nonnative PETSc >>> data; so I am not sure which recent methods would support shell >>> approach. All I want is to use PETSc's algorithm >>> for solving multicomponent system of equation. I am not bothering about >>> preconditioning yet, as I will be trying >>> a custom preconditioner down the road. >>> >> >> MatNest is just an optimization matrix format. Get your stuff working >> with FieldSplit and then think about optimization. >> >> Matt >> >> >>> >>> -- >>> Amneet >>> >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Amneet > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Feb 21 16:27:52 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 21 Feb 2013 16:27:52 -0600 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: Message-ID: Start by looking as src/snes/examples/tutorials/ex28.c, then try to structure your code similarly. You can also look at src/ksp/ksp/examples/tests/ex22.c. I will also strengthen my previous statement. You will be doing yourself a disservice in terms of understanding of PETSc, testability, and practicality if you try to write everything using MatShell now. Don't do it. Use normal matrices and follow the structure of ex28.c. Even if your eventual goal is to do everything with MatShell, you'll get there sooner (and have a better code) if you DON'T start out using MatShell. On Thu, Feb 21, 2013 at 4:19 PM, Amneet Bhalla wrote: > Can you guys comment on what example case would be best to start off for > shell > operators with FieldSplit? The examples I am looking into all start with > creating native > PETSc matrices and vectors. > > If possible could you write down steps that would be needed in setting > such system... > > a) Create matrices using XXXX (?) b) Create vectors using YYYY .... > > > On Thu, Feb 21, 2013 at 3:24 PM, Matthew Knepley wrote: > >> On Thu, Feb 21, 2013 at 4:20 PM, Amneet Bhalla wrote: >> >>> >>> On Thu, Feb 21, 2013 at 2:57 PM, Jed Brown wrote: >>> >>>> that you're happy with and that performs well with fieldsplit or custom >>>> preconditioning that will be amenable to your MatShells >>> >>> >>> Is fieldsplit an alternative approach to MatNest and VecNest? The >>> problem is that I am working on nonnative PETSc >>> data; so I am not sure which recent methods would support shell >>> approach. All I want is to use PETSc's algorithm >>> for solving multicomponent system of equation. I am not bothering about >>> preconditioning yet, as I will be trying >>> a custom preconditioner down the road. >>> >> >> MatNest is just an optimization matrix format. Get your stuff working >> with FieldSplit and then think about optimization. >> >> Matt >> >> >>> >>> -- >>> Amneet >>> >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Amneet > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Feb 21 17:06:48 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 21 Feb 2013 17:06:48 -0600 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: Message-ID: On Thu, Feb 21, 2013 at 4:19 PM, Amneet Bhalla wrote: > Can you guys comment on what example case would be best to start off for > shell > operators with FieldSplit? The examples I am looking into all start with > creating native > PETSc matrices and vectors. > Amneet, do you already have code that applies all the "blocks" of your coupled system or are you starting something new? If it's something new, please don't use MatNest/MatShell just yet. If you have tested existing code, then wrapping it in MatShell/MatNest is fine. If you are working on something new, I recommend the progression below. It will encourage better program structure and better debuggability, and will ultimately be faster than trying to "skip" steps. Step 1: Just write a residual and use -snes_mf to solve all matrix-free without preconditioning. Step 2: Assemble an approximate Jacobian and use -snes_mf_operator Step 3: use fd_coloring (if possible) to see how much solver benefit you could gain by implementing the full Jacobian. Also use fieldsplit solvers to find out which blocks of the Jacobian are most important to assemble. Step 4: Implement those blocks of the Jacobian that you need for effective preconditioning. Step 5: Profile, consider those preconditioners that are most effective and the suitability of the discretization for matrix-free application. If you spend a lot of time/memory in things like MatGetSubMatrix, then add an option to use MatNest. If you have overly heavy matrices (in terms of memory, bandwidth, or assembly time) that need not be completely assembled for effective preconditioning, add an option to use MatShell for those parts. -------------- next part -------------- An HTML attachment was scrubbed... URL: From griffith at cims.nyu.edu Thu Feb 21 17:12:58 2013 From: griffith at cims.nyu.edu (Boyce Griffith) Date: Thu, 21 Feb 2013 18:12:58 -0500 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: Message-ID: <5126A9FA.3060203@cims.nyu.edu> Sorry I couldn't jump in earlier --- I had a proposal going out today and am just now catching up on other email. Amneet is trying to solve a system the couples together some field data that is described using SAMRAI data structures along with particles that are stored in a standard PETSc Vec. The hook between the SAMRAI data and PETSc is via a version of Bobby Philip's SAMRAI-to-PETSc interface that I've bastardized for my own purposes. There is no support for {Mat,Vec}SetValues here --- the wrapper just provides standard vector space operations required by Krylov and Newton-Krylov solvers. I used to have some code to make collections of Mat's and Vec's to make it feasible to solve systems involving both native and non-native PETSc data structures. (It was similar to some code for making Vec's of Vec's that someone --- Dave May? --- used to maintain.) I ripped this stuff out of my code a while ago, and I suggested that Amneet investigate {Vec,Mat}Nest before we tried to resurrect that old code. -- Boyce On 2/21/13 5:27 PM, Jed Brown wrote: > Start by looking as src/snes/examples/tutorials/ex28.c, then try to > structure your code similarly. You can also look at > src/ksp/ksp/examples/tests/ex22.c. > > I will also strengthen my previous statement. You will be doing yourself > a disservice in terms of understanding of PETSc, testability, and > practicality if you try to write everything using MatShell now. Don't do > it. Use normal matrices and follow the structure of ex28.c. Even if your > eventual goal is to do everything with MatShell, you'll get there sooner > (and have a better code) if you DON'T start out using MatShell. > > > On Thu, Feb 21, 2013 at 4:19 PM, Amneet Bhalla > wrote: > > Can you guys comment on what example case would be best to start off > for shell > operators with FieldSplit? The examples I am looking into all start > with creating native > PETSc matrices and vectors. > > If possible could you write down steps that would be needed in > setting such system... > > a) Create matrices using XXXX (?) b) Create vectors using YYYY .... > > > On Thu, Feb 21, 2013 at 3:24 PM, Matthew Knepley > wrote: > > On Thu, Feb 21, 2013 at 4:20 PM, Amneet Bhalla > > wrote: > > > On Thu, Feb 21, 2013 at 2:57 PM, Jed Brown > > wrote: > > that you're happy with and that performs well with > fieldsplit or custom preconditioning that will be > amenable to your MatShells > > > Is fieldsplit an alternative approach to MatNest and > VecNest? The problem is that I am working on nonnative PETSc > data; so I am not sure which recent methods would support > shell approach. All I want is to use PETSc's algorithm > for solving multicomponent system of equation. I am not > bothering about preconditioning yet, as I will be trying > a custom preconditioner down the road. > > > MatNest is just an optimization matrix format. Get your stuff > working with FieldSplit and then think about optimization. > > Matt > > > -- > Amneet > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to > which their experiments lead. > -- Norbert Wiener > > > > > -- > Amneet > > > > From mail2amneet at gmail.com Thu Feb 21 17:16:14 2013 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Thu, 21 Feb 2013 17:16:14 -0600 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: Message-ID: Jed, the thing is that I have nonnative PETSc data to work with. It's from SAMRAI library. We have already defined MatShell to define the individual operators that act on nonnative vectors; so writing explicit matrices in PETSc would be very difficult (atleast for me) and time consuming. What we want is PETSc's algorithm (and NOT the data structures) to solve the problem. On Thu, Feb 21, 2013 at 5:06 PM, Jed Brown wrote: > > On Thu, Feb 21, 2013 at 4:19 PM, Amneet Bhalla wrote: > >> Can you guys comment on what example case would be best to start off for >> shell >> operators with FieldSplit? The examples I am looking into all start with >> creating native >> PETSc matrices and vectors. >> > > > Amneet, do you already have code that applies all the "blocks" of your > coupled system or are you starting something new? If it's something new, > please don't use MatNest/MatShell just yet. If you have tested existing > code, then wrapping it in MatShell/MatNest is fine. If you are working on > something new, I recommend the progression below. It will encourage better > program structure and better debuggability, and will ultimately be faster > than trying to "skip" steps. > > Step 1: Just write a residual and use -snes_mf to solve all matrix-free > without preconditioning. > > Step 2: Assemble an approximate Jacobian and use -snes_mf_operator > > Step 3: use fd_coloring (if possible) to see how much solver benefit you > could gain by implementing the full Jacobian. Also use fieldsplit solvers > to find out which blocks of the Jacobian are most important to assemble. > > Step 4: Implement those blocks of the Jacobian that you need for effective > preconditioning. > > Step 5: Profile, consider those preconditioners that are most effective > and the suitability of the discretization for matrix-free application. If > you spend a lot of time/memory in things like MatGetSubMatrix, then add an > option to use MatNest. If you have overly heavy matrices (in terms of > memory, bandwidth, or assembly time) that need not be completely assembled > for effective preconditioning, add an option to use MatShell for those > parts. > > > -- Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Feb 21 17:29:32 2013 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 21 Feb 2013 18:29:32 -0500 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: Message-ID: On Thu, Feb 21, 2013 at 6:16 PM, Amneet Bhalla wrote: > Jed, the thing is that I have nonnative PETSc data to work with. It's from > SAMRAI > library. We have already defined MatShell to define the individual > operators that act > on nonnative vectors; so writing explicit matrices in PETSc would be very > difficult (atleast for me) > and time consuming. What we want is PETSc's algorithm (and NOT the data > structures) > to solve the problem. > So I would reiterate: 1) Solve your system with a standard KSP solver, like GMRES 2) Tell PCFIELDSPLIT about your fields using PCFieldSplitSetIS(), http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCFieldSplitSetIS.html When running FieldSplit, crank down on all solvers (ksp_rtol 1.0e-10) to start and see if there is any hope of solving it. This will take forever to run. If it works, start backing off the tolerances. The tolerances that need to stay low represent the blocks you need to fill-in with an approximation that can be used to build a preconditioner. Matt > On Thu, Feb 21, 2013 at 5:06 PM, Jed Brown wrote: > >> >> On Thu, Feb 21, 2013 at 4:19 PM, Amneet Bhalla wrote: >> >>> Can you guys comment on what example case would be best to start off for >>> shell >>> operators with FieldSplit? The examples I am looking into all start with >>> creating native >>> PETSc matrices and vectors. >>> >> >> >> Amneet, do you already have code that applies all the "blocks" of your >> coupled system or are you starting something new? If it's something new, >> please don't use MatNest/MatShell just yet. If you have tested existing >> code, then wrapping it in MatShell/MatNest is fine. If you are working on >> something new, I recommend the progression below. It will encourage better >> program structure and better debuggability, and will ultimately be faster >> than trying to "skip" steps. >> >> Step 1: Just write a residual and use -snes_mf to solve all matrix-free >> without preconditioning. >> >> Step 2: Assemble an approximate Jacobian and use -snes_mf_operator >> >> Step 3: use fd_coloring (if possible) to see how much solver benefit you >> could gain by implementing the full Jacobian. Also use fieldsplit solvers >> to find out which blocks of the Jacobian are most important to assemble. >> >> Step 4: Implement those blocks of the Jacobian that you need for >> effective preconditioning. >> >> Step 5: Profile, consider those preconditioners that are most effective >> and the suitability of the discretization for matrix-free application. If >> you spend a lot of time/memory in things like MatGetSubMatrix, then add an >> option to use MatNest. If you have overly heavy matrices (in terms of >> memory, bandwidth, or assembly time) that need not be completely assembled >> for effective preconditioning, add an option to use MatShell for those >> parts. >> >> >> > > > -- > Amneet > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Feb 21 17:33:03 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 21 Feb 2013 17:33:03 -0600 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: <5126A9FA.3060203@cims.nyu.edu> References: <5126A9FA.3060203@cims.nyu.edu> Message-ID: On Thu, Feb 21, 2013 at 5:12 PM, Boyce Griffith wrote: > Sorry I couldn't jump in earlier --- I had a proposal going out today and > am just now catching up on other email. > > Amneet is trying to solve a system the couples together some field data > that is described using SAMRAI data structures along with particles that > are stored in a standard PETSc Vec. The hook between the SAMRAI data and > PETSc is via a version of Bobby Philip's SAMRAI-to-PETSc interface that > I've bastardized for my own purposes. There is no support for > {Mat,Vec}SetValues here --- the wrapper just provides standard vector space > operations required by Krylov and Newton-Krylov solvers. > Boyce, thanks for explaining. We *always* need to know this sort of background in order to make useful suggestions. > > I used to have some code to make collections of Mat's and Vec's to make it > feasible to solve systems involving both native and non-native PETSc data > structures. (It was similar to some code for making Vec's of Vec's that > someone --- Dave May? --- used to maintain.) I ripped this stuff out of my > code a while ago, and I suggested that Amneet investigate {Vec,Mat}Nest > before we tried to resurrect that old code. > That was called PetscExt. Dave and I wrote MatNest out of desire to do the sort of things that PetscExt did in a way that was more flexible and would interoperate better with the rest of PETSc. It, and PCFieldSplit should be a complete replacement for anything you might have used PetscExt for in the past. If you already have MatShells around for all the blocks, you can easily wire them together with MatNest. Note that MatNest does *not* depend on VecNest, and in general, I recommend not using VecNest. If you call VecGetSubVector() with a contiguous index set, you'll get a subvec back without a copy. This means that your residual evaluation only needs to provide a map from your non-contiguous local representation to a contiguous global representation. This should be no overhead and will interoperate better. But if you really want to use those other vector space wrappers, you can use VecNest. -------------- next part -------------- An HTML attachment was scrubbed... URL: From griffith at cims.nyu.edu Thu Feb 21 17:46:35 2013 From: griffith at cims.nyu.edu (Boyce Griffith) Date: Thu, 21 Feb 2013 18:46:35 -0500 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: <5126A9FA.3060203@cims.nyu.edu> Message-ID: <5126B1DB.90103@cims.nyu.edu> On 2/21/13 6:33 PM, Jed Brown wrote: > That was called PetscExt. Dave and I wrote MatNest out of desire to do > the sort of things that PetscExt did in a way that was more flexible and > would interoperate better with the rest of PETSc. It, and PCFieldSplit > should be a complete replacement for anything you might have used > PetscExt for in the past. > > If you already have MatShells around for all the blocks, you can easily > wire them together with MatNest. > > Note that MatNest does *not* depend on VecNest, and in general, I > recommend not using VecNest. If you call VecGetSubVector() with a > contiguous index set, you'll get a subvec back without a copy. This > means that your residual evaluation only needs to provide a map from > your non-contiguous local representation to a contiguous global > representation. This should be no overhead and will interoperate better. There are no index sets available for the SAMRAI vector data. The interface only provides access to "capital-V Vector Space" operations (vector addition, scalar multiplication, dot product, etc.). We've discussed this briefly before on some list --- petsc-dev? --- with Bobby Philip, but the basic story is that SAMRAI does patch-based AMR and separately allocates data for each patch as simple FORTRAN-style arrays (e.g. ghost cells are "co-mingled" with patch DOFs). This makes it cumbersome to generate a Vec representation that plays nice with PETSc. I am working on some code to make it feasible to map SAMRAI data onto a native PETSc Vec and to generate unique indices for the different data centerings provided by SAMRAI that we use regularly, but at least for now Amneet isn't trying to use that stuff. (And by "working on", here I really mean thinking about working on while I am instead writing grant applications. ;-) It also seems almost feasible to use one of the existing DMs to wrap the SAMRAI data. I haven't thought about this in a while and don't remember what I thought sounded like it was feasible. > But if you really want to use those other vector space wrappers, you can > use VecNest. Great --- this is what I had hoped and what Amneet was trying to figure out. Thanks, -- Boyce From mail2amneet at gmail.com Thu Feb 21 17:48:09 2013 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Thu, 21 Feb 2013 17:48:09 -0600 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: <5126A9FA.3060203@cims.nyu.edu> Message-ID: Jed, so to use VecGetSubVector() with a regular Vec; it would need to set Vec->map->bs = no_of_components that I want to solve for? I understand that a wrapper code needs to be written for gelling SAMRAI data field and PETSc Vec; I am referring to such vector wrapper here. On Thu, Feb 21, 2013 at 5:33 PM, Jed Brown wrote: > > On Thu, Feb 21, 2013 at 5:12 PM, Boyce Griffith wrote: > >> Sorry I couldn't jump in earlier --- I had a proposal going out today and >> am just now catching up on other email. >> >> Amneet is trying to solve a system the couples together some field data >> that is described using SAMRAI data structures along with particles that >> are stored in a standard PETSc Vec. The hook between the SAMRAI data and >> PETSc is via a version of Bobby Philip's SAMRAI-to-PETSc interface that >> I've bastardized for my own purposes. There is no support for >> {Mat,Vec}SetValues here --- the wrapper just provides standard vector space >> operations required by Krylov and Newton-Krylov solvers. >> > > Boyce, thanks for explaining. We *always* need to know this sort of > background in order to make useful suggestions. > > >> >> I used to have some code to make collections of Mat's and Vec's to make >> it feasible to solve systems involving both native and non-native PETSc >> data structures. (It was similar to some code for making Vec's of Vec's >> that someone --- Dave May? --- used to maintain.) I ripped this stuff out >> of my code a while ago, and I suggested that Amneet investigate >> {Vec,Mat}Nest before we tried to resurrect that old code. >> > > That was called PetscExt. Dave and I wrote MatNest out of desire to do the > sort of things that PetscExt did in a way that was more flexible and would > interoperate better with the rest of PETSc. It, and PCFieldSplit should be > a complete replacement for anything you might have used PetscExt for in the > past. > > If you already have MatShells around for all the blocks, you can easily > wire them together with MatNest. > > Note that MatNest does *not* depend on VecNest, and in general, I > recommend not using VecNest. If you call VecGetSubVector() with a > contiguous index set, you'll get a subvec back without a copy. This means > that your residual evaluation only needs to provide a map from your > non-contiguous local representation to a contiguous global representation. > This should be no overhead and will interoperate better. > > But if you really want to use those other vector space wrappers, you can > use VecNest. > -- Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Feb 21 17:51:29 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 21 Feb 2013 17:51:29 -0600 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: <5126A9FA.3060203@cims.nyu.edu> Message-ID: On Thu, Feb 21, 2013 at 5:48 PM, Amneet Bhalla wrote: > Jed, so to use VecGetSubVector() with a regular Vec; it would need to set > Vec->map->bs = no_of_components > that I want to solve for? I understand that a wrapper code needs to be > written for gelling SAMRAI data field and PETSc Vec; I am referring to such > vector wrapper here. > No, you should never touch vec->map->bs directly in any circumstance and it's totally wrong semantically here. You just need to define index sets for the pieces. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Feb 21 17:55:40 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 21 Feb 2013 17:55:40 -0600 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: <5126B1DB.90103@cims.nyu.edu> References: <5126A9FA.3060203@cims.nyu.edu> <5126B1DB.90103@cims.nyu.edu> Message-ID: On Thu, Feb 21, 2013 at 5:46 PM, Boyce Griffith wrote: > There are no index sets available for the SAMRAI vector data. The > interface only provides access to "capital-V Vector Space" operations > (vector addition, scalar multiplication, dot product, etc.). > > We've discussed this briefly before on some list --- petsc-dev? --- with > Bobby Philip, but the basic story is that SAMRAI does patch-based AMR and > separately allocates data for each patch as simple FORTRAN-style arrays > (e.g. ghost cells are "co-mingled" with patch DOFs). This makes it > cumbersome to generate a Vec representation that plays nice with PETSc. > Surely there is a way to distinguish ghost cells from non-ghost cells. Then you define a contiguous ordering/index set by simply marching through the non-ghost cells labeling them sequentially. > I am working on some code to make it feasible to map SAMRAI data onto a > native PETSc Vec and to generate unique indices for the different data > centerings provided by SAMRAI that we use regularly, but at least for now > Amneet isn't trying to use that stuff. (And by "working on", here I really > mean thinking about working on while I am instead writing grant > applications. ;-) > > It also seems almost feasible to use one of the existing DMs to wrap the > SAMRAI data. I haven't thought about this in a while and don't remember > what I thought sounded like it was feasible. > > > But if you really want to use those other vector space wrappers, you can >> use VecNest. >> > > Great --- this is what I had hoped and what Amneet was trying to figure > out. > If you really don't want to have an IS, you can use VecNestGetSubVec(), just be warned that this makes your code more brittle and harder to interoperate with. -------------- next part -------------- An HTML attachment was scrubbed... URL: From griffith at cims.nyu.edu Thu Feb 21 17:59:52 2013 From: griffith at cims.nyu.edu (Boyce Griffith) Date: Thu, 21 Feb 2013 18:59:52 -0500 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: <5126A9FA.3060203@cims.nyu.edu> <5126B1DB.90103@cims.nyu.edu> Message-ID: <5126B4F8.8060408@cims.nyu.edu> On 2/21/13 6:55 PM, Jed Brown wrote: > But if you really want to use those other vector space wrappers, > you can > use VecNest. > > > Great --- this is what I had hoped and what Amneet was trying to > figure out. > > > If you really don't want to have an IS, you can use VecNestGetSubVec(), > just be warned that this makes your code more brittle and harder to > interoperate with. There is no such thing as an IS for these data structures in the current implementation. -- Boyce From jedbrown at mcs.anl.gov Thu Feb 21 18:04:37 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 21 Feb 2013 18:04:37 -0600 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: <5126B4F8.8060408@cims.nyu.edu> References: <5126A9FA.3060203@cims.nyu.edu> <5126B1DB.90103@cims.nyu.edu> <5126B4F8.8060408@cims.nyu.edu> Message-ID: On Thu, Feb 21, 2013 at 5:59 PM, Boyce Griffith wrote: > There is no such thing as an IS for these data structures in the current > implementation. But there is a local and global size. (You had to pick something to create the Vec.) Therefore, the global space is _defined_ as the concatenation of all those contiguous local spaces. You need not store it that way, but the abstract space exists. If you use interfaces that ask for ISs, they will be with respect to that abstract space. -------------- next part -------------- An HTML attachment was scrubbed... URL: From griffith at cims.nyu.edu Thu Feb 21 18:07:55 2013 From: griffith at cims.nyu.edu (Boyce Griffith) Date: Thu, 21 Feb 2013 19:07:55 -0500 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: <5126A9FA.3060203@cims.nyu.edu> <5126B1DB.90103@cims.nyu.edu> <5126B4F8.8060408@cims.nyu.edu> Message-ID: <5126B6DB.4020207@cims.nyu.edu> On 2/21/13 7:04 PM, Jed Brown wrote: > > On Thu, Feb 21, 2013 at 5:59 PM, Boyce Griffith > wrote: > > There is no such thing as an IS for these data structures in the > current implementation. > > > > But there is a local and global size. (You had to pick something to > create the Vec.) Therefore, the global space is _defined_ as the > concatenation of all those contiguous local spaces. You need not store > it that way, but the abstract space exists. If you use interfaces that > ask for ISs, they will be with respect to that abstract space. Yes there is a local and global size. They are both 0. :-) At present, the wrappers don't provide any access whatsoever to the underlying data. -- Boyce From jedbrown at mcs.anl.gov Thu Feb 21 18:11:12 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 21 Feb 2013 18:11:12 -0600 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: <5126B6DB.4020207@cims.nyu.edu> References: <5126A9FA.3060203@cims.nyu.edu> <5126B1DB.90103@cims.nyu.edu> <5126B4F8.8060408@cims.nyu.edu> <5126B6DB.4020207@cims.nyu.edu> Message-ID: On Thu, Feb 21, 2013 at 6:07 PM, Boyce Griffith wrote: > > Yes there is a local and global size. They are both 0. :-) Heh, interesting. Most finite-dimensional vector spaces have a dimension. May as well use VecNestGetSubVec() when needed. I don't think there is any code that will short-circuit in the zero-size case, but I'd still encourage you to state a dimension that's not obviously bogus. -------------- next part -------------- An HTML attachment was scrubbed... URL: From griffith at cims.nyu.edu Thu Feb 21 18:12:49 2013 From: griffith at cims.nyu.edu (Boyce Griffith) Date: Thu, 21 Feb 2013 19:12:49 -0500 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: <5126A9FA.3060203@cims.nyu.edu> <5126B1DB.90103@cims.nyu.edu> <5126B4F8.8060408@cims.nyu.edu> Message-ID: <5126B801.1020505@cims.nyu.edu> On 2/21/13 7:04 PM, Jed Brown wrote: > > On Thu, Feb 21, 2013 at 5:59 PM, Boyce Griffith > wrote: > > There is no such thing as an IS for these data structures in the > current implementation. > > > > But there is a local and global size. (You had to pick something to > create the Vec.) Therefore, the global space is _defined_ as the > concatenation of all those contiguous local spaces. You need not store > it that way, but the abstract space exists. If you use interfaces that > ask for ISs, they will be with respect to that abstract space. Ah, I think I wasn't clear about what I mean here by "wrapper". Basically, we have a subclass of Vec that knows how to evaluate vector space operations using the "wrapped" SAMRAI data. For instance, there is no VecGetArray(). -- Boyce From jedbrown at mcs.anl.gov Thu Feb 21 18:21:16 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 21 Feb 2013 18:21:16 -0600 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: <5126B801.1020505@cims.nyu.edu> References: <5126A9FA.3060203@cims.nyu.edu> <5126B1DB.90103@cims.nyu.edu> <5126B4F8.8060408@cims.nyu.edu> <5126B801.1020505@cims.nyu.edu> Message-ID: On Thu, Feb 21, 2013 at 6:12 PM, Boyce Griffith wrote: > But there is a local and global size. (You had to pick something to >> create the Vec.) Therefore, the global space is _defined_ as the >> concatenation of all those contiguous local spaces. You need not store >> it that way, but the abstract space exists. If you use interfaces that >> ask for ISs, they will be with respect to that abstract space. >> > > Ah, I think I wasn't clear about what I mean here by "wrapper". > > Basically, we have a subclass of Vec that knows how to evaluate vector > space operations using the "wrapped" SAMRAI data. For instance, there is > no VecGetArray(). I understand that completely, but you cannot create a valid Vec without setting a size. Internally the PetscLayout has a size of -1 on creation and you will get errors if you try to use it without setting a size. Setting a size of 0 is patently dishonest and PETSc could justifiably short-circuit and not call into your implementation. (I don't think we do this, but it would be correct at the vector space level. PETSc Vec is a finite-dimensional thing, therefore it has a well-defined dimension.) I'm saying that you should be honest about the size even if it's not explicitly used in the functionality you happen to be using now. If you define a size, then you can also (optionally) use IS to refer to pieces, which will make interoperability with other codes, or eventual extensions (such as AMG preconditioning) less intrusive. -------------- next part -------------- An HTML attachment was scrubbed... URL: From griffith at cims.nyu.edu Thu Feb 21 18:22:08 2013 From: griffith at cims.nyu.edu (Boyce Griffith) Date: Thu, 21 Feb 2013 19:22:08 -0500 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: <5126A9FA.3060203@cims.nyu.edu> <5126B1DB.90103@cims.nyu.edu> <5126B4F8.8060408@cims.nyu.edu> <5126B6DB.4020207@cims.nyu.edu> Message-ID: <5126BA30.9090607@cims.nyu.edu> On 2/21/13 7:11 PM, Jed Brown wrote: > > On Thu, Feb 21, 2013 at 6:07 PM, Boyce Griffith > wrote: > > > Yes there is a local and global size. They are both 0. :-) > > > > Heh, interesting. Most finite-dimensional vector spaces have a > dimension. May as well use VecNestGetSubVec() when needed. I don't think > there is any code that will short-circuit in the zero-size case, but I'd > still encourage you to state a dimension that's not obviously bogus. For better or for worse, the "size" of the wrapped data can change dynamically. Somehow it seems less dishonest to set the size to 0 instead of, e.g., 1. At least when you get 0 you know that the size is bogus! :-) We've been using this stuff for ... a while now, and it hasn't caused any problems. That said, I would prefer to have tighter integration with PETSc. I think that DMComposite is what I was thinking might work. -- Boyce From irving at naml.us Thu Feb 21 18:25:19 2013 From: irving at naml.us (Geoffrey Irving) Date: Thu, 21 Feb 2013 16:25:19 -0800 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: <5126BA30.9090607@cims.nyu.edu> References: <5126A9FA.3060203@cims.nyu.edu> <5126B1DB.90103@cims.nyu.edu> <5126B4F8.8060408@cims.nyu.edu> <5126B6DB.4020207@cims.nyu.edu> <5126BA30.9090607@cims.nyu.edu> Message-ID: On Thu, Feb 21, 2013 at 4:22 PM, Boyce Griffith wrote: > > > On 2/21/13 7:11 PM, Jed Brown wrote: >> >> >> On Thu, Feb 21, 2013 at 6:07 PM, Boyce Griffith > > wrote: >> >> >> Yes there is a local and global size. They are both 0. :-) >> >> >> >> Heh, interesting. Most finite-dimensional vector spaces have a >> dimension. May as well use VecNestGetSubVec() when needed. I don't think >> there is any code that will short-circuit in the zero-size case, but I'd >> still encourage you to state a dimension that's not obviously bogus. > > > For better or for worse, the "size" of the wrapped data can change > dynamically. Somehow it seems less dishonest to set the size to 0 instead > of, e.g., 1. At least when you get 0 you know that the size is bogus! :-) > We've been using this stuff for ... a while now, and it hasn't caused any > problems. Jed is saying that if you have to use a sentinel size, use a large integer rather than zero. Petsc can't detect that you're lying in this case (unless it were to perform that many Krylov iterations in exact arithmetic), so it won't be able to break. > That said, I would prefer to have tighter integration with PETSc. I think > that DMComposite is what I was thinking might work. Geoffrey From jedbrown at mcs.anl.gov Thu Feb 21 18:31:48 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 21 Feb 2013 18:31:48 -0600 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: <5126BA30.9090607@cims.nyu.edu> References: <5126A9FA.3060203@cims.nyu.edu> <5126B1DB.90103@cims.nyu.edu> <5126B4F8.8060408@cims.nyu.edu> <5126B6DB.4020207@cims.nyu.edu> <5126BA30.9090607@cims.nyu.edu> Message-ID: On Thu, Feb 21, 2013 at 6:22 PM, Boyce Griffith wrote: > For better or for worse, the "size" of the wrapped data can change > dynamically. > Then your operators are not linear. If you are in a Vec implementation, you should be able to PetscLayoutSetSize(vec->map,newN). Changing the size dynamically without telling anyone is dangerous in the sense that it violates the API so you're totally on your own if it's breaking. Many things will work, but when I write software, I like to write it so that it works because it's correct, not just so that it works on my machine with a specific compiler, etc. If you want your code to work because it's correct, then you should only change the size at well-defined places, call KSPReset() or whatever, and generally follow the API. > Somehow it seems less dishonest to set the size to 0 instead of, e.g., 1. > At least when you get 0 you know that the size is bogus! :-) We've been > using this stuff for ... a while now, and it hasn't caused any problems. > > That said, I would prefer to have tighter integration with PETSc. I think > that DMComposite is what I was thinking might work. > DMComposite is sort of tied to contiguous memory. In its current form, you'd have to implement VecGetArray() and VecPlaceArray(). -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Feb 21 19:01:21 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 21 Feb 2013 19:01:21 -0600 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: <5126A9FA.3060203@cims.nyu.edu> <5126B1DB.90103@cims.nyu.edu> <5126B4F8.8060408@cims.nyu.edu> <5126B6DB.4020207@cims.nyu.edu> <5126BA30.9090607@cims.nyu.edu> Message-ID: On Thu, Feb 21, 2013 at 6:31 PM, Jed Brown wrote: > Changing the size dynamically without telling anyone is dangerous in the > sense that it violates the API so you're totally on your own if it's > breaking. Many things will work, but when I write software, I like to write > it so that it works because it's correct, not just so that it works on my > machine with a specific compiler, etc. If you want your code to work > because it's correct, then you should only change the size at well-defined > places, call KSPReset() or whatever, and generally follow the API. The issue here is by stating a size like 0, you're violating the contract, so anything could go wrong. Now perhaps PETSc Vecs should be generalized so that the interface supports a concept of "unknown" dimension. In that case, you would identify your Vecs as having unknown size and we would be obliged to respect it. If we make that conceptual change, we'd need to think carefully about any places in PETSc that would be broken in such circumstances. Such a generalization might involve zero lines of code (though realistically, we'd modify the validation code to find bugs from broken assemptions), but it's something that would have to be decided if you don't want a future release of PETSc to break your implementation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Feb 21 19:44:35 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 21 Feb 2013 19:44:35 -0600 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: <5126C9C6.80306@cims.nyu.edu> References: <5126A9FA.3060203@cims.nyu.edu> <5126B1DB.90103@cims.nyu.edu> <5126B4F8.8060408@cims.nyu.edu> <5126B6DB.4020207@cims.nyu.edu> <5126BA30.9090607@cims.nyu.edu> <5126C9C6.80306@cims.nyu.edu> Message-ID: On Thu, Feb 21, 2013 at 7:28 PM, Boyce Griffith wrote: > > I would very happily use an "undefined" vector size. (I will also happily > change the bogus local sizes from 0 to 1, and the global sizes accordingly. > :-) > > The PETSc solvers that we use in the code should be oblivious to the size > of the vector. I've actually never been all that happy with size "0", but > it was in some inherited code, and at least so far, it has never caused any > problems over the 10+ years it has been in use. (If ever did start causing > problems, I certainly would not come complaining to petsc-users or > petsc-dev. It isn't like I do not realize that "0" is a potentially > problematic value here.) > > I actually have always hoped that setting size to "0" would help smoke out > any places where something actually did use the vector size. > Okay, let's move the design discussion to petsc-dev. I want to understand the extent to your adaptivity and how your Vecs "behave". In which places do you (mathematically) change the dimension of a vector? If I start with Vecs X and Y and I write VecCopy(X,Y); YourSolver(X); /* solve F(X) = 0 with adaptivity */ can I then compare the result like this? VecAXPY(Y,-1.,X); Do vector-space properties of Y (e.g., norm) change when I call YourSolver(X). (Some codes keep a global view of all vectors, and when they adapt, they regrid all vectors. But regridding doesn't preserve things like norms, so you end up with tangled semantics.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Feb 21 19:50:07 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 21 Feb 2013 19:50:07 -0600 Subject: [petsc-users] [petsc-dev] MATNEST with shell matrices In-Reply-To: References: <5126A9FA.3060203@cims.nyu.edu> <5126B1DB.90103@cims.nyu.edu> <5126B4F8.8060408@cims.nyu.edu> <5126B6DB.4020207@cims.nyu.edu> <5126BA30.9090607@cims.nyu.edu> <5126C9C6.80306@cims.nyu.edu> Message-ID: The way I view changing sizes inside a (non)linear solver is that one does an approximate (non)linear solver, decides to adapt the mesh, then does another approximate (non)linear solver etc. So even though formally they may not be stopping the SNESSolve() and restarting one can think of it that way. Of course inside a Krylov linear solver this is nonsense because the inner produces cannot be done between old and new but with plain old linear multigrid it is ok. Barry On Feb 21, 2013, at 7:44 PM, Jed Brown wrote: > On Thu, Feb 21, 2013 at 7:28 PM, Boyce Griffith wrote: > > I would very happily use an "undefined" vector size. (I will also happily change the bogus local sizes from 0 to 1, and the global sizes accordingly. :-) > > The PETSc solvers that we use in the code should be oblivious to the size of the vector. I've actually never been all that happy with size "0", but it was in some inherited code, and at least so far, it has never caused any problems over the 10+ years it has been in use. (If ever did start causing problems, I certainly would not come complaining to petsc-users or petsc-dev. It isn't like I do not realize that "0" is a potentially problematic value here.) > > I actually have always hoped that setting size to "0" would help smoke out any places where something actually did use the vector size. > > Okay, let's move the design discussion to petsc-dev. > > I want to understand the extent to your adaptivity and how your Vecs "behave". In which places do you (mathematically) change the dimension of a vector? > > If I start with Vecs X and Y and I write > > VecCopy(X,Y); > YourSolver(X); /* solve F(X) = 0 with adaptivity */ > > can I then compare the result like this? > > VecAXPY(Y,-1.,X); > > Do vector-space properties of Y (e.g., norm) change when I call YourSolver(X). (Some codes keep a global view of all vectors, and when they adapt, they regrid all vectors. But regridding doesn't preserve things like norms, so you end up with tangled semantics.) From griffith at cims.nyu.edu Thu Feb 21 19:53:54 2013 From: griffith at cims.nyu.edu (Boyce Griffith) Date: Thu, 21 Feb 2013 20:53:54 -0500 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: <5126A9FA.3060203@cims.nyu.edu> <5126B1DB.90103@cims.nyu.edu> <5126B4F8.8060408@cims.nyu.edu> <5126B6DB.4020207@cims.nyu.edu> <5126BA30.9090607@cims.nyu.edu> <5126C9C6.80306@cims.nyu.edu> Message-ID: <1ACA5F14-5B26-43BB-ACFE-1B48F7CAA06F@cims.nyu.edu> On Feb 21, 2013, at 8:44 PM, Jed Brown wrote: > On Thu, Feb 21, 2013 at 7:28 PM, Boyce Griffith wrote: > > I would very happily use an "undefined" vector size. (I will also happily change the bogus local sizes from 0 to 1, and the global sizes accordingly. :-) > > The PETSc solvers that we use in the code should be oblivious to the size of the vector. I've actually never been all that happy with size "0", but it was in some inherited code, and at least so far, it has never caused any problems over the 10+ years it has been in use. (If ever did start causing problems, I certainly would not come complaining to petsc-users or petsc-dev. It isn't like I do not realize that "0" is a potentially problematic value here.) > > I actually have always hoped that setting size to "0" would help smoke out any places where something actually did use the vector size. > > Okay, let's move the design discussion to petsc-dev. > > I want to understand the extent to your adaptivity and how your Vecs "behave". In which places do you (mathematically) change the dimension of a vector? These applications are mainly transient solvers in which regridding happens "between" time steps. > If I start with Vecs X and Y and I write > > VecCopy(X,Y); > YourSolver(X); /* solve F(X) = 0 with adaptivity */ > > can I then compare the result like this? > > VecAXPY(Y,-1.,X); Absolutely. > Do vector-space properties of Y (e.g., norm) change when I call YourSolver(X). (Some codes keep a global view of all vectors, and when they adapt, they regrid all vectors. But regridding doesn't preserve things like norms, so you end up with tangled semantics.) At present all solvers operate only on fixed grid configurations. I do not expect this to change any time soon. -- Boyce -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Fri Feb 22 13:44:32 2013 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Fri, 22 Feb 2013 13:44:32 -0600 Subject: [petsc-users] using hybrid chebyshev In-Reply-To: References: <201301251456.r0PEuRVM001471@msw1.awe.co.uk> Message-ID: Wadud, The hybrid Chebyshev solver in petsc-3.3 was experimental and only partially done. I made an update to petsc-dev and added a test https://bitbucket.org/petsc/petsc-dev/commits/ee234e7e6a79c7eca4f3878f5409a67d7172e12e Currently, only runtime options are provided, e.g. petsc-dev/src/ksp/ksp/examples/tests>./ex39 -beta -0.6 -ksp_type chebyshev -h | grep cheby chebyshev gmres tcqmr bcgs ibcgs fbcgs fbcgsr bcgsl cgs tfqmr cr pipecr lsqr preonly qcg bicg fgmres minres symmlq lgmres lcd gcr pgmres specest dgmres (KSPSetType) -ksp_chebyshev_eigenvalues <0>: extreme eigenvalues (KSPChebyshevSetEigenvalues) -ksp_chebyshev_estimate_eigenvalues <-1>: estimate eigenvalues using a Krylov method, then use this transform for Chebyshev eigenvalue bounds (KSPChebyshevSetEstimateEigenvalues) -ksp_chebyshev_hybrid: Use hybrid Chebyshev () -ksp_chebyshev_hybrid_chebysteps <20000>: Number of Chebyshev steps in hybrid Chebyshev () -ksp_chebyshev_hybrid_gmressteps <10>: Number of adaptive/GMRES steps in hybrid Chebyshev () -ksp_chebyshev_hybrid_purification: Use purification in hybrid Chebyshev () Let us know if you encounter any problem or want any update when using Hybrid Chebyshev. Hong On Fri, Jan 25, 2013 at 9:16 AM, Jed Brown wrote: > KSPCHEBYSHEV (or -ksp_type chebyshev). The hybrid option is > -ksp_chebyshev_hybrid (see -help for related options). There is currently no > functional interface in code, just the options database. > > > On Fri, Jan 25, 2013 at 8:56 AM, wrote: >> >> The changelog of PETSc 3.3-p5 says that it now provides a hybrid Chebyshev >> solver. What parameter do you pass into the Fortran KSPSetType subroutine to >> use this solver? >> >> >> >> Regards, >> >> >> >> -------------------------- >> >> Wadud Miah >> HPC, Design Physics Division >> Direct: 0118 98 56220 >> AWE, Aldermaston, Reading, RG7 4PR >> >> >> >> >> >> ___________________________________________________ >> ____________________________ The information in this email and in any >> attachment(s) is commercial in confidence. If you are not the named >> addressee(s) or if you receive this email in error then any distribution, >> copying or use of this communication or the information in it is strictly >> prohibited. Please notify us immediately by email at >> admin.internet(at)awe.co.uk, and then delete this message from your >> computer. While attachments are virus checked, AWE plc does not accept any >> liability in respect of any virus which is not detected. AWE Plc Registered >> in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 >> 4PR > > From zhenglun.wei at gmail.com Fri Feb 22 14:57:28 2013 From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei) Date: Fri, 22 Feb 2013 14:57:28 -0600 Subject: [petsc-users] Non-uniform Cartisian Grid In-Reply-To: <818369A9-2984-4133-A62F-E5DCEE283EA1@mcs.anl.gov> References: <509B01D6.9030501@gmail.com> <818369A9-2984-4133-A62F-E5DCEE283EA1@mcs.anl.gov> Message-ID: <5127DBB8.6040704@gmail.com> Dear all, I hope you're having a nice day. I wonder if anyone could point out a example for me about the DMDASetCoordinates(). I found this (http://www.mcs.anl.gov/petsc/petsc-current/src/dm/impls/da/dacorn.c.html#DMDASetCoordinates) online; however it does not really demonstrate how to create the 'coordinate vector - c'. Also, I found that in the petsc sub-folder: /petsc-dev/src/dm/impls/patch has a patch.c, which I suppose that it does a local refinement for mesh. Does this a mature model that we can use or just a testing code? thanks, Alan On 11/7/2012 6:55 PM, Barry Smith wrote: > On Nov 7, 2012, at 6:50 PM, Alan wrote: > >> Dear folks, >> I hope you're having a nice day. >> I'm testing the Poisson solver in /src/ksp/ksp/examples/tutorials/ex45. >> It uses the 'DMDACreate3d' to generate a Cartisian structured grid and >> solve it with uniform grid size. However, I attempt to modify it to >> obtain solution with a Cartsian structured grid and non-uniform grid (It >> is not unstructured local refined mesh). Should I still use the >> 'DMDACreate3d' and implement different grid size in 'ComputeMatrix' and >> 'ComputeRHS'? In other words, does 'DMDACreate3d' itself include any >> mesh size information? > You can use DMDASetCoordinates() and DMDAGetCoordinates() (called DMSetCoordinates() and DMGetCoordinates() in petsc-dev) to keep track of the coordinates but you are responsible for generating the correctly scaled entries in the matrix etc. > > Barry > >> thanks, >> Alan From knepley at gmail.com Fri Feb 22 15:20:13 2013 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 22 Feb 2013 16:20:13 -0500 Subject: [petsc-users] Non-uniform Cartisian Grid In-Reply-To: <5127DBB8.6040704@gmail.com> References: <509B01D6.9030501@gmail.com> <818369A9-2984-4133-A62F-E5DCEE283EA1@mcs.anl.gov> <5127DBB8.6040704@gmail.com> Message-ID: On Fri, Feb 22, 2013 at 3:57 PM, Zhenglun (Alan) Wei wrote: > Dear all, > I hope you're having a nice day. > I wonder if anyone could point out a example for me about the > DMDASetCoordinates(). I found this > ( > http://www.mcs.anl.gov/petsc/petsc-current/src/dm/impls/da/dacorn.c.html#DMDASetCoordinates > ) > online; however it does not really demonstrate how to create the > 'coordinate vector - c'. > Also, I found that in the petsc sub-folder: > /petsc-dev/src/dm/impls/patch has a patch.c, which I suppose that it > does a local refinement for mesh. Does this a mature model that we can > use or just a testing code? > Look at KSP ex49. In the routine DMDACoordViewGnuplot2d() it reads out the coordinates. You could just as easily set them in this fashion. Matt > thanks, > Alan > > On 11/7/2012 6:55 PM, Barry Smith wrote: > > On Nov 7, 2012, at 6:50 PM, Alan wrote: > > > >> Dear folks, > >> I hope you're having a nice day. > >> I'm testing the Poisson solver in /src/ksp/ksp/examples/tutorials/ex45. > >> It uses the 'DMDACreate3d' to generate a Cartisian structured grid and > >> solve it with uniform grid size. However, I attempt to modify it to > >> obtain solution with a Cartsian structured grid and non-uniform grid (It > >> is not unstructured local refined mesh). Should I still use the > >> 'DMDACreate3d' and implement different grid size in 'ComputeMatrix' and > >> 'ComputeRHS'? In other words, does 'DMDACreate3d' itself include any > >> mesh size information? > > You can use DMDASetCoordinates() and DMDAGetCoordinates() (called > DMSetCoordinates() and DMGetCoordinates() in petsc-dev) to keep track of > the coordinates but you are responsible for generating the correctly scaled > entries in the matrix etc. > > > > Barry > > > >> thanks, > >> Alan > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bernardomartinsrocha at gmail.com Fri Feb 22 15:37:08 2013 From: bernardomartinsrocha at gmail.com (Bernardo Rocha) Date: Fri, 22 Feb 2013 18:37:08 -0300 Subject: [petsc-users] SNES Form FunctionAndJacobian Message-ID: Hi everyone, I?m using SNES to solve nonlinear problems in elasticity. I was wondering if it is possible to configure SNES for calling a single function like FormFunctionAndJacobian to compute the residual and jacobian at the same time since this is the way I was doing before using SNES in my finite element routines...An example would be a good starting point. Thanks, Bernardo M. R. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Feb 22 15:38:50 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 22 Feb 2013 15:38:50 -0600 Subject: [petsc-users] SNES Form FunctionAndJacobian In-Reply-To: References: Message-ID: http://www.mcs.anl.gov/petsc/documentation/faq.html#functionjacobian On Fri, Feb 22, 2013 at 3:37 PM, Bernardo Rocha < bernardomartinsrocha at gmail.com> wrote: > Hi everyone, > > I?m using SNES to solve nonlinear problems in elasticity. I was wondering > if it is possible to configure SNES for calling a single function like > FormFunctionAndJacobian to compute the residual and jacobian at the same > time since this is the way I was doing before using SNES in my finite > element routines...An example would be a good starting point. > > Thanks, > Bernardo M. R. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike.hui.zhang at hotmail.com Sat Feb 23 02:51:25 2013 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Sat, 23 Feb 2013 09:51:25 +0100 Subject: [petsc-users] MUMPS error Message-ID: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 70397575 megabytes The required memory is too large. Is that normal? I'm solving a Helmholtz equation on 100^3 mesh partitioned to 4^3 processors. From mike.hui.zhang at hotmail.com Sat Feb 23 03:09:29 2013 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Sat, 23 Feb 2013 10:09:29 +0100 Subject: [petsc-users] [mumps-users] MUMPS error In-Reply-To: <1361609793.52203.YahooMailNeo@web28904.mail.ir2.yahoo.com> References: <1361609793.52203.YahooMailNeo@web28904.mail.ir2.yahoo.com> Message-ID: No, it is a 27-stencil discretization. I use petsc MatXSetPreallocation with much larger estimates for off-diagonal (i.e. on other processors) zeros. Basically, I say 27 non-zeros per row are on other processors. Does this cause the MUMPS error? Thanks a lot! On Feb 23, 2013, at 9:56 AM, paresh murthy wrote: > Hello Mike, > > I would guess, as perhaps others might as well, that it is trying to solve a million x million dense matrix. > Are you sure you have removed the zeroes and given it a sparse matrix ? > > Paresh > > > From: Hui Zhang > To: mumps-users at listes.ens-lyon.fr; PETSc users list > Cc: Hui Zhang > Sent: Saturday, February 23, 2013 9:51 AM > Subject: [mumps-users] MUMPS error > > Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 70397575 megabytes > > The required memory is too large. Is that normal? I'm solving a Helmholtz equation on 100^3 mesh partitioned to 4^3 processors. > > > > From mike.hui.zhang at hotmail.com Sat Feb 23 06:29:00 2013 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Sat, 23 Feb 2013 13:29:00 +0100 Subject: [petsc-users] Scale the diagonal entries of a Mat Message-ID: I want to implement diagonal penalty method for enforcing the Dirichlet boundary conditions. That is, the diagonal entries corresponding to Dirichlet boundary are going to be scaled by a large number. What is the easiest way to do this? Thanks! From jedbrown at mcs.anl.gov Sat Feb 23 06:32:15 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 23 Feb 2013 06:32:15 -0600 Subject: [petsc-users] Scale the diagonal entries of a Mat In-Reply-To: References: Message-ID: Add big values to diagonal entries and scale the right hand side appropriately, as per the definition. It is a terrible method. On Feb 23, 2013 6:29 AM, "Hui Zhang" wrote: > I want to implement diagonal penalty method for enforcing the Dirichlet > boundary conditions. That is, the diagonal entries corresponding to > Dirichlet boundary are going to be scaled by a large number. > > What is the easiest way to do this? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Feb 23 06:32:39 2013 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 23 Feb 2013 07:32:39 -0500 Subject: [petsc-users] Scale the diagonal entries of a Mat In-Reply-To: References: Message-ID: On Sat, Feb 23, 2013 at 7:29 AM, Hui Zhang wrote: > I want to implement diagonal penalty method for enforcing the Dirichlet > boundary conditions. That is, the diagonal entries corresponding to > Dirichlet boundary are going to be scaled by a large number. > > What is the easiest way to do this? Thanks! http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatDiagonalScale.html Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike.hui.zhang at hotmail.com Sat Feb 23 06:35:15 2013 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Sat, 23 Feb 2013 13:35:15 +0100 Subject: [petsc-users] Scale the diagonal entries of a Mat In-Reply-To: References: Message-ID: On Feb 23, 2013, at 1:32 PM, Matthew Knepley wrote: > On Sat, Feb 23, 2013 at 7:29 AM, Hui Zhang wrote: > I want to implement diagonal penalty method for enforcing the Dirichlet boundary conditions. That is, the diagonal entries corresponding to Dirichlet boundary are going to be scaled by a large number. > > What is the easiest way to do this? Thanks! > > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatDiagonalScale.html But it seems not what I want. That routine scales all the entries of a mat. I want to scale only the diagonal entries. > > Matt > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From mike.hui.zhang at hotmail.com Sat Feb 23 06:36:16 2013 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Sat, 23 Feb 2013 13:36:16 +0100 Subject: [petsc-users] Scale the diagonal entries of a Mat In-Reply-To: References: Message-ID: On Feb 23, 2013, at 1:32 PM, Jed Brown wrote: > Add big values to diagonal entries and scale the right hand side appropriately, as per the definition. It is a terrible method. Why? That is what I was taught by textbooks and used by some FEM library like FreeFem++. > > On Feb 23, 2013 6:29 AM, "Hui Zhang" wrote: > I want to implement diagonal penalty method for enforcing the Dirichlet boundary conditions. That is, the diagonal entries corresponding to Dirichlet boundary are going to be scaled by a large number. > > What is the easiest way to do this? Thanks! From knepley at gmail.com Sat Feb 23 06:37:37 2013 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 23 Feb 2013 07:37:37 -0500 Subject: [petsc-users] Scale the diagonal entries of a Mat In-Reply-To: References: Message-ID: On Sat, Feb 23, 2013 at 7:35 AM, Hui Zhang wrote: > > On Feb 23, 2013, at 1:32 PM, Matthew Knepley wrote: > > > On Sat, Feb 23, 2013 at 7:29 AM, Hui Zhang > wrote: > > I want to implement diagonal penalty method for enforcing the Dirichlet > boundary conditions. That is, the diagonal entries corresponding to > Dirichlet boundary are going to be scaled by a large number. > > > > What is the easiest way to do this? Thanks! > > > > > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatDiagonalScale.html > > But it seems not what I want. That routine scales all the entries of a > mat. I want to scale only the diagonal entries. Then use http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatDiagonalSet.html As Jed says, you really really do not want to do this. Matt > > > > Matt > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Feb 23 06:47:58 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 23 Feb 2013 06:47:58 -0600 Subject: [petsc-users] Scale the diagonal entries of a Mat In-Reply-To: References: Message-ID: Use MatSetValues with ADD_VALUES. The method is used for toy examples because it is the most brain-dead simple thing to implement, not because it is a good method. Much better to lift and remove or to eliminate in-place. Any good book discusses this and there are useful answers on scicomp.stackexchange.com. On Feb 23, 2013 6:36 AM, "Hui Zhang" wrote: > > On Feb 23, 2013, at 1:32 PM, Jed Brown wrote: > > > Add big values to diagonal entries and scale the right hand side > appropriately, as per the definition. It is a terrible method. > > Why? That is what I was taught by textbooks and used by some FEM library > like FreeFem++. > > > > > On Feb 23, 2013 6:29 AM, "Hui Zhang" wrote: > > I want to implement diagonal penalty method for enforcing the Dirichlet > boundary conditions. That is, the diagonal entries corresponding to > Dirichlet boundary are going to be scaled by a large number. > > > > What is the easiest way to do this? Thanks! > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike.hui.zhang at hotmail.com Sat Feb 23 07:28:06 2013 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Sat, 23 Feb 2013 14:28:06 +0100 Subject: [petsc-users] Scale the diagonal entries of a Mat In-Reply-To: References: Message-ID: On Feb 23, 2013, at 1:37 PM, Matthew Knepley wrote: > On Sat, Feb 23, 2013 at 7:35 AM, Hui Zhang wrote: > > On Feb 23, 2013, at 1:32 PM, Matthew Knepley wrote: > > > On Sat, Feb 23, 2013 at 7:29 AM, Hui Zhang wrote: > > I want to implement diagonal penalty method for enforcing the Dirichlet boundary conditions. That is, the diagonal entries corresponding to Dirichlet boundary are going to be scaled by a large number. > > > > What is the easiest way to do this? Thanks! > > > > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatDiagonalScale.html > > But it seems not what I want. That routine scales all the entries of a mat. I want to scale only the diagonal entries. > > Then use http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatDiagonalSet.html I want to do the following on an already assembled matrix (FINAL_ASSEMBLY). MatGetLocalSubMatrix .. to get 'submat', MatGetDiagonal .. to get diagonal vector 'diag', VecScale .. to scale 'diag' MatDiagonalSet .. to set scaled 'diag' to 'submat' MatRestoreLocalSubMatrix .. to restore 'submat' After the above process, do I need MatAssemblyBegin/End again? I do not know why the penalty method is bad. Maybe because the bad conditioning? I just want to try out and compare with MatZeroRowsColumns (lift and remove) method. > > As Jed says, you really really do not want to do this. > > Matt > > > > > Matt > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From jedbrown at mcs.anl.gov Sat Feb 23 07:32:36 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 23 Feb 2013 07:32:36 -0600 Subject: [petsc-users] Scale the diagonal entries of a Mat In-Reply-To: References: Message-ID: You should really use MatSetValuesLocal with ADD_VALUES *before* assembling. On Feb 23, 2013 7:28 AM, "Hui Zhang" wrote: > > On Feb 23, 2013, at 1:37 PM, Matthew Knepley wrote: > > > On Sat, Feb 23, 2013 at 7:35 AM, Hui Zhang > wrote: > > > > On Feb 23, 2013, at 1:32 PM, Matthew Knepley wrote: > > > > > On Sat, Feb 23, 2013 at 7:29 AM, Hui Zhang > wrote: > > > I want to implement diagonal penalty method for enforcing the > Dirichlet boundary conditions. That is, the diagonal entries corresponding > to Dirichlet boundary are going to be scaled by a large number. > > > > > > What is the easiest way to do this? Thanks! > > > > > > > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatDiagonalScale.html > > > > But it seems not what I want. That routine scales all the entries of a > mat. I want to scale only the diagonal entries. > > > > Then use > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatDiagonalSet.html > > I want to do the following on an already assembled matrix (FINAL_ASSEMBLY). > > MatGetLocalSubMatrix .. to get 'submat', > MatGetDiagonal .. to get diagonal vector 'diag', > VecScale .. to scale 'diag' > MatDiagonalSet .. to set scaled 'diag' to 'submat' > MatRestoreLocalSubMatrix .. to restore 'submat' > > After the above process, do I need MatAssemblyBegin/End again? I do not > know why the penalty method is bad. Maybe because the bad conditioning? I > just want to try out and compare with MatZeroRowsColumns (lift and remove) > method. > > > > > As Jed says, you really really do not want to do this. > > > > Matt > > > > > > > > Matt > > > > > > -- > > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > > -- Norbert Wiener > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike.hui.zhang at hotmail.com Sat Feb 23 07:35:44 2013 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Sat, 23 Feb 2013 14:35:44 +0100 Subject: [petsc-users] Scale the diagonal entries of a Mat In-Reply-To: References: Message-ID: Thanks for advices! Follow you. On Feb 23, 2013, at 2:32 PM, Jed Brown wrote: > You should really use MatSetValuesLocal with ADD_VALUES *before* assembling. > > On Feb 23, 2013 7:28 AM, "Hui Zhang" wrote: > > On Feb 23, 2013, at 1:37 PM, Matthew Knepley wrote: > > > On Sat, Feb 23, 2013 at 7:35 AM, Hui Zhang wrote: > > > > On Feb 23, 2013, at 1:32 PM, Matthew Knepley wrote: > > > > > On Sat, Feb 23, 2013 at 7:29 AM, Hui Zhang wrote: > > > I want to implement diagonal penalty method for enforcing the Dirichlet boundary conditions. That is, the diagonal entries corresponding to Dirichlet boundary are going to be scaled by a large number. > > > > > > What is the easiest way to do this? Thanks! > > > > > > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatDiagonalScale.html > > > > But it seems not what I want. That routine scales all the entries of a mat. I want to scale only the diagonal entries. > > > > Then use http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatDiagonalSet.html > > I want to do the following on an already assembled matrix (FINAL_ASSEMBLY). > > MatGetLocalSubMatrix .. to get 'submat', > MatGetDiagonal .. to get diagonal vector 'diag', > VecScale .. to scale 'diag' > MatDiagonalSet .. to set scaled 'diag' to 'submat' > MatRestoreLocalSubMatrix .. to restore 'submat' > > After the above process, do I need MatAssemblyBegin/End again? I do not know why the penalty method is bad. Maybe because the bad conditioning? I just want to try out and compare with MatZeroRowsColumns (lift and remove) method. > > > > > As Jed says, you really really do not want to do this. > > > > Matt > > > > > > > > Matt > > > > > > -- > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > -- Norbert Wiener > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > From jedbrown at mcs.anl.gov Sat Feb 23 07:37:12 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 23 Feb 2013 07:37:12 -0600 Subject: [petsc-users] Scale the diagonal entries of a Mat In-Reply-To: References: Message-ID: (You should really not use penalties, but if you insist on doing it, you should implement it the way I describe. There is no point assembling before imposing boundary conditions.) On Feb 23, 2013 7:32 AM, "Jed Brown" wrote: > You should really use MatSetValuesLocal with ADD_VALUES *before* > assembling. > On Feb 23, 2013 7:28 AM, "Hui Zhang" wrote: > >> >> On Feb 23, 2013, at 1:37 PM, Matthew Knepley wrote: >> >> > On Sat, Feb 23, 2013 at 7:35 AM, Hui Zhang >> wrote: >> > >> > On Feb 23, 2013, at 1:32 PM, Matthew Knepley wrote: >> > >> > > On Sat, Feb 23, 2013 at 7:29 AM, Hui Zhang < >> mike.hui.zhang at hotmail.com> wrote: >> > > I want to implement diagonal penalty method for enforcing the >> Dirichlet boundary conditions. That is, the diagonal entries corresponding >> to Dirichlet boundary are going to be scaled by a large number. >> > > >> > > What is the easiest way to do this? Thanks! >> > > >> > > >> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatDiagonalScale.html >> > >> > But it seems not what I want. That routine scales all the entries of a >> mat. I want to scale only the diagonal entries. >> > >> > Then use >> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatDiagonalSet.html >> >> I want to do the following on an already assembled matrix >> (FINAL_ASSEMBLY). >> >> MatGetLocalSubMatrix .. to get 'submat', >> MatGetDiagonal .. to get diagonal vector 'diag', >> VecScale .. to scale 'diag' >> MatDiagonalSet .. to set scaled 'diag' to 'submat' >> MatRestoreLocalSubMatrix .. to restore 'submat' >> >> After the above process, do I need MatAssemblyBegin/End again? I do not >> know why the penalty method is bad. Maybe because the bad conditioning? I >> just want to try out and compare with MatZeroRowsColumns (lift and remove) >> method. >> >> > >> > As Jed says, you really really do not want to do this. >> > >> > Matt >> > >> > > >> > > Matt >> > > >> > > -- >> > > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > > -- Norbert Wiener >> > >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > -- Norbert Wiener >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike.hui.zhang at hotmail.com Sat Feb 23 07:43:47 2013 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Sat, 23 Feb 2013 14:43:47 +0100 Subject: [petsc-users] Scale the diagonal entries of a Mat In-Reply-To: References: Message-ID: Should MatZeroRowsColumnsLocalIS also be done before MatAssembly? I used MatZeroRows one year ago. At that time, petsc asked me the Mat must have already been assembled. On Feb 23, 2013, at 2:37 PM, Jed Brown wrote: > (You should really not use penalties, but if you insist on doing it, you should implement it the way I describe. There is no point assembling before imposing boundary conditions.) > > On Feb 23, 2013 7:32 AM, "Jed Brown" wrote: > You should really use MatSetValuesLocal with ADD_VALUES *before* assembling. > > On Feb 23, 2013 7:28 AM, "Hui Zhang" wrote: > > On Feb 23, 2013, at 1:37 PM, Matthew Knepley wrote: > > > On Sat, Feb 23, 2013 at 7:35 AM, Hui Zhang wrote: > > > > On Feb 23, 2013, at 1:32 PM, Matthew Knepley wrote: > > > > > On Sat, Feb 23, 2013 at 7:29 AM, Hui Zhang wrote: > > > I want to implement diagonal penalty method for enforcing the Dirichlet boundary conditions. That is, the diagonal entries corresponding to Dirichlet boundary are going to be scaled by a large number. > > > > > > What is the easiest way to do this? Thanks! > > > > > > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatDiagonalScale.html > > > > But it seems not what I want. That routine scales all the entries of a mat. I want to scale only the diagonal entries. > > > > Then use http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatDiagonalSet.html > > I want to do the following on an already assembled matrix (FINAL_ASSEMBLY). > > MatGetLocalSubMatrix .. to get 'submat', > MatGetDiagonal .. to get diagonal vector 'diag', > VecScale .. to scale 'diag' > MatDiagonalSet .. to set scaled 'diag' to 'submat' > MatRestoreLocalSubMatrix .. to restore 'submat' > > After the above process, do I need MatAssemblyBegin/End again? I do not know why the penalty method is bad. Maybe because the bad conditioning? I just want to try out and compare with MatZeroRowsColumns (lift and remove) method. > > > > > As Jed says, you really really do not want to do this. > > > > Matt > > > > > > > > Matt > > > > > > -- > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > -- Norbert Wiener > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > From mike.hui.zhang at hotmail.com Sat Feb 23 07:54:29 2013 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Sat, 23 Feb 2013 14:54:29 +0100 Subject: [petsc-users] Scale the diagonal entries of a Mat In-Reply-To: References: Message-ID: Sorry, I found the answer from the source code of Petsc. All the MatZero things are not for unassembled matrix. On Feb 23, 2013, at 2:43 PM, Hui Zhang wrote: > Should MatZeroRowsColumnsLocalIS also be done before MatAssembly? > > I used MatZeroRows one year ago. At that time, petsc asked me the Mat must have already been assembled. > > On Feb 23, 2013, at 2:37 PM, Jed Brown wrote: > >> (You should really not use penalties, but if you insist on doing it, you should implement it the way I describe. There is no point assembling before imposing boundary conditions.) >> >> On Feb 23, 2013 7:32 AM, "Jed Brown" wrote: >> You should really use MatSetValuesLocal with ADD_VALUES *before* assembling. >> >> On Feb 23, 2013 7:28 AM, "Hui Zhang" wrote: >> >> On Feb 23, 2013, at 1:37 PM, Matthew Knepley wrote: >> >>> On Sat, Feb 23, 2013 at 7:35 AM, Hui Zhang wrote: >>> >>> On Feb 23, 2013, at 1:32 PM, Matthew Knepley wrote: >>> >>>> On Sat, Feb 23, 2013 at 7:29 AM, Hui Zhang wrote: >>>> I want to implement diagonal penalty method for enforcing the Dirichlet boundary conditions. That is, the diagonal entries corresponding to Dirichlet boundary are going to be scaled by a large number. >>>> >>>> What is the easiest way to do this? Thanks! >>>> >>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatDiagonalScale.html >>> >>> But it seems not what I want. That routine scales all the entries of a mat. I want to scale only the diagonal entries. >>> >>> Then use http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatDiagonalSet.html >> >> I want to do the following on an already assembled matrix (FINAL_ASSEMBLY). >> >> MatGetLocalSubMatrix .. to get 'submat', >> MatGetDiagonal .. to get diagonal vector 'diag', >> VecScale .. to scale 'diag' >> MatDiagonalSet .. to set scaled 'diag' to 'submat' >> MatRestoreLocalSubMatrix .. to restore 'submat' >> >> After the above process, do I need MatAssemblyBegin/End again? I do not know why the penalty method is bad. Maybe because the bad conditioning? I just want to try out and compare with MatZeroRowsColumns (lift and remove) method. >> >>> >>> As Jed says, you really really do not want to do this. >>> >>> Matt >>> >>>> >>>> Matt >>>> >>>> -- >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>> -- Norbert Wiener >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >> > > From paresh_murthy at yahoo.com Sat Feb 23 02:56:33 2013 From: paresh_murthy at yahoo.com (paresh murthy) Date: Sat, 23 Feb 2013 08:56:33 +0000 (GMT) Subject: [petsc-users] [mumps-users] MUMPS error In-Reply-To: References: Message-ID: <1361609793.52203.YahooMailNeo@web28904.mail.ir2.yahoo.com> Hello Mike, I would guess, as perhaps others might as well, that it is trying to solve a million x million dense matrix. Are you sure you have removed the zeroes and given it a sparse matrix ? Paresh ________________________________ From: Hui Zhang To: mumps-users at listes.ens-lyon.fr; PETSc users list Cc: Hui Zhang Sent: Saturday, February 23, 2013 9:51 AM Subject: [mumps-users] MUMPS error Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 70397575 megabytes The required memory is too large. Is that normal? I'm solving a Helmholtz equation on 100^3 mesh partitioned to 4^3 processors. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrinal at umich.edu Sat Feb 23 10:27:18 2013 From: mrinal at umich.edu (Mrinal Iyer) Date: Sat, 23 Feb 2013 11:27:18 -0500 Subject: [petsc-users] [mumps-users] MUMPS error In-Reply-To: References: <1361609793.52203.YahooMailNeo@web28904.mail.ir2.yahoo.com> Message-ID: Hi, It seems to me that you have misunderstood the usage of Petsc's MatXSetPreallocation. With a 27 point stencil, the "total" number of non-zeros in a row must be 27. You need to split this up into those interactions that are local and those that are off-proc. Setting off-proc nnz to be 27 is way too high. You are running out of memory. Mrinal On Sat, Feb 23, 2013 at 4:09 AM, Hui Zhang wrote: > No, it is a 27-stencil discretization. I use petsc MatXSetPreallocation > with much larger estimates for off-diagonal (i.e. on other processors) > zeros. Basically, I say 27 non-zeros per row are on other processors. > Does this cause the MUMPS error? Thanks a lot! > > On Feb 23, 2013, at 9:56 AM, paresh murthy wrote: > > > Hello Mike, > > > > I would guess, as perhaps others might as well, that it is trying to > solve a million x million dense matrix. > > Are you sure you have removed the zeroes and given it a sparse matrix ? > > > > Paresh > > > > > > From: Hui Zhang > > To: mumps-users at listes.ens-lyon.fr; PETSc users list < > petsc-users at mcs.anl.gov> > > Cc: Hui Zhang > > Sent: Saturday, February 23, 2013 9:51 AM > > Subject: [mumps-users] MUMPS error > > > > Error reported by MUMPS in numerical factorization phase: Cannot > allocate required memory 70397575 megabytes > > > > The required memory is too large. Is that normal? I'm solving a > Helmholtz equation on 100^3 mesh partitioned to 4^3 processors. > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mihai at vt.edu Mon Feb 25 03:28:58 2013 From: mihai at vt.edu (Mihai Alexe) Date: Mon, 25 Feb 2013 10:28:58 +0100 Subject: [petsc-users] LSQR with Jacobi preconditioning Message-ID: Hello, I'm solving a small least-squares problem with LSQR. I want to set up Jacobi preconditioning, so I do something like the following: * MatCreateMPIAIJWithSplitArrays( PETSC_COMM_WORLD, *locrow, *loccol, nrow,* * *ncol, onrowidx, oncolidx,* * (PetscScalar*) onvals, offrowidx, offcolidx,* * (PetscScalar*) values, &A_mat );* */* Create Linear Solver Context (and set options) */* * KSPCreate( PETSC_COMM_WORLD, &solksp );* * KSPSetType( solksp, "lsqr" );* * * * MatCreateNormal(A_mat, &P_mat);* * MatSetUp(P_mat); * * KSPSetOperators( solksp, A_mat, P_mat, DIFFERENT_NONZERO_PATTERN );* * * * KSPGetPC( solksp, &pc);* * PCSetType( pc, PCJACOBI );* * * * KSPSetFromOptions( solksp );* * KSPSolve( solksp, b_vec, x_vec );* However, PETSc detects zeros on the diagonal of P_mat: [0] PCSetUp_Jacobi(): Zero detected in diagonal of matrix, using 1 at those locations I then printed out the indices at which the zeros are detected. It turns out that PETSC believes that all the diagonal elements are zero. There is no zero column in my A_mat though... I confirmed this by examining both A_mat and A_mat^T * A_mat in Matlab. What am I doing wrong? kind regards, Mihai -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Feb 25 05:32:28 2013 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 25 Feb 2013 06:32:28 -0500 Subject: [petsc-users] LSQR with Jacobi preconditioning In-Reply-To: References: Message-ID: On Mon, Feb 25, 2013 at 4:28 AM, Mihai Alexe wrote: > Hello, > > I'm solving a small least-squares problem with LSQR. I want to set up > Jacobi preconditioning, so I do something like the following: > Comments on the code below: Always check all the return codes with CHKERRQ() > * MatCreateMPIAIJWithSplitArrays( PETSC_COMM_WORLD, *locrow, *loccol, > nrow,* > * *ncol, onrowidx, oncolidx,* > * (PetscScalar*) onvals, offrowidx, offcolidx,* > * (PetscScalar*) values, &A_mat );* > ** > Why would you do this instead of just using MatSetValues(). Its fragile, more complex, and cannot use other matrix types. > */* Create Linear Solver Context (and set options) */* > * KSPCreate( PETSC_COMM_WORLD, &solksp );* > * KSPSetType( solksp, "lsqr" );* > I would replace this with KSPSetFromOptions(solksp) so that you can change solvers from the command line using -ksp_type. > * MatCreateNormal(A_mat, &P_mat);* > This is just a shell matrix to apply A^T A. In order to use Jacobi, you can MatShellSetOperation(P_mat, MATOP_GET_DIAGONAL, myDiagFunc) where myDiagFunc() return the vector with a^T_i a_i for each slot. > * MatSetUp(P_mat); * > * KSPSetOperators( solksp, A_mat, P_mat, DIFFERENT_NONZERO_PATTERN );* > * > * > * KSPGetPC( solksp, &pc);* > * PCSetType( pc, PCJACOBI );* > Again, why not just use -pc_type jacobi. Matt > * KSPSetFromOptions( solksp );* > * KSPSolve( solksp, b_vec, x_vec );* > > However, PETSc detects zeros on the diagonal of P_mat: > > [0] PCSetUp_Jacobi(): Zero detected in diagonal of matrix, using 1 at > those locations > > I then printed out the indices at which the zeros are detected. It turns > out that PETSC believes that all the diagonal elements are zero. > There is no zero column in my A_mat though... I confirmed this by > examining both A_mat and A_mat^T * A_mat in Matlab. > > What am I doing wrong? > > kind regards, > Mihai > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mihai at vt.edu Mon Feb 25 06:41:13 2013 From: mihai at vt.edu (Mihai Alexe) Date: Mon, 25 Feb 2013 13:41:13 +0100 Subject: [petsc-users] LSQR with Jacobi preconditioning In-Reply-To: References: Message-ID: Matt, Thank you. I'll try to implement some of the changes you suggested. As for the preconditioner, I can think of two ways of going about it: a. Define my "own" Jacobi preconditioner as in ksp/ex15.c & use MatGetColumnNorms on A_mat to get the diagonal entries of A_mat^T * A_mat. I don't very much like this option. or b. Leave P_mat as defined before and implement MATOP_GET_DIAGONAL in PetscErrorCode myDiagFunc(Mat mat, Vec diag_elements) But how can I access to A_mat from inside myDiagFunc? Through the shell matrix context object? I couldn't find an example of how to do so. Which option do you think would work best? cheers, Mihai On Mon, Feb 25, 2013 at 12:32 PM, Matthew Knepley wrote: > On Mon, Feb 25, 2013 at 4:28 AM, Mihai Alexe wrote: > >> Hello, >> >> I'm solving a small least-squares problem with LSQR. I want to set up >> Jacobi preconditioning, so I do something like the following: >> > > Comments on the code below: > > Always check all the return codes with CHKERRQ() > > >> * MatCreateMPIAIJWithSplitArrays( PETSC_COMM_WORLD, *locrow, *loccol, >> nrow,* >> * *ncol, onrowidx, oncolidx,* >> * (PetscScalar*) onvals, offrowidx, offcolidx,* >> * (PetscScalar*) values, &A_mat );* >> ** >> > > Why would you do this instead of just using MatSetValues(). Its fragile, > more complex, and cannot use other matrix types. > > >> */* Create Linear Solver Context (and set options) */* >> * KSPCreate( PETSC_COMM_WORLD, &solksp );* >> * KSPSetType( solksp, "lsqr" );* >> > > I would replace this with KSPSetFromOptions(solksp) so that you can change > solvers from the command line using -ksp_type. > > >> * MatCreateNormal(A_mat, &P_mat);* >> > > This is just a shell matrix to apply A^T A. In order to use Jacobi, you can > > MatShellSetOperation(P_mat, MATOP_GET_DIAGONAL, myDiagFunc) > > where myDiagFunc() return the vector with a^T_i a_i for each slot. > > >> * MatSetUp(P_mat); * >> * KSPSetOperators( solksp, A_mat, P_mat, DIFFERENT_NONZERO_PATTERN );* >> * >> * >> * KSPGetPC( solksp, &pc);* >> * PCSetType( pc, PCJACOBI );* >> > > Again, why not just use -pc_type jacobi. > > Matt > > >> * KSPSetFromOptions( solksp );* >> * KSPSolve( solksp, b_vec, x_vec );* >> >> However, PETSc detects zeros on the diagonal of P_mat: >> >> [0] PCSetUp_Jacobi(): Zero detected in diagonal of matrix, using 1 at >> those locations >> >> I then printed out the indices at which the zeros are detected. It turns >> out that PETSC believes that all the diagonal elements are zero. >> There is no zero column in my A_mat though... I confirmed this by >> examining both A_mat and A_mat^T * A_mat in Matlab. >> >> What am I doing wrong? >> >> kind regards, >> Mihai >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Feb 25 08:03:26 2013 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 25 Feb 2013 09:03:26 -0500 Subject: [petsc-users] LSQR with Jacobi preconditioning In-Reply-To: References: Message-ID: On Mon, Feb 25, 2013 at 7:41 AM, Mihai Alexe wrote: > Matt, > > Thank you. I'll try to implement some of the changes you suggested. > > As for the preconditioner, I can think of two ways of going about it: > > a. Define my "own" Jacobi preconditioner as in ksp/ex15.c & use > MatGetColumnNorms on A_mat to get the diagonal entries of A_mat^T * A_mat. > I don't very much like this option. > > or > > b. Leave P_mat as defined before and implement MATOP_GET_DIAGONAL in > > PetscErrorCode myDiagFunc(Mat mat, Vec diag_elements) > > But how can I access to A_mat from inside myDiagFunc? Through the shell > matrix context object? I couldn't find an example of how to do so. > Yes, put it in the context. struct MyStruct { Mat A; }; struct myStruct ctx; ctx.A = A_mat; MatShellSetContext(P_mat, &ctx); Then use MatShellGetContext() in myDiagFunc. Matt > Which option do you think would work best? > > cheers, > Mihai > > > On Mon, Feb 25, 2013 at 12:32 PM, Matthew Knepley wrote: > >> On Mon, Feb 25, 2013 at 4:28 AM, Mihai Alexe wrote: >> >>> Hello, >>> >>> I'm solving a small least-squares problem with LSQR. I want to set up >>> Jacobi preconditioning, so I do something like the following: >>> >> >> Comments on the code below: >> >> Always check all the return codes with CHKERRQ() >> >> >>> * MatCreateMPIAIJWithSplitArrays( PETSC_COMM_WORLD, *locrow, *loccol, >>> nrow,* >>> * *ncol, onrowidx, oncolidx,* >>> * (PetscScalar*) onvals, offrowidx, offcolidx,* >>> * (PetscScalar*) values, &A_mat );* >>> ** >>> >> >> Why would you do this instead of just using MatSetValues(). Its fragile, >> more complex, and cannot use other matrix types. >> >> >>> */* Create Linear Solver Context (and set options) */* >>> * KSPCreate( PETSC_COMM_WORLD, &solksp );* >>> * KSPSetType( solksp, "lsqr" );* >>> >> >> I would replace this with KSPSetFromOptions(solksp) so that you can >> change solvers from the command line using -ksp_type. >> >> >>> * MatCreateNormal(A_mat, &P_mat);* >>> >> >> This is just a shell matrix to apply A^T A. In order to use Jacobi, you >> can >> >> MatShellSetOperation(P_mat, MATOP_GET_DIAGONAL, myDiagFunc) >> >> where myDiagFunc() return the vector with a^T_i a_i for each slot. >> >> >>> * MatSetUp(P_mat); * >>> * KSPSetOperators( solksp, A_mat, P_mat, DIFFERENT_NONZERO_PATTERN );* >>> * >>> * >>> * KSPGetPC( solksp, &pc);* >>> * PCSetType( pc, PCJACOBI );* >>> >> >> Again, why not just use -pc_type jacobi. >> >> Matt >> >> >>> * KSPSetFromOptions( solksp );* >>> * KSPSolve( solksp, b_vec, x_vec );* >>> >>> However, PETSc detects zeros on the diagonal of P_mat: >>> >>> [0] PCSetUp_Jacobi(): Zero detected in diagonal of matrix, using 1 at >>> those locations >>> >>> I then printed out the indices at which the zeros are detected. It turns >>> out that PETSC believes that all the diagonal elements are zero. >>> There is no zero column in my A_mat though... I confirmed this by >>> examining both A_mat and A_mat^T * A_mat in Matlab. >>> >>> What am I doing wrong? >>> >>> kind regards, >>> Mihai >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From slivkaje at gmail.com Mon Feb 25 18:02:47 2013 From: slivkaje at gmail.com (Jelena Slivka) Date: Mon, 25 Feb 2013 19:02:47 -0500 Subject: [petsc-users] Determinant of the matrix Message-ID: Hello, I am trying to find the determinant of the matrix. I have found this thread: http://lists.mcs.anl.gov/pipermail/petsc-users/2010-July/006716.html But, could you please explain how to access the data structure directly? So far I have tried the following: #include #include <../src/mat/impls/aij/seq/aij.h> ... // (Solving the system A*x = rhs MatSetOption(A, MAT_SPD, PETSC_TRUE); Mat F; MatGetFactor(A, MATSOLVERMUMPS, MAT_FACTOR_CHOLESKY, &F); MatFactorInfo factinfo; MatFactorInfoInitialize(&factinfo); IS perm, iscol; MatGetOrdering(A, MATORDERINGNATURAL, &perm, &iscol); MatCholeskyFactorSymbolic(F, A, perm, &factinfo); MatCholeskyFactorNumeric(F, A, &factinfo); ISDestroy(&iscol); ISDestroy(&perm); MatSolve(F, rhs, x); // Now I would like to access the diagonal part of the matrix F, can I do something like: Mat_SeqAIJ *aa = (Mat_SeqAIJ*)F->data; Do you maybe have an example I could look at? Grateful in advance, Jelena -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Feb 25 18:13:16 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 25 Feb 2013 18:13:16 -0600 Subject: [petsc-users] Determinant of the matrix In-Reply-To: References: Message-ID: Jelena, MUMPS has its own (complicated) data structure for storing the factored matrix, you cannot easily look inside it. However, fortunately MUMPS has a way of accessing the determinate of the matrix directly. Look at src/mat/impls/aij/mpi/mumps.c and search for the word determinant, you will find the command line argument to have it computed and where it will be printed out if you call MatView() on the factored matrix. You mush pass the PetscViewerFormat PETSC_VIEWER_ASCII_INFO to PetscViewerSetFormat() before viewing the matrix. Good luck, Barry On Feb 25, 2013, at 6:02 PM, Jelena Slivka wrote: > Hello, > I am trying to find the determinant of the matrix. I have found this thread: > > http://lists.mcs.anl.gov/pipermail/petsc-users/2010-July/006716.html > > But, could you please explain how to access the data structure directly? > So far I have tried the following: > > #include > #include <../src/mat/impls/aij/seq/aij.h> > ... > // (Solving the system A*x = rhs > MatSetOption(A, MAT_SPD, PETSC_TRUE); > Mat F; > MatGetFactor(A, MATSOLVERMUMPS, MAT_FACTOR_CHOLESKY, &F); > MatFactorInfo factinfo; > MatFactorInfoInitialize(&factinfo); > IS perm, iscol; > MatGetOrdering(A, MATORDERINGNATURAL, &perm, &iscol); > MatCholeskyFactorSymbolic(F, A, perm, &factinfo); > MatCholeskyFactorNumeric(F, A, &factinfo); > ISDestroy(&iscol); > ISDestroy(&perm); > > MatSolve(F, rhs, x); > > // Now I would like to access the diagonal part of the matrix F, can I do something like: > > Mat_SeqAIJ *aa = (Mat_SeqAIJ*)F->data; > > Do you maybe have an example I could look at? > > Grateful in advance, > Jelena From slivkaje at gmail.com Mon Feb 25 18:47:43 2013 From: slivkaje at gmail.com (Jelena Slivka) Date: Mon, 25 Feb 2013 19:47:43 -0500 Subject: [petsc-users] Determinant of the matrix In-Reply-To: References: Message-ID: Thank you very much, but I need to access this value in my program and not just print it, is there a way to do this? Also, I need to use Cholesky factorization, is there a way to avoid MUMPS so I could access the diagonal? On Mon, Feb 25, 2013 at 7:13 PM, Barry Smith wrote: > > Jelena, > > MUMPS has its own (complicated) data structure for storing the > factored matrix, you cannot easily look inside it. > > However, fortunately MUMPS has a way of accessing the determinate of > the matrix directly. Look at src/mat/impls/aij/mpi/mumps.c and search for > the word determinant, you will find the command line argument to have it > computed and where it will be printed out if you call MatView() on the > factored matrix. You mush pass the PetscViewerFormat > PETSC_VIEWER_ASCII_INFO to PetscViewerSetFormat() before viewing the matrix. > > Good luck, > > Barry > > On Feb 25, 2013, at 6:02 PM, Jelena Slivka wrote: > > > Hello, > > I am trying to find the determinant of the matrix. I have found this > thread: > > > > http://lists.mcs.anl.gov/pipermail/petsc-users/2010-July/006716.html > > > > But, could you please explain how to access the data structure directly? > > So far I have tried the following: > > > > #include > > #include <../src/mat/impls/aij/seq/aij.h> > > ... > > // (Solving the system A*x = rhs > > MatSetOption(A, MAT_SPD, PETSC_TRUE); > > Mat F; > > MatGetFactor(A, MATSOLVERMUMPS, MAT_FACTOR_CHOLESKY, &F); > > MatFactorInfo factinfo; > > MatFactorInfoInitialize(&factinfo); > > IS perm, iscol; > > MatGetOrdering(A, MATORDERINGNATURAL, &perm, &iscol); > > MatCholeskyFactorSymbolic(F, A, perm, &factinfo); > > MatCholeskyFactorNumeric(F, A, &factinfo); > > ISDestroy(&iscol); > > ISDestroy(&perm); > > > > MatSolve(F, rhs, x); > > > > // Now I would like to access the diagonal part of the matrix F, can I > do something like: > > > > Mat_SeqAIJ *aa = (Mat_SeqAIJ*)F->data; > > > > Do you maybe have an example I could look at? > > > > Grateful in advance, > > Jelena > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Mon Feb 25 19:55:33 2013 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Mon, 25 Feb 2013 19:55:33 -0600 Subject: [petsc-users] Determinant of the matrix In-Reply-To: References: Message-ID: Jelena : > Thank you very much, but I need to access this value in my program and not > just print it, is there a way to do this? Also, I need to use Cholesky > factorization, is there a way to avoid MUMPS so I could access the diagonal? Google search 'determinant, mumps' leads to an email that I sent to an user: http://lists.mcs.anl.gov/pipermail/petsc-users/2011-September/010225.html I added mumps options ICNTL(30 - 33) to petsc-3.2. With the latest petsc-3.2, you can get determinant with '-mat_mumps_icntl_33 1' the result can be found using '-ksp_view'. For example, petsc-3.2/src/ksp/ksp/examples/tutorials>./ex2 -pc_type lu -pc_factor_mat_solver_package mumps -mat_mumps_icntl_33 1 -ksp_view ... (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0.900768,0)*(2^99) Let us know if you need additional support - I guess you might want MatGetMumpsRINFOG()? Hong > > > On Mon, Feb 25, 2013 at 7:13 PM, Barry Smith wrote: >> >> >> Jelena, >> >> MUMPS has its own (complicated) data structure for storing the >> factored matrix, you cannot easily look inside it. >> >> However, fortunately MUMPS has a way of accessing the determinate of >> the matrix directly. Look at src/mat/impls/aij/mpi/mumps.c and search for >> the word determinant, you will find the command line argument to have it >> computed and where it will be printed out if you call MatView() on the >> factored matrix. You mush pass the PetscViewerFormat PETSC_VIEWER_ASCII_INFO >> to PetscViewerSetFormat() before viewing the matrix. >> >> Good luck, >> >> Barry >> >> On Feb 25, 2013, at 6:02 PM, Jelena Slivka wrote: >> >> > Hello, >> > I am trying to find the determinant of the matrix. I have found this >> > thread: >> > >> > http://lists.mcs.anl.gov/pipermail/petsc-users/2010-July/006716.html >> > >> > But, could you please explain how to access the data structure directly? >> > So far I have tried the following: >> > >> > #include >> > #include <../src/mat/impls/aij/seq/aij.h> >> > ... >> > // (Solving the system A*x = rhs >> > MatSetOption(A, MAT_SPD, PETSC_TRUE); >> > Mat F; >> > MatGetFactor(A, MATSOLVERMUMPS, MAT_FACTOR_CHOLESKY, &F); >> > MatFactorInfo factinfo; >> > MatFactorInfoInitialize(&factinfo); >> > IS perm, iscol; >> > MatGetOrdering(A, MATORDERINGNATURAL, &perm, &iscol); >> > MatCholeskyFactorSymbolic(F, A, perm, &factinfo); >> > MatCholeskyFactorNumeric(F, A, &factinfo); >> > ISDestroy(&iscol); >> > ISDestroy(&perm); >> > >> > MatSolve(F, rhs, x); >> > >> > // Now I would like to access the diagonal part of the matrix F, can I >> > do something like: >> > >> > Mat_SeqAIJ *aa = (Mat_SeqAIJ*)F->data; >> > >> > Do you maybe have an example I could look at? >> > >> > Grateful in advance, >> > Jelena >> > From bsmith at mcs.anl.gov Mon Feb 25 20:45:06 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 25 Feb 2013 20:45:06 -0600 Subject: [petsc-users] Determinant of the matrix In-Reply-To: References: Message-ID: <14035F55-6D14-43C5-AAEE-EC8FD099F832@mcs.anl.gov> On Feb 25, 2013, at 6:47 PM, Jelena Slivka wrote: > Thank you very much, but I need to access this value in my program and not just print it, is there a way to do this? You need to look at the code that prints out the determinant and call the same routine from your code directly. > Also, I need to use Cholesky factorization, is there a way to avoid MUMPS so I could access the diagonal? If the problem is small enough to run sequentially then you can just the MPIBAIJ format Cholesky and access the diagonal directly. Barry > > > On Mon, Feb 25, 2013 at 7:13 PM, Barry Smith wrote: > > Jelena, > > MUMPS has its own (complicated) data structure for storing the factored matrix, you cannot easily look inside it. > > However, fortunately MUMPS has a way of accessing the determinate of the matrix directly. Look at src/mat/impls/aij/mpi/mumps.c and search for the word determinant, you will find the command line argument to have it computed and where it will be printed out if you call MatView() on the factored matrix. You mush pass the PetscViewerFormat PETSC_VIEWER_ASCII_INFO to PetscViewerSetFormat() before viewing the matrix. > > Good luck, > > Barry > > On Feb 25, 2013, at 6:02 PM, Jelena Slivka wrote: > > > Hello, > > I am trying to find the determinant of the matrix. I have found this thread: > > > > http://lists.mcs.anl.gov/pipermail/petsc-users/2010-July/006716.html > > > > But, could you please explain how to access the data structure directly? > > So far I have tried the following: > > > > #include > > #include <../src/mat/impls/aij/seq/aij.h> > > ... > > // (Solving the system A*x = rhs > > MatSetOption(A, MAT_SPD, PETSC_TRUE); > > Mat F; > > MatGetFactor(A, MATSOLVERMUMPS, MAT_FACTOR_CHOLESKY, &F); > > MatFactorInfo factinfo; > > MatFactorInfoInitialize(&factinfo); > > IS perm, iscol; > > MatGetOrdering(A, MATORDERINGNATURAL, &perm, &iscol); > > MatCholeskyFactorSymbolic(F, A, perm, &factinfo); > > MatCholeskyFactorNumeric(F, A, &factinfo); > > ISDestroy(&iscol); > > ISDestroy(&perm); > > > > MatSolve(F, rhs, x); > > > > // Now I would like to access the diagonal part of the matrix F, can I do something like: > > > > Mat_SeqAIJ *aa = (Mat_SeqAIJ*)F->data; > > > > Do you maybe have an example I could look at? > > > > Grateful in advance, > > Jelena > > From zhenglun.wei at gmail.com Tue Feb 26 16:49:47 2013 From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei) Date: Tue, 26 Feb 2013 16:49:47 -0600 Subject: [petsc-users] Non-uniform Cartisian Grid In-Reply-To: References: <509B01D6.9030501@gmail.com> <818369A9-2984-4133-A62F-E5DCEE283EA1@mcs.anl.gov> <5127DBB8.6040704@gmail.com> Message-ID: <512D3C0B.3010107@gmail.com> Dear folks, I have one question for the ex49. It do show a way to 'get' the 'coord' by the 'DMDAVecGetArray'. Should I just modified the coordinate on the 'coord' in order to obtain a non-uniform grid? and then, use DMDAVecRestoreArray to link the modified 'coord' back to 'cda' so that the 'coord' will be updated to the mesh I want? thanks, Alan On 2/22/2013 3:20 PM, Matthew Knepley wrote: > On Fri, Feb 22, 2013 at 3:57 PM, Zhenglun (Alan) Wei > > wrote: > > Dear all, > I hope you're having a nice day. > I wonder if anyone could point out a example for me about the > DMDASetCoordinates(). I found this > (http://www.mcs.anl.gov/petsc/petsc-current/src/dm/impls/da/dacorn.c.html#DMDASetCoordinates) > online; however it does not really demonstrate how to create the > 'coordinate vector - c'. > Also, I found that in the petsc sub-folder: > /petsc-dev/src/dm/impls/patch has a patch.c, which I suppose that it > does a local refinement for mesh. Does this a mature model that we can > use or just a testing code? > > > Look at KSP ex49. In the routine DMDACoordViewGnuplot2d() it reads out > the coordinates. > You could just as easily set them in this fashion. > > Matt > > thanks, > Alan > > On 11/7/2012 6:55 PM, Barry Smith wrote: > > On Nov 7, 2012, at 6:50 PM, Alan > wrote: > > > >> Dear folks, > >> I hope you're having a nice day. > >> I'm testing the Poisson solver in > /src/ksp/ksp/examples/tutorials/ex45. > >> It uses the 'DMDACreate3d' to generate a Cartisian structured > grid and > >> solve it with uniform grid size. However, I attempt to modify it to > >> obtain solution with a Cartsian structured grid and non-uniform > grid (It > >> is not unstructured local refined mesh). Should I still use the > >> 'DMDACreate3d' and implement different grid size in > 'ComputeMatrix' and > >> 'ComputeRHS'? In other words, does 'DMDACreate3d' itself > include any > >> mesh size information? > > You can use DMDASetCoordinates() and DMDAGetCoordinates() > (called DMSetCoordinates() and DMGetCoordinates() in petsc-dev) to > keep track of the coordinates but you are responsible for > generating the correctly scaled entries in the matrix etc. > > > > Barry > > > >> thanks, > >> Alan > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Feb 26 20:46:04 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 26 Feb 2013 20:46:04 -0600 Subject: [petsc-users] Non-uniform Cartisian Grid In-Reply-To: <512D3C0B.3010107@gmail.com> References: <509B01D6.9030501@gmail.com> <818369A9-2984-4133-A62F-E5DCEE283EA1@mcs.anl.gov> <5127DBB8.6040704@gmail.com> <512D3C0B.3010107@gmail.com> Message-ID: <5D146EFF-9F0A-4BAD-B3AC-F4D0C4E3F012@mcs.anl.gov> On Feb 26, 2013, at 4:49 PM, "Zhenglun (Alan) Wei" wrote: > Dear folks, > I have one question for the ex49. > It do show a way to 'get' the 'coord' by the 'DMDAVecGetArray'. Should I just modified the coordinate on the 'coord' in order to obtain a non-uniform grid? and then, use DMDAVecRestoreArray to link the modified 'coord' back to 'cda' so that the 'coord' will be updated to the mesh I want? Yes, you can do this. > > thanks, > Alan > > On 2/22/2013 3:20 PM, Matthew Knepley wrote: >> On Fri, Feb 22, 2013 at 3:57 PM, Zhenglun (Alan) Wei wrote: >> Dear all, >> I hope you're having a nice day. >> I wonder if anyone could point out a example for me about the >> DMDASetCoordinates(). I found this >> (http://www.mcs.anl.gov/petsc/petsc-current/src/dm/impls/da/dacorn.c.html#DMDASetCoordinates) >> online; however it does not really demonstrate how to create the >> 'coordinate vector - c'. >> Also, I found that in the petsc sub-folder: >> /petsc-dev/src/dm/impls/patch has a patch.c, which I suppose that it >> does a local refinement for mesh. Does this a mature model that we can >> use or just a testing code? >> >> Look at KSP ex49. In the routine DMDACoordViewGnuplot2d() it reads out the coordinates. >> You could just as easily set them in this fashion. >> >> Matt >> >> thanks, >> Alan >> >> On 11/7/2012 6:55 PM, Barry Smith wrote: >> > On Nov 7, 2012, at 6:50 PM, Alan wrote: >> > >> >> Dear folks, >> >> I hope you're having a nice day. >> >> I'm testing the Poisson solver in /src/ksp/ksp/examples/tutorials/ex45. >> >> It uses the 'DMDACreate3d' to generate a Cartisian structured grid and >> >> solve it with uniform grid size. However, I attempt to modify it to >> >> obtain solution with a Cartsian structured grid and non-uniform grid (It >> >> is not unstructured local refined mesh). Should I still use the >> >> 'DMDACreate3d' and implement different grid size in 'ComputeMatrix' and >> >> 'ComputeRHS'? In other words, does 'DMDACreate3d' itself include any >> >> mesh size information? >> > You can use DMDASetCoordinates() and DMDAGetCoordinates() (called DMSetCoordinates() and DMGetCoordinates() in petsc-dev) to keep track of the coordinates but you are responsible for generating the correctly scaled entries in the matrix etc. >> > >> > Barry >> > >> >> thanks, >> >> Alan >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > From james.j.ramsey14.civ at mail.mil Wed Feb 27 10:02:28 2013 From: james.j.ramsey14.civ at mail.mil (Ramsey, James J CIV (US)) Date: Wed, 27 Feb 2013 16:02:28 +0000 Subject: [petsc-users] Problems exceeding 192 million unknowns in FEM code Message-ID: I apologize in advance for any naivete. I'm new to running largeish-scale FEM problems. I have written a code intended to find the elastic properties of the unit cell of a microstructure rendered as a finite element mesh. [FWIW, basically I'm trying to solve equation (19) from Composites: Part A 32 (2001) 1291-1301.] The method of partitioning is kind of crude. The format of the file that the code reads in to determine element connectivity is the same as the format used by mpmetis. I run mpmetis on the element connectivity file, which produces two files indicating the partitioning of the nodes and elements. Each process then incrementally reads in the files used to define the mesh, using the output files from mpmetis to determine which nodes and elements to store in memory. (Yes, I've become aware that there are problems with multiple processes reading the same file, especially when the file is large.) For meshes with 50 x 50 x 50 or 100 x 100 x100 elements, the code seems to work reasonably well. The 100x100x100 mesh has run on a single 8-core node with 20 GB of RAM, for about 125,000 elements per core. If I try a mesh with 400 x 400 x 400 elements, I start running into problems. On one cluster (the same cluster on which I ran the 100x100x100 mesh), the 400x400x400 mesh wouldn't finish its run on 512 cores, which seems odd to me since (1) the number of elements per core is about the same as the case where the 100x100x100 mesh ran on 8 nodes and (2) an earlier version of the code using Epetra from Trilinos did work on that many cores. This might just be an issue with me preallocating too many non-zeros and running out of memory, though I'm not sure why that wouldn't have been a problem for the 100x100x100 run. On 512 cores, the code dies as it loops over the local elements to assemble its part of the global stiffness matrix. On 608 cores, the code dies differently. It finishes looping over the elements, but dies with "RETRY EXCEEDED" errors from OpenMPI. For 800 and 1024 cores, the code appears to work again. FYI, this cluster has 4x DDR Infiniband interconnects. (I don't know what "4x DDR" means, but maybe someone else does.) On a different--and newer--cluster, I get no joy with the 400x400x400 mesh at all. This cluster has 16 cores and 64 GB of RAM per node, and FDR-10 Infiniband interconnects. For 512 and 608 cores, the code seems to die as it loops over the elements, while for 704, 800, and 912 cores, the code finishes its calls to MatAssemblyBegin(), but during the calls to MatAssemblyEnd(), I get thousands of warning messages from OpenMPI, saying "rdma_recv.c:578 MXM WARN Many RX drops detected. Application performance may be affected". On this cluster, the Trilinos version of the code worked. even at 512 cores. For a 500x500x500 mesh, I have no luck on either cluster with PETSc, and only one cluster (the one with 16 cores per node) seems to work with the Trilinos version of the code. (It was actually the failures with the 500x500x500 mesh that led me to rewrite the relevant parts of the code using PETSc. For the cluster with 8 cores per node, running a 500x500x500 mesh on 1024 cores, the code usually dies during the calls to MatAssemblyEnd(), spewing out "RETRY EXCEEDED" errors from OpenMPI. I have done one trial with 1504 cores on the 8-core/node cluster, but it seems to have died before the code even starts. On the other cluster, I've tried cases with 1024 and 1504 cores, and during the calls to MatAssemblyEnd(), I get thousands of warning messages from OpenMPI, saying "rdma_recv.c:578 MXM WARN Many RX drops detected. Application performance may be affected." (Interestingly enough, in the output from the Trilinos version of my code, running on 1024 cores, I get one warning from OpenMPI about the RX drops, but the code appears to have finished successfully and gotten reasonably results.) I'm trying to make sense of what barriers I'm hitting here. If the main problem is, say, that the MatAssemblyXXX() calls entail a lot of communication, then why would increasing the number of cores solve the problem for the 400x400x400 case? And why does increasing the number of cores not seem to help for the other cluster? Am I doing something naive here? (Or maybe the question should be what naive things am I doing here.) From bsmith at mcs.anl.gov Wed Feb 27 10:53:31 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 27 Feb 2013 10:53:31 -0600 Subject: [petsc-users] Problems exceeding 192 million unknowns in FEM code In-Reply-To: References: Message-ID: <6B335D8D-1FD5-4FF1-A271-F23B7AE177E0@mcs.anl.gov> This sounds like an OpenMPI issue with the matrix element communication and assembly process. In this process PETSc "stashes" any values set on the "wrong" process, then at MatAssemblyBegin() each process computes how much data it is sending to each other process, tells each other process how much data to expect (the receivers then post receives), then actually sends the data. We have used this code over many years on many systems so it is likely to be relatively bug free. My feeling is that the OpenMPI hardware/software combination is getting overwhelmed with all the messages that need to be sent around. Since you are handling this communication completely differently with Trilinos it doesn't have this problem. Since we can't change the hardware the first thing to check is how much (what percentage) of matrix entries need to be stashed and moved between processes. Run the 100 by 100 by 100 on 8 cores with the -info option and send the resulting output (it may be a lot) to petsc-maint at mcs.anl.gov (not petsc-users) and we'll show you how to determine how much data is being moved around. Perhaps you could also send the output from the successful 800 core run (with -info). Barry On Feb 27, 2013, at 10:02 AM, "Ramsey, James J CIV (US)" wrote: > I apologize in advance for any naivete. I'm new to running largeish-scale FEM problems. > > I have written a code intended to find the elastic properties of the unit cell of a microstructure rendered as a finite element mesh. [FWIW, basically I'm trying to solve equation (19) from Composites: Part A 32 (2001) 1291-1301.] The method of partitioning is kind of crude. The format of the file that the code reads in to determine element connectivity is the same as the format used by mpmetis. I run mpmetis on the element connectivity file, which produces two files indicating the partitioning of the nodes and elements. Each process then incrementally reads in the files used to define the mesh, using the output files from mpmetis to determine which nodes and elements to store in memory. (Yes, I've become aware that there are problems with multiple processes reading the same file, especially when the file is large.) For meshes with 50 x 50 x 50 or 100 x 100 x100 elements, the code seems to work reasonably well. The 100x100x100 mesh has run on a single 8-core node with 20 GB of RAM, for about 125,000 elements per core. If I try a mesh with 400 x 400 x 400 elements, I start running into problems. > > On one cluster (the same cluster on which I ran the 100x100x100 mesh), the 400x400x400 mesh wouldn't finish its run on 512 cores, which seems odd to me since (1) the number of elements per core is about the same as the case where the 100x100x100 mesh ran on 8 nodes and (2) an earlier version of the code using Epetra from Trilinos did work on that many cores. This might just be an issue with me preallocating too many non-zeros and running out of memory, though I'm not sure why that wouldn't have been a problem for the 100x100x100 run. On 512 cores, the code dies as it loops over the local elements to assemble its part of the global stiffness matrix. On 608 cores, the code dies differently. It finishes looping over the elements, but dies with "RETRY EXCEEDED" errors from OpenMPI. For 800 and 1024 cores, the code appears to work again. FYI, this cluster has 4x DDR Infiniband interconnects. (I don't know what "4x DDR" means, but maybe someone else does.) > > On a different--and newer--cluster, I get no joy with the 400x400x400 mesh at all. This cluster has 16 cores and 64 GB of RAM per node, and FDR-10 Infiniband interconnects. For 512 and 608 cores, the code seems to die as it loops over the elements, while for 704, 800, and 912 cores, the code finishes its calls to MatAssemblyBegin(), but during the calls to MatAssemblyEnd(), I get thousands of warning messages from OpenMPI, saying "rdma_recv.c:578 MXM WARN Many RX drops detected. Application performance may be affected". On this cluster, the Trilinos version of the code worked. even at 512 cores. > > For a 500x500x500 mesh, I have no luck on either cluster with PETSc, and only one cluster (the one with 16 cores per node) seems to work with the Trilinos version of the code. (It was actually the failures with the 500x500x500 mesh that led me to rewrite the relevant parts of the code using PETSc. For the cluster with 8 cores per node, running a 500x500x500 mesh on 1024 cores, the code usually dies during the calls to MatAssemblyEnd(), spewing out "RETRY EXCEEDED" errors from OpenMPI. I have done one trial with 1504 cores on the 8-core/node cluster, but it seems to have died before the code even starts. On the other cluster, I've tried cases with 1024 and 1504 cores, and during the calls to MatAssemblyEnd(), I get thousands of warning messages from OpenMPI, saying "rdma_recv.c:578 MXM WARN Many RX drops detected. Application performance may be affected." (Interestingly enough, in the output from the Trilinos version of my code, running on 1024 cores, I get one warning from OpenMPI about the RX drops, but the code appears to have finished successfully and gotten reasonably results.) > > I'm trying to make sense of what barriers I'm hitting here. If the main problem is, say, that the MatAssemblyXXX() calls entail a lot of communication, then why would increasing the number of cores solve the problem for the 400x400x400 case? And why does increasing the number of cores not seem to help for the other cluster? Am I doing something naive here? (Or maybe the question should be what naive things am I doing here.) From balay at mcs.anl.gov Wed Feb 27 11:07:54 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 27 Feb 2013 11:07:54 -0600 (CST) Subject: [petsc-users] Problems exceeding 192 million unknowns in FEM code In-Reply-To: <6B335D8D-1FD5-4FF1-A271-F23B7AE177E0@mcs.anl.gov> References: <6B335D8D-1FD5-4FF1-A271-F23B7AE177E0@mcs.anl.gov> Message-ID: There could some inefficiency if the stash allocated for the off-proc messages is insufficient [-info will show how many mallocs were needed, and how much stash was needed.] One can use the option -matstash_initial_size to specify a sufficient size [that can avoid these mallocs] Also you can try reducing the message sizes [assuming MPI behaves better with smaller messages]. This can be done by duing multiple flush assemblies before the finaly assembly [each flush assembly will pack and send of data destined for off processors] i.e: MatSetVaules() ... MatAssemblyBegin/End(MAT_FLUSH_ASSEMBLY) MatSetVaules() ... MatAssemblyBegin/End(MAT_FLUSH_ASSEMBLY) MatSetVaules() ... MatAssemblyBegin/End(MAT_FINAL_ASSEMBLY) [in this case a smaller stash is used anyway] Satish On Wed, 27 Feb 2013, Barry Smith wrote: > > This sounds like an OpenMPI issue with the matrix element communication and assembly process. In this process PETSc "stashes" any values set on the "wrong" process, then at MatAssemblyBegin() each process computes > how much data it is sending to each other process, tells each other process how much data to expect (the receivers then post receives), then actually sends the data. We have used this code over many years on many systems so it is likely to be relatively bug free. My feeling is that the OpenMPI hardware/software combination is getting overwhelmed with all the messages that need to be sent around. > > Since you are handling this communication completely differently with Trilinos it doesn't have this problem. > > Since we can't change the hardware the first thing to check is how much (what percentage) of matrix entries need to be stashed and moved between processes. Run the 100 by 100 by 100 on 8 cores with the -info option and send the resulting output (it may be a lot) to petsc-maint at mcs.anl.gov (not petsc-users) and we'll show you how to determine how much data is being moved around. Perhaps you could also send the output from the successful 800 core run (with -info). > > Barry > > > > On Feb 27, 2013, at 10:02 AM, "Ramsey, James J CIV (US)" wrote: > > > I apologize in advance for any naivete. I'm new to running largeish-scale FEM problems. > > > > I have written a code intended to find the elastic properties of the unit cell of a microstructure rendered as a finite element mesh. [FWIW, basically I'm trying to solve equation (19) from Composites: Part A 32 (2001) 1291-1301.] The method of partitioning is kind of crude. The format of the file that the code reads in to determine element connectivity is the same as the format used by mpmetis. I run mpmetis on the element connectivity file, which produces two files indicating the partitioning of the nodes and elements. Each process then incrementally reads in the files used to define the mesh, using the output files from mpmetis to determine which nodes and elements to store in memory. (Yes, I've become aware that there are problems with multiple processes reading the same file, especially when the file is large.) For meshes with 50 x 50 x 50 or 100 x 100 x100 elements, the code seems to work reasonably well. The 100x100x100 mesh has run on a single 8-core node with 20 GB of RAM, for about 125,000 elements per core. If I try a mesh with 400 x 400 x 400 elements, I start running into problems. > > > > On one cluster (the same cluster on which I ran the 100x100x100 mesh), the 400x400x400 mesh wouldn't finish its run on 512 cores, which seems odd to me since (1) the number of elements per core is about the same as the case where the 100x100x100 mesh ran on 8 nodes and (2) an earlier version of the code using Epetra from Trilinos did work on that many cores. This might just be an issue with me preallocating too many non-zeros and running out of memory, though I'm not sure why that wouldn't have been a problem for the 100x100x100 run. On 512 cores, the code dies as it loops over the local elements to assemble its part of the global stiffness matrix. On 608 cores, the code dies differently. It finishes looping over the elements, but dies with "RETRY EXCEEDED" errors from OpenMPI. For 800 and 1024 cores, the code appears to work again. FYI, this cluster has 4x DDR Infiniband interconnects. (I don't know what "4x DDR" means, but maybe someone else does.) > > > > On a different--and newer--cluster, I get no joy with the 400x400x400 mesh at all. This cluster has 16 cores and 64 GB of RAM per node, and FDR-10 Infiniband interconnects. For 512 and 608 cores, the code seems to die as it loops over the elements, while for 704, 800, and 912 cores, the code finishes its calls to MatAssemblyBegin(), but during the calls to MatAssemblyEnd(), I get thousands of warning messages from OpenMPI, saying "rdma_recv.c:578 MXM WARN Many RX drops detected. Application performance may be affected". On this cluster, the Trilinos version of the code worked. even at 512 cores. > > > > For a 500x500x500 mesh, I have no luck on either cluster with PETSc, and only one cluster (the one with 16 cores per node) seems to work with the Trilinos version of the code. (It was actually the failures with the 500x500x500 mesh that led me to rewrite the relevant parts of the code using PETSc. For the cluster with 8 cores per node, running a 500x500x500 mesh on 1024 cores, the code usually dies during the calls to MatAssemblyEnd(), spewing out "RETRY EXCEEDED" errors from OpenMPI. I have done one trial with 1504 cores on the 8-core/node cluster, but it seems to have died before the code even starts. On the other cluster, I've tried cases with 1024 and 1504 cores, and during the calls to MatAssemblyEnd(), I get thousands of warning messages from OpenMPI, saying "rdma_recv.c:578 MXM WARN Many RX drops detected. Application performance may be affected." (Interestingly enough, in the output from the Trilinos version of my code, running on 1024 cores, I get one warning f rom OpenMPI about the RX drops, but the code appears to have finished successfully and gotten reasonably results.) > > > > I'm trying to make sense of what barriers I'm hitting here. If the main problem is, say, that the MatAssemblyXXX() calls entail a lot of communication, then why would increasing the number of cores solve the problem for the 400x400x400 case? And why does increasing the number of cores not seem to help for the other cluster? Am I doing something naive here? (Or maybe the question should be what naive things am I doing here.) > > From james.j.ramsey14.civ at mail.mil Wed Feb 27 12:44:21 2013 From: james.j.ramsey14.civ at mail.mil (Ramsey, James J CIV (US)) Date: Wed, 27 Feb 2013 18:44:21 +0000 Subject: [petsc-users] Problems exceeding 192 million unknowns in FEM code In-Reply-To: <6B335D8D-1FD5-4FF1-A271-F23B7AE177E0@mcs.anl.gov> References: , <6B335D8D-1FD5-4FF1-A271-F23B7AE177E0@mcs.anl.gov> Message-ID: ________________________________________ From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Barry Smith [bsmith at mcs.anl.gov] Sent: Wednesday, February 27, 2013 11:53 AM To: PETSc users list Subject: Re: [petsc-users] Problems exceeding 192 million unknowns in FEM code This sounds like an OpenMPI issue with the matrix element communication and assembly process. In this process PETSc "stashes" any values set on the "wrong" process, then at MatAssemblyBegin() each process computes how much data it is sending to each other process, tells each other process how much data to expect (the receivers then post receives), then actually sends the data. We have used this code over many years on many systems so it is likely to be relatively bug free. My feeling is that the OpenMPI hardware/software combination is getting overwhelmed with all the messages that need to be sent around. Since you are handling this communication completely differently with Trilinos it doesn't have this problem. ________________________________________ When a job using the Trilinos code goes south, if often dies at about the same place as the code with PETSc often does, at its equivalent of the MatAssemblyBegin()/MatAssemblyEnd() call (i.e. GlobalAssemble()). I suspect that I may have a similar underlying problem in both the PETSc and Trilinos versions of my code, though the details in how the problem is expressed may differ. ________________________________________ Since we can't change the hardware the first thing to check is how much (what percentage) of matrix entries need to be stashed and moved between processes. Run the 100 by 100 by 100 on 8 cores with the -info option and send the resulting output (it may be a lot) to petsc-maint at mcs.anl.gov (not petsc-users) and we'll show you how to determine how much data is being moved around. Perhaps you could also send the output from the successful 800 core run (with -info). ________________________________________ Done. From sourceforge.petsc at user.fastmail.fm Wed Feb 27 13:45:10 2013 From: sourceforge.petsc at user.fastmail.fm (Hugo Gagnon) Date: Wed, 27 Feb 2013 14:45:10 -0500 Subject: [petsc-users] ILU(k) with MATSBAIJ Message-ID: <1361994310.17743.140661197814777.4519F17C@webmail.messagingengine.com> Hi, I'm trying to use the MAT*SBAIJ matrix types in conjunction with ILU(1): call KSPGetPC(Pksp,Ppc,Pierr) if (nProc > 1) then call PCSetType(Ppc,PCBJACOBI,Pierr) call KSPSetup(Pksp,Pierr) call PCBJacobiGetSubKSP(Ppc,dumi,dumi,Psubksp,Pierr) call KSPGetPC(Psubksp(1),Ppc,Pierr) end if call PCSetType(Ppc,PCILU,Pierr) call PCFactorSetLevels(Ppc,1,Pierr) call KSPSetFromOptions(Pksp,Pierr) which gives the following error (for the serial case): [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: No support for this operation for this object type! [0]PETSC ERROR: Factor type not supported! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 6, Mon Feb 11 12:26:34 CST 2013 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /Users/hugo/Documents/jetstream/jetstream_x86_64 on a arch-darw named user202-44.wireless.utoronto.ca by hugo Wed Feb 27 14:34:04 2013 [0]PETSC ERROR: Libraries linked from /Users/hugo/Documents/petsc-3.3-p6/arch-darwin-c-opt/lib [0]PETSC ERROR: Configure run at Mon Feb 18 15:08:07 2013 [0]PETSC ERROR: Configure options --with-debugging=0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatGetFactor_seqsbaij_petsc() line 1820 in src/mat/impls/sbaij/seq/sbaij.c [0]PETSC ERROR: MatGetFactor() line 3881 in src/mat/interface/matrix.c [0]PETSC ERROR: PCSetUp_ILU() line 214 in src/ksp/pc/impls/factor/ilu/ilu.c [0]PETSC ERROR: PCSetUp() line 832 in src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPSetUp() line 278 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: KSPSolve() line 402 in src/ksp/ksp/interface/itfunc.c I don't get this error when I use the normal MAT*BAIJ matrix types, and I can use whatever fill levels I want. In the user's guide on page 79 it says that MATSEQSBAIJ is supported for ILU. Am I missing something? Any recommendations? Thanks, -- Hugo Gagnon From bsmith at mcs.anl.gov Wed Feb 27 14:36:16 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 27 Feb 2013 14:36:16 -0600 Subject: [petsc-users] ILU(k) with MATSBAIJ In-Reply-To: <1361994310.17743.140661197814777.4519F17C@webmail.messagingengine.com> References: <1361994310.17743.140661197814777.4519F17C@webmail.messagingengine.com> Message-ID: <007ABDB6-9B5C-4516-B8AD-A2F532645827@mcs.anl.gov> SBAIJ stores only a symmetric half of the matrix. With SBAIJ you can only use Cholesky and incomplete Cholesky factorization, not ILU. Barry On Feb 27, 2013, at 1:45 PM, Hugo Gagnon wrote: > Hi, > > I'm trying to use the MAT*SBAIJ matrix types in conjunction with ILU(1): > > call KSPGetPC(Pksp,Ppc,Pierr) > if (nProc > 1) then > call PCSetType(Ppc,PCBJACOBI,Pierr) > call KSPSetup(Pksp,Pierr) > call PCBJacobiGetSubKSP(Ppc,dumi,dumi,Psubksp,Pierr) > call KSPGetPC(Psubksp(1),Ppc,Pierr) > end if > call PCSetType(Ppc,PCILU,Pierr) > call PCFactorSetLevels(Ppc,1,Pierr) > call KSPSetFromOptions(Pksp,Pierr) > > which gives the following error (for the serial case): > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: Factor type not supported! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 6, Mon Feb 11 > 12:26:34 CST 2013 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: /Users/hugo/Documents/jetstream/jetstream_x86_64 on a > arch-darw named user202-44.wireless.utoronto.ca by hugo Wed Feb 27 > 14:34:04 2013 > [0]PETSC ERROR: Libraries linked from > /Users/hugo/Documents/petsc-3.3-p6/arch-darwin-c-opt/lib > [0]PETSC ERROR: Configure run at Mon Feb 18 15:08:07 2013 > [0]PETSC ERROR: Configure options --with-debugging=0 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatGetFactor_seqsbaij_petsc() line 1820 in > src/mat/impls/sbaij/seq/sbaij.c > [0]PETSC ERROR: MatGetFactor() line 3881 in src/mat/interface/matrix.c > [0]PETSC ERROR: PCSetUp_ILU() line 214 in > src/ksp/pc/impls/factor/ilu/ilu.c > [0]PETSC ERROR: PCSetUp() line 832 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSetUp() line 278 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: KSPSolve() line 402 in src/ksp/ksp/interface/itfunc.c > > I don't get this error when I use the normal MAT*BAIJ matrix types, and > I can use whatever fill levels I want. > > In the user's guide on page 79 it says that MATSEQSBAIJ is supported for > ILU. > > Am I missing something? Any recommendations? > > Thanks, > -- > Hugo Gagnon From mail2amneet at gmail.com Wed Feb 27 18:09:28 2013 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Wed, 27 Feb 2013 18:09:28 -0600 Subject: [petsc-users] Extracting Vec from VecNest Message-ID: I have packaged few vectors into a single Vec. I want to extract components of this composite Vec at some time in the code. Doing things this way is giving me an error. static const int NO_COMPS = ...; Vec array_x[NO_COMPS] = {.....}; Vec x; VecCreateNest(PETSC_COMM_WORLD, NO_COMPS, PETSC_NULL, array_x, &x); for(int n = 0; n < NO_COMPS; ++n) { Vec sub_x; IS iset; ISCreateGeneral(PETSC_COMM_WORLD, 1, &n, PETSC_COPY_VALUES, &iset); VecGetSubVector(x, iset, &sub_x); //*ERROR: Index set not found in nested Vec!* ISDestroy(&iset); } What's the proper way of extracting components? -- Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From vijay.m at gmail.com Wed Feb 27 18:50:14 2013 From: vijay.m at gmail.com (Vijay S. Mahadevan) Date: Wed, 27 Feb 2013 18:50:14 -0600 Subject: [petsc-users] Extracting Vec from VecNest In-Reply-To: References: Message-ID: Always give the full error message and use error checking for PETSc functions. (CHKERRQ, CHKERRABORT) The VecGetSubVector routine expects the entire index set for the component vector you need i.e., size of isset should be equal to size of the vector. Instead you are asking for the reference to the first component of the sub-vector in VecNest. This not possible. > What's the proper way of extracting components? For getting a single sub-component, it might be appropriate to use VecNestGetSubVec but you are tying yourself too closely to the VecNest data type. For example: VecNestGetSubVec(x, n, &sub_x); Vijay On Wed, Feb 27, 2013 at 6:09 PM, Amneet Bhalla wrote: > I have packaged few vectors into a single Vec. I want to extract > components of this composite Vec at some time in the code. Doing things > this way is giving me an error. > > static const int NO_COMPS = ...; > Vec array_x[NO_COMPS] = {.....}; > Vec x; > VecCreateNest(PETSC_COMM_WORLD, NO_COMPS, PETSC_NULL, array_x, &x); > > for(int n = 0; n < NO_COMPS; ++n) > { > Vec sub_x; > IS iset; > ISCreateGeneral(PETSC_COMM_WORLD, 1, &n, PETSC_COPY_VALUES, &iset); > > VecGetSubVector(x, iset, &sub_x); > //ERROR: Index set not found in nested Vec! > > ISDestroy(&iset); > > } > > What's the proper way of extracting components? > > -- > Amneet > > > From mike.hui.zhang at hotmail.com Thu Feb 28 05:33:45 2013 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Thu, 28 Feb 2013 12:33:45 +0100 Subject: [petsc-users] csqrt Message-ID: My cluster has an old version of glibc 2.3.3, which can not give usual principal value of square root. For example, it gives square root of -i as the value with negative real part. I 'grep' in the source tree of petsc and found csqrt in many places. Is this dangerous? Thank you! From Wadud.Miah at awe.co.uk Thu Feb 28 05:37:26 2013 From: Wadud.Miah at awe.co.uk (Wadud.Miah at awe.co.uk) Date: Thu, 28 Feb 2013 11:37:26 +0000 Subject: [petsc-users] OSKI building block Message-ID: <201302281137.r1SBbbhF014242@msw1.awe.co.uk> Hi PETSc users, What is the status of PETSc OSKI (Optimised Sparse Kernel Interface) implementation? I have noticed that there is a PETSc-OSKI implementation on the OSKI download page, but nothing on the official PETSc web page. Regards, Wadud. ___________________________________________________ ____________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail2amneet at gmail.com Thu Feb 28 11:58:45 2013 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Thu, 28 Feb 2013 11:58:45 -0600 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: <1ACA5F14-5B26-43BB-ACFE-1B48F7CAA06F@cims.nyu.edu> References: <5126A9FA.3060203@cims.nyu.edu> <5126B1DB.90103@cims.nyu.edu> <5126B4F8.8060408@cims.nyu.edu> <5126B6DB.4020207@cims.nyu.edu> <5126BA30.9090607@cims.nyu.edu> <5126C9C6.80306@cims.nyu.edu> <1ACA5F14-5B26-43BB-ACFE-1B48F7CAA06F@cims.nyu.edu> Message-ID: Hi Jed, I have registered MatShell that operates on VecNest (which is composed of shell vectors) with KSP. When I am solving sytem of equations using this KSP, I am getting some errors. The sequence of events from gdb are: #0 0x0000000000000000 in ?? () #1 0x00000000009856fd in VecGetArrayRead (x=0x3422a30, a=0x7fff8b8c6620) at /home/amneet/SOFTWARES/LIBRARIESIBAMR/PETSc-3.3-Lite/include/petsc-private/vecimpl.h:350 #2 0x000000000098f57f in VecGetArray_Nest (X=0x34291d0, x=0x7fff8b8c6720) at /home/amneet/SOFTWARES/LIBRARIESIBAMR/PETSc-3.3-Lite/src/vec/vec/impls/nest/vecnest.c:719 #3 0x0000000000a3f3df in VecGetArrayRead (x=0x34291d0, a=0x7fff8b8c6720) at /home/amneet/SOFTWARES/LIBRARIESIBAMR/PETSc-3.3-Lite/include/petsc-private/vecimpl.h:350 #4 0x0000000000a51c83 in MatMult (mat=0x32c34f0, x=0x336a1c0, y=0x34291d0) at /home/amneet/SOFTWARES/LIBRARIESIBAMR/PETSc-3.3-Lite/src/mat/interface/matrix.c:2092 #5 0x00000000012b2a78 in KSPInitialResidual (ksp=0x334bcb0, vsoln=0x336a1c0, vt1=0x34291d0, vt2=0x343c150, vres=0x344f220, vb=0x337d810) at /home/amneet/SOFTWARES/LIBRARIESIBAMR/PETSc-3.3-Lite/src/ksp/ksp/interface/itres.c:54 #6 0x0000000000da2eaa in KSPSolve_GMRES (ksp=0x334bcb0) at /home/amneet/SOFTWARES/LIBRARIESIBAMR/PETSc-3.3-Lite/src/ksp/ksp/impls/gmres/gmres.c:230 #7 0x0000000000d1ad6b in KSPSolve (ksp=0x334bcb0, b=0x337d810, x=0x336a1c0) at /home/amneet/SOFTWARES/LIBRARIESIBAMR/PETSc-3.3-Lite/src/ksp/ksp/interface/itfunc.c:446 #8 0x00000000006b8a08 in IBTK::PETScBlockKrylovLinearSolver::solveSystem ( this=0x3249380, x=0x32f3e20, b=0x330c6d0) at PETScBlockKrylovLinearSolver.C:384 #9 0x00000000005777e3 in main (argc=6, argv=0x7fff8b8ca988) at main.C:274 In frame 4, the user defined matrix-vector multiplication is successful ( I printed out a statement). This would imply in the routine MatVecMult(Mat A, Vec x, Vec y), Vec y would be a duplicated vector. This in turn would mean that user-defined routine to duplicate vectors MatGetVecs(Mat A, Vec *left, Vec* right) should have been called. But this is not happening. It is in this routine, that I am setting *left->ops->getarray = PETSC_NULL; Any thoughts on what might be going on? Thanks, Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Feb 28 13:12:15 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 28 Feb 2013 13:12:15 -0600 Subject: [petsc-users] OSKI building block In-Reply-To: <201302281137.r1SBbbhF014242@msw1.awe.co.uk> References: <201302281137.r1SBbbhF014242@msw1.awe.co.uk> Message-ID: <842195E1-1AC7-44D3-BCA5-F585C0218145@mcs.anl.gov> Wadud, They never communicated to us anything regarding their implementation so nothing is contained in the PETSc source code with regard to this. I believe that OSKI was just a research project and they've moved on to other research topics without providing support for OSKI. Barry On Feb 28, 2013, at 5:37 AM, Wadud.Miah at awe.co.uk wrote: > Hi PETSc users, > > What is the status of PETSc OSKI (Optimised Sparse Kernel Interface) implementation? I have noticed that there is a PETSc-OSKI implementation on the OSKI download page, but nothing on the official PETSc web page. > > Regards, > Wadud. > > ___________________________________________________ ____________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR > From bsmith at mcs.anl.gov Thu Feb 28 13:52:24 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 28 Feb 2013 13:52:24 -0600 Subject: [petsc-users] csqrt In-Reply-To: References: Message-ID: On Feb 28, 2013, at 5:33 AM, Hui Zhang wrote: > My cluster has an old version of glibc 2.3.3, which can not give usual principal value of square root. For example, it gives square root of -i as the value with negative real part. > > I 'grep' in the source tree of petsc and found csqrt in many places. Is this dangerous? Thank you! If you are using the default PETSc builds where it works with real numbers then it will have no effect on. If you are using PETSc with complex numbers you might consider changing the cluster software. Barry From balay at mcs.anl.gov Thu Feb 28 14:04:44 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 28 Feb 2013 14:04:44 -0600 (CST) Subject: [petsc-users] csqrt In-Reply-To: References: Message-ID: On Thu, 28 Feb 2013, Barry Smith wrote: > > On Feb 28, 2013, at 5:33 AM, Hui Zhang wrote: > > > My cluster has an old version of glibc 2.3.3, which can not give usual principal value of square root. For example, it gives square root of -i as the value with negative real part. > > > > I 'grep' in the source tree of petsc and found csqrt in many places. Is this dangerous? Thank you! > > If you are using the default PETSc builds where it works with real numbers then it will have no effect on. > > If you are using PETSc with complex numbers you might consider changing the cluster software. presumably csqrt() is c99 - so c++ std:sqrt() is perhaps ok ok? [so one could use --with-clanguage=cxx] Satish From jedbrown at mcs.anl.gov Thu Feb 28 14:27:32 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 28 Feb 2013 15:27:32 -0500 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: <5126A9FA.3060203@cims.nyu.edu> <5126B1DB.90103@cims.nyu.edu> <5126B4F8.8060408@cims.nyu.edu> <5126B6DB.4020207@cims.nyu.edu> <5126BA30.9090607@cims.nyu.edu> <5126C9C6.80306@cims.nyu.edu> <1ACA5F14-5B26-43BB-ACFE-1B48F7CAA06F@cims.nyu.edu> Message-ID: On Thu, Feb 28, 2013 at 12:58 PM, Amneet Bhalla wrote: > > Hi Jed, > > I have registered MatShell that operates on VecNest (which is composed of > shell vectors) with KSP. > When I am solving sytem of equations using this KSP, I am getting some > errors. The sequence of > events from gdb are: > > #0 0x0000000000000000 in ?? () > > #1 0x00000000009856fd in VecGetArrayRead (x=0x3422a30, a=0x7fff8b8c6620) > at > /home/amneet/SOFTWARES/LIBRARIESIBAMR/PETSc-3.3-Lite/include/petsc-private/vecimpl.h:350 > > #2 0x000000000098f57f in VecGetArray_Nest (X=0x34291d0, x=0x7fff8b8c6720) > at > /home/amneet/SOFTWARES/LIBRARIESIBAMR/PETSc-3.3-Lite/src/vec/vec/impls/nest/vecnest.c:719 > > #3 0x0000000000a3f3df in VecGetArrayRead (x=0x34291d0, a=0x7fff8b8c6720) > at > /home/amneet/SOFTWARES/LIBRARIESIBAMR/PETSc-3.3-Lite/include/petsc-private/vecimpl.h:350 > This is calling VecValidValues() checks for vec->ops->getarray. Since VecGetArray_Nest() is implemented, it calls the routine. I'm inclined to add a vec->ops->validvalues() and _not_ try to hack it in terms of VecGetArray(). > #4 0x0000000000a51c83 in MatMult (mat=0x32c34f0, x=0x336a1c0, y=0x34291d0) > at > /home/amneet/SOFTWARES/LIBRARIESIBAMR/PETSc-3.3-Lite/src/mat/interface/matrix.c:2092 > > #5 0x00000000012b2a78 in KSPInitialResidual (ksp=0x334bcb0, > vsoln=0x336a1c0, > vt1=0x34291d0, vt2=0x343c150, vres=0x344f220, vb=0x337d810) > at > /home/amneet/SOFTWARES/LIBRARIESIBAMR/PETSc-3.3-Lite/src/ksp/ksp/interface/itres.c:54 > > #6 0x0000000000da2eaa in KSPSolve_GMRES (ksp=0x334bcb0) > at > /home/amneet/SOFTWARES/LIBRARIESIBAMR/PETSc-3.3-Lite/src/ksp/ksp/impls/gmres/gmres.c:230 > > #7 0x0000000000d1ad6b in KSPSolve (ksp=0x334bcb0, b=0x337d810, > x=0x336a1c0) > at > /home/amneet/SOFTWARES/LIBRARIESIBAMR/PETSc-3.3-Lite/src/ksp/ksp/interface/itfunc.c:446 > > #8 0x00000000006b8a08 in IBTK::PETScBlockKrylovLinearSolver::solveSystem ( > this=0x3249380, x=0x32f3e20, b=0x330c6d0) > at PETScBlockKrylovLinearSolver.C:384 > > #9 0x00000000005777e3 in main (argc=6, argv=0x7fff8b8ca988) at main.C:274 > > In frame 4, the user defined matrix-vector multiplication is successful ( > I printed out a statement). > This would imply in the routine MatVecMult(Mat A, Vec x, Vec y), Vec y > would be a duplicated > vector. This in turn would mean that user-defined routine to duplicate > vectors MatGetVecs(Mat A, Vec *left, Vec* right) > should have been called. But this is not happening. It is in this routine, > that I am setting > > *left->ops->getarray = PETSC_NULL; > > Any thoughts on what might be going on? > > Thanks, > Amneet > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail2amneet at gmail.com Thu Feb 28 14:39:09 2013 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Thu, 28 Feb 2013 14:39:09 -0600 Subject: [petsc-users] MATNEST with shell matrices In-Reply-To: References: <5126A9FA.3060203@cims.nyu.edu> <5126B1DB.90103@cims.nyu.edu> <5126B4F8.8060408@cims.nyu.edu> <5126B6DB.4020207@cims.nyu.edu> <5126BA30.9090607@cims.nyu.edu> <5126C9C6.80306@cims.nyu.edu> <1ACA5F14-5B26-43BB-ACFE-1B48F7CAA06F@cims.nyu.edu> Message-ID: I am all for doing things the legal way :-) Where should this routine be added and how should it look like for shell vectors? I had assumed that such changes should be done in user defined MatGetVecs routine. Clearly this is not happening. Can you comment on proper construction of duplicated Vecs which are needed by Krylov solver of PETSc? The way I am doing is : a) Create a wrapper Vec (x), which holds SAMRAI vector. b) Define a x->ops->duplicatevector for this wrapper vector. c) Create a VecNest (X) of these wrapper Vecs. d) Define MatGetVecs routine, which will duplicate X by the call VecDuplicate(X,left/right) On Thu, Feb 28, 2013 at 2:27 PM, Jed Brown wrote: > This is calling VecValidValues() checks for vec->ops->getarray. Since > VecGetArray_Nest() is implemented, it calls the routine. I'm inclined to > add a vec->ops->validvalues() and _not_ try to hack it in terms of > VecGetArray(). -- Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike.hui.zhang at hotmail.com Thu Feb 28 16:08:56 2013 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Thu, 28 Feb 2013 23:08:56 +0100 Subject: [petsc-users] csqrt In-Reply-To: References: Message-ID: On Feb 28, 2013, at 9:04 PM, Satish Balay wrote: > On Thu, 28 Feb 2013, Barry Smith wrote: > >> >> On Feb 28, 2013, at 5:33 AM, Hui Zhang wrote: >> >>> My cluster has an old version of glibc 2.3.3, which can not give usual principal value of square root. For example, it gives square root of -i as the value with negative real part. >>> >>> I 'grep' in the source tree of petsc and found csqrt in many places. Is this dangerous? Thank you! >> >> If you are using the default PETSc builds where it works with real numbers then it will have no effect on. >> >> If you are using PETSc with complex numbers you might consider changing the cluster software. > > presumably csqrt() is c99 - so c++ std:sqrt() is perhaps ok ok? [so one could use --with-clanguage=cxx] Unfortunately, this does not work. Thanks any way! > > Satish > From zhenglun.wei at gmail.com Thu Feb 28 18:19:19 2013 From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei) Date: Thu, 28 Feb 2013 18:19:19 -0600 Subject: [petsc-users] Non-uniform Cartisian Grid In-Reply-To: <5D146EFF-9F0A-4BAD-B3AC-F4D0C4E3F012@mcs.anl.gov> References: <509B01D6.9030501@gmail.com> <818369A9-2984-4133-A62F-E5DCEE283EA1@mcs.anl.gov> <5127DBB8.6040704@gmail.com> <512D3C0B.3010107@gmail.com> <5D146EFF-9F0A-4BAD-B3AC-F4D0C4E3F012@mcs.anl.gov> Message-ID: <512FF407.6020408@gmail.com> Dear folks, Here are several questions following up my previous ones: 1, I modified my code based on ex49.c. However, I found that if I don't call 'DMDASetUniformCoordinates' before calling 'DMDAVecGetArray', then an error comes up: [0]PETSC ERROR: Null argument, when expecting valid pointer! [0]PETSC ERROR: Null Object: Parameter # 2! The code do works if I add a call for 'DMDASetUniformCoordinates' in it. However, I feel it is a waste of calling 'DMDASetUniformCoordinates' since what follows this calling is to change the coordinate to a non-uniform gird. Does it have any way to bypass this call or you think it is fine to keep it here. 2, Also, if I do by this way, the outputted mesh looks weird, as shown in the figure. I was trying to double the dy value manually in the SetNonUniformGrid(). However, the only mesh changed is just on the z=0 and x=0 plane. Here I attached my code. Do I need to do anything else except DMDAVecRestoreArray() after I modified the coordinate information? thanks, Alan On 2/26/2013 8:46 PM, Barry Smith wrote: > On Feb 26, 2013, at 4:49 PM, "Zhenglun (Alan) Wei" wrote: > >> Dear folks, >> I have one question for the ex49. >> It do show a way to 'get' the 'coord' by the 'DMDAVecGetArray'. Should I just modified the coordinate on the 'coord' in order to obtain a non-uniform grid? and then, use DMDAVecRestoreArray to link the modified 'coord' back to 'cda' so that the 'coord' will be updated to the mesh I want? > Yes, you can do this. > > >> thanks, >> Alan >> >> On 2/22/2013 3:20 PM, Matthew Knepley wrote: >>> On Fri, Feb 22, 2013 at 3:57 PM, Zhenglun (Alan) Wei wrote: >>> Dear all, >>> I hope you're having a nice day. >>> I wonder if anyone could point out a example for me about the >>> DMDASetCoordinates(). I found this >>> (http://www.mcs.anl.gov/petsc/petsc-current/src/dm/impls/da/dacorn.c.html#DMDASetCoordinates) >>> online; however it does not really demonstrate how to create the >>> 'coordinate vector - c'. >>> Also, I found that in the petsc sub-folder: >>> /petsc-dev/src/dm/impls/patch has a patch.c, which I suppose that it >>> does a local refinement for mesh. Does this a mature model that we can >>> use or just a testing code? >>> >>> Look at KSP ex49. In the routine DMDACoordViewGnuplot2d() it reads out the coordinates. >>> You could just as easily set them in this fashion. >>> >>> Matt >>> >>> thanks, >>> Alan >>> >>> On 11/7/2012 6:55 PM, Barry Smith wrote: >>>> On Nov 7, 2012, at 6:50 PM, Alan wrote: >>>> >>>>> Dear folks, >>>>> I hope you're having a nice day. >>>>> I'm testing the Poisson solver in /src/ksp/ksp/examples/tutorials/ex45. >>>>> It uses the 'DMDACreate3d' to generate a Cartisian structured grid and >>>>> solve it with uniform grid size. However, I attempt to modify it to >>>>> obtain solution with a Cartsian structured grid and non-uniform grid (It >>>>> is not unstructured local refined mesh). Should I still use the >>>>> 'DMDACreate3d' and implement different grid size in 'ComputeMatrix' and >>>>> 'ComputeRHS'? In other words, does 'DMDACreate3d' itself include any >>>>> mesh size information? >>>> You can use DMDASetCoordinates() and DMDAGetCoordinates() (called DMSetCoordinates() and DMGetCoordinates() in petsc-dev) to keep track of the coordinates but you are responsible for generating the correctly scaled entries in the matrix etc. >>>> >>>> Barry >>>> >>>>> thanks, >>>>> Alan >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener -------------- next part -------------- A non-text attachment was scrubbed... Name: export.jpg Type: image/jpeg Size: 358401 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Test.zip Type: application/zip Size: 3713 bytes Desc: not available URL: From knepley at gmail.com Thu Feb 28 18:31:58 2013 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 28 Feb 2013 19:31:58 -0500 Subject: [petsc-users] Non-uniform Cartisian Grid In-Reply-To: <512FF407.6020408@gmail.com> References: <509B01D6.9030501@gmail.com> <818369A9-2984-4133-A62F-E5DCEE283EA1@mcs.anl.gov> <5127DBB8.6040704@gmail.com> <512D3C0B.3010107@gmail.com> <5D146EFF-9F0A-4BAD-B3AC-F4D0C4E3F012@mcs.anl.gov> <512FF407.6020408@gmail.com> Message-ID: On Thu, Feb 28, 2013 at 7:19 PM, Zhenglun (Alan) Wei wrote: > Dear folks, > Here are several questions following up my previous ones: > 1, I modified my code based on ex49.c. However, I found that if I don't > call 'DMDASetUniformCoordinates' before calling 'DMDAVecGetArray', then an > error comes up: > [0]PETSC ERROR: Null argument, when expecting valid pointer! > [0]PETSC ERROR: Null Object: Parameter # 2! > The code do works if I add a call for 'DMDASetUniformCoordinates' in > it. However, I feel it is a waste of calling 'DMDASetUniformCoordinates' > since what follows this calling is to change the coordinate to a > non-uniform gird. Does it have any way to bypass this call or you think it > is fine to keep it here. > If you do not call it, then you have to create the coordinate vector yourself. I would just call it. > 2, Also, if I do by this way, the outputted mesh looks weird, as shown in > the figure. > I was trying to double the dy value manually in the > SetNonUniformGrid(). However, the only mesh changed is just on the z=0 and > x=0 plane. Here I attached my code. Do I need to do anything else except > DMDAVecRestoreArray() after I modified the coordinate information? > I do not understand what you want here. Matt > thanks, > Alan > > > On 2/26/2013 8:46 PM, Barry Smith wrote: > >> On Feb 26, 2013, at 4:49 PM, "Zhenglun (Alan) Wei" >> wrote: >> >> Dear folks, >>> I have one question for the ex49. >>> It do show a way to 'get' the 'coord' by the 'DMDAVecGetArray'. >>> Should I just modified the coordinate on the 'coord' in order to obtain a >>> non-uniform grid? and then, use DMDAVecRestoreArray to link the modified >>> 'coord' back to 'cda' so that the 'coord' will be updated to the mesh I >>> want? >>> >> Yes, you can do this. >> >> >> thanks, >>> Alan >>> >>> On 2/22/2013 3:20 PM, Matthew Knepley wrote: >>> >>>> On Fri, Feb 22, 2013 at 3:57 PM, Zhenglun (Alan) Wei< >>>> zhenglun.wei at gmail.com> wrote: >>>> Dear all, >>>> I hope you're having a nice day. >>>> I wonder if anyone could point out a example for me about the >>>> DMDASetCoordinates(). I found this >>>> (http://www.mcs.anl.gov/petsc/**petsc-current/src/dm/impls/da/** >>>> dacorn.c.html#**DMDASetCoordinates >>>> ) >>>> online; however it does not really demonstrate how to create the >>>> 'coordinate vector - c'. >>>> Also, I found that in the petsc sub-folder: >>>> /petsc-dev/src/dm/impls/patch has a patch.c, which I suppose that it >>>> does a local refinement for mesh. Does this a mature model that we can >>>> use or just a testing code? >>>> >>>> Look at KSP ex49. In the routine DMDACoordViewGnuplot2d() it reads out >>>> the coordinates. >>>> You could just as easily set them in this fashion. >>>> >>>> Matt >>>> thanks, >>>> Alan >>>> >>>> On 11/7/2012 6:55 PM, Barry Smith wrote: >>>> >>>>> On Nov 7, 2012, at 6:50 PM, Alan wrote: >>>>> >>>>> Dear folks, >>>>>> I hope you're having a nice day. >>>>>> I'm testing the Poisson solver in /src/ksp/ksp/examples/** >>>>>> tutorials/ex45. >>>>>> It uses the 'DMDACreate3d' to generate a Cartisian structured grid and >>>>>> solve it with uniform grid size. However, I attempt to modify it to >>>>>> obtain solution with a Cartsian structured grid and non-uniform grid >>>>>> (It >>>>>> is not unstructured local refined mesh). Should I still use the >>>>>> 'DMDACreate3d' and implement different grid size in 'ComputeMatrix' >>>>>> and >>>>>> 'ComputeRHS'? In other words, does 'DMDACreate3d' itself include any >>>>>> mesh size information? >>>>>> >>>>> You can use DMDASetCoordinates() and DMDAGetCoordinates() (called >>>>> DMSetCoordinates() and DMGetCoordinates() in petsc-dev) to keep track of >>>>> the coordinates but you are responsible for generating the correctly scaled >>>>> entries in the matrix etc. >>>>> >>>>> Barry >>>>> >>>>> thanks, >>>>>> Alan >>>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenglun.wei at gmail.com Thu Feb 28 19:05:13 2013 From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei) Date: Thu, 28 Feb 2013 19:05:13 -0600 Subject: [petsc-users] Non-uniform Cartisian Grid In-Reply-To: References: <509B01D6.9030501@gmail.com> <818369A9-2984-4133-A62F-E5DCEE283EA1@mcs.anl.gov> <5127DBB8.6040704@gmail.com> <512D3C0B.3010107@gmail.com> <5D146EFF-9F0A-4BAD-B3AC-F4D0C4E3F012@mcs.anl.gov> <512FF407.6020408@gmail.com> Message-ID: <512FFEC9.2060204@gmail.com> Dear Matt, Thanks for your reply on the 1st question and I'm sorry for the confusion on the 2nd one. In order to create a non-uniform Cartesian grid, I expected that the way I was programming can let me freely control the grid size. In the code attached in the previous E-mail, the 'DMDASetUniformCoordinate' generates a uniform coordinate with 100*48*48 grid number and 0.1*0.1*0.1 grid size. I was trying to manually double the grid size in the y-direction for the whole mesh with operations of DMDA in the 'SetNonUniformGrid()', which follows the method that /src/ksp/ksp/examples/tutorials/ex49.c used. However, as you can see in the figure attached in the previous E-mail, only the mesh on z = 0 plane and x = 0 plane has been changed to dy = 0.2; while I expected all mesh should have dy = 0.2 after I call 'SetNonUniformGrid()'. Any idea about this? thanks, Alan On 2/28/2013 6:31 PM, Matthew Knepley wrote: > On Thu, Feb 28, 2013 at 7:19 PM, Zhenglun (Alan) Wei > > wrote: > > Dear folks, > Here are several questions following up my previous ones: > 1, I modified my code based on ex49.c. However, I found that if I > don't call 'DMDASetUniformCoordinates' before calling > 'DMDAVecGetArray', then an error comes up: > [0]PETSC ERROR: Null argument, when expecting valid pointer! > [0]PETSC ERROR: Null Object: Parameter # 2! > The code do works if I add a call for > 'DMDASetUniformCoordinates' in it. However, I feel it is a waste > of calling 'DMDASetUniformCoordinates' since what follows this > calling is to change the coordinate to a non-uniform gird. Does it > have any way to bypass this call or you think it is fine to keep > it here. > > > If you do not call it, then you have to create the coordinate vector > yourself. I would just call it. > > 2, Also, if I do by this way, the outputted mesh looks weird, as > shown in the figure. > I was trying to double the dy value manually in the > SetNonUniformGrid(). However, the only mesh changed is just on the > z=0 and x=0 plane. Here I attached my code. Do I need to do > anything else except DMDAVecRestoreArray() after I modified the > coordinate information? > > > I do not understand what you want here. > > Matt > > thanks, > Alan > > > On 2/26/2013 8:46 PM, Barry Smith wrote: > > On Feb 26, 2013, at 4:49 PM, "Zhenglun (Alan) > Wei"> > wrote: > > Dear folks, > I have one question for the ex49. > It do show a way to 'get' the 'coord' by the > 'DMDAVecGetArray'. Should I just modified the coordinate > on the 'coord' in order to obtain a non-uniform grid? and > then, use DMDAVecRestoreArray to link the modified 'coord' > back to 'cda' so that the 'coord' will be updated to the > mesh I want? > > Yes, you can do this. > > > thanks, > Alan > > On 2/22/2013 3:20 PM, Matthew Knepley wrote: > > On Fri, Feb 22, 2013 at 3:57 PM, Zhenglun (Alan) > Wei > wrote: > Dear all, > I hope you're having a nice day. > I wonder if anyone could point out a example for me > about the > DMDASetCoordinates(). I found this > (http://www.mcs.anl.gov/petsc/petsc-current/src/dm/impls/da/dacorn.c.html#DMDASetCoordinates) > online; however it does not really demonstrate how to > create the > 'coordinate vector - c'. > Also, I found that in the petsc sub-folder: > /petsc-dev/src/dm/impls/patch has a patch.c, which I > suppose that it > does a local refinement for mesh. Does this a mature > model that we can > use or just a testing code? > > Look at KSP ex49. In the routine > DMDACoordViewGnuplot2d() it reads out the coordinates. > You could just as easily set them in this fashion. > > Matt > thanks, > Alan > > On 11/7/2012 6:55 PM, Barry Smith wrote: > > On Nov 7, 2012, at 6:50 PM, > Alan > wrote: > > Dear folks, > I hope you're having a nice day. > I'm testing the Poisson solver in > /src/ksp/ksp/examples/tutorials/ex45. > It uses the 'DMDACreate3d' to generate a > Cartisian structured grid and > solve it with uniform grid size. However, I > attempt to modify it to > obtain solution with a Cartsian structured > grid and non-uniform grid (It > is not unstructured local refined mesh). > Should I still use the > 'DMDACreate3d' and implement different grid > size in 'ComputeMatrix' and > 'ComputeRHS'? In other words, does > 'DMDACreate3d' itself include any > mesh size information? > > You can use DMDASetCoordinates() and > DMDAGetCoordinates() (called DMSetCoordinates() > and DMGetCoordinates() in petsc-dev) to keep track > of the coordinates but you are responsible for > generating the correctly scaled entries in the > matrix etc. > > Barry > > thanks, > Alan > > > > -- > What most experimenters take for granted before they > begin their experiments is infinitely more interesting > than any results to which their experiments lead. > -- Norbert Wiener > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Feb 28 19:25:42 2013 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 28 Feb 2013 20:25:42 -0500 Subject: [petsc-users] Non-uniform Cartisian Grid In-Reply-To: <512FFEC9.2060204@gmail.com> References: <509B01D6.9030501@gmail.com> <818369A9-2984-4133-A62F-E5DCEE283EA1@mcs.anl.gov> <5127DBB8.6040704@gmail.com> <512D3C0B.3010107@gmail.com> <5D146EFF-9F0A-4BAD-B3AC-F4D0C4E3F012@mcs.anl.gov> <512FF407.6020408@gmail.com> <512FFEC9.2060204@gmail.com> Message-ID: On Thu, Feb 28, 2013 at 8:05 PM, Zhenglun (Alan) Wei wrote: > Dear Matt, > Thanks for your reply on the 1st question and I'm sorry for the > confusion on the 2nd one. > In order to create a non-uniform Cartesian grid, I expected that the > way I was programming can let me freely control the grid size. In the code > attached in the previous E-mail, the 'DMDASetUniformCoordinate' generates a > uniform coordinate with 100*48*48 grid number and 0.1*0.1*0.1 grid size. I > was trying to manually double the grid size in the y-direction for the > whole mesh with operations of DMDA in the 'SetNonUniformGrid()', which > follows the method that /src/ksp/ksp/examples/tutorials/ex49.c used. > However, as you can see in the figure attached in the previous E-mail, only > the mesh on z = 0 plane and x = 0 plane has been changed to dy = 0.2; while > I expected all mesh should have dy = 0.2 after I call > 'SetNonUniformGrid()'. > Any idea about this? > Were you looking over all 3 directions? Matt > thanks, > Alan > > On 2/28/2013 6:31 PM, Matthew Knepley wrote: > > On Thu, Feb 28, 2013 at 7:19 PM, Zhenglun (Alan) Wei < > zhenglun.wei at gmail.com> wrote: > >> Dear folks, >> Here are several questions following up my previous ones: >> 1, I modified my code based on ex49.c. However, I found that if I don't >> call 'DMDASetUniformCoordinates' before calling 'DMDAVecGetArray', then an >> error comes up: >> [0]PETSC ERROR: Null argument, when expecting valid pointer! >> [0]PETSC ERROR: Null Object: Parameter # 2! >> The code do works if I add a call for 'DMDASetUniformCoordinates' in >> it. However, I feel it is a waste of calling 'DMDASetUniformCoordinates' >> since what follows this calling is to change the coordinate to a >> non-uniform gird. Does it have any way to bypass this call or you think it >> is fine to keep it here. >> > > If you do not call it, then you have to create the coordinate vector > yourself. I would just call it. > > >> 2, Also, if I do by this way, the outputted mesh looks weird, as shown in >> the figure. >> I was trying to double the dy value manually in the >> SetNonUniformGrid(). However, the only mesh changed is just on the z=0 and >> x=0 plane. Here I attached my code. Do I need to do anything else except >> DMDAVecRestoreArray() after I modified the coordinate information? >> > > I do not understand what you want here. > > Matt > > >> thanks, >> Alan >> >> >> On 2/26/2013 8:46 PM, Barry Smith wrote: >> >>> On Feb 26, 2013, at 4:49 PM, "Zhenglun (Alan) Wei"< >>> zhenglun.wei at gmail.com> wrote: >>> >>> Dear folks, >>>> I have one question for the ex49. >>>> It do show a way to 'get' the 'coord' by the 'DMDAVecGetArray'. >>>> Should I just modified the coordinate on the 'coord' in order to obtain a >>>> non-uniform grid? and then, use DMDAVecRestoreArray to link the modified >>>> 'coord' back to 'cda' so that the 'coord' will be updated to the mesh I >>>> want? >>>> >>> Yes, you can do this. >>> >>> >>> thanks, >>>> Alan >>>> >>>> On 2/22/2013 3:20 PM, Matthew Knepley wrote: >>>> >>>>> On Fri, Feb 22, 2013 at 3:57 PM, Zhenglun (Alan) Wei< >>>>> zhenglun.wei at gmail.com> wrote: >>>>> Dear all, >>>>> I hope you're having a nice day. >>>>> I wonder if anyone could point out a example for me about the >>>>> DMDASetCoordinates(). I found this >>>>> ( >>>>> http://www.mcs.anl.gov/petsc/petsc-current/src/dm/impls/da/dacorn.c.html#DMDASetCoordinates >>>>> ) >>>>> online; however it does not really demonstrate how to create the >>>>> 'coordinate vector - c'. >>>>> Also, I found that in the petsc sub-folder: >>>>> /petsc-dev/src/dm/impls/patch has a patch.c, which I suppose that it >>>>> does a local refinement for mesh. Does this a mature model that we can >>>>> use or just a testing code? >>>>> >>>>> Look at KSP ex49. In the routine DMDACoordViewGnuplot2d() it reads out >>>>> the coordinates. >>>>> You could just as easily set them in this fashion. >>>>> >>>>> Matt >>>>> thanks, >>>>> Alan >>>>> >>>>> On 11/7/2012 6:55 PM, Barry Smith wrote: >>>>> >>>>>> On Nov 7, 2012, at 6:50 PM, Alan wrote: >>>>>> >>>>>> Dear folks, >>>>>>> I hope you're having a nice day. >>>>>>> I'm testing the Poisson solver in >>>>>>> /src/ksp/ksp/examples/tutorials/ex45. >>>>>>> It uses the 'DMDACreate3d' to generate a Cartisian structured grid >>>>>>> and >>>>>>> solve it with uniform grid size. However, I attempt to modify it to >>>>>>> obtain solution with a Cartsian structured grid and non-uniform grid >>>>>>> (It >>>>>>> is not unstructured local refined mesh). Should I still use the >>>>>>> 'DMDACreate3d' and implement different grid size in 'ComputeMatrix' >>>>>>> and >>>>>>> 'ComputeRHS'? In other words, does 'DMDACreate3d' itself include any >>>>>>> mesh size information? >>>>>>> >>>>>> You can use DMDASetCoordinates() and DMDAGetCoordinates() (called >>>>>> DMSetCoordinates() and DMGetCoordinates() in petsc-dev) to keep track of >>>>>> the coordinates but you are responsible for generating the correctly scaled >>>>>> entries in the matrix etc. >>>>>> >>>>>> Barry >>>>>> >>>>>> thanks, >>>>>>> Alan >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Feb 28 19:40:16 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 28 Feb 2013 19:40:16 -0600 Subject: [petsc-users] Non-uniform Cartisian Grid In-Reply-To: <512FF407.6020408@gmail.com> References: <509B01D6.9030501@gmail.com> <818369A9-2984-4133-A62F-E5DCEE283EA1@mcs.anl.gov> <5127DBB8.6040704@gmail.com> <512D3C0B.3010107@gmail.com> <5D146EFF-9F0A-4BAD-B3AC-F4D0C4E3F012@mcs.anl.gov> <512FF407.6020408@gmail.com> Message-ID: Why do you have the -1 at the end of the loops. If you fix that it will run as you would like. Note use a small imax,jmax, and kmax for testing. for (k = sk; k < sk+nz-1; k++) { for (j = sj; j < sj+ny-1; j++) { for (i = si; i < si+nx-1; i++) { On Feb 28, 2013, at 6:19 PM, "Zhenglun (Alan) Wei" wrote: > Dear folks, > Here are several questions following up my previous ones: > 1, I modified my code based on ex49.c. However, I found that if I don't call 'DMDASetUniformCoordinates' before calling 'DMDAVecGetArray', then an error comes up: > [0]PETSC ERROR: Null argument, when expecting valid pointer! > [0]PETSC ERROR: Null Object: Parameter # 2! > The code do works if I add a call for 'DMDASetUniformCoordinates' in it. However, I feel it is a waste of calling 'DMDASetUniformCoordinates' since what follows this calling is to change the coordinate to a non-uniform gird. Does it have any way to bypass this call or you think it is fine to keep it here. > > 2, Also, if I do by this way, the outputted mesh looks weird, as shown in the figure. > I was trying to double the dy value manually in the SetNonUniformGrid(). However, the only mesh changed is just on the z=0 and x=0 plane. Here I attached my code. Do I need to do anything else except DMDAVecRestoreArray() after I modified the coordinate information? > > thanks, > Alan > > > On 2/26/2013 8:46 PM, Barry Smith wrote: >> On Feb 26, 2013, at 4:49 PM, "Zhenglun (Alan) Wei" wrote: >> >>> Dear folks, >>> I have one question for the ex49. >>> It do show a way to 'get' the 'coord' by the 'DMDAVecGetArray'. Should I just modified the coordinate on the 'coord' in order to obtain a non-uniform grid? and then, use DMDAVecRestoreArray to link the modified 'coord' back to 'cda' so that the 'coord' will be updated to the mesh I want? >> Yes, you can do this. >> >> >>> thanks, >>> Alan >>> >>> On 2/22/2013 3:20 PM, Matthew Knepley wrote: >>>> On Fri, Feb 22, 2013 at 3:57 PM, Zhenglun (Alan) Wei wrote: >>>> Dear all, >>>> I hope you're having a nice day. >>>> I wonder if anyone could point out a example for me about the >>>> DMDASetCoordinates(). I found this >>>> (http://www.mcs.anl.gov/petsc/petsc-current/src/dm/impls/da/dacorn.c.html#DMDASetCoordinates) >>>> online; however it does not really demonstrate how to create the >>>> 'coordinate vector - c'. >>>> Also, I found that in the petsc sub-folder: >>>> /petsc-dev/src/dm/impls/patch has a patch.c, which I suppose that it >>>> does a local refinement for mesh. Does this a mature model that we can >>>> use or just a testing code? >>>> >>>> Look at KSP ex49. In the routine DMDACoordViewGnuplot2d() it reads out the coordinates. >>>> You could just as easily set them in this fashion. >>>> >>>> Matt >>>> thanks, >>>> Alan >>>> >>>> On 11/7/2012 6:55 PM, Barry Smith wrote: >>>>> On Nov 7, 2012, at 6:50 PM, Alan wrote: >>>>> >>>>>> Dear folks, >>>>>> I hope you're having a nice day. >>>>>> I'm testing the Poisson solver in /src/ksp/ksp/examples/tutorials/ex45. >>>>>> It uses the 'DMDACreate3d' to generate a Cartisian structured grid and >>>>>> solve it with uniform grid size. However, I attempt to modify it to >>>>>> obtain solution with a Cartsian structured grid and non-uniform grid (It >>>>>> is not unstructured local refined mesh). Should I still use the >>>>>> 'DMDACreate3d' and implement different grid size in 'ComputeMatrix' and >>>>>> 'ComputeRHS'? In other words, does 'DMDACreate3d' itself include any >>>>>> mesh size information? >>>>> You can use DMDASetCoordinates() and DMDAGetCoordinates() (called DMSetCoordinates() and DMGetCoordinates() in petsc-dev) to keep track of the coordinates but you are responsible for generating the correctly scaled entries in the matrix etc. >>>>> >>>>> Barry >>>>> >>>>>> thanks, >>>>>> Alan >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>> -- Norbert Wiener > > From zhenglun.wei at gmail.com Thu Feb 28 19:35:36 2013 From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei) Date: Thu, 28 Feb 2013 19:35:36 -0600 Subject: [petsc-users] Non-uniform Cartisian Grid In-Reply-To: References: <509B01D6.9030501@gmail.com> <818369A9-2984-4133-A62F-E5DCEE283EA1@mcs.anl.gov> <5127DBB8.6040704@gmail.com> <512D3C0B.3010107@gmail.com> <5D146EFF-9F0A-4BAD-B3AC-F4D0C4E3F012@mcs.anl.gov> <512FF407.6020408@gmail.com> <512FFEC9.2060204@gmail.com> Message-ID: <513005E8.1050305@gmail.com> Dear Matt, Yes. if I only double the dx, then, the mesh has been changed are on z=0 plane and y = 0 plane. Similarly, if the dz is doubled, the mesh has been changed are only on x = 0 plane and y = 0 plane (see the picture attached). Moreover, if I double all direction (i.e. dx = dy = dz = 0.2), the mesh is shown in the picture 'doubleAll' in the attachment. It seems that the mesh generated by the 'DMDASetUniformCoordinate' is always there. best, Alan On 2/28/2013 7:25 PM, Matthew Knepley wrote: > On Thu, Feb 28, 2013 at 8:05 PM, Zhenglun (Alan) Wei > > wrote: > > Dear Matt, > Thanks for your reply on the 1st question and I'm sorry for > the confusion on the 2nd one. > In order to create a non-uniform Cartesian grid, I expected > that the way I was programming can let me freely control the grid > size. In the code attached in the previous E-mail, the > 'DMDASetUniformCoordinate' generates a uniform coordinate with > 100*48*48 grid number and 0.1*0.1*0.1 grid size. I was trying to > manually double the grid size in the y-direction for the whole > mesh with operations of DMDA in the 'SetNonUniformGrid()', which > follows the method that /src/ksp/ksp/examples/tutorials/ex49.c > used. However, as you can see in the figure attached in the > previous E-mail, only the mesh on z = 0 plane and x = 0 plane has > been changed to dy = 0.2; while I expected all mesh should have dy > = 0.2 after I call 'SetNonUniformGrid()'. > Any idea about this? > > > Were you looking over all 3 directions? > > Matt > > thanks, > Alan > > On 2/28/2013 6:31 PM, Matthew Knepley wrote: >> On Thu, Feb 28, 2013 at 7:19 PM, Zhenglun (Alan) Wei >> > wrote: >> >> Dear folks, >> Here are several questions following up my previous ones: >> 1, I modified my code based on ex49.c. However, I found that >> if I don't call 'DMDASetUniformCoordinates' before calling >> 'DMDAVecGetArray', then an error comes up: >> [0]PETSC ERROR: Null argument, when expecting valid pointer! >> [0]PETSC ERROR: Null Object: Parameter # 2! >> The code do works if I add a call for >> 'DMDASetUniformCoordinates' in it. However, I feel it is a >> waste of calling 'DMDASetUniformCoordinates' since what >> follows this calling is to change the coordinate to a >> non-uniform gird. Does it have any way to bypass this call or >> you think it is fine to keep it here. >> >> >> If you do not call it, then you have to create the coordinate >> vector yourself. I would just call it. >> >> 2, Also, if I do by this way, the outputted mesh looks weird, >> as shown in the figure. >> I was trying to double the dy value manually in the >> SetNonUniformGrid(). However, the only mesh changed is just >> on the z=0 and x=0 plane. Here I attached my code. Do I need >> to do anything else except DMDAVecRestoreArray() after I >> modified the coordinate information? >> >> >> I do not understand what you want here. >> >> Matt >> >> thanks, >> Alan >> >> >> On 2/26/2013 8:46 PM, Barry Smith wrote: >> >> On Feb 26, 2013, at 4:49 PM, "Zhenglun (Alan) >> Wei"> > wrote: >> >> Dear folks, >> I have one question for the ex49. >> It do show a way to 'get' the 'coord' by the >> 'DMDAVecGetArray'. Should I just modified the >> coordinate on the 'coord' in order to obtain a >> non-uniform grid? and then, use DMDAVecRestoreArray >> to link the modified 'coord' back to 'cda' so that >> the 'coord' will be updated to the mesh I want? >> >> Yes, you can do this. >> >> >> thanks, >> Alan >> >> On 2/22/2013 3:20 PM, Matthew Knepley wrote: >> >> On Fri, Feb 22, 2013 at 3:57 PM, Zhenglun (Alan) >> Wei> > wrote: >> Dear all, >> I hope you're having a nice day. >> I wonder if anyone could point out a example for >> me about the >> DMDASetCoordinates(). I found this >> (http://www.mcs.anl.gov/petsc/petsc-current/src/dm/impls/da/dacorn.c.html#DMDASetCoordinates) >> online; however it does not really demonstrate >> how to create the >> 'coordinate vector - c'. >> Also, I found that in the petsc sub-folder: >> /petsc-dev/src/dm/impls/patch has a patch.c, >> which I suppose that it >> does a local refinement for mesh. Does this a >> mature model that we can >> use or just a testing code? >> >> Look at KSP ex49. In the routine >> DMDACoordViewGnuplot2d() it reads out the >> coordinates. >> You could just as easily set them in this fashion. >> >> Matt >> thanks, >> Alan >> >> On 11/7/2012 6:55 PM, Barry Smith wrote: >> >> On Nov 7, 2012, at 6:50 PM, >> Alan> > wrote: >> >> Dear folks, >> I hope you're having a nice day. >> I'm testing the Poisson solver in >> /src/ksp/ksp/examples/tutorials/ex45. >> It uses the 'DMDACreate3d' to generate a >> Cartisian structured grid and >> solve it with uniform grid size. However, >> I attempt to modify it to >> obtain solution with a Cartsian >> structured grid and non-uniform grid (It >> is not unstructured local refined mesh). >> Should I still use the >> 'DMDACreate3d' and implement different >> grid size in 'ComputeMatrix' and >> 'ComputeRHS'? In other words, does >> 'DMDACreate3d' itself include any >> mesh size information? >> >> You can use DMDASetCoordinates() and >> DMDAGetCoordinates() (called >> DMSetCoordinates() and DMGetCoordinates() in >> petsc-dev) to keep track of the coordinates >> but you are responsible for generating the >> correctly scaled entries in the matrix etc. >> >> Barry >> >> thanks, >> Alan >> >> >> >> -- >> What most experimenters take for granted before >> they begin their experiments is infinitely more >> interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: doubleZ.jpg Type: image/jpeg Size: 197567 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: doubleAll.jpg Type: image/jpeg Size: 302638 bytes Desc: not available URL: