From manfred.gratt at uibk.ac.at Thu Mar 1 03:19:39 2012 From: manfred.gratt at uibk.ac.at (Manfred Gratt) Date: Thu, 01 Mar 2012 10:19:39 +0100 Subject: [petsc-users] Problem with KSPSetOperators Message-ID: <4F4F3F2B.70306@uibk.ac.at> Dear all, I am solving with cholesky finite element problems where I have to solve multiple times. To improve the performance I use KSPSetOperators and set the option SAME_NONZERO_PATTERN because the matrix structure is always the same in successive solves. This works great as long I use more than one processor. When I use MUMPS, only one processor and a matrix made with MATSEQSBAIJ the solving segfaults in the second solve. When I use spooles or use LU with AIJ it also works, but I need MUMPS as MatGetInertia only works with MUMPS with multiple processors. The segfault: [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] MatFactorNumeric_MUMPS line 642 src/mat/impls/aij/mpi/mumps/mumps.c [0]PETSC ERROR: [0] MatCholeskyFactorNumeric line 3037 src/mat/interface/matrix.c [0]PETSC ERROR: [0] PCSetUp_Cholesky line 92 src/ksp/pc/impls/factor/cholesky/cholesky.c [0]PETSC ERROR: [0] PCSetUp line 797 src/ksp/pc/interface/precon.c [0]PETSC ERROR: [0] KSPSetUp line 184 src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] KSPSolve line 331 src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./icona on a linux-gnu named sunray.dps.uibk.ac.at by manfred Thu Mar 1 10:06:47 2012 [0]PETSC ERROR: Libraries linked from /home/manfred/petsc-3.2-p6/linux-gnu-c/lib [0]PETSC ERROR: Configure run at Thu Mar 1 09:11:08 2012 [0]PETSC ERROR: Configure options --with-cc=/home/manfred/gcc-4.6.2/bin/gcc --download-mpich=1 --with-blas-lapack=1 --download-f-blas-lapack=1 PETSC_ARCH=linux-gnu-c --with-clanguage=cxx --with-cxx=/home/manfred/gcc-4.6.2/bin/g++ --with-fc=/home/manfred/gcc-4.6.2/bin/gfortran --with-scalapack=1 --download-scalapack=1 --with-blacs=1 --download-blacs=1 --with-parmetis=1 --download-parmetis=1 --with-spooles=1 --download-spooles=1 --with-mumps=1 --download-mumps=1 --with-blopex=1 --download-blopex=1 --with-debugging=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 [cli_0]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 with valgrind I get the error ==6853== Invalid read of size 8 ==6853== at 0x11CDDAA: dmumps_148_ (dmumps_part1.F:1299) ==6853== by 0x111093C: dmumps_142_ (dmumps_part5.F:5094) ==6853== by 0x11C9754: dmumps_ (dmumps_part1.F:642) ==6853== by 0x10B4DDB: dmumps_f77_ (dmumps_part3.F:6651) ==6853== by 0x108E0FF: dmumps_c (mumps_c.c:422) ==6853== by 0xAE6EB3: MatFactorNumeric_MUMPS(_p_Mat*, _p_Mat*, MatFactorInfo const*) (mumps.c:667) ==6853== by 0x9832D7: MatCholeskyFactorNumeric(_p_Mat*, _p_Mat*, MatFactorInfo const*) (matrix.c:3050) ==6853== by 0xDFFE2D: PCSetUp_Cholesky(_p_PC*) (cholesky.c:157) ==6853== by 0xD7B390: PCSetUp(_p_PC*) (precon.c:819) ==6853== by 0xE47BD2: KSPSetUp(_p_KSP*) (itfunc.c:260) ==6853== by 0xE48C47: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:379) ==6853== by 0x4914E0: Task::runCalc() (Task.cpp:1204) Is this a bug or doing I am something wrong? Should I go to MUMPS with this error? I have currently a workaround to simply not use SAME_NONZERO_PATTERN if using only one processor but as a result it is much slower in successive solves. Thank you in advance Manfred Gratt From friedmud at gmail.com Thu Mar 1 10:16:41 2012 From: friedmud at gmail.com (Derek Gaston) Date: Thu, 1 Mar 2012 09:16:41 -0700 Subject: [petsc-users] Number of Matrices with -snes_mf and -snes_mf_operator Message-ID: In trying to track down where all of our memory usage is coming from I noticed that PETSc is saying that more matrices than I expect are getting created. For instance, with -snes_mf I would expect to see just one matrix (we always create one, even with -snes_mf)... but instead I'm seeing 2 (from -log_summary): Matrix 2 2 1634161540 0 Then when using -snes_mf_operator with ilu as the preconditioner I'm actually seeing _3_! Matrix 3 3 3080557036 0 As you can see, our matrices are fairly heavy (cubic hermites in 3d have a heavy sparsity pattern) so it would be awesome to keep the number of them down. I've double checked that we're not calling MatCreate more than once in our code. So is this expected behavior or am I doing something wrong somewhere else? What's the best way to track this down... maybe print a stack trace inside MatCreate? Particulars: PETSc 3.1-p8 (yes yes old... ;-) Using it through libMesh Thanks, Derek -------------- next part -------------- An HTML attachment was scrubbed... URL: From karpeev at mcs.anl.gov Thu Mar 1 10:31:06 2012 From: karpeev at mcs.anl.gov (Dmitry Karpeev) Date: Thu, 1 Mar 2012 10:31:06 -0600 Subject: [petsc-users] Number of Matrices with -snes_mf and -snes_mf_operator In-Reply-To: References: Message-ID: MatCreate probably gets called inside SNES via MatDuplicate. Set a breakpoint in MatCreate to catch it. Dmitry. On Thu, Mar 1, 2012 at 10:16 AM, Derek Gaston wrote: > In trying to track down where all of our memory usage is coming from I > noticed that PETSc is saying that more matrices than I expect are getting > created. For instance, with -snes_mf I would expect to see just one matrix > (we always create one, even with -snes_mf)... but instead I'm seeing 2 > (from -log_summary): > > Matrix 2 2 1634161540 0 > > Then when using -snes_mf_operator with ilu as the preconditioner I'm > actually seeing _3_! > > Matrix 3 3 3080557036 0 > > As you can see, our matrices are fairly heavy (cubic hermites in 3d have a > heavy sparsity pattern) so it would be awesome to keep the number of them > down. I've double checked that we're not calling MatCreate more than once > in our code. So is this expected behavior or am I doing something wrong > somewhere else? What's the best way to track this down... maybe print a > stack trace inside MatCreate? > > Particulars: > PETSc 3.1-p8 (yes yes old... ;-) > Using it through libMesh > > Thanks, > Derek > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Mar 1 10:37:14 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 1 Mar 2012 10:37:14 -0600 Subject: [petsc-users] Number of Matrices with -snes_mf and -snes_mf_operator In-Reply-To: References: Message-ID: snes_mf_operator has one MFFD (almost free), your assembled matrix, and the factored matrix created by the preconditioner. On Mar 1, 2012 10:16 AM, "Derek Gaston" wrote: > In trying to track down where all of our memory usage is coming from I > noticed that PETSc is saying that more matrices than I expect are getting > created. For instance, with -snes_mf I would expect to see just one matrix > (we always create one, even with -snes_mf)... but instead I'm seeing 2 > (from -log_summary): > > Matrix 2 2 1634161540 0 > > Then when using -snes_mf_operator with ilu as the preconditioner I'm > actually seeing _3_! > > Matrix 3 3 3080557036 0 > > As you can see, our matrices are fairly heavy (cubic hermites in 3d have a > heavy sparsity pattern) so it would be awesome to keep the number of them > down. I've double checked that we're not calling MatCreate more than once > in our code. So is this expected behavior or am I doing something wrong > somewhere else? What's the best way to track this down... maybe print a > stack trace inside MatCreate? > > Particulars: > PETSc 3.1-p8 (yes yes old... ;-) > Using it through libMesh > > Thanks, > Derek > -------------- next part -------------- An HTML attachment was scrubbed... URL: From friedmud at gmail.com Thu Mar 1 10:40:05 2012 From: friedmud at gmail.com (Derek Gaston) Date: Thu, 1 Mar 2012 09:40:05 -0700 Subject: [petsc-users] Number of Matrices with -snes_mf and -snes_mf_operator In-Reply-To: References: Message-ID: On Thu, Mar 1, 2012 at 9:37 AM, Jed Brown wrote: > snes_mf_operator has one MFFD (almost free), your assembled matrix, and > the factored matrix created by the preconditioner. > I see, so one of those is the MFFD which really isn't taking up much space. I suppose that's why going from 2->3 matrices ~doubled the memory. So there's probably nothing wrong here... those matrices are needed and really do take up that much memory... damn ;-) Derek -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Mar 1 11:24:13 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 1 Mar 2012 11:24:13 -0600 Subject: [petsc-users] Number of Matrices with -snes_mf and -snes_mf_operator In-Reply-To: References: Message-ID: On Thu, Mar 1, 2012 at 10:40, Derek Gaston wrote: > I see, so one of those is the MFFD which really isn't taking up much > space. I suppose that's why going from 2->3 matrices ~doubled the memory. > > So there's probably nothing wrong here... those matrices are needed and > really do take up that much memory... damn ;-) > When you run with -snes_view, you see the number of nonzeros for all matrices in the system. Third order elements are really expensive. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Thu Mar 1 12:09:35 2012 From: john.fettig at gmail.com (John Fettig) Date: Thu, 1 Mar 2012 13:09:35 -0500 Subject: [petsc-users] ccgraph.c error In-Reply-To: References: Message-ID: On Sun, Nov 27, 2011 at 3:28 PM, Matthew Knepley wrote: > On Sun, Nov 27, 2011 at 2:25 PM, Sean Farley wrote: > >> So there would not be the problem with empty domains? I could see it >>> happening for this many processes. >> >> >> I'm saying that he's using the older parmetis (3.1) and SuperLU_DIST >> (2.5) (and also Jed was right, this looks like 3.2p1), which as far I >> remember still have the empty domain problem. >> > > Okay, executive summary: > > There is a problem with the interaction of SuperLU and ParMetis which is > fixed in petsc-dev, using the latest > releases of both packages. Consider upgrading. > Sorry to dig up an old thread (I seem to have a tendency of doing that here!), but this problem exists in petsc-3.2-p6 with MUMPS and Metis as well. Looking at the differences between the dev version of mump.c and the release version, I don't see anything that would have obviously fixed this. Could you give me a hint about what was changed? Is it as simple as using the updated Metis package (in dev)? How difficult would it be to backport the fix to the released version? John -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Mar 1 12:13:42 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 1 Mar 2012 12:13:42 -0600 Subject: [petsc-users] Number of Matrices with -snes_mf and -snes_mf_operator In-Reply-To: References: Message-ID: On Thu, Mar 1, 2012 at 11:24 AM, Jed Brown wrote: > On Thu, Mar 1, 2012 at 10:40, Derek Gaston wrote: > >> I see, so one of those is the MFFD which really isn't taking up much >> space. I suppose that's why going from 2->3 matrices ~doubled the memory. >> >> So there's probably nothing wrong here... those matrices are needed and >> really do take up that much memory... damn ;-) >> > > When you run with -snes_view, you see the number of nonzeros for all > matrices in the system. Third order elements are really expensive. > You might consider applying this operator matrix-free, and using a low order approximation for the preconditioning matrix. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Thu Mar 1 12:32:42 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Thu, 1 Mar 2012 12:32:42 -0600 Subject: [petsc-users] Problem with KSPSetOperators In-Reply-To: <4F4F3F2B.70306@uibk.ac.at> References: <4F4F3F2B.70306@uibk.ac.at> Message-ID: Manfred : I tested 'SAME_NONZERO_PATTERN' with sequential mumps using petsc-dev/src/ksp/ksp/examples/tutorials/ex5.c: petsc-dev/src/ksp/ksp/examples/tutorials>./ex5 -ksp_monitor -mat_type sbaij -pc_type cholesky -pc_factor_mat_solver_package mumps 0 KSP Residual norm 7.416198487096e+00 1 KSP Residual norm 8.281965561733e-16 0 KSP Residual norm 7.416198487096e+00 1 KSP Residual norm 9.211962217404e-16 Can you switch to petsc-dev (http://www.mcs.anl.gov/petsc/developers/index.html) to see if you still get the same error? Hong > > > I am solving with cholesky finite element problems where I have to solve > multiple times. To improve the performance I use KSPSetOperators and set > the option SAME_NONZERO_PATTERN because the matrix structure is always the > same in successive solves. This works great as long I use more than one > processor. When I use MUMPS, only one processor and a matrix made with > MATSEQSBAIJ the solving segfaults in the second solve. When I use spooles > or use LU with AIJ it also works, but I need MUMPS as MatGetInertia only > works with MUMPS with multiple processors. > > The segfault: > [0]PETSC ERROR: ------------------------------** > ------------------------------**------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/** > petsc-as/documentation/faq.**html#valgrind[0]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------**------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] MatFactorNumeric_MUMPS line 642 > src/mat/impls/aij/mpi/mumps/**mumps.c > [0]PETSC ERROR: [0] MatCholeskyFactorNumeric line 3037 > src/mat/interface/matrix.c > [0]PETSC ERROR: [0] PCSetUp_Cholesky line 92 src/ksp/pc/impls/factor/** > cholesky/cholesky.c > [0]PETSC ERROR: [0] PCSetUp line 797 src/ksp/pc/interface/precon.c > [0]PETSC ERROR: [0] KSPSetUp line 184 src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] KSPSolve line 331 src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: --------------------- Error Message > ------------------------------**------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: ------------------------------** > ------------------------------**------------ > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 > CST 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------** > ------------------------------**------------ > [0]PETSC ERROR: ./icona on a linux-gnu named sunray.dps.uibk.ac.at by > manfred Thu Mar 1 10:06:47 2012 > [0]PETSC ERROR: Libraries linked from /home/manfred/petsc-3.2-p6/** > linux-gnu-c/lib > [0]PETSC ERROR: Configure run at Thu Mar 1 09:11:08 2012 > [0]PETSC ERROR: Configure options --with-cc=/home/manfred/gcc-4.**6.2/bin/gcc > --download-mpich=1 --with-blas-lapack=1 --download-f-blas-lapack=1 > PETSC_ARCH=linux-gnu-c --with-clanguage=cxx --with-cxx=/home/manfred/gcc-* > *4.6.2/bin/g++ --with-fc=/home/manfred/gcc-4.**6.2/bin/gfortran > --with-scalapack=1 --download-scalapack=1 --with-blacs=1 --download-blacs=1 > --with-parmetis=1 --download-parmetis=1 --with-spooles=1 > --download-spooles=1 --with-mumps=1 --download-mumps=1 --with-blopex=1 > --download-blopex=1 --with-debugging=1 > [0]PETSC ERROR: ------------------------------** > ------------------------------**------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > [cli_0]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > > with valgrind I get the error > ==6853== Invalid read of size 8 > ==6853== at 0x11CDDAA: dmumps_148_ (dmumps_part1.F:1299) > ==6853== by 0x111093C: dmumps_142_ (dmumps_part5.F:5094) > ==6853== by 0x11C9754: dmumps_ (dmumps_part1.F:642) > ==6853== by 0x10B4DDB: dmumps_f77_ (dmumps_part3.F:6651) > ==6853== by 0x108E0FF: dmumps_c (mumps_c.c:422) > ==6853== by 0xAE6EB3: MatFactorNumeric_MUMPS(_p_Mat***, _p_Mat*, > MatFactorInfo const*) (mumps.c:667) > ==6853== by 0x9832D7: MatCholeskyFactorNumeric(_p_**Mat*, _p_Mat*, > MatFactorInfo const*) (matrix.c:3050) > ==6853== by 0xDFFE2D: PCSetUp_Cholesky(_p_PC*) (cholesky.c:157) > ==6853== by 0xD7B390: PCSetUp(_p_PC*) (precon.c:819) > ==6853== by 0xE47BD2: KSPSetUp(_p_KSP*) (itfunc.c:260) > ==6853== by 0xE48C47: KSPSolve(_p_KSP*, _p_Vec*, _p_Vec*) (itfunc.c:379) > ==6853== by 0x4914E0: Task::runCalc() (Task.cpp:1204) > > Is this a bug or doing I am something wrong? Should I go to MUMPS with > this error? I have currently a workaround to simply not use > SAME_NONZERO_PATTERN if using only one processor but as a result it is much > slower in successive solves. > > Thank you in advance > Manfred Gratt > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean at mcs.anl.gov Thu Mar 1 12:32:58 2012 From: sean at mcs.anl.gov (Sean Farley) Date: Thu, 1 Mar 2012 12:32:58 -0600 Subject: [petsc-users] ccgraph.c error In-Reply-To: References: Message-ID: On Thu, Mar 1, 2012 at 12:09 PM, John Fettig wrote: > Sorry to dig up an old thread (I seem to have a tendency of doing that > here!), but this problem exists in petsc-3.2-p6 with MUMPS and Metis as > well. Looking at the differences between the dev version of mump.c and the > release version, I don't see anything that would have obviously fixed > this. Could you give me a hint about what was changed? Is it as simple as > using the updated Metis package (in dev)? How difficult would it be to > backport the fix to the released version? It'd be pretty murky to do this. Start by looking at this changeset: http://petsc.cs.iit.edu/petsc/petsc-dev/rev/0c0bf07c7a32 Luckily, you just need to look at revisions that I authored ( sean at mcs.anl.gov): hg log -r 'user(sean)'. The *real* difficultly would be the changes I made to SuperLU_DIST, MUMPS, and SuiteSparse and hope that it all works together for 3.2p6. Related question, would it be a problem to just use petsc-dev? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Thu Mar 1 13:51:12 2012 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Thu, 1 Mar 2012 11:51:12 -0800 Subject: [petsc-users] ccgraph.c error In-Reply-To: References: Message-ID: Hi all, I was also having this problem (assertion fails) when using ParMetis 3.2 as a part of PETSc 3.2-p6. I could fix it by installing ParMetis 4.0.2 directly and passing the header/libs to petsc (check the chages in http://glaros.dtc.umn.edu/gkhome/metis/parmetis/changes) . I do not see this version of ParMetis in the petsc ftp site, what version is used in petsc-dev? Mohammad On Thu, Mar 1, 2012 at 10:32 AM, Sean Farley wrote: > On Thu, Mar 1, 2012 at 12:09 PM, John Fettig wrote: > >> Sorry to dig up an old thread (I seem to have a tendency of doing that >> here!), but this problem exists in petsc-3.2-p6 with MUMPS and Metis as >> well. Looking at the differences between the dev version of mump.c and the >> release version, I don't see anything that would have obviously fixed >> this. Could you give me a hint about what was changed? Is it as simple as >> using the updated Metis package (in dev)? How difficult would it be to >> backport the fix to the released version? > > > It'd be pretty murky to do this. Start by looking at this changeset: > > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/0c0bf07c7a32 > > Luckily, you just need to look at revisions that I authored ( > sean at mcs.anl.gov): hg log -r 'user(sean)'. The *real* difficultly would > be the changes I made to SuperLU_DIST, MUMPS, and SuiteSparse and hope that > it all works together for 3.2p6. > > Related question, would it be a problem to just use petsc-dev? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean at mcs.anl.gov Thu Mar 1 13:55:09 2012 From: sean at mcs.anl.gov (Sean Farley) Date: Thu, 1 Mar 2012 13:55:09 -0600 Subject: [petsc-users] ccgraph.c error In-Reply-To: References: Message-ID: > > I was also having this problem (assertion fails) when using ParMetis 3.2 > as a part of PETSc 3.2-p6. I could fix it by installing ParMetis 4.0.2 > directly and passing the header/libs to petsc (check the chages in > http://glaros.dtc.umn.edu/gkhome/metis/parmetis/changes) . > If you're monkeying around this much with PETSc, why not just switch to petsc-dev? > I do not see this version of ParMetis in the petsc ftp site, what version > is used in petsc-dev? > ftp://ftp.mcs.anl.gov/pub/petsc/externalpackages/parmetis-4.0.2-p1.tar.gz -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Thu Mar 1 13:58:08 2012 From: john.fettig at gmail.com (John Fettig) Date: Thu, 1 Mar 2012 14:58:08 -0500 Subject: [petsc-users] ccgraph.c error In-Reply-To: References: Message-ID: On Thu, Mar 1, 2012 at 1:32 PM, Sean Farley wrote: > On Thu, Mar 1, 2012 at 12:09 PM, John Fettig wrote: > >> Sorry to dig up an old thread (I seem to have a tendency of doing that >> here!), but this problem exists in petsc-3.2-p6 with MUMPS and Metis as >> well. Looking at the differences between the dev version of mump.c and the >> release version, I don't see anything that would have obviously fixed >> this. Could you give me a hint about what was changed? Is it as simple as >> using the updated Metis package (in dev)? How difficult would it be to >> backport the fix to the released version? > > > It'd be pretty murky to do this. Start by looking at this changeset: > > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/0c0bf07c7a32 > Do I understand correctly that this was fixed mainly by using the latest Metis? Were there any patches to MUMPS that weren't directly related to the updated Metis? I'm just trying to understand what fixed the ccgraph.c problem. It doesn't seem to happen in 3.1-p8, although that version of MUMPS is a lot slower (not sure why). Related question, would it be a problem to just use petsc-dev? > I could, but it's a little too unstable for use in production software. Using the released/patched version simplifies a lot of things on many levels. John -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Thu Mar 1 13:59:30 2012 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Thu, 1 Mar 2012 11:59:30 -0800 Subject: [petsc-users] ccgraph.c error In-Reply-To: References: Message-ID: The only reason I did it by hand was I could not find it -- its totally weird how I missed it there! thanks anyways. On Thu, Mar 1, 2012 at 11:55 AM, Sean Farley wrote: > I was also having this problem (assertion fails) when using ParMetis 3.2 >> as a part of PETSc 3.2-p6. I could fix it by installing ParMetis 4.0.2 >> directly and passing the header/libs to petsc (check the chages in >> http://glaros.dtc.umn.edu/gkhome/metis/parmetis/changes) . >> > > If you're monkeying around this much with PETSc, why not just switch to > petsc-dev? > > >> I do not see this version of ParMetis in the petsc ftp site, what >> version is used in petsc-dev? >> > > ftp://ftp.mcs.anl.gov/pub/petsc/externalpackages/parmetis-4.0.2-p1.tar.gz > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Mar 1 14:22:44 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 1 Mar 2012 14:22:44 -0600 Subject: [petsc-users] ccgraph.c error In-Reply-To: References: Message-ID: <5B68A5B3-C12C-42EC-A505-122E664CD763@mcs.anl.gov> On Mar 1, 2012, at 1:58 PM, John Fettig wrote: > On Thu, Mar 1, 2012 at 1:32 PM, Sean Farley wrote: > On Thu, Mar 1, 2012 at 12:09 PM, John Fettig wrote: > Sorry to dig up an old thread (I seem to have a tendency of doing that here!), but this problem exists in petsc-3.2-p6 with MUMPS and Metis as well. Looking at the differences between the dev version of mump.c and the release version, I don't see anything that would have obviously fixed this. Could you give me a hint about what was changed? Is it as simple as using the updated Metis package (in dev)? How difficult would it be to backport the fix to the released version? > > It'd be pretty murky to do this. Start by looking at this changeset: > > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/0c0bf07c7a32 > > Do I understand correctly that this was fixed mainly by using the latest Metis? Were there any patches to MUMPS that weren't directly related to the updated Metis? I'm just trying to understand what fixed the ccgraph.c problem. It doesn't seem to happen in 3.1-p8, although that version of MUMPS is a lot slower (not sure why). > > Related question, would it be a problem to just use petsc-dev? > > I could, but it's a little too unstable for use in production software. Using the released/patched version simplifies a lot of things on many levels. John, If you are using a different version of MUMPS or Metis than the petsc version was designed for then you are still using untested, unsupported code so you are not gaining anything. In fact you are in worse shape because we will not support that configuration, while we will help people with problems with petsc-dev and the latest MUMPS/Metis we support. Barry > > John > From sean at mcs.anl.gov Thu Mar 1 14:23:30 2012 From: sean at mcs.anl.gov (Sean Farley) Date: Thu, 1 Mar 2012 14:23:30 -0600 Subject: [petsc-users] ccgraph.c error In-Reply-To: References: Message-ID: > > Do I understand correctly that this was fixed mainly by using the latest > Metis? Were there any patches to MUMPS that weren't directly related to > the updated Metis? I'm just trying to understand what fixed the ccgraph.c > problem. It doesn't seem to happen in 3.1-p8, although that version of > MUMPS is a lot slower (not sure why). > I have no idea what's going on in ccgraph.c. I did have to patch a few places in PETSc but most of that was in sieve and Mat. The changes I made to metis, parmetis, mumps, superlu_dist, and suitesparse were only related to updating their interfaces to use the newest metis / parmetis for 64 bit compatibility. I could, but it's a little too unstable for use in production software. > Using the released/patched version simplifies a lot of things on many > levels. > Ok, but just a heads up that you could find a revision that works for you and stick to that one: hg clone http://petsc.cs.iit.edu/petsc/petsc/-dev -r d33f3ccece60 -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Thu Mar 1 15:52:58 2012 From: john.fettig at gmail.com (John Fettig) Date: Thu, 1 Mar 2012 16:52:58 -0500 Subject: [petsc-users] ccgraph.c error In-Reply-To: References: Message-ID: On Thu, Mar 1, 2012 at 3:23 PM, Sean Farley wrote: > Ok, but just a heads up that you could find a revision that works for you > and stick to that one: > > hg clone http://petsc.cs.iit.edu/petsc/petsc/-dev -r d33f3ccece60 > You and Barry make a compelling argument. I'll consider this option. Thanks! John -------------- next part -------------- An HTML attachment was scrubbed... URL: From liujuy at gmail.com Thu Mar 1 16:08:28 2012 From: liujuy at gmail.com (Ju LIU) Date: Thu, 1 Mar 2012 16:08:28 -0600 Subject: [petsc-users] how is PETSC_MAX_PATH_LEN defined? Message-ID: Hi: I am reading http://www.mcs.anl.gov/petsc/petsc-current/src/mat/examples/tutorials/ex9.c.html I am curious about the value of PETSC_MAX_PATH_LEN. Thanks, Ju -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Mar 1 16:14:35 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 1 Mar 2012 16:14:35 -0600 (CST) Subject: [petsc-users] how is PETSC_MAX_PATH_LEN defined? In-Reply-To: References: Message-ID: from include/petscsys.h >>>>>>>>> #if defined(PETSC_HAVE_LIMITS_H) #include #endif #if defined(PETSC_HAVE_SYS_PARAM_H) #include #endif #if defined(PETSC_HAVE_SYS_TYPES_H) #include #endif #if defined(MAXPATHLEN) # define PETSC_MAX_PATH_LEN MAXPATHLEN #elif defined(MAX_PATH) # define PETSC_MAX_PATH_LEN MAX_PATH #elif defined(_MAX_PATH) # define PETSC_MAX_PATH_LEN _MAX_PATH #else # define PETSC_MAX_PATH_LEN 4096 #endif <<<<<<<<<<< So usually the value is picked up from system include files via MAXPATHLEN etc - and if no value is found - it it set as 4096. Satish On Thu, 1 Mar 2012, Ju LIU wrote: > Hi: > > I am reading > http://www.mcs.anl.gov/petsc/petsc-current/src/mat/examples/tutorials/ex9.c.html > > I am curious about the value of PETSC_MAX_PATH_LEN. > > Thanks, > > Ju > From liujuy at gmail.com Thu Mar 1 16:15:34 2012 From: liujuy at gmail.com (Ju LIU) Date: Thu, 1 Mar 2012 16:15:34 -0600 Subject: [petsc-users] how is PETSC_MAX_PATH_LEN defined? In-Reply-To: References: Message-ID: All right. Thanks, Satish! Ju 2012/3/1 Satish Balay > from include/petscsys.h > > >>>>>>>>> > #if defined(PETSC_HAVE_LIMITS_H) > #include > #endif > #if defined(PETSC_HAVE_SYS_PARAM_H) > #include > #endif > #if defined(PETSC_HAVE_SYS_TYPES_H) > #include > #endif > #if defined(MAXPATHLEN) > # define PETSC_MAX_PATH_LEN MAXPATHLEN > #elif defined(MAX_PATH) > # define PETSC_MAX_PATH_LEN MAX_PATH > #elif defined(_MAX_PATH) > # define PETSC_MAX_PATH_LEN _MAX_PATH > #else > # define PETSC_MAX_PATH_LEN 4096 > #endif > <<<<<<<<<<< > > So usually the value is picked up from system include files via > MAXPATHLEN etc - and if no value is found - it it set as 4096. > > Satish > > On Thu, 1 Mar 2012, Ju LIU wrote: > > > Hi: > > > > I am reading > > > http://www.mcs.anl.gov/petsc/petsc-current/src/mat/examples/tutorials/ex9.c.html > > > > I am curious about the value of PETSC_MAX_PATH_LEN. > > > > Thanks, > > > > Ju > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From manfred.gratt at uibk.ac.at Fri Mar 2 04:18:21 2012 From: manfred.gratt at uibk.ac.at (Manfred Gratt) Date: Fri, 02 Mar 2012 11:18:21 +0100 Subject: [petsc-users] Problem with KSPSetOperators In-Reply-To: References: <4F4F3F2B.70306@uibk.ac.at> Message-ID: <4F509E6D.7030507@uibk.ac.at> An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Mar 2 06:23:56 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 2 Mar 2012 06:23:56 -0600 Subject: [petsc-users] Problem with KSPSetOperators In-Reply-To: <4F509E6D.7030507@uibk.ac.at> References: <4F4F3F2B.70306@uibk.ac.at> <4F509E6D.7030507@uibk.ac.at> Message-ID: On Fri, Mar 2, 2012 at 04:18, Manfred Gratt wrote: > thank you for your quick response. The petsc-dev did not solve the > segfault. I checked what the difference from ex5 to my code was and after I > changed my code to ex5 the segfault was gone when I used MatZeroEntries > instead of destroying and creating a new Matrix for the second solve. This > seems to be logical but, what I do not understand is why it works with > destroying and creating a new Matrix on more than one processor fine? I can't understand from this description exactly what works and what doesn't work for you. There shouldn't be a SEGV for any "reasonable" sequence of calls you make, so we should clarify the confusion and either fix the bug or make a better/earlier error message. Is there a way you can modify ex5 (or another example, or send your own code) to show the problem? -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Fri Mar 2 07:55:23 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Fri, 02 Mar 2012 14:55:23 +0100 Subject: [petsc-users] Using -ipo, -fast or -O3 during compilation of PETSc Message-ID: <4F50D14B.9060006@gmail.com> Hi, May I know if it is advisable to use -ipo, -fast or -O3 during compilation of PETSc for the non debug mode? Will it be faster or will there be unpredictable errors? Thanks! -- Yours sincerely, TAY wee-beng From jedbrown at mcs.anl.gov Fri Mar 2 07:58:22 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 2 Mar 2012 07:58:22 -0600 Subject: [petsc-users] Using -ipo, -fast or -O3 during compilation of PETSc In-Reply-To: <4F50D14B.9060006@gmail.com> References: <4F50D14B.9060006@gmail.com> Message-ID: On Fri, Mar 2, 2012 at 07:55, TAY wee-beng wrote: > May I know if it is advisable to use -ipo, -fast or -O3 during compilation > of PETSc for the non debug mode? > > Will it be faster or will there be unpredictable errors? > The only thing we can say for sure is that it will take longer to compile. (It may be faster or slower, probably not by much. There should not be errors.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From manfred.gratt at uibk.ac.at Fri Mar 2 08:25:52 2012 From: manfred.gratt at uibk.ac.at (Manfred Gratt) Date: Fri, 02 Mar 2012 15:25:52 +0100 Subject: [petsc-users] Problem with KSPSetOperators In-Reply-To: References: <4F4F3F2B.70306@uibk.ac.at> <4F509E6D.7030507@uibk.ac.at> Message-ID: <4F50D870.3010502@uibk.ac.at> An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex5.c Type: text/x-csrc Size: 12505 bytes Desc: not available URL: From mike.hui.zhang at hotmail.com Fri Mar 2 08:47:33 2012 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Fri, 2 Mar 2012 15:47:33 +0100 Subject: [petsc-users] error of AODestroy and ISLocalToGlobalMappingDestroy Message-ID: Hi, I got errors on a linux cluster from AODestroy and ISLocalToGlobalMappingDestroy with the petsc-dev hg version 882fc279ddfaa44915c56a5b227514ada881775d. And if I run the following test program with number of processes > 3, it hangs on. The linux is using mpich2. #include char help[]= "test ao destroy\n"; int main(int argc, char **argv) { AO ao; PetscInitialize(&argc,&argv,PETSC_NULL,help); AOCreate(PETSC_COMM_WORLD,&ao); AODestroy(&ao); PetscFinalize(); return 0; } I also run the same program with petsc-3.2 on my Mac, if I use mpirun it is ok but if I use mpiexec it gives the same error. The mac is using openmpi. Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike.hui.zhang at hotmail.com Fri Mar 2 08:55:00 2012 From: mike.hui.zhang at hotmail.com (Hui Zhang) Date: Fri, 2 Mar 2012 15:55:00 +0100 Subject: [petsc-users] error of AODestroy and ISLocalToGlobalMappingDestroy In-Reply-To: References: Message-ID: > Hi, > > I got errors on a linux cluster from AODestroy and ISLocalToGlobalMappingDestroy with > the petsc-dev hg version 882fc279ddfaa44915c56a5b227514ada881775d. And if I run the > following test program with number of processes > 3, it hangs on. The linux is using mpich2. > > #include > > char help[]= "test ao destroy\n"; > > int main(int argc, char **argv) > { > AO ao; > PetscInitialize(&argc,&argv,PETSC_NULL,help); > AOCreate(PETSC_COMM_WORLD,&ao); > AODestroy(&ao); > > PetscFinalize(); > return 0; > } > Sorry, I just have changed the order in the hosts file and then it runs ok and will not hang. So it must be some place wrong in my actual computing programs. I will check again. > I also run the same program with petsc-3.2 on my Mac, if I use mpirun it is ok > but if I use mpiexec it gives the same error. The mac is using openmpi. > > Thank you! > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Mar 2 09:08:49 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 2 Mar 2012 09:08:49 -0600 (CST) Subject: [petsc-users] Using -ipo, -fast or -O3 during compilation of PETSc In-Reply-To: References: <4F50D14B.9060006@gmail.com> Message-ID: On Fri, 2 Mar 2012, Jed Brown wrote: > On Fri, Mar 2, 2012 at 07:55, TAY wee-beng wrote: > > > May I know if it is advisable to use -ipo, -fast or -O3 during compilation > > of PETSc for the non debug mode? > > > > Will it be faster or will there be unpredictable errors? > > > > The only thing we can say for sure is that it will take longer to compile. > > (It may be faster or slower, probably not by much. There should not be > errors.) The best way to see if it helps is to do a separate build of PETSc with these options - and compare -log_summary for runs with and without this option.. Note: Some compilers misbehave with '-fast' type options.. [not returing error codes - when there are compile errors]. When this happens - configure won't work properly.. Satish From hzhang at mcs.anl.gov Fri Mar 2 09:13:07 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Fri, 2 Mar 2012 09:13:07 -0600 Subject: [petsc-users] Problem with KSPSetOperators In-Reply-To: <4F50D870.3010502@uibk.ac.at> References: <4F4F3F2B.70306@uibk.ac.at> <4F509E6D.7030507@uibk.ac.at> <4F50D870.3010502@uibk.ac.at> Message-ID: Manfred : Thanks for sending the modified code. I can repeat the error in 'MatFactorNumeric_MUMPS'. Running it with '-mat_type aij' works well, likely there is a bug in our code. I'll work on it and get back to you about the fix. Meanwhile, you may use aij format instead of sbaij. Hong ** > Jed Brown schrieb: > > On Fri, Mar 2, 2012 at 04:18, Manfred Gratt wrote: > >> thank you for your quick response. The petsc-dev did not solve the >> segfault. I checked what the difference from ex5 to my code was and after I >> changed my code to ex5 the segfault was gone when I used MatZeroEntries >> instead of destroying and creating a new Matrix for the second solve. This >> seems to be logical but, what I do not understand is why it works with >> destroying and creating a new Matrix on more than one processor fine? > > > I can't understand from this description exactly what works and what > doesn't work for you. There shouldn't be a SEGV for any "reasonable" > sequence of calls you make, so we should clarify the confusion and either > fix the bug or make a better/earlier error message. > > Is there a way you can modify ex5 (or another example, or send your own > code) to show the problem? > > I tried to change the ex5 to bring the same error but I am not able to get > the same error. I still get an error although it is an error I don't > understand. > I changed ex5 to calculate in second solve the same matrix as in the first > solve. As written in definition of SAME_NONZERO_PATTERN it should work. > When before the second solve the matrix C is set to zero with > MatZeroEntries it works well, but when I instead destroy the matrix C and > create a new C it does not work with the error that it is an Numerically > singular matrix. Should that happen? It is the same matrix as in the first > solve. So shouldnt it happen there? > When I use instead DIFFERENT_NONZERO_PATTERN it also works well in the > second solve. > > I modified the ex5 so that you can easily call the changes with > -mat_zeroent to call MatZeroEntries instead of destroy and create a new > matrix. > The second option -mat_different calls DIFFERENT_NONZERO_PATTERN instead > of SAME_NONZERO_PATTERN. > > Without these options it throws an error when you call it with > > ./ex5 -ksp_monitor -mat_type sbaij -pc_type cholesky > -pc_factor_mat_solver_package mumps > > Thanks > Manfred > > newex5.c: > > static char help[] = "Solves two linear systems in parallel with KSP. The > code\n\ > illustrates repeated solution of linear systems with the same > preconditioner\n\ > method but different matrices (having the same nonzero structure). The > code\n\ > also uses multiple profiling stages. Input arguments are\n\ > -m : problem size\n\ > -mat_nonsym : use nonsymmetric matrix (default is symmetric)\n\n"; > > /*T > Concepts: KSP^repeatedly solving linear systems; > Concepts: PetscLog^profiling multiple stages of code; > Processors: n > T*/ > > /* > Include "petscksp.h" so that we can use KSP solvers. Note that this file > automatically includes: > petscsys.h - base PETSc routines petscvec.h - vectors > petscmat.h - matrices > petscis.h - index sets petscksp.h - Krylov subspace > methods > petscviewer.h - viewers petscpc.h - preconditioners > */ > #include > > #undef __FUNCT__ > #define __FUNCT__ "main" > int main(int argc,char **args) > { > KSP ksp; /* linear solver context */ > Mat C; /* matrix */ > Vec x,u,b; /* approx solution, RHS, exact solution > */ > PetscReal norm; /* norm of solution error */ > PetscScalar v,none = -1.0; > PetscInt Ii,J,ldim,low,high,iglobal,Istart,Iend; > PetscErrorCode ierr; > PetscInt i,j,m = 3,n = 2,its; > PetscMPIInt size,rank; > PetscBool mat_nonsymmetric = PETSC_FALSE; > PetscBool matzeroent = PETSC_FALSE, different_pattern = PETSC_FALSE; > #if defined (PETSC_USE_LOG) > PetscLogStage stages[2]; > #endif > > PetscInitialize(&argc,&args,(char *)0,help); > ierr = PetscOptionsGetInt(PETSC_NULL,"-m",&m,PETSC_NULL);CHKERRQ(ierr); > ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); > ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size);CHKERRQ(ierr); > n = 2*size; > > /* > Set flag if we are doing a nonsymmetric problem; the default is > symmetric. > */ > ierr = > PetscOptionsGetBool(PETSC_NULL,"-mat_nonsym",&mat_nonsymmetric,PETSC_NULL);CHKERRQ(ierr); > ierr = > PetscOptionsGetBool(PETSC_NULL,"-mat_zeroent",&matzeroent,PETSC_NULL);CHKERRQ(ierr); > ierr = > PetscOptionsGetBool(PETSC_NULL,"-mat_different",&different_pattern,PETSC_NULL);CHKERRQ(ierr); > /* > Register two stages for separate profiling of the two linear solves. > Use the runtime option -log_summary for a printout of performance > statistics at the program's conlusion. > */ > ierr = PetscLogStageRegister("Original Solve",&stages[0]);CHKERRQ(ierr); > ierr = PetscLogStageRegister("Second Solve",&stages[1]);CHKERRQ(ierr); > > /* -------------- Stage 0: Solve Original System ---------------------- > */ > /* > Indicate to PETSc profiling that we're beginning the first stage > */ > ierr = PetscLogStagePush(stages[0]);CHKERRQ(ierr); > > /* > Create parallel matrix, specifying only its global dimensions. > When using MatCreate(), the matrix format can be specified at > runtime. Also, the parallel partitioning of the matrix is > determined by PETSc at runtime. > */ > ierr = MatCreate(PETSC_COMM_WORLD,&C);CHKERRQ(ierr); > ierr = MatSetSizes(C,PETSC_DECIDE,PETSC_DECIDE,m*n,m*n);CHKERRQ(ierr); > ierr = MatSetFromOptions(C);CHKERRQ(ierr); > ierr = MatSetUp(C);CHKERRQ(ierr); > > /* > Currently, all PETSc parallel matrix formats are partitioned by > contiguous chunks of rows across the processors. Determine which > rows of the matrix are locally owned. > */ > ierr = MatGetOwnershipRange(C,&Istart,&Iend);CHKERRQ(ierr); > > /* > Set matrix entries matrix in parallel. > - Each processor needs to insert only elements that it owns > locally (but any non-local elements will be sent to the > appropriate processor during matrix assembly). > - Always specify global row and columns of matrix entries. > */ > for (Ii=Istart; Ii v = -1.0; i = Ii/n; j = Ii - i*n; > if (i>0) {J = Ii - n; ierr = > MatSetValues(C,1,&Ii,1,&J,&v,ADD_VALUES);CHKERRQ(ierr);} > if (i MatSetValues(C,1,&Ii,1,&J,&v,ADD_VALUES);CHKERRQ(ierr);} > if (j>0) {J = Ii - 1; ierr = > MatSetValues(C,1,&Ii,1,&J,&v,ADD_VALUES);CHKERRQ(ierr);} > if (j MatSetValues(C,1,&Ii,1,&J,&v,ADD_VALUES);CHKERRQ(ierr);} > v = 4.0; ierr = MatSetValues(C,1,&Ii,1,&Ii,&v,ADD_VALUES); > } > > /* > Make the matrix nonsymmetric if desired > */ > if (mat_nonsymmetric) { > for (Ii=Istart; Ii v = -1.5; i = Ii/n; > if (i>1) {J = Ii-n-1; ierr = > MatSetValues(C,1,&Ii,1,&J,&v,ADD_VALUES);CHKERRQ(ierr);} > } > } else { > ierr = MatSetOption(C,MAT_SYMMETRIC,PETSC_TRUE);CHKERRQ(ierr); > ierr = MatSetOption(C,MAT_SYMMETRY_ETERNAL,PETSC_TRUE);CHKERRQ(ierr); > } > > /* > Assemble matrix, using the 2-step process: > MatAssemblyBegin(), MatAssemblyEnd() > Computations can be done while messages are in transition > by placing code between these two statements. > */ > ierr = MatAssemblyBegin(C,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(C,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > /* > Create parallel vectors. > - When using VecSetSizes(), we specify only the vector's global > dimension; the parallel partitioning is determined at runtime. > - Note: We form 1 vector from scratch and then duplicate as needed. > */ > ierr = VecCreate(PETSC_COMM_WORLD,&u);CHKERRQ(ierr); > ierr = VecSetSizes(u,PETSC_DECIDE,m*n);CHKERRQ(ierr); > ierr = VecSetFromOptions(u);CHKERRQ(ierr); > ierr = VecDuplicate(u,&b);CHKERRQ(ierr); > ierr = VecDuplicate(b,&x);CHKERRQ(ierr); > > /* > Currently, all parallel PETSc vectors are partitioned by > contiguous chunks across the processors. Determine which > range of entries are locally owned. > */ > ierr = VecGetOwnershipRange(x,&low,&high);CHKERRQ(ierr); > > /* > Set elements within the exact solution vector in parallel. > - Each processor needs to insert only elements that it owns > locally (but any non-local entries will be sent to the > appropriate processor during vector assembly). > - Always specify global locations of vector entries. > */ > ierr = VecGetLocalSize(x,&ldim);CHKERRQ(ierr); > for (i=0; i iglobal = i + low; > v = (PetscScalar)(i + 100*rank); > ierr = VecSetValues(u,1,&iglobal,&v,INSERT_VALUES);CHKERRQ(ierr); > } > > /* > Assemble vector, using the 2-step process: > VecAssemblyBegin(), VecAssemblyEnd() > Computations can be done while messages are in transition, > by placing code between these two statements. > */ > ierr = VecAssemblyBegin(u);CHKERRQ(ierr); > ierr = VecAssemblyEnd(u);CHKERRQ(ierr); > > /* > Compute right-hand-side vector > */ > ierr = MatMult(C,u,b);CHKERRQ(ierr); > > /* > Create linear solver context > */ > ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr); > > /* > Set operators. Here the matrix that defines the linear system > also serves as the preconditioning matrix. > */ > if(different_pattern) { > ierr = > KSPSetOperators(ksp,C,C,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); > }else { > ierr = KSPSetOperators(ksp,C,C,SAME_NONZERO_PATTERN);CHKERRQ(ierr); > } > /* > Set runtime options (e.g., -ksp_type -pc_type ) > */ > > ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr); > > /* > Solve linear system. Here we explicitly call KSPSetUp() for more > detailed performance monitoring of certain preconditioners, such > as ICC and ILU. This call is optional, as KSPSetUp() will > automatically be called within KSPSolve() if it hasn't been > called already. > */ > ierr = KSPSetUp(ksp);CHKERRQ(ierr); > ierr = KSPSolve(ksp,b,x);CHKERRQ(ierr); > > /* > Check the error > */ > // ierr = VecAXPY(x,none,u);CHKERRQ(ierr); > // ierr = VecNorm(x,NORM_2,&norm);CHKERRQ(ierr); > // ierr = KSPGetIterationNumber(ksp,&its);CHKERRQ(ierr); > // if (norm > 1.e-13){ > // ierr = PetscPrintf(PETSC_COMM_WORLD,"Norm of error %G, Iterations > %D\n",norm,its);CHKERRQ(ierr); > // } > > /* -------------- Stage 1: Solve Second System ---------------------- */ > /* > Solve another linear system with the same method. We reuse the KSP > context, matrix and vector data structures, and hence save the > overhead of creating new ones. > > Indicate to PETSc profiling that we're concluding the first > stage with PetscLogStagePop(), and beginning the second stage with > PetscLogStagePush(). > */ > ierr = PetscLogStagePop();CHKERRQ(ierr); > ierr = PetscLogStagePush(stages[1]);CHKERRQ(ierr); > > /* > Initialize all matrix entries to zero. MatZeroEntries() retains the > nonzero structure of the matrix for sparse formats. > */ > if(matzeroent) { > ierr = MatZeroEntries(C);CHKERRQ(ierr); > } else { > ierr = MatDestroy(&C); > > ierr = MatCreate(PETSC_COMM_WORLD,&C);CHKERRQ(ierr); > ierr = MatSetSizes(C,PETSC_DECIDE,PETSC_DECIDE,m*n,m*n);CHKERRQ(ierr); > ierr = MatSetFromOptions(C);CHKERRQ(ierr); > ierr = MatSetUp(C);CHKERRQ(ierr); > } > /* > Assemble matrix again. Note that we retain the same matrix data > structure and the same nonzero pattern; we just change the values > of the matrix entries. > */ > for (Ii=Istart; Ii v = -1.0; i = Ii/n; j = Ii - i*n; > if (i>0) {J = Ii - n; ierr = > MatSetValues(C,1,&Ii,1,&J,&v,ADD_VALUES);CHKERRQ(ierr);} > if (i MatSetValues(C,1,&Ii,1,&J,&v,ADD_VALUES);CHKERRQ(ierr);} > if (j>0) {J = Ii - 1; ierr = > MatSetValues(C,1,&Ii,1,&J,&v,ADD_VALUES);CHKERRQ(ierr);} > if (j MatSetValues(C,1,&Ii,1,&J,&v,ADD_VALUES);CHKERRQ(ierr);} > v = 4.0; ierr = MatSetValues(C,1,&Ii,1,&Ii,&v,ADD_VALUES); > } > if (mat_nonsymmetric) { > for (Ii=Istart; Ii v = -1.5; i = Ii/n; > if (i>1) {J = Ii-n-1; ierr = > MatSetValues(C,1,&Ii,1,&J,&v,ADD_VALUES);CHKERRQ(ierr);} > } > } else { > ierr = MatSetOption(C,MAT_SYMMETRIC,PETSC_TRUE);CHKERRQ(ierr); > ierr = MatSetOption(C,MAT_SYMMETRY_ETERNAL,PETSC_TRUE);CHKERRQ(ierr); > } > ierr = MatAssemblyBegin(C,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(C,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > /* > Compute another right-hand-side vector > */ > ierr = MatMult(C,u,b);CHKERRQ(ierr); > > /* > Set operators. Here the matrix that defines the linear system > also serves as the preconditioning matrix. > - The flag SAME_NONZERO_PATTERN indicates that the > preconditioning matrix has identical nonzero structure > as during the last linear solve (although the values of > the entries have changed). Thus, we can save some > work in setting up the preconditioner (e.g., no need to > redo symbolic factorization for ILU/ICC preconditioners). > - If the nonzero structure of the matrix is different during > the second linear solve, then the flag DIFFERENT_NONZERO_PATTERN > must be used instead. If you are unsure whether the > matrix structure has changed or not, use the flag > DIFFERENT_NONZERO_PATTERN. > - Caution: If you specify SAME_NONZERO_PATTERN, PETSc > believes your assertion and does not check the structure > of the matrix. If you erroneously claim that the structure > is the same when it actually is not, the new preconditioner > will not function correctly. Thus, use this optimization > feature with caution! > */ > > if(different_pattern) { > ierr = > KSPSetOperators(ksp,C,C,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); > }else { > ierr = KSPSetOperators(ksp,C,C,SAME_NONZERO_PATTERN);CHKERRQ(ierr); > } > > /* > Solve linear system > */ > ierr = KSPSetUp(ksp);CHKERRQ(ierr); > ierr = KSPSolve(ksp,b,x);CHKERRQ(ierr); > > /* > Check the error > */ > ierr = VecAXPY(x,none,u);CHKERRQ(ierr); > ierr = VecNorm(x,NORM_2,&norm);CHKERRQ(ierr); > ierr = KSPGetIterationNumber(ksp,&its);CHKERRQ(ierr); > if (norm > 1.e-4){ > ierr = PetscPrintf(PETSC_COMM_WORLD,"Norm of error %G, Iterations > %D\n",norm,its);CHKERRQ(ierr); > } > > /* > Free work space. All PETSc objects should be destroyed when they > are no longer needed. > */ > ierr = KSPDestroy(&ksp);CHKERRQ(ierr); > ierr = VecDestroy(&u);CHKERRQ(ierr); > ierr = VecDestroy(&x);CHKERRQ(ierr); > ierr = VecDestroy(&b);CHKERRQ(ierr); > ierr = MatDestroy(&C);CHKERRQ(ierr); > > /* > Indicate to PETSc profiling that we're concluding the second stage > */ > ierr = PetscLogStagePop();CHKERRQ(ierr); > > ierr = PetscFinalize(); > return 0; > } > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Fri Mar 2 11:31:05 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Fri, 2 Mar 2012 11:31:05 -0600 Subject: [petsc-users] Problem with KSPSetOperators In-Reply-To: <4F50D870.3010502@uibk.ac.at> References: <4F4F3F2B.70306@uibk.ac.at> <4F509E6D.7030507@uibk.ac.at> <4F50D870.3010502@uibk.ac.at> Message-ID: Manfred : Bug is fixed in petsc-3.2 and petsc-dev: http://petsc.cs.iit.edu/petsc/petsc-dev/rev/759927945bb3 You can update your petsc library and check your code. Let us know if you still have problem. Thanks for the bug report which helps improve petsc! Hong ** > Jed Brown schrieb: > > On Fri, Mar 2, 2012 at 04:18, Manfred Gratt wrote: > >> thank you for your quick response. The petsc-dev did not solve the >> segfault. I checked what the difference from ex5 to my code was and after I >> changed my code to ex5 the segfault was gone when I used MatZeroEntries >> instead of destroying and creating a new Matrix for the second solve. This >> seems to be logical but, what I do not understand is why it works with >> destroying and creating a new Matrix on more than one processor fine? > > > I can't understand from this description exactly what works and what > doesn't work for you. There shouldn't be a SEGV for any "reasonable" > sequence of calls you make, so we should clarify the confusion and either > fix the bug or make a better/earlier error message. > > Is there a way you can modify ex5 (or another example, or send your own > code) to show the problem? > > I tried to change the ex5 to bring the same error but I am not able to get > the same error. I still get an error although it is an error I don't > understand. > I changed ex5 to calculate in second solve the same matrix as in the first > solve. As written in definition of SAME_NONZERO_PATTERN it should work. > When before the second solve the matrix C is set to zero with > MatZeroEntries it works well, but when I instead destroy the matrix C and > create a new C it does not work with the error that it is an Numerically > singular matrix. Should that happen? It is the same matrix as in the first > solve. So shouldnt it happen there? > When I use instead DIFFERENT_NONZERO_PATTERN it also works well in the > second solve. > > I modified the ex5 so that you can easily call the changes with > -mat_zeroent to call MatZeroEntries instead of destroy and create a new > matrix. > The second option -mat_different calls DIFFERENT_NONZERO_PATTERN instead > of SAME_NONZERO_PATTERN. > > Without these options it throws an error when you call it with > > ./ex5 -ksp_monitor -mat_type sbaij -pc_type cholesky > -pc_factor_mat_solver_package mumps > > Thanks > Manfred > > newex5.c: > > static char help[] = "Solves two linear systems in parallel with KSP. The > code\n\ > illustrates repeated solution of linear systems with the same > preconditioner\n\ > method but different matrices (having the same nonzero structure). The > code\n\ > also uses multiple profiling stages. Input arguments are\n\ > -m : problem size\n\ > -mat_nonsym : use nonsymmetric matrix (default is symmetric)\n\n"; > > /*T > Concepts: KSP^repeatedly solving linear systems; > Concepts: PetscLog^profiling multiple stages of code; > Processors: n > T*/ > > /* > Include "petscksp.h" so that we can use KSP solvers. Note that this file > automatically includes: > petscsys.h - base PETSc routines petscvec.h - vectors > petscmat.h - matrices > petscis.h - index sets petscksp.h - Krylov subspace > methods > petscviewer.h - viewers petscpc.h - preconditioners > */ > #include > > #undef __FUNCT__ > #define __FUNCT__ "main" > int main(int argc,char **args) > { > KSP ksp; /* linear solver context */ > Mat C; /* matrix */ > Vec x,u,b; /* approx solution, RHS, exact solution > */ > PetscReal norm; /* norm of solution error */ > PetscScalar v,none = -1.0; > PetscInt Ii,J,ldim,low,high,iglobal,Istart,Iend; > PetscErrorCode ierr; > PetscInt i,j,m = 3,n = 2,its; > PetscMPIInt size,rank; > PetscBool mat_nonsymmetric = PETSC_FALSE; > PetscBool matzeroent = PETSC_FALSE, different_pattern = PETSC_FALSE; > #if defined (PETSC_USE_LOG) > PetscLogStage stages[2]; > #endif > > PetscInitialize(&argc,&args,(char *)0,help); > ierr = PetscOptionsGetInt(PETSC_NULL,"-m",&m,PETSC_NULL);CHKERRQ(ierr); > ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); > ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size);CHKERRQ(ierr); > n = 2*size; > > /* > Set flag if we are doing a nonsymmetric problem; the default is > symmetric. > */ > ierr = > PetscOptionsGetBool(PETSC_NULL,"-mat_nonsym",&mat_nonsymmetric,PETSC_NULL);CHKERRQ(ierr); > ierr = > PetscOptionsGetBool(PETSC_NULL,"-mat_zeroent",&matzeroent,PETSC_NULL);CHKERRQ(ierr); > ierr = > PetscOptionsGetBool(PETSC_NULL,"-mat_different",&different_pattern,PETSC_NULL);CHKERRQ(ierr); > /* > Register two stages for separate profiling of the two linear solves. > Use the runtime option -log_summary for a printout of performance > statistics at the program's conlusion. > */ > ierr = PetscLogStageRegister("Original Solve",&stages[0]);CHKERRQ(ierr); > ierr = PetscLogStageRegister("Second Solve",&stages[1]);CHKERRQ(ierr); > > /* -------------- Stage 0: Solve Original System ---------------------- > */ > /* > Indicate to PETSc profiling that we're beginning the first stage > */ > ierr = PetscLogStagePush(stages[0]);CHKERRQ(ierr); > > /* > Create parallel matrix, specifying only its global dimensions. > When using MatCreate(), the matrix format can be specified at > runtime. Also, the parallel partitioning of the matrix is > determined by PETSc at runtime. > */ > ierr = MatCreate(PETSC_COMM_WORLD,&C);CHKERRQ(ierr); > ierr = MatSetSizes(C,PETSC_DECIDE,PETSC_DECIDE,m*n,m*n);CHKERRQ(ierr); > ierr = MatSetFromOptions(C);CHKERRQ(ierr); > ierr = MatSetUp(C);CHKERRQ(ierr); > > /* > Currently, all PETSc parallel matrix formats are partitioned by > contiguous chunks of rows across the processors. Determine which > rows of the matrix are locally owned. > */ > ierr = MatGetOwnershipRange(C,&Istart,&Iend);CHKERRQ(ierr); > > /* > Set matrix entries matrix in parallel. > - Each processor needs to insert only elements that it owns > locally (but any non-local elements will be sent to the > appropriate processor during matrix assembly). > - Always specify global row and columns of matrix entries. > */ > for (Ii=Istart; Ii v = -1.0; i = Ii/n; j = Ii - i*n; > if (i>0) {J = Ii - n; ierr = > MatSetValues(C,1,&Ii,1,&J,&v,ADD_VALUES);CHKERRQ(ierr);} > if (i MatSetValues(C,1,&Ii,1,&J,&v,ADD_VALUES);CHKERRQ(ierr);} > if (j>0) {J = Ii - 1; ierr = > MatSetValues(C,1,&Ii,1,&J,&v,ADD_VALUES);CHKERRQ(ierr);} > if (j MatSetValues(C,1,&Ii,1,&J,&v,ADD_VALUES);CHKERRQ(ierr);} > v = 4.0; ierr = MatSetValues(C,1,&Ii,1,&Ii,&v,ADD_VALUES); > } > > /* > Make the matrix nonsymmetric if desired > */ > if (mat_nonsymmetric) { > for (Ii=Istart; Ii v = -1.5; i = Ii/n; > if (i>1) {J = Ii-n-1; ierr = > MatSetValues(C,1,&Ii,1,&J,&v,ADD_VALUES);CHKERRQ(ierr);} > } > } else { > ierr = MatSetOption(C,MAT_SYMMETRIC,PETSC_TRUE);CHKERRQ(ierr); > ierr = MatSetOption(C,MAT_SYMMETRY_ETERNAL,PETSC_TRUE);CHKERRQ(ierr); > } > > /* > Assemble matrix, using the 2-step process: > MatAssemblyBegin(), MatAssemblyEnd() > Computations can be done while messages are in transition > by placing code between these two statements. > */ > ierr = MatAssemblyBegin(C,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(C,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > /* > Create parallel vectors. > - When using VecSetSizes(), we specify only the vector's global > dimension; the parallel partitioning is determined at runtime. > - Note: We form 1 vector from scratch and then duplicate as needed. > */ > ierr = VecCreate(PETSC_COMM_WORLD,&u);CHKERRQ(ierr); > ierr = VecSetSizes(u,PETSC_DECIDE,m*n);CHKERRQ(ierr); > ierr = VecSetFromOptions(u);CHKERRQ(ierr); > ierr = VecDuplicate(u,&b);CHKERRQ(ierr); > ierr = VecDuplicate(b,&x);CHKERRQ(ierr); > > /* > Currently, all parallel PETSc vectors are partitioned by > contiguous chunks across the processors. Determine which > range of entries are locally owned. > */ > ierr = VecGetOwnershipRange(x,&low,&high);CHKERRQ(ierr); > > /* > Set elements within the exact solution vector in parallel. > - Each processor needs to insert only elements that it owns > locally (but any non-local entries will be sent to the > appropriate processor during vector assembly). > - Always specify global locations of vector entries. > */ > ierr = VecGetLocalSize(x,&ldim);CHKERRQ(ierr); > for (i=0; i iglobal = i + low; > v = (PetscScalar)(i + 100*rank); > ierr = VecSetValues(u,1,&iglobal,&v,INSERT_VALUES);CHKERRQ(ierr); > } > > /* > Assemble vector, using the 2-step process: > VecAssemblyBegin(), VecAssemblyEnd() > Computations can be done while messages are in transition, > by placing code between these two statements. > */ > ierr = VecAssemblyBegin(u);CHKERRQ(ierr); > ierr = VecAssemblyEnd(u);CHKERRQ(ierr); > > /* > Compute right-hand-side vector > */ > ierr = MatMult(C,u,b);CHKERRQ(ierr); > > /* > Create linear solver context > */ > ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr); > > /* > Set operators. Here the matrix that defines the linear system > also serves as the preconditioning matrix. > */ > if(different_pattern) { > ierr = > KSPSetOperators(ksp,C,C,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); > }else { > ierr = KSPSetOperators(ksp,C,C,SAME_NONZERO_PATTERN);CHKERRQ(ierr); > } > /* > Set runtime options (e.g., -ksp_type -pc_type ) > */ > > ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr); > > /* > Solve linear system. Here we explicitly call KSPSetUp() for more > detailed performance monitoring of certain preconditioners, such > as ICC and ILU. This call is optional, as KSPSetUp() will > automatically be called within KSPSolve() if it hasn't been > called already. > */ > ierr = KSPSetUp(ksp);CHKERRQ(ierr); > ierr = KSPSolve(ksp,b,x);CHKERRQ(ierr); > > /* > Check the error > */ > // ierr = VecAXPY(x,none,u);CHKERRQ(ierr); > // ierr = VecNorm(x,NORM_2,&norm);CHKERRQ(ierr); > // ierr = KSPGetIterationNumber(ksp,&its);CHKERRQ(ierr); > // if (norm > 1.e-13){ > // ierr = PetscPrintf(PETSC_COMM_WORLD,"Norm of error %G, Iterations > %D\n",norm,its);CHKERRQ(ierr); > // } > > /* -------------- Stage 1: Solve Second System ---------------------- */ > /* > Solve another linear system with the same method. We reuse the KSP > context, matrix and vector data structures, and hence save the > overhead of creating new ones. > > Indicate to PETSc profiling that we're concluding the first > stage with PetscLogStagePop(), and beginning the second stage with > PetscLogStagePush(). > */ > ierr = PetscLogStagePop();CHKERRQ(ierr); > ierr = PetscLogStagePush(stages[1]);CHKERRQ(ierr); > > /* > Initialize all matrix entries to zero. MatZeroEntries() retains the > nonzero structure of the matrix for sparse formats. > */ > if(matzeroent) { > ierr = MatZeroEntries(C);CHKERRQ(ierr); > } else { > ierr = MatDestroy(&C); > > ierr = MatCreate(PETSC_COMM_WORLD,&C);CHKERRQ(ierr); > ierr = MatSetSizes(C,PETSC_DECIDE,PETSC_DECIDE,m*n,m*n);CHKERRQ(ierr); > ierr = MatSetFromOptions(C);CHKERRQ(ierr); > ierr = MatSetUp(C);CHKERRQ(ierr); > } > /* > Assemble matrix again. Note that we retain the same matrix data > structure and the same nonzero pattern; we just change the values > of the matrix entries. > */ > for (Ii=Istart; Ii v = -1.0; i = Ii/n; j = Ii - i*n; > if (i>0) {J = Ii - n; ierr = > MatSetValues(C,1,&Ii,1,&J,&v,ADD_VALUES);CHKERRQ(ierr);} > if (i MatSetValues(C,1,&Ii,1,&J,&v,ADD_VALUES);CHKERRQ(ierr);} > if (j>0) {J = Ii - 1; ierr = > MatSetValues(C,1,&Ii,1,&J,&v,ADD_VALUES);CHKERRQ(ierr);} > if (j MatSetValues(C,1,&Ii,1,&J,&v,ADD_VALUES);CHKERRQ(ierr);} > v = 4.0; ierr = MatSetValues(C,1,&Ii,1,&Ii,&v,ADD_VALUES); > } > if (mat_nonsymmetric) { > for (Ii=Istart; Ii v = -1.5; i = Ii/n; > if (i>1) {J = Ii-n-1; ierr = > MatSetValues(C,1,&Ii,1,&J,&v,ADD_VALUES);CHKERRQ(ierr);} > } > } else { > ierr = MatSetOption(C,MAT_SYMMETRIC,PETSC_TRUE);CHKERRQ(ierr); > ierr = MatSetOption(C,MAT_SYMMETRY_ETERNAL,PETSC_TRUE);CHKERRQ(ierr); > } > ierr = MatAssemblyBegin(C,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(C,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > /* > Compute another right-hand-side vector > */ > ierr = MatMult(C,u,b);CHKERRQ(ierr); > > /* > Set operators. Here the matrix that defines the linear system > also serves as the preconditioning matrix. > - The flag SAME_NONZERO_PATTERN indicates that the > preconditioning matrix has identical nonzero structure > as during the last linear solve (although the values of > the entries have changed). Thus, we can save some > work in setting up the preconditioner (e.g., no need to > redo symbolic factorization for ILU/ICC preconditioners). > - If the nonzero structure of the matrix is different during > the second linear solve, then the flag DIFFERENT_NONZERO_PATTERN > must be used instead. If you are unsure whether the > matrix structure has changed or not, use the flag > DIFFERENT_NONZERO_PATTERN. > - Caution: If you specify SAME_NONZERO_PATTERN, PETSc > believes your assertion and does not check the structure > of the matrix. If you erroneously claim that the structure > is the same when it actually is not, the new preconditioner > will not function correctly. Thus, use this optimization > feature with caution! > */ > > if(different_pattern) { > ierr = > KSPSetOperators(ksp,C,C,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); > }else { > ierr = KSPSetOperators(ksp,C,C,SAME_NONZERO_PATTERN);CHKERRQ(ierr); > } > > /* > Solve linear system > */ > ierr = KSPSetUp(ksp);CHKERRQ(ierr); > ierr = KSPSolve(ksp,b,x);CHKERRQ(ierr); > > /* > Check the error > */ > ierr = VecAXPY(x,none,u);CHKERRQ(ierr); > ierr = VecNorm(x,NORM_2,&norm);CHKERRQ(ierr); > ierr = KSPGetIterationNumber(ksp,&its);CHKERRQ(ierr); > if (norm > 1.e-4){ > ierr = PetscPrintf(PETSC_COMM_WORLD,"Norm of error %G, Iterations > %D\n",norm,its);CHKERRQ(ierr); > } > > /* > Free work space. All PETSc objects should be destroyed when they > are no longer needed. > */ > ierr = KSPDestroy(&ksp);CHKERRQ(ierr); > ierr = VecDestroy(&u);CHKERRQ(ierr); > ierr = VecDestroy(&x);CHKERRQ(ierr); > ierr = VecDestroy(&b);CHKERRQ(ierr); > ierr = MatDestroy(&C);CHKERRQ(ierr); > > /* > Indicate to PETSc profiling that we're concluding the second stage > */ > ierr = PetscLogStagePop();CHKERRQ(ierr); > > ierr = PetscFinalize(); > return 0; > } > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guanglei.xiong.at.siemens at gmail.com Fri Mar 2 16:58:41 2012 From: guanglei.xiong.at.siemens at gmail.com (Guanglei Xiong) Date: Fri, 2 Mar 2012 17:58:41 -0500 Subject: [petsc-users] Siemens internship opportunity in computational solid mechanics Message-ID: Hi all, Please forgive me to post a job advertisement. I hope this is an opportunity for some people in this community. Siemens Corporate Research (SCR), a division of Siemens Corporation, located in Princeton, New Jersey (USA) is Siemens' largest research center outside of Europe. SCR excels in transferring academic research into innovative products, and works in close collaboration with clinical and industrial partners. With more than 250 scientists, engineers, and technical experts, the company helps its customers and strategic partners grow their businesses in the fields of healthcare, automation, production, energy, industry, information and communications. SCR has an immediate student internship opening in the area of computational solid mechanics with a focus on cardiovascular and orthopedic applications. A PhD or MS student with related R&D experience is preferred. The duration of internship is preferably 6 months. According to the progress, publications will be submitted and patents will be filed. The ideal candidate will have a strong theoretical and computational background on solid mechanics (in particular elasticity and plasticity), the awareness of the state-of-the-art, and the ability to translate the theory into working codes using finite element method in C/C++ or Fortran. He/she should have excellent oral and written communication skills and result-oriented attitude. If interested, please forward your resume to Guanglei Xiong (guanglei.xiong at siemens.com). Best regards, Guanglei Xiong -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanharen at nrg.eu Sun Mar 4 05:43:35 2012 From: vanharen at nrg.eu (Haren, S.W. van (Steven)) Date: Sun, 4 Mar 2012 12:43:35 +0100 Subject: [petsc-users] HDF5 installation problems References: <4F4A5AC3.6030203@wb.tu-darmstadt.de> <4F4A5AF9.3060006@gmx.de> <4F4C8690.3050700@psi.ch> <4F4C988A.8080004@wb.tu-darmstadt.de> <4F4C9C3B.4010809@psi.ch> <4F4D0986.5080601@psi.ch> <1EC3BC0BF3DF4945832ECD4A46E7DE2F01A6D29A@NRGPEX.nrgnet.intra> Message-ID: <1EC3BC0BF3DF4945832ECD4A46E7DE2F01A6D29E@NRGPEX.nrgnet.intra> Dear Jed, Thanks for your reply. These are the last lines from the configure.log: sh: cd /home/steven/C++/PETSc/petsc-3.2-p6/externalpackages/hdf5-1.8.6 && make clean && make && make install Executing: cd /home/steven/C++/PETSc/petsc-3.2-p6/externalpackages/hdf5-1.8.6 && make clean && make && make install Runaway process exceeded time limit of 2500s sh: ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error running make on HDF5: Could not execute "cd /home/steven/C++/PETSc/petsc-3.2-p6/externalpackages/hdf5-1.8.6 && make clean && make && make install": Runaway process exceeded time limit of 2500s ******************************************************************************* File "./config/configure.py", line 283, in petsc_configure framework.configure(out = sys.stdout) File "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/framework.py", line 925, in configure child.configure() File "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/package.py", line 506, in configure self.executeTest(self.configureLibrary) File "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/base.py", line 115, in executeTest ret = apply(test, args,kargs) File "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/packages/hdf5.py", line 63, in configureLibrary config.package.Package.configureLibrary(self) File "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/package.py", line 433, in configureLibrary for location, directory, lib, incl in self.generateGuesses(): File "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/package.py", line 228, in generateGuesses d = self.checkDownload(1) File "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/package.py", line 313, in checkDownload return self.getInstallDir() File "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/package.py", line 183, in getInstallDir return os.path.abspath(self.Install()) File "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/packages/hdf5.py", line 56, in Install raise RuntimeError('Error running make on HDF5: '+str(e)) Using the top command I cannot really identify a process that uses all the memory. The memory use increases untill completely full. But the maximum percentage used by any process does not go above 10 percent. Even when the compilation is stopped the memory is not freed. I don't really understand this. Could you give me a few pointers based on the configure.log output? Hoe should I attach a debugger? Any good links were I can read how to do that? Thanks in advance. Kind regards, Steven -----Original Message----- From: petsc-users-bounces at mcs.anl.gov on behalf of Jed Brown Sent: Wed 2/29/2012 10:43 PM To: PETSc users list Subject: Re: [petsc-users] HDF5 installation problems On Wed, Feb 29, 2012 at 15:35, Haren, S.W. van (Steven) wrote: > Dear all, > > I try to configure Petsc to use HDF5. > > During the HDF5 compilation stage the memory use increases untill the > computer basically stalls and almost all memory is used. After 2500s the > compilation is stopped because of a runaway process. > Check configure.log and/or attach a debugger to find out where it got stuck. At least use "top" to find out which process is taking all the time and memory. From jedbrown at mcs.anl.gov Sun Mar 4 08:15:17 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 4 Mar 2012 08:15:17 -0600 Subject: [petsc-users] HDF5 installation problems In-Reply-To: <1EC3BC0BF3DF4945832ECD4A46E7DE2F01A6D29E@NRGPEX.nrgnet.intra> References: <4F4A5AC3.6030203@wb.tu-darmstadt.de> <4F4A5AF9.3060006@gmx.de> <4F4C8690.3050700@psi.ch> <4F4C988A.8080004@wb.tu-darmstadt.de> <4F4C9C3B.4010809@psi.ch> <4F4D0986.5080601@psi.ch> <1EC3BC0BF3DF4945832ECD4A46E7DE2F01A6D29A@NRGPEX.nrgnet.intra> <1EC3BC0BF3DF4945832ECD4A46E7DE2F01A6D29E@NRGPEX.nrgnet.intra> Message-ID: On Sun, Mar 4, 2012 at 05:43, Haren, S.W. van (Steven) wrote: > Dear Jed, > > Thanks for your reply. > > These are the last lines from the configure.log: > > sh: cd /home/steven/C++/PETSc/petsc-3.2-p6/externalpackages/hdf5-1.8.6 && > make clean && make && make install > Executing: cd > /home/steven/C++/PETSc/petsc-3.2-p6/externalpackages/hdf5-1.8.6 && make > clean && make && make install > Runaway process exceeded time limit of 2500s > Possibilities: 1. System clock was changed or the machine was suspended, causing incorrect timeout. (As an aside, the Python docs are horrible. They act like time is one thing, so I have to read the source code to find out that threading.Thread.join() implements a timeout using time(2) which is neither monotone nor continuous.) 2. The compilation actually takes a long time on this machine, perhaps because it doesn't get much time to run due to other jobs. You could try increasing the timeout or running make manually from externalpackages/hdf5-1.8.6/. 3. The hdf5 build process is getting stuck somewhere, I suggest trying to complete the install manually from externalpackages/hdf5-1.8.6/. If you can't get it working, send configure.log and HDF5 attempted build logs to petsc-maint at mcs.anl.gov. > sh: > > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > > ------------------------------------------------------------------------------- > Error running make on HDF5: Could not execute "cd > /home/steven/C++/PETSc/petsc-3.2-p6/externalpackages/hdf5-1.8.6 && make > clean && make && make install": > Runaway process exceeded time limit of 2500s > > ******************************************************************************* > File "./config/configure.py", line 283, in petsc_configure > framework.configure(out = sys.stdout) > File > "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/framework.py", > line 925, in configure > child.configure() > File > "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/package.py", > line 506, in configure > self.executeTest(self.configureLibrary) > File > "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/base.py", > line 115, in executeTest > ret = apply(test, args,kargs) > File > "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/packages/hdf5.py", > line 63, in configureLibrary > config.package.Package.configureLibrary(self) > File > "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/package.py", > line 433, in configureLibrary > for location, directory, lib, incl in self.generateGuesses(): > File > "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/package.py", > line 228, in generateGuesses > d = self.checkDownload(1) > File > "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/package.py", > line 313, in checkDownload > return self.getInstallDir() > File > "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/package.py", > line 183, in getInstallDir > return os.path.abspath(self.Install()) > File > "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/packages/hdf5.py", > line 56, in Install > raise RuntimeError('Error running make on HDF5: '+str(e)) > > Using the top command I cannot really identify a process that uses all the > memory. The memory use increases untill completely full. But the maximum > percentage used by any process does not go above 10 percent. Even when the > compilation is stopped the memory is not freed. I don't really understand > this. Could you give me a few pointers based on the configure.log output? > > Hoe should I attach a debugger? Any good links were I can read how to do > that? > > Thanks in advance. > > Kind regards, > > Steven > > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov on behalf of Jed Brown > Sent: Wed 2/29/2012 10:43 PM > To: PETSc users list > Subject: Re: [petsc-users] HDF5 installation problems > > On Wed, Feb 29, 2012 at 15:35, Haren, S.W. van (Steven) >wrote: > > > Dear all, > > > > I try to configure Petsc to use HDF5. > > > > During the HDF5 compilation stage the memory use increases untill the > > computer basically stalls and almost all memory is used. After 2500s the > > compilation is stopped because of a runaway process. > > > > Check configure.log and/or attach a debugger to find out where it got > stuck. At least use "top" to find out which process is taking all the time > and memory. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From manfred.gratt at uibk.ac.at Mon Mar 5 03:22:28 2012 From: manfred.gratt at uibk.ac.at (Manfred Gratt) Date: Mon, 05 Mar 2012 10:22:28 +0100 Subject: [petsc-users] Problem with KSPSetOperators In-Reply-To: References: <4F4F3F2B.70306@uibk.ac.at> <4F509E6D.7030507@uibk.ac.at> <4F50D870.3010502@uibk.ac.at> Message-ID: <4F5485D4.70107@uibk.ac.at> An HTML attachment was scrubbed... URL: From samuelandjw at gmail.com Mon Mar 5 04:05:45 2012 From: samuelandjw at gmail.com (Degang Wu) Date: Mon, 05 Mar 2012 18:05:45 +0800 Subject: [petsc-users] compilation problems of ex1.c Message-ID: <4F548FF9.1030905@gmail.com> Hi, I am new to petsc and slepc. Right now I am trying to compile ex1.c (http://acts.nersc.gov/slepc/example1/ex1.c.html) in Eclipse. I specified the paths of include files and libraries, but the compiler/linker still complained about undefined references. The following is the error message: make all Building file: ../ex1.c Invoking: GCC C Compiler gcc -I/usr/include/slepc -I/usr/include/petsc -I/usr/include/mpi -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"ex1.d" -MT"ex1.d" -o"ex1.o" "../ex1.c" Finished building: ../ex1.c Building target: slepc-ex1 Invoking: GCC C Linker gcc -L/usr/lib/slepc -L/usr/lib/petsc -o"slepc-ex1" ./ex1.o ./ex1.o: In function `main': /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:43: undefined reference to `SlepcInitialize' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:45: undefined reference to `PetscOptionsGetInt' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:45: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:46: undefined reference to `PETSC_COMM_WORLD' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:46: undefined reference to `PetscPrintf' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:46: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:52: undefined reference to `PETSC_COMM_WORLD' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:52: undefined reference to `MatCreate' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:52: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:53: undefined reference to `MatSetSizes' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:53: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:54: undefined reference to `MatSetFromOptions' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:54: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:56: undefined reference to `MatGetOwnershipRange' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:56: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:62: undefined reference to `MatSetValues' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:62: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:66: undefined reference to `MatSetValues' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:66: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:70: undefined reference to `MatSetValues' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:70: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:73: undefined reference to `MatAssemblyBegin' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:73: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:74: undefined reference to `MatAssemblyEnd' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:74: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:76: undefined reference to `MatGetVecs' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:76: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:77: undefined reference to `MatGetVecs' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:77: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:85: undefined reference to `PETSC_COMM_WORLD' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:85: undefined reference to `EPSCreate' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:85: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:90: undefined reference to `EPSSetOperators' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:90: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:91: undefined reference to `EPSSetProblemType' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:91: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:96: undefined reference to `EPSSetFromOptions' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:96: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:102: undefined reference to `EPSSolve' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:102: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:106: undefined reference to `EPSGetIterationNumber' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:106: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:107: undefined reference to `PETSC_COMM_WORLD' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:107: undefined reference to `PetscPrintf' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:107: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:108: undefined reference to `EPSGetType' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:108: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:109: undefined reference to `PETSC_COMM_WORLD' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:109: undefined reference to `PetscPrintf' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:109: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:110: undefined reference to `EPSGetDimensions' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:110: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:111: undefined reference to `PETSC_COMM_WORLD' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:111: undefined reference to `PetscPrintf' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:111: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:112: undefined reference to `EPSGetTolerances' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:112: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:113: undefined reference to `PETSC_COMM_WORLD' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:113: undefined reference to `PetscPrintf' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:113: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:121: undefined reference to `EPSGetConverged' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:121: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:122: undefined reference to `PETSC_COMM_WORLD' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:122: undefined reference to `PetscPrintf' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:122: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:128: undefined reference to `PETSC_COMM_WORLD' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:128: undefined reference to `PetscPrintf' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:130: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:137: undefined reference to `EPSGetEigenpair' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:137: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:141: undefined reference to `EPSComputeRelativeError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:141: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:151: undefined reference to `PETSC_COMM_WORLD' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:151: undefined reference to `PetscPrintf' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:151: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:153: undefined reference to `PETSC_COMM_WORLD' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:153: undefined reference to `PetscPrintf' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:153: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:156: undefined reference to `PETSC_COMM_WORLD' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:156: undefined reference to `PetscPrintf' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:156: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:162: undefined reference to `EPSDestroy' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:162: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:163: undefined reference to `MatDestroy' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:163: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:164: undefined reference to `VecDestroy' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:164: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:165: undefined reference to `VecDestroy' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:165: undefined reference to `PetscError' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:166: undefined reference to `SlepcFinalize' /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:166: undefined reference to `PetscError' collect2: ld returned 1 exit status make: *** [slepc-ex1] Error 1 Regards, Wu Degang From jroman at dsic.upv.es Mon Mar 5 04:24:43 2012 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 5 Mar 2012 11:24:43 +0100 Subject: [petsc-users] compilation problems of ex1.c In-Reply-To: <4F548FF9.1030905@gmail.com> References: <4F548FF9.1030905@gmail.com> Message-ID: <0C7F42D1-6BAC-4845-99B5-B89E09467E50@dsic.upv.es> El 05/03/2012, a las 11:05, Degang Wu escribi?: > Hi, > > I am new to petsc and slepc. Right now I am trying to compile ex1.c (http://acts.nersc.gov/slepc/example1/ex1.c.html) in Eclipse. I specified the paths of include files and libraries, but the compiler/linker still complained about undefined references. > > The following is the error message: > make all > Building file: ../ex1.c > Invoking: GCC C Compiler > gcc -I/usr/include/slepc -I/usr/include/petsc -I/usr/include/mpi -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"ex1.d" -MT"ex1.d" -o"ex1.o" "../ex1.c" > Finished building: ../ex1.c > > Building target: slepc-ex1 > Invoking: GCC C Linker > gcc -L/usr/lib/slepc -L/usr/lib/petsc -o"slepc-ex1" ./ex1.o ^^^^^^^^^^^ Here you should have the libraries, such as -lslepc -lpetsc -lX11 -lpthread -llapack -lblas and maybe others. You may also want to use mpicc rather than gcc. My suggestion is that you use the example makefiles provided by PETSc and SLEPc, rather than trying to build from Eclipse, unless you are an Eclipse expert and know what to do. Jose > ./ex1.o: In function `main': > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:43: undefined reference to `SlepcInitialize' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:45: undefined reference to `PetscOptionsGetInt' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:45: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:46: undefined reference to `PETSC_COMM_WORLD' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:46: undefined reference to `PetscPrintf' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:46: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:52: undefined reference to `PETSC_COMM_WORLD' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:52: undefined reference to `MatCreate' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:52: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:53: undefined reference to `MatSetSizes' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:53: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:54: undefined reference to `MatSetFromOptions' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:54: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:56: undefined reference to `MatGetOwnershipRange' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:56: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:62: undefined reference to `MatSetValues' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:62: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:66: undefined reference to `MatSetValues' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:66: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:70: undefined reference to `MatSetValues' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:70: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:73: undefined reference to `MatAssemblyBegin' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:73: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:74: undefined reference to `MatAssemblyEnd' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:74: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:76: undefined reference to `MatGetVecs' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:76: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:77: undefined reference to `MatGetVecs' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:77: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:85: undefined reference to `PETSC_COMM_WORLD' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:85: undefined reference to `EPSCreate' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:85: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:90: undefined reference to `EPSSetOperators' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:90: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:91: undefined reference to `EPSSetProblemType' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:91: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:96: undefined reference to `EPSSetFromOptions' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:96: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:102: undefined reference to `EPSSolve' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:102: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:106: undefined reference to `EPSGetIterationNumber' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:106: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:107: undefined reference to `PETSC_COMM_WORLD' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:107: undefined reference to `PetscPrintf' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:107: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:108: undefined reference to `EPSGetType' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:108: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:109: undefined reference to `PETSC_COMM_WORLD' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:109: undefined reference to `PetscPrintf' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:109: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:110: undefined reference to `EPSGetDimensions' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:110: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:111: undefined reference to `PETSC_COMM_WORLD' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:111: undefined reference to `PetscPrintf' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:111: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:112: undefined reference to `EPSGetTolerances' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:112: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:113: undefined reference to `PETSC_COMM_WORLD' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:113: undefined reference to `PetscPrintf' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:113: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:121: undefined reference to `EPSGetConverged' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:121: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:122: undefined reference to `PETSC_COMM_WORLD' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:122: undefined reference to `PetscPrintf' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:122: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:128: undefined reference to `PETSC_COMM_WORLD' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:128: undefined reference to `PetscPrintf' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:130: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:137: undefined reference to `EPSGetEigenpair' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:137: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:141: undefined reference to `EPSComputeRelativeError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:141: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:151: undefined reference to `PETSC_COMM_WORLD' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:151: undefined reference to `PetscPrintf' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:151: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:153: undefined reference to `PETSC_COMM_WORLD' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:153: undefined reference to `PetscPrintf' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:153: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:156: undefined reference to `PETSC_COMM_WORLD' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:156: undefined reference to `PetscPrintf' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:156: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:162: undefined reference to `EPSDestroy' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:162: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:163: undefined reference to `MatDestroy' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:163: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:164: undefined reference to `VecDestroy' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:164: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:165: undefined reference to `VecDestroy' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:165: undefined reference to `PetscError' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:166: undefined reference to `SlepcFinalize' > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:166: undefined reference to `PetscError' > collect2: ld returned 1 exit status > make: *** [slepc-ex1] Error 1 > > Regards, > Wu Degang From aron.ahmadia at kaust.edu.sa Mon Mar 5 07:17:07 2012 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Mon, 5 Mar 2012 16:17:07 +0300 Subject: [petsc-users] compilation problems of ex1.c In-Reply-To: <0C7F42D1-6BAC-4845-99B5-B89E09467E50@dsic.upv.es> References: <4F548FF9.1030905@gmail.com> <0C7F42D1-6BAC-4845-99B5-B89E09467E50@dsic.upv.es> Message-ID: The PETSc user's manual has a section on working with Eclipse (13.10) which describes one way to add the libraries*, have you consulted this? A * I notice that TeX overflowed the library link line in the PDF in my copy for 3.2, but the idea should be clear On Mon, Mar 5, 2012 at 1:24 PM, Jose E. Roman wrote: > > El 05/03/2012, a las 11:05, Degang Wu escribi?: > > > Hi, > > > > I am new to petsc and slepc. Right now I am trying to compile ex1.c ( > http://acts.nersc.gov/slepc/example1/ex1.c.html) in Eclipse. I specified > the paths of include files and libraries, but the compiler/linker still > complained about undefined references. > > > > The following is the error message: > > make all > > Building file: ../ex1.c > > Invoking: GCC C Compiler > > gcc -I/usr/include/slepc -I/usr/include/petsc -I/usr/include/mpi -O0 -g3 > -Wall -c -fmessage-length=0 -MMD -MP -MF"ex1.d" -MT"ex1.d" -o"ex1.o" > "../ex1.c" > > Finished building: ../ex1.c > > > > Building target: slepc-ex1 > > Invoking: GCC C Linker > > gcc -L/usr/lib/slepc -L/usr/lib/petsc -o"slepc-ex1" ./ex1.o > > > ^^^^^^^^^^^ > Here you should have the libraries, such as -lslepc -lpetsc -lX11 > -lpthread -llapack -lblas > and maybe others. You may also want to use mpicc rather than gcc. > > My suggestion is that you use the example makefiles provided by PETSc and > SLEPc, rather than trying to build from Eclipse, unless you are an Eclipse > expert and know what to do. > > Jose > > > ./ex1.o: In function `main': > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:43: undefined reference > to `SlepcInitialize' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:45: undefined reference > to `PetscOptionsGetInt' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:45: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:46: undefined reference > to `PETSC_COMM_WORLD' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:46: undefined reference > to `PetscPrintf' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:46: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:52: undefined reference > to `PETSC_COMM_WORLD' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:52: undefined reference > to `MatCreate' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:52: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:53: undefined reference > to `MatSetSizes' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:53: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:54: undefined reference > to `MatSetFromOptions' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:54: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:56: undefined reference > to `MatGetOwnershipRange' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:56: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:62: undefined reference > to `MatSetValues' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:62: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:66: undefined reference > to `MatSetValues' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:66: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:70: undefined reference > to `MatSetValues' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:70: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:73: undefined reference > to `MatAssemblyBegin' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:73: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:74: undefined reference > to `MatAssemblyEnd' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:74: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:76: undefined reference > to `MatGetVecs' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:76: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:77: undefined reference > to `MatGetVecs' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:77: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:85: undefined reference > to `PETSC_COMM_WORLD' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:85: undefined reference > to `EPSCreate' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:85: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:90: undefined reference > to `EPSSetOperators' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:90: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:91: undefined reference > to `EPSSetProblemType' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:91: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:96: undefined reference > to `EPSSetFromOptions' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:96: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:102: undefined reference > to `EPSSolve' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:102: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:106: undefined reference > to `EPSGetIterationNumber' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:106: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:107: undefined reference > to `PETSC_COMM_WORLD' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:107: undefined reference > to `PetscPrintf' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:107: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:108: undefined reference > to `EPSGetType' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:108: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:109: undefined reference > to `PETSC_COMM_WORLD' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:109: undefined reference > to `PetscPrintf' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:109: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:110: undefined reference > to `EPSGetDimensions' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:110: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:111: undefined reference > to `PETSC_COMM_WORLD' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:111: undefined reference > to `PetscPrintf' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:111: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:112: undefined reference > to `EPSGetTolerances' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:112: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:113: undefined reference > to `PETSC_COMM_WORLD' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:113: undefined reference > to `PetscPrintf' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:113: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:121: undefined reference > to `EPSGetConverged' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:121: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:122: undefined reference > to `PETSC_COMM_WORLD' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:122: undefined reference > to `PetscPrintf' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:122: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:128: undefined reference > to `PETSC_COMM_WORLD' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:128: undefined reference > to `PetscPrintf' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:130: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:137: undefined reference > to `EPSGetEigenpair' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:137: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:141: undefined reference > to `EPSComputeRelativeError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:141: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:151: undefined reference > to `PETSC_COMM_WORLD' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:151: undefined reference > to `PetscPrintf' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:151: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:153: undefined reference > to `PETSC_COMM_WORLD' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:153: undefined reference > to `PetscPrintf' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:153: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:156: undefined reference > to `PETSC_COMM_WORLD' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:156: undefined reference > to `PetscPrintf' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:156: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:162: undefined reference > to `EPSDestroy' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:162: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:163: undefined reference > to `MatDestroy' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:163: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:164: undefined reference > to `VecDestroy' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:164: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:165: undefined reference > to `VecDestroy' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:165: undefined reference > to `PetscError' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:166: undefined reference > to `SlepcFinalize' > > /home/wdg/Dropbox/dev/slepc-ex1/Debug/../ex1.c:166: undefined reference > to `PetscError' > > collect2: ld returned 1 exit status > > make: *** [slepc-ex1] Error 1 > > > > Regards, > > Wu Degang > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengshixiangze at 163.com Mon Mar 5 08:35:01 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Mon, 5 Mar 2012 22:35:01 +0800 (CST) Subject: [petsc-users] error occurs when running ex19 in src/snes/examples/tutorials Message-ID: <2636a114.160a5.135e348eb48.Coremail.zengshixiangze@163.com> Hi, I want to run the code on the GPU, and I have installed the CUDA, Cusp, Thrust as the instruction. Now I can use nvcc successfully. But when I run the ex19. It terminate with the following message: "terminate called after throwing an instance of 'thrust::system::system_error' what(): invalid device function Aborted." I have tried closed all the X systems,but the same error still occurs. What's the problem? Thank you. Zeng -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Mon Mar 5 12:01:25 2012 From: u.tabak at tudelft.nl (Umut Tabak) Date: Mon, 05 Mar 2012 19:01:25 +0100 Subject: [petsc-users] routines called in LAPACK interface of SLEPc Message-ID: <4F54FF75.7050506@tudelft.nl> Dear SLEPc team, May I kindly ask which LAPACK routines are called in the LAPACK interface of SLEPc? And what kind of checks do you conduct to check if the matrix is symmetric or not? I have some matrices which are not completely symmetric however the off-diagonal terms are really small(orders: 1e-16) and I used Intel MKL Lapack routines for my problems. But, I could not get the correct eigenvectors, however I can get the correct eigenvalues on which I am puzzling with. Best regards, Umut From jroman at dsic.upv.es Mon Mar 5 13:10:22 2012 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 5 Mar 2012 20:10:22 +0100 Subject: [petsc-users] routines called in LAPACK interface of SLEPc In-Reply-To: <4F54FF75.7050506@tudelft.nl> References: <4F54FF75.7050506@tudelft.nl> Message-ID: <3C6D3341-E36C-4F3F-A5A8-767CF81FEEFD@dsic.upv.es> On 05/03/2012, Umut Tabak wrote: > Dear SLEPc team, > > May I kindly ask which LAPACK routines are called in the LAPACK interface of SLEPc? > > And what kind of checks do you conduct to check if the matrix is symmetric or not? > > I have some matrices which are not completely symmetric however the off-diagonal terms are really small(orders: 1e-16) and I used Intel MKL Lapack routines for my problems. > > But, I could not get the correct eigenvectors, however I can get the correct eigenvalues on which I am puzzling with. > > Best regards, > Umut When the problem in known to be symmetric, we symmetrize the matrix and call a symmetric Lapack subroutine such as dsyevr. Jose From zengshixiangze at 163.com Mon Mar 5 23:19:33 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Tue, 6 Mar 2012 13:19:33 +0800 (CST) Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> Message-ID: <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> Hi, Matt. We know that when we compile the CUDA-C code, we use nvcc. But if we run the PETSc-CUDA code in GPU, what we do is just to change the type of Mat and Vec. In the PETSc-CUDA code compilings progress ,does it use nvcc? And when implement the PETSc for using GPU, do you ever consider the comupute capibility of the GPU. And in which place, the parameter of the comupute capibility of the GPU has been set? Thank you. Zeng ? 2012-03-05 22:28:20?"Matthew Knepley" ??? On Mon, Mar 5, 2012 at 8:18 AM, Xiangze Zeng wrote: Hi, Matt. I have tried the method you told me( I closed all the X systems),but the same error still occurs. Is there something wrong with the installation of the thrust? And I find the same problem in several web pages, like this http://code.google.com/p/thrust/wiki/Debugging, they all say the error is related to the compute capability . Is that the case? My GPU is GeForce 310, which supports compute capability 1.2 . One of the downsides of CUDA, at least from our perspective, is that these errors are very hard to debug. If thrust compiles and the cudaInit() succeeds, there is not much else we can do to debug on this machine. Maybe you can try running thrust examples? Matt Thank you. ? 2012-03-02 23:55:13?"Matthew Knepley" ??? On Fri, Mar 2, 2012 at 9:40 AM, Xiangze Zeng wrote: Thank both of you, Satish and Jed. This warning message has disappeared. But another problem occurs, it terminates with the same message as I run my own code:" terminate called after throwing an instance of 'thrust::system::system_error' what(): invalid device function Aborted." (Several days ago, I thought I run the ex19 successfully, but today I found I was totally wrong when I find the warning message in the PETSc Performance Summary. So sorry about that!) Most likely, this a problem with another program locking your GPU (I assume you are running on your laptop/desktop). Start closing apps one at a time (especially graphical ones like a PDF reader) and try running each time. Matt May this new problem be related to the installation of the cuda( I can run cuda-C code successfully in my PC)? Thank you again! Zeng Xiangze At 2012-03-02 23:18:19,"Satish Balay" wrote: >On Fri, 2 Mar 2012, Xiangze Zeng wrote: > >> Dear all, >> >> After I have installed the PETSc to use the NVidia GPUS, I run the ex19. When I make ex19, it stopped with the message: >> >> >> "makefile:24: /conf/variables: No such file or directory >> makefile:25: /conf/rules: No such file or directory >> makefile:1023: /conf/test: No such file or directory >> make: *** No rule to make target `/conf/test'. Stop." >> >> >> Then I add the PETSC_DIR to the makefile. After that, I made ex19 successfully. But when I run it using the command : > >Yes - you need PETSC_DIR/PETSC_ARCH values to be specified to >make. You can set these as env variables - or specify to make on the >command line - or add to makefile. [we recommended not modifying the >makefile - to keep it portable..] > >> >> >> "./ex19 -da_vec_type mpicusp -da_mat_type mpiaijcusp -pc_type none -dmmg_nlevels 1 -da_grid_x 100 -da_grid_y 100 -log_summary -mat_no_inode -preload off -cusp_synchronize". >> >> >> In the PETSc Performance Summary, it says:" >> WARNING! There are options you set that were not used! >> WARNING! could be spelling mistake, etc! >> Option left: name:-da_mat_type value: mpiaijcusp >> Option left: name:-da_vec_type value: mpicusp" > >Looks like the options are now changed to -dm_mat_type and -dm_vec_type. > >Satish > >> >> >> Is there any problem with it? >> Any response will be appreciated! Thank you so much! >> >> >> Zeng Xiangze >> >> >> >> >> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From petsc-maint at mcs.anl.gov Mon Mar 5 23:31:06 2012 From: petsc-maint at mcs.anl.gov (Matthew Knepley) Date: Mon, 5 Mar 2012 23:31:06 -0600 Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> Message-ID: 2012/3/5 Xiangze Zeng > Hi, Matt. > > We know that when we compile the CUDA-C code, we use nvcc. But if we run > the PETSc-CUDA code in GPU, what we do is just to change the type of Mat > and Vec. In the PETSc-CUDA code compilings progress ,does it use nvcc? > Yes. > And when implement the PETSc for using GPU, do you ever consider the > comupute capibility of the GPU. And in which place, the parameter of the > comupute capibility of the GPU has been set? > You configure using --with-cuda-arch=sm_13 for instance, or compute_12. And you have given me an idea what your problem is. The GeForce 310 has no double precision floating point units. Therefore, you must configure PETSc using --with-precision=single. Matt > Thank you. > Zeng > > ? 2012-03-05 22:28:20?"Matthew Knepley" ??? > > On Mon, Mar 5, 2012 at 8:18 AM, Xiangze Zeng wrote: > >> Hi, Matt. >> I have tried the method you told me( I closed all the X systems),but the >> same error still occurs. Is there something wrong with the installation of >> the thrust? >> >> >> And I find the same problem in several web pages, like this >> http://code.google.com/p/thrust/wiki/Debugging, they all say the error >> is related to the compute capability . Is that the case? My GPU is GeForce >> 310, which supports compute capability 1.2 . >> > > One of the downsides of CUDA, at least from our perspective, is that these > errors are very > hard to debug. If thrust compiles and the cudaInit() succeeds, there is > not much else we > can do to debug on this machine. Maybe you can try running thrust examples? > > Matt > > >> Thank you. >> >> ? 2012-03-02 23:55:13?"Matthew Knepley" ??? >> On Fri, Mar 2, 2012 at 9:40 AM, Xiangze Zeng >> wrote: >> >> Thank both of you, Satish and Jed. >> >> >> This warning message has disappeared. But another problem occurs, it >> terminates with the same message as I run my own code:" >> terminate called after throwing an instance of >> 'thrust::system::system_error' >> what(): invalid device function >> Aborted." >> (Several days ago, I thought I run the ex19 successfully, but today I >> found I was totally wrong when I find the warning message in the PETSc >> Performance Summary. So sorry about that!) >> >> >> >> Most likely, this a problem with another program locking your GPU (I >> assume you are running on your laptop/desktop). Start >> closing apps one at a time (especially graphical ones like a PDF reader) >> and try running each time. >> >> >> Matt >> >> May this new problem be related to the installation of the cuda( I can >> run cuda-C code successfully in my PC)? >> >> >> Thank you again! >> Zeng Xiangze >> >> At 2012-03-02 23:18:19,"Satish Balay" wrote: >> >On Fri, 2 Mar 2012, Xiangze Zeng wrote: >> > >> >> >> Dear all, >> >> >> >> After I have installed the PETSc to use the NVidia GPUS, I run the >> ex19. When I make ex19, it stopped with the message: >> >> >> >> >> >> "makefile:24: /conf/variables: No such file or directory >> >> makefile:25: /conf/rules: No such file or directory >> >> makefile:1023: /conf/test: No such file or directory >> >> make: *** No rule to make target `/conf/test'. Stop." >> >> >> >> >> >> Then I add the PETSC_DIR to the makefile. After that, I made ex19 >> successfully. But when I run it using the command : >> > >> >> >Yes - you need PETSC_DIR/PETSC_ARCH values to be specified to >> >make. You can set these as env variables - or specify to make on the >> >command line - or add to makefile. [we recommended not modifying the >> >makefile - to keep it portable..] >> > >> >> >> >> >> >> >> "./ex19 -da_vec_type mpicusp -da_mat_type mpiaijcusp -pc_type none >> -dmmg_nlevels 1 -da_grid_x 100 -da_grid_y 100 -log_summary -mat_no_inode >> -preload off -cusp_synchronize". >> >> >> >> >> >> In the PETSc Performance Summary, it says:" >> >> WARNING! There are options you set that were not used! >> >> WARNING! could be spelling mistake, etc! >> >> Option left: name:-da_mat_type value: mpiaijcusp >> >> Option left: name:-da_vec_type value: mpicusp" >> > >> >> >Looks like the options are now changed to -dm_mat_type and -dm_vec_type. >> > >> >Satish >> > >> >> >> >> >> >> >> Is there any problem with it? >> >> Any response will be appreciated! Thank you so much! >> >> >> >> >> >> Zeng Xiangze >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengshixiangze at 163.com Mon Mar 5 23:42:07 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Tue, 6 Mar 2012 13:42:07 +0800 (CST) Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> Message-ID: <3100000a.1e163.135e687676f.Coremail.zengshixiangze@163.com> Thank you. I'll try it. ? 2012-03-06 13:31:06?"Matthew Knepley" ??? 2012/3/5 Xiangze Zeng Hi, Matt. We know that when we compile the CUDA-C code, we use nvcc. But if we run the PETSc-CUDA code in GPU, what we do is just to change the type of Mat and Vec. In the PETSc-CUDA code compilings progress ,does it use nvcc? Yes. And when implement the PETSc for using GPU, do you ever consider the comupute capibility of the GPU. And in which place, the parameter of the comupute capibility of the GPU has been set? You configure using --with-cuda-arch=sm_13 for instance, or compute_12. And you have given me an idea what your problem is. The GeForce 310 has no double precision floating point units. Therefore, you must configure PETSc using --with-precision=single. Matt Thank you. Zeng ? 2012-03-05 22:28:20?"Matthew Knepley" ??? On Mon, Mar 5, 2012 at 8:18 AM, Xiangze Zeng wrote: Hi, Matt. I have tried the method you told me( I closed all the X systems),but the same error still occurs. Is there something wrong with the installation of the thrust? And I find the same problem in several web pages, like this http://code.google.com/p/thrust/wiki/Debugging, they all say the error is related to the compute capability . Is that the case? My GPU is GeForce 310, which supports compute capability 1.2 . One of the downsides of CUDA, at least from our perspective, is that these errors are very hard to debug. If thrust compiles and the cudaInit() succeeds, there is not much else we can do to debug on this machine. Maybe you can try running thrust examples? Matt Thank you. ? 2012-03-02 23:55:13?"Matthew Knepley" ??? On Fri, Mar 2, 2012 at 9:40 AM, Xiangze Zeng wrote: Thank both of you, Satish and Jed. This warning message has disappeared. But another problem occurs, it terminates with the same message as I run my own code:" terminate called after throwing an instance of 'thrust::system::system_error' what(): invalid device function Aborted." (Several days ago, I thought I run the ex19 successfully, but today I found I was totally wrong when I find the warning message in the PETSc Performance Summary. So sorry about that!) Most likely, this a problem with another program locking your GPU (I assume you are running on your laptop/desktop). Start closing apps one at a time (especially graphical ones like a PDF reader) and try running each time. Matt May this new problem be related to the installation of the cuda( I can run cuda-C code successfully in my PC)? Thank you again! Zeng Xiangze At 2012-03-02 23:18:19,"Satish Balay" wrote: >On Fri, 2 Mar 2012, Xiangze Zeng wrote: > >> Dear all, >> >> After I have installed the PETSc to use the NVidia GPUS, I run the ex19. When I make ex19, it stopped with the message: >> >> >> "makefile:24: /conf/variables: No such file or directory >> makefile:25: /conf/rules: No such file or directory >> makefile:1023: /conf/test: No such file or directory >> make: *** No rule to make target `/conf/test'. Stop." >> >> >> Then I add the PETSC_DIR to the makefile. After that, I made ex19 successfully. But when I run it using the command : > >Yes - you need PETSC_DIR/PETSC_ARCH values to be specified to >make. You can set these as env variables - or specify to make on the >command line - or add to makefile. [we recommended not modifying the >makefile - to keep it portable..] > >> >> >> "./ex19 -da_vec_type mpicusp -da_mat_type mpiaijcusp -pc_type none -dmmg_nlevels 1 -da_grid_x 100 -da_grid_y 100 -log_summary -mat_no_inode -preload off -cusp_synchronize". >> >> >> In the PETSc Performance Summary, it says:" >> WARNING! There are options you set that were not used! >> WARNING! could be spelling mistake, etc! >> Option left: name:-da_mat_type value: mpiaijcusp >> Option left: name:-da_vec_type value: mpicusp" > >Looks like the options are now changed to -dm_mat_type and -dm_vec_type. > >Satish > >> >> >> Is there any problem with it? >> Any response will be appreciated! Thank you so much! >> >> >> Zeng Xiangze >> >> >> >> >> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Tue Mar 6 02:13:00 2012 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Tue, 6 Mar 2012 08:13:00 +0000 Subject: [petsc-users] segv in VecSum when using nested vec Message-ID: In a larger code I'm getting a segmentation fault when setting a null space with KSPSetNullSpace. After some trying I can reproduce the problem with the small code below. It happens when calling VecSum on a nested vec. /* test program to find VecSum segmentation fault */ #include class vecsumsegv { public: PetscErrorCode ierr; PetscInt nx, ny; Vec x, subx[2]; IS isg[2]; vecsumsegv(PetscInt m, PetscInt n); private: PetscErrorCode setupVectors(); PetscErrorCode setupIndexSets(); PetscErrorCode setupVecBlock0(); PetscErrorCode setupVecBlock1(); }; vecsumsegv::vecsumsegv(PetscInt m, PetscInt n) { nx=m; ny=n; ierr = setupVectors(); ierr = setupIndexSets(); ierr = VecCreateNest(PETSC_COMM_WORLD,2,isg,subx,&x); // nested vec ierr = VecView(x,(PetscViewer)PETSC_VIEWER_DEFAULT); // this causes the segv: PetscScalar val; ierr = VecSum(x,&val); } PetscErrorCode vecsumsegv::setupVectors() { // the two vectors ierr = setupVecBlock0(); CHKERRQ(ierr); ierr = setupVecBlock1(); CHKERRQ(ierr); return 0; } PetscErrorCode vecsumsegv::setupVecBlock0() { // 2N vector corresponding to block 0 ierr = VecCreate(PETSC_COMM_WORLD,&subx[0]); CHKERRQ(ierr); ierr = VecSetSizes(subx[0],PETSC_DECIDE,2*nx*ny); CHKERRQ(ierr); ierr = VecSetType(subx[0],VECMPI); CHKERRQ(ierr); return 0; } PetscErrorCode vecsumsegv::setupVecBlock1() { // N vector corresponding to block 1 ierr = VecCreate(PETSC_COMM_WORLD,&subx[1]); CHKERRQ(ierr); ierr = VecSetSizes(subx[1],PETSC_DECIDE,nx*ny); CHKERRQ(ierr); ierr = VecSetType(subx[1],VECMPI); CHKERRQ(ierr); return 0; } PetscErrorCode vecsumsegv::setupIndexSets() { PetscInt rank,m0,m1; ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank); CHKERRQ(ierr); ierr = VecGetLocalSize(subx[0],&m0); CHKERRQ(ierr); ierr = VecGetLocalSize(subx[1],&m1); CHKERRQ(ierr); // first index set PetscInt *idx0=new PetscInt[m0]; for (int i=0; i References: Message-ID: On Tue, Mar 6, 2012 at 2:13 AM, Klaij, Christiaan wrote: > In a larger code I'm getting a segmentation fault when setting a null space > with KSPSetNullSpace. After some trying I can reproduce the problem with > the small code below. It happens when calling VecSum on a nested vec. > Runs for me: Vector Object: type=nest, rows=2 VecNest structure: (0) : type=mpi, rows=24 Vector Object: 1 MPI processes type: mpi Process [0] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (1) : type=mpi, rows=12 Vector Object: 1 MPI processes type: mpi Process [0] 0 0 0 0 0 0 0 0 0 0 0 0 and it was valgrind clean. Are you running on multiple processes? Matt > /* test program to find VecSum segmentation fault */ > > #include > > class vecsumsegv { > public: > PetscErrorCode ierr; > PetscInt nx, ny; > Vec x, subx[2]; > IS isg[2]; > vecsumsegv(PetscInt m, PetscInt n); > private: > PetscErrorCode setupVectors(); > PetscErrorCode setupIndexSets(); > PetscErrorCode setupVecBlock0(); > PetscErrorCode setupVecBlock1(); > }; > > vecsumsegv::vecsumsegv(PetscInt m, PetscInt n) { > nx=m; > ny=n; > ierr = setupVectors(); > ierr = setupIndexSets(); > ierr = VecCreateNest(PETSC_COMM_WORLD,2,isg,subx,&x); // nested vec > ierr = VecView(x,(PetscViewer)PETSC_VIEWER_DEFAULT); > // this causes the segv: > PetscScalar val; > ierr = VecSum(x,&val); > } > > PetscErrorCode vecsumsegv::setupVectors() { > // the two vectors > ierr = setupVecBlock0(); CHKERRQ(ierr); > ierr = setupVecBlock1(); CHKERRQ(ierr); > return 0; > } > > PetscErrorCode vecsumsegv::setupVecBlock0() { > // 2N vector corresponding to block 0 > ierr = VecCreate(PETSC_COMM_WORLD,&subx[0]); CHKERRQ(ierr); > ierr = VecSetSizes(subx[0],PETSC_DECIDE,2*nx*ny); CHKERRQ(ierr); > ierr = VecSetType(subx[0],VECMPI); CHKERRQ(ierr); > return 0; > } > > PetscErrorCode vecsumsegv::setupVecBlock1() { > // N vector corresponding to block 1 > ierr = VecCreate(PETSC_COMM_WORLD,&subx[1]); CHKERRQ(ierr); > ierr = VecSetSizes(subx[1],PETSC_DECIDE,nx*ny); CHKERRQ(ierr); > ierr = VecSetType(subx[1],VECMPI); CHKERRQ(ierr); > return 0; > } > > PetscErrorCode vecsumsegv::setupIndexSets() { > PetscInt rank,m0,m1; > ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank); CHKERRQ(ierr); > ierr = VecGetLocalSize(subx[0],&m0); CHKERRQ(ierr); > ierr = VecGetLocalSize(subx[1],&m1); CHKERRQ(ierr); > // first index set > PetscInt *idx0=new PetscInt[m0]; > for (int i=0; i ierr = > ISCreateGeneral(PETSC_COMM_WORLD,m0,idx0,PETSC_COPY_VALUES,&isg[0]); > CHKERRQ(ierr); > delete[] idx0; > // ISView(isg[0],PETSC_VIEWER_STDOUT_WORLD); > // second index set > PetscInt *idx1=new PetscInt[m1]; > for (int i=0; i ierr = > ISCreateGeneral(PETSC_COMM_WORLD,m1,idx1,PETSC_COPY_VALUES,&isg[1]); > CHKERRQ(ierr); > delete[] idx1; > // ISView(isg[1],PETSC_VIEWER_STDOUT_WORLD); > return 0; > } > > int main(int argc, char **argv) { > PetscErrorCode ierr; > PetscReal val; > ierr = PetscInitialize(&argc, &argv, PETSC_NULL, PETSC_NULL); > CHKERRQ(ierr); > vecsumsegv tryme(3,4); > ierr = PetscFinalize(); CHKERRQ(ierr); > return 0; > } > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] VecSum line 1301 > /home/CKlaij/ReFRESCO/Libraries/build/petsc-3.2-p5/src/vec/vec/utils/vinv.c > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54 > CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./vecsumsegv on a linux_64b named lin0133 by cklaij Tue > Mar 6 08:55:25 2012 > [0]PETSC ERROR: Libraries linked from > /home/CKlaij/ReFRESCO/Libraries/build/petsc-3.2-p5/linux_64bit_debug/lib > [0]PETSC ERROR: Configure run at Thu Mar 1 10:59:35 2012 > [0]PETSC ERROR: Configure options > --with-mpi-dir=/opt/refresco/64bit_intelv11.1_openmpi/openmpi-1.4.5 > --with-clanguage=c++ --with-x=1 --with-debugging=1 > --with-hypre-include=/opt/refresco/64bit_intelv11.1_openmpi/hypre-2.7.0b/include > --with-hypre-lib=/opt/refresco/64bit_intelv11.1_openmpi/hypre-2.7.0b/lib/libHYPRE.a > --with-ml-include=/opt/refresco/64bit_intelv11.1_openmpi/ml-6.2/include > --with-ml-lib=/opt/refresco/64bit_intelv11.1_openmpi/ml-6.2/lib/libml.a > --with-blas-lapack-dir=/opt/intel/mkl > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 59. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > -------------------------------------------------------------------------- > mpiexec has exited due to process rank 0 with PID 565 on > node lin0133 exiting without calling "finalize". This may > have caused other processes in the application to be > terminated by signals sent by mpiexec (as reported here). > -------------------------------------------------------------------------- > > > dr. ir. Christiaan Klaij > CFD Researcher > Research & Development > E mailto:C.Klaij at marin.nl > T +31 317 49 33 44 > > MARIN > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Tue Mar 6 08:19:16 2012 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Tue, 6 Mar 2012 14:19:16 +0000 Subject: [petsc-users] segv in VecSum when using nested vec Message-ID: >> In a larger code I'm getting a segmentation fault when setting a null space >> with KSPSetNullSpace. After some trying I can reproduce the problem with >> the small code below. It happens when calling VecSum on a nested vec. > >Runs for me: > >and it was valgrind clean. Are you running on multiple processes? > > Matt Strange, I get the same error both with single and multiple processes... dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From knepley at gmail.com Tue Mar 6 08:27:23 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 6 Mar 2012 08:27:23 -0600 Subject: [petsc-users] segv in VecSum when using nested vec In-Reply-To: References: Message-ID: On Tue, Mar 6, 2012 at 8:19 AM, Klaij, Christiaan wrote: > >> In a larger code I'm getting a segmentation fault when setting a null > space > >> with KSPSetNullSpace. After some trying I can reproduce the problem with > >> the small code below. It happens when calling VecSum on a nested vec. > > > >Runs for me: > > > >and it was valgrind clean. Are you running on multiple processes? > > > > Matt > > Strange, I get the same error both with single and multiple processes... > Could you get a stack trace with gdb? Also, I ran on petsc-dev. Matt dr. ir. Christiaan Klaij > CFD Researcher > Research & Development > E mailto:C.Klaij at marin.nl > T +31 317 49 33 44 > > MARIN > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hanangul12 at yahoo.co.uk Tue Mar 6 08:31:35 2012 From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh) Date: Tue, 6 Mar 2012 14:31:35 +0000 (GMT) Subject: [petsc-users] PETSc.: Projection preconditioner as PCMG Message-ID: <1331044295.65746.YahooMailNeo@web28515.mail.ukl.yahoo.com> Dear, I had to implement the projection preconditioner (P = I - A*R*(A_H)^1 *P) along with an standard preconditioner (M). And I read in some thread of users list that multilevel preconditioner can better handled by PCMG too, thus? I had tried to implement the later (M) as pre smoother, where as the earlier (P) as CGC in PCMG framework. 1. Does it make sense ? or the coarse grid correction of PCMG only does this part: R*(A_H)^1 *P?? ?? 2. How can I adapt my projection preconditioner (P) as (P1 = I - A*R*(A_H)^1 *P + R*(A_H)^1 *P ) ? I hope made you to get my problem. Even sorry for not a good narration. Thank you in anticipation, Abdul Hanan TechnischeUniversiteit Delft Faculteit Elektrotechniek, Wiskunde en Informatica Numerical Analysis -------------- next part -------------- An HTML attachment was scrubbed... URL: From hanangul12 at yahoo.co.uk Tue Mar 6 08:50:58 2012 From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh) Date: Tue, 6 Mar 2012 14:50:58 +0000 (GMT) Subject: [petsc-users] Projection preconditioner as PCMG Message-ID: <1331045458.64662.YahooMailNeo@web28501.mail.ukl.yahoo.com> Dear members, I had to implement the projection preconditioner (P = I - A*R*(A_H)^1 *P) along with an standard preconditioner (M). And I read in some thread of users list that multilevel preconditioner can better handled by PCMG too, thus? I had tried to implement the later (M) as pre smoother, where as the earlier (P) as CGC in PCMG framework. 1. Does it make sense ? or the coarse grid correction of PCMG only does this part: R*(A_H)^1 *P?? ?? 2. How can I adapt my projection preconditioner (P) as (P1 = I - A*R*(A_H)^1 *P + R*(A_H)^1 *P ) ? I hope made you to get my problem. Even sorry for not a good narration. Thank you in anticipation, ? Abdul H. TechnischeUniversiteit Delft Faculteit Elektrotechniek, Wiskunde en Informatica Numerical Analysis -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Mar 6 09:04:59 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 6 Mar 2012 09:04:59 -0600 Subject: [petsc-users] Projection preconditioner as PCMG In-Reply-To: <1331045458.64662.YahooMailNeo@web28501.mail.ukl.yahoo.com> References: <1331045458.64662.YahooMailNeo@web28501.mail.ukl.yahoo.com> Message-ID: On Tue, Mar 6, 2012 at 08:50, Abdul Hanan Sheikh wrote: > I had to implement the projection preconditioner (P = I - A*R*(A_H)^1 *P) > along with an standard preconditioner (M). > And I read in some thread of users list that multilevel preconditioner can better > handled by PCMG too, thus > I had tried to implement the later (M) as pre smoother, where as the > earlier (P) as CGC in PCMG framework. > > 1. Does it make sense ? > or the coarse grid correction of PCMG only does this part: R*(A_H)^1 *P > ?? > 2. How can I adapt my projection preconditioner (P) as (P1 = I - > A*R*(A_H)^1 *P + R*(A_H)^1 *P ) ? It's still not clear exactly what you want the method to do. PCMG can do additive or multiplicative cycles. For more general composition, see http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCCOMPOSITE.html http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGALERKIN.html I see the man pages are lacking details, the user's manual has more details. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sanjay.Kharche at liverpool.ac.uk Tue Mar 6 09:10:34 2012 From: Sanjay.Kharche at liverpool.ac.uk (Kharche, Sanjay) Date: Tue, 6 Mar 2012 15:10:34 +0000 Subject: [petsc-users] ADI Message-ID: <04649ABFF695C94F8E6CF3BBBA9B16650F326FAF@BHEXMBX1.livad.liv.ac.uk> Dear All Can someone suggest me for how to solve the 3D heat equation implicitly using ADI please. In brief, I would like to solve the 3D heat equation with a constant diffusion using finite differences. In 1D, I am using the Thomas algorithm in serial. However, since the 3D problem is large, I need to parallelise it. Any pointers will be appreciated. thanks Sanjay -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Mar 6 09:38:26 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 6 Mar 2012 09:38:26 -0600 Subject: [petsc-users] ADI In-Reply-To: <04649ABFF695C94F8E6CF3BBBA9B16650F326FAF@BHEXMBX1.livad.liv.ac.uk> References: <04649ABFF695C94F8E6CF3BBBA9B16650F326FAF@BHEXMBX1.livad.liv.ac.uk> Message-ID: On Tue, Mar 6, 2012 at 9:10 AM, Kharche, Sanjay < Sanjay.Kharche at liverpool.ac.uk> wrote: > > Dear All > > Can someone suggest me for how to solve the 3D heat equation implicitly > using ADI please. > > In brief, I would like to solve the 3D heat equation with a constant > diffusion using finite differences. In 1D, I am using the Thomas algorithm > in serial. However, since the 3D problem is large, I need to parallelise > it. Any pointers will be appreciated. > ADI is from the last century. Use PETSc's Geometric Multigrid. If you look at SNES ex50 and the tutorial runs for it, you can see how this works. Matt > thanks > Sanjay > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Mar 6 09:47:42 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 6 Mar 2012 09:47:42 -0600 Subject: [petsc-users] ADI In-Reply-To: References: <04649ABFF695C94F8E6CF3BBBA9B16650F326FAF@BHEXMBX1.livad.liv.ac.uk> Message-ID: On Tue, Mar 6, 2012 at 09:38, Matthew Knepley wrote: > ADI is from the last century. Use PETSc's Geometric Multigrid. If you look > at SNES ex50 and the tutorial runs for it, you can > see how this works. > ex5.c is a better example if you are solving a scalar problem. Sanjay, ADI has poor memory memory characteristics, does not scale well in parallel, and is very restrictive. As Matt says, multigrid is a better and more general approach, it will converge faster and scale better. -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Tue Mar 6 09:52:43 2012 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Tue, 6 Mar 2012 15:52:43 +0000 Subject: [petsc-users] segv in VecSum when using nested vec Message-ID: >> >> In a larger code I'm getting a segmentation fault when setting a null >> space >> >> with KSPSetNullSpace. After some trying I can reproduce the problem with >> >> the small code below. It happens when calling VecSum on a nested vec. >> > >> >Runs for me: >> > >> >and it was valgrind clean. Are you running on multiple processes? >> > >> > Matt >> >> Strange, I get the same error both with single and multiple processes... >> > >Could you get a stack trace with gdb? Also, I ran on petsc-dev. Breakpoint 1, vecsumsegv::vecsumsegv (this=0x7fffffffd310, m=3, n=4) at vecsumsegv.cc:28 28 ierr = VecSum(x,&val); (gdb) next Program received signal SIGSEGV, Segmentation fault. 0x00000000004fda7c in VecSum (v=0xbad750, sum=0x7fffffffd2c8) at /home/CKlaij/ReFRESCO/Libraries/build/petsc-3.2-p5/src/vec/vec/utils/vinv.c:1307 1307 lsum += x[i]; (gdb) backtrace #0 0x00000000004fda7c in VecSum (v=0xbad750, sum=0x7fffffffd2c8) at /home/CKlaij/ReFRESCO/Libraries/build/petsc-3.2-p5/src/vec/vec/utils/vinv.c:1307 #1 0x000000000041f93e in vecsumsegv::vecsumsegv (this=0x7fffffffd310, m=3, n=4) at vecsumsegv.cc:28 #2 0x00000000004202fd in main (argc=1, argv=0x7fffffffd468) at vecsumsegv.cc:78 (gdb) bt full #0 0x00000000004fda7c in VecSum (v=0xbad750, sum=0x7fffffffd2c8) at /home/CKlaij/ReFRESCO/Libraries/build/petsc-3.2-p5/src/vec/vec/utils/vinv.c:1307 ierr = 0 i = 0 n = 36 x = 0x2 lsum = 0 #1 0x000000000041f93e in vecsumsegv::vecsumsegv (this=0x7fffffffd310, m=3, n=4) at vecsumsegv.cc:28 val = 0 #2 0x00000000004202fd in main (argc=1, argv=0x7fffffffd468) at vecsumsegv.cc:78 ierr = 0 val = 6.3659873728958169e-314 tryme = {ierr = 0, nx = 3, ny = 4, x = 0xbad750, subx = {0xb954e0, 0xb9b640}, isg = {0xba4ac0, 0xba9140}} dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From jedbrown at mcs.anl.gov Tue Mar 6 10:07:29 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 6 Mar 2012 10:07:29 -0600 Subject: [petsc-users] segv in VecSum when using nested vec In-Reply-To: References: Message-ID: On Tue, Mar 6, 2012 at 09:52, Klaij, Christiaan wrote: > >Could you get a stack trace with gdb? Also, I ran on petsc-dev. > > Breakpoint 1, vecsumsegv::vecsumsegv (this=0x7fffffffd310, m=3, n=4) > at vecsumsegv.cc:28 > 28 ierr = VecSum(x,&val); > (gdb) next This was fixed in petsc-dev. http://petsc.cs.iit.edu/petsc/petsc-dev/rev/3b9bc425b217 Why are you using VecNest? It causes more harm than good in all but some very special circumstances. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanharen at nrg.eu Tue Mar 6 10:54:15 2012 From: vanharen at nrg.eu (Haren, S.W. van (Steven)) Date: Tue, 6 Mar 2012 17:54:15 +0100 Subject: [petsc-users] HDF5 installation problems References: <4F4A5AC3.6030203@wb.tu-darmstadt.de> <4F4A5AF9.3060006@gmx.de> <4F4C8690.3050700@psi.ch> <4F4C988A.8080004@wb.tu-darmstadt.de> <4F4C9C3B.4010809@psi.ch> <4F4D0986.5080601@psi.ch> <1EC3BC0BF3DF4945832ECD4A46E7DE2F01A6D29A@NRGPEX.nrgnet.intra> <1EC3BC0BF3DF4945832ECD4A46E7DE2F01A6D29E@NRGPEX.nrgnet.intra> Message-ID: <1EC3BC0BF3DF4945832ECD4A46E7DE2F01A6D2A5@NRGPEX.nrgnet.intra> Dear Jed, Thanks for your help. After some more investigation it turned out that hdf5 was testing its parallel abilities with 6 processes during the testing if (if I installed it via PETSC configure). After this I installed hdf5 manually and then configured Petsc using: ./configure --with-shared-libraries --with-hdf5-lib=/home/steven/C++/PETSc/petsc-3.2-p6/externalpackages/hdf5-1.8.6/hdf5/lib/libhdf5.so --with-hdf5 --with-hdf5-include=/home/steven/C++/PETSc/petsc-3.2-p6/externalpackages/hdf5-1.8.6/hdf5/include/ Now it works like a charm! Thanks again for your help. Kind regards, Steven -----Original Message----- From: petsc-users-bounces at mcs.anl.gov on behalf of Jed Brown Sent: Sun 3/4/2012 3:15 PM To: PETSc users list Subject: Re: [petsc-users] HDF5 installation problems On Sun, Mar 4, 2012 at 05:43, Haren, S.W. van (Steven) wrote: > Dear Jed, > > Thanks for your reply. > > These are the last lines from the configure.log: > > sh: cd /home/steven/C++/PETSc/petsc-3.2-p6/externalpackages/hdf5-1.8.6 && > make clean && make && make install > Executing: cd > /home/steven/C++/PETSc/petsc-3.2-p6/externalpackages/hdf5-1.8.6 && make > clean && make && make install > Runaway process exceeded time limit of 2500s > Possibilities: 1. System clock was changed or the machine was suspended, causing incorrect timeout. (As an aside, the Python docs are horrible. They act like time is one thing, so I have to read the source code to find out that threading.Thread.join() implements a timeout using time(2) which is neither monotone nor continuous.) 2. The compilation actually takes a long time on this machine, perhaps because it doesn't get much time to run due to other jobs. You could try increasing the timeout or running make manually from externalpackages/hdf5-1.8.6/. 3. The hdf5 build process is getting stuck somewhere, I suggest trying to complete the install manually from externalpackages/hdf5-1.8.6/. If you can't get it working, send configure.log and HDF5 attempted build logs to petsc-maint at mcs.anl.gov. > sh: > > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > > ------------------------------------------------------------------------------- > Error running make on HDF5: Could not execute "cd > /home/steven/C++/PETSc/petsc-3.2-p6/externalpackages/hdf5-1.8.6 && make > clean && make && make install": > Runaway process exceeded time limit of 2500s > > ******************************************************************************* > File "./config/configure.py", line 283, in petsc_configure > framework.configure(out = sys.stdout) > File > "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/framework.py", > line 925, in configure > child.configure() > File > "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/package.py", > line 506, in configure > self.executeTest(self.configureLibrary) > File > "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/base.py", > line 115, in executeTest > ret = apply(test, args,kargs) > File > "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/packages/hdf5.py", > line 63, in configureLibrary > config.package.Package.configureLibrary(self) > File > "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/package.py", > line 433, in configureLibrary > for location, directory, lib, incl in self.generateGuesses(): > File > "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/package.py", > line 228, in generateGuesses > d = self.checkDownload(1) > File > "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/package.py", > line 313, in checkDownload > return self.getInstallDir() > File > "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/package.py", > line 183, in getInstallDir > return os.path.abspath(self.Install()) > File > "/home/steven/C++/PETSc/petsc-3.2-p6/config/BuildSystem/config/packages/hdf5.py", > line 56, in Install > raise RuntimeError('Error running make on HDF5: '+str(e)) > > Using the top command I cannot really identify a process that uses all the > memory. The memory use increases untill completely full. But the maximum > percentage used by any process does not go above 10 percent. Even when the > compilation is stopped the memory is not freed. I don't really understand > this. Could you give me a few pointers based on the configure.log output? > > Hoe should I attach a debugger? Any good links were I can read how to do > that? > > Thanks in advance. > > Kind regards, > > Steven > > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov on behalf of Jed Brown > Sent: Wed 2/29/2012 10:43 PM > To: PETSc users list > Subject: Re: [petsc-users] HDF5 installation problems > > On Wed, Feb 29, 2012 at 15:35, Haren, S.W. van (Steven) >wrote: > > > Dear all, > > > > I try to configure Petsc to use HDF5. > > > > During the HDF5 compilation stage the memory use increases untill the > > computer basically stalls and almost all memory is used. After 2500s the > > compilation is stopped because of a runaway process. > > > > Check configure.log and/or attach a debugger to find out where it got > stuck. At least use "top" to find out which process is taking all the time > and memory. > > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/ms-tnef Size: 5689 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Tue Mar 6 14:14:48 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 6 Mar 2012 14:14:48 -0600 Subject: [petsc-users] Projection preconditioner as PCMG In-Reply-To: <1331053255.72750.YahooMailNeo@web28504.mail.ukl.yahoo.com> References: <1331045458.64662.YahooMailNeo@web28501.mail.ukl.yahoo.com> <1331053255.72750.YahooMailNeo@web28504.mail.ukl.yahoo.com> Message-ID: On Tue, Mar 6, 2012 at 11:00, Abdul Hanan Sheikh wrote: > I make my query simpler: how can I implement a preconditioner of form > > P = I - A*P*(A_H)^-1 * R > with any Krylov , where > I : Identity > P : Prolongation > A_H : Galerkin coarse matrix > R : Restriction, > > Can I exploit the CGC part of PCMG , by making its pre and post smoother > null ? > You use P twice in the above, but I think I see what you mean. You can certainly use PCMG with no pre- or post-smoother or you could do the same here with PCComposite, but you probably don't want that extra application of A in the preconditioner unless you are going to use a post-smoother. That is, suppose C = P*A_H^{-1}*R is the coarse grid solver. Then with a post-smoother S, application of the preconditioner to the vector b would be C b + S (b - A C b). If you drop the post-smoother by setting S = I, you get C b + b - A C b. Keeping the extra multiplication with A and not using a post-smoother will likely destabilize the method. As an example, take src/snes/examples/tutorials/ex5.c and just look at the first linear solve with 2-level method. First, with an additive coarse grid correction and no fine grid smoother (i.e. the preconditioner is I + C where C is an Prolongate*Exact_Coarse_Solve*Restrict). $ ./ex5 -da_grid_x 201 -da_grid_y 201 -snes_max_it 1 -ksp_monitor_singular_value -pc_type mg -pc_mg_levels 2 -pc_mg_type additive -mg_levels_ksp_type preonly -mg_levels_pc_type none 0 KSP Residual norm 1.991818008559e+01 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 1 KSP Residual norm 8.562399481020e+00 % max 9.703742797548e-01 min 9.703742797548e-01 max/min 1.000000000000e+00 2 KSP Residual norm 1.547282124742e+00 % max 4.106911489507e+00 min 9.455565835409e-01 max/min 4.343379931984e+00 3 KSP Residual norm 5.193012838962e-01 % max 7.849972917628e+00 min 9.390711734984e-01 max/min 8.359294949267e+00 4 KSP Residual norm 1.844773292053e-01 % max 8.102437955088e+00 min 9.157291371635e-01 max/min 8.848072673744e+00 5 KSP Residual norm 6.904332859839e-02 % max 8.102847554075e+00 min 8.906430581537e-01 max/min 9.097749631454e+00 6 KSP Residual norm 3.747280928850e-02 % max 8.131998087718e+00 min 8.796142392701e-01 max/min 9.244959579630e+00 7 KSP Residual norm 1.781146506612e-02 % max 8.153370848515e+00 min 8.294602459028e-01 max/min 9.829730705949e+00 8 KSP Residual norm 7.506605928852e-03 % max 8.160488227056e+00 min 8.035220795876e-01 max/min 1.015589793282e+01 9 KSP Residual norm 3.771683771569e-03 % max 8.185296438244e+00 min 7.986463109034e-01 max/min 1.024896293452e+01 10 KSP Residual norm 1.900344739083e-03 % max 8.246526708637e+00 min 7.835579388152e-01 max/min 1.052446322107e+01 11 KSP Residual norm 8.969235041181e-04 % max 8.246542749144e+00 min 7.698718962697e-01 max/min 1.071157784704e+01 12 KSP Residual norm 3.771331016150e-04 % max 8.253249205705e+00 min 7.665454362492e-01 max/min 1.076681017904e+01 13 KSP Residual norm 1.998592159889e-04 % max 8.323950632412e+00 min 7.632295016946e-01 max/min 1.090622232753e+01 14 KSP Residual norm 9.624660396860e-05 % max 8.327354837686e+00 min 7.567183120253e-01 max/min 1.100456366042e+01 This is identical with one step of Richardson (-mg_levels_ksp_type richardson) because the "initial guess" is 0. The convergence is somewhat better if we improve the additive fine-grid smoother, e.g. ILU or SOR, though the condition number is actually worse in this case (because the preconditioned operator is now non-normal, the eigenvalues are well-clustered). $ ./ex5 -da_grid_x 201 -da_grid_y 201 -snes_max_it 1 -ksp_monitor_singular_value -pc_type mg -pc_mg_levels 2 -pc_mg_type additive -mg_levels_ksp_type preonly -mg_levels_pc_type ilu 0 KSP Residual norm 1.997267177320e+01 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 1 KSP Residual norm 5.755612216067e+00 % max 9.658916552787e-01 min 9.658916552787e-01 max/min 1.000000000000e+00 2 KSP Residual norm 2.828695990580e-01 % max 2.491505412300e+00 min 9.583727874764e-01 max/min 2.599724705102e+00 3 KSP Residual norm 1.097671648754e-01 % max 7.900124346250e+00 min 5.695048906547e-01 max/min 1.387191659964e+01 4 KSP Residual norm 2.203949856452e-02 % max 9.291338218054e+00 min 3.581794670449e-01 max/min 2.594045464055e+01 5 KSP Residual norm 8.497097452426e-03 % max 1.013788428797e+01 min 3.272689944775e-01 max/min 3.097722197653e+01 6 KSP Residual norm 1.376443389574e-03 % max 1.055586782597e+01 min 2.933041209977e-01 max/min 3.598949714743e+01 7 KSP Residual norm 2.753520742081e-04 % max 1.061464021647e+01 min 2.893817762727e-01 max/min 3.668040314489e+01 8 KSP Residual norm 6.552848547677e-05 % max 1.062831125267e+01 min 2.883290857853e-01 max/min 3.686173812025e+01 "Kaskade" multigrid applies no pre-smoother, but uses a multiplicative post-smoother. That is what I wrote above and probably what you wanted. $ ./ex5 -da_grid_x 201 -da_grid_y 201 -snes_max_it 1 -ksp_monitor_singular_value -pc_type mg -pc_mg_levels 2 -pc_mg_type kaskade -mg_levels_ksp_type richardson -mg_levels_pc_type none 0 KSP Residual norm 1.981089588505e+01 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 1 KSP Residual norm 3.481667117996e+00 % max 9.904507054699e-01 min 9.904507054699e-01 max/min 1.000000000000e+00 2 KSP Residual norm 1.652167261481e+00 % max 5.608992736195e+00 min 9.879184804293e-01 max/min 5.677586609937e+00 3 KSP Residual norm 1.627219442360e+00 % max 6.798397388800e+00 min 9.550178771471e-01 max/min 7.118607464301e+00 4 KSP Residual norm 7.880741343988e-01 % max 7.451381264321e+00 min 7.145924145893e-01 max/min 1.042745642438e+01 5 KSP Residual norm 3.537498074218e-01 % max 7.663873901737e+00 min 5.751738420032e-01 max/min 1.332444791829e+01 6 KSP Residual norm 1.078704665358e-01 % max 7.800309555756e+00 min 5.120187805734e-01 max/min 1.523442079023e+01 7 KSP Residual norm 4.407455476078e-02 % max 7.855365072615e+00 min 5.037708740493e-01 max/min 1.559313068156e+01 8 KSP Residual norm 2.107104852559e-02 % max 7.893144033146e+00 min 5.025484383148e-01 max/min 1.570623532254e+01 9 KSP Residual norm 1.763159978441e-02 % max 7.912218583050e+00 min 5.024504696471e-01 max/min 1.574726079688e+01 10 KSP Residual norm 1.745405855993e-02 % max 7.928099602729e+00 min 5.023138582042e-01 max/min 1.578315922056e+01 11 KSP Residual norm 1.376977330994e-02 % max 7.939644155950e+00 min 5.017479535724e-01 max/min 1.582396918497e+01 12 KSP Residual norm 9.446347797152e-03 % max 7.949376050758e+00 min 5.015718142316e-01 max/min 1.584892895734e+01 13 KSP Residual norm 6.843342509359e-03 % max 7.956996132919e+00 min 4.969226601251e-01 max/min 1.601254434828e+01 14 KSP Residual norm 6.437072325338e-03 % max 7.961936315836e+00 min 4.932296836382e-01 max/min 1.614245164059e+01 15 KSP Residual norm 6.091589527973e-03 % max 7.966811508176e+00 min 4.932019218895e-01 max/min 1.615324505966e+01 16 KSP Residual norm 4.652185214024e-03 % max 7.970648347219e+00 min 4.916668205282e-01 max/min 1.621148309064e+01 17 KSP Residual norm 3.701849685772e-03 % max 7.973893800218e+00 min 4.915344876340e-01 max/min 1.622245030781e+01 18 KSP Residual norm 3.482133374032e-03 % max 7.976475806296e+00 min 4.911811815837e-01 max/min 1.623937582580e+01 19 KSP Residual norm 3.465608263872e-03 % max 7.978688479804e+00 min 4.898709616560e-01 max/min 1.628732687652e+01 20 KSP Residual norm 3.149472026786e-03 % max 7.980652326863e+00 min 4.891722853032e-01 max/min 1.631460441778e+01 21 KSP Residual norm 2.763464672775e-03 % max 7.982383839733e+00 min 4.891419493795e-01 max/min 1.631915612607e+01 22 KSP Residual norm 2.689646479190e-03 % max 7.983807057995e+00 min 4.876132849055e-01 max/min 1.637323531811e+01 23 KSP Residual norm 2.661061400431e-03 % max 7.985119113668e+00 min 4.836465184065e-01 max/min 1.651023797292e+01 24 KSP Residual norm 2.383953808700e-03 % max 7.986314359135e+00 min 4.718949191329e-01 max/min 1.692392529636e+01 ^C Okay, that didn't work. But it was because the Richardson step was too large. We can get away without a traditional smoothing step by fixing the relaxation parameter for Richardson: $ ./ex5 -da_grid_x 201 -da_grid_y 201 -snes_max_it 1 -ksp_monitor_singular_value -pc_type mg -pc_mg_levels 2 -pc_mg_type kaskade -mg_levels_ksp_type richardson -mg_levels_pc_type none -mg_levels_ksp_richardson_self_scale 0 KSP Residual norm 1.975857018746e+01 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 1 KSP Residual norm 2.476038171898e+00 % max 9.622055302029e-01 min 9.622055302029e-01 max/min 1.000000000000e+00 2 KSP Residual norm 3.723935286980e-02 % max 1.019358214307e+00 min 8.790661889416e-01 max/min 1.159592107091e+00 3 KSP Residual norm 1.048082002961e-02 % max 2.231045830559e+00 min 2.203725693734e-01 max/min 1.012397249305e+01 4 KSP Residual norm 4.815690407780e-03 % max 3.167321755561e+00 min 1.859713575290e-01 max/min 1.703123425911e+01 5 KSP Residual norm 2.282674490066e-03 % max 5.220845319337e+00 min 1.603210979291e-01 max/min 3.256492992360e+01 6 KSP Residual norm 5.030060118813e-04 % max 6.078990414170e+00 min 1.535348150609e-01 max/min 3.959356326940e+01 7 KSP Residual norm 2.084516564815e-04 % max 6.195047474999e+00 min 1.466558297789e-01 max/min 4.224208123426e+01 8 KSP Residual norm 6.654981023267e-05 % max 6.379643313848e+00 min 1.442472810392e-01 max/min 4.422713043801e+01 Unfortunately, convergence is very sensitive to that step and for variable coefficient problems, we won't be able to choose a good value. If we know the matrix diagonal, we can get a lot more robustness by post-smoothing with Jacobi, and now we don't need that pesky scaling parameter. $ ./ex5 -da_grid_x 201 -da_grid_y 201 -snes_max_it 1 -ksp_monitor_singular_value -pc_type mg -pc_mg_levels 2 -pc_mg_type kaskade -mg_levels_ksp_type richardson -mg_levels_pc_type jacobi 0 KSP Residual norm 1.975846409514e+01 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 1 KSP Residual norm 2.482202615145e+00 % max 9.621966917526e-01 min 9.621966917526e-01 max/min 1.000000000000e+00 2 KSP Residual norm 4.457401425338e-02 % max 1.020264072916e+00 min 8.821750286291e-01 max/min 1.156532479163e+00 3 KSP Residual norm 1.637472997384e-02 % max 1.776622548172e+00 min 7.524631434334e-01 max/min 2.361075839629e+00 4 KSP Residual norm 3.644899164209e-03 % max 1.907783379671e+00 min 4.245698995769e-01 max/min 4.493449445127e+00 5 KSP Residual norm 1.443735788456e-03 % max 1.935858887793e+00 min 3.898702611314e-01 max/min 4.965392544111e+00 6 KSP Residual norm 3.929697132002e-04 % max 1.961850264833e+00 min 3.669296462786e-01 max/min 5.346666001861e+00 7 KSP Residual norm 1.446207699005e-04 % max 1.970859698383e+00 min 3.614358036844e-01 max/min 5.452862384670e+00 -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Tue Mar 6 17:22:15 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Tue, 6 Mar 2012 18:22:15 -0500 Subject: [petsc-users] petsc4py KSP Question Message-ID: Hello I'm in the process of using petsc4py to solve a large multidisciplinary, non-linear system and its adjoint. I have the non-linear solution with snes() working correctly and I'm now doing the linear solution. For the non-linear solve, I create the snes and set my user context as follows: # Create SNES Object ASContext = ASNKSolver(self) # Context to hold data snes = PETSc.SNES().createPython(ASContext, comm=self.gcomm) snes.setFunction(ASContext.formFunction, resVec) For the linear part, I'm a little confused. I currently have # Create Python Context and KSP Object KSPContext = AdjointKSPSolver(self) # Context to hold data ksp = PETSc.KSP().createPython(KSPContext, comm=self.gcomm) What I'm not sure of is how to specify the operator for the KSP solver in the KSPContext object. Is this possible? Is there something like def apply() you must do? The petsc4py poisson2d.py example, first creates a Mat, then sets a context for that, and then uses ksp.setOperators() to use that matrix. Is this the only way to do it? If, so, what is the use of the PETSc.KSP().createPython() command? Thank you, Gaetan -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Wed Mar 7 01:54:01 2012 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Wed, 7 Mar 2012 07:54:01 +0000 Subject: [petsc-users] segv in VecSum when using nested vec Message-ID: > > >Could you get a stack trace with gdb? Also, I ran on petsc-dev. > > > > Breakpoint 1, vecsumsegv::vecsumsegv (this=0x7fffffffd310, m=3, n=4) > > at vecsumsegv.cc:28 > > 28 ierr = VecSum(x,&val); > > (gdb) next > > > This was fixed in petsc-dev. > > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/3b9bc425b217 Thanks, I'll switch to petsc-dev for this. > > > Why are you using VecNest? It causes more harm than good in all but some > very special circumstances. I'm using it to experiment with PETSc's block preconditioners. In short, I have a block matrix for the 2D Stokes eqs with the variables arranged as (u1,...,uN,v1,...,vN,p1,...pN). The matrix has the form A = [Q G, D 0]. I'm using VecCreateNest to assemble the vector x = [u p], just like I use MatCreateNest to assemble the matrix A from the blocks Q, G and D. The index sets are the ones you helped me with in a previous thread (https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2012-February/012262.html) dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From dalcinl at gmail.com Wed Mar 7 02:16:39 2012 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 7 Mar 2012 11:16:39 +0300 Subject: [petsc-users] petsc4py KSP Question In-Reply-To: References: Message-ID: On 7 March 2012 02:22, Gaetan Kenway wrote: > Hello > > I'm in the process of using petsc4py to solve a large multidisciplinary, > non-linear system and its adjoint. I have the non-linear solution with > snes() working correctly and I'm now doing the linear solution. > > For the non-linear solve, I create the snes and set my user context as > follows: > ? ?# Create SNES Object > ? ASContext = ASNKSolver(self) # Context to hold data > ? snes = PETSc.SNES().createPython(ASContext, comm=self.gcomm) > ? snes.setFunction(ASContext.formFunction, resVec) > > For the linear part, I'm a little confused. I currently have > > ? ? # Create Python Context and KSP Object > ? ? KSPContext = AdjointKSPSolver(self) # Context to hold data > ? ? ksp = PETSc.KSP().createPython(KSPContext, comm=self.gcomm) > > What I'm not sure of is how to specify the operator for the KSP solver in > the KSPContext object. Is this possible? Is there something like def apply() > you must do? > > The petsc4py poisson2d.py example, first creates a Mat, then sets a context > for that, and then uses ksp.setOperators() to use that matrix. Is this the > only way to do it? If, so, what is the use of > the??PETSc.KSP().createPython() command? > Unless you want to implement a custom KSP/SNES in Python, you should just use KSP/SNES.create(). KSP/SNES.createPython() is a rather advanced feature, it is not obvious at all how to use them, there are no demos nor documentation about it (however, there is some code in test/test_{ksp|snes}_py.py ). Please take a look at the demo/bratu2d and demo/bratu3d about how to setup a SNES solver. For KSP, take a look at demo/kspsolve/petsc-mat.py and demo/kspsolve/petsc-ksp.py -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From knepley at gmail.com Wed Mar 7 06:53:23 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 7 Mar 2012 06:53:23 -0600 Subject: [petsc-users] segv in VecSum when using nested vec In-Reply-To: References: Message-ID: On Wed, Mar 7, 2012 at 1:54 AM, Klaij, Christiaan wrote: > > > >Could you get a stack trace with gdb? Also, I ran on petsc-dev. > > > > > > Breakpoint 1, vecsumsegv::vecsumsegv (this=0x7fffffffd310, m=3, n=4) > > > at vecsumsegv.cc:28 > > > 28 ierr = VecSum(x,&val); > > > (gdb) next > > > > > > This was fixed in petsc-dev. > > > > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/3b9bc425b217 > > Thanks, I'll switch to petsc-dev for this. > > > > > > > Why are you using VecNest? It causes more harm than good in all but some > > very special circumstances. > > I'm using it to experiment with PETSc's block preconditioners. In > short, I have a block matrix for the 2D Stokes eqs with the > variables arranged as (u1,...,uN,v1,...,vN,p1,...pN). The matrix > has the form A = [Q G, D 0]. I'm using VecCreateNest to assemble > the vector x = [u p], just like I use MatCreateNest to assemble > the matrix A from the blocks Q, G and D. The index sets are the > ones you helped me with in a previous thread > ( > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2012-February/012262.html > ) > This is just an optimization for block PCs, and for vectors I think it makes almost no difference. Matt > dr. ir. Christiaan Klaij > CFD Researcher > Research & Development > E mailto:C.Klaij at marin.nl > T +31 317 49 33 44 > > MARIN > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Mar 7 07:07:27 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 7 Mar 2012 07:07:27 -0600 Subject: [petsc-users] segv in VecSum when using nested vec In-Reply-To: References: Message-ID: On Wed, Mar 7, 2012 at 06:53, Matthew Knepley wrote: > > Why are you using VecNest? It causes more harm than good in all but some >> > very special circumstances. >> >> I'm using it to experiment with PETSc's block preconditioners. In >> short, I have a block matrix for the 2D Stokes eqs with the >> variables arranged as (u1,...,uN,v1,...,vN,p1,...pN). The matrix >> has the form A = [Q G, D 0]. I'm using VecCreateNest to assemble >> the vector x = [u p], just like I use MatCreateNest to assemble >> the matrix A from the blocks Q, G and D. The index sets are the >> ones you helped me with in a previous thread >> ( >> https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2012-February/012262.html >> ) >> > > This is just an optimization for block PCs, and for vectors I think it > makes almost > no difference. > Agreed, I recommend using normal Vecs. You can access part of it with VecGetSubVector() if you want to set values for each part separately (it will be no-copy if possible). VecNest only matters if you insist on storing the subvectors as non-contiguous parts of the composite Vec, but also insist on not making copies. Even then, since VecScatter does not have native support for VecNest, there will be some copies in the preconditioner (but they could eventually be supported without copies, note that it's mostly moot because you need a very specialized scenario to make copies account for more than 5% of run-time). So you should basically never use VecNest unless you are really sure that it's the right thing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Wed Mar 7 08:18:42 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Wed, 7 Mar 2012 09:18:42 -0500 Subject: [petsc-users] petsc4py KSP Question In-Reply-To: References: Message-ID: Thanks. I have both the SNES and the linear KSP solver working the way I want. One more question: At the beginning of my non-linear SNES function I have something like def formFunction(self, snes, X, F): states = X.getArray() and a X.resetArray() at the end. Am I correct to assume that this is an efficient pointer assignment to the numpy array and does not involve a memory copy? Thanks Gaetan On Wed, Mar 7, 2012 at 3:16 AM, Lisandro Dalcin wrote: > On 7 March 2012 02:22, Gaetan Kenway wrote: > > Hello > > > > I'm in the process of using petsc4py to solve a large multidisciplinary, > > non-linear system and its adjoint. I have the non-linear solution with > > snes() working correctly and I'm now doing the linear solution. > > > > For the non-linear solve, I create the snes and set my user context as > > follows: > > # Create SNES Object > > ASContext = ASNKSolver(self) # Context to hold data > > snes = PETSc.SNES().createPython(ASContext, comm=self.gcomm) > > snes.setFunction(ASContext.formFunction, resVec) > > > > For the linear part, I'm a little confused. I currently have > > > > # Create Python Context and KSP Object > > KSPContext = AdjointKSPSolver(self) # Context to hold data > > ksp = PETSc.KSP().createPython(KSPContext, comm=self.gcomm) > > > > What I'm not sure of is how to specify the operator for the KSP solver in > > the KSPContext object. Is this possible? Is there something like def > apply() > > you must do? > > > > The petsc4py poisson2d.py example, first creates a Mat, then sets a > context > > for that, and then uses ksp.setOperators() to use that matrix. Is this > the > > only way to do it? If, so, what is the use of > > the PETSc.KSP().createPython() command? > > > > Unless you want to implement a custom KSP/SNES in Python, you should > just use KSP/SNES.create(). KSP/SNES.createPython() is a rather > advanced feature, it is not obvious at all how to use them, there are > no demos nor documentation about it (however, there is some code in > test/test_{ksp|snes}_py.py ). > > Please take a look at the demo/bratu2d and demo/bratu3d about how to > setup a SNES solver. For KSP, take a look at > demo/kspsolve/petsc-mat.py and demo/kspsolve/petsc-ksp.py > > > > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Mar 7 08:22:59 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 7 Mar 2012 08:22:59 -0600 Subject: [petsc-users] petsc4py KSP Question In-Reply-To: References: Message-ID: On Wed, Mar 7, 2012 at 8:18 AM, Gaetan Kenway wrote: > Thanks. I have both the SNES and the linear KSP solver working the way I > want. > > One more question: At the beginning of my non-linear SNES function I have > something like > > def formFunction(self, snes, X, F): > > states = X.getArray() > > and a X.resetArray() at the end. Am I correct to assume that this is an > efficient pointer assignment to the numpy array and does not involve a > memory copy? > It is no-copy. You can use a 'with block, which is a little nicer, and I think you mean restoreArray(). Matt > Thanks > > Gaetan > > > > On Wed, Mar 7, 2012 at 3:16 AM, Lisandro Dalcin wrote: > >> On 7 March 2012 02:22, Gaetan Kenway wrote: >> > Hello >> > >> > I'm in the process of using petsc4py to solve a large multidisciplinary, >> > non-linear system and its adjoint. I have the non-linear solution with >> > snes() working correctly and I'm now doing the linear solution. >> > >> > For the non-linear solve, I create the snes and set my user context as >> > follows: >> > # Create SNES Object >> > ASContext = ASNKSolver(self) # Context to hold data >> > snes = PETSc.SNES().createPython(ASContext, comm=self.gcomm) >> > snes.setFunction(ASContext.formFunction, resVec) >> > >> > For the linear part, I'm a little confused. I currently have >> > >> > # Create Python Context and KSP Object >> > KSPContext = AdjointKSPSolver(self) # Context to hold data >> > ksp = PETSc.KSP().createPython(KSPContext, comm=self.gcomm) >> > >> > What I'm not sure of is how to specify the operator for the KSP solver >> in >> > the KSPContext object. Is this possible? Is there something like def >> apply() >> > you must do? >> > >> > The petsc4py poisson2d.py example, first creates a Mat, then sets a >> context >> > for that, and then uses ksp.setOperators() to use that matrix. Is this >> the >> > only way to do it? If, so, what is the use of >> > the PETSc.KSP().createPython() command? >> > >> >> Unless you want to implement a custom KSP/SNES in Python, you should >> just use KSP/SNES.create(). KSP/SNES.createPython() is a rather >> advanced feature, it is not obvious at all how to use them, there are >> no demos nor documentation about it (however, there is some code in >> test/test_{ksp|snes}_py.py ). >> >> Please take a look at the demo/bratu2d and demo/bratu3d about how to >> setup a SNES solver. For KSP, take a look at >> demo/kspsolve/petsc-mat.py and demo/kspsolve/petsc-ksp.py >> >> >> >> -- >> Lisandro Dalcin >> --------------- >> CIMEC (INTEC/CONICET-UNL) >> Predio CONICET-Santa Fe >> Colectora RN 168 Km 472, Paraje El Pozo >> 3000 Santa Fe, Argentina >> Tel: +54-342-4511594 (ext 1011) >> Tel/Fax: +54-342-4511169 >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Wed Mar 7 08:49:21 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Wed, 7 Mar 2012 09:49:21 -0500 Subject: [petsc-users] petsc4py KSP Question In-Reply-To: References: Message-ID: Thanks for the reply. What is the 'with block' option. I don't see anything related to that when I run help(PETSc.Vec()). Does it return a (N/bs,bs) array instead of an (N) array? Also in petsc4py the call for resetting the pointer from getArray() is actually resetArray() which is different from C/Fortran where it is restoreArray(). I've got them confused as well. Gaetan On Wed, Mar 7, 2012 at 9:22 AM, Matthew Knepley wrote: > On Wed, Mar 7, 2012 at 8:18 AM, Gaetan Kenway wrote: > >> Thanks. I have both the SNES and the linear KSP solver working the way I >> want. >> >> One more question: At the beginning of my non-linear SNES function I have >> something like >> >> def formFunction(self, snes, X, F): >> >> states = X.getArray() >> >> and a X.resetArray() at the end. Am I correct to assume that this is an >> efficient pointer assignment to the numpy array and does not involve a >> memory copy? >> > > It is no-copy. You can use a 'with block, which is a little nicer, and I > think you mean restoreArray(). > > Matt > > >> Thanks >> >> Gaetan >> >> >> >> On Wed, Mar 7, 2012 at 3:16 AM, Lisandro Dalcin wrote: >> >>> On 7 March 2012 02:22, Gaetan Kenway wrote: >>> > Hello >>> > >>> > I'm in the process of using petsc4py to solve a large >>> multidisciplinary, >>> > non-linear system and its adjoint. I have the non-linear solution with >>> > snes() working correctly and I'm now doing the linear solution. >>> > >>> > For the non-linear solve, I create the snes and set my user context as >>> > follows: >>> > # Create SNES Object >>> > ASContext = ASNKSolver(self) # Context to hold data >>> > snes = PETSc.SNES().createPython(ASContext, comm=self.gcomm) >>> > snes.setFunction(ASContext.formFunction, resVec) >>> > >>> > For the linear part, I'm a little confused. I currently have >>> > >>> > # Create Python Context and KSP Object >>> > KSPContext = AdjointKSPSolver(self) # Context to hold data >>> > ksp = PETSc.KSP().createPython(KSPContext, comm=self.gcomm) >>> > >>> > What I'm not sure of is how to specify the operator for the KSP solver >>> in >>> > the KSPContext object. Is this possible? Is there something like def >>> apply() >>> > you must do? >>> > >>> > The petsc4py poisson2d.py example, first creates a Mat, then sets a >>> context >>> > for that, and then uses ksp.setOperators() to use that matrix. Is this >>> the >>> > only way to do it? If, so, what is the use of >>> > the PETSc.KSP().createPython() command? >>> > >>> >>> Unless you want to implement a custom KSP/SNES in Python, you should >>> just use KSP/SNES.create(). KSP/SNES.createPython() is a rather >>> advanced feature, it is not obvious at all how to use them, there are >>> no demos nor documentation about it (however, there is some code in >>> test/test_{ksp|snes}_py.py ). >>> >>> Please take a look at the demo/bratu2d and demo/bratu3d about how to >>> setup a SNES solver. For KSP, take a look at >>> demo/kspsolve/petsc-mat.py and demo/kspsolve/petsc-ksp.py >>> >>> >>> >>> -- >>> Lisandro Dalcin >>> --------------- >>> CIMEC (INTEC/CONICET-UNL) >>> Predio CONICET-Santa Fe >>> Colectora RN 168 Km 472, Paraje El Pozo >>> 3000 Santa Fe, Argentina >>> Tel: +54-342-4511594 (ext 1011) >>> Tel/Fax: +54-342-4511169 >>> >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Mar 7 09:14:13 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 7 Mar 2012 09:14:13 -0600 Subject: [petsc-users] petsc4py KSP Question In-Reply-To: References: Message-ID: On Wed, Mar 7, 2012 at 08:49, Gaetan Kenway wrote: > What is the 'with block' option. I don't see anything related to that when > I run help(PETSc.Vec()). Does it return a (N/bs,bs) array instead of an (N) > array? with X as x: print(numpy.sin(x)) # x is a Numpy array > Also in petsc4py the call for resetting the pointer from getArray() is > actually resetArray() which is different from C/Fortran where it is > restoreArray(). I've got them confused as well. No, resetArray() is VecResetArray() which is a different thing, getArray() in Python does not need an explicit restore. (I believe the restore is called when the "gotten" array falls out of scope.) I recommend using the 'with' statement, it's much clearer and more explicit about resource management. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Wed Mar 7 15:04:00 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Wed, 7 Mar 2012 16:04:00 -0500 Subject: [petsc-users] petsc4py KSP Question In-Reply-To: References: Message-ID: Thanks for the info. I've recoded the functions using the 'with' statements and I agree is makes it more explicit the scope of the temporary numpy array. Would the following snippet of code be the preferred way to use the petsc vectors in python? Thanks, Gaetan class ASPC(object): def apply(self, pc, X, Y): """y <-- M^-1 * x""" # Extract Vector Pointers with X as x: with Y as y: # Apply preconditioners in parallel if self.AS.isAero: y = self.AS.solver.globalNKPreCon(x, y) if self.AS.isStruct: y = self.AS.solver.globalNKPreCon(x, y) On Wed, Mar 7, 2012 at 10:14 AM, Jed Brown wrote: > On Wed, Mar 7, 2012 at 08:49, Gaetan Kenway wrote: > >> What is the 'with block' option. I don't see anything related to that >> when I run help(PETSc.Vec()). Does it return a (N/bs,bs) array instead of >> an (N) array? > > > with X as x: > print(numpy.sin(x)) # x is a Numpy array > > >> Also in petsc4py the call for resetting the pointer from getArray() is >> actually resetArray() which is different from C/Fortran where it is >> restoreArray(). I've got them confused as well. > > > No, resetArray() is VecResetArray() which is a different thing, getArray() > in Python does not need an explicit restore. (I believe the restore is > called when the "gotten" array falls out of scope.) I recommend using the > 'with' statement, it's much clearer and more explicit about resource > management. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Mar 7 15:08:38 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 7 Mar 2012 15:08:38 -0600 Subject: [petsc-users] petsc4py KSP Question In-Reply-To: References: Message-ID: On Wed, Mar 7, 2012 at 3:04 PM, Gaetan Kenway wrote: > Thanks for the info. I've recoded the functions using the 'with' > statements and I agree is makes it more explicit the scope of the temporary > numpy array. Would the following snippet of code be the preferred way to > use the petsc vectors in python? 1) You can use with X as x, Y as y: 2) For linear algebra, this is fine. For things with topology, you generally want "ghost" regions, so you first get a local vector, instead of the global vector, and then pull out the array. Matt > Thanks, > Gaetan > > class ASPC(object): > def apply(self, pc, X, Y): > """y <-- M^-1 * x""" > > # Extract Vector Pointers > with X as x: > with Y as y: > > # Apply preconditioners in parallel > if self.AS.isAero: > y = self.AS.solver.globalNKPreCon(x, y) > > if self.AS.isStruct: > y = self.AS.solver.globalNKPreCon(x, y) > > > > On Wed, Mar 7, 2012 at 10:14 AM, Jed Brown wrote: > >> On Wed, Mar 7, 2012 at 08:49, Gaetan Kenway wrote: >> >>> What is the 'with block' option. I don't see anything related to that >>> when I run help(PETSc.Vec()). Does it return a (N/bs,bs) array instead of >>> an (N) array? >> >> >> with X as x: >> print(numpy.sin(x)) # x is a Numpy array >> >> >>> Also in petsc4py the call for resetting the pointer from getArray() is >>> actually resetArray() which is different from C/Fortran where it is >>> restoreArray(). I've got them confused as well. >> >> >> No, resetArray() is VecResetArray() which is a different thing, >> getArray() in Python does not need an explicit restore. (I believe the >> restore is called when the "gotten" array falls out of scope.) I recommend >> using the 'with' statement, it's much clearer and more explicit about >> resource management. >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Wed Mar 7 15:32:43 2012 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Wed, 7 Mar 2012 16:32:43 -0500 Subject: [petsc-users] petsc4py KSP Question In-Reply-To: References: Message-ID: Excellent. Thanks I think the "with X as x, Y as y" syntax only works with python 2.7 and up and one computer I run this on still has 2.6, so I'll leave them nested. In my particular application I'm using petsc4py to solve a coupled problem with two codes that never talk directly to each other. Each code takes care of its own topology, communication etc, so my petsc4py objects just have the owned rows with no halos. Thanks for all your help Gaetan On Wed, Mar 7, 2012 at 4:08 PM, Matthew Knepley wrote: > On Wed, Mar 7, 2012 at 3:04 PM, Gaetan Kenway wrote: > >> Thanks for the info. I've recoded the functions using the 'with' >> statements and I agree is makes it more explicit the scope of the temporary >> numpy array. Would the following snippet of code be the preferred way to >> use the petsc vectors in python? > > > 1) You can use with X as x, Y as y: > > 2) For linear algebra, this is fine. For things with topology, you > generally want "ghost" regions, so you > first get a local vector, instead of the global vector, and then pull > out the array. > > Matt > > >> Thanks, >> Gaetan >> >> class ASPC(object): >> def apply(self, pc, X, Y): >> """y <-- M^-1 * x""" >> >> # Extract Vector Pointers >> with X as x: >> with Y as y: >> >> # Apply preconditioners in parallel >> if self.AS.isAero: >> y = self.AS.solver.globalNKPreCon(x, y) >> >> if self.AS.isStruct: >> y = self.AS.solver.globalNKPreCon(x, y) >> >> >> >> On Wed, Mar 7, 2012 at 10:14 AM, Jed Brown wrote: >> >>> On Wed, Mar 7, 2012 at 08:49, Gaetan Kenway wrote: >>> >>>> What is the 'with block' option. I don't see anything related to that >>>> when I run help(PETSc.Vec()). Does it return a (N/bs,bs) array instead of >>>> an (N) array? >>> >>> >>> with X as x: >>> print(numpy.sin(x)) # x is a Numpy array >>> >>> >>>> Also in petsc4py the call for resetting the pointer from getArray() is >>>> actually resetArray() which is different from C/Fortran where it is >>>> restoreArray(). I've got them confused as well. >>> >>> >>> No, resetArray() is VecResetArray() which is a different thing, >>> getArray() in Python does not need an explicit restore. (I believe the >>> restore is called when the "gotten" array falls out of scope.) I recommend >>> using the 'with' statement, it's much clearer and more explicit about >>> resource management. >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From samuelandjw at gmail.com Thu Mar 8 01:29:24 2012 From: samuelandjw at gmail.com (Wu Degang) Date: Thu, 08 Mar 2012 15:29:24 +0800 Subject: [petsc-users] storing a matrix in petsc binary form Message-ID: <4F585FD4.2030302@ust.hk> Hi, I need to convert a matrix (in standard c array form) to the "pestc binary form" so that I can pass it to a slepc program which solves the eigenvalue spectrum for the matrix. I found someone else has raised this question long before (http://lists.mcs.anl.gov/pipermail/petsc-users/2011-July/009326.html), but the code cannot be compiled with petsc 3.2 (perhaps the code is for an older version?). Can anyone give me some guidance? or even some code? Thanks. Regards, Wu Degang From jroman at dsic.upv.es Thu Mar 8 01:42:39 2012 From: jroman at dsic.upv.es (Jose E. Roman) Date: Thu, 8 Mar 2012 08:42:39 +0100 Subject: [petsc-users] storing a matrix in petsc binary form In-Reply-To: <4F585FD4.2030302@ust.hk> References: <4F585FD4.2030302@ust.hk> Message-ID: <7A052E90-35FC-4F55-968A-C6CD08EB7D92@dsic.upv.es> El 08/03/2012, a las 08:29, Wu Degang escribi?: > Hi, > > I need to convert a matrix (in standard c array form) to the "pestc binary form" so that I can pass it to a slepc program which solves the eigenvalue spectrum for the matrix. I found someone else has raised this question long before (http://lists.mcs.anl.gov/pipermail/petsc-users/2011-July/009326.html), but the code cannot be compiled with petsc 3.2 (perhaps the code is for an older version?). Can anyone give me some guidance? or even some code? > > Thanks. > > Regards, > Wu Degang In 3.2 it would be something like this: PetscViewer viewer; Mat A; MatCreateSeqDense(PETSC_COMM_SELF,n,n,values,&A); PetscViewerBinaryOpen(PETSC_COMM_WORLD,"mymatrix.petsc",FILE_MODE_WRITE,&viewer); MatView(A,viewer); PetscViewerDestroy(&viewer); MatDestroy(A); Jose From C.Klaij at marin.nl Thu Mar 8 03:17:08 2012 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Thu, 8 Mar 2012 09:17:08 +0000 Subject: [petsc-users] using PCFieldSplitGetSubKSP in c++ Message-ID: This may be a dumb question (new to c++), well anyway: I'm trying to get the sub KSP associated with the Schur complement S, in order to set the null space. This is the code: KSP subksp[2]; PetscInt n=2; ierr = PCFieldSplitGetSubKSP(pc,&n,&subksp); CHKERRQ(ierr); ierr = KSPSetNullSpace(subksp[1],subnullsp); CHKERRQ(ierr); It gives the following compiler error: error: argument of type "KSP (*)[2]" is incompatible with parameter of type "KSP **" ierr = PCFieldSplitGetSubKSP(pc,&n,&subksp); CHKERRQ(ierr); ^ So I changed the code to: KSP *subksp[2]; PetscInt n=2; ierr = PCFieldSplitGetSubKSP(pc,&n,subksp); CHKERRQ(ierr); ierr = KSPSetNullSpace(subksp[1],subnullsp); CHKERRQ(ierr); Now I get the error: error: argument of type "KSP *" is incompatible with parameter of type "KSP" ierr = KSPSetNullSpace(subksp[1],subnullsp); CHKERRQ(ierr); ^ What is the proper way to set the null space of the Schur complement? I already have the null space of the matrix [A00 A01, A10 A11], the null space of A11 - A10 inv(A00) A01 is clearly related and could perhaps be deduced somehow? dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From knepley at gmail.com Thu Mar 8 05:59:05 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 8 Mar 2012 05:59:05 -0600 Subject: [petsc-users] using PCFieldSplitGetSubKSP in c++ In-Reply-To: References: Message-ID: On Thu, Mar 8, 2012 at 3:17 AM, Klaij, Christiaan wrote: > This may be a dumb question (new to c++), well anyway: I'm trying > to get the sub KSP associated with the Schur complement S, in > order to set the null space. This is the code: > > KSP subksp[2]; > Declare this as KSP *subksp; Matt > PetscInt n=2; > ierr = PCFieldSplitGetSubKSP(pc,&n,&subksp); CHKERRQ(ierr); > ierr = KSPSetNullSpace(subksp[1],subnullsp); CHKERRQ(ierr); > > It gives the following compiler error: > > error: argument of type "KSP (*)[2]" is incompatible with parameter of > type "KSP **" > ierr = PCFieldSplitGetSubKSP(pc,&n,&subksp); CHKERRQ(ierr); > ^ > So I changed the code to: > > KSP *subksp[2]; > PetscInt n=2; > ierr = PCFieldSplitGetSubKSP(pc,&n,subksp); CHKERRQ(ierr); > ierr = KSPSetNullSpace(subksp[1],subnullsp); CHKERRQ(ierr); > > Now I get the error: > > error: argument of type "KSP *" is incompatible with parameter of type > "KSP" > ierr = KSPSetNullSpace(subksp[1],subnullsp); CHKERRQ(ierr); > ^ > > What is the proper way to set the null space of the Schur > complement? I already have the null space of the matrix [A00 A01, > A10 A11], the null space of A11 - A10 inv(A00) A01 is clearly > related and could perhaps be deduced somehow? > > > dr. ir. Christiaan Klaij > CFD Researcher > Research & Development > E mailto:C.Klaij at marin.nl > T +31 317 49 33 44 > > MARIN > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From maxwellr at gmail.com Thu Mar 8 11:46:20 2012 From: maxwellr at gmail.com (Max Rudolph) Date: Thu, 8 Mar 2012 09:46:20 -0800 Subject: [petsc-users] Binary VTK viewer Message-ID: I was looking into the various PetscViewerFormats available and it appears that only ASCII vtk format is supported. I found an old thread on this mailing list with talk between Blaise Bourdin and Matt Knepley of adding support for output of binary vtk files. Was this ever done? Thanks for your help, Max -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Mar 8 11:48:09 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 8 Mar 2012 11:48:09 -0600 Subject: [petsc-users] Binary VTK viewer In-Reply-To: References: Message-ID: On Thu, Mar 8, 2012 at 11:46, Max Rudolph wrote: > I was looking into the various PetscViewerFormats available and it appears > that only ASCII vtk format is supported. I found an old thread on this > mailing list with talk between Blaise Bourdin and Matt Knepley of adding > support for output of binary vtk files. Was this ever done? Binary VTK output for DMDA is in petsc-dev. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bourdin at lsu.edu Thu Mar 8 11:49:02 2012 From: bourdin at lsu.edu (Blaise Bourdin) Date: Thu, 8 Mar 2012 11:49:02 -0600 Subject: [petsc-users] Binary VTK viewer In-Reply-To: References: Message-ID: <34006BF2-DF77-4775-88E5-64DCCD206B97@lsu.edu> No. Imtiaz had to leave LSU before we could accomplish anything meaningful. Blaise On Mar 8, 2012, at 11:46 AM, Max Rudolph wrote: > I was looking into the various PetscViewerFormats available and it appears that only ASCII vtk format is supported. I found an old thread on this mailing list with talk between Blaise Bourdin and Matt Knepley of adding support for output of binary vtk files. Was this ever done? > > Thanks for your help, > Max -- Department of Mathematics and Center for Computation & Technology Louisiana State University, Baton Rouge, LA 70803, USA Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin From hanangul12 at yahoo.co.uk Thu Mar 8 12:57:28 2012 From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh) Date: Thu, 8 Mar 2012 18:57:28 +0000 (GMT) Subject: [petsc-users] Projection preconditioner as PCMG In-Reply-To: References: <1331045458.64662.YahooMailNeo@web28501.mail.ukl.yahoo.com> <1331053255.72750.YahooMailNeo@web28504.mail.ukl.yahoo.com> Message-ID: <1331233048.52170.YahooMailNeo@web28512.mail.ukl.yahoo.com> Dear, Thank you Jed for reply!? I dont want to try snes, since my problem is linear, but I successfully applied the KSP preconditioned with PCMG (not kaskade, but multiplicative; for some reasons) . I make pre-smoother dummy by setting KSP_PRE context as? PREONLY alongwith PCNONE. CGC and post-smoother serves my purpose to apply preconditioner Prec = I - A*P*(A_H)^-1 * R?? with KSP_POST as RICHARDSON alongwith PCNONE. I hope PCNONE make smoother S = I in the framework you suggested earlier i.e. C b + S (b - A C b)~~~~~~ where C is read as CGC. It seems working well with levels 2, but when I force RICHARDSON_max_it 0. To my understanding, it should work with RICHARDSON_max_it = 1;? What if I want to approximate my all coarse matrices with any Krylov iteration ? Appreciations for helping out! Abdul H. TechnischeUniversiteit Delft Faculteit Elektrotechniek, Wiskunde en Informatica Numerical Analysis Room No. HB.07.050 Mekelweg 4, 2628 CD Delft Tel: (015) - 27 81692 Fax: (015) - 27 87209 E-mail: hanangul12 at yahoo.co.uk (Primary) ??? ??? ? a.h.sheikh at tudelft.nl >________________________________ > From: Jed Brown >To: Abdul Hanan Sheikh ; PETSc users list >Sent: Tuesday, 6 March 2012, 21:14 >Subject: Re: [petsc-users] Projection preconditioner as PCMG > > >On Tue, Mar 6, 2012 at 11:00, Abdul Hanan Sheikh wrote: > >I make my query simpler: how can I implement a preconditioner of form >> >> >> >>P = I - A*P*(A_H)^-1 * R? >> >>with any Krylov , where >>I : Identity >>P : Prolongation >>A_H : Galerkin coarse matrix >>R : Restriction, >> >> >> >>Can I exploit the CGC part of PCMG , by making its pre and post smoother null ? > >You use P twice in the above, but I think I see what you mean. You can certainly use PCMG with no pre- or post-smoother or you could do the same here with PCComposite, but you probably don't want that extra application of A in the preconditioner unless you are going to use a post-smoother. > > >That is, suppose C = P*A_H^{-1}*R is the coarse grid solver. Then with a post-smoother S, application of the preconditioner to the vector b would be C b + S (b - A C b). If you drop the post-smoother by setting S = I, you get C b + b - A C b. Keeping the extra multiplication with A and not using a post-smoother will likely destabilize the method. As an example, take src/snes/examples/tutorials/ex5.c and just look at the first linear solve with 2-level method. First, with an additive coarse grid correction and no fine grid smoother (i.e. the preconditioner is I + C where C is an Prolongate*Exact_Coarse_Solve*Restrict). > > >$ ./ex5 -da_grid_x 201 -da_grid_y 201 -snes_max_it 1 -ksp_monitor_singular_value -pc_type mg -pc_mg_levels 2 -pc_mg_type additive -mg_levels_ksp_type preonly -mg_levels_pc_type??none >? ? 0 KSP Residual norm 1.991818008559e+01 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 >? ? 1 KSP Residual norm 8.562399481020e+00 % max 9.703742797548e-01 min 9.703742797548e-01 max/min 1.000000000000e+00 >? ? 2 KSP Residual norm 1.547282124742e+00 % max 4.106911489507e+00 min 9.455565835409e-01 max/min 4.343379931984e+00 >? ? 3 KSP Residual norm 5.193012838962e-01 % max 7.849972917628e+00 min 9.390711734984e-01 max/min 8.359294949267e+00 >? ? 4 KSP Residual norm 1.844773292053e-01 % max 8.102437955088e+00 min 9.157291371635e-01 max/min 8.848072673744e+00 >? ? 5 KSP Residual norm 6.904332859839e-02 % max 8.102847554075e+00 min 8.906430581537e-01 max/min 9.097749631454e+00 >? ? 6 KSP Residual norm 3.747280928850e-02 % max 8.131998087718e+00 min 8.796142392701e-01 max/min 9.244959579630e+00 >? ? 7 KSP Residual norm 1.781146506612e-02 % max 8.153370848515e+00 min 8.294602459028e-01 max/min 9.829730705949e+00 >? ? 8 KSP Residual norm 7.506605928852e-03 % max 8.160488227056e+00 min 8.035220795876e-01 max/min 1.015589793282e+01 >? ? 9 KSP Residual norm 3.771683771569e-03 % max 8.185296438244e+00 min 7.986463109034e-01 max/min 1.024896293452e+01 >? ?10 KSP Residual norm 1.900344739083e-03 % max 8.246526708637e+00 min 7.835579388152e-01 max/min 1.052446322107e+01 >? ?11 KSP Residual norm 8.969235041181e-04 % max 8.246542749144e+00 min 7.698718962697e-01 max/min 1.071157784704e+01 >? ?12 KSP Residual norm 3.771331016150e-04 % max 8.253249205705e+00 min 7.665454362492e-01 max/min 1.076681017904e+01 >? ?13 KSP Residual norm 1.998592159889e-04 % max 8.323950632412e+00 min 7.632295016946e-01 max/min 1.090622232753e+01 >? ?14 KSP Residual norm 9.624660396860e-05 % max 8.327354837686e+00 min 7.567183120253e-01 max/min 1.100456366042e+01? > > >This is identical with one step of Richardson (-mg_levels_ksp_type richardson) because the "initial guess" is 0. The convergence is somewhat better if we improve the additive fine-grid smoother, e.g. ILU or SOR, though the condition number is actually worse in this case (because the preconditioned operator is now non-normal, the eigenvalues are well-clustered). > > >$ ./ex5 -da_grid_x 201 -da_grid_y 201 -snes_max_it 1 -ksp_monitor_singular_value -pc_type mg -pc_mg_levels 2 -pc_mg_type additive -mg_levels_ksp_type preonly -mg_levels_pc_type ilu >? ? 0 KSP Residual norm 1.997267177320e+01 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 >? ? 1 KSP Residual norm 5.755612216067e+00 % max 9.658916552787e-01 min 9.658916552787e-01 max/min 1.000000000000e+00 >? ? 2 KSP Residual norm 2.828695990580e-01 % max 2.491505412300e+00 min 9.583727874764e-01 max/min 2.599724705102e+00 >? ? 3 KSP Residual norm 1.097671648754e-01 % max 7.900124346250e+00 min 5.695048906547e-01 max/min 1.387191659964e+01 >? ? 4 KSP Residual norm 2.203949856452e-02 % max 9.291338218054e+00 min 3.581794670449e-01 max/min 2.594045464055e+01 >? ? 5 KSP Residual norm 8.497097452426e-03 % max 1.013788428797e+01 min 3.272689944775e-01 max/min 3.097722197653e+01 >? ? 6 KSP Residual norm 1.376443389574e-03 % max 1.055586782597e+01 min 2.933041209977e-01 max/min 3.598949714743e+01 >? ? 7 KSP Residual norm 2.753520742081e-04 % max 1.061464021647e+01 min 2.893817762727e-01 max/min 3.668040314489e+01 >? ? 8 KSP Residual norm 6.552848547677e-05 % max 1.062831125267e+01 min 2.883290857853e-01 max/min 3.686173812025e+01 > > >"Kaskade" multigrid applies no pre-smoother, but uses a multiplicative post-smoother. That is what I wrote above and probably what you wanted. > > >$ ./ex5 -da_grid_x 201 -da_grid_y 201 -snes_max_it 1 -ksp_monitor_singular_value -pc_type mg -pc_mg_levels 2 -pc_mg_type kaskade -mg_levels_ksp_type richardson -mg_levels_pc_type none >? ? 0 KSP Residual norm 1.981089588505e+01 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 >? ? 1 KSP Residual norm 3.481667117996e+00 % max 9.904507054699e-01 min 9.904507054699e-01 max/min 1.000000000000e+00 >? ? 2 KSP Residual norm 1.652167261481e+00 % max 5.608992736195e+00 min 9.879184804293e-01 max/min 5.677586609937e+00 >? ? 3 KSP Residual norm 1.627219442360e+00 % max 6.798397388800e+00 min 9.550178771471e-01 max/min 7.118607464301e+00 >? ? 4 KSP Residual norm 7.880741343988e-01 % max 7.451381264321e+00 min 7.145924145893e-01 max/min 1.042745642438e+01 >? ? 5 KSP Residual norm 3.537498074218e-01 % max 7.663873901737e+00 min 5.751738420032e-01 max/min 1.332444791829e+01 >? ? 6 KSP Residual norm 1.078704665358e-01 % max 7.800309555756e+00 min 5.120187805734e-01 max/min 1.523442079023e+01 >? ? 7 KSP Residual norm 4.407455476078e-02 % max 7.855365072615e+00 min 5.037708740493e-01 max/min 1.559313068156e+01 >? ? 8 KSP Residual norm 2.107104852559e-02 % max 7.893144033146e+00 min 5.025484383148e-01 max/min 1.570623532254e+01 >? ? 9 KSP Residual norm 1.763159978441e-02 % max 7.912218583050e+00 min 5.024504696471e-01 max/min 1.574726079688e+01 >? ?10 KSP Residual norm 1.745405855993e-02 % max 7.928099602729e+00 min 5.023138582042e-01 max/min 1.578315922056e+01 >? ?11 KSP Residual norm 1.376977330994e-02 % max 7.939644155950e+00 min 5.017479535724e-01 max/min 1.582396918497e+01 >? ?12 KSP Residual norm 9.446347797152e-03 % max 7.949376050758e+00 min 5.015718142316e-01 max/min 1.584892895734e+01 >? ?13 KSP Residual norm 6.843342509359e-03 % max 7.956996132919e+00 min 4.969226601251e-01 max/min 1.601254434828e+01 >? ?14 KSP Residual norm 6.437072325338e-03 % max 7.961936315836e+00 min 4.932296836382e-01 max/min 1.614245164059e+01 >? ?15 KSP Residual norm 6.091589527973e-03 % max 7.966811508176e+00 min 4.932019218895e-01 max/min 1.615324505966e+01 >? ?16 KSP Residual norm 4.652185214024e-03 % max 7.970648347219e+00 min 4.916668205282e-01 max/min 1.621148309064e+01 >? ?17 KSP Residual norm 3.701849685772e-03 % max 7.973893800218e+00 min 4.915344876340e-01 max/min 1.622245030781e+01 >? ?18 KSP Residual norm 3.482133374032e-03 % max 7.976475806296e+00 min 4.911811815837e-01 max/min 1.623937582580e+01 >? ?19 KSP Residual norm 3.465608263872e-03 % max 7.978688479804e+00 min 4.898709616560e-01 max/min 1.628732687652e+01 >? ?20 KSP Residual norm 3.149472026786e-03 % max 7.980652326863e+00 min 4.891722853032e-01 max/min 1.631460441778e+01 >? ?21 KSP Residual norm 2.763464672775e-03 % max 7.982383839733e+00 min 4.891419493795e-01 max/min 1.631915612607e+01 >? ?22 KSP Residual norm 2.689646479190e-03 % max 7.983807057995e+00 min 4.876132849055e-01 max/min 1.637323531811e+01 >? ?23 KSP Residual norm 2.661061400431e-03 % max 7.985119113668e+00 min 4.836465184065e-01 max/min 1.651023797292e+01 >? ?24 KSP Residual norm 2.383953808700e-03 % max 7.986314359135e+00 min 4.718949191329e-01 max/min 1.692392529636e+01 >^C > > >Okay, that didn't work. But it was because the Richardson step was too large. We can get away without a traditional smoothing step by fixing the relaxation parameter for Richardson: > > >$ ./ex5 -da_grid_x 201 -da_grid_y 201 -snes_max_it 1 -ksp_monitor_singular_value -pc_type mg -pc_mg_levels 2 -pc_mg_type kaskade -mg_levels_ksp_type richardson -mg_levels_pc_type none -mg_levels_ksp_richardson_self_scale >? ? 0 KSP Residual norm 1.975857018746e+01 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 >? ? 1 KSP Residual norm 2.476038171898e+00 % max 9.622055302029e-01 min 9.622055302029e-01 max/min 1.000000000000e+00 >? ? 2 KSP Residual norm 3.723935286980e-02 % max 1.019358214307e+00 min 8.790661889416e-01 max/min 1.159592107091e+00 >? ? 3 KSP Residual norm 1.048082002961e-02 % max 2.231045830559e+00 min 2.203725693734e-01 max/min 1.012397249305e+01 >? ? 4 KSP Residual norm 4.815690407780e-03 % max 3.167321755561e+00 min 1.859713575290e-01 max/min 1.703123425911e+01 >? ? 5 KSP Residual norm 2.282674490066e-03 % max 5.220845319337e+00 min 1.603210979291e-01 max/min 3.256492992360e+01 >? ? 6 KSP Residual norm 5.030060118813e-04 % max 6.078990414170e+00 min 1.535348150609e-01 max/min 3.959356326940e+01 >? ? 7 KSP Residual norm 2.084516564815e-04 % max 6.195047474999e+00 min 1.466558297789e-01 max/min 4.224208123426e+01 >? ? 8 KSP Residual norm 6.654981023267e-05 % max 6.379643313848e+00 min 1.442472810392e-01 max/min 4.422713043801e+01 > > >Unfortunately, convergence is very sensitive to that step and for variable coefficient problems, we won't be able to choose a good value. If we know the matrix diagonal, we can get a lot more robustness by post-smoothing with Jacobi, and now we don't need that pesky scaling parameter. > > >$ ./ex5 -da_grid_x 201 -da_grid_y 201 -snes_max_it 1 -ksp_monitor_singular_value -pc_type mg -pc_mg_levels 2 -pc_mg_type kaskade -mg_levels_ksp_type richardson -mg_levels_pc_type jacobi >? ? 0 KSP Residual norm 1.975846409514e+01 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 >? ? 1 KSP Residual norm 2.482202615145e+00 % max 9.621966917526e-01 min 9.621966917526e-01 max/min 1.000000000000e+00 >? ? 2 KSP Residual norm 4.457401425338e-02 % max 1.020264072916e+00 min 8.821750286291e-01 max/min 1.156532479163e+00 >? ? 3 KSP Residual norm 1.637472997384e-02 % max 1.776622548172e+00 min 7.524631434334e-01 max/min 2.361075839629e+00 >? ? 4 KSP Residual norm 3.644899164209e-03 % max 1.907783379671e+00 min 4.245698995769e-01 max/min 4.493449445127e+00 >? ? 5 KSP Residual norm 1.443735788456e-03 % max 1.935858887793e+00 min 3.898702611314e-01 max/min 4.965392544111e+00 >? ? 6 KSP Residual norm 3.929697132002e-04 % max 1.961850264833e+00 min 3.669296462786e-01 max/min 5.346666001861e+00 >? ? 7 KSP Residual norm 1.446207699005e-04 % max 1.970859698383e+00 min 3.614358036844e-01 max/min 5.452862384670e+00 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Mar 8 13:16:35 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 8 Mar 2012 13:16:35 -0600 Subject: [petsc-users] Projection preconditioner as PCMG In-Reply-To: <1331233048.52170.YahooMailNeo@web28512.mail.ukl.yahoo.com> References: <1331045458.64662.YahooMailNeo@web28501.mail.ukl.yahoo.com> <1331053255.72750.YahooMailNeo@web28504.mail.ukl.yahoo.com> <1331233048.52170.YahooMailNeo@web28512.mail.ukl.yahoo.com> Message-ID: On Thu, Mar 8, 2012 at 12:57, Abdul Hanan Sheikh wrote: > I dont want to try snes, since my problem is linear, > Note that you can still use SNES for linear problems. (I prefer the interface, you can choose -snes_type ksponly so there is no extra overhead, but use whatever you like.) > but I successfully applied the KSP > preconditioned with PCMG (not kaskade, but multiplicative; for some > reasons) . > I make pre-smoother dummy by setting KSP_PRE context as PREONLY alongwith > PCNONE. > You can just set the number of pre-smoothing steps to 0. > CGC and post-smoother serves my purpose to apply preconditioner > Prec = I - A*P*(A_H)^-1 * R with KSP_POST as RICHARDSON alongwith > PCNONE. > Read my last message again. You either want the additive Prec = I + C (where C = P*A_H^{-1}*R is the coarse solve) or the multiplicative Prec = C + S (I - A C) perhaps with S = I (or some scaling factor). I don't think your definition of Prec makes sense or is the operation that you intend to be applying. > I hope PCNONE make smoother S = I in the framework you suggested earlier > i.e. > C b + S (b - A C b)~~~~~~ where C is read as CGC. > It seems working well with levels 2, but when I force RICHARDSON_max_it 0. > To my understanding, it should work with RICHARDSON_max_it = 1; > Read through my last message again. I explained that this does not work and that it is caused by the destabilizing effect of the extra multiply without a smoother. You can fix it for easy problems with -mg_levels_ksp_richardson_self_scale. > What if I want to approximate my all coarse matrices with any Krylov > iteration ? > The methods on each level are independent, you can set them with -mg_coarse_ksp_type gmres -mg_levels_1_ksp_type cg -mg_levels_1_ksp_max_it 100 -mg_levels_2_ksp_type minres ... -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Mar 8 13:21:32 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 8 Mar 2012 13:21:32 -0600 Subject: [petsc-users] Binary VTK viewer In-Reply-To: References: <34006BF2-DF77-4775-88E5-64DCCD206B97@lsu.edu> Message-ID: On Thu, Mar 8, 2012 at 13:10, Max Rudolph wrote: > I get an implicit declaration warning when compiling with a call to > DMDAVTKWriteAll: > > io.c:209: warning: implicit declaration of function 'DMDAVTKWriteAll' > > Should the prototype be in petscdmda.h ? I do not see it there. Code still > compiles successfully. > That is a developer-level function that you should not be calling, it's declared in private/daimpl.h. I didn't hack together some funky new API when I added the VTK viewer. You use it like any other viewer, by calling VecView(). Here's some sample code. ierr = PetscOptionsGetString(((PetscObject)snes)->prefix,"-snes_view_solution_vtk",filename,PETSC_MAX_PATH_LEN,&flg);CHKERRQ(ierr); if (flg) { PetscViewer viewer; ierr = PetscViewerCreate(((PetscObject)snes)->comm,&viewer);CHKERRQ(ierr); ierr = PetscViewerSetType(viewer,PETSCVIEWERVTK);CHKERRQ(ierr); ierr = PetscViewerFileSetName(viewer,filename);CHKERRQ(ierr); ierr = VecView(snes->vec_sol,viewer);CHKERRQ(ierr); ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr); } -------------- next part -------------- An HTML attachment was scrubbed... URL: From hanangul12 at yahoo.co.uk Thu Mar 8 13:59:56 2012 From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh) Date: Thu, 8 Mar 2012 19:59:56 +0000 (GMT) Subject: [petsc-users] Projection preconditioner as PCMG In-Reply-To: References: <1331045458.64662.YahooMailNeo@web28501.mail.ukl.yahoo.com> <1331053255.72750.YahooMailNeo@web28504.mail.ukl.yahoo.com> <1331233048.52170.YahooMailNeo@web28512.mail.ukl.yahoo.com> Message-ID: <1331236796.16988.YahooMailNeo@web28510.mail.ukl.yahoo.com> Dear, First two answers are appreciated; ? And last one is more appreciated; -mg_levels_1_ksp_type cg can this be written in routine ? yes, with KSPSetType() but which ksp-context ? ksp_pre ? My Preconditioner: I? need a preconditioner which read as: Prec = \lambda_max*C + (I - AC) , [C = CGC] where (I - AC) deflates spectrum to zero, and this part \lambda_max *C then shifts spectrum to \lambda_max (which is always 1 in my case) . Dont I get my Prec by multiplicative combination of CGC with post smoother S = Identity? using mg_type_multiplicative? Thanks in anticipation, regards, Abdul multiplicative_ >________________________________ >I dont want to try snes, since my problem is linear, > > >Note that you can still use SNES for linear problems. (I prefer the interface, you can choose -snes_type ksponly so there is no extra overhead, but use whatever you like.) >Thanks! > >but I successfully applied the KSP >>preconditioned with PCMG (not kaskade, but multiplicative; for some reasons) . >>I make pre-smoother dummy by setting KSP_PRE context as? PREONLY alongwith PCNONE. >> > > >You can just set the number of pre-smoothing steps to 0. >?Done! :-) > >CGC and post-smoother serves my purpose to apply preconditioner >>Prec = I - A*P*(A_H)^-1 * R?? with KSP_POST as RICHARDSON alongwith PCNONE. >> > > >Read my last message again. You either want the additive > > >Prec = I + C > > >(where C = P*A_H^{-1}*R is the coarse solve)?or the multiplicative > > >Prec = C + S (I - A C) > > >perhaps with S = I (or some scaling factor). I don't think your definition of Prec makes sense or is the operation that you intend to be applying. > > >I hope PCNONE make smoother S = I in the framework you suggested earlier i.e. >> >>C b + S (b - A C b)~~~~~~ where C is read as CGC. >> >>It seems working well with levels 2, but when I force RICHARDSON_max_it 0. To my understanding, it should work with RICHARDSON_max_it = 1;? > > >Read through my last message again. I explained that this does not work and that it is caused by the destabilizing effect of the extra multiply without a smoother. You can fix it for easy problems with -mg_levels_ksp_richardson_self_scale. >? >What if I want to approximate my all coarse matrices with any Krylov iteration ? > > >The methods on each level are independent, you can set them with -mg_coarse_ksp_type gmres -mg_levels_1_ksp_type cg -mg_levels_1_ksp_max_it 100 -mg_levels_2_ksp_type minres ...? > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Mar 8 14:09:10 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 8 Mar 2012 14:09:10 -0600 Subject: [petsc-users] Projection preconditioner as PCMG In-Reply-To: <1331236796.16988.YahooMailNeo@web28510.mail.ukl.yahoo.com> References: <1331045458.64662.YahooMailNeo@web28501.mail.ukl.yahoo.com> <1331053255.72750.YahooMailNeo@web28504.mail.ukl.yahoo.com> <1331233048.52170.YahooMailNeo@web28512.mail.ukl.yahoo.com> <1331236796.16988.YahooMailNeo@web28510.mail.ukl.yahoo.com> Message-ID: On Thu, Mar 8, 2012 at 13:59, Abdul Hanan Sheikh wrote: > And last one is more appreciated; > -mg_levels_1_ksp_type cg can this be written in routine ? yes, with > KSPSetType() but which ksp-context ? ksp_pre ? > That is the command line option, see PCMGGetSmoother() or PCMGGetSmootherUp() to get the KSP for a level from code. > > My Preconditioner: > I need a preconditioner which read as: Prec = \lambda_max*C + (I - AC) , > [C = CGC] where (I - AC) deflates spectrum to zero, > and this part \lambda_max *C then shifts spectrum to \lambda_max (which > is always 1 in my case) . > You can always multiply preconditioners by a arbitrary constant, so multiply through by S = 1/\lambda_max to get the standard form Prec = C + S(I - AC) > Dont I get my Prec by multiplicative combination of CGC with post smoother > S = Identity using mg_type_multiplicative ? If we put in S = I, we get my Prec = C + I - AC which is not equal to your stated Prec = I - AC unless C = 0 which would make Prec = I. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hanangul12 at yahoo.co.uk Thu Mar 8 14:16:37 2012 From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh) Date: Thu, 8 Mar 2012 20:16:37 +0000 (GMT) Subject: [petsc-users] Projection preconditioner as PCMG In-Reply-To: References: <1331045458.64662.YahooMailNeo@web28501.mail.ukl.yahoo.com> <1331053255.72750.YahooMailNeo@web28504.mail.ukl.yahoo.com> <1331233048.52170.YahooMailNeo@web28512.mail.ukl.yahoo.com> <1331236796.16988.YahooMailNeo@web28510.mail.ukl.yahoo.com> Message-ID: <1331237797.65871.YahooMailNeo@web28511.mail.ukl.yahoo.com> Dear, I almost got you; and thus I got my preconditoner? Prec = \lambda_max*C + (I - AC) . Only question is : how to make S = I. S is my post_smoother in PCMG framwork and post_smoother is a KSP_POST context with PC_POST ? Does following choices make S =I; KSP_POST => RICHARDSON PC_POST => PCNONE THANKS ALOT. Abdul, ? >________________________________ > From: Jed Brown >To: Abdul Hanan Sheikh >Cc: "petsc-users at mcs.anl.gov" >Sent: Thursday, 8 March 2012, 21:09 >Subject: Re: [petsc-users] Projection preconditioner as PCMG > > >On Thu, Mar 8, 2012 at 13:59, Abdul Hanan Sheikh wrote: > >And last one is more appreciated; >>-mg_levels_1_ksp_type cg can this be written in routine ? yes, with KSPSetType() but which ksp-context ? ksp_pre ? >> > > >That is the command line option, see PCMGGetSmoother() or PCMGGetSmootherUp() to get the KSP for a level from code. >? > >>My Preconditioner: >>I? need a preconditioner which read as: Prec = \lambda_max*C + (I - AC) , [C = CGC] where (I - AC) deflates spectrum to zero, >>and this part \lambda_max *C then shifts spectrum to \lambda_max (which is always 1 in my case) . >> > > >You can always multiply preconditioners by a arbitrary constant, so multiply through by S = 1/\lambda_max to get the standard form > > >Prec = C + S(I - AC) >? >Dont I get my Prec by multiplicative combination of CGC with post smoother S = Identity? using mg_type_multiplicative? > >If we put in S = I, we get my > > >Prec = C + I - AC > > >which is not equal to your stated > > >Prec = I - AC > > >unless C = 0 which would make Prec = I. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Mar 8 14:19:13 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 8 Mar 2012 14:19:13 -0600 Subject: [petsc-users] Projection preconditioner as PCMG In-Reply-To: <1331237797.65871.YahooMailNeo@web28511.mail.ukl.yahoo.com> References: <1331045458.64662.YahooMailNeo@web28501.mail.ukl.yahoo.com> <1331053255.72750.YahooMailNeo@web28504.mail.ukl.yahoo.com> <1331233048.52170.YahooMailNeo@web28512.mail.ukl.yahoo.com> <1331236796.16988.YahooMailNeo@web28510.mail.ukl.yahoo.com> <1331237797.65871.YahooMailNeo@web28511.mail.ukl.yahoo.com> Message-ID: On Thu, Mar 8, 2012 at 14:16, Abdul Hanan Sheikh wrote: > I almost got you; and thus I got my preconditoner Prec = \lambda_max*C + > (I - AC) . > Only question is : how to make S = I. S is my post_smoother in PCMG > framwork and post_smoother > is a KSP_POST context with PC_POST ? > Does following choices make S =I; > KSP_POST => RICHARDSON > PC_POST => PCNONE Yes, you are asking for one post-smoothing Richardson iteration with no preconditioner. That is what I demonstrated in the command line options of my earlier message. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Mar 8 16:25:26 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 8 Mar 2012 16:25:26 -0600 Subject: [petsc-users] Binary VTK viewer In-Reply-To: References: <34006BF2-DF77-4775-88E5-64DCCD206B97@lsu.edu> Message-ID: On Thu, Mar 8, 2012 at 16:19, Max Rudolph wrote: > Is it expected behavior for field names not to be included in the .vts > file produced using this viewer? Did you use DMDASetFieldNames()? $ mpiexec -n 2 ./ex50 -da_refine 4 -snes_monitor -snes_view_solution_vtk foo.vts now foo.vts contains > > Max > > On Thu, Mar 8, 2012 at 11:21 AM, Jed Brown wrote: > >> On Thu, Mar 8, 2012 at 13:10, Max Rudolph wrote: >> >>> I get an implicit declaration warning when compiling with a call to >>> DMDAVTKWriteAll: >>> >>> io.c:209: warning: implicit declaration of function 'DMDAVTKWriteAll' >>> >>> Should the prototype be in petscdmda.h ? I do not see it there. Code >>> still compiles successfully. >>> >> >> That is a developer-level function that you should not be calling, it's >> declared in private/daimpl.h. I didn't hack together some funky new API >> when I added the VTK viewer. You use it like any other viewer, by calling >> VecView(). Here's some sample code. >> >> ierr = >> PetscOptionsGetString(((PetscObject)snes)->prefix,"-snes_view_solution_vtk",filename,PETSC_MAX_PATH_LEN,&flg);CHKERRQ(ierr); >> if (flg) { >> PetscViewer viewer; >> ierr = >> PetscViewerCreate(((PetscObject)snes)->comm,&viewer);CHKERRQ(ierr); >> ierr = PetscViewerSetType(viewer,PETSCVIEWERVTK);CHKERRQ(ierr); >> ierr = PetscViewerFileSetName(viewer,filename);CHKERRQ(ierr); >> ierr = VecView(snes->vec_sol,viewer);CHKERRQ(ierr); >> ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr); >> } >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Mar 8 17:20:07 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 8 Mar 2012 17:20:07 -0600 Subject: [petsc-users] Binary VTK viewer In-Reply-To: References: <34006BF2-DF77-4775-88E5-64DCCD206B97@lsu.edu> Message-ID: On Thu, Mar 8, 2012 at 16:49, Max Rudolph wrote: > Is there a way to do this on a vector, independently of the DA? I was > hoping to save some calculated fields like strain rate and viscous > dissipation, and the way that I have done this previously is to create a DA > with 1 dof per node, then create a global vector for each quantity of > interest, give it a name using, then populate these arrays and write it to > disk. Is this a bad design decision? If you ever communicate those values, I suggest making it one Vec. It should cost a similar amount to communicate all the values as to communicate just one. Similarly, if you usually use several of the values at once, I would put them all in the same Vec because the memory access is better on most hardware. Can you pull petsc-dev, what you want should work now. > One kludgey workaround might be to set the DA field name for component 0 > using the name returned by PetscObjectGetName prior to writing each > vector. The alternative that I see, which would allow me to use the method > you suggested, is to create a DA with as many DOFs per node as there are > quantities of interest, then create one global vector with multiple DOFs > per node, set the field names, and write it to disk. This would involve > some significant re-coding but might ultimately be a better solution. > > Max > > > On Thu, Mar 8, 2012 at 2:38 PM, Jed Brown wrote: > >> On Thu, Mar 8, 2012 at 16:36, Max Rudolph wrote: >> >>> No. I created vectors with DMDACreateGlobalVector and then set vector >>> names using PetscObjectSetName. >>> >> >> Set the field name, not the PetscObject name. VTK does not provide a >> useful mechanism to write multiple "steps" into the same file, so you would >> write separate files per step. >> >> >>> I will look into how hard it would be to change this. I am viewing the >>> vectors using VecView currently. Thanks again for your help. >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hanangul12 at yahoo.co.uk Fri Mar 9 05:16:01 2012 From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh) Date: Fri, 9 Mar 2012 11:16:01 +0000 (GMT) Subject: [petsc-users] Projection preconditioner as PCMG In-Reply-To: References: <1331045458.64662.YahooMailNeo@web28501.mail.ukl.yahoo.com> <1331053255.72750.YahooMailNeo@web28504.mail.ukl.yahoo.com> <1331233048.52170.YahooMailNeo@web28512.mail.ukl.yahoo.com> <1331236796.16988.YahooMailNeo@web28510.mail.ukl.yahoo.com> <1331237797.65871.YahooMailNeo@web28511.mail.ukl.yahoo.com> Message-ID: <1331291761.89699.YahooMailNeo@web28504.mail.ukl.yahoo.com> Dear,? To my earlier querry: What if I want to approximate my all coarse matrices with any Krylov iteration ? reply: The methods on each level are independent, you can set them with -mg_coarse_ksp_type gmres -mg_levels_1_ksp_type cg -mg_levels_1_ksp_max_it 100 -mg_levels_2_ksp_type minres ... ? This command line option changes the ksp_type on both ksp_pre and ksp_post (pre and post smoothing) . Where as I need to fix the ksp_post_type as RICHARDSON to get my Prec = C + I - AC ; If adapt pre_smoother as follows : ierr = MGGetSmootherDown(pcmg,lev,&ksp_pre); CHKERRQ(ierr); ierr = KSPSetType(ksp_pre, KSPGMRES);CHKERRQ(ierr); Does ONLY setting pre_smoother as GMRES means that GMRES iterates on coarse matrix at level = lev ? Thanks in advance. Abdul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Mar 9 07:08:33 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 9 Mar 2012 07:08:33 -0600 Subject: [petsc-users] Projection preconditioner as PCMG In-Reply-To: <1331291761.89699.YahooMailNeo@web28504.mail.ukl.yahoo.com> References: <1331045458.64662.YahooMailNeo@web28501.mail.ukl.yahoo.com> <1331053255.72750.YahooMailNeo@web28504.mail.ukl.yahoo.com> <1331233048.52170.YahooMailNeo@web28512.mail.ukl.yahoo.com> <1331236796.16988.YahooMailNeo@web28510.mail.ukl.yahoo.com> <1331237797.65871.YahooMailNeo@web28511.mail.ukl.yahoo.com> <1331291761.89699.YahooMailNeo@web28504.mail.ukl.yahoo.com> Message-ID: On Fri, Mar 9, 2012 at 05:16, Abdul Hanan Sheikh wrote: > Dear, > > To my earlier querry: > > What if I want to approximate my all coarse matrices with any Krylov > iteration ? > > reply: > The methods on each level are independent, you can set them with > -mg_coarse_ksp_type gmres -mg_levels_1_ksp_type cg -mg_levels_1_ksp_max_it > 100 -mg_levels_2_ksp_type minres ... > > This command line option changes the ksp_type on both ksp_pre and ksp_post > (pre and post smoothing) . > Where as I need to fix the ksp_post_type as RICHARDSON to get my Prec = C > + I - AC ; > Do you want this on every level or still just the finest? > > If adapt pre_smoother as follows : > > ierr = MGGetSmootherDown(pcmg,lev,&ksp_pre); CHKERRQ(ierr); > ierr = KSPSetType(ksp_pre, KSPGMRES);CHKERRQ(ierr); > > Does ONLY setting pre_smoother as GMRES means that GMRES iterates on > coarse matrix at level = lev ? > Yes, this will only affect the pre-smoother (so it will then go to the next coarser grid, come back, and do a post-smoother). You may have to increase the number of iterations. I strongly recommend configuring the solver using the command line and checking that it is the method you intended by reading the output of -ksp_view. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Mar 9 09:24:58 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 9 Mar 2012 09:24:58 -0600 Subject: [petsc-users] Projection preconditioner as PCMG In-Reply-To: <1331304806.94034.YahooMailNeo@web28515.mail.ukl.yahoo.com> References: <1331045458.64662.YahooMailNeo@web28501.mail.ukl.yahoo.com> <1331053255.72750.YahooMailNeo@web28504.mail.ukl.yahoo.com> <1331233048.52170.YahooMailNeo@web28512.mail.ukl.yahoo.com> <1331236796.16988.YahooMailNeo@web28510.mail.ukl.yahoo.com> <1331237797.65871.YahooMailNeo@web28511.mail.ukl.yahoo.com> <1331291761.89699.YahooMailNeo@web28504.mail.ukl.yahoo.com> <1331304806.94034.YahooMailNeo@web28515.mail.ukl.yahoo.com> Message-ID: On Fri, Mar 9, 2012 at 08:53, Abdul Hanan Sheikh wrote: > > > > ------------------------------ > *From:* Jed Brown > *To:* Abdul Hanan Sheikh > *Cc:* "petsc-users at mcs.anl.gov" > *Sent:* Friday, 9 March 2012, 14:08 > > *Subject:* Re: [petsc-users] Projection preconditioner as PCMG > > On Fri, Mar 9, 2012 at 05:16, Abdul Hanan Sheikh wrote: > > Dear, > > To my earlier querry: > > What if I want to approximate my all coarse matrices with any Krylov > iteration ? > > reply: > The methods on each level are independent, you can set them with > -mg_coarse_ksp_type gmres -mg_levels_1_ksp_type cg -mg_levels_1_ksp_max_it > 100 -mg_levels_2_ksp_type minres ... > > This command line option changes the ksp_type on both ksp_pre and ksp_post > (pre and post smoothing) . > Where as I need to fix the ksp_post_type as RICHARDSON to get my Prec = C > + I - AC ; > > > Do you want this on every level or still just the finest? > I need this on every level. > > You can set each with code or you can do it all with the following options. Some notes: 1. I use Chebychev instead of Richardson because I don't like tuning Richardson. The option -ksp_richardson_self_scale make the preconditioner nonlinear, in which case we'd have to use FGMRES or GCR on the outside. Chebychev(1) is equivalent to Richardson with some damping factor. I found the Chebychev bounds (3,9) by running with -mg_levels_ksp_chebychev_estimate_eigenvalues 0,0.3,0,1.1 which targets the upper end of the spectrum (from 0.3*maxeig to 1.1*maxeig where maxeig is an estimate of the largest eigenvalue on each level). I ran with -snes_view and noticed that those bounds happened to be about the same on all levels, so I just set them directly. You can translate that to a Richardson relaxation if you like, but it tends to be more fragile. 2. The unpreconditioned norm gives much more accurate condition number estimates here. 3. You can also do Kaskade multigrid (which is exactly what you are asking for) with -pc_mg_smoothdown 0 -pc_mg_smoothup 1. 4. If you have variable coefficients, you will need to use the diagonal in the Richardson smoother for scaling (-mg_levels_pc_type jacobi). Remember that this will force you to recompute your Chebychev/Richardson bounds (or keep doing it automatically with -mg_levels_ksp_chebychev_estimate_eigenvalues 0,0.3,0,1.1). $ ./ex5 -da_grid_x 257 -da_grid_y 257 -snes_max_it 1 -ksp_type gmres -ksp_norm_type unpreconditioned -ksp_monitor_singular_value -ksp_compute_eigenvalues -pc_type mg -pc_mg_levels 9 -pc_mg_type kaskade -mg_levels_ksp_type chebychev -mg_levels_ksp_chebychev_eigenvalues 3,9 -mg_levels_pc_type none 0 KSP Residual norm 1.062542528908e+00 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 1 KSP Residual norm 2.376610964454e-01 % max 1.001957907202e+00 min 1.001957907202e+00 max/min 1.000000000000e+00 2 KSP Residual norm 4.282525183085e-02 % max 1.139043558191e+00 min 8.730762970080e-01 max/min 1.304632323767e+00 3 KSP Residual norm 4.732778291888e-03 % max 1.148710370654e+00 min 7.632757295550e-01 max/min 1.504974318158e+00 4 KSP Residual norm 7.292121371221e-04 % max 1.148751387861e+00 min 7.091101704841e-01 max/min 1.619990003919e+00 5 KSP Residual norm 8.739575412872e-05 % max 1.152691180981e+00 min 6.760500858171e-01 max/min 1.705038140166e+00 6 KSP Residual norm 1.511456466330e-05 % max 1.154316560884e+00 min 6.659327122249e-01 max/min 1.733383177750e+00 7 KSP Residual norm 2.020623105869e-06 % max 1.154629497508e+00 min 6.591981750318e-01 max/min 1.751566586865e+00 Iteratively computed eigenvalues 0.659531 + 0i 0.742062 + 0i 0.815314 + 0i 0.96316 + 0.0106016i 0.96316 - 0.0106016i 1.08027 + 0i 1.11294 + 0i $ ./ex5 -da_grid_x 257 -da_grid_y 257 -snes_max_it 1 -ksp_type gmres -ksp_norm_type unpreconditioned -ksp_monitor_singular_value -ksp_compute_eigenvalues -pc_type mg -pc_mg_levels 9 -pc_mg_type multiplicative -mg_levels_ksp_type chebychev -mg_levels_ksp_chebychev_eigenvalues 3,9 -mg_levels_pc_type none 0 KSP Residual norm 1.062542528908e+00 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 1 KSP Residual norm 3.178344085522e-02 % max 9.704936480137e-01 min 9.704936480137e-01 max/min 1.000000000000e+00 2 KSP Residual norm 1.968390067011e-03 % max 9.897660772730e-01 min 9.220434626490e-01 max/min 1.073448397356e+00 3 KSP Residual norm 1.145527676866e-04 % max 9.992147688651e-01 min 8.443712510420e-01 max/min 1.183383218735e+00 4 KSP Residual norm 5.642680559393e-06 % max 1.001387939323e+00 min 8.242353430425e-01 max/min 1.214929628747e+00 Iteratively computed eigenvalues 0.824026 + 0i 0.918072 + 0i 0.968702 + 0i 1.00085 + 0i > > > > > If adapt pre_smoother as follows : > > ierr = MGGetSmootherDown(pcmg,lev,&ksp_pre); CHKERRQ(ierr); > ierr = KSPSetType(ksp_pre, KSPGMRES);CHKERRQ(ierr); > > Does ONLY setting pre_smoother as GMRES means that GMRES iterates on > coarse matrix at level = lev ? > > > Yes, this will only affect the pre-smoother (so it will then go to the > next coarser grid, come back, and do a post-smoother). You may have to > increase the number of iterations. I strongly recommend configuring the > solver using the command line and checking that it is the method you > intended by reading the output of -ksp_view. > Yes I always see what Multilevel solver do by -ksp_view.. Thanks a lot. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xavier.garnaud at ladhyx.polytechnique.fr Sat Mar 10 09:59:51 2012 From: xavier.garnaud at ladhyx.polytechnique.fr (Xavier Garnaud) Date: Sat, 10 Mar 2012 16:59:51 +0100 Subject: [petsc-users] performance issue Message-ID: Dear PETSc team, I am solving the compressible Navier--Stokes equations in compressible form, so in order to apply the operator, I 1. apply BCs on the flow field 2. compute the flux 3. take the derivative using finite differences 4. apply BCs on the derivatives of the flux In order to apply the linearized operator, I wish to linearize steps 2 and 4 (the other are linear). For this I assemble sparse matrices (MPIAIJ). The matrices should be block diagonal -- with square or rectangular blocks -- so I preallocate the whole diagonal blocks (but I only use MatSetValues for nonzero entries). When I do this, the linearized code runs approximately 50% slower (the computation of derivatives takes more that 70% of the time in the non-linear code), so steps 2 and 4 are much slower for the linear operator although the number of operations is very similar. Is this be due to the poor preallocation? Is there a way to improve the performance? Thank you very much, Xavier -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Mar 10 10:10:51 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 10 Mar 2012 10:10:51 -0600 Subject: [petsc-users] performance issue In-Reply-To: References: Message-ID: On Sat, Mar 10, 2012 at 09:59, Xavier Garnaud < xavier.garnaud at ladhyx.polytechnique.fr> wrote: > am solving the compressible Navier--Stokes equations in compressible form, > so in order to apply the operator, I > > 1. apply BCs on the flow field > 2. compute the flux > 3. take the derivative using finite differences > 4. apply BCs on the derivatives of the flux > > > In order to apply the linearized operator, I wish to linearize steps 2 and > 4 (the other are linear). For this I assemble sparse matrices (MPIAIJ). The > matrices should be block diagonal -- with square or rectangular blocks -- > so I preallocate the whole diagonal blocks (but I only use MatSetValues for > nonzero entries). When I do this, the linearized code runs approximately > 50% slower (the computation of derivatives takes more that 70% of the time > in the non-linear code), so steps 2 and 4 are much slower for the linear > operator although the number of operations is very similar. Is this be due > to the poor preallocation? Is there a way to improve the performance? > It's not clear to me from this description if you are even using an implicit method. Is the linearization for use in a Newton iteration? How often do you have to reassemble? Please always send -log_summary output with performance questions. -------------- next part -------------- An HTML attachment was scrubbed... URL: From xavier.garnaud at ladhyx.polytechnique.fr Sat Mar 10 11:05:45 2012 From: xavier.garnaud at ladhyx.polytechnique.fr (Xavier Garnaud) Date: Sat, 10 Mar 2012 18:05:45 +0100 Subject: [petsc-users] performance issue In-Reply-To: References: Message-ID: I am using an explicit time stepper. The matrices are assembled only once, and then I use the linear operator for example to compute the least stable eigenmode(s). I attached the output of log_summary for performing the same number of time steps using the linear and nonlinear operators. On Sat, Mar 10, 2012 at 5:10 PM, Jed Brown wrote: > On Sat, Mar 10, 2012 at 09:59, Xavier Garnaud < > xavier.garnaud at ladhyx.polytechnique.fr> wrote: > >> am solving the compressible Navier--Stokes equations in compressible >> form, so in order to apply the operator, I >> >> 1. apply BCs on the flow field >> 2. compute the flux >> 3. take the derivative using finite differences >> 4. apply BCs on the derivatives of the flux >> >> >> In order to apply the linearized operator, I wish to linearize steps 2 >> and 4 (the other are linear). For this I assemble sparse matrices (MPIAIJ). >> The matrices should be block diagonal -- with square or rectangular blocks >> -- so I preallocate the whole diagonal blocks (but I only use MatSetValues >> for nonzero entries). When I do this, the linearized code runs >> approximately 50% slower (the computation of derivatives takes more that >> 70% of the time in the non-linear code), so steps 2 and 4 are much slower >> for the linear operator although the number of operations is very similar. >> Is this be due to the poor preallocation? Is there a way to improve the >> performance? >> > > It's not clear to me from this description if you are even using an > implicit method. Is the linearization for use in a Newton iteration? How > often do you have to reassemble? Please always send -log_summary output > with performance questions. > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- nx = 256 ny = 128 JET MESH ix = 74 iy_in = 44 iy_out = 65 rank: 0, i0-i1xj0-j1 : 0 - 74 x 44 - 63 rank: 2, i0-i1xj0-j1 : 0 - 74 x 64 - 65 TIME STEPPING Tf = 1 CFL = 2 State loaded from q0.h5 32768 40960 32768 40960 32768 40960 32768 40960 Euler - x Euler - y LODI - x LODIq - x LODI - y LODIq - y Stress - x Stress - x dFv/dq - x dFv/dtau - x dFv/dq - y dFv/dtau - y |MatEulerx | = 21.7871 |MatEulery | = 10.4999 |MatLODIx | = 13.3652 |MatLODIy | = 15.0075 |MatLODIqx | = 4.58531e+06 |MatLODIqy | = 1.00002 |MatViscousx_q | = 0.00122487 |MatViscousy_q | = 0.00125045 |MatViscousx_tau | = 1.99893 |MatViscousy_tau | = 1.99893 dt = 0.00571429 |q(1.000000)|/|q(0)| = 1.84842 Elapsed time (CPU) = 27.2226 s ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./ns2d on a real_opt named muzo.polytechnique.fr with 4 processors, by garnaud Sat Mar 10 18:02:03 2012 Using Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54 CDT 2011 Max Max/Min Avg Total Time (sec): 2.762e+01 1.00000 2.762e+01 Objects: 1.900e+02 1.00000 1.900e+02 Flops: 1.068e+10 1.01222 1.065e+10 4.258e+10 Flops/sec: 3.869e+08 1.01222 3.855e+08 1.542e+09 MPI Messages: 3.260e+04 1.00000 3.260e+04 1.304e+05 MPI Message Lengths: 2.277e+08 1.00000 6.984e+03 9.108e+08 MPI Reductions: 4.280e+02 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 2.7615e+01 100.0% 4.2584e+10 100.0% 1.304e+05 100.0% 6.984e+03 100.0% 4.270e+02 99.8% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage DERIVATIVES 10508 1.0 1.4299e+01 1.0 8.21e+09 1.0 0.0e+00 0.0e+00 0.0e+00 51 77 0 0 0 51 77 0 0 0 2295 FILTERS 350 1.0 1.9905e-01 1.0 1.26e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 2535 VECMANIP 21716 1.0 2.8288e+00 1.2 0.00e+00 0.0 1.3e+05 7.0e+03 6.0e+00 9 0100100 1 9 0100100 1 0 VecView 6 1.0 3.7352e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecNorm 2 1.0 5.9009e-04 3.3 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScale 1800 1.0 7.1079e-02 1.2 5.90e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 3323 VecCopy 414 1.0 3.7731e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 7051 1.0 7.2879e-01 1.1 4.97e+08 1.0 0.0e+00 0.0e+00 0.0e+00 3 5 0 0 0 3 5 0 0 0 2726 VecAXPBYCZ 350 1.0 6.3609e-02 1.0 4.01e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2524 VecLoad 1 1.0 1.8210e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 31858 1.0 1.5961e+00 1.1 0.00e+00 0.0 1.3e+05 7.0e+03 0.0e+00 6 0100100 0 6 0100100 0 0 VecScatterEnd 31858 1.0 8.4421e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 IMPOSEBC_VISC 5251 1.0 7.2675e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 IMPOSEBC_EULER 1945 1.0 8.8332e-03 4.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 FLUXES_VISC 22 1.0 4.4665e-03 1.1 4.33e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 3874 FLUXES_EULER 14 1.0 2.4092e-03 1.3 2.75e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4570 STRESSES 12 1.0 1.9977e-03 1.1 1.67e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 3346 MatMult 12250 1.0 6.6647e+00 1.0 1.34e+09 1.1 0.0e+00 0.0e+00 0.0e+00 24 12 0 0 0 24 12 0 0 0 784 MatMultAdd 8750 1.0 2.5075e+00 1.0 4.13e+08 1.1 0.0e+00 0.0e+00 0.0e+00 9 4 0 0 0 9 4 0 0 0 642 MatAssemblyBegin 12 1.0 7.1454e-04 3.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01 0 0 0 0 6 0 0 0 0 6 0 MatAssemblyEnd 12 1.0 2.1005e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.1e+02 0 0 0 0 25 0 0 0 0 25 0 TSStep 175 1.0 2.6759e+01 1.0 1.05e+10 1.0 1.3e+05 7.0e+03 6.0e+00 97 99 97 97 1 97 99 97 97 1 1570 TSFunctionEval 1750 1.0 2.6487e+01 1.0 1.04e+10 1.0 1.3e+05 7.0e+03 0.0e+00 96 97 97 97 0 96 97 97 97 0 1562 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Distributed Mesh 5 5 905272 0 Vector 59 59 6793272 0 Vector Scatter 22 22 23320 0 Index Set 49 49 220252 0 IS L to G Mapping 10 10 703172 0 Viewer 4 3 2064 0 Matrix 39 36 33641936 0 TS 1 1 1088 0 SNES 1 1 1200 0 ======================================================================================================================== Average time to get PetscTime(): 9.53674e-08 Average time for MPI_Barrier(): 3.57628e-06 Average time for zero size MPI_Send(): 5.24521e-06 #PETSc Option Table entries: -Tf 1. -cfl 2 -lints -log_summary -nx 256 -ny 128 -save 1. #End of PETSc Option Table entries Compiled with FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 Configure run at: Thu Feb 16 17:51:17 2012 Configure options: --with-mpi=yes --with-shared-libraries --with-scalar-type=real --with-fortran-interfaces=1 --FFLAGS=-I/usr/include --with-fortran --with-fortran-kernels=1 --with-clanguage=c COPTFLAGS=-O3 FOPTFLAGS=-O3 --download-mumps=MUMPS_4.10.0.tar.gz --download-scalapack=SCALAPACK-1.7.tar.gz --download-blacs=blacs-dev.tar.gz --download-parmetis=ParMetis-3.2.0-p1.tar.gz --download-superlu=superlu_4.2.tar.gz --download-superlu_dist=superlu_dist_2.5.tar.gz --download-spooles=spooles-2.2-dec-2008.tar.gz --download-umfpack=UMFPACK-5.5.1.tar.gz --with-debugging=0 --with-mpi-dir=/home/garnaud/local/openmpi-1.4.4 --download-hdf5 --download-f-blas-lapack ----------------------------------------- Libraries compiled on Thu Feb 16 17:51:17 2012 on muzo.polytechnique.fr Machine characteristics: Linux-2.6.39.4-4.2-desktop-x86_64-with-mandrake-2011.0-Official Using PETSc directory: /home/garnaud/local/petsc/petsc-3.2-p5 Using PETSc arch: real_opt ----------------------------------------- Using C compiler: /home/garnaud/local/openmpi-1.4.4/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O3 ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /home/garnaud/local/openmpi-1.4.4/bin/mpif90 -I/usr/include -fPIC -O3 ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/home/garnaud/local/petsc/petsc-3.2-p5/real_opt/include -I/home/garnaud/local/petsc/petsc-3.2-p5/include -I/home/garnaud/local/petsc/petsc-3.2-p5/include -I/home/garnaud/local/petsc/petsc-3.2-p5/real_opt/include -I/home/garnaud/local/openmpi-1.4.4/include ----------------------------------------- Using C linker: /home/garnaud/local/openmpi-1.4.4/bin/mpicc Using Fortran linker: /home/garnaud/local/openmpi-1.4.4/bin/mpif90 Using libraries: -Wl,-rpath,/home/garnaud/local/petsc/petsc-3.2-p5/real_opt/lib -L/home/garnaud/local/petsc/petsc-3.2-p5/real_opt/lib -lpetsc -lX11 -lpthread -Wl,-rpath,/home/garnaud/local/petsc/petsc-3.2-p5/real_opt/lib -L/home/garnaud/local/petsc/petsc-3.2-p5/real_opt/lib -lsuperlu_dist_2.5 -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lspooles -lscalapack -lblacs -lsuperlu_4.2 -lumfpack -lamd -lflapack -lfblas -lhdf5_fortran -lhdf5 -lz -Wl,-rpath,/home/garnaud/local/openmpi-1.4.4/lib -L/home/garnaud/local/openmpi-1.4.4/lib -Wl,-rpath,/usr/lib64/gcc/x86_64-mandriva-linux-gnu/4.6.1 -L/usr/lib64/gcc/x86_64-mandriva-linux-gnu/4.6.1 -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lm -lquadmath -lm -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl ----------------------------------------- -------------- next part -------------- nx = 256 ny = 128 JET MESH ix = 74 iy_in = 44 iy_out = 65 rank: 0, i0-i1xj0-j1 : 0 - 74 x 44 - 63 rank: 2, i0-i1xj0-j1 : 0 - 74 x 64 - 65 TIME STEPPING Tf = 1 CFL = 2 dt = 0.00571429 |q(1.000000)|/|q(0)| = 1.0005 Elapsed time (CPU) = 19.2814 s Final state saved in q0.h5 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./ns2d on a real_opt named muzo.polytechnique.fr with 4 processors, by garnaud Sat Mar 10 18:03:09 2012 Using Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54 CDT 2011 Max Max/Min Avg Total Time (sec): 1.955e+01 1.00000 1.955e+01 Objects: 8.400e+01 1.00000 8.400e+01 Flops: 1.090e+10 1.00000 1.090e+10 4.358e+10 Flops/sec: 5.574e+08 1.00000 5.574e+08 2.229e+09 MPI Messages: 3.259e+04 1.00000 3.259e+04 1.303e+05 MPI Message Lengths: 2.276e+08 1.00000 6.984e+03 9.103e+08 MPI Reductions: 1.180e+02 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.9549e+01 100.0% 4.3584e+10 100.0% 1.303e+05 100.0% 6.984e+03 100.0% 1.170e+02 99.2% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage DERIVATIVES 10502 1.0 1.4136e+01 1.0 8.20e+09 1.0 0.0e+00 0.0e+00 0.0e+00 72 75 0 0 0 72 75 0 0 0 2321 FILTERS 350 1.0 1.9755e-01 1.0 1.26e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 2554 VECMANIP 21704 1.0 2.4231e+00 1.2 0.00e+00 0.0 1.3e+05 7.0e+03 6.0e+00 11 0100100 5 11 0100100 5 0 VecView 7 1.0 4.2857e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 VecNorm 2 1.0 6.0606e-04 3.7 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 2 0 0 0 0 2 0 VecScale 1750 1.0 6.4685e-02 1.1 5.73e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 3546 VecCopy 352 1.0 3.0717e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 8750 1.0 8.0684e-01 1.1 6.08e+08 1.0 0.0e+00 0.0e+00 0.0e+00 4 6 0 0 0 4 6 0 0 0 3013 VecAXPBYCZ 350 1.0 6.5956e-02 1.0 4.01e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2434 VecScatterBegin 10852 1.0 1.4070e+00 1.1 0.00e+00 0.0 1.3e+05 7.0e+03 0.0e+00 7 0100100 0 7 0100100 0 0 VecScatterEnd 10852 1.0 7.2064e-01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 3 0 0 0 0 3 0 0 0 0 0 IMPOSEBC_VISC 5250 1.0 7.5293e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 IMPOSEBC_EULER 5425 1.0 5.2320e-02 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 FLUXES_VISC 3500 1.0 6.5287e-01 1.1 6.88e+08 1.0 0.0e+00 0.0e+00 0.0e+00 3 6 0 0 0 3 6 0 0 0 4216 FLUXES_EULER 3500 1.0 4.5190e-01 1.1 6.88e+08 1.0 0.0e+00 0.0e+00 0.0e+00 2 6 0 0 0 2 6 0 0 0 6091 STRESSES 3500 1.0 4.7757e-01 1.1 4.87e+08 1.0 0.0e+00 0.0e+00 0.0e+00 2 4 0 0 0 2 4 0 0 0 4083 TSStep 175 1.0 1.8815e+01 1.0 1.08e+10 1.0 1.3e+05 7.0e+03 1.0e+01 96 99 97 97 8 96 99 97 97 9 2289 TSFunctionEval 1750 1.0 1.8553e+01 1.0 1.06e+10 1.0 1.3e+05 7.0e+03 4.0e+00 95 97 97 97 3 95 97 97 97 3 2287 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Distributed Mesh 5 5 905272 0 Vector 28 28 4715800 0 Vector Scatter 10 10 10600 0 Index Set 25 25 202300 0 IS L to G Mapping 10 10 703172 0 Viewer 4 3 2064 0 TS 1 1 1088 0 SNES 1 1 1200 0 ======================================================================================================================== Average time to get PetscTime(): 2.14577e-07 Average time for MPI_Barrier(): 3.57628e-06 Average time for zero size MPI_Send(): 5.00679e-06 #PETSc Option Table entries: -Tf 1. -cfl 2 -log_summary -nx 256 -ny 128 -save 1. -ts #End of PETSc Option Table entries Compiled with FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 Configure run at: Thu Feb 16 17:51:17 2012 Configure options: --with-mpi=yes --with-shared-libraries --with-scalar-type=real --with-fortran-interfaces=1 --FFLAGS=-I/usr/include --with-fortran --with-fortran-kernels=1 --with-clanguage=c COPTFLAGS=-O3 FOPTFLAGS=-O3 --download-mumps=MUMPS_4.10.0.tar.gz --download-scalapack=SCALAPACK-1.7.tar.gz --download-blacs=blacs-dev.tar.gz --download-parmetis=ParMetis-3.2.0-p1.tar.gz --download-superlu=superlu_4.2.tar.gz --download-superlu_dist=superlu_dist_2.5.tar.gz --download-spooles=spooles-2.2-dec-2008.tar.gz --download-umfpack=UMFPACK-5.5.1.tar.gz --with-debugging=0 --with-mpi-dir=/home/garnaud/local/openmpi-1.4.4 --download-hdf5 --download-f-blas-lapack ----------------------------------------- Libraries compiled on Thu Feb 16 17:51:17 2012 on muzo.polytechnique.fr Machine characteristics: Linux-2.6.39.4-4.2-desktop-x86_64-with-mandrake-2011.0-Official Using PETSc directory: /home/garnaud/local/petsc/petsc-3.2-p5 Using PETSc arch: real_opt ----------------------------------------- Using C compiler: /home/garnaud/local/openmpi-1.4.4/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O3 ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /home/garnaud/local/openmpi-1.4.4/bin/mpif90 -I/usr/include -fPIC -O3 ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/home/garnaud/local/petsc/petsc-3.2-p5/real_opt/include -I/home/garnaud/local/petsc/petsc-3.2-p5/include -I/home/garnaud/local/petsc/petsc-3.2-p5/include -I/home/garnaud/local/petsc/petsc-3.2-p5/real_opt/include -I/home/garnaud/local/openmpi-1.4.4/include ----------------------------------------- Using C linker: /home/garnaud/local/openmpi-1.4.4/bin/mpicc Using Fortran linker: /home/garnaud/local/openmpi-1.4.4/bin/mpif90 Using libraries: -Wl,-rpath,/home/garnaud/local/petsc/petsc-3.2-p5/real_opt/lib -L/home/garnaud/local/petsc/petsc-3.2-p5/real_opt/lib -lpetsc -lX11 -lpthread -Wl,-rpath,/home/garnaud/local/petsc/petsc-3.2-p5/real_opt/lib -L/home/garnaud/local/petsc/petsc-3.2-p5/real_opt/lib -lsuperlu_dist_2.5 -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lspooles -lscalapack -lblacs -lsuperlu_4.2 -lumfpack -lamd -lflapack -lfblas -lhdf5_fortran -lhdf5 -lz -Wl,-rpath,/home/garnaud/local/openmpi-1.4.4/lib -L/home/garnaud/local/openmpi-1.4.4/lib -Wl,-rpath,/usr/lib64/gcc/x86_64-mandriva-linux-gnu/4.6.1 -L/usr/lib64/gcc/x86_64-mandriva-linux-gnu/4.6.1 -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lm -lquadmath -lm -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl ----------------------------------------- From jedbrown at mcs.anl.gov Sat Mar 10 11:21:48 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 10 Mar 2012 11:21:48 -0600 Subject: [petsc-users] performance issue In-Reply-To: References: Message-ID: On Sat, Mar 10, 2012 at 11:05, Xavier Garnaud < xavier.garnaud at ladhyx.polytechnique.fr> wrote: > I am using an explicit time stepper. The matrices are assembled only once, > and then I use the linear operator for example to compute the least stable > eigenmode(s). I attached the output of log_summary for performing the same > number of time steps using the linear and nonlinear operators. Do you assemble more than one matrix as part of defining its action? I ask because there is about 3 times more VecScatterBegin/Ends for the linear version (although they send the same amount of data, so some calls don't do any communication). I don't see anything here indicating an implicit solve, just TSFunctionEval. If TS did an implicit solve, there should be SNES/KSP/PC events. Why do you want an assembled matrix? The matrix uses more memory, so if your nonlinear function evaluation is efficient, it may well be faster to evaluate than to multiply by the matrix. > > > > On Sat, Mar 10, 2012 at 5:10 PM, Jed Brown wrote: > >> On Sat, Mar 10, 2012 at 09:59, Xavier Garnaud < >> xavier.garnaud at ladhyx.polytechnique.fr> wrote: >> >>> am solving the compressible Navier--Stokes equations in compressible >>> form, so in order to apply the operator, I >>> >>> 1. apply BCs on the flow field >>> 2. compute the flux >>> 3. take the derivative using finite differences >>> 4. apply BCs on the derivatives of the flux >>> >>> >>> In order to apply the linearized operator, I wish to linearize steps 2 >>> and 4 (the other are linear). For this I assemble sparse matrices (MPIAIJ). >>> The matrices should be block diagonal -- with square or rectangular blocks >>> -- so I preallocate the whole diagonal blocks (but I only use MatSetValues >>> for nonzero entries). When I do this, the linearized code runs >>> approximately 50% slower (the computation of derivatives takes more that >>> 70% of the time in the non-linear code), so steps 2 and 4 are much slower >>> for the linear operator although the number of operations is very similar. >>> Is this be due to the poor preallocation? Is there a way to improve the >>> performance? >>> >> >> It's not clear to me from this description if you are even using an >> implicit method. Is the linearization for use in a Newton iteration? How >> often do you have to reassemble? Please always send -log_summary output >> with performance questions. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xavier.garnaud at ladhyx.polytechnique.fr Sat Mar 10 11:47:30 2012 From: xavier.garnaud at ladhyx.polytechnique.fr (Xavier Garnaud) Date: Sat, 10 Mar 2012 18:47:30 +0100 Subject: [petsc-users] performance issue In-Reply-To: References: Message-ID: I assemble 12 matrices (for each dimension : viscous and inviscid fluxes (2 matrices for the dependency on the flow field and the stresses), viscous stresses, inviscid BCs (x2 as they depend on the flow field and its derivative)). There is no linear or non-linear solve. I want an assembled matrix for two reasons. First, it allows me to linearize automatically the operators (by using a variation of the variable of ~1e-8: as all the matrices correspond to functions local to each discretization points, this can be done with as many function evaluations as DOFs per discretization point) without doing it "on the fly", and second it allows to take the adjoint of the linear operator. On Sat, Mar 10, 2012 at 6:21 PM, Jed Brown wrote: > On Sat, Mar 10, 2012 at 11:05, Xavier Garnaud < > xavier.garnaud at ladhyx.polytechnique.fr> wrote: > >> I am using an explicit time stepper. The matrices are assembled only >> once, and then I use the linear operator for example to compute the least >> stable eigenmode(s). I attached the output of log_summary for performing >> the same number of time steps using the linear and nonlinear operators. > > > Do you assemble more than one matrix as part of defining its action? I ask > because there is about 3 times more VecScatterBegin/Ends for the linear > version (although they send the same amount of data, so some calls don't do > any communication). > > I don't see anything here indicating an implicit solve, just > TSFunctionEval. If TS did an implicit solve, there should be SNES/KSP/PC > events. > > Why do you want an assembled matrix? The matrix uses more memory, so if > your nonlinear function evaluation is efficient, it may well be faster to > evaluate than to multiply by the matrix. > > >> >> >> >> On Sat, Mar 10, 2012 at 5:10 PM, Jed Brown wrote: >> >>> On Sat, Mar 10, 2012 at 09:59, Xavier Garnaud < >>> xavier.garnaud at ladhyx.polytechnique.fr> wrote: >>> >>>> am solving the compressible Navier--Stokes equations in compressible >>>> form, so in order to apply the operator, I >>>> >>>> 1. apply BCs on the flow field >>>> 2. compute the flux >>>> 3. take the derivative using finite differences >>>> 4. apply BCs on the derivatives of the flux >>>> >>>> >>>> In order to apply the linearized operator, I wish to linearize steps 2 >>>> and 4 (the other are linear). For this I assemble sparse matrices (MPIAIJ). >>>> The matrices should be block diagonal -- with square or rectangular blocks >>>> -- so I preallocate the whole diagonal blocks (but I only use MatSetValues >>>> for nonzero entries). When I do this, the linearized code runs >>>> approximately 50% slower (the computation of derivatives takes more that >>>> 70% of the time in the non-linear code), so steps 2 and 4 are much slower >>>> for the linear operator although the number of operations is very similar. >>>> Is this be due to the poor preallocation? Is there a way to improve the >>>> performance? >>>> >>> >>> It's not clear to me from this description if you are even using an >>> implicit method. Is the linearization for use in a Newton iteration? How >>> often do you have to reassemble? Please always send -log_summary output >>> with performance questions. >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Mar 10 11:58:08 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 10 Mar 2012 11:58:08 -0600 Subject: [petsc-users] performance issue In-Reply-To: References: Message-ID: On Sat, Mar 10, 2012 at 11:47, Xavier Garnaud < xavier.garnaud at ladhyx.polytechnique.fr> wrote: > I assemble 12 matrices (for each dimension : viscous and inviscid fluxes > (2 matrices for the dependency on the flow field and the stresses), viscous > stresses, inviscid BCs (x2 as they depend on the flow field and its > derivative)). > Okay, this effectively involves more traversals of memory, so it's expected to be slower than the nonlinear version that does it all in one shot. > There is no linear or non-linear solve. > > I want an assembled matrix for two reasons. First, it allows me to > linearize automatically the operators (by using a variation of the variable > of ~1e-8: as all the matrices correspond to functions local to each > discretization points, this can be done with as many function evaluations > as DOFs per discretization point) without doing it "on the fly", and second > it allows to take the adjoint of the linear operator. > Okay, the adjoint is the key reason. I can imagine code being structured so that this is a good way to apply the adjoint. If that's the case, just go with it, it's entirely reasonable for an adjoint to cost twice as much as the forward operator. Note that you might want to consider assembling one operator that is the entire action (instead of many separate sparse matrices for different terms). The single operator should be faster to apply and is what you generally want to use for preconditioning (if you wanted to use an implicit time integration method). -------------- next part -------------- An HTML attachment was scrubbed... URL: From Markus.Iivonen at metropolia.fi Sat Mar 10 11:49:33 2012 From: Markus.Iivonen at metropolia.fi (Markus Iivonen) Date: Sat, 10 Mar 2012 17:49:33 +0000 Subject: [petsc-users] Creating a dll on 64-bit environment Message-ID: Hello, I've written a program on 32-bit environment and I want to use a dll. Everything worked fine until I uploaded the code on the other machine which run 64-bit os. At first I got an error: "/usr/bin/ld: MyClass.o: relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC MyClass.o: could not read symbols: Bad value" what was new for me because I haven't worked on 64bit environment before. Anyway at first I just tried to modify "-${CLINKER} -shared -Wl,-soname,${SONAME} -o ${VERS} *.o ${PETSC_MAT_LIB}" as "-${CLINKER} -shared -fPIC -Wl,-soname,${SONAME} -o ${VERS} *.o ${PETSC_MAT_LIB}" even after that I got the same error, after that I added -fPIC to CFLAGS and the error changed: "../petsc/petsc-3.2-p6/arch-linux2-cxx-debug/lib/libpetsc.a(err.o): relocation R_X86_64_32 against `ompi_mpi_comm_self' can not be used when making a shared object; recompile with -fPIC" So here is my current makefile CFLAGS = - fPIC CCPPFLAGS = LOCDIR = /home/user/petsc/project/ EXAMPLESC = x1.cpp x2.cpp x3.cpp PETSC_DIR = /home/user/petsc/petsc-3.2-p6/ MYLIB = -L/home/user/petsc/project/opt/lib/ -libmyclass LIBADD = /.../user/petsc/project/opt/lib/ VERS = libmyclass.so.1.0 SONAME = libmyclass.so.1 SOWOV = libmyclass.so include ${PETSC_DIR}/conf/variables include ${PETSC_DIR}/conf/rules MyProgram: main.o chkopts -${CLINKER} -o $@ $< ${MYLIB} ${PETSC_MAT_LIB} ${RM} main.o export LD_LIBRARY_PATH=${LIBADD}:$LD_LIBRARY_PATH LibMyProgram.so: MyClass.o chkopts -${CLINKER} -shared -Wl,-soname,${SONAME} -o ${VERS} *.o ${PETSC_MAT_LIB} mv ${VERS} ${LIBADD} ln -sf ${LIBADD}${VERS} ${LIBADD}${SOWOV} ln -sf ${LIBADD}${VERS} ${LIBADD}${SONAME} So would there be any kind of solution to this problem ? Thank You. Sincerely Markus Iivonen From knepley at gmail.com Sat Mar 10 12:56:16 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 10 Mar 2012 12:56:16 -0600 Subject: [petsc-users] Creating a dll on 64-bit environment In-Reply-To: References: Message-ID: On Sat, Mar 10, 2012 at 11:49 AM, Markus Iivonen < Markus.Iivonen at metropolia.fi> wrote: > Hello, > > I've written a program on 32-bit environment and I want to use a dll. > Everything worked fine until I uploaded the code on the other machine which > run 64-bit os. > At first I got an error: > > "/usr/bin/ld: MyClass.o: relocation R_X86_64_32S against `.rodata' can not > be used when making a shared object; recompile with -fPIC > MyClass.o: could not read symbols: Bad value" > > what was new for me because I haven't worked on 64bit environment before. > Anyway at first I just tried to modify "-${CLINKER} -shared > -Wl,-soname,${SONAME} -o ${VERS} *.o ${PETSC_MAT_LIB}" as "-${CLINKER} > -shared -fPIC -Wl,-soname,${SONAME} -o ${VERS} *.o ${PETSC_MAT_LIB}" > > even after that I got the same error, after that I added -fPIC to CFLAGS > and the error changed: > > "../petsc/petsc-3.2-p6/arch-linux2-cxx-debug/lib/libpetsc.a(err.o): > relocation R_X86_64_32 against `ompi_mpi_comm_self' can not be used when > making a shared object; recompile with -fPIC" > This is a known problem with OpenMPI. There response was "cry me a river". Switch to MPICH and make sure you configure with --with-shared-libraries. Matt > So here is my current makefile > > > CFLAGS = - fPIC > CCPPFLAGS = > LOCDIR = /home/user/petsc/project/ > EXAMPLESC = x1.cpp x2.cpp x3.cpp > PETSC_DIR = /home/user/petsc/petsc-3.2-p6/ > > MYLIB = -L/home/user/petsc/project/opt/lib/ -libmyclass > LIBADD = /.../user/petsc/project/opt/lib/ > > VERS = libmyclass.so.1.0 > SONAME = libmyclass.so.1 > SOWOV = libmyclass.so > > include ${PETSC_DIR}/conf/variables > include ${PETSC_DIR}/conf/rules > > > > > MyProgram: main.o chkopts > -${CLINKER} -o $@ $< ${MYLIB} ${PETSC_MAT_LIB} > ${RM} main.o > > export LD_LIBRARY_PATH=${LIBADD}:$LD_LIBRARY_PATH > > LibMyProgram.so: MyClass.o chkopts > -${CLINKER} -shared -Wl,-soname,${SONAME} -o ${VERS} *.o > ${PETSC_MAT_LIB} > > mv ${VERS} ${LIBADD} > ln -sf ${LIBADD}${VERS} ${LIBADD}${SOWOV} > ln -sf ${LIBADD}${VERS} ${LIBADD}${SONAME} > > So would there be any kind of solution to this problem ? > > Thank You. > > Sincerely Markus Iivonen -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sat Mar 10 12:57:09 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Sat, 10 Mar 2012 12:57:09 -0600 (CST) Subject: [petsc-users] Creating a dll on 64-bit environment In-Reply-To: References: Message-ID: Everything that goes into .so file should be built with -fPIC. So you have to make sure MPI/PETSc are built with -fPIC flag. One way to do this is to build woth PETSc and MPI as shared libraries. Not sure what MPI you are using - but wrt PETSc - use the configure option --with-shared-libraries=1 Satish On Sat, 10 Mar 2012, Markus Iivonen wrote: > Hello, > > I've written a program on 32-bit environment and I want to use a dll. Everything worked fine until I uploaded the code on the other machine which run 64-bit os. > At first I got an error: > > "/usr/bin/ld: MyClass.o: relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC > MyClass.o: could not read symbols: Bad value" > > what was new for me because I haven't worked on 64bit environment before. Anyway at first I just tried to modify "-${CLINKER} -shared -Wl,-soname,${SONAME} -o ${VERS} *.o ${PETSC_MAT_LIB}" as "-${CLINKER} -shared -fPIC -Wl,-soname,${SONAME} -o ${VERS} *.o ${PETSC_MAT_LIB}" > > even after that I got the same error, after that I added -fPIC to CFLAGS and the error changed: > > "../petsc/petsc-3.2-p6/arch-linux2-cxx-debug/lib/libpetsc.a(err.o): relocation R_X86_64_32 against `ompi_mpi_comm_self' can not be used when making a shared object; recompile with -fPIC" > > So here is my current makefile > > > CFLAGS = - fPIC > CCPPFLAGS = > LOCDIR = /home/user/petsc/project/ > EXAMPLESC = x1.cpp x2.cpp x3.cpp > PETSC_DIR = /home/user/petsc/petsc-3.2-p6/ > > MYLIB = -L/home/user/petsc/project/opt/lib/ -libmyclass > LIBADD = /.../user/petsc/project/opt/lib/ > > VERS = libmyclass.so.1.0 > SONAME = libmyclass.so.1 > SOWOV = libmyclass.so > > include ${PETSC_DIR}/conf/variables > include ${PETSC_DIR}/conf/rules > > > > > MyProgram: main.o chkopts > -${CLINKER} -o $@ $< ${MYLIB} ${PETSC_MAT_LIB} > ${RM} main.o > > export LD_LIBRARY_PATH=${LIBADD}:$LD_LIBRARY_PATH > > LibMyProgram.so: MyClass.o chkopts > -${CLINKER} -shared -Wl,-soname,${SONAME} -o ${VERS} *.o ${PETSC_MAT_LIB} > > mv ${VERS} ${LIBADD} > ln -sf ${LIBADD}${VERS} ${LIBADD}${SOWOV} > ln -sf ${LIBADD}${VERS} ${LIBADD}${SONAME} > > So would there be any kind of solution to this problem ? > > Thank You. > > Sincerely Markus Iivonen From jedbrown at mcs.anl.gov Sat Mar 10 12:59:29 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 10 Mar 2012 12:59:29 -0600 Subject: [petsc-users] Creating a dll on 64-bit environment In-Reply-To: References: Message-ID: On Sat, Mar 10, 2012 at 11:49, Markus Iivonen wrote: > Hello, > > I've written a program on 32-bit environment and I want to use a dll. > Everything worked fine until I uploaded the code on the other machine which > run 64-bit os. > At first I got an error: > > "/usr/bin/ld: MyClass.o: relocation R_X86_64_32S against `.rodata' can not > be used when making a shared object; recompile with -fPIC > MyClass.o: could not read symbols: Bad value" > And are you intending to build a 64-bit native binary? If so, you should be sure to compile everything with the appropriate compiler flags (-m64 or -march=native, for example). If you want a shared PETSc library, be sure to configure --with-shared-libraries > > what was new for me because I haven't worked on 64bit environment before. > Anyway at first I just tried to modify "-${CLINKER} -shared > -Wl,-soname,${SONAME} -o ${VERS} *.o ${PETSC_MAT_LIB}" as "-${CLINKER} > -shared -fPIC -Wl,-soname,${SONAME} -o ${VERS} *.o ${PETSC_MAT_LIB}" > > even after that I got the same error, after that I added -fPIC to CFLAGS > and the error changed: > > "../petsc/petsc-3.2-p6/arch-linux2-cxx-debug/lib/libpetsc.a(err.o): > relocation R_X86_64_32 against `ompi_mpi_comm_self' can not be used when > making a shared object; recompile with -fPIC" > > So here is my current makefile > > > CFLAGS = - fPIC > CCPPFLAGS = > LOCDIR = /home/user/petsc/project/ > EXAMPLESC = x1.cpp x2.cpp x3.cpp > PETSC_DIR = /home/user/petsc/petsc-3.2-p6/ > > MYLIB = -L/home/user/petsc/project/opt/lib/ -libmyclass > LIBADD = /.../user/petsc/project/opt/lib/ > > VERS = libmyclass.so.1.0 > SONAME = libmyclass.so.1 > SOWOV = libmyclass.so > > include ${PETSC_DIR}/conf/variables > include ${PETSC_DIR}/conf/rules > > > > > MyProgram: main.o chkopts > -${CLINKER} -o $@ $< ${MYLIB} ${PETSC_MAT_LIB} > ${RM} main.o > > export LD_LIBRARY_PATH=${LIBADD}:$LD_LIBRARY_PATH > > LibMyProgram.so: MyClass.o chkopts > -${CLINKER} -shared -Wl,-soname,${SONAME} -o ${VERS} *.o > ${PETSC_MAT_LIB} > > mv ${VERS} ${LIBADD} > ln -sf ${LIBADD}${VERS} ${LIBADD}${SOWOV} > ln -sf ${LIBADD}${VERS} ${LIBADD}${SONAME} > > So would there be any kind of solution to this problem ? > > Thank You. > > Sincerely Markus Iivonen -------------- next part -------------- An HTML attachment was scrubbed... URL: From Markus.Iivonen at metropolia.fi Sat Mar 10 13:14:39 2012 From: Markus.Iivonen at metropolia.fi (Markus Iivonen) Date: Sat, 10 Mar 2012 19:14:39 +0000 Subject: [petsc-users] Creating a dll on 64-bit environment In-Reply-To: References: , Message-ID: I'm using openmpi, I'll try to compile PETSc and openmpi with shared libraries. Thank you so much! ________________________________________ From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Satish Balay [balay at mcs.anl.gov] Sent: Saturday, March 10, 2012 12:57 PM To: PETSc users list Subject: Re: [petsc-users] Creating a dll on 64-bit environment Everything that goes into .so file should be built with -fPIC. So you have to make sure MPI/PETSc are built with -fPIC flag. One way to do this is to build woth PETSc and MPI as shared libraries. Not sure what MPI you are using - but wrt PETSc - use the configure option --with-shared-libraries=1 Satish On Sat, 10 Mar 2012, Markus Iivonen wrote: > Hello, > > I've written a program on 32-bit environment and I want to use a dll. Everything worked fine until I uploaded the code on the other machine which run 64-bit os. > At first I got an error: > > "/usr/bin/ld: MyClass.o: relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC > MyClass.o: could not read symbols: Bad value" > > what was new for me because I haven't worked on 64bit environment before. Anyway at first I just tried to modify "-${CLINKER} -shared -Wl,-soname,${SONAME} -o ${VERS} *.o ${PETSC_MAT_LIB}" as "-${CLINKER} -shared -fPIC -Wl,-soname,${SONAME} -o ${VERS} *.o ${PETSC_MAT_LIB}" > > even after that I got the same error, after that I added -fPIC to CFLAGS and the error changed: > > "../petsc/petsc-3.2-p6/arch-linux2-cxx-debug/lib/libpetsc.a(err.o): relocation R_X86_64_32 against `ompi_mpi_comm_self' can not be used when making a shared object; recompile with -fPIC" > > So here is my current makefile > > > CFLAGS = - fPIC > CCPPFLAGS = > LOCDIR = /home/user/petsc/project/ > EXAMPLESC = x1.cpp x2.cpp x3.cpp > PETSC_DIR = /home/user/petsc/petsc-3.2-p6/ > > MYLIB = -L/home/user/petsc/project/opt/lib/ -libmyclass > LIBADD = /.../user/petsc/project/opt/lib/ > > VERS = libmyclass.so.1.0 > SONAME = libmyclass.so.1 > SOWOV = libmyclass.so > > include ${PETSC_DIR}/conf/variables > include ${PETSC_DIR}/conf/rules > > > > > MyProgram: main.o chkopts > -${CLINKER} -o $@ $< ${MYLIB} ${PETSC_MAT_LIB} > ${RM} main.o > > export LD_LIBRARY_PATH=${LIBADD}:$LD_LIBRARY_PATH > > LibMyProgram.so: MyClass.o chkopts > -${CLINKER} -shared -Wl,-soname,${SONAME} -o ${VERS} *.o ${PETSC_MAT_LIB} > > mv ${VERS} ${LIBADD} > ln -sf ${LIBADD}${VERS} ${LIBADD}${SOWOV} > ln -sf ${LIBADD}${VERS} ${LIBADD}${SONAME} > > So would there be any kind of solution to this problem ? > > Thank You. > > Sincerely Markus Iivonen From tisaac at ices.utexas.edu Sun Mar 11 00:38:56 2012 From: tisaac at ices.utexas.edu (Tobin Isaac) Date: Sun, 11 Mar 2012 00:38:56 -0600 Subject: [petsc-users] Bug in pc_right + nonzero initial guess + {tfqmr, tcqmr, cgs} Message-ID: <20120311063856.GA12186@ices.utexas.edu> Hi, The changes made to bcgs in the commit below should be applied to tfqmr, tcqmr, and cgs as well. Cheers, Toby author Barry Smith bsmith at mcs.anl.gov Wed, 19 Jan 2011 20:10:29 -0600 (13 months ago) changeset 18159 a526acdfdc95 parent 18153 3c53022524fb child 18160 4db6569c26af fixed KSPBCGS to work with right preconditioning and nonzero initial guess -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: Digital signature URL: From C.Klaij at marin.nl Mon Mar 12 02:56:46 2012 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Mon, 12 Mar 2012 07:56:46 +0000 Subject: [petsc-users] using PCFieldSplitGetSubKSP in c++ Message-ID: >> This may be a dumb question (new to c++), well anyway: I'm trying >> to get the sub KSP associated with the Schur complement S, in >> order to set the null space. This is the code: >> >> KSP subksp[2]; >> > >Declare this as KSP *subksp; > > Matt > Thanks Matt, but when I do that I cannot use KSPSetNullSpace: KSP *subksp[2]; PetscInt n=2; ierr = PCFieldSplitGetSubKSP(pc,&n,subksp); CHKERRQ(ierr); ierr = KSPSetNullSpace(subksp[1],subnullsp); CHKERRQ(ierr); This gives the error: error: argument of type "KSP *" is incompatible with parameter of type "KSP" ierr = KSPSetNullSpace(subksp[1],subnullsp); CHKERRQ(ierr); ^ dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From jedbrown at mcs.anl.gov Mon Mar 12 07:59:18 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 12 Mar 2012 07:59:18 -0500 Subject: [petsc-users] using PCFieldSplitGetSubKSP in c++ In-Reply-To: References: Message-ID: On Mon, Mar 12, 2012 at 02:56, Klaij, Christiaan wrote: > >Declare this as KSP *subksp; > Compare this ^^^^^^^^^^^ > > > > Matt > > > > Thanks Matt, but when I do that I cannot use KSPSetNullSpace: > > KSP *subksp[2]; > To this ^^^^^^^^^^ > PetscInt n=2; > ierr = PCFieldSplitGetSubKSP(pc,&n,subksp); CHKERRQ(ierr); > ierr = KSPSetNullSpace(subksp[1],subnullsp); CHKERRQ(ierr); > > This gives the error: > > error: argument of type "KSP *" is incompatible with parameter of type > "KSP" > ierr = KSPSetNullSpace(subksp[1],subnullsp); CHKERRQ(ierr); > -------------- next part -------------- An HTML attachment was scrubbed... URL: From laurent.berenguer at gmail.com Mon Mar 12 11:50:49 2012 From: laurent.berenguer at gmail.com (Laurent Berenguer) Date: Mon, 12 Mar 2012 17:50:49 +0100 Subject: [petsc-users] PCASM and dof > 1 Message-ID: Hello, I would like to solve nonlinear PDEs on a regular grid (DMDACreate2D, 2 dof per node) using: - MatFDColoring to compute the Jacobian matrix - The ASM preconditioner Here is an overview of the code: ierr = DMGetMatrix(da,MATAIJ,&J); ierr = TSGetSNES(ts,&snes); ierr = DMGetColoring(da,IS_COLORING_GLOBAL,MATAIJ,&iscoloring); ierr = MatFDColoringCreate(J,iscoloring,&matfdcoloring); ierr = MatFDColoringSetFromOptions(matfdcoloring); ierr = ISColoringDestroy(&iscoloring); ierr = MatFDColoringSetFunction(matfdcoloring,(PetscErrorCode (*)(void))SNESTSFormFunction,ts); ierr = SNESSetJacobian(snes,J,J,SNESDefaultComputeJacobianColor,matfdcoloring); ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); ierr = KSPSetOperators(ksp,J,J,DIFFERENT_NONZERO_PATTERN); ierr = KSPGetPC(ksp,&pc); ierr = PCSetType(pc,PCASM); ierr = PCASMSetOverlap(pc,1); ierr = KSPSetUp(ksp); ierr = SNESSetKSP(snes,ksp); ierr = TSSetFromOptions(ts); ierr = TSSolve(ts,u,&ftime); And I have the following error for overlap>0. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [1]PETSC ERROR: likely location of problem given in stack below [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [1]PETSC ERROR: INSTEAD the line number of the start of the function [1]PETSC ERROR: is given. [1]PETSC ERROR: [1] MatIncreaseOverlap_MPIAIJ_Receive line 427 src/mat/impls/aij/mpi/mpiov.c [1]PETSC ERROR: [1] MatIncreaseOverlap_MPIAIJ_Once line 71 src/mat/impls/aij/mpi/mpiov.c [1]PETSC ERROR: [1] MatIncreaseOverlap_MPIAIJ line 22 src/mat/impls/aij/mpi/mpiov.c [1]PETSC ERROR: [1] MatIncreaseOverlap line 6543 src/mat/interface/matrix.c [1]PETSC ERROR: [1] PCSetUp_ASM line 155 src/ksp/pc/impls/asm/asm.c [1]PETSC ERROR: [1] PCSetUp line 797 src/ksp/pc/interface/precon.c [1]PETSC ERROR: [1] KSPSetUp line 184 src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: [1] KSPSolve line 331 src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: [1] SNES_KSPSolve line 3394 src/snes/interface/snes.c [1]PETSC ERROR: [1] SNESSolve_LS line 142 src/snes/impls/ls/ls.c [1]PETSC ERROR: [1] SNESSolve line 2647 src/snes/interface/snes.c [1]PETSC ERROR: [1] TSStep_Theta line 25 src/ts/impls/implicit/theta/theta.c [1]PETSC ERROR: [1] TSStep line 1768 src/ts/interface/ts.c [1]PETSC ERROR: [1] TSSolve line 1824 src/ts/interface/ts.c [1]PETSC ERROR: --------------------- Error Message ------------------------------------ The same error happens when I run the example /snes/examples/tutorials/ex32.c with the option -'pc_type asm'. How to properly set up the preconditioner in that case (dof > 1) ? Thanks, Laurent Berenguer PhD student, Universit? Lyon 1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Mar 12 12:56:56 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 12 Mar 2012 12:56:56 -0500 Subject: [petsc-users] PCASM and dof > 1 In-Reply-To: References: Message-ID: On Mon, Mar 12, 2012 at 11:50 AM, Laurent Berenguer < laurent.berenguer at gmail.com> wrote: > Hello, > > I would like to solve nonlinear PDEs on a regular grid (DMDACreate2D, 2 > dof per node) using: > - MatFDColoring to compute the Jacobian matrix > - The ASM preconditioner > I have just run mpiexec -n 2 ./ex32 -pc_type asm without a problem.Please send yourconfigure.log to petsc-maint at mcs.anl.gov Matt > Here is an overview of the code: > > ierr = DMGetMatrix(da,MATAIJ,&J); > ierr = TSGetSNES(ts,&snes); > ierr = DMGetColoring(da,IS_COLORING_GLOBAL,MATAIJ,&iscoloring); > ierr = MatFDColoringCreate(J,iscoloring,&matfdcoloring); > ierr = MatFDColoringSetFromOptions(matfdcoloring); > ierr = ISColoringDestroy(&iscoloring); > ierr = MatFDColoringSetFunction(matfdcoloring,(PetscErrorCode > (*)(void))SNESTSFormFunction,ts); > ierr = > SNESSetJacobian(snes,J,J,SNESDefaultComputeJacobianColor,matfdcoloring); > ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); > ierr = KSPSetOperators(ksp,J,J,DIFFERENT_NONZERO_PATTERN); > ierr = KSPGetPC(ksp,&pc); > ierr = PCSetType(pc,PCASM); > ierr = PCASMSetOverlap(pc,1); > ierr = KSPSetUp(ksp); > ierr = SNESSetKSP(snes,ksp); > ierr = TSSetFromOptions(ts); > ierr = TSSolve(ts,u,&ftime); > > And I have the following error for overlap>0. > > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [1]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [1]PETSC ERROR: likely location of problem given in stack below > [1]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [1]PETSC ERROR: INSTEAD the line number of the start of the function > [1]PETSC ERROR: is given. > [1]PETSC ERROR: [1] MatIncreaseOverlap_MPIAIJ_Receive line 427 > src/mat/impls/aij/mpi/mpiov.c > [1]PETSC ERROR: [1] MatIncreaseOverlap_MPIAIJ_Once line 71 > src/mat/impls/aij/mpi/mpiov.c > [1]PETSC ERROR: [1] MatIncreaseOverlap_MPIAIJ line 22 > src/mat/impls/aij/mpi/mpiov.c > [1]PETSC ERROR: [1] MatIncreaseOverlap line 6543 src/mat/interface/matrix.c > [1]PETSC ERROR: [1] PCSetUp_ASM line 155 src/ksp/pc/impls/asm/asm.c > [1]PETSC ERROR: [1] PCSetUp line 797 src/ksp/pc/interface/precon.c > [1]PETSC ERROR: [1] KSPSetUp line 184 src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: [1] KSPSolve line 331 src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: [1] SNES_KSPSolve line 3394 src/snes/interface/snes.c > [1]PETSC ERROR: [1] SNESSolve_LS line 142 src/snes/impls/ls/ls.c > [1]PETSC ERROR: [1] SNESSolve line 2647 src/snes/interface/snes.c > [1]PETSC ERROR: [1] TSStep_Theta line 25 > src/ts/impls/implicit/theta/theta.c > [1]PETSC ERROR: [1] TSStep line 1768 src/ts/interface/ts.c > [1]PETSC ERROR: [1] TSSolve line 1824 src/ts/interface/ts.c > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > The same error happens when I run the example > /snes/examples/tutorials/ex32.c with the option -'pc_type asm'. > > How to properly set up the preconditioner in that case (dof > 1) ? > > Thanks, > > Laurent Berenguer > > PhD student, > Universit? Lyon 1 > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From fpoulin at uwaterloo.ca Mon Mar 12 15:37:30 2012 From: fpoulin at uwaterloo.ca (Francis Poulin) Date: Mon, 12 Mar 2012 16:37:30 -0400 Subject: [petsc-users] calling PETSc from C++ Message-ID: Hello, I am trying to call Petsc from a C++ program and having difficulties. I'm using v3.2.6 and I can run all the examples so I assumed that everything was installed ok. The body of the function is very simple, see below. I get a segmentation fault. In my installation I used mpich I have a colleague who also installed openmp and this works for him on the same version of Petsc. Could it be that Petsc is confused because I have two different MPI's installed? I am hoping that it will use the one that it configured but I do have the other MPI in the same path. Sorry for my confusion but any help would be appreciated. Francis P.S. I have a .txt file that has the output for the seg fault. int main(int argc, char** argv) { // Initialize Petsc PetscInitialize(&argc, &argv, 0, 0); // Finalize Petsc PetscFinalize(); } -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: output.txt URL: From jedbrown at mcs.anl.gov Mon Mar 12 15:41:20 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 12 Mar 2012 15:41:20 -0500 Subject: [petsc-users] calling PETSc from C++ In-Reply-To: References: Message-ID: On Mon, Mar 12, 2012 at 15:37, Francis Poulin wrote: > I am trying to call Petsc from a C++ program and having difficulties. I'm > using v3.2.6 and I can run all the examples so I assumed that everything > was installed ok. The body of the function is very simple, see below. I > get a segmentation fault. In my installation I used mpich > > I have a colleague who also installed openmp and this works for him on the > same version of Petsc. Could it be that Petsc is confused because I have > two different MPI's installed? I am hoping that it will use the one that > it configured but I do have the other MPI in the same path. > Yes, the most likely scenario is that your application was linked with a different MPI than PETSc was compiled with. It is probably a makefile that uses the wrong MPI. Can you build PETSc examples? -------------- next part -------------- An HTML attachment was scrubbed... URL: From fpoulin at uwaterloo.ca Mon Mar 12 16:49:06 2012 From: fpoulin at uwaterloo.ca (Francis Poulin) Date: Mon, 12 Mar 2012 17:49:06 -0400 Subject: [petsc-users] calling PETSc from C++ In-Reply-To: References: Message-ID: <8CD615B9-65F7-49E9-B732-19C927D00D09@uwaterloo.ca> Hello, Yes my examples work fine. I have looked in the makefile for the examples and it looks fairly complicated. One thing I noticed is that I wasn't using the mpicc that is in my petsc directory. I went back and made sure that I am using the mpicc that was build by PETSc. That changes things, but now I cannot compile the code whereas before I could. Do you have a suggestion as to how I can most easily adapt the makefile from the examples into the code we are building? Thanks for the help, Francis On 2012-03-12, at 4:41 PM, Jed Brown wrote: > On Mon, Mar 12, 2012 at 15:37, Francis Poulin wrote: > I am trying to call Petsc from a C++ program and having difficulties. I'm using v3.2.6 and I can run all the examples so I assumed that everything was installed ok. The body of the function is very simple, see below. I get a segmentation fault. In my installation I used mpich > > I have a colleague who also installed openmp and this works for him on the same version of Petsc. Could it be that Petsc is confused because I have two different MPI's installed? I am hoping that it will use the one that it configured but I do have the other MPI in the same path. > > Yes, the most likely scenario is that your application was linked with a different MPI than PETSc was compiled with. It is probably a makefile that uses the wrong MPI. Can you build PETSc examples? -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Mar 12 16:52:05 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 12 Mar 2012 16:52:05 -0500 Subject: [petsc-users] calling PETSc from C++ In-Reply-To: <8CD615B9-65F7-49E9-B732-19C927D00D09@uwaterloo.ca> References: <8CD615B9-65F7-49E9-B732-19C927D00D09@uwaterloo.ca> Message-ID: On Mon, Mar 12, 2012 at 4:49 PM, Francis Poulin wrote: > Hello, > > Yes my examples work fine. I have looked in the makefile for the > examples and it looks fairly complicated. One thing I noticed is that I > wasn't using the mpicc that is in my petsc directory. I went back and > made sure that I am using the mpicc that was build by PETSc. That changes > things, but now I cannot compile the code whereas before I could. > > Do you have a suggestion as to how I can most easily adapt the makefile > from the examples into the code we are building? > The makefiles are simple. Here is building an executable: myProg: myProg.o otherSource.o anotherSource.o ${CLINKER} -o myProg ${PETSC_TS_LIB} Matt > Thanks for the help, > Francis > > On 2012-03-12, at 4:41 PM, Jed Brown wrote: > > On Mon, Mar 12, 2012 at 15:37, Francis Poulin wrote: > >> I am trying to call Petsc from a C++ program and having difficulties. >> I'm using v3.2.6 and I can run all the examples so I assumed that >> everything was installed ok. The body of the function is very simple, see >> below. I get a segmentation fault. In my installation I used mpich >> >> I have a colleague who also installed openmp and this works for him on >> the same version of Petsc. Could it be that Petsc is confused because I >> have two different MPI's installed? I am hoping that it will use the one >> that it configured but I do have the other MPI in the same path. >> > > Yes, the most likely scenario is that your application was linked with a > different MPI than PETSc was compiled with. It is probably a makefile that > uses the wrong MPI. Can you build PETSc examples? > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Mar 12 16:55:06 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 12 Mar 2012 16:55:06 -0500 (CDT) Subject: [petsc-users] calling PETSc from C++ In-Reply-To: References: <8CD615B9-65F7-49E9-B732-19C927D00D09@uwaterloo.ca> Message-ID: On Mon, 12 Mar 2012, Matthew Knepley wrote: > > The makefiles are simple. Here is building an executable: you need some addtional stuf.. include ${PETSC_DIR}/conf/variables include ${PETSC_DIR}/conf/rules > myProg: myProg.o otherSource.o anotherSource.o > ${CLINKER} -o myProg ${PETSC_TS_LIB} attaching a sample makefile. Also its best to build petsc with option --with-clanguage=cxx [if using from c++]. Satish -------------- next part -------------- CFLAGS = FFLAGS = CPPFLAGS = FPPFLAGS = CLEANFILES = include ${PETSC_DIR}/conf/variables include ${PETSC_DIR}/conf/rules ex1: ex1.o chkopts -${CLINKER} -o ex1 ex1.o ${PETSC_LIB} ${RM} ex1.o From j.alyn.roberts at gmail.com Mon Mar 12 17:27:56 2012 From: j.alyn.roberts at gmail.com (Jeremy Roberts) Date: Mon, 12 Mar 2012 18:27:56 -0400 Subject: [petsc-users] MATLAB Classes Message-ID: Hi all, I'm trying access PETSc from MATLAB via the classes. I'm working on a 64 bit Ubuntu 11.04 system with MATLAB R2011b and PETSc 3.2-p6. I get PETSc built, but running e.g. "exKSP.m" leads to a segmentation fault at ksp.Solve(A,b,x). As a second test, I added MatMult to matlabheader.h, and that worked using A,x, and b of the exKSP example. My guess is that something goofy is happening with BLAS/LAPACK. I tried to run "valgrind matlab -nodesktop -nosplash < exKSP.m" without success (maybe there's another way to include valgrind?). I configured via: ./configure \ PETSC_ARCH=gcc-matlab \ --with-debugging \ --with-cc=gcc-4.3 \ --with-fc=gfortran-4.3 \ --with-cxx=g++-4.3 \ --with-shared-libraries \ --with-matlab-engine \ --with-matlab \ --download-f2cblaslapack \ --with-mpi=0 \ --with-x=0 \ --with-x11=0 Any suggestions are greatly appreciated. Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Mar 12 19:16:03 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 12 Mar 2012 19:16:03 -0500 Subject: [petsc-users] MATLAB Classes In-Reply-To: References: Message-ID: <08A4B788-1309-406A-A22F-33BD49CE2EA6@mcs.anl.gov> Please send all the error output from MATLAB when it crashes to petsc-maint at mcs.anl.gov. In particular you may need to run MATLAB from a terminal with matlab -nodesktop to get the traceback as it crashes. Barry On Mar 12, 2012, at 5:27 PM, Jeremy Roberts wrote: > Hi all, > > I'm trying access PETSc from MATLAB via the classes. I'm working on a 64 bit Ubuntu 11.04 system with MATLAB R2011b and PETSc 3.2-p6. > > I get PETSc built, but running e.g. "exKSP.m" leads to a segmentation fault at ksp.Solve(A,b,x). As a second test, I added MatMult to matlabheader.h, and that worked using A,x, and b of the exKSP example. My guess is that something goofy is happening with BLAS/LAPACK. I tried to run "valgrind matlab -nodesktop -nosplash < exKSP.m" without success (maybe there's another way to include valgrind?). > > I configured via: > > ./configure \ > PETSC_ARCH=gcc-matlab \ > --with-debugging \ > --with-cc=gcc-4.3 \ > --with-fc=gfortran-4.3 \ > --with-cxx=g++-4.3 \ > --with-shared-libraries \ > --with-matlab-engine \ > --with-matlab \ > --download-f2cblaslapack \ > --with-mpi=0 \ > --with-x=0 \ > --with-x11=0 > > Any suggestions are greatly appreciated. > > Jeremy > > From fpoulin at uwaterloo.ca Mon Mar 12 19:35:46 2012 From: fpoulin at uwaterloo.ca (Francis Poulin) Date: Mon, 12 Mar 2012 20:35:46 -0400 Subject: [petsc-users] calling PETSc from C++ In-Reply-To: References: <8CD615B9-65F7-49E9-B732-19C927D00D09@uwaterloo.ca> Message-ID: <860678E2-6138-4289-8F8A-A3FFC66B5DE7@uwaterloo.ca> Hello, Thank for the help. When I insert these lines at the end of the mail file and I try make clean I get Francis-Poulins-MacBook-Pro:Ocean3D fpoulin$ make clean /Users/fpoulin/soft/petsc-3.2-p6/conf/rules:137: *** target file `clean' has both : and :: entries. Stop. I see that in the rules file there's a line with clean:: That looks strange but I presume it's suppose to be there. Any ideas how I should do? Thanks, Francis On 2012-03-12, at 5:55 PM, Satish Balay wrote: > On Mon, 12 Mar 2012, Matthew Knepley wrote: > >> >> The makefiles are simple. Here is building an executable: > > you need some addtional stuf.. > > include ${PETSC_DIR}/conf/variables > include ${PETSC_DIR}/conf/rules > >> myProg: myProg.o otherSource.o anotherSource.o >> ${CLINKER} -o myProg ${PETSC_TS_LIB} > > > attaching a sample makefile. Also its best to build petsc with option > --with-clanguage=cxx [if using from c++]. > > Satish > From sean at mcs.anl.gov Mon Mar 12 19:39:23 2012 From: sean at mcs.anl.gov (Sean Farley) Date: Mon, 12 Mar 2012 19:39:23 -0500 Subject: [petsc-users] calling PETSc from C++ In-Reply-To: <860678E2-6138-4289-8F8A-A3FFC66B5DE7@uwaterloo.ca> References: <8CD615B9-65F7-49E9-B732-19C927D00D09@uwaterloo.ca> <860678E2-6138-4289-8F8A-A3FFC66B5DE7@uwaterloo.ca> Message-ID: > > Francis-Poulins-MacBook-Pro:Ocean3D fpoulin$ make clean > /Users/fpoulin/soft/petsc-3.2-p6/conf/rules:137: *** target file `clean' > has both : and :: entries. Stop. > > I see that in the rules file there's a line with > > clean:: > > That looks strange but I presume it's suppose to be there. > > Any ideas how I should do? Add a colon to the clean rule that you have defined in your own makefiles -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengshixiangze at 163.com Tue Mar 13 03:37:59 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Tue, 13 Mar 2012 16:37:59 +0800 (CST) Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> Message-ID: Hi, Matt. After I configure PETSc using --with-precision=single, I can run both ex19 and my own code. Good news! But it seems lots of time is using for copping the data from CPU to GPU. In all, thank you for your generously help! Zeng ? 2012-03-06 13:31:06?"Matthew Knepley" ??? 2012/3/5 Xiangze Zeng Hi, Matt. We know that when we compile the CUDA-C code, we use nvcc. But if we run the PETSc-CUDA code in GPU, what we do is just to change the type of Mat and Vec. In the PETSc-CUDA code compilings progress ,does it use nvcc? Yes. And when implement the PETSc for using GPU, do you ever consider the comupute capibility of the GPU. And in which place, the parameter of the comupute capibility of the GPU has been set? You configure using --with-cuda-arch=sm_13 for instance, or compute_12. And you have given me an idea what your problem is. The GeForce 310 has no double precision floating point units. Therefore, you must configure PETSc using --with-precision=single. Matt Thank you. Zeng ? 2012-03-05 22:28:20?"Matthew Knepley" ??? On Mon, Mar 5, 2012 at 8:18 AM, Xiangze Zeng wrote: Hi, Matt. I have tried the method you told me( I closed all the X systems),but the same error still occurs. Is there something wrong with the installation of the thrust? And I find the same problem in several web pages, like this http://code.google.com/p/thrust/wiki/Debugging, they all say the error is related to the compute capability . Is that the case? My GPU is GeForce 310, which supports compute capability 1.2 . One of the downsides of CUDA, at least from our perspective, is that these errors are very hard to debug. If thrust compiles and the cudaInit() succeeds, there is not much else we can do to debug on this machine. Maybe you can try running thrust examples? Matt Thank you. ? 2012-03-02 23:55:13?"Matthew Knepley" ??? On Fri, Mar 2, 2012 at 9:40 AM, Xiangze Zeng wrote: Thank both of you, Satish and Jed. This warning message has disappeared. But another problem occurs, it terminates with the same message as I run my own code:" terminate called after throwing an instance of 'thrust::system::system_error' what(): invalid device function Aborted." (Several days ago, I thought I run the ex19 successfully, but today I found I was totally wrong when I find the warning message in the PETSc Performance Summary. So sorry about that!) Most likely, this a problem with another program locking your GPU (I assume you are running on your laptop/desktop). Start closing apps one at a time (especially graphical ones like a PDF reader) and try running each time. Matt May this new problem be related to the installation of the cuda( I can run cuda-C code successfully in my PC)? Thank you again! Zeng Xiangze At 2012-03-02 23:18:19,"Satish Balay" wrote: >On Fri, 2 Mar 2012, Xiangze Zeng wrote: > >> Dear all, >> >> After I have installed the PETSc to use the NVidia GPUS, I run the ex19. When I make ex19, it stopped with the message: >> >> >> "makefile:24: /conf/variables: No such file or directory >> makefile:25: /conf/rules: No such file or directory >> makefile:1023: /conf/test: No such file or directory >> make: *** No rule to make target `/conf/test'. Stop." >> >> >> Then I add the PETSC_DIR to the makefile. After that, I made ex19 successfully. But when I run it using the command : > >Yes - you need PETSC_DIR/PETSC_ARCH values to be specified to >make. You can set these as env variables - or specify to make on the >command line - or add to makefile. [we recommended not modifying the >makefile - to keep it portable..] > >> >> >> "./ex19 -da_vec_type mpicusp -da_mat_type mpiaijcusp -pc_type none -dmmg_nlevels 1 -da_grid_x 100 -da_grid_y 100 -log_summary -mat_no_inode -preload off -cusp_synchronize". >> >> >> In the PETSc Performance Summary, it says:" >> WARNING! There are options you set that were not used! >> WARNING! could be spelling mistake, etc! >> Option left: name:-da_mat_type value: mpiaijcusp >> Option left: name:-da_vec_type value: mpicusp" > >Looks like the options are now changed to -dm_mat_type and -dm_vec_type. > >Satish > >> >> >> Is there any problem with it? >> Any response will be appreciated! Thank you so much! >> >> >> Zeng Xiangze >> >> >> >> >> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Mar 13 07:12:09 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 13 Mar 2012 07:12:09 -0500 Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> Message-ID: 2012/3/13 Xiangze Zeng > After I configure PETSc using --with-precision=single, I can run both > ex19 and my own code. Good news! But it seems lots of time is using for > copping the data from CPU to GPU. > How are you measuring? What preconditioner are you using and how many iterations are typically required? -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Tue Mar 13 07:32:43 2012 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Tue, 13 Mar 2012 12:32:43 +0000 Subject: [petsc-users] using PCFieldSplitGetSubKSP in c++ Message-ID: > > >Declare this as KSP *subksp; > > > > Compare this ^^^^^^^^^^^ > > > > > > > > Matt > > > > > > > Thanks Matt, but when I do that I cannot use KSPSetNullSpace: > > > > KSP *subksp[2]; > > > > To this ^^^^^^^^^^ Matt, Jed Thanks, I get the idea now and changed the code to: KSP *subksp; PetscInt n=2; ierr = PCFieldSplitGetSubKSP(pc,&n,&subksp); CHKERRQ(ierr); which removes the compiler error but gives the following runtime error in MatSchurComplementGetKSP(): [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Null argument, when expecting valid pointer! [0]PETSC ERROR: Null Object: Parameter # 1! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: 3e5cbcd09c0cdca7d193329b251809bec1adb9e3 HG Date: Tue Mar 06 22:49:22 2012 -0600 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./stokes on a linux_64b named lin0133 by cklaij Tue Mar 13 11:55:39 2012 [0]PETSC ERROR: Libraries linked from /home/CKlaij/ReFRESCO/Libraries/build/petsc-dev/linux_64bit_debug/lib [0]PETSC ERROR: Configure run at Wed Mar 7 09:40:17 2012 [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/refresco/64bit_intelv11.1_openmpi/openmpi-1.4.5 --with-clanguage=c++ --with-x=1 --with-debugging=1 --with-hypre-include=/opt/refresco/64bit_intelv11.1_openmpi/hypre-2.7.0b/include --with-hypre-lib=/opt/refresco/64bit_intelv11.1_openmpi/hypre-2.7.0b/lib/libHYPRE.a --with-ml-include=/opt/refresco/64bit_intelv11.1_openmpi/ml-6.2/include --with-ml-lib=/opt/refresco/64bit_intelv11.1_openmpi/ml-6.2/lib/libml.a --with-blas-lapack-dir=/opt/intel/mkl [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatSchurComplementGetKSP() line 218 in /home/CKlaij/ReFRESCO/Libraries/build/petsc-dev/src/ksp/ksp/utils/schurm.c [0]PETSC ERROR: PCFieldSplitGetSubKSP_FieldSplit_Schur() line 957 in /home/CKlaij/ReFRESCO/Libraries/build/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: PCFieldSplitGetSubKSP() line 1221 in /home/CKlaij/ReFRESCO/Libraries/build/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: solve() line 101 in stokes.cc [0]PETSC ERROR: main() line 581 in stokes.cc dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From jedbrown at mcs.anl.gov Tue Mar 13 07:47:55 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 13 Mar 2012 07:47:55 -0500 Subject: [petsc-users] Bug in pc_right + nonzero initial guess + {tfqmr, tcqmr, cgs} In-Reply-To: <20120311063856.GA12186@ices.utexas.edu> References: <20120311063856.GA12186@ices.utexas.edu> Message-ID: On Sun, Mar 11, 2012 at 00:38, Tobin Isaac wrote: > > Hi, > > The changes made to bcgs in the commit below should be applied to > tfqmr, tcqmr, and cgs as well. > Toby, thanks for hunting this down. Barry, do you think we can reasonably share code between these implementations (and any others that might need it)? > > Cheers, > Toby > > author Barry Smith bsmith at mcs.anl.gov > Wed, 19 Jan 2011 20:10:29 -0600 (13 months ago) > changeset 18159 a526acdfdc95 > parent 18153 3c53022524fb > child 18160 4db6569c26af > > fixed KSPBCGS to work with right preconditioning and nonzero initial > guess > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > > iEYEARECAAYFAk9cSIAACgkQk/TrNolnueUqMQCcD9gSdldhOY14sIFpwEC4nhpD > pTEAniP556Lh3+5MgYB9cTUzHj7I4jg5 > =FvoR > -----END PGP SIGNATURE----- > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Mar 13 07:56:26 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 13 Mar 2012 07:56:26 -0500 Subject: [petsc-users] using PCFieldSplitGetSubKSP in c++ In-Reply-To: References: Message-ID: On Tue, Mar 13, 2012 at 07:32, Klaij, Christiaan wrote: > Thanks, I get the idea now and changed the code to: > > KSP *subksp; > PetscInt n=2; > ierr = PCFieldSplitGetSubKSP(pc,&n,&subksp); CHKERRQ(ierr); > > which removes the compiler error but gives the following runtime > error in MatSchurComplementGetKSP(): > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Null argument, when expecting valid pointer! > [0]PETSC ERROR: Null Object: Parameter # 1! > You have to have set the matrix and called PCSetUp() for this inner context to have been created. This is an unfortunate property of nested methods (with every programming system I know of), it's difficult to bootstrap so that a certain inner object is configured without setting up more than is strictly necessary. If it's just the constant null space, then you can do it on the command line. Note that with petsc-dev, you can use MatSetNullSpace() and it will be used by the KSP, thus avoiding the need to drill down into the KSP like this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengshixiangze at 163.com Tue Mar 13 08:59:34 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Tue, 13 Mar 2012 21:59:34 +0800 (CST) Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> Message-ID: <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> Hi, Jed. At the beginning and end of the codes for setting the matrices values, I add "printf", and compute the time of this period. It is much longer than that when I don't use the GPU. I just guess the time is used for copping data. My PCTYPE is sor. And 2000 iterations. Do you have any suggestion about this? Zeng ? 2012-03-13 20:12:09?"Jed Brown" ??? 2012/3/13 Xiangze Zeng After I configure PETSc using --with-precision=single, I can run both ex19 and my own code. Good news! But it seems lots of time is using for copping the data from CPU to GPU. How are you measuring? What preconditioner are you using and how many iterations are typically required? -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Mar 13 09:03:12 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 13 Mar 2012 09:03:12 -0500 Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> Message-ID: On Tue, Mar 13, 2012 at 8:59 AM, Xiangze Zeng wrote: > Hi, Jed. > At the beginning and end of the codes for setting the matrices values, I > add "printf", and compute the time of this period. It is much longer than > that when I don't use the GPU. I just guess the time is used for copping > data. My PCTYPE is sor. And 2000 iterations. Do you have any suggestion > about this? > 1) You do not have to guess. Use -log_summary, and there are explicit events for copying to the GPU 2) GPUs only really become effective for large systems due to this overhead. I suggest looking at the performance and overhead as a function of system size. Matt > Zeng > > ? 2012-03-13 20:12:09?"Jed Brown" ??? > > 2012/3/13 Xiangze Zeng > >> After I configure PETSc using --with-precision=single, I can run both >> ex19 and my own code. Good news! But it seems lots of time is using for >> copping the data from CPU to GPU. >> > > How are you measuring? What preconditioner are you using and how many > iterations are typically required? > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Mar 13 09:14:12 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 13 Mar 2012 09:14:12 -0500 Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> Message-ID: 2012/3/13 Xiangze Zeng > At the beginning and end of the codes for setting the matrices values, I > add "printf", and compute the time of this period. It is much longer than > that when I don't use the GPU. I just guess the time is used for copping > data. My PCTYPE is sor. And 2000 iterations. Do you have any suggestion > about this? I suspect you have lost preallocation information when the matrix type was changed. Make sure you set preallocation after setting the matrix type. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengshixiangze at 163.com Tue Mar 13 09:25:27 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Tue, 13 Mar 2012 22:25:27 +0800 (CST) Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> Message-ID: <53eb88fc.10e14.1360c730adb.Coremail.zengshixiangze@163.com> Hi, Matt. I get it. I just test a so little system. I'll try a larger one. Zeng ? 2012-03-13 22:03:12?"Matthew Knepley" ??? On Tue, Mar 13, 2012 at 8:59 AM, Xiangze Zeng wrote: Hi, Jed. At the beginning and end of the codes for setting the matrices values, I add "printf", and compute the time of this period. It is much longer than that when I don't use the GPU. I just guess the time is used for copping data. My PCTYPE is sor. And 2000 iterations. Do you have any suggestion about this? 1) You do not have to guess. Use -log_summary, and there are explicit events for copying to the GPU 2) GPUs only really become effective for large systems due to this overhead. I suggest looking at the performance and overhead as a function of system size. Matt Zeng ? 2012-03-13 20:12:09?"Jed Brown" ??? 2012/3/13 Xiangze Zeng After I configure PETSc using --with-precision=single, I can run both ex19 and my own code. Good news! But it seems lots of time is using for copping the data from CPU to GPU. How are you measuring? What preconditioner are you using and how many iterations are typically required? -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengshixiangze at 163.com Tue Mar 13 09:30:15 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Tue, 13 Mar 2012 22:30:15 +0800 (CST) Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> Message-ID: <97d3e2d.10edc.1360c777086.Coremail.zengshixiangze@163.com> Hi, Jed. I do set preallocation after setting the matrix type. Zeng ? 2012-03-13 22:14:12?"Jed Brown" ??? 2012/3/13 Xiangze Zeng At the beginning and end of the codes for setting the matrices values, I add "printf", and compute the time of this period. It is much longer than that when I don't use the GPU. I just guess the time is used for copping data. My PCTYPE is sor. And 2000 iterations. Do you have any suggestion about this? I suspect you have lost preallocation information when the matrix type was changed. Make sure you set preallocation after setting the matrix type. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Mar 13 09:36:46 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 13 Mar 2012 09:36:46 -0500 Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: <97d3e2d.10edc.1360c777086.Coremail.zengshixiangze@163.com> References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> <97d3e2d.10edc.1360c777086.Coremail.zengshixiangze@163.com> Message-ID: 2012/3/13 Xiangze Zeng > I do set preallocation after setting the matrix type. > Run with -info to make sure it is used (i.e. that the matrix type isn't changed later). Note that PCSOR does not run on the GPU, so it will do lots of copying (run with -log_summary to see). You should start by running PCJACOBI on the GPU. > > Zeng > > ? 2012-03-13 22:14:12?"Jed Brown" ??? > > 2012/3/13 Xiangze Zeng > >> At the beginning and end of the codes for setting the matrices values, I >> add "printf", and compute the time of this period. It is much longer than >> that when I don't use the GPU. I just guess the time is used for copping >> data. My PCTYPE is sor. And 2000 iterations. Do you have any suggestion >> about this? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengshixiangze at 163.com Tue Mar 13 09:54:59 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Tue, 13 Mar 2012 22:54:59 +0800 (CST) Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> <97d3e2d.10edc.1360c777086.Coremail.zengshixiangze@163.com> Message-ID: <6f6af5bc.16f7e.1360c8e139b.Coremail.zengshixiangze@163.com> Thank you. I'll try it. Zeng ? 2012-03-13 22:36:46?"Jed Brown" ??? 2012/3/13 Xiangze Zeng I do set preallocation after setting the matrix type. Run with -info to make sure it is used (i.e. that the matrix type isn't changed later). Note that PCSOR does not run on the GPU, so it will do lots of copying (run with -log_summary to see). You should start by running PCJACOBI on the GPU. Zeng ? 2012-03-13 22:14:12?"Jed Brown" ??? 2012/3/13 Xiangze Zeng At the beginning and end of the codes for setting the matrices values, I add "printf", and compute the time of this period. It is much longer than that when I don't use the GPU. I just guess the time is used for copping data. My PCTYPE is sor. And 2000 iterations. Do you have any suggestion about this? -------------- next part -------------- An HTML attachment was scrubbed... URL: From paulm at txcorp.com Tue Mar 13 10:18:35 2012 From: paulm at txcorp.com (Paul Mullowney) Date: Tue, 13 Mar 2012 09:18:35 -0600 Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> <97d3e2d.10edc.1360c777086.Coremail.zengshixiangze@163.com> Message-ID: <4F5F654B.6040902@txcorp.com> I don't think PCJACOBI is working on the GPU yet. I have some local fixes to make it work efficiently (without extra copies), but changes won't be pushed too soon because the design is not sufficiently general. -Paul > 2012/3/13 Xiangze Zeng > > > I do set preallocation after setting the matrix type. > > > Run with -info to make sure it is used (i.e. that the matrix type > isn't changed later). Note that PCSOR does not run on the GPU, so it > will do lots of copying (run with -log_summary to see). You should > start by running PCJACOBI on the GPU. > > > Zeng > > ? 2012-03-13 22:14:12?"Jed Brown" > ??? > > 2012/3/13 Xiangze Zeng > > > At the beginning and end of the codes for setting the > matrices values, I add "printf", and compute the time of > this period. It is much longer than that when I don't use > the GPU. I just guess the time is used for copping > data. My PCTYPE is sor. And 2000 iterations. Do you have > any suggestion about this? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Mar 13 10:27:10 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 13 Mar 2012 10:27:10 -0500 Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: <4F5F654B.6040902@txcorp.com> References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> <97d3e2d.10edc.1360c777086.Coremail.zengshixiangze@163.com> <4F5F654B.6040902@txcorp.com> Message-ID: On Tue, Mar 13, 2012 at 10:18, Paul Mullowney wrote: > I don't think PCJACOBI is working on the GPU yet. > It damn well better be. (It only calls MatGetDiagonal(), everything else is just a Vec operation and the VecType will be set so that it runs on the GPU.) > > I have some local fixes to make it work efficiently (without extra > copies), but changes won't be pushed too soon because the design is not > sufficiently general. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paulm at txcorp.com Tue Mar 13 10:38:08 2012 From: paulm at txcorp.com (Paul Mullowney) Date: Tue, 13 Mar 2012 09:38:08 -0600 Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> <97d3e2d.10edc.1360c777086.Coremail.zengshixiangze@163.com> <4F5F654B.6040902@txcorp.com> Message-ID: <4F5F69E0.9030905@txcorp.com> In bjacobi.c, the following comments are given for PCCreate_BJacobi. Developer Notes: This preconditioner does not currently work with CUDA/CUSP for a couple of reasons. (1) It creates seq vectors as work vectors that should be cusp (2) The use of VecPlaceArray() is not handled properly by CUSP (that is it will not know where the ownership of the vector is so may use wrong values) even if it did know the ownership it may induce extra copy ups and downs. Satish suggests a VecTransplantArray() to handle two vectors sharing the same pointer and handling the CUSP side as well instead of VecGetArray()/VecPlaceArray(). I've had some success with comment (2) above in regards to Singleblock BJacobi. I haven't yet tried it for Multiblock or Multiproc. -Paul > On Tue, Mar 13, 2012 at 10:18, Paul Mullowney > wrote: > > I don't think PCJACOBI is working on the GPU yet. > > > It damn well better be. > > (It only calls MatGetDiagonal(), everything else is just a Vec > operation and the VecType will be set so that it runs on the GPU.) > > > I have some local fixes to make it work efficiently (without extra > copies), but changes won't be pushed too soon because the design > is not sufficiently general. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paulm at txcorp.com Tue Mar 13 10:44:02 2012 From: paulm at txcorp.com (Paul Mullowney) Date: Tue, 13 Mar 2012 09:44:02 -0600 Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: <4F5F69E0.9030905@txcorp.com> References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> <97d3e2d.10edc.1360c777086.Coremail.zengshixiangze@163.com> <4F5F654B.6040902@txcorp.com> <4F5F69E0.9030905@txcorp.! com> Message-ID: <4F5F6B42.2030107@txcorp.com> Sorry. You are asking for Jacobi, and not BJacobi. D'oh! -Paul > In bjacobi.c, the following comments are given for PCCreate_BJacobi. > > Developer Notes: This preconditioner does not currently work with > CUDA/CUSP for a couple of reasons. > (1) It creates seq vectors as work vectors that should be cusp > (2) The use of VecPlaceArray() is not handled properly by CUSP (that > is it will not know where the ownership of the vector is so may use > wrong values) even if it did know the ownership it may induce extra > copy ups and downs. Satish suggests a VecTransplantArray() to handle > two vectors sharing the same pointer and handling the CUSP side as > well instead of VecGetArray()/VecPlaceArray(). > > > I've had some success with comment (2) above in regards to Singleblock > BJacobi. I haven't yet tried it for Multiblock or Multiproc. > > -Paul > >> On Tue, Mar 13, 2012 at 10:18, Paul Mullowney > > wrote: >> >> I don't think PCJACOBI is working on the GPU yet. >> >> >> It damn well better be. >> >> (It only calls MatGetDiagonal(), everything else is just a Vec >> operation and the VecType will be set so that it runs on the GPU.) >> >> >> I have some local fixes to make it work efficiently (without >> extra copies), but changes won't be pushed too soon because the >> design is not sufficiently general. >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Mar 13 10:44:41 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 13 Mar 2012 10:44:41 -0500 Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: <4F5F69E0.9030905@txcorp.com> References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> <97d3e2d.10edc.1360c777086.Coremail.zengshixiangze@163.com> <4F5F654B.6040902@txcorp.com> <4F5F69E0.9030905@txcorp.com> Message-ID: On Tue, Mar 13, 2012 at 10:38, Paul Mullowney wrote: > In bjacobi.c, the following comments are given for PCCreate_BJacobi. > These things should definitely be fixed, but I said "jacobi" (without the "b"). > > Developer Notes: This preconditioner does not currently work with > CUDA/CUSP for a couple of > reasons. > > (1) It creates seq vectors as work vectors that should be > cusp > > (2) The use of VecPlaceArray() is not handled properly by CUSP (that is it > will not know where the ownership of the vector is so may use wrong values) > even if it did know the ownership it may induce extra copy ups and downs. > Satish suggests a VecTransplantArray() to handle two vectors sharing the > same pointer and handling the CUSP side as well instead of > VecGetArray()/VecPlaceArray(). > > > I've had some success with comment (2) above in regards to Singleblock > BJacobi. I haven't yet tried it for Multiblock or Multiproc. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Tue Mar 13 15:22:30 2012 From: dave.mayhem23 at gmail.com (Dave May) Date: Tue, 13 Mar 2012 21:22:30 +0100 Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> Message-ID: Hey Matt, Do you have any guidance or ideas regarding how large the subdomains should be to offset the cost of this copy? Cheers, Dave On 13 March 2012 15:03, Matthew Knepley wrote: > On Tue, Mar 13, 2012 at 8:59 AM, Xiangze Zeng > wrote: >> >> Hi, Jed. >> At the beginning and end of ?the codes for setting the matrices values, I >> add "printf", and compute the time of this period. It is much longer than >> that when I don't use the GPU. I just guess the time is used for copping >> data.?My PCTYPE is sor. And 2000 iterations. ?Do you have any suggestion >> about this? > > > 1) You do not have to guess. Use -log_summary, and there are explicit events > for copying to the GPU > > 2) GPUs only really become effective for large systems due to this overhead. > I suggest looking at the > ? ? performance and overhead as a function of system size. > > ? ?Matt > >> >> Zeng >> >> ? 2012-03-13 20:12:09?"Jed?Brown"? ??? >> >> 2012/3/13 Xiangze Zeng >>> >>> After I??configure PETSc using --with-precision=single, I can run both >>> ex19 and my own code. Good news! But it seems lots of time is using for >>> copping the data from CPU to GPU. >> >> >> How are you measuring? What preconditioner are you using and how many >> iterations are typically required? >> >> >> > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From knepley at gmail.com Tue Mar 13 16:16:51 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 13 Mar 2012 16:16:51 -0500 Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> Message-ID: On Tue, Mar 13, 2012 at 3:22 PM, Dave May wrote: > Hey Matt, > Do you have any guidance or ideas regarding how large the subdomains > should be to offset the cost of this copy? > They have to be substantial, but it depends on your arithmetic intensity. What I can say for sure is that we maxed out the memory on the machines we have (like my laptop) and saw 3-5x speed up with SpMV. Also, I saw a TON of overhead on my FEM benchmark, even though it is screaming on the GPU, but I now think that is cudaMalloc()/Free() rather than all communication. Matt > Cheers, > Dave > > > On 13 March 2012 15:03, Matthew Knepley wrote: > > On Tue, Mar 13, 2012 at 8:59 AM, Xiangze Zeng > > wrote: > >> > >> Hi, Jed. > >> At the beginning and end of the codes for setting the matrices values, > I > >> add "printf", and compute the time of this period. It is much longer > than > >> that when I don't use the GPU. I just guess the time is used for copping > >> data. My PCTYPE is sor. And 2000 iterations. Do you have any suggestion > >> about this? > > > > > > 1) You do not have to guess. Use -log_summary, and there are explicit > events > > for copying to the GPU > > > > 2) GPUs only really become effective for large systems due to this > overhead. > > I suggest looking at the > > performance and overhead as a function of system size. > > > > Matt > > > >> > >> Zeng > >> > >> ? 2012-03-13 20:12:09?"Jed Brown" ??? > >> > >> 2012/3/13 Xiangze Zeng > >>> > >>> After I configure PETSc using --with-precision=single, I can run both > >>> ex19 and my own code. Good news! But it seems lots of time is using for > >>> copping the data from CPU to GPU. > >> > >> > >> How are you measuring? What preconditioner are you using and how many > >> iterations are typically required? > >> > >> > >> > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments > > is infinitely more interesting than any results to which their > experiments > > lead. > > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From xdliang at gmail.com Tue Mar 13 16:39:33 2012 From: xdliang at gmail.com (Xiangdong Liang) Date: Tue, 13 Mar 2012 17:39:33 -0400 Subject: [petsc-users] using OpenMP in PETSc Message-ID: Hello everyone, Can someone provide me advice on using OpenMP in PETSc? I am solving a problem like this: int main() { vec vsum; for(i=0; i< N; i++) { vec vi; fcomputev(i, vi); VecAXPY(vsum, 1.0, vi); // vsum +=vi; } } Can I use OpenMP "omp parallel for" to do the loop in parallel? For example, suppose I have 8 processes. It would be nice if each petsc subrountine fcomputev uses 2 processes while 4 different i's are computed in parallel (since different i's are independent). Any helps or hints on this would be appreciated. Best, Xiangdong From fpoulin at uwaterloo.ca Tue Mar 13 16:45:43 2012 From: fpoulin at uwaterloo.ca (Francis Poulin) Date: Tue, 13 Mar 2012 17:45:43 -0400 Subject: [petsc-users] system upgrade Message-ID: <2574D312-D95C-4FB8-AEF8-8EB9C2AF86C5@uwaterloo.ca> Hello, I was using petsc on my Mac OS x 10.6 perfectly fine and then decided (perhaps foolishly) to upgrade to lion, 10.7. When I type the same configure line as before, ./configure --with-cc=/usr/local/bin/gcc --with-fc=/usr/local/bin/gfortran --with-clanguage=cxx --download-f-blas-lapack=1 --download-mpich=1 Any ideas what I need to do to configure it like I had before (that doesn't involve going back to 10.6)? Thanks, Francis P.S. I tried before with both log files but the first was too big so i'm only including the one that failed. I presume that's the more interesting one anyhow. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: configure.txt URL: From knepley at gmail.com Tue Mar 13 16:57:05 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 13 Mar 2012 16:57:05 -0500 Subject: [petsc-users] using OpenMP in PETSc In-Reply-To: References: Message-ID: On Tue, Mar 13, 2012 at 4:39 PM, Xiangdong Liang wrote: > Hello everyone, > > Can someone provide me advice on using OpenMP in PETSc? I am solving a > problem like this: > > int main() > { > vec vsum; > > for(i=0; i< N; i++) > { > vec vi; > fcomputev(i, vi); > VecAXPY(vsum, 1.0, vi); // vsum +=vi; > } > > } > > Can I use OpenMP "omp parallel for" to do the loop in parallel? For > example, suppose I have 8 processes. It would be nice if each petsc > subrountine fcomputev uses 2 processes while 4 different i's are > computed in parallel (since different i's are independent). > PETSc is not threadsafe. This is trivial to do in MPI where the comm for eah Vec is a group of 2 procs. Matt > Any helps or hints on this would be appreciated. > > Best, > Xiangdong > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Mar 13 16:59:55 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 13 Mar 2012 16:59:55 -0500 Subject: [petsc-users] system upgrade In-Reply-To: <2574D312-D95C-4FB8-AEF8-8EB9C2AF86C5@uwaterloo.ca> References: <2574D312-D95C-4FB8-AEF8-8EB9C2AF86C5@uwaterloo.ca> Message-ID: On Tue, Mar 13, 2012 at 4:45 PM, Francis Poulin wrote: > Hello, > > I was using petsc on my Mac OS x 10.6 perfectly fine and then decided > (perhaps foolishly) to upgrade to lion, 10.7. When I type the same > configure line as before, > > ./configure --with-cc=/usr/local/bin/gcc --with-fc=/usr/local/bin/gfortran > --with-clanguage=cxx --download-f-blas-lapack=1 --download-mpich=1 > > Any ideas what I need to do to configure it like I had before (that > doesn't involve going back to 10.6)? > I am guessing you have not installed XCode. It cannot find 'make', just like the error message says. Matt > Thanks, > Francis > > P.S. I tried before with both log files but the first was too big so i'm > only including the one that failed. I presume that's the more interesting > one anyhow. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean at mcs.anl.gov Tue Mar 13 17:00:04 2012 From: sean at mcs.anl.gov (Sean Farley) Date: Tue, 13 Mar 2012 17:00:04 -0500 Subject: [petsc-users] system upgrade In-Reply-To: <2574D312-D95C-4FB8-AEF8-8EB9C2AF86C5@uwaterloo.ca> References: <2574D312-D95C-4FB8-AEF8-8EB9C2AF86C5@uwaterloo.ca> Message-ID: > > I was using petsc on my Mac OS x 10.6 perfectly fine and then decided > (perhaps foolishly) to upgrade to lion, 10.7. When I type the same > configure line as before, > > ./configure --with-cc=/usr/local/bin/gcc --with-fc=/usr/local/bin/gfortran > --with-clanguage=cxx --download-f-blas-lapack=1 --download-mpich=1 > > Any ideas what I need to do to configure it like I had before (that > doesn't involve going back to 10.6)? > Depending on how you upgraded (or perhaps did a clean install), you'll need to install Xcode 4.3.1 and make sure that you check (it's not checked by default) to install command line tools. This is done from the 'Downloads' section of Xcode's preferences. P.S. I tried before with both log files but the first was too big so i'm > only including the one that failed. I presume that's the more interesting > one anyhow. You could send to petsc-maint instead (and / or try zipping) -------------- next part -------------- An HTML attachment was scrubbed... URL: From fpoulin at uwaterloo.ca Tue Mar 13 18:47:57 2012 From: fpoulin at uwaterloo.ca (Francis Poulin) Date: Tue, 13 Mar 2012 19:47:57 -0400 Subject: [petsc-users] system upgrade In-Reply-To: References: <2574D312-D95C-4FB8-AEF8-8EB9C2AF86C5@uwaterloo.ca> Message-ID: <145E3965-C38C-41D5-B4B8-DA7944ABB849@uwaterloo.ca> Hello, Sorry for not thinking of this myself. Thanks for pointing out that it was Xcode and also telling me about the command line tools. That saved me from sending another email. It seems to be installing very nicely right now. Cheers, Francis On 2012-03-13, at 6:00 PM, Sean Farley wrote: > I was using petsc on my Mac OS x 10.6 perfectly fine and then decided (perhaps foolishly) to upgrade to lion, 10.7. When I type the same configure line as before, > > ./configure --with-cc=/usr/local/bin/gcc --with-fc=/usr/local/bin/gfortran --with-clanguage=cxx --download-f-blas-lapack=1 --download-mpich=1 > > Any ideas what I need to do to configure it like I had before (that doesn't involve going back to 10.6)? > > Depending on how you upgraded (or perhaps did a clean install), you'll need to install Xcode 4.3.1 and make sure that you check (it's not checked by default) to install command line tools. This is done from the 'Downloads' section of Xcode's preferences. > > P.S. I tried before with both log files but the first was too big so i'm only including the one that failed. I presume that's the more interesting one anyhow. > > You could send to petsc-maint instead (and / or try zipping) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Daniel.Weflen at Colorado.EDU Tue Mar 13 18:19:36 2012 From: Daniel.Weflen at Colorado.EDU (Daniel Christian Weflen) Date: Tue, 13 Mar 2012 17:19:36 -0600 Subject: [petsc-users] PetscBag Question Message-ID: <4F5FD608.4050900@colorado.edu> I have a question about PetscBagCreate (the documentation for which can be found here [1]). The documentation says that the size of the struct cannot be larger than a PetscInt, which is 4 bytes. Is this referring to the entire struct, or just a pointer to it? If the size of the struct itself cannot be larger than one PetscInt, what exactly is are PetscBags supposed to do? -Dan [1] http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscBagCreate.html#PetscBagCreate From knepley at gmail.com Tue Mar 13 19:18:00 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 13 Mar 2012 19:18:00 -0500 Subject: [petsc-users] PetscBag Question In-Reply-To: <4F5FD608.4050900@colorado.edu> References: <4F5FD608.4050900@colorado.edu> Message-ID: On Tue, Mar 13, 2012 at 6:19 PM, Daniel Christian Weflen < Daniel.Weflen at colorado.edu> wrote: > I have a question about PetscBagCreate (the documentation for which can be > found here [1]). > > The documentation says that the size of the struct cannot be larger than a > PetscInt, which is 4 bytes. Is this referring to the entire struct, or just > a pointer to it? > > If the size of the struct itself cannot be larger than one PetscInt, what > exactly is are PetscBags supposed to do? > What this means is that you cannot have a struct bigger than 2G, since we use an integer to hold the size in bytes. Matt > -Dan > > > > > > [1] http://www.mcs.anl.gov/petsc/**petsc-current/docs/**manualpages/Sys/** > PetscBagCreate.html#**PetscBagCreate > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Mar 13 19:26:13 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 13 Mar 2012 19:26:13 -0500 Subject: [petsc-users] PetscBag Question In-Reply-To: References: <4F5FD608.4050900@colorado.edu> Message-ID: On Tue, Mar 13, 2012 at 19:18, Matthew Knepley wrote: > What this means is that you cannot have a struct bigger than 2G, since we > use an integer to hold the size in bytes. Unless you configure --with-64-bit-indices. In either case, it would be an egregious abuse of the PetscBag to exceed this size. -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Wed Mar 14 02:53:16 2012 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Wed, 14 Mar 2012 07:53:16 +0000 Subject: [petsc-users] using PCFieldSplitGetSubKSP in c++ Message-ID: > > Thanks, I get the idea now and changed the code to: > > > > KSP *subksp; > > PetscInt n=2; > > ierr = PCFieldSplitGetSubKSP(pc,&n,&subksp); CHKERRQ(ierr); > > > > which removes the compiler error but gives the following runtime > > error in MatSchurComplementGetKSP(): > > > > [0]PETSC ERROR: --------------------- Error Message > > ------------------------------------ > > [0]PETSC ERROR: Null argument, when expecting valid pointer! > > [0]PETSC ERROR: Null Object: Parameter # 1! > > > > You have to have set the matrix and called PCSetUp() for this inner context > to have been created. This is an unfortunate property of nested methods > (with every programming system I know of), it's difficult to bootstrap so > that a certain inner object is configured without setting up more than is > strictly necessary. If it's just the constant null space, then you can do > it on the command line. Note that with petsc-dev, you can use > MatSetNullSpace() and it will be used by the KSP, thus avoiding the need to > drill down into the KSP like this. Thanks, no more problems after PCSetUP(). I am using petsc-dev, so are you saying that MatSetNullSpace() on the block matrix [A00 A01, A10 A11] will be enough? Can/will the null space of the Schur complement be deduced from this? Otherwise one would still need to "drill down" to get Schur complement matrix, right? dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From jarunan at ascomp.ch Wed Mar 14 04:00:11 2012 From: jarunan at ascomp.ch (Jarunan Panyasantisuk) Date: Wed, 14 Mar 2012 10:00:11 +0100 Subject: [petsc-users] AMG options Message-ID: <4F605E1B.50108@ascomp.ch> Dear PETSc team, I have a problem setting options for Hypre/BoomerAMG. I am trying to set these options in the code: 1 call PetscOptionsSetValue('-pc_hypre_boomeramg_max_iter','0',ierr) 2 call PetscOptionsSetValue('-pc_hypre_boomeramg_coarsen_type','HMIS',ierr) 3 call PetscOptionsSetValue('-pc_hypre_boomeramg_interp_type','ext+i',ierr) 4 call PetscOptionsSetValue('-pc_hypre_boomeramg_P_max','4',ierr) 5 call PetscOptionsSetValue('-pc_hypre_boomeramg_strong_threshold','4',ierr) The only option it takes is -pc_hypre_boomeramg_max_iter, the others appear as left over options at the end. My questions are: - Can the options in line 2 to 5 used only on a specific architecture e.g. only on a cluster? - Is there a certain order in setting these options? Can I set it anywhere in my code e.g. after set -ksp_type? The things I am trying to do is to reuse the solvers and preconditioners for several equations and reset only their solver/preconds type. If you have any advice, I would appreciate. Thank you in advance Jarunan -- Development Engineer ASCOMP GmbH Technoparkstrasse 1 Newton 1004 8005 Zurich Switzerland Tel: +41 44 445 4072 Fax: +41 44 445 4075 www.ascomp.ch From jedbrown at mcs.anl.gov Wed Mar 14 05:11:58 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 14 Mar 2012 05:11:58 -0500 Subject: [petsc-users] using PCFieldSplitGetSubKSP in c++ In-Reply-To: References: Message-ID: On Wed, Mar 14, 2012 at 02:53, Klaij, Christiaan wrote: > Thanks, no more problems after PCSetUP(). I am using petsc-dev, > so are you saying that MatSetNullSpace() on the block matrix [A00 > A01, A10 A11] will be enough? Can/will the null space of the > Schur complement be deduced from this? Otherwise one would still > need to "drill down" to get Schur complement matrix, right? > Unfortunately, no, and I don't think it's possible to do this automatically. Barry/Matt, this is the recurring issue of whether MatGetSubMatrix()/MatGetSchurComplement() should have the "big" matrix retain ownership, such that the user could decompose, set null spaces and auxiliary operators, restore, and have PCFieldSplit and everyone else do the right thing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Mar 14 05:21:51 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 14 Mar 2012 05:21:51 -0500 Subject: [petsc-users] AMG options In-Reply-To: <4F605E1B.50108@ascomp.ch> References: <4F605E1B.50108@ascomp.ch> Message-ID: On Wed, Mar 14, 2012 at 04:00, Jarunan Panyasantisuk wrote: > Dear PETSc team, > > I have a problem setting options for Hypre/BoomerAMG. I am trying to set > these options in the code: > > 1 call PetscOptionsSetValue('-pc_**hypre_boomeramg_max_iter','0',**ierr) > 2 call PetscOptionsSetValue('-pc_**hypre_boomeramg_coarsen_type',** > 'HMIS',ierr) > 3 call PetscOptionsSetValue('-pc_**hypre_boomeramg_interp_type','** > ext+i',ierr) > 4 call PetscOptionsSetValue('-pc_**hypre_boomeramg_P_max','4',**ierr) > 5 call PetscOptionsSetValue('-pc_**hypre_boomeramg_strong_** > threshold','4',ierr) > > The only option it takes is -pc_hypre_boomeramg_max_iter, the others > appear as left over options at the end. Are you sure there isn't an error raised and not checked? The options are all checked in the same function, so there is no reason the latter ones would not be reached. > My questions are: > > - Can the options in line 2 to 5 used only on a specific architecture e.g. > only on a cluster? > no > - Is there a certain order in setting these options? Can I set it anywhere > in my code e.g. after set -ksp_type? > You can set it anywhere before PCSetFromOptions() which is probably called via KSPSetFromOptions or SNESSetFromOptions(). > > The things I am trying to do is to reuse the solvers and preconditioners > for several equations and reset only their solver/preconds type. If you > have any advice, I would appreciate. > You are solving different equations several times and trying to reuse? Or you are iterating on a nonlinear problem? > > Thank you in advance > Jarunan > > -- > Development Engineer > > ASCOMP GmbH > Technoparkstrasse 1 > Newton 1004 > 8005 Zurich > Switzerland > Tel: +41 44 445 4072 > Fax: +41 44 445 4075 > www.ascomp.ch > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jarunan at ascomp.ch Wed Mar 14 05:53:19 2012 From: jarunan at ascomp.ch (Jarunan Panyasantisuk) Date: Wed, 14 Mar 2012 11:53:19 +0100 Subject: [petsc-users] AMG options In-Reply-To: References: <4F605E1B.50108@ascomp.ch> Message-ID: <4F60789F.6080604@ascomp.ch> > - Is there a certain order in setting these options? Can I set it > anywhere in my code e.g. after set -ksp_type? > > > You can set it anywhere before PCSetFromOptions() which is probably > called via KSPSetFromOptions or SNESSetFromOptions(). This could be one of the reason. I will check if there is any error as well. > > > The things I am trying to do is to reuse the solvers and > preconditioners for several equations and reset only their > solver/preconds type. If you have any advice, I would appreciate. > > > You are solving different equations several times and trying to reuse? > Or you are iterating on a nonlinear problem? The equations will be solved one after another. They are not solved at the same time. Yes, it is iterative. Thank you for you advice. Jarunan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Mar 14 06:06:53 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 14 Mar 2012 06:06:53 -0500 Subject: [petsc-users] AMG options In-Reply-To: <4F60789F.6080604@ascomp.ch> References: <4F605E1B.50108@ascomp.ch> <4F60789F.6080604@ascomp.ch> Message-ID: On Wed, Mar 14, 2012 at 05:53, Jarunan Panyasantisuk wrote: > The equations will be solved one after another. They are not solved at the > same time. Yes, it is iterative. We usually recommend using a different KSP for each solve since that allows information to be reused between iterations, and for less communication on each iteration. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Mar 14 07:47:41 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 14 Mar 2012 07:47:41 -0500 Subject: [petsc-users] using PCFieldSplitGetSubKSP in c++ In-Reply-To: References: Message-ID: On Wed, Mar 14, 2012 at 5:11 AM, Jed Brown wrote: > On Wed, Mar 14, 2012 at 02:53, Klaij, Christiaan wrote: > >> Thanks, no more problems after PCSetUP(). I am using petsc-dev, >> so are you saying that MatSetNullSpace() on the block matrix [A00 >> A01, A10 A11] will be enough? Can/will the null space of the >> Schur complement be deduced from this? Otherwise one would still >> need to "drill down" to get Schur complement matrix, right? >> > > Unfortunately, no, and I don't think it's possible to do this > automatically. > > Barry/Matt, this is the recurring issue of whether > MatGetSubMatrix()/MatGetSchurComplement() should have the "big" matrix > retain ownership, such that the user could decompose, set null spaces and > auxiliary operators, restore, and have PCFieldSplit and everyone else do > the right thing. > I thought MatSchurComplement() made the original matrix available? I still disagree with this for MatGetSubMatrix() since it will create a nightmare for references. I am still tracking down a reference bug in FS when you use a custom preconditoner for Schur. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Mar 14 07:52:14 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 14 Mar 2012 07:52:14 -0500 Subject: [petsc-users] using PCFieldSplitGetSubKSP in c++ In-Reply-To: References: Message-ID: On Wed, Mar 14, 2012 at 07:47, Matthew Knepley wrote: > I thought MatSchurComplement() made the original matrix available? > Not sure what you mean. The submatrices are inside a MatSchurComplement. But the user doesn't get access to the Schur complement to set near-null spaces. > I still disagree with this for > MatGetSubMatrix() since it will create a nightmare for references. I am > still tracking down a > reference bug in FS when you use a custom preconditoner for Schur. > FieldSplit in petsc-dev has some new bugs, Jungho is working on them. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.mousel at gmail.com Wed Mar 14 10:08:09 2012 From: john.mousel at gmail.com (John Mousel) Date: Wed, 14 Mar 2012 10:08:09 -0500 Subject: [petsc-users] GAMG issue Message-ID: I'm getting the following error when using GAMG. petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion `sgid==-1' failed. When I try to alter the type of aggregation at the command line using -pc_gamg_type pa, I'm getting [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type: see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! [1]PETSC ERROR: Unknown GAMG type pa given! Has there been a change in the aggregation options? I just pulled petsc-dev this morning. John -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Wed Mar 14 10:54:06 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Wed, 14 Mar 2012 11:54:06 -0400 Subject: [petsc-users] GAMG issue In-Reply-To: References: Message-ID: On Mar 14, 2012, at 11:08 AM, John Mousel wrote: > I'm getting the following error when using GAMG. > > petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion `sgid==-1' failed. Is it possible that your matrix is structurally asymmetric? This code is evolving fast and so you will need to move to the dev version if you are not already using it. (I think I fixed a bug that hit this assert). > > When I try to alter the type of aggregation at the command line using -pc_gamg_type pa, I'm getting > > [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ > [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type: > see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! > [1]PETSC ERROR: Unknown GAMG type pa given! > > Has there been a change in the aggregation options? I just pulled petsc-dev this morning. > Yes, this option is gone now. You can use -pc_gamg_type agg for now. Mark > John -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.mousel at gmail.com Wed Mar 14 10:56:31 2012 From: john.mousel at gmail.com (John Mousel) Date: Wed, 14 Mar 2012 10:56:31 -0500 Subject: [petsc-users] GAMG issue In-Reply-To: References: Message-ID: Mark, The matrix is asymmetric. Does this require the setting of an option? I pulled petsc-dev this morning, so I should have (at least close to) the latest code. John On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams wrote: > > On Mar 14, 2012, at 11:08 AM, John Mousel wrote: > > I'm getting the following error when using GAMG. > > petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion > `sgid==-1' failed. > > > Is it possible that your matrix is structurally asymmetric? > > This code is evolving fast and so you will need to move to the dev version > if you are not already using it. (I think I fixed a bug that hit this > assert). > > > When I try to alter the type of aggregation at the command line using > -pc_gamg_type pa, I'm getting > > [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external > package needed for type: > see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! > [1]PETSC ERROR: Unknown GAMG type pa given! > > Has there been a change in the aggregation options? I just pulled > petsc-dev this morning. > > > Yes, this option is gone now. You can use -pc_gamg_type agg for now. > > Mark > > John > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Wed Mar 14 11:27:44 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Wed, 14 Mar 2012 12:27:44 -0400 Subject: [petsc-users] GAMG issue In-Reply-To: References: Message-ID: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: > Mark, > > The matrix is asymmetric. Does this require the setting of an option? Yes: -pc_gamg_sym_graph Mark > I pulled petsc-dev this morning, so I should have (at least close to) the latest code. > > John > > On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams wrote: > > On Mar 14, 2012, at 11:08 AM, John Mousel wrote: > >> I'm getting the following error when using GAMG. >> >> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion `sgid==-1' failed. > > Is it possible that your matrix is structurally asymmetric? > > This code is evolving fast and so you will need to move to the dev version if you are not already using it. (I think I fixed a bug that hit this assert). > >> >> When I try to alter the type of aggregation at the command line using -pc_gamg_type pa, I'm getting >> >> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type: >> see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! >> [1]PETSC ERROR: Unknown GAMG type pa given! >> >> Has there been a change in the aggregation options? I just pulled petsc-dev this morning. >> > > Yes, this option is gone now. You can use -pc_gamg_type agg for now. > > Mark > >> John > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.mousel at gmail.com Wed Mar 14 11:54:02 2012 From: john.mousel at gmail.com (John Mousel) Date: Wed, 14 Mar 2012 11:54:02 -0500 Subject: [petsc-users] GAMG issue In-Reply-To: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> Message-ID: Mark, I have a non-symmetric matrix. I am running with the following options. -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual and with the inclusion of -pc_gamg_sym_graph, I get a new malloc error: 0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 -0500 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named wv.iihr.uiowa.eduby jmousel Wed Mar 14 11:51:35 2012 [0]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 [0]PETSC ERROR: Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c [0]PETSC ERROR: MatSetValues() line 1141 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c [0]PETSC ERROR: scaleFilterGraph() line 155 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c [0]PETSC ERROR: PCSetUp_GAMG() line 516 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c [0]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPSetUp() line 261 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: KSPSolve() line 385 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c John On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams wrote: > > On Mar 14, 2012, at 11:56 AM, John Mousel wrote: > > Mark, > > The matrix is asymmetric. Does this require the setting of an option? > > > Yes: -pc_gamg_sym_graph > > Mark > > I pulled petsc-dev this morning, so I should have (at least close to) the > latest code. > > John > > On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams wrote: > >> >> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >> >> I'm getting the following error when using GAMG. >> >> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion >> `sgid==-1' failed. >> >> >> Is it possible that your matrix is structurally asymmetric? >> >> This code is evolving fast and so you will need to move to the dev >> version if you are not already using it. (I think I fixed a bug that hit >> this assert). >> >> >> When I try to alter the type of aggregation at the command line using >> -pc_gamg_type pa, I'm getting >> >> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external >> package needed for type: >> see http://www.mcs.anl.gov/petsc/documentation/installation.html#external >> ! >> [1]PETSC ERROR: Unknown GAMG type pa given! >> >> Has there been a change in the aggregation options? I just pulled >> petsc-dev this morning. >> >> >> Yes, this option is gone now. You can use -pc_gamg_type agg for now. >> >> Mark >> >> John >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xdliang at gmail.com Wed Mar 14 13:33:21 2012 From: xdliang at gmail.com (Xiangdong Liang) Date: Wed, 14 Mar 2012 14:33:21 -0400 Subject: [petsc-users] using OpenMP in PETSc In-Reply-To: References: Message-ID: On Tue, Mar 13, 2012 at 5:57 PM, Matthew Knepley wrote: > On Tue, Mar 13, 2012 at 4:39 PM, Xiangdong Liang wrote: >> >> Hello everyone, >> >> Can someone provide me advice on using OpenMP in PETSc? I am solving a >> problem like this: >> >> int main() >> { >> ?vec vsum; >> >> ?for(i=0; i< N; i++) >> ? ?{ >> ? ? ?vec vi; >> ? ? ?fcomputev(i, vi); >> ? ? ?VecAXPY(vsum, 1.0, vi); // vsum +=vi; >> ? ?} >> >> } >> >> Can I use OpenMP "omp parallel for" to do the loop in parallel? For >> example, suppose I have 8 processes. ? It would be nice if each petsc >> subrountine fcomputev uses 2 processes while 4 different i's are >> computed in parallel (since different i's are independent). > > > PETSc is not threadsafe. This is trivial to do in MPI where the comm for eah > Vec is a group of 2 procs. Do you mean the use of MPI_Reduce? If I use MPI_Reduce, should I convert the Vec objects into regular array by vecgetarray first, then apply the MPI_Op (MPI_SUM) on these arrays? Is there any way to circumvent this vec-arrary converting by define MPI_Op and MPI_datatype on Vec ? Another quick question, where can I find the implementation of vec ops? For example, In petscvec.h, VecAXPY is implemented like (*y->ops->axpy)(y,alpha,x). Can you point me to the implementation of methods ops->axpy? Thanks. Xiangdong > > ? Matt > >> >> Any helps or hints on this would be appreciated. >> >> Best, >> Xiangdong > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From mark.adams at columbia.edu Wed Mar 14 13:38:25 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Wed, 14 Mar 2012 14:38:25 -0400 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> Message-ID: <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> Damn, I'm not preallocating the graph perfectly for unsymmetric matrices and PETSc now dies on this. I have a fix but I committed it with other changes that I do not want to commit. The changes are all in one file so I should be able to just commit this file. Anyone know how to delete a commit? I've tried: ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f hg: unknown command 'strip' 'strip' is provided by the following extension: mq manage a stack of patches use "hg help extensions" for information on enabling extensions But have not figured out how to load extensions. Mark On Mar 14, 2012, at 12:54 PM, John Mousel wrote: > Mark, > > I have a non-symmetric matrix. I am running with the following options. > > -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual > > and with the inclusion of -pc_gamg_sym_graph, I get a new malloc error: > > > 0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Argument out of range! > [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Development HG revision: 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 -0500 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 > [0]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib > [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 > [0]PETSC ERROR: Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c > [0]PETSC ERROR: MatSetValues() line 1141 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c > [0]PETSC ERROR: scaleFilterGraph() line 155 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c > [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c > [0]PETSC ERROR: PCSetUp_GAMG() line 516 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c > [0]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSetUp() line 261 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: KSPSolve() line 385 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c > > > John > > > On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams wrote: > > On Mar 14, 2012, at 11:56 AM, John Mousel wrote: > >> Mark, >> >> The matrix is asymmetric. Does this require the setting of an option? > > Yes: -pc_gamg_sym_graph > > Mark > >> I pulled petsc-dev this morning, so I should have (at least close to) the latest code. >> >> John >> >> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams wrote: >> >> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >> >>> I'm getting the following error when using GAMG. >>> >>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion `sgid==-1' failed. >> >> Is it possible that your matrix is structurally asymmetric? >> >> This code is evolving fast and so you will need to move to the dev version if you are not already using it. (I think I fixed a bug that hit this assert). >> >>> >>> When I try to alter the type of aggregation at the command line using -pc_gamg_type pa, I'm getting >>> >>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type: >>> see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! >>> [1]PETSC ERROR: Unknown GAMG type pa given! >>> >>> Has there been a change in the aggregation options? I just pulled petsc-dev this morning. >>> >> >> Yes, this option is gone now. You can use -pc_gamg_type agg for now. >> >> Mark >> >>> John >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Mar 14 13:44:04 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 14 Mar 2012 13:44:04 -0500 Subject: [petsc-users] GAMG issue In-Reply-To: <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> Message-ID: On Wed, Mar 14, 2012 at 13:38, Mark F. Adams wrote: > Damn, I'm not preallocating the graph perfectly for unsymmetric matrices > and PETSc now dies on this. > > I have a fix but I committed it with other changes that I do not want to > commit. The changes are all in one file so I should be able to just commit > this file. > > Anyone know how to delete a commit? > Be careful to make a backup. Edit your ~/.hgrc [extensions] mq = crecord = ~/src/crecord/crecord For partial commits, I use this kinda clunky non-standard extension because I don't know of anything better for Hg. hg clone https://bitbucket.org/edgimar/crecord ~/src/crecord Then you can "hg crecord" which will let you select specific lines to commit. > > I've tried: > > ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f > hg: unknown command 'strip' > 'strip' is provided by the following extension: > > mq manage a stack of patches > > use "hg help extensions" for information on enabling extensions > > But have not figured out how to load extensions. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Mar 14 13:44:39 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 14 Mar 2012 13:44:39 -0500 (CDT) Subject: [petsc-users] GAMG issue In-Reply-To: <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> Message-ID: If commit is the last hg operation that you've done - then 'hg rollback' would undo this commit. Satish On Wed, 14 Mar 2012, Mark F. Adams wrote: > Damn, I'm not preallocating the graph perfectly for unsymmetric matrices and PETSc now dies on this. > > I have a fix but I committed it with other changes that I do not want to commit. The changes are all in one file so I should be able to just commit this file. > > Anyone know how to delete a commit? > > I've tried: > > ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f > hg: unknown command 'strip' > 'strip' is provided by the following extension: > > mq manage a stack of patches > > use "hg help extensions" for information on enabling extensions > > But have not figured out how to load extensions. > > Mark > > On Mar 14, 2012, at 12:54 PM, John Mousel wrote: > > > Mark, > > > > I have a non-symmetric matrix. I am running with the following options. > > > > -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual > > > > and with the inclusion of -pc_gamg_sym_graph, I get a new malloc error: > > > > > > 0]PETSC ERROR: --------------------- Error Message ------------------------------------ > > [0]PETSC ERROR: Argument out of range! > > [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: Petsc Development HG revision: 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 -0500 > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [0]PETSC ERROR: See docs/index.html for manual pages. > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 > > [0]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib > > [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 > > [0]PETSC ERROR: Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c > > [0]PETSC ERROR: MatSetValues() line 1141 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c > > [0]PETSC ERROR: scaleFilterGraph() line 155 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c > > [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c > > [0]PETSC ERROR: PCSetUp_GAMG() line 516 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c > > [0]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: KSPSetUp() line 261 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: KSPSolve() line 385 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c > > > > > > John > > > > > > On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams wrote: > > > > On Mar 14, 2012, at 11:56 AM, John Mousel wrote: > > > >> Mark, > >> > >> The matrix is asymmetric. Does this require the setting of an option? > > > > Yes: -pc_gamg_sym_graph > > > > Mark > > > >> I pulled petsc-dev this morning, so I should have (at least close to) the latest code. > >> > >> John > >> > >> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams wrote: > >> > >> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: > >> > >>> I'm getting the following error when using GAMG. > >>> > >>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion `sgid==-1' failed. > >> > >> Is it possible that your matrix is structurally asymmetric? > >> > >> This code is evolving fast and so you will need to move to the dev version if you are not already using it. (I think I fixed a bug that hit this assert). > >> > >>> > >>> When I try to alter the type of aggregation at the command line using -pc_gamg_type pa, I'm getting > >>> > >>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type: > >>> see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! > >>> [1]PETSC ERROR: Unknown GAMG type pa given! > >>> > >>> Has there been a change in the aggregation options? I just pulled petsc-dev this morning. > >>> > >> > >> Yes, this option is gone now. You can use -pc_gamg_type agg for now. > >> > >> Mark > >> > >>> John > >> > >> > > > > > > From mark.adams at columbia.edu Wed Mar 14 14:35:02 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Wed, 14 Mar 2012 15:35:02 -0400 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> Message-ID: <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> Great, that seems to work. I did a 'hg commit tools.c' and I want to push this file only. I guess its the only thing in the change set so 'hg push' should be fine. But I see this: ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update abort: crosses branches (merge branches or use --clean to discard changes) ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge abort: outstanding uncommitted changes (use 'hg status' to list changes) ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status M include/petscmat.h M include/private/matimpl.h M src/ksp/pc/impls/gamg/agg.c M src/ksp/pc/impls/gamg/gamg.c M src/ksp/pc/impls/gamg/gamg.h M src/ksp/pc/impls/gamg/geo.c M src/mat/coarsen/coarsen.c M src/mat/coarsen/impls/hem/hem.c M src/mat/coarsen/impls/mis/mis.c Am I ready to do a push? Thanks, Mark On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: > If commit is the last hg operation that you've done - then 'hg rollback' would undo this commit. > > Satish > > On Wed, 14 Mar 2012, Mark F. Adams wrote: > >> Damn, I'm not preallocating the graph perfectly for unsymmetric matrices and PETSc now dies on this. >> >> I have a fix but I committed it with other changes that I do not want to commit. The changes are all in one file so I should be able to just commit this file. >> >> Anyone know how to delete a commit? >> >> I've tried: >> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f >> hg: unknown command 'strip' >> 'strip' is provided by the following extension: >> >> mq manage a stack of patches >> >> use "hg help extensions" for information on enabling extensions >> >> But have not figured out how to load extensions. >> >> Mark >> >> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: >> >>> Mark, >>> >>> I have a non-symmetric matrix. I am running with the following options. >>> >>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual >>> >>> and with the inclusion of -pc_gamg_sym_graph, I get a new malloc error: >>> >>> >>> 0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [0]PETSC ERROR: Argument out of range! >>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Development HG revision: 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 -0500 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 >>> [0]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib >>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 >>> [0]PETSC ERROR: Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >>> [0]PETSC ERROR: MatSetValues() line 1141 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c >>> [0]PETSC ERROR: scaleFilterGraph() line 155 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c >>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c >>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>> [0]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c >>> [0]PETSC ERROR: KSPSetUp() line 261 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: KSPSolve() line 385 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>> >>> >>> John >>> >>> >>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams wrote: >>> >>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: >>> >>>> Mark, >>>> >>>> The matrix is asymmetric. Does this require the setting of an option? >>> >>> Yes: -pc_gamg_sym_graph >>> >>> Mark >>> >>>> I pulled petsc-dev this morning, so I should have (at least close to) the latest code. >>>> >>>> John >>>> >>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams wrote: >>>> >>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >>>> >>>>> I'm getting the following error when using GAMG. >>>>> >>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion `sgid==-1' failed. >>>> >>>> Is it possible that your matrix is structurally asymmetric? >>>> >>>> This code is evolving fast and so you will need to move to the dev version if you are not already using it. (I think I fixed a bug that hit this assert). >>>> >>>>> >>>>> When I try to alter the type of aggregation at the command line using -pc_gamg_type pa, I'm getting >>>>> >>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type: >>>>> see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! >>>>> [1]PETSC ERROR: Unknown GAMG type pa given! >>>>> >>>>> Has there been a change in the aggregation options? I just pulled petsc-dev this morning. >>>>> >>>> >>>> Yes, this option is gone now. You can use -pc_gamg_type agg for now. >>>> >>>> Mark >>>> >>>>> John >>>> >>>> >>> >>> >> >> > > From balay at mcs.anl.gov Wed Mar 14 14:46:14 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 14 Mar 2012 14:46:14 -0500 (CDT) Subject: [petsc-users] GAMG issue In-Reply-To: <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> Message-ID: This is the usual merge [with uncommited changes] issue. You could use 'hg shelf' extension to shelve your local changes and then do a merge [as Sean would suggest] - or do the merge in a separate/clean clone [I normally do this..] i.e cd ~/Codes hg clone petsc-dev petsc-dev-merge cd petsc-dev-merge hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just to be sure, look for latest chagnes before merge.. hg merge hg commit hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev [now update your petsc-dev to latest] cd ~/Codes/petsc-dev hg pull hg update Satish On Wed, 14 Mar 2012, Mark F. Adams wrote: > Great, that seems to work. > > I did a 'hg commit tools.c' > > and I want to push this file only. I guess its the only thing in the change set so 'hg push' should be fine. But I see this: > > ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update > abort: crosses branches (merge branches or use --clean to discard changes) > ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge > abort: outstanding uncommitted changes (use 'hg status' to list changes) > ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status > M include/petscmat.h > M include/private/matimpl.h > M src/ksp/pc/impls/gamg/agg.c > M src/ksp/pc/impls/gamg/gamg.c > M src/ksp/pc/impls/gamg/gamg.h > M src/ksp/pc/impls/gamg/geo.c > M src/mat/coarsen/coarsen.c > M src/mat/coarsen/impls/hem/hem.c > M src/mat/coarsen/impls/mis/mis.c > > Am I ready to do a push? > > Thanks, > Mark > > On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: > > > If commit is the last hg operation that you've done - then 'hg rollback' would undo this commit. > > > > Satish > > > > On Wed, 14 Mar 2012, Mark F. Adams wrote: > > > >> Damn, I'm not preallocating the graph perfectly for unsymmetric matrices and PETSc now dies on this. > >> > >> I have a fix but I committed it with other changes that I do not want to commit. The changes are all in one file so I should be able to just commit this file. > >> > >> Anyone know how to delete a commit? > >> > >> I've tried: > >> > >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f > >> hg: unknown command 'strip' > >> 'strip' is provided by the following extension: > >> > >> mq manage a stack of patches > >> > >> use "hg help extensions" for information on enabling extensions > >> > >> But have not figured out how to load extensions. > >> > >> Mark > >> > >> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: > >> > >>> Mark, > >>> > >>> I have a non-symmetric matrix. I am running with the following options. > >>> > >>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual > >>> > >>> and with the inclusion of -pc_gamg_sym_graph, I get a new malloc error: > >>> > >>> > >>> 0]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>> [0]PETSC ERROR: Argument out of range! > >>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! > >>> [0]PETSC ERROR: ------------------------------------------------------------------------ > >>> [0]PETSC ERROR: Petsc Development HG revision: 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 -0500 > >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. > >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >>> [0]PETSC ERROR: See docs/index.html for manual pages. > >>> [0]PETSC ERROR: ------------------------------------------------------------------------ > >>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 > >>> [0]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib > >>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 > >>> [0]PETSC ERROR: Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug > >>> [0]PETSC ERROR: ------------------------------------------------------------------------ > >>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c > >>> [0]PETSC ERROR: MatSetValues() line 1141 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c > >>> [0]PETSC ERROR: scaleFilterGraph() line 155 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c > >>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c > >>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c > >>> [0]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c > >>> [0]PETSC ERROR: KSPSetUp() line 261 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c > >>> [0]PETSC ERROR: KSPSolve() line 385 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c > >>> > >>> > >>> John > >>> > >>> > >>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams wrote: > >>> > >>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: > >>> > >>>> Mark, > >>>> > >>>> The matrix is asymmetric. Does this require the setting of an option? > >>> > >>> Yes: -pc_gamg_sym_graph > >>> > >>> Mark > >>> > >>>> I pulled petsc-dev this morning, so I should have (at least close to) the latest code. > >>>> > >>>> John > >>>> > >>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams wrote: > >>>> > >>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: > >>>> > >>>>> I'm getting the following error when using GAMG. > >>>>> > >>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion `sgid==-1' failed. > >>>> > >>>> Is it possible that your matrix is structurally asymmetric? > >>>> > >>>> This code is evolving fast and so you will need to move to the dev version if you are not already using it. (I think I fixed a bug that hit this assert). > >>>> > >>>>> > >>>>> When I try to alter the type of aggregation at the command line using -pc_gamg_type pa, I'm getting > >>>>> > >>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type: > >>>>> see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! > >>>>> [1]PETSC ERROR: Unknown GAMG type pa given! > >>>>> > >>>>> Has there been a change in the aggregation options? I just pulled petsc-dev this morning. > >>>>> > >>>> > >>>> Yes, this option is gone now. You can use -pc_gamg_type agg for now. > >>>> > >>>> Mark > >>>> > >>>>> John > >>>> > >>>> > >>> > >>> > >> > >> > > > > > > From knepley at gmail.com Wed Mar 14 15:17:49 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 14 Mar 2012 15:17:49 -0500 Subject: [petsc-users] using OpenMP in PETSc In-Reply-To: References: Message-ID: On Wed, Mar 14, 2012 at 1:33 PM, Xiangdong Liang wrote: > On Tue, Mar 13, 2012 at 5:57 PM, Matthew Knepley > wrote: > > On Tue, Mar 13, 2012 at 4:39 PM, Xiangdong Liang > wrote: > >> > >> Hello everyone, > >> > >> Can someone provide me advice on using OpenMP in PETSc? I am solving a > >> problem like this: > >> > >> int main() > >> { > >> vec vsum; > >> > >> for(i=0; i< N; i++) > >> { > >> vec vi; > >> fcomputev(i, vi); > >> VecAXPY(vsum, 1.0, vi); // vsum +=vi; > >> } > >> > >> } > >> > >> Can I use OpenMP "omp parallel for" to do the loop in parallel? For > >> example, suppose I have 8 processes. It would be nice if each petsc > >> subrountine fcomputev uses 2 processes while 4 different i's are > >> computed in parallel (since different i's are independent). > > > > > > PETSc is not threadsafe. This is trivial to do in MPI where the comm for > eah > > Vec is a group of 2 procs. > > Do you mean the use of MPI_Reduce? If I use MPI_Reduce, should I > convert the Vec objects into regular array by vecgetarray first, then > apply the MPI_Op (MPI_SUM) on these arrays? Is there any way to > circumvent this vec-arrary converting by define MPI_Op and > MPI_datatype on Vec ? > I think you miss my point. This is the difference between writing SPMD programs and threaded programs. PETSc is SPMD, and I don't ever expect that to change. CUDA, for instance, is also SPMD. > Another quick question, where can I find the implementation of vec > ops? For example, In petscvec.h, VecAXPY is implemented like > (*y->ops->axpy)(y,alpha,x). Can you point me to the implementation of > methods ops->axpy? > src/vec/vec/impls Matt > Thanks. > Xiangdong > > > > > > > Matt > > > >> > >> Any helps or hints on this would be appreciated. > >> > >> Best, > >> Xiangdong > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments > > is infinitely more interesting than any results to which their > experiments > > lead. > > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From xdliang at gmail.com Wed Mar 14 15:57:35 2012 From: xdliang at gmail.com (Xiangdong Liang) Date: Wed, 14 Mar 2012 16:57:35 -0400 Subject: [petsc-users] using OpenMP in PETSc In-Reply-To: References: Message-ID: On Wed, Mar 14, 2012 at 4:17 PM, Matthew Knepley wrote: > On Wed, Mar 14, 2012 at 1:33 PM, Xiangdong Liang wrote: >> >> On Tue, Mar 13, 2012 at 5:57 PM, Matthew Knepley >> wrote: >> > On Tue, Mar 13, 2012 at 4:39 PM, Xiangdong Liang >> > wrote: >> >> >> >> Hello everyone, >> >> >> >> Can someone provide me advice on using OpenMP in PETSc? I am solving a >> >> problem like this: >> >> >> >> int main() >> >> { >> >> ?vec vsum; >> >> >> >> ?for(i=0; i< N; i++) >> >> ? ?{ >> >> ? ? ?vec vi; >> >> ? ? ?fcomputev(i, vi); >> >> ? ? ?VecAXPY(vsum, 1.0, vi); // vsum +=vi; >> >> ? ?} >> >> >> >> } >> >> >> >> Can I use OpenMP "omp parallel for" to do the loop in parallel? For >> >> example, suppose I have 8 processes. ? It would be nice if each petsc >> >> subrountine fcomputev uses 2 processes while 4 different i's are >> >> computed in parallel (since different i's are independent). >> > >> > >> > PETSc is not threadsafe. This is trivial to do in MPI where the comm for >> > eah >> > Vec is a group of 2 procs. >> >> Do you mean the use of MPI_Reduce? If I use MPI_Reduce, should I >> convert the Vec objects into regular array by vecgetarray first, then >> apply the MPI_Op (MPI_SUM) on these arrays? Is there any way to >> circumvent this vec-arrary converting by define MPI_Op and >> MPI_datatype on Vec ? > > > I think you miss my point. This is the difference between writing SPMD > programs > and threaded programs. PETSc is SPMD, and I don't ever expect that to > change. I am still not clear about this. Is MPI_Reduce or converting vec to array is unnecessary? Can you explain more about that? Thanks. Xiangdong > CUDA, for instance, is also SPMD. > >> >> Another quick question, where can I find the implementation of vec >> ops? ?For example, In petscvec.h, VecAXPY is implemented like >> (*y->ops->axpy)(y,alpha,x). ?Can you point me to the implementation of >> methods ops->axpy? > > > src/vec/vec/impls > > ? ?Matt > >> >> Thanks. >> Xiangdong >> >> >> >> > >> > ? Matt >> > >> >> >> >> Any helps or hints on this would be appreciated. >> >> >> >> Best, >> >> Xiangdong >> > >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> > experiments >> > is infinitely more interesting than any results to which their >> > experiments >> > lead. >> > -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From mark.adams at columbia.edu Wed Mar 14 16:40:54 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Wed, 14 Mar 2012 17:40:54 -0400 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> Message-ID: <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> John, I've committed these changes, give a try. Mark On Mar 14, 2012, at 3:46 PM, Satish Balay wrote: > This is the usual merge [with uncommited changes] issue. > > You could use 'hg shelf' extension to shelve your local changes and > then do a merge [as Sean would suggest] - or do the merge in a > separate/clean clone [I normally do this..] > > i.e > cd ~/Codes > hg clone petsc-dev petsc-dev-merge > cd petsc-dev-merge > hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just to be sure, look for latest chagnes before merge.. > hg merge > hg commit > hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev > > [now update your petsc-dev to latest] > cd ~/Codes/petsc-dev > hg pull > hg update > > Satish > > On Wed, 14 Mar 2012, Mark F. Adams wrote: > >> Great, that seems to work. >> >> I did a 'hg commit tools.c' >> >> and I want to push this file only. I guess its the only thing in the change set so 'hg push' should be fine. But I see this: >> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update >> abort: crosses branches (merge branches or use --clean to discard changes) >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge >> abort: outstanding uncommitted changes (use 'hg status' to list changes) >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status >> M include/petscmat.h >> M include/private/matimpl.h >> M src/ksp/pc/impls/gamg/agg.c >> M src/ksp/pc/impls/gamg/gamg.c >> M src/ksp/pc/impls/gamg/gamg.h >> M src/ksp/pc/impls/gamg/geo.c >> M src/mat/coarsen/coarsen.c >> M src/mat/coarsen/impls/hem/hem.c >> M src/mat/coarsen/impls/mis/mis.c >> >> Am I ready to do a push? >> >> Thanks, >> Mark >> >> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: >> >>> If commit is the last hg operation that you've done - then 'hg rollback' would undo this commit. >>> >>> Satish >>> >>> On Wed, 14 Mar 2012, Mark F. Adams wrote: >>> >>>> Damn, I'm not preallocating the graph perfectly for unsymmetric matrices and PETSc now dies on this. >>>> >>>> I have a fix but I committed it with other changes that I do not want to commit. The changes are all in one file so I should be able to just commit this file. >>>> >>>> Anyone know how to delete a commit? >>>> >>>> I've tried: >>>> >>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f >>>> hg: unknown command 'strip' >>>> 'strip' is provided by the following extension: >>>> >>>> mq manage a stack of patches >>>> >>>> use "hg help extensions" for information on enabling extensions >>>> >>>> But have not figured out how to load extensions. >>>> >>>> Mark >>>> >>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: >>>> >>>>> Mark, >>>>> >>>>> I have a non-symmetric matrix. I am running with the following options. >>>>> >>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual >>>>> >>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new malloc error: >>>>> >>>>> >>>>> 0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [0]PETSC ERROR: Argument out of range! >>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: Petsc Development HG revision: 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 -0500 >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 >>>>> [0]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib >>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 >>>>> [0]PETSC ERROR: Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >>>>> [0]PETSC ERROR: MatSetValues() line 1141 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c >>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c >>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>>>> [0]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: KSPSolve() line 385 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>> >>>>> >>>>> John >>>>> >>>>> >>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams wrote: >>>>> >>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: >>>>> >>>>>> Mark, >>>>>> >>>>>> The matrix is asymmetric. Does this require the setting of an option? >>>>> >>>>> Yes: -pc_gamg_sym_graph >>>>> >>>>> Mark >>>>> >>>>>> I pulled petsc-dev this morning, so I should have (at least close to) the latest code. >>>>>> >>>>>> John >>>>>> >>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams wrote: >>>>>> >>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >>>>>> >>>>>>> I'm getting the following error when using GAMG. >>>>>>> >>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion `sgid==-1' failed. >>>>>> >>>>>> Is it possible that your matrix is structurally asymmetric? >>>>>> >>>>>> This code is evolving fast and so you will need to move to the dev version if you are not already using it. (I think I fixed a bug that hit this assert). >>>>>> >>>>>>> >>>>>>> When I try to alter the type of aggregation at the command line using -pc_gamg_type pa, I'm getting >>>>>>> >>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type: >>>>>>> see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! >>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given! >>>>>>> >>>>>>> Has there been a change in the aggregation options? I just pulled petsc-dev this morning. >>>>>>> >>>>>> >>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg for now. >>>>>> >>>>>> Mark >>>>>> >>>>>>> John >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > > From john.mousel at gmail.com Wed Mar 14 16:53:58 2012 From: john.mousel at gmail.com (John Mousel) Date: Wed, 14 Mar 2012 16:53:58 -0500 Subject: [petsc-users] GAMG issue In-Reply-To: <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> Message-ID: Mark, No change. Can you give me the location that you patched so I can check to make sure it pulled? I don't see it on the petsc-dev change log. John On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams wrote: > John, I've committed these changes, give a try. > > Mark > > On Mar 14, 2012, at 3:46 PM, Satish Balay wrote: > > > This is the usual merge [with uncommited changes] issue. > > > > You could use 'hg shelf' extension to shelve your local changes and > > then do a merge [as Sean would suggest] - or do the merge in a > > separate/clean clone [I normally do this..] > > > > i.e > > cd ~/Codes > > hg clone petsc-dev petsc-dev-merge > > cd petsc-dev-merge > > hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just to be > sure, look for latest chagnes before merge.. > > hg merge > > hg commit > > hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev > > > > [now update your petsc-dev to latest] > > cd ~/Codes/petsc-dev > > hg pull > > hg update > > > > Satish > > > > On Wed, 14 Mar 2012, Mark F. Adams wrote: > > > >> Great, that seems to work. > >> > >> I did a 'hg commit tools.c' > >> > >> and I want to push this file only. I guess its the only thing in the > change set so 'hg push' should be fine. But I see this: > >> > >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update > >> abort: crosses branches (merge branches or use --clean to discard > changes) > >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge > >> abort: outstanding uncommitted changes (use 'hg status' to list changes) > >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status > >> M include/petscmat.h > >> M include/private/matimpl.h > >> M src/ksp/pc/impls/gamg/agg.c > >> M src/ksp/pc/impls/gamg/gamg.c > >> M src/ksp/pc/impls/gamg/gamg.h > >> M src/ksp/pc/impls/gamg/geo.c > >> M src/mat/coarsen/coarsen.c > >> M src/mat/coarsen/impls/hem/hem.c > >> M src/mat/coarsen/impls/mis/mis.c > >> > >> Am I ready to do a push? > >> > >> Thanks, > >> Mark > >> > >> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: > >> > >>> If commit is the last hg operation that you've done - then 'hg > rollback' would undo this commit. > >>> > >>> Satish > >>> > >>> On Wed, 14 Mar 2012, Mark F. Adams wrote: > >>> > >>>> Damn, I'm not preallocating the graph perfectly for unsymmetric > matrices and PETSc now dies on this. > >>>> > >>>> I have a fix but I committed it with other changes that I do not want > to commit. The changes are all in one file so I should be able to just > commit this file. > >>>> > >>>> Anyone know how to delete a commit? > >>>> > >>>> I've tried: > >>>> > >>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f > >>>> hg: unknown command 'strip' > >>>> 'strip' is provided by the following extension: > >>>> > >>>> mq manage a stack of patches > >>>> > >>>> use "hg help extensions" for information on enabling extensions > >>>> > >>>> But have not figured out how to load extensions. > >>>> > >>>> Mark > >>>> > >>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: > >>>> > >>>>> Mark, > >>>>> > >>>>> I have a non-symmetric matrix. I am running with the following > options. > >>>>> > >>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual > >>>>> > >>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new malloc > error: > >>>>> > >>>>> > >>>>> 0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > >>>>> [0]PETSC ERROR: Argument out of range! > >>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! > >>>>> [0]PETSC ERROR: > ------------------------------------------------------------------------ > >>>>> [0]PETSC ERROR: Petsc Development HG revision: > 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 > -0500 > >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. > >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. > >>>>> [0]PETSC ERROR: > ------------------------------------------------------------------------ > >>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named > wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 > >>>>> [0]PETSC ERROR: Libraries linked from > /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib > >>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 > >>>>> [0]PETSC ERROR: Configure options --download-blacs=1 > --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 > --download-parmetis=1 --download-scalapack=1 > --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc > --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort > PETSC_ARCH=linux-debug > >>>>> [0]PETSC ERROR: > ------------------------------------------------------------------------ > >>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in > /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c > >>>>> [0]PETSC ERROR: MatSetValues() line 1141 in > /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c > >>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in > /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c > >>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in > /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c > >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in > /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c > >>>>> [0]PETSC ERROR: PCSetUp() line 832 in > /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c > >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in > /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c > >>>>> [0]PETSC ERROR: KSPSolve() line 385 in > /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c > >>>>> > >>>>> > >>>>> John > >>>>> > >>>>> > >>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams < > mark.adams at columbia.edu> wrote: > >>>>> > >>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: > >>>>> > >>>>>> Mark, > >>>>>> > >>>>>> The matrix is asymmetric. Does this require the setting of an > option? > >>>>> > >>>>> Yes: -pc_gamg_sym_graph > >>>>> > >>>>> Mark > >>>>> > >>>>>> I pulled petsc-dev this morning, so I should have (at least close > to) the latest code. > >>>>>> > >>>>>> John > >>>>>> > >>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams < > mark.adams at columbia.edu> wrote: > >>>>>> > >>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: > >>>>>> > >>>>>>> I'm getting the following error when using GAMG. > >>>>>>> > >>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion > `sgid==-1' failed. > >>>>>> > >>>>>> Is it possible that your matrix is structurally asymmetric? > >>>>>> > >>>>>> This code is evolving fast and so you will need to move to the dev > version if you are not already using it. (I think I fixed a bug that hit > this assert). > >>>>>> > >>>>>>> > >>>>>>> When I try to alter the type of aggregation at the command line > using -pc_gamg_type pa, I'm getting > >>>>>>> > >>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error > Message ------------------------------------ > >>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing > external package needed for type: > >>>>>>> see > http://www.mcs.anl.gov/petsc/documentation/installation.html#external! > >>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given! > >>>>>>> > >>>>>>> Has there been a change in the aggregation options? I just pulled > petsc-dev this morning. > >>>>>>> > >>>>>> > >>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg for > now. > >>>>>> > >>>>>> Mark > >>>>>> > >>>>>>> John > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >> > >> > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Wed Mar 14 18:04:08 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Wed, 14 Mar 2012 19:04:08 -0400 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> Message-ID: <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> Humm, I see it with hg view (appended). Satish, my main repo looks hosed. I see this: ~/Codes/petsc-dev>hg update abort: crosses branches (merge branches or use --clean to discard changes) ~/Codes/petsc-dev>hg merge abort: branch 'default' has 3 heads - please merge with an explicit rev (run 'hg heads .' to see heads) ~/Codes/petsc-dev>hg heads changeset: 22496:8e2a98268179 tag: tip user: Barry Smith date: Wed Mar 14 16:42:25 2012 -0500 files: src/vec/is/interface/f90-custom/zindexf90.c src/vec/vec/interface/f90-custom/zvectorf90.c description: undoing manually changes I put in because Satish had a better fix changeset: 22492:bda4df63072d user: Mark F. Adams date: Wed Mar 14 17:39:52 2012 -0400 files: src/ksp/pc/impls/gamg/tools.c description: fix for unsymmetric matrices. changeset: 22469:b063baf366e4 user: Mark F. Adams date: Wed Mar 14 14:22:28 2012 -0400 files: src/ksp/pc/impls/gamg/tools.c description: added fix for preallocation for unsymetric matrices. Mark my 'hg view' on my merge repo: Revision: 22492 Branch: default Author: Mark F. Adams 2012-03-14 17:39:52 Committer: Mark F. Adams 2012-03-14 17:39:52 Tags: tip Parent: 22491:451bbbd291c2 (Small fixes to the BT linesearch) fix for unsymmetric matrices. ------------------------ src/ksp/pc/impls/gamg/tools.c ------------------------ @@ -103,7 +103,7 @@ PetscErrorCode ierr; PetscInt Istart,Iend,Ii,jj,ncols,nnz0,nnz1, NN, MM, nloc; PetscMPIInt mype, npe; - Mat Gmat = *a_Gmat, tGmat; + Mat Gmat = *a_Gmat, tGmat, matTrans; MPI_Comm wcomm = ((PetscObject)Gmat)->comm; const PetscScalar *vals; const PetscInt *idx; @@ -127,6 +127,10 @@ ierr = MatDiagonalScale( Gmat, diag, diag ); CHKERRQ(ierr); ierr = VecDestroy( &diag ); CHKERRQ(ierr); + if( symm ) { + ierr = MatTranspose( Gmat, MAT_INITIAL_MATRIX, &matTrans ); CHKERRQ(ierr); + } + /* filter - dup zeros out matrix */ ierr = PetscMalloc( nloc*sizeof(PetscInt), &d_nnz ); CHKERRQ(ierr); ierr = PetscMalloc( nloc*sizeof(PetscInt), &o_nnz ); CHKERRQ(ierr); @@ -135,6 +139,12 @@ d_nnz[jj] = ncols; o_nnz[jj] = ncols; ierr = MatRestoreRow(Gmat,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); + if( symm ) { + ierr = MatGetRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); + d_nnz[jj] += ncols; + o_nnz[jj] += ncols; + ierr = MatRestoreRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); + } if( d_nnz[jj] > nloc ) d_nnz[jj] = nloc; if( o_nnz[jj] > (MM-nloc) ) o_nnz[jj] = MM - nloc; } @@ -142,6 +152,9 @@ CHKERRQ(ierr); ierr = PetscFree( d_nnz ); CHKERRQ(ierr); ierr = PetscFree( o_nnz ); CHKERRQ(ierr); + if( symm ) { + ierr = MatDestroy( &matTrans ); CHKERRQ(ierr); + } On Mar 14, 2012, at 5:53 PM, John Mousel wrote: > Mark, > > No change. Can you give me the location that you patched so I can check to make sure it pulled? > I don't see it on the petsc-dev change log. > > John > > On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams wrote: > John, I've committed these changes, give a try. > > Mark > > On Mar 14, 2012, at 3:46 PM, Satish Balay wrote: > > > This is the usual merge [with uncommited changes] issue. > > > > You could use 'hg shelf' extension to shelve your local changes and > > then do a merge [as Sean would suggest] - or do the merge in a > > separate/clean clone [I normally do this..] > > > > i.e > > cd ~/Codes > > hg clone petsc-dev petsc-dev-merge > > cd petsc-dev-merge > > hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just to be sure, look for latest chagnes before merge.. > > hg merge > > hg commit > > hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev > > > > [now update your petsc-dev to latest] > > cd ~/Codes/petsc-dev > > hg pull > > hg update > > > > Satish > > > > On Wed, 14 Mar 2012, Mark F. Adams wrote: > > > >> Great, that seems to work. > >> > >> I did a 'hg commit tools.c' > >> > >> and I want to push this file only. I guess its the only thing in the change set so 'hg push' should be fine. But I see this: > >> > >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update > >> abort: crosses branches (merge branches or use --clean to discard changes) > >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge > >> abort: outstanding uncommitted changes (use 'hg status' to list changes) > >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status > >> M include/petscmat.h > >> M include/private/matimpl.h > >> M src/ksp/pc/impls/gamg/agg.c > >> M src/ksp/pc/impls/gamg/gamg.c > >> M src/ksp/pc/impls/gamg/gamg.h > >> M src/ksp/pc/impls/gamg/geo.c > >> M src/mat/coarsen/coarsen.c > >> M src/mat/coarsen/impls/hem/hem.c > >> M src/mat/coarsen/impls/mis/mis.c > >> > >> Am I ready to do a push? > >> > >> Thanks, > >> Mark > >> > >> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: > >> > >>> If commit is the last hg operation that you've done - then 'hg rollback' would undo this commit. > >>> > >>> Satish > >>> > >>> On Wed, 14 Mar 2012, Mark F. Adams wrote: > >>> > >>>> Damn, I'm not preallocating the graph perfectly for unsymmetric matrices and PETSc now dies on this. > >>>> > >>>> I have a fix but I committed it with other changes that I do not want to commit. The changes are all in one file so I should be able to just commit this file. > >>>> > >>>> Anyone know how to delete a commit? > >>>> > >>>> I've tried: > >>>> > >>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f > >>>> hg: unknown command 'strip' > >>>> 'strip' is provided by the following extension: > >>>> > >>>> mq manage a stack of patches > >>>> > >>>> use "hg help extensions" for information on enabling extensions > >>>> > >>>> But have not figured out how to load extensions. > >>>> > >>>> Mark > >>>> > >>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: > >>>> > >>>>> Mark, > >>>>> > >>>>> I have a non-symmetric matrix. I am running with the following options. > >>>>> > >>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual > >>>>> > >>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new malloc error: > >>>>> > >>>>> > >>>>> 0]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>>>> [0]PETSC ERROR: Argument out of range! > >>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! > >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ > >>>>> [0]PETSC ERROR: Petsc Development HG revision: 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 -0500 > >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. > >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. > >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ > >>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 > >>>>> [0]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib > >>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 > >>>>> [0]PETSC ERROR: Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug > >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ > >>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c > >>>>> [0]PETSC ERROR: MatSetValues() line 1141 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c > >>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c > >>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c > >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c > >>>>> [0]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c > >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c > >>>>> [0]PETSC ERROR: KSPSolve() line 385 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c > >>>>> > >>>>> > >>>>> John > >>>>> > >>>>> > >>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams wrote: > >>>>> > >>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: > >>>>> > >>>>>> Mark, > >>>>>> > >>>>>> The matrix is asymmetric. Does this require the setting of an option? > >>>>> > >>>>> Yes: -pc_gamg_sym_graph > >>>>> > >>>>> Mark > >>>>> > >>>>>> I pulled petsc-dev this morning, so I should have (at least close to) the latest code. > >>>>>> > >>>>>> John > >>>>>> > >>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams wrote: > >>>>>> > >>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: > >>>>>> > >>>>>>> I'm getting the following error when using GAMG. > >>>>>>> > >>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion `sgid==-1' failed. > >>>>>> > >>>>>> Is it possible that your matrix is structurally asymmetric? > >>>>>> > >>>>>> This code is evolving fast and so you will need to move to the dev version if you are not already using it. (I think I fixed a bug that hit this assert). > >>>>>> > >>>>>>> > >>>>>>> When I try to alter the type of aggregation at the command line using -pc_gamg_type pa, I'm getting > >>>>>>> > >>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type: > >>>>>>> see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! > >>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given! > >>>>>>> > >>>>>>> Has there been a change in the aggregation options? I just pulled petsc-dev this morning. > >>>>>>> > >>>>>> > >>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg for now. > >>>>>> > >>>>>> Mark > >>>>>> > >>>>>>> John > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >> > >> > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengshixiangze at 163.com Thu Mar 15 08:00:54 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Thu, 15 Mar 2012 21:00:54 +0800 (CST) Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> <97d3e2d.10edc.1360c777086.Coremail.zengshixiangze@163.com> Message-ID: <56e287e6.15a37.13616725ab4.Coremail.zengshixiangze@163.com> Hi, When i run my code today, it appears "out of memory" which not happened last time. The following is the error message: [0]PETSC ERROR: Out of memory. This could be due to allocating [0]PETSC ERROR: too large an object or bleeding by not properly [0]PETSC ERROR: destroying unneeded objects. [0]PETSC ERROR: Memory allocated 5659280320 Memory used by process 5719199744 [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [0]PETSC ERROR: Memory requested 18446744070677811200! [0]PETSC ERROR: ---------------------------------------- It happens at the process of creating the matrices. Another problem, when I add -log_summary, there is no message about the performance appearing. What's wrong? Thank you. Zeng ? 2012-03-13 22:36:46?"Jed Brown" ??? 2012/3/13 Xiangze Zeng I do set preallocation after setting the matrix type. Run with -info to make sure it is used (i.e. that the matrix type isn't changed later). Note that PCSOR does not run on the GPU, so it will do lots of copying (run with -log_summary to see). You should start by running PCJACOBI on the GPU. Zeng ? 2012-03-13 22:14:12?"Jed Brown" ??? 2012/3/13 Xiangze Zeng At the beginning and end of the codes for setting the matrices values, I add "printf", and compute the time of this period. It is much longer than that when I don't use the GPU. I just guess the time is used for copping data. My PCTYPE is sor. And 2000 iterations. Do you have any suggestion about this? -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengshixiangze at 163.com Thu Mar 15 08:08:27 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Thu, 15 Mar 2012 21:08:27 +0800 (CST) Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: <56e287e6.15a37.13616725ab4.Coremail.zengshixiangze@163.com> References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> <97d3e2d.10edc.1360c777086.Coremail.zengshixiangze@163.com> <56e287e6.15a37.13616725ab4.Coremail.zengshixiangze@163.com> Message-ID: <1eba23.15cde.13616794188.Coremail.zengshixiangze@163.com> And it seems to request so large memory. At 2012-03-15 21:00:54,"Xiangze Zeng" wrote: Hi, When i run my code today, it appears "out of memory" which not happened last time. The following is the error message: [0]PETSC ERROR: Out of memory. This could be due to allocating [0]PETSC ERROR: too large an object or bleeding by not properly [0]PETSC ERROR: destroying unneeded objects. [0]PETSC ERROR: Memory allocated 5659280320 Memory used by process 5719199744 [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [0]PETSC ERROR: Memory requested 18446744070677811200! [0]PETSC ERROR: ---------------------------------------- It happens at the process of creating the matrices. Another problem, when I add -log_summary, there is no message about the performance appearing. What's wrong? Thank you. Zeng ? 2012-03-13 22:36:46?"Jed Brown" ??? 2012/3/13 Xiangze Zeng I do set preallocation after setting the matrix type. Run with -info to make sure it is used (i.e. that the matrix type isn't changed later). Note that PCSOR does not run on the GPU, so it will do lots of copying (run with -log_summary to see). You should start by running PCJACOBI on the GPU. Zeng ? 2012-03-13 22:14:12?"Jed Brown" ??? 2012/3/13 Xiangze Zeng At the beginning and end of the codes for setting the matrices values, I add "printf", and compute the time of this period. It is much longer than that when I don't use the GPU. I just guess the time is used for copping data. My PCTYPE is sor. And 2000 iterations. Do you have any suggestion about this? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Mar 15 08:12:43 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 15 Mar 2012 08:12:43 -0500 Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: <56e287e6.15a37.13616725ab4.Coremail.zengshixiangze@163.com> References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> <97d3e2d.10edc.1360c777086.Coremail.zengshixiangze@163.com> <56e287e6.15a37.13616725ab4.Coremail.zengshixiangze@163.com> Message-ID: 2012/3/15 Xiangze Zeng > When i run my code today, it appears "out of memory" which not happened > last time. The following is the error message: > > [0]PETSC ERROR: Out of memory. This could be due to allocating > [0]PETSC ERROR: too large an object or bleeding by not properly > [0]PETSC ERROR: destroying unneeded objects. > [0]PETSC ERROR: Memory allocated 5659280320 Memory used by process > 5719199744 > [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. > [0]PETSC ERROR: Memory requested 18446744070677811200! > [0]PETSC ERROR: ---------------------------------------- > This is not the error message, this is PART OF the error message. Always send the whole thing so that we don't have to exchange these emails to get the rest of the interesting part of the message. (There is memory corruption somewhere, but this message doesn't tell us where to suggest you look.) > > It happens at the process of creating the matrices. > Another problem, when I add -log_summary, there is no message about the > performance appearing. > What's wrong? > The run has to complete successfully to see -log_summary output. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengshixiangze at 163.com Thu Mar 15 09:23:06 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Thu, 15 Mar 2012 22:23:06 +0800 (CST) Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> <97d3e2d.10edc.1360c777086.Coremail.zengshixiangze@163.com> <56e287e6.15a37.13616725ab4.Coremail.zengshixiangze@163.com> Message-ID: <633e1e3b.16bcc.13616bd9bf2.Coremail.zengshixiangze@163.com> Sorry, someting is wrong with my file which stores the size of the matriex. But I still can't see -log_summary output. I just see at the first time, after that, never. ? 2012-03-15 21:12:43?"Jed Brown" ??? 2012/3/15 Xiangze Zeng When i run my code today, it appears "out of memory" which not happened last time. The following is the error message: [0]PETSC ERROR: Out of memory. This could be due to allocating [0]PETSC ERROR: too large an object or bleeding by not properly [0]PETSC ERROR: destroying unneeded objects. [0]PETSC ERROR: Memory allocated 5659280320 Memory used by process 5719199744 [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [0]PETSC ERROR: Memory requested 18446744070677811200! [0]PETSC ERROR: ---------------------------------------- This is not the error message, this is PART OF the error message. Always send the whole thing so that we don't have to exchange these emails to get the rest of the interesting part of the message. (There is memory corruption somewhere, but this message doesn't tell us where to suggest you look.) It happens at the process of creating the matrices. Another problem, when I add -log_summary, there is no message about the performance appearing. What's wrong? The run has to complete successfully to see -log_summary output. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Mar 15 09:26:41 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 15 Mar 2012 09:26:41 -0500 Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: <633e1e3b.16bcc.13616bd9bf2.Coremail.zengshixiangze@163.com> References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> <97d3e2d.10edc.1360c777086.Coremail.zengshixiangze@163.com> <56e287e6.15a37.13616725ab4.Coremail.zengshixiangze@163.com> <633e1e3b.16bcc.13616bd9bf2.Coremail.zengshixiangze@163.com> Message-ID: 2012/3/15 Xiangze Zeng > Sorry, someting is wrong with my file which stores the size of the > matriex. > But I still can't see -log_summary output. I just see at the first time, > after that, never. > It is written by PetscFinalize(). Find out why you don't get there. Also note that the option needs to be visible at PetscInitialize (through the command line/options file/environment). -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.mousel at gmail.com Thu Mar 15 10:13:10 2012 From: john.mousel at gmail.com (John Mousel) Date: Thu, 15 Mar 2012 10:13:10 -0500 Subject: [petsc-users] GAMG issue In-Reply-To: <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> Message-ID: Mark, The changes pulled through this morning. I've run it with the options -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale -ksp_diagonal_scale_fix -pc_mg_levels 4 -mg_levels_ksp_type richardson -mg_levels_pc_type sor -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 and it converges in the true residual, but it's not converging as fast as anticpated. The matrix arises from a non-symmetric discretization of the Poisson equation. The solve takes GAMG 114 iterations, whereas ML takes 24 iterations, BoomerAMG takes 22 iterations, and -ksp_type bcgsl -pc_type bjacobi -sub_pc_type ilu -sub_pc_factor_levels 4 takes around 170. I've attached the -ksp_view results for ML,GAMG, and HYPRE. I've attempted to make all the options the same on all levels for ML and GAMG. Any thoughts? John On Wed, Mar 14, 2012 at 6:04 PM, Mark F. Adams wrote: > Humm, I see it with hg view (appended). > > Satish, my main repo looks hosed. I see this: > > ~/Codes/petsc-dev>hg update > abort: crosses branches (merge branches or use --clean to discard changes) > ~/Codes/petsc-dev>hg merge > abort: branch 'default' has 3 heads - please merge with an explicit rev > (run 'hg heads .' to see heads) > ~/Codes/petsc-dev>hg heads > changeset: 22496:8e2a98268179 > tag: tip > user: Barry Smith > date: Wed Mar 14 16:42:25 2012 -0500 > files: src/vec/is/interface/f90-custom/zindexf90.c > src/vec/vec/interface/f90-custom/zvectorf90.c > description: > undoing manually changes I put in because Satish had a better fix > > > changeset: 22492:bda4df63072d > user: Mark F. Adams > date: Wed Mar 14 17:39:52 2012 -0400 > files: src/ksp/pc/impls/gamg/tools.c > description: > fix for unsymmetric matrices. > > > changeset: 22469:b063baf366e4 > user: Mark F. Adams > date: Wed Mar 14 14:22:28 2012 -0400 > files: src/ksp/pc/impls/gamg/tools.c > description: > added fix for preallocation for unsymetric matrices. > > Mark > > my 'hg view' on my merge repo: > > Revision: 22492 > Branch: default > Author: Mark F. Adams 2012-03-14 17:39:52 > Committer: Mark F. Adams 2012-03-14 17:39:52 > Tags: tip > Parent: 22491:451bbbd291c2 (Small fixes to the BT linesearch) > > fix for unsymmetric matrices. > > > ------------------------ src/ksp/pc/impls/gamg/tools.c > ------------------------ > @@ -103,7 +103,7 @@ > PetscErrorCode ierr; > PetscInt Istart,Iend,Ii,jj,ncols,nnz0,nnz1, NN, MM, nloc; > PetscMPIInt mype, npe; > - Mat Gmat = *a_Gmat, tGmat; > + Mat Gmat = *a_Gmat, tGmat, matTrans; > MPI_Comm wcomm = ((PetscObject)Gmat)->comm; > const PetscScalar *vals; > const PetscInt *idx; > @@ -127,6 +127,10 @@ > ierr = MatDiagonalScale( Gmat, diag, diag ); CHKERRQ(ierr); > ierr = VecDestroy( &diag ); CHKERRQ(ierr); > > + if( symm ) { > + ierr = MatTranspose( Gmat, MAT_INITIAL_MATRIX, &matTrans ); > CHKERRQ(ierr); > + } > + > /* filter - dup zeros out matrix */ > ierr = PetscMalloc( nloc*sizeof(PetscInt), &d_nnz ); CHKERRQ(ierr); > ierr = PetscMalloc( nloc*sizeof(PetscInt), &o_nnz ); CHKERRQ(ierr); > @@ -135,6 +139,12 @@ > d_nnz[jj] = ncols; > o_nnz[jj] = ncols; > ierr = MatRestoreRow(Gmat,Ii,&ncols,PETSC_NULL,PETSC_NULL); > CHKERRQ(ierr); > + if( symm ) { > + ierr = MatGetRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); > CHKERRQ(ierr); > + d_nnz[jj] += ncols; > + o_nnz[jj] += ncols; > + ierr = MatRestoreRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); > CHKERRQ(ierr); > + } > if( d_nnz[jj] > nloc ) d_nnz[jj] = nloc; > if( o_nnz[jj] > (MM-nloc) ) o_nnz[jj] = MM - nloc; > } > @@ -142,6 +152,9 @@ > CHKERRQ(ierr); > ierr = PetscFree( d_nnz ); CHKERRQ(ierr); > ierr = PetscFree( o_nnz ); CHKERRQ(ierr); > + if( symm ) { > + ierr = MatDestroy( &matTrans ); CHKERRQ(ierr); > + } > > > > > On Mar 14, 2012, at 5:53 PM, John Mousel wrote: > > Mark, > > No change. Can you give me the location that you patched so I can check to > make sure it pulled? > I don't see it on the petsc-dev change log. > > John > > On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams wrote: > >> John, I've committed these changes, give a try. >> >> Mark >> >> On Mar 14, 2012, at 3:46 PM, Satish Balay wrote: >> >> > This is the usual merge [with uncommited changes] issue. >> > >> > You could use 'hg shelf' extension to shelve your local changes and >> > then do a merge [as Sean would suggest] - or do the merge in a >> > separate/clean clone [I normally do this..] >> > >> > i.e >> > cd ~/Codes >> > hg clone petsc-dev petsc-dev-merge >> > cd petsc-dev-merge >> > hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just to be >> sure, look for latest chagnes before merge.. >> > hg merge >> > hg commit >> > hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev >> > >> > [now update your petsc-dev to latest] >> > cd ~/Codes/petsc-dev >> > hg pull >> > hg update >> > >> > Satish >> > >> > On Wed, 14 Mar 2012, Mark F. Adams wrote: >> > >> >> Great, that seems to work. >> >> >> >> I did a 'hg commit tools.c' >> >> >> >> and I want to push this file only. I guess its the only thing in the >> change set so 'hg push' should be fine. But I see this: >> >> >> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update >> >> abort: crosses branches (merge branches or use --clean to discard >> changes) >> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge >> >> abort: outstanding uncommitted changes (use 'hg status' to list >> changes) >> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status >> >> M include/petscmat.h >> >> M include/private/matimpl.h >> >> M src/ksp/pc/impls/gamg/agg.c >> >> M src/ksp/pc/impls/gamg/gamg.c >> >> M src/ksp/pc/impls/gamg/gamg.h >> >> M src/ksp/pc/impls/gamg/geo.c >> >> M src/mat/coarsen/coarsen.c >> >> M src/mat/coarsen/impls/hem/hem.c >> >> M src/mat/coarsen/impls/mis/mis.c >> >> >> >> Am I ready to do a push? >> >> >> >> Thanks, >> >> Mark >> >> >> >> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: >> >> >> >>> If commit is the last hg operation that you've done - then 'hg >> rollback' would undo this commit. >> >>> >> >>> Satish >> >>> >> >>> On Wed, 14 Mar 2012, Mark F. Adams wrote: >> >>> >> >>>> Damn, I'm not preallocating the graph perfectly for unsymmetric >> matrices and PETSc now dies on this. >> >>>> >> >>>> I have a fix but I committed it with other changes that I do not >> want to commit. The changes are all in one file so I should be able to >> just commit this file. >> >>>> >> >>>> Anyone know how to delete a commit? >> >>>> >> >>>> I've tried: >> >>>> >> >>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f >> >>>> hg: unknown command 'strip' >> >>>> 'strip' is provided by the following extension: >> >>>> >> >>>> mq manage a stack of patches >> >>>> >> >>>> use "hg help extensions" for information on enabling extensions >> >>>> >> >>>> But have not figured out how to load extensions. >> >>>> >> >>>> Mark >> >>>> >> >>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: >> >>>> >> >>>>> Mark, >> >>>>> >> >>>>> I have a non-symmetric matrix. I am running with the following >> options. >> >>>>> >> >>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual >> >>>>> >> >>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new malloc >> error: >> >>>>> >> >>>>> >> >>>>> 0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> >>>>> [0]PETSC ERROR: Argument out of range! >> >>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! >> >>>>> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> >>>>> [0]PETSC ERROR: Petsc Development HG revision: >> 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 >> -0500 >> >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >> >>>>> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> >>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named >> wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 >> >>>>> [0]PETSC ERROR: Libraries linked from >> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib >> >>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 >> >>>>> [0]PETSC ERROR: Configure options --download-blacs=1 >> --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 >> --download-parmetis=1 --download-scalapack=1 >> --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc >> --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort >> PETSC_ARCH=linux-debug >> >>>>> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> >>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in >> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >> >>>>> [0]PETSC ERROR: MatSetValues() line 1141 in >> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c >> >>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in >> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c >> >>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in >> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c >> >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in >> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >> >>>>> [0]PETSC ERROR: PCSetUp() line 832 in >> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c >> >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in >> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >> >>>>> [0]PETSC ERROR: KSPSolve() line 385 in >> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >> >>>>> >> >>>>> >> >>>>> John >> >>>>> >> >>>>> >> >>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams < >> mark.adams at columbia.edu> wrote: >> >>>>> >> >>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: >> >>>>> >> >>>>>> Mark, >> >>>>>> >> >>>>>> The matrix is asymmetric. Does this require the setting of an >> option? >> >>>>> >> >>>>> Yes: -pc_gamg_sym_graph >> >>>>> >> >>>>> Mark >> >>>>> >> >>>>>> I pulled petsc-dev this morning, so I should have (at least close >> to) the latest code. >> >>>>>> >> >>>>>> John >> >>>>>> >> >>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams < >> mark.adams at columbia.edu> wrote: >> >>>>>> >> >>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >> >>>>>> >> >>>>>>> I'm getting the following error when using GAMG. >> >>>>>>> >> >>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion >> `sgid==-1' failed. >> >>>>>> >> >>>>>> Is it possible that your matrix is structurally asymmetric? >> >>>>>> >> >>>>>> This code is evolving fast and so you will need to move to the dev >> version if you are not already using it. (I think I fixed a bug that hit >> this assert). >> >>>>>> >> >>>>>>> >> >>>>>>> When I try to alter the type of aggregation at the command line >> using -pc_gamg_type pa, I'm getting >> >>>>>>> >> >>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error >> Message ------------------------------------ >> >>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing >> external package needed for type: >> >>>>>>> see >> http://www.mcs.anl.gov/petsc/documentation/installation.html#external! >> >>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given! >> >>>>>>> >> >>>>>>> Has there been a change in the aggregation options? I just pulled >> petsc-dev this morning. >> >>>>>>> >> >>>>>> >> >>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg for >> now. >> >>>>>> >> >>>>>> Mark >> >>>>>> >> >>>>>>> John >> >>>>>> >> >>>>>> >> >>>>> >> >>>>> >> >>>> >> >>>> >> >>> >> >>> >> >> >> >> >> > >> > >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- Residual norms for pres_ solve. 0 KSP Residual norm 4.059775183487e+03 2 KSP Residual norm 5.981316947830e+02 4 KSP Residual norm 4.442881917561e+02 6 KSP Residual norm 3.949125263925e+02 8 KSP Residual norm 7.982748331499e+03 10 KSP Residual norm 2.543280382461e+02 12 KSP Residual norm 2.185515850048e+02 14 KSP Residual norm 1.802232035923e+02 16 KSP Residual norm 1.583506303369e+02 18 KSP Residual norm 1.599584824650e+02 20 KSP Residual norm 2.888585316347e+02 22 KSP Residual norm 1.561026662942e+02 24 KSP Residual norm 1.771271404110e+02 26 KSP Residual norm 1.988800994293e+02 28 KSP Residual norm 2.199783177060e+02 30 KSP Residual norm 2.216968849511e+02 32 KSP Residual norm 2.072764401471e+02 34 KSP Residual norm 1.860542194649e+02 36 KSP Residual norm 1.659181700778e+02 38 KSP Residual norm 1.407464754713e+02 40 KSP Residual norm 9.615564181863e+01 42 KSP Residual norm 6.703321281230e+01 44 KSP Residual norm 3.561383031838e+01 46 KSP Residual norm 1.856503785272e+01 48 KSP Residual norm 1.055079242233e+01 50 KSP Residual norm 6.356112490758e+00 52 KSP Residual norm 3.425262242318e+00 54 KSP Residual norm 3.077183029782e+00 56 KSP Residual norm 2.176818497656e+00 58 KSP Residual norm 1.078873696603e+00 60 KSP Residual norm 5.918603863161e-01 62 KSP Residual norm 3.752380791312e-01 64 KSP Residual norm 1.412075511011e-01 66 KSP Residual norm 2.671083368701e-02 68 KSP Residual norm 4.773262749061e-03 70 KSP Residual norm 7.963615707246e-03 72 KSP Residual norm 9.995350897604e-03 74 KSP Residual norm 9.622695628075e-03 76 KSP Residual norm 2.242543288576e-03 78 KSP Residual norm 3.841006934453e-03 80 KSP Residual norm 8.987952547562e-03 82 KSP Residual norm 9.558153629986e-04 84 KSP Residual norm 2.646555390162e-04 86 KSP Residual norm 1.186132284573e-04 88 KSP Residual norm 3.628058439929e-05 90 KSP Residual norm 1.793225729106e-05 92 KSP Residual norm 4.744894233256e-06 94 KSP Residual norm 2.508223070292e-06 96 KSP Residual norm 7.000610692450e-07 98 KSP Residual norm 3.780248712706e-07 100 KSP Residual norm 2.502747308880e-07 102 KSP Residual norm 1.378104516997e-07 104 KSP Residual norm 1.109001062835e-07 106 KSP Residual norm 6.202415071048e-08 108 KSP Residual norm 2.871737213539e-08 110 KSP Residual norm 1.871854343209e-08 112 KSP Residual norm 1.163299109304e-08 114 KSP Residual norm 9.392404490548e-09 KSP Object:(pres_) 4 MPI processes type: bcgsl BCGSL: Ell = 2 BCGSL: Delta = 0 maximum iterations=5000 tolerances: relative=1e-12, absolute=1e-50, divergence=10000 left preconditioning diagonally scaled system has attached null space using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object:(pres_) 4 MPI processes type: gamg MG: type is MULTIPLICATIVE, levels=4 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (pres_mg_coarse_) 4 MPI processes type: preonly maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (pres_mg_coarse_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 8, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=139, cols=139 total: nonzeros=679, allocated nonzeros=679 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (pres_mg_levels_1_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_1_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=895, cols=895 total: nonzeros=4941, allocated nonzeros=4941 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (pres_mg_levels_2_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_2_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=7604, cols=7604 total: nonzeros=49366, allocated nonzeros=49366 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (pres_mg_levels_3_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_3_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=58507, cols=58507 total: nonzeros=383336, allocated nonzeros=675924 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=58507, cols=58507 total: nonzeros=383336, allocated nonzeros=675924 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines -------------- next part -------------- Residual norms for pres_ solve. 0 KSP Residual norm 1.083611831517e+04 2 KSP Residual norm 3.063605374040e+03 4 KSP Residual norm 1.926278721105e+03 6 KSP Residual norm 2.562396342913e+02 8 KSP Residual norm 1.472308108162e+01 10 KSP Residual norm 2.062803551544e+00 12 KSP Residual norm 1.101533503605e-01 14 KSP Residual norm 1.007101075709e-02 16 KSP Residual norm 9.417367089322e-04 18 KSP Residual norm 1.016475488904e-04 20 KSP Residual norm 7.187713879600e-06 22 KSP Residual norm 1.458531748697e-07 24 KSP Residual norm 6.829597900465e-09 KSP Object:(pres_) 4 MPI processes type: bcgsl BCGSL: Ell = 2 BCGSL: Delta = 0 maximum iterations=5000 tolerances: relative=1e-12, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object:(pres_) 4 MPI processes type: ml MG: type is MULTIPLICATIVE, levels=4 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (pres_mg_coarse_) 4 MPI processes type: preonly maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (pres_mg_coarse_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 8, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=35, cols=35 total: nonzeros=479, allocated nonzeros=479 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (pres_mg_levels_1_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_1_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=532, cols=532 total: nonzeros=8539, allocated nonzeros=8539 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (pres_mg_levels_2_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_2_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=8439, cols=8439 total: nonzeros=110975, allocated nonzeros=110975 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (pres_mg_levels_3_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_3_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=58507, cols=58507 total: nonzeros=383336, allocated nonzeros=675924 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=58507, cols=58507 total: nonzeros=383336, allocated nonzeros=675924 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines -------------- next part -------------- Residual norms for pres_ solve. 0 KSP preconditioned resid norm 1.767885052835e+04 true resid norm 5.490506953240e+02 ||r(i)||/||b|| 1.173737492770e-01 2 KSP preconditioned resid norm 2.066096994623e+03 true resid norm 2.745377688272e+03 ||r(i)||/||b|| 5.868952998296e-01 4 KSP preconditioned resid norm 1.106321516621e+02 true resid norm 1.033500617653e+02 ||r(i)||/||b|| 2.209374169035e-02 6 KSP preconditioned resid norm 2.570916205201e+01 true resid norm 2.235586678502e+01 ||r(i)||/||b|| 4.779143210710e-03 8 KSP preconditioned resid norm 1.619194902332e+00 true resid norm 3.773126954763e-01 ||r(i)||/||b|| 8.066032170621e-05 10 KSP preconditioned resid norm 6.319088912041e-01 true resid norm 1.630872902347e-01 ||r(i)||/||b|| 3.486411523980e-05 12 KSP preconditioned resid norm 3.808406632980e-02 true resid norm 8.627569048343e-03 ||r(i)||/||b|| 1.844365438336e-06 14 KSP preconditioned resid norm 2.683282198467e-03 true resid norm 5.026895110387e-04 ||r(i)||/||b|| 1.074628502164e-07 16 KSP preconditioned resid norm 1.517361222053e-05 true resid norm 9.443042302108e-06 ||r(i)||/||b|| 2.018693882038e-09 18 KSP preconditioned resid norm 3.351216887604e-06 true resid norm 1.563343028855e-06 ||r(i)||/||b|| 3.342048999581e-10 20 KSP preconditioned resid norm 1.331870500595e-07 true resid norm 3.754585860708e-08 ||r(i)||/||b|| 8.026395799272e-12 22 KSP preconditioned resid norm 5.782696971967e-09 true resid norm 3.780795199311e-09 ||r(i)||/||b|| 8.082425021418e-13 KSP Object:(pres_) 4 MPI processes type: bcgsl BCGSL: Ell = 2 BCGSL: Delta = 0 maximum iterations=5000 tolerances: relative=1e-12, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object:(pres_) 4 MPI processes type: hypre HYPRE BoomerAMG preconditioning HYPRE BoomerAMG: Cycle type V HYPRE BoomerAMG: Maximum number of levels 25 HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1 HYPRE BoomerAMG: Convergence tolerance PER hypre call 0 HYPRE BoomerAMG: Threshold for strong coupling 0.25 HYPRE BoomerAMG: Interpolation truncation factor 0 HYPRE BoomerAMG: Interpolation: max elements per row 0 HYPRE BoomerAMG: Number of levels of aggressive coarsening 0 HYPRE BoomerAMG: Number of paths for aggressive coarsening 1 HYPRE BoomerAMG: Maximum row sums 0.9 HYPRE BoomerAMG: Sweeps down 1 HYPRE BoomerAMG: Sweeps up 1 HYPRE BoomerAMG: Sweeps on coarse 4 HYPRE BoomerAMG: Relax down symmetric-SOR/Jacobi HYPRE BoomerAMG: Relax up symmetric-SOR/Jacobi HYPRE BoomerAMG: Relax on coarse symmetric-SOR/Jacobi HYPRE BoomerAMG: Relax weight (all) 1 HYPRE BoomerAMG: Outer relax weight (all) 1 HYPRE BoomerAMG: Using CF-relaxation HYPRE BoomerAMG: Measure type local HYPRE BoomerAMG: Coarsen type PMIS HYPRE BoomerAMG: Interpolation type classical linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=58507, cols=58507 total: nonzeros=383336, allocated nonzeros=675924 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines From jedbrown at mcs.anl.gov Thu Mar 15 10:24:14 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 15 Mar 2012 10:24:14 -0500 Subject: [petsc-users] GAMG issue In-Reply-To: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> Message-ID: On Wed, Mar 14, 2012 at 11:27, Mark F. Adams wrote: > The matrix is asymmetric. Does this require the setting of an option? > > > Yes: -pc_gamg_sym_graph > Mark, I don't like this name. First, it's imperative in the sense that it's describing a procedure for fixing something rather than declarative (stating some property of the problem). Second, there is already a way to state symmetry, MatSetOption(): Options Describing Matrix Structure: + MAT_SPD - symmetric positive definite - MAT_SYMMETRIC - symmetric in terms of both structure and value . MAT_HERMITIAN - transpose is the complex conjugation . MAT_STRUCTURALLY_SYMMETRIC - symmetric nonzero structure - MAT_SYMMETRY_ETERNAL - if you would like the symmetry/Hermitian flag you set to be kept with all future use of the matrix including after MatAssemblyBegin/End() which could potentially change the symmetry structure, i.e. you KNOW the matrix will ALWAYS have the property you set. Is there a reason you can't just use this information? If you have multiple methods for handling nonsymmetric structure, then I would suggest using PetscOptionsList() or PetscOptionsEnum() to select which to use. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Thu Mar 15 10:55:04 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Thu, 15 Mar 2012 11:55:04 -0400 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> Message-ID: <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> John, can you run again with: -pc_gamg_verbose 1 And I would not use: -pc_mg_levels 4 -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 1) I think -mg_coarse_ksp_type preonly and -mg_coarse_pc_sor_its 8 do not do what you think. I think this is the same as 1 iteration. I think you want 'richardson' not 'preonly'. 2) Why are you using sor as the coarse solver? If your problem is singular then you want to use as many levels as possible to get the coarse grid to be tiny. I'm pretty sure HYPRE ignores the coarse solver parameters. But ML uses them and it is converging well. 3) I would not specify the number of levels. GAMG, and I think the rest, have internal logic for stopping a the right level. If the coarse level is large and you use just 8 iterations of sor then convergence will suffer. Mark On Mar 15, 2012, at 11:13 AM, John Mousel wrote: > Mark, > > The changes pulled through this morning. I've run it with the options > > -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale -ksp_diagonal_scale_fix -pc_mg_levels 4 -mg_levels_ksp_type richardson -mg_levels_pc_type sor -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 > > and it converges in the true residual, but it's not converging as fast as anticpated. The matrix arises from a non-symmetric discretization of the Poisson equation. The solve takes GAMG 114 iterations, whereas ML takes 24 iterations, BoomerAMG takes 22 iterations, and -ksp_type bcgsl -pc_type bjacobi -sub_pc_type ilu -sub_pc_factor_levels 4 takes around 170. I've attached the -ksp_view results for ML,GAMG, and HYPRE. I've attempted to make all the options the same on all levels for ML and GAMG. > > Any thoughts? > > John > > > On Wed, Mar 14, 2012 at 6:04 PM, Mark F. Adams wrote: > Humm, I see it with hg view (appended). > > Satish, my main repo looks hosed. I see this: > > ~/Codes/petsc-dev>hg update > abort: crosses branches (merge branches or use --clean to discard changes) > ~/Codes/petsc-dev>hg merge > abort: branch 'default' has 3 heads - please merge with an explicit rev > (run 'hg heads .' to see heads) > ~/Codes/petsc-dev>hg heads > changeset: 22496:8e2a98268179 > tag: tip > user: Barry Smith > date: Wed Mar 14 16:42:25 2012 -0500 > files: src/vec/is/interface/f90-custom/zindexf90.c src/vec/vec/interface/f90-custom/zvectorf90.c > description: > undoing manually changes I put in because Satish had a better fix > > > changeset: 22492:bda4df63072d > user: Mark F. Adams > date: Wed Mar 14 17:39:52 2012 -0400 > files: src/ksp/pc/impls/gamg/tools.c > description: > fix for unsymmetric matrices. > > > changeset: 22469:b063baf366e4 > user: Mark F. Adams > date: Wed Mar 14 14:22:28 2012 -0400 > files: src/ksp/pc/impls/gamg/tools.c > description: > added fix for preallocation for unsymetric matrices. > > Mark > > my 'hg view' on my merge repo: > > Revision: 22492 > Branch: default > Author: Mark F. Adams 2012-03-14 17:39:52 > Committer: Mark F. Adams 2012-03-14 17:39:52 > Tags: tip > Parent: 22491:451bbbd291c2 (Small fixes to the BT linesearch) > > fix for unsymmetric matrices. > > > ------------------------ src/ksp/pc/impls/gamg/tools.c ------------------------ > @@ -103,7 +103,7 @@ > PetscErrorCode ierr; > PetscInt Istart,Iend,Ii,jj,ncols,nnz0,nnz1, NN, MM, nloc; > PetscMPIInt mype, npe; > - Mat Gmat = *a_Gmat, tGmat; > + Mat Gmat = *a_Gmat, tGmat, matTrans; > MPI_Comm wcomm = ((PetscObject)Gmat)->comm; > const PetscScalar *vals; > const PetscInt *idx; > @@ -127,6 +127,10 @@ > ierr = MatDiagonalScale( Gmat, diag, diag ); CHKERRQ(ierr); > ierr = VecDestroy( &diag ); CHKERRQ(ierr); > > + if( symm ) { > + ierr = MatTranspose( Gmat, MAT_INITIAL_MATRIX, &matTrans ); CHKERRQ(ierr); > + } > + > /* filter - dup zeros out matrix */ > ierr = PetscMalloc( nloc*sizeof(PetscInt), &d_nnz ); CHKERRQ(ierr); > ierr = PetscMalloc( nloc*sizeof(PetscInt), &o_nnz ); CHKERRQ(ierr); > @@ -135,6 +139,12 @@ > d_nnz[jj] = ncols; > o_nnz[jj] = ncols; > ierr = MatRestoreRow(Gmat,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); > + if( symm ) { > + ierr = MatGetRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); > + d_nnz[jj] += ncols; > + o_nnz[jj] += ncols; > + ierr = MatRestoreRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); > + } > if( d_nnz[jj] > nloc ) d_nnz[jj] = nloc; > if( o_nnz[jj] > (MM-nloc) ) o_nnz[jj] = MM - nloc; > } > @@ -142,6 +152,9 @@ > CHKERRQ(ierr); > ierr = PetscFree( d_nnz ); CHKERRQ(ierr); > ierr = PetscFree( o_nnz ); CHKERRQ(ierr); > + if( symm ) { > + ierr = MatDestroy( &matTrans ); CHKERRQ(ierr); > + } > > > > > On Mar 14, 2012, at 5:53 PM, John Mousel wrote: > >> Mark, >> >> No change. Can you give me the location that you patched so I can check to make sure it pulled? >> I don't see it on the petsc-dev change log. >> >> John >> >> On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams wrote: >> John, I've committed these changes, give a try. >> >> Mark >> >> On Mar 14, 2012, at 3:46 PM, Satish Balay wrote: >> >> > This is the usual merge [with uncommited changes] issue. >> > >> > You could use 'hg shelf' extension to shelve your local changes and >> > then do a merge [as Sean would suggest] - or do the merge in a >> > separate/clean clone [I normally do this..] >> > >> > i.e >> > cd ~/Codes >> > hg clone petsc-dev petsc-dev-merge >> > cd petsc-dev-merge >> > hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just to be sure, look for latest chagnes before merge.. >> > hg merge >> > hg commit >> > hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev >> > >> > [now update your petsc-dev to latest] >> > cd ~/Codes/petsc-dev >> > hg pull >> > hg update >> > >> > Satish >> > >> > On Wed, 14 Mar 2012, Mark F. Adams wrote: >> > >> >> Great, that seems to work. >> >> >> >> I did a 'hg commit tools.c' >> >> >> >> and I want to push this file only. I guess its the only thing in the change set so 'hg push' should be fine. But I see this: >> >> >> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update >> >> abort: crosses branches (merge branches or use --clean to discard changes) >> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge >> >> abort: outstanding uncommitted changes (use 'hg status' to list changes) >> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status >> >> M include/petscmat.h >> >> M include/private/matimpl.h >> >> M src/ksp/pc/impls/gamg/agg.c >> >> M src/ksp/pc/impls/gamg/gamg.c >> >> M src/ksp/pc/impls/gamg/gamg.h >> >> M src/ksp/pc/impls/gamg/geo.c >> >> M src/mat/coarsen/coarsen.c >> >> M src/mat/coarsen/impls/hem/hem.c >> >> M src/mat/coarsen/impls/mis/mis.c >> >> >> >> Am I ready to do a push? >> >> >> >> Thanks, >> >> Mark >> >> >> >> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: >> >> >> >>> If commit is the last hg operation that you've done - then 'hg rollback' would undo this commit. >> >>> >> >>> Satish >> >>> >> >>> On Wed, 14 Mar 2012, Mark F. Adams wrote: >> >>> >> >>>> Damn, I'm not preallocating the graph perfectly for unsymmetric matrices and PETSc now dies on this. >> >>>> >> >>>> I have a fix but I committed it with other changes that I do not want to commit. The changes are all in one file so I should be able to just commit this file. >> >>>> >> >>>> Anyone know how to delete a commit? >> >>>> >> >>>> I've tried: >> >>>> >> >>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f >> >>>> hg: unknown command 'strip' >> >>>> 'strip' is provided by the following extension: >> >>>> >> >>>> mq manage a stack of patches >> >>>> >> >>>> use "hg help extensions" for information on enabling extensions >> >>>> >> >>>> But have not figured out how to load extensions. >> >>>> >> >>>> Mark >> >>>> >> >>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: >> >>>> >> >>>>> Mark, >> >>>>> >> >>>>> I have a non-symmetric matrix. I am running with the following options. >> >>>>> >> >>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual >> >>>>> >> >>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new malloc error: >> >>>>> >> >>>>> >> >>>>> 0]PETSC ERROR: --------------------- Error Message ------------------------------------ >> >>>>> [0]PETSC ERROR: Argument out of range! >> >>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! >> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >> >>>>> [0]PETSC ERROR: Petsc Development HG revision: 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 -0500 >> >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >> >>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 >> >>>>> [0]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib >> >>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 >> >>>>> [0]PETSC ERROR: Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug >> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >> >>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >> >>>>> [0]PETSC ERROR: MatSetValues() line 1141 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c >> >>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c >> >>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c >> >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >> >>>>> [0]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c >> >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >> >>>>> [0]PETSC ERROR: KSPSolve() line 385 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >> >>>>> >> >>>>> >> >>>>> John >> >>>>> >> >>>>> >> >>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams wrote: >> >>>>> >> >>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: >> >>>>> >> >>>>>> Mark, >> >>>>>> >> >>>>>> The matrix is asymmetric. Does this require the setting of an option? >> >>>>> >> >>>>> Yes: -pc_gamg_sym_graph >> >>>>> >> >>>>> Mark >> >>>>> >> >>>>>> I pulled petsc-dev this morning, so I should have (at least close to) the latest code. >> >>>>>> >> >>>>>> John >> >>>>>> >> >>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams wrote: >> >>>>>> >> >>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >> >>>>>> >> >>>>>>> I'm getting the following error when using GAMG. >> >>>>>>> >> >>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion `sgid==-1' failed. >> >>>>>> >> >>>>>> Is it possible that your matrix is structurally asymmetric? >> >>>>>> >> >>>>>> This code is evolving fast and so you will need to move to the dev version if you are not already using it. (I think I fixed a bug that hit this assert). >> >>>>>> >> >>>>>>> >> >>>>>>> When I try to alter the type of aggregation at the command line using -pc_gamg_type pa, I'm getting >> >>>>>>> >> >>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >> >>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type: >> >>>>>>> see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! >> >>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given! >> >>>>>>> >> >>>>>>> Has there been a change in the aggregation options? I just pulled petsc-dev this morning. >> >>>>>>> >> >>>>>> >> >>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg for now. >> >>>>>> >> >>>>>> Mark >> >>>>>> >> >>>>>>> John >> >>>>>> >> >>>>>> >> >>>>> >> >>>>> >> >>>> >> >>>> >> >>> >> >>> >> >> >> >> >> > >> > >> >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.mousel at gmail.com Thu Mar 15 11:21:20 2012 From: john.mousel at gmail.com (John Mousel) Date: Thu, 15 Mar 2012 11:21:20 -0500 Subject: [petsc-users] GAMG issue In-Reply-To: <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> Message-ID: Mark, I ran with those options removed (see the run options listed below). Things actually got slightly worse. Now it's up to 142 iterations. I have attached the ksp_view output. -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale -ksp_diagonal_scale_fix -mg_levels_ksp_type richardson -mg_levels_pc_type sor -pc_gamg_verbose 1 John On Thu, Mar 15, 2012 at 10:55 AM, Mark F. Adams wrote: > John, can you run again with: -pc_gamg_verbose 1 > > And I would not use: -pc_mg_levels 4 -mg_coarse_ksp_type preonly > -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 > > 1) I think -mg_coarse_ksp_type preonly and -mg_coarse_pc_sor_its 8 do not > do what you think. I think this is the same as 1 iteration. I think you > want 'richardson' not 'preonly'. > > 2) Why are you using sor as the coarse solver? If your problem is > singular then you want to use as many levels as possible to get the coarse > grid to be tiny. I'm pretty sure HYPRE ignores the coarse solver > parameters. But ML uses them and it is converging well. > > 3) I would not specify the number of levels. GAMG, and I think the rest, > have internal logic for stopping a the right level. If the coarse level is > large and you use just 8 iterations of sor then convergence will suffer. > > Mark > > On Mar 15, 2012, at 11:13 AM, John Mousel wrote: > > Mark, > > The changes pulled through this morning. I've run it with the options > > -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale > -ksp_diagonal_scale_fix -pc_mg_levels 4 -mg_levels_ksp_type richardson > -mg_levels_pc_type sor -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor > -mg_coarse_pc_sor_its 8 > > and it converges in the true residual, but it's not converging as fast as > anticpated. The matrix arises from a non-symmetric discretization of the > Poisson equation. The solve takes GAMG 114 iterations, whereas ML takes 24 > iterations, BoomerAMG takes 22 iterations, and -ksp_type bcgsl -pc_type > bjacobi -sub_pc_type ilu -sub_pc_factor_levels 4 takes around 170. I've > attached the -ksp_view results for ML,GAMG, and HYPRE. I've attempted to > make all the options the same on all levels for ML and GAMG. > > Any thoughts? > > John > > > On Wed, Mar 14, 2012 at 6:04 PM, Mark F. Adams wrote: > >> Humm, I see it with hg view (appended). >> >> Satish, my main repo looks hosed. I see this: >> >> ~/Codes/petsc-dev>hg update >> abort: crosses branches (merge branches or use --clean to discard changes) >> ~/Codes/petsc-dev>hg merge >> abort: branch 'default' has 3 heads - please merge with an explicit rev >> (run 'hg heads .' to see heads) >> ~/Codes/petsc-dev>hg heads >> changeset: 22496:8e2a98268179 >> tag: tip >> user: Barry Smith >> date: Wed Mar 14 16:42:25 2012 -0500 >> files: src/vec/is/interface/f90-custom/zindexf90.c >> src/vec/vec/interface/f90-custom/zvectorf90.c >> description: >> undoing manually changes I put in because Satish had a better fix >> >> >> changeset: 22492:bda4df63072d >> user: Mark F. Adams >> date: Wed Mar 14 17:39:52 2012 -0400 >> files: src/ksp/pc/impls/gamg/tools.c >> description: >> fix for unsymmetric matrices. >> >> >> changeset: 22469:b063baf366e4 >> user: Mark F. Adams >> date: Wed Mar 14 14:22:28 2012 -0400 >> files: src/ksp/pc/impls/gamg/tools.c >> description: >> added fix for preallocation for unsymetric matrices. >> >> Mark >> >> my 'hg view' on my merge repo: >> >> Revision: 22492 >> Branch: default >> Author: Mark F. Adams 2012-03-14 17:39:52 >> Committer: Mark F. Adams 2012-03-14 17:39:52 >> Tags: tip >> Parent: 22491:451bbbd291c2 (Small fixes to the BT linesearch) >> >> fix for unsymmetric matrices. >> >> >> ------------------------ src/ksp/pc/impls/gamg/tools.c >> ------------------------ >> @@ -103,7 +103,7 @@ >> PetscErrorCode ierr; >> PetscInt Istart,Iend,Ii,jj,ncols,nnz0,nnz1, NN, MM, nloc; >> PetscMPIInt mype, npe; >> - Mat Gmat = *a_Gmat, tGmat; >> + Mat Gmat = *a_Gmat, tGmat, matTrans; >> MPI_Comm wcomm = ((PetscObject)Gmat)->comm; >> const PetscScalar *vals; >> const PetscInt *idx; >> @@ -127,6 +127,10 @@ >> ierr = MatDiagonalScale( Gmat, diag, diag ); CHKERRQ(ierr); >> ierr = VecDestroy( &diag ); CHKERRQ(ierr); >> >> + if( symm ) { >> + ierr = MatTranspose( Gmat, MAT_INITIAL_MATRIX, &matTrans ); >> CHKERRQ(ierr); >> + } >> + >> /* filter - dup zeros out matrix */ >> ierr = PetscMalloc( nloc*sizeof(PetscInt), &d_nnz ); CHKERRQ(ierr); >> ierr = PetscMalloc( nloc*sizeof(PetscInt), &o_nnz ); CHKERRQ(ierr); >> @@ -135,6 +139,12 @@ >> d_nnz[jj] = ncols; >> o_nnz[jj] = ncols; >> ierr = MatRestoreRow(Gmat,Ii,&ncols,PETSC_NULL,PETSC_NULL); >> CHKERRQ(ierr); >> + if( symm ) { >> + ierr = MatGetRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); >> CHKERRQ(ierr); >> + d_nnz[jj] += ncols; >> + o_nnz[jj] += ncols; >> + ierr = MatRestoreRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); >> CHKERRQ(ierr); >> + } >> if( d_nnz[jj] > nloc ) d_nnz[jj] = nloc; >> if( o_nnz[jj] > (MM-nloc) ) o_nnz[jj] = MM - nloc; >> } >> @@ -142,6 +152,9 @@ >> CHKERRQ(ierr); >> ierr = PetscFree( d_nnz ); CHKERRQ(ierr); >> ierr = PetscFree( o_nnz ); CHKERRQ(ierr); >> + if( symm ) { >> + ierr = MatDestroy( &matTrans ); CHKERRQ(ierr); >> + } >> >> >> >> >> On Mar 14, 2012, at 5:53 PM, John Mousel wrote: >> >> Mark, >> >> No change. Can you give me the location that you patched so I can check >> to make sure it pulled? >> I don't see it on the petsc-dev change log. >> >> John >> >> On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams wrote: >> >>> John, I've committed these changes, give a try. >>> >>> Mark >>> >>> On Mar 14, 2012, at 3:46 PM, Satish Balay wrote: >>> >>> > This is the usual merge [with uncommited changes] issue. >>> > >>> > You could use 'hg shelf' extension to shelve your local changes and >>> > then do a merge [as Sean would suggest] - or do the merge in a >>> > separate/clean clone [I normally do this..] >>> > >>> > i.e >>> > cd ~/Codes >>> > hg clone petsc-dev petsc-dev-merge >>> > cd petsc-dev-merge >>> > hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just to >>> be sure, look for latest chagnes before merge.. >>> > hg merge >>> > hg commit >>> > hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev >>> > >>> > [now update your petsc-dev to latest] >>> > cd ~/Codes/petsc-dev >>> > hg pull >>> > hg update >>> > >>> > Satish >>> > >>> > On Wed, 14 Mar 2012, Mark F. Adams wrote: >>> > >>> >> Great, that seems to work. >>> >> >>> >> I did a 'hg commit tools.c' >>> >> >>> >> and I want to push this file only. I guess its the only thing in the >>> change set so 'hg push' should be fine. But I see this: >>> >> >>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update >>> >> abort: crosses branches (merge branches or use --clean to discard >>> changes) >>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge >>> >> abort: outstanding uncommitted changes (use 'hg status' to list >>> changes) >>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status >>> >> M include/petscmat.h >>> >> M include/private/matimpl.h >>> >> M src/ksp/pc/impls/gamg/agg.c >>> >> M src/ksp/pc/impls/gamg/gamg.c >>> >> M src/ksp/pc/impls/gamg/gamg.h >>> >> M src/ksp/pc/impls/gamg/geo.c >>> >> M src/mat/coarsen/coarsen.c >>> >> M src/mat/coarsen/impls/hem/hem.c >>> >> M src/mat/coarsen/impls/mis/mis.c >>> >> >>> >> Am I ready to do a push? >>> >> >>> >> Thanks, >>> >> Mark >>> >> >>> >> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: >>> >> >>> >>> If commit is the last hg operation that you've done - then 'hg >>> rollback' would undo this commit. >>> >>> >>> >>> Satish >>> >>> >>> >>> On Wed, 14 Mar 2012, Mark F. Adams wrote: >>> >>> >>> >>>> Damn, I'm not preallocating the graph perfectly for unsymmetric >>> matrices and PETSc now dies on this. >>> >>>> >>> >>>> I have a fix but I committed it with other changes that I do not >>> want to commit. The changes are all in one file so I should be able to >>> just commit this file. >>> >>>> >>> >>>> Anyone know how to delete a commit? >>> >>>> >>> >>>> I've tried: >>> >>>> >>> >>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f >>> >>>> hg: unknown command 'strip' >>> >>>> 'strip' is provided by the following extension: >>> >>>> >>> >>>> mq manage a stack of patches >>> >>>> >>> >>>> use "hg help extensions" for information on enabling extensions >>> >>>> >>> >>>> But have not figured out how to load extensions. >>> >>>> >>> >>>> Mark >>> >>>> >>> >>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: >>> >>>> >>> >>>>> Mark, >>> >>>>> >>> >>>>> I have a non-symmetric matrix. I am running with the following >>> options. >>> >>>>> >>> >>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual >>> >>>>> >>> >>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new malloc >>> error: >>> >>>>> >>> >>>>> >>> >>>>> 0]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> >>>>> [0]PETSC ERROR: Argument out of range! >>> >>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! >>> >>>>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> >>>>> [0]PETSC ERROR: Petsc Development HG revision: >>> 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 >>> -0500 >>> >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> >>>>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> >>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named >>> wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 >>> >>>>> [0]PETSC ERROR: Libraries linked from >>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib >>> >>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 >>> >>>>> [0]PETSC ERROR: Configure options --download-blacs=1 >>> --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 >>> --download-parmetis=1 --download-scalapack=1 >>> --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc >>> --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort >>> PETSC_ARCH=linux-debug >>> >>>>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> >>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in >>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >>> >>>>> [0]PETSC ERROR: MatSetValues() line 1141 in >>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c >>> >>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in >>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c >>> >>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in >>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c >>> >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in >>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>> >>>>> [0]PETSC ERROR: PCSetUp() line 832 in >>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c >>> >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in >>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>> >>>>> [0]PETSC ERROR: KSPSolve() line 385 in >>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>> >>>>> >>> >>>>> >>> >>>>> John >>> >>>>> >>> >>>>> >>> >>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams < >>> mark.adams at columbia.edu> wrote: >>> >>>>> >>> >>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: >>> >>>>> >>> >>>>>> Mark, >>> >>>>>> >>> >>>>>> The matrix is asymmetric. Does this require the setting of an >>> option? >>> >>>>> >>> >>>>> Yes: -pc_gamg_sym_graph >>> >>>>> >>> >>>>> Mark >>> >>>>> >>> >>>>>> I pulled petsc-dev this morning, so I should have (at least close >>> to) the latest code. >>> >>>>>> >>> >>>>>> John >>> >>>>>> >>> >>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams < >>> mark.adams at columbia.edu> wrote: >>> >>>>>> >>> >>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >>> >>>>>> >>> >>>>>>> I'm getting the following error when using GAMG. >>> >>>>>>> >>> >>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion >>> `sgid==-1' failed. >>> >>>>>> >>> >>>>>> Is it possible that your matrix is structurally asymmetric? >>> >>>>>> >>> >>>>>> This code is evolving fast and so you will need to move to the >>> dev version if you are not already using it. (I think I fixed a bug that >>> hit this assert). >>> >>>>>> >>> >>>>>>> >>> >>>>>>> When I try to alter the type of aggregation at the command line >>> using -pc_gamg_type pa, I'm getting >>> >>>>>>> >>> >>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error >>> Message ------------------------------------ >>> >>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing >>> external package needed for type: >>> >>>>>>> see >>> http://www.mcs.anl.gov/petsc/documentation/installation.html#external! >>> >>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given! >>> >>>>>>> >>> >>>>>>> Has there been a change in the aggregation options? I just >>> pulled petsc-dev this morning. >>> >>>>>>> >>> >>>>>> >>> >>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg for >>> now. >>> >>>>>> >>> >>>>>> Mark >>> >>>>>> >>> >>>>>>> John >>> >>>>>> >>> >>>>>> >>> >>>>> >>> >>>>> >>> >>>> >>> >>>> >>> >>> >>> >>> >>> >> >>> >> >>> > >>> > >>> >>> >> >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- [0]PCSetUp_GAMG level 0 N=58507, n data rows=1, n data cols=1, nnz/row (ave)=1, np=4 [0]scaleFilterGraph 80.6997% nnz after filtering, with threshold 0.05, 6.43289 nnz ave. [0]maxIndSetAgg removed 0 of 13276 vertices. [0]PCGAMGprolongator_AGG New grid 7604 nodes [0]PCSetUp_GAMG 1) N=7604, n data cols=1, nnz/row (ave)=6, 4 active pes [0]scaleFilterGraph 93.3889% nnz after filtering, with threshold 0.05, 6.58382 nnz ave. [0]maxIndSetAgg removed 0 of 1730 vertices. [0]PCGAMGprolongator_AGG New grid 895 nodes [0]PCSetUp_GAMG 2) N=895, n data cols=1, nnz/row (ave)=5, 4 active pes [0]scaleFilterGraph 91.3043% nnz after filtering, with threshold 0.05, 5.54359 nnz ave. [0]maxIndSetAgg removed 0 of 195 vertices. [0]PCGAMGprolongator_AGG New grid 139 nodes [0]createLevel aggregate processors: npe: 4 --> 1, neq=139 [0]PCSetUp_GAMG 3) N=139, n data cols=1, nnz/row (ave)=4, 1 active pes [0]PCSetUp_GAMG 4 levels, grid compexity = 1.15398 PCSetUp_GAMG PC setup max eigen=1.875451e+00 min=2.696859e-02 PCSetUp_GAMG PC setup max eigen=1.742779e+00 min=1.770445e-02 PCSetUp_GAMG PC setup max eigen=1.948789e+00 min=3.955759e-02 Residual norms for pres_ solve. 0 KSP Residual norm 5.141026204165e+03 2 KSP Residual norm 1.284239020662e+03 4 KSP Residual norm 9.967668419130e+02 6 KSP Residual norm 5.783283077358e+02 8 KSP Residual norm 5.566458446657e+02 10 KSP Residual norm 3.783499634403e+02 12 KSP Residual norm 3.053155044665e+02 14 KSP Residual norm 2.726314646644e+02 16 KSP Residual norm 1.960952298636e+02 18 KSP Residual norm 1.865981998220e+02 20 KSP Residual norm 1.554042318839e+02 22 KSP Residual norm 1.497631349479e+02 24 KSP Residual norm 4.598812937768e+02 26 KSP Residual norm 1.678435395838e+02 28 KSP Residual norm 2.040759429929e+02 30 KSP Residual norm 2.288608663089e+02 32 KSP Residual norm 2.733802693469e+02 34 KSP Residual norm 1.640058147647e+02 36 KSP Residual norm 1.172045953032e+02 38 KSP Residual norm 1.157357240159e+02 40 KSP Residual norm 1.122520337943e+02 42 KSP Residual norm 9.825251767781e+01 44 KSP Residual norm 1.208285004589e+02 46 KSP Residual norm 6.786930614669e+01 48 KSP Residual norm 4.590732680097e+01 50 KSP Residual norm 3.841598045058e+01 52 KSP Residual norm 2.625665852109e+01 54 KSP Residual norm 1.941467729850e+01 56 KSP Residual norm 1.365056340101e+01 58 KSP Residual norm 6.773865959980e+00 60 KSP Residual norm 5.087803263599e+00 62 KSP Residual norm 4.056350834310e+00 64 KSP Residual norm 2.885709078944e+00 66 KSP Residual norm 3.319001782669e+00 68 KSP Residual norm 2.786670792779e+00 70 KSP Residual norm 5.516775828892e+00 72 KSP Residual norm 4.496277080798e+00 74 KSP Residual norm 1.004981312474e-01 76 KSP Residual norm 5.360056134540e-02 78 KSP Residual norm 4.464985177773e-02 80 KSP Residual norm 3.483114821741e-02 82 KSP Residual norm 1.514538931552e-02 84 KSP Residual norm 1.223770508292e-02 86 KSP Residual norm 1.423206397892e-02 88 KSP Residual norm 9.108998551955e-03 90 KSP Residual norm 7.266553500351e-03 92 KSP Residual norm 9.600829160875e-04 94 KSP Residual norm 2.730957811323e-04 96 KSP Residual norm 1.481227697383e-04 98 KSP Residual norm 8.426581071258e-05 100 KSP Residual norm 7.380041287915e-05 102 KSP Residual norm 5.553587538116e-05 104 KSP Residual norm 4.563527252379e-05 106 KSP Residual norm 3.110759145615e-05 108 KSP Residual norm 1.785372995034e-05 110 KSP Residual norm 1.720065891767e-05 112 KSP Residual norm 1.213804492876e-05 114 KSP Residual norm 2.835812681505e-05 116 KSP Residual norm 8.426402773025e-06 118 KSP Residual norm 3.833374276573e-06 120 KSP Residual norm 4.812273681701e-06 122 KSP Residual norm 1.117614658717e-06 124 KSP Residual norm 7.404391742242e-07 126 KSP Residual norm 5.129803443305e-07 128 KSP Residual norm 4.260359119617e-07 130 KSP Residual norm 4.109770534864e-07 132 KSP Residual norm 5.183915037437e-07 134 KSP Residual norm 1.091650015290e-06 136 KSP Residual norm 1.271770801341e-07 138 KSP Residual norm 2.534603085234e-08 140 KSP Residual norm 2.290908824239e-08 142 KSP Residual norm 5.846953963288e-09 KSP Object:(pres_) 4 MPI processes type: bcgsl BCGSL: Ell = 2 BCGSL: Delta = 0 maximum iterations=5000 tolerances: relative=1e-12, absolute=1e-50, divergence=10000 left preconditioning diagonally scaled system has attached null space using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object:(pres_) 4 MPI processes type: gamg MG: type is MULTIPLICATIVE, levels=4 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (pres_mg_coarse_) 4 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (pres_mg_coarse_) 4 MPI processes type: bjacobi block Jacobi: number of blocks = 4 Local solve info for each block is in the following KSP and PC objects: [0] number of local blocks = 1, first local block number = 0 [0] local block number 0 KSP Object: (pres_mg_coarse_sub_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (pres_mg_coarse_sub_) 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 5, needed 1.66568 Factored matrix follows: Matrix Object: 1 MPI processes type: seqaij rows=139, cols=139 package used to perform factorization: petsc total: nonzeros=1131, allocated nonzeros=1131 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=139, cols=139 total: nonzeros=679, allocated nonzeros=679 total number of mallocs used during MatSetValues calls =0 not using I-node routines - - - - - - - - - - - - - - - - - - [1] number of local blocks = 1, first local block number = 1 [1] local block number 0 KSP Object: (pres_mg_coarse_sub_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (pres_mg_coarse_sub_) 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 5, needed 0 Factored matrix follows: Matrix Object: 1 MPI processes type: seqaij rows=0, cols=0 package used to perform factorization: petsc total: nonzeros=1, allocated nonzeros=1 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=0, cols=0 total: nonzeros=0, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 not using I-node routines - - - - - - - - - - - - - - - - - - [2] number of local blocks = 1, first local block number = 2 [2] local block number 0 KSP Object: (pres_mg_coarse_sub_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (pres_mg_coarse_sub_) 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 5, needed 0 Factored matrix follows: Matrix Object: 1 MPI processes type: seqaij rows=0, cols=0 package used to perform factorization: petsc total: nonzeros=1, allocated nonzeros=1 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=0, cols=0 total: nonzeros=0, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 not using I-node routines - - - - - - - - - - - - - - - - - - [3] number of local blocks = 1, first local block number = 3 [3] local block number 0 KSP Object: (pres_mg_coarse_sub_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (pres_mg_coarse_sub_) 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 5, needed 0 Factored matrix follows: Matrix Object: 1 MPI processes type: seqaij rows=0, cols=0 package used to perform factorization: petsc total: nonzeros=1, allocated nonzeros=1 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=0, cols=0 total: nonzeros=0, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 not using I-node routines - - - - - - - - - - - - - - - - - - linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=139, cols=139 total: nonzeros=679, allocated nonzeros=679 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (pres_mg_levels_1_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_1_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=895, cols=895 total: nonzeros=4941, allocated nonzeros=4941 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (pres_mg_levels_2_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_2_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=7604, cols=7604 total: nonzeros=49366, allocated nonzeros=49366 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (pres_mg_levels_3_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_3_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=58507, cols=58507 total: nonzeros=383336, allocated nonzeros=675924 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=58507, cols=58507 total: nonzeros=383336, allocated nonzeros=675924 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines From mark.adams at columbia.edu Thu Mar 15 12:04:07 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Thu, 15 Mar 2012 13:04:07 -0400 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> Message-ID: <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> You also want: -pc_gamg_agg_nsmooths 1 You are running plain aggregation. If it is Poisson then smoothing is good. Is this problem singular? Can you try running ML with these parameters and see if its performance degrades? The ML implementation uses the PETSC infrastructure and uses a very similar algorithm to GAMG-SA. We should be able to get these two to match pretty well. Mark On Mar 15, 2012, at 12:21 PM, John Mousel wrote: > Mark, > > I ran with those options removed (see the run options listed below). Things actually got slightly worse. Now it's up to 142 iterations. I have attached the ksp_view output. > > -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale -ksp_diagonal_scale_fix -mg_levels_ksp_type richardson -mg_levels_pc_type sor -pc_gamg_verbose 1 > > > John > > > On Thu, Mar 15, 2012 at 10:55 AM, Mark F. Adams wrote: > John, can you run again with: -pc_gamg_verbose 1 > > And I would not use: -pc_mg_levels 4 -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 > > 1) I think -mg_coarse_ksp_type preonly and -mg_coarse_pc_sor_its 8 do not do what you think. I think this is the same as 1 iteration. I think you want 'richardson' not 'preonly'. > > 2) Why are you using sor as the coarse solver? If your problem is singular then you want to use as many levels as possible to get the coarse grid to be tiny. I'm pretty sure HYPRE ignores the coarse solver parameters. But ML uses them and it is converging well. > > 3) I would not specify the number of levels. GAMG, and I think the rest, have internal logic for stopping a the right level. If the coarse level is large and you use just 8 iterations of sor then convergence will suffer. > > Mark > > On Mar 15, 2012, at 11:13 AM, John Mousel wrote: > >> Mark, >> >> The changes pulled through this morning. I've run it with the options >> >> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale -ksp_diagonal_scale_fix -pc_mg_levels 4 -mg_levels_ksp_type richardson -mg_levels_pc_type sor -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 >> >> and it converges in the true residual, but it's not converging as fast as anticpated. The matrix arises from a non-symmetric discretization of the Poisson equation. The solve takes GAMG 114 iterations, whereas ML takes 24 iterations, BoomerAMG takes 22 iterations, and -ksp_type bcgsl -pc_type bjacobi -sub_pc_type ilu -sub_pc_factor_levels 4 takes around 170. I've attached the -ksp_view results for ML,GAMG, and HYPRE. I've attempted to make all the options the same on all levels for ML and GAMG. >> >> Any thoughts? >> >> John >> >> >> On Wed, Mar 14, 2012 at 6:04 PM, Mark F. Adams wrote: >> Humm, I see it with hg view (appended). >> >> Satish, my main repo looks hosed. I see this: >> >> ~/Codes/petsc-dev>hg update >> abort: crosses branches (merge branches or use --clean to discard changes) >> ~/Codes/petsc-dev>hg merge >> abort: branch 'default' has 3 heads - please merge with an explicit rev >> (run 'hg heads .' to see heads) >> ~/Codes/petsc-dev>hg heads >> changeset: 22496:8e2a98268179 >> tag: tip >> user: Barry Smith >> date: Wed Mar 14 16:42:25 2012 -0500 >> files: src/vec/is/interface/f90-custom/zindexf90.c src/vec/vec/interface/f90-custom/zvectorf90.c >> description: >> undoing manually changes I put in because Satish had a better fix >> >> >> changeset: 22492:bda4df63072d >> user: Mark F. Adams >> date: Wed Mar 14 17:39:52 2012 -0400 >> files: src/ksp/pc/impls/gamg/tools.c >> description: >> fix for unsymmetric matrices. >> >> >> changeset: 22469:b063baf366e4 >> user: Mark F. Adams >> date: Wed Mar 14 14:22:28 2012 -0400 >> files: src/ksp/pc/impls/gamg/tools.c >> description: >> added fix for preallocation for unsymetric matrices. >> >> Mark >> >> my 'hg view' on my merge repo: >> >> Revision: 22492 >> Branch: default >> Author: Mark F. Adams 2012-03-14 17:39:52 >> Committer: Mark F. Adams 2012-03-14 17:39:52 >> Tags: tip >> Parent: 22491:451bbbd291c2 (Small fixes to the BT linesearch) >> >> fix for unsymmetric matrices. >> >> >> ------------------------ src/ksp/pc/impls/gamg/tools.c ------------------------ >> @@ -103,7 +103,7 @@ >> PetscErrorCode ierr; >> PetscInt Istart,Iend,Ii,jj,ncols,nnz0,nnz1, NN, MM, nloc; >> PetscMPIInt mype, npe; >> - Mat Gmat = *a_Gmat, tGmat; >> + Mat Gmat = *a_Gmat, tGmat, matTrans; >> MPI_Comm wcomm = ((PetscObject)Gmat)->comm; >> const PetscScalar *vals; >> const PetscInt *idx; >> @@ -127,6 +127,10 @@ >> ierr = MatDiagonalScale( Gmat, diag, diag ); CHKERRQ(ierr); >> ierr = VecDestroy( &diag ); CHKERRQ(ierr); >> >> + if( symm ) { >> + ierr = MatTranspose( Gmat, MAT_INITIAL_MATRIX, &matTrans ); CHKERRQ(ierr); >> + } >> + >> /* filter - dup zeros out matrix */ >> ierr = PetscMalloc( nloc*sizeof(PetscInt), &d_nnz ); CHKERRQ(ierr); >> ierr = PetscMalloc( nloc*sizeof(PetscInt), &o_nnz ); CHKERRQ(ierr); >> @@ -135,6 +139,12 @@ >> d_nnz[jj] = ncols; >> o_nnz[jj] = ncols; >> ierr = MatRestoreRow(Gmat,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >> + if( symm ) { >> + ierr = MatGetRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >> + d_nnz[jj] += ncols; >> + o_nnz[jj] += ncols; >> + ierr = MatRestoreRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >> + } >> if( d_nnz[jj] > nloc ) d_nnz[jj] = nloc; >> if( o_nnz[jj] > (MM-nloc) ) o_nnz[jj] = MM - nloc; >> } >> @@ -142,6 +152,9 @@ >> CHKERRQ(ierr); >> ierr = PetscFree( d_nnz ); CHKERRQ(ierr); >> ierr = PetscFree( o_nnz ); CHKERRQ(ierr); >> + if( symm ) { >> + ierr = MatDestroy( &matTrans ); CHKERRQ(ierr); >> + } >> >> >> >> >> On Mar 14, 2012, at 5:53 PM, John Mousel wrote: >> >>> Mark, >>> >>> No change. Can you give me the location that you patched so I can check to make sure it pulled? >>> I don't see it on the petsc-dev change log. >>> >>> John >>> >>> On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams wrote: >>> John, I've committed these changes, give a try. >>> >>> Mark >>> >>> On Mar 14, 2012, at 3:46 PM, Satish Balay wrote: >>> >>> > This is the usual merge [with uncommited changes] issue. >>> > >>> > You could use 'hg shelf' extension to shelve your local changes and >>> > then do a merge [as Sean would suggest] - or do the merge in a >>> > separate/clean clone [I normally do this..] >>> > >>> > i.e >>> > cd ~/Codes >>> > hg clone petsc-dev petsc-dev-merge >>> > cd petsc-dev-merge >>> > hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just to be sure, look for latest chagnes before merge.. >>> > hg merge >>> > hg commit >>> > hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev >>> > >>> > [now update your petsc-dev to latest] >>> > cd ~/Codes/petsc-dev >>> > hg pull >>> > hg update >>> > >>> > Satish >>> > >>> > On Wed, 14 Mar 2012, Mark F. Adams wrote: >>> > >>> >> Great, that seems to work. >>> >> >>> >> I did a 'hg commit tools.c' >>> >> >>> >> and I want to push this file only. I guess its the only thing in the change set so 'hg push' should be fine. But I see this: >>> >> >>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update >>> >> abort: crosses branches (merge branches or use --clean to discard changes) >>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge >>> >> abort: outstanding uncommitted changes (use 'hg status' to list changes) >>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status >>> >> M include/petscmat.h >>> >> M include/private/matimpl.h >>> >> M src/ksp/pc/impls/gamg/agg.c >>> >> M src/ksp/pc/impls/gamg/gamg.c >>> >> M src/ksp/pc/impls/gamg/gamg.h >>> >> M src/ksp/pc/impls/gamg/geo.c >>> >> M src/mat/coarsen/coarsen.c >>> >> M src/mat/coarsen/impls/hem/hem.c >>> >> M src/mat/coarsen/impls/mis/mis.c >>> >> >>> >> Am I ready to do a push? >>> >> >>> >> Thanks, >>> >> Mark >>> >> >>> >> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: >>> >> >>> >>> If commit is the last hg operation that you've done - then 'hg rollback' would undo this commit. >>> >>> >>> >>> Satish >>> >>> >>> >>> On Wed, 14 Mar 2012, Mark F. Adams wrote: >>> >>> >>> >>>> Damn, I'm not preallocating the graph perfectly for unsymmetric matrices and PETSc now dies on this. >>> >>>> >>> >>>> I have a fix but I committed it with other changes that I do not want to commit. The changes are all in one file so I should be able to just commit this file. >>> >>>> >>> >>>> Anyone know how to delete a commit? >>> >>>> >>> >>>> I've tried: >>> >>>> >>> >>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f >>> >>>> hg: unknown command 'strip' >>> >>>> 'strip' is provided by the following extension: >>> >>>> >>> >>>> mq manage a stack of patches >>> >>>> >>> >>>> use "hg help extensions" for information on enabling extensions >>> >>>> >>> >>>> But have not figured out how to load extensions. >>> >>>> >>> >>>> Mark >>> >>>> >>> >>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: >>> >>>> >>> >>>>> Mark, >>> >>>>> >>> >>>>> I have a non-symmetric matrix. I am running with the following options. >>> >>>>> >>> >>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual >>> >>>>> >>> >>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new malloc error: >>> >>>>> >>> >>>>> >>> >>>>> 0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> >>>>> [0]PETSC ERROR: Argument out of range! >>> >>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! >>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> >>>>> [0]PETSC ERROR: Petsc Development HG revision: 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 -0500 >>> >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> >>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 >>> >>>>> [0]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib >>> >>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 >>> >>>>> [0]PETSC ERROR: Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug >>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> >>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >>> >>>>> [0]PETSC ERROR: MatSetValues() line 1141 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c >>> >>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c >>> >>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c >>> >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>> >>>>> [0]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c >>> >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>> >>>>> [0]PETSC ERROR: KSPSolve() line 385 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>> >>>>> >>> >>>>> >>> >>>>> John >>> >>>>> >>> >>>>> >>> >>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams wrote: >>> >>>>> >>> >>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: >>> >>>>> >>> >>>>>> Mark, >>> >>>>>> >>> >>>>>> The matrix is asymmetric. Does this require the setting of an option? >>> >>>>> >>> >>>>> Yes: -pc_gamg_sym_graph >>> >>>>> >>> >>>>> Mark >>> >>>>> >>> >>>>>> I pulled petsc-dev this morning, so I should have (at least close to) the latest code. >>> >>>>>> >>> >>>>>> John >>> >>>>>> >>> >>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams wrote: >>> >>>>>> >>> >>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >>> >>>>>> >>> >>>>>>> I'm getting the following error when using GAMG. >>> >>>>>>> >>> >>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion `sgid==-1' failed. >>> >>>>>> >>> >>>>>> Is it possible that your matrix is structurally asymmetric? >>> >>>>>> >>> >>>>>> This code is evolving fast and so you will need to move to the dev version if you are not already using it. (I think I fixed a bug that hit this assert). >>> >>>>>> >>> >>>>>>> >>> >>>>>>> When I try to alter the type of aggregation at the command line using -pc_gamg_type pa, I'm getting >>> >>>>>>> >>> >>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> >>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type: >>> >>>>>>> see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! >>> >>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given! >>> >>>>>>> >>> >>>>>>> Has there been a change in the aggregation options? I just pulled petsc-dev this morning. >>> >>>>>>> >>> >>>>>> >>> >>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg for now. >>> >>>>>> >>> >>>>>> Mark >>> >>>>>> >>> >>>>>>> John >>> >>>>>> >>> >>>>>> >>> >>>>> >>> >>>>> >>> >>>> >>> >>>> >>> >>> >>> >>> >>> >> >>> >> >>> > >>> > >>> >>> >> >> >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Thu Mar 15 12:19:43 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Thu, 15 Mar 2012 13:19:43 -0400 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> Message-ID: On Mar 15, 2012, at 11:24 AM, Jed Brown wrote: > On Wed, Mar 14, 2012 at 11:27, Mark F. Adams wrote: >> The matrix is asymmetric. Does this require the setting of an option? > > Yes: -pc_gamg_sym_graph > > Mark, I don't like this name. First, it's imperative in the sense that it's describing a procedure for fixing something rather than declarative (stating some property of the problem). Second, there is already a way to state symmetry, MatSetOption(): > Good point, I've added: ierr = MatIsSymmetricKnown(Amat, &set, &flg); CHKERRQ(ierr); symm = pc_gamg_agg->sym_graph || !(set && flg); > Options Describing Matrix Structure: > + MAT_SPD - symmetric positive definite > - MAT_SYMMETRIC - symmetric in terms of both structure and value > . MAT_HERMITIAN - transpose is the complex conjugation > . MAT_STRUCTURALLY_SYMMETRIC - symmetric nonzero structure > - MAT_SYMMETRY_ETERNAL - if you would like the symmetry/Hermitian flag > you set to be kept with all future use of the matrix > including after MatAssemblyBegin/End() which could > potentially change the symmetry structure, i.e. you > KNOW the matrix will ALWAYS have the property you set. > > > Is there a reason you can't just use this information? If you have multiple methods for handling nonsymmetric structure, then I would suggest using PetscOptionsList() or PetscOptionsEnum() to select which to use. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.mousel at gmail.com Thu Mar 15 15:24:32 2012 From: john.mousel at gmail.com (John Mousel) Date: Thu, 15 Mar 2012 15:24:32 -0500 Subject: [petsc-users] GAMG issue In-Reply-To: <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> Message-ID: Mark, Running without the options you mentioned before leads to slightly worse performance (175 iterations). I have not been able to get run coarse grid solve to work with LU while running ML. It keeps experiencing a zero pivot, and all the combinations of shifting i've tried haven't lead me anywhere, hence the SOR on the course grid. Also, the ML manual suggests limiting the number of levels to 3 or 4 and performing a few sweeps of an iterative method as opposed to a direct solve. John On Thu, Mar 15, 2012 at 12:04 PM, Mark F. Adams wrote: > You also want: -pc_gamg_agg_nsmooths 1 > > You are running plain aggregation. If it is Poisson then smoothing is > good. > > Is this problem singular? Can you try running ML with these parameters > and see if its performance degrades? The ML implementation uses the PETSC > infrastructure and uses a very similar algorithm to GAMG-SA. We should be > able to get these two to match pretty well. > > Mark > > > On Mar 15, 2012, at 12:21 PM, John Mousel wrote: > > Mark, > > I ran with those options removed (see the run options listed below). > Things actually got slightly worse. Now it's up to 142 iterations. I have > attached the ksp_view output. > > -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale > -ksp_diagonal_scale_fix -mg_levels_ksp_type richardson -mg_levels_pc_type > sor -pc_gamg_verbose 1 > > > John > > > On Thu, Mar 15, 2012 at 10:55 AM, Mark F. Adams wrote: > >> John, can you run again with: -pc_gamg_verbose 1 >> >> And I would not use: -pc_mg_levels 4 -mg_coarse_ksp_type preonly >> -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 >> >> 1) I think -mg_coarse_ksp_type preonly and -mg_coarse_pc_sor_its 8 do not >> do what you think. I think this is the same as 1 iteration. I think you >> want 'richardson' not 'preonly'. >> >> 2) Why are you using sor as the coarse solver? If your problem is >> singular then you want to use as many levels as possible to get the coarse >> grid to be tiny. I'm pretty sure HYPRE ignores the coarse solver >> parameters. But ML uses them and it is converging well. >> >> 3) I would not specify the number of levels. GAMG, and I think the rest, >> have internal logic for stopping a the right level. If the coarse level is >> large and you use just 8 iterations of sor then convergence will suffer. >> >> Mark >> >> On Mar 15, 2012, at 11:13 AM, John Mousel wrote: >> >> Mark, >> >> The changes pulled through this morning. I've run it with the options >> >> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale >> -ksp_diagonal_scale_fix -pc_mg_levels 4 -mg_levels_ksp_type richardson >> -mg_levels_pc_type sor -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor >> -mg_coarse_pc_sor_its 8 >> >> and it converges in the true residual, but it's not converging as fast as >> anticpated. The matrix arises from a non-symmetric discretization of the >> Poisson equation. The solve takes GAMG 114 iterations, whereas ML takes 24 >> iterations, BoomerAMG takes 22 iterations, and -ksp_type bcgsl -pc_type >> bjacobi -sub_pc_type ilu -sub_pc_factor_levels 4 takes around 170. I've >> attached the -ksp_view results for ML,GAMG, and HYPRE. I've attempted to >> make all the options the same on all levels for ML and GAMG. >> >> Any thoughts? >> >> John >> >> >> On Wed, Mar 14, 2012 at 6:04 PM, Mark F. Adams wrote: >> >>> Humm, I see it with hg view (appended). >>> >>> Satish, my main repo looks hosed. I see this: >>> >>> ~/Codes/petsc-dev>hg update >>> abort: crosses branches (merge branches or use --clean to discard >>> changes) >>> ~/Codes/petsc-dev>hg merge >>> abort: branch 'default' has 3 heads - please merge with an explicit rev >>> (run 'hg heads .' to see heads) >>> ~/Codes/petsc-dev>hg heads >>> changeset: 22496:8e2a98268179 >>> tag: tip >>> user: Barry Smith >>> date: Wed Mar 14 16:42:25 2012 -0500 >>> files: src/vec/is/interface/f90-custom/zindexf90.c >>> src/vec/vec/interface/f90-custom/zvectorf90.c >>> description: >>> undoing manually changes I put in because Satish had a better fix >>> >>> >>> changeset: 22492:bda4df63072d >>> user: Mark F. Adams >>> date: Wed Mar 14 17:39:52 2012 -0400 >>> files: src/ksp/pc/impls/gamg/tools.c >>> description: >>> fix for unsymmetric matrices. >>> >>> >>> changeset: 22469:b063baf366e4 >>> user: Mark F. Adams >>> date: Wed Mar 14 14:22:28 2012 -0400 >>> files: src/ksp/pc/impls/gamg/tools.c >>> description: >>> added fix for preallocation for unsymetric matrices. >>> >>> Mark >>> >>> my 'hg view' on my merge repo: >>> >>> Revision: 22492 >>> Branch: default >>> Author: Mark F. Adams 2012-03-14 17:39:52 >>> Committer: Mark F. Adams 2012-03-14 17:39:52 >>> Tags: tip >>> Parent: 22491:451bbbd291c2 (Small fixes to the BT linesearch) >>> >>> fix for unsymmetric matrices. >>> >>> >>> ------------------------ src/ksp/pc/impls/gamg/tools.c >>> ------------------------ >>> @@ -103,7 +103,7 @@ >>> PetscErrorCode ierr; >>> PetscInt Istart,Iend,Ii,jj,ncols,nnz0,nnz1, NN, MM, nloc; >>> PetscMPIInt mype, npe; >>> - Mat Gmat = *a_Gmat, tGmat; >>> + Mat Gmat = *a_Gmat, tGmat, matTrans; >>> MPI_Comm wcomm = ((PetscObject)Gmat)->comm; >>> const PetscScalar *vals; >>> const PetscInt *idx; >>> @@ -127,6 +127,10 @@ >>> ierr = MatDiagonalScale( Gmat, diag, diag ); CHKERRQ(ierr); >>> ierr = VecDestroy( &diag ); CHKERRQ(ierr); >>> >>> + if( symm ) { >>> + ierr = MatTranspose( Gmat, MAT_INITIAL_MATRIX, &matTrans ); >>> CHKERRQ(ierr); >>> + } >>> + >>> /* filter - dup zeros out matrix */ >>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &d_nnz ); CHKERRQ(ierr); >>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &o_nnz ); CHKERRQ(ierr); >>> @@ -135,6 +139,12 @@ >>> d_nnz[jj] = ncols; >>> o_nnz[jj] = ncols; >>> ierr = MatRestoreRow(Gmat,Ii,&ncols,PETSC_NULL,PETSC_NULL); >>> CHKERRQ(ierr); >>> + if( symm ) { >>> + ierr = MatGetRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); >>> CHKERRQ(ierr); >>> + d_nnz[jj] += ncols; >>> + o_nnz[jj] += ncols; >>> + ierr = MatRestoreRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); >>> CHKERRQ(ierr); >>> + } >>> if( d_nnz[jj] > nloc ) d_nnz[jj] = nloc; >>> if( o_nnz[jj] > (MM-nloc) ) o_nnz[jj] = MM - nloc; >>> } >>> @@ -142,6 +152,9 @@ >>> CHKERRQ(ierr); >>> ierr = PetscFree( d_nnz ); CHKERRQ(ierr); >>> ierr = PetscFree( o_nnz ); CHKERRQ(ierr); >>> + if( symm ) { >>> + ierr = MatDestroy( &matTrans ); CHKERRQ(ierr); >>> + } >>> >>> >>> >>> >>> On Mar 14, 2012, at 5:53 PM, John Mousel wrote: >>> >>> Mark, >>> >>> No change. Can you give me the location that you patched so I can check >>> to make sure it pulled? >>> I don't see it on the petsc-dev change log. >>> >>> John >>> >>> On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams wrote: >>> >>>> John, I've committed these changes, give a try. >>>> >>>> Mark >>>> >>>> On Mar 14, 2012, at 3:46 PM, Satish Balay wrote: >>>> >>>> > This is the usual merge [with uncommited changes] issue. >>>> > >>>> > You could use 'hg shelf' extension to shelve your local changes and >>>> > then do a merge [as Sean would suggest] - or do the merge in a >>>> > separate/clean clone [I normally do this..] >>>> > >>>> > i.e >>>> > cd ~/Codes >>>> > hg clone petsc-dev petsc-dev-merge >>>> > cd petsc-dev-merge >>>> > hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just to >>>> be sure, look for latest chagnes before merge.. >>>> > hg merge >>>> > hg commit >>>> > hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev >>>> > >>>> > [now update your petsc-dev to latest] >>>> > cd ~/Codes/petsc-dev >>>> > hg pull >>>> > hg update >>>> > >>>> > Satish >>>> > >>>> > On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>> > >>>> >> Great, that seems to work. >>>> >> >>>> >> I did a 'hg commit tools.c' >>>> >> >>>> >> and I want to push this file only. I guess its the only thing in >>>> the change set so 'hg push' should be fine. But I see this: >>>> >> >>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update >>>> >> abort: crosses branches (merge branches or use --clean to discard >>>> changes) >>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge >>>> >> abort: outstanding uncommitted changes (use 'hg status' to list >>>> changes) >>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status >>>> >> M include/petscmat.h >>>> >> M include/private/matimpl.h >>>> >> M src/ksp/pc/impls/gamg/agg.c >>>> >> M src/ksp/pc/impls/gamg/gamg.c >>>> >> M src/ksp/pc/impls/gamg/gamg.h >>>> >> M src/ksp/pc/impls/gamg/geo.c >>>> >> M src/mat/coarsen/coarsen.c >>>> >> M src/mat/coarsen/impls/hem/hem.c >>>> >> M src/mat/coarsen/impls/mis/mis.c >>>> >> >>>> >> Am I ready to do a push? >>>> >> >>>> >> Thanks, >>>> >> Mark >>>> >> >>>> >> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: >>>> >> >>>> >>> If commit is the last hg operation that you've done - then 'hg >>>> rollback' would undo this commit. >>>> >>> >>>> >>> Satish >>>> >>> >>>> >>> On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>> >>> >>>> >>>> Damn, I'm not preallocating the graph perfectly for unsymmetric >>>> matrices and PETSc now dies on this. >>>> >>>> >>>> >>>> I have a fix but I committed it with other changes that I do not >>>> want to commit. The changes are all in one file so I should be able to >>>> just commit this file. >>>> >>>> >>>> >>>> Anyone know how to delete a commit? >>>> >>>> >>>> >>>> I've tried: >>>> >>>> >>>> >>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f >>>> >>>> hg: unknown command 'strip' >>>> >>>> 'strip' is provided by the following extension: >>>> >>>> >>>> >>>> mq manage a stack of patches >>>> >>>> >>>> >>>> use "hg help extensions" for information on enabling extensions >>>> >>>> >>>> >>>> But have not figured out how to load extensions. >>>> >>>> >>>> >>>> Mark >>>> >>>> >>>> >>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: >>>> >>>> >>>> >>>>> Mark, >>>> >>>>> >>>> >>>>> I have a non-symmetric matrix. I am running with the following >>>> options. >>>> >>>>> >>>> >>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual >>>> >>>>> >>>> >>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new malloc >>>> error: >>>> >>>>> >>>> >>>>> >>>> >>>>> 0]PETSC ERROR: --------------------- Error Message >>>> ------------------------------------ >>>> >>>>> [0]PETSC ERROR: Argument out of range! >>>> >>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! >>>> >>>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> >>>>> [0]PETSC ERROR: Petsc Development HG revision: >>>> 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 >>>> -0500 >>>> >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble >>>> shooting. >>>> >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> >>>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> >>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named >>>> wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 >>>> >>>>> [0]PETSC ERROR: Libraries linked from >>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib >>>> >>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 >>>> >>>>> [0]PETSC ERROR: Configure options --download-blacs=1 >>>> --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 >>>> --download-parmetis=1 --download-scalapack=1 >>>> --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc >>>> --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort >>>> PETSC_ARCH=linux-debug >>>> >>>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> >>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in >>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >>>> >>>>> [0]PETSC ERROR: MatSetValues() line 1141 in >>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c >>>> >>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in >>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c >>>> >>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in >>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c >>>> >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in >>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>>> >>>>> [0]PETSC ERROR: PCSetUp() line 832 in >>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c >>>> >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in >>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>> >>>>> [0]PETSC ERROR: KSPSolve() line 385 in >>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>> >>>>> >>>> >>>>> >>>> >>>>> John >>>> >>>>> >>>> >>>>> >>>> >>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams < >>>> mark.adams at columbia.edu> wrote: >>>> >>>>> >>>> >>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: >>>> >>>>> >>>> >>>>>> Mark, >>>> >>>>>> >>>> >>>>>> The matrix is asymmetric. Does this require the setting of an >>>> option? >>>> >>>>> >>>> >>>>> Yes: -pc_gamg_sym_graph >>>> >>>>> >>>> >>>>> Mark >>>> >>>>> >>>> >>>>>> I pulled petsc-dev this morning, so I should have (at least >>>> close to) the latest code. >>>> >>>>>> >>>> >>>>>> John >>>> >>>>>> >>>> >>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams < >>>> mark.adams at columbia.edu> wrote: >>>> >>>>>> >>>> >>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >>>> >>>>>> >>>> >>>>>>> I'm getting the following error when using GAMG. >>>> >>>>>>> >>>> >>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: >>>> Assertion `sgid==-1' failed. >>>> >>>>>> >>>> >>>>>> Is it possible that your matrix is structurally asymmetric? >>>> >>>>>> >>>> >>>>>> This code is evolving fast and so you will need to move to the >>>> dev version if you are not already using it. (I think I fixed a bug that >>>> hit this assert). >>>> >>>>>> >>>> >>>>>>> >>>> >>>>>>> When I try to alter the type of aggregation at the command line >>>> using -pc_gamg_type pa, I'm getting >>>> >>>>>>> >>>> >>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error >>>> Message ------------------------------------ >>>> >>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or >>>> missing external package needed for type: >>>> >>>>>>> see >>>> http://www.mcs.anl.gov/petsc/documentation/installation.html#external! >>>> >>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given! >>>> >>>>>>> >>>> >>>>>>> Has there been a change in the aggregation options? I just >>>> pulled petsc-dev this morning. >>>> >>>>>>> >>>> >>>>>> >>>> >>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg for >>>> now. >>>> >>>>>> >>>> >>>>>> Mark >>>> >>>>>> >>>> >>>>>>> John >>>> >>>>>> >>>> >>>>>> >>>> >>>>> >>>> >>>>> >>>> >>>> >>>> >>>> >>>> >>> >>>> >>> >>>> >> >>>> >> >>>> > >>>> > >>>> >>>> >>> >>> >> >> >> >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Mar 15 15:54:08 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 15 Mar 2012 15:54:08 -0500 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> Message-ID: On Thu, Mar 15, 2012 at 3:24 PM, John Mousel wrote: > Mark, > > Running without the options you mentioned before leads to slightly worse > performance (175 iterations). > I have not been able to get run coarse grid solve to work with LU while > running ML. It keeps experiencing a zero pivot, and all the combinations of > shifting i've tried haven't lead me anywhere, hence the SOR on the course > grid. Also, the ML manual suggests limiting the number of levels to 3 or 4 > and performing a few sweeps of an iterative method as opposed to a direct > solve. > That is suspicious behavior for LU. I now suspect something is wrong with this system, namely it is rank deficient or the aggregates are and you are not seeing real convergence. Can you add -ksp_monitor_true_residual to make sure the solver is actually converging. Matt > John > > On Thu, Mar 15, 2012 at 12:04 PM, Mark F. Adams wrote: > >> You also want: -pc_gamg_agg_nsmooths 1 >> >> You are running plain aggregation. If it is Poisson then smoothing is >> good. >> >> Is this problem singular? Can you try running ML with these parameters >> and see if its performance degrades? The ML implementation uses the PETSC >> infrastructure and uses a very similar algorithm to GAMG-SA. We should be >> able to get these two to match pretty well. >> >> Mark >> >> >> On Mar 15, 2012, at 12:21 PM, John Mousel wrote: >> >> Mark, >> >> I ran with those options removed (see the run options listed below). >> Things actually got slightly worse. Now it's up to 142 iterations. I have >> attached the ksp_view output. >> >> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale >> -ksp_diagonal_scale_fix -mg_levels_ksp_type richardson -mg_levels_pc_type >> sor -pc_gamg_verbose 1 >> >> >> John >> >> >> On Thu, Mar 15, 2012 at 10:55 AM, Mark F. Adams wrote: >> >>> John, can you run again with: -pc_gamg_verbose 1 >>> >>> And I would not use: -pc_mg_levels 4 -mg_coarse_ksp_type preonly >>> -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 >>> >>> 1) I think -mg_coarse_ksp_type preonly and -mg_coarse_pc_sor_its 8 do >>> not do what you think. I think this is the same as 1 iteration. I think >>> you want 'richardson' not 'preonly'. >>> >>> 2) Why are you using sor as the coarse solver? If your problem is >>> singular then you want to use as many levels as possible to get the coarse >>> grid to be tiny. I'm pretty sure HYPRE ignores the coarse solver >>> parameters. But ML uses them and it is converging well. >>> >>> 3) I would not specify the number of levels. GAMG, and I think the >>> rest, have internal logic for stopping a the right level. If the coarse >>> level is large and you use just 8 iterations of sor then convergence will >>> suffer. >>> >>> Mark >>> >>> On Mar 15, 2012, at 11:13 AM, John Mousel wrote: >>> >>> Mark, >>> >>> The changes pulled through this morning. I've run it with the options >>> >>> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale >>> -ksp_diagonal_scale_fix -pc_mg_levels 4 -mg_levels_ksp_type richardson >>> -mg_levels_pc_type sor -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor >>> -mg_coarse_pc_sor_its 8 >>> >>> and it converges in the true residual, but it's not converging as fast >>> as anticpated. The matrix arises from a non-symmetric discretization of the >>> Poisson equation. The solve takes GAMG 114 iterations, whereas ML takes 24 >>> iterations, BoomerAMG takes 22 iterations, and -ksp_type bcgsl -pc_type >>> bjacobi -sub_pc_type ilu -sub_pc_factor_levels 4 takes around 170. I've >>> attached the -ksp_view results for ML,GAMG, and HYPRE. I've attempted to >>> make all the options the same on all levels for ML and GAMG. >>> >>> Any thoughts? >>> >>> John >>> >>> >>> On Wed, Mar 14, 2012 at 6:04 PM, Mark F. Adams wrote: >>> >>>> Humm, I see it with hg view (appended). >>>> >>>> Satish, my main repo looks hosed. I see this: >>>> >>>> ~/Codes/petsc-dev>hg update >>>> abort: crosses branches (merge branches or use --clean to discard >>>> changes) >>>> ~/Codes/petsc-dev>hg merge >>>> abort: branch 'default' has 3 heads - please merge with an explicit rev >>>> (run 'hg heads .' to see heads) >>>> ~/Codes/petsc-dev>hg heads >>>> changeset: 22496:8e2a98268179 >>>> tag: tip >>>> user: Barry Smith >>>> date: Wed Mar 14 16:42:25 2012 -0500 >>>> files: src/vec/is/interface/f90-custom/zindexf90.c >>>> src/vec/vec/interface/f90-custom/zvectorf90.c >>>> description: >>>> undoing manually changes I put in because Satish had a better fix >>>> >>>> >>>> changeset: 22492:bda4df63072d >>>> user: Mark F. Adams >>>> date: Wed Mar 14 17:39:52 2012 -0400 >>>> files: src/ksp/pc/impls/gamg/tools.c >>>> description: >>>> fix for unsymmetric matrices. >>>> >>>> >>>> changeset: 22469:b063baf366e4 >>>> user: Mark F. Adams >>>> date: Wed Mar 14 14:22:28 2012 -0400 >>>> files: src/ksp/pc/impls/gamg/tools.c >>>> description: >>>> added fix for preallocation for unsymetric matrices. >>>> >>>> Mark >>>> >>>> my 'hg view' on my merge repo: >>>> >>>> Revision: 22492 >>>> Branch: default >>>> Author: Mark F. Adams 2012-03-14 17:39:52 >>>> Committer: Mark F. Adams 2012-03-14 17:39:52 >>>> Tags: tip >>>> Parent: 22491:451bbbd291c2 (Small fixes to the BT linesearch) >>>> >>>> fix for unsymmetric matrices. >>>> >>>> >>>> ------------------------ src/ksp/pc/impls/gamg/tools.c >>>> ------------------------ >>>> @@ -103,7 +103,7 @@ >>>> PetscErrorCode ierr; >>>> PetscInt Istart,Iend,Ii,jj,ncols,nnz0,nnz1, NN, MM, nloc; >>>> PetscMPIInt mype, npe; >>>> - Mat Gmat = *a_Gmat, tGmat; >>>> + Mat Gmat = *a_Gmat, tGmat, matTrans; >>>> MPI_Comm wcomm = ((PetscObject)Gmat)->comm; >>>> const PetscScalar *vals; >>>> const PetscInt *idx; >>>> @@ -127,6 +127,10 @@ >>>> ierr = MatDiagonalScale( Gmat, diag, diag ); CHKERRQ(ierr); >>>> ierr = VecDestroy( &diag ); CHKERRQ(ierr); >>>> >>>> + if( symm ) { >>>> + ierr = MatTranspose( Gmat, MAT_INITIAL_MATRIX, &matTrans ); >>>> CHKERRQ(ierr); >>>> + } >>>> + >>>> /* filter - dup zeros out matrix */ >>>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &d_nnz ); CHKERRQ(ierr); >>>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &o_nnz ); CHKERRQ(ierr); >>>> @@ -135,6 +139,12 @@ >>>> d_nnz[jj] = ncols; >>>> o_nnz[jj] = ncols; >>>> ierr = MatRestoreRow(Gmat,Ii,&ncols,PETSC_NULL,PETSC_NULL); >>>> CHKERRQ(ierr); >>>> + if( symm ) { >>>> + ierr = MatGetRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); >>>> CHKERRQ(ierr); >>>> + d_nnz[jj] += ncols; >>>> + o_nnz[jj] += ncols; >>>> + ierr = MatRestoreRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); >>>> CHKERRQ(ierr); >>>> + } >>>> if( d_nnz[jj] > nloc ) d_nnz[jj] = nloc; >>>> if( o_nnz[jj] > (MM-nloc) ) o_nnz[jj] = MM - nloc; >>>> } >>>> @@ -142,6 +152,9 @@ >>>> CHKERRQ(ierr); >>>> ierr = PetscFree( d_nnz ); CHKERRQ(ierr); >>>> ierr = PetscFree( o_nnz ); CHKERRQ(ierr); >>>> + if( symm ) { >>>> + ierr = MatDestroy( &matTrans ); CHKERRQ(ierr); >>>> + } >>>> >>>> >>>> >>>> >>>> On Mar 14, 2012, at 5:53 PM, John Mousel wrote: >>>> >>>> Mark, >>>> >>>> No change. Can you give me the location that you patched so I can check >>>> to make sure it pulled? >>>> I don't see it on the petsc-dev change log. >>>> >>>> John >>>> >>>> On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams >>> > wrote: >>>> >>>>> John, I've committed these changes, give a try. >>>>> >>>>> Mark >>>>> >>>>> On Mar 14, 2012, at 3:46 PM, Satish Balay wrote: >>>>> >>>>> > This is the usual merge [with uncommited changes] issue. >>>>> > >>>>> > You could use 'hg shelf' extension to shelve your local changes and >>>>> > then do a merge [as Sean would suggest] - or do the merge in a >>>>> > separate/clean clone [I normally do this..] >>>>> > >>>>> > i.e >>>>> > cd ~/Codes >>>>> > hg clone petsc-dev petsc-dev-merge >>>>> > cd petsc-dev-merge >>>>> > hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just to >>>>> be sure, look for latest chagnes before merge.. >>>>> > hg merge >>>>> > hg commit >>>>> > hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev >>>>> > >>>>> > [now update your petsc-dev to latest] >>>>> > cd ~/Codes/petsc-dev >>>>> > hg pull >>>>> > hg update >>>>> > >>>>> > Satish >>>>> > >>>>> > On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>>> > >>>>> >> Great, that seems to work. >>>>> >> >>>>> >> I did a 'hg commit tools.c' >>>>> >> >>>>> >> and I want to push this file only. I guess its the only thing in >>>>> the change set so 'hg push' should be fine. But I see this: >>>>> >> >>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update >>>>> >> abort: crosses branches (merge branches or use --clean to discard >>>>> changes) >>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge >>>>> >> abort: outstanding uncommitted changes (use 'hg status' to list >>>>> changes) >>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status >>>>> >> M include/petscmat.h >>>>> >> M include/private/matimpl.h >>>>> >> M src/ksp/pc/impls/gamg/agg.c >>>>> >> M src/ksp/pc/impls/gamg/gamg.c >>>>> >> M src/ksp/pc/impls/gamg/gamg.h >>>>> >> M src/ksp/pc/impls/gamg/geo.c >>>>> >> M src/mat/coarsen/coarsen.c >>>>> >> M src/mat/coarsen/impls/hem/hem.c >>>>> >> M src/mat/coarsen/impls/mis/mis.c >>>>> >> >>>>> >> Am I ready to do a push? >>>>> >> >>>>> >> Thanks, >>>>> >> Mark >>>>> >> >>>>> >> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: >>>>> >> >>>>> >>> If commit is the last hg operation that you've done - then 'hg >>>>> rollback' would undo this commit. >>>>> >>> >>>>> >>> Satish >>>>> >>> >>>>> >>> On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>>> >>> >>>>> >>>> Damn, I'm not preallocating the graph perfectly for unsymmetric >>>>> matrices and PETSc now dies on this. >>>>> >>>> >>>>> >>>> I have a fix but I committed it with other changes that I do not >>>>> want to commit. The changes are all in one file so I should be able to >>>>> just commit this file. >>>>> >>>> >>>>> >>>> Anyone know how to delete a commit? >>>>> >>>> >>>>> >>>> I've tried: >>>>> >>>> >>>>> >>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip >>>>> 22487:26ffb9eef17f >>>>> >>>> hg: unknown command 'strip' >>>>> >>>> 'strip' is provided by the following extension: >>>>> >>>> >>>>> >>>> mq manage a stack of patches >>>>> >>>> >>>>> >>>> use "hg help extensions" for information on enabling extensions >>>>> >>>> >>>>> >>>> But have not figured out how to load extensions. >>>>> >>>> >>>>> >>>> Mark >>>>> >>>> >>>>> >>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: >>>>> >>>> >>>>> >>>>> Mark, >>>>> >>>>> >>>>> >>>>> I have a non-symmetric matrix. I am running with the following >>>>> options. >>>>> >>>>> >>>>> >>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual >>>>> >>>>> >>>>> >>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new malloc >>>>> error: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> 0]PETSC ERROR: --------------------- Error Message >>>>> ------------------------------------ >>>>> >>>>> [0]PETSC ERROR: Argument out of range! >>>>> >>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! >>>>> >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> >>>>> [0]PETSC ERROR: Petsc Development HG revision: >>>>> 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 >>>>> -0500 >>>>> >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble >>>>> shooting. >>>>> >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>> >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> >>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named >>>>> wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 >>>>> >>>>> [0]PETSC ERROR: Libraries linked from >>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib >>>>> >>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 >>>>> >>>>> [0]PETSC ERROR: Configure options --download-blacs=1 >>>>> --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 >>>>> --download-parmetis=1 --download-scalapack=1 >>>>> --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc >>>>> --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort >>>>> PETSC_ARCH=linux-debug >>>>> >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> >>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in >>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >>>>> >>>>> [0]PETSC ERROR: MatSetValues() line 1141 in >>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c >>>>> >>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in >>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c >>>>> >>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in >>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c >>>>> >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in >>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>>>> >>>>> [0]PETSC ERROR: PCSetUp() line 832 in >>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c >>>>> >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in >>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>> >>>>> [0]PETSC ERROR: KSPSolve() line 385 in >>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> John >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams < >>>>> mark.adams at columbia.edu> wrote: >>>>> >>>>> >>>>> >>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: >>>>> >>>>> >>>>> >>>>>> Mark, >>>>> >>>>>> >>>>> >>>>>> The matrix is asymmetric. Does this require the setting of an >>>>> option? >>>>> >>>>> >>>>> >>>>> Yes: -pc_gamg_sym_graph >>>>> >>>>> >>>>> >>>>> Mark >>>>> >>>>> >>>>> >>>>>> I pulled petsc-dev this morning, so I should have (at least >>>>> close to) the latest code. >>>>> >>>>>> >>>>> >>>>>> John >>>>> >>>>>> >>>>> >>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams < >>>>> mark.adams at columbia.edu> wrote: >>>>> >>>>>> >>>>> >>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >>>>> >>>>>> >>>>> >>>>>>> I'm getting the following error when using GAMG. >>>>> >>>>>>> >>>>> >>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: >>>>> Assertion `sgid==-1' failed. >>>>> >>>>>> >>>>> >>>>>> Is it possible that your matrix is structurally asymmetric? >>>>> >>>>>> >>>>> >>>>>> This code is evolving fast and so you will need to move to the >>>>> dev version if you are not already using it. (I think I fixed a bug that >>>>> hit this assert). >>>>> >>>>>> >>>>> >>>>>>> >>>>> >>>>>>> When I try to alter the type of aggregation at the command >>>>> line using -pc_gamg_type pa, I'm getting >>>>> >>>>>>> >>>>> >>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error >>>>> Message ------------------------------------ >>>>> >>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or >>>>> missing external package needed for type: >>>>> >>>>>>> see >>>>> http://www.mcs.anl.gov/petsc/documentation/installation.html#external! >>>>> >>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given! >>>>> >>>>>>> >>>>> >>>>>>> Has there been a change in the aggregation options? I just >>>>> pulled petsc-dev this morning. >>>>> >>>>>>> >>>>> >>>>>> >>>>> >>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg >>>>> for now. >>>>> >>>>>> >>>>> >>>>>> Mark >>>>> >>>>>> >>>>> >>>>>>> John >>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>>> >>>> >>>>> >>> >>>>> >>> >>>>> >> >>>>> >> >>>>> > >>>>> > >>>>> >>>>> >>>> >>>> >>> >>> >>> >>> >> >> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Thu Mar 15 16:28:51 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Thu, 15 Mar 2012 17:28:51 -0400 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> Message-ID: <640F531B-7804-4FE4-9AFF-ED40F300D42A@columbia.edu> On Mar 15, 2012, at 4:54 PM, Matthew Knepley wrote: > On Thu, Mar 15, 2012 at 3:24 PM, John Mousel wrote: > Mark, > > Running without the options you mentioned before leads to slightly worse performance (175 iterations). > I have not been able to get run coarse grid solve to work with LU while running ML. It keeps experiencing a zero pivot, and all the combinations of shifting i've tried haven't lead me anywhere, hence the SOR on the course grid. Also, the ML manual suggests limiting the number of levels to 3 or 4 and performing a few sweeps of an iterative method as opposed to a direct solve. > This system is singular; it is a Neumann problem. That is fine. LU survived in GAMG because GAMG added just enough floating point error to keep it alive but it is a terrible solver, so SOR is fine. But I think that you do want to use richardson or GMRES (default) and not preonly with say jacobi preconditioning. Its not clear to me why GAMG is so bad. I suspect that the coarse grid is larger in the GAMG solve than ML, and so the simple SOR solver is not adequate. GAMG stops coarsening with 139 equations on your test. You can run with -pc_gamg_coarse_eq_limit 10 This will coarsen to a tiny problem (<10) and 8 iterations of SOR should be fine. Mark > That is suspicious behavior for LU. I now suspect something is wrong with this system, namely it is rank deficient > or the aggregates are and you are not seeing real convergence. Can you add -ksp_monitor_true_residual to make > sure the solver is actually converging. > > Matt > > John > > On Thu, Mar 15, 2012 at 12:04 PM, Mark F. Adams wrote: > You also want: -pc_gamg_agg_nsmooths 1 > > You are running plain aggregation. If it is Poisson then smoothing is good. > > Is this problem singular? Can you try running ML with these parameters and see if its performance degrades? The ML implementation uses the PETSC infrastructure and uses a very similar algorithm to GAMG-SA. We should be able to get these two to match pretty well. > > Mark > > > On Mar 15, 2012, at 12:21 PM, John Mousel wrote: > >> Mark, >> >> I ran with those options removed (see the run options listed below). Things actually got slightly worse. Now it's up to 142 iterations. I have attached the ksp_view output. >> >> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale -ksp_diagonal_scale_fix -mg_levels_ksp_type richardson -mg_levels_pc_type sor -pc_gamg_verbose 1 >> >> >> John >> >> >> On Thu, Mar 15, 2012 at 10:55 AM, Mark F. Adams wrote: >> John, can you run again with: -pc_gamg_verbose 1 >> >> And I would not use: -pc_mg_levels 4 -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 >> >> 1) I think -mg_coarse_ksp_type preonly and -mg_coarse_pc_sor_its 8 do not do what you think. I think this is the same as 1 iteration. I think you want 'richardson' not 'preonly'. >> >> 2) Why are you using sor as the coarse solver? If your problem is singular then you want to use as many levels as possible to get the coarse grid to be tiny. I'm pretty sure HYPRE ignores the coarse solver parameters. But ML uses them and it is converging well. >> >> 3) I would not specify the number of levels. GAMG, and I think the rest, have internal logic for stopping a the right level. If the coarse level is large and you use just 8 iterations of sor then convergence will suffer. >> >> Mark >> >> On Mar 15, 2012, at 11:13 AM, John Mousel wrote: >> >>> Mark, >>> >>> The changes pulled through this morning. I've run it with the options >>> >>> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale -ksp_diagonal_scale_fix -pc_mg_levels 4 -mg_levels_ksp_type richardson -mg_levels_pc_type sor -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 >>> >>> and it converges in the true residual, but it's not converging as fast as anticpated. The matrix arises from a non-symmetric discretization of the Poisson equation. The solve takes GAMG 114 iterations, whereas ML takes 24 iterations, BoomerAMG takes 22 iterations, and -ksp_type bcgsl -pc_type bjacobi -sub_pc_type ilu -sub_pc_factor_levels 4 takes around 170. I've attached the -ksp_view results for ML,GAMG, and HYPRE. I've attempted to make all the options the same on all levels for ML and GAMG. >>> >>> Any thoughts? >>> >>> John >>> >>> >>> On Wed, Mar 14, 2012 at 6:04 PM, Mark F. Adams wrote: >>> Humm, I see it with hg view (appended). >>> >>> Satish, my main repo looks hosed. I see this: >>> >>> ~/Codes/petsc-dev>hg update >>> abort: crosses branches (merge branches or use --clean to discard changes) >>> ~/Codes/petsc-dev>hg merge >>> abort: branch 'default' has 3 heads - please merge with an explicit rev >>> (run 'hg heads .' to see heads) >>> ~/Codes/petsc-dev>hg heads >>> changeset: 22496:8e2a98268179 >>> tag: tip >>> user: Barry Smith >>> date: Wed Mar 14 16:42:25 2012 -0500 >>> files: src/vec/is/interface/f90-custom/zindexf90.c src/vec/vec/interface/f90-custom/zvectorf90.c >>> description: >>> undoing manually changes I put in because Satish had a better fix >>> >>> >>> changeset: 22492:bda4df63072d >>> user: Mark F. Adams >>> date: Wed Mar 14 17:39:52 2012 -0400 >>> files: src/ksp/pc/impls/gamg/tools.c >>> description: >>> fix for unsymmetric matrices. >>> >>> >>> changeset: 22469:b063baf366e4 >>> user: Mark F. Adams >>> date: Wed Mar 14 14:22:28 2012 -0400 >>> files: src/ksp/pc/impls/gamg/tools.c >>> description: >>> added fix for preallocation for unsymetric matrices. >>> >>> Mark >>> >>> my 'hg view' on my merge repo: >>> >>> Revision: 22492 >>> Branch: default >>> Author: Mark F. Adams 2012-03-14 17:39:52 >>> Committer: Mark F. Adams 2012-03-14 17:39:52 >>> Tags: tip >>> Parent: 22491:451bbbd291c2 (Small fixes to the BT linesearch) >>> >>> fix for unsymmetric matrices. >>> >>> >>> ------------------------ src/ksp/pc/impls/gamg/tools.c ------------------------ >>> @@ -103,7 +103,7 @@ >>> PetscErrorCode ierr; >>> PetscInt Istart,Iend,Ii,jj,ncols,nnz0,nnz1, NN, MM, nloc; >>> PetscMPIInt mype, npe; >>> - Mat Gmat = *a_Gmat, tGmat; >>> + Mat Gmat = *a_Gmat, tGmat, matTrans; >>> MPI_Comm wcomm = ((PetscObject)Gmat)->comm; >>> const PetscScalar *vals; >>> const PetscInt *idx; >>> @@ -127,6 +127,10 @@ >>> ierr = MatDiagonalScale( Gmat, diag, diag ); CHKERRQ(ierr); >>> ierr = VecDestroy( &diag ); CHKERRQ(ierr); >>> >>> + if( symm ) { >>> + ierr = MatTranspose( Gmat, MAT_INITIAL_MATRIX, &matTrans ); CHKERRQ(ierr); >>> + } >>> + >>> /* filter - dup zeros out matrix */ >>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &d_nnz ); CHKERRQ(ierr); >>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &o_nnz ); CHKERRQ(ierr); >>> @@ -135,6 +139,12 @@ >>> d_nnz[jj] = ncols; >>> o_nnz[jj] = ncols; >>> ierr = MatRestoreRow(Gmat,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >>> + if( symm ) { >>> + ierr = MatGetRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >>> + d_nnz[jj] += ncols; >>> + o_nnz[jj] += ncols; >>> + ierr = MatRestoreRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >>> + } >>> if( d_nnz[jj] > nloc ) d_nnz[jj] = nloc; >>> if( o_nnz[jj] > (MM-nloc) ) o_nnz[jj] = MM - nloc; >>> } >>> @@ -142,6 +152,9 @@ >>> CHKERRQ(ierr); >>> ierr = PetscFree( d_nnz ); CHKERRQ(ierr); >>> ierr = PetscFree( o_nnz ); CHKERRQ(ierr); >>> + if( symm ) { >>> + ierr = MatDestroy( &matTrans ); CHKERRQ(ierr); >>> + } >>> >>> >>> >>> >>> On Mar 14, 2012, at 5:53 PM, John Mousel wrote: >>> >>>> Mark, >>>> >>>> No change. Can you give me the location that you patched so I can check to make sure it pulled? >>>> I don't see it on the petsc-dev change log. >>>> >>>> John >>>> >>>> On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams wrote: >>>> John, I've committed these changes, give a try. >>>> >>>> Mark >>>> >>>> On Mar 14, 2012, at 3:46 PM, Satish Balay wrote: >>>> >>>> > This is the usual merge [with uncommited changes] issue. >>>> > >>>> > You could use 'hg shelf' extension to shelve your local changes and >>>> > then do a merge [as Sean would suggest] - or do the merge in a >>>> > separate/clean clone [I normally do this..] >>>> > >>>> > i.e >>>> > cd ~/Codes >>>> > hg clone petsc-dev petsc-dev-merge >>>> > cd petsc-dev-merge >>>> > hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just to be sure, look for latest chagnes before merge.. >>>> > hg merge >>>> > hg commit >>>> > hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev >>>> > >>>> > [now update your petsc-dev to latest] >>>> > cd ~/Codes/petsc-dev >>>> > hg pull >>>> > hg update >>>> > >>>> > Satish >>>> > >>>> > On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>> > >>>> >> Great, that seems to work. >>>> >> >>>> >> I did a 'hg commit tools.c' >>>> >> >>>> >> and I want to push this file only. I guess its the only thing in the change set so 'hg push' should be fine. But I see this: >>>> >> >>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update >>>> >> abort: crosses branches (merge branches or use --clean to discard changes) >>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge >>>> >> abort: outstanding uncommitted changes (use 'hg status' to list changes) >>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status >>>> >> M include/petscmat.h >>>> >> M include/private/matimpl.h >>>> >> M src/ksp/pc/impls/gamg/agg.c >>>> >> M src/ksp/pc/impls/gamg/gamg.c >>>> >> M src/ksp/pc/impls/gamg/gamg.h >>>> >> M src/ksp/pc/impls/gamg/geo.c >>>> >> M src/mat/coarsen/coarsen.c >>>> >> M src/mat/coarsen/impls/hem/hem.c >>>> >> M src/mat/coarsen/impls/mis/mis.c >>>> >> >>>> >> Am I ready to do a push? >>>> >> >>>> >> Thanks, >>>> >> Mark >>>> >> >>>> >> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: >>>> >> >>>> >>> If commit is the last hg operation that you've done - then 'hg rollback' would undo this commit. >>>> >>> >>>> >>> Satish >>>> >>> >>>> >>> On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>> >>> >>>> >>>> Damn, I'm not preallocating the graph perfectly for unsymmetric matrices and PETSc now dies on this. >>>> >>>> >>>> >>>> I have a fix but I committed it with other changes that I do not want to commit. The changes are all in one file so I should be able to just commit this file. >>>> >>>> >>>> >>>> Anyone know how to delete a commit? >>>> >>>> >>>> >>>> I've tried: >>>> >>>> >>>> >>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f >>>> >>>> hg: unknown command 'strip' >>>> >>>> 'strip' is provided by the following extension: >>>> >>>> >>>> >>>> mq manage a stack of patches >>>> >>>> >>>> >>>> use "hg help extensions" for information on enabling extensions >>>> >>>> >>>> >>>> But have not figured out how to load extensions. >>>> >>>> >>>> >>>> Mark >>>> >>>> >>>> >>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: >>>> >>>> >>>> >>>>> Mark, >>>> >>>>> >>>> >>>>> I have a non-symmetric matrix. I am running with the following options. >>>> >>>>> >>>> >>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual >>>> >>>>> >>>> >>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new malloc error: >>>> >>>>> >>>> >>>>> >>>> >>>>> 0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>> >>>>> [0]PETSC ERROR: Argument out of range! >>>> >>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! >>>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> >>>>> [0]PETSC ERROR: Petsc Development HG revision: 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 -0500 >>>> >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> >>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 >>>> >>>>> [0]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib >>>> >>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 >>>> >>>>> [0]PETSC ERROR: Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug >>>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> >>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >>>> >>>>> [0]PETSC ERROR: MatSetValues() line 1141 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c >>>> >>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c >>>> >>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c >>>> >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>>> >>>>> [0]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c >>>> >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>> >>>>> [0]PETSC ERROR: KSPSolve() line 385 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>> >>>>> >>>> >>>>> >>>> >>>>> John >>>> >>>>> >>>> >>>>> >>>> >>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams wrote: >>>> >>>>> >>>> >>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: >>>> >>>>> >>>> >>>>>> Mark, >>>> >>>>>> >>>> >>>>>> The matrix is asymmetric. Does this require the setting of an option? >>>> >>>>> >>>> >>>>> Yes: -pc_gamg_sym_graph >>>> >>>>> >>>> >>>>> Mark >>>> >>>>> >>>> >>>>>> I pulled petsc-dev this morning, so I should have (at least close to) the latest code. >>>> >>>>>> >>>> >>>>>> John >>>> >>>>>> >>>> >>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams wrote: >>>> >>>>>> >>>> >>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >>>> >>>>>> >>>> >>>>>>> I'm getting the following error when using GAMG. >>>> >>>>>>> >>>> >>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion `sgid==-1' failed. >>>> >>>>>> >>>> >>>>>> Is it possible that your matrix is structurally asymmetric? >>>> >>>>>> >>>> >>>>>> This code is evolving fast and so you will need to move to the dev version if you are not already using it. (I think I fixed a bug that hit this assert). >>>> >>>>>> >>>> >>>>>>> >>>> >>>>>>> When I try to alter the type of aggregation at the command line using -pc_gamg_type pa, I'm getting >>>> >>>>>>> >>>> >>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>> >>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type: >>>> >>>>>>> see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! >>>> >>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given! >>>> >>>>>>> >>>> >>>>>>> Has there been a change in the aggregation options? I just pulled petsc-dev this morning. >>>> >>>>>>> >>>> >>>>>> >>>> >>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg for now. >>>> >>>>>> >>>> >>>>>> Mark >>>> >>>>>> >>>> >>>>>>> John >>>> >>>>>> >>>> >>>>>> >>>> >>>>> >>>> >>>>> >>>> >>>> >>>> >>>> >>>> >>> >>>> >>> >>>> >> >>>> >> >>>> > >>>> > >>>> >>>> >>> >>> >>> >> >> >> > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengshixiangze at 163.com Thu Mar 15 20:24:24 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Fri, 16 Mar 2012 09:24:24 +0800 (CST) Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> <97d3e2d.10edc.1360c777086.Coremail.zengshixiangze@163.com> <56e287e6.15a37.13616725ab4.Coremail.zengshixiangze@163.com> <633e1e3b.16bcc.13616bd9bf2.Coremail.zengshixiangze@163.com> Message-ID: <6d1d77a7.1c6ef.136191b0a3b.Coremail.zengshixiangze@163.com> I can see log_summary output when I use -log_summary [filename]. But the time of inserting the matrices values is almost the same when I use PCJACOBI to run the code on GPU instead of PCSOR.They are all much longer than that when I run the code in CPU(I still use a small system to test it). The attachments are outputs of log_summary, c_jacobi_sum is the output of using CPU, PCJACOBI;g_sor_sum using GPU, PCSOR;g_jac_sum using GPU, PCJACOBI. Zeng ? 2012-03-15 22:26:41?"Jed Brown" ??? 2012/3/15 Xiangze Zeng Sorry, someting is wrong with my file which stores the size of the matriex. But I still can't see -log_summary output. I just see at the first time, after that, never. It is written by PetscFinalize(). Find out why you don't get there. Also note that the option needs to be visible at PetscInitialize (through the command line/options file/environment). -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: c_jacobi_sum Type: application/octet-stream Size: 9205 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: c_sor_sum Type: application/octet-stream Size: 9217 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: g_jac_sum Type: application/octet-stream Size: 9635 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: g_sor_sum Type: application/octet-stream Size: 9655 bytes Desc: not available URL: From pham_ha at yahoo.com Fri Mar 16 00:23:20 2012 From: pham_ha at yahoo.com (Pham Van) Date: Thu, 15 Mar 2012 22:23:20 -0700 (PDT) Subject: [petsc-users] External package SCALAPACK does not work on Microsoft Windows In-Reply-To: References: Message-ID: <1331875400.57128.YahooMailNeo@web163102.mail.bf1.yahoo.com> Dear Team, I used to compile petsc with external package mumps in version 3.1 However, today I tried 3.2 and get many of such error massage: External package SCALAPACK does not work on Microsoft Windows It happens also with blacs. How can I build mumps without those pakages? Kind regards, PVH ________________________________ From: "petsc-users-request at mcs.anl.gov" To: petsc-users at mcs.anl.gov Sent: Friday, March 16, 2012 8:24 AM Subject: petsc-users Digest, Vol 39, Issue 48 Send petsc-users mailing list submissions to ??? petsc-users at mcs.anl.gov To subscribe or unsubscribe via the World Wide Web, visit ??? https://lists.mcs.anl.gov/mailman/listinfo/petsc-users or, via email, send a message with subject or body 'help' to ??? petsc-users-request at mcs.anl.gov You can reach the person managing the list at ??? petsc-users-owner at mcs.anl.gov When replying, please edit your Subject line so it is more specific than "Re: Contents of petsc-users digest..." Today's Topics: ? 1. Re:? [petsc-maint #107932] run a simple example ex19 in ? ? ? src/snes/examples/tutorials (Xiangze Zeng) ---------------------------------------------------------------------- Message: 1 Date: Fri, 16 Mar 2012 09:24:24 +0800 (CST) From: "Xiangze Zeng" Subject: Re: [petsc-users] [petsc-maint #107932] run a simple example ??? ex19 in src/snes/examples/tutorials To: "PETSc users list" Message-ID: ??? <6d1d77a7.1c6ef.136191b0a3b.Coremail.zengshixiangze at 163.com> Content-Type: text/plain; charset="gbk" I can see log_summary output when I use -log_summary [filename]. But the time of inserting the matrices values is almost the same when I use PCJACOBI? to run the code on GPU instead of PCSOR.They are all much longer than that when I run the code in CPU(I still use a small system to test it). The attachments are outputs of log_summary, c_jacobi_sum is the output of using CPU, PCJACOBI;g_sor_sum using GPU, PCSOR;g_jac_sum using GPU, PCJACOBI. Zeng ? 2012-03-15 22:26:41?"Jed Brown" ??? 2012/3/15 Xiangze Zeng Sorry, someting is wrong with my file which stores the size of the matriex. But I still can't see -log_summary output. I just see at the first time, after that, never. It is written by PetscFinalize(). Find out why you don't get there. Also note that the option needs to be visible at PetscInitialize (through the command line/options file/environment). -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: c_jacobi_sum Type: application/octet-stream Size: 9205 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: c_sor_sum Type: application/octet-stream Size: 9217 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: g_jac_sum Type: application/octet-stream Size: 9635 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: g_sor_sum Type: application/octet-stream Size: 9655 bytes Desc: not available URL: ------------------------------ _______________________________________________ petsc-users mailing list petsc-users at mcs.anl.gov https://lists.mcs.anl.gov/mailman/listinfo/petsc-users End of petsc-users Digest, Vol 39, Issue 48 ******************************************* -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Mar 16 06:43:38 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 16 Mar 2012 06:43:38 -0500 Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: <6d1d77a7.1c6ef.136191b0a3b.Coremail.zengshixiangze@163.com> References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> <97d3e2d.10edc.1360c777086.Coremail.zengshixiangze@163.com> <56e287e6.15a37.13616725ab4.Coremail.zengshixiangze@163.com> <633e1e3b.16bcc.13616bd9bf2.Coremail.zengshixiangze@163.com> <6d1d77a7.1c6ef.136191b0a3b.Coremail.zengshixiangze@163.com> Message-ID: 2012/3/15 Xiangze Zeng > I can see log_summary output when I use -log_summary [filename]. > > But the time of inserting the matrices values is almost the same when I > use PCJACOBI to run the code on GPU instead of PCSOR.They are all much > longer than that when I run the code in CPU(I still use a small system to > test it). The attachments are outputs of log_summary, c_jacobi_sum is the > output of using CPU, PCJACOBI;g_sor_sum using GPU, PCSOR;g_jac_sum using > GPU, PCJACOBI. > ########################################################## # # # WARNING!!! # # # # This code was compiled with a debugging option, # # To get timing results run ./configure # # using --with-debugging=no, the performance will # # be generally two or three times faster. # # # ########################################################## It's a waste of time to look at performance of a debug build, especially when it only takes half a second. You have probably lost preallocation information. I don't see an assembly event, so I can't tell if that is really where the time is. (SNES makes one automatically, you can PetscLogEventRegister(), PetscLogEventBegin/End(). -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Mar 16 06:50:54 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 16 Mar 2012 06:50:54 -0500 Subject: [petsc-users] External package SCALAPACK does not work on Microsoft Windows In-Reply-To: <1331875400.57128.YahooMailNeo@web163102.mail.bf1.yahoo.com> References: <1331875400.57128.YahooMailNeo@web163102.mail.bf1.yahoo.com> Message-ID: On Fri, Mar 16, 2012 at 00:23, Pham Van wrote: > I used to compile petsc with external package mumps in version 3.1 > How did you build them? If you had them built, you could add this line to the __init__() function in config/PETSc/packages/SCALAPACK.py and config/PETSc/packages/blacs.py self.worksonWindows = 1 If you want to try --download-{scalapack,blacs}, you can also add this line self.downloadonWindows= 1 Let us know if it works, I assumed it was disabled because it didn't. > However, today I tried 3.2 and get many of such error massage: > > External package SCALAPACK does not work on Microsoft Windows > > It happens also with blacs. How can I build mumps without those pakages? > You can't, those are hard dependencies. You could use SuperLU_DIST which works with --download-superlu_dist. -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Fri Mar 16 08:40:41 2012 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Fri, 16 Mar 2012 13:40:41 +0000 Subject: [petsc-users] function name and meaning of -pc_fieldsplit_schur_factorization_type Message-ID: The manual page 87 states that "there are several variants of the Schur complement preconditioner obtained by dropping some of the terms, these can be obtained with -pc_fieldsplit_schur_factorization_type " 1) What is (will be) the name of the corresponding function? 2) It would be nice to know what exactly is being dropped. is obvious and apparently drops the right most matrix, but what do lower and upper drop? 3) I'm confused about the minus sign in the form and its motivation. It doesn't come from the factorization. Is the rhs multiplied with a minus sign as well? Why not follow the sign convention of Elman's taxonomy (eq 3 and 4, JCP 227, 2008)? dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From knepley at gmail.com Fri Mar 16 09:07:00 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 16 Mar 2012 09:07:00 -0500 Subject: [petsc-users] function name and meaning of -pc_fieldsplit_schur_factorization_type In-Reply-To: References: Message-ID: On Fri, Mar 16, 2012 at 8:40 AM, Klaij, Christiaan wrote: > The manual page 87 states that "there are several variants > of the Schur complement preconditioner obtained by dropping some > of the terms, these can be obtained with > -pc_fieldsplit_schur_factorization_type " > > 1) What is (will be) the name of the corresponding function? > There isn't one. > 2) It would be nice to know what exactly is being dropped. > is obvious and apparently drops the right most matrix, but > what do lower and upper drop? > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCFIELDSPLIT.html > 3) I'm confused about the minus sign in the form and its > motivation. It doesn't come from the factorization. Is the rhs > multiplied with a minus sign as well? Why not follow the sign > convention of Elman's taxonomy (eq 3 and 4, JCP 227, 2008)? > It makes the PC positive definite. Matt > > dr. ir. Christiaan Klaij > CFD Researcher > Research & Development > E mailto:C.Klaij at marin.nl > T +31 317 49 33 44 > > MARIN > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Mar 16 09:11:19 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 16 Mar 2012 09:11:19 -0500 Subject: [petsc-users] function name and meaning of -pc_fieldsplit_schur_factorization_type In-Reply-To: References: Message-ID: On Fri, Mar 16, 2012 at 08:40, Klaij, Christiaan wrote: > The manual page 87 states that "there are several variants > of the Schur complement preconditioner obtained by dropping some > of the terms, these can be obtained with > -pc_fieldsplit_schur_factorization_type " > > 1) What is (will be) the name of the corresponding function? > It looks like PCFieldSplitSetSchurFactorizationType() is missing. > 2) It would be nice to know what exactly is being dropped. > is obvious and apparently drops the right most matrix, but > what do lower and upper drop? > Lower uses the lower-triangular factor (dropping the strictly upper triangular factor). > > 3) I'm confused about the minus sign in the form and its > motivation. It doesn't come from the factorization. Is the rhs > multiplied with a minus sign as well? Why not follow the sign > convention of Elman's taxonomy (eq 3 and 4, JCP 227, 2008)? > Elman's convention is rather specific to finite element discretization of incompressible flow. He arbitrarily slaps a negative on one of the matrices in the system because it represents a certain kind of stabilization. That minus sign would be pretty silly if the matrix was actually positive (mixed elasticity, finite-time step stiff waves, etc). The negative sign in the DIAG form appears because the minimal polynomial with that preconditioner has degree 3 (for a nonsingular problem) if you have a minus sign there. Read Murphy's note: http://eprints.maths.ox.ac.uk/1291/1/NA-99-07.pdf If you skip that minus sign, that property goes away. Also, if you are solving Stokes flow or constrained problems in elasticity, that sign convention causes the DIAG form to be positive definite which means that you could use it with MINRES which was is the main motivation for using DIAG in the first place. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengshixiangze at 163.com Fri Mar 16 09:35:22 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Fri, 16 Mar 2012 22:35:22 +0800 (CST) Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> <97d3e2d.10edc.1360c777086.Coremail.zengshixiangze@163.com> <56e287e6.15a37.13616725ab4.Coremail.zengshixiangze@163.com> <633e1e3b.16bcc.13616bd9bf2.Coremail.zengshixiangze@163.com> <6d1d77a7.1c6ef.136191b0a3b.Coremail.zengshixiangze@163.com> Message-ID: <45dd8f25.2ccb7.1361bef32e2.Coremail.zengshixiangze@163.com> After I changed the Mat type for using GPU, can I still use MatMPIAIJSetPreallocation to set preallocation? I have add Petsc events in the code.The attachments are the outputs of log_summary (I'll configure --with-debugging=no later). The fileinterface.c is my code in which set preallocation and insert matrices values. Thank you. Zeng ? 2012-03-16 19:43:38?"Jed Brown" ??? 2012/3/15 Xiangze Zeng I can see log_summary output when I use -log_summary [filename]. But the time of inserting the matrices values is almost the same when I use PCJACOBI to run the code on GPU instead of PCSOR.They are all much longer than that when I run the code in CPU(I still use a small system to test it). The attachments are outputs of log_summary, c_jacobi_sum is the output of using CPU, PCJACOBI;g_sor_sum using GPU, PCSOR;g_jac_sum using GPU, PCJACOBI. ########################################################## # # # WARNING!!! # # # # This code was compiled with a debugging option, # # To get timing results run ./configure # # using --with-debugging=no, the performance will # # be generally two or three times faster. # # # ########################################################## It's a waste of time to look at performance of a debug build, especially when it only takes half a second. You have probably lost preallocation information. I don't see an assembly event, so I can't tell if that is really where the time is. (SNES makes one automatically, you can PetscLogEventRegister(), PetscLogEventBegin/End(). -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: c_jac_sum Type: application/octet-stream Size: 9639 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: c_sor_sum Type: application/octet-stream Size: 9644 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: g_jac_sum Type: application/octet-stream Size: 9992 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: g_sor_sum Type: application/octet-stream Size: 10007 bytes Desc: not available URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: fileinterface.c URL: From jedbrown at mcs.anl.gov Fri Mar 16 10:00:29 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 16 Mar 2012 10:00:29 -0500 Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: <45dd8f25.2ccb7.1361bef32e2.Coremail.zengshixiangze@163.com> References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> <97d3e2d.10edc.1360c777086.Coremail.zengshixiangze@163.com> <56e287e6.15a37.13616725ab4.Coremail.zengshixiangze@163.com> <633e1e3b.16bcc.13616bd9bf2.Coremail.zengshixiangze@163.com> <6d1d77a7.1c6ef.136191b0a3b.Coremail.zengshixiangze@163.com> <45dd8f25.2ccb7.1361bef32e2.Coremail.zengshixiangze@163.com> Message-ID: 2012/3/16 Xiangze Zeng > After I changed the Mat type for using GPU, can I still > use MatMPIAIJSetPreallocation to set preallocation? > Yes, the code looks okay to me. > > I have add Petsc events in the code.The attachments are the outputs of > log_summary (I'll configure --with-debugging=no later). > The fileinterface.c is my code in which set preallocation and insert > matrices values. > You are running in serial so you should also call MatSeqAIJSetPreallocation(). The MPI allocation information is skipped because it's not an MPI matrix. It is safe to call all the preallocation routines, whichever matches will be used. Alternatively, you could call this function which will do all the variants for you: http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatXAIJSetPreallocation.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengshixiangze at 163.com Fri Mar 16 10:59:31 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Fri, 16 Mar 2012 23:59:31 +0800 (CST) Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> <97d3e2d.10edc.1360c777086.Coremail.zengshixiangze@163.com> <56e287e6.15a37.13616725ab4.Coremail.zengshixiangze@163.com> <633e1e3b.16bcc.13616bd9bf2.Coremail.zengshixiangze@163.com> <6d1d77a7.1c6ef.136191b0a3b.Coremail.zengshixiangze@163.com> <45dd8f25.2ccb7.1361bef32e2.Coremail.zengshixiangze@163.com> Message-ID: <192a2148.2c982.1361c3c3ce9.Coremail.zengshixiangze@163.com> I replace MATAIJCUSP with MATMPIAIJCUSP. It works well. The attachments are the outputs of log_summary. MATAIJCUSP is used for serial matrices? ? 2012-03-16 23:00:29?"Jed Brown" ??? 2012/3/16 Xiangze Zeng After I changed the Mat type for using GPU, can I still use MatMPIAIJSetPreallocation to set preallocation? Yes, the code looks okay to me. I have add Petsc events in the code.The attachments are the outputs of log_summary (I'll configure --with-debugging=no later). The fileinterface.c is my code in which set preallocation and insert matrices values. You are running in serial so you should also call MatSeqAIJSetPreallocation(). The MPI allocation information is skipped because it's not an MPI matrix. It is safe to call all the preallocation routines, whichever matches will be used. Alternatively, you could call this function which will do all the variants for you: http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatXAIJSetPreallocation.html -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: g_jacobi_sum2 Type: application/octet-stream Size: 9986 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: g_sor_sum2 Type: application/octet-stream Size: 10008 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Fri Mar 16 11:09:48 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 16 Mar 2012 11:09:48 -0500 Subject: [petsc-users] [petsc-maint #107932] run a simple example ex19 in src/snes/examples/tutorials In-Reply-To: <192a2148.2c982.1361c3c3ce9.Coremail.zengshixiangze@163.com> References: <46512e7.26ab8.135d411d694.Coremail.zengshixiangze@163.com> <13122e75.15c5b.135e33990e7.Coremail.zengshixiangze@163.com> <7a211ad.24906.135e672bd23.Coremail.zengshixiangze@163.com> <1b2c0bb2.109a0.1360c5b563d.Coremail.zengshixiangze@163.com> <97d3e2d.10edc.1360c777086.Coremail.zengshixiangze@163.com> <56e287e6.15a37.13616725ab4.Coremail.zengshixiangze@163.com> <633e1e3b.16bcc.13616bd9bf2.Coremail.zengshixiangze@163.com> <6d1d77a7.1c6ef.136191b0a3b.Coremail.zengshixiangze@163.com> <45dd8f25.2ccb7.1361bef32e2.Coremail.zengshixiangze@163.com> <192a2148.2c982.1361c3c3ce9.Coremail.zengshixiangze@163.com> Message-ID: 2012/3/16 Xiangze Zeng > I replace MATAIJCUSP with MATMPIAIJCUSP. It works well. The attachments > are the outputs of log_summary. > MATAIJCUSP is used for serial matrices? > MATAIJCUSP automatically chooses MATSEQAIJCUSP if on serial communicator and MATMPIAIJCUSP if on a parallel communicator. Practically speaking, either will work, but when you run in serial, there is more overhead going through the MPIAIJCUSP container and fewer preconditioners will be available. So it's better to call bath MatSeqAIJSetPreallocation() and MatMPIAIJSetPreallocation(). -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Mar 16 13:47:27 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 16 Mar 2012 13:47:27 -0500 Subject: [petsc-users] External package SCALAPACK does not work on Microsoft Windows In-Reply-To: <1331922991.98054.YahooMailNeo@web163103.mail.bf1.yahoo.com> References: <1331875400.57128.YahooMailNeo@web163102.mail.bf1.yahoo.com> <1331922991.98054.YahooMailNeo@web163103.mail.bf1.yahoo.com> Message-ID: On Fri, Mar 16, 2012 at 13:36, Pham Van wrote: > Thank Jed for very quick reply. > > The two magic lines did the trick. The question is why they have been > excluded? For me its just fine right now. > I assume because people reported that it didn't work. I have enabled them again in petsc-dev. > > I have one question thought: I have unsymmetric matrix. It is solved just > fine with 1 processor, but if I increased to 2 processors mumps produce the > following error (the matrix is solved just fine when I use superlu_dist > package both uni and multi-proccesors, the problem is superlu_dist is too > slow): > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Error in external library! > [0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: > INFO(1)=-9, INFO(2)=2055660 > As far as we can tell, MUMPS is competing for the most arcane error reporting system. You have to read the user's manual to translate those numbers into anything understandable. ?9 Main internal real/complex workarray S too small. If INFO(2) is positive, then the number of entries that are missing in S at the moment when the error is raised is available in INFO(2). If INFO(2) is negative, then its absolute value should be multiplied by 1 million. If an error ?9 occurs, the user should increase the value of ICNTL(14) before calling the factorization (JOB=2) again, except if ICNTL(23) is provided, in which case ICNTL(23) should be increased. You can use -mat_mumps_icntl_14 to control that parameter. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Fri Mar 16 14:04:29 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Fri, 16 Mar 2012 15:04:29 -0400 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> Message-ID: John, did this get resolved? Mark On Mar 15, 2012, at 4:24 PM, John Mousel wrote: > Mark, > > Running without the options you mentioned before leads to slightly worse performance (175 iterations). > I have not been able to get run coarse grid solve to work with LU while running ML. It keeps experiencing a zero pivot, and all the combinations of shifting i've tried haven't lead me anywhere, hence the SOR on the course grid. Also, the ML manual suggests limiting the number of levels to 3 or 4 and performing a few sweeps of an iterative method as opposed to a direct solve. > > John > > On Thu, Mar 15, 2012 at 12:04 PM, Mark F. Adams wrote: > You also want: -pc_gamg_agg_nsmooths 1 > > You are running plain aggregation. If it is Poisson then smoothing is good. > > Is this problem singular? Can you try running ML with these parameters and see if its performance degrades? The ML implementation uses the PETSC infrastructure and uses a very similar algorithm to GAMG-SA. We should be able to get these two to match pretty well. > > Mark > > > On Mar 15, 2012, at 12:21 PM, John Mousel wrote: > >> Mark, >> >> I ran with those options removed (see the run options listed below). Things actually got slightly worse. Now it's up to 142 iterations. I have attached the ksp_view output. >> >> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale -ksp_diagonal_scale_fix -mg_levels_ksp_type richardson -mg_levels_pc_type sor -pc_gamg_verbose 1 >> >> >> John >> >> >> On Thu, Mar 15, 2012 at 10:55 AM, Mark F. Adams wrote: >> John, can you run again with: -pc_gamg_verbose 1 >> >> And I would not use: -pc_mg_levels 4 -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 >> >> 1) I think -mg_coarse_ksp_type preonly and -mg_coarse_pc_sor_its 8 do not do what you think. I think this is the same as 1 iteration. I think you want 'richardson' not 'preonly'. >> >> 2) Why are you using sor as the coarse solver? If your problem is singular then you want to use as many levels as possible to get the coarse grid to be tiny. I'm pretty sure HYPRE ignores the coarse solver parameters. But ML uses them and it is converging well. >> >> 3) I would not specify the number of levels. GAMG, and I think the rest, have internal logic for stopping a the right level. If the coarse level is large and you use just 8 iterations of sor then convergence will suffer. >> >> Mark >> >> On Mar 15, 2012, at 11:13 AM, John Mousel wrote: >> >>> Mark, >>> >>> The changes pulled through this morning. I've run it with the options >>> >>> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale -ksp_diagonal_scale_fix -pc_mg_levels 4 -mg_levels_ksp_type richardson -mg_levels_pc_type sor -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 >>> >>> and it converges in the true residual, but it's not converging as fast as anticpated. The matrix arises from a non-symmetric discretization of the Poisson equation. The solve takes GAMG 114 iterations, whereas ML takes 24 iterations, BoomerAMG takes 22 iterations, and -ksp_type bcgsl -pc_type bjacobi -sub_pc_type ilu -sub_pc_factor_levels 4 takes around 170. I've attached the -ksp_view results for ML,GAMG, and HYPRE. I've attempted to make all the options the same on all levels for ML and GAMG. >>> >>> Any thoughts? >>> >>> John >>> >>> >>> On Wed, Mar 14, 2012 at 6:04 PM, Mark F. Adams wrote: >>> Humm, I see it with hg view (appended). >>> >>> Satish, my main repo looks hosed. I see this: >>> >>> ~/Codes/petsc-dev>hg update >>> abort: crosses branches (merge branches or use --clean to discard changes) >>> ~/Codes/petsc-dev>hg merge >>> abort: branch 'default' has 3 heads - please merge with an explicit rev >>> (run 'hg heads .' to see heads) >>> ~/Codes/petsc-dev>hg heads >>> changeset: 22496:8e2a98268179 >>> tag: tip >>> user: Barry Smith >>> date: Wed Mar 14 16:42:25 2012 -0500 >>> files: src/vec/is/interface/f90-custom/zindexf90.c src/vec/vec/interface/f90-custom/zvectorf90.c >>> description: >>> undoing manually changes I put in because Satish had a better fix >>> >>> >>> changeset: 22492:bda4df63072d >>> user: Mark F. Adams >>> date: Wed Mar 14 17:39:52 2012 -0400 >>> files: src/ksp/pc/impls/gamg/tools.c >>> description: >>> fix for unsymmetric matrices. >>> >>> >>> changeset: 22469:b063baf366e4 >>> user: Mark F. Adams >>> date: Wed Mar 14 14:22:28 2012 -0400 >>> files: src/ksp/pc/impls/gamg/tools.c >>> description: >>> added fix for preallocation for unsymetric matrices. >>> >>> Mark >>> >>> my 'hg view' on my merge repo: >>> >>> Revision: 22492 >>> Branch: default >>> Author: Mark F. Adams 2012-03-14 17:39:52 >>> Committer: Mark F. Adams 2012-03-14 17:39:52 >>> Tags: tip >>> Parent: 22491:451bbbd291c2 (Small fixes to the BT linesearch) >>> >>> fix for unsymmetric matrices. >>> >>> >>> ------------------------ src/ksp/pc/impls/gamg/tools.c ------------------------ >>> @@ -103,7 +103,7 @@ >>> PetscErrorCode ierr; >>> PetscInt Istart,Iend,Ii,jj,ncols,nnz0,nnz1, NN, MM, nloc; >>> PetscMPIInt mype, npe; >>> - Mat Gmat = *a_Gmat, tGmat; >>> + Mat Gmat = *a_Gmat, tGmat, matTrans; >>> MPI_Comm wcomm = ((PetscObject)Gmat)->comm; >>> const PetscScalar *vals; >>> const PetscInt *idx; >>> @@ -127,6 +127,10 @@ >>> ierr = MatDiagonalScale( Gmat, diag, diag ); CHKERRQ(ierr); >>> ierr = VecDestroy( &diag ); CHKERRQ(ierr); >>> >>> + if( symm ) { >>> + ierr = MatTranspose( Gmat, MAT_INITIAL_MATRIX, &matTrans ); CHKERRQ(ierr); >>> + } >>> + >>> /* filter - dup zeros out matrix */ >>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &d_nnz ); CHKERRQ(ierr); >>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &o_nnz ); CHKERRQ(ierr); >>> @@ -135,6 +139,12 @@ >>> d_nnz[jj] = ncols; >>> o_nnz[jj] = ncols; >>> ierr = MatRestoreRow(Gmat,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >>> + if( symm ) { >>> + ierr = MatGetRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >>> + d_nnz[jj] += ncols; >>> + o_nnz[jj] += ncols; >>> + ierr = MatRestoreRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >>> + } >>> if( d_nnz[jj] > nloc ) d_nnz[jj] = nloc; >>> if( o_nnz[jj] > (MM-nloc) ) o_nnz[jj] = MM - nloc; >>> } >>> @@ -142,6 +152,9 @@ >>> CHKERRQ(ierr); >>> ierr = PetscFree( d_nnz ); CHKERRQ(ierr); >>> ierr = PetscFree( o_nnz ); CHKERRQ(ierr); >>> + if( symm ) { >>> + ierr = MatDestroy( &matTrans ); CHKERRQ(ierr); >>> + } >>> >>> >>> >>> >>> On Mar 14, 2012, at 5:53 PM, John Mousel wrote: >>> >>>> Mark, >>>> >>>> No change. Can you give me the location that you patched so I can check to make sure it pulled? >>>> I don't see it on the petsc-dev change log. >>>> >>>> John >>>> >>>> On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams wrote: >>>> John, I've committed these changes, give a try. >>>> >>>> Mark >>>> >>>> On Mar 14, 2012, at 3:46 PM, Satish Balay wrote: >>>> >>>> > This is the usual merge [with uncommited changes] issue. >>>> > >>>> > You could use 'hg shelf' extension to shelve your local changes and >>>> > then do a merge [as Sean would suggest] - or do the merge in a >>>> > separate/clean clone [I normally do this..] >>>> > >>>> > i.e >>>> > cd ~/Codes >>>> > hg clone petsc-dev petsc-dev-merge >>>> > cd petsc-dev-merge >>>> > hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just to be sure, look for latest chagnes before merge.. >>>> > hg merge >>>> > hg commit >>>> > hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev >>>> > >>>> > [now update your petsc-dev to latest] >>>> > cd ~/Codes/petsc-dev >>>> > hg pull >>>> > hg update >>>> > >>>> > Satish >>>> > >>>> > On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>> > >>>> >> Great, that seems to work. >>>> >> >>>> >> I did a 'hg commit tools.c' >>>> >> >>>> >> and I want to push this file only. I guess its the only thing in the change set so 'hg push' should be fine. But I see this: >>>> >> >>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update >>>> >> abort: crosses branches (merge branches or use --clean to discard changes) >>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge >>>> >> abort: outstanding uncommitted changes (use 'hg status' to list changes) >>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status >>>> >> M include/petscmat.h >>>> >> M include/private/matimpl.h >>>> >> M src/ksp/pc/impls/gamg/agg.c >>>> >> M src/ksp/pc/impls/gamg/gamg.c >>>> >> M src/ksp/pc/impls/gamg/gamg.h >>>> >> M src/ksp/pc/impls/gamg/geo.c >>>> >> M src/mat/coarsen/coarsen.c >>>> >> M src/mat/coarsen/impls/hem/hem.c >>>> >> M src/mat/coarsen/impls/mis/mis.c >>>> >> >>>> >> Am I ready to do a push? >>>> >> >>>> >> Thanks, >>>> >> Mark >>>> >> >>>> >> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: >>>> >> >>>> >>> If commit is the last hg operation that you've done - then 'hg rollback' would undo this commit. >>>> >>> >>>> >>> Satish >>>> >>> >>>> >>> On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>> >>> >>>> >>>> Damn, I'm not preallocating the graph perfectly for unsymmetric matrices and PETSc now dies on this. >>>> >>>> >>>> >>>> I have a fix but I committed it with other changes that I do not want to commit. The changes are all in one file so I should be able to just commit this file. >>>> >>>> >>>> >>>> Anyone know how to delete a commit? >>>> >>>> >>>> >>>> I've tried: >>>> >>>> >>>> >>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f >>>> >>>> hg: unknown command 'strip' >>>> >>>> 'strip' is provided by the following extension: >>>> >>>> >>>> >>>> mq manage a stack of patches >>>> >>>> >>>> >>>> use "hg help extensions" for information on enabling extensions >>>> >>>> >>>> >>>> But have not figured out how to load extensions. >>>> >>>> >>>> >>>> Mark >>>> >>>> >>>> >>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: >>>> >>>> >>>> >>>>> Mark, >>>> >>>>> >>>> >>>>> I have a non-symmetric matrix. I am running with the following options. >>>> >>>>> >>>> >>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual >>>> >>>>> >>>> >>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new malloc error: >>>> >>>>> >>>> >>>>> >>>> >>>>> 0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>> >>>>> [0]PETSC ERROR: Argument out of range! >>>> >>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! >>>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> >>>>> [0]PETSC ERROR: Petsc Development HG revision: 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 -0500 >>>> >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> >>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 >>>> >>>>> [0]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib >>>> >>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 >>>> >>>>> [0]PETSC ERROR: Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug >>>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> >>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >>>> >>>>> [0]PETSC ERROR: MatSetValues() line 1141 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c >>>> >>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c >>>> >>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c >>>> >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>>> >>>>> [0]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c >>>> >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>> >>>>> [0]PETSC ERROR: KSPSolve() line 385 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>> >>>>> >>>> >>>>> >>>> >>>>> John >>>> >>>>> >>>> >>>>> >>>> >>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams wrote: >>>> >>>>> >>>> >>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: >>>> >>>>> >>>> >>>>>> Mark, >>>> >>>>>> >>>> >>>>>> The matrix is asymmetric. Does this require the setting of an option? >>>> >>>>> >>>> >>>>> Yes: -pc_gamg_sym_graph >>>> >>>>> >>>> >>>>> Mark >>>> >>>>> >>>> >>>>>> I pulled petsc-dev this morning, so I should have (at least close to) the latest code. >>>> >>>>>> >>>> >>>>>> John >>>> >>>>>> >>>> >>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams wrote: >>>> >>>>>> >>>> >>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >>>> >>>>>> >>>> >>>>>>> I'm getting the following error when using GAMG. >>>> >>>>>>> >>>> >>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion `sgid==-1' failed. >>>> >>>>>> >>>> >>>>>> Is it possible that your matrix is structurally asymmetric? >>>> >>>>>> >>>> >>>>>> This code is evolving fast and so you will need to move to the dev version if you are not already using it. (I think I fixed a bug that hit this assert). >>>> >>>>>> >>>> >>>>>>> >>>> >>>>>>> When I try to alter the type of aggregation at the command line using -pc_gamg_type pa, I'm getting >>>> >>>>>>> >>>> >>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>> >>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type: >>>> >>>>>>> see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! >>>> >>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given! >>>> >>>>>>> >>>> >>>>>>> Has there been a change in the aggregation options? I just pulled petsc-dev this morning. >>>> >>>>>>> >>>> >>>>>> >>>> >>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg for now. >>>> >>>>>> >>>> >>>>>> Mark >>>> >>>>>> >>>> >>>>>>> John >>>> >>>>>> >>>> >>>>>> >>>> >>>>> >>>> >>>>> >>>> >>>> >>>> >>>> >>>> >>> >>>> >>> >>>> >> >>>> >> >>>> > >>>> > >>>> >>>> >>> >>> >>> >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nan.Jia at Dartmouth.edu Fri Mar 16 14:52:00 2012 From: Nan.Jia at Dartmouth.edu (Nan.Jia at Dartmouth.edu) Date: Fri, 16 Mar 2012 15:52:00 -0400 Subject: [petsc-users] Petsc Performance Help Message-ID: <20120316155200.rbvlenhc2sw8ccs4@webmail.dartmouth.edu> Dear PETSc Group, I am tuning the efficiency of my PETSc code for a while, but get very little progress. So can anyone help me to analysis the log? Any suggestions will be appreciated. My problem is time dependent. At every time step, two about 6000 by 6000 sparse matrices need to be solved, which come from a Poisson equation. I use both sequential and parallel AIJ format to store matrices, but the performances are both not very good. Please let me know if you need more information of the code or the problem. Thanks in advance! Best, Nan -------------- next part -------------- ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./3dtest5 on a linux-gnu named babylon3 with 1 processor, by nan_jia Fri Mar 16 15:13:12 2012 Using Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 Max Max/Min Avg Total Time (sec): 8.810e+02 1.00000 8.810e+02 Objects: 9.000e+01 1.00000 9.000e+01 Flops: 4.963e+09 1.00000 4.963e+09 4.963e+09 Flops/sec: 5.634e+06 1.00000 5.634e+06 5.634e+06 Memory: 7.493e+06 1.00000 7.493e+06 MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 MPI Reductions: 2.732e+03 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 8.8098e+02 100.0% 4.9632e+09 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 2.645e+03 96.8% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ ########################################################## # # # WARNING!!! # # # # This code was compiled with a debugging option, # # To get timing results run config/configure.py # # using --with-debugging=no, the performance will # # be generally two or three times faster. # # # ########################################################## Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage KSPGMRESOrthog 3134 1.0 2.4914e-01 1.0 1.82e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 4 0 0 0 0 4 0 0 0 729 KSPSetup 20001 1.0 1.0103e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 20001 1.0 1.8803e+01 1.0 4.96e+09 1.0 0.0e+00 0.0e+00 2.6e+03 2100 0 0 96 2100 0 0100 264 VecMDot 3134 1.0 8.5895e-02 1.0 9.08e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 1057 VecNorm 43141 1.0 3.4036e-01 1.0 5.47e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 11 0 0 0 0 11 0 0 0 1606 VecScale 23143 1.0 1.1922e-01 1.0 1.47e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 1230 VecCopy 20009 1.0 4.4645e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 2553 1.0 1.6527e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 22556 1.0 2.2632e-01 1.0 2.86e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 6 0 0 0 0 6 0 0 0 1263 VecMAXPY 5684 1.0 2.3512e-01 1.0 1.31e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 555 VecAssemblyBegin 40002 1.0 1.8107e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 40002 1.0 1.3301e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 23143 1.0 3.9494e-01 1.0 4.40e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 9 0 0 0 0 9 0 0 0 1114 MatMult 23140 1.0 5.4058e+00 1.0 1.31e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 26 0 0 0 1 26 0 0 0 243 MatSolve 43141 1.0 1.1149e+01 1.0 2.45e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 49 0 0 0 1 49 0 0 0 220 MatLUFactorNum 2 1.0 2.5282e-03 1.0 1.14e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 45 MatILUFactorSym 2 1.0 3.0980e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 20001 1.0 1.8443e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 20001 1.0 4.5861e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 MatGetRowIJ 2 1.0 2.8610e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 2 1.0 2.2390e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 2 1.0 8.2610e-03 1.0 1.14e+05 1.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 0 14 PCApply 43141 1.0 1.1242e+01 1.0 2.45e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 49 0 0 0 1 49 0 0 0 218 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Krylov Solver 2 2 36112 0 Vec 73 73 3794248 0 Matrix 7 5 2412812 0 Preconditioner 2 2 1520 0 Index Set 6 6 155232 0 ======================================================================================================================== Average time to get PetscTime(): 0 #PETSc Option Table entries: -log_summary #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 Configure run at: Thu Feb 2 15:41:42 2012 Configure options: FOPTFLAGS=-O3 ----------------------------------------- Libraries compiled on Thu Feb 2 15:43:28 EST 2012 on babylon1 Machine characteristics: Linux babylon1 2.6.32-37-generic #81-Ubuntu SMP Fri Dec 2 20:32:42 UTC 2011 x86_64 GNU/Linux Using PETSc directory: /thayerfs/research/anatoly/NAN/Petsc/petsc-3.1-p8 Using PETSc arch: linux-gnu-c-debug ----------------------------------------- Using C compiler: mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -g3 Using Fortran compiler: mpif90 -Wall -Wno-unused-variable -O3 ----------------------------------------- Using include paths: -I/thayerfs/research/anatoly/NAN/Petsc/petsc-3.1-p8/linux-gnu-c-debug/include -I/thayerfs/research/anatoly/NAN/Petsc/petsc-3.1-p8/include -I/usr/lib/openmpi/include -I/usr/lib/openmpi/lib ------------------------------------------ Using C linker: mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -g3 Using Fortran linker: mpif90 -Wall -Wno-unused-variable -O3 Using libraries: -Wl,-rpath,/thayerfs/research/anatoly/NAN/Petsc/petsc-3.1-p8/linux-gnu-c-debug/lib -L/thayerfs/research/anatoly/NAN/Petsc/petsc-3.1-p8/linux-gnu-c-debug/lib -lpetsc -lX11 -llapack -lblas -L/usr/lib/openmpi/lib -L/thayerfs/apps/intel/Compiler/11.0/081/ipp/em64t/lib -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -L/thayerfs/apps/intel/Compiler/11.0/081/mkl/lib/em64t -L/mnt/thayerfs/anatoly/NAN/Petsc/petsc-3.1-p8 -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -lmpi_f90 -lmpi_f77 -lgfortran -lm -lm -lm -lm -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl ------------------------------------------ From jedbrown at mcs.anl.gov Fri Mar 16 15:05:24 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 16 Mar 2012 15:05:24 -0500 Subject: [petsc-users] Petsc Performance Help In-Reply-To: <20120316155200.rbvlenhc2sw8ccs4@webmail.dartmouth.edu> References: <20120316155200.rbvlenhc2sw8ccs4@webmail.dartmouth.edu> Message-ID: On Fri, Mar 16, 2012 at 14:52, wrote: > Dear PETSc Group, > I am tuning the efficiency of my PETSc code for a while, but get very > little progress. So can anyone help me to analysis the log? Any suggestions > will be appreciated. > > My problem is time dependent. At every time step, two about 6000 by 6000 > sparse matrices need to be solved, which come from a Poisson equation. I > use both sequential and parallel AIJ format to store matrices, but the > performances are both not very good. > 1. You need to heed the huge warning ########################################################## # # # WARNING!!! # # # # This code was compiled with a debugging option, # # To get timing results run config/configure.py # # using --with-debugging=no, the performance will # # be generally two or three times faster. # # # ########################################################## 2. You only spend 18 of 880 seconds in the solver. What do you want? 3. The problem is too small to get significant parallel speedup. http://www.mcs.anl.gov/petsc/documentation/faq.html#slowerparallel -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhyshr at mcs.anl.gov Fri Mar 16 15:28:55 2012 From: abhyshr at mcs.anl.gov (Shri) Date: Fri, 16 Mar 2012 15:28:55 -0500 (CDT) Subject: [petsc-users] Petsc Performance Help In-Reply-To: Message-ID: <1615930846.28121.1331929735274.JavaMail.root@zimbra-mb2.anl.gov> As Jed points out, most of the time (~98%) is spent in your code rather than in the PETSc solver. Clearly, you need to profile your own source code and figure out where the most time is spent. You can do this using the tools for user code profiling (PetscLogEvent and PetscLogStage) in PETSc. Read Sections 11.2 and 11.3 of the User's Manual for more information on PetscLogEvent and PetscLogStage and also look at examples such as http://www.mcs.anl.gov/petsc/petsc-current/src/vec/vec/examples/tutorials/ex5.c.html http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex2.c.html http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex46.c.html Shri ----- Original Message ----- > On Fri, Mar 16, 2012 at 14:52, < Nan.Jia at dartmouth.edu > wrote: > > Dear PETSc Group, > > I am tuning the efficiency of my PETSc code for a while, but get > > very > > little progress. So can anyone help me to analysis the log? Any > > suggestions will be appreciated. > > My problem is time dependent. At every time step, two about 6000 by > > 6000 sparse matrices need to be solved, which come from a Poisson > > equation. I use both sequential and parallel AIJ format to store > > matrices, but the performances are both not very good. > 1. You need to heed the huge warning > ########################################################## > # # > # WARNING!!! # > # # > # This code was compiled with a debugging option, # > # To get timing results run config/configure.py # > # using --with-debugging=no, the performance will # > # be generally two or three times faster. # > # # > ########################################################## > 2. You only spend 18 of 880 seconds in the solver. What do you want? > 3. The problem is too small to get significant parallel speedup. > http://www.mcs.anl.gov/petsc/documentation/faq.html#slowerparallel -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Mar 16 15:53:41 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 16 Mar 2012 15:53:41 -0500 (CDT) Subject: [petsc-users] Petsc Performance Help In-Reply-To: References: <20120316155200.rbvlenhc2sw8ccs4@webmail.dartmouth.edu> Message-ID: On Fri, 16 Mar 2012, Jed Brown wrote: > On Fri, Mar 16, 2012 at 14:52, wrote: > > > Dear PETSc Group, > > I am tuning the efficiency of my PETSc code for a while, but get very > > little progress. So can anyone help me to analysis the log? Any suggestions > > will be appreciated. > > > > My problem is time dependent. At every time step, two about 6000 by 6000 > > sparse matrices need to be solved, which come from a Poisson equation. I > > use both sequential and parallel AIJ format to store matrices, but the > > performances are both not very good. > > > > 1. You need to heed the huge warning > > ########################################################## > # # > # WARNING!!! # > # # > # This code was compiled with a debugging option, # > # To get timing results run config/configure.py # > # using --with-debugging=no, the performance will # > # be generally two or three times faster. # > # # > ########################################################## > > > > 2. You only spend 18 of 880 seconds in the solver. What do you want? > > 3. The problem is too small to get significant parallel speedup. > > http://www.mcs.anl.gov/petsc/documentation/faq.html#slowerparallel 4. MatSetValues() is not logged by default - you should run with -info and make sure there are no mallocs during assembly. Also check http://www.mcs.anl.gov/petsc/documentation/faq.html#efficient-assembly satish From xavier.garnaud at ladhyx.polytechnique.fr Sun Mar 18 09:43:45 2012 From: xavier.garnaud at ladhyx.polytechnique.fr (Xavier Garnaud) Date: Sun, 18 Mar 2012 15:43:45 +0100 Subject: [petsc-users] real and complex numbers Message-ID: Dear PETSc team, I am performing nonlinear and linearized computations. Nonlinear computations are performed using real numbers (it would be approximately twice slower to use complex numbers instead). For linearized computations however, it is much more convenient to use complex numbers. These two types of computations are performed separately: first a run a nonlinear simulation, I save the result on the disk, and then I load it in a different code to perform linear operations. I would therefore like to be able to use VecView using a version of PETSc compiled with real numbers and VecLoad using a version compiled for complex numbers. I would like to do this using binary or HDF5 format. I tried to create a real vector with twice the size of my non-linear result and fill every other element with a 0, but it does not work. Is there a way to do this using existing functions? Thank you very much, Sincerely, Xavier -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Mar 18 10:17:02 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 18 Mar 2012 10:17:02 -0500 Subject: [petsc-users] real and complex numbers In-Reply-To: References: Message-ID: On Sun, Mar 18, 2012 at 9:43 AM, Xavier Garnaud < xavier.garnaud at ladhyx.polytechnique.fr> wrote: > Dear PETSc team, > > I am performing nonlinear and linearized computations. Nonlinear > computations are performed using real numbers (it would be approximately > twice slower to use complex numbers instead). For linearized computations > however, it is much more convenient to use complex numbers. These two types > of computations are performed separately: first a run a nonlinear > simulation, I save the result on the disk, and then I load it in a > different code to perform linear operations. > I would therefore like to be able to use VecView using a version of PETSc > compiled with real numbers and VecLoad using a version compiled for complex > numbers. I would like to do this using binary or HDF5 format. I tried to > create a real vector with twice the size of my non-linear result and fill > every other element with a 0, but it does not work. Is there a way to do > this using existing functions? > Thank you very much, > I think the easiest thing to do here is write a custom viewer (start with binary) to writes complex numbers. Matt > Sincerely, > > Xavier > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Mar 18 14:52:41 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 18 Mar 2012 14:52:41 -0500 Subject: [petsc-users] real and complex numbers In-Reply-To: References: Message-ID: <8CF5B21F-394A-4704-814E-ECFD95083495@mcs.anl.gov> Just run your single code in complex numbers (where the nonlinear part may have zero imaginary part). You are introducing a complicated work flow (saving to files ...) (that is also not well supported in PETSc) just to save some work on the nonlinear part when generally most nonlinear solvers spend much more time in the linear solve than the nonlinear computations. Make your life easy, just use complex numbers; your time is much more valuable than the computers. Barry On Mar 18, 2012, at 9:43 AM, Xavier Garnaud wrote: > Dear PETSc team, > > I am performing nonlinear and linearized computations. Nonlinear computations are performed using real numbers (it would be approximately twice slower to use complex numbers instead). For linearized computations however, it is much more convenient to use complex numbers. These two types of computations are performed separately: first a run a nonlinear simulation, I save the result on the disk, and then I load it in a different code to perform linear operations. > I would therefore like to be able to use VecView using a version of PETSc compiled with real numbers and VecLoad using a version compiled for complex numbers. I would like to do this using binary or HDF5 format. I tried to create a real vector with twice the size of my non-linear result and fill every other element with a 0, but it does not work. Is there a way to do this using existing functions? > Thank you very much, > > Sincerely, > > Xavier From jedbrown at mcs.anl.gov Sun Mar 18 14:57:42 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 18 Mar 2012 14:57:42 -0500 Subject: [petsc-users] real and complex numbers In-Reply-To: <8CF5B21F-394A-4704-814E-ECFD95083495@mcs.anl.gov> References: <8CF5B21F-394A-4704-814E-ECFD95083495@mcs.anl.gov> Message-ID: On Sun, Mar 18, 2012 at 14:52, Barry Smith wrote: > Just run your single code in complex numbers (where the nonlinear part > may have zero imaginary part). You are introducing a complicated work flow > (saving to files ...) (that is also not well supported in PETSc) just to > save some work on the nonlinear part when generally most nonlinear solvers > spend much more time in the linear solve than the nonlinear computations. > Make your life easy, just use complex numbers; your time is much more > valuable than the computers. Also, if the nonlinear function is very expensive, you can read the state from the Vec into PetscReal locally (e.g. one stencil at a time) and compute with reals. Writing to files creates a _serious_ bottleneck that will likely take orders of magnitude longer than everything else your code does if you scale it to large problems/computers. -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Mon Mar 19 08:25:41 2012 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Mon, 19 Mar 2012 13:25:41 +0000 Subject: [petsc-users] function name and meaning of -pc_fieldsplit_schur_factorization_type Message-ID: > > 2) It would be nice to know what exactly is being dropped. > > is obvious and apparently drops the right most matrix, but > > what do lower and upper drop? > > > > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCFIELDSPLIT.html > > > > 3) I'm confused about the minus sign in the form and its > > motivation. It doesn't come from the factorization. Is the rhs > > multiplied with a minus sign as well? Why not follow the sign > > convention of Elman's taxonomy (eq 3 and 4, JCP 227, 2008)? > > > > It makes the PC positive definite. > > Matt Thanks Matt, these petsc-dev manual pages are much clearer than current. dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From C.Klaij at marin.nl Mon Mar 19 08:29:54 2012 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Mon, 19 Mar 2012 13:29:54 +0000 Subject: [petsc-users] function name and meaning of -pc_fieldsplit_schur_factorization_type Message-ID: > > The manual page 87 states that "there are several variants > > of the Schur complement preconditioner obtained by dropping some > > of the terms, these can be obtained with > > -pc_fieldsplit_schur_factorization_type " > > > > 1) What is (will be) the name of the corresponding function? > > > > It looks like PCFieldSplitSetSchurFactorizationType() is missing. Would be nice to have. > > > > > > 3) I'm confused about the minus sign in the form and its > > motivation. It doesn't come from the factorization. Is the rhs > > multiplied with a minus sign as well? Why not follow the sign > > convention of Elman's taxonomy (eq 3 and 4, JCP 227, 2008)? > > > > Elman's convention is rather specific to finite element discretization of > incompressible flow. He arbitrarily slaps a negative on one of the matrices > in the system because it represents a certain kind of stabilization. That > minus sign would be pretty silly if the matrix was actually positive (mixed > elasticity, finite-time step stiff waves, etc). > > The negative sign in the DIAG form appears because the minimal polynomial > with that preconditioner has degree 3 (for a nonsingular problem) if you > have a minus sign there. Read Murphy's note: > http://eprints.maths.ox.ac.uk/1291/1/NA-99-07.pdf > > If you skip that minus sign, that property goes away. Also, if you are > solving Stokes flow or constrained problems in elasticity, that sign > convention causes the DIAG form to be positive definite which means that > you could use it with MINRES which was is the main motivation for using > DIAG in the first place. Jed, thanks for pointing me to Murphy's note, I didn't realize we were allowed to choose the sign in this case. It's rather counter-intuitive to have a preconditioner which is not an approximate inverse. dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From xavier.garnaud at ladhyx.polytechnique.fr Mon Mar 19 13:15:09 2012 From: xavier.garnaud at ladhyx.polytechnique.fr (Xavier Garnaud) Date: Mon, 19 Mar 2012 19:15:09 +0100 Subject: [petsc-users] real and complex numbers In-Reply-To: References: <8CF5B21F-394A-4704-814E-ECFD95083495@mcs.anl.gov> Message-ID: Thank you for your help. Using h5py I finally wrote a 10 lines python script that translates a real hdf5 file to a complex one. On Sun, Mar 18, 2012 at 8:57 PM, Jed Brown wrote: > On Sun, Mar 18, 2012 at 14:52, Barry Smith wrote: > >> Just run your single code in complex numbers (where the nonlinear part >> may have zero imaginary part). You are introducing a complicated work flow >> (saving to files ...) (that is also not well supported in PETSc) just to >> save some work on the nonlinear part when generally most nonlinear solvers >> spend much more time in the linear solve than the nonlinear computations. >> Make your life easy, just use complex numbers; your time is much more >> valuable than the computers. > > > Also, if the nonlinear function is very expensive, you can read the state > from the Vec into PetscReal locally (e.g. one stencil at a time) and > compute with reals. > > Writing to files creates a _serious_ bottleneck that will likely take > orders of magnitude longer than everything else your code does if you scale > it to large problems/computers. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From torres.pedrozpk at gmail.com Mon Mar 19 15:47:49 2012 From: torres.pedrozpk at gmail.com (Pedro Torres) Date: Mon, 19 Mar 2012 17:47:49 -0300 Subject: [petsc-users] Solving a Linear System with two different Solvers (one each thread)..? Message-ID: Hello Petsc Team, I has some experience with PETSc + MPI, but neither with PETSc+Pthreads nor Pthreads alone. Taking in to account that pthread was recenltly added to petsc, is it possible to code a program that solve the same linear system with two different solvers, one on each thread?. I guess that can be done calling different ksp object, one on each thread, but I don't have clear, first if this is possible and second how to do that. Sorry if my question is meaningless, I just started to study pthreads. I will really appreciated any advice. Thanks in advance. Best Regards -- Pedro Torres -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Mar 19 16:02:46 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 19 Mar 2012 16:02:46 -0500 Subject: [petsc-users] Solving a Linear System with two different Solvers (one each thread)..? In-Reply-To: References: Message-ID: On Mon, Mar 19, 2012 at 15:47, Pedro Torres wrote: > I has some experience with PETSc + MPI, but neither with PETSc+Pthreads > nor Pthreads alone. Taking in to account that pthread was recenltly added > to petsc, is it possible to code a program that solve the same linear > system with two different solvers, one on each thread?. I guess that can be > done calling different ksp object, one on each thread, but I don't have > clear, first if this is possible and second how to do that. Sorry if my > question is meaningless, I just started to study pthreads. > > I will really appreciated any advice. Thanks in advance. > No, the threading model does not work like this (and the performance of a system that did work that way would be a delicate beast). -------------- next part -------------- An HTML attachment was scrubbed... URL: From torres.pedrozpk at gmail.com Mon Mar 19 16:16:32 2012 From: torres.pedrozpk at gmail.com (Pedro Torres) Date: Mon, 19 Mar 2012 18:16:32 -0300 Subject: [petsc-users] Solving a Linear System with two different Solvers (one each thread)..? In-Reply-To: References: Message-ID: Jed, thanks for your response. 2012/3/19 Jed Brown > On Mon, Mar 19, 2012 at 15:47, Pedro Torres wrote: > >> I has some experience with PETSc + MPI, but neither with PETSc+Pthreads >> nor Pthreads alone. Taking in to account that pthread was recenltly added >> to petsc, is it possible to code a program that solve the same linear >> system with two different solvers, one on each thread?. I guess that can be >> done calling different ksp object, one on each thread, but I don't have >> clear, first if this is possible and second how to do that. Sorry if my >> question is meaningless, I just started to study pthreads. >> >> I will really appreciated any advice. Thanks in advance. >> > > No, the threading model does not work like this (and the performance of a > system that did work that way would be a delicate beast). > Ok, so the only way to implement that is coding by hand each algorithm for each thread, rigth?. Anyway, at least can I use petsc object like Vec/Mat on each thread?. Thanks for your help! -- Pedro Torres -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhyshr at mcs.anl.gov Mon Mar 19 16:18:57 2012 From: abhyshr at mcs.anl.gov (Shri) Date: Mon, 19 Mar 2012 16:18:57 -0500 (CDT) Subject: [petsc-users] Solving a Linear System with two different Solvers (one each thread)..? In-Reply-To: Message-ID: <670639195.37020.1332191937294.JavaMail.root@zimbra-mb2.anl.gov> Pedro, > Taking in to account that pthread was recenltly added to petsc, is it possible to code a program that solve the same linear system with two different solvers, one on each thread?. I guess that can be done calling different ksp object, one on each thread, but I don't have clear, first if this is possible and second how to do that. No, this is not possible currently and may not be available in the future too. I don't think there may be any performance benefit by solving the same linear system using different KSPs on different threads. What are you trying to do using pthreads in PETSc? Shri ----- Original Message ----- Hello Petsc Team, I has some experience with PETSc + MPI, but neither with PETSc+Pthreads nor Pthreads alone. Taking in to account that pthread was recenltly added to petsc, is it possible to code a program that solve the same linear system with two different solvers, one on each thread?. I guess that can be done calling different ksp object, one on each thread, but I don't have clear, first if this is possible and second how to do that. Sorry if my question is meaningless, I just started to study pthreads. I will really appreciated any advice. Thanks in advance. Best Regards -- Pedro Torres -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Mar 19 16:22:08 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 19 Mar 2012 16:22:08 -0500 Subject: [petsc-users] Solving a Linear System with two different Solvers (one each thread)..? In-Reply-To: References: Message-ID: On Mon, Mar 19, 2012 at 16:16, Pedro Torres wrote: > Ok, so the only way to implement that is coding by hand each algorithm for > each thread, rigth?. > > Anyway, at least can I use petsc object like Vec/Mat on each thread?. > No, the threading model in PETSc does not allow you to call it simultaneously from different threads. You can set locks around every call, but this whole approach is almost certainly a waste of your time. Just solve those systems one after the other. It will be much simpler code and faster in many cases. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhyshr at mcs.anl.gov Mon Mar 19 16:32:54 2012 From: abhyshr at mcs.anl.gov (Shri) Date: Mon, 19 Mar 2012 16:32:54 -0500 (CDT) Subject: [petsc-users] Solving a Linear System with two different Solvers (one each thread)..? In-Reply-To: Message-ID: <1496045599.37153.1332192774219.JavaMail.root@zimbra-mb2.anl.gov> I'm not sure what you mean by 'coding by hand each algorithm for each thread'. Do you want an entire Vec/Mat on each thread? This is not how our model works currently and doing so will require considerable time and effort and may not be the worth it. The way the pthread model in PETSc works is that every pthread vector/matrix uses threads from a thread pool for its operations. You can control how many and which threads you want from the thread pool. Shri ----- Original Message ----- Jed, thanks for your response. 2012/3/19 Jed Brown < jedbrown at mcs.anl.gov >
On Mon, Mar 19, 2012 at 15:47, Pedro Torres < torres.pedrozpk at gmail.com > wrote:
I has some experience with PETSc + MPI, but neither with PETSc+Pthreads nor Pthreads alone. Taking in to account that pthread was recenltly added to petsc, is it possible to code a program that solve the same linear system with two different solvers, one on each thread?. I guess that can be done calling different ksp object, one on each thread, but I don't have clear, first if this is possible and second how to do that. Sorry if my question is meaningless, I just started to study pthreads. I will really appreciated any advice. Thanks in advance. No, the threading model does not work like this (and the performance of a system that did work that way would be a delicate beast).
Ok, so the only way to implement that is coding by hand each algorithm for each thread, rigth?. Anyway, at least can I use petsc object like Vec/Mat on each thread?. Thanks for your help! -- Pedro Torres
-------------- next part -------------- An HTML attachment was scrubbed... URL: From maxwellr at gmail.com Mon Mar 19 17:06:08 2012 From: maxwellr at gmail.com (Max Rudolph) Date: Mon, 19 Mar 2012 15:06:08 -0700 Subject: [petsc-users] -ksp_diagonal_scale question Message-ID: I was looking at the documentation for KSPSetDiagonalScale and it contains this comment: "This routine is only used if the matrix and preconditioner matrix are the same thing." However,?I have found that even if the matrix and preconditioner are different, using the -ksp_diagonal_scale command line argument changes the convergence behavior of the solvers. Is the man page correct? Also, can you point me towards documentation of the type of scaling that is applied? Thanks for your help, Max Rudolph From mirzadeh at gmail.com Mon Mar 19 19:19:28 2012 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Mon, 19 Mar 2012 17:19:28 -0700 Subject: [petsc-users] AOCreateMapping Message-ID: Hi, When using AOCreateMapping, it is not possible to ask for mapping of indecies that do not exist in the AO -- You get "[0]PETSC ERROR: Argument out of range!" : [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Invalid input index 21! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./petsc on a arch-linu named mohammad-laptop by mohammad Mon Mar 19 17:16:13 2012 [0]PETSC ERROR: Libraries linked from /home/mohammad/soft/petsc-3.2-p6/arch-linux2-cxx-debug/lib [0]PETSC ERROR: Configure run at Thu Feb 16 02:16:40 2012 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-clanguage=cxx --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 --download-ml=1 --with-parmetis-include=/home/mohammad/soft/parmetis/include --with-parmetis-lib="-L/home/mohammad/soft/parmetis/lib -lparmetis -lmetis" --download-superlu_dist=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: AOApplicationToPetsc_Mapping() line 136 in /home/mohammad/soft/petsc-3.2-p6/src/dm/ao/impls/mapping/aomapping.c [0]PETSC ERROR: AOApplicationToPetsc() line 250 in /home/mohammad/soft/petsc-3.2-p6/src/dm/ao/interface/ao.c Is there an easy way to have the AO return a flag integer (like -1) for nodes that do not exist in the AO? Mohammad -------------- next part -------------- An HTML attachment was scrubbed... URL: From pengxwang at hotmail.com Mon Mar 19 21:18:07 2012 From: pengxwang at hotmail.com (Roc Wang) Date: Mon, 19 Mar 2012 21:18:07 -0500 Subject: [petsc-users] errors in compiling ~\petsc-3.2-p7\src\ksp\ksp\examples\tutorials\ex2f.F Message-ID: Hello, I am trying to compile the example source code of ~\petsc-3.2-p7\src\ksp\ksp\examples\tutorials\ex2f.F. However the error information shows: ex2f.o: In function `MAIN__':ex2f.F90:(.text+0x1b): undefined reference to `_gfortran_set_std'ex2f.F90:(.text+0x6f1): undefined reference to `_gfortran_st_write'ex2f.F90:(.text+0x706): undefined reference to `_gfortran_transfer_real'ex2f.F90:(.text+0x71b): undefined reference to `_gfortran_transfer_integer'ex2f.F90:(.text+0x727): undefined reference to `_gfortran_st_write_done'ex2f.F90:(.text+0x773): undefined reference to `_gfortran_st_write'ex2f.F90:(.text+0x788): undefined reference to `_gfortran_transfer_integer'ex2f.F90:(.text+0x794): undefined reference to `_gfortran_st_write_done'ex2f.o: In function `mykspmonitor_':ex2f.F90:(.text+0x896): undefined reference to `_gfortran_st_write'ex2f.F90:(.text+0x8ae): undefined reference to `_gfortran_transfer_integer'ex2f.F90:(.text+0x8ba): undefined reference to `_gfortran_st_write_done'ex2f.F90:(.text+0x924): undefined reference to `_gfortran_st_write'ex2f.F90:(.text+0x93c): undefined reference to `_gfortran_transfer_integer'ex2f.F90:(.text+0x954): undefined reference to `_gfortran_transfer_real'ex2f.F90:(.text+0x960): undefined reference to `_gfortran_st_write_done'make: *** [ex2f] Error 1 My makefile is like this: ########################################################################PETSC_DIR =/usr/global/petsc/3.1-p8PETSC_ARCH =linux-intel11-debugFFLAGS = -I${PETSC_DIR}/include -I${PETSC_DIR}/${PETSC_ARCH}/includeLFLAGS = -L${PETSC_DIR}/${PETSC_ARCH}/lib -lpetsc\ -L/usr/global/intel/mkl/10.3.1.107/mkl/lib/intel64\ -Wl,-R/usr/global/intel/mkl/10.3.1.107/mkl/lib/intel64\ -lmkl_solver_lp64_sequential\ -Wl,--start-group -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -Wl,--end-group\ -lX11 FC = mpif90BIN = ex2fOBJS = ex2f.o ${BIN}: ${OBJS} ${FC} -o ex2f ${OBJS} ${LFLAGS} ex2f.o: ex2f.F90 ${FC} -c ${FFLAGS} ex2f.F90 clean: rm -f ex2f *.o Should there any more libraries be included in the makefile? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Mar 19 21:47:25 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 19 Mar 2012 21:47:25 -0500 Subject: [petsc-users] errors in compiling ~\petsc-3.2-p7\src\ksp\ksp\examples\tutorials\ex2f.F In-Reply-To: References: Message-ID: <3F85DDAD-6F7B-4E2B-9D6B-2F4C53613321@mcs.anl.gov> Run make ex5f on src/snes/examples/tutorials Does that compile and link correctly? Note all the libraries that being linked (they are printed to the screen) and make sure you have them all. Also note that you must use the same BLAS in building PETSc that you link against. So make sure you built PETSc with the mkl libraries. Barry On Mar 19, 2012, at 9:18 PM, Roc Wang wrote: > Hello, > > I am trying to compile the example source code of ~\petsc-3.2-p7\src\ksp\ksp\examples\tutorials\ex2f.F. However the error information shows: > > ex2f.o: In function `MAIN__': > ex2f.F90:(.text+0x1b): undefined reference to `_gfortran_set_std' > ex2f.F90:(.text+0x6f1): undefined reference to `_gfortran_st_write' > ex2f.F90:(.text+0x706): undefined reference to `_gfortran_transfer_real' > ex2f.F90:(.text+0x71b): undefined reference to `_gfortran_transfer_integer' > ex2f.F90:(.text+0x727): undefined reference to `_gfortran_st_write_done' > ex2f.F90:(.text+0x773): undefined reference to `_gfortran_st_write' > ex2f.F90:(.text+0x788): undefined reference to `_gfortran_transfer_integer' > ex2f.F90:(.text+0x794): undefined reference to `_gfortran_st_write_done' > ex2f.o: In function `mykspmonitor_': > ex2f.F90:(.text+0x896): undefined reference to `_gfortran_st_write' > ex2f.F90:(.text+0x8ae): undefined reference to `_gfortran_transfer_integer' > ex2f.F90:(.text+0x8ba): undefined reference to `_gfortran_st_write_done' > ex2f.F90:(.text+0x924): undefined reference to `_gfortran_st_write' > ex2f.F90:(.text+0x93c): undefined reference to `_gfortran_transfer_integer' > ex2f.F90:(.text+0x954): undefined reference to `_gfortran_transfer_real' > ex2f.F90:(.text+0x960): undefined reference to `_gfortran_st_write_done' > make: *** [ex2f] Error 1 > > > My makefile is like this: > > ######################################################################## > PETSC_DIR =/usr/global/petsc/3.1-p8 > PETSC_ARCH =linux-intel11-debug > FFLAGS = -I${PETSC_DIR}/include -I${PETSC_DIR}/${PETSC_ARCH}/include > LFLAGS = -L${PETSC_DIR}/${PETSC_ARCH}/lib -lpetsc\ > -L/usr/global/intel/mkl/10.3.1.107/mkl/lib/intel64\ > -Wl,-R/usr/global/intel/mkl/10.3.1.107/mkl/lib/intel64\ > -lmkl_solver_lp64_sequential\ > -Wl,--start-group -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -Wl,--end-group\ > -lX11 > > > FC = mpif90 > BIN = ex2f > OBJS = ex2f.o > > > ${BIN}: ${OBJS} > ${FC} -o ex2f ${OBJS} ${LFLAGS} > > ex2f.o: ex2f.F90 > ${FC} -c ${FFLAGS} ex2f.F90 > > clean: > rm -f ex2f *.o > > > Should there any more libraries be included in the makefile? From knepley at gmail.com Mon Mar 19 23:37:30 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 19 Mar 2012 23:37:30 -0500 Subject: [petsc-users] AOCreateMapping In-Reply-To: References: Message-ID: On Mon, Mar 19, 2012 at 7:19 PM, Mohammad Mirzadeh wrote: > Hi, > > When using AOCreateMapping, it is not possible to ask for mapping of > indecies that do not exist in the AO -- You get "[0]PETSC ERROR: Argument > out of range!" : > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Argument out of range! > [0]PETSC ERROR: Invalid input index 21! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 > CST 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./petsc on a arch-linu named mohammad-laptop by mohammad > Mon Mar 19 17:16:13 2012 > [0]PETSC ERROR: Libraries linked from > /home/mohammad/soft/petsc-3.2-p6/arch-linux2-cxx-debug/lib > [0]PETSC ERROR: Configure run at Thu Feb 16 02:16:40 2012 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --with-clanguage=cxx --download-f-blas-lapack=1 > --download-mpich=1 --download-hypre=1 --download-ml=1 > --with-parmetis-include=/home/mohammad/soft/parmetis/include > --with-parmetis-lib="-L/home/mohammad/soft/parmetis/lib -lparmetis -lmetis" > --download-superlu_dist=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: AOApplicationToPetsc_Mapping() line 136 in > /home/mohammad/soft/petsc-3.2-p6/src/dm/ao/impls/mapping/aomapping.c > [0]PETSC ERROR: AOApplicationToPetsc() line 250 in > /home/mohammad/soft/petsc-3.2-p6/src/dm/ao/interface/ao.c > > > Is there an easy way to have the AO return a flag integer (like -1) for > nodes that do not exist in the AO? > You can get the indices and check yourself. Matt > Mohammad > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From popov at uni-mainz.de Tue Mar 20 07:23:10 2012 From: popov at uni-mainz.de (Anton Popov) Date: Tue, 20 Mar 2012 13:23:10 +0100 Subject: [petsc-users] errors in compiling ~\petsc-3.2-p7\src\ksp\ksp\examples\tutorials\ex2f.F In-Reply-To: References: Message-ID: <4F6876AE.9030403@uni-mainz.de> Try adding -lgfortran to LFLAGS variable On 3/20/12 3:18 AM, Roc Wang wrote: > Hello, > > I am trying to compile the example source code of > ~\petsc-3.2-p7\src\ksp\ksp\examples\tutorials\ex2f.F. However the > error information shows: > > > ex2f.o: In function `MAIN__': > ex2f.F90:(.text+0x1b): undefined reference to `_gfortran_set_std' > ex2f.F90:(.text+0x6f1): undefined reference to `_gfortran_st_write' > ex2f.F90:(.text+0x706): undefined reference to `_gfortran_transfer_real' > ex2f.F90:(.text+0x71b): undefined reference to > `_gfortran_transfer_integer' > ex2f.F90:(.text+0x727): undefined reference to `_gfortran_st_write_done' > ex2f.F90:(.text+0x773): undefined reference to `_gfortran_st_write' > ex2f.F90:(.text+0x788): undefined reference to > `_gfortran_transfer_integer' > ex2f.F90:(.text+0x794): undefined reference to `_gfortran_st_write_done' > ex2f.o: In function `mykspmonitor_': > ex2f.F90:(.text+0x896): undefined reference to `_gfortran_st_write' > ex2f.F90:(.text+0x8ae): undefined reference to > `_gfortran_transfer_integer' > ex2f.F90:(.text+0x8ba): undefined reference to `_gfortran_st_write_done' > ex2f.F90:(.text+0x924): undefined reference to `_gfortran_st_write' > ex2f.F90:(.text+0x93c): undefined reference to > `_gfortran_transfer_integer' > ex2f.F90:(.text+0x954): undefined reference to `_gfortran_transfer_real' > ex2f.F90:(.text+0x960): undefined reference to `_gfortran_st_write_done' > make: *** [ex2f] Error 1 > > My makefile is like this: > > ######################################################################## > PETSC_DIR =/usr/global/petsc/3.1-p8 > PETSC_ARCH =linux-intel11-debug > FFLAGS = -I${PETSC_DIR}/include -I${PETSC_DIR}/${PETSC_ARCH}/include > LFLAGS = -L${PETSC_DIR}/${PETSC_ARCH}/lib -lpetsc\ > -L/usr/global/intel/mkl/10.3.1.107/mkl/lib/intel64\ > -Wl,-R/usr/global/intel/mkl/10.3.1.107/mkl/lib/intel64\ > -lmkl_solver_lp64_sequential\ > -Wl,--start-group -lmkl_intel_lp64 -lmkl_sequential -lmkl_core > -Wl,--end-group\ > -lX11 > > > FC = mpif90 > BIN = ex2f > OBJS = ex2f.o > > > ${BIN}: ${OBJS} > ${FC} -o ex2f ${OBJS} ${LFLAGS} > > ex2f.o: ex2f.F90 > ${FC} -c ${FFLAGS} ex2f.F90 > > clean: > rm -f ex2f *.o > > > Should there any more libraries be included in the makefile? -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.mousel at gmail.com Tue Mar 20 09:19:02 2012 From: john.mousel at gmail.com (John Mousel) Date: Tue, 20 Mar 2012 09:19:02 -0500 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> Message-ID: Mark, Sorry for the late reply. I've been on travel and hadn't had a chance to pick this back up. I've tried running with the suggested options: -ksp_type bcgsl -pc_type gamg -pc_gamg_coarse_eq_limit 10 -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_diagonal_scale -ksp_diagonal_scale_fix -ksp_monitor_true_residual -ksp_view -pc_gamg_verbose 1 With these options, the convergence starts to hang (see attached GAMG_kspview.txt). The hanging happens for both -mg_coarse_ksp_type richardson and preonly. It was my understanding from previous emails that using preonly made it so that only the preconditioner was run, which in this case would be 8 sweeps of SOR. If I get rid of the -pc_gamg_agg_nsmooths 1 (see GAMG_kspview_nosmooth.txt), the problem converges, but again the convergence is slow. Without this option, both Richardson and preonly converge in 172 iterations. Matt, I've checked and the problem does converge in the true residual using GAMG, ML, HYPRE, and ILU preconditioned BiCG. I explicitly ensure that a solution exists by projecting the rhs vector out of the nullity of the transpose of operator. John On Fri, Mar 16, 2012 at 2:04 PM, Mark F. Adams wrote: > John, did this get resolved? > Mark > > On Mar 15, 2012, at 4:24 PM, John Mousel wrote: > > Mark, > > Running without the options you mentioned before leads to slightly worse > performance (175 iterations). > I have not been able to get run coarse grid solve to work with LU while > running ML. It keeps experiencing a zero pivot, and all the combinations of > shifting i've tried haven't lead me anywhere, hence the SOR on the course > grid. Also, the ML manual suggests limiting the number of levels to 3 or 4 > and performing a few sweeps of an iterative method as opposed to a direct > solve. > > John > > On Thu, Mar 15, 2012 at 12:04 PM, Mark F. Adams wrote: > >> You also want: -pc_gamg_agg_nsmooths 1 >> >> You are running plain aggregation. If it is Poisson then smoothing is >> good. >> >> Is this problem singular? Can you try running ML with these parameters >> and see if its performance degrades? The ML implementation uses the PETSC >> infrastructure and uses a very similar algorithm to GAMG-SA. We should be >> able to get these two to match pretty well. >> >> Mark >> >> >> On Mar 15, 2012, at 12:21 PM, John Mousel wrote: >> >> Mark, >> >> I ran with those options removed (see the run options listed below). >> Things actually got slightly worse. Now it's up to 142 iterations. I have >> attached the ksp_view output. >> >> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale >> -ksp_diagonal_scale_fix -mg_levels_ksp_type richardson -mg_levels_pc_type >> sor -pc_gamg_verbose 1 >> >> >> John >> >> >> On Thu, Mar 15, 2012 at 10:55 AM, Mark F. Adams wrote: >> >>> John, can you run again with: -pc_gamg_verbose 1 >>> >>> And I would not use: -pc_mg_levels 4 -mg_coarse_ksp_type preonly >>> -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 >>> >>> 1) I think -mg_coarse_ksp_type preonly and -mg_coarse_pc_sor_its 8 do >>> not do what you think. I think this is the same as 1 iteration. I think >>> you want 'richardson' not 'preonly'. >>> >>> 2) Why are you using sor as the coarse solver? If your problem is >>> singular then you want to use as many levels as possible to get the coarse >>> grid to be tiny. I'm pretty sure HYPRE ignores the coarse solver >>> parameters. But ML uses them and it is converging well. >>> >>> 3) I would not specify the number of levels. GAMG, and I think the >>> rest, have internal logic for stopping a the right level. If the coarse >>> level is large and you use just 8 iterations of sor then convergence will >>> suffer. >>> >>> Mark >>> >>> On Mar 15, 2012, at 11:13 AM, John Mousel wrote: >>> >>> Mark, >>> >>> The changes pulled through this morning. I've run it with the options >>> >>> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale >>> -ksp_diagonal_scale_fix -pc_mg_levels 4 -mg_levels_ksp_type richardson >>> -mg_levels_pc_type sor -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor >>> -mg_coarse_pc_sor_its 8 >>> >>> and it converges in the true residual, but it's not converging as fast >>> as anticpated. The matrix arises from a non-symmetric discretization of the >>> Poisson equation. The solve takes GAMG 114 iterations, whereas ML takes 24 >>> iterations, BoomerAMG takes 22 iterations, and -ksp_type bcgsl -pc_type >>> bjacobi -sub_pc_type ilu -sub_pc_factor_levels 4 takes around 170. I've >>> attached the -ksp_view results for ML,GAMG, and HYPRE. I've attempted to >>> make all the options the same on all levels for ML and GAMG. >>> >>> Any thoughts? >>> >>> John >>> >>> >>> On Wed, Mar 14, 2012 at 6:04 PM, Mark F. Adams wrote: >>> >>>> Humm, I see it with hg view (appended). >>>> >>>> Satish, my main repo looks hosed. I see this: >>>> >>>> ~/Codes/petsc-dev>hg update >>>> abort: crosses branches (merge branches or use --clean to discard >>>> changes) >>>> ~/Codes/petsc-dev>hg merge >>>> abort: branch 'default' has 3 heads - please merge with an explicit rev >>>> (run 'hg heads .' to see heads) >>>> ~/Codes/petsc-dev>hg heads >>>> changeset: 22496:8e2a98268179 >>>> tag: tip >>>> user: Barry Smith >>>> date: Wed Mar 14 16:42:25 2012 -0500 >>>> files: src/vec/is/interface/f90-custom/zindexf90.c >>>> src/vec/vec/interface/f90-custom/zvectorf90.c >>>> description: >>>> undoing manually changes I put in because Satish had a better fix >>>> >>>> >>>> changeset: 22492:bda4df63072d >>>> user: Mark F. Adams >>>> date: Wed Mar 14 17:39:52 2012 -0400 >>>> files: src/ksp/pc/impls/gamg/tools.c >>>> description: >>>> fix for unsymmetric matrices. >>>> >>>> >>>> changeset: 22469:b063baf366e4 >>>> user: Mark F. Adams >>>> date: Wed Mar 14 14:22:28 2012 -0400 >>>> files: src/ksp/pc/impls/gamg/tools.c >>>> description: >>>> added fix for preallocation for unsymetric matrices. >>>> >>>> Mark >>>> >>>> my 'hg view' on my merge repo: >>>> >>>> Revision: 22492 >>>> Branch: default >>>> Author: Mark F. Adams 2012-03-14 17:39:52 >>>> Committer: Mark F. Adams 2012-03-14 17:39:52 >>>> Tags: tip >>>> Parent: 22491:451bbbd291c2 (Small fixes to the BT linesearch) >>>> >>>> fix for unsymmetric matrices. >>>> >>>> >>>> ------------------------ src/ksp/pc/impls/gamg/tools.c >>>> ------------------------ >>>> @@ -103,7 +103,7 @@ >>>> PetscErrorCode ierr; >>>> PetscInt Istart,Iend,Ii,jj,ncols,nnz0,nnz1, NN, MM, nloc; >>>> PetscMPIInt mype, npe; >>>> - Mat Gmat = *a_Gmat, tGmat; >>>> + Mat Gmat = *a_Gmat, tGmat, matTrans; >>>> MPI_Comm wcomm = ((PetscObject)Gmat)->comm; >>>> const PetscScalar *vals; >>>> const PetscInt *idx; >>>> @@ -127,6 +127,10 @@ >>>> ierr = MatDiagonalScale( Gmat, diag, diag ); CHKERRQ(ierr); >>>> ierr = VecDestroy( &diag ); CHKERRQ(ierr); >>>> >>>> + if( symm ) { >>>> + ierr = MatTranspose( Gmat, MAT_INITIAL_MATRIX, &matTrans ); >>>> CHKERRQ(ierr); >>>> + } >>>> + >>>> /* filter - dup zeros out matrix */ >>>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &d_nnz ); CHKERRQ(ierr); >>>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &o_nnz ); CHKERRQ(ierr); >>>> @@ -135,6 +139,12 @@ >>>> d_nnz[jj] = ncols; >>>> o_nnz[jj] = ncols; >>>> ierr = MatRestoreRow(Gmat,Ii,&ncols,PETSC_NULL,PETSC_NULL); >>>> CHKERRQ(ierr); >>>> + if( symm ) { >>>> + ierr = MatGetRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); >>>> CHKERRQ(ierr); >>>> + d_nnz[jj] += ncols; >>>> + o_nnz[jj] += ncols; >>>> + ierr = MatRestoreRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); >>>> CHKERRQ(ierr); >>>> + } >>>> if( d_nnz[jj] > nloc ) d_nnz[jj] = nloc; >>>> if( o_nnz[jj] > (MM-nloc) ) o_nnz[jj] = MM - nloc; >>>> } >>>> @@ -142,6 +152,9 @@ >>>> CHKERRQ(ierr); >>>> ierr = PetscFree( d_nnz ); CHKERRQ(ierr); >>>> ierr = PetscFree( o_nnz ); CHKERRQ(ierr); >>>> + if( symm ) { >>>> + ierr = MatDestroy( &matTrans ); CHKERRQ(ierr); >>>> + } >>>> >>>> >>>> >>>> >>>> On Mar 14, 2012, at 5:53 PM, John Mousel wrote: >>>> >>>> Mark, >>>> >>>> No change. Can you give me the location that you patched so I can check >>>> to make sure it pulled? >>>> I don't see it on the petsc-dev change log. >>>> >>>> John >>>> >>>> On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams >>> > wrote: >>>> >>>>> John, I've committed these changes, give a try. >>>>> >>>>> Mark >>>>> >>>>> On Mar 14, 2012, at 3:46 PM, Satish Balay wrote: >>>>> >>>>> > This is the usual merge [with uncommited changes] issue. >>>>> > >>>>> > You could use 'hg shelf' extension to shelve your local changes and >>>>> > then do a merge [as Sean would suggest] - or do the merge in a >>>>> > separate/clean clone [I normally do this..] >>>>> > >>>>> > i.e >>>>> > cd ~/Codes >>>>> > hg clone petsc-dev petsc-dev-merge >>>>> > cd petsc-dev-merge >>>>> > hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just to >>>>> be sure, look for latest chagnes before merge.. >>>>> > hg merge >>>>> > hg commit >>>>> > hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev >>>>> > >>>>> > [now update your petsc-dev to latest] >>>>> > cd ~/Codes/petsc-dev >>>>> > hg pull >>>>> > hg update >>>>> > >>>>> > Satish >>>>> > >>>>> > On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>>> > >>>>> >> Great, that seems to work. >>>>> >> >>>>> >> I did a 'hg commit tools.c' >>>>> >> >>>>> >> and I want to push this file only. I guess its the only thing in >>>>> the change set so 'hg push' should be fine. But I see this: >>>>> >> >>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update >>>>> >> abort: crosses branches (merge branches or use --clean to discard >>>>> changes) >>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge >>>>> >> abort: outstanding uncommitted changes (use 'hg status' to list >>>>> changes) >>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status >>>>> >> M include/petscmat.h >>>>> >> M include/private/matimpl.h >>>>> >> M src/ksp/pc/impls/gamg/agg.c >>>>> >> M src/ksp/pc/impls/gamg/gamg.c >>>>> >> M src/ksp/pc/impls/gamg/gamg.h >>>>> >> M src/ksp/pc/impls/gamg/geo.c >>>>> >> M src/mat/coarsen/coarsen.c >>>>> >> M src/mat/coarsen/impls/hem/hem.c >>>>> >> M src/mat/coarsen/impls/mis/mis.c >>>>> >> >>>>> >> Am I ready to do a push? >>>>> >> >>>>> >> Thanks, >>>>> >> Mark >>>>> >> >>>>> >> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: >>>>> >> >>>>> >>> If commit is the last hg operation that you've done - then 'hg >>>>> rollback' would undo this commit. >>>>> >>> >>>>> >>> Satish >>>>> >>> >>>>> >>> On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>>> >>> >>>>> >>>> Damn, I'm not preallocating the graph perfectly for unsymmetric >>>>> matrices and PETSc now dies on this. >>>>> >>>> >>>>> >>>> I have a fix but I committed it with other changes that I do not >>>>> want to commit. The changes are all in one file so I should be able to >>>>> just commit this file. >>>>> >>>> >>>>> >>>> Anyone know how to delete a commit? >>>>> >>>> >>>>> >>>> I've tried: >>>>> >>>> >>>>> >>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip >>>>> 22487:26ffb9eef17f >>>>> >>>> hg: unknown command 'strip' >>>>> >>>> 'strip' is provided by the following extension: >>>>> >>>> >>>>> >>>> mq manage a stack of patches >>>>> >>>> >>>>> >>>> use "hg help extensions" for information on enabling extensions >>>>> >>>> >>>>> >>>> But have not figured out how to load extensions. >>>>> >>>> >>>>> >>>> Mark >>>>> >>>> >>>>> >>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: >>>>> >>>> >>>>> >>>>> Mark, >>>>> >>>>> >>>>> >>>>> I have a non-symmetric matrix. I am running with the following >>>>> options. >>>>> >>>>> >>>>> >>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual >>>>> >>>>> >>>>> >>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new malloc >>>>> error: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> 0]PETSC ERROR: --------------------- Error Message >>>>> ------------------------------------ >>>>> >>>>> [0]PETSC ERROR: Argument out of range! >>>>> >>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! >>>>> >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> >>>>> [0]PETSC ERROR: Petsc Development HG revision: >>>>> 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 >>>>> -0500 >>>>> >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble >>>>> shooting. >>>>> >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>> >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> >>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named >>>>> wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 >>>>> >>>>> [0]PETSC ERROR: Libraries linked from >>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib >>>>> >>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 >>>>> >>>>> [0]PETSC ERROR: Configure options --download-blacs=1 >>>>> --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 >>>>> --download-parmetis=1 --download-scalapack=1 >>>>> --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc >>>>> --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort >>>>> PETSC_ARCH=linux-debug >>>>> >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> >>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in >>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >>>>> >>>>> [0]PETSC ERROR: MatSetValues() line 1141 in >>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c >>>>> >>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in >>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c >>>>> >>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in >>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c >>>>> >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in >>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>>>> >>>>> [0]PETSC ERROR: PCSetUp() line 832 in >>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c >>>>> >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in >>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>> >>>>> [0]PETSC ERROR: KSPSolve() line 385 in >>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> John >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams < >>>>> mark.adams at columbia.edu> wrote: >>>>> >>>>> >>>>> >>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: >>>>> >>>>> >>>>> >>>>>> Mark, >>>>> >>>>>> >>>>> >>>>>> The matrix is asymmetric. Does this require the setting of an >>>>> option? >>>>> >>>>> >>>>> >>>>> Yes: -pc_gamg_sym_graph >>>>> >>>>> >>>>> >>>>> Mark >>>>> >>>>> >>>>> >>>>>> I pulled petsc-dev this morning, so I should have (at least >>>>> close to) the latest code. >>>>> >>>>>> >>>>> >>>>>> John >>>>> >>>>>> >>>>> >>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams < >>>>> mark.adams at columbia.edu> wrote: >>>>> >>>>>> >>>>> >>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >>>>> >>>>>> >>>>> >>>>>>> I'm getting the following error when using GAMG. >>>>> >>>>>>> >>>>> >>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: >>>>> Assertion `sgid==-1' failed. >>>>> >>>>>> >>>>> >>>>>> Is it possible that your matrix is structurally asymmetric? >>>>> >>>>>> >>>>> >>>>>> This code is evolving fast and so you will need to move to the >>>>> dev version if you are not already using it. (I think I fixed a bug that >>>>> hit this assert). >>>>> >>>>>> >>>>> >>>>>>> >>>>> >>>>>>> When I try to alter the type of aggregation at the command >>>>> line using -pc_gamg_type pa, I'm getting >>>>> >>>>>>> >>>>> >>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error >>>>> Message ------------------------------------ >>>>> >>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or >>>>> missing external package needed for type: >>>>> >>>>>>> see >>>>> http://www.mcs.anl.gov/petsc/documentation/installation.html#external! >>>>> >>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given! >>>>> >>>>>>> >>>>> >>>>>>> Has there been a change in the aggregation options? I just >>>>> pulled petsc-dev this morning. >>>>> >>>>>>> >>>>> >>>>>> >>>>> >>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg >>>>> for now. >>>>> >>>>>> >>>>> >>>>>> Mark >>>>> >>>>>> >>>>> >>>>>>> John >>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>>> >>>> >>>>> >>> >>>>> >>> >>>>> >> >>>>> >> >>>>> > >>>>> > >>>>> >>>>> >>>> >>>> >>> >>> >>> >>> >> >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- [0]PCSetUp_GAMG level 0 N=46330, n data rows=1, n data cols=1, nnz/row (ave)=1, np=4 [0]scaleFilterGraph 75.4421% nnz after filtering, with threshold 0.05, 6.96664 nnz ave. [0]maxIndSetAgg removed 0 of 10552 vertices. [0]PCGAMGprolongator_AGG New grid 5903 nodes PCGAMGoptprol_AGG smooth P0: max eigen=1.934105e+00 min=3.878260e-02 PC=jacobi [0]PCSetUp_GAMG 1) N=5903, n data cols=1, nnz/row (ave)=13, 4 active pes [0]scaleFilterGraph 50.3194% nnz after filtering, with threshold 0.05, 13.7456 nnz ave. [0]maxIndSetAgg removed 0 of 1321 vertices. [0]PCGAMGprolongator_AGG New grid 610 nodes PCGAMGoptprol_AGG smooth P0: max eigen=1.594176e+00 min=2.021619e-02 PC=jacobi [0]PCSetUp_GAMG 2) N=610, n data cols=1, nnz/row (ave)=21, 4 active pes [0]scaleFilterGraph 22.9069% nnz after filtering, with threshold 0.05, 21.3876 nnz ave. [0]maxIndSetAgg removed 0 of 129 vertices. [0]PCGAMGprolongator_AGG New grid 96 nodes PCGAMGoptprol_AGG smooth P0: max eigen=9.698899e+00 min=1.546704e-01 PC=jacobi [0]createLevel aggregate processors: npe: 4 --> 1, neq=96 [0]PCSetUp_GAMG 3) N=96, n data cols=1, nnz/row (ave)=39, 1 active pes [0]scaleFilterGraph 10.4645% nnz after filtering, with threshold 0.05, 39.0208 nnz ave. [0]maxIndSetAgg removed 0 of 96 vertices. [0]PCGAMGprolongator_AGG New grid 19 nodes PCGAMGoptprol_AGG smooth P0: max eigen=2.744061e+00 min=3.561007e-02 PC=jacobi [0]PCSetUp_GAMG 4) N=19, n data cols=1, nnz/row (ave)=18, 1 active pes [0]scaleFilterGraph 9.45559% nnz after filtering, with threshold 0.05, 18.3684 nnz ave. [0]maxIndSetAgg removed 7 of 19 vertices. [0]PCGAMGprolongator_AGG New grid 5 nodes PCGAMGoptprol_AGG smooth P0: max eigen=2.463197e+00 min=1.316806e-01 PC=jacobi [0]PCSetUp_GAMG 5) N=5, n data cols=1, nnz/row (ave)=5, 1 active pes [0]PCSetUp_GAMG 6 levels, grid compexity = 1.34058 Residual norms for pres_ solve. 0 KSP preconditioned resid norm 3.612110338948e+05 true resid norm 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 2 KSP preconditioned resid norm 2.687590175950e+08 true resid norm 3.313319530843e+07 ||r(i)||/||b|| 2.527950803566e+04 4 KSP preconditioned resid norm 1.718513684798e+05 true resid norm 2.068416025165e+04 ||r(i)||/||b|| 1.578131509578e+01 6 KSP preconditioned resid norm 1.711727556016e+05 true resid norm 1.448181412902e+04 ||r(i)||/||b|| 1.104913465899e+01 8 KSP preconditioned resid norm 7.680280259767e+04 true resid norm 3.841577666956e+03 ||r(i)||/||b|| 2.930993904977e+00 10 KSP preconditioned resid norm 7.284099753086e+04 true resid norm 2.280215579888e+03 ||r(i)||/||b|| 1.739727410478e+00 12 KSP preconditioned resid norm 7.802756964719e+04 true resid norm 3.407495334944e+03 ||r(i)||/||b|| 2.599803758718e+00 14 KSP preconditioned resid norm 7.160374210384e+04 true resid norm 3.172235297295e+03 ||r(i)||/||b|| 2.420308302368e+00 16 KSP preconditioned resid norm 8.637141413866e+04 true resid norm 3.670932784518e+03 ||r(i)||/||b|| 2.800797627900e+00 18 KSP preconditioned resid norm 8.606984257347e+04 true resid norm 3.584264866912e+03 ||r(i)||/||b|| 2.734672936358e+00 20 KSP preconditioned resid norm 9.533640175279e+04 true resid norm 6.402375095182e+03 ||r(i)||/||b|| 4.884795781371e+00 22 KSP preconditioned resid norm 8.215290019906e+04 true resid norm 4.696169451039e+03 ||r(i)||/||b|| 3.583018548898e+00 24 KSP preconditioned resid norm 8.043855900983e+04 true resid norm 4.084316108581e+03 ||r(i)||/||b|| 3.116195130772e+00 26 KSP preconditioned resid norm 7.030803769831e+04 true resid norm 3.310652743124e+03 ||r(i)||/||b|| 2.525916134680e+00 28 KSP preconditioned resid norm 4.112954031889e+04 true resid norm 3.170745816926e+03 ||r(i)||/||b|| 2.419171879194e+00 30 KSP preconditioned resid norm 8.472322642800e+04 true resid norm 3.462065578656e+03 ||r(i)||/||b|| 2.641439010060e+00 32 KSP preconditioned resid norm 7.166398327286e+04 true resid norm 4.853626644992e+03 ||r(i)||/||b|| 3.703153065438e+00 34 KSP preconditioned resid norm 6.306754157168e+04 true resid norm 3.310461332692e+03 ||r(i)||/||b|| 2.525770094991e+00 36 KSP preconditioned resid norm 6.309721154408e+04 true resid norm 3.325407554352e+03 ||r(i)||/||b|| 2.537173556898e+00 38 KSP preconditioned resid norm 2.708710273084e+04 true resid norm 1.351570554381e+03 ||r(i)||/||b|| 1.031202646536e+00 40 KSP preconditioned resid norm 1.556004637642e+04 true resid norm 1.027186285216e+03 ||r(i)||/||b|| 7.837084141603e-01 42 KSP preconditioned resid norm 9.165999428820e+03 true resid norm 3.565338720300e+02 ||r(i)||/||b|| 2.720232926244e-01 44 KSP preconditioned resid norm 4.960846975829e+03 true resid norm 2.748395120845e+02 ||r(i)||/||b|| 2.096932574592e-01 46 KSP preconditioned resid norm 3.112191236107e+03 true resid norm 1.762355295070e+02 ||r(i)||/||b|| 1.344617518132e-01 48 KSP preconditioned resid norm 2.543443891823e+03 true resid norm 1.070094485030e+02 ||r(i)||/||b|| 8.164459202145e-02 50 KSP preconditioned resid norm 8.420615408330e+02 true resid norm 9.643382609945e+01 ||r(i)||/||b|| 7.357574960994e-02 52 KSP preconditioned resid norm 5.236007088398e+02 true resid norm 3.159819451051e+01 ||r(i)||/||b|| 2.410835431370e-02 54 KSP preconditioned resid norm 5.040203463563e+02 true resid norm 1.896920617172e+01 ||r(i)||/||b|| 1.447286310252e-02 56 KSP preconditioned resid norm 2.633181310275e+02 true resid norm 1.011352036822e+01 ||r(i)||/||b|| 7.716274178731e-03 58 KSP preconditioned resid norm 2.466280371039e+02 true resid norm 9.227453336468e+00 ||r(i)||/||b|| 7.040234984779e-03 60 KSP preconditioned resid norm 2.312633390333e+02 true resid norm 1.197301878110e+01 ||r(i)||/||b|| 9.135008612070e-03 62 KSP preconditioned resid norm 2.360698749045e+02 true resid norm 5.863557739326e+00 ||r(i)||/||b|| 4.473696352224e-03 64 KSP preconditioned resid norm 2.228010266419e+02 true resid norm 5.855056953673e+00 ||r(i)||/||b|| 4.467210540119e-03 66 KSP preconditioned resid norm 2.122377720607e+02 true resid norm 4.957796602145e+00 ||r(i)||/||b|| 3.782631221541e-03 68 KSP preconditioned resid norm 2.045503230399e+02 true resid norm 4.887322884286e+00 ||r(i)||/||b|| 3.728862157002e-03 70 KSP preconditioned resid norm 2.997956905450e+02 true resid norm 7.050792714872e+00 ||r(i)||/||b|| 5.379516507061e-03 72 KSP preconditioned resid norm 2.111369460493e+01 true resid norm 9.129783421612e-01 ||r(i)||/||b|| 6.965716141230e-04 74 KSP preconditioned resid norm 1.261967389115e+01 true resid norm 4.965472453812e-01 ||r(i)||/||b|| 3.788487636900e-04 76 KSP preconditioned resid norm 1.289050806041e+01 true resid norm 6.218083277440e-01 ||r(i)||/||b|| 4.744187353957e-04 78 KSP preconditioned resid norm 1.083352610697e+01 true resid norm 4.398776101309e-01 ||r(i)||/||b|| 3.356117475691e-04 80 KSP preconditioned resid norm 2.274409904236e+01 true resid norm 1.215038053296e+00 ||r(i)||/||b|| 9.270329633470e-04 82 KSP preconditioned resid norm 2.847352576438e+01 true resid norm 1.220115820373e+00 ||r(i)||/||b|| 9.309071279853e-04 84 KSP preconditioned resid norm 7.250797645802e+00 true resid norm 3.906201125100e-01 ||r(i)||/||b|| 2.980299419107e-04 86 KSP preconditioned resid norm 7.048473788478e+00 true resid norm 3.936854723961e-01 ||r(i)||/||b|| 3.003687078870e-04 88 KSP preconditioned resid norm 1.098249410053e+01 true resid norm 6.062928999480e-01 ||r(i)||/||b|| 4.625809884475e-04 90 KSP preconditioned resid norm 1.062480789565e+00 true resid norm 4.724208767154e-02 ||r(i)||/||b|| 3.604411599294e-05 92 KSP preconditioned resid norm 9.552895552568e-01 true resid norm 4.451250942715e-02 ||r(i)||/||b|| 3.396154005903e-05 94 KSP preconditioned resid norm 5.719134548913e-01 true resid norm 3.152141258463e-02 ||r(i)||/||b|| 2.404977229968e-05 96 KSP preconditioned resid norm 9.675788485492e-01 true resid norm 5.651537251153e-02 ||r(i)||/||b|| 4.311931886570e-05 98 KSP preconditioned resid norm 9.664671699486e-02 true resid norm 4.520511717286e-03 ||r(i)||/||b|| 3.448997635714e-06 100 KSP preconditioned resid norm 9.130467777174e-03 true resid norm 3.299763651766e-04 ||r(i)||/||b|| 2.517608126053e-07 102 KSP preconditioned resid norm 1.126791794400e-02 true resid norm 6.788892804581e-04 ||r(i)||/||b|| 5.179695728379e-07 104 KSP preconditioned resid norm 9.838420281303e-03 true resid norm 5.065570335783e-04 ||r(i)||/||b|| 3.864858937286e-07 106 KSP preconditioned resid norm 1.084444455419e-02 true resid norm 4.770227150710e-04 ||r(i)||/||b|| 3.639522070412e-07 108 KSP preconditioned resid norm 5.720579076324e-02 true resid norm 2.844060861586e-03 ||r(i)||/||b|| 2.169922301037e-06 110 KSP preconditioned resid norm 3.556058070390e-02 true resid norm 1.603587024377e-03 ||r(i)||/||b|| 1.223482694357e-06 112 KSP preconditioned resid norm 1.336438885062e-03 true resid norm 6.132214983120e-05 ||r(i)||/||b|| 4.678672747954e-08 114 KSP preconditioned resid norm 1.644242123526e-04 true resid norm 2.454816318596e-05 ||r(i)||/||b|| 1.872941872172e-08 116 KSP preconditioned resid norm 2.077302118590e-05 true resid norm 1.814016140734e-05 ||r(i)||/||b|| 1.384032997108e-08 118 KSP preconditioned resid norm 2.062148780542e-05 true resid norm 1.814203749554e-05 ||r(i)||/||b|| 1.384176136296e-08 120 KSP preconditioned resid norm 3.987467373484e-05 true resid norm 1.847262282651e-05 ||r(i)||/||b|| 1.409398679589e-08 122 KSP preconditioned resid norm 1.577748457132e-05 true resid norm 1.821983976764e-05 ||r(i)||/||b|| 1.390112186666e-08 124 KSP preconditioned resid norm 1.408767972144e-05 true resid norm 1.819870889010e-05 ||r(i)||/||b|| 1.388499972138e-08 126 KSP preconditioned resid norm 1.408271904199e-05 true resid norm 1.820121027523e-05 ||r(i)||/||b|| 1.388690819368e-08 128 KSP preconditioned resid norm 1.404851203992e-05 true resid norm 1.820287394307e-05 ||r(i)||/||b|| 1.388817751600e-08 130 KSP preconditioned resid norm 1.405039492404e-05 true resid norm 1.820273914085e-05 ||r(i)||/||b|| 1.388807466646e-08 132 KSP preconditioned resid norm 1.401867587595e-05 true resid norm 1.819712041489e-05 ||r(i)||/||b|| 1.388378776849e-08 134 KSP preconditioned resid norm 1.401302217409e-05 true resid norm 1.819932172647e-05 ||r(i)||/||b|| 1.388546729481e-08 136 KSP preconditioned resid norm 1.401277226860e-05 true resid norm 1.819913560733e-05 ||r(i)||/||b|| 1.388532529220e-08 138 KSP preconditioned resid norm 1.401269117590e-05 true resid norm 1.819912527486e-05 ||r(i)||/||b|| 1.388531740887e-08 140 KSP preconditioned resid norm 1.401269566176e-05 true resid norm 1.819927213240e-05 ||r(i)||/||b|| 1.388542945622e-08 142 KSP preconditioned resid norm 1.401278503672e-05 true resid norm 1.819939545249e-05 ||r(i)||/||b|| 1.388552354527e-08 144 KSP preconditioned resid norm 1.401278982467e-05 true resid norm 1.819941480329e-05 ||r(i)||/||b|| 1.388553830928e-08 146 KSP preconditioned resid norm 1.401293263197e-05 true resid norm 1.819944180498e-05 ||r(i)||/||b|| 1.388555891066e-08 148 KSP preconditioned resid norm 1.401282452159e-05 true resid norm 1.819933079847e-05 ||r(i)||/||b|| 1.388547421644e-08 150 KSP preconditioned resid norm 1.401257583064e-05 true resid norm 1.819907987066e-05 ||r(i)||/||b|| 1.388528276700e-08 152 KSP preconditioned resid norm 1.401260524504e-05 true resid norm 1.819908631586e-05 ||r(i)||/||b|| 1.388528768447e-08 154 KSP preconditioned resid norm 1.401507585649e-05 true resid norm 1.819925585968e-05 ||r(i)||/||b|| 1.388541704068e-08 156 KSP preconditioned resid norm 1.401252030737e-05 true resid norm 1.819910061311e-05 ||r(i)||/||b|| 1.388529859279e-08 158 KSP preconditioned resid norm 1.401252841472e-05 true resid norm 1.819914630889e-05 ||r(i)||/||b|| 1.388533345712e-08 160 KSP preconditioned resid norm 1.401235849762e-05 true resid norm 1.819914148389e-05 ||r(i)||/||b|| 1.388532977581e-08 162 KSP preconditioned resid norm 1.401233765711e-05 true resid norm 1.819915840374e-05 ||r(i)||/||b|| 1.388534268509e-08 164 KSP preconditioned resid norm 1.401234398567e-05 true resid norm 1.819916534902e-05 ||r(i)||/||b|| 1.388534798410e-08 166 KSP preconditioned resid norm 1.401231028718e-05 true resid norm 1.819913969356e-05 ||r(i)||/||b|| 1.388532840986e-08 168 KSP preconditioned resid norm 1.401234331067e-05 true resid norm 1.819916846357e-05 ||r(i)||/||b|| 1.388535036040e-08 170 KSP preconditioned resid norm 1.401235885972e-05 true resid norm 1.819917751038e-05 ||r(i)||/||b|| 1.388535726281e-08 172 KSP preconditioned resid norm 1.401233900388e-05 true resid norm 1.819923941500e-05 ||r(i)||/||b|| 1.388540449394e-08 174 KSP preconditioned resid norm 1.401225082189e-05 true resid norm 1.819918916295e-05 ||r(i)||/||b|| 1.388536615333e-08 176 KSP preconditioned resid norm 1.401226552751e-05 true resid norm 1.819915213085e-05 ||r(i)||/||b|| 1.388533789909e-08 178 KSP preconditioned resid norm 1.401219943890e-05 true resid norm 1.819913939571e-05 ||r(i)||/||b|| 1.388532818261e-08 180 KSP preconditioned resid norm 1.401199263035e-05 true resid norm 1.819910733407e-05 ||r(i)||/||b|| 1.388530372065e-08 182 KSP preconditioned resid norm 1.401287854824e-05 true resid norm 1.819930416585e-05 ||r(i)||/||b|| 1.388545389665e-08 184 KSP preconditioned resid norm 1.401167159645e-05 true resid norm 1.819872545239e-05 ||r(i)||/||b|| 1.388501235785e-08 186 KSP preconditioned resid norm 1.401103238265e-05 true resid norm 1.819860547689e-05 ||r(i)||/||b|| 1.388492082059e-08 188 KSP preconditioned resid norm 1.401323897340e-05 true resid norm 1.819922279518e-05 ||r(i)||/||b|| 1.388539181359e-08 190 KSP preconditioned resid norm 1.401226979551e-05 true resid norm 1.819904247531e-05 ||r(i)||/||b|| 1.388525423562e-08 192 KSP preconditioned resid norm 1.401256483397e-05 true resid norm 1.819910162959e-05 ||r(i)||/||b|| 1.388529936833e-08 194 KSP preconditioned resid norm 1.401189092748e-05 true resid norm 1.819899255266e-05 ||r(i)||/||b|| 1.388521614632e-08 196 KSP preconditioned resid norm 1.401269823835e-05 true resid norm 1.819928036837e-05 ||r(i)||/||b|| 1.388543573998e-08 198 KSP preconditioned resid norm 1.401223408377e-05 true resid norm 1.819904630195e-05 ||r(i)||/||b|| 1.388525715521e-08 200 KSP preconditioned resid norm 1.401252572765e-05 true resid norm 1.819908365033e-05 ||r(i)||/||b|| 1.388528565076e-08 202 KSP preconditioned resid norm 1.401235007102e-05 true resid norm 1.819914581432e-05 ||r(i)||/||b|| 1.388533307979e-08 204 KSP preconditioned resid norm 1.401348753852e-05 true resid norm 1.819829709741e-05 ||r(i)||/||b|| 1.388468553747e-08 206 KSP preconditioned resid norm 1.401258540428e-05 true resid norm 1.819916724139e-05 ||r(i)||/||b|| 1.388534942792e-08 208 KSP preconditioned resid norm 1.401263706129e-05 true resid norm 1.819919858881e-05 ||r(i)||/||b|| 1.388537334494e-08 210 KSP preconditioned resid norm 1.401238565223e-05 true resid norm 1.819905505425e-05 ||r(i)||/||b|| 1.388526383292e-08 212 KSP preconditioned resid norm 1.401208932589e-05 true resid norm 1.819902354098e-05 ||r(i)||/||b|| 1.388523978936e-08 214 KSP preconditioned resid norm 1.538450958191e-05 true resid norm 1.819523499266e-05 ||r(i)||/||b|| 1.388234925506e-08 216 KSP preconditioned resid norm 1.401335550976e-05 true resid norm 1.819931191443e-05 ||r(i)||/||b|| 1.388545980855e-08 218 KSP preconditioned resid norm 1.401199672215e-05 true resid norm 1.819921984869e-05 ||r(i)||/||b|| 1.388538956551e-08 220 KSP preconditioned resid norm 1.401176169548e-05 true resid norm 1.819913798571e-05 ||r(i)||/||b|| 1.388532710682e-08 222 KSP preconditioned resid norm 1.401290941148e-05 true resid norm 1.819933634161e-05 ||r(i)||/||b|| 1.388547844567e-08 224 KSP preconditioned resid norm 1.401220439981e-05 true resid norm 1.81990505 -------------- next part -------------- [0]PCSetUp_GAMG level 0 N=46330, n data rows=1, n data cols=1, nnz/row (ave)=1, np=4 [0]scaleFilterGraph 75.4421% nnz after filtering, with threshold 0.05, 6.96664 nnz ave. [0]maxIndSetAgg removed 0 of 10552 vertices. [0]PCGAMGprolongator_AGG New grid 5903 nodes [0]PCSetUp_GAMG 1) N=5903, n data cols=1, nnz/row (ave)=6, 4 active pes [0]scaleFilterGraph 91.3687% nnz after filtering, with threshold 0.05, 6.49886 nnz ave. [0]maxIndSetAgg removed 0 of 1321 vertices. [0]PCGAMGprolongator_AGG New grid 718 nodes [0]PCSetUp_GAMG 2) N=718, n data cols=1, nnz/row (ave)=5, 4 active pes [0]scaleFilterGraph 90.3509% nnz after filtering, with threshold 0.05, 5.11538 nnz ave. [0]maxIndSetAgg removed 0 of 156 vertices. [0]PCGAMGprolongator_AGG New grid 127 nodes [0]createLevel aggregate processors: npe: 4 --> 1, neq=127 [0]PCSetUp_GAMG 3) N=127, n data cols=1, nnz/row (ave)=4, 1 active pes [0]scaleFilterGraph 86.4013% nnz after filtering, with threshold 0.05, 4.74803 nnz ave. [0]maxIndSetAgg removed 0 of 127 vertices. [0]PCGAMGprolongator_AGG New grid 25 nodes [0]PCSetUp_GAMG 4) N=25, n data cols=1, nnz/row (ave)=4, 1 active pes [0]scaleFilterGraph 98.0583% nnz after filtering, with threshold 0.05, 4.12 nnz ave. [0]maxIndSetAgg removed 0 of 25 vertices. [0]PCGAMGprolongator_AGG New grid 5 nodes [0]PCSetUp_GAMG 5) N=5, n data cols=1, nnz/row (ave)=2, 1 active pes [0]PCSetUp_GAMG 6 levels, grid compexity = 1.13742 PCSetUp_GAMG PC setup max eigen=1.693353e+00 min=1.205252e-01 PCSetUp_GAMG PC setup max eigen=1.999649e+00 min=5.459423e-02 PCSetUp_GAMG PC setup max eigen=1.844191e+00 min=3.600443e-02 PCSetUp_GAMG PC setup max eigen=1.768003e+00 min=2.083195e-02 PCSetUp_GAMG PC setup max eigen=1.934105e+00 min=3.878260e-02 Residual norms for pres_ solve. 0 KSP preconditioned resid norm 1.663347231219e+05 true resid norm 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 2 KSP preconditioned resid norm 6.567796713567e+04 true resid norm 2.752296371149e+04 ||r(i)||/||b|| 2.099909096702e+01 4 KSP preconditioned resid norm 5.466106023057e+05 true resid norm 1.980240237000e+05 ||r(i)||/||b|| 1.510856363770e+02 6 KSP preconditioned resid norm 3.269775589446e+05 true resid norm 1.589713021070e+05 ||r(i)||/||b|| 1.212897298810e+02 8 KSP preconditioned resid norm 3.203649214780e+04 true resid norm 1.156144810931e+04 ||r(i)||/||b|| 8.820994101611e+00 10 KSP preconditioned resid norm 8.667598714391e+04 true resid norm 2.478167249197e+04 ||r(i)||/||b|| 1.890757842901e+01 12 KSP preconditioned resid norm 1.769725563561e+04 true resid norm 4.147840907249e+03 ||r(i)||/||b|| 3.164662404859e+00 14 KSP preconditioned resid norm 1.312323526170e+04 true resid norm 1.948461028455e+03 ||r(i)||/||b|| 1.486609901867e+00 16 KSP preconditioned resid norm 1.156457361259e+04 true resid norm 1.266774145072e+03 ||r(i)||/||b|| 9.665058525630e-01 18 KSP preconditioned resid norm 1.027678853385e+04 true resid norm 1.488971189709e+03 ||r(i)||/||b|| 1.136034686807e+00 20 KSP preconditioned resid norm 1.039522693159e+04 true resid norm 1.455905421918e+03 ||r(i)||/||b|| 1.110806623687e+00 22 KSP preconditioned resid norm 9.852763686063e+03 true resid norm 1.429220248656e+03 ||r(i)||/||b|| 1.090446738514e+00 24 KSP preconditioned resid norm 8.767670423772e+03 true resid norm 1.274497701051e+03 ||r(i)||/||b|| 9.723986646997e-01 26 KSP preconditioned resid norm 8.328695373033e+03 true resid norm 8.258935539966e+02 ||r(i)||/||b|| 6.301288644366e-01 28 KSP preconditioned resid norm 8.016139977502e+03 true resid norm 1.227648373812e+03 ||r(i)||/||b|| 9.366542116403e-01 30 KSP preconditioned resid norm 7.918495667087e+03 true resid norm 8.305221324851e+02 ||r(i)||/||b|| 6.336603133658e-01 32 KSP preconditioned resid norm 7.990512819579e+03 true resid norm 7.548039142022e+02 ||r(i)||/||b|| 5.758898722807e-01 34 KSP preconditioned resid norm 7.999257819915e+03 true resid norm 1.004252641256e+03 ||r(i)||/||b|| 7.662108190335e-01 36 KSP preconditioned resid norm 7.835545253091e+03 true resid norm 1.444231992624e+03 ||r(i)||/||b|| 1.101900191727e+00 38 KSP preconditioned resid norm 7.670995008661e+03 true resid norm 8.083283362773e+02 ||r(i)||/||b|| 6.167271970649e-01 40 KSP preconditioned resid norm 7.577687720138e+03 true resid norm 1.016051521051e+03 ||r(i)||/||b|| 7.752129654855e-01 42 KSP preconditioned resid norm 7.909824866411e+03 true resid norm 1.358212988013e+03 ||r(i)||/||b|| 1.036270598866e+00 44 KSP preconditioned resid norm 8.773425010803e+03 true resid norm 1.002009994050e+03 ||r(i)||/||b|| 7.644997550220e-01 46 KSP preconditioned resid norm 7.954187546293e+03 true resid norm 2.698635503179e+03 ||r(i)||/||b|| 2.058967668312e+00 48 KSP preconditioned resid norm 7.643168470330e+03 true resid norm 1.607019669270e+03 ||r(i)||/||b|| 1.226101686378e+00 50 KSP preconditioned resid norm 7.467053286167e+03 true resid norm 1.238908135793e+03 ||r(i)||/||b|| 9.452450294239e-01 52 KSP preconditioned resid norm 7.504066288874e+03 true resid norm 1.065449375972e+03 ||r(i)||/||b|| 8.129018590196e-01 54 KSP preconditioned resid norm 7.961829184352e+03 true resid norm 1.705190760618e+03 ||r(i)||/||b|| 1.301002910649e+00 56 KSP preconditioned resid norm 5.280109275365e+03 true resid norm 1.230007549400e+03 ||r(i)||/||b|| 9.384541828682e-01 58 KSP preconditioned resid norm 7.285589173580e+03 true resid norm 1.248789115263e+03 ||r(i)||/||b|| 9.527838827573e-01 60 KSP preconditioned resid norm 5.372930205337e+03 true resid norm 2.055209806116e+03 ||r(i)||/||b|| 1.568055610847e+00 62 KSP preconditioned resid norm 5.317455508650e+03 true resid norm 1.569883420506e+03 ||r(i)||/||b|| 1.197767983869e+00 64 KSP preconditioned resid norm 4.651478346787e+03 true resid norm 1.125464473759e+03 ||r(i)||/||b|| 8.586913499711e-01 66 KSP preconditioned resid norm 5.029458030703e+03 true resid norm 1.283911390936e+03 ||r(i)||/||b|| 9.795809918756e-01 68 KSP preconditioned resid norm 3.359247179976e+03 true resid norm 7.487032294139e+02 ||r(i)||/||b|| 5.712352560055e-01 70 KSP preconditioned resid norm 2.298226566553e+03 true resid norm 4.810633996817e+02 ||r(i)||/||b|| 3.670351128139e-01 72 KSP preconditioned resid norm 2.437279220201e+03 true resid norm 3.451338187828e+02 ||r(i)||/||b|| 2.633254373470e-01 74 KSP preconditioned resid norm 1.121862058899e+03 true resid norm 3.739958480903e+02 ||r(i)||/||b|| 2.853461901001e-01 76 KSP preconditioned resid norm 5.764976452342e+02 true resid norm 1.296514348922e+02 ||r(i)||/||b|| 9.891966228073e-02 78 KSP preconditioned resid norm 2.959527805479e+02 true resid norm 3.850107180615e+01 ||r(i)||/||b|| 2.937501635580e-02 80 KSP preconditioned resid norm 2.767666324669e+02 true resid norm 3.598111843300e+01 ||r(i)||/||b|| 2.745237711280e-02 82 KSP preconditioned resid norm 2.017812198522e+02 true resid norm 2.553182349442e+01 ||r(i)||/||b|| 1.947991828690e-02 84 KSP preconditioned resid norm 1.420504762647e+02 true resid norm 2.316682808482e+01 ||r(i)||/||b|| 1.767550673212e-02 86 KSP preconditioned resid norm 2.161763577525e+02 true resid norm 3.055059796487e+01 ||r(i)||/||b|| 2.330907356075e-02 88 KSP preconditioned resid norm 8.797736673019e+01 true resid norm 1.775948996075e+01 ||r(i)||/||b|| 1.354989052498e-02 90 KSP preconditioned resid norm 5.212224911849e+01 true resid norm 9.037670663578e+00 ||r(i)||/||b|| 6.895437220491e-03 92 KSP preconditioned resid norm 2.582485513792e+01 true resid norm 3.695016858770e+00 ||r(i)||/||b|| 2.819172962452e-03 94 KSP preconditioned resid norm 2.511753395563e+01 true resid norm 4.331444191820e+00 ||r(i)||/||b|| 3.304745504737e-03 96 KSP preconditioned resid norm 1.340286627338e+01 true resid norm 2.043360937970e+00 ||r(i)||/||b|| 1.559015324973e-03 98 KSP preconditioned resid norm 7.788283769267e+00 true resid norm 1.365025541649e+00 ||r(i)||/||b|| 1.041468347009e-03 100 KSP preconditioned resid norm 2.330399915648e+00 true resid norm 5.473386771246e-01 ||r(i)||/||b|| 4.176009092331e-04 102 KSP preconditioned resid norm 1.841162449845e+00 true resid norm 5.640944100940e-01 ||r(i)||/||b|| 4.303849671031e-04 104 KSP preconditioned resid norm 9.849647493643e-01 true resid norm 2.556801524986e-01 ||r(i)||/||b|| 1.950753137293e-04 106 KSP preconditioned resid norm 8.669422296393e-01 true resid norm 1.344088995700e-01 ||r(i)||/||b|| 1.025494470157e-04 108 KSP preconditioned resid norm 8.980014218288e-01 true resid norm 1.374441239490e-01 ||r(i)||/||b|| 1.048652206187e-04 110 KSP preconditioned resid norm 2.748683080221e-01 true resid norm 4.397420547304e-02 ||r(i)||/||b|| 3.355083233806e-05 112 KSP preconditioned resid norm 1.605746404788e-01 true resid norm 1.718733424203e-02 ||r(i)||/||b|| 1.311335505189e-05 114 KSP preconditioned resid norm 1.065880847345e-01 true resid norm 1.394296024957e-02 ||r(i)||/||b|| 1.063800736358e-05 116 KSP preconditioned resid norm 5.089068176336e-02 true resid norm 1.010497577591e-02 ||r(i)||/||b|| 7.709754943628e-06 118 KSP preconditioned resid norm 3.131857220882e-02 true resid norm 3.323217069673e-03 ||r(i)||/||b|| 2.535502291132e-06 120 KSP preconditioned resid norm 2.965821428510e-02 true resid norm 3.276233382478e-03 ||r(i)||/||b|| 2.499655325968e-06 122 KSP preconditioned resid norm 1.597891045514e-02 true resid norm 4.759139650005e-03 ||r(i)||/||b|| 3.631062682159e-06 124 KSP preconditioned resid norm 9.680012737053e-03 true resid norm 1.054103556414e-03 ||r(i)||/||b|| 8.042453822137e-07 126 KSP preconditioned resid norm 6.153122965798e-03 true resid norm 1.604398742466e-03 ||r(i)||/||b|| 1.224102007821e-06 128 KSP preconditioned resid norm 3.954214764783e-03 true resid norm 4.156609639169e-04 ||r(i)||/||b|| 3.171352650909e-07 130 KSP preconditioned resid norm 2.562330445863e-03 true resid norm 4.095016203896e-04 ||r(i)||/||b|| 3.124358941807e-07 132 KSP preconditioned resid norm 2.112111566426e-03 true resid norm 2.387672371692e-04 ||r(i)||/||b|| 1.821713310317e-07 134 KSP preconditioned resid norm 1.080595713577e-03 true resid norm 9.445872749660e-05 ||r(i)||/||b|| 7.206881613923e-08 136 KSP preconditioned resid norm 9.019488537597e-04 true resid norm 2.261853037116e-04 ||r(i)||/||b|| 1.725717411043e-07 138 KSP preconditioned resid norm 1.655643753743e-03 true resid norm 6.326769898827e-04 ||r(i)||/||b|| 4.827111572196e-07 140 KSP preconditioned resid norm 3.354786964176e-04 true resid norm 4.665899166324e-05 ||r(i)||/||b|| 3.559923344872e-08 142 KSP preconditioned resid norm 1.955417849395e-03 true resid norm 4.173899603468e-04 ||r(i)||/||b|| 3.184544309225e-07 144 KSP preconditioned resid norm 2.157529609788e-04 true resid norm 2.953353419131e-05 ||r(i)||/||b|| 2.253308828082e-08 146 KSP preconditioned resid norm 1.486973480783e-04 true resid norm 1.855777018380e-05 ||r(i)||/||b|| 1.415895135130e-08 148 KSP preconditioned resid norm 1.055226071274e-04 true resid norm 2.066226516026e-05 ||r(i)||/||b|| 1.576460988116e-08 150 KSP preconditioned resid norm 8.947908647545e-05 true resid norm 9.528537251226e-06 ||r(i)||/||b|| 7.269951834351e-09 152 KSP preconditioned resid norm 4.953708167034e-05 true resid norm 8.318389913741e-06 ||r(i)||/||b|| 6.346650321850e-09 154 KSP preconditioned resid norm 2.410529397250e-05 true resid norm 2.433007111899e-06 ||r(i)||/||b|| 1.856302184668e-09 156 KSP preconditioned resid norm 1.431008972002e-05 true resid norm 3.007953192313e-06 ||r(i)||/||b|| 2.294966609412e-09 158 KSP preconditioned resid norm 8.344138040342e-06 true resid norm 8.950909955055e-07 ||r(i)||/||b|| 6.829241732617e-10 160 KSP preconditioned resid norm 5.305429174321e-06 true resid norm 8.751180974573e-07 ||r(i)||/||b|| 6.676855271847e-10 162 KSP preconditioned resid norm 3.407936113633e-06 true resid norm 5.080913148796e-07 ||r(i)||/||b|| 3.876564984989e-10 164 KSP preconditioned resid norm 3.307812191488e-06 true resid norm 5.928124531795e-07 ||r(i)||/||b|| 4.522958632359e-10 166 KSP preconditioned resid norm 1.279802768847e-06 true resid norm 5.915416474155e-07 ||r(i)||/||b|| 4.513262813943e-10 168 KSP preconditioned resid norm 5.578434397034e-07 true resid norm 4.219723792327e-07 ||r(i)||/||b|| 3.219506616353e-10 170 KSP preconditioned resid norm 4.050142686203e-07 true resid norm 4.209005270951e-07 ||r(i)||/||b|| 3.211328746856e-10 172 KSP preconditioned resid norm 1.279720451106e-07 true resid norm 4.065801879654e-07 ||r(i)||/||b|| 3.102069399928e-10 KSP Object:(pres_) 4 MPI processes type: bcgsl BCGSL: Ell = 2 BCGSL: Delta = 0 maximum iterations=5000 tolerances: relative=1e-12, absolute=1e-50, divergence=10000 left preconditioning diagonally scaled system has attached null space using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object:(pres_) 4 MPI processes type: gamg MG: type is MULTIPLICATIVE, levels=6 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (pres_mg_coarse_) 4 MPI processes type: preonly maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (pres_mg_coarse_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 8, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=5, cols=5 total: nonzeros=13, allocated nonzeros=13 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (pres_mg_levels_1_) 4 MPI processes type: chebychev Chebychev: eigenvalue estimates: min = 0.338671, max = 1.77802 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_1_) 4 MPI processes type: jacobi linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=25, cols=25 total: nonzeros=103, allocated nonzeros=103 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (pres_mg_levels_2_) 4 MPI processes type: chebychev Chebychev: eigenvalue estimates: min = 0.393632, max = 2.09963 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_2_) 4 MPI processes type: jacobi linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=127, cols=127 total: nonzeros=603, allocated nonzeros=603 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (pres_mg_levels_3_) 4 MPI processes type: chebychev Chebychev: eigenvalue estimates: min = 0.326201, max = 1.9364 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_3_) 4 MPI processes type: jacobi linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=718, cols=718 total: nonzeros=3765, allocated nonzeros=3765 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 4 ------------------------------- KSP Object: (pres_mg_levels_4_) 4 MPI processes type: chebychev Chebychev: eigenvalue estimates: min = 0.215048, max = 1.8564 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_4_) 4 MPI processes type: jacobi linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=5903, cols=5903 total: nonzeros=38087, allocated nonzeros=38087 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 5 ------------------------------- KSP Object: (pres_mg_levels_5_) 4 MPI processes type: chebychev Chebychev: eigenvalue estimates: min = 0.246428, max = 2.03081 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_5_) 4 MPI processes type: jacobi linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=46330, cols=46330 total: nonzeros=322437, allocated nonzeros=615417 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=46330, cols=46330 total: nonzeros=322437, allocated nonzeros=615417 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines From knepley at gmail.com Tue Mar 20 09:24:50 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 20 Mar 2012 09:24:50 -0500 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> Message-ID: On Tue, Mar 20, 2012 at 9:19 AM, John Mousel wrote: > Mark, > > Sorry for the late reply. I've been on travel and hadn't had a chance to > pick this back up. I've tried running with the suggested options: > > -ksp_type bcgsl -pc_type gamg -pc_gamg_coarse_eq_limit 10 > -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph -mg_coarse_ksp_type richardson > -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_diagonal_scale > -ksp_diagonal_scale_fix -ksp_monitor_true_residual -ksp_view > -pc_gamg_verbose 1 > > With these options, the convergence starts to hang (see attached > GAMG_kspview.txt). The hanging happens for both -mg_coarse_ksp_type > richardson and preonly. It was my understanding from previous emails that > using preonly made it so that only the preconditioner was run, which in > this case would be 8 sweeps of SOR. If I get rid of the > -pc_gamg_agg_nsmooths 1 (see GAMG_kspview_nosmooth.txt), the problem > converges, but again the convergence is slow. Without this option, both > Richardson and preonly converge in 172 iterations. > > Matt, I've checked and the problem does converge in the true residual > using GAMG, ML, HYPRE, and ILU preconditioned BiCG. I explicitly ensure > that a solution exists by projecting the rhs vector out of the nullity of > the transpose of operator. > Would you mind sending the matrix in binary format, -ksp_view_binary, to petsc-maint at mcs.anl.gov? Then we can figure out what is happening here. Thanks, Matt > John > > > On Fri, Mar 16, 2012 at 2:04 PM, Mark F. Adams wrote: > >> John, did this get resolved? >> Mark >> >> On Mar 15, 2012, at 4:24 PM, John Mousel wrote: >> >> Mark, >> >> Running without the options you mentioned before leads to slightly worse >> performance (175 iterations). >> I have not been able to get run coarse grid solve to work with LU while >> running ML. It keeps experiencing a zero pivot, and all the combinations of >> shifting i've tried haven't lead me anywhere, hence the SOR on the course >> grid. Also, the ML manual suggests limiting the number of levels to 3 or 4 >> and performing a few sweeps of an iterative method as opposed to a direct >> solve. >> >> John >> >> On Thu, Mar 15, 2012 at 12:04 PM, Mark F. Adams wrote: >> >>> You also want: -pc_gamg_agg_nsmooths 1 >>> >>> You are running plain aggregation. If it is Poisson then smoothing is >>> good. >>> >>> Is this problem singular? Can you try running ML with these parameters >>> and see if its performance degrades? The ML implementation uses the PETSC >>> infrastructure and uses a very similar algorithm to GAMG-SA. We should be >>> able to get these two to match pretty well. >>> >>> Mark >>> >>> >>> On Mar 15, 2012, at 12:21 PM, John Mousel wrote: >>> >>> Mark, >>> >>> I ran with those options removed (see the run options listed below). >>> Things actually got slightly worse. Now it's up to 142 iterations. I have >>> attached the ksp_view output. >>> >>> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale >>> -ksp_diagonal_scale_fix -mg_levels_ksp_type richardson -mg_levels_pc_type >>> sor -pc_gamg_verbose 1 >>> >>> >>> John >>> >>> >>> On Thu, Mar 15, 2012 at 10:55 AM, Mark F. Adams >> > wrote: >>> >>>> John, can you run again with: -pc_gamg_verbose 1 >>>> >>>> And I would not use: -pc_mg_levels 4 -mg_coarse_ksp_type preonly >>>> -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 >>>> >>>> 1) I think -mg_coarse_ksp_type preonly and -mg_coarse_pc_sor_its 8 do >>>> not do what you think. I think this is the same as 1 iteration. I think >>>> you want 'richardson' not 'preonly'. >>>> >>>> 2) Why are you using sor as the coarse solver? If your problem is >>>> singular then you want to use as many levels as possible to get the coarse >>>> grid to be tiny. I'm pretty sure HYPRE ignores the coarse solver >>>> parameters. But ML uses them and it is converging well. >>>> >>>> 3) I would not specify the number of levels. GAMG, and I think the >>>> rest, have internal logic for stopping a the right level. If the coarse >>>> level is large and you use just 8 iterations of sor then convergence will >>>> suffer. >>>> >>>> Mark >>>> >>>> On Mar 15, 2012, at 11:13 AM, John Mousel wrote: >>>> >>>> Mark, >>>> >>>> The changes pulled through this morning. I've run it with the options >>>> >>>> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale >>>> -ksp_diagonal_scale_fix -pc_mg_levels 4 -mg_levels_ksp_type richardson >>>> -mg_levels_pc_type sor -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor >>>> -mg_coarse_pc_sor_its 8 >>>> >>>> and it converges in the true residual, but it's not converging as fast >>>> as anticpated. The matrix arises from a non-symmetric discretization of the >>>> Poisson equation. The solve takes GAMG 114 iterations, whereas ML takes 24 >>>> iterations, BoomerAMG takes 22 iterations, and -ksp_type bcgsl -pc_type >>>> bjacobi -sub_pc_type ilu -sub_pc_factor_levels 4 takes around 170. I've >>>> attached the -ksp_view results for ML,GAMG, and HYPRE. I've attempted to >>>> make all the options the same on all levels for ML and GAMG. >>>> >>>> Any thoughts? >>>> >>>> John >>>> >>>> >>>> On Wed, Mar 14, 2012 at 6:04 PM, Mark F. Adams >>> > wrote: >>>> >>>>> Humm, I see it with hg view (appended). >>>>> >>>>> Satish, my main repo looks hosed. I see this: >>>>> >>>>> ~/Codes/petsc-dev>hg update >>>>> abort: crosses branches (merge branches or use --clean to discard >>>>> changes) >>>>> ~/Codes/petsc-dev>hg merge >>>>> abort: branch 'default' has 3 heads - please merge with an explicit rev >>>>> (run 'hg heads .' to see heads) >>>>> ~/Codes/petsc-dev>hg heads >>>>> changeset: 22496:8e2a98268179 >>>>> tag: tip >>>>> user: Barry Smith >>>>> date: Wed Mar 14 16:42:25 2012 -0500 >>>>> files: src/vec/is/interface/f90-custom/zindexf90.c >>>>> src/vec/vec/interface/f90-custom/zvectorf90.c >>>>> description: >>>>> undoing manually changes I put in because Satish had a better fix >>>>> >>>>> >>>>> changeset: 22492:bda4df63072d >>>>> user: Mark F. Adams >>>>> date: Wed Mar 14 17:39:52 2012 -0400 >>>>> files: src/ksp/pc/impls/gamg/tools.c >>>>> description: >>>>> fix for unsymmetric matrices. >>>>> >>>>> >>>>> changeset: 22469:b063baf366e4 >>>>> user: Mark F. Adams >>>>> date: Wed Mar 14 14:22:28 2012 -0400 >>>>> files: src/ksp/pc/impls/gamg/tools.c >>>>> description: >>>>> added fix for preallocation for unsymetric matrices. >>>>> >>>>> Mark >>>>> >>>>> my 'hg view' on my merge repo: >>>>> >>>>> Revision: 22492 >>>>> Branch: default >>>>> Author: Mark F. Adams 2012-03-14 17:39:52 >>>>> Committer: Mark F. Adams 2012-03-14 >>>>> 17:39:52 >>>>> Tags: tip >>>>> Parent: 22491:451bbbd291c2 (Small fixes to the BT linesearch) >>>>> >>>>> fix for unsymmetric matrices. >>>>> >>>>> >>>>> ------------------------ src/ksp/pc/impls/gamg/tools.c >>>>> ------------------------ >>>>> @@ -103,7 +103,7 @@ >>>>> PetscErrorCode ierr; >>>>> PetscInt Istart,Iend,Ii,jj,ncols,nnz0,nnz1, NN, MM, nloc; >>>>> PetscMPIInt mype, npe; >>>>> - Mat Gmat = *a_Gmat, tGmat; >>>>> + Mat Gmat = *a_Gmat, tGmat, matTrans; >>>>> MPI_Comm wcomm = ((PetscObject)Gmat)->comm; >>>>> const PetscScalar *vals; >>>>> const PetscInt *idx; >>>>> @@ -127,6 +127,10 @@ >>>>> ierr = MatDiagonalScale( Gmat, diag, diag ); CHKERRQ(ierr); >>>>> ierr = VecDestroy( &diag ); CHKERRQ(ierr); >>>>> >>>>> + if( symm ) { >>>>> + ierr = MatTranspose( Gmat, MAT_INITIAL_MATRIX, &matTrans ); >>>>> CHKERRQ(ierr); >>>>> + } >>>>> + >>>>> /* filter - dup zeros out matrix */ >>>>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &d_nnz ); CHKERRQ(ierr); >>>>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &o_nnz ); CHKERRQ(ierr); >>>>> @@ -135,6 +139,12 @@ >>>>> d_nnz[jj] = ncols; >>>>> o_nnz[jj] = ncols; >>>>> ierr = MatRestoreRow(Gmat,Ii,&ncols,PETSC_NULL,PETSC_NULL); >>>>> CHKERRQ(ierr); >>>>> + if( symm ) { >>>>> + ierr = MatGetRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); >>>>> CHKERRQ(ierr); >>>>> + d_nnz[jj] += ncols; >>>>> + o_nnz[jj] += ncols; >>>>> + ierr = MatRestoreRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); >>>>> CHKERRQ(ierr); >>>>> + } >>>>> if( d_nnz[jj] > nloc ) d_nnz[jj] = nloc; >>>>> if( o_nnz[jj] > (MM-nloc) ) o_nnz[jj] = MM - nloc; >>>>> } >>>>> @@ -142,6 +152,9 @@ >>>>> CHKERRQ(ierr); >>>>> ierr = PetscFree( d_nnz ); CHKERRQ(ierr); >>>>> ierr = PetscFree( o_nnz ); CHKERRQ(ierr); >>>>> + if( symm ) { >>>>> + ierr = MatDestroy( &matTrans ); CHKERRQ(ierr); >>>>> + } >>>>> >>>>> >>>>> >>>>> >>>>> On Mar 14, 2012, at 5:53 PM, John Mousel wrote: >>>>> >>>>> Mark, >>>>> >>>>> No change. Can you give me the location that you patched so I can >>>>> check to make sure it pulled? >>>>> I don't see it on the petsc-dev change log. >>>>> >>>>> John >>>>> >>>>> On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams < >>>>> mark.adams at columbia.edu> wrote: >>>>> >>>>>> John, I've committed these changes, give a try. >>>>>> >>>>>> Mark >>>>>> >>>>>> On Mar 14, 2012, at 3:46 PM, Satish Balay wrote: >>>>>> >>>>>> > This is the usual merge [with uncommited changes] issue. >>>>>> > >>>>>> > You could use 'hg shelf' extension to shelve your local changes and >>>>>> > then do a merge [as Sean would suggest] - or do the merge in a >>>>>> > separate/clean clone [I normally do this..] >>>>>> > >>>>>> > i.e >>>>>> > cd ~/Codes >>>>>> > hg clone petsc-dev petsc-dev-merge >>>>>> > cd petsc-dev-merge >>>>>> > hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just >>>>>> to be sure, look for latest chagnes before merge.. >>>>>> > hg merge >>>>>> > hg commit >>>>>> > hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev >>>>>> > >>>>>> > [now update your petsc-dev to latest] >>>>>> > cd ~/Codes/petsc-dev >>>>>> > hg pull >>>>>> > hg update >>>>>> > >>>>>> > Satish >>>>>> > >>>>>> > On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>>>> > >>>>>> >> Great, that seems to work. >>>>>> >> >>>>>> >> I did a 'hg commit tools.c' >>>>>> >> >>>>>> >> and I want to push this file only. I guess its the only thing in >>>>>> the change set so 'hg push' should be fine. But I see this: >>>>>> >> >>>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update >>>>>> >> abort: crosses branches (merge branches or use --clean to discard >>>>>> changes) >>>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge >>>>>> >> abort: outstanding uncommitted changes (use 'hg status' to list >>>>>> changes) >>>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status >>>>>> >> M include/petscmat.h >>>>>> >> M include/private/matimpl.h >>>>>> >> M src/ksp/pc/impls/gamg/agg.c >>>>>> >> M src/ksp/pc/impls/gamg/gamg.c >>>>>> >> M src/ksp/pc/impls/gamg/gamg.h >>>>>> >> M src/ksp/pc/impls/gamg/geo.c >>>>>> >> M src/mat/coarsen/coarsen.c >>>>>> >> M src/mat/coarsen/impls/hem/hem.c >>>>>> >> M src/mat/coarsen/impls/mis/mis.c >>>>>> >> >>>>>> >> Am I ready to do a push? >>>>>> >> >>>>>> >> Thanks, >>>>>> >> Mark >>>>>> >> >>>>>> >> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: >>>>>> >> >>>>>> >>> If commit is the last hg operation that you've done - then 'hg >>>>>> rollback' would undo this commit. >>>>>> >>> >>>>>> >>> Satish >>>>>> >>> >>>>>> >>> On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>>>> >>> >>>>>> >>>> Damn, I'm not preallocating the graph perfectly for unsymmetric >>>>>> matrices and PETSc now dies on this. >>>>>> >>>> >>>>>> >>>> I have a fix but I committed it with other changes that I do not >>>>>> want to commit. The changes are all in one file so I should be able to >>>>>> just commit this file. >>>>>> >>>> >>>>>> >>>> Anyone know how to delete a commit? >>>>>> >>>> >>>>>> >>>> I've tried: >>>>>> >>>> >>>>>> >>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip >>>>>> 22487:26ffb9eef17f >>>>>> >>>> hg: unknown command 'strip' >>>>>> >>>> 'strip' is provided by the following extension: >>>>>> >>>> >>>>>> >>>> mq manage a stack of patches >>>>>> >>>> >>>>>> >>>> use "hg help extensions" for information on enabling extensions >>>>>> >>>> >>>>>> >>>> But have not figured out how to load extensions. >>>>>> >>>> >>>>>> >>>> Mark >>>>>> >>>> >>>>>> >>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: >>>>>> >>>> >>>>>> >>>>> Mark, >>>>>> >>>>> >>>>>> >>>>> I have a non-symmetric matrix. I am running with the following >>>>>> options. >>>>>> >>>>> >>>>>> >>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual >>>>>> >>>>> >>>>>> >>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new >>>>>> malloc error: >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> 0]PETSC ERROR: --------------------- Error Message >>>>>> ------------------------------------ >>>>>> >>>>> [0]PETSC ERROR: Argument out of range! >>>>>> >>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! >>>>>> >>>>> [0]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> >>>>> [0]PETSC ERROR: Petsc Development HG revision: >>>>>> 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 >>>>>> -0500 >>>>>> >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>> >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble >>>>>> shooting. >>>>>> >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>>> >>>>> [0]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> >>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named >>>>>> wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 >>>>>> >>>>> [0]PETSC ERROR: Libraries linked from >>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib >>>>>> >>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 >>>>>> >>>>> [0]PETSC ERROR: Configure options --download-blacs=1 >>>>>> --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 >>>>>> --download-parmetis=1 --download-scalapack=1 >>>>>> --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc >>>>>> --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort >>>>>> PETSC_ARCH=linux-debug >>>>>> >>>>> [0]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> >>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in >>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >>>>>> >>>>> [0]PETSC ERROR: MatSetValues() line 1141 in >>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c >>>>>> >>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in >>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c >>>>>> >>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in >>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c >>>>>> >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in >>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>>>>> >>>>> [0]PETSC ERROR: PCSetUp() line 832 in >>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c >>>>>> >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in >>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>>> >>>>> [0]PETSC ERROR: KSPSolve() line 385 in >>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> John >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams < >>>>>> mark.adams at columbia.edu> wrote: >>>>>> >>>>> >>>>>> >>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: >>>>>> >>>>> >>>>>> >>>>>> Mark, >>>>>> >>>>>> >>>>>> >>>>>> The matrix is asymmetric. Does this require the setting of an >>>>>> option? >>>>>> >>>>> >>>>>> >>>>> Yes: -pc_gamg_sym_graph >>>>>> >>>>> >>>>>> >>>>> Mark >>>>>> >>>>> >>>>>> >>>>>> I pulled petsc-dev this morning, so I should have (at least >>>>>> close to) the latest code. >>>>>> >>>>>> >>>>>> >>>>>> John >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams < >>>>>> mark.adams at columbia.edu> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >>>>>> >>>>>> >>>>>> >>>>>>> I'm getting the following error when using GAMG. >>>>>> >>>>>>> >>>>>> >>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: >>>>>> Assertion `sgid==-1' failed. >>>>>> >>>>>> >>>>>> >>>>>> Is it possible that your matrix is structurally asymmetric? >>>>>> >>>>>> >>>>>> >>>>>> This code is evolving fast and so you will need to move to the >>>>>> dev version if you are not already using it. (I think I fixed a bug that >>>>>> hit this assert). >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>> >>>>>>> When I try to alter the type of aggregation at the command >>>>>> line using -pc_gamg_type pa, I'm getting >>>>>> >>>>>>> >>>>>> >>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error >>>>>> Message ------------------------------------ >>>>>> >>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or >>>>>> missing external package needed for type: >>>>>> >>>>>>> see >>>>>> http://www.mcs.anl.gov/petsc/documentation/installation.html#external >>>>>> ! >>>>>> >>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given! >>>>>> >>>>>>> >>>>>> >>>>>>> Has there been a change in the aggregation options? I just >>>>>> pulled petsc-dev this morning. >>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg >>>>>> for now. >>>>>> >>>>>> >>>>>> >>>>>> Mark >>>>>> >>>>>> >>>>>> >>>>>>> John >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>> >>>>>> >>> >>>>>> >> >>>>>> >> >>>>>> > >>>>>> > >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> >>> >>> >>> >>> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zengshixiangze at 163.com Tue Mar 20 09:31:37 2012 From: zengshixiangze at 163.com (Xiangze Zeng) Date: Tue, 20 Mar 2012 22:31:37 +0800 (CST) Subject: [petsc-users] Performances of different PCTYPE and KSPTYPE Message-ID: <26b0fa07.2fe09.136308531b1.Coremail.zengshixiangze@163.com> Hi, Is there any information about performances of different PCTYPE and KSPTYPE? Thank you very much! Zeng -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Mar 20 09:37:12 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 20 Mar 2012 09:37:12 -0500 Subject: [petsc-users] Performances of different PCTYPE and KSPTYPE In-Reply-To: <26b0fa07.2fe09.136308531b1.Coremail.zengshixiangze@163.com> References: <26b0fa07.2fe09.136308531b1.Coremail.zengshixiangze@163.com> Message-ID: On Tue, Mar 20, 2012 at 9:31 AM, Xiangze Zeng wrote: > Hi, > > Is there any information about performances of different PCTYPE and > KSPTYPE? > > Thank you very much! > The question does not make sense since the performance is highly dependent on the system being solved. Thanks, Matt > Zeng > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Tue Mar 20 10:29:52 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Tue, 20 Mar 2012 11:29:52 -0400 Subject: [petsc-users] SNES convergence tol question Message-ID: <16670314-575E-4A0B-B9E4-CA1E61DC046A@columbia.edu> We've been trying to drive the nonlinear residual down and SNES seems to return prematurely. Appended is Dan's note. We are asking for -snes_rtol 1.e-12 but only getting ~1.e-8. Anyone have any ideas? Mark > Nonlinear solve converged due to CONVERGED_PNORM_RELATIVE iterations 8 However, here are the initial and final norms: > 0 SNES Function norm 1.999411334131e+08 > 8 SNES Function norm 5.999267565004e-01 2e8/6e-1 = 3.3e8. So, I'm confused. Am I misunderstanding the convergence criterion? here's the dump of petsc options at the end: > #PETSc Option Table entries: > -ksp_converged_use_initial_residual_norm > -ksp_max_it 100 > -ksp_monitor > -ksp_rtol 1.e-6 > -ksp_type gmres > -mg_levels_ksp_max_it 4 > -options_left > -pc_gamg_agg_nsmooths 1 > -pc_gamg_sym_graph > -pc_gamg_threshold .01 > -pc_gamg_type agg > -pc_gamg_verbose 2 > -pc_hypre_boomeramg_agg_nl 1 > -pc_hypre_boomeramg_coarsen_type HMIS > -pc_hypre_boomeramg_interp_type ext+i > -pc_hypre_boomeramg_no CF > -pc_hypre_type boomeramg > -pc_type gamg > -snes_converged_reason > -snes_mf_operator > -snes_monitor > -snes_rtol 1.e-12 > #End of PETSc Option Table entries Mark From prbrune at gmail.com Tue Mar 20 10:40:19 2012 From: prbrune at gmail.com (Peter Brune) Date: Tue, 20 Mar 2012 10:40:19 -0500 Subject: [petsc-users] SNES convergence tol question In-Reply-To: <16670314-575E-4A0B-B9E4-CA1E61DC046A@columbia.edu> References: <16670314-575E-4A0B-B9E4-CA1E61DC046A@columbia.edu> Message-ID: What you're seeing is convergence in the step tolerance, not the relative tolerance applied to the norm of the residual. This means that the norm of the steplength between successive iterations has gone below snes->xtol times the norm of the solution. This appears to be set through -snes_stol. - Peter On Tue, Mar 20, 2012 at 10:29 AM, Mark F. Adams wrote: > We've been trying to drive the nonlinear residual down and SNES seems to > return prematurely. > > Appended is Dan's note. We are asking for -snes_rtol 1.e-12 but only > getting ~1.e-8. > > Anyone have any ideas? > > Mark > > > Nonlinear solve converged due to CONVERGED_PNORM_RELATIVE iterations 8 > > However, here are the initial and final norms: > > > 0 SNES Function norm 1.999411334131e+08 > > 8 SNES Function norm 5.999267565004e-01 > > 2e8/6e-1 = 3.3e8. So, I'm confused. Am I misunderstanding the convergence > criterion? > > here's the dump of petsc options at the end: > > > #PETSc Option Table entries: > > -ksp_converged_use_initial_residual_norm > > -ksp_max_it 100 > > -ksp_monitor > > -ksp_rtol 1.e-6 > > -ksp_type gmres > > -mg_levels_ksp_max_it 4 > > -options_left > > -pc_gamg_agg_nsmooths 1 > > -pc_gamg_sym_graph > > -pc_gamg_threshold .01 > > -pc_gamg_type agg > > -pc_gamg_verbose 2 > > -pc_hypre_boomeramg_agg_nl 1 > > -pc_hypre_boomeramg_coarsen_type HMIS > > -pc_hypre_boomeramg_interp_type ext+i > > -pc_hypre_boomeramg_no CF > > -pc_hypre_type boomeramg > > -pc_type gamg > > -snes_converged_reason > > -snes_mf_operator > > -snes_monitor > > -snes_rtol 1.e-12 > > #End of PETSc Option Table entries > > > > Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Tue Mar 20 11:11:14 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Tue, 20 Mar 2012 12:11:14 -0400 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> Message-ID: <824A7702-AB87-4F0C-B3F6-3529A5BDB420@columbia.edu> On Mar 20, 2012, at 10:19 AM, John Mousel wrote: > Mark, > > Sorry for the late reply. I've been on travel and hadn't had a chance to pick this back up. I've tried running with the suggested options: > > -ksp_type bcgsl -pc_type gamg -pc_gamg_coarse_eq_limit 10 -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_diagonal_scale -ksp_diagonal_scale_fix -ksp_monitor_true_residual -ksp_view -pc_gamg_verbose 1 > -mg_levels_ksp_type richardson -mg_levels_pc_type sor seems to have gotten lost. HYPRE uses this and I believe your ML runs used this as well. I will test this with your matrix. Mark > With these options, the convergence starts to hang (see attached GAMG_kspview.txt). The hanging happens for both -mg_coarse_ksp_type richardson and preonly. It was my understanding from previous emails that using preonly made it so that only the preconditioner was run, which in this case would be 8 sweeps of SOR. If I get rid of the -pc_gamg_agg_nsmooths 1 (see GAMG_kspview_nosmooth.txt), the problem converges, but again the convergence is slow. Without this option, both Richardson and preonly converge in 172 iterations. > > Matt, I've checked and the problem does converge in the true residual using GAMG, ML, HYPRE, and ILU preconditioned BiCG. I explicitly ensure that a solution exists by projecting the rhs vector out of the nullity of the transpose of operator. > > John > > > On Fri, Mar 16, 2012 at 2:04 PM, Mark F. Adams wrote: > John, did this get resolved? > Mark > > On Mar 15, 2012, at 4:24 PM, John Mousel wrote: > >> Mark, >> >> Running without the options you mentioned before leads to slightly worse performance (175 iterations). >> I have not been able to get run coarse grid solve to work with LU while running ML. It keeps experiencing a zero pivot, and all the combinations of shifting i've tried haven't lead me anywhere, hence the SOR on the course grid. Also, the ML manual suggests limiting the number of levels to 3 or 4 and performing a few sweeps of an iterative method as opposed to a direct solve. >> >> John >> >> On Thu, Mar 15, 2012 at 12:04 PM, Mark F. Adams wrote: >> You also want: -pc_gamg_agg_nsmooths 1 >> >> You are running plain aggregation. If it is Poisson then smoothing is good. >> >> Is this problem singular? Can you try running ML with these parameters and see if its performance degrades? The ML implementation uses the PETSC infrastructure and uses a very similar algorithm to GAMG-SA. We should be able to get these two to match pretty well. >> >> Mark >> >> >> On Mar 15, 2012, at 12:21 PM, John Mousel wrote: >> >>> Mark, >>> >>> I ran with those options removed (see the run options listed below). Things actually got slightly worse. Now it's up to 142 iterations. I have attached the ksp_view output. >>> >>> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale -ksp_diagonal_scale_fix -mg_levels_ksp_type richardson -mg_levels_pc_type sor -pc_gamg_verbose 1 >>> >>> >>> John >>> >>> >>> On Thu, Mar 15, 2012 at 10:55 AM, Mark F. Adams wrote: >>> John, can you run again with: -pc_gamg_verbose 1 >>> >>> And I would not use: -pc_mg_levels 4 -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 >>> >>> 1) I think -mg_coarse_ksp_type preonly and -mg_coarse_pc_sor_its 8 do not do what you think. I think this is the same as 1 iteration. I think you want 'richardson' not 'preonly'. >>> >>> 2) Why are you using sor as the coarse solver? If your problem is singular then you want to use as many levels as possible to get the coarse grid to be tiny. I'm pretty sure HYPRE ignores the coarse solver parameters. But ML uses them and it is converging well. >>> >>> 3) I would not specify the number of levels. GAMG, and I think the rest, have internal logic for stopping a the right level. If the coarse level is large and you use just 8 iterations of sor then convergence will suffer. >>> >>> Mark >>> >>> On Mar 15, 2012, at 11:13 AM, John Mousel wrote: >>> >>>> Mark, >>>> >>>> The changes pulled through this morning. I've run it with the options >>>> >>>> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale -ksp_diagonal_scale_fix -pc_mg_levels 4 -mg_levels_ksp_type richardson -mg_levels_pc_type sor -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 >>>> >>>> and it converges in the true residual, but it's not converging as fast as anticpated. The matrix arises from a non-symmetric discretization of the Poisson equation. The solve takes GAMG 114 iterations, whereas ML takes 24 iterations, BoomerAMG takes 22 iterations, and -ksp_type bcgsl -pc_type bjacobi -sub_pc_type ilu -sub_pc_factor_levels 4 takes around 170. I've attached the -ksp_view results for ML,GAMG, and HYPRE. I've attempted to make all the options the same on all levels for ML and GAMG. >>>> >>>> Any thoughts? >>>> >>>> John >>>> >>>> >>>> On Wed, Mar 14, 2012 at 6:04 PM, Mark F. Adams wrote: >>>> Humm, I see it with hg view (appended). >>>> >>>> Satish, my main repo looks hosed. I see this: >>>> >>>> ~/Codes/petsc-dev>hg update >>>> abort: crosses branches (merge branches or use --clean to discard changes) >>>> ~/Codes/petsc-dev>hg merge >>>> abort: branch 'default' has 3 heads - please merge with an explicit rev >>>> (run 'hg heads .' to see heads) >>>> ~/Codes/petsc-dev>hg heads >>>> changeset: 22496:8e2a98268179 >>>> tag: tip >>>> user: Barry Smith >>>> date: Wed Mar 14 16:42:25 2012 -0500 >>>> files: src/vec/is/interface/f90-custom/zindexf90.c src/vec/vec/interface/f90-custom/zvectorf90.c >>>> description: >>>> undoing manually changes I put in because Satish had a better fix >>>> >>>> >>>> changeset: 22492:bda4df63072d >>>> user: Mark F. Adams >>>> date: Wed Mar 14 17:39:52 2012 -0400 >>>> files: src/ksp/pc/impls/gamg/tools.c >>>> description: >>>> fix for unsymmetric matrices. >>>> >>>> >>>> changeset: 22469:b063baf366e4 >>>> user: Mark F. Adams >>>> date: Wed Mar 14 14:22:28 2012 -0400 >>>> files: src/ksp/pc/impls/gamg/tools.c >>>> description: >>>> added fix for preallocation for unsymetric matrices. >>>> >>>> Mark >>>> >>>> my 'hg view' on my merge repo: >>>> >>>> Revision: 22492 >>>> Branch: default >>>> Author: Mark F. Adams 2012-03-14 17:39:52 >>>> Committer: Mark F. Adams 2012-03-14 17:39:52 >>>> Tags: tip >>>> Parent: 22491:451bbbd291c2 (Small fixes to the BT linesearch) >>>> >>>> fix for unsymmetric matrices. >>>> >>>> >>>> ------------------------ src/ksp/pc/impls/gamg/tools.c ------------------------ >>>> @@ -103,7 +103,7 @@ >>>> PetscErrorCode ierr; >>>> PetscInt Istart,Iend,Ii,jj,ncols,nnz0,nnz1, NN, MM, nloc; >>>> PetscMPIInt mype, npe; >>>> - Mat Gmat = *a_Gmat, tGmat; >>>> + Mat Gmat = *a_Gmat, tGmat, matTrans; >>>> MPI_Comm wcomm = ((PetscObject)Gmat)->comm; >>>> const PetscScalar *vals; >>>> const PetscInt *idx; >>>> @@ -127,6 +127,10 @@ >>>> ierr = MatDiagonalScale( Gmat, diag, diag ); CHKERRQ(ierr); >>>> ierr = VecDestroy( &diag ); CHKERRQ(ierr); >>>> >>>> + if( symm ) { >>>> + ierr = MatTranspose( Gmat, MAT_INITIAL_MATRIX, &matTrans ); CHKERRQ(ierr); >>>> + } >>>> + >>>> /* filter - dup zeros out matrix */ >>>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &d_nnz ); CHKERRQ(ierr); >>>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &o_nnz ); CHKERRQ(ierr); >>>> @@ -135,6 +139,12 @@ >>>> d_nnz[jj] = ncols; >>>> o_nnz[jj] = ncols; >>>> ierr = MatRestoreRow(Gmat,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >>>> + if( symm ) { >>>> + ierr = MatGetRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >>>> + d_nnz[jj] += ncols; >>>> + o_nnz[jj] += ncols; >>>> + ierr = MatRestoreRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >>>> + } >>>> if( d_nnz[jj] > nloc ) d_nnz[jj] = nloc; >>>> if( o_nnz[jj] > (MM-nloc) ) o_nnz[jj] = MM - nloc; >>>> } >>>> @@ -142,6 +152,9 @@ >>>> CHKERRQ(ierr); >>>> ierr = PetscFree( d_nnz ); CHKERRQ(ierr); >>>> ierr = PetscFree( o_nnz ); CHKERRQ(ierr); >>>> + if( symm ) { >>>> + ierr = MatDestroy( &matTrans ); CHKERRQ(ierr); >>>> + } >>>> >>>> >>>> >>>> >>>> On Mar 14, 2012, at 5:53 PM, John Mousel wrote: >>>> >>>>> Mark, >>>>> >>>>> No change. Can you give me the location that you patched so I can check to make sure it pulled? >>>>> I don't see it on the petsc-dev change log. >>>>> >>>>> John >>>>> >>>>> On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams wrote: >>>>> John, I've committed these changes, give a try. >>>>> >>>>> Mark >>>>> >>>>> On Mar 14, 2012, at 3:46 PM, Satish Balay wrote: >>>>> >>>>> > This is the usual merge [with uncommited changes] issue. >>>>> > >>>>> > You could use 'hg shelf' extension to shelve your local changes and >>>>> > then do a merge [as Sean would suggest] - or do the merge in a >>>>> > separate/clean clone [I normally do this..] >>>>> > >>>>> > i.e >>>>> > cd ~/Codes >>>>> > hg clone petsc-dev petsc-dev-merge >>>>> > cd petsc-dev-merge >>>>> > hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just to be sure, look for latest chagnes before merge.. >>>>> > hg merge >>>>> > hg commit >>>>> > hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev >>>>> > >>>>> > [now update your petsc-dev to latest] >>>>> > cd ~/Codes/petsc-dev >>>>> > hg pull >>>>> > hg update >>>>> > >>>>> > Satish >>>>> > >>>>> > On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>>> > >>>>> >> Great, that seems to work. >>>>> >> >>>>> >> I did a 'hg commit tools.c' >>>>> >> >>>>> >> and I want to push this file only. I guess its the only thing in the change set so 'hg push' should be fine. But I see this: >>>>> >> >>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update >>>>> >> abort: crosses branches (merge branches or use --clean to discard changes) >>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge >>>>> >> abort: outstanding uncommitted changes (use 'hg status' to list changes) >>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status >>>>> >> M include/petscmat.h >>>>> >> M include/private/matimpl.h >>>>> >> M src/ksp/pc/impls/gamg/agg.c >>>>> >> M src/ksp/pc/impls/gamg/gamg.c >>>>> >> M src/ksp/pc/impls/gamg/gamg.h >>>>> >> M src/ksp/pc/impls/gamg/geo.c >>>>> >> M src/mat/coarsen/coarsen.c >>>>> >> M src/mat/coarsen/impls/hem/hem.c >>>>> >> M src/mat/coarsen/impls/mis/mis.c >>>>> >> >>>>> >> Am I ready to do a push? >>>>> >> >>>>> >> Thanks, >>>>> >> Mark >>>>> >> >>>>> >> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: >>>>> >> >>>>> >>> If commit is the last hg operation that you've done - then 'hg rollback' would undo this commit. >>>>> >>> >>>>> >>> Satish >>>>> >>> >>>>> >>> On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>>> >>> >>>>> >>>> Damn, I'm not preallocating the graph perfectly for unsymmetric matrices and PETSc now dies on this. >>>>> >>>> >>>>> >>>> I have a fix but I committed it with other changes that I do not want to commit. The changes are all in one file so I should be able to just commit this file. >>>>> >>>> >>>>> >>>> Anyone know how to delete a commit? >>>>> >>>> >>>>> >>>> I've tried: >>>>> >>>> >>>>> >>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f >>>>> >>>> hg: unknown command 'strip' >>>>> >>>> 'strip' is provided by the following extension: >>>>> >>>> >>>>> >>>> mq manage a stack of patches >>>>> >>>> >>>>> >>>> use "hg help extensions" for information on enabling extensions >>>>> >>>> >>>>> >>>> But have not figured out how to load extensions. >>>>> >>>> >>>>> >>>> Mark >>>>> >>>> >>>>> >>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: >>>>> >>>> >>>>> >>>>> Mark, >>>>> >>>>> >>>>> >>>>> I have a non-symmetric matrix. I am running with the following options. >>>>> >>>>> >>>>> >>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual >>>>> >>>>> >>>>> >>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new malloc error: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> 0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> >>>>> [0]PETSC ERROR: Argument out of range! >>>>> >>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! >>>>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> >>>>> [0]PETSC ERROR: Petsc Development HG revision: 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 -0500 >>>>> >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> >>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 >>>>> >>>>> [0]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib >>>>> >>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 >>>>> >>>>> [0]PETSC ERROR: Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug >>>>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> >>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >>>>> >>>>> [0]PETSC ERROR: MatSetValues() line 1141 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c >>>>> >>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c >>>>> >>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c >>>>> >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>>>> >>>>> [0]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c >>>>> >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>> >>>>> [0]PETSC ERROR: KSPSolve() line 385 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> John >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams wrote: >>>>> >>>>> >>>>> >>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: >>>>> >>>>> >>>>> >>>>>> Mark, >>>>> >>>>>> >>>>> >>>>>> The matrix is asymmetric. Does this require the setting of an option? >>>>> >>>>> >>>>> >>>>> Yes: -pc_gamg_sym_graph >>>>> >>>>> >>>>> >>>>> Mark >>>>> >>>>> >>>>> >>>>>> I pulled petsc-dev this morning, so I should have (at least close to) the latest code. >>>>> >>>>>> >>>>> >>>>>> John >>>>> >>>>>> >>>>> >>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams wrote: >>>>> >>>>>> >>>>> >>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >>>>> >>>>>> >>>>> >>>>>>> I'm getting the following error when using GAMG. >>>>> >>>>>>> >>>>> >>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion `sgid==-1' failed. >>>>> >>>>>> >>>>> >>>>>> Is it possible that your matrix is structurally asymmetric? >>>>> >>>>>> >>>>> >>>>>> This code is evolving fast and so you will need to move to the dev version if you are not already using it. (I think I fixed a bug that hit this assert). >>>>> >>>>>> >>>>> >>>>>>> >>>>> >>>>>>> When I try to alter the type of aggregation at the command line using -pc_gamg_type pa, I'm getting >>>>> >>>>>>> >>>>> >>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> >>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type: >>>>> >>>>>>> see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! >>>>> >>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given! >>>>> >>>>>>> >>>>> >>>>>>> Has there been a change in the aggregation options? I just pulled petsc-dev this morning. >>>>> >>>>>>> >>>>> >>>>>> >>>>> >>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg for now. >>>>> >>>>>> >>>>> >>>>>> Mark >>>>> >>>>>> >>>>> >>>>>>> John >>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>>> >>>> >>>>> >>> >>>>> >>> >>>>> >> >>>>> >> >>>>> > >>>>> > >>>>> >>>>> >>>> >>>> >>>> >>> >>> >>> >> >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From huangsc at gmail.com Tue Mar 20 12:26:27 2012 From: huangsc at gmail.com (Shao-Ching Huang) Date: Tue, 20 Mar 2012 10:26:27 -0700 Subject: [petsc-users] MatNullSpaceCreate Message-ID: Hi, I have two linear systems: A1*x1=b1 A2*x2=b2 Each of A1 and A2 has a null space with a constant vector (like [1 1 ... 1]') in it, so I currently use MatNullSpaceCreate(with n=0,has_cnst=PETSC_TRUE). Now I want to solve these two systems in one solve, i.e. A*x = b where A = [A1 0; 0 A2] and x=[x1 x2]', b=[b1 b2]' Can I just put two vectors [1 1 1 ... 1 0 0 ... 0]' (i.e. null space from A1) [0 0 0 ... 0 1 1 ... 1]' (i.e. null space from A2) into MatNullSpaceCreate (with n=2,has_cnst=PETSC_FALSE) to span the null space? Thanks. Shao-Ching From knepley at gmail.com Tue Mar 20 13:14:03 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 20 Mar 2012 13:14:03 -0500 Subject: [petsc-users] MatNullSpaceCreate In-Reply-To: References: Message-ID: On Tue, Mar 20, 2012 at 12:26 PM, Shao-Ching Huang wrote: > Hi, > > I have two linear systems: > > A1*x1=b1 > A2*x2=b2 > > Each of A1 and A2 has a null space with a constant vector (like [1 1 > ... 1]') in it, so I currently use MatNullSpaceCreate(with > n=0,has_cnst=PETSC_TRUE). > > Now I want to solve these two systems in one solve, i.e. > > A*x = b > > where A = [A1 0; 0 A2] and x=[x1 x2]', b=[b1 b2]' > > Can I just put two vectors > > [1 1 1 ... 1 0 0 ... 0]' (i.e. null space from A1) > [0 0 0 ... 0 1 1 ... 1]' (i.e. null space from A2) > > into MatNullSpaceCreate (with n=2,has_cnst=PETSC_FALSE) to span the null > space? > Yes, Thanks Matt > Thanks. > > Shao-Ching > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Tue Mar 20 13:33:38 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Tue, 20 Mar 2012 14:33:38 -0400 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> Message-ID: <9CB8B9D0-BC04-4471-BCFC-BEE61031A123@columbia.edu> John, I am getting poor results (diverging) from ML also: 0 KSP preconditioned resid norm 3.699832960909e+22 true resid norm 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 2 KSP preconditioned resid norm 5.706378365783e+11 true resid norm 1.563902233018e+03 ||r(i)||/||b|| 1.193204539995e+00 4 KSP preconditioned resid norm 5.570291685152e+11 true resid norm 1.564235542744e+03 ||r(i)||/||b|| 1.193458844050e+00 6 KSP preconditioned resid norm 5.202150407298e+10 true resid norm 1.749929789082e+03 ||r(i)||/||b|| 1.335137277077e+00 Linear solve converged due to CONVERGED_RTOL iterations 6 With GAMG I get: 0 KSP preconditioned resid norm 7.731260075891e+06 true resid norm 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 2 KSP preconditioned resid norm 2.856415184685e+05 true resid norm 1.310410242531e+03 ||r(i)||/||b|| 9.997987199150e-01 4 KSP preconditioned resid norm 1.528467019258e+05 true resid norm 1.284856538976e+03 ||r(i)||/||b|| 9.803021078816e-01 6 KSP preconditioned resid norm 1.451091957899e+05 true resid norm 1.564309254168e+03 ||r(i)||/||b|| 1.193515083375e+00 122 KSP preconditioned resid norm 2.486245341783e+01 true resid norm 1.404397185367e+00 ||r(i)||/||b|| 1.071507580306e-03 124 KSP preconditioned resid norm 1.482316853621e+01 true resid norm 4.488661881759e-01 ||r(i)||/||b|| 3.424697287811e-04 126 KSP preconditioned resid norm 1.481941150253e+01 true resid norm 4.484480100832e-01 ||r(i)||/||b|| 3.421506730318e-04 128 KSP preconditioned resid norm 8.191887347033e+00 true resid norm 6.678630367218e-01 ||r(i)||/||b|| 5.095569215816e-04 And HYPRE: 0 KSP preconditioned resid norm 3.774510769907e+04 true resid norm 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 2 KSP preconditioned resid norm 1.843165835831e+04 true resid norm 8.502433792869e+02 ||r(i)||/||b|| 6.487069580482e-01 4 KSP preconditioned resid norm 1.573176624705e+04 true resid norm 1.167264367302e+03 ||r(i)||/||b|| 8.905832558033e-01 6 KSP preconditioned resid norm 1.657958380765e+04 true resid norm 8.684701624902e+02 ||r(i)||/||b|| 6.626133775216e-01 8 KSP preconditioned resid norm 2.190304455083e+04 true resid norm 6.969893263600e+02 ||r(i)||/||b|| 5.317792960344e-01 10 KSP preconditioned resid norm 2.485714630000e+04 true resid norm 6.642641436830e+02 ||r(i)||/||b|| 5.068110878446e-01 62 KSP preconditioned resid norm 6.432516040957e+00 true resid norm 2.124960171419e-01 ||r(i)||/||b|| 1.621272781837e-04 64 KSP preconditioned resid norm 5.731033176541e+00 true resid norm 1.338816774003e-01 ||r(i)||/||b|| 1.021471943216e-04 66 KSP preconditioned resid norm 1.600285935522e-01 true resid norm 3.352408932031e-03 ||r(i)||/||b|| 2.557774695353e-06 ML and GAMG should act similarly, but ML seems to have a problem (see the preconditioned norm difference and its diverging). ML has a parameter: -pc_ml_Threshold [.0] I setting this to 0.05 (GAMG default) helps a bit but it still diverges. So it would be nice to figure out the difference between ML and GAMG, but that is secondary for you as the both suck. HYPRE is a very different algorithm. It looks like the smoothing in GAMG (and ML) may be the problem. If I turn smoothing off (-pc_gamg_agg_nsmooths 0) and I get for GAMG: 0 KSP preconditioned resid norm 2.186148437534e+05 true resid norm 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 2 KSP preconditioned resid norm 2.916843959765e+04 true resid norm 3.221533667508e+03 ||r(i)||/||b|| 2.457921292432e+00 4 KSP preconditioned resid norm 2.396374655925e+04 true resid norm 1.834299897412e+03 ||r(i)||/||b|| 1.399508817812e+00 6 KSP preconditioned resid norm 2.509576275453e+04 true resid norm 1.035475461174e+03 ||r(i)||/||b|| 7.900327752214e-01 64 KSP preconditioned resid norm 1.973859758284e+01 true resid norm 7.322674977169e+00 ||r(i)||/||b|| 5.586953482895e-03 66 KSP preconditioned resid norm 3.371598890438e+01 true resid norm 7.152754930495e+00 ||r(i)||/||b|| 5.457310231004e-03 68 KSP preconditioned resid norm 4.687839294418e+00 true resid norm 4.605726307025e-01 ||r(i)||/||b|| 3.514013487219e-04 70 KSP preconditioned resid norm 1.487545519493e+00 true resid norm 1.558723789416e-01 ||r(i)||/||b|| 1.189253562571e-04 72 KSP preconditioned resid norm 5.317329808718e-01 true resid norm 5.027178038177e-02 ||r(i)||/||b|| 3.835566911967e-05 74 KSP preconditioned resid norm 3.405339702462e-01 true resid norm 1.897059263835e-02 ||r(i)||/||b|| 1.447392092969e-05 This is almost as good as HYPRE. An other thing to keep in mind is the cost of each iteration, not just then number of iterations, You can use -pc_hypre_boomeramg_print_statistics to get some data on this from HYPRE: Average Convergence Factor = 0.537664 Complexity: grid = 1.780207 operator = 2.624910 cycle = 5.249670 And GAMG prints this with verbose set: [0]PCSetUp_GAMG 6 levels, grid compexity [sic] = 1.1316 I believe that the hypre "Complexity: grid" is the same as my "grid complexity". So hypre actually looks more expensive at this point. I've worked on optimizing parameters for hypre with the hypre people and here are a set of arguments that I've used: -pc_hypre_boomeramg_no_CF -pc_hypre_boomeramg_agg_nl 1 -pc_hypre_boomeramg_coarsen_type HMIS -pc_hypre_boomeramg_interp_type ext+i -pc_hypre_boomeramg_P_max 4 -pc_hypre_boomeramg_agg_num_paths 2 With these parmeters I get: Complexity: grid = 1.244140 operator = 1.396722 cycle = 2.793442 and: 0 KSP preconditioned resid norm 4.698624821403e+04 true resid norm 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 2 KSP preconditioned resid norm 2.207967626172e+04 true resid norm 3.466160021150e+03 ||r(i)||/||b|| 2.644562931280e+00 4 KSP preconditioned resid norm 2.278468320876e+04 true resid norm 1.246784122467e+03 ||r(i)||/||b|| 9.512541410282e-01 56 KSP preconditioned resid norm 1.108460232262e+00 true resid norm 8.276869475681e-02 ||r(i)||/||b|| 6.314971631105e-05 58 KSP preconditioned resid norm 3.617217454336e-01 true resid norm 3.764556404754e-02 ||r(i)||/||b|| 2.872229285428e-05 60 KSP preconditioned resid norm 1.666532560770e-01 true resid norm 5.149302513338e-03 ||r(i)||/||b|| 3.928743758404e-06 Linear solve converged due to CONVERGED_RTOL iterations 60 So this actually converged faster with lower complexity. Anyway these result seem different from what you are getting, so I've appended my options. This uses ex10 in the KSP tutorials to read in your binary file. Mark #PETSc Option Table entries: -f ./binaryoutput -ksp_converged_reason -ksp_diagonal_scale -ksp_diagonal_scale_fix -ksp_monitor_true_residual -ksp_type bcgsl -mg_coarse_ksp_type richardson -mg_coarse_pc_sor_its 8 -mg_coarse_pc_type sor -mg_levels_ksp_type richardson -mg_levels_pc_type sor -options_left -pc_gamg_agg_nsmooths 0 -pc_gamg_coarse_eq_limit 10 -pc_gamg_sym_graph -pc_gamg_verbose 2 -pc_hypre_boomeramg_P_max 4 -pc_hypre_boomeramg_agg_nl 1 -pc_hypre_boomeramg_agg_num_paths 2 -pc_hypre_boomeramg_coarsen_type HMIS -pc_hypre_boomeramg_interp_type ext+i -pc_hypre_boomeramg_no_CF -pc_ml_Threshold .01 -pc_type gamg -vecload_block_size 1 #End of PETSc Option Table entries There are 7 unused database options. They are: Option left: name:-pc_hypre_boomeramg_P_max value: 4 Option left: name:-pc_hypre_boomeramg_agg_nl value: 1 Option left: name:-pc_hypre_boomeramg_agg_num_paths value: 2 Option left: name:-pc_hypre_boomeramg_coarsen_type value: HMIS Option left: name:-pc_hypre_boomeramg_interp_type value: ext+i Option left: name:-pc_hypre_boomeramg_no_CF no value Option left: name:-pc_ml_Threshold value: .01 On Mar 20, 2012, at 10:19 AM, John Mousel wrote: > Mark, > > Sorry for the late reply. I've been on travel and hadn't had a chance to pick this back up. I've tried running with the suggested options: > > -ksp_type bcgsl -pc_type gamg -pc_gamg_coarse_eq_limit 10 -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_diagonal_scale -ksp_diagonal_scale_fix -ksp_monitor_true_residual -ksp_view -pc_gamg_verbose 1 > > With these options, the convergence starts to hang (see attached GAMG_kspview.txt). The hanging happens for both -mg_coarse_ksp_type richardson and preonly. It was my understanding from previous emails that using preonly made it so that only the preconditioner was run, which in this case would be 8 sweeps of SOR. If I get rid of the -pc_gamg_agg_nsmooths 1 (see GAMG_kspview_nosmooth.txt), the problem converges, but again the convergence is slow. Without this option, both Richardson and preonly converge in 172 iterations. > > Matt, I've checked and the problem does converge in the true residual using GAMG, ML, HYPRE, and ILU preconditioned BiCG. I explicitly ensure that a solution exists by projecting the rhs vector out of the nullity of the transpose of operator. > > John > > > On Fri, Mar 16, 2012 at 2:04 PM, Mark F. Adams wrote: > John, did this get resolved? > Mark > > On Mar 15, 2012, at 4:24 PM, John Mousel wrote: > >> Mark, >> >> Running without the options you mentioned before leads to slightly worse performance (175 iterations). >> I have not been able to get run coarse grid solve to work with LU while running ML. It keeps experiencing a zero pivot, and all the combinations of shifting i've tried haven't lead me anywhere, hence the SOR on the course grid. Also, the ML manual suggests limiting the number of levels to 3 or 4 and performing a few sweeps of an iterative method as opposed to a direct solve. >> >> John >> >> On Thu, Mar 15, 2012 at 12:04 PM, Mark F. Adams wrote: >> You also want: -pc_gamg_agg_nsmooths 1 >> >> You are running plain aggregation. If it is Poisson then smoothing is good. >> >> Is this problem singular? Can you try running ML with these parameters and see if its performance degrades? The ML implementation uses the PETSC infrastructure and uses a very similar algorithm to GAMG-SA. We should be able to get these two to match pretty well. >> >> Mark >> >> >> On Mar 15, 2012, at 12:21 PM, John Mousel wrote: >> >>> Mark, >>> >>> I ran with those options removed (see the run options listed below). Things actually got slightly worse. Now it's up to 142 iterations. I have attached the ksp_view output. >>> >>> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale -ksp_diagonal_scale_fix -mg_levels_ksp_type richardson -mg_levels_pc_type sor -pc_gamg_verbose 1 >>> >>> >>> John >>> >>> >>> On Thu, Mar 15, 2012 at 10:55 AM, Mark F. Adams wrote: >>> John, can you run again with: -pc_gamg_verbose 1 >>> >>> And I would not use: -pc_mg_levels 4 -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 >>> >>> 1) I think -mg_coarse_ksp_type preonly and -mg_coarse_pc_sor_its 8 do not do what you think. I think this is the same as 1 iteration. I think you want 'richardson' not 'preonly'. >>> >>> 2) Why are you using sor as the coarse solver? If your problem is singular then you want to use as many levels as possible to get the coarse grid to be tiny. I'm pretty sure HYPRE ignores the coarse solver parameters. But ML uses them and it is converging well. >>> >>> 3) I would not specify the number of levels. GAMG, and I think the rest, have internal logic for stopping a the right level. If the coarse level is large and you use just 8 iterations of sor then convergence will suffer. >>> >>> Mark >>> >>> On Mar 15, 2012, at 11:13 AM, John Mousel wrote: >>> >>>> Mark, >>>> >>>> The changes pulled through this morning. I've run it with the options >>>> >>>> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale -ksp_diagonal_scale_fix -pc_mg_levels 4 -mg_levels_ksp_type richardson -mg_levels_pc_type sor -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 >>>> >>>> and it converges in the true residual, but it's not converging as fast as anticpated. The matrix arises from a non-symmetric discretization of the Poisson equation. The solve takes GAMG 114 iterations, whereas ML takes 24 iterations, BoomerAMG takes 22 iterations, and -ksp_type bcgsl -pc_type bjacobi -sub_pc_type ilu -sub_pc_factor_levels 4 takes around 170. I've attached the -ksp_view results for ML,GAMG, and HYPRE. I've attempted to make all the options the same on all levels for ML and GAMG. >>>> >>>> Any thoughts? >>>> >>>> John >>>> >>>> >>>> On Wed, Mar 14, 2012 at 6:04 PM, Mark F. Adams wrote: >>>> Humm, I see it with hg view (appended). >>>> >>>> Satish, my main repo looks hosed. I see this: >>>> >>>> ~/Codes/petsc-dev>hg update >>>> abort: crosses branches (merge branches or use --clean to discard changes) >>>> ~/Codes/petsc-dev>hg merge >>>> abort: branch 'default' has 3 heads - please merge with an explicit rev >>>> (run 'hg heads .' to see heads) >>>> ~/Codes/petsc-dev>hg heads >>>> changeset: 22496:8e2a98268179 >>>> tag: tip >>>> user: Barry Smith >>>> date: Wed Mar 14 16:42:25 2012 -0500 >>>> files: src/vec/is/interface/f90-custom/zindexf90.c src/vec/vec/interface/f90-custom/zvectorf90.c >>>> description: >>>> undoing manually changes I put in because Satish had a better fix >>>> >>>> >>>> changeset: 22492:bda4df63072d >>>> user: Mark F. Adams >>>> date: Wed Mar 14 17:39:52 2012 -0400 >>>> files: src/ksp/pc/impls/gamg/tools.c >>>> description: >>>> fix for unsymmetric matrices. >>>> >>>> >>>> changeset: 22469:b063baf366e4 >>>> user: Mark F. Adams >>>> date: Wed Mar 14 14:22:28 2012 -0400 >>>> files: src/ksp/pc/impls/gamg/tools.c >>>> description: >>>> added fix for preallocation for unsymetric matrices. >>>> >>>> Mark >>>> >>>> my 'hg view' on my merge repo: >>>> >>>> Revision: 22492 >>>> Branch: default >>>> Author: Mark F. Adams 2012-03-14 17:39:52 >>>> Committer: Mark F. Adams 2012-03-14 17:39:52 >>>> Tags: tip >>>> Parent: 22491:451bbbd291c2 (Small fixes to the BT linesearch) >>>> >>>> fix for unsymmetric matrices. >>>> >>>> >>>> ------------------------ src/ksp/pc/impls/gamg/tools.c ------------------------ >>>> @@ -103,7 +103,7 @@ >>>> PetscErrorCode ierr; >>>> PetscInt Istart,Iend,Ii,jj,ncols,nnz0,nnz1, NN, MM, nloc; >>>> PetscMPIInt mype, npe; >>>> - Mat Gmat = *a_Gmat, tGmat; >>>> + Mat Gmat = *a_Gmat, tGmat, matTrans; >>>> MPI_Comm wcomm = ((PetscObject)Gmat)->comm; >>>> const PetscScalar *vals; >>>> const PetscInt *idx; >>>> @@ -127,6 +127,10 @@ >>>> ierr = MatDiagonalScale( Gmat, diag, diag ); CHKERRQ(ierr); >>>> ierr = VecDestroy( &diag ); CHKERRQ(ierr); >>>> >>>> + if( symm ) { >>>> + ierr = MatTranspose( Gmat, MAT_INITIAL_MATRIX, &matTrans ); CHKERRQ(ierr); >>>> + } >>>> + >>>> /* filter - dup zeros out matrix */ >>>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &d_nnz ); CHKERRQ(ierr); >>>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &o_nnz ); CHKERRQ(ierr); >>>> @@ -135,6 +139,12 @@ >>>> d_nnz[jj] = ncols; >>>> o_nnz[jj] = ncols; >>>> ierr = MatRestoreRow(Gmat,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >>>> + if( symm ) { >>>> + ierr = MatGetRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >>>> + d_nnz[jj] += ncols; >>>> + o_nnz[jj] += ncols; >>>> + ierr = MatRestoreRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >>>> + } >>>> if( d_nnz[jj] > nloc ) d_nnz[jj] = nloc; >>>> if( o_nnz[jj] > (MM-nloc) ) o_nnz[jj] = MM - nloc; >>>> } >>>> @@ -142,6 +152,9 @@ >>>> CHKERRQ(ierr); >>>> ierr = PetscFree( d_nnz ); CHKERRQ(ierr); >>>> ierr = PetscFree( o_nnz ); CHKERRQ(ierr); >>>> + if( symm ) { >>>> + ierr = MatDestroy( &matTrans ); CHKERRQ(ierr); >>>> + } >>>> >>>> >>>> >>>> >>>> On Mar 14, 2012, at 5:53 PM, John Mousel wrote: >>>> >>>>> Mark, >>>>> >>>>> No change. Can you give me the location that you patched so I can check to make sure it pulled? >>>>> I don't see it on the petsc-dev change log. >>>>> >>>>> John >>>>> >>>>> On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams wrote: >>>>> John, I've committed these changes, give a try. >>>>> >>>>> Mark >>>>> >>>>> On Mar 14, 2012, at 3:46 PM, Satish Balay wrote: >>>>> >>>>> > This is the usual merge [with uncommited changes] issue. >>>>> > >>>>> > You could use 'hg shelf' extension to shelve your local changes and >>>>> > then do a merge [as Sean would suggest] - or do the merge in a >>>>> > separate/clean clone [I normally do this..] >>>>> > >>>>> > i.e >>>>> > cd ~/Codes >>>>> > hg clone petsc-dev petsc-dev-merge >>>>> > cd petsc-dev-merge >>>>> > hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just to be sure, look for latest chagnes before merge.. >>>>> > hg merge >>>>> > hg commit >>>>> > hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev >>>>> > >>>>> > [now update your petsc-dev to latest] >>>>> > cd ~/Codes/petsc-dev >>>>> > hg pull >>>>> > hg update >>>>> > >>>>> > Satish >>>>> > >>>>> > On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>>> > >>>>> >> Great, that seems to work. >>>>> >> >>>>> >> I did a 'hg commit tools.c' >>>>> >> >>>>> >> and I want to push this file only. I guess its the only thing in the change set so 'hg push' should be fine. But I see this: >>>>> >> >>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update >>>>> >> abort: crosses branches (merge branches or use --clean to discard changes) >>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge >>>>> >> abort: outstanding uncommitted changes (use 'hg status' to list changes) >>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status >>>>> >> M include/petscmat.h >>>>> >> M include/private/matimpl.h >>>>> >> M src/ksp/pc/impls/gamg/agg.c >>>>> >> M src/ksp/pc/impls/gamg/gamg.c >>>>> >> M src/ksp/pc/impls/gamg/gamg.h >>>>> >> M src/ksp/pc/impls/gamg/geo.c >>>>> >> M src/mat/coarsen/coarsen.c >>>>> >> M src/mat/coarsen/impls/hem/hem.c >>>>> >> M src/mat/coarsen/impls/mis/mis.c >>>>> >> >>>>> >> Am I ready to do a push? >>>>> >> >>>>> >> Thanks, >>>>> >> Mark >>>>> >> >>>>> >> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: >>>>> >> >>>>> >>> If commit is the last hg operation that you've done - then 'hg rollback' would undo this commit. >>>>> >>> >>>>> >>> Satish >>>>> >>> >>>>> >>> On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>>> >>> >>>>> >>>> Damn, I'm not preallocating the graph perfectly for unsymmetric matrices and PETSc now dies on this. >>>>> >>>> >>>>> >>>> I have a fix but I committed it with other changes that I do not want to commit. The changes are all in one file so I should be able to just commit this file. >>>>> >>>> >>>>> >>>> Anyone know how to delete a commit? >>>>> >>>> >>>>> >>>> I've tried: >>>>> >>>> >>>>> >>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f >>>>> >>>> hg: unknown command 'strip' >>>>> >>>> 'strip' is provided by the following extension: >>>>> >>>> >>>>> >>>> mq manage a stack of patches >>>>> >>>> >>>>> >>>> use "hg help extensions" for information on enabling extensions >>>>> >>>> >>>>> >>>> But have not figured out how to load extensions. >>>>> >>>> >>>>> >>>> Mark >>>>> >>>> >>>>> >>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: >>>>> >>>> >>>>> >>>>> Mark, >>>>> >>>>> >>>>> >>>>> I have a non-symmetric matrix. I am running with the following options. >>>>> >>>>> >>>>> >>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual >>>>> >>>>> >>>>> >>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new malloc error: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> 0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> >>>>> [0]PETSC ERROR: Argument out of range! >>>>> >>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! >>>>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> >>>>> [0]PETSC ERROR: Petsc Development HG revision: 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 -0500 >>>>> >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> >>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 >>>>> >>>>> [0]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib >>>>> >>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 >>>>> >>>>> [0]PETSC ERROR: Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug >>>>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> >>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >>>>> >>>>> [0]PETSC ERROR: MatSetValues() line 1141 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c >>>>> >>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c >>>>> >>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c >>>>> >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>>>> >>>>> [0]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c >>>>> >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>> >>>>> [0]PETSC ERROR: KSPSolve() line 385 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> John >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams wrote: >>>>> >>>>> >>>>> >>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: >>>>> >>>>> >>>>> >>>>>> Mark, >>>>> >>>>>> >>>>> >>>>>> The matrix is asymmetric. Does this require the setting of an option? >>>>> >>>>> >>>>> >>>>> Yes: -pc_gamg_sym_graph >>>>> >>>>> >>>>> >>>>> Mark >>>>> >>>>> >>>>> >>>>>> I pulled petsc-dev this morning, so I should have (at least close to) the latest code. >>>>> >>>>>> >>>>> >>>>>> John >>>>> >>>>>> >>>>> >>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams wrote: >>>>> >>>>>> >>>>> >>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >>>>> >>>>>> >>>>> >>>>>>> I'm getting the following error when using GAMG. >>>>> >>>>>>> >>>>> >>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion `sgid==-1' failed. >>>>> >>>>>> >>>>> >>>>>> Is it possible that your matrix is structurally asymmetric? >>>>> >>>>>> >>>>> >>>>>> This code is evolving fast and so you will need to move to the dev version if you are not already using it. (I think I fixed a bug that hit this assert). >>>>> >>>>>> >>>>> >>>>>>> >>>>> >>>>>>> When I try to alter the type of aggregation at the command line using -pc_gamg_type pa, I'm getting >>>>> >>>>>>> >>>>> >>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> >>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type: >>>>> >>>>>>> see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! >>>>> >>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given! >>>>> >>>>>>> >>>>> >>>>>>> Has there been a change in the aggregation options? I just pulled petsc-dev this morning. >>>>> >>>>>>> >>>>> >>>>>> >>>>> >>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg for now. >>>>> >>>>>> >>>>> >>>>>> Mark >>>>> >>>>>> >>>>> >>>>>>> John >>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>>> >>>> >>>>> >>> >>>>> >>> >>>>> >> >>>>> >> >>>>> > >>>>> > >>>>> >>>>> >>>> >>>> >>>> >>> >>> >>> >> >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.mousel at gmail.com Tue Mar 20 13:45:30 2012 From: john.mousel at gmail.com (John Mousel) Date: Tue, 20 Mar 2012 13:45:30 -0500 Subject: [petsc-users] GAMG issue In-Reply-To: <9CB8B9D0-BC04-4471-BCFC-BEE61031A123@columbia.edu> References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> <9CB8B9D0-BC04-4471-BCFC-BEE61031A123@columbia.edu> Message-ID: Mark, I run ML with the following options. -ksp_type bcgsl -pc_type ml -pc_ml_maxNlevels 5 -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_monitor -ksp_view Note the lack of scaling. For some reason scaling seems to mess with ML. As you can see below, ML converges very nicely. With regards to HYPRE, this one took a bit of work to get convergence. The options that work are: -ksp_type bcgsl -pc_type hypre -pc_hypre_type boomeramg -ksp_monitor_true_residual -pc_hypre_boomeramg_relax_type_coarse symmetric-SOR/Jacobi -pc_hypre_boomeramg_grid_sweeps_coarse 4 -pc_hypre_boomeramg_coarsen_type PMIS The problem is that neither of ML or HYPRE seem to scale at all. ML output: 0 KSP preconditioned resid norm 1.538968715109e+06 true resid norm 2.621342052504e+03 ||r(i)||/||b|| 1.000000000000e+00 2 KSP preconditioned resid norm 1.263129058693e+05 true resid norm 1.096298699671e+04 ||r(i)||/||b|| 4.182203915830e+00 4 KSP preconditioned resid norm 2.277379585186e+04 true resid norm 2.999721659930e+03 ||r(i)||/||b|| 1.144345758717e+00 6 KSP preconditioned resid norm 4.766504457975e+03 true resid norm 6.733421603796e+02 ||r(i)||/||b|| 2.568692474667e-01 8 KSP preconditioned resid norm 2.139020425406e+03 true resid norm 1.360842101250e+02 ||r(i)||/||b|| 5.191394613876e-02 10 KSP preconditioned resid norm 6.621380459944e+02 true resid norm 1.522758800025e+02 ||r(i)||/||b|| 5.809080881188e-02 12 KSP preconditioned resid norm 2.973409610262e+01 true resid norm 1.161046206089e+01 ||r(i)||/||b|| 4.429205280479e-03 14 KSP preconditioned resid norm 2.532665482573e+00 true resid norm 2.557425874623e+00 ||r(i)||/||b|| 9.756170020543e-04 16 KSP preconditioned resid norm 2.375585214826e+00 true resid norm 2.441783841415e+00 ||r(i)||/||b|| 9.315014189327e-04 18 KSP preconditioned resid norm 1.436338060675e-02 true resid norm 1.305304828818e-02 ||r(i)||/||b|| 4.979528816437e-06 20 KSP preconditioned resid norm 4.088293864561e-03 true resid norm 9.841243465634e-04 ||r(i)||/||b|| 3.754276728683e-07 22 KSP preconditioned resid norm 6.140822977383e-04 true resid norm 1.682184150207e-04 ||r(i)||/||b|| 6.417263052718e-08 24 KSP preconditioned resid norm 2.685415483430e-05 true resid norm 1.065041542336e-05 ||r(i)||/||b|| 4.062962867890e-09 26 KSP preconditioned resid norm 1.620776166579e-06 true resid norm 9.563268703474e-07 ||r(i)||/||b|| 3.648233809982e-10 28 KSP preconditioned resid norm 2.823291105652e-07 true resid norm 7.705418741178e-08 ||r(i)||/||b|| 2.939493811507e-11 KSP Object:(pres_) 4 MPI processes type: bcgsl BCGSL: Ell = 2 BCGSL: Delta = 0 maximum iterations=5000 tolerances: relative=1e-12, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object:(pres_) 4 MPI processes type: ml MG: type is MULTIPLICATIVE, levels=5 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (pres_mg_coarse_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_coarse_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 8, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=4, cols=4 total: nonzeros=16, allocated nonzeros=16 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (pres_mg_levels_1_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_1_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=25, cols=25 total: nonzeros=303, allocated nonzeros=303 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 4 nodes, limit used is 5 Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (pres_mg_levels_2_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_2_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=423, cols=423 total: nonzeros=7437, allocated nonzeros=7437 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (pres_mg_levels_3_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_3_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=6617, cols=6617 total: nonzeros=88653, allocated nonzeros=88653 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 4 ------------------------------- KSP Object: (pres_mg_levels_4_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_4_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=46330, cols=46330 total: nonzeros=322437, allocated nonzeros=615417 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=46330, cols=46330 total: nonzeros=322437, allocated nonzeros=615417 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines John On Tue, Mar 20, 2012 at 1:33 PM, Mark F. Adams wrote: > John, > > I am getting poor results (diverging) from ML also: > > 0 KSP preconditioned resid norm 3.699832960909e+22 true resid norm > 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP preconditioned resid norm 5.706378365783e+11 true resid norm > 1.563902233018e+03 ||r(i)||/||b|| 1.193204539995e+00 > 4 KSP preconditioned resid norm 5.570291685152e+11 true resid norm > 1.564235542744e+03 ||r(i)||/||b|| 1.193458844050e+00 > 6 KSP preconditioned resid norm 5.202150407298e+10 true resid norm > 1.749929789082e+03 ||r(i)||/||b|| 1.335137277077e+00 > Linear solve converged due to CONVERGED_RTOL iterations 6 > > With GAMG I get: > > 0 KSP preconditioned resid norm 7.731260075891e+06 true resid norm > 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP preconditioned resid norm 2.856415184685e+05 true resid norm > 1.310410242531e+03 ||r(i)||/||b|| 9.997987199150e-01 > 4 KSP preconditioned resid norm 1.528467019258e+05 true resid norm > 1.284856538976e+03 ||r(i)||/||b|| 9.803021078816e-01 > 6 KSP preconditioned resid norm 1.451091957899e+05 true resid norm > 1.564309254168e+03 ||r(i)||/||b|| 1.193515083375e+00 > > > > 122 KSP preconditioned resid norm 2.486245341783e+01 true resid norm > 1.404397185367e+00 ||r(i)||/||b|| 1.071507580306e-03 > 124 KSP preconditioned resid norm 1.482316853621e+01 true resid norm > 4.488661881759e-01 ||r(i)||/||b|| 3.424697287811e-04 > 126 KSP preconditioned resid norm 1.481941150253e+01 true resid norm > 4.484480100832e-01 ||r(i)||/||b|| 3.421506730318e-04 > 128 KSP preconditioned resid norm 8.191887347033e+00 true resid norm > 6.678630367218e-01 ||r(i)||/||b|| 5.095569215816e-04 > > And HYPRE: > > 0 KSP preconditioned resid norm 3.774510769907e+04 true resid norm > 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP preconditioned resid norm 1.843165835831e+04 true resid norm > 8.502433792869e+02 ||r(i)||/||b|| 6.487069580482e-01 > 4 KSP preconditioned resid norm 1.573176624705e+04 true resid norm > 1.167264367302e+03 ||r(i)||/||b|| 8.905832558033e-01 > 6 KSP preconditioned resid norm 1.657958380765e+04 true resid norm > 8.684701624902e+02 ||r(i)||/||b|| 6.626133775216e-01 > 8 KSP preconditioned resid norm 2.190304455083e+04 true resid norm > 6.969893263600e+02 ||r(i)||/||b|| 5.317792960344e-01 > 10 KSP preconditioned resid norm 2.485714630000e+04 true resid norm > 6.642641436830e+02 ||r(i)||/||b|| 5.068110878446e-01 > > > > 62 KSP preconditioned resid norm 6.432516040957e+00 true resid norm > 2.124960171419e-01 ||r(i)||/||b|| 1.621272781837e-04 > 64 KSP preconditioned resid norm 5.731033176541e+00 true resid norm > 1.338816774003e-01 ||r(i)||/||b|| 1.021471943216e-04 > 66 KSP preconditioned resid norm 1.600285935522e-01 true resid norm > 3.352408932031e-03 ||r(i)||/||b|| 2.557774695353e-06 > > ML and GAMG should act similarly, but ML seems to have a problem (see the > preconditioned norm difference and its diverging). ML has a parameter: > > -pc_ml_Threshold [.0] > > I setting this to 0.05 (GAMG default) helps a bit but it still diverges. > > So it would be nice to figure out the difference between ML and GAMG, but > that is secondary for you as the both suck. > > HYPRE is a very different algorithm. It looks like the smoothing in GAMG > (and ML) may be the problem. If I turn smoothing off > (-pc_gamg_agg_nsmooths 0) and I get for GAMG: > > 0 KSP preconditioned resid norm 2.186148437534e+05 true resid norm > 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP preconditioned resid norm 2.916843959765e+04 true resid norm > 3.221533667508e+03 ||r(i)||/||b|| 2.457921292432e+00 > 4 KSP preconditioned resid norm 2.396374655925e+04 true resid norm > 1.834299897412e+03 ||r(i)||/||b|| 1.399508817812e+00 > 6 KSP preconditioned resid norm 2.509576275453e+04 true resid norm > 1.035475461174e+03 ||r(i)||/||b|| 7.900327752214e-01 > > > > 64 KSP preconditioned resid norm 1.973859758284e+01 true resid norm > 7.322674977169e+00 ||r(i)||/||b|| 5.586953482895e-03 > 66 KSP preconditioned resid norm 3.371598890438e+01 true resid norm > 7.152754930495e+00 ||r(i)||/||b|| 5.457310231004e-03 > 68 KSP preconditioned resid norm 4.687839294418e+00 true resid norm > 4.605726307025e-01 ||r(i)||/||b|| 3.514013487219e-04 > 70 KSP preconditioned resid norm 1.487545519493e+00 true resid norm > 1.558723789416e-01 ||r(i)||/||b|| 1.189253562571e-04 > 72 KSP preconditioned resid norm 5.317329808718e-01 true resid norm > 5.027178038177e-02 ||r(i)||/||b|| 3.835566911967e-05 > 74 KSP preconditioned resid norm 3.405339702462e-01 true resid norm > 1.897059263835e-02 ||r(i)||/||b|| 1.447392092969e-05 > > This is almost as good as HYPRE. > > An other thing to keep in mind is the cost of each iteration, not just > then number of iterations, You can > use -pc_hypre_boomeramg_print_statistics to get some data on this from > HYPRE: > > Average Convergence Factor = 0.537664 > > Complexity: grid = 1.780207 > operator = 2.624910 > cycle = 5.249670 > > And GAMG prints this with verbose set: > > [0]PCSetUp_GAMG 6 levels, grid compexity [sic] = 1.1316 > > I believe that the hypre "Complexity: grid" is the same as my "grid > complexity". So hypre actually looks more expensive at this point. > > I've worked on optimizing parameters for hypre with the hypre people and > here are a set of arguments that I've used: > > -pc_hypre_boomeramg_no_CF -pc_hypre_boomeramg_agg_nl 1 > -pc_hypre_boomeramg_coarsen_type HMIS -pc_hypre_boomeramg_interp_type ext+i > -pc_hypre_boomeramg_P_max 4 -pc_hypre_boomeramg_agg_num_paths 2 > > With these parmeters I get: > > Complexity: grid = 1.244140 > operator = 1.396722 > cycle = 2.793442 > > and: > > 0 KSP preconditioned resid norm 4.698624821403e+04 true resid norm > 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP preconditioned resid norm 2.207967626172e+04 true resid norm > 3.466160021150e+03 ||r(i)||/||b|| 2.644562931280e+00 > 4 KSP preconditioned resid norm 2.278468320876e+04 true resid norm > 1.246784122467e+03 ||r(i)||/||b|| 9.512541410282e-01 > > > > 56 KSP preconditioned resid norm 1.108460232262e+00 true resid norm > 8.276869475681e-02 ||r(i)||/||b|| 6.314971631105e-05 > 58 KSP preconditioned resid norm 3.617217454336e-01 true resid norm > 3.764556404754e-02 ||r(i)||/||b|| 2.872229285428e-05 > 60 KSP preconditioned resid norm 1.666532560770e-01 true resid norm > 5.149302513338e-03 ||r(i)||/||b|| 3.928743758404e-06 > Linear solve converged due to CONVERGED_RTOL iterations 60 > > So this actually converged faster with lower complexity. > > Anyway these result seem different from what you are getting, so I've > appended my options. This uses ex10 in the KSP tutorials to read in your > binary file. > > Mark > > #PETSc Option Table entries: > -f ./binaryoutput > -ksp_converged_reason > -ksp_diagonal_scale > -ksp_diagonal_scale_fix > -ksp_monitor_true_residual > -ksp_type bcgsl > -mg_coarse_ksp_type richardson > -mg_coarse_pc_sor_its 8 > -mg_coarse_pc_type sor > -mg_levels_ksp_type richardson > -mg_levels_pc_type sor > -options_left > -pc_gamg_agg_nsmooths 0 > -pc_gamg_coarse_eq_limit 10 > -pc_gamg_sym_graph > -pc_gamg_verbose 2 > -pc_hypre_boomeramg_P_max 4 > -pc_hypre_boomeramg_agg_nl 1 > -pc_hypre_boomeramg_agg_num_paths 2 > -pc_hypre_boomeramg_coarsen_type HMIS > -pc_hypre_boomeramg_interp_type ext+i > -pc_hypre_boomeramg_no_CF > -pc_ml_Threshold .01 > -pc_type gamg > -vecload_block_size 1 > #End of PETSc Option Table entries > There are 7 unused database options. They are: > Option left: name:-pc_hypre_boomeramg_P_max value: 4 > Option left: name:-pc_hypre_boomeramg_agg_nl value: 1 > Option left: name:-pc_hypre_boomeramg_agg_num_paths value: 2 > Option left: name:-pc_hypre_boomeramg_coarsen_type value: HMIS > Option left: name:-pc_hypre_boomeramg_interp_type value: ext+i > Option left: name:-pc_hypre_boomeramg_no_CF no value > Option left: name:-pc_ml_Threshold value: .01 > > > On Mar 20, 2012, at 10:19 AM, John Mousel wrote: > > Mark, > > Sorry for the late reply. I've been on travel and hadn't had a chance to > pick this back up. I've tried running with the suggested options: > > -ksp_type bcgsl -pc_type gamg -pc_gamg_coarse_eq_limit 10 > -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph -mg_coarse_ksp_type richardson > -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_diagonal_scale > -ksp_diagonal_scale_fix -ksp_monitor_true_residual -ksp_view > -pc_gamg_verbose 1 > > With these options, the convergence starts to hang (see attached > GAMG_kspview.txt). The hanging happens for both -mg_coarse_ksp_type > richardson and preonly. It was my understanding from previous emails that > using preonly made it so that only the preconditioner was run, which in > this case would be 8 sweeps of SOR. If I get rid of the > -pc_gamg_agg_nsmooths 1 (see GAMG_kspview_nosmooth.txt), the problem > converges, but again the convergence is slow. Without this option, both > Richardson and preonly converge in 172 iterations. > > Matt, I've checked and the problem does converge in the true residual > using GAMG, ML, HYPRE, and ILU preconditioned BiCG. I explicitly ensure > that a solution exists by projecting the rhs vector out of the nullity of > the transpose of operator. > > John > > > On Fri, Mar 16, 2012 at 2:04 PM, Mark F. Adams wrote: > >> John, did this get resolved? >> Mark >> >> On Mar 15, 2012, at 4:24 PM, John Mousel wrote: >> >> Mark, >> >> Running without the options you mentioned before leads to slightly worse >> performance (175 iterations). >> I have not been able to get run coarse grid solve to work with LU while >> running ML. It keeps experiencing a zero pivot, and all the combinations of >> shifting i've tried haven't lead me anywhere, hence the SOR on the course >> grid. Also, the ML manual suggests limiting the number of levels to 3 or 4 >> and performing a few sweeps of an iterative method as opposed to a direct >> solve. >> >> John >> >> On Thu, Mar 15, 2012 at 12:04 PM, Mark F. Adams wrote: >> >>> You also want: -pc_gamg_agg_nsmooths 1 >>> >>> You are running plain aggregation. If it is Poisson then smoothing is >>> good. >>> >>> Is this problem singular? Can you try running ML with these parameters >>> and see if its performance degrades? The ML implementation uses the PETSC >>> infrastructure and uses a very similar algorithm to GAMG-SA. We should be >>> able to get these two to match pretty well. >>> >>> Mark >>> >>> >>> On Mar 15, 2012, at 12:21 PM, John Mousel wrote: >>> >>> Mark, >>> >>> I ran with those options removed (see the run options listed below). >>> Things actually got slightly worse. Now it's up to 142 iterations. I have >>> attached the ksp_view output. >>> >>> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale >>> -ksp_diagonal_scale_fix -mg_levels_ksp_type richardson -mg_levels_pc_type >>> sor -pc_gamg_verbose 1 >>> >>> >>> John >>> >>> >>> On Thu, Mar 15, 2012 at 10:55 AM, Mark F. Adams >> > wrote: >>> >>>> John, can you run again with: -pc_gamg_verbose 1 >>>> >>>> And I would not use: -pc_mg_levels 4 -mg_coarse_ksp_type preonly >>>> -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 >>>> >>>> 1) I think -mg_coarse_ksp_type preonly and -mg_coarse_pc_sor_its 8 do >>>> not do what you think. I think this is the same as 1 iteration. I think >>>> you want 'richardson' not 'preonly'. >>>> >>>> 2) Why are you using sor as the coarse solver? If your problem is >>>> singular then you want to use as many levels as possible to get the coarse >>>> grid to be tiny. I'm pretty sure HYPRE ignores the coarse solver >>>> parameters. But ML uses them and it is converging well. >>>> >>>> 3) I would not specify the number of levels. GAMG, and I think the >>>> rest, have internal logic for stopping a the right level. If the coarse >>>> level is large and you use just 8 iterations of sor then convergence will >>>> suffer. >>>> >>>> Mark >>>> >>>> On Mar 15, 2012, at 11:13 AM, John Mousel wrote: >>>> >>>> Mark, >>>> >>>> The changes pulled through this morning. I've run it with the options >>>> >>>> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale >>>> -ksp_diagonal_scale_fix -pc_mg_levels 4 -mg_levels_ksp_type richardson >>>> -mg_levels_pc_type sor -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor >>>> -mg_coarse_pc_sor_its 8 >>>> >>>> and it converges in the true residual, but it's not converging as fast >>>> as anticpated. The matrix arises from a non-symmetric discretization of the >>>> Poisson equation. The solve takes GAMG 114 iterations, whereas ML takes 24 >>>> iterations, BoomerAMG takes 22 iterations, and -ksp_type bcgsl -pc_type >>>> bjacobi -sub_pc_type ilu -sub_pc_factor_levels 4 takes around 170. I've >>>> attached the -ksp_view results for ML,GAMG, and HYPRE. I've attempted to >>>> make all the options the same on all levels for ML and GAMG. >>>> >>>> Any thoughts? >>>> >>>> John >>>> >>>> >>>> On Wed, Mar 14, 2012 at 6:04 PM, Mark F. Adams >>> > wrote: >>>> >>>>> Humm, I see it with hg view (appended). >>>>> >>>>> Satish, my main repo looks hosed. I see this: >>>>> >>>>> ~/Codes/petsc-dev>hg update >>>>> abort: crosses branches (merge branches or use --clean to discard >>>>> changes) >>>>> ~/Codes/petsc-dev>hg merge >>>>> abort: branch 'default' has 3 heads - please merge with an explicit rev >>>>> (run 'hg heads .' to see heads) >>>>> ~/Codes/petsc-dev>hg heads >>>>> changeset: 22496:8e2a98268179 >>>>> tag: tip >>>>> user: Barry Smith >>>>> date: Wed Mar 14 16:42:25 2012 -0500 >>>>> files: src/vec/is/interface/f90-custom/zindexf90.c >>>>> src/vec/vec/interface/f90-custom/zvectorf90.c >>>>> description: >>>>> undoing manually changes I put in because Satish had a better fix >>>>> >>>>> >>>>> changeset: 22492:bda4df63072d >>>>> user: Mark F. Adams >>>>> date: Wed Mar 14 17:39:52 2012 -0400 >>>>> files: src/ksp/pc/impls/gamg/tools.c >>>>> description: >>>>> fix for unsymmetric matrices. >>>>> >>>>> >>>>> changeset: 22469:b063baf366e4 >>>>> user: Mark F. Adams >>>>> date: Wed Mar 14 14:22:28 2012 -0400 >>>>> files: src/ksp/pc/impls/gamg/tools.c >>>>> description: >>>>> added fix for preallocation for unsymetric matrices. >>>>> >>>>> Mark >>>>> >>>>> my 'hg view' on my merge repo: >>>>> >>>>> Revision: 22492 >>>>> Branch: default >>>>> Author: Mark F. Adams 2012-03-14 17:39:52 >>>>> Committer: Mark F. Adams 2012-03-14 >>>>> 17:39:52 >>>>> Tags: tip >>>>> Parent: 22491:451bbbd291c2 (Small fixes to the BT linesearch) >>>>> >>>>> fix for unsymmetric matrices. >>>>> >>>>> >>>>> ------------------------ src/ksp/pc/impls/gamg/tools.c >>>>> ------------------------ >>>>> @@ -103,7 +103,7 @@ >>>>> PetscErrorCode ierr; >>>>> PetscInt Istart,Iend,Ii,jj,ncols,nnz0,nnz1, NN, MM, nloc; >>>>> PetscMPIInt mype, npe; >>>>> - Mat Gmat = *a_Gmat, tGmat; >>>>> + Mat Gmat = *a_Gmat, tGmat, matTrans; >>>>> MPI_Comm wcomm = ((PetscObject)Gmat)->comm; >>>>> const PetscScalar *vals; >>>>> const PetscInt *idx; >>>>> @@ -127,6 +127,10 @@ >>>>> ierr = MatDiagonalScale( Gmat, diag, diag ); CHKERRQ(ierr); >>>>> ierr = VecDestroy( &diag ); CHKERRQ(ierr); >>>>> >>>>> + if( symm ) { >>>>> + ierr = MatTranspose( Gmat, MAT_INITIAL_MATRIX, &matTrans ); >>>>> CHKERRQ(ierr); >>>>> + } >>>>> + >>>>> /* filter - dup zeros out matrix */ >>>>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &d_nnz ); CHKERRQ(ierr); >>>>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &o_nnz ); CHKERRQ(ierr); >>>>> @@ -135,6 +139,12 @@ >>>>> d_nnz[jj] = ncols; >>>>> o_nnz[jj] = ncols; >>>>> ierr = MatRestoreRow(Gmat,Ii,&ncols,PETSC_NULL,PETSC_NULL); >>>>> CHKERRQ(ierr); >>>>> + if( symm ) { >>>>> + ierr = MatGetRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); >>>>> CHKERRQ(ierr); >>>>> + d_nnz[jj] += ncols; >>>>> + o_nnz[jj] += ncols; >>>>> + ierr = MatRestoreRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); >>>>> CHKERRQ(ierr); >>>>> + } >>>>> if( d_nnz[jj] > nloc ) d_nnz[jj] = nloc; >>>>> if( o_nnz[jj] > (MM-nloc) ) o_nnz[jj] = MM - nloc; >>>>> } >>>>> @@ -142,6 +152,9 @@ >>>>> CHKERRQ(ierr); >>>>> ierr = PetscFree( d_nnz ); CHKERRQ(ierr); >>>>> ierr = PetscFree( o_nnz ); CHKERRQ(ierr); >>>>> + if( symm ) { >>>>> + ierr = MatDestroy( &matTrans ); CHKERRQ(ierr); >>>>> + } >>>>> >>>>> >>>>> >>>>> >>>>> On Mar 14, 2012, at 5:53 PM, John Mousel wrote: >>>>> >>>>> Mark, >>>>> >>>>> No change. Can you give me the location that you patched so I can >>>>> check to make sure it pulled? >>>>> I don't see it on the petsc-dev change log. >>>>> >>>>> John >>>>> >>>>> On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams < >>>>> mark.adams at columbia.edu> wrote: >>>>> >>>>>> John, I've committed these changes, give a try. >>>>>> >>>>>> Mark >>>>>> >>>>>> On Mar 14, 2012, at 3:46 PM, Satish Balay wrote: >>>>>> >>>>>> > This is the usual merge [with uncommited changes] issue. >>>>>> > >>>>>> > You could use 'hg shelf' extension to shelve your local changes and >>>>>> > then do a merge [as Sean would suggest] - or do the merge in a >>>>>> > separate/clean clone [I normally do this..] >>>>>> > >>>>>> > i.e >>>>>> > cd ~/Codes >>>>>> > hg clone petsc-dev petsc-dev-merge >>>>>> > cd petsc-dev-merge >>>>>> > hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just >>>>>> to be sure, look for latest chagnes before merge.. >>>>>> > hg merge >>>>>> > hg commit >>>>>> > hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev >>>>>> > >>>>>> > [now update your petsc-dev to latest] >>>>>> > cd ~/Codes/petsc-dev >>>>>> > hg pull >>>>>> > hg update >>>>>> > >>>>>> > Satish >>>>>> > >>>>>> > On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>>>> > >>>>>> >> Great, that seems to work. >>>>>> >> >>>>>> >> I did a 'hg commit tools.c' >>>>>> >> >>>>>> >> and I want to push this file only. I guess its the only thing in >>>>>> the change set so 'hg push' should be fine. But I see this: >>>>>> >> >>>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update >>>>>> >> abort: crosses branches (merge branches or use --clean to discard >>>>>> changes) >>>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge >>>>>> >> abort: outstanding uncommitted changes (use 'hg status' to list >>>>>> changes) >>>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status >>>>>> >> M include/petscmat.h >>>>>> >> M include/private/matimpl.h >>>>>> >> M src/ksp/pc/impls/gamg/agg.c >>>>>> >> M src/ksp/pc/impls/gamg/gamg.c >>>>>> >> M src/ksp/pc/impls/gamg/gamg.h >>>>>> >> M src/ksp/pc/impls/gamg/geo.c >>>>>> >> M src/mat/coarsen/coarsen.c >>>>>> >> M src/mat/coarsen/impls/hem/hem.c >>>>>> >> M src/mat/coarsen/impls/mis/mis.c >>>>>> >> >>>>>> >> Am I ready to do a push? >>>>>> >> >>>>>> >> Thanks, >>>>>> >> Mark >>>>>> >> >>>>>> >> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: >>>>>> >> >>>>>> >>> If commit is the last hg operation that you've done - then 'hg >>>>>> rollback' would undo this commit. >>>>>> >>> >>>>>> >>> Satish >>>>>> >>> >>>>>> >>> On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>>>> >>> >>>>>> >>>> Damn, I'm not preallocating the graph perfectly for unsymmetric >>>>>> matrices and PETSc now dies on this. >>>>>> >>>> >>>>>> >>>> I have a fix but I committed it with other changes that I do not >>>>>> want to commit. The changes are all in one file so I should be able to >>>>>> just commit this file. >>>>>> >>>> >>>>>> >>>> Anyone know how to delete a commit? >>>>>> >>>> >>>>>> >>>> I've tried: >>>>>> >>>> >>>>>> >>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip >>>>>> 22487:26ffb9eef17f >>>>>> >>>> hg: unknown command 'strip' >>>>>> >>>> 'strip' is provided by the following extension: >>>>>> >>>> >>>>>> >>>> mq manage a stack of patches >>>>>> >>>> >>>>>> >>>> use "hg help extensions" for information on enabling extensions >>>>>> >>>> >>>>>> >>>> But have not figured out how to load extensions. >>>>>> >>>> >>>>>> >>>> Mark >>>>>> >>>> >>>>>> >>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: >>>>>> >>>> >>>>>> >>>>> Mark, >>>>>> >>>>> >>>>>> >>>>> I have a non-symmetric matrix. I am running with the following >>>>>> options. >>>>>> >>>>> >>>>>> >>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual >>>>>> >>>>> >>>>>> >>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new >>>>>> malloc error: >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> 0]PETSC ERROR: --------------------- Error Message >>>>>> ------------------------------------ >>>>>> >>>>> [0]PETSC ERROR: Argument out of range! >>>>>> >>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! >>>>>> >>>>> [0]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> >>>>> [0]PETSC ERROR: Petsc Development HG revision: >>>>>> 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 >>>>>> -0500 >>>>>> >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>> >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble >>>>>> shooting. >>>>>> >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>>> >>>>> [0]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> >>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named >>>>>> wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 >>>>>> >>>>> [0]PETSC ERROR: Libraries linked from >>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib >>>>>> >>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 >>>>>> >>>>> [0]PETSC ERROR: Configure options --download-blacs=1 >>>>>> --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 >>>>>> --download-parmetis=1 --download-scalapack=1 >>>>>> --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc >>>>>> --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort >>>>>> PETSC_ARCH=linux-debug >>>>>> >>>>> [0]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> >>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in >>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >>>>>> >>>>> [0]PETSC ERROR: MatSetValues() line 1141 in >>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c >>>>>> >>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in >>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c >>>>>> >>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in >>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c >>>>>> >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in >>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>>>>> >>>>> [0]PETSC ERROR: PCSetUp() line 832 in >>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c >>>>>> >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in >>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>>> >>>>> [0]PETSC ERROR: KSPSolve() line 385 in >>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> John >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams < >>>>>> mark.adams at columbia.edu> wrote: >>>>>> >>>>> >>>>>> >>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: >>>>>> >>>>> >>>>>> >>>>>> Mark, >>>>>> >>>>>> >>>>>> >>>>>> The matrix is asymmetric. Does this require the setting of an >>>>>> option? >>>>>> >>>>> >>>>>> >>>>> Yes: -pc_gamg_sym_graph >>>>>> >>>>> >>>>>> >>>>> Mark >>>>>> >>>>> >>>>>> >>>>>> I pulled petsc-dev this morning, so I should have (at least >>>>>> close to) the latest code. >>>>>> >>>>>> >>>>>> >>>>>> John >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams < >>>>>> mark.adams at columbia.edu> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >>>>>> >>>>>> >>>>>> >>>>>>> I'm getting the following error when using GAMG. >>>>>> >>>>>>> >>>>>> >>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: >>>>>> Assertion `sgid==-1' failed. >>>>>> >>>>>> >>>>>> >>>>>> Is it possible that your matrix is structurally asymmetric? >>>>>> >>>>>> >>>>>> >>>>>> This code is evolving fast and so you will need to move to the >>>>>> dev version if you are not already using it. (I think I fixed a bug that >>>>>> hit this assert). >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>> >>>>>>> When I try to alter the type of aggregation at the command >>>>>> line using -pc_gamg_type pa, I'm getting >>>>>> >>>>>>> >>>>>> >>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error >>>>>> Message ------------------------------------ >>>>>> >>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or >>>>>> missing external package needed for type: >>>>>> >>>>>>> see >>>>>> http://www.mcs.anl.gov/petsc/documentation/installation.html#external >>>>>> ! >>>>>> >>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given! >>>>>> >>>>>>> >>>>>> >>>>>>> Has there been a change in the aggregation options? I just >>>>>> pulled petsc-dev this morning. >>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg >>>>>> for now. >>>>>> >>>>>> >>>>>> >>>>>> Mark >>>>>> >>>>>> >>>>>> >>>>>>> John >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>> >>>>>> >>> >>>>>> >> >>>>>> >> >>>>>> > >>>>>> > >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> >>> >>> >>> >>> >> >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Mar 20 14:05:42 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 20 Mar 2012 14:05:42 -0500 Subject: [petsc-users] -ksp_diagonal_scale question In-Reply-To: References: Message-ID: Max, In petsc-dev it will scale both the mat and pmat by the diagonal of the pmat if they are different so it appears the documentation is simply out of date. Note that -ksp_monitor and the various convergence tests are run in the scaled system so there is no certainty that convergence in this scaled system results in the same amount of convergence as in the original system, though for reasonably conditioned systems it should be fine. In other words if the condition number of your systems is 10^14 this will not help you get a better convergence, you had better use quad precision available with that latest gcc compilers and petsc-dev Barry On Mar 19, 2012, at 5:06 PM, Max Rudolph wrote: > I was looking at the documentation for KSPSetDiagonalScale and it > contains this comment: > > "This routine is only used if the matrix and preconditioner matrix are > the same thing." > > However, I have found that even if the matrix and preconditioner are > different, using the -ksp_diagonal_scale command line argument changes > the convergence behavior of the solvers. Is the man page correct? > Also, can you point me towards documentation of the type of scaling > that is applied? > > Thanks for your help, > Max Rudolph From mark.adams at columbia.edu Tue Mar 20 14:21:44 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Tue, 20 Mar 2012 15:21:44 -0400 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> <9CB8B9D0-BC04-4471-BCFC-BEE61031A123@columbia.edu> Message-ID: <8909A66D-5CB4-479B-ADBC-34A1133D64F4@columbia.edu> John, I had dome diagonal scaling stuff in my input which seemed to mess things up. I don't understand that. With your hypre parameters I get Complexity: grid = 1.408828 operator = 1.638900 cycle = 3.277856 0 KSP preconditioned resid norm 2.246209947341e+06 true resid norm 2.621342052504e+03 ||r(i)||/||b|| 1.000000000000e+00 2 KSP preconditioned resid norm 6.591054866442e+04 true resid norm 5.518411654910e+03 ||r(i)||/||b|| 2.105185643224e+00 4 KSP preconditioned resid norm 2.721184454964e+03 true resid norm 1.937153214559e+03 ||r(i)||/||b|| 7.389929188022e-01 6 KSP preconditioned resid norm 2.942012838854e+02 true resid norm 5.614763956317e+01 ||r(i)||/||b|| 2.141942502679e-02 8 KSP preconditioned resid norm 2.143421596353e+01 true resid norm 5.306843482279e+00 ||r(i)||/||b|| 2.024475774617e-03 10 KSP preconditioned resid norm 3.689048280659e+00 true resid norm 2.482945300243e-01 ||r(i)||/||b|| 9.472038560826e-05 Linear solve converged due to CONVERGED_RTOL iterations 10 with ML I get 18 iterations but if I add -pc_ml_Threshold .01 I get it to 12: -@${MPIEXEC} -n 1 ./ex10 -f ./binaryoutput -pc_type ml -ksp_type bcgsl -pc_gamg_coarse_eq_limit 10 -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_monitor_true_residual -pc_gamg_verbose 2 -ksp_converged_reason -options_left -mg_levels_ksp_type richardson -mg_levels_pc_type sor -pc_ml_maxNlevels 5 -pc_ml_Threshold .01 0 KSP preconditioned resid norm 1.987800354481e+06 true resid norm 2.621342052504e+03 ||r(i)||/||b|| 1.000000000000e+00 2 KSP preconditioned resid norm 4.845840795806e+04 true resid norm 9.664923970856e+03 ||r(i)||/||b|| 3.687013666005e+00 4 KSP preconditioned resid norm 4.086337251141e+03 true resid norm 1.111442892542e+03 ||r(i)||/||b|| 4.239976585582e-01 6 KSP preconditioned resid norm 1.496117919395e+03 true resid norm 4.243682354730e+02 ||r(i)||/||b|| 1.618896835946e-01 8 KSP preconditioned resid norm 1.019912311314e+02 true resid norm 6.165476121107e+01 ||r(i)||/||b|| 2.352030371320e-02 10 KSP preconditioned resid norm 1.282179114927e+01 true resid norm 4.434755525096e+00 ||r(i)||/||b|| 1.691788189512e-03 12 KSP preconditioned resid norm 2.801790417375e+00 true resid norm 4.876299030996e-01 ||r(i)||/||b|| 1.860229963632e-04 Linear solve converged due to CONVERGED_RTOL iterations 12 and gamg: -@${MPIEXEC} -n 1 ./ex10 -f ./binaryoutput -pc_type gamg -ksp_type bcgsl -pc_gamg_coarse_eq_limit 10 -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_monitor_true_residual -pc_gamg_verbose 2 -ksp_converged_reason -options_left -mg_levels_ksp_type richardson -mg_levels_pc_type sor [0]PCSetUp_GAMG 5 levels, grid compexity = 1.2916 0 KSP preconditioned resid norm 6.288750978813e+06 true resid norm 2.621342052504e+03 ||r(i)||/||b|| 1.000000000000e+00 2 KSP preconditioned resid norm 3.009668424006e+04 true resid norm 4.394363256786e+02 ||r(i)||/||b|| 1.676379186222e-01 4 KSP preconditioned resid norm 2.079756553216e+01 true resid norm 5.094584609440e+00 ||r(i)||/||b|| 1.943502414946e-03 6 KSP preconditioned resid norm 4.323447593442e+00 true resid norm 3.146656048880e-01 ||r(i)||/||b|| 1.200398874261e-04 Linear solve converged due to CONVERGED_RTOL iterations 6 So this looks pretty different from what you are getting. Is your code hardwiring anything? Can you reproduce my results with ksp ex10.c? Actually, I just realized that I am using petsc-dev. What version of PETSc are you using? Also, here is the makefile that I use to run this jobs: ALL: runex10 include ${PETSC_DIR}/conf/variables include ${PETSC_DIR}/conf/rules runex10: -@${MPIEXEC} -n 1 ./ex10 -f ./binaryoutput -pc_type gamg -ksp_type bcgsl -pc_gamg_coarse_eq_limit 10 -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_monitor_true_residual -pc_gamg_verbose 2 -ksp_converged_reason -options_left -mg_levels_ksp_type richardson -mg_levels_pc_type sor -pc_ml_maxNlevels 5 -pc_ml_Threshold .01 -pc_hypre_boomeramg_relax_type_coarse symmetric-SOR/Jacobi -pc_hypre_boomeramg_grid_sweeps_coarse 4 -pc_hypre_boomeramg_coarsen_type PMIS You just need to run 'make ex10' and then 'make -f this-file'. Mark On Mar 20, 2012, at 2:45 PM, John Mousel wrote: > Mark, > > I run ML with the following options. > > -ksp_type bcgsl -pc_type ml -pc_ml_maxNlevels 5 -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_monitor -ksp_view > > Note the lack of scaling. For some reason scaling seems to mess with ML. > As you can see below, ML converges very nicely. > > With regards to HYPRE, this one took a bit of work to get convergence. The options that work are: > > -ksp_type bcgsl -pc_type hypre -pc_hypre_type boomeramg -ksp_monitor_true_residual -pc_hypre_boomeramg_relax_type_coarse symmetric-SOR/Jacobi -pc_hypre_boomeramg_grid_sweeps_coarse 4 -pc_hypre_boomeramg_coarsen_type PMIS > > The problem is that neither of ML or HYPRE seem to scale at all. > > ML output: > 0 KSP preconditioned resid norm 1.538968715109e+06 true resid norm 2.621342052504e+03 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP preconditioned resid norm 1.263129058693e+05 true resid norm 1.096298699671e+04 ||r(i)||/||b|| 4.182203915830e+00 > 4 KSP preconditioned resid norm 2.277379585186e+04 true resid norm 2.999721659930e+03 ||r(i)||/||b|| 1.144345758717e+00 > 6 KSP preconditioned resid norm 4.766504457975e+03 true resid norm 6.733421603796e+02 ||r(i)||/||b|| 2.568692474667e-01 > 8 KSP preconditioned resid norm 2.139020425406e+03 true resid norm 1.360842101250e+02 ||r(i)||/||b|| 5.191394613876e-02 > 10 KSP preconditioned resid norm 6.621380459944e+02 true resid norm 1.522758800025e+02 ||r(i)||/||b|| 5.809080881188e-02 > 12 KSP preconditioned resid norm 2.973409610262e+01 true resid norm 1.161046206089e+01 ||r(i)||/||b|| 4.429205280479e-03 > 14 KSP preconditioned resid norm 2.532665482573e+00 true resid norm 2.557425874623e+00 ||r(i)||/||b|| 9.756170020543e-04 > 16 KSP preconditioned resid norm 2.375585214826e+00 true resid norm 2.441783841415e+00 ||r(i)||/||b|| 9.315014189327e-04 > 18 KSP preconditioned resid norm 1.436338060675e-02 true resid norm 1.305304828818e-02 ||r(i)||/||b|| 4.979528816437e-06 > 20 KSP preconditioned resid norm 4.088293864561e-03 true resid norm 9.841243465634e-04 ||r(i)||/||b|| 3.754276728683e-07 > 22 KSP preconditioned resid norm 6.140822977383e-04 true resid norm 1.682184150207e-04 ||r(i)||/||b|| 6.417263052718e-08 > 24 KSP preconditioned resid norm 2.685415483430e-05 true resid norm 1.065041542336e-05 ||r(i)||/||b|| 4.062962867890e-09 > 26 KSP preconditioned resid norm 1.620776166579e-06 true resid norm 9.563268703474e-07 ||r(i)||/||b|| 3.648233809982e-10 > 28 KSP preconditioned resid norm 2.823291105652e-07 true resid norm 7.705418741178e-08 ||r(i)||/||b|| 2.939493811507e-11 > KSP Object:(pres_) 4 MPI processes > type: bcgsl > BCGSL: Ell = 2 > BCGSL: Delta = 0 > maximum iterations=5000 > tolerances: relative=1e-12, absolute=1e-50, divergence=10000 > left preconditioning > has attached null space > using nonzero initial guess > using PRECONDITIONED norm type for convergence test > PC Object:(pres_) 4 MPI processes > type: ml > MG: type is MULTIPLICATIVE, levels=5 cycles=v > Cycles per PCApply=1 > Using Galerkin computed coarse grid matrices > Coarse grid solver -- level ------------------------------- > KSP Object: (pres_mg_coarse_) 4 MPI processes > type: richardson > Richardson: damping factor=1 > maximum iterations=1, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: (pres_mg_coarse_) 4 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 8, local iterations = 1, omega = 1 > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=4, cols=4 > total: nonzeros=16, allocated nonzeros=16 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Down solver (pre-smoother) on level 1 ------------------------------- > KSP Object: (pres_mg_levels_1_) 4 MPI processes > type: richardson > Richardson: damping factor=1 > maximum iterations=1 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using nonzero initial guess > using PRECONDITIONED norm type for convergence test > PC Object: (pres_mg_levels_1_) 4 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=25, cols=25 > total: nonzeros=303, allocated nonzeros=303 > total number of mallocs used during MatSetValues calls =0 > using I-node (on process 0) routines: found 4 nodes, limit used is 5 > Up solver (post-smoother) same as down solver (pre-smoother) > Down solver (pre-smoother) on level 2 ------------------------------- > KSP Object: (pres_mg_levels_2_) 4 MPI processes > type: richardson > Richardson: damping factor=1 > maximum iterations=1 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using nonzero initial guess > using PRECONDITIONED norm type for convergence test > PC Object: (pres_mg_levels_2_) 4 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=423, cols=423 > total: nonzeros=7437, allocated nonzeros=7437 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Up solver (post-smoother) same as down solver (pre-smoother) > Down solver (pre-smoother) on level 3 ------------------------------- > KSP Object: (pres_mg_levels_3_) 4 MPI processes > type: richardson > Richardson: damping factor=1 > maximum iterations=1 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using nonzero initial guess > using PRECONDITIONED norm type for convergence test > PC Object: (pres_mg_levels_3_) 4 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=6617, cols=6617 > total: nonzeros=88653, allocated nonzeros=88653 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Up solver (post-smoother) same as down solver (pre-smoother) > Down solver (pre-smoother) on level 4 ------------------------------- > KSP Object: (pres_mg_levels_4_) 4 MPI processes > type: richardson > Richardson: damping factor=1 > maximum iterations=1 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > has attached null space > using nonzero initial guess > using PRECONDITIONED norm type for convergence test > PC Object: (pres_mg_levels_4_) 4 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=46330, cols=46330 > total: nonzeros=322437, allocated nonzeros=615417 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Up solver (post-smoother) same as down solver (pre-smoother) > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=46330, cols=46330 > total: nonzeros=322437, allocated nonzeros=615417 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > > > John > > > > On Tue, Mar 20, 2012 at 1:33 PM, Mark F. Adams wrote: > John, > > I am getting poor results (diverging) from ML also: > > 0 KSP preconditioned resid norm 3.699832960909e+22 true resid norm 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP preconditioned resid norm 5.706378365783e+11 true resid norm 1.563902233018e+03 ||r(i)||/||b|| 1.193204539995e+00 > 4 KSP preconditioned resid norm 5.570291685152e+11 true resid norm 1.564235542744e+03 ||r(i)||/||b|| 1.193458844050e+00 > 6 KSP preconditioned resid norm 5.202150407298e+10 true resid norm 1.749929789082e+03 ||r(i)||/||b|| 1.335137277077e+00 > Linear solve converged due to CONVERGED_RTOL iterations 6 > > With GAMG I get: > > 0 KSP preconditioned resid norm 7.731260075891e+06 true resid norm 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP preconditioned resid norm 2.856415184685e+05 true resid norm 1.310410242531e+03 ||r(i)||/||b|| 9.997987199150e-01 > 4 KSP preconditioned resid norm 1.528467019258e+05 true resid norm 1.284856538976e+03 ||r(i)||/||b|| 9.803021078816e-01 > 6 KSP preconditioned resid norm 1.451091957899e+05 true resid norm 1.564309254168e+03 ||r(i)||/||b|| 1.193515083375e+00 > > > > 122 KSP preconditioned resid norm 2.486245341783e+01 true resid norm 1.404397185367e+00 ||r(i)||/||b|| 1.071507580306e-03 > 124 KSP preconditioned resid norm 1.482316853621e+01 true resid norm 4.488661881759e-01 ||r(i)||/||b|| 3.424697287811e-04 > 126 KSP preconditioned resid norm 1.481941150253e+01 true resid norm 4.484480100832e-01 ||r(i)||/||b|| 3.421506730318e-04 > 128 KSP preconditioned resid norm 8.191887347033e+00 true resid norm 6.678630367218e-01 ||r(i)||/||b|| 5.095569215816e-04 > > And HYPRE: > > 0 KSP preconditioned resid norm 3.774510769907e+04 true resid norm 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP preconditioned resid norm 1.843165835831e+04 true resid norm 8.502433792869e+02 ||r(i)||/||b|| 6.487069580482e-01 > 4 KSP preconditioned resid norm 1.573176624705e+04 true resid norm 1.167264367302e+03 ||r(i)||/||b|| 8.905832558033e-01 > 6 KSP preconditioned resid norm 1.657958380765e+04 true resid norm 8.684701624902e+02 ||r(i)||/||b|| 6.626133775216e-01 > 8 KSP preconditioned resid norm 2.190304455083e+04 true resid norm 6.969893263600e+02 ||r(i)||/||b|| 5.317792960344e-01 > 10 KSP preconditioned resid norm 2.485714630000e+04 true resid norm 6.642641436830e+02 ||r(i)||/||b|| 5.068110878446e-01 > > > > 62 KSP preconditioned resid norm 6.432516040957e+00 true resid norm 2.124960171419e-01 ||r(i)||/||b|| 1.621272781837e-04 > 64 KSP preconditioned resid norm 5.731033176541e+00 true resid norm 1.338816774003e-01 ||r(i)||/||b|| 1.021471943216e-04 > 66 KSP preconditioned resid norm 1.600285935522e-01 true resid norm 3.352408932031e-03 ||r(i)||/||b|| 2.557774695353e-06 > > ML and GAMG should act similarly, but ML seems to have a problem (see the preconditioned norm difference and its diverging). ML has a parameter: > > -pc_ml_Threshold [.0] > > I setting this to 0.05 (GAMG default) helps a bit but it still diverges. > > So it would be nice to figure out the difference between ML and GAMG, but that is secondary for you as the both suck. > > HYPRE is a very different algorithm. It looks like the smoothing in GAMG (and ML) may be the problem. If I turn smoothing off (-pc_gamg_agg_nsmooths 0) and I get for GAMG: > > 0 KSP preconditioned resid norm 2.186148437534e+05 true resid norm 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP preconditioned resid norm 2.916843959765e+04 true resid norm 3.221533667508e+03 ||r(i)||/||b|| 2.457921292432e+00 > 4 KSP preconditioned resid norm 2.396374655925e+04 true resid norm 1.834299897412e+03 ||r(i)||/||b|| 1.399508817812e+00 > 6 KSP preconditioned resid norm 2.509576275453e+04 true resid norm 1.035475461174e+03 ||r(i)||/||b|| 7.900327752214e-01 > > > > 64 KSP preconditioned resid norm 1.973859758284e+01 true resid norm 7.322674977169e+00 ||r(i)||/||b|| 5.586953482895e-03 > 66 KSP preconditioned resid norm 3.371598890438e+01 true resid norm 7.152754930495e+00 ||r(i)||/||b|| 5.457310231004e-03 > 68 KSP preconditioned resid norm 4.687839294418e+00 true resid norm 4.605726307025e-01 ||r(i)||/||b|| 3.514013487219e-04 > 70 KSP preconditioned resid norm 1.487545519493e+00 true resid norm 1.558723789416e-01 ||r(i)||/||b|| 1.189253562571e-04 > 72 KSP preconditioned resid norm 5.317329808718e-01 true resid norm 5.027178038177e-02 ||r(i)||/||b|| 3.835566911967e-05 > 74 KSP preconditioned resid norm 3.405339702462e-01 true resid norm 1.897059263835e-02 ||r(i)||/||b|| 1.447392092969e-05 > > This is almost as good as HYPRE. > > An other thing to keep in mind is the cost of each iteration, not just then number of iterations, You can use -pc_hypre_boomeramg_print_statistics to get some data on this from HYPRE: > > Average Convergence Factor = 0.537664 > > Complexity: grid = 1.780207 > operator = 2.624910 > cycle = 5.249670 > > And GAMG prints this with verbose set: > > [0]PCSetUp_GAMG 6 levels, grid compexity [sic] = 1.1316 > > I believe that the hypre "Complexity: grid" is the same as my "grid complexity". So hypre actually looks more expensive at this point. > > I've worked on optimizing parameters for hypre with the hypre people and here are a set of arguments that I've used: > > -pc_hypre_boomeramg_no_CF -pc_hypre_boomeramg_agg_nl 1 -pc_hypre_boomeramg_coarsen_type HMIS -pc_hypre_boomeramg_interp_type ext+i -pc_hypre_boomeramg_P_max 4 -pc_hypre_boomeramg_agg_num_paths 2 > > With these parmeters I get: > > Complexity: grid = 1.244140 > operator = 1.396722 > cycle = 2.793442 > > and: > > 0 KSP preconditioned resid norm 4.698624821403e+04 true resid norm 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP preconditioned resid norm 2.207967626172e+04 true resid norm 3.466160021150e+03 ||r(i)||/||b|| 2.644562931280e+00 > 4 KSP preconditioned resid norm 2.278468320876e+04 true resid norm 1.246784122467e+03 ||r(i)||/||b|| 9.512541410282e-01 > > > > 56 KSP preconditioned resid norm 1.108460232262e+00 true resid norm 8.276869475681e-02 ||r(i)||/||b|| 6.314971631105e-05 > 58 KSP preconditioned resid norm 3.617217454336e-01 true resid norm 3.764556404754e-02 ||r(i)||/||b|| 2.872229285428e-05 > 60 KSP preconditioned resid norm 1.666532560770e-01 true resid norm 5.149302513338e-03 ||r(i)||/||b|| 3.928743758404e-06 > Linear solve converged due to CONVERGED_RTOL iterations 60 > > So this actually converged faster with lower complexity. > > Anyway these result seem different from what you are getting, so I've appended my options. This uses ex10 in the KSP tutorials to read in your binary file. > > Mark > > #PETSc Option Table entries: > -f ./binaryoutput > -ksp_converged_reason > -ksp_diagonal_scale > -ksp_diagonal_scale_fix > -ksp_monitor_true_residual > -ksp_type bcgsl > -mg_coarse_ksp_type richardson > -mg_coarse_pc_sor_its 8 > -mg_coarse_pc_type sor > -mg_levels_ksp_type richardson > -mg_levels_pc_type sor > -options_left > -pc_gamg_agg_nsmooths 0 > -pc_gamg_coarse_eq_limit 10 > -pc_gamg_sym_graph > -pc_gamg_verbose 2 > -pc_hypre_boomeramg_P_max 4 > -pc_hypre_boomeramg_agg_nl 1 > -pc_hypre_boomeramg_agg_num_paths 2 > -pc_hypre_boomeramg_coarsen_type HMIS > -pc_hypre_boomeramg_interp_type ext+i > -pc_hypre_boomeramg_no_CF > -pc_ml_Threshold .01 > -pc_type gamg > -vecload_block_size 1 > #End of PETSc Option Table entries > There are 7 unused database options. They are: > Option left: name:-pc_hypre_boomeramg_P_max value: 4 > Option left: name:-pc_hypre_boomeramg_agg_nl value: 1 > Option left: name:-pc_hypre_boomeramg_agg_num_paths value: 2 > Option left: name:-pc_hypre_boomeramg_coarsen_type value: HMIS > Option left: name:-pc_hypre_boomeramg_interp_type value: ext+i > Option left: name:-pc_hypre_boomeramg_no_CF no value > Option left: name:-pc_ml_Threshold value: .01 > > > On Mar 20, 2012, at 10:19 AM, John Mousel wrote: > >> Mark, >> >> Sorry for the late reply. I've been on travel and hadn't had a chance to pick this back up. I've tried running with the suggested options: >> >> -ksp_type bcgsl -pc_type gamg -pc_gamg_coarse_eq_limit 10 -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_diagonal_scale -ksp_diagonal_scale_fix -ksp_monitor_true_residual -ksp_view -pc_gamg_verbose 1 >> >> With these options, the convergence starts to hang (see attached GAMG_kspview.txt). The hanging happens for both -mg_coarse_ksp_type richardson and preonly. It was my understanding from previous emails that using preonly made it so that only the preconditioner was run, which in this case would be 8 sweeps of SOR. If I get rid of the -pc_gamg_agg_nsmooths 1 (see GAMG_kspview_nosmooth.txt), the problem converges, but again the convergence is slow. Without this option, both Richardson and preonly converge in 172 iterations. >> >> Matt, I've checked and the problem does converge in the true residual using GAMG, ML, HYPRE, and ILU preconditioned BiCG. I explicitly ensure that a solution exists by projecting the rhs vector out of the nullity of the transpose of operator. >> >> John >> >> >> On Fri, Mar 16, 2012 at 2:04 PM, Mark F. Adams wrote: >> John, did this get resolved? >> Mark >> >> On Mar 15, 2012, at 4:24 PM, John Mousel wrote: >> >>> Mark, >>> >>> Running without the options you mentioned before leads to slightly worse performance (175 iterations). >>> I have not been able to get run coarse grid solve to work with LU while running ML. It keeps experiencing a zero pivot, and all the combinations of shifting i've tried haven't lead me anywhere, hence the SOR on the course grid. Also, the ML manual suggests limiting the number of levels to 3 or 4 and performing a few sweeps of an iterative method as opposed to a direct solve. >>> >>> John >>> >>> On Thu, Mar 15, 2012 at 12:04 PM, Mark F. Adams wrote: >>> You also want: -pc_gamg_agg_nsmooths 1 >>> >>> You are running plain aggregation. If it is Poisson then smoothing is good. >>> >>> Is this problem singular? Can you try running ML with these parameters and see if its performance degrades? The ML implementation uses the PETSC infrastructure and uses a very similar algorithm to GAMG-SA. We should be able to get these two to match pretty well. >>> >>> Mark >>> >>> >>> On Mar 15, 2012, at 12:21 PM, John Mousel wrote: >>> >>>> Mark, >>>> >>>> I ran with those options removed (see the run options listed below). Things actually got slightly worse. Now it's up to 142 iterations. I have attached the ksp_view output. >>>> >>>> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale -ksp_diagonal_scale_fix -mg_levels_ksp_type richardson -mg_levels_pc_type sor -pc_gamg_verbose 1 >>>> >>>> >>>> John >>>> >>>> >>>> On Thu, Mar 15, 2012 at 10:55 AM, Mark F. Adams wrote: >>>> John, can you run again with: -pc_gamg_verbose 1 >>>> >>>> And I would not use: -pc_mg_levels 4 -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 >>>> >>>> 1) I think -mg_coarse_ksp_type preonly and -mg_coarse_pc_sor_its 8 do not do what you think. I think this is the same as 1 iteration. I think you want 'richardson' not 'preonly'. >>>> >>>> 2) Why are you using sor as the coarse solver? If your problem is singular then you want to use as many levels as possible to get the coarse grid to be tiny. I'm pretty sure HYPRE ignores the coarse solver parameters. But ML uses them and it is converging well. >>>> >>>> 3) I would not specify the number of levels. GAMG, and I think the rest, have internal logic for stopping a the right level. If the coarse level is large and you use just 8 iterations of sor then convergence will suffer. >>>> >>>> Mark >>>> >>>> On Mar 15, 2012, at 11:13 AM, John Mousel wrote: >>>> >>>>> Mark, >>>>> >>>>> The changes pulled through this morning. I've run it with the options >>>>> >>>>> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale -ksp_diagonal_scale_fix -pc_mg_levels 4 -mg_levels_ksp_type richardson -mg_levels_pc_type sor -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 >>>>> >>>>> and it converges in the true residual, but it's not converging as fast as anticpated. The matrix arises from a non-symmetric discretization of the Poisson equation. The solve takes GAMG 114 iterations, whereas ML takes 24 iterations, BoomerAMG takes 22 iterations, and -ksp_type bcgsl -pc_type bjacobi -sub_pc_type ilu -sub_pc_factor_levels 4 takes around 170. I've attached the -ksp_view results for ML,GAMG, and HYPRE. I've attempted to make all the options the same on all levels for ML and GAMG. >>>>> >>>>> Any thoughts? >>>>> >>>>> John >>>>> >>>>> >>>>> On Wed, Mar 14, 2012 at 6:04 PM, Mark F. Adams wrote: >>>>> Humm, I see it with hg view (appended). >>>>> >>>>> Satish, my main repo looks hosed. I see this: >>>>> >>>>> ~/Codes/petsc-dev>hg update >>>>> abort: crosses branches (merge branches or use --clean to discard changes) >>>>> ~/Codes/petsc-dev>hg merge >>>>> abort: branch 'default' has 3 heads - please merge with an explicit rev >>>>> (run 'hg heads .' to see heads) >>>>> ~/Codes/petsc-dev>hg heads >>>>> changeset: 22496:8e2a98268179 >>>>> tag: tip >>>>> user: Barry Smith >>>>> date: Wed Mar 14 16:42:25 2012 -0500 >>>>> files: src/vec/is/interface/f90-custom/zindexf90.c src/vec/vec/interface/f90-custom/zvectorf90.c >>>>> description: >>>>> undoing manually changes I put in because Satish had a better fix >>>>> >>>>> >>>>> changeset: 22492:bda4df63072d >>>>> user: Mark F. Adams >>>>> date: Wed Mar 14 17:39:52 2012 -0400 >>>>> files: src/ksp/pc/impls/gamg/tools.c >>>>> description: >>>>> fix for unsymmetric matrices. >>>>> >>>>> >>>>> changeset: 22469:b063baf366e4 >>>>> user: Mark F. Adams >>>>> date: Wed Mar 14 14:22:28 2012 -0400 >>>>> files: src/ksp/pc/impls/gamg/tools.c >>>>> description: >>>>> added fix for preallocation for unsymetric matrices. >>>>> >>>>> Mark >>>>> >>>>> my 'hg view' on my merge repo: >>>>> >>>>> Revision: 22492 >>>>> Branch: default >>>>> Author: Mark F. Adams 2012-03-14 17:39:52 >>>>> Committer: Mark F. Adams 2012-03-14 17:39:52 >>>>> Tags: tip >>>>> Parent: 22491:451bbbd291c2 (Small fixes to the BT linesearch) >>>>> >>>>> fix for unsymmetric matrices. >>>>> >>>>> >>>>> ------------------------ src/ksp/pc/impls/gamg/tools.c ------------------------ >>>>> @@ -103,7 +103,7 @@ >>>>> PetscErrorCode ierr; >>>>> PetscInt Istart,Iend,Ii,jj,ncols,nnz0,nnz1, NN, MM, nloc; >>>>> PetscMPIInt mype, npe; >>>>> - Mat Gmat = *a_Gmat, tGmat; >>>>> + Mat Gmat = *a_Gmat, tGmat, matTrans; >>>>> MPI_Comm wcomm = ((PetscObject)Gmat)->comm; >>>>> const PetscScalar *vals; >>>>> const PetscInt *idx; >>>>> @@ -127,6 +127,10 @@ >>>>> ierr = MatDiagonalScale( Gmat, diag, diag ); CHKERRQ(ierr); >>>>> ierr = VecDestroy( &diag ); CHKERRQ(ierr); >>>>> >>>>> + if( symm ) { >>>>> + ierr = MatTranspose( Gmat, MAT_INITIAL_MATRIX, &matTrans ); CHKERRQ(ierr); >>>>> + } >>>>> + >>>>> /* filter - dup zeros out matrix */ >>>>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &d_nnz ); CHKERRQ(ierr); >>>>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &o_nnz ); CHKERRQ(ierr); >>>>> @@ -135,6 +139,12 @@ >>>>> d_nnz[jj] = ncols; >>>>> o_nnz[jj] = ncols; >>>>> ierr = MatRestoreRow(Gmat,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >>>>> + if( symm ) { >>>>> + ierr = MatGetRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >>>>> + d_nnz[jj] += ncols; >>>>> + o_nnz[jj] += ncols; >>>>> + ierr = MatRestoreRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >>>>> + } >>>>> if( d_nnz[jj] > nloc ) d_nnz[jj] = nloc; >>>>> if( o_nnz[jj] > (MM-nloc) ) o_nnz[jj] = MM - nloc; >>>>> } >>>>> @@ -142,6 +152,9 @@ >>>>> CHKERRQ(ierr); >>>>> ierr = PetscFree( d_nnz ); CHKERRQ(ierr); >>>>> ierr = PetscFree( o_nnz ); CHKERRQ(ierr); >>>>> + if( symm ) { >>>>> + ierr = MatDestroy( &matTrans ); CHKERRQ(ierr); >>>>> + } >>>>> >>>>> >>>>> >>>>> >>>>> On Mar 14, 2012, at 5:53 PM, John Mousel wrote: >>>>> >>>>>> Mark, >>>>>> >>>>>> No change. Can you give me the location that you patched so I can check to make sure it pulled? >>>>>> I don't see it on the petsc-dev change log. >>>>>> >>>>>> John >>>>>> >>>>>> On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams wrote: >>>>>> John, I've committed these changes, give a try. >>>>>> >>>>>> Mark >>>>>> >>>>>> On Mar 14, 2012, at 3:46 PM, Satish Balay wrote: >>>>>> >>>>>> > This is the usual merge [with uncommited changes] issue. >>>>>> > >>>>>> > You could use 'hg shelf' extension to shelve your local changes and >>>>>> > then do a merge [as Sean would suggest] - or do the merge in a >>>>>> > separate/clean clone [I normally do this..] >>>>>> > >>>>>> > i.e >>>>>> > cd ~/Codes >>>>>> > hg clone petsc-dev petsc-dev-merge >>>>>> > cd petsc-dev-merge >>>>>> > hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just to be sure, look for latest chagnes before merge.. >>>>>> > hg merge >>>>>> > hg commit >>>>>> > hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev >>>>>> > >>>>>> > [now update your petsc-dev to latest] >>>>>> > cd ~/Codes/petsc-dev >>>>>> > hg pull >>>>>> > hg update >>>>>> > >>>>>> > Satish >>>>>> > >>>>>> > On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>>>> > >>>>>> >> Great, that seems to work. >>>>>> >> >>>>>> >> I did a 'hg commit tools.c' >>>>>> >> >>>>>> >> and I want to push this file only. I guess its the only thing in the change set so 'hg push' should be fine. But I see this: >>>>>> >> >>>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update >>>>>> >> abort: crosses branches (merge branches or use --clean to discard changes) >>>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge >>>>>> >> abort: outstanding uncommitted changes (use 'hg status' to list changes) >>>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status >>>>>> >> M include/petscmat.h >>>>>> >> M include/private/matimpl.h >>>>>> >> M src/ksp/pc/impls/gamg/agg.c >>>>>> >> M src/ksp/pc/impls/gamg/gamg.c >>>>>> >> M src/ksp/pc/impls/gamg/gamg.h >>>>>> >> M src/ksp/pc/impls/gamg/geo.c >>>>>> >> M src/mat/coarsen/coarsen.c >>>>>> >> M src/mat/coarsen/impls/hem/hem.c >>>>>> >> M src/mat/coarsen/impls/mis/mis.c >>>>>> >> >>>>>> >> Am I ready to do a push? >>>>>> >> >>>>>> >> Thanks, >>>>>> >> Mark >>>>>> >> >>>>>> >> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: >>>>>> >> >>>>>> >>> If commit is the last hg operation that you've done - then 'hg rollback' would undo this commit. >>>>>> >>> >>>>>> >>> Satish >>>>>> >>> >>>>>> >>> On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>>>> >>> >>>>>> >>>> Damn, I'm not preallocating the graph perfectly for unsymmetric matrices and PETSc now dies on this. >>>>>> >>>> >>>>>> >>>> I have a fix but I committed it with other changes that I do not want to commit. The changes are all in one file so I should be able to just commit this file. >>>>>> >>>> >>>>>> >>>> Anyone know how to delete a commit? >>>>>> >>>> >>>>>> >>>> I've tried: >>>>>> >>>> >>>>>> >>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f >>>>>> >>>> hg: unknown command 'strip' >>>>>> >>>> 'strip' is provided by the following extension: >>>>>> >>>> >>>>>> >>>> mq manage a stack of patches >>>>>> >>>> >>>>>> >>>> use "hg help extensions" for information on enabling extensions >>>>>> >>>> >>>>>> >>>> But have not figured out how to load extensions. >>>>>> >>>> >>>>>> >>>> Mark >>>>>> >>>> >>>>>> >>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: >>>>>> >>>> >>>>>> >>>>> Mark, >>>>>> >>>>> >>>>>> >>>>> I have a non-symmetric matrix. I am running with the following options. >>>>>> >>>>> >>>>>> >>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual >>>>>> >>>>> >>>>>> >>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new malloc error: >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> 0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>> >>>>> [0]PETSC ERROR: Argument out of range! >>>>>> >>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! >>>>>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> >>>>> [0]PETSC ERROR: Petsc Development HG revision: 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 -0500 >>>>>> >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>> >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>> >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> >>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 >>>>>> >>>>> [0]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib >>>>>> >>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 >>>>>> >>>>> [0]PETSC ERROR: Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug >>>>>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> >>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >>>>>> >>>>> [0]PETSC ERROR: MatSetValues() line 1141 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c >>>>>> >>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c >>>>>> >>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c >>>>>> >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>>>>> >>>>> [0]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c >>>>>> >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>>> >>>>> [0]PETSC ERROR: KSPSolve() line 385 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> John >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams wrote: >>>>>> >>>>> >>>>>> >>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: >>>>>> >>>>> >>>>>> >>>>>> Mark, >>>>>> >>>>>> >>>>>> >>>>>> The matrix is asymmetric. Does this require the setting of an option? >>>>>> >>>>> >>>>>> >>>>> Yes: -pc_gamg_sym_graph >>>>>> >>>>> >>>>>> >>>>> Mark >>>>>> >>>>> >>>>>> >>>>>> I pulled petsc-dev this morning, so I should have (at least close to) the latest code. >>>>>> >>>>>> >>>>>> >>>>>> John >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >>>>>> >>>>>> >>>>>> >>>>>>> I'm getting the following error when using GAMG. >>>>>> >>>>>>> >>>>>> >>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion `sgid==-1' failed. >>>>>> >>>>>> >>>>>> >>>>>> Is it possible that your matrix is structurally asymmetric? >>>>>> >>>>>> >>>>>> >>>>>> This code is evolving fast and so you will need to move to the dev version if you are not already using it. (I think I fixed a bug that hit this assert). >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>> >>>>>>> When I try to alter the type of aggregation at the command line using -pc_gamg_type pa, I'm getting >>>>>> >>>>>>> >>>>>> >>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>> >>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type: >>>>>> >>>>>>> see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! >>>>>> >>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given! >>>>>> >>>>>>> >>>>>> >>>>>>> Has there been a change in the aggregation options? I just pulled petsc-dev this morning. >>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg for now. >>>>>> >>>>>> >>>>>> >>>>>> Mark >>>>>> >>>>>> >>>>>> >>>>>>> John >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>> >>>>>> >>> >>>>>> >> >>>>>> >> >>>>>> > >>>>>> > >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>> >>> >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From huangsc at gmail.com Tue Mar 20 14:34:01 2012 From: huangsc at gmail.com (Shao-Ching Huang) Date: Tue, 20 Mar 2012 12:34:01 -0700 Subject: [petsc-users] MatNullSpaceCreate In-Reply-To: References: Message-ID: Thanks Matt. From john.mousel at gmail.com Tue Mar 20 14:39:38 2012 From: john.mousel at gmail.com (John Mousel) Date: Tue, 20 Mar 2012 14:39:38 -0500 Subject: [petsc-users] GAMG issue In-Reply-To: <8909A66D-5CB4-479B-ADBC-34A1133D64F4@columbia.edu> References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> <9CB8B9D0-BC04-4471-BCFC-BEE61031A123@columbia.edu> <8909A66D-5CB4-479B-ADBC-34A1133D64F4@columbia.edu> Message-ID: Mark, I am using petsc-dev that I pulled after you made the changes for the non-symmetric discretization allocations last week. I think the difference in our results comes from different convergence tolerances. I'm using an rtol of 1.D-012. It seems to be converging very nicely now. I think I dropped the option to set ksp and pc on levels after a bit, and that seems to have made the difference. GAMG should scale much better than HYPRE and ML right? They both seem to work efficiently for really small core counts, but deteriorate with impressive speed as you go up the ladder. [0]PCSetUp_GAMG level 0 N=46330, n data rows=1, n data cols=1, nnz/row (ave)=6, np=4 [0]scaleFilterGraph 75.5527% nnz after filtering, with threshold 0.05, 6.95957 nnz ave. [0]maxIndSetAgg removed 0 of 46330 vertices. (0 local) [0]PCGAMGprolongator_AGG New grid 5903 nodes PCGAMGoptprol_AGG smooth P0: max eigen=1.923098e+00 min=3.858220e-02 PC=jacobi [0]PCSetUp_GAMG 1) N=5903, n data cols=1, nnz/row (ave)=13, 4 active pes [0]scaleFilterGraph 52.8421% nnz after filtering, with threshold 0.05, 13.3249 nnz ave. [0]maxIndSetAgg removed 0 of 5903 vertices. (0 local) [0]PCGAMGprolongator_AGG New grid 615 nodes PCGAMGoptprol_AGG smooth P0: max eigen=1.575363e+00 min=2.167886e-03 PC=jacobi [0]PCSetUp_GAMG 2) N=615, n data cols=1, nnz/row (ave)=21, 4 active pes [0]scaleFilterGraph 24.7174% nnz after filtering, with threshold 0.05, 21.722 nnz ave. [0]maxIndSetAgg removed 0 of 615 vertices. (0 local) [0]PCGAMGprolongator_AGG New grid 91 nodes PCGAMGoptprol_AGG smooth P0: max eigen=1.676442e+00 min=2.270745e-03 PC=jacobi [0]createLevel aggregate processors: npe: 4 --> 1, neq=91 [0]PCSetUp_GAMG 3) N=91, n data cols=1, nnz/row (ave)=37, 1 active pes [0]scaleFilterGraph 16.4384% nnz after filtering, with threshold 0.05, 37.7033 nnz ave. [0]maxIndSetAgg removed 0 of 91 vertices. (0 local) [0]PCGAMGprolongator_AGG New grid 10 nodes PCGAMGoptprol_AGG smooth P0: max eigen=1.538313e+00 min=8.923063e-04 PC=jacobi [0]PCSetUp_GAMG 4) N=10, n data cols=1, nnz/row (ave)=10, 1 active pes [0]PCSetUp_GAMG 5 levels, grid compexity = 1.29633 Residual norms for pres_ solve. 0 KSP preconditioned resid norm 4.680688832182e+06 true resid norm 2.621342052504e+03 ||r(i)||/||b|| 1.000000000000e+00 2 KSP preconditioned resid norm 1.728993898497e+04 true resid norm 2.888375221014e+03 ||r(i)||/||b|| 1.101868876004e+00 4 KSP preconditioned resid norm 4.510102902646e+02 true resid norm 5.677727287161e+01 ||r(i)||/||b|| 2.165962004744e-02 6 KSP preconditioned resid norm 3.959846836748e+01 true resid norm 1.973580779699e+00 ||r(i)||/||b|| 7.528894513455e-04 8 KSP preconditioned resid norm 3.175473803927e-01 true resid norm 4.315977395174e-02 ||r(i)||/||b|| 1.646476235732e-05 10 KSP preconditioned resid norm 7.502408552205e-04 true resid norm 1.016040400933e-04 ||r(i)||/||b|| 3.876031363257e-08 12 KSP preconditioned resid norm 2.868067261023e-06 true resid norm 1.194542164810e-06 ||r(i)||/||b|| 4.556986997056e-10 KSP Object:(pres_) 4 MPI processes type: bcgsl BCGSL: Ell = 2 BCGSL: Delta = 0 maximum iterations=5000 tolerances: relative=1e-12, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object:(pres_) 4 MPI processes type: gamg MG: type is MULTIPLICATIVE, levels=5 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (pres_mg_coarse_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (pres_mg_coarse_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 8, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=10, cols=10 total: nonzeros=100, allocated nonzeros=100 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 2 nodes, limit used is 5 Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (pres_mg_levels_1_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_1_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=91, cols=91 total: nonzeros=3431, allocated nonzeros=3431 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (pres_mg_levels_2_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_2_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=615, cols=615 total: nonzeros=13359, allocated nonzeros=13359 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (pres_mg_levels_3_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_3_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=5903, cols=5903 total: nonzeros=78657, allocated nonzeros=78657 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 4 ------------------------------- KSP Object: (pres_mg_levels_4_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_4_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=46330, cols=46330 total: nonzeros=322437, allocated nonzeros=615417 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=46330, cols=46330 total: nonzeros=322437, allocated nonzeros=615417 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines On Tue, Mar 20, 2012 at 2:21 PM, Mark F. Adams wrote: > John, > > I had dome diagonal scaling stuff in my input which seemed to mess things > up. I don't understand that. With your hypre parameters I get > > Complexity: grid = 1.408828 > operator = 1.638900 > cycle = 3.277856 > > 0 KSP preconditioned resid norm 2.246209947341e+06 true resid norm > 2.621342052504e+03 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP preconditioned resid norm 6.591054866442e+04 true resid norm > 5.518411654910e+03 ||r(i)||/||b|| 2.105185643224e+00 > 4 KSP preconditioned resid norm 2.721184454964e+03 true resid norm > 1.937153214559e+03 ||r(i)||/||b|| 7.389929188022e-01 > 6 KSP preconditioned resid norm 2.942012838854e+02 true resid norm > 5.614763956317e+01 ||r(i)||/||b|| 2.141942502679e-02 > 8 KSP preconditioned resid norm 2.143421596353e+01 true resid norm > 5.306843482279e+00 ||r(i)||/||b|| 2.024475774617e-03 > 10 KSP preconditioned resid norm 3.689048280659e+00 true resid norm > 2.482945300243e-01 ||r(i)||/||b|| 9.472038560826e-05 > Linear solve converged due to CONVERGED_RTOL iterations 10 > > with ML I get 18 iterations but if I add -pc_ml_Threshold .01 I get it to > 12: > > -@${MPIEXEC} -n 1 ./ex10 -f ./binaryoutput -pc_type ml -ksp_type bcgsl > -pc_gamg_coarse_eq_limit 10 -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph > -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its > 8 -ksp_monitor_true_residual -pc_gamg_verbose 2 -ksp_converged_reason > -options_left -mg_levels_ksp_type richardson -mg_levels_pc_type sor > -pc_ml_maxNlevels 5 -pc_ml_Threshold .01 > > 0 KSP preconditioned resid norm 1.987800354481e+06 true resid norm > 2.621342052504e+03 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP preconditioned resid norm 4.845840795806e+04 true resid norm > 9.664923970856e+03 ||r(i)||/||b|| 3.687013666005e+00 > 4 KSP preconditioned resid norm 4.086337251141e+03 true resid norm > 1.111442892542e+03 ||r(i)||/||b|| 4.239976585582e-01 > 6 KSP preconditioned resid norm 1.496117919395e+03 true resid norm > 4.243682354730e+02 ||r(i)||/||b|| 1.618896835946e-01 > 8 KSP preconditioned resid norm 1.019912311314e+02 true resid norm > 6.165476121107e+01 ||r(i)||/||b|| 2.352030371320e-02 > 10 KSP preconditioned resid norm 1.282179114927e+01 true resid norm > 4.434755525096e+00 ||r(i)||/||b|| 1.691788189512e-03 > 12 KSP preconditioned resid norm 2.801790417375e+00 true resid norm > 4.876299030996e-01 ||r(i)||/||b|| 1.860229963632e-04 > Linear solve converged due to CONVERGED_RTOL iterations 12 > > and gamg: > > -@${MPIEXEC} -n 1 ./ex10 -f ./binaryoutput -pc_type gamg -ksp_type bcgsl > -pc_gamg_coarse_eq_limit 10 -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph > -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its > 8 -ksp_monitor_true_residual -pc_gamg_verbose 2 -ksp_converged_reason > -options_left -mg_levels_ksp_type richardson -mg_levels_pc_type sor > > [0]PCSetUp_GAMG 5 levels, grid compexity = 1.2916 > 0 KSP preconditioned resid norm 6.288750978813e+06 true resid norm > 2.621342052504e+03 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP preconditioned resid norm 3.009668424006e+04 true resid norm > 4.394363256786e+02 ||r(i)||/||b|| 1.676379186222e-01 > 4 KSP preconditioned resid norm 2.079756553216e+01 true resid norm > 5.094584609440e+00 ||r(i)||/||b|| 1.943502414946e-03 > 6 KSP preconditioned resid norm 4.323447593442e+00 true resid norm > 3.146656048880e-01 ||r(i)||/||b|| 1.200398874261e-04 > Linear solve converged due to CONVERGED_RTOL iterations 6 > > So this looks pretty different from what you are getting. Is your code > hardwiring anything? Can you reproduce my results with ksp ex10.c? > > Actually, I just realized that I am using petsc-dev. What version of > PETSc are you using? > > Also, here is the makefile that I use to run this jobs: > > ALL: runex10 > > include ${PETSC_DIR}/conf/variables > include ${PETSC_DIR}/conf/rules > > runex10: > -@${MPIEXEC} -n 1 ./ex10 -f ./binaryoutput -pc_type gamg -ksp_type bcgsl > -pc_gamg_coarse_eq_limit 10 -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph > -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its > 8 -ksp_monitor_true_residual -pc_gamg_verbose 2 -ksp_converged_reason > -options_left -mg_levels_ksp_type richardson -mg_levels_pc_type sor > -pc_ml_maxNlevels 5 -pc_ml_Threshold .01 > -pc_hypre_boomeramg_relax_type_coarse symmetric-SOR/Jacobi > -pc_hypre_boomeramg_grid_sweeps_coarse 4 -pc_hypre_boomeramg_coarsen_type > PMIS > > You just need to run 'make ex10' and then 'make -f this-file'. > > Mark > > On Mar 20, 2012, at 2:45 PM, John Mousel wrote: > > Mark, > > I run ML with the following options. > > -ksp_type bcgsl -pc_type ml -pc_ml_maxNlevels 5 -mg_coarse_ksp_type > richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_monitor > -ksp_view > > Note the lack of scaling. For some reason scaling seems to mess with ML. > As you can see below, ML converges very nicely. > > With regards to HYPRE, this one took a bit of work to get convergence. The > options that work are: > > -ksp_type bcgsl -pc_type hypre -pc_hypre_type boomeramg > -ksp_monitor_true_residual -pc_hypre_boomeramg_relax_type_coarse > symmetric-SOR/Jacobi -pc_hypre_boomeramg_grid_sweeps_coarse 4 > -pc_hypre_boomeramg_coarsen_type PMIS > > The problem is that neither of ML or HYPRE seem to scale at all. > > ML output: > 0 KSP preconditioned resid norm 1.538968715109e+06 true resid norm > 2.621342052504e+03 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP preconditioned resid norm 1.263129058693e+05 true resid norm > 1.096298699671e+04 ||r(i)||/||b|| 4.182203915830e+00 > 4 KSP preconditioned resid norm 2.277379585186e+04 true resid norm > 2.999721659930e+03 ||r(i)||/||b|| 1.144345758717e+00 > 6 KSP preconditioned resid norm 4.766504457975e+03 true resid norm > 6.733421603796e+02 ||r(i)||/||b|| 2.568692474667e-01 > 8 KSP preconditioned resid norm 2.139020425406e+03 true resid norm > 1.360842101250e+02 ||r(i)||/||b|| 5.191394613876e-02 > 10 KSP preconditioned resid norm 6.621380459944e+02 true resid norm > 1.522758800025e+02 ||r(i)||/||b|| 5.809080881188e-02 > 12 KSP preconditioned resid norm 2.973409610262e+01 true resid norm > 1.161046206089e+01 ||r(i)||/||b|| 4.429205280479e-03 > 14 KSP preconditioned resid norm 2.532665482573e+00 true resid norm > 2.557425874623e+00 ||r(i)||/||b|| 9.756170020543e-04 > 16 KSP preconditioned resid norm 2.375585214826e+00 true resid norm > 2.441783841415e+00 ||r(i)||/||b|| 9.315014189327e-04 > 18 KSP preconditioned resid norm 1.436338060675e-02 true resid norm > 1.305304828818e-02 ||r(i)||/||b|| 4.979528816437e-06 > 20 KSP preconditioned resid norm 4.088293864561e-03 true resid norm > 9.841243465634e-04 ||r(i)||/||b|| 3.754276728683e-07 > 22 KSP preconditioned resid norm 6.140822977383e-04 true resid norm > 1.682184150207e-04 ||r(i)||/||b|| 6.417263052718e-08 > 24 KSP preconditioned resid norm 2.685415483430e-05 true resid norm > 1.065041542336e-05 ||r(i)||/||b|| 4.062962867890e-09 > 26 KSP preconditioned resid norm 1.620776166579e-06 true resid norm > 9.563268703474e-07 ||r(i)||/||b|| 3.648233809982e-10 > 28 KSP preconditioned resid norm 2.823291105652e-07 true resid norm > 7.705418741178e-08 ||r(i)||/||b|| 2.939493811507e-11 > KSP Object:(pres_) 4 MPI processes > type: bcgsl > BCGSL: Ell = 2 > BCGSL: Delta = 0 > maximum iterations=5000 > tolerances: relative=1e-12, absolute=1e-50, divergence=10000 > left preconditioning > has attached null space > using nonzero initial guess > using PRECONDITIONED norm type for convergence test > PC Object:(pres_) 4 MPI processes > type: ml > MG: type is MULTIPLICATIVE, levels=5 cycles=v > Cycles per PCApply=1 > Using Galerkin computed coarse grid matrices > Coarse grid solver -- level ------------------------------- > KSP Object: (pres_mg_coarse_) 4 MPI processes > type: richardson > Richardson: damping factor=1 > maximum iterations=1, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: (pres_mg_coarse_) 4 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 8, local iterations = 1, > omega = 1 > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=4, cols=4 > total: nonzeros=16, allocated nonzeros=16 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Down solver (pre-smoother) on level 1 ------------------------------- > KSP Object: (pres_mg_levels_1_) 4 MPI processes > type: richardson > Richardson: damping factor=1 > maximum iterations=1 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using nonzero initial guess > using PRECONDITIONED norm type for convergence test > PC Object: (pres_mg_levels_1_) 4 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, > omega = 1 > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=25, cols=25 > total: nonzeros=303, allocated nonzeros=303 > total number of mallocs used during MatSetValues calls =0 > using I-node (on process 0) routines: found 4 nodes, limit used > is 5 > Up solver (post-smoother) same as down solver (pre-smoother) > Down solver (pre-smoother) on level 2 ------------------------------- > KSP Object: (pres_mg_levels_2_) 4 MPI processes > type: richardson > Richardson: damping factor=1 > maximum iterations=1 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using nonzero initial guess > using PRECONDITIONED norm type for convergence test > PC Object: (pres_mg_levels_2_) 4 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, > omega = 1 > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=423, cols=423 > total: nonzeros=7437, allocated nonzeros=7437 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Up solver (post-smoother) same as down solver (pre-smoother) > Down solver (pre-smoother) on level 3 ------------------------------- > KSP Object: (pres_mg_levels_3_) 4 MPI processes > type: richardson > Richardson: damping factor=1 > maximum iterations=1 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using nonzero initial guess > using PRECONDITIONED norm type for convergence test > PC Object: (pres_mg_levels_3_) 4 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, > omega = 1 > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=6617, cols=6617 > total: nonzeros=88653, allocated nonzeros=88653 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Up solver (post-smoother) same as down solver (pre-smoother) > Down solver (pre-smoother) on level 4 ------------------------------- > KSP Object: (pres_mg_levels_4_) 4 MPI processes > type: richardson > Richardson: damping factor=1 > maximum iterations=1 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > has attached null space > using nonzero initial guess > using PRECONDITIONED norm type for convergence test > PC Object: (pres_mg_levels_4_) 4 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, > omega = 1 > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=46330, cols=46330 > total: nonzeros=322437, allocated nonzeros=615417 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Up solver (post-smoother) same as down solver (pre-smoother) > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=46330, cols=46330 > total: nonzeros=322437, allocated nonzeros=615417 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > > > John > > > > On Tue, Mar 20, 2012 at 1:33 PM, Mark F. Adams wrote: > >> John, >> >> I am getting poor results (diverging) from ML also: >> >> 0 KSP preconditioned resid norm 3.699832960909e+22 true resid norm >> 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 >> 2 KSP preconditioned resid norm 5.706378365783e+11 true resid norm >> 1.563902233018e+03 ||r(i)||/||b|| 1.193204539995e+00 >> 4 KSP preconditioned resid norm 5.570291685152e+11 true resid norm >> 1.564235542744e+03 ||r(i)||/||b|| 1.193458844050e+00 >> 6 KSP preconditioned resid norm 5.202150407298e+10 true resid norm >> 1.749929789082e+03 ||r(i)||/||b|| 1.335137277077e+00 >> Linear solve converged due to CONVERGED_RTOL iterations 6 >> >> With GAMG I get: >> >> 0 KSP preconditioned resid norm 7.731260075891e+06 true resid norm >> 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 >> 2 KSP preconditioned resid norm 2.856415184685e+05 true resid norm >> 1.310410242531e+03 ||r(i)||/||b|| 9.997987199150e-01 >> 4 KSP preconditioned resid norm 1.528467019258e+05 true resid norm >> 1.284856538976e+03 ||r(i)||/||b|| 9.803021078816e-01 >> 6 KSP preconditioned resid norm 1.451091957899e+05 true resid norm >> 1.564309254168e+03 ||r(i)||/||b|| 1.193515083375e+00 >> >> >> >> 122 KSP preconditioned resid norm 2.486245341783e+01 true resid norm >> 1.404397185367e+00 ||r(i)||/||b|| 1.071507580306e-03 >> 124 KSP preconditioned resid norm 1.482316853621e+01 true resid norm >> 4.488661881759e-01 ||r(i)||/||b|| 3.424697287811e-04 >> 126 KSP preconditioned resid norm 1.481941150253e+01 true resid norm >> 4.484480100832e-01 ||r(i)||/||b|| 3.421506730318e-04 >> 128 KSP preconditioned resid norm 8.191887347033e+00 true resid norm >> 6.678630367218e-01 ||r(i)||/||b|| 5.095569215816e-04 >> >> And HYPRE: >> >> 0 KSP preconditioned resid norm 3.774510769907e+04 true resid norm >> 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 >> 2 KSP preconditioned resid norm 1.843165835831e+04 true resid norm >> 8.502433792869e+02 ||r(i)||/||b|| 6.487069580482e-01 >> 4 KSP preconditioned resid norm 1.573176624705e+04 true resid norm >> 1.167264367302e+03 ||r(i)||/||b|| 8.905832558033e-01 >> 6 KSP preconditioned resid norm 1.657958380765e+04 true resid norm >> 8.684701624902e+02 ||r(i)||/||b|| 6.626133775216e-01 >> 8 KSP preconditioned resid norm 2.190304455083e+04 true resid norm >> 6.969893263600e+02 ||r(i)||/||b|| 5.317792960344e-01 >> 10 KSP preconditioned resid norm 2.485714630000e+04 true resid norm >> 6.642641436830e+02 ||r(i)||/||b|| 5.068110878446e-01 >> >> >> >> 62 KSP preconditioned resid norm 6.432516040957e+00 true resid norm >> 2.124960171419e-01 ||r(i)||/||b|| 1.621272781837e-04 >> 64 KSP preconditioned resid norm 5.731033176541e+00 true resid norm >> 1.338816774003e-01 ||r(i)||/||b|| 1.021471943216e-04 >> 66 KSP preconditioned resid norm 1.600285935522e-01 true resid norm >> 3.352408932031e-03 ||r(i)||/||b|| 2.557774695353e-06 >> >> ML and GAMG should act similarly, but ML seems to have a problem (see the >> preconditioned norm difference and its diverging). ML has a parameter: >> >> -pc_ml_Threshold [.0] >> >> I setting this to 0.05 (GAMG default) helps a bit but it still diverges. >> >> So it would be nice to figure out the difference between ML and GAMG, but >> that is secondary for you as the both suck. >> >> HYPRE is a very different algorithm. It looks like the smoothing in GAMG >> (and ML) may be the problem. If I turn smoothing off >> (-pc_gamg_agg_nsmooths 0) and I get for GAMG: >> >> 0 KSP preconditioned resid norm 2.186148437534e+05 true resid norm >> 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 >> 2 KSP preconditioned resid norm 2.916843959765e+04 true resid norm >> 3.221533667508e+03 ||r(i)||/||b|| 2.457921292432e+00 >> 4 KSP preconditioned resid norm 2.396374655925e+04 true resid norm >> 1.834299897412e+03 ||r(i)||/||b|| 1.399508817812e+00 >> 6 KSP preconditioned resid norm 2.509576275453e+04 true resid norm >> 1.035475461174e+03 ||r(i)||/||b|| 7.900327752214e-01 >> >> >> >> 64 KSP preconditioned resid norm 1.973859758284e+01 true resid norm >> 7.322674977169e+00 ||r(i)||/||b|| 5.586953482895e-03 >> 66 KSP preconditioned resid norm 3.371598890438e+01 true resid norm >> 7.152754930495e+00 ||r(i)||/||b|| 5.457310231004e-03 >> 68 KSP preconditioned resid norm 4.687839294418e+00 true resid norm >> 4.605726307025e-01 ||r(i)||/||b|| 3.514013487219e-04 >> 70 KSP preconditioned resid norm 1.487545519493e+00 true resid norm >> 1.558723789416e-01 ||r(i)||/||b|| 1.189253562571e-04 >> 72 KSP preconditioned resid norm 5.317329808718e-01 true resid norm >> 5.027178038177e-02 ||r(i)||/||b|| 3.835566911967e-05 >> 74 KSP preconditioned resid norm 3.405339702462e-01 true resid norm >> 1.897059263835e-02 ||r(i)||/||b|| 1.447392092969e-05 >> >> This is almost as good as HYPRE. >> >> An other thing to keep in mind is the cost of each iteration, not just >> then number of iterations, You can >> use -pc_hypre_boomeramg_print_statistics to get some data on this from >> HYPRE: >> >> Average Convergence Factor = 0.537664 >> >> Complexity: grid = 1.780207 >> operator = 2.624910 >> cycle = 5.249670 >> >> And GAMG prints this with verbose set: >> >> [0]PCSetUp_GAMG 6 levels, grid compexity [sic] = 1.1316 >> >> I believe that the hypre "Complexity: grid" is the same as my "grid >> complexity". So hypre actually looks more expensive at this point. >> >> I've worked on optimizing parameters for hypre with the hypre people and >> here are a set of arguments that I've used: >> >> -pc_hypre_boomeramg_no_CF -pc_hypre_boomeramg_agg_nl 1 >> -pc_hypre_boomeramg_coarsen_type HMIS -pc_hypre_boomeramg_interp_type ext+i >> -pc_hypre_boomeramg_P_max 4 -pc_hypre_boomeramg_agg_num_paths 2 >> >> With these parmeters I get: >> >> Complexity: grid = 1.244140 >> operator = 1.396722 >> cycle = 2.793442 >> >> and: >> >> 0 KSP preconditioned resid norm 4.698624821403e+04 true resid norm >> 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 >> 2 KSP preconditioned resid norm 2.207967626172e+04 true resid norm >> 3.466160021150e+03 ||r(i)||/||b|| 2.644562931280e+00 >> 4 KSP preconditioned resid norm 2.278468320876e+04 true resid norm >> 1.246784122467e+03 ||r(i)||/||b|| 9.512541410282e-01 >> >> >> >> 56 KSP preconditioned resid norm 1.108460232262e+00 true resid norm >> 8.276869475681e-02 ||r(i)||/||b|| 6.314971631105e-05 >> 58 KSP preconditioned resid norm 3.617217454336e-01 true resid norm >> 3.764556404754e-02 ||r(i)||/||b|| 2.872229285428e-05 >> 60 KSP preconditioned resid norm 1.666532560770e-01 true resid norm >> 5.149302513338e-03 ||r(i)||/||b|| 3.928743758404e-06 >> Linear solve converged due to CONVERGED_RTOL iterations 60 >> >> So this actually converged faster with lower complexity. >> >> Anyway these result seem different from what you are getting, so I've >> appended my options. This uses ex10 in the KSP tutorials to read in your >> binary file. >> >> Mark >> >> #PETSc Option Table entries: >> -f ./binaryoutput >> -ksp_converged_reason >> -ksp_diagonal_scale >> -ksp_diagonal_scale_fix >> -ksp_monitor_true_residual >> -ksp_type bcgsl >> -mg_coarse_ksp_type richardson >> -mg_coarse_pc_sor_its 8 >> -mg_coarse_pc_type sor >> -mg_levels_ksp_type richardson >> -mg_levels_pc_type sor >> -options_left >> -pc_gamg_agg_nsmooths 0 >> -pc_gamg_coarse_eq_limit 10 >> -pc_gamg_sym_graph >> -pc_gamg_verbose 2 >> -pc_hypre_boomeramg_P_max 4 >> -pc_hypre_boomeramg_agg_nl 1 >> -pc_hypre_boomeramg_agg_num_paths 2 >> -pc_hypre_boomeramg_coarsen_type HMIS >> -pc_hypre_boomeramg_interp_type ext+i >> -pc_hypre_boomeramg_no_CF >> -pc_ml_Threshold .01 >> -pc_type gamg >> -vecload_block_size 1 >> #End of PETSc Option Table entries >> There are 7 unused database options. They are: >> Option left: name:-pc_hypre_boomeramg_P_max value: 4 >> Option left: name:-pc_hypre_boomeramg_agg_nl value: 1 >> Option left: name:-pc_hypre_boomeramg_agg_num_paths value: 2 >> Option left: name:-pc_hypre_boomeramg_coarsen_type value: HMIS >> Option left: name:-pc_hypre_boomeramg_interp_type value: ext+i >> Option left: name:-pc_hypre_boomeramg_no_CF no value >> Option left: name:-pc_ml_Threshold value: .01 >> >> >> On Mar 20, 2012, at 10:19 AM, John Mousel wrote: >> >> Mark, >> >> Sorry for the late reply. I've been on travel and hadn't had a chance to >> pick this back up. I've tried running with the suggested options: >> >> -ksp_type bcgsl -pc_type gamg -pc_gamg_coarse_eq_limit 10 >> -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph -mg_coarse_ksp_type richardson >> -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_diagonal_scale >> -ksp_diagonal_scale_fix -ksp_monitor_true_residual -ksp_view >> -pc_gamg_verbose 1 >> >> With these options, the convergence starts to hang (see attached >> GAMG_kspview.txt). The hanging happens for both -mg_coarse_ksp_type >> richardson and preonly. It was my understanding from previous emails that >> using preonly made it so that only the preconditioner was run, which in >> this case would be 8 sweeps of SOR. If I get rid of the >> -pc_gamg_agg_nsmooths 1 (see GAMG_kspview_nosmooth.txt), the problem >> converges, but again the convergence is slow. Without this option, both >> Richardson and preonly converge in 172 iterations. >> >> Matt, I've checked and the problem does converge in the true residual >> using GAMG, ML, HYPRE, and ILU preconditioned BiCG. I explicitly ensure >> that a solution exists by projecting the rhs vector out of the nullity of >> the transpose of operator. >> >> John >> >> >> On Fri, Mar 16, 2012 at 2:04 PM, Mark F. Adams wrote: >> >>> John, did this get resolved? >>> Mark >>> >>> On Mar 15, 2012, at 4:24 PM, John Mousel wrote: >>> >>> Mark, >>> >>> Running without the options you mentioned before leads to slightly worse >>> performance (175 iterations). >>> I have not been able to get run coarse grid solve to work with LU while >>> running ML. It keeps experiencing a zero pivot, and all the combinations of >>> shifting i've tried haven't lead me anywhere, hence the SOR on the course >>> grid. Also, the ML manual suggests limiting the number of levels to 3 or 4 >>> and performing a few sweeps of an iterative method as opposed to a direct >>> solve. >>> >>> John >>> >>> On Thu, Mar 15, 2012 at 12:04 PM, Mark F. Adams >> > wrote: >>> >>>> You also want: -pc_gamg_agg_nsmooths 1 >>>> >>>> You are running plain aggregation. If it is Poisson then smoothing is >>>> good. >>>> >>>> Is this problem singular? Can you try running ML with these parameters >>>> and see if its performance degrades? The ML implementation uses the PETSC >>>> infrastructure and uses a very similar algorithm to GAMG-SA. We should be >>>> able to get these two to match pretty well. >>>> >>>> Mark >>>> >>>> >>>> On Mar 15, 2012, at 12:21 PM, John Mousel wrote: >>>> >>>> Mark, >>>> >>>> I ran with those options removed (see the run options listed below). >>>> Things actually got slightly worse. Now it's up to 142 iterations. I have >>>> attached the ksp_view output. >>>> >>>> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale >>>> -ksp_diagonal_scale_fix -mg_levels_ksp_type richardson -mg_levels_pc_type >>>> sor -pc_gamg_verbose 1 >>>> >>>> >>>> John >>>> >>>> >>>> On Thu, Mar 15, 2012 at 10:55 AM, Mark F. Adams < >>>> mark.adams at columbia.edu> wrote: >>>> >>>>> John, can you run again with: -pc_gamg_verbose 1 >>>>> >>>>> And I would not use: -pc_mg_levels 4 -mg_coarse_ksp_type preonly >>>>> -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 >>>>> >>>>> 1) I think -mg_coarse_ksp_type preonly and -mg_coarse_pc_sor_its 8 do >>>>> not do what you think. I think this is the same as 1 iteration. I think >>>>> you want 'richardson' not 'preonly'. >>>>> >>>>> 2) Why are you using sor as the coarse solver? If your problem is >>>>> singular then you want to use as many levels as possible to get the coarse >>>>> grid to be tiny. I'm pretty sure HYPRE ignores the coarse solver >>>>> parameters. But ML uses them and it is converging well. >>>>> >>>>> 3) I would not specify the number of levels. GAMG, and I think the >>>>> rest, have internal logic for stopping a the right level. If the coarse >>>>> level is large and you use just 8 iterations of sor then convergence will >>>>> suffer. >>>>> >>>>> Mark >>>>> >>>>> On Mar 15, 2012, at 11:13 AM, John Mousel wrote: >>>>> >>>>> Mark, >>>>> >>>>> The changes pulled through this morning. I've run it with the options >>>>> >>>>> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale >>>>> -ksp_diagonal_scale_fix -pc_mg_levels 4 -mg_levels_ksp_type richardson >>>>> -mg_levels_pc_type sor -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor >>>>> -mg_coarse_pc_sor_its 8 >>>>> >>>>> and it converges in the true residual, but it's not converging as fast >>>>> as anticpated. The matrix arises from a non-symmetric discretization of the >>>>> Poisson equation. The solve takes GAMG 114 iterations, whereas ML takes 24 >>>>> iterations, BoomerAMG takes 22 iterations, and -ksp_type bcgsl -pc_type >>>>> bjacobi -sub_pc_type ilu -sub_pc_factor_levels 4 takes around 170. I've >>>>> attached the -ksp_view results for ML,GAMG, and HYPRE. I've attempted to >>>>> make all the options the same on all levels for ML and GAMG. >>>>> >>>>> Any thoughts? >>>>> >>>>> John >>>>> >>>>> >>>>> On Wed, Mar 14, 2012 at 6:04 PM, Mark F. Adams < >>>>> mark.adams at columbia.edu> wrote: >>>>> >>>>>> Humm, I see it with hg view (appended). >>>>>> >>>>>> Satish, my main repo looks hosed. I see this: >>>>>> >>>>>> ~/Codes/petsc-dev>hg update >>>>>> abort: crosses branches (merge branches or use --clean to discard >>>>>> changes) >>>>>> ~/Codes/petsc-dev>hg merge >>>>>> abort: branch 'default' has 3 heads - please merge with an explicit >>>>>> rev >>>>>> (run 'hg heads .' to see heads) >>>>>> ~/Codes/petsc-dev>hg heads >>>>>> changeset: 22496:8e2a98268179 >>>>>> tag: tip >>>>>> user: Barry Smith >>>>>> date: Wed Mar 14 16:42:25 2012 -0500 >>>>>> files: src/vec/is/interface/f90-custom/zindexf90.c >>>>>> src/vec/vec/interface/f90-custom/zvectorf90.c >>>>>> description: >>>>>> undoing manually changes I put in because Satish had a better fix >>>>>> >>>>>> >>>>>> changeset: 22492:bda4df63072d >>>>>> user: Mark F. Adams >>>>>> date: Wed Mar 14 17:39:52 2012 -0400 >>>>>> files: src/ksp/pc/impls/gamg/tools.c >>>>>> description: >>>>>> fix for unsymmetric matrices. >>>>>> >>>>>> >>>>>> changeset: 22469:b063baf366e4 >>>>>> user: Mark F. Adams >>>>>> date: Wed Mar 14 14:22:28 2012 -0400 >>>>>> files: src/ksp/pc/impls/gamg/tools.c >>>>>> description: >>>>>> added fix for preallocation for unsymetric matrices. >>>>>> >>>>>> Mark >>>>>> >>>>>> my 'hg view' on my merge repo: >>>>>> >>>>>> Revision: 22492 >>>>>> Branch: default >>>>>> Author: Mark F. Adams 2012-03-14 17:39:52 >>>>>> Committer: Mark F. Adams 2012-03-14 >>>>>> 17:39:52 >>>>>> Tags: tip >>>>>> Parent: 22491:451bbbd291c2 (Small fixes to the BT linesearch) >>>>>> >>>>>> fix for unsymmetric matrices. >>>>>> >>>>>> >>>>>> ------------------------ src/ksp/pc/impls/gamg/tools.c >>>>>> ------------------------ >>>>>> @@ -103,7 +103,7 @@ >>>>>> PetscErrorCode ierr; >>>>>> PetscInt Istart,Iend,Ii,jj,ncols,nnz0,nnz1, NN, MM, nloc; >>>>>> PetscMPIInt mype, npe; >>>>>> - Mat Gmat = *a_Gmat, tGmat; >>>>>> + Mat Gmat = *a_Gmat, tGmat, matTrans; >>>>>> MPI_Comm wcomm = ((PetscObject)Gmat)->comm; >>>>>> const PetscScalar *vals; >>>>>> const PetscInt *idx; >>>>>> @@ -127,6 +127,10 @@ >>>>>> ierr = MatDiagonalScale( Gmat, diag, diag ); CHKERRQ(ierr); >>>>>> ierr = VecDestroy( &diag ); CHKERRQ(ierr); >>>>>> >>>>>> + if( symm ) { >>>>>> + ierr = MatTranspose( Gmat, MAT_INITIAL_MATRIX, &matTrans ); >>>>>> CHKERRQ(ierr); >>>>>> + } >>>>>> + >>>>>> /* filter - dup zeros out matrix */ >>>>>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &d_nnz ); CHKERRQ(ierr); >>>>>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &o_nnz ); CHKERRQ(ierr); >>>>>> @@ -135,6 +139,12 @@ >>>>>> d_nnz[jj] = ncols; >>>>>> o_nnz[jj] = ncols; >>>>>> ierr = MatRestoreRow(Gmat,Ii,&ncols,PETSC_NULL,PETSC_NULL); >>>>>> CHKERRQ(ierr); >>>>>> + if( symm ) { >>>>>> + ierr = MatGetRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); >>>>>> CHKERRQ(ierr); >>>>>> + d_nnz[jj] += ncols; >>>>>> + o_nnz[jj] += ncols; >>>>>> + ierr = >>>>>> MatRestoreRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >>>>>> + } >>>>>> if( d_nnz[jj] > nloc ) d_nnz[jj] = nloc; >>>>>> if( o_nnz[jj] > (MM-nloc) ) o_nnz[jj] = MM - nloc; >>>>>> } >>>>>> @@ -142,6 +152,9 @@ >>>>>> CHKERRQ(ierr); >>>>>> ierr = PetscFree( d_nnz ); CHKERRQ(ierr); >>>>>> ierr = PetscFree( o_nnz ); CHKERRQ(ierr); >>>>>> + if( symm ) { >>>>>> + ierr = MatDestroy( &matTrans ); CHKERRQ(ierr); >>>>>> + } >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mar 14, 2012, at 5:53 PM, John Mousel wrote: >>>>>> >>>>>> Mark, >>>>>> >>>>>> No change. Can you give me the location that you patched so I can >>>>>> check to make sure it pulled? >>>>>> I don't see it on the petsc-dev change log. >>>>>> >>>>>> John >>>>>> >>>>>> On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams < >>>>>> mark.adams at columbia.edu> wrote: >>>>>> >>>>>>> John, I've committed these changes, give a try. >>>>>>> >>>>>>> Mark >>>>>>> >>>>>>> On Mar 14, 2012, at 3:46 PM, Satish Balay wrote: >>>>>>> >>>>>>> > This is the usual merge [with uncommited changes] issue. >>>>>>> > >>>>>>> > You could use 'hg shelf' extension to shelve your local changes and >>>>>>> > then do a merge [as Sean would suggest] - or do the merge in a >>>>>>> > separate/clean clone [I normally do this..] >>>>>>> > >>>>>>> > i.e >>>>>>> > cd ~/Codes >>>>>>> > hg clone petsc-dev petsc-dev-merge >>>>>>> > cd petsc-dev-merge >>>>>>> > hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just >>>>>>> to be sure, look for latest chagnes before merge.. >>>>>>> > hg merge >>>>>>> > hg commit >>>>>>> > hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev >>>>>>> > >>>>>>> > [now update your petsc-dev to latest] >>>>>>> > cd ~/Codes/petsc-dev >>>>>>> > hg pull >>>>>>> > hg update >>>>>>> > >>>>>>> > Satish >>>>>>> > >>>>>>> > On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>>>>> > >>>>>>> >> Great, that seems to work. >>>>>>> >> >>>>>>> >> I did a 'hg commit tools.c' >>>>>>> >> >>>>>>> >> and I want to push this file only. I guess its the only thing in >>>>>>> the change set so 'hg push' should be fine. But I see this: >>>>>>> >> >>>>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update >>>>>>> >> abort: crosses branches (merge branches or use --clean to discard >>>>>>> changes) >>>>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge >>>>>>> >> abort: outstanding uncommitted changes (use 'hg status' to list >>>>>>> changes) >>>>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status >>>>>>> >> M include/petscmat.h >>>>>>> >> M include/private/matimpl.h >>>>>>> >> M src/ksp/pc/impls/gamg/agg.c >>>>>>> >> M src/ksp/pc/impls/gamg/gamg.c >>>>>>> >> M src/ksp/pc/impls/gamg/gamg.h >>>>>>> >> M src/ksp/pc/impls/gamg/geo.c >>>>>>> >> M src/mat/coarsen/coarsen.c >>>>>>> >> M src/mat/coarsen/impls/hem/hem.c >>>>>>> >> M src/mat/coarsen/impls/mis/mis.c >>>>>>> >> >>>>>>> >> Am I ready to do a push? >>>>>>> >> >>>>>>> >> Thanks, >>>>>>> >> Mark >>>>>>> >> >>>>>>> >> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: >>>>>>> >> >>>>>>> >>> If commit is the last hg operation that you've done - then 'hg >>>>>>> rollback' would undo this commit. >>>>>>> >>> >>>>>>> >>> Satish >>>>>>> >>> >>>>>>> >>> On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>>>>> >>> >>>>>>> >>>> Damn, I'm not preallocating the graph perfectly for unsymmetric >>>>>>> matrices and PETSc now dies on this. >>>>>>> >>>> >>>>>>> >>>> I have a fix but I committed it with other changes that I do >>>>>>> not want to commit. The changes are all in one file so I should be able to >>>>>>> just commit this file. >>>>>>> >>>> >>>>>>> >>>> Anyone know how to delete a commit? >>>>>>> >>>> >>>>>>> >>>> I've tried: >>>>>>> >>>> >>>>>>> >>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip >>>>>>> 22487:26ffb9eef17f >>>>>>> >>>> hg: unknown command 'strip' >>>>>>> >>>> 'strip' is provided by the following extension: >>>>>>> >>>> >>>>>>> >>>> mq manage a stack of patches >>>>>>> >>>> >>>>>>> >>>> use "hg help extensions" for information on enabling extensions >>>>>>> >>>> >>>>>>> >>>> But have not figured out how to load extensions. >>>>>>> >>>> >>>>>>> >>>> Mark >>>>>>> >>>> >>>>>>> >>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: >>>>>>> >>>> >>>>>>> >>>>> Mark, >>>>>>> >>>>> >>>>>>> >>>>> I have a non-symmetric matrix. I am running with the following >>>>>>> options. >>>>>>> >>>>> >>>>>>> >>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual >>>>>>> >>>>> >>>>>>> >>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new >>>>>>> malloc error: >>>>>>> >>>>> >>>>>>> >>>>> >>>>>>> >>>>> 0]PETSC ERROR: --------------------- Error Message >>>>>>> ------------------------------------ >>>>>>> >>>>> [0]PETSC ERROR: Argument out of range! >>>>>>> >>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! >>>>>>> >>>>> [0]PETSC ERROR: >>>>>>> ------------------------------------------------------------------------ >>>>>>> >>>>> [0]PETSC ERROR: Petsc Development HG revision: >>>>>>> 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 >>>>>>> -0500 >>>>>>> >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>> >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble >>>>>>> shooting. >>>>>>> >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>>>> >>>>> [0]PETSC ERROR: >>>>>>> ------------------------------------------------------------------------ >>>>>>> >>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named >>>>>>> wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 >>>>>>> >>>>> [0]PETSC ERROR: Libraries linked from >>>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib >>>>>>> >>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 >>>>>>> >>>>> [0]PETSC ERROR: Configure options --download-blacs=1 >>>>>>> --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 >>>>>>> --download-parmetis=1 --download-scalapack=1 >>>>>>> --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc >>>>>>> --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort >>>>>>> PETSC_ARCH=linux-debug >>>>>>> >>>>> [0]PETSC ERROR: >>>>>>> ------------------------------------------------------------------------ >>>>>>> >>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in >>>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >>>>>>> >>>>> [0]PETSC ERROR: MatSetValues() line 1141 in >>>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c >>>>>>> >>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in >>>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c >>>>>>> >>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in >>>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c >>>>>>> >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in >>>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>>>>>> >>>>> [0]PETSC ERROR: PCSetUp() line 832 in >>>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c >>>>>>> >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in >>>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>>>> >>>>> [0]PETSC ERROR: KSPSolve() line 385 in >>>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>>>> >>>>> >>>>>>> >>>>> >>>>>>> >>>>> John >>>>>>> >>>>> >>>>>>> >>>>> >>>>>>> >>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams < >>>>>>> mark.adams at columbia.edu> wrote: >>>>>>> >>>>> >>>>>>> >>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: >>>>>>> >>>>> >>>>>>> >>>>>> Mark, >>>>>>> >>>>>> >>>>>>> >>>>>> The matrix is asymmetric. Does this require the setting of an >>>>>>> option? >>>>>>> >>>>> >>>>>>> >>>>> Yes: -pc_gamg_sym_graph >>>>>>> >>>>> >>>>>>> >>>>> Mark >>>>>>> >>>>> >>>>>>> >>>>>> I pulled petsc-dev this morning, so I should have (at least >>>>>>> close to) the latest code. >>>>>>> >>>>>> >>>>>>> >>>>>> John >>>>>>> >>>>>> >>>>>>> >>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams < >>>>>>> mark.adams at columbia.edu> wrote: >>>>>>> >>>>>> >>>>>>> >>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >>>>>>> >>>>>> >>>>>>> >>>>>>> I'm getting the following error when using GAMG. >>>>>>> >>>>>>> >>>>>>> >>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: >>>>>>> Assertion `sgid==-1' failed. >>>>>>> >>>>>> >>>>>>> >>>>>> Is it possible that your matrix is structurally asymmetric? >>>>>>> >>>>>> >>>>>>> >>>>>> This code is evolving fast and so you will need to move to >>>>>>> the dev version if you are not already using it. (I think I fixed a bug >>>>>>> that hit this assert). >>>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> When I try to alter the type of aggregation at the command >>>>>>> line using -pc_gamg_type pa, I'm getting >>>>>>> >>>>>>> >>>>>>> >>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error >>>>>>> Message ------------------------------------ >>>>>>> >>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or >>>>>>> missing external package needed for type: >>>>>>> >>>>>>> see >>>>>>> http://www.mcs.anl.gov/petsc/documentation/installation.html#external >>>>>>> ! >>>>>>> >>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given! >>>>>>> >>>>>>> >>>>>>> >>>>>>> Has there been a change in the aggregation options? I just >>>>>>> pulled petsc-dev this morning. >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>>> >>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg >>>>>>> for now. >>>>>>> >>>>>> >>>>>>> >>>>>> Mark >>>>>>> >>>>>> >>>>>>> >>>>>>> John >>>>>>> >>>>>> >>>>>>> >>>>>> >>>>>>> >>>>> >>>>>>> >>>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>> >>>>>>> >>> >>>>>>> >> >>>>>>> >> >>>>>>> > >>>>>>> > >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> >>> >>> >> >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Tue Mar 20 15:02:12 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Tue, 20 Mar 2012 16:02:12 -0400 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> <9CB8! B9D0-BC04-4471-BCFC-BEE61031A123@columbia.edu> <8909A66D-5CB4-479B-ADBC-34A1133D64F4@columbia.edu> Message-ID: On Mar 20, 2012, at 3:39 PM, John Mousel wrote: > Mark, > > I am using petsc-dev that I pulled after you made the changes for the non-symmetric discretization allocations last week. > I think the difference in our results comes from different convergence tolerances. I'm using an rtol of 1.D-012. It seems to be converging very nicely now. Good, > I think I dropped the option to set ksp and pc on levels after a bit, and that seems to have made the difference. GAMG should scale much better than HYPRE and ML right? > They both seem to work efficiently for really small core counts, but deteriorate with impressive speed as you go up the ladder. ML, should scale pretty similar to GAMG. The drop tolerance can effect complexity but if these are about the same the ML interface uses PETSc infrastructure. HYPRE is its own solver and its has been optimized for scalability. You do have to watch for complexity getting out of hand with all AMG solvers but they are more or less good scalable codes. Mark > > [0]PCSetUp_GAMG level 0 N=46330, n data rows=1, n data cols=1, nnz/row (ave)=6, np=4 > [0]scaleFilterGraph 75.5527% nnz after filtering, with threshold 0.05, 6.95957 nnz ave. > [0]maxIndSetAgg removed 0 of 46330 vertices. (0 local) > [0]PCGAMGprolongator_AGG New grid 5903 nodes > PCGAMGoptprol_AGG smooth P0: max eigen=1.923098e+00 min=3.858220e-02 PC=jacobi > [0]PCSetUp_GAMG 1) N=5903, n data cols=1, nnz/row (ave)=13, 4 active pes > [0]scaleFilterGraph 52.8421% nnz after filtering, with threshold 0.05, 13.3249 nnz ave. > [0]maxIndSetAgg removed 0 of 5903 vertices. (0 local) > [0]PCGAMGprolongator_AGG New grid 615 nodes > PCGAMGoptprol_AGG smooth P0: max eigen=1.575363e+00 min=2.167886e-03 PC=jacobi > [0]PCSetUp_GAMG 2) N=615, n data cols=1, nnz/row (ave)=21, 4 active pes > [0]scaleFilterGraph 24.7174% nnz after filtering, with threshold 0.05, 21.722 nnz ave. > [0]maxIndSetAgg removed 0 of 615 vertices. (0 local) > [0]PCGAMGprolongator_AGG New grid 91 nodes > PCGAMGoptprol_AGG smooth P0: max eigen=1.676442e+00 min=2.270745e-03 PC=jacobi > [0]createLevel aggregate processors: npe: 4 --> 1, neq=91 > [0]PCSetUp_GAMG 3) N=91, n data cols=1, nnz/row (ave)=37, 1 active pes > [0]scaleFilterGraph 16.4384% nnz after filtering, with threshold 0.05, 37.7033 nnz ave. > [0]maxIndSetAgg removed 0 of 91 vertices. (0 local) > [0]PCGAMGprolongator_AGG New grid 10 nodes > PCGAMGoptprol_AGG smooth P0: max eigen=1.538313e+00 min=8.923063e-04 PC=jacobi > [0]PCSetUp_GAMG 4) N=10, n data cols=1, nnz/row (ave)=10, 1 active pes > [0]PCSetUp_GAMG 5 levels, grid compexity = 1.29633 > Residual norms for pres_ solve. > 0 KSP preconditioned resid norm 4.680688832182e+06 true resid norm 2.621342052504e+03 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP preconditioned resid norm 1.728993898497e+04 true resid norm 2.888375221014e+03 ||r(i)||/||b|| 1.101868876004e+00 > 4 KSP preconditioned resid norm 4.510102902646e+02 true resid norm 5.677727287161e+01 ||r(i)||/||b|| 2.165962004744e-02 > 6 KSP preconditioned resid norm 3.959846836748e+01 true resid norm 1.973580779699e+00 ||r(i)||/||b|| 7.528894513455e-04 > 8 KSP preconditioned resid norm 3.175473803927e-01 true resid norm 4.315977395174e-02 ||r(i)||/||b|| 1.646476235732e-05 > 10 KSP preconditioned resid norm 7.502408552205e-04 true resid norm 1.016040400933e-04 ||r(i)||/||b|| 3.876031363257e-08 > 12 KSP preconditioned resid norm 2.868067261023e-06 true resid norm 1.194542164810e-06 ||r(i)||/||b|| 4.556986997056e-10 > KSP Object:(pres_) 4 MPI processes > type: bcgsl > BCGSL: Ell = 2 > BCGSL: Delta = 0 > maximum iterations=5000 > tolerances: relative=1e-12, absolute=1e-50, divergence=10000 > left preconditioning > has attached null space > using nonzero initial guess > using PRECONDITIONED norm type for convergence test > PC Object:(pres_) 4 MPI processes > type: gamg > MG: type is MULTIPLICATIVE, levels=5 cycles=v > Cycles per PCApply=1 > Using Galerkin computed coarse grid matrices > Coarse grid solver -- level ------------------------------- > KSP Object: (pres_mg_coarse_) 4 MPI processes > type: richardson > Richardson: damping factor=1 > maximum iterations=1, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (pres_mg_coarse_) 4 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 8, local iterations = 1, omega = 1 > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=10, cols=10 > total: nonzeros=100, allocated nonzeros=100 > total number of mallocs used during MatSetValues calls =0 > using I-node (on process 0) routines: found 2 nodes, limit used is 5 > Down solver (pre-smoother) on level 1 ------------------------------- > KSP Object: (pres_mg_levels_1_) 4 MPI processes > type: richardson > Richardson: damping factor=1 > maximum iterations=1 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using nonzero initial guess > using NONE norm type for convergence test > PC Object: (pres_mg_levels_1_) 4 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=91, cols=91 > total: nonzeros=3431, allocated nonzeros=3431 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Up solver (post-smoother) same as down solver (pre-smoother) > Down solver (pre-smoother) on level 2 ------------------------------- > KSP Object: (pres_mg_levels_2_) 4 MPI processes > type: richardson > Richardson: damping factor=1 > maximum iterations=1 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using nonzero initial guess > using NONE norm type for convergence test > PC Object: (pres_mg_levels_2_) 4 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=615, cols=615 > total: nonzeros=13359, allocated nonzeros=13359 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Up solver (post-smoother) same as down solver (pre-smoother) > Down solver (pre-smoother) on level 3 ------------------------------- > KSP Object: (pres_mg_levels_3_) 4 MPI processes > type: richardson > Richardson: damping factor=1 > maximum iterations=1 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using nonzero initial guess > using NONE norm type for convergence test > PC Object: (pres_mg_levels_3_) 4 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=5903, cols=5903 > total: nonzeros=78657, allocated nonzeros=78657 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Up solver (post-smoother) same as down solver (pre-smoother) > Down solver (pre-smoother) on level 4 ------------------------------- > KSP Object: (pres_mg_levels_4_) 4 MPI processes > type: richardson > Richardson: damping factor=1 > maximum iterations=1 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > has attached null space > using nonzero initial guess > using NONE norm type for convergence test > PC Object: (pres_mg_levels_4_) 4 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=46330, cols=46330 > total: nonzeros=322437, allocated nonzeros=615417 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Up solver (post-smoother) same as down solver (pre-smoother) > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=46330, cols=46330 > total: nonzeros=322437, allocated nonzeros=615417 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > > > > On Tue, Mar 20, 2012 at 2:21 PM, Mark F. Adams wrote: > John, > > I had dome diagonal scaling stuff in my input which seemed to mess things up. I don't understand that. With your hypre parameters I get > > Complexity: grid = 1.408828 > operator = 1.638900 > cycle = 3.277856 > > 0 KSP preconditioned resid norm 2.246209947341e+06 true resid norm 2.621342052504e+03 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP preconditioned resid norm 6.591054866442e+04 true resid norm 5.518411654910e+03 ||r(i)||/||b|| 2.105185643224e+00 > 4 KSP preconditioned resid norm 2.721184454964e+03 true resid norm 1.937153214559e+03 ||r(i)||/||b|| 7.389929188022e-01 > 6 KSP preconditioned resid norm 2.942012838854e+02 true resid norm 5.614763956317e+01 ||r(i)||/||b|| 2.141942502679e-02 > 8 KSP preconditioned resid norm 2.143421596353e+01 true resid norm 5.306843482279e+00 ||r(i)||/||b|| 2.024475774617e-03 > 10 KSP preconditioned resid norm 3.689048280659e+00 true resid norm 2.482945300243e-01 ||r(i)||/||b|| 9.472038560826e-05 > Linear solve converged due to CONVERGED_RTOL iterations 10 > > with ML I get 18 iterations but if I add -pc_ml_Threshold .01 I get it to 12: > > -@${MPIEXEC} -n 1 ./ex10 -f ./binaryoutput -pc_type ml -ksp_type bcgsl -pc_gamg_coarse_eq_limit 10 -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_monitor_true_residual -pc_gamg_verbose 2 -ksp_converged_reason -options_left -mg_levels_ksp_type richardson -mg_levels_pc_type sor -pc_ml_maxNlevels 5 -pc_ml_Threshold .01 > > 0 KSP preconditioned resid norm 1.987800354481e+06 true resid norm 2.621342052504e+03 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP preconditioned resid norm 4.845840795806e+04 true resid norm 9.664923970856e+03 ||r(i)||/||b|| 3.687013666005e+00 > 4 KSP preconditioned resid norm 4.086337251141e+03 true resid norm 1.111442892542e+03 ||r(i)||/||b|| 4.239976585582e-01 > 6 KSP preconditioned resid norm 1.496117919395e+03 true resid norm 4.243682354730e+02 ||r(i)||/||b|| 1.618896835946e-01 > 8 KSP preconditioned resid norm 1.019912311314e+02 true resid norm 6.165476121107e+01 ||r(i)||/||b|| 2.352030371320e-02 > 10 KSP preconditioned resid norm 1.282179114927e+01 true resid norm 4.434755525096e+00 ||r(i)||/||b|| 1.691788189512e-03 > 12 KSP preconditioned resid norm 2.801790417375e+00 true resid norm 4.876299030996e-01 ||r(i)||/||b|| 1.860229963632e-04 > Linear solve converged due to CONVERGED_RTOL iterations 12 > > and gamg: > > -@${MPIEXEC} -n 1 ./ex10 -f ./binaryoutput -pc_type gamg -ksp_type bcgsl -pc_gamg_coarse_eq_limit 10 -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_monitor_true_residual -pc_gamg_verbose 2 -ksp_converged_reason -options_left -mg_levels_ksp_type richardson -mg_levels_pc_type sor > > [0]PCSetUp_GAMG 5 levels, grid compexity = 1.2916 > 0 KSP preconditioned resid norm 6.288750978813e+06 true resid norm 2.621342052504e+03 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP preconditioned resid norm 3.009668424006e+04 true resid norm 4.394363256786e+02 ||r(i)||/||b|| 1.676379186222e-01 > 4 KSP preconditioned resid norm 2.079756553216e+01 true resid norm 5.094584609440e+00 ||r(i)||/||b|| 1.943502414946e-03 > 6 KSP preconditioned resid norm 4.323447593442e+00 true resid norm 3.146656048880e-01 ||r(i)||/||b|| 1.200398874261e-04 > Linear solve converged due to CONVERGED_RTOL iterations 6 > > So this looks pretty different from what you are getting. Is your code hardwiring anything? Can you reproduce my results with ksp ex10.c? > > Actually, I just realized that I am using petsc-dev. What version of PETSc are you using? > > Also, here is the makefile that I use to run this jobs: > > ALL: runex10 > > include ${PETSC_DIR}/conf/variables > include ${PETSC_DIR}/conf/rules > > runex10: > -@${MPIEXEC} -n 1 ./ex10 -f ./binaryoutput -pc_type gamg -ksp_type bcgsl -pc_gamg_coarse_eq_limit 10 -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_monitor_true_residual -pc_gamg_verbose 2 -ksp_converged_reason -options_left -mg_levels_ksp_type richardson -mg_levels_pc_type sor -pc_ml_maxNlevels 5 -pc_ml_Threshold .01 -pc_hypre_boomeramg_relax_type_coarse symmetric-SOR/Jacobi -pc_hypre_boomeramg_grid_sweeps_coarse 4 -pc_hypre_boomeramg_coarsen_type PMIS > > You just need to run 'make ex10' and then 'make -f this-file'. > > Mark > > On Mar 20, 2012, at 2:45 PM, John Mousel wrote: > >> Mark, >> >> I run ML with the following options. >> >> -ksp_type bcgsl -pc_type ml -pc_ml_maxNlevels 5 -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_monitor -ksp_view >> >> Note the lack of scaling. For some reason scaling seems to mess with ML. >> As you can see below, ML converges very nicely. >> >> With regards to HYPRE, this one took a bit of work to get convergence. The options that work are: >> >> -ksp_type bcgsl -pc_type hypre -pc_hypre_type boomeramg -ksp_monitor_true_residual -pc_hypre_boomeramg_relax_type_coarse symmetric-SOR/Jacobi -pc_hypre_boomeramg_grid_sweeps_coarse 4 -pc_hypre_boomeramg_coarsen_type PMIS >> >> The problem is that neither of ML or HYPRE seem to scale at all. >> >> ML output: >> 0 KSP preconditioned resid norm 1.538968715109e+06 true resid norm 2.621342052504e+03 ||r(i)||/||b|| 1.000000000000e+00 >> 2 KSP preconditioned resid norm 1.263129058693e+05 true resid norm 1.096298699671e+04 ||r(i)||/||b|| 4.182203915830e+00 >> 4 KSP preconditioned resid norm 2.277379585186e+04 true resid norm 2.999721659930e+03 ||r(i)||/||b|| 1.144345758717e+00 >> 6 KSP preconditioned resid norm 4.766504457975e+03 true resid norm 6.733421603796e+02 ||r(i)||/||b|| 2.568692474667e-01 >> 8 KSP preconditioned resid norm 2.139020425406e+03 true resid norm 1.360842101250e+02 ||r(i)||/||b|| 5.191394613876e-02 >> 10 KSP preconditioned resid norm 6.621380459944e+02 true resid norm 1.522758800025e+02 ||r(i)||/||b|| 5.809080881188e-02 >> 12 KSP preconditioned resid norm 2.973409610262e+01 true resid norm 1.161046206089e+01 ||r(i)||/||b|| 4.429205280479e-03 >> 14 KSP preconditioned resid norm 2.532665482573e+00 true resid norm 2.557425874623e+00 ||r(i)||/||b|| 9.756170020543e-04 >> 16 KSP preconditioned resid norm 2.375585214826e+00 true resid norm 2.441783841415e+00 ||r(i)||/||b|| 9.315014189327e-04 >> 18 KSP preconditioned resid norm 1.436338060675e-02 true resid norm 1.305304828818e-02 ||r(i)||/||b|| 4.979528816437e-06 >> 20 KSP preconditioned resid norm 4.088293864561e-03 true resid norm 9.841243465634e-04 ||r(i)||/||b|| 3.754276728683e-07 >> 22 KSP preconditioned resid norm 6.140822977383e-04 true resid norm 1.682184150207e-04 ||r(i)||/||b|| 6.417263052718e-08 >> 24 KSP preconditioned resid norm 2.685415483430e-05 true resid norm 1.065041542336e-05 ||r(i)||/||b|| 4.062962867890e-09 >> 26 KSP preconditioned resid norm 1.620776166579e-06 true resid norm 9.563268703474e-07 ||r(i)||/||b|| 3.648233809982e-10 >> 28 KSP preconditioned resid norm 2.823291105652e-07 true resid norm 7.705418741178e-08 ||r(i)||/||b|| 2.939493811507e-11 >> KSP Object:(pres_) 4 MPI processes >> type: bcgsl >> BCGSL: Ell = 2 >> BCGSL: Delta = 0 >> maximum iterations=5000 >> tolerances: relative=1e-12, absolute=1e-50, divergence=10000 >> left preconditioning >> has attached null space >> using nonzero initial guess >> using PRECONDITIONED norm type for convergence test >> PC Object:(pres_) 4 MPI processes >> type: ml >> MG: type is MULTIPLICATIVE, levels=5 cycles=v >> Cycles per PCApply=1 >> Using Galerkin computed coarse grid matrices >> Coarse grid solver -- level ------------------------------- >> KSP Object: (pres_mg_coarse_) 4 MPI processes >> type: richardson >> Richardson: damping factor=1 >> maximum iterations=1, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> using PRECONDITIONED norm type for convergence test >> PC Object: (pres_mg_coarse_) 4 MPI processes >> type: sor >> SOR: type = local_symmetric, iterations = 8, local iterations = 1, omega = 1 >> linear system matrix = precond matrix: >> Matrix Object: 4 MPI processes >> type: mpiaij >> rows=4, cols=4 >> total: nonzeros=16, allocated nonzeros=16 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node (on process 0) routines >> Down solver (pre-smoother) on level 1 ------------------------------- >> KSP Object: (pres_mg_levels_1_) 4 MPI processes >> type: richardson >> Richardson: damping factor=1 >> maximum iterations=1 >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> using nonzero initial guess >> using PRECONDITIONED norm type for convergence test >> PC Object: (pres_mg_levels_1_) 4 MPI processes >> type: sor >> SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 >> linear system matrix = precond matrix: >> Matrix Object: 4 MPI processes >> type: mpiaij >> rows=25, cols=25 >> total: nonzeros=303, allocated nonzeros=303 >> total number of mallocs used during MatSetValues calls =0 >> using I-node (on process 0) routines: found 4 nodes, limit used is 5 >> Up solver (post-smoother) same as down solver (pre-smoother) >> Down solver (pre-smoother) on level 2 ------------------------------- >> KSP Object: (pres_mg_levels_2_) 4 MPI processes >> type: richardson >> Richardson: damping factor=1 >> maximum iterations=1 >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> using nonzero initial guess >> using PRECONDITIONED norm type for convergence test >> PC Object: (pres_mg_levels_2_) 4 MPI processes >> type: sor >> SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 >> linear system matrix = precond matrix: >> Matrix Object: 4 MPI processes >> type: mpiaij >> rows=423, cols=423 >> total: nonzeros=7437, allocated nonzeros=7437 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node (on process 0) routines >> Up solver (post-smoother) same as down solver (pre-smoother) >> Down solver (pre-smoother) on level 3 ------------------------------- >> KSP Object: (pres_mg_levels_3_) 4 MPI processes >> type: richardson >> Richardson: damping factor=1 >> maximum iterations=1 >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> using nonzero initial guess >> using PRECONDITIONED norm type for convergence test >> PC Object: (pres_mg_levels_3_) 4 MPI processes >> type: sor >> SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 >> linear system matrix = precond matrix: >> Matrix Object: 4 MPI processes >> type: mpiaij >> rows=6617, cols=6617 >> total: nonzeros=88653, allocated nonzeros=88653 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node (on process 0) routines >> Up solver (post-smoother) same as down solver (pre-smoother) >> Down solver (pre-smoother) on level 4 ------------------------------- >> KSP Object: (pres_mg_levels_4_) 4 MPI processes >> type: richardson >> Richardson: damping factor=1 >> maximum iterations=1 >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> has attached null space >> using nonzero initial guess >> using PRECONDITIONED norm type for convergence test >> PC Object: (pres_mg_levels_4_) 4 MPI processes >> type: sor >> SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 >> linear system matrix = precond matrix: >> Matrix Object: 4 MPI processes >> type: mpiaij >> rows=46330, cols=46330 >> total: nonzeros=322437, allocated nonzeros=615417 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node (on process 0) routines >> Up solver (post-smoother) same as down solver (pre-smoother) >> linear system matrix = precond matrix: >> Matrix Object: 4 MPI processes >> type: mpiaij >> rows=46330, cols=46330 >> total: nonzeros=322437, allocated nonzeros=615417 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node (on process 0) routines >> >> >> >> John >> >> >> >> On Tue, Mar 20, 2012 at 1:33 PM, Mark F. Adams wrote: >> John, >> >> I am getting poor results (diverging) from ML also: >> >> 0 KSP preconditioned resid norm 3.699832960909e+22 true resid norm 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 >> 2 KSP preconditioned resid norm 5.706378365783e+11 true resid norm 1.563902233018e+03 ||r(i)||/||b|| 1.193204539995e+00 >> 4 KSP preconditioned resid norm 5.570291685152e+11 true resid norm 1.564235542744e+03 ||r(i)||/||b|| 1.193458844050e+00 >> 6 KSP preconditioned resid norm 5.202150407298e+10 true resid norm 1.749929789082e+03 ||r(i)||/||b|| 1.335137277077e+00 >> Linear solve converged due to CONVERGED_RTOL iterations 6 >> >> With GAMG I get: >> >> 0 KSP preconditioned resid norm 7.731260075891e+06 true resid norm 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 >> 2 KSP preconditioned resid norm 2.856415184685e+05 true resid norm 1.310410242531e+03 ||r(i)||/||b|| 9.997987199150e-01 >> 4 KSP preconditioned resid norm 1.528467019258e+05 true resid norm 1.284856538976e+03 ||r(i)||/||b|| 9.803021078816e-01 >> 6 KSP preconditioned resid norm 1.451091957899e+05 true resid norm 1.564309254168e+03 ||r(i)||/||b|| 1.193515083375e+00 >> >> >> >> 122 KSP preconditioned resid norm 2.486245341783e+01 true resid norm 1.404397185367e+00 ||r(i)||/||b|| 1.071507580306e-03 >> 124 KSP preconditioned resid norm 1.482316853621e+01 true resid norm 4.488661881759e-01 ||r(i)||/||b|| 3.424697287811e-04 >> 126 KSP preconditioned resid norm 1.481941150253e+01 true resid norm 4.484480100832e-01 ||r(i)||/||b|| 3.421506730318e-04 >> 128 KSP preconditioned resid norm 8.191887347033e+00 true resid norm 6.678630367218e-01 ||r(i)||/||b|| 5.095569215816e-04 >> >> And HYPRE: >> >> 0 KSP preconditioned resid norm 3.774510769907e+04 true resid norm 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 >> 2 KSP preconditioned resid norm 1.843165835831e+04 true resid norm 8.502433792869e+02 ||r(i)||/||b|| 6.487069580482e-01 >> 4 KSP preconditioned resid norm 1.573176624705e+04 true resid norm 1.167264367302e+03 ||r(i)||/||b|| 8.905832558033e-01 >> 6 KSP preconditioned resid norm 1.657958380765e+04 true resid norm 8.684701624902e+02 ||r(i)||/||b|| 6.626133775216e-01 >> 8 KSP preconditioned resid norm 2.190304455083e+04 true resid norm 6.969893263600e+02 ||r(i)||/||b|| 5.317792960344e-01 >> 10 KSP preconditioned resid norm 2.485714630000e+04 true resid norm 6.642641436830e+02 ||r(i)||/||b|| 5.068110878446e-01 >> >> >> >> 62 KSP preconditioned resid norm 6.432516040957e+00 true resid norm 2.124960171419e-01 ||r(i)||/||b|| 1.621272781837e-04 >> 64 KSP preconditioned resid norm 5.731033176541e+00 true resid norm 1.338816774003e-01 ||r(i)||/||b|| 1.021471943216e-04 >> 66 KSP preconditioned resid norm 1.600285935522e-01 true resid norm 3.352408932031e-03 ||r(i)||/||b|| 2.557774695353e-06 >> >> ML and GAMG should act similarly, but ML seems to have a problem (see the preconditioned norm difference and its diverging). ML has a parameter: >> >> -pc_ml_Threshold [.0] >> >> I setting this to 0.05 (GAMG default) helps a bit but it still diverges. >> >> So it would be nice to figure out the difference between ML and GAMG, but that is secondary for you as the both suck. >> >> HYPRE is a very different algorithm. It looks like the smoothing in GAMG (and ML) may be the problem. If I turn smoothing off (-pc_gamg_agg_nsmooths 0) and I get for GAMG: >> >> 0 KSP preconditioned resid norm 2.186148437534e+05 true resid norm 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 >> 2 KSP preconditioned resid norm 2.916843959765e+04 true resid norm 3.221533667508e+03 ||r(i)||/||b|| 2.457921292432e+00 >> 4 KSP preconditioned resid norm 2.396374655925e+04 true resid norm 1.834299897412e+03 ||r(i)||/||b|| 1.399508817812e+00 >> 6 KSP preconditioned resid norm 2.509576275453e+04 true resid norm 1.035475461174e+03 ||r(i)||/||b|| 7.900327752214e-01 >> >> >> >> 64 KSP preconditioned resid norm 1.973859758284e+01 true resid norm 7.322674977169e+00 ||r(i)||/||b|| 5.586953482895e-03 >> 66 KSP preconditioned resid norm 3.371598890438e+01 true resid norm 7.152754930495e+00 ||r(i)||/||b|| 5.457310231004e-03 >> 68 KSP preconditioned resid norm 4.687839294418e+00 true resid norm 4.605726307025e-01 ||r(i)||/||b|| 3.514013487219e-04 >> 70 KSP preconditioned resid norm 1.487545519493e+00 true resid norm 1.558723789416e-01 ||r(i)||/||b|| 1.189253562571e-04 >> 72 KSP preconditioned resid norm 5.317329808718e-01 true resid norm 5.027178038177e-02 ||r(i)||/||b|| 3.835566911967e-05 >> 74 KSP preconditioned resid norm 3.405339702462e-01 true resid norm 1.897059263835e-02 ||r(i)||/||b|| 1.447392092969e-05 >> >> This is almost as good as HYPRE. >> >> An other thing to keep in mind is the cost of each iteration, not just then number of iterations, You can use -pc_hypre_boomeramg_print_statistics to get some data on this from HYPRE: >> >> Average Convergence Factor = 0.537664 >> >> Complexity: grid = 1.780207 >> operator = 2.624910 >> cycle = 5.249670 >> >> And GAMG prints this with verbose set: >> >> [0]PCSetUp_GAMG 6 levels, grid compexity [sic] = 1.1316 >> >> I believe that the hypre "Complexity: grid" is the same as my "grid complexity". So hypre actually looks more expensive at this point. >> >> I've worked on optimizing parameters for hypre with the hypre people and here are a set of arguments that I've used: >> >> -pc_hypre_boomeramg_no_CF -pc_hypre_boomeramg_agg_nl 1 -pc_hypre_boomeramg_coarsen_type HMIS -pc_hypre_boomeramg_interp_type ext+i -pc_hypre_boomeramg_P_max 4 -pc_hypre_boomeramg_agg_num_paths 2 >> >> With these parmeters I get: >> >> Complexity: grid = 1.244140 >> operator = 1.396722 >> cycle = 2.793442 >> >> and: >> >> 0 KSP preconditioned resid norm 4.698624821403e+04 true resid norm 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 >> 2 KSP preconditioned resid norm 2.207967626172e+04 true resid norm 3.466160021150e+03 ||r(i)||/||b|| 2.644562931280e+00 >> 4 KSP preconditioned resid norm 2.278468320876e+04 true resid norm 1.246784122467e+03 ||r(i)||/||b|| 9.512541410282e-01 >> >> >> >> 56 KSP preconditioned resid norm 1.108460232262e+00 true resid norm 8.276869475681e-02 ||r(i)||/||b|| 6.314971631105e-05 >> 58 KSP preconditioned resid norm 3.617217454336e-01 true resid norm 3.764556404754e-02 ||r(i)||/||b|| 2.872229285428e-05 >> 60 KSP preconditioned resid norm 1.666532560770e-01 true resid norm 5.149302513338e-03 ||r(i)||/||b|| 3.928743758404e-06 >> Linear solve converged due to CONVERGED_RTOL iterations 60 >> >> So this actually converged faster with lower complexity. >> >> Anyway these result seem different from what you are getting, so I've appended my options. This uses ex10 in the KSP tutorials to read in your binary file. >> >> Mark >> >> #PETSc Option Table entries: >> -f ./binaryoutput >> -ksp_converged_reason >> -ksp_diagonal_scale >> -ksp_diagonal_scale_fix >> -ksp_monitor_true_residual >> -ksp_type bcgsl >> -mg_coarse_ksp_type richardson >> -mg_coarse_pc_sor_its 8 >> -mg_coarse_pc_type sor >> -mg_levels_ksp_type richardson >> -mg_levels_pc_type sor >> -options_left >> -pc_gamg_agg_nsmooths 0 >> -pc_gamg_coarse_eq_limit 10 >> -pc_gamg_sym_graph >> -pc_gamg_verbose 2 >> -pc_hypre_boomeramg_P_max 4 >> -pc_hypre_boomeramg_agg_nl 1 >> -pc_hypre_boomeramg_agg_num_paths 2 >> -pc_hypre_boomeramg_coarsen_type HMIS >> -pc_hypre_boomeramg_interp_type ext+i >> -pc_hypre_boomeramg_no_CF >> -pc_ml_Threshold .01 >> -pc_type gamg >> -vecload_block_size 1 >> #End of PETSc Option Table entries >> There are 7 unused database options. They are: >> Option left: name:-pc_hypre_boomeramg_P_max value: 4 >> Option left: name:-pc_hypre_boomeramg_agg_nl value: 1 >> Option left: name:-pc_hypre_boomeramg_agg_num_paths value: 2 >> Option left: name:-pc_hypre_boomeramg_coarsen_type value: HMIS >> Option left: name:-pc_hypre_boomeramg_interp_type value: ext+i >> Option left: name:-pc_hypre_boomeramg_no_CF no value >> Option left: name:-pc_ml_Threshold value: .01 >> >> >> On Mar 20, 2012, at 10:19 AM, John Mousel wrote: >> >>> Mark, >>> >>> Sorry for the late reply. I've been on travel and hadn't had a chance to pick this back up. I've tried running with the suggested options: >>> >>> -ksp_type bcgsl -pc_type gamg -pc_gamg_coarse_eq_limit 10 -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_diagonal_scale -ksp_diagonal_scale_fix -ksp_monitor_true_residual -ksp_view -pc_gamg_verbose 1 >>> >>> With these options, the convergence starts to hang (see attached GAMG_kspview.txt). The hanging happens for both -mg_coarse_ksp_type richardson and preonly. It was my understanding from previous emails that using preonly made it so that only the preconditioner was run, which in this case would be 8 sweeps of SOR. If I get rid of the -pc_gamg_agg_nsmooths 1 (see GAMG_kspview_nosmooth.txt), the problem converges, but again the convergence is slow. Without this option, both Richardson and preonly converge in 172 iterations. >>> >>> Matt, I've checked and the problem does converge in the true residual using GAMG, ML, HYPRE, and ILU preconditioned BiCG. I explicitly ensure that a solution exists by projecting the rhs vector out of the nullity of the transpose of operator. >>> >>> John >>> >>> >>> On Fri, Mar 16, 2012 at 2:04 PM, Mark F. Adams wrote: >>> John, did this get resolved? >>> Mark >>> >>> On Mar 15, 2012, at 4:24 PM, John Mousel wrote: >>> >>>> Mark, >>>> >>>> Running without the options you mentioned before leads to slightly worse performance (175 iterations). >>>> I have not been able to get run coarse grid solve to work with LU while running ML. It keeps experiencing a zero pivot, and all the combinations of shifting i've tried haven't lead me anywhere, hence the SOR on the course grid. Also, the ML manual suggests limiting the number of levels to 3 or 4 and performing a few sweeps of an iterative method as opposed to a direct solve. >>>> >>>> John >>>> >>>> On Thu, Mar 15, 2012 at 12:04 PM, Mark F. Adams wrote: >>>> You also want: -pc_gamg_agg_nsmooths 1 >>>> >>>> You are running plain aggregation. If it is Poisson then smoothing is good. >>>> >>>> Is this problem singular? Can you try running ML with these parameters and see if its performance degrades? The ML implementation uses the PETSC infrastructure and uses a very similar algorithm to GAMG-SA. We should be able to get these two to match pretty well. >>>> >>>> Mark >>>> >>>> >>>> On Mar 15, 2012, at 12:21 PM, John Mousel wrote: >>>> >>>>> Mark, >>>>> >>>>> I ran with those options removed (see the run options listed below). Things actually got slightly worse. Now it's up to 142 iterations. I have attached the ksp_view output. >>>>> >>>>> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale -ksp_diagonal_scale_fix -mg_levels_ksp_type richardson -mg_levels_pc_type sor -pc_gamg_verbose 1 >>>>> >>>>> >>>>> John >>>>> >>>>> >>>>> On Thu, Mar 15, 2012 at 10:55 AM, Mark F. Adams wrote: >>>>> John, can you run again with: -pc_gamg_verbose 1 >>>>> >>>>> And I would not use: -pc_mg_levels 4 -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 >>>>> >>>>> 1) I think -mg_coarse_ksp_type preonly and -mg_coarse_pc_sor_its 8 do not do what you think. I think this is the same as 1 iteration. I think you want 'richardson' not 'preonly'. >>>>> >>>>> 2) Why are you using sor as the coarse solver? If your problem is singular then you want to use as many levels as possible to get the coarse grid to be tiny. I'm pretty sure HYPRE ignores the coarse solver parameters. But ML uses them and it is converging well. >>>>> >>>>> 3) I would not specify the number of levels. GAMG, and I think the rest, have internal logic for stopping a the right level. If the coarse level is large and you use just 8 iterations of sor then convergence will suffer. >>>>> >>>>> Mark >>>>> >>>>> On Mar 15, 2012, at 11:13 AM, John Mousel wrote: >>>>> >>>>>> Mark, >>>>>> >>>>>> The changes pulled through this morning. I've run it with the options >>>>>> >>>>>> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale -ksp_diagonal_scale_fix -pc_mg_levels 4 -mg_levels_ksp_type richardson -mg_levels_pc_type sor -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 >>>>>> >>>>>> and it converges in the true residual, but it's not converging as fast as anticpated. The matrix arises from a non-symmetric discretization of the Poisson equation. The solve takes GAMG 114 iterations, whereas ML takes 24 iterations, BoomerAMG takes 22 iterations, and -ksp_type bcgsl -pc_type bjacobi -sub_pc_type ilu -sub_pc_factor_levels 4 takes around 170. I've attached the -ksp_view results for ML,GAMG, and HYPRE. I've attempted to make all the options the same on all levels for ML and GAMG. >>>>>> >>>>>> Any thoughts? >>>>>> >>>>>> John >>>>>> >>>>>> >>>>>> On Wed, Mar 14, 2012 at 6:04 PM, Mark F. Adams wrote: >>>>>> Humm, I see it with hg view (appended). >>>>>> >>>>>> Satish, my main repo looks hosed. I see this: >>>>>> >>>>>> ~/Codes/petsc-dev>hg update >>>>>> abort: crosses branches (merge branches or use --clean to discard changes) >>>>>> ~/Codes/petsc-dev>hg merge >>>>>> abort: branch 'default' has 3 heads - please merge with an explicit rev >>>>>> (run 'hg heads .' to see heads) >>>>>> ~/Codes/petsc-dev>hg heads >>>>>> changeset: 22496:8e2a98268179 >>>>>> tag: tip >>>>>> user: Barry Smith >>>>>> date: Wed Mar 14 16:42:25 2012 -0500 >>>>>> files: src/vec/is/interface/f90-custom/zindexf90.c src/vec/vec/interface/f90-custom/zvectorf90.c >>>>>> description: >>>>>> undoing manually changes I put in because Satish had a better fix >>>>>> >>>>>> >>>>>> changeset: 22492:bda4df63072d >>>>>> user: Mark F. Adams >>>>>> date: Wed Mar 14 17:39:52 2012 -0400 >>>>>> files: src/ksp/pc/impls/gamg/tools.c >>>>>> description: >>>>>> fix for unsymmetric matrices. >>>>>> >>>>>> >>>>>> changeset: 22469:b063baf366e4 >>>>>> user: Mark F. Adams >>>>>> date: Wed Mar 14 14:22:28 2012 -0400 >>>>>> files: src/ksp/pc/impls/gamg/tools.c >>>>>> description: >>>>>> added fix for preallocation for unsymetric matrices. >>>>>> >>>>>> Mark >>>>>> >>>>>> my 'hg view' on my merge repo: >>>>>> >>>>>> Revision: 22492 >>>>>> Branch: default >>>>>> Author: Mark F. Adams 2012-03-14 17:39:52 >>>>>> Committer: Mark F. Adams 2012-03-14 17:39:52 >>>>>> Tags: tip >>>>>> Parent: 22491:451bbbd291c2 (Small fixes to the BT linesearch) >>>>>> >>>>>> fix for unsymmetric matrices. >>>>>> >>>>>> >>>>>> ------------------------ src/ksp/pc/impls/gamg/tools.c ------------------------ >>>>>> @@ -103,7 +103,7 @@ >>>>>> PetscErrorCode ierr; >>>>>> PetscInt Istart,Iend,Ii,jj,ncols,nnz0,nnz1, NN, MM, nloc; >>>>>> PetscMPIInt mype, npe; >>>>>> - Mat Gmat = *a_Gmat, tGmat; >>>>>> + Mat Gmat = *a_Gmat, tGmat, matTrans; >>>>>> MPI_Comm wcomm = ((PetscObject)Gmat)->comm; >>>>>> const PetscScalar *vals; >>>>>> const PetscInt *idx; >>>>>> @@ -127,6 +127,10 @@ >>>>>> ierr = MatDiagonalScale( Gmat, diag, diag ); CHKERRQ(ierr); >>>>>> ierr = VecDestroy( &diag ); CHKERRQ(ierr); >>>>>> >>>>>> + if( symm ) { >>>>>> + ierr = MatTranspose( Gmat, MAT_INITIAL_MATRIX, &matTrans ); CHKERRQ(ierr); >>>>>> + } >>>>>> + >>>>>> /* filter - dup zeros out matrix */ >>>>>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &d_nnz ); CHKERRQ(ierr); >>>>>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &o_nnz ); CHKERRQ(ierr); >>>>>> @@ -135,6 +139,12 @@ >>>>>> d_nnz[jj] = ncols; >>>>>> o_nnz[jj] = ncols; >>>>>> ierr = MatRestoreRow(Gmat,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >>>>>> + if( symm ) { >>>>>> + ierr = MatGetRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >>>>>> + d_nnz[jj] += ncols; >>>>>> + o_nnz[jj] += ncols; >>>>>> + ierr = MatRestoreRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >>>>>> + } >>>>>> if( d_nnz[jj] > nloc ) d_nnz[jj] = nloc; >>>>>> if( o_nnz[jj] > (MM-nloc) ) o_nnz[jj] = MM - nloc; >>>>>> } >>>>>> @@ -142,6 +152,9 @@ >>>>>> CHKERRQ(ierr); >>>>>> ierr = PetscFree( d_nnz ); CHKERRQ(ierr); >>>>>> ierr = PetscFree( o_nnz ); CHKERRQ(ierr); >>>>>> + if( symm ) { >>>>>> + ierr = MatDestroy( &matTrans ); CHKERRQ(ierr); >>>>>> + } >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mar 14, 2012, at 5:53 PM, John Mousel wrote: >>>>>> >>>>>>> Mark, >>>>>>> >>>>>>> No change. Can you give me the location that you patched so I can check to make sure it pulled? >>>>>>> I don't see it on the petsc-dev change log. >>>>>>> >>>>>>> John >>>>>>> >>>>>>> On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams wrote: >>>>>>> John, I've committed these changes, give a try. >>>>>>> >>>>>>> Mark >>>>>>> >>>>>>> On Mar 14, 2012, at 3:46 PM, Satish Balay wrote: >>>>>>> >>>>>>> > This is the usual merge [with uncommited changes] issue. >>>>>>> > >>>>>>> > You could use 'hg shelf' extension to shelve your local changes and >>>>>>> > then do a merge [as Sean would suggest] - or do the merge in a >>>>>>> > separate/clean clone [I normally do this..] >>>>>>> > >>>>>>> > i.e >>>>>>> > cd ~/Codes >>>>>>> > hg clone petsc-dev petsc-dev-merge >>>>>>> > cd petsc-dev-merge >>>>>>> > hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just to be sure, look for latest chagnes before merge.. >>>>>>> > hg merge >>>>>>> > hg commit >>>>>>> > hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev >>>>>>> > >>>>>>> > [now update your petsc-dev to latest] >>>>>>> > cd ~/Codes/petsc-dev >>>>>>> > hg pull >>>>>>> > hg update >>>>>>> > >>>>>>> > Satish >>>>>>> > >>>>>>> > On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>>>>> > >>>>>>> >> Great, that seems to work. >>>>>>> >> >>>>>>> >> I did a 'hg commit tools.c' >>>>>>> >> >>>>>>> >> and I want to push this file only. I guess its the only thing in the change set so 'hg push' should be fine. But I see this: >>>>>>> >> >>>>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update >>>>>>> >> abort: crosses branches (merge branches or use --clean to discard changes) >>>>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge >>>>>>> >> abort: outstanding uncommitted changes (use 'hg status' to list changes) >>>>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status >>>>>>> >> M include/petscmat.h >>>>>>> >> M include/private/matimpl.h >>>>>>> >> M src/ksp/pc/impls/gamg/agg.c >>>>>>> >> M src/ksp/pc/impls/gamg/gamg.c >>>>>>> >> M src/ksp/pc/impls/gamg/gamg.h >>>>>>> >> M src/ksp/pc/impls/gamg/geo.c >>>>>>> >> M src/mat/coarsen/coarsen.c >>>>>>> >> M src/mat/coarsen/impls/hem/hem.c >>>>>>> >> M src/mat/coarsen/impls/mis/mis.c >>>>>>> >> >>>>>>> >> Am I ready to do a push? >>>>>>> >> >>>>>>> >> Thanks, >>>>>>> >> Mark >>>>>>> >> >>>>>>> >> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: >>>>>>> >> >>>>>>> >>> If commit is the last hg operation that you've done - then 'hg rollback' would undo this commit. >>>>>>> >>> >>>>>>> >>> Satish >>>>>>> >>> >>>>>>> >>> On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>>>>> >>> >>>>>>> >>>> Damn, I'm not preallocating the graph perfectly for unsymmetric matrices and PETSc now dies on this. >>>>>>> >>>> >>>>>>> >>>> I have a fix but I committed it with other changes that I do not want to commit. The changes are all in one file so I should be able to just commit this file. >>>>>>> >>>> >>>>>>> >>>> Anyone know how to delete a commit? >>>>>>> >>>> >>>>>>> >>>> I've tried: >>>>>>> >>>> >>>>>>> >>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f >>>>>>> >>>> hg: unknown command 'strip' >>>>>>> >>>> 'strip' is provided by the following extension: >>>>>>> >>>> >>>>>>> >>>> mq manage a stack of patches >>>>>>> >>>> >>>>>>> >>>> use "hg help extensions" for information on enabling extensions >>>>>>> >>>> >>>>>>> >>>> But have not figured out how to load extensions. >>>>>>> >>>> >>>>>>> >>>> Mark >>>>>>> >>>> >>>>>>> >>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: >>>>>>> >>>> >>>>>>> >>>>> Mark, >>>>>>> >>>>> >>>>>>> >>>>> I have a non-symmetric matrix. I am running with the following options. >>>>>>> >>>>> >>>>>>> >>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual >>>>>>> >>>>> >>>>>>> >>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new malloc error: >>>>>>> >>>>> >>>>>>> >>>>> >>>>>>> >>>>> 0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>>> >>>>> [0]PETSC ERROR: Argument out of range! >>>>>>> >>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! >>>>>>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>> >>>>> [0]PETSC ERROR: Petsc Development HG revision: 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 -0500 >>>>>>> >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>> >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>> >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>>>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>> >>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 >>>>>>> >>>>> [0]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib >>>>>>> >>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 >>>>>>> >>>>> [0]PETSC ERROR: Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug >>>>>>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>> >>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >>>>>>> >>>>> [0]PETSC ERROR: MatSetValues() line 1141 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c >>>>>>> >>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c >>>>>>> >>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c >>>>>>> >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>>>>>> >>>>> [0]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c >>>>>>> >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>>>> >>>>> [0]PETSC ERROR: KSPSolve() line 385 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>>>> >>>>> >>>>>>> >>>>> >>>>>>> >>>>> John >>>>>>> >>>>> >>>>>>> >>>>> >>>>>>> >>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams wrote: >>>>>>> >>>>> >>>>>>> >>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: >>>>>>> >>>>> >>>>>>> >>>>>> Mark, >>>>>>> >>>>>> >>>>>>> >>>>>> The matrix is asymmetric. Does this require the setting of an option? >>>>>>> >>>>> >>>>>>> >>>>> Yes: -pc_gamg_sym_graph >>>>>>> >>>>> >>>>>>> >>>>> Mark >>>>>>> >>>>> >>>>>>> >>>>>> I pulled petsc-dev this morning, so I should have (at least close to) the latest code. >>>>>>> >>>>>> >>>>>>> >>>>>> John >>>>>>> >>>>>> >>>>>>> >>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams wrote: >>>>>>> >>>>>> >>>>>>> >>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >>>>>>> >>>>>> >>>>>>> >>>>>>> I'm getting the following error when using GAMG. >>>>>>> >>>>>>> >>>>>>> >>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion `sgid==-1' failed. >>>>>>> >>>>>> >>>>>>> >>>>>> Is it possible that your matrix is structurally asymmetric? >>>>>>> >>>>>> >>>>>>> >>>>>> This code is evolving fast and so you will need to move to the dev version if you are not already using it. (I think I fixed a bug that hit this assert). >>>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> When I try to alter the type of aggregation at the command line using -pc_gamg_type pa, I'm getting >>>>>>> >>>>>>> >>>>>>> >>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>>> >>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type: >>>>>>> >>>>>>> see http://www.mcs.anl.gov/petsc/documentation/installation.html#external! >>>>>>> >>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given! >>>>>>> >>>>>>> >>>>>>> >>>>>>> Has there been a change in the aggregation options? I just pulled petsc-dev this morning. >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>>> >>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg for now. >>>>>>> >>>>>> >>>>>>> >>>>>> Mark >>>>>>> >>>>>> >>>>>>> >>>>>>> John >>>>>>> >>>>>> >>>>>>> >>>>>> >>>>>>> >>>>> >>>>>>> >>>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>> >>>>>>> >>> >>>>>>> >> >>>>>>> >> >>>>>>> > >>>>>>> > >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>> >>>> >>> >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Wed Mar 21 04:03:39 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Wed, 21 Mar 2012 10:03:39 +0100 Subject: [petsc-users] Convergence behavior of different solvers Message-ID: <4F69996B.4030407@gfz-potsdam.de> Hello, Plotting values of relative residual norm versus iteration I noticed that these curves have different behavior. For instance, ones I got with BiCGStab are oscillatory and rather noisy with large jumps within few iterations. Curves for QMR are smooth, however it's usual that they stagnate for ~10-20 iterations and then go down further. Same preconditioner is applied for both and both methods are Lanczos based, so I don't understand this. I'm wondering is this just my specific case or there is general explanation for that? -- Regards, Alexander From jedbrown at mcs.anl.gov Wed Mar 21 05:38:39 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 21 Mar 2012 05:38:39 -0500 Subject: [petsc-users] Convergence behavior of different solvers In-Reply-To: <4F69996B.4030407@gfz-potsdam.de> References: <4F69996B.4030407@gfz-potsdam.de> Message-ID: On Wed, Mar 21, 2012 at 04:03, Alexander Grayver wrote: > Hello, > > Plotting values of relative residual norm versus iteration I noticed that > these curves have different behavior. For instance, ones I got with > BiCGStab are oscillatory and rather noisy with large jumps within few > iterations. Curves for QMR are smooth, however it's usual that they > stagnate for ~10-20 iterations and then go down further. Same > preconditioner is applied for both and both methods are Lanczos based, so I > don't understand this. > I'm wondering is this just my specific case or there is general > explanation for that? > This is normal. http://people.maths.ox.ac.uk/trefethen/publication/PDF/1992_52.pdf -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Wed Mar 21 06:01:59 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Wed, 21 Mar 2012 12:01:59 +0100 Subject: [petsc-users] Convergence behavior of different solvers In-Reply-To: References: <4F69996B.4030407@gfz-potsdam.de> Message-ID: <4F69B527.20104@gfz-potsdam.de> Ok, thank you for pointing me to this paper. On 21.03.2012 11:38, Jed Brown wrote: > On Wed, Mar 21, 2012 at 04:03, Alexander Grayver > > wrote: > > Hello, > > Plotting values of relative residual norm versus iteration I > noticed that these curves have different behavior. For instance, > ones I got with BiCGStab are oscillatory and rather noisy with > large jumps within few iterations. Curves for QMR are smooth, > however it's usual that they stagnate for ~10-20 iterations and > then go down further. Same preconditioner is applied for both and > both methods are Lanczos based, so I don't understand this. > I'm wondering is this just my specific case or there is general > explanation for that? > > > This is normal. > > http://people.maths.ox.ac.uk/trefethen/publication/PDF/1992_52.pdf -- Regards, Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From Markus.Iivonen at metropolia.fi Wed Mar 21 11:05:03 2012 From: Markus.Iivonen at metropolia.fi (Markus Iivonen) Date: Wed, 21 Mar 2012 16:05:03 +0000 Subject: [petsc-users] Inaccuracy while calculating numbers on different machines 32-bit and 64-bit. Message-ID: I've read that inaccuracy is quite common problem but on the other hand my values aren't that small that there should be any inaccuracy at least I think so... So the problem is that when I run the code on the other machine it causes inaccuracy which happens as early as 0.00000 digits. My program calculates two different values at the same time. First value is really accurate and it is calculated by boost bisection the second value is produced by two different functions which makes matrix and vector calculations using PETSc. Here are some example values: Where to compare: 0.01985308798 -0.8497140415 1 0.3683436681 0.509926544 -0.2559051838 0.754963272 0.06256485518 Here is what my one computer produces: 0.0198531 -0.849714 1 0.368344 0.509927 -0.255905 0.754963 0.0625649 And here is what happens when I'll run the program on other machine. 0.0198531 -0.849714 1 0.368321 0.509927 -0.255922 0.754963 0.062544 My one machine is running ubuntu 32-bit and other is running 64-bit. The actual values where I'm comparing to are calculated on the 64-bit system but different program. If I run the example value program which generated my base values on my 32-bit desktop it will give the same values as my C++ program and the example program on 64-bit system. When I run the C++ program on the 64-bit system the inaccuracy occurs. The values that I'm using are always doubles. Somehow the inaccuracy seems to happen mostly in the values that are produced by PETSc matrix and vector operations. Compilers, libraries and library confs are exactly same on the both machines. I did also try to set locales but it didn't have any affect, I just searched something and found this about PetscInt: PETSc type that represents integer - used primarily to represent size of arrays and indexing into arrays. Its size can be configured with the option --with-64-bit-indices - to be either 32bit or 64bit [default 32 bit ints] So should I maybe conf 64-bit machine with --with-64-bit-indices or should I try to find the problem somewhere else ? Thank You. From knepley at gmail.com Wed Mar 21 11:08:06 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 21 Mar 2012 11:08:06 -0500 Subject: [petsc-users] Inaccuracy while calculating numbers on different machines 32-bit and 64-bit. In-Reply-To: References: Message-ID: On Wed, Mar 21, 2012 at 11:05 AM, Markus Iivonen < Markus.Iivonen at metropolia.fi> wrote: > I've read that inaccuracy is quite common problem but on the other hand my > values aren't that small that there should be any inaccuracy at least I > think so... > > So the problem is that when I run the code on the other machine it causes > inaccuracy which happens as early as 0.00000 digits. > > My program calculates two different values at the same time. First value > is really accurate and it is calculated by boost bisection the second value > is produced by two different functions which makes matrix and vector > calculations using PETSc. Here are some example values: > Lists of numbers are meaningless. I guarantee you that these do not result from roundoff. I will bet that you are running some sort of solve, which is sensitive to the tolerance. Send a simple example to petsc-maint at mcs.anl.gov. Matt > Where to compare: > > 0.01985308798 -0.8497140415 > 1 0.3683436681 > 0.509926544 -0.2559051838 > 0.754963272 0.06256485518 > > Here is what my one computer produces: > > 0.0198531 -0.849714 > 1 0.368344 > 0.509927 -0.255905 > 0.754963 0.0625649 > > And here is what happens when I'll run the program on other machine. > > 0.0198531 -0.849714 > 1 0.368321 > 0.509927 -0.255922 > 0.754963 0.062544 > > My one machine is running ubuntu 32-bit and other is running 64-bit. The > actual values where I'm comparing to are calculated on the 64-bit system > but different program. > > If I run the example value program which generated my base values on my > 32-bit desktop it will give the same values as my C++ program and the > example program on 64-bit system. When I run the C++ program on the 64-bit > system the inaccuracy occurs. The values that I'm using are always doubles. > > Somehow the inaccuracy seems to happen mostly in the values that are > produced by PETSc matrix and vector operations. > > Compilers, libraries and library confs are exactly same on the both > machines. I did also try to set locales but it didn't have any affect, I > just searched something and found this about PetscInt: > > PETSc type that represents integer - used primarily to represent size of > arrays and indexing into arrays. Its size can be configured with the option > --with-64-bit-indices - to be either 32bit or 64bit [default 32 bit ints] > > So should I maybe conf 64-bit machine with --with-64-bit-indices or should > I try to find the problem somewhere else ? > > Thank You. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From tautges at mcs.anl.gov Wed Mar 21 14:38:13 2012 From: tautges at mcs.anl.gov (Tim Tautges) Date: Wed, 21 Mar 2012 14:38:13 -0500 Subject: [petsc-users] openmpi/petsc 3.1/gfortran and multiply-defined symbols Message-ID: <4F6A2E25.2000401@mcs.anl.gov> Hi all, I'm almost embarrassed to be asking this, but haven't found anything that looks similar in the archives... I'm compiling fortran code and get: mpif90 -c -g -ffree-line-length-none -fcray-pointer -DMOAB -fPIC -Wall -Wno-unused-variable -O -I/usr/local/petsc-3.1-p8-opt/include -I/usr/local/petsc-3.1-p8-opt/include -I/usr/include/scotch -I/usr/lib/openmpi/include -I/usr/lib/openmpi/lib -I/usr/include/spooles -I/usr/include/suitesparse -I/usr/local/petsc-3.1-p8-opt/include -I/usr/include/scotch -I/usr/lib/openmpi/include -I/usr/lib/openmpi/lib -I/usr/include/spooles -I/usr/include/suitesparse -I. -I./MeshLibrary/RTelements -I/usr/local/hdf5-1.8.8-par-openmpi-gcc/include -I/home/tautges/code/MOABpar/include -o MeshLibrary/PNTmesh_Import.o MeshLibrary/PNTmesh_Import.F90 mpif-mpi-io.h:64.27: Included at mpif-config.h:65: Included at mpif-common.h:70: Included at /usr/lib/openmpi/include/mpif.h:59: Included at /usr/local/petsc-3.1-p8-opt/include/finclude/petscsys.h:11: Included at /usr/local/petsc-3.1-p8-opt/include/finclude/petsc.h:6: Included at MeshLibrary/PNTmesh_Import.F90:16: integer MPI_FILE_NULL 1 Error: Symbol 'mpi_file_null' at (1) already has basic type of INTEGER My petsc configure options (as reported in arch/configure.log) are: Configure Options: --configModules=PETSc.Configure --optionsModule=PETSc.compilerOptions --with-clanguage=C++ --with-blas-lapack-dir=/usr/lib --with-mpi-dir=/usr --with-debugging=1 --prefix=/usr/local/petsc-3.1-p8-dbg Working directory: /home/tautges/linux/petsc-3.1-p8 This is an Ubuntu 10.04 system, synaptic-installed openmpi, gfortran 4.4.3, all pretty standard stuff. I think the problem started showing up when I inserted the "--with-clanguage=C++" in the configure line, expecting that would take care of finding the C++ std libs for me (which it did). Clues? Thanks. - tim -- ================================================================ "You will keep in perfect peace him whose mind is steadfast, because he trusts in you." Isaiah 26:3 Tim Tautges Argonne National Laboratory (tautges at mcs.anl.gov) (telecommuting from UW-Madison) phone (gvoice): (608) 354-1459 1500 Engineering Dr. fax: (608) 263-4499 Madison, WI 53706 From balay at mcs.anl.gov Wed Mar 21 14:48:04 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 21 Mar 2012 14:48:04 -0500 (CDT) Subject: [petsc-users] openmpi/petsc 3.1/gfortran and multiply-defined symbols In-Reply-To: <4F6A2E25.2000401@mcs.anl.gov> References: <4F6A2E25.2000401@mcs.anl.gov> Message-ID: a couple of issues: - Looks like you've installed petsc at --prefix=/usr/local/petsc-3.1-p8-dbg - but using from -I/usr/local/petsc-3.1-p8-opt - Should avoid pointing to std locations as configure will always look there. --with-blas-lapack-dir=/usr/lib --with-mpi-dir=/usr - when reinstalling - make sure the prefix location does not have any old install files. And to track the duplicate location of MPI_FILE_NULL - suggest compiling the code with '-E' preprocessor directive - and then locate MPI_FILE_NULL there.. i.e mpif90 -c -g -ffree-line-length-none -fcray-pointer -DMOAB -fPIC -Wall -Wno-unused-variable -O -I/usr/local/petsc-3.1-p8-opt/include -I/usr/local/petsc-3.1-p8-opt/include -I/usr/include/scotch -I/usr/lib/openmpi/include -I/usr/lib/openmpi/lib -I/usr/include/spooles -I/usr/include/suitesparse -I/usr/local/petsc-3.1-p8-opt/include -I/usr/include/scotch -I/usr/lib/openmpi/include -I/usr/lib/openmpi/lib -I/usr/include/spooles -I/usr/include/suitesparse -I. -I./MeshLibrary/RTelements -I/usr/local/hdf5-1.8.8-par-openmpi-gcc/include -I/home/tautges/code/MOABpar/include MeshLibrary/PNTmesh_Import.F90 -E > out.f Satish On Wed, 21 Mar 2012, Tim Tautges wrote: > Hi all, > I'm almost embarrassed to be asking this, but haven't found anything that > looks similar in the archives... > > I'm compiling fortran code and get: > > mpif90 -c -g -ffree-line-length-none -fcray-pointer -DMOAB -fPIC -Wall > -Wno-unused-variable -O -I/usr/local/petsc-3.1-p8-opt/include > -I/usr/local/petsc-3.1-p8-opt/include -I/usr/include/scotch > -I/usr/lib/openmpi/include -I/usr/lib/openmpi/lib -I/usr/include/spooles > -I/usr/include/suitesparse -I/usr/local/petsc-3.1-p8-opt/include > -I/usr/include/scotch -I/usr/lib/openmpi/include -I/usr/lib/openmpi/lib > -I/usr/include/spooles -I/usr/include/suitesparse -I. > -I./MeshLibrary/RTelements -I/usr/local/hdf5-1.8.8-par-openmpi-gcc/include > -I/home/tautges/code/MOABpar/include -o MeshLibrary/PNTmesh_Import.o > MeshLibrary/PNTmesh_Import.F90 > mpif-mpi-io.h:64.27: > Included at mpif-config.h:65: > Included at mpif-common.h:70: > Included at /usr/lib/openmpi/include/mpif.h:59: > Included at /usr/local/petsc-3.1-p8-opt/include/finclude/petscsys.h:11: > Included at /usr/local/petsc-3.1-p8-opt/include/finclude/petsc.h:6: > Included at MeshLibrary/PNTmesh_Import.F90:16: > > integer MPI_FILE_NULL > 1 > Error: Symbol 'mpi_file_null' at (1) already has basic type of INTEGER > > > My petsc configure options (as reported in arch/configure.log) are: > > Configure Options: --configModules=PETSc.Configure > --optionsModule=PETSc.compilerOptions --with-clanguage=C++ > --with-blas-lapack-dir=/usr/lib --with-mpi-dir=/usr --with-debugging=1 > --prefix=/usr/local/petsc-3.1-p8-dbg > Working directory: /home/tautges/linux/petsc-3.1-p8 > > This is an Ubuntu 10.04 system, synaptic-installed openmpi, gfortran 4.4.3, > all pretty standard stuff. I think the problem started showing up when I > inserted the "--with-clanguage=C++" in the configure line, expecting that > would take care of finding the C++ std libs for me (which it did). Clues? > > Thanks. > > - tim > > From Feras.A.Habbal at jpl.nasa.gov Wed Mar 21 17:11:15 2012 From: Feras.A.Habbal at jpl.nasa.gov (Habbal, Feras (353G-Affiliate)) Date: Wed, 21 Mar 2012 15:11:15 -0700 Subject: [petsc-users] Saving graphical matrix structure to a file Message-ID: Hello, I am trying to save the matrix structure visualization from (-mat_view_draw) to a file (jpeg, ps, or anything else). I saw the post: "help in capturing matrix patterns to files," but adding the options did not work for me: ?mat_view_draw ?draw_type ps I have been trying to do this with the example file: ksp/examples/tutorials/ex1.c I included "petscdraw.h" $ mpiexec ex1 -mat_view_draw -draw_type ps [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type seehttp://www.mcs.anl.gov/petsc/petsc-as/documentation/installation.html#external! [0]PETSC ERROR: Unknown PetscDraw type given: ps! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 10:28:33 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ex1 on a macosx-gn named habbalf.jpl.nasa.gov by habbalf Wed Mar 21 14:56:30 2012 [0]PETSC ERROR: Libraries linked from /Users/habbalf/issm/trunk-jpl/externalpackages/petsc/install/lib [0]PETSC ERROR: Configure run at Tue Mar 6 11:39:23 2012 [0]PETSC ERROR: Configure options --prefix=/Users/habbalf/issm/trunk-jpl/externalpackages/petsc/install --PETSC_DIR=/Users/habbalf/issm/trunk-jpl/externalpackages/petsc/src --PETSC_ARCH=macosx-gnu --with-mpi-dir=/Users/habbalf/issm/trunk-jpl/externalpackages/mpich2/install --with-debugging=1 --download-c2html=1 --with-shared-libraries=0 --download-mumps=yes --download-scalapack=yes --download-blacs=yes --download-blas=yes --download-plapack=yes --download-parmetis=yes --download-f-blas-lapack=yes [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PetscDrawSetType() line 147 in src/sys/draw/interface/drawreg.c [0]PETSC ERROR: PetscDrawSetFromOptions() line 278 in src/sys/draw/interface/drawreg.c [0]PETSC ERROR: PetscViewerDrawGetDraw() line 109 in src/sys/viewer/impls/draw/drawv.c [0]PETSC ERROR: MatView_SeqAIJ_Draw() line 752 in src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: MatView_SeqAIJ() line 781 in src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: MatView() line 757 in src/mat/interface/matrix.c [0]PETSC ERROR: MatView_Private() line 4881 in src/mat/interface/matrix.c [0]PETSC ERROR: MatAssemblyEnd() line 4965 in src/mat/interface/matrix.c [0]PETSC ERROR: main() line 89 in src/ksp/ksp/examples/tutorials/ex1.c application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0 Thank you for your assistance, Feras -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Mar 21 17:14:39 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 21 Mar 2012 17:14:39 -0500 Subject: [petsc-users] Saving graphical matrix structure to a file In-Reply-To: References: Message-ID: On Wed, Mar 21, 2012 at 5:11 PM, Habbal, Feras (353G-Affiliate) < Feras.A.Habbal at jpl.nasa.gov> wrote: > Hello, > > I am trying to save the matrix structure visualization from > (-mat_view_draw) to a file (jpeg, ps, or anything else). > I saw the post: "help in capturing matrix patterns to files," > The easiest thing to do is use screen capture for that window. Matt > but adding the options did not work for me: > ?mat_view_draw ?draw_type ps > > I have been trying to do this with the example > file: ksp/examples/tutorials/ex1.c > > I included "petscdraw.h" > > $ mpiexec ex1 -mat_view_draw -draw_type ps > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external > package needed for type > see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/installation.html#external > ! > [0]PETSC ERROR: Unknown PetscDraw type given: ps! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 10:28:33 > CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ex1 on a macosx-gn named habbalf.jpl.nasa.gov by habbalf > Wed Mar 21 14:56:30 2012 > [0]PETSC ERROR: Libraries linked from > /Users/habbalf/issm/trunk-jpl/externalpackages/petsc/install/lib > [0]PETSC ERROR: Configure run at Tue Mar 6 11:39:23 2012 > [0]PETSC ERROR: Configure options > --prefix=/Users/habbalf/issm/trunk-jpl/externalpackages/petsc/install > --PETSC_DIR=/Users/habbalf/issm/trunk-jpl/externalpackages/petsc/src > --PETSC_ARCH=macosx-gnu > --with-mpi-dir=/Users/habbalf/issm/trunk-jpl/externalpackages/mpich2/install > --with-debugging=1 --download-c2html=1 --with-shared-libraries=0 > --download-mumps=yes --download-scalapack=yes --download-blacs=yes > --download-blas=yes --download-plapack=yes --download-parmetis=yes > --download-f-blas-lapack=yes > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: PetscDrawSetType() line 147 in > src/sys/draw/interface/drawreg.c > [0]PETSC ERROR: PetscDrawSetFromOptions() line 278 in > src/sys/draw/interface/drawreg.c > [0]PETSC ERROR: PetscViewerDrawGetDraw() line 109 in > src/sys/viewer/impls/draw/drawv.c > [0]PETSC ERROR: MatView_SeqAIJ_Draw() line 752 in > src/mat/impls/aij/seq/aij.c > [0]PETSC ERROR: MatView_SeqAIJ() line 781 in src/mat/impls/aij/seq/aij.c > [0]PETSC ERROR: MatView() line 757 in src/mat/interface/matrix.c > [0]PETSC ERROR: MatView_Private() line 4881 in src/mat/interface/matrix.c > [0]PETSC ERROR: MatAssemblyEnd() line 4965 in src/mat/interface/matrix.c > [0]PETSC ERROR: main() line 89 in src/ksp/ksp/examples/tutorials/ex1.c > application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0 > > Thank you for your assistance, > > Feras > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Feras.A.Habbal at jpl.nasa.gov Wed Mar 21 17:31:57 2012 From: Feras.A.Habbal at jpl.nasa.gov (Habbal, Feras (353G-Affiliate)) Date: Wed, 21 Mar 2012 15:31:57 -0700 Subject: [petsc-users] Saving graphical matrix structure to a file In-Reply-To: Message-ID: I tried using the commands from the previous post: xwd | xpr -device ps > foo.ps But the output was bizarre: reversed and had some artifacts. I can certainly use the mac screen capture tools, but I am needing an automated capability to capture the structure of several matrices (~1000) during a run. Feras From: Matthew Knepley > Reply-To: PETSc users list > Date: Wed, 21 Mar 2012 15:14:39 -0700 To: PETSc users list > Subject: Re: [petsc-users] Saving graphical matrix structure to a file On Wed, Mar 21, 2012 at 5:11 PM, Habbal, Feras (353G-Affiliate) > wrote: Hello, I am trying to save the matrix structure visualization from (-mat_view_draw) to a file (jpeg, ps, or anything else). I saw the post: "help in capturing matrix patterns to files," The easiest thing to do is use screen capture for that window. Matt but adding the options did not work for me: ?mat_view_draw ?draw_type ps I have been trying to do this with the example file: ksp/examples/tutorials/ex1.c I included "petscdraw.h" $ mpiexec ex1 -mat_view_draw -draw_type ps [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type seehttp://www.mcs.anl.gov/petsc/petsc-as/documentation/installation.html#external! [0]PETSC ERROR: Unknown PetscDraw type given: ps! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 10:28:33 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ex1 on a macosx-gn named habbalf.jpl.nasa.gov by habbalf Wed Mar 21 14:56:30 2012 [0]PETSC ERROR: Libraries linked from /Users/habbalf/issm/trunk-jpl/externalpackages/petsc/install/lib [0]PETSC ERROR: Configure run at Tue Mar 6 11:39:23 2012 [0]PETSC ERROR: Configure options --prefix=/Users/habbalf/issm/trunk-jpl/externalpackages/petsc/install --PETSC_DIR=/Users/habbalf/issm/trunk-jpl/externalpackages/petsc/src --PETSC_ARCH=macosx-gnu --with-mpi-dir=/Users/habbalf/issm/trunk-jpl/externalpackages/mpich2/install --with-debugging=1 --download-c2html=1 --with-shared-libraries=0 --download-mumps=yes --download-scalapack=yes --download-blacs=yes --download-blas=yes --download-plapack=yes --download-parmetis=yes --download-f-blas-lapack=yes [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PetscDrawSetType() line 147 in src/sys/draw/interface/drawreg.c [0]PETSC ERROR: PetscDrawSetFromOptions() line 278 in src/sys/draw/interface/drawreg.c [0]PETSC ERROR: PetscViewerDrawGetDraw() line 109 in src/sys/viewer/impls/draw/drawv.c [0]PETSC ERROR: MatView_SeqAIJ_Draw() line 752 in src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: MatView_SeqAIJ() line 781 in src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: MatView() line 757 in src/mat/interface/matrix.c [0]PETSC ERROR: MatView_Private() line 4881 in src/mat/interface/matrix.c [0]PETSC ERROR: MatAssemblyEnd() line 4965 in src/mat/interface/matrix.c [0]PETSC ERROR: main() line 89 in src/ksp/ksp/examples/tutorials/ex1.c application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0 Thank you for your assistance, Feras -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Mar 21 17:43:07 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 21 Mar 2012 17:43:07 -0500 Subject: [petsc-users] Saving graphical matrix structure to a file In-Reply-To: References: Message-ID: On Wed, Mar 21, 2012 at 5:31 PM, Habbal, Feras (353G-Affiliate) < Feras.A.Habbal at jpl.nasa.gov> wrote: > I tried using the commands from the previous post: > xwd | xpr -device ps > foo.ps > That worked fine for me when I made movies this way. I'm not sure what else to recommend. Matt > But the output was bizarre: reversed and had some artifacts. > > I can certainly use the mac screen capture tools, but I am needing an > automated capability to capture the structure of several > matrices (~1000) during a run. > > Feras > > From: Matthew Knepley > Reply-To: PETSc users list > Date: Wed, 21 Mar 2012 15:14:39 -0700 > To: PETSc users list > Subject: Re: [petsc-users] Saving graphical matrix structure to a file > > On Wed, Mar 21, 2012 at 5:11 PM, Habbal, Feras (353G-Affiliate) < > Feras.A.Habbal at jpl.nasa.gov> wrote: > >> Hello, >> >> I am trying to save the matrix structure visualization from >> (-mat_view_draw) to a file (jpeg, ps, or anything else). >> I saw the post: "help in capturing matrix patterns to files," >> > > The easiest thing to do is use screen capture for that window. > > Matt > > >> but adding the options did not work for me: >> ?mat_view_draw ?draw_type ps >> >> I have been trying to do this with the example >> file: ksp/examples/tutorials/ex1.c >> >> I included "petscdraw.h" >> >> $ mpiexec ex1 -mat_view_draw -draw_type ps >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external >> package needed for type >> see >> http://www.mcs.anl.gov/petsc/petsc-as/documentation/installation.html#external >> ! >> [0]PETSC ERROR: Unknown PetscDraw type given: ps! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 10:28:33 >> CDT 2011 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: ex1 on a macosx-gn named habbalf.jpl.nasa.gov by habbalf >> Wed Mar 21 14:56:30 2012 >> [0]PETSC ERROR: Libraries linked from >> /Users/habbalf/issm/trunk-jpl/externalpackages/petsc/install/lib >> [0]PETSC ERROR: Configure run at Tue Mar 6 11:39:23 2012 >> [0]PETSC ERROR: Configure options >> --prefix=/Users/habbalf/issm/trunk-jpl/externalpackages/petsc/install >> --PETSC_DIR=/Users/habbalf/issm/trunk-jpl/externalpackages/petsc/src >> --PETSC_ARCH=macosx-gnu >> --with-mpi-dir=/Users/habbalf/issm/trunk-jpl/externalpackages/mpich2/install >> --with-debugging=1 --download-c2html=1 --with-shared-libraries=0 >> --download-mumps=yes --download-scalapack=yes --download-blacs=yes >> --download-blas=yes --download-plapack=yes --download-parmetis=yes >> --download-f-blas-lapack=yes >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: PetscDrawSetType() line 147 in >> src/sys/draw/interface/drawreg.c >> [0]PETSC ERROR: PetscDrawSetFromOptions() line 278 in >> src/sys/draw/interface/drawreg.c >> [0]PETSC ERROR: PetscViewerDrawGetDraw() line 109 in >> src/sys/viewer/impls/draw/drawv.c >> [0]PETSC ERROR: MatView_SeqAIJ_Draw() line 752 in >> src/mat/impls/aij/seq/aij.c >> [0]PETSC ERROR: MatView_SeqAIJ() line 781 in src/mat/impls/aij/seq/aij.c >> [0]PETSC ERROR: MatView() line 757 in src/mat/interface/matrix.c >> [0]PETSC ERROR: MatView_Private() line 4881 in src/mat/interface/matrix.c >> [0]PETSC ERROR: MatAssemblyEnd() line 4965 in src/mat/interface/matrix.c >> [0]PETSC ERROR: main() line 89 in src/ksp/ksp/examples/tutorials/ex1.c >> application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0 >> >> Thank you for your assistance, >> >> Feras >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Feras.A.Habbal at jpl.nasa.gov Wed Mar 21 18:02:50 2012 From: Feras.A.Habbal at jpl.nasa.gov (Habbal, Feras (353G-Affiliate)) Date: Wed, 21 Mar 2012 16:02:50 -0700 Subject: [petsc-users] Saving graphical matrix structure to a file In-Reply-To: Message-ID: Ok, no problem. Thank you for your assistance! From: Matthew Knepley > Reply-To: PETSc users list > Date: Wed, 21 Mar 2012 15:43:07 -0700 To: PETSc users list > Subject: Re: [petsc-users] Saving graphical matrix structure to a file On Wed, Mar 21, 2012 at 5:31 PM, Habbal, Feras (353G-Affiliate) > wrote: I tried using the commands from the previous post: xwd | xpr -device ps > foo.ps That worked fine for me when I made movies this way. I'm not sure what else to recommend. Matt But the output was bizarre: reversed and had some artifacts. I can certainly use the mac screen capture tools, but I am needing an automated capability to capture the structure of several matrices (~1000) during a run. Feras From: Matthew Knepley > Reply-To: PETSc users list > Date: Wed, 21 Mar 2012 15:14:39 -0700 To: PETSc users list > Subject: Re: [petsc-users] Saving graphical matrix structure to a file On Wed, Mar 21, 2012 at 5:11 PM, Habbal, Feras (353G-Affiliate) > wrote: Hello, I am trying to save the matrix structure visualization from (-mat_view_draw) to a file (jpeg, ps, or anything else). I saw the post: "help in capturing matrix patterns to files," The easiest thing to do is use screen capture for that window. Matt but adding the options did not work for me: ?mat_view_draw ?draw_type ps I have been trying to do this with the example file: ksp/examples/tutorials/ex1.c I included "petscdraw.h" $ mpiexec ex1 -mat_view_draw -draw_type ps [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type seehttp://www.mcs.anl.gov/petsc/petsc-as/documentation/installation.html#external! [0]PETSC ERROR: Unknown PetscDraw type given: ps! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 10:28:33 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ex1 on a macosx-gn named habbalf.jpl.nasa.gov by habbalf Wed Mar 21 14:56:30 2012 [0]PETSC ERROR: Libraries linked from /Users/habbalf/issm/trunk-jpl/externalpackages/petsc/install/lib [0]PETSC ERROR: Configure run at Tue Mar 6 11:39:23 2012 [0]PETSC ERROR: Configure options --prefix=/Users/habbalf/issm/trunk-jpl/externalpackages/petsc/install --PETSC_DIR=/Users/habbalf/issm/trunk-jpl/externalpackages/petsc/src --PETSC_ARCH=macosx-gnu --with-mpi-dir=/Users/habbalf/issm/trunk-jpl/externalpackages/mpich2/install --with-debugging=1 --download-c2html=1 --with-shared-libraries=0 --download-mumps=yes --download-scalapack=yes --download-blacs=yes --download-blas=yes --download-plapack=yes --download-parmetis=yes --download-f-blas-lapack=yes [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PetscDrawSetType() line 147 in src/sys/draw/interface/drawreg.c [0]PETSC ERROR: PetscDrawSetFromOptions() line 278 in src/sys/draw/interface/drawreg.c [0]PETSC ERROR: PetscViewerDrawGetDraw() line 109 in src/sys/viewer/impls/draw/drawv.c [0]PETSC ERROR: MatView_SeqAIJ_Draw() line 752 in src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: MatView_SeqAIJ() line 781 in src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: MatView() line 757 in src/mat/interface/matrix.c [0]PETSC ERROR: MatView_Private() line 4881 in src/mat/interface/matrix.c [0]PETSC ERROR: MatAssemblyEnd() line 4965 in src/mat/interface/matrix.c [0]PETSC ERROR: main() line 89 in src/ksp/ksp/examples/tutorials/ex1.c application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0 Thank you for your assistance, Feras -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdiso at ustc.edu Thu Mar 22 05:14:18 2012 From: gdiso at ustc.edu (Gong Ding) Date: Thu, 22 Mar 2012 18:14:18 +0800 (CST) Subject: [petsc-users] FAIJ patch has bug in parallel Message-ID: <9158501.98251332411258716.JavaMail.coremail@mail.ustc.edu> Hi, Last year I had committed a patch for flexable AIJ matrix, which uses dynamic array for the matrix entries. This patch only works for serial AIJ matrix. However, for parallel MPIAIJ matrix, it seems MatZeroEntries and/or MatAssemblyEnd_MPIAIJ will destroy the nonzero pattern, and MAT_REUSE_MATRIX will report error. I used to set DIFFERENT_NONZERO_PATTERN flag for SNES matrix, thus FAIJ works. However, when I set SAME_NONZERO_PATTERN, my code crash. I tried to fix the problem but failed, sigh. Gong Ding From bin.gao at uit.no Thu Mar 22 15:13:49 2012 From: bin.gao at uit.no (Gao Bin) Date: Thu, 22 Mar 2012 20:13:49 +0000 Subject: [petsc-users] storage of parallel dense matrices and (anti)symmetric matrices Message-ID: Hi there, I am a beginner of using PETSc, and confused about the parallel distribution of matrix in PETSc. For instance, as regards the parallel dense matrices, as described in docs/developers.pdf, "The parallel dense matrices are partitioned by rows across the processors, so that each local rectangular submatrix is stored in the dense format described above." Does it mean each processor will have several continuous rows and all columns of the matrix? If yes, why do we need to specify "n" -- the number of local columns when calling MatCreateMPIDense? I am sorry to raise this simple question, since I have read the manual and tutorials, but I have not found a clear answer. Moreover, the reason I am asking this question is that I would like to use PETSc for matrix operations, but the elements of matrices need to be calculate via my own code. If I know the distribution of the matrix, I could let each processor only calculate and set local values (the rows and columns possessed on the processor itself) for efficiency. My second question is if PETSc provides symmetric and anti-symmetric matrices. I have read the manual, the answer seems to be no. Am I right? Thank you in advance. Cheers Gao -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Mar 22 15:17:47 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 22 Mar 2012 15:17:47 -0500 Subject: [petsc-users] storage of parallel dense matrices and (anti)symmetric matrices In-Reply-To: References: Message-ID: 2012/3/22 Gao Bin > "The parallel dense matrices are partitioned by rows across the > processors, so that each local rectangular submatrix is stored in the dense > format described above." > > Does it mean each processor will have several continuous rows and all > columns of the matrix? If yes, why do we need to specify "n" -- the number > of local columns when calling MatCreateMPIDense? > Interpret the local column size n as the local size of the Vec that the Mat will be applied to. > I am sorry to raise this simple question, since I have read the manual and > tutorials, but I have not found a clear answer. Moreover, the reason I am > asking this question is that I would like to use PETSc for matrix > operations, but the elements of matrices need to be calculate via my own > code. If I know the distribution of the matrix, I could let each processor > only calculate and set local values (the rows and columns possessed on the > processor itself) for efficiency. > > My second question is if PETSc provides symmetric and anti-symmetric > matrices. I have read the manual, the answer seems to be no. Am I right? See the SBAIJ format (it is sparse). With a parallel dense matrix, there isn't any point using a symmetric format unless you use a different distribution of the entries. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bin.gao at uit.no Thu Mar 22 15:32:35 2012 From: bin.gao at uit.no (Gao Bin) Date: Thu, 22 Mar 2012 20:32:35 +0000 Subject: [petsc-users] storage of parallel dense matrices and (anti)symmetric matrices In-Reply-To: References: , Message-ID: Hi, Jed Thank you very much for your quick reply. May I ask two more further questions? (1) Why does not PETSc also partition the columns so that each processor could use less memory? (2) If the matrix I use is a square matrix, the number of local columns "n" should be equal to the number of local rows "m" when calling MatCreateMPIDense, am I right? Thank you again for your answer. Cheers Gao ________________________________ From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Jed Brown [jedbrown at mcs.anl.gov] Sent: Thursday, March 22, 2012 9:17 PM To: PETSc users list Subject: Re: [petsc-users] storage of parallel dense matrices and (anti)symmetric matrices 2012/3/22 Gao Bin > "The parallel dense matrices are partitioned by rows across the processors, so that each local rectangular submatrix is stored in the dense format described above." Does it mean each processor will have several continuous rows and all columns of the matrix? If yes, why do we need to specify "n" -- the number of local columns when calling MatCreateMPIDense? Interpret the local column size n as the local size of the Vec that the Mat will be applied to. I am sorry to raise this simple question, since I have read the manual and tutorials, but I have not found a clear answer. Moreover, the reason I am asking this question is that I would like to use PETSc for matrix operations, but the elements of matrices need to be calculate via my own code. If I know the distribution of the matrix, I could let each processor only calculate and set local values (the rows and columns possessed on the processor itself) for efficiency. My second question is if PETSc provides symmetric and anti-symmetric matrices. I have read the manual, the answer seems to be no. Am I right? See the SBAIJ format (it is sparse). With a parallel dense matrix, there isn't any point using a symmetric format unless you use a different distribution of the entries. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Mar 22 15:38:18 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 22 Mar 2012 15:38:18 -0500 Subject: [petsc-users] storage of parallel dense matrices and (anti)symmetric matrices In-Reply-To: References: Message-ID: On Thu, Mar 22, 2012 at 3:32 PM, Gao Bin wrote: > Hi, Jed > > Thank you very much for your quick reply. May I ask two more further > questions? > > (1) Why does not PETSc also partition the columns so that each processor > could use less memory? > 2D distributions are not efficient for sparse matrices. They are sometimes used for dense. > (2) If the matrix I use is a square matrix, the number of local columns > "n" should be equal to the number of local rows "m" when calling > MatCreateMPIDense, am I right? > Yes. You can always let PETSc choose by giving PETSC_DETERMINE. Matt > Thank you again for your answer. > > Cheers > > Gao > ------------------------------ > *From:* petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] > on behalf of Jed Brown [jedbrown at mcs.anl.gov] > *Sent:* Thursday, March 22, 2012 9:17 PM > *To:* PETSc users list > *Subject:* Re: [petsc-users] storage of parallel dense matrices and > (anti)symmetric matrices > > 2012/3/22 Gao Bin > >> "The parallel dense matrices are partitioned by rows across the >> processors, so that each local rectangular submatrix is stored in the dense >> format described above." >> >> Does it mean each processor will have several continuous rows and all >> columns of the matrix? If yes, why do we need to specify "n" -- the number >> of local columns when calling MatCreateMPIDense? >> > > Interpret the local column size n as the local size of the Vec that the > Mat will be applied to. > > >> I am sorry to raise this simple question, since I have read the manual >> and tutorials, but I have not found a clear answer. Moreover, the reason I >> am asking this question is that I would like to use PETSc for matrix >> operations, but the elements of matrices need to be calculate via my own >> code. If I know the distribution of the matrix, I could let each processor >> only calculate and set local values (the rows and columns possessed on the >> processor itself) for efficiency. >> >> My second question is if PETSc provides symmetric and anti-symmetric >> matrices. I have read the manual, the answer seems to be no. Am I right? > > > See the SBAIJ format (it is sparse). > > With a parallel dense matrix, there isn't any point using a symmetric > format unless you use a different distribution of the entries. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bin.gao at uit.no Thu Mar 22 15:45:26 2012 From: bin.gao at uit.no (Gao Bin) Date: Thu, 22 Mar 2012 20:45:26 +0000 Subject: [petsc-users] storage of parallel dense matrices and (anti)symmetric matrices In-Reply-To: References: , Message-ID: Hi, Matt Thank you. I see the point :-) Cheers Gao ________________________________ From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Matthew Knepley [knepley at gmail.com] Sent: Thursday, March 22, 2012 9:38 PM To: PETSc users list Subject: Re: [petsc-users] storage of parallel dense matrices and (anti)symmetric matrices On Thu, Mar 22, 2012 at 3:32 PM, Gao Bin > wrote: Hi, Jed Thank you very much for your quick reply. May I ask two more further questions? (1) Why does not PETSc also partition the columns so that each processor could use less memory? 2D distributions are not efficient for sparse matrices. They are sometimes used for dense. (2) If the matrix I use is a square matrix, the number of local columns "n" should be equal to the number of local rows "m" when calling MatCreateMPIDense, am I right? Yes. You can always let PETSc choose by giving PETSC_DETERMINE. Matt Thank you again for your answer. Cheers Gao ________________________________ From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Jed Brown [jedbrown at mcs.anl.gov] Sent: Thursday, March 22, 2012 9:17 PM To: PETSc users list Subject: Re: [petsc-users] storage of parallel dense matrices and (anti)symmetric matrices 2012/3/22 Gao Bin > "The parallel dense matrices are partitioned by rows across the processors, so that each local rectangular submatrix is stored in the dense format described above." Does it mean each processor will have several continuous rows and all columns of the matrix? If yes, why do we need to specify "n" -- the number of local columns when calling MatCreateMPIDense? Interpret the local column size n as the local size of the Vec that the Mat will be applied to. I am sorry to raise this simple question, since I have read the manual and tutorials, but I have not found a clear answer. Moreover, the reason I am asking this question is that I would like to use PETSc for matrix operations, but the elements of matrices need to be calculate via my own code. If I know the distribution of the matrix, I could let each processor only calculate and set local values (the rows and columns possessed on the processor itself) for efficiency. My second question is if PETSc provides symmetric and anti-symmetric matrices. I have read the manual, the answer seems to be no. Am I right? See the SBAIJ format (it is sparse). With a parallel dense matrix, there isn't any point using a symmetric format unless you use a different distribution of the entries. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Mar 22 15:58:27 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 22 Mar 2012 15:58:27 -0500 Subject: [petsc-users] Saving graphical matrix structure to a file In-Reply-To: References: Message-ID: <07E6F3D5-75B1-4D01-955F-7893C3841194@mcs.anl.gov> Feras, You are in luck, we do have a mechanism that automatically save PETSc X images as Gif files. I delayed responding so I could clean up the code for you to use. 1) It is only in petsc-dev http://www.mcs.anl.gov/petsc/developers/index.html 2) you must first install afterimage (easy) which installs in /usr/local 3) ./configure PETSc with the additional option --with-afterimage 4) when running your program just use the extra command line option -draw_save or use the function PetscDrawSetSave() on the draw object you create before using it 5) If you want to generate a movie of all the images once the problem is complete also install ffmpeg and also use the run time option -draw_save_movie 6) When generating your images make sure they are all open and fulling displayed on the screen, afterimage will not function if the window is not fully visible. 7) Report all problems to petsc-maint at mcs.anl.gov 8) Have fun 9) Like all PETSc graphics this is a useful tool for debugging and algorithmic development etc, it is not intended to produce publication quality graphics Barry On Mar 21, 2012, at 6:02 PM, Habbal, Feras (353G-Affiliate) wrote: > Ok, no problem. Thank you for your assistance! > > From: Matthew Knepley > Reply-To: PETSc users list > Date: Wed, 21 Mar 2012 15:43:07 -0700 > To: PETSc users list > Subject: Re: [petsc-users] Saving graphical matrix structure to a file > > On Wed, Mar 21, 2012 at 5:31 PM, Habbal, Feras (353G-Affiliate) wrote: >> I tried using the commands from the previous post: >> xwd | xpr -device ps > foo.ps > > That worked fine for me when I made movies this way. I'm not sure what else to recommend. > > Matt > >> But the output was bizarre: reversed and had some artifacts. >> >> I can certainly use the mac screen capture tools, but I am needing an automated capability to capture the structure of several matrices (~1000) during a run. >> >> Feras >> >> From: Matthew Knepley >> Reply-To: PETSc users list >> Date: Wed, 21 Mar 2012 15:14:39 -0700 >> To: PETSc users list >> Subject: Re: [petsc-users] Saving graphical matrix structure to a file >> >> On Wed, Mar 21, 2012 at 5:11 PM, Habbal, Feras (353G-Affiliate) wrote: >>> Hello, >>> >>> I am trying to save the matrix structure visualization from (-mat_view_draw) to a file (jpeg, ps, or anything else). >>> I saw the post: "help in capturing matrix patterns to files," >> >> The easiest thing to do is use screen capture for that window. >> >> Matt >> >>> but adding the options did not work for me: >>> ?mat_view_draw ?draw_type ps >>> >>> I have been trying to do this with the example file: ksp/examples/tutorials/ex1.c >>> >>> I included "petscdraw.h" >>> >>> $ mpiexec ex1 -mat_view_draw -draw_type ps >>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type >>> seehttp://www.mcs.anl.gov/petsc/petsc-as/documentation/installation.html#external! >>> [0]PETSC ERROR: Unknown PetscDraw type given: ps! >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 10:28:33 CDT 2011 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: ex1 on a macosx-gn named habbalf.jpl.nasa.gov by habbalf Wed Mar 21 14:56:30 2012 >>> [0]PETSC ERROR: Libraries linked from /Users/habbalf/issm/trunk-jpl/externalpackages/petsc/install/lib >>> [0]PETSC ERROR: Configure run at Tue Mar 6 11:39:23 2012 >>> [0]PETSC ERROR: Configure options --prefix=/Users/habbalf/issm/trunk-jpl/externalpackages/petsc/install --PETSC_DIR=/Users/habbalf/issm/trunk-jpl/externalpackages/petsc/src --PETSC_ARCH=macosx-gnu --with-mpi-dir=/Users/habbalf/issm/trunk-jpl/externalpackages/mpich2/install --with-debugging=1 --download-c2html=1 --with-shared-libraries=0 --download-mumps=yes --download-scalapack=yes --download-blacs=yes --download-blas=yes --download-plapack=yes --download-parmetis=yes --download-f-blas-lapack=yes >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: PetscDrawSetType() line 147 in src/sys/draw/interface/drawreg.c >>> [0]PETSC ERROR: PetscDrawSetFromOptions() line 278 in src/sys/draw/interface/drawreg.c >>> [0]PETSC ERROR: PetscViewerDrawGetDraw() line 109 in src/sys/viewer/impls/draw/drawv.c >>> [0]PETSC ERROR: MatView_SeqAIJ_Draw() line 752 in src/mat/impls/aij/seq/aij.c >>> [0]PETSC ERROR: MatView_SeqAIJ() line 781 in src/mat/impls/aij/seq/aij.c >>> [0]PETSC ERROR: MatView() line 757 in src/mat/interface/matrix.c >>> [0]PETSC ERROR: MatView_Private() line 4881 in src/mat/interface/matrix.c >>> [0]PETSC ERROR: MatAssemblyEnd() line 4965 in src/mat/interface/matrix.c >>> [0]PETSC ERROR: main() line 89 in src/ksp/ksp/examples/tutorials/ex1.c >>> application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0 >>> >>> Thank you for your assistance, >>> >>> Feras >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From Feras.A.Habbal at jpl.nasa.gov Thu Mar 22 16:01:13 2012 From: Feras.A.Habbal at jpl.nasa.gov (Habbal, Feras (353G-Affiliate)) Date: Thu, 22 Mar 2012 14:01:13 -0700 Subject: [petsc-users] Saving graphical matrix structure to a file In-Reply-To: <07E6F3D5-75B1-4D01-955F-7893C3841194@mcs.anl.gov> Message-ID: Thank you very much for your response! I will definitely make use of this feature. On 3/22/12 1:58 PM, "Barry Smith" wrote: > > Feras, > > You are in luck, we do have a mechanism that automatically save >PETSc X images as Gif files. I delayed responding so I could clean up >the code for you to use. > > 1) It is only in petsc-dev >http://www.mcs.anl.gov/petsc/developers/index.html > > 2) you must first install afterimage (easy) which installs in >/usr/local > > 3) ./configure PETSc with the additional option --with-afterimage > > 4) when running your program just use the extra command line option >-draw_save or use the function PetscDrawSetSave() on the draw object you >create before using it > > 5) If you want to generate a movie of all the images once the problem >is complete also install ffmpeg and also use the run time option >-draw_save_movie > > 6) When generating your images make sure they are all open and >fulling displayed on the screen, afterimage will not function if the >window is not fully visible. > > 7) Report all problems to petsc-maint at mcs.anl.gov > > 8) Have fun > > 9) Like all PETSc graphics this is a useful tool for debugging and >algorithmic development etc, it is not intended to produce publication >quality graphics > > Barry > >On Mar 21, 2012, at 6:02 PM, Habbal, Feras (353G-Affiliate) wrote: > >> Ok, no problem. Thank you for your assistance! >> >> From: Matthew Knepley >> Reply-To: PETSc users list >> Date: Wed, 21 Mar 2012 15:43:07 -0700 >> To: PETSc users list >> Subject: Re: [petsc-users] Saving graphical matrix structure to a file >> >> On Wed, Mar 21, 2012 at 5:31 PM, Habbal, Feras (353G-Affiliate) >> wrote: >>> I tried using the commands from the previous post: >>> xwd | xpr -device ps > foo.ps >> >> That worked fine for me when I made movies this way. I'm not sure what >>else to recommend. >> >> Matt >> >>> But the output was bizarre: reversed and had some artifacts. >>> >>> I can certainly use the mac screen capture tools, but I am needing an >>>automated capability to capture the structure of several matrices >>>(~1000) during a run. >>> >>> Feras >>> >>> From: Matthew Knepley >>> Reply-To: PETSc users list >>> Date: Wed, 21 Mar 2012 15:14:39 -0700 >>> To: PETSc users list >>> Subject: Re: [petsc-users] Saving graphical matrix structure to a file >>> >>> On Wed, Mar 21, 2012 at 5:11 PM, Habbal, Feras (353G-Affiliate) >>> wrote: >>>> Hello, >>>> >>>> I am trying to save the matrix structure visualization from >>>>(-mat_view_draw) to a file (jpeg, ps, or anything else). >>>> I saw the post: "help in capturing matrix patterns to files," >>> >>> The easiest thing to do is use screen capture for that window. >>> >>> Matt >>> >>>> but adding the options did not work for me: >>>> ?mat_view_draw ?draw_type ps >>>> >>>> I have been trying to do this with the example file: >>>>ksp/examples/tutorials/ex1.c >>>> >>>> I included "petscdraw.h" >>>> >>>> $ mpiexec ex1 -mat_view_draw -draw_type ps >>>> [0]PETSC ERROR: --------------------- Error Message >>>>------------------------------------ >>>> [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing >>>>external package needed for type >>>> >>>>seehttp://www.mcs.anl.gov/petsc/petsc-as/documentation/installation.htm >>>>l#external! >>>> [0]PETSC ERROR: Unknown PetscDraw type given: ps! >>>> [0]PETSC ERROR: >>>>----------------------------------------------------------------------- >>>>- >>>> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 >>>>10:28:33 CDT 2011 >>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> [0]PETSC ERROR: >>>>----------------------------------------------------------------------- >>>>- >>>> [0]PETSC ERROR: ex1 on a macosx-gn named habbalf.jpl.nasa.gov by >>>>habbalf Wed Mar 21 14:56:30 2012 >>>> [0]PETSC ERROR: Libraries linked from >>>>/Users/habbalf/issm/trunk-jpl/externalpackages/petsc/install/lib >>>> [0]PETSC ERROR: Configure run at Tue Mar 6 11:39:23 2012 >>>> [0]PETSC ERROR: Configure options >>>>--prefix=/Users/habbalf/issm/trunk-jpl/externalpackages/petsc/install >>>>--PETSC_DIR=/Users/habbalf/issm/trunk-jpl/externalpackages/petsc/src >>>>--PETSC_ARCH=macosx-gnu >>>>--with-mpi-dir=/Users/habbalf/issm/trunk-jpl/externalpackages/mpich2/in >>>>stall --with-debugging=1 --download-c2html=1 --with-shared-libraries=0 >>>>--download-mumps=yes --download-scalapack=yes --download-blacs=yes >>>>--download-blas=yes --download-plapack=yes --download-parmetis=yes >>>>--download-f-blas-lapack=yes >>>> [0]PETSC ERROR: >>>>----------------------------------------------------------------------- >>>>- >>>> [0]PETSC ERROR: PetscDrawSetType() line 147 in >>>>src/sys/draw/interface/drawreg.c >>>> [0]PETSC ERROR: PetscDrawSetFromOptions() line 278 in >>>>src/sys/draw/interface/drawreg.c >>>> [0]PETSC ERROR: PetscViewerDrawGetDraw() line 109 in >>>>src/sys/viewer/impls/draw/drawv.c >>>> [0]PETSC ERROR: MatView_SeqAIJ_Draw() line 752 in >>>>src/mat/impls/aij/seq/aij.c >>>> [0]PETSC ERROR: MatView_SeqAIJ() line 781 in >>>>src/mat/impls/aij/seq/aij.c >>>> [0]PETSC ERROR: MatView() line 757 in src/mat/interface/matrix.c >>>> [0]PETSC ERROR: MatView_Private() line 4881 in >>>>src/mat/interface/matrix.c >>>> [0]PETSC ERROR: MatAssemblyEnd() line 4965 in >>>>src/mat/interface/matrix.c >>>> [0]PETSC ERROR: main() line 89 in src/ksp/ksp/examples/tutorials/ex1.c >>>> application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0 >>>> >>>> Thank you for your assistance, >>>> >>>> Feras >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>>experiments is infinitely more interesting than any results to which >>>their experiments lead. >>> -- Norbert Wiener >> >> >> >> -- >> What most experimenters take for granted before they begin their >>experiments is infinitely more interesting than any results to which >>their experiments lead. >> -- Norbert Wiener > From gdiso at ustc.edu Mon Mar 26 00:50:10 2012 From: gdiso at ustc.edu (Gong Ding) Date: Mon, 26 Mar 2012 13:50:10 +0800 (CST) Subject: [petsc-users] How to solve option conflict during several KSP object Message-ID: <16748967.102251332741010024.JavaMail.coremail@mail.ustc.edu> Hi, I created several KSP objects, each solves different problem. Some problems are well conditioned, thus GMRES+ASM(ilu) works well. However, there's a bad conditioned problem. I have to use GMRES + ASM(lu). The problem is I set the KSP and PC BEFORE matrix assembled. For this situation, I can not call KSPSetUp then PCASMGetSubKSP to set each KSP/PC for local block. At the same time, set sub pc by petsc options i.e PetscOptionsSetValue("-sub_pc_type","lu") will cause conflict since options are global. The solver use GMRES + ASM(ILU) will use PetscOptionsSetValue("-sub_pc_type","ilu") But solver use GMRES + ASM(LU) will set PetscOptionsSetValue("-sub_pc_type","lu") and PetscOptionsSetValue("-sub_pc_factor_mat_solver_package","mumps") Any comment to solve this problem? I wonder if petsc options can be attached to a specified petsc object? Gong Ding From sean at mcs.anl.gov Mon Mar 26 00:52:11 2012 From: sean at mcs.anl.gov (Sean Farley) Date: Mon, 26 Mar 2012 00:52:11 -0500 Subject: [petsc-users] How to solve option conflict during several KSP object In-Reply-To: <16748967.102251332741010024.JavaMail.coremail@mail.ustc.edu> References: <16748967.102251332741010024.JavaMail.coremail@mail.ustc.edu> Message-ID: > > I created several KSP objects, each solves different problem. > Some problems are well conditioned, thus GMRES+ASM(ilu) works well. > However, there's a bad conditioned problem. I have to use GMRES + ASM(lu). > > The problem is I set the KSP and PC BEFORE matrix assembled. > For this situation, I can not call KSPSetUp then PCASMGetSubKSP to set > each KSP/PC for local block. > At the same time, set sub pc by petsc options i.e > PetscOptionsSetValue("-sub_pc_type","lu") > will cause conflict since options are global. http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/KSP/KSPSetOptionsPrefix.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.s.seljebotn at astro.uio.no Mon Mar 26 13:53:03 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Mon, 26 Mar 2012 11:53:03 -0700 Subject: [petsc-users] Repacking/scattering of multi-dimensional arrays Message-ID: <4F70BB0F.1080508@astro.uio.no> Hi list, I'm wondering whether there's any code in PETSc that I can either use directly or lift out and adapt for my purposes (which are described in footnote [1]). What I need to do is a number of different 2D array ("multi-vector", "dense matrix") repacking operations, in order to make the data available to different operations (spherical harmonic transforms, dense linear algebra, sparse linear algebra, etc.). There's a number of repacking operations I'm in need of: - From each process having a given contiguous set of rows of a 2D array, to an element-cyclic distribution where one distributes over both rows and columns (the distribution of Elemental [2]) - Arbitrary redistribution of rows between processes (but each row is kept in a process), potentially with overlaps. - More structured redistribution of rows between processes where indices can be easily computed on the fly (but still a pattern quite specific to my application). While I found PETSc routines for arbitrary redistribution of 1D arrays, I couldn't find anything for 2D arrays. Any pointers into documentation or papers etc. explaining these aspects of PETSc is very welcome. What I'd *really* like (and what I'll do if I end up doing it myself) is something more generic: A small, focused library which, given two descriptors of how an N-dimensional array is distributed, does the redistribution from one to the other. What I'd imagine is that you for each axis of the array pick an axis distribution ("chunked", "(block-)cyclic", "provided by indices", "non-distributed"), and an MPI datatype. (The array would have to be distributed on a process grid of the same dimension as the array). Then, given two N-dimensions descriptors which mixes structured and unstructured information, the library can make and execute a redistribution plan. Perhaps too ambitious, which is why I'm hoping that somebody can point me to at least something existing. I can imagine doing most of it from scratch, but reusing PETSc for the unstructured axis-distribution type. Comments of any kind welcome. Dag [1] What I'm doing is solving a linear system by CG where the LHS matrix can only be written as a (rather large) symbolic equation, containing lots of spherical harmonic transforms, Fourier transform, both dense and sparse linear algebra, and so on. [2] http://code.google.com/p/elemental/ From jedbrown at mcs.anl.gov Mon Mar 26 14:06:05 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 26 Mar 2012 14:06:05 -0500 Subject: [petsc-users] Repacking/scattering of multi-dimensional arrays In-Reply-To: <4F70BB0F.1080508@astro.uio.no> References: <4F70BB0F.1080508@astro.uio.no> Message-ID: Matt, Jack, and myself have discussed adding proper support for Elemental with PETSc, but we just haven't had the time. Matt and I don't have a direct use case and we all have plenty of other things going. On Mon, Mar 26, 2012 at 13:53, Dag Sverre Seljebotn < d.s.seljebotn at astro.uio.no> wrote: > Hi list, > > I'm wondering whether there's any code in PETSc that I can either use > directly or lift out and adapt for my purposes (which are described in > footnote [1]). > > What I need to do is a number of different 2D array ("multi-vector", > "dense matrix") repacking operations, in order to make the data available > to different operations (spherical harmonic transforms, dense linear > algebra, sparse linear algebra, etc.). > > There's a number of repacking operations I'm in need of: > > - From each process having a given contiguous set of rows of a 2D array, > to an element-cyclic distribution where one distributes over both rows and > columns (the distribution of Elemental [2]) > > - Arbitrary redistribution of rows between processes (but each row is > kept in a process), potentially with overlaps. > > - More structured redistribution of rows between processes where indices > can be easily computed on the fly (but still a pattern quite specific to my > application). > > While I found PETSc routines for arbitrary redistribution of 1D arrays, I > couldn't find anything for 2D arrays. Any pointers into documentation or > papers etc. explaining these aspects of PETSc is very welcome. > It sounds like you just want help with indexing. VecScatter can move data of any dimensionality, you just have to know how to index it. (You could ask for something more specialized if you were using the topology routines in MPI and wanted control over how messages are routed.) > > What I'd *really* like (and what I'll do if I end up doing it myself) is > something more generic: A small, focused library which, given two > descriptors of how an N-dimensional array is distributed, does the > redistribution from one to the other. What I'd imagine is that you for each > axis of the array pick an axis distribution ("chunked", "(block-)cyclic", > "provided by indices", "non-distributed"), and an MPI datatype. (The array > would have to be distributed on a process grid of the same dimension as the > array). > > Then, given two N-dimensions descriptors which mixes structured and > unstructured information, the library can make and execute a redistribution > plan. > > Perhaps too ambitious, which is why I'm hoping that somebody can point me > to at least something existing. I can imagine doing most of it from > scratch, but reusing PETSc for the unstructured axis-distribution type. > > Comments of any kind welcome. > > Dag > > [1] What I'm doing is solving a linear system by CG where the LHS matrix > can only be written as a (rather large) symbolic equation, containing lots > of spherical harmonic transforms, Fourier transform, both dense and sparse > linear algebra, and so on. > > [2] http://code.google.com/p/**elemental/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.s.seljebotn at astro.uio.no Mon Mar 26 14:15:15 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Mon, 26 Mar 2012 12:15:15 -0700 Subject: [petsc-users] Repacking/scattering of multi-dimensional arrays In-Reply-To: References: <4F70BB0F.1080508@astro.uio.no> Message-ID: <4F70C043.1000200@astro.uio.no> On 03/26/2012 12:06 PM, Jed Brown wrote: > Matt, Jack, and myself have discussed adding proper support for > Elemental with PETSc, but we just haven't had the time. Matt and I don't > have a direct use case and we all have plenty of other things going. > > On Mon, Mar 26, 2012 at 13:53, Dag Sverre Seljebotn > > wrote: > > Hi list, > > I'm wondering whether there's any code in PETSc that I can either > use directly or lift out and adapt for my purposes (which are > described in footnote [1]). > > What I need to do is a number of different 2D array ("multi-vector", > "dense matrix") repacking operations, in order to make the data > available to different operations (spherical harmonic transforms, > dense linear algebra, sparse linear algebra, etc.). > > There's a number of repacking operations I'm in need of: > > - From each process having a given contiguous set of rows of a 2D > array, to an element-cyclic distribution where one distributes over > both rows and columns (the distribution of Elemental [2]) > > - Arbitrary redistribution of rows between processes (but each row > is kept in a process), potentially with overlaps. > > - More structured redistribution of rows between processes where > indices can be easily computed on the fly (but still a pattern quite > specific to my application). > > While I found PETSc routines for arbitrary redistribution of 1D > arrays, I couldn't find anything for 2D arrays. Any pointers into > documentation or papers etc. explaining these aspects of PETSc is > very welcome. > > > It sounds like you just want help with indexing. VecScatter can move > data of any dimensionality, you just have to know how to index it. (You > could ask for something more specialized if you were using the topology > routines in MPI and wanted control over how messages are routed.) Thanks for your response. Now I know what question to ask: The problem with VecScatter is that it requires indexing all elements, which can be rather wasteful in some situations. So I'm wondering if VecScatter could potentially work with elements of any size, so that one could send a bunch of elements (a picture of this is only indexing rows of a row-major 2D array, thus amortizing the indexing overhead and memory use by the number of columns). (That doesn't cover the Elemental case, where one would need to index every element (rather wasteful), but it does cover some of my other needs). Dag From knepley at gmail.com Mon Mar 26 14:18:36 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 26 Mar 2012 14:18:36 -0500 Subject: [petsc-users] Repacking/scattering of multi-dimensional arrays In-Reply-To: <4F70C043.1000200@astro.uio.no> References: <4F70BB0F.1080508@astro.uio.no> <4F70C043.1000200@astro.uio.no> Message-ID: On Mon, Mar 26, 2012 at 2:15 PM, Dag Sverre Seljebotn < d.s.seljebotn at astro.uio.no> wrote: > On 03/26/2012 12:06 PM, Jed Brown wrote: > >> Matt, Jack, and myself have discussed adding proper support for >> Elemental with PETSc, but we just haven't had the time. Matt and I don't >> have a direct use case and we all have plenty of other things going. >> >> On Mon, Mar 26, 2012 at 13:53, Dag Sverre Seljebotn >> >> >> wrote: >> >> Hi list, >> >> I'm wondering whether there's any code in PETSc that I can either >> use directly or lift out and adapt for my purposes (which are >> described in footnote [1]). >> >> What I need to do is a number of different 2D array ("multi-vector", >> "dense matrix") repacking operations, in order to make the data >> available to different operations (spherical harmonic transforms, >> dense linear algebra, sparse linear algebra, etc.). >> >> There's a number of repacking operations I'm in need of: >> >> - From each process having a given contiguous set of rows of a 2D >> array, to an element-cyclic distribution where one distributes over >> both rows and columns (the distribution of Elemental [2]) >> >> - Arbitrary redistribution of rows between processes (but each row >> is kept in a process), potentially with overlaps. >> >> - More structured redistribution of rows between processes where >> indices can be easily computed on the fly (but still a pattern quite >> specific to my application). >> >> While I found PETSc routines for arbitrary redistribution of 1D >> arrays, I couldn't find anything for 2D arrays. Any pointers into >> documentation or papers etc. explaining these aspects of PETSc is >> very welcome. >> >> >> It sounds like you just want help with indexing. VecScatter can move >> data of any dimensionality, you just have to know how to index it. (You >> could ask for something more specialized if you were using the topology >> routines in MPI and wanted control over how messages are routed.) >> > > Thanks for your response. Now I know what question to ask: > > The problem with VecScatter is that it requires indexing all elements, > which can be rather wasteful in some situations. So I'm wondering if > VecScatter could potentially work with elements of any size, so that one > could send a bunch of elements (a picture of this is only indexing rows of > a row-major 2D array, thus amortizing the indexing overhead and memory use > by the number of columns). > Use PetscSF instead, which can use any MPIDataatype. Its the VecScatter generalization. Matt > (That doesn't cover the Elemental case, where one would need to index > every element (rather wasteful), but it does cover some of my other needs). > > Dag > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Mar 26 14:22:52 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 26 Mar 2012 14:22:52 -0500 Subject: [petsc-users] Repacking/scattering of multi-dimensional arrays In-Reply-To: <4F70C043.1000200@astro.uio.no> References: <4F70BB0F.1080508@astro.uio.no> <4F70C043.1000200@astro.uio.no> Message-ID: On Mon, Mar 26, 2012 at 14:15, Dag Sverre Seljebotn < d.s.seljebotn at astro.uio.no> wrote: > The problem with VecScatter is that it requires indexing all elements, > which can be rather wasteful in some situations. So I'm wondering if > VecScatter could potentially work with elements of any size, so that one > could send a bunch of elements (a picture of this is only indexing rows of > a row-major 2D array, thus amortizing the indexing overhead and memory use > by the number of columns). IS is abstract, it doesn't have to enumerate every entry. http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/IS/ISCreateStride.html http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/IS/ISCreateBlock.html As Matt says, PetscSF has a similarly convenient API (although somewhat lower level), but is more flexible. It requires MPI-2 right now because it uses one-sided communication (low setup cost), but I plan to write an implementation over point-to-point so that it would have similar flexibility to in which MPI calls actually get used. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdiso at ustc.edu Tue Mar 27 03:23:08 2012 From: gdiso at ustc.edu (Gong Ding) Date: Tue, 27 Mar 2012 16:23:08 +0800 (CST) Subject: [petsc-users] KSPSetOptionsPrefix usage Message-ID: <7524995.104361332836588897.JavaMail.coremail@mail.ustc.edu> Hi I now use KSPSetOptionsPrefix to avoid option conflict to several KSP objects. I know KSPSetOptionsPrefix will call PCSetOptionsPrefix, PC options can be safely added the same prefix. However, what about the matrix prefix? i.e. I would like to use following code ierr = PCSetType (pc, (char*) PCILU); ierr = PCFactorSetMatSolverPackage (pc, "superlu"); ierr = PetscOptionsSetValue("-mat_superlu_rowperm","LargeDiag"); ierr = PetscOptionsSetValue("-mat_superlu_ilu_droptol","1e-4"); ierr = PetscOptionsSetValue("-mat_superlu_ilu_filltol","1e-2"); ierr = PetscOptionsSetValue("-mat_superlu_ilu_fillfactor","20"); Suppose I use prefix as "KSP1_", should I change mat option name to, i.e. -KSP1_mat_superlu_rowperm? Gong Ding From jarunan at ascomp.ch Tue Mar 27 09:32:49 2012 From: jarunan at ascomp.ch (Jarunan Panyasantisuk) Date: Tue, 27 Mar 2012 16:32:49 +0200 Subject: [petsc-users] Preconditioners memory use Message-ID: <4F71CF91.80600@ascomp.ch> Dear Petsc List, In case you have an experience about precondtiioners performance tests: Does the preconditioner Block Jacobi generally consume memory more than HYPRE/BoomerAMG? I use GMRES as a solver and -log_summary to print out the performance result. The results of my test case shows that BJACOBI uses more memory than AMG. I thought it is other way round. If you have similar results, please share your experience. Thank you Jarunan From jedbrown at mcs.anl.gov Tue Mar 27 10:17:46 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 27 Mar 2012 09:17:46 -0600 Subject: [petsc-users] Preconditioners memory use In-Reply-To: <4F71CF91.80600@ascomp.ch> References: <4F71CF91.80600@ascomp.ch> Message-ID: On Tue, Mar 27, 2012 at 08:32, Jarunan Panyasantisuk wrote: > In case you have an experience about precondtiioners performance tests: > Does the preconditioner Block Jacobi generally consume memory more than > HYPRE/BoomerAMG? I use GMRES as a solver and -log_summary to print out the > performance result. The results of my test case shows that BJACOBI uses > more memory than AMG. I thought it is other way round. If you have similar > results, please share your experience. BoomerAMG uses much more memory (unless you use direct subdomain solves, depending on size), but BoomerAMG does not use PetscMalloc, so PETSc can't reliably monitor how much memory it is using. You can use PetscMemoryGetCurrentUsage() to ask the system how much memory it has given to the application. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Tue Mar 27 10:30:34 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 27 Mar 2012 10:30:34 -0500 Subject: [petsc-users] KSPSetOptionsPrefix usage In-Reply-To: <7524995.104361332836588897.JavaMail.coremail@mail.ustc.edu> References: <7524995.104361332836588897.JavaMail.coremail@mail.ustc.edu> Message-ID: 2012/3/27 Gong Ding > Hi > I now use KSPSetOptionsPrefix to avoid option conflict to several KSP > objects. > I know KSPSetOptionsPrefix will call PCSetOptionsPrefix, PC options can be > safely added the same prefix. > However, what about the matrix prefix? i.e. I would like to use following > code > > ierr = PCSetType (pc, (char*) PCILU); > ierr = PCFactorSetMatSolverPackage (pc, "superlu"); > ierr = PetscOptionsSetValue("-mat_superlu_rowperm","LargeDiag"); > ierr = PetscOptionsSetValue("-mat_superlu_ilu_droptol","1e-4"); > ierr = PetscOptionsSetValue("-mat_superlu_ilu_filltol","1e-2"); > ierr = PetscOptionsSetValue("-mat_superlu_ilu_fillfactor","20"); > > Suppose I use prefix as "KSP1_", should I change mat option name to, i.e. > -KSP1_mat_superlu_rowperm? > No, you still use '-mat_superlu_rowperm'. You can add ierr = MatSetOptionsPrefix(mat,"KSP1_");CHKERRQ(ierr); to use '-KSP1_mat_superlu_rowperm'. Hong > > Gong Ding > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuentesdt at gmail.com Tue Mar 27 12:58:49 2012 From: fuentesdt at gmail.com (David Fuentes) Date: Tue, 27 Mar 2012 12:58:49 -0500 Subject: [petsc-users] ex52_integrateElement.cu Message-ID: Hi, I had a question about the status of example 52. http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu Can this example be used with a DM object created from an unstructured exodusII mesh, DMMeshCreateExodus, And the FEM assembly done on GPU ? Thanks, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Mar 27 13:23:07 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 27 Mar 2012 13:23:07 -0500 Subject: [petsc-users] ex52_integrateElement.cu In-Reply-To: References: Message-ID: On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes wrote: > Hi, > > I had a question about the status of example 52. > > > http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c > > http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu > > > Can this example be used with a DM object created from an unstructured > exodusII mesh, DMMeshCreateExodus, And the FEM assembly done on GPU ? > 1) I have pushed many more tests for it now. They can be run using the Python build system ./config/builder2.py check src/snes/examples/tutorials/ex52.c in fact, you can build any set of files this way. 2) The Exodus creation has to be converted to DMComplex from DMMesh. That should not take me very long. Blaise maintains that so maybe there will be help :) You will just replace DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request it, I will bump it up the list. Let me know if you can run the tests. Thanks Matt > Thanks, > David > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bourdin at lsu.edu Tue Mar 27 14:10:55 2012 From: bourdin at lsu.edu (Blaise Bourdin) Date: Tue, 27 Mar 2012 14:10:55 -0500 Subject: [petsc-users] ex52_integrateElement.cu In-Reply-To: References: Message-ID: <99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu> On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote: > On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes wrote: > Hi, > > I had a question about the status of example 52. > > http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c > http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu > > Can this example be used with a DM object created from an unstructured exodusII mesh, DMMeshCreateExodus, And the FEM assembly done on GPU ? > > 1) I have pushed many more tests for it now. They can be run using the Python build system > > ./config/builder2.py check src/snes/examples/tutorials/ex52.c > > in fact, you can build any set of files this way. > > 2) The Exodus creation has to be converted to DMComplex from DMMesh. That should not take me very long. Blaise maintains that > so maybe there will be help :) You will just replace DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request > it, I will bump it up the list. DMMeshCreateExodusNG is much more flexible than DMMeshCreateExodus in that it can read meshes with multiple element types and should have a much lower memory footprint. The code should be fairly easy to read. you can email me directly if you have specific questions. I had looked at creating a DMComplex and it did not look too difficult, as long as interpolation is not needed. I have plans to write DMComplexCreateExodus, but haven't had time too so far. Updating the Vec viewers and readers may be a bit more involved. In perfect world, one would write an EXODUS viewer following the lines of the VTK and HDF5 ones. Blaise > > Let me know if you can run the tests. > > Thanks > > Matt > > Thanks, > David > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener -- Department of Mathematics and Center for Computation & Technology Louisiana State University, Baton Rouge, LA 70803, USA Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuentesdt at gmail.com Tue Mar 27 19:18:35 2012 From: fuentesdt at gmail.com (David Fuentes) Date: Tue, 27 Mar 2012 19:18:35 -0500 Subject: [petsc-users] ex52_integrateElement.cu In-Reply-To: <99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu> References: <99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu> Message-ID: thanks! when generating the header file ex52.h what should the input arguments be to the PetscGenerateFEMQuadrature.py file ? SCRGP2$ $PETSC_DIR/bin/pythonscripts/PetscGenerateFEMQuadrature.py 3 1 8 1 laplacian ex52.h [{(-1.0, -1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0, -1.0): [(1.0, ())]}, {(-1.0, 1.0, -1.0): [(1.0, ())]}, {(-1.0, -1.0, 1.0): [(1.0, ())]}] {0: {0: [0], 1: [1], 2: [2], 3: [3]}, 1: {0: [], 1: [], 2: [], 3: [], 4: [], 5: []}, 2: {0: [], 1: [], 2: [], 3: []}, 3: {0: []}} Perm: [0, 1, 2, 3] Creating /home/fuentes/snestutorials/ex52.h Traceback (most recent call last): File "/opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py", line 33, in generator.run(elements, numBlocks, operator, filename) File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 967, in run self.outputElementSource(self.getElementSource(elements, numBlocks, operator, sourceType = 'GPU'), os.path.splitext(filename)[0]+'_gpu'+os.path.splitext(filename)[1]) File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 926, in getElementSource defns.extend(self.getComputationTypes(element, n)) File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 440, in getComputationTypes vecType.type = self.Cxx.typeMap['float'+str(dim)] KeyError: 'float3' On Tue, Mar 27, 2012 at 2:10 PM, Blaise Bourdin wrote: > > On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote: > > On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes wrote: > >> Hi, >> >> I had a question about the status of example 52. >> >> >> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c >> >> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu >> >> >> Can this example be used with a DM object created from an unstructured >> exodusII mesh, DMMeshCreateExodus, And the FEM assembly done on GPU ? >> > > 1) I have pushed many more tests for it now. They can be run using the > Python build system > > ./config/builder2.py check src/snes/examples/tutorials/ex52.c > > in fact, you can build any set of files this way. > > 2) The Exodus creation has to be converted to DMComplex from DMMesh. That > should not take me very long. Blaise maintains that > so maybe there will be help :) You will just replace > DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request > it, I will bump it up the list. > > > DMMeshCreateExodusNG is much more flexible than DMMeshCreateExodus in that > it can read meshes with multiple element types and should have a much lower > memory footprint. The code should be fairly easy to read. you can email me > directly if you have specific questions. I had looked at creating a > DMComplex and it did not look too difficult, as long as interpolation is > not needed. I have plans to write DMComplexCreateExodus, but haven't had > time too so far. Updating the Vec viewers and readers may be a bit more > involved. In perfect world, one would write an EXODUS viewer following the > lines of the VTK and HDF5 ones. > > Blaise > > > > Let me know if you can run the tests. > > Thanks > > Matt > > >> Thanks, >> David >> > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- > Department of Mathematics and Center for Computation & Technology > Louisiana State University, Baton Rouge, LA 70803, USA > Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 > http://www.math.lsu.edu/~bourdin > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Mar 27 19:24:44 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 27 Mar 2012 19:24:44 -0500 Subject: [petsc-users] ex52_integrateElement.cu In-Reply-To: References: <99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu> Message-ID: On Tue, Mar 27, 2012 at 7:18 PM, David Fuentes wrote: > thanks! when generating the header file ex52.h > > what should the input arguments be to the PetscGenerateFEMQuadrature.py > file ? > Andy Terrel already busted me about this today. I pushed a new tarball to the ftp site, so this will work rm externalpackages/Generator ./$PETSC_ARCH/conf/reconfigure-${PETSC_ARCH}.py or if you are more sophisticated with Mercurial cd externalpackages/Generator hg revert --all hg pull -u https://bitbucket.org/knepley/code-generator Then it should run. Thanks, Matt > SCRGP2$ $PETSC_DIR/bin/pythonscripts/PetscGenerateFEMQuadrature.py 3 1 8 1 > laplacian ex52.h > [{(-1.0, -1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0, -1.0): [(1.0, ())]}, > {(-1.0, 1.0, -1.0): [(1.0, ())]}, {(-1.0, -1.0, 1.0): [(1.0, ())]}] > {0: {0: [0], 1: [1], 2: [2], 3: [3]}, 1: {0: [], 1: [], 2: [], 3: [], 4: > [], 5: []}, 2: {0: [], 1: [], 2: [], 3: []}, 3: {0: []}} > Perm: [0, 1, 2, 3] > Creating /home/fuentes/snestutorials/ex52.h > Traceback (most recent call last): > File > "/opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py", > line 33, in > generator.run(elements, numBlocks, operator, filename) > File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 967, in run > self.outputElementSource(self.getElementSource(elements, numBlocks, > operator, sourceType = 'GPU'), > os.path.splitext(filename)[0]+'_gpu'+os.path.splitext(filename)[1]) > File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 926, in > getElementSource > defns.extend(self.getComputationTypes(element, n)) > File "/opt/apps/PETSC/petsc-dev/config/PETSc/FEM.py", line 440, in > getComputationTypes > vecType.type = self.Cxx.typeMap['float'+str(dim)] > KeyError: 'float3' > > > > > On Tue, Mar 27, 2012 at 2:10 PM, Blaise Bourdin wrote: > >> >> On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote: >> >> On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes wrote: >> >>> Hi, >>> >>> I had a question about the status of example 52. >>> >>> >>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c >>> >>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu >>> >>> >>> Can this example be used with a DM object created from an unstructured >>> exodusII mesh, DMMeshCreateExodus, And the FEM assembly done on GPU ? >>> >> >> 1) I have pushed many more tests for it now. They can be run using the >> Python build system >> >> ./config/builder2.py check src/snes/examples/tutorials/ex52.c >> >> in fact, you can build any set of files this way. >> >> 2) The Exodus creation has to be converted to DMComplex from DMMesh. That >> should not take me very long. Blaise maintains that >> so maybe there will be help :) You will just replace >> DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request >> it, I will bump it up the list. >> >> >> DMMeshCreateExodusNG is much more flexible than DMMeshCreateExodus in >> that it can read meshes with multiple element types and should have a much >> lower memory footprint. The code should be fairly easy to read. you can >> email me directly if you have specific questions. I had looked at creating >> a DMComplex and it did not look too difficult, as long as interpolation is >> not needed. I have plans to write DMComplexCreateExodus, but haven't had >> time too so far. Updating the Vec viewers and readers may be a bit more >> involved. In perfect world, one would write an EXODUS viewer following the >> lines of the VTK and HDF5 ones. >> >> Blaise >> >> >> >> Let me know if you can run the tests. >> >> Thanks >> >> Matt >> >> >>> Thanks, >>> David >>> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> -- >> Department of Mathematics and Center for Computation & Technology >> Louisiana State University, Baton Rouge, LA 70803, USA >> Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 >> http://www.math.lsu.edu/~bourdin >> >> >> >> >> >> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Mar 27 20:37:18 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 27 Mar 2012 20:37:18 -0500 Subject: [petsc-users] ex52_integrateElement.cu In-Reply-To: <99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu> References: <99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu> Message-ID: On Tue, Mar 27, 2012 at 2:10 PM, Blaise Bourdin wrote: > > On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote: > > On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes wrote: > >> Hi, >> >> I had a question about the status of example 52. >> >> >> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c >> >> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu >> >> >> Can this example be used with a DM object created from an unstructured >> exodusII mesh, DMMeshCreateExodus, And the FEM assembly done on GPU ? >> > > 1) I have pushed many more tests for it now. They can be run using the > Python build system > > ./config/builder2.py check src/snes/examples/tutorials/ex52.c > > in fact, you can build any set of files this way. > > 2) The Exodus creation has to be converted to DMComplex from DMMesh. That > should not take me very long. Blaise maintains that > so maybe there will be help :) You will just replace > DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request > it, I will bump it up the list. > > > DMMeshCreateExodusNG is much more flexible than DMMeshCreateExodus in that > it can read meshes with multiple element types and should have a much lower > memory footprint. The code should be fairly easy to read. you can email me > directly if you have specific questions. I had looked at creating a > DMComplex and it did not look too difficult, as long as interpolation is > not needed. I have plans to write DMComplexCreateExodus, but haven't had > time too so far. Updating the Vec viewers and readers may be a bit more > involved. In perfect world, one would write an EXODUS viewer following the > lines of the VTK and HDF5 ones. > David and Blaise, I have converted this function, now DMComplexCreateExodus(). Its not tested, but I think Blaise has some stuff we can use to test it. Thanks, Matt > Blaise > > > > Let me know if you can run the tests. > > Thanks > > Matt > > >> Thanks, >> David >> > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- > Department of Mathematics and Center for Computation & Technology > Louisiana State University, Baton Rouge, LA 70803, USA > Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 > http://www.math.lsu.edu/~bourdin > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From fredva at ifi.uio.no Wed Mar 28 07:30:05 2012 From: fredva at ifi.uio.no (Fredrik Heffer Valdmanis) Date: Wed, 28 Mar 2012 14:30:05 +0200 Subject: [petsc-users] Statistics on the popularity of PETSc Message-ID: Hi, Do you guys have any recent up-to-date numbers on for instance: - The number of PETSc downloads per month - The number of projects that use PETSc - The number of users of PETSc or anything else that may be used when discussing the popularity of PETSc? I have seen some numbers on Matt's home page regarding hits on Google, references on Google Books etc., so I can easily get updated such numbers myself, but I would greatly appreciate it if you have anything to share with me. Thanks, Fredrik -------------- next part -------------- An HTML attachment was scrubbed... URL: From recrusader at gmail.com Wed Mar 28 10:46:30 2012 From: recrusader at gmail.com (recrusader) Date: Wed, 28 Mar 2012 10:46:30 -0500 Subject: [petsc-users] Can I reorder the matrix for iterative methods? Message-ID: Dear PETSc developers, In order to reorder the matrix, I just find relevant functions for LU decomposition in "PC". My question is whether i can reorder the matrix and then use iterative method like gmres. If it works, how to do it? Thank you very much. Best, Yujie -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Mar 28 10:50:43 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 28 Mar 2012 10:50:43 -0500 Subject: [petsc-users] Can I reorder the matrix for iterative methods? In-Reply-To: References: Message-ID: On Wed, Mar 28, 2012 at 10:46 AM, recrusader wrote: > Dear PETSc developers, > > > In order to reorder the matrix, I just find relevant functions for LU > decomposition in "PC". > My question is whether i can reorder the matrix and then use iterative > method like gmres. > If it works, how to do it? Thank you very much. > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/MatOrderings/MatGetOrdering.html Matt > Best, > Yujie > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Mar 28 11:00:27 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 28 Mar 2012 11:00:27 -0500 Subject: [petsc-users] Can I reorder the matrix for iterative methods? In-Reply-To: References: Message-ID: You can also use runtime option: -pc_factor_mat_ordering_type supported types: natural nd 1wd rcm qmd rowlength Hong On Wed, Mar 28, 2012 at 10:50 AM, Matthew Knepley wrote: > On Wed, Mar 28, 2012 at 10:46 AM, recrusader wrote: > >> Dear PETSc developers, >> >> >> In order to reorder the matrix, I just find relevant functions for LU >> decomposition in "PC". >> My question is whether i can reorder the matrix and then use iterative >> method like gmres. >> If it works, how to do it? Thank you very much. >> > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/MatOrderings/MatGetOrdering.html > > Matt > > >> Best, >> Yujie >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From recrusader at gmail.com Wed Mar 28 11:07:26 2012 From: recrusader at gmail.com (recrusader) Date: Wed, 28 Mar 2012 11:07:26 -0500 Subject: [petsc-users] Can I reorder the matrix for iterative methods? In-Reply-To: References: Message-ID: Thank you very much, Matt and Hong. Best, Yujie On Wed, Mar 28, 2012 at 11:00 AM, Hong Zhang wrote: > You can also use runtime option: > -pc_factor_mat_ordering_type > supported types: > natural nd 1wd rcm qmd rowlength > > Hong > > > > On Wed, Mar 28, 2012 at 10:50 AM, Matthew Knepley wrote: > >> On Wed, Mar 28, 2012 at 10:46 AM, recrusader wrote: >> >>> Dear PETSc developers, >>> >>> >>> In order to reorder the matrix, I just find relevant functions for LU >>> decomposition in "PC". >>> My question is whether i can reorder the matrix and then use iterative >>> method like gmres. >>> If it works, how to do it? Thank you very much. >>> >> >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/MatOrderings/MatGetOrdering.html >> >> Matt >> >> >>> Best, >>> Yujie >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuentesdt at gmail.com Wed Mar 28 11:45:14 2012 From: fuentesdt at gmail.com (David Fuentes) Date: Wed, 28 Mar 2012 11:45:14 -0500 Subject: [petsc-users] ex52_integrateElement.cu In-Reply-To: References: <99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu> Message-ID: The example seems to be running on cpu with '-batch' but i'm getting errors in line 323 with the '-gpu' option [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in src/snes/examples/tutorials/ex52_integrateElement.cu should this possibly be PetscScalar ? - ierr = cudaMalloc((void**) &d_coefficients, Ne*N_bt * sizeof(float));CHKERRQ(ierr); + ierr = cudaMalloc((void**) &d_coefficients, Ne*N_bt * sizeof(PetscScalar));CHKERRQ(ierr); SCRGP2$ python $PETSC_DIR/bin/pythonscripts/PetscGenerateFEMQuadrature.py 2 1 1 1 laplacian ex52.h ['/opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py', '2', '1', '1', '1', 'laplacian', 'ex52.h'] 2 1 1 1 laplacian [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0, 1.0): [(1.0, ())]}] {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}} Perm: [0, 1, 2] Creating /home/fuentes/snestutorial/ex52.h Creating /home/fuentes/snestutorial/ex52_gpu.h [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0, 1.0): [(1.0, ())]}] {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}} Perm: [0, 1, 2] Creating /home/fuentes/snestutorial/ex52_gpu_inline.h SCRGP2$ make ex52 /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC -I/opt/apps/PETSC/petsc-dev/include -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve -I/opt/MATLAB/R2011a/extern/include -I/usr/include -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/ ex52.c nvcc -O0 -g -arch=sm_20 -c --compiler-options="-O0 -g -fPIC -I/opt/apps/PETSC/petsc-dev/include -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve -I/opt/MATLAB/R2011a/extern/include -I/usr/include -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/" ex52_integrateElement.cu ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never referenced ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never referenced ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never referenced ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but never referenced ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never referenced ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was declared but never referenced ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never referenced ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never referenced ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never referenced ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but never referenced ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never referenced ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was declared but never referenced /usr/bin/mpicxx -O0 -g -o ex52 ex52.o ex52_integrateElement.o -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib -lpetsc -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib -ltriangle -lX11 -lpthread -lsuperlu_dist_3.0 -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -Wl,-rpath,/opt/apps/cuda/4.1/cuda/lib64 -L/opt/apps/cuda/4.1/cuda/lib64 -lcufft -lcublas -lcudart -lcusparse -Wl,-rpath,/opt/MATLAB/R2011a/sys/os/glnxa64:/opt/MATLAB/R2011a/bin/glnxa64:/opt/MATLAB/R2011a/extern/lib/glnxa64 -L/opt/MATLAB/R2011a/bin/glnxa64 -L/opt/MATLAB/R2011a/extern/lib/glnxa64 -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 -lexoIIv2for -lexodus -lnetcdf_c++ -lnetcdf -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl /bin/rm -f ex52.o ex52_integrateElement.o SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch Residual: Vector Object: 1 MPI processes type: seq -0.25 -0.5 0.25 -0.5 -1 0.5 0.25 0.5 0.75 SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in src/snes/examples/tutorials/ex52_integrateElement.cu [0]PETSC ERROR: FormFunctionLocalBatch() line 679 in src/snes/examples/tutorials/ex52.c [0]PETSC ERROR: SNESDMComplexComputeFunction() line 431 in src/snes/utils/damgsnes.c [0]PETSC ERROR: main() line 1021 in src/snes/examples/tutorials/ex52.c application called MPI_Abort(MPI_COMM_WORLD, 35) - process 0 On Tue, Mar 27, 2012 at 8:37 PM, Matthew Knepley wrote: > On Tue, Mar 27, 2012 at 2:10 PM, Blaise Bourdin wrote: > >> >> On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote: >> >> On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes wrote: >> >>> Hi, >>> >>> I had a question about the status of example 52. >>> >>> >>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c >>> >>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu >>> >>> >>> Can this example be used with a DM object created from an unstructured >>> exodusII mesh, DMMeshCreateExodus, And the FEM assembly done on GPU ? >>> >> >> 1) I have pushed many more tests for it now. They can be run using the >> Python build system >> >> ./config/builder2.py check src/snes/examples/tutorials/ex52.c >> >> in fact, you can build any set of files this way. >> >> 2) The Exodus creation has to be converted to DMComplex from DMMesh. That >> should not take me very long. Blaise maintains that >> so maybe there will be help :) You will just replace >> DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request >> it, I will bump it up the list. >> >> >> DMMeshCreateExodusNG is much more flexible than DMMeshCreateExodus in >> that it can read meshes with multiple element types and should have a much >> lower memory footprint. The code should be fairly easy to read. you can >> email me directly if you have specific questions. I had looked at creating >> a DMComplex and it did not look too difficult, as long as interpolation is >> not needed. I have plans to write DMComplexCreateExodus, but haven't had >> time too so far. Updating the Vec viewers and readers may be a bit more >> involved. In perfect world, one would write an EXODUS viewer following the >> lines of the VTK and HDF5 ones. >> > > David and Blaise, I have converted this function, now > DMComplexCreateExodus(). Its not tested, but I think > Blaise has some stuff we can use to test it. > > Thanks, > > Matt > > >> Blaise >> >> >> >> Let me know if you can run the tests. >> >> Thanks >> >> Matt >> >> >>> Thanks, >>> David >>> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> -- >> Department of Mathematics and Center for Computation & Technology >> Louisiana State University, Baton Rouge, LA 70803, USA >> Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 >> http://www.math.lsu.edu/~bourdin >> >> >> >> >> >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Mar 28 11:54:24 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 28 Mar 2012 11:54:24 -0500 (CDT) Subject: [petsc-users] Statistics on the popularity of PETSc In-Reply-To: References: Message-ID: On Wed, 28 Mar 2012, Fredrik Heffer Valdmanis wrote: > Hi, > > Do you guys have any recent up-to-date numbers on for instance: > > - The number of PETSc downloads per month > - The number of projects that use PETSc > - The number of users of PETSc > > or anything else that may be used when discussing the popularity of PETSc? > I have seen some numbers on Matt's home page regarding hits on Google, > references on Google Books etc., so I can easily get updated such numbers > myself, but I would greatly appreciate it if you have anything to share > with me. I have the following statistics.. [2011] Total petsc tarball downloads : 43335 [2011] Unique machines requesting tarball downloads : 12979 [Today]Total unique e-mail ids in mailing lists : 1138 Regarding projects using PETSc - we have some incomplete info at: http://www.mcs.anl.gov/petsc/publications/index.html Satish From knepley at gmail.com Wed Mar 28 11:55:46 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 28 Mar 2012 11:55:46 -0500 Subject: [petsc-users] ex52_integrateElement.cu In-Reply-To: References: <99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu> Message-ID: On Wed, Mar 28, 2012 at 11:45 AM, David Fuentes wrote: > The example seems to be running on cpu with '-batch' but i'm getting > errors in line 323 with the '-gpu' option > > [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in > src/snes/examples/tutorials/ex52_integrateElement.cu > > should this possibly be PetscScalar ? > No. > - ierr = cudaMalloc((void**) &d_coefficients, Ne*N_bt * > sizeof(float));CHKERRQ(ierr); > + ierr = cudaMalloc((void**) &d_coefficients, Ne*N_bt * > sizeof(PetscScalar));CHKERRQ(ierr); > > > SCRGP2$ python $PETSC_DIR/bin/pythonscripts/PetscGenerateFEMQuadrature.py > 2 1 1 1 laplacian ex52.h > ['/opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py', > '2', '1', '1', '1', 'laplacian', 'ex52.h'] > 2 1 1 1 laplacian > [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0, 1.0): > [(1.0, ())]}] > {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}} > Perm: [0, 1, 2] > Creating /home/fuentes/snestutorial/ex52.h > Creating /home/fuentes/snestutorial/ex52_gpu.h > [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0, 1.0): > [(1.0, ())]}] > {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}} > Perm: [0, 1, 2] > Creating /home/fuentes/snestutorial/ex52_gpu_inline.h > > SCRGP2$ make ex52 > /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC > -I/opt/apps/PETSC/petsc-dev/include > -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include > -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve > -I/opt/MATLAB/R2011a/extern/include -I/usr/include > -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include > -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include > -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/ ex52.c > nvcc -O0 -g -arch=sm_20 -c --compiler-options="-O0 -g -fPIC > -I/opt/apps/PETSC/petsc-dev/include > -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include > -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve > -I/opt/MATLAB/R2011a/extern/include -I/usr/include > -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include > -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include > -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/" > ex52_integrateElement.cu > ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never > referenced > > ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never > referenced > > ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never > referenced > > ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but > never referenced > > ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never > referenced > > ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was declared > but never referenced > > ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never > referenced > > ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never > referenced > > ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never > referenced > > ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but > never referenced > > ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never > referenced > > ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was declared > but never referenced > > /usr/bin/mpicxx -O0 -g -o ex52 ex52.o ex52_integrateElement.o > -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib > -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib -lpetsc > -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib > -ltriangle -lX11 -lpthread -lsuperlu_dist_3.0 -lcmumps -ldmumps -lsmumps > -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs > -Wl,-rpath,/opt/apps/cuda/4.1/cuda/lib64 -L/opt/apps/cuda/4.1/cuda/lib64 > -lcufft -lcublas -lcudart -lcusparse > -Wl,-rpath,/opt/MATLAB/R2011a/sys/os/glnxa64:/opt/MATLAB/R2011a/bin/glnxa64:/opt/MATLAB/R2011a/extern/lib/glnxa64 > -L/opt/MATLAB/R2011a/bin/glnxa64 -L/opt/MATLAB/R2011a/extern/lib/glnxa64 > -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc > -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib > -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 -lexoIIv2for -lexodus > -lnetcdf_c++ -lnetcdf -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3 > -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt > -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx > -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl > /bin/rm -f ex52.o ex52_integrateElement.o > > > SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch > Residual: > Vector Object: 1 MPI processes > type: seq > -0.25 > -0.5 > 0.25 > -0.5 > -1 > 0.5 > 0.25 > 0.5 > 0.75 > SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu > [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in > src/snes/examples/tutorials/ex52_integrateElement.cu > [0]PETSC ERROR: FormFunctionLocalBatch() line 679 in > src/snes/examples/tutorials/ex52.c > [0]PETSC ERROR: SNESDMComplexComputeFunction() line 431 in > src/snes/utils/damgsnes.c > [0]PETSC ERROR: main() line 1021 in src/snes/examples/tutorials/ex52.c > application called MPI_Abort(MPI_COMM_WORLD, 35) - process 0 > This is failing on cudaMalloc(), which means your card is not available for running. Are you trying to run on your laptop? If so, applications like Preview can lock up the GPU. I know of no way to test this in CUDA while running. I just close apps until it runs. Thanks, Matt > > On Tue, Mar 27, 2012 at 8:37 PM, Matthew Knepley wrote: > >> On Tue, Mar 27, 2012 at 2:10 PM, Blaise Bourdin wrote: >> >>> >>> On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote: >>> >>> On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes wrote: >>> >>>> Hi, >>>> >>>> I had a question about the status of example 52. >>>> >>>> >>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c >>>> >>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu >>>> >>>> >>>> Can this example be used with a DM object created from an unstructured >>>> exodusII mesh, DMMeshCreateExodus, And the FEM assembly done on GPU ? >>>> >>> >>> 1) I have pushed many more tests for it now. They can be run using the >>> Python build system >>> >>> ./config/builder2.py check src/snes/examples/tutorials/ex52.c >>> >>> in fact, you can build any set of files this way. >>> >>> 2) The Exodus creation has to be converted to DMComplex from DMMesh. >>> That should not take me very long. Blaise maintains that >>> so maybe there will be help :) You will just replace >>> DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request >>> it, I will bump it up the list. >>> >>> >>> DMMeshCreateExodusNG is much more flexible than DMMeshCreateExodus in >>> that it can read meshes with multiple element types and should have a much >>> lower memory footprint. The code should be fairly easy to read. you can >>> email me directly if you have specific questions. I had looked at creating >>> a DMComplex and it did not look too difficult, as long as interpolation is >>> not needed. I have plans to write DMComplexCreateExodus, but haven't had >>> time too so far. Updating the Vec viewers and readers may be a bit more >>> involved. In perfect world, one would write an EXODUS viewer following the >>> lines of the VTK and HDF5 ones. >>> >> >> David and Blaise, I have converted this function, now >> DMComplexCreateExodus(). Its not tested, but I think >> Blaise has some stuff we can use to test it. >> >> Thanks, >> >> Matt >> >> >>> Blaise >>> >>> >>> >>> Let me know if you can run the tests. >>> >>> Thanks >>> >>> Matt >>> >>> >>>> Thanks, >>>> David >>>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >>> -- >>> Department of Mathematics and Center for Computation & Technology >>> Louisiana State University, Baton Rouge, LA 70803, USA >>> Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 >>> http://www.math.lsu.edu/~bourdin >>> >>> >>> >>> >>> >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuentesdt at gmail.com Wed Mar 28 13:14:50 2012 From: fuentesdt at gmail.com (David Fuentes) Date: Wed, 28 Mar 2012 13:14:50 -0500 Subject: [petsc-users] ex52_integrateElement.cu In-Reply-To: References: <99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu> Message-ID: thanks! its running, but I seem to be getting different answer for cpu/gpu ? i had some floating point problems on this Tesla M2070 gpu before, but adding the '-arch=sm_20' option seemed to fix it last time. is the assembly in single precision ? my 'const PetscReal jacobianInverse' being passed in are doubles SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu GPU layout grid(1,2,1) block(3,1,1) with 1 batches N_t: 3, N_cb: 1 Residual: Vector Object: 1 MPI processes type: seq 0 755712 0 -58720 -2953.13 0.375 1.50323e+07 0.875 0 SCRGP2$ SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch Residual: Vector Object: 1 MPI processes type: seq -0.25 -0.5 0.25 -0.5 -1 0.5 0.25 0.5 0.75 On Wed, Mar 28, 2012 at 11:55 AM, Matthew Knepley wrote: > On Wed, Mar 28, 2012 at 11:45 AM, David Fuentes wrote: > >> The example seems to be running on cpu with '-batch' but i'm getting >> errors in line 323 with the '-gpu' option >> >> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in >> src/snes/examples/tutorials/ex52_integrateElement.cu >> >> should this possibly be PetscScalar ? >> > > No. > > >> - ierr = cudaMalloc((void**) &d_coefficients, Ne*N_bt * >> sizeof(float));CHKERRQ(ierr); >> + ierr = cudaMalloc((void**) &d_coefficients, Ne*N_bt * >> sizeof(PetscScalar));CHKERRQ(ierr); >> >> >> SCRGP2$ python $PETSC_DIR/bin/pythonscripts/PetscGenerateFEMQuadrature.py >> 2 1 1 1 laplacian ex52.h >> ['/opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py', >> '2', '1', '1', '1', 'laplacian', 'ex52.h'] >> 2 1 1 1 laplacian >> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0, 1.0): >> [(1.0, ())]}] >> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}} >> Perm: [0, 1, 2] >> Creating /home/fuentes/snestutorial/ex52.h >> Creating /home/fuentes/snestutorial/ex52_gpu.h >> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0, 1.0): >> [(1.0, ())]}] >> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}} >> Perm: [0, 1, 2] >> Creating /home/fuentes/snestutorial/ex52_gpu_inline.h >> >> SCRGP2$ make ex52 >> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC >> -I/opt/apps/PETSC/petsc-dev/include >> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include >> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve >> -I/opt/MATLAB/R2011a/extern/include -I/usr/include >> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include >> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include >> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/ ex52.c >> nvcc -O0 -g -arch=sm_20 -c --compiler-options="-O0 -g -fPIC >> -I/opt/apps/PETSC/petsc-dev/include >> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include >> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve >> -I/opt/MATLAB/R2011a/extern/include -I/usr/include >> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include >> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include >> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/" >> ex52_integrateElement.cu >> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never >> referenced >> >> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never >> referenced >> >> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never >> referenced >> >> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but >> never referenced >> >> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never >> referenced >> >> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was >> declared but never referenced >> >> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never >> referenced >> >> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never >> referenced >> >> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never >> referenced >> >> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but >> never referenced >> >> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never >> referenced >> >> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was >> declared but never referenced >> >> /usr/bin/mpicxx -O0 -g -o ex52 ex52.o ex52_integrateElement.o >> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib >> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib -lpetsc >> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib >> -ltriangle -lX11 -lpthread -lsuperlu_dist_3.0 -lcmumps -ldmumps -lsmumps >> -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs >> -Wl,-rpath,/opt/apps/cuda/4.1/cuda/lib64 -L/opt/apps/cuda/4.1/cuda/lib64 >> -lcufft -lcublas -lcudart -lcusparse >> -Wl,-rpath,/opt/MATLAB/R2011a/sys/os/glnxa64:/opt/MATLAB/R2011a/bin/glnxa64:/opt/MATLAB/R2011a/extern/lib/glnxa64 >> -L/opt/MATLAB/R2011a/bin/glnxa64 -L/opt/MATLAB/R2011a/extern/lib/glnxa64 >> -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc >> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib >> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 -lexoIIv2for -lexodus >> -lnetcdf_c++ -lnetcdf -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3 >> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt >> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx >> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl >> /bin/rm -f ex52.o ex52_integrateElement.o >> >> >> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch >> Residual: >> Vector Object: 1 MPI processes >> type: seq >> -0.25 >> -0.5 >> 0.25 >> -0.5 >> -1 >> 0.5 >> 0.25 >> 0.5 >> 0.75 >> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu >> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in >> src/snes/examples/tutorials/ex52_integrateElement.cu >> [0]PETSC ERROR: FormFunctionLocalBatch() line 679 in >> src/snes/examples/tutorials/ex52.c >> [0]PETSC ERROR: SNESDMComplexComputeFunction() line 431 in >> src/snes/utils/damgsnes.c >> [0]PETSC ERROR: main() line 1021 in src/snes/examples/tutorials/ex52.c >> application called MPI_Abort(MPI_COMM_WORLD, 35) - process 0 >> > > This is failing on cudaMalloc(), which means your card is not available > for running. Are you trying to run on your laptop? > If so, applications like Preview can lock up the GPU. I know of no way to > test this in CUDA while running. I just close > apps until it runs. > > Thanks, > > Matt > > >> >> On Tue, Mar 27, 2012 at 8:37 PM, Matthew Knepley wrote: >> >>> On Tue, Mar 27, 2012 at 2:10 PM, Blaise Bourdin wrote: >>> >>>> >>>> On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote: >>>> >>>> On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes wrote: >>>> >>>>> Hi, >>>>> >>>>> I had a question about the status of example 52. >>>>> >>>>> >>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c >>>>> >>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu >>>>> >>>>> >>>>> Can this example be used with a DM object created from an unstructured >>>>> exodusII mesh, DMMeshCreateExodus, And the FEM assembly done on GPU ? >>>>> >>>> >>>> 1) I have pushed many more tests for it now. They can be run using the >>>> Python build system >>>> >>>> ./config/builder2.py check src/snes/examples/tutorials/ex52.c >>>> >>>> in fact, you can build any set of files this way. >>>> >>>> 2) The Exodus creation has to be converted to DMComplex from DMMesh. >>>> That should not take me very long. Blaise maintains that >>>> so maybe there will be help :) You will just replace >>>> DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request >>>> it, I will bump it up the list. >>>> >>>> >>>> DMMeshCreateExodusNG is much more flexible than DMMeshCreateExodus in >>>> that it can read meshes with multiple element types and should have a much >>>> lower memory footprint. The code should be fairly easy to read. you can >>>> email me directly if you have specific questions. I had looked at creating >>>> a DMComplex and it did not look too difficult, as long as interpolation is >>>> not needed. I have plans to write DMComplexCreateExodus, but haven't had >>>> time too so far. Updating the Vec viewers and readers may be a bit more >>>> involved. In perfect world, one would write an EXODUS viewer following the >>>> lines of the VTK and HDF5 ones. >>>> >>> >>> David and Blaise, I have converted this function, now >>> DMComplexCreateExodus(). Its not tested, but I think >>> Blaise has some stuff we can use to test it. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Blaise >>>> >>>> >>>> >>>> Let me know if you can run the tests. >>>> >>>> Thanks >>>> >>>> Matt >>>> >>>> >>>>> Thanks, >>>>> David >>>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> >>>> -- >>>> Department of Mathematics and Center for Computation & Technology >>>> Louisiana State University, Baton Rouge, LA 70803, USA >>>> Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 >>>> http://www.math.lsu.edu/~bourdin >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Mar 28 13:23:37 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 28 Mar 2012 13:23:37 -0500 Subject: [petsc-users] ex52_integrateElement.cu In-Reply-To: References: <99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu> Message-ID: On Wed, Mar 28, 2012 at 1:14 PM, David Fuentes wrote: > thanks! its running, but I seem to be getting different answer for cpu/gpu > ? > i had some floating point problems on this Tesla M2070 gpu before, but > adding the '-arch=sm_20' option seemed to fix it last time. > > > is the assembly in single precision ? my 'const PetscReal jacobianInverse' > being passed in are doubles > Yep, that is the problem. I have not tested anything in double. I have not decided exactly how to handle it. Can you make another ARCH --with-precision=single and make sure it works, and then we can fix the double issue? Thanks, Matt > SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu > GPU layout grid(1,2,1) block(3,1,1) with 1 batches > N_t: 3, N_cb: 1 > Residual: > Vector Object: 1 MPI processes > type: seq > 0 > 755712 > 0 > -58720 > -2953.13 > 0.375 > 1.50323e+07 > 0.875 > 0 > SCRGP2$ > SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch > Residual: > Vector Object: 1 MPI processes > type: seq > -0.25 > -0.5 > 0.25 > -0.5 > -1 > 0.5 > 0.25 > 0.5 > 0.75 > > > > > > On Wed, Mar 28, 2012 at 11:55 AM, Matthew Knepley wrote: > >> On Wed, Mar 28, 2012 at 11:45 AM, David Fuentes wrote: >> >>> The example seems to be running on cpu with '-batch' but i'm getting >>> errors in line 323 with the '-gpu' option >>> >>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in >>> src/snes/examples/tutorials/ex52_integrateElement.cu >>> >>> should this possibly be PetscScalar ? >>> >> >> No. >> >> >>> - ierr = cudaMalloc((void**) &d_coefficients, Ne*N_bt * >>> sizeof(float));CHKERRQ(ierr); >>> + ierr = cudaMalloc((void**) &d_coefficients, Ne*N_bt * >>> sizeof(PetscScalar));CHKERRQ(ierr); >>> >>> >>> SCRGP2$ python >>> $PETSC_DIR/bin/pythonscripts/PetscGenerateFEMQuadrature.py 2 1 1 1 >>> laplacian ex52.h >>> ['/opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py', >>> '2', '1', '1', '1', 'laplacian', 'ex52.h'] >>> 2 1 1 1 laplacian >>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0, 1.0): >>> [(1.0, ())]}] >>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}} >>> Perm: [0, 1, 2] >>> Creating /home/fuentes/snestutorial/ex52.h >>> Creating /home/fuentes/snestutorial/ex52_gpu.h >>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0, 1.0): >>> [(1.0, ())]}] >>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}} >>> Perm: [0, 1, 2] >>> Creating /home/fuentes/snestutorial/ex52_gpu_inline.h >>> >>> SCRGP2$ make ex52 >>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC >>> -I/opt/apps/PETSC/petsc-dev/include >>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include >>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve >>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include >>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include >>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include >>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/ ex52.c >>> nvcc -O0 -g -arch=sm_20 -c --compiler-options="-O0 -g -fPIC >>> -I/opt/apps/PETSC/petsc-dev/include >>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include >>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve >>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include >>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include >>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include >>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/" >>> ex52_integrateElement.cu >>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>> never referenced >>> >>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>> never referenced >>> >>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>> never referenced >>> >>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but >>> never referenced >>> >>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>> never referenced >>> >>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was >>> declared but never referenced >>> >>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>> never referenced >>> >>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>> never referenced >>> >>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>> never referenced >>> >>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but >>> never referenced >>> >>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>> never referenced >>> >>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was >>> declared but never referenced >>> >>> /usr/bin/mpicxx -O0 -g -o ex52 ex52.o ex52_integrateElement.o >>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib >>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib -lpetsc >>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib >>> -ltriangle -lX11 -lpthread -lsuperlu_dist_3.0 -lcmumps -ldmumps -lsmumps >>> -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs >>> -Wl,-rpath,/opt/apps/cuda/4.1/cuda/lib64 -L/opt/apps/cuda/4.1/cuda/lib64 >>> -lcufft -lcublas -lcudart -lcusparse >>> -Wl,-rpath,/opt/MATLAB/R2011a/sys/os/glnxa64:/opt/MATLAB/R2011a/bin/glnxa64:/opt/MATLAB/R2011a/extern/lib/glnxa64 >>> -L/opt/MATLAB/R2011a/bin/glnxa64 -L/opt/MATLAB/R2011a/extern/lib/glnxa64 >>> -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc >>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib >>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 -lexoIIv2for -lexodus >>> -lnetcdf_c++ -lnetcdf -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3 >>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt >>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx >>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl >>> /bin/rm -f ex52.o ex52_integrateElement.o >>> >>> >>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch >>> Residual: >>> Vector Object: 1 MPI processes >>> type: seq >>> -0.25 >>> -0.5 >>> 0.25 >>> -0.5 >>> -1 >>> 0.5 >>> 0.25 >>> 0.5 >>> 0.75 >>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu >>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in >>> src/snes/examples/tutorials/ex52_integrateElement.cu >>> [0]PETSC ERROR: FormFunctionLocalBatch() line 679 in >>> src/snes/examples/tutorials/ex52.c >>> [0]PETSC ERROR: SNESDMComplexComputeFunction() line 431 in >>> src/snes/utils/damgsnes.c >>> [0]PETSC ERROR: main() line 1021 in src/snes/examples/tutorials/ex52.c >>> application called MPI_Abort(MPI_COMM_WORLD, 35) - process 0 >>> >> >> This is failing on cudaMalloc(), which means your card is not available >> for running. Are you trying to run on your laptop? >> If so, applications like Preview can lock up the GPU. I know of no way to >> test this in CUDA while running. I just close >> apps until it runs. >> >> Thanks, >> >> Matt >> >> >>> >>> On Tue, Mar 27, 2012 at 8:37 PM, Matthew Knepley wrote: >>> >>>> On Tue, Mar 27, 2012 at 2:10 PM, Blaise Bourdin wrote: >>>> >>>>> >>>>> On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote: >>>>> >>>>> On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I had a question about the status of example 52. >>>>>> >>>>>> >>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c >>>>>> >>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu >>>>>> >>>>>> >>>>>> Can this example be used with a DM object created from an >>>>>> unstructured exodusII mesh, DMMeshCreateExodus, And the FEM assembly done >>>>>> on GPU ? >>>>>> >>>>> >>>>> 1) I have pushed many more tests for it now. They can be run using the >>>>> Python build system >>>>> >>>>> ./config/builder2.py check src/snes/examples/tutorials/ex52.c >>>>> >>>>> in fact, you can build any set of files this way. >>>>> >>>>> 2) The Exodus creation has to be converted to DMComplex from DMMesh. >>>>> That should not take me very long. Blaise maintains that >>>>> so maybe there will be help :) You will just replace >>>>> DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request >>>>> it, I will bump it up the list. >>>>> >>>>> >>>>> DMMeshCreateExodusNG is much more flexible than DMMeshCreateExodus in >>>>> that it can read meshes with multiple element types and should have a much >>>>> lower memory footprint. The code should be fairly easy to read. you can >>>>> email me directly if you have specific questions. I had looked at creating >>>>> a DMComplex and it did not look too difficult, as long as interpolation is >>>>> not needed. I have plans to write DMComplexCreateExodus, but haven't had >>>>> time too so far. Updating the Vec viewers and readers may be a bit more >>>>> involved. In perfect world, one would write an EXODUS viewer following the >>>>> lines of the VTK and HDF5 ones. >>>>> >>>> >>>> David and Blaise, I have converted this function, now >>>> DMComplexCreateExodus(). Its not tested, but I think >>>> Blaise has some stuff we can use to test it. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Blaise >>>>> >>>>> >>>>> >>>>> Let me know if you can run the tests. >>>>> >>>>> Thanks >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thanks, >>>>>> David >>>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> -- >>>>> Department of Mathematics and Center for Computation & Technology >>>>> Louisiana State University, Baton Rouge, LA 70803, USA >>>>> Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 >>>>> http://www.math.lsu.edu/~bourdin >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuentesdt at gmail.com Wed Mar 28 13:37:59 2012 From: fuentesdt at gmail.com (David Fuentes) Date: Wed, 28 Mar 2012 13:37:59 -0500 Subject: [petsc-users] ex52_integrateElement.cu In-Reply-To: References: <99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu> Message-ID: sure. will do. On Wed, Mar 28, 2012 at 1:23 PM, Matthew Knepley wrote: > On Wed, Mar 28, 2012 at 1:14 PM, David Fuentes wrote: > >> thanks! its running, but I seem to be getting different answer for >> cpu/gpu ? >> i had some floating point problems on this Tesla M2070 gpu before, but >> adding the '-arch=sm_20' option seemed to fix it last time. >> >> >> is the assembly in single precision ? my 'const PetscReal >> jacobianInverse' being passed in are doubles >> > > Yep, that is the problem. I have not tested anything in double. I have not > decided exactly how to handle it. Can you > make another ARCH --with-precision=single and make sure it works, and then > we can fix the double issue? > > Thanks, > > Matt > > >> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu >> GPU layout grid(1,2,1) block(3,1,1) with 1 batches >> N_t: 3, N_cb: 1 >> Residual: >> Vector Object: 1 MPI processes >> type: seq >> 0 >> 755712 >> 0 >> -58720 >> -2953.13 >> 0.375 >> 1.50323e+07 >> 0.875 >> 0 >> SCRGP2$ >> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch >> Residual: >> Vector Object: 1 MPI processes >> type: seq >> -0.25 >> -0.5 >> 0.25 >> -0.5 >> -1 >> 0.5 >> 0.25 >> 0.5 >> 0.75 >> >> >> >> >> >> On Wed, Mar 28, 2012 at 11:55 AM, Matthew Knepley wrote: >> >>> On Wed, Mar 28, 2012 at 11:45 AM, David Fuentes wrote: >>> >>>> The example seems to be running on cpu with '-batch' but i'm getting >>>> errors in line 323 with the '-gpu' option >>>> >>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in >>>> src/snes/examples/tutorials/ex52_integrateElement.cu >>>> >>>> should this possibly be PetscScalar ? >>>> >>> >>> No. >>> >>> >>>> - ierr = cudaMalloc((void**) &d_coefficients, Ne*N_bt * >>>> sizeof(float));CHKERRQ(ierr); >>>> + ierr = cudaMalloc((void**) &d_coefficients, Ne*N_bt * >>>> sizeof(PetscScalar));CHKERRQ(ierr); >>>> >>>> >>>> SCRGP2$ python >>>> $PETSC_DIR/bin/pythonscripts/PetscGenerateFEMQuadrature.py 2 1 1 1 >>>> laplacian ex52.h >>>> ['/opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py', >>>> '2', '1', '1', '1', 'laplacian', 'ex52.h'] >>>> 2 1 1 1 laplacian >>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0, 1.0): >>>> [(1.0, ())]}] >>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}} >>>> Perm: [0, 1, 2] >>>> Creating /home/fuentes/snestutorial/ex52.h >>>> Creating /home/fuentes/snestutorial/ex52_gpu.h >>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0, 1.0): >>>> [(1.0, ())]}] >>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}} >>>> Perm: [0, 1, 2] >>>> Creating /home/fuentes/snestutorial/ex52_gpu_inline.h >>>> >>>> SCRGP2$ make ex52 >>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC >>>> -I/opt/apps/PETSC/petsc-dev/include >>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include >>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve >>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include >>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include >>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include >>>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/ ex52.c >>>> nvcc -O0 -g -arch=sm_20 -c --compiler-options="-O0 -g -fPIC >>>> -I/opt/apps/PETSC/petsc-dev/include >>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include >>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve >>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include >>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include >>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include >>>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/" >>>> ex52_integrateElement.cu >>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>>> never referenced >>>> >>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>>> never referenced >>>> >>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>>> never referenced >>>> >>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but >>>> never referenced >>>> >>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>>> never referenced >>>> >>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was >>>> declared but never referenced >>>> >>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>>> never referenced >>>> >>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>>> never referenced >>>> >>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>>> never referenced >>>> >>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but >>>> never referenced >>>> >>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>>> never referenced >>>> >>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was >>>> declared but never referenced >>>> >>>> /usr/bin/mpicxx -O0 -g -o ex52 ex52.o ex52_integrateElement.o >>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib >>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib -lpetsc >>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib >>>> -ltriangle -lX11 -lpthread -lsuperlu_dist_3.0 -lcmumps -ldmumps -lsmumps >>>> -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs >>>> -Wl,-rpath,/opt/apps/cuda/4.1/cuda/lib64 -L/opt/apps/cuda/4.1/cuda/lib64 >>>> -lcufft -lcublas -lcudart -lcusparse >>>> -Wl,-rpath,/opt/MATLAB/R2011a/sys/os/glnxa64:/opt/MATLAB/R2011a/bin/glnxa64:/opt/MATLAB/R2011a/extern/lib/glnxa64 >>>> -L/opt/MATLAB/R2011a/bin/glnxa64 -L/opt/MATLAB/R2011a/extern/lib/glnxa64 >>>> -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc >>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib >>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 -lexoIIv2for -lexodus >>>> -lnetcdf_c++ -lnetcdf -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3 >>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt >>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx >>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl >>>> /bin/rm -f ex52.o ex52_integrateElement.o >>>> >>>> >>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch >>>> Residual: >>>> Vector Object: 1 MPI processes >>>> type: seq >>>> -0.25 >>>> -0.5 >>>> 0.25 >>>> -0.5 >>>> -1 >>>> 0.5 >>>> 0.25 >>>> 0.5 >>>> 0.75 >>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu >>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in >>>> src/snes/examples/tutorials/ex52_integrateElement.cu >>>> [0]PETSC ERROR: FormFunctionLocalBatch() line 679 in >>>> src/snes/examples/tutorials/ex52.c >>>> [0]PETSC ERROR: SNESDMComplexComputeFunction() line 431 in >>>> src/snes/utils/damgsnes.c >>>> [0]PETSC ERROR: main() line 1021 in src/snes/examples/tutorials/ex52.c >>>> application called MPI_Abort(MPI_COMM_WORLD, 35) - process 0 >>>> >>> >>> This is failing on cudaMalloc(), which means your card is not available >>> for running. Are you trying to run on your laptop? >>> If so, applications like Preview can lock up the GPU. I know of no way >>> to test this in CUDA while running. I just close >>> apps until it runs. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> >>>> On Tue, Mar 27, 2012 at 8:37 PM, Matthew Knepley wrote: >>>> >>>>> On Tue, Mar 27, 2012 at 2:10 PM, Blaise Bourdin wrote: >>>>> >>>>>> >>>>>> On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote: >>>>>> >>>>>> On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I had a question about the status of example 52. >>>>>>> >>>>>>> >>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c >>>>>>> >>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu >>>>>>> >>>>>>> >>>>>>> Can this example be used with a DM object created from an >>>>>>> unstructured exodusII mesh, DMMeshCreateExodus, And the FEM assembly done >>>>>>> on GPU ? >>>>>>> >>>>>> >>>>>> 1) I have pushed many more tests for it now. They can be run using >>>>>> the Python build system >>>>>> >>>>>> ./config/builder2.py check src/snes/examples/tutorials/ex52.c >>>>>> >>>>>> in fact, you can build any set of files this way. >>>>>> >>>>>> 2) The Exodus creation has to be converted to DMComplex from DMMesh. >>>>>> That should not take me very long. Blaise maintains that >>>>>> so maybe there will be help :) You will just replace >>>>>> DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request >>>>>> it, I will bump it up the list. >>>>>> >>>>>> >>>>>> DMMeshCreateExodusNG is much more flexible than DMMeshCreateExodus in >>>>>> that it can read meshes with multiple element types and should have a much >>>>>> lower memory footprint. The code should be fairly easy to read. you can >>>>>> email me directly if you have specific questions. I had looked at creating >>>>>> a DMComplex and it did not look too difficult, as long as interpolation is >>>>>> not needed. I have plans to write DMComplexCreateExodus, but haven't had >>>>>> time too so far. Updating the Vec viewers and readers may be a bit more >>>>>> involved. In perfect world, one would write an EXODUS viewer following the >>>>>> lines of the VTK and HDF5 ones. >>>>>> >>>>> >>>>> David and Blaise, I have converted this function, now >>>>> DMComplexCreateExodus(). Its not tested, but I think >>>>> Blaise has some stuff we can use to test it. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Blaise >>>>>> >>>>>> >>>>>> >>>>>> Let me know if you can run the tests. >>>>>> >>>>>> Thanks >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Thanks, >>>>>>> David >>>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> >>>>>> -- >>>>>> Department of Mathematics and Center for Computation & Technology >>>>>> Louisiana State University, Baton Rouge, LA 70803, USA >>>>>> Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 >>>>>> http://www.math.lsu.edu/~bourdin >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuentesdt at gmail.com Wed Mar 28 16:12:25 2012 From: fuentesdt at gmail.com (David Fuentes) Date: Wed, 28 Mar 2012 16:12:25 -0500 Subject: [petsc-users] ex52_integrateElement.cu In-Reply-To: References: <99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu> Message-ID: works! SCRGP2$ make ex52 /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC -I/opt/apps/PETSC/petsc-dev/include -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/ ex52.c nvcc -G -O0 -g -arch=sm_10 -c --compiler-options="-O0 -g -fPIC -I/opt/apps/PETSC/petsc-dev/include -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/" ex52_integrateElement.cu ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never referenced ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never referenced ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never referenced ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but never referenced ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never referenced ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was declared but never referenced ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never referenced ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never referenced ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never referenced ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but never referenced ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never referenced ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was declared but never referenced /usr/bin/mpicxx -O0 -g -o ex52 ex52.o ex52_integrateElement.o -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib -lpetsc -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib -ltriangle -lX11 -lpthread -lmetis -Wl,-rpath,/opt/apps/cuda/4.0/cuda/lib64 -L/opt/apps/cuda/4.0/cuda/lib64 -lcufft -lcublas -lcudart -lcusparse -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu GPU layout grid(1,2,1) block(3,1,1) with 1 batches N_t: 3, N_cb: 1 Residual: Vector Object: 1 MPI processes type: seq -0.25 -0.5 0.25 -0.5 -1 0.5 0.25 0.5 0.75 SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch Residual: Vector Object: 1 MPI processes type: seq -0.25 -0.5 0.25 -0.5 -1 0.5 0.25 0.5 0.75 On Wed, Mar 28, 2012 at 1:37 PM, David Fuentes wrote: > sure. will do. > > > On Wed, Mar 28, 2012 at 1:23 PM, Matthew Knepley wrote: > >> On Wed, Mar 28, 2012 at 1:14 PM, David Fuentes wrote: >> >>> thanks! its running, but I seem to be getting different answer for >>> cpu/gpu ? >>> i had some floating point problems on this Tesla M2070 gpu before, but >>> adding the '-arch=sm_20' option seemed to fix it last time. >>> >>> >>> is the assembly in single precision ? my 'const PetscReal >>> jacobianInverse' being passed in are doubles >>> >> >> Yep, that is the problem. I have not tested anything in double. I have >> not decided exactly how to handle it. Can you >> make another ARCH --with-precision=single and make sure it works, and >> then we can fix the double issue? >> >> Thanks, >> >> Matt >> >> >>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu >>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches >>> N_t: 3, N_cb: 1 >>> Residual: >>> Vector Object: 1 MPI processes >>> type: seq >>> 0 >>> 755712 >>> 0 >>> -58720 >>> -2953.13 >>> 0.375 >>> 1.50323e+07 >>> 0.875 >>> 0 >>> SCRGP2$ >>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch >>> Residual: >>> Vector Object: 1 MPI processes >>> type: seq >>> -0.25 >>> -0.5 >>> 0.25 >>> -0.5 >>> -1 >>> 0.5 >>> 0.25 >>> 0.5 >>> 0.75 >>> >>> >>> >>> >>> >>> On Wed, Mar 28, 2012 at 11:55 AM, Matthew Knepley wrote: >>> >>>> On Wed, Mar 28, 2012 at 11:45 AM, David Fuentes wrote: >>>> >>>>> The example seems to be running on cpu with '-batch' but i'm getting >>>>> errors in line 323 with the '-gpu' option >>>>> >>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in >>>>> src/snes/examples/tutorials/ex52_integrateElement.cu >>>>> >>>>> should this possibly be PetscScalar ? >>>>> >>>> >>>> No. >>>> >>>> >>>>> - ierr = cudaMalloc((void**) &d_coefficients, Ne*N_bt * >>>>> sizeof(float));CHKERRQ(ierr); >>>>> + ierr = cudaMalloc((void**) &d_coefficients, Ne*N_bt * >>>>> sizeof(PetscScalar));CHKERRQ(ierr); >>>>> >>>>> >>>>> SCRGP2$ python >>>>> $PETSC_DIR/bin/pythonscripts/PetscGenerateFEMQuadrature.py 2 1 1 1 >>>>> laplacian ex52.h >>>>> ['/opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py', >>>>> '2', '1', '1', '1', 'laplacian', 'ex52.h'] >>>>> 2 1 1 1 laplacian >>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0, >>>>> 1.0): [(1.0, ())]}] >>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}} >>>>> Perm: [0, 1, 2] >>>>> Creating /home/fuentes/snestutorial/ex52.h >>>>> Creating /home/fuentes/snestutorial/ex52_gpu.h >>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0, >>>>> 1.0): [(1.0, ())]}] >>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}} >>>>> Perm: [0, 1, 2] >>>>> Creating /home/fuentes/snestutorial/ex52_gpu_inline.h >>>>> >>>>> SCRGP2$ make ex52 >>>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC >>>>> -I/opt/apps/PETSC/petsc-dev/include >>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include >>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve >>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include >>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include >>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include >>>>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/ ex52.c >>>>> nvcc -O0 -g -arch=sm_20 -c --compiler-options="-O0 -g -fPIC >>>>> -I/opt/apps/PETSC/petsc-dev/include >>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include >>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve >>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include >>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include >>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include >>>>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/" >>>>> ex52_integrateElement.cu >>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>>>> never referenced >>>>> >>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>>>> never referenced >>>>> >>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>>>> never referenced >>>>> >>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but >>>>> never referenced >>>>> >>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>>>> never referenced >>>>> >>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was >>>>> declared but never referenced >>>>> >>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>>>> never referenced >>>>> >>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>>>> never referenced >>>>> >>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>>>> never referenced >>>>> >>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but >>>>> never referenced >>>>> >>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>>>> never referenced >>>>> >>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was >>>>> declared but never referenced >>>>> >>>>> /usr/bin/mpicxx -O0 -g -o ex52 ex52.o ex52_integrateElement.o >>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib >>>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib -lpetsc >>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib >>>>> -ltriangle -lX11 -lpthread -lsuperlu_dist_3.0 -lcmumps -ldmumps -lsmumps >>>>> -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs >>>>> -Wl,-rpath,/opt/apps/cuda/4.1/cuda/lib64 -L/opt/apps/cuda/4.1/cuda/lib64 >>>>> -lcufft -lcublas -lcudart -lcusparse >>>>> -Wl,-rpath,/opt/MATLAB/R2011a/sys/os/glnxa64:/opt/MATLAB/R2011a/bin/glnxa64:/opt/MATLAB/R2011a/extern/lib/glnxa64 >>>>> -L/opt/MATLAB/R2011a/bin/glnxa64 -L/opt/MATLAB/R2011a/extern/lib/glnxa64 >>>>> -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc >>>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib >>>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 -lexoIIv2for -lexodus >>>>> -lnetcdf_c++ -lnetcdf -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3 >>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt >>>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx >>>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl >>>>> /bin/rm -f ex52.o ex52_integrateElement.o >>>>> >>>>> >>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch >>>>> Residual: >>>>> Vector Object: 1 MPI processes >>>>> type: seq >>>>> -0.25 >>>>> -0.5 >>>>> 0.25 >>>>> -0.5 >>>>> -1 >>>>> 0.5 >>>>> 0.25 >>>>> 0.5 >>>>> 0.75 >>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu >>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in >>>>> src/snes/examples/tutorials/ex52_integrateElement.cu >>>>> [0]PETSC ERROR: FormFunctionLocalBatch() line 679 in >>>>> src/snes/examples/tutorials/ex52.c >>>>> [0]PETSC ERROR: SNESDMComplexComputeFunction() line 431 in >>>>> src/snes/utils/damgsnes.c >>>>> [0]PETSC ERROR: main() line 1021 in src/snes/examples/tutorials/ex52.c >>>>> application called MPI_Abort(MPI_COMM_WORLD, 35) - process 0 >>>>> >>>> >>>> This is failing on cudaMalloc(), which means your card is not available >>>> for running. Are you trying to run on your laptop? >>>> If so, applications like Preview can lock up the GPU. I know of no way >>>> to test this in CUDA while running. I just close >>>> apps until it runs. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> >>>>> On Tue, Mar 27, 2012 at 8:37 PM, Matthew Knepley wrote: >>>>> >>>>>> On Tue, Mar 27, 2012 at 2:10 PM, Blaise Bourdin wrote: >>>>>> >>>>>>> >>>>>>> On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote: >>>>>>> >>>>>>> On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes >>>>>> > wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I had a question about the status of example 52. >>>>>>>> >>>>>>>> >>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c >>>>>>>> >>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu >>>>>>>> >>>>>>>> >>>>>>>> Can this example be used with a DM object created from an >>>>>>>> unstructured exodusII mesh, DMMeshCreateExodus, And the FEM assembly done >>>>>>>> on GPU ? >>>>>>>> >>>>>>> >>>>>>> 1) I have pushed many more tests for it now. They can be run using >>>>>>> the Python build system >>>>>>> >>>>>>> ./config/builder2.py check src/snes/examples/tutorials/ex52.c >>>>>>> >>>>>>> in fact, you can build any set of files this way. >>>>>>> >>>>>>> 2) The Exodus creation has to be converted to DMComplex from DMMesh. >>>>>>> That should not take me very long. Blaise maintains that >>>>>>> so maybe there will be help :) You will just replace >>>>>>> DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request >>>>>>> it, I will bump it up the list. >>>>>>> >>>>>>> >>>>>>> DMMeshCreateExodusNG is much more flexible than DMMeshCreateExodus >>>>>>> in that it can read meshes with multiple element types and should have a >>>>>>> much lower memory footprint. The code should be fairly easy to read. you >>>>>>> can email me directly if you have specific questions. I had looked at >>>>>>> creating a DMComplex and it did not look too difficult, as long as >>>>>>> interpolation is not needed. I have plans to write DMComplexCreateExodus, >>>>>>> but haven't had time too so far. Updating the Vec viewers and readers may >>>>>>> be a bit more involved. In perfect world, one would write an EXODUS viewer >>>>>>> following the lines of the VTK and HDF5 ones. >>>>>>> >>>>>> >>>>>> David and Blaise, I have converted this function, now >>>>>> DMComplexCreateExodus(). Its not tested, but I think >>>>>> Blaise has some stuff we can use to test it. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Blaise >>>>>>> >>>>>>> >>>>>>> >>>>>>> Let me know if you can run the tests. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Thanks, >>>>>>>> David >>>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Department of Mathematics and Center for Computation & Technology >>>>>>> Louisiana State University, Baton Rouge, LA 70803, USA >>>>>>> Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 >>>>>>> http://www.math.lsu.edu/~bourdin >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Mar 28 16:57:45 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 28 Mar 2012 16:57:45 -0500 Subject: [petsc-users] ex52_integrateElement.cu In-Reply-To: References: <99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu> Message-ID: On Wed, Mar 28, 2012 at 4:12 PM, David Fuentes wrote: > works! > Excellent. Now, my thinking was that GPUs are most useful doing single work, but I can see the utility of double accuracy for a residual. My inclination is to define another type, say GPUReal, and use it for all kernels. Would that do what you want? Matt > SCRGP2$ make ex52 > /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC > -I/opt/apps/PETSC/petsc-dev/include > -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include > -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2 > -D__INSDIR__=src/snes/examples/tutorials/ ex52.c > nvcc -G -O0 -g -arch=sm_10 -c --compiler-options="-O0 -g -fPIC > -I/opt/apps/PETSC/petsc-dev/include > -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include > -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2 > -D__INSDIR__=src/snes/examples/tutorials/" ex52_integrateElement.cu > ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never > referenced > > ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never > referenced > > ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never > referenced > > ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but > never referenced > > ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never > referenced > > ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was declared > but never referenced > > ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never > referenced > > ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never > referenced > > ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never > referenced > > ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but > never referenced > > ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never > referenced > > ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was declared > but never referenced > > /usr/bin/mpicxx -O0 -g -o ex52 ex52.o ex52_integrateElement.o > -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib > -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib > -lpetsc > -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib > -ltriangle -lX11 -lpthread -lmetis -Wl,-rpath,/opt/apps/cuda/4.0/cuda/lib64 > -L/opt/apps/cuda/4.0/cuda/lib64 -lcufft -lcublas -lcudart -lcusparse > -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib > -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3 > -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt > -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx > -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl > > > SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu > GPU layout grid(1,2,1) block(3,1,1) with 1 batches > N_t: 3, N_cb: 1 > Residual: > Vector Object: 1 MPI processes > type: seq > -0.25 > -0.5 > 0.25 > -0.5 > -1 > 0.5 > 0.25 > 0.5 > 0.75 > SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch > Residual: > Vector Object: 1 MPI processes > type: seq > -0.25 > -0.5 > 0.25 > -0.5 > -1 > 0.5 > 0.25 > 0.5 > 0.75 > > > > > > On Wed, Mar 28, 2012 at 1:37 PM, David Fuentes wrote: > >> sure. will do. >> >> >> On Wed, Mar 28, 2012 at 1:23 PM, Matthew Knepley wrote: >> >>> On Wed, Mar 28, 2012 at 1:14 PM, David Fuentes wrote: >>> >>>> thanks! its running, but I seem to be getting different answer for >>>> cpu/gpu ? >>>> i had some floating point problems on this Tesla M2070 gpu before, but >>>> adding the '-arch=sm_20' option seemed to fix it last time. >>>> >>>> >>>> is the assembly in single precision ? my 'const PetscReal >>>> jacobianInverse' being passed in are doubles >>>> >>> >>> Yep, that is the problem. I have not tested anything in double. I have >>> not decided exactly how to handle it. Can you >>> make another ARCH --with-precision=single and make sure it works, and >>> then we can fix the double issue? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu >>>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches >>>> N_t: 3, N_cb: 1 >>>> Residual: >>>> Vector Object: 1 MPI processes >>>> type: seq >>>> 0 >>>> 755712 >>>> 0 >>>> -58720 >>>> -2953.13 >>>> 0.375 >>>> 1.50323e+07 >>>> 0.875 >>>> 0 >>>> SCRGP2$ >>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch >>>> Residual: >>>> Vector Object: 1 MPI processes >>>> type: seq >>>> -0.25 >>>> -0.5 >>>> 0.25 >>>> -0.5 >>>> -1 >>>> 0.5 >>>> 0.25 >>>> 0.5 >>>> 0.75 >>>> >>>> >>>> >>>> >>>> >>>> On Wed, Mar 28, 2012 at 11:55 AM, Matthew Knepley wrote: >>>> >>>>> On Wed, Mar 28, 2012 at 11:45 AM, David Fuentes wrote: >>>>> >>>>>> The example seems to be running on cpu with '-batch' but i'm getting >>>>>> errors in line 323 with the '-gpu' option >>>>>> >>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in >>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu >>>>>> >>>>>> should this possibly be PetscScalar ? >>>>>> >>>>> >>>>> No. >>>>> >>>>> >>>>>> - ierr = cudaMalloc((void**) &d_coefficients, Ne*N_bt * >>>>>> sizeof(float));CHKERRQ(ierr); >>>>>> + ierr = cudaMalloc((void**) &d_coefficients, Ne*N_bt * >>>>>> sizeof(PetscScalar));CHKERRQ(ierr); >>>>>> >>>>>> >>>>>> SCRGP2$ python >>>>>> $PETSC_DIR/bin/pythonscripts/PetscGenerateFEMQuadrature.py 2 1 1 1 >>>>>> laplacian ex52.h >>>>>> ['/opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py', >>>>>> '2', '1', '1', '1', 'laplacian', 'ex52.h'] >>>>>> 2 1 1 1 laplacian >>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0, >>>>>> 1.0): [(1.0, ())]}] >>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}} >>>>>> Perm: [0, 1, 2] >>>>>> Creating /home/fuentes/snestutorial/ex52.h >>>>>> Creating /home/fuentes/snestutorial/ex52_gpu.h >>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0, >>>>>> 1.0): [(1.0, ())]}] >>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}} >>>>>> Perm: [0, 1, 2] >>>>>> Creating /home/fuentes/snestutorial/ex52_gpu_inline.h >>>>>> >>>>>> SCRGP2$ make ex52 >>>>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC >>>>>> -I/opt/apps/PETSC/petsc-dev/include >>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include >>>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve >>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include >>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include >>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include >>>>>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/ ex52.c >>>>>> nvcc -O0 -g -arch=sm_20 -c --compiler-options="-O0 -g -fPIC >>>>>> -I/opt/apps/PETSC/petsc-dev/include >>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include >>>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve >>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include >>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include >>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include >>>>>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/" >>>>>> ex52_integrateElement.cu >>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>>>>> never referenced >>>>>> >>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>>>>> never referenced >>>>>> >>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>>>>> never referenced >>>>>> >>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but >>>>>> never referenced >>>>>> >>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>>>>> never referenced >>>>>> >>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was >>>>>> declared but never referenced >>>>>> >>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>>>>> never referenced >>>>>> >>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>>>>> never referenced >>>>>> >>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>>>>> never referenced >>>>>> >>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but >>>>>> never referenced >>>>>> >>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>>>>> never referenced >>>>>> >>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was >>>>>> declared but never referenced >>>>>> >>>>>> /usr/bin/mpicxx -O0 -g -o ex52 ex52.o ex52_integrateElement.o >>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib >>>>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib -lpetsc >>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib >>>>>> -ltriangle -lX11 -lpthread -lsuperlu_dist_3.0 -lcmumps -ldmumps -lsmumps >>>>>> -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs >>>>>> -Wl,-rpath,/opt/apps/cuda/4.1/cuda/lib64 -L/opt/apps/cuda/4.1/cuda/lib64 >>>>>> -lcufft -lcublas -lcudart -lcusparse >>>>>> -Wl,-rpath,/opt/MATLAB/R2011a/sys/os/glnxa64:/opt/MATLAB/R2011a/bin/glnxa64:/opt/MATLAB/R2011a/extern/lib/glnxa64 >>>>>> -L/opt/MATLAB/R2011a/bin/glnxa64 -L/opt/MATLAB/R2011a/extern/lib/glnxa64 >>>>>> -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc >>>>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib >>>>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 -lexoIIv2for -lexodus >>>>>> -lnetcdf_c++ -lnetcdf -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3 >>>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt >>>>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx >>>>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl >>>>>> /bin/rm -f ex52.o ex52_integrateElement.o >>>>>> >>>>>> >>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch >>>>>> Residual: >>>>>> Vector Object: 1 MPI processes >>>>>> type: seq >>>>>> -0.25 >>>>>> -0.5 >>>>>> 0.25 >>>>>> -0.5 >>>>>> -1 >>>>>> 0.5 >>>>>> 0.25 >>>>>> 0.5 >>>>>> 0.75 >>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu >>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in >>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu >>>>>> [0]PETSC ERROR: FormFunctionLocalBatch() line 679 in >>>>>> src/snes/examples/tutorials/ex52.c >>>>>> [0]PETSC ERROR: SNESDMComplexComputeFunction() line 431 in >>>>>> src/snes/utils/damgsnes.c >>>>>> [0]PETSC ERROR: main() line 1021 in src/snes/examples/tutorials/ex52.c >>>>>> application called MPI_Abort(MPI_COMM_WORLD, 35) - process 0 >>>>>> >>>>> >>>>> This is failing on cudaMalloc(), which means your card is not >>>>> available for running. Are you trying to run on your laptop? >>>>> If so, applications like Preview can lock up the GPU. I know of no way >>>>> to test this in CUDA while running. I just close >>>>> apps until it runs. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> >>>>>> On Tue, Mar 27, 2012 at 8:37 PM, Matthew Knepley wrote: >>>>>> >>>>>>> On Tue, Mar 27, 2012 at 2:10 PM, Blaise Bourdin wrote: >>>>>>> >>>>>>>> >>>>>>>> On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote: >>>>>>>> >>>>>>>> On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes < >>>>>>>> fuentesdt at gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I had a question about the status of example 52. >>>>>>>>> >>>>>>>>> >>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c >>>>>>>>> >>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu >>>>>>>>> >>>>>>>>> >>>>>>>>> Can this example be used with a DM object created from an >>>>>>>>> unstructured exodusII mesh, DMMeshCreateExodus, And the FEM assembly done >>>>>>>>> on GPU ? >>>>>>>>> >>>>>>>> >>>>>>>> 1) I have pushed many more tests for it now. They can be run using >>>>>>>> the Python build system >>>>>>>> >>>>>>>> ./config/builder2.py check src/snes/examples/tutorials/ex52.c >>>>>>>> >>>>>>>> in fact, you can build any set of files this way. >>>>>>>> >>>>>>>> 2) The Exodus creation has to be converted to DMComplex from >>>>>>>> DMMesh. That should not take me very long. Blaise maintains that >>>>>>>> so maybe there will be help :) You will just replace >>>>>>>> DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request >>>>>>>> it, I will bump it up the list. >>>>>>>> >>>>>>>> >>>>>>>> DMMeshCreateExodusNG is much more flexible than DMMeshCreateExodus >>>>>>>> in that it can read meshes with multiple element types and should have a >>>>>>>> much lower memory footprint. The code should be fairly easy to read. you >>>>>>>> can email me directly if you have specific questions. I had looked at >>>>>>>> creating a DMComplex and it did not look too difficult, as long as >>>>>>>> interpolation is not needed. I have plans to write DMComplexCreateExodus, >>>>>>>> but haven't had time too so far. Updating the Vec viewers and readers may >>>>>>>> be a bit more involved. In perfect world, one would write an EXODUS viewer >>>>>>>> following the lines of the VTK and HDF5 ones. >>>>>>>> >>>>>>> >>>>>>> David and Blaise, I have converted this function, now >>>>>>> DMComplexCreateExodus(). Its not tested, but I think >>>>>>> Blaise has some stuff we can use to test it. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Blaise >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Let me know if you can run the tests. >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> David >>>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Department of Mathematics and Center for Computation & Technology >>>>>>>> Louisiana State University, Baton Rouge, LA 70803, USA >>>>>>>> Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 >>>>>>>> http://www.math.lsu.edu/~bourdin >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuentesdt at gmail.com Wed Mar 28 18:25:40 2012 From: fuentesdt at gmail.com (David Fuentes) Date: Wed, 28 Mar 2012 18:25:40 -0500 Subject: [petsc-users] ex52_integrateElement.cu In-Reply-To: References: <99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu> Message-ID: Yes. This would work. I had trouble compiling in single precision using the some of the external package options I was using for double. On Wed, Mar 28, 2012 at 4:57 PM, Matthew Knepley wrote: > On Wed, Mar 28, 2012 at 4:12 PM, David Fuentes wrote: > >> works! >> > > Excellent. Now, my thinking was that GPUs are most useful doing single > work, but > I can see the utility of double accuracy for a residual. > > My inclination is to define another type, say GPUReal, and use it for all > kernels. > Would that do what you want? > > Matt > > >> SCRGP2$ make ex52 >> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC >> -I/opt/apps/PETSC/petsc-dev/include >> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include >> -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2 >> -D__INSDIR__=src/snes/examples/tutorials/ ex52.c >> nvcc -G -O0 -g -arch=sm_10 -c --compiler-options="-O0 -g -fPIC >> -I/opt/apps/PETSC/petsc-dev/include >> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include >> -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2 >> -D__INSDIR__=src/snes/examples/tutorials/" ex52_integrateElement.cu >> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >> never referenced >> >> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never >> referenced >> >> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never >> referenced >> >> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but >> never referenced >> >> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never >> referenced >> >> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was >> declared but never referenced >> >> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never >> referenced >> >> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never >> referenced >> >> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but never >> referenced >> >> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but >> never referenced >> >> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but never >> referenced >> >> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was >> declared but never referenced >> >> /usr/bin/mpicxx -O0 -g -o ex52 ex52.o ex52_integrateElement.o >> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib >> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib >> -lpetsc >> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib >> -ltriangle -lX11 -lpthread -lmetis -Wl,-rpath,/opt/apps/cuda/4.0/cuda/lib64 >> -L/opt/apps/cuda/4.0/cuda/lib64 -lcufft -lcublas -lcudart -lcusparse >> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib >> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3 >> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt >> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx >> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl >> >> >> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu >> GPU layout grid(1,2,1) block(3,1,1) with 1 batches >> N_t: 3, N_cb: 1 >> Residual: >> Vector Object: 1 MPI processes >> type: seq >> -0.25 >> -0.5 >> 0.25 >> -0.5 >> -1 >> 0.5 >> 0.25 >> 0.5 >> 0.75 >> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch >> Residual: >> Vector Object: 1 MPI processes >> type: seq >> -0.25 >> -0.5 >> 0.25 >> -0.5 >> -1 >> 0.5 >> 0.25 >> 0.5 >> 0.75 >> >> >> >> >> >> On Wed, Mar 28, 2012 at 1:37 PM, David Fuentes wrote: >> >>> sure. will do. >>> >>> >>> On Wed, Mar 28, 2012 at 1:23 PM, Matthew Knepley wrote: >>> >>>> On Wed, Mar 28, 2012 at 1:14 PM, David Fuentes wrote: >>>> >>>>> thanks! its running, but I seem to be getting different answer for >>>>> cpu/gpu ? >>>>> i had some floating point problems on this Tesla M2070 gpu before, but >>>>> adding the '-arch=sm_20' option seemed to fix it last time. >>>>> >>>>> >>>>> is the assembly in single precision ? my 'const PetscReal >>>>> jacobianInverse' being passed in are doubles >>>>> >>>> >>>> Yep, that is the problem. I have not tested anything in double. I have >>>> not decided exactly how to handle it. Can you >>>> make another ARCH --with-precision=single and make sure it works, and >>>> then we can fix the double issue? >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu >>>>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches >>>>> N_t: 3, N_cb: 1 >>>>> Residual: >>>>> Vector Object: 1 MPI processes >>>>> type: seq >>>>> 0 >>>>> 755712 >>>>> 0 >>>>> -58720 >>>>> -2953.13 >>>>> 0.375 >>>>> 1.50323e+07 >>>>> 0.875 >>>>> 0 >>>>> SCRGP2$ >>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch >>>>> Residual: >>>>> Vector Object: 1 MPI processes >>>>> type: seq >>>>> -0.25 >>>>> -0.5 >>>>> 0.25 >>>>> -0.5 >>>>> -1 >>>>> 0.5 >>>>> 0.25 >>>>> 0.5 >>>>> 0.75 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Mar 28, 2012 at 11:55 AM, Matthew Knepley wrote: >>>>> >>>>>> On Wed, Mar 28, 2012 at 11:45 AM, David Fuentes wrote: >>>>>> >>>>>>> The example seems to be running on cpu with '-batch' but i'm getting >>>>>>> errors in line 323 with the '-gpu' option >>>>>>> >>>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in >>>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu >>>>>>> >>>>>>> should this possibly be PetscScalar ? >>>>>>> >>>>>> >>>>>> No. >>>>>> >>>>>> >>>>>>> - ierr = cudaMalloc((void**) &d_coefficients, Ne*N_bt * >>>>>>> sizeof(float));CHKERRQ(ierr); >>>>>>> + ierr = cudaMalloc((void**) &d_coefficients, Ne*N_bt * >>>>>>> sizeof(PetscScalar));CHKERRQ(ierr); >>>>>>> >>>>>>> >>>>>>> SCRGP2$ python >>>>>>> $PETSC_DIR/bin/pythonscripts/PetscGenerateFEMQuadrature.py 2 1 1 1 >>>>>>> laplacian ex52.h >>>>>>> ['/opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py', >>>>>>> '2', '1', '1', '1', 'laplacian', 'ex52.h'] >>>>>>> 2 1 1 1 laplacian >>>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0, >>>>>>> 1.0): [(1.0, ())]}] >>>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}} >>>>>>> Perm: [0, 1, 2] >>>>>>> Creating /home/fuentes/snestutorial/ex52.h >>>>>>> Creating /home/fuentes/snestutorial/ex52_gpu.h >>>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0, >>>>>>> 1.0): [(1.0, ())]}] >>>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}} >>>>>>> Perm: [0, 1, 2] >>>>>>> Creating /home/fuentes/snestutorial/ex52_gpu_inline.h >>>>>>> >>>>>>> SCRGP2$ make ex52 >>>>>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC >>>>>>> -I/opt/apps/PETSC/petsc-dev/include >>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include >>>>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve >>>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include >>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include >>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include >>>>>>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/ ex52.c >>>>>>> nvcc -O0 -g -arch=sm_20 -c --compiler-options="-O0 -g -fPIC >>>>>>> -I/opt/apps/PETSC/petsc-dev/include >>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include >>>>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve >>>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include >>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include >>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include >>>>>>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/" >>>>>>> ex52_integrateElement.cu >>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>>>>>> never referenced >>>>>>> >>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>>>>>> never referenced >>>>>>> >>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>>>>>> never referenced >>>>>>> >>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared >>>>>>> but never referenced >>>>>>> >>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>>>>>> never referenced >>>>>>> >>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was >>>>>>> declared but never referenced >>>>>>> >>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>>>>>> never referenced >>>>>>> >>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>>>>>> never referenced >>>>>>> >>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>>>>>> never referenced >>>>>>> >>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared >>>>>>> but never referenced >>>>>>> >>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>>>>>> never referenced >>>>>>> >>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was >>>>>>> declared but never referenced >>>>>>> >>>>>>> /usr/bin/mpicxx -O0 -g -o ex52 ex52.o ex52_integrateElement.o >>>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib >>>>>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib -lpetsc >>>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib >>>>>>> -ltriangle -lX11 -lpthread -lsuperlu_dist_3.0 -lcmumps -ldmumps -lsmumps >>>>>>> -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs >>>>>>> -Wl,-rpath,/opt/apps/cuda/4.1/cuda/lib64 -L/opt/apps/cuda/4.1/cuda/lib64 >>>>>>> -lcufft -lcublas -lcudart -lcusparse >>>>>>> -Wl,-rpath,/opt/MATLAB/R2011a/sys/os/glnxa64:/opt/MATLAB/R2011a/bin/glnxa64:/opt/MATLAB/R2011a/extern/lib/glnxa64 >>>>>>> -L/opt/MATLAB/R2011a/bin/glnxa64 -L/opt/MATLAB/R2011a/extern/lib/glnxa64 >>>>>>> -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc >>>>>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib >>>>>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 -lexoIIv2for -lexodus >>>>>>> -lnetcdf_c++ -lnetcdf -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3 >>>>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt >>>>>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx >>>>>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl >>>>>>> /bin/rm -f ex52.o ex52_integrateElement.o >>>>>>> >>>>>>> >>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch >>>>>>> Residual: >>>>>>> Vector Object: 1 MPI processes >>>>>>> type: seq >>>>>>> -0.25 >>>>>>> -0.5 >>>>>>> 0.25 >>>>>>> -0.5 >>>>>>> -1 >>>>>>> 0.5 >>>>>>> 0.25 >>>>>>> 0.5 >>>>>>> 0.75 >>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu >>>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in >>>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu >>>>>>> [0]PETSC ERROR: FormFunctionLocalBatch() line 679 in >>>>>>> src/snes/examples/tutorials/ex52.c >>>>>>> [0]PETSC ERROR: SNESDMComplexComputeFunction() line 431 in >>>>>>> src/snes/utils/damgsnes.c >>>>>>> [0]PETSC ERROR: main() line 1021 in >>>>>>> src/snes/examples/tutorials/ex52.c >>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 35) - process 0 >>>>>>> >>>>>> >>>>>> This is failing on cudaMalloc(), which means your card is not >>>>>> available for running. Are you trying to run on your laptop? >>>>>> If so, applications like Preview can lock up the GPU. I know of no >>>>>> way to test this in CUDA while running. I just close >>>>>> apps until it runs. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> >>>>>>> On Tue, Mar 27, 2012 at 8:37 PM, Matthew Knepley wrote: >>>>>>> >>>>>>>> On Tue, Mar 27, 2012 at 2:10 PM, Blaise Bourdin wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote: >>>>>>>>> >>>>>>>>> On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes < >>>>>>>>> fuentesdt at gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I had a question about the status of example 52. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c >>>>>>>>>> >>>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Can this example be used with a DM object created from an >>>>>>>>>> unstructured exodusII mesh, DMMeshCreateExodus, And the FEM assembly done >>>>>>>>>> on GPU ? >>>>>>>>>> >>>>>>>>> >>>>>>>>> 1) I have pushed many more tests for it now. They can be run using >>>>>>>>> the Python build system >>>>>>>>> >>>>>>>>> ./config/builder2.py check src/snes/examples/tutorials/ex52.c >>>>>>>>> >>>>>>>>> in fact, you can build any set of files this way. >>>>>>>>> >>>>>>>>> 2) The Exodus creation has to be converted to DMComplex from >>>>>>>>> DMMesh. That should not take me very long. Blaise maintains that >>>>>>>>> so maybe there will be help :) You will just replace >>>>>>>>> DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request >>>>>>>>> it, I will bump it up the list. >>>>>>>>> >>>>>>>>> >>>>>>>>> DMMeshCreateExodusNG is much more flexible than DMMeshCreateExodus >>>>>>>>> in that it can read meshes with multiple element types and should have a >>>>>>>>> much lower memory footprint. The code should be fairly easy to read. you >>>>>>>>> can email me directly if you have specific questions. I had looked at >>>>>>>>> creating a DMComplex and it did not look too difficult, as long as >>>>>>>>> interpolation is not needed. I have plans to write DMComplexCreateExodus, >>>>>>>>> but haven't had time too so far. Updating the Vec viewers and readers may >>>>>>>>> be a bit more involved. In perfect world, one would write an EXODUS viewer >>>>>>>>> following the lines of the VTK and HDF5 ones. >>>>>>>>> >>>>>>>> >>>>>>>> David and Blaise, I have converted this function, now >>>>>>>> DMComplexCreateExodus(). Its not tested, but I think >>>>>>>> Blaise has some stuff we can use to test it. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>>> Blaise >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Let me know if you can run the tests. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> David >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Department of Mathematics and Center for Computation & Technology >>>>>>>>> Louisiana State University, Baton Rouge, LA 70803, USA >>>>>>>>> Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 >>>>>>>>> http://www.math.lsu.edu/~bourdin >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Mar 28 18:42:39 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 28 Mar 2012 18:42:39 -0500 Subject: [petsc-users] ex52_integrateElement.cu In-Reply-To: References: <99B812CB-3215-402A-BEC4-E78A90D08443@lsu.edu> Message-ID: On Wed, Mar 28, 2012 at 6:25 PM, David Fuentes wrote: > Yes. This would work. > I had trouble compiling in single precision using the some of the external > package options I was using for double. > Okay, getting on it now. Matt > On Wed, Mar 28, 2012 at 4:57 PM, Matthew Knepley wrote: > >> On Wed, Mar 28, 2012 at 4:12 PM, David Fuentes wrote: >> >>> works! >>> >> >> Excellent. Now, my thinking was that GPUs are most useful doing single >> work, but >> I can see the utility of double accuracy for a residual. >> >> My inclination is to define another type, say GPUReal, and use it for all >> kernels. >> Would that do what you want? >> >> Matt >> >> >>> SCRGP2$ make ex52 >>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC >>> -I/opt/apps/PETSC/petsc-dev/include >>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include >>> -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2 >>> -D__INSDIR__=src/snes/examples/tutorials/ ex52.c >>> nvcc -G -O0 -g -arch=sm_10 -c --compiler-options="-O0 -g -fPIC >>> -I/opt/apps/PETSC/petsc-dev/include >>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include >>> -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2 >>> -D__INSDIR__=src/snes/examples/tutorials/" ex52_integrateElement.cu >>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>> never referenced >>> >>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>> never referenced >>> >>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>> never referenced >>> >>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but >>> never referenced >>> >>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>> never referenced >>> >>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was >>> declared but never referenced >>> >>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>> never referenced >>> >>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>> never referenced >>> >>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>> never referenced >>> >>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but >>> never referenced >>> >>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>> never referenced >>> >>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was >>> declared but never referenced >>> >>> /usr/bin/mpicxx -O0 -g -o ex52 ex52.o ex52_integrateElement.o >>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib >>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib >>> -lpetsc >>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib >>> -ltriangle -lX11 -lpthread -lmetis -Wl,-rpath,/opt/apps/cuda/4.0/cuda/lib64 >>> -L/opt/apps/cuda/4.0/cuda/lib64 -lcufft -lcublas -lcudart -lcusparse >>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib >>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 >>> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3 >>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt >>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx >>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl >>> >>> >>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu >>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches >>> N_t: 3, N_cb: 1 >>> Residual: >>> Vector Object: 1 MPI processes >>> type: seq >>> -0.25 >>> -0.5 >>> 0.25 >>> -0.5 >>> -1 >>> 0.5 >>> 0.25 >>> 0.5 >>> 0.75 >>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch >>> Residual: >>> Vector Object: 1 MPI processes >>> type: seq >>> -0.25 >>> -0.5 >>> 0.25 >>> -0.5 >>> -1 >>> 0.5 >>> 0.25 >>> 0.5 >>> 0.75 >>> >>> >>> >>> >>> >>> On Wed, Mar 28, 2012 at 1:37 PM, David Fuentes wrote: >>> >>>> sure. will do. >>>> >>>> >>>> On Wed, Mar 28, 2012 at 1:23 PM, Matthew Knepley wrote: >>>> >>>>> On Wed, Mar 28, 2012 at 1:14 PM, David Fuentes wrote: >>>>> >>>>>> thanks! its running, but I seem to be getting different answer for >>>>>> cpu/gpu ? >>>>>> i had some floating point problems on this Tesla M2070 gpu before, >>>>>> but adding the '-arch=sm_20' option seemed to fix it last time. >>>>>> >>>>>> >>>>>> is the assembly in single precision ? my 'const PetscReal >>>>>> jacobianInverse' being passed in are doubles >>>>>> >>>>> >>>>> Yep, that is the problem. I have not tested anything in double. I have >>>>> not decided exactly how to handle it. Can you >>>>> make another ARCH --with-precision=single and make sure it works, and >>>>> then we can fix the double issue? >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu >>>>>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches >>>>>> N_t: 3, N_cb: 1 >>>>>> Residual: >>>>>> Vector Object: 1 MPI processes >>>>>> type: seq >>>>>> 0 >>>>>> 755712 >>>>>> 0 >>>>>> -58720 >>>>>> -2953.13 >>>>>> 0.375 >>>>>> 1.50323e+07 >>>>>> 0.875 >>>>>> 0 >>>>>> SCRGP2$ >>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch >>>>>> Residual: >>>>>> Vector Object: 1 MPI processes >>>>>> type: seq >>>>>> -0.25 >>>>>> -0.5 >>>>>> 0.25 >>>>>> -0.5 >>>>>> -1 >>>>>> 0.5 >>>>>> 0.25 >>>>>> 0.5 >>>>>> 0.75 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Mar 28, 2012 at 11:55 AM, Matthew Knepley wrote: >>>>>> >>>>>>> On Wed, Mar 28, 2012 at 11:45 AM, David Fuentes >>>>>> > wrote: >>>>>>> >>>>>>>> The example seems to be running on cpu with '-batch' but >>>>>>>> i'm getting errors in line 323 with the '-gpu' option >>>>>>>> >>>>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in >>>>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu >>>>>>>> >>>>>>>> should this possibly be PetscScalar ? >>>>>>>> >>>>>>> >>>>>>> No. >>>>>>> >>>>>>> >>>>>>>> - ierr = cudaMalloc((void**) &d_coefficients, Ne*N_bt * >>>>>>>> sizeof(float));CHKERRQ(ierr); >>>>>>>> + ierr = cudaMalloc((void**) &d_coefficients, Ne*N_bt * >>>>>>>> sizeof(PetscScalar));CHKERRQ(ierr); >>>>>>>> >>>>>>>> >>>>>>>> SCRGP2$ python >>>>>>>> $PETSC_DIR/bin/pythonscripts/PetscGenerateFEMQuadrature.py 2 1 1 1 >>>>>>>> laplacian ex52.h >>>>>>>> ['/opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py', >>>>>>>> '2', '1', '1', '1', 'laplacian', 'ex52.h'] >>>>>>>> 2 1 1 1 laplacian >>>>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0, >>>>>>>> 1.0): [(1.0, ())]}] >>>>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}} >>>>>>>> Perm: [0, 1, 2] >>>>>>>> Creating /home/fuentes/snestutorial/ex52.h >>>>>>>> Creating /home/fuentes/snestutorial/ex52_gpu.h >>>>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0, >>>>>>>> 1.0): [(1.0, ())]}] >>>>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}} >>>>>>>> Perm: [0, 1, 2] >>>>>>>> Creating /home/fuentes/snestutorial/ex52_gpu_inline.h >>>>>>>> >>>>>>>> SCRGP2$ make ex52 >>>>>>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC >>>>>>>> -I/opt/apps/PETSC/petsc-dev/include >>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include >>>>>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve >>>>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include >>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include >>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include >>>>>>>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/ ex52.c >>>>>>>> nvcc -O0 -g -arch=sm_20 -c --compiler-options="-O0 -g -fPIC >>>>>>>> -I/opt/apps/PETSC/petsc-dev/include >>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include >>>>>>>> -I/opt/apps/cuda/4.1/cuda/include -I/opt/apps/PETSC/petsc-dev/include/sieve >>>>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include >>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include >>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include >>>>>>>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/" >>>>>>>> ex52_integrateElement.cu >>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>>>>>>> never referenced >>>>>>>> >>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>>>>>>> never referenced >>>>>>>> >>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>>>>>>> never referenced >>>>>>>> >>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared >>>>>>>> but never referenced >>>>>>>> >>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>>>>>>> never referenced >>>>>>>> >>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was >>>>>>>> declared but never referenced >>>>>>>> >>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>>>>>>> never referenced >>>>>>>> >>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>>>>>>> never referenced >>>>>>>> >>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but >>>>>>>> never referenced >>>>>>>> >>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared >>>>>>>> but never referenced >>>>>>>> >>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but >>>>>>>> never referenced >>>>>>>> >>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was >>>>>>>> declared but never referenced >>>>>>>> >>>>>>>> /usr/bin/mpicxx -O0 -g -o ex52 ex52.o ex52_integrateElement.o >>>>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib >>>>>>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib -lpetsc >>>>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib >>>>>>>> -ltriangle -lX11 -lpthread -lsuperlu_dist_3.0 -lcmumps -ldmumps -lsmumps >>>>>>>> -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs >>>>>>>> -Wl,-rpath,/opt/apps/cuda/4.1/cuda/lib64 -L/opt/apps/cuda/4.1/cuda/lib64 >>>>>>>> -lcufft -lcublas -lcudart -lcusparse >>>>>>>> -Wl,-rpath,/opt/MATLAB/R2011a/sys/os/glnxa64:/opt/MATLAB/R2011a/bin/glnxa64:/opt/MATLAB/R2011a/extern/lib/glnxa64 >>>>>>>> -L/opt/MATLAB/R2011a/bin/glnxa64 -L/opt/MATLAB/R2011a/extern/lib/glnxa64 >>>>>>>> -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc >>>>>>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib >>>>>>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 -lexoIIv2for -lexodus >>>>>>>> -lnetcdf_c++ -lnetcdf -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3 >>>>>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt >>>>>>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx >>>>>>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl >>>>>>>> /bin/rm -f ex52.o ex52_integrateElement.o >>>>>>>> >>>>>>>> >>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch >>>>>>>> Residual: >>>>>>>> Vector Object: 1 MPI processes >>>>>>>> type: seq >>>>>>>> -0.25 >>>>>>>> -0.5 >>>>>>>> 0.25 >>>>>>>> -0.5 >>>>>>>> -1 >>>>>>>> 0.5 >>>>>>>> 0.25 >>>>>>>> 0.5 >>>>>>>> 0.75 >>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu >>>>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in >>>>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu >>>>>>>> [0]PETSC ERROR: FormFunctionLocalBatch() line 679 in >>>>>>>> src/snes/examples/tutorials/ex52.c >>>>>>>> [0]PETSC ERROR: SNESDMComplexComputeFunction() line 431 in >>>>>>>> src/snes/utils/damgsnes.c >>>>>>>> [0]PETSC ERROR: main() line 1021 in >>>>>>>> src/snes/examples/tutorials/ex52.c >>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 35) - process 0 >>>>>>>> >>>>>>> >>>>>>> This is failing on cudaMalloc(), which means your card is not >>>>>>> available for running. Are you trying to run on your laptop? >>>>>>> If so, applications like Preview can lock up the GPU. I know of no >>>>>>> way to test this in CUDA while running. I just close >>>>>>> apps until it runs. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> On Tue, Mar 27, 2012 at 8:37 PM, Matthew Knepley >>>>>>> > wrote: >>>>>>>> >>>>>>>>> On Tue, Mar 27, 2012 at 2:10 PM, Blaise Bourdin wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote: >>>>>>>>>> >>>>>>>>>> On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes < >>>>>>>>>> fuentesdt at gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I had a question about the status of example 52. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c >>>>>>>>>>> >>>>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Can this example be used with a DM object created from an >>>>>>>>>>> unstructured exodusII mesh, DMMeshCreateExodus, And the FEM assembly done >>>>>>>>>>> on GPU ? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 1) I have pushed many more tests for it now. They can be run >>>>>>>>>> using the Python build system >>>>>>>>>> >>>>>>>>>> ./config/builder2.py check src/snes/examples/tutorials/ex52.c >>>>>>>>>> >>>>>>>>>> in fact, you can build any set of files this way. >>>>>>>>>> >>>>>>>>>> 2) The Exodus creation has to be converted to DMComplex from >>>>>>>>>> DMMesh. That should not take me very long. Blaise maintains that >>>>>>>>>> so maybe there will be help :) You will just replace >>>>>>>>>> DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you request >>>>>>>>>> it, I will bump it up the list. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> DMMeshCreateExodusNG is much more flexible >>>>>>>>>> than DMMeshCreateExodus in that it can read meshes with multiple element >>>>>>>>>> types and should have a much lower memory footprint. The code should be >>>>>>>>>> fairly easy to read. you can email me directly if you have specific >>>>>>>>>> questions. I had looked at creating a DMComplex and it did not look too >>>>>>>>>> difficult, as long as interpolation is not needed. I have plans to >>>>>>>>>> write DMComplexCreateExodus, but haven't had time too so far. Updating the >>>>>>>>>> Vec viewers and readers may be a bit more involved. In perfect world, one >>>>>>>>>> would write an EXODUS viewer following the lines of the VTK and HDF5 ones. >>>>>>>>>> >>>>>>>>> >>>>>>>>> David and Blaise, I have converted this function, now >>>>>>>>> DMComplexCreateExodus(). Its not tested, but I think >>>>>>>>> Blaise has some stuff we can use to test it. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>>> Blaise >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Let me know if you can run the tests. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> David >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>>> experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Department of Mathematics and Center for Computation & Technology >>>>>>>>>> Louisiana State University, Baton Rouge, LA 70803, USA >>>>>>>>>> Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 >>>>>>>>>> http://www.math.lsu.edu/~bourdin >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.mousel at gmail.com Thu Mar 29 10:18:50 2012 From: john.mousel at gmail.com (John Mousel) Date: Thu, 29 Mar 2012 10:18:50 -0500 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> <8909A66D-5CB4-479B-ADBC-34A1133D64F4@columbia.edu> Message-ID: Mark, I'm working with a new matrix, which again converges with ML and HYPRE on 4 cores. I pulled petsc-dev this morning, and I'm getting [0]PCSetFromOptions_GAMG threshold set 1.000000e-02 [0]PCSetUp_GAMG level 0 N=556240, n data rows=1, n data cols=1, nnz/row (ave)=27, np=4 [0]PCGAMGFilterGraph 65.8687% nnz after filtering, with threshold 0.01, 27.0672 nnz ave. [0]maxIndSetAgg removed 0 of 556240 vertices. (0 local) 20114 selected. [0]PCGAMGProlongator_AGG New grid 20114 nodes [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Error in external library! [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Error in external library! [0]PETSC ERROR: Cannot disable floating point exceptions! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: 7d8a276dc2a168a3596c060afce69a229eb409ea HG Date: Thu Mar 29 07:30:18 2012 -0500 [0]PETSC ERROR: [1]PETSC ERROR: Cannot disable floating point exceptions! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Development HG revision: 7d8a276dc2a168a3596c060afce69a229eb409ea HG Date: Thu Mar 29 07:30:18 2012 -0500 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [2]PETSC ERROR: --------------------- Error Message ------------------------------------ [2]PETSC ERROR: Error in external library! [3]PETSC ERROR: --------------------- Error Message ------------------------------------ [3]PETSC ERROR: Error in external library! [3]PETSC ERROR: Cannot disable floating point exceptions! See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named wv.iihr.uiowa.eduby jmousel Thu Mar 29 10:03:45 2012 [0]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib [0]PETSC ERROR: Configure run at Thu Mar 29 09:29:37 2012 [0]PETSC ERROR: Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named wv.iihr.uiowa.eduby jmousel Thu Mar 29 10:03:45 2012 [1]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib [1]PETSC ERROR: Configure run at Thu Mar 29 09:29:37 2012 [1]PETSC ERROR: [2]PETSC ERROR: Cannot disable floating point exceptions! [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Petsc Development HG revision: 7d8a276dc2a168a3596c060afce69a229eb409ea HG Date: Thu Mar 29 07:30:18 2012 -0500 [2]PETSC ERROR: See docs/changes/index.html for recent updates. [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [2]PETSC ERROR: [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Petsc Development HG revision: 7d8a276dc2a168a3596c060afce69a229eb409ea HG Date: Thu Mar 29 07:30:18 2012 -0500 [3]PETSC ERROR: See docs/changes/index.html for recent updates. [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [3]PETSC ERROR: See docs/index.html for manual pages. [3]PETSC ERROR: [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PetscSetFPTrap() line 465 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/sys/error/fp.c [0]PETSC ERROR: PetscFPTrapPush() line 56 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/sys/error/fp.c [0]PETSC ERROR: KSPComputeExtremeSingularValues_GMRES() line 42 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/impls/gmres/gmreig.c [0]PETSC ERROR: KSPComputeExtremeSingularValues() line 47 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: PetscSetFPTrap() line 465 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/sys/error/fp.c See docs/index.html for manual pages. [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named wv.iihr.uiowa.eduby jmousel Thu Mar 29 10:03:45 2012 [2]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib [2]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named wv.iihr.uiowa.eduby jmousel Thu Mar 29 10:03:45 2012 [3]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib [3]PETSC ERROR: Configure run at Thu Mar 29 09:29:37 2012 [3]PETSC ERROR: [0]PETSC ERROR: PCGAMGOptprol_AGG() line 1293 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c [0]PETSC ERROR: PCSetUp_GAMG() line 545 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c [0]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPSetUp() line 266 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c Configure run at Thu Mar 29 09:29:37 2012 [2]PETSC ERROR: Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: PetscSetFPTrap() line 465 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/sys/error/fp.c Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: PetscSetFPTrap() line 465 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/sys/error/fp.c [3]PETSC ERROR: PetscFPTrapPush() line 56 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/sys/error/fp.c [0]PETSC ERROR: KSPSolve() line 390 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: PetscFPTrapPush() line 56 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/sys/error/fp.c [1]PETSC ERROR: KSPComputeExtremeSingularValues_GMRES() line 42 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/impls/gmres/gmreig.c [1]PETSC ERROR: KSPComputeExtremeSingularValues() line 47 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: [2]PETSC ERROR: PetscFPTrapPush() line 56 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/sys/error/fp.c [2]PETSC ERROR: KSPComputeExtremeSingularValues_GMRES() line 42 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/impls/gmres/gmreig.c [2]PETSC ERROR: KSPComputeExtremeSingularValues() line 47 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c [2]PETSC ERROR: [3]PETSC ERROR: KSPComputeExtremeSingularValues_GMRES() line 42 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/impls/gmres/gmreig.c [3]PETSC ERROR: KSPComputeExtremeSingularValues() line 47 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c [3]PETSC ERROR: PCGAMGOptprol_AGG() line 1293 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c PCGAMGOptprol_AGG() line 1293 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c [1]PETSC ERROR: PCSetUp_GAMG() line 545 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c [1]PETSC ERROR: PCGAMGOptprol_AGG() line 1293 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c [2]PETSC ERROR: PCSetUp_GAMG() line 545 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c [2]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c [3]PETSC ERROR: PCSetUp_GAMG() line 545 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c [3]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c [3]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c [1]PETSC ERROR: KSPSetUp() line 266 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: KSPSolve() line 390 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c [2]PETSC ERROR: KSPSetUp() line 266 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c [2]PETSC ERROR: KSPSolve() line 390 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c KSPSetUp() line 266 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c [3]PETSC ERROR: KSPSolve() line 390 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c I'm using the options: -pc_type gamg -ksp_type bcgsl -pc_gamg_coarse_eq_limit 10 -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_monitor_true_residual -pc_gamg_verbose 2 -ksp_converged_reason -options_left -mg_levels_ksp_type richardson -mg_levels_pc_type sor -pc_gamg_threshold 0.01 -pc_gamg_repartition John On Tue, Mar 20, 2012 at 3:02 PM, Mark F. Adams wrote: > > On Mar 20, 2012, at 3:39 PM, John Mousel wrote: > > Mark, > > I am using petsc-dev that I pulled after you made the changes for the > non-symmetric discretization allocations last week. > I think the difference in our results comes from different convergence > tolerances. I'm using an rtol of 1.D-012. It seems to be converging very > nicely now. > > > Good, > > I think I dropped the option to set ksp and pc on levels after a bit, and > that seems to have made the difference. GAMG should scale much better than > HYPRE and ML right? > They both seem to work efficiently for really small core counts, but > deteriorate with impressive speed as you go up the ladder. > > > ML, should scale pretty similar to GAMG. The drop tolerance can effect > complexity but if these are about the same the ML interface uses PETSc > infrastructure. > > HYPRE is its own solver and its has been optimized for scalability. You > do have to watch for complexity getting out of hand with all AMG solvers > but they are more or less good scalable codes. > > Mark > > > [0]PCSetUp_GAMG level 0 N=46330, n data rows=1, n data cols=1, nnz/row > (ave)=6, np=4 > [0]scaleFilterGraph 75.5527% nnz after filtering, with threshold 0.05, > 6.95957 nnz ave. > [0]maxIndSetAgg removed 0 of 46330 vertices. (0 local) > [0]PCGAMGprolongator_AGG New grid 5903 nodes > PCGAMGoptprol_AGG smooth P0: max eigen=1.923098e+00 > min=3.858220e-02 PC=jacobi > [0]PCSetUp_GAMG 1) N=5903, n data cols=1, nnz/row (ave)=13, 4 > active pes > [0]scaleFilterGraph 52.8421% nnz after filtering, with threshold 0.05, > 13.3249 nnz ave. > [0]maxIndSetAgg removed 0 of 5903 vertices. (0 local) > [0]PCGAMGprolongator_AGG New grid 615 nodes > PCGAMGoptprol_AGG smooth P0: max eigen=1.575363e+00 > min=2.167886e-03 PC=jacobi > [0]PCSetUp_GAMG 2) N=615, n data cols=1, nnz/row (ave)=21, 4 > active pes > [0]scaleFilterGraph 24.7174% nnz after filtering, with threshold 0.05, > 21.722 nnz ave. > [0]maxIndSetAgg removed 0 of 615 vertices. (0 local) > [0]PCGAMGprolongator_AGG New grid 91 nodes > PCGAMGoptprol_AGG smooth P0: max eigen=1.676442e+00 > min=2.270745e-03 PC=jacobi > [0]createLevel aggregate processors: npe: 4 --> 1, neq=91 > [0]PCSetUp_GAMG 3) N=91, n data cols=1, nnz/row (ave)=37, 1 active > pes > [0]scaleFilterGraph 16.4384% nnz after filtering, with threshold 0.05, > 37.7033 nnz ave. > [0]maxIndSetAgg removed 0 of 91 vertices. (0 local) > [0]PCGAMGprolongator_AGG New grid 10 nodes > PCGAMGoptprol_AGG smooth P0: max eigen=1.538313e+00 > min=8.923063e-04 PC=jacobi > [0]PCSetUp_GAMG 4) N=10, n data cols=1, nnz/row (ave)=10, 1 active > pes > [0]PCSetUp_GAMG 5 levels, grid compexity = 1.29633 > Residual norms for pres_ solve. > 0 KSP preconditioned resid norm 4.680688832182e+06 true resid norm > 2.621342052504e+03 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP preconditioned resid norm 1.728993898497e+04 true resid norm > 2.888375221014e+03 ||r(i)||/||b|| 1.101868876004e+00 > 4 KSP preconditioned resid norm 4.510102902646e+02 true resid norm > 5.677727287161e+01 ||r(i)||/||b|| 2.165962004744e-02 > 6 KSP preconditioned resid norm 3.959846836748e+01 true resid norm > 1.973580779699e+00 ||r(i)||/||b|| 7.528894513455e-04 > 8 KSP preconditioned resid norm 3.175473803927e-01 true resid norm > 4.315977395174e-02 ||r(i)||/||b|| 1.646476235732e-05 > 10 KSP preconditioned resid norm 7.502408552205e-04 true resid norm > 1.016040400933e-04 ||r(i)||/||b|| 3.876031363257e-08 > 12 KSP preconditioned resid norm 2.868067261023e-06 true resid norm > 1.194542164810e-06 ||r(i)||/||b|| 4.556986997056e-10 > KSP Object:(pres_) 4 MPI processes > type: bcgsl > BCGSL: Ell = 2 > BCGSL: Delta = 0 > maximum iterations=5000 > tolerances: relative=1e-12, absolute=1e-50, divergence=10000 > left preconditioning > has attached null space > using nonzero initial guess > using PRECONDITIONED norm type for convergence test > PC Object:(pres_) 4 MPI processes > type: gamg > MG: type is MULTIPLICATIVE, levels=5 cycles=v > Cycles per PCApply=1 > Using Galerkin computed coarse grid matrices > Coarse grid solver -- level ------------------------------- > KSP Object: (pres_mg_coarse_) 4 MPI processes > type: richardson > Richardson: damping factor=1 > maximum iterations=1, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NONE norm type for convergence test > PC Object: (pres_mg_coarse_) 4 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 8, local iterations = 1, > omega = 1 > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=10, cols=10 > total: nonzeros=100, allocated nonzeros=100 > total number of mallocs used during MatSetValues calls =0 > using I-node (on process 0) routines: found 2 nodes, limit used > is 5 > Down solver (pre-smoother) on level 1 ------------------------------- > KSP Object: (pres_mg_levels_1_) 4 MPI processes > type: richardson > Richardson: damping factor=1 > maximum iterations=1 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using nonzero initial guess > using NONE norm type for convergence test > PC Object: (pres_mg_levels_1_) 4 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, > omega = 1 > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=91, cols=91 > total: nonzeros=3431, allocated nonzeros=3431 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Up solver (post-smoother) same as down solver (pre-smoother) > Down solver (pre-smoother) on level 2 ------------------------------- > KSP Object: (pres_mg_levels_2_) 4 MPI processes > type: richardson > Richardson: damping factor=1 > maximum iterations=1 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using nonzero initial guess > using NONE norm type for convergence test > PC Object: (pres_mg_levels_2_) 4 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, > omega = 1 > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=615, cols=615 > total: nonzeros=13359, allocated nonzeros=13359 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Up solver (post-smoother) same as down solver (pre-smoother) > Down solver (pre-smoother) on level 3 ------------------------------- > KSP Object: (pres_mg_levels_3_) 4 MPI processes > type: richardson > Richardson: damping factor=1 > maximum iterations=1 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using nonzero initial guess > using NONE norm type for convergence test > PC Object: (pres_mg_levels_3_) 4 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, > omega = 1 > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=5903, cols=5903 > total: nonzeros=78657, allocated nonzeros=78657 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Up solver (post-smoother) same as down solver (pre-smoother) > Down solver (pre-smoother) on level 4 ------------------------------- > KSP Object: (pres_mg_levels_4_) 4 MPI processes > type: richardson > Richardson: damping factor=1 > maximum iterations=1 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > has attached null space > using nonzero initial guess > using NONE norm type for convergence test > PC Object: (pres_mg_levels_4_) 4 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, > omega = 1 > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=46330, cols=46330 > total: nonzeros=322437, allocated nonzeros=615417 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > Up solver (post-smoother) same as down solver (pre-smoother) > linear system matrix = precond matrix: > Matrix Object: 4 MPI processes > type: mpiaij > rows=46330, cols=46330 > total: nonzeros=322437, allocated nonzeros=615417 > total number of mallocs used during MatSetValues calls =0 > not using I-node (on process 0) routines > > > > > On Tue, Mar 20, 2012 at 2:21 PM, Mark F. Adams wrote: > >> John, >> >> I had dome diagonal scaling stuff in my input which seemed to mess things >> up. I don't understand that. With your hypre parameters I get >> >> Complexity: grid = 1.408828 >> operator = 1.638900 >> cycle = 3.277856 >> >> 0 KSP preconditioned resid norm 2.246209947341e+06 true resid norm >> 2.621342052504e+03 ||r(i)||/||b|| 1.000000000000e+00 >> 2 KSP preconditioned resid norm 6.591054866442e+04 true resid norm >> 5.518411654910e+03 ||r(i)||/||b|| 2.105185643224e+00 >> 4 KSP preconditioned resid norm 2.721184454964e+03 true resid norm >> 1.937153214559e+03 ||r(i)||/||b|| 7.389929188022e-01 >> 6 KSP preconditioned resid norm 2.942012838854e+02 true resid norm >> 5.614763956317e+01 ||r(i)||/||b|| 2.141942502679e-02 >> 8 KSP preconditioned resid norm 2.143421596353e+01 true resid norm >> 5.306843482279e+00 ||r(i)||/||b|| 2.024475774617e-03 >> 10 KSP preconditioned resid norm 3.689048280659e+00 true resid norm >> 2.482945300243e-01 ||r(i)||/||b|| 9.472038560826e-05 >> Linear solve converged due to CONVERGED_RTOL iterations 10 >> >> with ML I get 18 iterations but if I add -pc_ml_Threshold .01 I get it to >> 12: >> >> -@${MPIEXEC} -n 1 ./ex10 -f ./binaryoutput -pc_type ml -ksp_type bcgsl >> -pc_gamg_coarse_eq_limit 10 -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph >> -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its >> 8 -ksp_monitor_true_residual -pc_gamg_verbose 2 -ksp_converged_reason >> -options_left -mg_levels_ksp_type richardson -mg_levels_pc_type sor >> -pc_ml_maxNlevels 5 -pc_ml_Threshold .01 >> >> 0 KSP preconditioned resid norm 1.987800354481e+06 true resid norm >> 2.621342052504e+03 ||r(i)||/||b|| 1.000000000000e+00 >> 2 KSP preconditioned resid norm 4.845840795806e+04 true resid norm >> 9.664923970856e+03 ||r(i)||/||b|| 3.687013666005e+00 >> 4 KSP preconditioned resid norm 4.086337251141e+03 true resid norm >> 1.111442892542e+03 ||r(i)||/||b|| 4.239976585582e-01 >> 6 KSP preconditioned resid norm 1.496117919395e+03 true resid norm >> 4.243682354730e+02 ||r(i)||/||b|| 1.618896835946e-01 >> 8 KSP preconditioned resid norm 1.019912311314e+02 true resid norm >> 6.165476121107e+01 ||r(i)||/||b|| 2.352030371320e-02 >> 10 KSP preconditioned resid norm 1.282179114927e+01 true resid norm >> 4.434755525096e+00 ||r(i)||/||b|| 1.691788189512e-03 >> 12 KSP preconditioned resid norm 2.801790417375e+00 true resid norm >> 4.876299030996e-01 ||r(i)||/||b|| 1.860229963632e-04 >> Linear solve converged due to CONVERGED_RTOL iterations 12 >> >> and gamg: >> >> -@${MPIEXEC} -n 1 ./ex10 -f ./binaryoutput -pc_type gamg -ksp_type bcgsl >> -pc_gamg_coarse_eq_limit 10 -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph >> -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its >> 8 -ksp_monitor_true_residual -pc_gamg_verbose 2 -ksp_converged_reason >> -options_left -mg_levels_ksp_type richardson -mg_levels_pc_type sor >> >> [0]PCSetUp_GAMG 5 levels, grid compexity = 1.2916 >> 0 KSP preconditioned resid norm 6.288750978813e+06 true resid norm >> 2.621342052504e+03 ||r(i)||/||b|| 1.000000000000e+00 >> 2 KSP preconditioned resid norm 3.009668424006e+04 true resid norm >> 4.394363256786e+02 ||r(i)||/||b|| 1.676379186222e-01 >> 4 KSP preconditioned resid norm 2.079756553216e+01 true resid norm >> 5.094584609440e+00 ||r(i)||/||b|| 1.943502414946e-03 >> 6 KSP preconditioned resid norm 4.323447593442e+00 true resid norm >> 3.146656048880e-01 ||r(i)||/||b|| 1.200398874261e-04 >> Linear solve converged due to CONVERGED_RTOL iterations 6 >> >> So this looks pretty different from what you are getting. Is your code >> hardwiring anything? Can you reproduce my results with ksp ex10.c? >> >> Actually, I just realized that I am using petsc-dev. What version of >> PETSc are you using? >> >> Also, here is the makefile that I use to run this jobs: >> >> ALL: runex10 >> >> include ${PETSC_DIR}/conf/variables >> include ${PETSC_DIR}/conf/rules >> >> runex10: >> -@${MPIEXEC} -n 1 ./ex10 -f ./binaryoutput -pc_type gamg -ksp_type bcgsl >> -pc_gamg_coarse_eq_limit 10 -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph >> -mg_coarse_ksp_type richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its >> 8 -ksp_monitor_true_residual -pc_gamg_verbose 2 -ksp_converged_reason >> -options_left -mg_levels_ksp_type richardson -mg_levels_pc_type sor >> -pc_ml_maxNlevels 5 -pc_ml_Threshold .01 >> -pc_hypre_boomeramg_relax_type_coarse symmetric-SOR/Jacobi >> -pc_hypre_boomeramg_grid_sweeps_coarse 4 -pc_hypre_boomeramg_coarsen_type >> PMIS >> >> You just need to run 'make ex10' and then 'make -f this-file'. >> >> Mark >> >> On Mar 20, 2012, at 2:45 PM, John Mousel wrote: >> >> Mark, >> >> I run ML with the following options. >> >> -ksp_type bcgsl -pc_type ml -pc_ml_maxNlevels 5 -mg_coarse_ksp_type >> richardson -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_monitor >> -ksp_view >> >> Note the lack of scaling. For some reason scaling seems to mess with ML. >> As you can see below, ML converges very nicely. >> >> With regards to HYPRE, this one took a bit of work to get convergence. >> The options that work are: >> >> -ksp_type bcgsl -pc_type hypre -pc_hypre_type boomeramg >> -ksp_monitor_true_residual -pc_hypre_boomeramg_relax_type_coarse >> symmetric-SOR/Jacobi -pc_hypre_boomeramg_grid_sweeps_coarse 4 >> -pc_hypre_boomeramg_coarsen_type PMIS >> >> The problem is that neither of ML or HYPRE seem to scale at all. >> >> ML output: >> 0 KSP preconditioned resid norm 1.538968715109e+06 true resid norm >> 2.621342052504e+03 ||r(i)||/||b|| 1.000000000000e+00 >> 2 KSP preconditioned resid norm 1.263129058693e+05 true resid norm >> 1.096298699671e+04 ||r(i)||/||b|| 4.182203915830e+00 >> 4 KSP preconditioned resid norm 2.277379585186e+04 true resid norm >> 2.999721659930e+03 ||r(i)||/||b|| 1.144345758717e+00 >> 6 KSP preconditioned resid norm 4.766504457975e+03 true resid norm >> 6.733421603796e+02 ||r(i)||/||b|| 2.568692474667e-01 >> 8 KSP preconditioned resid norm 2.139020425406e+03 true resid norm >> 1.360842101250e+02 ||r(i)||/||b|| 5.191394613876e-02 >> 10 KSP preconditioned resid norm 6.621380459944e+02 true resid norm >> 1.522758800025e+02 ||r(i)||/||b|| 5.809080881188e-02 >> 12 KSP preconditioned resid norm 2.973409610262e+01 true resid norm >> 1.161046206089e+01 ||r(i)||/||b|| 4.429205280479e-03 >> 14 KSP preconditioned resid norm 2.532665482573e+00 true resid norm >> 2.557425874623e+00 ||r(i)||/||b|| 9.756170020543e-04 >> 16 KSP preconditioned resid norm 2.375585214826e+00 true resid norm >> 2.441783841415e+00 ||r(i)||/||b|| 9.315014189327e-04 >> 18 KSP preconditioned resid norm 1.436338060675e-02 true resid norm >> 1.305304828818e-02 ||r(i)||/||b|| 4.979528816437e-06 >> 20 KSP preconditioned resid norm 4.088293864561e-03 true resid norm >> 9.841243465634e-04 ||r(i)||/||b|| 3.754276728683e-07 >> 22 KSP preconditioned resid norm 6.140822977383e-04 true resid norm >> 1.682184150207e-04 ||r(i)||/||b|| 6.417263052718e-08 >> 24 KSP preconditioned resid norm 2.685415483430e-05 true resid norm >> 1.065041542336e-05 ||r(i)||/||b|| 4.062962867890e-09 >> 26 KSP preconditioned resid norm 1.620776166579e-06 true resid norm >> 9.563268703474e-07 ||r(i)||/||b|| 3.648233809982e-10 >> 28 KSP preconditioned resid norm 2.823291105652e-07 true resid norm >> 7.705418741178e-08 ||r(i)||/||b|| 2.939493811507e-11 >> KSP Object:(pres_) 4 MPI processes >> type: bcgsl >> BCGSL: Ell = 2 >> BCGSL: Delta = 0 >> maximum iterations=5000 >> tolerances: relative=1e-12, absolute=1e-50, divergence=10000 >> left preconditioning >> has attached null space >> using nonzero initial guess >> using PRECONDITIONED norm type for convergence test >> PC Object:(pres_) 4 MPI processes >> type: ml >> MG: type is MULTIPLICATIVE, levels=5 cycles=v >> Cycles per PCApply=1 >> Using Galerkin computed coarse grid matrices >> Coarse grid solver -- level ------------------------------- >> KSP Object: (pres_mg_coarse_) 4 MPI processes >> type: richardson >> Richardson: damping factor=1 >> maximum iterations=1, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> using PRECONDITIONED norm type for convergence test >> PC Object: (pres_mg_coarse_) 4 MPI processes >> type: sor >> SOR: type = local_symmetric, iterations = 8, local iterations = >> 1, omega = 1 >> linear system matrix = precond matrix: >> Matrix Object: 4 MPI processes >> type: mpiaij >> rows=4, cols=4 >> total: nonzeros=16, allocated nonzeros=16 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node (on process 0) routines >> Down solver (pre-smoother) on level 1 ------------------------------- >> KSP Object: (pres_mg_levels_1_) 4 MPI processes >> type: richardson >> Richardson: damping factor=1 >> maximum iterations=1 >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> using nonzero initial guess >> using PRECONDITIONED norm type for convergence test >> PC Object: (pres_mg_levels_1_) 4 MPI processes >> type: sor >> SOR: type = local_symmetric, iterations = 1, local iterations = >> 1, omega = 1 >> linear system matrix = precond matrix: >> Matrix Object: 4 MPI processes >> type: mpiaij >> rows=25, cols=25 >> total: nonzeros=303, allocated nonzeros=303 >> total number of mallocs used during MatSetValues calls =0 >> using I-node (on process 0) routines: found 4 nodes, limit used >> is 5 >> Up solver (post-smoother) same as down solver (pre-smoother) >> Down solver (pre-smoother) on level 2 ------------------------------- >> KSP Object: (pres_mg_levels_2_) 4 MPI processes >> type: richardson >> Richardson: damping factor=1 >> maximum iterations=1 >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> using nonzero initial guess >> using PRECONDITIONED norm type for convergence test >> PC Object: (pres_mg_levels_2_) 4 MPI processes >> type: sor >> SOR: type = local_symmetric, iterations = 1, local iterations = >> 1, omega = 1 >> linear system matrix = precond matrix: >> Matrix Object: 4 MPI processes >> type: mpiaij >> rows=423, cols=423 >> total: nonzeros=7437, allocated nonzeros=7437 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node (on process 0) routines >> Up solver (post-smoother) same as down solver (pre-smoother) >> Down solver (pre-smoother) on level 3 ------------------------------- >> KSP Object: (pres_mg_levels_3_) 4 MPI processes >> type: richardson >> Richardson: damping factor=1 >> maximum iterations=1 >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> using nonzero initial guess >> using PRECONDITIONED norm type for convergence test >> PC Object: (pres_mg_levels_3_) 4 MPI processes >> type: sor >> SOR: type = local_symmetric, iterations = 1, local iterations = >> 1, omega = 1 >> linear system matrix = precond matrix: >> Matrix Object: 4 MPI processes >> type: mpiaij >> rows=6617, cols=6617 >> total: nonzeros=88653, allocated nonzeros=88653 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node (on process 0) routines >> Up solver (post-smoother) same as down solver (pre-smoother) >> Down solver (pre-smoother) on level 4 ------------------------------- >> KSP Object: (pres_mg_levels_4_) 4 MPI processes >> type: richardson >> Richardson: damping factor=1 >> maximum iterations=1 >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> has attached null space >> using nonzero initial guess >> using PRECONDITIONED norm type for convergence test >> PC Object: (pres_mg_levels_4_) 4 MPI processes >> type: sor >> SOR: type = local_symmetric, iterations = 1, local iterations = >> 1, omega = 1 >> linear system matrix = precond matrix: >> Matrix Object: 4 MPI processes >> type: mpiaij >> rows=46330, cols=46330 >> total: nonzeros=322437, allocated nonzeros=615417 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node (on process 0) routines >> Up solver (post-smoother) same as down solver (pre-smoother) >> linear system matrix = precond matrix: >> Matrix Object: 4 MPI processes >> type: mpiaij >> rows=46330, cols=46330 >> total: nonzeros=322437, allocated nonzeros=615417 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node (on process 0) routines >> >> >> >> John >> >> >> >> On Tue, Mar 20, 2012 at 1:33 PM, Mark F. Adams wrote: >> >>> John, >>> >>> I am getting poor results (diverging) from ML also: >>> >>> 0 KSP preconditioned resid norm 3.699832960909e+22 true resid norm >>> 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 >>> 2 KSP preconditioned resid norm 5.706378365783e+11 true resid norm >>> 1.563902233018e+03 ||r(i)||/||b|| 1.193204539995e+00 >>> 4 KSP preconditioned resid norm 5.570291685152e+11 true resid norm >>> 1.564235542744e+03 ||r(i)||/||b|| 1.193458844050e+00 >>> 6 KSP preconditioned resid norm 5.202150407298e+10 true resid norm >>> 1.749929789082e+03 ||r(i)||/||b|| 1.335137277077e+00 >>> Linear solve converged due to CONVERGED_RTOL iterations 6 >>> >>> With GAMG I get: >>> >>> 0 KSP preconditioned resid norm 7.731260075891e+06 true resid norm >>> 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 >>> 2 KSP preconditioned resid norm 2.856415184685e+05 true resid norm >>> 1.310410242531e+03 ||r(i)||/||b|| 9.997987199150e-01 >>> 4 KSP preconditioned resid norm 1.528467019258e+05 true resid norm >>> 1.284856538976e+03 ||r(i)||/||b|| 9.803021078816e-01 >>> 6 KSP preconditioned resid norm 1.451091957899e+05 true resid norm >>> 1.564309254168e+03 ||r(i)||/||b|| 1.193515083375e+00 >>> >>> >>> >>> 122 KSP preconditioned resid norm 2.486245341783e+01 true resid norm >>> 1.404397185367e+00 ||r(i)||/||b|| 1.071507580306e-03 >>> 124 KSP preconditioned resid norm 1.482316853621e+01 true resid norm >>> 4.488661881759e-01 ||r(i)||/||b|| 3.424697287811e-04 >>> 126 KSP preconditioned resid norm 1.481941150253e+01 true resid norm >>> 4.484480100832e-01 ||r(i)||/||b|| 3.421506730318e-04 >>> 128 KSP preconditioned resid norm 8.191887347033e+00 true resid norm >>> 6.678630367218e-01 ||r(i)||/||b|| 5.095569215816e-04 >>> >>> And HYPRE: >>> >>> 0 KSP preconditioned resid norm 3.774510769907e+04 true resid norm >>> 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 >>> 2 KSP preconditioned resid norm 1.843165835831e+04 true resid norm >>> 8.502433792869e+02 ||r(i)||/||b|| 6.487069580482e-01 >>> 4 KSP preconditioned resid norm 1.573176624705e+04 true resid norm >>> 1.167264367302e+03 ||r(i)||/||b|| 8.905832558033e-01 >>> 6 KSP preconditioned resid norm 1.657958380765e+04 true resid norm >>> 8.684701624902e+02 ||r(i)||/||b|| 6.626133775216e-01 >>> 8 KSP preconditioned resid norm 2.190304455083e+04 true resid norm >>> 6.969893263600e+02 ||r(i)||/||b|| 5.317792960344e-01 >>> 10 KSP preconditioned resid norm 2.485714630000e+04 true resid norm >>> 6.642641436830e+02 ||r(i)||/||b|| 5.068110878446e-01 >>> >>> >>> >>> 62 KSP preconditioned resid norm 6.432516040957e+00 true resid norm >>> 2.124960171419e-01 ||r(i)||/||b|| 1.621272781837e-04 >>> 64 KSP preconditioned resid norm 5.731033176541e+00 true resid norm >>> 1.338816774003e-01 ||r(i)||/||b|| 1.021471943216e-04 >>> 66 KSP preconditioned resid norm 1.600285935522e-01 true resid norm >>> 3.352408932031e-03 ||r(i)||/||b|| 2.557774695353e-06 >>> >>> ML and GAMG should act similarly, but ML seems to have a problem (see >>> the preconditioned norm difference and its diverging). ML has a parameter: >>> >>> -pc_ml_Threshold [.0] >>> >>> I setting this to 0.05 (GAMG default) helps a bit but it still diverges. >>> >>> So it would be nice to figure out the difference between ML and GAMG, >>> but that is secondary for you as the both suck. >>> >>> HYPRE is a very different algorithm. It looks like the smoothing in >>> GAMG (and ML) may be the problem. If I turn smoothing off >>> (-pc_gamg_agg_nsmooths 0) and I get for GAMG: >>> >>> 0 KSP preconditioned resid norm 2.186148437534e+05 true resid norm >>> 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 >>> 2 KSP preconditioned resid norm 2.916843959765e+04 true resid norm >>> 3.221533667508e+03 ||r(i)||/||b|| 2.457921292432e+00 >>> 4 KSP preconditioned resid norm 2.396374655925e+04 true resid norm >>> 1.834299897412e+03 ||r(i)||/||b|| 1.399508817812e+00 >>> 6 KSP preconditioned resid norm 2.509576275453e+04 true resid norm >>> 1.035475461174e+03 ||r(i)||/||b|| 7.900327752214e-01 >>> >>> >>> >>> 64 KSP preconditioned resid norm 1.973859758284e+01 true resid norm >>> 7.322674977169e+00 ||r(i)||/||b|| 5.586953482895e-03 >>> 66 KSP preconditioned resid norm 3.371598890438e+01 true resid norm >>> 7.152754930495e+00 ||r(i)||/||b|| 5.457310231004e-03 >>> 68 KSP preconditioned resid norm 4.687839294418e+00 true resid norm >>> 4.605726307025e-01 ||r(i)||/||b|| 3.514013487219e-04 >>> 70 KSP preconditioned resid norm 1.487545519493e+00 true resid norm >>> 1.558723789416e-01 ||r(i)||/||b|| 1.189253562571e-04 >>> 72 KSP preconditioned resid norm 5.317329808718e-01 true resid norm >>> 5.027178038177e-02 ||r(i)||/||b|| 3.835566911967e-05 >>> 74 KSP preconditioned resid norm 3.405339702462e-01 true resid norm >>> 1.897059263835e-02 ||r(i)||/||b|| 1.447392092969e-05 >>> >>> This is almost as good as HYPRE. >>> >>> An other thing to keep in mind is the cost of each iteration, not just >>> then number of iterations, You can >>> use -pc_hypre_boomeramg_print_statistics to get some data on this from >>> HYPRE: >>> >>> Average Convergence Factor = 0.537664 >>> >>> Complexity: grid = 1.780207 >>> operator = 2.624910 >>> cycle = 5.249670 >>> >>> And GAMG prints this with verbose set: >>> >>> [0]PCSetUp_GAMG 6 levels, grid compexity [sic] = 1.1316 >>> >>> I believe that the hypre "Complexity: grid" is the same as my "grid >>> complexity". So hypre actually looks more expensive at this point. >>> >>> I've worked on optimizing parameters for hypre with the hypre people and >>> here are a set of arguments that I've used: >>> >>> -pc_hypre_boomeramg_no_CF -pc_hypre_boomeramg_agg_nl 1 >>> -pc_hypre_boomeramg_coarsen_type HMIS -pc_hypre_boomeramg_interp_type ext+i >>> -pc_hypre_boomeramg_P_max 4 -pc_hypre_boomeramg_agg_num_paths 2 >>> >>> With these parmeters I get: >>> >>> Complexity: grid = 1.244140 >>> operator = 1.396722 >>> cycle = 2.793442 >>> >>> and: >>> >>> 0 KSP preconditioned resid norm 4.698624821403e+04 true resid norm >>> 1.310674055116e+03 ||r(i)||/||b|| 1.000000000000e+00 >>> 2 KSP preconditioned resid norm 2.207967626172e+04 true resid norm >>> 3.466160021150e+03 ||r(i)||/||b|| 2.644562931280e+00 >>> 4 KSP preconditioned resid norm 2.278468320876e+04 true resid norm >>> 1.246784122467e+03 ||r(i)||/||b|| 9.512541410282e-01 >>> >>> >>> >>> 56 KSP preconditioned resid norm 1.108460232262e+00 true resid norm >>> 8.276869475681e-02 ||r(i)||/||b|| 6.314971631105e-05 >>> 58 KSP preconditioned resid norm 3.617217454336e-01 true resid norm >>> 3.764556404754e-02 ||r(i)||/||b|| 2.872229285428e-05 >>> 60 KSP preconditioned resid norm 1.666532560770e-01 true resid norm >>> 5.149302513338e-03 ||r(i)||/||b|| 3.928743758404e-06 >>> Linear solve converged due to CONVERGED_RTOL iterations 60 >>> >>> So this actually converged faster with lower complexity. >>> >>> Anyway these result seem different from what you are getting, so I've >>> appended my options. This uses ex10 in the KSP tutorials to read in your >>> binary file. >>> >>> Mark >>> >>> #PETSc Option Table entries: >>> -f ./binaryoutput >>> -ksp_converged_reason >>> -ksp_diagonal_scale >>> -ksp_diagonal_scale_fix >>> -ksp_monitor_true_residual >>> -ksp_type bcgsl >>> -mg_coarse_ksp_type richardson >>> -mg_coarse_pc_sor_its 8 >>> -mg_coarse_pc_type sor >>> -mg_levels_ksp_type richardson >>> -mg_levels_pc_type sor >>> -options_left >>> -pc_gamg_agg_nsmooths 0 >>> -pc_gamg_coarse_eq_limit 10 >>> -pc_gamg_sym_graph >>> -pc_gamg_verbose 2 >>> -pc_hypre_boomeramg_P_max 4 >>> -pc_hypre_boomeramg_agg_nl 1 >>> -pc_hypre_boomeramg_agg_num_paths 2 >>> -pc_hypre_boomeramg_coarsen_type HMIS >>> -pc_hypre_boomeramg_interp_type ext+i >>> -pc_hypre_boomeramg_no_CF >>> -pc_ml_Threshold .01 >>> -pc_type gamg >>> -vecload_block_size 1 >>> #End of PETSc Option Table entries >>> There are 7 unused database options. They are: >>> Option left: name:-pc_hypre_boomeramg_P_max value: 4 >>> Option left: name:-pc_hypre_boomeramg_agg_nl value: 1 >>> Option left: name:-pc_hypre_boomeramg_agg_num_paths value: 2 >>> Option left: name:-pc_hypre_boomeramg_coarsen_type value: HMIS >>> Option left: name:-pc_hypre_boomeramg_interp_type value: ext+i >>> Option left: name:-pc_hypre_boomeramg_no_CF no value >>> Option left: name:-pc_ml_Threshold value: .01 >>> >>> >>> On Mar 20, 2012, at 10:19 AM, John Mousel wrote: >>> >>> Mark, >>> >>> Sorry for the late reply. I've been on travel and hadn't had a chance to >>> pick this back up. I've tried running with the suggested options: >>> >>> -ksp_type bcgsl -pc_type gamg -pc_gamg_coarse_eq_limit 10 >>> -pc_gamg_agg_nsmooths 1 -pc_gamg_sym_graph -mg_coarse_ksp_type richardson >>> -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 -ksp_diagonal_scale >>> -ksp_diagonal_scale_fix -ksp_monitor_true_residual -ksp_view >>> -pc_gamg_verbose 1 >>> >>> With these options, the convergence starts to hang (see attached >>> GAMG_kspview.txt). The hanging happens for both -mg_coarse_ksp_type >>> richardson and preonly. It was my understanding from previous emails that >>> using preonly made it so that only the preconditioner was run, which in >>> this case would be 8 sweeps of SOR. If I get rid of the >>> -pc_gamg_agg_nsmooths 1 (see GAMG_kspview_nosmooth.txt), the problem >>> converges, but again the convergence is slow. Without this option, both >>> Richardson and preonly converge in 172 iterations. >>> >>> Matt, I've checked and the problem does converge in the true residual >>> using GAMG, ML, HYPRE, and ILU preconditioned BiCG. I explicitly ensure >>> that a solution exists by projecting the rhs vector out of the nullity of >>> the transpose of operator. >>> >>> John >>> >>> >>> On Fri, Mar 16, 2012 at 2:04 PM, Mark F. Adams wrote: >>> >>>> John, did this get resolved? >>>> Mark >>>> >>>> On Mar 15, 2012, at 4:24 PM, John Mousel wrote: >>>> >>>> Mark, >>>> >>>> Running without the options you mentioned before leads to slightly >>>> worse performance (175 iterations). >>>> I have not been able to get run coarse grid solve to work with LU while >>>> running ML. It keeps experiencing a zero pivot, and all the combinations of >>>> shifting i've tried haven't lead me anywhere, hence the SOR on the course >>>> grid. Also, the ML manual suggests limiting the number of levels to 3 or 4 >>>> and performing a few sweeps of an iterative method as opposed to a direct >>>> solve. >>>> >>>> John >>>> >>>> On Thu, Mar 15, 2012 at 12:04 PM, Mark F. Adams < >>>> mark.adams at columbia.edu> wrote: >>>> >>>>> You also want: -pc_gamg_agg_nsmooths 1 >>>>> >>>>> You are running plain aggregation. If it is Poisson then smoothing is >>>>> good. >>>>> >>>>> Is this problem singular? Can you try running ML with these >>>>> parameters and see if its performance degrades? The ML implementation uses >>>>> the PETSC infrastructure and uses a very similar algorithm to GAMG-SA. We >>>>> should be able to get these two to match pretty well. >>>>> >>>>> Mark >>>>> >>>>> >>>>> On Mar 15, 2012, at 12:21 PM, John Mousel wrote: >>>>> >>>>> Mark, >>>>> >>>>> I ran with those options removed (see the run options listed below). >>>>> Things actually got slightly worse. Now it's up to 142 iterations. I have >>>>> attached the ksp_view output. >>>>> >>>>> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale >>>>> -ksp_diagonal_scale_fix -mg_levels_ksp_type richardson -mg_levels_pc_type >>>>> sor -pc_gamg_verbose 1 >>>>> >>>>> >>>>> John >>>>> >>>>> >>>>> On Thu, Mar 15, 2012 at 10:55 AM, Mark F. Adams < >>>>> mark.adams at columbia.edu> wrote: >>>>> >>>>>> John, can you run again with: -pc_gamg_verbose 1 >>>>>> >>>>>> And I would not use: -pc_mg_levels 4 -mg_coarse_ksp_type preonly >>>>>> -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8 >>>>>> >>>>>> 1) I think -mg_coarse_ksp_type preonly and -mg_coarse_pc_sor_its 8 do >>>>>> not do what you think. I think this is the same as 1 iteration. I think >>>>>> you want 'richardson' not 'preonly'. >>>>>> >>>>>> 2) Why are you using sor as the coarse solver? If your problem is >>>>>> singular then you want to use as many levels as possible to get the coarse >>>>>> grid to be tiny. I'm pretty sure HYPRE ignores the coarse solver >>>>>> parameters. But ML uses them and it is converging well. >>>>>> >>>>>> 3) I would not specify the number of levels. GAMG, and I think the >>>>>> rest, have internal logic for stopping a the right level. If the coarse >>>>>> level is large and you use just 8 iterations of sor then convergence will >>>>>> suffer. >>>>>> >>>>>> Mark >>>>>> >>>>>> On Mar 15, 2012, at 11:13 AM, John Mousel wrote: >>>>>> >>>>>> Mark, >>>>>> >>>>>> The changes pulled through this morning. I've run it with the options >>>>>> >>>>>> -ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale >>>>>> -ksp_diagonal_scale_fix -pc_mg_levels 4 -mg_levels_ksp_type richardson >>>>>> -mg_levels_pc_type sor -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor >>>>>> -mg_coarse_pc_sor_its 8 >>>>>> >>>>>> and it converges in the true residual, but it's not converging as >>>>>> fast as anticpated. The matrix arises from a non-symmetric discretization >>>>>> of the Poisson equation. The solve takes GAMG 114 iterations, whereas ML >>>>>> takes 24 iterations, BoomerAMG takes 22 iterations, and -ksp_type bcgsl >>>>>> -pc_type bjacobi -sub_pc_type ilu -sub_pc_factor_levels 4 takes around 170. >>>>>> I've attached the -ksp_view results for ML,GAMG, and HYPRE. I've attempted >>>>>> to make all the options the same on all levels for ML and GAMG. >>>>>> >>>>>> Any thoughts? >>>>>> >>>>>> John >>>>>> >>>>>> >>>>>> On Wed, Mar 14, 2012 at 6:04 PM, Mark F. Adams < >>>>>> mark.adams at columbia.edu> wrote: >>>>>> >>>>>>> Humm, I see it with hg view (appended). >>>>>>> >>>>>>> Satish, my main repo looks hosed. I see this: >>>>>>> >>>>>>> ~/Codes/petsc-dev>hg update >>>>>>> abort: crosses branches (merge branches or use --clean to discard >>>>>>> changes) >>>>>>> ~/Codes/petsc-dev>hg merge >>>>>>> abort: branch 'default' has 3 heads - please merge with an explicit >>>>>>> rev >>>>>>> (run 'hg heads .' to see heads) >>>>>>> ~/Codes/petsc-dev>hg heads >>>>>>> changeset: 22496:8e2a98268179 >>>>>>> tag: tip >>>>>>> user: Barry Smith >>>>>>> date: Wed Mar 14 16:42:25 2012 -0500 >>>>>>> files: src/vec/is/interface/f90-custom/zindexf90.c >>>>>>> src/vec/vec/interface/f90-custom/zvectorf90.c >>>>>>> description: >>>>>>> undoing manually changes I put in because Satish had a better fix >>>>>>> >>>>>>> >>>>>>> changeset: 22492:bda4df63072d >>>>>>> user: Mark F. Adams >>>>>>> date: Wed Mar 14 17:39:52 2012 -0400 >>>>>>> files: src/ksp/pc/impls/gamg/tools.c >>>>>>> description: >>>>>>> fix for unsymmetric matrices. >>>>>>> >>>>>>> >>>>>>> changeset: 22469:b063baf366e4 >>>>>>> user: Mark F. Adams >>>>>>> date: Wed Mar 14 14:22:28 2012 -0400 >>>>>>> files: src/ksp/pc/impls/gamg/tools.c >>>>>>> description: >>>>>>> added fix for preallocation for unsymetric matrices. >>>>>>> >>>>>>> Mark >>>>>>> >>>>>>> my 'hg view' on my merge repo: >>>>>>> >>>>>>> Revision: 22492 >>>>>>> Branch: default >>>>>>> Author: Mark F. Adams 2012-03-14 17:39:52 >>>>>>> Committer: Mark F. Adams 2012-03-14 >>>>>>> 17:39:52 >>>>>>> Tags: tip >>>>>>> Parent: 22491:451bbbd291c2 (Small fixes to the BT linesearch) >>>>>>> >>>>>>> fix for unsymmetric matrices. >>>>>>> >>>>>>> >>>>>>> ------------------------ src/ksp/pc/impls/gamg/tools.c >>>>>>> ------------------------ >>>>>>> @@ -103,7 +103,7 @@ >>>>>>> PetscErrorCode ierr; >>>>>>> PetscInt Istart,Iend,Ii,jj,ncols,nnz0,nnz1, NN, MM, nloc; >>>>>>> PetscMPIInt mype, npe; >>>>>>> - Mat Gmat = *a_Gmat, tGmat; >>>>>>> + Mat Gmat = *a_Gmat, tGmat, matTrans; >>>>>>> MPI_Comm wcomm = ((PetscObject)Gmat)->comm; >>>>>>> const PetscScalar *vals; >>>>>>> const PetscInt *idx; >>>>>>> @@ -127,6 +127,10 @@ >>>>>>> ierr = MatDiagonalScale( Gmat, diag, diag ); CHKERRQ(ierr); >>>>>>> ierr = VecDestroy( &diag ); CHKERRQ(ierr); >>>>>>> >>>>>>> + if( symm ) { >>>>>>> + ierr = MatTranspose( Gmat, MAT_INITIAL_MATRIX, &matTrans ); >>>>>>> CHKERRQ(ierr); >>>>>>> + } >>>>>>> + >>>>>>> /* filter - dup zeros out matrix */ >>>>>>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &d_nnz ); >>>>>>> CHKERRQ(ierr); >>>>>>> ierr = PetscMalloc( nloc*sizeof(PetscInt), &o_nnz ); >>>>>>> CHKERRQ(ierr); >>>>>>> @@ -135,6 +139,12 @@ >>>>>>> d_nnz[jj] = ncols; >>>>>>> o_nnz[jj] = ncols; >>>>>>> ierr = MatRestoreRow(Gmat,Ii,&ncols,PETSC_NULL,PETSC_NULL); >>>>>>> CHKERRQ(ierr); >>>>>>> + if( symm ) { >>>>>>> + ierr = MatGetRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); >>>>>>> CHKERRQ(ierr); >>>>>>> + d_nnz[jj] += ncols; >>>>>>> + o_nnz[jj] += ncols; >>>>>>> + ierr = >>>>>>> MatRestoreRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); >>>>>>> + } >>>>>>> if( d_nnz[jj] > nloc ) d_nnz[jj] = nloc; >>>>>>> if( o_nnz[jj] > (MM-nloc) ) o_nnz[jj] = MM - nloc; >>>>>>> } >>>>>>> @@ -142,6 +152,9 @@ >>>>>>> CHKERRQ(ierr); >>>>>>> ierr = PetscFree( d_nnz ); CHKERRQ(ierr); >>>>>>> ierr = PetscFree( o_nnz ); CHKERRQ(ierr); >>>>>>> + if( symm ) { >>>>>>> + ierr = MatDestroy( &matTrans ); CHKERRQ(ierr); >>>>>>> + } >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mar 14, 2012, at 5:53 PM, John Mousel wrote: >>>>>>> >>>>>>> Mark, >>>>>>> >>>>>>> No change. Can you give me the location that you patched so I can >>>>>>> check to make sure it pulled? >>>>>>> I don't see it on the petsc-dev change log. >>>>>>> >>>>>>> John >>>>>>> >>>>>>> On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams < >>>>>>> mark.adams at columbia.edu> wrote: >>>>>>> >>>>>>>> John, I've committed these changes, give a try. >>>>>>>> >>>>>>>> Mark >>>>>>>> >>>>>>>> On Mar 14, 2012, at 3:46 PM, Satish Balay wrote: >>>>>>>> >>>>>>>> > This is the usual merge [with uncommited changes] issue. >>>>>>>> > >>>>>>>> > You could use 'hg shelf' extension to shelve your local changes >>>>>>>> and >>>>>>>> > then do a merge [as Sean would suggest] - or do the merge in a >>>>>>>> > separate/clean clone [I normally do this..] >>>>>>>> > >>>>>>>> > i.e >>>>>>>> > cd ~/Codes >>>>>>>> > hg clone petsc-dev petsc-dev-merge >>>>>>>> > cd petsc-dev-merge >>>>>>>> > hg pull ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev #just >>>>>>>> to be sure, look for latest chagnes before merge.. >>>>>>>> > hg merge >>>>>>>> > hg commit >>>>>>>> > hg push ssh://petsc at petsc.cs.iit.edu//hg/petsc/petsc-dev >>>>>>>> > >>>>>>>> > [now update your petsc-dev to latest] >>>>>>>> > cd ~/Codes/petsc-dev >>>>>>>> > hg pull >>>>>>>> > hg update >>>>>>>> > >>>>>>>> > Satish >>>>>>>> > >>>>>>>> > On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>>>>>> > >>>>>>>> >> Great, that seems to work. >>>>>>>> >> >>>>>>>> >> I did a 'hg commit tools.c' >>>>>>>> >> >>>>>>>> >> and I want to push this file only. I guess its the only thing >>>>>>>> in the change set so 'hg push' should be fine. But I see this: >>>>>>>> >> >>>>>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update >>>>>>>> >> abort: crosses branches (merge branches or use --clean to >>>>>>>> discard changes) >>>>>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge >>>>>>>> >> abort: outstanding uncommitted changes (use 'hg status' to list >>>>>>>> changes) >>>>>>>> >> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status >>>>>>>> >> M include/petscmat.h >>>>>>>> >> M include/private/matimpl.h >>>>>>>> >> M src/ksp/pc/impls/gamg/agg.c >>>>>>>> >> M src/ksp/pc/impls/gamg/gamg.c >>>>>>>> >> M src/ksp/pc/impls/gamg/gamg.h >>>>>>>> >> M src/ksp/pc/impls/gamg/geo.c >>>>>>>> >> M src/mat/coarsen/coarsen.c >>>>>>>> >> M src/mat/coarsen/impls/hem/hem.c >>>>>>>> >> M src/mat/coarsen/impls/mis/mis.c >>>>>>>> >> >>>>>>>> >> Am I ready to do a push? >>>>>>>> >> >>>>>>>> >> Thanks, >>>>>>>> >> Mark >>>>>>>> >> >>>>>>>> >> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote: >>>>>>>> >> >>>>>>>> >>> If commit is the last hg operation that you've done - then 'hg >>>>>>>> rollback' would undo this commit. >>>>>>>> >>> >>>>>>>> >>> Satish >>>>>>>> >>> >>>>>>>> >>> On Wed, 14 Mar 2012, Mark F. Adams wrote: >>>>>>>> >>> >>>>>>>> >>>> Damn, I'm not preallocating the graph perfectly for >>>>>>>> unsymmetric matrices and PETSc now dies on this. >>>>>>>> >>>> >>>>>>>> >>>> I have a fix but I committed it with other changes that I do >>>>>>>> not want to commit. The changes are all in one file so I should be able to >>>>>>>> just commit this file. >>>>>>>> >>>> >>>>>>>> >>>> Anyone know how to delete a commit? >>>>>>>> >>>> >>>>>>>> >>>> I've tried: >>>>>>>> >>>> >>>>>>>> >>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip >>>>>>>> 22487:26ffb9eef17f >>>>>>>> >>>> hg: unknown command 'strip' >>>>>>>> >>>> 'strip' is provided by the following extension: >>>>>>>> >>>> >>>>>>>> >>>> mq manage a stack of patches >>>>>>>> >>>> >>>>>>>> >>>> use "hg help extensions" for information on enabling extensions >>>>>>>> >>>> >>>>>>>> >>>> But have not figured out how to load extensions. >>>>>>>> >>>> >>>>>>>> >>>> Mark >>>>>>>> >>>> >>>>>>>> >>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote: >>>>>>>> >>>> >>>>>>>> >>>>> Mark, >>>>>>>> >>>>> >>>>>>>> >>>>> I have a non-symmetric matrix. I am running with the >>>>>>>> following options. >>>>>>>> >>>>> >>>>>>>> >>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual >>>>>>>> >>>>> >>>>>>>> >>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new >>>>>>>> malloc error: >>>>>>>> >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> 0]PETSC ERROR: --------------------- Error Message >>>>>>>> ------------------------------------ >>>>>>>> >>>>> [0]PETSC ERROR: Argument out of range! >>>>>>>> >>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc! >>>>>>>> >>>>> [0]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> >>>>> [0]PETSC ERROR: Petsc Development HG revision: >>>>>>>> 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 >>>>>>>> -0500 >>>>>>>> >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent >>>>>>>> updates. >>>>>>>> >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble >>>>>>>> shooting. >>>>>>>> >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>> >>>>> [0]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> >>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named >>>>>>>> wv.iihr.uiowa.edu by jmousel Wed Mar 14 11:51:35 2012 >>>>>>>> >>>>> [0]PETSC ERROR: Libraries linked from >>>>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib >>>>>>>> >>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012 >>>>>>>> >>>>> [0]PETSC ERROR: Configure options --download-blacs=1 >>>>>>>> --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 >>>>>>>> --download-parmetis=1 --download-scalapack=1 >>>>>>>> --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc >>>>>>>> --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort >>>>>>>> PETSC_ARCH=linux-debug >>>>>>>> >>>>> [0]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> >>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in >>>>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c >>>>>>>> >>>>> [0]PETSC ERROR: MatSetValues() line 1141 in >>>>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c >>>>>>>> >>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in >>>>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c >>>>>>>> >>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in >>>>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c >>>>>>>> >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in >>>>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>>>>>>> >>>>> [0]PETSC ERROR: PCSetUp() line 832 in >>>>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c >>>>>>>> >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in >>>>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>>>>> >>>>> [0]PETSC ERROR: KSPSolve() line 385 in >>>>>>>> /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>>>>> >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> John >>>>>>>> >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams < >>>>>>>> mark.adams at columbia.edu> wrote: >>>>>>>> >>>>> >>>>>>>> >>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote: >>>>>>>> >>>>> >>>>>>>> >>>>>> Mark, >>>>>>>> >>>>>> >>>>>>>> >>>>>> The matrix is asymmetric. Does this require the setting of >>>>>>>> an option? >>>>>>>> >>>>> >>>>>>>> >>>>> Yes: -pc_gamg_sym_graph >>>>>>>> >>>>> >>>>>>>> >>>>> Mark >>>>>>>> >>>>> >>>>>>>> >>>>>> I pulled petsc-dev this morning, so I should have (at least >>>>>>>> close to) the latest code. >>>>>>>> >>>>>> >>>>>>>> >>>>>> John >>>>>>>> >>>>>> >>>>>>>> >>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams < >>>>>>>> mark.adams at columbia.edu> wrote: >>>>>>>> >>>>>> >>>>>>>> >>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote: >>>>>>>> >>>>>> >>>>>>>> >>>>>>> I'm getting the following error when using GAMG. >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: >>>>>>>> Assertion `sgid==-1' failed. >>>>>>>> >>>>>> >>>>>>>> >>>>>> Is it possible that your matrix is structurally asymmetric? >>>>>>>> >>>>>> >>>>>>>> >>>>>> This code is evolving fast and so you will need to move to >>>>>>>> the dev version if you are not already using it. (I think I fixed a bug >>>>>>>> that hit this assert). >>>>>>>> >>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> When I try to alter the type of aggregation at the command >>>>>>>> line using -pc_gamg_type pa, I'm getting >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error >>>>>>>> Message ------------------------------------ >>>>>>>> >>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or >>>>>>>> missing external package needed for type: >>>>>>>> >>>>>>> see >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/installation.html#external >>>>>>>> ! >>>>>>>> >>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given! >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> Has there been a change in the aggregation options? I just >>>>>>>> pulled petsc-dev this morning. >>>>>>>> >>>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg >>>>>>>> for now. >>>>>>>> >>>>>> >>>>>>>> >>>>>> Mark >>>>>>>> >>>>>> >>>>>>>> >>>>>>> John >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >> >>>>>>>> >> >>>>>>>> > >>>>>>>> > >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>> >>> >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Mar 29 11:03:11 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 29 Mar 2012 10:03:11 -0600 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> <8909A66D-5CB4-479B-ADBC-34A1133D64F4@columbia.edu> Message-ID: On Thu, Mar 29, 2012 at 09:18, John Mousel wrote: > [0]PETSC ERROR: Error in external library! > [0]PETSC ERROR: Cannot disable floating point exceptions! > Looks like something is strange with your environment because fesetenv() is returning an error. I have disabled the call if the trap mode is not changing. http://petsc.cs.iit.edu/petsc/petsc-dev/rev/352b4c19e451 -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.mousel at gmail.com Thu Mar 29 13:40:28 2012 From: john.mousel at gmail.com (John Mousel) Date: Thu, 29 Mar 2012 13:40:28 -0500 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> <8909A66D-5CB4-479B-ADBC-34A1133D64F4@columbia.edu> Message-ID: I'm attempting to solve a non-symmetric discretization of a 3D Poisson problem. The problem is singular. I've attached the results of KSPView from runs with ML and GAMG. When I run ML, I get convergence in 30 iterations. When I attempt to use the same settings with GAMG, I'm not getting convergence at all. The two things I notice are: 1. GAMG is using KSPType preonly, even though I've set it to be Richardson in my command line options. 2. ML only coarsens down to 4 rows while GAMG coarsens to 2. My problem is singular, and whenever I try to use LU, I get zero pivot problems. To mitigate this, I've been using Richardson with SOR on the coarse matrix. Could the smaller coarse grid size of GAMG be causing problems with SOR. If so, is there a way to put a lower limit on the coarse grid size? John On Thu, Mar 29, 2012 at 11:03 AM, Jed Brown wrote: > On Thu, Mar 29, 2012 at 09:18, John Mousel wrote: > >> [0]PETSC ERROR: Error in external library! >> [0]PETSC ERROR: Cannot disable floating point exceptions! >> > > Looks like something is strange with your environment because fesetenv() > is returning an error. I have disabled the call if the trap mode is not > changing. > > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/352b4c19e451 > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- [0]PCSetFromOptions_GAMG threshold set 1.000000e-02 [0]PCSetUp_GAMG level 0 N=556240, n data rows=1, n data cols=1, nnz/row (ave)=27, np=4 [0]PCGAMGFilterGraph 65.8687% nnz after filtering, with threshold 0.01, 27.0672 nnz ave. [0]maxIndSetAgg removed 0 of 556240 vertices. (0 local) 20114 selected. [0]PCGAMGProlongator_AGG New grid 20114 nodes PCGAMGOptprol_AGG smooth P0: max eigen=1.925528e+00 min=3.616018e-02 PC=jacobi [0]PCSetUp_GAMG 1) N=20114, n data cols=1, nnz/row (ave)=65, 4 active pes [0]PCGAMGFilterGraph 31.9285% nnz after filtering, with threshold 0.01, 65.3544 nnz ave. [0]maxIndSetAgg removed 0 of 20114 vertices. (0 local) 443 selected. [0]PCGAMGProlongator_AGG New grid 443 nodes PCGAMGOptprol_AGG smooth P0: max eigen=3.782372e+00 min=1.065309e-01 PC=jacobi [0]PCSetUp_GAMG 2) N=443, n data cols=1, nnz/row (ave)=113, 4 active pes [0]PCGAMGFilterGraph 11.1902% nnz after filtering, with threshold 0.01, 113.813 nnz ave. [0]maxIndSetAgg removed 0 of 443 vertices. (0 local) 20 selected. [0]PCGAMGProlongator_AGG New grid 20 nodes PCGAMGOptprol_AGG smooth P0: max eigen=2.141505e+00 min=5.132666e-03 PC=jacobi [0]createLevel aggregate processors: npe: 4 --> 1, neq=20 [0]PCSetUp_GAMG 3) N=20, n data cols=1, nnz/row (ave)=20, 1 active pes [0]PCGAMGFilterGraph 45% nnz after filtering, with threshold 0.01, 20 nnz ave. [0]maxIndSetAgg removed 0 of 20 vertices. (0 local) 2 selected. [0]PCGAMGProlongator_AGG New grid 2 nodes PCGAMGOptprol_AGG smooth P0: max eigen=1.444685e+00 min=9.769929e-05 PC=jacobi [0]PCSetUp_GAMG 4) N=2, n data cols=1, nnz/row (ave)=2, 1 active pes [0]PCSetUp_GAMG 5 levels, grid complexity = 1.09069 Residual norms for pres_ solve. 0 KSP preconditioned resid norm 1.653389026762e+05 true resid norm 1.643826511377e+01 ||r(i)||/||b|| 1.000000000000e+00 2 KSP preconditioned resid norm 3.498722572696e+04 true resid norm 1.676304035805e+01 ||r(i)||/||b|| 1.019757270128e+00 4 KSP preconditioned resid norm 3.466542979714e+04 true resid norm 1.682142736387e+01 ||r(i)||/||b|| 1.023309165988e+00 6 KSP preconditioned resid norm 3.544823492105e+04 true resid norm 2.096113365180e+01 ||r(i)||/||b|| 1.275142693388e+00 8 KSP preconditioned resid norm 3.830052737912e+04 true resid norm 4.734563796599e+01 ||r(i)||/||b|| 2.880208929489e+00 10 KSP preconditioned resid norm 4.928240285009e+04 true resid norm 1.505114395192e+02 ||r(i)||/||b|| 9.156163285936e+00 12 KSP preconditioned resid norm 2.692615589856e+04 true resid norm 1.610600319827e+02 ||r(i)||/||b|| 9.797872881841e+00 14 KSP preconditioned resid norm 3.258672961867e+04 true resid norm 3.930018988644e+01 ||r(i)||/||b|| 2.390774793717e+00 16 KSP preconditioned resid norm 5.254695737537e+04 true resid norm 3.184054936988e+02 ||r(i)||/||b|| 1.936977482083e+01 18 KSP preconditioned resid norm 4.664982713877e+04 true resid norm 2.267717910337e+02 ||r(i)||/||b|| 1.379536036584e+01 20 KSP preconditioned resid norm 3.556402200695e+04 true resid norm 3.209651885839e+01 ||r(i)||/||b|| 1.952549045550e+00 22 KSP preconditioned resid norm 3.751970755236e+04 true resid norm 6.277578808422e+01 ||r(i)||/||b|| 3.818881594241e+00 24 KSP preconditioned resid norm 3.528015422621e+04 true resid norm 2.472314669019e+01 ||r(i)||/||b|| 1.503999754176e+00 26 KSP preconditioned resid norm 3.520149611893e+04 true resid norm 2.349210422028e+01 ||r(i)||/||b|| 1.429110922454e+00 28 KSP preconditioned resid norm 3.541077679608e+04 true resid norm 2.187432271791e+01 ||r(i)||/||b|| 1.330695335944e+00 30 KSP preconditioned resid norm 3.517654266134e+04 true resid norm 2.171356997182e+01 ||r(i)||/||b|| 1.320916156391e+00 32 KSP preconditioned resid norm 3.515900089315e+04 true resid norm 2.158840469643e+01 ||r(i)||/||b|| 1.313301893297e+00 34 KSP preconditioned resid norm 3.559564844912e+04 true resid norm 2.273499576040e+01 ||r(i)||/||b|| 1.383053236035e+00 36 KSP preconditioned resid norm 3.558002879717e+04 true resid norm 2.230901057983e+01 ||r(i)||/||b|| 1.357138994013e+00 38 KSP preconditioned resid norm 3.536033027489e+04 true resid norm 2.255313539785e+01 ||r(i)||/||b|| 1.371990002702e+00 40 KSP preconditioned resid norm 3.530445288118e+04 true resid norm 2.222344798566e+01 ||r(i)||/||b|| 1.351933907371e+00 42 KSP preconditioned resid norm 3.531238381899e+04 true resid norm 2.203611003220e+01 ||r(i)||/||b|| 1.340537452078e+00 44 KSP preconditioned resid norm 3.526814846743e+04 true resid norm 2.162375692510e+01 ||r(i)||/||b|| 1.315452499120e+00 46 KSP preconditioned resid norm 3.549420786877e+04 true resid norm 2.658362282001e+01 ||r(i)||/||b|| 1.617179345632e+00 48 KSP preconditioned resid norm 3.475902657029e+04 true resid norm 6.863778415755e+01 ||r(i)||/||b|| 4.175488330582e+00 50 KSP preconditioned resid norm 3.455004242441e+04 true resid norm 8.709937353956e+01 ||r(i)||/||b|| 5.298574571997e+00 52 KSP preconditioned resid norm 3.397528825543e+04 true resid norm 1.124159495041e+02 ||r(i)||/||b|| 6.838674806984e+00 54 KSP preconditioned resid norm 3.312054456379e+04 true resid norm 1.290515134210e+02 ||r(i)||/||b|| 7.850677217321e+00 56 KSP preconditioned resid norm 3.451647821637e+04 true resid norm 2.295572351071e+01 ||r(i)||/||b|| 1.396480915221e+00 58 KSP preconditioned resid norm 3.415274707483e+04 true resid norm 1.478357632082e+02 ||r(i)||/||b|| 8.993392075441e+00 60 KSP preconditioned resid norm 3.301948214819e+04 true resid norm 1.684691863247e+02 ||r(i)||/||b|| 1.024859893418e+01 62 KSP preconditioned resid norm 3.312565770380e+04 true resid norm 1.525060310355e+02 ||r(i)||/||b|| 9.277501608592e+00 64 KSP preconditioned resid norm 3.185347314326e+04 true resid norm 1.829054885160e+02 ||r(i)||/||b|| 1.112681218183e+01 66 KSP preconditioned resid norm 3.171608219114e+04 true resid norm 1.875039031777e+02 ||r(i)||/||b|| 1.140655062319e+01 68 KSP preconditioned resid norm 3.171847298154e+04 true resid norm 1.880370116219e+02 ||r(i)||/||b|| 1.143898156652e+01 70 KSP preconditioned resid norm 3.178619322639e+04 true resid norm 1.860704776770e+02 ||r(i)||/||b|| 1.131935008890e+01 72 KSP preconditioned resid norm 3.184456847066e+04 true resid norm 1.852753590050e+02 ||r(i)||/||b|| 1.127098010177e+01 74 KSP preconditioned resid norm 3.189769214425e+04 true resid norm 1.842324138668e+02 ||r(i)||/||b|| 1.120753392111e+01 76 KSP preconditioned resid norm 3.199081602250e+04 true resid norm 1.822782258561e+02 ||r(i)||/||b|| 1.108865349199e+01 78 KSP preconditioned resid norm 3.102476484074e+04 true resid norm 2.097078712241e+02 ||r(i)||/||b|| 1.275729949436e+01 80 KSP preconditioned resid norm 3.065784272235e+04 true resid norm 2.065139379058e+02 ||r(i)||/||b|| 1.256300080796e+01 82 KSP preconditioned resid norm 2.394546049264e+04 true resid norm 4.274889221658e+02 ||r(i)||/||b|| 2.600572014182e+01 84 KSP preconditioned resid norm 5.540308447940e+04 true resid norm 2.519566526625e+02 ||r(i)||/||b|| 1.532744793436e+01 86 KSP preconditioned resid norm 8.474143362887e+05 true resid norm 1.261372272928e+04 ||r(i)||/||b|| 7.673390495884e+02 88 KSP preconditioned resid norm 1.848633122514e+06 true resid norm 2.777035833045e+04 ||r(i)||/||b|| 1.689372822391e+03 90 KSP preconditioned resid norm 7.565704677060e+05 true resid norm 1.143785512560e+04 ||r(i)||/||b|| 6.958067074863e+02 92 KSP preconditioned resid norm 6.757814515708e+05 true resid norm 1.024954173515e+04 ||r(i)||/||b|| 6.235172424959e+02 94 KSP preconditioned resid norm 6.614703098041e+05 true resid norm 1.004656691097e+04 ||r(i)||/||b|| 6.111695389652e+02 96 KSP preconditioned resid norm 2.934757771515e+05 true resid norm 4.802602009791e+03 ||r(i)||/||b|| 2.921599071771e+02 98 KSP preconditioned resid norm 2.585058956777e+05 true resid norm 4.004734942226e+03 ||r(i)||/||b|| 2.436227250570e+02 100 KSP preconditioned resid norm 2.309935778893e+05 true resid norm 4.002669905640e+03 ||r(i)||/||b|| 2.434971013022e+02 102 KSP preconditioned resid norm 2.015988421948e+05 true resid norm 4.016332928729e+03 ||r(i)||/||b|| 2.443282731439e+02 104 KSP preconditioned resid norm 2.045801533779e+05 true resid norm 4.000349066110e+03 ||r(i)||/||b|| 2.433559161154e+02 106 KSP preconditioned resid norm 2.264469374791e+05 true resid norm 3.893162890422e+03 ||r(i)||/||b|| 2.368353876445e+02 108 KSP preconditioned resid norm 3.080238001769e+05 true resid norm 3.906970507584e+03 ||r(i)||/||b|| 2.376753556743e+02 110 KSP preconditioned resid norm 2.793144888698e+05 true resid norm 4.313926252234e+03 ||r(i)||/||b|| 2.624319672652e+02 112 KSP preconditioned resid norm 2.403158281465e+05 true resid norm 5.534097030480e+03 ||r(i)||/||b|| 3.366594340813e+02 114 KSP preconditioned resid norm 1.045230913669e+06 true resid norm 2.054226795354e+04 ||r(i)||/||b|| 1.249661555606e+03 116 KSP preconditioned resid norm 1.059410089666e+06 true resid norm 2.765007218808e+04 ||r(i)||/||b|| 1.682055374865e+03 118 KSP preconditioned resid norm 7.620229766528e+05 true resid norm 2.851741985732e+04 ||r(i)||/||b|| 1.734819316999e+03 Linear solve did not converge due to DIVERGED_ITS iterations 118 KSP Object:(pres_) 4 MPI processes type: bcgsl BCGSL: Ell = 2 BCGSL: Delta = 0 maximum iterations=120 tolerances: relative=1e-08, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object:(pres_) 4 MPI processes type: gamg MG: type is MULTIPLICATIVE, levels=5 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (pres_mg_coarse_) 4 MPI processes type: preonly maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (pres_mg_coarse_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 8, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=2, cols=2 total: nonzeros=4, allocated nonzeros=4 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 1 nodes, limit used is 5 Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (pres_mg_levels_1_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_1_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=20, cols=20 total: nonzeros=400, allocated nonzeros=400 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 4 nodes, limit used is 5 Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (pres_mg_levels_2_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_2_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=443, cols=443 total: nonzeros=50419, allocated nonzeros=50419 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (pres_mg_levels_3_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_3_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=20114, cols=20114 total: nonzeros=1314538, allocated nonzeros=1314538 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 4 ------------------------------- KSP Object: (pres_mg_levels_4_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_4_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=556240, cols=556240 total: nonzeros=15055870, allocated nonzeros=28055391 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=556240, cols=556240 total: nonzeros=15055870, allocated nonzeros=28055391 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines -------------- next part -------------- Residual norms for pres_ solve. 0 KSP preconditioned resid norm 2.993219493794e+03 true resid norm 1.643826511377e+01 ||r(i)||/||b|| 1.000000000000e+00 2 KSP preconditioned resid norm 4.012627047075e+03 true resid norm 2.722662316913e+02 ||r(i)||/||b|| 1.656295416864e+01 4 KSP preconditioned resid norm 7.152372463858e+02 true resid norm 3.245016507811e+01 ||r(i)||/||b|| 1.974062643078e+00 6 KSP preconditioned resid norm 2.615419780458e+02 true resid norm 2.745979195842e+01 ||r(i)||/||b|| 1.670479930112e+00 8 KSP preconditioned resid norm 1.626445578620e+02 true resid norm 8.455143190817e+00 ||r(i)||/||b|| 5.143573930885e-01 10 KSP preconditioned resid norm 3.412604554263e+01 true resid norm 6.517443353179e-01 ||r(i)||/||b|| 3.964800000530e-02 12 KSP preconditioned resid norm 4.448092876396e+01 true resid norm 2.990775751163e+00 ||r(i)||/||b|| 1.819398659448e-01 14 KSP preconditioned resid norm 1.003757428948e+00 true resid norm 2.495627994699e-01 ||r(i)||/||b|| 1.518182105853e-02 16 KSP preconditioned resid norm 6.831960905152e-01 true resid norm 2.110987118511e-02 ||r(i)||/||b|| 1.284190943449e-03 18 KSP preconditioned resid norm 8.821616336254e-02 true resid norm 1.421049860455e-02 ||r(i)||/||b|| 8.644767867046e-04 20 KSP preconditioned resid norm 2.018914933890e-02 true resid norm 1.749749201962e-03 ||r(i)||/||b|| 1.064436660349e-04 22 KSP preconditioned resid norm 2.149723645607e-02 true resid norm 7.137329046115e-04 ||r(i)||/||b|| 4.341899219119e-05 24 KSP preconditioned resid norm 1.085565650332e-03 true resid norm 9.250592958305e-05 ||r(i)||/||b|| 5.627475219728e-06 26 KSP preconditioned resid norm 5.850921166024e-04 true resid norm 3.554584676434e-05 ||r(i)||/||b|| 2.162384322088e-06 28 KSP preconditioned resid norm 2.636412182917e-04 true resid norm 8.740872361198e-06 ||r(i)||/||b|| 5.317393472305e-07 30 KSP preconditioned resid norm 8.051486995446e-06 true resid norm 5.329776180424e-06 ||r(i)||/||b|| 3.242298468565e-07 KSP Object:(pres_) 4 MPI processes type: bcgsl BCGSL: Ell = 2 BCGSL: Delta = 0 maximum iterations=5000 tolerances: relative=1e-08, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object:(pres_) 4 MPI processes type: ml MG: type is MULTIPLICATIVE, levels=6 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (pres_mg_coarse_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_coarse_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 8, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=4, cols=4 total: nonzeros=16, allocated nonzeros=16 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (pres_mg_levels_1_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_1_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=23, cols=23 total: nonzeros=481, allocated nonzeros=481 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 2 nodes, limit used is 5 Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (pres_mg_levels_2_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_2_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=204, cols=204 total: nonzeros=19040, allocated nonzeros=19040 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 37 nodes, limit used is 5 Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (pres_mg_levels_3_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_3_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=2704, cols=2704 total: nonzeros=277205, allocated nonzeros=277205 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 4 ------------------------------- KSP Object: (pres_mg_levels_4_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_4_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=52276, cols=52276 total: nonzeros=5628023, allocated nonzeros=5628023 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 5 ------------------------------- KSP Object: (pres_mg_levels_5_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_5_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=556240, cols=556240 total: nonzeros=15055870, allocated nonzeros=28055391 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=556240, cols=556240 total: nonzeros=15055870, allocated nonzeros=28055391 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines From andrew.spott at gmail.com Thu Mar 29 20:09:16 2012 From: andrew.spott at gmail.com (Andrew Spott) Date: Thu, 29 Mar 2012 19:09:16 -0600 Subject: [petsc-users] TSSetMatrices structure? Message-ID: In the Petsc manual, there is a TSSetMatrices function, however there is no further documentation for it. It is referenced in the manual as: TSSetMatrices(TS ts, Mat A,PetscErrorCode (*frhs)(TS,PetscReal,Mat*,Mat*,MatStructure*,void*), Mat B,PetscErrorCode (*flhs)(TS,PetscReal,Mat*,Mat*,MatStructure*,void*), MatStructure flag,void *ctx) However, what is passed to the function pointers isn't mentioned anywhere, and it isn't in the online documentation. Also, I assume that "MatStructure flag" tells if the structure of A and B are the same or different, if B is PETSC_NULL, is flag "SAME_NONZERO_PATERN" or is it different? (B can be considered to not exist, or to exist as an identity matrix, hence the confusion). How does the ctx work? Is the same context passed to frhs each time? Thanks for the help. -Andrew From bsmith at mcs.anl.gov Thu Mar 29 20:37:51 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 29 Mar 2012 20:37:51 -0500 Subject: [petsc-users] TSSetMatrices structure? In-Reply-To: References: Message-ID: Andrew, These are outdated manual pages; you'll want to avoid this. Jed and Emil have done a major update of the TS interface and solvers, much more powerful and less confusing. Likely you'll want to use TSSetIFunction() and TSSetIJacobian() http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/TS/index.html and work with petsc-dev http://www.mcs.anl.gov/petsc/developers/index.html Barry On Mar 29, 2012, at 8:09 PM, Andrew Spott wrote: > In the Petsc manual, there is a TSSetMatrices function, however there is no further documentation for it. > > It is referenced in the manual as: > > TSSetMatrices(TS ts, > Mat A,PetscErrorCode (*frhs)(TS,PetscReal,Mat*,Mat*,MatStructure*,void*), Mat B,PetscErrorCode (*flhs)(TS,PetscReal,Mat*,Mat*,MatStructure*,void*), MatStructure flag,void *ctx) > > However, what is passed to the function pointers isn't mentioned anywhere, and it isn't in the online documentation. > > Also, I assume that "MatStructure flag" tells if the structure of A and B are the same or different, if B is PETSC_NULL, is flag "SAME_NONZERO_PATERN" or is it different? (B can be considered to not exist, or to exist as an identity matrix, hence the confusion). > > How does the ctx work? Is the same context passed to frhs each time? > > Thanks for the help. > > -Andrew From andrew.spott at gmail.com Thu Mar 29 20:58:16 2012 From: andrew.spott at gmail.com (Andrew Spott) Date: Thu, 29 Mar 2012 19:58:16 -0600 Subject: [petsc-users] TSSetMatrices structure? In-Reply-To: References: Message-ID: This: http://www.mcs.anl.gov/petsc/petsc-3.2/src/ts/examples/tutorials/ex4.c.html Looks like what I'm interested in doing, though I would like to use the latest. When is Petsc 3.3 (or 3.4, I'm not sure how your versioning works) coming out? Is the dev version stable enough (in API and in execution) to be used? Until then, does the 3.2 version of the TS interface work? (the link above, and the corresponding one for petsc-dev don't appear to have much changed between them). Thanks for the prompt reply, -Andrew On Mar 29, 2012, at 7:37 PM, Barry Smith wrote: > > Andrew, > > These are outdated manual pages; you'll want to avoid this. > > Jed and Emil have done a major update of the TS interface and solvers, much more powerful and less confusing. > > Likely you'll want to use TSSetIFunction() and TSSetIJacobian() http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/TS/index.html and work with petsc-dev http://www.mcs.anl.gov/petsc/developers/index.html > > > > Barry > > On Mar 29, 2012, at 8:09 PM, Andrew Spott wrote: > >> In the Petsc manual, there is a TSSetMatrices function, however there is no further documentation for it. >> >> It is referenced in the manual as: >> >> TSSetMatrices(TS ts, >> Mat A,PetscErrorCode (*frhs)(TS,PetscReal,Mat*,Mat*,MatStructure*,void*), Mat B,PetscErrorCode (*flhs)(TS,PetscReal,Mat*,Mat*,MatStructure*,void*), MatStructure flag,void *ctx) >> >> However, what is passed to the function pointers isn't mentioned anywhere, and it isn't in the online documentation. >> >> Also, I assume that "MatStructure flag" tells if the structure of A and B are the same or different, if B is PETSC_NULL, is flag "SAME_NONZERO_PATERN" or is it different? (B can be considered to not exist, or to exist as an identity matrix, hence the confusion). >> >> How does the ctx work? Is the same context passed to frhs each time? >> >> Thanks for the help. >> >> -Andrew > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Mar 29 21:07:48 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 29 Mar 2012 21:07:48 -0500 Subject: [petsc-users] TSSetMatrices structure? In-Reply-To: References: Message-ID: On Mar 29, 2012, at 8:58 PM, Andrew Spott wrote: > This: http://www.mcs.anl.gov/petsc/petsc-3.2/src/ts/examples/tutorials/ex4.c.html Looks like what I'm interested in doing, though I would like to use the latest. With this simple a situation using the current version or switching to dev is both fine. The changes you would have to make are small > When is Petsc 3.3 (or 3.4, I'm not sure how your versioning works) coming out? Is the dev version stable enough (in API and in execution) to be used? The dev version is always stable enough to use :-). At this point yes because we are in the testing phase for the next release. > > Until then, does the 3.2 version of the TS interface work? (the link above, and the corresponding one for petsc-dev don't appear to have much changed between them). Yes. Barry Note that in 3.2 if you have a mass matrix (for example M U_t = A U_xx)how to formulate the problem is different between 3.2 and dev so I would recommend working immediately with dev. But for simple U_t = something, 3.2 is fine and then you can change it slightly for the next release. > > Thanks for the prompt reply, > > -Andrew > > On Mar 29, 2012, at 7:37 PM, Barry Smith wrote: > >> >> Andrew, >> >> These are outdated manual pages; you'll want to avoid this. >> >> Jed and Emil have done a major update of the TS interface and solvers, much more powerful and less confusing. >> >> Likely you'll want to use TSSetIFunction() and TSSetIJacobian() http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/TS/index.html and work with petsc-dev http://www.mcs.anl.gov/petsc/developers/index.html >> >> >> >> Barry >> >> On Mar 29, 2012, at 8:09 PM, Andrew Spott wrote: >> >>> In the Petsc manual, there is a TSSetMatrices function, however there is no further documentation for it. >>> >>> It is referenced in the manual as: >>> >>> TSSetMatrices(TS ts, >>> Mat A,PetscErrorCode (*frhs)(TS,PetscReal,Mat*,Mat*,MatStructure*,void*), Mat B,PetscErrorCode (*flhs)(TS,PetscReal,Mat*,Mat*,MatStructure*,void*), MatStructure flag,void *ctx) >>> >>> However, what is passed to the function pointers isn't mentioned anywhere, and it isn't in the online documentation. >>> >>> Also, I assume that "MatStructure flag" tells if the structure of A and B are the same or different, if B is PETSC_NULL, is flag "SAME_NONZERO_PATERN" or is it different? (B can be considered to not exist, or to exist as an identity matrix, hence the confusion). >>> >>> How does the ctx work? Is the same context passed to frhs each time? >>> >>> Thanks for the help. >>> >>> -Andrew >> > From andrew.spott at gmail.com Thu Mar 29 21:09:51 2012 From: andrew.spott at gmail.com (Andrew Spott) Date: Thu, 29 Mar 2012 20:09:51 -0600 Subject: [petsc-users] TSSetMatrices structure? In-Reply-To: References: Message-ID: Thanks again for the quick reply! Thankfully I don't have a mass matrix, I'm just solving the schrodinger equation. Thanks again. -Andrew On Mar 29, 2012, at 8:07 PM, Barry Smith wrote: > > On Mar 29, 2012, at 8:58 PM, Andrew Spott wrote: > >> This: http://www.mcs.anl.gov/petsc/petsc-3.2/src/ts/examples/tutorials/ex4.c.html Looks like what I'm interested in doing, though I would like to use the latest. > > With this simple a situation using the current version or switching to dev is both fine. The changes you would have to make are small > >> When is Petsc 3.3 (or 3.4, I'm not sure how your versioning works) coming out? Is the dev version stable enough (in API and in execution) to be used? > > The dev version is always stable enough to use :-). At this point yes because we are in the testing phase for the next release. > >> >> Until then, does the 3.2 version of the TS interface work? (the link above, and the corresponding one for petsc-dev don't appear to have much changed between them). > > Yes. > > Barry > > Note that in 3.2 if you have a mass matrix (for example M U_t = A U_xx)how to formulate the problem is different between 3.2 and dev so I would recommend working immediately with dev. But for simple U_t = something, 3.2 is fine and then you can change it slightly for the next release. > >> >> Thanks for the prompt reply, >> >> -Andrew >> >> On Mar 29, 2012, at 7:37 PM, Barry Smith wrote: >> >>> >>> Andrew, >>> >>> These are outdated manual pages; you'll want to avoid this. >>> >>> Jed and Emil have done a major update of the TS interface and solvers, much more powerful and less confusing. >>> >>> Likely you'll want to use TSSetIFunction() and TSSetIJacobian() http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/TS/index.html and work with petsc-dev http://www.mcs.anl.gov/petsc/developers/index.html >>> >>> >>> >>> Barry >>> >>> On Mar 29, 2012, at 8:09 PM, Andrew Spott wrote: >>> >>>> In the Petsc manual, there is a TSSetMatrices function, however there is no further documentation for it. >>>> >>>> It is referenced in the manual as: >>>> >>>> TSSetMatrices(TS ts, >>>> Mat A,PetscErrorCode (*frhs)(TS,PetscReal,Mat*,Mat*,MatStructure*,void*), Mat B,PetscErrorCode (*flhs)(TS,PetscReal,Mat*,Mat*,MatStructure*,void*), MatStructure flag,void *ctx) >>>> >>>> However, what is passed to the function pointers isn't mentioned anywhere, and it isn't in the online documentation. >>>> >>>> Also, I assume that "MatStructure flag" tells if the structure of A and B are the same or different, if B is PETSC_NULL, is flag "SAME_NONZERO_PATERN" or is it different? (B can be considered to not exist, or to exist as an identity matrix, hence the confusion). >>>> >>>> How does the ctx work? Is the same context passed to frhs each time? >>>> >>>> Thanks for the help. >>>> >>>> -Andrew >>> >> > From bsmith at mcs.anl.gov Thu Mar 29 21:13:08 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 29 Mar 2012 21:13:08 -0500 Subject: [petsc-users] TSSetMatrices structure? In-Reply-To: References: Message-ID: Thats the first time I've seen __"just"__ solving the schrodinger equation :-) Barry On Mar 29, 2012, at 9:09 PM, Andrew Spott wrote: > Thanks again for the quick reply! Thankfully I don't have a mass matrix, I'm just solving the schrodinger equation. > > Thanks again. > > -Andrew > > On Mar 29, 2012, at 8:07 PM, Barry Smith wrote: > >> >> On Mar 29, 2012, at 8:58 PM, Andrew Spott wrote: >> >>> This: http://www.mcs.anl.gov/petsc/petsc-3.2/src/ts/examples/tutorials/ex4.c.html Looks like what I'm interested in doing, though I would like to use the latest. >> >> With this simple a situation using the current version or switching to dev is both fine. The changes you would have to make are small >> >>> When is Petsc 3.3 (or 3.4, I'm not sure how your versioning works) coming out? Is the dev version stable enough (in API and in execution) to be used? >> >> The dev version is always stable enough to use :-). At this point yes because we are in the testing phase for the next release. >> >>> >>> Until then, does the 3.2 version of the TS interface work? (the link above, and the corresponding one for petsc-dev don't appear to have much changed between them). >> >> Yes. >> >> Barry >> >> Note that in 3.2 if you have a mass matrix (for example M U_t = A U_xx)how to formulate the problem is different between 3.2 and dev so I would recommend working immediately with dev. But for simple U_t = something, 3.2 is fine and then you can change it slightly for the next release. >> >>> >>> Thanks for the prompt reply, >>> >>> -Andrew >>> >>> On Mar 29, 2012, at 7:37 PM, Barry Smith wrote: >>> >>>> >>>> Andrew, >>>> >>>> These are outdated manual pages; you'll want to avoid this. >>>> >>>> Jed and Emil have done a major update of the TS interface and solvers, much more powerful and less confusing. >>>> >>>> Likely you'll want to use TSSetIFunction() and TSSetIJacobian() http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/TS/index.html and work with petsc-dev http://www.mcs.anl.gov/petsc/developers/index.html >>>> >>>> >>>> >>>> Barry >>>> >>>> On Mar 29, 2012, at 8:09 PM, Andrew Spott wrote: >>>> >>>>> In the Petsc manual, there is a TSSetMatrices function, however there is no further documentation for it. >>>>> >>>>> It is referenced in the manual as: >>>>> >>>>> TSSetMatrices(TS ts, >>>>> Mat A,PetscErrorCode (*frhs)(TS,PetscReal,Mat*,Mat*,MatStructure*,void*), Mat B,PetscErrorCode (*flhs)(TS,PetscReal,Mat*,Mat*,MatStructure*,void*), MatStructure flag,void *ctx) >>>>> >>>>> However, what is passed to the function pointers isn't mentioned anywhere, and it isn't in the online documentation. >>>>> >>>>> Also, I assume that "MatStructure flag" tells if the structure of A and B are the same or different, if B is PETSC_NULL, is flag "SAME_NONZERO_PATERN" or is it different? (B can be considered to not exist, or to exist as an identity matrix, hence the confusion). >>>>> >>>>> How does the ctx work? Is the same context passed to frhs each time? >>>>> >>>>> Thanks for the help. >>>>> >>>>> -Andrew >>>> >>> >> > From knepley at gmail.com Thu Mar 29 21:40:34 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 29 Mar 2012 21:40:34 -0500 Subject: [petsc-users] TSSetMatrices structure? In-Reply-To: References: Message-ID: On Thu, Mar 29, 2012 at 9:09 PM, Andrew Spott wrote: > Thanks again for the quick reply! Thankfully I don't have a mass matrix, > I'm just solving the schrodinger equation. > Depending on how you discretize, you can have a mass matrix for Schrodinger. Matt > Thanks again. > > -Andrew > > On Mar 29, 2012, at 8:07 PM, Barry Smith wrote: > > > > > On Mar 29, 2012, at 8:58 PM, Andrew Spott wrote: > > > >> This: > http://www.mcs.anl.gov/petsc/petsc-3.2/src/ts/examples/tutorials/ex4.c.html Looks like what I'm interested in doing, though I would like to use the > latest. > > > > With this simple a situation using the current version or switching to > dev is both fine. The changes you would have to make are small > > > >> When is Petsc 3.3 (or 3.4, I'm not sure how your versioning works) > coming out? Is the dev version stable enough (in API and in execution) to > be used? > > > > The dev version is always stable enough to use :-). At this point yes > because we are in the testing phase for the next release. > > > >> > >> Until then, does the 3.2 version of the TS interface work? (the link > above, and the corresponding one for petsc-dev don't appear to have much > changed between them). > > > > Yes. > > > > Barry > > > > Note that in 3.2 if you have a mass matrix (for example M U_t = A > U_xx)how to formulate the problem is different between 3.2 and dev so I > would recommend working immediately with dev. But for simple U_t = > something, 3.2 is fine and then you can change it slightly for the next > release. > > > >> > >> Thanks for the prompt reply, > >> > >> -Andrew > >> > >> On Mar 29, 2012, at 7:37 PM, Barry Smith wrote: > >> > >>> > >>> Andrew, > >>> > >>> These are outdated manual pages; you'll want to avoid this. > >>> > >>> Jed and Emil have done a major update of the TS interface and > solvers, much more powerful and less confusing. > >>> > >>> Likely you'll want to use TSSetIFunction() and TSSetIJacobian() > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/TS/index.html and > work with petsc-dev http://www.mcs.anl.gov/petsc/developers/index.html > >>> > >>> > >>> > >>> Barry > >>> > >>> On Mar 29, 2012, at 8:09 PM, Andrew Spott wrote: > >>> > >>>> In the Petsc manual, there is a TSSetMatrices function, however there > is no further documentation for it. > >>>> > >>>> It is referenced in the manual as: > >>>> > >>>> TSSetMatrices(TS ts, > >>>> Mat A,PetscErrorCode > (*frhs)(TS,PetscReal,Mat*,Mat*,MatStructure*,void*), Mat B,PetscErrorCode > (*flhs)(TS,PetscReal,Mat*,Mat*,MatStructure*,void*), MatStructure flag,void > *ctx) > >>>> > >>>> However, what is passed to the function pointers isn't mentioned > anywhere, and it isn't in the online documentation. > >>>> > >>>> Also, I assume that "MatStructure flag" tells if the structure of A > and B are the same or different, if B is PETSC_NULL, is flag > "SAME_NONZERO_PATERN" or is it different? (B can be considered to not > exist, or to exist as an identity matrix, hence the confusion). > >>>> > >>>> How does the ctx work? Is the same context passed to frhs each time? > >>>> > >>>> Thanks for the help. > >>>> > >>>> -Andrew > >>> > >> > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Fri Mar 30 08:42:18 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Fri, 30 Mar 2012 09:42:18 -0400 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> <8909A66D-5CB4-479B-ADBC-34A1133D64F4@columbia.edu> Message-ID: On Mar 29, 2012, at 2:40 PM, John Mousel wrote: > I'm attempting to solve a non-symmetric discretization of a 3D Poisson problem. The problem is singular. I've attached the results of KSPView from runs with ML and GAMG. When I run ML, I get convergence in 30 iterations. When I attempt to use the same settings with GAMG, I'm not getting convergence at all. The two things I notice are: > > 1. GAMG is using KSPType preonly, even though I've set it to be Richardson in my command line options. PETSc seems to switch the coarse grid solver to GMRES in Setup. This seems to be a bug and I unwisely decide to override this manually. I will undo this in the next checkin. This should not be the problem however. > 2. ML only coarsens down to 4 rows while GAMG coarsens to 2. My problem is singular, and whenever I try to use LU, I get zero pivot problems. To mitigate this, I've been using Richardson with SOR on the coarse matrix. Could the smaller coarse grid size of GAMG be causing problems with SOR. If so, is there a way to put a lower limit on the coarse grid size? > I'm thinking that with a 2x2 coarse grid 8 iterations of SOR is picking up the null space. Maybe try just one SOR iteration on the coarse grid. Also, can you run with options_left so that I can see your arguments. One known bug is the mat_diagaonal_scale breaks GAMG, but it should also break ML. Mark > John > > On Thu, Mar 29, 2012 at 11:03 AM, Jed Brown wrote: > On Thu, Mar 29, 2012 at 09:18, John Mousel wrote: > [0]PETSC ERROR: Error in external library! > [0]PETSC ERROR: Cannot disable floating point exceptions! > > Looks like something is strange with your environment because fesetenv() is returning an error. I have disabled the call if the trap mode is not changing. > > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/352b4c19e451 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.mousel at gmail.com Fri Mar 30 09:52:45 2012 From: john.mousel at gmail.com (John Mousel) Date: Fri, 30 Mar 2012 09:52:45 -0500 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> <8909A66D-5CB4-479B-ADBC-34A1133D64F4@columbia.edu> Message-ID: Mark, I've run GAMG twice with different coarse grid sizes of 2 and 8 with 1 sweep of SOR on the coarse grid. For a size of 8 it converges nicely, but for a size of 2, I think the null space is causing too many problems. If GAMG were to coarsen to a size of 1, then there would be no hope because only the null space would remain, right? This doesn't ever seem to occur with ML because there are at least as many rows as processors. John On Fri, Mar 30, 2012 at 8:42 AM, Mark F. Adams wrote: > > On Mar 29, 2012, at 2:40 PM, John Mousel wrote: > > I'm attempting to solve a non-symmetric discretization of a 3D Poisson > problem. The problem is singular. I've attached the results of KSPView from > runs with ML and GAMG. When I run ML, I get convergence in 30 iterations. > When I attempt to use the same settings with GAMG, I'm not getting > convergence at all. The two things I notice are: > > 1. GAMG is using KSPType preonly, even though I've set it to be Richardson > in my command line options. > > > PETSc seems to switch the coarse grid solver to GMRES in Setup. This > seems to be a bug and I unwisely decide to override this manually. I will > undo this in the next checkin. This should not be the problem however. > > 2. ML only coarsens down to 4 rows while GAMG coarsens to 2. My problem is > singular, and whenever I try to use LU, I get zero pivot problems. To > mitigate this, I've been using Richardson with SOR on the coarse matrix. > Could the smaller coarse grid size of GAMG be causing problems with SOR. If > so, is there a way to put a lower limit on the coarse grid size? > > > I'm thinking that with a 2x2 coarse grid 8 iterations of SOR is picking up > the null space. Maybe try just one SOR iteration on the coarse grid. > > Also, can you run with options_left so that I can see your arguments. One > known bug is the mat_diagaonal_scale breaks GAMG, but it should also break > ML. > > Mark > > John > > On Thu, Mar 29, 2012 at 11:03 AM, Jed Brown wrote: > >> On Thu, Mar 29, 2012 at 09:18, John Mousel wrote: >> >>> [0]PETSC ERROR: Error in external library! >>> [0]PETSC ERROR: Cannot disable floating point exceptions! >>> >> >> Looks like something is strange with your environment because fesetenv() >> is returning an error. I have disabled the call if the trap mode is not >> changing. >> >> http://petsc.cs.iit.edu/petsc/petsc-dev/rev/352b4c19e451 >> > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- Residual norms for pres_ solve. 0 KSP preconditioned resid norm 5.016797113851e+03 true resid norm 1.537048121012e+01 ||r(i)||/||b|| 1.000000000000e+00 2 KSP preconditioned resid norm 2.593195031348e+03 true resid norm 1.577483110619e+02 ||r(i)||/||b|| 1.026306911966e+01 4 KSP preconditioned resid norm 4.642597221361e+02 true resid norm 3.165516339207e+01 ||r(i)||/||b|| 2.059477706608e+00 6 KSP preconditioned resid norm 1.121148735724e+02 true resid norm 1.634561891940e+01 ||r(i)||/||b|| 1.063442236840e+00 8 KSP preconditioned resid norm 4.092437868938e+02 true resid norm 4.290066631217e+01 ||r(i)||/||b|| 2.791107560375e+00 10 KSP preconditioned resid norm 2.364490073524e+01 true resid norm 9.278095018184e-01 ||r(i)||/||b|| 6.036307446299e-02 12 KSP preconditioned resid norm 1.118938046010e+01 true resid norm 8.235420783716e-01 ||r(i)||/||b|| 5.357945968727e-02 14 KSP preconditioned resid norm 8.827495203651e-01 true resid norm 1.253023900141e-02 ||r(i)||/||b|| 8.152144900421e-04 16 KSP preconditioned resid norm 5.786122993849e-02 true resid norm 2.785770732954e-03 ||r(i)||/||b|| 1.812416081755e-04 18 KSP preconditioned resid norm 1.482851639037e-02 true resid norm 7.572356321341e-04 ||r(i)||/||b|| 4.926557742613e-05 20 KSP preconditioned resid norm 2.833429772564e-02 true resid norm 4.152766547014e-04 ||r(i)||/||b|| 2.701780438911e-05 22 KSP preconditioned resid norm 2.727719252108e-03 true resid norm 1.753161236491e-04 ||r(i)||/||b|| 1.140602699762e-05 24 KSP preconditioned resid norm 5.417522429762e-05 true resid norm 1.308360805994e-05 ||r(i)||/||b|| 8.512165547116e-07 26 KSP preconditioned resid norm 1.274290782774e-05 true resid norm 1.317046021128e-05 ||r(i)||/||b|| 8.568671358587e-07 KSP Object:(pres_) 4 MPI processes type: bcgsl BCGSL: Ell = 2 BCGSL: Delta = 0 maximum iterations=1000 tolerances: relative=1e-08, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object:(pres_) 4 MPI processes type: ml MG: type is MULTIPLICATIVE, levels=6 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (pres_mg_coarse_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_coarse_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 8, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=4, cols=4 total: nonzeros=16, allocated nonzeros=16 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (pres_mg_levels_1_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_1_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=7, cols=7 total: nonzeros=39, allocated nonzeros=39 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 1 nodes, limit used is 5 Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (pres_mg_levels_2_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_2_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=78, cols=78 total: nonzeros=2914, allocated nonzeros=2914 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 4 nodes, limit used is 5 Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (pres_mg_levels_3_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_3_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=889, cols=889 total: nonzeros=75418, allocated nonzeros=75418 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 4 ------------------------------- KSP Object: (pres_mg_levels_4_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_4_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=16562, cols=16562 total: nonzeros=1527407, allocated nonzeros=1527407 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 5 ------------------------------- KSP Object: (pres_mg_levels_5_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_5_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=178085, cols=178085 total: nonzeros=4676387, allocated nonzeros=8374105 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=178085, cols=178085 total: nonzeros=4676387, allocated nonzeros=8374105 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-pres_options_left no value Options: -pres_ksp_type bcgsl -pres_pc_type ml -pres_pc_ml_Threshold .01 -pres_mg_coarse_ksp_type richardson -pres_mg_coarse_pc_type sor -pres_mg_coarse_pc_sor_its 8 -pres_ksp_monitor_true_residual -pres_ksp_view -pres_options_left -------------- next part -------------- RUN 1: Options: -pres_ksp_type bcgsl -pres_pc_type gamg -pres_pc_gamg_sym_graph -pres_pc_gamg_coarse_eq_limit 2 -pres_pc_gamg_agg_nsmooths 1 -pres_pc_gamg_threshold 0.01 -pres_mg_levels_ksp_type richardson -pres_mg_levels_pc_type sor -pres_mg_coarse_ksp_type richardson -pres_mg_coarse_pc_type sor -pres_mg_coarse_pc_sor_its 1 -pres_ksp_monitor_true_residual -pres_pc_gamg_verbose 2 -pres_ksp_view -pres_options_left [0]PCSetFromOptions_GAMG threshold set 1.000000e-02 [0]PCSetUp_GAMG level 0 N=178085, n data rows=1, n data cols=1, nnz/row (ave)=26, np=4 [0]PCGAMGFilterGraph 64.0278% nnz after filtering, with threshold 0.01, 26.2593 nnz ave. [0]maxIndSetAgg removed 0 of 178085 vertices. (0 local) 6937 selected. [0]PCGAMGProlongator_AGG New grid 6937 nodes PCGAMGOptprol_AGG smooth P0: max eigen=1.927344e+00 min=4.441888e-02 PC=jacobi [0]PCSetUp_GAMG 1) N=6937, n data cols=1, nnz/row (ave)=63, 4 active pes [0]PCGAMGFilterGraph 32.3853% nnz after filtering, with threshold 0.01, 63.4622 nnz ave. [0]maxIndSetAgg removed 0 of 6937 vertices. (0 local) 167 selected. [0]PCGAMGProlongator_AGG New grid 167 nodes PCGAMGOptprol_AGG smooth P0: max eigen=4.364217e+00 min=5.712156e-03 PC=jacobi [0]createLevel aggregate processors: npe: 4 --> 2, neq=167 [0]PCSetUp_GAMG 2) N=167, n data cols=1, nnz/row (ave)=58, 2 active pes [0]PCGAMGFilterGraph 19.7613% nnz after filtering, with threshold 0.01, 58.2096 nnz ave. [0]maxIndSetAgg removed 0 of 167 vertices. (0 local) 8 selected. [0]PCGAMGProlongator_AGG New grid 8 nodes PCGAMGOptprol_AGG smooth P0: max eigen=1.459581e+00 min=1.292445e-03 PC=jacobi [0]createLevel aggregate processors: npe: 2 --> 1, neq=8 [0]PCSetUp_GAMG 3) N=8, n data cols=1, nnz/row (ave)=8, 1 active pes [0]PCGAMGFilterGraph 56.25% nnz after filtering, with threshold 0.01, 8 nnz ave. [0]maxIndSetAgg removed 0 of 8 vertices. (0 local) 2 selected. [0]PCGAMGProlongator_AGG New grid 2 nodes PCGAMGOptprol_AGG smooth P0: max eigen=2.426409e+00 min=3.017187e-14 PC=jacobi [0]PCSetUp_GAMG 4) N=2, n data cols=1, nnz/row (ave)=2, 1 active pes [0]PCSetUp_GAMG 5 levels, grid complexity = 1.09623 Residual norms for pres_ solve. 0 KSP preconditioned resid norm 3.214004976835e+04 true resid norm 1.537048121012e+01 ||r(i)||/||b|| 1.000000000000e+00 2 KSP preconditioned resid norm 9.962575667075e+03 true resid norm 1.666562709330e+01 ||r(i)||/||b|| 1.084261895609e+00 4 KSP preconditioned resid norm 9.964216763945e+03 true resid norm 1.668511403642e+01 ||r(i)||/||b|| 1.085529711681e+00 6 KSP preconditioned resid norm 1.009624086413e+04 true resid norm 1.899567590778e+01 ||r(i)||/||b|| 1.235854339764e+00 8 KSP preconditioned resid norm 1.119961784740e+04 true resid norm 4.344299871311e+01 ||r(i)||/||b|| 2.826391582620e+00 10 KSP preconditioned resid norm 3.270474238322e+04 true resid norm 5.485695179461e+02 ||r(i)||/||b|| 3.568980765449e+01 12 KSP preconditioned resid norm 6.862047100670e+04 true resid norm 1.392424526518e+03 ||r(i)||/||b|| 9.059082194521e+01 14 KSP preconditioned resid norm 8.020735170110e+04 true resid norm 1.708524744698e+03 ||r(i)||/||b|| 1.111562300062e+02 16 KSP preconditioned resid norm 7.957648213223e+04 true resid norm 1.697241325110e+03 ||r(i)||/||b|| 1.104221333027e+02 18 KSP preconditioned resid norm 7.987861582440e+04 true resid norm 1.720588721880e+03 ||r(i)||/||b|| 1.119411096087e+02 20 KSP preconditioned resid norm 7.781283925015e+04 true resid norm 1.570743019621e+03 ||r(i)||/||b|| 1.021921824144e+02 22 KSP preconditioned resid norm 7.613701956079e+04 true resid norm 1.437142018083e+03 ||r(i)||/||b|| 9.350013174194e+01 24 KSP preconditioned resid norm 7.441069686260e+04 true resid norm 1.291829388291e+03 ||r(i)||/||b|| 8.404612520790e+01 26 KSP preconditioned resid norm 7.381709754141e+04 true resid norm 1.239993290391e+03 ||r(i)||/||b|| 8.067368050748e+01 28 KSP preconditioned resid norm 4.660758247799e+05 true resid norm 1.986383272864e+04 ||r(i)||/||b|| 1.292336424416e+03 30 KSP preconditioned resid norm 3.843023238140e+05 true resid norm 1.626650509205e+04 ||r(i)||/||b|| 1.058295109287e+03 32 KSP preconditioned resid norm 3.807302107274e+05 true resid norm 1.621607637493e+04 ||r(i)||/||b|| 1.055014228459e+03 34 KSP preconditioned resid norm 1.576847461172e+05 true resid norm 3.774334039228e+03 ||r(i)||/||b|| 2.455573112925e+02 36 KSP preconditioned resid norm 1.485636406237e+05 true resid norm 3.409895452504e+03 ||r(i)||/||b|| 2.218470200047e+02 38 KSP preconditioned resid norm 1.453089145230e+05 true resid norm 3.119776988055e+03 ||r(i)||/||b|| 2.029719789125e+02 40 KSP preconditioned resid norm 1.494647504254e+05 true resid norm 2.103792395208e+03 ||r(i)||/||b|| 1.368722531487e+02 42 KSP preconditioned resid norm 1.492528384367e+05 true resid norm 2.105571540467e+03 ||r(i)||/||b|| 1.369880039332e+02 44 KSP preconditioned resid norm 1.495172887683e+05 true resid norm 2.098100475867e+03 ||r(i)||/||b|| 1.365019381752e+02 46 KSP preconditioned resid norm 1.486262786592e+05 true resid norm 2.164181004195e+03 ||r(i)||/||b|| 1.408011222687e+02 48 KSP preconditioned resid norm 1.660465856506e+05 true resid norm 4.172426666954e+03 ||r(i)||/||b|| 2.714571268079e+02 50 KSP preconditioned resid norm 1.482759191221e+05 true resid norm 2.856097129241e+03 ||r(i)||/||b|| 1.858170274696e+02 52 KSP preconditioned resid norm 1.685729081672e+05 true resid norm 5.947178479153e+03 ||r(i)||/||b|| 3.869220747128e+02 54 KSP preconditioned resid norm 1.569362853302e+05 true resid norm 5.573721373030e+03 ||r(i)||/||b|| 3.626250406110e+02 56 KSP preconditioned resid norm 1.572502232797e+05 true resid norm 5.614318099048e+03 ||r(i)||/||b|| 3.652662543416e+02 58 KSP preconditioned resid norm 1.563201382830e+05 true resid norm 5.430932820022e+03 ||r(i)||/||b|| 3.533352499365e+02 60 KSP preconditioned resid norm 1.575394042371e+05 true resid norm 5.517707804827e+03 ||r(i)||/||b|| 3.589808106460e+02 62 KSP preconditioned resid norm 1.574611644503e+05 true resid norm 5.514851203197e+03 ||r(i)||/||b|| 3.587949607958e+02 64 KSP preconditioned resid norm 1.584082961150e+05 true resid norm 5.573531260888e+03 ||r(i)||/||b|| 3.626126719584e+02 66 KSP preconditioned resid norm 1.634951153589e+05 true resid norm 5.910135540989e+03 ||r(i)||/||b|| 3.845120696090e+02 68 KSP preconditioned resid norm 1.647079886374e+05 true resid norm 6.000249380861e+03 ||r(i)||/||b|| 3.903748554672e+02 70 KSP preconditioned resid norm 1.650191144351e+05 true resid norm 6.022517993959e+03 ||r(i)||/||b|| 3.918236463536e+02 72 KSP preconditioned resid norm 1.542157148610e+05 true resid norm 5.403131962522e+03 ||r(i)||/||b|| 3.515265324917e+02 74 KSP preconditioned resid norm 1.554195380312e+05 true resid norm 5.450251819441e+03 ||r(i)||/||b|| 3.545921396302e+02 76 KSP preconditioned resid norm 1.532469232837e+05 true resid norm 5.409499245974e+03 ||r(i)||/||b|| 3.519407865001e+02 78 KSP preconditioned resid norm 1.222695504853e+05 true resid norm 8.087451810483e+03 ||r(i)||/||b|| 5.261677692406e+02 80 KSP preconditioned resid norm 1.144385291371e+05 true resid norm 8.086187300487e+03 ||r(i)||/||b|| 5.260855005089e+02 82 KSP preconditioned resid norm 1.139628084163e+05 true resid norm 8.074654763408e+03 ||r(i)||/||b|| 5.253351962782e+02 84 KSP preconditioned resid norm 9.849803475030e+04 true resid norm 8.340298610024e+03 ||r(i)||/||b|| 5.426179243193e+02 86 KSP preconditioned resid norm 9.841631558189e+04 true resid norm 8.334019324969e+03 ||r(i)||/||b|| 5.422093954666e+02 88 KSP preconditioned resid norm 9.602058427463e+04 true resid norm 8.471718963816e+03 ||r(i)||/||b|| 5.511681025470e+02 90 KSP preconditioned resid norm 9.627762519493e+04 true resid norm 8.455558776172e+03 ||r(i)||/||b|| 5.501167244267e+02 92 KSP preconditioned resid norm 9.628212912916e+04 true resid norm 8.455709739147e+03 ||r(i)||/||b|| 5.501265460433e+02 94 KSP preconditioned resid norm 9.944415339754e+04 true resid norm 8.303497811687e+03 ||r(i)||/||b|| 5.402236727774e+02 96 KSP preconditioned resid norm 1.360957681260e+05 true resid norm 9.342234268251e+03 ||r(i)||/||b|| 6.078036296028e+02 98 KSP preconditioned resid norm 1.028254804925e+06 true resid norm 1.639833993379e+05 ||r(i)||/||b|| 1.066872254006e+04 100 KSP preconditioned resid norm 2.878267699013e+06 true resid norm 5.284924897012e+05 ||r(i)||/||b|| 3.438360077844e+04 102 KSP preconditioned resid norm 3.284723758217e+06 true resid norm 6.019019818184e+05 ||r(i)||/||b|| 3.915960558360e+04 104 KSP preconditioned resid norm 3.490898600321e+06 true resid norm 6.386257805102e+05 ||r(i)||/||b|| 4.154884754615e+04 106 KSP preconditioned resid norm 4.292371780598e+06 true resid norm 7.825387697929e+05 ||r(i)||/||b|| 5.091179378807e+04 108 KSP preconditioned resid norm 4.272469798938e+06 true resid norm 7.783966012792e+05 ||r(i)||/||b|| 5.064230524980e+04 110 KSP preconditioned resid norm 3.932496986313e+06 true resid norm 7.284977182706e+05 ||r(i)||/||b|| 4.739589530814e+04 112 KSP preconditioned resid norm 3.938646883327e+06 true resid norm 7.296330278220e+05 ||r(i)||/||b|| 4.746975828848e+04 114 KSP preconditioned resid norm 3.108284574985e+06 true resid norm 6.043424330292e+05 ||r(i)||/||b|| 3.931838078246e+04 116 KSP preconditioned resid norm 3.382223114783e+06 true resid norm 3.235035948244e+05 ||r(i)||/||b|| 2.104707005604e+04 118 KSP preconditioned resid norm 3.034407011811e+06 true resid norm 3.294035179204e+05 ||r(i)||/||b|| 2.143091770630e+04 Linear solve did not converge due to DIVERGED_ITS iterations 118 KSP Object:(pres_) 4 MPI processes type: bcgsl BCGSL: Ell = 2 BCGSL: Delta = 0 maximum iterations=120 tolerances: relative=1e-08, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object:(pres_) 4 MPI processes type: gamg MG: type is MULTIPLICATIVE, levels=5 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (pres_mg_coarse_) 4 MPI processes type: preonly maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (pres_mg_coarse_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=2, cols=2 total: nonzeros=4, allocated nonzeros=4 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 1 nodes, limit used is 5 Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (pres_mg_levels_1_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_1_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=8, cols=8 total: nonzeros=64, allocated nonzeros=64 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 2 nodes, limit used is 5 Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (pres_mg_levels_2_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_2_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=167, cols=167 total: nonzeros=9721, allocated nonzeros=9721 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (pres_mg_levels_3_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_3_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=6937, cols=6937 total: nonzeros=440237, allocated nonzeros=440237 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 4 ------------------------------- KSP Object: (pres_mg_levels_4_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_4_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=178085, cols=178085 total: nonzeros=4676387, allocated nonzeros=8374105 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=178085, cols=178085 total: nonzeros=4676387, allocated nonzeros=8374105 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name: -pres_options_left no value RUN2: Options: -pres_ksp_type bcgsl -pres_pc_type gamg -pres_pc_gamg_sym_graph -pres_pc_gamg_coarse_eq_limit 8 -pres_pc_gamg_agg_nsmooths 1 -pres_pc_gamg_threshold 0.01 -pres_mg_levels_ksp_type richardson -pres_mg_levels_pc_type sor -pres_mg_coarse_ksp_type richardson -pres_mg_coarse_pc_type sor -pres_mg_coarse_pc_sor_its 1 -pres_ksp_monitor_true_residual -pres_pc_gamg_verbose 2 -pres_ksp_view -pres_options_left [0]PCSetFromOptions_GAMG threshold set 1.000000e-02 [0]PCSetUp_GAMG level 0 N=178085, n data rows=1, n data cols=1, nnz/row (ave)=26, np=4 [0]PCGAMGFilterGraph 64.0278% nnz after filtering, with threshold 0.01, 26.2593 nnz ave. [0]maxIndSetAgg removed 0 of 178085 vertices. (0 local) 6937 selected. [0]PCGAMGProlongator_AGG New grid 6937 nodes PCGAMGOptprol_AGG smooth P0: max eigen=1.927344e+00 min=4.441888e-02 PC=jacobi [0]PCSetUp_GAMG 1) N=6937, n data cols=1, nnz/row (ave)=63, 4 active pes [0]PCGAMGFilterGraph 32.3853% nnz after filtering, with threshold 0.01, 63.4622 nnz ave. [0]maxIndSetAgg removed 0 of 6937 vertices. (0 local) 167 selected. [0]PCGAMGProlongator_AGG New grid 167 nodes PCGAMGOptprol_AGG smooth P0: max eigen=4.364217e+00 min=5.712156e-03 PC=jacobi [0]createLevel aggregate processors: npe: 4 --> 2, neq=167 [0]PCSetUp_GAMG 2) N=167, n data cols=1, nnz/row (ave)=58, 2 active pes [0]PCGAMGFilterGraph 19.7613% nnz after filtering, with threshold 0.01, 58.2096 nnz ave. [0]maxIndSetAgg removed 0 of 167 vertices. (0 local) 8 selected. [0]PCGAMGProlongator_AGG New grid 8 nodes PCGAMGOptprol_AGG smooth P0: max eigen=1.459581e+00 min=1.292445e-03 PC=jacobi [0]createLevel aggregate processors: npe: 2 --> 1, neq=8 [0]PCSetUp_GAMG 3) N=8, n data cols=1, nnz/row (ave)=8, 1 active pes [0]PCSetUp_GAMG 4 levels, grid complexity = 1.09623 Residual norms for pres_ solve. 0 KSP preconditioned resid norm 1.210558693457e+04 true resid norm 1.537048121012e+01 ||r(i)||/||b|| 1.000000000000e+00 2 KSP preconditioned resid norm 4.471234066863e+03 true resid norm 5.782528108610e+01 ||r(i)||/||b|| 3.762099591783e+00 4 KSP preconditioned resid norm 5.705586099552e+02 true resid norm 1.766618406384e+01 ||r(i)||/||b|| 1.149357903786e+00 6 KSP preconditioned resid norm 2.630493890369e+02 true resid norm 4.130300513941e+00 ||r(i)||/||b|| 2.687164089060e-01 8 KSP preconditioned resid norm 1.622096002911e+01 true resid norm 3.985823191611e-01 ||r(i)||/||b|| 2.593167472849e-02 10 KSP preconditioned resid norm 1.166276779499e+01 true resid norm 1.933128706628e-01 ||r(i)||/||b|| 1.257689125149e-02 12 KSP preconditioned resid norm 8.600869344082e-01 true resid norm 1.497599703252e-02 ||r(i)||/||b|| 9.743349494267e-04 14 KSP preconditioned resid norm 1.478956802605e-01 true resid norm 1.829141074894e-03 ||r(i)||/||b|| 1.190035009242e-04 16 KSP preconditioned resid norm 2.089299129489e-02 true resid norm 3.275473585379e-04 ||r(i)||/||b|| 2.131015640046e-05 18 KSP preconditioned resid norm 1.222936568425e-03 true resid norm 4.389356922168e-05 ||r(i)||/||b|| 2.855705597088e-06 20 KSP preconditioned resid norm 2.108428382409e-03 true resid norm 2.784330122700e-05 ||r(i)||/||b|| 1.811478824012e-06 22 KSP preconditioned resid norm 1.051218867616e-04 true resid norm 1.044274089421e-05 ||r(i)||/||b|| 6.794023395530e-07 Linear solve converged due to CONVERGED_RTOL iterations 22 KSP Object:(pres_) 4 MPI processes type: bcgsl BCGSL: Ell = 2 BCGSL: Delta = 0 maximum iterations=120 tolerances: relative=1e-08, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object:(pres_) 4 MPI processes type: gamg MG: type is MULTIPLICATIVE, levels=4 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (pres_mg_coarse_) 4 MPI processes type: preonly maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (pres_mg_coarse_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=8, cols=8 total: nonzeros=64, allocated nonzeros=64 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 2 nodes, limit used is 5 Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (pres_mg_levels_1_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_1_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=167, cols=167 total: nonzeros=9721, allocated nonzeros=9721 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (pres_mg_levels_2_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_2_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=6937, cols=6937 total: nonzeros=440237, allocated nonzeros=440237 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (pres_mg_levels_3_) 4 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_3_) 4 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=178085, cols=178085 total: nonzeros=4676387, allocated nonzeros=8374105 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=178085, cols=178085 total: nonzeros=4676387, allocated nonzeros=8374105 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-pres_options_left no value From mark.adams at columbia.edu Fri Mar 30 10:19:27 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Fri, 30 Mar 2012 11:19:27 -0400 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> <8909! A66D-5CB4-479B-ADBC-34A1133D64F4@columbia.edu> Message-ID: <9D189143-5EC3-4FCF-8947-2F1BBDB7D7AB@columbia.edu> On Mar 30, 2012, at 10:52 AM, John Mousel wrote: > Mark, > > I've run GAMG twice with different coarse grid sizes of 2 and 8 with 1 sweep of SOR on the coarse grid. For a size of 8 it converges nicely, but for a size of 2, I think the null space is causing too many problems. YEs, the iterative method is seeing the null space because of floating point error. > If GAMG were to coarsen to a size of 1, then there would be no hope because only the null space would remain, right? This doesn't ever seem to occur with ML because there are at least as many rows as processors. Yes that seems like a good assumption. The right thing to do here would probably be to do and SVD and filter out the very low modes explicitly. For now I guess tweaking -pc_gamg_coarse_eq_limit n is al that can be done. Not very satisfying. We will think about this ... any thoughts anyone? Mark > > John > > On Fri, Mar 30, 2012 at 8:42 AM, Mark F. Adams wrote: > > On Mar 29, 2012, at 2:40 PM, John Mousel wrote: > >> I'm attempting to solve a non-symmetric discretization of a 3D Poisson problem. The problem is singular. I've attached the results of KSPView from runs with ML and GAMG. When I run ML, I get convergence in 30 iterations. When I attempt to use the same settings with GAMG, I'm not getting convergence at all. The two things I notice are: >> >> 1. GAMG is using KSPType preonly, even though I've set it to be Richardson in my command line options. > > PETSc seems to switch the coarse grid solver to GMRES in Setup. This seems to be a bug and I unwisely decide to override this manually. I will undo this in the next checkin. This should not be the problem however. > >> 2. ML only coarsens down to 4 rows while GAMG coarsens to 2. My problem is singular, and whenever I try to use LU, I get zero pivot problems. To mitigate this, I've been using Richardson with SOR on the coarse matrix. Could the smaller coarse grid size of GAMG be causing problems with SOR. If so, is there a way to put a lower limit on the coarse grid size? >> > > I'm thinking that with a 2x2 coarse grid 8 iterations of SOR is picking up the null space. Maybe try just one SOR iteration on the coarse grid. > > Also, can you run with options_left so that I can see your arguments. One known bug is the mat_diagaonal_scale breaks GAMG, but it should also break ML. > > Mark > >> John >> >> On Thu, Mar 29, 2012 at 11:03 AM, Jed Brown wrote: >> On Thu, Mar 29, 2012 at 09:18, John Mousel wrote: >> [0]PETSC ERROR: Error in external library! >> [0]PETSC ERROR: Cannot disable floating point exceptions! >> >> Looks like something is strange with your environment because fesetenv() is returning an error. I have disabled the call if the trap mode is not changing. >> >> http://petsc.cs.iit.edu/petsc/petsc-dev/rev/352b4c19e451 >> >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Mar 30 10:24:23 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 30 Mar 2012 09:24:23 -0600 Subject: [petsc-users] GAMG issue In-Reply-To: <9D189143-5EC3-4FCF-8947-2F1BBDB7D7AB@columbia.edu> References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> <9D189143-5EC3-4FCF-8947-2F1BBDB7D7AB@columbia.edu> Message-ID: -mg_coarse_pc_type svd? (Use redundant for parallel.) On Mar 30, 2012 9:21 AM, "Mark F. Adams" wrote: > > On Mar 30, 2012, at 10:52 AM, John Mousel wrote: > > Mark, > > I've run GAMG twice with different coarse grid sizes of 2 and 8 with 1 > sweep of SOR on the coarse grid. For a size of 8 it converges nicely, but > for a size of 2, I think the null space is causing too many problems. > > > YEs, the iterative method is seeing the null space because of floating > point error. > > If GAMG were to coarsen to a size of 1, then there would be no hope > because only the null space would remain, right? This doesn't ever seem to > occur with ML because there are at least as many rows as processors. > > > Yes that seems like a good assumption. The right thing to do here would > probably be to do and SVD and filter out the very low modes explicitly. > For now I guess tweaking -pc_gamg_coarse_eq_limit n is al that can be > done. Not very satisfying. We will think about this ... any thoughts > anyone? > > Mark > > > John > > On Fri, Mar 30, 2012 at 8:42 AM, Mark F. Adams wrote: > >> >> On Mar 29, 2012, at 2:40 PM, John Mousel wrote: >> >> I'm attempting to solve a non-symmetric discretization of a 3D Poisson >> problem. The problem is singular. I've attached the results of KSPView from >> runs with ML and GAMG. When I run ML, I get convergence in 30 iterations. >> When I attempt to use the same settings with GAMG, I'm not getting >> convergence at all. The two things I notice are: >> >> 1. GAMG is using KSPType preonly, even though I've set it to be >> Richardson in my command line options. >> >> >> PETSc seems to switch the coarse grid solver to GMRES in Setup. This >> seems to be a bug and I unwisely decide to override this manually. I will >> undo this in the next checkin. This should not be the problem however. >> >> 2. ML only coarsens down to 4 rows while GAMG coarsens to 2. My problem >> is singular, and whenever I try to use LU, I get zero pivot problems. To >> mitigate this, I've been using Richardson with SOR on the coarse matrix. >> Could the smaller coarse grid size of GAMG be causing problems with SOR. If >> so, is there a way to put a lower limit on the coarse grid size? >> >> >> I'm thinking that with a 2x2 coarse grid 8 iterations of SOR is picking >> up the null space. Maybe try just one SOR iteration on the coarse grid. >> >> Also, can you run with options_left so that I can see your arguments. >> One known bug is the mat_diagaonal_scale breaks GAMG, but it should also >> break ML. >> >> Mark >> >> John >> >> On Thu, Mar 29, 2012 at 11:03 AM, Jed Brown wrote: >> >>> On Thu, Mar 29, 2012 at 09:18, John Mousel wrote: >>> >>>> [0]PETSC ERROR: Error in external library! >>>> [0]PETSC ERROR: Cannot disable floating point exceptions! >>>> >>> >>> Looks like something is strange with your environment because fesetenv() >>> is returning an error. I have disabled the call if the trap mode is not >>> changing. >>> >>> http://petsc.cs.iit.edu/petsc/petsc-dev/rev/352b4c19e451 >>> >> >> >> >> >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.mousel at gmail.com Fri Mar 30 10:58:40 2012 From: john.mousel at gmail.com (John Mousel) Date: Fri, 30 Mar 2012 10:58:40 -0500 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> <9D189143-5EC3-4FCF-8947-2F1BBDB7D7AB@columbia.edu> Message-ID: Mark, I've just run on one core which allows me to get ML to produce a one row coarse grid. The problem converges. I'm a bit confused. John On Fri, Mar 30, 2012 at 10:24 AM, Jed Brown wrote: > -mg_coarse_pc_type svd? > > (Use redundant for parallel.) > On Mar 30, 2012 9:21 AM, "Mark F. Adams" wrote: > >> >> On Mar 30, 2012, at 10:52 AM, John Mousel wrote: >> >> Mark, >> >> I've run GAMG twice with different coarse grid sizes of 2 and 8 with 1 >> sweep of SOR on the coarse grid. For a size of 8 it converges nicely, but >> for a size of 2, I think the null space is causing too many problems. >> >> >> YEs, the iterative method is seeing the null space because of floating >> point error. >> >> If GAMG were to coarsen to a size of 1, then there would be no hope >> because only the null space would remain, right? This doesn't ever seem to >> occur with ML because there are at least as many rows as processors. >> >> >> Yes that seems like a good assumption. The right thing to do here would >> probably be to do and SVD and filter out the very low modes explicitly. >> For now I guess tweaking -pc_gamg_coarse_eq_limit n is al that can be >> done. Not very satisfying. We will think about this ... any thoughts >> anyone? >> >> Mark >> >> >> John >> >> On Fri, Mar 30, 2012 at 8:42 AM, Mark F. Adams wrote: >> >>> >>> On Mar 29, 2012, at 2:40 PM, John Mousel wrote: >>> >>> I'm attempting to solve a non-symmetric discretization of a 3D Poisson >>> problem. The problem is singular. I've attached the results of KSPView from >>> runs with ML and GAMG. When I run ML, I get convergence in 30 iterations. >>> When I attempt to use the same settings with GAMG, I'm not getting >>> convergence at all. The two things I notice are: >>> >>> 1. GAMG is using KSPType preonly, even though I've set it to be >>> Richardson in my command line options. >>> >>> >>> PETSc seems to switch the coarse grid solver to GMRES in Setup. This >>> seems to be a bug and I unwisely decide to override this manually. I will >>> undo this in the next checkin. This should not be the problem however. >>> >>> 2. ML only coarsens down to 4 rows while GAMG coarsens to 2. My problem >>> is singular, and whenever I try to use LU, I get zero pivot problems. To >>> mitigate this, I've been using Richardson with SOR on the coarse matrix. >>> Could the smaller coarse grid size of GAMG be causing problems with SOR. If >>> so, is there a way to put a lower limit on the coarse grid size? >>> >>> >>> I'm thinking that with a 2x2 coarse grid 8 iterations of SOR is picking >>> up the null space. Maybe try just one SOR iteration on the coarse grid. >>> >>> Also, can you run with options_left so that I can see your arguments. >>> One known bug is the mat_diagaonal_scale breaks GAMG, but it should also >>> break ML. >>> >>> Mark >>> >>> John >>> >>> On Thu, Mar 29, 2012 at 11:03 AM, Jed Brown wrote: >>> >>>> On Thu, Mar 29, 2012 at 09:18, John Mousel wrote: >>>> >>>>> [0]PETSC ERROR: Error in external library! >>>>> [0]PETSC ERROR: Cannot disable floating point exceptions! >>>>> >>>> >>>> Looks like something is strange with your environment because >>>> fesetenv() is returning an error. I have disabled the call if the trap mode >>>> is not changing. >>>> >>>> http://petsc.cs.iit.edu/petsc/petsc-dev/rev/352b4c19e451 >>>> >>> >>> >>> >>> >>> >> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- -pres_ksp_type bcgsl -pres_pc_type ml -pres_mg_coarse_ksp_type richardson -pres_mg_coarse_pc_type sor -pres_mg_coarse_pc_sor_its 8 -pres_ksp_monitor_true_residual -pres_ksp_view -pres_pc_ml_Threshold .01 -pres_options_left Residual norms for pres_ solve. 0 KSP preconditioned resid norm 2.484438808878e+03 true resid norm 1.537048121260e+01 ||r(i)||/||b|| 1.000000000000e+00 2 KSP preconditioned resid norm 4.803121617032e+03 true resid norm 2.730837669491e+02 ||r(i)||/||b|| 1.776676755736e+01 4 KSP preconditioned resid norm 6.585025011351e+02 true resid norm 3.800284284730e+01 ||r(i)||/||b|| 2.472456283031e+00 6 KSP preconditioned resid norm 3.225377737134e+02 true resid norm 1.185212296990e+01 ||r(i)||/||b|| 7.710964156533e-01 8 KSP preconditioned resid norm 4.960746023374e+01 true resid norm 4.118225945036e+00 ||r(i)||/||b|| 2.679308401652e-01 10 KSP preconditioned resid norm 9.030392220992e+00 true resid norm 1.475499217091e+00 ||r(i)||/||b|| 9.599564234079e-02 12 KSP preconditioned resid norm 3.084127960093e+00 true resid norm 1.384958803077e-01 ||r(i)||/||b|| 9.010510366727e-03 14 KSP preconditioned resid norm 4.079306062909e+00 true resid norm 4.367252084412e-01 ||r(i)||/||b|| 2.841324239629e-02 16 KSP preconditioned resid norm 5.156398532433e-01 true resid norm 5.165106549016e-02 ||r(i)||/||b|| 3.360406533519e-03 18 KSP preconditioned resid norm 9.880823156494e-01 true resid norm 4.287333629430e-02 ||r(i)||/||b|| 2.789329475199e-03 20 KSP preconditioned resid norm 2.420500561140e-02 true resid norm 5.178810721229e-03 ||r(i)||/||b|| 3.369322436687e-04 22 KSP preconditioned resid norm 2.874890434196e-02 true resid norm 3.193794305528e-03 ||r(i)||/||b|| 2.077875286631e-04 24 KSP preconditioned resid norm 3.412881480874e-03 true resid norm 1.504951460520e-04 ||r(i)||/||b|| 9.791179857704e-06 26 KSP preconditioned resid norm 4.854602551459e-04 true resid norm 7.508313761033e-05 ||r(i)||/||b|| 4.884891798233e-06 28 KSP preconditioned resid norm 5.821234231049e-06 true resid norm 4.848903629291e-05 ||r(i)||/||b|| 3.154685635552e-06 KSP Object:(pres_) 1 MPI processes type: bcgsl BCGSL: Ell = 2 BCGSL: Delta = 0 maximum iterations=1000 tolerances: relative=1e-08, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object:(pres_) 1 MPI processes type: ml MG: type is MULTIPLICATIVE, levels=7 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (pres_mg_coarse_) 1 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_coarse_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 8, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=1, cols=1 total: nonzeros=1, allocated nonzeros=1 total number of mallocs used during MatSetValues calls =0 not using I-node routines Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (pres_mg_levels_1_) 1 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_1_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=2, cols=2 total: nonzeros=4, allocated nonzeros=4 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 1 nodes, limit used is 5 Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (pres_mg_levels_2_) 1 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_2_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=5, cols=5 total: nonzeros=21, allocated nonzeros=21 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 3 nodes, limit used is 5 Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (pres_mg_levels_3_) 1 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_3_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=59, cols=59 total: nonzeros=1661, allocated nonzeros=1661 total number of mallocs used during MatSetValues calls =0 not using I-node routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 4 ------------------------------- KSP Object: (pres_mg_levels_4_) 1 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_4_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=833, cols=833 total: nonzeros=68657, allocated nonzeros=68657 total number of mallocs used during MatSetValues calls =0 not using I-node routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 5 ------------------------------- KSP Object: (pres_mg_levels_5_) 1 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_5_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=16364, cols=16364 total: nonzeros=1501443, allocated nonzeros=1501443 total number of mallocs used during MatSetValues calls =0 not using I-node routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 6 ------------------------------- KSP Object: (pres_mg_levels_6_) 1 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object: (pres_mg_levels_6_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=178085, cols=178085 total: nonzeros=4676387, allocated nonzeros=8385165 total number of mallocs used during MatSetValues calls =0 not using I-node routines Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=178085, cols=178085 total: nonzeros=4676387, allocated nonzeros=8385165 total number of mallocs used during MatSetValues calls =0 not using I-node routines total run time = 284.970678000000 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! -------------- next part -------------- Options: -pres_ksp_type bcgsl -pres_pc_type gamg -pres_pc_gamg_threshold 0.01 -pres_pc_gamg_sym_graph -pres_pc_gamg_coarse_eq_limit 8 -pres_pc_gamg_agg_nsmooths 1 -pres_mg_levels_ksp_type richardson -pres_mg_levels_pc_type sor -pres_mg_coarse_ksp_type richardson -pres_mg_coarse_pc_type sor -pres_mg_coarse_pc_sor_its 8 -pres_pc_gamg_verbose 2 -pres_ksp_converged_reason -pres_options_left -pres_ksp_view -pres_ksp_max_it 120 -pres_options_left -pres_ksp_monitor_true_residual [0]PCSetFromOptions_GAMG threshold set 1.000000e-02 [0]PCSetUp_GAMG level 0 N=178085, n data rows=1, n data cols=1, nnz/row (ave)=26, np=1 [0]PCGAMGFilterGraph 64.0278% nnz after filtering, with threshold 0.01, 26.2593 nnz ave. [0]maxIndSetAgg removed 0 of 178085 vertices. (0 local) 6889 selected. [0]PCGAMGProlongator_AGG New grid 6889 nodes PCGAMGOptprol_AGG smooth P0: max eigen=1.926427e+00 min=4.469907e-02 PC=jacobi [0]PCSetUp_GAMG 1) N=6889, n data cols=1, nnz/row (ave)=62, 1 active pes [0]PCGAMGFilterGraph 32.4218% nnz after filtering, with threshold 0.01, 62.979 nnz ave. [0]maxIndSetAgg removed 0 of 6889 vertices. (0 local) 162 selected. [0]PCGAMGProlongator_AGG New grid 162 nodes PCGAMGOptprol_AGG smooth P0: max eigen=1.619990e+00 min=3.528137e-03 PC=jacobi [0]PCSetUp_GAMG 2) N=162, n data cols=1, nnz/row (ave)=55, 1 active pes [0]PCGAMGFilterGraph 28.2109% nnz after filtering, with threshold 0.01, 55.2716 nnz ave. [0]maxIndSetAgg removed 0 of 162 vertices. (0 local) 5 selected. [0]PCGAMGProlongator_AGG New grid 5 nodes PCGAMGOptprol_AGG smooth P0: max eigen=1.569842e+00 min=2.465992e-03 PC=jacobi [0]PCSetUp_GAMG 3) N=5, n data cols=1, nnz/row (ave)=5, 1 active pes [0]PCGAMGFilterGraph 72% nnz after filtering, with threshold 0.01, 5 nnz ave. [0]maxIndSetAgg removed 0 of 5 vertices. (0 local) 1 selected. [0]PCGAMGProlongator_AGG New grid 1 nodes PCGAMGOptprol_AGG smooth P0: max eigen=3.686042e+00 min=1.557703e-13 PC=jacobi [0]PCSetUp_GAMG 4) N=1, n data cols=1, nnz/row (ave)=1, 1 active pes [0]PCSetUp_GAMG 5 levels, grid complexity = 1.0947 Residual norms for pres_ solve. 0 KSP preconditioned resid norm 7.588646719755e+04 true resid norm 1.537048121260e+01 ||r(i)||/||b|| 1.000000000000e+00 2 KSP preconditioned resid norm 1.326222725745e+03 true resid norm 1.669373471357e+01 ||r(i)||/||b|| 1.086090570794e+00 4 KSP preconditioned resid norm 8.694481142514e+02 true resid norm 1.002676416276e+01 ||r(i)||/||b|| 6.523389882247e-01 6 KSP preconditioned resid norm 9.189263949949e+01 true resid norm 6.222035822438e+00 ||r(i)||/||b|| 4.048042306793e-01 8 KSP preconditioned resid norm 3.703995815828e+01 true resid norm 5.983853333761e+00 ||r(i)||/||b|| 3.893081323215e-01 10 KSP preconditioned resid norm 2.183186527735e+01 true resid norm 5.952245622356e+00 ||r(i)||/||b|| 3.872517418307e-01 12 KSP preconditioned resid norm 3.493925445567e+02 true resid norm 7.329828465587e+00 ||r(i)||/||b|| 4.768769672336e-01 14 KSP preconditioned resid norm 1.356658375642e+03 true resid norm 1.739768043232e+01 ||r(i)||/||b|| 1.131889118609e+00 16 KSP preconditioned resid norm 5.412294172351e+02 true resid norm 6.711159067106e+00 ||r(i)||/||b|| 4.366264773548e-01 18 KSP preconditioned resid norm 4.616224894440e+02 true resid norm 7.521105420582e+00 ||r(i)||/||b|| 4.893214022744e-01 20 KSP preconditioned resid norm 2.577285776652e+02 true resid norm 6.858357846531e+00 ||r(i)||/||b|| 4.462031963520e-01 22 KSP preconditioned resid norm 1.231090948563e+02 true resid norm 6.138029315352e+00 ||r(i)||/||b|| 3.993387864994e-01 24 KSP preconditioned resid norm 2.852537299632e+03 true resid norm 1.740135209172e+01 ||r(i)||/||b|| 1.132127995931e+00 26 KSP preconditioned resid norm 2.236778478113e+04 true resid norm 2.346805733410e+02 ||r(i)||/||b|| 1.526826454520e+01 28 KSP preconditioned resid norm 1.869320270064e+04 true resid norm 1.805684545024e+02 ||r(i)||/||b|| 1.174774244247e+01 30 KSP preconditioned resid norm 2.308247715341e+04 true resid norm 2.257355295321e+02 ||r(i)||/||b|| 1.468630203633e+01 32 KSP preconditioned resid norm 2.261854214989e+04 true resid norm 2.331192380366e+02 ||r(i)||/||b|| 1.516668442661e+01 34 KSP preconditioned resid norm 2.497512693081e+04 true resid norm 2.734577063670e+02 ||r(i)||/||b|| 1.779109597055e+01 36 KSP preconditioned resid norm 2.596144404138e+04 true resid norm 3.014136843412e+02 ||r(i)||/||b|| 1.960990551773e+01 38 KSP preconditioned resid norm 2.350126041193e+04 true resid norm 2.252989797587e+02 ||r(i)||/||b|| 1.465790020771e+01 40 KSP preconditioned resid norm 2.477262261324e+04 true resid norm 2.564872285424e+02 ||r(i)||/||b|| 1.668700055611e+01 42 KSP preconditioned resid norm 6.931302940514e+04 true resid norm 8.605328793761e+02 ||r(i)||/||b|| 5.598607275032e+01 44 KSP preconditioned resid norm 8.688325842110e+04 true resid norm 1.094833786804e+03 ||r(i)||/||b|| 7.122963631788e+01 46 KSP preconditioned resid norm 4.479201456138e+04 true resid norm 2.694563200163e+02 ||r(i)||/||b|| 1.753076668773e+01 48 KSP preconditioned resid norm 2.165097859297e+04 true resid norm 3.074224494605e+02 ||r(i)||/||b|| 2.000083440514e+01 50 KSP preconditioned resid norm 2.055902332706e+04 true resid norm 2.890284413681e+02 ||r(i)||/||b|| 1.880412443633e+01 52 KSP preconditioned resid norm 2.226003271579e+04 true resid norm 3.190481467683e+02 ||r(i)||/||b|| 2.075719961888e+01 54 KSP preconditioned resid norm 2.649136916398e+04 true resid norm 3.633403292566e+02 ||r(i)||/||b|| 2.363883890367e+01 56 KSP preconditioned resid norm 3.636407292301e+03 true resid norm 1.770809887959e+02 ||r(i)||/||b|| 1.152084871947e+01 58 KSP preconditioned resid norm 1.911384813961e+03 true resid norm 1.764445605915e+02 ||r(i)||/||b|| 1.147944284574e+01 60 KSP preconditioned resid norm 3.026919479329e+02 true resid norm 1.794750740713e+02 ||r(i)||/||b|| 1.167660736114e+01 62 KSP preconditioned resid norm 3.308470165040e+02 true resid norm 1.771235549660e+02 ||r(i)||/||b|| 1.152361806479e+01 64 KSP preconditioned resid norm 1.051625068085e+02 true resid norm 1.775402024438e+02 ||r(i)||/||b|| 1.155072505461e+01 66 KSP preconditioned resid norm 2.235446953063e+01 true resid norm 1.778283859610e+02 ||r(i)||/||b|| 1.156947420847e+01 68 KSP preconditioned resid norm 2.112406298937e+01 true resid norm 1.777642739041e+02 ||r(i)||/||b|| 1.156530309268e+01 70 KSP preconditioned resid norm 2.326066510932e+01 true resid norm 1.777623800257e+02 ||r(i)||/||b|| 1.156517987738e+01 72 KSP preconditioned resid norm 3.398049334914e+00 true resid norm 1.777679162542e+02 ||r(i)||/||b|| 1.156554006315e+01 74 KSP preconditioned resid norm 5.201069185149e+00 true resid norm 1.777678703221e+02 ||r(i)||/||b|| 1.156553707482e+01 76 KSP preconditioned resid norm 5.533064413464e+00 true resid norm 1.777679308795e+02 ||r(i)||/||b|| 1.156554101466e+01 78 KSP preconditioned resid norm 3.913801853323e+00 true resid norm 1.777680240223e+02 ||r(i)||/||b|| 1.156554707452e+01 80 KSP preconditioned resid norm 2.537859018317e+00 true resid norm 1.777681223249e+02 ||r(i)||/||b|| 1.156555347006e+01 82 KSP preconditioned resid norm 7.741748315959e+00 true resid norm 1.777654007295e+02 ||r(i)||/||b|| 1.156537640369e+01 84 KSP preconditioned resid norm 4.440928932749e+00 true resid norm 1.777677225227e+02 ||r(i)||/||b|| 1.156552745902e+01 86 KSP preconditioned resid norm 3.704557177890e+00 true resid norm 1.777669720986e+02 ||r(i)||/||b|| 1.156547863660e+01 88 KSP preconditioned resid norm 2.046666429633e+00 true resid norm 1.777675304674e+02 ||r(i)||/||b|| 1.156551496395e+01 90 KSP preconditioned resid norm 2.774055639762e+00 true resid norm 1.777673010008e+02 ||r(i)||/||b|| 1.156550003490e+01 92 KSP preconditioned resid norm 2.777963966142e+00 true resid norm 1.777673492779e+02 ||r(i)||/||b|| 1.156550317580e+01 94 KSP preconditioned resid norm 3.047451777681e-01 true resid norm 1.777673215532e+02 ||r(i)||/||b|| 1.156550137203e+01 96 KSP preconditioned resid norm 1.391897161136e-01 true resid norm 1.777673352268e+02 ||r(i)||/||b|| 1.156550226164e+01 98 KSP preconditioned resid norm 4.877455333649e-02 true resid norm 1.777673302262e+02 ||r(i)||/||b|| 1.156550193630e+01 100 KSP preconditioned resid norm 2.319094057406e-02 true resid norm 1.777673333910e+02 ||r(i)||/||b|| 1.156550214220e+01 102 KSP preconditioned resid norm 1.902403543163e-01 true resid norm 1.777673373724e+02 ||r(i)||/||b|| 1.156550240123e+01 104 KSP preconditioned resid norm 2.088352126122e-02 true resid norm 1.777673337586e+02 ||r(i)||/||b|| 1.156550216612e+01 106 KSP preconditioned resid norm 1.557333341580e-03 true resid norm 1.777673349772e+02 ||r(i)||/||b|| 1.156550224540e+01 108 KSP preconditioned resid norm 1.505375301694e-03 true resid norm 1.777673349419e+02 ||r(i)||/||b|| 1.156550224310e+01 110 KSP preconditioned resid norm 1.671356769285e-03 true resid norm 1.777673349043e+02 ||r(i)||/||b|| 1.156550224066e+01 112 KSP preconditioned resid norm 4.450900294941e-04 true resid norm 1.777673343536e+02 ||r(i)||/||b|| 1.156550220483e+01 Linear solve converged due to CONVERGED_RTOL iterations 112 KSP Object:(pres_) 1 MPI processes type: bcgsl BCGSL: Ell = 2 BCGSL: Delta = 0 maximum iterations=120 tolerances: relative=1e-08, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using PRECONDITIONED norm type for convergence test PC Object:(pres_) 1 MPI processes type: gamg MG: type is MULTIPLICATIVE, levels=5 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (pres_mg_coarse_) 1 MPI processes type: preonly maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (pres_mg_coarse_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 8, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=1, cols=1 total: nonzeros=1, allocated nonzeros=1 total number of mallocs used during MatSetValues calls =0 not using I-node routines Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (pres_mg_levels_1_) 1 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_1_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=5, cols=5 total: nonzeros=25, allocated nonzeros=25 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 1 nodes, limit used is 5 Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (pres_mg_levels_2_) 1 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_2_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=162, cols=162 total: nonzeros=8954, allocated nonzeros=8954 total number of mallocs used during MatSetValues calls =0 not using I-node routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (pres_mg_levels_3_) 1 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_3_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=6889, cols=6889 total: nonzeros=433862, allocated nonzeros=433862 total number of mallocs used during MatSetValues calls =0 not using I-node routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 4 ------------------------------- KSP Object: (pres_mg_levels_4_) 1 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning has attached null space using nonzero initial guess using NONE norm type for convergence test PC Object: (pres_mg_levels_4_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=178085, cols=178085 total: nonzeros=4676387, allocated nonzeros=8385165 total number of mallocs used during MatSetValues calls =0 not using I-node routines Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=178085, cols=178085 total: nonzeros=4676387, allocated nonzeros=8385165 total number of mallocs used during MatSetValues calls =0 not using I-node routines total run time = 326.601349000000 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-lspaint_options_left no value Option left: name:-nullity_options_left no value Option left: name:-pres_options_left no value From mark.adams at columbia.edu Fri Mar 30 12:55:55 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Fri, 30 Mar 2012 13:55:55 -0400 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> <9D189143-5EC3-4FCF-8947-2F1BBDB7D7AB@columbia.edu> Message-ID: <8EC8C111-E0A3-4E87-9AB3-F3523E3BDEF9@columbia.edu> Humm, that is mysterious. ML uses the same PETSc MG infrastructure ... so there must be something in its prolongation operators... I thought of what _might_ be a better way to run your problems. Set "-pc_gamg_coarse_eq_limit 1 [or a very small integer]" and then use PC "none" on the coarse grid. Maybe even PREONLY so the coarse grid does nothing. Mark On Mar 30, 2012, at 11:58 AM, John Mousel wrote: > Mark, > > I've just run on one core which allows me to get ML to produce a one row coarse grid. The problem converges. I'm a bit confused. > > John > > On Fri, Mar 30, 2012 at 10:24 AM, Jed Brown wrote: > -mg_coarse_pc_type svd? > > (Use redundant for parallel.) > > On Mar 30, 2012 9:21 AM, "Mark F. Adams" wrote: > > On Mar 30, 2012, at 10:52 AM, John Mousel wrote: > >> Mark, >> >> I've run GAMG twice with different coarse grid sizes of 2 and 8 with 1 sweep of SOR on the coarse grid. For a size of 8 it converges nicely, but for a size of 2, I think the null space is causing too many problems. > > YEs, the iterative method is seeing the null space because of floating point error. > >> If GAMG were to coarsen to a size of 1, then there would be no hope because only the null space would remain, right? This doesn't ever seem to occur with ML because there are at least as many rows as processors. > > Yes that seems like a good assumption. The right thing to do here would probably be to do and SVD and filter out the very low modes explicitly. For now I guess tweaking -pc_gamg_coarse_eq_limit n is al that can be done. Not very satisfying. We will think about this ... any thoughts anyone? > > Mark > >> >> John >> >> On Fri, Mar 30, 2012 at 8:42 AM, Mark F. Adams wrote: >> >> On Mar 29, 2012, at 2:40 PM, John Mousel wrote: >> >>> I'm attempting to solve a non-symmetric discretization of a 3D Poisson problem. The problem is singular. I've attached the results of KSPView from runs with ML and GAMG. When I run ML, I get convergence in 30 iterations. When I attempt to use the same settings with GAMG, I'm not getting convergence at all. The two things I notice are: >>> >>> 1. GAMG is using KSPType preonly, even though I've set it to be Richardson in my command line options. >> >> PETSc seems to switch the coarse grid solver to GMRES in Setup. This seems to be a bug and I unwisely decide to override this manually. I will undo this in the next checkin. This should not be the problem however. >> >>> 2. ML only coarsens down to 4 rows while GAMG coarsens to 2. My problem is singular, and whenever I try to use LU, I get zero pivot problems. To mitigate this, I've been using Richardson with SOR on the coarse matrix. Could the smaller coarse grid size of GAMG be causing problems with SOR. If so, is there a way to put a lower limit on the coarse grid size? >>> >> >> I'm thinking that with a 2x2 coarse grid 8 iterations of SOR is picking up the null space. Maybe try just one SOR iteration on the coarse grid. >> >> Also, can you run with options_left so that I can see your arguments. One known bug is the mat_diagaonal_scale breaks GAMG, but it should also break ML. >> >> Mark >> >>> John >>> >>> On Thu, Mar 29, 2012 at 11:03 AM, Jed Brown wrote: >>> On Thu, Mar 29, 2012 at 09:18, John Mousel wrote: >>> [0]PETSC ERROR: Error in external library! >>> [0]PETSC ERROR: Cannot disable floating point exceptions! >>> >>> Looks like something is strange with your environment because fesetenv() is returning an error. I have disabled the call if the trap mode is not changing. >>> >>> http://petsc.cs.iit.edu/petsc/petsc-dev/rev/352b4c19e451 >>> >>> >> >> >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Fri Mar 30 12:56:57 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Fri, 30 Mar 2012 13:56:57 -0400 Subject: [petsc-users] GAMG issue In-Reply-To: References: <3FD99791-1A35-4E50-8CF3-002E3F8E8D33@columbia.edu> <362A4431-F9D5-49C9-B02D-7F5076AD5C82@columbia.edu> <2CC0EE2A-D7B9-40B6-96EE-4EFBAA886DFB@columbia.edu> <36A0D9E7-E6F2-4F1A-8CC3-4DFDA92A9E84@columbia.edu> <4115E470-A868-44D4-A32A-31D2B21BB5D2@columbia.edu> <9D3034B9-AE48-46FE-93D8-ED7C3A60EBBA@columbia.edu> <68A01909-D54E-401A-A231-71614E09A986@columbia.edu> <9D189143-5EC3-4FCF-8947-2F1BBDB7D7AB@columbia.edu> Message-ID: John, try that. Will this be smart enough to filter out very small values?... On Mar 30, 2012, at 11:24 AM, Jed Brown wrote: > -mg_coarse_pc_type svd? > > (Use redundant for parallel.) > > On Mar 30, 2012 9:21 AM, "Mark F. Adams" wrote: > > On Mar 30, 2012, at 10:52 AM, John Mousel wrote: > >> Mark, >> >> I've run GAMG twice with different coarse grid sizes of 2 and 8 with 1 sweep of SOR on the coarse grid. For a size of 8 it converges nicely, but for a size of 2, I think the null space is causing too many problems. > > YEs, the iterative method is seeing the null space because of floating point error. > >> If GAMG were to coarsen to a size of 1, then there would be no hope because only the null space would remain, right? This doesn't ever seem to occur with ML because there are at least as many rows as processors. > > Yes that seems like a good assumption. The right thing to do here would probably be to do and SVD and filter out the very low modes explicitly. For now I guess tweaking -pc_gamg_coarse_eq_limit n is al that can be done. Not very satisfying. We will think about this ... any thoughts anyone? > > Mark > >> >> John >> >> On Fri, Mar 30, 2012 at 8:42 AM, Mark F. Adams wrote: >> >> On Mar 29, 2012, at 2:40 PM, John Mousel wrote: >> >>> I'm attempting to solve a non-symmetric discretization of a 3D Poisson problem. The problem is singular. I've attached the results of KSPView from runs with ML and GAMG. When I run ML, I get convergence in 30 iterations. When I attempt to use the same settings with GAMG, I'm not getting convergence at all. The two things I notice are: >>> >>> 1. GAMG is using KSPType preonly, even though I've set it to be Richardson in my command line options. >> >> PETSc seems to switch the coarse grid solver to GMRES in Setup. This seems to be a bug and I unwisely decide to override this manually. I will undo this in the next checkin. This should not be the problem however. >> >>> 2. ML only coarsens down to 4 rows while GAMG coarsens to 2. My problem is singular, and whenever I try to use LU, I get zero pivot problems. To mitigate this, I've been using Richardson with SOR on the coarse matrix. Could the smaller coarse grid size of GAMG be causing problems with SOR. If so, is there a way to put a lower limit on the coarse grid size? >>> >> >> I'm thinking that with a 2x2 coarse grid 8 iterations of SOR is picking up the null space. Maybe try just one SOR iteration on the coarse grid. >> >> Also, can you run with options_left so that I can see your arguments. One known bug is the mat_diagaonal_scale breaks GAMG, but it should also break ML. >> >> Mark >> >>> John >>> >>> On Thu, Mar 29, 2012 at 11:03 AM, Jed Brown wrote: >>> On Thu, Mar 29, 2012 at 09:18, John Mousel wrote: >>> [0]PETSC ERROR: Error in external library! >>> [0]PETSC ERROR: Cannot disable floating point exceptions! >>> >>> Looks like something is strange with your environment because fesetenv() is returning an error. I have disabled the call if the trap mode is not changing. >>> >>> http://petsc.cs.iit.edu/petsc/petsc-dev/rev/352b4c19e451 >>> >>> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.spott at gmail.com Fri Mar 30 17:59:55 2012 From: andrew.spott at gmail.com (Andrew Spott) Date: Fri, 30 Mar 2012 16:59:55 -0600 Subject: [petsc-users] Matrix/Vector Assembly Message-ID: <9370B6EE-DD0D-481E-8344-B019A58D2344@gmail.com> When is it necessary to assemble a matrix/vector? Do I need to do it after every change in the matrix/vector? Or just after "AddValue(s)" type functions? -Andrew From knepley at gmail.com Fri Mar 30 18:15:56 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 30 Mar 2012 18:15:56 -0500 Subject: [petsc-users] Matrix/Vector Assembly In-Reply-To: <9370B6EE-DD0D-481E-8344-B019A58D2344@gmail.com> References: <9370B6EE-DD0D-481E-8344-B019A58D2344@gmail.com> Message-ID: On Fri, Mar 30, 2012 at 5:59 PM, Andrew Spott wrote: > When is it necessary to assemble a matrix/vector? Do I need to do it > after every change in the matrix/vector? Or just after "AddValue(s)" type > functions? Every change. Matt > -Andrew -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Mar 30 20:14:52 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 30 Mar 2012 20:14:52 -0500 Subject: [petsc-users] Matrix/Vector Assembly In-Reply-To: References: <9370B6EE-DD0D-481E-8344-B019A58D2344@gmail.com> Message-ID: On Mar 30, 2012, at 6:15 PM, Matthew Knepley wrote: > On Fri, Mar 30, 2012 at 5:59 PM, Andrew Spott wrote: > When is it necessary to assemble a matrix/vector? Do I need to do it after every change in the matrix/vector? Or just after "AddValue(s)" type functions? > > Every change. Well after calls to MatSetValues and friends but not MatScale() MatShift() MatAXPY() etc > > Matt > > -Andrew > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener