From saswata at umd.edu Sat Nov 1 11:55:35 2008 From: saswata at umd.edu (Saswata Hier-Majumder) Date: Sat, 01 Nov 2008 12:55:35 -0400 Subject: Running petsc on infiniband Message-ID: <490C8A07.9020309@umd.edu> Greetings, I am new to petsc and not sure how to interpret the result I get when running petsc example codes (bratu problem and hello world) in an SGI Altix 1300 cluster. Any help will be greatly appreciated! I compile the petsc code using a makefile that came with the petsc distribution, submit the job to the queue using pbspro, and run using mpirun. The solution returned by running the bratu problem ($PETSC_DIR/src/snes/examples/tutorials/ex5f90.F) looks correct, but I am not sure if petsc is using all processors supplied to it by mpirun. MPI_COMM_SIZE always returns a size of 1 and MPI_COMM_RANK always returns rank 0, when I call these routines from inside my code, despite using a command like mpirun -n 16 -machinefile $PBS_NODEFILE myexecutable.exe in my submit shell script. I get similar results when I run hello world with Petsc. It prints 16 lines displaying rank=0 and size=1, when run with 16 processors. Run just with mpi, helloworld prints the size and ranks as expected, i.e. 16 different lines with size 16 and ranks from 0 to 15. Am I correct in assuming that petsc is somehow not able use the version of mpirun implemented in vltmpi? I reran the petsc using the install shell script given below, but it did not help export PETSC_DIR=$PWD ./config/configure.py --with-mpi-dir=/opt/vltmpi/OPENIB/mpi.icc.rsh/bin make all test Sash Hier-Majumder -- www.geol.umd.edu/~saswata From balay at mcs.anl.gov Sat Nov 1 12:14:03 2008 From: balay at mcs.anl.gov (Satish Balay) Date: Sat, 1 Nov 2008 12:14:03 -0500 (CDT) Subject: Running petsc on infiniband In-Reply-To: <490C8A07.9020309@umd.edu> References: <490C8A07.9020309@umd.edu> Message-ID: On Sat, 1 Nov 2008, Saswata Hier-Majumder wrote: > Greetings, > I am new to petsc and not sure how to interpret the result I get when running > petsc example codes (bratu problem and hello world) in an SGI Altix 1300 > cluster. Any help will be greatly appreciated! > > I compile the petsc code using a makefile that came with the petsc > distribution, submit the job to the queue using pbspro, and run using mpirun. > The solution returned by running the bratu problem > ($PETSC_DIR/src/snes/examples/tutorials/ex5f90.F) looks correct, but I am > not sure if petsc is using all processors supplied to it by mpirun. > MPI_COMM_SIZE always returns a size of 1 and MPI_COMM_RANK always returns rank > 0, when I call these routines from inside my code, despite using a command > like > > mpirun -n 16 -machinefile $PBS_NODEFILE myexecutable.exe > > in my submit shell script. > > I get similar results when I run hello world with Petsc. It prints 16 lines > displaying rank=0 and size=1, when run with 16 processors. Run just with mpi, > helloworld prints the size and ranks as expected, i.e. 16 different lines with > size 16 and ranks from 0 to 15. > > Am I correct in assuming that petsc is somehow not able use the version of > mpirun implemented in vltmpi? I reran the petsc using the install shell > script given below, but it did not help > > export PETSC_DIR=$PWD > ./config/configure.py --with-mpi-dir=/opt/vltmpi/OPENIB/mpi.icc.rsh/bin This should be --with-mpi-dir=/opt/vltmpi/OPENIB/mpi.icc.rsh Configure should print a summary of compilers being used. [it should be /opt/vltmpi/OPENIB/mpi.icc.rsh/bin/mpicc etc..] If you can use /opt/vltmpi/OPENIB/mpi.icc.rsh/bin/mpicc & mpirun with a sample MPI test code and run it parallelly - then you should be able to do the exact same thing with PETSc examples [using the same tools] Satish > make all test > > Sash Hier-Majumder > > From saswata at umd.edu Sat Nov 1 13:20:34 2008 From: saswata at umd.edu (Saswata Hier-Majumder) Date: Sat, 01 Nov 2008 14:20:34 -0400 Subject: Running petsc on infiniband In-Reply-To: References: <490C8A07.9020309@umd.edu> Message-ID: <490C9DF2.7080301@umd.edu> Satish, Thanks for the prompt response! After rerunning the installation script with the correct path, I found a bunch of error messages in the configure.log file which look like sh: /opt/vltmpi/OPENIB/mpi.icc.rsh/bin/mpif90 -o conftest conftest.o Executing: /opt/vltmpi/OPENIB/mpi.icc.rsh/bin/mpif90 -o conftest conftest.o sh: Possible ERROR while running linker: /opt/intel/fce/10.0.023/lib/libimf.so: warning: warning: feupdateenv is not implemented and will always fail I recognize the last part of the error message. I need to use the option -nochoicemod while compiling an mpi code with mpif90 to avoid this error. Is there any way for me to pass that option to the petsc installer? I tried --with-fc=mpif90 -nochoicemod in the install shell script, but that did not seem to work. Thanks Sash Satish Balay wrote: > On Sat, 1 Nov 2008, Saswata Hier-Majumder wrote: > > >> Greetings, >> I am new to petsc and not sure how to interpret the result I get when running >> petsc example codes (bratu problem and hello world) in an SGI Altix 1300 >> cluster. Any help will be greatly appreciated! >> >> I compile the petsc code using a makefile that came with the petsc >> distribution, submit the job to the queue using pbspro, and run using mpirun. >> The solution returned by running the bratu problem >> ($PETSC_DIR/src/snes/examples/tutorials/ex5f90.F) looks correct, but I am >> not sure if petsc is using all processors supplied to it by mpirun. >> MPI_COMM_SIZE always returns a size of 1 and MPI_COMM_RANK always returns rank >> 0, when I call these routines from inside my code, despite using a command >> like >> >> mpirun -n 16 -machinefile $PBS_NODEFILE myexecutable.exe >> >> in my submit shell script. >> >> I get similar results when I run hello world with Petsc. It prints 16 lines >> displaying rank=0 and size=1, when run with 16 processors. Run just with mpi, >> helloworld prints the size and ranks as expected, i.e. 16 different lines with >> size 16 and ranks from 0 to 15. >> >> Am I correct in assuming that petsc is somehow not able use the version of >> mpirun implemented in vltmpi? I reran the petsc using the install shell >> script given below, but it did not help >> >> export PETSC_DIR=$PWD >> ./config/configure.py --with-mpi-dir=/opt/vltmpi/OPENIB/mpi.icc.rsh/bin >> > > This should be --with-mpi-dir=/opt/vltmpi/OPENIB/mpi.icc.rsh > > Configure should print a summary of compilers being used. [it should > be /opt/vltmpi/OPENIB/mpi.icc.rsh/bin/mpicc etc..] > > If you can use /opt/vltmpi/OPENIB/mpi.icc.rsh/bin/mpicc & mpirun with a > sample MPI test code and run it parallelly - then you should be able to do > the exact same thing with PETSc examples [using the same tools] > > Satish > > >> make all test >> >> Sash Hier-Majumder >> >> >> > > -- www.geol.umd.edu/~saswata From balay at mcs.anl.gov Sat Nov 1 13:24:17 2008 From: balay at mcs.anl.gov (Satish Balay) Date: Sat, 1 Nov 2008 13:24:17 -0500 (CDT) Subject: Running petsc on infiniband In-Reply-To: <490C9DF2.7080301@umd.edu> References: <490C8A07.9020309@umd.edu> <490C9DF2.7080301@umd.edu> Message-ID: On Sat, 1 Nov 2008, Saswata Hier-Majumder wrote: > Satish, > Thanks for the prompt response! > > After rerunning the installation script with the correct path, I found a bunch > of error messages in the configure.log file which look like > > sh: /opt/vltmpi/OPENIB/mpi.icc.rsh/bin/mpif90 -o conftest conftest.o > Executing: /opt/vltmpi/OPENIB/mpi.icc.rsh/bin/mpif90 -o conftest conftest.o > sh: > Possible ERROR while running linker: /opt/intel/fce/10.0.023/lib/libimf.so: > warning: warning: feupdateenv is not implemented and will always fail > > I recognize the last part of the error message. I need to use the option > -nochoicemod while compiling an mpi code with mpif90 to avoid this error. Is > there any way for me to pass that option to the petsc installer? I tried > --with-fc=mpif90 -nochoicemod > in the install shell script, but that did not seem to work. -with-fc="mpif90 -nochoicemod" should work. Alternatively you can use FFLAGS=-nochoicemod Satish From saswata at umd.edu Sat Nov 1 16:00:35 2008 From: saswata at umd.edu (Saswata Hier-Majumder) Date: Sat, 01 Nov 2008 17:00:35 -0400 Subject: Running petsc on infiniband In-Reply-To: References: <490C8A07.9020309@umd.edu> <490C9DF2.7080301@umd.edu> Message-ID: <490CC373.1050604@umd.edu> Thanks. Using FFLAGS=-nochoicemod seemed to take care of most of the error messages in the configure.log file, but the original problem still persists. Satish Balay wrote: > On Sat, 1 Nov 2008, Saswata Hier-Majumder wrote: > > >> Satish, >> Thanks for the prompt response! >> >> After rerunning the installation script with the correct path, I found a bunch >> of error messages in the configure.log file which look like >> >> sh: /opt/vltmpi/OPENIB/mpi.icc.rsh/bin/mpif90 -o conftest conftest.o >> Executing: /opt/vltmpi/OPENIB/mpi.icc.rsh/bin/mpif90 -o conftest conftest.o >> sh: >> Possible ERROR while running linker: /opt/intel/fce/10.0.023/lib/libimf.so: >> warning: warning: feupdateenv is not implemented and will always fail >> >> I recognize the last part of the error message. I need to use the option >> -nochoicemod while compiling an mpi code with mpif90 to avoid this error. Is >> there any way for me to pass that option to the petsc installer? I tried >> --with-fc=mpif90 -nochoicemod >> in the install shell script, but that did not seem to work. >> > > -with-fc="mpif90 -nochoicemod" should work. Alternatively you can use FFLAGS=-nochoicemod > > Satish > > > > > -- www.geol.umd.edu/~saswata From knepley at gmail.com Sat Nov 1 16:11:04 2008 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 1 Nov 2008 16:11:04 -0500 Subject: Running petsc on infiniband In-Reply-To: <490CC373.1050604@umd.edu> References: <490C8A07.9020309@umd.edu> <490C9DF2.7080301@umd.edu> <490CC373.1050604@umd.edu> Message-ID: Send the configure.log to petsc-maint at mcs.anl.gov. There is obviously a mismatch between the MPI libraries being linked and the mpirun being used. Matt On Sat, Nov 1, 2008 at 4:00 PM, Saswata Hier-Majumder wrote: > Thanks. Using FFLAGS=-nochoicemod seemed to take care of most of the error > messages in the configure.log file, but the original problem still persists. > > Satish Balay wrote: >> >> On Sat, 1 Nov 2008, Saswata Hier-Majumder wrote: >> >> >>> >>> Satish, >>> Thanks for the prompt response! >>> >>> After rerunning the installation script with the correct path, I found a >>> bunch >>> of error messages in the configure.log file which look like >>> >>> sh: /opt/vltmpi/OPENIB/mpi.icc.rsh/bin/mpif90 -o conftest conftest.o >>> Executing: /opt/vltmpi/OPENIB/mpi.icc.rsh/bin/mpif90 -o conftest >>> conftest.o >>> sh: >>> Possible ERROR while running linker: >>> /opt/intel/fce/10.0.023/lib/libimf.so: >>> warning: warning: feupdateenv is not implemented and will always fail >>> >>> I recognize the last part of the error message. I need to use the option >>> -nochoicemod while compiling an mpi code with mpif90 to avoid this error. >>> Is >>> there any way for me to pass that option to the petsc installer? I tried >>> --with-fc=mpif90 -nochoicemod >>> in the install shell script, but that did not seem to work. >>> >> >> -with-fc="mpif90 -nochoicemod" should work. Alternatively you can use >> FFLAGS=-nochoicemod >> >> Satish >> >> >> >> >> > > -- > www.geol.umd.edu/~saswata > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From w_subber at yahoo.com Sun Nov 2 15:25:46 2008 From: w_subber at yahoo.com (Waad Subber) Date: Sun, 2 Nov 2008 13:25:46 -0800 (PST) Subject: hello Message-ID: <95253.45321.qm@web38202.mail.mud.yahoo.com> Hi friends: ?? We are wholesale company which can offer you laptops, digital cameras, videos, GPS?cell phone, mp4, game console and many other electron products. We can offer you both highest quality products and best price. Also we could give you favorable discount if you order more. All of our products are brand new and original; if you need any help, please contact us. Site:???? www.okvqq.com MSN:????? qqokv at hotmail.com? /? qqokv at yahoo.cn E-mail:?? qqokv at yahoo.cn Telphone: 0086-01083031356 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mafunk at nmsu.edu Tue Nov 4 13:33:11 2008 From: mafunk at nmsu.edu (Matt Funk) Date: Tue, 4 Nov 2008 12:33:11 -0700 Subject: analyze preconditioned operator? In-Reply-To: References: <200810091223.00078.mafunk@nmsu.edu> <200810101411.02514.mafunk@nmsu.edu> Message-ID: <200811041233.12030.mafunk@nmsu.edu> Hi, first of all: sorry to be picking this back up so late. (If you guys cannot answer because of that i understand. I was busy with other stuff and am now just returning to this ... ) As was described earlier (see below) i create a Shell Matrix and registered the mulitplying fcn with it via MatSetShellOperation(...). Regarding the myApply(...) fcn (which provides the custom multiply) I looked at the Fortran example for using MattSetShellOperations at: http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/src/ksp/ksp/examples/tutorials/ex14f.F.html Question #1: In it (=>the fortran code) the MatGetContext fcn (which was included in the myApply fcn (see below)) is never used. So i took it out as well. Things appear to work (at least nothing crashes and i get an answer), so am i still missing something? (I simply do this: 1) ORIGINAL_MATRIX*VECTOR_X => WORKSPACE 2) PCAPPLY(PC, WORKSPACE, VECTOR_Y) where ORIGINAL_MATRIX is a pointer to the original matrix. This pointer is stored in a global structure that can be accessed in the myApply fcn) I guess i never understood why it (=>MatGetContext fcn) was used anyway. The main point of the myApply() fcn below is to return the vector y, correct? So what is the purpose of the MatGetContext fcn? I'd like to understand it, but if it is too complicated to explain, then don't worry about it. One sanity check i did to see whether things make sense was to replace: ... MatMult(ctx->M, x, ctx->work); PCApply(ctx->pc, ctx->work, y); ... in the myApply fcn with simply: ... MatMult(ctx->M, x, y); ... which should then return the same answer as if i had just used my original matrix in the first place: This checks out. So things look ok, however, just out of curiosity i have another question: Question #2 : I registered the MULT operation with the shell matrix. However, since there are many other operations defined in petscmat.h and i don't know whether SLEPC uses those or not, what happens if were to to use another operation? Will it fall back to the original operator since it is not defined for the new (shell) matrix? Will it crash? What is the behavior going to be? thanks for all the help matt On Friday 10 October 2008, Matthew Knepley wrote: > On Fri, Oct 10, 2008 at 3:11 PM, Matt Funk wrote: > > Ok, > > > > thanks, i think i finally got the main idea (the emphasize here being on > > 'i think' which might not mean much ...) > > > > One other question though: > > What do i declare ctx to be: PetscObject * ctx ? > > No, you declare a struct which holds > > a) The original matrix > > b) The preconditioner > > c) A work vector > > or you would not be able to code the apply method. > > > Another question related to KSPComputeExplicitOperator: > > > > I was trying to use KSPComputeExplicitOperator and i am having issues > > with it as well. My matrix is a sparse matrix of length ((2*73)^3) with a > > maximum of ~30 entries per row. > > This is way way way too big to form the operator explicitly. > > > I believe that in order to use KSPComputeExplicitOperator i need to set > > up a matrix, allocate the memory for it and then pass it to the routine. > > From what i gather on the reference page this matrix needs to be of type > > MatDense? (if so then i cannot use KSPComputeExplicitOperator due to the > > size of the resulting matrix) > > However, it said on the website that when multiple procs are used it uses > > a sparse format. > > So why cannot i not use a sparse format in serial? > > 1) This operator is in general not sparse at all > > 2) We only use a sparse format in parllel because matMPIDense was > having problems > > It is instructive to look at what the code is doing. The function is > very short and you > can get to it right from the link on the manpage. > > Matt > > > thanks > > matt > > > > On Thursday 09 October 2008, Matthew Knepley wrote: > >> On Thu, Oct 9, 2008 at 5:58 PM, Matt Funk wrote: > >> > So then, > >> > > >> > is it then correct to say that this code "registers" this extra > >> > operation (i.e. applying the preconditioner) with the matrix context > >> > such that whenever a matrix operation involving this matrix is invoked > >> > (like MatMult for example) the pc-applying fcn (i.e. myApply() ) is > >> > called first? > >> > >> The idea here is to replace a given matrix A, which you are passing to > >> SLEPc, with another matrix M, which is a shell matrix. When MatMult() is > >> called on M, we call MatMult on A, and then PCApply on the result. > >> > >> Matt > >> > >> > matt > >> > > >> > ps: sorry for being a little slow on the uptake here ... > >> > > >> > On Thursday 09 October 2008, Matthew Knepley wrote: > >> >> On Thu, Oct 9, 2008 at 5:25 PM, Matt Funk wrote: > >> >> > Hi Matt, > >> >> > > >> >> > so, the basic idea with this code is to apply the pc to each column > >> >> > vector of the matrix? Is that right? > >> >> > >> >> No, that is what KSPGetExplicitOperator() does. This applies the > >> >> operator in a matrix-free way. > >> >> > >> >> > Also, in your example: when is myApply actually invoked? I also > >> >> > looked at the example listed under the MatShellSetOperation > >> >> > reference page. > >> >> > >> >> When MatMult(A) is called. > >> >> > >> >> Matt > >> >> > >> >> > Is the function then actually internally called when > >> >> > MatShellSetOperation is called, or when KSPSetOperators is called > >> >> > or KSPSolve? > >> >> > > >> >> > The reason i am asking is that if it is called when KSPSolve called > >> >> > then there is a problem because for the analysis i never call > >> >> > KSPSolve directly. > >> >> > > >> >> > thanks > >> >> > matt > >> >> > > >> >> > On Thursday 09 October 2008, Matthew Knepley wrote: > >> >> >> On Thu, Oct 9, 2008 at 4:32 PM, Matt Funk wrote: > >> >> >> > mmhh, > >> >> >> > > >> >> >> > i think i am missing something. Doesn't PCApply() apply the > >> >> >> > preconditioner to a vector? So how would that work (easily) with > >> >> >> > a matrix? > >> >> >> > >> >> >> You do not apply it to the matrix. Here is a skeleton (maybe has > >> >> >> mistakes) > >> >> >> > >> >> >> void myApply(Mat A, Vec x, Vec y) { > >> >> >> MatShellGetContext(A, &ctx); > >> >> >> MatMult(ctx->M, x, ctx->work); > >> >> >> PCApply(ctx->pc, ctx->work, y); > >> >> >> } > >> >> >> > >> >> >> MatShellSetOperation(A, MATOP_MULT, myApply) > >> >> >> > >> >> >> Matt > >> >> >> > >> >> >> > matt > >> >> >> > > >> >> >> > On Thursday 09 October 2008, Matthew Knepley wrote: > >> >> >> >> CApply(). From knepley at gmail.com Tue Nov 4 13:41:45 2008 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 4 Nov 2008 14:41:45 -0500 Subject: analyze preconditioned operator? In-Reply-To: <200811041233.12030.mafunk@nmsu.edu> References: <200810091223.00078.mafunk@nmsu.edu> <200810101411.02514.mafunk@nmsu.edu> <200811041233.12030.mafunk@nmsu.edu> Message-ID: On Tue, Nov 4, 2008 at 2:33 PM, Matt Funk wrote: > Hi, > > first of all: sorry to be picking this back up so late. > (If you guys cannot answer because of that i understand. I was busy with other > stuff and am now just returning to this ... ) > > As was described earlier (see below) i create a Shell Matrix and registered > the mulitplying fcn with it via MatSetShellOperation(...). > > Regarding the myApply(...) fcn (which provides the custom multiply) > I looked at the Fortran example for using MattSetShellOperations at: > http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/src/ksp/ksp/examples/tutorials/ex14f.F.html > > > Question #1: > > In it (=>the fortran code) the MatGetContext fcn (which was included in the > myApply fcn (see below)) is never used. So i took it out as well. Things > appear to work (at least nothing crashes and i get an answer), so am i still > missing something? > > (I simply do this: > 1) ORIGINAL_MATRIX*VECTOR_X => WORKSPACE > 2) PCAPPLY(PC, WORKSPACE, VECTOR_Y) > > where ORIGINAL_MATRIX is a pointer to the original matrix. This pointer is > stored in a global structure that can be accessed in the myApply fcn) > > I guess i never understood why it (=>MatGetContext fcn) was used anyway. The > main point of the myApply() fcn below is to return the vector y, correct? So > what is the purpose of the MatGetContext fcn? > I'd like to understand it, but if it is too complicated to explain, then don't > worry about it. Simple to explain. You get the ORIGINAL_MATRIX from a global location. If you can't or won't use a global variable, then you need the context. > One sanity check i did to see whether things make sense was to replace: > > ... > MatMult(ctx->M, x, ctx->work); > PCApply(ctx->pc, ctx->work, y); > ... > > in the myApply fcn with simply: > > ... > MatMult(ctx->M, x, y); > ... > > which should then return the same answer as if i had just used my original > matrix in the first place: This checks out. > > So things look ok, however, just out of curiosity i have another question: > > > Question #2 : > > I registered the MULT operation with the shell matrix. However, since there > are many other operations defined in petscmat.h and i don't know whether > SLEPC uses those or not, what happens if were to to use another operation? > > Will it fall back to the original operator since it is not defined for the new > (shell) matrix? Will it crash? > What is the behavior going to be? It will raise a PetscError, which SLEPc should propagate back to you. Matt > thanks for all the help > matt > > > > On Friday 10 October 2008, Matthew Knepley wrote: >> On Fri, Oct 10, 2008 at 3:11 PM, Matt Funk wrote: >> > Ok, >> > >> > thanks, i think i finally got the main idea (the emphasize here being on >> > 'i think' which might not mean much ...) >> > >> > One other question though: >> > What do i declare ctx to be: PetscObject * ctx ? >> >> No, you declare a struct which holds >> >> a) The original matrix >> >> b) The preconditioner >> >> c) A work vector >> >> or you would not be able to code the apply method. >> >> > Another question related to KSPComputeExplicitOperator: >> > >> > I was trying to use KSPComputeExplicitOperator and i am having issues >> > with it as well. My matrix is a sparse matrix of length ((2*73)^3) with a >> > maximum of ~30 entries per row. >> >> This is way way way too big to form the operator explicitly. >> >> > I believe that in order to use KSPComputeExplicitOperator i need to set >> > up a matrix, allocate the memory for it and then pass it to the routine. >> > From what i gather on the reference page this matrix needs to be of type >> > MatDense? (if so then i cannot use KSPComputeExplicitOperator due to the >> > size of the resulting matrix) >> > However, it said on the website that when multiple procs are used it uses >> > a sparse format. >> > So why cannot i not use a sparse format in serial? >> >> 1) This operator is in general not sparse at all >> >> 2) We only use a sparse format in parllel because matMPIDense was >> having problems >> >> It is instructive to look at what the code is doing. The function is >> very short and you >> can get to it right from the link on the manpage. >> >> Matt >> >> > thanks >> > matt >> > >> > On Thursday 09 October 2008, Matthew Knepley wrote: >> >> On Thu, Oct 9, 2008 at 5:58 PM, Matt Funk wrote: >> >> > So then, >> >> > >> >> > is it then correct to say that this code "registers" this extra >> >> > operation (i.e. applying the preconditioner) with the matrix context >> >> > such that whenever a matrix operation involving this matrix is invoked >> >> > (like MatMult for example) the pc-applying fcn (i.e. myApply() ) is >> >> > called first? >> >> >> >> The idea here is to replace a given matrix A, which you are passing to >> >> SLEPc, with another matrix M, which is a shell matrix. When MatMult() is >> >> called on M, we call MatMult on A, and then PCApply on the result. >> >> >> >> Matt >> >> >> >> > matt >> >> > >> >> > ps: sorry for being a little slow on the uptake here ... >> >> > >> >> > On Thursday 09 October 2008, Matthew Knepley wrote: >> >> >> On Thu, Oct 9, 2008 at 5:25 PM, Matt Funk wrote: >> >> >> > Hi Matt, >> >> >> > >> >> >> > so, the basic idea with this code is to apply the pc to each column >> >> >> > vector of the matrix? Is that right? >> >> >> >> >> >> No, that is what KSPGetExplicitOperator() does. This applies the >> >> >> operator in a matrix-free way. >> >> >> >> >> >> > Also, in your example: when is myApply actually invoked? I also >> >> >> > looked at the example listed under the MatShellSetOperation >> >> >> > reference page. >> >> >> >> >> >> When MatMult(A) is called. >> >> >> >> >> >> Matt >> >> >> >> >> >> > Is the function then actually internally called when >> >> >> > MatShellSetOperation is called, or when KSPSetOperators is called >> >> >> > or KSPSolve? >> >> >> > >> >> >> > The reason i am asking is that if it is called when KSPSolve called >> >> >> > then there is a problem because for the analysis i never call >> >> >> > KSPSolve directly. >> >> >> > >> >> >> > thanks >> >> >> > matt >> >> >> > >> >> >> > On Thursday 09 October 2008, Matthew Knepley wrote: >> >> >> >> On Thu, Oct 9, 2008 at 4:32 PM, Matt Funk wrote: >> >> >> >> > mmhh, >> >> >> >> > >> >> >> >> > i think i am missing something. Doesn't PCApply() apply the >> >> >> >> > preconditioner to a vector? So how would that work (easily) with >> >> >> >> > a matrix? >> >> >> >> >> >> >> >> You do not apply it to the matrix. Here is a skeleton (maybe has >> >> >> >> mistakes) >> >> >> >> >> >> >> >> void myApply(Mat A, Vec x, Vec y) { >> >> >> >> MatShellGetContext(A, &ctx); >> >> >> >> MatMult(ctx->M, x, ctx->work); >> >> >> >> PCApply(ctx->pc, ctx->work, y); >> >> >> >> } >> >> >> >> >> >> >> >> MatShellSetOperation(A, MATOP_MULT, myApply) >> >> >> >> >> >> >> >> Matt >> >> >> >> >> >> >> >> > matt >> >> >> >> > >> >> >> >> > On Thursday 09 October 2008, Matthew Knepley wrote: >> >> >> >> >> CApply(). > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From mafunk at nmsu.edu Tue Nov 4 16:19:02 2008 From: mafunk at nmsu.edu (Matt Funk) Date: Tue, 4 Nov 2008 15:19:02 -0700 Subject: HYPRE 2.4 In-Reply-To: References: <200810091223.00078.mafunk@nmsu.edu> <200811041233.12030.mafunk@nmsu.edu> Message-ID: <200811041519.02316.mafunk@nmsu.edu> Hi, are there any plans to incorporate hypre-2.4 with PETSC instead of hypre-2.0 sometime soon? matt From bsmith at mcs.anl.gov Thu Nov 6 13:30:05 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 6 Nov 2008 13:30:05 -0600 Subject: HYPRE 2.4 In-Reply-To: <200811041519.02316.mafunk@nmsu.edu> References: <200810091223.00078.mafunk@nmsu.edu> <200811041233.12030.mafunk@nmsu.edu> <200811041519.02316.mafunk@nmsu.edu> Message-ID: <02752C4F-B344-490E-9D64-D8F0239A23FC@mcs.anl.gov> Matt, Satish has switched petsc-dev http://www-unix.mcs.anl.gov/petsc/petsc-as/developers/index.html to use hypre 2.4. Please let us know if you have difficulties. Barry It is not possible to change petsc-2.3.3 to use hypre 2.4 On Nov 4, 2008, at 4:19 PM, Matt Funk wrote: > Hi, > > are there any plans to incorporate hypre-2.4 with PETSC instead of > hypre-2.0 > sometime soon? > > matt > From balay at mcs.anl.gov Thu Nov 6 13:40:18 2008 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 6 Nov 2008 13:40:18 -0600 (CST) Subject: HYPRE 2.4 In-Reply-To: <02752C4F-B344-490E-9D64-D8F0239A23FC@mcs.anl.gov> References: <200810091223.00078.mafunk@nmsu.edu> <200811041233.12030.mafunk@nmsu.edu> <200811041519.02316.mafunk@nmsu.edu> <02752C4F-B344-490E-9D64-D8F0239A23FC@mcs.anl.gov> Message-ID: On Thu, 6 Nov 2008, Barry Smith wrote: > It is not possible to change petsc-2.3.3 to use hypre 2.4 Actually - you might be able to just do the following with petsc-2.3.3 and it should work.. wget ftp://ftp.mcs.anl.gov/pub/petsc/externalpackages/hypre-2.4.0b.tar.gz ./config/configure.py --download-hypre=hypre-2.4.0b.tar.gz Satish From griffith at cims.nyu.edu Fri Nov 7 12:06:21 2008 From: griffith at cims.nyu.edu (Boyce Griffith) Date: Fri, 07 Nov 2008 13:06:21 -0500 Subject: changing default values in options database Message-ID: <4914839D.9050209@cims.nyu.edu> Hi, Folks -- I'm using some KSP solvers in a preconditioner (using FGMRES as the "outer" Krylov solver). For these preconditioner KSPs, I generally want to use relatively loose relative convergence tolerances. Presently, I check to see if, e.g., -solver1_ksp_rtol, -solver2_ksp_atol, -solver3_max_it, etc., are set in the options database. If a particular value is not set in the options database, I manually set that value to some value which is different from the default PETSc value. Of course, when I use the `-help' flag, the help for each of the solvers indicates the default values for rtol, atol, max_it, etc. are the standard PETSc defaults. Is there a simple way to change these default values for individual solvers? In particular, I would like to be able to ensure that the default values listed by `-help' correspond to the values that I'm specifying in the code. Thanks, -- Boyce From knepley at gmail.com Fri Nov 7 12:12:52 2008 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 7 Nov 2008 13:12:52 -0500 Subject: changing default values in options database In-Reply-To: <4914839D.9050209@cims.nyu.edu> References: <4914839D.9050209@cims.nyu.edu> Message-ID: I think you can just set those values before you call SetFromOptions() and they will show up. However, I have not tried it. Matt On Fri, Nov 7, 2008 at 1:06 PM, Boyce Griffith wrote: > Hi, Folks -- > > I'm using some KSP solvers in a preconditioner (using FGMRES as the "outer" > Krylov solver). For these preconditioner KSPs, I generally want to use > relatively loose relative convergence tolerances. > > Presently, I check to see if, e.g., -solver1_ksp_rtol, -solver2_ksp_atol, > -solver3_max_it, etc., are set in the options database. If a particular > value is not set in the options database, I manually set that value to some > value which is different from the default PETSc value. Of course, when I > use the `-help' flag, the help for each of the solvers indicates the default > values for rtol, atol, max_it, etc. are the standard PETSc defaults. > > Is there a simple way to change these default values for individual solvers? > In particular, I would like to be able to ensure that the default values > listed by `-help' correspond to the values that I'm specifying in the code. > > Thanks, > > -- Boyce > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From dalcinl at gmail.com Fri Nov 7 12:32:28 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 7 Nov 2008 15:32:28 -0300 Subject: changing default values in options database In-Reply-To: References: <4914839D.9050209@cims.nyu.edu> Message-ID: On Fri, Nov 7, 2008 at 3:12 PM, Matthew Knepley wrote: > I think you can just set those values before you call SetFromOptions() > and they will show up. However, I have not tried it. Yep, that's the way options work in most cases. > On Fri, Nov 7, 2008 at 1:06 PM, Boyce Griffith wrote: >> Hi, Folks -- >> >> Is there a simple way to change these default values for individual solvers? >> In particular, I would like to be able to ensure that the default values >> listed by `-help' correspond to the values that I'm specifying in the code. >> As Matt, suggested, use this KSPSetTolerances(ksp, ....) KSPSetFromOptions(ksp) -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From griffith at cims.nyu.edu Fri Nov 7 12:41:09 2008 From: griffith at cims.nyu.edu (Boyce Griffith) Date: Fri, 07 Nov 2008 13:41:09 -0500 Subject: changing default values in options database In-Reply-To: References: <4914839D.9050209@cims.nyu.edu> Message-ID: <49148BC5.4020900@cims.nyu.edu> Yes, you're right --- I was being stupid and was setting things in the wrong order. A related question: Is there any way to change the list of PC's associated with a solver as listed by -help? For my solver, it only makes sense for -pc_type to be "shell" or "none". Thanks, -- Boyce Lisandro Dalcin wrote: > On Fri, Nov 7, 2008 at 3:12 PM, Matthew Knepley wrote: >> I think you can just set those values before you call SetFromOptions() >> and they will show up. However, I have not tried it. > > Yep, that's the way options work in most cases. > >> On Fri, Nov 7, 2008 at 1:06 PM, Boyce Griffith wrote: >>> Hi, Folks -- >>> >>> Is there a simple way to change these default values for individual solvers? >>> In particular, I would like to be able to ensure that the default values >>> listed by `-help' correspond to the values that I'm specifying in the code. >>> > > As Matt, suggested, use this > > KSPSetTolerances(ksp, ....) > KSPSetFromOptions(ksp) > > From knepley at gmail.com Fri Nov 7 12:52:01 2008 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 7 Nov 2008 13:52:01 -0500 Subject: changing default values in options database In-Reply-To: <49148BC5.4020900@cims.nyu.edu> References: <4914839D.9050209@cims.nyu.edu> <49148BC5.4020900@cims.nyu.edu> Message-ID: On Fri, Nov 7, 2008 at 1:41 PM, Boyce Griffith wrote > Yes, you're right --- I was being stupid and was setting things in the wrong > order. > > A related question: Is there any way to change the list of PC's associated > with a solver as listed by -help? For my solver, it only makes sense for > -pc_type to be "shell" or "none". There is no way to do this, since these are "generic" KSP objects. Matt > Thanks, > > -- Boyce > > Lisandro Dalcin wrote: >> >> On Fri, Nov 7, 2008 at 3:12 PM, Matthew Knepley wrote: >>> >>> I think you can just set those values before you call SetFromOptions() >>> and they will show up. However, I have not tried it. >> >> Yep, that's the way options work in most cases. >> >>> On Fri, Nov 7, 2008 at 1:06 PM, Boyce Griffith >>> wrote: >>>> >>>> Hi, Folks -- >>>> >>>> Is there a simple way to change these default values for individual >>>> solvers? >>>> In particular, I would like to be able to ensure that the default >>>> values >>>> listed by `-help' correspond to the values that I'm specifying in the >>>> code. >>>> >> >> As Matt, suggested, use this >> >> KSPSetTolerances(ksp, ....) >> KSPSetFromOptions(ksp) >> >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From mafunk at nmsu.edu Fri Nov 7 18:55:09 2008 From: mafunk at nmsu.edu (Matt Funk) Date: Fri, 7 Nov 2008 17:55:09 -0700 Subject: HYPRE 2.4 In-Reply-To: References: <200810091223.00078.mafunk@nmsu.edu> <02752C4F-B344-490E-9D64-D8F0239A23FC@mcs.anl.gov> Message-ID: <200811071755.09373.mafunk@nmsu.edu> Thanks, that seems to work. matt On Thursday 06 November 2008, you wrote: > On Thu, 6 Nov 2008, Barry Smith wrote: > > It is not possible to change petsc-2.3.3 to use hypre 2.4 > > Actually - you might be able to just do the following with petsc-2.3.3 > and it should work.. > > wget ftp://ftp.mcs.anl.gov/pub/petsc/externalpackages/hypre-2.4.0b.tar.gz > ./config/configure.py --download-hypre=hypre-2.4.0b.tar.gz > > Satish From schuang at ats.ucla.edu Tue Nov 11 18:16:48 2008 From: schuang at ats.ucla.edu (Shao-Ching Huang) Date: Tue, 11 Nov 2008 16:16:48 -0800 Subject: ksp_monitor and hypre/boomeramg Message-ID: <20081112001648.GA18040@ats.ucla.edu> Hi, I am using PETSc-2.3.3p15, compiled with Hypre 2.4.0b, to solve a Poisson equation (100x100 mesh) using Hypre's boomeramg. The same matrix equation has been solved by PETSc's CG or GMRES successfully. When I run with "-ksp_monitor" (see below), the iterations show convergence and everything seems fine. When I remove the "ksp_monitor" flag (keeping all other flags the same), it takes relatively long time to finish. It appears to me (just a guess) that the KSP runs through the default ksp_max_it=10000 iterations without looking at the residual level (even when ksp_rtol=1.e-5). My question is: why does "-ksp_monitor" change the convergence criteria/behavior? Or am I doing something wrong? Some more details are attached below. Thanks, Shao-Ching --------------------------------- In the following, "solve time" is in seconds. #### with "-ksp_monitor" $ mpiexec -np 1 -machinefile mach ./ibm -ksp_type richardson \ -pc_type hypre -pc_hypre_type boomeramg -ksp_monitor |grep poisson_solve [poisson_solve] solve time = 0.043164 The convergence looks like: 0 KSP Residual norm 1.263975733203e-01 1 KSP Residual norm 4.825832685316e-03 2 KSP Residual norm 1.980816153418e-04 3 KSP Residual norm 9.187303727632e-06 4 KSP Residual norm 5.022156966290e-07 #### without "-ksp_monitor" $ mpiexec -np 1 -machinefile mach ./ibm -ksp_type richardson \ -pc_type hypre -pc_hypre_type boomeramg |grep poisson_solve [poisson_solve] solve time = 66.463 #### without "-ksp_monitor" but set "-ksp_max_it 10" $ mpiexec -np 1 -machinefile mach ./ibm -ksp_type richardson \ -pc_type hypre -pc_hypre_type boomeramg -ksp_max_it 10 |grep poisson_solve [poisson_solve] solve time = 0.0682499 -------------------------------------- ksp_view: KSP Object: type: richardson Richardson: damping factor=1 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning PC Object: type: hypre HYPRE BoomerAMG preconditioning HYPRE BoomerAMG: Cycle type V HYPRE BoomerAMG: Maximum number of levels 25 HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1 HYPRE BoomerAMG: Convergence tolerance PER hypre call 0 HYPRE BoomerAMG: Threshold for strong coupling 0.25 HYPRE BoomerAMG: Interpolation truncation factor 0 HYPRE BoomerAMG: Interpolation: max elements per row 0 HYPRE BoomerAMG: Number of levels of aggressive coarsening 0 HYPRE BoomerAMG: Number of paths for aggressive coarsening 1 HYPRE BoomerAMG: Maximum row sums 0.9 HYPRE BoomerAMG: Sweeps down 1 HYPRE BoomerAMG: Sweeps up 1 HYPRE BoomerAMG: Sweeps on coarse 1 HYPRE BoomerAMG: Relax down symmetric-SOR/Jacobi HYPRE BoomerAMG: Relax up symmetric-SOR/Jacobi HYPRE BoomerAMG: Relax on coarse Gaussian-elimination HYPRE BoomerAMG: Relax weight (all) 1 HYPRE BoomerAMG: Outer relax weight (all) 1 HYPRE BoomerAMG: Using CF-relaxation HYPRE BoomerAMG: Measure type local HYPRE BoomerAMG: Coarsen type Falgout HYPRE BoomerAMG: Interpolation type classical linear system matrix = precond matrix: Matrix Object: type=mpiaij, rows=10000, cols=10000 total: nonzeros=90000, allocated nonzeros=90000 not using I-node (on process 0) routines -------------------------------------- $ uname -a Linux n125 2.6.18-92.1.6.el5 #1 SMP Wed Jun 25 13:45:47 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux From knepley at gmail.com Tue Nov 11 19:45:11 2008 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 11 Nov 2008 19:45:11 -0600 Subject: ksp_monitor and hypre/boomeramg In-Reply-To: <20081112001648.GA18040@ats.ucla.edu> References: <20081112001648.GA18040@ats.ucla.edu> Message-ID: On Tue, Nov 11, 2008 at 6:16 PM, Shao-Ching Huang wrote: > Hi, > > I am using PETSc-2.3.3p15, compiled with Hypre 2.4.0b, to solve a > Poisson equation (100x100 mesh) using Hypre's boomeramg. The same > matrix equation has been solved by PETSc's CG or GMRES successfully. > > When I run with "-ksp_monitor" (see below), the iterations show > convergence and everything seems fine. > > When I remove the "ksp_monitor" flag (keeping all other flags the > same), it takes relatively long time to finish. It appears to me (just > a guess) that the KSP runs through the default ksp_max_it=10000 > iterations without looking at the residual level (even when > ksp_rtol=1.e-5). > > My question is: why does "-ksp_monitor" change the convergence > criteria/behavior? Or am I doing something wrong? Nothing should change. You can try -ksp_converged_reason to see why it terminates. The best way is to put a breakpoint in KSPDefaultConverged() and check the convergence test. Matt > Some more details are attached below. Thanks, > > Shao-Ching > > --------------------------------- > > In the following, "solve time" is in seconds. > > #### with "-ksp_monitor" > > $ mpiexec -np 1 -machinefile mach ./ibm -ksp_type richardson \ > -pc_type hypre -pc_hypre_type boomeramg -ksp_monitor |grep poisson_solve > > [poisson_solve] solve time = 0.043164 > > The convergence looks like: > > 0 KSP Residual norm 1.263975733203e-01 > 1 KSP Residual norm 4.825832685316e-03 > 2 KSP Residual norm 1.980816153418e-04 > 3 KSP Residual norm 9.187303727632e-06 > 4 KSP Residual norm 5.022156966290e-07 > > #### without "-ksp_monitor" > > $ mpiexec -np 1 -machinefile mach ./ibm -ksp_type richardson \ > -pc_type hypre -pc_hypre_type boomeramg |grep poisson_solve > > [poisson_solve] solve time = 66.463 > > > #### without "-ksp_monitor" but set "-ksp_max_it 10" > > $ mpiexec -np 1 -machinefile mach ./ibm -ksp_type richardson \ > -pc_type hypre -pc_hypre_type boomeramg -ksp_max_it 10 |grep poisson_solve > > [poisson_solve] solve time = 0.0682499 > > -------------------------------------- > > ksp_view: > > KSP Object: > type: richardson > Richardson: damping factor=1 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > PC Object: > type: hypre > HYPRE BoomerAMG preconditioning > HYPRE BoomerAMG: Cycle type V > HYPRE BoomerAMG: Maximum number of levels 25 > HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1 > HYPRE BoomerAMG: Convergence tolerance PER hypre call 0 > HYPRE BoomerAMG: Threshold for strong coupling 0.25 > HYPRE BoomerAMG: Interpolation truncation factor 0 > HYPRE BoomerAMG: Interpolation: max elements per row 0 > HYPRE BoomerAMG: Number of levels of aggressive coarsening 0 > HYPRE BoomerAMG: Number of paths for aggressive coarsening 1 > HYPRE BoomerAMG: Maximum row sums 0.9 > HYPRE BoomerAMG: Sweeps down 1 > HYPRE BoomerAMG: Sweeps up 1 > HYPRE BoomerAMG: Sweeps on coarse 1 > HYPRE BoomerAMG: Relax down symmetric-SOR/Jacobi > HYPRE BoomerAMG: Relax up symmetric-SOR/Jacobi > HYPRE BoomerAMG: Relax on coarse Gaussian-elimination > HYPRE BoomerAMG: Relax weight (all) 1 > HYPRE BoomerAMG: Outer relax weight (all) 1 > HYPRE BoomerAMG: Using CF-relaxation > HYPRE BoomerAMG: Measure type local > HYPRE BoomerAMG: Coarsen type Falgout > HYPRE BoomerAMG: Interpolation type classical > linear system matrix = precond matrix: > Matrix Object: > type=mpiaij, rows=10000, cols=10000 > total: nonzeros=90000, allocated nonzeros=90000 > not using I-node (on process 0) routines > > -------------------------------------- > > $ uname -a > Linux n125 2.6.18-92.1.6.el5 #1 SMP Wed Jun 25 13:45:47 EDT 2008 > x86_64 x86_64 x86_64 GNU/Linux > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From bsmith at mcs.anl.gov Tue Nov 11 21:26:27 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 11 Nov 2008 21:26:27 -0600 Subject: ksp_monitor and hypre/boomeramg In-Reply-To: <20081112001648.GA18040@ats.ucla.edu> References: <20081112001648.GA18040@ats.ucla.edu> Message-ID: <7E919A73-783E-49E4-A3D6-62FDD92E0E40@mcs.anl.gov> You are not doing anything wrong. If you run with the additional option -pc_hypre_boomeramg_tol where you use the same rtol here as you use for -ksp_rtol (default is 1.e-5) then you will get the behavior you expect. This is a "feature"/"design bug" in our 2.3.3 interface to HYPRE's BoomerAMG. It uses a default -pc_hypre_boomeramg_tol of 0.0 which causes it to run the BoomerAMG cycle for the full default 10,000 cycles when run with Richardson. In src/ksp/pc/impls/hypre/hypre.c line 608 you will find ierr = HYPRE_BoomerAMGSetTol(jac->hsolver,PetscMin(rtol,jac- >tol));CHKERRQ(ierr); it has been changed in petsc-dev to ierr = HYPRE_BoomerAMGSetTol(jac->hsolver,rtol);CHKERRQ(ierr); this causes it to only run BoomerAMG cycles until it gets below the KSP rtol. Barry Note: running Richardson with and without monitoring is different code for many preconditioners. This is because monitoring reguires the || residual|| at each iteration. Many Richardson application of preconditioners like SOR, and BoomerAMG never compute the residual hence cannot compute || residual||. These preconditioners have a separate PCApplyRichardson() method rather then just calling PCApply() and computing the residual at each iteration. On Nov 11, 2008, at 6:16 PM, Shao-Ching Huang wrote: > Hi, > > I am using PETSc-2.3.3p15, compiled with Hypre 2.4.0b, to solve a > Poisson equation (100x100 mesh) using Hypre's boomeramg. The same > matrix equation has been solved by PETSc's CG or GMRES successfully. > > When I run with "-ksp_monitor" (see below), the iterations show > convergence and everything seems fine. > > When I remove the "ksp_monitor" flag (keeping all other flags the > same), it takes relatively long time to finish. It appears to me (just > a guess) that the KSP runs through the default ksp_max_it=10000 > iterations without looking at the residual level (even when > ksp_rtol=1.e-5). > > My question is: why does "-ksp_monitor" change the convergence > criteria/behavior? Or am I doing something wrong? > > Some more details are attached below. Thanks, > > Shao-Ching > > --------------------------------- > > In the following, "solve time" is in seconds. > > #### with "-ksp_monitor" > > $ mpiexec -np 1 -machinefile mach ./ibm -ksp_type richardson \ > -pc_type hypre -pc_hypre_type boomeramg -ksp_monitor |grep > poisson_solve > > [poisson_solve] solve time = 0.043164 > > The convergence looks like: > > 0 KSP Residual norm 1.263975733203e-01 > 1 KSP Residual norm 4.825832685316e-03 > 2 KSP Residual norm 1.980816153418e-04 > 3 KSP Residual norm 9.187303727632e-06 > 4 KSP Residual norm 5.022156966290e-07 > > #### without "-ksp_monitor" > > $ mpiexec -np 1 -machinefile mach ./ibm -ksp_type richardson \ > -pc_type hypre -pc_hypre_type boomeramg |grep poisson_solve > > [poisson_solve] solve time = 66.463 > > > #### without "-ksp_monitor" but set "-ksp_max_it 10" > > $ mpiexec -np 1 -machinefile mach ./ibm -ksp_type richardson \ > -pc_type hypre -pc_hypre_type boomeramg -ksp_max_it 10 |grep > poisson_solve > > [poisson_solve] solve time = 0.0682499 > > -------------------------------------- > > ksp_view: > > KSP Object: > type: richardson > Richardson: damping factor=1 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > PC Object: > type: hypre > HYPRE BoomerAMG preconditioning > HYPRE BoomerAMG: Cycle type V > HYPRE BoomerAMG: Maximum number of levels 25 > HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1 > HYPRE BoomerAMG: Convergence tolerance PER hypre call 0 > HYPRE BoomerAMG: Threshold for strong coupling 0.25 > HYPRE BoomerAMG: Interpolation truncation factor 0 > HYPRE BoomerAMG: Interpolation: max elements per row 0 > HYPRE BoomerAMG: Number of levels of aggressive coarsening 0 > HYPRE BoomerAMG: Number of paths for aggressive coarsening 1 > HYPRE BoomerAMG: Maximum row sums 0.9 > HYPRE BoomerAMG: Sweeps down 1 > HYPRE BoomerAMG: Sweeps up 1 > HYPRE BoomerAMG: Sweeps on coarse 1 > HYPRE BoomerAMG: Relax down symmetric-SOR/Jacobi > HYPRE BoomerAMG: Relax up symmetric-SOR/Jacobi > HYPRE BoomerAMG: Relax on coarse Gaussian-elimination > HYPRE BoomerAMG: Relax weight (all) 1 > HYPRE BoomerAMG: Outer relax weight (all) 1 > HYPRE BoomerAMG: Using CF-relaxation > HYPRE BoomerAMG: Measure type local > HYPRE BoomerAMG: Coarsen type Falgout > HYPRE BoomerAMG: Interpolation type classical > linear system matrix = precond matrix: > Matrix Object: > type=mpiaij, rows=10000, cols=10000 > total: nonzeros=90000, allocated nonzeros=90000 > not using I-node (on process 0) routines > > -------------------------------------- > > $ uname -a > Linux n125 2.6.18-92.1.6.el5 #1 SMP Wed Jun 25 13:45:47 EDT 2008 > x86_64 x86_64 x86_64 GNU/Linux > > > > > From m.schauer at tu-bs.de Wed Nov 12 05:01:05 2008 From: m.schauer at tu-bs.de (Marco Schauer) Date: Wed, 12 Nov 2008 12:01:05 +0100 Subject: how to get bandwidth of a matrix Message-ID: <491AB771.9000704@tu-bs.de> Hello, I try to find out the bandwidth of my matrix. Currently I use MatInfo to get nonzero entities, but MatInfo returns only the allocated, used and unneeded entities of the whole matrix. Is it possible to get the nonzero entities for each row? Kind regards Marco From dave.mayhem23 at gmail.com Wed Nov 12 05:39:27 2008 From: dave.mayhem23 at gmail.com (Dave May) Date: Wed, 12 Nov 2008 22:39:27 +1100 Subject: how to get bandwidth of a matrix In-Reply-To: <491AB771.9000704@tu-bs.de> References: <491AB771.9000704@tu-bs.de> Message-ID: <956373f0811120339g49d78e6cmbbdf7428995cccc2@mail.gmail.com> Hey Marco, I think MatGetRow() should do what you are after. http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatGetRow.html Cheers, Dave On Wed, Nov 12, 2008 at 10:01 PM, Marco Schauer wrote: > Hello, > I try to find out the bandwidth of my matrix. Currently I use MatInfo to > get nonzero entities, but MatInfo returns only the allocated, used and > unneeded entities of the whole matrix. Is it possible to get the nonzero > entities for each row? > Kind regards > Marco > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.schauer at tu-bs.de Wed Nov 12 07:46:08 2008 From: m.schauer at tu-bs.de (Marco Schauer) Date: Wed, 12 Nov 2008 14:46:08 +0100 Subject: how to get bandwidth of a matrix In-Reply-To: <956373f0811120339g49d78e6cmbbdf7428995cccc2@mail.gmail.com> References: <491AB771.9000704@tu-bs.de> <956373f0811120339g49d78e6cmbbdf7428995cccc2@mail.gmail.com> Message-ID: <491ADE20.9070603@tu-bs.de> Hey Dave, It works! Thanks a lot. Regards, Marco Dave May schrieb: > Hey Marco, > I think MatGetRow() should do what you are after. > > http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatGetRow.html > > Cheers, > Dave > > > On Wed, Nov 12, 2008 at 10:01 PM, Marco Schauer > wrote: > > Hello, > I try to find out the bandwidth of my matrix. Currently I use > MatInfo to get nonzero entities, but MatInfo returns only the > allocated, used and unneeded entities of the whole matrix. Is it > possible to get the nonzero entities for each row? > Kind regards > Marco > > From schuang at ats.ucla.edu Wed Nov 12 11:44:21 2008 From: schuang at ats.ucla.edu (Shao-Ching Huang) Date: Wed, 12 Nov 2008 09:44:21 -0800 Subject: ksp_monitor and hypre/boomeramg In-Reply-To: References: <20081112001648.GA18040@ats.ucla.edu> Message-ID: <20081112174421.GA20260@ats.ucla.edu> Hi Matt: I tried "-ksp_converged_reason" you suggested. With "-ksp_monitor", it shows "due to CONVERGED_RTOL". Without "-ksp_monitor", it shows "due to CONVERGED_ITS". ------------------ $ mpiexec -np 1 -machinefile mach ./ibm -ksp_type richardson \ -pc_type hypre -pc_hypre_type boomeramg -ksp_monitor \ -ksp_converged_reason | grep Linear Linear solve converged due to CONVERGED_RTOL iterations 5 $ mpiexec -np 1 -machinefile mach ./ibm -ksp_type richardson \ -pc_type hypre -pc_hypre_type boomeramg \ -ksp_converged_reason | grep Linear Linear solve converged due to CONVERGED_ITS iterations 0 ------------------ Thanks, Shao-Ching On Tue, Nov 11, 2008 at 07:45:11PM -0600, Matthew Knepley wrote: > > Nothing should change. You can try -ksp_converged_reason to see why it > terminates. The best way is to put a breakpoint in KSPDefaultConverged() > and check the convergence test. > > Matt > From knepley at gmail.com Wed Nov 12 11:45:50 2008 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 12 Nov 2008 11:45:50 -0600 Subject: ksp_monitor and hypre/boomeramg In-Reply-To: <20081112174421.GA20260@ats.ucla.edu> References: <20081112001648.GA18040@ats.ucla.edu> <20081112174421.GA20260@ats.ucla.edu> Message-ID: Barry gave the correct answer after me. I was unaware that the behavior was different in the release. Thanks, Matt On Wed, Nov 12, 2008 at 11:44 AM, Shao-Ching Huang wrote: > Hi Matt: > > I tried "-ksp_converged_reason" you suggested. With "-ksp_monitor", it > shows "due to CONVERGED_RTOL". Without "-ksp_monitor", it shows "due > to CONVERGED_ITS". > > ------------------ > $ mpiexec -np 1 -machinefile mach ./ibm -ksp_type richardson \ > -pc_type hypre -pc_hypre_type boomeramg -ksp_monitor \ > -ksp_converged_reason | grep Linear > > Linear solve converged due to CONVERGED_RTOL iterations 5 > > $ mpiexec -np 1 -machinefile mach ./ibm -ksp_type richardson \ > -pc_type hypre -pc_hypre_type boomeramg \ > -ksp_converged_reason | grep Linear > > Linear solve converged due to CONVERGED_ITS iterations 0 > ------------------ > > Thanks, > > Shao-Ching > > > On Tue, Nov 11, 2008 at 07:45:11PM -0600, Matthew Knepley wrote: >> >> Nothing should change. You can try -ksp_converged_reason to see why it >> terminates. The best way is to put a breakpoint in KSPDefaultConverged() >> and check the convergence test. >> >> Matt >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From schuang at ats.ucla.edu Wed Nov 12 11:51:31 2008 From: schuang at ats.ucla.edu (Shao-Ching Huang) Date: Wed, 12 Nov 2008 09:51:31 -0800 Subject: ksp_monitor and hypre/boomeramg In-Reply-To: <7E919A73-783E-49E4-A3D6-62FDD92E0E40@mcs.anl.gov> References: <20081112001648.GA18040@ats.ucla.edu> <7E919A73-783E-49E4-A3D6-62FDD92E0E40@mcs.anl.gov> Message-ID: <20081112175131.GB20260@ats.ucla.edu> Hi Barry: Thank you for your comment. I will set -pc_hypre_boomeramg_tol explicitly for now. Shao-Ching On Tue, Nov 11, 2008 at 09:26:27PM -0600, Barry Smith wrote: > > You are not doing anything wrong. If you run with the additional > option -pc_hypre_boomeramg_tol where you use the same rtol > here as you use for -ksp_rtol (default is 1.e-5) then you will > get the behavior you expect. > > This is a "feature"/"design bug" in our 2.3.3 interface to HYPRE's > BoomerAMG. > It uses a default -pc_hypre_boomeramg_tol of 0.0 which causes it to > run the BoomerAMG cycle for the full default 10,000 cycles when run > with Richardson. > > In src/ksp/pc/impls/hypre/hypre.c line 608 you will find > ierr = HYPRE_BoomerAMGSetTol(jac->hsolver,PetscMin(rtol,jac- > >tol));CHKERRQ(ierr); > it has been changed in petsc-dev to > ierr = HYPRE_BoomerAMGSetTol(jac->hsolver,rtol);CHKERRQ(ierr); > this causes it to only run BoomerAMG cycles until it gets below the KSP > rtol. > > Barry > > Note: running Richardson with and without monitoring is different code > for many preconditioners. This is because monitoring reguires the || > residual|| > at each iteration. Many Richardson application of preconditioners like > SOR, > and BoomerAMG never compute the residual hence cannot compute || > residual||. > These preconditioners have a separate PCApplyRichardson() method rather > then just calling PCApply() and computing the residual at each > iteration. > > From schuang at ats.ucla.edu Wed Nov 12 18:24:23 2008 From: schuang at ats.ucla.edu (Shao-Ching Huang) Date: Wed, 12 Nov 2008 16:24:23 -0800 Subject: process id in DA Message-ID: <20081113002423.GA22482@ats.ucla.edu> Hi, Is there a PETSc call to obtain the "process coordinates" in a process grid, partitioned by a DA? (Something like what MPI_Cart_coords does.) Thanks, Shao-Ching From knepley at gmail.com Wed Nov 12 18:28:39 2008 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 12 Nov 2008 18:28:39 -0600 Subject: process id in DA In-Reply-To: <20081113002423.GA22482@ats.ucla.edu> References: <20081113002423.GA22482@ats.ucla.edu> Message-ID: On Wed, Nov 12, 2008 at 6:24 PM, Shao-Ching Huang wrote: > Hi, > > Is there a PETSc call to obtain the "process coordinates" in a process > grid, partitioned by a DA? (Something like what MPI_Cart_coords does.) No, I don't think it has what you want. Why do you want it? Thanks, Matt > Thanks, > > Shao-Ching -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From sdettrick at gmail.com Wed Nov 12 18:39:33 2008 From: sdettrick at gmail.com (Sean Dettrick) Date: Wed, 12 Nov 2008 16:39:33 -0800 Subject: process id in DA In-Reply-To: <20081113002423.GA22482@ats.ucla.edu> References: <20081113002423.GA22482@ats.ucla.edu> Message-ID: <7886389A-C722-4C9F-B1BC-92F99DAB4255@gmail.com> On Nov 12, 2008, at 4:24 PM, Shao-Ching Huang wrote: > Hi, > > Is there a PETSc call to obtain the "process coordinates" in a process > grid, partitioned by a DA? (Something like what MPI_Cart_coords does.) > > Thanks, > > Shao-Ching > A while ago (May 2006) I found that the PETSc DA has its processes ordered in the TRANSPOSE way to that of MPI Cartesian communicators, so you might have to consider mapping back and forth between the two cartesian representations. Barry said that he might change the DA so that the two agree with each other, but I don't know if he ever did... Does anybody know the status of that? In case it helps, below is part of our exchange on the matter. Sean --------------------------- Ok, so it is laid-out like a matrix, not like x, y coordinates. Will put the change to DA on the list of things to do. Barry On Thu, 18 May 2006, Sean Dettrick wrote: > Barry Smith wrote: > >> Bill, does the MPI standard dictate this decomposition or >> could different implementations do it the opposite way? >> Then we'd have to make the DA logic a bit more complicated. > > I don't have a copy of the standard, but to quote page 255 of "MPI, > the complete reference" by Snir et al: > "Row-major numbering is always used for the processes in a Cartesian > structure". > Their diagram in figure 6.1 matches my code output for coords > couplets (i,j): > > 0 1 2 3 > (0,0) (0,1) (0,2) (0,3) > > 4 5 6 7 > (1,0) (1,1) (1,2) (1,3) > > 8 9 10 11 > (2,0) (2,1) (2,2) (2,3) > > > By the way I agree with you, I *should* be able to swap the x and y > myself. Just haven't had much luck yet in that regard. > > Sean > >> On Thu, 18 May 2006, Sean Dettrick wrote: >>> Barry Smith wrote: >>>> On Thu, 18 May 2006, Sean Dettrick wrote: >>>>> Hi Barry, >>>>> the order is determined by MPI_Cart_create. >>>> >>>> Do you mean that MPI_Cart_create() orders across the 2nd (y-axis) >>>> fastest and then the first (x-axis)? Hmmm, maybe we should change >>>> the >>>> DA? Changing it once and for all (not supporting both) is probably >>>> not a big deal and shouldn't break much (I hope). >>> Hi Barry, >>> it depends, what do you call x and what do you call y? >>> MPI_Cart_coords returns a vector, coords - I tend to say x is >>> coords[0], y is coords[1] and z is coords[2]. For what it's worth, >>> there's a short code appended to this email, which produces: >>> rank = 0 has Cartesian coords = { 0, 0 } >>> rank = 1 has Cartesian coords = { 0, 1 } >>> rank = 2 has Cartesian coords = { 1, 0 } >>> rank = 3 has Cartesian coords = { 1, 1 } >>> rank = 0 has DA range x=[0,50) and y=[0,50) >>> rank = 1 has DA range x=[50,100) and y=[0,50) >>> rank = 2 has DA range x=[0,50) and y=[50,100) >>> rank = 3 has DA range x=[50,100) and y=[50,100) >>> >>>>>> I don't completely understand what goes wrong. Is it because >>>>>> YOUR >>>>>> application orders the processors related to geometry in the >>>>>> following way? >>>>>> >>>>>> ^ y direction >>>>>> | >>>>>> 2 5 8 >>>>>> 1 4 7 >>>>>> 0 3 6 >>>>>> >>>>>> -> x direction >>>>>> Or is this something inherent in MPI_Cart_create? >>> For my interpretation of x and y, MPI_Cart_create produces the >>> above layout. But if I said x=coords[1] and y=coords[0], then it >>> would match the one below. >>>>>> PETSc does it so >>>>>> >>>>>> ^ y direction >>>>>> | >>>>>> 6 7 8 >>>>>> 3 4 5 >>>>>> 0 1 2 >>>>>> >>>>>> -> x direction >>> Code and makefile attached ... hopefully within the message size >>> limit. >>> Just make cartcommtest. >>> Sean > > From schuang at ats.ucla.edu Wed Nov 12 19:01:56 2008 From: schuang at ats.ucla.edu (Shao-Ching Huang) Date: Wed, 12 Nov 2008 17:01:56 -0800 Subject: process id in DA In-Reply-To: References: <20081113002423.GA22482@ats.ucla.edu> Message-ID: <20081113010156.GA22563@ats.ucla.edu> On the first and last processes along each partitioned direction, I have some extra operations to do. Currently I test the global indices (using DAGetCorners) to achieve a similar effect, but was wondering if there is already some one-shot function call available. Thanks, Shao-Ching On Wed, Nov 12, 2008 at 06:28:39PM -0600, Matthew Knepley wrote: > On Wed, Nov 12, 2008 at 6:24 PM, Shao-Ching Huang wrote: > > Hi, > > > > Is there a PETSc call to obtain the "process coordinates" in a process > > grid, partitioned by a DA? (Something like what MPI_Cart_coords does.) > > No, I don't think it has what you want. Why do you want it? > > Thanks, > > Matt > > > Thanks, > > > > Shao-Ching From bsmith at mcs.anl.gov Wed Nov 12 19:15:22 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 12 Nov 2008 19:15:22 -0600 Subject: process id in DA In-Reply-To: <20081113002423.GA22482@ats.ucla.edu> References: <20081113002423.GA22482@ats.ucla.edu> Message-ID: <210E5146-802C-40B5-9736-E3D3B407E74A@mcs.anl.gov> Doesn't look like it: but from the formula rank = i + j*Mp + k*Mp*Np i = rank % Mp k = rank/(Mp*Np) j = (rank - k*Mp*Np)/Mp Barry Or something similar. Mp, Np are the number of processes in the x and y directions On Nov 12, 2008, at 6:24 PM, Shao-Ching Huang wrote: > Hi, > > Is there a PETSc call to obtain the "process coordinates" in a process > grid, partitioned by a DA? (Something like what MPI_Cart_coords does.) > > Thanks, > > Shao-Ching > From bsmith at mcs.anl.gov Wed Nov 12 19:15:27 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 12 Nov 2008 19:15:27 -0600 Subject: process id in DA In-Reply-To: <7886389A-C722-4C9F-B1BC-92F99DAB4255@gmail.com> References: <20081113002423.GA22482@ats.ucla.edu> <7886389A-C722-4C9F-B1BC-92F99DAB4255@gmail.com> Message-ID: On Nov 12, 2008, at 6:39 PM, Sean Dettrick wrote: > > On Nov 12, 2008, at 4:24 PM, Shao-Ching Huang wrote: > >> Hi, >> >> Is there a PETSc call to obtain the "process coordinates" in a >> process >> grid, partitioned by a DA? (Something like what MPI_Cart_coords >> does.) >> >> Thanks, >> >> Shao-Ching >> > > A while ago (May 2006) I found that the PETSc DA has its processes > ordered in the TRANSPOSE way to that of MPI Cartesian communicators, > so you might have to consider mapping back and forth between the two > cartesian representations. > > Barry said that he might change the DA so that the two agree with > each other, but I don't know if he ever did... I got lazy and never did it. In the abstract changing this is easy but getting every right requires care. It would be nice if someone did this. Barry > > > Does anybody know the status of that? > > In case it helps, below is part of our exchange on the matter. > > Sean > --------------------------- > > > > Ok, so it is laid-out like a matrix, not like x, y coordinates. > > Will put the change to DA on the list of things to do. > > > Barry > > On Thu, 18 May 2006, Sean Dettrick wrote: > >> Barry Smith wrote: >> >>> Bill, does the MPI standard dictate this decomposition or >>> could different implementations do it the opposite way? >>> Then we'd have to make the DA logic a bit more complicated. >> >> I don't have a copy of the standard, but to quote page 255 of "MPI, >> the complete reference" by Snir et al: >> "Row-major numbering is always used for the processes in a >> Cartesian structure". >> Their diagram in figure 6.1 matches my code output for coords >> couplets (i,j): >> >> 0 1 2 3 >> (0,0) (0,1) (0,2) (0,3) >> >> 4 5 6 7 >> (1,0) (1,1) (1,2) (1,3) >> >> 8 9 10 11 >> (2,0) (2,1) (2,2) (2,3) >> >> >> By the way I agree with you, I *should* be able to swap the x and y >> myself. Just haven't had much luck yet in that regard. >> >> Sean >> >>> On Thu, 18 May 2006, Sean Dettrick wrote: >>>> Barry Smith wrote: >>>>> On Thu, 18 May 2006, Sean Dettrick wrote: >>>>>> Hi Barry, >>>>>> the order is determined by MPI_Cart_create. >>>>> >>>>> Do you mean that MPI_Cart_create() orders across the 2nd (y-axis) >>>>> fastest and then the first (x-axis)? Hmmm, maybe we should >>>>> change the >>>>> DA? Changing it once and for all (not supporting both) is probably >>>>> not a big deal and shouldn't break much (I hope). >>>> Hi Barry, >>>> it depends, what do you call x and what do you call y? >>>> MPI_Cart_coords returns a vector, coords - I tend to say x is >>>> coords[0], y is coords[1] and z is coords[2]. For what it's >>>> worth, there's a short code appended to this email, which produces: >>>> rank = 0 has Cartesian coords = { 0, 0 } >>>> rank = 1 has Cartesian coords = { 0, 1 } >>>> rank = 2 has Cartesian coords = { 1, 0 } >>>> rank = 3 has Cartesian coords = { 1, 1 } >>>> rank = 0 has DA range x=[0,50) and y=[0,50) >>>> rank = 1 has DA range x=[50,100) and y=[0,50) >>>> rank = 2 has DA range x=[0,50) and y=[50,100) >>>> rank = 3 has DA range x=[50,100) and y=[50,100) >>>> >>>>>>> I don't completely understand what goes wrong. Is it because >>>>>>> YOUR >>>>>>> application orders the processors related to geometry in the >>>>>>> following way? >>>>>>> >>>>>>> ^ y direction >>>>>>> | >>>>>>> 2 5 8 >>>>>>> 1 4 7 >>>>>>> 0 3 6 >>>>>>> >>>>>>> -> x direction >>>>>>> Or is this something inherent in MPI_Cart_create? >>>> For my interpretation of x and y, MPI_Cart_create produces the >>>> above layout. But if I said x=coords[1] and y=coords[0], then it >>>> would match the one below. >>>>>>> PETSc does it so >>>>>>> >>>>>>> ^ y direction >>>>>>> | >>>>>>> 6 7 8 >>>>>>> 3 4 5 >>>>>>> 0 1 2 >>>>>>> >>>>>>> -> x direction >>>> Code and makefile attached ... hopefully within the message size >>>> limit. >>>> Just make cartcommtest. >>>> Sean >> >> > From schuang at ats.ucla.edu Wed Nov 12 20:48:31 2008 From: schuang at ats.ucla.edu (Shao-Ching Huang) Date: Wed, 12 Nov 2008 18:48:31 -0800 Subject: process id in DA In-Reply-To: <210E5146-802C-40B5-9736-E3D3B407E74A@mcs.anl.gov> References: <20081113002423.GA22482@ats.ucla.edu> <210E5146-802C-40B5-9736-E3D3B407E74A@mcs.anl.gov> Message-ID: <20081113024831.GB22906@ats.ucla.edu> So it's in the "fortran order" (in the sense of mapping the memory address of A(i,j,k)). I will write my own query routine following this line. Thanks, Shao-Ching On Wed, Nov 12, 2008 at 07:15:22PM -0600, Barry Smith wrote: > > Doesn't look like it: but from the formula > > rank = i + j*Mp + k*Mp*Np > > i = rank % Mp > > k = rank/(Mp*Np) > > j = (rank - k*Mp*Np)/Mp > > Barry > > Or something similar. > > Mp, Np are the number of processes in the x and y directions > From tim.kroeger at cevis.uni-bremen.de Thu Nov 13 05:22:42 2008 From: tim.kroeger at cevis.uni-bremen.de (Tim Kroeger) Date: Thu, 13 Nov 2008 12:22:42 +0100 (CET) Subject: ILU for MatAIJ Message-ID: Dear PETSc team, In my application, I got this (on 4 CPUs): [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: No support for this operation for this object type! [0]PETSC ERROR: Matrix type mpiaij symbolic ILU! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 2.3.3, Patch 11, Thu Mar 27 09:16:38 CDT 2008 HG revision: 0a79e1469a317b0468b3b3310524bf13867c66f5 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./checkConvergence on a linux-gnu named inf056 by tkroger Thu Nov 13 04:06:32 2008 [0]PETSC ERROR: Libraries linked from /home/collab/tkroger/packages/petsc-2.3.3-p11/lib/linux-gnu-c-debug [0]PETSC ERROR: Configure run at Fri Oct 17 06:57:08 2008 [0]PETSC ERROR: Configure options --with-mpi-dir=/usr/local/lam-mpi/ --with-mpi-compilers=1 --download-f-blas-lapack=1 --with-shared=0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatILUFactorSymbolic() line 4788 in src/mat/interface/matrix.c [0]PETSC ERROR: PCSetUp_ILU() line 541 in src/ksp/pc/impls/factor/ilu/ilu.c [0]PETSC ERROR: PCSetUp() line 787 in src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPSetUp() line 234 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: KSPSolve() line 347 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: User provided function() line 687 in unknowndirectory/src/numerics/petsc_linear_solver.C I wonder what that means because the AIJ matrix type should be able to do ILU preconditioning, shouldn't it? In other words: If not, which matrix type will then be able to do that? Best Regards, Tim -- Dr. Tim Kroeger Phone +49-421-218-7710 tim.kroeger at mevis.de, tim.kroeger at cevis.uni-bremen.de Fax +49-421-218-4236 MeVis Research GmbH, Universitaetsallee 29, 28359 Bremen, Germany Amtsgericht Bremen HRB 16222 Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen From dave.mayhem23 at gmail.com Thu Nov 13 06:16:29 2008 From: dave.mayhem23 at gmail.com (Dave May) Date: Thu, 13 Nov 2008 23:16:29 +1100 Subject: ILU for MatAIJ In-Reply-To: References: Message-ID: <956373f0811130416o12494df2pf949c91e4bd89f45@mail.gmail.com> Hey Tim, There is no parallel ILU support in petsc. In parallel, the closest thing is block jacobi with ilu on each block. This is the default parallel preconditioner in petsc. If you have installed hypre, it provide a parallel ILU, via pilut. Cheers, Dave On Thu, Nov 13, 2008 at 10:22 PM, Tim Kroeger < tim.kroeger at cevis.uni-bremen.de> wrote: > Dear PETSc team, > > In my application, I got this (on 4 CPUs): > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: Matrix type mpiaij symbolic ILU! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 2.3.3, Patch 11, Thu Mar 27 09:16:38 > CDT 2008 HG revision: 0a79e1469a317b0468b3b3310524bf13867c66f5 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./checkConvergence on a linux-gnu named inf056 by tkroger > Thu Nov 13 04:06:32 2008 > [0]PETSC ERROR: Libraries linked from > /home/collab/tkroger/packages/petsc-2.3.3-p11/lib/linux-gnu-c-debug > [0]PETSC ERROR: Configure run at Fri Oct 17 06:57:08 2008 > [0]PETSC ERROR: Configure options --with-mpi-dir=/usr/local/lam-mpi/ > --with-mpi-compilers=1 --download-f-blas-lapack=1 --with-shared=0 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatILUFactorSymbolic() line 4788 in > src/mat/interface/matrix.c > [0]PETSC ERROR: PCSetUp_ILU() line 541 in src/ksp/pc/impls/factor/ilu/ilu.c > [0]PETSC ERROR: PCSetUp() line 787 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSetUp() line 234 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: KSPSolve() line 347 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: User provided function() line 687 in > unknowndirectory/src/numerics/petsc_linear_solver.C > > > I wonder what that means because the AIJ matrix type should be able to do > ILU preconditioning, shouldn't it? In other words: If not, which matrix > type will then be able to do that? > > Best Regards, > > Tim > > -- > Dr. Tim Kroeger Phone > +49-421-218-7710 > tim.kroeger at mevis.de, tim.kroeger at cevis.uni-bremen.de Fax > +49-421-218-4236 > > MeVis Research GmbH, Universitaetsallee 29, 28359 Bremen, Germany > > Amtsgericht Bremen HRB 16222 > Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Thu Nov 13 06:22:05 2008 From: jed at 59A2.org (Jed Brown) Date: Thu, 13 Nov 2008 13:22:05 +0100 Subject: ILU for MatAIJ In-Reply-To: References: Message-ID: <20081113122205.GC29410@brakk.ethz.ch> On Thu 2008-11-13 12:22, Tim Kroeger wrote: > I wonder what that means because the AIJ matrix type should be able to > do ILU preconditioning, shouldn't it? In other words: If not, which > matrix type will then be able to do that? ILU has very poor parallel scaling and is not implemented in PETSc. With only four processors, it may still be a reasonable choice and Euclid from Hypre (--download-hypre) is probably the best parallel implementation. -pc_type hypre -pc_hypre_type euclid Jed -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From tim.kroeger at cevis.uni-bremen.de Thu Nov 13 08:07:55 2008 From: tim.kroeger at cevis.uni-bremen.de (Tim Kroeger) Date: Thu, 13 Nov 2008 15:07:55 +0100 (CET) Subject: ILU for MatAIJ In-Reply-To: <956373f0811130416o12494df2pf949c91e4bd89f45@mail.gmail.com> References: <956373f0811130416o12494df2pf949c91e4bd89f45@mail.gmail.com> Message-ID: Dear Dave and Jed, On Thu, 13 Nov 2008, Dave May wrote: > There is no parallel ILU support in petsc. > In parallel, the closest thing is block jacobi with ilu on each block. Thank you very much. I mixed that up. It's doing well now. Best Regards, Tim -- Dr. Tim Kroeger Phone +49-421-218-7710 tim.kroeger at mevis.de, tim.kroeger at cevis.uni-bremen.de Fax +49-421-218-4236 MeVis Research GmbH, Universitaetsallee 29, 28359 Bremen, Germany Amtsgericht Bremen HRB 16222 Geschaeftsfuehrer: Prof. Dr. H.-O. Peitgen From bhatiamanav at gmail.com Thu Nov 13 20:51:41 2008 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Thu, 13 Nov 2008 21:51:41 -0500 Subject: make test on sgi Message-ID: <8A1523AA-26A5-4287-B691-FC6CAC5210E5@gmail.com> Hi, I have built a static petsc library on sgi running suse 10. When I run make test, I get the following message, which I am not able to decide if it is harmless or not. Please let me know if anyone has seen this in the past. Regards, Manav manav at ericjohnson:petsc-2.3.3-p13> make test Running test examples to verify correct installation C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process ex19(10331): unaligned access to 0x6000000000f7c0e4, ip=0x40000000006a5b71 ex19(10330): unaligned access to 0x6000000000f9bcb4, ip=0x40000000006a5b71 ex19(10330): unaligned access to 0x6000000000f9bcac, ip=0x40000000006a5b71 ex19(10331): unaligned access to 0x6000000000f7c0dc, ip=0x40000000006a5b71 ex19(10330): unaligned access to 0x6000000000f9bcac, ip=0x40000000002d9750 ex19(10330): unaligned access to 0x6000000000f9bc6c, ip=0x200000000051b250 C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI processes Graphics example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process Completed test examples my configuration options are: ./config/configure.py --with-clanguage=C++ --with-c-support --with- fc=0 --with-shared=0 --with-debugging=no --with-mpi=true --with- mpirun=mpirun --with-mpi-include=/usr/include --with-mpi-lib=[/usr/lib/ libmpi++.so,/usr/lib/libmpi.so] --with-parmetis=true --with-parmetis- dir=/l/manav/codes/ParMetis-3.1 --with-blas-lib=[/usr/lib/libblas.so,- lgfortran] --with-lapack-lib=[/usr/lib/liblapack.so,-lgfortran] --with-x From bsmith at mcs.anl.gov Thu Nov 13 21:17:36 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 13 Nov 2008 21:17:36 -0600 Subject: make test on sgi In-Reply-To: <8A1523AA-26A5-4287-B691-FC6CAC5210E5@gmail.com> References: <8A1523AA-26A5-4287-B691-FC6CAC5210E5@gmail.com> Message-ID: <2C620663-F330-40E0-A5B7-D8156F002405@mcs.anl.gov> This message is harmless. Barry On Nov 13, 2008, at 8:51 PM, Manav Bhatia wrote: > Hi, > > I have built a static petsc library on sgi running suse 10. > When I run make test, I get the following message, which I am not > able to decide if it is harmless or not. Please let me know if > anyone has seen this in the past. > > Regards, > Manav > > > manav at ericjohnson:petsc-2.3.3-p13> make test > Running test examples to verify correct installation > C/C++ example src/snes/examples/tutorials/ex19 run successfully with > 1 MPI process > ex19(10331): unaligned access to 0x6000000000f7c0e4, > ip=0x40000000006a5b71 > ex19(10330): unaligned access to 0x6000000000f9bcb4, > ip=0x40000000006a5b71 > ex19(10330): unaligned access to 0x6000000000f9bcac, > ip=0x40000000006a5b71 > ex19(10331): unaligned access to 0x6000000000f7c0dc, > ip=0x40000000006a5b71 > ex19(10330): unaligned access to 0x6000000000f9bcac, > ip=0x40000000002d9750 > ex19(10330): unaligned access to 0x6000000000f9bc6c, > ip=0x200000000051b250 > C/C++ example src/snes/examples/tutorials/ex19 run successfully with > 2 MPI processes > Graphics example src/snes/examples/tutorials/ex19 run successfully > with 1 MPI process > Completed test examples > > > my configuration options are: > > ./config/configure.py --with-clanguage=C++ --with-c-support --with- > fc=0 --with-shared=0 --with-debugging=no --with-mpi=true --with- > mpirun=mpirun --with-mpi-include=/usr/include --with-mpi-lib=[/usr/ > lib/libmpi++.so,/usr/lib/libmpi.so] --with-parmetis=true --with- > parmetis-dir=/l/manav/codes/ParMetis-3.1 --with-blas-lib=[/usr/lib/ > libblas.so,-lgfortran] --with-lapack-lib=[/usr/lib/liblapack.so,- > lgfortran] --with-x > > From stvtiegh at ulb.ac.be Mon Nov 17 06:04:48 2008 From: stvtiegh at ulb.ac.be (stijn vantieghem) Date: Mon, 17 Nov 2008 13:04:48 +0100 Subject: convergence of BoomerAMG Message-ID: <740A7075-D485-4B4D-94F3-ED9C138A994C@ulb.ac.be> Dear PETSc team I'm using PETSC with HYPRE's BoomerAMG to solve a pressure Poisson equation in an incompressible Navier-Stokes solver. When working on a 64x129x64-grid, it converges reasonbly well on 8 (1 bi-quad-core) processors, However, when changing. to 16 (2 bi-quad-core) procs, BoomerAMG is hardly able to converge. Below, you find an excerpt of my code with all the flags I'm using . Have you any suggestions to overcome this problem? Thanks in advance Sincerely yours Stijn Vantieghem ! KSP parameters call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) call KSPSetType(ksp,KSPRICHARDSON,ierr) call KSPSetOperators(ksp,M,M,DIFFERENT_NONZERO_PATTERN,ierr) call KSPGetPC(ksp,pc,ierr) call PCSetType(pc,PCHYPRE,ierr) call PCHYPRESetType(pc,"boomeramg",ierr) ! HYPRE Options call PetscOptionsSetValue("- pc_hypre_boomeramg_print_statistics","2",ierr) call PetscOptionsSetValue("- pc_hypre_boomeramg_tol",trim(co),ierr) call PetscOptionsSetValue("- pc_hypre_boomeramg_max_iter","25",ierr) call PetscOptionsSetValue("- pc_hypre_boomeramg_min_iter","2",ierr) call PetscOptionsSetValue("- pc_hypre_boomeramg_grid_sweeps_down","2",ierr) call PetscOptionsSetValue("- pc_hypre_boomeramg_grid_sweeps_up","2",ierr) call PetscOptionsSetValue("- pc_hypre_boomeramg_grid_sweeps_coarse","1",ierr) call PetscOptionsSetValue("- pc_hypre_boomeramg_relax_type_down","SOR/Jacobi",ierr) call PetscOptionsSetValue("- pc_hypre_boomeramg_relax_type_up","SOR/Jacobi",ierr) call PetscOptionsSetValue("- pc_hypre_boomeramg_relax_type_coarse","Jacobi",ierr) call PetscOptionsSetValue("- pc_hypre_boomeramg_relax_weight_all","1.0",ierr) call PetscOptionsSetValue("- pc_hypre_boomeramg_outer_relax_weight_all","1.0",ierr) call PetscOptionsSetValue("- pc_hypre_boomeramg_grid_sweeps_all","1",ierr) call PetscOptionsSetValue("- pc_hypre_boomeramg_max_row_sum","1.0",ierr) call PetscOptionsSetValue("- pc_hypre_boomeramg_max_levels","25",ierr) ! call PetscOptionsSetValue("- pc_hypre_boomeramg_nodal_coarsen",PETSC_NULL_CHARACTER,ierr) call KSPSetTolerances(ksp,1.0e-10_WP,1.0e-6_WP,1.0e15_WP, 1,ierr) call KSPSetFromOptions(ksp,ierr) From bsmith at mcs.anl.gov Mon Nov 17 11:04:30 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 17 Nov 2008 11:04:30 -0600 Subject: convergence of BoomerAMG In-Reply-To: <740A7075-D485-4B4D-94F3-ED9C138A994C@ulb.ac.be> References: <740A7075-D485-4B4D-94F3-ED9C138A994C@ulb.ac.be> Message-ID: First run with call KSPSetType(ksp,GMRES,ierr) and see what difference that makes. You can switch to petsc-dev http://www-unix.mcs.anl.gov/petsc/petsc-as/developers/index.html if you are not currently using it. This with the config/configure.py option --download-hypre will use the latest version of hypre. If none of this helps then run both cases printing the statistics and send them to petsc-maint at mcs.anl.gov and I'll take a look at them. Barry On Nov 17, 2008, at 6:04 AM, stijn vantieghem wrote: > Dear PETSc team > > I'm using PETSC with HYPRE's BoomerAMG to solve a pressure Poisson > equation in an incompressible Navier-Stokes solver. When working on > a 64x129x64-grid, it converges reasonbly well on 8 (1 bi-quad-core) > processors, However, when changing. to 16 (2 bi-quad-core) procs, > BoomerAMG is hardly able to converge. Below, you find an excerpt of > my code with all the flags I'm using . Have you any suggestions to > overcome this problem? > Thanks in advance > > Sincerely yours > Stijn Vantieghem > > ! KSP parameters > call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > call KSPSetType(ksp,KSPRICHARDSON,ierr) > call KSPSetOperators(ksp,M,M,DIFFERENT_NONZERO_PATTERN,ierr) > > call KSPGetPC(ksp,pc,ierr) > call PCSetType(pc,PCHYPRE,ierr) > call PCHYPRESetType(pc,"boomeramg",ierr) > > ! HYPRE Options > call PetscOptionsSetValue("- > pc_hypre_boomeramg_print_statistics","2",ierr) > > call PetscOptionsSetValue("- > pc_hypre_boomeramg_tol",trim(co),ierr) > call PetscOptionsSetValue("- > pc_hypre_boomeramg_max_iter","25",ierr) > call PetscOptionsSetValue("- > pc_hypre_boomeramg_min_iter","2",ierr) > > call PetscOptionsSetValue("- > pc_hypre_boomeramg_grid_sweeps_down","2",ierr) > call PetscOptionsSetValue("- > pc_hypre_boomeramg_grid_sweeps_up","2",ierr) > call PetscOptionsSetValue("- > pc_hypre_boomeramg_grid_sweeps_coarse","1",ierr) > > call PetscOptionsSetValue("- > pc_hypre_boomeramg_relax_type_down","SOR/Jacobi",ierr) > call PetscOptionsSetValue("- > pc_hypre_boomeramg_relax_type_up","SOR/Jacobi",ierr) > call PetscOptionsSetValue("- > pc_hypre_boomeramg_relax_type_coarse","Jacobi",ierr) > > call PetscOptionsSetValue("- > pc_hypre_boomeramg_relax_weight_all","1.0",ierr) > call PetscOptionsSetValue("- > pc_hypre_boomeramg_outer_relax_weight_all","1.0",ierr) > > call PetscOptionsSetValue("- > pc_hypre_boomeramg_grid_sweeps_all","1",ierr) > call PetscOptionsSetValue("- > pc_hypre_boomeramg_max_row_sum","1.0",ierr) > call PetscOptionsSetValue("- > pc_hypre_boomeramg_max_levels","25",ierr) > ! call PetscOptionsSetValue("- > pc_hypre_boomeramg_nodal_coarsen",PETSC_NULL_CHARACTER,ierr) > > call KSPSetTolerances(ksp,1.0e-10_WP,1.0e-6_WP,1.0e15_WP, > 1,ierr) > call KSPSetFromOptions(ksp,ierr) > From rafaelsantoscoelho at gmail.com Tue Nov 18 09:59:56 2008 From: rafaelsantoscoelho at gmail.com (Rafael Santos Coelho) Date: Tue, 18 Nov 2008 13:59:56 -0200 Subject: Doubts about SNESMF and matrix-free preconditioners Message-ID: <3b6f83d40811180759m2d2674d3p4139682e91d6df5e@mail.gmail.com> Hi, everyone (: For a considerable amount of time, I've used PETSc a lot in my numerical experiments, but then I stopped for a while and got rusty at it. So now, I'm trying to get back on track by implementing my own serial matrix-free preconditioner for the JFNK (Jacobian-free Newton-Krylov) method. Basicly, it's a uniparametric LU-SGS (LU Symmetric Gauss-Seidel) preconditioner as bellow: 1. Let J(x), F(x) and di be the jacobian matrix, the nonlinear function and ith diagonal element of J(x), respectively 2. Suppose J(x) = L + D + U and consider J(x)s = -F(x), where 's' is the Newton correction 3. Let M = (D + wL)D^(-1)(D + wU), where w >0 4. Now, we have M^(-1)J(x)s = -M^(-1)F(x), knowing that M^(-1)J(x)s = (F(x + hM^(-1)s) - F(x)) / h 5. Let y = M^(-1)s = (D + wU)^(-1)D(D + wL)^(-1)s and z = (D + wL)^(-1)s <=> (D + wL)z = s 6. To find z, we follow steps 7 and 8 7. x' = x, z1 = s1 / d1, x'1 = x1 + hz1 8. for(i = 2; i <= n; i++) { zi = (si - w(Fi(x') - Fi(x)) / h) / di ; x'i = xi + hzi ; } 9. Now, we have y = (D + wU)^(-1)Dz 10. Let v = Dz and we got y = (D + wU)^(-1)v 11. Let u = (D + wU)^(-1)v <=> (D + wU)u = v, and to find u we go through steps 12 and 13 12. x' = x, un = vn / dn, x'n = xn + hvn 13. for(i = n - 1; i >= 1; i--) { ui = (vi - w(Fi(x') - Fi(x)) / h) / di ; x'i = xi + hui ; } 14. Finally, we compute M^(-1)J(x)s = (F(x + hy) - F(x)) / h As for the right-hand side, a similar approach can be done. I only need to store diag[J(x)]. Well, I'm not sure how to code the steps above. However, looking at "src/snes/examples/tutorials/ex6.c", things got cleared out a little bit for me. I think it suffices to type "-snes_mf" on the command-line, to write my own "MatrixFreePreconditioner" with all the steps above and then 1. ierr = PCSetType(pc, PCSHELL); CHKERRQ(ierr); 2. ierr = PCShellSetApply(pc, MatrixFreePreconditioner); CHKERRQ(ierr); Is my reasoning correct? -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Nov 18 10:39:21 2008 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 18 Nov 2008 10:39:21 -0600 Subject: Doubts about SNESMF and matrix-free preconditioners In-Reply-To: <3b6f83d40811180759m2d2674d3p4139682e91d6df5e@mail.gmail.com> References: <3b6f83d40811180759m2d2674d3p4139682e91d6df5e@mail.gmail.com> Message-ID: On Tue, Nov 18, 2008 at 9:59 AM, Rafael Santos Coelho wrote: > Hi, everyone (: > > For a considerable amount of time, I've used PETSc a lot in my numerical > experiments, but then I stopped for a while and got rusty at it. So now, I'm > trying to get back on track by implementing my own serial matrix-free > preconditioner for the JFNK (Jacobian-free Newton-Krylov) method. Basicly, > it's a uniparametric LU-SGS (LU Symmetric Gauss-Seidel) preconditioner as > bellow: > > Let J(x), F(x) and di be the jacobian matrix, the nonlinear function and ith > diagonal element of J(x), respectively > Suppose J(x) = L + D + U and consider J(x)s = -F(x), where 's' is the Newton > correction > Let M = (D + wL)D^(-1)(D + wU), where w >0 > Now, we have M^(-1)J(x)s = -M^(-1)F(x), knowing that M^(-1)J(x)s = (F(x + > hM^(-1)s) - F(x)) / h > Let y = M^(-1)s = (D + wU)^(-1)D(D + wL)^(-1)s and z = (D + wL)^(-1)s <=> (D > + wL)z = s > To find z, we follow steps 7 and 8 > x' = x, z1 = s1 / d1, x'1 = x1 + hz1 > for(i = 2; i <= n; i++) { zi = (si - w(Fi(x') - Fi(x)) / h) / di ; x'i = xi > + hzi ; } > Now, we have y = (D + wU)^(-1)Dz > Let v = Dz and we got y = (D + wU)^(-1)v > Let u = (D + wU)^(-1)v <=> (D + wU)u = v, and to find u we go through steps > 12 and 13 > x' = x, un = vn / dn, x'n = xn + hvn > for(i = n - 1; i >= 1; i--) { ui = (vi - w(Fi(x') - Fi(x)) / h) / di ; x'i = > xi + hui ; } > Finally, we compute M^(-1)J(x)s = (F(x + hy) - F(x)) / h > > As for the right-hand side, a similar approach can be done. I only need to > store diag[J(x)]. Well, I'm not sure how to code the steps above. However, > looking at "src/snes/examples/tutorials/ex6.c", things got cleared out a > little bit for me. I think it suffices to type "-snes_mf" on the > command-line, to write my own "MatrixFreePreconditioner" with all the steps > above and then > > ierr = PCSetType(pc, PCSHELL); CHKERRQ(ierr); > ierr = PCShellSetApply(pc, MatrixFreePreconditioner); CHKERRQ(ierr); > > Is my reasoning correct? I did not read the PC discussion closely, however, you can do two things: 1) Use -snes_mf_operator described here: http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/SNES/SNESCreate.html What this means is that matrix-vector products with the Jacobian are performed using finite differences with F, and then a preconditioner is computed using a simpler matrix you specify using SNESSetJacobian(). This is easy, but not what you describe above. 2) Use -snes_mf, which again does matvecs with FD, but now no preconditioner is set by default. Then you would have to extract the KSP and set your matrix-free PC there. The matrix it will receive will be the MFFD that is created by -mat_mf. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From rafaelsantoscoelho at gmail.com Tue Nov 18 13:22:13 2008 From: rafaelsantoscoelho at gmail.com (Rafael Santos Coelho) Date: Tue, 18 Nov 2008 17:22:13 -0200 Subject: Doubts about SNESMF and matrix-free preconditioners In-Reply-To: References: <3b6f83d40811180759m2d2674d3p4139682e91d6df5e@mail.gmail.com> Message-ID: <3b6f83d40811181122t7182d94bj1d6bcd603844a85e@mail.gmail.com> Originally, my idea is to implement this preconditioner. As far as I`m concerned, there`s no need to form and store any matrices, only the diagonal elements of the jacobian like I said. This way, I believe I`d have to stick with option #2. With regard to "The matrix it will receive will be the MFFD that is created by -mat_mf.", what matrix is that actually? The one that is created by the `MatCreateMFFD` routine? "Then you would have to extract the KSP and set your matrix-free PC there." Do the calls to PCSetType and PCShellSetApply plus my own implementation of the `MatrixFreePreconditioner` function do that? Best regards, Rafael -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Nov 18 13:54:34 2008 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 18 Nov 2008 13:54:34 -0600 Subject: Doubts about SNESMF and matrix-free preconditioners In-Reply-To: <3b6f83d40811181122t7182d94bj1d6bcd603844a85e@mail.gmail.com> References: <3b6f83d40811180759m2d2674d3p4139682e91d6df5e@mail.gmail.com> <3b6f83d40811181122t7182d94bj1d6bcd603844a85e@mail.gmail.com> Message-ID: On Tue, Nov 18, 2008 at 1:22 PM, Rafael Santos Coelho wrote: > Originally, my idea is to implement this preconditioner. As far as I`m > concerned, there`s no need to form and store any matrices, only the diagonal > elements of the jacobian like I said. This way, I believe I`d have to stick > with option #2. > > With regard to "The matrix it will receive will be the MFFD that is created > by -mat_mf.", what matrix is that actually? The one that is created by the > `MatCreateMFFD` routine? Yes. > "Then you would have to extract the KSP and set your matrix-free PC there." > > Do the calls to PCSetType and PCShellSetApply plus my own implementation of > the `MatrixFreePreconditioner` function do that? SNESGetKSP KSPGetPC then your functions Matt > Best regards, > > Rafael -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From Hung.V.Nguyen at usace.army.mil Thu Nov 20 08:57:26 2008 From: Hung.V.Nguyen at usace.army.mil (Nguyen, Hung V ERDC-ITL-MS) Date: Thu, 20 Nov 2008 08:57:26 -0600 Subject: Question with pc_type redundant Message-ID: All, I try to test the pc_type of redundant with ksp_type of stcg and cg. For both cases, why does the elapsed times are large when running with large number of processors? Thanks for your help. -Hung hvnguyen:jade17% head A.m % Size = 59409 59409 % Nonzeros = 1113875 zzz = zeros(1113875,3); 1. Test with stcg and redundant: hvnguyen:jade17% aprun -n 1 ./test_matrix_read -ksp_type stcg -pc_type redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP Residual norm 1.573381744277e+04 0 KSP Residual norm 1.117929341660e-06 0 KSP Residual norm 2.394961013957e-16 Time in PETSc solver: 26.436834 seconds The number of iteration = 2 The solution residual error = 2.394961e-16 2 norm 1.848728e-08 infinity norm 4.014424e-10 1 norm 2.288013e-06 hvnguyen:jade17% aprun -n 2 ./test_matrix_read -ksp_type stcg -pc_type redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP Residual norm 1.573381744277e+04 0 KSP Residual norm 1.108986752645e-06 0 KSP Residual norm 2.388254585478e-16 Time in PETSc solver: 28.498422 seconds The number of iteration = 2 The solution residual error = 2.388255e-16 2 norm 1.763764e-08 infinity norm 3.500773e-10 1 norm 2.543252e-06 hvnguyen:jade17% aprun -n 4 ./test_matrix_read -ksp_type stcg -pc_type redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP Residual norm 1.573381744277e+04 0 KSP Residual norm 1.076617230240e-06 0 KSP Residual norm 2.364985060528e-16 Time in PETSc solver: 34.782896 seconds The number of iteration = 2 The solution residual error = 2.364985e-16 2 norm 3.544232e-08 infinity norm 4.848555e-10 1 norm 5.478173e-06 hvnguyen:jade17% setenv MPICH_UNEX_BUFFER_SIZE 100M hvnguyen:jade17% setenv MPICH_PTL_MATCH_OFF 1 hvnguyen:jade17% aprun -n 8 ./test_matrix_read -ksp_type stcg -pc_type redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP Residual norm 1.573381744277e+04 0 KSP Residual norm 1.076187797630e-06 0 KSP Residual norm 2.327317778542e-16 Time in PETSc solver: 34.810215 seconds The number of iteration = 2 The solution residual error = 2.327318e-16 2 norm 3.448040e-08 infinity norm 4.582703e-10 1 norm 5.306073e-06 hvnguyen:jade17% aprun -n 16 ./test_matrix_read -ksp_type stcg -pc_type redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP Residual norm 1.573381744277e+04 0 KSP Residual norm 1.170520566185e-06 0 KSP Residual norm 2.480424058413e-16 Time in PETSc solver: 34.851701 seconds The number of iteration = 2 The solution residual error = 2.480424e-16 2 norm 6.318255e-08 infinity norm 8.093555e-10 1 norm 9.681583e-06 hvnguyen:jade17% aprun -n 32 ./test_matrix_read -ksp_type stcg -pc_type redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP Residual norm 1.573381744277e+04 0 KSP Residual norm 1.072600099318e-06 0 KSP Residual norm 2.340608377807e-16 Time in PETSc solver: 34.951315 seconds The number of iteration = 2 The solution residual error = 2.340608e-16 2 norm 3.767959e-08 infinity norm 5.213003e-10 1 norm 5.790317e-06 2. Test with cg and redundant: hvnguyen:jade17% aprun -n 1 ./test_matrix_read -ksp_type cg -pc_type redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP Residual norm 1.573381744277e+04 1 KSP Residual norm 1.117929341660e-06 2 KSP Residual norm 2.394961013957e-16 Time in PETSc solver: 26.460811 seconds The number of iteration = 2 The solution residual error = 2.394961e-16 2 norm 1.848728e-08 infinity norm 4.014424e-10 1 norm 2.288013e-06 hvnguyen:jade17% aprun -n 2 ./test_matrix_read -ksp_type cg -pc_type redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP Residual norm 1.573381744277e+04 1 KSP Residual norm 1.108986752645e-06 2 KSP Residual norm 2.388254585478e-16 Time in PETSc solver: 28.494262 seconds The number of iteration = 2 The solution residual error = 2.388255e-16 2 norm 1.763764e-08 infinity norm 3.500773e-10 1 norm 2.543252e-06 hvnguyen:jade17% aprun -n 4 ./test_matrix_read -ksp_type cg -pc_type redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP Residual norm 1.573381744277e+04 1 KSP Residual norm 1.076617230240e-06 2 KSP Residual norm 2.364985060528e-16 Time in PETSc solver: 34.746018 seconds The number of iteration = 2 The solution residual error = 2.364985e-16 2 norm 3.544232e-08 infinity norm 4.848555e-10 1 norm 5.478173e-06 hvnguyen:jade17% aprun -n 8 ./test_matrix_read -ksp_type cg -pc_type redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP Residual norm 1.573381744277e+04 1 KSP Residual norm 1.076187797630e-06 2 KSP Residual norm 2.327317778542e-16 Time in PETSc solver: 34.830512 seconds The number of iteration = 2 The solution residual error = 2.327318e-16 2 norm 3.448040e-08 infinity norm 4.582703e-10 1 norm 5.306073e-06 hvnguyen:jade17% aprun -n 16 ./test_matrix_read -ksp_type cg -pc_type redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP Residual norm 1.573381744277e+04 1 KSP Residual norm 1.170520566185e-06 2 KSP Residual norm 2.480424058413e-16 Time in PETSc solver: 34.866815 seconds The number of iteration = 2 The solution residual error = 2.480424e-16 2 norm 6.318255e-08 infinity norm 8.093555e-10 1 norm 9.681583e-06 hvnguyen:jade17% aprun -n 32 ./test_matrix_read -ksp_type cg -pc_type redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP Residual norm 1.573381744277e+04 1 KSP Residual norm 1.072600099318e-06 2 KSP Residual norm 2.340608377807e-16 Time in PETSc solver: 34.968400 seconds The number of iteration = 2 The solution residual error = 2.340608e-16 2 norm 3.767959e-08 infinity norm 5.213003e-10 1 norm 5.790317e-06 From hzhang at mcs.anl.gov Thu Nov 20 09:31:24 2008 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Thu, 20 Nov 2008 09:31:24 -0600 (CST) Subject: Question with pc_type redundant In-Reply-To: References: Message-ID: Hung, PC redundant is used for solving coarse-grid linear problem, for which the problem is too small for parallel processing. Thus, we assign each(default) or a subset of procesors to solve the problem redundantly with LU direct solver. In your case, it is likely all processors implement LU sequentially. Adding the time of broadcasting matrix entries to all processors, more processors take longer time is anticipated. Hong On Thu, 20 Nov 2008, Nguyen, Hung V ERDC-ITL-MS wrote: > All, > > I try to test the pc_type of redundant with ksp_type of stcg and cg. For both > cases, why does the elapsed times are large when running with large number of > processors? > > Thanks for your help. > > -Hung > > hvnguyen:jade17% head A.m > % Size = 59409 59409 > % Nonzeros = 1113875 > zzz = zeros(1113875,3); > > 1. Test with stcg and redundant: > > hvnguyen:jade17% aprun -n 1 ./test_matrix_read -ksp_type stcg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor > 0 KSP Residual norm 1.573381744277e+04 > 0 KSP Residual norm 1.117929341660e-06 > 0 KSP Residual norm 2.394961013957e-16 > Time in PETSc solver: 26.436834 seconds > The number of iteration = 2 > The solution residual error = 2.394961e-16 > 2 norm 1.848728e-08 > infinity norm 4.014424e-10 > 1 norm 2.288013e-06 > > hvnguyen:jade17% aprun -n 2 ./test_matrix_read -ksp_type stcg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor > 0 KSP Residual norm 1.573381744277e+04 > 0 KSP Residual norm 1.108986752645e-06 > 0 KSP Residual norm 2.388254585478e-16 > Time in PETSc solver: 28.498422 seconds > The number of iteration = 2 > The solution residual error = 2.388255e-16 > 2 norm 1.763764e-08 > infinity norm 3.500773e-10 > 1 norm 2.543252e-06 > > hvnguyen:jade17% aprun -n 4 ./test_matrix_read -ksp_type stcg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor > 0 KSP Residual norm 1.573381744277e+04 > 0 KSP Residual norm 1.076617230240e-06 > 0 KSP Residual norm 2.364985060528e-16 > Time in PETSc solver: 34.782896 seconds > The number of iteration = 2 > The solution residual error = 2.364985e-16 > 2 norm 3.544232e-08 > infinity norm 4.848555e-10 > 1 norm 5.478173e-06 > > hvnguyen:jade17% setenv MPICH_UNEX_BUFFER_SIZE 100M > hvnguyen:jade17% setenv MPICH_PTL_MATCH_OFF 1 > hvnguyen:jade17% aprun -n 8 ./test_matrix_read -ksp_type stcg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor > 0 KSP Residual norm 1.573381744277e+04 > 0 KSP Residual norm 1.076187797630e-06 > 0 KSP Residual norm 2.327317778542e-16 > Time in PETSc solver: 34.810215 seconds > The number of iteration = 2 > The solution residual error = 2.327318e-16 > 2 norm 3.448040e-08 > infinity norm 4.582703e-10 > 1 norm 5.306073e-06 > > hvnguyen:jade17% aprun -n 16 ./test_matrix_read -ksp_type stcg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor > 0 KSP Residual norm 1.573381744277e+04 > 0 KSP Residual norm 1.170520566185e-06 > 0 KSP Residual norm 2.480424058413e-16 > Time in PETSc solver: 34.851701 seconds > The number of iteration = 2 > The solution residual error = 2.480424e-16 > 2 norm 6.318255e-08 > infinity norm 8.093555e-10 > 1 norm 9.681583e-06 > > > hvnguyen:jade17% aprun -n 32 ./test_matrix_read -ksp_type stcg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor > 0 KSP Residual norm 1.573381744277e+04 > 0 KSP Residual norm 1.072600099318e-06 > 0 KSP Residual norm 2.340608377807e-16 > Time in PETSc solver: 34.951315 seconds > The number of iteration = 2 > The solution residual error = 2.340608e-16 > 2 norm 3.767959e-08 > infinity norm 5.213003e-10 > 1 norm 5.790317e-06 > > > 2. Test with cg and redundant: > > hvnguyen:jade17% aprun -n 1 ./test_matrix_read -ksp_type cg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor > 0 KSP Residual norm 1.573381744277e+04 > 1 KSP Residual norm 1.117929341660e-06 > 2 KSP Residual norm 2.394961013957e-16 > Time in PETSc solver: 26.460811 seconds > The number of iteration = 2 > The solution residual error = 2.394961e-16 > 2 norm 1.848728e-08 > infinity norm 4.014424e-10 > 1 norm 2.288013e-06 > > hvnguyen:jade17% aprun -n 2 ./test_matrix_read -ksp_type cg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor > 0 KSP Residual norm 1.573381744277e+04 > 1 KSP Residual norm 1.108986752645e-06 > 2 KSP Residual norm 2.388254585478e-16 > Time in PETSc solver: 28.494262 seconds > The number of iteration = 2 > The solution residual error = 2.388255e-16 > 2 norm 1.763764e-08 > infinity norm 3.500773e-10 > 1 norm 2.543252e-06 > > hvnguyen:jade17% aprun -n 4 ./test_matrix_read -ksp_type cg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor > 0 KSP Residual norm 1.573381744277e+04 > 1 KSP Residual norm 1.076617230240e-06 > 2 KSP Residual norm 2.364985060528e-16 > Time in PETSc solver: 34.746018 seconds > The number of iteration = 2 > The solution residual error = 2.364985e-16 > 2 norm 3.544232e-08 > infinity norm 4.848555e-10 > 1 norm 5.478173e-06 > > hvnguyen:jade17% aprun -n 8 ./test_matrix_read -ksp_type cg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor > 0 KSP Residual norm 1.573381744277e+04 > 1 KSP Residual norm 1.076187797630e-06 > 2 KSP Residual norm 2.327317778542e-16 > Time in PETSc solver: 34.830512 seconds > The number of iteration = 2 > The solution residual error = 2.327318e-16 > 2 norm 3.448040e-08 > infinity norm 4.582703e-10 > 1 norm 5.306073e-06 > > hvnguyen:jade17% aprun -n 16 ./test_matrix_read -ksp_type cg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor > 0 KSP Residual norm 1.573381744277e+04 > 1 KSP Residual norm 1.170520566185e-06 > 2 KSP Residual norm 2.480424058413e-16 > Time in PETSc solver: 34.866815 seconds > The number of iteration = 2 > The solution residual error = 2.480424e-16 > 2 norm 6.318255e-08 > infinity norm 8.093555e-10 > 1 norm 9.681583e-06 > > hvnguyen:jade17% aprun -n 32 ./test_matrix_read -ksp_type cg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor > 0 KSP Residual norm 1.573381744277e+04 > 1 KSP Residual norm 1.072600099318e-06 > 2 KSP Residual norm 2.340608377807e-16 > Time in PETSc solver: 34.968400 seconds > The number of iteration = 2 > The solution residual error = 2.340608e-16 > 2 norm 3.767959e-08 > infinity norm 5.213003e-10 > 1 norm 5.790317e-06 > > > From Hung.V.Nguyen at usace.army.mil Thu Nov 20 11:47:05 2008 From: Hung.V.Nguyen at usace.army.mil (Nguyen, Hung V ERDC-ITL-MS) Date: Thu, 20 Nov 2008 11:47:05 -0600 Subject: Question with pc_type redundant In-Reply-To: References: Message-ID: Hong, Thank you for the answer. -Hung -----Original Message----- From: owner-petsc-users at mcs.anl.gov [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Hong Zhang Sent: Thursday, November 20, 2008 9:31 AM To: petsc-users at mcs.anl.gov Subject: Re: Question with pc_type redundant Hung, PC redundant is used for solving coarse-grid linear problem, for which the problem is too small for parallel processing. Thus, we assign each(default) or a subset of procesors to solve the problem redundantly with LU direct solver. In your case, it is likely all processors implement LU sequentially. Adding the time of broadcasting matrix entries to all processors, more processors take longer time is anticipated. Hong On Thu, 20 Nov 2008, Nguyen, Hung V ERDC-ITL-MS wrote: > All, > > I try to test the pc_type of redundant with ksp_type of stcg and cg. > For both cases, why does the elapsed times are large when running with > large number of processors? > > Thanks for your help. > > -Hung > > hvnguyen:jade17% head A.m > % Size = 59409 59409 > % Nonzeros = 1113875 > zzz = zeros(1113875,3); > > 1. Test with stcg and redundant: > > hvnguyen:jade17% aprun -n 1 ./test_matrix_read -ksp_type stcg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP > Residual norm 1.573381744277e+04 0 KSP Residual norm > 1.117929341660e-06 0 KSP Residual norm 2.394961013957e-16 Time in > PETSc solver: 26.436834 seconds > The number of iteration = 2 > The solution residual error = 2.394961e-16 > 2 norm 1.848728e-08 > infinity norm 4.014424e-10 > 1 norm 2.288013e-06 > > hvnguyen:jade17% aprun -n 2 ./test_matrix_read -ksp_type stcg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP > Residual norm 1.573381744277e+04 0 KSP Residual norm > 1.108986752645e-06 0 KSP Residual norm 2.388254585478e-16 Time in > PETSc solver: 28.498422 seconds > The number of iteration = 2 > The solution residual error = 2.388255e-16 > 2 norm 1.763764e-08 > infinity norm 3.500773e-10 > 1 norm 2.543252e-06 > > hvnguyen:jade17% aprun -n 4 ./test_matrix_read -ksp_type stcg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP > Residual norm 1.573381744277e+04 0 KSP Residual norm > 1.076617230240e-06 0 KSP Residual norm 2.364985060528e-16 Time in > PETSc solver: 34.782896 seconds > The number of iteration = 2 > The solution residual error = 2.364985e-16 > 2 norm 3.544232e-08 > infinity norm 4.848555e-10 > 1 norm 5.478173e-06 > > hvnguyen:jade17% setenv MPICH_UNEX_BUFFER_SIZE 100M > hvnguyen:jade17% setenv MPICH_PTL_MATCH_OFF 1 > hvnguyen:jade17% aprun -n 8 ./test_matrix_read -ksp_type stcg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP > Residual norm 1.573381744277e+04 0 KSP Residual norm > 1.076187797630e-06 0 KSP Residual norm 2.327317778542e-16 Time in > PETSc solver: 34.810215 seconds > The number of iteration = 2 > The solution residual error = 2.327318e-16 > 2 norm 3.448040e-08 > infinity norm 4.582703e-10 > 1 norm 5.306073e-06 > > hvnguyen:jade17% aprun -n 16 ./test_matrix_read -ksp_type stcg > -pc_type redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 > KSP Residual norm 1.573381744277e+04 0 KSP Residual norm > 1.170520566185e-06 0 KSP Residual norm 2.480424058413e-16 Time in > PETSc solver: 34.851701 seconds > The number of iteration = 2 > The solution residual error = 2.480424e-16 > 2 norm 6.318255e-08 > infinity norm 8.093555e-10 > 1 norm 9.681583e-06 > > > hvnguyen:jade17% aprun -n 32 ./test_matrix_read -ksp_type stcg > -pc_type redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 > KSP Residual norm 1.573381744277e+04 0 KSP Residual norm > 1.072600099318e-06 0 KSP Residual norm 2.340608377807e-16 Time in > PETSc solver: 34.951315 seconds > The number of iteration = 2 > The solution residual error = 2.340608e-16 > 2 norm 3.767959e-08 > infinity norm 5.213003e-10 > 1 norm 5.790317e-06 > > > 2. Test with cg and redundant: > > hvnguyen:jade17% aprun -n 1 ./test_matrix_read -ksp_type cg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP > Residual norm 1.573381744277e+04 > 1 KSP Residual norm 1.117929341660e-06 > 2 KSP Residual norm 2.394961013957e-16 Time in PETSc solver: > 26.460811 seconds > The number of iteration = 2 > The solution residual error = 2.394961e-16 > 2 norm 1.848728e-08 > infinity norm 4.014424e-10 > 1 norm 2.288013e-06 > > hvnguyen:jade17% aprun -n 2 ./test_matrix_read -ksp_type cg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP > Residual norm 1.573381744277e+04 > 1 KSP Residual norm 1.108986752645e-06 > 2 KSP Residual norm 2.388254585478e-16 Time in PETSc solver: > 28.494262 seconds > The number of iteration = 2 > The solution residual error = 2.388255e-16 > 2 norm 1.763764e-08 > infinity norm 3.500773e-10 > 1 norm 2.543252e-06 > > hvnguyen:jade17% aprun -n 4 ./test_matrix_read -ksp_type cg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP > Residual norm 1.573381744277e+04 > 1 KSP Residual norm 1.076617230240e-06 > 2 KSP Residual norm 2.364985060528e-16 Time in PETSc solver: > 34.746018 seconds > The number of iteration = 2 > The solution residual error = 2.364985e-16 > 2 norm 3.544232e-08 > infinity norm 4.848555e-10 > 1 norm 5.478173e-06 > > hvnguyen:jade17% aprun -n 8 ./test_matrix_read -ksp_type cg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP > Residual norm 1.573381744277e+04 > 1 KSP Residual norm 1.076187797630e-06 > 2 KSP Residual norm 2.327317778542e-16 Time in PETSc solver: > 34.830512 seconds > The number of iteration = 2 > The solution residual error = 2.327318e-16 > 2 norm 3.448040e-08 > infinity norm 4.582703e-10 > 1 norm 5.306073e-06 > > hvnguyen:jade17% aprun -n 16 ./test_matrix_read -ksp_type cg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP > Residual norm 1.573381744277e+04 > 1 KSP Residual norm 1.170520566185e-06 > 2 KSP Residual norm 2.480424058413e-16 Time in PETSc solver: > 34.866815 seconds > The number of iteration = 2 > The solution residual error = 2.480424e-16 > 2 norm 6.318255e-08 > infinity norm 8.093555e-10 > 1 norm 9.681583e-06 > > hvnguyen:jade17% aprun -n 32 ./test_matrix_read -ksp_type cg -pc_type > redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor 0 KSP > Residual norm 1.573381744277e+04 > 1 KSP Residual norm 1.072600099318e-06 > 2 KSP Residual norm 2.340608377807e-16 Time in PETSc solver: > 34.968400 seconds > The number of iteration = 2 > The solution residual error = 2.340608e-16 > 2 norm 3.767959e-08 > infinity norm 5.213003e-10 > 1 norm 5.790317e-06 > > > From ahuramazda10 at gmail.com Fri Nov 21 07:52:43 2008 From: ahuramazda10 at gmail.com (Santolo Felaco) Date: Fri, 21 Nov 2008 14:52:43 +0100 Subject: Checkpoint in PETSc Message-ID: <5f76eef60811210552r1a06fba1h8c503381f97563ed@mail.gmail.com> Hi, I have to implement the checkpoint of function KSPSolve_CG. I have many problem. 1- Can I to duplicate the KSP object? Can I send the KSP objcet to checkpoint process? 2 - I'm using FT-MPI, in case of fault can I set (do not recreate) the new communicator to object of alive process? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jarunan.Panyasantisuk at eleves.ec-nantes.fr Mon Nov 24 08:09:56 2008 From: Jarunan.Panyasantisuk at eleves.ec-nantes.fr (Panyasantisuk Jarunan) Date: Mon, 24 Nov 2008 15:09:56 +0100 Subject: Interface to BlockSolve95 and ML/Trilinos Message-ID: <20081124150956.2ahumilcwqgwww8o@webmail.ec-nantes.fr> Hello, I need some advice for interfacing PETSc to BlockSolve95 and ML/Trilinos. For both I can compile but .. For BlockSolve95, I am trying to use the command MatCreateMPIRowbs When I execute, PETSc error said... [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type! [0]PETSC ERROR: Unknown Mat type given: mpirowbs! Almost the same for ML/Trilinos, I am trying to use call PCSetType(pc,PCML,ierr) And when I execute, [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type! [1]PETSC ERROR: Unable to find requested PC type ml! Do I have to include includefile directly in the beginning of my program? Best regards, Jarunan PANYASANTISUK -- Jarunan PANYASANTISUK MSc. in Computational Mechanics Erasmus Mundus Master Program Ecole Centrale de Nantes 1, rue de la no?, 44321 NANTES, FRANCE From balay at mcs.anl.gov Mon Nov 24 09:15:19 2008 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 24 Nov 2008 09:15:19 -0600 (CST) Subject: Interface to BlockSolve95 and ML/Trilinos In-Reply-To: <20081124150956.2ahumilcwqgwww8o@webmail.ec-nantes.fr> References: <20081124150956.2ahumilcwqgwww8o@webmail.ec-nantes.fr> Message-ID: You have to install PETSc with BlockSolve95 and ML. For ML - you can install it with PETSc configure - but BlockSolve95 - you have to install it separately and specify it to PETSc configure. For eg: ./config/configure.py --download-ml=1 --with-blocksolve95-include=-I/PATH_TO_INC --with-blocksolve95-lib=/Path_T0/libBS95.a [other options] Satish On Mon, 24 Nov 2008, Panyasantisuk Jarunan wrote: > Hello, > > I need some advice for interfacing PETSc to BlockSolve95 and ML/Trilinos. > For both I can compile but .. > > For BlockSolve95, I am trying to use the command MatCreateMPIRowbs > When I execute, PETSc error said... > [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external > package needed for type! > [0]PETSC ERROR: Unknown Mat type given: mpirowbs! > > Almost the same for ML/Trilinos, I am trying to use call > PCSetType(pc,PCML,ierr) > And when I execute, [1]PETSC ERROR: Unknown type. Check for miss-spelling or > missing external package needed for type! > [1]PETSC ERROR: Unable to find requested PC type ml! > > Do I have to include includefile directly in the beginning of my program? > > > Best regards, > Jarunan PANYASANTISUK > > From 100039048 at alumnos.uc3m.es Tue Nov 25 16:46:03 2008 From: 100039048 at alumnos.uc3m.es (=?ISO-8859-1?Q?Ignacio_Mart=EDnez?=) Date: Tue, 25 Nov 2008 23:46:03 +0100 Subject: PETSc Global Vector to local process vector Message-ID: <2ee2d8330811251446p2ce27d8bia7ed6b32fb8f46db@mail.gmail.com> Hello, I'm a beginner user, and I have a problem: I have a routine in Fortran and I just want to parallelize few lines with PETSc, I'm using DA in order to gather the results, from local vectors to a global vector. It works fine. But I need to return those values to the Fortran routine, and I only get access to local process values. I try to create a sequential vectors with an scatter to get that values. But, PETSc returns an error, something like: "Cannot create VECSEQ on more than one process!" (I try to create the vector only in one process). Exactly in using DAGlobalToNaturalAllCreate and VecScatterBegin to get global values in to a process. And my question is if someone has tried to do something like this, and how has done it. Or if could you give me some orientation? Thank you in advance, Ignacio From bsmith at mcs.anl.gov Tue Nov 25 17:08:10 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 25 Nov 2008 17:08:10 -0600 Subject: PETSc Global Vector to local process vector In-Reply-To: <2ee2d8330811251446p2ce27d8bia7ed6b32fb8f46db@mail.gmail.com> References: <2ee2d8330811251446p2ce27d8bia7ed6b32fb8f46db@mail.gmail.com> Message-ID: Your message is unclear. DAGlobalToNaturalAllCreate() will give you a scatter to get ALL the values on EACH process in the "natural ordering". Do you wish to get ALL the values onto a SINGLE process (and nothing on the rest)? If so, here is how you do it. Vec natural,allonme; VecScatter tozero; DACreateNaturalVector(da,&natural); VecScatterCreateToZero(natural,&tozero,&allonme); /* now to get the values on the that 0th process */ DAGlobalToNaturalBegin(da,vec,INSERT_VALUES,natural); DAGlobalToNaturalEnd(da,vec,INSERT_VALUES,natural); VecScatterBegin(tozero,natural,allonme,INSERT_VALUES,SCATTER_FORWARD); VecScatterEnd(tozero,natural,allonme,INSERT_VALUES,SCATTER_FORWARD); Now process 0 will have a sequential vector that contains the entire global vector. All other processes will have a sequential vector of size 0. ALL processes, of course, must call the DAGlobalToNatural.... and VecScatter... routines. It is possible to reverse this operation too, to get values from process 0 out to the other processes. If this was not your question, then please clarify what you wish to do. Barry On Nov 25, 2008, at 4:46 PM, Ignacio Mart?nez wrote: > Hello, > > I'm a beginner user, and I have a problem: > > I have a routine in Fortran and I just want to parallelize few lines > with PETSc, I'm using DA in order to gather the results, from local > vectors to a global vector. > It works fine. But I need to return those values to the Fortran > routine, and I only get access to local process values. > > I try to create a sequential vectors with an scatter to get that > values. But, PETSc returns an error, something like: "Cannot create > VECSEQ on more than one process!" (I try to create the vector only in > one process). > > Exactly in using DAGlobalToNaturalAllCreate and VecScatterBegin to > get global values in to a process. > > And my question is if someone has tried to do something like this, and > how has done it. Or if could you give me some orientation? > > Thank you in advance, > Ignacio > From 100039048 at alumnos.uc3m.es Tue Nov 25 17:54:38 2008 From: 100039048 at alumnos.uc3m.es (=?ISO-8859-1?Q?Ignacio_Mart=EDnez?=) Date: Wed, 26 Nov 2008 00:54:38 +0100 Subject: PETSc Global Vector to local process vector In-Reply-To: References: <2ee2d8330811251446p2ce27d8bia7ed6b32fb8f46db@mail.gmail.com> Message-ID: <2ee2d8330811251554v6ed03341o862c82a994d1790d@mail.gmail.com> My apologies, I want all values in all processor, following your indications I guess I need to use something like: Vec natural,allonme; VecScatter tozero; DACreateNaturalVector(da,&natural); /***********************************************************/ VecScatterCreateToAll(natural,&tozero,&allonme); /***********************************************************/ /* now to get the values on the that 0th process */ DAGlobalToNaturalBegin(da,vec,INSERT_VALUES,natural); DAGlobalToNaturalEnd(da,vec,INSERT_VALUES,natural); VecScatterBegin(tozero,natural,allonme,INSERT_VALUES,SCATTER_FORWARD); VecScatterEnd(tozero,natural,allonme,INSERT_VALUES,SCATTER_FORWARD); And all process should have all data. Is this right? Thank you very much, Ignacio 2008/11/26 Barry Smith : > > Your message is unclear. > > DAGlobalToNaturalAllCreate() will give you a scatter to get ALL the values > on EACH process in the "natural ordering". > > Do you wish to get ALL the values onto a SINGLE process (and nothing > on the rest)? If so, here is how you do it. > > Vec natural,allonme; > VecScatter tozero; > > DACreateNaturalVector(da,&natural); > VecScatterCreateToZero(natural,&tozero,&allonme); > > /* now to get the values on the that 0th process */ > > DAGlobalToNaturalBegin(da,vec,INSERT_VALUES,natural); > DAGlobalToNaturalEnd(da,vec,INSERT_VALUES,natural); > VecScatterBegin(tozero,natural,allonme,INSERT_VALUES,SCATTER_FORWARD); > VecScatterEnd(tozero,natural,allonme,INSERT_VALUES,SCATTER_FORWARD); > > Now process 0 will have a sequential vector that contains the entire > global vector. > All other processes will have a sequential vector of size 0. ALL processes, > of course, > must call the DAGlobalToNatural.... and VecScatter... routines. > > It is possible to reverse this operation too, to get values from process 0 > out to the other processes. > > If this was not your question, then please clarify what you wish to do. > > Barry > > > On Nov 25, 2008, at 4:46 PM, Ignacio Mart?nez wrote: > >> Hello, >> >> I'm a beginner user, and I have a problem: >> >> I have a routine in Fortran and I just want to parallelize few lines >> with PETSc, I'm using DA in order to gather the results, from local >> vectors to a global vector. >> It works fine. But I need to return those values to the Fortran >> routine, and I only get access to local process values. >> >> I try to create a sequential vectors with an scatter to get that >> values. But, PETSc returns an error, something like: "Cannot create >> VECSEQ on more than one process!" (I try to create the vector only in >> one process). >> >> Exactly in using DAGlobalToNaturalAllCreate and VecScatterBegin to >> get global values in to a process. >> >> And my question is if someone has tried to do something like this, and >> how has done it. Or if could you give me some orientation? >> >> Thank you in advance, >> Ignacio >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Nov 25 19:41:03 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 25 Nov 2008 19:41:03 -0600 Subject: PETSc Global Vector to local process vector In-Reply-To: <2ee2d8330811251554v6ed03341o862c82a994d1790d@mail.gmail.com> References: <2ee2d8330811251446p2ce27d8bia7ed6b32fb8f46db@mail.gmail.com> <2ee2d8330811251554v6ed03341o862c82a994d1790d@mail.gmail.com> Message-ID: Looks ok. On Nov 25, 2008, at 5:54 PM, Ignacio Mart?nez wrote: > > My apologies, I want all values in all processor, following your > indications I guess I need to use something like: > > Vec natural,allonme; > VecScatter tozero; > > DACreateNaturalVector(da,&natural); > /***********************************************************/ > VecScatterCreateToAll(natural,&tozero,&allonme); > /***********************************************************/ > /* now to get the values on the that 0th process */ > > DAGlobalToNaturalBegin(da,vec,INSERT_VALUES,natural); > DAGlobalToNaturalEnd(da,vec,INSERT_VALUES,natural); > > VecScatterBegin(tozero,natural,allonme,INSERT_VALUES,SCATTER_FORWARD); > VecScatterEnd(tozero,natural,allonme,INSERT_VALUES,SCATTER_FORWARD); > > And all process should have all data. Is this right? > > Thank you very much, > Ignacio > > 2008/11/26 Barry Smith : > > > > Your message is unclear. > > > > DAGlobalToNaturalAllCreate() will give you a scatter to get ALL > the values > > on EACH process in the "natural ordering". > > > > Do you wish to get ALL the values onto a SINGLE process (and > nothing > > on the rest)? If so, here is how you do it. > > > > Vec natural,allonme; > > VecScatter tozero; > > > > DACreateNaturalVector(da,&natural); > > VecScatterCreateToZero(natural,&tozero,&allonme); > > > > /* now to get the values on the that 0th process */ > > > > DAGlobalToNaturalBegin(da,vec,INSERT_VALUES,natural); > > DAGlobalToNaturalEnd(da,vec,INSERT_VALUES,natural); > > > VecScatterBegin(tozero,natural,allonme,INSERT_VALUES,SCATTER_FORWARD); > > > VecScatterEnd(tozero,natural,allonme,INSERT_VALUES,SCATTER_FORWARD); > > > > Now process 0 will have a sequential vector that contains the > entire > > global vector. > > All other processes will have a sequential vector of size 0. ALL > processes, > > of course, > > must call the DAGlobalToNatural.... and VecScatter... routines. > > > > It is possible to reverse this operation too, to get values from > process 0 > > out to the other processes. > > > > If this was not your question, then please clarify what you wish > to do. > > > > Barry > > > > > > On Nov 25, 2008, at 4:46 PM, Ignacio Mart?nez wrote: > > > >> Hello, > >> > >> I'm a beginner user, and I have a problem: > >> > >> I have a routine in Fortran and I just want to parallelize few > lines > >> with PETSc, I'm using DA in order to gather the results, from local > >> vectors to a global vector. > >> It works fine. But I need to return those values to the Fortran > >> routine, and I only get access to local process values. > >> > >> I try to create a sequential vectors with an scatter to get that > >> values. But, PETSc returns an error, something like: "Cannot create > >> VECSEQ on more than one process!" (I try to create the vector > only in > >> one process). > >> > >> Exactly in using DAGlobalToNaturalAllCreate and VecScatterBegin to > >> get global values in to a process. > >> > >> And my question is if someone has tried to do something like > this, and > >> how has done it. Or if could you give me some orientation? > >> > >> Thank you in advance, > >> Ignacio > >> > > > > > From Jarunan.Panyasantisuk at eleves.ec-nantes.fr Wed Nov 26 08:43:53 2008 From: Jarunan.Panyasantisuk at eleves.ec-nantes.fr (Panyasantisuk Jarunan) Date: Wed, 26 Nov 2008 15:43:53 +0100 Subject: BlockSolve95 error Message-ID: <20081126154353.rrn4jl3k440sco4w@webmail.ec-nantes.fr> Hello, Could you please explain the error I got from running code with blocksolve95?: ERROR: Code -30 occured at Line 103 in File BMcomp_msg.c: Too many messages in comp_msg ERROR: Code -30 occured at Line 48 in File BSsetup_factor.c I cannot find BMcomp_msg.c and BSsetup_factor.c anywhere. For running I use: mpirun -np 3 -machinefile machines ./test2 It works with other test cases though. In the fortran code: call MatCreate(PETSC_COMM_WORLD,D,ierr) call MatSetSizes(D,N,N,PETSC_DETERMINE,PETSC_DETERMINE, $ ierr) call MatSetType(D,MATMPIROWBS,ierr) call MatSetFromOptions(D,ierr) call MatMPIRowbsSetPreallocation(D,nz,PETSC_NULL_INTEGER,ierr) ... ... call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) call KSPSetOperators(ksp,D,D,DIFFERENT_NONZERO_PATTERN,ierr) call KSPGetPC(ksp,pc,ierr) call PCSetType(pc,PCILU,ierr) Thank you before hand Regards, Jarunan -- Jarunan PANYASANTISUK MSc. in Computational Mechanics Erasmus Mundus Master Program Ecole Centrale de Nantes 1, rue de la no?, 44321 NANTES, FRANCE From ahuramazda10 at gmail.com Wed Nov 26 13:24:32 2008 From: ahuramazda10 at gmail.com (Santolo Felaco) Date: Wed, 26 Nov 2008 20:24:32 +0100 Subject: Cannot Message-ID: <5f76eef60811261124h4d6d68b0j234f4ff8caade3b6@mail.gmail.com> Hi, I cannot to KPS's struct field. For exemple if I try to access to field its or vec_sol I have this error of compiling: dereferencing pointer to incomplete type. Can you help me? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From ahuramazda10 at gmail.com Wed Nov 26 13:25:17 2008 From: ahuramazda10 at gmail.com (Santolo Felaco) Date: Wed, 26 Nov 2008 20:25:17 +0100 Subject: Cannot access KSP's struct field Message-ID: <5f76eef60811261125u10263343h9173f1e4d0c511d5@mail.gmail.com> Hi, I cannot to KPS's struct field. For exemple if I try to access to field its or vec_sol I have this error of compiling: dereferencing pointer to incomplete type. Can you help me? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Wed Nov 26 13:41:27 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 26 Nov 2008 16:41:27 -0300 Subject: Cannot access KSP's struct field In-Reply-To: <5f76eef60811261125u10263343h9173f1e4d0c511d5@mail.gmail.com> References: <5f76eef60811261125u10263343h9173f1e4d0c511d5@mail.gmail.com> Message-ID: On Wed, Nov 26, 2008 at 4:25 PM, Santolo Felaco wrote: > Hi, I cannot to KPS's struct field. For exemple if I try to access to field > its or vec_sol I have this error of compiling: dereferencing pointer to > incomplete type. Can you help me? Thanks Use KSPGetSolution() to get the solution vector, or KSPGetRhs() for the rhs. In short, see the KSP manual pages, there are a lot of calls for getting internal stuff. Anyway, If you need to access some struct internals, then you have to add into your source files #include "private/kspimpl.h" but NEVER do that unless you REALLY need it (e.g., because you are implementing a brand-new KSP subtype) -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From ahuramazda10 at gmail.com Wed Nov 26 14:21:17 2008 From: ahuramazda10 at gmail.com (Santolo Felaco) Date: Wed, 26 Nov 2008 21:21:17 +0100 Subject: Cannot access KSP's struct field In-Reply-To: References: <5f76eef60811261125u10263343h9173f1e4d0c511d5@mail.gmail.com> Message-ID: <5f76eef60811261221w6f96aca6m40997f378054bf57@mail.gmail.com> Thanks. 2008/11/26 Lisandro Dalcin > On Wed, Nov 26, 2008 at 4:25 PM, Santolo Felaco > wrote: > > Hi, I cannot to KPS's struct field. For exemple if I try to access to > field > > its or vec_sol I have this error of compiling: dereferencing pointer to > > incomplete type. Can you help me? Thanks > > Use KSPGetSolution() to get the solution vector, or KSPGetRhs() for > the rhs. In short, see the KSP manual pages, there are a lot of calls > for getting internal stuff. > > Anyway, If you need to access some struct internals, then you have to > add into your source files > > #include "private/kspimpl.h" > > but NEVER do that unless you REALLY need it (e.g., because you are > implementing a brand-new KSP subtype) > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Nov 26 15:19:33 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 26 Nov 2008 15:19:33 -0600 Subject: BlockSolve95 error In-Reply-To: <20081126154353.rrn4jl3k440sco4w@webmail.ec-nantes.fr> References: <20081126154353.rrn4jl3k440sco4w@webmail.ec-nantes.fr> Message-ID: I have no idea. BlockSolve has not been supported by its developers for like 10 years. Those messages are coming from inside BlockSolve which we don't support. I'm sorry for the terse response but we cannot support all the orphaned parallel numerical software in the world. Barry On Nov 26, 2008, at 8:43 AM, Panyasantisuk Jarunan wrote: > Hello, > > Could you please explain the error I got from running code with > blocksolve95?: > ERROR: Code -30 occured at Line 103 in File BMcomp_msg.c: Too many > messages in comp_msg > ERROR: Code -30 occured at Line 48 in File BSsetup_factor.c > > I cannot find BMcomp_msg.c and BSsetup_factor.c anywhere. > > For running I use: mpirun -np 3 -machinefile machines ./test2 > It works with other test cases though. > > In the fortran code: > > call MatCreate(PETSC_COMM_WORLD,D,ierr) > call MatSetSizes(D,N,N,PETSC_DETERMINE,PETSC_DETERMINE, > $ ierr) > call MatSetType(D,MATMPIROWBS,ierr) > call MatSetFromOptions(D,ierr) > call MatMPIRowbsSetPreallocation(D,nz,PETSC_NULL_INTEGER,ierr) > ... > ... > > call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > call KSPSetOperators(ksp,D,D,DIFFERENT_NONZERO_PATTERN,ierr) > call KSPGetPC(ksp,pc,ierr) > call PCSetType(pc,PCILU,ierr) > > Thank you before hand > > Regards, > Jarunan > > -- > Jarunan PANYASANTISUK > MSc. in Computational Mechanics > Erasmus Mundus Master Program > Ecole Centrale de Nantes > 1, rue de la no?, 44321 NANTES, FRANCE > > > From bsmith at mcs.anl.gov Wed Nov 26 15:34:08 2008 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 26 Nov 2008 15:34:08 -0600 Subject: Cannot In-Reply-To: <5f76eef60811261124h4d6d68b0j234f4ff8caade3b6@mail.gmail.com> References: <5f76eef60811261124h4d6d68b0j234f4ff8caade3b6@mail.gmail.com> Message-ID: <25DA43CF-A81E-467A-ADD1-A8963C6E4F83@mcs.anl.gov> PETSc uses "data encapsulation" to NOT allow most code to access the internals of its objects (like KSP, Mat etc.) This is a very powerful feature of object oriented programming. KSP provides a large number of routines (defined in include/ petscksp.h) to accessing/changing information in the KSP object. It is intended that this is how one accesses the data. If you really, really want to access the data then #include "private/kspimpl.h" and that gives access to the structure definition. Barry On Nov 26, 2008, at 1:24 PM, Santolo Felaco wrote: > Hi, I cannot to KPS's struct field. For exemple if I try to access > to field its or vec_sol I have this error of compiling: > dereferencing pointer to incomplete type. Can you help me? Thanks From keita at cray.com Wed Nov 26 15:55:01 2008 From: keita at cray.com (Keita Teranishi) Date: Wed, 26 Nov 2008 15:55:01 -0600 Subject: BlockSolve95 error In-Reply-To: References: <20081126154353.rrn4jl3k440sco4w@webmail.ec-nantes.fr> Message-ID: <925346A443D4E340BEB20248BAFCDBDF088166EA@CFEVS1-IP.americas.cray.com> Jarunan, Make the value for MAX_NUM_MSGS bigger in BlockSolve95/include/BSprivate.h. > #define MAX_NUM_MSGS 10000000 You will need to change the values of other "MAX_XXXX" variables in the header files because computers nowadays have much bigger capacity/capability than those in mid 90's. ================================ Keita Teranishi Math Software Group Cray, Inc. keita at cray.com ================================ -----Original Message----- From: owner-petsc-users at mcs.anl.gov [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry Smith Sent: Wednesday, November 26, 2008 3:20 PM To: petsc-users at mcs.anl.gov Subject: Re: BlockSolve95 error I have no idea. BlockSolve has not been supported by its developers for like 10 years. Those messages are coming from inside BlockSolve which we don't support. I'm sorry for the terse response but we cannot support all the orphaned parallel numerical software in the world. Barry On Nov 26, 2008, at 8:43 AM, Panyasantisuk Jarunan wrote: > Hello, > > Could you please explain the error I got from running code with > blocksolve95?: > ERROR: Code -30 occured at Line 103 in File BMcomp_msg.c: Too many > messages in comp_msg > ERROR: Code -30 occured at Line 48 in File BSsetup_factor.c > > I cannot find BMcomp_msg.c and BSsetup_factor.c anywhere. > > For running I use: mpirun -np 3 -machinefile machines ./test2 > It works with other test cases though. > > In the fortran code: > > call MatCreate(PETSC_COMM_WORLD,D,ierr) > call MatSetSizes(D,N,N,PETSC_DETERMINE,PETSC_DETERMINE, > $ ierr) > call MatSetType(D,MATMPIROWBS,ierr) > call MatSetFromOptions(D,ierr) > call MatMPIRowbsSetPreallocation(D,nz,PETSC_NULL_INTEGER,ierr) > ... > ... > > call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > call KSPSetOperators(ksp,D,D,DIFFERENT_NONZERO_PATTERN,ierr) > call KSPGetPC(ksp,pc,ierr) > call PCSetType(pc,PCILU,ierr) > > Thank you before hand > > Regards, > Jarunan > > -- > Jarunan PANYASANTISUK > MSc. in Computational Mechanics > Erasmus Mundus Master Program > Ecole Centrale de Nantes > 1, rue de la no?, 44321 NANTES, FRANCE > > > From e.tadeu at gmail.com Fri Nov 28 13:18:03 2008 From: e.tadeu at gmail.com (Edson Tadeu) Date: Fri, 28 Nov 2008 17:18:03 -0200 Subject: PreCheck on SNES Line Search and solution constraints (PreCheck not working) Message-ID: Hi, I'm using the Cubic Line Search algorithm to solve some non-linear systems where the solution vector is constrained to a given space, e.g., some of its elements are only allowed in the range 0 <= x <= 1... outside this domain, the evaluation of the Jacobian is "undefined", and may lead to wrong directions or unwanted solutions. I'm trying to use a PreCheck to limit the value of lambda, so that the solution vector would never leave the domain. Is this the right approach? Anyway, I'm facing the following problem: I'm calling SNESLineSearchSetPreCheck(snes, MyPreCheck, ctx), but it seems that the PreCheck is not being set correctly... it actually never enters the MyPreCheck function when solving. I'm using petsc-2.3.3-p13... I've tried to trace the problem inside Petsc code, and it seems that the problem is that this line: ierr = PetscObjectQueryFunction((PetscObject)snes,"SNESLineSearchSetPreCheck_C",(void (**)(void))&f);CHKERRQ(ierr); is actually returning PETSC_NULL on f. The calling sequence is the following: SNESCreate SNESSetFunction SNESSetJacobian SNESLineSearchSetPreCheck SNESMonitorSet SNESSetFromOptions SNESSetApplicationContext SNESSolve Thanks for any help, Edson -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Nov 28 13:37:50 2008 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 28 Nov 2008 13:37:50 -0600 Subject: PreCheck on SNES Line Search and solution constraints (PreCheck not working) In-Reply-To: References: Message-ID: On Fri, Nov 28, 2008 at 1:18 PM, Edson Tadeu wrote: > Hi, > > I'm using the Cubic Line Search algorithm to solve some non-linear systems > where the solution vector is constrained to a given space, e.g., some of its > elements are only allowed in the range 0 <= x <= 1... outside this domain, > the evaluation of the Jacobian is "undefined", and may lead to wrong > directions or unwanted solutions. > > I'm trying to use a PreCheck to limit the value of lambda, so that the > solution vector would never leave the domain. Is this the right approach? Yes. > Anyway, I'm facing the following problem: I'm calling > SNESLineSearchSetPreCheck(snes, MyPreCheck, ctx), but it seems that the > PreCheck is not being set correctly... it actually never enters the > MyPreCheck function when solving. I'm using petsc-2.3.3-p13... I've tried to > trace the problem inside Petsc code, and it seems that the problem is that > this line: > > ierr = > PetscObjectQueryFunction((PetscObject)snes,"SNESLineSearchSetPreCheck_C",(void > (**)(void))&f);CHKERRQ(ierr); > > is actually returning PETSC_NULL on f. > > > The calling sequence is the following: > > SNESCreate > SNESSetFunction > SNESSetJacobian > SNESLineSearchSetPreCheck > SNESMonitorSet > SNESSetFromOptions The problem is that the SNES type is not set until SNESSetFromOptions(). Therefore, you called SNESLineSearch*() without knowing whether the solver actually did line searches, and thus it was ignored. If you move the call after SetFromOptions, it should work. Matt > SNESSetApplicationContext > SNESSolve > > > Thanks for any help, > Edson > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From Lars.Rindorf at teknologisk.dk Fri Nov 28 14:40:19 2008 From: Lars.Rindorf at teknologisk.dk (Lars Rindorf) Date: Fri, 28 Nov 2008 21:40:19 +0100 Subject: Timesolver: Newmark method? Message-ID: Dear all Is it planned to do the Newmark time scheme for second order time differential equations in petsc? KR _____________________ Lars Rindorf Technology consultant, Ph.D. Mobile +45 7220 3367 Surfaces and microtechnology Danish Technological Institute Gregersensvej 2630 Taastrup Denmark http://www.dti.dk Phone +45 7220 2000 -------------- next part -------------- An HTML attachment was scrubbed... URL: