From renzhengyong at gmail.com Wed Jun 1 01:01:08 2011 From: renzhengyong at gmail.com (RenZhengYong) Date: Wed, 1 Jun 2011 08:01:08 +0200 Subject: [petsc-users] question on extracting a column from a matrix In-Reply-To: <97B4BA05-E924-4D6F-A316-2CDDC819267F@mcs.anl.gov> References: <97B4BA05-E924-4D6F-A316-2CDDC819267F@mcs.anl.gov> Message-ID: Hi, Hong and Barry, Thanks a lot. Have a nice day. Zhengyong On Tue, May 31, 2011 at 9:48 PM, Barry Smith wrote: > > With petsc-dev > http://www.mcs.anl.gov/petsc/petsc-as/developers/index.html there are very > easy ways to solve the Schur complement systems using PCFIELDSPLIT see > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/PC/PCFIELDSPLIT.html > > > Barry > > > On May 31, 2011, at 2:41 PM, Hong Zhang wrote: > > > This is virtually SCHUR complement. You may take a look at > > petsc-dev/src/ksp/ksp/examples/tests/ex11.c and > > petsc-dev/src/ksp/ksp/examples/tests/makefile: runex11 > > > > Hong > > > > On Tue, May 31, 2011 at 8:29 AM, RenZhengYong > wrote: > >> I have a linear system of equation like > >> [A B] [E]= [C] > >> [F G] [S]= [D], > >> where size of matrix A is NxN, B is NxNs, F is Ns_N, G is Ns_Ns; size of > >> vector E is N, S is Ns, C is N and D is N. > >> Instead of solving them together, I prefer to solve it by: > >> (1) AX=B, AY=C; > >> (2) QS=P, Q= G-FX, P=D-FY, > >> due to the dense property of G. > >> > >> In step (1), I need to extract a new "vec" Bi from "mat" B, according to > the > >> way of PETSc telling me. My question is that could I do it as follows, > >> for a given sparse matrix B, first get transpose of B by > "MatTranspose(B, > >> MatReuse, B_t); then, form a new "vec" Bi by "MatGetRow(...)"? > >> If answer is yes, is it expensive? > >> > >> > >> > >> > >> -- > >> Zhengyong Ren > >> AUG Group, Institute of Geophysics > >> Department of Geosciences, ETH Zurich > >> NO H 47 Sonneggstrasse 5 > >> CH-8092, Z?rich, Switzerland > >> Tel: +41 44 633 37561 > >> e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch > >> Gmail: renzhengyong at gmail.com > >> > > -- Zhengyong Ren AUG Group, Institute of Geophysics Department of Geosciences, ETH Zurich NO H 47 Sonneggstrasse 5 CH-8092, Z?rich, Switzerland Tel: +41 44 633 37561 e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch Gmail: renzhengyong at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Wed Jun 1 03:02:17 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 1 Jun 2011 10:02:17 +0200 Subject: [petsc-users] [petsc-maint #73793] Weird behavior of the Jacobian of a complex system In-Reply-To: <4DE54D3B.7020500@gatech.edu> References: <4DDC37CE.70201@gatech.edu> <4DDD5CC5.1020506@gatech.edu> <4DDD5ECD.3050009@gatech.edu> <4DDD63B4.5070502@gatech.edu> <4DDD6665.7060401@gatech.edu> <4DDD6956.5080802@gatech.edu> <4DDD8616.2070103@gatech.edu> <4DDEB696.7000307@gatech.edu> <4DDECDDE.1050009@gatech.edu> <4DDECEF0.9000503@gatech.edu> <4DDEDEC1.5070207@gatech.edu> <4DE54D3B.7020500@gatech.edu> Message-ID: Please email the list rather than me personally. On Tue, May 31, 2011 at 22:19, Jeremy Riousset wrote: > Am I correct if I say that the alpha method becomes an implicit Runge-Kutta > for alpha 0.5 and a classic Euler method for alpha = 1? There are two values of "alpha" which could be taken to be the same which would give you a theta method. If they are both equal to one half, you get the midpoint method. Please read the cited papers for specific details. Our implementation recasts the method into a more convenient form to use, but it is equivalent. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenglun.wei at gmail.com Wed Jun 1 11:47:38 2011 From: zhenglun.wei at gmail.com (Alan Wei) Date: Wed, 1 Jun 2011 11:47:38 -0500 Subject: [petsc-users] A question on "memory access out of range" in mpi running Message-ID: Hi guys, I hope you're having a nice day. I was trying to run ex29.c in src/ksp/ksp/example/tutorial. It is normal to go through without MPI. However, with the same grid size, if I run it with 4 processes with MPI, a error message of "Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range" will show up. Did anyone have this kind of issue before. I guess this is probably caused by the way I launched the 'mpiexec'. Any idea on it? The full error message is attached. Thanks in advance. best, Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [1]PETSC ERROR: likely location of problem given in stack below [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [1]PETSC ERROR: INSTEAD the line number of the start of the function [1]PETSC ERROR: is given. [1]PETSC ERROR: [1] VecView_VTK line 230 src/ksp/ksp/examples/tutorials/ex29.c [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Signal received! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Development HG revision: f4c0045a3598c355546a497a4596696abe625181 HG Date: Thu May 26 14:04:32 2011 -0500 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------[2]PETSC ERROR: likely location of problem given in stack below [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [2]PETSC ERROR: INSTEAD the line number of the start of the function [2]PETSC ERROR: is given. [2]PETSC ERROR: [2] VecView_VTK line 230 src/ksp/ksp/examples/tutorials/ex29.c [2]PETSC ERROR: --------------------- Error Message ------------------------------------ [2]PETSC ERROR: Signal received! [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Petsc Development HG revision: f4c0045a3598c355546a497a4596696abe625181 HG Date: Thu May 26 14:04:32 2011 -0500 [2]PETSC ERROR: See docs/changes/index.html for recent updates. [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [2]PETSC ERROR: See docs/index.html for manual pages. [2]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ./ex29 on a arch-linu named l2118a-linux.soecs.ku.edu by zlwei Tue May 31 15:16:53 2011 [1]PETSC ERROR: Libraries linked from /home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib [1]PETSC ERROR: Configure run at Thu May 26 15:32:02 2011 [1]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran --download-f-blas-lapack=1 --download-mpich=1 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file [cli_1]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 ------ [2]PETSC ERROR: ./ex29 on a arch-linu named l2118a-linux.soecs.ku.edu by zlwei Tue May 31 15:16:53 2011 [2]PETSC ERROR: Libraries linked from /home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib [2]PETSC ERROR: Configure run at Thu May 26 15:32:02 2011 [2]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran --download-f-blas-lapack=1 --download-mpich=1 [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: User provided function() line 0 in unknown directory unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2 [cli_2]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2 [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [3]PETSC ERROR: likely location of problem given in stack below [3]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [3]PETSC ERROR: INSTEAD the line number of the start of the function [3]PETSC ERROR: is given. [3]PETSC ERROR: [3] VecView_VTK line 230 src/ksp/ksp/examples/tutorials/ex29.c [3]PETSC ERROR: --------------------- Error Message ------------------------------------ [3]PETSC ERROR: Signal received! [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Petsc Development HG revision: f4c0045a3598c355546a497a4596696abe625181 HG Date: Thu May 26 14:04:32 2011 -0500 [3]PETSC ERROR: See docs/changes/index.html for recent updates. [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [3]PETSC ERROR: See docs/index.html for manual pages. [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: ./ex29 on a arch-linu named l2118a-linux.soecs.ku.edu by zlwei Tue May 31 15:16:53 2011 [3]PETSC ERROR: Libraries linked from /home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib [3]PETSC ERROR: Configure run at Thu May 26 15:32:02 2011 [3]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran --download-f-blas-lapack=1 --download-mpich=1 [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: User provided function() line 0 in unknown directory unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3 [cli_3]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] VecView_VTK line 230 src/ksp/ksp/examples/tutorials/ex29.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: f4c0045a3598c355546a497a4596696abe625181 HG Date: Thu May 26 14:04:32 2011 -0500 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ex29 on a arch-linu named l2118a-linux.soecs.ku.edu by zlwei Tue May 31 15:16:53 2011 [0]PETSC ERROR: Libraries linked from /home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Thu May 26 15:32:02 2011 [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran --download-f-blas-lapack=1 --download-mpich=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 [cli_0]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 From knepley at gmail.com Wed Jun 1 11:50:23 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 1 Jun 2011 11:50:23 -0500 Subject: [petsc-users] A question on "memory access out of range" in mpi running In-Reply-To: References: Message-ID: On Wed, Jun 1, 2011 at 11:47 AM, Alan Wei wrote: > Hi guys, > I hope you're having a nice day. > I was trying to run ex29.c in src/ksp/ksp/example/tutorial. It is normal > to go through without MPI. However, with the same grid size, if I run it > with 4 processes with MPI, a error message of "Caught signal number 11 SEGV: > Segmentation Violation, probably memory access out of range" will show up. > Did anyone have this kind of issue before. I guess this is probably caused > by the way I launched the 'mpiexec'. Any idea on it? The full error message > is attached. > Thanks in advance. > No, there is some problem with the VTK output for that. You can comment it out. I will look at fixing it. Matt > best, > Alan > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenglun.wei at gmail.com Wed Jun 1 11:57:57 2011 From: zhenglun.wei at gmail.com (zhenglun.wei at gmail.com) Date: Wed, 01 Jun 2011 16:57:57 +0000 Subject: [petsc-users] A question on "memory access out of range" in mpi running In-Reply-To: Message-ID: <90e6ba6135a827dbb604a4a96c43@google.com> Thanks Matt, Do I need to comment out the whole function of VTK, or just certain line(s)? Alan On Jun 1, 2011 11:50am, Matthew Knepley wrote: > On Wed, Jun 1, 2011 at 11:47 AM, Alan Wei zhenglun.wei at gmail.com> wrote: > Hi guys, I hope you're having a nice day. > I was trying to run ex29.c in src/ksp/ksp/example/tutorial. It is normal > to go through without MPI. However, with the same grid size, if I run it > with 4 processes with MPI, a error message of "Caught signal number 11 > SEGV: Segmentation Violation, probably memory access out of range" will > show up. Did anyone have this kind of issue before. I guess this is > probably caused by the way I launched the 'mpiexec'. Any idea on it? The > full error message is attached. > Thanks in advance. > No, there is some problem with the VTK output for that. You can comment > it out. I will look at fixing it. > Matt > best, > Alan > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Wed Jun 1 11:58:16 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 1 Jun 2011 18:58:16 +0200 Subject: [petsc-users] A question on "memory access out of range" in mpi running In-Reply-To: References: Message-ID: On Wed, Jun 1, 2011 at 18:50, Matthew Knepley wrote: > No, there is some problem with the VTK output for that. You can comment it > out. I will look at fixing it. Even if it didn't deadlock, that code never worked since it is totally wrong for parallel. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Wed Jun 1 11:58:33 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 1 Jun 2011 18:58:33 +0200 Subject: [petsc-users] A question on "memory access out of range" in mpi running In-Reply-To: <90e6ba6135a827dbb604a4a96c43@google.com> References: <90e6ba6135a827dbb604a4a96c43@google.com> Message-ID: On Wed, Jun 1, 2011 at 18:57, wrote: > Do I need to comment out the whole function of VTK, or just certain > line(s)? Just don't call that function. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenglun.wei at gmail.com Wed Jun 1 12:01:52 2011 From: zhenglun.wei at gmail.com (zhenglun.wei at gmail.com) Date: Wed, 01 Jun 2011 17:01:52 +0000 Subject: [petsc-users] A question on "memory access out of range" in mpi running In-Reply-To: Message-ID: <20cf30549f171e473604a4a97a9a@google.com> Thanks, ^_^ On Jun 1, 2011 11:58am, Jed Brown wrote: > On Wed, Jun 1, 2011 at 18:57, zhenglun.wei at gmail.com> wrote: > Do I need to comment out the whole function of VTK, or just certain > line(s)? > Just don't call that function. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Wed Jun 1 14:53:07 2011 From: john.fettig at gmail.com (John Fettig) Date: Wed, 1 Jun 2011 15:53:07 -0400 Subject: [petsc-users] Reducing cost of MatSetValues Message-ID: What are the recommendations for reducing the cost of inserting values into an AIJ matrix? In my application (transient finite element solution of flow and heat, linear elements), this is accounting for up to 20% of overall runtime. Is this expected? I have double checked that the matrices are preallocated correctly, and I have set MAT_NEW_NONZERO_ALLOCATION_ERR and it runs without error. The matrices periodically change size/nonzero pattern, but until then the values are zeroed out and MatSetOption( mat, MAT_KEEP_NONZERO_PATTERN, PETSC_TRUE); is called. The call to MatSetValues happens on a per-element basis on the local element matrix, so one call per element. I re-activated the MAT_SetValues event and have included a -log_summary from a short run. Thanks, John ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./test on a intel-opt named lagrange with 1 processor, by jfe Wed Jun 1 15:49:20 2011 Using Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 Max Max/Min Avg Total Time (sec): 2.242e+02 1.00000 2.242e+02 Objects: 2.718e+03 1.00000 2.718e+03 Flops: 2.239e+10 1.00000 2.239e+10 2.239e+10 Flops/sec: 9.986e+07 1.00000 9.986e+07 9.986e+07 MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 MPI Reductions: 2.062e+04 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 2.2423e+02 100.0% 2.2391e+10 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 1.796e+04 87.1% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage KSPSetup 532 1.0 2.6963e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+02 0 0 0 0 1 0 0 0 0 1 0 KSPSolve 277 1.0 3.2930e+01 1.0 2.17e+10 1.0 0.0e+00 0.0e+00 9.9e+03 15 97 0 0 48 15 97 0 0 55 660 PCSetUp 277 1.0 8.3035e+00 1.0 7.52e+08 1.0 0.0e+00 0.0e+00 1.4e+03 4 3 0 0 7 4 3 0 0 8 91 PCApply 4907 1.0 9.1353e+00 1.0 7.08e+09 1.0 0.0e+00 0.0e+00 2.0e+00 4 32 0 0 0 4 32 0 0 0 775 MatMult 12821 1.0 1.5110e+01 1.0 1.34e+10 1.0 0.0e+00 0.0e+00 0.0e+00 7 60 0 0 0 7 60 0 0 0 888 MatMultAdd 3668 1.0 6.9999e-01 1.0 2.83e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 404 MatSolve 917 1.0 5.3644e-04 1.0 9.17e+02 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2 MatSOR 7336 1.0 5.7982e+00 1.0 4.62e+09 1.0 0.0e+00 0.0e+00 0.0e+00 3 21 0 0 0 3 21 0 0 0 797 MatLUFactorSym 51 1.0 3.1686e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 5.1e+01 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 51 1.0 1.2851e-04 1.0 5.10e+01 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 1068 1.0 1.4639e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 1068 1.0 7.9752e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatSetValues 89511956 1.0 4.6221e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 21 0 0 0 0 21 0 0 0 0 0 MatGetRowIJ 51 1.0 7.7248e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 51 1.0 5.0688e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+02 0 0 0 0 0 0 0 0 0 1 0 MatZeroEntries 133 1.0 1.7668e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecMax 85 1.0 6.9754e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 8.5e+01 0 0 0 0 0 0 0 0 0 0 0 VecDot 5479 1.0 3.5107e-01 1.0 8.84e+08 1.0 0.0e+00 0.0e+00 5.5e+03 0 4 0 0 27 0 4 0 0 31 2517 VecNorm 2937 1.0 1.5535e+00 1.0 4.81e+08 1.0 0.0e+00 0.0e+00 2.9e+03 1 2 0 0 14 1 2 0 0 16 310 VecCopy 955 1.0 8.6689e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 6718 1.0 9.4952e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 6937 1.0 6.1062e-01 1.0 1.15e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 5 0 0 0 0 5 0 0 0 1891 VecAYPX 4483 1.0 1.4344e-01 1.0 1.54e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 1073 VecWAXPY 6624 1.0 9.5558e-01 1.0 1.04e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 5 0 0 0 0 5 0 0 0 1089 VecAssemblyBegin 1286 1.0 5.1773e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 3.9e+03 0 0 0 0 19 0 0 0 0 21 0 VecAssemblyEnd 1286 1.0 1.2426e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 3990 1.0 6.7400e-01 1.0 3.57e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 529 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Krylov Solver 258 258 218416 0 Preconditioner 258 258 193856 0 Matrix 666 666 140921396 0 Vec 1383 1382 244199352 0 Index Set 153 153 81396 0 ======================================================================================================================== Average time to get PetscTime(): 0 #PETSc Option Table entries: -log_summary #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 Configure run at: Sun May 22 22:30:16 2011 Configure options: --with-x=0 --download-f-blas-lapack=0 --with-blas-lapack-dir=/opt/intel/Compiler/11.1/072/mkl/lib/em64t --with-mpi=1 --with-mpi-shared=1 --with-mpi-include=/usr/local/encap/hpmpi-8.01/include --with-mpi-lib="[/usr/local/encap/hpmpi-8.01/lib/linux_amd64/libmpi.so,/usr/local/encap/hpmpi-8.01/lib/linux_amd64/libfmpi.so]" --with-mpi=1 --download-mpich=no --with-debugging=0 --with-gnu-compilers=no --with-vendor-compilers=intel --with-cc=icc --with-cxx=icpc --with-fc=ifort --with-shared=1 --with-c++-support --with-clanguage=C --COPTFLAGS="-fPIC -O3 -xSSE4.2 -g -debug inline_debug_info" --CXXOPTFLAGS="-fPIC -O3 -xSSE4.2 -g -debug inline_debug_info" --FOPTFLAGS="-fPIC -O3 -xSSE4.2 -g -debug inline_debug_info" --download-scalapack=no --download-blacs=no --with-blacs=1 --with-blacs-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_blacs_lp64.a --with-blacs-include=/opt/intel/Compiler/11.1/072/mkl/include --with-scalapack=1 --with-scalapack-lib="[/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_scalapack_lp64.a,/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_lp64.a,/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_thread.a,/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_core.a]" --with-scalapack-include=/opt/intel/Compiler/11.1/072/mkl/include --download-umfpack=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-ml=1 --with-hypre=1 --download-hypre=yes ----------------------------------------- Libraries compiled on Thu May 26 16:49:30 EDT 2011 on lagrange Machine characteristics: Linux lagrange 2.6.39-0.el5.elrepo #1 SMP PREEMPT Sat May 21 04:48:38 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux Using PETSc directory: /home/jfe/local/centos/petsc-3.1-p8 Using PETSc arch: intel-opt ----------------------------------------- Using C compiler: icc -fPIC -fPIC -O3 -xSSE4.2 -g -debug inline_debug_info Using Fortran compiler: ifort -fPIC -fPIC -O3 -xSSE4.2 -g -debug inline_debug_info ----------------------------------------- Using include paths: -I/home/jfe/local/centos/petsc-3.1-p8/intel-opt/include -I/home/jfe/local/centos/petsc-3.1-p8/include -I/home/jfe/local/centos/petsc-3.1-p8/intel-opt/include -I/usr/local/encap/hpmpi-8.01/include ------------------------------------------ Using C linker: icc -fPIC -fPIC -O3 -xSSE4.2 -g -debug inline_debug_info Using Fortran linker: ifort -fPIC -fPIC -O3 -xSSE4.2 -g -debug inline_debug_info Using libraries: -Wl,-rpath,/home/jfe/local/centos/petsc-3.1-p8/intel-opt/lib -L/home/jfe/local/centos/petsc-3.1-p8/intel-opt/lib -lpetsc -Wl,-rpath,/home/jfe/local/centos/petsc-3.1-p8/intel-opt/lib -L/home/jfe/local/centos/petsc-3.1-p8/intel-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -Wl,-rpath,/opt/intel/Compiler/11.1/072/mkl/lib/em64t -L/opt/intel/Compiler/11.1/072/mkl/lib/em64t -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_lp64 -lHYPRE -Wl,-rpath,/opt/intel/Compiler/11.1/072/lib/intel64 -Wl,-rpath,/opt/intel/Compiler/11.1/072/ipp/em64t/lib -Wl,-rpath,/opt/intel/Compiler/11.1/072/tbb/intel64/cc4.1.0_libc2.4_kernel2.6.16.21/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -lstdc++ -lml -lstdc++ -lsuperlu_dist_2.4 -lparmetis -lmetis -lumfpack -lamd -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -Wl,-rpath,/usr/local/encap/hpmpi-8.01/lib/linux_amd64 -L/usr/local/encap/hpmpi-8.01/lib/linux_amd64 -lmpi -lfmpi -ldl -Wl,-rpath,/opt/intel/Compiler/11.1/072/lib/intel64 -L/opt/intel/Compiler/11.1/072/lib/intel64 -Wl,-rpath,/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -Wl,-rpath,/opt/intel/Compiler/11.1/072/mkl/lib/em64t -L/opt/intel/Compiler/11.1/072/mkl/lib/em64t -Wl,-rpath,/opt/intel/Compiler/11.1/072/tbb/intel64/cc4.1.0_libc2.4_kernel2.6.16.21/lib -L/opt/intel/Compiler/11.1/072/tbb/intel64/cc4.1.0_libc2.4_kernel2.6.16.21/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lirc_s -lifport -lifcore -lm -lpthread -lm -lstdc++ -lstdc++ -ldl -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lirc_s -ldl ------------------------------------------ From knepley at gmail.com Wed Jun 1 15:06:01 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 1 Jun 2011 15:06:01 -0500 Subject: [petsc-users] Reducing cost of MatSetValues In-Reply-To: References: Message-ID: On Wed, Jun 1, 2011 at 2:53 PM, John Fettig wrote: > What are the recommendations for reducing the cost of inserting values > into an AIJ matrix? In my application (transient finite element > solution of flow and heat, linear elements), this is accounting for up > to 20% of overall runtime. Is this expected? > It looks like the calls are taking 0.5 microseconds, but there are 89G insertions against only 13K MatMults that take about 1s apiece. This seems like an awful lot of insertions into a matrix that can be applied in 1s. The one thing I can think of is for you to try something matrix-free. However, this will typically degrade your solver performance. Matt > I have double checked that the matrices are preallocated correctly, > and I have set MAT_NEW_NONZERO_ALLOCATION_ERR and it runs without > error. > > The matrices periodically change size/nonzero pattern, but until then > the values are zeroed out and MatSetOption( mat, > MAT_KEEP_NONZERO_PATTERN, PETSC_TRUE); is called. > > The call to MatSetValues happens on a per-element basis on the local > element matrix, so one call per element. I re-activated the > MAT_SetValues event and have included a -log_summary from a short run. > > Thanks, > John > > > ************************************************************************************************************************ > *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r > -fCourier9' to print this document *** > > ************************************************************************************************************************ > > ---------------------------------------------- PETSc Performance > Summary: ---------------------------------------------- > > ./test on a intel-opt named lagrange with 1 processor, by jfe Wed Jun > 1 15:49:20 2011 > Using Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 > > Max Max/Min Avg Total > Time (sec): 2.242e+02 1.00000 2.242e+02 > Objects: 2.718e+03 1.00000 2.718e+03 > Flops: 2.239e+10 1.00000 2.239e+10 2.239e+10 > Flops/sec: 9.986e+07 1.00000 9.986e+07 9.986e+07 > MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 > MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 > MPI Reductions: 2.062e+04 1.00000 > > Flop counting convention: 1 flop = 1 real number operation of type > (multiply/divide/add/subtract) > e.g., VecAXPY() for real vectors of length > N --> 2N flops > and VecAXPY() for complex vectors of > length N --> 8N flops > > Summary of Stages: ----- Time ------ ----- Flops ----- --- > Messages --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total counts > %Total Avg %Total counts %Total > 0: Main Stage: 2.2423e+02 100.0% 2.2391e+10 100.0% 0.000e+00 > 0.0% 0.000e+00 0.0% 1.796e+04 87.1% > > > ------------------------------------------------------------------------------------------------------------------------ > See the 'Profiling' chapter of the users' manual for details on > interpreting output. > Phase summary info: > Count: number of times phase was executed > Time and Flops: Max - maximum over all processors > Ratio - ratio of maximum to minimum over all processors > Mess: number of messages sent > Avg. len: average message length > Reduct: number of global reductions > Global: entire computation > Stage: stages of a computation. Set stages with PetscLogStagePush() > and PetscLogStagePop(). > %T - percent time in this phase %F - percent flops in this > phase > %M - percent messages in this phase %L - percent message > lengths in this phase > %R - percent reductions in this phase > Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time > over all processors) > > ------------------------------------------------------------------------------------------------------------------------ > Event Count Time (sec) Flops > --- Global --- --- Stage --- Total > Max Ratio Max Ratio Max Ratio Mess Avg > len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > KSPSetup 532 1.0 2.6963e-02 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 2.0e+02 0 0 0 0 1 0 0 0 0 1 0 > KSPSolve 277 1.0 3.2930e+01 1.0 2.17e+10 1.0 0.0e+00 > 0.0e+00 9.9e+03 15 97 0 0 48 15 97 0 0 55 660 > PCSetUp 277 1.0 8.3035e+00 1.0 7.52e+08 1.0 0.0e+00 > 0.0e+00 1.4e+03 4 3 0 0 7 4 3 0 0 8 91 > PCApply 4907 1.0 9.1353e+00 1.0 7.08e+09 1.0 0.0e+00 > 0.0e+00 2.0e+00 4 32 0 0 0 4 32 0 0 0 775 > MatMult 12821 1.0 1.5110e+01 1.0 1.34e+10 1.0 0.0e+00 > 0.0e+00 0.0e+00 7 60 0 0 0 7 60 0 0 0 888 > MatMultAdd 3668 1.0 6.9999e-01 1.0 2.83e+08 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 404 > MatSolve 917 1.0 5.3644e-04 1.0 9.17e+02 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2 > MatSOR 7336 1.0 5.7982e+00 1.0 4.62e+09 1.0 0.0e+00 > 0.0e+00 0.0e+00 3 21 0 0 0 3 21 0 0 0 797 > MatLUFactorSym 51 1.0 3.1686e-04 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 5.1e+01 0 0 0 0 0 0 0 0 0 0 0 > MatLUFactorNum 51 1.0 1.2851e-04 1.0 5.10e+01 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyBegin 1068 1.0 1.4639e-04 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 1068 1.0 7.9752e-01 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatSetValues 89511956 1.0 4.6221e+01 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 21 0 0 0 0 21 0 0 0 0 0 > MatGetRowIJ 51 1.0 7.7248e-05 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatGetOrdering 51 1.0 5.0688e-04 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 1.0e+02 0 0 0 0 0 0 0 0 0 1 0 > MatZeroEntries 133 1.0 1.7668e-01 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecMax 85 1.0 6.9754e-03 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 8.5e+01 0 0 0 0 0 0 0 0 0 0 0 > VecDot 5479 1.0 3.5107e-01 1.0 8.84e+08 1.0 0.0e+00 > 0.0e+00 5.5e+03 0 4 0 0 27 0 4 0 0 31 2517 > VecNorm 2937 1.0 1.5535e+00 1.0 4.81e+08 1.0 0.0e+00 > 0.0e+00 2.9e+03 1 2 0 0 14 1 2 0 0 16 310 > VecCopy 955 1.0 8.6689e-02 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecSet 6718 1.0 9.4952e-02 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAXPY 6937 1.0 6.1062e-01 1.0 1.15e+09 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 5 0 0 0 0 5 0 0 0 1891 > VecAYPX 4483 1.0 1.4344e-01 1.0 1.54e+08 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 1073 > VecWAXPY 6624 1.0 9.5558e-01 1.0 1.04e+09 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 5 0 0 0 0 5 0 0 0 1089 > VecAssemblyBegin 1286 1.0 5.1773e-03 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 3.9e+03 0 0 0 0 19 0 0 0 0 21 0 > VecAssemblyEnd 1286 1.0 1.2426e-03 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecPointwiseMult 3990 1.0 6.7400e-01 1.0 3.57e+08 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 529 > > ------------------------------------------------------------------------------------------------------------------------ > > Memory usage is given in bytes: > > Object Type Creations Destructions Memory Descendants' Mem. > Reports information only for process 0. > > --- Event Stage 0: Main Stage > > Krylov Solver 258 258 218416 0 > Preconditioner 258 258 193856 0 > Matrix 666 666 140921396 0 > Vec 1383 1382 244199352 0 > Index Set 153 153 81396 0 > > ======================================================================================================================== > Average time to get PetscTime(): 0 > #PETSc Option Table entries: > -log_summary > #End of PETSc Option Table entries > Compiled without FORTRAN kernels > Compiled with full precision matrices (default) > sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 > sizeof(PetscScalar) 8 > Configure run at: Sun May 22 22:30:16 2011 > Configure options: --with-x=0 --download-f-blas-lapack=0 > --with-blas-lapack-dir=/opt/intel/Compiler/11.1/072/mkl/lib/em64t > --with-mpi=1 --with-mpi-shared=1 > --with-mpi-include=/usr/local/encap/hpmpi-8.01/include > > --with-mpi-lib="[/usr/local/encap/hpmpi-8.01/lib/linux_amd64/libmpi.so,/usr/local/encap/hpmpi-8.01/lib/linux_amd64/libfmpi.so]" > --with-mpi=1 --download-mpich=no --with-debugging=0 > --with-gnu-compilers=no --with-vendor-compilers=intel --with-cc=icc > --with-cxx=icpc --with-fc=ifort --with-shared=1 --with-c++-support > --with-clanguage=C --COPTFLAGS="-fPIC -O3 -xSSE4.2 -g -debug > inline_debug_info" --CXXOPTFLAGS="-fPIC -O3 -xSSE4.2 -g -debug > inline_debug_info" --FOPTFLAGS="-fPIC -O3 -xSSE4.2 -g -debug > inline_debug_info" --download-scalapack=no --download-blacs=no > --with-blacs=1 > --with-blacs-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_blacs_lp64.a > --with-blacs-include=/opt/intel/Compiler/11.1/072/mkl/include > --with-scalapack=1 > > --with-scalapack-lib="[/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_scalapack_lp64.a,/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_lp64.a,/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_thread.a,/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_core.a]" > --with-scalapack-include=/opt/intel/Compiler/11.1/072/mkl/include > --download-umfpack=1 --download-parmetis=1 --download-superlu_dist=1 > --download-mumps=1 --download-ml=1 --with-hypre=1 --download-hypre=yes > ----------------------------------------- > Libraries compiled on Thu May 26 16:49:30 EDT 2011 on lagrange > Machine characteristics: Linux lagrange 2.6.39-0.el5.elrepo #1 SMP > PREEMPT Sat May 21 04:48:38 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux > Using PETSc directory: /home/jfe/local/centos/petsc-3.1-p8 > Using PETSc arch: intel-opt > ----------------------------------------- > Using C compiler: icc -fPIC -fPIC -O3 -xSSE4.2 -g -debug inline_debug_info > Using Fortran compiler: ifort -fPIC -fPIC -O3 -xSSE4.2 -g -debug > inline_debug_info > ----------------------------------------- > Using include paths: > -I/home/jfe/local/centos/petsc-3.1-p8/intel-opt/include > -I/home/jfe/local/centos/petsc-3.1-p8/include > -I/home/jfe/local/centos/petsc-3.1-p8/intel-opt/include > -I/usr/local/encap/hpmpi-8.01/include > ------------------------------------------ > Using C linker: icc -fPIC -fPIC -O3 -xSSE4.2 -g -debug inline_debug_info > Using Fortran linker: ifort -fPIC -fPIC -O3 -xSSE4.2 -g -debug > inline_debug_info > Using libraries: > -Wl,-rpath,/home/jfe/local/centos/petsc-3.1-p8/intel-opt/lib > -L/home/jfe/local/centos/petsc-3.1-p8/intel-opt/lib -lpetsc > -Wl,-rpath,/home/jfe/local/centos/petsc-3.1-p8/intel-opt/lib > -L/home/jfe/local/centos/petsc-3.1-p8/intel-opt/lib -lcmumps -ldmumps > -lsmumps -lzmumps -lmumps_common -lpord > -Wl,-rpath,/opt/intel/Compiler/11.1/072/mkl/lib/em64t > -L/opt/intel/Compiler/11.1/072/mkl/lib/em64t -lmkl_scalapack_lp64 > -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_lp64 > -lHYPRE -Wl,-rpath,/opt/intel/Compiler/11.1/072/lib/intel64 > -Wl,-rpath,/opt/intel/Compiler/11.1/072/ipp/em64t/lib > > -Wl,-rpath,/opt/intel/Compiler/11.1/072/tbb/intel64/cc4.1.0_libc2.4_kernel2.6.16.21/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -lstdc++ -lml > -lstdc++ -lsuperlu_dist_2.4 -lparmetis -lmetis -lumfpack -lamd > -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread > -Wl,-rpath,/usr/local/encap/hpmpi-8.01/lib/linux_amd64 > -L/usr/local/encap/hpmpi-8.01/lib/linux_amd64 -lmpi -lfmpi -ldl > -Wl,-rpath,/opt/intel/Compiler/11.1/072/lib/intel64 > -L/opt/intel/Compiler/11.1/072/lib/intel64 > -Wl,-rpath,/opt/intel/Compiler/11.1/072/ipp/em64t/lib > -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib > -Wl,-rpath,/opt/intel/Compiler/11.1/072/mkl/lib/em64t > -L/opt/intel/Compiler/11.1/072/mkl/lib/em64t > > -Wl,-rpath,/opt/intel/Compiler/11.1/072/tbb/intel64/cc4.1.0_libc2.4_kernel2.6.16.21/lib > > -L/opt/intel/Compiler/11.1/072/tbb/intel64/cc4.1.0_libc2.4_kernel2.6.16.21/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.1.2 > -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -limf -lsvml -lipgo -ldecimal > -lgcc_s -lirc -lirc_s -lifport -lifcore -lm -lpthread -lm -lstdc++ > -lstdc++ -ldl -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lirc_s -ldl > ------------------------------------------ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Wed Jun 1 15:11:43 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 1 Jun 2011 22:11:43 +0200 Subject: [petsc-users] Reducing cost of MatSetValues In-Reply-To: References: Message-ID: On Wed, Jun 1, 2011 at 21:53, John Fettig wrote: > What are the recommendations for reducing the cost of inserting values > into an AIJ matrix? In my application (transient finite element > solution of flow and heat, linear elements), this is accounting for up > to 20% of overall runtime. Is this expected? > 2D or 3D? What kind of elements? > > I have double checked that the matrices are preallocated correctly, > and I have set MAT_NEW_NONZERO_ALLOCATION_ERR and it runs without > error. > > The matrices periodically change size/nonzero pattern, but until then > the values are zeroed out and MatSetOption( mat, > MAT_KEEP_NONZERO_PATTERN, PETSC_TRUE); is called. > > The call to MatSetValues happens on a per-element basis on the local > element matrix, so one call per element. I re-activated the > MAT_SetValues event and have included a -log_summary from a short run. > The overhead of the profiling can be nontrivial for small elements. How does the run-time change if you deactivate that event? How much of the overall time is spent in matrix assembly (your code plus MatSetValues)? It looks like the linear solve is around 18 iterations. For first order elements, that is probably too many for matrix-free with a lagged Jacobian to pay off. The trade-off is much more favorable for higher order elements, or if the preconditioner is better. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Wed Jun 1 15:43:28 2011 From: john.fettig at gmail.com (John Fettig) Date: Wed, 1 Jun 2011 16:43:28 -0400 Subject: [petsc-users] Reducing cost of MatSetValues In-Reply-To: References: Message-ID: Hi Matt, On Wed, Jun 1, 2011 at 4:06 PM, Matthew Knepley wrote: > It looks like the calls are taking 0.5 microseconds, but there are 89G > insertions > against only 13K MatMults that take about 1s apiece. > This seems like an awful lot of insertions into a matrix that can be applied > in 1s. Unless I'm reading it wrong, there are 89M insertions. This is from 10 timesteps with nonlinear iteration on each timestep, with ~1.5M elements. The matrices are reconstructed at virtually every substep. I don't quite understand what you mean by "can be applied in 1s", doesn't it take ~1ms per call? Anyways, I agree that this is a poor ratio of work to assemble/use of the matrices. We had thought about constructing the matrix outside of PETSc and then passing PETSc pointers to the matrix, but maybe matrix-free would be better for some of the equations where jacobi preconditioning is sufficient. Thanks, John From john.fettig at gmail.com Wed Jun 1 15:53:46 2011 From: john.fettig at gmail.com (John Fettig) Date: Wed, 1 Jun 2011 16:53:46 -0400 Subject: [petsc-users] Reducing cost of MatSetValues In-Reply-To: References: Message-ID: Hi Jed, On Wed, Jun 1, 2011 at 4:11 PM, Jed Brown wrote: > 2D or 3D? What kind of elements? This is 3D with linear tets. > The overhead of the profiling can be nontrivial for small elements. How does > the run-time change if you deactivate that event? > How much of the overall time is spent in matrix assembly (your code plus > MatSetValues)? I actually started off by profiling the application with HPCToolkit, and only turned on the PETSc profiling as a sanity check and to send to this list. The results are pretty consistent with the profiling from HPCToolkit. I ran without -log_summary and it was about 8% faster. > It looks like the linear solve is around 18 iterations. For first order > elements, that is probably too many for matrix-free with a lagged Jacobian > to pay off. The trade-off is much more favorable for higher order elements, > or if the preconditioner is better. There are several linear solves, and some of them do pretty well with just cg/jacobi. Perhaps we could move a few of the solves to matrix-free. Thanks, John From knepley at gmail.com Wed Jun 1 15:53:26 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 1 Jun 2011 15:53:26 -0500 Subject: [petsc-users] Reducing cost of MatSetValues In-Reply-To: References: Message-ID: On Wed, Jun 1, 2011 at 3:43 PM, John Fettig wrote: > Hi Matt, > > On Wed, Jun 1, 2011 at 4:06 PM, Matthew Knepley wrote: > > It looks like the calls are taking 0.5 microseconds, but there are 89G > > insertions > > against only 13K MatMults that take about 1s apiece. > > This seems like an awful lot of insertions into a matrix that can be > applied > > in 1s. > > Unless I'm reading it wrong, there are 89M insertions. This is from > 10 timesteps with nonlinear iteration on each timestep, with ~1.5M > elements. The matrices are reconstructed at virtually every substep. > I don't quite understand what you mean by "can be applied in 1s", > doesn't it take ~1ms per call? Anyways, I agree that this is a poor > ratio of work to assemble/use of the matrices. > Yes, both units were wrong. > We had thought about constructing the matrix outside of PETSc and then > passing PETSc pointers to the matrix, but maybe matrix-free would be > better for some of the equations where jacobi preconditioning is > sufficient. > I think it would be hard to speedup the insertion. Why are you reassembling? Just for Newton? You could use a lagged version of the matrix as a preconditioner, and run matrix-free for the matvecs. Matt > Thanks, > John > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Wed Jun 1 15:59:19 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 1 Jun 2011 22:59:19 +0200 Subject: [petsc-users] Reducing cost of MatSetValues In-Reply-To: References: Message-ID: On Wed, Jun 1, 2011 at 22:43, John Fettig wrote: > Unless I'm reading it wrong, there are 89M insertions. This is from > 10 timesteps with nonlinear iteration on each timestep, > Needing 27 linear solves per time step would normally be a better place to direct your effort, but strangely, very little of the total run time is currently in the solve. > with ~1.5M > elements. > Is this 1.5M finite elements or dofs or nonzeros in the matrix? Doing more than 1000 SOR sweeps per second would be weird if the matrix is big. > > We had thought about constructing the matrix outside of PETSc and then > passing PETSc pointers to the matrix, but maybe matrix-free would be > better for some of the equations where jacobi preconditioning is > sufficient. > Have you considered solving the coupled system instead of a bunch of separate systems. Solving the coupled thing allows you to use block matrix formats and gets better data locality. It also reduces the number of synchronization points (parallel reductions and scatters). And it should converge faster on the coupled system. 27 separate solves per time step is really high, and you currently have over 800 reductions per time step. That is going to hurt a lot in parallel, especially for strong scaling. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jun 1 16:15:41 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 1 Jun 2011 16:15:41 -0500 Subject: [petsc-users] Reducing cost of MatSetValues In-Reply-To: References: Message-ID: <25835B8B-B623-4265-AF46-37677C5E43A5@mcs.anl.gov> If you run with -info and grep the results for "Stash has" do you see much in the way of stashed values? These are values computed on one process that must be shipped to another process. Likely you have done a good job of partitioning and there are not that many relative to local number of values set. If the number of mallocs is high for this you can run with a big enough -matstash_initial_size so that no additional mallocs are needed. Also if you are hardwired to the MPIAIJ format you can call MatSetValues_MPIAIJ() directly instead of MatSetValues() (just add a prototype for this function to your source code). This could shave off a couple percent. I am hard pressed at seeing anyway to really shrink down the code in MatSetValues_MPIAIJ() anymore than it already is. Since your linear solves are pretty easy it is tough to make the setting values a small percent of the total time. Using -snes_lag_jacobian 2 or 3 (or even bigger) might help. Barry On Jun 1, 2011, at 2:53 PM, John Fettig wrote: > What are the recommendations for reducing the cost of inserting values > into an AIJ matrix? In my application (transient finite element > solution of flow and heat, linear elements), this is accounting for up > to 20% of overall runtime. Is this expected? > > I have double checked that the matrices are preallocated correctly, > and I have set MAT_NEW_NONZERO_ALLOCATION_ERR and it runs without > error. > > The matrices periodically change size/nonzero pattern, but until then > the values are zeroed out and MatSetOption( mat, > MAT_KEEP_NONZERO_PATTERN, PETSC_TRUE); is called. > > The call to MatSetValues happens on a per-element basis on the local > element matrix, so one call per element. I re-activated the > MAT_SetValues event and have included a -log_summary from a short run. > > Thanks, > John > > ************************************************************************************************************************ > *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r > -fCourier9' to print this document *** > ************************************************************************************************************************ > > ---------------------------------------------- PETSc Performance > Summary: ---------------------------------------------- > > ./test on a intel-opt named lagrange with 1 processor, by jfe Wed Jun > 1 15:49:20 2011 > Using Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 > > Max Max/Min Avg Total > Time (sec): 2.242e+02 1.00000 2.242e+02 > Objects: 2.718e+03 1.00000 2.718e+03 > Flops: 2.239e+10 1.00000 2.239e+10 2.239e+10 > Flops/sec: 9.986e+07 1.00000 9.986e+07 9.986e+07 > MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 > MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 > MPI Reductions: 2.062e+04 1.00000 > > Flop counting convention: 1 flop = 1 real number operation of type > (multiply/divide/add/subtract) > e.g., VecAXPY() for real vectors of length > N --> 2N flops > and VecAXPY() for complex vectors of > length N --> 8N flops > > Summary of Stages: ----- Time ------ ----- Flops ----- --- > Messages --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total counts > %Total Avg %Total counts %Total > 0: Main Stage: 2.2423e+02 100.0% 2.2391e+10 100.0% 0.000e+00 > 0.0% 0.000e+00 0.0% 1.796e+04 87.1% > > ------------------------------------------------------------------------------------------------------------------------ > See the 'Profiling' chapter of the users' manual for details on > interpreting output. > Phase summary info: > Count: number of times phase was executed > Time and Flops: Max - maximum over all processors > Ratio - ratio of maximum to minimum over all processors > Mess: number of messages sent > Avg. len: average message length > Reduct: number of global reductions > Global: entire computation > Stage: stages of a computation. Set stages with PetscLogStagePush() > and PetscLogStagePop(). > %T - percent time in this phase %F - percent flops in this phase > %M - percent messages in this phase %L - percent message > lengths in this phase > %R - percent reductions in this phase > Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time > over all processors) > ------------------------------------------------------------------------------------------------------------------------ > Event Count Time (sec) Flops > --- Global --- --- Stage --- Total > Max Ratio Max Ratio Max Ratio Mess Avg > len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > KSPSetup 532 1.0 2.6963e-02 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 2.0e+02 0 0 0 0 1 0 0 0 0 1 0 > KSPSolve 277 1.0 3.2930e+01 1.0 2.17e+10 1.0 0.0e+00 > 0.0e+00 9.9e+03 15 97 0 0 48 15 97 0 0 55 660 > PCSetUp 277 1.0 8.3035e+00 1.0 7.52e+08 1.0 0.0e+00 > 0.0e+00 1.4e+03 4 3 0 0 7 4 3 0 0 8 91 > PCApply 4907 1.0 9.1353e+00 1.0 7.08e+09 1.0 0.0e+00 > 0.0e+00 2.0e+00 4 32 0 0 0 4 32 0 0 0 775 > MatMult 12821 1.0 1.5110e+01 1.0 1.34e+10 1.0 0.0e+00 > 0.0e+00 0.0e+00 7 60 0 0 0 7 60 0 0 0 888 > MatMultAdd 3668 1.0 6.9999e-01 1.0 2.83e+08 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 404 > MatSolve 917 1.0 5.3644e-04 1.0 9.17e+02 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2 > MatSOR 7336 1.0 5.7982e+00 1.0 4.62e+09 1.0 0.0e+00 > 0.0e+00 0.0e+00 3 21 0 0 0 3 21 0 0 0 797 > MatLUFactorSym 51 1.0 3.1686e-04 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 5.1e+01 0 0 0 0 0 0 0 0 0 0 0 > MatLUFactorNum 51 1.0 1.2851e-04 1.0 5.10e+01 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyBegin 1068 1.0 1.4639e-04 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 1068 1.0 7.9752e-01 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatSetValues 89511956 1.0 4.6221e+01 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 21 0 0 0 0 21 0 0 0 0 0 > MatGetRowIJ 51 1.0 7.7248e-05 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatGetOrdering 51 1.0 5.0688e-04 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 1.0e+02 0 0 0 0 0 0 0 0 0 1 0 > MatZeroEntries 133 1.0 1.7668e-01 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecMax 85 1.0 6.9754e-03 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 8.5e+01 0 0 0 0 0 0 0 0 0 0 0 > VecDot 5479 1.0 3.5107e-01 1.0 8.84e+08 1.0 0.0e+00 > 0.0e+00 5.5e+03 0 4 0 0 27 0 4 0 0 31 2517 > VecNorm 2937 1.0 1.5535e+00 1.0 4.81e+08 1.0 0.0e+00 > 0.0e+00 2.9e+03 1 2 0 0 14 1 2 0 0 16 310 > VecCopy 955 1.0 8.6689e-02 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecSet 6718 1.0 9.4952e-02 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAXPY 6937 1.0 6.1062e-01 1.0 1.15e+09 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 5 0 0 0 0 5 0 0 0 1891 > VecAYPX 4483 1.0 1.4344e-01 1.0 1.54e+08 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 1073 > VecWAXPY 6624 1.0 9.5558e-01 1.0 1.04e+09 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 5 0 0 0 0 5 0 0 0 1089 > VecAssemblyBegin 1286 1.0 5.1773e-03 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 3.9e+03 0 0 0 0 19 0 0 0 0 21 0 > VecAssemblyEnd 1286 1.0 1.2426e-03 1.0 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecPointwiseMult 3990 1.0 6.7400e-01 1.0 3.57e+08 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 529 > ------------------------------------------------------------------------------------------------------------------------ > > Memory usage is given in bytes: > > Object Type Creations Destructions Memory Descendants' Mem. > Reports information only for process 0. > > --- Event Stage 0: Main Stage > > Krylov Solver 258 258 218416 0 > Preconditioner 258 258 193856 0 > Matrix 666 666 140921396 0 > Vec 1383 1382 244199352 0 > Index Set 153 153 81396 0 > ======================================================================================================================== > Average time to get PetscTime(): 0 > #PETSc Option Table entries: > -log_summary > #End of PETSc Option Table entries > Compiled without FORTRAN kernels > Compiled with full precision matrices (default) > sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 > sizeof(PetscScalar) 8 > Configure run at: Sun May 22 22:30:16 2011 > Configure options: --with-x=0 --download-f-blas-lapack=0 > --with-blas-lapack-dir=/opt/intel/Compiler/11.1/072/mkl/lib/em64t > --with-mpi=1 --with-mpi-shared=1 > --with-mpi-include=/usr/local/encap/hpmpi-8.01/include > --with-mpi-lib="[/usr/local/encap/hpmpi-8.01/lib/linux_amd64/libmpi.so,/usr/local/encap/hpmpi-8.01/lib/linux_amd64/libfmpi.so]" > --with-mpi=1 --download-mpich=no --with-debugging=0 > --with-gnu-compilers=no --with-vendor-compilers=intel --with-cc=icc > --with-cxx=icpc --with-fc=ifort --with-shared=1 --with-c++-support > --with-clanguage=C --COPTFLAGS="-fPIC -O3 -xSSE4.2 -g -debug > inline_debug_info" --CXXOPTFLAGS="-fPIC -O3 -xSSE4.2 -g -debug > inline_debug_info" --FOPTFLAGS="-fPIC -O3 -xSSE4.2 -g -debug > inline_debug_info" --download-scalapack=no --download-blacs=no > --with-blacs=1 --with-blacs-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_blacs_lp64.a > --with-blacs-include=/opt/intel/Compiler/11.1/072/mkl/include > --with-scalapack=1 > --with-scalapack-lib="[/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_scalapack_lp64.a,/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_lp64.a,/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_thread.a,/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_core.a]" > --with-scalapack-include=/opt/intel/Compiler/11.1/072/mkl/include > --download-umfpack=1 --download-parmetis=1 --download-superlu_dist=1 > --download-mumps=1 --download-ml=1 --with-hypre=1 --download-hypre=yes > ----------------------------------------- > Libraries compiled on Thu May 26 16:49:30 EDT 2011 on lagrange > Machine characteristics: Linux lagrange 2.6.39-0.el5.elrepo #1 SMP > PREEMPT Sat May 21 04:48:38 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux > Using PETSc directory: /home/jfe/local/centos/petsc-3.1-p8 > Using PETSc arch: intel-opt > ----------------------------------------- > Using C compiler: icc -fPIC -fPIC -O3 -xSSE4.2 -g -debug inline_debug_info > Using Fortran compiler: ifort -fPIC -fPIC -O3 -xSSE4.2 -g -debug > inline_debug_info > ----------------------------------------- > Using include paths: > -I/home/jfe/local/centos/petsc-3.1-p8/intel-opt/include > -I/home/jfe/local/centos/petsc-3.1-p8/include > -I/home/jfe/local/centos/petsc-3.1-p8/intel-opt/include > -I/usr/local/encap/hpmpi-8.01/include > ------------------------------------------ > Using C linker: icc -fPIC -fPIC -O3 -xSSE4.2 -g -debug inline_debug_info > Using Fortran linker: ifort -fPIC -fPIC -O3 -xSSE4.2 -g -debug > inline_debug_info > Using libraries: > -Wl,-rpath,/home/jfe/local/centos/petsc-3.1-p8/intel-opt/lib > -L/home/jfe/local/centos/petsc-3.1-p8/intel-opt/lib -lpetsc > -Wl,-rpath,/home/jfe/local/centos/petsc-3.1-p8/intel-opt/lib > -L/home/jfe/local/centos/petsc-3.1-p8/intel-opt/lib -lcmumps -ldmumps > -lsmumps -lzmumps -lmumps_common -lpord > -Wl,-rpath,/opt/intel/Compiler/11.1/072/mkl/lib/em64t > -L/opt/intel/Compiler/11.1/072/mkl/lib/em64t -lmkl_scalapack_lp64 > -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_lp64 > -lHYPRE -Wl,-rpath,/opt/intel/Compiler/11.1/072/lib/intel64 > -Wl,-rpath,/opt/intel/Compiler/11.1/072/ipp/em64t/lib > -Wl,-rpath,/opt/intel/Compiler/11.1/072/tbb/intel64/cc4.1.0_libc2.4_kernel2.6.16.21/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -lstdc++ -lml > -lstdc++ -lsuperlu_dist_2.4 -lparmetis -lmetis -lumfpack -lamd > -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread > -Wl,-rpath,/usr/local/encap/hpmpi-8.01/lib/linux_amd64 > -L/usr/local/encap/hpmpi-8.01/lib/linux_amd64 -lmpi -lfmpi -ldl > -Wl,-rpath,/opt/intel/Compiler/11.1/072/lib/intel64 > -L/opt/intel/Compiler/11.1/072/lib/intel64 > -Wl,-rpath,/opt/intel/Compiler/11.1/072/ipp/em64t/lib > -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib > -Wl,-rpath,/opt/intel/Compiler/11.1/072/mkl/lib/em64t > -L/opt/intel/Compiler/11.1/072/mkl/lib/em64t > -Wl,-rpath,/opt/intel/Compiler/11.1/072/tbb/intel64/cc4.1.0_libc2.4_kernel2.6.16.21/lib > -L/opt/intel/Compiler/11.1/072/tbb/intel64/cc4.1.0_libc2.4_kernel2.6.16.21/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.1.2 > -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -limf -lsvml -lipgo -ldecimal > -lgcc_s -lirc -lirc_s -lifport -lifcore -lm -lpthread -lm -lstdc++ > -lstdc++ -ldl -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lirc_s -ldl > ------------------------------------------ From gdiso at ustc.edu Wed Jun 1 21:36:23 2011 From: gdiso at ustc.edu (Gong Ding) Date: Thu, 2 Jun 2011 10:36:23 +0800 (CST) Subject: [petsc-users] Have you received patch to AIJ matrix with hash table Message-ID: <30453840.56511306982183239.JavaMail.coremail@mail.ustc.edu> Hi, I had committed the patch (AIJ matrix with hash table for fast entry re-allcation) to petsc-maint at mcs.anl.gov for a week. Have you received it yet? Gong Ding From bsmith at mcs.anl.gov Wed Jun 1 21:40:21 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 1 Jun 2011 21:40:21 -0500 Subject: [petsc-users] Have you received patch to AIJ matrix with hash table In-Reply-To: <30453840.56511306982183239.JavaMail.coremail@mail.ustc.edu> References: <30453840.56511306982183239.JavaMail.coremail@mail.ustc.edu> Message-ID: <285F3B63-D9DD-4AB4-934E-1F2DEEC40818@mcs.anl.gov> Satish will be getting to it soon. Barry On Jun 1, 2011, at 9:36 PM, Gong Ding wrote: > Hi, > I had committed the patch (AIJ matrix with hash table for fast entry re-allcation) to petsc-maint at mcs.anl.gov for a week. > Have you received it yet? > > Gong Ding > > > > > From stijn.vantieghem at erdw.ethz.ch Fri Jun 3 03:54:19 2011 From: stijn.vantieghem at erdw.ethz.ch (Stijn A. M. Vantieghem) Date: Fri, 3 Jun 2011 08:54:19 +0000 Subject: [petsc-users] ML - zero pivot error Message-ID: <69AD671D-5C83-4A1A-9100-6E5ACDA29224@erdw.ethz.ch> Dear all, I am using PETSc (Fortran interface) to solve a Poisson equation with Neumann boundary conditions. Up till now, I managed to do this with Hypre's BoomerAMG. Now, I am investigating whether I can improve the performance of my code by using ML. However, at execution I receive a zero pivot error; I tried to remove the (constant) null space with KSPSetNullSpace, but this didn't solve my problem. Do you have an idea of what I'm doing wrong? Thanks. The relevant portions of my code are as follows: !**************************************************** call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) call KSPSetOperators(ksp,M,M,DIFFERENT_NONZERO_PATTERN,ierr) call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL,sp,ierr) call KSPSetNullSpace(ksp,sp,ierr) call KSPGetPC(ksp,pc,ierr) call PCSetType(pc,PCML,ierr) call KSPSetFromOptions(ksp,ierr) ... call KSPSetInitialGuessNonzero(ksp,PETSC_TRUE,ierr) call KSPSolve(ksp,petsc_rhs,petsc_pressure,ierr) !**************************************************** and the error message is: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Detected zero pivot in LU factorization see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#ZeroPivot! [0]PETSC ERROR: Zero pivot row 0 value 3.57045e-20 tolerance 1e-12! [0]PETSC ERROR: ------------------------------------------------------------------------ ... [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatLUFactorNumeric_SeqAIJ() line 574 in src/mat/impls/aij/seq/aijfact.c [0]PETSC ERROR: MatLUFactorNumeric() line 2587 in src/mat/interface/matrix.c [0]PETSC ERROR: PCSetUp_LU() line 158 in src/ksp/pc/impls/factor/lu/lu.c [0]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: PCSetUp_MG() line 602 in src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: PCSetUp_ML() line 668 in src/ksp/pc/impls/ml/ml.c [0]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c Regards Stijn Stijn A.M. Vantieghem Earth and Planetary Magnetism Institute for Geophysics ETH Z?rich Sonneggstrasse 5 - CH 8092 Z?rich tel: +41 44 632 39 90 e-mail: stijn.vantieghem at erdw.ethz.ch From bsmith at mcs.anl.gov Fri Jun 3 07:26:25 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 3 Jun 2011 07:26:25 -0500 Subject: [petsc-users] ML - zero pivot error In-Reply-To: <69AD671D-5C83-4A1A-9100-6E5ACDA29224@erdw.ethz.ch> References: <69AD671D-5C83-4A1A-9100-6E5ACDA29224@erdw.ethz.ch> Message-ID: <0422B860-8CBC-40BB-986F-0B128E98C85B@mcs.anl.gov> It is the direct solver on the the coarse grid that is finding the zero pivot (since the coarse grid problem like all the levels has a null space). You can use the option -mg_coarse_pc_factor_shift_type nonzero (in petsc-3.1 or petsc-dev) Also keep the KSPSetNullSpace() function you are using. Barry On Jun 3, 2011, at 3:54 AM, Stijn A. M. Vantieghem wrote: > Dear all, > > I am using PETSc (Fortran interface) to solve a Poisson equation with Neumann boundary conditions. Up till now, I managed to do this with Hypre's BoomerAMG. Now, I am investigating whether I can improve the performance of my code by using ML. However, at execution I receive a zero pivot error; I tried to remove the (constant) null space with KSPSetNullSpace, but this didn't solve my problem. Do you have an idea of what I'm doing wrong? Thanks. > > The relevant portions of my code are as follows: > > !**************************************************** > call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > call KSPSetOperators(ksp,M,M,DIFFERENT_NONZERO_PATTERN,ierr) > > call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL,sp,ierr) > call KSPSetNullSpace(ksp,sp,ierr) > > call KSPGetPC(ksp,pc,ierr) > call PCSetType(pc,PCML,ierr) > > call KSPSetFromOptions(ksp,ierr) > ... > call KSPSetInitialGuessNonzero(ksp,PETSC_TRUE,ierr) > call KSPSolve(ksp,petsc_rhs,petsc_pressure,ierr) > !**************************************************** > > and the error message is: > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Detected zero pivot in LU factorization > see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#ZeroPivot! > [0]PETSC ERROR: Zero pivot row 0 value 3.57045e-20 tolerance 1e-12! > [0]PETSC ERROR: ------------------------------------------------------------------------ > ... > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatLUFactorNumeric_SeqAIJ() line 574 in src/mat/impls/aij/seq/aijfact.c > [0]PETSC ERROR: MatLUFactorNumeric() line 2587 in src/mat/interface/matrix.c > [0]PETSC ERROR: PCSetUp_LU() line 158 in src/ksp/pc/impls/factor/lu/lu.c > [0]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: PCSetUp_MG() line 602 in src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: PCSetUp_ML() line 668 in src/ksp/pc/impls/ml/ml.c > [0]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c > > Regards > Stijn > > Stijn A.M. Vantieghem > Earth and Planetary Magnetism > Institute for Geophysics > ETH Z?rich > Sonneggstrasse 5 - CH 8092 Z?rich > tel: +41 44 632 39 90 > e-mail: stijn.vantieghem at erdw.ethz.ch > > > > From danesh.daroui at ltu.se Fri Jun 3 09:03:02 2011 From: danesh.daroui at ltu.se (Danesh Daroui) Date: Fri, 03 Jun 2011 16:03:02 +0200 Subject: [petsc-users] ILU preconditioner In-Reply-To: References: <1306821768.7496.3.camel@linux-sclt.site> Message-ID: <1307109782.3303.28.camel@linux-sclt.site> Hi, In this case, can I create a preconditioner in double precision format and apply it to the system while I am using PETSc to solve a system of complex numbers? The reason is that I am solving equations in several iterations and I want to create an ILU-based preconditioner only on real part of my coefficient matrix because it is unchanged and I can create and store is once. The imaginary parts will change but they do not have big effect. Regards, D. On Tue, 2011-05-31 at 15:13 +0200, Jed Brown wrote: > On Tue, May 31, 2011 at 08:02, Danesh Daroui > wrote: > I have a complex system of equations and I use PETSc to solve > the > system. I also use MKL to create the preconditioner using ILU. > The ILU > preconditioner is created only on the real part of the > coefficient > matrix and is stored in CSR format with the data type of > double > precision. I wanted to ask if it is possible to use this > preconditioner > and pass it to PETSc in CSR format? Can it be used with PETSc > complex > solver although the preconditioner is in double precision > format? > > You can create a PCShell() that manages the factorization using MKL > and applies the preconditioner stored in whatever format MKL likes. > PETSc would not know that it is ILU or otherwise. It sounds like that > would do what you want. From bsmith at mcs.anl.gov Fri Jun 3 09:07:34 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 3 Jun 2011 09:07:34 -0500 Subject: [petsc-users] ILU preconditioner In-Reply-To: <1307109782.3303.28.camel@linux-sclt.site> References: <1306821768.7496.3.camel@linux-sclt.site> <1307109782.3303.28.camel@linux-sclt.site> Message-ID: <9F648F50-DC9B-4F04-A509-DEF6787C7529@mcs.anl.gov> Unfortunately PETSc is restricted to working with only real or complex objects at one time and one cannot mix some real and some complex. Barry On Jun 3, 2011, at 9:03 AM, Danesh Daroui wrote: > > Hi, > > In this case, can I create a preconditioner in double precision format > and apply it to the system while I am using PETSc to solve a system of > complex numbers? The reason is that I am solving equations in several > iterations and I want to create an ILU-based preconditioner only on real > part of my coefficient matrix because it is unchanged and I can create > and store is once. The imaginary parts will change but they do not have > big effect. > > Regards, > > D. > > > On Tue, 2011-05-31 at 15:13 +0200, Jed Brown wrote: >> On Tue, May 31, 2011 at 08:02, Danesh Daroui >> wrote: >> I have a complex system of equations and I use PETSc to solve >> the >> system. I also use MKL to create the preconditioner using ILU. >> The ILU >> preconditioner is created only on the real part of the >> coefficient >> matrix and is stored in CSR format with the data type of >> double >> precision. I wanted to ask if it is possible to use this >> preconditioner >> and pass it to PETSc in CSR format? Can it be used with PETSc >> complex >> solver although the preconditioner is in double precision >> format? >> >> You can create a PCShell() that manages the factorization using MKL >> and applies the preconditioner stored in whatever format MKL likes. >> PETSc would not know that it is ILU or otherwise. It sounds like that >> would do what you want. > > > From jed at 59A2.org Fri Jun 3 09:12:24 2011 From: jed at 59A2.org (Jed Brown) Date: Fri, 3 Jun 2011 16:12:24 +0200 Subject: [petsc-users] ILU preconditioner In-Reply-To: <9F648F50-DC9B-4F04-A509-DEF6787C7529@mcs.anl.gov> References: <1306821768.7496.3.camel@linux-sclt.site> <1307109782.3303.28.camel@linux-sclt.site> <9F648F50-DC9B-4F04-A509-DEF6787C7529@mcs.anl.gov> Message-ID: On Fri, Jun 3, 2011 at 16:07, Barry Smith wrote: > Unfortunately PETSc is restricted to working with only real or complex > objects at one time and one cannot mix some real and some complex. If I understand correctly, Danesh intends to use an external package (MKL) for the ILU on the real part. That will work fine with PCShell. His setup function will pull out the real part of the matrix, put it into an MKL data structure, and do the factorization. Then the apply function will take the input vector apart, solve with the MKL factorization, and put the result back together in the output vector. A bit awkward perhaps, but it should work fine. -------------- next part -------------- An HTML attachment was scrubbed... URL: From danesh.daroui at ltu.se Fri Jun 3 09:17:03 2011 From: danesh.daroui at ltu.se (Danesh Daroui) Date: Fri, 03 Jun 2011 16:17:03 +0200 Subject: [petsc-users] ILU preconditioner In-Reply-To: <9F648F50-DC9B-4F04-A509-DEF6787C7529@mcs.anl.gov> References: <1306821768.7496.3.camel@linux-sclt.site> <1307109782.3303.28.camel@linux-sclt.site> <9F648F50-DC9B-4F04-A509-DEF6787C7529@mcs.anl.gov> Message-ID: <1307110623.3303.44.camel@linux-sclt.site> Do I have any other choice for my needs? What about PCShell? It seems to be quite flexible. Can't I use this to only calculate preconditioner on real parts? I can calculate them using MKL and pass it to PETSc. On Fri, 2011-06-03 at 09:07 -0500, Barry Smith wrote: > Unfortunately PETSc is restricted to working with only real or complex objects at one time and one cannot mix some real and some complex. > > Barry > > On Jun 3, 2011, at 9:03 AM, Danesh Daroui wrote: > > > > > Hi, > > > > In this case, can I create a preconditioner in double precision format > > and apply it to the system while I am using PETSc to solve a system of > > complex numbers? The reason is that I am solving equations in several > > iterations and I want to create an ILU-based preconditioner only on real > > part of my coefficient matrix because it is unchanged and I can create > > and store is once. The imaginary parts will change but they do not have > > big effect. > > > > Regards, > > > > D. > > > > > > On Tue, 2011-05-31 at 15:13 +0200, Jed Brown wrote: > >> On Tue, May 31, 2011 at 08:02, Danesh Daroui > >> wrote: > >> I have a complex system of equations and I use PETSc to solve > >> the > >> system. I also use MKL to create the preconditioner using ILU. > >> The ILU > >> preconditioner is created only on the real part of the > >> coefficient > >> matrix and is stored in CSR format with the data type of > >> double > >> precision. I wanted to ask if it is possible to use this > >> preconditioner > >> and pass it to PETSc in CSR format? Can it be used with PETSc > >> complex > >> solver although the preconditioner is in double precision > >> format? > >> > >> You can create a PCShell() that manages the factorization using MKL > >> and applies the preconditioner stored in whatever format MKL likes. > >> PETSc would not know that it is ILU or otherwise. It sounds like that > >> would do what you want. > > > > > > > > -- Danesh Daroui, Ph.D Student EISLAB Department of Computer Science, Electrical and Space Engineering Lulea University of Technology SE-971 87 Lulea Sweden http://www.ltu.se email: danesh.daroui at ltu.se phone: +46-(0)920-492451 mobile: +46-(0)704-399847 From danesh.daroui at ltu.se Fri Jun 3 09:20:35 2011 From: danesh.daroui at ltu.se (Danesh Daroui) Date: Fri, 03 Jun 2011 16:20:35 +0200 Subject: [petsc-users] ILU preconditioner In-Reply-To: References: <1306821768.7496.3.camel@linux-sclt.site> <1307109782.3303.28.camel@linux-sclt.site> <9F648F50-DC9B-4F04-A509-DEF6787C7529@mcs.anl.gov> Message-ID: <1307110835.3303.49.camel@linux-sclt.site> Yes that's exactly what I want to do. I can also use PETSc to calculate the preconditioner, but can I use it to calculate real part's ILU while I am using complex solver? On Fri, 2011-06-03 at 16:12 +0200, Jed Brown wrote: > On Fri, Jun 3, 2011 at 16:07, Barry Smith wrote: > Unfortunately PETSc is restricted to working with only real or > complex objects at one time and one cannot mix some real and > some complex. > > If I understand correctly, Danesh intends to use an external package > (MKL) for the ILU on the real part. That will work fine with PCShell. > His setup function will pull out the real part of the matrix, put it > into an MKL data structure, and do the factorization. Then the apply > function will take the input vector apart, solve with the MKL > factorization, and put the result back together in the output vector. > A bit awkward perhaps, but it should work fine. From jed at 59A2.org Fri Jun 3 09:27:55 2011 From: jed at 59A2.org (Jed Brown) Date: Fri, 3 Jun 2011 16:27:55 +0200 Subject: [petsc-users] ILU preconditioner In-Reply-To: <1307110835.3303.49.camel@linux-sclt.site> References: <1306821768.7496.3.camel@linux-sclt.site> <1307109782.3303.28.camel@linux-sclt.site> <9F648F50-DC9B-4F04-A509-DEF6787C7529@mcs.anl.gov> <1307110835.3303.49.camel@linux-sclt.site> Message-ID: On Fri, Jun 3, 2011 at 16:20, Danesh Daroui wrote: > Yes that's exactly what I want to do. I can also use PETSc to calculate > the preconditioner, but can I use it to calculate real part's ILU while > I am using complex solver? > No, the scalar type is chosen at configuration time. You can't mix real and complex Mat and Vec types in the same program (without weird non-portable shared library tricks, and I'm not sure that would even work). I would just use complex everywhere to check that the algorithm works. You can even create a preconditioning matrix that is just the real part (still using complex arithmetic) to check that the algorithm you propose really works. Then you can do the PCShell implementation, but still put the result in a PETSc Mat which uses complex scalars. Once that works, you can swap out the PETSc Mat for some external library that does ILU in real arithmetic. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jun 3 10:20:41 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 3 Jun 2011 10:20:41 -0500 Subject: [petsc-users] ILU and Block Gauss Seidel smoothing In-Reply-To: References: Message-ID: <9F52876E-C796-407A-8608-53D30D666A83@mcs.anl.gov> On May 25, 2011, at 2:45 AM, Gianluca Meneghello wrote: > Dear all, > > I am writing a multigrid solver for the Navier Stokes Equations, using > PETSc. I am not using the dmmg structure because I need grid > refinement and, as far as I can see, that is not supported. > > I have two questions concerning two different possibilities of smoothing: > > 1) ILU relaxation: in Multigrid (U. Trottenberg,Cornelis W. > Oosterlee,Anton Sch?ller), it is suggested that alternating ILU > represents a robust smoother. Given a routine for ILU decomposition, > my understanding is that alternation depends on the ordering of the > matrix. My code builds a linearized Navier Stokes operator where first > the u momentum equation, then the v momentum and then the p equation > are stored. Each equation is built in EN order ? first comes the > bottom horizontal line, from West to East, then I move line by line > from South to North. Otherwise stated, the position of each unknown in > the matrix is given by P = cf*nx*ny + j*nx + i, where cf is the > variable number (0 = u, 1 = v, 2 = p) and i,j,nx,ny are as usual. > > For "historical" reasons I'm not using the DM structure even if the > grids are logically rectangular, but I can change that. > > My question: is there in PETSc a way of doing alternating ILU which > does not require rebuilding the matrix with different ordering each > time? There is a way for you to set the ordering of the ILU factorization without building the matrix in that ordering. It requires that you write a function with the calling sequence PetscErrorCode YourOrdering(Mat mat,const MatOrderingType type,IS *isrow, IS *iscol) { Your function fills up isrow with the order of the rows you want the ILU to visit, for example if you want the first row visited to be ten then you would create an isrow whose first entry is 10. You can/and probably should use the same ordering for rows and columns. Your function will igore the type name that is passed in. } In your main program after PetscInitialize() call MatOrderingRegister("yourname",0,yourname,YourOrdering); now when you run the program you can use the option -pc_factor_mat_ordering_type yourname and it will use the function you provided to generate the ordering that is used for ILU. Good luck and sorry for not responding to this message sooner, Barry > > Somehow related questions: what is the best ordering for the matrix > and which one is the one used by the DM structure? > > > 2) Decoupled block-line Gauss Seidel relaxation: this is another > smoother I'm considering, which explains the matrix ordering above (I > use block horizontal lines smoothing, so that each line is stored > consecutively in the matrix). At the moment the full linearized > operator is first built and then parts of the matrix are extracted. > In reality though, I never need the full linearized operator to be > built except on the coarsest grid. Rather, I need it to be constructed > each block-line at a time. I've seen that for parallel applications > each part of the matrix is built on its corresponding processor. Is > there a way to do it sequentially and to have control on which part is > built? > > > Any other suggestion on both subjects is of course welcome. Thanks in advance > > Gianluca > > > > -- > "[Je pense que] l'homme est un monde qui vaut des fois les mondes et > que les plus ardentes ambitions sont celles qui ont eu l'orgueil de > l'Anonymat" -- Non omnibus, sed mihi et tibi > Amedeo Modigliani From zhenglun.wei at gmail.com Fri Jun 3 14:57:41 2011 From: zhenglun.wei at gmail.com (Alan Wei) Date: Fri, 3 Jun 2011 14:57:41 -0500 Subject: [petsc-users] Optimized Option Message-ID: Dear Sir/Madam, Lately, I read there is a optimized version of PETSc by using '--with-debugging=0'. I have two questions here: 1, Do I need to use '/config/configure.py ... --with-debugging=0' when I configure the PETSc in order to use the non-debugging version of PETSc? Therefore, if I want to switch PETSc between these two versions, I have to keep configuring. 2, I was using the '--with-debugging=0' to configure the PETSc. However, I tested the computational rate by executing the ex29.c. The non-debugging version does not improve the speed much; in the contrary, it slow down the speed a little bit. The comparison is attached here. Any idea on this. thanks in advance, Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ComputationalRateComparisons.xlsx Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet Size: 8601 bytes Desc: not available URL: From knepley at gmail.com Fri Jun 3 15:02:32 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 3 Jun 2011 15:02:32 -0500 Subject: [petsc-users] Optimized Option In-Reply-To: References: Message-ID: On Fri, Jun 3, 2011 at 2:57 PM, Alan Wei wrote: > Dear Sir/Madam, > Lately, I read there is a optimized version of PETSc by using > '--with-debugging=0'. I have two questions here: > 1, Do I need to use '/config/configure.py ... --with-debugging=0' when I > configure the PETSc in order to use the non-debugging version of PETSc? > Therefore, if I want to switch PETSc between these two versions, I have to > keep configuring. > Configure makes a directory with all the information in it (including libraries) for that particular configuration. You set the PETSC_ARCH env var to that string (e.g. linxu-gnu-c-debug) when you want to use a particular configuration. > 2, I was using the '--with-debugging=0' to configure the PETSc. However, I > tested the computational rate by executing the ex29.c. The non-debugging > version does not improve the speed much; in the contrary, it slow down the > speed a little bit. The comparison is attached here. Any idea on this. > The only way we can help on performance questions is to see the output of -log_summary. Matt > thanks in advance, > Alan > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jun 3 15:06:38 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 3 Jun 2011 15:06:38 -0500 Subject: [petsc-users] Optimized Option In-Reply-To: References: Message-ID: <06A3A57E-3783-40F9-8B6A-A60E53551DDD@mcs.anl.gov> On Jun 3, 2011, at 2:57 PM, Alan Wei wrote: > Dear Sir/Madam, > Lately, I read there is a optimized version of PETSc by using '--with-debugging=0'. I have two questions here: > 1, Do I need to use '/config/configure.py ... --with-debugging=0' when I configure the PETSc in order to use the non-debugging version of PETSc? Therefore, if I want to switch PETSc between these two versions, I have to keep configuring. ABSOLUTELY NOT. You use two different values of the environmental variable PETSC_ARCH and switch back and forth by changing the variable. You can even change it on the make command line. For example make mycode PETSC_ARCH=arch-opt Note that you can use any name you want for the values of PETSC_ARCH. I often use arch-debug and arch-opt to keep track of them. > 2, I was using the '--with-debugging=0' to configure the PETSc. However, I tested the computational rate by executing the ex29.c. The non-debugging version does not improve the speed much; in the contrary, it slow down the speed a little bit. The comparison is attached here. Any idea on this. It depends on the compiler and particular code you are running. We recommend using the option -log_summary to see how much time is spent in each part of the computation and how much it gets better with optimization. Barry > > thanks in advance, > Alan > > From balay at mcs.anl.gov Fri Jun 3 15:21:30 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 3 Jun 2011 15:21:30 -0500 (CDT) Subject: [petsc-users] Optimized Option In-Reply-To: <06A3A57E-3783-40F9-8B6A-A60E53551DDD@mcs.anl.gov> References: <06A3A57E-3783-40F9-8B6A-A60E53551DDD@mcs.anl.gov> Message-ID: On Fri, 3 Jun 2011, Barry Smith wrote: > > 2, I was using the '--with-debugging=0' to configure the PETSc. However, I tested the computational rate by executing the ex29.c. The non-debugging version does not improve the speed much; in the contrary, it slow down the speed a little bit. The comparison is attached here. Any idea on this. > > It depends on the compiler and particular code you are running. We recommend using the option -log_summary to see how much time is spent in each part of the computation and how much it gets better with optimization. > Not sure about slowdown - but if you have prebuilt mpi with mpicc automatically using -O2 internally - you might not see a difference between --with-debugging=0 or 1 Satish From zhenglun.wei at gmail.com Fri Jun 3 15:28:44 2011 From: zhenglun.wei at gmail.com (zhenglun.wei at gmail.com) Date: Fri, 03 Jun 2011 20:28:44 +0000 Subject: [petsc-users] Optimized Option In-Reply-To: <06A3A57E-3783-40F9-8B6A-A60E53551DDD@mcs.anl.gov> Message-ID: <90e6ba6e8ea4a813fb04a4d499e6@google.com> Thanks for your reply. So, if I want to use '--with-debugging=0', what is the PETSC_ARCH equal to? I just tried to use PETSC = arch-opt, and a error message comes up: ~/petsc-dev/conf/variables:95: ~/petsc-dev/arch-opt/conf/petscvariables: No such file or directory ~l/petsc-dev/conf/rules:1018: ~/petsc-dev/arch-opt/conf/petscrules: No such file or directory make: *** No rule to make target `~l/petsc-dev/arch-opt/conf/petscrules'. Stop. this really confused me. thanks, Alan On , Barry Smith wrote: > On Jun 3, 2011, at 2:57 PM, Alan Wei wrote: > > Dear Sir/Madam, > > Lately, I read there is a optimized version of PETSc by > using '--with-debugging=0'. I have two questions here: > > 1, Do I need to use '/config/configure.py ... --with-debugging=0' when > I configure the PETSc in order to use the non-debugging version of PETSc? > Therefore, if I want to switch PETSc between these two versions, I have > to keep configuring. > ABSOLUTELY NOT. You use two different values of the environmental > variable PETSC_ARCH and switch back and forth by changing the variable. > You can even change it on the make command line. For example make mycode > PETSC_ARCH=arch-opt > Note that you can use any name you want for the values of PETSC_ARCH. I > often use arch-debug and arch-opt to keep track of them. > > 2, I was using the '--with-debugging=0' to configure the PETSc. > However, I tested the computational rate by executing the ex29.c. The > non-debugging version does not improve the speed much; in the contrary, > it slow down the speed a little bit. The comparison is attached here. Any > idea on this. > It depends on the compiler and particular code you are running. We > recommend using the option -log_summary to see how much time is spent in > each part of the computation and how much it gets better with > optimization. > Barry > > > > thanks in advance, > > Alan > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 3 15:30:28 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 3 Jun 2011 15:30:28 -0500 Subject: [petsc-users] Optimized Option In-Reply-To: <90e6ba6e8ea4a813fb04a4d499e6@google.com> References: <06A3A57E-3783-40F9-8B6A-A60E53551DDD@mcs.anl.gov> <90e6ba6e8ea4a813fb04a4d499e6@google.com> Message-ID: On Fri, Jun 3, 2011 at 3:28 PM, wrote: > Thanks for your reply. > So, if I want to use '--with-debugging=0', what is the PETSC_ARCH equal to? > When you run configure, at the end it prints out a LARGE message that indicates - What PETSC_ARCH it has chosen - Exactly the command line to run when building This information is also stored in configure.log Matt I just tried to use PETSC = arch-opt, and a error message comes up: > ~/petsc-dev/conf/variables:95: ~/petsc-dev/arch-opt/conf/petscvariables: No > such file or directory > ~l/petsc-dev/conf/rules:1018: ~/petsc-dev/arch-opt/conf/petscrules: No such > file or directory > make: *** No rule to make target `~l/petsc-dev/arch-opt/conf/petscrules'. > Stop. > this really confused me. > > thanks, > Alan > > On , Barry Smith wrote: > > > > > > On Jun 3, 2011, at 2:57 PM, Alan Wei wrote: > > > > > > > > > Dear Sir/Madam, > > > > > Lately, I read there is a optimized version of PETSc by using > '--with-debugging=0'. I have two questions here: > > > > > 1, Do I need to use '/config/configure.py ... --with-debugging=0' when > I configure the PETSc in order to use the non-debugging version of PETSc? > Therefore, if I want to switch PETSc between these two versions, I have to > keep configuring. > > > > > > > > > > > > ABSOLUTELY NOT. You use two different values of the environmental > variable PETSC_ARCH and switch back and forth by changing the variable. You > can even change it on the make command line. For example make mycode > PETSC_ARCH=arch-opt > > > > > > > > Note that you can use any name you want for the values of PETSC_ARCH. I > often use arch-debug and arch-opt to keep track of them. > > > > > > > > > > > > > 2, I was using the '--with-debugging=0' to configure the PETSc. > However, I tested the computational rate by executing the ex29.c. The > non-debugging version does not improve the speed much; in the contrary, it > slow down the speed a little bit. The comparison is attached here. Any idea > on this. > > > > > > > > It depends on the compiler and particular code you are running. We > recommend using the option -log_summary to see how much time is spent in > each part of the computation and how much it gets better with optimization. > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > thanks in advance, > > > > > Alan > > > > > > > > > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jun 3 15:38:07 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 3 Jun 2011 15:38:07 -0500 Subject: [petsc-users] Optimized Option In-Reply-To: References: <06A3A57E-3783-40F9-8B6A-A60E53551DDD@mcs.anl.gov> <90e6ba6e8ea4a813fb04a4d499e6@google.com> Message-ID: <6F565F94-4D52-48EE-B168-A1B24BDCBE67@mcs.anl.gov> On Jun 3, 2011, at 3:30 PM, Matthew Knepley wrote: > On Fri, Jun 3, 2011 at 3:28 PM, wrote: > Thanks for your reply. > So, if I want to use '--with-debugging=0', what is the PETSC_ARCH equal to? > > When you run configure, at the end it prints out a LARGE message that indicates > > - What PETSC_ARCH it has chosen > > - Exactly the command line to run when building > > This information is also stored in configure.log You should provide a value for the environmental variable PETSC_ARCH before running ./configure For example PETSC_ARCH=arch-debug ./configure --with-debugging=1 otherstuff then PETSC_ARCH=arch-opt ./configure --with-debugging=0 otherstuff > > Matt > > I just tried to use PETSC = arch-opt, and a error message comes up: > ~/petsc-dev/conf/variables:95: ~/petsc-dev/arch-opt/conf/petscvariables: No such file or directory > ~l/petsc-dev/conf/rules:1018: ~/petsc-dev/arch-opt/conf/petscrules: No such file or directory > make: *** No rule to make target `~l/petsc-dev/arch-opt/conf/petscrules'. Stop. > this really confused me. > > thanks, > Alan > > On , Barry Smith wrote: > > > > > > On Jun 3, 2011, at 2:57 PM, Alan Wei wrote: > > > > > > > > > Dear Sir/Madam, > > > > > Lately, I read there is a optimized version of PETSc by using '--with-debugging=0'. I have two questions here: > > > > > 1, Do I need to use '/config/configure.py ... --with-debugging=0' when I configure the PETSc in order to use the non-debugging version of PETSc? Therefore, if I want to switch PETSc between these two versions, I have to keep configuring. > > > > > > > > > > > > ABSOLUTELY NOT. You use two different values of the environmental variable PETSC_ARCH and switch back and forth by changing the variable. You can even change it on the make command line. For example make mycode PETSC_ARCH=arch-opt > > > > > > > > Note that you can use any name you want for the values of PETSC_ARCH. I often use arch-debug and arch-opt to keep track of them. > > > > > > > > > > > > > 2, I was using the '--with-debugging=0' to configure the PETSc. However, I tested the computational rate by executing the ex29.c. The non-debugging version does not improve the speed much; in the contrary, it slow down the speed a little bit. The comparison is attached here. Any idea on this. > > > > > > > > It depends on the compiler and particular code you are running. We recommend using the option -log_summary to see how much time is spent in each part of the computation and how much it gets better with optimization. > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > thanks in advance, > > > > > Alan > > > > > > > > > > > > > > > > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From knepley at gmail.com Fri Jun 3 15:39:48 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 3 Jun 2011 15:39:48 -0500 Subject: [petsc-users] Optimized Option In-Reply-To: <6F565F94-4D52-48EE-B168-A1B24BDCBE67@mcs.anl.gov> References: <06A3A57E-3783-40F9-8B6A-A60E53551DDD@mcs.anl.gov> <90e6ba6e8ea4a813fb04a4d499e6@google.com> <6F565F94-4D52-48EE-B168-A1B24BDCBE67@mcs.anl.gov> Message-ID: On Fri, Jun 3, 2011 at 3:38 PM, Barry Smith wrote: > > On Jun 3, 2011, at 3:30 PM, Matthew Knepley wrote: > > > On Fri, Jun 3, 2011 at 3:28 PM, wrote: > > Thanks for your reply. > > So, if I want to use '--with-debugging=0', what is the PETSC_ARCH equal > to? > > > > When you run configure, at the end it prints out a LARGE message that > indicates > > > > - What PETSC_ARCH it has chosen > > > > - Exactly the command line to run when building > > > > This information is also stored in configure.log > > You should provide a value for the environmental variable PETSC_ARCH > before running ./configure > > For example PETSC_ARCH=arch-debug ./configure --with-debugging=1 > otherstuff > then PETSC_ARCH=arch-opt ./configure --with-debugging=0 otherstuff This is fragile since it depends on your shell. Just use ./configure --with-debugging=0 --PETSC_ARCH=arch-opt Matt > > > > Matt > > > > I just tried to use PETSC = arch-opt, and a error message comes up: > > ~/petsc-dev/conf/variables:95: ~/petsc-dev/arch-opt/conf/petscvariables: > No such file or directory > > ~l/petsc-dev/conf/rules:1018: ~/petsc-dev/arch-opt/conf/petscrules: No > such file or directory > > make: *** No rule to make target `~l/petsc-dev/arch-opt/conf/petscrules'. > Stop. > > this really confused me. > > > > thanks, > > Alan > > > > On , Barry Smith wrote: > > > > > > > > > On Jun 3, 2011, at 2:57 PM, Alan Wei wrote: > > > > > > > > > > > > > Dear Sir/Madam, > > > > > > > Lately, I read there is a optimized version of PETSc by using > '--with-debugging=0'. I have two questions here: > > > > > > > 1, Do I need to use '/config/configure.py ... --with-debugging=0' > when I configure the PETSc in order to use the non-debugging version of > PETSc? Therefore, if I want to switch PETSc between these two versions, I > have to keep configuring. > > > > > > > > > > > > > > > > > > ABSOLUTELY NOT. You use two different values of the environmental > variable PETSC_ARCH and switch back and forth by changing the variable. You > can even change it on the make command line. For example make mycode > PETSC_ARCH=arch-opt > > > > > > > > > > > > Note that you can use any name you want for the values of PETSC_ARCH. > I often use arch-debug and arch-opt to keep track of them. > > > > > > > > > > > > > > > > > > > 2, I was using the '--with-debugging=0' to configure the PETSc. > However, I tested the computational rate by executing the ex29.c. The > non-debugging version does not improve the speed much; in the contrary, it > slow down the speed a little bit. The comparison is attached here. Any idea > on this. > > > > > > > > > > > > It depends on the compiler and particular code you are running. We > recommend using the option -log_summary to see how much time is spent in > each part of the computation and how much it gets better with optimization. > > > > > > > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > > > > > > > thanks in advance, > > > > > > > Alan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenglun.wei at gmail.com Fri Jun 3 15:50:43 2011 From: zhenglun.wei at gmail.com (zhenglun.wei at gmail.com) Date: Fri, 03 Jun 2011 20:50:43 +0000 Subject: [petsc-users] Optimized Option In-Reply-To: Message-ID: <90e6ba61359447c04104a4d4e8ca@google.com> Thanks Matt, I think I got it. Does that mean I need to configure a PETSC_ARCH as --with-debugging=0 so that I can used it later on. best, Alan On , Matthew Knepley wrote: > On Fri, Jun 3, 2011 at 3:38 PM, Barry Smith bsmith at mcs.anl.gov> wrote: > On Jun 3, 2011, at 3:30 PM, Matthew Knepley wrote: > > On Fri, Jun 3, 2011 at 3:28 PM, zhenglun.wei at gmail.com> wrote: > > Thanks for your reply. > > So, if I want to use '--with-debugging=0', what is the PETSC_ARCH equal > to? > > > > When you run configure, at the end it prints out a LARGE message that > indicates > > > > - What PETSC_ARCH it has chosen > > > > - Exactly the command line to run when building > > > > This information is also stored in configure.log > You should provide a value for the environmental variable PETSC_ARCH > before running ./configure > For example PETSC_ARCH=arch-debug ./configure --with-debugging=1 > otherstuff > then PETSC_ARCH=arch-opt ./configure --with-debugging=0 otherstuff > This is fragile since it depends on your shell. Just use > ./configure --with-debugging=0 --PETSC_ARCH=arch-opt > Matt > > > > Matt > > > > I just tried to use PETSC = arch-opt, and a error message comes up: > > ~/petsc-dev/conf/variables:95: > ~/petsc-dev/arch-opt/conf/petscvariables: No such file or directory > > ~l/petsc-dev/conf/rules:1018: ~/petsc-dev/arch-opt/conf/petscrules: No > such file or directory > > make: *** No rule to make target > `~l/petsc-dev/arch-opt/conf/petscrules'. Stop. > > this really confused me. > > > > thanks, > > Alan > > > > On , Barry Smith bsmith at mcs.anl.gov> wrote: > > > > > > > > > On Jun 3, 2011, at 2:57 PM, Alan Wei wrote: > > > > > > > > > > > > > Dear Sir/Madam, > > > > > > > Lately, I read there is a optimized version of PETSc by > using '--with-debugging=0'. I have two questions here: > > > > > > > 1, Do I need to use '/config/configure.py ... --with-debugging=0' > when I configure the PETSc in order to use the non-debugging version of > PETSc? Therefore, if I want to switch PETSc between these two versions, I > have to keep configuring. > > > > > > > > > > > > > > > > > > ABSOLUTELY NOT. You use two different values of the environmental > variable PETSC_ARCH and switch back and forth by changing the variable. > You can even change it on the make command line. For example make mycode > PETSC_ARCH=arch-opt > > > > > > > > > > > > Note that you can use any name you want for the values of PETSC_ARCH. > I often use arch-debug and arch-opt to keep track of them. > > > > > > > > > > > > > > > > > > > 2, I was using the '--with-debugging=0' to configure the PETSc. > However, I tested the computational rate by executing the ex29.c. The > non-debugging version does not improve the speed much; in the contrary, > it slow down the speed a little bit. The comparison is attached here. Any > idea on this. > > > > > > > > > > > > It depends on the compiler and particular code you are running. We > recommend using the option -log_summary to see how much time is spent in > each part of the computation and how much it gets better with > optimization. > > > > > > > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > > > > > > > thanks in advance, > > > > > > > Alan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > > -- Norbert Wiener > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 3 15:52:10 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 3 Jun 2011 15:52:10 -0500 Subject: [petsc-users] Optimized Option In-Reply-To: <90e6ba61359447c04104a4d4e8ca@google.com> References: <90e6ba61359447c04104a4d4e8ca@google.com> Message-ID: On Fri, Jun 3, 2011 at 3:50 PM, wrote: > Thanks Matt, > I think I got it. Does that mean I need to configure a PETSC_ARCH as > --with-debugging=0 so that I can used it later on. > Yes, and build it. Matt > best, > Alan > > > On , Matthew Knepley wrote: > > On Fri, Jun 3, 2011 at 3:38 PM, Barry Smith bsmith at mcs.anl.gov> wrote: > > > > > > > > On Jun 3, 2011, at 3:30 PM, Matthew Knepley wrote: > > > > > > > > > On Fri, Jun 3, 2011 at 3:28 PM, zhenglun.wei at gmail.com> wrote: > > > > > Thanks for your reply. > > > > > So, if I want to use '--with-debugging=0', what is the PETSC_ARCH equal > to? > > > > > > > > > > When you run configure, at the end it prints out a LARGE message that > indicates > > > > > > > > > > - What PETSC_ARCH it has chosen > > > > > > > > > > - Exactly the command line to run when building > > > > > > > > > > This information is also stored in configure.log > > > > > > > > > > You should provide a value for the environmental variable PETSC_ARCH > before running ./configure > > > > > > > > For example PETSC_ARCH=arch-debug ./configure --with-debugging=1 > otherstuff > > > > then PETSC_ARCH=arch-opt ./configure --with-debugging=0 otherstuff > > > > This is fragile since it depends on your shell. Just use > > > > > > ./configure --with-debugging=0 --PETSC_ARCH=arch-opt > > > > > > > > Matt > > > > > > > > > > > > Matt > > > > > > > > > > I just tried to use PETSC = arch-opt, and a error message comes up: > > > > > ~/petsc-dev/conf/variables:95: > ~/petsc-dev/arch-opt/conf/petscvariables: No such file or directory > > > > > ~l/petsc-dev/conf/rules:1018: ~/petsc-dev/arch-opt/conf/petscrules: No > such file or directory > > > > > make: *** No rule to make target > `~l/petsc-dev/arch-opt/conf/petscrules'. Stop. > > > > > this really confused me. > > > > > > > > > > thanks, > > > > > Alan > > > > > > > > > > On , Barry Smith bsmith at mcs.anl.gov> wrote: > > > > > > > > > > > > > > > > > > On Jun 3, 2011, at 2:57 PM, Alan Wei wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Dear Sir/Madam, > > > > > > > > > > > > > Lately, I read there is a optimized version of PETSc by using > '--with-debugging=0'. I have two questions here: > > > > > > > > > > > > > 1, Do I need to use '/config/configure.py ... --with-debugging=0' > when I configure the PETSc in order to use the non-debugging version of > PETSc? Therefore, if I want to switch PETSc between these two versions, I > have to keep configuring. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ABSOLUTELY NOT. You use two different values of the environmental > variable PETSC_ARCH and switch back and forth by changing the variable. You > can even change it on the make command line. For example make mycode > PETSC_ARCH=arch-opt > > > > > > > > > > > > > > > > > > > > > > > > > > Note that you can use any name you want for the values of > PETSC_ARCH. I often use arch-debug and arch-opt to keep track of them. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2, I was using the '--with-debugging=0' to configure the PETSc. > However, I tested the computational rate by executing the ex29.c. The > non-debugging version does not improve the speed much; in the contrary, it > slow down the speed a little bit. The comparison is attached here. Any idea > on this. > > > > > > > > > > > > > > > > > > > > > > > > > > It depends on the compiler and particular code you are running. We > recommend using the option -log_summary to see how much time is spent in > each part of the computation and how much it gets better with optimization. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks in advance, > > > > > > > > > > > > > Alan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > > > > -- Norbert Wiener > > > > > > > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > > > -- Norbert Wiener > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From khalid_eee at yahoo.com Sat Jun 4 05:01:41 2011 From: khalid_eee at yahoo.com (khalid ashraf) Date: Sat, 4 Jun 2011 03:01:41 -0700 (PDT) Subject: [petsc-users] Copy entries of unequal size vector Message-ID: <593054.56877.qm@web112603.mail.gq1.yahoo.com> Hi, I want to copy 3 parallel vectors of size N into a vector of size 3N where the 3 vector entries goes as dest(3*i)=vec1(i); dest(1+3*i)=vec2(i); dest(2+3*i)=vec3(i); Could you please suggest what would be the most efficient way to copy for parallel performance. I guess it would be best if all the vectors have the same parallel layout. Is there a way to make sure that the vectors vec1(N elements) and dest(3N elements) have similar parallel layout ? Thanks in advance. Khalid -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Jun 4 06:21:07 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 4 Jun 2011 06:21:07 -0500 Subject: [petsc-users] Copy entries of unequal size vector In-Reply-To: <593054.56877.qm@web112603.mail.gq1.yahoo.com> References: <593054.56877.qm@web112603.mail.gq1.yahoo.com> Message-ID: http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Vec/VecStrideScatterAll.html Matt On Sat, Jun 4, 2011 at 5:01 AM, khalid ashraf wrote: > Hi, > I want to copy 3 parallel vectors of size N into a vector of size 3N where > the 3 vector entries goes as > dest(3*i)=vec1(i); > dest(1+3*i)=vec2(i); > dest(2+3*i)=vec3(i); > > Could you please suggest what would be the most efficient way to copy for > parallel performance. > > I guess it would be best if all the vectors have the same parallel layout. > Is there a way to make sure that the vectors vec1(N elements) and dest(3N > elements) have similar parallel layout ? > > Thanks in advance. > > Khalid > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Juergen.Duerrwang at iosb.fraunhofer.de Mon Jun 6 01:47:00 2011 From: Juergen.Duerrwang at iosb.fraunhofer.de (=?iso-8859-1?Q?D=FCrrwang=2C_J=FCrgen?=) Date: Mon, 6 Jun 2011 08:47:00 +0200 Subject: [petsc-users] Some papers for additive schwarz and blocked jacobi? Message-ID: Hi, I am working on a NVIDIA CUDA improved cg solver, with fast preconditioning. Since I have tested PETSC PCJACOBI with ILU(0) on each block I have some nice results. Unfortunately the pc is on CPU, so I try to port them on GPU. Now my question is, has body some material about this PETSC implementation? Kind regards, J?rgen D?rrwang -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jun 6 02:39:09 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 6 Jun 2011 02:39:09 -0500 Subject: [petsc-users] Some papers for additive schwarz and blocked jacobi? In-Reply-To: References: Message-ID: 2011/6/6 D?rrwang, J?rgen > Hi, > > > > I am working on a NVIDIA CUDA improved cg solver, with fast > preconditioning. Since I have tested PETSC PCJACOBI with ILU(0) on each > block I have some nice results. Unfortunately the pc is on CPU, so I try to > port them on GPU. Now my question is, has body some material about this > PETSC implementation? > The ILU(0) algorithm is described in Yousef Saad's book on Iterative Methods. It is notoriously hard to parallelize. ASM and Block-Jacobi jsut decompose the system into blocks. They are also described in the book. Matt > > > Kind regards, > > J?rgen D?rrwang > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Juergen.Duerrwang at iosb.fraunhofer.de Mon Jun 6 07:53:45 2011 From: Juergen.Duerrwang at iosb.fraunhofer.de (=?iso-8859-1?Q?D=FCrrwang=2C_J=FCrgen?=) Date: Mon, 6 Jun 2011 14:53:45 +0200 Subject: [petsc-users] Some papers for additive schwarz and blocked jacobi? In-Reply-To: References: Message-ID: Thanks, for help, I have read both algorithms in Saad. My main problem in understanding of the procedure is how the decompose the domain. At the moment I decompose with jacobi block method(without overlapping) and I run on each block LIU(0) in parallel after that I put each solution together for an CG solver. With that my iteration number raises instead of break down, so there must be something wrong :). Perhaps you know the order of the methods? First Jacobi-block then additive-Schwarz then ILU(0) and at least cg loop, or is it another order? Kind regards J?rgen D?rrwang Von: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] Im Auftrag von Matthew Knepley Gesendet: Montag, 6. Juni 2011 09:39 An: PETSc users list Betreff: Re: [petsc-users] Some papers for additive schwarz and blocked jacobi? 2011/6/6 D?rrwang, J?rgen > Hi, I am working on a NVIDIA CUDA improved cg solver, with fast preconditioning. Since I have tested PETSC PCJACOBI with ILU(0) on each block I have some nice results. Unfortunately the pc is on CPU, so I try to port them on GPU. Now my question is, has body some material about this PETSC implementation? The ILU(0) algorithm is described in Yousef Saad's book on Iterative Methods. It is notoriously hard to parallelize. ASM and Block-Jacobi jsut decompose the system into blocks. They are also described in the book. Matt Kind regards, J?rgen D?rrwang -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Mon Jun 6 08:38:29 2011 From: jed at 59A2.org (Jed Brown) Date: Mon, 6 Jun 2011 15:38:29 +0200 Subject: [petsc-users] Some papers for additive schwarz and blocked jacobi? In-Reply-To: References: Message-ID: 2011/6/6 D?rrwang, J?rgen > Thanks, > > > > for help, I have read both algorithms in Saad. My main problem in > understanding of the procedure is how the decompose the domain. At the > moment I decompose with jacobi block method(without overlapping) and I run > on each block LIU(0) in parallel after that I put each solution together for > an CG solver. With that my iteration number raises instead of break down, so > there must be something wrong J. > > > > Perhaps you know the order of the methods? First Jacobi-block then > additive-Schwarz then ILU(0) and at least cg loop, or is it another order? > I don't understand what you're asking. Have you looked at some PETSc examples? You should be able to do this stuff with run-time options. If you are trying something that doesn't work, send the options you ran with and the output. Running with -ksp_view is helpful to see what method is being used. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Juergen.Duerrwang at iosb.fraunhofer.de Mon Jun 6 09:27:47 2011 From: Juergen.Duerrwang at iosb.fraunhofer.de (=?utf-8?B?RMO8cnJ3YW5nLCBKw7xyZ2Vu?=) Date: Mon, 6 Jun 2011 16:27:47 +0200 Subject: [petsc-users] Some papers for additive schwarz and blocked jacobi? In-Reply-To: References: Message-ID: @ Jed Brown Yes, I tried some PETSC examples and I modified one for my stuff. It works very well on my Xeon quadcore, but my intention is to mix CPU and GPU code. I want a paralell domain decomposition using jacobi block method for runing ILU(0) on each block(number of blocks = number of CPU cores). Then I want to take the results of each blocksolution as a preconditioner for a cg solver on GPU. At the moment I can decompose my matrix in four jacobi block matrices. I compared my results with petsc and they are the same. But now I don?t know if I have to run my cg solver on each block or could I put the results of each blocked-ILU together and the use this as preconditioner for the non blocked matrix(my large input matrix). When I run my PETSC program with ?ksp_monitor/-ksp_view, I can only see that the matrix will be blocked and then ILU(0) will applied and after that there comes a cg solver. But there is no information about how to bring the results of each solved block together for using in the cg computing?.thats my issue Jed Brown wrote: I don't understand what you're asking. Have you looked at some PETSc examples? You should be able to do this stuff with run-time options. If you are trying something that doesn't work, send the options you ran with and the output. Running with -ksp_view is helpful to see what method is being used. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Mon Jun 6 09:42:46 2011 From: jed at 59A2.org (Jed Brown) Date: Mon, 6 Jun 2011 16:42:46 +0200 Subject: [petsc-users] Some papers for additive schwarz and blocked jacobi? In-Reply-To: References: Message-ID: 2011/6/6 D?rrwang, J?rgen > Yes, I tried some PETSC examples and I modified one for my stuff. It works > very well on my Xeon quadcore, but my intention is to mix CPU and GPU code. > I want a paralell domain decomposition using jacobi block method for runing > ILU(0) on each block(number of blocks = number of CPU cores). Then I want to > take the results of each blocksolution as a preconditioner for a cg solver > on GPU. > What is the GPU going to do while this is taking place on the CPU? I don't see much point doing CG on the GPU if you don't also move the matrix and preconditioner there. (The performance may even be worse than doing everything on the CPU.) Have you read the docs on running PETSc on GPUs? http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#gpus http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/installation.html#CUDA There is no ILU on the GPU because nobody has written it (because it seems to be ill-suited to the execution model). > > > At the moment I can decompose my matrix in four jacobi block matrices. I > compared my results with petsc and they are the same. But now I don?t know > if I have to run my cg solver on each block or could I put the results of > each blocked-ILU together and the use this as preconditioner for the non > blocked matrix(my large input matrix). > You can do either of these; -pc_type asm -sub_ksp_type cg -sub_pc_type icc, for example. Be careful about symmetry and remember to use FGMRES if you make the preconditioner nonlinear. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Juergen.Duerrwang at iosb.fraunhofer.de Mon Jun 6 10:16:19 2011 From: Juergen.Duerrwang at iosb.fraunhofer.de (=?utf-8?B?RMO8cnJ3YW5nLCBKw7xyZ2Vu?=) Date: Mon, 6 Jun 2011 17:16:19 +0200 Subject: [petsc-users] Some papers for additive schwarz and blocked jacobi? Message-ID: @Jed Brown I copy the hole matrix and the solved vectors from each ILU-Block(=preconditioner) to GPU where I can solve with cg. At the moment I have finished an cg solver on GPU using an algorithm from Saad. It is very fast. By a matrix of size 640000x640000 and about 4.500.000 non zero elements I need for a failure tolerance of 10e-3 only 900ms. But I want to have a mix of an stabile and fast solver, so I implemented a cg solver with ILU(0) preconditioning. Where the ILU is unfortunately a serial CPU implementation(ILU decompose and solve on CPU, cg operations on GPU). It computes for the same Matrix size the solution in 2,6s. So I thought if I can use all of my cpu cores instead of only one would be nice. And perhaps I can get the the 1,5s for computing. That?s the way I want to go: 1.Load Matrix which should be solve to CPU and GPU 2.Decompose in blocks, so on each block an ILU(0) can run in ?parallel?. : CPU 3.Loop until tolerance is reached 4.Solve each block in parallel to get an preconditioner : CPU 5.Solve CG with preconditioner to break down iteration number :GPU 6.End loop There are about 4 copies between CPU /GPU per step, but that isn?t a problem I haven?t seen the PETSC GPU manually until now?. Yes, I tried some PETSC examples and I modified one for my stuff. It works very well on my Xeon quadcore, but my intention is to mix CPU and GPU code. I want a paralell domain decomposition using jacobi block method for runing ILU(0) on each block(number of blocks = number of CPU cores). Then I want to take the results of each blocksolution as a preconditioner for a cg solver on GPU. What is the GPU going to do while this is taking place on the CPU? I don't see much point doing CG on the GPU if you don't also move the matrix and preconditioner there. (The performance may even be worse than doing everything on the CPU.) Have you read the docs on running PETSc on GPUs? http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#gpus http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/installation.html#CUDA There is no ILU on the GPU because nobody has written it (because it seems to be ill-suited to the execution model). At the moment I can decompose my matrix in four jacobi block matrices. I compared my results with petsc and they are the same. But now I don?t know if I have to run my cg solver on each block or could I put the results of each blocked-ILU together and the use this as preconditioner for the non blocked matrix(my large input matrix). You can do either of these; -pc_type asm -sub_ksp_type cg -sub_pc_type icc, for example. Be careful about symmetry and remember to use FGMRES if you make the preconditioner nonlinear. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Mon Jun 6 10:22:55 2011 From: jed at 59A2.org (Jed Brown) Date: Mon, 6 Jun 2011 17:22:55 +0200 Subject: [petsc-users] Some papers for additive schwarz and blocked jacobi? In-Reply-To: References: Message-ID: 2011/6/6 D?rrwang, J?rgen > 1.Load Matrix which should be solve to CPU and GPU > > 2.Decompose in blocks, so on each block an ILU(0) can run in > ?parallel?. : CPU > > 3.Loop until tolerance is reached > > 4.Solve each block in parallel to get an preconditioner > : CPU > > 5.Solve CG with preconditioner to break down iteration number > :GPU > Step 5 is not all on the GPU. You do a matrix multiple and a dot product on the GPU, then move the vector over to the CPU, put the pieces on different cores, solve, put it back on the GPU. > 6.End loop > > > > There are about 4 copies between CPU /GPU per step, but that isn?t a > problem > You have a copy each way *per CG iteration*. I think it is a problem. -------------- next part -------------- An HTML attachment was scrubbed... URL: From danesh.daroui at ltu.se Tue Jun 7 10:20:36 2011 From: danesh.daroui at ltu.se (Danesh Daroui) Date: Tue, 07 Jun 2011 17:20:36 +0200 Subject: [petsc-users] incomplete LU preconditioner Message-ID: <1307460036.9126.10.camel@linux-sclt.site> Hi, I want to use incomplete LU preconditioner. My code works OK when I use no preconditioner and direct solver for test: ierr=KSPCreate(PETSC_COMM_WORLD, &ksp); ierr=KSPSetOperators(ksp, Mp, Mp, DIFFERENT_NONZERO_PATTERN); ierr=KSPSetTolerances(ksp, 1.e-2/Msize, 1.e-50, PETSC_DEFAULT, PETSC_DEFAULT); // GMRES iterative solver is used ierr=KSPSetType(ksp, KSPGMRES); // no PETSc preconditioner is used ierr=KSPGetPC(ksp, &prec); ierr=PCSetType(prec, PCLU); // set up the solver according to the options ierr=KSPSetFromOptions(ksp); ierr=KSPSetUp(ksp); // solve the equation using an iterative solver ierr=KSPSolve(ksp, bp, xp); Again for some tests, at the first step, I want to use ILU but create the exact LU and then change the parameters to tune the preconditioner and I almost know what values to set according to my tests in MATLAB. I use the code below to create the get exact LU, using ILU preconditioner: ierr=KSPCreate(PETSC_COMM_WORLD, &ksp); ierr=KSPSetOperators(ksp, Mp, Mp, DIFFERENT_NONZERO_PATTERN); ierr=KSPSetTolerances(ksp, 1.e-2/Msize, 1.e-50, PETSC_DEFAULT, PETSC_DEFAULT); // GMRES iterative solver is used ierr=KSPSetType(ksp, KSPGMRES); // no PETSc preconditioner is used ierr=KSPGetPC(ksp, &prec); ierr=PCSetType(prec, PCILU); PetscReal dt=1e-10; PetscReal dtcol=0.1; PetscInt maxrowcount=M; PCFactorSetDropTolerance(prec, dt, dtcol, maxrowcount); // set up the solver according to the options ierr=KSPSetFromOptions(ksp); ierr=KSPSetUp(ksp); // solve the equation using an iterative solver ierr=KSPSolve(ksp, bp, xp); and my coefficient matrix is a square "MxM" matrix and I pass "M" as "maxrowcount". Using this code, the solver never converges while it solved the equation using previous code (direct solver). Can anybody let me know how can I get exact LU using ILU and what parameters should I change in the above code? Regards, D. From bsmith at mcs.anl.gov Tue Jun 7 10:28:54 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 7 Jun 2011 10:28:54 -0500 Subject: [petsc-users] incomplete LU preconditioner In-Reply-To: <1307460036.9126.10.camel@linux-sclt.site> References: <1307460036.9126.10.camel@linux-sclt.site> Message-ID: <054EBEDC-090F-4A00-A591-E34F6C3BF0E1@mcs.anl.gov> Sorry for the confusion, it is our fault. We don't have a drop tolerance ILU hence PCFactorSetDropTolerance() isn't doing anything. You can run with -pc_factor_levels M or call PCFactorSetLevels(pc,M) in the code where M is the size of the matrix. Barry On Jun 7, 2011, at 10:20 AM, Danesh Daroui wrote: > > Hi, > > I want to use incomplete LU preconditioner. My code works OK when I use > no preconditioner and direct solver for test: > > > ierr=KSPCreate(PETSC_COMM_WORLD, &ksp); > ierr=KSPSetOperators(ksp, Mp, Mp, DIFFERENT_NONZERO_PATTERN); > ierr=KSPSetTolerances(ksp, 1.e-2/Msize, 1.e-50, PETSC_DEFAULT, > PETSC_DEFAULT); > > // GMRES iterative solver is used > ierr=KSPSetType(ksp, KSPGMRES); > > // no PETSc preconditioner is used > ierr=KSPGetPC(ksp, &prec); > ierr=PCSetType(prec, PCLU); > > // set up the solver according to the options > ierr=KSPSetFromOptions(ksp); > > ierr=KSPSetUp(ksp); > > // solve the equation using an iterative solver > ierr=KSPSolve(ksp, bp, xp); > > > Again for some tests, at the first step, I want to use ILU but create > the exact LU and then change the parameters to tune the preconditioner > and I almost know what values to set according to my tests in MATLAB. I > use the code below to create the get exact LU, using ILU preconditioner: > > > > > > ierr=KSPCreate(PETSC_COMM_WORLD, &ksp); > ierr=KSPSetOperators(ksp, Mp, Mp, DIFFERENT_NONZERO_PATTERN); > ierr=KSPSetTolerances(ksp, 1.e-2/Msize, 1.e-50, PETSC_DEFAULT, > PETSC_DEFAULT); > > // GMRES iterative solver is used > ierr=KSPSetType(ksp, KSPGMRES); > > // no PETSc preconditioner is used > ierr=KSPGetPC(ksp, &prec); > ierr=PCSetType(prec, PCILU); > > PetscReal dt=1e-10; > PetscReal dtcol=0.1; > PetscInt maxrowcount=M; > PCFactorSetDropTolerance(prec, dt, dtcol, maxrowcount); > > // set up the solver according to the options > ierr=KSPSetFromOptions(ksp); > > ierr=KSPSetUp(ksp); > > // solve the equation using an iterative solver > ierr=KSPSolve(ksp, bp, xp); > > > and my coefficient matrix is a square "MxM" matrix and I pass "M" as > "maxrowcount". Using this code, the solver never converges while it > solved the equation using previous code (direct solver). Can anybody let > me know how can I get exact LU using ILU and what parameters should I > change in the above code? > > Regards, > > D. > > From danesh.daroui at ltu.se Tue Jun 7 10:44:15 2011 From: danesh.daroui at ltu.se (Danesh Daroui) Date: Tue, 07 Jun 2011 17:44:15 +0200 Subject: [petsc-users] incomplete LU preconditioner In-Reply-To: <054EBEDC-090F-4A00-A591-E34F6C3BF0E1@mcs.anl.gov> References: <1307460036.9126.10.camel@linux-sclt.site> <054EBEDC-090F-4A00-A591-E34F6C3BF0E1@mcs.anl.gov> Message-ID: <1307461455.9126.14.camel@linux-sclt.site> So, it is not implemented? That's too bad! ILU with drop threshold is very useful so I am wondering why it is not implemented yet? Do you have any plan to implement it? Setting levels did work OK for me. I have another question. If I want to calculate the preconditioner once, and use it later to solve other equations, since I know that the preconditioner would be OK because I solve several equations in several iterations and I know that my coefficient matrix will not change that much so calculating the preconditioner at once would be OK. What routines should I call once and how can I apply the pre-calculated preconditioner each time I want to solve the equation? Thanks, D. On Tue, 2011-06-07 at 10:28 -0500, Barry Smith wrote: > Sorry for the confusion, it is our fault. We don't have a drop tolerance ILU hence PCFactorSetDropTolerance() isn't doing anything. > > You can run with -pc_factor_levels M or call PCFactorSetLevels(pc,M) in the code where M is the size of the matrix. > > Barry > > On Jun 7, 2011, at 10:20 AM, Danesh Daroui wrote: > > > > > Hi, > > > > I want to use incomplete LU preconditioner. My code works OK when I use > > no preconditioner and direct solver for test: > > > > > > ierr=KSPCreate(PETSC_COMM_WORLD, &ksp); > > ierr=KSPSetOperators(ksp, Mp, Mp, DIFFERENT_NONZERO_PATTERN); > > ierr=KSPSetTolerances(ksp, 1.e-2/Msize, 1.e-50, PETSC_DEFAULT, > > PETSC_DEFAULT); > > > > // GMRES iterative solver is used > > ierr=KSPSetType(ksp, KSPGMRES); > > > > // no PETSc preconditioner is used > > ierr=KSPGetPC(ksp, &prec); > > ierr=PCSetType(prec, PCLU); > > > > // set up the solver according to the options > > ierr=KSPSetFromOptions(ksp); > > > > ierr=KSPSetUp(ksp); > > > > // solve the equation using an iterative solver > > ierr=KSPSolve(ksp, bp, xp); > > > > > > Again for some tests, at the first step, I want to use ILU but create > > the exact LU and then change the parameters to tune the preconditioner > > and I almost know what values to set according to my tests in MATLAB. I > > use the code below to create the get exact LU, using ILU preconditioner: > > > > > > > > > > > > ierr=KSPCreate(PETSC_COMM_WORLD, &ksp); > > ierr=KSPSetOperators(ksp, Mp, Mp, DIFFERENT_NONZERO_PATTERN); > > ierr=KSPSetTolerances(ksp, 1.e-2/Msize, 1.e-50, PETSC_DEFAULT, > > PETSC_DEFAULT); > > > > // GMRES iterative solver is used > > ierr=KSPSetType(ksp, KSPGMRES); > > > > // no PETSc preconditioner is used > > ierr=KSPGetPC(ksp, &prec); > > ierr=PCSetType(prec, PCILU); > > > > PetscReal dt=1e-10; > > PetscReal dtcol=0.1; > > PetscInt maxrowcount=M; > > PCFactorSetDropTolerance(prec, dt, dtcol, maxrowcount); > > > > // set up the solver according to the options > > ierr=KSPSetFromOptions(ksp); > > > > ierr=KSPSetUp(ksp); > > > > // solve the equation using an iterative solver > > ierr=KSPSolve(ksp, bp, xp); > > > > > > and my coefficient matrix is a square "MxM" matrix and I pass "M" as > > "maxrowcount". Using this code, the solver never converges while it > > solved the equation using previous code (direct solver). Can anybody let > > me know how can I get exact LU using ILU and what parameters should I > > change in the above code? > > > > Regards, > > > > D. > > > > > > From knepley at gmail.com Tue Jun 7 10:49:18 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 7 Jun 2011 10:49:18 -0500 Subject: [petsc-users] incomplete LU preconditioner In-Reply-To: <1307461455.9126.14.camel@linux-sclt.site> References: <1307460036.9126.10.camel@linux-sclt.site> <054EBEDC-090F-4A00-A591-E34F6C3BF0E1@mcs.anl.gov> <1307461455.9126.14.camel@linux-sclt.site> Message-ID: On Tue, Jun 7, 2011 at 10:44 AM, Danesh Daroui wrote: > > So, it is not implemented? That's too bad! ILU with drop threshold is > very useful so I am wondering why it is not implemented yet? Do you have > any plan to implement it? > It is not very scalable, so it is only really useful in a serial code. We tend to focus on parallel algorithms which are scalable. We don't have any plans to implement it, but we do take code submissions. > Setting levels did work OK for me. I have another question. If I want to > calculate the preconditioner once, and use it later to solve other > equations, since I know that the preconditioner would be OK because I > solve several equations in several iterations and I know that my > coefficient matrix will not change that much so calculating the > preconditioner at once would be OK. What routines should I call once and > how can I apply the pre-calculated preconditioner each time I want to > solve the equation? > You would use KSPSetOperators() with SAME_PRECONDITIONER. Matt > > Thanks, > > D. > > > > On Tue, 2011-06-07 at 10:28 -0500, Barry Smith wrote: > > Sorry for the confusion, it is our fault. We don't have a drop tolerance > ILU hence PCFactorSetDropTolerance() isn't doing anything. > > > > You can run with -pc_factor_levels M or call PCFactorSetLevels(pc,M) > in the code where M is the size of the matrix. > > > > Barry > > > > On Jun 7, 2011, at 10:20 AM, Danesh Daroui wrote: > > > > > > > > Hi, > > > > > > I want to use incomplete LU preconditioner. My code works OK when I use > > > no preconditioner and direct solver for test: > > > > > > > > > ierr=KSPCreate(PETSC_COMM_WORLD, &ksp); > > > ierr=KSPSetOperators(ksp, Mp, Mp, DIFFERENT_NONZERO_PATTERN); > > > ierr=KSPSetTolerances(ksp, 1.e-2/Msize, 1.e-50, PETSC_DEFAULT, > > > PETSC_DEFAULT); > > > > > > // GMRES iterative solver is used > > > ierr=KSPSetType(ksp, KSPGMRES); > > > > > > // no PETSc preconditioner is used > > > ierr=KSPGetPC(ksp, &prec); > > > ierr=PCSetType(prec, PCLU); > > > > > > // set up the solver according to the options > > > ierr=KSPSetFromOptions(ksp); > > > > > > ierr=KSPSetUp(ksp); > > > > > > // solve the equation using an iterative solver > > > ierr=KSPSolve(ksp, bp, xp); > > > > > > > > > Again for some tests, at the first step, I want to use ILU but create > > > the exact LU and then change the parameters to tune the preconditioner > > > and I almost know what values to set according to my tests in MATLAB. I > > > use the code below to create the get exact LU, using ILU > preconditioner: > > > > > > > > > > > > > > > > > > ierr=KSPCreate(PETSC_COMM_WORLD, &ksp); > > > ierr=KSPSetOperators(ksp, Mp, Mp, DIFFERENT_NONZERO_PATTERN); > > > ierr=KSPSetTolerances(ksp, 1.e-2/Msize, 1.e-50, PETSC_DEFAULT, > > > PETSC_DEFAULT); > > > > > > // GMRES iterative solver is used > > > ierr=KSPSetType(ksp, KSPGMRES); > > > > > > // no PETSc preconditioner is used > > > ierr=KSPGetPC(ksp, &prec); > > > ierr=PCSetType(prec, PCILU); > > > > > > PetscReal dt=1e-10; > > > PetscReal dtcol=0.1; > > > PetscInt maxrowcount=M; > > > PCFactorSetDropTolerance(prec, dt, dtcol, maxrowcount); > > > > > > // set up the solver according to the options > > > ierr=KSPSetFromOptions(ksp); > > > > > > ierr=KSPSetUp(ksp); > > > > > > // solve the equation using an iterative solver > > > ierr=KSPSolve(ksp, bp, xp); > > > > > > > > > and my coefficient matrix is a square "MxM" matrix and I pass "M" as > > > "maxrowcount". Using this code, the solver never converges while it > > > solved the equation using previous code (direct solver). Can anybody > let > > > me know how can I get exact LU using ILU and what parameters should I > > > change in the above code? > > > > > > Regards, > > > > > > D. > > > > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jun 7 10:49:49 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 7 Jun 2011 10:49:49 -0500 Subject: [petsc-users] incomplete LU preconditioner In-Reply-To: <1307461455.9126.14.camel@linux-sclt.site> References: <1307460036.9126.10.camel@linux-sclt.site> <054EBEDC-090F-4A00-A591-E34F6C3BF0E1@mcs.anl.gov> <1307461455.9126.14.camel@linux-sclt.site> Message-ID: <27ED11A3-49B8-4F52-BA6E-4BC882812297@mcs.anl.gov> On Jun 7, 2011, at 10:44 AM, Danesh Daroui wrote: > > So, it is not implemented? That's too bad! ILU with drop threshold is > very useful so I am wondering why it is not implemented yet? Do you have > any plan to implement it? We've found that providing a high-quality drop tolerance ILU is far from trivial and haven't had the time/person with expertise to devote the time needed to implement it. > > Setting levels did work OK for me. I have another question. If I want to > calculate the preconditioner once, and use it later to solve other > equations, since I know that the preconditioner would be OK because I > solve several equations in several iterations and I know that my > coefficient matrix will not change that much so calculating the > preconditioner at once would be OK. What routines should I call once and > how can I apply the pre-calculated preconditioner each time I want to > solve the equation? Since you are using the linear solvers directly. If you change the right hand side but NOT the matrix you simply call KSPSolve() again with the new right hand side. If you are changing the matrix but do not want to update the preconditioner call KSPSetOperators() with the new matrix but use a final argument of SAME_PRECONDITIONER Barry > > Thanks, > > D. > > > > On Tue, 2011-06-07 at 10:28 -0500, Barry Smith wrote: >> Sorry for the confusion, it is our fault. We don't have a drop tolerance ILU hence PCFactorSetDropTolerance() isn't doing anything. >> >> You can run with -pc_factor_levels M or call PCFactorSetLevels(pc,M) in the code where M is the size of the matrix. >> >> Barry >> >> On Jun 7, 2011, at 10:20 AM, Danesh Daroui wrote: >> >>> >>> Hi, >>> >>> I want to use incomplete LU preconditioner. My code works OK when I use >>> no preconditioner and direct solver for test: >>> >>> >>> ierr=KSPCreate(PETSC_COMM_WORLD, &ksp); >>> ierr=KSPSetOperators(ksp, Mp, Mp, DIFFERENT_NONZERO_PATTERN); >>> ierr=KSPSetTolerances(ksp, 1.e-2/Msize, 1.e-50, PETSC_DEFAULT, >>> PETSC_DEFAULT); >>> >>> // GMRES iterative solver is used >>> ierr=KSPSetType(ksp, KSPGMRES); >>> >>> // no PETSc preconditioner is used >>> ierr=KSPGetPC(ksp, &prec); >>> ierr=PCSetType(prec, PCLU); >>> >>> // set up the solver according to the options >>> ierr=KSPSetFromOptions(ksp); >>> >>> ierr=KSPSetUp(ksp); >>> >>> // solve the equation using an iterative solver >>> ierr=KSPSolve(ksp, bp, xp); >>> >>> >>> Again for some tests, at the first step, I want to use ILU but create >>> the exact LU and then change the parameters to tune the preconditioner >>> and I almost know what values to set according to my tests in MATLAB. I >>> use the code below to create the get exact LU, using ILU preconditioner: >>> >>> >>> >>> >>> >>> ierr=KSPCreate(PETSC_COMM_WORLD, &ksp); >>> ierr=KSPSetOperators(ksp, Mp, Mp, DIFFERENT_NONZERO_PATTERN); >>> ierr=KSPSetTolerances(ksp, 1.e-2/Msize, 1.e-50, PETSC_DEFAULT, >>> PETSC_DEFAULT); >>> >>> // GMRES iterative solver is used >>> ierr=KSPSetType(ksp, KSPGMRES); >>> >>> // no PETSc preconditioner is used >>> ierr=KSPGetPC(ksp, &prec); >>> ierr=PCSetType(prec, PCILU); >>> >>> PetscReal dt=1e-10; >>> PetscReal dtcol=0.1; >>> PetscInt maxrowcount=M; >>> PCFactorSetDropTolerance(prec, dt, dtcol, maxrowcount); >>> >>> // set up the solver according to the options >>> ierr=KSPSetFromOptions(ksp); >>> >>> ierr=KSPSetUp(ksp); >>> >>> // solve the equation using an iterative solver >>> ierr=KSPSolve(ksp, bp, xp); >>> >>> >>> and my coefficient matrix is a square "MxM" matrix and I pass "M" as >>> "maxrowcount". Using this code, the solver never converges while it >>> solved the equation using previous code (direct solver). Can anybody let >>> me know how can I get exact LU using ILU and what parameters should I >>> change in the above code? >>> >>> Regards, >>> >>> D. >>> >>> >> >> > > From danesh.daroui at ltu.se Tue Jun 7 11:05:40 2011 From: danesh.daroui at ltu.se (Danesh Daroui) Date: Tue, 07 Jun 2011 18:05:40 +0200 Subject: [petsc-users] incomplete LU preconditioner In-Reply-To: <27ED11A3-49B8-4F52-BA6E-4BC882812297@mcs.anl.gov> References: <1307460036.9126.10.camel@linux-sclt.site> <054EBEDC-090F-4A00-A591-E34F6C3BF0E1@mcs.anl.gov> <1307461455.9126.14.camel@linux-sclt.site> <27ED11A3-49B8-4F52-BA6E-4BC882812297@mcs.anl.gov> Message-ID: <1307462740.9126.17.camel@linux-sclt.site> Thanks, I changed my code to use same preconditioner. Some more questions: 1. I changed the level from 1730 to 28 and I expected "no convergence" or at convergence at more iterations or higher residual, but nothing changed. What can be the reason? 2. Using same preconditioner I see no change in the solution time. In all cases I get the convergence at max iteration 2 with very low residual, but it is a bit slow. Thanks, D. On Tue, 2011-06-07 at 10:49 -0500, Barry Smith wrote: > On Jun 7, 2011, at 10:44 AM, Danesh Daroui wrote: > > > > > So, it is not implemented? That's too bad! ILU with drop threshold is > > very useful so I am wondering why it is not implemented yet? Do you have > > any plan to implement it? > > We've found that providing a high-quality drop tolerance ILU is far from trivial and haven't had the time/person with expertise to devote the time needed to implement it. > > > > > Setting levels did work OK for me. I have another question. If I want to > > calculate the preconditioner once, and use it later to solve other > > equations, since I know that the preconditioner would be OK because I > > solve several equations in several iterations and I know that my > > coefficient matrix will not change that much so calculating the > > preconditioner at once would be OK. What routines should I call once and > > how can I apply the pre-calculated preconditioner each time I want to > > solve the equation? > > Since you are using the linear solvers directly. If you change the right hand side but NOT the matrix you simply call KSPSolve() again with the new right hand side. > > If you are changing the matrix but do not want to update the preconditioner call KSPSetOperators() with the new matrix but use a final argument of SAME_PRECONDITIONER > > Barry > > > > > Thanks, > > > > D. > > > > > > > > On Tue, 2011-06-07 at 10:28 -0500, Barry Smith wrote: > >> Sorry for the confusion, it is our fault. We don't have a drop tolerance ILU hence PCFactorSetDropTolerance() isn't doing anything. > >> > >> You can run with -pc_factor_levels M or call PCFactorSetLevels(pc,M) in the code where M is the size of the matrix. > >> > >> Barry > >> > >> On Jun 7, 2011, at 10:20 AM, Danesh Daroui wrote: > >> > >>> > >>> Hi, > >>> > >>> I want to use incomplete LU preconditioner. My code works OK when I use > >>> no preconditioner and direct solver for test: > >>> > >>> > >>> ierr=KSPCreate(PETSC_COMM_WORLD, &ksp); > >>> ierr=KSPSetOperators(ksp, Mp, Mp, DIFFERENT_NONZERO_PATTERN); > >>> ierr=KSPSetTolerances(ksp, 1.e-2/Msize, 1.e-50, PETSC_DEFAULT, > >>> PETSC_DEFAULT); > >>> > >>> // GMRES iterative solver is used > >>> ierr=KSPSetType(ksp, KSPGMRES); > >>> > >>> // no PETSc preconditioner is used > >>> ierr=KSPGetPC(ksp, &prec); > >>> ierr=PCSetType(prec, PCLU); > >>> > >>> // set up the solver according to the options > >>> ierr=KSPSetFromOptions(ksp); > >>> > >>> ierr=KSPSetUp(ksp); > >>> > >>> // solve the equation using an iterative solver > >>> ierr=KSPSolve(ksp, bp, xp); > >>> > >>> > >>> Again for some tests, at the first step, I want to use ILU but create > >>> the exact LU and then change the parameters to tune the preconditioner > >>> and I almost know what values to set according to my tests in MATLAB. I > >>> use the code below to create the get exact LU, using ILU preconditioner: > >>> > >>> > >>> > >>> > >>> > >>> ierr=KSPCreate(PETSC_COMM_WORLD, &ksp); > >>> ierr=KSPSetOperators(ksp, Mp, Mp, DIFFERENT_NONZERO_PATTERN); > >>> ierr=KSPSetTolerances(ksp, 1.e-2/Msize, 1.e-50, PETSC_DEFAULT, > >>> PETSC_DEFAULT); > >>> > >>> // GMRES iterative solver is used > >>> ierr=KSPSetType(ksp, KSPGMRES); > >>> > >>> // no PETSc preconditioner is used > >>> ierr=KSPGetPC(ksp, &prec); > >>> ierr=PCSetType(prec, PCILU); > >>> > >>> PetscReal dt=1e-10; > >>> PetscReal dtcol=0.1; > >>> PetscInt maxrowcount=M; > >>> PCFactorSetDropTolerance(prec, dt, dtcol, maxrowcount); > >>> > >>> // set up the solver according to the options > >>> ierr=KSPSetFromOptions(ksp); > >>> > >>> ierr=KSPSetUp(ksp); > >>> > >>> // solve the equation using an iterative solver > >>> ierr=KSPSolve(ksp, bp, xp); > >>> > >>> > >>> and my coefficient matrix is a square "MxM" matrix and I pass "M" as > >>> "maxrowcount". Using this code, the solver never converges while it > >>> solved the equation using previous code (direct solver). Can anybody let > >>> me know how can I get exact LU using ILU and what parameters should I > >>> change in the above code? > >>> > >>> Regards, > >>> > >>> D. > >>> > >>> > >> > >> > > > > > > From bsmith at mcs.anl.gov Tue Jun 7 11:08:22 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 7 Jun 2011 11:08:22 -0500 Subject: [petsc-users] incomplete LU preconditioner In-Reply-To: <1307462740.9126.17.camel@linux-sclt.site> References: <1307460036.9126.10.camel@linux-sclt.site> <054EBEDC-090F-4A00-A591-E34F6C3BF0E1@mcs.anl.gov> <1307461455.9126.14.camel@linux-sclt.site> <27ED11A3-49B8-4F52-BA6E-4BC882812297@mcs.anl.gov> <1307462740.9126.17.camel@linux-sclt.site> Message-ID: On Jun 7, 2011, at 11:05 AM, Danesh Daroui wrote: > > Thanks, I changed my code to use same preconditioner. Some more > questions: > > 1. I changed the level from 1730 to 28 and I expected "no convergence" > or at convergence at more iterations or higher residual, but nothing > changed. What can be the reason? 28 levels of fill is equivalent/results in a comple LU factorization > > 2. Using same preconditioner I see no change in the solution time. Run with -log_summary to see where the code is spending the time. If the ILU numeric factorization is not taking much of the run time then doing it less often will not be noticable. Barry > > In all cases I get the convergence at max iteration 2 with very low > residual, but it is a bit slow. > > Thanks, > > D. > > > On Tue, 2011-06-07 at 10:49 -0500, Barry Smith wrote: >> On Jun 7, 2011, at 10:44 AM, Danesh Daroui wrote: >> >>> >>> So, it is not implemented? That's too bad! ILU with drop threshold is >>> very useful so I am wondering why it is not implemented yet? Do you have >>> any plan to implement it? >> >> We've found that providing a high-quality drop tolerance ILU is far from trivial and haven't had the time/person with expertise to devote the time needed to implement it. >> >>> >>> Setting levels did work OK for me. I have another question. If I want to >>> calculate the preconditioner once, and use it later to solve other >>> equations, since I know that the preconditioner would be OK because I >>> solve several equations in several iterations and I know that my >>> coefficient matrix will not change that much so calculating the >>> preconditioner at once would be OK. What routines should I call once and >>> how can I apply the pre-calculated preconditioner each time I want to >>> solve the equation? >> >> Since you are using the linear solvers directly. If you change the right hand side but NOT the matrix you simply call KSPSolve() again with the new right hand side. >> >> If you are changing the matrix but do not want to update the preconditioner call KSPSetOperators() with the new matrix but use a final argument of SAME_PRECONDITIONER >> >> Barry >> >>> >>> Thanks, >>> >>> D. >>> >>> >>> >>> On Tue, 2011-06-07 at 10:28 -0500, Barry Smith wrote: >>>> Sorry for the confusion, it is our fault. We don't have a drop tolerance ILU hence PCFactorSetDropTolerance() isn't doing anything. >>>> >>>> You can run with -pc_factor_levels M or call PCFactorSetLevels(pc,M) in the code where M is the size of the matrix. >>>> >>>> Barry >>>> >>>> On Jun 7, 2011, at 10:20 AM, Danesh Daroui wrote: >>>> >>>>> >>>>> Hi, >>>>> >>>>> I want to use incomplete LU preconditioner. My code works OK when I use >>>>> no preconditioner and direct solver for test: >>>>> >>>>> >>>>> ierr=KSPCreate(PETSC_COMM_WORLD, &ksp); >>>>> ierr=KSPSetOperators(ksp, Mp, Mp, DIFFERENT_NONZERO_PATTERN); >>>>> ierr=KSPSetTolerances(ksp, 1.e-2/Msize, 1.e-50, PETSC_DEFAULT, >>>>> PETSC_DEFAULT); >>>>> >>>>> // GMRES iterative solver is used >>>>> ierr=KSPSetType(ksp, KSPGMRES); >>>>> >>>>> // no PETSc preconditioner is used >>>>> ierr=KSPGetPC(ksp, &prec); >>>>> ierr=PCSetType(prec, PCLU); >>>>> >>>>> // set up the solver according to the options >>>>> ierr=KSPSetFromOptions(ksp); >>>>> >>>>> ierr=KSPSetUp(ksp); >>>>> >>>>> // solve the equation using an iterative solver >>>>> ierr=KSPSolve(ksp, bp, xp); >>>>> >>>>> >>>>> Again for some tests, at the first step, I want to use ILU but create >>>>> the exact LU and then change the parameters to tune the preconditioner >>>>> and I almost know what values to set according to my tests in MATLAB. I >>>>> use the code below to create the get exact LU, using ILU preconditioner: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> ierr=KSPCreate(PETSC_COMM_WORLD, &ksp); >>>>> ierr=KSPSetOperators(ksp, Mp, Mp, DIFFERENT_NONZERO_PATTERN); >>>>> ierr=KSPSetTolerances(ksp, 1.e-2/Msize, 1.e-50, PETSC_DEFAULT, >>>>> PETSC_DEFAULT); >>>>> >>>>> // GMRES iterative solver is used >>>>> ierr=KSPSetType(ksp, KSPGMRES); >>>>> >>>>> // no PETSc preconditioner is used >>>>> ierr=KSPGetPC(ksp, &prec); >>>>> ierr=PCSetType(prec, PCILU); >>>>> >>>>> PetscReal dt=1e-10; >>>>> PetscReal dtcol=0.1; >>>>> PetscInt maxrowcount=M; >>>>> PCFactorSetDropTolerance(prec, dt, dtcol, maxrowcount); >>>>> >>>>> // set up the solver according to the options >>>>> ierr=KSPSetFromOptions(ksp); >>>>> >>>>> ierr=KSPSetUp(ksp); >>>>> >>>>> // solve the equation using an iterative solver >>>>> ierr=KSPSolve(ksp, bp, xp); >>>>> >>>>> >>>>> and my coefficient matrix is a square "MxM" matrix and I pass "M" as >>>>> "maxrowcount". Using this code, the solver never converges while it >>>>> solved the equation using previous code (direct solver). Can anybody let >>>>> me know how can I get exact LU using ILU and what parameters should I >>>>> change in the above code? >>>>> >>>>> Regards, >>>>> >>>>> D. >>>>> >>>>> >>>> >>>> >>> >>> >> >> > > > From hzhang at mcs.anl.gov Tue Jun 7 12:30:39 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 7 Jun 2011 12:30:39 -0500 Subject: [petsc-users] incomplete LU preconditioner In-Reply-To: References: <1307460036.9126.10.camel@linux-sclt.site> <054EBEDC-090F-4A00-A591-E34F6C3BF0E1@mcs.anl.gov> <1307461455.9126.14.camel@linux-sclt.site> <27ED11A3-49B8-4F52-BA6E-4BC882812297@mcs.anl.gov> <1307462740.9126.17.camel@linux-sclt.site> Message-ID: Danesh: >> Thanks, I changed my code to use same preconditioner. Some more >> questions: >> >> 1. I changed the level from 1730 to 28 and I expected "no convergence" >> or at convergence at more iterations or higher residual, but nothing >> changed. What can be the reason? > > ? 28 levels of fill is equivalent/results in a comple LU factorization Note, 28 levels of fill in petsc ilu() is different from what your mentioned "M" as "maxrowcount" in iludt(). What Barry means here is that ilu(28) is mathematically equivalent to a complete LU, but not as efficient as complete LU. Try to use a direct solver, e.g., mumps, (use runtime option '-pc_type lu -pc_factor_mat_solver_package mumps), likely you'll get much faster time than ilu(28). Few years ago, I wrote iludt for petsc and tested it on a collection of matrices. Comparing it with ilu(p) in petsc, I did not see any advantage of iludt over ilu. Do any of you have reference that demonstrates the advantage of iludt over ilu? Computationally, iludt requires full LU computation for each row, then discard entries that either smaller than droptol or take more spaces than "maxrowcount" with row sort. Analytically, dropping small entries not necessarily maintains the quality of the LU factor. Without row/column permutation, L factor gives larger multipliers resulting bad convergences. We realized that much more are needed to make our iludt useful. Due to time constraint and lack of good idea, I stopped this work :-( Hong >> 2. Using same preconditioner I see no change in the solution time. > > ? Run with -log_summary to see where the code is spending the time. If the ILU numeric factorization is not taking much of the run time then doing it less often will not be noticable. > > ? Barry > >> >> In all cases I get the convergence at max iteration 2 with very low >> residual, but it is a bit slow. >> >> Thanks, >> >> D. >> >> >> On Tue, 2011-06-07 at 10:49 -0500, Barry Smith wrote: >>> On Jun 7, 2011, at 10:44 AM, Danesh Daroui wrote: >>> >>>> >>>> So, it is not implemented? That's too bad! ILU with drop threshold is >>>> very useful so I am wondering why it is not implemented yet? Do you have >>>> any plan to implement it? >>> >>> ? We've found that providing a high-quality drop tolerance ILU is far from trivial and haven't had the time/person with expertise to devote the time needed to implement it. >>> >>>> >>>> Setting levels did work OK for me. I have another question. If I want to >>>> calculate the preconditioner once, and use it later to solve other >>>> equations, since I know that the preconditioner would be OK because I >>>> solve several equations in several iterations and I know that my >>>> coefficient matrix will not change that much so calculating the >>>> preconditioner at once would be OK. What routines should I call once and >>>> how can I apply the pre-calculated preconditioner each time I want to >>>> solve the equation? >>> >>> ? Since you are using the linear solvers directly. If you change the right hand side but NOT the matrix you simply call KSPSolve() again with the new right hand side. >>> >>> ? If you are changing the matrix but do not want to update the preconditioner call KSPSetOperators() with the new matrix but use a final argument of SAME_PRECONDITIONER >>> >>> ? Barry >>> >>>> >>>> Thanks, >>>> >>>> D. >>>> >>>> >>>> >>>> On Tue, 2011-06-07 at 10:28 -0500, Barry Smith wrote: >>>>> Sorry for the confusion, it is our fault. We don't have a drop tolerance ILU hence PCFactorSetDropTolerance() isn't doing anything. >>>>> >>>>> You can run with -pc_factor_levels ?M ? or call PCFactorSetLevels(pc,M) in the code ?where M is the size of the matrix. >>>>> >>>>> ?Barry >>>>> >>>>> On Jun 7, 2011, at 10:20 AM, Danesh Daroui wrote: >>>>> >>>>>> >>>>>> Hi, >>>>>> >>>>>> I want to use incomplete LU preconditioner. My code works OK when I use >>>>>> no preconditioner and direct solver for test: >>>>>> >>>>>> >>>>>> ierr=KSPCreate(PETSC_COMM_WORLD, &ksp); >>>>>> ierr=KSPSetOperators(ksp, Mp, Mp, DIFFERENT_NONZERO_PATTERN); >>>>>> ierr=KSPSetTolerances(ksp, 1.e-2/Msize, 1.e-50, PETSC_DEFAULT, >>>>>> PETSC_DEFAULT); >>>>>> >>>>>> // GMRES iterative solver is used >>>>>> ierr=KSPSetType(ksp, KSPGMRES); >>>>>> >>>>>> // no PETSc preconditioner is used >>>>>> ierr=KSPGetPC(ksp, &prec); >>>>>> ierr=PCSetType(prec, PCLU); >>>>>> >>>>>> // set up the solver according to the options >>>>>> ierr=KSPSetFromOptions(ksp); >>>>>> >>>>>> ierr=KSPSetUp(ksp); >>>>>> >>>>>> // solve the equation using an iterative solver >>>>>> ierr=KSPSolve(ksp, bp, xp); >>>>>> >>>>>> >>>>>> Again for some tests, at the first step, I want to use ILU but create >>>>>> the exact LU and then change the parameters to tune the preconditioner >>>>>> and I almost know what values to set according to my tests in MATLAB. I >>>>>> use the code below to create the get exact LU, using ILU preconditioner: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> ierr=KSPCreate(PETSC_COMM_WORLD, &ksp); >>>>>> ierr=KSPSetOperators(ksp, Mp, Mp, DIFFERENT_NONZERO_PATTERN); >>>>>> ierr=KSPSetTolerances(ksp, 1.e-2/Msize, 1.e-50, PETSC_DEFAULT, >>>>>> PETSC_DEFAULT); >>>>>> >>>>>> // GMRES iterative solver is used >>>>>> ierr=KSPSetType(ksp, KSPGMRES); >>>>>> >>>>>> // no PETSc preconditioner is used >>>>>> ierr=KSPGetPC(ksp, &prec); >>>>>> ierr=PCSetType(prec, PCILU); >>>>>> >>>>>> PetscReal dt=1e-10; >>>>>> PetscReal dtcol=0.1; >>>>>> PetscInt maxrowcount=M; >>>>>> PCFactorSetDropTolerance(prec, dt, dtcol, maxrowcount); >>>>>> >>>>>> // set up the solver according to the options >>>>>> ierr=KSPSetFromOptions(ksp); >>>>>> >>>>>> ierr=KSPSetUp(ksp); >>>>>> >>>>>> // solve the equation using an iterative solver >>>>>> ierr=KSPSolve(ksp, bp, xp); >>>>>> >>>>>> >>>>>> and my coefficient matrix is a square "MxM" matrix and I pass "M" as >>>>>> "maxrowcount". Using this code, the solver never converges while it >>>>>> solved the equation using previous code (direct solver). Can anybody let >>>>>> me know how can I get exact LU using ILU and what parameters should I >>>>>> change in the above code? >>>>>> >>>>>> Regards, >>>>>> >>>>>> D. >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> >> > > From haoxiang at yahoo.cn Tue Jun 7 22:20:57 2011 From: haoxiang at yahoo.cn (Xiang Hao) Date: Tue, 7 Jun 2011 21:20:57 -0600 Subject: [petsc-users] Examples on solving linear systems using GMRES and/or matrix-free method Message-ID: Hi all, I need to solve a PDE ( basically a Poisson's equation with Neumann boundary condition ), and I solved it using a steepest descent method, which is very slow. Now I want to solve the PDE using GMRES. In addition, I do have the matrix A, but I have a function which computes Ax, so I also need to use the matrix free method. I am very new to PETSc, so I am looking for examples on solving linear systems using GMRES and/or matrix-free method. Thanks, Xiang -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jun 8 06:38:34 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 8 Jun 2011 06:38:34 -0500 Subject: [petsc-users] Examples on solving linear systems using GMRES and/or matrix-free method In-Reply-To: References: Message-ID: On Tue, Jun 7, 2011 at 10:20 PM, Xiang Hao wrote: > Hi all, > > I need to solve a PDE ( basically a Poisson's equation with Neumann > boundary condition ), and I solved it using a steepest descent method, which > is very slow. Now I want to solve the PDE using GMRES. In addition, I do > have the matrix A, but I have a function which computes Ax, so I also need > to use the matrix free method. > > I am very new to PETSc, so I am looking for examples on solving linear > systems using GMRES and/or matrix-free method. > You can look at SNES ex5 for the Poisson equation (after setting lambda = 0.0). SNES ex19 also shows how to use geometric Multigrid. Its not hard to do this for Poisson. Thanks, Matt > Thanks, > Xiang > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From klaus.zimmermann at physik.uni-freiburg.de Wed Jun 8 07:05:14 2011 From: klaus.zimmermann at physik.uni-freiburg.de (Klaus Zimmermann) Date: Wed, 08 Jun 2011 14:05:14 +0200 Subject: [petsc-users] Advise on choice of iterative solver Message-ID: <4DEF657A.8010204@physik.uni-freiburg.de> Hi, I want to extract the most relevant eigenstates from a quadratic, complex symmetric generalized eigenvalue problem Ax = \lambda Bx. A is complex symmetric with 3*36 non-zeros per row, B is real with 36 non-zeros per row (subpattern), both with dimension of the order of up to 1e7. We use Krylov-Schur with shift-and-invert from slepc. With a direct solver for the inner part (at the moment mumps) everything works fine. To circumvent memory problems and improve parallelizability I would like to use an iterative solver for the inner part, but was unable to actually get one to work. What is also strange is that the setup as it is now works fine only with out true residuals. If we use the command line switch for true residuals the Krylov-Schur does not seem to convert. So my questions are: 1) Do you have any idea on why true residuals would work worse than the Krylov estimate? 2) What iterative solver/preconditioner would you suggest? Thanks in advance! Klaus From knepley at gmail.com Wed Jun 8 07:17:03 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 8 Jun 2011 07:17:03 -0500 Subject: [petsc-users] Advise on choice of iterative solver In-Reply-To: <4DEF657A.8010204@physik.uni-freiburg.de> References: <4DEF657A.8010204@physik.uni-freiburg.de> Message-ID: On Wed, Jun 8, 2011 at 7:05 AM, Klaus Zimmermann < klaus.zimmermann at physik.uni-freiburg.de> wrote: > Hi, > > I want to extract the most relevant eigenstates from a quadratic, complex > symmetric generalized eigenvalue problem Ax = \lambda Bx. A is complex > symmetric with 3*36 non-zeros per row, B is real with 36 non-zeros per row > (subpattern), both with dimension of the order of up to 1e7. We use > Krylov-Schur with shift-and-invert from slepc. With a direct solver for the > inner part (at the moment mumps) everything works fine. To circumvent memory > problems and improve parallelizability I would like to use an iterative > solver for the inner part, but was unable to actually get one to work. > What is also strange is that the setup as it is now works fine only with > out true residuals. If we use the command line switch for true residuals the > Krylov-Schur does not seem to convert. > So my questions are: > 1) Do you have any idea on why true residuals would work worse than the > Krylov estimate? > I am not sure what you mean here. The "true" residual distinction only has to do with stopping criteria, not with the solution returned. > 2) What iterative solver/preconditioner would you suggest? > Iterative solvers do not work without good preconditioners (which are really just other approximate solvers). And parallel preconditioners all depend on the operator. So, without knowing anything about the operator, we cannot make an intelligent recommendation. However, you might look at Elemental (http://code.google.com/p/elemental/) which solves the complex symmetric eigenproblem and is very scalable. Matt > Thanks in advance! > Klaus > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Wed Jun 8 07:23:45 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 8 Jun 2011 14:23:45 +0200 Subject: [petsc-users] Advise on choice of iterative solver In-Reply-To: References: <4DEF657A.8010204@physik.uni-freiburg.de> Message-ID: On Wed, Jun 8, 2011 at 14:17, Matthew Knepley wrote: > However, you might look at Elemental (http://code.google.com/p/elemental/) > which solves the complex symmetric eigenproblem and is very scalable. > Note that Elemental is for dense systems. To solve your problem, it's important to know where it came from. The average number of nonzeros per row doesn't tell us anything about it's mathematical structure which is needed to design a good solver. -------------- next part -------------- An HTML attachment was scrubbed... URL: From klaus.zimmermann at physik.uni-freiburg.de Wed Jun 8 07:34:45 2011 From: klaus.zimmermann at physik.uni-freiburg.de (Klaus Zimmermann) Date: Wed, 08 Jun 2011 14:34:45 +0200 Subject: [petsc-users] Advise on choice of iterative solver In-Reply-To: References: <4DEF657A.8010204@physik.uni-freiburg.de> Message-ID: <4DEF6C65.4070604@physik.uni-freiburg.de> Hi Jed, Hi Matthew, thanks for your quick responses! On 06/08/2011 02:23 PM, Jed Brown wrote: > On Wed, Jun 8, 2011 at 14:17, Matthew Knepley > wrote: > > However, you might look at Elemental > (http://code.google.com/p/elemental/) which solves the complex > symmetric eigenproblem and is very scalable. > > > Note that Elemental is for dense systems. > > > To solve your problem, it's important to know where it came from. The > average number of nonzeros per row doesn't tell us anything about it's > mathematical structure which is needed to design a good solver. We are doing quantum mechanical ab initio calculations. The Matrix stems from a two particle Hamiltonian in a product basis. Thus we have basis vectors S_{nm}. The sparseness is now due to the fact that the matrix element can only be non-zero if |n-n'|<4 and |m-m'|<4. Does this help or do you need more information? Like the matrix construction code? Thanks, Klaus From knepley at gmail.com Wed Jun 8 07:51:24 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 8 Jun 2011 07:51:24 -0500 Subject: [petsc-users] Advise on choice of iterative solver In-Reply-To: <4DEF6C65.4070604@physik.uni-freiburg.de> References: <4DEF657A.8010204@physik.uni-freiburg.de> <4DEF6C65.4070604@physik.uni-freiburg.de> Message-ID: On Wed, Jun 8, 2011 at 7:34 AM, Klaus Zimmermann < klaus.zimmermann at physik.uni-freiburg.de> wrote: > Hi Jed, Hi Matthew, > > thanks for your quick responses! > > On 06/08/2011 02:23 PM, Jed Brown wrote: > > On Wed, Jun 8, 2011 at 14:17, Matthew Knepley > > wrote: > > > > However, you might look at Elemental > > (http://code.google.com/p/elemental/) which solves the complex > > symmetric eigenproblem and is very scalable. > > > > > > Note that Elemental is for dense systems. > > > > > > To solve your problem, it's important to know where it came from. The > > average number of nonzeros per row doesn't tell us anything about it's > > mathematical structure which is needed to design a good solver. > > We are doing quantum mechanical ab initio calculations. The Matrix stems > from a two particle Hamiltonian in a product basis. Thus we have basis > vectors S_{nm}. The sparseness is now due to the fact that the matrix > element can only be non-zero if |n-n'|<4 and |m-m'|<4. > > Does this help or do you need more information? Like the matrix > construction code? > This does not just sound sparse, it sounds banded. Is this true? If so, you can use dense, banded solvers instead. Matt > Thanks, > Klaus > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From klaus.zimmermann at physik.uni-freiburg.de Wed Jun 8 07:59:40 2011 From: klaus.zimmermann at physik.uni-freiburg.de (Klaus Zimmermann) Date: Wed, 08 Jun 2011 14:59:40 +0200 Subject: [petsc-users] Advise on choice of iterative solver In-Reply-To: References: <4DEF657A.8010204@physik.uni-freiburg.de> <4DEF6C65.4070604@physik.uni-freiburg.de> Message-ID: <4DEF723C.8090402@physik.uni-freiburg.de> On 06/08/2011 02:51 PM, Matthew Knepley wrote: > On Wed, Jun 8, 2011 at 7:34 AM, Klaus Zimmermann > > wrote: > > Hi Jed, Hi Matthew, > > thanks for your quick responses! > > On 06/08/2011 02:23 PM, Jed Brown wrote: > > On Wed, Jun 8, 2011 at 14:17, Matthew Knepley > > >> wrote: > > > > However, you might look at Elemental > > (http://code.google.com/p/elemental/) which solves the complex > > symmetric eigenproblem and is very scalable. > > > > > > Note that Elemental is for dense systems. > > > > > > To solve your problem, it's important to know where it came from. The > > average number of nonzeros per row doesn't tell us anything about > it's > > mathematical structure which is needed to design a good solver. > > We are doing quantum mechanical ab initio calculations. The Matrix > stems from a two particle Hamiltonian in a product basis. Thus we > have basis vectors S_{nm}. The sparseness is now due to the fact > that the matrix element can only be non-zero if > |n-n'|<4 and |m-m'|<4. > > Does this help or do you need more information? Like the matrix > construction code? > > > This does not just sound sparse, it sounds banded. Is this true? If so, > you can use dense, banded solvers instead. Well, yes. Actually this is how it was done initially. However due to the product structure and another extension we want to do later this proved prohibitive in terms of memory usage. Klaus From jroman at dsic.upv.es Wed Jun 8 08:11:04 2011 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 8 Jun 2011 15:11:04 +0200 Subject: [petsc-users] Advise on choice of iterative solver In-Reply-To: <4DEF657A.8010204@physik.uni-freiburg.de> References: <4DEF657A.8010204@physik.uni-freiburg.de> Message-ID: <8E740483-BB9C-40EC-868E-1CD7D73285D7@dsic.upv.es> El 08/06/2011, a las 14:05, Klaus Zimmermann escribi?: > Hi, > > I want to extract the most relevant eigenstates from a quadratic, complex symmetric generalized eigenvalue problem Ax = \lambda Bx. A is complex symmetric with 3*36 non-zeros per row, B is real with 36 non-zeros per row (subpattern), both with dimension of the order of up to 1e7. We use Krylov-Schur with shift-and-invert from slepc. With a direct solver for the inner part (at the moment mumps) everything works fine. To circumvent memory problems and improve parallelizability I would like to use an iterative solver for the inner part, but was unable to actually get one to work. > What is also strange is that the setup as it is now works fine only with out true residuals. If we use the command line switch for true residuals the Krylov-Schur does not seem to convert. > So my questions are: > 1) Do you have any idea on why true residuals would work worse than the Krylov estimate? With shift-and-invert the Krylov estimate might be too optimistic, especially if the shift is far from the eigenvalue. The true residual may be more accurate, but if the tolerance is too small then the solver might fail to attain it. Try with a larger tolerance. Anyway, the true residual option should be used only if the actual residual norms that you get are large. > 2) What iterative solver/preconditioner would you suggest? Note that using an iterative method for the inner linear system usually requires a small tolerance in order to maintain stability of the Krylov-Schur process. Unless you have a very good preconditioner, this is normally not competitive with respect to MUMPS. As an alternative, consider using the new GD and JD solvers. Jose > > Thanks in advance! > Klaus From uerland at gmail.com Thu Jun 9 07:57:48 2011 From: uerland at gmail.com (Henning Sauerland) Date: Thu, 9 Jun 2011 14:57:48 +0200 Subject: [petsc-users] Error when switching from MPIAIJ to MPIBAIJ Message-ID: I'd like to get an idea of the solver performance when using mpibaij. Using my existing code based on mpiaij I simply swapped the preallocation routine and switched to mpibaij using the runtime flag. I basically perform the following function calls: MatCreate(PETSC_COMM_WORLD, &SystMat); MatSetSizes(SystMat, PETSC_DECIDE, PETSC_DECIDE, nDOFg, nDOFg); MatSetFromOptions(SystMat); MatMPIBAIJSetPreallocation(SystMat, 5, 30, PETSC_NULL,10, PETSC_NULL); VecCreate(PETSC_COMM_WORLD, &Rhs); VecSetSizes(Rhs, PETSC_DECIDE, nDOFg); VecSetFromOptions(Rhs); /* Filling SystMat and Rhs (currently using MatSetValues() ) */ ... VecDuplicate(*Rhs, &Sol); KSPCreate(PETSC_COMM_WORLD, &ksp); KSPSetOperators(ksp, *SystMat, *SystMat, DIFFERENT_NONZERO_PATTERN); KSPSetFromOptions(ksp); KSPSolve(ksp, *Rhs, Sol); My runtime options are: -mat_type mpibaij -ksp_type lgmres -ksp_rtol 1.e-8 -ksp_gmres_restart 200 -pc_type asm -log_summary This is what I get using 2 processors: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Nonconforming object sizes! [0]PETSC ERROR: Vector wrong size 34178 for scatter 34180 (scatter forward and vector from != ctx from size)! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 5, Mon Sep 27 11:51:54 CDT 2010 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ... on a ... Thu Jun 9 14:01:13 2011 [0]PETSC ERROR: Libraries linked from ... [0]PETSC ERROR: Configure run at Wed Oct 6 11:03:30 2010 [0]PETSC ERROR: Configure options --with-cc=mpicc --download-scalapack=1 --with-cxx=mpicxx --download-mumps=1 --download-ml=1 --download-parmetis=1 --download-f-blas-lapack=1 PETSC_ARCH=linux-gnu-c-debug --download-blacs=1 --with-fc=mpif90 --download-hypre=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: VecScatterBegin() line 1549 in src/vec/vec/utils/vscat.c [0]PETSC ERROR: PCApply_ASM() line 366 in src/ksp/pc/impls/asm/asm.c [0]PETSC ERROR: PCApply() line 357 in src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPInitialResidual() line 65 in src/ksp/ksp/interface/itres.c [0]PETSC ERROR: KSPSolve_LGMRES() line 376 in src/ksp/ksp/impls/gmres/lgmres/lgmres.c [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Nonconforming object sizes! [1]PETSC ERROR: Vector wrong size 34177 for scatter 34175 (scatter forward and vector from != ctx from size)! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 5, Mon Sep 27 11:51:54 CDT 2010 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ... on a ... Thu Jun 9 14:01:13 2011 [1]PETSC ERROR: Libraries linked from ... [1]PETSC ERROR: Configure run at Wed Oct 6 11:03:30 2010 [1]PETSC ERROR: Configure options --with-cc=mpicc --download-scalapack=1 --with-cxx=mpicxx --download-mumps=1 --download-ml=1 --download-parmetis=1 --download-f-blas-lapack=1 PETSC_ARCH=linux-gnu-c-debug --download-blacs=1 --with-fc=mpif90 --download-hypre=1 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: VecScatterBegin() line 1549 in src/vec/vec/utils/vscat.c [1]PETSC ERROR: PCApply_ASM() line 366 in src/ksp/pc/impls/asm/asm.c [1]PETSC ERROR: PCApply() line 357 in src/ksp/pc/interface/precon.c [1]PETSC ERROR: KSPInitialResidual() line 65 in src/ksp/ksp/interface/itres.c [1]PETSC ERROR: KSPSolve_LGMRES() line 376 in src/ksp/ksp/impls/gmres/lgmres/lgmres.c [1]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c Is there a basic mistake in my procedure? Thanks Henning From knepley at gmail.com Thu Jun 9 08:03:18 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Jun 2011 08:03:18 -0500 Subject: [petsc-users] Error when switching from MPIAIJ to MPIBAIJ In-Reply-To: References: Message-ID: On Thu, Jun 9, 2011 at 7:57 AM, Henning Sauerland wrote: > I'd like to get an idea of the solver performance when using mpibaij. Using > my existing code based on mpiaij I simply swapped the preallocation routine > and switched to mpibaij using the runtime flag. I basically perform the > following function calls: > 1) You can call both preallocation routines, but note that the block version takes smaller dnz/onz 2) If you want the matrix and vector to be divided the same way automatically, give them the same blocksize Thanks, Matt > MatCreate(PETSC_COMM_WORLD, &SystMat); > MatSetSizes(SystMat, PETSC_DECIDE, PETSC_DECIDE, nDOFg, nDOFg); > MatSetFromOptions(SystMat); > MatMPIBAIJSetPreallocation(SystMat, 5, 30, PETSC_NULL,10, PETSC_NULL); > > VecCreate(PETSC_COMM_WORLD, &Rhs); > VecSetSizes(Rhs, PETSC_DECIDE, nDOFg); > VecSetFromOptions(Rhs); > > /* Filling SystMat and Rhs (currently using MatSetValues() ) */ > ... > > VecDuplicate(*Rhs, &Sol); > > KSPCreate(PETSC_COMM_WORLD, &ksp); > KSPSetOperators(ksp, *SystMat, *SystMat, DIFFERENT_NONZERO_PATTERN); > KSPSetFromOptions(ksp); > KSPSolve(ksp, *Rhs, Sol); > > > My runtime options are: -mat_type mpibaij -ksp_type lgmres -ksp_rtol 1.e-8 > -ksp_gmres_restart 200 -pc_type asm -log_summary > > > This is what I get using 2 processors: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Nonconforming object sizes! > [0]PETSC ERROR: Vector wrong size 34178 for scatter 34180 (scatter forward > and vector from != ctx from size)! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 5, Mon Sep 27 11:51:54 > CDT 2010 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ... on a ... Thu Jun 9 14:01:13 2011 > [0]PETSC ERROR: Libraries linked from ... > [0]PETSC ERROR: Configure run at Wed Oct 6 11:03:30 2010 > [0]PETSC ERROR: Configure options --with-cc=mpicc --download-scalapack=1 > --with-cxx=mpicxx --download-mumps=1 --download-ml=1 --download-parmetis=1 > --download-f-blas-lapack=1 PETSC_ARCH=linux-gnu-c-debug --download-blacs=1 > --with-fc=mpif90 --download-hypre=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: VecScatterBegin() line 1549 in src/vec/vec/utils/vscat.c > [0]PETSC ERROR: PCApply_ASM() line 366 in src/ksp/pc/impls/asm/asm.c > [0]PETSC ERROR: PCApply() line 357 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPInitialResidual() line 65 in > src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: KSPSolve_LGMRES() line 376 in > src/ksp/ksp/impls/gmres/lgmres/lgmres.c > [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Nonconforming object sizes! > [1]PETSC ERROR: Vector wrong size 34177 for scatter 34175 (scatter forward > and vector from != ctx from size)! > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 5, Mon Sep 27 11:51:54 > CDT 2010 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: ... on a ... Thu Jun 9 14:01:13 2011 > [1]PETSC ERROR: Libraries linked from ... > [1]PETSC ERROR: Configure run at Wed Oct 6 11:03:30 2010 > [1]PETSC ERROR: Configure options --with-cc=mpicc --download-scalapack=1 > --with-cxx=mpicxx --download-mumps=1 --download-ml=1 --download-parmetis=1 > --download-f-blas-lapack=1 PETSC_ARCH=linux-gnu-c-debug --download-blacs=1 > --with-fc=mpif90 --download-hypre=1 > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: VecScatterBegin() line 1549 in src/vec/vec/utils/vscat.c > [1]PETSC ERROR: PCApply_ASM() line 366 in src/ksp/pc/impls/asm/asm.c > [1]PETSC ERROR: PCApply() line 357 in src/ksp/pc/interface/precon.c > [1]PETSC ERROR: KSPInitialResidual() line 65 in > src/ksp/ksp/interface/itres.c > [1]PETSC ERROR: KSPSolve_LGMRES() line 376 in > src/ksp/ksp/impls/gmres/lgmres/lgmres.c > [1]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c > > Is there a basic mistake in my procedure? > > > Thanks > Henning -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Thu Jun 9 08:12:52 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 9 Jun 2011 15:12:52 +0200 Subject: [petsc-users] Error when switching from MPIAIJ to MPIBAIJ In-Reply-To: References: Message-ID: On Thu, Jun 9, 2011 at 15:03, Matthew Knepley wrote: > 2) If you want the matrix and vector to be divided the same way > automatically, give them the same blocksize Alternatively, just call MatGetVecs() to get compatible vectors after the matrix is configured. Depending on how your spatial discretization is managed, you may want to specify the local sizes instead of letting PETSc choose something based on the global sizes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hung.thanh.nguyen at petrell.no Thu Jun 9 08:08:48 2011 From: hung.thanh.nguyen at petrell.no (Hung Thanh Nguyen) Date: Thu, 09 Jun 2011 15:08:48 +0200 Subject: [petsc-users] Petsc-Link-error? In-Reply-To: References: Message-ID: Hei all I used Microsoft Visual Stud . Run Petsc in this platform and got the following error: Error 3 error LNK2005: __initp_misc_invarg already defined in LIBCMTD.lib(invarg.obj) C:\Users\Hung\Documents\Visual Studio 2005\Projects\petsc-testing\petsc-testing\LIBCMT.lib(invarg.obj) Error 4 error LNK2005: __invoke_watson already defined in LIBCMTD.lib(invarg.obj) C:\Users\Hung\Documents\Visual Studio 2005\Projects\petsc-testing\petsc-testing\LIBCMT.lib(invarg.obj) ... Error 10 error LNK2005: ___pInvalidArgHandler already defined in LIBCMTD.lib(invarg.obj) C:\Users\Hung\Documents\Visual Studio 2005\Projects\petsc-testing\petsc-testing\LIBCMT.lib(invarg.obj) Error 12 fatal error LNK1169: one or more multiply defined symbols found C:\Users\Hung\Documents\Visual Studio 2005\Projects\petsc-testing\Debug\petsc-testing.exe I try : Linker -Input-Ignore Specific library: LIBCMT.lib, and then got new error message : Error 3 error LNK2019: unresolved external symbol __invalid_parameter_noinfo referenced in function "public: char const & __thiscall std::basic_string,class std::allocator >::operator[](unsigned int)const " (??A?$basic_string at DU?$char_traits at D@std@@V?$allocator at D@2@@std@@QBEABDI at Z) C:\Users\Hung\Documents\Visual Studio 2005\Projects\petsc-testing\petsc-testing\libpetsc.lib(err.o) Error 4 error LNK2001: unresolved external symbol __invalid_parameter_noinfo C:\Users\Hung\Documents\Visual Studio 2005\Projects\petsc-testing\petsc-testing\libpetsc.lib(errtrace.o) ... Error 6 fatal error LNK1120: 1 unresolved externals C:\Users\Hung\Documents\Visual Studio 2005\Projects\petsc-testing\Debug\petsc-testing.exe Could someone tell me how to resolve it. Best regards hung Nguyen From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Xiang Hao Sent: 8. juni 2011 05:21 To: petsc-users at mcs.anl.gov Subject: [petsc-users] Examples on solving linear systems using GMRES and/or matrix-free method Hi all, I need to solve a PDE ( basically a Poisson's equation with Neumann boundary condition ), and I solved it using a steepest descent method, which is very slow. Now I want to solve the PDE using GMRES. In addition, I do have the matrix A, but I have a function which computes Ax, so I also need to use the matrix free method. I am very new to PETSc, so I am looking for examples on solving linear systems using GMRES and/or matrix-free method. Thanks, Xiang -------------- next part -------------- An HTML attachment was scrubbed... URL: From uerland at gmail.com Thu Jun 9 08:25:57 2011 From: uerland at gmail.com (Henning Sauerland) Date: Thu, 9 Jun 2011 15:25:57 +0200 Subject: [petsc-users] Error when switching from MPIAIJ to MPIBAIJ In-Reply-To: References: Message-ID: Thanks for the quick reply. On 09.06.2011, at 15:12, Jed Brown wrote: > On Thu, Jun 9, 2011 at 15:03, Matthew Knepley wrote: > 2) If you want the matrix and vector to be divided the same way automatically, give them the same blocksize > > Alternatively, just call MatGetVecs() to get compatible vectors after the matrix is configured. > > Depending on how your spatial discretization is managed, you may want to specify the local sizes instead of letting PETSc choose something based on the global sizes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lizs at mail.uc.edu Thu Jun 9 17:05:33 2011 From: lizs at mail.uc.edu (Li, Zhisong (lizs)) Date: Thu, 9 Jun 2011 22:05:33 +0000 Subject: [petsc-users] Multiple vectors read/written into one file in binary/HDF5 format Message-ID: Hi, Petsc Team, I know that we can write multiple vectors or matrices into one binary file by doing PetscViewerBinaryOpen(COMM, "./vector.dat", FILE_MODE_WRITE, &viewer1); VecView(a, viewer1); PetscViewerBinaryOpen(COMM, "./vector.dat", FILE_MODE_APPEND, &viewer1); VecView(b, viewer1); But if we want to read from "./vector.dat" and distribute the respective values to vector a and b, how to realize that? What I previously knew is "one file for one vector only" when reading. Another question is about the HDF5 format. I've included "petscsys.h" at the code header. But when I complied (make file) the code, the error message indicated "undefined reference to PetscViewerHDF5Open". The error also happens to the example codes for this function. What's wrong with is? Is it no more supported? Thank you. Zhisong Li -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 9 17:08:41 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Jun 2011 17:08:41 -0500 Subject: [petsc-users] Multiple vectors read/written into one file in binary/HDF5 format In-Reply-To: References: Message-ID: On Thu, Jun 9, 2011 at 5:05 PM, Li, Zhisong (lizs) wrote: > Hi, Petsc Team, > > > I know that we can write multiple vectors or matrices into one binary file > by doing > > PetscViewerBinaryOpen(COMM, "./vector.dat", FILE_MODE_WRITE, &viewer1); > > VecView(a, viewer1); > PetscViewerBinaryOpen(COMM, "./vector.dat", FILE_MODE_APPEND, &viewer1); > VecView(b, viewer1); > Just call VecView twice without reopening the file. > > But if we want to read from "./vector.dat" and distribute the respective > values to vector a and b, how to realize that? What I previously knew is > "one file for one vector only" when reading. > Just call VecLoad twice. > Another question is about the HDF5 format. I've included "petscsys.h" at > the code header. But when I complied (make file) the code, the error > message indicated "undefined reference to PetscViewerHDF5Open". The error > also happens to the example codes for this function. What's wrong with is? > Is it no more supported? > Did you configure with support for HDF5, --with-hdf5? Thanks, Matt > > Thank you. > > > > Zhisong Li > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenglun.wei at gmail.com Thu Jun 9 18:01:30 2011 From: zhenglun.wei at gmail.com (zhenglun.wei at gmail.com) Date: Thu, 09 Jun 2011 23:01:30 +0000 Subject: [petsc-users] Output question for ex29. Message-ID: <90e6ba2121bd074c2104a54f6fbe@google.com> Dear Sir/Madam, I'm still studying on the ex29 of /src/ksp/ksp/example/tutorial. Before I met a problem on VecView_VTK in parallel computation. I'm trying to modify it in order to output some data from the computation. 1) My first questions is that what does this section do in VecView_VTK: 272: MPI_Comm_rank(comm, &rank); 273: MPI_Comm_size(comm, &size); 274: MPI_Reduce(&n, &maxn, 1, MPIU_INT, MPI_MAX, 0, comm); 275: tag = ((PetscObject) viewer)->tag; 276: if (!rank) { 277: PetscMalloc((maxn+1) * sizeof(PetscScalar), &values); 278: for(i = 0; i < n; i++) { 279: PetscViewerASCIIPrintf(viewer, "%G\n", PetscRealPart(array[i])); 280: } 281: for(p = 1; p < size; p++) { 282: MPI_Recv(values, (PetscMPIInt) n, MPIU_SCALAR, p, tag, comm, &status); 283: MPI_Get_count(&status, MPIU_SCALAR, &nn); 284: for(i = 0; i < nn; i++) { 285: PetscViewerASCIIPrintf(viewer, "%G\n", PetscRealPart(array[i])); 286: } 287: } 288: PetscFree(values); 289: } else { 290: MPI_Send(array, n, MPIU_SCALAR, 0, tag, comm); 291: } What I understand is: it gather all the data from different process in parallel computation, and output it to the 'viewer'. I comment out everything in VecView_VTK except this part, there is no error message coming up in my parallel computation so far. 2) however, I really don't know how did it split the domain for parallel computation. For example, if I use 4 processes, is the domain split like: a) | 0 | 1 | ---------------|-------------- | 2 | 3 | b) | 0 | 2 | ---------------|-------------- | 1 | 3 | c) | | | | | | | | 0 | | 2 | | | 1 | | 3 | | | | | | | | | d) 0 ------------------------ 1 ------------------------ 2 ------------------------ 3 thanks in advance, Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 9 18:18:28 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Jun 2011 18:18:28 -0500 Subject: [petsc-users] Output question for ex29. In-Reply-To: <90e6ba2121bd074c2104a54f6fbe@google.com> References: <90e6ba2121bd074c2104a54f6fbe@google.com> Message-ID: On Thu, Jun 9, 2011 at 6:01 PM, wrote: > Dear Sir/Madam, > I'm still studying on the ex29 of /src/ksp/ksp/example/tutorial. Before I > met a problem on VecView_VTK in parallel computation. I'm trying to modify > it in order to output some data from the computation. > > 1) My first questions is that what does this section do in VecView_VTK: > > 272: MPI_Comm_rank(comm, &rank); > 273: MPI_Comm_size(comm, &size); > 274: MPI_Reduce(&n, &maxn, 1, MPIU_INT, MPI_MAX, 0, comm); > 275: tag = ((PetscObject) viewer)->tag; > 276: if (!rank) { > 277: PetscMalloc((maxn+1) * sizeof(PetscScalar), &values); > 278: for(i = 0; i < n; i++) { > 279: PetscViewerASCIIPrintf(viewer, "%G\n", PetscRealPart(array[i])); > 280: } > 281: for(p = 1; p < size; p++) { > 282: MPI_Recv(values, (PetscMPIInt) n, MPIU_SCALAR, p, tag, comm, &status); > 283: MPI_Get_count(&status, MPIU_SCALAR, &nn); > 284: for(i = 0; i < nn; i++) { > 285: PetscViewerASCIIPrintf(viewer, "%G\n", PetscRealPart(array[i])); > 286: } > 287: } > 288: PetscFree(values); > 289: } else { > 290: MPI_Send(array, n, MPIU_SCALAR, 0, tag, comm); > 291: } > > What I understand is: it gather all the data from different process in > parallel computation, and output it to the 'viewer'. I comment out > everything in VecView_VTK except this part, there is no error message coming > up in my parallel computation so far. > > 2) however, I really don't know how did it split the domain for parallel > computation. For example, if I use 4 processes, is the domain split like: > The DMDA describes the domain splitting. Matt > a) > | > 0 | 1 > | > ---------------|-------------- > | > 2 | 3 > | > > b) > | > 0 | 2 > | > ---------------|-------------- > | > 1 | 3 > | > > c) > | | | | > | | | | > 0 | | 2 | | > | 1 | | 3 | > | | | | > | | | | > > d) > 0 > ------------------------ > 1 > ------------------------ > 2 > ------------------------ > 3 > > thanks in advance, > Alan -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59a2.org Thu Jun 9 18:21:21 2011 From: jed at 59a2.org (Jed Brown) Date: Fri, 10 Jun 2011 01:21:21 +0200 Subject: [petsc-users] Output question for ex29. In-Reply-To: <90e6ba2121bd074c2104a54f6fbe@google.com> References: <90e6ba2121bd074c2104a54f6fbe@google.com> Message-ID: On Fri, Jun 10, 2011 at 01:01, wrote: > I'm still studying on the ex29 of /src/ksp/ksp/example/tutorial. Before I > met a problem on VecView_VTK in parallel computation. I'm trying to modify > it in order to output some data from the computation. > We really need to delete this function because it's wrong and leads to confusion like this. You can view to VTK with PetscViewer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 9 18:26:07 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Jun 2011 18:26:07 -0500 Subject: [petsc-users] Output question for ex29. In-Reply-To: References: <90e6ba2121bd074c2104a54f6fbe@google.com> Message-ID: On Thu, Jun 9, 2011 at 6:18 PM, Matthew Knepley wrote: > On Thu, Jun 9, 2011 at 6:01 PM, wrote: > >> Dear Sir/Madam, >> I'm still studying on the ex29 of /src/ksp/ksp/example/tutorial. Before I >> met a problem on VecView_VTK in parallel computation. I'm trying to modify >> it in order to output some data from the computation. >> > Here is a better answer. If you want output, throw away this old function which is broken, and use PetscViewerASCIIOpen() PetscViewerASCIISetFormat(PETSC_VIEWER_ASCII_VTK) DMView() VecView() PetscViewerDestroy() Thanks, Matt > 1) My first questions is that what does this section do in VecView_VTK: >> >> 272: MPI_Comm_rank(comm, &rank); >> 273: MPI_Comm_size(comm, &size); >> 274: MPI_Reduce(&n, &maxn, 1, MPIU_INT, MPI_MAX, 0, comm); >> 275: tag = ((PetscObject) viewer)->tag; >> 276: if (!rank) { >> 277: PetscMalloc((maxn+1) * sizeof(PetscScalar), &values); >> 278: for(i = 0; i < n; i++) { >> 279: PetscViewerASCIIPrintf(viewer, "%G\n", PetscRealPart(array[i])); >> 280: } >> 281: for(p = 1; p < size; p++) { >> 282: MPI_Recv(values, (PetscMPIInt) n, MPIU_SCALAR, p, tag, comm, >> &status); >> 283: MPI_Get_count(&status, MPIU_SCALAR, &nn); >> 284: for(i = 0; i < nn; i++) { >> 285: PetscViewerASCIIPrintf(viewer, "%G\n", PetscRealPart(array[i])); >> 286: } >> 287: } >> 288: PetscFree(values); >> 289: } else { >> 290: MPI_Send(array, n, MPIU_SCALAR, 0, tag, comm); >> 291: } >> >> What I understand is: it gather all the data from different process in >> parallel computation, and output it to the 'viewer'. I comment out >> everything in VecView_VTK except this part, there is no error message coming >> up in my parallel computation so far. >> >> 2) however, I really don't know how did it split the domain for parallel >> computation. For example, if I use 4 processes, is the domain split like: >> > > The DMDA describes the domain splitting. > > Matt > > >> a) >> | >> 0 | 1 >> | >> ---------------|-------------- >> | >> 2 | 3 >> | >> >> b) >> | >> 0 | 2 >> | >> ---------------|-------------- >> | >> 1 | 3 >> | >> >> c) >> | | | | >> | | | | >> 0 | | 2 | | >> | 1 | | 3 | >> | | | | >> | | | | >> >> d) >> 0 >> ------------------------ >> 1 >> ------------------------ >> 2 >> ------------------------ >> 3 >> >> thanks in advance, >> Alan > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenglun.wei at gmail.com Fri Jun 10 15:05:21 2011 From: zhenglun.wei at gmail.com (zhenglun.wei at gmail.com) Date: Fri, 10 Jun 2011 20:05:21 +0000 Subject: [petsc-users] Output question for ex29. In-Reply-To: Message-ID: <90e6ba6e8366edf58804a561160d@google.com> Dear Matt, Following your suggestions, I modified the VecView_VTK in /src/ksp/ksp/example/tutorial/ex29.c to PetscFunctionBegin; PetscViewerASCIIOpen(comm, filename, &viewer); PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_MATLAB); DMView(x, viewer); VecView(x, PETSC_VIEWER_STDOUT_WORLD); PetscViewerDestroy(&viewer); PetscFunctionReturn(0); However, I always give me some error. Could you please briefly check if there is any obvious coding problem here. thanks, Alan On , Matthew Knepley wrote: > On Thu, Jun 9, 2011 at 6:18 PM, Matthew Knepley knepley at gmail.com> wrote: > On Thu, Jun 9, 2011 at 6:01 PM, zhenglun.wei at gmail.com> wrote: > Dear Sir/Madam, > I'm still studying on the ex29 of /src/ksp/ksp/example/tutorial. Before I > met a problem on VecView_VTK in parallel computation. I'm trying to > modify it in order to output some data from the computation. > Here is a better answer. If you want output, throw away this old function > which is broken, and use > PetscViewerASCIIOpen() > PetscViewerASCIISetFormat(PETSC_VIEWER_ASCII_VTK) > DMView() > VecView() > PetscViewerDestroy() > Thanks, > Matt > 1) My first questions is that what does this section do in VecView_VTK: > 272: MPI_Comm_rank(comm, &rank); > 273: MPI_Comm_size(comm, &size); > 274: MPI_Reduce(&n, &maxn, 1, MPIU_INT, MPI_MAX, 0, comm); > 275: tag = ((PetscObject) viewer)->tag; > 276: if (!rank) { > 277: PetscMalloc((maxn+1) * sizeof(PetscScalar), &values); > 278: for(i = 0; i 279: PetscViewerASCIIPrintf(viewer, "%G\n", > PetscRealPart(array[i])); > 280: } > 281: for(p = 1; p 282: MPI_Recv(values, (PetscMPIInt) n, MPIU_SCALAR, p, > tag, comm, &status); > 283: MPI_Get_count(&status, MPIU_SCALAR, &nn); > 284: for(i = 0; i > 285: PetscViewerASCIIPrintf(viewer, "%G\n", PetscRealPart(array[i])); > 286: } > 287: } > 288: PetscFree(values); > 289: } else { > 290: MPI_Send(array, n, MPIU_SCALAR, 0, tag, comm); > 291: } > What I understand is: it gather all the data from different process in > parallel computation, and output it to the 'viewer'. I comment out > everything in VecView_VTK except this part, there is no error message > coming up in my parallel computation so far. > 2) however, I really don't know how did it split the domain for parallel > computation. For example, if I use 4 processes, is the domain split like: > The DMDA describes the domain splitting. > Matt > a) > | > 0 | 1 > | > ---------------|-------------- > | > 2 | 3 > | > b) > | > 0 | 2 > | > ---------------|-------------- > | > 1 | 3 > | > c) > | | | | > | | | | > 0 | | 2 | | > | 1 | | 3 | > | | | | > | | | | > d) > 0 > ------------------------ > 1 > ------------------------ > 2 > ------------------------ > 3 > thanks in advance, > Alan > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 10 15:09:09 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 10 Jun 2011 15:09:09 -0500 Subject: [petsc-users] Output question for ex29. In-Reply-To: <90e6ba6e8366edf58804a561160d@google.com> References: <90e6ba6e8366edf58804a561160d@google.com> Message-ID: On Fri, Jun 10, 2011 at 3:05 PM, wrote: > Dear Matt, > Following your suggestions, I modified the VecView_VTK in > /src/ksp/ksp/example/tutorial/ex29.c to > > PetscFunctionBegin; > PetscViewerASCIIOpen(comm, filename, &viewer); > PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_MATLAB); > I told you to use ASCII_VTK, not ASCII_MATLAB. > DMView(x, viewer); > VecView(x, PETSC_VIEWER_STDOUT_WORLD); > PetscViewerDestroy(&viewer); > PetscFunctionReturn(0); > > However, I always give me some error. Could you please briefly check if > there is any obvious coding problem here. > How can I possibly know what went wrong if you do not send the ENTIRE error message. Matt > thanks, > Alan > > > On , Matthew Knepley wrote: > > On Thu, Jun 9, 2011 at 6:18 PM, Matthew Knepley knepley at gmail.com> > wrote: > > > > On Thu, Jun 9, 2011 at 6:01 PM, zhenglun.wei at gmail.com> wrote: > > > > > > Dear Sir/Madam, > > I'm still studying on the ex29 of /src/ksp/ksp/example/tutorial. Before I > met a problem on VecView_VTK in parallel computation. I'm trying to modify > it in order to output some data from the computation. > > > > > > > > > > Here is a better answer. If you want output, throw away this old function > which is broken, and use > > > > > > PetscViewerASCIIOpen() > > PetscViewerASCIISetFormat(PETSC_VIEWER_ASCII_VTK) > > > > DMView() > > VecView() > > PetscViewerDestroy() > > > > > > Thanks, > > > > > > Matt > > > > > > 1) My first questions is that what does this section do in VecView_VTK: > > > > 272: MPI_Comm_rank(comm, &rank); > > > > 273: MPI_Comm_size(comm, &size); > > 274: MPI_Reduce(&n, &maxn, 1, MPIU_INT, MPI_MAX, 0, comm); > > > > 275: tag = ((PetscObject) viewer)->tag; > > 276: if (!rank) { > > 277: PetscMalloc((maxn+1) * sizeof(PetscScalar), &values); > > 278: for(i = 0; i 279: PetscViewerASCIIPrintf(viewer, "%G\n", > PetscRealPart(array[i])); > > > > > > 280: } > > 281: for(p = 1; p 282: MPI_Recv(values, (PetscMPIInt) n, MPIU_SCALAR, p, > tag, comm, &status); > > > 283: MPI_Get_count(&status, MPIU_SCALAR, &nn); > > 284: for(i = 0; i > > > > 285: PetscViewerASCIIPrintf(viewer, "%G\n", PetscRealPart(array[i])); > > 286: } > > 287: } > > 288: PetscFree(values); > > 289: } else { > > 290: MPI_Send(array, n, MPIU_SCALAR, 0, tag, comm); > > > > > > 291: } > > > > What I understand is: it gather all the data from different process in > parallel computation, and output it to the 'viewer'. I comment out > everything in VecView_VTK except this part, there is no error message coming > up in my parallel computation so far. > > > > > > > > 2) however, I really don't know how did it split the domain for parallel > computation. For example, if I use 4 processes, is the domain split like: > > > > > > The DMDA describes the domain splitting. > > > > > > > > > > Matt > > > > a) > > | > > 0 | 1 > > | > > > > > > ---------------|-------------- > > | > > 2 | 3 > > | > > > > b) > > | > > 0 | 2 > > | > > ---------------|-------------- > > | > > > > > > 1 | 3 > > | > > > > c) > > | | | | > > | | | | > > 0 | | 2 | | > > | 1 | | 3 | > > | | | | > > | | | | > > > > > > > > d) > > 0 > > ------------------------ > > 1 > > ------------------------ > > 2 > > ------------------------ > > 3 > > > > thanks in advance, > > Alan > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenglun.wei at gmail.com Fri Jun 10 15:16:22 2011 From: zhenglun.wei at gmail.com (Alan Wei) Date: Fri, 10 Jun 2011 15:16:22 -0500 Subject: [petsc-users] Output question for ex29. In-Reply-To: References: <90e6ba6e8366edf58804a561160d@google.com> Message-ID: ooops, sorry, here it is. ^_^ Alan On Fri, Jun 10, 2011 at 3:09 PM, Matthew Knepley wrote: > On Fri, Jun 10, 2011 at 3:05 PM, wrote: > >> Dear Matt, >> Following your suggestions, I modified the VecView_VTK in >> /src/ksp/ksp/example/tutorial/ex29.c to >> >> PetscFunctionBegin; >> PetscViewerASCIIOpen(comm, filename, &viewer); >> PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_MATLAB); >> > > I told you to use ASCII_VTK, not ASCII_MATLAB. > > >> DMView(x, viewer); >> VecView(x, PETSC_VIEWER_STDOUT_WORLD); >> PetscViewerDestroy(&viewer); >> PetscFunctionReturn(0); >> >> However, I always give me some error. Could you please briefly check if >> there is any obvious coding problem here. >> > > How can I possibly know what went wrong if you do not send the ENTIRE error > message. > > Matt > > >> thanks, >> Alan >> >> >> On , Matthew Knepley wrote: >> > On Thu, Jun 9, 2011 at 6:18 PM, Matthew Knepley knepley at gmail.com> >> wrote: >> > >> > On Thu, Jun 9, 2011 at 6:01 PM, zhenglun.wei at gmail.com> wrote: >> > >> > >> > Dear Sir/Madam, >> > I'm still studying on the ex29 of /src/ksp/ksp/example/tutorial. Before >> I met a problem on VecView_VTK in parallel computation. I'm trying to modify >> it in order to output some data from the computation. >> > >> > >> > >> > >> > Here is a better answer. If you want output, throw away this old >> function which is broken, and use >> > >> > >> > PetscViewerASCIIOpen() >> > PetscViewerASCIISetFormat(PETSC_VIEWER_ASCII_VTK) >> > >> > DMView() >> > VecView() >> > PetscViewerDestroy() >> > >> > >> > Thanks, >> > >> > >> > Matt >> > >> > >> > 1) My first questions is that what does this section do in VecView_VTK: >> > >> > 272: MPI_Comm_rank(comm, &rank); >> > >> > 273: MPI_Comm_size(comm, &size); >> > 274: MPI_Reduce(&n, &maxn, 1, MPIU_INT, MPI_MAX, 0, comm); >> > >> > 275: tag = ((PetscObject) viewer)->tag; >> > 276: if (!rank) { >> > 277: PetscMalloc((maxn+1) * sizeof(PetscScalar), &values); >> > 278: for(i = 0; i 279: PetscViewerASCIIPrintf(viewer, "%G\n", >> PetscRealPart(array[i])); >> > >> > >> > 280: } >> > 281: for(p = 1; p 282: MPI_Recv(values, (PetscMPIInt) n, MPIU_SCALAR, p, >> tag, comm, &status); >> >> > 283: MPI_Get_count(&status, MPIU_SCALAR, &nn); >> > 284: for(i = 0; i >> > >> > 285: PetscViewerASCIIPrintf(viewer, "%G\n", PetscRealPart(array[i])); >> > 286: } >> > 287: } >> > 288: PetscFree(values); >> > 289: } else { >> > 290: MPI_Send(array, n, MPIU_SCALAR, 0, tag, comm); >> > >> > >> > 291: } >> > >> > What I understand is: it gather all the data from different process in >> parallel computation, and output it to the 'viewer'. I comment out >> everything in VecView_VTK except this part, there is no error message coming >> up in my parallel computation so far. >> > >> > >> > >> > 2) however, I really don't know how did it split the domain for parallel >> computation. For example, if I use 4 processes, is the domain split like: >> > >> > >> > The DMDA describes the domain splitting. >> > >> > >> > >> > >> > Matt >> > >> > a) >> > | >> > 0 | 1 >> > | >> > >> > >> > ---------------|-------------- >> > | >> > 2 | 3 >> > | >> > >> > b) >> > | >> > 0 | 2 >> > | >> > ---------------|-------------- >> > | >> > >> > >> > 1 | 3 >> > | >> > >> > c) >> > | | | | >> > | | | | >> > 0 | | 2 | | >> > | 1 | | 3 | >> > | | | | >> > | | | | >> > >> > >> > >> > d) >> > 0 >> > ------------------------ >> > 1 >> > ------------------------ >> > 2 >> > ------------------------ >> > 3 >> > >> > thanks in advance, >> > Alan >> > >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > -- Norbert Wiener >> > >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > -- Norbert Wiener >> > >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: errormessage Type: application/octet-stream Size: 2349 bytes Desc: not available URL: From knepley at gmail.com Fri Jun 10 15:23:13 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 10 Jun 2011 15:23:13 -0500 Subject: [petsc-users] Output question for ex29. In-Reply-To: References: <90e6ba6e8366edf58804a561160d@google.com> Message-ID: On Fri, Jun 10, 2011 at 3:16 PM, Alan Wei wrote: > ooops, sorry, here it is. > You did not pass a valid communicator to PetscVIewerASCIIOpen(). Use PETSC_COMM_WORLD. Matt > ^_^ > > Alan > > > On Fri, Jun 10, 2011 at 3:09 PM, Matthew Knepley wrote: > >> On Fri, Jun 10, 2011 at 3:05 PM, wrote: >> >>> Dear Matt, >>> Following your suggestions, I modified the VecView_VTK in >>> /src/ksp/ksp/example/tutorial/ex29.c to >>> >>> PetscFunctionBegin; >>> PetscViewerASCIIOpen(comm, filename, &viewer); >>> PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_MATLAB); >>> >> >> I told you to use ASCII_VTK, not ASCII_MATLAB. >> >> >>> DMView(x, viewer); >>> VecView(x, PETSC_VIEWER_STDOUT_WORLD); >>> PetscViewerDestroy(&viewer); >>> PetscFunctionReturn(0); >>> >>> However, I always give me some error. Could you please briefly check if >>> there is any obvious coding problem here. >>> >> >> How can I possibly know what went wrong if you do not send the ENTIRE >> error message. >> >> Matt >> >> >>> thanks, >>> Alan >>> >>> >>> On , Matthew Knepley wrote: >>> > On Thu, Jun 9, 2011 at 6:18 PM, Matthew Knepley knepley at gmail.com> >>> wrote: >>> > >>> > On Thu, Jun 9, 2011 at 6:01 PM, zhenglun.wei at gmail.com> wrote: >>> > >>> > >>> > Dear Sir/Madam, >>> > I'm still studying on the ex29 of /src/ksp/ksp/example/tutorial. Before >>> I met a problem on VecView_VTK in parallel computation. I'm trying to modify >>> it in order to output some data from the computation. >>> > >>> > >>> > >>> > >>> > Here is a better answer. If you want output, throw away this old >>> function which is broken, and use >>> > >>> > >>> > PetscViewerASCIIOpen() >>> > PetscViewerASCIISetFormat(PETSC_VIEWER_ASCII_VTK) >>> > >>> > DMView() >>> > VecView() >>> > PetscViewerDestroy() >>> > >>> > >>> > Thanks, >>> > >>> > >>> > Matt >>> > >>> > >>> > 1) My first questions is that what does this section do in VecView_VTK: >>> > >>> > 272: MPI_Comm_rank(comm, &rank); >>> > >>> > 273: MPI_Comm_size(comm, &size); >>> > 274: MPI_Reduce(&n, &maxn, 1, MPIU_INT, MPI_MAX, 0, comm); >>> > >>> > 275: tag = ((PetscObject) viewer)->tag; >>> > 276: if (!rank) { >>> > 277: PetscMalloc((maxn+1) * sizeof(PetscScalar), &values); >>> > 278: for(i = 0; i 279: PetscViewerASCIIPrintf(viewer, "%G\n", >>> PetscRealPart(array[i])); >>> > >>> > >>> > 280: } >>> > 281: for(p = 1; p 282: MPI_Recv(values, (PetscMPIInt) n, MPIU_SCALAR, >>> p, tag, comm, &status); >>> >>> > 283: MPI_Get_count(&status, MPIU_SCALAR, &nn); >>> > 284: for(i = 0; i >>> > >>> > 285: PetscViewerASCIIPrintf(viewer, "%G\n", PetscRealPart(array[i])); >>> > 286: } >>> > 287: } >>> > 288: PetscFree(values); >>> > 289: } else { >>> > 290: MPI_Send(array, n, MPIU_SCALAR, 0, tag, comm); >>> > >>> > >>> > 291: } >>> > >>> > What I understand is: it gather all the data from different process in >>> parallel computation, and output it to the 'viewer'. I comment out >>> everything in VecView_VTK except this part, there is no error message coming >>> up in my parallel computation so far. >>> > >>> > >>> > >>> > 2) however, I really don't know how did it split the domain for >>> parallel computation. For example, if I use 4 processes, is the domain split >>> like: >>> > >>> > >>> > The DMDA describes the domain splitting. >>> > >>> > >>> > >>> > >>> > Matt >>> > >>> > a) >>> > | >>> > 0 | 1 >>> > | >>> > >>> > >>> > ---------------|-------------- >>> > | >>> > 2 | 3 >>> > | >>> > >>> > b) >>> > | >>> > 0 | 2 >>> > | >>> > ---------------|-------------- >>> > | >>> > >>> > >>> > 1 | 3 >>> > | >>> > >>> > c) >>> > | | | | >>> > | | | | >>> > 0 | | 2 | | >>> > | 1 | | 3 | >>> > | | | | >>> > | | | | >>> > >>> > >>> > >>> > d) >>> > 0 >>> > ------------------------ >>> > 1 >>> > ------------------------ >>> > 2 >>> > ------------------------ >>> > 3 >>> > >>> > thanks in advance, >>> > Alan >>> > >>> > >>> > >>> > >>> > -- >>> > What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> > -- Norbert Wiener >>> > >>> > >>> > >>> > >>> > -- >>> > What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> > -- Norbert Wiener >>> > >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Sun Jun 12 01:22:07 2011 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Sat, 11 Jun 2011 23:22:07 -0700 Subject: [petsc-users] long int Message-ID: Hi guys, How can I use long int instead of PetscInt in the functions? Does simply changing the definition of PetscInt solve the problem? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Sun Jun 12 01:48:31 2011 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Sat, 11 Jun 2011 23:48:31 -0700 Subject: [petsc-users] long int In-Reply-To: References: Message-ID: I guess I figured it out. On 64-bit OS long long and long are the same and I may as well just use --with-64-bit-ints flag. Thanks *** * On Sat, Jun 11, 2011 at 11:22 PM, Mohammad Mirzadeh wrote: > Hi guys, > > How can I use long int instead of PetscInt in the functions? Does simply > changing the definition of PetscInt solve the problem? > > Thanks > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dharmareddy84 at gmail.com Sun Jun 12 02:02:28 2011 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Sun, 12 Jun 2011 02:02:28 -0500 Subject: [petsc-users] Error using MatCreateSeqAIJ Message-ID: Hello, I am getting argument out of range error when i use MatCreateSeqAIJ, in my code. I am attaching a test case with this email, can someone help me fix this. Note the the error comes when the matrix size is grater then 49 ( this number was changing for different runs but for the last few test runs, it was 49). thanks Reddy -- ----------------------------------------------------- Dharmendar Reddy Palle Graduate Student Microelectronics Research center, University of Texas at Austin, 10100 Burnet Road, Bldg. 160 MER 2.608F, TX 78758-4445 e-mail: dharmareddy84 at gmail.com Phone: +1-512-350-9082 United States of America. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PetscError.tar Type: application/x-tar Size: 6656 bytes Desc: not available URL: From jed at 59A2.org Sun Jun 12 08:36:10 2011 From: jed at 59A2.org (Jed Brown) Date: Sun, 12 Jun 2011 15:36:10 +0200 Subject: [petsc-users] Error using MatCreateSeqAIJ In-Reply-To: References: Message-ID: On Sun, Jun 12, 2011 at 09:02, Dharmendar Reddy wrote: > I am getting argument out of range error when i use > MatCreateSeqAIJ, in my code. I am attaching a test case with this email, can > someone help me fix this. Note the the error comes when the matrix size is > grater then 49 ( this number was changing for different runs but for the > last few test runs, it was 49). call MatCreateSeqAIJ(PETSC_COMM_SELF,123,123,PETSC_DECIDE,PETSC_NULL,A,ierr) should be call MatCreateSeqAIJ(PETSC_COMM_SELF,123,123,PETSC_DECIDE,PETSC_NULL_INTEGER,A,ierr) See the Fortran section in the user's manual, especially "Passing Null Pointers". You should also call MatDestroy(). -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Jun 12 11:39:34 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 12 Jun 2011 11:39:34 -0500 Subject: [petsc-users] long int In-Reply-To: References: Message-ID: On Jun 12, 2011, at 1:48 AM, Mohammad Mirzadeh wrote: > I guess I figured it out. On 64-bit OS long long and long are the same and I may as well just use --with-64-bit-ints flag. You only need to configure PETSc with --with-64-bit-indices if you are dealing with problems with over 2 billion unknowns (in parallel) or if the sparse matrix will have more than 2 billion nonzeros on a single process. You do not need to use --with-64-bit-indices just because you are using 64 bit pointers, only for really large problems. Barry > > Thanks > > > On Sat, Jun 11, 2011 at 11:22 PM, Mohammad Mirzadeh wrote: > Hi guys, > > How can I use long int instead of PetscInt in the functions? Does simply changing the definition of PetscInt solve the problem? > > Thanks > From mirzadeh at gmail.com Sun Jun 12 13:33:40 2011 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Sun, 12 Jun 2011 11:33:40 -0700 Subject: [petsc-users] long int In-Reply-To: References: Message-ID: Thanks Barry. Problem is my indices are all long and if I use the with petsc the compiler will complain. Do you think casting them to int will solve the problem? Also how does 64 bit international affect petsc performance? Thanks On Jun 12, 2011 9:39 AM, "Barry Smith" wrote: > > On Jun 12, 2011, at 1:48 AM, Mohammad Mirzadeh wrote: > >> I guess I figured it out. On 64-bit OS long long and long are the same and I may as well just use --with-64-bit-ints flag. > > You only need to configure PETSc with --with-64-bit-indices if you are dealing with problems with over 2 billion unknowns (in parallel) or if the sparse matrix will have more than 2 billion nonzeros on a single process. You do not need to use --with-64-bit-indices just because you are using 64 bit pointers, only for really large problems. > > > Barry > >> >> Thanks >> >> >> On Sat, Jun 11, 2011 at 11:22 PM, Mohammad Mirzadeh wrote: >> Hi guys, >> >> How can I use long int instead of PetscInt in the functions? Does simply changing the definition of PetscInt solve the problem? >> >> Thanks >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Sun Jun 12 13:35:53 2011 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Sun, 12 Jun 2011 11:35:53 -0700 Subject: [petsc-users] long int In-Reply-To: References: Message-ID: Sorry for misspells . It's my phone's auto correct! On Jun 12, 2011 11:33 AM, "Mohammad Mirzadeh" wrote: > Thanks Barry. Problem is my indices are all long and if I use the with petsc > the compiler will complain. Do you think casting them to int will solve the > problem? Also how does 64 bit international affect petsc performance? > > Thanks > On Jun 12, 2011 9:39 AM, "Barry Smith" wrote: >> >> On Jun 12, 2011, at 1:48 AM, Mohammad Mirzadeh wrote: >> >>> I guess I figured it out. On 64-bit OS long long and long are the same > and I may as well just use --with-64-bit-ints flag. >> >> You only need to configure PETSc with --with-64-bit-indices if you are > dealing with problems with over 2 billion unknowns (in parallel) or if the > sparse matrix will have more than 2 billion nonzeros on a single process. > You do not need to use --with-64-bit-indices just because you are using 64 > bit pointers, only for really large problems. >> >> >> Barry >> >>> >>> Thanks >>> >>> >>> On Sat, Jun 11, 2011 at 11:22 PM, Mohammad Mirzadeh > wrote: >>> Hi guys, >>> >>> How can I use long int instead of PetscInt in the functions? Does simply > changing the definition of PetscInt solve the problem? >>> >>> Thanks >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Jun 12 13:38:32 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 12 Jun 2011 13:38:32 -0500 Subject: [petsc-users] long int In-Reply-To: References: Message-ID: On Sun, Jun 12, 2011 at 1:33 PM, Mohammad Mirzadeh wrote: > Thanks Barry. Problem is my indices are all long and if I use the with > petsc the compiler will complain. Do you think casting them to int will > solve the problem? Also how does 64 bit international affect petsc > performance? > The words here are too imprecise: short int: A 2 byte integer long int: A 4 byte integer int: An integer which can be short or long, depending on the compiler, but most often long int today long long int: An 8 byte integer The --with-64-bit-indices flag gives long long ints. If you have > 2B unknowns, use this, otherwise don't. Yes, it slows things down. On every 64-bit architecture I know, 'int' is still 'long int', not 'long long int'. The only thing that is 64-bit is the pointers. Matt > Thanks > On Jun 12, 2011 9:39 AM, "Barry Smith" wrote: > > > > On Jun 12, 2011, at 1:48 AM, Mohammad Mirzadeh wrote: > > > >> I guess I figured it out. On 64-bit OS long long and long are the same > and I may as well just use --with-64-bit-ints flag. > > > > You only need to configure PETSc with --with-64-bit-indices if you are > dealing with problems with over 2 billion unknowns (in parallel) or if the > sparse matrix will have more than 2 billion nonzeros on a single process. > You do not need to use --with-64-bit-indices just because you are using 64 > bit pointers, only for really large problems. > > > > > > Barry > > > >> > >> Thanks > >> > >> > >> On Sat, Jun 11, 2011 at 11:22 PM, Mohammad Mirzadeh > wrote: > >> Hi guys, > >> > >> How can I use long int instead of PetscInt in the functions? Does simply > changing the definition of PetscInt solve the problem? > >> > >> Thanks > >> > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Sun Jun 12 13:45:14 2011 From: jed at 59A2.org (Jed Brown) Date: Sun, 12 Jun 2011 20:45:14 +0200 Subject: [petsc-users] long int In-Reply-To: References: Message-ID: On Sun, Jun 12, 2011 at 20:38, Matthew Knepley wrote: > The words here are too imprecise: > > short int: A 2 byte integer > long int: A 4 byte integer > No, "long int" must be at least 4 bytes, but it may be larger. "long int" is 8 bytes on 64-bit Linux and OS X. It is 4 bytes on LLP64 such as 64-bit Windows. > int: An integer which can be short or long, depending on the > compiler, but most often long int today > Int must be at least 2 bytes. In practice, it is either 4 or 8 bytes. Int cannot be larger than long. It is 4 bytes on the most commonly used platforms. > long long int: An 8 byte integer > This is a minimum, but I don't know any architectures that make it larger. > > The --with-64-bit-indices flag gives long long ints. If you have > 2B > unknowns, use this, otherwise don't. Yes, it slows things down. > In particular, it doubles the amount of memory required for column indices in the matrix and for index sets. This is not a huge difference, but it is noticeable. > On every 64-bit architecture I know, 'int' is still 'long int', not 'long > long int'. The only thing that is 64-bit is the pointers. > Nonsense, sizeof(long) is 8 on YOUR COMPUTER, Matt. sizeof(int) is 4 on your computer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59a2.org Sun Jun 12 13:47:58 2011 From: jed at 59a2.org (Jed Brown) Date: Sun, 12 Jun 2011 20:47:58 +0200 Subject: [petsc-users] long int In-Reply-To: References: Message-ID: On Sun, Jun 12, 2011 at 20:35, Mohammad Mirzadeh wrote: > On Jun 12, 2011 11:33 AM, "Mohammad Mirzadeh" wrote: > > Thanks Barry. Problem is my indices are all long and if I use the with > petsc > > the compiler will complain. Do you think casting them to int will solve > the > > problem? > No, if your code is hard-coded to work with "long" and you are on a LP64 platform (e.g. 64-bit x86 Linux or OS-X), then you must use --with-64-bit-indices. Note that hard-coding long may make your code non-portable to an LLP64 platform like 64-bit Windows. In any case, you would *not* use --with-64-bit-indices there because long is the same as int on LLP64. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dharmareddy84 at gmail.com Sun Jun 12 15:46:08 2011 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Sun, 12 Jun 2011 15:46:08 -0500 Subject: [petsc-users] Error using MatCreateSeqAIJ In-Reply-To: References: Message-ID: Thank you. On Sun, Jun 12, 2011 at 8:36 AM, Jed Brown wrote: > On Sun, Jun 12, 2011 at 09:02, Dharmendar Reddy wrote: > >> I am getting argument out of range error when i use >> MatCreateSeqAIJ, in my code. I am attaching a test case with this email, can >> someone help me fix this. Note the the error comes when the matrix size is >> grater then 49 ( this number was changing for different runs but for the >> last few test runs, it was 49). > > > call > MatCreateSeqAIJ(PETSC_COMM_SELF,123,123,PETSC_DECIDE,PETSC_NULL,A,ierr) > > should be > > call > MatCreateSeqAIJ(PETSC_COMM_SELF,123,123,PETSC_DECIDE,PETSC_NULL_INTEGER,A,ierr) > > See the Fortran section in the user's manual, especially "Passing Null > Pointers". > > You should also call MatDestroy(). > -- ----------------------------------------------------- Dharmendar Reddy Palle Graduate Student Microelectronics Research center, University of Texas at Austin, 10100 Burnet Road, Bldg. 160 MER 2.608F, TX 78758-4445 e-mail: dharmareddy84 at gmail.com Phone: +1-512-350-9082 United States of America. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Sun Jun 12 17:58:26 2011 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Sun, 12 Jun 2011 15:58:26 -0700 Subject: [petsc-users] long int In-Reply-To: References: Message-ID: Thanks guys. This is helpful. Actually I am using a Linux 64-bit OS and my tests show that on my system: sizeof(int) = 4 sizeof(long int) = 8 sizeof(long long int) = 8 So as Jed was saying on my system long int is the same as long long int and I need the --with-64-bit-indices flag. But if it leads to too much cost I'm willing to change it back to int as currently I wont have 2B points on each processor Thanks again On Sun, Jun 12, 2011 at 11:47 AM, Jed Brown wrote: > On Sun, Jun 12, 2011 at 20:35, Mohammad Mirzadeh wrote: > >> On Jun 12, 2011 11:33 AM, "Mohammad Mirzadeh" wrote: >> > Thanks Barry. Problem is my indices are all long and if I use the with >> petsc >> > the compiler will complain. Do you think casting them to int will solve >> the >> > problem? >> > > No, if your code is hard-coded to work with "long" and you are on a LP64 > platform (e.g. 64-bit x86 Linux or OS-X), then you must use > --with-64-bit-indices. > > Note that hard-coding long may make your code non-portable to an LLP64 > platform like 64-bit Windows. In any case, you would *not* use > --with-64-bit-indices there because long is the same as int on LLP64. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Sun Jun 12 18:05:58 2011 From: jed at 59A2.org (Jed Brown) Date: Mon, 13 Jun 2011 01:05:58 +0200 Subject: [petsc-users] long int In-Reply-To: References: Message-ID: On Mon, Jun 13, 2011 at 00:58, Mohammad Mirzadeh wrote: > But if it leads to too much cost I'm willing to change it back to int as > currently I wont have 2B points on each processor Just use a typedef. You can use PetscInt directly, or create your own typedef (that has to match PetscInt for integer arrays that you pass to PETSc). Then you don't have to touch the code to switch between 32- and 64-bit integers. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Eric.Chamberland at giref.ulaval.ca Mon Jun 13 10:45:40 2011 From: Eric.Chamberland at giref.ulaval.ca (Eric Chamberland) Date: Mon, 13 Jun 2011 11:45:40 -0400 Subject: [petsc-users] Problem installing PETSC with openmpi 1.4.1 In-Reply-To: References: Message-ID: <4DF630A4.10106@giref.ulaval.ca> Hi, I am unable to install PETSC compiled with openmpi 1.4.1 (or 1.4.3). The configure/make phase seems to go well, but the "make install" return an error: ================================================================= *** using PETSC_DIR=/home/mefpp_ericc/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-opt *** *** Installing PETSc at /opt/petsc-3.1-p8_openmpi *** Traceback (most recent call last): File "./config/install.py", line 261, in Installer(sys.argv[1:]).run() File "./config/install.py", line 245, in run self.installBin() File "./config/install.py", line 175, in installBin self.copies.extend(self.copytree(self.archBinDir, self.installBinDir)) File "./config/install.py", line 131, in copytree raise shutil.Error, errors shutil.Error: [('/home/mefpp_ericc/petsc-3.1-p8/linux-gnu-c-opt/bin/ompi-checkpoint', '/opt/petsc-3.1-p8_openmpi/bin/ompi-checkpoint', "[Errno 2] No such file or directory: '/home/mefpp_ericc/petsc-3.1-p8/linux-gnu-c-opt/bin/ompi-checkpoint'"), ('/home/mefpp_ericc/petsc-3.1-p8/linux-gnu-c-opt/bin/ompi-restart', '/opt/petsc-3.1-p8_openmpi/bin/ompi-restart', "[Errno 2] No such file or directory: '/home/mefpp_ericc/petsc-3.1-p8/linux-gnu-c-opt/bin/ompi-restart'")] make: *** [install] Error 1 ================================================================= Here is what I did: ################ #1- downloaded and installed openmpi 1.4.1 (or 1.4.3) like this: ################ ./configure --prefix=/opt/openmpi-1.4.1 CC=gcc CFLAGS="-Wall -Wwrite-strings -Wno-strict-aliasing -O" CXX=g++ CXXFLAGS="-Wall -Wwrite-strings -Wno-strict-aliasing -O" F77=gfortran FFLAGS="-Wall -Wno-unused-variable -O" FC=gfortran FCFLAGS="-Wall -Wno-unused-variable -O" --enable-shared=no --enable-static=yes make all su make install and it seems to work well (I am able do use /opt/openmpi-1.4.1/bin/mpirun...) ################ #2- for PETSC 3.1-p8 ################ setenv PATH /opt/openmpi-1.4.1/bin/:$PATH setenv LD_LIBRARY_PATH /opt/openmpi-1.4.1/lib/:$LD_LIBRARY_PATH ./config/configure.py \ --prefix=/opt/petsc-3.1-p8_openmpi \ --with-mpi-dir=/opt/openmpi-1.4.1 \ --with-mpi-compilers=1 \ --with-debugging=no \ --download-f-blas-lapack=1 \ --download-mumps=yes \ --download-superlu=yes \ --download-superlu_dist=yes \ --with-shared=0 \ --download-scalapack=yes \ --download-blacs=yes \ --download-parmetis=yes make PETSC_DIR=/home/mefpp_ericc/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-opt all su setenv PATH /opt/openmpi-1.4.1/bin/:$PATH setenv LD_LIBRARY_PATH /opt/openmpi-1.4.1/lib/:$LD_LIBRARY_PATH make PETSC_DIR=/home/mefpp_ericc/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-opt install and I get the error I copied at the beginning of this message... What did I missed? Thank you! Eric From knepley at gmail.com Mon Jun 13 12:10:57 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 13 Jun 2011 17:10:57 +0000 Subject: [petsc-users] Problem installing PETSC with openmpi 1.4.1 In-Reply-To: <4DF630A4.10106@giref.ulaval.ca> References: <4DF630A4.10106@giref.ulaval.ca> Message-ID: On Mon, Jun 13, 2011 at 3:45 PM, Eric Chamberland < Eric.Chamberland at giref.ulaval.ca> wrote: > Hi, > > I am unable to install PETSC compiled with openmpi 1.4.1 (or 1.4.3). The > configure/make phase seems to go well, but the "make install" return an > error: > > ================================================================= > *** using PETSC_DIR=/home/mefpp_ericc/petsc-3.1-p8 > PETSC_ARCH=linux-gnu-c-opt *** > *** Installing PETSc at /opt/petsc-3.1-p8_openmpi *** > Traceback (most recent call last): > File "./config/install.py", line 261, in > Installer(sys.argv[1:]).run() > File "./config/install.py", line 245, in run > self.installBin() > File "./config/install.py", line 175, in installBin > self.copies.extend(self.copytree(self.archBinDir, self.installBinDir)) > File "./config/install.py", line 131, in copytree > raise shutil.Error, errors > shutil.Error: > [('/home/mefpp_ericc/petsc-3.1-p8/linux-gnu-c-opt/bin/ompi-checkpoint', > '/opt/petsc-3.1-p8_openmpi/bin/ompi-checkpoint', "[Errno 2] No such file or > directory: > '/home/mefpp_ericc/petsc-3.1-p8/linux-gnu-c-opt/bin/ompi-checkpoint'"), > ('/home/mefpp_ericc/petsc-3.1-p8/linux-gnu-c-opt/bin/ompi-restart', > '/opt/petsc-3.1-p8_openmpi/bin/ompi-restart', "[Errno 2] No such file or > directory: > '/home/mefpp_ericc/petsc-3.1-p8/linux-gnu-c-opt/bin/ompi-restart'")] > make: *** [install] Error 1 > 1) I do not yet understand this error. It is saying that the source file: /home/mefpp_ericc/petsc-3.1-p8/linux-gnu-c-opt/bin/ompi-checkpoint does not exist. This is very strange since os.listdir() just told us that said file exists. Does this file actually exist? With correct permissions? 2) How did OpenMPI get into your PETSc build? Didn't you install it yourself somewhere else? Did you rm -rf $PETSC_DIR/$PETSC_ARCH before you rebuilt? Thanks, Matt > ================================================================= > > Here is what I did: > > ################ > #1- downloaded and installed openmpi 1.4.1 (or 1.4.3) like this: > ################ > > ./configure --prefix=/opt/openmpi-1.4.1 CC=gcc CFLAGS="-Wall > -Wwrite-strings -Wno-strict-aliasing -O" CXX=g++ CXXFLAGS="-Wall > -Wwrite-strings -Wno-strict-aliasing -O" F77=gfortran FFLAGS="-Wall > -Wno-unused-variable -O" FC=gfortran FCFLAGS="-Wall -Wno-unused-variable -O" > --enable-shared=no --enable-static=yes > > make all > su > make install > > and it seems to work well (I am able do use > /opt/openmpi-1.4.1/bin/mpirun...) > > ################ > #2- for PETSC 3.1-p8 > ################ > setenv PATH /opt/openmpi-1.4.1/bin/:$PATH > setenv LD_LIBRARY_PATH /opt/openmpi-1.4.1/lib/:$LD_LIBRARY_PATH > > ./config/configure.py \ > --prefix=/opt/petsc-3.1-p8_openmpi \ > --with-mpi-dir=/opt/openmpi-1.4.1 \ > --with-mpi-compilers=1 \ > --with-debugging=no \ > --download-f-blas-lapack=1 \ > --download-mumps=yes \ > --download-superlu=yes \ > --download-superlu_dist=yes \ > --with-shared=0 \ > --download-scalapack=yes \ > --download-blacs=yes \ > --download-parmetis=yes > > make PETSC_DIR=/home/mefpp_ericc/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-opt > all > su > setenv PATH /opt/openmpi-1.4.1/bin/:$PATH > setenv LD_LIBRARY_PATH /opt/openmpi-1.4.1/lib/:$LD_LIBRARY_PATH > make PETSC_DIR=/home/mefpp_ericc/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-opt > install > > and I get the error I copied at the beginning of this message... > > What did I missed? > > Thank you! > > Eric > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Eric.Chamberland at giref.ulaval.ca Mon Jun 13 13:06:03 2011 From: Eric.Chamberland at giref.ulaval.ca (Eric Chamberland) Date: Mon, 13 Jun 2011 14:06:03 -0400 Subject: [petsc-users] Problem installing PETSC with openmpi 1.4.1 In-Reply-To: References: <4DF630A4.10106@giref.ulaval.ca> Message-ID: <4DF6518B.10504@giref.ulaval.ca> On 06/13/2011 01:10 PM, Matthew Knepley wrote: > 2) How did OpenMPI get into your PETSc build? Didn't you install it > yourself somewhere else? Did you > > rm -rf $PETSC_DIR/$PETSC_ARCH > > before you rebuilt? > Oups... No... but I tried a "make clean" and "make distclean"... Now, I removed the $PETSC_DIR/$PETSC_ARCH and it did the trick! Thanks a lot! > Thanks, > > Matt If I can suggest: I would propose to add the "distclean" target to the makefile so everything returns to its original state after someone did a configuration mistake/error (like me)... Thank you! Eric From carver at cs.ua.edu Mon Jun 13 13:34:20 2011 From: carver at cs.ua.edu (Carver, Jeffrey C.) Date: Mon, 13 Jun 2011 13:34:20 -0500 Subject: [petsc-users] Participation Requested: Survey about Software Practices in Computational Science Message-ID: <2FE9F619592E8A4382310F62E7F06311013BB1E61656@MAIL1.ua-net.ua.edu> Dr. Roscoe Bartlett, Sandia National Laboratory, Dr. Jeffrey Carver, University of Alabama, and Dr. Lorin Hochstein, University of Southern California, are conducting a survey of software development practices among computational scientists. This survey seeks to understand current software development practices and identify areas of need. The survey should take approximately 15 minutes to complete. This survey has been approved by The University of Alabama IRB board. We would appreciate your participation in our survey. If you have any questions, please do not hesitate to contact me. The survey is available at: https://spreadsheets.google.com/spreadsheet/viewform?hl=en_US&formkey=dEF6QXRHbGd4Wjktbk5LRUlRYWlnVVE6MQ Thank you, Jeffrey Carver Assistant Professor University of Alabama (v) 205-348-9829 (f) 205-348-0219 http://www.cs.ua.edu/~carver From dharmareddy84 at gmail.com Tue Jun 14 22:00:07 2011 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Tue, 14 Jun 2011 22:00:07 -0500 Subject: [petsc-users] [error] undefined reference to TSSetPostStep Message-ID: Hello, I get the following error when i try to compile the Fortran code attached with this email. Can you please tell me if i am missing any thing in the code. undefined reference to `tssetpoststep_' ( i also get the error for tssetprestep_) Thanks Reddy -- ----------------------------------------------------- Dharmendar Reddy Palle Graduate Student Microelectronics Research center, University of Texas at Austin, 10100 Burnet Road, Bldg. 160 MER 2.608F, TX 78758-4445 e-mail: dharmareddy84 at gmail.com Phone: +1-512-350-9082 United States of America. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: TsTest.F90 Type: application/octet-stream Size: 935 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Makefile Type: application/octet-stream Size: 90 bytes Desc: not available URL: From bsmith at mcs.anl.gov Tue Jun 14 22:35:58 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Jun 2011 22:35:58 -0500 Subject: [petsc-users] [error] undefined reference to TSSetPostStep In-Reply-To: References: Message-ID: The fortran stub for this function is missing. Someone will prepare one and send it to you shortly. Barry On Jun 14, 2011, at 10:00 PM, Dharmendar Reddy wrote: > Hello, > I get the following error when i try to compile the Fortran code attached with this email. Can you please tell me if i am missing any thing in the code. > undefined reference to `tssetpoststep_' ( i also get the error for tssetprestep_) > > Thanks > Reddy > > -- > ----------------------------------------------------- > Dharmendar Reddy Palle > Graduate Student > Microelectronics Research center, > University of Texas at Austin, > 10100 Burnet Road, Bldg. 160 > MER 2.608F, TX 78758-4445 > e-mail: dharmareddy84 at gmail.com > Phone: +1-512-350-9082 > United States of America. > > From dharmareddy84 at gmail.com Tue Jun 14 22:39:26 2011 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Tue, 14 Jun 2011 22:39:26 -0500 Subject: [petsc-users] [error] undefined reference to TSSetPostStep In-Reply-To: References: Message-ID: Thanks. I hope it will be for both TsSetPostStep and TsSetPreStep. On Tue, Jun 14, 2011 at 10:35 PM, Barry Smith wrote: > > The fortran stub for this function is missing. Someone will prepare one > and send it to you shortly. > > Barry > > On Jun 14, 2011, at 10:00 PM, Dharmendar Reddy wrote: > > > Hello, > > I get the following error when i try to compile the Fortran code > attached with this email. Can you please tell me if i am missing any thing > in the code. > > undefined reference to `tssetpoststep_' ( i also get the error for > tssetprestep_) > > > > Thanks > > Reddy > > > > -- > > ----------------------------------------------------- > > Dharmendar Reddy Palle > > Graduate Student > > Microelectronics Research center, > > University of Texas at Austin, > > 10100 Burnet Road, Bldg. 160 > > MER 2.608F, TX 78758-4445 > > e-mail: dharmareddy84 at gmail.com > > Phone: +1-512-350-9082 > > United States of America. > > > > > > -- ----------------------------------------------------- Dharmendar Reddy Palle Graduate Student Microelectronics Research center, University of Texas at Austin, 10100 Burnet Road, Bldg. 160 MER 2.608F, TX 78758-4445 e-mail: dharmareddy84 at gmail.com Phone: +1-512-350-9082 United States of America. -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Wed Jun 15 11:54:54 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Wed, 15 Jun 2011 18:54:54 +0200 Subject: [petsc-users] Efficient computation of A'*A Message-ID: <4DF8E3DE.7060308@gfz-potsdam.de> Hello! I've got simple question. What is the most efficient way to compute B = A'*A using petsc? Matrix A is sparse MPIAIJ. Thanks. Regards, Alexander From jed at 59A2.org Wed Jun 15 12:00:15 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 15 Jun 2011 19:00:15 +0200 Subject: [petsc-users] Efficient computation of A'*A In-Reply-To: <4DF8E3DE.7060308@gfz-potsdam.de> References: <4DF8E3DE.7060308@gfz-potsdam.de> Message-ID: On Wed, Jun 15, 2011 at 18:54, Alexander Grayver wrote: > I've got simple question. > What is the most efficient way to compute B = A'*A using petsc? > Matrix A is sparse MPIAIJ. > What are you going to do with B = A'*A? The most efficient thing is MatCreateTranspose() which does no computation. http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatCreateTranspose.html If you need B assembled, then create a diagonal matrix and use MatPtAP() http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatPtAP.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From haoxiang at yahoo.cn Wed Jun 15 12:05:23 2011 From: haoxiang at yahoo.cn (Xiang Hao) Date: Wed, 15 Jun 2011 11:05:23 -0600 Subject: [petsc-users] Examples on solving linear systems using GMRES and/or matrix-free method In-Reply-To: References: Message-ID: Hi, I modify the KSP ex1 to a matrix free version, everything works. Basically, I define a usermult function and pass it to MatShellSetOperation, as shown in http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-3.0.0/docs/manualpages/Mat/MatShellSetOperation.html PetscErrorCode usermult(Mat ,Vec ,Vec ); ierr = MatCreateShell (comm ,m,n,M,N,ctx,&A); ierr = MatShellSetOperation (A,MATOP_MULT,(void(*)(void))usermult); Now, I have a problem when I try to change the above to a c++ version(I want to use a class). Let's see I define the following class and it has a function usermult Class classA{ PetscErrorCode usermult(Mat ,Vec ,Vec ); .... }; PetscErrorCode classA::usermult(Mat ,Vec ,Vec ) { .... } If I want to use the matrix free method, in the main function, I tried several different ways to pass the function usermult to MatShellSetOperation, but they all fails. Does anyone know how to do this? int main(int argc, char ** argv) { classA ob; ... ierr = MatCreateShell (comm ,m,n,M,N,ctx,&A); ierr = MatShellSetOperation (A,MATOP_MULT,(void(*)(void))ob.usermult); ? } On Wed, Jun 8, 2011 at 5:38 AM, Matthew Knepley wrote: > On Tue, Jun 7, 2011 at 10:20 PM, Xiang Hao wrote: > >> Hi all, >> >> I need to solve a PDE ( basically a Poisson's equation with Neumann >> boundary condition ), and I solved it using a steepest descent method, which >> is very slow. Now I want to solve the PDE using GMRES. In addition, I do >> have the matrix A, but I have a function which computes Ax, so I also need >> to use the matrix free method. >> >> I am very new to PETSc, so I am looking for examples on solving linear >> systems using GMRES and/or matrix-free method. >> > > You can look at SNES ex5 for the Poisson equation (after setting lambda = > 0.0). SNES ex19 also shows how to use geometric Multigrid. > Its not hard to do this for Poisson. > > Thanks, > > Matt > > >> Thanks, >> Xiang >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Wed Jun 15 12:08:35 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 15 Jun 2011 19:08:35 +0200 Subject: [petsc-users] Examples on solving linear systems using GMRES and/or matrix-free method In-Reply-To: References: Message-ID: On Wed, Jun 15, 2011 at 19:05, Xiang Hao wrote: > Now, I have a problem when I try to change the above to a c++ version(I > want to use a class). Set the class pointer as the application context and use a simple wrapper: MatMult_Wrapped(Mat A,Vec X,Vec Y) { TheClass *cls; MatShellGetContext(A,&cls); return cls->matmult(A,X,Y); } -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Wed Jun 15 12:35:27 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Wed, 15 Jun 2011 19:35:27 +0200 Subject: [petsc-users] Efficient computation of A'*A In-Reply-To: References: <4DF8E3DE.7060308@gfz-potsdam.de> Message-ID: An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jun 15 12:46:06 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 15 Jun 2011 17:46:06 +0000 Subject: [petsc-users] Efficient computation of A'*A In-Reply-To: References: <4DF8E3DE.7060308@gfz-potsdam.de> Message-ID: On Wed, Jun 15, 2011 at 5:35 PM, Alexander Grayver wrote: > Thanks Jed, > > Actually, I'm going to use this product as a part of the following > expressions: > > v'*A'*A*v > A'*A*v > > I need it many times with different v, but constant A. > > MatPtAP with unit matrix seems to be ok, I guess. > No, you do not need that at all for the above. Use MatMult(A, v, w) VecDot(w, w, &prod) prod = v' A' A v MatMultTranspose(A, w, y) y = A' A v Matt Regards, > Alexander > > > On Wed, 15 Jun 2011 19:00:15 +0200 > Jed Brown wrote: > > On Wed, Jun 15, 2011 at 18:54, Alexander Grayver > >wrote: > > > >> I've got simple question. > >> What is the most efficient way to compute B = A'*A using > >>petsc? > >> Matrix A is sparse MPIAIJ. > >> > > > > What are you going to do with B = A'*A? > > > > The most efficient thing is MatCreateTranspose() which > >does no computation. > > > > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatCreateTranspose.html > > > > > > If you need B assembled, then create a diagonal matrix > >and use MatPtAP() > > > > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatPtAP.html > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Wed Jun 15 12:46:44 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 15 Jun 2011 19:46:44 +0200 Subject: [petsc-users] Efficient computation of A'*A In-Reply-To: References: <4DF8E3DE.7060308@gfz-potsdam.de> Message-ID: On Wed, Jun 15, 2011 at 19:35, Alexander Grayver wrote: > Actually, I'm going to use this product as a part of the following > expressions: > > v'*A'*A*v > A'*A*v > Two choices here w = A*v result1 = norm(w) result2 = A'*w preprocess: B = A'*A w = B*v result1 = dot(v,w) result2 = w Note that B will require a bit more storage it is less sparse. It is probably problem-dependent which of these is faster. -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Wed Jun 15 13:00:11 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Wed, 15 Jun 2011 20:00:11 +0200 Subject: [petsc-users] Efficient computation of A'*A In-Reply-To: References: <4DF8E3DE.7060308@gfz-potsdam.de> Message-ID: An HTML attachment was scrubbed... URL: From haoxiang at yahoo.cn Wed Jun 15 14:20:07 2011 From: haoxiang at yahoo.cn (Xiang Hao) Date: Wed, 15 Jun 2011 13:20:07 -0600 Subject: [petsc-users] Examples on solving linear systems using GMRES and/or matrix-free method In-Reply-To: References: Message-ID: Thank you, Jed. It works :) On Wed, Jun 15, 2011 at 11:08 AM, Jed Brown wrote: > On Wed, Jun 15, 2011 at 19:05, Xiang Hao wrote: > >> Now, I have a problem when I try to change the above to a c++ version(I >> want to use a class). > > > Set the class pointer as the application context and use a simple wrapper: > > MatMult_Wrapped(Mat A,Vec X,Vec Y) { > TheClass *cls; > MatShellGetContext(A,&cls); > return cls->matmult(A,X,Y); > } > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jun 15 17:44:01 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 15 Jun 2011 17:44:01 -0500 Subject: [petsc-users] [error] undefined reference to TSSetPostStep In-Reply-To: References: Message-ID: <45F68E1A-D02E-417B-BCBD-EFF95200824B@mcs.anl.gov> Please find attached a new copy of src/ts/interface/ftn-custom/ztsf.c Just drop it into that directory and run make in that directory. Any errors please send entire error messages to petsc-maint at mcs.anl.gov Barry [see attached file: ztsf.c] On Jun 14, 2011, at 10:39 PM, Dharmendar Reddy wrote: > Thanks. I hope it will be for both TsSetPostStep and TsSetPreStep. > > > > On Tue, Jun 14, 2011 at 10:35 PM, Barry Smith wrote: > > The fortran stub for this function is missing. Someone will prepare one and send it to you shortly. > > Barry > > On Jun 14, 2011, at 10:00 PM, Dharmendar Reddy wrote: > > > Hello, > > I get the following error when i try to compile the Fortran code attached with this email. Can you please tell me if i am missing any thing in the code. > > undefined reference to `tssetpoststep_' ( i also get the error for tssetprestep_) > > > > Thanks > > Reddy > > > > -- > > ----------------------------------------------------- > > Dharmendar Reddy Palle > > Graduate Student > > Microelectronics Research center, > > University of Texas at Austin, > > 10100 Burnet Road, Bldg. 160 > > MER 2.608F, TX 78758-4445 > > e-mail: dharmareddy84 at gmail.com > > Phone: +1-512-350-9082 > > United States of America. > > > > > > > > > -- > ----------------------------------------------------- > Dharmendar Reddy Palle > Graduate Student > Microelectronics Research center, > University of Texas at Austin, > 10100 Burnet Road, Bldg. 160 > MER 2.608F, TX 78758-4445 > e-mail: dharmareddy84 at gmail.com > Phone: +1-512-350-9082 > United States of America. > -------------- next part -------------- A non-text attachment was scrubbed... Name: ztsf.c Type: application/octet-stream Size: 9123 bytes Desc: not available URL: From rabartl at sandia.gov Wed Jun 15 21:02:46 2011 From: rabartl at sandia.gov (Bartlett, Roscoe A) Date: Thu, 16 Jun 2011 02:02:46 +0000 Subject: [petsc-users] Participation Requested: Survey about Software Practices in Computational Science Message-ID: <6CC53491CB5E4E4FA1423EFA578E4BC70ED2BD49@EXMB03.srn.sandia.gov> Hello, Myself (Dr. Roscoe Bartlett, Sandia National Laboratory), Dr. Jeffrey Carver, University of Alabama, and Dr. Lorin Hochstein, University of Southern California, are conducting a survey of software development practices among computational scientists. This survey seeks to understand current software development practices and identify areas of need. The survey should take approximately 15 minutes to complete. This survey has been approved by The University of Alabama IRB board. We would appreciate your participation in our survey. If you have any questions, please do not hesitate to contact us. The survey is available at: https://spreadsheets.google.com/spreadsheet/viewform?hl=en_US&formkey=dEF6QXRHbGd4Wjktbk5LRUlRYWlnVVE6MQ Thanks, -Ross P.S. This is an updated version of a survey that we sent out in August of 2010. We have refined the survey based on feedback and detailed analysis. Please participate if you can as we will not be merging in the results from the old survey for this survey and analysis. P.S. Sorry if you get spammed by multiple emails for this survey which will happen if you are on multiple mail lists we are sending this too. ----------------------------------------------------------------------- Dr. Roscoe A. Bartlett, PhD Sandia National Laboratories Department of Optimization and Uncertainty Estimation Trilinos Software Engineering Technologies and Integration Lead CASL Virtual Reactor Integration Software Engineering Lead Phone: (505) 844-5097 Website: www.cs.sandia.gov/~rabartl/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From khalid_eee at yahoo.com Thu Jun 16 04:14:36 2011 From: khalid_eee at yahoo.com (khalid ashraf) Date: Thu, 16 Jun 2011 02:14:36 -0700 (PDT) Subject: [petsc-users] profiling the total time Message-ID: <400501.33830.qm@web112602.mail.gq1.yahoo.com> Hi, I am trying to benchmark the performance of my code on 8 processors and am trying to find where most of the time is used. When I look at the breakdown of the stages time required, the total add up to ~7s however, the main stage time is ~350s. I am not being able to find out the stage which is taking so much extra time. Could you please suggest something ? Thanks. Time (sec): 3.508e+02 1.00000 3.508e+02 Objects: 3.310e+02 1.00000 3.310e+02 Flops: 1.279e+08 1.03856 1.267e+08 1.014e+09 Flops/sec: 3.646e+05 1.03856 3.612e+05 2.890e+06 Memory: 2.817e+07 1.06078 2.221e+08 MPI Messages: 3.150e+02 1.26506 2.985e+02 2.388e+03 MPI Message Lengths: 4.011e+06 1.83678 1.191e+04 2.843e+07 MPI Reductions: 6.970e+02 1.00000 VecMDot 51 1.0 6.0108e-02 2.4 5.65e+06 1.0 0.0e+00 0.0e+00 5.1e+01 0 4 0 0 7 0 4 0 0 11 752 VecNorm 67 1.0 1.1708e-02 1.1 7.41e+05 1.0 0.0e+00 0.0e+00 6.7e+01 0 1 0 0 10 0 1 0 0 14 507 VecScale 75 1.0 1.2923e-03 1.0 3.69e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2282 VecCopy 20 1.0 6.3189e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 66 1.0 5.0619e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 51 1.0 8.9883e-04 1.1 2.95e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2625 VecWAXPY 30 1.0 2.0204e-03 1.2 1.09e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 430 VecMAXPY 54 1.0 8.1123e-03 1.0 6.28e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 5 0 0 0 0 5 0 0 0 6192 VecAssemblyBegin 3 1.0 4.9893e-04 3.0 0.00e+00 0.0 0.0e+00 0.0e+00 9.0e+00 0 0 0 0 1 0 0 0 0 2 0 VecAssemblyEnd 3 1.0 2.0169e-05 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 9 1.0 1.7315e-04 1.0 1.84e+04 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 852 VecScatterBegin 118 1.0 7.5530e-03 1.5 0.00e+00 0.0 2.2e+03 6.0e+03 0.0e+00 0 0 93 47 0 0 0 93 47 0 0 VecScatterEnd 118 1.0 3.5505e-02 3.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 54 1.0 7.4800e-03 1.4 9.95e+05 1.0 0.0e+00 0.0e+00 5.4e+01 0 1 0 0 8 0 1 0 0 11 1065 MatMult 66 1.0 6.4065e-01 1.1 5.67e+07 1.1 9.7e+02 1.1e+04 0.0e+00 0 44 41 38 0 0 44 41 38 0 694 MatSolve 81 1.0 3.9077e-01 1.0 4.40e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 35 0 0 0 0 35 0 0 0 901 MatLUFactorSym 8 1.0 1.0749e-05 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 9 1.0 4.7825e-02 1.0 1.38e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 11 0 0 0 0 11 0 0 0 2288 MatILUFactorSym 1 1.0 1.5453e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 0 0 0 1 0 MatConvert 2 1.0 3.7113e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 1 0 0 0 0 2 0 MatScale 16 1.0 3.7416e-05 1.2 4.90e+03 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1047 MatAssemblyBegin 59 1.0 3.1884e+00484.1 0.00e+00 0.0 4.2e+01 3.6e+05 6.0e+00 0 0 2 53 1 0 0 2 53 1 0 MatAssemblyEnd 59 1.0 1.2491e+00 1.0 0.00e+00 0.0 7.6e+01 1.5e+03 2.1e+01 0 0 3 0 3 0 0 3 0 4 0 MatGetValues 792 1.0 1.7098e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 1.0 5.9657e-06 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 2.0580e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 1 0 0 0 0 1 0 MatZeroEntries 1 1.0 5.6988e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatTranspose 16 1.0 6.8201e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 3.2e+01 0 0 0 0 5 0 0 0 0 7 0 MatMatMult 32 1.0 1.3172e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 6.4e+01 0 0 0 0 9 0 0 0 0 14 0 MatMatSolve 8 1.0 1.3842e-03 1.1 3.60e+02 1.0 0.0e+00 0.0e+00 3.2e+01 0 0 0 0 5 0 0 0 0 7 2 KSPGMRESOrthog 51 1.0 6.8248e-02 2.1 1.13e+07 1.0 0.0e+00 0.0e+00 5.1e+01 0 9 0 0 7 0 9 0 0 11 1325 KSPSetup 4 1.0 2.2410e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 8 1.0 1.0945e+00 1.0 1.21e+08 1.0 8.8e+02 1.1e+04 1.9e+02 0 95 37 34 27 0 95 37 34 40 877 PCSetUp 3 1.0 6.8884e-02 1.0 1.38e+07 1.0 0.0e+00 0.0e+00 1.1e+01 0 11 0 0 2 0 11 0 0 2 1588 PCSetUpOnBlocks 3 1.0 6.7468e-02 1.0 1.38e+07 1.0 0.0e+00 0.0e+00 7.0e+00 0 11 0 0 1 0 11 0 0 1 1622 PCApply 57 1.0 4.0561e-01 1.0 4.40e+07 1.0 0.0e+00 0.0e+00 5.7e+01 0 35 0 0 8 0 35 0 0 12 868 ------------------------------------------------------------------------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sperif at gmail.com Thu Jun 16 04:40:27 2011 From: sperif at gmail.com (Pierre-Yves Aquilanti) Date: Thu, 16 Jun 2011 11:40:27 +0200 Subject: [petsc-users] cholesky factorization Memory regions overlap Message-ID: Hello, i'm using petsc to proceed to the cholesky factorization of a square matrix using last stable version of PETSc using real scalars. However i am faced to a memory problem than i am not able to solve depiste spending hours on it. What i try to do is to proceed to a cholesky factorization using the preconditionner as it was recommanded on this list instead of using directly the matrix cholesky factorization. For this i create a KSP context with no solver, just the preconditionner that is fixed to PCCHOLESKY. When using KSPSetup, the factorization should occur. However, the program crash saying that there's a memory overlap. I checked with valgrind memcheck but didn't found any obvious error on allocation or deallocation. Also i attached the error and concerned code below. If anyone had any idea on what's happening that would be of great help :) Thanks a lot. Regards. Pierre-Yves [3]PETSC ERROR: --------------------- Error Message ------------------------------------ [3]PETSC ERROR: Arguments are incompatible! [3]PETSC ERROR: Memory regions overlap: either use PetscMemmov() or make sure your copy regions and lengths are correct. Length (bytes) -4 first address 178909960 second address 178900956! [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [3]PETSC ERROR: See docs/changes/index.html for recent updates. [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [3]PETSC ERROR: See docs/index.html for manual pages. [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: ./hyperh on a linux-c-d named Tripel by perif Thu Jun 16 11:06:23 2011 [3]PETSC ERROR: Libraries linked from /home/perif/Utils/Libs/petsc-3.1-p8/linux-c-debug-real/lib [3]PETSC ERROR: Configure run at Tue May 24 14:49:00 2011 [3]PETSC ERROR: Configure options --download-c-blas-lapack=yes --with-scalar-type=complex --with-fortran=0 --with-debugging=yes --with-shared=0 --with-petsc-arch=linux-c-debug-real [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: User provided function() line 1755 in src/mat/utils//home/perif/Utils/Libs/petsc-3.1-p8/include/petscsys.h [3]PETSC ERROR: PetscFreeSpaceContiguous_Cholesky() line 164 in src/mat/utils/freespace.c [3]PETSC ERROR: MatCholeskyFactorSymbolic_SeqAIJ() line 2922 in src/mat/impls/aij/seq/aijfact.c [3]PETSC ERROR: MatCholeskyFactorSymbolic() line 2702 in src/mat/interface/matrix.c [3]PETSC ERROR: PCSetUp_Cholesky() line 132 in src/ksp/pc/impls/factor/cholesky/cholesky.c [3]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c [3]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c [3]PETSC ERROR: User provided function() line 162 in precond.c MatCreateSeqAIJ(PETSC_COMM_WORLD,(*nb_eigen_all)+1,(*nb_eigen_all)+1,0,PETSC_NULL,&MM); MatSetFromOptions(MM); for(j=0;j<=*nb_eigen_all;j++) for(i=0;i From jed at 59A2.org Thu Jun 16 04:43:02 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 16 Jun 2011 11:43:02 +0200 Subject: [petsc-users] profiling the total time In-Reply-To: <400501.33830.qm@web112602.mail.gq1.yahoo.com> References: <400501.33830.qm@web112602.mail.gq1.yahoo.com> Message-ID: On Thu, Jun 16, 2011 at 11:14, khalid ashraf wrote: > When I look at the breakdown of the stages time required, the total add up > to ~7s however, the main stage time is ~350s. Two possibilities: 1. The time is not in PETSc. 2. The matrix is not preallocated correctly. http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#efficient-assembly -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Thu Jun 16 04:50:34 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 16 Jun 2011 11:50:34 +0200 Subject: [petsc-users] cholesky factorization Memory regions overlap In-Reply-To: References: Message-ID: On Thu, Jun 16, 2011 at 11:40, Pierre-Yves Aquilanti wrote: > > MatCreateSeqAIJ(PETSC_COMM_WORLD,(*nb_eigen_all)+1,(*nb_eigen_all)+1,0,PETSC_NULL,&MM); > MatSetFromOptions(MM); > > > for(j=0;j<=*nb_eigen_all;j++) > for(i=0;i MatSetValue(MM,i,j,(PetscReal)mm_tmp[i][j],INSERT_VALUES); > MatSetValue(MM,j,i,(PetscReal)mm_tmp[i][j],INSERT_VALUES); > Do you intend to have 0 on the diagonal? The rest of your code looks okay to me, but if you problem really is dense, then you may as well use a dense matrix format, see MatCreateSeqDense(). Could you send a test case so that we can reproduce? -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Thu Jun 16 05:59:24 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Thu, 16 Jun 2011 12:59:24 +0200 Subject: [petsc-users] Getting access to matrix rows without and setting values simultaneously Message-ID: <4DF9E20C.5060704@gfz-potsdam.de> Hello! Sorry for weird subject. The problem is very simple actually. I have big distributed dense matrix and I want to change the element of this matrix without copying whole matrix. If I do it this way: call MatGetOwnershipRange(A,istart,iend,ierr);CHKERRQ(ierr) do i=istart,iend-1 ! get one row from A rowidx(1) = i call MatGetRow(A,i,nvals,cols,vals,ierr);CHKERRQ(ierr); ! transform all elements in the row do j=1,nvals transformedvals(j) = func(vals(j)) enddo ! store transformed elements call MatSetValues(A,1,rowidx,nvals,cols,transformedvals,INSERT_VALUES,ierr);CHKERRQ(ierr); call MatRestoreRow(A,i,nvals,cols,vals,ierr);CHKERRQ(ierr); enddo It tells me that I can't use MatGetRow/MatRestoreRow and MatSetValues at the same time. So, what would be the better way of doing this? Maybe MatGetArrayF90? I would like to try it but don't find the description of output element arrays. Regards, Alexander From jed at 59A2.org Thu Jun 16 06:10:22 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 16 Jun 2011 13:10:22 +0200 Subject: [petsc-users] Getting access to matrix rows without and setting values simultaneously In-Reply-To: <4DF9E20C.5060704@gfz-potsdam.de> References: <4DF9E20C.5060704@gfz-potsdam.de> Message-ID: On Thu, Jun 16, 2011 at 12:59, Alexander Grayver wrote: > ! store transformed elements > call > MatSetValues(A,1,rowidx,nvals,cols,transformedvals,INSERT_VALUES,ierr);CHKERRQ(ierr); > call MatRestoreRow(A,i,nvals,cols,vals,ierr);CHKERRQ(ierr); > You can't mix these because MatGetRow() needs an assembled matrix, and MatSetValues() makes the matrix be unassembled. What do you want the parallel semantics to be during this transformation? The MatGetArray() for MPIDense just gives you access to the "diagonal" block. What is the high-level description of the operation you're trying to do? Is this "matrix" actually being used as a linear operator, or is it a grid of values? -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Thu Jun 16 07:08:26 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Thu, 16 Jun 2011 14:08:26 +0200 Subject: [petsc-users] Getting access to matrix rows without and setting values simultaneously In-Reply-To: References: <4DF9E20C.5060704@gfz-potsdam.de> Message-ID: <4DF9F23A.5020704@gfz-potsdam.de> Hi Jed, This matrix is rather grid o values. The operation is very simply, it's scalar. Let's say we have this matrix A of size MxN and I want to apply thw following operation to each element of the matrix: A(i,j) = log( (A(i,j) - a(j)) / (b(j) - A(i,j)) ), i=0..M-1 j=0..N-1 the vectors a,b are just arrays size of N and have them on each processor. Regards, Alexander On 16.06.2011 13:10, Jed Brown wrote: > On Thu, Jun 16, 2011 at 12:59, Alexander Grayver > > wrote: > > ! store transformed elements > call > MatSetValues(A,1,rowidx,nvals,cols,transformedvals,INSERT_VALUES,ierr);CHKERRQ(ierr); > call MatRestoreRow(A,i,nvals,cols,vals,ierr);CHKERRQ(ierr); > > > You can't mix these because MatGetRow() needs an assembled matrix, and > MatSetValues() makes the matrix be unassembled. > > What do you want the parallel semantics to be during this transformation? > > The MatGetArray() for MPIDense just gives you access to the "diagonal" > block. > > What is the high-level description of the operation you're trying to > do? Is this "matrix" actually being used as a linear operator, or is > it a grid of values? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Thu Jun 16 07:30:14 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 16 Jun 2011 14:30:14 +0200 Subject: [petsc-users] Getting access to matrix rows without and setting values simultaneously In-Reply-To: <4DF9F23A.5020704@gfz-potsdam.de> References: <4DF9E20C.5060704@gfz-potsdam.de> <4DF9F23A.5020704@gfz-potsdam.de> Message-ID: On Thu, Jun 16, 2011 at 14:08, Alexander Grayver wrote: > This matrix is rather grid o values. > Okay, then you can manage it with a DA (DMDA in petsc-dev). This will give a better parallel distribution and give you access to entries in convenient ways (e.g. using arrays). See the section of the user's manual Structured Grids Using Distributed Arrays. In PETSc, the Mat type is for linear operators. A Mat might represent a transformation from one of your grids to another. A given state on the grid is given by a Vec, and DMDA will help you manage that. > The operation is very simply, it's scalar. Let's say we have this matrix A > of size MxN and I want to apply thw following operation to each element of > the matrix: > > A(i,j) = log( (A(i,j) - a(j)) / (b(j) - A(i,j)) ), > i=0..M-1 j=0..N-1 > > the vectors a,b are just arrays size of N and have them on each processor. > This will be very easy using DMDA. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ozergedem at gmail.com Thu Jun 16 07:48:38 2011 From: ozergedem at gmail.com (=?ISO-8859-1?Q?=F6zge_erdem?=) Date: Thu, 16 Jun 2011 14:48:38 +0200 Subject: [petsc-users] PETSC_VIEWER_MATLAB Message-ID: Hi, I am using a library based on PETSc. This library creates all the matrices and vectors based PETSc. Now, I am computing some vectors. I need to save and visualize them. I checked the documentation about viewers, and found that it is also possible to save them as a matlab file. The explanation on the webpage about how to use this function seemed a bit confusing for me since I am not very much experienced with C. Can you give me more information how to use that viewer(PETSC_VIEWER_MATLAB) Best -- Ozge Erdem -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Thu Jun 16 07:48:48 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Thu, 16 Jun 2011 14:48:48 +0200 Subject: [petsc-users] Getting access to matrix rows without and setting values simultaneously In-Reply-To: References: <4DF9E20C.5060704@gfz-potsdam.de> <4DF9F23A.5020704@gfz-potsdam.de> Message-ID: <4DF9FBB0.1060104@gfz-potsdam.de> Jed, I'm afraid we are talking about different "grids", by the grid I meant that this matrix is 2d array, not a real grid of some physical parameters or whatever and it's also not a linear operator itself. Sorry that I possibly made you confused. It's very big and that is why I chose PETSc. I compute this matrix as a product of three other petsc matrices, one of which is also dense. Now I need this transformation to be applied and then I'm planning to use this matrix to create my operator, more precisely I need to use A'*A as the linear operator and Matrix-Free notation since A'*A is dense and even larger than original A. So I guess I need A as a matrix, since (A'*A)*v operations will be required further. I completely understand that petsc Mat has certain field of applications and my current problem is out of it, but still I wonder if there is a way to apply a scalar operation to each element of the matrix? Could you maybe give me an idea how to implement it? Any workaround? Regards, Alexander On 16.06.2011 14:30, Jed Brown wrote: > On Thu, Jun 16, 2011 at 14:08, Alexander Grayver > > wrote: > > This matrix is rather grid o values. > > > Okay, then you can manage it with a DA (DMDA in petsc-dev). This will > give a better parallel distribution and give you access to entries in > convenient ways (e.g. using arrays). See the section of the user's > manual Structured Grids Using Distributed Arrays. > > In PETSc, the Mat type is for linear operators. A Mat might represent > a transformation from one of your grids to another. A given state on > the grid is given by a Vec, and DMDA will help you manage that. > > The operation is very simply, it's scalar. Let's say we have this > matrix A of size MxN and I want to apply thw following operation > to each element of the matrix: > > A(i,j) = log( (A(i,j) - a(j)) / (b(j) - A(i,j)) ), > i=0..M-1 j=0..N-1 > > the vectors a,b are just arrays size of N and have them on each > processor. > > > This will be very easy using DMDA. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Thu Jun 16 07:55:18 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 16 Jun 2011 14:55:18 +0200 Subject: [petsc-users] Getting access to matrix rows without and setting values simultaneously In-Reply-To: <4DF9FBB0.1060104@gfz-potsdam.de> References: <4DF9E20C.5060704@gfz-potsdam.de> <4DF9F23A.5020704@gfz-potsdam.de> <4DF9FBB0.1060104@gfz-potsdam.de> Message-ID: On Thu, Jun 16, 2011 at 14:48, Alexander Grayver wrote: > by the grid I meant that this matrix is 2d array, not a real grid of some > physical parameters or whatever and it's also not a linear operator itself. Could you explain more about the task you're trying to do. Pseudocode or Matlab for the whole process would be useful. There might be a better way to do this without using Mat to store something that is not a linear operator. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Thu Jun 16 08:02:04 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 16 Jun 2011 15:02:04 +0200 Subject: [petsc-users] PETSC_VIEWER_MATLAB In-Reply-To: References: Message-ID: On Thu, Jun 16, 2011 at 14:48, ?zge erdem wrote: > Can you give me more information how to use that > viewer(PETSC_VIEWER_MATLAB) This does not save it as a Matlab file, it talks to a running Matlab session. To write a Matlab file, you can use http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/DM/PetscViewerBinaryMatlabOpen.html But, especially for large files, it is much better to write a binary file in PETSc format and load it from Matlab using $PETSC_DIR/bin/matlab/PetscBinaryRead.m -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Thu Jun 16 08:43:46 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Thu, 16 Jun 2011 15:43:46 +0200 Subject: [petsc-users] Getting access to matrix rows without and setting values simultaneously In-Reply-To: References: <4DF9E20C.5060704@gfz-potsdam.de> <4DF9F23A.5020704@gfz-potsdam.de> <4DF9FBB0.1060104@gfz-potsdam.de> Message-ID: <4DFA0892.6030301@gfz-potsdam.de> >> Could you explain more about the task you're trying to do. Well, I can try. That is pretty specific task, I wouldn't like to go deep into details. Let's say we have a discretized model of some physical parameter m (say acoustic velocity). Number of model parameters is N. We need to take M measurements d within model (say time traveling of acoustic wave) based on M different sets of receiver/source positions. In our case N >> M (e.g. 10^7 >> 10^3). We have operator F (which might linear or not) that defines relationship between m and d: d=F(m) The operator F is just a system of equations actually. I have no problem now to solve it for any m. What I need now is to compute the variation of this operator: A = \frac{\partial d_i}{\partial m_j}, for all i=1..M, j=1..N After some maths this calculation could be reformulated as a triple product: A_i = C*F(m)^-1*v Where C some sparse matrix. Once the A is computed I want to solve another problem: (A'*A)b=A'*r Which is a Gauss-Newton system now, A'*A is truncated Hessian, A'*r is gradient, r = d - d_obs, where d_obs is the real observed data. b -- model change which have to applied to original model m in order to explain your observed data better The latter problem I want to solve using petsc matrix-free formulation and some iterative solver (haven't decided yet which, could you advice one?). But the POINT here is that the latter problem must be solved not in the original m-space, but in transformed x-space. For that we need another A: A_tr = \frac{\partial d_i}{\partial x_j} However, using chain rule you can represent A_tr in terms of A: A = \frac{\partial d_i}{\partial m_j} * \frac{\partial m_j}{\partial x_j} \frac{\partial m_j}{\partial x_j} - is the exactly transformation I want to apply to matrix A. It's a simple scalar expression, but has to be applied to each element of A. Did it help? :) Regards, Alexander On 16.06.2011 14:55, Jed Brown wrote: > On Thu, Jun 16, 2011 at 14:48, Alexander Grayver > > wrote: > > by the grid I meant that this matrix is 2d array, not a real grid > of some physical parameters or whatever and it's also not a linear > operator itself. > > > Could you explain more about the task you're trying to do. Pseudocode > or Matlab for the whole process would be useful. There might be a > better way to do this without using Mat to store something that is not > a linear operator. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 16 09:07:48 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 16 Jun 2011 14:07:48 +0000 Subject: [petsc-users] Getting access to matrix rows without and setting values simultaneously In-Reply-To: <4DFA0892.6030301@gfz-potsdam.de> References: <4DF9E20C.5060704@gfz-potsdam.de> <4DF9F23A.5020704@gfz-potsdam.de> <4DF9FBB0.1060104@gfz-potsdam.de> <4DFA0892.6030301@gfz-potsdam.de> Message-ID: On Thu, Jun 16, 2011 at 1:43 PM, Alexander Grayver wrote: > >> Could you explain more about the task you're trying to do. > > Well, I can try. That is pretty specific task, I wouldn't like to go deep > into details. > > Let's say we have a discretized model of some physical parameter m (say > acoustic velocity). Number of model parameters is N. We need to take M > measurements d within model (say time traveling of acoustic wave) based on M > different sets of receiver/source positions. > In our case N >> M (e.g. 10^7 >> 10^3). > We have operator F (which might linear or not) that defines relationship > between m and d: > d=F(m) > > The operator F is just a system of equations actually. I have no problem > now to solve it for any m. > > What I need now is to compute the variation of this operator: > > A = \frac{\partial d_i}{\partial m_j}, for all i=1..M, j=1..N > > After some maths this calculation could be reformulated as a triple > product: > > A_i = C*F(m)^-1*v > > Where C some sparse matrix. > > Once the A is computed I want to solve another problem: > > (A'*A)b=A'*r > > Which is a Gauss-Newton system now, > A'*A is truncated Hessian, > A'*r is gradient, > r = d - d_obs, where d_obs is the real observed data. > b -- model change which have to applied to original model m in order to > explain your observed data better > > The latter problem I want to solve using petsc matrix-free formulation and > some iterative solver (haven't decided yet which, could you advice one?). > > But the POINT here is that the latter problem must be solved not in the > original m-space, but in transformed x-space. For that we need another A: > > A_tr = \frac{\partial d_i}{\partial x_j} > > However, using chain rule you can represent A_tr in terms of A: > > A = \frac{\partial d_i}{\partial m_j} * \frac{\partial m_j}{\partial x_j} > > \frac{\partial m_j}{\partial x_j} - is the exactly transformation I want to > apply to matrix A. It's a simple scalar expression, but has to be applied to > each element of A. > > Did it help? :) > Do you ever actually use A? If not, why not just build the transformed operator directly? Matt > Regards, > Alexander > > > > On 16.06.2011 14:55, Jed Brown wrote: > > On Thu, Jun 16, 2011 at 14:48, Alexander Grayver wrote: > >> by the grid I meant that this matrix is 2d array, not a real grid of some >> physical parameters or whatever and it's also not a linear operator itself. > > > Could you explain more about the task you're trying to do. Pseudocode or > Matlab for the whole process would be useful. There might be a better way to > do this without using Mat to store something that is not a linear operator. > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Thu Jun 16 09:19:51 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Thu, 16 Jun 2011 16:19:51 +0200 Subject: [petsc-users] Getting access to matrix rows without and setting values simultaneously In-Reply-To: References: <4DF9E20C.5060704@gfz-potsdam.de> <4DF9F23A.5020704@gfz-potsdam.de> <4DF9FBB0.1060104@gfz-potsdam.de> <4DFA0892.6030301@gfz-potsdam.de> Message-ID: <4DFA1107.1020302@gfz-potsdam.de> On 16.06.2011 16:07, Matthew Knepley wrote: > On Thu, Jun 16, 2011 at 1:43 PM, Alexander Grayver > > wrote: > > >> Could you explain more about the task you're trying to do. > > Well, I can try. That is pretty specific task, I wouldn't like to > go deep into details. > > Let's say we have a discretized model of some physical parameter m > (say acoustic velocity). Number of model parameters is N. We need > to take M measurements d within model (say time traveling of > acoustic wave) based on M different sets of receiver/source positions. > In our case N >> M (e.g. 10^7 >> 10^3). > We have operator F (which might linear or not) that defines > relationship between m and d: > d=F(m) > > The operator F is just a system of equations actually. I have no > problem now to solve it for any m. > > What I need now is to compute the variation of this operator: > > A = \frac{\partial d_i}{\partial m_j}, for all i=1..M, j=1..N > > After some maths this calculation could be reformulated as a > triple product: > > A_i = C*F(m)^-1*v > > Where C some sparse matrix. > > Once the A is computed I want to solve another problem: > > (A'*A)b=A'*r > > Which is a Gauss-Newton system now, > A'*A is truncated Hessian, > A'*r is gradient, > r = d - d_obs, where d_obs is the real observed data. > b -- model change which have to applied to original model m in > order to explain your observed data better > > The latter problem I want to solve using petsc matrix-free > formulation and some iterative solver (haven't decided yet which, > could you advice one?). > > But the POINT here is that the latter problem must be solved not > in the original m-space, but in transformed x-space. For that we > need another A: > > A_tr = \frac{\partial d_i}{\partial x_j} > > However, using chain rule you can represent A_tr in terms of A: > > A = \frac{\partial d_i}{\partial m_j} * \frac{\partial > m_j}{\partial x_j} > > \frac{\partial m_j}{\partial x_j} - is the exactly transformation > I want to apply to matrix A. It's a simple scalar expression, but > has to be applied to each element of A. > > Did it help? :) > > > Do you ever actually use A? If not, why not just build the transformed > operator directly? Two reasons: 1. It's wrong from the architectural point view. At the point where I compute A I prefer know nothing about transformation, just working with original physical parameters is much more natural. 2. This original A might be used in the future. I really suggested that would be easier to apply this transformation to the whole matrix. If it's impossible than I have to find another way. Just want to be sure of that. > > Matt > > Regards, > Alexander > > > > On 16.06.2011 14:55, Jed Brown wrote: >> On Thu, Jun 16, 2011 at 14:48, Alexander Grayver >> > wrote: >> >> by the grid I meant that this matrix is 2d array, not a real >> grid of some physical parameters or whatever and it's also >> not a linear operator itself. >> >> >> Could you explain more about the task you're trying to do. >> Pseudocode or Matlab for the whole process would be useful. There >> might be a better way to do this without using Mat to store >> something that is not a linear operator. > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Thu Jun 16 09:38:58 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 16 Jun 2011 16:38:58 +0200 Subject: [petsc-users] Getting access to matrix rows without and setting values simultaneously In-Reply-To: <4DFA0892.6030301@gfz-potsdam.de> References: <4DF9E20C.5060704@gfz-potsdam.de> <4DF9F23A.5020704@gfz-potsdam.de> <4DF9FBB0.1060104@gfz-potsdam.de> <4DFA0892.6030301@gfz-potsdam.de> Message-ID: On Thu, Jun 16, 2011 at 15:43, Alexander Grayver wrote: > Let's say we have a discretized model of some physical parameter m (say > acoustic velocity). Number of model parameters is N. We need to take M > measurements d within model (say time traveling of acoustic wave) based on M > different sets of receiver/source positions. > In our case N >> M (e.g. 10^7 >> 10^3). > We have operator F (which might linear or not) that defines relationship > between m and d: > d=F(m) > > The operator F is just a system of equations actually. I have no problem > now to solve it for any m. > > What I need now is to compute the variation of this operator: > > A = \frac{\partial d_i}{\partial m_j}, for all i=1..M, j=1..N > > After some maths this calculation could be reformulated as a triple > product: > > A_i = C*F(m)^-1*v > > Where C some sparse matrix. > > Once the A is computed I want to solve another problem: > > (A'*A)b=A'*r > > Which is a Gauss-Newton system now, > A'*A is truncated Hessian, > A'*r is gradient, > r = d - d_obs, where d_obs is the real observed data. > b -- model change which have to applied to original model m in order to > explain your observed data better > Excellent, great description. It looks to me like you are assembling A because you want to be able to multiply by A' (which you can't do with finite differencing). But there is a very efficient way to apply A' that does not require assembling it: automatic differentiation. The adjoint costs about three times as much as a single forward solve. Depending on the nature of the nonlinearities in your problem, it may also be reasonably easy to write the adjoint without using AD. You shouldn't need to assemble any of these matrices to solve the optimization/inverse problem. Assembling the matrices seriously limits scalability and there are optimal algorithms that don't use them. Depending on the problem, you may need a preconditioner for the reduced Hessian, but that can also be done without assembling it. I imagine that F() is nonlinear. The adjoint of the nonlinear solve is the adjoint of the Jacobian at the converged state. For more specifics on algorithms and pointers to the literature, you could look at this paper and some of the more specific applications it cites. http://users.ices.utexas.edu/~omar/papers/siam-par-pde-opt.pdf -------------- next part -------------- An HTML attachment was scrubbed... URL: From dharmareddy84 at gmail.com Thu Jun 16 10:10:48 2011 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Thu, 16 Jun 2011 10:10:48 -0500 Subject: [petsc-users] [error] undefined reference to TSSetPostStep In-Reply-To: <45F68E1A-D02E-417B-BCBD-EFF95200824B@mcs.anl.gov> References: <45F68E1A-D02E-417B-BCBD-EFF95200824B@mcs.anl.gov> Message-ID: Thanks. It works now. I get a warning when i run make, is that okay? ztsf.c(172): warning #167: argument of type "PetscErrorCode={int} (*)(void **)" is incompatible with paramet er of type "PetscErrorCode={int} (*)(void *)" *ierr = TSMonitorSet(*ts,ourtsmonitor,*ts,ourmonitordestroy); On Wed, Jun 15, 2011 at 5:44 PM, Barry Smith wrote: > > Please find attached a new copy of src/ts/interface/ftn-custom/ztsf.c > Just drop it into that directory and run make in that directory. > > Any errors please send entire error messages to petsc-maint at mcs.anl.gov > > Barry > > [see attached file: ztsf.c] > On Jun 14, 2011, at 10:39 PM, Dharmendar Reddy wrote: > > > Thanks. I hope it will be for both TsSetPostStep and TsSetPreStep. > > > > > > > > On Tue, Jun 14, 2011 at 10:35 PM, Barry Smith > wrote: > > > > The fortran stub for this function is missing. Someone will prepare one > and send it to you shortly. > > > > Barry > > > > On Jun 14, 2011, at 10:00 PM, Dharmendar Reddy wrote: > > > > > Hello, > > > I get the following error when i try to compile the Fortran > code attached with this email. Can you please tell me if i am missing any > thing in the code. > > > undefined reference to `tssetpoststep_' ( i also get the error for > tssetprestep_) > > > > > > Thanks > > > Reddy > > > > > > -- > > > ----------------------------------------------------- > > > Dharmendar Reddy Palle > > > Graduate Student > > > Microelectronics Research center, > > > University of Texas at Austin, > > > 10100 Burnet Road, Bldg. 160 > > > MER 2.608F, TX 78758-4445 > > > e-mail: dharmareddy84 at gmail.com > > > Phone: +1-512-350-9082 > > > United States of America. > > > > > > > > > > > > > > > > -- > > ----------------------------------------------------- > > Dharmendar Reddy Palle > > Graduate Student > > Microelectronics Research center, > > University of Texas at Austin, > > 10100 Burnet Road, Bldg. 160 > > MER 2.608F, TX 78758-4445 > > e-mail: dharmareddy84 at gmail.com > > Phone: +1-512-350-9082 > > United States of America. > > > > -- ----------------------------------------------------- Dharmendar Reddy Palle Graduate Student Microelectronics Research center, University of Texas at Austin, 10100 Burnet Road, Bldg. 160 MER 2.608F, TX 78758-4445 e-mail: dharmareddy84 at gmail.com Phone: +1-512-350-9082 United States of America. -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Thu Jun 16 10:16:20 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Thu, 16 Jun 2011 17:16:20 +0200 Subject: [petsc-users] Getting access to matrix rows without and setting values simultaneously In-Reply-To: References: <4DF9E20C.5060704@gfz-potsdam.de> <4DF9F23A.5020704@gfz-potsdam.de> <4DF9FBB0.1060104@gfz-potsdam.de> <4DFA0892.6030301@gfz-potsdam.de> Message-ID: <4DFA1E44.7040507@gfz-potsdam.de> Jed, thanks a lot for paper, it seems to be very good! You're right, there are ways to avoid calculating the whole A, but there are actually advantages having A computed, which you don't have without A being computed (at least with approaches I know so far). I know several technique from paper you cited, maybe they could be better (I personally believe they are not), but anyway now I'm at the point when no serious changes are reasonable. That is right, F is nonlinear. The adjoint in my case costs the same as forward problem. I can afford A for my problems. The only thing which is really has to be avoided is the explicit assembling of A'*A in Gauss-Newton. That I want to do using matrix-free formulation and one of the iterative solvers from petsc. Would that be possible? If not, then I really should think about different approach, maybe one presented in paper you cited. Regards, Alexander On 16.06.2011 16:38, Jed Brown wrote: > On Thu, Jun 16, 2011 at 15:43, Alexander Grayver > > wrote: > > Let's say we have a discretized model of some physical parameter m > (say acoustic velocity). Number of model parameters is N. We need > to take M measurements d within model (say time traveling of > acoustic wave) based on M different sets of receiver/source positions. > In our case N >> M (e.g. 10^7 >> 10^3). > We have operator F (which might linear or not) that defines > relationship between m and d: > d=F(m) > > The operator F is just a system of equations actually. I have no > problem now to solve it for any m. > > What I need now is to compute the variation of this operator: > > A = \frac{\partial d_i}{\partial m_j}, for all i=1..M, j=1..N > > After some maths this calculation could be reformulated as a > triple product: > > A_i = C*F(m)^-1*v > > Where C some sparse matrix. > > Once the A is computed I want to solve another problem: > > (A'*A)b=A'*r > > Which is a Gauss-Newton system now, > A'*A is truncated Hessian, > A'*r is gradient, > r = d - d_obs, where d_obs is the real observed data. > b -- model change which have to applied to original model m in > order to explain your observed data better > > > Excellent, great description. > > It looks to me like you are assembling A because you want to be able > to multiply by A' (which you can't do with finite differencing). But > there is a very efficient way to apply A' that does not require > assembling it: automatic differentiation. The adjoint costs about > three times as much as a single forward solve. Depending on the nature > of the nonlinearities in your problem, it may also be reasonably easy > to write the adjoint without using AD. > > You shouldn't need to assemble any of these matrices to solve the > optimization/inverse problem. Assembling the matrices seriously limits > scalability and there are optimal algorithms that don't use them. > Depending on the problem, you may need a preconditioner for the > reduced Hessian, but that can also be done without assembling it. > > I imagine that F() is nonlinear. The adjoint of the nonlinear solve is > the adjoint of the Jacobian at the converged state. > > For more specifics on algorithms and pointers to the literature, you > could look at this paper and some of the more specific applications it > cites. > > http://users.ices.utexas.edu/~omar/papers/siam-par-pde-opt.pdf > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Thu Jun 16 10:24:55 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 16 Jun 2011 17:24:55 +0200 Subject: [petsc-users] [error] undefined reference to TSSetPostStep In-Reply-To: References: <45F68E1A-D02E-417B-BCBD-EFF95200824B@mcs.anl.gov> Message-ID: On Thu, Jun 16, 2011 at 17:10, Dharmendar Reddy wrote: > Thanks. It works now. > I get a warning when i run make, is that okay? > ztsf.c(172): warning #167: argument of type "PetscErrorCode={int} (*)(void > **)" is incompatible with paramet > er of type "PetscErrorCode={int} (*)(void *)" > *ierr = TSMonitorSet(*ts,ourtsmonitor,*ts,ourmonitordestroy); This warning is harmless. The new prototype is more permissive, so there is no warning in petsc-dev. If the warning bothers you, you can add an explicit cast. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 16 10:34:03 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 16 Jun 2011 15:34:03 +0000 Subject: [petsc-users] Getting access to matrix rows without and setting values simultaneously In-Reply-To: <4DFA1E44.7040507@gfz-potsdam.de> References: <4DF9E20C.5060704@gfz-potsdam.de> <4DF9F23A.5020704@gfz-potsdam.de> <4DF9FBB0.1060104@gfz-potsdam.de> <4DFA0892.6030301@gfz-potsdam.de> <4DFA1E44.7040507@gfz-potsdam.de> Message-ID: On Thu, Jun 16, 2011 at 3:16 PM, Alexander Grayver wrote: > Jed, thanks a lot for paper, it seems to be very good! > > You're right, there are ways to avoid calculating the whole A, but there > are actually advantages having A computed, which you don't have without A > being computed (at least with approaches I know so far). I know several > technique from paper you cited, maybe they could be better (I personally > believe they are not), but anyway now I'm at the point when no serious > changes are reasonable. > That is right, F is nonlinear. The adjoint in my case costs the same as > forward problem. I can afford A for my problems. The only thing which is > really has to be avoided is the explicit assembling of A'*A in Gauss-Newton. > That I want to do using matrix-free formulation and one of the iterative > solvers from petsc. Would that be possible? If not, then I really should > think about different approach, maybe one presented in paper you cited. > Yes, you can do MF applications of A' A. You could just make a MatShell that called MatMult and MatMultTranspose. For modifying the matrix, MatGetRow() IF you are using an AIJ matrix, return a pointer directly to the values. Thus you could alter them in place. This is not portable, and might break later, but at least you could run now. Thanks, Matt > Regards, > Alexander > > On 16.06.2011 16:38, Jed Brown wrote: > > On Thu, Jun 16, 2011 at 15:43, Alexander Grayver wrote: > >> Let's say we have a discretized model of some physical parameter m (say >> acoustic velocity). Number of model parameters is N. We need to take M >> measurements d within model (say time traveling of acoustic wave) based on M >> different sets of receiver/source positions. >> In our case N >> M (e.g. 10^7 >> 10^3). >> We have operator F (which might linear or not) that defines relationship >> between m and d: >> d=F(m) >> >> The operator F is just a system of equations actually. I have no problem >> now to solve it for any m. >> >> What I need now is to compute the variation of this operator: >> >> A = \frac{\partial d_i}{\partial m_j}, for all i=1..M, j=1..N >> >> After some maths this calculation could be reformulated as a triple >> product: >> >> A_i = C*F(m)^-1*v >> >> Where C some sparse matrix. >> >> Once the A is computed I want to solve another problem: >> >> (A'*A)b=A'*r >> >> Which is a Gauss-Newton system now, >> A'*A is truncated Hessian, >> A'*r is gradient, >> r = d - d_obs, where d_obs is the real observed data. >> b -- model change which have to applied to original model m in order to >> explain your observed data better >> > > Excellent, great description. > > It looks to me like you are assembling A because you want to be able to > multiply by A' (which you can't do with finite differencing). But there is a > very efficient way to apply A' that does not require assembling it: > automatic differentiation. The adjoint costs about three times as much as a > single forward solve. Depending on the nature of the nonlinearities in your > problem, it may also be reasonably easy to write the adjoint without using > AD. > > You shouldn't need to assemble any of these matrices to solve the > optimization/inverse problem. Assembling the matrices seriously limits > scalability and there are optimal algorithms that don't use them. Depending > on the problem, you may need a preconditioner for the reduced Hessian, but > that can also be done without assembling it. > > I imagine that F() is nonlinear. The adjoint of the nonlinear solve is > the adjoint of the Jacobian at the converged state. > > For more specifics on algorithms and pointers to the literature, you > could look at this paper and some of the more specific applications it > cites. > > http://users.ices.utexas.edu/~omar/papers/siam-par-pde-opt.pdf > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Thu Jun 16 10:40:07 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 16 Jun 2011 17:40:07 +0200 Subject: [petsc-users] Getting access to matrix rows without and setting values simultaneously In-Reply-To: References: <4DF9E20C.5060704@gfz-potsdam.de> <4DF9F23A.5020704@gfz-potsdam.de> <4DF9FBB0.1060104@gfz-potsdam.de> <4DFA0892.6030301@gfz-potsdam.de> <4DFA1E44.7040507@gfz-potsdam.de> Message-ID: On Thu, Jun 16, 2011 at 17:34, Matthew Knepley wrote: > Yes, you can do MF applications of A' A. You could just make a MatShell > that called MatMult and MatMultTranspose. > MatShell is not necessary, just use MatCreateTranspose(). http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Mat/MatCreateTranspose.html But I think Alexander is clear on that part. > For modifying the matrix, MatGetRow() IF you are using an AIJ matrix, > return a pointer directly to the values. > This is only for SeqAIJ. Alexander, could you clarify where in the Gauss-Newton procedure you outlined is the dense matrix on which you need to do the nonlinear row operation? Also, the matrix A'*A has a huge null space. What sort of regularization are you using. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 16 10:57:03 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 16 Jun 2011 15:57:03 +0000 Subject: [petsc-users] Getting access to matrix rows without and setting values simultaneously In-Reply-To: References: <4DF9E20C.5060704@gfz-potsdam.de> <4DF9F23A.5020704@gfz-potsdam.de> <4DF9FBB0.1060104@gfz-potsdam.de> <4DFA0892.6030301@gfz-potsdam.de> <4DFA1E44.7040507@gfz-potsdam.de> Message-ID: On Thu, Jun 16, 2011 at 3:40 PM, Jed Brown wrote: > On Thu, Jun 16, 2011 at 17:34, Matthew Knepley wrote: > >> Yes, you can do MF applications of A' A. You could just make a MatShell >> that called MatMult and MatMultTranspose. >> > > MatShell is not necessary, just use MatCreateTranspose(). > > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Mat/MatCreateTranspose.html > > But I think Alexander is clear on that part. > Doesn't he need A' A (normal equations) for the solve, so a MatShell? > > >> For modifying the matrix, MatGetRow() IF you are using an AIJ matrix, >> return a pointer directly to the values. >> > > This is only for SeqAIJ. > True. It would not be that hard to make a SetRow() which would alter values in place for MPIAIJ as well. > Alexander, could you clarify where in the Gauss-Newton procedure you > outlined is the dense matrix on which you need to do the nonlinear row > operation? > > Also, the matrix A'*A has a huge null space. What sort of regularization > are you using. > You could use a range space method I guess. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Thu Jun 16 11:03:16 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Thu, 16 Jun 2011 18:03:16 +0200 Subject: [petsc-users] Getting access to matrix rows without and setting values simultaneously In-Reply-To: References: <4DF9E20C.5060704@gfz-potsdam.de> <4DF9F23A.5020704@gfz-potsdam.de> <4DF9FBB0.1060104@gfz-potsdam.de> <4DFA0892.6030301@gfz-potsdam.de> <4DFA1E44.7040507@gfz-potsdam.de> Message-ID: <4DFA2944.1020004@gfz-potsdam.de> On 16.06.2011 17:40, Jed Brown wrote: > On Thu, Jun 16, 2011 at 17:34, Matthew Knepley > wrote: > > Yes, you can do MF applications of A' A. You could just make a > MatShell that called MatMult and MatMultTranspose. > > > MatShell is not necessary, just use MatCreateTranspose(). > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Mat/MatCreateTranspose.html That's not very clear to me. I understand what Matthew mean, but can I use MF without MatShell at all? So far I imagine it like this: 1. I create matrix using MatCreateShell 2. I set MATOP_MULT and MATOP_MULTTRANSPOSE using MatShellSetOperation 3. Inside my operations I represent the product (A'*A)v as w = Av -- using MatMult f = A'w -- using MatMultTranspose Where should I use MatCreateTranspose? > > But I think Alexander is clear on that part. > > For modifying the matrix, MatGetRow() IF you are using an AIJ > matrix, return a pointer directly to the values. > > > This is only for SeqAIJ. > > > Alexander, could you clarify where in the Gauss-Newton procedure you > outlined is the dense matrix on which you need to do the nonlinear row > operation? So if my system is: (A'*A)v=A'r then A everywhere is transformed. There is possibility to compute transformed A directly instead of applying it afterward as Matthew suggested. It could be through the using C_tr instead of C in this formula: A_i = C*F(m)^-1*v Since I assemble C myself I can apply transformation per element. That is not as elegant as using chain rule, but still possible. > > Also, the matrix A'*A has a huge null space. What sort of > regularization are you using. I use Tikhonov regularization. And there is variety of choices for regularization matrix you use. Could you (Matthew and Jed) suggest any solver which is better for this kind of problem? Thanks a lot for your help. Regards, Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Thu Jun 16 11:23:24 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 16 Jun 2011 18:23:24 +0200 Subject: [petsc-users] Getting access to matrix rows without and setting values simultaneously In-Reply-To: References: <4DF9E20C.5060704@gfz-potsdam.de> <4DF9F23A.5020704@gfz-potsdam.de> <4DF9FBB0.1060104@gfz-potsdam.de> <4DFA0892.6030301@gfz-potsdam.de> <4DFA1E44.7040507@gfz-potsdam.de> Message-ID: On Thu, Jun 16, 2011 at 17:57, Matthew Knepley wrote: > > > Doesn't he need A' A (normal equations) > Note that A has many more columns than rows, therefore A'*A has huge null space. > for the solve, so a MatShell? > I'm sorry, yes. Or MatComposite: http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Mat/MatCreateComposite.html But there is also likely some regularization so MatShell makes sense. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pabloarganaras at gmail.com Thu Jun 16 13:24:01 2011 From: pabloarganaras at gmail.com (=?ISO-8859-1?Q?Pablo_Arga=F1aras?=) Date: Thu, 16 Jun 2011 15:24:01 -0300 Subject: [petsc-users] Running PETSC_DIR/src/snes/examples/tutorials/ex19.c Message-ID: Hi PETSc developers, I have installed PETSc and Cuda 3.2 on computer, and when I try to run a tutorial example for the nonlinear solvers PETSC_DIR/src/snes/examples/tutorials/ex19.c I get an error message in the mpiaijcuda mat type. Any help? Thanks, Pablo. ========================================================================================= $ ./ex19 -da_vec_type mpicuda -da_mat_type mpiaijcuda -pc_type none -dmmg_nlevels 1 -da_grid_x 100 -da_grid_y 100 -log_summary -mat_no_inode -preload off -cuda_synchronize [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type seehttp:// www.mcs.anl.gov/petsc/petsc-as/documentation/installation.html#external! [0]PETSC ERROR: Unknown Mat type given: mpiaijcuda! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ex19 on a x86-64-op named mecom14 by arganap Thu Jun 16 15:04:23 2011 [0]PETSC ERROR: Libraries linked from /usr/local/petsc-201105O/lib [0]PETSC ERROR: Configure run at Tue Jun 14 15:55:30 2011 [0]PETSC ERROR: Configure options --prefix=/usr/local/petsc-201105O --with-petsc-arch=x86-64-opt --with-cuda=1 --with-cusp=1 --with-thrust=1 --doCleanup=0 --with-debugging=0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatSetType() line 51 in src/mat/interface/matreg.c [0]PETSC ERROR: DAGetMatrix() line 643 in src/dm/da/utils/fdda.c [0]PETSC ERROR: DMGetMatrix() line 231 in src/dm/da/utils/dm.c [0]PETSC ERROR: DMMGSetSNES() line 640 in src/snes/utils/damgsnes.c [0]PETSC ERROR: DMMGSetSNESLocal_Private() line 952 in src/snes/utils/damgsnes.c [0]PETSC ERROR: main() line 140 in src/snes/examples/tutorials/ex19.c application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0 arganap at mecom14:~/PETSc/petsc-3.1-p8/src/snes/examples/tutorials$ =================================================================================== -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 16 13:36:32 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 16 Jun 2011 18:36:32 +0000 Subject: [petsc-users] Running PETSC_DIR/src/snes/examples/tutorials/ex19.c In-Reply-To: References: Message-ID: Can you send configure.log to petsc-maint at mcs.anl.gov? Thanks, Matt 2011/6/16 Pablo Arga?aras > Hi PETSc developers, > I have installed PETSc and Cuda 3.2 on computer, and when I try to run a > tutorial example for the nonlinear solvers > PETSC_DIR/src/snes/examples/tutorials/ex19.c > > I get an error message in the mpiaijcuda mat type. > > Any help? > > Thanks, > Pablo. > > > > ========================================================================================= > > $ ./ex19 -da_vec_type mpicuda -da_mat_type mpiaijcuda -pc_type none > -dmmg_nlevels 1 -da_grid_x 100 -da_grid_y 100 -log_summary -mat_no_inode > -preload off -cuda_synchronize > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external > package needed for type > seehttp:// > www.mcs.anl.gov/petsc/petsc-as/documentation/installation.html#external! > [0]PETSC ERROR: Unknown Mat type given: mpiaijcuda! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 > CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex19 on a x86-64-op named mecom14 by arganap Thu Jun 16 > 15:04:23 2011 > [0]PETSC ERROR: Libraries linked from /usr/local/petsc-201105O/lib > [0]PETSC ERROR: Configure run at Tue Jun 14 15:55:30 2011 > [0]PETSC ERROR: Configure options --prefix=/usr/local/petsc-201105O > --with-petsc-arch=x86-64-opt --with-cuda=1 --with-cusp=1 --with-thrust=1 > --doCleanup=0 --with-debugging=0 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatSetType() line 51 in src/mat/interface/matreg.c > [0]PETSC ERROR: DAGetMatrix() line 643 in src/dm/da/utils/fdda.c > [0]PETSC ERROR: DMGetMatrix() line 231 in src/dm/da/utils/dm.c > [0]PETSC ERROR: DMMGSetSNES() line 640 in src/snes/utils/damgsnes.c > [0]PETSC ERROR: DMMGSetSNESLocal_Private() line 952 in > src/snes/utils/damgsnes.c > [0]PETSC ERROR: main() line 140 in src/snes/examples/tutorials/ex19.c > application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0 > arganap at mecom14:~/PETSc/petsc-3.1-p8/src/snes/examples/tutorials$ > > =================================================================================== > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenglun.wei at gmail.com Thu Jun 16 16:49:24 2011 From: zhenglun.wei at gmail.com (Alan Wei) Date: Thu, 16 Jun 2011 16:49:24 -0500 Subject: [petsc-users] Output question for ex29. In-Reply-To: References: <90e6ba6e8366edf58804a561160d@google.com> Message-ID: I still have some problem with the output by using VecView(). I read its source code but still confusing. 1) Does it just execute in 0 rank process after a MPI reduce or it execute in every process; 2) for every execution of VecView, does it loop 'i' first or 'j' first if it is x[j][i] in VecView(x,viewer). There is a minor question here. If VecView does do loop, is it generally faster than that I do loop by for() command manually ^_^ I believe it is, but I want to confirm it; 3) the VTK file output by VecView is a single column file with some header content in the beginning. I want a 2 column file with, say MPI rank number as the first column, can I do that? Could you please suggest a function to call or I need to loop it by myself. thanks in advance, ^_^ Alan > On Fri, Jun 10, 2011 at 5:03 PM, Alan Wei wrote: > >> Oh, I got it. Indeed I found the original subroutine has the coordinate. >> However, I may not need it since I want to plot it to a software named: >> Tecplot. It needs data only. Thanks a lot, Matt. >> >> sincerely, >> Alan >> >> >> On Fri, Jun 10, 2011 at 4:57 PM, Matthew Knepley wrote: >> >>> On Fri, Jun 10, 2011 at 4:54 PM, Alan Wei wrote: >>> >>>> oh, really. what do you meant 'valid'? it is created and data is inside. >>>> If I have to use it, what should I do for this function? actually, I do not >>>> have any thing in 'DM' format other than the 'da'. However, the 'da' does >>>> not contain anything. >_< >>> >>> >>> A VTK file must have a mesh in it. If you have not used VTK before, I do >>> not think you should start with this function. >>> >>> Matt >>> >>> >>>> >>>> Alan >>>> >>>> >>>> On Fri, Jun 10, 2011 at 4:44 PM, Matthew Knepley wrote: >>>> >>>>> On Fri, Jun 10, 2011 at 4:40 PM, Alan Wei wrote: >>>>> >>>>>> Thanks, Matt. I appreciate a lot for your help. I just delete 'DMView' >>>>>> part since I do not need to see the plot of 'dm'. Right now it really plot >>>>>> out everything. >>>>>> >>>>> >>>>> Without the DMView(), the VTK file will not be valid. >>>>> >>>>> Matt >>>>> >>>>> >>>>>> ^_^ thank you so much, again. >>>>>> Alan >>>>>> >>>>>> >>>>>> On Fri, Jun 10, 2011 at 4:26 PM, Matthew Knepley wrote: >>>>>> >>>>>>> On Fri, Jun 10, 2011 at 4:18 PM, Alan Wei wrote: >>>>>>> >>>>>>>> Here is my code right now, and the error message is attached. ^_^ >>>>>>>> >>>>>>>> >>>>>>>> PetscErrorCode VecView_VTK(Vec x, const char filename[], const char >>>>>>>> bcName[]) >>>>>>>> { >>>>>>>> >>>>>>>> MPI_Comm comm; >>>>>>>> DM da; >>>>>>>> Vec coords; >>>>>>>> PetscViewer viewer; >>>>>>>> PetscScalar *array, *values; >>>>>>>> PetscInt n, N, maxn, mx, my, dof; >>>>>>>> PetscInt i, p; >>>>>>>> MPI_Status status; >>>>>>>> PetscMPIInt rank, size, tag, nn; >>>>>>>> PetscErrorCode ierr; >>>>>>>> >>>>>>>> PetscFunctionBegin; >>>>>>>> PetscViewerASCIIOpen(PETSC_COMM_WORLD, filename, &viewer); >>>>>>>> >>>>>>> >>>>>>> ALWAYS check the return values: >>>>>>> >>>>>>> ierr = PetscVIewerASCIIOpen(PETSC_COMM_WORLD, filename, >>>>>>> &viewer);CHKERRQ(ierr); >>>>>>> >>>>>>> Do this for all the function calls. >>>>>>> >>>>>>> >>>>>>>> PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_VTK); >>>>>>>> DMView(x, viewer); >>>>>>>> >>>>>>> >>>>>>> The first argument here is a Vec, x, not the DM. Pass in the DM as >>>>>>> well. >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> VecView(x, viewer); >>>>>>>> PetscViewerDestroy(&viewer); >>>>>>>> PetscFunctionReturn(0); >>>>>>>> } >>>>>>>> >>>>>>>> thanks a lot, >>>>>>>> Alan >>>>>>>> >>>>>>>> On Fri, Jun 10, 2011 at 4:03 PM, Matthew Knepley >>>>>>> > wrote: >>>>>>>> >>>>>>>>> On Fri, Jun 10, 2011 at 3:47 PM, Alan Wei wrote: >>>>>>>>> >>>>>>>>>> Thanks, Matt. You're brilliant. >>>>>>>>>> I re-write it and the subroutine is like this. It indeed works. >>>>>>>>>> However, there are two questions. >>>>>>>>>> >>>>>>>>>> PetscErrorCode VecView_VTK(Vec x, const char filename[], const >>>>>>>>>> char bcName[]) >>>>>>>>>> { >>>>>>>>>> >>>>>>>>>> MPI_Comm comm; >>>>>>>>>> DM da; >>>>>>>>>> Vec coords; >>>>>>>>>> PetscViewer viewer; >>>>>>>>>> PetscScalar *array, *values; >>>>>>>>>> PetscInt n, N, maxn, mx, my, dof; >>>>>>>>>> PetscInt i, p; >>>>>>>>>> MPI_Status status; >>>>>>>>>> PetscMPIInt rank, size, tag, nn; >>>>>>>>>> PetscErrorCode ierr; >>>>>>>>>> >>>>>>>>>> PetscFunctionBegin; >>>>>>>>>> PetscViewerASCIIOpen(PETSC_COMM_WORLD, filename, &viewer); >>>>>>>>>> >>>>>>>>>> PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_MATLAB); >>>>>>>>>> >>>>>>>>> >>>>>>>>> This still reads ASCII_MATLAB. It should be ASCII_VTK. I put that >>>>>>>>> in my last email. >>>>>>>>> >>>>>>>>> >>>>>>>>>> DMView(x, viewer); >>>>>>>>>> VecView(x, PETSC_VIEWER_STDOUT_SELF); >>>>>>>>>> >>>>>>>>> >>>>>>>>> This should be 'viewer', not PETSC_VIEWER_STDOUT_SELF. >>>>>>>>> >>>>>>>>> >>>>>>>>>> PetscViewerDestroy(&viewer); >>>>>>>>>> PetscFunctionReturn(0); >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> 1) It does plot out the matrix to the console, but it does not >>>>>>>>>> create the file and write it to the file; >>>>>>>>>> >>>>>>>>> >>>>>>>>> There is nothing to do with any matrix in this function, only a >>>>>>>>> vector. The vector is written to the >>>>>>>>> console because the argument is wrong above. >>>>>>>>> >>>>>>>>> >>>>>>>>>> 2) everytime I execute it, it pop up a error message at the end of >>>>>>>>>> the computation, as I attached here. >>>>>>>>>> Could you please help me about these. ^_^ >>>>>>>>>> >>>>>>>>> >>>>>>>>> Its taking forever to download this message (3M) so I cannot see >>>>>>>>> the error right now. >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>>> thanks in advance, >>>>>>>>>> Alan >>>>>>>>>> >>>>>>>>>> On Fri, Jun 10, 2011 at 3:23 PM, Matthew Knepley < >>>>>>>>>> knepley at gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> On Fri, Jun 10, 2011 at 3:16 PM, Alan Wei < >>>>>>>>>>> zhenglun.wei at gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> ooops, sorry, here it is. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> You did not pass a valid communicator to PetscVIewerASCIIOpen(). >>>>>>>>>>> Use PETSC_COMM_WORLD. >>>>>>>>>>> >>>>>>>>>>> Matt >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> ^_^ >>>>>>>>>>>> >>>>>>>>>>>> Alan >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Jun 10, 2011 at 3:09 PM, Matthew Knepley < >>>>>>>>>>>> knepley at gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Jun 10, 2011 at 3:05 PM, wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Dear Matt, >>>>>>>>>>>>>> Following your suggestions, I modified the VecView_VTK in >>>>>>>>>>>>>> /src/ksp/ksp/example/tutorial/ex29.c to >>>>>>>>>>>>>> >>>>>>>>>>>>>> PetscFunctionBegin; >>>>>>>>>>>>>> PetscViewerASCIIOpen(comm, filename, &viewer); >>>>>>>>>>>>>> PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_MATLAB); >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I told you to use ASCII_VTK, not ASCII_MATLAB. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> DMView(x, viewer); >>>>>>>>>>>>>> VecView(x, PETSC_VIEWER_STDOUT_WORLD); >>>>>>>>>>>>>> PetscViewerDestroy(&viewer); >>>>>>>>>>>>>> PetscFunctionReturn(0); >>>>>>>>>>>>>> >>>>>>>>>>>>>> However, I always give me some error. Could you please briefly >>>>>>>>>>>>>> check if there is any obvious coding problem here. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> How can I possibly know what went wrong if you do not send the >>>>>>>>>>>>> ENTIRE error message. >>>>>>>>>>>>> >>>>>>>>>>>>> Matt >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> thanks, >>>>>>>>>>>>>> Alan >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On , Matthew Knepley wrote: >>>>>>>>>>>>>> > On Thu, Jun 9, 2011 at 6:18 PM, Matthew Knepley >>>>>>>>>>>>>> knepley at gmail.com> wrote: >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > On Thu, Jun 9, 2011 at 6:01 PM, zhenglun.wei at gmail.com> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Dear Sir/Madam, >>>>>>>>>>>>>> > I'm still studying on the ex29 of >>>>>>>>>>>>>> /src/ksp/ksp/example/tutorial. Before I met a problem on VecView_VTK in >>>>>>>>>>>>>> parallel computation. I'm trying to modify it in order to output some data >>>>>>>>>>>>>> from the computation. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Here is a better answer. If you want output, throw away this >>>>>>>>>>>>>> old function which is broken, and use >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > PetscViewerASCIIOpen() >>>>>>>>>>>>>> > PetscViewerASCIISetFormat(PETSC_VIEWER_ASCII_VTK) >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > DMView() >>>>>>>>>>>>>> > VecView() >>>>>>>>>>>>>> > PetscViewerDestroy() >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Thanks, >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Matt >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > 1) My first questions is that what does this section do in >>>>>>>>>>>>>> VecView_VTK: >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > 272: MPI_Comm_rank(comm, &rank); >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > 273: MPI_Comm_size(comm, &size); >>>>>>>>>>>>>> > 274: MPI_Reduce(&n, &maxn, 1, MPIU_INT, MPI_MAX, 0, comm); >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > 275: tag = ((PetscObject) viewer)->tag; >>>>>>>>>>>>>> > 276: if (!rank) { >>>>>>>>>>>>>> > 277: PetscMalloc((maxn+1) * sizeof(PetscScalar), &values); >>>>>>>>>>>>>> > 278: for(i = 0; i 279: PetscViewerASCIIPrintf(viewer, >>>>>>>>>>>>>> "%G\n", PetscRealPart(array[i])); >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > 280: } >>>>>>>>>>>>>> > 281: for(p = 1; p 282: MPI_Recv(values, (PetscMPIInt) n, >>>>>>>>>>>>>> MPIU_SCALAR, p, tag, comm, &status); >>>>>>>>>>>>>> >>>>>>>>>>>>>> > 283: MPI_Get_count(&status, MPIU_SCALAR, &nn); >>>>>>>>>>>>>> > 284: for(i = 0; i >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > 285: PetscViewerASCIIPrintf(viewer, "%G\n", >>>>>>>>>>>>>> PetscRealPart(array[i])); >>>>>>>>>>>>>> > 286: } >>>>>>>>>>>>>> > 287: } >>>>>>>>>>>>>> > 288: PetscFree(values); >>>>>>>>>>>>>> > 289: } else { >>>>>>>>>>>>>> > 290: MPI_Send(array, n, MPIU_SCALAR, 0, tag, comm); >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > 291: } >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > What I understand is: it gather all the data from different >>>>>>>>>>>>>> process in parallel computation, and output it to the 'viewer'. I comment >>>>>>>>>>>>>> out everything in VecView_VTK except this part, there is no error message >>>>>>>>>>>>>> coming up in my parallel computation so far. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > 2) however, I really don't know how did it split the domain >>>>>>>>>>>>>> for parallel computation. For example, if I use 4 processes, is the domain >>>>>>>>>>>>>> split like: >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > The DMDA describes the domain splitting. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Matt >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > a) >>>>>>>>>>>>>> > | >>>>>>>>>>>>>> > 0 | 1 >>>>>>>>>>>>>> > | >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > ---------------|-------------- >>>>>>>>>>>>>> > | >>>>>>>>>>>>>> > 2 | 3 >>>>>>>>>>>>>> > | >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > b) >>>>>>>>>>>>>> > | >>>>>>>>>>>>>> > 0 | 2 >>>>>>>>>>>>>> > | >>>>>>>>>>>>>> > ---------------|-------------- >>>>>>>>>>>>>> > | >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > 1 | 3 >>>>>>>>>>>>>> > | >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > c) >>>>>>>>>>>>>> > | | | | >>>>>>>>>>>>>> > | | | | >>>>>>>>>>>>>> > 0 | | 2 | | >>>>>>>>>>>>>> > | 1 | | 3 | >>>>>>>>>>>>>> > | | | | >>>>>>>>>>>>>> > | | | | >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > d) >>>>>>>>>>>>>> > 0 >>>>>>>>>>>>>> > ------------------------ >>>>>>>>>>>>>> > 1 >>>>>>>>>>>>>> > ------------------------ >>>>>>>>>>>>>> > 2 >>>>>>>>>>>>>> > ------------------------ >>>>>>>>>>>>>> > 3 >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > thanks in advance, >>>>>>>>>>>>>> > Alan >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > -- >>>>>>>>>>>>>> > What most experimenters take for granted before they begin >>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which >>>>>>>>>>>>>> their experiments lead. >>>>>>>>>>>>>> > -- Norbert Wiener >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > -- >>>>>>>>>>>>>> > What most experimenters take for granted before they begin >>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which >>>>>>>>>>>>>> their experiments lead. >>>>>>>>>>>>>> > -- Norbert Wiener >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which >>>>>>>>>>>>> their experiments lead. >>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>>>> experiments lead. >>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jun 16 18:17:20 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 16 Jun 2011 18:17:20 -0500 Subject: [petsc-users] Getting access to matrix rows without and setting values simultaneously In-Reply-To: References: <4DF9E20C.5060704@gfz-potsdam.de> Message-ID: I am afraid that Jed gave you some misinformation on this response that led you guys off the trail a bit. This is the first mistake I've seen Jed make in the three years I've known him (as opposed to Matt and I who make mistakes everyday :-)) so don't be too hard on him. MatGetArray() on dense matrices returns a pointer to the entire local matrix stored in column major order (Fortran style), thus you can call MatGetArray() and then muck with all the local entries very efficiently since you are just doing direct array access. Hope I am right (and this is not my daily mistake) and this helps you, Barry Dang proposal issues most of the day so not able to read email until now. On Jun 16, 2011, at 6:10 AM, Jed Brown wrote: > On Thu, Jun 16, 2011 at 12:59, Alexander Grayver wrote: > ! store transformed elements > call MatSetValues(A,1,rowidx,nvals,cols,transformedvals,INSERT_VALUES,ierr);CHKERRQ(ierr); > call MatRestoreRow(A,i,nvals,cols,vals,ierr);CHKERRQ(ierr); > > You can't mix these because MatGetRow() needs an assembled matrix, and MatSetValues() makes the matrix be unassembled. > > What do you want the parallel semantics to be during this transformation? > > The MatGetArray() for MPIDense just gives you access to the "diagonal" block. > > What is the high-level description of the operation you're trying to do? Is this "matrix" actually being used as a linear operator, or is it a grid of values? From bsmith at mcs.anl.gov Thu Jun 16 20:37:01 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 16 Jun 2011 20:37:01 -0500 Subject: [petsc-users] Running PETSC_DIR/src/snes/examples/tutorials/ex19.c In-Reply-To: References: Message-ID: It appears that you are using the PETSc 3.1.8 RELEASE? The directions for our GPU stuff http://www.mcs.anl.gov/petsc/petsc-as/features/gpus.html clearly state that it only works in petsc-dev. You must install petsc-dev to use the CUDA stuff. Barry On Jun 16, 2011, at 1:24 PM, Pablo Arga?aras wrote: > Hi PETSc developers, > I have installed PETSc and Cuda 3.2 on computer, and when I try to run a tutorial example for the nonlinear solvers > PETSC_DIR/src/snes/examples/tutorials/ex19.c > > I get an error message in the mpiaijcuda mat type. > > Any help? > > Thanks, > Pablo. > > > ========================================================================================= > > $ ./ex19 -da_vec_type mpicuda -da_mat_type mpiaijcuda -pc_type none -dmmg_nlevels 1 -da_grid_x 100 -da_grid_y 100 -log_summary -mat_no_inode -preload off -cuda_synchronize > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type > seehttp://www.mcs.anl.gov/petsc/petsc-as/documentation/installation.html#external! > [0]PETSC ERROR: Unknown Mat type given: mpiaijcuda! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex19 on a x86-64-op named mecom14 by arganap Thu Jun 16 15:04:23 2011 > [0]PETSC ERROR: Libraries linked from /usr/local/petsc-201105O/lib > [0]PETSC ERROR: Configure run at Tue Jun 14 15:55:30 2011 > [0]PETSC ERROR: Configure options --prefix=/usr/local/petsc-201105O --with-petsc-arch=x86-64-opt --with-cuda=1 --with-cusp=1 --with-thrust=1 --doCleanup=0 --with-debugging=0 > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatSetType() line 51 in src/mat/interface/matreg.c > [0]PETSC ERROR: DAGetMatrix() line 643 in src/dm/da/utils/fdda.c > [0]PETSC ERROR: DMGetMatrix() line 231 in src/dm/da/utils/dm.c > [0]PETSC ERROR: DMMGSetSNES() line 640 in src/snes/utils/damgsnes.c > [0]PETSC ERROR: DMMGSetSNESLocal_Private() line 952 in src/snes/utils/damgsnes.c > [0]PETSC ERROR: main() line 140 in src/snes/examples/tutorials/ex19.c > application called MPI_Abort(MPI_COMM_WORLD, 86) - process 0 > arganap at mecom14:~/PETSc/petsc-3.1-p8/src/snes/examples/tutorials$ > =================================================================================== From agrayver at gfz-potsdam.de Fri Jun 17 04:41:56 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 17 Jun 2011 11:41:56 +0200 Subject: [petsc-users] Getting access to matrix rows without and setting values simultaneously In-Reply-To: References: <4DF9E20C.5060704@gfz-potsdam.de> Message-ID: <4DFB2164.1000601@gfz-potsdam.de> Hi Barry, It happens to all of us. ;) So probably you're right about MatGetArray, I would be happy to check it if not another error I got on MatGetArrayF90... I sent it as a separate branch. Thanks a lot for clarification. Regards, Alexander On 17.06.2011 01:17, Barry Smith wrote: > I am afraid that Jed gave you some misinformation on this response that led you guys off the trail a bit. This is the first mistake I've seen Jed make in the three years I've known him (as opposed to Matt and I who make mistakes everyday :-)) so don't be too hard on him. > > MatGetArray() on dense matrices returns a pointer to the entire local matrix stored in column major order (Fortran style), thus you can call MatGetArray() and then muck with all the local entries very efficiently since you are just doing direct array access. > > Hope I am right (and this is not my daily mistake) and this helps you, > > > Barry > > Dang proposal issues most of the day so not able to read email until now. > > > On Jun 16, 2011, at 6:10 AM, Jed Brown wrote: > >> On Thu, Jun 16, 2011 at 12:59, Alexander Grayver wrote: >> ! store transformed elements >> call MatSetValues(A,1,rowidx,nvals,cols,transformedvals,INSERT_VALUES,ierr);CHKERRQ(ierr); >> call MatRestoreRow(A,i,nvals,cols,vals,ierr);CHKERRQ(ierr); >> >> You can't mix these because MatGetRow() needs an assembled matrix, and MatSetValues() makes the matrix be unassembled. >> >> What do you want the parallel semantics to be during this transformation? >> >> The MatGetArray() for MPIDense just gives you access to the "diagonal" block. >> >> What is the high-level description of the operation you're trying to do? Is this "matrix" actually being used as a linear operator, or is it a grid of values? From agrayver at gfz-potsdam.de Fri Jun 17 04:42:02 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 17 Jun 2011 11:42:02 +0200 Subject: [petsc-users] Error in VecGetArrayF90 and MatGetArrayF90 Message-ID: <4DFB216A.3080008@gfz-potsdam.de> Hello! I use petsc-3.1.7 on Windows x64 and Intel Fortran Compiler 10.1. Whenever I call VecGetArrayF90/MatGetArrayF90 it crashes. For example: subroutine VecRealPart(v) implicit none #include "finclude/petscsys.h" #include "finclude/petscvec.h" #include "finclude/petscis.h" ! input variables Vec :: v ! local variables integer(kind=int32) :: ierr !error index integer(kind=int32) :: i,istart,iend !counters for loops and indices PetscScalar,pointer :: lv_v(:) ! localy owned part of the vector call VecGetOwnershipRange(v,istart,iend,ierr) call VecGetArrayF90(v,lv_v,ierr) ! <<<<<<<<<<<<<< crash do i=1,iend-istart lv_v(i) = dcmplx(real(lv_v(i))) enddo call VecRestoreArrayF90(v,lv_v,ierr) end subroutine VecRealPart When I trace into these procedures during debuging I see that parameter ptr is always 0x00000000: void PETSC_STDCALL matgetarrayf90_(Mat *mat,F90Array2d *ptr,int *ierr PETSC_F90_2PTR_PROTO(ptrd)) void PETSC_STDCALL vecgetarrayf90_(Vec *x,F90Array1d *ptr,int *__ierr PETSC_F90_2PTR_PROTO(ptrd)) What could be the possible reason for such behavior? Regards, Alexander From jed at 59A2.org Fri Jun 17 05:00:26 2011 From: jed at 59A2.org (Jed Brown) Date: Fri, 17 Jun 2011 12:00:26 +0200 Subject: [petsc-users] Error in VecGetArrayF90 and MatGetArrayF90 In-Reply-To: <4DFB216A.3080008@gfz-potsdam.de> References: <4DFB216A.3080008@gfz-potsdam.de> Message-ID: On Fri, Jun 17, 2011 at 11:42, Alexander Grayver wrote: > Hello! > > I use petsc-3.1.7 on Windows x64 and Intel Fortran Compiler 10.1. > Whenever I call VecGetArrayF90/MatGetArrayF90 it crashes. > Hmm, maybe Intel Fortran has a different ABI for pointer parameters on 64-bit Windows. Does this examples also crash? src/vec/vec/examples/tutorials/ex4f90.F If this doesn't work, then someone will need to learn what the ABI is so that the F90 wrappers work. How about this one (using F77 arrays)? src/mat/examples/tests/ex36f.F As a short-term solution, perhaps you can use this API? -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Fri Jun 17 05:05:29 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 17 Jun 2011 12:05:29 +0200 Subject: [petsc-users] Error in VecGetArrayF90 and MatGetArrayF90 In-Reply-To: References: <4DFB216A.3080008@gfz-potsdam.de> Message-ID: <4DFB26E9.9060009@gfz-potsdam.de> On 17.06.2011 12:00, Jed Brown wrote: > On Fri, Jun 17, 2011 at 11:42, Alexander Grayver > > wrote: > > Hello! > > I use petsc-3.1.7 on Windows x64 and Intel Fortran Compiler 10.1. > Whenever I call VecGetArrayF90/MatGetArrayF90 it crashes. > > > Hmm, maybe Intel Fortran has a different ABI for pointer parameters on > 64-bit Windows. What is ABI? > > Does this examples also crash? > > src/vec/vec/examples/tutorials/ex4f90.F > > If this doesn't work, then someone will need to learn what the ABI is > so that the F90 wrappers work. > > How about this one (using F77 arrays)? > > src/mat/examples/tests/ex36f.F Both examples work without crashing. Pretty weird... I can tell more. I found place in my code, just different fortran module, where VecGetArrayF90 works both on seq and mpi vectors. > > As a short-term solution, perhaps you can use this API? > What API do you mean exactly? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Fri Jun 17 05:23:02 2011 From: jed at 59A2.org (Jed Brown) Date: Fri, 17 Jun 2011 12:23:02 +0200 Subject: [petsc-users] Error in VecGetArrayF90 and MatGetArrayF90 In-Reply-To: <4DFB26E9.9060009@gfz-potsdam.de> References: <4DFB216A.3080008@gfz-potsdam.de> <4DFB26E9.9060009@gfz-potsdam.de> Message-ID: On Fri, Jun 17, 2011 at 12:05, Alexander Grayver wrote: > What is ABI? > Application Binary Interface, basically the assembly-level calling semantics. If the tests run successfully, then this is not the problem. Both examples work without crashing. Pretty weird... > One more to try: mat/examples/tests/ex16f90.F > I can tell more. I found place in my code, just different fortran module, > where VecGetArrayF90 works both on seq and mpi vectors. > This call works from one Fortran module, but not from another? I suspect stack corruption, perhaps from something as simple as the wrong number of arguments to an earlier call. (Fortran having nearly worthless type checking.) > What API do you mean exactly? > The non-F90 VecGetArray() and MatGetArray(). But if MatGetArrayF90() is working when called in a different context, then this shouldn't be necessary. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rongtian at ncic.ac.cn Fri Jun 17 05:35:08 2011 From: rongtian at ncic.ac.cn (Tian(ICT)) Date: Fri, 17 Jun 2011 18:35:08 +0800 Subject: [petsc-users] Analytic Jacobian verfication In-Reply-To: References: <4DFB216A.3080008@gfz-potsdam.de><4DFB26E9.9060009@gfz-potsdam.de> Message-ID: Hi, a quick question. If I test the code with and without -snes_mf_operator and the SNES solve gives the "exactly" same solution (can they be the exactly same?), can I say the analytic Jacobian is correct? Rong -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Fri Jun 17 05:50:53 2011 From: jed at 59A2.org (Jed Brown) Date: Fri, 17 Jun 2011 12:50:53 +0200 Subject: [petsc-users] Analytic Jacobian verfication In-Reply-To: References: <4DFB216A.3080008@gfz-potsdam.de> <4DFB26E9.9060009@gfz-potsdam.de> Message-ID: On Fri, Jun 17, 2011 at 12:35, Tian(ICT) wrote: > If I test the code with and without -snes_mf_operator and the SNES > solve gives the "exactly" same solution (can they be the exactly same? They won't be exactly the same because the finite differenced Jacobian has more and different rounding errors from an analytic Jacobian. But they should agree to about sqrt(epsilon) which is about 7 significant digits for double precision. > , can I say the analytic Jacobian is correct? Yes, provided the problem you ran it on exercises all nonlinear terms in your equations. -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Fri Jun 17 05:50:35 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 17 Jun 2011 12:50:35 +0200 Subject: [petsc-users] Error in VecGetArrayF90 and MatGetArrayF90 In-Reply-To: References: <4DFB216A.3080008@gfz-potsdam.de> <4DFB26E9.9060009@gfz-potsdam.de> Message-ID: <4DFB317B.4050206@gfz-potsdam.de> Sorry Jed, t was my fault. I had to include: #include "finclude/petscvec.h90" #include "finclude/petscmat.h90" I read about that in documentation 3 months ago and forgot to do that in new module. On 17.06.2011 12:23, Jed Brown wrote: > On Fri, Jun 17, 2011 at 12:05, Alexander Grayver > > wrote: > > What is ABI? > > > > Application Binary Interface, basically the assembly-level calling > semantics. If the tests run successfully, then this is not the problem. > > Both examples work without crashing. Pretty weird... > > > One more to try: mat/examples/tests/ex16f90.F > > I can tell more. I found place in my code, just different fortran > module, where VecGetArrayF90 works both on seq and mpi vectors. > > > This call works from one Fortran module, but not from another? I > suspect stack corruption, perhaps from something as simple as the > wrong number of arguments to an earlier call. (Fortran having nearly > worthless type checking.) > > What API do you mean exactly? > > > The non-F90 VecGetArray() and MatGetArray(). But if MatGetArrayF90() > is working when called in a different context, then this shouldn't be > necessary. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rongtian at ncic.ac.cn Fri Jun 17 05:59:52 2011 From: rongtian at ncic.ac.cn (Tian(ICT)) Date: Fri, 17 Jun 2011 18:59:52 +0800 Subject: [petsc-users] Analytic Jacobian verfication In-Reply-To: References: <4DFB216A.3080008@gfz-potsdam.de><4DFB26E9.9060009@gfz-potsdam.de> Message-ID: <03D9ED7ED78A494AAB0280AE825E80A1@rti458laptop> Appreciate the quick answering. I DID get the identical solution, exact to all digital numbers (w and w/o -snes_mf_operator). so second quick question: if the mesh happen to be structured (say one square element), can finite differenced Jacobian be the exactly same as the analytic one? Or the exactly same solution whether -snes_mf_operator or not implies something wrong in my code? Rong ----- Original Message ----- From: Jed Brown To: PETSc users list Sent: Friday, June 17, 2011 6:50 PM Subject: Re: [petsc-users] Analytic Jacobian verfication On Fri, Jun 17, 2011 at 12:35, Tian(ICT) wrote: If I test the code with and without -snes_mf_operator and the SNES solve gives the "exactly" same solution (can they be the exactly same? They won't be exactly the same because the finite differenced Jacobian has more and different rounding errors from an analytic Jacobian. But they should agree to about sqrt(epsilon) which is about 7 significant digits for double precision. , can I say the analytic Jacobian is correct? Yes, provided the problem you ran it on exercises all nonlinear terms in your equations. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Fri Jun 17 06:19:03 2011 From: jed at 59A2.org (Jed Brown) Date: Fri, 17 Jun 2011 13:19:03 +0200 Subject: [petsc-users] Analytic Jacobian verfication In-Reply-To: <03D9ED7ED78A494AAB0280AE825E80A1@rti458laptop> References: <4DFB216A.3080008@gfz-potsdam.de> <4DFB26E9.9060009@gfz-potsdam.de> <03D9ED7ED78A494AAB0280AE825E80A1@rti458laptop> Message-ID: On Fri, Jun 17, 2011 at 12:59, Tian(ICT) wrote: > Appreciate the quick answering. I DID get the identical solution, exact to > all digital numbers (w and w/o -snes_mf_operator). > Please run both ways with -snes_view -snes_monitor and send the output. > so second quick question: if the mesh happen to be structured (say one > square element), can finite differenced Jacobian be the exactly same as the > analytic one? > Suppose there is no grid at all. Let f(x) = 3*x, the 3 is arbitrary. This is the simplest case because the function is actually linear, thus df(x,y) = 3*y. Now we compare z1 = f(x + eps*y) - f(x) / eps = (3*(x + eps*y) - 3*x)/eps; z2 = df(x,y) = 3*y; Since floating point arithmetic is not associative, these are not the same. For example, taking x=1, y=2, eps = 1e-8, we have z1 = 6.000000052353016 z2 = 6.000000000000000 If it's not exact for a linear scalar problem with all coefficients, inputs, and (exact) outputs exactly representable in floating point arithmetic, then it's pretty unlikely to be exact for any PDE. Or the exactly same solution whether -snes_mf_operator or not implies > something wrong in my code? > It's more likely that the same algorithm is used in both cases, -snes_view should tell us what's happening. -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Fri Jun 17 06:18:04 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 17 Jun 2011 13:18:04 +0200 Subject: [petsc-users] Error in VecGetArrayF90 and MatGetArrayF90 In-Reply-To: <4DFB317B.4050206@gfz-potsdam.de> References: <4DFB216A.3080008@gfz-potsdam.de> <4DFB26E9.9060009@gfz-potsdam.de> <4DFB317B.4050206@gfz-potsdam.de> Message-ID: <4DFB37EC.7060109@gfz-potsdam.de> I find it strange that in the documentation of MatGetArrayF90 you write: Synopsis /MatGetArrayF90(Mat x,{Scalar, pointer :: xx_v(:)},integer ierr)/ Which doesn't compile: /Error 1 Error: The shape matching rules of actual arguments and dummy arguments have been violated. [xx_v]/ What really works is /MatGetArrayF90(Mat x,{Scalar, pointer :: xx_v(:,:)},integer ierr)/ On 17.06.2011 12:50, Alexander Grayver wrote: > Sorry Jed, t was my fault. > I had to include: > #include "finclude/petscvec.h90" > #include "finclude/petscmat.h90" > > I read about that in documentation 3 months ago and forgot to do that > in new module. > > On 17.06.2011 12:23, Jed Brown wrote: >> On Fri, Jun 17, 2011 at 12:05, Alexander Grayver >> > wrote: >> >> What is ABI? >> >> >> >> Application Binary Interface, basically the assembly-level calling >> semantics. If the tests run successfully, then this is not the problem. >> >> Both examples work without crashing. Pretty weird... >> >> >> One more to try: mat/examples/tests/ex16f90.F >> >> I can tell more. I found place in my code, just different fortran >> module, where VecGetArrayF90 works both on seq and mpi vectors. >> >> >> This call works from one Fortran module, but not from another? I >> suspect stack corruption, perhaps from something as simple as the >> wrong number of arguments to an earlier call. (Fortran having nearly >> worthless type checking.) >> >> What API do you mean exactly? >> >> >> The non-F90 VecGetArray() and MatGetArray(). But if MatGetArrayF90() >> is working when called in a different context, then this shouldn't be >> necessary. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Fri Jun 17 06:23:03 2011 From: jed at 59A2.org (Jed Brown) Date: Fri, 17 Jun 2011 13:23:03 +0200 Subject: [petsc-users] Error in VecGetArrayF90 and MatGetArrayF90 In-Reply-To: <4DFB37EC.7060109@gfz-potsdam.de> References: <4DFB216A.3080008@gfz-potsdam.de> <4DFB26E9.9060009@gfz-potsdam.de> <4DFB317B.4050206@gfz-potsdam.de> <4DFB37EC.7060109@gfz-potsdam.de> Message-ID: On Fri, Jun 17, 2011 at 13:18, Alexander Grayver wrote: > > What really works is > *MatGetArrayF90(Mat x,{Scalar, pointer :: xx_v(:,:)},integer ierr)* > I've fixed the documentation, thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ozergedem at gmail.com Fri Jun 17 07:46:53 2011 From: ozergedem at gmail.com (=?ISO-8859-1?Q?=F6zge_erdem?=) Date: Fri, 17 Jun 2011 14:46:53 +0200 Subject: [petsc-users] PETSC_VIEWER_MATLAB In-Reply-To: References: Message-ID: Ok, thanks but for writing in PETSC format what do you suggest? On Thu, Jun 16, 2011 at 3:02 PM, Jed Brown wrote: On Thu, Jun 16, 2011 at 14:48, ?zge erdem wrote: > >> Can you give me more information how to use that >> viewer(PETSC_VIEWER_MATLAB) > > > This does not save it as a Matlab file, it talks to a running Matlab > session. To write a Matlab file, you can use > > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/DM/PetscViewerBinaryMatlabOpen.html > > But, especially for large files, it is much better to write a binary file > in PETSc format and load it from Matlab using > > $PETSC_DIR/bin/matlab/PetscBinaryRead.m > -- Ozge Erdem -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Fri Jun 17 07:51:22 2011 From: jed at 59A2.org (Jed Brown) Date: Fri, 17 Jun 2011 14:51:22 +0200 Subject: [petsc-users] PETSC_VIEWER_MATLAB In-Reply-To: References: Message-ID: On Fri, Jun 17, 2011 at 14:46, ?zge erdem wrote: > > Ok, thanks but for writing in PETSC format what do you suggest? Item 2: http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#matlab -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Fri Jun 17 12:16:15 2011 From: jed at 59A2.org (Jed Brown) Date: Fri, 17 Jun 2011 19:16:15 +0200 Subject: [petsc-users] AGU session: IN34 - Robust Methods and Software for Multiphysics Simulation Message-ID: For those in the community working on geophysics: I like to draw your attention to this computationally-oriented session at the American Geophysical Union Fall Meeting, San Francisco, California (5-9 Dec, 2011). IN34 - Robust Methods and Software for Multiphysics Simulation Many computational problems in geosciences, such as plate boundary dynamics and coupled ocean/atmosphere/land surface problems, involve multiple physical processes / domains interacting over a wide range of spatial and temporal scales. A single discretization technique rarely suits the entire problem. How do we guarantee numerical stability and accuracy over the timescales of interest? Can we move beyond the common approach of using first-order accurate operator splitting, semi-implicit / explicit time integration methods and decoupled nonlinear solution methods? How do we make the software maintainable and performant? We seek submissions from all disciplines addressing the challenge of solving stiffly-coupled multi-physics problems. Conveners: Jed Brown - ETH Zurich / Argonne National Laboratory Matt Knepley - Computational Institute, U of Chicago Louis Moresi - Monash University Marc Spiegelman - Columbia / LDEO Sponsor: Earth and Space Science Informatics (IN) Co-Sponsors: Atmospheric Sciences (A) Cryosphere (C) Study of Earth's Deep Interior (DI) Hydrology (H) Natural Hazards (NH) Ocean Sciences (OS) Planetary Sciences (P) Volcanology, Geochemistry, and Petrology (V) Deadlines: 4 August - Abstract Submissions http://sites.agu.org/fallmeeting/scientific-program/session-search/547 General meeting information: http://sites.agu.org/fallmeeting/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From gianmail at gmail.com Mon Jun 20 09:26:21 2011 From: gianmail at gmail.com (Gianluca Meneghello) Date: Mon, 20 Jun 2011 16:26:21 +0200 Subject: [petsc-users] ILU and Block Gauss Seidel smoothing In-Reply-To: <9F52876E-C796-407A-8608-53D30D666A83@mcs.anl.gov> References: <9F52876E-C796-407A-8608-53D30D666A83@mcs.anl.gov> Message-ID: Barry, thanks for your answer and sorry for my late reply. While your suggestion is in the right direction for what I need, I think I would need something a little different. I would like to use different ordering in the same run, corresponding to the different ordering of the unknown in a rectangular grid ? that is in NW, NE, SW, SE directions. My goal is to perform some relaxation sweeps for each ordering at each level of a multigrid process, probably using ksptype PREONLY and pctype ILU. Is that possible? Is there the equivalent of -pc_sor_its with ILU (-pc_ilu_its maybe)? I also have the problem that in order to build the ordering I would need to have access to a structure containing some grid informations, and it seems I cannot pass that structure to the YourOrdering function you suggested. I guess a solution for me could be to build the IS from an external function (not used by petsc) and then attach them directly to the mat structure. I also guess that the one to use are the ones at line 36 of http://www.mcs.anl.gov/petsc/petsc-2/snapshots/petsc-dev/src/mat/impls/aij/seq/aij.h.html Not being an expert in the underlying structure of PETSc, I wonder whether this is a good idea or not. Is there a more standard alternative? Thanks Gianluca On 3 June 2011 17:20, Barry Smith wrote: > > On May 25, 2011, at 2:45 AM, Gianluca Meneghello wrote: > >> Dear all, >> >> I am writing a multigrid solver for the Navier Stokes Equations, using >> PETSc. I am not using the dmmg structure because I need grid >> refinement and, as far as I can see, that is not supported. >> >> I have two questions concerning two different possibilities of smoothing: >> >> 1) ILU relaxation: in Multigrid (U. Trottenberg,Cornelis W. >> Oosterlee,Anton Sch?ller), it is suggested that alternating ILU >> represents a robust smoother. Given a routine for ILU decomposition, >> my understanding is that alternation depends on the ordering of the >> matrix. My code builds a linearized Navier Stokes operator where first >> the u momentum equation, then the v momentum and then the p equation >> are stored. Each equation is built in EN order ? first comes the >> bottom horizontal line, from West to East, then I move line by line >> from South to North. Otherwise stated, the position of each unknown in >> the matrix is given by P = cf*nx*ny + j*nx + i, where cf is the >> variable number (0 = u, 1 = v, 2 = p) and i,j,nx,ny are as usual. >> >> For "historical" reasons I'm not using the DM structure even if the >> grids are logically rectangular, but I can change that. >> >> My question: is there in PETSc a way of doing alternating ILU which >> does not require rebuilding the matrix with different ordering each >> time? > > ? ?There is a way for you to set the ordering of the ILU factorization without building the matrix in that ordering. It requires that you write a function with the calling sequence > > PetscErrorCode ?YourOrdering(Mat mat,const MatOrderingType type,IS *isrow, IS *iscol) > { > ? ?Your function fills up isrow with the order of the rows you want the ILU to visit, for example if you want the first row visited to be ten then you would create an isrow whose first entry is 10. You can/and probably should use the same ordering for rows and columns. Your function will igore the type name that is passed in. > } > > ? In your main program after PetscInitialize() call MatOrderingRegister("yourname",0,yourname,YourOrdering); > > now when you run the program you can use the option -pc_factor_mat_ordering_type yourname ?and it will use the function you provided to generate the ordering that is used for ILU. > > ? Good luck and sorry for not responding to this message sooner, > > ? Barry > > >> >> Somehow related questions: what is the best ordering for the matrix >> and which one is the one used by the DM structure? >> >> >> 2) Decoupled block-line Gauss Seidel relaxation: this is another >> smoother I'm considering, which explains the matrix ordering above (I >> use block horizontal lines smoothing, so that each line is stored >> consecutively in the matrix). At the moment the full linearized >> operator is first built and then parts of the matrix are extracted. >> In reality though, I never need the full linearized operator to be >> built except on the coarsest grid. Rather, I need it to be constructed >> each block-line at a time. I've seen that for parallel applications >> each part of the matrix is built on its corresponding processor. Is >> there a way to do it sequentially and to have control on which part is >> built? >> >> >> Any other suggestion on both subjects is of course welcome. Thanks in advance >> >> Gianluca >> >> >> >> -- >> "[Je pense que] l'homme est un monde qui vaut des fois les mondes et >> que les plus ardentes ambitions sont celles qui ont eu l'orgueil de >> l'Anonymat" -- Non omnibus, sed mihi et tibi >> Amedeo Modigliani > > -- "[Je pense que] l'homme est un monde qui vaut des fois les mondes et que les plus ardentes ambitions sont celles qui ont eu l'orgueil de l'Anonymat" -- Non omnibus, sed mihi et tibi Amedeo Modigliani From jed at 59A2.org Mon Jun 20 09:48:56 2011 From: jed at 59A2.org (Jed Brown) Date: Mon, 20 Jun 2011 16:48:56 +0200 Subject: [petsc-users] ILU and Block Gauss Seidel smoothing In-Reply-To: References: <9F52876E-C796-407A-8608-53D30D666A83@mcs.anl.gov> Message-ID: On Mon, Jun 20, 2011 at 16:26, Gianluca Meneghello wrote: > My goal is to perform some relaxation sweeps for each ordering at each > level of a multigrid process, probably using ksptype PREONLY and > pctype ILU. > Is that possible? > Yes, but you would need to perform a separate factorization for each ordering. Also, the orderings that are not aligned with the grid in memory will have poor performance. This is a general problem with using different orderings. Is there the equivalent of -pc_sor_its with ILU > (-pc_ilu_its maybe)? > -ksp_type richardson is the same thing, put it inside -pc_type bjacobi for -pc_sor_lits. (-pc_sor_its is just a lower overhead way to do that cycling). > I also have the problem that in order to build the ordering I would > need to have access to a structure containing some grid informations, > and it seems I cannot pass that structure to the YourOrdering function > you suggested. > You can PetscObjectCompose() your structure to the Mat. You might need PetscContainerCreate() to wrap your struct. > > I guess a solution for me could be to build the IS from an external > function (not used by petsc) and then attach them directly to the mat > structure. I also guess that the one to use are the ones at line 36 of > No, those slots are not public. Use PetscObjectCompose(). -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Mon Jun 20 14:42:26 2011 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Mon, 20 Jun 2011 12:42:26 -0700 Subject: [petsc-users] MatSetValues and Zero enteries Message-ID: Hi folks, I've noticed that MatSetValues explicitly inserts zeros in the matrix if an element is really zero. Is there a way to drop elements (or not save them in the first place) that are below a certain tolerance? I realize one way of doing it is in the application code and when using MatSetValues but I'm just wondering if PETSc has a function for it? Thanks, Mohammad -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.mousel at gmail.com Mon Jun 20 15:12:55 2011 From: john.mousel at gmail.com (John Mousel) Date: Mon, 20 Jun 2011 15:12:55 -0500 Subject: [petsc-users] MatSetValues and Zero enteries In-Reply-To: References: Message-ID: See the matrix options available in the man pages. I think the function below is what you are looking for. http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatSetOption.html MatSetOption (A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE ); Best Regards, John On Mon, Jun 20, 2011 at 2:42 PM, Mohammad Mirzadeh wrote: > Hi folks, > > I've noticed that MatSetValues explicitly inserts zeros in the matrix if an > element is really zero. Is there a way to drop elements (or not save them in > the first place) that are below a certain tolerance? I realize one way of > doing it is in the application code and when using MatSetValues but I'm just > wondering if PETSc has a function for it? > > Thanks, > Mohammad > -------------- next part -------------- An HTML attachment was scrubbed... URL: From py.aquilanti at gmail.com Thu Jun 16 06:35:05 2011 From: py.aquilanti at gmail.com (Pierre-Yves Aquilanti) Date: Thu, 16 Jun 2011 13:35:05 +0200 Subject: [petsc-users] cholesky factorization Memory regions overlap Message-ID: <201106161335.06093.py.aquilanti@gmail.com> Thank you Jed ! The problem was the 0 in the diagonal, which confirmed a matlab spy plot. I introduced this error because i'm actually converting an old fortran code to PETSc. Thanks again for your help :) From mirzadeh at gmail.com Mon Jun 20 16:31:54 2011 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Mon, 20 Jun 2011 14:31:54 -0700 Subject: [petsc-users] MatSetValues and Zero enteries In-Reply-To: References: Message-ID: Thanks John -- taht's helpful. Do you also know how I can change the tolerance for dropping elements? Mohammad On Mon, Jun 20, 2011 at 1:12 PM, John Mousel wrote: > See the matrix options available in the man pages. I think the function > below is what you are looking for. > > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatSetOption.html > > > MatSetOption (A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE ); > > > > Best Regards, > John > > > > > On Mon, Jun 20, 2011 at 2:42 PM, Mohammad Mirzadeh wrote: > >> Hi folks, >> >> I've noticed that MatSetValues explicitly inserts zeros in the matrix if >> an element is really zero. Is there a way to drop elements (or not save them >> in the first place) that are below a certain tolerance? I realize one way of >> doing it is in the application code and when using MatSetValues but I'm just >> wondering if PETSc has a function for it? >> >> Thanks, >> Mohammad >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jun 20 16:34:05 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 20 Jun 2011 21:34:05 +0000 Subject: [petsc-users] MatSetValues and Zero enteries In-Reply-To: References: Message-ID: On Mon, Jun 20, 2011 at 9:31 PM, Mohammad Mirzadeh wrote: > Thanks John -- taht's helpful. Do you also know how I can change the > tolerance for dropping elements? > Right now, this only discards entires which are exactly floating point 0.0. Matt > Mohammad > > On Mon, Jun 20, 2011 at 1:12 PM, John Mousel wrote: > >> See the matrix options available in the man pages. I think the function >> below is what you are looking for. >> >> >> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatSetOption.html >> >> >> MatSetOption (A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE ); >> >> >> >> >> Best Regards, >> John >> >> >> >> >> On Mon, Jun 20, 2011 at 2:42 PM, Mohammad Mirzadeh wrote: >> >>> Hi folks, >>> >>> I've noticed that MatSetValues explicitly inserts zeros in the matrix if >>> an element is really zero. Is there a way to drop elements (or not save them >>> in the first place) that are below a certain tolerance? I realize one way of >>> doing it is in the application code and when using MatSetValues but I'm just >>> wondering if PETSc has a function for it? >>> >>> Thanks, >>> Mohammad >>> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From rongtian at ncic.ac.cn Mon Jun 20 16:40:42 2011 From: rongtian at ncic.ac.cn (Tian(ICT)) Date: Tue, 21 Jun 2011 05:40:42 +0800 Subject: [petsc-users] MSVC debug runs in a different branch. In-Reply-To: <201106161335.06093.py.aquilanti@gmail.com> References: <201106161335.06093.py.aquilanti@gmail.com> Message-ID: Hi, Good morning/evening! I have this issue for quite a while. Using MSVC with Petsc, when debuging the code, the FormJacobian was never called, instead calling "MatMFFDComputeJacobian(SNES snes,Vec x,Mat *jac,Mat *B,MatStructure *flag,void *dummy)", though I donot use matrix free option. However, when runing the same thing in command line (still a debug compilation), I can see FormJacobian was called correctly. Any hint? Rong From jed at 59A2.org Mon Jun 20 16:48:22 2011 From: jed at 59A2.org (Jed Brown) Date: Mon, 20 Jun 2011 23:48:22 +0200 Subject: [petsc-users] MSVC debug runs in a different branch. In-Reply-To: References: <201106161335.06093.py.aquilanti@gmail.com> Message-ID: On Mon, Jun 20, 2011 at 23:40, Tian(ICT) wrote: > Using MSVC with Petsc, when debuging the code, the FormJacobian was never > called, instead calling "MatMFFDComputeJacobian(SNES snes,Vec x,Mat *jac,Mat > *B,MatStructure *flag,void *dummy)", though I donot use matrix free option. Is there something in the environment, in a ".petscrc" in the directory MSVC is running it in, or in an extra command-line parameter that MSVC is implicitly passing? If you can you use a command-line debugger like gdb, then you can set a breakpoint in SNESSetJacobian() and see where it's getting changed from. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jun 20 16:46:12 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 20 Jun 2011 21:46:12 +0000 Subject: [petsc-users] MSVC debug runs in a different branch. In-Reply-To: References: <201106161335.06093.py.aquilanti@gmail.com> Message-ID: On Mon, Jun 20, 2011 at 9:40 PM, Tian(ICT) wrote: > Hi, > Good morning/evening! > I have this issue for quite a while. Using MSVC with Petsc, when debuging > the code, the FormJacobian was never called, instead calling > "MatMFFDComputeJacobian(SNES snes,Vec x,Mat *jac,Mat *B,MatStructure > *flag,void *dummy)", though I donot use matrix free option. > > However, when runing the same thing in command line (still a debug > compilation), I can see FormJacobian was called correctly. > > Any hint? > My guess is that you are getting the matrix-free option from somewhere (maybe a .petscrc file?). You can use -options_dump to see everything passed in. If not, look at SNESSetFromOptions() in the debugger, and see why it is choosing MF. Matt > Rong > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From rongtian at ncic.ac.cn Mon Jun 20 17:19:27 2011 From: rongtian at ncic.ac.cn (Tian(ICT)) Date: Tue, 21 Jun 2011 06:19:27 +0800 Subject: [petsc-users] MSVC debug runs in a different branch. In-Reply-To: References: <201106161335.06093.py.aquilanti@gmail.com> Message-ID: <210DA4F3867C42DDA8AB2947C8901038@rti458laptop> Thanks for quick answering. I looked into SNESSetFromOptions() in the debugger and MF was not chosen. Still the wired thing is the code never breaks at the point I set inside FormJacobian when running in the debug mode. I will leave this issue alone again and look at it later. Actually I am struggling with a SNES solve. The followings are some outputs. -snes_ratiomonitor shows that residual norm almost equals to one but the SNES converged, is this a good convergence? atol=1e-050, rtol=1e-008, stol=1e-008, maxit=50, maxf=10000 function................. residual force................. 0 0 0 0 0 0 0 0 0 0 0 0 0 40000 <== initial value 0 0 40000 0 0 40000 0 0 40000 0 0 SNES Function norm 8.000000000000e+004 0 SNES Function norm 8.000000000000e+004 ------------------->Jacobian function................. residual force................. -8.2245e+007 -5.12049e+006 1.03104e+008 2.44985e+007 -1.11118e+007 -3.144e+007 -2.37583e+007 -535905 2.79206e+007 5.63514e+007 -2.26332e+007 -7.54711e+007 -5.76723e+007 1.544e+007 7.56104e+007 2.34587e+007 9.57426e+006 -2.56718e+007 2.7089e+007 3.32996e+007 -3.09772e+007 3.2278e+007 -1.87524e+007 -4.30752e+007 function................. residual force................. -8.2245e+007 -5.12049e+006 1.03104e+008 2.44985e+007 -1.11118e+007 -3.144e+007 -2.37583e+007 -535905 2.79206e+007 5.63514e+007 -2.26332e+007 -7.54711e+007 -5.76723e+007 1.544e+007 7.56104e+007 2.34587e+007 9.57426e+006 -2.56718e+007 2.7089e+007 3.32996e+007 -3.09772e+007 3.2278e+007 -1.87524e+007 -4.30752e+007 function................. residual force................. -2.05272e+007 3.44926e+006 2.5202e+007 -1.55108e+006 -9.56839e+006 2.17527e+006 3.56516e+006 6.12324e+006 -5.9437e+006 1.41997e+007 -1.21796e+007 -1.92224e+007 -1.63165e+007 1.00523e+007 2.14974e+007 -1.87655e+006 -3.44175e+006 4.3047e+006 1.50918e+007 1.66532e+007 -1.8379e+007 7.41472e+006 -1.09283e+007 -9.63423e+006 function................. residual force................. -4.83477e+006 1.9056e+006 5.06631e+006 -1.74833e+006 -3.81725e+006 3.0505e+006 1.93196e+006 2.614e+006 -3.9812e+006 3.71686e+006 -4.00712e+006 -4.46187e+006 -4.24801e+006 3.70415e+006 5.07447e+006 -1.59935e+006 -2.18778e+006 3.63332e+006 3.74435e+006 5.0913e+006 -5.46539e+006 3.03729e+006 -3.14292e+006 -2.91613e+006 function................. residual force................. -1.62518e+006 635442 1.14687e+006 -395026 -1.35624e+006 1.30091e+006 258576 922003 -1.54824e+006 1.34306e+006 -1.18937e+006 -1.19905e+006 -1.44957e+006 1.18631e+006 1.32948e+006 -233476 -848651 1.52147e+006 630822 1.59179e+006 -1.62993e+006 1.4708e+006 -781281 -921506 function................. residual force................. -680584 150805 298173 -4235.39 -556100 495315 -83367.9 401881 -609708 583119 -325632 -382676 -608984 356434 416726 79947.2 -362413 611645 24823.1 640620 -552863 689283 -145594 -276612 function................. residual force................. -252778 -43702.9 107996 39434.6 -252471 108972 -56021.1 202917 -167131 237714 -13235.2 -160064 -245478 46386.1 170214 53958.6 -164661 167576 -41053 309826 -119435 264223 74941.4 -108128 function................. residual force................. -37722.5 -30354.3 12055.9 19055.1 -45688.3 -1869.42 -17961.6 41506 -11175 39667.1 24371.9 -24644.8 -40629.2 12856 25503.9 17749.7 -1976.73 10689.7 -20792.3 89625.4 2237.53 40633.6 69659.9 -12797.9 function................. residual force................. -3191.46 -3795.77 508.943 2612.15 -4047.01 -1039.77 -2394.34 3957.38 -341.114 3526.13 3531.34 -1843.35 -3574.59 36099.4 1775.2 2391.73 35967.4 179.561 -2883.93 44445.2 1264.25 3514.31 43842.1 -503.71 function................. residual force................. -268.146 -347.447 46.5142 257.768 -335.67 -97.6787 -234.991 341.388 -35.3256 299.924 338.987 -161.929 -306.308 39615 132.745 236.496 39642.5 -7.34212 -285.009 40378.6 141.125 300.268 40366.6 -18.108 function................. residual force................. -22.758 -27.2585 11.605 23.791 -26.2078 -3.38911 -21.3068 28.26 -10.762 25.2783 28.3454 -19.8467 -27.0516 39963.9 11.1365 21.8233 39967.5 -1.27698 -26.2771 40032.7 13.3758 26.5009 40032.8 -0.842484 1 SNES Function norm 7.999847670241e+004 1 SNES Function norm 7.999847670241e+004 0.999981 ------------------->Jacobian function................. residual force................. -223.361 -326.156 36.1377 229.267 -321.922 -82.1057 -210.332 329.183 -52.6793 249.815 322.469 -153.907 -255.672 39637.8 127.173 211.072 39655.1 12.6163 -250.93 40359 119.951 250.141 40344.5 -7.18602 function................. residual force................. -69.9794 -97.0147 18.7512 72.7704 -94.1566 -22.1285 -66.1397 98.02 -20.4693 78.3769 97.6066 -52.1529 -81.3498 39886.4 37.8587 66.7162 39894.4 0.222982 -79.8688 40108.4 40.0624 79.4743 40106.3 -2.14464 2 SNES Function norm 7.999823436070e+004 2 SNES Function norm 7.999823436070e+004 0.999997 ------------------->Jacobian function................. residual force................. -85.9273 -122.698 20.3604 89.7785 -119.971 -28.2927 -81.9324 124.468 -25.2852 96.1213 123.187 -63.7311 -99.4852 39858.1 47.741 82.4929 39866.9 2.51439 -98.3175 40136.8 48.6046 97.2698 40133.2 <== almost no change -1.91145 3 SNES Function norm 7.999822446658e+004 3 SNES Function norm 7.999822446658e+004 1 ------------------->Jacobian function................. residual force................. -90.9631 -130.918 20.7647 95.1669 -128.284 -30.1982 -86.9654 132.973 -26.9162 101.696 131.372 -67.3645 -105.178 39849.1 50.8645 87.519 39858 3.38427 -104.154 40145.9 51.2208 102.878 40141.8 -1.7555 function................. residual force................. -87.4304 -125.151 20.4826 91.387 -122.451 -28.8623 -83.4344 127.006 -25.7711 97.7856 125.63 -64.8168 -101.185 39855.4 48.6736 83.9928 39864.2 2.77216 -100.06 40139.5 49.3875 98.944 40135.8 -1.86551 function................. residual force................. -86.3776 -123.432 20.3972 90.2603 -120.714 -28.4634 -82.3823 125.228 -25.4307 96.6198 123.919 -64.0564 -99.9944 39857.3 48.0204 82.9421 39866.1 2.59144 -98.8394 40137.6 48.8393 97.7713 40134 -1.89774 function................. residual force................. -86.0623 -122.918 20.3714 89.923 -120.194 -28.3439 -82.0673 124.696 -25.3288 96.2708 123.407 -63.8287 -99.6379 39857.9 47.8248 82.6276 39866.6 2.53748 -98.474 40137.1 48.675 97.4202 40133.5 -1.90734 function................. residual force................. -85.9678 -122.764 20.3637 89.8218 -120.038 -28.3081 -81.9729 124.536 -25.2983 96.1661 123.253 -63.7604 -99.531 39858 47.7661 82.5333 39866.8 2.52132 -98.3644 40136.9 48.6258 97.3149 40133.3 -1.91022 function................. residual force................. -85.9395 -122.717 20.3614 89.7915 -119.991 -28.2973 -81.9445 124.489 -25.2891 96.1347 123.207 -63.7399 -99.4989 39858.1 47.7485 82.505 39866.9 2.51647 -98.3316 40136.8 48.611 97.2833 40133.2 -1.91108 function................. residual force................. -85.931 -122.704 20.3607 89.7824 -119.977 -28.2941 -81.936 124.474 -25.2863 96.1253 123.193 -63.7337 -99.4893 39858.1 47.7432 82.4965 39866.9 2.51502 -98.3217 40136.8 48.6065 97.2738 40133.2 -1.91134 function................. residual force................. -85.9284 -122.699 20.3605 89.7796 -119.973 -28.2931 -81.9335 124.47 -25.2855 96.1225 123.189 -63.7319 -99.4864 39858.1 47.7416 82.494 39866.9 2.51458 -98.3187 40136.8 48.6052 97.271 40133.2 -1.91142 function................. residual force................. -85.9277 -122.698 20.3604 89.7788 -119.972 -28.2928 -81.9327 124.469 -25.2853 96.1216 123.188 -63.7313 -99.4856 39858.1 47.7412 82.4932 39866.9 2.51445 -98.3179 40136.8 48.6048 97.2702 40133.2 -1.91144 function................. residual force................. -85.9274 -122.698 20.3604 89.7786 -119.972 -28.2927 -81.9325 124.468 -25.2852 96.1214 123.187 -63.7312 -99.4853 39858.1 47.741 82.493 39866.9 2.51441 -98.3176 40136.8 48.6047 97.2699 40133.2 -1.91145 function................. residual force................. -85.9274 -122.698 20.3604 89.7785 -119.971 -28.2927 -81.9324 124.468 -25.2852 96.1213 123.187 -63.7311 -99.4852 39858.1 47.741 82.4929 39866.9 2.5144 -98.3175 40136.8 48.6047 97.2698 40133.2 -1.91145 function................. residual force................. -85.9273 -122.698 20.3604 89.7785 -119.971 -28.2927 -81.9324 124.468 -25.2852 96.1213 123.187 -63.7311 -99.4852 39858.1 47.741 82.4929 39866.9 2.5144 -98.3175 40136.8 48.6046 97.2698 40133.2 -1.91145 function................. residual force................. -85.9273 -122.698 20.3604 89.7785 -119.971 -28.2927 -81.9324 124.468 -25.2852 96.1213 123.187 -63.7311 -99.4852 39858.1 47.741 82.4929 39866.9 2.5144 -98.3175 40136.8 48.6046 97.2698 40133.2 -1.91145 function................. residual force................. -85.9273 -122.698 20.3604 89.7785 -119.971 -28.2927 -81.9324 124.468 -25.2852 96.1213 123.187 -63.7311 -99.4852 39858.1 47.741 82.4929 39866.9 2.51439 -98.3175 40136.8 48.6046 97.2698 40133.2 -1.91145 function................. residual force................. -85.9273 -122.698 20.3604 89.7785 -119.971 -28.2927 -81.9324 124.468 -25.2852 96.1213 123.187 -63.7311 -99.4852 39858.1 47.741 82.4929 39866.9 2.51439 -98.3175 40136.8 48.6046 97.2698 40133.2 -1.91145 function................. residual force................. -85.9273 -122.698 20.3604 89.7785 -119.971 -28.2927 -81.9324 124.468 -25.2852 96.1213 123.187 -63.7311 -99.4852 39858.1 47.741 82.4929 39866.9 2.51439 -98.3175 40136.8 48.6046 97.2698 40133.2 -1.91145 function................. residual force................. -85.9273 -122.698 20.3604 89.7785 -119.971 -28.2927 -81.9324 124.468 -25.2852 96.1213 123.187 -63.7311 -99.4852 39858.1 47.741 82.4929 39866.9 2.51439 -98.3175 40136.8 48.6046 97.2698 40133.2 -1.91145 function................. residual force................. -85.9273 -122.698 20.3604 89.7785 -119.971 -28.2927 -81.9324 124.468 -25.2852 96.1213 123.187 -63.7311 -99.4852 39858.1 47.741 82.4929 39866.9 2.51439 -98.3175 40136.8 48.6046 97.2698 40133.2 -1.91145 function................. residual force................. -85.9273 -122.698 20.3604 89.7785 -119.971 -28.2927 -81.9324 124.468 -25.2852 96.1213 123.187 -63.7311 -99.4852 39858.1 47.741 82.4929 39866.9 2.51439 -98.3175 40136.8 48.6046 97.2698 40133.2 -1.91145 function................. residual force................. -85.9273 -122.698 20.3604 89.7785 -119.971 -28.2927 -81.9324 124.468 -25.2852 96.1213 123.187 -63.7311 -99.4852 39858.1 47.741 82.4929 39866.9 2.51439 -98.3175 40136.8 48.6046 97.2698 40133.2 -1.91145 function................. residual force................. -85.9273 -122.698 20.3604 89.7785 -119.971 -28.2927 -81.9324 124.468 -25.2852 96.1213 123.187 -63.7311 -99.4852 39858.1 47.741 82.4929 39866.9 2.51439 -98.3175 40136.8 48.6046 97.2698 40133.2 -1.91145 4 SNES Function norm 7.999822446658e+004 4 SNES Function norm 7.999822446658e+004 1 SNES Object: type: tr mu=0.25, eta=0.75, sigma=0.0001 delta0=0.2, delta1=0.3, delta2=0.75, delta3=2 maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-008, absolute=1e-050, solution=1e-008 total number of linear solver iterations=4 total number of function evaluations=36 KSP Object: type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-030 maximum iterations=10000, initial guess is zero tolerances: relative=1e-005, absolute=1e-050, divergence=10000 left preconditioning PC Object: type: ilu ILU: 0 levels of fill ILU: factor fill ratio allocated 1 ILU: tolerance for zero pivot 1e-012 out-of-place factorization matrix ordering: natural ILU: factor fill ratio needed 1 Factored matrix follows Matrix Object: type=seqaij, rows=24, cols=24 total: nonzeros=300, allocated nonzeros=840 using I-node routines: found 15 nodes, limit used is 5 linear system matrix = precond matrix: Matrix Object: type=seqaij, rows=24, cols=24 total: nonzeros=300, allocated nonzeros=1920 using I-node routines: found 15 nodes, limit used is 5 Nonlinear solve converged due to CONVERGED_TR_DELTA STEP 0 (Newton iterations: 3) WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-options_dump no value ----- Original Message ----- From: Matthew Knepley To: PETSc users list Sent: Tuesday, June 21, 2011 5:46 AM Subject: Re: [petsc-users] MSVC debug runs in a different branch. On Mon, Jun 20, 2011 at 9:40 PM, Tian(ICT) wrote: Hi, Good morning/evening! I have this issue for quite a while. Using MSVC with Petsc, when debuging the code, the FormJacobian was never called, instead calling "MatMFFDComputeJacobian(SNES snes,Vec x,Mat *jac,Mat *B,MatStructure *flag,void *dummy)", though I donot use matrix free option. However, when runing the same thing in command line (still a debug compilation), I can see FormJacobian was called correctly. Any hint? My guess is that you are getting the matrix-free option from somewhere (maybe a .petscrc file?). You can use -options_dump to see everything passed in. If not, look at SNESSetFromOptions() in the debugger, and see why it is choosing MF. Matt Rong -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jun 20 17:40:37 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 20 Jun 2011 22:40:37 +0000 Subject: [petsc-users] MSVC debug runs in a different branch. In-Reply-To: <210DA4F3867C42DDA8AB2947C8901038@rti458laptop> References: <201106161335.06093.py.aquilanti@gmail.com> <210DA4F3867C42DDA8AB2947C8901038@rti458laptop> Message-ID: On Mon, Jun 20, 2011 at 10:19 PM, Tian(ICT) wrote: > ** > Thanks for quick answering. > I looked into SNESSetFromOptions() in the debugger and MF was not chosen. > Still the wired thing is the code never breaks at the point I set inside > FormJacobian when running in the debug mode. > I will leave this issue alone again and look at it later. > > Actually I am struggling with a SNES solve. The followings are some > outputs. > -snes_ratiomonitor shows that residual norm almost equals to one but the > SNES converged, > is this a good convergence? > No. Are you sure this system is consistent, meaning it has a solution? Have you tried putting in an exact solution for a simpler problem? Matt > > atol=1e-050, rtol=1e-008, stol=1e-008, maxit=50, maxf=10000 > function................. > residual force................. > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 40000 <== initial value > 0 > 0 > 40000 > 0 > 0 > 40000 > 0 > 0 > 40000 > 0 > 0 SNES Function norm 8.000000000000e+004 > 0 SNES Function norm 8.000000000000e+004 > ------------------->Jacobian > function................. > residual force................. > -8.2245e+007 > -5.12049e+006 > 1.03104e+008 > 2.44985e+007 > -1.11118e+007 > -3.144e+007 > -2.37583e+007 > -535905 > 2.79206e+007 > 5.63514e+007 > -2.26332e+007 > -7.54711e+007 > -5.76723e+007 > 1.544e+007 > 7.56104e+007 > 2.34587e+007 > 9.57426e+006 > -2.56718e+007 > 2.7089e+007 > 3.32996e+007 > -3.09772e+007 > 3.2278e+007 > -1.87524e+007 > -4.30752e+007 > function................. > residual force................. > -8.2245e+007 > -5.12049e+006 > 1.03104e+008 > 2.44985e+007 > -1.11118e+007 > -3.144e+007 > -2.37583e+007 > -535905 > 2.79206e+007 > 5.63514e+007 > -2.26332e+007 > -7.54711e+007 > -5.76723e+007 > 1.544e+007 > 7.56104e+007 > 2.34587e+007 > 9.57426e+006 > -2.56718e+007 > 2.7089e+007 > 3.32996e+007 > -3.09772e+007 > 3.2278e+007 > -1.87524e+007 > -4.30752e+007 > function................. > residual force................. > -2.05272e+007 > 3.44926e+006 > 2.5202e+007 > -1.55108e+006 > -9.56839e+006 > 2.17527e+006 > 3.56516e+006 > 6.12324e+006 > -5.9437e+006 > 1.41997e+007 > -1.21796e+007 > -1.92224e+007 > -1.63165e+007 > 1.00523e+007 > 2.14974e+007 > -1.87655e+006 > -3.44175e+006 > 4.3047e+006 > 1.50918e+007 > 1.66532e+007 > -1.8379e+007 > 7.41472e+006 > -1.09283e+007 > -9.63423e+006 > function................. > residual force................. > -4.83477e+006 > 1.9056e+006 > 5.06631e+006 > -1.74833e+006 > -3.81725e+006 > 3.0505e+006 > 1.93196e+006 > 2.614e+006 > -3.9812e+006 > 3.71686e+006 > -4.00712e+006 > -4.46187e+006 > -4.24801e+006 > 3.70415e+006 > 5.07447e+006 > -1.59935e+006 > -2.18778e+006 > 3.63332e+006 > 3.74435e+006 > 5.0913e+006 > -5.46539e+006 > 3.03729e+006 > -3.14292e+006 > -2.91613e+006 > function................. > residual force................. > -1.62518e+006 > 635442 > 1.14687e+006 > -395026 > -1.35624e+006 > 1.30091e+006 > 258576 > 922003 > -1.54824e+006 > 1.34306e+006 > -1.18937e+006 > -1.19905e+006 > -1.44957e+006 > 1.18631e+006 > 1.32948e+006 > -233476 > -848651 > 1.52147e+006 > 630822 > 1.59179e+006 > -1.62993e+006 > 1.4708e+006 > -781281 > -921506 > function................. > residual force................. > -680584 > 150805 > 298173 > -4235.39 > -556100 > 495315 > -83367.9 > 401881 > -609708 > 583119 > -325632 > -382676 > -608984 > 356434 > 416726 > 79947.2 > -362413 > 611645 > 24823.1 > 640620 > -552863 > 689283 > -145594 > -276612 > function................. > residual force................. > -252778 > -43702.9 > 107996 > 39434.6 > -252471 > 108972 > -56021.1 > 202917 > -167131 > 237714 > -13235.2 > -160064 > -245478 > 46386.1 > 170214 > 53958.6 > -164661 > 167576 > -41053 > 309826 > -119435 > 264223 > 74941.4 > -108128 > function................. > residual force................. > -37722.5 > -30354.3 > 12055.9 > 19055.1 > -45688.3 > -1869.42 > -17961.6 > 41506 > -11175 > 39667.1 > 24371.9 > -24644.8 > -40629.2 > 12856 > 25503.9 > 17749.7 > -1976.73 > 10689.7 > -20792.3 > 89625.4 > 2237.53 > 40633.6 > 69659.9 > -12797.9 > function................. > residual force................. > -3191.46 > -3795.77 > 508.943 > 2612.15 > -4047.01 > -1039.77 > -2394.34 > 3957.38 > -341.114 > 3526.13 > 3531.34 > -1843.35 > -3574.59 > 36099.4 > 1775.2 > 2391.73 > 35967.4 > 179.561 > -2883.93 > 44445.2 > 1264.25 > 3514.31 > 43842.1 > -503.71 > function................. > residual force................. > -268.146 > -347.447 > 46.5142 > 257.768 > -335.67 > -97.6787 > -234.991 > 341.388 > -35.3256 > 299.924 > 338.987 > -161.929 > -306.308 > 39615 > 132.745 > 236.496 > 39642.5 > -7.34212 > -285.009 > 40378.6 > 141.125 > 300.268 > 40366.6 > -18.108 > function................. > residual force................. > -22.758 > -27.2585 > 11.605 > 23.791 > -26.2078 > -3.38911 > -21.3068 > 28.26 > -10.762 > 25.2783 > 28.3454 > -19.8467 > -27.0516 > 39963.9 > 11.1365 > 21.8233 > 39967.5 > -1.27698 > -26.2771 > 40032.7 > 13.3758 > 26.5009 > 40032.8 > -0.842484 > 1 SNES Function norm 7.999847670241e+004 > 1 SNES Function norm 7.999847670241e+004 0.999981 > ------------------->Jacobian > function................. > residual force................. > -223.361 > -326.156 > 36.1377 > 229.267 > -321.922 > -82.1057 > -210.332 > 329.183 > -52.6793 > 249.815 > 322.469 > -153.907 > -255.672 > 39637.8 > 127.173 > 211.072 > 39655.1 > 12.6163 > -250.93 > 40359 > 119.951 > 250.141 > 40344.5 > -7.18602 > function................. > residual force................. > -69.9794 > -97.0147 > 18.7512 > 72.7704 > -94.1566 > -22.1285 > -66.1397 > 98.02 > -20.4693 > 78.3769 > 97.6066 > -52.1529 > -81.3498 > 39886.4 > 37.8587 > 66.7162 > 39894.4 > 0.222982 > -79.8688 > 40108.4 > 40.0624 > 79.4743 > 40106.3 > -2.14464 > 2 SNES Function norm 7.999823436070e+004 > 2 SNES Function norm 7.999823436070e+004 0.999997 > ------------------->Jacobian > function................. > residual force................. > -85.9273 > -122.698 > 20.3604 > 89.7785 > -119.971 > -28.2927 > -81.9324 > 124.468 > -25.2852 > 96.1213 > 123.187 > -63.7311 > -99.4852 > 39858.1 > 47.741 > 82.4929 > 39866.9 > 2.51439 > -98.3175 > 40136.8 > 48.6046 > 97.2698 > 40133.2 <== almost no change > -1.91145 > 3 SNES Function norm 7.999822446658e+004 > 3 SNES Function norm 7.999822446658e+004 1 > ------------------->Jacobian > function................. > residual force................. > -90.9631 > -130.918 > 20.7647 > 95.1669 > -128.284 > -30.1982 > -86.9654 > 132.973 > -26.9162 > 101.696 > 131.372 > -67.3645 > -105.178 > 39849.1 > 50.8645 > 87.519 > 39858 > 3.38427 > -104.154 > 40145.9 > 51.2208 > 102.878 > 40141.8 > -1.7555 > function................. > residual force................. > -87.4304 > -125.151 > 20.4826 > 91.387 > -122.451 > -28.8623 > -83.4344 > 127.006 > -25.7711 > 97.7856 > 125.63 > -64.8168 > -101.185 > 39855.4 > 48.6736 > 83.9928 > 39864.2 > 2.77216 > -100.06 > 40139.5 > 49.3875 > 98.944 > 40135.8 > -1.86551 > function................. > residual force................. > -86.3776 > -123.432 > 20.3972 > 90.2603 > -120.714 > -28.4634 > -82.3823 > 125.228 > -25.4307 > 96.6198 > 123.919 > -64.0564 > -99.9944 > 39857.3 > 48.0204 > 82.9421 > 39866.1 > 2.59144 > -98.8394 > 40137.6 > 48.8393 > 97.7713 > 40134 > -1.89774 > function................. > residual force................. > -86.0623 > -122.918 > 20.3714 > 89.923 > -120.194 > -28.3439 > -82.0673 > 124.696 > -25.3288 > 96.2708 > 123.407 > -63.8287 > -99.6379 > 39857.9 > 47.8248 > 82.6276 > 39866.6 > 2.53748 > -98.474 > 40137.1 > 48.675 > 97.4202 > 40133.5 > -1.90734 > function................. > residual force................. > -85.9678 > -122.764 > 20.3637 > 89.8218 > -120.038 > -28.3081 > -81.9729 > 124.536 > -25.2983 > 96.1661 > 123.253 > -63.7604 > -99.531 > 39858 > 47.7661 > 82.5333 > 39866.8 > 2.52132 > -98.3644 > 40136.9 > 48.6258 > 97.3149 > 40133.3 > -1.91022 > function................. > residual force................. > -85.9395 > -122.717 > 20.3614 > 89.7915 > -119.991 > -28.2973 > -81.9445 > 124.489 > -25.2891 > 96.1347 > 123.207 > -63.7399 > -99.4989 > 39858.1 > 47.7485 > 82.505 > 39866.9 > 2.51647 > -98.3316 > 40136.8 > 48.611 > 97.2833 > 40133.2 > -1.91108 > function................. > residual force................. > -85.931 > -122.704 > 20.3607 > 89.7824 > -119.977 > -28.2941 > -81.936 > 124.474 > -25.2863 > 96.1253 > 123.193 > -63.7337 > -99.4893 > 39858.1 > 47.7432 > 82.4965 > 39866.9 > 2.51502 > -98.3217 > 40136.8 > 48.6065 > 97.2738 > 40133.2 > -1.91134 > function................. > residual force................. > -85.9284 > -122.699 > 20.3605 > 89.7796 > -119.973 > -28.2931 > -81.9335 > 124.47 > -25.2855 > 96.1225 > 123.189 > -63.7319 > -99.4864 > 39858.1 > 47.7416 > 82.494 > 39866.9 > 2.51458 > -98.3187 > 40136.8 > 48.6052 > 97.271 > 40133.2 > -1.91142 > function................. > residual force................. > -85.9277 > -122.698 > 20.3604 > 89.7788 > -119.972 > -28.2928 > -81.9327 > 124.469 > -25.2853 > 96.1216 > 123.188 > -63.7313 > -99.4856 > 39858.1 > 47.7412 > 82.4932 > 39866.9 > 2.51445 > -98.3179 > 40136.8 > 48.6048 > 97.2702 > 40133.2 > -1.91144 > function................. > residual force................. > -85.9274 > -122.698 > 20.3604 > 89.7786 > -119.972 > -28.2927 > -81.9325 > 124.468 > -25.2852 > 96.1214 > 123.187 > -63.7312 > -99.4853 > 39858.1 > 47.741 > 82.493 > 39866.9 > 2.51441 > -98.3176 > 40136.8 > 48.6047 > 97.2699 > 40133.2 > -1.91145 > function................. > residual force................. > -85.9274 > -122.698 > 20.3604 > 89.7785 > -119.971 > -28.2927 > -81.9324 > 124.468 > -25.2852 > 96.1213 > 123.187 > -63.7311 > -99.4852 > 39858.1 > 47.741 > 82.4929 > 39866.9 > 2.5144 > -98.3175 > 40136.8 > 48.6047 > 97.2698 > 40133.2 > -1.91145 > function................. > residual force................. > -85.9273 > -122.698 > 20.3604 > 89.7785 > -119.971 > -28.2927 > -81.9324 > 124.468 > -25.2852 > 96.1213 > 123.187 > -63.7311 > -99.4852 > 39858.1 > 47.741 > 82.4929 > 39866.9 > 2.5144 > -98.3175 > 40136.8 > 48.6046 > 97.2698 > 40133.2 > -1.91145 > function................. > residual force................. > -85.9273 > -122.698 > 20.3604 > 89.7785 > -119.971 > -28.2927 > -81.9324 > 124.468 > -25.2852 > 96.1213 > 123.187 > -63.7311 > -99.4852 > 39858.1 > 47.741 > 82.4929 > 39866.9 > 2.5144 > -98.3175 > 40136.8 > 48.6046 > 97.2698 > 40133.2 > -1.91145 > function................. > residual force................. > -85.9273 > -122.698 > 20.3604 > 89.7785 > -119.971 > -28.2927 > -81.9324 > 124.468 > -25.2852 > 96.1213 > 123.187 > -63.7311 > -99.4852 > 39858.1 > 47.741 > 82.4929 > 39866.9 > 2.51439 > -98.3175 > 40136.8 > 48.6046 > 97.2698 > 40133.2 > -1.91145 > function................. > residual force................. > -85.9273 > -122.698 > 20.3604 > 89.7785 > -119.971 > -28.2927 > -81.9324 > 124.468 > -25.2852 > 96.1213 > 123.187 > -63.7311 > -99.4852 > 39858.1 > 47.741 > 82.4929 > 39866.9 > 2.51439 > -98.3175 > 40136.8 > 48.6046 > 97.2698 > 40133.2 > -1.91145 > function................. > residual force................. > -85.9273 > -122.698 > 20.3604 > 89.7785 > -119.971 > -28.2927 > -81.9324 > 124.468 > -25.2852 > 96.1213 > 123.187 > -63.7311 > -99.4852 > 39858.1 > 47.741 > 82.4929 > 39866.9 > 2.51439 > -98.3175 > 40136.8 > 48.6046 > 97.2698 > 40133.2 > -1.91145 > function................. > residual force................. > -85.9273 > -122.698 > 20.3604 > 89.7785 > -119.971 > -28.2927 > -81.9324 > 124.468 > -25.2852 > 96.1213 > 123.187 > -63.7311 > -99.4852 > 39858.1 > 47.741 > 82.4929 > 39866.9 > 2.51439 > -98.3175 > 40136.8 > 48.6046 > 97.2698 > 40133.2 > -1.91145 > function................. > residual force................. > -85.9273 > -122.698 > 20.3604 > 89.7785 > -119.971 > -28.2927 > -81.9324 > 124.468 > -25.2852 > 96.1213 > 123.187 > -63.7311 > -99.4852 > 39858.1 > 47.741 > 82.4929 > 39866.9 > 2.51439 > -98.3175 > 40136.8 > 48.6046 > 97.2698 > 40133.2 > -1.91145 > function................. > residual force................. > -85.9273 > -122.698 > 20.3604 > 89.7785 > -119.971 > -28.2927 > -81.9324 > 124.468 > -25.2852 > 96.1213 > 123.187 > -63.7311 > -99.4852 > 39858.1 > 47.741 > 82.4929 > 39866.9 > 2.51439 > -98.3175 > 40136.8 > 48.6046 > 97.2698 > 40133.2 > -1.91145 > function................. > residual force................. > -85.9273 > -122.698 > 20.3604 > 89.7785 > -119.971 > -28.2927 > -81.9324 > 124.468 > -25.2852 > 96.1213 > 123.187 > -63.7311 > -99.4852 > 39858.1 > 47.741 > 82.4929 > 39866.9 > 2.51439 > -98.3175 > 40136.8 > 48.6046 > 97.2698 > 40133.2 > -1.91145 > function................. > residual force................. > -85.9273 > -122.698 > 20.3604 > 89.7785 > -119.971 > -28.2927 > -81.9324 > 124.468 > -25.2852 > 96.1213 > 123.187 > -63.7311 > -99.4852 > 39858.1 > 47.741 > 82.4929 > 39866.9 > 2.51439 > -98.3175 > 40136.8 > 48.6046 > 97.2698 > 40133.2 > -1.91145 > 4 SNES Function norm 7.999822446658e+004 > 4 SNES Function norm 7.999822446658e+004 1 > SNES Object: > type: tr > mu=0.25, eta=0.75, sigma=0.0001 > delta0=0.2, delta1=0.3, delta2=0.75, delta3=2 > maximum iterations=50, maximum function evaluations=10000 > tolerances: relative=1e-008, absolute=1e-050, solution=1e-008 > total number of linear solver iterations=4 > total number of function evaluations=36 > KSP Object: > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-030 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-005, absolute=1e-050, divergence=10000 > left preconditioning > PC Object: > type: ilu > ILU: 0 levels of fill > ILU: factor fill ratio allocated 1 > ILU: tolerance for zero pivot 1e-012 > out-of-place factorization > matrix ordering: natural > ILU: factor fill ratio needed 1 > Factored matrix follows > Matrix Object: > type=seqaij, rows=24, cols=24 > total: nonzeros=300, allocated nonzeros=840 > using I-node routines: found 15 nodes, limit used is 5 > linear system matrix = precond matrix: > Matrix Object: > type=seqaij, rows=24, cols=24 > total: nonzeros=300, allocated nonzeros=1920 > using I-node routines: found 15 nodes, limit used is 5 > Nonlinear solve converged due to CONVERGED_TR_DELTA > STEP 0 (Newton iterations: 3) > > WARNING! There are options you set that were not used! > WARNING! could be spelling mistake, etc! > Option left: name:-options_dump no value > > > ----- Original Message ----- > *From:* Matthew Knepley > *To:* PETSc users list > *Sent:* Tuesday, June 21, 2011 5:46 AM > *Subject:* Re: [petsc-users] MSVC debug runs in a different branch. > > On Mon, Jun 20, 2011 at 9:40 PM, Tian(ICT) wrote: > >> Hi, >> Good morning/evening! >> I have this issue for quite a while. Using MSVC with Petsc, when debuging >> the code, the FormJacobian was never called, instead calling >> "MatMFFDComputeJacobian(SNES snes,Vec x,Mat *jac,Mat *B,MatStructure >> *flag,void *dummy)", though I donot use matrix free option. >> >> However, when runing the same thing in command line (still a debug >> compilation), I can see FormJacobian was called correctly. >> >> Any hint? >> > > My guess is that you are getting the matrix-free option from somewhere > (maybe a .petscrc file?). You can use -options_dump to see > everything passed in. If not, look at SNESSetFromOptions() in the debugger, > and see why it is choosing MF. > > Matt > > >> Rong >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jun 20 15:20:06 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 20 Jun 2011 14:20:06 -0600 Subject: [petsc-users] ILU and Block Gauss Seidel smoothing In-Reply-To: References: <9F52876E-C796-407A-8608-53D30D666A83@mcs.anl.gov> Message-ID: <86F0EB88-6914-4E20-9775-483E8EC2CBD7@mcs.anl.gov> I would suggest implementing your own PCSHELL that contains the four factored matrices and orders. The PCSetUp() you write would compute the orderings and call the Mat factorization routines. The PCApply() you write would do the appropriate applications of the triangular solves. The PCILU in PETSc is just for one ordering and factorization so is not a good starting point for your algorithm. Barry On Jun 20, 2011, at 8:48 AM, Jed Brown wrote: > On Mon, Jun 20, 2011 at 16:26, Gianluca Meneghello wrote: > My goal is to perform some relaxation sweeps for each ordering at each > level of a multigrid process, probably using ksptype PREONLY and > pctype ILU. > Is that possible? > > Yes, but you would need to perform a separate factorization for each ordering. Also, the orderings that are not aligned with the grid in memory will have poor performance. This is a general problem with using different orderings. > > Is there the equivalent of -pc_sor_its with ILU > (-pc_ilu_its maybe)? > > -ksp_type richardson is the same thing, put it inside -pc_type bjacobi for -pc_sor_lits. (-pc_sor_its is just a lower overhead way to do that cycling). > > > I also have the problem that in order to build the ordering I would > need to have access to a structure containing some grid informations, > and it seems I cannot pass that structure to the YourOrdering function > you suggested. > > You can PetscObjectCompose() your structure to the Mat. You might need PetscContainerCreate() to wrap your struct. > > > I guess a solution for me could be to build the IS from an external > function (not used by petsc) and then attach them directly to the mat > structure. I also guess that the one to use are the ones at line 36 of > > No, those slots are not public. Use PetscObjectCompose(). From mirzadeh at gmail.com Mon Jun 20 18:16:52 2011 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Mon, 20 Jun 2011 16:16:52 -0700 Subject: [petsc-users] MatSetValues and Zero enteries In-Reply-To: References: Message-ID: Matt, I see. Please forgive my ignorance, but I don't quite understand why you should not be able to set a tolerance for it. Could you please point me to the source file that applies the MAT_IGNORE_ZERO_ENTRIES option to the matrix? I may be able to manually modify it. Thanks, Mohammad On Mon, Jun 20, 2011 at 2:34 PM, Matthew Knepley wrote: > On Mon, Jun 20, 2011 at 9:31 PM, Mohammad Mirzadeh wrote: > >> Thanks John -- taht's helpful. Do you also know how I can change the >> tolerance for dropping elements? >> > > Right now, this only discards entires which are exactly floating point 0.0. > > Matt > > >> Mohammad >> >> On Mon, Jun 20, 2011 at 1:12 PM, John Mousel wrote: >> >>> See the matrix options available in the man pages. I think the function >>> below is what you are looking for. >>> >>> >>> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatSetOption.html >>> >>> >>> MatSetOption (A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE ); >>> >>> >>> >>> >>> >>> >>> Best Regards, >>> John >>> >>> >>> >>> >>> On Mon, Jun 20, 2011 at 2:42 PM, Mohammad Mirzadeh wrote: >>> >>>> Hi folks, >>>> >>>> I've noticed that MatSetValues explicitly inserts zeros in the matrix if >>>> an element is really zero. Is there a way to drop elements (or not save them >>>> in the first place) that are below a certain tolerance? I realize one way of >>>> doing it is in the application code and when using MatSetValues but I'm just >>>> wondering if PETSc has a function for it? >>>> >>>> Thanks, >>>> Mohammad >>>> >>> >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jun 20 18:20:48 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 20 Jun 2011 23:20:48 +0000 Subject: [petsc-users] MatSetValues and Zero enteries In-Reply-To: References: Message-ID: On Mon, Jun 20, 2011 at 11:16 PM, Mohammad Mirzadeh wrote: > Matt, > > I see. Please forgive my ignorance, but I don't quite understand why you > should not be able to set a tolerance for it. Could you please point me to > the source file that applies the MAT_IGNORE_ZERO_ENTRIES option to the > matrix? I may be able to manually modify it. > src/mat/impls/aij/seq/aij.c. Search for ignorezeroentries. I think changing it here would be bad design. Its simple to process your input matrix to MatSetValues() to chop small values to 0.0 Matt > Thanks, > Mohammad > > > On Mon, Jun 20, 2011 at 2:34 PM, Matthew Knepley wrote: > >> On Mon, Jun 20, 2011 at 9:31 PM, Mohammad Mirzadeh wrote: >> >>> Thanks John -- taht's helpful. Do you also know how I can change the >>> tolerance for dropping elements? >>> >> >> Right now, this only discards entires which are exactly floating point >> 0.0. >> >> Matt >> >> >>> Mohammad >>> >>> On Mon, Jun 20, 2011 at 1:12 PM, John Mousel wrote: >>> >>>> See the matrix options available in the man pages. I think the function >>>> below is what you are looking for. >>>> >>>> >>>> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatSetOption.html >>>> >>>> >>>> MatSetOption (A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE ); >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Best Regards, >>>> John >>>> >>>> >>>> >>>> >>>> On Mon, Jun 20, 2011 at 2:42 PM, Mohammad Mirzadeh wrote: >>>> >>>>> Hi folks, >>>>> >>>>> I've noticed that MatSetValues explicitly inserts zeros in the matrix >>>>> if an element is really zero. Is there a way to drop elements (or not save >>>>> them in the first place) that are below a certain tolerance? I realize one >>>>> way of doing it is in the application code and when using MatSetValues but >>>>> I'm just wondering if PETSc has a function for it? >>>>> >>>>> Thanks, >>>>> Mohammad >>>>> >>>> >>>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From khalid_eee at yahoo.com Tue Jun 21 00:34:47 2011 From: khalid_eee at yahoo.com (khalid ashraf) Date: Mon, 20 Jun 2011 22:34:47 -0700 (PDT) Subject: [petsc-users] profiling the total time In-Reply-To: References: Message-ID: <940946.45055.qm@web112602.mail.gq1.yahoo.com> Hi Jed, In order to find where the extra time is consumed, I started from ksp/ksp/example/tutorials/ex22.c and changed it one line at a time. I found that the time is consumed in the call: ierr = MatSetValuesStencil(B,7,row,7,col,&val[0][0],INSERT_VALUES);CHKERRQ(ierr); The same with the ierr = MatSetValuesBlockedStencil(B,7,row,7,col,&val[0][0],INSERT_VALUES);CHKERRQ(ierr); (total time=1e1) (here row is same as col) Now since ierr = MatSetValuesStencil(B,1,&row,7,col,&val[0][0],INSERT_VALUES);CHKERRQ(ierr); is very fast (total time=1e-2) (here row=(i,j,k)) I tried to call it 7 times for(l=0;l<7;l++){ ierr = MatSetValuesBlockedStencil(B,1,&row[l],7,col,&val[0][0],INSERT_VALUES);CHKERRQ(ierr);} } is again slow (total time=1e1) This is because I expected the computation time to be 7*1e-2 by this repetitive call. However, I find that the execution time is ~1e-2 only for l=3. For any other value of l, it is ~1e0. I can see that the only speciality of l=3 is that it corresponds to row=(i,j,k). Any other combination like (i+1,j,k) causes the call to MatSetValuesBlockedStencil(B,1,&row[l],7,col,&val[0][0],INSERT_VALUES);CHKERRQ(ierr);} to be slower by 2 orders of magnitude. Could you please suggest why the performance goes down so drastically. The difference in performance becomes visible when the matrix size gets closer to 3000X3000 and becomes worse very fast. I want to assemble a matrix for FEM application. I started from the example ksp/ksp/examples/tutorials/ex3.c This example shows the similar problem stated above when m>=120. Please let me know what could be done for any/both of the above two problems. Best regards. Khalid On Thu, Jun 16, 2011 at 11:14, khalid ashraf wrote: > When I look at the breakdown of the stages time required, the total add up > to ~7s however, the main stage time is ~350s. Two possibilities: 1. The time is not in PETSc. 2. The matrix is not preallocated correctly. http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#efficient-assembly -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhyshr at mcs.anl.gov Tue Jun 21 01:15:59 2011 From: abhyshr at mcs.anl.gov (Shri) Date: Tue, 21 Jun 2011 01:15:59 -0500 (CDT) Subject: [petsc-users] Error for ilu In-Reply-To: <618162191.27911.1308636681818.JavaMail.root@zimbra.anl.gov> Message-ID: <1364528461.27919.1308636959511.JavaMail.root@zimbra.anl.gov> Hi, I am getting this error for ilu. Did anything get changed for ilu? I get the same error for debug mode as well. ./TS -pc_type ilu [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: No support for this operation for this object type! [0]PETSC ERROR: Matrix type seqaij symbolic ILU! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: e36155ef11881fd94f40c202d1aacce6d908f14b HG Date: Mon Apr 25 11:24:07 2011 -0500 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./TS on a opt-mode named karlin.math.iit.edu by abhyshr Tue Jun 21 01:09:20 2011 [0]PETSC ERROR: Libraries linked from /home/abhyshr/petsc-dev/opt-mode/lib [0]PETSC ERROR: Configure run at Mon Apr 25 18:22:34 2011 [0]PETSC ERROR: Configure options --download-blacs --download-hypre --download-mpich --download-mumps --download-parmetis --download-plapack --download-scalapack --download-scotch --download-superlu --download-superlu_dist --download-umfpack --with-cc=gcc --with-cxx=g++ --with-debugging=0 --with-fc=gfortran --with-clanguage=cxx --download-f-blas-lapack [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatILUFactorSymbolic() line 6011 in src/mat/interface/matrix.c [0]PETSC ERROR: PCSetUp_ILU() line 216 in src/ksp/pc/impls/factor/ilu/ilu.c [0]PETSC ERROR: PCSetUp() line 819 in src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPSetUp() line 261 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: KSPSolve() line 383 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: SNES_KSPSolve() line 3124 in src/snes/interface/snes.c [0]PETSC ERROR: SNESSolve_LS() line 190 in src/snes/impls/ls/ls.c [0]PETSC ERROR: SNESSolve() line 2424 in src/snes/interface/snes.c [0]PETSC ERROR: main() line 160 in TS-dir/TS.c application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 [unset]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 Thanks, Shri From jed at 59A2.org Tue Jun 21 04:12:05 2011 From: jed at 59A2.org (Jed Brown) Date: Tue, 21 Jun 2011 11:12:05 +0200 Subject: [petsc-users] Error for ilu In-Reply-To: <1364528461.27919.1308636959511.JavaMail.root@zimbra.anl.gov> References: <618162191.27911.1308636681818.JavaMail.root@zimbra.anl.gov> <1364528461.27919.1308636959511.JavaMail.root@zimbra.anl.gov> Message-ID: On Tue, Jun 21, 2011 at 08:15, Shri wrote: > I am getting this error for ilu. Did anything get changed for ilu? I get > the same error for debug mode as well. > > ./TS -pc_type ilu > Can you reproduce with an example? I haven't noticed any problems. Maybe you have memory corruption or something else unexpected causing MatGetFactor to not be called. Maybe try setting a breakpoint at ilu.c:213. -------------- next part -------------- An HTML attachment was scrubbed... URL: From koen.janssens at psi.ch Tue Jun 21 04:18:49 2011 From: koen.janssens at psi.ch (Koenraad Janssens) Date: Tue, 21 Jun 2011 11:18:49 +0200 Subject: [petsc-users] Check creation of an object Message-ID: Is there a way to check whether an object has been created? I noticed PetSc crashes when calling PCView on a declared PC that has not been created yet. PC pc1; PCView(pc1, PETSC_VIEWER_STDOUT_WORLD); - Koen -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Tue Jun 21 04:25:37 2011 From: jed at 59A2.org (Jed Brown) Date: Tue, 21 Jun 2011 11:25:37 +0200 Subject: [petsc-users] profiling the total time In-Reply-To: <940946.45055.qm@web112602.mail.gq1.yahoo.com> References: <940946.45055.qm@web112602.mail.gq1.yahoo.com> Message-ID: On Tue, Jun 21, 2011 at 07:34, khalid ashraf wrote: > In order to find where the extra time is consumed, I started from > ksp/ksp/example/tutorials/ex22.c and changed it one line at a time. I found > that the time is consumed in the call: > > ierr = > MatSetValuesStencil(B,7,row,7,col,&val[0][0],INSERT_VALUES);CHKERRQ(ierr); > You must have changed more than just this because the arrays just aren't that big. This is a finite difference example. It inserts one row at a time. Inserting 7 rows at a time doesn't make sense. > This is because I expected the computation time to be 7*1e-2 by this > repetitive call. However, I find that the execution time is ~1e-2 only for > l=3. For any other value of l, it is ~1e0. I can see that the only > speciality of l=3 is that it corresponds to row=(i,j,k). Any other > combination like (i+1,j,k) causes the call to > > MatSetValuesBlockedStencil(B,1,&row[l],7,col,&val[0][0],INSERT_VALUES);CHKERRQ(ierr);} > to be slower by 2 orders of magnitude. Could you please suggest why the > performance goes down so drastically. > You have to preallocate correctly to get high performance. When you change the insertion code to set values arbitrarily, the preallocation becomes incorrect. This is all explained in the link I gave you earlier as well as the Users Manual. http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#efficient-assembly > I want to assemble a matrix for FEM application. I started from the example > ksp/ksp/examples/tutorials/ex3.c > This example shows the similar problem stated above when m>=120. > For simplicity, this example did not preallocate. I have added very naive preallocation to this example in petsc-dev. If you add the two lines below, assembly will be fast for large sizes. changeset: b961a1bfd123 user: Jed Brown date: Tue Jun 21 11:25:30 2011 +0200 summary: Add naive preallocation to example diff --git a/src/ksp/ksp/examples/tutorials/ex3.c b/src/ksp/ksp/examples/tutorials/ex3.c --- a/src/ksp/ksp/examples/tutorials/ex3.c +++ b/src/ksp/ksp/examples/tutorials/ex3.c @@ -64,6 +64,8 @@ ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N);CHKERRQ(ierr); ierr = MatSetFromOptions(A);CHKERRQ(ierr); + ierr = MatSeqAIJSetPreallocation(A,9,PETSC_NULL);CHKERRQ(ierr); + ierr = MatMPIAIJSetPreallocation(A,9,PETSC_NULL,5,PETSC_NULL);CHKERRQ(ierr); /* More than necessary */ start = rank*(M/size) + ((M%size) < rank ? (M%size) : rank); end = start + M/size + ((M%size) > rank); -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Tue Jun 21 04:49:11 2011 From: jed at 59A2.org (Jed Brown) Date: Tue, 21 Jun 2011 11:49:11 +0200 Subject: [petsc-users] Check creation of an object In-Reply-To: References: Message-ID: On Tue, Jun 21, 2011 at 11:18, Koenraad Janssens wrote: > Is there a way to check whether an object has been created? I noticed PetSc > crashes when calling PCView on a declared PC that has not been created yet. > > > PC pc1; > PCView(pc1, PETSC_VIEWER_STDOUT_WORLD); No, pc1 has not been initialized. Compilers should warn about this, the latest releases usually do. If you initialize pc1 to NULL (e.g. when it is declared), then PETSc functions will give you an error message saying that it is NULL. If this occurs in main, the stack will usually be cleared, in which case pc1 will automatically be NULL (but don't rely on this). In other functions, you have to expect that the stack is dirty. You can also run with valgrind which detects the uninitialized access. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Tue Jun 21 09:34:41 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 21 Jun 2011 09:34:41 -0500 Subject: [petsc-users] Error for ilu In-Reply-To: References: <618162191.27911.1308636681818.JavaMail.root@zimbra.anl.gov> <1364528461.27919.1308636959511.JavaMail.root@zimbra.anl.gov> Message-ID: Shri, I notice this is on Karlin. Does the same code work on other machines? Does petsc examples work on this machine? Hong On Tue, Jun 21, 2011 at 4:12 AM, Jed Brown wrote: > On Tue, Jun 21, 2011 at 08:15, Shri wrote: >> >> I am getting this error for ilu. Did anything get changed for ilu? I get >> the same error for debug mode as well. >> >> ./TS -pc_type ilu > > Can you reproduce with an example? I haven't noticed any problems. Maybe you > have memory corruption or something else unexpected causing MatGetFactor to > not be called. Maybe try setting a breakpoint at ilu.c:213. From amryazanov at gmail.com Tue Jun 21 09:45:10 2011 From: amryazanov at gmail.com (=?KOI8-R?B?4czFy9PFyiDy0drBzs/X?=) Date: Tue, 21 Jun 2011 18:45:10 +0400 Subject: [petsc-users] Output for TecPlot Message-ID: Hello. Help me please with my output trouble. I have a solution vector X and i need it to be visualized via TecPlot 360. X is a DA2D vector and PetscSnchtnizedFPrintf writes it down according to its petsc ordering, but i need the output to be arranged according to application ordering. My goal is to have a file with lines of this format: "i j x(i,j)". And Lines should be arranged firstly by j and secondly by i without any relation with DA processor distribution. Actually i need to realize more complex file structure, but the problem of automaticly propperly arranged output is the main stuck point. A combination of printf() and mpibarrier() didnt bring me luck. I need an advice. Thank you!! Alexey Ryazanov ______________________________________ Nuclear Safety Institute of Russian Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jun 21 09:49:14 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 21 Jun 2011 14:49:14 +0000 Subject: [petsc-users] Output for TecPlot In-Reply-To: References: Message-ID: On Tue, Jun 21, 2011 at 2:45 PM, ??????? ??????? wrote: > Hello. > > Help me please with my output trouble. > > I have a solution vector X and i need it to be visualized via TecPlot 360. > X is a DA2D vector and PetscSnchtnizedFPrintf writes it down according to > its petsc ordering, but i need the output to be arranged according to > application ordering. > > My goal is to have a file with lines of this format: "i j x(i,j)". > And Lines should be arranged firstly by j and secondly by i without any > relation with DA processor distribution. > Actually i need to realize more complex file structure, but the problem of > automaticly propperly arranged output is the main stuck point. > > A combination of printf() and mpibarrier() didnt bring me luck. I need an > advice. > You want to use http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/DM/DMDAGlobalToNaturalBegin.html to get a Vec with the natural ordering, and then VecView() to output it. You can postprocess that output to get whatever you want, or use VecGetArray() and print what you like. Matt > Thank you!! > > Alexey Ryazanov > ______________________________________ > Nuclear Safety Institute of Russian Academy of Sciences > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From amryazanov at gmail.com Tue Jun 21 09:54:12 2011 From: amryazanov at gmail.com (=?KOI8-R?B?4czFy9PFyiDy0drBzs/X?=) Date: Tue, 21 Jun 2011 18:54:12 +0400 Subject: [petsc-users] Output for TecPlot In-Reply-To: References: Message-ID: Thank you, Matt! I will try it 2011/6/21 Matthew Knepley > On Tue, Jun 21, 2011 at 2:45 PM, ??????? ??????? wrote: > >> Hello. >> >> Help me please with my output trouble. >> >> I have a solution vector X and i need it to be visualized via TecPlot 360. >> X is a DA2D vector and PetscSnchtnizedFPrintf writes it down according to >> its petsc ordering, but i need the output to be arranged according to >> application ordering. >> >> My goal is to have a file with lines of this format: "i j x(i,j)". >> And Lines should be arranged firstly by j and secondly by i without any >> relation with DA processor distribution. >> Actually i need to realize more complex file structure, but the problem of >> automaticly propperly arranged output is the main stuck point. >> >> A combination of printf() and mpibarrier() didnt bring me luck. I need an >> advice. >> > > You want to use > > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/DM/DMDAGlobalToNaturalBegin.html > > to get a Vec with the natural ordering, and then VecView() to output it. > You can postprocess that output to get whatever you want, > or use VecGetArray() and print what you like. > > Matt > > >> Thank you!! >> >> Alexey Ryazanov >> ______________________________________ >> Nuclear Safety Institute of Russian Academy of Sciences >> >> >> >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhyshr at mcs.anl.gov Tue Jun 21 12:14:28 2011 From: abhyshr at mcs.anl.gov (Shri) Date: Tue, 21 Jun 2011 12:14:28 -0500 (CDT) Subject: [petsc-users] Error for ilu In-Reply-To: Message-ID: <70699576.29772.1308676467999.JavaMail.root@zimbra.anl.gov> Sorry, my mistake..since 3 other people are working on this project someone added pc_factor_package mumps in the options file and hence ilu was giving that error. Thanks, Shri ----- Original Message ----- > Shri, > I notice this is on Karlin. > Does the same code work on other machines? > Does petsc examples work on this machine? > > Hong > > On Tue, Jun 21, 2011 at 4:12 AM, Jed Brown wrote: > > On Tue, Jun 21, 2011 at 08:15, Shri wrote: > >> > >> I am getting this error for ilu. Did anything get changed for ilu? > >> I get > >> the same error for debug mode as well. > >> > >> ./TS -pc_type ilu > > > > Can you reproduce with an example? I haven't noticed any problems. > > Maybe you > > have memory corruption or something else unexpected causing > > MatGetFactor to > > not be called. Maybe try setting a breakpoint at ilu.c:213. From bsmith at mcs.anl.gov Tue Jun 21 12:20:31 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 21 Jun 2011 11:20:31 -0600 Subject: [petsc-users] Error for ilu In-Reply-To: <70699576.29772.1308676467999.JavaMail.root@zimbra.anl.gov> References: <70699576.29772.1308676467999.JavaMail.root@zimbra.anl.gov> Message-ID: Hmm, then the error message is not complete enough. It printed seqaij but nothing about mumps hence confused everyone who read your email message. So how come mumps doesn't come up somewhere in the error message? Barry On Jun 21, 2011, at 11:14 AM, Shri wrote: > Sorry, my mistake..since 3 other people are working on this project someone added pc_factor_package mumps in the options file and hence ilu was giving that error. > > Thanks, > Shri > > ----- Original Message ----- >> Shri, >> I notice this is on Karlin. >> Does the same code work on other machines? >> Does petsc examples work on this machine? >> >> Hong >> >> On Tue, Jun 21, 2011 at 4:12 AM, Jed Brown wrote: >>> On Tue, Jun 21, 2011 at 08:15, Shri wrote: >>>> >>>> I am getting this error for ilu. Did anything get changed for ilu? >>>> I get >>>> the same error for debug mode as well. >>>> >>>> ./TS -pc_type ilu >>> >>> Can you reproduce with an example? I haven't noticed any problems. >>> Maybe you >>> have memory corruption or something else unexpected causing >>> MatGetFactor to >>> not be called. Maybe try setting a breakpoint at ilu.c:213. From jed at 59A2.org Tue Jun 21 12:21:04 2011 From: jed at 59A2.org (Jed Brown) Date: Tue, 21 Jun 2011 19:21:04 +0200 Subject: [petsc-users] Error for ilu In-Reply-To: References: <70699576.29772.1308676467999.JavaMail.root@zimbra.anl.gov> Message-ID: On Tue, Jun 21, 2011 at 19:20, Barry Smith wrote: > Hmm, then the error message is not complete enough. It printed seqaij but > nothing about mumps hence confused everyone who read your email message. So > how come mumps doesn't come up somewhere in the error message? I've got a fix for it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhyshr at mcs.anl.gov Tue Jun 21 13:02:52 2011 From: abhyshr at mcs.anl.gov (Shri) Date: Tue, 21 Jun 2011 13:02:52 -0500 (CDT) Subject: [petsc-users] Error for ilu In-Reply-To: Message-ID: <567588901.29934.1308679372355.JavaMail.root@zimbra.anl.gov> The error message is indeed confusing. Perhaps a better error message would be 'Cannot use -pc_type ilu with mumps'. ----- Original Message ----- > Hmm, then the error message is not complete enough. It printed seqaij > but nothing about mumps hence confused everyone who read your email > message. So how come mumps doesn't come up somewhere in the error > message? > > Barry > > > On Jun 21, 2011, at 11:14 AM, Shri wrote: > > > Sorry, my mistake..since 3 other people are working on this project > > someone added pc_factor_package mumps in the options file and hence > > ilu was giving that error. > > > > Thanks, > > Shri > > > > ----- Original Message ----- > >> Shri, > >> I notice this is on Karlin. > >> Does the same code work on other machines? > >> Does petsc examples work on this machine? > >> > >> Hong > >> > >> On Tue, Jun 21, 2011 at 4:12 AM, Jed Brown wrote: > >>> On Tue, Jun 21, 2011 at 08:15, Shri wrote: > >>>> > >>>> I am getting this error for ilu. Did anything get changed for > >>>> ilu? > >>>> I get > >>>> the same error for debug mode as well. > >>>> > >>>> ./TS -pc_type ilu > >>> > >>> Can you reproduce with an example? I haven't noticed any problems. > >>> Maybe you > >>> have memory corruption or something else unexpected causing > >>> MatGetFactor to > >>> not be called. Maybe try setting a breakpoint at ilu.c:213. From jed at 59A2.org Tue Jun 21 13:06:12 2011 From: jed at 59A2.org (Jed Brown) Date: Tue, 21 Jun 2011 20:06:12 +0200 Subject: [petsc-users] Error for ilu In-Reply-To: <567588901.29934.1308679372355.JavaMail.root@zimbra.anl.gov> References: <567588901.29934.1308679372355.JavaMail.root@zimbra.anl.gov> Message-ID: On Tue, Jun 21, 2011 at 20:02, Shri wrote: > The error message is indeed confusing. Perhaps a better error message would > be 'Cannot use -pc_type ilu with mumps'. Did you try with the patch I pushed 45 minutes ago? [0]PETSC ERROR: No support for this operation for this object type! [0]PETSC ERROR: Matrix type seqaij symbolic ILU using solver package mumps! -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhyshr at mcs.anl.gov Tue Jun 21 14:19:12 2011 From: abhyshr at mcs.anl.gov (Shri) Date: Tue, 21 Jun 2011 14:19:12 -0500 (CDT) Subject: [petsc-users] Error for ilu In-Reply-To: Message-ID: <138970854.30349.1308683952463.JavaMail.root@zimbra.anl.gov> Thanks for the patch. I tried to run the code but it spews an error Undefined symbols: "MatMFFDSetBase_MFFD(_p_Mat*, _p_Vec*, _p_Vec*)", referenced from: MatAssemblyEnd_SNESMF(_p_Mat*, MatAssemblyType) in libpetsc.a(snesmfj.o) _MatMFFDSetBase_SNESMF in libpetsc.a(snesmfj.o) make test is also giving the same error ----- Original Message ----- On Tue, Jun 21, 2011 at 20:02, Shri < abhyshr at mcs.anl.gov > wrote: The error message is indeed confusing. Perhaps a better error message would be 'Cannot use -pc_type ilu with mumps'. Did you try with the patch I pushed 45 minutes ago? [0]PETSC ERROR: No support for this operation for this object type! [0]PETSC ERROR: Matrix type seqaij symbolic ILU using solver package mumps! -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jun 21 14:26:51 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 21 Jun 2011 13:26:51 -0600 Subject: [petsc-users] Error for ilu In-Reply-To: <138970854.30349.1308683952463.JavaMail.root@zimbra.anl.gov> References: <138970854.30349.1308683952463.JavaMail.root@zimbra.anl.gov> Message-ID: <744CAE23-21D7-455E-BF9B-6880D9560945@mcs.anl.gov> Your make all must have generated an error. Barry On Jun 21, 2011, at 1:19 PM, Shri wrote: > Thanks for the patch. I tried to run the code but it spews an error > Undefined symbols: > "MatMFFDSetBase_MFFD(_p_Mat*, _p_Vec*, _p_Vec*)", referenced from: > MatAssemblyEnd_SNESMF(_p_Mat*, MatAssemblyType) in libpetsc.a(snesmfj.o) > _MatMFFDSetBase_SNESMF in libpetsc.a(snesmfj.o) > > make test is also giving the same error > > On Tue, Jun 21, 2011 at 20:02, Shri wrote: > The error message is indeed confusing. Perhaps a better error message would be 'Cannot use -pc_type ilu with mumps'. > > Did you try with the patch I pushed 45 minutes ago? > > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: Matrix type seqaij symbolic ILU using solver package mumps! > > > From jed at 59A2.org Tue Jun 21 14:28:40 2011 From: jed at 59A2.org (Jed Brown) Date: Tue, 21 Jun 2011 21:28:40 +0200 Subject: [petsc-users] Error for ilu In-Reply-To: <138970854.30349.1308683952463.JavaMail.root@zimbra.anl.gov> References: <138970854.30349.1308683952463.JavaMail.root@zimbra.anl.gov> Message-ID: On Tue, Jun 21, 2011 at 21:19, Shri wrote: > Thanks for the patch. I tried to run the code but it spews an error > Undefined symbols: > "MatMFFDSetBase_MFFD(_p_Mat*, _p_Vec*, _p_Vec*)", referenced from: > MatAssemblyEnd_SNESMF(_p_Mat*, MatAssemblyType) in > libpetsc.a(snesmfj.o) > _MatMFFDSetBase_SNESMF in libpetsc.a(snesmfj.o) > Pull and try again. I think it was just a missing EXTERN_C -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Tue Jun 21 20:07:22 2011 From: john.fettig at gmail.com (John Fettig) Date: Tue, 21 Jun 2011 21:07:22 -0400 Subject: [petsc-users] Attaching to gdb within screen Message-ID: Fellow petsc users, I am trying to start my program in gdb, using the options -start_in_debugger -debug_terminal "screen -X -S debug screen" where I have a running screen named debug. If I run with e.g. -np 4, all 4 processes report that they are being attached by gdb, but in the screen session only 2 new windows show up. If I run with -np 2, only 1 new window shows up. If I de-attach these windows by quitting gdb, the program runs. On the other hand, if I use "xterm -e", all of the xterms pop up. Any ideas? John From milanm at student.ethz.ch Wed Jun 22 04:45:10 2011 From: milanm at student.ethz.ch (Milan Mitrovic) Date: Wed, 22 Jun 2011 11:45:10 +0200 Subject: [petsc-users] Ghost communication Message-ID: Hello everyone, I'm trying to use petsc with ppm and here is what I have so far: First I create parallel vectors with ghost locations using an existing array that already holds the data: CALL VecCreateGhostWithArray(PETSC_COMM_WORLD, & Particles%Npart, & PETSC_DECIDE, & Particles%Mpart-Particles%Npart, & gi(Particles%Npart:Particles%Mpart), & wp, petsc_vector, info) where gi is the global indexing, wp is the existing data array, Npart is the number of real, and Mpart the number of real+ghost particles. Then, I create a matrix shell, and provide my own multiplication routine (this part works fine I think). Inside the multiplication routine I do this: CALL VecGhostUpdateBegin(from,INSERT_VALUES,SCATTER_FORWARD,info) CALL VecGhostUpdateEnd(from,INSERT_VALUES,SCATTER_FORWARD,info) ! get ghosted versions of vectors CALL VecGhostGetLocalForm(from,xl,info) CALL VecSet(to,0.0_MK,info) ! get arrays from petsc vectors CALL VecGetArray(xl,xx,xxi,info) CALL VecGetArray(to,yy,yyi,info) Then I modify the data, and do the following: CALL VecRestoreArray(xl,xx,xxi,info) CALL VecRestoreArray(to,yy,yyi,info) ! release local form of vectors CALL VecGhostRestoreLocalForm(from,xl,info) CALL VecGhostUpdateBegin(to,ADD_VALUES,SCATTER_REVERSE,info) CALL VecGhostUpdateEnd(to,ADD_VALUES,SCATTER_REVERSE,info) And then I call KSP solve: call KSPCreate(PETSC_COMM_WORLD, ksp, info) call KSPSetOperators(ksp, A, A, DIFFERENT_NONZERO_PATTERN, info) call KSPSolve(ksp, b, x, info) I noticed that the values of the ghost particles never change. The communication never seems to take place... Can you give me some hints on what to try? Or what I do wrong? Thanks! Milan Mitrovic From jed at 59A2.org Wed Jun 22 06:01:06 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 22 Jun 2011 13:01:06 +0200 Subject: [petsc-users] Ghost communication In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 11:45, Milan Mitrovic wrote: > Inside the multiplication routine I do this: > > CALL VecGhostUpdateBegin(from,INSERT_VALUES,SCATTER_FORWARD,info) > CALL VecGhostUpdateEnd(from,INSERT_VALUES,SCATTER_FORWARD,info) > This does update the ghost entries, right? > > ! get ghosted versions of vectors > CALL VecGhostGetLocalForm(from,xl,info) > CALL VecSet(to,0.0_MK,info) > ! get arrays from petsc vectors > CALL VecGetArray(xl,xx,xxi,info) > CALL VecGetArray(to,yy,yyi,info) > Is "to" a ghosted vector in which the residual should be the sum of contributions to all ghost copies (as in most finite element implementations)? If so, then you should call VecGhostGetLocalForm(to,yl,info) call VecGetArray(yl,yy,yyi,info) ... > > Then I modify the data, and do the following: > > CALL VecRestoreArray(xl,xx,xxi,info) > CALL VecRestoreArray(to,yy,yyi,info) > ! release local form of vectors > CALL VecGhostRestoreLocalForm(from,xl,info) > > CALL VecGhostUpdateBegin(to,ADD_VALUES,SCATTER_REVERSE,info) > CALL VecGhostUpdateEnd(to,ADD_VALUES,SCATTER_REVERSE,info) > You can always test this by getting the local form, doing VecSet(yl,1.0), restore the local form, do the scatter above, and view to. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Wed Jun 22 06:25:30 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 22 Jun 2011 13:25:30 +0200 Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 03:07, John Fettig wrote: > I am trying to start my program in gdb, using the options > > -start_in_debugger -debug_terminal "screen -X -S debug screen" > > where I have a running screen named debug. If I run with e.g. -np 4, > all 4 processes report that they are being attached by gdb, but in the > screen session only 2 new windows show up. If I run with -np 2, only > 1 new window shows up. > I've seen this, but only inconsistently (race condition) and only with Open MPI (not with MPICH2). Are you using MPICH2 or Open MPI? I think MPICH2 might be setting SA_NOCLDSTOP which is saving us from the following race condition: 1. process is forked 2a. parent sleeps 2b. child gets parent PID and execs the screen attach command 3. screen attach command returns immediately, before actually running the command (invoking gdb) 4. child exits, generates SIGCHLD which wakes up parent 5. parent continues onward, possibly completing and exiting 6. screen session gets around to invoking gdb and attaching to the parent process, but it has already exited (sometimes some of the processes have exited while others are still in MPI_Finalize or similar) I think the solution is to use sigaction(3) to set SA_NOCLDSTOP so that the child terminating does not generate SIGCHLD. The man page points out that this is typically the desired behavior. Is there some obscure platform that supports signal(), but not sigaction() or can we use sigaction() for everything? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jun 22 06:54:20 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 22 Jun 2011 05:54:20 -0600 Subject: [petsc-users] Ghost communication In-Reply-To: References: Message-ID: How are b and x obtained? They need to be optained via VecDuplicate() from petsc_vector, if you obtain them via a regular VecCreate() they won't be of ghosted style hence mess up your plan. Barry Plus you need to do as Jed suggested. On Jun 22, 2011, at 3:45 AM, Milan Mitrovic wrote: > Hello everyone, > > I'm trying to use petsc with ppm and here is what I have so far: > > First I create parallel vectors with ghost locations using an existing > array that already holds the data: > > CALL VecCreateGhostWithArray(PETSC_COMM_WORLD, & > Particles%Npart, & > PETSC_DECIDE, & > Particles%Mpart-Particles%Npart, & > gi(Particles%Npart:Particles%Mpart), & > wp, petsc_vector, info) > > where gi is the global indexing, wp is the existing data array, Npart > is the number of real, and Mpart the number of real+ghost particles. > > Then, I create a matrix shell, and provide my own multiplication > routine (this part works fine I think). > > Inside the multiplication routine I do this: > > CALL VecGhostUpdateBegin(from,INSERT_VALUES,SCATTER_FORWARD,info) > CALL VecGhostUpdateEnd(from,INSERT_VALUES,SCATTER_FORWARD,info) > > ! get ghosted versions of vectors > CALL VecGhostGetLocalForm(from,xl,info) > CALL VecSet(to,0.0_MK,info) > ! get arrays from petsc vectors > CALL VecGetArray(xl,xx,xxi,info) > CALL VecGetArray(to,yy,yyi,info) > > Then I modify the data, and do the following: > > CALL VecRestoreArray(xl,xx,xxi,info) > CALL VecRestoreArray(to,yy,yyi,info) > ! release local form of vectors > CALL VecGhostRestoreLocalForm(from,xl,info) > > CALL VecGhostUpdateBegin(to,ADD_VALUES,SCATTER_REVERSE,info) > CALL VecGhostUpdateEnd(to,ADD_VALUES,SCATTER_REVERSE,info) > > And then I call KSP solve: > > call KSPCreate(PETSC_COMM_WORLD, ksp, info) > call KSPSetOperators(ksp, A, A, DIFFERENT_NONZERO_PATTERN, info) > call KSPSolve(ksp, b, x, info) > > I noticed that the values of the ghost particles never change. The > communication never seems to take place... > > Can you give me some hints on what to try? Or what I do wrong? > > Thanks! > > Milan Mitrovic From john.fettig at gmail.com Wed Jun 22 07:43:55 2011 From: john.fettig at gmail.com (John Fettig) Date: Wed, 22 Jun 2011 08:43:55 -0400 Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 7:25 AM, Jed Brown wrote: > Are you using MPICH2 or Open MPI? I think MPICH2 might be setting > SA_NOCLDSTOP which is saving us from the following race condition: I'm using mpich2-1.3.2p1 My screen version (which is hopefully not important) is 4.00.03. John From jed at 59A2.org Wed Jun 22 07:53:12 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 22 Jun 2011 14:53:12 +0200 Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 14:43, John Fettig wrote: > I'm using mpich2-1.3.2p1 My screen version (which is hopefully not > important) is 4.00.03. > Interesting. Which operating system? When some processes attach successfully, what does the backtrace from gdb look like? -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Wed Jun 22 07:59:20 2011 From: john.fettig at gmail.com (John Fettig) Date: Wed, 22 Jun 2011 08:59:20 -0400 Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 8:53 AM, Jed Brown wrote: > On Wed, Jun 22, 2011 at 14:43, John Fettig wrote: >> >> I'm using mpich2-1.3.2p1 My screen version (which is hopefully not >> important) is ?4.00.03. > > Interesting. Which operating system? When some processes attach > successfully, what does the backtrace from gdb look like? I'm using CentOS 5.6. Here's the backtrace from -np 2 (where only one gdb is launched): #0 0x000000310ee9a510 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x000000310ee9a364 in sleep () from /lib64/libc.so.6 #2 0x00002b37a6601a83 in PetscSleep (s=10) at psleep.c:56 #3 0x00002b37a5ef049b in PetscAttachDebugger () at adebug.c:410 #4 0x00002b37a6302351 in PetscOptionsCheckInitial_Private () at init.c:392 #5 0x00002b37a65cdeb6 in PetscInitialize (argc=0x7fff636a6d3c, args=0x7fff636a6d30, file=0x0, help=0x9f3d4d "") at pinit.c:639 #6 0x0000000000409ac4 in main (argc=2, argv=0x7fff636a6ee8) at main.c:694 John From jed at 59A2.org Wed Jun 22 08:09:01 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 22 Jun 2011 15:09:01 +0200 Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 14:59, John Fettig wrote: > On Wed, Jun 22, 2011 at 8:53 AM, Jed Brown wrote: > > On Wed, Jun 22, 2011 at 14:43, John Fettig > wrote: > >> > >> I'm using mpich2-1.3.2p1 My screen version (which is hopefully not > >> important) is 4.00.03. > > > > Interesting. Which operating system? When some processes attach > > successfully, what does the backtrace from gdb look like? > > I'm using CentOS 5.6. Here's the backtrace from -np 2 (where only one > gdb is launched): > > #0 0x000000310ee9a510 in __nanosleep_nocancel () from /lib64/libc.so.6 > #1 0x000000310ee9a364 in sleep () from /lib64/libc.so.6 > #2 0x00002b37a6601a83 in PetscSleep (s=10) at psleep.c:56 > #3 0x00002b37a5ef049b in PetscAttachDebugger () at adebug.c:410 > #4 0x00002b37a6302351 in PetscOptionsCheckInitial_Private () at init.c:392 > #5 0x00002b37a65cdeb6 in PetscInitialize (argc=0x7fff636a6d3c, > args=0x7fff636a6d30, file=0x0, help=0x9f3d4d "") at pinit.c:639 > #6 0x0000000000409ac4 in main (argc=2, argv=0x7fff636a6ee8) at main.c:694 Okay, this is the right place. Can you do a couple experiments: $ cat > redirect.sh #!/bin/sh echo ppid=$PPID pwd=$PWD cmd="$@" sleep 10 exec "$@" $ chmod +x redirect.sh Then show me the output of: $ mpiexec -n 2 .... -debug_terminal "redirect.sh echo" and $ mpiexec -n 2 .... -debug_terminal "screen -X -S debug screen redirect.sh" You should be able to see some output in the screen session before gdb is invoked. -------------- next part -------------- An HTML attachment was scrubbed... URL: From maxime.moge at fr.michelin.com Wed Jun 22 08:14:57 2011 From: maxime.moge at fr.michelin.com (maxime.moge at fr.michelin.com) Date: Wed, 22 Jun 2011 15:14:57 +0200 Subject: [petsc-users] PETSc 3.2 release Message-ID: An HTML attachment was scrubbed... URL: From jed at 59A2.org Wed Jun 22 08:15:35 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 22 Jun 2011 15:15:35 +0200 Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 15:09, Jed Brown wrote: > $ cat > redirect.sh > #!/bin/sh > > echo ppid=$PPID pwd=$PWD cmd="$@" > sleep 10 > exec "$@" > This might be more useful written as echo ... "$@" sleep 10 Also, if you do sleep for 10 before, then be sure to add -debugger_pause 20 to the PETSc command line because 10 happens to be the default sleep time. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Wed Jun 22 08:23:38 2011 From: john.fettig at gmail.com (John Fettig) Date: Wed, 22 Jun 2011 09:23:38 -0400 Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 9:09 AM, Jed Brown wrote: > $ mpiexec -n 2 .... -debug_terminal "redirect.sh echo" The command passed to the debug_terminal seems to be wrong: ppid=24296 pwd=/home/jfe/test cmd=-display localhost:0.0 echo gdb /home/jfe/test/example 24296 The exec line fails because of this. John From jed at 59A2.org Wed Jun 22 08:35:14 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 22 Jun 2011 15:35:14 +0200 Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 15:23, John Fettig wrote: > The command passed to the debug_terminal seems to be wrong: > > ppid=24296 pwd=/home/jfe/test cmd=-display localhost:0.0 echo gdb > /home/jfe/test/example 24296 > > The exec line fails because of this. > Whoops, it should still work when you pass it with: -debug_terminal "screen -X -S debug screen redirect.sh" because there is special treatment for "screen" with regard to -display. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aron.ahmadia at kaust.edu.sa Wed Jun 22 08:40:18 2011 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Wed, 22 Jun 2011 16:40:18 +0300 Subject: [petsc-users] PETSc 3.2 release In-Reply-To: References: Message-ID: We (KAUST) are also interested in seeing a release soon, and I am happy to help verify the 64-bit index code on our BG/P here when you have a release candidate ready. A On Wed, Jun 22, 2011 at 4:14 PM, wrote: > Hi, > > A few week weeks ago you wrote about releasing PETSc 3.2 "before the summer > rush". Did you set a more precise date already ? > Will PETSc 3.2 use the new Cusp 2.0 with support for CUDA 4.0 ? > > Regards, > > Maxime > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Wed Jun 22 08:50:36 2011 From: john.fettig at gmail.com (John Fettig) Date: Wed, 22 Jun 2011 09:50:36 -0400 Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 9:35 AM, Jed Brown wrote: > Whoops, it should still work when you pass it with: > -debug_terminal "screen -X -S debug screen redirect.sh" > because there is special treatment for "screen" with regard to -display. This still doesn't work, although I have no idea why. The title of the screen changes briefly to Cannot exec '/correct/path/garbage': No such file or directory. where "garbage" means non-ascii characters. I gave it the full path to redirect.sh and it displayed that correctly, just not the script. John From balay at mcs.anl.gov Wed Jun 22 08:54:21 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 22 Jun 2011 08:54:21 -0500 (CDT) Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, 22 Jun 2011, John Fettig wrote: > On Wed, Jun 22, 2011 at 8:53 AM, Jed Brown wrote: > > On Wed, Jun 22, 2011 at 14:43, John Fettig wrote: > >> > >> I'm using mpich2-1.3.2p1 My screen version (which is hopefully not > >> important) is ?4.00.03. > > > > Interesting. Which operating system? When some processes attach > > successfully, what does the backtrace from gdb look like? > > I'm using CentOS 5.6. Here's the backtrace from -np 2 (where only one > gdb is launched): > > #0 0x000000310ee9a510 in __nanosleep_nocancel () from /lib64/libc.so.6 > #1 0x000000310ee9a364 in sleep () from /lib64/libc.so.6 > #2 0x00002b37a6601a83 in PetscSleep (s=10) at psleep.c:56 > #3 0x00002b37a5ef049b in PetscAttachDebugger () at adebug.c:410 > #4 0x00002b37a6302351 in PetscOptionsCheckInitial_Private () at init.c:392 > #5 0x00002b37a65cdeb6 in PetscInitialize (argc=0x7fff636a6d3c, > args=0x7fff636a6d30, file=0x0, help=0x9f3d4d "") at pinit.c:639 > #6 0x0000000000409ac4 in main (argc=2, argv=0x7fff636a6ee8) at main.c:694 John, what pm do you use for MPICH startup? i.e what configure options did you use to build it? Have a centos 5.6 box so can try reproducing it. Satish From john.fettig at gmail.com Wed Jun 22 08:55:12 2011 From: john.fettig at gmail.com (John Fettig) Date: Wed, 22 Jun 2011 09:55:12 -0400 Subject: [petsc-users] Size of off-diagonal matrix Message-ID: I'm having trouble understanding how the size of off-diagonal matrix is determined in the MPIAIJ format. For example, if I run on 2 processors the global size is 4177x4177, but the local matrices are Rank 1: 2089x2089 (diagonal block) 2089x259 (off diagonal block) Rank2: 2088x274 (off diagonal block) 2088x2088 (diagonal block) Why isn't the off-diagonal block sized for all of the off-diagonal columns? John From knepley at gmail.com Wed Jun 22 08:56:28 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 22 Jun 2011 08:56:28 -0500 Subject: [petsc-users] Size of off-diagonal matrix In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 8:55 AM, John Fettig wrote: > I'm having trouble understanding how the size of off-diagonal matrix > is determined in the MPIAIJ format. For example, if I run on 2 > processors the global size is 4177x4177, but the local matrices are > > Rank 1: > 2089x2089 (diagonal block) > 2089x259 (off diagonal block) > > Rank2: > 2088x274 (off diagonal block) > 2088x2088 (diagonal block) > > Why isn't the off-diagonal block sized for all of the off-diagonal columns? It is compressed, so it only has columns that have a nonzero. Matt > > John > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Wed Jun 22 08:57:53 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 22 Jun 2011 15:57:53 +0200 Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 15:50, John Fettig wrote: > This still doesn't work, although I have no idea why. The title of > the screen changes briefly to > > Cannot exec '/correct/path/garbage': No such file or directory. > > where "garbage" means non-ascii characters. I gave it the full path > to redirect.sh and it displayed that correctly, just not the script. > By title, I'm guessing you mean the screen session title. Could you try changing the script to just have: echo ppid=$PPID pwd=$PWD cmd="$@" sleep 10 Screen's remote session creation unfortunately resets PWD (at least on my machine), so the gdb command is expected to "fail" at opening the executable (unless the original program was invoked with full path), but succeed at attaching to the process ID. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Wed Jun 22 08:57:41 2011 From: john.fettig at gmail.com (John Fettig) Date: Wed, 22 Jun 2011 09:57:41 -0400 Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 9:54 AM, Satish Balay wrote: > what pm do you use for MPICH startup? i.e what configure options did > you use to build it? Satish, Here's what mpiexec --version reports that I used for the pertinent configure options: '--enable-shared' '--with-mpe' '--enable-fast=-O2 -g' '--with-device=ch3:nemesis' 'CC=gcc' 'CPP=cpp' 'F77=gfortran' 'FC=gfortran' 'CXX=g++' '--disable-option-checking' John From john.fettig at gmail.com Wed Jun 22 09:02:33 2011 From: john.fettig at gmail.com (John Fettig) Date: Wed, 22 Jun 2011 10:02:33 -0400 Subject: [petsc-users] Size of off-diagonal matrix In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 9:56 AM, Matthew Knepley wrote: > It is compressed, so it only has columns that have a nonzero. Thanks for answering so quickly. I had just assumed that it would logically still report sizes that added up to the global column size so I was confused. John From john.fettig at gmail.com Wed Jun 22 09:05:59 2011 From: john.fettig at gmail.com (John Fettig) Date: Wed, 22 Jun 2011 10:05:59 -0400 Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 9:57 AM, Jed Brown wrote: > On Wed, Jun 22, 2011 at 15:50, John Fettig wrote: >> >> This still doesn't work, although I have no idea why. ?The title of >> the screen changes briefly to >> >> Cannot exec '/correct/path/garbage': No such file or directory. >> >> where "garbage" means non-ascii characters. ?I gave it the full path >> to redirect.sh and it displayed that correctly, just not the script. > > By title, I'm guessing you mean the screen session title. Yes, that's what I meant. The title bar in the terminal window briefly changes to that, and then it returns to "screen 0: bash". > Could you try changing the script to just have: > echo ppid=$PPID pwd=$PWD cmd="$@" > sleep 10 This makes no difference, because the script isn't executing. If it makes any difference, I'm setting the options through the environment variable "PETSC_OPTIONS" John > Screen's remote session creation unfortunately resets PWD (at least on my > machine), so the gdb command is expected to "fail" at opening the executable > (unless the original program was invoked with full path), but succeed at > attaching to the process ID. From jed at 59A2.org Wed Jun 22 09:12:18 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 22 Jun 2011 16:12:18 +0200 Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 16:05, John Fettig wrote: > This makes no difference, because the script isn't executing. If it > makes any difference, I'm setting the options through the environment > variable "PETSC_OPTIONS" > Hmm, only in that the arguments would need to be doubly escaped. I would expect that is working if you are getting one screen terminal to come up, but could you try using the command line with a PETSc example, e.g. src/ksp/ksp/examples/tutorials/ex2.c. I'm running out of efficient remote debugging ideas so it would help a lot of we can figure out how to reproduce this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Wed Jun 22 09:40:29 2011 From: john.fettig at gmail.com (John Fettig) Date: Wed, 22 Jun 2011 10:40:29 -0400 Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 10:12 AM, Jed Brown wrote: > On Wed, Jun 22, 2011 at 16:05, John Fettig wrote: >> >> This makes no difference, because the script isn't executing. If it >> makes any difference, I'm setting the options through the environment >> variable "PETSC_OPTIONS" > > Hmm, only in that the arguments would need to be doubly escaped. I would > expect that is working if you are getting one screen terminal to come up, > but could you try using the command line with a PETSc example, e.g. > src/ksp/ksp/examples/tutorials/ex2.c. This is how I set PETSC_OPTIONS: export PETSC_OPTIONS="-start_in_debugger -debug_terminal \"screen -X -S debug screen\"" I did try using the command line options only (unset $PETSC_OPTIONS and used a petsc example), and that has the same problem. If I only run the command screen -X -S debug screen redirect.sh gdb It loads a new window in screen without issue. So something funky is happening inside petsc. John From milan.v.mitrovic at gmail.com Wed Jun 22 09:45:51 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Wed, 22 Jun 2011 16:45:51 +0200 Subject: [petsc-users] Ghost communication In-Reply-To: References: Message-ID: Both vectors are created with the VecCreateGhostWithArray as pasted above. The ghosts never get updated... I put a dummy value and it stays untouched. 'to' is also a ghosted vector but the contributions from the ghosts should be discarded. The ghosts are only used to compute the new local values. Any ideas? From jed at 59A2.org Wed Jun 22 09:48:39 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 22 Jun 2011 16:48:39 +0200 Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 16:40, John Fettig wrote: > > I did try using the command line options only (unset $PETSC_OPTIONS > and used a petsc example), and that has the same problem. If I only > run the command > > screen -X -S debug screen redirect.sh gdb > > It loads a new window in screen without issue. What command line does it report using for that? > So something funky is > happening inside petsc. > > Does the executable path include any special characters or spaces? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Wed Jun 22 09:50:45 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 22 Jun 2011 16:50:45 +0200 Subject: [petsc-users] Ghost communication In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 16:45, Milan Mitrovic wrote: > Both vectors are created with the VecCreateGhostWithArray as pasted above. > > The ghosts never get updated... I put a dummy value and it stays untouched. > Please paste the code you used to conclude this and what you expected to see. > > 'to' is also a ghosted vector but the contributions from the ghosts > should be discarded. The ghosts are only used to compute the new local > values. > Then why were you using this? CALL VecGhostUpdateBegin(to,ADD_VALUES,SCATTER_REVERSE,info) CALL VecGhostUpdateEnd(to,ADD_VALUES,SCATTER_REVERSE,info) What did you expect that to do? -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jun 22 09:52:21 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 22 Jun 2011 09:52:21 -0500 Subject: [petsc-users] Ghost communication In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 9:45 AM, Milan Mitrovic wrote: > Both vectors are created with the VecCreateGhostWithArray as pasted above. > > The ghosts never get updated... I put a dummy value and it stays untouched. > > 'to' is also a ghosted vector but the contributions from the ghosts > should be discarded. The ghosts are only used to compute the new local > values. > > Any ideas? > Can you run Vec ex9? It demonstrates the use of ghosted vectors. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From milan.v.mitrovic at gmail.com Wed Jun 22 09:54:04 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Wed, 22 Jun 2011 16:54:04 +0200 Subject: [petsc-users] Ghost communication In-Reply-To: References: Message-ID: vec ex9? From balay at mcs.anl.gov Wed Jun 22 09:57:25 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 22 Jun 2011 09:57:25 -0500 (CDT) Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, 22 Jun 2011, John Fettig wrote: > On Wed, Jun 22, 2011 at 9:54 AM, Satish Balay wrote: > > what pm do you use for MPICH startup? i.e what configure options did > > you use to build it? > > Satish, > > Here's what mpiexec --version reports that I used for the pertinent > configure options: > > '--enable-shared' '--with-mpe' '--enable-fast=-O2 -g' > '--with-device=ch3:nemesis' 'CC=gcc' 'CPP=cpp' 'F77=gfortran' > 'FC=gfortran' 'CXX=g++' '--disable-option-checking' Ok - I just did the following on 'CentOS release 5.6 (Final)' 32bit: install mpich2-1.3.2p1.tar.gz with: ./configure --enable-shared --enable-fast='-O2 -g' --with-device=ch3:nemesis CC=gcc CPP=cpp --disable-f77 --disable-fc --disable-option-checking --prefix=$HOME/mpich-install --disable-cxx install petsc-dev with: 1047 ./configure --with-mpi-dir=${HOME}/mpich-install --with-fc=0 --download-c-blas-lapack=1 --with-sowing=0 --with-c2html=0 start screen on one terminal screen -S debug And run an example: src/ksp/ksp/examples/tutorials/ex2.c [balay at petsc tutorials]$ ~/mpich-install/bin/mpiexec -n 4 ./ex2 -start_in_debugger -debug_terminal "screen -X -S debug screen" [3]PETSC ERROR: PETSC: Attaching gdb to ./ex2 on pid 25786 on petsc.cs.iit.edu [0]PETSC ERROR: PETSC: Attaching gdb to ./ex2 on pid 25783 on petsc.cs.iit.edu [1]PETSC ERROR: PETSC: Attaching gdb to ./ex2 on pid 25784 on petsc.cs.iit.edu [2]PETSC ERROR: PETSC: Attaching gdb to ./ex2 on pid 25785 on petsc.cs.iit.edu And I see all 4 gdb sessions on the screen session. Satish From jed at 59A2.org Wed Jun 22 09:57:39 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 22 Jun 2011 16:57:39 +0200 Subject: [petsc-users] Ghost communication In-Reply-To: References: Message-ID: 2011/6/22 Milan Mitrovic > vec ex9? > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/src/vec/vec/examples/tutorials/ex9.c.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Wed Jun 22 09:57:38 2011 From: john.fettig at gmail.com (John Fettig) Date: Wed, 22 Jun 2011 10:57:38 -0400 Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 10:48 AM, Jed Brown wrote: > On Wed, Jun 22, 2011 at 16:40, John Fettig wrote: >> >> I did try using the command line options only (unset $PETSC_OPTIONS >> and used a petsc example), and that has the same problem. ?If I only >> run the command >> >> screen -X -S debug screen redirect.sh gdb >> >> It loads a new window in screen without issue. > > What command line does it report using for that? The output is: ppid=28218 pwd=/home/jfe cmd=gdb And the screen title is "screen 1: redirect.sh" >> ?So something funky is >> happening inside petsc. >> > > Does the executable path include any special characters or spaces? No spaces, but there are a couple of underscores. As a test I moved redirect.sh into my path and re-ran. This works! I get both gdb's and they give meaningful backtraces. What does this mean? John From qzhu at mcs.anl.gov Wed Jun 22 09:58:50 2011 From: qzhu at mcs.anl.gov (Qian Zhu) Date: Wed, 22 Jun 2011 09:58:50 -0500 Subject: [petsc-users] non-convergence for SNES ex5 and ex20 Message-ID: <89615C74-8B9A-4ABD-ABE8-A4077AD57162@mcs.anl.gov> Hi, I am running ex5 and ex20 under $PETSC_DIR/src/snes/examples/tutorials. I need to see some non-converged cases. For ex5, I tried different values for "par" while for ex20, "beta" was varied (to create different inputs thus different problems to solve). Now it seems that no matter which solver+pc I used, all cases converged for both applications. Let me explain what I need using an example...For ex5, par=xxx, using "-ksp_type gmres -pc_type ilu" will make the solution converge while using "-ksp_type cg -pc_type jacobi" cannot (all ksp's and pc's are just randomly picked...they may not make sense). What would this "xxx" value be? Or is there any input parameters I am missing out for these applications? Thanks a lot! Qian Qian Zhu, Ph.D. Mathematics and Computer Science Division Argonne National Laboratory 9700 S.Cass Avenue Argonne, IL 60439-4844 Phone: 630-252-0086 From balay at mcs.anl.gov Wed Jun 22 10:02:26 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 22 Jun 2011 10:02:26 -0500 (CDT) Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, 22 Jun 2011, Satish Balay wrote: > On Wed, 22 Jun 2011, John Fettig wrote: > > > On Wed, Jun 22, 2011 at 9:54 AM, Satish Balay wrote: > > > what pm do you use for MPICH startup? i.e what configure options did > > > you use to build it? > > > > Satish, > > > > Here's what mpiexec --version reports that I used for the pertinent > > configure options: > > > > '--enable-shared' '--with-mpe' '--enable-fast=-O2 -g' > > '--with-device=ch3:nemesis' 'CC=gcc' 'CPP=cpp' 'F77=gfortran' > > 'FC=gfortran' 'CXX=g++' '--disable-option-checking' > > Ok - I just did the following on 'CentOS release 5.6 (Final)' 32bit: > > install mpich2-1.3.2p1.tar.gz with: > ./configure --enable-shared --enable-fast='-O2 -g' --with-device=ch3:nemesis CC=gcc CPP=cpp --disable-f77 --disable-fc --disable-option-checking --prefix=$HOME/mpich-install --disable-cxx > > install petsc-dev with: > 1047 ./configure --with-mpi-dir=${HOME}/mpich-install --with-fc=0 --download-c-blas-lapack=1 --with-sowing=0 --with-c2html=0 > > start screen on one terminal > screen -S debug > > And run an example: src/ksp/ksp/examples/tutorials/ex2.c > [balay at petsc tutorials]$ ~/mpich-install/bin/mpiexec -n 4 ./ex2 -start_in_debugger -debug_terminal "screen -X -S debug screen" > [3]PETSC ERROR: PETSC: Attaching gdb to ./ex2 on pid 25786 on petsc.cs.iit.edu > [0]PETSC ERROR: PETSC: Attaching gdb to ./ex2 on pid 25783 on petsc.cs.iit.edu > [1]PETSC ERROR: PETSC: Attaching gdb to ./ex2 on pid 25784 on petsc.cs.iit.edu > [2]PETSC ERROR: PETSC: Attaching gdb to ./ex2 on pid 25785 on petsc.cs.iit.edu > > > And I see all 4 gdb sessions on the screen session. I retry - and now I see 3 gdb sessions.. hm.. Satish From jed at 59A2.org Wed Jun 22 10:04:06 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 22 Jun 2011 17:04:06 +0200 Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 17:02, Satish Balay wrote: > I retry - and now I see 3 gdb sessions.. hm.. That may be the race condition discussed earlier. -------------- next part -------------- An HTML attachment was scrubbed... URL: From milan.v.mitrovic at gmail.com Wed Jun 22 10:12:28 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Wed, 22 Jun 2011 17:12:28 +0200 Subject: [petsc-users] Ghost communication In-Reply-To: References: Message-ID: should I do it in c or in fortran? It seems identical to what I am doing... From xdliang at gmail.com Wed Jun 22 10:18:21 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Wed, 22 Jun 2011 11:18:21 -0400 Subject: [petsc-users] Fwd: superlu_dist with same_nonzero_pattern In-Reply-To: References: Message-ID: Hello everyone, I had some problem in using superlu_dist with same_nonzero_pattern. Sometimes superlu_dist solver halts there and never finishes. However, when I switched to different_nonzero_pattern or other solvers, the program works nicely. ?Ding had reported this problem a few months ago: ?http://lists.mcs.anl.gov/pipermail/petsc-users/2011-January/007623.html Following Barry's suggestion, I checked that the nonzero pattern of my matrix is still the same via ?ierr=MatSetOption(M, MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE); CHKERRQ(ierr); I tried two other options: 1 First install the latest superlu_dist (v2.5), then compile petsc with this new superlu_dist package. However, the problem is still there, and the performance is even worse ( comparing to the option --download-superlu_dist=yes (v2.4)). 2 I compiled petsc-dev with --download-superlu_dist. However, I got some errors when compiling my program with petsc-dev. It said that ResonatorOpt.c: In function ?main?: ResonatorOpt.c:154: error: ?MAT_SOLVER_SUPERLU_DIST? undeclared (first use in this function) ResonatorOpt.c:154: error: (Each undeclared identifier is reported only once ResonatorOpt.c:154: error: for each function it appears in.) Besides superlu_dist, MAT_SOLVER_PASTIX and MAT_SOLVER_PASTIX are undeclared either. Am I missing some header files? I used the same procedure as I complied regular petsc, which had no such problems. (I tried to attach the configure log in a previous mail, but it is bounced back.) Can you give me some suggestions on using superlu_dist with same_nonzero_pattern? Thank you. Best, Xiangdong From jed at 59A2.org Wed Jun 22 10:26:55 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 22 Jun 2011 17:26:55 +0200 Subject: [petsc-users] non-convergence for SNES ex5 and ex20 In-Reply-To: <89615C74-8B9A-4ABD-ABE8-A4077AD57162@mcs.anl.gov> References: <89615C74-8B9A-4ABD-ABE8-A4077AD57162@mcs.anl.gov> Message-ID: On Wed, Jun 22, 2011 at 16:58, Qian Zhu wrote: > For ex5, par=xxx, using "-ksp_type gmres -pc_type ilu" will make the > solution converge while using "-ksp_type cg -pc_type jacobi" cannot (all > ksp's and pc's are just randomly picked...they may not make sense). boundary conditions are not symmetric in this example > What would this "xxx" value be? Or is there any input parameters I am > missing out for these applications? look at snes ex19 and use -lidvelocity 100 -grashof 1e5 if you are looking for something that doesn't converge. Or try a Stokes problem. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jun 22 10:29:50 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 22 Jun 2011 10:29:50 -0500 Subject: [petsc-users] Ghost communication In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 10:12 AM, Milan Mitrovic wrote: > should I do it in c or in fortran? It seems identical to what I am doing... > cd src/vec/vec/examples/tutorials make ex9 ./ex9 Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From qzhu at mcs.anl.gov Wed Jun 22 10:32:36 2011 From: qzhu at mcs.anl.gov (Qian Zhu) Date: Wed, 22 Jun 2011 10:32:36 -0500 Subject: [petsc-users] non-convergence for SNES ex5 and ex20 In-Reply-To: References: <89615C74-8B9A-4ABD-ABE8-A4077AD57162@mcs.anl.gov> Message-ID: On Jun 22, 2011, at 10:26 AM, Jed Brown wrote: > On Wed, Jun 22, 2011 at 16:58, Qian Zhu wrote: > For ex5, par=xxx, using "-ksp_type gmres -pc_type ilu" will make the solution converge while using "-ksp_type cg -pc_type jacobi" cannot (all ksp's and pc's are just randomly picked...they may not make sense). > > boundary conditions are not symmetric in this example > > What would this "xxx" value be? Or is there any input parameters I am missing out for these applications? > > look at snes ex19 and use -lidvelocity 100 -grashof 1e5 if you are looking for something that doesn't converge. Or try a Stokes problem. Thanks for the reply, Jed. I worked on ex27 so I know setting certain values for the input parameters can make the case converge/non-converge...I am just wondering whether this is also true with ex5 and ex20 (We picked them for other interesting characteristics). Which examples are solving the Stokers problem? Thanks a lot! Qian Zhu, Ph.D. Mathematics and Computer Science Division Argonne National Laboratory 9700 S.Cass Avenue Argonne, IL 60439-4844 Phone: 630-252-0086 -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Jun 22 10:38:55 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 22 Jun 2011 10:38:55 -0500 (CDT) Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, 22 Jun 2011, Jed Brown wrote: > On Wed, Jun 22, 2011 at 17:02, Satish Balay wrote: > > > I retry - and now I see 3 gdb sessions.. hm.. > > > That may be the race condition discussed earlier. My reproducability factor is low [i.e its working most of the time]. John, does restarting the screen session [perhaps with a different name] or using '-debugger_puase 10' make a difference? Satish From john.mousel at gmail.com Wed Jun 22 10:37:23 2011 From: john.mousel at gmail.com (John Mousel) Date: Wed, 22 Jun 2011 10:37:23 -0500 Subject: [petsc-users] ML spectral radius estimate Message-ID: I'm using ML as a preconditioner, and I want to get an estimate of the spectral radius of my preconditioned matrix. I see a flag -pc_ml_SpectralNormScheme_Anorm. Does this need to be set in conjunction with another flag like -ksp_compute_eigenvalues or something cool I don't know about? John -------------- next part -------------- An HTML attachment was scrubbed... URL: From sperif at gmail.com Wed Jun 22 10:44:03 2011 From: sperif at gmail.com (Pierre-Yves Aquilanti) Date: Wed, 22 Jun 2011 17:44:03 +0200 Subject: [petsc-users] KSPLSQR Incompatible vector local lengths Message-ID: Hello, i'm trying to use the KSPLSQR solver in order to solve the Ax=b where b and x are of size "n" and matrix A is of size "n+1 x n". A is in fact a superior hessenberg matrix (i upload a figure at http://img824.imageshack.us/i/totog.jpg in case of). I don't use any preconditionning and n=20. Also, to understand where the program was crashing, i runned Valgrind which tells me that the error is related to a VecCopy operation withing the KSPSolve_LSQR. I checked the vectors and matrix size, but it doesn't seem to come from the sizes. Also, i'm using sequential matrix and vectors. However, despite my search, i am not able to make the solver work as intended, it return a "Incompatible vector local lengths!" error. Do you have any idea on what would be my mistake ? Thanks a lot Regards PYA Ps: i attached above the petsc error log, valgrind log and related code. *************************** Error : *************************** [3]PETSC ERROR: --------------------- Error Message ------------------------------------ [3]PETSC ERROR: Arguments are incompatible! [3]PETSC ERROR: Incompatible vector local lengths! [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [3]PETSC ERROR: See docs/changes/index.html for recent updates. [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [3]PETSC ERROR: See docs/index.html for manual pages. [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: ./hyperh on a linux-c-d named Tripel by perif Wed Jun 22 16:43:26 2011 [3]PETSC ERROR: Libraries linked from /home/perif/Utils/Libs/petsc-3.1-p8/linux-c-debug-real/lib [3]PETSC ERROR: Configure run at Tue May 24 14:49:00 2011 [3]PETSC ERROR: Configure options --download-c-blas-lapack=yes --with-scalar-type=complex --with-fortran=0 --with-debugging=yes --with-shared=0 --with-petsc-arch=linux-c-debug-real [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: VecCopy() line 1694 in src/vec/vec/interface/vector.c [3]PETSC ERROR: KSPSolve_LSQR() line 135 in src/ksp/ksp/impls/lsqr/lsqr.c [3]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c [3]PETSC ERROR: User provided function() line 217 in precond.c *************************** Code : *************************** /*get factor matrix*/ MatCreateSeqDense(PETSC_COMM_WORLD,(*nb_eigen_all)+1,(*nb_eigen_all)+1,PETSC_NULL,&fact); MatSetFromOptions(fact); MatAssemblyBegin(fact,MAT_FINAL_ASSEMBLY); MatAssemblyEnd(fact,MAT_FINAL_ASSEMBLY); /*Create the matrix operator that will be used in the QR factorization*/ MatCreateSeqDense(PETSC_COMM_WORLD,(*nb_eigen)+1,(*nb_eigen),PETSC_NULL,&F); MatSetFromOptions(F); /* Set the matrix values */ ................................ /*assemble F for processing*/ MatAssemblyBegin(F,MAT_FINAL_ASSEMBLY); MatAssemblyEnd(F,MAT_FINAL_ASSEMBLY); /* set the vectors*/ ierr=VecCreate(PETSC_COMM_WORLD,&rhs);CHKERRQ(ierr); ierr=VecCreate(PETSC_COMM_WORLD,&soln);CHKERRQ(ierr); ierr=VecSetSizes(rhs, PETSC_DECIDE, (*nb_eigen));CHKERRQ(ierr); ierr=VecSetSizes(soln, PETSC_DECIDE, (*nb_eigen));CHKERRQ(ierr); ierr=VecSetType(soln,VECSEQ);CHKERRQ(ierr); ierr=VecSetType(rhs,VECSEQ);CHKERRQ(ierr); /*set the solution to zero*/ ierr=VecSet(soln,(PetscScalar)0.0);CHKERRQ(ierr); ierr=VecSet(rhs,(PetscScalar)0.0);CHKERRQ(ierr); /* rhs[0] must be setted to beta*/ ierr=VecSetValue(rhs,0,(PetscReal)fact_tmp[0],INSERT_VALUES);CHKERRQ(ierr); ierr = VecAssemblyBegin(soln);CHKERRQ(ierr); ierr = VecAssemblyEnd(soln);CHKERRQ(ierr); ierr = VecAssemblyBegin(rhs);CHKERRQ(ierr); ierr = VecAssemblyEnd(rhs);CHKERRQ(ierr); #ifdef DEBUG PetscViewerSetFormat(PETSC_VIEWER_STDOUT_WORLD,PETSC_VIEWER_ASCII_MATLAB ); MatView(F,PETSC_VIEWER_STDOUT_WORLD); #endif /*create the lsqr solver context and set it up*/ ierr=KSPCreate(PETSC_COMM_WORLD,&ksplsqr);CHKERRQ(ierr); ierr=KSPSetOperators(ksplsqr,F,F,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); ierr=KSPGetPC(ksplsqr,&pclsqr);CHKERRQ(ierr); ierr=PCSetType(pclsqr,PCNONE);CHKERRQ(ierr); ierr=KSPSetType(ksplsqr,KSPLSQR);CHKERRQ(ierr); ierr=KSPSetInitialGuessNonzero(ksplsqr,PETSC_TRUE);CHKERRQ(ierr); ierr=KSPSetUp(ksplsqr);CHKERRQ(ierr); /* now we are ready to kick some ass and chew some bubble gum unfortunately i'm all out of gum */ ierr = KSPSolve(ksplsqr, rhs, soln); CHKERRQ(ierr); *************************** Valgrind Error : *************************** ==13892== Invalid read of size 8 ==13892== at 0x678B76: VecAYPX_Seq (dvec2.c:667) ==13892== by 0x65B5DE: VecAYPX (rvector.c:692) ==13892== by 0x8636D3: KSPSolve_LSQR (lsqr.c:119) ==13892== by 0x80B7C5: KSPSolve (itfunc.c:396) ==13892== by 0x4174B8: LSPrecond (precond.c:217) ==13892== by 0x414B1E: LSQR (lsqr.c:172) ==13892== by 0x407FCD: main (main.c:105) ==13892== Address 0x8f5f350 is 816 bytes inside a block of size 820 alloc'd ==13892== at 0x4C27870: memalign (vg_replace_malloc.c:581) ==13892== by 0x5948FB: PetscMallocAlign (mal.c:30) ==13892== by 0x595DDD: PetscTrMallocDefault (mtr.c:192) ==13892== by 0x66AEEA: VecCreate_Seq (bvec2.c:823) ==13892== by 0x650478: VecSetType (vecreg.c:54) ==13892== by 0x416EA4: LSPrecond (precond.c:179) ==13892== by 0x414B1E: LSQR (lsqr.c:172) ==13892== by 0x407FCD: main (main.c:105) ==13892== ==13892== Invalid read of size 8 ==13892== at 0x678B7A: VecAYPX_Seq (dvec2.c:667) ==13892== by 0x65B5DE: VecAYPX (rvector.c:692) ==13892== by 0x8636D3: KSPSolve_LSQR (lsqr.c:119) ==13892== by 0x80B7C5: KSPSolve (itfunc.c:396) ==13892== by 0x4174B8: LSPrecond (precond.c:217) ==13892== by 0x414B1E: LSQR (lsqr.c:172) ==13892== by 0x407FCD: main (main.c:105) ==13892== Address 0x8f5f358 is 4 bytes after a block of size 820 alloc'd ==13892== at 0x4C27870: memalign (vg_replace_malloc.c:581) ==13892== by 0x5948FB: PetscMallocAlign (mal.c:30) ==13892== by 0x595DDD: PetscTrMallocDefault (mtr.c:192) ==13892== by 0x66AEEA: VecCreate_Seq (bvec2.c:823) ==13892== by 0x650478: VecSetType (vecreg.c:54) ==13892== by 0x416EA4: LSPrecond (precond.c:179) ==13892== by 0x414B1E: LSQR (lsqr.c:172) ==13892== by 0x407FCD: main (main.c:105) ==13892== -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Wed Jun 22 10:46:34 2011 From: john.fettig at gmail.com (John Fettig) Date: Wed, 22 Jun 2011 11:46:34 -0400 Subject: [petsc-users] Attaching to gdb within screen In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 11:38 AM, Satish Balay wrote: > On Wed, 22 Jun 2011, Jed Brown wrote: > >> On Wed, Jun 22, 2011 at 17:02, Satish Balay wrote: >> >> > I retry - and now I see 3 gdb sessions.. hm.. >> >> >> That may be the race condition discussed earlier. > > My reproducability factor is low [i.e its working most of the time]. I'm guessing this might be a difference in speed of our machines? > John, > > does restarting the screen session [perhaps with a different name] or > using '-debugger_puase 10' make a difference? No. I've tried running a few more times without the "redirect.sh" script and it seems that I get both of the gdb's some of the time. I think Jed must be right about a race condition. John From knepley at gmail.com Wed Jun 22 10:59:03 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 22 Jun 2011 10:59:03 -0500 Subject: [petsc-users] ML spectral radius estimate In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 10:37 AM, John Mousel wrote: > I'm using ML as a preconditioner, and I want to get an estimate of the > spectral radius of my preconditioned matrix. I see a flag > -pc_ml_SpectralNormScheme_Anorm. Does this need to be set in conjunction > with another flag like -ksp_compute_eigenvalues or something cool I don't > know about? I do not see a way to pull out the estimate from ML. However, you can easily replicate their estimate using -ksp_monitor_singular_values and run GMRES with -pc_type none. If you run with ML, it will estimate the extreme values for the preconditioned operators. Matt > > John > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From milan.v.mitrovic at gmail.com Wed Jun 22 10:59:52 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Wed, 22 Jun 2011 17:59:52 +0200 Subject: [petsc-users] Ghost communication In-Reply-To: References: Message-ID: mpirun -n 2 ./ex9 0 0 1 1 2 2 3 3 4 4 5 5 6 11 7 6 0 6 1 7 2 8 3 9 4 10 5 11 6 0 7 5 I am also doing it in fortran to test if it works: if (rank .eq. 0) then order(1) = 11 order(2) = 6 else order(1) = 0 order(2) = 5 end if call VecCreateGhost(PETSC_COMM_WORLD, 6, PETSC_DECIDE, 2, order, p, info) call VecGhostGetLocalForm(p, pl, info) call VecGetOwnershipRange(p, ps, pe, info) do i=ps,pe-1 call VecSetValues(p,1,i,real(i,8),INSERT_VALUES,info) end do call VecAssemblyBegin(p, info) call VecAssemblyEnd(p, info) call VecGhostUpdateBegin(p,INSERT_VALUES,SCATTER_FORWARD,info) call VecGhostUpdateEnd(p,INSERT_VALUES,SCATTER_FORWARD,info) call VecGetArray(pl,pla,plai,info) and the output is: [0] p(1) = 0 [0] p(2) = 1 [0] p(3) = 2 [0] p(4) = 3 [0] p(5) = 4 [0] p(6) = 5 [0] p(7) = 11 [0] p(8) = 6 [1] p(1) = 6 [1] p(2) = 7 [1] p(3) = 8 [1] p(4) = 9 [1] p(5) = 10 [1] p(6) = 11 [1] p(7) = 0 [1] p(8) = 5 so it seems to work... but the other code still doesn't From knepley at gmail.com Wed Jun 22 11:02:12 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 22 Jun 2011 11:02:12 -0500 Subject: [petsc-users] Ghost communication In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 10:59 AM, Milan Mitrovic wrote: > mpirun -n 2 ./ex9 > 0 0 > 1 1 > 2 2 > 3 3 > 4 4 > 5 5 > 6 11 > 7 6 > 0 6 > 1 7 > 2 8 > 3 9 > 4 10 > 5 11 > 6 0 > 7 5 > > I am also doing it in fortran to test if it works: > > if (rank .eq. 0) then > order(1) = 11 > order(2) = 6 > else > order(1) = 0 > order(2) = 5 > end if > > call VecCreateGhost(PETSC_COMM_WORLD, 6, PETSC_DECIDE, 2, order, p, info) > > call VecGhostGetLocalForm(p, pl, info) > call VecGetOwnershipRange(p, ps, pe, info) > > do i=ps,pe-1 > call VecSetValues(p,1,i,real(i,8),INSERT_VALUES,info) > end do > > call VecAssemblyBegin(p, info) > call VecAssemblyEnd(p, info) > > call VecGhostUpdateBegin(p,INSERT_VALUES,SCATTER_FORWARD,info) > call VecGhostUpdateEnd(p,INSERT_VALUES,SCATTER_FORWARD,info) > > call VecGetArray(pl,pla,plai,info) > > and the output is: > > [0] p(1) = 0 > [0] p(2) = 1 > [0] p(3) = 2 > [0] p(4) = 3 > [0] p(5) = 4 > [0] p(6) = 5 > [0] p(7) = 11 > [0] p(8) = 6 > > [1] p(1) = 6 > [1] p(2) = 7 > [1] p(3) = 8 > [1] p(4) = 9 > [1] p(5) = 10 > [1] p(6) = 11 > [1] p(7) = 0 > [1] p(8) = 5 > > so it seems to work... but the other code still doesn't > So we know that ghost update works, so something is coded incorrectly in your other example. I recommend changing ex9 step-by-step until you can do what you want. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Jun 22 13:06:31 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 22 Jun 2011 13:06:31 -0500 Subject: [petsc-users] Fwd: superlu_dist with same_nonzero_pattern In-Reply-To: References: Message-ID: Xiangdong : Does it hangs inside superlu_dist? Try runtime option '-mat_superlu_dist_equil NO' If you can send us a stand-alone short code or your matrix in petsc binary format that repeats this behavior, then we can investigate it. Likely memory problem. Hong > Hello everyone, > > I had some problem in using superlu_dist with same_nonzero_pattern. > Sometimes superlu_dist solver halts there and never finishes. However, > when I switched to different_nonzero_pattern or other solvers, the > program works nicely. ?Ding had reported this problem a few months > ago: > ?http://lists.mcs.anl.gov/pipermail/petsc-users/2011-January/007623.html > > Following Barry's suggestion, I checked that the nonzero pattern of my > matrix is still the same via > ?ierr=MatSetOption(M, MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE); CHKERRQ(ierr); > > I tried two other options: > > 1 First install the latest superlu_dist (v2.5), then compile petsc > with this new superlu_dist package. However, the problem is still > there, and the performance is even worse ( comparing to the option > --download-superlu_dist=yes (v2.4)). > > 2 I compiled petsc-dev with --download-superlu_dist. However, I got > some errors when compiling my program with petsc-dev. It said that > > ResonatorOpt.c: In function ?main?: > ResonatorOpt.c:154: error: ?MAT_SOLVER_SUPERLU_DIST? undeclared (first > use in this function) > ResonatorOpt.c:154: error: (Each undeclared identifier is reported only once > ResonatorOpt.c:154: error: for each function it appears in.) > > Besides superlu_dist, MAT_SOLVER_PASTIX and MAT_SOLVER_PASTIX are > undeclared either. Am I missing some header files? I used the same > procedure as I complied regular petsc, which had no such problems. ?(I > tried to attach the configure log in a previous mail, but it is > bounced back.) > > Can you give me some suggestions on using superlu_dist with > same_nonzero_pattern? Thank you. > > Best, > Xiangdong > From jed at 59A2.org Wed Jun 22 15:26:12 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 22 Jun 2011 22:26:12 +0200 Subject: [petsc-users] non-convergence for SNES ex5 and ex20 In-Reply-To: References: <89615C74-8B9A-4ABD-ABE8-A4077AD57162@mcs.anl.gov> Message-ID: On Wed, Jun 22, 2011 at 17:32, Qian Zhu wrote: > Thanks for the reply, Jed. I worked on ex27 so I know setting certain > values for the input parameters can make the case converge/non-converge...I > am just wondering whether this is also true with ex5 and ex20 (We picked > them for other interesting characteristics). Well, ex5 is a really simple problem which converges pretty easily when lambda < 6.81 (the bifurcation point). I don't know how hard ex20 is, but you could certainly increase the power beta. What characteristics are you looking for? Asking for a solver that doesn't converge just seems like an odd request considering that we spend lots of time figuring out how to make them converge robustly. > Which examples are solving the Stokers problem? src/ksp/ksp/examples/tutorials/ex43.c This will happily not converge: $ ./ex43 -mx 40 -my 40 -c_str 2 -sinker_eta0 1e-8 -sinker_eta1 1 -stokes_ksp_monitor_true_residual -stokes_ksp_type fgmres Run this to see some configurations that do work: $ make -n runex43 runex43_2 The discretization above is not particularly robust, but is relatively easy to solve. This one is harder [1] src/ksp/ksp/examples/tests/ex11.c This will pretend to converge, but the true residuals are nonsense. ./ex11 -f $DATAFILESPATH/matrices/underworld32.gz -fc_ksp_monitor_true_residual If you add -fc_ksp_type fgmres to the above, it will stop pretending (and just not converge). To see some solver configurations that work, use $ make -n runex11 runex11_2 [1] Follow the directions here to get the test matrix: http://www.mcs.anl.gov/petsc/petsc-2/documentation/faq.html#datafiles -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Wed Jun 22 15:50:47 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 22 Jun 2011 22:50:47 +0200 Subject: [petsc-users] KSPLSQR Incompatible vector local lengths In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 17:44, Pierre-Yves Aquilanti wrote: > i'm trying to use the KSPLSQR solver in order to solve the Ax=b where b and > x are of size "n" and matrix A is of size "n+1 x n". > Shouldn't the right hand side vector have length n+1? > A is in fact a superior hessenberg matrix > Is your "real" system small and dense, or is this a model for some large system? In the small dense case, the natural thing is to use a direct solver, e.g. with Lapack. -------------- next part -------------- An HTML attachment was scrubbed... URL: From juhaj at iki.fi Wed Jun 22 18:52:09 2011 From: juhaj at iki.fi (Juha =?iso-8859-1?q?J=E4ykk=E4?=) Date: Thu, 23 Jun 2011 00:52:09 +0100 Subject: [petsc-users] SNES_DIVERGED_LS_FAILURE Message-ID: <201106230052.12923.juhaj@iki.fi> Hi all! I am having nasty problems with SNES. As you probably guess already, my result is non-converging line search. I have a good candidate reason for this, but I do not know how to fix it. Jacobian should be fine: -snes_ls_type test gives nice small numbers (not quite as small as I would like: around 1e-7). And even if Jacobian was wrong, -snes_fd behaves in exactly the same way. When snes gives up with its line search, the values in the residual vector are: 0.00000000e+00, 1.02404374e+00, 2.28558501e-01, 1.94195641e-02, -4.82401091e-02, -7.07456591e-02, -7.64790651e-02, -7.55281623e-02, -7.19176452e-02 before they stabilise to interval (1.e-5, 8e-2). Now, my candidate is the huge (compared to others) value of the rhs at the second lattice point and slightly smaller, but still large at the third. The diverged solution vector grows very fast in the beginning and then settles to an almost constant value after about one tenth of the lattice, so my guess is that the big change needed at i=1 causes precision to be lost at i>>1 and therefore line search fails. Do you have any ideas how to get around this or if my theory is even correct? Trying trust region search does not really help: it suffers from the same deficiency - except that it thinks it converges even though the function norm is of the order of 0.5. (It does fare slightly better, though, its rhs starts like 0.00000000e+00, 3.93745063e-01, 1.79187741e-01, 1.03100385e-01, 6.76686117e-02, 4.82967347e-02, 3.65243377e-02, 2.88059666e-02, 2.34452314e-02, which is better, but still not good enough.) Cheers, Juha -- ----------------------------------------------- | Juha J?ykk?, juhaj at iki.fi | | http://www.maths.leeds.ac.uk/~juhaj | ----------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. URL: From knepley at gmail.com Wed Jun 22 20:20:47 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 22 Jun 2011 20:20:47 -0500 Subject: [petsc-users] SNES_DIVERGED_LS_FAILURE In-Reply-To: <201106230052.12923.juhaj@iki.fi> References: <201106230052.12923.juhaj@iki.fi> Message-ID: On Wed, Jun 22, 2011 at 6:52 PM, Juha J?ykk? wrote: > Hi all! > > I am having nasty problems with SNES. As you probably guess already, my > result > is non-converging line search. I have a good candidate reason for this, but > I > do not know how to fix it. > > Jacobian should be fine: -snes_ls_type test gives nice small numbers (not > quite as small as I would like: around 1e-7). > > And even if Jacobian was wrong, -snes_fd behaves in exactly the same way. > > When snes gives up with its line search, the values in the residual vector > are: > > 0.00000000e+00, 1.02404374e+00, 2.28558501e-01, > 1.94195641e-02, -4.82401091e-02, -7.07456591e-02, > -7.64790651e-02, -7.55281623e-02, -7.19176452e-02 > > before they stabilise to interval (1.e-5, 8e-2). Now, my candidate is the > huge > (compared to others) value of the rhs at the second lattice point and > slightly > smaller, but still large at the third. The diverged solution vector grows > very > fast in the beginning and then settles to an almost constant value after > about > one tenth of the lattice, so my guess is that the big change needed at i=1 > causes precision to be lost at i>>1 and therefore line search fails. > > Do you have any ideas how to get around this or if my theory is even > correct? > Its hard to say anything without knowing what equations are being solved. However, does the linear problem converge? If so, you could try continuation in a parameter in the nonlinear term. Matt > Trying trust region search does not really help: it suffers from the same > deficiency - except that it thinks it converges even though the function > norm > is of the order of 0.5. (It does fare slightly better, though, its rhs > starts > like > > 0.00000000e+00, 3.93745063e-01, 1.79187741e-01, > 1.03100385e-01, 6.76686117e-02, 4.82967347e-02, > 3.65243377e-02, 2.88059666e-02, 2.34452314e-02, > > which is better, but still not good enough.) > > Cheers, > Juha > > -- > ----------------------------------------------- > | Juha J?ykk?, juhaj at iki.fi | > | http://www.maths.leeds.ac.uk/~juhaj | > ----------------------------------------------- > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jun 22 21:30:56 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 22 Jun 2011 21:30:56 -0500 Subject: [petsc-users] Size of off-diagonal matrix In-Reply-To: References: Message-ID: On Jun 22, 2011, at 9:02 AM, John Fettig wrote: > On Wed, Jun 22, 2011 at 9:56 AM, Matthew Knepley wrote: >> It is compressed, so it only has columns that have a nonzero. > > Thanks for answering so quickly. I had just assumed that it would > logically still report sizes that added up to the global column size > so I was confused. What do you mean "report"? The sequential matrix that represents the off-diagonal part of the matrix actually does not contain the "zero columns" so does have fewer columns and hence the smaller number. Barry > > John From bsmith at mcs.anl.gov Wed Jun 22 22:12:53 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 22 Jun 2011 22:12:53 -0500 Subject: [petsc-users] SNES_DIVERGED_LS_FAILURE In-Reply-To: References: <201106230052.12923.juhaj@iki.fi> Message-ID: As Matt notes you absolutely need to run with -pc_type lu -ksp_monitor_true_residual -ksp_converged_reason to make sure that the linear system solve is not the issue. It is also possible that a bug in the function evaluation (using uninitialized values or something) is causing the difficulty: run with valgrind (http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind) to check for these types of errors and review the code. Barry On Jun 22, 2011, at 8:20 PM, Matthew Knepley wrote: > On Wed, Jun 22, 2011 at 6:52 PM, Juha J?ykk? wrote: > Hi all! > > I am having nasty problems with SNES. As you probably guess already, my result > is non-converging line search. I have a good candidate reason for this, but I > do not know how to fix it. > > Jacobian should be fine: -snes_ls_type test gives nice small numbers (not > quite as small as I would like: around 1e-7). > > And even if Jacobian was wrong, -snes_fd behaves in exactly the same way. > > When snes gives up with its line search, the values in the residual vector > are: > > 0.00000000e+00, 1.02404374e+00, 2.28558501e-01, > 1.94195641e-02, -4.82401091e-02, -7.07456591e-02, > -7.64790651e-02, -7.55281623e-02, -7.19176452e-02 > > before they stabilise to interval (1.e-5, 8e-2). Now, my candidate is the huge > (compared to others) value of the rhs at the second lattice point and slightly > smaller, but still large at the third. The diverged solution vector grows very > fast in the beginning and then settles to an almost constant value after about > one tenth of the lattice, so my guess is that the big change needed at i=1 > causes precision to be lost at i>>1 and therefore line search fails. > > Do you have any ideas how to get around this or if my theory is even correct? > > Its hard to say anything without knowing what equations are being solved. However, > does the linear problem converge? If so, you could try continuation in a parameter > in the nonlinear term. > > Matt > > Trying trust region search does not really help: it suffers from the same > deficiency - except that it thinks it converges even though the function norm > is of the order of 0.5. (It does fare slightly better, though, its rhs starts > like > > 0.00000000e+00, 3.93745063e-01, 1.79187741e-01, > 1.03100385e-01, 6.76686117e-02, 4.82967347e-02, > 3.65243377e-02, 2.88059666e-02, 2.34452314e-02, > > which is better, but still not good enough.) > > Cheers, > Juha > > -- > ----------------------------------------------- > | Juha J?ykk?, juhaj at iki.fi | > | http://www.maths.leeds.ac.uk/~juhaj | > ----------------------------------------------- > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From bsmith at mcs.anl.gov Wed Jun 22 22:14:38 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 22 Jun 2011 22:14:38 -0500 Subject: [petsc-users] PETSc 3.2 release In-Reply-To: References: Message-ID: <407EC8EB-61C9-4585-BB25-6A207464BFE5@mcs.anl.gov> On Jun 22, 2011, at 8:14 AM, maxime.moge at fr.michelin.com wrote: > Hi, > > A few week weeks ago you wrote about releasing PETSc 3.2 "before the summer rush". Did you set a more precise date already ? > Will PETSc 3.2 use the new Cusp 2.0 with support for CUDA 4.0 ? > > Regards, > > Maxime Unfortunately we have missed the "summer rush", at this point I don't see doing a release until maybe the later portion of August. Barry From Andrew.Barker at Colorado.EDU Thu Jun 23 04:13:50 2011 From: Andrew.Barker at Colorado.EDU (Andrew T Barker) Date: Thu, 23 Jun 2011 03:13:50 -0600 (MDT) Subject: [petsc-users] PCCOMPOSITE and ApplyTranspose Message-ID: <20110623031350.AIV11763@batman.int.colorado.edu> I've made a PCCOMPOSITE that contains two PCSHELLS, both of which have a transpose set with PCShellSetApplyTranspose(). However the composite PC does not seem to support the transpose: [0]PETSC ERROR: No support for this operation for this object type! [0]PETSC ERROR: PC does not have apply transpose! The code runs fine with the composite replaced with either of the shells by itself. Of course I can implement my own composite as a PCShell with a transpose easily enough, but I was wondering if this was expected or if I am making some other mistake. I am using Petsc 3.0.0. Thanks, Andrew From jed at 59A2.org Thu Jun 23 04:29:32 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 23 Jun 2011 11:29:32 +0200 Subject: [petsc-users] PCCOMPOSITE and ApplyTranspose In-Reply-To: <20110623031350.AIV11763@batman.int.colorado.edu> References: <20110623031350.AIV11763@batman.int.colorado.edu> Message-ID: On Thu, Jun 23, 2011 at 11:13, Andrew T Barker wrote: > I've made a PCCOMPOSITE that contains two PCSHELLS, both of > which have a transpose set with PCShellSetApplyTranspose(). > However the composite PC does not seem to support the > transpose: > Yeah, PCApplyTranspose_Composite() has not been implemented. Are you using additive or multiplicative? Could you explain what the preconditioner is? (It's possible that there is a different way to do it that would have a nicer interface, for example, using PCFIELDSPLIT). -------------- next part -------------- An HTML attachment was scrubbed... URL: From dharmareddy84 at gmail.com Thu Jun 23 05:17:47 2011 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Thu, 23 Jun 2011 05:17:47 -0500 Subject: [petsc-users] PETSs Mesh Examples Message-ID: Hello, Any FEM examples (Fortran) using PETSC Mesh ?? Thanks Reddy -- ----------------------------------------------------- Dharmendar Reddy Palle Graduate Student Microelectronics Research center, University of Texas at Austin, 10100 Burnet Road, Bldg. 160 MER 2.608F, TX 78758-4445 e-mail: dharmareddy84 at gmail.com Phone: +1-512-350-9082 United States of America. -------------- next part -------------- An HTML attachment was scrubbed... URL: From juhaj at iki.fi Thu Jun 23 05:18:37 2011 From: juhaj at iki.fi (Juha =?utf-8?q?J=C3=A4ykk=C3=A4?=) Date: Thu, 23 Jun 2011 11:18:37 +0100 Subject: [petsc-users] SNES_DIVERGED_LS_FAILURE In-Reply-To: References: <201106230052.12923.juhaj@iki.fi> Message-ID: <201106231118.42039.juhaj@iki.fi> > Its hard to say anything without knowing what equations are being solved. I was expecting that, but the equation is rather long to write in ascii. I do have a latex version somewhere that I can send if necessary. It is horribly nonlinear, though, including a term like (d/dx f) * (d^2/dx^2) f. > However, > does the linear problem converge? If so, you could try continuation in a > parameter > in the nonlinear term. Linear solve is fine, it starts with (SNES iteration 1) [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 8.93296e-12 is less than relative tolerance 1e-05 times initial right hand side norm 9.16829 at iteration 1 and goes to (SNES iteration 107 - which is the one which diverges) [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 4.10794e-06 is less than relative tolerance 1e-05 times initial right hand side norm 1.07761 at iteration 1 Decreasing ksp_rtol to 1e-15 makes no difference, so I think the problem is not in the linear solve. As to the continuation, I was trying to avoid that since I am already doing continuation in two parameters! That is because I am scanning a 2-dim parameter space. I know an exact solution for one pair of these two (namely, a straight line) and then start changing them always using the previous result as the new initial guess. I will test the continuation in the worst nonlinear parts and see what happens. Cheers, -Juha -- ----------------------------------------------- | Juha J?ykk?, juhaj at iki.fi | | http://www.maths.leeds.ac.uk/~juhaj | ----------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. URL: From jed at 59A2.org Thu Jun 23 05:31:05 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 23 Jun 2011 12:31:05 +0200 Subject: [petsc-users] SNES_DIVERGED_LS_FAILURE In-Reply-To: <201106231118.42039.juhaj@iki.fi> References: <201106230052.12923.juhaj@iki.fi> <201106231118.42039.juhaj@iki.fi> Message-ID: On Thu, Jun 23, 2011 at 12:18, Juha J?ykk? wrote: > I was expecting that, but the equation is rather long to write in ascii. I > do > have a latex version somewhere that I can send if necessary. It is horribly > nonlinear, though, including a term like (d/dx f) * (d^2/dx^2) f. > What physical system does it represent and what sort of discretization are you using? Do you know that the equations have a solution for all values of the parameter? Even simple problems may not have solutions for all values of a parameter, or they may be non-unique. For example, see the diagram for the Bratu problem on page 2 here: http://www.nada.kth.se/kurser/kth/2D1290/comp03_3.pdf -------------- next part -------------- An HTML attachment was scrubbed... URL: From juhaj at iki.fi Thu Jun 23 05:49:44 2011 From: juhaj at iki.fi (Juha =?iso-8859-1?q?J=E4ykk=E4?=) Date: Thu, 23 Jun 2011 11:49:44 +0100 Subject: [petsc-users] SNES_DIVERGED_LS_FAILURE In-Reply-To: References: <201106230052.12923.juhaj@iki.fi> Message-ID: <201106231149.50594.juhaj@iki.fi> > As Matt notes you absolutely need to run with -pc_type lu > -ksp_monitor_true_residual -ksp_converged_reason to make sure that the Why lu? I was running with bjacobi for whatever reason, which I no longer recall. Probably gave the fastest convergence before introducing the horribly nonlinear terms. I just tried lu again: it converges, too, but snes still diverges. Running with -pc_type none requires too many KSP iterations, I had a limit of 1e6 and ran out. I will revisit this while I wait for any further replies, but as it takes so long, I will finish this reply first. > linear system solve is not the issue. It is also possible that a bug in > the function evaluation (using uninitialized values or something) is > causing the difficulty: run with valgrind Valgrind complains a LOT about openmpi, which guess is not the issue here. It also complains about PetscStrcasecmp (sic!) and some other Petsc* functions, but my code is clean: the last complaint from valgrind (apart from the summary at the end) comes from SNESSetFromOptions, which is called before I construct the initial guess. Cheers, -Juha -- ----------------------------------------------- | Juha J?ykk?, juhaj at iki.fi | | http://www.maths.leeds.ac.uk/~juhaj | ----------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. URL: From knepley at gmail.com Thu Jun 23 05:53:41 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 23 Jun 2011 05:53:41 -0500 Subject: [petsc-users] SNES_DIVERGED_LS_FAILURE In-Reply-To: <201106231149.50594.juhaj@iki.fi> References: <201106230052.12923.juhaj@iki.fi> <201106231149.50594.juhaj@iki.fi> Message-ID: On Thu, Jun 23, 2011 at 5:49 AM, Juha J?ykk? wrote: > > As Matt notes you absolutely need to run with -pc_type lu > > -ksp_monitor_true_residual -ksp_converged_reason to make sure that the > > Why lu? I was running with bjacobi for whatever reason, which I no longer > recall. Probably gave the fastest convergence before introducing the > horribly > nonlinear terms. > > I just tried lu again: it converges, too, but snes still diverges. Running > with -pc_type none requires too many KSP iterations, I had a limit of 1e6 > and > ran out. I will revisit this while I wait for any further replies, but as > it > takes so long, I will finish this reply first. > As Jed points out, there are no a priori guarantees that your system has a solution. Do you have a proof? If not, and if the linear systems converge, the most likely conclusion is that a) Your system has no solution at this parameter value or b) Your initial guess is far from the solution In either case, more theoretical investigation of the solution space is warranted. Thanks, Matt > > linear system solve is not the issue. It is also possible that a bug in > > the function evaluation (using uninitialized values or something) is > > causing the difficulty: run with valgrind > > Valgrind complains a LOT about openmpi, which guess is not the issue here. > It > also complains about PetscStrcasecmp (sic!) and some other Petsc* > functions, > but my code is clean: the last complaint from valgrind (apart from the > summary > at the end) comes from SNESSetFromOptions, which is called before I > construct > the initial guess. > > Cheers, > -Juha > > -- > ----------------------------------------------- > | Juha J?ykk?, juhaj at iki.fi | > | http://www.maths.leeds.ac.uk/~juhaj | > ----------------------------------------------- > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 23 05:59:50 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 23 Jun 2011 05:59:50 -0500 Subject: [petsc-users] PETSs Mesh Examples In-Reply-To: References: Message-ID: On Thu, Jun 23, 2011 at 5:17 AM, Dharmendar Reddy wrote: > Hello, > Any FEM examples (Fortran) using PETSC Mesh ?? > I am in the process of slowly converting Mesh from something special and experimental (in C++) to a more integrated part of PETSc (in C). There is a single example of the way it will look, SNES ex12, in dev which is mostly finished. The best completely working example of the fully C++ approach is the PyLith code (http://www.geodynamics.org/cig/software/pylith) which is a parallel FEM code for crustal deformation. FEM is wide and varied, and PETSc is not intended to include FEM per se but rather support for constructing any FEM method you would like. What do you intend to do? Thanks, Matt > Thanks > Reddy > > -- > ----------------------------------------------------- > Dharmendar Reddy Palle > Graduate Student > Microelectronics Research center, > University of Texas at Austin, > 10100 Burnet Road, Bldg. 160 > MER 2.608F, TX 78758-4445 > e-mail: dharmareddy84 at gmail.com > Phone: +1-512-350-9082 > United States of America. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dharmareddy84 at gmail.com Thu Jun 23 06:08:56 2011 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Thu, 23 Jun 2011 06:08:56 -0500 Subject: [petsc-users] PETSs Mesh Examples In-Reply-To: References: Message-ID: For learning purposes, I want to bring in a mesh form an exodus file and solve a nonlinear Poisson equation (Semiconductor devices). On Thu, Jun 23, 2011 at 5:59 AM, Matthew Knepley wrote: > On Thu, Jun 23, 2011 at 5:17 AM, Dharmendar Reddy > wrote: > >> Hello, >> Any FEM examples (Fortran) using PETSC Mesh ?? >> > > I am in the process of slowly converting Mesh from something special and > experimental (in C++) > to a more integrated part of PETSc (in C). There is a single example of the > way it will look, SNES > ex12, in dev which is mostly finished. The best completely working example > of the fully C++ > approach is the PyLith code ( > http://www.geodynamics.org/cig/software/pylith) which is a parallel > FEM code for crustal deformation. FEM is wide and varied, and PETSc is not > intended to include > FEM per se but rather support for constructing any FEM method you would > like. What do you > intend to do? > > Thanks, > > Matt > > >> Thanks >> Reddy >> >> -- >> ----------------------------------------------------- >> Dharmendar Reddy Palle >> Graduate Student >> Microelectronics Research center, >> University of Texas at Austin, >> 10100 Burnet Road, Bldg. 160 >> MER 2.608F, TX 78758-4445 >> e-mail: dharmareddy84 at gmail.com >> Phone: +1-512-350-9082 >> United States of America. >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- ----------------------------------------------------- Dharmendar Reddy Palle Graduate Student Microelectronics Research center, University of Texas at Austin, 10100 Burnet Road, Bldg. 160 MER 2.608F, TX 78758-4445 e-mail: dharmareddy84 at gmail.com Phone: +1-512-350-9082 United States of America. -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Thu Jun 23 06:10:26 2011 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Thu, 23 Jun 2011 11:10:26 +0000 Subject: [petsc-users] SNES_DIVERGED_LS_FAILURE Message-ID: you can have valgrind suppress the false errors from openmpi, see http://www.open-mpi.de/faq/?category=debugging#valgrind_clean > Date: Thu, 23 Jun 2011 11:49:44 +0100 > From: Juha J?ykk? > Subject: Re: [petsc-users] SNES_DIVERGED_LS_FAILURE > To: PETSc users list > Message-ID: <201106231149.50594.juhaj at iki.fi> > Content-Type: text/plain; charset="iso-8859-1" > Valgrind complains a LOT about openmpi dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From knepley at gmail.com Thu Jun 23 06:13:33 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 23 Jun 2011 06:13:33 -0500 Subject: [petsc-users] PETSs Mesh Examples In-Reply-To: References: Message-ID: On Thu, Jun 23, 2011 at 6:08 AM, Dharmendar Reddy wrote: > For learning purposes, I want to bring in a mesh form an exodus file and > solve a nonlinear Poisson equation (Semiconductor devices). DMMeshCreateExodus() will read in the mesh. SNES ex12 should then be able to use that mesh. It is P1 Lagrange for Bratu, which is close. Let me know if you have trouble modifying it. Thanks, Matt > On Thu, Jun 23, 2011 at 5:59 AM, Matthew Knepley wrote: > >> On Thu, Jun 23, 2011 at 5:17 AM, Dharmendar Reddy < >> dharmareddy84 at gmail.com> wrote: >> >>> Hello, >>> Any FEM examples (Fortran) using PETSC Mesh ?? >>> >> >> I am in the process of slowly converting Mesh from something special and >> experimental (in C++) >> to a more integrated part of PETSc (in C). There is a single example of >> the way it will look, SNES >> ex12, in dev which is mostly finished. The best completely working example >> of the fully C++ >> approach is the PyLith code ( >> http://www.geodynamics.org/cig/software/pylith) which is a parallel >> FEM code for crustal deformation. FEM is wide and varied, and PETSc is not >> intended to include >> FEM per se but rather support for constructing any FEM method you would >> like. What do you >> intend to do? >> >> Thanks, >> >> Matt >> >> >>> Thanks >>> Reddy >>> >>> -- >>> ----------------------------------------------------- >>> Dharmendar Reddy Palle >>> Graduate Student >>> Microelectronics Research center, >>> University of Texas at Austin, >>> 10100 Burnet Road, Bldg. 160 >>> MER 2.608F, TX 78758-4445 >>> e-mail: dharmareddy84 at gmail.com >>> Phone: +1-512-350-9082 >>> United States of America. >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > ----------------------------------------------------- > Dharmendar Reddy Palle > Graduate Student > Microelectronics Research center, > University of Texas at Austin, > 10100 Burnet Road, Bldg. 160 > MER 2.608F, TX 78758-4445 > e-mail: dharmareddy84 at gmail.com > Phone: +1-512-350-9082 > United States of America. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From juhaj at iki.fi Thu Jun 23 06:13:53 2011 From: juhaj at iki.fi (Juha =?utf-8?q?J=C3=A4ykk=C3=A4?=) Date: Thu, 23 Jun 2011 12:13:53 +0100 Subject: [petsc-users] SNES_DIVERGED_LS_FAILURE In-Reply-To: References: <201106230052.12923.juhaj@iki.fi> <201106231118.42039.juhaj@iki.fi> Message-ID: <201106231213.59092.juhaj@iki.fi> > What physical system does it represent and what sort of discretization are > you using? Please see arXiv:0809.4303 for details. The equation is obtained from the Lagrangian (4) by imposing cylindrically symmetric u with z and t appearing in complex exponential in a certain way, which decouples z and t from the planar variables. Furthermore, the angular variable vanishes (the whole point of the ansatz), leaving one with just the equation for the radial profile of u. (This is all similar to what is done in the article at Eq. (17), but the article has further constraints imposed, which eventually gives exact solutions.) After some rescaling of the domain and the codomain, one ends up with unknown g:[0,1] -> [0,1], which is what I am solving. > Do you know that the equations have a solution for all values of the > parameter? Even simple problems may not have solutions for all values of a Given the origin of the equation - a well defined Hamiltonian/Lagrangian, I would be very surprised if there were no solutions. It is hard to prove, either way, though. If I treat the problem as 3D energy minimisation one, I do find solutions (of course I do: the energy is bouded from below!), which look very much like what the diverged SNES line searches end up with, but not quite. Therefore I believe there are solutions and end up with the theory I explained in the original post. Oh, now that the equations are there, the parameters I am scanning are the product \beta e^2 and n, where n is comparable to the n in Eq (17). Obviously, n is an integer so cannot be continued as such, but \beta e^2 is real, so I start with \beta e^2 = 1, n=1, where the solution is g(y) = y. Which reminds me of another oddity: if I start with the exact solution, my function value is ~ 1e-11, so I know for sure to keep -snes_atol ~1e-10 because that's the best my function evaluation can do. Could this be the problem? Too little accuracy in function? I did try up to 6th order central differences, but it does not help. Cheers, -Juha -- ----------------------------------------------- | Juha J?ykk?, juhaj at iki.fi | | http://www.maths.leeds.ac.uk/~juhaj | ----------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. URL: From knepley at gmail.com Thu Jun 23 06:21:06 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 23 Jun 2011 06:21:06 -0500 Subject: [petsc-users] SNES_DIVERGED_LS_FAILURE In-Reply-To: <201106231213.59092.juhaj@iki.fi> References: <201106230052.12923.juhaj@iki.fi> <201106231118.42039.juhaj@iki.fi> <201106231213.59092.juhaj@iki.fi> Message-ID: On Thu, Jun 23, 2011 at 6:13 AM, Juha J?ykk? wrote: > > What physical system does it represent and what sort of discretization > are > > you using? > > Please see arXiv:0809.4303 for details. The equation is obtained from the > Lagrangian (4) by imposing cylindrically symmetric u with z and t appearing > in > complex exponential in a certain way, which decouples z and t from the > planar > variables. Furthermore, the angular variable vanishes (the whole point of > the > ansatz), leaving one with just the equation for the radial profile of u. > (This > is all similar to what is done in the article at Eq. (17), but the article > has > further constraints imposed, which eventually gives exact solutions.) > > After some rescaling of the domain and the codomain, one ends up with > unknown > g:[0,1] -> [0,1], which is what I am solving. > > > Do you know that the equations have a solution for all values of the > > parameter? Even simple problems may not have solutions for all values of > a > > Given the origin of the equation - a well defined Hamiltonian/Lagrangian, I > would be very surprised if there were no solutions. It is hard to prove, > either way, though. If I treat the problem as 3D energy minimisation one, I > do > find solutions (of course I do: the energy is bouded from below!), which > look > very much like what the diverged SNES line searches end up with, but not > quite. Therefore I believe there are solutions and end up with the theory I > explained in the original post. > > Oh, now that the equations are there, the parameters I am scanning are the > product \beta e^2 and n, where n is comparable to the n in Eq (17). > Obviously, > n is an integer so cannot be continued as such, but \beta e^2 is real, so I > start with \beta e^2 = 1, n=1, where the solution is g(y) = y. > You could relax integrality. > Which reminds me of another oddity: if I start with the exact solution, my > function value is ~ 1e-11, so I know for sure to keep -snes_atol ~1e-10 > because that's the best my function evaluation can do. Could this be the > problem? Too little accuracy in function? I did try up to 6th order central > differences, but it does not help. > It is possible for the discrete equation to have no real solutions, while the continuous equation does. Even if it is expensive, I suggest continuing in the nonlinearity to try to get to a solution. If you find one, it could give you insight into designing a search strategy that will work for your equation. Matt > Cheers, > -Juha > > -- > ----------------------------------------------- > | Juha J?ykk?, juhaj at iki.fi | > | http://www.maths.leeds.ac.uk/~juhaj | > ----------------------------------------------- > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dharmareddy84 at gmail.com Thu Jun 23 06:26:17 2011 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Thu, 23 Jun 2011 06:26:17 -0500 Subject: [petsc-users] PETSs Mesh Examples In-Reply-To: References: Message-ID: Where is the ex12 located? I do not find it in snes/example/tutorials/ . Can you please send me the src file On Thu, Jun 23, 2011 at 6:13 AM, Matthew Knepley wrote: > On Thu, Jun 23, 2011 at 6:08 AM, Dharmendar Reddy > wrote: > >> For learning purposes, I want to bring in a mesh form an exodus file and >> solve a nonlinear Poisson equation (Semiconductor devices). > > > DMMeshCreateExodus() will read in the mesh. SNES ex12 should then be able > to use that mesh. It is P1 > Lagrange for Bratu, which is close. Let me know if you have trouble > modifying it. > > Thanks, > > Matt > > >> On Thu, Jun 23, 2011 at 5:59 AM, Matthew Knepley wrote: >> >>> On Thu, Jun 23, 2011 at 5:17 AM, Dharmendar Reddy < >>> dharmareddy84 at gmail.com> wrote: >>> >>>> Hello, >>>> Any FEM examples (Fortran) using PETSC Mesh ?? >>>> >>> >>> I am in the process of slowly converting Mesh from something special and >>> experimental (in C++) >>> to a more integrated part of PETSc (in C). There is a single example of >>> the way it will look, SNES >>> ex12, in dev which is mostly finished. The best completely working >>> example of the fully C++ >>> approach is the PyLith code ( >>> http://www.geodynamics.org/cig/software/pylith) which is a parallel >>> FEM code for crustal deformation. FEM is wide and varied, and PETSc is >>> not intended to include >>> FEM per se but rather support for constructing any FEM method you would >>> like. What do you >>> intend to do? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks >>>> Reddy >>>> >>>> -- >>>> ----------------------------------------------------- >>>> Dharmendar Reddy Palle >>>> Graduate Student >>>> Microelectronics Research center, >>>> University of Texas at Austin, >>>> 10100 Burnet Road, Bldg. 160 >>>> MER 2.608F, TX 78758-4445 >>>> e-mail: dharmareddy84 at gmail.com >>>> Phone: +1-512-350-9082 >>>> United States of America. >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> ----------------------------------------------------- >> Dharmendar Reddy Palle >> Graduate Student >> Microelectronics Research center, >> University of Texas at Austin, >> 10100 Burnet Road, Bldg. 160 >> MER 2.608F, TX 78758-4445 >> e-mail: dharmareddy84 at gmail.com >> Phone: +1-512-350-9082 >> United States of America. >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- ----------------------------------------------------- Dharmendar Reddy Palle Graduate Student Microelectronics Research center, University of Texas at Austin, 10100 Burnet Road, Bldg. 160 MER 2.608F, TX 78758-4445 e-mail: dharmareddy84 at gmail.com Phone: +1-512-350-9082 United States of America. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Thu Jun 23 07:53:53 2011 From: john.fettig at gmail.com (John Fettig) Date: Thu, 23 Jun 2011 08:53:53 -0400 Subject: [petsc-users] Size of off-diagonal matrix In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 10:30 PM, Barry Smith wrote: > > On Jun 22, 2011, at 9:02 AM, John Fettig wrote: > >> On Wed, Jun 22, 2011 at 9:56 AM, Matthew Knepley wrote: >>> It is compressed, so it only has columns that have a nonzero. >> >> Thanks for answering so quickly. ?I had just assumed that it would >> logically still report sizes that added up to the global column size >> so I was confused. > > ? What do you mean "report"? The sequential matrix that represents the off-diagonal part of the matrix actually does not contain the "zero columns" so does have fewer columns and hence the smaller number. I mean if you run with "-mat_view_info", it tells you the size of the sequential matrices (which are the sizes I gave in the initial email). I had expected a couple of things: 1) the logical row size would be (global size - diagonal block size). 2) the indices would follow similar logic, i.e. entries before the diagonal block would have the same column number as in the global matrix, and entries after the diagonal would have (column number - diagonal block size). I was wrong on both counts, and I still don't understand how the column numbering for the off-diagonal block is calculated. It appears you are doing something more than just compressing the columns, you are also shifting the indices if an entire column is zero? Is there anyplace in the documentation that would help me understand this? Alternatively, where would I look in mpiaij.c? Thanks, John From jed at 59A2.org Thu Jun 23 06:32:46 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 23 Jun 2011 13:32:46 +0200 Subject: [petsc-users] PETSs Mesh Examples In-Reply-To: References: Message-ID: On Thu, Jun 23, 2011 at 13:26, Dharmendar Reddy wrote: > Where is the ex12 located? I do not find it in snes/example/tutorials/ You'll need to download petsc-dev, that is the correct location. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 23 06:29:40 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 23 Jun 2011 06:29:40 -0500 Subject: [petsc-users] PETSs Mesh Examples In-Reply-To: References: Message-ID: On Thu, Jun 23, 2011 at 6:26 AM, Dharmendar Reddy wrote: > Where is the ex12 located? I do not find it in snes/example/tutorials/ . > Can you please send me the src file You would need petsc-dev: http://www.mcs.anl.gov/petsc/petsc-as/developers/index.html Matt > On Thu, Jun 23, 2011 at 6:13 AM, Matthew Knepley wrote: > >> On Thu, Jun 23, 2011 at 6:08 AM, Dharmendar Reddy < >> dharmareddy84 at gmail.com> wrote: >> >>> For learning purposes, I want to bring in a mesh form an exodus file and >>> solve a nonlinear Poisson equation (Semiconductor devices). >> >> >> DMMeshCreateExodus() will read in the mesh. SNES ex12 should then be able >> to use that mesh. It is P1 >> Lagrange for Bratu, which is close. Let me know if you have trouble >> modifying it. >> >> Thanks, >> >> Matt >> >> >>> On Thu, Jun 23, 2011 at 5:59 AM, Matthew Knepley wrote: >>> >>>> On Thu, Jun 23, 2011 at 5:17 AM, Dharmendar Reddy < >>>> dharmareddy84 at gmail.com> wrote: >>>> >>>>> Hello, >>>>> Any FEM examples (Fortran) using PETSC Mesh ?? >>>>> >>>> >>>> I am in the process of slowly converting Mesh from something special and >>>> experimental (in C++) >>>> to a more integrated part of PETSc (in C). There is a single example of >>>> the way it will look, SNES >>>> ex12, in dev which is mostly finished. The best completely working >>>> example of the fully C++ >>>> approach is the PyLith code ( >>>> http://www.geodynamics.org/cig/software/pylith) which is a parallel >>>> FEM code for crustal deformation. FEM is wide and varied, and PETSc is >>>> not intended to include >>>> FEM per se but rather support for constructing any FEM method you would >>>> like. What do you >>>> intend to do? >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thanks >>>>> Reddy >>>>> >>>>> -- >>>>> ----------------------------------------------------- >>>>> Dharmendar Reddy Palle >>>>> Graduate Student >>>>> Microelectronics Research center, >>>>> University of Texas at Austin, >>>>> 10100 Burnet Road, Bldg. 160 >>>>> MER 2.608F, TX 78758-4445 >>>>> e-mail: dharmareddy84 at gmail.com >>>>> Phone: +1-512-350-9082 >>>>> United States of America. >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >>> >>> -- >>> ----------------------------------------------------- >>> Dharmendar Reddy Palle >>> Graduate Student >>> Microelectronics Research center, >>> University of Texas at Austin, >>> 10100 Burnet Road, Bldg. 160 >>> MER 2.608F, TX 78758-4445 >>> e-mail: dharmareddy84 at gmail.com >>> Phone: +1-512-350-9082 >>> United States of America. >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > ----------------------------------------------------- > Dharmendar Reddy Palle > Graduate Student > Microelectronics Research center, > University of Texas at Austin, > 10100 Burnet Road, Bldg. 160 > MER 2.608F, TX 78758-4445 > e-mail: dharmareddy84 at gmail.com > Phone: +1-512-350-9082 > United States of America. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Thu Jun 23 09:25:53 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 23 Jun 2011 16:25:53 +0200 Subject: [petsc-users] Size of off-diagonal matrix In-Reply-To: References: Message-ID: On Thu, Jun 23, 2011 at 14:53, John Fettig wrote: > I was wrong on both counts, and I still don't understand how the > column numbering for the off-diagonal block is calculated. It appears > you are doing something more than just compressing the columns, you > are also shifting the indices if an entire column is zero? > Yeah, that's what we mean by "compressing the columns". > Is there > anyplace in the documentation that would help me understand this? > Alternatively, where would I look in mpiaij.c? > The MPIAIJ format works with blocks that are in SeqAIJ format. In most cases, it's not necessary to be familiar with the internal storage format. Are you just curious how it's implemented, or is there something you are trying to do that depends on the storage format? -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Thu Jun 23 09:47:11 2011 From: john.fettig at gmail.com (John Fettig) Date: Thu, 23 Jun 2011 10:47:11 -0400 Subject: [petsc-users] Size of off-diagonal matrix In-Reply-To: References: Message-ID: On Thu, Jun 23, 2011 at 10:25 AM, Jed Brown wrote: > On Thu, Jun 23, 2011 at 14:53, John Fettig wrote: >> ?Is there >> anyplace in the documentation that would help me understand this? >> Alternatively, where would I look in mpiaij.c? > > The MPIAIJ format works with blocks that are in SeqAIJ format. In most > cases, it's not necessary to be familiar with the internal storage format. > Are you just curious how it's implemented, or is there something you are > trying to do that depends on the storage format? I'm trying to use MatCreateMPIAIJWithSplitArrays, but it isn't clear to me how the off-diagonal portion is supposed to be constructed. John From jed at 59A2.org Thu Jun 23 09:50:39 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 23 Jun 2011 16:50:39 +0200 Subject: [petsc-users] Size of off-diagonal matrix In-Reply-To: References: Message-ID: On Thu, Jun 23, 2011 at 16:47, John Fettig wrote: > I'm trying to use MatCreateMPIAIJWithSplitArrays, but it isn't clear > to me how the off-diagonal portion is supposed to be constructed. > Okay, it's defined by compressing all the sorted off-process entries and then defining the sequential matrix operating on those values. Unless you already happen to use the same format, you're better off using MatSetValues(). -------------- next part -------------- An HTML attachment was scrubbed... URL: From milan.v.mitrovic at gmail.com Thu Jun 23 09:58:13 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Thu, 23 Jun 2011 16:58:13 +0200 Subject: [petsc-users] Ghost communication In-Reply-To: References: Message-ID: Well, I found where the problem is. I wanted to us KSP to solve a system for which I created a matrix-free matrix, and I provide the MatMult method. I created the vectors the same way I showed before, but when I ran KSPSolve I found that the ghost communication did not happen automatically during the solve. So I tried to add it to the multiplication method and it did not work. Thats when I sent the e-mail... But I found out in the meantime that when I call MatMult(A,b,x,info) the communication does take place! So I guess my real question is how should I get KSPSolve to update the ghosts while solving a system? (before or after every multiplication) From john.fettig at gmail.com Thu Jun 23 09:59:15 2011 From: john.fettig at gmail.com (John Fettig) Date: Thu, 23 Jun 2011 10:59:15 -0400 Subject: [petsc-users] Size of off-diagonal matrix In-Reply-To: References: Message-ID: On Thu, Jun 23, 2011 at 10:50 AM, Jed Brown wrote: > On Thu, Jun 23, 2011 at 16:47, John Fettig wrote: >> >> I'm trying to use MatCreateMPIAIJWithSplitArrays, but it isn't clear >> to me how the off-diagonal portion is supposed to be constructed. > > Okay, it's defined by compressing all the sorted off-process entries and > then defining the sequential matrix operating on those values. I understand everything you say here except what "compressing" means. It is compressed by virtue of being in CSR format, but then you do some additional mapping with the column indices that further compresses it? What purpose does changing the column indices serve? > Unless you already happen to use the same format, you're better off using > MatSetValues(). We already have a distributed CSR matrix that works with MatCreateMPIAIJWithArrays, but I was hoping to avoid the copy if possible. If it is too difficult for me to understand how the off-diagonal matrix is stored, then I will just stick with the copying version. John From jed at 59A2.org Thu Jun 23 10:03:05 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 23 Jun 2011 17:03:05 +0200 Subject: [petsc-users] Size of off-diagonal matrix In-Reply-To: References: Message-ID: On Thu, Jun 23, 2011 at 16:59, John Fettig wrote: > I understand everything you say here except what "compressing" means. > It is compressed by virtue of being in CSR format, but then you do > some additional mapping with the column indices that further > compresses it? What purpose does changing the column indices serve? > It operates on the dense vector consisting of only those entries on other processors that interact with the current one. If this thing operated on the full global vector, there would either need to be a matrix format that could operator on "sparse vectors" (i.e. an additional layer of indirection) or the whole dense global vector would need to be gathered to every process which would be horribly non-scalable. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Thu Jun 23 10:06:44 2011 From: john.fettig at gmail.com (John Fettig) Date: Thu, 23 Jun 2011 11:06:44 -0400 Subject: [petsc-users] Size of off-diagonal matrix In-Reply-To: References: Message-ID: On Thu, Jun 23, 2011 at 11:03 AM, Jed Brown wrote: > On Thu, Jun 23, 2011 at 16:59, John Fettig wrote: >> >> I understand everything you say here except what "compressing" means. >> It is compressed by virtue of being in CSR format, but then you do >> some additional mapping with the column indices that further >> compresses it? What purpose does changing the column indices serve? > > It operates on the dense vector consisting of only those entries on other > processors that interact with the current one. If this thing operated on the > full global vector, there would either need to be a matrix format that could > operator on "sparse vectors" (i.e. an additional layer of indirection) or > the whole dense global vector would need to be gathered to every process > which would be horribly non-scalable. Thanks for the explanation, it makes sense. John From john.fettig at gmail.com Thu Jun 23 10:09:21 2011 From: john.fettig at gmail.com (John Fettig) Date: Thu, 23 Jun 2011 11:09:21 -0400 Subject: [petsc-users] Size of off-diagonal matrix In-Reply-To: References: Message-ID: On Thu, Jun 23, 2011 at 11:06 AM, John Fettig wrote: > On Thu, Jun 23, 2011 at 11:03 AM, Jed Brown wrote: >> On Thu, Jun 23, 2011 at 16:59, John Fettig wrote: >>> >>> I understand everything you say here except what "compressing" means. >>> It is compressed by virtue of being in CSR format, but then you do >>> some additional mapping with the column indices that further >>> compresses it? What purpose does changing the column indices serve? >> >> It operates on the dense vector consisting of only those entries on other >> processors that interact with the current one. If this thing operated on the >> full global vector, there would either need to be a matrix format that could >> operator on "sparse vectors" (i.e. an additional layer of indirection) or >> the whole dense global vector would need to be gathered to every process >> which would be horribly non-scalable. > > Thanks for the explanation, it makes sense. Is there any way to access the map from global numbering to the compressed numbering? John From jed at 59A2.org Thu Jun 23 10:11:09 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 23 Jun 2011 17:11:09 +0200 Subject: [petsc-users] Ghost communication In-Reply-To: References: Message-ID: On Thu, Jun 23, 2011 at 16:58, Milan Mitrovic wrote: > But I found out in the meantime that when I call > MatMult(A,b,x,info) the communication does take place! > > So I guess my real question is how should I get KSPSolve to update the > ghosts while solving a system? (before or after every multiplication) > KSPSolve does not do any "ghost update" for you, you have to do the communication inside of your MatMult() implementation. Can you paste that function so we can see what you're doing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Thu Jun 23 10:17:13 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 23 Jun 2011 17:17:13 +0200 Subject: [petsc-users] Size of off-diagonal matrix In-Reply-To: References: Message-ID: On Thu, Jun 23, 2011 at 17:09, John Fettig wrote: > Is there any way to access the map from global numbering to the > compressed numbering? > 1. Certainly not before you create the matrix. 2. You have this information in some form if you can call MatCreateMPIAIJWithArrays(). Scan through the whole local block and make a sorted list of all the off-process entries. Then scan through again and look up each off-process entry in that sorted list to get its index. 3. But this is a waste of time, just call MatCreateMPIAIJWithArrays() or (better) use MatSetValues(). -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Thu Jun 23 10:19:00 2011 From: john.fettig at gmail.com (John Fettig) Date: Thu, 23 Jun 2011 11:19:00 -0400 Subject: [petsc-users] Size of off-diagonal matrix In-Reply-To: References: Message-ID: On Thu, Jun 23, 2011 at 11:17 AM, Jed Brown wrote: > On Thu, Jun 23, 2011 at 17:09, John Fettig wrote: >> >> Is there any way to access the map from global numbering to the >> compressed numbering? > > 1. Certainly not before you create the matrix. > 2. You have this information in some form if you can call > MatCreateMPIAIJWithArrays(). Scan through the whole local block and make a > sorted list of all the off-process entries. Then scan through again and look > up each off-process entry in that sorted list to get its index. > 3. But this is a waste of time, just call MatCreateMPIAIJWithArrays() or > (better) use MatSetValues(). Ok, thanks! John From milan.v.mitrovic at gmail.com Thu Jun 23 10:38:24 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Thu, 23 Jun 2011 17:38:24 +0200 Subject: [petsc-users] Ghost communication In-Reply-To: References: Message-ID: SUBROUTINE dcop_mult(A,x,y,info) ! args Mat A Vec x Vec y INTEGER, INTENT( OUT) :: info ! vars INTEGER :: id info = 0 ! update ghosts CALL VecGhostUpdateBegin(x,INSERT_VALUES,SCATTER_FORWARD,info) CALL VecGhostUpdateEnd(x,INSERT_VALUES,SCATTER_FORWARD,info) CALL VecGhostUpdateBegin(y,INSERT_VALUES,SCATTER_FORWARD,info) CALL VecGhostUpdateEnd(y,INSERT_VALUES,SCATTER_FORWARD,info) ! get matrix context CALL MatShellGetContext(A,id,info) ! apply dcops CALL particles_dcop_apply_petsc(AParticles,x,y,id,info) END SUBROUTINE dcop_mult From jed at 59A2.org Thu Jun 23 10:41:06 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 23 Jun 2011 17:41:06 +0200 Subject: [petsc-users] Ghost communication In-Reply-To: References: Message-ID: On Thu, Jun 23, 2011 at 17:38, Milan Mitrovic wrote: > SUBROUTINE dcop_mult(A,x,y,info) > ! args > Mat A > Vec x > Vec y > INTEGER, INTENT( OUT) :: info > ! vars > INTEGER :: id > info = 0 > ! update ghosts > CALL VecGhostUpdateBegin(x,INSERT_VALUES,SCATTER_FORWARD,info) > CALL VecGhostUpdateEnd(x,INSERT_VALUES,SCATTER_FORWARD,info) > CALL VecGhostUpdateBegin(y,INSERT_VALUES,SCATTER_FORWARD,info) > CALL VecGhostUpdateEnd(y,INSERT_VALUES,SCATTER_FORWARD,info) > y is an output-only argument, it doesn't make sense to do this second scatter. > ! get matrix context > CALL MatShellGetContext(A,id,info) > ! apply dcops > CALL particles_dcop_apply_petsc(AParticles,x,y,id,info) > END SUBROUTINE dcop_mult > Where do you call VecGhostGetLocalForm()? -------------- next part -------------- An HTML attachment was scrubbed... URL: From milanm at student.ethz.ch Thu Jun 23 11:29:34 2011 From: milanm at student.ethz.ch (Milan Mitrovic) Date: Thu, 23 Jun 2011 18:29:34 +0200 Subject: [petsc-users] Ghost communication In-Reply-To: References: Message-ID: I thought at some point that I maybe have the order of the arguments wrong... The first few lines in particles_dcop_apply_petsc are: ! get ghosted versions of vectors CALL VecGhostGetLocalForm(from,xl,info) CALL VecSet(to,0.0_MK,info) ! get arrays from petsc vectors CALL VecGetArray(xl,xx,xxi,info) CALL VecGetArray(to,yy,yyi,info) CALL VecGetLocalSize(xl,xxn,info) CALL VecGetLocalSize(to,yyn,info) and its closed with: ! release petsc arrays CALL VecRestoreArray(xl,xx,xxi,info) CALL VecRestoreArray(to,yy,yyi,info) ! release local form of vectors CALL VecGhostRestoreLocalForm(from,xl,info) From xdliang at gmail.com Thu Jun 23 15:42:21 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Thu, 23 Jun 2011 16:42:21 -0400 Subject: [petsc-users] Fwd: superlu_dist with same_nonzero_pattern In-Reply-To: References: Message-ID: Thanks, Hong. I tried the runtime option -mat_superlu_dist_equil NO. However, the problem is still there. I've upload my short codes here: http://math.mit.edu/~xdliang/superlu_dist_test.zip In this code, I first generate my sparse matrix M with dimension N-by-N, then solve Mx=J for a few times. Each time, I only modify the entries of two diagonals: (i,i) and ( i, (i+N/2)%N). Althougth I modify these entries, I thought the nonzero pattern should not change. Based on my tests, I found that: 1 superlu_dist works fine with DIFFERENT_NONZERO_PATTERN. However, if I switch to SAME_NONZERO_PATTERN, the program either crashes (with np >1 ) or does not converge ( with single processor). 2 pastix works fine with SAME_NONZERO_PATTERN, but it halts there if I switch to DIFFERENT_NONZERO_PATTERN ( with np>1). 3 spooles works fine with both SAME and DIFFERENT nonzero pattern. Can you give me some hints why superlu_dist fails on my matrix with SAME_NONZERO_PATTERN? I tried both superlu_dist v2.4 and v2.5 ( latest one). Thanks. Best, Xiangdong On Wed, Jun 22, 2011 at 2:06 PM, Hong Zhang wrote: > Xiangdong : > > Does it hangs inside superlu_dist? > Try runtime option '-mat_superlu_dist_equil NO' > > If you can send us a stand-alone short code or your matrix in petsc > binary format that repeats this behavior, then we can investigate it. > Likely memory problem. > > Hong > > >> Hello everyone, >> >> I had some problem in using superlu_dist with same_nonzero_pattern. >> Sometimes superlu_dist solver halts there and never finishes. However, >> when I switched to different_nonzero_pattern or other solvers, the >> program works nicely. ?Ding had reported this problem a few months >> ago: >> ?http://lists.mcs.anl.gov/pipermail/petsc-users/2011-January/007623.html >> >> Following Barry's suggestion, I checked that the nonzero pattern of my >> matrix is still the same via >> ?ierr=MatSetOption(M, MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE); CHKERRQ(ierr); >> >> I tried two other options: >> >> 1 First install the latest superlu_dist (v2.5), then compile petsc >> with this new superlu_dist package. However, the problem is still >> there, and the performance is even worse ( comparing to the option >> --download-superlu_dist=yes (v2.4)). >> >> 2 I compiled petsc-dev with --download-superlu_dist. However, I got >> some errors when compiling my program with petsc-dev. It said that >> >> ResonatorOpt.c: In function ?main?: >> ResonatorOpt.c:154: error: ?MAT_SOLVER_SUPERLU_DIST? undeclared (first >> use in this function) >> ResonatorOpt.c:154: error: (Each undeclared identifier is reported only once >> ResonatorOpt.c:154: error: for each function it appears in.) >> >> Besides superlu_dist, MAT_SOLVER_PASTIX and MAT_SOLVER_PASTIX are >> undeclared either. Am I missing some header files? I used the same >> procedure as I complied regular petsc, which had no such problems. ?(I >> tried to attach the configure log in a previous mail, but it is >> bounced back.) >> >> Can you give me some suggestions on using superlu_dist with >> same_nonzero_pattern? Thank you. >> >> Best, >> Xiangdong >> > From hzhang at mcs.anl.gov Thu Jun 23 22:27:03 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Thu, 23 Jun 2011 22:27:03 -0500 Subject: [petsc-users] Fwd: superlu_dist with same_nonzero_pattern In-Reply-To: References: Message-ID: Xiangdong : I tested your code. Here is what I get: 1. your code does not build. I made following changes: < PetscErrorCode MoperatorGeneral(MPI_Comm comm, int Nx, int Ny, int Nz, Vec muinvpml,Mat *Aout) --- > Mat MoperatorGeneral(MPI_Comm comm, int Nx, int Ny, int Nz, Vec muinvpml) and get it build on an iMac and a linux machine 2. code hangs at if (rank == 0) fgetc(stdin); I comment it out. 3. then I can reproduce your error with superlu_dist, np=2, and SAME_NONZERO_PATTERN. KSPSetOperators(ksp,M,M,SAME_NONZERO_PATTERN) should be called once, not inside your iteration loop. Move it out of the loop fix the problem. Why other packages work? I do not know :-( 4. the modified code works well on the linux machine with np=1, 2, 4 etc. However, on iMac, even with np=1, at the end of 1st iteration, I get [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Infinite or not-a-number generated in norm! ... [0]PETSC ERROR: VecNorm() line 167 in src/vec/vec/interface/rvector.c [0]PETSC ERROR: main() line 93 in test.c Checking solution x, I see '-inf' in several components. Using mumps ('-pc_factor_mat_solver_package mumps') with np=2, I get same error. However, sequential superlu, mumps and petsc all run well. It is likely your model is numerically sensitive, not problem with these packages. The modified code is attached. Good luck! Hong > Thanks, Hong. I tried the runtime option -mat_superlu_dist_equil NO. > However, the problem is still there. > > I've upload my short codes here: > http://math.mit.edu/~xdliang/superlu_dist_test.zip > > In this code, I first generate my sparse matrix M with dimension > N-by-N, then solve Mx=J for a few times. Each time, I only modify the > entries of two diagonals: (i,i) and ?( i, (i+N/2)%N). Althougth I > modify these entries, I thought the nonzero pattern should not change. > Based on my tests, I found that: > > 1 superlu_dist works fine with DIFFERENT_NONZERO_PATTERN. However, if > I switch to SAME_NONZERO_PATTERN, the program either crashes (with np >>1 ) or does not converge ( with single processor). > > 2 pastix works fine with SAME_NONZERO_PATTERN, but it halts there if I > switch to DIFFERENT_NONZERO_PATTERN ( with np>1). > > 3 spooles works fine with both SAME and DIFFERENT nonzero pattern. > > Can you give me some hints why superlu_dist fails on my matrix with > SAME_NONZERO_PATTERN? I tried both superlu_dist v2.4 and v2.5 ( latest > one). Thanks. > > Best, > Xiangdong > > > > On Wed, Jun 22, 2011 at 2:06 PM, Hong Zhang wrote: >> Xiangdong : >> >> Does it hangs inside superlu_dist? >> Try runtime option '-mat_superlu_dist_equil NO' >> >> If you can send us a stand-alone short code or your matrix in petsc >> binary format that repeats this behavior, then we can investigate it. >> Likely memory problem. >> >> Hong >> >> >>> Hello everyone, >>> >>> I had some problem in using superlu_dist with same_nonzero_pattern. >>> Sometimes superlu_dist solver halts there and never finishes. However, >>> when I switched to different_nonzero_pattern or other solvers, the >>> program works nicely. ?Ding had reported this problem a few months >>> ago: >>> ?http://lists.mcs.anl.gov/pipermail/petsc-users/2011-January/007623.html >>> >>> Following Barry's suggestion, I checked that the nonzero pattern of my >>> matrix is still the same via >>> ?ierr=MatSetOption(M, MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE); CHKERRQ(ierr); >>> >>> I tried two other options: >>> >>> 1 First install the latest superlu_dist (v2.5), then compile petsc >>> with this new superlu_dist package. However, the problem is still >>> there, and the performance is even worse ( comparing to the option >>> --download-superlu_dist=yes (v2.4)). >>> >>> 2 I compiled petsc-dev with --download-superlu_dist. However, I got >>> some errors when compiling my program with petsc-dev. It said that >>> >>> ResonatorOpt.c: In function ?main?: >>> ResonatorOpt.c:154: error: ?MAT_SOLVER_SUPERLU_DIST? undeclared (first >>> use in this function) >>> ResonatorOpt.c:154: error: (Each undeclared identifier is reported only once >>> ResonatorOpt.c:154: error: for each function it appears in.) >>> >>> Besides superlu_dist, MAT_SOLVER_PASTIX and MAT_SOLVER_PASTIX are >>> undeclared either. Am I missing some header files? I used the same >>> procedure as I complied regular petsc, which had no such problems. ?(I >>> tried to attach the configure log in a previous mail, but it is >>> bounced back.) >>> >>> Can you give me some suggestions on using superlu_dist with >>> same_nonzero_pattern? Thank you. >>> >>> Best, >>> Xiangdong >>> >> > -------------- next part -------------- A non-text attachment was scrubbed... Name: superlu_dist_test2.tar.gz Type: application/x-gzip Size: 3750 bytes Desc: not available URL: From milanm at student.ethz.ch Fri Jun 24 02:38:19 2011 From: milanm at student.ethz.ch (Milan Mitrovic) Date: Fri, 24 Jun 2011 09:38:19 +0200 Subject: [petsc-users] Ghost communication In-Reply-To: References: Message-ID: So, any ideas? when I call MatMult(A,b,x,info) the ghosts get updated. When I call KSPSolve(ksp,b,x,info) the ghosts dont get updated (the multiplication routine is called... I checked that)... It seems that KSP calles MatMult with different vectors... not the original b and x... From knepley at gmail.com Fri Jun 24 08:23:49 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 24 Jun 2011 08:23:49 -0500 Subject: [petsc-users] Ghost communication In-Reply-To: References: Message-ID: On Fri, Jun 24, 2011 at 2:38 AM, Milan Mitrovic wrote: > So, any ideas? > > when I call MatMult(A,b,x,info) the ghosts get updated. When I call > KSPSolve(ksp,b,x,info) the ghosts dont get updated (the multiplication > routine is called... I checked that)... It seems that KSP calles > MatMult with different vectors... not the original b and x... > Yes, that definitely happens. You cannot count on the same vector being used internally. They are the same type (ghosted), but they may come from VecDuplicate/Copy(), so anytime you need ghost values, you need to update. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From xdliang at gmail.com Fri Jun 24 10:20:46 2011 From: xdliang at gmail.com (Xiangdong Liang) Date: Fri, 24 Jun 2011 11:20:46 -0400 Subject: [petsc-users] Fwd: superlu_dist with same_nonzero_pattern In-Reply-To: References: Message-ID: Thanks for helping me check the code. I agree that moving the KSPSetOperators(ksp,M,M,SAME_NONZERO_PATTERN) outside the iteration loop lifted the crash of superlu_dist. However, I have the following concerns: 1) If we only call KSPSetOperators once outside the loop, the ksp solver would be the same for all the iterations, although matrix M is changing during each iteration. I added the following code to manually check the error of the solution, ierr = MatMult(M,x, xdiff);CHKERRQ(ierr); ierr = VecAXPY(xdiff,-1.0,J);CHKERRQ(ierr); ierr = VecNorm(xdiff,NORM_2,&norm);CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"---Norm of error %A----\n ",norm);CHKERRQ(ierr); As far as I get, the error are huge (1e+6) except the first iteration. It seems that it keeps solving Mx=J by using the non-updated M. 2) Actually, I really want to switch between SAME_NONZERO_PATTERN and SAME_PRECONDITIONER. In my problem, matrix M changes only a little bit during each step. I am planing to only use the sparse direct solver every 10 steps, because the LU decomposition is very expensive. The other steps I simply use the iterative methods (GMRES) with the preconditioner obtained in the previous LU decomposition. As we tested, Pastix performs well for this switch. It would be nice if superlu_dist also works for this switch. The other sparse direct solvers are just too slow for my 3D problem. 3) I am also think to use the low level functions like: Outside the loop: MatGetOrdering(Mat matrix,MatOrderingType type,IS* rowperm,IS* colperm); MatGetFactor(Mat matrix,const MatSolverPackage package,MatFactorType ftype,Mat *factor); MatLUFactorSymbolic(Mat factor,Mat matrix,IS rowperm,IS colperm,const MatFactorInfo *info); Inside the loop: MatLUFactorNumeric(Mat factor,Mat matrix,const MatFactorInfo *info); MatSolve(Mat A,Vec x, Vec y); Are they equivalent to SAME_NONZERO_PATTERN? Do you have an idea whether it will perform well or worse than the ksp version? Thank you very much. Xiangdong On Thu, Jun 23, 2011 at 11:27 PM, Hong Zhang wrote: > Xiangdong : > I tested your code. Here is what I get: > > 1. your code does not build. I made following changes: > < PetscErrorCode MoperatorGeneral(MPI_Comm comm, int Nx, int Ny, int > Nz, Vec muinvpml,Mat *Aout) > --- >> Mat MoperatorGeneral(MPI_Comm comm, int Nx, int Ny, int Nz, Vec muinvpml) > > and get it build on an iMac and a linux machine > > 2. code hangs at > if (rank == 0) fgetc(stdin); > I comment it out. > > 3. then I can reproduce your error with superlu_dist, np=2, and > SAME_NONZERO_PATTERN. > KSPSetOperators(ksp,M,M,SAME_NONZERO_PATTERN) > should be called once, not inside your iteration loop. Move it out of > the loop fix the problem. > Why other packages work? I do not know :-( > > 4. the modified code works well on the linux machine with np=1, 2, 4 etc. > However, on iMac, even with np=1, at the end of 1st iteration, I get > > ?[0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Infinite or not-a-number generated in norm! > ... > [0]PETSC ERROR: VecNorm() line 167 in src/vec/vec/interface/rvector.c > [0]PETSC ERROR: main() line 93 in test.c > > Checking solution x, I see > '-inf' in several components. > Using mumps ('-pc_factor_mat_solver_package mumps') with np=2, I get same error. > However, sequential superlu, mumps and petsc all run well. > It is likely your model is numerically sensitive, not problem with > these packages. > > The modified code is attached. Good luck! > > Hong > > > >> Thanks, Hong. I tried the runtime option -mat_superlu_dist_equil NO. >> However, the problem is still there. >> >> I've upload my short codes here: >> http://math.mit.edu/~xdliang/superlu_dist_test.zip >> >> In this code, I first generate my sparse matrix M with dimension >> N-by-N, then solve Mx=J for a few times. Each time, I only modify the >> entries of two diagonals: (i,i) and ?( i, (i+N/2)%N). Althougth I >> modify these entries, I thought the nonzero pattern should not change. >> Based on my tests, I found that: >> >> 1 superlu_dist works fine with DIFFERENT_NONZERO_PATTERN. However, if >> I switch to SAME_NONZERO_PATTERN, the program either crashes (with np >>>1 ) or does not converge ( with single processor). >> >> 2 pastix works fine with SAME_NONZERO_PATTERN, but it halts there if I >> switch to DIFFERENT_NONZERO_PATTERN ( with np>1). >> >> 3 spooles works fine with both SAME and DIFFERENT nonzero pattern. >> >> Can you give me some hints why superlu_dist fails on my matrix with >> SAME_NONZERO_PATTERN? I tried both superlu_dist v2.4 and v2.5 ( latest >> one). Thanks. >> >> Best, >> Xiangdong >> >> >> >> On Wed, Jun 22, 2011 at 2:06 PM, Hong Zhang wrote: >>> Xiangdong : >>> >>> Does it hangs inside superlu_dist? >>> Try runtime option '-mat_superlu_dist_equil NO' >>> >>> If you can send us a stand-alone short code or your matrix in petsc >>> binary format that repeats this behavior, then we can investigate it. >>> Likely memory problem. >>> >>> Hong >>> >>> >>>> Hello everyone, >>>> >>>> I had some problem in using superlu_dist with same_nonzero_pattern. >>>> Sometimes superlu_dist solver halts there and never finishes. However, >>>> when I switched to different_nonzero_pattern or other solvers, the >>>> program works nicely. ?Ding had reported this problem a few months >>>> ago: >>>> ?http://lists.mcs.anl.gov/pipermail/petsc-users/2011-January/007623.html >>>> >>>> Following Barry's suggestion, I checked that the nonzero pattern of my >>>> matrix is still the same via >>>> ?ierr=MatSetOption(M, MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE); CHKERRQ(ierr); >>>> >>>> I tried two other options: >>>> >>>> 1 First install the latest superlu_dist (v2.5), then compile petsc >>>> with this new superlu_dist package. However, the problem is still >>>> there, and the performance is even worse ( comparing to the option >>>> --download-superlu_dist=yes (v2.4)). >>>> >>>> 2 I compiled petsc-dev with --download-superlu_dist. However, I got >>>> some errors when compiling my program with petsc-dev. It said that >>>> >>>> ResonatorOpt.c: In function ?main?: >>>> ResonatorOpt.c:154: error: ?MAT_SOLVER_SUPERLU_DIST? undeclared (first >>>> use in this function) >>>> ResonatorOpt.c:154: error: (Each undeclared identifier is reported only once >>>> ResonatorOpt.c:154: error: for each function it appears in.) >>>> >>>> Besides superlu_dist, MAT_SOLVER_PASTIX and MAT_SOLVER_PASTIX are >>>> undeclared either. Am I missing some header files? I used the same >>>> procedure as I complied regular petsc, which had no such problems. ?(I >>>> tried to attach the configure log in a previous mail, but it is >>>> bounced back.) >>>> >>>> Can you give me some suggestions on using superlu_dist with >>>> same_nonzero_pattern? Thank you. >>>> >>>> Best, >>>> Xiangdong >>>> >>> >> > From bsmith at mcs.anl.gov Fri Jun 24 11:11:05 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 24 Jun 2011 11:11:05 -0500 Subject: [petsc-users] Fwd: superlu_dist with same_nonzero_pattern In-Reply-To: References: Message-ID: <3400366C-93EB-4734-B147-EBF22C0E1F2A@mcs.anl.gov> Use the KSP routines, there is no reason to use the low-level routines, they will not be any less efficient. Barry On Jun 24, 2011, at 10:20 AM, Xiangdong Liang wrote: > Thanks for helping me check the code. > > I agree that moving the KSPSetOperators(ksp,M,M,SAME_NONZERO_PATTERN) > outside the iteration loop lifted the crash of superlu_dist. However, > I have the following concerns: > > 1) If we only call KSPSetOperators once outside the loop, the ksp > solver would be the same for all the iterations, although matrix M is > changing during each iteration. I added the following code to > manually check the error of the solution, > > ierr = MatMult(M,x, xdiff);CHKERRQ(ierr); > ierr = VecAXPY(xdiff,-1.0,J);CHKERRQ(ierr); > ierr = VecNorm(xdiff,NORM_2,&norm);CHKERRQ(ierr); > ierr = PetscPrintf(PETSC_COMM_WORLD,"---Norm of error %A----\n > ",norm);CHKERRQ(ierr); > > As far as I get, the error are huge (1e+6) except the first iteration. > It seems that it keeps solving Mx=J by using the non-updated M. > > 2) Actually, I really want to switch between SAME_NONZERO_PATTERN and > SAME_PRECONDITIONER. In my problem, matrix M changes only a little bit > during each step. I am planing to only use the sparse direct solver > every 10 steps, because the LU decomposition is very expensive. The > other steps I simply use the iterative methods (GMRES) with the > preconditioner obtained in the previous LU decomposition. > > As we tested, Pastix performs well for this switch. It would be nice > if superlu_dist also works for this switch. The other sparse direct > solvers are just too slow for my 3D problem. > > 3) I am also think to use the low level functions like: > > Outside the loop: > MatGetOrdering(Mat matrix,MatOrderingType type,IS* rowperm,IS* colperm); > MatGetFactor(Mat matrix,const MatSolverPackage package,MatFactorType > ftype,Mat *factor); > MatLUFactorSymbolic(Mat factor,Mat matrix,IS rowperm,IS colperm,const > MatFactorInfo *info); > > Inside the loop: > MatLUFactorNumeric(Mat factor,Mat matrix,const MatFactorInfo *info); > MatSolve(Mat A,Vec x, Vec y); > > Are they equivalent to SAME_NONZERO_PATTERN? Do you have an idea > whether it will perform well or worse than the ksp version? > > Thank you very much. > > Xiangdong > > > On Thu, Jun 23, 2011 at 11:27 PM, Hong Zhang wrote: >> Xiangdong : >> I tested your code. Here is what I get: >> >> 1. your code does not build. I made following changes: >> < PetscErrorCode MoperatorGeneral(MPI_Comm comm, int Nx, int Ny, int >> Nz, Vec muinvpml,Mat *Aout) >> --- >>> Mat MoperatorGeneral(MPI_Comm comm, int Nx, int Ny, int Nz, Vec muinvpml) >> >> and get it build on an iMac and a linux machine >> >> 2. code hangs at >> if (rank == 0) fgetc(stdin); >> I comment it out. >> >> 3. then I can reproduce your error with superlu_dist, np=2, and >> SAME_NONZERO_PATTERN. >> KSPSetOperators(ksp,M,M,SAME_NONZERO_PATTERN) >> should be called once, not inside your iteration loop. Move it out of >> the loop fix the problem. >> Why other packages work? I do not know :-( >> >> 4. the modified code works well on the linux machine with np=1, 2, 4 etc. >> However, on iMac, even with np=1, at the end of 1st iteration, I get >> >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Floating point exception! >> [0]PETSC ERROR: Infinite or not-a-number generated in norm! >> ... >> [0]PETSC ERROR: VecNorm() line 167 in src/vec/vec/interface/rvector.c >> [0]PETSC ERROR: main() line 93 in test.c >> >> Checking solution x, I see >> '-inf' in several components. >> Using mumps ('-pc_factor_mat_solver_package mumps') with np=2, I get same error. >> However, sequential superlu, mumps and petsc all run well. >> It is likely your model is numerically sensitive, not problem with >> these packages. >> >> The modified code is attached. Good luck! >> >> Hong >> >> >> >>> Thanks, Hong. I tried the runtime option -mat_superlu_dist_equil NO. >>> However, the problem is still there. >>> >>> I've upload my short codes here: >>> http://math.mit.edu/~xdliang/superlu_dist_test.zip >>> >>> In this code, I first generate my sparse matrix M with dimension >>> N-by-N, then solve Mx=J for a few times. Each time, I only modify the >>> entries of two diagonals: (i,i) and ( i, (i+N/2)%N). Althougth I >>> modify these entries, I thought the nonzero pattern should not change. >>> Based on my tests, I found that: >>> >>> 1 superlu_dist works fine with DIFFERENT_NONZERO_PATTERN. However, if >>> I switch to SAME_NONZERO_PATTERN, the program either crashes (with np >>>> 1 ) or does not converge ( with single processor). >>> >>> 2 pastix works fine with SAME_NONZERO_PATTERN, but it halts there if I >>> switch to DIFFERENT_NONZERO_PATTERN ( with np>1). >>> >>> 3 spooles works fine with both SAME and DIFFERENT nonzero pattern. >>> >>> Can you give me some hints why superlu_dist fails on my matrix with >>> SAME_NONZERO_PATTERN? I tried both superlu_dist v2.4 and v2.5 ( latest >>> one). Thanks. >>> >>> Best, >>> Xiangdong >>> >>> >>> >>> On Wed, Jun 22, 2011 at 2:06 PM, Hong Zhang wrote: >>>> Xiangdong : >>>> >>>> Does it hangs inside superlu_dist? >>>> Try runtime option '-mat_superlu_dist_equil NO' >>>> >>>> If you can send us a stand-alone short code or your matrix in petsc >>>> binary format that repeats this behavior, then we can investigate it. >>>> Likely memory problem. >>>> >>>> Hong >>>> >>>> >>>>> Hello everyone, >>>>> >>>>> I had some problem in using superlu_dist with same_nonzero_pattern. >>>>> Sometimes superlu_dist solver halts there and never finishes. However, >>>>> when I switched to different_nonzero_pattern or other solvers, the >>>>> program works nicely. Ding had reported this problem a few months >>>>> ago: >>>>> http://lists.mcs.anl.gov/pipermail/petsc-users/2011-January/007623.html >>>>> >>>>> Following Barry's suggestion, I checked that the nonzero pattern of my >>>>> matrix is still the same via >>>>> ierr=MatSetOption(M, MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE); CHKERRQ(ierr); >>>>> >>>>> I tried two other options: >>>>> >>>>> 1 First install the latest superlu_dist (v2.5), then compile petsc >>>>> with this new superlu_dist package. However, the problem is still >>>>> there, and the performance is even worse ( comparing to the option >>>>> --download-superlu_dist=yes (v2.4)). >>>>> >>>>> 2 I compiled petsc-dev with --download-superlu_dist. However, I got >>>>> some errors when compiling my program with petsc-dev. It said that >>>>> >>>>> ResonatorOpt.c: In function ?main?: >>>>> ResonatorOpt.c:154: error: ?MAT_SOLVER_SUPERLU_DIST? undeclared (first >>>>> use in this function) >>>>> ResonatorOpt.c:154: error: (Each undeclared identifier is reported only once >>>>> ResonatorOpt.c:154: error: for each function it appears in.) >>>>> >>>>> Besides superlu_dist, MAT_SOLVER_PASTIX and MAT_SOLVER_PASTIX are >>>>> undeclared either. Am I missing some header files? I used the same >>>>> procedure as I complied regular petsc, which had no such problems. (I >>>>> tried to attach the configure log in a previous mail, but it is >>>>> bounced back.) >>>>> >>>>> Can you give me some suggestions on using superlu_dist with >>>>> same_nonzero_pattern? Thank you. >>>>> >>>>> Best, >>>>> Xiangdong >>>>> >>>> >>> >> From bsmith at mcs.anl.gov Fri Jun 24 22:22:04 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 24 Jun 2011 22:22:04 -0500 Subject: [petsc-users] PCCOMPOSITE and ApplyTranspose In-Reply-To: <20110623031350.AIV11763@batman.int.colorado.edu> References: <20110623031350.AIV11763@batman.int.colorado.edu> Message-ID: I have added PCApplyTranspose() for PCCOMPOSITE in petsc-dev. You will need to switch to petsc-dev http://www.mcs.anl.gov/petsc/petsc-as/developers/index.html to access it. Sorry for the inconvenience. Barry On Jun 23, 2011, at 4:13 AM, Andrew T Barker wrote: > I've made a PCCOMPOSITE that contains two PCSHELLS, both of > which have a transpose set with PCShellSetApplyTranspose(). > However the composite PC does not seem to support the > transpose: > > [0]PETSC ERROR: No support for this operation for this object > type! > [0]PETSC ERROR: PC does not have apply transpose! > > The code runs fine with the composite replaced with either of > the shells by itself. Of course I can implement my own > composite as a PCShell with a transpose easily enough, but I > was wondering if this was expected or if I am making some > other mistake. > > I am using Petsc 3.0.0. > > Thanks, > > Andrew From njumeiyi at gmail.com Sat Jun 25 09:07:37 2011 From: njumeiyi at gmail.com (Yi Mei) Date: Sat, 25 Jun 2011 09:07:37 -0500 Subject: [petsc-users] make test error after petsc installation Message-ID: Hi, I am new to petsc. I downloaded the latest version of petsc, and followed the installation guide, did the following: cd petsc-3.1-p8/ PETSC_DIR=$PWD; export PETSC_DIR ./config/configure.py --with-cc=gcc --with-fc=gfortran --download-f-blas-lapack=1 --download-mpich=1 make all make test Then I got the following error and warning message: --------------Error detected during compile or link!----------------------- See http://www.mcs.anl.gov/petsc/petsc-2/documentation/troubleshooting.html /home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/bin/mpif90 -c -Wall -Wno-unused-variable -g -I/home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/include -I/home/yimei/Linuxsoft/petsc/petsc-3.1-p8/include -I/home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/include -I/home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/include -I/home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/include -o ex5f.o ex5f.F ex5f.F:92.72: call PetscOptionsGetReal(PETSC_NULL_CHARACTER,'-par',lambda, 1 Warning: Line truncated at (1) ex5f.F:113.72: call DACreate2d(PETSC_COMM_WORLD,DA_NONPERIODIC,DA_STENCIL_STAR, 1 Warning: Line truncated at (1) ex5f.F:114.72: & i4,i4,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER, 1 Warning: Line truncated at (1) ex5f.F:125.72: call DAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER, 1 Warning: Line truncated at (1) ex5f.F:126.72: & PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, 1 Warning: Line truncated at (1) ex5f.F:127.72: & PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, 1 Warning: Line truncated at (1) ex5f.F:128.72: & PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, 1 Warning: Line truncated at (1) ex5f.F:130.72: call DAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym, 1 Warning: Line truncated at (1) ex5f.F:132.72: call DAGetGhostCorners(da,gxs,gys,PETSC_NULL_INTEGER,gxm,gym, 1 Warning: Line truncated at (1) ex5f.F:188.72: call SNESSetJacobian(snes,A,J,SNESDAComputeJacobian, 1 Warning: Line truncated at (1) ex5f.F:344.72: if (i .eq. 1 .or. j .eq. 1 1 Warning: Line truncated at (1) ex5f.F:348.72: x(i,j) = temp1 * 1 Warning: Line truncated at (1) ex5f.F:412.72: if (i .eq. 1 .or. j .eq. 1 1 Warning: Line truncated at (1) ex5f.F:417.72: uxx = hydhx * (two*u 1 Warning: Line truncated at (1) ex5f.F:517.72: if (i .eq. 1 .or. j .eq. 1 1 Warning: Line truncated at (1) ex5f.F:522.72: call MatSetValuesLocal(jac,i1,row,i1,col,v, 1 Warning: Line truncated at (1) ex5f.F:528.72: v(3) = two*(hydhx + hxdhy) 1 Warning: Line truncated at (1) ex5f.F:537.72: call MatSetValuesLocal(jac,i1,row,i5,col,v, 1 Warning: Line truncated at (1) /home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/bin/mpif90 -Wall -Wno-unused-variable -g -o ex5f ex5f.o -Wl,-rpath,/home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/lib -L/home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/lib -lpetsc -lX11 -Wl,-rpath,/home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/lib -L/home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/lib -lflapack -lfblas -lnsl -lrt -lm -L/home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/lib -L/usr/lib/i386-linux-gnu/gcc/i686-linux-gnu/4.5.2 -L/usr/lib/i386-linux-gnu -ldl -lmpich -lpthread -lrt -lgcc_s -lmpichf90 -lgfortran -lm -lm -ldl -lmpich -lpthread -lrt -lgcc_s -ldl /bin/rm -f ex5f.o Fortran example src/snes/examples/tutorials/ex5f run successfully with 1 MPI process Could you please help me figure out what I did wrong during the installation of this package? Thank you so much! Best Yi -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Jun 25 09:10:33 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 25 Jun 2011 09:10:33 -0500 Subject: [petsc-users] make test error after petsc installation In-Reply-To: References: Message-ID: On Sat, Jun 25, 2011 at 9:07 AM, Yi Mei wroteHi, > > > I am new to petsc. > > I downloaded the latest version of petsc, and followed the installation > guide, did the following: > You can ignore these warnings. The Fortran example ran correctly. These will be fixed in the next release. Thanks, Matt > cd petsc-3.1-p8/ > PETSC_DIR=$PWD; export PETSC_DIR > ./config/configure.py --with-cc=gcc --with-fc=gfortran > --download-f-blas-lapack=1 --download-mpich=1 > make all > make test > > Then I got the following error and warning message: > > --------------Error detected during compile or link!----------------------- > See > http://www.mcs.anl.gov/petsc/petsc-2/documentation/troubleshooting.html > /home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/bin/mpif90 -c > -Wall -Wno-unused-variable -g > -I/home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/include > -I/home/yimei/Linuxsoft/petsc/petsc-3.1-p8/include > -I/home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/include > -I/home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/include > -I/home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/include -o > ex5f.o ex5f.F > ex5f.F:92.72: > > call PetscOptionsGetReal(PETSC_NULL_CHARACTER,'-par',lambda, > 1 > Warning: Line truncated at (1) > ex5f.F:113.72: > > call DACreate2d(PETSC_COMM_WORLD,DA_NONPERIODIC,DA_STENCIL_STAR, > 1 > Warning: Line truncated at (1) > ex5f.F:114.72: > > & i4,i4,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER, > 1 > Warning: Line truncated at (1) > ex5f.F:125.72: > > call DAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER, > 1 > Warning: Line truncated at (1) > ex5f.F:126.72: > > & PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, > 1 > Warning: Line truncated at (1) > ex5f.F:127.72: > > & PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, > 1 > Warning: Line truncated at (1) > ex5f.F:128.72: > > & PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, > 1 > Warning: Line truncated at (1) > ex5f.F:130.72: > > call DAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym, > 1 > Warning: Line truncated at (1) > ex5f.F:132.72: > > call DAGetGhostCorners(da,gxs,gys,PETSC_NULL_INTEGER,gxm,gym, > 1 > Warning: Line truncated at (1) > ex5f.F:188.72: > > call SNESSetJacobian(snes,A,J,SNESDAComputeJacobian, > 1 > Warning: Line truncated at (1) > ex5f.F:344.72: > > if (i .eq. 1 .or. j .eq. 1 > 1 > Warning: Line truncated at (1) > ex5f.F:348.72: > > x(i,j) = temp1 * > 1 > Warning: Line truncated at (1) > ex5f.F:412.72: > > if (i .eq. 1 .or. j .eq. 1 > 1 > Warning: Line truncated at (1) > ex5f.F:417.72: > > uxx = hydhx * (two*u > 1 > Warning: Line truncated at (1) > ex5f.F:517.72: > > if (i .eq. 1 .or. j .eq. 1 > 1 > Warning: Line truncated at (1) > ex5f.F:522.72: > > call MatSetValuesLocal(jac,i1,row,i1,col,v, > 1 > Warning: Line truncated at (1) > ex5f.F:528.72: > > v(3) = two*(hydhx + hxdhy) > 1 > Warning: Line truncated at (1) > ex5f.F:537.72: > > call MatSetValuesLocal(jac,i1,row,i5,col,v, > 1 > Warning: Line truncated at (1) > /home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/bin/mpif90 -Wall > -Wno-unused-variable -g -o ex5f ex5f.o > -Wl,-rpath,/home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/lib > -L/home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/lib -lpetsc > -lX11 > -Wl,-rpath,/home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/lib > -L/home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/lib -lflapack > -lfblas -lnsl -lrt -lm > -L/home/yimei/Linuxsoft/petsc/petsc-3.1-p8/linux-gnu-c-debug/lib > -L/usr/lib/i386-linux-gnu/gcc/i686-linux-gnu/4.5.2 -L/usr/lib/i386-linux-gnu > -ldl -lmpich -lpthread -lrt -lgcc_s -lmpichf90 -lgfortran -lm -lm -ldl > -lmpich -lpthread -lrt -lgcc_s -ldl > /bin/rm -f ex5f.o > Fortran example src/snes/examples/tutorials/ex5f run successfully with 1 > MPI process > > > Could you please help me figure out what I did wrong during the > installation of this package? > > Thank you so much! > > Best > Yi > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Sat Jun 25 09:55:52 2011 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sat, 25 Jun 2011 16:55:52 +0200 Subject: [petsc-users] eigenvalues in a band Message-ID: <4E05F6F8.3030202@tudelft.nl> Dear all, I am sorry for a question that is not directly related to PETSc however there is large community with a lot experienced people. So the question fits I guess. I would like to extract the eigenvalues/vectors from a band, say such a range as [a,b]. I found out that there are some packages on the netlib pages however they are really outdated fortran codes, at least the ones I checked and there should be a lot of changes on the codes. I was wondering if there is already a package that does what I would like to reach at the end, the range eigenvalues and vectors. So I would be glad if someone who knows some packages that is capable of this operation, would share his/her experience. P.S. Ther////// e is SLEPc however they are not supporting the band option yet, however use of two shifts might be an option and then I can do some post-processing later on to extract the information that I would like. So on that I am aware of. Best regards, Umut -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Jun 25 10:00:21 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 25 Jun 2011 10:00:21 -0500 Subject: [petsc-users] eigenvalues in a band In-Reply-To: <4E05F6F8.3030202@tudelft.nl> References: <4E05F6F8.3030202@tudelft.nl> Message-ID: <01DB0EFB-02EC-4DD8-9280-2FA1B7CC6A27@mcs.anl.gov> Umut, You might find this article to be of use, http://portal.acm.org/citation.cfm?id=1236464 also here http://www.mcs.anl.gov/uploads/cels/papers/P1263.pdf Barry On Jun 25, 2011, at 9:55 AM, Umut Tabak wrote: > Dear all, > > I am sorry for a question that is not directly related to PETSc however there is large community with a lot experienced people. So the question fits I guess. > > I would like to extract the eigenvalues/vectors from a band, say such a range as [a,b]. I found out that there are some packages on the netlib pages however they are really outdated fortran codes, at least the ones I checked and there should be a lot of changes on the codes. I was wondering if there is already a package that does what I would like to reach at the end, the range eigenvalues and vectors. So I would be glad if someone who knows some packages that is capable of this operation, would share his/her experience. > > P.S. There is SLEPc however they are not supporting the band option yet, however use of two shifts might be an option and then I can do some post-processing later on to extract the information that I would like. So on that I am aware of. > > Best regards, > Umut From u.tabak at tudelft.nl Sat Jun 25 10:07:09 2011 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sat, 25 Jun 2011 17:07:09 +0200 Subject: [petsc-users] eigenvalues in a band In-Reply-To: <01DB0EFB-02EC-4DD8-9280-2FA1B7CC6A27@mcs.anl.gov> References: <4E05F6F8.3030202@tudelft.nl> <01DB0EFB-02EC-4DD8-9280-2FA1B7CC6A27@mcs.anl.gov> Message-ID: <4E05F99D.2010902@tudelft.nl> On 06/25/2011 05:00 PM, Barry Smith wrote: > Umut, > > You might find this article to be of use, http://portal.acm.org/citation.cfm?id=1236464 also here http://www.mcs.anl.gov/uploads/cels/papers/P1263.pdf > > Barry > > Dear Barry, Thanks for the super fast reply, I was wondering if there is a way to use this package SIPs, I can not read the acm portal paper from my location since I am outside the university now however the report shows that this seems like an optimal package that I would like to be a customer of ;) Best wishes, Umut From bsmith at mcs.anl.gov Sat Jun 25 10:20:39 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 25 Jun 2011 10:20:39 -0500 Subject: [petsc-users] eigenvalues in a band In-Reply-To: <4E05F99D.2010902@tudelft.nl> References: <4E05F6F8.3030202@tudelft.nl> <01DB0EFB-02EC-4DD8-9280-2FA1B7CC6A27@mcs.anl.gov> <4E05F99D.2010902@tudelft.nl> Message-ID: <1B6195B3-66C9-400E-8F60-FCF9441E1E26@mcs.anl.gov> On Jun 25, 2011, at 10:07 AM, Umut Tabak wrote: > On 06/25/2011 05:00 PM, Barry Smith wrote: >> Umut, >> >> You might find this article to be of use, http://portal.acm.org/citation.cfm?id=1236464 also here http://www.mcs.anl.gov/uploads/cels/papers/P1263.pdf >> >> Barry >> >> > Dear Barry, > > Thanks for the super fast reply, I was wondering if there is a way to use this package SIPs, I can not read the acm portal paper from my location since I am outside the university now however the report shows that this seems like an optimal package that I would like to be a customer of ;) The report is the same as the ACM paper. We can share the code with you but it is a little fragile and you may need to do a little work to get it going for your problems. Barry > > Best wishes, > Umut From u.tabak at tudelft.nl Sat Jun 25 10:22:46 2011 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sat, 25 Jun 2011 17:22:46 +0200 Subject: [petsc-users] eigenvalues in a band In-Reply-To: <1B6195B3-66C9-400E-8F60-FCF9441E1E26@mcs.anl.gov> References: <4E05F6F8.3030202@tudelft.nl> <01DB0EFB-02EC-4DD8-9280-2FA1B7CC6A27@mcs.anl.gov> <4E05F99D.2010902@tudelft.nl> <1B6195B3-66C9-400E-8F60-FCF9441E1E26@mcs.anl.gov> Message-ID: <4E05FD46.6000801@tudelft.nl> > The report is the same as the ACM paper. > > We can share the code with you but it is a little fragile and you may need to do a little work to get it going for your problems. > > > Barry > That would be perfect and I will refer to that paper in my publication if I can make it get published ;) How should I continue to get the code and start testing the code? Best wishes, Umut From hzhang at mcs.anl.gov Sat Jun 25 13:43:43 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Sat, 25 Jun 2011 13:43:43 -0500 Subject: [petsc-users] eigenvalues in a band In-Reply-To: <4E05FD46.6000801@tudelft.nl> References: <4E05F6F8.3030202@tudelft.nl> <01DB0EFB-02EC-4DD8-9280-2FA1B7CC6A27@mcs.anl.gov> <4E05F99D.2010902@tudelft.nl> <1B6195B3-66C9-400E-8F60-FCF9441E1E26@mcs.anl.gov> <4E05FD46.6000801@tudelft.nl> Message-ID: Umut, SIPs was developed from a project in material simulation more than 9 years ago. Since then, both petsc and slepc have several new releases while SIPs lacks resource for its update and further development. Thus far, I was the only SIPs coder, with obligation on several funded demanding projects, plus teaching and student advising. The current SIPs code remains as "research code" and fails to work with the latest version of petsc and slepc. Until I'm able to devote a period of concentrated effort to its update, it cannot be used. I attempted to update it to the current petsc/slepc release several times, but failed to accomplish due to other tasks. I'm sorry that I do not have anything to offer at this time. I'll let you know when SIPs is updated to be usable. Thanks for your interest, Hong On Sat, Jun 25, 2011 at 10:22 AM, Umut Tabak wrote: > >> ? The report is the same as the ACM paper. >> >> ? ?We can share the code with you but it is a little fragile and you may >> need to do a little work to get it going for your problems. >> >> >> ? ?Barry >> > That would be perfect and I will refer to that paper in my publication if I > can make it get published ;) > > How should I continue to get the code and start testing the code? > > Best wishes, > > Umut > From u.tabak at tudelft.nl Sat Jun 25 13:47:54 2011 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sat, 25 Jun 2011 20:47:54 +0200 Subject: [petsc-users] eigenvalues in a band In-Reply-To: References: <4E05F6F8.3030202@tudelft.nl> <01DB0EFB-02EC-4DD8-9280-2FA1B7CC6A27@mcs.anl.gov> <4E05F99D.2010902@tudelft.nl> <1B6195B3-66C9-400E-8F60-FCF9441E1E26@mcs.anl.gov> <4E05FD46.6000801@tudelft.nl> Message-ID: <4E062D5A.8040505@tudelft.nl> On 06/25/2011 08:43 PM, Hong Zhang wrote: > Umut, > > SIPs was developed from a project in material simulation more > than 9 years ago. Since then, both petsc and slepc have several new releases > while SIPs lacks resource for its update and further development. > Thus far, I was the only SIPs coder, with obligation on several > funded demanding projects, plus teaching and student advising. > > The current SIPs code remains as "research code" and fails to work with > the latest version of petsc and slepc. > Until I'm able to devote a period of concentrated effort to its > update, it cannot be used. I attempted to update it to the current > petsc/slepc release several times, > but failed to accomplish due to other tasks. > > I'm sorry that I do not have anything to offer at this time. I'll let > you know when SIPs is updated to be usable. > > Thanks for your interest, > > Hong Thanks for the sincere reply and interest. Please, let me know if you make progress in short time, I would be interested to know. Best wishes, Umut From zhaonanavril at gmail.com Sat Jun 25 14:49:25 2011 From: zhaonanavril at gmail.com (NAN ZHAO) Date: Sat, 25 Jun 2011 14:49:25 -0500 Subject: [petsc-users] help on how to use petsc in a smart way Message-ID: Dear all, I have a question about how to use the ksp in a smart way to solve a linear system. I had a simple test to generate a random matrix and vector, but I put only one nonzero value in a row (let's say 1), and in the respect location of the RHS vector I put a zero. The size of the matrix is kind of big, I saw petsc some time give unreasonable value at that loaction (it should be zero or some really small number). I want to know if there is a way to avoid it? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Sat Jun 25 14:54:21 2011 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sat, 25 Jun 2011 21:54:21 +0200 Subject: [petsc-users] petscstack undefined reference from SLEPc Message-ID: <4E063CED.7050709@tudelft.nl> Dear all, I was trying to update an old eigenvalue solver wrapper of mine that is dependent on Slepc, however I am getting an error on the linking phase as such /home/utab/external_programs/slepc-3.1-p6/src/eps/interface/basic.c:43: undefined reference to `petscstack' So basically the error comes from the SLEPc side but the nm command on the libpetsc.a also does not return a symbol with this name, so a bit puzzled. I am using 3.1.patch 8 and slepc 3.1 patch 6. Any pointer on the whereabouts of my problem is appreciated. Best regards, Umut From knepley at gmail.com Sat Jun 25 14:58:04 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 25 Jun 2011 14:58:04 -0500 Subject: [petsc-users] help on how to use petsc in a smart way In-Reply-To: References: Message-ID: On Sat, Jun 25, 2011 at 2:49 PM, NAN ZHAO wrote: > Dear all, > > I have a question about how to use the ksp in a smart way to solve a linear > system. I had a simple test to generate a random matrix and vector, but I > put only one nonzero value in a row (let's say 1), and in the respect > location of the RHS vector I put a zero. The size of the matrix is kind of > big, I saw petsc some time give unreasonable value at that loaction (it > should be zero or some really small number). I want to know if there is a > way to avoid it? > The problem sounds degenerate. A full rank A with 1 nonzero/row and b = 0 would not have a solution other than 0. What will this tell you? Matt > Thanks > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Jun 25 16:00:35 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 25 Jun 2011 16:00:35 -0500 Subject: [petsc-users] petscstack undefined reference from SLEPc In-Reply-To: <4E063CED.7050709@tudelft.nl> References: <4E063CED.7050709@tudelft.nl> Message-ID: On Sat, Jun 25, 2011 at 2:54 PM, Umut Tabak wrote: > Dear all, > > I was trying to update an old eigenvalue solver wrapper of mine that is > dependent on Slepc, however I am getting an error on the linking phase as > such > > /home/utab/external_programs/**slepc-3.1-p6/src/eps/**interface/basic.c:43: > undefined reference to `petscstack' > > So basically the error comes from the SLEPc side but the nm command on the > libpetsc.a also does not return a symbol with this name, so a bit puzzled. I > am using 3.1.patch 8 and slepc 3.1 patch 6. > > Any pointer on the whereabouts of my problem is appreciated. > Its defined in src/sys/err/pstack.c. Its only defined if PETSC_USE_DEBUG is #defined. Are you combining debug and non-debug libraries? Matt > Best regards, > Umut > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Sat Jun 25 16:06:07 2011 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sat, 25 Jun 2011 23:06:07 +0200 Subject: [petsc-users] petscstack undefined reference from SLEPc In-Reply-To: References: <4E063CED.7050709@tudelft.nl> Message-ID: <4E064DBF.4080207@tudelft.nl> On 06/25/2011 11:00 PM, Matthew Knepley wrote: > Its defined in src/sys/err/pstack.c. Its only defined if > PETSC_USE_DEBUG is #defined. Are you combining debug and non-debug > libraries? > Hi Matt, That was the problem, I also realized that I have a non-debug version of Petsc linked and later recompiled slepc with the debug version, problem solved. Thanks, Umut From zhaonanavril at gmail.com Sat Jun 25 20:57:50 2011 From: zhaonanavril at gmail.com (NAN ZHAO) Date: Sat, 25 Jun 2011 20:57:50 -0500 Subject: [petsc-users] help on how to use petsc in a smart way In-Reply-To: References: Message-ID: I think I may rephrase my problem like that, I have a huge matrix with n by n, at row m, I put a 1 in the location (m,m), other element in the row is zero. Then I construct a vector with size n, put zero in the m location. All other location I just put random number. I should get a answer vector with 0 in the mth element, But I some times got unreasonable number in this location. I am wondering if there is some option in petsc to avoid this case. Thanks On Sat, Jun 25, 2011 at 2:58 PM, Matthew Knepley wrote: > On Sat, Jun 25, 2011 at 2:49 PM, NAN ZHAO wrote: > >> Dear all, >> >> I have a question about how to use the ksp in a smart way to solve a >> linear system. I had a simple test to generate a random matrix and vector, >> but I put only one nonzero value in a row (let's say 1), and in the respect >> location of the RHS vector I put a zero. The size of the matrix is kind of >> big, I saw petsc some time give unreasonable value at that loaction (it >> should be zero or some really small number). I want to know if there is a >> way to avoid it? >> > > The problem sounds degenerate. A full rank A with 1 nonzero/row and b = 0 > would not have a solution other than 0. What will this tell you? > > Matt > > >> Thanks >> > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Jun 25 21:01:47 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 25 Jun 2011 21:01:47 -0500 Subject: [petsc-users] help on how to use petsc in a smart way In-Reply-To: References: Message-ID: On Sat, Jun 25, 2011 at 8:57 PM, NAN ZHAO wrote: > I think I may rephrase my problem like that, I have a huge matrix with n by > n, at row m, I put a 1 in the location (m,m), other element in the row is > zero. Then I construct a vector with size n, put zero in the m location. All > other location I just put random number. I should get a answer vector with 0 > in the mth element, But I some times got unreasonable number in this > location. I am wondering if there is some option in petsc to avoid this > case. > Use -pc_type lu -ksp_type preonly. You will get zero or it will tell you that the system is singular. Matt > Thanks > > > On Sat, Jun 25, 2011 at 2:58 PM, Matthew Knepley wrote: > >> On Sat, Jun 25, 2011 at 2:49 PM, NAN ZHAO wrote: >> >>> Dear all, >>> >>> I have a question about how to use the ksp in a smart way to solve a >>> linear system. I had a simple test to generate a random matrix and vector, >>> but I put only one nonzero value in a row (let's say 1), and in the respect >>> location of the RHS vector I put a zero. The size of the matrix is kind of >>> big, I saw petsc some time give unreasonable value at that loaction (it >>> should be zero or some really small number). I want to know if there is a >>> way to avoid it? >>> >> >> The problem sounds degenerate. A full rank A with 1 nonzero/row and b = 0 >> would not have a solution other than 0. What will this tell you? >> >> Matt >> >> >>> Thanks >>> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From ckontzialis at lycos.com Sun Jun 26 22:37:13 2011 From: ckontzialis at lycos.com (Kostas Kontzialis) Date: Mon, 27 Jun 2011 06:37:13 +0300 Subject: [petsc-users] Compiling Message-ID: <4E07FAE9.3050405@lycos.com> Dear all, I have installed Petsc and everything is going well. But, if I use the icc compiler with the -fast flag, it won't compile my program. I do this in order to get the best optimization for my application. Any suggestions? Kostas From knepley at gmail.com Sun Jun 26 22:48:52 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 26 Jun 2011 22:48:52 -0500 Subject: [petsc-users] Compiling In-Reply-To: <4E07FAE9.3050405@lycos.com> References: <4E07FAE9.3050405@lycos.com> Message-ID: On Sun, Jun 26, 2011 at 10:37 PM, Kostas Kontzialis wrote: > Dear all, > > I have installed Petsc and everything is going well. But, if I use the icc > compiler with the -fast flag, it won't compile my program. I do this in > order to get the best optimization for my application. Any suggestions? >From this mail, we can have absolutely no idea what went wrong. Send configure.log and make.log to petsc-maint at mcs.anl.gov. Matt > > Kostas > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From cbergstrom at pathscale.com Sun Jun 26 22:57:28 2011 From: cbergstrom at pathscale.com (=?ISO-8859-1?Q?=22C=2E_Bergstr=F6m=22?=) Date: Mon, 27 Jun 2011 10:57:28 +0700 Subject: [petsc-users] Compiling In-Reply-To: <4E07FAE9.3050405@lycos.com> References: <4E07FAE9.3050405@lycos.com> Message-ID: <4E07FFA8.5080308@pathscale.com> On 06/27/11 10:37 AM, Kostas Kontzialis wrote: > Dear all, > > I have installed Petsc and everything is going well. But, if I use > the icc compiler with the -fast flag, it won't compile my program. I > do this in order to get the best optimization for my application. Any > suggestions? Hi Kostas, I don't mean to hi-jack your thread, but if you're willing to also try building with EKOPath we can offer help for optimizing. (EKOPath is now open source and there's a link to the free nightly builds.) http://www.pathscale.com/ekopath-compiler-suite ./C From klaus.zimmermann at physik.uni-freiburg.de Mon Jun 27 05:46:22 2011 From: klaus.zimmermann at physik.uni-freiburg.de (Klaus Zimmermann) Date: Mon, 27 Jun 2011 12:46:22 +0200 Subject: [petsc-users] petsc-dev: configure problem with metis Message-ID: <4E085F7E.6020907@physik.uni-freiburg.de> Hi all, in order to profit from cuda I tried to compile petsc-dev (latest hg). Unfortunately it fails on configuring metis (see below). Is this a known problem? Thanks in advance, Klaus -------------------- kz38 at globus:~/src/extern/petsc-dev$ ./configure --download-metis --with-fortran --with-blacs-include=/opt/bwgrid/compiler/intel/ct_4.0/mkl/10.2.5.035/include --with-blas-lapack-dir=/opt/bwgrid/compiler/intel/ct_4.0/mkl/10.2.5.035 --with-scalar-type=complex --with-blacs --with-scalapack-lib=/opt/bwgrid/compiler/intel/ct_4.0/mkl/10.2.5.035/lib/em64t/libmkl_scalapack_lp64.a --with-clanguage=c++ --with-precision=double --with-scalapack --with-scalapack-include=/opt/bwgrid/compiler/intel/ct_4.0/mkl/10.2.5.035/include --with-blacs-lib=/opt/bwgrid/compiler/intel/ct_4.0/mkl/10.2.5.035/lib/em64t/libmkl_blacs_openmpi_lp64.a --with-cuda --with-cusp --with-thrust =============================================================================== Configuring PETSc to compile on your system =============================================================================== =============================================================================== Compiling & installing Metis; this may take several minutes =============================================================================== ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error running make on Metis: Could not execute "cd /home/kz38/src/extern/petsc-dev/externalpackages/metis-4.0.3 && make clean && make library && make minstall && make clean": (cd Lib ; make clean ) make[1]: Entering directory `/home/kz38/src/extern/petsc-dev/externalpackages/metis-4.0.3/Lib' rm -f *.o make[1]: Leaving directory `/home/kz38/src/extern/petsc-dev/externalpackages/metis-4.0.3/Lib' (cd Programs ; make clean ) make[1]: Entering directory `/home/kz38/src/extern/petsc-dev/externalpackages/metis-4.0.3/Programs' rm -f *.o make[1]: Leaving directory `/home/kz38/src/extern/petsc-dev/externalpackages/metis-4.0.3/Programs' (cd Test ; make clean ) make[1]: Entering directory `/home/kz38/src/extern/petsc-dev/externalpackages/metis-4.0.3/Test' rm -f *.o make[1]: Leaving directory `/home/kz38/src/extern/petsc-dev/externalpackages/metis-4.0.3/Test' make: *** No rule to make target `library'. Stop. ******************************************************************************* From bsmith at mcs.anl.gov Mon Jun 27 08:16:01 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 27 Jun 2011 08:16:01 -0500 Subject: [petsc-users] petsc-dev: configure problem with metis In-Reply-To: <4E085F7E.6020907@physik.uni-freiburg.de> References: <4E085F7E.6020907@physik.uni-freiburg.de> Message-ID: You need to send configure.log to petsc-maint at mcs.anl.gov On Jun 27, 2011, at 5:46 AM, Klaus Zimmermann wrote: > Hi all, > > in order to profit from cuda I tried to compile petsc-dev (latest hg). Unfortunately it fails on configuring metis (see below). > Is this a known problem? > > Thanks in advance, > Klaus > > -------------------- > kz38 at globus:~/src/extern/petsc-dev$ ./configure --download-metis --with-fortran --with-blacs-include=/opt/bwgrid/compiler/intel/ct_4.0/mkl/10.2.5.035/include --with-blas-lapack-dir=/opt/bwgrid/compiler/intel/ct_4.0/mkl/10.2.5.035 --with-scalar-type=complex --with-blacs --with-scalapack-lib=/opt/bwgrid/compiler/intel/ct_4.0/mkl/10.2.5.035/lib/em64t/libmkl_scalapack_lp64.a --with-clanguage=c++ --with-precision=double --with-scalapack --with-scalapack-include=/opt/bwgrid/compiler/intel/ct_4.0/mkl/10.2.5.035/include --with-blacs-lib=/opt/bwgrid/compiler/intel/ct_4.0/mkl/10.2.5.035/lib/em64t/libmkl_blacs_openmpi_lp64.a --with-cuda --with-cusp --with-thrust > =============================================================================== > Configuring PETSc to compile on your system > =============================================================================== > =============================================================================== Compiling & installing Metis; this may take several minutes =============================================================================== ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > Error running make on Metis: Could not execute "cd /home/kz38/src/extern/petsc-dev/externalpackages/metis-4.0.3 && make clean && make library && make minstall && make clean": > (cd Lib ; make clean ) > make[1]: Entering directory `/home/kz38/src/extern/petsc-dev/externalpackages/metis-4.0.3/Lib' > rm -f *.o > make[1]: Leaving directory `/home/kz38/src/extern/petsc-dev/externalpackages/metis-4.0.3/Lib' > (cd Programs ; make clean ) > make[1]: Entering directory `/home/kz38/src/extern/petsc-dev/externalpackages/metis-4.0.3/Programs' > rm -f *.o > make[1]: Leaving directory `/home/kz38/src/extern/petsc-dev/externalpackages/metis-4.0.3/Programs' > (cd Test ; make clean ) > make[1]: Entering directory `/home/kz38/src/extern/petsc-dev/externalpackages/metis-4.0.3/Test' > rm -f *.o > make[1]: Leaving directory `/home/kz38/src/extern/petsc-dev/externalpackages/metis-4.0.3/Test' > make: *** No rule to make target `library'. Stop. > ******************************************************************************* > From agrayver at gfz-potsdam.de Mon Jun 27 09:23:42 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Mon, 27 Jun 2011 16:23:42 +0200 Subject: [petsc-users] matrix-free KSPLSQR and matrix norm Message-ID: <4E08926E.10303@gfz-potsdam.de> Hello! I'm using matrix-free formulation and KSPLSQR to solve normal equations system: (A'*A)b = -g I managed to write all necessary code: call MatCreateShell(MPI_COMM_WORLD,m,N,N,N,PETSC_NULL_INTEGER,H,ierr) call MatShellSetOperation(H,MATOP_MULT,MFMatMult,ierr) call KSPCreate(MPI_COMM_WORLD,ksp,ierr);CHKERRQ(ierr) call KSPSetType(ksp,KSPLSQR,ierr);CHKERRQ(ierr) call KSPGetPC(ksp,pc,ierr);CHKERRQ(ierr) call PCSetType(pc,PCNONE,ierr);CHKERRQ(ierr) call KSPSetOperators(ksp,H,PETSC_NULL_OBJECT,SAME_PRECONDITIONER,ierr);CHKERRQ(ierr) call KSPSolve(ksp,g,b,ierr);CHKERRQ(ierr) but fail in KSPSolve stage with that: [0]PETSC ERROR: --------------------- Error Message ---------------------------- -------- [0]PETSC ERROR: Object is in wrong state! [0]PETSC ERROR: Mat object's type is not set: Argument # 1! [0]PETSC ERROR: ---------------------------------------------------------------- -------- [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 20 10 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ---------------------------------------------------------------- -------- [0]PETSC ERROR: D:\mt-soft\multiem\INV3D\INV3Dwin\x64\debug\em3d.exe on a cygwin -cx named MT0. by Unknown Mon Jun 27 16:18:43 2011 [0]PETSC ERROR: Libraries linked from /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-de bug/lib [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor t" --with-cxx="win32fe cl" --with-windows-graphics=0 --download-f-blas-lapack=1 --with-precision=double --with-scalar-type=complex --with-clanguage=cxx --with-m pi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include --with-mpi-lib="[/cygdrive/d/Dev /HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpife c.lib]" --useThreads=0 --useThreads=0 [0]PETSC ERROR: ---------------------------------------------------------------- -------- [0]PETSC ERROR: MatNorm() line 4507 in src/mat/interface/D:\dev\PETSC-3.1-P\src\mat\INTERF~1\matrix.c [0]PETSC ERROR: KSPSolve_LSQR() line 90 in src/ksp/ksp/impls/lsqr/D:\dev\PETSC-3.1-P\src\ksp\ksp\impls\lsqr\lsqr.c Any suggestions? Do I have to set MATOP_NORM operation? Then how? Regards, Alexander From bsmith at mcs.anl.gov Mon Jun 27 09:32:02 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 27 Jun 2011 09:32:02 -0500 Subject: [petsc-users] matrix-free KSPLSQR and matrix norm In-Reply-To: <4E08926E.10303@gfz-potsdam.de> References: <4E08926E.10303@gfz-potsdam.de> Message-ID: <02A939F1-6F10-4F41-A7F2-FD83AD767DBC@mcs.anl.gov> You should pass H as third argument to KSPSetOperators() as well as the second. Barry On Jun 27, 2011, at 9:23 AM, Alexander Grayver wrote: > Hello! > > I'm using matrix-free formulation and KSPLSQR to solve normal equations system: > > (A'*A)b = -g > > I managed to write all necessary code: > > call MatCreateShell(MPI_COMM_WORLD,m,N,N,N,PETSC_NULL_INTEGER,H,ierr) > call MatShellSetOperation(H,MATOP_MULT,MFMatMult,ierr) > > call KSPCreate(MPI_COMM_WORLD,ksp,ierr);CHKERRQ(ierr) > call KSPSetType(ksp,KSPLSQR,ierr);CHKERRQ(ierr) > call KSPGetPC(ksp,pc,ierr);CHKERRQ(ierr) > call PCSetType(pc,PCNONE,ierr);CHKERRQ(ierr) > call KSPSetOperators(ksp,H,PETSC_NULL_OBJECT,SAME_PRECONDITIONER,ierr);CHKERRQ(ierr) > call KSPSolve(ksp,g,b,ierr);CHKERRQ(ierr) > > > but fail in KSPSolve stage with that: > [0]PETSC ERROR: --------------------- Error Message ---------------------------- > -------- > [0]PETSC ERROR: Object is in wrong state! > [0]PETSC ERROR: Mat object's type is not set: Argument # 1! > [0]PETSC ERROR: ---------------------------------------------------------------- > -------- > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 20 > 10 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ---------------------------------------------------------------- > -------- > [0]PETSC ERROR: D:\mt-soft\multiem\INV3D\INV3Dwin\x64\debug\em3d.exe on a cygwin > -cx named MT0. by Unknown Mon Jun 27 16:18:43 2011 > [0]PETSC ERROR: Libraries linked from /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-de > bug/lib > [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 > [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor > t" --with-cxx="win32fe cl" --with-windows-graphics=0 --download-f-blas-lapack=1 > --with-precision=double --with-scalar-type=complex --with-clanguage=cxx --with-m > pi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include --with-mpi-lib="[/cygdrive/d/Dev > /HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpife > c.lib]" --useThreads=0 --useThreads=0 > [0]PETSC ERROR: ---------------------------------------------------------------- > -------- > [0]PETSC ERROR: MatNorm() line 4507 in src/mat/interface/D:\dev\PETSC-3.1-P\src\mat\INTERF~1\matrix.c > [0]PETSC ERROR: KSPSolve_LSQR() line 90 in src/ksp/ksp/impls/lsqr/D:\dev\PETSC-3.1-P\src\ksp\ksp\impls\lsqr\lsqr.c > > Any suggestions? Do I have to set MATOP_NORM operation? Then how? > > Regards, > Alexander From agrayver at gfz-potsdam.de Mon Jun 27 09:34:22 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Mon, 27 Jun 2011 16:34:22 +0200 Subject: [petsc-users] matrix-free KSPLSQR and matrix norm In-Reply-To: <02A939F1-6F10-4F41-A7F2-FD83AD767DBC@mcs.anl.gov> References: <4E08926E.10303@gfz-potsdam.de> <02A939F1-6F10-4F41-A7F2-FD83AD767DBC@mcs.anl.gov> Message-ID: <4E0894EE.3030900@gfz-potsdam.de> Thanks Barry, Now I get this: [0]PETSC ERROR: --------------------- Error Message ---------------------------- -------- [0]PETSC ERROR: No support for this operation for this object type! [0]PETSC ERROR: Mat type shell! [0]PETSC ERROR: ---------------------------------------------------------------- -------- [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 20 10 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ---------------------------------------------------------------- -------- [0]PETSC ERROR: test.exe on a cygwin -cx named MT0. by Unknown Mon Jun 27 16:30:42 2011 [0]PETSC ERROR: Libraries linked from /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-de bug/lib [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor t" --with-cxx="win32fe cl" --with-windows-graphics=0 --download-f-blas-lapack=1 --with-precision=double --with-scalar-type=complex --with-clanguage=cxx --with-m pi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include --with-mpi-lib="[/cygdrive/d/Dev /HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpife c.lib]" --useThreads=0 --useThreads=0 [0]PETSC ERROR: ---------------------------------------------------------------- -------- [0]PETSC ERROR: MatNorm() line 4512 in src/mat/interface/D:\dev\PETSC-3.1-P\src\mat\INTERF~1\matrix.c On 27.06.2011 16:32, Barry Smith wrote: > You should pass H as third argument to KSPSetOperators() as well as the second. > > Barry > > On Jun 27, 2011, at 9:23 AM, Alexander Grayver wrote: > >> Hello! >> >> I'm using matrix-free formulation and KSPLSQR to solve normal equations system: >> >> (A'*A)b = -g >> >> I managed to write all necessary code: >> >> call MatCreateShell(MPI_COMM_WORLD,m,N,N,N,PETSC_NULL_INTEGER,H,ierr) >> call MatShellSetOperation(H,MATOP_MULT,MFMatMult,ierr) >> >> call KSPCreate(MPI_COMM_WORLD,ksp,ierr);CHKERRQ(ierr) >> call KSPSetType(ksp,KSPLSQR,ierr);CHKERRQ(ierr) >> call KSPGetPC(ksp,pc,ierr);CHKERRQ(ierr) >> call PCSetType(pc,PCNONE,ierr);CHKERRQ(ierr) >> call KSPSetOperators(ksp,H,PETSC_NULL_OBJECT,SAME_PRECONDITIONER,ierr);CHKERRQ(ierr) >> call KSPSolve(ksp,g,b,ierr);CHKERRQ(ierr) >> >> >> but fail in KSPSolve stage with that: >> [0]PETSC ERROR: --------------------- Error Message ---------------------------- >> -------- >> [0]PETSC ERROR: Object is in wrong state! >> [0]PETSC ERROR: Mat object's type is not set: Argument # 1! >> [0]PETSC ERROR: ---------------------------------------------------------------- >> -------- >> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 20 >> 10 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: ---------------------------------------------------------------- >> -------- >> [0]PETSC ERROR: test.exe on a cygwin >> -cx named MT0. by Unknown Mon Jun 27 16:18:43 2011 >> [0]PETSC ERROR: Libraries linked from /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-de >> bug/lib >> [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 >> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor >> t" --with-cxx="win32fe cl" --with-windows-graphics=0 --download-f-blas-lapack=1 >> --with-precision=double --with-scalar-type=complex --with-clanguage=cxx --with-m >> pi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include --with-mpi-lib="[/cygdrive/d/Dev >> /HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpife >> c.lib]" --useThreads=0 --useThreads=0 >> [0]PETSC ERROR: ---------------------------------------------------------------- >> -------- >> [0]PETSC ERROR: MatNorm() line 4507 in src/mat/interface/D:\dev\PETSC-3.1-P\src\mat\INTERF~1\matrix.c >> [0]PETSC ERROR: KSPSolve_LSQR() line 90 in src/ksp/ksp/impls/lsqr/D:\dev\PETSC-3.1-P\src\ksp\ksp\impls\lsqr\lsqr.c >> >> Any suggestions? Do I have to set MATOP_NORM operation? Then how? >> >> Regards, >> Alexander From knepley at gmail.com Mon Jun 27 09:47:30 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 27 Jun 2011 09:47:30 -0500 Subject: [petsc-users] matrix-free KSPLSQR and matrix norm In-Reply-To: <4E0894EE.3030900@gfz-potsdam.de> References: <4E08926E.10303@gfz-potsdam.de> <02A939F1-6F10-4F41-A7F2-FD83AD767DBC@mcs.anl.gov> <4E0894EE.3030900@gfz-potsdam.de> Message-ID: On Mon, Jun 27, 2011 at 9:34 AM, Alexander Grayver wrote: > Thanks Barry, > > Now I get this: Please send the entire error message (this is missing the stack), but I am guessing you are not changed the default preconditioner (ILU) which cannot act on a shell matrix. Use -ksp_view to see what solver is begin used. Matt > [0]PETSC ERROR: --------------------- Error Message > ---------------------------- > -------- > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: Mat type shell! > > [0]PETSC ERROR: ------------------------------** > ------------------------------**---- > -------- > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 > CST 20 > 10 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------** > ------------------------------**---- > -------- > [0]PETSC ERROR: test.exe on a cygwin > -cx named MT0. by Unknown Mon Jun 27 16:30:42 2011 > > [0]PETSC ERROR: Libraries linked from /cygdrive/d/dev/petsc-3.1-p7/** > cygwin-cxx-de > bug/lib > [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 > [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe > ifor > t" --with-cxx="win32fe cl" --with-windows-graphics=0 > --download-f-blas-lapack=1 > --with-precision=double --with-scalar-type=complex --with-clanguage=cxx > --with-m > pi-include=/cygdrive/d/Dev/**HPC_2008_SDK/Include > --with-mpi-lib="[/cygdrive/d/**Dev > /HPC_2008_SDK/Lib/amd64/msmpi.**lib,/cygdrive/d/Dev/HPC_2008_** > SDK/Lib/amd64/msmpife > c.lib]" --useThreads=0 --useThreads=0 > [0]PETSC ERROR: ------------------------------** > ------------------------------**---- > -------- > [0]PETSC ERROR: MatNorm() line 4512 in src/mat/interface/D:\dev\** > PETSC-3.1-P\src\mat\INTERF~1\**matrix.c > > > On 27.06.2011 16:32, Barry Smith wrote: > >> You should pass H as third argument to KSPSetOperators() as well as the >> second. >> >> Barry >> >> On Jun 27, 2011, at 9:23 AM, Alexander Grayver wrote: >> >> Hello! >>> >>> I'm using matrix-free formulation and KSPLSQR to solve normal equations >>> system: >>> >>> (A'*A)b = -g >>> >>> I managed to write all necessary code: >>> >>> call MatCreateShell(MPI_COMM_WORLD,**m,N,N,N,PETSC_NULL_INTEGER,H,** >>> ierr) >>> call MatShellSetOperation(H,MATOP_**MULT,MFMatMult,ierr) >>> >>> call KSPCreate(MPI_COMM_WORLD,ksp,**ierr);CHKERRQ(ierr) >>> call KSPSetType(ksp,KSPLSQR,ierr);**CHKERRQ(ierr) >>> call KSPGetPC(ksp,pc,ierr);CHKERRQ(**ierr) >>> call PCSetType(pc,PCNONE,ierr);**CHKERRQ(ierr) >>> call KSPSetOperators(ksp,H,PETSC_**NULL_OBJECT,SAME_** >>> PRECONDITIONER,ierr);CHKERRQ(**ierr) >>> call KSPSolve(ksp,g,b,ierr);**CHKERRQ(ierr) >>> >>> >>> but fail in KSPSolve stage with that: >>> [0]PETSC ERROR: --------------------- Error Message >>> ---------------------------- >>> -------- >>> [0]PETSC ERROR: Object is in wrong state! >>> [0]PETSC ERROR: Mat object's type is not set: Argument # 1! >>> [0]PETSC ERROR: ------------------------------** >>> ------------------------------**---- >>> -------- >>> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 >>> CST 20 >>> 10 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: ------------------------------** >>> ------------------------------**---- >>> -------- >>> [0]PETSC ERROR: test.exe on a cygwin >>> >>> -cx named MT0. by Unknown Mon Jun 27 16:18:43 2011 >>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/dev/petsc-3.1-p7/** >>> cygwin-cxx-de >>> bug/lib >>> [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 >>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" >>> --with-fc="win32fe ifor >>> t" --with-cxx="win32fe cl" --with-windows-graphics=0 >>> --download-f-blas-lapack=1 >>> --with-precision=double --with-scalar-type=complex --with-clanguage=cxx >>> --with-m >>> pi-include=/cygdrive/d/Dev/**HPC_2008_SDK/Include >>> --with-mpi-lib="[/cygdrive/d/**Dev >>> /HPC_2008_SDK/Lib/amd64/msmpi.**lib,/cygdrive/d/Dev/HPC_2008_** >>> SDK/Lib/amd64/msmpife >>> c.lib]" --useThreads=0 --useThreads=0 >>> [0]PETSC ERROR: ------------------------------** >>> ------------------------------**---- >>> -------- >>> [0]PETSC ERROR: MatNorm() line 4507 in src/mat/interface/D:\dev\** >>> PETSC-3.1-P\src\mat\INTERF~1\**matrix.c >>> [0]PETSC ERROR: KSPSolve_LSQR() line 90 in src/ksp/ksp/impls/lsqr/D:\dev\ >>> **PETSC-3.1-P\src\ksp\ksp\impls\**lsqr\lsqr.c >>> >>> Any suggestions? Do I have to set MATOP_NORM operation? Then how? >>> >>> Regards, >>> Alexander >>> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Mon Jun 27 09:54:22 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Mon, 27 Jun 2011 16:54:22 +0200 Subject: [petsc-users] matrix-free KSPLSQR and matrix norm In-Reply-To: References: <4E08926E.10303@gfz-potsdam.de> <02A939F1-6F10-4F41-A7F2-FD83AD767DBC@mcs.anl.gov> <4E0894EE.3030900@gfz-potsdam.de> Message-ID: <4E08999E.5060901@gfz-potsdam.de> >> Please send the entire error message (this is missing the stack), but I am guessing you >> are not changed the default preconditioner (ILU) which cannot act on a shell matrix. Use >> -ksp_view to see what solver is begin used. The full error message: test.exe -log_summary -ksp_view [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: No support for this operation for this object type! [0]PETSC ERROR: Mat type shell! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 2010 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: test.exe on a cygwin-cx named M0. by Unknown Mon Jun 27 16:51:32 2011 [0]PETSC ERROR: Libraries linked from /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifort" --with-cxx="win32fe cl" --with-windows-graphics=0 --with-precision=double --with-scalar-type=complex --with-clanguage=cxx --with-mpi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include --wi /HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib]" --useThreads=0 --useThreads=0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatNorm() line 4512 in src/mat/interface/D:\dev\PETSC-~3.1-P\src\mat\INTERF~1\matrix.c [0]PETSC ERROR: KSPSolve_LSQR() line 90 in src/ksp/ksp/impls/lsqr/D:\dev\PETSC-~3.1-P\src\ksp\ksp\impls\lsqr\lsqr.c [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/D:\dev\PETSC-~3.1-P\src\ksp\ksp\INTERF~1\itfunc.c job aborted: [ranks] message [0] application aborted aborting MPI_COMM_WORLD, error 56, comm rank 0 On 27.06.2011 16:47, Matthew Knepley wrote: > On Mon, Jun 27, 2011 at 9:34 AM, Alexander Grayver > > wrote: > > Thanks Barry, > > Now I get this: > > > Please send the entire error message (this is missing the stack), but > I am guessing you > are not changed the default preconditioner (ILU) which cannot act on a > shell matrix. Use > -ksp_view to see what solver is begin used. > > Matt > > [0]PETSC ERROR: --------------------- Error Message > ---------------------------- > -------- > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: Mat type shell! > > [0]PETSC ERROR: > ---------------------------------------------------------------- > -------- > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 > 14:26:37 CST 20 > 10 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ---------------------------------------------------------------- > -------- > [0]PETSC ERROR: test.exe on a cygwin > -cx named MT0. by Unknown Mon Jun 27 16:30:42 2011 > > [0]PETSC ERROR: Libraries linked from > /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-de > bug/lib > [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 > [0]PETSC ERROR: Configure options --with-cc="win32fe cl" > --with-fc="win32fe ifor > t" --with-cxx="win32fe cl" --with-windows-graphics=0 > --download-f-blas-lapack=1 > --with-precision=double --with-scalar-type=complex > --with-clanguage=cxx --with-m > pi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include > --with-mpi-lib="[/cygdrive/d/Dev > /HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpife > c.lib]" --useThreads=0 --useThreads=0 > [0]PETSC ERROR: > ---------------------------------------------------------------- > -------- > [0]PETSC ERROR: MatNorm() line 4512 in > src/mat/interface/D:\dev\PETSC-3.1-P\src\mat\INTERF~1\matrix.c > > > On 27.06.2011 16 :32, Barry Smith wrote: > > You should pass H as third argument to KSPSetOperators() as > well as the second. > > Barry > > On Jun 27, 2011, at 9:23 AM, Alexander Grayver wrote: > > Hello! > > I'm using matrix-free formulation and KSPLSQR to solve > normal equations system: > > (A'*A)b = -g > > I managed to write all necessary code: > > call > MatCreateShell(MPI_COMM_WORLD,m,N,N,N,PETSC_NULL_INTEGER,H,ierr) > call MatShellSetOperation(H,MATOP_MULT,MFMatMult,ierr) > > call KSPCreate(MPI_COMM_WORLD,ksp,ierr);CHKERRQ(ierr) > call KSPSetType(ksp,KSPLSQR,ierr);CHKERRQ(ierr) > call KSPGetPC(ksp,pc,ierr);CHKERRQ(ierr) > call PCSetType(pc,PCNONE,ierr);CHKERRQ(ierr) > call > KSPSetOperators(ksp,H,PETSC_NULL_OBJECT,SAME_PRECONDITIONER,ierr);CHKERRQ(ierr) > call KSPSolve(ksp,g,b,ierr);CHKERRQ(ierr) > > > but fail in KSPSolve stage with that: > [0]PETSC ERROR: --------------------- Error Message > ---------------------------- > -------- > [0]PETSC ERROR: Object is in wrong state! > [0]PETSC ERROR: Mat object's type is not set: Argument # 1! > [0]PETSC ERROR: > ---------------------------------------------------------------- > -------- > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon > Dec 20 14:26:37 CST 20 > 10 > [0]PETSC ERROR: See docs/changes/index.html for recent > updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble > shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ---------------------------------------------------------------- > -------- > [0]PETSC ERROR: test.exe on a cygwin > > -cx named MT0. by Unknown Mon Jun 27 16:18:43 2011 > [0]PETSC ERROR: Libraries linked from > /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-de > bug/lib > [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 > [0]PETSC ERROR: Configure options --with-cc="win32fe cl" > --with-fc="win32fe ifor > t" --with-cxx="win32fe cl" --with-windows-graphics=0 > --download-f-blas-lapack=1 > --with-precision=double --with-scalar-type=complex > --with-clanguage=cxx --with-m > pi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include > --with-mpi-lib="[/cygdrive/d/Dev > /HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpife > c.lib]" --useThreads=0 --useThreads=0 > [0]PETSC ERROR: > ---------------------------------------------------------------- > -------- > [0]PETSC ERROR: MatNorm() line 4507 in > src/mat/interface/D:\dev\PETSC-3.1-P\src\mat\INTERF~1\matrix.c > [0]PETSC ERROR: KSPSolve_LSQR() line 90 in > src/ksp/ksp/impls/lsqr/D:\dev\PETSC-3.1-P\src\ksp\ksp\impls\lsqr\lsqr.c > > Any suggestions? Do I have to set MATOP_NORM operation? > Then how? > > Regards, > Alexander > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jun 27 10:12:49 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 27 Jun 2011 10:12:49 -0500 Subject: [petsc-users] matrix-free KSPLSQR and matrix norm In-Reply-To: <4E08999E.5060901@gfz-potsdam.de> References: <4E08926E.10303@gfz-potsdam.de> <02A939F1-6F10-4F41-A7F2-FD83AD767DBC@mcs.anl.gov> <4E0894EE.3030900@gfz-potsdam.de> <4E08999E.5060901@gfz-potsdam.de> Message-ID: On Mon, Jun 27, 2011 at 9:54 AM, Alexander Grayver wrote: > ** > >> Please send the entire error message (this is missing the stack), but I > am guessing you > >> are not changed the default preconditioner (ILU) which cannot act on a > shell matrix. Use > >> -ksp_view to see what solver is begin used. > > The full error message: > > test.exe -log_summary -ksp_view > The LSQR solver uses the Frobenius norm of the matrix. In order to use a MATSHELL, you must implement MatNorm using http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Mat/MatShellSetOperation.html Thanks, Matt > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: Mat type shell! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 > CST 2010 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: test.exe on a cygwin-cx named M0. by Unknown Mon Jun 27 > 16:51:32 2011 > [0]PETSC ERROR: Libraries linked from > /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib > > [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 > [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe > ifort" --with-cxx="win32fe cl" --with-windows-graphics=0 > --with-precision=double --with-scalar-type=complex --with-clanguage=cxx > --with-mpi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include --wi > /HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib]" > --useThreads=0 --useThreads=0 > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatNorm() line 4512 in > src/mat/interface/D:\dev\PETSC-~3.1-P\src\mat\INTERF~1\matrix.c > [0]PETSC ERROR: KSPSolve_LSQR() line 90 in > src/ksp/ksp/impls/lsqr/D:\dev\PETSC-~3.1-P\src\ksp\ksp\impls\lsqr\lsqr.c > [0]PETSC ERROR: KSPSolve() line 396 in > src/ksp/ksp/interface/D:\dev\PETSC-~3.1-P\src\ksp\ksp\INTERF~1\itfunc.c > > job aborted: > [ranks] message > > [0] application aborted > aborting MPI_COMM_WORLD, error 56, comm rank 0 > > > On 27.06.2011 16:47, Matthew Knepley wrote: > > On Mon, Jun 27, 2011 at 9:34 AM, Alexander Grayver < > agrayver at gfz-potsdam.de> wrote: > >> Thanks Barry, >> >> Now I get this: > > > Please send the entire error message (this is missing the stack), but I > am guessing you > are not changed the default preconditioner (ILU) which cannot act on a > shell matrix. Use > -ksp_view to see what solver is begin used. > > Matt > > >> [0]PETSC ERROR: --------------------- Error Message >> ---------------------------- >> -------- >> [0]PETSC ERROR: No support for this operation for this object type! >> [0]PETSC ERROR: Mat type shell! >> >> [0]PETSC ERROR: >> ---------------------------------------------------------------- >> -------- >> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 >> CST 20 >> 10 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ---------------------------------------------------------------- >> -------- >> [0]PETSC ERROR: test.exe on a cygwin >> -cx named MT0. by Unknown Mon Jun 27 16:30:42 2011 >> >> [0]PETSC ERROR: Libraries linked from >> /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-de >> bug/lib >> [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 >> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" >> --with-fc="win32fe ifor >> t" --with-cxx="win32fe cl" --with-windows-graphics=0 >> --download-f-blas-lapack=1 >> --with-precision=double --with-scalar-type=complex --with-clanguage=cxx >> --with-m >> pi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include >> --with-mpi-lib="[/cygdrive/d/Dev >> >> /HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpife >> c.lib]" --useThreads=0 --useThreads=0 >> [0]PETSC ERROR: >> ---------------------------------------------------------------- >> -------- >> [0]PETSC ERROR: MatNorm() line 4512 in >> src/mat/interface/D:\dev\PETSC-3.1-P\src\mat\INTERF~1\matrix.c >> >> >> On 27.06.2011 16:32, Barry Smith wrote: >> >>> You should pass H as third argument to KSPSetOperators() as well as >>> the second. >>> >>> Barry >>> >>> On Jun 27, 2011, at 9:23 AM, Alexander Grayver wrote: >>> >>> Hello! >>>> >>>> I'm using matrix-free formulation and KSPLSQR to solve normal equations >>>> system: >>>> >>>> (A'*A)b = -g >>>> >>>> I managed to write all necessary code: >>>> >>>> call MatCreateShell(MPI_COMM_WORLD,m,N,N,N,PETSC_NULL_INTEGER,H,ierr) >>>> call MatShellSetOperation(H,MATOP_MULT,MFMatMult,ierr) >>>> >>>> call KSPCreate(MPI_COMM_WORLD,ksp,ierr);CHKERRQ(ierr) >>>> call KSPSetType(ksp,KSPLSQR,ierr);CHKERRQ(ierr) >>>> call KSPGetPC(ksp,pc,ierr);CHKERRQ(ierr) >>>> call PCSetType(pc,PCNONE,ierr);CHKERRQ(ierr) >>>> call >>>> KSPSetOperators(ksp,H,PETSC_NULL_OBJECT,SAME_PRECONDITIONER,ierr);CHKERRQ(ierr) >>>> call KSPSolve(ksp,g,b,ierr);CHKERRQ(ierr) >>>> >>>> >>>> but fail in KSPSolve stage with that: >>>> [0]PETSC ERROR: --------------------- Error Message >>>> ---------------------------- >>>> -------- >>>> [0]PETSC ERROR: Object is in wrong state! >>>> [0]PETSC ERROR: Mat object's type is not set: Argument # 1! >>>> [0]PETSC ERROR: >>>> ---------------------------------------------------------------- >>>> -------- >>>> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 >>>> 14:26:37 CST 20 >>>> 10 >>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> [0]PETSC ERROR: >>>> ---------------------------------------------------------------- >>>> -------- >>>> [0]PETSC ERROR: test.exe on a cygwin >>>> >>>> -cx named MT0. by Unknown Mon Jun 27 16:18:43 2011 >>>> [0]PETSC ERROR: Libraries linked from >>>> /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-de >>>> bug/lib >>>> [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 >>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" >>>> --with-fc="win32fe ifor >>>> t" --with-cxx="win32fe cl" --with-windows-graphics=0 >>>> --download-f-blas-lapack=1 >>>> --with-precision=double --with-scalar-type=complex --with-clanguage=cxx >>>> --with-m >>>> pi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include >>>> --with-mpi-lib="[/cygdrive/d/Dev >>>> >>>> /HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpife >>>> c.lib]" --useThreads=0 --useThreads=0 >>>> [0]PETSC ERROR: >>>> ---------------------------------------------------------------- >>>> -------- >>>> [0]PETSC ERROR: MatNorm() line 4507 in >>>> src/mat/interface/D:\dev\PETSC-3.1-P\src\mat\INTERF~1\matrix.c >>>> [0]PETSC ERROR: KSPSolve_LSQR() line 90 in >>>> src/ksp/ksp/impls/lsqr/D:\dev\PETSC-3.1-P\src\ksp\ksp\impls\lsqr\lsqr.c >>>> >>>> Any suggestions? Do I have to set MATOP_NORM operation? Then how? >>>> >>>> Regards, >>>> Alexander >>>> >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Mon Jun 27 12:44:46 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Mon, 27 Jun 2011 19:44:46 +0200 Subject: [petsc-users] matrix-free KSPLSQR and matrix norm In-Reply-To: References: <4E08926E.10303@gfz-potsdam.de> <02A939F1-6F10-4F41-A7F2-FD83AD767DBC@mcs.anl.gov> <4E0894EE.3030900@gfz-potsdam.de> <4E08999E.5060901@gfz-potsdam.de> Message-ID: <4E08C18E.7030406@gfz-potsdam.de> Actually, I tried, but haven't managed to do this in fortran: test.exe -log_summary -ksp_view [0]PETSC ERROR: MatShellSetOperation_Fortran() line 107 in src/mat/impls/shell/ftn-custom/D:\dev\PETSC-~3.1-P\src\mat\impls\shell\FTN-CU~1\zshellf.c job aborted: [ranks] message [0] application aborted aborting MPI_COMM_WORLD, error 1, comm rank 0 It seems that MATOP_NORM is not supported from Fortran. Why is that so? Regards, Alexander On 27.06.2011 17:12, Matthew Knepley wrote: > On Mon, Jun 27, 2011 at 9:54 AM, Alexander Grayver > > wrote: > > >> Please send the entire error message (this is missing the > stack), but I am guessing you > >> are not changed the default preconditioner (ILU) which cannot > act on a shell matrix. Use > >> -ksp_view to see what solver is begin used. > > The full error message: > > test.exe -log_summary -ksp_view > > > The LSQR solver uses the Frobenius norm of the matrix. In order to use > a MATSHELL, you > must implement MatNorm using > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Mat/MatShellSetOperation.html > > Thanks, > > Matt > > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: Mat type shell! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 > 14:26:37 CST 2010 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: test.exe on a cygwin-cx named M0. by Unknown Mon > Jun 27 16:51:32 2011 > [0]PETSC ERROR: Libraries linked from > /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib > > [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 > [0]PETSC ERROR: Configure options --with-cc="win32fe cl" > --with-fc="win32fe ifort" --with-cxx="win32fe cl" > --with-windows-graphics=0 > --with-precision=double --with-scalar-type=complex > --with-clanguage=cxx > --with-mpi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include --wi > /HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib]" > --useThreads=0 --useThreads=0 > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatNorm() line 4512 in > src/mat/interface/D:\dev\PETSC-~3.1-P\src\mat\INTERF~1\matrix.c > [0]PETSC ERROR: KSPSolve_LSQR() line 90 in > src/ksp/ksp/impls/lsqr/D:\dev\PETSC-~3.1-P\src\ksp\ksp\impls\lsqr\lsqr.c > [0]PETSC ERROR: KSPSolve() line 396 in > src/ksp/ksp/interface/D:\dev\PETSC-~3.1-P\src\ksp\ksp\INTERF~1\itfunc.c > > job aborted: > [ranks] message > > [0] application aborted > aborting MPI_COMM_WORLD, error 56, comm rank 0 > > > On 27.06.2011 16 :47, Matthew Knepley wrote: >> On Mon, Jun 27, 2011 at 9:34 AM, Alexander Grayver >> > wrote: >> >> Thanks Barry, >> >> Now I get this: >> >> >> Please send the entire error message (this is missing the stack), >> but I am guessing you >> are not changed the default preconditioner (ILU) which cannot act >> on a shell matrix. Use >> -ksp_view to see what solver is begin used. >> >> Matt >> >> [0]PETSC ERROR: --------------------- Error Message >> ---------------------------- >> -------- >> [0]PETSC ERROR: No support for this operation for this object >> type! >> [0]PETSC ERROR: Mat type shell! >> >> [0]PETSC ERROR: >> ---------------------------------------------------------------- >> -------- >> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec >> 20 14:26:37 CST 20 >> 10 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble >> shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ---------------------------------------------------------------- >> -------- >> [0]PETSC ERROR: test.exe on a cygwin >> -cx named MT0. by Unknown Mon Jun 27 16:30:42 2011 >> >> [0]PETSC ERROR: Libraries linked from >> /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-de >> bug/lib >> [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 >> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" >> --with-fc="win32fe ifor >> t" --with-cxx="win32fe cl" --with-windows-graphics=0 >> --download-f-blas-lapack=1 >> --with-precision=double --with-scalar-type=complex >> --with-clanguage=cxx --with-m >> pi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include >> --with-mpi-lib="[/cygdrive/d/Dev >> /HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpife >> c.lib]" --useThreads=0 --useThreads=0 >> [0]PETSC ERROR: >> ---------------------------------------------------------------- >> -------- >> [0]PETSC ERROR: MatNorm() line 4512 in >> src/mat/interface/D:\dev\PETSC-3.1-P\src\mat\INTERF~1\matrix.c >> >> >> On 27.06.2011 16 :32, Barry Smith wrote: >> >> You should pass H as third argument to >> KSPSetOperators() as well as the second. >> >> Barry >> >> On Jun 27, 2011, at 9:23 AM, Alexander Grayver wrote: >> >> Hello! >> >> I'm using matrix-free formulation and KSPLSQR to >> solve normal equations system: >> >> (A'*A)b = -g >> >> I managed to write all necessary code: >> >> call >> MatCreateShell(MPI_COMM_WORLD,m,N,N,N,PETSC_NULL_INTEGER,H,ierr) >> call MatShellSetOperation(H,MATOP_MULT,MFMatMult,ierr) >> >> call KSPCreate(MPI_COMM_WORLD,ksp,ierr);CHKERRQ(ierr) >> call KSPSetType(ksp,KSPLSQR,ierr);CHKERRQ(ierr) >> call KSPGetPC(ksp,pc,ierr);CHKERRQ(ierr) >> call PCSetType(pc,PCNONE,ierr);CHKERRQ(ierr) >> call >> KSPSetOperators(ksp,H,PETSC_NULL_OBJECT,SAME_PRECONDITIONER,ierr);CHKERRQ(ierr) >> call KSPSolve(ksp,g,b,ierr);CHKERRQ(ierr) >> >> >> but fail in KSPSolve stage with that: >> [0]PETSC ERROR: --------------------- Error Message >> ---------------------------- >> -------- >> [0]PETSC ERROR: Object is in wrong state! >> [0]PETSC ERROR: Mat object's type is not set: >> Argument # 1! >> [0]PETSC ERROR: >> ---------------------------------------------------------------- >> -------- >> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, >> Mon Dec 20 14:26:37 CST 20 >> 10 >> [0]PETSC ERROR: See docs/changes/index.html for >> recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about >> trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ---------------------------------------------------------------- >> -------- >> [0]PETSC ERROR: test.exe on a cygwin >> >> -cx named MT0. by Unknown Mon Jun 27 16:18:43 2011 >> [0]PETSC ERROR: Libraries linked from >> /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-de >> bug/lib >> [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 >> [0]PETSC ERROR: Configure options --with-cc="win32fe >> cl" --with-fc="win32fe ifor >> t" --with-cxx="win32fe cl" --with-windows-graphics=0 >> --download-f-blas-lapack=1 >> --with-precision=double --with-scalar-type=complex >> --with-clanguage=cxx --with-m >> pi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include >> --with-mpi-lib="[/cygdrive/d/Dev >> /HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpife >> c.lib]" --useThreads=0 --useThreads=0 >> [0]PETSC ERROR: >> ---------------------------------------------------------------- >> -------- >> [0]PETSC ERROR: MatNorm() line 4507 in >> src/mat/interface/D:\dev\PETSC-3.1-P\src\mat\INTERF~1\matrix.c >> [0]PETSC ERROR: KSPSolve_LSQR() line 90 in >> src/ksp/ksp/impls/lsqr/D:\dev\PETSC-3.1-P\src\ksp\ksp\impls\lsqr\lsqr.c >> >> Any suggestions? Do I have to set MATOP_NORM >> operation? Then how? >> >> Regards, >> Alexander >> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenglun.wei at gmail.com Mon Jun 27 14:28:32 2011 From: zhenglun.wei at gmail.com (Alan Wei) Date: Mon, 27 Jun 2011 14:28:32 -0500 Subject: [petsc-users] Nested Poisson Solver Message-ID: Dear Sir/Madam, Recently, I studied the /src/ksp/ksp/example/tutorial/ex29.c of solving a 2D Poisson equation. It employs the Multigrid Method on a uniform coordinate. I wonder if there is a way to solve the Poisson equation with a nested grid in PETSc. Any suggestions on any example or thoughts is greatly appreciated. ^_^ best, Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jun 27 14:53:24 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 27 Jun 2011 14:53:24 -0500 Subject: [petsc-users] Nested Poisson Solver In-Reply-To: References: Message-ID: <1EF8F053-DB26-4620-AD86-3D25DD9935FB@mcs.anl.gov> On Jun 27, 2011, at 2:28 PM, Alan Wei wrote: > Dear Sir/Madam, > Recently, I studied the /src/ksp/ksp/example/tutorial/ex29.c of solving a 2D Poisson equation. It employs the Multigrid Method on a uniform coordinate. I wonder if there is a way to solve the Poisson equation with a nested grid in PETSc. Any suggestions on any example or thoughts is greatly appreciated. ^_^ > > best, > Alan I am not aware of anyone whose used PETSc to solver on a nested grid. Barry From bsmith at mcs.anl.gov Mon Jun 27 16:16:16 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 27 Jun 2011 16:16:16 -0500 Subject: [petsc-users] matrix-free KSPLSQR and matrix norm In-Reply-To: <4E08C18E.7030406@gfz-potsdam.de> References: <4E08926E.10303@gfz-potsdam.de> <02A939F1-6F10-4F41-A7F2-FD83AD767DBC@mcs.anl.gov> <4E0894EE.3030900@gfz-potsdam.de> <4E08999E.5060901@gfz-potsdam.de> <4E08C18E.7030406@gfz-potsdam.de> Message-ID: <23AACEF1-BC05-428D-A8FA-00D6087BB4E7@mcs.anl.gov> This code is kind of silly because it doesn't actually ever use that Matrix norm by default so in petsc-3.1 just edit src/ksp/ksp/impls/lsqr/lsqr.c and comment out the line where it calls MatNorm() then run make in that one directory Good luck Barry On Jun 27, 2011, at 12:44 PM, Alexander Grayver wrote: > Actually, I tried, but haven't managed to do this in fortran: > > test.exe -log_summary -ksp_view > [0]PETSC ERROR: MatShellSetOperation_Fortran() line 107 in src/mat/impls/shell/ftn-custom/D:\dev\PETSC-~3.1-P\src\mat\impls\shell\FTN-CU~1\zshellf.c > > job aborted: > [ranks] message > > [0] application aborted > aborting MPI_COMM_WORLD, error 1, comm rank 0 > > It seems that MATOP_NORM is not supported from Fortran. Why is that so? > > Regards, > Alexander > > On 27.06.2011 17:12, Matthew Knepley wrote: >> On Mon, Jun 27, 2011 at 9:54 AM, Alexander Grayver wrote: >> >> Please send the entire error message (this is missing the stack), but I am guessing you >> >> are not changed the default preconditioner (ILU) which cannot act on a shell matrix. Use >> >> -ksp_view to see what solver is begin used. >> >> The full error message: >> >> test.exe -log_summary -ksp_view >> >> The LSQR solver uses the Frobenius norm of the matrix. In order to use a MATSHELL, you >> must implement MatNorm using >> >> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Mat/MatShellSetOperation.html >> >> Thanks, >> >> Matt >> >> >> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [0]PETSC ERROR: No support for this operation for this object type! >> [0]PETSC ERROR: Mat type shell! >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 2010 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: test.exe on a cygwin-cx named M0. by Unknown Mon Jun 27 16:51:32 2011 >> [0]PETSC ERROR: Libraries linked from /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib >> >> [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 >> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifort" --with-cxx="win32fe cl" --with-windows-graphics=0 >> --with-precision=double --with-scalar-type=complex --with-clanguage=cxx --with-mpi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include --wi >> /HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib]" --useThreads=0 --useThreads=0 >> >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: MatNorm() line 4512 in src/mat/interface/D:\dev\PETSC-~3.1-P\src\mat\INTERF~1\matrix.c >> [0]PETSC ERROR: KSPSolve_LSQR() line 90 in src/ksp/ksp/impls/lsqr/D:\dev\PETSC-~3.1-P\src\ksp\ksp\impls\lsqr\lsqr.c >> [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/D:\dev\PETSC-~3.1-P\src\ksp\ksp\INTERF~1\itfunc.c >> >> job aborted: >> [ranks] message >> >> [0] application aborted >> aborting MPI_COMM_WORLD, error 56, comm rank 0 >> >> >> On 27.06.2011 16:47, Matthew Knepley wrote: >>> On Mon, Jun 27, 2011 at 9:34 AM, Alexander Grayver wrote: >>> Thanks Barry, >>> >>> Now I get this: >>> >>> Please send the entire error message (this is missing the stack), but I am guessing you >>> are not changed the default preconditioner (ILU) which cannot act on a shell matrix. Use >>> -ksp_view to see what solver is begin used. >>> >>> Matt >>> >>> [0]PETSC ERROR: --------------------- Error Message ---------------------------- >>> -------- >>> [0]PETSC ERROR: No support for this operation for this object type! >>> [0]PETSC ERROR: Mat type shell! >>> >>> [0]PETSC ERROR: ---------------------------------------------------------------- >>> -------- >>> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 20 >>> 10 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: ---------------------------------------------------------------- >>> -------- >>> [0]PETSC ERROR: test.exe on a cygwin >>> -cx named MT0. by Unknown Mon Jun 27 16:30:42 2011 >>> >>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-de >>> bug/lib >>> [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 >>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor >>> t" --with-cxx="win32fe cl" --with-windows-graphics=0 --download-f-blas-lapack=1 >>> --with-precision=double --with-scalar-type=complex --with-clanguage=cxx --with-m >>> pi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include --with-mpi-lib="[/cygdrive/d/Dev >>> /HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpife >>> c.lib]" --useThreads=0 --useThreads=0 >>> [0]PETSC ERROR: ---------------------------------------------------------------- >>> -------- >>> [0]PETSC ERROR: MatNorm() line 4512 in src/mat/interface/D:\dev\PETSC-3.1-P\src\mat\INTERF~1\matrix.c >>> >>> >>> On 27.06.2011 16:32, Barry Smith wrote: >>> You should pass H as third argument to KSPSetOperators() as well as the second. >>> >>> Barry >>> >>> On Jun 27, 2011, at 9:23 AM, Alexander Grayver wrote: >>> >>> Hello! >>> >>> I'm using matrix-free formulation and KSPLSQR to solve normal equations system: >>> >>> (A'*A)b = -g >>> >>> I managed to write all necessary code: >>> >>> call MatCreateShell(MPI_COMM_WORLD,m,N,N,N,PETSC_NULL_INTEGER,H,ierr) >>> call MatShellSetOperation(H,MATOP_MULT,MFMatMult,ierr) >>> >>> call KSPCreate(MPI_COMM_WORLD,ksp,ierr);CHKERRQ(ierr) >>> call KSPSetType(ksp,KSPLSQR,ierr);CHKERRQ(ierr) >>> call KSPGetPC(ksp,pc,ierr);CHKERRQ(ierr) >>> call PCSetType(pc,PCNONE,ierr);CHKERRQ(ierr) >>> call KSPSetOperators(ksp,H,PETSC_NULL_OBJECT,SAME_PRECONDITIONER,ierr);CHKERRQ(ierr) >>> call KSPSolve(ksp,g,b,ierr);CHKERRQ(ierr) >>> >>> >>> but fail in KSPSolve stage with that: >>> [0]PETSC ERROR: --------------------- Error Message ---------------------------- >>> -------- >>> [0]PETSC ERROR: Object is in wrong state! >>> [0]PETSC ERROR: Mat object's type is not set: Argument # 1! >>> [0]PETSC ERROR: ---------------------------------------------------------------- >>> -------- >>> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 20 >>> 10 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: ---------------------------------------------------------------- >>> -------- >>> [0]PETSC ERROR: test.exe on a cygwin >>> >>> -cx named MT0. by Unknown Mon Jun 27 16:18:43 2011 >>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-de >>> bug/lib >>> [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 >>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor >>> t" --with-cxx="win32fe cl" --with-windows-graphics=0 --download-f-blas-lapack=1 >>> --with-precision=double --with-scalar-type=complex --with-clanguage=cxx --with-m >>> pi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include --with-mpi-lib="[/cygdrive/d/Dev >>> /HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpife >>> c.lib]" --useThreads=0 --useThreads=0 >>> [0]PETSC ERROR: ---------------------------------------------------------------- >>> -------- >>> [0]PETSC ERROR: MatNorm() line 4507 in src/mat/interface/D:\dev\PETSC-3.1-P\src\mat\INTERF~1\matrix.c >>> [0]PETSC ERROR: KSPSolve_LSQR() line 90 in src/ksp/ksp/impls/lsqr/D:\dev\PETSC-3.1-P\src\ksp\ksp\impls\lsqr\lsqr.c >>> >>> Any suggestions? Do I have to set MATOP_NORM operation? Then how? >>> >>> Regards, >>> Alexander >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > From agrayver at gfz-potsdam.de Tue Jun 28 04:59:26 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Tue, 28 Jun 2011 11:59:26 +0200 Subject: [petsc-users] matrix-free KSPLSQR and matrix norm In-Reply-To: <23AACEF1-BC05-428D-A8FA-00D6087BB4E7@mcs.anl.gov> References: <4E08926E.10303@gfz-potsdam.de> <02A939F1-6F10-4F41-A7F2-FD83AD767DBC@mcs.anl.gov> <4E0894EE.3030900@gfz-potsdam.de> <4E08999E.5060901@gfz-potsdam.de> <4E08C18E.7030406@gfz-potsdam.de> <23AACEF1-BC05-428D-A8FA-00D6087BB4E7@mcs.anl.gov> Message-ID: <4E09A5FE.6060605@gfz-potsdam.de> Thanks Barry & Matthew, it works now. Regards, Alexander On 27.06.2011 23:16, Barry Smith wrote: > This code is kind of silly because it doesn't actually ever use that Matrix norm by default so in petsc-3.1 just edit src/ksp/ksp/impls/lsqr/lsqr.c and comment out the line where it calls MatNorm() > then run make in that one directory > > Good luck > > Barry > > On Jun 27, 2011, at 12:44 PM, Alexander Grayver wrote: > >> Actually, I tried, but haven't managed to do this in fortran: >> >> test.exe -log_summary -ksp_view >> [0]PETSC ERROR: MatShellSetOperation_Fortran() line 107 in src/mat/impls/shell/ftn-custom/D:\dev\PETSC-~3.1-P\src\mat\impls\shell\FTN-CU~1\zshellf.c >> >> job aborted: >> [ranks] message >> >> [0] application aborted >> aborting MPI_COMM_WORLD, error 1, comm rank 0 >> >> It seems that MATOP_NORM is not supported from Fortran. Why is that so? >> >> Regards, >> Alexander >> >> On 27.06.2011 17:12, Matthew Knepley wrote: >>> On Mon, Jun 27, 2011 at 9:54 AM, Alexander Grayver wrote: >>>>> Please send the entire error message (this is missing the stack), but I am guessing you >>>>> are not changed the default preconditioner (ILU) which cannot act on a shell matrix. Use >>>>> -ksp_view to see what solver is begin used. >>> The full error message: >>> >>> test.exe -log_summary -ksp_view >>> >>> The LSQR solver uses the Frobenius norm of the matrix. In order to use a MATSHELL, you >>> must implement MatNorm using >>> >>> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Mat/MatShellSetOperation.html >>> >>> Thanks, >>> >>> Matt >>> >>> >>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [0]PETSC ERROR: No support for this operation for this object type! >>> [0]PETSC ERROR: Mat type shell! >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 2010 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: test.exe on a cygwin-cx named M0. by Unknown Mon Jun 27 16:51:32 2011 >>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib >>> >>> [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 >>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifort" --with-cxx="win32fe cl" --with-windows-graphics=0 >>> --with-precision=double --with-scalar-type=complex --with-clanguage=cxx --with-mpi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include --wi >>> /HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib]" --useThreads=0 --useThreads=0 >>> >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: MatNorm() line 4512 in src/mat/interface/D:\dev\PETSC-~3.1-P\src\mat\INTERF~1\matrix.c >>> [0]PETSC ERROR: KSPSolve_LSQR() line 90 in src/ksp/ksp/impls/lsqr/D:\dev\PETSC-~3.1-P\src\ksp\ksp\impls\lsqr\lsqr.c >>> [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/D:\dev\PETSC-~3.1-P\src\ksp\ksp\INTERF~1\itfunc.c >>> >>> job aborted: >>> [ranks] message >>> >>> [0] application aborted >>> aborting MPI_COMM_WORLD, error 56, comm rank 0 >>> >>> >>> On 27.06.2011 16:47, Matthew Knepley wrote: >>>> On Mon, Jun 27, 2011 at 9:34 AM, Alexander Grayver wrote: >>>> Thanks Barry, >>>> >>>> Now I get this: >>>> >>>> Please send the entire error message (this is missing the stack), but I am guessing you >>>> are not changed the default preconditioner (ILU) which cannot act on a shell matrix. Use >>>> -ksp_view to see what solver is begin used. >>>> >>>> Matt >>>> >>>> [0]PETSC ERROR: --------------------- Error Message ---------------------------- >>>> -------- >>>> [0]PETSC ERROR: No support for this operation for this object type! >>>> [0]PETSC ERROR: Mat type shell! >>>> >>>> [0]PETSC ERROR: ---------------------------------------------------------------- >>>> -------- >>>> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 20 >>>> 10 >>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> [0]PETSC ERROR: ---------------------------------------------------------------- >>>> -------- >>>> [0]PETSC ERROR: test.exe on a cygwin >>>> -cx named MT0. by Unknown Mon Jun 27 16:30:42 2011 >>>> >>>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-de >>>> bug/lib >>>> [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 >>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor >>>> t" --with-cxx="win32fe cl" --with-windows-graphics=0 --download-f-blas-lapack=1 >>>> --with-precision=double --with-scalar-type=complex --with-clanguage=cxx --with-m >>>> pi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include --with-mpi-lib="[/cygdrive/d/Dev >>>> /HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpife >>>> c.lib]" --useThreads=0 --useThreads=0 >>>> [0]PETSC ERROR: ---------------------------------------------------------------- >>>> -------- >>>> [0]PETSC ERROR: MatNorm() line 4512 in src/mat/interface/D:\dev\PETSC-3.1-P\src\mat\INTERF~1\matrix.c >>>> >>>> >>>> On 27.06.2011 16:32, Barry Smith wrote: >>>> You should pass H as third argument to KSPSetOperators() as well as the second. >>>> >>>> Barry >>>> >>>> On Jun 27, 2011, at 9:23 AM, Alexander Grayver wrote: >>>> >>>> Hello! >>>> >>>> I'm using matrix-free formulation and KSPLSQR to solve normal equations system: >>>> >>>> (A'*A)b = -g >>>> >>>> I managed to write all necessary code: >>>> >>>> call MatCreateShell(MPI_COMM_WORLD,m,N,N,N,PETSC_NULL_INTEGER,H,ierr) >>>> call MatShellSetOperation(H,MATOP_MULT,MFMatMult,ierr) >>>> >>>> call KSPCreate(MPI_COMM_WORLD,ksp,ierr);CHKERRQ(ierr) >>>> call KSPSetType(ksp,KSPLSQR,ierr);CHKERRQ(ierr) >>>> call KSPGetPC(ksp,pc,ierr);CHKERRQ(ierr) >>>> call PCSetType(pc,PCNONE,ierr);CHKERRQ(ierr) >>>> call KSPSetOperators(ksp,H,PETSC_NULL_OBJECT,SAME_PRECONDITIONER,ierr);CHKERRQ(ierr) >>>> call KSPSolve(ksp,g,b,ierr);CHKERRQ(ierr) >>>> >>>> >>>> but fail in KSPSolve stage with that: >>>> [0]PETSC ERROR: --------------------- Error Message ---------------------------- >>>> -------- >>>> [0]PETSC ERROR: Object is in wrong state! >>>> [0]PETSC ERROR: Mat object's type is not set: Argument # 1! >>>> [0]PETSC ERROR: ---------------------------------------------------------------- >>>> -------- >>>> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 20 >>>> 10 >>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> [0]PETSC ERROR: ---------------------------------------------------------------- >>>> -------- >>>> [0]PETSC ERROR: test.exe on a cygwin >>>> >>>> -cx named MT0. by Unknown Mon Jun 27 16:18:43 2011 >>>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-de >>>> bug/lib >>>> [0]PETSC ERROR: Configure run at Fri Mar 4 10:10:06 2011 >>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor >>>> t" --with-cxx="win32fe cl" --with-windows-graphics=0 --download-f-blas-lapack=1 >>>> --with-precision=double --with-scalar-type=complex --with-clanguage=cxx --with-m >>>> pi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include --with-mpi-lib="[/cygdrive/d/Dev >>>> /HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpife >>>> c.lib]" --useThreads=0 --useThreads=0 >>>> [0]PETSC ERROR: ---------------------------------------------------------------- >>>> -------- >>>> [0]PETSC ERROR: MatNorm() line 4507 in src/mat/interface/D:\dev\PETSC-3.1-P\src\mat\INTERF~1\matrix.c >>>> [0]PETSC ERROR: KSPSolve_LSQR() line 90 in src/ksp/ksp/impls/lsqr/D:\dev\PETSC-3.1-P\src\ksp\ksp\impls\lsqr\lsqr.c >>>> >>>> Any suggestions? Do I have to set MATOP_NORM operation? Then how? >>>> >>>> Regards, >>>> Alexander >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>> -- Norbert Wiener >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener From adam1.byrd at gmail.com Tue Jun 28 08:55:12 2011 From: adam1.byrd at gmail.com (Adam Byrd) Date: Tue, 28 Jun 2011 09:55:12 -0400 Subject: [petsc-users] Matrix Construction Question In-Reply-To: References: Message-ID: Hi, I'm rather new to PETSc and trying to work out the best way to create and fill a large sparse matrix distributed over many processors. Currently, my goal is to create a 3600x3600 matrix in units of 12x12 blocks with several blocks on any given node. I'd like to create the matrix in such a way that each node only holds the information in it's handful of blocks and not the entire matrix. Eventually, this matrix is to be inverted (I know, inversion should be avoided, but as this is a Hamiltonian matrix from which I need the Green's function, I'm unaware of a way to forgo carrying out the inversion). Additionally, the values will be changed slightly and the matrix will be repeatedly inverted. It's structure will remain the same. In order to learn how to do this is I am starting with a small 6x6 matrix broken into four 3x3 blocks and distributed one block per node. I've been able to create a local 3x3 matrix on each node, with it's own values, and with the global row/column IDs correctly set to [0, 1, 2] or [3, 4, 5] depending on where the block is in the matrix. My problem manifests when I try to create the larger matrix from the individual smaller ones. When the matrix is constructed I'm trying to use MatSetValues and having each node pass in it's 3x3 block. I end up with an error that the sum of local lengths 12x12 does not match the global length 6x6. It appears as though this is from passing in four 3x3s and the program interpreting that as a 12x12 instead of as a 6x6 with the blocks in a grid. My question is then: is it possible to fill a matrix as a grid of blocks, or can I only fill it in groups of rows or columns? Also, am I approaching this problem the correct way, or are there more efficient ways of building this matrix with the ultimate goal of inverting it? I have included my copy of a modified example if it helps. I do apologize if this is answered somewhere in the documentation, I have been unable to find a solution. Respectfully, Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex1.c Type: text/x-csrc Size: 2279 bytes Desc: not available URL: From aron.ahmadia at kaust.edu.sa Tue Jun 28 09:09:44 2011 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Tue, 28 Jun 2011 17:09:44 +0300 Subject: [petsc-users] Matrix Construction Question In-Reply-To: References: Message-ID: Hi Adam, It sounds like you are creating a 'blocked' matrix. PETSc's format for this is described briefly here: http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatCreateMPIBAIJ.html#MatCreateMPIBAIJ You will also be interested in adding values blocked: http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatSetValuesBlocked.html *Also, am I approaching this problem the correct way, or are there more efficient ways of building this matrix with the ultimate goal of inverting it?* I will let somebody else on the list tell you why this is or isn't a good idea. Generally if you are going to go through the trouble of inverting a matrix, you should use a direct solver and at least factor it first, then construct the inverse from the decomposed matrix (though the situations where you MUST construct the inverse are quite rare). A On Tue, Jun 28, 2011 at 4:55 PM, Adam Byrd wrote: > Hi, > > I'm rather new to PETSc and trying to work out the best way to create and > fill a large sparse matrix distributed over many processors. Currently, my > goal is to create a 3600x3600 matrix in units of 12x12 blocks with several > blocks on any given node. I'd like to create the matrix in such a way that > each node only holds the information in it's handful of blocks and not the > entire matrix. Eventually, this matrix is to be inverted (I know, inversion > should be avoided, but as this is a Hamiltonian matrix from which I need the > Green's function, I'm unaware of a way to forgo carrying out the inversion). > Additionally, the values will be changed slightly and the matrix will be > repeatedly inverted. It's structure will remain the same. In order to learn > how to do this is I am starting with a small 6x6 matrix broken into four 3x3 > blocks and distributed one block per node. I've been able to create a local > 3x3 matrix on each node, with it's own values, and with the global > row/column IDs correctly set to [0, 1, 2] or [3, 4, 5] depending on where > the block is in the matrix. My problem manifests when I try to create the > larger matrix from the individual smaller ones. When the matrix is > constructed I'm trying to use MatSetValues and having each node pass in it's > 3x3 block. I end up with an error that the sum of local lengths 12x12 does > not match the global length 6x6. It appears as though this is from passing > in four 3x3s and the program interpreting that as a 12x12 instead of as a > 6x6 with the blocks in a grid. > > My question is then: is it possible to fill a matrix as a grid of blocks, > or can I only fill it in groups of rows or columns? Also, am I approaching > this problem the correct way, or are there more efficient ways of building > this matrix with the ultimate goal of inverting it? > > I have included my copy of a modified example if it helps. I do apologize > if this is answered somewhere in the documentation, I have been unable to > find a solution. > > Respectfully, > Adam > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jun 28 16:55:09 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 28 Jun 2011 16:55:09 -0500 Subject: [petsc-users] Matrix Construction Question In-Reply-To: References: Message-ID: It sounds like you have a dense matrix (from your example). Is this true? If so, you should use Elemental (on Google Code). Thanks, Matt On Tue, Jun 28, 2011 at 8:55 AM, Adam Byrd wrote: > Hi, > > I'm rather new to PETSc and trying to work out the best way to create and > fill a large sparse matrix distributed over many processors. Currently, my > goal is to create a 3600x3600 matrix in units of 12x12 blocks with several > blocks on any given node. I'd like to create the matrix in such a way that > each node only holds the information in it's handful of blocks and not the > entire matrix. Eventually, this matrix is to be inverted (I know, inversion > should be avoided, but as this is a Hamiltonian matrix from which I need the > Green's function, I'm unaware of a way to forgo carrying out the inversion). > Additionally, the values will be changed slightly and the matrix will be > repeatedly inverted. It's structure will remain the same. In order to learn > how to do this is I am starting with a small 6x6 matrix broken into four 3x3 > blocks and distributed one block per node. I've been able to create a local > 3x3 matrix on each node, with it's own values, and with the global > row/column IDs correctly set to [0, 1, 2] or [3, 4, 5] depending on where > the block is in the matrix. My problem manifests when I try to create the > larger matrix from the individual smaller ones. When the matrix is > constructed I'm trying to use MatSetValues and having each node pass in it's > 3x3 block. I end up with an error that the sum of local lengths 12x12 does > not match the global length 6x6. It appears as though this is from passing > in four 3x3s and the program interpreting that as a 12x12 instead of as a > 6x6 with the blocks in a grid. > > My question is then: is it possible to fill a matrix as a grid of blocks, > or can I only fill it in groups of rows or columns? Also, am I approaching > this problem the correct way, or are there more efficient ways of building > this matrix with the ultimate goal of inverting it? > > I have included my copy of a modified example if it helps. I do apologize > if this is answered somewhere in the documentation, I have been unable to > find a solution. > > Respectfully, > Adam > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam1.byrd at gmail.com Tue Jun 28 17:01:53 2011 From: adam1.byrd at gmail.com (Adam Byrd) Date: Tue, 28 Jun 2011 18:01:53 -0400 Subject: [petsc-users] Matrix Construction Question In-Reply-To: References: Message-ID: Actually, it's quite sparse. In the 3600x3600 there are only just 4 nonzero entries in each row. This means it's 99.9% empty. My smaller 6x6 example is dense, but it's only practice building and manipulating matrices. Respectfully, Adam On Tue, Jun 28, 2011 at 5:55 PM, Matthew Knepley wrote: > It sounds like you have a dense matrix (from your example). Is this true? > If so, you should use Elemental (on Google Code). > > Thanks, > > Matt > > On Tue, Jun 28, 2011 at 8:55 AM, Adam Byrd wrote: > >> Hi, >> >> I'm rather new to PETSc and trying to work out the best way to create and >> fill a large sparse matrix distributed over many processors. Currently, my >> goal is to create a 3600x3600 matrix in units of 12x12 blocks with several >> blocks on any given node. I'd like to create the matrix in such a way that >> each node only holds the information in it's handful of blocks and not the >> entire matrix. Eventually, this matrix is to be inverted (I know, inversion >> should be avoided, but as this is a Hamiltonian matrix from which I need the >> Green's function, I'm unaware of a way to forgo carrying out the inversion). >> Additionally, the values will be changed slightly and the matrix will be >> repeatedly inverted. It's structure will remain the same. In order to learn >> how to do this is I am starting with a small 6x6 matrix broken into four 3x3 >> blocks and distributed one block per node. I've been able to create a local >> 3x3 matrix on each node, with it's own values, and with the global >> row/column IDs correctly set to [0, 1, 2] or [3, 4, 5] depending on where >> the block is in the matrix. My problem manifests when I try to create the >> larger matrix from the individual smaller ones. When the matrix is >> constructed I'm trying to use MatSetValues and having each node pass in it's >> 3x3 block. I end up with an error that the sum of local lengths 12x12 does >> not match the global length 6x6. It appears as though this is from passing >> in four 3x3s and the program interpreting that as a 12x12 instead of as a >> 6x6 with the blocks in a grid. >> >> My question is then: is it possible to fill a matrix as a grid of blocks, >> or can I only fill it in groups of rows or columns? Also, am I approaching >> this problem the correct way, or are there more efficient ways of building >> this matrix with the ultimate goal of inverting it? >> >> I have included my copy of a modified example if it helps. I do apologize >> if this is answered somewhere in the documentation, I have been unable to >> find a solution. >> >> Respectfully, >> Adam >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jack.poulson at gmail.com Tue Jun 28 17:06:05 2011 From: jack.poulson at gmail.com (Jack Poulson) Date: Tue, 28 Jun 2011 17:06:05 -0500 Subject: [petsc-users] Matrix Construction Question In-Reply-To: References: Message-ID: Adam, What operations do you need to perform with the inverses of your Hamiltonian matrices? The inverse of a sparse matrix is still, in general, dense. That is what Matt was getting at. Using a sparse matrix format will be a bad idea. If your operator was close to Hermitian positive-definite then you could look into the H-matrix/semi-separable literature for means of forming the inverse in roughly O(n log^2(n)) arithmetic, but Hamiltonian matrices are not quite as well-behaved and I'm not aware of readily-available techniques for forming approximations to their inverses. With the unstructured dense linear algebra approach, it will cost you O(n^3) operations to form the inverse even if your sparse matrix only contains O(n) entries. If I were you, I would think carefully about how to avoid forming the inverse. Regards, Jack Poulson On Tue, Jun 28, 2011 at 5:01 PM, Adam Byrd wrote: > Actually, it's quite sparse. In the 3600x3600 there are only just 4 nonzero > entries in each row. This means it's 99.9% empty. My smaller 6x6 example is > dense, but it's only practice building and manipulating matrices. > > Respectfully, > Adam > > > On Tue, Jun 28, 2011 at 5:55 PM, Matthew Knepley wrote: > >> It sounds like you have a dense matrix (from your example). Is this true? >> If so, you should use Elemental (on Google Code). >> >> Thanks, >> >> Matt >> >> On Tue, Jun 28, 2011 at 8:55 AM, Adam Byrd wrote: >> >>> Hi, >>> >>> I'm rather new to PETSc and trying to work out the best way to create and >>> fill a large sparse matrix distributed over many processors. Currently, my >>> goal is to create a 3600x3600 matrix in units of 12x12 blocks with several >>> blocks on any given node. I'd like to create the matrix in such a way that >>> each node only holds the information in it's handful of blocks and not the >>> entire matrix. Eventually, this matrix is to be inverted (I know, inversion >>> should be avoided, but as this is a Hamiltonian matrix from which I need the >>> Green's function, I'm unaware of a way to forgo carrying out the inversion). >>> Additionally, the values will be changed slightly and the matrix will be >>> repeatedly inverted. It's structure will remain the same. In order to learn >>> how to do this is I am starting with a small 6x6 matrix broken into four 3x3 >>> blocks and distributed one block per node. I've been able to create a local >>> 3x3 matrix on each node, with it's own values, and with the global >>> row/column IDs correctly set to [0, 1, 2] or [3, 4, 5] depending on where >>> the block is in the matrix. My problem manifests when I try to create the >>> larger matrix from the individual smaller ones. When the matrix is >>> constructed I'm trying to use MatSetValues and having each node pass in it's >>> 3x3 block. I end up with an error that the sum of local lengths 12x12 does >>> not match the global length 6x6. It appears as though this is from passing >>> in four 3x3s and the program interpreting that as a 12x12 instead of as a >>> 6x6 with the blocks in a grid. >>> >>> My question is then: is it possible to fill a matrix as a grid of blocks, >>> or can I only fill it in groups of rows or columns? Also, am I approaching >>> this problem the correct way, or are there more efficient ways of building >>> this matrix with the ultimate goal of inverting it? >>> >>> I have included my copy of a modified example if it helps. I do apologize >>> if this is answered somewhere in the documentation, I have been unable to >>> find a solution. >>> >>> Respectfully, >>> Adam >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jun 28 17:08:48 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 28 Jun 2011 17:08:48 -0500 Subject: [petsc-users] Matrix Construction Question In-Reply-To: References: Message-ID: On Tue, Jun 28, 2011 at 5:01 PM, Adam Byrd wrote: > Actually, it's quite sparse. In the 3600x3600 there are only just 4 nonzero > entries in each row. This means it's 99.9% empty. My smaller 6x6 example is > dense, but it's only practice building and manipulating matrices. > Ah, then its easy. Just call MatSetValues() with each block. Then use MUMPS to do a sparse direct solve. Matt > Respectfully, > Adam > > > On Tue, Jun 28, 2011 at 5:55 PM, Matthew Knepley wrote: > >> It sounds like you have a dense matrix (from your example). Is this true? >> If so, you should use Elemental (on Google Code). >> >> Thanks, >> >> Matt >> >> On Tue, Jun 28, 2011 at 8:55 AM, Adam Byrd wrote: >> >>> Hi, >>> >>> I'm rather new to PETSc and trying to work out the best way to create and >>> fill a large sparse matrix distributed over many processors. Currently, my >>> goal is to create a 3600x3600 matrix in units of 12x12 blocks with several >>> blocks on any given node. I'd like to create the matrix in such a way that >>> each node only holds the information in it's handful of blocks and not the >>> entire matrix. Eventually, this matrix is to be inverted (I know, inversion >>> should be avoided, but as this is a Hamiltonian matrix from which I need the >>> Green's function, I'm unaware of a way to forgo carrying out the inversion). >>> Additionally, the values will be changed slightly and the matrix will be >>> repeatedly inverted. It's structure will remain the same. In order to learn >>> how to do this is I am starting with a small 6x6 matrix broken into four 3x3 >>> blocks and distributed one block per node. I've been able to create a local >>> 3x3 matrix on each node, with it's own values, and with the global >>> row/column IDs correctly set to [0, 1, 2] or [3, 4, 5] depending on where >>> the block is in the matrix. My problem manifests when I try to create the >>> larger matrix from the individual smaller ones. When the matrix is >>> constructed I'm trying to use MatSetValues and having each node pass in it's >>> 3x3 block. I end up with an error that the sum of local lengths 12x12 does >>> not match the global length 6x6. It appears as though this is from passing >>> in four 3x3s and the program interpreting that as a 12x12 instead of as a >>> 6x6 with the blocks in a grid. >>> >>> My question is then: is it possible to fill a matrix as a grid of blocks, >>> or can I only fill it in groups of rows or columns? Also, am I approaching >>> this problem the correct way, or are there more efficient ways of building >>> this matrix with the ultimate goal of inverting it? >>> >>> I have included my copy of a modified example if it helps. I do apologize >>> if this is answered somewhere in the documentation, I have been unable to >>> find a solution. >>> >>> Respectfully, >>> Adam >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam1.byrd at gmail.com Tue Jun 28 17:20:07 2011 From: adam1.byrd at gmail.com (Adam Byrd) Date: Tue, 28 Jun 2011 18:20:07 -0400 Subject: [petsc-users] Matrix Construction Question In-Reply-To: References: Message-ID: Matt, Alright, that means I need to continue learning how to use MatSetValues(). With my 6x6 example I tried filling it with four 3x3 sub matrices, but when I do that I get the error 'sum of local sizes 12 does not equal global size.' I had 4 processors each calling MatSetValues for their own 3x3. Graphically, I arranged the nodes 0 1 2 3 where process 0 had global rows 0-2 and global columns 0-2; process 1 had 0-2, 3-5; process 2 had 3-5, 0-2; and process 3 had 3-5, 3-5. From the documentation, I think this should be correct, but I'm not sure. Also, which format would you recommend for storing the matrix? Jack, I'm a summer intern just getting started with this project, so I don't know all the details yet (I can ask though). I know I need to find the Green's function which will involve the trace of the inverted Hamiltonian, as well as the rest of the matrix. I have inquired about avoiding the inversion altogether, but my instructor doesn't believe there is a way around it. Once I've worked through the math I want to explore other options though. Respectfully, Adam On Tue, Jun 28, 2011 at 6:08 PM, Matthew Knepley wrote: > On Tue, Jun 28, 2011 at 5:01 PM, Adam Byrd wrote: > >> Actually, it's quite sparse. In the 3600x3600 there are only just 4 >> nonzero entries in each row. This means it's 99.9% empty. My smaller 6x6 >> example is dense, but it's only practice building and manipulating matrices. >> > > Ah, then its easy. Just call MatSetValues() with each block. Then use MUMPS > to do a sparse direct solve. > > Matt > > >> Respectfully, >> Adam >> >> >> On Tue, Jun 28, 2011 at 5:55 PM, Matthew Knepley wrote: >> >>> It sounds like you have a dense matrix (from your example). Is this true? >>> If so, you should use Elemental (on Google Code). >>> >>> Thanks, >>> >>> Matt >>> >>> On Tue, Jun 28, 2011 at 8:55 AM, Adam Byrd wrote: >>> >>>> Hi, >>>> >>>> I'm rather new to PETSc and trying to work out the best way to create >>>> and fill a large sparse matrix distributed over many processors. Currently, >>>> my goal is to create a 3600x3600 matrix in units of 12x12 blocks with >>>> several blocks on any given node. I'd like to create the matrix in such a >>>> way that each node only holds the information in it's handful of blocks and >>>> not the entire matrix. Eventually, this matrix is to be inverted (I know, >>>> inversion should be avoided, but as this is a Hamiltonian matrix from which >>>> I need the Green's function, I'm unaware of a way to forgo carrying out the >>>> inversion). Additionally, the values will be changed slightly and the matrix >>>> will be repeatedly inverted. It's structure will remain the same. In order >>>> to learn how to do this is I am starting with a small 6x6 matrix broken into >>>> four 3x3 blocks and distributed one block per node. I've been able to create >>>> a local 3x3 matrix on each node, with it's own values, and with the global >>>> row/column IDs correctly set to [0, 1, 2] or [3, 4, 5] depending on where >>>> the block is in the matrix. My problem manifests when I try to create the >>>> larger matrix from the individual smaller ones. When the matrix is >>>> constructed I'm trying to use MatSetValues and having each node pass in it's >>>> 3x3 block. I end up with an error that the sum of local lengths 12x12 does >>>> not match the global length 6x6. It appears as though this is from passing >>>> in four 3x3s and the program interpreting that as a 12x12 instead of as a >>>> 6x6 with the blocks in a grid. >>>> >>>> My question is then: is it possible to fill a matrix as a grid of >>>> blocks, or can I only fill it in groups of rows or columns? Also, am I >>>> approaching this problem the correct way, or are there more efficient ways >>>> of building this matrix with the ultimate goal of inverting it? >>>> >>>> I have included my copy of a modified example if it helps. I do >>>> apologize if this is answered somewhere in the documentation, I have been >>>> unable to find a solution. >>>> >>>> Respectfully, >>>> Adam >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jack.poulson at gmail.com Tue Jun 28 17:26:42 2011 From: jack.poulson at gmail.com (Jack Poulson) Date: Tue, 28 Jun 2011 17:26:42 -0500 Subject: [petsc-users] Matrix Construction Question In-Reply-To: References: Message-ID: On Tue, Jun 28, 2011 at 5:20 PM, Adam Byrd wrote: > Jack, > > I'm a summer intern just getting started with this project, so I don't know > all the details yet (I can ask though). I know I need to find the Green's > function which will involve the trace of the inverted Hamiltonian, as well > as the rest of the matrix. I have inquired about avoiding the inversion > altogether, but my instructor doesn't believe there is a way around it. Once > I've worked through the math I want to explore other options though. > > Respectfully, > Adam > > Ah, it's not a discretized Hamiltonian, not a Hamiltonian matrix ( http://en.wikipedia.org/wiki/Hamiltonian_matrix). For 2d problems, there is a nice approach for efficiently computing the trace of the inverse here: http://www.ma.utexas.edu/users/lexing/publications/siscpar.pdf. For 3d problems it is more difficult and I'm not sure what the best approach is. Jack -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jun 29 00:26:14 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 29 Jun 2011 05:26:14 +0000 Subject: [petsc-users] Matrix Construction Question In-Reply-To: References: Message-ID: On Tue, Jun 28, 2011 at 10:20 PM, Adam Byrd wrote: > Matt, > > Alright, that means I need to continue learning how to use MatSetValues(). > With my 6x6 example I tried filling it with four 3x3 sub matrices, but when > I do that I get the error 'sum of local sizes 12 does not equal global > size.' I had 4 processors each calling MatSetValues for their own 3x3. > Graphically, I arranged the nodes 0 1 > > 2 3 > where process 0 had global rows 0-2 and global columns 0-2; process 1 had > 0-2, 3-5; process 2 had 3-5, 0-2; and process 3 had 3-5, 3-5. From the > documentation, I think this should be correct, but I'm not sure. Also, which > format would you recommend for storing the matrix? > 1) With any error, send the Entire error message. 2) PETSc matrices are divided by rows, not rows and columns, see the manual section. Rows & columns only makes sense for dense matrices 3) You can still set arbitrary blocks no matter how the matrix is divided 4) The error means you tried to set both local and global dimensions, and they do not add up correctly. Just set the global dimensions Matt > Jack, > > I'm a summer intern just getting started with this project, so I don't know > all the details yet (I can ask though). I know I need to find the Green's > function which will involve the trace of the inverted Hamiltonian, as well > as the rest of the matrix. I have inquired about avoiding the inversion > altogether, but my instructor doesn't believe there is a way around it. Once > I've worked through the math I want to explore other options though. > > Respectfully, > Adam > > On Tue, Jun 28, 2011 at 6:08 PM, Matthew Knepley wrote: > >> On Tue, Jun 28, 2011 at 5:01 PM, Adam Byrd wrote: >> >>> Actually, it's quite sparse. In the 3600x3600 there are only just 4 >>> nonzero entries in each row. This means it's 99.9% empty. My smaller 6x6 >>> example is dense, but it's only practice building and manipulating matrices. >>> >> >> Ah, then its easy. Just call MatSetValues() with each block. Then use >> MUMPS to do a sparse direct solve. >> >> Matt >> >> >>> Respectfully, >>> Adam >>> >>> >>> On Tue, Jun 28, 2011 at 5:55 PM, Matthew Knepley wrote: >>> >>>> It sounds like you have a dense matrix (from your example). Is this >>>> true? If so, you should use Elemental (on Google Code). >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> On Tue, Jun 28, 2011 at 8:55 AM, Adam Byrd wrote: >>>> >>>>> Hi, >>>>> >>>>> I'm rather new to PETSc and trying to work out the best way to create >>>>> and fill a large sparse matrix distributed over many processors. Currently, >>>>> my goal is to create a 3600x3600 matrix in units of 12x12 blocks with >>>>> several blocks on any given node. I'd like to create the matrix in such a >>>>> way that each node only holds the information in it's handful of blocks and >>>>> not the entire matrix. Eventually, this matrix is to be inverted (I know, >>>>> inversion should be avoided, but as this is a Hamiltonian matrix from which >>>>> I need the Green's function, I'm unaware of a way to forgo carrying out the >>>>> inversion). Additionally, the values will be changed slightly and the matrix >>>>> will be repeatedly inverted. It's structure will remain the same. In order >>>>> to learn how to do this is I am starting with a small 6x6 matrix broken into >>>>> four 3x3 blocks and distributed one block per node. I've been able to create >>>>> a local 3x3 matrix on each node, with it's own values, and with the global >>>>> row/column IDs correctly set to [0, 1, 2] or [3, 4, 5] depending on where >>>>> the block is in the matrix. My problem manifests when I try to create the >>>>> larger matrix from the individual smaller ones. When the matrix is >>>>> constructed I'm trying to use MatSetValues and having each node pass in it's >>>>> 3x3 block. I end up with an error that the sum of local lengths 12x12 does >>>>> not match the global length 6x6. It appears as though this is from passing >>>>> in four 3x3s and the program interpreting that as a 12x12 instead of as a >>>>> 6x6 with the blocks in a grid. >>>>> >>>>> My question is then: is it possible to fill a matrix as a grid of >>>>> blocks, or can I only fill it in groups of rows or columns? Also, am I >>>>> approaching this problem the correct way, or are there more efficient ways >>>>> of building this matrix with the ultimate goal of inverting it? >>>>> >>>>> I have included my copy of a modified example if it helps. I do >>>>> apologize if this is answered somewhere in the documentation, I have been >>>>> unable to find a solution. >>>>> >>>>> Respectfully, >>>>> Adam >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From goldmann at itp.uni-bremen.de Wed Jun 29 09:37:32 2011 From: goldmann at itp.uni-bremen.de (Elias Goldmann) Date: Wed, 29 Jun 2011 16:37:32 +0200 Subject: [petsc-users] Read from file failes Message-ID: <4E0B38AC.6090503@itp.uni-bremen.de> Hello, i just did my first steps with SLEPc working my way through the given examples. Apperently i encounter the error message below when i try to read from binary files. If i do understand this correctly, this could be related to the combination of 64 bit systems and complex binary files. Since i want to solve complex eigenproblems from input files im confused that this causes problems. Any help available? Thanks, Elias ./ex7 -f1 /home/egoldmann/elias/bfw62a.petsc -f2 /home/egoldmann/elias/bfw62b.petsc Generalized eigenproblem stored in file. Reading COMPLEX matrices from binary files... [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Read from file failed! [0]PETSC ERROR: Read past end of file! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ex7 on a linux-gnu named node5 by egoldmann Wed Jun 29 15:54:06 2011 [0]PETSC ERROR: Libraries linked from /home/egoldmann/petsc-3.1-p8/linux-gnu-c-debug/lib [0]PETSC ERROR: Configure run at Wed Jun 29 10:19:42 2011 [0]PETSC ERROR: Configure options --with-scalar-type=complex --download-blopex [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PetscBinaryRead() line 249 in src/sys/fileio/sysio.c [0]PETSC ERROR: MatLoad_SeqAIJ() line 3572 in src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: MatLoad() line 124 in src/mat/utils/matio.c [0]PETSC ERROR: main() line 64 in ex7.c [unset]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 66) - process 0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jun 29 09:38:21 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 29 Jun 2011 14:38:21 +0000 Subject: [petsc-users] Read from file failes In-Reply-To: <4E0B38AC.6090503@itp.uni-bremen.de> References: <4E0B38AC.6090503@itp.uni-bremen.de> Message-ID: On Wed, Jun 29, 2011 at 2:37 PM, Elias Goldmann wrote: > ** > Hello, > > i just did my first steps with SLEPc working my way through the given > examples. > Apperently i encounter the error message below when i try to read from > binary files. > If i do understand thiscorrectly, this could be related to the combination of 64 bit systems and > complex binary files. Since i want to solve complex eigenproblems from input > files im confused that this causes problems. > It appears the file is not valid. If you used Matlab to make the binary file, this will not work because PETSc can only read real numbers from Matlab. However, valid complex binary files are fine (say from SLEPc or PETSc itself). I would check the file I am trying to load. Matt > Any help available? > Thanks, > Elias > > > ./ex7 -f1 /home/egoldmann/elias/bfw62a.petsc -f2 > /home/egoldmann/elias/bfw62b.petsc > > Generalized eigenproblem stored in file. > > Reading COMPLEX matrices from binary files... > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Read from file failed! > [0]PETSC ERROR: Read past end of file! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 > CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex7 on a linux-gnu named node5 by egoldmann Wed Jun 29 > 15:54:06 2011 > [0]PETSC ERROR: Libraries linked from > /home/egoldmann/petsc-3.1-p8/linux-gnu-c-debug/lib > [0]PETSC ERROR: Configure run at Wed Jun 29 10:19:42 2011 > [0]PETSC ERROR: Configure options --with-scalar-type=complex > --download-blopex > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: PetscBinaryRead() line 249 in src/sys/fileio/sysio.c > [0]PETSC ERROR: MatLoad_SeqAIJ() line 3572 in src/mat/impls/aij/seq/aij.c > [0]PETSC ERROR: MatLoad() line 124 in src/mat/utils/matio.c > [0]PETSC ERROR: main() line 64 in ex7.c > [unset]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 66) - process 0 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Wed Jun 29 09:41:57 2011 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 29 Jun 2011 16:41:57 +0200 Subject: [petsc-users] Read from file failes In-Reply-To: <4E0B38AC.6090503@itp.uni-bremen.de> References: <4E0B38AC.6090503@itp.uni-bremen.de> Message-ID: If bfw62a.petsc and bfw62a.petsc are from the SLEPc distribution, then they contain real scalars, so you cannot read them from a program that goes --with-scalar-type=complex You should read a file generated by you, or test examples that do not read files. Jose El 29/06/2011, a las 16:37, Elias Goldmann escribi?: > Hello, > > i just did my first steps with SLEPc working my way through the given examples. > Apperently i encounter the error message below when i try to read from binary files. > If i do understand this correctly, this could be related to the combination of 64 bit systems and complex binary files. Since i want to solve complex eigenproblems from input files im confused that this causes problems. > > Any help available? > Thanks, > Elias > > > ./ex7 -f1 /home/egoldmann/elias/bfw62a.petsc -f2 /home/egoldmann/elias/bfw62b.petsc > > Generalized eigenproblem stored in file. > > Reading COMPLEX matrices from binary files... > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Read from file failed! > [0]PETSC ERROR: Read past end of file! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex7 on a linux-gnu named node5 by egoldmann Wed Jun 29 15:54:06 2011 > [0]PETSC ERROR: Libraries linked from /home/egoldmann/petsc-3.1-p8/linux-gnu-c-debug/lib > [0]PETSC ERROR: Configure run at Wed Jun 29 10:19:42 2011 > [0]PETSC ERROR: Configure options --with-scalar-type=complex --download-blopex > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: PetscBinaryRead() line 249 in src/sys/fileio/sysio.c > [0]PETSC ERROR: MatLoad_SeqAIJ() line 3572 in src/mat/impls/aij/seq/aij.c > [0]PETSC ERROR: MatLoad() line 124 in src/mat/utils/matio.c > [0]PETSC ERROR: main() line 64 in ex7.c > [unset]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 66) - process 0 > From goldmann at itp.uni-bremen.de Wed Jun 29 09:48:38 2011 From: goldmann at itp.uni-bremen.de (Elias Goldmann) Date: Wed, 29 Jun 2011 16:48:38 +0200 Subject: [petsc-users] Read from file failes In-Reply-To: References: <4E0B38AC.6090503@itp.uni-bremen.de> Message-ID: <4E0B3B46.1050000@itp.uni-bremen.de> Actually the files came with the installation files. I tried ex7 and ex14, both giving the same error message while trying to read the associated files. On 29.06.2011 16:38, Matthew Knepley wrote: > On Wed, Jun 29, 2011 at 2:37 PM, Elias Goldmann > wrote: > >> ** >> Hello, >> >> i just did my first steps with SLEPc working my way through the given >> examples. >> Apperently i encounter the error message below when i try to read from >> binary files. >> If i do understand thiscorrectly, this could be related to the combination of 64 bit systems and >> complex binary files. Since i want to solve complex eigenproblems from input >> files im confused that this causes problems. >> > It appears the file is not valid. If you used Matlab to make the binary > file, this will not work > because PETSc can only read real numbers from Matlab. However, valid complex > binary > files are fine (say from SLEPc or PETSc itself). > > I would check the file I am trying to load. > > Matt > > >> Any help available? >> Thanks, >> Elias >> >> >> ./ex7 -f1 /home/egoldmann/elias/bfw62a.petsc -f2 >> /home/egoldmann/elias/bfw62b.petsc >> >> Generalized eigenproblem stored in file. >> >> Reading COMPLEX matrices from binary files... >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Read from file failed! >> [0]PETSC ERROR: Read past end of file! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 >> CDT 2011 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./ex7 on a linux-gnu named node5 by egoldmann Wed Jun 29 >> 15:54:06 2011 >> [0]PETSC ERROR: Libraries linked from >> /home/egoldmann/petsc-3.1-p8/linux-gnu-c-debug/lib >> [0]PETSC ERROR: Configure run at Wed Jun 29 10:19:42 2011 >> [0]PETSC ERROR: Configure options --with-scalar-type=complex >> --download-blopex >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: PetscBinaryRead() line 249 in src/sys/fileio/sysio.c >> [0]PETSC ERROR: MatLoad_SeqAIJ() line 3572 in src/mat/impls/aij/seq/aij.c >> [0]PETSC ERROR: MatLoad() line 124 in src/mat/utils/matio.c >> [0]PETSC ERROR: main() line 64 in ex7.c >> [unset]: aborting job: >> application called MPI_Abort(MPI_COMM_WORLD, 66) - process 0 >> >> > From goldmann at itp.uni-bremen.de Wed Jun 29 09:49:58 2011 From: goldmann at itp.uni-bremen.de (Elias Goldmann) Date: Wed, 29 Jun 2011 16:49:58 +0200 Subject: [petsc-users] Read from file failes In-Reply-To: References: <4E0B38AC.6090503@itp.uni-bremen.de> Message-ID: <4E0B3B96.4000801@itp.uni-bremen.de> Yes they are, so that makes sense. The examples without file input are working fine. Thanks to both of you for the quick help! E On 29.06.2011 16:41, Jose E. Roman wrote: > If bfw62a.petsc and bfw62a.petsc are from the SLEPc distribution, then they contain real scalars, so you cannot read them from a program that goes --with-scalar-type=complex > You should read a file generated by you, or test examples that do not read files. > Jose > > > > El 29/06/2011, a las 16:37, Elias Goldmann escribi?: > >> Hello, >> >> i just did my first steps with SLEPc working my way through the given examples. >> Apperently i encounter the error message below when i try to read from binary files. >> If i do understand this correctly, this could be related to the combination of 64 bit systems and complex binary files. Since i want to solve complex eigenproblems from input files im confused that this causes problems. >> >> Any help available? >> Thanks, >> Elias >> >> >> ./ex7 -f1 /home/egoldmann/elias/bfw62a.petsc -f2 /home/egoldmann/elias/bfw62b.petsc >> >> Generalized eigenproblem stored in file. >> >> Reading COMPLEX matrices from binary files... >> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [0]PETSC ERROR: Read from file failed! >> [0]PETSC ERROR: Read past end of file! >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./ex7 on a linux-gnu named node5 by egoldmann Wed Jun 29 15:54:06 2011 >> [0]PETSC ERROR: Libraries linked from /home/egoldmann/petsc-3.1-p8/linux-gnu-c-debug/lib >> [0]PETSC ERROR: Configure run at Wed Jun 29 10:19:42 2011 >> [0]PETSC ERROR: Configure options --with-scalar-type=complex --download-blopex >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: PetscBinaryRead() line 249 in src/sys/fileio/sysio.c >> [0]PETSC ERROR: MatLoad_SeqAIJ() line 3572 in src/mat/impls/aij/seq/aij.c >> [0]PETSC ERROR: MatLoad() line 124 in src/mat/utils/matio.c >> [0]PETSC ERROR: main() line 64 in ex7.c >> [unset]: aborting job: >> application called MPI_Abort(MPI_COMM_WORLD, 66) - process 0 >> > From john.fettig at gmail.com Wed Jun 29 10:50:58 2011 From: john.fettig at gmail.com (John Fettig) Date: Wed, 29 Jun 2011 11:50:58 -0400 Subject: [petsc-users] Reusing ML preconditioner Message-ID: I'm trying to reuse a ML preconditioner but it's not working correctly. The A matrix is changing (even being destroyed), so here's what I do: 1) The first time I'm setting up the KSP, I duplicate the A matrix with MatDuplicate and pass it to the KSP: MatDuplicate(Amat,MAT_COPY_VALUES,&Pmat); KSPSetOperators( ksp, Amat, Pmat, SAME_PRECONDITIONER ); 2) I call KSPSolve, then Amat gets destroyed. The KSP persists. 3) The next time through, I follow the instructions on the manpage for KSPSetOperators and do the following, note that Amat is a totally different matrix: Mat tmp_A, tmp_P; MatStructure tmp_flag; KSPGetOperators(ksp,&tmp_A,&tmp_P,&tmp_flag); PetscObjectReference((PetscObject)(tmp_P)); KSPSetOperators( ksp, new_Amat, tmp_P, SAME_PRECONDITIONER ); However it doesn't work. The iteration count is too high (I verified through a slower method that the preconditioner is still good for new_Amat). What am I doing wrong? Thanks, John From jbacks at mcmillan-mcgee.com Wed Jun 29 18:23:35 2011 From: jbacks at mcmillan-mcgee.com (Jonathan Backs) Date: Wed, 29 Jun 2011 17:23:35 -0600 Subject: [petsc-users] Coupled multi-component problems with TS Message-ID: <7EF40573-CC56-4C8E-92C6-247EF9BF198D@mcmillan-mcgee.com> Hi, I am developing my first application with PETSc in an attempt to solve two coupled non-linear PDEs: (1) Poisson's equation for electric potential with a temperature-dependent electrical conductivity, and (2) the heat diffusion equation with a voltage-dependent thermal conductivity and a voltage-and-temperature-dependent source term (ohmic heating from the electric potential and the temperature-dependent electrical conductivity). So far I have solved the two problems separately without problems: Poisson's equation with a voltage-dependent electric conductivity (no timestepping), and the heat equation with a temperature-dependent thermal conductivity (timestepping, but no source term; the heat came from the boundary conditions in this case). However, I am not sure how to set up PETSc for solving these non-linear equations simultaneously. For example, when setting up a non-linear problem with time stepping, I understand I have to set the RHSFunction and RHSJacobian, assuming the left hand side provided by PETSc is the time-derivative of the independent variable; When setting up a non-linear problem without time stepping, I understand I have to set the Function assuming a zero left hand side (and then set the Jacobian according to the Function). I have tried to implement the coupled problem using a three-dimensional DA with two degrees of freedom (V and T). While PETSc seems to facilitate the use of multiple degrees of freedom in the same RHSFunction (and RHSJacobian), I cannot assign an RHSFunction for the Poisson's equation since the Poisson's equation does not include a time derivative. I have not been able to find any examples in the documentation or on this mailing list that illustrated this particular type of problem. Could anyone offer a hint as to the proper strategy for implementing this coupled system of PDEs? Thank you for your time, Jon From bsmith at mcs.anl.gov Wed Jun 29 21:08:40 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 29 Jun 2011 21:08:40 -0500 Subject: [petsc-users] Coupled multi-component problems with TS In-Reply-To: <7EF40573-CC56-4C8E-92C6-247EF9BF198D@mcmillan-mcgee.com> References: <7EF40573-CC56-4C8E-92C6-247EF9BF198D@mcmillan-mcgee.com> Message-ID: Jonathan, You need to switch to PETSc-dev http://www.mcs.anl.gov/petsc/petsc-as/developers/index.html to access the DAE solvers that Jed Brown has developed. For that you use TSSetIFunction() and TSSetIJacobian() . Examples in src/ts/examples/tutorials ex14.c: ierr = TSSetIFunction(ts,THIFunction,thi);CHKERRQ(ierr); ex15.c: ierr = TSSetIFunction(ts,FormIFunction,&user);CHKERRQ(ierr); ex17.c: ierr = TSSetIFunction(ts,FormIFunction,&user);CHKERRQ(ierr); ex18.c: ierr = TSSetIFunction(ts,FormResidual,&user);CHKERRQ(ierr); ex8.c: ierr = TSSetIFunction(ts,problem->function,problem->data);CHKERRQ(ierr); all use this functionality. Good luck and feel free to send more questions once you are going, but I suspect it will be easy for you with this hint. Barry On Jun 29, 2011, at 6:23 PM, Jonathan Backs wrote: > Hi, > > I am developing my first application with PETSc in an attempt to solve two coupled non-linear PDEs: (1) Poisson's equation for electric potential with a temperature-dependent electrical conductivity, and (2) the heat diffusion equation with a voltage-dependent thermal conductivity and a voltage-and-temperature-dependent source term (ohmic heating from the electric potential and the temperature-dependent electrical conductivity). > > So far I have solved the two problems separately without problems: Poisson's equation with a voltage-dependent electric conductivity (no timestepping), and the heat equation with a temperature-dependent thermal conductivity (timestepping, but no source term; the heat came from the boundary conditions in this case). However, I am not sure how to set up PETSc for solving these non-linear equations simultaneously. For example, when setting up a non-linear problem with time stepping, I understand I have to set the RHSFunction and RHSJacobian, assuming the left hand side provided by PETSc is the time-derivative of the independent variable; When setting up a non-linear problem without time stepping, I understand I have to set the Function assuming a zero left hand side (and then set the Jacobian according to the Function). > > I have tried to implement the coupled problem using a three-dimensional DA with two degrees of freedom (V and T). While PETSc seems to facilitate the use of multiple degrees of freedom in the same RHSFunction (and RHSJacobian), I cannot assign an RHSFunction for the Poisson's equation since the Poisson's equation does not include a time derivative. I have not been able to find any examples in the documentation or on this mailing list that illustrated this particular type of problem. > > Could anyone offer a hint as to the proper strategy for implementing this coupled system of PDEs? > > Thank you for your time, > > Jon From bsmith at mcs.anl.gov Wed Jun 29 21:38:34 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 29 Jun 2011 21:38:34 -0500 Subject: [petsc-users] Reusing ML preconditioner In-Reply-To: References: Message-ID: <19008F56-E40F-4BDC-800E-A18CEF286099@mcs.anl.gov> On Jun 29, 2011, at 10:50 AM, John Fettig wrote: > I'm trying to reuse a ML preconditioner but it's not working > correctly. The A matrix is changing (even being destroyed), so here's > what I do: > > 1) The first time I'm setting up the KSP, I duplicate the A matrix > with MatDuplicate and pass it to the KSP: > MatDuplicate(Amat,MAT_COPY_VALUES,&Pmat); > KSPSetOperators( ksp, Amat, Pmat, SAME_PRECONDITIONER ); > > 2) I call KSPSolve, then Amat gets destroyed. The KSP persists. > > 3) The next time through, I follow the instructions on the manpage for > KSPSetOperators and do the following, note that Amat is a totally > different matrix: > > Mat tmp_A, tmp_P; > MatStructure tmp_flag; > > KSPGetOperators(ksp,&tmp_A,&tmp_P,&tmp_flag); > PetscObjectReference((PetscObject)(tmp_P)); > KSPSetOperators( ksp, new_Amat, tmp_P, SAME_PRECONDITIONER ); > > However it doesn't work. The iteration count is too high (I verified > through a slower method that the preconditioner is still good for > new_Amat). What am I doing wrong? What do you mean a slower method? How can you verify that it is still a good preconditioner without actually running the old preconditioner on the new matrix. BTW: You don't need that complicated stuff above. Just do KSPSetOperators( ksp, Amat, Amat, SAME_PRECONDITIONER ); KSPSolve(). change Amat, or destroy it and make a new one KSPSetOperators( ksp, Amat, Amat, SAME_PRECONDITIONER ); KSPSolve(). It will solve the new linear system using the old preconditioner built during the first KSPSolve(). There is no reason to pass in the old version of Amat you copied to Pmat I suspect that the linear system changes more than you think and that the old preconditioner simply isn't good enough for the new system. Barry > > Thanks, > John From john.fettig at gmail.com Wed Jun 29 21:54:41 2011 From: john.fettig at gmail.com (John Fettig) Date: Wed, 29 Jun 2011 22:54:41 -0400 Subject: [petsc-users] Reusing ML preconditioner In-Reply-To: <19008F56-E40F-4BDC-800E-A18CEF286099@mcs.anl.gov> References: <19008F56-E40F-4BDC-800E-A18CEF286099@mcs.anl.gov> Message-ID: On Wed, Jun 29, 2011 at 10:38 PM, Barry Smith wrote: > ? What do you mean a slower method? How can you verify that it is still a good preconditioner without actually running the old preconditioner on the new matrix. I have two equivalent ways of building the matrix: 1) MatSetValues, (standard route) 2) MatCreate*WithArrays. Both routines produce identical matrices and identical solutions. However, if I run with version 1), the preconditioner is pretty good for many solves. If I run with version 2), the preconditioner only works well for a couple of solves. > ? ?BTW: You don't need that complicated stuff above. Just do > > ? ?KSPSetOperators( ksp, Amat, Amat, SAME_PRECONDITIONER ); > ? ?KSPSolve(). > ? ?change Amat, or destroy it and make a new one > ? ?KSPSetOperators( ksp, Amat, Amat, SAME_PRECONDITIONER ); > ? ?KSPSolve(). > > ? ?It will solve the new linear system using the old preconditioner built during the first KSPSolve(). There is no reason to pass in the old version of Amat you copied to Pmat I tried this and it works ok for a couple of solves (a couple of nonlinear iterations), but on the next timestep it takes a lot more iterations than what I referenced above as version 1). The total KSPSolve time in version 1) strongly indicates to me that it is indeed using the old preconditioner. Thanks, John From bsmith at mcs.anl.gov Wed Jun 29 22:01:56 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 29 Jun 2011 22:01:56 -0500 Subject: [petsc-users] Reusing ML preconditioner In-Reply-To: References: <19008F56-E40F-4BDC-800E-A18CEF286099@mcs.anl.gov> Message-ID: <36F75DB1-D7C3-4407-8C24-1892138CB5A0@mcs.anl.gov> John, The MatCreateSeqAIJWithArrays() uses the memory (arrays) that you provide so you cannot free those arrays or muck with them until you are finished with the solve. What happens if you ALWAYS copy with MatDuplicate() the MatCreate*WithArrays() matrix immediately after you create it and use that matrix as I describe below in the setoperators? Does it start behaving like the 1) MatSetValues() or does it still behave "badly"? Barry On Jun 29, 2011, at 9:54 PM, John Fettig wrote: > On Wed, Jun 29, 2011 at 10:38 PM, Barry Smith wrote: >> What do you mean a slower method? How can you verify that it is still a good preconditioner without actually running the old preconditioner on the new matrix. > > I have two equivalent ways of building the matrix: 1) MatSetValues, > (standard route) 2) MatCreate*WithArrays. Both routines produce > identical matrices and identical solutions. However, if I run with > version 1), the preconditioner is pretty good for many solves. If I > run with version 2), the preconditioner only works well for a couple > of solves. > >> BTW: You don't need that complicated stuff above. Just do >> >> KSPSetOperators( ksp, Amat, Amat, SAME_PRECONDITIONER ); >> KSPSolve(). >> change Amat, or destroy it and make a new one >> KSPSetOperators( ksp, Amat, Amat, SAME_PRECONDITIONER ); >> KSPSolve(). >> >> It will solve the new linear system using the old preconditioner built during the first KSPSolve(). There is no reason to pass in the old version of Amat you copied to Pmat > > I tried this and it works ok for a couple of solves (a couple of > nonlinear iterations), but on the next timestep it takes a lot more > iterations than what I referenced above as version 1). The total > KSPSolve time in version 1) strongly indicates to me that it is indeed > using the old preconditioner. > > Thanks, > John From john.fettig at gmail.com Wed Jun 29 22:12:45 2011 From: john.fettig at gmail.com (John Fettig) Date: Wed, 29 Jun 2011 23:12:45 -0400 Subject: [petsc-users] Reusing ML preconditioner In-Reply-To: <36F75DB1-D7C3-4407-8C24-1892138CB5A0@mcs.anl.gov> References: <19008F56-E40F-4BDC-800E-A18CEF286099@mcs.anl.gov> <36F75DB1-D7C3-4407-8C24-1892138CB5A0@mcs.anl.gov> Message-ID: On Wed, Jun 29, 2011 at 11:01 PM, Barry Smith wrote: > > ?John, > > ? The MatCreateSeqAIJWithArrays() uses the memory (arrays) that you provide so you cannot free those arrays or muck with them until you are finished with the solve. I figured that was the case. I wasn't sure how far down the hierarchy the memory I provided was referenced, so I figured if I passed in a Pmat that used petsc allocated memory I could get around this. > ? What happens if you ALWAYS copy with MatDuplicate() the MatCreate*WithArrays() matrix immediately after you create it and use that matrix as I describe below in the setoperators? Does it start behaving like > the 1) MatSetValues() ?or does it still behave "badly"? It takes roughly the same number of CG iterations as 1), but the solve time indicates to me that PCSetUp is being called every solve. The solutions are also different, which also indicates that the preconditioner is being rebuilt every solve. Thanks, John From bsmith at mcs.anl.gov Wed Jun 29 22:21:37 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 29 Jun 2011 22:21:37 -0500 Subject: [petsc-users] Reusing ML preconditioner In-Reply-To: References: <19008F56-E40F-4BDC-800E-A18CEF286099@mcs.anl.gov> <36F75DB1-D7C3-4407-8C24-1892138CB5A0@mcs.anl.gov> Message-ID: <78447D8F-CDB7-49D1-BEEE-704BE3D4918C@mcs.anl.gov> On Jun 29, 2011, at 10:12 PM, John Fettig wrote: > On Wed, Jun 29, 2011 at 11:01 PM, Barry Smith wrote: >> >> John, >> >> The MatCreateSeqAIJWithArrays() uses the memory (arrays) that you provide so you cannot free those arrays or muck with them until you are finished with the solve. > > I figured that was the case. I wasn't sure how far down the hierarchy > the memory I provided was referenced, so I figured if I passed in a > Pmat that used petsc allocated memory I could get around this. Once the preconditioner is built the "old" Pmat is not used for later linear solves only the "built" preconditioner is used so passing the old Pmat is not doing anything for you. > >> What happens if you ALWAYS copy with MatDuplicate() the MatCreate*WithArrays() matrix immediately after you create it and use that matrix as I describe below in the setoperators? Does it start behaving like >> the 1) MatSetValues() or does it still behave "badly"? > > It takes roughly the same number of CG iterations as 1), but the solve > time indicates to me that PCSetUp is being called every solve. The > solutions are also different, which also indicates that the > preconditioner is being rebuilt every solve. If you pass in SAME_PRECONDITIONER to the KSPSetOperators() it is simply impossible that it will rebuild the preconditioner. There must be another explanation as to what is going on. Are you calling anything else on the KSP? If you run with -info it should print the message "Setting PC with identical preconditioner\n" when it reuses the preconditioner otherwise it will print "Setting up new PC\n") or "Setting up PC with same nonzero pattern\n") or "Setting up PC with different nonzero pattern" Is this only happing with PCML and not with other preconditioners like PCILU? Barry > > Thanks, > John From john.fettig at gmail.com Wed Jun 29 22:35:07 2011 From: john.fettig at gmail.com (John Fettig) Date: Wed, 29 Jun 2011 23:35:07 -0400 Subject: [petsc-users] Reusing ML preconditioner In-Reply-To: <78447D8F-CDB7-49D1-BEEE-704BE3D4918C@mcs.anl.gov> References: <19008F56-E40F-4BDC-800E-A18CEF286099@mcs.anl.gov> <36F75DB1-D7C3-4407-8C24-1892138CB5A0@mcs.anl.gov> <78447D8F-CDB7-49D1-BEEE-704BE3D4918C@mcs.anl.gov> Message-ID: On Wed, Jun 29, 2011 at 11:21 PM, Barry Smith wrote: > ?If you pass in SAME_PRECONDITIONER to the KSPSetOperators() it is simply impossible that it will rebuild the preconditioner. There must be another explanation as to what is going on. Are you calling anything else on the KSP? You are right. I was playing around with options and had set it to SAME_NONZERO_PATTERN for the first iteration at each timestep to make sure that would work (it does), and forgot to change it back. When it is actually set to SAME_PRECONDITIONER and I duplicate Amat to Pmat and pass them to KSPSetOperators() before every KSPSolve, it definitely uses the same preconditioner (as reported by -info), but unfortunately it reverts back to the behavior I saw previously. The number of iterations jumps on the second timestep. > ? Is this only happing with PCML and not with other preconditioners like PCILU? I have only tried PCML. Let me try PCILU and get back to you. John From john.fettig at gmail.com Wed Jun 29 23:26:22 2011 From: john.fettig at gmail.com (John Fettig) Date: Thu, 30 Jun 2011 00:26:22 -0400 Subject: [petsc-users] Reusing ML preconditioner In-Reply-To: References: <19008F56-E40F-4BDC-800E-A18CEF286099@mcs.anl.gov> <36F75DB1-D7C3-4407-8C24-1892138CB5A0@mcs.anl.gov> <78447D8F-CDB7-49D1-BEEE-704BE3D4918C@mcs.anl.gov> Message-ID: On Wed, Jun 29, 2011 at 11:35 PM, John Fettig wrote: >> ? Is this only happing with PCML and not with other preconditioners like PCILU? > > I have only tried PCML. ?Let me try PCILU and get back to you. PCILU doesn't work at all on the second timestep in either version. The solution hasn't met tolerances after 10000 iterations, whereas it only took ~100 in the first timestep. But the solutions match in both versions. PCHYPRE works, and the behavior is identical in both versions of my code. The preconditioner also loses effectiveness in the second timestep (similar to the MatCreate*WithArrays version with ML), but at least the behavior is consistent between versions. The fact that ML can do so well on the second iteration is somewhat surprising. If it is keeping the factorization of the coarse grid, I would think that would need to be updated. However, shouldn't the restriction/interpolation operators remain relatively decent if the structure of the matrix doesn't change? John From aldo.bonfiglioli at unibas.it Thu Jun 30 08:17:59 2011 From: aldo.bonfiglioli at unibas.it (Aldo Bonfiglioli) Date: Thu, 30 Jun 2011 15:17:59 +0200 Subject: [petsc-users] Interpreting ksp_view Message-ID: <4E0C7787.7030608@unibas.it> Dear all, I have a doubt concerning the interpretation of the output from -vec_view. I am using ILU(k) on a BAIJ matrix; the run is uni-processor. > KSP Object: > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=1e+10 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: > type: ilu > ILU: out-of-place factorization > ILU: Reusing fill from past factorization > ILU: Reusing reordering from past factorization > 1 level of fill > tolerance for zero pivot 1e-12 > using diagonal shift to prevent zero pivot > matrix ordering: natural > factor fill ratio given 1.30284, needed 1.30284 > Factored matrix follows: > Matrix Object: > type=seqbaij, rows=52995, cols=52995 > package used to perform factorization: petsc > total: nonzeros=2398825, allocated nonzeros=95953 > block size is 5 > [0] PetscCommDuplicate(): returning tag 1073741814 > linear system matrix = precond matrix: > Matrix Object: > type=seqbaij, rows=52995, cols=52995 > total: nonzeros=1841225, allocated nonzeros=73649 > block size is 5 I have set the fill ratio and this is aknowledged. I do NOT understand why the nof allocated nonzeros does NOT match the nonzeros; this applies to both the A matrix and its ILU(k) preconditioner the ratio allocated nonzeros/nonzeros = 25 = blocksize^2 in both cases. Is there a bs*bs factor missing or am I misinterpreting the output? If a look at the MatInfo structure of the preconditioner > info = 1 25.000000000000000 > info = 2 95953.000000000000 > info = 3 2398825.0000000000 > info = 4 -2302872.0000000000 > info = 5 19576580.000000000 > info = 6 0.0000000000000000 > info = 7 0.0000000000000000 > info = 8 1.3028418580021452 > info = 9 1.3028418580021452 > info = 10 0.0000000000000000 a negative number of nz_unneeded seems to indicate that not enough memory has been provided. on the other hand, the -info option seems to confirm that the linear system matrix had been correctly pre-allocated > [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 52995 X 52995, block size > 5; storage space: 0 unneeded, 1841225 used > [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues is 0 Thanks, Aldo From adam1.byrd at gmail.com Thu Jun 30 14:30:06 2011 From: adam1.byrd at gmail.com (Adam Byrd) Date: Thu, 30 Jun 2011 15:30:06 -0400 Subject: [petsc-users] Matrix Construction Question In-Reply-To: References: Message-ID: I'm trying to work through what I need to do, again by practicing with a small scale random problem. The general order of events seems to be: create a matrix, fill it, assemble it, factor it, then one can use solvers with it. When I use MatLUFactor on my matrix before using it with a solver I get this error: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Null argument, when expecting valid pointer! [0]PETSC ERROR: Null Object: Parameter # 1! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./test on a osx-gnu-c named Macintosh-3.local by adambyrd Thu Jun 30 15:27:30 2011 [0]PETSC ERROR: Libraries linked from /Users/adambyrd/soft/petsc-3.1-p8/osx-gnu-cpp/lib [0]PETSC ERROR: Configure run at Tue Jun 28 12:56:55 2011 [0]PETSC ERROR: Configure options PETSC_ARCH=osx-gnu-cpp --with-fc=gfortran -download-f-blas-lapack=1 --download-mpich=1 --with-scalar-type=complex --with-clanguage=c++ [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ISInvertPermutation() line 209 in src/vec/is/interface/index.c [0]PETSC ERROR: MatLUFactorSymbolic_SeqAIJ() line 306 in src/mat/impls/aij/seq/aijfact.c [0]PETSC ERROR: MatLUFactorSymbolic() line 2534 in src/mat/interface/matrix.c [0]PETSC ERROR: MatLUFactor_SeqAIJ() line 945 in src/mat/impls/aij/seq/aijfact.c [0]PETSC ERROR: MatLUFactor() line 2417 in src/mat/interface/matrix.c [0]PETSC ERROR: main() line 62 in WDtest.cpp application called MPI_Abort(MPI_COMM_WORLD, 85) - process 0[unset]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 85) - process 0 I don't understand what I'm doing wrong. Respectfully, Adam On Wed, Jun 29, 2011 at 1:26 AM, Matthew Knepley wrote: > On Tue, Jun 28, 2011 at 10:20 PM, Adam Byrd wrote: > >> Matt, >> >> Alright, that means I need to continue learning how to use MatSetValues(). >> With my 6x6 example I tried filling it with four 3x3 sub matrices, but when >> I do that I get the error 'sum of local sizes 12 does not equal global >> size.' I had 4 processors each calling MatSetValues for their own 3x3. >> Graphically, I arranged the nodes 0 1 >> >> 2 3 >> where process 0 had global rows 0-2 and global columns 0-2; process 1 had >> 0-2, 3-5; process 2 had 3-5, 0-2; and process 3 had 3-5, 3-5. From the >> documentation, I think this should be correct, but I'm not sure. Also, which >> format would you recommend for storing the matrix? >> > > 1) With any error, send the Entire error message. > > 2) PETSc matrices are divided by rows, not rows and columns, see the manual > section. Rows & columns only makes sense for dense matrices > > 3) You can still set arbitrary blocks no matter how the matrix is divided > > 4) The error means you tried to set both local and global dimensions, and > they do not add up correctly. Just set the global dimensions > > Matt > > >> Jack, >> >> I'm a summer intern just getting started with this project, so I don't >> know all the details yet (I can ask though). I know I need to find the >> Green's function which will involve the trace of the inverted Hamiltonian, >> as well as the rest of the matrix. I have inquired about avoiding the >> inversion altogether, but my instructor doesn't believe there is a way >> around it. Once I've worked through the math I want to explore other options >> though. >> >> Respectfully, >> Adam >> >> On Tue, Jun 28, 2011 at 6:08 PM, Matthew Knepley wrote: >> >>> On Tue, Jun 28, 2011 at 5:01 PM, Adam Byrd wrote: >>> >>>> Actually, it's quite sparse. In the 3600x3600 there are only just 4 >>>> nonzero entries in each row. This means it's 99.9% empty. My smaller 6x6 >>>> example is dense, but it's only practice building and manipulating matrices. >>>> >>> >>> Ah, then its easy. Just call MatSetValues() with each block. Then use >>> MUMPS to do a sparse direct solve. >>> >>> Matt >>> >>> >>>> Respectfully, >>>> Adam >>>> >>>> >>>> On Tue, Jun 28, 2011 at 5:55 PM, Matthew Knepley wrote: >>>> >>>>> It sounds like you have a dense matrix (from your example). Is this >>>>> true? If so, you should use Elemental (on Google Code). >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> On Tue, Jun 28, 2011 at 8:55 AM, Adam Byrd wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I'm rather new to PETSc and trying to work out the best way to create >>>>>> and fill a large sparse matrix distributed over many processors. Currently, >>>>>> my goal is to create a 3600x3600 matrix in units of 12x12 blocks with >>>>>> several blocks on any given node. I'd like to create the matrix in such a >>>>>> way that each node only holds the information in it's handful of blocks and >>>>>> not the entire matrix. Eventually, this matrix is to be inverted (I know, >>>>>> inversion should be avoided, but as this is a Hamiltonian matrix from which >>>>>> I need the Green's function, I'm unaware of a way to forgo carrying out the >>>>>> inversion). Additionally, the values will be changed slightly and the matrix >>>>>> will be repeatedly inverted. It's structure will remain the same. In order >>>>>> to learn how to do this is I am starting with a small 6x6 matrix broken into >>>>>> four 3x3 blocks and distributed one block per node. I've been able to create >>>>>> a local 3x3 matrix on each node, with it's own values, and with the global >>>>>> row/column IDs correctly set to [0, 1, 2] or [3, 4, 5] depending on where >>>>>> the block is in the matrix. My problem manifests when I try to create the >>>>>> larger matrix from the individual smaller ones. When the matrix is >>>>>> constructed I'm trying to use MatSetValues and having each node pass in it's >>>>>> 3x3 block. I end up with an error that the sum of local lengths 12x12 does >>>>>> not match the global length 6x6. It appears as though this is from passing >>>>>> in four 3x3s and the program interpreting that as a 12x12 instead of as a >>>>>> 6x6 with the blocks in a grid. >>>>>> >>>>>> My question is then: is it possible to fill a matrix as a grid of >>>>>> blocks, or can I only fill it in groups of rows or columns? Also, am I >>>>>> approaching this problem the correct way, or are there more efficient ways >>>>>> of building this matrix with the ultimate goal of inverting it? >>>>>> >>>>>> I have included my copy of a modified example if it helps. I do >>>>>> apologize if this is answered somewhere in the documentation, I have been >>>>>> unable to find a solution. >>>>>> >>>>>> Respectfully, >>>>>> Adam >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam1.byrd at gmail.com Thu Jun 30 14:33:13 2011 From: adam1.byrd at gmail.com (Adam Byrd) Date: Thu, 30 Jun 2011 15:33:13 -0400 Subject: [petsc-users] Matrix Construction Question In-Reply-To: References: Message-ID: Forgot to attach the code, if needed. On Thu, Jun 30, 2011 at 3:30 PM, Adam Byrd wrote: > I'm trying to work through what I need to do, again by practicing with a > small scale random problem. The general order of events seems to be: create > a matrix, fill it, assemble it, factor it, then one can use solvers with it. > When I use MatLUFactor on my matrix before using it with a solver I get this > error: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Null argument, when expecting valid pointer! > [0]PETSC ERROR: Null Object: Parameter # 1! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 > CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./test on a osx-gnu-c named Macintosh-3.local by adambyrd > Thu Jun 30 15:27:30 2011 > [0]PETSC ERROR: Libraries linked from > /Users/adambyrd/soft/petsc-3.1-p8/osx-gnu-cpp/lib > [0]PETSC ERROR: Configure run at Tue Jun 28 12:56:55 2011 > [0]PETSC ERROR: Configure options PETSC_ARCH=osx-gnu-cpp --with-fc=gfortran > -download-f-blas-lapack=1 --download-mpich=1 --with-scalar-type=complex > --with-clanguage=c++ > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ISInvertPermutation() line 209 in > src/vec/is/interface/index.c > [0]PETSC ERROR: MatLUFactorSymbolic_SeqAIJ() line 306 in > src/mat/impls/aij/seq/aijfact.c > [0]PETSC ERROR: MatLUFactorSymbolic() line 2534 in > src/mat/interface/matrix.c > [0]PETSC ERROR: MatLUFactor_SeqAIJ() line 945 in > src/mat/impls/aij/seq/aijfact.c > [0]PETSC ERROR: MatLUFactor() line 2417 in src/mat/interface/matrix.c > [0]PETSC ERROR: main() line 62 in WDtest.cpp > application called MPI_Abort(MPI_COMM_WORLD, 85) - process 0[unset]: > aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 85) - process 0 > > I don't understand what I'm doing wrong. > > Respectfully, > Adam > > On Wed, Jun 29, 2011 at 1:26 AM, Matthew Knepley wrote: > >> On Tue, Jun 28, 2011 at 10:20 PM, Adam Byrd wrote: >> >>> Matt, >>> >>> Alright, that means I need to continue learning how to use >>> MatSetValues(). With my 6x6 example I tried filling it with four 3x3 sub >>> matrices, but when I do that I get the error 'sum of local sizes 12 does not >>> equal global size.' I had 4 processors each calling MatSetValues for their >>> own 3x3. Graphically, I arranged the nodes 0 1 >>> >>> 2 3 >>> where process 0 had global rows 0-2 and global columns 0-2; process 1 had >>> 0-2, 3-5; process 2 had 3-5, 0-2; and process 3 had 3-5, 3-5. From the >>> documentation, I think this should be correct, but I'm not sure. Also, which >>> format would you recommend for storing the matrix? >>> >> >> 1) With any error, send the Entire error message. >> >> 2) PETSc matrices are divided by rows, not rows and columns, see the >> manual section. Rows & columns only makes sense for dense matrices >> >> 3) You can still set arbitrary blocks no matter how the matrix is divided >> >> 4) The error means you tried to set both local and global dimensions, and >> they do not add up correctly. Just set the global dimensions >> >> Matt >> >> >>> Jack, >>> >>> I'm a summer intern just getting started with this project, so I don't >>> know all the details yet (I can ask though). I know I need to find the >>> Green's function which will involve the trace of the inverted Hamiltonian, >>> as well as the rest of the matrix. I have inquired about avoiding the >>> inversion altogether, but my instructor doesn't believe there is a way >>> around it. Once I've worked through the math I want to explore other options >>> though. >>> >>> Respectfully, >>> Adam >>> >>> On Tue, Jun 28, 2011 at 6:08 PM, Matthew Knepley wrote: >>> >>>> On Tue, Jun 28, 2011 at 5:01 PM, Adam Byrd wrote: >>>> >>>>> Actually, it's quite sparse. In the 3600x3600 there are only just 4 >>>>> nonzero entries in each row. This means it's 99.9% empty. My smaller 6x6 >>>>> example is dense, but it's only practice building and manipulating matrices. >>>>> >>>> >>>> Ah, then its easy. Just call MatSetValues() with each block. Then use >>>> MUMPS to do a sparse direct solve. >>>> >>>> Matt >>>> >>>> >>>>> Respectfully, >>>>> Adam >>>>> >>>>> >>>>> On Tue, Jun 28, 2011 at 5:55 PM, Matthew Knepley wrote: >>>>> >>>>>> It sounds like you have a dense matrix (from your example). Is this >>>>>> true? If so, you should use Elemental (on Google Code). >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> On Tue, Jun 28, 2011 at 8:55 AM, Adam Byrd wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I'm rather new to PETSc and trying to work out the best way to create >>>>>>> and fill a large sparse matrix distributed over many processors. Currently, >>>>>>> my goal is to create a 3600x3600 matrix in units of 12x12 blocks with >>>>>>> several blocks on any given node. I'd like to create the matrix in such a >>>>>>> way that each node only holds the information in it's handful of blocks and >>>>>>> not the entire matrix. Eventually, this matrix is to be inverted (I know, >>>>>>> inversion should be avoided, but as this is a Hamiltonian matrix from which >>>>>>> I need the Green's function, I'm unaware of a way to forgo carrying out the >>>>>>> inversion). Additionally, the values will be changed slightly and the matrix >>>>>>> will be repeatedly inverted. It's structure will remain the same. In order >>>>>>> to learn how to do this is I am starting with a small 6x6 matrix broken into >>>>>>> four 3x3 blocks and distributed one block per node. I've been able to create >>>>>>> a local 3x3 matrix on each node, with it's own values, and with the global >>>>>>> row/column IDs correctly set to [0, 1, 2] or [3, 4, 5] depending on where >>>>>>> the block is in the matrix. My problem manifests when I try to create the >>>>>>> larger matrix from the individual smaller ones. When the matrix is >>>>>>> constructed I'm trying to use MatSetValues and having each node pass in it's >>>>>>> 3x3 block. I end up with an error that the sum of local lengths 12x12 does >>>>>>> not match the global length 6x6. It appears as though this is from passing >>>>>>> in four 3x3s and the program interpreting that as a 12x12 instead of as a >>>>>>> 6x6 with the blocks in a grid. >>>>>>> >>>>>>> My question is then: is it possible to fill a matrix as a grid of >>>>>>> blocks, or can I only fill it in groups of rows or columns? Also, am I >>>>>>> approaching this problem the correct way, or are there more efficient ways >>>>>>> of building this matrix with the ultimate goal of inverting it? >>>>>>> >>>>>>> I have included my copy of a modified example if it helps. I do >>>>>>> apologize if this is answered somewhere in the documentation, I have been >>>>>>> unable to find a solution. >>>>>>> >>>>>>> Respectfully, >>>>>>> Adam >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test.cpp Type: text/x-c++src Size: 2903 bytes Desc: not available URL: From hzhang at mcs.anl.gov Thu Jun 30 15:18:48 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Thu, 30 Jun 2011 15:18:48 -0500 Subject: [petsc-users] Matrix Construction Question In-Reply-To: References: Message-ID: Add ierr = MatGetOrdering(testMat,MATORDERING_ND,&isrow,&iscol);CHKERRQ(ierr); before the line ierr = MatFactorInfoInitialize(&luinfo);CHKERRQ(ierr); Somehow, your matrix is numerically singular. With MATORDERING_NATURAL I get [0]PETSC ERROR: Detected zero pivot in LU factorization even using MATDENSE matrix format, which calls lapack. With MATORDERING_ND, I get useless inverseMat. The modified code I used is attached. Hong On Thu, Jun 30, 2011 at 2:30 PM, Adam Byrd wrote: > I'm trying to work through what I need to do, again by practicing with a > small scale random problem. The general order of events seems to be: create > a matrix, fill it, assemble it, factor it, then one can use solvers with it. > When I use MatLUFactor on my matrix before using it with a solver I get this > error: > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Null argument, when expecting valid pointer! > [0]PETSC ERROR: Null Object: Parameter # 1! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 > CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./test on a osx-gnu-c named Macintosh-3.local by adambyrd > Thu Jun 30 15:27:30 2011 > [0]PETSC ERROR: Libraries linked from > /Users/adambyrd/soft/petsc-3.1-p8/osx-gnu-cpp/lib > [0]PETSC ERROR: Configure run at Tue Jun 28 12:56:55 2011 > [0]PETSC ERROR: Configure options PETSC_ARCH=osx-gnu-cpp --with-fc=gfortran > -download-f-blas-lapack=1 --download-mpich=1 --with-scalar-type=complex > --with-clanguage=c++ > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ISInvertPermutation() line 209 in > src/vec/is/interface/index.c > [0]PETSC ERROR: MatLUFactorSymbolic_SeqAIJ() line 306 in > src/mat/impls/aij/seq/aijfact.c > [0]PETSC ERROR: MatLUFactorSymbolic() line 2534 in > src/mat/interface/matrix.c > [0]PETSC ERROR: MatLUFactor_SeqAIJ() line 945 in > src/mat/impls/aij/seq/aijfact.c > [0]PETSC ERROR: MatLUFactor() line 2417 in src/mat/interface/matrix.c > [0]PETSC ERROR: main() line 62 in WDtest.cpp > application called MPI_Abort(MPI_COMM_WORLD, 85) - process 0[unset]: > aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 85) - process 0 > I don't understand what I'm doing wrong. > Respectfully, > Adam > On Wed, Jun 29, 2011 at 1:26 AM, Matthew Knepley wrote: >> >> On Tue, Jun 28, 2011 at 10:20 PM, Adam Byrd wrote: >>> >>> Matt, >>> >>> Alright, that means I need to continue learning how to use >>> MatSetValues(). With my 6x6 example I tried filling it with four 3x3 sub >>> matrices, but when I do that I get the error 'sum of local sizes 12 does not >>> equal global size.' I had 4 processors each calling MatSetValues for their >>> own 3x3. Graphically, I arranged the nodes 0 1 >>> >>> ????????????????????????????????????????????? ???????????????????????? 2 3 >>> where process 0 had global rows 0-2 and global columns 0-2; process 1 had >>> 0-2, 3-5; process 2 had 3-5, 0-2; and process 3 had 3-5, 3-5. From the >>> documentation, I think this should be correct, but I'm not sure. Also, which >>> format would you recommend for storing the matrix? >> >> 1) With any error, send the Entire error message. >> 2) PETSc matrices are divided by rows, not rows and columns, see the >> manual section. Rows & columns only makes sense for dense matrices >> 3) You can still set arbitrary blocks no matter how the matrix is divided >> 4) The error means you tried to set both local and global dimensions, and >> they do not add up correctly. Just set the global dimensions >> ? ?Matt >> >>> >>> Jack, >>> >>> I'm a summer intern just getting started with this project, so I don't >>> know all the details yet (I can ask though). I know I need to find the >>> Green's function which will involve the trace of the inverted Hamiltonian, >>> as well as the rest of the matrix. I have inquired about avoiding the >>> inversion altogether, but my instructor doesn't believe there is a way >>> around it. Once I've worked through the math I want to explore other options >>> though. >>> >>> Respectfully, >>> Adam >>> >>> On Tue, Jun 28, 2011 at 6:08 PM, Matthew Knepley >>> wrote: >>>> >>>> On Tue, Jun 28, 2011 at 5:01 PM, Adam Byrd wrote: >>>>> >>>>> Actually, it's quite sparse. In the 3600x3600 there are only just 4 >>>>> nonzero entries in each row. This means it's 99.9% empty. My smaller 6x6 >>>>> example is dense, but it's only practice building and manipulating matrices. >>>> >>>> Ah, then its easy. Just call MatSetValues() with each block. Then use >>>> MUMPS to do a sparse direct solve. >>>> ? Matt >>>> >>>>> >>>>> Respectfully, >>>>> Adam >>>>> >>>>> On Tue, Jun 28, 2011 at 5:55 PM, Matthew Knepley >>>>> wrote: >>>>>> >>>>>> It sounds like you have a dense matrix (from your example). Is this >>>>>> true? If so, you should use Elemental (on Google Code). >>>>>> ? Thanks, >>>>>> ? ? ?Matt >>>>>> >>>>>> On Tue, Jun 28, 2011 at 8:55 AM, Adam Byrd >>>>>> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> I'm rather new to PETSc and trying to work out the best way to create >>>>>>> and fill a large sparse matrix distributed over many processors. Currently, >>>>>>> my goal is to create a 3600x3600 matrix in units of 12x12 blocks with >>>>>>> several blocks on any given node. I'd like to create the matrix in such a >>>>>>> way that each node only holds the information in it's handful of blocks and >>>>>>> not the entire matrix. Eventually, this matrix is to be inverted (I know, >>>>>>> inversion should be avoided, but as this is a Hamiltonian matrix from which >>>>>>> I need the Green's function, I'm unaware of a way to forgo carrying out the >>>>>>> inversion). Additionally, the values will be changed slightly and the matrix >>>>>>> will be repeatedly inverted. It's structure will remain the same. In order >>>>>>> to learn how to do this is I am starting with a small 6x6 matrix broken into >>>>>>> four 3x3 blocks and distributed one block per node. I've been able to create >>>>>>> a local 3x3 matrix on each node, with it's own values, and with the global >>>>>>> row/column IDs correctly set to [0, 1, 2] or [3, 4, 5] depending on where >>>>>>> the block is in the matrix. My problem manifests when I try to create the >>>>>>> larger matrix from the individual smaller ones. When the matrix is >>>>>>> constructed I'm trying to use MatSetValues and having each node pass in it's >>>>>>> 3x3 block. I end up with an error that the sum of local lengths 12x12 does >>>>>>> not match the global length 6x6. It appears as though this is from passing >>>>>>> in four 3x3s and the program interpreting that as a 12x12 instead of as a >>>>>>> 6x6 with the blocks in a grid. >>>>>>> My question is then: is it possible to fill a matrix as a grid of >>>>>>> blocks, or can I only fill it in groups of rows or columns? Also, am I >>>>>>> approaching this problem the correct way, or are there more efficient ways >>>>>>> of building this matrix with the ultimate goal of inverting it? >>>>>>> I have included my copy of a modified example if it helps. I do >>>>>>> apologize if this is answered somewhere in the documentation, I have been >>>>>>> unable to find a solution. >>>>>>> Respectfully, >>>>>>> Adam >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener > > -------------- next part -------------- A non-text attachment was scrubbed... Name: test.cpp Type: text/x-c++src Size: 3138 bytes Desc: not available URL: From bsmith at mcs.anl.gov Thu Jun 30 16:11:58 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 30 Jun 2011 16:11:58 -0500 Subject: [petsc-users] Reusing ML preconditioner In-Reply-To: References: <19008F56-E40F-4BDC-800E-A18CEF286099@mcs.anl.gov> <36F75DB1-D7C3-4407-8C24-1892138CB5A0@mcs.anl.gov> <78447D8F-CDB7-49D1-BEEE-704BE3D4918C@mcs.anl.gov> Message-ID: <527B8F7E-7DBD-40BA-A199-C1D973FFFF7F@mcs.anl.gov> On Jun 29, 2011, at 10:35 PM, John Fettig wrote: > On Wed, Jun 29, 2011 at 11:21 PM, Barry Smith wrote: >> If you pass in SAME_PRECONDITIONER to the KSPSetOperators() it is simply impossible that it will rebuild the preconditioner. There must be another explanation as to what is going on. Are you calling anything else on the KSP? > > You are right. I was playing around with options and had set it to > SAME_NONZERO_PATTERN for the first iteration at each timestep to make > sure that would work (it does), and forgot to change it back. > > When it is actually set to SAME_PRECONDITIONER and I duplicate Amat to > Pmat and pass them to KSPSetOperators() before every KSPSolve, it > definitely uses the same preconditioner (as reported by -info), but > unfortunately it reverts back to the behavior I saw previously. What do you mean? Reverts back to? After some iterations? Copying the matrix and passing it into KSPSetOperators() as the Pmat cannot make any difference. Can you send us a code that reproduces the problem you think is happening (petsc-maint at mcs.anl.gov), I cannot get a consistent handle on the issue. Barry > The > number of iterations jumps on the second timestep. > >> Is this only happing with PCML and not with other preconditioners like PCILU? > > I have only tried PCML. Let me try PCILU and get back to you. > > John From john.fettig at gmail.com Thu Jun 30 16:20:17 2011 From: john.fettig at gmail.com (John Fettig) Date: Thu, 30 Jun 2011 17:20:17 -0400 Subject: [petsc-users] Reusing ML preconditioner In-Reply-To: <527B8F7E-7DBD-40BA-A199-C1D973FFFF7F@mcs.anl.gov> References: <19008F56-E40F-4BDC-800E-A18CEF286099@mcs.anl.gov> <36F75DB1-D7C3-4407-8C24-1892138CB5A0@mcs.anl.gov> <78447D8F-CDB7-49D1-BEEE-704BE3D4918C@mcs.anl.gov> <527B8F7E-7DBD-40BA-A199-C1D973FFFF7F@mcs.anl.gov> Message-ID: On Thu, Jun 30, 2011 at 5:11 PM, Barry Smith wrote: > > On Jun 29, 2011, at 10:35 PM, John Fettig wrote: > >> On Wed, Jun 29, 2011 at 11:21 PM, Barry Smith wrote: >>> ?If you pass in SAME_PRECONDITIONER to the KSPSetOperators() it is simply impossible that it will rebuild the preconditioner. There must be another explanation as to what is going on. Are you calling anything else on the KSP? >> >> You are right. ?I was playing around with options and had set it to >> SAME_NONZERO_PATTERN for the first iteration at each timestep to make >> sure that would work (it does), and forgot to change it back. >> >> When it is actually set to SAME_PRECONDITIONER and I duplicate Amat to >> Pmat and pass them to KSPSetOperators() before every KSPSolve, it >> definitely uses the same preconditioner (as reported by -info), but >> unfortunately it reverts back to the behavior I saw previously. > > ? What do you mean? Reverts back to? After some iterations? Copying the matrix and passing it into KSPSetOperators() as the Pmat cannot make any difference. I mean that it doesn't make any difference, as you expect. > ? Can you send us a code that reproduces the problem you think is happening (petsc-maint at mcs.anl.gov), I cannot get a consistent handle on the issue. I will see what I can do. John From bsmith at mcs.anl.gov Thu Jun 30 16:43:28 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 30 Jun 2011 16:43:28 -0500 Subject: [petsc-users] Interpreting ksp_view In-Reply-To: <4E0C7787.7030608@unibas.it> References: <4E0C7787.7030608@unibas.it> Message-ID: <99F83178-0480-4A23-913E-5E2E546FBB70@mcs.anl.gov> Aldo, This is a bug. The allocated number was not multipled by the block size squared. Because of the this the compute unneeded number was garbage. We fixed this a while ago in petsc-dev (yes it should have been fixed as a patch for 3.1 but that didn't happen.) Barry On Jun 30, 2011, at 8:17 AM, Aldo Bonfiglioli wrote: > Dear all, > I have a doubt concerning the interpretation of the output from -vec_view. > I am using ILU(k) on a BAIJ matrix; the run is uni-processor. >> KSP Object: >> type: gmres >> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement >> GMRES: happy breakdown tolerance 1e-30 >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=1e+10 >> left preconditioning >> using PRECONDITIONED norm type for convergence test >> PC Object: >> type: ilu >> ILU: out-of-place factorization >> ILU: Reusing fill from past factorization >> ILU: Reusing reordering from past factorization >> 1 level of fill >> tolerance for zero pivot 1e-12 >> using diagonal shift to prevent zero pivot >> matrix ordering: natural >> factor fill ratio given 1.30284, needed 1.30284 >> Factored matrix follows: >> Matrix Object: >> type=seqbaij, rows=52995, cols=52995 >> package used to perform factorization: petsc >> total: nonzeros=2398825, allocated nonzeros=95953 >> block size is 5 >> [0] PetscCommDuplicate(): returning tag 1073741814 >> linear system matrix = precond matrix: >> Matrix Object: >> type=seqbaij, rows=52995, cols=52995 >> total: nonzeros=1841225, allocated nonzeros=73649 >> block size is 5 > I have set the fill ratio and this is aknowledged. > I do NOT understand why the nof allocated nonzeros does NOT match > the nonzeros; this applies to both the A matrix and its ILU(k) preconditioner > the ratio allocated nonzeros/nonzeros = 25 = blocksize^2 in both cases. > Is there a bs*bs factor missing or am I misinterpreting the output? > If a look at the MatInfo structure of the preconditioner >> info = 1 25.000000000000000 >> info = 2 95953.000000000000 >> info = 3 2398825.0000000000 >> info = 4 -2302872.0000000000 >> info = 5 19576580.000000000 >> info = 6 0.0000000000000000 >> info = 7 0.0000000000000000 >> info = 8 1.3028418580021452 >> info = 9 1.3028418580021452 >> info = 10 0.0000000000000000 > a negative number of nz_unneeded seems to indicate that not enough memory has been provided. > on the other hand, the -info option seems to confirm that the linear system matrix had been correctly pre-allocated >> [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 52995 X 52995, block size 5; storage space: 0 unneeded, 1841225 used >> [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues is 0 > Thanks, > Aldo > >