From fkar at nemesis-project.org Mon Apr 2 19:58:55 2007 From: fkar at nemesis-project.org (F.E. Karaoulanis) Date: Tue, 03 Apr 2007 03:58:55 +0300 Subject: A simple question. Message-ID: <4611A6CF.6010502@nemesis-project.org> I am new to PETSc, so my question may sound trivial. This is what I've done: 1. I configured PETSc on WinXP under cygwin, using: *************************************************************************** ./config/configure.py \ --with-cc='win32fe cl' \ --with-cxx='win32fe cl' \ --with-fortran=0 \ --with-parmetis=0 \ --download-c-blas-lapack=yes \ --with-mpi-dir=/cygdrive/C/MPICH2 *************************************************************************** 2. I wrote and compiled the following (simple) code: *************************************************************************** static char help[] = "Testing PETSc installation."; #include "petscksp.h" #include using namespace std; #undef __FUNCT__ #define __FUNCT__ "main" int main(int argc,char **args) { Vec x; PetscInt n =10; PetscErrorCode ierr; PetscMPIInt size; PetscScalar one=1.0; double vd[3]={2.,2.,2.}; int indices[3]={0,5,7}; PetscScalar* v=&vd[0]; ierr = PetscInitialize(&argc,&args,(char *)0,help); CHKERRQ(ierr); ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size); CHKERRQ(ierr); printf("Number of processors : %d\n",size); ierr = VecCreate(PETSC_COMM_WORLD,&x); CHKERRQ(ierr); ierr = VecSetSizes(x,PETSC_DECIDE,n); CHKERRQ(ierr); ierr = VecSetType(x,"mpi"); CHKERRQ(ierr); ierr = VecSetFromOptions(x); CHKERRQ(ierr); ierr = VecSet(x,one); CHKERRQ(ierr); VecSetValues(x,3,indices,v,ADD_VALUES); VecAssemblyBegin(x); VecAssemblyEnd(x); ierr = VecView(x,PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); VecDestroy(x); CHKERRQ(ierr); ierr = PetscFinalize(); CHKERRQ(ierr); return 0; } *************************************************************************** 3. And now I'm running it on a dual core processor for -n 1 & 2. These are the results. *************************************************************************** Number of processors : 1 Process [0] 3 1 1 1 1 3 1 3 1 1 *************************************************************************** Number of processors : 2 Process [0] 5 1 1 1 1 Process [1] 5 1 5 1 1 Number of processors : 2 *************************************************************************** This is not I was really looking for. I intended to have the same output (concerning the vector entries) and no double-printings (like "Number of processors : 2"). Do you think this is an MPI setup problem, or have I not really understood what the above code does? Kind regards, Fotios Karaoulanis._ ps. Congratulations on your excellent work! -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Fotios E. Karaoulanis Dipl. Civil Engineer, MSc TUM tel +30 2310 458913 fax +30 2310 458913 mob +30 6948 179452 e-mail fkar at nemesis-project.org -------------------------------------------- Consider visiting www.nemesis-project.org. Home of an experimental finite element code. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From hzhang at mcs.anl.gov Mon Apr 2 20:48:41 2007 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Mon, 2 Apr 2007 20:48:41 -0500 (CDT) Subject: A simple question. In-Reply-To: <4611A6CF.6010502@nemesis-project.org> References: <4611A6CF.6010502@nemesis-project.org> Message-ID: The problem is at VecSetValues(). > Vec x; > PetscInt n =10; > PetscErrorCode ierr; > PetscMPIInt size; > PetscScalar one=1.0; > double vd[3]={2.,2.,2.}; > int indices[3]={0,5,7}; > PetscScalar* v=&vd[0]; > > ierr = PetscInitialize(&argc,&args,(char *)0,help); CHKERRQ(ierr); > ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size); CHKERRQ(ierr); > printf("Number of processors : %d\n",size); > ierr = VecCreate(PETSC_COMM_WORLD,&x); CHKERRQ(ierr); > ierr = VecSetSizes(x,PETSC_DECIDE,n); CHKERRQ(ierr); > ierr = VecSetType(x,"mpi"); CHKERRQ(ierr); > ierr = VecSetFromOptions(x); CHKERRQ(ierr); > ierr = VecSet(x,one); CHKERRQ(ierr); > VecSetValues(x,3,indices,v,ADD_VALUES); ^^^^^^^^^^ Here, when np = 2, both processors add v=2.0 into the vector at 0,5 7th component, resulting different output than np=1. You should call VecGetOwnershipRange(Vec x,PetscInt *low,PetscInt *high), then have each processor set values into its own part. Hong > VecAssemblyBegin(x); > VecAssemblyEnd(x); > > ierr = VecView(x,PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); > VecDestroy(x); CHKERRQ(ierr); > ierr = PetscFinalize(); CHKERRQ(ierr); > return 0; > } > *************************************************************************** > > 3. And now I'm running it on a dual core processor for -n 1 & 2. These are > the results. > *************************************************************************** > Number of processors : 1 > Process [0] > 3 > 1 > 1 > 1 > 1 > 3 > 1 > 3 > 1 > 1 > *************************************************************************** > Number of processors : 2 > Process [0] > 5 > 1 > 1 > 1 > 1 > Process [1] > 5 > 1 > 5 > 1 > 1 > Number of processors : 2 > *************************************************************************** > > This is not I was really looking for. I intended to have the same output > (concerning the vector entries) and no double-printings (like "Number of > processors : 2"). > > Do you think this is an MPI setup problem, or have I not really understood > what the above code does? > > Kind regards, > > Fotios Karaoulanis._ > > ps. Congratulations on your excellent work! > > > > -- > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Fotios E. Karaoulanis > Dipl. Civil Engineer, MSc TUM > tel +30 2310 458913 > fax +30 2310 458913 > mob +30 6948 179452 > e-mail fkar at nemesis-project.org > -------------------------------------------- > Consider visiting www.nemesis-project.org. > Home of an experimental finite element code. > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > From bsmith at mcs.anl.gov Mon Apr 2 21:37:56 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 2 Apr 2007 21:37:56 -0500 (CDT) Subject: A simple question. In-Reply-To: References: <4611A6CF.6010502@nemesis-project.org> Message-ID: The double printing of > > Number of processors : 2 is do to the > > printf("Number of processors : %d\n",size); since each of the two processes is running the code they both print it, hence you get the message twice. You can user PetscPrintf(PETSC_COMM_WORLD,"Number of processors : %d\n",size); and the message will only be printed once. Barry On Mon, 2 Apr 2007, Hong Zhang wrote: > > The problem is at VecSetValues(). > > > Vec x; > > PetscInt n =10; > > PetscErrorCode ierr; > > PetscMPIInt size; > > PetscScalar one=1.0; > > double vd[3]={2.,2.,2.}; > > int indices[3]={0,5,7}; > > PetscScalar* v=&vd[0]; > > > > ierr = PetscInitialize(&argc,&args,(char *)0,help); CHKERRQ(ierr); > > ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size); CHKERRQ(ierr); > > printf("Number of processors : %d\n",size); > > ierr = VecCreate(PETSC_COMM_WORLD,&x); CHKERRQ(ierr); > > ierr = VecSetSizes(x,PETSC_DECIDE,n); CHKERRQ(ierr); > > ierr = VecSetType(x,"mpi"); CHKERRQ(ierr); > > ierr = VecSetFromOptions(x); CHKERRQ(ierr); > > ierr = VecSet(x,one); CHKERRQ(ierr); > > VecSetValues(x,3,indices,v,ADD_VALUES); > ^^^^^^^^^^ > Here, when np = 2, both processors add v=2.0 into the vector at 0,5 > 7th component, resulting different output than np=1. > You should call VecGetOwnershipRange(Vec x,PetscInt *low,PetscInt *high), > then have each processor set values into its own part. > > Hong > > > VecAssemblyBegin(x); > > VecAssemblyEnd(x); > > > > ierr = VecView(x,PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); > > VecDestroy(x); CHKERRQ(ierr); > > ierr = PetscFinalize(); CHKERRQ(ierr); > > return 0; > > } > > *************************************************************************** > > > > 3. And now I'm running it on a dual core processor for -n 1 & 2. These are > > the results. > > *************************************************************************** > > Number of processors : 1 > > Process [0] > > 3 > > 1 > > 1 > > 1 > > 1 > > 3 > > 1 > > 3 > > 1 > > 1 > > *************************************************************************** > > Number of processors : 2 > > Process [0] > > 5 > > 1 > > 1 > > 1 > > 1 > > Process [1] > > 5 > > 1 > > 5 > > 1 > > 1 > > Number of processors : 2 > > *************************************************************************** > > > > This is not I was really looking for. I intended to have the same output > > (concerning the vector entries) and no double-printings (like "Number of > > processors : 2"). > > > > Do you think this is an MPI setup problem, or have I not really understood > > what the above code does? > > > > Kind regards, > > > > Fotios Karaoulanis._ > > > > ps. Congratulations on your excellent work! > > > > > > > > -- > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Fotios E. Karaoulanis > > Dipl. Civil Engineer, MSc TUM > > tel +30 2310 458913 > > fax +30 2310 458913 > > mob +30 6948 179452 > > e-mail fkar at nemesis-project.org > > -------------------------------------------- > > Consider visiting www.nemesis-project.org. > > Home of an experimental finite element code. > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > > > > > From yaron at oak-research.com Sun Apr 8 18:22:27 2007 From: yaron at oak-research.com (yaron at oak-research.com) Date: Sun, 08 Apr 2007 15:22:27 -0800 Subject: Viewing a reordered rectangular matrix... Message-ID: <20070408232227.11134.qmail@s402.sureserver.com> I have a non-square SEQAIJ matrix, which I'd like to reorder. The reordering (Nested Dissection, in case that matters) goes through silently, but when I want to display the nonzero structure of the new matrix (with -mat_view_ordering_draw), I get a runtime error: ====================================== 0]PETSC ERROR: MatSetValues_SeqAIJ() line 141 in src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Column too large: col 905218 max 790999! [0]PETSC ERROR: MatPermute_SeqAIJ() line 1824 in src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: MatPermute() line 3158 in src/mat/interface/matrix.c [0]PETSC ERROR: MatGetOrdering() line 251 in ============================================== What gives? Yaron -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Sun Apr 8 19:49:32 2007 From: zonexo at gmail.com (Ben Tay) Date: Mon, 9 Apr 2007 08:49:32 +0800 Subject: Option with HYPRE and KSP Message-ID: <804ab5d40704081749w5c3db81fh48477e13aad65c31@mail.gmail.com> Hi, I am solving a poisson eqn and using hypre as a preconditioner has greatly increased the speed. I am trying to do some more tuning and I have some questions: 1. Is boomerAMG the only option? What are the other hypre conditioners available? Is there any documentation somewhere? Is hypre just a preconditioner or can it be a solver as well? 2. what does the preonly in ksp_type mean? Is the eqns solved? 3. why can't I use bcgs or bicg with hypre? Also, can cg be used for non-symmetric matrix. It seems to work. Thank you very much. Regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Apr 8 19:59:22 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 8 Apr 2007 19:59:22 -0500 (CDT) Subject: Option with HYPRE and KSP In-Reply-To: <804ab5d40704081749w5c3db81fh48477e13aad65c31@mail.gmail.com> References: <804ab5d40704081749w5c3db81fh48477e13aad65c31@mail.gmail.com> Message-ID: On Mon, 9 Apr 2007, Ben Tay wrote: > Hi, > > I am solving a poisson eqn and using hypre as a preconditioner has greatly > increased the speed. I am trying to do some more tuning and I have some > questions: > > 1. Is boomerAMG the only option? Yes, but all will suck on the poisson compared to boomeramg, so stick to boomer What are the other hypre conditioners > available? Is there any documentation somewhere? Run with -help and check the hypre users manual >Is hypre just a > preconditioner or can it be a solver as well? Run with -ksp_type richardson to use it as a "solver" > > 2. what does the preonly in ksp_type mean? Is the eqns solved? It means apply the preconditioner ONCE, unless the preconditioner is a direct solver it does not solve the equations. Richardson means run some number of iterations of the preconditioner and can solve the equations > > 3. why can't I use bcgs or bicg with hypre? hypre doesn't support applying the transpose of the preconditioner that bcgs and bicg need? > Also, can cg be used for > non-symmetric matrix. It seems to work. Sometimes it works, but it is dangerous and not worth risking. Barry > > Thank you very much. > > Regards. > From bsmith at mcs.anl.gov Sun Apr 8 20:11:20 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 8 Apr 2007 20:11:20 -0500 (CDT) Subject: Viewing a reordered rectangular matrix... In-Reply-To: <20070408232227.11134.qmail@s402.sureserver.com> References: <20070408232227.11134.qmail@s402.sureserver.com> Message-ID: Please run with -mat_view_info -mat_no_inode -mat_view_info_detailed and send to petsc-maint at mcs.anl.gov Barry On Sun, 8 Apr 2007, yaron at oak-research.com wrote: > I have a non-square SEQAIJ matrix, which I'd like to reorder. The > reordering (Nested Dissection, in case that matters) goes through > silently, but when I want to display the nonzero structure of the new > matrix (with -mat_view_ordering_draw), I get a runtime error: > > ====================================== > 0]PETSC ERROR: MatSetValues_SeqAIJ() line 141 in > src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: Argument out of range! > [0]PETSC ERROR: Column too large: col 905218 max 790999! > [0]PETSC ERROR: MatPermute_SeqAIJ() line 1824 in > src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: MatPermute() line 3158 in > src/mat/interface/matrix.c [0]PETSC ERROR: MatGetOrdering() line 251 in > ============================================== > > What gives? > > Yaron > From zonexo at gmail.com Sun Apr 8 20:13:55 2007 From: zonexo at gmail.com (Ben Tay) Date: Mon, 9 Apr 2007 09:13:55 +0800 Subject: Option with HYPRE and KSP In-Reply-To: References: <804ab5d40704081749w5c3db81fh48477e13aad65c31@mail.gmail.com> Message-ID: <804ab5d40704081813j6f145a5dkf9f42c4c9361b24c@mail.gmail.com> Ok thank you very much On 4/9/07, Barry Smith wrote: > > > > On Mon, 9 Apr 2007, Ben Tay wrote: > > > Hi, > > > > I am solving a poisson eqn and using hypre as a preconditioner has > greatly > > increased the speed. I am trying to do some more tuning and I have some > > questions: > > > > 1. Is boomerAMG the only option? > > Yes, but all will suck on the poisson compared to boomeramg, so stick to > boomer > > What are the other hypre conditioners > > available? Is there any documentation somewhere? > > Run with -help and check the hypre users manual > > >Is hypre just a > > preconditioner or can it be a solver as well? > > Run with -ksp_type richardson to use it as a "solver" > > > > 2. what does the preonly in ksp_type mean? Is the eqns solved? > > It means apply the preconditioner ONCE, unless the preconditioner is > a direct solver it does not solve the equations. Richardson means run > some number of iterations of the preconditioner and can solve the > equations > > > > > 3. why can't I use bcgs or bicg with hypre? > > hypre doesn't support applying the transpose of the preconditioner that > bcgs and bicg need? > > > > Also, can cg be used for > > non-symmetric matrix. It seems to work. > > Sometimes it works, but it is dangerous and not worth risking. > > Barry > > > > > Thank you very much. > > > > Regards. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From griffith at cims.nyu.edu Sun Apr 8 21:02:24 2007 From: griffith at cims.nyu.edu (Boyce Griffith) Date: Sun, 08 Apr 2007 22:02:24 -0400 Subject: Option with HYPRE and KSP In-Reply-To: References: <804ab5d40704081749w5c3db81fh48477e13aad65c31@mail.gmail.com> Message-ID: <46199EB0.7020703@cims.nyu.edu> Barry Smith wrote: > > On Mon, 9 Apr 2007, Ben Tay wrote: >> 3. why can't I use bcgs or bicg with hypre? > > hypre doesn't support applying the transpose of the preconditioner that > bcgs and bicg need? I don't believe that Bi-CGSTAB (-ksp_type bcgs) uses matrix or preconditioner transposes. On the other hand, bicg does. -- Boyce From zonexo at gmail.com Sun Apr 8 22:29:21 2007 From: zonexo at gmail.com (Ben Tay) Date: Mon, 9 Apr 2007 11:29:21 +0800 Subject: Error with using downloaded mpich and lammpi shared Message-ID: <804ab5d40704082029n32f6a054i225a007b6f32999b@mail.gmail.com> Hi, when trying to configure usign download-mpich=1, I got this error: ================================================================================= Running configure on MPICH; this may take several minutes ================================================================================= ********************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): --------------------------------------------------------------------------------------- Error running configure on MPICH: Could not execute 'cd /lsftmp/g0306332/petsc- 2.3.2-p8-new/externalpackages/mpich2-1.0.4p1;./configure --prefix=/lsftmp/g0306332/petsc-2.3.2-p8-new/externalpackages/mpich2-1.0.4p1/ia64 CC="/scratch1/g0306332/intel/cc/bin/icc -fPIC -O " CXX="g++ -Wall -Wwrite-strings -O -fPIC " F90="/scratch1/g0306332/intel/fc/bin/ifort -fPIC -O " F77="/scratch1/g0306332/intel/fc/bin/ifort -fPIC -O " --enable-sharedlibs=gcc --without-mpe --with-pm=gforker': Configuring MPICH2 version 1.0.4p1 with '--prefix=/lsftmp/g0306332/petsc- 2.3.2-p8-new/externalpackages/mpich2-1.0.4p1/ia64' 'CC=/scratch1/g0306332/intel/cc/bin/icc -fPIC -O ' 'CXX=g++ -Wall -Wwrite-strings -O -fPIC ' 'F90=/scratch1/g0306332/intel/fc/bin/ifort -fPIC -O ' 'F77=/scratch1/g0306332/intel/fc/bin/ifort -fPIC -O ' '--enable-sharedlibs=gcc' '--without-mpe' '--with-pm=gforker' 'CFLAGS=-mp -DANSI -I/app1/ia64/imsl/imsl/cnl600/itanium/include' 'F90FLAGS=-w -nologo -fpp -I/app1/ia64/imsl/CTT6.0/include/linx64_i9 -I/usr/local/mpich- 1.2.7p1/include' 'FFLAGS=-w -nologo -fpp -I/app1/ia64/imsl/CTT6.0/include/linx64_i9 -I/usr/local/mpich-1.2.7p1/include' Executing mpich2prereq in /lsftmp/g0306332/petsc-2.3.2-p8-new/externalpackages/mpich2- 1.0.4p1/src/mpid/ch3 with Executing mpich2prereq in /lsftmp/g0306332/petsc-2.3.2-p8-new /externalpackages/mpich2-1.0.4p1/src/mpid/ch3/channels/sock sourcing /lsftmp/g0306332/petsc-2.3.2-p8-new/externalpackages/mpich2- 1.0.4p1/src/pm/gforker/mpich2prereq checking for gcc... /scratch1/g0306332/intel/cc/bin/icc -fPIC -O checking for C compiler default output file name... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for suffix of executables... checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether /scratch1/g0306332/intel/cc/bin/icc -fPIC -O accepts -g... yes checking for /scratch1/g0306332/intel/cc/bin/icc -fPIC -O option to accept ANSI C... none needed checking for type of weak symbol support... pragma weak checking whether __attribute__ ((weak)) allowed... yes checking for multiple weak symbol support... yes configure: error: The values of the environment variables F77 (=/scratch1/g0306332/intel/fc/bin/ifort -fPIC -O ) and FC (=ifort) are not the same. Unset one or both and rerun configure ********************************************************************************* it seems that the F77 and FC environment variables are different. I want to use (=/scratch1/g0306332/intel/fc/bin/ifort). How can I configure them? If I use the lammpi which the sch provide, I got this error msg: ./config/configure.py --with-cc=icc --with-fc=ifort --download-f-blas-lapack=1 --download-hypre=1 --with-debugging=0 --with-mpi-include=/app1/ia64/lammpi/include/ --with-mpi-lib=/app1/ia64/lammpi/lib/libmpi.la --with-mpirun=/app1/ia64/lammpi/bin/mpirun --with-x=0 --with-shared I also replace libmpi.la with libmpi.a but both cannot work. Shared libraries cannot be built using MPI provided. Either rebuild with --with-shared=0 or rebuild MPI with shared library support What's the extension for shared libraries for ia64 platform? Is it *.so? thanks alot -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Sun Apr 8 22:30:19 2007 From: zonexo at gmail.com (Ben Tay) Date: Mon, 9 Apr 2007 11:30:19 +0800 Subject: Option with HYPRE and KSP In-Reply-To: <46199EB0.7020703@cims.nyu.edu> References: <804ab5d40704081749w5c3db81fh48477e13aad65c31@mail.gmail.com> <46199EB0.7020703@cims.nyu.edu> Message-ID: <804ab5d40704082030n3392bf3dic0938fe07f765f6e@mail.gmail.com> Ya, bcgs worked, solving my own problem. But it doesn't seem to work with the test examples like ex1f or ex2f. On 4/9/07, Boyce Griffith wrote: > > > > Barry Smith wrote: > > > > On Mon, 9 Apr 2007, Ben Tay wrote: > >> 3. why can't I use bcgs or bicg with hypre? > > > > hypre doesn't support applying the transpose of the preconditioner > that > > bcgs and bicg need? > > I don't believe that Bi-CGSTAB (-ksp_type bcgs) uses matrix or > preconditioner transposes. On the other hand, bicg does. > > -- Boyce > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Apr 8 22:52:08 2007 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 8 Apr 2007 22:52:08 -0500 Subject: Error with using downloaded mpich and lammpi shared In-Reply-To: <804ab5d40704082029n32f6a054i225a007b6f32999b@mail.gmail.com> References: <804ab5d40704082029n32f6a054i225a007b6f32999b@mail.gmail.com> Message-ID: On 4/8/07, Ben Tay wrote: > Hi, > > when trying to configure usign download-mpich=1, I got this error: > ================================================================================= > > > Running configure on MPICH; this may take several > minutes > > > ================================================================================= > > > > > > > ********************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > --------------------------------------------------------------------------------------- > Error running configure on MPICH: Could not execute 'cd > /lsftmp/g0306332/petsc- > 2.3.2-p8-new/externalpackages/mpich2-1.0.4p1;./configure > --prefix=/lsftmp/g0306332/petsc-2.3.2-p8-new/externalpackages/mpich2-1.0.4p1/ia64 > CC="/scratch1/g0306332/intel/cc/bin/icc -fPIC -O " > CXX="g++ -Wall -Wwrite-strings -O -fPIC " > F90="/scratch1/g0306332/intel/fc/bin/ifort -fPIC -O " > F77="/scratch1/g0306332/intel/fc/bin/ifort -fPIC -O " > --enable-sharedlibs=gcc --without-mpe --with-pm=gforker': > Configuring MPICH2 version 1.0.4p1 with > '--prefix=/lsftmp/g0306332/petsc-2.3.2-p8-new/externalpackages/mpich2-1.0.4p1/ia64' > 'CC=/scratch1/g0306332/intel/cc/bin/icc -fPIC -O ' > 'CXX=g++ -Wall -Wwrite-strings -O -fPIC ' > 'F90=/scratch1/g0306332/intel/fc/bin/ifort -fPIC -O ' > 'F77=/scratch1/g0306332/intel/fc/bin/ifort -fPIC -O ' > '--enable-sharedlibs=gcc' '--without-mpe' '--with-pm=gforker' 'CFLAGS=-mp > -DANSI -I/app1/ia64/imsl/imsl/cnl600/itanium/include' > 'F90FLAGS=-w -nologo -fpp > -I/app1/ia64/imsl/CTT6.0/include/linx64_i9 > -I/usr/local/mpich- 1.2.7p1/include' 'FFLAGS=-w -nologo -fpp > -I/app1/ia64/imsl/CTT6.0/include/linx64_i9 > -I/usr/local/mpich-1.2.7p1/include' > Executing mpich2prereq in > /lsftmp/g0306332/petsc-2.3.2-p8-new/externalpackages/mpich2- > 1.0.4p1/src/mpid/ch3 with > Executing mpich2prereq in > /lsftmp/g0306332/petsc-2.3.2-p8-new/externalpackages/mpich2-1.0.4p1/src/mpid/ch3/channels/sock > sourcing /lsftmp/g0306332/petsc-2.3.2-p8-new/externalpackages/mpich2- > 1.0.4p1/src/pm/gforker/mpich2prereq > checking for gcc... /scratch1/g0306332/intel/cc/bin/icc > -fPIC -O > checking for C compiler default output file name... a.out > checking whether the C compiler works... yes > checking whether we are cross compiling... no > checking for suffix of executables... > checking for suffix of object files... o > checking whether we are using the GNU C compiler... yes > checking whether /scratch1/g0306332/intel/cc/bin/icc -fPIC > -O accepts -g... yes > checking for /scratch1/g0306332/intel/cc/bin/icc -fPIC -O > option to accept ANSI C... none needed > checking for type of weak symbol support... pragma weak > checking whether __attribute__ ((weak)) allowed... yes > checking for multiple weak symbol support... yes > configure: error: The values of the environment variables F77 > (=/scratch1/g0306332/intel/fc/bin/ifort -fPIC -O ) and FC > (=ifort) are not the same. Unset one or both and rerun configure > ********************************************************************************* > > it seems that the F77 and FC environment variables are different. I want to > use (=/scratch1/g0306332/intel/fc/bin/ifort). How can I > configure them? > Do not set any environment variables. Let configure control it. > If I use the lammpi which the sch provide, I got this error msg: > ./config/configure.py --with-cc=icc --with-fc=ifort > --download-f-blas-lapack=1 --download-hypre=1 --with-debugging=0 > --with-mpi-include=/app1/ia64/lammpi/include/ > --with-mpi-lib=/app1/ia64/lammpi/lib/libmpi.la > --with-mpirun=/app1/ia64/lammpi/bin/mpirun --with-x=0 > --with-shared > > I also replace libmpi.la with libmpi.a but both cannot work. > > > Shared libraries cannot be built using MPI provided. > Either rebuild with --with-shared=0 or rebuild MPI with shared library > support > > What's the extension for shared libraries for ia64 platform? Is it *.so? I am fairly sure since it is 64-bit that the shared library is either not PIC, or we cannot build shared libraries for this system. Matt > thanks alot -- The government saving money is like me spilling beer. It happens, but never on purpose. From zonexo at gmail.com Sun Apr 8 23:19:38 2007 From: zonexo at gmail.com (Ben Tay) Date: Mon, 9 Apr 2007 12:19:38 +0800 Subject: Error with using downloaded mpich and lammpi shared In-Reply-To: References: <804ab5d40704082029n32f6a054i225a007b6f32999b@mail.gmail.com> Message-ID: <804ab5d40704082119k662f82aendc4ba5c709b324a7@mail.gmail.com> Hi, I tried using shared=0 and got this when I used lammpi: --with-mpi-lib=['/app1/ia64/lammpi/lib/libmpi.la'] and --with-mpi-include=/app1/ia64/lammpi/include/ did not work same for libmpi.a. Did I specify the wrong files? thanks. On 4/9/07, Matthew Knepley wrote: > > On 4/8/07, Ben Tay wrote: > > Hi, > > > > when trying to configure usign download-mpich=1, I got this error: > > > ================================================================================= > > > > > > Running configure on MPICH; this may take several > > minutes > > > > > > > ================================================================================= > > > > > > > > > > > > > > > ********************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.logfor > > details): > > > --------------------------------------------------------------------------------------- > > Error running configure on MPICH: Could not execute 'cd > > /lsftmp/g0306332/petsc- > > 2.3.2-p8-new/externalpackages/mpich2-1.0.4p1;./configure > > --prefix=/lsftmp/g0306332/petsc-2.3.2-p8-new/externalpackages/mpich2- > 1.0.4p1/ia64 > > CC="/scratch1/g0306332/intel/cc/bin/icc -fPIC -O " > > CXX="g++ -Wall -Wwrite-strings -O -fPIC " > > F90="/scratch1/g0306332/intel/fc/bin/ifort -fPIC -O " > > F77="/scratch1/g0306332/intel/fc/bin/ifort -fPIC -O " > > --enable-sharedlibs=gcc --without-mpe --with-pm=gforker': > > Configuring MPICH2 version 1.0.4p1 with > > '--prefix=/lsftmp/g0306332/petsc-2.3.2-p8-new/externalpackages/mpich2- > 1.0.4p1/ia64' > > 'CC=/scratch1/g0306332/intel/cc/bin/icc -fPIC -O ' > > 'CXX=g++ -Wall -Wwrite-strings -O -fPIC ' > > 'F90=/scratch1/g0306332/intel/fc/bin/ifort -fPIC -O ' > > 'F77=/scratch1/g0306332/intel/fc/bin/ifort -fPIC -O ' > > '--enable-sharedlibs=gcc' '--without-mpe' '--with-pm=gforker' > 'CFLAGS=-mp > > -DANSI -I/app1/ia64/imsl/imsl/cnl600/itanium/include' > > 'F90FLAGS=-w -nologo -fpp > > -I/app1/ia64/imsl/CTT6.0/include/linx64_i9 > > -I/usr/local/mpich- 1.2.7p1/include' 'FFLAGS=-w -nologo -fpp > > -I/app1/ia64/imsl/CTT6.0/include/linx64_i9 > > -I/usr/local/mpich-1.2.7p1/include' > > Executing mpich2prereq in > > /lsftmp/g0306332/petsc-2.3.2-p8-new/externalpackages/mpich2- > > 1.0.4p1/src/mpid/ch3 with > > Executing mpich2prereq in > > /lsftmp/g0306332/petsc-2.3.2-p8-new/externalpackages/mpich2-1.0.4p1 > /src/mpid/ch3/channels/sock > > sourcing /lsftmp/g0306332/petsc-2.3.2-p8-new/externalpackages/mpich2- > > 1.0.4p1/src/pm/gforker/mpich2prereq > > checking for gcc... /scratch1/g0306332/intel/cc/bin/icc > > -fPIC -O > > checking for C compiler default output file name... a.out > > checking whether the C compiler works... yes > > checking whether we are cross compiling... no > > checking for suffix of executables... > > checking for suffix of object files... o > > checking whether we are using the GNU C compiler... yes > > checking whether /scratch1/g0306332/intel/cc/bin/icc -fPIC > > -O accepts -g... yes > > checking for /scratch1/g0306332/intel/cc/bin/icc -fPIC -O > > option to accept ANSI C... none needed > > checking for type of weak symbol support... pragma weak > > checking whether __attribute__ ((weak)) allowed... yes > > checking for multiple weak symbol support... yes > > configure: error: The values of the environment variables F77 > > (=/scratch1/g0306332/intel/fc/bin/ifort -fPIC -O ) and FC > > (=ifort) are not the same. Unset one or both and rerun configure > > > ********************************************************************************* > > > > it seems that the F77 and FC environment variables are different. I > want to > > use (=/scratch1/g0306332/intel/fc/bin/ifort). How can I > > configure them? > > > > Do not set any environment variables. Let configure control it. > > > If I use the lammpi which the sch provide, I got this error msg: > > ./config/configure.py --with-cc=icc --with-fc=ifort > > --download-f-blas-lapack=1 --download-hypre=1 --with-debugging=0 > > --with-mpi-include=/app1/ia64/lammpi/include/ > > --with-mpi-lib=/app1/ia64/lammpi/lib/libmpi.la > > --with-mpirun=/app1/ia64/lammpi/bin/mpirun --with-x=0 > > --with-shared > > > > I also replace libmpi.la with libmpi.a but both cannot work. > > > > > > Shared libraries cannot be built using MPI provided. > > Either rebuild with --with-shared=0 or rebuild MPI with shared library > > support > > > > What's the extension for shared libraries for ia64 platform? Is it *.so? > > I am fairly sure since it is 64-bit that the shared library is either > not PIC, or > we cannot build shared libraries for this system. > > Matt > > > thanks alot > -- > The government saving money is like me spilling beer. It happens, but > never on purpose. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Apr 8 23:28:58 2007 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 8 Apr 2007 23:28:58 -0500 Subject: Error with using downloaded mpich and lammpi shared In-Reply-To: <804ab5d40704082119k662f82aendc4ba5c709b324a7@mail.gmail.com> References: <804ab5d40704082029n32f6a054i225a007b6f32999b@mail.gmail.com> <804ab5d40704082119k662f82aendc4ba5c709b324a7@mail.gmail.com> Message-ID: On 4/8/07, Ben Tay wrote: > Hi, > > I tried using shared=0 and got this when I used lammpi: > > --with-mpi-lib=['/app1/ia64/lammpi/lib/libmpi.la'] and > --with-mpi-include=/app1/ia64/lammpi/include/ did not work > > same for libmpi.a. Did I specify the wrong files? 1) Send this to petsc-maint at mcs.anl.gov 2) I cannot tell anything without the configure.log Matt > thanks. > > > > On 4/9/07, Matthew Knepley wrote: > > On 4/8/07, Ben Tay wrote: > > > Hi, > > > > > > when trying to configure usign download-mpich=1, I got this error: > > > > ================================================================================= > > > > > > > > > Running configure on MPICH; this may take several > > > minutes > > > > > > > > > > ================================================================================= > > > > > > > > > > > > > > > > > > > > > > ********************************************************************************* > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > for > > > details): > > > > --------------------------------------------------------------------------------------- > > > Error running configure on MPICH: Could not execute 'cd > > > /lsftmp/g0306332/petsc- > > > 2.3.2-p8-new/externalpackages/mpich2-1.0.4p1;./configure > > > > --prefix=/lsftmp/g0306332/petsc-2.3.2-p8-new/externalpackages/mpich2-1.0.4p1/ia64 > > > CC="/scratch1/g0306332/intel/cc/bin/icc -fPIC -O " > > > CXX="g++ -Wall -Wwrite-strings -O -fPIC " > > > F90="/scratch1/g0306332/intel/fc/bin/ifort -fPIC -O " > > > F77="/scratch1/g0306332/intel/fc/bin/ifort -fPIC -O " > > > --enable-sharedlibs=gcc --without-mpe --with-pm=gforker': > > > Configuring MPICH2 version 1.0.4p1 with > > > > '--prefix=/lsftmp/g0306332/petsc-2.3.2-p8-new/externalpackages/mpich2-1.0.4p1/ia64' > > > 'CC=/scratch1/g0306332/intel/cc/bin/icc -fPIC -O ' > > > 'CXX=g++ -Wall -Wwrite-strings -O -fPIC ' > > > 'F90=/scratch1/g0306332/intel/fc/bin/ifort -fPIC -O ' > > > 'F77=/scratch1/g0306332/intel/fc/bin/ifort -fPIC -O ' > > > '--enable-sharedlibs=gcc' '--without-mpe' '--with-pm=gforker' > 'CFLAGS=-mp > > > -DANSI -I/app1/ia64/imsl/imsl/cnl600/itanium/include' > > > 'F90FLAGS=-w -nologo -fpp > > > -I/app1/ia64/imsl/CTT6.0/include/linx64_i9 > > > -I/usr/local/mpich- 1.2.7p1/include' 'FFLAGS=-w -nologo -fpp > > > -I/app1/ia64/imsl/CTT6.0/include/linx64_i9 > > > -I/usr/local/mpich-1.2.7p1/include' > > > Executing mpich2prereq in > > > /lsftmp/g0306332/petsc-2.3.2-p8-new/externalpackages/mpich2- > > > 1.0.4p1/src/mpid/ch3 with > > > Executing mpich2prereq in > > > > /lsftmp/g0306332/petsc-2.3.2-p8-new/externalpackages/mpich2-1.0.4p1/src/mpid/ch3/channels/sock > > > sourcing /lsftmp/g0306332/petsc-2.3.2-p8-new/externalpackages/mpich2- > > > 1.0.4p1/src/pm/gforker/mpich2prereq > > > checking for gcc... /scratch1/g0306332/intel/cc/bin/icc > > > -fPIC -O > > > checking for C compiler default output file name... a.out > > > checking whether the C compiler works... yes > > > checking whether we are cross compiling... no > > > checking for suffix of executables... > > > checking for suffix of object files... o > > > checking whether we are using the GNU C compiler... yes > > > checking whether /scratch1/g0306332/intel/cc/bin/icc > -fPIC > > > -O accepts -g... yes > > > checking for /scratch1/g0306332/intel/cc/bin/icc -fPIC > -O > > > option to accept ANSI C... none needed > > > checking for type of weak symbol support... pragma weak > > > checking whether __attribute__ ((weak)) allowed... yes > > > checking for multiple weak symbol support... yes > > > configure: error: The values of the environment variables F77 > > > (=/scratch1/g0306332/intel/fc/bin/ifort -fPIC -O ) > and FC > > > (=ifort) are not the same. Unset one or both and rerun configure > > > > ********************************************************************************* > > > > > > it seems that the F77 and FC environment variables are different. I > want to > > > use (=/scratch1/g0306332/intel/fc/bin/ifort). How can I > > > configure them? > > > > > > > Do not set any environment variables. Let configure control it. > > > > > If I use the lammpi which the sch provide, I got this error msg: > > > ./config/configure.py --with-cc=icc --with-fc=ifort > > > --download-f-blas-lapack=1 --download-hypre=1 --with-debugging=0 > > > --with-mpi-include=/app1/ia64/lammpi/include/ > > > --with-mpi-lib=/app1/ia64/lammpi/lib/libmpi.la > > > --with-mpirun=/app1/ia64/lammpi/bin/mpirun --with-x=0 > > > --with-shared > > > > > > I also replace libmpi.la with libmpi.a but both cannot work. > > > > > > > > > Shared libraries cannot be built using MPI provided. > > > Either rebuild with --with-shared=0 or rebuild MPI with shared library > > > support > > > > > > What's the extension for shared libraries for ia64 platform? Is it *.so? > > > > I am fairly sure since it is 64-bit that the shared library is either > > not PIC, or > > we cannot build shared libraries for this system. > > > > Matt > > > > > thanks alot > > -- > > The government saving money is like me spilling beer. It happens, but > > never on purpose. > > > > > > -- The government saving money is like me spilling beer. It happens, but never on purpose. From randy at geosystem.us Tue Apr 10 15:05:05 2007 From: randy at geosystem.us (Randall Mackie) Date: Tue, 10 Apr 2007 13:05:05 -0700 Subject: how to clear number of iterations Message-ID: <461BEDF1.5020700@geosystem.us> I have a situation where I am calling a KSP to solve various systems of equations (same matrix, different right hand sides). Once in a while, one of the solutions does not require any iterations, ie, it's starting solution that I set is correct. The problem is that when I call KSPGetIterationNumber, it has retained the number of iterations from the previous solve. Is there some call I can make that would clear this or set it to zero in between KSP calls? That way, I can print out the true number of iterations for each solve. Thanks, Randy -- Randall Mackie GSY-USA, Inc. PMB# 643 2261 Market St., San Francisco, CA 94114-1600 Tel (415) 469-8649 Fax (415) 469-5044 California Registered Geophysicist License No. GP 1034 From mafunk at nmsu.edu Tue Apr 10 17:23:16 2007 From: mafunk at nmsu.edu (Matt Funk) Date: Tue, 10 Apr 2007 16:23:16 -0600 Subject: petsc c and c++ In-Reply-To: <461BEDF1.5020700@geosystem.us> References: <461BEDF1.5020700@geosystem.us> Message-ID: <200704101623.18704.mafunk@nmsu.edu> Is there a performance difference between CLanguage: C and CLanguage: Cxx using gnu compilers as far as anyone knows? thanks mat From knutert at stud.ntnu.no Wed Apr 11 03:52:05 2007 From: knutert at stud.ntnu.no (Knut Erik Teigen) Date: Wed, 11 Apr 2007 10:52:05 +0200 Subject: Process interrupted error In-Reply-To: References: <4611A6CF.6010502@nemesis-project.org> Message-ID: <1176281525.7365.16.camel@iept0415.ivt.ntnu.no> Hello, I get the following error when trying to copy a solution vector to process zero using VecScatterCreateToZero: [0] VecScatterCreate(): Special case: processor zero gets entire parallel vector, rest get none forrtl: error (69): process interrupted (SIGINT) forrtl: error (69): process interrupted (SIGINT) When running with one or two processes, the code runs fine, but with three or more, the above error occurs, with one "process interrupted" error for each process minus one. Could someone help me figure out what's wrong? Regards, Knut Erik Teigen From bsmith at mcs.anl.gov Wed Apr 11 06:55:10 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 11 Apr 2007 06:55:10 -0500 (CDT) Subject: Process interrupted error In-Reply-To: <1176281525.7365.16.camel@iept0415.ivt.ntnu.no> References: <4611A6CF.6010502@nemesis-project.org> <1176281525.7365.16.camel@iept0415.ivt.ntnu.no> Message-ID: Something is sending an interupt signal to all the proccess except one. We've seen this happen where the "batch or node scheduler does this to kill a long running job". Does it happen even if the parallel vector is real short? Does the machine seem to hang at that point or does the sigint come immediately? Can you us the runtime option -start_in_debugger or totalview to catch the signal? Barry On Wed, 11 Apr 2007, Knut Erik Teigen wrote: > Hello, > > I get the following error when trying to copy a solution vector to > process zero using VecScatterCreateToZero: > > [0] VecScatterCreate(): Special case: processor zero gets entire > parallel vector, rest get none > forrtl: error (69): process interrupted (SIGINT) > forrtl: error (69): process interrupted (SIGINT) > > When running with one or two processes, the code runs fine, but with > three or more, the above error occurs, with one "process interrupted" > error for each process minus one. Could someone help me figure out > what's wrong? > > Regards, > Knut Erik Teigen > > > > From Stephen.R.Ball at awe.co.uk Wed Apr 11 10:39:13 2007 From: Stephen.R.Ball at awe.co.uk (Stephen R Ball) Date: Wed, 11 Apr 2007 16:39:13 +0100 Subject: Using ML via PETSc Message-ID: <74BGj5002963@awe.co.uk> Hi I am looking to have a go at using Sandia National Laboratory's ML preconditioning package via PETSc. I have looked in your python script for ML and the download link refers to v4.0. However, looking at your ftp download site I can see that v5.0 is included. Can you tell me if it is to safe to use the v5.0 version with PETSc v2.3.2 or should I use v4.0. Also, can you tell me if there are any examples of using ML via PETSc. Regards Stephen -- _______________________________________________________________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR From knepley at gmail.com Wed Apr 11 11:38:06 2007 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 11 Apr 2007 11:38:06 -0500 Subject: Using ML via PETSc In-Reply-To: <74BGj5002963@awe.co.uk> References: <74BGj5002963@awe.co.uk> Message-ID: On 4/11/07, Stephen R Ball wrote: > > Hi > > I am looking to have a go at using Sandia National Laboratory's ML > preconditioning package via PETSc. I have looked in your python script > for ML and the download link refers to v4.0. However, looking at your > ftp download site I can see that v5.0 is included. Can you tell me if it > is to safe to use the v5.0 version with PETSc v2.3.2 or should I use > v4.0. I think you should use 4.0 with 2.3.2. 5.0 works with the dev version. > Also, can you tell me if there are any examples of using ML via PETSc. You should be able to use it on any of the PETSc examples (like KSP ex2) using -pc_type ml. Matt > Regards > > Stephen > -- > _______________________________________________________________________________ > > The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. > > AWE Plc > Registered in England and Wales > Registration No 02763902 > AWE, Aldermaston, Reading, RG7 4PR > > -- The government saving money is like me spilling beer. It happens, but never on purpose. From balay at mcs.anl.gov Wed Apr 11 11:42:20 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 11 Apr 2007 11:42:20 -0500 (CDT) Subject: Using ML via PETSc In-Reply-To: <74BGj5002963@awe.co.uk> References: <74BGj5002963@awe.co.uk> Message-ID: On Wed, 11 Apr 2007, Stephen R Ball wrote: > I am looking to have a go at using Sandia National Laboratory's ML > preconditioning package via PETSc. I have looked in your python script > for ML and the download link refers to v4.0. However, looking at your > ftp download site I can see that v5.0 is included. Can you tell me if it > is to safe to use the v5.0 version with PETSc v2.3.2 or should I use > v4.0. You might just need the following patch to get ML5.0 working with petsc-2.3.2 http://petsc.cs.iit.edu/petsc/petsc-dev/diff/882b30353a21/src/ksp/pc/impls/ml/ml.c > > Also, can you tell me if there are any examples of using ML via PETSc. Any ksp example with '-pc_type ml' shold work... Satish From hzhang at mcs.anl.gov Wed Apr 11 12:44:04 2007 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 11 Apr 2007 12:44:04 -0500 (CDT) Subject: Using ML via PETSc In-Reply-To: References: <74BGj5002963@awe.co.uk> Message-ID: Stephen, > > > I am looking to have a go at using Sandia National Laboratory's ML > > preconditioning package via PETSc. I have looked in your python script > > for ML and the download link refers to v4.0. However, looking at your > > ftp download site I can see that v5.0 is included. Can you tell me if it > > is to safe to use the v5.0 version with PETSc v2.3.2 or should I use > > v4.0. > > You might just need the following patch to get ML5.0 working with > petsc-2.3.2 > > http://petsc.cs.iit.edu/petsc/petsc-dev/diff/882b30353a21/src/ksp/pc/impls/ml/ml.c > Petsc-dev is compatible with ML5.0. petsc-2.3.2 is interfaced with ML4.0 config petsc with '--download-ml=1' gives you compatible ML. > > > > Also, can you tell me if there are any examples of using ML via PETSc. > > Any ksp example with '-pc_type ml' shold work... Not necessarily. ML requires input of fine-grid matrix, then it determines nlevels; while petsc let user define nlevels. See petsc-2.3.2/src/ksp/ksp/examples/tests/ex26.c (run with -pc_type ml -ksp_view) or petsc-dev/src/ksp/ksp/examples/tests/ex29.c Hong > From knutert at stud.ntnu.no Thu Apr 12 02:20:42 2007 From: knutert at stud.ntnu.no (Knut Erik Teigen) Date: Thu, 12 Apr 2007 09:20:42 +0200 Subject: Process interrupted error In-Reply-To: References: <4611A6CF.6010502@nemesis-project.org> <1176281525.7365.16.camel@iept0415.ivt.ntnu.no> Message-ID: <1176362443.7365.28.camel@iept0415.ivt.ntnu.no> On Wed, 2007-04-11 at 06:55 -0500, Barry Smith wrote: > Something is sending an interupt signal to all the proccess > except one. We've seen this happen where the "batch or node > scheduler does this to kill a long running job". > > Does it happen even if the parallel vector is real short? Yes, even with a vector of size 1, this happens. > > Does the machine seem to hang at that point or does the sigint > come immediately? It hangs for a short period, and then displays the sigint error. When running with 2 processes, it also hangs a bit, but then continues with the correct results. > > Can you us the runtime option -start_in_debugger or totalview > to catch the signal? Something really weird happens here. The first time I run the program with -start_in_debugger noxterm,idb , it finishes correctly, no matter the number of processes or problem size. The second time, however, I get the sigint errors. When I run with idb I get: (idb) Program exited normally. (idb) Program received signal SIGINT With gdb I get: 0xffffe410 in __kernel_vsyscall () and then the program hangs. However I get this message in the beginning with both: [3]PETSC ERROR: PETSC: Attaching gdb to ./out of pid 30549 on ivt0415 -Knut Erik- > > Barry > > On Wed, 11 Apr 2007, Knut Erik Teigen wrote: > > > Hello, > > > > I get the following error when trying to copy a solution vector to > > process zero using VecScatterCreateToZero: > > > > [0] VecScatterCreate(): Special case: processor zero gets entire > > parallel vector, rest get none > > forrtl: error (69): process interrupted (SIGINT) > > forrtl: error (69): process interrupted (SIGINT) > > > > When running with one or two processes, the code runs fine, but with > > three or more, the above error occurs, with one "process interrupted" > > error for each process minus one. Could someone help me figure out > > what's wrong? > > > > Regards, > > Knut Erik Teigen > > > > > > > > > > From nicolas.bathfield at chalmers.se Thu Apr 12 08:00:52 2007 From: nicolas.bathfield at chalmers.se (Nicolas Bathfield) Date: Thu, 12 Apr 2007 15:00:52 +0200 (CEST) Subject: petsc: bommeramg for system of PDE's Message-ID: <11590.193.183.3.2.1176382852.squirrel@webmail.chalmers.se> Dear PETSc users, How can I force boomeramg to solve for systems of PDE's (multiple variables)? The Hypre manual is unclear about this, and I could not find the option in PETSc that would allow to do it. Here is what I found in the manual from Hypre: "If the users wants to solve systems of PDEs and can provide information on which variables belong to which function, BoomerAMG?s systems AMG version can also be used." (paragraph 6.3 page 52). Thanks for your help. Best regards, Nicolas Bathfield. -- Nicolas BATHFIELD Chalmers University of Technology Shipping and Marine Technology phone: +46 (0)31 772 1476 fax: +46 (0)31 772 3699 From zonexo at gmail.com Thu Apr 12 08:09:55 2007 From: zonexo at gmail.com (Ben Tay) Date: Thu, 12 Apr 2007 21:09:55 +0800 Subject: ld: skipping incompatible Message-ID: <804ab5d40704120609s1008e612n8d761ba0de3b2d99@mail.gmail.com> Hi, i've just compiled PETSc on a 64bit system. No error was shown. However, when using make ex1f (or other similar compilation), I got this error msg: ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libnsl.so when searching for -lnsl ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libnsl.a when searching for -lnsl ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../librt.so when searching for -lrt ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../librt.a when searching for -lrt ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libdl.so when searching for -ldl ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libdl.a when searching for -ldl ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libm.so when searching for -lm ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libm.a when searching for -lm ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libm.so when searching for -lm ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libm.a when searching for -lm ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libdl.so when searching for -ldl ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libdl.a when searching for -ldl ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libdl.so when searching for -ldl ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libdl.a when searching for -ldl ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libm.so when searching for -lm ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libm.a when searching for -lm ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libc.so when searching for -lc ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libc.a when searching for -lc ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libdl.so when searching for -ldl ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libdl.a when searching for -ldl ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libc.so when searching for -lc ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libc.a when searching for -lc The test file worked fine, same as some of my test codes. But i'm not sure if there 'll be problems in the future. Is this a serious problem? How can I fix it. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Apr 12 08:13:00 2007 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 12 Apr 2007 08:13:00 -0500 Subject: ld: skipping incompatible In-Reply-To: <804ab5d40704120609s1008e612n8d761ba0de3b2d99@mail.gmail.com> References: <804ab5d40704120609s1008e612n8d761ba0de3b2d99@mail.gmail.com> Message-ID: It is not using these 32-bit libraries that someone, probably your compiler for some reason, reported to configure. If everything runs, you are fine. Matt On 4/12/07, Ben Tay wrote: > Hi, > > i've just compiled PETSc on a 64bit system. No error was shown. However, > when using make ex1f (or other similar compilation), I got this error msg: > > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libnsl.so > when searching for -lnsl > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libnsl.a > when searching for -lnsl > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../librt.so > when searching for -lrt > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../librt.a > when searching for -lrt > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libdl.so > when searching for -ldl > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libdl.a > when searching for -ldl > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libm.so > when searching for -lm > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libm.a when > searching for -lm > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libm.so > when searching for -lm > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libm.a when > searching for -lm > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libdl.so > when searching for -ldl > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libdl.a > when searching for -ldl > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libdl.so > when searching for -ldl > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libdl.a > when searching for -ldl > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libm.so > when searching for -lm > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libm.a when > searching for -lm > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libc.so > when searching for -lc > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libc.a when > searching for -lc > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libdl.so > when searching for -ldl > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libdl.a > when searching for -ldl > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libc.so > when searching for -lc > ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../libc.a when > searching for -lc > > The test file worked fine, same as some of my test codes. But i'm not sure > if there 'll be problems in the future. > > Is this a serious problem? How can I fix it. Thanks. -- The government saving money is like me spilling beer. It happens, but never on purpose. From laurent.nguyen at idris.fr Thu Apr 12 09:49:25 2007 From: laurent.nguyen at idris.fr (Laurent Nguyen) Date: Thu, 12 Apr 2007 16:49:25 +0200 Subject: Question about memory used by PetsC Objects Message-ID: <461E46F5.90606@idris.fr> Hi, I'm a Petsc user, but I'm beginning to use petsc in a supercomputation context. So, I'm trying parallel versions of my old mono-processed programs. But I've some difficulties to determine how much memory my program use. (I give a little example). In this example on one processor (with these command line arguments: -mat_type mpiaij -pc_type none -log_summary), I created one mpiaij matrix (size = 500*500) and two vectors (size = 500). I've this information from output: Matrix Object: type=mpiaij, rows=250000, cols=250000 total: nonzeros=749996, allocated nonzeros=1750000 To me, there is an array of 1750000 double precision number of 8 bytes = 14000000 bytes (14Mo). Same for vectors: 2*8*250000 = 4 Mo But in the log summary, I've: Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. --- Event Stage 0: Main Stage Viewer 1 1 0 0 Index Set 2 2 672 0 Map 16 16 4992 0 Vec 10 10 2006576 0 Vec Scatter 1 1 0 0 Matrix 3 3 1068 0 Krylov Solver 1 1 17208 0 Preconditioner 1 1 0 0 ============================================================= I don't understand this output...If you have some documentation about how Petsc deals with memory or if you can explain, I would be very interested. This is a problem because I'm working on IBM SP4, and I've limitations of memory (700mb for data and 700mb for stack). I think the objects created by Petsc is created in the stack memory, but as I cannot determine exactly the memory usage, I cannot verify. Thank you for your advices, Best regards -- ************************************** NGUYEN Anh-Khai Laurent Equipe Support Utilisateur Email : laurent.nguyen at idris.fr T?l : 01.69.35.85.66 Adresse : IDRIS - Institut du D?veloppement et des Ressources en Informatique Scientifique CNRS Batiment 506 BP 167 F - 91403 ORSAY Cedex Site Web : http://www.idris.fr ************************************** -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: poisson.F90 URL: From jordi.poblet at gmail.com Thu Apr 12 11:51:22 2007 From: jordi.poblet at gmail.com (jordi poblet) Date: Thu, 12 Apr 2007 18:51:22 +0200 Subject: memory GMRES preconditioner Message-ID: Dear all, I think that the GMRES ILU(p) preconditioner in my program is consuming too much memory. I just would like to check if it is correct or not and compare this experience with other PETSc users. For a quite small sparse matrix (dimension = 3500 and 2.5 Mb) the memory consumed while the preconditioner is created is more or less: ILU level approximate increase of memory waste (Mb) 4 9 8 26 12 43 16 53 20 68 Moreover a fast convergence is reached suddenly (i.e. for ILU level 25 convergence is not reached after 3000 iterations but for ILU level 30 the convergence is reached in 7 iterations). I suppose that some of the problems are caused by the large condition number of the matrix but in any case I guess that the memory used by the preconditioner is too large. I have tried to improve the performance using: PCFactorSetFill, PCFactorSetAllowDiagonalFill and PCFactorSetUseDropTolerance but I have not succeed. I would also like to know how to estimate a priori the memory required for a PETSc sparse matrix if the non null entries are known. Is the following rule: 1 double/complex and 2 integers per non null coefficient, a good approximation (taking into account PETSc implementation)? Is the same rule valid for the case of the preconditioner? I will give some details on the particular situation being solved: ------------------------------------------------------------------------------------------------------------ Type of matrix: FEM resolution of an structural dynamic problem (K + w*w*M) | L^T ------------------------- = System matrix L | 0 K == stiffness matrix (square banded) M == consistent mass matrix (square banded, and same nonzero pattern than K) w == pulsation of the problem (scalar) L == Lagrange multipliers matrix (rectangular sparse) The matrix is in general ill conditioned and non positive definite. ------------------------------------------------------------------------------------------------------------ And my use of PETSc functions is as follows: ierr = KSPCreate(PETSC_COMM_WORLD,&MyKsp);CHKERRQ(ierr); ierr = KSPSetType(MyKsp, KSPGMRES);CHKERRQ(ierr); ierr = KSPSetOperators(MyKsp,MyMat,MyMat,DIFFERENT_NONZERO_PATTERN); CHKERRQ(ierr); ierr = KSPGetPC(MyKsp,&(MyPc));CHKERRQ(ierr); ierr = PCSetType(MyPc,PCILU);CHKERRQ(ierr); ierr = PCFactorSetLevels(MyPc, LevelNumber);CHKERRQ(ierr); PetscReal realshift = 1.0; ierr = PCFactorSetShiftNonzero(MyPc, realshift); CHKERRQ(ierr); //With and without this line ierr = KSPSetTolerances(MyKsp,tol,PETSC_DEFAULT,PETSC_DEFAULT,itmax);CHKERRQ(ierr); ierr = KSPSetInitialGuessNonzero(MyKsp,PETSC_TRUE);CHKERRQ(ierr); ierr = KSPGMRESSetRestart(MyKsp, max_steps_restart); CHKERRQ(ierr); //With and without this line ierr = KSPSetFromOptions(MyKsp);CHKERRQ(ierr); ierr = KSPSolve(MyKsp,MyVector,x);CHKERRQ(ierr); ----------------------------------------------------------------------------------------------------------- Thank you very much in advance, Jordi Poblet Puig -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Apr 12 12:11:56 2007 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 12 Apr 2007 12:11:56 -0500 Subject: memory GMRES preconditioner In-Reply-To: References: Message-ID: On 4/12/07, jordi poblet wrote: > > > Dear all, > > I think that the GMRES ILU(p) preconditioner in my program is consuming too > much memory. I just would like to check if it is correct or not and compare > this experience with other PETSc users. > For a quite small sparse matrix (dimension = 3500 and 2.5 Mb) the memory > consumed while the preconditioner is created is more or less: This memory usage does not really sound outrageous to me. The full matrix would cost 3500*4 + 3500*3500*(8+4) = 147M. Using so many levels has to be getting close to a full matrix. In general, high levels of fill (I would say > 2) are counterproductive with ILU. For such as small problem, consider using a sparse direct solver like MUMPS. YOu can get this automatically by configureing with --download-mumps. Until your problems are at least 100K, I would say that this is one of the best options. Matt > ILU level approximate > increase of memory waste (Mb) > > 4 9 > 8 26 > 12 43 > 16 53 > 20 68 > > > Moreover a fast convergence is reached suddenly (i.e. for ILU level 25 > convergence is not reached after 3000 iterations but for ILU level 30 the > convergence is reached in 7 iterations). > I suppose that some of the problems are caused by the large condition > number of the matrix but in any case I guess that the memory used by the > preconditioner is too large. > I have tried to improve the performance using: PCFactorSetFill, > PCFactorSetAllowDiagonalFill and PCFactorSetUseDropTolerance but I have not > succeed. > > > > I would also like to know how to estimate a priori the memory required for a > PETSc sparse matrix if the non null entries are known. Is the following > rule: 1 double/complex and 2 integers per non null coefficient, a > good approximation (taking into account PETSc implementation)? Is the same > rule valid for the case of the preconditioner? > > I will give some details on the particular situation being solved: > ------------------------------------------------------------------------------------------------------------ > > Type of matrix: FEM resolution of an structural dynamic problem > > (K + w*w*M) | L^T > > ------------------------- = System matrix > > L | 0 > > > > > K == stiffness matrix (square banded) > M == consistent mass matrix (square banded, and same nonzero pattern than > K) > w == pulsation of the problem (scalar) > L == Lagrange multipliers matrix (rectangular sparse) > > > The matrix is in general ill conditioned and non positive definite. > > ------------------------------------------------------------------------------------------------------------ > > > > And my use of PETSc functions is as follows: > > ierr = KSPCreate(PETSC_COMM_WORLD,&MyKsp);CHKERRQ(ierr); > > ierr = KSPSetType(MyKsp, KSPGMRES);CHKERRQ(ierr); > > ierr = > KSPSetOperators(MyKsp,MyMat,MyMat,DIFFERENT_NONZERO_PATTERN); > CHKERRQ(ierr); > > > > > ierr = KSPGetPC(MyKsp,&(MyPc));CHKERRQ(ierr); > > > > > > ierr = PCSetType(MyPc,PCILU);CHKERRQ(ierr); > > ierr = PCFactorSetLevels(MyPc, LevelNumber);CHKERRQ(ierr); > > > > > PetscReal realshift = 1.0; > > ierr = PCFactorSetShiftNonzero(MyPc, realshift); CHKERRQ(ierr); //With and > without this line > > > > > ierr = > KSPSetTolerances(MyKsp,tol,PETSC_DEFAULT,PETSC_DEFAULT,itmax);CHKERRQ(ierr); > > ierr = > KSPSetInitialGuessNonzero(MyKsp,PETSC_TRUE);CHKERRQ(ierr); > > > > > ierr = KSPGMRESSetRestart(MyKsp, max_steps_restart); CHKERRQ(ierr); //With > and without this line > > > > > ierr = KSPSetFromOptions(MyKsp);CHKERRQ(ierr); > > > > ierr = KSPSolve(MyKsp,MyVector,x);CHKERRQ(ierr); > ----------------------------------------------------------------------------------------------------------- > > Thank you very much in advance, > > > Jordi Poblet Puig > > > > > > -- The government saving money is like me spilling beer. It happens, but never on purpose. From junwang at uwm.edu Thu Apr 12 18:27:52 2007 From: junwang at uwm.edu (junwang at uwm.edu) Date: Thu, 12 Apr 2007 18:27:52 -0500 Subject: Cholesky factorization problem Message-ID: <1176420472.461ec07872b1d@panthermail.uwm.edu> Hello: I have written a test sample program for using the Cholesky Factorization function, here is the code i wrote, it compiled successfully. But the result is not correct, is anyone who can help me out? Thank you! *************************************************************************** #include "petsc.h" #include "petscmat.h" #include #include int main(int argc,char **args) { PetscErrorCode ierr; Mat mat; int ind1[2],ind2[2]; PetscScalar temp[2*2]; PetscInt *nnz=new PetscInt[3]; PetscInitialize(&argc,&args,0,0); nnz[0]=2;nnz[1]=1;nnz[1]=1; ierr = MatCreateSeqSBAIJ(PETSC_COMM_SELF,2,6,6,0,nnz,&mat);CHKERRQ(ierr); ind1[0]=0;ind1[1]=1; temp[0]=3;temp[1]=2;temp[2]=0;temp[3]=3; ierr = MatSetValues(mat,2,ind1,2,ind1,temp,INSERT_VALUES);CHKERRQ(ierr); ind2[0]=4;ind2[1]=5; temp[0]=1;temp[1]=1;temp[2]=2;temp[3]=1; ierr = MatSetValues(mat,2,ind1,2,ind2,temp,INSERT_VALUES);CHKERRQ(ierr); ind1[0]=2;ind1[1]=3; temp[0]=4;temp[1]=1;temp[2]=1;temp[3]=5; ierr = MatSetValues(mat,2,ind1,2,ind1,temp,INSERT_VALUES);CHKERRQ(ierr); ind1[0]=4;ind1[1]=5; temp[0]=5;temp[1]=1;temp[2]=1;temp[3]=6; ierr = MatSetValues(mat,2,ind1,2,ind1,temp,INSERT_VALUES);CHKERRQ(ierr); ierr = MatAssemblyBegin(mat,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ierr = MatAssemblyEnd(mat,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); MatView(mat,PETSC_VIEWER_STDOUT_SELF); // begin cholesky factorization IS perm,colp; MatGetOrdering(mat,MATORDERING_NATURAL,&perm,&colp); ierr = ISDestroy(colp);CHKERRQ(ierr); MatFactorInfo info; info.fill=1.0; Mat result; ierr = MatCholeskyFactorSymbolic(mat,perm,&info,&result); CHKERRQ(ierr); ierr = MatCholeskyFactorNumeric(mat,&info,&result);CHKERRQ(ierr); MatView(result, PETSC_VIEWER_STDOUT_SELF); ierr = ISDestroy(perm);CHKERRQ(ierr); ierr= MatDestroy(mat);CHKERRQ(ierr); PetscFinalize(); return 0; } *************************************************************************** the result given below: row0: (0,3),(1,2),(4,1),(5,1) row1: (0,2),(1,3),(4,2),(5,1) row2: (2,4),(3,1) row3: (2,1),(3,5) row4: (4,5),(5,1) row5: (4,1),(5,6) row0: (4,0.2),(5,-0.2) row1: (4,-0.8) (5, -0.2) row2: row3: row4: row5: *************************************************************************** The matrix is in SBAIJ format, block size is 2, which cholesky factorization should i call? really need that for the project! Thank you! Jun From bsmith at mcs.anl.gov Thu Apr 12 21:40:48 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 12 Apr 2007 21:40:48 -0500 (CDT) Subject: Question about memory used by PetsC Objects In-Reply-To: <461E46F5.90606@idris.fr> References: <461E46F5.90606@idris.fr> Message-ID: Laurent, The memory usage printed with -log_summary is not correct. Much of the memory used is not printed out. You can use PetscMemoryGetCurrentUsage() and PetscMemoryGetMaximumUsage() in your code before and after creating a filling in PETSc objects to see how much memory they are used. You can also use PetscMallocDump() at any point to see exactly where in PETSc and how much all of PETSc's memory was allocated. Barry On Thu, 12 Apr 2007, Laurent Nguyen wrote: > Hi, > > I'm a Petsc user, but I'm beginning to use petsc in a supercomputation > context. So, I'm trying parallel versions of my old mono-processed programs. > But I've some difficulties to determine how much memory my program use. (I > give a little example). In this example on one processor (with these command > line arguments: -mat_type mpiaij -pc_type none -log_summary), I created one > mpiaij matrix (size = 500*500) and two vectors (size = 500). I've this > information from output: > > Matrix Object: > type=mpiaij, rows=250000, cols=250000 > total: nonzeros=749996, allocated nonzeros=1750000 > > To me, there is an array of 1750000 double precision number of 8 bytes = > 14000000 bytes (14Mo). > > Same for vectors: 2*8*250000 = 4 Mo > > But in the log summary, I've: > > Memory usage is given in bytes: > > Object Type Creations Destructions Memory Descendants' Mem. > > --- Event Stage 0: Main Stage > > Viewer 1 1 0 0 > Index Set 2 2 672 0 > Map 16 16 4992 0 > Vec 10 10 2006576 0 > Vec Scatter 1 1 0 0 > Matrix 3 3 1068 0 > Krylov Solver 1 1 17208 0 > Preconditioner 1 1 0 0 > ============================================================= > > I don't understand this output...If you have some documentation about how > Petsc deals with memory or if you can explain, I would be very interested. > > This is a problem because I'm working on IBM SP4, and I've limitations of > memory (700mb for data and 700mb for stack). I think the objects created by > Petsc is created in the stack memory, but as I cannot determine exactly the > memory usage, I cannot verify. > > Thank you for your advices, > > Best regards > > From laurent.nguyen at idris.fr Fri Apr 13 03:39:45 2007 From: laurent.nguyen at idris.fr (Laurent Nguyen) Date: Fri, 13 Apr 2007 10:39:45 +0200 Subject: Question about memory used by PetsC Objects In-Reply-To: References: <461E46F5.90606@idris.fr> Message-ID: <461F41D1.3050801@idris.fr> Barry, I thank you for your help. So I tried to apply the functions. It seems that PetscMemoryGetCurrentUsage() gives the total amount of memory allocated at the point of the program where it is placed. But, if a object is deallocated, PetscMemoryGetCurrentUsage() doesn't take it. I would like to try PetscMemoryGetMaximumUsage() at the end of the program hoping it gives me the maximum memory used by my program but there are problems. When I put PetscMemoryGetMaximumUsage(), I've this error at the execution: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Object is in wrong state! [0]PETSC ERROR: To use this function you must first call PetscMemorySetGetMaximumUsage()! So I put PetscMemorySetGetMaximumUsage() before, but at linking, it didn't find PetscMemorySetGetMaximumUsage in the petsc libraries. I thought I made a bad install but when I tried on another machine, I've the same problem. Best regards, ************************************** NGUYEN Anh-Khai Laurent Equipe Support Utilisateur Email : laurent.nguyen at idris.fr T?l : 01.69.35.85.66 Adresse : IDRIS - Institut du D?veloppement et des Ressources en Informatique Scientifique CNRS Batiment 506 BP 167 F - 91403 ORSAY Cedex Site Web : http://www.idris.fr ************************************** Barry Smith a ?crit : > Laurent, > > The memory usage printed with -log_summary is not correct. > Much of the memory used is not printed out. > > You can use PetscMemoryGetCurrentUsage() and PetscMemoryGetMaximumUsage() > in your code before and after creating a filling in PETSc objects > to see how much memory they are used. You can also use PetscMallocDump() > at any point to see exactly where in PETSc and how much all of PETSc's > memory was allocated. > > Barry > > > On Thu, 12 Apr 2007, Laurent Nguyen wrote: > >> Hi, >> >> I'm a Petsc user, but I'm beginning to use petsc in a supercomputation >> context. So, I'm trying parallel versions of my old mono-processed programs. >> But I've some difficulties to determine how much memory my program use. (I >> give a little example). In this example on one processor (with these command >> line arguments: -mat_type mpiaij -pc_type none -log_summary), I created one >> mpiaij matrix (size = 500*500) and two vectors (size = 500). I've this >> information from output: >> >> Matrix Object: >> type=mpiaij, rows=250000, cols=250000 >> total: nonzeros=749996, allocated nonzeros=1750000 >> >> To me, there is an array of 1750000 double precision number of 8 bytes = >> 14000000 bytes (14Mo). >> >> Same for vectors: 2*8*250000 = 4 Mo >> >> But in the log summary, I've: >> >> Memory usage is given in bytes: >> >> Object Type Creations Destructions Memory Descendants' Mem. >> >> --- Event Stage 0: Main Stage >> >> Viewer 1 1 0 0 >> Index Set 2 2 672 0 >> Map 16 16 4992 0 >> Vec 10 10 2006576 0 >> Vec Scatter 1 1 0 0 >> Matrix 3 3 1068 0 >> Krylov Solver 1 1 17208 0 >> Preconditioner 1 1 0 0 >> ============================================================= >> >> I don't understand this output...If you have some documentation about how >> Petsc deals with memory or if you can explain, I would be very interested. >> >> This is a problem because I'm working on IBM SP4, and I've limitations of >> memory (700mb for data and 700mb for stack). I think the objects created by >> Petsc is created in the stack memory, but as I cannot determine exactly the >> memory usage, I cannot verify. >> >> Thank you for your advices, >> >> Best regards >> >> > > From nicolas.bathfield at chalmers.se Fri Apr 13 03:45:00 2007 From: nicolas.bathfield at chalmers.se (Nicolas Bathfield) Date: Fri, 13 Apr 2007 10:45:00 +0200 (CEST) Subject: HYPRE with multiple variables Message-ID: <26157.193.183.3.2.1176453900.squirrel@webmail.chalmers.se> Hi, I am solving the Navier-stokes equations and try to use Hypre as preconditioner. Until now, I used PETSc as non-segregated solver and it worked perfectly. Things got worse when I decided to use Boomeramg (Hypre). As I solve a system of PDEs, each cell is represented by 5 rows in my matrix (I solve for 5 variables). PETSc handles that without problem apparently, but the coarsening scheme of Boomeramg needs more input in order to work properly. Is there an option in PETSc to tell HYPRE that we are dealing with a system of PDEs? (something like: -pc_hypre_boomeramg_...) Thanks for your help. Best regards, Nicolas -- Nicolas BATHFIELD Chalmers University of Technology Shipping and Marine Technology phone: +46 (0)31 772 1476 fax: +46 (0)31 772 3699 From shma7099 at student.uu.se Fri Apr 13 05:54:07 2007 From: shma7099 at student.uu.se (Shaman Mahmoudi) Date: Fri, 13 Apr 2007 12:54:07 +0200 Subject: HYPRE with multiple variables In-Reply-To: <26157.193.183.3.2.1176453900.squirrel@webmail.chalmers.se> References: <26157.193.183.3.2.1176453900.squirrel@webmail.chalmers.se> Message-ID: Hi, You want to exploit the structure of the model? As far as I know, boomeramg can not treat a set of rows or blocks as a molecule, a so called block-smoother? ML 2.0 smoothed aggregation does support it. With best regards, Shaman Mahmoudi On Apr 13, 2007, at 10:45 AM, Nicolas Bathfield wrote: > Hi, > > I am solving the Navier-stokes equations and try to use Hypre as > preconditioner. > Until now, I used PETSc as non-segregated solver and it worked > perfectly. > Things got worse when I decided to use Boomeramg (Hypre). > As I solve a system of PDEs, each cell is represented by 5 rows in my > matrix (I solve for 5 variables). PETSc handles that without problem > apparently, but the coarsening scheme of Boomeramg needs more input in > order to work properly. > > Is there an option in PETSc to tell HYPRE that we are dealing with a > system of PDEs? (something like: -pc_hypre_boomeramg_...) > > > Thanks for your help. > > Best regards, > > Nicolas > > > -- > Nicolas BATHFIELD > Chalmers University of Technology > Shipping and Marine Technology > phone: +46 (0)31 772 1476 > fax: +46 (0)31 772 3699 > From nicolas.bathfield at chalmers.se Fri Apr 13 06:40:53 2007 From: nicolas.bathfield at chalmers.se (Nicolas Bathfield) Date: Fri, 13 Apr 2007 13:40:53 +0200 (CEST) Subject: HYPRE with multiple variables In-Reply-To: References: <26157.193.183.3.2.1176453900.squirrel@webmail.chalmers.se> Message-ID: <9329.193.183.3.2.1176464453.squirrel@webmail.chalmers.se> Dear Shaman, As far as I could understand, there is a BoomerAMG?s systems AMG version available. This seems to be exactly what I am looking for, but I just don't know how to access it, either through PETSc or directly. Best regards, Nicolas > Hi, > > You want to exploit the structure of the model? > As far as I know, boomeramg can not treat a set of rows or blocks as > a molecule, a so called block-smoother? > ML 2.0 smoothed aggregation does support it. > > With best regards, Shaman Mahmoudi > > On Apr 13, 2007, at 10:45 AM, Nicolas Bathfield wrote: > >> Hi, >> >> I am solving the Navier-stokes equations and try to use Hypre as >> preconditioner. >> Until now, I used PETSc as non-segregated solver and it worked >> perfectly. >> Things got worse when I decided to use Boomeramg (Hypre). >> As I solve a system of PDEs, each cell is represented by 5 rows in my >> matrix (I solve for 5 variables). PETSc handles that without problem >> apparently, but the coarsening scheme of Boomeramg needs more input in >> order to work properly. >> >> Is there an option in PETSc to tell HYPRE that we are dealing with a >> system of PDEs? (something like: -pc_hypre_boomeramg_...) >> >> >> Thanks for your help. >> >> Best regards, >> >> Nicolas >> >> >> -- >> Nicolas BATHFIELD >> Chalmers University of Technology >> Shipping and Marine Technology >> phone: +46 (0)31 772 1476 >> fax: +46 (0)31 772 3699 >> > > -- Nicolas BATHFIELD Chalmers University of Technology Shipping and Marine Technology phone: +46 (0)31 772 1476 fax: +46 (0)31 772 3699 From shma7099 at student.uu.se Fri Apr 13 07:04:50 2007 From: shma7099 at student.uu.se (Shaman Mahmoudi) Date: Fri, 13 Apr 2007 14:04:50 +0200 Subject: HYPRE with multiple variables In-Reply-To: <9329.193.183.3.2.1176464453.squirrel@webmail.chalmers.se> References: <26157.193.183.3.2.1176453900.squirrel@webmail.chalmers.se> <9329.193.183.3.2.1176464453.squirrel@webmail.chalmers.se> Message-ID: <8614E121-37D8-4AF3-A67A-A142D27B7B62@student.uu.se> Hi Nicolas, You are right. hypre has changed a lot since the version I used. I found this interesting information: int HYPRE_BOOMERAMGSetNodal(....) Sets whether to use the nodal systems version. Default is 0. Then information about smoothers: One interesting thing there is this, HYPRE_BoomerAMGSetDomainType(....) 0 - each point is a domain (default) 1 each node is a domain (only of interest in systems AMG) 2 .... I could not find how you define the nodal displacement ordering. But it should be there somewhere. I read the reference manual for hypre 2.0 With best regards, Shaman Mahmoudi On Apr 13, 2007, at 1:40 PM, Nicolas Bathfield wrote: > Dear Shaman, > > As far as I could understand, there is a BoomerAMG?s systems AMG > version > available. This seems to be exactly what I am looking for, but I just > don't know how to access it, either through PETSc or directly. > > Best regards, > > Nicolas > >> Hi, >> >> You want to exploit the structure of the model? >> As far as I know, boomeramg can not treat a set of rows or blocks as >> a molecule, a so called block-smoother? >> ML 2.0 smoothed aggregation does support it. >> >> With best regards, Shaman Mahmoudi >> >> On Apr 13, 2007, at 10:45 AM, Nicolas Bathfield wrote: >> >>> Hi, >>> >>> I am solving the Navier-stokes equations and try to use Hypre as >>> preconditioner. >>> Until now, I used PETSc as non-segregated solver and it worked >>> perfectly. >>> Things got worse when I decided to use Boomeramg (Hypre). >>> As I solve a system of PDEs, each cell is represented by 5 rows >>> in my >>> matrix (I solve for 5 variables). PETSc handles that without problem >>> apparently, but the coarsening scheme of Boomeramg needs more >>> input in >>> order to work properly. >>> >>> Is there an option in PETSc to tell HYPRE that we are dealing with a >>> system of PDEs? (something like: -pc_hypre_boomeramg_...) >>> >>> >>> Thanks for your help. >>> >>> Best regards, >>> >>> Nicolas >>> >>> >>> -- >>> Nicolas BATHFIELD >>> Chalmers University of Technology >>> Shipping and Marine Technology >>> phone: +46 (0)31 772 1476 >>> fax: +46 (0)31 772 3699 >>> >> >> > > > -- > Nicolas BATHFIELD > Chalmers University of Technology > Shipping and Marine Technology > phone: +46 (0)31 772 1476 > fax: +46 (0)31 772 3699 > From shma7099 at student.uu.se Fri Apr 13 07:13:23 2007 From: shma7099 at student.uu.se (Shaman Mahmoudi) Date: Fri, 13 Apr 2007 14:13:23 +0200 Subject: HYPRE with multiple variables In-Reply-To: <8614E121-37D8-4AF3-A67A-A142D27B7B62@student.uu.se> References: <26157.193.183.3.2.1176453900.squirrel@webmail.chalmers.se> <9329.193.183.3.2.1176464453.squirrel@webmail.chalmers.se> <8614E121-37D8-4AF3-A67A-A142D27B7B62@student.uu.se> Message-ID: Hi, int HYPRE_BoomerAMGSetNumFunctions (.....) sets the size of the system of PDEs. With best regards, Shaman Mahmoudi On Apr 13, 2007, at 2:04 PM, Shaman Mahmoudi wrote: > Hi Nicolas, > > You are right. hypre has changed a lot since the version I used. > > I found this interesting information: > > int HYPRE_BOOMERAMGSetNodal(....) > > Sets whether to use the nodal systems version. Default is 0. > > Then information about smoothers: > > One interesting thing there is this, > > HYPRE_BoomerAMGSetDomainType(....) > > 0 - each point is a domain (default) > 1 each node is a domain (only of interest in systems AMG) > 2 .... > > I could not find how you define the nodal displacement ordering. > But it should be there somewhere. > > I read the reference manual for hypre 2.0 > > With best regards, Shaman Mahmoudi > > > On Apr 13, 2007, at 1:40 PM, Nicolas Bathfield wrote: > >> Dear Shaman, >> >> As far as I could understand, there is a BoomerAMG?s systems AMG >> version >> available. This seems to be exactly what I am looking for, but I just >> don't know how to access it, either through PETSc or directly. >> >> Best regards, >> >> Nicolas >> >>> Hi, >>> >>> You want to exploit the structure of the model? >>> As far as I know, boomeramg can not treat a set of rows or blocks as >>> a molecule, a so called block-smoother? >>> ML 2.0 smoothed aggregation does support it. >>> >>> With best regards, Shaman Mahmoudi >>> >>> On Apr 13, 2007, at 10:45 AM, Nicolas Bathfield wrote: >>> >>>> Hi, >>>> >>>> I am solving the Navier-stokes equations and try to use Hypre as >>>> preconditioner. >>>> Until now, I used PETSc as non-segregated solver and it worked >>>> perfectly. >>>> Things got worse when I decided to use Boomeramg (Hypre). >>>> As I solve a system of PDEs, each cell is represented by 5 rows >>>> in my >>>> matrix (I solve for 5 variables). PETSc handles that without >>>> problem >>>> apparently, but the coarsening scheme of Boomeramg needs more >>>> input in >>>> order to work properly. >>>> >>>> Is there an option in PETSc to tell HYPRE that we are dealing >>>> with a >>>> system of PDEs? (something like: -pc_hypre_boomeramg_...) >>>> >>>> >>>> Thanks for your help. >>>> >>>> Best regards, >>>> >>>> Nicolas >>>> >>>> >>>> -- >>>> Nicolas BATHFIELD >>>> Chalmers University of Technology >>>> Shipping and Marine Technology >>>> phone: +46 (0)31 772 1476 >>>> fax: +46 (0)31 772 3699 >>>> >>> >>> >> >> >> -- >> Nicolas BATHFIELD >> Chalmers University of Technology >> Shipping and Marine Technology >> phone: +46 (0)31 772 1476 >> fax: +46 (0)31 772 3699 >> > From knepley at gmail.com Fri Apr 13 08:33:11 2007 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 13 Apr 2007 08:33:11 -0500 Subject: Question about memory used by PetsC Objects In-Reply-To: <461F41D1.3050801@idris.fr> References: <461E46F5.90606@idris.fr> <461F41D1.3050801@idris.fr> Message-ID: On 4/13/07, Laurent Nguyen wrote: > Barry, > > I thank you for your help. So I tried to apply the functions. It seems > that PetscMemoryGetCurrentUsage() gives the total amount of memory > allocated at the point of the program where it is placed. But, if a > object is deallocated, PetscMemoryGetCurrentUsage() doesn't take it. > > I would like to try PetscMemoryGetMaximumUsage() at the end of the > program hoping it gives me the maximum memory used by my program but > there are problems. When I put PetscMemoryGetMaximumUsage(), I've this > error at the execution: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Object is in wrong state! > [0]PETSC ERROR: To use this function you must first call > PetscMemorySetGetMaximumUsage()! > > So I put PetscMemorySetGetMaximumUsage() before, but at linking, it > didn't find PetscMemorySetGetMaximumUsage in the petsc libraries. I > thought I made a bad install but when I tried on another machine, I've > the same problem. The function is definitely there (check src/sys/memory/mem.c). You can try a short test: cd src/ksp/ksp/examples/tutorials make ex2 ./ex2 -memory_info ./ex2 -malloc_log Also, I have fixed the specific logging errors you pointed out in the dev copy, but we are doing a code review to make sure no other omissions are lurking. Matt > Best regards, > > ************************************** > NGUYEN Anh-Khai Laurent > Equipe Support Utilisateur > > Email : laurent.nguyen at idris.fr > T?l : 01.69.35.85.66 > Adresse : IDRIS - Institut du D?veloppement et des Ressources en > Informatique Scientifique > CNRS > Batiment 506 > BP 167 > F - 91403 ORSAY Cedex > Site Web : http://www.idris.fr > ************************************** > > Barry Smith a ?crit : > > Laurent, > > > > The memory usage printed with -log_summary is not correct. > > Much of the memory used is not printed out. > > > > You can use PetscMemoryGetCurrentUsage() and PetscMemoryGetMaximumUsage() > > in your code before and after creating a filling in PETSc objects > > to see how much memory they are used. You can also use PetscMallocDump() > > at any point to see exactly where in PETSc and how much all of PETSc's > > memory was allocated. > > > > Barry > > > > > > On Thu, 12 Apr 2007, Laurent Nguyen wrote: > > > >> Hi, > >> > >> I'm a Petsc user, but I'm beginning to use petsc in a supercomputation > >> context. So, I'm trying parallel versions of my old mono-processed programs. > >> But I've some difficulties to determine how much memory my program use. (I > >> give a little example). In this example on one processor (with these command > >> line arguments: -mat_type mpiaij -pc_type none -log_summary), I created one > >> mpiaij matrix (size = 500*500) and two vectors (size = 500). I've this > >> information from output: > >> > >> Matrix Object: > >> type=mpiaij, rows=250000, cols=250000 > >> total: nonzeros=749996, allocated nonzeros=1750000 > >> > >> To me, there is an array of 1750000 double precision number of 8 bytes = > >> 14000000 bytes (14Mo). > >> > >> Same for vectors: 2*8*250000 = 4 Mo > >> > >> But in the log summary, I've: > >> > >> Memory usage is given in bytes: > >> > >> Object Type Creations Destructions Memory Descendants' Mem. > >> > >> --- Event Stage 0: Main Stage > >> > >> Viewer 1 1 0 0 > >> Index Set 2 2 672 0 > >> Map 16 16 4992 0 > >> Vec 10 10 2006576 0 > >> Vec Scatter 1 1 0 0 > >> Matrix 3 3 1068 0 > >> Krylov Solver 1 1 17208 0 > >> Preconditioner 1 1 0 0 > >> ============================================================= > >> > >> I don't understand this output...If you have some documentation about how > >> Petsc deals with memory or if you can explain, I would be very interested. > >> > >> This is a problem because I'm working on IBM SP4, and I've limitations of > >> memory (700mb for data and 700mb for stack). I think the objects created by > >> Petsc is created in the stack memory, but as I cannot determine exactly the > >> memory usage, I cannot verify. > >> > >> Thank you for your advices, > >> > >> Best regards > >> > >> > > > > > > -- The government saving money is like me spilling beer. It happens, but never on purpose. From laurent.nguyen at idris.fr Fri Apr 13 09:18:11 2007 From: laurent.nguyen at idris.fr (Laurent Nguyen) Date: Fri, 13 Apr 2007 16:18:11 +0200 Subject: Question about memory used by PetsC Objects In-Reply-To: References: <461E46F5.90606@idris.fr> <461F41D1.3050801@idris.fr> Message-ID: <461F9123.8070908@idris.fr> Yes, I've tested the argument -memory_info. I've seen the memory given to me is the same memory given by a PetscMemoryGetCurrentUsage() before PetscFinalize(). I didn't know it was the same ^^. It is not the memory I need. I would like to have the real memory used by my program (It is a little difficult to explain clearly): for example, if I create 2 vectors of size 100, the program is consuming 2*8*100 bytes. But later, when I destroy only 1 vector, the programs is consuming 1*8*100 bytes. This is this kind of information that I need (more precisely, the max). I tell you that because, in my context of computation, I've only 1.5Go of memory per processors. And the memory given by the argument -memory_info was 2.4Go. So I was perplex about that. But now I understand how your function is running. But thanks to yours functions, I can now better manage the memory of my objects! Regards, ************************************** NGUYEN Anh-Khai Laurent Equipe Support Utilisateur Email : laurent.nguyen at idris.fr T?l : 01.69.35.85.66 Adresse : IDRIS - Institut du D?veloppement et des Ressources en Informatique Scientifique CNRS Batiment 506 BP 167 F - 91403 ORSAY Cedex Site Web : http://www.idris.fr ************************************** Matthew Knepley a ?crit : > On 4/13/07, Laurent Nguyen wrote: >> Barry, >> >> I thank you for your help. So I tried to apply the functions. It seems >> that PetscMemoryGetCurrentUsage() gives the total amount of memory >> allocated at the point of the program where it is placed. But, if a >> object is deallocated, PetscMemoryGetCurrentUsage() doesn't take it. >> >> I would like to try PetscMemoryGetMaximumUsage() at the end of the >> program hoping it gives me the maximum memory used by my program but >> there are problems. When I put PetscMemoryGetMaximumUsage(), I've this >> error at the execution: >> >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Object is in wrong state! >> [0]PETSC ERROR: To use this function you must first call >> PetscMemorySetGetMaximumUsage()! >> >> So I put PetscMemorySetGetMaximumUsage() before, but at linking, it >> didn't find PetscMemorySetGetMaximumUsage in the petsc libraries. I >> thought I made a bad install but when I tried on another machine, I've >> the same problem. > > The function is definitely there (check src/sys/memory/mem.c). You can try > a short test: > > cd src/ksp/ksp/examples/tutorials > make ex2 > ./ex2 -memory_info > ./ex2 -malloc_log > > Also, I have fixed the specific logging errors you pointed out in the dev > copy, but we are doing a code review to make sure no other omissions > are lurking. > > Matt > >> Best regards, >> >> ************************************** >> NGUYEN Anh-Khai Laurent >> Equipe Support Utilisateur >> >> Email : laurent.nguyen at idris.fr >> T?l : 01.69.35.85.66 >> Adresse : IDRIS - Institut du D?veloppement et des Ressources en >> Informatique Scientifique >> CNRS >> Batiment 506 >> BP 167 >> F - 91403 ORSAY Cedex >> Site Web : http://www.idris.fr >> ************************************** >> >> Barry Smith a ?crit : >> > Laurent, >> > >> > The memory usage printed with -log_summary is not correct. >> > Much of the memory used is not printed out. >> > >> > You can use PetscMemoryGetCurrentUsage() and >> PetscMemoryGetMaximumUsage() >> > in your code before and after creating a filling in PETSc objects >> > to see how much memory they are used. You can also use >> PetscMallocDump() >> > at any point to see exactly where in PETSc and how much all of PETSc's >> > memory was allocated. >> > >> > Barry >> > >> > >> > On Thu, 12 Apr 2007, Laurent Nguyen wrote: >> > >> >> Hi, >> >> >> >> I'm a Petsc user, but I'm beginning to use petsc in a supercomputation >> >> context. So, I'm trying parallel versions of my old mono-processed >> programs. >> >> But I've some difficulties to determine how much memory my program >> use. (I >> >> give a little example). In this example on one processor (with >> these command >> >> line arguments: -mat_type mpiaij -pc_type none -log_summary), I >> created one >> >> mpiaij matrix (size = 500*500) and two vectors (size = 500). I've this >> >> information from output: >> >> >> >> Matrix Object: >> >> type=mpiaij, rows=250000, cols=250000 >> >> total: nonzeros=749996, allocated nonzeros=1750000 >> >> >> >> To me, there is an array of 1750000 double precision number of 8 >> bytes = >> >> 14000000 bytes (14Mo). >> >> >> >> Same for vectors: 2*8*250000 = 4 Mo >> >> >> >> But in the log summary, I've: >> >> >> >> Memory usage is given in bytes: >> >> >> >> Object Type Creations Destructions Memory >> Descendants' Mem. >> >> >> >> --- Event Stage 0: Main Stage >> >> >> >> Viewer 1 1 0 0 >> >> Index Set 2 2 672 0 >> >> Map 16 16 4992 0 >> >> Vec 10 10 2006576 0 >> >> Vec Scatter 1 1 0 0 >> >> Matrix 3 3 1068 0 >> >> Krylov Solver 1 1 17208 0 >> >> Preconditioner 1 1 0 0 >> >> ============================================================= >> >> >> >> I don't understand this output...If you have some documentation >> about how >> >> Petsc deals with memory or if you can explain, I would be very >> interested. >> >> >> >> This is a problem because I'm working on IBM SP4, and I've >> limitations of >> >> memory (700mb for data and 700mb for stack). I think the objects >> created by >> >> Petsc is created in the stack memory, but as I cannot determine >> exactly the >> >> memory usage, I cannot verify. >> >> >> >> Thank you for your advices, >> >> >> >> Best regards >> >> >> >> >> > >> > >> >> > > From knepley at gmail.com Fri Apr 13 09:34:22 2007 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 13 Apr 2007 09:34:22 -0500 Subject: Question about memory used by PetsC Objects In-Reply-To: <461F9123.8070908@idris.fr> References: <461E46F5.90606@idris.fr> <461F41D1.3050801@idris.fr> <461F9123.8070908@idris.fr> Message-ID: There is a way to get exactly what you want which is not really very hard. Petsc allows you to easily override PetscMalloc() and PetscFree() which we use for everything. You could just keep a running static tally of the sizes and then call the regular version. This would only take a few lines of code. Take a look at http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Sys/PetscSetMalloc.html Matt On 4/13/07, Laurent Nguyen wrote: > Yes, I've tested the argument -memory_info. I've seen the memory given > to me is the same memory given by a PetscMemoryGetCurrentUsage() before > PetscFinalize(). I didn't know it was the same ^^. > > It is not the memory I need. I would like to have the real memory used > by my program (It is a little difficult to explain clearly): for > example, if I create 2 vectors of size 100, the program is consuming > 2*8*100 bytes. But later, when I destroy only 1 vector, the programs is > consuming 1*8*100 bytes. This is this kind of information that I need > (more precisely, the max). I tell you that because, in my context of > computation, I've only 1.5Go of memory per processors. And the memory > given by the argument -memory_info was 2.4Go. So I was perplex about > that. But now I understand how your function is running. > > But thanks to yours functions, I can now better manage the memory of my > objects! > > Regards, > > ************************************** > NGUYEN Anh-Khai Laurent > Equipe Support Utilisateur > > Email : laurent.nguyen at idris.fr > T?l : 01.69.35.85.66 > Adresse : IDRIS - Institut du D?veloppement et des Ressources en > Informatique Scientifique > CNRS > Batiment 506 > BP 167 > F - 91403 ORSAY Cedex > Site Web : http://www.idris.fr > ************************************** > > Matthew Knepley a ?crit : > > On 4/13/07, Laurent Nguyen wrote: > >> Barry, > >> > >> I thank you for your help. So I tried to apply the functions. It seems > >> that PetscMemoryGetCurrentUsage() gives the total amount of memory > >> allocated at the point of the program where it is placed. But, if a > >> object is deallocated, PetscMemoryGetCurrentUsage() doesn't take it. > >> > >> I would like to try PetscMemoryGetMaximumUsage() at the end of the > >> program hoping it gives me the maximum memory used by my program but > >> there are problems. When I put PetscMemoryGetMaximumUsage(), I've this > >> error at the execution: > >> > >> [0]PETSC ERROR: --------------------- Error Message > >> ------------------------------------ > >> [0]PETSC ERROR: Object is in wrong state! > >> [0]PETSC ERROR: To use this function you must first call > >> PetscMemorySetGetMaximumUsage()! > >> > >> So I put PetscMemorySetGetMaximumUsage() before, but at linking, it > >> didn't find PetscMemorySetGetMaximumUsage in the petsc libraries. I > >> thought I made a bad install but when I tried on another machine, I've > >> the same problem. > > > > The function is definitely there (check src/sys/memory/mem.c). You can try > > a short test: > > > > cd src/ksp/ksp/examples/tutorials > > make ex2 > > ./ex2 -memory_info > > ./ex2 -malloc_log > > > > Also, I have fixed the specific logging errors you pointed out in the dev > > copy, but we are doing a code review to make sure no other omissions > > are lurking. > > > > Matt > > > >> Best regards, > >> > >> ************************************** > >> NGUYEN Anh-Khai Laurent > >> Equipe Support Utilisateur > >> > >> Email : laurent.nguyen at idris.fr > >> T?l : 01.69.35.85.66 > >> Adresse : IDRIS - Institut du D?veloppement et des Ressources en > >> Informatique Scientifique > >> CNRS > >> Batiment 506 > >> BP 167 > >> F - 91403 ORSAY Cedex > >> Site Web : http://www.idris.fr > >> ************************************** > >> > >> Barry Smith a ?crit : > >> > Laurent, > >> > > >> > The memory usage printed with -log_summary is not correct. > >> > Much of the memory used is not printed out. > >> > > >> > You can use PetscMemoryGetCurrentUsage() and > >> PetscMemoryGetMaximumUsage() > >> > in your code before and after creating a filling in PETSc objects > >> > to see how much memory they are used. You can also use > >> PetscMallocDump() > >> > at any point to see exactly where in PETSc and how much all of PETSc's > >> > memory was allocated. > >> > > >> > Barry > >> > > >> > > >> > On Thu, 12 Apr 2007, Laurent Nguyen wrote: > >> > > >> >> Hi, > >> >> > >> >> I'm a Petsc user, but I'm beginning to use petsc in a supercomputation > >> >> context. So, I'm trying parallel versions of my old mono-processed > >> programs. > >> >> But I've some difficulties to determine how much memory my program > >> use. (I > >> >> give a little example). In this example on one processor (with > >> these command > >> >> line arguments: -mat_type mpiaij -pc_type none -log_summary), I > >> created one > >> >> mpiaij matrix (size = 500*500) and two vectors (size = 500). I've this > >> >> information from output: > >> >> > >> >> Matrix Object: > >> >> type=mpiaij, rows=250000, cols=250000 > >> >> total: nonzeros=749996, allocated nonzeros=1750000 > >> >> > >> >> To me, there is an array of 1750000 double precision number of 8 > >> bytes = > >> >> 14000000 bytes (14Mo). > >> >> > >> >> Same for vectors: 2*8*250000 = 4 Mo > >> >> > >> >> But in the log summary, I've: > >> >> > >> >> Memory usage is given in bytes: > >> >> > >> >> Object Type Creations Destructions Memory > >> Descendants' Mem. > >> >> > >> >> --- Event Stage 0: Main Stage > >> >> > >> >> Viewer 1 1 0 0 > >> >> Index Set 2 2 672 0 > >> >> Map 16 16 4992 0 > >> >> Vec 10 10 2006576 0 > >> >> Vec Scatter 1 1 0 0 > >> >> Matrix 3 3 1068 0 > >> >> Krylov Solver 1 1 17208 0 > >> >> Preconditioner 1 1 0 0 > >> >> ============================================================= > >> >> > >> >> I don't understand this output...If you have some documentation > >> about how > >> >> Petsc deals with memory or if you can explain, I would be very > >> interested. > >> >> > >> >> This is a problem because I'm working on IBM SP4, and I've > >> limitations of > >> >> memory (700mb for data and 700mb for stack). I think the objects > >> created by > >> >> Petsc is created in the stack memory, but as I cannot determine > >> exactly the > >> >> memory usage, I cannot verify. > >> >> > >> >> Thank you for your advices, > >> >> > >> >> Best regards > >> >> > >> >> > >> > > >> > > >> > >> > > > > > > -- The government saving money is like me spilling beer. It happens, but never on purpose. From bsmith at mcs.anl.gov Fri Apr 13 11:07:05 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 13 Apr 2007 11:07:05 -0500 (CDT) Subject: HYPRE with multiple variables In-Reply-To: References: <26157.193.183.3.2.1176453900.squirrel@webmail.chalmers.se> <9329.193.183.3.2.1176464453.squirrel@webmail.chalmers.se> <8614E121-37D8-4AF3-A67A-A142D27B7B62@student.uu.se> Message-ID: From PETSc MPIAIJ matrices you need to set the block size of the matrix with MatSetBlockSize(A,5) after you have called MatSetType() or MatCreateMPIAIJ(). Then HYPRE_BoomerAMGSetNumFunctions() is automatically called by PETSc. Barry The reason this is done this way instead of as -pc_hypre_boomeramg_block_size is the idea that hypre will use the properties of the matrix it is given in building the preconditioner so the user does not have to pass those properties in seperately directly to hypre. On Fri, 13 Apr 2007, Shaman Mahmoudi wrote: > Hi, > > int HYPRE_BoomerAMGSetNumFunctions (.....) > > sets the size of the system of PDEs. > > With best regards, Shaman Mahmoudi > > On Apr 13, 2007, at 2:04 PM, Shaman Mahmoudi wrote: > > > Hi Nicolas, > > > > You are right. hypre has changed a lot since the version I used. > > > > I found this interesting information: > > > > int HYPRE_BOOMERAMGSetNodal(....) > > > > Sets whether to use the nodal systems version. Default is 0. > > > > Then information about smoothers: > > > > One interesting thing there is this, > > > > HYPRE_BoomerAMGSetDomainType(....) > > > > 0 - each point is a domain (default) > > 1 each node is a domain (only of interest in systems AMG) > > 2 .... > > > > I could not find how you define the nodal displacement ordering. But it > > should be there somewhere. > > > > I read the reference manual for hypre 2.0 > > > > With best regards, Shaman Mahmoudi > > > > > > On Apr 13, 2007, at 1:40 PM, Nicolas Bathfield wrote: > > > > > Dear Shaman, > > > > > > As far as I could understand, there is a BoomerAMG?s systems AMG version > > > available. This seems to be exactly what I am looking for, but I just > > > don't know how to access it, either through PETSc or directly. > > > > > > Best regards, > > > > > > Nicolas > > > > > > > Hi, > > > > > > > > You want to exploit the structure of the model? > > > > As far as I know, boomeramg can not treat a set of rows or blocks as > > > > a molecule, a so called block-smoother? > > > > ML 2.0 smoothed aggregation does support it. > > > > > > > > With best regards, Shaman Mahmoudi > > > > > > > > On Apr 13, 2007, at 10:45 AM, Nicolas Bathfield wrote: > > > > > > > > > Hi, > > > > > > > > > > I am solving the Navier-stokes equations and try to use Hypre as > > > > > preconditioner. > > > > > Until now, I used PETSc as non-segregated solver and it worked > > > > > perfectly. > > > > > Things got worse when I decided to use Boomeramg (Hypre). > > > > > As I solve a system of PDEs, each cell is represented by 5 rows in my > > > > > matrix (I solve for 5 variables). PETSc handles that without problem > > > > > apparently, but the coarsening scheme of Boomeramg needs more input in > > > > > order to work properly. > > > > > > > > > > Is there an option in PETSc to tell HYPRE that we are dealing with a > > > > > system of PDEs? (something like: -pc_hypre_boomeramg_...) > > > > > > > > > > > > > > > Thanks for your help. > > > > > > > > > > Best regards, > > > > > > > > > > Nicolas > > > > > > > > > > > > > > > -- > > > > > Nicolas BATHFIELD > > > > > Chalmers University of Technology > > > > > Shipping and Marine Technology > > > > > phone: +46 (0)31 772 1476 > > > > > fax: +46 (0)31 772 3699 > > > > > > > > > > > > > > > > > > > > > > -- > > > Nicolas BATHFIELD > > > Chalmers University of Technology > > > Shipping and Marine Technology > > > phone: +46 (0)31 772 1476 > > > fax: +46 (0)31 772 3699 > > > > > > From junwang at uwm.edu Fri Apr 13 11:11:20 2007 From: junwang at uwm.edu (junwang at uwm.edu) Date: Fri, 13 Apr 2007 11:11:20 -0500 Subject: Cholesky factorization problem Message-ID: <1176480680.461faba88335b@panthermail.uwm.edu> Hello: I have written a test sample program for using the Cholesky Factorization function, here is the code i wrote, it compiled successfully. But the result is not correct, is anyone who can help me out? Thank you! *************************************************************************** #include "petsc.h" #include "petscmat.h" #include #include int main(int argc,char **args) { PetscErrorCode ierr; Mat mat; int ind1[2],ind2[2]; PetscScalar temp[2*2]; PetscInt *nnz=new PetscInt[3]; PetscInitialize(&argc,&args,0,0); nnz[0]=2;nnz[1]=1;nnz[1]=1; ierr = MatCreateSeqSBAIJ(PETSC_COMM_SELF,2,6,6,0,nnz,&mat);CHKERRQ(ierr); ind1[0]=0;ind1[1]=1; temp[0]=3;temp[1]=2;temp[2]=0;temp[3]=3; ierr = MatSetValues(mat,2,ind1,2,ind1,temp,INSERT_VALUES);CHKERRQ(ierr); ind2[0]=4;ind2[1]=5; temp[0]=1;temp[1]=1;temp[2]=2;temp[3]=1; ierr = MatSetValues(mat,2,ind1,2,ind2,temp,INSERT_VALUES);CHKERRQ(ierr); ind1[0]=2;ind1[1]=3; temp[0]=4;temp[1]=1;temp[2]=1;temp[3]=5; ierr = MatSetValues(mat,2,ind1,2,ind1,temp,INSERT_VALUES);CHKERRQ(ierr); ind1[0]=4;ind1[1]=5; temp[0]=5;temp[1]=1;temp[2]=1;temp[3]=6; ierr = MatSetValues(mat,2,ind1,2,ind1,temp,INSERT_VALUES);CHKERRQ(ierr); ierr = MatAssemblyBegin(mat,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ierr = MatAssemblyEnd(mat,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); MatView(mat,PETSC_VIEWER_STDOUT_SELF); // begin cholesky factorization IS perm,colp; MatGetOrdering(mat,MATORDERING_NATURAL,&perm,&colp); ierr = ISDestroy(colp);CHKERRQ(ierr); MatFactorInfo info; info.fill=1.0; Mat result; ierr = MatCholeskyFactorSymbolic(mat,perm,&info,&result); CHKERRQ(ierr); ierr = MatCholeskyFactorNumeric(mat,&info,&result);CHKERRQ(ierr); MatView(result, PETSC_VIEWER_STDOUT_SELF); ierr = ISDestroy(perm);CHKERRQ(ierr); ierr= MatDestroy(mat);CHKERRQ(ierr); PetscFinalize(); return 0; } *************************************************************************** the result given below: mat: row0: (0,3),(1,2),(4,1),(5,1) row1: (0,2),(1,3),(4,2),(5,1) row2: (2,4),(3,1) row3: (2,1),(3,5) row4: (4,5),(5,1) row5: (4,1),(5,6) result: row0: (4,0.2),(5,-0.2) row1: (4,-0.8) (5, -0.2) row2: row3: row4: row5: *************************************************************************** The matrix is in SBAIJ format, block size is 2, which cholesky factorization function should i call? really need that for the project! Thank you! Jun From bsmith at mcs.anl.gov Fri Apr 13 11:14:34 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 13 Apr 2007 11:14:34 -0500 (CDT) Subject: Question about memory used by PetsC Objects In-Reply-To: References: <461E46F5.90606@idris.fr> <461F41D1.3050801@idris.fr> <461F9123.8070908@idris.fr> Message-ID: Matt, I do not understand why one would want this, doesn't PETSc already keep a current amount used (memory malloced but not freed) and the maximum current amount used? NGUYEN, Once PETSc frees memory (or any program) it is not returned to the operating system, rather it is reused in a future malloc if possible. PETSc cannot give the TOTAL memory used by the process since that includes pages of the program and any memory your code is using and there is no good Unix command to get that information on all computer systems. On Fri, 13 Apr 2007, Matthew Knepley wrote: > There is a way to get exactly what you want which is not really very hard. > Petsc allows you to easily override PetscMalloc() and PetscFree() which > we use for everything. You could just keep a running static tally of the sizes > and then call the regular version. This would only take a few lines of code. > Take a look at > > http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Sys/PetscSetMalloc.html > > Matt > > On 4/13/07, Laurent Nguyen wrote: > > Yes, I've tested the argument -memory_info. I've seen the memory given > > to me is the same memory given by a PetscMemoryGetCurrentUsage() before > > PetscFinalize(). I didn't know it was the same ^^. > > > > It is not the memory I need. I would like to have the real memory used > > by my program (It is a little difficult to explain clearly): for > > example, if I create 2 vectors of size 100, the program is consuming > > 2*8*100 bytes. But later, when I destroy only 1 vector, the programs is > > consuming 1*8*100 bytes. This is this kind of information that I need > > (more precisely, the max). I tell you that because, in my context of > > computation, I've only 1.5Go of memory per processors. And the memory > > given by the argument -memory_info was 2.4Go. So I was perplex about > > that. But now I understand how your function is running. > > > > But thanks to yours functions, I can now better manage the memory of my > > objects! > > > > Regards, > > > > ************************************** > > NGUYEN Anh-Khai Laurent > > Equipe Support Utilisateur > > > > Email : laurent.nguyen at idris.fr > > T?l : 01.69.35.85.66 > > Adresse : IDRIS - Institut du D?veloppement et des Ressources en > > Informatique Scientifique > > CNRS > > Batiment 506 > > BP 167 > > F - 91403 ORSAY Cedex > > Site Web : http://www.idris.fr > > ************************************** > > > > Matthew Knepley a ?crit : > > > On 4/13/07, Laurent Nguyen wrote: > > >> Barry, > > >> > > >> I thank you for your help. So I tried to apply the functions. It seems > > >> that PetscMemoryGetCurrentUsage() gives the total amount of memory > > >> allocated at the point of the program where it is placed. But, if a > > >> object is deallocated, PetscMemoryGetCurrentUsage() doesn't take it. > > >> > > >> I would like to try PetscMemoryGetMaximumUsage() at the end of the > > >> program hoping it gives me the maximum memory used by my program but > > >> there are problems. When I put PetscMemoryGetMaximumUsage(), I've this > > >> error at the execution: > > >> > > >> [0]PETSC ERROR: --------------------- Error Message > > >> ------------------------------------ > > >> [0]PETSC ERROR: Object is in wrong state! > > >> [0]PETSC ERROR: To use this function you must first call > > >> PetscMemorySetGetMaximumUsage()! > > >> > > >> So I put PetscMemorySetGetMaximumUsage() before, but at linking, it > > >> didn't find PetscMemorySetGetMaximumUsage in the petsc libraries. I > > >> thought I made a bad install but when I tried on another machine, I've > > >> the same problem. > > > > > > The function is definitely there (check src/sys/memory/mem.c). You can try > > > a short test: > > > > > > cd src/ksp/ksp/examples/tutorials > > > make ex2 > > > ./ex2 -memory_info > > > ./ex2 -malloc_log > > > > > > Also, I have fixed the specific logging errors you pointed out in the dev > > > copy, but we are doing a code review to make sure no other omissions > > > are lurking. > > > > > > Matt > > > > > >> Best regards, > > >> > > >> ************************************** > > >> NGUYEN Anh-Khai Laurent > > >> Equipe Support Utilisateur > > >> > > >> Email : laurent.nguyen at idris.fr > > >> T?l : 01.69.35.85.66 > > >> Adresse : IDRIS - Institut du D?veloppement et des Ressources en > > >> Informatique Scientifique > > >> CNRS > > >> Batiment 506 > > >> BP 167 > > >> F - 91403 ORSAY Cedex > > >> Site Web : http://www.idris.fr > > >> ************************************** > > >> > > >> Barry Smith a ?crit : > > >> > Laurent, > > >> > > > >> > The memory usage printed with -log_summary is not correct. > > >> > Much of the memory used is not printed out. > > >> > > > >> > You can use PetscMemoryGetCurrentUsage() and > > >> PetscMemoryGetMaximumUsage() > > >> > in your code before and after creating a filling in PETSc objects > > >> > to see how much memory they are used. You can also use > > >> PetscMallocDump() > > >> > at any point to see exactly where in PETSc and how much all of PETSc's > > >> > memory was allocated. > > >> > > > >> > Barry > > >> > > > >> > > > >> > On Thu, 12 Apr 2007, Laurent Nguyen wrote: > > >> > > > >> >> Hi, > > >> >> > > >> >> I'm a Petsc user, but I'm beginning to use petsc in a supercomputation > > >> >> context. So, I'm trying parallel versions of my old mono-processed > > >> programs. > > >> >> But I've some difficulties to determine how much memory my program > > >> use. (I > > >> >> give a little example). In this example on one processor (with > > >> these command > > >> >> line arguments: -mat_type mpiaij -pc_type none -log_summary), I > > >> created one > > >> >> mpiaij matrix (size = 500*500) and two vectors (size = 500). I've this > > >> >> information from output: > > >> >> > > >> >> Matrix Object: > > >> >> type=mpiaij, rows=250000, cols=250000 > > >> >> total: nonzeros=749996, allocated nonzeros=1750000 > > >> >> > > >> >> To me, there is an array of 1750000 double precision number of 8 > > >> bytes = > > >> >> 14000000 bytes (14Mo). > > >> >> > > >> >> Same for vectors: 2*8*250000 = 4 Mo > > >> >> > > >> >> But in the log summary, I've: > > >> >> > > >> >> Memory usage is given in bytes: > > >> >> > > >> >> Object Type Creations Destructions Memory > > >> Descendants' Mem. > > >> >> > > >> >> --- Event Stage 0: Main Stage > > >> >> > > >> >> Viewer 1 1 0 0 > > >> >> Index Set 2 2 672 0 > > >> >> Map 16 16 4992 0 > > >> >> Vec 10 10 2006576 0 > > >> >> Vec Scatter 1 1 0 0 > > >> >> Matrix 3 3 1068 0 > > >> >> Krylov Solver 1 1 17208 0 > > >> >> Preconditioner 1 1 0 0 > > >> >> ============================================================= > > >> >> > > >> >> I don't understand this output...If you have some documentation > > >> about how > > >> >> Petsc deals with memory or if you can explain, I would be very > > >> interested. > > >> >> > > >> >> This is a problem because I'm working on IBM SP4, and I've > > >> limitations of > > >> >> memory (700mb for data and 700mb for stack). I think the objects > > >> created by > > >> >> Petsc is created in the stack memory, but as I cannot determine > > >> exactly the > > >> >> memory usage, I cannot verify. > > >> >> > > >> >> Thank you for your advices, > > >> >> > > >> >> Best regards > > >> >> > > >> >> > > >> > > > >> > > > >> > > >> > > > > > > > > > > > > > From bsmith at mcs.anl.gov Fri Apr 13 11:24:12 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 13 Apr 2007 11:24:12 -0500 (CDT) Subject: Process interrupted error In-Reply-To: <1176362443.7365.28.camel@iept0415.ivt.ntnu.no> References: <4611A6CF.6010502@nemesis-project.org> <1176281525.7365.16.camel@iept0415.ivt.ntnu.no> <1176362443.7365.28.camel@iept0415.ivt.ntnu.no> Message-ID: Runs in debugger sometimes and not outside debugger is often a sign of memory corruption. Run with -malloc_debug and put a CHKMEMQ; directly before you create the scatter context see if that gives you any useful information. Please respond to petsc-maint and not petsc-users. Barry On Thu, 12 Apr 2007, Knut Erik Teigen wrote: > On Wed, 2007-04-11 at 06:55 -0500, Barry Smith wrote: > > Something is sending an interupt signal to all the proccess > > except one. We've seen this happen where the "batch or node > > scheduler does this to kill a long running job". > > > > Does it happen even if the parallel vector is real short? > > Yes, even with a vector of size 1, this happens. > > > > Does the machine seem to hang at that point or does the sigint > > come immediately? > It hangs for a short period, and then displays the sigint error. > When running with 2 processes, it also hangs a bit, but then continues > with the correct results. > > > > Can you us the runtime option -start_in_debugger or totalview > > to catch the signal? > Something really weird happens here. The first time I run the program > with -start_in_debugger noxterm,idb , it finishes correctly, no matter > the number of processes or problem size. The second time, however, I get > the sigint errors. > When I run with idb I get: > (idb) Program exited normally. > (idb) Program received signal SIGINT > > With gdb I get: > 0xffffe410 in __kernel_vsyscall () > and then the program hangs. > > However I get this message in the beginning with both: > [3]PETSC ERROR: PETSC: Attaching gdb to ./out of pid 30549 on ivt0415 > > -Knut Erik- > > > > > Barry > > > > On Wed, 11 Apr 2007, Knut Erik Teigen wrote: > > > > > Hello, > > > > > > I get the following error when trying to copy a solution vector to > > > process zero using VecScatterCreateToZero: > > > > > > [0] VecScatterCreate(): Special case: processor zero gets entire > > > parallel vector, rest get none > > > forrtl: error (69): process interrupted (SIGINT) > > > forrtl: error (69): process interrupted (SIGINT) > > > > > > When running with one or two processes, the code runs fine, but with > > > three or more, the above error occurs, with one "process interrupted" > > > error for each process minus one. Could someone help me figure out > > > what's wrong? > > > > > > Regards, > > > Knut Erik Teigen > > > > > > > > > > > > > > > > > > From hzhang at mcs.anl.gov Fri Apr 13 14:25:46 2007 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Fri, 13 Apr 2007 14:25:46 -0500 (CDT) Subject: Cholesky factorization problem In-Reply-To: <1176480680.461faba88335b@panthermail.uwm.edu> References: <1176480680.461faba88335b@panthermail.uwm.edu> Message-ID: Jun, Your code looks fine. The factored matrix "result" is an internal matrix to be used by MatSolve, thus, its actural storage doesn't match the logical matrix elements. MatView() only display off-diagonal blocks of the factored matrix (these values may not match the logical factored matrix values!). We'll add a note to MatView() when user display an internal factored matrix. If you get wrong MatSolve() with your code, let us know. Thanks for reporting if, Hong On Fri, 13 Apr 2007 junwang at uwm.edu wrote: > > > Hello: > I have written a test sample program for using the Cholesky Factorization > function, here is the code i wrote, it compiled successfully. But the result is > not correct, is anyone who can help me out? Thank you! > *************************************************************************** > > #include "petsc.h" > #include "petscmat.h" > #include > #include > > int main(int argc,char **args) > { > PetscErrorCode ierr; > Mat mat; > int ind1[2],ind2[2]; > PetscScalar temp[2*2]; > PetscInt *nnz=new PetscInt[3]; > > PetscInitialize(&argc,&args,0,0); > nnz[0]=2;nnz[1]=1;nnz[1]=1; > > ierr = MatCreateSeqSBAIJ(PETSC_COMM_SELF,2,6,6,0,nnz,&mat);CHKERRQ(ierr); > ind1[0]=0;ind1[1]=1; > temp[0]=3;temp[1]=2;temp[2]=0;temp[3]=3; > ierr = MatSetValues(mat,2,ind1,2,ind1,temp,INSERT_VALUES);CHKERRQ(ierr); > ind2[0]=4;ind2[1]=5; > temp[0]=1;temp[1]=1;temp[2]=2;temp[3]=1; > ierr = MatSetValues(mat,2,ind1,2,ind2,temp,INSERT_VALUES);CHKERRQ(ierr); > ind1[0]=2;ind1[1]=3; > temp[0]=4;temp[1]=1;temp[2]=1;temp[3]=5; > ierr = MatSetValues(mat,2,ind1,2,ind1,temp,INSERT_VALUES);CHKERRQ(ierr); > ind1[0]=4;ind1[1]=5; > temp[0]=5;temp[1]=1;temp[2]=1;temp[3]=6; > ierr = MatSetValues(mat,2,ind1,2,ind1,temp,INSERT_VALUES);CHKERRQ(ierr); > > ierr = MatAssemblyBegin(mat,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > ierr = MatAssemblyEnd(mat,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > > MatView(mat,PETSC_VIEWER_STDOUT_SELF); > // begin cholesky factorization > IS perm,colp; > > MatGetOrdering(mat,MATORDERING_NATURAL,&perm,&colp); > ierr = ISDestroy(colp);CHKERRQ(ierr); > > MatFactorInfo info; > info.fill=1.0; > Mat result; > > ierr = MatCholeskyFactorSymbolic(mat,perm,&info,&result); CHKERRQ(ierr); > ierr = MatCholeskyFactorNumeric(mat,&info,&result);CHKERRQ(ierr); > MatView(result, PETSC_VIEWER_STDOUT_SELF); > > ierr = ISDestroy(perm);CHKERRQ(ierr); > ierr= MatDestroy(mat);CHKERRQ(ierr); > PetscFinalize(); > return 0; > } > *************************************************************************** > > the result given below: > mat: > row0: (0,3),(1,2),(4,1),(5,1) > row1: (0,2),(1,3),(4,2),(5,1) > row2: (2,4),(3,1) > row3: (2,1),(3,5) > row4: (4,5),(5,1) > row5: (4,1),(5,6) > result: > row0: (4,0.2),(5,-0.2) > row1: (4,-0.8) (5, -0.2) > row2: > row3: > row4: > row5: > *************************************************************************** > > The matrix is in SBAIJ format, block size is 2, which cholesky factorization > function should i call? really need that for the project! Thank you! > > Jun > > From JUNWANG at uwm.edu Fri Apr 13 17:26:32 2007 From: JUNWANG at uwm.edu (JUNWANG at uwm.edu) Date: Fri, 13 Apr 2007 17:26:32 -0500 Subject: Cholesky factorization problem In-Reply-To: References: <1176480680.461faba88335b@panthermail.uwm.edu> Message-ID: <1176503192.46200398e7b82@panthermail.uwm.edu> Hi, Hong: Really appreciate your help! The MatSolve() works fine. It provides the right solution. How about if i wanna see the elements in the factorized up-triangle matix R, where A =R'R. Since the MatView() cannot show the logical matrix elements, which function should i call? Thank you ! Jun Quoting Hong Zhang : > > Jun, > > Your code looks fine. The factored matrix > "result" is an internal matrix to be used by MatSolve, > thus, its actural storage doesn't match the logical > matrix elements. MatView() only display off-diagonal blocks > of the factored matrix (these values may not match the logical > factored matrix values!). > > We'll add a note to MatView() when user display > an internal factored matrix. > > If you get wrong MatSolve() with your code, > let us know. > > Thanks for reporting if, > > Hong > > > On Fri, 13 Apr 2007 junwang at uwm.edu wrote: > > > > > > > Hello: > > I have written a test sample program for using the Cholesky > Factorization > > function, here is the code i wrote, it compiled successfully. But the > result is > > not correct, is anyone who can help me out? Thank you! > > *************************************************************************** > > > > #include "petsc.h" > > #include "petscmat.h" > > #include > > #include > > > > int main(int argc,char **args) > > { > > PetscErrorCode ierr; > > Mat mat; > > int ind1[2],ind2[2]; > > PetscScalar temp[2*2]; > > PetscInt *nnz=new PetscInt[3]; > > > > PetscInitialize(&argc,&args,0,0); > > nnz[0]=2;nnz[1]=1;nnz[1]=1; > > > > ierr = MatCreateSeqSBAIJ(PETSC_COMM_SELF,2,6,6,0,nnz,&mat);CHKERRQ(ierr); > > ind1[0]=0;ind1[1]=1; > > temp[0]=3;temp[1]=2;temp[2]=0;temp[3]=3; > > ierr = MatSetValues(mat,2,ind1,2,ind1,temp,INSERT_VALUES);CHKERRQ(ierr); > > ind2[0]=4;ind2[1]=5; > > temp[0]=1;temp[1]=1;temp[2]=2;temp[3]=1; > > ierr = MatSetValues(mat,2,ind1,2,ind2,temp,INSERT_VALUES);CHKERRQ(ierr); > > ind1[0]=2;ind1[1]=3; > > temp[0]=4;temp[1]=1;temp[2]=1;temp[3]=5; > > ierr = MatSetValues(mat,2,ind1,2,ind1,temp,INSERT_VALUES);CHKERRQ(ierr); > > ind1[0]=4;ind1[1]=5; > > temp[0]=5;temp[1]=1;temp[2]=1;temp[3]=6; > > ierr = MatSetValues(mat,2,ind1,2,ind1,temp,INSERT_VALUES);CHKERRQ(ierr); > > > > ierr = MatAssemblyBegin(mat,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > > ierr = MatAssemblyEnd(mat,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > > > > MatView(mat,PETSC_VIEWER_STDOUT_SELF); > > // begin cholesky factorization > > IS perm,colp; > > > > MatGetOrdering(mat,MATORDERING_NATURAL,&perm,&colp); > > ierr = ISDestroy(colp);CHKERRQ(ierr); > > > > MatFactorInfo info; > > info.fill=1.0; > > Mat result; > > > > ierr = MatCholeskyFactorSymbolic(mat,perm,&info,&result); CHKERRQ(ierr); > > ierr = MatCholeskyFactorNumeric(mat,&info,&result);CHKERRQ(ierr); > > MatView(result, PETSC_VIEWER_STDOUT_SELF); > > > > ierr = ISDestroy(perm);CHKERRQ(ierr); > > ierr= MatDestroy(mat);CHKERRQ(ierr); > > PetscFinalize(); > > return 0; > > } > > *************************************************************************** > > > > the result given below: > > mat: > > row0: (0,3),(1,2),(4,1),(5,1) > > row1: (0,2),(1,3),(4,2),(5,1) > > row2: (2,4),(3,1) > > row3: (2,1),(3,5) > > row4: (4,5),(5,1) > > row5: (4,1),(5,6) > > result: > > row0: (4,0.2),(5,-0.2) > > row1: (4,-0.8) (5, -0.2) > > row2: > > row3: > > row4: > > row5: > > *************************************************************************** > > > > The matrix is in SBAIJ format, block size is 2, which cholesky > factorization > > function should i call? really need that for the project! Thank you! > > > > Jun > > > > > > From hzhang at mcs.anl.gov Fri Apr 13 19:24:38 2007 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Fri, 13 Apr 2007 19:24:38 -0500 (CDT) Subject: Cholesky factorization problem In-Reply-To: <1176503192.46200398e7b82@panthermail.uwm.edu> References: <1176480680.461faba88335b@panthermail.uwm.edu> <1176503192.46200398e7b82@panthermail.uwm.edu> Message-ID: In petsc, we do A = R*D*R^T, in which D acturally stores inv(d_i). When A is in blocked form, i.e., bs=2 in your case, d_i is a bsxbs block, D holds inverse(d_i) which is computed by LAPACK. Our MatView() only displays R without diagonal blocks because this is the data structure used by MatSolve(). Why do you want to see R? For varifying the correctness of the code or its numerical values? Hong On Fri, 13 Apr 2007 JUNWANG at uwm.edu wrote: > Hi, Hong: > > Really appreciate your help! > The MatSolve() works fine. It provides the right solution. How about if i > wanna see the elements in the factorized up-triangle matix R, where A =R'R. > Since the MatView() cannot show the logical matrix elements, which function > should i call? > Thank you ! > > Jun > > Quoting Hong Zhang : > > > > > Jun, > > > > Your code looks fine. The factored matrix > > "result" is an internal matrix to be used by MatSolve, > > thus, its actural storage doesn't match the logical > > matrix elements. MatView() only display off-diagonal blocks > > of the factored matrix (these values may not match the logical > > factored matrix values!). > > > > We'll add a note to MatView() when user display > > an internal factored matrix. > > > > If you get wrong MatSolve() with your code, > > let us know. > > > > Thanks for reporting if, > > > > Hong > > > > > > On Fri, 13 Apr 2007 junwang at uwm.edu wrote: > > > > > > > > > > > Hello: > > > I have written a test sample program for using the Cholesky > > Factorization > > > function, here is the code i wrote, it compiled successfully. But the > > result is > > > not correct, is anyone who can help me out? Thank you! > > > *************************************************************************** > > > > > > #include "petsc.h" > > > #include "petscmat.h" > > > #include > > > #include > > > > > > int main(int argc,char **args) > > > { > > > PetscErrorCode ierr; > > > Mat mat; > > > int ind1[2],ind2[2]; > > > PetscScalar temp[2*2]; > > > PetscInt *nnz=new PetscInt[3]; > > > > > > PetscInitialize(&argc,&args,0,0); > > > nnz[0]=2;nnz[1]=1;nnz[1]=1; > > > > > > ierr = MatCreateSeqSBAIJ(PETSC_COMM_SELF,2,6,6,0,nnz,&mat);CHKERRQ(ierr); > > > ind1[0]=0;ind1[1]=1; > > > temp[0]=3;temp[1]=2;temp[2]=0;temp[3]=3; > > > ierr = MatSetValues(mat,2,ind1,2,ind1,temp,INSERT_VALUES);CHKERRQ(ierr); > > > ind2[0]=4;ind2[1]=5; > > > temp[0]=1;temp[1]=1;temp[2]=2;temp[3]=1; > > > ierr = MatSetValues(mat,2,ind1,2,ind2,temp,INSERT_VALUES);CHKERRQ(ierr); > > > ind1[0]=2;ind1[1]=3; > > > temp[0]=4;temp[1]=1;temp[2]=1;temp[3]=5; > > > ierr = MatSetValues(mat,2,ind1,2,ind1,temp,INSERT_VALUES);CHKERRQ(ierr); > > > ind1[0]=4;ind1[1]=5; > > > temp[0]=5;temp[1]=1;temp[2]=1;temp[3]=6; > > > ierr = MatSetValues(mat,2,ind1,2,ind1,temp,INSERT_VALUES);CHKERRQ(ierr); > > > > > > ierr = MatAssemblyBegin(mat,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > > > ierr = MatAssemblyEnd(mat,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > > > > > > MatView(mat,PETSC_VIEWER_STDOUT_SELF); > > > // begin cholesky factorization > > > IS perm,colp; > > > > > > MatGetOrdering(mat,MATORDERING_NATURAL,&perm,&colp); > > > ierr = ISDestroy(colp);CHKERRQ(ierr); > > > > > > MatFactorInfo info; > > > info.fill=1.0; > > > Mat result; > > > > > > ierr = MatCholeskyFactorSymbolic(mat,perm,&info,&result); CHKERRQ(ierr); > > > ierr = MatCholeskyFactorNumeric(mat,&info,&result);CHKERRQ(ierr); > > > MatView(result, PETSC_VIEWER_STDOUT_SELF); > > > > > > ierr = ISDestroy(perm);CHKERRQ(ierr); > > > ierr= MatDestroy(mat);CHKERRQ(ierr); > > > PetscFinalize(); > > > return 0; > > > } > > > *************************************************************************** > > > > > > the result given below: > > > mat: > > > row0: (0,3),(1,2),(4,1),(5,1) > > > row1: (0,2),(1,3),(4,2),(5,1) > > > row2: (2,4),(3,1) > > > row3: (2,1),(3,5) > > > row4: (4,5),(5,1) > > > row5: (4,1),(5,6) > > > result: > > > row0: (4,0.2),(5,-0.2) > > > row1: (4,-0.8) (5, -0.2) > > > row2: > > > row3: > > > row4: > > > row5: > > > *************************************************************************** > > > > > > The matrix is in SBAIJ format, block size is 2, which cholesky > > factorization > > > function should i call? really need that for the project! Thank you! > > > > > > Jun > > > > > > > > > > > > > > From laurent.nguyen at idris.fr Fri Apr 13 21:16:41 2007 From: laurent.nguyen at idris.fr (Laurent Nguyen) Date: Sat, 14 Apr 2007 04:16:41 +0200 Subject: Question about memory used by PetsC Objects In-Reply-To: References: <461E46F5.90606@idris.fr> <461F41D1.3050801@idris.fr> <461F9123.8070908@idris.fr> Message-ID: <20070414021641.GA3902@lin2.idris.fr> Yes, I know, this is a problem on all computer systems.But, I hoped that your profiling functions could do something like that (this is the name of functions which give me this hope) The reason why I need that is because of my computer system (IBM SP4). I've limits of memory and if I use more memory than allowed, the system break my program. For example, in a C code without allocation (only static memory), I can calculate the maximum of memory used by my program. With PetsC, I don't know all the mechanism behind Petsc Objects. This why I sent you theses mails ^^ I would have thought this is a problem for users who make computations on this sort of system (with limits etc...), but it seems this is not the case. Sorry for the disturbance and thank you for your help Laurent On Fri, Apr 13, 2007 at 11:14:34AM -0500, Barry Smith wrote: > > Matt, > > I do not understand why one would want this, doesn't PETSc already keep > a current amount used (memory malloced but not freed) and the maximum current > amount used? > > NGUYEN, > > Once PETSc frees memory (or any program) it is not returned to the operating > system, rather it is reused in a future malloc if possible. PETSc cannot give > the TOTAL memory used by the process since that includes pages of the program > and any memory your code is using and there is no good Unix command to get that > information on all computer systems. > > On Fri, 13 Apr 2007, Matthew Knepley wrote: > > > There is a way to get exactly what you want which is not really very hard. > > Petsc allows you to easily override PetscMalloc() and PetscFree() which > > we use for everything. You could just keep a running static tally of the sizes > > and then call the regular version. This would only take a few lines of code. > > Take a look at > > > > http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Sys/PetscSetMalloc.html > > > > Matt > > > > On 4/13/07, Laurent Nguyen wrote: > > > Yes, I've tested the argument -memory_info. I've seen the memory given > > > to me is the same memory given by a PetscMemoryGetCurrentUsage() before > > > PetscFinalize(). I didn't know it was the same ^^. > > > > > > It is not the memory I need. I would like to have the real memory used > > > by my program (It is a little difficult to explain clearly): for > > > example, if I create 2 vectors of size 100, the program is consuming > > > 2*8*100 bytes. But later, when I destroy only 1 vector, the programs is > > > consuming 1*8*100 bytes. This is this kind of information that I need > > > (more precisely, the max). I tell you that because, in my context of > > > computation, I've only 1.5Go of memory per processors. And the memory > > > given by the argument -memory_info was 2.4Go. So I was perplex about > > > that. But now I understand how your function is running. > > > > > > But thanks to yours functions, I can now better manage the memory of my > > > objects! > > > > > > Regards, > > > > > > ************************************** > > > NGUYEN Anh-Khai Laurent > > > Equipe Support Utilisateur > > > > > > Email : laurent.nguyen at idris.fr > > > T?l : 01.69.35.85.66 > > > Adresse : IDRIS - Institut du D?veloppement et des Ressources en > > > Informatique Scientifique > > > CNRS > > > Batiment 506 > > > BP 167 > > > F - 91403 ORSAY Cedex > > > Site Web : http://www.idris.fr > > > ************************************** > > > > > > Matthew Knepley a ?crit : > > > > On 4/13/07, Laurent Nguyen wrote: > > > >> Barry, > > > >> > > > >> I thank you for your help. So I tried to apply the functions. It seems > > > >> that PetscMemoryGetCurrentUsage() gives the total amount of memory > > > >> allocated at the point of the program where it is placed. But, if a > > > >> object is deallocated, PetscMemoryGetCurrentUsage() doesn't take it. > > > >> > > > >> I would like to try PetscMemoryGetMaximumUsage() at the end of the > > > >> program hoping it gives me the maximum memory used by my program but > > > >> there are problems. When I put PetscMemoryGetMaximumUsage(), I've this > > > >> error at the execution: > > > >> > > > >> [0]PETSC ERROR: --------------------- Error Message > > > >> ------------------------------------ > > > >> [0]PETSC ERROR: Object is in wrong state! > > > >> [0]PETSC ERROR: To use this function you must first call > > > >> PetscMemorySetGetMaximumUsage()! > > > >> > > > >> So I put PetscMemorySetGetMaximumUsage() before, but at linking, it > > > >> didn't find PetscMemorySetGetMaximumUsage in the petsc libraries. I > > > >> thought I made a bad install but when I tried on another machine, I've > > > >> the same problem. > > > > > > > > The function is definitely there (check src/sys/memory/mem.c). You can try > > > > a short test: > > > > > > > > cd src/ksp/ksp/examples/tutorials > > > > make ex2 > > > > ./ex2 -memory_info > > > > ./ex2 -malloc_log > > > > > > > > Also, I have fixed the specific logging errors you pointed out in the dev > > > > copy, but we are doing a code review to make sure no other omissions > > > > are lurking. > > > > > > > > Matt > > > > > > > >> Best regards, > > > >> > > > >> ************************************** > > > >> NGUYEN Anh-Khai Laurent > > > >> Equipe Support Utilisateur > > > >> > > > >> Email : laurent.nguyen at idris.fr > > > >> T?l : 01.69.35.85.66 > > > >> Adresse : IDRIS - Institut du D?veloppement et des Ressources en > > > >> Informatique Scientifique > > > >> CNRS > > > >> Batiment 506 > > > >> BP 167 > > > >> F - 91403 ORSAY Cedex > > > >> Site Web : http://www.idris.fr > > > >> ************************************** > > > >> > > > >> Barry Smith a ?crit : > > > >> > Laurent, > > > >> > > > > >> > The memory usage printed with -log_summary is not correct. > > > >> > Much of the memory used is not printed out. > > > >> > > > > >> > You can use PetscMemoryGetCurrentUsage() and > > > >> PetscMemoryGetMaximumUsage() > > > >> > in your code before and after creating a filling in PETSc objects > > > >> > to see how much memory they are used. You can also use > > > >> PetscMallocDump() > > > >> > at any point to see exactly where in PETSc and how much all of PETSc's > > > >> > memory was allocated. > > > >> > > > > >> > Barry > > > >> > > > > >> > > > > >> > On Thu, 12 Apr 2007, Laurent Nguyen wrote: > > > >> > > > > >> >> Hi, > > > >> >> > > > >> >> I'm a Petsc user, but I'm beginning to use petsc in a supercomputation > > > >> >> context. So, I'm trying parallel versions of my old mono-processed > > > >> programs. > > > >> >> But I've some difficulties to determine how much memory my program > > > >> use. (I > > > >> >> give a little example). In this example on one processor (with > > > >> these command > > > >> >> line arguments: -mat_type mpiaij -pc_type none -log_summary), I > > > >> created one > > > >> >> mpiaij matrix (size = 500*500) and two vectors (size = 500). I've this > > > >> >> information from output: > > > >> >> > > > >> >> Matrix Object: > > > >> >> type=mpiaij, rows=250000, cols=250000 > > > >> >> total: nonzeros=749996, allocated nonzeros=1750000 > > > >> >> > > > >> >> To me, there is an array of 1750000 double precision number of 8 > > > >> bytes = > > > >> >> 14000000 bytes (14Mo). > > > >> >> > > > >> >> Same for vectors: 2*8*250000 = 4 Mo > > > >> >> > > > >> >> But in the log summary, I've: > > > >> >> > > > >> >> Memory usage is given in bytes: > > > >> >> > > > >> >> Object Type Creations Destructions Memory > > > >> Descendants' Mem. > > > >> >> > > > >> >> --- Event Stage 0: Main Stage > > > >> >> > > > >> >> Viewer 1 1 0 0 > > > >> >> Index Set 2 2 672 0 > > > >> >> Map 16 16 4992 0 > > > >> >> Vec 10 10 2006576 0 > > > >> >> Vec Scatter 1 1 0 0 > > > >> >> Matrix 3 3 1068 0 > > > >> >> Krylov Solver 1 1 17208 0 > > > >> >> Preconditioner 1 1 0 0 > > > >> >> ============================================================= > > > >> >> > > > >> >> I don't understand this output...If you have some documentation > > > >> about how > > > >> >> Petsc deals with memory or if you can explain, I would be very > > > >> interested. > > > >> >> > > > >> >> This is a problem because I'm working on IBM SP4, and I've > > > >> limitations of > > > >> >> memory (700mb for data and 700mb for stack). I think the objects > > > >> created by > > > >> >> Petsc is created in the stack memory, but as I cannot determine > > > >> exactly the > > > >> >> memory usage, I cannot verify. > > > >> >> > > > >> >> Thank you for your advices, > > > >> >> > > > >> >> Best regards > > > >> >> > > > >> >> > > > >> > > > > >> > > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > -- ************************************** NGUYEN Anh-Khai Laurent Equipe Support Utilisateur Email : laurent.nguyen at idris.fr Tel : 01.69.35.85.66 Adresse : IDRIS - Institut du D??veloppement et des Ressources en Informatique Scientifique CNRS Batiment 506 BP 167 F - 91403 ORSAY Cedex Site Web: http://www.idris.fr ************************************** From bsmith at mcs.anl.gov Fri Apr 13 21:31:58 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 13 Apr 2007 21:31:58 -0500 (CDT) Subject: Question about memory used by PetsC Objects In-Reply-To: <20070414021641.GA3902@lin2.idris.fr> References: <461E46F5.90606@idris.fr> <461F41D1.3050801@idris.fr> <461F9123.8070908@idris.fr> <20070414021641.GA3902@lin2.idris.fr> Message-ID: Laurent, It is a problem; we just don't have a good solution to the control of memory and measuring how much is used. Barry On Sat, 14 Apr 2007, Laurent Nguyen wrote: > Yes, I know, this is a problem on all computer systems.But, I hoped that your profiling functions could do something like that (this is the name of functions which give me this hope) > > The reason why I need that is because of my computer system (IBM SP4). I've limits of memory and if I use more memory than allowed, the system break my program. For example, in a C code without allocation (only static memory), I can calculate the maximum of memory used by my program. With PetsC, I don't know all the mechanism behind Petsc Objects. This why I sent you theses mails ^^ > > I would have thought this is a problem for users who make computations on this sort of system (with limits etc...), but it seems this is not the case. > > Sorry for the disturbance and thank you for your help > > Laurent > > On Fri, Apr 13, 2007 at 11:14:34AM -0500, Barry Smith wrote: > > > > Matt, > > > > I do not understand why one would want this, doesn't PETSc already keep > > a current amount used (memory malloced but not freed) and the maximum current > > amount used? > > > > NGUYEN, > > > > Once PETSc frees memory (or any program) it is not returned to the operating > > system, rather it is reused in a future malloc if possible. PETSc cannot give > > the TOTAL memory used by the process since that includes pages of the program > > and any memory your code is using and there is no good Unix command to get that > > information on all computer systems. > > > > On Fri, 13 Apr 2007, Matthew Knepley wrote: > > > > > There is a way to get exactly what you want which is not really very hard. > > > Petsc allows you to easily override PetscMalloc() and PetscFree() which > > > we use for everything. You could just keep a running static tally of the sizes > > > and then call the regular version. This would only take a few lines of code. > > > Take a look at > > > > > > http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Sys/PetscSetMalloc.html > > > > > > Matt > > > > > > On 4/13/07, Laurent Nguyen wrote: > > > > Yes, I've tested the argument -memory_info. I've seen the memory given > > > > to me is the same memory given by a PetscMemoryGetCurrentUsage() before > > > > PetscFinalize(). I didn't know it was the same ^^. > > > > > > > > It is not the memory I need. I would like to have the real memory used > > > > by my program (It is a little difficult to explain clearly): for > > > > example, if I create 2 vectors of size 100, the program is consuming > > > > 2*8*100 bytes. But later, when I destroy only 1 vector, the programs is > > > > consuming 1*8*100 bytes. This is this kind of information that I need > > > > (more precisely, the max). I tell you that because, in my context of > > > > computation, I've only 1.5Go of memory per processors. And the memory > > > > given by the argument -memory_info was 2.4Go. So I was perplex about > > > > that. But now I understand how your function is running. > > > > > > > > But thanks to yours functions, I can now better manage the memory of my > > > > objects! > > > > > > > > Regards, > > > > > > > > ************************************** > > > > NGUYEN Anh-Khai Laurent > > > > Equipe Support Utilisateur > > > > > > > > Email : laurent.nguyen at idris.fr > > > > T?l : 01.69.35.85.66 > > > > Adresse : IDRIS - Institut du D?veloppement et des Ressources en > > > > Informatique Scientifique > > > > CNRS > > > > Batiment 506 > > > > BP 167 > > > > F - 91403 ORSAY Cedex > > > > Site Web : http://www.idris.fr > > > > ************************************** > > > > > > > > Matthew Knepley a ?crit : > > > > > On 4/13/07, Laurent Nguyen wrote: > > > > >> Barry, > > > > >> > > > > >> I thank you for your help. So I tried to apply the functions. It seems > > > > >> that PetscMemoryGetCurrentUsage() gives the total amount of memory > > > > >> allocated at the point of the program where it is placed. But, if a > > > > >> object is deallocated, PetscMemoryGetCurrentUsage() doesn't take it. > > > > >> > > > > >> I would like to try PetscMemoryGetMaximumUsage() at the end of the > > > > >> program hoping it gives me the maximum memory used by my program but > > > > >> there are problems. When I put PetscMemoryGetMaximumUsage(), I've this > > > > >> error at the execution: > > > > >> > > > > >> [0]PETSC ERROR: --------------------- Error Message > > > > >> ------------------------------------ > > > > >> [0]PETSC ERROR: Object is in wrong state! > > > > >> [0]PETSC ERROR: To use this function you must first call > > > > >> PetscMemorySetGetMaximumUsage()! > > > > >> > > > > >> So I put PetscMemorySetGetMaximumUsage() before, but at linking, it > > > > >> didn't find PetscMemorySetGetMaximumUsage in the petsc libraries. I > > > > >> thought I made a bad install but when I tried on another machine, I've > > > > >> the same problem. > > > > > > > > > > The function is definitely there (check src/sys/memory/mem.c). You can try > > > > > a short test: > > > > > > > > > > cd src/ksp/ksp/examples/tutorials > > > > > make ex2 > > > > > ./ex2 -memory_info > > > > > ./ex2 -malloc_log > > > > > > > > > > Also, I have fixed the specific logging errors you pointed out in the dev > > > > > copy, but we are doing a code review to make sure no other omissions > > > > > are lurking. > > > > > > > > > > Matt > > > > > > > > > >> Best regards, > > > > >> > > > > >> ************************************** > > > > >> NGUYEN Anh-Khai Laurent > > > > >> Equipe Support Utilisateur > > > > >> > > > > >> Email : laurent.nguyen at idris.fr > > > > >> T?l : 01.69.35.85.66 > > > > >> Adresse : IDRIS - Institut du D?veloppement et des Ressources en > > > > >> Informatique Scientifique > > > > >> CNRS > > > > >> Batiment 506 > > > > >> BP 167 > > > > >> F - 91403 ORSAY Cedex > > > > >> Site Web : http://www.idris.fr > > > > >> ************************************** > > > > >> > > > > >> Barry Smith a ?crit : > > > > >> > Laurent, > > > > >> > > > > > >> > The memory usage printed with -log_summary is not correct. > > > > >> > Much of the memory used is not printed out. > > > > >> > > > > > >> > You can use PetscMemoryGetCurrentUsage() and > > > > >> PetscMemoryGetMaximumUsage() > > > > >> > in your code before and after creating a filling in PETSc objects > > > > >> > to see how much memory they are used. You can also use > > > > >> PetscMallocDump() > > > > >> > at any point to see exactly where in PETSc and how much all of PETSc's > > > > >> > memory was allocated. > > > > >> > > > > > >> > Barry > > > > >> > > > > > >> > > > > > >> > On Thu, 12 Apr 2007, Laurent Nguyen wrote: > > > > >> > > > > > >> >> Hi, > > > > >> >> > > > > >> >> I'm a Petsc user, but I'm beginning to use petsc in a supercomputation > > > > >> >> context. So, I'm trying parallel versions of my old mono-processed > > > > >> programs. > > > > >> >> But I've some difficulties to determine how much memory my program > > > > >> use. (I > > > > >> >> give a little example). In this example on one processor (with > > > > >> these command > > > > >> >> line arguments: -mat_type mpiaij -pc_type none -log_summary), I > > > > >> created one > > > > >> >> mpiaij matrix (size = 500*500) and two vectors (size = 500). I've this > > > > >> >> information from output: > > > > >> >> > > > > >> >> Matrix Object: > > > > >> >> type=mpiaij, rows=250000, cols=250000 > > > > >> >> total: nonzeros=749996, allocated nonzeros=1750000 > > > > >> >> > > > > >> >> To me, there is an array of 1750000 double precision number of 8 > > > > >> bytes = > > > > >> >> 14000000 bytes (14Mo). > > > > >> >> > > > > >> >> Same for vectors: 2*8*250000 = 4 Mo > > > > >> >> > > > > >> >> But in the log summary, I've: > > > > >> >> > > > > >> >> Memory usage is given in bytes: > > > > >> >> > > > > >> >> Object Type Creations Destructions Memory > > > > >> Descendants' Mem. > > > > >> >> > > > > >> >> --- Event Stage 0: Main Stage > > > > >> >> > > > > >> >> Viewer 1 1 0 0 > > > > >> >> Index Set 2 2 672 0 > > > > >> >> Map 16 16 4992 0 > > > > >> >> Vec 10 10 2006576 0 > > > > >> >> Vec Scatter 1 1 0 0 > > > > >> >> Matrix 3 3 1068 0 > > > > >> >> Krylov Solver 1 1 17208 0 > > > > >> >> Preconditioner 1 1 0 0 > > > > >> >> ============================================================= > > > > >> >> > > > > >> >> I don't understand this output...If you have some documentation > > > > >> about how > > > > >> >> Petsc deals with memory or if you can explain, I would be very > > > > >> interested. > > > > >> >> > > > > >> >> This is a problem because I'm working on IBM SP4, and I've > > > > >> limitations of > > > > >> >> memory (700mb for data and 700mb for stack). I think the objects > > > > >> created by > > > > >> >> Petsc is created in the stack memory, but as I cannot determine > > > > >> exactly the > > > > >> >> memory usage, I cannot verify. > > > > >> >> > > > > >> >> Thank you for your advices, > > > > >> >> > > > > >> >> Best regards > > > > >> >> > > > > >> >> > > > > >> > > > > > >> > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From pbauman at ices.utexas.edu Sat Apr 14 15:52:34 2007 From: pbauman at ices.utexas.edu (Paul T. Bauman) Date: Sat, 14 Apr 2007 15:52:34 -0500 Subject: Preallocation for external direct solvers Message-ID: <46213F12.9050906@ices.utexas.edu> Hello, I'm developing some code and am using direct solvers (for the moment), in particular MUMPS and SuperLU. I noticed that PETSc is not preallocating the memory for it even though I specify maximum number of entries per row. call MatCreateSeqAIJ(PETSC_COMM_WORLD,total_num_dofs,total_num_dofs, & max_number_nonzeros_row,PETSC_NULL_INTEGER, J ,ierr) call MatSetFromOptions(J,ierr) call SNESSetJacobian(snes,J,J,FormJacobian,PETSC_NULL_OBJECT,ierr) When I print -info: [0] User provided function(): (Fortran):PETSc successfully started: procs 1 [0] User provided function(): Running on machine: dhcp-67-30.ices.utexas.edu [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374784 max tags = 2147483647 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374784 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374784 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374784 [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374784 [0] MatConvert_AIJ_AIJMUMPS(): Using MUMPS for LU factorization and solves. [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374784 0 SNES Function norm 8.500000000000e-01 [0] MatSetUpPreallocation(): Warning not preallocating matrix storage So, of course, this really slows down the code for even small problems (20000 dof). Same story for SuperLU. Should I use a different call to do the preallocation? Thanks for any suggestions. Best, Paul From bsmith at mcs.anl.gov Sat Apr 14 16:14:03 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 14 Apr 2007 16:14:03 -0500 (CDT) Subject: Preallocation for external direct solvers In-Reply-To: <46213F12.9050906@ices.utexas.edu> References: <46213F12.9050906@ices.utexas.edu> Message-ID: Paul, This is a bug in how MatSetFromOptions() changes the type. You can use instead MatConvert(J,MATAIJMUMPS,MAT_REUSE_MATRIX,J,ierr) for now and I will investigate how the bug can be fixed. Another, maybe better alternative is to use MatCreate(comm,J,ierr) MatSetSizes(J,....) MatSetFromOptions(J,ierr) MatSeqAIJSetPreallocation(....) MatMPIAIJSetPreallocation(...) Sorry and thanks for reporting the problem, Barry The bug is that MatSetType() does not know that AIJMUMPS is inherited from AIJ and hence destroys the AIJ data structure (including the preallocation information) before creating a whole new AIJMUMPS matrix. On Sat, 14 Apr 2007, Paul T. Bauman wrote: > Hello, > > I'm developing some code and am using direct solvers (for the moment), in > particular MUMPS and SuperLU. I noticed that PETSc is not preallocating the > memory for it even though I specify maximum number of entries per row. > > call MatCreateSeqAIJ(PETSC_COMM_WORLD,total_num_dofs,total_num_dofs, & > max_number_nonzeros_row,PETSC_NULL_INTEGER, J ,ierr) > > call MatSetFromOptions(J,ierr) > > call SNESSetJacobian(snes,J,J,FormJacobian,PETSC_NULL_OBJECT,ierr) > > When I print -info: > > [0] User provided function(): (Fortran):PETSc successfully started: procs 1 > [0] User provided function(): Running on machine: dhcp-67-30.ices.utexas.edu > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374784 > max tags = 2147483647 > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374784 > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374784 > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374784 > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374784 > [0] MatConvert_AIJ_AIJMUMPS(): Using MUMPS for LU factorization and solves. > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374784 > 0 SNES Function norm 8.500000000000e-01 > [0] MatSetUpPreallocation(): Warning not preallocating matrix storage > > So, of course, this really slows down the code for even small problems (20000 > dof). Same story for SuperLU. Should I use a different call to do the > preallocation? > > Thanks for any suggestions. > > Best, > > Paul > > From pbauman at ices.utexas.edu Sat Apr 14 17:01:58 2007 From: pbauman at ices.utexas.edu (Paul T. Bauman) Date: Sat, 14 Apr 2007 17:01:58 -0500 Subject: Preallocation for external direct solvers In-Reply-To: References: <46213F12.9050906@ices.utexas.edu> Message-ID: <46214F56.6020303@ices.utexas.edu> Barry Smith wrote: > Paul, > > This is a bug in how MatSetFromOptions() changes the type. You can > use instead > MatConvert(J,MATAIJMUMPS,MAT_REUSE_MATRIX,J,ierr) > for now and I will investigate how the bug can be fixed. > > Another, maybe better alternative is to use > MatCreate(comm,J,ierr) > MatSetSizes(J,....) > MatSetFromOptions(J,ierr) > MatSeqAIJSetPreallocation(....) > MatMPIAIJSetPreallocation(...) > This option worked well. Thank you very much. Paul > Sorry and thanks for reporting the problem, > > Barry > > The bug is that MatSetType() does not know that AIJMUMPS is inherited from > AIJ and hence destroys the AIJ data structure (including the preallocation > information) before creating a whole new AIJMUMPS matrix. > > On Sat, 14 Apr 2007, Paul T. Bauman wrote: > > >> Hello, >> >> I'm developing some code and am using direct solvers (for the moment), in >> particular MUMPS and SuperLU. I noticed that PETSc is not preallocating the >> memory for it even though I specify maximum number of entries per row. >> >> call MatCreateSeqAIJ(PETSC_COMM_WORLD,total_num_dofs,total_num_dofs, & >> max_number_nonzeros_row,PETSC_NULL_INTEGER, J ,ierr) >> >> call MatSetFromOptions(J,ierr) >> >> call SNESSetJacobian(snes,J,J,FormJacobian,PETSC_NULL_OBJECT,ierr) >> >> When I print -info: >> >> [0] User provided function(): (Fortran):PETSc successfully started: procs 1 >> [0] User provided function(): Running on machine: dhcp-67-30.ices.utexas.edu >> [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374784 >> max tags = 2147483647 >> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 >> -2080374784 >> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 >> -2080374784 >> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 >> -2080374784 >> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 >> -2080374784 >> [0] MatConvert_AIJ_AIJMUMPS(): Using MUMPS for LU factorization and solves. >> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 >> -2080374784 >> 0 SNES Function norm 8.500000000000e-01 >> [0] MatSetUpPreallocation(): Warning not preallocating matrix storage >> >> So, of course, this really slows down the code for even small problems (20000 >> dof). Same story for SuperLU. Should I use a different call to do the >> preallocation? >> >> Thanks for any suggestions. >> >> Best, >> >> Paul >> >> >> From zonexo at gmail.com Sun Apr 15 22:01:59 2007 From: zonexo at gmail.com (Ben Tay) Date: Mon, 16 Apr 2007 11:01:59 +0800 Subject: Deleting lapack*, blas* etc Message-ID: <804ab5d40704152001h12629221ybcda84a9ccb07bd7@mail.gmail.com> Hi, I've finished compiling PETSc and it use the fblaslapack external package. May I know if I can delete the directories blas1-3, lapack1-6 and the lapack-extra1-4? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Apr 15 22:17:09 2007 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 15 Apr 2007 22:17:09 -0500 Subject: Deleting lapack*, blas* etc In-Reply-To: <804ab5d40704152001h12629221ybcda84a9ccb07bd7@mail.gmail.com> References: <804ab5d40704152001h12629221ybcda84a9ccb07bd7@mail.gmail.com> Message-ID: Its a really small package. Do you need memory that bad? Deleting things will wreck the ability to rebuild. Matt On 4/15/07, Ben Tay wrote: > Hi, > > I've finished compiling PETSc and it use the fblaslapack external package. > May I know if I can delete the directories blas1-3, lapack1-6 and the > lapack-extra1-4? > > Thanks -- The government saving money is like me spilling beer. It happens, but never on purpose. From h.a.slim at durham.ac.uk Mon Apr 16 10:49:21 2007 From: h.a.slim at durham.ac.uk (SLIM H.A.) Date: Mon, 16 Apr 2007 16:49:21 +0100 Subject: building petsc for myrinet Message-ID: Dear users I have been able to build petsc for ethernet interconnect with static and shared object libraries using the intel compiler and intel's mkl libraries for lapack/blas. Now I want to build the libraries for mpi over myrinet as well. Using the same configuration arguments as before (and mpiCC wrapping intel compiler and myrinet libraries) eg >./config/configure.py --with-blas-lapack-lib=\[/usr/local/Cluster-Apps/intel/mkl/8.0/lib/em64t /libmkl_lapack64.so\,libmkl_def.so\,libguide.so\] --with-vendor-compilers=intel --with-gnu-compilers=0 --with-shared now I get the error ************************************************************************ ********* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------ --------------- You set a value for --with-blas-lapack-lib=, but ['/usr/local/Cluster-Apps/intel/mkl/8.0/lib/em64t/libmkl_lapack64.so', 'libmkl_def.so', 'libguide.so'] cannot be used ************************************************************************ ********* Examining the configure.log shows that a reference to a pthread function is undefined: sh: Possible ERROR while running linker: /usr/local/Cluster-Apps/intel/mkl/8.0/lib/em64t/libguide.so: undefined reference to `pthread_atfork' output: ret = 256 error message = {/usr/local/Cluster-Apps/intel/mkl/8.0/lib/em64t/libguide.so: undefined reference to `pthread_atfork' Should pthread.so added somewhere in the configuration? Thanks for any advice Henk From balay at mcs.anl.gov Mon Apr 16 11:44:10 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 16 Apr 2007 11:44:10 -0500 (CDT) Subject: building petsc for myrinet In-Reply-To: References: Message-ID: > {/usr/local/Cluster-Apps/intel/mkl/8.0/lib/em64t/libguide.so: undefined > reference to `pthread_atfork' try using: --with-blas-lapack-lib=\[/usr/local/Cluster-Apps/intel/mkl/8.0/lib/em64t/libmkl_lapack64.so,libmkl_def.so\,libguide.so\,-lpthread\] or just: --with-blas-lapack-dir=/usr/local/Cluster-Apps/intel/mkl/8.0 Satish On Mon, 16 Apr 2007, SLIM H.A. wrote: > Dear users > > I have been able to build petsc for ethernet interconnect with static > and shared object libraries using the intel compiler and intel's mkl > libraries for lapack/blas. > > Now I want to build the libraries for mpi over myrinet as well. > Using the same configuration arguments as before (and mpiCC wrapping > intel compiler and myrinet libraries) eg > > >./config/configure.py > --with-blas-lapack-lib=\[/usr/local/Cluster-Apps/intel/mkl/8.0/lib/em64t > /libmkl_lapack64.so\,libmkl_def.so\,libguide.so\] > --with-vendor-compilers=intel --with-gnu-compilers=0 --with-shared > > now I get the error > > ************************************************************************ > ********* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > for details): > ------------------------------------------------------------------------ > --------------- > You set a value for --with-blas-lapack-lib=, but > ['/usr/local/Cluster-Apps/intel/mkl/8.0/lib/em64t/libmkl_lapack64.so', > 'libmkl_def.so', 'libguide.so'] cannot be used > ************************************************************************ > ********* > > Examining the configure.log shows that a reference to a pthread function > is undefined: > > sh: > Possible ERROR while running linker: > /usr/local/Cluster-Apps/intel/mkl/8.0/lib/em64t/libguide.so: undefined > reference to `pthread_atfork' > output: ret = 256 > error message = > {/usr/local/Cluster-Apps/intel/mkl/8.0/lib/em64t/libguide.so: undefined > reference to `pthread_atfork' > > > Should pthread.so added somewhere in the configuration? > > Thanks for any advice > > Henk > > From randy at geosystem.us Wed Apr 18 10:27:32 2007 From: randy at geosystem.us (Randall Mackie) Date: Wed, 18 Apr 2007 08:27:32 -0700 Subject: PetscViewerBinaryWrite Message-ID: <462638E4.2080609@geosystem.us> In my fortran program, I save the results of calculations in a binary file for later use, like this: IF (rank == 0) THEN irec = irec + 1 write(7,rec=irec) hsavx(:,ipol) irec = irec + 1 write(7,rec=irec) hsavy(:,ipol) irec = irec + 1 write(7,rec=irec) hsavz(:,ipol) END IF And then read them back later in a similar fashion. Is it possible to replace this with PetscViewerBinaryWrite? There is a flag in this call called istemp, which specifies if the data can be overwritten. I believe if you set this to false, then with each call you would simply append the vector to the file? My question is when you go back to re-read in these vectors, how does PetscViewerBinaryRead know where to start? In other words, I don't see a way to specify to start at the beginning, or rewind the file, or maybe these Fortran concepts are not valid? Thanks for any suggestions. Randy -- Randall Mackie GSY-USA, Inc. PMB# 643 2261 Market St., San Francisco, CA 94114-1600 Tel (415) 469-8649 Fax (415) 469-5044 California Registered Geophysicist License No. GP 1034 From randy at geosystem.us Wed Apr 18 11:45:43 2007 From: randy at geosystem.us (Randall Mackie) Date: Wed, 18 Apr 2007 09:45:43 -0700 Subject: create parallel vectors (allocatable type question) Message-ID: <46264B37.1000102@geosystem.us> I want to create a doubly-indexed MPI global vector in my Fortran code, but I won't know the number to create until I start the program (it is model dependent). In Fortran, I would use allocatable type statements, and just allocate the number once known. (This is different from allocating the length of the vector). If I knew I wanted 5 of these, I could say: Vec X(5) And then create them later with calls to VecCreateMPI, and VecDuplicate, for example. In my case, I want to do something like: Vec X(:) And then determine the number later in my code. Is there a way to do this that I'm just overlooking? Do I just need to use a larger number than expected, and then create only the number of vectors actually needed later on with VecCreateMPI and VecDuplicate? Thanks, Randy -- Randall Mackie GSY-USA, Inc. PMB# 643 2261 Market St., San Francisco, CA 94114-1600 Tel (415) 469-8649 Fax (415) 469-5044 California Registered Geophysicist License No. GP 1034 From knepley at gmail.com Wed Apr 18 14:00:55 2007 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 18 Apr 2007 14:00:55 -0500 Subject: PetscViewerBinaryWrite In-Reply-To: <462638E4.2080609@geosystem.us> References: <462638E4.2080609@geosystem.us> Message-ID: On 4/18/07, Randall Mackie wrote: > In my fortran program, I save the results of calculations in a binary file > for later use, like this: > > IF (rank == 0) THEN > irec = irec + 1 > write(7,rec=irec) hsavx(:,ipol) > irec = irec + 1 > write(7,rec=irec) hsavy(:,ipol) > irec = irec + 1 > write(7,rec=irec) hsavz(:,ipol) > END IF > > And then read them back later in a similar fashion. > > > Is it possible to replace this with PetscViewerBinaryWrite? Should be. > There is a flag in this call called istemp, which specifies if the > data can be overwritten. I believe if you set this to false, then > with each call you would simply append the vector to the file? Actually, the flag is a little more obscure than that. You should just not set it. You would open a ViewerBinary and then use VecView() I think. > My question is when you go back to re-read in these vectors, how > does PetscViewerBinaryRead know where to start? In other words, > I don't see a way to specify to start at the beginning, or rewind > the file, or maybe these Fortran concepts are not valid? I am not sure I understand this part. However, I think what you want is to close the inital viewer when you are done writing. When you want to read in, you open a new viewer and use VecLoad(). If you need something fancier, there are functions for changing the file offset, but that demands knowedge of the underlying format. Matt > Thanks for any suggestions. > > Randy > > -- > Randall Mackie > GSY-USA, Inc. > PMB# 643 > 2261 Market St., > San Francisco, CA 94114-1600 > Tel (415) 469-8649 > Fax (415) 469-5044 > > California Registered Geophysicist > License No. GP 1034 > > -- The government saving money is like me spilling beer. It happens, but never on purpose. From bsmith at mcs.anl.gov Wed Apr 18 14:23:04 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 18 Apr 2007 14:23:04 -0500 (CDT) Subject: create parallel vectors (allocatable type question) In-Reply-To: <46264B37.1000102@geosystem.us> References: <46264B37.1000102@geosystem.us> Message-ID: On Wed, 18 Apr 2007, Randall Mackie wrote: > I want to create a doubly-indexed MPI global vector in my Fortran code, > but I won't know the number to create until I start the program (it is > model dependent). In Fortran, I would use allocatable type statements, > and just allocate the number once known. (This is different from allocating > the length of the vector). > > If I knew I wanted 5 of these, I could say: > > Vec X(5) > > And then create them later with calls to VecCreateMPI, and VecDuplicate, for > example. > > In my case, I want to do something like: > > Vec X(:) > > And then determine the number later in my code. You can use fortran allocation, I think something like Vec, pointer :: X(:) n = 22 allocate(X(n)) VecCreate(....,x(2),ierr) Barry > > Is there a way to do this that I'm just overlooking? Do I just need to > use a larger number than expected, and then create only the number > of vectors actually needed later on with VecCreateMPI and VecDuplicate? > > Thanks, Randy > > From timothy.stitt at ichec.ie Sat Apr 21 06:50:21 2007 From: timothy.stitt at ichec.ie (Dr. Timothy Stitt) Date: Sat, 21 Apr 2007 12:50:21 +0100 Subject: CSR Sparse Matrix Query Message-ID: <4629FA7D.9020701@ichec.ie> Hi all, I have just begun to use PETSc and I have already ran into a problem. Apologies if this has been dealt with before. My code currently generates its own non-distributed Hamiltonian sparse matrix in CSR format (due to the large size of the dense representation). I want to exploit PETSc's (actually SLEPc's) eigensolver routines and hence I need to convert my CSR representation into PETSc CSR representation for use by the eigensolver. From reading the documentation I am under the impression that you initially supply the dense matrix to the PETSc matrix routines for conversion to PETSc CSR representation. In my case I do not generate the dense matrix. I would be grateful if someone could guide me on to how to create a PETSC sparse matrix from my own in-code generated sparse matrix. Thanks in advance for any assistance that can be provided. Regards, Tim. -- Dr. Timothy Stitt HPC Application Consultant - ICHEC (www.ichec.ie) Dublin Institute for Advanced Studies 5 Merrion Square - Dublin 2 - Ireland +353-1-6621333 (tel) / +353-1-6621477 (fax) / +353-874195427 (mobile) From randy at geosystem.us Sat Apr 21 08:53:28 2007 From: randy at geosystem.us (Randall Mackie) Date: Sat, 21 Apr 2007 06:53:28 -0700 Subject: CSR Sparse Matrix Query In-Reply-To: <4629FA7D.9020701@ichec.ie> References: <4629FA7D.9020701@ichec.ie> Message-ID: <462A1758.9010506@geosystem.us> Hi Tim, There's no need to generate a dense matrix if you're dealing with sparse matrices. There are many approaches, but if you just want to get your existing matrix into PETSc, you can use something like the following (written in Fortran, but almost the same in C): Assume the global vector length is np: call VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,np,b,ierr) call VecDuplicate(b,xsol,ierr) call VecGetLocalSize(b,mloc,ierr) call VecGetOwnershipRange(b,Istart,Iend,ierr) do i=Istart+1,Iend loc(i)=i-1 end do ALLOCATE (d_nnz(mloc), o_nnz(mloc), STAT=istat) Note: the d_nnz and o_nnz are used to determine the pre-allocation of the global matrix according to the PETSc manual (basically just looping through and figuring out which elements are in d_nnz and which are in o_nnz). Then you can create the matrix: call MatCreateMPIAIJ(PETSC_COMM_WORLD,mloc,mloc,np,np, . PETSC_NULL_INTEGER, d_nnz, PETSC_NULL_INTEGER, . o_nnz,P,ierr) And then I had a subroutine to fill it like this (the critical part is the check whether that part of the matrix is owned by the current processor): jj=0 do i=1,l do k=2,n do j=2,m jj=jj+1 row = jj-1 IF (jj >= Istart+1 .and. jj <= Iend) THEN if(k.ne.2) then fact=-rijcy*dx*dy/z(k-1) ! Hxijc ic=ic+1 v(ic)=fact col(ic)=ijkhx(i,j,k-1)-1 end if [snip code] call MatSetValues(A,i1,row,ic,col,v,INSERT_VALUES, . ierr) [and finally at the end you assemble the matrix] call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) Since these early days, I've switched over to using Distributed Arrays, which in my opinion are much better and much easier to use. Hope this helps, Randy Dr. Timothy Stitt wrote: > Hi all, > > I have just begun to use PETSc and I have already ran into a problem. > Apologies if this has been dealt with before. > > My code currently generates its own non-distributed Hamiltonian sparse > matrix in CSR format (due to the large size of the dense > representation). I want to exploit PETSc's (actually SLEPc's) > eigensolver routines and hence I need to convert my CSR representation > into PETSc CSR representation for use by the eigensolver. > From reading the documentation I am under the impression that you > initially supply the dense matrix to the PETSc matrix routines for > conversion to PETSc CSR representation. In my case I do not generate the > dense matrix. I would be grateful if someone could guide me on to how to > create a PETSC sparse matrix from my own in-code generated sparse matrix. > > Thanks in advance for any assistance that can be provided. > > Regards, > > Tim. > -- Randall Mackie GSY-USA, Inc. PMB# 643 2261 Market St., San Francisco, CA 94114-1600 Tel (415) 469-8649 Fax (415) 469-5044 California Registered Geophysicist License No. GP 1034 From timothy.stitt at ichec.ie Sat Apr 21 09:15:52 2007 From: timothy.stitt at ichec.ie (Tim Stitt) Date: Sat, 21 Apr 2007 15:15:52 +0100 Subject: CSR Sparse Matrix Query In-Reply-To: <462A1758.9010506@geosystem.us> References: <4629FA7D.9020701@ichec.ie> <462A1758.9010506@geosystem.us> Message-ID: <462A1C98.8030506@ichec.ie> Randall, Many thanks for that...very helpful indeed. Just out of curiosity...what do you mean by "Distributed Arrays". This is not the same as PETSc distributed arrays which I thought are implicit in the Matrix and Vector generation routines (depending on whether multiple nodes are available or not)? Regards, Tim. Randall Mackie wrote: > Hi Tim, > > There's no need to generate a dense matrix if you're dealing with sparse > matrices. There are many approaches, but if you just want to get your > existing matrix into PETSc, you can use something like the following > (written > in Fortran, but almost the same in C): > > Assume the global vector length is np: > > call VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,np,b,ierr) > call VecDuplicate(b,xsol,ierr) > call VecGetLocalSize(b,mloc,ierr) > call VecGetOwnershipRange(b,Istart,Iend,ierr) > > do i=Istart+1,Iend > loc(i)=i-1 > end do > > > ALLOCATE (d_nnz(mloc), o_nnz(mloc), STAT=istat) > > Note: the d_nnz and o_nnz are used to determine the pre-allocation of > the global matrix > according to the PETSc manual (basically just looping through and > figuring out which > elements are in d_nnz and which are in o_nnz). > > Then you can create the matrix: > > > call MatCreateMPIAIJ(PETSC_COMM_WORLD,mloc,mloc,np,np, > . PETSC_NULL_INTEGER, d_nnz, PETSC_NULL_INTEGER, > . o_nnz,P,ierr) > > > And then I had a subroutine to fill it like this (the critical part is > the check > whether that part of the matrix is owned by the current processor): > > jj=0 > > do i=1,l > do k=2,n > do j=2,m > > jj=jj+1 > row = jj-1 > > IF (jj >= Istart+1 .and. jj <= Iend) THEN > > if(k.ne.2) then > fact=-rijcy*dx*dy/z(k-1) ! Hxijc > ic=ic+1 > v(ic)=fact > col(ic)=ijkhx(i,j,k-1)-1 > end if > > [snip code] > > call MatSetValues(A,i1,row,ic,col,v,INSERT_VALUES, > . ierr) > > > [and finally at the end you assemble the matrix] > > call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) > call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) > > > Since these early days, I've switched over to using Distributed Arrays, > which in my opinion are much better and much easier to use. > > Hope this helps, > > Randy > > > Dr. Timothy Stitt wrote: >> Hi all, >> >> I have just begun to use PETSc and I have already ran into a problem. >> Apologies if this has been dealt with before. >> >> My code currently generates its own non-distributed Hamiltonian >> sparse matrix in CSR format (due to the large size of the dense >> representation). I want to exploit PETSc's (actually SLEPc's) >> eigensolver routines and hence I need to convert my CSR >> representation into PETSc CSR representation for use by the eigensolver. >> From reading the documentation I am under the impression that you >> initially supply the dense matrix to the PETSc matrix routines for >> conversion to PETSc CSR representation. In my case I do not generate >> the dense matrix. I would be grateful if someone could guide me on to >> how to create a PETSC sparse matrix from my own in-code generated >> sparse matrix. >> >> Thanks in advance for any assistance that can be provided. >> >> Regards, >> >> Tim. >> > -- Dr. Timothy Stitt HPC Application Consultant - ICHEC (www.ichec.ie) Dublin Institute for Advanced Studies 5 Merrion Square - Dublin 2 - Ireland +353-1-6621333 (tel) / +353-1-6621477 (fax) / +353-874195427 (mobile) From randy at geosystem.us Sat Apr 21 09:29:28 2007 From: randy at geosystem.us (Randall Mackie) Date: Sat, 21 Apr 2007 07:29:28 -0700 Subject: CSR Sparse Matrix Query In-Reply-To: <462A1C98.8030506@ichec.ie> References: <4629FA7D.9020701@ichec.ie> <462A1758.9010506@geosystem.us> <462A1C98.8030506@ichec.ie> Message-ID: <462A1FC8.9050304@geosystem.us> Hi Tim, The way I understand it, Distributed Arrays are used for structured problems and they automatically take care of ghost values and so forth, and for me were much more efficient and easier to use (once I got them figured out, but then I'm not an expert). There is a good section on Distributed Arrays in the users manual. So, now in my more recent code, I replaced what I outlined in the first email with calls like: call DACreate3d(PETSC_COMM_WORLD,DA_NONPERIODIC,DA_STENCIL_STAR, . l+1,m+1,nzn,PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,i3,i1, . PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, . da,ierr) call DACreateGlobalVector(da,b,ierr) call VecDuplicate(b,xsol,ierr) ngrow=3*(l+1)*(m+1)*(n+1) call MatCreateMPIAIJ(PETSC_COMM_WORLD,mloc,mloc,ngrow,ngrow, . i15,PETSC_NULL_INTEGER,i7,PETSC_NULL_INTEGER,A,ierr) call DAGetInfo(da,PETSC_NULL_INTEGER,mx,my,mz, . PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, . PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, . PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, . PETSC_NULL_INTEGER,ierr) call DAGetCorners(da,xs,ys,zs,xm,ym,zm,ierr) call DAGetGhostCorners(da,gxs,gys,gzs,gxm,gym,gzm,ierr) [and then to set the matrix elements, calls like]: do kk=zs,zs+zm-1 do jj=ys,ys+ym-1 do ii=xs,xs+xm-1 row=ii-gxs + (jj-gys)*gxm + (kk-gzs)*gxm*gym i=ii+1 j=jj+1 k=kk+1 !--Begin-Hx--==================================================================== grow=ltog(idltog + 3*row + 1) IF (jj == 0 .or. jj == my-1 .or. kk == 0 .or. . ii == mx-1) THEN v(1) = 1.d0 col(1) = grow call MatSetValues(A,i1,grow,i1,col,v,INSERT_VALUES, . ierr) ELSE fact=-rijcy*dx*dy/z(k-1) ! Hxijc ic=ic+1 v(ic)=fact col(ic)=ltog(idltog + 3*(row-gxm*gym) +1) [etc] Regards, Randy Tim Stitt wrote: > Randall, > > Many thanks for that...very helpful indeed. > > Just out of curiosity...what do you mean by "Distributed Arrays". This > is not the same as PETSc distributed arrays which I thought are implicit > in the Matrix and Vector generation routines (depending on whether > multiple nodes are available or not)? > > Regards, > > Tim. > > Randall Mackie wrote: >> Hi Tim, >> >> There's no need to generate a dense matrix if you're dealing with sparse >> matrices. There are many approaches, but if you just want to get your >> existing matrix into PETSc, you can use something like the following >> (written >> in Fortran, but almost the same in C): >> >> Assume the global vector length is np: >> >> call VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,np,b,ierr) >> call VecDuplicate(b,xsol,ierr) >> call VecGetLocalSize(b,mloc,ierr) >> call VecGetOwnershipRange(b,Istart,Iend,ierr) >> >> do i=Istart+1,Iend >> loc(i)=i-1 >> end do >> >> >> ALLOCATE (d_nnz(mloc), o_nnz(mloc), STAT=istat) >> >> Note: the d_nnz and o_nnz are used to determine the pre-allocation of >> the global matrix >> according to the PETSc manual (basically just looping through and >> figuring out which >> elements are in d_nnz and which are in o_nnz). >> >> Then you can create the matrix: >> >> >> call MatCreateMPIAIJ(PETSC_COMM_WORLD,mloc,mloc,np,np, >> . PETSC_NULL_INTEGER, d_nnz, PETSC_NULL_INTEGER, >> . o_nnz,P,ierr) >> >> >> And then I had a subroutine to fill it like this (the critical part is >> the check >> whether that part of the matrix is owned by the current processor): >> >> jj=0 >> >> do i=1,l >> do k=2,n >> do j=2,m >> >> jj=jj+1 >> row = jj-1 >> >> IF (jj >= Istart+1 .and. jj <= Iend) THEN >> >> if(k.ne.2) then >> fact=-rijcy*dx*dy/z(k-1) ! Hxijc >> ic=ic+1 >> v(ic)=fact >> col(ic)=ijkhx(i,j,k-1)-1 >> end if >> >> [snip code] >> >> call MatSetValues(A,i1,row,ic,col,v,INSERT_VALUES, >> . ierr) >> >> >> [and finally at the end you assemble the matrix] >> >> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) >> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) >> >> >> Since these early days, I've switched over to using Distributed Arrays, >> which in my opinion are much better and much easier to use. >> >> Hope this helps, >> >> Randy >> >> >> Dr. Timothy Stitt wrote: >>> Hi all, >>> >>> I have just begun to use PETSc and I have already ran into a problem. >>> Apologies if this has been dealt with before. >>> >>> My code currently generates its own non-distributed Hamiltonian >>> sparse matrix in CSR format (due to the large size of the dense >>> representation). I want to exploit PETSc's (actually SLEPc's) >>> eigensolver routines and hence I need to convert my CSR >>> representation into PETSc CSR representation for use by the eigensolver. >>> From reading the documentation I am under the impression that you >>> initially supply the dense matrix to the PETSc matrix routines for >>> conversion to PETSc CSR representation. In my case I do not generate >>> the dense matrix. I would be grateful if someone could guide me on to >>> how to create a PETSC sparse matrix from my own in-code generated >>> sparse matrix. >>> >>> Thanks in advance for any assistance that can be provided. >>> >>> Regards, >>> >>> Tim. >>> >> > > -- Randall Mackie GSY-USA, Inc. PMB# 643 2261 Market St., San Francisco, CA 94114-1600 Tel (415) 469-8649 Fax (415) 469-5044 California Registered Geophysicist License No. GP 1034 From bsmith at mcs.anl.gov Sat Apr 21 10:56:34 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 21 Apr 2007 10:56:34 -0500 (CDT) Subject: CSR Sparse Matrix Query In-Reply-To: <462A1758.9010506@geosystem.us> References: <4629FA7D.9020701@ichec.ie> <462A1758.9010506@geosystem.us> Message-ID: You can also use MatCreateSeqAIJWithArrays() and in parallel MatCreateMPIAIJWithArrays(). These are intended for those who already form the matrix in CSR format. Barry On Sat, 21 Apr 2007, Randall Mackie wrote: > Hi Tim, > > There's no need to generate a dense matrix if you're dealing with sparse > matrices. There are many approaches, but if you just want to get your > existing matrix into PETSc, you can use something like the following (written > in Fortran, but almost the same in C): > > Assume the global vector length is np: > > call VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,np,b,ierr) > call VecDuplicate(b,xsol,ierr) > call VecGetLocalSize(b,mloc,ierr) > call VecGetOwnershipRange(b,Istart,Iend,ierr) > > do i=Istart+1,Iend > loc(i)=i-1 > end do > > > ALLOCATE (d_nnz(mloc), o_nnz(mloc), STAT=istat) > > Note: the d_nnz and o_nnz are used to determine the pre-allocation of the > global matrix > according to the PETSc manual (basically just looping through and figuring out > which > elements are in d_nnz and which are in o_nnz). > > Then you can create the matrix: > > > call MatCreateMPIAIJ(PETSC_COMM_WORLD,mloc,mloc,np,np, > . PETSC_NULL_INTEGER, d_nnz, PETSC_NULL_INTEGER, > . o_nnz,P,ierr) > > > And then I had a subroutine to fill it like this (the critical part is the > check > whether that part of the matrix is owned by the current processor): > > jj=0 > > do i=1,l > do k=2,n > do j=2,m > > jj=jj+1 > row = jj-1 > > IF (jj >= Istart+1 .and. jj <= Iend) THEN > > if(k.ne.2) then > fact=-rijcy*dx*dy/z(k-1) ! Hxijc > ic=ic+1 > v(ic)=fact > col(ic)=ijkhx(i,j,k-1)-1 > end if > > [snip code] > > call MatSetValues(A,i1,row,ic,col,v,INSERT_VALUES, > . ierr) > > > [and finally at the end you assemble the matrix] > > call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) > call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) > > > Since these early days, I've switched over to using Distributed Arrays, > which in my opinion are much better and much easier to use. > > Hope this helps, > > Randy > > > Dr. Timothy Stitt wrote: > > Hi all, > > > > I have just begun to use PETSc and I have already ran into a problem. > > Apologies if this has been dealt with before. > > > > My code currently generates its own non-distributed Hamiltonian sparse > > matrix in CSR format (due to the large size of the dense representation). I > > want to exploit PETSc's (actually SLEPc's) eigensolver routines and hence I > > need to convert my CSR representation into PETSc CSR representation for use > > by the eigensolver. > > From reading the documentation I am under the impression that you initially > > supply the dense matrix to the PETSc matrix routines for conversion to PETSc > > CSR representation. In my case I do not generate the dense matrix. I would > > be grateful if someone could guide me on to how to create a PETSC sparse > > matrix from my own in-code generated sparse matrix. > > > > Thanks in advance for any assistance that can be provided. > > > > Regards, > > > > Tim. > > > > From mfhoel at ifi.uio.no Sun Apr 22 19:36:19 2007 From: mfhoel at ifi.uio.no (Mads Fredrik Skoge Hoel) Date: Mon, 23 Apr 2007 02:36:19 +0200 (CEST) Subject: CSR Sparse Matrix Query In-Reply-To: References: <4629FA7D.9020701@ichec.ie> <462A1758.9010506@geosystem.us> Message-ID: <5171.62.16.186.162.1177288579.squirrel@webmail.uio.no> I was just dealing with this a few moments ago. Remember to make sure the row and col array start at 0 (and stop at nnz and n-1 respectively). I just did that mistake. Regards, Mads > > You can also use MatCreateSeqAIJWithArrays() and in parallel > MatCreateMPIAIJWithArrays(). These are intended for those who already > form the matrix in CSR format. > > Barry > > > On Sat, 21 Apr 2007, Randall Mackie wrote: > >> Hi Tim, >> >> There's no need to generate a dense matrix if you're dealing with sparse >> matrices. There are many approaches, but if you just want to get your >> existing matrix into PETSc, you can use something like the following >> (written >> in Fortran, but almost the same in C): >> >> Assume the global vector length is np: >> >> call VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,np,b,ierr) >> call VecDuplicate(b,xsol,ierr) >> call VecGetLocalSize(b,mloc,ierr) >> call VecGetOwnershipRange(b,Istart,Iend,ierr) >> >> do i=Istart+1,Iend >> loc(i)=i-1 >> end do >> >> >> ALLOCATE (d_nnz(mloc), o_nnz(mloc), STAT=istat) >> >> Note: the d_nnz and o_nnz are used to determine the pre-allocation of >> the >> global matrix >> according to the PETSc manual (basically just looping through and >> figuring out >> which >> elements are in d_nnz and which are in o_nnz). >> >> Then you can create the matrix: >> >> >> call MatCreateMPIAIJ(PETSC_COMM_WORLD,mloc,mloc,np,np, >> . PETSC_NULL_INTEGER, d_nnz, PETSC_NULL_INTEGER, >> . o_nnz,P,ierr) >> >> >> And then I had a subroutine to fill it like this (the critical part is >> the >> check >> whether that part of the matrix is owned by the current processor): >> >> jj=0 >> >> do i=1,l >> do k=2,n >> do j=2,m >> >> jj=jj+1 >> row = jj-1 >> >> IF (jj >= Istart+1 .and. jj <= Iend) THEN >> >> if(k.ne.2) then >> fact=-rijcy*dx*dy/z(k-1) ! Hxijc >> ic=ic+1 >> v(ic)=fact >> col(ic)=ijkhx(i,j,k-1)-1 >> end if >> >> [snip code] >> >> call MatSetValues(A,i1,row,ic,col,v,INSERT_VALUES, >> . ierr) >> >> >> [and finally at the end you assemble the matrix] >> >> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) >> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) >> >> >> Since these early days, I've switched over to using Distributed Arrays, >> which in my opinion are much better and much easier to use. >> >> Hope this helps, >> >> Randy >> >> >> Dr. Timothy Stitt wrote: >> > Hi all, >> > >> > I have just begun to use PETSc and I have already ran into a problem. >> > Apologies if this has been dealt with before. >> > >> > My code currently generates its own non-distributed Hamiltonian sparse >> > matrix in CSR format (due to the large size of the dense >> representation). I >> > want to exploit PETSc's (actually SLEPc's) eigensolver routines and >> hence I >> > need to convert my CSR representation into PETSc CSR representation >> for use >> > by the eigensolver. >> > From reading the documentation I am under the impression that you >> initially >> > supply the dense matrix to the PETSc matrix routines for conversion to >> PETSc >> > CSR representation. In my case I do not generate the dense matrix. I >> would >> > be grateful if someone could guide me on to how to create a PETSC >> sparse >> > matrix from my own in-code generated sparse matrix. >> > >> > Thanks in advance for any assistance that can be provided. >> > >> > Regards, >> > >> > Tim. >> > >> >> > > From timothy.stitt at ichec.ie Mon Apr 23 03:48:47 2007 From: timothy.stitt at ichec.ie (Dr. Timothy Stitt) Date: Mon, 23 Apr 2007 09:48:47 +0100 Subject: CSR Sparse Matrix Query In-Reply-To: <5171.62.16.186.162.1177288579.squirrel@webmail.uio.no> References: <4629FA7D.9020701@ichec.ie> <462A1758.9010506@geosystem.us> <5171.62.16.186.162.1177288579.squirrel@webmail.uio.no> Message-ID: <462C72EF.1050304@ichec.ie> Many thanks to all who replied to my original posting. I believe I have now enough excellent advice to solve the problem. Thanks Mads also for the final tip which could have been easily overlooked. Thanks all. Mads Fredrik Skoge Hoel wrote: > I was just dealing with this a few moments ago. Remember to make sure the > row and col array start at 0 (and stop at nnz and n-1 respectively). I > just did that mistake. > > Regards, > Mads > > >> You can also use MatCreateSeqAIJWithArrays() and in parallel >> MatCreateMPIAIJWithArrays(). These are intended for those who already >> form the matrix in CSR format. >> >> Barry >> >> >> On Sat, 21 Apr 2007, Randall Mackie wrote: >> >> >>> Hi Tim, >>> >>> There's no need to generate a dense matrix if you're dealing with sparse >>> matrices. There are many approaches, but if you just want to get your >>> existing matrix into PETSc, you can use something like the following >>> (written >>> in Fortran, but almost the same in C): >>> >>> Assume the global vector length is np: >>> >>> call VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,np,b,ierr) >>> call VecDuplicate(b,xsol,ierr) >>> call VecGetLocalSize(b,mloc,ierr) >>> call VecGetOwnershipRange(b,Istart,Iend,ierr) >>> >>> do i=Istart+1,Iend >>> loc(i)=i-1 >>> end do >>> >>> >>> ALLOCATE (d_nnz(mloc), o_nnz(mloc), STAT=istat) >>> >>> Note: the d_nnz and o_nnz are used to determine the pre-allocation of >>> the >>> global matrix >>> according to the PETSc manual (basically just looping through and >>> figuring out >>> which >>> elements are in d_nnz and which are in o_nnz). >>> >>> Then you can create the matrix: >>> >>> >>> call MatCreateMPIAIJ(PETSC_COMM_WORLD,mloc,mloc,np,np, >>> . PETSC_NULL_INTEGER, d_nnz, PETSC_NULL_INTEGER, >>> . o_nnz,P,ierr) >>> >>> >>> And then I had a subroutine to fill it like this (the critical part is >>> the >>> check >>> whether that part of the matrix is owned by the current processor): >>> >>> jj=0 >>> >>> do i=1,l >>> do k=2,n >>> do j=2,m >>> >>> jj=jj+1 >>> row = jj-1 >>> >>> IF (jj >= Istart+1 .and. jj <= Iend) THEN >>> >>> if(k.ne.2) then >>> fact=-rijcy*dx*dy/z(k-1) ! Hxijc >>> ic=ic+1 >>> v(ic)=fact >>> col(ic)=ijkhx(i,j,k-1)-1 >>> end if >>> >>> [snip code] >>> >>> call MatSetValues(A,i1,row,ic,col,v,INSERT_VALUES, >>> . ierr) >>> >>> >>> [and finally at the end you assemble the matrix] >>> >>> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) >>> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) >>> >>> >>> Since these early days, I've switched over to using Distributed Arrays, >>> which in my opinion are much better and much easier to use. >>> >>> Hope this helps, >>> >>> Randy >>> >>> >>> Dr. Timothy Stitt wrote: >>> >>>> Hi all, >>>> >>>> I have just begun to use PETSc and I have already ran into a problem. >>>> Apologies if this has been dealt with before. >>>> >>>> My code currently generates its own non-distributed Hamiltonian sparse >>>> matrix in CSR format (due to the large size of the dense >>>> >>> representation). I >>> >>>> want to exploit PETSc's (actually SLEPc's) eigensolver routines and >>>> >>> hence I >>> >>>> need to convert my CSR representation into PETSc CSR representation >>>> >>> for use >>> >>>> by the eigensolver. >>>> From reading the documentation I am under the impression that you >>>> >>> initially >>> >>>> supply the dense matrix to the PETSc matrix routines for conversion to >>>> >>> PETSc >>> >>>> CSR representation. In my case I do not generate the dense matrix. I >>>> >>> would >>> >>>> be grateful if someone could guide me on to how to create a PETSC >>>> >>> sparse >>> >>>> matrix from my own in-code generated sparse matrix. >>>> >>>> Thanks in advance for any assistance that can be provided. >>>> >>>> Regards, >>>> >>>> Tim. >>>> >>>> >>> >> > > > -- Dr. Timothy Stitt HPC Application Consultant - ICHEC (www.ichec.ie) Dublin Institute for Advanced Studies 5 Merrion Square - Dublin 2 - Ireland +353-1-6621333 (tel) / +353-1-6621477 (fax) / +353-874195427 (mobile) From timothy.stitt at ichec.ie Mon Apr 23 06:33:20 2007 From: timothy.stitt at ichec.ie (Dr. Timothy Stitt) Date: Mon, 23 Apr 2007 12:33:20 +0100 Subject: CSR Sparse Matrix Query In-Reply-To: <5171.62.16.186.162.1177288579.squirrel@webmail.uio.no> References: <4629FA7D.9020701@ichec.ie> <462A1758.9010506@geosystem.us> <5171.62.16.186.162.1177288579.squirrel@webmail.uio.no> Message-ID: <462C9980.8000203@ichec.ie> Sorry just a couple more questions. I am using Fortran. Do I have to perform some operation to switch from column-major to row-major representation before calling the MatCreateSeqAIJWithArrays routine? Also does this routine replace the call to the MatCreate() routine? I am assuming I still need to do the assembling of the matrix afterwards? Regards, Tim. Mads Fredrik Skoge Hoel wrote: > I was just dealing with this a few moments ago. Remember to make sure the > row and col array start at 0 (and stop at nnz and n-1 respectively). I > just did that mistake. > > Regards, > Mads > > >> You can also use MatCreateSeqAIJWithArrays() and in parallel >> MatCreateMPIAIJWithArrays(). These are intended for those who already >> form the matrix in CSR format. >> >> Barry >> >> >> On Sat, 21 Apr 2007, Randall Mackie wrote: >> >> >>> Hi Tim, >>> >>> There's no need to generate a dense matrix if you're dealing with sparse >>> matrices. There are many approaches, but if you just want to get your >>> existing matrix into PETSc, you can use something like the following >>> (written >>> in Fortran, but almost the same in C): >>> >>> Assume the global vector length is np: >>> >>> call VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,np,b,ierr) >>> call VecDuplicate(b,xsol,ierr) >>> call VecGetLocalSize(b,mloc,ierr) >>> call VecGetOwnershipRange(b,Istart,Iend,ierr) >>> >>> do i=Istart+1,Iend >>> loc(i)=i-1 >>> end do >>> >>> >>> ALLOCATE (d_nnz(mloc), o_nnz(mloc), STAT=istat) >>> >>> Note: the d_nnz and o_nnz are used to determine the pre-allocation of >>> the >>> global matrix >>> according to the PETSc manual (basically just looping through and >>> figuring out >>> which >>> elements are in d_nnz and which are in o_nnz). >>> >>> Then you can create the matrix: >>> >>> >>> call MatCreateMPIAIJ(PETSC_COMM_WORLD,mloc,mloc,np,np, >>> . PETSC_NULL_INTEGER, d_nnz, PETSC_NULL_INTEGER, >>> . o_nnz,P,ierr) >>> >>> >>> And then I had a subroutine to fill it like this (the critical part is >>> the >>> check >>> whether that part of the matrix is owned by the current processor): >>> >>> jj=0 >>> >>> do i=1,l >>> do k=2,n >>> do j=2,m >>> >>> jj=jj+1 >>> row = jj-1 >>> >>> IF (jj >= Istart+1 .and. jj <= Iend) THEN >>> >>> if(k.ne.2) then >>> fact=-rijcy*dx*dy/z(k-1) ! Hxijc >>> ic=ic+1 >>> v(ic)=fact >>> col(ic)=ijkhx(i,j,k-1)-1 >>> end if >>> >>> [snip code] >>> >>> call MatSetValues(A,i1,row,ic,col,v,INSERT_VALUES, >>> . ierr) >>> >>> >>> [and finally at the end you assemble the matrix] >>> >>> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) >>> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) >>> >>> >>> Since these early days, I've switched over to using Distributed Arrays, >>> which in my opinion are much better and much easier to use. >>> >>> Hope this helps, >>> >>> Randy >>> >>> >>> Dr. Timothy Stitt wrote: >>> >>>> Hi all, >>>> >>>> I have just begun to use PETSc and I have already ran into a problem. >>>> Apologies if this has been dealt with before. >>>> >>>> My code currently generates its own non-distributed Hamiltonian sparse >>>> matrix in CSR format (due to the large size of the dense >>>> >>> representation). I >>> >>>> want to exploit PETSc's (actually SLEPc's) eigensolver routines and >>>> >>> hence I >>> >>>> need to convert my CSR representation into PETSc CSR representation >>>> >>> for use >>> >>>> by the eigensolver. >>>> From reading the documentation I am under the impression that you >>>> >>> initially >>> >>>> supply the dense matrix to the PETSc matrix routines for conversion to >>>> >>> PETSc >>> >>>> CSR representation. In my case I do not generate the dense matrix. I >>>> >>> would >>> >>>> be grateful if someone could guide me on to how to create a PETSC >>>> >>> sparse >>> >>>> matrix from my own in-code generated sparse matrix. >>>> >>>> Thanks in advance for any assistance that can be provided. >>>> >>>> Regards, >>>> >>>> Tim. >>>> >>>> >>> >> > > > -- Dr. Timothy Stitt HPC Application Consultant - ICHEC (www.ichec.ie) Dublin Institute for Advanced Studies 5 Merrion Square - Dublin 2 - Ireland +353-1-6621333 (tel) / +353-1-6621477 (fax) / +353-874195427 (mobile) From balay at mcs.anl.gov Mon Apr 23 10:15:28 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 23 Apr 2007 10:15:28 -0500 (CDT) Subject: CSR Sparse Matrix Query In-Reply-To: <462C9980.8000203@ichec.ie> References: <4629FA7D.9020701@ichec.ie> <462A1758.9010506@geosystem.us> <5171.62.16.186.162.1177288579.squirrel@webmail.uio.no> <462C9980.8000203@ichec.ie> Message-ID: On Mon, 23 Apr 2007, Dr. Timothy Stitt wrote: > I am using Fortran. Do I have to perform some operation to switch > from column-major to row-major representation before calling the > MatCreateSeqAIJWithArrays routine? MatCreateSeqAIJWithArrays expects a 1d array for the values [and rows,column indicies as well] - so this issue of row-major vs column-major does not exist. > Also does this routine replace the call to the MatCreate() routine? > I am assuming I still need to do the assembling of the matrix > afterwards? You will get a fully created & assembled matrix after this call. Satish From timothy.stitt at ichec.ie Mon Apr 23 10:29:41 2007 From: timothy.stitt at ichec.ie (Dr. Timothy Stitt) Date: Mon, 23 Apr 2007 16:29:41 +0100 Subject: CSR Sparse Matrix Query In-Reply-To: References: <4629FA7D.9020701@ichec.ie> <462A1758.9010506@geosystem.us> <5171.62.16.186.162.1177288579.squirrel@webmail.uio.no> <462C9980.8000203@ichec.ie> Message-ID: <462CD0E5.4060906@ichec.ie> Great...thanks for that Satish. Satish Balay wrote: > On Mon, 23 Apr 2007, Dr. Timothy Stitt wrote: > > >> I am using Fortran. Do I have to perform some operation to switch >> from column-major to row-major representation before calling the >> MatCreateSeqAIJWithArrays routine? >> > > MatCreateSeqAIJWithArrays expects a 1d array for the values [and > rows,column indicies as well] - so this issue of row-major vs > column-major does not exist. > > >> Also does this routine replace the call to the MatCreate() routine? >> I am assuming I still need to do the assembling of the matrix >> afterwards? >> > > You will get a fully created & assembled matrix after this call. > > Satish > > -- Dr. Timothy Stitt HPC Application Consultant - ICHEC (www.ichec.ie) Dublin Institute for Advanced Studies 5 Merrion Square - Dublin 2 - Ireland +353-1-6621333 (tel) / +353-1-6621477 (fax) / +353-874195427 (mobile) From balay at mcs.anl.gov Mon Apr 23 10:31:51 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 23 Apr 2007 10:31:51 -0500 (CDT) Subject: CSR Sparse Matrix Query In-Reply-To: References: <4629FA7D.9020701@ichec.ie> <462A1758.9010506@geosystem.us> <5171.62.16.186.162.1177288579.squirrel@webmail.uio.no> <462C9980.8000203@ichec.ie> Message-ID: On Mon, 23 Apr 2007, Satish Balay wrote: > On Mon, 23 Apr 2007, Dr. Timothy Stitt wrote: > > > I am using Fortran. Do I have to perform some operation to switch > > from column-major to row-major representation before calling the > > MatCreateSeqAIJWithArrays routine? > > MatCreateSeqAIJWithArrays expects a 1d array for the values [and > rows,column indicies as well] - so this issue of row-major vs > column-major does not exist. However the CSR format which is used for the input, is equivalent to a row-major ordering.. i.e for the following matrix: 1 0 0 2 0 3 0 4 5 The call expects the following i = {0,1,3,5} [size = nrow+1 = 3+1] j = {0,0,2,1,2} [size = nz = 5] v = {1,2,3,4,5} [size = nz = 5] Satish From timothy.stitt at ichec.ie Tue Apr 24 05:11:58 2007 From: timothy.stitt at ichec.ie (Dr. Timothy Stitt) Date: Tue, 24 Apr 2007 11:11:58 +0100 Subject: CSR Sparse Matrix Query In-Reply-To: References: <4629FA7D.9020701@ichec.ie> <462A1758.9010506@geosystem.us> <5171.62.16.186.162.1177288579.squirrel@webmail.uio.no> <462C9980.8000203@ichec.ie> Message-ID: <462DD7EE.1030304@ichec.ie> Hi everyone...again. Thanks to the great support I have now got my solver working (in Fortran) albeit in a sequential non-distributed version. My problem is that I planned to substitute the call to MatCreateSeqAIJWithArrays() with a call to MatCreateMPIAIJWithArrays() (along with suitable modification to the arguments) to obtain a parallel version. Anyhow, I have come upon a few problems() unfortunately. Firstly here are the code segments I am trying to update (my sparse matrix is square with global dimensions [rows,rows], size is returned by MPI_COMM_SIZE()): --------------------------------------------------------------------------------------------------------------- call MatCreateSeqAIJWithArrays(PETSC_COMM_WORLD,rows,rows,& csrRowPointers,csrColumnIndices,csrElements,A,ierr) ---------------------------------------------------------------------------------------------------------------- TO --------------------------------------------------------------------------------------------------------------- call MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD,rows/size,rows/size,rows,rows,& csrRowPointers,csrColumnIndices,csrElements,A,ierr) ---------------------------------------------------------------------------------------------------------------- 1. Firstly, when I compile the code I keep getting: solver.o(.text+0x202): In function `MAIN__': : undefined reference to `matcreatempiaijwitharrays_' 2. Do the arguments to the routine look correct ? I would prefer for PETSc to determine as many of the arguments as possible but I wasn't sure which ones to supply explicitly or allow PETSc to infer implicitly. 3. When constructing the operator matrix is it o.k. to build it only on node with rank 0 or does it have to be duplicated across all nodes before the call to the MatCreateMPIxxxxx() routine? The matrix is very large and hence wouldn't be suitable for replication. I hope this is the last of my worries. Thanks again for your support and patience. Regards, Tim. Satish Balay wrote: > On Mon, 23 Apr 2007, Satish Balay wrote: > > >> On Mon, 23 Apr 2007, Dr. Timothy Stitt wrote: >> >> >>> I am using Fortran. Do I have to perform some operation to switch >>> from column-major to row-major representation before calling the >>> MatCreateSeqAIJWithArrays routine? >>> >> MatCreateSeqAIJWithArrays expects a 1d array for the values [and >> rows,column indicies as well] - so this issue of row-major vs >> column-major does not exist. >> > > However the CSR format which is used for the input, is equivalent to a > row-major ordering.. i.e for the following matrix: > > 1 0 0 > 2 0 3 > 0 4 5 > > The call expects the following > > i = {0,1,3,5} [size = nrow+1 = 3+1] > j = {0,0,2,1,2} [size = nz = 5] > v = {1,2,3,4,5} [size = nz = 5] > > Satish > > -- Dr. Timothy Stitt HPC Application Consultant - ICHEC (www.ichec.ie) Dublin Institute for Advanced Studies 5 Merrion Square - Dublin 2 - Ireland +353-1-6621333 (tel) / +353-1-6621477 (fax) / +353-874195427 (mobile) From knepley at gmail.com Tue Apr 24 07:09:50 2007 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 24 Apr 2007 07:09:50 -0500 Subject: CSR Sparse Matrix Query In-Reply-To: <462DD7EE.1030304@ichec.ie> References: <4629FA7D.9020701@ichec.ie> <462A1758.9010506@geosystem.us> <5171.62.16.186.162.1177288579.squirrel@webmail.uio.no> <462C9980.8000203@ichec.ie> <462DD7EE.1030304@ichec.ie> Message-ID: On 4/24/07, Dr. Timothy Stitt wrote: > Hi everyone...again. > > Thanks to the great support I have now got my solver working (in > Fortran) albeit in a sequential non-distributed version. My problem is > that I planned to substitute the call to MatCreateSeqAIJWithArrays() > with a call to MatCreateMPIAIJWithArrays() (along with suitable > modification to the arguments) to obtain a parallel version. > > Anyhow, I have come upon a few problems() unfortunately. Firstly here > are the code segments I am trying to update (my sparse matrix is square > with global dimensions [rows,rows], size is returned by MPI_COMM_SIZE()): > > --------------------------------------------------------------------------------------------------------------- > call MatCreateSeqAIJWithArrays(PETSC_COMM_WORLD,rows,rows,& > csrRowPointers,csrColumnIndices,csrElements,A,ierr) > ---------------------------------------------------------------------------------------------------------------- > > TO > > --------------------------------------------------------------------------------------------------------------- > call > MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD,rows/size,rows/size,rows,rows,& > csrRowPointers,csrColumnIndices,csrElements,A,ierr) > ---------------------------------------------------------------------------------------------------------------- > > 1. Firstly, when I compile the code I keep getting: > > solver.o(.text+0x202): In function `MAIN__': > : undefined reference to `matcreatempiaijwitharrays_' There is no Fortran interface for this function (since no one had ever asked). > 2. Do the arguments to the routine look correct ? I would prefer for > PETSc to determine as many of the arguments as possible but I wasn't > sure which ones to supply explicitly or allow PETSc to infer implicitly. No, I think you are going to have some problems here. This function is intended for people who already have a parallel CSR matrix constructed. From your discussion above, this does not appear to be the case. I see several pitfalls: a) size may not evenly divide rows b) the arrays must be only for rows local to the process, but contain global indices I would encourage you to follow the steps that Randy outlined, namely create the matrix (letting PETSc partition the rows), preallocate the storage by determining local row lengths, and the setting values with MatSetValues for each row. Thanks, Matt > 3. When constructing the operator matrix is it o.k. to build it only on > node with rank 0 or does it have to be duplicated across all nodes > before the call to the MatCreateMPIxxxxx() routine? The matrix is very > large and hence wouldn't be suitable for replication. > > I hope this is the last of my worries. Thanks again for your support and > patience. > > Regards, > > Tim. > > > > Satish Balay wrote: > > On Mon, 23 Apr 2007, Satish Balay wrote: > > > > > >> On Mon, 23 Apr 2007, Dr. Timothy Stitt wrote: > >> > >> > >>> I am using Fortran. Do I have to perform some operation to switch > >>> from column-major to row-major representation before calling the > >>> MatCreateSeqAIJWithArrays routine? > >>> > >> MatCreateSeqAIJWithArrays expects a 1d array for the values [and > >> rows,column indicies as well] - so this issue of row-major vs > >> column-major does not exist. > >> > > > > However the CSR format which is used for the input, is equivalent to a > > row-major ordering.. i.e for the following matrix: > > > > 1 0 0 > > 2 0 3 > > 0 4 5 > > > > The call expects the following > > > > i = {0,1,3,5} [size = nrow+1 = 3+1] > > j = {0,0,2,1,2} [size = nz = 5] > > v = {1,2,3,4,5} [size = nz = 5] > > > > Satish > > > > > > > -- > Dr. Timothy Stitt > HPC Application Consultant - ICHEC (www.ichec.ie) > > Dublin Institute for Advanced Studies > 5 Merrion Square - Dublin 2 - Ireland > > +353-1-6621333 (tel) / +353-1-6621477 (fax) / +353-874195427 (mobile) > > -- The government saving money is like me spilling beer. It happens, but never on purpose. From timothy.stitt at ichec.ie Tue Apr 24 07:39:12 2007 From: timothy.stitt at ichec.ie (Dr. Timothy Stitt) Date: Tue, 24 Apr 2007 13:39:12 +0100 Subject: CSR Sparse Matrix Query In-Reply-To: References: <4629FA7D.9020701@ichec.ie> <462A1758.9010506@geosystem.us> <5171.62.16.186.162.1177288579.squirrel@webmail.uio.no> <462C9980.8000203@ichec.ie> <462DD7EE.1030304@ichec.ie> Message-ID: <462DFA70.7020107@ichec.ie> Matt, Thanks for the reply. I was hoping that there was some sort of automatic distribution from the global operator matrix to a distributed form (in the one call) but obviously there is more to be done. I will go back and follow Randy's suggestions now that I know the conversion from global to distributed matrices isn't as straightforward as I hoped. Regards, Tim. Matthew Knepley wrote: > On 4/24/07, Dr. Timothy Stitt wrote: >> Hi everyone...again. >> >> Thanks to the great support I have now got my solver working (in >> Fortran) albeit in a sequential non-distributed version. My problem is >> that I planned to substitute the call to MatCreateSeqAIJWithArrays() >> with a call to MatCreateMPIAIJWithArrays() (along with suitable >> modification to the arguments) to obtain a parallel version. >> >> Anyhow, I have come upon a few problems() unfortunately. Firstly here >> are the code segments I am trying to update (my sparse matrix is square >> with global dimensions [rows,rows], size is returned by >> MPI_COMM_SIZE()): >> >> --------------------------------------------------------------------------------------------------------------- >> >> call MatCreateSeqAIJWithArrays(PETSC_COMM_WORLD,rows,rows,& >> csrRowPointers,csrColumnIndices,csrElements,A,ierr) >> ---------------------------------------------------------------------------------------------------------------- >> >> >> TO >> >> --------------------------------------------------------------------------------------------------------------- >> >> call >> MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD,rows/size,rows/size,rows,rows,& >> >> csrRowPointers,csrColumnIndices,csrElements,A,ierr) >> ---------------------------------------------------------------------------------------------------------------- >> >> >> 1. Firstly, when I compile the code I keep getting: >> >> solver.o(.text+0x202): In function `MAIN__': >> : undefined reference to `matcreatempiaijwitharrays_' > > There is no Fortran interface for this function (since no one had ever > asked). > >> 2. Do the arguments to the routine look correct ? I would prefer for >> PETSc to determine as many of the arguments as possible but I wasn't >> sure which ones to supply explicitly or allow PETSc to infer implicitly. > > No, I think you are going to have some problems here. This function is > intended > for people who already have a parallel CSR matrix constructed. From your > discussion above, this does not appear to be the case. I see several > pitfalls: > > a) size may not evenly divide rows > > b) the arrays must be only for rows local to the process, but contain > global > indices > > I would encourage you to follow the steps that Randy outlined, namely > create > the matrix (letting PETSc partition the rows), preallocate the storage by > determining local row lengths, and the setting values with > MatSetValues for > each row. > > Thanks, > > Matt > >> 3. When constructing the operator matrix is it o.k. to build it only on >> node with rank 0 or does it have to be duplicated across all nodes >> before the call to the MatCreateMPIxxxxx() routine? The matrix is very >> large and hence wouldn't be suitable for replication. >> >> I hope this is the last of my worries. Thanks again for your support and >> patience. >> >> Regards, >> >> Tim. >> >> >> >> Satish Balay wrote: >> > On Mon, 23 Apr 2007, Satish Balay wrote: >> > >> > >> >> On Mon, 23 Apr 2007, Dr. Timothy Stitt wrote: >> >> >> >> >> >>> I am using Fortran. Do I have to perform some operation to switch >> >>> from column-major to row-major representation before calling the >> >>> MatCreateSeqAIJWithArrays routine? >> >>> >> >> MatCreateSeqAIJWithArrays expects a 1d array for the values [and >> >> rows,column indicies as well] - so this issue of row-major vs >> >> column-major does not exist. >> >> >> > >> > However the CSR format which is used for the input, is equivalent to a >> > row-major ordering.. i.e for the following matrix: >> > >> > 1 0 0 >> > 2 0 3 >> > 0 4 5 >> > >> > The call expects the following >> > >> > i = {0,1,3,5} [size = nrow+1 = 3+1] >> > j = {0,0,2,1,2} [size = nz = 5] >> > v = {1,2,3,4,5} [size = nz = 5] >> > >> > Satish >> > >> > >> >> >> -- >> Dr. Timothy Stitt >> HPC Application Consultant - ICHEC (www.ichec.ie) >> >> Dublin Institute for Advanced Studies >> 5 Merrion Square - Dublin 2 - Ireland >> >> +353-1-6621333 (tel) / +353-1-6621477 (fax) / +353-874195427 (mobile) >> >> > > -- Dr. Timothy Stitt HPC Application Consultant - ICHEC (www.ichec.ie) Dublin Institute for Advanced Studies 5 Merrion Square - Dublin 2 - Ireland +353-1-6621333 (tel) / +353-1-6621477 (fax) / +353-874195427 (mobile) From bsmith at mcs.anl.gov Tue Apr 24 07:55:21 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 24 Apr 2007 07:55:21 -0500 (CDT) Subject: CSR Sparse Matrix Query In-Reply-To: <462DFA70.7020107@ichec.ie> References: <4629FA7D.9020701@ichec.ie> <462A1758.9010506@geosystem.us> <5171.62.16.186.162.1177288579.squirrel@webmail.uio.no> <462C9980.8000203@ichec.ie> <462DD7EE.1030304@ichec.ie> <462DFA70.7020107@ichec.ie> Message-ID: In petsc-dev http://www-unix.mcs.anl.gov/petsc/petsc-as/developers/index.html there is code that allows creating a matrix/linear system on one process and solving it in parallel, See PCType PCOPENMP http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/PC/PCOPENMP.html This is a quick-and-dirty stop-gap way of having a parallel code (where your code is not parallel). It is not a long term solution but can get good speedup for several processes. Barry There is also a low-level routine that takes a seqaij and distributes it to a parallel form called MatDistribute_MPIAIJ(), but using PCOPENMP is much easier to use. On Tue, 24 Apr 2007, Dr. Timothy Stitt wrote: > Matt, > > Thanks for the reply. I was hoping that there was some sort of automatic > distribution from the global operator matrix to a distributed form (in the one > call) but obviously there is more to be done. I will go back and follow > Randy's suggestions now that I know the conversion from global to distributed > matrices isn't as straightforward as I hoped. > > Regards, > > Tim. > > Matthew Knepley wrote: > > On 4/24/07, Dr. Timothy Stitt wrote: > > > Hi everyone...again. > > > > > > Thanks to the great support I have now got my solver working (in > > > Fortran) albeit in a sequential non-distributed version. My problem is > > > that I planned to substitute the call to MatCreateSeqAIJWithArrays() > > > with a call to MatCreateMPIAIJWithArrays() (along with suitable > > > modification to the arguments) to obtain a parallel version. > > > > > > Anyhow, I have come upon a few problems() unfortunately. Firstly here > > > are the code segments I am trying to update (my sparse matrix is square > > > with global dimensions [rows,rows], size is returned by MPI_COMM_SIZE()): > > > > > > > > > --------------------------------------------------------------------------------------------------------------- > > > call MatCreateSeqAIJWithArrays(PETSC_COMM_WORLD,rows,rows,& > > > csrRowPointers,csrColumnIndices,csrElements,A,ierr) > > > > > > ---------------------------------------------------------------------------------------------------------------- > > > > > > TO > > > > > > > > > --------------------------------------------------------------------------------------------------------------- > > > call > > > MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD,rows/size,rows/size,rows,rows,& > > > csrRowPointers,csrColumnIndices,csrElements,A,ierr) > > > > > > ---------------------------------------------------------------------------------------------------------------- > > > > > > 1. Firstly, when I compile the code I keep getting: > > > > > > solver.o(.text+0x202): In function `MAIN__': > > > : undefined reference to `matcreatempiaijwitharrays_' > > > > There is no Fortran interface for this function (since no one had ever > > asked). > > > > > 2. Do the arguments to the routine look correct ? I would prefer for > > > PETSc to determine as many of the arguments as possible but I wasn't > > > sure which ones to supply explicitly or allow PETSc to infer implicitly. > > > > No, I think you are going to have some problems here. This function is > > intended > > for people who already have a parallel CSR matrix constructed. From your > > discussion above, this does not appear to be the case. I see several > > pitfalls: > > > > a) size may not evenly divide rows > > > > b) the arrays must be only for rows local to the process, but contain > > global > > indices > > > > I would encourage you to follow the steps that Randy outlined, namely create > > the matrix (letting PETSc partition the rows), preallocate the storage by > > determining local row lengths, and the setting values with MatSetValues for > > each row. > > > > Thanks, > > > > Matt > > > > > 3. When constructing the operator matrix is it o.k. to build it only on > > > node with rank 0 or does it have to be duplicated across all nodes > > > before the call to the MatCreateMPIxxxxx() routine? The matrix is very > > > large and hence wouldn't be suitable for replication. > > > > > > I hope this is the last of my worries. Thanks again for your support and > > > patience. > > > > > > Regards, > > > > > > Tim. > > > > > > > > > > > > Satish Balay wrote: > > > > On Mon, 23 Apr 2007, Satish Balay wrote: > > > > > > > > > > > >> On Mon, 23 Apr 2007, Dr. Timothy Stitt wrote: > > > >> > > > >> > > > >>> I am using Fortran. Do I have to perform some operation to switch > > > >>> from column-major to row-major representation before calling the > > > >>> MatCreateSeqAIJWithArrays routine? > > > >>> > > > >> MatCreateSeqAIJWithArrays expects a 1d array for the values [and > > > >> rows,column indicies as well] - so this issue of row-major vs > > > >> column-major does not exist. > > > >> > > > > > > > > However the CSR format which is used for the input, is equivalent to a > > > > row-major ordering.. i.e for the following matrix: > > > > > > > > 1 0 0 > > > > 2 0 3 > > > > 0 4 5 > > > > > > > > The call expects the following > > > > > > > > i = {0,1,3,5} [size = nrow+1 = 3+1] > > > > j = {0,0,2,1,2} [size = nz = 5] > > > > v = {1,2,3,4,5} [size = nz = 5] > > > > > > > > Satish > > > > > > > > > > > > > > > > > -- > > > Dr. Timothy Stitt > > > HPC Application Consultant - ICHEC (www.ichec.ie) > > > > > > Dublin Institute for Advanced Studies > > > 5 Merrion Square - Dublin 2 - Ireland > > > > > > +353-1-6621333 (tel) / +353-1-6621477 (fax) / +353-874195427 (mobile) > > > > > > > > > > > > > From timothy.stitt at ichec.ie Tue Apr 24 08:07:26 2007 From: timothy.stitt at ichec.ie (Dr. Timothy Stitt) Date: Tue, 24 Apr 2007 14:07:26 +0100 Subject: CSR Sparse Matrix Query In-Reply-To: References: <4629FA7D.9020701@ichec.ie> <462A1758.9010506@geosystem.us> <5171.62.16.186.162.1177288579.squirrel@webmail.uio.no> <462C9980.8000203@ichec.ie> <462DD7EE.1030304@ichec.ie> <462DFA70.7020107@ichec.ie> Message-ID: <462E010E.7010008@ichec.ie> Thanks Barry....I am now becoming overwhelmed with the different approaches possible. Incidentally my target platform is a large shared-memory machine (with 256GB RAM and 32 processors) so I am not overly concerned right now with the matrix storage size. I just want to reduce my time to completion. As I scale up though I will need to think about distributing the data but a quick fix as you say Barry is ideal for me now. Thanks for that. Tim. Barry Smith wrote: > In petsc-dev http://www-unix.mcs.anl.gov/petsc/petsc-as/developers/index.html > there is code that allows creating a matrix/linear system on one process and > solving it in parallel, See PCType PCOPENMP > http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/PC/PCOPENMP.html > > This is a quick-and-dirty stop-gap way of having a parallel code (where your > code is not parallel). It is not a long term solution but can get good speedup > for several processes. > > Barry > > There is also a low-level routine that takes a seqaij and distributes it to a > parallel form called MatDistribute_MPIAIJ(), but using PCOPENMP is much easier > to use. > > > > On Tue, 24 Apr 2007, Dr. Timothy Stitt wrote: > > >> Matt, >> >> Thanks for the reply. I was hoping that there was some sort of automatic >> distribution from the global operator matrix to a distributed form (in the one >> call) but obviously there is more to be done. I will go back and follow >> Randy's suggestions now that I know the conversion from global to distributed >> matrices isn't as straightforward as I hoped. >> >> Regards, >> >> Tim. >> >> Matthew Knepley wrote: >> >>> On 4/24/07, Dr. Timothy Stitt wrote: >>> >>>> Hi everyone...again. >>>> >>>> Thanks to the great support I have now got my solver working (in >>>> Fortran) albeit in a sequential non-distributed version. My problem is >>>> that I planned to substitute the call to MatCreateSeqAIJWithArrays() >>>> with a call to MatCreateMPIAIJWithArrays() (along with suitable >>>> modification to the arguments) to obtain a parallel version. >>>> >>>> Anyhow, I have come upon a few problems() unfortunately. Firstly here >>>> are the code segments I am trying to update (my sparse matrix is square >>>> with global dimensions [rows,rows], size is returned by MPI_COMM_SIZE()): >>>> >>>> >>>> --------------------------------------------------------------------------------------------------------------- >>>> call MatCreateSeqAIJWithArrays(PETSC_COMM_WORLD,rows,rows,& >>>> csrRowPointers,csrColumnIndices,csrElements,A,ierr) >>>> >>>> ---------------------------------------------------------------------------------------------------------------- >>>> >>>> TO >>>> >>>> >>>> --------------------------------------------------------------------------------------------------------------- >>>> call >>>> MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD,rows/size,rows/size,rows,rows,& >>>> csrRowPointers,csrColumnIndices,csrElements,A,ierr) >>>> >>>> ---------------------------------------------------------------------------------------------------------------- >>>> >>>> 1. Firstly, when I compile the code I keep getting: >>>> >>>> solver.o(.text+0x202): In function `MAIN__': >>>> : undefined reference to `matcreatempiaijwitharrays_' >>>> >>> There is no Fortran interface for this function (since no one had ever >>> asked). >>> >>> >>>> 2. Do the arguments to the routine look correct ? I would prefer for >>>> PETSc to determine as many of the arguments as possible but I wasn't >>>> sure which ones to supply explicitly or allow PETSc to infer implicitly. >>>> >>> No, I think you are going to have some problems here. This function is >>> intended >>> for people who already have a parallel CSR matrix constructed. From your >>> discussion above, this does not appear to be the case. I see several >>> pitfalls: >>> >>> a) size may not evenly divide rows >>> >>> b) the arrays must be only for rows local to the process, but contain >>> global >>> indices >>> >>> I would encourage you to follow the steps that Randy outlined, namely create >>> the matrix (letting PETSc partition the rows), preallocate the storage by >>> determining local row lengths, and the setting values with MatSetValues for >>> each row. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> 3. When constructing the operator matrix is it o.k. to build it only on >>>> node with rank 0 or does it have to be duplicated across all nodes >>>> before the call to the MatCreateMPIxxxxx() routine? The matrix is very >>>> large and hence wouldn't be suitable for replication. >>>> >>>> I hope this is the last of my worries. Thanks again for your support and >>>> patience. >>>> >>>> Regards, >>>> >>>> Tim. >>>> >>>> >>>> >>>> Satish Balay wrote: >>>> >>>>> On Mon, 23 Apr 2007, Satish Balay wrote: >>>>> >>>>> >>>>> >>>>>> On Mon, 23 Apr 2007, Dr. Timothy Stitt wrote: >>>>>> >>>>>> >>>>>> >>>>>>> I am using Fortran. Do I have to perform some operation to switch >>>>>>> from column-major to row-major representation before calling the >>>>>>> MatCreateSeqAIJWithArrays routine? >>>>>>> >>>>>>> >>>>>> MatCreateSeqAIJWithArrays expects a 1d array for the values [and >>>>>> rows,column indicies as well] - so this issue of row-major vs >>>>>> column-major does not exist. >>>>>> >>>>>> >>>>> However the CSR format which is used for the input, is equivalent to a >>>>> row-major ordering.. i.e for the following matrix: >>>>> >>>>> 1 0 0 >>>>> 2 0 3 >>>>> 0 4 5 >>>>> >>>>> The call expects the following >>>>> >>>>> i = {0,1,3,5} [size = nrow+1 = 3+1] >>>>> j = {0,0,2,1,2} [size = nz = 5] >>>>> v = {1,2,3,4,5} [size = nz = 5] >>>>> >>>>> Satish >>>>> >>>>> >>>>> >>>> -- >>>> Dr. Timothy Stitt >>>> HPC Application Consultant - ICHEC (www.ichec.ie) >>>> >>>> Dublin Institute for Advanced Studies >>>> 5 Merrion Square - Dublin 2 - Ireland >>>> >>>> +353-1-6621333 (tel) / +353-1-6621477 (fax) / +353-874195427 (mobile) >>>> >>>> >>>> >>> >> >> > > -- Dr. Timothy Stitt HPC Application Consultant - ICHEC (www.ichec.ie) Dublin Institute for Advanced Studies 5 Merrion Square - Dublin 2 - Ireland +353-1-6621333 (tel) / +353-1-6621477 (fax) / +353-874195427 (mobile) From mfhoel at ifi.uio.no Tue Apr 24 08:45:46 2007 From: mfhoel at ifi.uio.no (Mads Fredrik Skoge Hoel) Date: Tue, 24 Apr 2007 15:45:46 +0200 (CEST) Subject: Questions regarding MatCreateMPIAIJWithArrays Message-ID: <2556.129.240.228.53.1177422346.squirrel@webmail.uio.no> Hi, I am in a similar situation as Tim. I am starting out with a sequential program using PETSc as a solver and this other library to discretisize. Which hopefully will become a parallel program once run under mpirun. But calling MatCreateMPIAIJWithArrays i get the error message: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Object is in wrong state! [0]PETSC ERROR: Must call MatSetSizes() first! [0]PETSC ERROR: ------------------------------------------------------------------------ This is the call generating the error message: MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, n, m, n, m, irow, jcol, &dpA(1), &A); I am probably using the MatCreateMPIAIJWithArrays function wrong or in a wrong context. Is it allowed to call MatCreateMPIAIJWithArrays when running as a uniprocessor application? Or must I use in a multiprocessor context calling MatCreateSeqAIJWithArrays on each processor? And how would a PETSc solver know that local Mat A is part of a global matrix? Any help is highly appreciated. Regards, Mads From knepley at gmail.com Tue Apr 24 08:51:39 2007 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 24 Apr 2007 08:51:39 -0500 Subject: Questions regarding MatCreateMPIAIJWithArrays In-Reply-To: <2556.129.240.228.53.1177422346.squirrel@webmail.uio.no> References: <2556.129.240.228.53.1177422346.squirrel@webmail.uio.no> Message-ID: On 4/24/07, Mads Fredrik Skoge Hoel wrote: > Hi, > > I am in a similar situation as Tim. I am starting out with a sequential > program using PETSc as a solver and this other library to discretisize. > Which hopefully will become a parallel program once run under mpirun. > > But calling MatCreateMPIAIJWithArrays i get the error message: > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Object is in wrong state! > [0]PETSC ERROR: Must call MatSetSizes() first! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > This is the call generating the error message: > MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, n, m, n, m, irow, jcol, > &dpA(1), &A); > > I am probably using the MatCreateMPIAIJWithArrays function wrong or in a > wrong context. > > Is it allowed to call MatCreateMPIAIJWithArrays when running as a > uniprocessor application? Or must I use in a multiprocessor context > calling MatCreateSeqAIJWithArrays on each processor? And how would a PETSc > solver know that local Mat A is part of a global matrix? No, this method is for people who already have partitioned their matrix. For a uniprocessor application, please follow Barry's instructions for the OPENMP PC. Otherwise, you will need to rework your code as Randy pointed out before. Thanks, Matt > Any help is highly appreciated. > > Regards, > Mads > > -- The government saving money is like me spilling beer. It happens, but never on purpose. From mfhoel at ifi.uio.no Tue Apr 24 11:45:44 2007 From: mfhoel at ifi.uio.no (Mads Fredrik Skoge Hoel) Date: Tue, 24 Apr 2007 18:45:44 +0200 (CEST) Subject: Questions regarding MatCreateMPIAIJWithArrays In-Reply-To: References: <2556.129.240.228.53.1177422346.squirrel@webmail.uio.no> Message-ID: <2965.129.240.228.21.1177433144.squirrel@webmail.uio.no> Thank you for your answer. I did not at first understand Barrys suggestion, but looking at it a second time helped. So if i got it right there is two options: 1. Barrys suggestion: use code that distributes a SeqAIJ matrix owned by a single process to multiple processes. Ie. code from http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/src/ksp/pc/impls/openmp/openmp.c.html#PCOPENMP 2. Randys suggestion: create an empty distributed MPIAIJ matrix, using MatCreateMPIAIJ. Have it filled with existing data, distributed or non- distributed, in an CRS format, and have it assembeled. Regards, Mads > On 4/24/07, Mads Fredrik Skoge Hoel wrote: >> Hi, >> >> I am in a similar situation as Tim. I am starting out with a sequential >> program using PETSc as a solver and this other library to discretisize. >> Which hopefully will become a parallel program once run under mpirun. >> >> But calling MatCreateMPIAIJWithArrays i get the error message: >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Object is in wrong state! >> [0]PETSC ERROR: Must call MatSetSizes() first! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> >> This is the call generating the error message: >> MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, n, m, n, m, irow, jcol, >> &dpA(1), &A); >> >> I am probably using the MatCreateMPIAIJWithArrays function wrong or in a >> wrong context. >> >> Is it allowed to call MatCreateMPIAIJWithArrays when running as a >> uniprocessor application? Or must I use in a multiprocessor context >> calling MatCreateSeqAIJWithArrays on each processor? And how would a >> PETSc >> solver know that local Mat A is part of a global matrix? > > No, this method is for people who already have partitioned their matrix. > For a > uniprocessor application, please follow Barry's instructions for the > OPENMP PC. > Otherwise, you will need to rework your code as Randy pointed out before. > > Thanks, > > Matt > >> Any help is highly appreciated. >> >> Regards, >> Mads >> >> > > > -- > The government saving money is like me spilling beer. It happens, but > never on purpose. > > From knepley at gmail.com Tue Apr 24 11:51:44 2007 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 24 Apr 2007 11:51:44 -0500 Subject: Questions regarding MatCreateMPIAIJWithArrays In-Reply-To: <2965.129.240.228.21.1177433144.squirrel@webmail.uio.no> References: <2556.129.240.228.53.1177422346.squirrel@webmail.uio.no> <2965.129.240.228.21.1177433144.squirrel@webmail.uio.no> Message-ID: On 4/24/07, Mads Fredrik Skoge Hoel wrote: > Thank you for your answer. I did not at first understand Barrys > suggestion, but looking at it a second time helped. > > So if i got it right there is two options: > > 1. Barrys suggestion: use code that distributes a SeqAIJ matrix owned by a > single process to multiple processes. Ie. code from > http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/src/ksp/pc/impls/openmp/openmp.c.html#PCOPENMP > > 2. Randys suggestion: create an empty distributed MPIAIJ matrix, using > MatCreateMPIAIJ. Have it filled with existing data, distributed or non- > distributed, in an CRS format, and have it assembeled. Yes, although you really want the data "distributed" for option 2. If you are reading from disk, have all procs read and just ignore rows which you do not own. Matt > Regards, > Mads > > > > On 4/24/07, Mads Fredrik Skoge Hoel wrote: > >> Hi, > >> > >> I am in a similar situation as Tim. I am starting out with a sequential > >> program using PETSc as a solver and this other library to discretisize. > >> Which hopefully will become a parallel program once run under mpirun. > >> > >> But calling MatCreateMPIAIJWithArrays i get the error message: > >> [0]PETSC ERROR: --------------------- Error Message > >> ------------------------------------ > >> [0]PETSC ERROR: Object is in wrong state! > >> [0]PETSC ERROR: Must call MatSetSizes() first! > >> [0]PETSC ERROR: > >> ------------------------------------------------------------------------ > >> > >> This is the call generating the error message: > >> MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, n, m, n, m, irow, jcol, > >> &dpA(1), &A); > >> > >> I am probably using the MatCreateMPIAIJWithArrays function wrong or in a > >> wrong context. > >> > >> Is it allowed to call MatCreateMPIAIJWithArrays when running as a > >> uniprocessor application? Or must I use in a multiprocessor context > >> calling MatCreateSeqAIJWithArrays on each processor? And how would a > >> PETSc > >> solver know that local Mat A is part of a global matrix? > > > > No, this method is for people who already have partitioned their matrix. > > For a > > uniprocessor application, please follow Barry's instructions for the > > OPENMP PC. > > Otherwise, you will need to rework your code as Randy pointed out before. > > > > Thanks, > > > > Matt > > > >> Any help is highly appreciated. > >> > >> Regards, > >> Mads > >> > >> > > > > > > -- > > The government saving money is like me spilling beer. It happens, but > > never on purpose. > > > > > > > -- The government saving money is like me spilling beer. It happens, but never on purpose. From mfhoel at ifi.uio.no Tue Apr 24 12:20:03 2007 From: mfhoel at ifi.uio.no (Mads Fredrik Skoge Hoel) Date: Tue, 24 Apr 2007 19:20:03 +0200 (CEST) Subject: Questions regarding MatCreateMPIAIJWithArrays In-Reply-To: References: <2556.129.240.228.53.1177422346.squirrel@webmail.uio.no> <2965.129.240.228.21.1177433144.squirrel@webmail.uio.no> Message-ID: <2502.129.240.228.53.1177435203.squirrel@webmail.uio.no> Thanks, for clearing that up. Truly helps alot. From an efficiency point of view I will go with option 2. And maybe implement a third option: to create an SeqAIJ if run as a uniprocessor program. Regards, Mads > On 4/24/07, Mads Fredrik Skoge Hoel wrote: >> Thank you for your answer. I did not at first understand Barrys >> suggestion, but looking at it a second time helped. >> >> So if i got it right there is two options: >> >> 1. Barrys suggestion: use code that distributes a SeqAIJ matrix owned by >> a >> single process to multiple processes. Ie. code from >> http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/src/ksp/pc/impls/openmp/openmp.c.html#PCOPENMP >> >> 2. Randys suggestion: create an empty distributed MPIAIJ matrix, using >> MatCreateMPIAIJ. Have it filled with existing data, distributed or non- >> distributed, in an CRS format, and have it assembeled. > > Yes, although you really want the data "distributed" for option 2. If > you are reading > from disk, have all procs read and just ignore rows which you do not own. > > Matt > >> Regards, >> Mads >> >> >> > On 4/24/07, Mads Fredrik Skoge Hoel wrote: >> >> Hi, >> >> >> >> I am in a similar situation as Tim. I am starting out with a >> sequential >> >> program using PETSc as a solver and this other library to >> discretisize. >> >> Which hopefully will become a parallel program once run under mpirun. >> >> >> >> But calling MatCreateMPIAIJWithArrays i get the error message: >> >> [0]PETSC ERROR: --------------------- Error Message >> >> ------------------------------------ >> >> [0]PETSC ERROR: Object is in wrong state! >> >> [0]PETSC ERROR: Must call MatSetSizes() first! >> >> [0]PETSC ERROR: >> >> ------------------------------------------------------------------------ >> >> >> >> This is the call generating the error message: >> >> MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, n, m, n, m, irow, jcol, >> >> &dpA(1), &A); >> >> >> >> I am probably using the MatCreateMPIAIJWithArrays function wrong or >> in a >> >> wrong context. >> >> >> >> Is it allowed to call MatCreateMPIAIJWithArrays when running as a >> >> uniprocessor application? Or must I use in a multiprocessor context >> >> calling MatCreateSeqAIJWithArrays on each processor? And how would a >> >> PETSc >> >> solver know that local Mat A is part of a global matrix? >> > >> > No, this method is for people who already have partitioned their >> matrix. >> > For a >> > uniprocessor application, please follow Barry's instructions for the >> > OPENMP PC. >> > Otherwise, you will need to rework your code as Randy pointed out >> before. >> > >> > Thanks, >> > >> > Matt >> > >> >> Any help is highly appreciated. >> >> >> >> Regards, >> >> Mads >> >> >> >> >> > >> > >> > -- >> > The government saving money is like me spilling beer. It happens, but >> > never on purpose. >> > >> > >> >> >> > > > -- > The government saving money is like me spilling beer. It happens, but > never on purpose. > > From balay at mcs.anl.gov Tue Apr 24 12:25:28 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 24 Apr 2007 12:25:28 -0500 (CDT) Subject: Questions regarding MatCreateMPIAIJWithArrays In-Reply-To: <2502.129.240.228.53.1177435203.squirrel@webmail.uio.no> References: <2556.129.240.228.53.1177422346.squirrel@webmail.uio.no> <2965.129.240.228.21.1177433144.squirrel@webmail.uio.no> <2502.129.240.228.53.1177435203.squirrel@webmail.uio.no> Message-ID: You might want to check src/mat/examples/tests/ex109.c for this usage.. Satish On Tue, 24 Apr 2007, Mads Fredrik Skoge Hoel wrote: > Thanks, for clearing that up. Truly helps alot. From an efficiency point > of view I will go with option 2. And maybe implement a third option: to > create an SeqAIJ if run as a uniprocessor program. > From mfhoel at ifi.uio.no Tue Apr 24 16:13:51 2007 From: mfhoel at ifi.uio.no (Mads Fredrik Skoge Hoel) Date: Tue, 24 Apr 2007 23:13:51 +0200 (CEST) Subject: Questions regarding MatCreateMPIAIJWithArrays In-Reply-To: References: <2556.129.240.228.53.1177422346.squirrel@webmail.uio.no> <2965.129.240.228.21.1177433144.squirrel@webmail.uio.no> <2502.129.240.228.53.1177435203.squirrel@webmail.uio.no> Message-ID: <19882.62.16.186.162.1177449231.squirrel@webmail.uio.no> Thanks for the tip Satish. I am looking at it right now, it is quite helpful. Regards, Mads > You might want to check src/mat/examples/tests/ex109.c for this usage.. > > Satish > > > On Tue, 24 Apr 2007, Mads Fredrik Skoge Hoel wrote: > >> Thanks, for clearing that up. Truly helps alot. From an efficiency point >> of view I will go with option 2. And maybe implement a third option: to >> create an SeqAIJ if run as a uniprocessor program. >> > > From nicolas.bathfield at chalmers.se Wed Apr 25 10:38:09 2007 From: nicolas.bathfield at chalmers.se (Nicolas Bathfield) Date: Wed, 25 Apr 2007 17:38:09 +0200 (CEST) Subject: HYPRE with multiple variables In-Reply-To: References: <26157.193.183.3.2.1176453900.squirrel@webmail.chalmers.se> <9329.193.183.3.2.1176464453.squirrel@webmail.chalmers.se> <8614E121-37D8-4AF3-A67A-A142D27B7B62@student.uu.se> Message-ID: <35010.129.16.81.46.1177515489.squirrel@webmail.chalmers.se> Dear Barry, Using MatSetBlockSize(A,5) improved my results greatly. Boomemramg is now solving the system of equations. Still, the equations I solve are coupled, and my discretization scheme is meant for a non-segregated solver. As a consequence (I believe), boomeramg still diverges. I would therefore like to use the nodal relaxation in boomeramg (the hypre command is HYPRE_BOOMERAMGSetNodal) in order to couple the coarse grid choice for all my variables. How can I achieve this from PETSc? Best regards, Nicolas > > From PETSc MPIAIJ matrices you need to set the block size of the matrix > with MatSetBlockSize(A,5) after you have called MatSetType() or > MatCreateMPIAIJ(). Then HYPRE_BoomerAMGSetNumFunctions() is automatically > called by PETSc. > > Barry > > The reason this is done this way instead of as > -pc_hypre_boomeramg_block_size is the idea that hypre will use the > properties of the matrix it is given in building the preconditioner so > the user does not have to pass those properties in seperately directly > to hypre. > > > On Fri, 13 Apr 2007, Shaman Mahmoudi wrote: > >> Hi, >> >> int HYPRE_BoomerAMGSetNumFunctions (.....) >> >> sets the size of the system of PDEs. >> >> With best regards, Shaman Mahmoudi >> >> On Apr 13, 2007, at 2:04 PM, Shaman Mahmoudi wrote: >> >> > Hi Nicolas, >> > >> > You are right. hypre has changed a lot since the version I used. >> > >> > I found this interesting information: >> > >> > int HYPRE_BOOMERAMGSetNodal(....) >> > >> > Sets whether to use the nodal systems version. Default is 0. >> > >> > Then information about smoothers: >> > >> > One interesting thing there is this, >> > >> > HYPRE_BoomerAMGSetDomainType(....) >> > >> > 0 - each point is a domain (default) >> > 1 each node is a domain (only of interest in systems AMG) >> > 2 .... >> > >> > I could not find how you define the nodal displacement ordering. But >> it >> > should be there somewhere. >> > >> > I read the reference manual for hypre 2.0 >> > >> > With best regards, Shaman Mahmoudi >> > >> > >> > On Apr 13, 2007, at 1:40 PM, Nicolas Bathfield wrote: >> > >> > > Dear Shaman, >> > > >> > > As far as I could understand, there is a BoomerAMG?s systems AMG >> version >> > > available. This seems to be exactly what I am looking for, but I >> just >> > > don't know how to access it, either through PETSc or directly. >> > > >> > > Best regards, >> > > >> > > Nicolas >> > > >> > > > Hi, >> > > > >> > > > You want to exploit the structure of the model? >> > > > As far as I know, boomeramg can not treat a set of rows or blocks >> as >> > > > a molecule, a so called block-smoother? >> > > > ML 2.0 smoothed aggregation does support it. >> > > > >> > > > With best regards, Shaman Mahmoudi >> > > > >> > > > On Apr 13, 2007, at 10:45 AM, Nicolas Bathfield wrote: >> > > > >> > > > > Hi, >> > > > > >> > > > > I am solving the Navier-stokes equations and try to use Hypre as >> > > > > preconditioner. >> > > > > Until now, I used PETSc as non-segregated solver and it worked >> > > > > perfectly. >> > > > > Things got worse when I decided to use Boomeramg (Hypre). >> > > > > As I solve a system of PDEs, each cell is represented by 5 rows >> in my >> > > > > matrix (I solve for 5 variables). PETSc handles that without >> problem >> > > > > apparently, but the coarsening scheme of Boomeramg needs more >> input in >> > > > > order to work properly. >> > > > > >> > > > > Is there an option in PETSc to tell HYPRE that we are dealing >> with a >> > > > > system of PDEs? (something like: -pc_hypre_boomeramg_...) >> > > > > >> > > > > >> > > > > Thanks for your help. >> > > > > >> > > > > Best regards, >> > > > > >> > > > > Nicolas >> > > > > >> > > > > >> > > > > -- >> > > > > Nicolas BATHFIELD >> > > > > Chalmers University of Technology >> > > > > Shipping and Marine Technology >> > > > > phone: +46 (0)31 772 1476 >> > > > > fax: +46 (0)31 772 3699 >> > > > > >> > > > >> > > > >> > > >> > > >> > > -- >> > > Nicolas BATHFIELD >> > > Chalmers University of Technology >> > > Shipping and Marine Technology >> > > phone: +46 (0)31 772 1476 >> > > fax: +46 (0)31 772 3699 >> > > >> > >> -- Nicolas BATHFIELD Chalmers University of Technology Shipping and Marine Technology phone: +46 (0)31 772 1476 fax: +46 (0)31 772 3699 From bsmith at mcs.anl.gov Wed Apr 25 11:35:28 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 25 Apr 2007 11:35:28 -0500 (CDT) Subject: HYPRE with multiple variables In-Reply-To: <35010.129.16.81.46.1177515489.squirrel@webmail.chalmers.se> References: <26157.193.183.3.2.1176453900.squirrel@webmail.chalmers.se> <9329.193.183.3.2.1176464453.squirrel@webmail.chalmers.se> <8614E121-37D8-4AF3-A67A-A142D27B7B62@student.uu.se> <35010.129.16.81.46.1177515489.squirrel@webmail.chalmers.se> Message-ID: Nicolas, On Wed, 25 Apr 2007, Nicolas Bathfield wrote: > Dear Barry, > > Using MatSetBlockSize(A,5) improved my results greatly. Boomemramg is now > solving the system of equations. Good > Still, the equations I solve are coupled, and my discretization scheme is > meant for a non-segregated solver. As a consequence (I believe), boomeramg > still diverges. How can "Boomeramg be now solving the system of equations" but also diverge? I am so confused. > I would therefore like to use the nodal relaxation in > boomeramg (the hypre command is HYPRE_BOOMERAMGSetNodal) in order to > couple the coarse grid choice for all my variables. I can add this this afternoon. I have to admit I do not understand the difference between HYPRE_BOOMERAMGSetNodal() and hypre_BoomerAMGSetNumFunctions(). Do you? Barry > > How can I achieve this from PETSc? > > Best regards, > > Nicolas > > > > > From PETSc MPIAIJ matrices you need to set the block size of the matrix > > with MatSetBlockSize(A,5) after you have called MatSetType() or > > MatCreateMPIAIJ(). Then HYPRE_BoomerAMGSetNumFunctions() is automatically > > called by PETSc. > > > > Barry > > > > The reason this is done this way instead of as > > -pc_hypre_boomeramg_block_size is the idea that hypre will use the > > properties of the matrix it is given in building the preconditioner so > > the user does not have to pass those properties in seperately directly > > to hypre. > > > > > > On Fri, 13 Apr 2007, Shaman Mahmoudi wrote: > > > >> Hi, > >> > >> int HYPRE_BoomerAMGSetNumFunctions (.....) > >> > >> sets the size of the system of PDEs. > >> > >> With best regards, Shaman Mahmoudi > >> > >> On Apr 13, 2007, at 2:04 PM, Shaman Mahmoudi wrote: > >> > >> > Hi Nicolas, > >> > > >> > You are right. hypre has changed a lot since the version I used. > >> > > >> > I found this interesting information: > >> > > >> > int HYPRE_BOOMERAMGSetNodal(....) > >> > > >> > Sets whether to use the nodal systems version. Default is 0. > >> > > >> > Then information about smoothers: > >> > > >> > One interesting thing there is this, > >> > > >> > HYPRE_BoomerAMGSetDomainType(....) > >> > > >> > 0 - each point is a domain (default) > >> > 1 each node is a domain (only of interest in systems AMG) > >> > 2 .... > >> > > >> > I could not find how you define the nodal displacement ordering. But > >> it > >> > should be there somewhere. > >> > > >> > I read the reference manual for hypre 2.0 > >> > > >> > With best regards, Shaman Mahmoudi > >> > > >> > > >> > On Apr 13, 2007, at 1:40 PM, Nicolas Bathfield wrote: > >> > > >> > > Dear Shaman, > >> > > > >> > > As far as I could understand, there is a BoomerAMG?s systems AMG > >> version > >> > > available. This seems to be exactly what I am looking for, but I > >> just > >> > > don't know how to access it, either through PETSc or directly. > >> > > > >> > > Best regards, > >> > > > >> > > Nicolas > >> > > > >> > > > Hi, > >> > > > > >> > > > You want to exploit the structure of the model? > >> > > > As far as I know, boomeramg can not treat a set of rows or blocks > >> as > >> > > > a molecule, a so called block-smoother? > >> > > > ML 2.0 smoothed aggregation does support it. > >> > > > > >> > > > With best regards, Shaman Mahmoudi > >> > > > > >> > > > On Apr 13, 2007, at 10:45 AM, Nicolas Bathfield wrote: > >> > > > > >> > > > > Hi, > >> > > > > > >> > > > > I am solving the Navier-stokes equations and try to use Hypre as > >> > > > > preconditioner. > >> > > > > Until now, I used PETSc as non-segregated solver and it worked > >> > > > > perfectly. > >> > > > > Things got worse when I decided to use Boomeramg (Hypre). > >> > > > > As I solve a system of PDEs, each cell is represented by 5 rows > >> in my > >> > > > > matrix (I solve for 5 variables). PETSc handles that without > >> problem > >> > > > > apparently, but the coarsening scheme of Boomeramg needs more > >> input in > >> > > > > order to work properly. > >> > > > > > >> > > > > Is there an option in PETSc to tell HYPRE that we are dealing > >> with a > >> > > > > system of PDEs? (something like: -pc_hypre_boomeramg_...) > >> > > > > > >> > > > > > >> > > > > Thanks for your help. > >> > > > > > >> > > > > Best regards, > >> > > > > > >> > > > > Nicolas > >> > > > > > >> > > > > > >> > > > > -- > >> > > > > Nicolas BATHFIELD > >> > > > > Chalmers University of Technology > >> > > > > Shipping and Marine Technology > >> > > > > phone: +46 (0)31 772 1476 > >> > > > > fax: +46 (0)31 772 3699 > >> > > > > > >> > > > > >> > > > > >> > > > >> > > > >> > > -- > >> > > Nicolas BATHFIELD > >> > > Chalmers University of Technology > >> > > Shipping and Marine Technology > >> > > phone: +46 (0)31 772 1476 > >> > > fax: +46 (0)31 772 3699 > >> > > > >> > > >> > > > From shma7099 at student.uu.se Wed Apr 25 11:53:22 2007 From: shma7099 at student.uu.se (Shaman Mahmoudi) Date: Wed, 25 Apr 2007 18:53:22 +0200 Subject: HYPRE with multiple variables In-Reply-To: References: <26157.193.183.3.2.1176453900.squirrel@webmail.chalmers.se> <9329.193.183.3.2.1176464453.squirrel@webmail.chalmers.se> <8614E121-37D8-4AF3-A67A-A142D27B7B62@student.uu.se> <35010.129.16.81.46.1177515489.squirrel@webmail.chalmers.se> Message-ID: <8954FB82-3B27-48CA-8A7C-39DCA7BC3445@student.uu.se> Hi, int HYPRE_BOOMERAMGSetNodal(....) Sets whether to use the nodal systems version. Default is 0, i.e. do not. int HYPRE_BoomerAMGSetNumFunctions (.....) sets the size of the system of PDEs. With best regards, Shaman Mahmoudi On Apr 25, 2007, at 6:35 PM, Barry Smith wrote: > > Nicolas, > > On Wed, 25 Apr 2007, Nicolas Bathfield wrote: > >> Dear Barry, >> >> Using MatSetBlockSize(A,5) improved my results greatly. Boomemramg >> is now >> solving the system of equations. > > Good > >> Still, the equations I solve are coupled, and my discretization >> scheme is >> meant for a non-segregated solver. As a consequence (I believe), >> boomeramg >> still diverges. > > How can "Boomeramg be now solving the system of equations" but also > diverge? I am so confused. > >> I would therefore like to use the nodal relaxation in >> boomeramg (the hypre command is HYPRE_BOOMERAMGSetNodal) in order to >> couple the coarse grid choice for all my variables. > > I can add this this afternoon. > > I have to admit I do not understand the difference between > HYPRE_BOOMERAMGSetNodal() and hypre_BoomerAMGSetNumFunctions(). Do > you? > > Barry >> >> How can I achieve this from PETSc? >> >> Best regards, >> >> Nicolas >> >>> >>> From PETSc MPIAIJ matrices you need to set the block size of >>> the matrix >>> with MatSetBlockSize(A,5) after you have called MatSetType() or >>> MatCreateMPIAIJ(). Then HYPRE_BoomerAMGSetNumFunctions() is >>> automatically >>> called by PETSc. >>> >>> Barry >>> >>> The reason this is done this way instead of as >>> -pc_hypre_boomeramg_block_size is the idea that hypre will >>> use the >>> properties of the matrix it is given in building the >>> preconditioner so >>> the user does not have to pass those properties in seperately >>> directly >>> to hypre. >>> >>> >>> On Fri, 13 Apr 2007, Shaman Mahmoudi wrote: >>> >>>> Hi, >>>> >>>> int HYPRE_BoomerAMGSetNumFunctions (.....) >>>> >>>> sets the size of the system of PDEs. >>>> >>>> With best regards, Shaman Mahmoudi >>>> >>>> On Apr 13, 2007, at 2:04 PM, Shaman Mahmoudi wrote: >>>> >>>>> Hi Nicolas, >>>>> >>>>> You are right. hypre has changed a lot since the version I used. >>>>> >>>>> I found this interesting information: >>>>> >>>>> int HYPRE_BOOMERAMGSetNodal(....) >>>>> >>>>> Sets whether to use the nodal systems version. Default is 0. >>>>> >>>>> Then information about smoothers: >>>>> >>>>> One interesting thing there is this, >>>>> >>>>> HYPRE_BoomerAMGSetDomainType(....) >>>>> >>>>> 0 - each point is a domain (default) >>>>> 1 each node is a domain (only of interest in systems AMG) >>>>> 2 .... >>>>> >>>>> I could not find how you define the nodal displacement >>>>> ordering. But >>>> it >>>>> should be there somewhere. >>>>> >>>>> I read the reference manual for hypre 2.0 >>>>> >>>>> With best regards, Shaman Mahmoudi >>>>> >>>>> >>>>> On Apr 13, 2007, at 1:40 PM, Nicolas Bathfield wrote: >>>>> >>>>>> Dear Shaman, >>>>>> >>>>>> As far as I could understand, there is a BoomerAMG?s systems AMG >>>> version >>>>>> available. This seems to be exactly what I am looking for, but I >>>> just >>>>>> don't know how to access it, either through PETSc or directly. >>>>>> >>>>>> Best regards, >>>>>> >>>>>> Nicolas >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> You want to exploit the structure of the model? >>>>>>> As far as I know, boomeramg can not treat a set of rows or >>>>>>> blocks >>>> as >>>>>>> a molecule, a so called block-smoother? >>>>>>> ML 2.0 smoothed aggregation does support it. >>>>>>> >>>>>>> With best regards, Shaman Mahmoudi >>>>>>> >>>>>>> On Apr 13, 2007, at 10:45 AM, Nicolas Bathfield wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I am solving the Navier-stokes equations and try to use >>>>>>>> Hypre as >>>>>>>> preconditioner. >>>>>>>> Until now, I used PETSc as non-segregated solver and it worked >>>>>>>> perfectly. >>>>>>>> Things got worse when I decided to use Boomeramg (Hypre). >>>>>>>> As I solve a system of PDEs, each cell is represented by 5 >>>>>>>> rows >>>> in my >>>>>>>> matrix (I solve for 5 variables). PETSc handles that without >>>> problem >>>>>>>> apparently, but the coarsening scheme of Boomeramg needs more >>>> input in >>>>>>>> order to work properly. >>>>>>>> >>>>>>>> Is there an option in PETSc to tell HYPRE that we are dealing >>>> with a >>>>>>>> system of PDEs? (something like: -pc_hypre_boomeramg_...) >>>>>>>> >>>>>>>> >>>>>>>> Thanks for your help. >>>>>>>> >>>>>>>> Best regards, >>>>>>>> >>>>>>>> Nicolas >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Nicolas BATHFIELD >>>>>>>> Chalmers University of Technology >>>>>>>> Shipping and Marine Technology >>>>>>>> phone: +46 (0)31 772 1476 >>>>>>>> fax: +46 (0)31 772 3699 >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Nicolas BATHFIELD >>>>>> Chalmers University of Technology >>>>>> Shipping and Marine Technology >>>>>> phone: +46 (0)31 772 1476 >>>>>> fax: +46 (0)31 772 3699 >>>>>> >>>>> >>>> >> >> >> > From nicolas.bathfield at chalmers.se Wed Apr 25 15:14:43 2007 From: nicolas.bathfield at chalmers.se (Nicolas Bathfield) Date: Wed, 25 Apr 2007 22:14:43 +0200 (CEST) Subject: HYPRE with multiple variables Message-ID: <62207.213.66.85.126.1177532083.squirrel@webmail.chalmers.se> Barry, Boomeramg used to diverge right away for all Reynolds numbers and size of grids...After I used MatSetBlockSize, I could see an improvement in some cases probably when the coupling between my variables was not that strong (just a guess). >From papers written by Rob Falgout I read, I could see that without HYPRE_BOOMERAMGSetNodal, boomermang deals with the variables independently, like a segregated solver. Thanks a lot for your help. Best regards, Nicolas > > Nicolas, > > On Wed, 25 Apr 2007, Nicolas Bathfield wrote: > >> Dear Barry, >> >> Using MatSetBlockSize(A,5) improved my results greatly. Boomemramg is >> now >> solving the system of equations. > > Good > >> Still, the equations I solve are coupled, and my discretization scheme >> is >> meant for a non-segregated solver. As a consequence (I believe), >> boomeramg >> still diverges. > > How can "Boomeramg be now solving the system of equations" but also > diverge? I am so confused. > >> I would therefore like to use the nodal relaxation in >> boomeramg (the hypre command is HYPRE_BOOMERAMGSetNodal) in order to >> couple% From abaker at llnl.gov Wed Apr 25 15:44:59 2007 From: abaker at llnl.gov (Allison Baker) Date: Wed, 25 Apr 2007 13:44:59 -0700 Subject: HYPRE with multiple variables In-Reply-To: References: <26157.193.183.3.2.1176453900.squirrel@webmail.chalmers.se> <9329.193.183.3.2.1176464453.squirrel@webmail.chalmers.se> <8614E121-37D8-4AF3-A67A-A142D27B7B62@student.uu.se> <35010.129.16.81.46.1177515489.squirrel@webmail.chalmers.se> Message-ID: <462FBDCB.3010309@llnl.gov> Hi Barry and Nicolas, To clarify, HYPRE_BoomerAMGSetNumFunctions(solver, int num_functions) tells AMG to solve a system of equations with the specified number of functions/unknowns. The default AMG scheme to solve a PDE system is the "unknown" approach. (The coarsening and interpolation are determined by looking at each unknown/function independently - therefore you can imagine that the coarse grids are generally not the same for each variable. This approach generally works well unless you have strong coupling between unknowns.) HYPRE_BOOMERAMGSetNodal(solver, int nodal ) tells AMG to coarsen such that each variable has the same coarse grid - sometimes this is more "physical" for a particular problem. The value chosen here for nodal determines how strength of connection is determined between the coupled system. I suggest setting nodal = 1, which uses a Frobenius norm. This does NOT tell AMG to use nodal relaxation. If you want to use nodal relaxation in hypre there are two choices: (1) If you call HYPRE_BOOMERAMGSetNodal, then you can additionally do nodal relaxation via the schwarz smoother option in hypre. I did not implement this in the Petsc interface, but it could be done easy enough. The following four functions need to be called: HYPRE_BoomerAMGSetSmoothType(solver, 6); HYPRE_BoomerAMGSetDomainType(solver, 1); HYPRE_BoomerAMGSetOverlap(solver, 0); HYPRE_BoomerAMGSetSmoothNumLevels(solver, num_levels); (Set num_levels to number of levels on which you want nodal smoothing, i.e. 1=just the fine grid, 2= fine grid and the grid below, etc. I find that doing nodal relaxation on just the finest level is generally sufficient.) Note that the interpolation scheme used will be the same as in the unknown approach - so this is what we call a hybrid systems method. (2) You can do both nodal smoothing and a nodal interpolation scheme. While this is currently implemented in 2.0.0, it is not advertised (i.e., mentioned at all in the user's manual) because it is not yet implemented very efficiently (the fine grid matrix is converted to a block matrix - and both are stored), and we have not found it to be as effective as advertised elsewhere (this is an area of current research for us)..... If you want to try it anyway, let me know and I will provide more info. Hope this helps, Allison Barry Smith wrote: > Nicolas, > > On Wed, 25 Apr 2007, Nicolas Bathfield wrote: > > >> Dear Barry, >> >> Using MatSetBlockSize(A,5) improved my results greatly. Boomemramg is now >> solving the system of equations. >> > > Good > > >> Still, the equations I solve are coupled, and my discretization scheme is >> meant for a non-segregated solver. As a consequence (I believe), boomeramg >> still diverges. >> > > How can "Boomeramg be now solving the system of equations" but also > diverge? I am so confused. > > >> I would therefore like to use the nodal relaxation in >> boomeramg (the hypre command is HYPRE_BOOMERAMGSetNodal) in order to >> couple the coarse grid choice for all my variables. >> > > I can add this this afternoon. > > I have to admit I do not understand the difference between > HYPRE_BOOMERAMGSetNodal() and hypre_BoomerAMGSetNumFunctions(). Do you? > > Barry > >> How can I achieve this from PETSc? >> >> Best regards, >> >> Nicolas >> >> >>> From PETSc MPIAIJ matrices you need to set the block size of the matrix >>> with MatSetBlockSize(A,5) after you have called MatSetType() or >>> MatCreateMPIAIJ(). Then HYPRE_BoomerAMGSetNumFunctions() is automatically >>> called by PETSc. >>> >>> Barry >>> >>> The reason this is done this way instead of as >>> -pc_hypre_boomeramg_block_size is the idea that hypre will use the >>> properties of the matrix it is given in building the preconditioner so >>> the user does not have to pass those properties in seperately directly >>> to hypre. >>> >>> >>> On Fri, 13 Apr 2007, Shaman Mahmoudi wrote: >>> >>> >>>> Hi, >>>> >>>> int HYPRE_BoomerAMGSetNumFunctions (.....) >>>> >>>> sets the size of the system of PDEs. >>>> >>>> With best regards, Shaman Mahmoudi >>>> >>>> On Apr 13, 2007, at 2:04 PM, Shaman Mahmoudi wrote: >>>> >>>> >>>>> Hi Nicolas, >>>>> >>>>> You are right. hypre has changed a lot since the version I used. >>>>> >>>>> I found this interesting information: >>>>> >>>>> int HYPRE_BOOMERAMGSetNodal(....) >>>>> >>>>> Sets whether to use the nodal systems version. Default is 0. >>>>> >>>>> Then information about smoothers: >>>>> >>>>> One interesting thing there is this, >>>>> >>>>> HYPRE_BoomerAMGSetDomainType(....) >>>>> >>>>> 0 - each point is a domain (default) >>>>> 1 each node is a domain (only of interest in systems AMG) >>>>> 2 .... >>>>> >>>>> I could not find how you define the nodal displacement ordering. But >>>>> >>>> it >>>> >>>>> should be there somewhere. >>>>> >>>>> I read the reference manual for hypre 2.0 >>>>> >>>>> With best regards, Shaman Mahmoudi >>>>> >>>>> >>>>> On Apr 13, 2007, at 1:40 PM, Nicolas Bathfield wrote: >>>>> >>>>> >>>>>> Dear Shaman, >>>>>> >>>>>> As far as I could understand, there is a BoomerAMG?s systems AMG >>>>>> >>>> version >>>> >>>>>> available. This seems to be exactly what I am looking for, but I >>>>>> >>>> just >>>> >>>>>> don't know how to access it, either through PETSc or directly. >>>>>> >>>>>> Best regards, >>>>>> >>>>>> Nicolas >>>>>> >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> You want to exploit the structure of the model? >>>>>>> As far as I know, boomeramg can not treat a set of rows or blocks >>>>>>> >>>> as >>>> >>>>>>> a molecule, a so called block-smoother? >>>>>>> ML 2.0 smoothed aggregation does support it. >>>>>>> >>>>>>> With best regards, Shaman Mahmoudi >>>>>>> >>>>>>> On Apr 13, 2007, at 10:45 AM, Nicolas Bathfield wrote: >>>>>>> >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I am solving the Navier-stokes equations and try to use Hypre as >>>>>>>> preconditioner. >>>>>>>> Until now, I used PETSc as non-segregated solver and it worked >>>>>>>> perfectly. >>>>>>>> Things got worse when I decided to use Boomeramg (Hypre). >>>>>>>> As I solve a system of PDEs, each cell is represented by 5 rows >>>>>>>> >>>> in my >>>> >>>>>>>> matrix (I solve for 5 variables). PETSc handles that without >>>>>>>> >>>> problem >>>> >>>>>>>> apparently, but the coarsening scheme of Boomeramg needs more >>>>>>>> >>>> input in >>>> >>>>>>>> order to work properly. >>>>>>>> >>>>>>>> Is there an option in PETSc to tell HYPRE that we are dealing >>>>>>>> >>>> with a >>>> >>>>>>>> system of PDEs? (something like: -pc_hypre_boomeramg_...) >>>>>>>> >>>>>>>> >>>>>>>> Thanks for your help. >>>>>>>> >>>>>>>> Best regards, >>>>>>>> >>>>>>>> Nicolas >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Nicolas BATHFIELD >>>>>>>> Chalmers University of Technology >>>>>>>> Shipping and Marine Technology >>>>>>>> phone: +46 (0)31 772 1476 >>>>>>>> fax: +46 (0)31 772 3699 >>>>>>>> >>>>>>>> >>>>>>> >>>>>> -- >>>>>> Nicolas BATHFIELD >>>>>> Chalmers University of Technology >>>>>> Shipping and Marine Technology >>>>>> phone: +46 (0)31 772 1476 >>>>>> fax: +46 (0)31 772 3699 >>>>>> >>>>>> >> >> > > From bsmith at mcs.anl.gov Wed Apr 25 16:59:46 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 25 Apr 2007 16:59:46 -0500 (CDT) Subject: HYPRE with multiple variables In-Reply-To: <462FBDCB.3010309@llnl.gov> References: <26157.193.183.3.2.1176453900.squirrel@webmail.chalmers.se> <9329.193.183.3.2.1176464453.squirrel@webmail.chalmers.se> <8614E121-37D8-4AF3-A67A-A142D27B7B62@student.uu.se> <35010.129.16.81.46.1177515489.squirrel@webmail.chalmers.se> <462FBDCB.3010309@llnl.gov> Message-ID: Nicolas, I have added support for both 1 and/or 2 to petsc-dev http://www-unix.mcs.anl.gov/petsc/petsc-as/developers/index.html The two new options are -pc_hypre_boomeramg_nodal_coarsen and -pc_hypre_boomeramg_nodal_relaxation Hi Barry and Nicolas, > > To clarify, > > HYPRE_BoomerAMGSetNumFunctions(solver, int num_functions) tells AMG to solve a > system of equations with the specified number of functions/unknowns. The > default AMG scheme to solve a PDE system is the "unknown" approach. (The > coarsening and interpolation are determined by looking at each > unknown/function independently - therefore you can imagine that the coarse > grids are generally not the same for each variable. This approach generally > works well unless you have strong coupling between unknowns.) > > HYPRE_BOOMERAMGSetNodal(solver, int nodal ) tells AMG to coarsen such that > each variable has the same coarse grid - sometimes this is more "physical" for > a particular problem. The value chosen here for nodal determines how strength > of connection is determined between the coupled system. I suggest setting > nodal = 1, which uses a Frobenius norm. This does NOT tell AMG to use nodal > relaxation. > > If you want to use nodal relaxation in hypre there are two choices: > > (1) If you call HYPRE_BOOMERAMGSetNodal, then you can additionally do nodal > relaxation via the schwarz smoother option in hypre. I did not implement this > in the Petsc interface, but it could be done easy enough. The following four > functions need to be called: > > HYPRE_BoomerAMGSetSmoothType(solver, 6); > HYPRE_BoomerAMGSetDomainType(solver, 1); > HYPRE_BoomerAMGSetOverlap(solver, 0); > HYPRE_BoomerAMGSetSmoothNumLevels(solver, num_levels); (Set num_levels > to number of levels on which you want nodal smoothing, i.e. 1=just the fine > grid, 2= fine grid and the grid below, etc. I find that doing nodal > relaxation on just the finest level is generally sufficient.) Note that the > interpolation scheme used will be the same as in the unknown approach - so > this is what we call a hybrid systems method. > > (2) You can do both nodal smoothing and a nodal interpolation scheme. While > this is currently implemented in 2.0.0, it is not advertised (i.e., mentioned > at all in the user's manual) because it is not yet implemented very > efficiently (the fine grid matrix is converted to a block matrix - and both > are stored), and we have not found it to be as effective as advertised > elsewhere (this is an area of current research for us)..... If you want to try > it anyway, let me know and I will provide more info. > > Hope this helps, > Allison > > > Barry Smith wrote: > > Nicolas, > > > > On Wed, 25 Apr 2007, Nicolas Bathfield wrote: > > > > > > > Dear Barry, > > > > > > Using MatSetBlockSize(A,5) improved my results greatly. Boomemramg is now > > > solving the system of equations. > > > > > > > Good > > > > > > > Still, the equations I solve are coupled, and my discretization scheme is > > > meant for a non-segregated solver. As a consequence (I believe), boomeramg > > > still diverges. > > > > > > > How can "Boomeramg be now solving the system of equations" but also > > diverge? I am so confused. > > > > > > > I would therefore like to use the nodal relaxation in > > > boomeramg (the hypre command is HYPRE_BOOMERAMGSetNodal) in order to > > > couple the coarse grid choice for all my variables. > > > > > > > I can add this this afternoon. > > > > I have to admit I do not understand the difference between > > HYPRE_BOOMERAMGSetNodal() and hypre_BoomerAMGSetNumFunctions(). Do you? > > > > Barry > > > > > How can I achieve this from PETSc? > > > > > > Best regards, > > > > > > Nicolas > > > > > > > > > > From PETSc MPIAIJ matrices you need to set the block size of the > > > > matrix > > > > with MatSetBlockSize(A,5) after you have called MatSetType() or > > > > MatCreateMPIAIJ(). Then HYPRE_BoomerAMGSetNumFunctions() is > > > > automatically > > > > called by PETSc. > > > > > > > > Barry > > > > > > > > The reason this is done this way instead of as > > > > -pc_hypre_boomeramg_block_size is the idea that hypre will use the > > > > properties of the matrix it is given in building the preconditioner so > > > > the user does not have to pass those properties in seperately directly > > > > to hypre. > > > > > > > > > > > > On Fri, 13 Apr 2007, Shaman Mahmoudi wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > int HYPRE_BoomerAMGSetNumFunctions (.....) > > > > > > > > > > sets the size of the system of PDEs. > > > > > > > > > > With best regards, Shaman Mahmoudi > > > > > > > > > > On Apr 13, 2007, at 2:04 PM, Shaman Mahmoudi wrote: > > > > > > > > > > > > > > > > Hi Nicolas, > > > > > > > > > > > > You are right. hypre has changed a lot since the version I used. > > > > > > > > > > > > I found this interesting information: > > > > > > > > > > > > int HYPRE_BOOMERAMGSetNodal(....) > > > > > > > > > > > > Sets whether to use the nodal systems version. Default is 0. > > > > > > > > > > > > Then information about smoothers: > > > > > > > > > > > > One interesting thing there is this, > > > > > > > > > > > > HYPRE_BoomerAMGSetDomainType(....) > > > > > > > > > > > > 0 - each point is a domain (default) > > > > > > 1 each node is a domain (only of interest in systems AMG) > > > > > > 2 .... > > > > > > > > > > > > I could not find how you define the nodal displacement ordering. But > > > > > > > > > > > it > > > > > > > > > > > should be there somewhere. > > > > > > > > > > > > I read the reference manual for hypre 2.0 > > > > > > > > > > > > With best regards, Shaman Mahmoudi > > > > > > > > > > > > > > > > > > On Apr 13, 2007, at 1:40 PM, Nicolas Bathfield wrote: > > > > > > > > > > > > > > > > > > > Dear Shaman, > > > > > > > > > > > > > > As far as I could understand, there is a BoomerAMG?s systems AMG > > > > > > > > > > > > version > > > > > > > > > > > > available. This seems to be exactly what I am looking for, but I > > > > > > > > > > > > just > > > > > > > > > > > > don't know how to access it, either through PETSc or directly. > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > Nicolas > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > You want to exploit the structure of the model? > > > > > > > > As far as I know, boomeramg can not treat a set of rows or > > > > > > > > blocks > > > > > > > > > > > > > as > > > > > > > > > > > > > a molecule, a so called block-smoother? > > > > > > > > ML 2.0 smoothed aggregation does support it. > > > > > > > > > > > > > > > > With best regards, Shaman Mahmoudi > > > > > > > > > > > > > > > > On Apr 13, 2007, at 10:45 AM, Nicolas Bathfield wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > I am solving the Navier-stokes equations and try to use Hypre > > > > > > > > > as > > > > > > > > > preconditioner. > > > > > > > > > Until now, I used PETSc as non-segregated solver and it worked > > > > > > > > > perfectly. > > > > > > > > > Things got worse when I decided to use Boomeramg (Hypre). > > > > > > > > > As I solve a system of PDEs, each cell is represented by 5 > > > > > > > > > rows > > > > > > > > > > > > > > in my > > > > > > > > > > > > > > matrix (I solve for 5 variables). PETSc handles that without > > > > > > > > > > > > > > problem > > > > > > > > > > > > > > apparently, but the coarsening scheme of Boomeramg needs more > > > > > > > > > > > > > > input in > > > > > > > > > > > > > > order to work properly. > > > > > > > > > > > > > > > > > > Is there an option in PETSc to tell HYPRE that we are dealing > > > > > > > > > > > > > > with a > > > > > > > > > > > > > > system of PDEs? (something like: -pc_hypre_boomeramg_...) > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your help. > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > > > Nicolas > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Nicolas BATHFIELD > > > > > > > > > Chalmers University of Technology > > > > > > > > > Shipping and Marine Technology > > > > > > > > > phone: +46 (0)31 772 1476 > > > > > > > > > fax: +46 (0)31 772 3699 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Nicolas BATHFIELD > > > > > > > Chalmers University of Technology > > > > > > > Shipping and Marine Technology > > > > > > > phone: +46 (0)31 772 1476 > > > > > > > fax: +46 (0)31 772 3699 > > > > > > > > > > > > > > > > > > > > > > > > > > From wsubber at connect.carleton.ca Wed Apr 25 18:24:13 2007 From: wsubber at connect.carleton.ca (Waad Subber) Date: Wed, 25 Apr 2007 19:24:13 -0400 (EDT) Subject: Domain Decomposition Method Message-ID: <2860074.1177543453174.JavaMail.wsubber@connect.carleton.ca> Hello everyone: I am new to PETSC. Reading the tutorials, I understand that PETSC supports domain decomposition with additive Schwartz and iterative substructuring (balancing Neumann-Neumann). I am looking for some example codes involving these domain decomposition methods for 1D and 2D PDE (preferable linear PDE) so that I can get started. Can anyone kindly point me to the right place where I can find them. Thanks :) waad From ondrej at certik.cz Wed Apr 25 18:48:10 2007 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 26 Apr 2007 01:48:10 +0200 Subject: a current state of python bindings Message-ID: <85b5c3130704251648m39d41531s7d2e260da49c2119@mail.gmail.com> Hi, I am using PETSc from Python. I read the post: http://www-unix.mcs.anl.gov/web-mail-archive/lists/petsc-users/2006/08/msg00090.html I am currently using the petsc4py, but unfortunately, they don't seem to have uptodate webpage. The most recent version, that works with petsc 2.3.2 can be found here: http://cheeseshop.python.org/pypi/petsc4py/0.7.2 However, there are some bindings inside the petsc tarball, but I didn't find any documentation for them - are they deprecated? Also, I would like to create a debian package "python-petsc" with python bindings to petsc (and get it to debian main distribution), so I wanted to check with you, what the most advanced python bindigs actually are (and which I should package). Thanks, Ondrej From bsmith at mcs.anl.gov Wed Apr 25 18:49:54 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 25 Apr 2007 18:49:54 -0500 (CDT) Subject: a current state of python bindings In-Reply-To: <85b5c3130704251648m39d41531s7d2e260da49c2119@mail.gmail.com> References: <85b5c3130704251648m39d41531s7d2e260da49c2119@mail.gmail.com> Message-ID: On Thu, 26 Apr 2007, Ondrej Certik wrote: > Hi, > > I am using PETSc from Python. I read the post: > > http://www-unix.mcs.anl.gov/web-mail-archive/lists/petsc-users/2006/08/msg00090.html > > I am currently using the petsc4py, but unfortunately, they don't seem > to have uptodate webpage. The most recent version, that works with > petsc 2.3.2 can be found here: > > http://cheeseshop.python.org/pypi/petsc4py/0.7.2 > > However, there are some bindings inside the petsc tarball, but I > didn't find any documentation for them - are they deprecated? Yes. > > Also, I would like to create a debian package "python-petsc" with > python bindings to petsc (and get it to debian main distribution), so > I wanted to check with you, what the most advanced python bindigs > actually are (and which I should package). At this time we are recommending petsc4py. Barry > > Thanks, > Ondrej > > From knepley at gmail.com Wed Apr 25 19:45:28 2007 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 25 Apr 2007 19:45:28 -0500 Subject: Domain Decomposition Method In-Reply-To: <2860074.1177543453174.JavaMail.wsubber@connect.carleton.ca> References: <2860074.1177543453174.JavaMail.wsubber@connect.carleton.ca> Message-ID: If you have a structured problem, I recommend looking at a DA example. If it is unstructured, the approach is to create the system matrix as a Mat. The Additive Schwarz method is a preconditioner (PC) you specify on the command line -pc_type asm. Theway to find example is to go to the online webpage help. Below each function is a ist of the examples it is used in. Thanks, Matt On 4/25/07, Waad Subber wrote: > Hello everyone: > > I am new to PETSC. Reading the tutorials, I understand that PETSC > supports domain decomposition with additive Schwartz and iterative > substructuring (balancing Neumann-Neumann). I am looking for some > example codes involving these domain decomposition methods for 1D and > 2D PDE (preferable linear PDE) so that I can get started. Can anyone > kindly point me to the right place where I can find them. > > Thanks :) > > waad > > -- The government saving money is like me spilling beer. It happens, but never on purpose. From mfhoel at ifi.uio.no Thu Apr 26 05:09:18 2007 From: mfhoel at ifi.uio.no (Mads Fredrik Skoge Hoel) Date: Thu, 26 Apr 2007 12:09:18 +0200 (CEST) Subject: a current state of python bindings In-Reply-To: References: <85b5c3130704251648m39d41531s7d2e260da49c2119@mail.gmail.com> Message-ID: <4483.129.240.176.76.1177582158.squirrel@webmail.uio.no> Hi Ondrej, I tried out petsc4py, i could not find any documentation. I used alot of time just trying to get it to work. And without documentation, i tried to guess, and finally managed to construct a Mat object. It seems that petsc4py is currently in an experimental version. You might also want to have a look at PyPetsc: https://elefant.developer.nicta.com.au/documentation/userguide/PyPetscManual.pdf and https://elefant.developer.nicta.com.au/developer/designdocs/PyPetscDesignRef.pdf I have not tried the PyPetsc, but at least there is documentation. Regards, Mads > > > On Thu, 26 Apr 2007, Ondrej Certik wrote: > >> Hi, >> >> I am using PETSc from Python. I read the post: >> >> http://www-unix.mcs.anl.gov/web-mail-archive/lists/petsc-users/2006/08/msg00090.html >> >> I am currently using the petsc4py, but unfortunately, they don't seem >> to have uptodate webpage. The most recent version, that works with >> petsc 2.3.2 can be found here: >> >> http://cheeseshop.python.org/pypi/petsc4py/0.7.2 >> >> However, there are some bindings inside the petsc tarball, but I >> didn't find any documentation for them - are they deprecated? > > Yes. >> >> Also, I would like to create a debian package "python-petsc" with >> python bindings to petsc (and get it to debian main distribution), so >> I wanted to check with you, what the most advanced python bindigs >> actually are (and which I should package). > > At this time we are recommending petsc4py. > > Barry > >> >> Thanks, >> Ondrej >> >> > > From ondrej at certik.cz Thu Apr 26 05:52:16 2007 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 26 Apr 2007 12:52:16 +0200 Subject: a current state of python bindings In-Reply-To: <4483.129.240.176.76.1177582158.squirrel@webmail.uio.no> References: <85b5c3130704251648m39d41531s7d2e260da49c2119@mail.gmail.com> <4483.129.240.176.76.1177582158.squirrel@webmail.uio.no> Message-ID: <85b5c3130704260352r1339a62bi53b3ef2f6243b8eb@mail.gmail.com> Hi Mads, yes, I know about the bindings from elefant, but they seem to be just a direct translation of C commands to Python, but petsc4py seem to be more pythonic. I know that there is no documentation to them, but reading the sources (there are a lot of docstrings) was enough for me. If you want to look at an example: http://grainmodel.googlecode.com/svn/trunk/femlib/ look into the "__init__.py", where I set up paths for petsc4py, and then into "fem.py", where I import petsc4py and construct matrices and solve them. But I just wanted to check what people at this mailinglist use. Experimenting with petsc in python is just fun. And as to the installation - there are many options to the petsc package, so it's difficult (almost impossible) to create one (binary) package for all, on the other hand, when developing my program, I want petsc to just work (I can play with the options later). I use Debian, thus I just want to "apt-get install petsc-dev" to have a functioning petsc installation and then "apt-get install python-petsc" to have functioning python bindings. So my contribution to this is to create the python-petsc Debian package and get it into Debian. Ondrej On 4/26/07, Mads Fredrik Skoge Hoel wrote: > Hi Ondrej, > > I tried out petsc4py, i could not find any documentation. I used alot of > time just trying to get it to work. And without documentation, i tried to > guess, and finally managed to construct a Mat object. It seems that > petsc4py is currently in an experimental version. > > You might also want to have a look at PyPetsc: > https://elefant.developer.nicta.com.au/documentation/userguide/PyPetscManual.pdf > and > https://elefant.developer.nicta.com.au/developer/designdocs/PyPetscDesignRef.pdf > > I have not tried the PyPetsc, but at least there is documentation. > > Regards, > Mads > > > > > > > On Thu, 26 Apr 2007, Ondrej Certik wrote: > > > >> Hi, > >> > >> I am using PETSc from Python. I read the post: > >> > >> http://www-unix.mcs.anl.gov/web-mail-archive/lists/petsc-users/2006/08/msg00090.html > >> > >> I am currently using the petsc4py, but unfortunately, they don't seem > >> to have uptodate webpage. The most recent version, that works with > >> petsc 2.3.2 can be found here: > >> > >> http://cheeseshop.python.org/pypi/petsc4py/0.7.2 > >> > >> However, there are some bindings inside the petsc tarball, but I > >> didn't find any documentation for them - are they deprecated? > > > > Yes. > >> > >> Also, I would like to create a debian package "python-petsc" with > >> python bindings to petsc (and get it to debian main distribution), so > >> I wanted to check with you, what the most advanced python bindigs > >> actually are (and which I should package). > > > > At this time we are recommending petsc4py. > > > > Barry > > > >> > >> Thanks, > >> Ondrej > >> > >> > > > > > > >