From aja2111 at columbia.edu Mon Jan 1 19:19:24 2007 From: aja2111 at columbia.edu (Aron Ahmadia) Date: Mon, 1 Jan 2007 15:19:24 -1000 Subject: How to code for parallel computing using PETSC in CFD In-Reply-To: <804ab5d40612302331m1c49689ax88f178dc377412c3@mail.gmail.com> References: <804ab5d40612302331m1c49689ax88f178dc377412c3@mail.gmail.com> Message-ID: <37604ab40701011719t58c2ed70v890968e84d844d95@mail.gmail.com> Dear Ben, If I was doing this for an optimized code on a stationary grid, I would run the if tests once at the beginning to generate lists of points for each boundary, and then run the for loops over those lists. ~Aron On 12/30/06, Ben Tay wrote: > Hi, > > I'm now trying to modify my source code to enable parallel computing. I > tried to use the tutorial example and I managed to get it working. However, > in the example, there are a lot of "if"s statement which is not efficient. > > In my CFD code, I've subroutine meant for corner cells, north, south, east , > west and finally internal cells. How should I modify my code then? > > My current idea is to insert a check statement into each subroutine ie " if > II>=Istart and II<=Iend-1 then ...." > > That's the simplest and least modification required. But does anyone know a > better way? > > > Thanks and happy new year to all! From dimitri.lecas at free.fr Mon Jan 8 13:16:54 2007 From: dimitri.lecas at free.fr (Dimitri Lecas) Date: Mon, 08 Jan 2007 20:16:54 +0100 Subject: Using parMetis in petsc for ordering Message-ID: <45A298A6.3020400@free.fr> Hello, I have to test the ParMetis ordering for factorization and i would like to known if it's possible to use a user ordering ? If i understand the manual correctly, i have to use MatOrderingRegisterDynamic and PCFactorSetMatOrdering but the sentence "Currently we support orderings only for sequential matrices" in 16.2 of manual, disturb me. What does it means ? Best regards -- Dimitri Lecas **** ** From bsmith at mcs.anl.gov Mon Jan 8 15:14:12 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 8 Jan 2007 15:14:12 -0600 (CST) Subject: Using parMetis in petsc for ordering In-Reply-To: <45A298A6.3020400@free.fr> References: <45A298A6.3020400@free.fr> Message-ID: 1) The PETSc LU and Cholesky solvers only run sequentially. 2) The parallel LU and Cholesky solvers PETSc interfaces to, SuperLU_dist, MUMPS, Spooles, DSCPACK do NOT accept an external ordering provided for them. Hence we do not have any setup for doing parallel matrix orderings for factorizations, since we cannot use them. We could allow calling a parallel ordering but I'm not sure what it would be useful for. Barry On Mon, 8 Jan 2007, Dimitri Lecas wrote: > Hello, > > I have to test the ParMetis ordering for factorization and i would like to > known if it's possible to use a user ordering ? > > If i understand the manual correctly, i have to use MatOrderingRegisterDynamic > and PCFactorSetMatOrdering but the sentence "Currently we support orderings > only for sequential matrices" in 16.2 of manual, disturb me. What does it > means ? > > Best regards > > From mafunk at nmsu.edu Mon Jan 8 20:26:08 2007 From: mafunk at nmsu.edu (Matt Funk) Date: Mon, 8 Jan 2007 19:26:08 -0700 Subject: MatSetValues Message-ID: <200701081926.09781.mafunk@nmsu.edu> Hi, i was wondering in MatSetValues, when i need to specify the number of columns, whether that is the number of column entries per row? So, if i want to insert two rows at five entries each, do i specify 5 or 10 for the number of columns? So something like this: m_ierr = MatSetValues(m_globalMatrix, //the matrix 2, //the number of rows GlobalRowIndices, //and their global indices 5, //the number of columns GlobalColIndices, //and their global indices Values, //the value to be inserted INSERT_VALUES); and where the size of GlobalRowIndices is of size 2 and the size of GlobalColIndices is 10 as well as the size of Values? thanks mat From bsmith at mcs.anl.gov Mon Jan 8 20:51:35 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 8 Jan 2007 20:51:35 -0600 (CST) Subject: MatSetValues In-Reply-To: <200701081926.09781.mafunk@nmsu.edu> References: <200701081926.09781.mafunk@nmsu.edu> Message-ID: It is 5, also the length of GlobalColIndices is 5, not 10. Each row you pass in has the values in the same columns. Barry On Mon, 8 Jan 2007, Matt Funk wrote: > Hi, > > i was wondering in MatSetValues, when i need to specify the number of columns, > whether that is the number of column entries per row? > > So, if i want to insert two rows at five entries each, do i specify 5 or 10 > for the number of columns? > > So something like this: > > m_ierr = MatSetValues(m_globalMatrix, //the matrix > 2, //the number of rows > GlobalRowIndices, //and their global indices > 5, //the number of columns > GlobalColIndices, //and their global indices > Values, //the value to be inserted > INSERT_VALUES); > > and where the size of GlobalRowIndices is of size 2 and the size of > GlobalColIndices is 10 as well as the size of Values? > > > thanks > mat > > From misabel at litec.csic.es Tue Jan 9 04:42:44 2007 From: misabel at litec.csic.es (Isabel Gil) Date: Tue, 09 Jan 2007 11:42:44 +0100 Subject: Solve a system with two processors Message-ID: <45A371A4.3050802@litec.csic.es> Hello PETSc's users! I have a problem with my parallel program. If I run it with only one processor (mpirun -np 1...), everything is all right but if I use 2 processors (mpi -np 2...), the problem appears. When I want to solve my matricial system through a solver, after having created my matrices without any hitch, I have the following problem: The two processes that are running in two different processors using 100% of CPU and 14.6% of memory each of them, end up using 0% of CPU and 14.6% of memory and, although the program doesn't break down, both of the processes don't do anything. This looks like a lockup: one processor is waiting for a message from the second one, but the second one thinks that it doesn't have to send anything and is instead waiting for something else from the first processor. But I don't know how to avoid this problem since it is supposed that I don't have to use "MPI_Send()" and "MPI_Recv()" functions because PETSc itself deals with it, do I?. Any advice will be welcome. Thanks in advance and best regards Isa From bsmith at mcs.anl.gov Tue Jan 9 08:31:08 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 9 Jan 2007 08:31:08 -0600 (CST) Subject: Solve a system with two processors In-Reply-To: <45A371A4.3050802@litec.csic.es> References: <45A371A4.3050802@litec.csic.es> Message-ID: Isa, The PETSc solvers are all deadlock free, this means that if used with a proper MPI implementation it will never deadlock inside the solvers. The easiest way (and best and the one you should use) to determine what is causing the problem is to use the additional option -start_in_debugger when the two xterms popup then type cont in both debuggers, wait and wait until you think it is stuck and then do control-c in each window and then where, it will show you exactly where it is hanging. (Do this with the debug version of PETSc, that is config/configure.py should not be run with --with-debugging=0. Good luck, Barry Should become a FAQ question On Tue, 9 Jan 2007, Isabel Gil wrote: > Hello PETSc's users! > > I have a problem with my parallel program. > If I run it with only one processor (mpirun -np 1...), everything is all right > but if I use 2 processors (mpi -np 2...), the problem appears. > > When I want to solve my matricial system through a solver, after having > created my matrices without any hitch, I have the following problem: > > The two processes that are running in two different processors using 100% of > CPU and 14.6% of memory each of them, end up using 0% of CPU and 14.6% of > memory and, although the program doesn't break down, both of the processes > don't do anything. This looks like a lockup: one processor is waiting for a > message from the second one, but the second one thinks that it doesn't have to > send anything and is instead waiting for something else from the first > processor. > > But I don't know how to avoid this problem since it is supposed that I don't > have to use "MPI_Send()" and "MPI_Recv()" functions because PETSc itself deals > with it, do I?. > Any advice will be welcome. > > Thanks in advance and best regards > Isa > > From zonexo at gmail.com Tue Jan 9 20:55:26 2007 From: zonexo at gmail.com (Ben Tay) Date: Wed, 10 Jan 2007 10:55:26 +0800 Subject: Using library from another linux system Message-ID: <804ab5d40701091855x16f5a757vdada44d9f9bd2587@mail.gmail.com> Hi, I've some problems compiling PETSC on one of my school's server. I've already compiled a working library on another linux system. Is it possible for me to just copy the library from one to another. They are both redhat linux system and I think they're running on xeon processors. What are the things which I must looked out for? e.g. directory location etc... Can the compiler be of different version? The server I compiled on uses ifort 9 or 7 while on the problem system ifort 8 is used. Can the original library be shared or must it be static? Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Jan 9 21:11:52 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 9 Jan 2007 21:11:52 -0600 (CST) Subject: Using library from another linux system In-Reply-To: <804ab5d40701091855x16f5a757vdada44d9f9bd2587@mail.gmail.com> References: <804ab5d40701091855x16f5a757vdada44d9f9bd2587@mail.gmail.com> Message-ID: On Wed, 10 Jan 2007, Ben Tay wrote: > Hi, > > I've some problems compiling PETSC on one of my school's server. send us configure.log at petsc-maint at mcs.anl.gov and we take a look at the problem. Its probably easier to get this working than trying the alternatives below.. > I've already compiled a working library on another linux system. Is > it possible for me to just copy the library from one to > another. They are both redhat linux system and I think they're > running on xeon processors. > > What are the things which I must looked out for? e.g. directory location > etc... It might work [see notes below] > Can the compiler be of different version? The server I compiled on uses > ifort 9 or 7 while on the problem system ifort 8 is used. yeah difference in compiler verson can cause problems.. [I'm not sure if there are issues between these intel compiler versions]. If it were to work - you can copy the whole petsc tree with 'rsync -a' > Can the original library be shared or must it be static? If PETSc was configured with Sharedlibs, there could be issues after the move. Rebuilding sharedlibs might work arround this problem.. make deleteshraed shared There could also be issues with external libraries. They need to exist on the other machine - in the exact same location as the first one. [otherwise you will get errors at compile time. Satish From zonexo at gmail.com Tue Jan 9 22:13:18 2007 From: zonexo at gmail.com (Ben Tay) Date: Wed, 10 Jan 2007 12:13:18 +0800 Subject: Must mpif90 be used to compile parallel version of code? Message-ID: <804ab5d40701092013o6152c445ib3c61d98343bf37@mail.gmail.com> Hi, I'm new to mpi. If I'm coding my code to use PETSC to solve my linear eqn in parallel, must I compile in mpif90 (for fortran) or can I compile in ifort and then instruct to send the job to n processors? Thank you. From knepley at gmail.com Tue Jan 9 22:16:57 2007 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 9 Jan 2007 22:16:57 -0600 Subject: Must mpif90 be used to compile parallel version of code? In-Reply-To: <804ab5d40701092013o6152c445ib3c61d98343bf37@mail.gmail.com> References: <804ab5d40701092013o6152c445ib3c61d98343bf37@mail.gmail.com> Message-ID: MPI is a library. mpif90 just adds the correct flags. Underneath it is calling whatever compiler you configured with. Matt On 1/9/07, Ben Tay wrote: > Hi, > > I'm new to mpi. If I'm coding my code to use PETSC to solve my linear > eqn in parallel, must I compile in mpif90 (for fortran) or can I > compile in ifort and then instruct to send the job to n processors? > > Thank you. > > -- One trouble is that despite this system, anyone who reads journals widely and critically is forced to realize that there are scarcely any bars to eventual publication. There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print. -- Drummond Rennie From zonexo at gmail.com Wed Jan 10 00:44:38 2007 From: zonexo at gmail.com (Ben Tay) Date: Wed, 10 Jan 2007 14:44:38 +0800 Subject: undefined reference to .... Message-ID: <804ab5d40701092244o6be0a8b4qeb72b95794d5fb00@mail.gmail.com> Hi, I have a very simple fortran code. It compiles on a 32bit system with mkl with no errors but on em64t, it gives "undefined reference to ...." error. It works when I compiled with the supplied blas/lapack. However if I use Intel mkl, it gives the error as stated above. My code is global.F module global_data implicit none save #include "include/finclude/petsc.h" #include "include/finclude/petscvec.h" #include "include/finclude/petscmat.h" #include "include/finclude/petscksp.h" #include "include/finclude/petscpc.h" #include "include/finclude/petscmat.h90" integer :: i,j,k Vec xx,b_rhs,xx_uv,b_rhs_uv ! /* solution vector, right hand side vector and work vector */ Mat A_mat,A_mat_uv ! /* sparse matrix */ end module global_data main.f90 program ns2d_c use global_data implicit none integer :: ierr i=1 call PetscInitialize(PETSC_NULL_CHARACTER,ierr) call MatCreateSeqAIJ(PETSC_COMM_SELF,9,9,9,PETSC_NULL_INTEGER,A_mat,ierr) end program ns2d_c The error msg is /tmp/ifort0JBYUf.o(.text+0x46): In function `ns2d_c': /nfs/home/enduser/g0306332/test/main.F:11: undefined reference to `petscinitialize_' /tmp/ifort0JBYUf.o(.text+0xaf):/nfs/home/enduser/g0306332/test/main.F:13: undefined reference to `matcreateseqaij_' The compiling commands, which I rephrase from the "make ex1f" are ifort -132 -fPIC -g -c -I/nfs/lsftmp/g0306332/petsc-2.3.2-p8-I/nfs/lsftmp/g0306332/petsc- 2.3.2-p8/bmake/l64-nompi-noshared -I/nfs/lsftmp/g0306332/petsc-2.3.2-p8/include -I/nfs/lsftmp/g0306332/petsc-2.3.2-p8/include/mpiuni global.F ifort -fPIC -g -Wl,-rpath,/nfs/lsftmp/g0306332/petsc-2.3.2-p8/lib/l64-nompi-noshared -L/nfs/lsftmp/g0306332/petsc-2.3.2-p8/lib/l64-nompi-noshared -lpetscksp -lpetscdm -lpetscmat -lpetscvec -lpetsc -Wl,-rpath,/nfs/lsftmp/g0306332/petsc-2.3.2-p8/lib/l64-nompi-noshared -L/nfs/lsftmp/g0306332/petsc-2.3.2-p8/lib/l64-nompi-noshared -lmpiuni -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/em64t -L/lsftmp/g0306332/inter/mkl/lib/em64t -lmkl_lapack -lmkl_em64t -lguide -lpthread -ldl -Wl,-rpath,/usr/local/intel/cce9.0/lib -L/usr/local/intel/cce9.0/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 -lsvml -limf -lirc -lgcc_s -lirc_s -Wl,-rpath,"/usr/local/intel/cce9.0/lib" -Wl,-rpath,"/usr/local/intel/cce9.0/lib" -L"/usr/local/intel/cce9.0/lib" -Wl,-rpath,"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/" -Wl,-rpath,"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/" -L"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/" -Wl,-rpath,"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64" -Wl,-rpath,"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64" -L"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64" -Wl,-rpath,/usr/local/intel/fc9.0/lib -L/usr/local/intel/fc9.0/lib -lifport -lifcore -lm -Wl,-rpath,/usr/local/intel/cce9.0/lib -Wl,-rpath,/usr/local/intel/cce9.0/lib -L/usr/local/intel/cce9.0/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 -lm -ldl -Wl,-rpath,/usr/local/intel/cce9.0/lib -L/usr/local/intel/cce9.0/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 -lsvml -limf -lirc -lgcc_s -lirc_s -ldl -o a.out global.o main.f90 I have used shared,static library. I wonder if it is a problem with mkl em64t or there's something wrong with my code/compilation. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Jan 10 13:41:22 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 10 Jan 2007 13:41:22 -0600 (CST) Subject: undefined reference to .... In-Reply-To: <804ab5d40701092244o6be0a8b4qeb72b95794d5fb00@mail.gmail.com> References: <804ab5d40701092244o6be0a8b4qeb72b95794d5fb00@mail.gmail.com> Message-ID: Do PETSc examples work? Send us the output from make test Staish On Wed, 10 Jan 2007, Ben Tay wrote: > Hi, > > I have a very simple fortran code. It compiles on a 32bit system with mkl > with no errors but on em64t, it gives "undefined reference to ...." error. > > It works when I compiled with the supplied blas/lapack. However if I use > Intel mkl, it gives the error as stated above. > > My code is > > global.F > > module global_data > > implicit none > > save > > #include "include/finclude/petsc.h" > #include "include/finclude/petscvec.h" > #include "include/finclude/petscmat.h" > #include "include/finclude/petscksp.h" > #include "include/finclude/petscpc.h" > #include "include/finclude/petscmat.h90" > > integer :: i,j,k > > Vec xx,b_rhs,xx_uv,b_rhs_uv ! /* solution vector, right hand > side vector and work vector */ > > Mat A_mat,A_mat_uv ! /* sparse matrix */ > > end module global_data > > main.f90 > > program ns2d_c > > use global_data > > implicit none > > integer :: ierr > > i=1 > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > call MatCreateSeqAIJ(PETSC_COMM_SELF,9,9,9,PETSC_NULL_INTEGER,A_mat,ierr) > > end program ns2d_c > > > The error msg is > > /tmp/ifort0JBYUf.o(.text+0x46): In function `ns2d_c': > /nfs/home/enduser/g0306332/test/main.F:11: undefined reference to > `petscinitialize_' > /tmp/ifort0JBYUf.o(.text+0xaf):/nfs/home/enduser/g0306332/test/main.F:13: > undefined reference to `matcreateseqaij_' > > The compiling commands, which I rephrase from the "make ex1f" are > > ifort -132 -fPIC -g -c > -I/nfs/lsftmp/g0306332/petsc-2.3.2-p8-I/nfs/lsftmp/g0306332/petsc- > 2.3.2-p8/bmake/l64-nompi-noshared > -I/nfs/lsftmp/g0306332/petsc-2.3.2-p8/include > -I/nfs/lsftmp/g0306332/petsc-2.3.2-p8/include/mpiuni global.F > > ifort -fPIC -g > -Wl,-rpath,/nfs/lsftmp/g0306332/petsc-2.3.2-p8/lib/l64-nompi-noshared > -L/nfs/lsftmp/g0306332/petsc-2.3.2-p8/lib/l64-nompi-noshared -lpetscksp > -lpetscdm -lpetscmat -lpetscvec -lpetsc > -Wl,-rpath,/nfs/lsftmp/g0306332/petsc-2.3.2-p8/lib/l64-nompi-noshared > -L/nfs/lsftmp/g0306332/petsc-2.3.2-p8/lib/l64-nompi-noshared -lmpiuni > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/em64t > -L/lsftmp/g0306332/inter/mkl/lib/em64t -lmkl_lapack -lmkl_em64t -lguide > -lpthread -ldl -Wl,-rpath,/usr/local/intel/cce9.0/lib > -L/usr/local/intel/cce9.0/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 -lsvml -limf > -lirc -lgcc_s -lirc_s -Wl,-rpath,"/usr/local/intel/cce9.0/lib" > -Wl,-rpath,"/usr/local/intel/cce9.0/lib" -L"/usr/local/intel/cce9.0/lib" > -Wl,-rpath,"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/" > -Wl,-rpath,"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/" > -L"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/" > -Wl,-rpath,"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64" > -Wl,-rpath,"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64" > -L"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64" > -Wl,-rpath,/usr/local/intel/fc9.0/lib -L/usr/local/intel/fc9.0/lib -lifport > -lifcore -lm -Wl,-rpath,/usr/local/intel/cce9.0/lib > -Wl,-rpath,/usr/local/intel/cce9.0/lib -L/usr/local/intel/cce9.0/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 -lm -ldl > -Wl,-rpath,/usr/local/intel/cce9.0/lib -L/usr/local/intel/cce9.0/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 -lsvml -limf > -lirc -lgcc_s -lirc_s -ldl -o a.out global.o main.f90 > > I have used shared,static library. I wonder if it is a problem with mkl > em64t or there's something wrong with my code/compilation. > > > > Thank you. > From dimitri.lecas at free.fr Wed Jan 10 17:18:06 2007 From: dimitri.lecas at free.fr (Dimitri Lecas) Date: Thu, 11 Jan 2007 00:18:06 +0100 Subject: Using parMetis in petsc for ordering In-Reply-To: References: <45A298A6.3020400@free.fr> Message-ID: <45A5742E.5080008@free.fr> Barry Smith a ?crit : > 1) The PETSc LU and Cholesky solvers only run sequentially. > 2) The parallel LU and Cholesky solvers PETSc interfaces to, SuperLU_dist, > MUMPS, Spooles, DSCPACK do NOT accept an external ordering provided for > them. > > Hence we do not have any setup for doing parallel matrix orderings for > factorizations, since we cannot use them. We could allow calling a parallel > ordering but I'm not sure what it would be useful for. > > Barry > > Ok i see that i was looking for a wrong direction. Just in ksp/examples/tutorials/ex10.c, Partitioning is used on the linear system matrix. I don't understand why ? What i understand it's, with MatPartitioning we try to partitioning the graph build from the matrix (vertices is the row/columns and edge between i and j if aij or aji is non zero value). But in my mind, a good partitioning for solving linear system with iterative algorithm is to load balance the non zero value between processors, so we have to use weight, number of non zero value in the row, to have a good partitioning. Do i have it right ? -- Dimitri Lecas From pbauman at ices.utexas.edu Wed Jan 10 17:24:28 2007 From: pbauman at ices.utexas.edu (Paul T. Bauman) Date: Wed, 10 Jan 2007 17:24:28 -0600 Subject: PETSc and gfortran Message-ID: <45A575AC.4020701@ices.utexas.edu> Hello, Has anyone had any experience using PETSc and gfortran together? My code compiles, but when I run it, it crashes with the following error (mac Tiger 10.4.8, power pc, latest build of gfortran, MPICH2, petsc 2.3.2): [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 10 BUS: Bus Error, possibly illegal memory access [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC ERROR: or try http://valgrind.org on linux or man libgmalloc on Apple to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 2.3.2, Patch 8, Tue Jan 2 14:33:59 PST 2007 HG revision: ebeddcedcc065e32fc252af32cf1d01ed4fc7a80 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: EXEC/shrink_tao on a Apple_Pow named dhcp-67-30.ices.utexas.edu by pbauman Wed Jan 10 17:20:03 2007 [0]PETSC ERROR: Libraries linked from /Users/pbauman/LIBRARIES/PETSC/petsc-2.3.2-p8/lib/Apple_PowerPC_Tiger_Debug [0]PETSC ERROR: Configure run at Tue Jan 9 15:42:42 2007 [0]PETSC ERROR: Configure options --with-shared=0 --with-clanguage=C++ --with-mpi-dir=/Users/pbauman/LIBRARIES/MPICH/mpich2-1.0.5_Apple_PowerPC_Tiger_gcc_gfortran --with-blas-lapack-dir=/Users/pbauman/LIBRARIES/PETSC/petsc-2.3.2-p8/externalpackages/fblaslapack/Apple_PowerPC_Tiger_Debug/ [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file [cli_0]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 rank 0 in job 2 dhcp-67-30.ices.utexas.edu_50482 caused collective abort of all ranks exit status of rank 0: return code 59 make: *** [run_shrink_tao_tr] Error 59 This happens at the very beginning of the code (I put a write statement in at the first line of the program and nothing came out). This code runs successfully on linux with intel compilers. I'm sure the problem is somehow with gfortran/my build, I'm just not sure where to begin with this kind of problem, so I thought I'd see if anyone had a similar experience. Thanks, Paul From balay at mcs.anl.gov Wed Jan 10 17:28:25 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 10 Jan 2007 17:28:25 -0600 (CST) Subject: PETSc and gfortran In-Reply-To: <45A575AC.4020701@ices.utexas.edu> References: <45A575AC.4020701@ices.utexas.edu> Message-ID: Can you reproduce this with a PETSc example? make test Satish On Wed, 10 Jan 2007, Paul T. Bauman wrote: > Hello, > > Has anyone had any experience using PETSc and gfortran together? My code > compiles, but when I run it, it crashes with the following error (mac Tiger > 10.4.8, power pc, latest build of gfortran, MPICH2, petsc 2.3.2): > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 10 BUS: Bus Error, possibly illegal > memory access > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC > ERROR: or try http://valgrind.org on linux or man libgmalloc on Apple to find > memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 2.3.2, Patch 8, Tue Jan 2 14:33:59 PST > 2007 HG revision: ebeddcedcc065e32fc252af32cf1d01ed4fc7a80 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: EXEC/shrink_tao on a Apple_Pow named > dhcp-67-30.ices.utexas.edu by pbauman Wed Jan 10 17:20:03 2007 > [0]PETSC ERROR: Libraries linked from > /Users/pbauman/LIBRARIES/PETSC/petsc-2.3.2-p8/lib/Apple_PowerPC_Tiger_Debug > [0]PETSC ERROR: Configure run at Tue Jan 9 15:42:42 2007 > [0]PETSC ERROR: Configure options --with-shared=0 --with-clanguage=C++ > --with-mpi-dir=/Users/pbauman/LIBRARIES/MPICH/mpich2-1.0.5_Apple_PowerPC_Tiger_gcc_gfortran > --with-blas-lapack-dir=/Users/pbauman/LIBRARIES/PETSC/petsc-2.3.2-p8/externalpackages/fblaslapack/Apple_PowerPC_Tiger_Debug/ > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown > file > [cli_0]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > rank 0 in job 2 dhcp-67-30.ices.utexas.edu_50482 caused collective abort of > all ranks > exit status of rank 0: return code 59 > make: *** [run_shrink_tao_tr] Error 59 > > This happens at the very beginning of the code (I put a write statement in at > the first line of the program and nothing came out). This code runs > successfully on linux with intel compilers. I'm sure the problem is somehow > with gfortran/my build, I'm just not sure where to begin with this kind of > problem, so I thought I'd see if anyone had a similar experience. > > Thanks, > > Paul > > From pbauman at ices.utexas.edu Wed Jan 10 17:37:44 2007 From: pbauman at ices.utexas.edu (Paul T. Bauman) Date: Wed, 10 Jan 2007 17:37:44 -0600 Subject: PETSc and gfortran In-Reply-To: References: <45A575AC.4020701@ices.utexas.edu> Message-ID: <45A578C8.8050201@ices.utexas.edu> Yes, example ex5f broke in the exact same way. Note the preceding c/c++ examples worked perfectly. How would you suggest I proceed? Thanks, Paul Satish Balay wrote: > Can you reproduce this with a PETSc example? > > make test > > Satish > > On Wed, 10 Jan 2007, Paul T. Bauman wrote: > > >> Hello, >> >> Has anyone had any experience using PETSc and gfortran together? My code >> compiles, but when I run it, it crashes with the following error (mac Tiger >> 10.4.8, power pc, latest build of gfortran, MPICH2, petsc 2.3.2): >> >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Caught signal number 10 BUS: Bus Error, possibly illegal >> memory access >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC >> ERROR: or try http://valgrind.org on linux or man libgmalloc on Apple to find >> memory corruption errors >> [0]PETSC ERROR: likely location of problem given in stack below >> [0]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> [0]PETSC ERROR: INSTEAD the line number of the start of the function >> [0]PETSC ERROR: is given. >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Signal received! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 2.3.2, Patch 8, Tue Jan 2 14:33:59 PST >> 2007 HG revision: ebeddcedcc065e32fc252af32cf1d01ed4fc7a80 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: EXEC/shrink_tao on a Apple_Pow named >> dhcp-67-30.ices.utexas.edu by pbauman Wed Jan 10 17:20:03 2007 >> [0]PETSC ERROR: Libraries linked from >> /Users/pbauman/LIBRARIES/PETSC/petsc-2.3.2-p8/lib/Apple_PowerPC_Tiger_Debug >> [0]PETSC ERROR: Configure run at Tue Jan 9 15:42:42 2007 >> [0]PETSC ERROR: Configure options --with-shared=0 --with-clanguage=C++ >> --with-mpi-dir=/Users/pbauman/LIBRARIES/MPICH/mpich2-1.0.5_Apple_PowerPC_Tiger_gcc_gfortran >> --with-blas-lapack-dir=/Users/pbauman/LIBRARIES/PETSC/petsc-2.3.2-p8/externalpackages/fblaslapack/Apple_PowerPC_Tiger_Debug/ >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown >> file >> [cli_0]: aborting job: >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >> rank 0 in job 2 dhcp-67-30.ices.utexas.edu_50482 caused collective abort of >> all ranks >> exit status of rank 0: return code 59 >> make: *** [run_shrink_tao_tr] Error 59 >> >> This happens at the very beginning of the code (I put a write statement in at >> the first line of the program and nothing came out). This code runs >> successfully on linux with intel compilers. I'm sure the problem is somehow >> with gfortran/my build, I'm just not sure where to begin with this kind of >> problem, so I thought I'd see if anyone had a similar experience. >> >> Thanks, >> >> Paul >> >> >> From balay at mcs.anl.gov Wed Jan 10 17:46:10 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 10 Jan 2007 17:46:10 -0600 (CST) Subject: PETSc and gfortran In-Reply-To: <45A578C8.8050201@ices.utexas.edu> References: <45A575AC.4020701@ices.utexas.edu> <45A578C8.8050201@ices.utexas.edu> Message-ID: BUS error is a bit suspicious. What can you send the output from all of the following commands.. cd src/ksp/ksp/examples/tutorials make ex2f ./ex2f gdb ex2f (gdb) run Satish On Wed, 10 Jan 2007, Paul T. Bauman wrote: > Yes, example ex5f broke in the exact same way. Note the preceding c/c++ > examples worked perfectly. How would you suggest I proceed? Thanks, > > Paul > > Satish Balay wrote: > > Can you reproduce this with a PETSc example? > > > > make test > > > > Satish > > > > On Wed, 10 Jan 2007, Paul T. Bauman wrote: > > > > > > > Hello, > > > > > > Has anyone had any experience using PETSc and gfortran together? My code > > > compiles, but when I run it, it crashes with the following error (mac > > > Tiger > > > 10.4.8, power pc, latest build of gfortran, MPICH2, petsc 2.3.2): > > > > > > [0]PETSC ERROR: > > > ------------------------------------------------------------------------ > > > [0]PETSC ERROR: Caught signal number 10 BUS: Bus Error, possibly illegal > > > memory access > > > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > > [0]PETSC ERROR: or see > > > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC > > > ERROR: or try http://valgrind.org on linux or man libgmalloc on Apple to > > > find > > > memory corruption errors > > > [0]PETSC ERROR: likely location of problem given in stack below > > > [0]PETSC ERROR: --------------------- Stack Frames > > > ------------------------------------ > > > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > > > available, > > > [0]PETSC ERROR: INSTEAD the line number of the start of the function > > > [0]PETSC ERROR: is given. > > > [0]PETSC ERROR: --------------------- Error Message > > > ------------------------------------ > > > [0]PETSC ERROR: Signal received! > > > [0]PETSC ERROR: > > > ------------------------------------------------------------------------ > > > [0]PETSC ERROR: Petsc Release Version 2.3.2, Patch 8, Tue Jan 2 14:33:59 > > > PST > > > 2007 HG revision: ebeddcedcc065e32fc252af32cf1d01ed4fc7a80 > > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > > [0]PETSC ERROR: See docs/index.html for manual pages. > > > [0]PETSC ERROR: > > > ------------------------------------------------------------------------ > > > [0]PETSC ERROR: EXEC/shrink_tao on a Apple_Pow named > > > dhcp-67-30.ices.utexas.edu by pbauman Wed Jan 10 17:20:03 2007 > > > [0]PETSC ERROR: Libraries linked from > > > /Users/pbauman/LIBRARIES/PETSC/petsc-2.3.2-p8/lib/Apple_PowerPC_Tiger_Debug > > > [0]PETSC ERROR: Configure run at Tue Jan 9 15:42:42 2007 > > > [0]PETSC ERROR: Configure options --with-shared=0 --with-clanguage=C++ > > > > > > --with-mpi-dir=/Users/pbauman/LIBRARIES/MPICH/mpich2-1.0.5_Apple_PowerPC_Tiger_gcc_gfortran > > > > > > --with-blas-lapack-dir=/Users/pbauman/LIBRARIES/PETSC/petsc-2.3.2-p8/externalpackages/fblaslapack/Apple_PowerPC_Tiger_Debug/ > > > [0]PETSC ERROR: > > > ------------------------------------------------------------------------ > > > [0]PETSC ERROR: User provided function() line 0 in unknown directory > > > unknown > > > file > > > [cli_0]: aborting job: > > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > > rank 0 in job 2 dhcp-67-30.ices.utexas.edu_50482 caused collective > > > abort of > > > all ranks > > > exit status of rank 0: return code 59 > > > make: *** [run_shrink_tao_tr] Error 59 > > > > > > This happens at the very beginning of the code (I put a write statement in > > > at > > > the first line of the program and nothing came out). This code runs > > > successfully on linux with intel compilers. I'm sure the problem is > > > somehow > > > with gfortran/my build, I'm just not sure where to begin with this kind of > > > problem, so I thought I'd see if anyone had a similar experience. > > > > > > Thanks, > > > > > > Paul > > > > > > > > > > > From bsmith at mcs.anl.gov Wed Jan 10 18:00:49 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 10 Jan 2007 18:00:49 -0600 (CST) Subject: Using parMetis in petsc for ordering In-Reply-To: <45A5742E.5080008@free.fr> References: <45A298A6.3020400@free.fr> <45A5742E.5080008@free.fr> Message-ID: Dimitri, No, I think this is not the correct way to look at things. Load balancing the original matrix is not neccessarily a good thing for doing an LU factorization (in fact it is likely just to make the LU factorization have much more fill and require much more floating operations). Packages like SuperLU_dist and Mumps have their own internal ordering routines that are specifically for getting a good ordering for doing the parallel LU factorization, you should just have these solvers use them (which they do automatically). Barry On Thu, 11 Jan 2007, Dimitri Lecas wrote: > Barry Smith a ?crit : > > 1) The PETSc LU and Cholesky solvers only run sequentially. > > 2) The parallel LU and Cholesky solvers PETSc interfaces to, SuperLU_dist, > > MUMPS, Spooles, DSCPACK do NOT accept an external ordering provided for > > them. > > Hence we do not have any setup for doing parallel matrix orderings for > > factorizations, since we cannot use them. We could allow calling a parallel > > ordering but I'm not sure what it would be useful for. > > > > Barry > > > > > Ok i see that i was looking for a wrong direction. > > Just in ksp/examples/tutorials/ex10.c, Partitioning is used on the linear > system matrix. I don't understand why ? > > What i understand it's, with MatPartitioning we try to partitioning the graph > build from the matrix (vertices is the row/columns and edge between i and j > if aij or aji is non zero value). But in my mind, a good partitioning for > solving linear system with iterative algorithm is to load balance the non zero > value between processors, so we have to use weight, number of non zero value > in the row, to have a good partitioning. > Do i have it right ? > > From zonexo at gmail.com Wed Jan 10 18:42:11 2007 From: zonexo at gmail.com (Ben Tay) Date: Thu, 11 Jan 2007 08:42:11 +0800 Subject: undefined reference to .... In-Reply-To: References: <804ab5d40701092244o6be0a8b4qeb72b95794d5fb00@mail.gmail.com> Message-ID: <804ab5d40701101642h36ad001gd9a6ededa0c68cab@mail.gmail.com> Yes it ran successfully. I've attached the output. thank you very much. On 1/11/07, Satish Balay wrote: > > Do PETSc examples work? > > Send us the output from > > make test > > Staish > > On Wed, 10 Jan 2007, Ben Tay wrote: > > > Hi, > > > > I have a very simple fortran code. It compiles on a 32bit system with > mkl > > with no errors but on em64t, it gives "undefined reference to ...." > error. > > > > It works when I compiled with the supplied blas/lapack. However if I use > > Intel mkl, it gives the error as stated above. > > > > My code is > > > > global.F > > > > module global_data > > > > implicit none > > > > save > > > > #include "include/finclude/petsc.h" > > #include "include/finclude/petscvec.h" > > #include "include/finclude/petscmat.h" > > #include "include/finclude/petscksp.h" > > #include "include/finclude/petscpc.h" > > #include "include/finclude/petscmat.h90" > > > > integer :: i,j,k > > > > Vec xx,b_rhs,xx_uv,b_rhs_uv ! /* solution vector, right > hand > > side vector and work vector */ > > > > Mat A_mat,A_mat_uv ! /* sparse matrix */ > > > > end module global_data > > > > main.f90 > > > > program ns2d_c > > > > use global_data > > > > implicit none > > > > integer :: ierr > > > > i=1 > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > > > call > MatCreateSeqAIJ(PETSC_COMM_SELF,9,9,9,PETSC_NULL_INTEGER,A_mat,ierr) > > > > end program ns2d_c > > > > > > The error msg is > > > > /tmp/ifort0JBYUf.o(.text+0x46): In function `ns2d_c': > > /nfs/home/enduser/g0306332/test/main.F:11: undefined reference to > > `petscinitialize_' > > > /tmp/ifort0JBYUf.o(.text+0xaf):/nfs/home/enduser/g0306332/test/main.F:13: > > undefined reference to `matcreateseqaij_' > > > > The compiling commands, which I rephrase from the "make ex1f" are > > > > ifort -132 -fPIC -g -c > > -I/nfs/lsftmp/g0306332/petsc-2.3.2-p8-I/nfs/lsftmp/g0306332/petsc- > > 2.3.2-p8/bmake/l64-nompi-noshared > > -I/nfs/lsftmp/g0306332/petsc-2.3.2-p8/include > > -I/nfs/lsftmp/g0306332/petsc-2.3.2-p8/include/mpiuni global.F > > > > ifort -fPIC -g > > -Wl,-rpath,/nfs/lsftmp/g0306332/petsc-2.3.2-p8/lib/l64-nompi-noshared > > -L/nfs/lsftmp/g0306332/petsc-2.3.2-p8/lib/l64-nompi-noshared -lpetscksp > > -lpetscdm -lpetscmat -lpetscvec -lpetsc > > -Wl,-rpath,/nfs/lsftmp/g0306332/petsc-2.3.2-p8/lib/l64-nompi-noshared > > -L/nfs/lsftmp/g0306332/petsc-2.3.2-p8/lib/l64-nompi-noshared -lmpiuni > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/em64t > > -L/lsftmp/g0306332/inter/mkl/lib/em64t -lmkl_lapack -lmkl_em64t -lguide > > -lpthread -ldl -Wl,-rpath,/usr/local/intel/cce9.0/lib > > -L/usr/local/intel/cce9.0/lib > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 > > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 -lsvml -limf > > -lirc -lgcc_s -lirc_s -Wl,-rpath,"/usr/local/intel/cce9.0/lib" > > -Wl,-rpath,"/usr/local/intel/cce9.0/lib" -L"/usr/local/intel/cce9.0/lib" > > -Wl,-rpath,"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/" > > -Wl,-rpath,"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/" > > -L"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/" > > -Wl,-rpath,"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64" > > -Wl,-rpath,"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64" > > -L"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64" > > -Wl,-rpath,/usr/local/intel/fc9.0/lib -L/usr/local/intel/fc9.0/lib > -lifport > > -lifcore -lm -Wl,-rpath,/usr/local/intel/cce9.0/lib > > -Wl,-rpath,/usr/local/intel/cce9.0/lib -L/usr/local/intel/cce9.0/lib > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 > > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 -lm -ldl > > -Wl,-rpath,/usr/local/intel/cce9.0/lib -L/usr/local/intel/cce9.0/lib > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 > > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 -lsvml -limf > > -lirc -lgcc_s -lirc_s -ldl -o a.out global.o main.f90 > > > > I have used shared,static library. I wonder if it is a problem with mkl > > em64t or there's something wrong with my code/compilation. > > > > > > > > Thank you. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: ex5f_1.out URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: ex5f_1.testout URL: From pbauman at ices.utexas.edu Wed Jan 10 18:46:00 2007 From: pbauman at ices.utexas.edu (Paul T. Bauman) Date: Wed, 10 Jan 2007 18:46:00 -0600 Subject: PETSc and gfortran In-Reply-To: References: <45A575AC.4020701@ices.utexas.edu> <45A578C8.8050201@ices.utexas.edu> Message-ID: <45A588C8.4080704@ices.utexas.edu> I'm such an idiot... PETSc was compiled with an older version of gfortran than what I compiled my code with. gdb illuminated this for me (it was looking for the old gfortran libraries that weren't there). Sorry to waste your time. Everything is working now. Thanks again for your prompt help. Paul Satish Balay wrote: > BUS error is a bit suspicious. What can you send the output from all > of the following commands.. > > cd src/ksp/ksp/examples/tutorials > make ex2f > ./ex2f > gdb ex2f > (gdb) run > > > Satish > > On Wed, 10 Jan 2007, Paul T. Bauman wrote: > > >> Yes, example ex5f broke in the exact same way. Note the preceding c/c++ >> examples worked perfectly. How would you suggest I proceed? Thanks, >> >> Paul >> >> Satish Balay wrote: >> >>> Can you reproduce this with a PETSc example? >>> >>> make test >>> >>> Satish >>> >>> On Wed, 10 Jan 2007, Paul T. Bauman wrote: >>> >>> >>> >>>> Hello, >>>> >>>> Has anyone had any experience using PETSc and gfortran together? My code >>>> compiles, but when I run it, it crashes with the following error (mac >>>> Tiger >>>> 10.4.8, power pc, latest build of gfortran, MPICH2, petsc 2.3.2): >>>> >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Caught signal number 10 BUS: Bus Error, possibly illegal >>>> memory access >>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>> [0]PETSC ERROR: or see >>>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC >>>> ERROR: or try http://valgrind.org on linux or man libgmalloc on Apple to >>>> find >>>> memory corruption errors >>>> [0]PETSC ERROR: likely location of problem given in stack below >>>> [0]PETSC ERROR: --------------------- Stack Frames >>>> ------------------------------------ >>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>> available, >>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>> [0]PETSC ERROR: is given. >>>> [0]PETSC ERROR: --------------------- Error Message >>>> ------------------------------------ >>>> [0]PETSC ERROR: Signal received! >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Petsc Release Version 2.3.2, Patch 8, Tue Jan 2 14:33:59 >>>> PST >>>> 2007 HG revision: ebeddcedcc065e32fc252af32cf1d01ed4fc7a80 >>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: EXEC/shrink_tao on a Apple_Pow named >>>> dhcp-67-30.ices.utexas.edu by pbauman Wed Jan 10 17:20:03 2007 >>>> [0]PETSC ERROR: Libraries linked from >>>> /Users/pbauman/LIBRARIES/PETSC/petsc-2.3.2-p8/lib/Apple_PowerPC_Tiger_Debug >>>> [0]PETSC ERROR: Configure run at Tue Jan 9 15:42:42 2007 >>>> [0]PETSC ERROR: Configure options --with-shared=0 --with-clanguage=C++ >>>> >>>> --with-mpi-dir=/Users/pbauman/LIBRARIES/MPICH/mpich2-1.0.5_Apple_PowerPC_Tiger_gcc_gfortran >>>> >>>> --with-blas-lapack-dir=/Users/pbauman/LIBRARIES/PETSC/petsc-2.3.2-p8/externalpackages/fblaslapack/Apple_PowerPC_Tiger_Debug/ >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: User provided function() line 0 in unknown directory >>>> unknown >>>> file >>>> [cli_0]: aborting job: >>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >>>> rank 0 in job 2 dhcp-67-30.ices.utexas.edu_50482 caused collective >>>> abort of >>>> all ranks >>>> exit status of rank 0: return code 59 >>>> make: *** [run_shrink_tao_tr] Error 59 >>>> >>>> This happens at the very beginning of the code (I put a write statement in >>>> at >>>> the first line of the program and nothing came out). This code runs >>>> successfully on linux with intel compilers. I'm sure the problem is >>>> somehow >>>> with gfortran/my build, I'm just not sure where to begin with this kind of >>>> problem, so I thought I'd see if anyone had a similar experience. >>>> >>>> Thanks, >>>> >>>> Paul >>>> >>>> >>>> >>>> >> From balay at mcs.anl.gov Wed Jan 10 19:32:08 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 10 Jan 2007 19:32:08 -0600 (CST) Subject: undefined reference to .... In-Reply-To: <804ab5d40701101642h36ad001gd9a6ededa0c68cab@mail.gmail.com> References: <804ab5d40701092244o6be0a8b4qeb72b95794d5fb00@mail.gmail.com> <804ab5d40701101642h36ad001gd9a6ededa0c68cab@mail.gmail.com> Message-ID: Ah - I wasn't checking your make output closly.. Looks like you are listing your libs after petsclibs. You'll have to list the object files - in the correct order of dependencies.. > > > -lirc -lgcc_s -lirc_s -ldl -o a.out global.o main.f90 Also - looks like you are compiling & linking at the same step.. [i.e using .o and .f90 in the same command]. Sugest crearing .o files before linking.. i.e ${FLINKER} -o a.out global.o main.o ${PETSC_LIB} Satish On Thu, 11 Jan 2007, Ben Tay wrote: > Yes it ran successfully. I've attached the output. > > thank you very much. > > > On 1/11/07, Satish Balay wrote: > > > > Do PETSc examples work? > > > > Send us the output from > > > > make test > > > > Staish > > > > On Wed, 10 Jan 2007, Ben Tay wrote: > > > > > Hi, > > > > > > I have a very simple fortran code. It compiles on a 32bit system with > > mkl > > > with no errors but on em64t, it gives "undefined reference to ...." > > error. > > > > > > It works when I compiled with the supplied blas/lapack. However if I use > > > Intel mkl, it gives the error as stated above. > > > > > > My code is > > > > > > global.F > > > > > > module global_data > > > > > > implicit none > > > > > > save > > > > > > #include "include/finclude/petsc.h" > > > #include "include/finclude/petscvec.h" > > > #include "include/finclude/petscmat.h" > > > #include "include/finclude/petscksp.h" > > > #include "include/finclude/petscpc.h" > > > #include "include/finclude/petscmat.h90" > > > > > > integer :: i,j,k > > > > > > Vec xx,b_rhs,xx_uv,b_rhs_uv ! /* solution vector, right > > hand > > > side vector and work vector */ > > > > > > Mat A_mat,A_mat_uv ! /* sparse matrix */ > > > > > > end module global_data > > > > > > main.f90 > > > > > > program ns2d_c > > > > > > use global_data > > > > > > implicit none > > > > > > integer :: ierr > > > > > > i=1 > > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > > > > > call > > MatCreateSeqAIJ(PETSC_COMM_SELF,9,9,9,PETSC_NULL_INTEGER,A_mat,ierr) > > > > > > end program ns2d_c > > > > > > > > > The error msg is > > > > > > /tmp/ifort0JBYUf.o(.text+0x46): In function `ns2d_c': > > > /nfs/home/enduser/g0306332/test/main.F:11: undefined reference to > > > `petscinitialize_' > > > > > /tmp/ifort0JBYUf.o(.text+0xaf):/nfs/home/enduser/g0306332/test/main.F:13: > > > undefined reference to `matcreateseqaij_' > > > > > > The compiling commands, which I rephrase from the "make ex1f" are > > > > > > ifort -132 -fPIC -g -c > > > -I/nfs/lsftmp/g0306332/petsc-2.3.2-p8-I/nfs/lsftmp/g0306332/petsc- > > > 2.3.2-p8/bmake/l64-nompi-noshared > > > -I/nfs/lsftmp/g0306332/petsc-2.3.2-p8/include > > > -I/nfs/lsftmp/g0306332/petsc-2.3.2-p8/include/mpiuni global.F > > > > > > ifort -fPIC -g > > > -Wl,-rpath,/nfs/lsftmp/g0306332/petsc-2.3.2-p8/lib/l64-nompi-noshared > > > -L/nfs/lsftmp/g0306332/petsc-2.3.2-p8/lib/l64-nompi-noshared -lpetscksp > > > -lpetscdm -lpetscmat -lpetscvec -lpetsc > > > -Wl,-rpath,/nfs/lsftmp/g0306332/petsc-2.3.2-p8/lib/l64-nompi-noshared > > > -L/nfs/lsftmp/g0306332/petsc-2.3.2-p8/lib/l64-nompi-noshared -lmpiuni > > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/em64t > > > -L/lsftmp/g0306332/inter/mkl/lib/em64t -lmkl_lapack -lmkl_em64t -lguide > > > -lpthread -ldl -Wl,-rpath,/usr/local/intel/cce9.0/lib > > > -L/usr/local/intel/cce9.0/lib > > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 > > > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 -lsvml -limf > > > -lirc -lgcc_s -lirc_s -Wl,-rpath,"/usr/local/intel/cce9.0/lib" > > > -Wl,-rpath,"/usr/local/intel/cce9.0/lib" -L"/usr/local/intel/cce9.0/lib" > > > -Wl,-rpath,"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/" > > > -Wl,-rpath,"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/" > > > -L"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/" > > > -Wl,-rpath,"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64" > > > -Wl,-rpath,"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64" > > > -L"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64" > > > -Wl,-rpath,/usr/local/intel/fc9.0/lib -L/usr/local/intel/fc9.0/lib > > -lifport > > > -lifcore -lm -Wl,-rpath,/usr/local/intel/cce9.0/lib > > > -Wl,-rpath,/usr/local/intel/cce9.0/lib -L/usr/local/intel/cce9.0/lib > > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 > > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 > > > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 -lm -ldl > > > -Wl,-rpath,/usr/local/intel/cce9.0/lib -L/usr/local/intel/cce9.0/lib > > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 > > > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 -lsvml -limf > > > -lirc -lgcc_s -lirc_s -ldl -o a.out global.o main.f90 > > > > > > I have used shared,static library. I wonder if it is a problem with mkl > > > em64t or there's something wrong with my code/compilation. > > > > > > > > > > > > Thank you. > > > > > > > > From zonexo at gmail.com Wed Jan 10 21:44:25 2007 From: zonexo at gmail.com (Ben Tay) Date: Thu, 11 Jan 2007 11:44:25 +0800 Subject: undefined reference to .... In-Reply-To: References: <804ab5d40701092244o6be0a8b4qeb72b95794d5fb00@mail.gmail.com> <804ab5d40701101642h36ad001gd9a6ededa0c68cab@mail.gmail.com> Message-ID: <804ab5d40701101944i5a10be4cye8259ab606976a0f@mail.gmail.com> Hi, regarding "> > > -lirc -lgcc_s -lirc_s -ldl -o a.out global.o main.f90" is it the correct order? my make file is in that order. anyway, compiling 1st then linking solve the problem. Strange that it didn't work compiling/linking at the same step. it worked for other configuration. thanks anyway. On 1/11/07, Satish Balay wrote: > > Ah - I wasn't checking your make output closly.. > > Looks like you are listing your libs after petsclibs. You'll have to > list the object files - in the correct order of dependencies.. > > > > > -lirc -lgcc_s -lirc_s -ldl -o a.out global.o main.f90 > > Also - looks like you are compiling & linking at the same step.. [i.e > using .o and .f90 in the same command]. Sugest crearing .o files before > linking.. > > i.e > > ${FLINKER} -o a.out global.o main.o ${PETSC_LIB} > > Satish > > > On Thu, 11 Jan 2007, Ben Tay wrote: > > > Yes it ran successfully. I've attached the output. > > > > thank you very much. > > > > > > On 1/11/07, Satish Balay wrote: > > > > > > Do PETSc examples work? > > > > > > Send us the output from > > > > > > make test > > > > > > Staish > > > > > > On Wed, 10 Jan 2007, Ben Tay wrote: > > > > > > > Hi, > > > > > > > > I have a very simple fortran code. It compiles on a 32bit system > with > > > mkl > > > > with no errors but on em64t, it gives "undefined reference to ...." > > > error. > > > > > > > > It works when I compiled with the supplied blas/lapack. However if I > use > > > > Intel mkl, it gives the error as stated above. > > > > > > > > My code is > > > > > > > > global.F > > > > > > > > module global_data > > > > > > > > implicit none > > > > > > > > save > > > > > > > > #include "include/finclude/petsc.h" > > > > #include "include/finclude/petscvec.h" > > > > #include "include/finclude/petscmat.h" > > > > #include "include/finclude/petscksp.h" > > > > #include "include/finclude/petscpc.h" > > > > #include "include/finclude/petscmat.h90" > > > > > > > > integer :: i,j,k > > > > > > > > Vec xx,b_rhs,xx_uv,b_rhs_uv ! /* solution vector, > right > > > hand > > > > side vector and work vector */ > > > > > > > > Mat A_mat,A_mat_uv ! /* sparse matrix */ > > > > > > > > end module global_data > > > > > > > > main.f90 > > > > > > > > program ns2d_c > > > > > > > > use global_data > > > > > > > > implicit none > > > > > > > > integer :: ierr > > > > > > > > i=1 > > > > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > > > > > > > call > > > MatCreateSeqAIJ(PETSC_COMM_SELF,9,9,9,PETSC_NULL_INTEGER,A_mat,ierr) > > > > > > > > end program ns2d_c > > > > > > > > > > > > The error msg is > > > > > > > > /tmp/ifort0JBYUf.o(.text+0x46): In function `ns2d_c': > > > > /nfs/home/enduser/g0306332/test/main.F:11: undefined reference to > > > > `petscinitialize_' > > > > > > > > /tmp/ifort0JBYUf.o(.text+0xaf):/nfs/home/enduser/g0306332/test/main.F:13: > > > > undefined reference to `matcreateseqaij_' > > > > > > > > The compiling commands, which I rephrase from the "make ex1f" are > > > > > > > > ifort -132 -fPIC -g -c > > > > -I/nfs/lsftmp/g0306332/petsc-2.3.2-p8-I/nfs/lsftmp/g0306332/petsc- > > > > 2.3.2-p8/bmake/l64-nompi-noshared > > > > -I/nfs/lsftmp/g0306332/petsc-2.3.2-p8/include > > > > -I/nfs/lsftmp/g0306332/petsc-2.3.2-p8/include/mpiuni global.F > > > > > > > > ifort -fPIC -g > > > > -Wl,-rpath,/nfs/lsftmp/g0306332/petsc-2.3.2-p8 > /lib/l64-nompi-noshared > > > > -L/nfs/lsftmp/g0306332/petsc-2.3.2-p8/lib/l64-nompi-noshared > -lpetscksp > > > > -lpetscdm -lpetscmat -lpetscvec -lpetsc > > > > -Wl,-rpath,/nfs/lsftmp/g0306332/petsc-2.3.2-p8 > /lib/l64-nompi-noshared > > > > -L/nfs/lsftmp/g0306332/petsc-2.3.2-p8/lib/l64-nompi-noshared > -lmpiuni > > > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/em64t > > > > -L/lsftmp/g0306332/inter/mkl/lib/em64t -lmkl_lapack -lmkl_em64t > -lguide > > > > -lpthread -ldl -Wl,-rpath,/usr/local/intel/cce9.0/lib > > > > -L/usr/local/intel/cce9.0/lib > > > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > > > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 > > > > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 -lsvml > -limf > > > > -lirc -lgcc_s -lirc_s -Wl,-rpath,"/usr/local/intel/cce9.0/lib" > > > > -Wl,-rpath,"/usr/local/intel/cce9.0/lib" > -L"/usr/local/intel/cce9.0/lib" > > > > -Wl,-rpath,"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/" > > > > -Wl,-rpath,"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/" > > > > -L"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/" > > > > > -Wl,-rpath,"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64" > > > > > -Wl,-rpath,"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64" > > > > -L"/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64" > > > > -Wl,-rpath,/usr/local/intel/fc9.0/lib -L/usr/local/intel/fc9.0/lib > > > -lifport > > > > -lifcore -lm -Wl,-rpath,/usr/local/intel/cce9.0/lib > > > > -Wl,-rpath,/usr/local/intel/cce9.0/lib -L/usr/local/intel/cce9.0/lib > > > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > > > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 > > > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 > > > > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 -lm -ldl > > > > -Wl,-rpath,/usr/local/intel/cce9.0/lib -L/usr/local/intel/cce9.0/lib > > > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > > > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/ > > > > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 > > > > -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 -lsvml > -limf > > > > -lirc -lgcc_s -lirc_s -ldl -o a.out global.o main.f90 > > > > > > > > I have used shared,static library. I wonder if it is a problem with > mkl > > > > em64t or there's something wrong with my code/compilation. > > > > > > > > > > > > > > > > Thank you. > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Jan 10 21:57:40 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 10 Jan 2007 21:57:40 -0600 (CST) Subject: undefined reference to .... In-Reply-To: <804ab5d40701101944i5a10be4cye8259ab606976a0f@mail.gmail.com> References: <804ab5d40701092244o6be0a8b4qeb72b95794d5fb00@mail.gmail.com> <804ab5d40701101642h36ad001gd9a6ededa0c68cab@mail.gmail.com> <804ab5d40701101944i5a10be4cye8259ab606976a0f@mail.gmail.com> Message-ID: On Thu, 11 Jan 2007, Ben Tay wrote: > Hi, > > regarding "> > > -lirc -lgcc_s -lirc_s -ldl -o a.out global.o main.f90" > > is it the correct order? my make file is in that order. No its the wrong order. The correct order was indicated in the previous e-mail. > > ${FLINKER} -o a.out global.o main.o ${PETSC_LIB} [note: -lirc -lgcc_s -lirc_s -ldl etc are the tail end of the PETSC_LIB listing] To check for the correct order - do: cd src/ksp/ksp/examples/tutorials make ex2f Satish From dimitri.lecas at free.fr Thu Jan 11 13:42:39 2007 From: dimitri.lecas at free.fr (Dimitri Lecas) Date: Thu, 11 Jan 2007 20:42:39 +0100 Subject: Using parMetis in petsc for ordering In-Reply-To: References: <45A298A6.3020400@free.fr> <45A5742E.5080008@free.fr> Message-ID: <45A6932F.2010009@free.fr> Barry Smith a ?crit : > Dimitri, > > No, I think this is not the correct way to look at things. Load > balancing the original matrix is not neccessarily a good thing for > doing an LU factorization (in fact it is likely just to make the LU > factorization have much more fill and require much more floating > operations). > > Packages like SuperLU_dist and Mumps have their own internal ordering > routines that are specifically for getting a good ordering for doing > the parallel LU factorization, you should just have these solvers > use them (which they do automatically). > > Barry > > > On Thu, 11 Jan 2007, Dimitri Lecas wrote: > > I'am no longer talking about doing LU factorization. But use iterative method for solving a linear system, like bicg. Like in the ex10. In this example i don't understand why using MatPartitioning. -- Dimitri Lecas From bsmith at mcs.anl.gov Thu Jan 11 13:54:36 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 11 Jan 2007 14:54:36 -0500 (EST) Subject: Using parMetis in petsc for ordering In-Reply-To: <45A6932F.2010009@free.fr> References: <45A298A6.3020400@free.fr> <45A5742E.5080008@free.fr> <45A6932F.2010009@free.fr> Message-ID: On Thu, 11 Jan 2007, Dimitri Lecas wrote: > Barry Smith a ?crit : > > Dimitri, > > > > No, I think this is not the correct way to look at things. Load > > balancing the original matrix is not neccessarily a good thing for > > doing an LU factorization (in fact it is likely just to make the LU > > factorization have much more fill and require much more floating > > operations). > > Packages like SuperLU_dist and Mumps have their own internal ordering > > routines that are specifically for getting a good ordering for doing > > the parallel LU factorization, you should just have these solvers > > use them (which they do automatically). > > > > Barry > > > > > > On Thu, 11 Jan 2007, Dimitri Lecas wrote: > > > > > I'am no longer talking about doing LU factorization. But use iterative method > for solving a linear system, like bicg. Like in the ex10. In this example i > don't understand why using MatPartitioning. Please rephrase the question. Are you asking why one should do the partition or why one should not? Are you asking in the case where the matrix is read from disk or generated in a parallel program? > > From dimitri.lecas at free.fr Thu Jan 11 15:00:11 2007 From: dimitri.lecas at free.fr (Dimitri Lecas) Date: Thu, 11 Jan 2007 22:00:11 +0100 Subject: Using parMetis in petsc for ordering In-Reply-To: References: <45A298A6.3020400@free.fr> <45A5742E.5080008@free.fr> <45A6932F.2010009@free.fr> Message-ID: <45A6A55B.8050108@free.fr> Barry Smith a ?crit : > > > On Thu, 11 Jan 2007, Dimitri Lecas wrote: > > >> Barry Smith a ?crit : >> >>> Dimitri, >>> >>> No, I think this is not the correct way to look at things. Load >>> balancing the original matrix is not neccessarily a good thing for >>> doing an LU factorization (in fact it is likely just to make the LU >>> factorization have much more fill and require much more floating >>> operations). >>> Packages like SuperLU_dist and Mumps have their own internal ordering >>> routines that are specifically for getting a good ordering for doing >>> the parallel LU factorization, you should just have these solvers >>> use them (which they do automatically). >>> >>> Barry >>> >>> >>> On Thu, 11 Jan 2007, Dimitri Lecas wrote: >>> >>> >>> >> I'am no longer talking about doing LU factorization. But use iterative method >> for solving a linear system, like bicg. Like in the ex10. In this example i >> don't understand why using MatPartitioning. >> > > Please rephrase the question. Are you asking why one should do the partition > or why one should not? Are you asking in the case where the matrix is read from > disk or generated in a parallel program? > > I try to understand the interest to call MatPartitioning before solving the linear system with the same matrix. (Like ksp/examples/tutorials/ex10.c). -- Dimitri Lecas From tmyue at westnet.com.au Thu Jan 11 15:13:25 2007 From: tmyue at westnet.com.au (Tan Meng YUE) Date: Fri, 12 Jan 2007 06:13:25 +0900 Subject: Visual Studio compiler and PETSc Message-ID: <380-220071411211325745@westnet.com.au> Hi, I have tried out PETSc on OS X and Linux in the past. My current place of work is a Windows only environment with a small (less than 1000) cluster of PCs/blade. I like to demostrate PETSc with MPICH2 on Windows to the developers here working on some fluid simulation code for digital visual effects. Are there any deployable binaries for PETSc (we only use C++, no Fortran)? I have tried compiling PETSc with Cygwin but was getting into various difficult like BLAS/LAPACK and when that was fixed, other problems cropped up. I have already installed MPICH2. Regards Cheers -- http://www.proceduralinsight.com/ From balay at mcs.anl.gov Thu Jan 11 15:38:35 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 11 Jan 2007 15:38:35 -0600 (CST) Subject: Visual Studio compiler and PETSc In-Reply-To: <380-220071411211325745@westnet.com.au> References: <380-220071411211325745@westnet.com.au> Message-ID: We don't have prebuild binaries.. Sugets configuring with: config/configure.py --with-cc='win32fe cl' --with-cxx='win32fe cl' --with-fc=0 --with-clanguage=cxx --download-c-blas-lapack=1 If you encounter problems - send us configure.log at petsc-maint at mcs.anl.gov Satish On Fri, 12 Jan 2007, Tan Meng YUE wrote: > Hi, > > I have tried out PETSc on OS X and Linux in the past. > > My current place of work is a Windows only environment with a small > (less than 1000) cluster of PCs/blade. > > I like to demostrate PETSc with MPICH2 on Windows to the developers > here working on some fluid simulation code for digital visual effects. > > Are there any deployable binaries for PETSc (we only use C++, no > Fortran)? > > I have tried compiling PETSc with Cygwin but was getting into > various difficult like BLAS/LAPACK and when that was fixed, other > problems cropped up. > > I have already installed MPICH2. > > Regards > > Cheers > -- > http://www.proceduralinsight.com/ > > From knepley at gmail.com Thu Jan 11 21:06:21 2007 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 11 Jan 2007 21:06:21 -0600 Subject: Using parMetis in petsc for ordering In-Reply-To: <45A6A55B.8050108@free.fr> References: <45A298A6.3020400@free.fr> <45A5742E.5080008@free.fr> <45A6932F.2010009@free.fr> <45A6A55B.8050108@free.fr> Message-ID: Reordering a matrix can result in fewer iterations for an iterative solver. Matt On 1/11/07, Dimitri Lecas wrote: > Barry Smith a ?crit : > > > > > > On Thu, 11 Jan 2007, Dimitri Lecas wrote: > > > > > >> Barry Smith a ?crit : > >> > >>> Dimitri, > >>> > >>> No, I think this is not the correct way to look at things. Load > >>> balancing the original matrix is not neccessarily a good thing for > >>> doing an LU factorization (in fact it is likely just to make the LU > >>> factorization have much more fill and require much more floating > >>> operations). > >>> Packages like SuperLU_dist and Mumps have their own internal ordering > >>> routines that are specifically for getting a good ordering for doing > >>> the parallel LU factorization, you should just have these solvers > >>> use them (which they do automatically). > >>> > >>> Barry > >>> > >>> > >>> On Thu, 11 Jan 2007, Dimitri Lecas wrote: > >>> > >>> > >>> > >> I'am no longer talking about doing LU factorization. But use iterative method > >> for solving a linear system, like bicg. Like in the ex10. In this example i > >> don't understand why using MatPartitioning. > >> > > > > Please rephrase the question. Are you asking why one should do the partition > > or why one should not? Are you asking in the case where the matrix is read from > > disk or generated in a parallel program? > > > > > I try to understand the interest to call MatPartitioning before solving > the linear system with the same matrix. (Like > ksp/examples/tutorials/ex10.c). > > -- > Dimitri Lecas > > -- One trouble is that despite this system, anyone who reads journals widely and critically is forced to realize that there are scarcely any bars to eventual publication. There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print. -- Drummond Rennie From bsmith at mcs.anl.gov Thu Jan 11 21:53:24 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 11 Jan 2007 22:53:24 -0500 (EST) Subject: Using parMetis in petsc for ordering In-Reply-To: <45A6A55B.8050108@free.fr> References: <45A298A6.3020400@free.fr> <45A5742E.5080008@free.fr> <45A6932F.2010009@free.fr> <45A6A55B.8050108@free.fr> Message-ID: In parallel matrix-vector products (used by all the KSP methods) the amount of communication is the number of "cut-edges" of the graph of the matrix. By repartitioning with metis this reduces the number of cut edges. Note: we don't actually advocate doing it this way. One should partition the underlying grid (finite element etc) then generate the matrix. If one does this then you do not repartition the matrix. Barry On Thu, 11 Jan 2007, Dimitri Lecas wrote: > Barry Smith a ?crit : > > > > On Thu, 11 Jan 2007, Dimitri Lecas wrote: > > > > > > > Barry Smith a ?crit : > > > > > > > Dimitri, > > > > > > > > No, I think this is not the correct way to look at things. Load > > > > balancing the original matrix is not neccessarily a good thing for > > > > doing an LU factorization (in fact it is likely just to make the LU > > > > factorization have much more fill and require much more floating > > > > operations). Packages like SuperLU_dist and Mumps have their own > > > > internal ordering > > > > routines that are specifically for getting a good ordering for doing > > > > the parallel LU factorization, you should just have these solvers > > > > use them (which they do automatically). > > > > > > > > Barry > > > > > > > > > > > > On Thu, 11 Jan 2007, Dimitri Lecas wrote: > > > > > > > > > > > I'am no longer talking about doing LU factorization. But use iterative > > > method > > > for solving a linear system, like bicg. Like in the ex10. In this example > > > i > > > don't understand why using MatPartitioning. > > > > > > > Please rephrase the question. Are you asking why one should do the > > partition or why one should not? Are you asking in the case where the matrix > > is read from > > disk or generated in a parallel program? > > > > > I try to understand the interest to call MatPartitioning before solving the > linear system with the same matrix. (Like ksp/examples/tutorials/ex10.c). > > From bessen at cs.columbia.edu Fri Jan 12 15:41:49 2007 From: bessen at cs.columbia.edu (Arvid Bessen) Date: Fri, 12 Jan 2007 13:41:49 -0800 Subject: No subject Message-ID: <5E722367-1998-4740-A9FF-B64B448E2616@cs.columbia.edu> Dear all, I have a problem with MPI in PETSc that is probably simple and you will hopefully know the answer to. In my code I treat the local rows of a MPIAIJ matrix one by one. If the number of local rows is equal on all processors, everything works well. However, if the number of rows is different, the program crashes. I have attached a simple example program to demonstrate what I want to do. Suppose process A has one more local row than process B. In the debugger I found out that the crash occurs in "MPI_Allreduce". Process A will call MPI_Allreduce one more time than process B, because it processes an additional row. While process A is still waiting for the result of the last MPI_Allreduce, process B will proceed and call MPI_Allreduce in a different context. This is, when the program crashes. What is the best way to make sure that all MPI calls are matched on the different processes? Thanks, Arvid static char help[] = "Example program with different row sizes\n\n"; #include "petscmat.h" #undef __FUNCT__ #define __FUNCT__ "main" int main(int Argc,char **Args) { const PetscInt size = 3; PetscErrorCode ierr; Mat cmat, dmat; PetscInt cmat_loc_start, cmat_loc_end, cmat_loc_size; PetscInt *rowsize_arr = 0; const PetscInt **row_arr = 0; IS *is_arr = 0; PetscInt a; Mat *submat_arr = 0; PetscInitialize(&Argc,&Args,(char *)0,help); ierr = MatCreateMPIAIJ(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, size, size, 1, PETSC_NULL, 2, PETSC_NULL, &cmat);CHKERRQ(ierr); ierr = MatAssemblyBegin(cmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = MatAssemblyEnd(cmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = MatCreateMPIAIJ(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, size, size, 1, PETSC_NULL, 2, PETSC_NULL, &dmat);CHKERRQ(ierr); ierr = MatAssemblyBegin(dmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = MatAssemblyEnd(dmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = MatGetOwnershipRange(cmat, &cmat_loc_start, &cmat_loc_end); CHKERRQ(ierr); cmat_loc_size = cmat_loc_end-cmat_loc_start; ierr = PetscMalloc(sizeof(PetscInt *)*cmat_loc_size, &row_arr); CHKERRQ(ierr); ierr = PetscMalloc(sizeof(PetscInt)*cmat_loc_size, &rowsize_arr); CHKERRQ(ierr); ierr = PetscMalloc(sizeof(IS)*cmat_loc_size, &is_arr); CHKERRQ(ierr); for( a=0; a References: <5E722367-1998-4740-A9FF-B64B448E2616@cs.columbia.edu> Message-ID: Why are you creating that ISGeneral with PETSC_COMM_WORLD. It does not seem to be global? Shouldn't it be PETSC_COMM_SELF? Also, what are you trying to do? Matt On 1/12/07, Arvid Bessen wrote: > > Dear all, > > I have a problem with MPI in PETSc that is probably simple and you > will hopefully know the answer to. In my code I treat the local rows > of a MPIAIJ matrix one by one. If the number of local rows is equal > on all processors, everything works well. However, if the number of > rows is different, the program crashes. I have attached a simple > example program to demonstrate what I want to do. > Suppose process A has one more local row than process B. In the > debugger I found out that the crash occurs in "MPI_Allreduce". > Process A will call MPI_Allreduce one more time than process B, > because it processes an additional row. While process A is still > waiting for the result of the last MPI_Allreduce, process B will > proceed and call MPI_Allreduce in a different context. This is, when > the program crashes. > > What is the best way to make sure that all MPI calls are matched on > the different processes? > > Thanks, > Arvid > > static char help[] = "Example program with different row sizes\n\n"; > > #include "petscmat.h" > > #undef __FUNCT__ > #define __FUNCT__ "main" > int main(int Argc,char **Args) > { > const PetscInt size = 3; > PetscErrorCode ierr; > Mat cmat, dmat; > PetscInt cmat_loc_start, cmat_loc_end, cmat_loc_size; > PetscInt *rowsize_arr = 0; > const PetscInt **row_arr = 0; > IS *is_arr = 0; > PetscInt a; > Mat *submat_arr = 0; > > PetscInitialize(&Argc,&Args,(char *)0,help); > > ierr = MatCreateMPIAIJ(PETSC_COMM_WORLD, PETSC_DECIDE, > PETSC_DECIDE, size, size, > 1, PETSC_NULL, 2, PETSC_NULL, &cmat);CHKERRQ(ierr); > ierr = MatAssemblyBegin(cmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(cmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatCreateMPIAIJ(PETSC_COMM_WORLD, PETSC_DECIDE, > PETSC_DECIDE, size, size, > 1, PETSC_NULL, 2, PETSC_NULL, &dmat);CHKERRQ(ierr); > ierr = MatAssemblyBegin(dmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(dmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > ierr = MatGetOwnershipRange(cmat, &cmat_loc_start, &cmat_loc_end); > CHKERRQ(ierr); > cmat_loc_size = cmat_loc_end-cmat_loc_start; > > ierr = PetscMalloc(sizeof(PetscInt *)*cmat_loc_size, &row_arr); > CHKERRQ(ierr); > ierr = PetscMalloc(sizeof(PetscInt)*cmat_loc_size, &rowsize_arr); > CHKERRQ(ierr); > ierr = PetscMalloc(sizeof(IS)*cmat_loc_size, &is_arr); CHKERRQ(ierr); > > for( a=0; a ierr = MatGetRow(cmat, a+cmat_loc_start, &(rowsize_arr[a]), & > (row_arr[a]), 0); CHKERRQ(ierr); > ierr = ISCreateGeneral(PETSC_COMM_WORLD, rowsize_arr[a], row_arr > [a], &(is_arr[a])); CHKERRQ(ierr); > ierr = MatRestoreRow(cmat, a+cmat_loc_start, &(rowsize_arr[a]), & > (row_arr[a]), 0); CHKERRQ(ierr); > } > > ierr = PetscMalloc(sizeof(Mat)*cmat_loc_size, &submat_arr); CHKERRQ > (ierr); > ierr = MatGetSubMatrices(dmat, cmat_loc_size, is_arr, is_arr, > MAT_INITIAL_MATRIX, &submat_arr); CHKERRQ(ierr); > > ierr = PetscFinalize();CHKERRQ(ierr); > return 0; > } > > > -- One trouble is that despite this system, anyone who reads journals widely and critically is forced to realize that there are scarcely any bars to eventual publication. There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print. -- Drummond Rennie From bessen at cs.columbia.edu Fri Jan 12 16:20:55 2007 From: bessen at cs.columbia.edu (Arvid Bessen) Date: Fri, 12 Jan 2007 14:20:55 -0800 Subject: In-Reply-To: References: <5E722367-1998-4740-A9FF-B64B448E2616@cs.columbia.edu> Message-ID: <484BF138-52E0-4D5C-BD6F-A6368CA593FE@cs.columbia.edu> On Jan 12, 2007, at 1:51 PM, Matthew Knepley wrote: > Why are you creating that ISGeneral with PETSC_COMM_WORLD. It does not > seem to be global? Shouldn't it be PETSC_COMM_SELF? Also, what are you > trying to do? > Thanks, that's a very good call, it works much better with PETSC_COMM_SELF, because then I don't have any MPI calls. What I was trying to do is basically determine the nonzero elements of a matrix row by row and then use this information to get exactly those entries from a different matrix. The actual code was too long and I just included everything that was needed for a crash. Thanks for your help, Arvid > Matt > > On 1/12/07, Arvid Bessen wrote: >> >> Dear all, >> >> I have a problem with MPI in PETSc that is probably simple and you >> will hopefully know the answer to. In my code I treat the local rows >> of a MPIAIJ matrix one by one. If the number of local rows is equal >> on all processors, everything works well. However, if the number of >> rows is different, the program crashes. I have attached a simple >> example program to demonstrate what I want to do. >> Suppose process A has one more local row than process B. In the >> debugger I found out that the crash occurs in "MPI_Allreduce". >> Process A will call MPI_Allreduce one more time than process B, >> because it processes an additional row. While process A is still >> waiting for the result of the last MPI_Allreduce, process B will >> proceed and call MPI_Allreduce in a different context. This is, when >> the program crashes. >> >> What is the best way to make sure that all MPI calls are matched on >> the different processes? >> >> Thanks, >> Arvid >> >> static char help[] = "Example program with different row sizes\n\n"; >> >> #include "petscmat.h" >> >> #undef __FUNCT__ >> #define __FUNCT__ "main" >> int main(int Argc,char **Args) >> { >> const PetscInt size = 3; >> PetscErrorCode ierr; >> Mat cmat, dmat; >> PetscInt cmat_loc_start, cmat_loc_end, cmat_loc_size; >> PetscInt *rowsize_arr = 0; >> const PetscInt **row_arr = 0; >> IS *is_arr = 0; >> PetscInt a; >> Mat *submat_arr = 0; >> >> PetscInitialize(&Argc,&Args,(char *)0,help); >> >> ierr = MatCreateMPIAIJ(PETSC_COMM_WORLD, PETSC_DECIDE, >> PETSC_DECIDE, size, size, >> 1, PETSC_NULL, 2, PETSC_NULL, >> &cmat);CHKERRQ(ierr); >> ierr = MatAssemblyBegin(cmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> ierr = MatAssemblyEnd(cmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> ierr = MatCreateMPIAIJ(PETSC_COMM_WORLD, PETSC_DECIDE, >> PETSC_DECIDE, size, size, >> 1, PETSC_NULL, 2, PETSC_NULL, >> &dmat);CHKERRQ(ierr); >> ierr = MatAssemblyBegin(dmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> ierr = MatAssemblyEnd(dmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> >> ierr = MatGetOwnershipRange(cmat, &cmat_loc_start, &cmat_loc_end); >> CHKERRQ(ierr); >> cmat_loc_size = cmat_loc_end-cmat_loc_start; >> >> ierr = PetscMalloc(sizeof(PetscInt *)*cmat_loc_size, &row_arr); >> CHKERRQ(ierr); >> ierr = PetscMalloc(sizeof(PetscInt)*cmat_loc_size, &rowsize_arr); >> CHKERRQ(ierr); >> ierr = PetscMalloc(sizeof(IS)*cmat_loc_size, &is_arr); CHKERRQ >> (ierr); >> >> for( a=0; a> ierr = MatGetRow(cmat, a+cmat_loc_start, &(rowsize_arr[a]), & >> (row_arr[a]), 0); CHKERRQ(ierr); >> ierr = ISCreateGeneral(PETSC_COMM_WORLD, rowsize_arr[a], row_arr >> [a], &(is_arr[a])); CHKERRQ(ierr); >> ierr = MatRestoreRow(cmat, a+cmat_loc_start, &(rowsize_arr >> [a]), & >> (row_arr[a]), 0); CHKERRQ(ierr); >> } >> >> ierr = PetscMalloc(sizeof(Mat)*cmat_loc_size, &submat_arr); >> CHKERRQ >> (ierr); >> ierr = MatGetSubMatrices(dmat, cmat_loc_size, is_arr, is_arr, >> MAT_INITIAL_MATRIX, &submat_arr); CHKERRQ(ierr); >> >> ierr = PetscFinalize();CHKERRQ(ierr); >> return 0; >> } >> >> >> > > > -- > One trouble is that despite this system, anyone who reads journals > widely > and critically is forced to realize that there are scarcely any > bars to eventual > publication. There seems to be no study too fragmented, no > hypothesis too > trivial, no literature citation too biased or too egotistical, no > design too > warped, no methodology too bungled, no presentation of results too > inaccurate, too obscure, and too contradictory, no analysis too > self-serving, > no argument too circular, no conclusions too trifling or too > unjustified, and > no grammar and syntax too offensive for a paper to end up in print. -- > Drummond Rennie From zonexo at gmail.com Fri Jan 12 19:45:29 2007 From: zonexo at gmail.com (Ben Tay) Date: Sat, 13 Jan 2007 09:45:29 +0800 Subject: Can I use ATLAS for the blas/lapack library Message-ID: <804ab5d40701121745t789f062cvd3dbe01a196d0f2f@mail.gmail.com> Hi, I've ATLAS on my server. Can I use it for the blas/lapack library? I tried to specify the directory and also the the library files but the PETSc configuration keep saying it's not valid. The library files are libatlas.a libcblas.a libf77blas.a liblapack.a libptcblas.a libptf77blas.a libtstatlas.a Which one should I specify? Lastly, have other users post any comments about using ATLAS? Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Fri Jan 12 20:46:06 2007 From: zonexo at gmail.com (Ben Tay) Date: Sat, 13 Jan 2007 10:46:06 +0800 Subject: Some questions about the parallel implementation of PETSc Message-ID: <804ab5d40701121846t3c134237ga1fe03ff05ce1330@mail.gmail.com> Hi, I've the sequential version of PETSc working and I have some questions about the parallel implementation. My code is to solve the NS eqn. Mainly it has to solve 2 linear eqns - momentum & poisson. The poisson eqn solving takes the most % of time so it going to be solved in parallel. Most likely the momentum eqn 'll be as well. The other parts of the code may be parallelized later using MPI. The parallel part is coded as follows: call MatCreate(PETSC_COMM_WORLD,A,ierr) call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m*n,m*n,ierr) call MatSetFromOptions(A,ierr) call MatGetOwnershipRange(A,Istart,Iend,ierr) call VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,m*n,b,ierr) call VecSetFromOptions(b,ierr) call VecDuplicate(b,x,ierr) ... insert matrix in parallel making use of Istart,Iend. ... assembly ... solve Since I've used PETSC_DECIDE, PETSc will determine the local dimension of A and b and x automatically. Is the no. of rows of A and b and x (Iend-Istart+1) locally? After the eqn is solved, I need to obtain the answer x I've used: call VecGetArray(x,ppv,i_vec,ierr) do j=1,size_y do i=1,size_x k=(j-1)*size_x+i p(i,j)=ppv(k+i_vec) end do end do call VecRestoreArray(x,ppv,i_vec,ierr) I'm trying to get the address of the 1st value of x and map the values onto a rectangular grid to update p. Is this correct? Or am I only able to get the local value, as in the local value of x. Do I need to use some MPI routines if I need to update the p values on the whole global grid? thanks alot! -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jan 12 21:02:16 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 12 Jan 2007 21:02:16 -0600 (CST) Subject: Some questions about the parallel implementation of PETSc In-Reply-To: <804ab5d40701121846t3c134237ga1fe03ff05ce1330@mail.gmail.com> References: <804ab5d40701121846t3c134237ga1fe03ff05ce1330@mail.gmail.com> Message-ID: Ben, Sounds like you are using a logically rectangular grid in two dimensions? (or 3). If so I highly recommend using the DMMG infrastructure in PETSc it handles all the decomposition of the domain into subrectangles, preallocates the matrix, handles ghost point updates and even lets you using multigrid. Check src/ksp/ksp/examples/tutorials/ex29.c and ex22.c Good luck, Barry On Sat, 13 Jan 2007, Ben Tay wrote: > Hi, > > I've the sequential version of PETSc working and I have some questions about > the parallel implementation. > > My code is to solve the NS eqn. Mainly it has to solve 2 linear eqns - > momentum & poisson. The poisson eqn solving takes the most % of time so it > going to be solved in parallel. Most likely the momentum eqn 'll be as well. > The other parts of the code may be parallelized later using MPI. > > > > The parallel part is coded as follows: > > call MatCreate(PETSC_COMM_WORLD,A,ierr) > > call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m*n,m*n,ierr) > > call MatSetFromOptions(A,ierr) > > call MatGetOwnershipRange(A,Istart,Iend,ierr) > > call VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,m*n,b,ierr) > > call VecSetFromOptions(b,ierr) > > call VecDuplicate(b,x,ierr) > > ... insert matrix in parallel making use of Istart,Iend. > > ... assembly > > ... solve > > Since I've used PETSC_DECIDE, PETSc will determine the local dimension of A > and b and x automatically. > > Is the no. of rows of A and b and x (Iend-Istart+1) locally? > > After the eqn is solved, I need to obtain the answer x > > I've used: > > > call VecGetArray(x,ppv,i_vec,ierr) > > do j=1,size_y > > do i=1,size_x > > k=(j-1)*size_x+i > > p(i,j)=ppv(k+i_vec) > > end do > > end do > > call VecRestoreArray(x,ppv,i_vec,ierr) > > I'm trying to get the address of the 1st value of x and map the values onto > a rectangular grid to update p. Is this correct? > > Or am I only able to get the local value, as in the local value of x. Do I > need to use some MPI routines if I need to update the p values on the whole > global grid? > > thanks alot! > From bsmith at mcs.anl.gov Fri Jan 12 21:04:30 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 12 Jan 2007 21:04:30 -0600 (CST) Subject: Can I use ATLAS for the blas/lapack library In-Reply-To: <804ab5d40701121745t789f062cvd3dbe01a196d0f2f@mail.gmail.com> References: <804ab5d40701121745t789f062cvd3dbe01a196d0f2f@mail.gmail.com> Message-ID: Ben, Please send the configure.log to petsc-maint at mcs.anl.gov It is suppose to work with the ATLAS libraries, but they are often temperamental. Barry On Sat, 13 Jan 2007, Ben Tay wrote: > Hi, > > I've ATLAS on my server. Can I use it for the blas/lapack library? > > I tried to specify the directory and also the the library files but the > PETSc configuration keep saying it's not valid. > > The library files are > > libatlas.a libcblas.a libf77blas.a liblapack.a libptcblas.a > libptf77blas.a libtstatlas.a > > Which one should I specify? > > Lastly, have other users post any comments about using ATLAS? > > Thank you! > From balay at mcs.anl.gov Sat Jan 13 01:23:12 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Sat, 13 Jan 2007 01:23:12 -0600 (CST) Subject: Can I use ATLAS for the blas/lapack library In-Reply-To: References: <804ab5d40701121745t789f062cvd3dbe01a196d0f2f@mail.gmail.com> Message-ID: You can specify with the option '--with-blas-lapack-dir=PATH' and configure will check if it can find comaptible libraries [wrt the compilers used with configure] However - unless you have a specific need for it [i.e make performance measurements for publishing] - its ok to just use regular blas/lapack. We recommend doing all the development with a debug build of PETSc libraries anyway.. [which is the default for configure] Satish On Fri, 12 Jan 2007, Barry Smith wrote: > > Ben, > > Please send the configure.log to petsc-maint at mcs.anl.gov > It is suppose to work with the ATLAS libraries, but they > are often temperamental. > > Barry > > On Sat, 13 Jan 2007, Ben Tay wrote: > > > Hi, > > > > I've ATLAS on my server. Can I use it for the blas/lapack library? > > > > I tried to specify the directory and also the the library files but the > > PETSc configuration keep saying it's not valid. > > > > The library files are > > > > libatlas.a libcblas.a libf77blas.a liblapack.a libptcblas.a > > libptf77blas.a libtstatlas.a > > > > Which one should I specify? > > > > Lastly, have other users post any comments about using ATLAS? > > > > Thank you! > > > > From thierry at pompo.net Sat Jan 13 01:18:32 2007 From: thierry at pompo.net (Thierry Thomas) Date: Sat, 13 Jan 2007 08:18:32 +0100 Subject: Can I use ATLAS for the blas/lapack library In-Reply-To: <804ab5d40701121745t789f062cvd3dbe01a196d0f2f@mail.gmail.com> References: <804ab5d40701121745t789f062cvd3dbe01a196d0f2f@mail.gmail.com> Message-ID: <20070113071832.GJ32885@graf.pompo.net> Le Sam 13 jan 07 ? 2:45:29 +0100, Ben Tay ?crivait?: > Hi, Hello, > I've ATLAS on my server. Can I use it for the blas/lapack library? > > I tried to specify the directory and also the the library files but the > PETSc configuration keep saying it's not valid. Yes, you can! Here is an extract of the FreeBSD port's Makefile: .if defined(WITH_ATLAS) LIB_DEPENDS+= atlas.2:${PORTSDIR}/math/atlas LIBBLASLAPACK= "-L${LOCALBASE}/lib -lalapack -lf77blas -latlas -lm" .else LIB_DEPENDS+= lapack.4:${PORTSDIR}/math/lapack LIBBLASLAPACK= "-L${LOCALBASE}/lib -llapack -lblas -lm" .endif CONFIGURE_ARGS+= --with-blas-lapack-lib=${LIBBLASLAPACK} Regards, -- Th. Thomas. From zonexo at gmail.com Sat Jan 13 01:44:24 2007 From: zonexo at gmail.com (Ben Tay) Date: Sat, 13 Jan 2007 15:44:24 +0800 Subject: Some questions about the parallel implementation of PETSc In-Reply-To: References: <804ab5d40701121846t3c134237ga1fe03ff05ce1330@mail.gmail.com> Message-ID: <804ab5d40701122344g69626cd2y23984b475c548dcd@mail.gmail.com> hi, actually it's a c grid which forms an airfoil. can I use DA & multi grid too? btw , regarding my previous question , is the answer for each process the I ocal values only or is it global? thanks ! On 1/13/07, Barry Smith wrote: > > Ben, > > Sounds like you are using a logically rectangular grid in two dimensions? > (or 3). If so I highly recommend using the DMMG infrastructure in PETSc it > handles all the decomposition of the domain into subrectangles, preallocates > the > matrix, handles ghost point updates and even lets you using multigrid. Check > src/ksp/ksp/examples/tutorials/ex29.c and ex22.c > > Good luck, > > Barry > > > On Sat, 13 Jan 2007, Ben Tay wrote: > > > Hi, > > > > I've the sequential version of PETSc working and I have some questions > about > > the parallel implementation. > > > > My code is to solve the NS eqn. Mainly it has to solve 2 linear eqns - > > momentum & poisson. The poisson eqn solving takes the most % of time so it > > going to be solved in parallel. Most likely the momentum eqn 'll be as > well. > > The other parts of the code may be parallelized later using MPI. > > > > > > > > The parallel part is coded as follows: > > > > call MatCreate(PETSC_COMM_WORLD,A,ierr) > > > > call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m*n,m*n,ierr) > > > > call MatSetFromOptions(A,ierr) > > > > call MatGetOwnershipRange(A,Istart,Iend,ierr) > > > > call VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,m*n,b,ierr) > > > > call VecSetFromOptions(b,ierr) > > > > call VecDuplicate(b,x,ierr) > > > > ... insert matrix in parallel making use of Istart,Iend. > > > > ... assembly > > > > ... solve > > > > Since I've used PETSC_DECIDE, PETSc will determine the local dimension of > A > > and b and x automatically. > > > > Is the no. of rows of A and b and x (Iend-Istart+1) locally? > > > > After the eqn is solved, I need to obtain the answer x > > > > I've used: > > > > > > call VecGetArray(x,ppv,i_vec,ierr) > > > > do j=1,size_y > > > > do i=1,size_x > > > > k=(j-1)*size_x+i > > > > p(i,j)=ppv(k+i_vec) > > > > end do > > > > end do > > > > call VecRestoreArray(x,ppv,i_vec,ierr) > > > > I'm trying to get the address of the 1st value of x and map the values > onto > > a rectangular grid to update p. Is this correct? > > > > Or am I only able to get the local value, as in the local value of x. Do I > > need to use some MPI routines if I need to update the p values on the > whole > > global grid? > > > > thanks alot! > > > > From zonexo at gmail.com Sat Jan 13 07:18:55 2007 From: zonexo at gmail.com (Ben Tay) Date: Sat, 13 Jan 2007 21:18:55 +0800 Subject: disk space requirement for mpich2 during PETSc library compilation Message-ID: <804ab5d40701130518n3f2b3f07ud2460ca5d874d94b@mail.gmail.com> Hi, I am trying to compile PETSc with mpi using --download-mpich=1 in linux. The command is ./config/configure.py --with-fc=/lsftmp/g0306332/inter/fc/bin/ifort --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/ --download-mpich=1 --with-x=0 --with-shared It displays: ================================================================================= Running configure on MPICH; this may take several minutes ================================================================================= ================================================================================= Running make on MPICH; this may take several minutes ================================================================================= then it says disk quota exceeded. I've about 450mb free space, which is all filled up when the error shows. May I know how much disk space is required? Also can I compile just mpich on a scratch directory and then moved it to the PETSc externalpackages directory? Or do I have to compile everything (including PETSc) on a scratch directory and moved it my my directory? thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aja2111 at columbia.edu Sat Jan 13 07:47:17 2007 From: aja2111 at columbia.edu (Aron Ahmadia) Date: Sat, 13 Jan 2007 08:47:17 -0500 Subject: disk space requirement for mpich2 during PETSc library compilation In-Reply-To: <804ab5d40701130518n3f2b3f07ud2460ca5d874d94b@mail.gmail.com> References: <804ab5d40701130518n3f2b3f07ud2460ca5d874d94b@mail.gmail.com> Message-ID: <37604ab40701130547h10cdad23x59048fa4e8267497@mail.gmail.com> Hi Ben, My PETSc install on an OS X machine requires about 343 MB of space, about 209 MB of which is MPICH. Unfortunately this has the potential of exceeding 500 MB temporarily I believe as the make process generates a lot of object files during the software build. I think what you want to do is compile and install your own copy of MPICH (using a scratch directory or whatever tools you have at your disposal), then use the --with-mpi-dir=/location/to/mpich/install argument into configure. I've never staged a PETSc build on a machine with an extremely limited quota, the developers might have some suggestions on how to do this. ~A On 1/13/07, Ben Tay wrote: > Hi, > > I am trying to compile PETSc with mpi using --download-mpich=1 in linux. The > command is > > > > ./config/configure.py > --with-fc=/lsftmp/g0306332/inter/fc/bin/ifort > --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/ > --download-mpich=1 --with-x=0 --with-shared > > It displays: > > ================================================================================= > Running configure on MPICH; this may take several > minutes > ================================================================================= > ================================================================================= > Running make on MPICH; this may take several > minutes > ================================================================================= > > then it says disk quota exceeded. I've about 450mb free space, which is all > filled up when the error shows. May I know how much disk space is required? > > Also can I compile just mpich on a scratch directory and then moved it to > the PETSc externalpackages directory? Or do I have to compile everything > (including PETSc) on a scratch directory and moved it my my directory? > > thank you. From bsmith at mcs.anl.gov Sat Jan 13 09:01:02 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 13 Jan 2007 09:01:02 -0600 (CST) Subject: disk space requirement for mpich2 during PETSc library compilation In-Reply-To: <37604ab40701130547h10cdad23x59048fa4e8267497@mail.gmail.com> References: <804ab5d40701130518n3f2b3f07ud2460ca5d874d94b@mail.gmail.com> <37604ab40701130547h10cdad23x59048fa4e8267497@mail.gmail.com> Message-ID: Ben, This is partially our fault. We never run "make clean" on MPICH after the install so there are lots of .o and .a files lying around. I've updated BuildSystem/config/packages/MPI.py to run make clean after the install. Barry On Sat, 13 Jan 2007, Aron Ahmadia wrote: > Hi Ben, > > My PETSc install on an OS X machine requires about 343 MB of space, > about 209 MB of which is MPICH. Unfortunately this has the potential > of exceeding 500 MB temporarily I believe as the make process > generates a lot of object files during the software build. > > I think what you want to do is compile and install your own copy of > MPICH (using a scratch directory or whatever tools you have at your > disposal), then use the --with-mpi-dir=/location/to/mpich/install > argument into configure. > > I've never staged a PETSc build on a machine with an extremely limited > quota, the developers might have some suggestions on how to do this. > > ~A > > On 1/13/07, Ben Tay wrote: > > Hi, > > > > I am trying to compile PETSc with mpi using --download-mpich=1 in linux. The > > command is > > > > > > > > ./config/configure.py > > --with-fc=/lsftmp/g0306332/inter/fc/bin/ifort > > --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/ > > --download-mpich=1 --with-x=0 --with-shared > > > > It displays: > > > > ================================================================================= > > Running configure on MPICH; this may take several > > minutes > > ================================================================================= > > ================================================================================= > > Running make on MPICH; this may take several > > minutes > > ================================================================================= > > > > then it says disk quota exceeded. I've about 450mb free space, which is all > > filled up when the error shows. May I know how much disk space is required? > > > > Also can I compile just mpich on a scratch directory and then moved it to > > the PETSc externalpackages directory? Or do I have to compile everything > > (including PETSc) on a scratch directory and moved it my my directory? > > > > thank you. > > From zonexo at gmail.com Sat Jan 13 09:13:38 2007 From: zonexo at gmail.com (Ben Tay) Date: Sat, 13 Jan 2007 23:13:38 +0800 Subject: disk space requirement for mpich2 during PETSc library compilation In-Reply-To: References: <804ab5d40701130518n3f2b3f07ud2460ca5d874d94b@mail.gmail.com> <37604ab40701130547h10cdad23x59048fa4e8267497@mail.gmail.com> Message-ID: <804ab5d40701130713o364ffa8eo3d236da6f2def1b8@mail.gmail.com> Thanks Barry & Aron. I've tried to install mpich2 on a scratch directory and it finished in a short while. On 1/13/07, Barry Smith wrote: > > > Ben, > > This is partially our fault. We never run "make clean" on MPICH after the > install so there are lots of .o and .a files lying around. I've updated > BuildSystem/config/packages/MPI.py to run make clean after the install. > > Barry > > On Sat, 13 Jan 2007, Aron Ahmadia wrote: > > > Hi Ben, > > > > My PETSc install on an OS X machine requires about 343 MB of space, > > about 209 MB of which is MPICH. Unfortunately this has the potential > > of exceeding 500 MB temporarily I believe as the make process > > generates a lot of object files during the software build. > > > > I think what you want to do is compile and install your own copy of > > MPICH (using a scratch directory or whatever tools you have at your > > disposal), then use the --with-mpi-dir=/location/to/mpich/install > > argument into configure. > > > > I've never staged a PETSc build on a machine with an extremely limited > > quota, the developers might have some suggestions on how to do this. > > > > ~A > > > > On 1/13/07, Ben Tay wrote: > > > Hi, > > > > > > I am trying to compile PETSc with mpi using --download-mpich=1 in > linux. The > > > command is > > > > > > > > > > > > ./config/configure.py > > > --with-fc=/lsftmp/g0306332/inter/fc/bin/ifort > > > --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/ > > > --download-mpich=1 --with-x=0 --with-shared > > > > > > It displays: > > > > > > > ================================================================================= > > > Running configure on MPICH; this may take several > > > minutes > > > > ================================================================================= > > > > ================================================================================= > > > Running make on MPICH; this may take several > > > minutes > > > > ================================================================================= > > > > > > then it says disk quota exceeded. I've about 450mb free space, which > is all > > > filled up when the error shows. May I know how much disk space is > required? > > > > > > Also can I compile just mpich on a scratch directory and then moved it > to > > > the PETSc externalpackages directory? Or do I have to compile > everything > > > (including PETSc) on a scratch directory and moved it my my directory? > > > > > > thank you. > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Sun Jan 14 18:32:32 2007 From: zonexo at gmail.com (Ben Tay) Date: Mon, 15 Jan 2007 08:32:32 +0800 Subject: disk space requirement for mpich2 during PETSc library compilation In-Reply-To: <804ab5d40701130713o364ffa8eo3d236da6f2def1b8@mail.gmail.com> References: <804ab5d40701130518n3f2b3f07ud2460ca5d874d94b@mail.gmail.com> <37604ab40701130547h10cdad23x59048fa4e8267497@mail.gmail.com> <804ab5d40701130713o364ffa8eo3d236da6f2def1b8@mail.gmail.com> Message-ID: <804ab5d40701141632vbf32697w36af205a85faf244@mail.gmail.com> Hi, I've tried to use the shared version of mpich2 which I installed seperately (due to above problem) with PETSc. The command is ./config/configure.py --with-fc=/lsftmp/g0306332/inter/fc/bin/ifort --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/ --with-mpi-dir=/lsftmp/g0306332/mpich2-l32 --with-x=0 --with-shared During the test, this error msg was shown: gcc -c -fPIC -Wall -Wwrite-strings -g3 -I/nas/lsftmp/g0306332/petsc-2.3.2-p8-I/nas/lsftmp/g0306332/petsc- 2.3.2-p8/bmake/linux-mpich2 -I/nas/lsftmp/g0306332/petsc-2.3.2-p8/include -I/lsftmp/g0306332/mpich2-l32/include -D__SDIR__="src/snes/examples/tutorials/" ex19.c gcc -fPIC -Wall -Wwrite-strings -g3 -o ex19 ex19.o-Wl,-rpath,/nas/lsftmp/g0306332/petsc- 2.3.2-p8/lib/linux-mpich2 -L/nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpich2 -lpetscsnes -lpetscksp -lpetscdm -lpetscmat -lpetscvec -lpetsc -Wl,-rpath,/lsftmp/g0306332/mpich2-l32/lib -L/lsftmp/g0306332/mpich2-l32/lib -lmpich -lnsl -laio -lrt -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack -lmkl_def -lguide -lvml -lpthread -lm -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3 -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3 -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.. -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.. -ldl -lgcc_eh -Wl,-rpath,"/usr/lib/gcc-lib/i386-pc-linux/3.2.3" -Wl,-rpath,"/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.." -Wl,-rpath,/lsftmp/g0306332/inter/fc/lib -L/lsftmp/g0306332/inter/fc/lib -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3/ -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3/ -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../../ -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../../ -lifport -lifcore -limf -lm -lipgo -lirc -lgcc_s -lirc_s -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3 -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.. -lm -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3 -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3 -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.. -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.. -ldl -lgcc_eh -ldl /nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpich2/libpetsc.so: undefined reference to `mpi_conversion_fn_null_' collect2: ld returned 1 exit status Then I tried to my fortran example. I realised that if I compile using ifort (using command similar to "make ex1f", there is no problem but the result shown that 4 processors are running 4 *individual* codes. I then tried to use the mpif90 in the mpich2 directory. compiling was ok but during linking the error was: /lsftmp/g0306332/mpich2-l32/bin/mpif90 -fPIC -g -o ex2f ex2f.o -Wl,-rpath,/nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpich2 -L/nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpich2 -lpetscksp -lpetscdm -lpetscmat -lpetscvec -lpetsc -Wl,-rpath,/lsftmp/g0306332/mpich2-l32/lib -L/lsftmp/g0306332/mpich2-l32/lib -lmpich -lnsl -laio -lrt -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack -lmkl_def -lguide -lvml -lpthread -lm -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3 -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3 -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.. -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.. -ldl -lgcc_eh -Wl,-rpath,"/usr/lib/gcc-lib/i386-pc-linux/3.2.3" -Wl,-rpath,"/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.." -Wl,-rpath,/lsftmp/g0306332/inter/fc/lib -L/lsftmp/g0306332/inter/fc/lib -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3/ -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3/ -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../../ -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../../ -lifport -lifcore -limf -lm -lipgo -lirc -lgcc_s -lirc_s -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3 -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.. -lm -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3 -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3 -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.. -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.. -ldl -lgcc_eh -ldl /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to `__clog10q' /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to `__cexp10q' /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to `__csqrtq' /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to `__ccoshq' /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to `__ctanhq' /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to `__ccosq' /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to `__clogq' /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to `__csinhq' /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to `__ctanq' /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to `__cpowq' /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to `__exp10q' /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to `__cexpq' /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to `__cabsq' /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to `__fabsq' /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to `__csinq' /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to `__ldexpq' /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to `__frexpq' So what is wrong? I am a novice in mpi so I hope someone can give some advice. Thank you. PS: Has the "make clean" on MPICH been updated on BuildSystem/config/packages/MPI.py? On 1/13/07, Ben Tay wrote: > > Thanks Barry & Aron. > > I've tried to install mpich2 on a scratch directory and it finished in a > short while. > > > On 1/13/07, Barry Smith wrote: > > > > > > Ben, > > > > This is partially our fault. We never run "make clean" on MPICH after > > the > > install so there are lots of .o and .a files lying around. I've updated > > BuildSystem/config/packages/MPI.py to run make clean after the install. > > > > Barry > > > > On Sat, 13 Jan 2007, Aron Ahmadia wrote: > > > > > Hi Ben, > > > > > > My PETSc install on an OS X machine requires about 343 MB of space, > > > about 209 MB of which is MPICH. Unfortunately this has the potential > > > of exceeding 500 MB temporarily I believe as the make process > > > generates a lot of object files during the software build. > > > > > > I think what you want to do is compile and install your own copy of > > > MPICH (using a scratch directory or whatever tools you have at your > > > disposal), then use the --with-mpi-dir=/location/to/mpich/install > > > argument into configure. > > > > > > I've never staged a PETSc build on a machine with an extremely limited > > > quota, the developers might have some suggestions on how to do this. > > > > > > ~A > > > > > > On 1/13/07, Ben Tay wrote: > > > > Hi, > > > > > > > > I am trying to compile PETSc with mpi using --download-mpich=1 in > > linux. The > > > > command is > > > > > > > > > > > > > > > > ./config/configure.py > > > > --with-fc=/lsftmp/g0306332/inter/fc/bin/ifort > > > > --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/ > > > > --download-mpich=1 --with-x=0 --with-shared > > > > > > > > It displays: > > > > > > > > > > ================================================================================= > > > > Running configure on MPICH; this may take several > > > > minutes > > > > > > ================================================================================= > > > > > > ================================================================================= > > > > Running make on MPICH; this may take several > > > > minutes > > > > > > ================================================================================= > > > > > > > > then it says disk quota exceeded. I've about 450mb free space, which > > is all > > > > filled up when the error shows. May I know how much disk space is > > required? > > > > > > > > Also can I compile just mpich on a scratch directory and then moved > > it to > > > > the PETSc externalpackages directory? Or do I have to compile > > everything > > > > (including PETSc) on a scratch directory and moved it my my > > directory? > > > > > > > > thank you. > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Mon Jan 15 08:53:10 2007 From: zonexo at gmail.com (Ben Tay) Date: Mon, 15 Jan 2007 22:53:10 +0800 Subject: mpi not working Message-ID: <804ab5d40701150653r1e522500s551eaad8fea116d5@mail.gmail.com> hi, i install PETSc using the following command: ./config/configure.py --with-vendor-compilers=intel --with-gnu-compilers=0 --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/lib/32 --with-mpi-dir=/opt/mpich/intel/ --with-x=0 --with-shared then i got: Compilers: C Compiler: /opt/mpich/intel/bin/mpicc -fPIC -g Fortran Compiler: /opt/mpich/intel/bin/mpif90 -I. -fPIC -g -w90 -w Linkers: Shared linker: /opt/mpich/intel/bin/mpicc -shared -fPIC -g Dynamic linker: /opt/mpich/intel/bin/mpicc -shared -fPIC -g PETSc: PETSC_ARCH: linux-mpif90 PETSC_DIR: /nas/lsftmp/g0306332/petsc-2.3.2-p8 ** ** Now build and test the libraries with "make all test" ** Clanguage: C Scalar type:real MPI: Includes: ['/opt/mpich/intel/include'] PETSc shared libraries: enabled PETSc dynamic libraries: disabled BLAS/LAPACK: -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack -lmkl_ia32 -lguide i ran "make all test" and everything seems fine /opt/mpich/intel/bin/mpicc -c -fPIC -g -I/nas/lsftmp/g0306332/petsc-2.3.2-p8-I/nas/lsftmp/g0306332/petsc- 2.3.2-p8/bmake/linux-mpif90 -I/nas/lsftmp/g0306332/petsc-2.3.2-p8/include -I/opt/mpich/intel/include -D__SDIR__="src/snes/examples/tutorials/" ex19.c /opt/mpich/intel/bin/mpicc -fPIC -g -o ex19 ex19.o-Wl,-rpath,/nas/lsftmp/g0306332/petsc- 2.3.2-p8/lib/linux-mpif90 -L/nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpif90 -lpetscsnes -lpetscksp -lpetscdm -lpetscmat -lpetscvec -lpetsc -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack -lmkl_ia32 -lguide -lPEPCF90 -Wl,-rpath,/opt/intel/compiler70/ia32/lib -Wl,-rpath,/opt/mpich/intel/lib -L/opt/mpich/intel/lib -Wl,-rpath,-rpath -Wl,-rpath,-ldl -L-ldl -lmpich -Wl,-rpath,/opt/intel/compiler70/ia32/lib -Wl,-rpath,/opt/intel/compiler70/ia32/lib -L/opt/intel/compiler70/ia32/lib -Wl,-rpath,/usr/lib -Wl,-rpath,/usr/lib -L/usr/lib -limf -lirc -lcprts -lcxa -lunwind -ldl -lmpichf90 -lPEPCF90 -Wl,-rpath,/opt/intel/compiler70/ia32/lib -L/opt/intel/compiler70/ia32/lib -Wl,-rpath,/usr/lib -L/usr/lib -lintrins -lIEPCF90 -lF90 -lm -Wl,-rpath,\ -Wl,-rpath,\ -L\ -ldl -lmpich -Wl,-rpath,/opt/intel/compiler70/ia32/lib -L/opt/intel/compiler70/ia32/lib -Wl,-rpath,/usr/lib -L/usr/lib -limf -lirc -lcprts -lcxa -lunwind -ldl /bin/rm -f ex19.o C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI processes Fortran example src/snes/examples/tutorials/ex5f run successfully with 1 MPI process Completed test examples I then tried to run my own parallel code. It's a simple code which prints the rank of each processor. If I compile the code using just mpif90 test.F(using just mpif.h) I get 0,1,2,3 (4 processors). however, if i change the code to use petsc.h etc ie. program ns2d_c implicit none #include "include/finclude/petsc.h" #include "include/finclude/petscvec.h" #include "include/finclude/petscmat.h" #include "include/finclude/petscksp.h" #include "include/finclude/petscpc.h" #include "include/finclude/petscmat.h90" integer,parameter :: size_x=8,size_y=4 integer :: ierr,Istart_p,Iend_p,Ntot,Istart_m,Iend_m,k PetscMPIInt nprocs,rank call PetscInitialize(PETSC_NULL_CHARACTER,ierr) call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr) call MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr) end program ns2d_c i then rename the filename to ex2f.F and use "make ex2f" the result I get is something like 0,0,0,0. Why is this so? Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jan 15 10:17:57 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 15 Jan 2007 10:17:57 -0600 (CST) Subject: disk space requirement for mpich2 during PETSc library compilation In-Reply-To: <804ab5d40701141632vbf32697w36af205a85faf244@mail.gmail.com> References: <804ab5d40701130518n3f2b3f07ud2460ca5d874d94b@mail.gmail.com> <37604ab40701130547h10cdad23x59048fa4e8267497@mail.gmail.com> <804ab5d40701130713o364ffa8eo3d236da6f2def1b8@mail.gmail.com> <804ab5d40701141632vbf32697w36af205a85faf244@mail.gmail.com> Message-ID: On Mon, 15 Jan 2007, Ben Tay wrote: > Hi, > > I've tried to use the shared version of mpich2 which I installed seperately > (due to above problem) with PETSc. The command is > > ./config/configure.py --with-fc=/lsftmp/g0306332/inter/fc/bin/ifort ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Do not use this option; since you pass in --with-mpi-dir= it will use the proper mpif90 script. Barry > --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/ > --with-mpi-dir=/lsftmp/g0306332/mpich2-l32 --with-x=0 --with-shared > > During the test, this error msg was shown: > > gcc -c -fPIC -Wall -Wwrite-strings -g3 > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8-I/nas/lsftmp/g0306332/petsc- > 2.3.2-p8/bmake/linux-mpich2 -I/nas/lsftmp/g0306332/petsc-2.3.2-p8/include > -I/lsftmp/g0306332/mpich2-l32/include > -D__SDIR__="src/snes/examples/tutorials/" ex19.c > gcc -fPIC -Wall -Wwrite-strings -g3 -o ex19 > ex19.o-Wl,-rpath,/nas/lsftmp/g0306332/petsc- > 2.3.2-p8/lib/linux-mpich2 > -L/nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpich2 > -lpetscsnes -lpetscksp -lpetscdm -lpetscmat -lpetscvec -lpetsc > -Wl,-rpath,/lsftmp/g0306332/mpich2-l32/lib -L/lsftmp/g0306332/mpich2-l32/lib > -lmpich -lnsl -laio -lrt -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack -lmkl_def -lguide -lvml > -lpthread -lm -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3 > -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3 > -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.. > -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.. -ldl -lgcc_eh > -Wl,-rpath,"/usr/lib/gcc-lib/i386-pc-linux/3.2.3" > -Wl,-rpath,"/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.." > -Wl,-rpath,/lsftmp/g0306332/inter/fc/lib -L/lsftmp/g0306332/inter/fc/lib > -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3/ > -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3/ > -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../../ > -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../../ -lifport -lifcore -limf -lm > -lipgo -lirc -lgcc_s -lirc_s -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3 > -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.. -lm > -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3 > -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3 > -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.. > -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.. -ldl -lgcc_eh -ldl > /nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpich2/libpetsc.so: undefined > reference to `mpi_conversion_fn_null_' > collect2: ld returned 1 exit status > > Then I tried to my fortran example. > > I realised that if I compile using ifort (using command similar to "make > ex1f", there is no problem but the result shown that 4 processors are > running 4 *individual* codes. > > I then tried to use the mpif90 in the mpich2 directory. compiling was ok but > during linking the error was: > > /lsftmp/g0306332/mpich2-l32/bin/mpif90 -fPIC -g -o ex2f ex2f.o > -Wl,-rpath,/nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpich2 > -L/nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpich2 -lpetscksp -lpetscdm > -lpetscmat -lpetscvec -lpetsc -Wl,-rpath,/lsftmp/g0306332/mpich2-l32/lib > -L/lsftmp/g0306332/mpich2-l32/lib -lmpich -lnsl -laio -lrt > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack -lmkl_def -lguide -lvml > -lpthread -lm -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3 > -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3 > -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.. > -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.. -ldl -lgcc_eh > -Wl,-rpath,"/usr/lib/gcc-lib/i386-pc-linux/3.2.3" > -Wl,-rpath,"/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.." > -Wl,-rpath,/lsftmp/g0306332/inter/fc/lib -L/lsftmp/g0306332/inter/fc/lib > -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3/ > -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3/ > -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../../ > -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../../ -lifport -lifcore -limf -lm > -lipgo -lirc -lgcc_s -lirc_s -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3 > -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.. -lm > -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3 > -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3 > -Wl,-rpath,/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.. > -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3/../../.. -ldl -lgcc_eh -ldl > /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to > `__clog10q' > /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to > `__cexp10q' > /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to > `__csqrtq' > /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to > `__ccoshq' > /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to > `__ctanhq' > /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to > `__ccosq' > /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to > `__clogq' > /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to > `__csinhq' > /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to > `__ctanq' > /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to > `__cpowq' > /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to > `__exp10q' > /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to > `__cexpq' > /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to > `__cabsq' > /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to > `__fabsq' > /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to > `__csinq' > /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to > `__ldexpq' > /lsftmp/g0306332/inter/fc/lib/libifcore.so.5: undefined reference to > `__frexpq' > > > So what is wrong? I am a novice in mpi so I hope someone can give some > advice. Thank you. > > PS: Has the "make clean" on MPICH been updated on > BuildSystem/config/packages/MPI.py? > > > On 1/13/07, Ben Tay wrote: > > > > Thanks Barry & Aron. > > > > I've tried to install mpich2 on a scratch directory and it finished in a > > short while. > > > > > > On 1/13/07, Barry Smith wrote: > > > > > > > > > Ben, > > > > > > This is partially our fault. We never run "make clean" on MPICH after > > > the > > > install so there are lots of .o and .a files lying around. I've updated > > > BuildSystem/config/packages/MPI.py to run make clean after the install. > > > > > > Barry > > > > > > On Sat, 13 Jan 2007, Aron Ahmadia wrote: > > > > > > > Hi Ben, > > > > > > > > My PETSc install on an OS X machine requires about 343 MB of space, > > > > about 209 MB of which is MPICH. Unfortunately this has the potential > > > > of exceeding 500 MB temporarily I believe as the make process > > > > generates a lot of object files during the software build. > > > > > > > > I think what you want to do is compile and install your own copy of > > > > MPICH (using a scratch directory or whatever tools you have at your > > > > disposal), then use the --with-mpi-dir=/location/to/mpich/install > > > > argument into configure. > > > > > > > > I've never staged a PETSc build on a machine with an extremely limited > > > > quota, the developers might have some suggestions on how to do this. > > > > > > > > ~A > > > > > > > > On 1/13/07, Ben Tay wrote: > > > > > Hi, > > > > > > > > > > I am trying to compile PETSc with mpi using --download-mpich=1 in > > > linux. The > > > > > command is > > > > > > > > > > > > > > > > > > > > ./config/configure.py > > > > > --with-fc=/lsftmp/g0306332/inter/fc/bin/ifort > > > > > --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/ > > > > > --download-mpich=1 --with-x=0 --with-shared > > > > > > > > > > It displays: > > > > > > > > > > > > > > > ================================================================================= > > > > > Running configure on MPICH; this may take several > > > > > minutes > > > > > > > > > > ================================================================================= > > > > > > > > > > ================================================================================= > > > > > Running make on MPICH; this may take several > > > > > minutes > > > > > > > > > > ================================================================================= > > > > > > > > > > then it says disk quota exceeded. I've about 450mb free space, which > > > is all > > > > > filled up when the error shows. May I know how much disk space is > > > required? > > > > > > > > > > Also can I compile just mpich on a scratch directory and then moved > > > it to > > > > > the PETSc externalpackages directory? Or do I have to compile > > > everything > > > > > (including PETSc) on a scratch directory and moved it my my > > > directory? > > > > > > > > > > thank you. > > > > > > > > > > > > > > > > > From li76pan at yahoo.com Mon Jan 15 10:31:59 2007 From: li76pan at yahoo.com (li pan) Date: Mon, 15 Jan 2007 08:31:59 -0800 (PST) Subject: mpi not working In-Reply-To: <804ab5d40701150653r1e522500s551eaad8fea116d5@mail.gmail.com> Message-ID: <25804.18538.qm@web36802.mail.mud.yahoo.com> I did try to download and install petsc2.3.2, but end up with error: mpich can not be download & installed, please install mpich for windows manually. In the homepage of mpich2, I didn't choose the version for windows but the source code version. And compiled it by myself. Then, I gave the --with-mpi-dir="install dir". Petsc was configured, and now it's doing "make". One interesting thing is, I installed petsc in linux before. The mpi libraries were very large (libmpich.a==60 mb). But this time in cygwin it was only several mbs. best pan --- Ben Tay wrote: > hi, > > i install PETSc using the following command: > > ./config/configure.py --with-vendor-compilers=intel > --with-gnu-compilers=0 > --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/lib/32 > --with-mpi-dir=/opt/mpich/intel/ --with-x=0 > --with-shared > > then i got: > > Compilers: > > C Compiler: /opt/mpich/intel/bin/mpicc > -fPIC -g > Fortran Compiler: /opt/mpich/intel/bin/mpif90 > -I. -fPIC -g -w90 -w > Linkers: > Shared linker: /opt/mpich/intel/bin/mpicc > -shared -fPIC -g > Dynamic linker: /opt/mpich/intel/bin/mpicc > -shared -fPIC -g > PETSc: > PETSC_ARCH: linux-mpif90 > PETSC_DIR: /nas/lsftmp/g0306332/petsc-2.3.2-p8 > ** > ** Now build and test the libraries with "make all > test" > ** > Clanguage: C > Scalar type:real > MPI: > Includes: ['/opt/mpich/intel/include'] > PETSc shared libraries: enabled > PETSc dynamic libraries: disabled > BLAS/LAPACK: > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack > -lmkl_ia32 -lguide > > i ran "make all test" and everything seems fine > > /opt/mpich/intel/bin/mpicc -c -fPIC -g > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8-I/nas/lsftmp/g0306332/petsc- > 2.3.2-p8/bmake/linux-mpif90 > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8/include > -I/opt/mpich/intel/include > -D__SDIR__="src/snes/examples/tutorials/" ex19.c > /opt/mpich/intel/bin/mpicc -fPIC -g -o ex19 > ex19.o-Wl,-rpath,/nas/lsftmp/g0306332/petsc- > 2.3.2-p8/lib/linux-mpif90 > -L/nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpif90 > -lpetscsnes -lpetscksp -lpetscdm -lpetscmat > -lpetscvec -lpetsc > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack > -lmkl_ia32 -lguide > -lPEPCF90 -Wl,-rpath,/opt/intel/compiler70/ia32/lib > -Wl,-rpath,/opt/mpich/intel/lib > -L/opt/mpich/intel/lib -Wl,-rpath,-rpath > -Wl,-rpath,-ldl -L-ldl -lmpich > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > -L/opt/intel/compiler70/ia32/lib > -Wl,-rpath,/usr/lib -Wl,-rpath,/usr/lib -L/usr/lib > -limf -lirc -lcprts -lcxa > -lunwind -ldl -lmpichf90 -lPEPCF90 > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > -L/opt/intel/compiler70/ia32/lib -Wl,-rpath,/usr/lib > -L/usr/lib -lintrins > -lIEPCF90 -lF90 -lm -Wl,-rpath,\ -Wl,-rpath,\ -L\ > -ldl -lmpich > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > -L/opt/intel/compiler70/ia32/lib > -Wl,-rpath,/usr/lib -L/usr/lib -limf -lirc -lcprts > -lcxa -lunwind -ldl > /bin/rm -f ex19.o > C/C++ example src/snes/examples/tutorials/ex19 run > successfully with 1 MPI > process > C/C++ example src/snes/examples/tutorials/ex19 run > successfully with 2 MPI > processes > Fortran example src/snes/examples/tutorials/ex5f run > successfully with 1 MPI > process > Completed test examples > > I then tried to run my own parallel code. It's a > simple code which prints > the rank of each processor. > > If I compile the code using just mpif90 test.F(using > just mpif.h) > > I get 0,1,2,3 (4 processors). > > however, if i change the code to use petsc.h etc > ie. > > > program ns2d_c > > implicit none > > > #include "include/finclude/petsc.h" > #include "include/finclude/petscvec.h" > #include "include/finclude/petscmat.h" > #include "include/finclude/petscksp.h" > #include "include/finclude/petscpc.h" > #include "include/finclude/petscmat.h90" > > integer,parameter :: size_x=8,size_y=4 > > integer :: > ierr,Istart_p,Iend_p,Ntot,Istart_m,Iend_m,k > > PetscMPIInt nprocs,rank > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > call > MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr) > > call > MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr) > > end program ns2d_c > > > > i then rename the filename to ex2f.F and use "make > ex2f" > > the result I get is something like 0,0,0,0. > > > > Why is this so? > > Thank you. > ____________________________________________________________________________________ Never Miss an Email Stay connected with Yahoo! Mail on your mobile. Get started! http://mobile.yahoo.com/services?promote=mail From zonexo at gmail.com Tue Jan 16 01:56:26 2007 From: zonexo at gmail.com (Ben Tay) Date: Tue, 16 Jan 2007 15:56:26 +0800 Subject: mpi not working In-Reply-To: <25804.18538.qm@web36802.mail.mud.yahoo.com> References: <804ab5d40701150653r1e522500s551eaad8fea116d5@mail.gmail.com> <25804.18538.qm@web36802.mail.mud.yahoo.com> Message-ID: <804ab5d40701152356u6bf38de1u438fb28b19b0922c@mail.gmail.com> Hi Pan, I also got very big library files if I use PETSc with mpich2. Btw, I have tried several options but I still don't understand why I can't get mpi to work with PETSc. The 4 processors are running together but each running its own code. I just use integer :: nprocs,rank,ierr call PetscInitialize(PETSC_NULL_CHARACTER,ierr) call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr) call MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr) print *, rank, nprocs call PetscFinalize(ierr) The answers I get is 0,1 repeated 4 times instead of 0,4 1,4 2,4 3,4. I'm using my school's server's mpich and it work if I just compile in pure mpi. Btw, if I need to send the job to 4 processors, I need to use a script file: #BSUB -o std-output #BSUB -q linux_parallel_test #BSUB -n 4 /usr/lsf6/bin/mpijob_gm /opt/mpich/myrinet/intel/bin/mpirun a.out I wonder if the problem lies here... Thank you. On 1/16/07, li pan wrote: > > I did try to download and install petsc2.3.2, but end > up with error: mpich can not be download & installed, > please install mpich for windows manually. > In the homepage of mpich2, I didn't choose the version > for windows but the source code version. And compiled > it by myself. Then, I gave the --with-mpi-dir="install > dir". Petsc was configured, and now it's doing "make". > One interesting thing is, I installed petsc in linux > before. The mpi libraries were very large > (libmpich.a==60 mb). But this time in cygwin it was > only several mbs. > > best > > pan > > > --- Ben Tay wrote: > > > hi, > > > > i install PETSc using the following command: > > > > ./config/configure.py --with-vendor-compilers=intel > > --with-gnu-compilers=0 > > > --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/lib/32 > > --with-mpi-dir=/opt/mpich/intel/ --with-x=0 > > --with-shared > > > > then i got: > > > > Compilers: > > > > C Compiler: /opt/mpich/intel/bin/mpicc > > -fPIC -g > > Fortran Compiler: /opt/mpich/intel/bin/mpif90 > > -I. -fPIC -g -w90 -w > > Linkers: > > Shared linker: /opt/mpich/intel/bin/mpicc > > -shared -fPIC -g > > Dynamic linker: /opt/mpich/intel/bin/mpicc > > -shared -fPIC -g > > PETSc: > > PETSC_ARCH: linux-mpif90 > > PETSC_DIR: /nas/lsftmp/g0306332/petsc-2.3.2-p8 > > ** > > ** Now build and test the libraries with "make all > > test" > > ** > > Clanguage: C > > Scalar type:real > > MPI: > > Includes: ['/opt/mpich/intel/include'] > > PETSc shared libraries: enabled > > PETSc dynamic libraries: disabled > > BLAS/LAPACK: > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack > > -lmkl_ia32 -lguide > > > > i ran "make all test" and everything seems fine > > > > /opt/mpich/intel/bin/mpicc -c -fPIC -g > > > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8-I/nas/lsftmp/g0306332/petsc- > > 2.3.2-p8/bmake/linux-mpif90 > > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8/include > > -I/opt/mpich/intel/include > > -D__SDIR__="src/snes/examples/tutorials/" ex19.c > > /opt/mpich/intel/bin/mpicc -fPIC -g -o ex19 > > ex19.o-Wl,-rpath,/nas/lsftmp/g0306332/petsc- > > 2.3.2-p8/lib/linux-mpif90 > > > -L/nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpif90 > > -lpetscsnes -lpetscksp -lpetscdm -lpetscmat > > -lpetscvec -lpetsc > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack > > -lmkl_ia32 -lguide > > -lPEPCF90 -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > -Wl,-rpath,/opt/mpich/intel/lib > > -L/opt/mpich/intel/lib -Wl,-rpath,-rpath > > -Wl,-rpath,-ldl -L-ldl -lmpich > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > -L/opt/intel/compiler70/ia32/lib > > -Wl,-rpath,/usr/lib -Wl,-rpath,/usr/lib -L/usr/lib > > -limf -lirc -lcprts -lcxa > > -lunwind -ldl -lmpichf90 -lPEPCF90 > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > -L/opt/intel/compiler70/ia32/lib -Wl,-rpath,/usr/lib > > -L/usr/lib -lintrins > > -lIEPCF90 -lF90 -lm -Wl,-rpath,\ -Wl,-rpath,\ -L\ > > -ldl -lmpich > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > -L/opt/intel/compiler70/ia32/lib > > -Wl,-rpath,/usr/lib -L/usr/lib -limf -lirc -lcprts > > -lcxa -lunwind -ldl > > /bin/rm -f ex19.o > > C/C++ example src/snes/examples/tutorials/ex19 run > > successfully with 1 MPI > > process > > C/C++ example src/snes/examples/tutorials/ex19 run > > successfully with 2 MPI > > processes > > Fortran example src/snes/examples/tutorials/ex5f run > > successfully with 1 MPI > > process > > Completed test examples > > > > I then tried to run my own parallel code. It's a > > simple code which prints > > the rank of each processor. > > > > If I compile the code using just mpif90 test.F(using > > just mpif.h) > > > > I get 0,1,2,3 (4 processors). > > > > however, if i change the code to use petsc.h etc > > ie. > > > > > > program ns2d_c > > > > implicit none > > > > > > #include "include/finclude/petsc.h" > > #include "include/finclude/petscvec.h" > > #include "include/finclude/petscmat.h" > > #include "include/finclude/petscksp.h" > > #include "include/finclude/petscpc.h" > > #include "include/finclude/petscmat.h90" > > > > integer,parameter :: size_x=8,size_y=4 > > > > integer :: > > ierr,Istart_p,Iend_p,Ntot,Istart_m,Iend_m,k > > > > PetscMPIInt nprocs,rank > > > > > > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > > > call > > MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr) > > > > call > > MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr) > > > > end program ns2d_c > > > > > > > > i then rename the filename to ex2f.F and use "make > > ex2f" > > > > the result I get is something like 0,0,0,0. > > > > > > > > Why is this so? > > > > Thank you. > > > > > > > > ____________________________________________________________________________________ > Never Miss an Email > Stay connected with Yahoo! Mail on your mobile. Get started! > http://mobile.yahoo.com/services?promote=mail > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shma7099 at student.uu.se Tue Jan 16 06:57:30 2007 From: shma7099 at student.uu.se (Shaman Mahmoudi) Date: Tue, 16 Jan 2007 13:57:30 +0100 Subject: mpi not working In-Reply-To: <804ab5d40701152356u6bf38de1u438fb28b19b0922c@mail.gmail.com> References: <804ab5d40701150653r1e522500s551eaad8fea116d5@mail.gmail.com> <25804.18538.qm@web36802.mail.mud.yahoo.com> <804ab5d40701152356u6bf38de1u438fb28b19b0922c@mail.gmail.com> Message-ID: <42FB7732-5791-4EC6-B298-40040D153818@student.uu.se> Hi, Maybe there is a problem with the output-to-screen buffer. A not very uncommon problem if you use a job-scheduler. Try to add unbuffer (if it is available in your system) in your script file and see if it improves. More information regarding unbuffer can be found here http:// expect.nist.gov/example/unbuffer.man.html With best regards, Shaman Mahmoudi On Jan 16, 2007, at 8:56 AM, Ben Tay wrote: > Hi Pan, > > I also got very big library files if I use PETSc with mpich2. > > Btw, I have tried several options but I still don't understand why > I can't get mpi to work with PETSc. > > The 4 processors are running together but each running its own code. > > I just use > > integer :: nprocs,rank,ierr > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr) > > call MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr) > > print *, rank, nprocs > > call PetscFinalize(ierr) > > > The answers I get is 0,1 repeated 4 times instead of 0,4 1,4 2,4 3,4. > > I'm using my school's server's mpich and it work if I just compile > in pure mpi. > > Btw, if I need to send the job to 4 processors, I need to use a > script file: > > #BSUB -o std-output > #BSUB -q linux_parallel_test > #BSUB -n 4 > /usr/lsf6/bin/mpijob_gm /opt/mpich/myrinet/intel/bin/mpirun a.out > > I wonder if the problem lies here... > > > Thank you. > > > > > On 1/16/07, li pan wrote: > I did try to download and install petsc2.3.2, but end > up with error: mpich can not be download & installed, > please install mpich for windows manually. > In the homepage of mpich2, I didn't choose the version > for windows but the source code version. And compiled > it by myself. Then, I gave the --with-mpi-dir="install > dir". Petsc was configured, and now it's doing "make". > One interesting thing is, I installed petsc in linux > before. The mpi libraries were very large > (libmpich.a==60 mb). But this time in cygwin it was > only several mbs. > > best > > pan > > > --- Ben Tay wrote: > > > hi, > > > > i install PETSc using the following command: > > > > ./config/configure.py --with-vendor-compilers=intel > > --with-gnu-compilers=0 > > > --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/lib/32 > > --with-mpi-dir=/opt/mpich/intel/ --with-x=0 > > --with-shared > > > > then i got: > > > > Compilers: > > > > C Compiler: /opt/mpich/intel/bin/mpicc > > -fPIC -g > > Fortran Compiler: /opt/mpich/intel/bin/mpif90 > > -I. -fPIC -g -w90 -w > > Linkers: > > Shared linker: /opt/mpich/intel/bin/mpicc > > -shared -fPIC -g > > Dynamic linker: /opt/mpich/intel/bin/mpicc > > -shared -fPIC -g > > PETSc: > > PETSC_ARCH: linux-mpif90 > > PETSC_DIR: /nas/lsftmp/g0306332/petsc-2.3.2-p8 > > ** > > ** Now build and test the libraries with "make all > > test" > > ** > > Clanguage: C > > Scalar type:real > > MPI: > > Includes: ['/opt/mpich/intel/include'] > > PETSc shared libraries: enabled > > PETSc dynamic libraries: disabled > > BLAS/LAPACK: > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack > > -lmkl_ia32 -lguide > > > > i ran "make all test" and everything seems fine > > > > /opt/mpich/intel/bin/mpicc -c -fPIC -g > > > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8-I/nas/lsftmp/g0306332/petsc- > > 2.3.2-p8/bmake/linux-mpif90 > > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8/include > > -I/opt/mpich/intel/include > > -D__SDIR__="src/snes/examples/tutorials/" ex19.c > > /opt/mpich/intel/bin/mpicc -fPIC -g -o ex19 > > ex19.o-Wl,-rpath,/nas/lsftmp/g0306332/petsc- > > 2.3.2-p8/lib/linux-mpif90 > > > -L/nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpif90 > > -lpetscsnes -lpetscksp -lpetscdm -lpetscmat > > -lpetscvec -lpetsc > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack > > -lmkl_ia32 -lguide > > -lPEPCF90 -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > -Wl,-rpath,/opt/mpich/intel/lib > > -L/opt/mpich/intel/lib -Wl,-rpath,-rpath > > -Wl,-rpath,-ldl -L-ldl -lmpich > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > -L/opt/intel/compiler70/ia32/lib > > -Wl,-rpath,/usr/lib -Wl,-rpath,/usr/lib -L/usr/lib > > -limf -lirc -lcprts -lcxa > > -lunwind -ldl -lmpichf90 -lPEPCF90 > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > -L/opt/intel/compiler70/ia32/lib -Wl,-rpath,/usr/lib > > -L/usr/lib -lintrins > > -lIEPCF90 -lF90 -lm -Wl,-rpath,\ -Wl,-rpath,\ -L\ > > -ldl -lmpich > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > -L/opt/intel/compiler70/ia32/lib > > -Wl,-rpath,/usr/lib -L/usr/lib -limf -lirc -lcprts > > -lcxa -lunwind -ldl > > /bin/rm -f ex19.o > > C/C++ example src/snes/examples/tutorials/ex19 run > > successfully with 1 MPI > > process > > C/C++ example src/snes/examples/tutorials/ex19 run > > successfully with 2 MPI > > processes > > Fortran example src/snes/examples/tutorials/ex5f run > > successfully with 1 MPI > > process > > Completed test examples > > > > I then tried to run my own parallel code. It's a > > simple code which prints > > the rank of each processor. > > > > If I compile the code using just mpif90 test.F(using > > just mpif.h) > > > > I get 0,1,2,3 (4 processors). > > > > however, if i change the code to use petsc.h etc > > ie. > > > > > > program ns2d_c > > > > implicit none > > > > > > #include "include/finclude/petsc.h" > > #include "include/finclude/petscvec.h" > > #include "include/finclude/petscmat.h" > > #include "include/finclude/petscksp.h" > > #include "include/finclude/petscpc.h" > > #include "include/finclude/petscmat.h90" > > > > integer,parameter :: size_x=8,size_y=4 > > > > integer :: > > ierr,Istart_p,Iend_p,Ntot,Istart_m,Iend_m,k > > > > PetscMPIInt nprocs,rank > > > > > > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > > > call > > MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr) > > > > call > > MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr) > > > > end program ns2d_c > > > > > > > > i then rename the filename to ex2f.F and use "make > > ex2f" > > > > the result I get is something like 0,0,0,0. > > > > > > > > Why is this so? > > > > Thank you. > > > > > > > ______________________________________________________________________ > ______________ > Never Miss an Email > Stay connected with Yahoo! Mail on your mobile. Get started! > http://mobile.yahoo.com/services?promote=mail > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jan 16 07:32:18 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 16 Jan 2007 07:32:18 -0600 (CST) Subject: mpi not working In-Reply-To: <804ab5d40701152356u6bf38de1u438fb28b19b0922c@mail.gmail.com> References: <804ab5d40701150653r1e522500s551eaad8fea116d5@mail.gmail.com> <25804.18538.qm@web36802.mail.mud.yahoo.com> <804ab5d40701152356u6bf38de1u438fb28b19b0922c@mail.gmail.com> Message-ID: Ben, You definitely have to submit a PETSc job just like any other MPI job. So please try using the script. Barry On Tue, 16 Jan 2007, Ben Tay wrote: > Hi Pan, > > I also got very big library files if I use PETSc with mpich2. > > Btw, I have tried several options but I still don't understand why I can't > get mpi to work with PETSc. > > The 4 processors are running together but each running its own code. > > I just use > > > integer :: nprocs,rank,ierr > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr) > > call MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr) > > print *, rank, nprocs > > call PetscFinalize(ierr) > > > > The answers I get is 0,1 repeated 4 times instead of 0,4 1,4 2,4 3,4. > > I'm using my school's server's mpich and it work if I just compile in pure > mpi. > > Btw, if I need to send the job to 4 processors, I need to use a script file: > > #BSUB -o std-output > #BSUB -q linux_parallel_test > #BSUB -n 4 > /usr/lsf6/bin/mpijob_gm /opt/mpich/myrinet/intel/bin/mpirun a.out > > I wonder if the problem lies here... > > > > Thank you. > > > > On 1/16/07, li pan wrote: > > > > I did try to download and install petsc2.3.2, but end > > up with error: mpich can not be download & installed, > > please install mpich for windows manually. > > In the homepage of mpich2, I didn't choose the version > > for windows but the source code version. And compiled > > it by myself. Then, I gave the --with-mpi-dir="install > > dir". Petsc was configured, and now it's doing "make". > > One interesting thing is, I installed petsc in linux > > before. The mpi libraries were very large > > (libmpich.a==60 mb). But this time in cygwin it was > > only several mbs. > > > > best > > > > pan > > > > > > --- Ben Tay wrote: > > > > > hi, > > > > > > i install PETSc using the following command: > > > > > > ./config/configure.py --with-vendor-compilers=intel > > > --with-gnu-compilers=0 > > > > > --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/lib/32 > > > --with-mpi-dir=/opt/mpich/intel/ --with-x=0 > > > --with-shared > > > > > > then i got: > > > > > > Compilers: > > > > > > C Compiler: /opt/mpich/intel/bin/mpicc > > > -fPIC -g > > > Fortran Compiler: /opt/mpich/intel/bin/mpif90 > > > -I. -fPIC -g -w90 -w > > > Linkers: > > > Shared linker: /opt/mpich/intel/bin/mpicc > > > -shared -fPIC -g > > > Dynamic linker: /opt/mpich/intel/bin/mpicc > > > -shared -fPIC -g > > > PETSc: > > > PETSC_ARCH: linux-mpif90 > > > PETSC_DIR: /nas/lsftmp/g0306332/petsc-2.3.2-p8 > > > ** > > > ** Now build and test the libraries with "make all > > > test" > > > ** > > > Clanguage: C > > > Scalar type:real > > > MPI: > > > Includes: ['/opt/mpich/intel/include'] > > > PETSc shared libraries: enabled > > > PETSc dynamic libraries: disabled > > > BLAS/LAPACK: > > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > > > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack > > > -lmkl_ia32 -lguide > > > > > > i ran "make all test" and everything seems fine > > > > > > /opt/mpich/intel/bin/mpicc -c -fPIC -g > > > > > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8-I/nas/lsftmp/g0306332/petsc- > > > 2.3.2-p8/bmake/linux-mpif90 > > > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8/include > > > -I/opt/mpich/intel/include > > > -D__SDIR__="src/snes/examples/tutorials/" ex19.c > > > /opt/mpich/intel/bin/mpicc -fPIC -g -o ex19 > > > ex19.o-Wl,-rpath,/nas/lsftmp/g0306332/petsc- > > > 2.3.2-p8/lib/linux-mpif90 > > > > > -L/nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpif90 > > > -lpetscsnes -lpetscksp -lpetscdm -lpetscmat > > > -lpetscvec -lpetsc > > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > > > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack > > > -lmkl_ia32 -lguide > > > -lPEPCF90 -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > -Wl,-rpath,/opt/mpich/intel/lib > > > -L/opt/mpich/intel/lib -Wl,-rpath,-rpath > > > -Wl,-rpath,-ldl -L-ldl -lmpich > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > -L/opt/intel/compiler70/ia32/lib > > > -Wl,-rpath,/usr/lib -Wl,-rpath,/usr/lib -L/usr/lib > > > -limf -lirc -lcprts -lcxa > > > -lunwind -ldl -lmpichf90 -lPEPCF90 > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > -L/opt/intel/compiler70/ia32/lib -Wl,-rpath,/usr/lib > > > -L/usr/lib -lintrins > > > -lIEPCF90 -lF90 -lm -Wl,-rpath,\ -Wl,-rpath,\ -L\ > > > -ldl -lmpich > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > -L/opt/intel/compiler70/ia32/lib > > > -Wl,-rpath,/usr/lib -L/usr/lib -limf -lirc -lcprts > > > -lcxa -lunwind -ldl > > > /bin/rm -f ex19.o > > > C/C++ example src/snes/examples/tutorials/ex19 run > > > successfully with 1 MPI > > > process > > > C/C++ example src/snes/examples/tutorials/ex19 run > > > successfully with 2 MPI > > > processes > > > Fortran example src/snes/examples/tutorials/ex5f run > > > successfully with 1 MPI > > > process > > > Completed test examples > > > > > > I then tried to run my own parallel code. It's a > > > simple code which prints > > > the rank of each processor. > > > > > > If I compile the code using just mpif90 test.F(using > > > just mpif.h) > > > > > > I get 0,1,2,3 (4 processors). > > > > > > however, if i change the code to use petsc.h etc > > > ie. > > > > > > > > > program ns2d_c > > > > > > implicit none > > > > > > > > > #include "include/finclude/petsc.h" > > > #include "include/finclude/petscvec.h" > > > #include "include/finclude/petscmat.h" > > > #include "include/finclude/petscksp.h" > > > #include "include/finclude/petscpc.h" > > > #include "include/finclude/petscmat.h90" > > > > > > integer,parameter :: size_x=8,size_y=4 > > > > > > integer :: > > > ierr,Istart_p,Iend_p,Ntot,Istart_m,Iend_m,k > > > > > > PetscMPIInt nprocs,rank > > > > > > > > > > > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > > > > > call > > > MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr) > > > > > > call > > > MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr) > > > > > > end program ns2d_c > > > > > > > > > > > > i then rename the filename to ex2f.F and use "make > > > ex2f" > > > > > > the result I get is something like 0,0,0,0. > > > > > > > > > > > > Why is this so? > > > > > > Thank you. > > > > > > > > > > > > > > > ____________________________________________________________________________________ > > Never Miss an Email > > Stay connected with Yahoo! Mail on your mobile. Get started! > > http://mobile.yahoo.com/services?promote=mail > > > > > From yaronkretchmer at gmail.com Tue Jan 16 15:26:38 2007 From: yaronkretchmer at gmail.com (Yaron Kretchmer) Date: Tue, 16 Jan 2007 13:26:38 -0800 Subject: UCSC users? Message-ID: Hi Petsc-users I'm looking for Petsc users in UCSC (UC Santa Cruz)- if you're one, and contact me off-list, I'd appreciate it Thanks Yaron -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Tue Jan 16 18:34:36 2007 From: zonexo at gmail.com (Ben Tay) Date: Wed, 17 Jan 2007 08:34:36 +0800 Subject: mpi not working In-Reply-To: References: <804ab5d40701150653r1e522500s551eaad8fea116d5@mail.gmail.com> <25804.18538.qm@web36802.mail.mud.yahoo.com> <804ab5d40701152356u6bf38de1u438fb28b19b0922c@mail.gmail.com> Message-ID: <804ab5d40701161634u58b77c1cr16daa8d72e28f8a7@mail.gmail.com> Thanks Shaman. But the problem is that I get 0,1 0,1 0,1 0,1 instead of 0,4 1,4 2,4 3,4 which means there are 4 processors instead of the above, whereby it seems that 4 serial jobs are running. Barry: The script was given by my school when parallel jobs are to be submitted. I use the same script when submitting pure mpi job and it works. On the other hand, the PETSc parallel code ran successfully on another of my school's server. It was also submitted using a script, but a slightly different one since it's another system. However, that server is very busy hence I usually use the current server. Do you have other other solution? Or should I try other ways of compilation? Btw, I am using ifc 7.0 and icc 7.0. The codes are written in fortran. Thank you. On 1/16/07, Barry Smith wrote: > > > Ben, > > You definitely have to submit a PETSc job just like > any other MPI job. So please try using the script. > > Barry > > > On Tue, 16 Jan 2007, Ben Tay wrote: > > > Hi Pan, > > > > I also got very big library files if I use PETSc with mpich2. > > > > Btw, I have tried several options but I still don't understand why I > can't > > get mpi to work with PETSc. > > > > The 4 processors are running together but each running its own code. > > > > I just use > > > > > > integer :: nprocs,rank,ierr > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > > > call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr) > > > > call MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr) > > > > print *, rank, nprocs > > > > call PetscFinalize(ierr) > > > > > > > > The answers I get is 0,1 repeated 4 times instead of 0,4 1,4 2,4 3,4. > > > > I'm using my school's server's mpich and it work if I just compile in > pure > > mpi. > > > > Btw, if I need to send the job to 4 processors, I need to use a script > file: > > > > #BSUB -o std-output > > #BSUB -q linux_parallel_test > > #BSUB -n 4 > > /usr/lsf6/bin/mpijob_gm /opt/mpich/myrinet/intel/bin/mpirun a.out > > > > I wonder if the problem lies here... > > > > > > > > Thank you. > > > > > > > > On 1/16/07, li pan wrote: > > > > > > I did try to download and install petsc2.3.2, but end > > > up with error: mpich can not be download & installed, > > > please install mpich for windows manually. > > > In the homepage of mpich2, I didn't choose the version > > > for windows but the source code version. And compiled > > > it by myself. Then, I gave the --with-mpi-dir="install > > > dir". Petsc was configured, and now it's doing "make". > > > One interesting thing is, I installed petsc in linux > > > before. The mpi libraries were very large > > > (libmpich.a==60 mb). But this time in cygwin it was > > > only several mbs. > > > > > > best > > > > > > pan > > > > > > > > > --- Ben Tay wrote: > > > > > > > hi, > > > > > > > > i install PETSc using the following command: > > > > > > > > ./config/configure.py --with-vendor-compilers=intel > > > > --with-gnu-compilers=0 > > > > > > > --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/lib/32 > > > > --with-mpi-dir=/opt/mpich/intel/ --with-x=0 > > > > --with-shared > > > > > > > > then i got: > > > > > > > > Compilers: > > > > > > > > C Compiler: /opt/mpich/intel/bin/mpicc > > > > -fPIC -g > > > > Fortran Compiler: /opt/mpich/intel/bin/mpif90 > > > > -I. -fPIC -g -w90 -w > > > > Linkers: > > > > Shared linker: /opt/mpich/intel/bin/mpicc > > > > -shared -fPIC -g > > > > Dynamic linker: /opt/mpich/intel/bin/mpicc > > > > -shared -fPIC -g > > > > PETSc: > > > > PETSC_ARCH: linux-mpif90 > > > > PETSC_DIR: /nas/lsftmp/g0306332/petsc-2.3.2-p8 > > > > ** > > > > ** Now build and test the libraries with "make all > > > > test" > > > > ** > > > > Clanguage: C > > > > Scalar type:real > > > > MPI: > > > > Includes: ['/opt/mpich/intel/include'] > > > > PETSc shared libraries: enabled > > > > PETSc dynamic libraries: disabled > > > > BLAS/LAPACK: > > > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > > > > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack > > > > -lmkl_ia32 -lguide > > > > > > > > i ran "make all test" and everything seems fine > > > > > > > > /opt/mpich/intel/bin/mpicc -c -fPIC -g > > > > > > > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8-I/nas/lsftmp/g0306332/petsc- > > > > 2.3.2-p8/bmake/linux-mpif90 > > > > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8/include > > > > -I/opt/mpich/intel/include > > > > -D__SDIR__="src/snes/examples/tutorials/" ex19.c > > > > /opt/mpich/intel/bin/mpicc -fPIC -g -o ex19 > > > > ex19.o-Wl,-rpath,/nas/lsftmp/g0306332/petsc- > > > > 2.3.2-p8/lib/linux-mpif90 > > > > > > > -L/nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpif90 > > > > -lpetscsnes -lpetscksp -lpetscdm -lpetscmat > > > > -lpetscvec -lpetsc > > > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > > > > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack > > > > -lmkl_ia32 -lguide > > > > -lPEPCF90 -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > -Wl,-rpath,/opt/mpich/intel/lib > > > > -L/opt/mpich/intel/lib -Wl,-rpath,-rpath > > > > -Wl,-rpath,-ldl -L-ldl -lmpich > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > -L/opt/intel/compiler70/ia32/lib > > > > -Wl,-rpath,/usr/lib -Wl,-rpath,/usr/lib -L/usr/lib > > > > -limf -lirc -lcprts -lcxa > > > > -lunwind -ldl -lmpichf90 -lPEPCF90 > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > -L/opt/intel/compiler70/ia32/lib -Wl,-rpath,/usr/lib > > > > -L/usr/lib -lintrins > > > > -lIEPCF90 -lF90 -lm -Wl,-rpath,\ -Wl,-rpath,\ -L\ > > > > -ldl -lmpich > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > -L/opt/intel/compiler70/ia32/lib > > > > -Wl,-rpath,/usr/lib -L/usr/lib -limf -lirc -lcprts > > > > -lcxa -lunwind -ldl > > > > /bin/rm -f ex19.o > > > > C/C++ example src/snes/examples/tutorials/ex19 run > > > > successfully with 1 MPI > > > > process > > > > C/C++ example src/snes/examples/tutorials/ex19 run > > > > successfully with 2 MPI > > > > processes > > > > Fortran example src/snes/examples/tutorials/ex5f run > > > > successfully with 1 MPI > > > > process > > > > Completed test examples > > > > > > > > I then tried to run my own parallel code. It's a > > > > simple code which prints > > > > the rank of each processor. > > > > > > > > If I compile the code using just mpif90 test.F(using > > > > just mpif.h) > > > > > > > > I get 0,1,2,3 (4 processors). > > > > > > > > however, if i change the code to use petsc.h etc > > > > ie. > > > > > > > > > > > > program ns2d_c > > > > > > > > implicit none > > > > > > > > > > > > #include "include/finclude/petsc.h" > > > > #include "include/finclude/petscvec.h" > > > > #include "include/finclude/petscmat.h" > > > > #include "include/finclude/petscksp.h" > > > > #include "include/finclude/petscpc.h" > > > > #include "include/finclude/petscmat.h90" > > > > > > > > integer,parameter :: size_x=8,size_y=4 > > > > > > > > integer :: > > > > ierr,Istart_p,Iend_p,Ntot,Istart_m,Iend_m,k > > > > > > > > PetscMPIInt nprocs,rank > > > > > > > > > > > > > > > > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > > > > > > > call > > > > MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr) > > > > > > > > call > > > > MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr) > > > > > > > > end program ns2d_c > > > > > > > > > > > > > > > > i then rename the filename to ex2f.F and use "make > > > > ex2f" > > > > > > > > the result I get is something like 0,0,0,0. > > > > > > > > > > > > > > > > Why is this so? > > > > > > > > Thank you. > > > > > > > > > > > > > > > > > > > > > > > ____________________________________________________________________________________ > > > Never Miss an Email > > > Stay connected with Yahoo! Mail on your mobile. Get started! > > > http://mobile.yahoo.com/services?promote=mail > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jan 16 19:02:04 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 16 Jan 2007 19:02:04 -0600 (CST) Subject: mpi not working In-Reply-To: <804ab5d40701161634u58b77c1cr16daa8d72e28f8a7@mail.gmail.com> References: <804ab5d40701150653r1e522500s551eaad8fea116d5@mail.gmail.com> <25804.18538.qm@web36802.mail.mud.yahoo.com> <804ab5d40701152356u6bf38de1u438fb28b19b0922c@mail.gmail.com> <804ab5d40701161634u58b77c1cr16daa8d72e28f8a7@mail.gmail.com> Message-ID: Ben, I don't know what to say; what you report is inherently contradictory. What happens when you run src/vec/vec/examples/tutorials/ex1.c on 2 processors? Barry On Wed, 17 Jan 2007, Ben Tay wrote: > Thanks Shaman. But the problem is that I get > > 0,1 > 0,1 > 0,1 > 0,1 > > instead of > > 0,4 > 1,4 > 2,4 > 3,4 which means there are 4 processors instead of the above, whereby it > seems that 4 serial jobs are running. > > Barry: > > The script was given by my school when parallel jobs are to be submitted. I > use the same script when submitting pure mpi job and it works. On the other > hand, the PETSc parallel code ran successfully on another of my school's > server. It was also submitted using a script, but a slightly different one > since it's another system. However, that server is very busy hence I usually > use the current server. > > Do you have other other solution? Or should I try other ways of compilation? > Btw, I am using ifc 7.0 and icc 7.0. The codes are written in fortran. > > Thank you. > > > On 1/16/07, Barry Smith wrote: > > > > > > Ben, > > > > You definitely have to submit a PETSc job just like > > any other MPI job. So please try using the script. > > > > Barry > > > > > > On Tue, 16 Jan 2007, Ben Tay wrote: > > > > > Hi Pan, > > > > > > I also got very big library files if I use PETSc with mpich2. > > > > > > Btw, I have tried several options but I still don't understand why I > > can't > > > get mpi to work with PETSc. > > > > > > The 4 processors are running together but each running its own code. > > > > > > I just use > > > > > > > > > integer :: nprocs,rank,ierr > > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > > > > > call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr) > > > > > > call MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr) > > > > > > print *, rank, nprocs > > > > > > call PetscFinalize(ierr) > > > > > > > > > > > > The answers I get is 0,1 repeated 4 times instead of 0,4 1,4 2,4 3,4. > > > > > > I'm using my school's server's mpich and it work if I just compile in > > pure > > > mpi. > > > > > > Btw, if I need to send the job to 4 processors, I need to use a script > > file: > > > > > > #BSUB -o std-output > > > #BSUB -q linux_parallel_test > > > #BSUB -n 4 > > > /usr/lsf6/bin/mpijob_gm /opt/mpich/myrinet/intel/bin/mpirun a.out > > > > > > I wonder if the problem lies here... > > > > > > > > > > > > Thank you. > > > > > > > > > > > > On 1/16/07, li pan wrote: > > > > > > > > I did try to download and install petsc2.3.2, but end > > > > up with error: mpich can not be download & installed, > > > > please install mpich for windows manually. > > > > In the homepage of mpich2, I didn't choose the version > > > > for windows but the source code version. And compiled > > > > it by myself. Then, I gave the --with-mpi-dir="install > > > > dir". Petsc was configured, and now it's doing "make". > > > > One interesting thing is, I installed petsc in linux > > > > before. The mpi libraries were very large > > > > (libmpich.a==60 mb). But this time in cygwin it was > > > > only several mbs. > > > > > > > > best > > > > > > > > pan > > > > > > > > > > > > --- Ben Tay wrote: > > > > > > > > > hi, > > > > > > > > > > i install PETSc using the following command: > > > > > > > > > > ./config/configure.py --with-vendor-compilers=intel > > > > > --with-gnu-compilers=0 > > > > > > > > > --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/lib/32 > > > > > --with-mpi-dir=/opt/mpich/intel/ --with-x=0 > > > > > --with-shared > > > > > > > > > > then i got: > > > > > > > > > > Compilers: > > > > > > > > > > C Compiler: /opt/mpich/intel/bin/mpicc > > > > > -fPIC -g > > > > > Fortran Compiler: /opt/mpich/intel/bin/mpif90 > > > > > -I. -fPIC -g -w90 -w > > > > > Linkers: > > > > > Shared linker: /opt/mpich/intel/bin/mpicc > > > > > -shared -fPIC -g > > > > > Dynamic linker: /opt/mpich/intel/bin/mpicc > > > > > -shared -fPIC -g > > > > > PETSc: > > > > > PETSC_ARCH: linux-mpif90 > > > > > PETSC_DIR: /nas/lsftmp/g0306332/petsc-2.3.2-p8 > > > > > ** > > > > > ** Now build and test the libraries with "make all > > > > > test" > > > > > ** > > > > > Clanguage: C > > > > > Scalar type:real > > > > > MPI: > > > > > Includes: ['/opt/mpich/intel/include'] > > > > > PETSc shared libraries: enabled > > > > > PETSc dynamic libraries: disabled > > > > > BLAS/LAPACK: > > > > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > > > > > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack > > > > > -lmkl_ia32 -lguide > > > > > > > > > > i ran "make all test" and everything seems fine > > > > > > > > > > /opt/mpich/intel/bin/mpicc -c -fPIC -g > > > > > > > > > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8-I/nas/lsftmp/g0306332/petsc- > > > > > 2.3.2-p8/bmake/linux-mpif90 > > > > > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8/include > > > > > -I/opt/mpich/intel/include > > > > > -D__SDIR__="src/snes/examples/tutorials/" ex19.c > > > > > /opt/mpich/intel/bin/mpicc -fPIC -g -o ex19 > > > > > ex19.o-Wl,-rpath,/nas/lsftmp/g0306332/petsc- > > > > > 2.3.2-p8/lib/linux-mpif90 > > > > > > > > > -L/nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpif90 > > > > > -lpetscsnes -lpetscksp -lpetscdm -lpetscmat > > > > > -lpetscvec -lpetsc > > > > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > > > > > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack > > > > > -lmkl_ia32 -lguide > > > > > -lPEPCF90 -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > -Wl,-rpath,/opt/mpich/intel/lib > > > > > -L/opt/mpich/intel/lib -Wl,-rpath,-rpath > > > > > -Wl,-rpath,-ldl -L-ldl -lmpich > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > -L/opt/intel/compiler70/ia32/lib > > > > > -Wl,-rpath,/usr/lib -Wl,-rpath,/usr/lib -L/usr/lib > > > > > -limf -lirc -lcprts -lcxa > > > > > -lunwind -ldl -lmpichf90 -lPEPCF90 > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > -L/opt/intel/compiler70/ia32/lib -Wl,-rpath,/usr/lib > > > > > -L/usr/lib -lintrins > > > > > -lIEPCF90 -lF90 -lm -Wl,-rpath,\ -Wl,-rpath,\ -L\ > > > > > -ldl -lmpich > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > -L/opt/intel/compiler70/ia32/lib > > > > > -Wl,-rpath,/usr/lib -L/usr/lib -limf -lirc -lcprts > > > > > -lcxa -lunwind -ldl > > > > > /bin/rm -f ex19.o > > > > > C/C++ example src/snes/examples/tutorials/ex19 run > > > > > successfully with 1 MPI > > > > > process > > > > > C/C++ example src/snes/examples/tutorials/ex19 run > > > > > successfully with 2 MPI > > > > > processes > > > > > Fortran example src/snes/examples/tutorials/ex5f run > > > > > successfully with 1 MPI > > > > > process > > > > > Completed test examples > > > > > > > > > > I then tried to run my own parallel code. It's a > > > > > simple code which prints > > > > > the rank of each processor. > > > > > > > > > > If I compile the code using just mpif90 test.F(using > > > > > just mpif.h) > > > > > > > > > > I get 0,1,2,3 (4 processors). > > > > > > > > > > however, if i change the code to use petsc.h etc > > > > > ie. > > > > > > > > > > > > > > > program ns2d_c > > > > > > > > > > implicit none > > > > > > > > > > > > > > > #include "include/finclude/petsc.h" > > > > > #include "include/finclude/petscvec.h" > > > > > #include "include/finclude/petscmat.h" > > > > > #include "include/finclude/petscksp.h" > > > > > #include "include/finclude/petscpc.h" > > > > > #include "include/finclude/petscmat.h90" > > > > > > > > > > integer,parameter :: size_x=8,size_y=4 > > > > > > > > > > integer :: > > > > > ierr,Istart_p,Iend_p,Ntot,Istart_m,Iend_m,k > > > > > > > > > > PetscMPIInt nprocs,rank > > > > > > > > > > > > > > > > > > > > > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > > > > > > > > > call > > > > > MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr) > > > > > > > > > > call > > > > > MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr) > > > > > > > > > > end program ns2d_c > > > > > > > > > > > > > > > > > > > > i then rename the filename to ex2f.F and use "make > > > > > ex2f" > > > > > > > > > > the result I get is something like 0,0,0,0. > > > > > > > > > > > > > > > > > > > > Why is this so? > > > > > > > > > > Thank you. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ____________________________________________________________________________________ > > > > Never Miss an Email > > > > Stay connected with Yahoo! Mail on your mobile. Get started! > > > > http://mobile.yahoo.com/services?promote=mail > > > > > > > > > > > > > > > > From zonexo at gmail.com Tue Jan 16 20:48:56 2007 From: zonexo at gmail.com (Ben Tay) Date: Wed, 17 Jan 2007 10:48:56 +0800 Subject: mpi not working In-Reply-To: References: <804ab5d40701150653r1e522500s551eaad8fea116d5@mail.gmail.com> <25804.18538.qm@web36802.mail.mud.yahoo.com> <804ab5d40701152356u6bf38de1u438fb28b19b0922c@mail.gmail.com> <804ab5d40701161634u58b77c1cr16daa8d72e28f8a7@mail.gmail.com> Message-ID: <804ab5d40701161848s13015c43vb553eae7ff3d0a51@mail.gmail.com> Hi, My school's server limit to minimum 4 processors. The output is Vector length 20 Vector length 20 Vector length 20 40 60 All other values should be near zero VecScale 0 VecCopy 0 VecAXPY 0 VecAYPX 0 VecSwap 0 VecSwap 0 VecWAXPY 0 VecPointwiseMult 0 VecPointwiseDivide 0 VecMAXPY 0 0 0 Vector length 20 40 60 All other values should be near zero VecScale 0 VecCopy 0 VecAXPY 0 VecAYPX 0 VecSwap 0 VecSwap 0 VecWAXPY 0 VecPointwiseMult 0 VecPointwiseDivide 0 VecMAXPY 0 0 0 Vector length 20 Vector length 20 40 60 All other values should be near zero Vector length 20 Vector length 20 40 60 All other values should be near zero VecScale 0 VecCopy 0 VecAXPY 0 VecAYPX 0 VecSwap 0 VecSwap 0 VecWAXPY 0 VecPointwiseMult 0 VecPointwiseDivide 0 VecMAXPY 0 0 0 VecScale 0 VecCopy 0 VecAXPY 0 VecAYPX 0 VecSwap 0 VecSwap 0 VecWAXPY 0 VecPointwiseMult 0 VecPointwiseDivide 0 VecMAXPY 0 0 0 So what's the verdict? Thank you. On 1/17/07, Barry Smith wrote: > > > Ben, > > I don't know what to say; what you report is inherently contradictory. > What happens when you run src/vec/vec/examples/tutorials/ex1.c on 2 > processors? > > Barry > > > On Wed, 17 Jan 2007, Ben Tay wrote: > > > Thanks Shaman. But the problem is that I get > > > > 0,1 > > 0,1 > > 0,1 > > 0,1 > > > > instead of > > > > 0,4 > > 1,4 > > 2,4 > > 3,4 which means there are 4 processors instead of the above, whereby it > > seems that 4 serial jobs are running. > > > > Barry: > > > > The script was given by my school when parallel jobs are to be > submitted. I > > use the same script when submitting pure mpi job and it works. On the > other > > hand, the PETSc parallel code ran successfully on another of my school's > > server. It was also submitted using a script, but a slightly different > one > > since it's another system. However, that server is very busy hence I > usually > > use the current server. > > > > Do you have other other solution? Or should I try other ways of > compilation? > > Btw, I am using ifc 7.0 and icc 7.0. The codes are written in fortran. > > > > Thank you. > > > > > > On 1/16/07, Barry Smith wrote: > > > > > > > > > Ben, > > > > > > You definitely have to submit a PETSc job just like > > > any other MPI job. So please try using the script. > > > > > > Barry > > > > > > > > > On Tue, 16 Jan 2007, Ben Tay wrote: > > > > > > > Hi Pan, > > > > > > > > I also got very big library files if I use PETSc with mpich2. > > > > > > > > Btw, I have tried several options but I still don't understand why I > > > can't > > > > get mpi to work with PETSc. > > > > > > > > The 4 processors are running together but each running its own code. > > > > > > > > I just use > > > > > > > > > > > > integer :: nprocs,rank,ierr > > > > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > > > > > > > call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr) > > > > > > > > call MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr) > > > > > > > > print *, rank, nprocs > > > > > > > > call PetscFinalize(ierr) > > > > > > > > > > > > > > > > The answers I get is 0,1 repeated 4 times instead of 0,4 1,4 2,4 > 3,4. > > > > > > > > I'm using my school's server's mpich and it work if I just compile > in > > > pure > > > > mpi. > > > > > > > > Btw, if I need to send the job to 4 processors, I need to use a > script > > > file: > > > > > > > > #BSUB -o std-output > > > > #BSUB -q linux_parallel_test > > > > #BSUB -n 4 > > > > /usr/lsf6/bin/mpijob_gm /opt/mpich/myrinet/intel/bin/mpirun a.out > > > > > > > > I wonder if the problem lies here... > > > > > > > > > > > > > > > > Thank you. > > > > > > > > > > > > > > > > On 1/16/07, li pan wrote: > > > > > > > > > > I did try to download and install petsc2.3.2, but end > > > > > up with error: mpich can not be download & installed, > > > > > please install mpich for windows manually. > > > > > In the homepage of mpich2, I didn't choose the version > > > > > for windows but the source code version. And compiled > > > > > it by myself. Then, I gave the --with-mpi-dir="install > > > > > dir". Petsc was configured, and now it's doing "make". > > > > > One interesting thing is, I installed petsc in linux > > > > > before. The mpi libraries were very large > > > > > (libmpich.a==60 mb). But this time in cygwin it was > > > > > only several mbs. > > > > > > > > > > best > > > > > > > > > > pan > > > > > > > > > > > > > > > --- Ben Tay wrote: > > > > > > > > > > > hi, > > > > > > > > > > > > i install PETSc using the following command: > > > > > > > > > > > > ./config/configure.py --with-vendor-compilers=intel > > > > > > --with-gnu-compilers=0 > > > > > > > > > > > --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/lib/32 > > > > > > --with-mpi-dir=/opt/mpich/intel/ --with-x=0 > > > > > > --with-shared > > > > > > > > > > > > then i got: > > > > > > > > > > > > Compilers: > > > > > > > > > > > > C Compiler: /opt/mpich/intel/bin/mpicc > > > > > > -fPIC -g > > > > > > Fortran Compiler: /opt/mpich/intel/bin/mpif90 > > > > > > -I. -fPIC -g -w90 -w > > > > > > Linkers: > > > > > > Shared linker: /opt/mpich/intel/bin/mpicc > > > > > > -shared -fPIC -g > > > > > > Dynamic linker: /opt/mpich/intel/bin/mpicc > > > > > > -shared -fPIC -g > > > > > > PETSc: > > > > > > PETSC_ARCH: linux-mpif90 > > > > > > PETSC_DIR: /nas/lsftmp/g0306332/petsc-2.3.2-p8 > > > > > > ** > > > > > > ** Now build and test the libraries with "make all > > > > > > test" > > > > > > ** > > > > > > Clanguage: C > > > > > > Scalar type:real > > > > > > MPI: > > > > > > Includes: ['/opt/mpich/intel/include'] > > > > > > PETSc shared libraries: enabled > > > > > > PETSc dynamic libraries: disabled > > > > > > BLAS/LAPACK: > > > > > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > > > > > > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack > > > > > > -lmkl_ia32 -lguide > > > > > > > > > > > > i ran "make all test" and everything seems fine > > > > > > > > > > > > /opt/mpich/intel/bin/mpicc -c -fPIC -g > > > > > > > > > > > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8-I/nas/lsftmp/g0306332/petsc- > > > > > > 2.3.2-p8/bmake/linux-mpif90 > > > > > > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8/include > > > > > > -I/opt/mpich/intel/include > > > > > > -D__SDIR__="src/snes/examples/tutorials/" ex19.c > > > > > > /opt/mpich/intel/bin/mpicc -fPIC -g -o ex19 > > > > > > ex19.o-Wl,-rpath,/nas/lsftmp/g0306332/petsc- > > > > > > 2.3.2-p8/lib/linux-mpif90 > > > > > > > > > > > -L/nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpif90 > > > > > > -lpetscsnes -lpetscksp -lpetscdm -lpetscmat > > > > > > -lpetscvec -lpetsc > > > > > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > > > > > > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack > > > > > > -lmkl_ia32 -lguide > > > > > > -lPEPCF90 -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > > -Wl,-rpath,/opt/mpich/intel/lib > > > > > > -L/opt/mpich/intel/lib -Wl,-rpath,-rpath > > > > > > -Wl,-rpath,-ldl -L-ldl -lmpich > > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > > -L/opt/intel/compiler70/ia32/lib > > > > > > -Wl,-rpath,/usr/lib -Wl,-rpath,/usr/lib -L/usr/lib > > > > > > -limf -lirc -lcprts -lcxa > > > > > > -lunwind -ldl -lmpichf90 -lPEPCF90 > > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > > -L/opt/intel/compiler70/ia32/lib -Wl,-rpath,/usr/lib > > > > > > -L/usr/lib -lintrins > > > > > > -lIEPCF90 -lF90 -lm -Wl,-rpath,\ -Wl,-rpath,\ -L\ > > > > > > -ldl -lmpich > > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > > -L/opt/intel/compiler70/ia32/lib > > > > > > -Wl,-rpath,/usr/lib -L/usr/lib -limf -lirc -lcprts > > > > > > -lcxa -lunwind -ldl > > > > > > /bin/rm -f ex19.o > > > > > > C/C++ example src/snes/examples/tutorials/ex19 run > > > > > > successfully with 1 MPI > > > > > > process > > > > > > C/C++ example src/snes/examples/tutorials/ex19 run > > > > > > successfully with 2 MPI > > > > > > processes > > > > > > Fortran example src/snes/examples/tutorials/ex5f run > > > > > > successfully with 1 MPI > > > > > > process > > > > > > Completed test examples > > > > > > > > > > > > I then tried to run my own parallel code. It's a > > > > > > simple code which prints > > > > > > the rank of each processor. > > > > > > > > > > > > If I compile the code using just mpif90 test.F(using > > > > > > just mpif.h) > > > > > > > > > > > > I get 0,1,2,3 (4 processors). > > > > > > > > > > > > however, if i change the code to use petsc.h etc > > > > > > ie. > > > > > > > > > > > > > > > > > > program ns2d_c > > > > > > > > > > > > implicit none > > > > > > > > > > > > > > > > > > #include "include/finclude/petsc.h" > > > > > > #include "include/finclude/petscvec.h" > > > > > > #include "include/finclude/petscmat.h" > > > > > > #include "include/finclude/petscksp.h" > > > > > > #include "include/finclude/petscpc.h" > > > > > > #include "include/finclude/petscmat.h90" > > > > > > > > > > > > integer,parameter :: size_x=8,size_y=4 > > > > > > > > > > > > integer :: > > > > > > ierr,Istart_p,Iend_p,Ntot,Istart_m,Iend_m,k > > > > > > > > > > > > PetscMPIInt nprocs,rank > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > > > > > > > > > > > call > > > > > > MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr) > > > > > > > > > > > > call > > > > > > MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr) > > > > > > > > > > > > end program ns2d_c > > > > > > > > > > > > > > > > > > > > > > > > i then rename the filename to ex2f.F and use "make > > > > > > ex2f" > > > > > > > > > > > > the result I get is something like 0,0,0,0. > > > > > > > > > > > > > > > > > > > > > > > > Why is this so? > > > > > > > > > > > > Thank you. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ____________________________________________________________________________________ > > > > > Never Miss an Email > > > > > Stay connected with Yahoo! Mail on your mobile. Get started! > > > > > http://mobile.yahoo.com/services?promote=mail > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yaronkretchmer at gmail.com Wed Jan 17 10:49:59 2007 From: yaronkretchmer at gmail.com (Yaron Kretchmer) Date: Wed, 17 Jan 2007 08:49:59 -0800 Subject: test Message-ID: From balay at mcs.anl.gov Wed Jan 17 12:11:13 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 17 Jan 2007 12:11:13 -0600 (CST) Subject: mpi not working In-Reply-To: <804ab5d40701161848s13015c43vb553eae7ff3d0a51@mail.gmail.com> References: <804ab5d40701150653r1e522500s551eaad8fea116d5@mail.gmail.com> <25804.18538.qm@web36802.mail.mud.yahoo.com> <804ab5d40701152356u6bf38de1u438fb28b19b0922c@mail.gmail.com> <804ab5d40701161634u58b77c1cr16daa8d72e28f8a7@mail.gmail.com> <804ab5d40701161848s13015c43vb553eae7ff3d0a51@mail.gmail.com> Message-ID: You've built PETSc with the following. > > > > > > > --with-mpi-dir=/opt/mpich/intel/ --with-x=0 However you are comparing simple MPI test with a different MPI [from /opt/mpich/myrinet/intel] > > > > > /usr/lsf6/bin/mpijob_gm /opt/mpich/myrinet/intel/bin/mpirun a.out You are using different MPI imps for each of these cases - hence the results are different. I guess you should be using --with-mpi-dir=/opt/mpich/myrinet/intel with PETSc configure. If you still encounter problems - please send us the COMPLETE info wrt the 2 tests. i.e compile, run , output , with the location of compilers used [which mpif90] Satish On Wed, 17 Jan 2007, Ben Tay wrote: > Hi, > > My school's server limit to minimum 4 processors. The output is > > Vector length 20 > Vector length 20 > Vector length 20 40 60 > All other values should be near zero > VecScale 0 > VecCopy 0 > VecAXPY 0 > VecAYPX 0 > VecSwap 0 > VecSwap 0 > VecWAXPY 0 > VecPointwiseMult 0 > VecPointwiseDivide 0 > VecMAXPY 0 0 0 > Vector length 20 40 60 > All other values should be near zero > VecScale 0 > VecCopy 0 > VecAXPY 0 > VecAYPX 0 > VecSwap 0 > VecSwap 0 > VecWAXPY 0 > VecPointwiseMult 0 > VecPointwiseDivide 0 > VecMAXPY 0 0 0 > Vector length 20 > Vector length 20 40 60 > All other values should be near zero > Vector length 20 > Vector length 20 40 60 > All other values should be near zero > VecScale 0 > VecCopy 0 > VecAXPY 0 > VecAYPX 0 > VecSwap 0 > VecSwap 0 > VecWAXPY 0 > VecPointwiseMult 0 > VecPointwiseDivide 0 > VecMAXPY 0 0 0 > VecScale 0 > VecCopy 0 > VecAXPY 0 > VecAYPX 0 > VecSwap 0 > VecSwap 0 > VecWAXPY 0 > VecPointwiseMult 0 > VecPointwiseDivide 0 > VecMAXPY 0 0 0 > > So what's the verdict? > > Thank you. > > > On 1/17/07, Barry Smith wrote: > > > > > > Ben, > > > > I don't know what to say; what you report is inherently contradictory. > > What happens when you run src/vec/vec/examples/tutorials/ex1.c on 2 > > processors? > > > > Barry > > > > > > On Wed, 17 Jan 2007, Ben Tay wrote: > > > > > Thanks Shaman. But the problem is that I get > > > > > > 0,1 > > > 0,1 > > > 0,1 > > > 0,1 > > > > > > instead of > > > > > > 0,4 > > > 1,4 > > > 2,4 > > > 3,4 which means there are 4 processors instead of the above, whereby it > > > seems that 4 serial jobs are running. > > > > > > Barry: > > > > > > The script was given by my school when parallel jobs are to be > > submitted. I > > > use the same script when submitting pure mpi job and it works. On the > > other > > > hand, the PETSc parallel code ran successfully on another of my school's > > > server. It was also submitted using a script, but a slightly different > > one > > > since it's another system. However, that server is very busy hence I > > usually > > > use the current server. > > > > > > Do you have other other solution? Or should I try other ways of > > compilation? > > > Btw, I am using ifc 7.0 and icc 7.0. The codes are written in fortran. > > > > > > Thank you. > > > > > > > > > On 1/16/07, Barry Smith wrote: > > > > > > > > > > > > Ben, > > > > > > > > You definitely have to submit a PETSc job just like > > > > any other MPI job. So please try using the script. > > > > > > > > Barry > > > > > > > > > > > > On Tue, 16 Jan 2007, Ben Tay wrote: > > > > > > > > > Hi Pan, > > > > > > > > > > I also got very big library files if I use PETSc with mpich2. > > > > > > > > > > Btw, I have tried several options but I still don't understand why I > > > > can't > > > > > get mpi to work with PETSc. > > > > > > > > > > The 4 processors are running together but each running its own code. > > > > > > > > > > I just use > > > > > > > > > > > > > > > integer :: nprocs,rank,ierr > > > > > > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > > > > > > > > > call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr) > > > > > > > > > > call MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr) > > > > > > > > > > print *, rank, nprocs > > > > > > > > > > call PetscFinalize(ierr) > > > > > > > > > > > > > > > > > > > > The answers I get is 0,1 repeated 4 times instead of 0,4 1,4 2,4 > > 3,4. > > > > > > > > > > I'm using my school's server's mpich and it work if I just compile > > in > > > > pure > > > > > mpi. > > > > > > > > > > Btw, if I need to send the job to 4 processors, I need to use a > > script > > > > file: > > > > > > > > > > #BSUB -o std-output > > > > > #BSUB -q linux_parallel_test > > > > > #BSUB -n 4 > > > > > /usr/lsf6/bin/mpijob_gm /opt/mpich/myrinet/intel/bin/mpirun a.out > > > > > > > > > > I wonder if the problem lies here... > > > > > > > > > > > > > > > > > > > > Thank you. > > > > > > > > > > > > > > > > > > > > On 1/16/07, li pan wrote: > > > > > > > > > > > > I did try to download and install petsc2.3.2, but end > > > > > > up with error: mpich can not be download & installed, > > > > > > please install mpich for windows manually. > > > > > > In the homepage of mpich2, I didn't choose the version > > > > > > for windows but the source code version. And compiled > > > > > > it by myself. Then, I gave the --with-mpi-dir="install > > > > > > dir". Petsc was configured, and now it's doing "make". > > > > > > One interesting thing is, I installed petsc in linux > > > > > > before. The mpi libraries were very large > > > > > > (libmpich.a==60 mb). But this time in cygwin it was > > > > > > only several mbs. > > > > > > > > > > > > best > > > > > > > > > > > > pan > > > > > > > > > > > > > > > > > > --- Ben Tay wrote: > > > > > > > > > > > > > hi, > > > > > > > > > > > > > > i install PETSc using the following command: > > > > > > > > > > > > > > ./config/configure.py --with-vendor-compilers=intel > > > > > > > --with-gnu-compilers=0 > > > > > > > > > > > > > --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/lib/32 > > > > > > > --with-mpi-dir=/opt/mpich/intel/ --with-x=0 > > > > > > > --with-shared > > > > > > > > > > > > > > then i got: > > > > > > > > > > > > > > Compilers: > > > > > > > > > > > > > > C Compiler: /opt/mpich/intel/bin/mpicc > > > > > > > -fPIC -g > > > > > > > Fortran Compiler: /opt/mpich/intel/bin/mpif90 > > > > > > > -I. -fPIC -g -w90 -w > > > > > > > Linkers: > > > > > > > Shared linker: /opt/mpich/intel/bin/mpicc > > > > > > > -shared -fPIC -g > > > > > > > Dynamic linker: /opt/mpich/intel/bin/mpicc > > > > > > > -shared -fPIC -g > > > > > > > PETSc: > > > > > > > PETSC_ARCH: linux-mpif90 > > > > > > > PETSC_DIR: /nas/lsftmp/g0306332/petsc-2.3.2-p8 > > > > > > > ** > > > > > > > ** Now build and test the libraries with "make all > > > > > > > test" > > > > > > > ** > > > > > > > Clanguage: C > > > > > > > Scalar type:real > > > > > > > MPI: > > > > > > > Includes: ['/opt/mpich/intel/include'] > > > > > > > PETSc shared libraries: enabled > > > > > > > PETSc dynamic libraries: disabled > > > > > > > BLAS/LAPACK: > > > > > > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > > > > > > > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack > > > > > > > -lmkl_ia32 -lguide > > > > > > > > > > > > > > i ran "make all test" and everything seems fine > > > > > > > > > > > > > > /opt/mpich/intel/bin/mpicc -c -fPIC -g > > > > > > > > > > > > > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8-I/nas/lsftmp/g0306332/petsc- > > > > > > > 2.3.2-p8/bmake/linux-mpif90 > > > > > > > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8/include > > > > > > > -I/opt/mpich/intel/include > > > > > > > -D__SDIR__="src/snes/examples/tutorials/" ex19.c > > > > > > > /opt/mpich/intel/bin/mpicc -fPIC -g -o ex19 > > > > > > > ex19.o-Wl,-rpath,/nas/lsftmp/g0306332/petsc- > > > > > > > 2.3.2-p8/lib/linux-mpif90 > > > > > > > > > > > > > -L/nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpif90 > > > > > > > -lpetscsnes -lpetscksp -lpetscdm -lpetscmat > > > > > > > -lpetscvec -lpetsc > > > > > > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > > > > > > > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack > > > > > > > -lmkl_ia32 -lguide > > > > > > > -lPEPCF90 -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > > > -Wl,-rpath,/opt/mpich/intel/lib > > > > > > > -L/opt/mpich/intel/lib -Wl,-rpath,-rpath > > > > > > > -Wl,-rpath,-ldl -L-ldl -lmpich > > > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > > > -L/opt/intel/compiler70/ia32/lib > > > > > > > -Wl,-rpath,/usr/lib -Wl,-rpath,/usr/lib -L/usr/lib > > > > > > > -limf -lirc -lcprts -lcxa > > > > > > > -lunwind -ldl -lmpichf90 -lPEPCF90 > > > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > > > -L/opt/intel/compiler70/ia32/lib -Wl,-rpath,/usr/lib > > > > > > > -L/usr/lib -lintrins > > > > > > > -lIEPCF90 -lF90 -lm -Wl,-rpath,\ -Wl,-rpath,\ -L\ > > > > > > > -ldl -lmpich > > > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > > > -L/opt/intel/compiler70/ia32/lib > > > > > > > -Wl,-rpath,/usr/lib -L/usr/lib -limf -lirc -lcprts > > > > > > > -lcxa -lunwind -ldl > > > > > > > /bin/rm -f ex19.o > > > > > > > C/C++ example src/snes/examples/tutorials/ex19 run > > > > > > > successfully with 1 MPI > > > > > > > process > > > > > > > C/C++ example src/snes/examples/tutorials/ex19 run > > > > > > > successfully with 2 MPI > > > > > > > processes > > > > > > > Fortran example src/snes/examples/tutorials/ex5f run > > > > > > > successfully with 1 MPI > > > > > > > process > > > > > > > Completed test examples > > > > > > > > > > > > > > I then tried to run my own parallel code. It's a > > > > > > > simple code which prints > > > > > > > the rank of each processor. > > > > > > > > > > > > > > If I compile the code using just mpif90 test.F(using > > > > > > > just mpif.h) > > > > > > > > > > > > > > I get 0,1,2,3 (4 processors). > > > > > > > > > > > > > > however, if i change the code to use petsc.h etc > > > > > > > ie. > > > > > > > > > > > > > > > > > > > > > program ns2d_c > > > > > > > > > > > > > > implicit none > > > > > > > > > > > > > > > > > > > > > #include "include/finclude/petsc.h" > > > > > > > #include "include/finclude/petscvec.h" > > > > > > > #include "include/finclude/petscmat.h" > > > > > > > #include "include/finclude/petscksp.h" > > > > > > > #include "include/finclude/petscpc.h" > > > > > > > #include "include/finclude/petscmat.h90" > > > > > > > > > > > > > > integer,parameter :: size_x=8,size_y=4 > > > > > > > > > > > > > > integer :: > > > > > > > ierr,Istart_p,Iend_p,Ntot,Istart_m,Iend_m,k > > > > > > > > > > > > > > PetscMPIInt nprocs,rank > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > > > > > > > > > > > > > call > > > > > > > MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr) > > > > > > > > > > > > > > call > > > > > > > MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr) > > > > > > > > > > > > > > end program ns2d_c > > > > > > > > > > > > > > > > > > > > > > > > > > > > i then rename the filename to ex2f.F and use "make > > > > > > > ex2f" > > > > > > > > > > > > > > the result I get is something like 0,0,0,0. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Why is this so? > > > > > > > > > > > > > > Thank you. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ____________________________________________________________________________________ > > > > > > Never Miss an Email > > > > > > Stay connected with Yahoo! Mail on your mobile. Get started! > > > > > > http://mobile.yahoo.com/services?promote=mail > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From balay at mcs.anl.gov Wed Jan 17 12:15:23 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 17 Jan 2007 12:15:23 -0600 (CST) Subject: mpi not working In-Reply-To: References: <804ab5d40701150653r1e522500s551eaad8fea116d5@mail.gmail.com> <25804.18538.qm@web36802.mail.mud.yahoo.com> <804ab5d40701152356u6bf38de1u438fb28b19b0922c@mail.gmail.com> <804ab5d40701161634u58b77c1cr16daa8d72e28f8a7@mail.gmail.com> <804ab5d40701161848s13015c43vb553eae7ff3d0a51@mail.gmail.com> Message-ID: It could be possible that both installs of MPI are functional [however myrinet version should performan better than the regular version]. And the problem could be - you are using the wrong mpirun with PETSc binaries.. Satish On Wed, 17 Jan 2007, Satish Balay wrote: > You've built PETSc with the following. > > > > > > > > > --with-mpi-dir=/opt/mpich/intel/ --with-x=0 > > However you are comparing simple MPI test with a different MPI [from > /opt/mpich/myrinet/intel] > > > > > > > /usr/lsf6/bin/mpijob_gm /opt/mpich/myrinet/intel/bin/mpirun a.out > > You are using different MPI imps for each of these cases - hence the > results are different. > > I guess you should be using --with-mpi-dir=/opt/mpich/myrinet/intel > with PETSc configure. > > If you still encounter problems - please send us the COMPLETE info wrt > the 2 tests. > > i.e compile, run , output , with the location of compilers used [which mpif90] > > Satish > > On Wed, 17 Jan 2007, Ben Tay wrote: > > > Hi, > > > > My school's server limit to minimum 4 processors. The output is > > > > Vector length 20 > > Vector length 20 > > Vector length 20 40 60 > > All other values should be near zero > > VecScale 0 > > VecCopy 0 > > VecAXPY 0 > > VecAYPX 0 > > VecSwap 0 > > VecSwap 0 > > VecWAXPY 0 > > VecPointwiseMult 0 > > VecPointwiseDivide 0 > > VecMAXPY 0 0 0 > > Vector length 20 40 60 > > All other values should be near zero > > VecScale 0 > > VecCopy 0 > > VecAXPY 0 > > VecAYPX 0 > > VecSwap 0 > > VecSwap 0 > > VecWAXPY 0 > > VecPointwiseMult 0 > > VecPointwiseDivide 0 > > VecMAXPY 0 0 0 > > Vector length 20 > > Vector length 20 40 60 > > All other values should be near zero > > Vector length 20 > > Vector length 20 40 60 > > All other values should be near zero > > VecScale 0 > > VecCopy 0 > > VecAXPY 0 > > VecAYPX 0 > > VecSwap 0 > > VecSwap 0 > > VecWAXPY 0 > > VecPointwiseMult 0 > > VecPointwiseDivide 0 > > VecMAXPY 0 0 0 > > VecScale 0 > > VecCopy 0 > > VecAXPY 0 > > VecAYPX 0 > > VecSwap 0 > > VecSwap 0 > > VecWAXPY 0 > > VecPointwiseMult 0 > > VecPointwiseDivide 0 > > VecMAXPY 0 0 0 > > > > So what's the verdict? > > > > Thank you. > > > > > > On 1/17/07, Barry Smith wrote: > > > > > > > > > Ben, > > > > > > I don't know what to say; what you report is inherently contradictory. > > > What happens when you run src/vec/vec/examples/tutorials/ex1.c on 2 > > > processors? > > > > > > Barry > > > > > > > > > On Wed, 17 Jan 2007, Ben Tay wrote: > > > > > > > Thanks Shaman. But the problem is that I get > > > > > > > > 0,1 > > > > 0,1 > > > > 0,1 > > > > 0,1 > > > > > > > > instead of > > > > > > > > 0,4 > > > > 1,4 > > > > 2,4 > > > > 3,4 which means there are 4 processors instead of the above, whereby it > > > > seems that 4 serial jobs are running. > > > > > > > > Barry: > > > > > > > > The script was given by my school when parallel jobs are to be > > > submitted. I > > > > use the same script when submitting pure mpi job and it works. On the > > > other > > > > hand, the PETSc parallel code ran successfully on another of my school's > > > > server. It was also submitted using a script, but a slightly different > > > one > > > > since it's another system. However, that server is very busy hence I > > > usually > > > > use the current server. > > > > > > > > Do you have other other solution? Or should I try other ways of > > > compilation? > > > > Btw, I am using ifc 7.0 and icc 7.0. The codes are written in fortran. > > > > > > > > Thank you. > > > > > > > > > > > > On 1/16/07, Barry Smith wrote: > > > > > > > > > > > > > > > Ben, > > > > > > > > > > You definitely have to submit a PETSc job just like > > > > > any other MPI job. So please try using the script. > > > > > > > > > > Barry > > > > > > > > > > > > > > > On Tue, 16 Jan 2007, Ben Tay wrote: > > > > > > > > > > > Hi Pan, > > > > > > > > > > > > I also got very big library files if I use PETSc with mpich2. > > > > > > > > > > > > Btw, I have tried several options but I still don't understand why I > > > > > can't > > > > > > get mpi to work with PETSc. > > > > > > > > > > > > The 4 processors are running together but each running its own code. > > > > > > > > > > > > I just use > > > > > > > > > > > > > > > > > > integer :: nprocs,rank,ierr > > > > > > > > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > > > > > > > > > > > call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr) > > > > > > > > > > > > call MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr) > > > > > > > > > > > > print *, rank, nprocs > > > > > > > > > > > > call PetscFinalize(ierr) > > > > > > > > > > > > > > > > > > > > > > > > The answers I get is 0,1 repeated 4 times instead of 0,4 1,4 2,4 > > > 3,4. > > > > > > > > > > > > I'm using my school's server's mpich and it work if I just compile > > > in > > > > > pure > > > > > > mpi. > > > > > > > > > > > > Btw, if I need to send the job to 4 processors, I need to use a > > > script > > > > > file: > > > > > > > > > > > > #BSUB -o std-output > > > > > > #BSUB -q linux_parallel_test > > > > > > #BSUB -n 4 > > > > > > /usr/lsf6/bin/mpijob_gm /opt/mpich/myrinet/intel/bin/mpirun a.out > > > > > > > > > > > > I wonder if the problem lies here... > > > > > > > > > > > > > > > > > > > > > > > > Thank you. > > > > > > > > > > > > > > > > > > > > > > > > On 1/16/07, li pan wrote: > > > > > > > > > > > > > > I did try to download and install petsc2.3.2, but end > > > > > > > up with error: mpich can not be download & installed, > > > > > > > please install mpich for windows manually. > > > > > > > In the homepage of mpich2, I didn't choose the version > > > > > > > for windows but the source code version. And compiled > > > > > > > it by myself. Then, I gave the --with-mpi-dir="install > > > > > > > dir". Petsc was configured, and now it's doing "make". > > > > > > > One interesting thing is, I installed petsc in linux > > > > > > > before. The mpi libraries were very large > > > > > > > (libmpich.a==60 mb). But this time in cygwin it was > > > > > > > only several mbs. > > > > > > > > > > > > > > best > > > > > > > > > > > > > > pan > > > > > > > > > > > > > > > > > > > > > --- Ben Tay wrote: > > > > > > > > > > > > > > > hi, > > > > > > > > > > > > > > > > i install PETSc using the following command: > > > > > > > > > > > > > > > > ./config/configure.py --with-vendor-compilers=intel > > > > > > > > --with-gnu-compilers=0 > > > > > > > > > > > > > > > --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/lib/32 > > > > > > > > --with-mpi-dir=/opt/mpich/intel/ --with-x=0 > > > > > > > > --with-shared > > > > > > > > > > > > > > > > then i got: > > > > > > > > > > > > > > > > Compilers: > > > > > > > > > > > > > > > > C Compiler: /opt/mpich/intel/bin/mpicc > > > > > > > > -fPIC -g > > > > > > > > Fortran Compiler: /opt/mpich/intel/bin/mpif90 > > > > > > > > -I. -fPIC -g -w90 -w > > > > > > > > Linkers: > > > > > > > > Shared linker: /opt/mpich/intel/bin/mpicc > > > > > > > > -shared -fPIC -g > > > > > > > > Dynamic linker: /opt/mpich/intel/bin/mpicc > > > > > > > > -shared -fPIC -g > > > > > > > > PETSc: > > > > > > > > PETSC_ARCH: linux-mpif90 > > > > > > > > PETSC_DIR: /nas/lsftmp/g0306332/petsc-2.3.2-p8 > > > > > > > > ** > > > > > > > > ** Now build and test the libraries with "make all > > > > > > > > test" > > > > > > > > ** > > > > > > > > Clanguage: C > > > > > > > > Scalar type:real > > > > > > > > MPI: > > > > > > > > Includes: ['/opt/mpich/intel/include'] > > > > > > > > PETSc shared libraries: enabled > > > > > > > > PETSc dynamic libraries: disabled > > > > > > > > BLAS/LAPACK: > > > > > > > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > > > > > > > > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack > > > > > > > > -lmkl_ia32 -lguide > > > > > > > > > > > > > > > > i ran "make all test" and everything seems fine > > > > > > > > > > > > > > > > /opt/mpich/intel/bin/mpicc -c -fPIC -g > > > > > > > > > > > > > > > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8-I/nas/lsftmp/g0306332/petsc- > > > > > > > > 2.3.2-p8/bmake/linux-mpif90 > > > > > > > > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8/include > > > > > > > > -I/opt/mpich/intel/include > > > > > > > > -D__SDIR__="src/snes/examples/tutorials/" ex19.c > > > > > > > > /opt/mpich/intel/bin/mpicc -fPIC -g -o ex19 > > > > > > > > ex19.o-Wl,-rpath,/nas/lsftmp/g0306332/petsc- > > > > > > > > 2.3.2-p8/lib/linux-mpif90 > > > > > > > > > > > > > > > -L/nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpif90 > > > > > > > > -lpetscsnes -lpetscksp -lpetscdm -lpetscmat > > > > > > > > -lpetscvec -lpetsc > > > > > > > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > > > > > > > > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack > > > > > > > > -lmkl_ia32 -lguide > > > > > > > > -lPEPCF90 -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > > > > -Wl,-rpath,/opt/mpich/intel/lib > > > > > > > > -L/opt/mpich/intel/lib -Wl,-rpath,-rpath > > > > > > > > -Wl,-rpath,-ldl -L-ldl -lmpich > > > > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > > > > -L/opt/intel/compiler70/ia32/lib > > > > > > > > -Wl,-rpath,/usr/lib -Wl,-rpath,/usr/lib -L/usr/lib > > > > > > > > -limf -lirc -lcprts -lcxa > > > > > > > > -lunwind -ldl -lmpichf90 -lPEPCF90 > > > > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > > > > -L/opt/intel/compiler70/ia32/lib -Wl,-rpath,/usr/lib > > > > > > > > -L/usr/lib -lintrins > > > > > > > > -lIEPCF90 -lF90 -lm -Wl,-rpath,\ -Wl,-rpath,\ -L\ > > > > > > > > -ldl -lmpich > > > > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > > > > -L/opt/intel/compiler70/ia32/lib > > > > > > > > -Wl,-rpath,/usr/lib -L/usr/lib -limf -lirc -lcprts > > > > > > > > -lcxa -lunwind -ldl > > > > > > > > /bin/rm -f ex19.o > > > > > > > > C/C++ example src/snes/examples/tutorials/ex19 run > > > > > > > > successfully with 1 MPI > > > > > > > > process > > > > > > > > C/C++ example src/snes/examples/tutorials/ex19 run > > > > > > > > successfully with 2 MPI > > > > > > > > processes > > > > > > > > Fortran example src/snes/examples/tutorials/ex5f run > > > > > > > > successfully with 1 MPI > > > > > > > > process > > > > > > > > Completed test examples > > > > > > > > > > > > > > > > I then tried to run my own parallel code. It's a > > > > > > > > simple code which prints > > > > > > > > the rank of each processor. > > > > > > > > > > > > > > > > If I compile the code using just mpif90 test.F(using > > > > > > > > just mpif.h) > > > > > > > > > > > > > > > > I get 0,1,2,3 (4 processors). > > > > > > > > > > > > > > > > however, if i change the code to use petsc.h etc > > > > > > > > ie. > > > > > > > > > > > > > > > > > > > > > > > > program ns2d_c > > > > > > > > > > > > > > > > implicit none > > > > > > > > > > > > > > > > > > > > > > > > #include "include/finclude/petsc.h" > > > > > > > > #include "include/finclude/petscvec.h" > > > > > > > > #include "include/finclude/petscmat.h" > > > > > > > > #include "include/finclude/petscksp.h" > > > > > > > > #include "include/finclude/petscpc.h" > > > > > > > > #include "include/finclude/petscmat.h90" > > > > > > > > > > > > > > > > integer,parameter :: size_x=8,size_y=4 > > > > > > > > > > > > > > > > integer :: > > > > > > > > ierr,Istart_p,Iend_p,Ntot,Istart_m,Iend_m,k > > > > > > > > > > > > > > > > PetscMPIInt nprocs,rank > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > > > > > > > > > > > > > > > call > > > > > > > > MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr) > > > > > > > > > > > > > > > > call > > > > > > > > MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr) > > > > > > > > > > > > > > > > end program ns2d_c > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > i then rename the filename to ex2f.F and use "make > > > > > > > > ex2f" > > > > > > > > > > > > > > > > the result I get is something like 0,0,0,0. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Why is this so? > > > > > > > > > > > > > > > > Thank you. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ____________________________________________________________________________________ > > > > > > > Never Miss an Email > > > > > > > Stay connected with Yahoo! Mail on your mobile. Get started! > > > > > > > http://mobile.yahoo.com/services?promote=mail > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From yaron at oak-research.com Wed Jan 17 15:55:29 2007 From: yaron at oak-research.com (yaron at oak-research.com) Date: Wed, 17 Jan 2007 13:55:29 -0800 Subject: Petsc users in UC Santa Cruz? Message-ID: <20070117215529.17403.qmail@s402.sureserver.com> Hi Petsc-users I'm looking for Petsc users in UCSC (UC Santa Cruz)- if you're one, and contact me off-list, I'd appreciate it Thanks Yaron From zonexo at gmail.com Thu Jan 18 09:32:55 2007 From: zonexo at gmail.com (Ben Tay) Date: Thu, 18 Jan 2007 23:32:55 +0800 Subject: Visual Studio compiler and PETSc In-Reply-To: <380-220071411211325745@westnet.com.au> References: <380-220071411211325745@westnet.com.au> Message-ID: <804ab5d40701180732t24bdc6b4k659bb53c4beeb1b7@mail.gmail.com> Hi, I can't get it to work with mpich2 (pre-built) although someone else told me he got mpich2 working with PETSc. however his mpich2 was compiled from source. on the other hand, i got mpich1 (pre-built) to work and built with PETSc. hope it helps. On 1/12/07, Tan Meng YUE wrote: > > Hi, > > I have tried out PETSc on OS X and Linux in the past. > > My current place of work is a Windows only environment with a small > (less than 1000) cluster of PCs/blade. > > I like to demostrate PETSc with MPICH2 on Windows to the developers > here working on some fluid simulation code for digital visual effects. > > Are there any deployable binaries for PETSc (we only use C++, no > Fortran)? > > I have tried compiling PETSc with Cygwin but was getting into > various difficult like BLAS/LAPACK and when that was fixed, other > problems cropped up. > > I have already installed MPICH2. > > Regards > > Cheers > -- > http://www.proceduralinsight.com/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From berend at chalmers.se Thu Jan 18 15:28:44 2007 From: berend at chalmers.se (Berend van Wachem) Date: Thu, 18 Jan 2007 22:28:44 +0100 Subject: Vector containing one number Message-ID: <200701182228.45229.berend@chalmers.se> Hi, In one of the input/output routines, I want to write one number to a viewer. I created a vector with global size one, wrote the number to the vector, and then viewed the vector. In the latest version of Petsc, this does not seem to work anymore. What do you suggest as a work around? Hereby an example code which crashes on more than 1 processor: static char help[] = "Just a simple test"; #undef __FUNCT__ #define __FUNCT__ "main" int main(int argc,char **args) { Vec Onevec; PetscInt size,ierr; PetscScalar var = 3.0; PetscInitialize(&argc,&args,(char *)0,help); ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size);CHKERRQ(ierr); ierr = VecCreateMPI(PETSC_COMM_WORLD, PETSC_DECIDE, 1, &Onevec);CHKERRQ(ierr); ierr = VecSetValue(Onevec, 0, var, INSERT_VALUES); CHKERRQ(ierr); ierr = VecAssemblyBegin(Onevec); CHKERRQ(ierr); ierr = VecAssemblyEnd(Onevec); CHKERRQ(ierr); VecDestroy(Onevec); ierr = PetscFinalize();CHKERRQ(ierr); return 0; } Thanks, Berend. From knepley at gmail.com Thu Jan 18 17:12:04 2007 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 18 Jan 2007 17:12:04 -0600 Subject: Vector containing one number In-Reply-To: <200701182228.45229.berend@chalmers.se> References: <200701182228.45229.berend@chalmers.se> Message-ID: Runs fine for me: knepley at knepley-laptop:/PETSc3/petsc/petsc-dev/src/vec/vec/examples/tutorials$ $MPIRUN -np 4 ./ex1001 -vec_view Process [0] 3 Process [1] Process [2] Process [3] Matt On 1/18/07, Berend van Wachem wrote: > Hi, > > In one of the input/output routines, I want to write one number to a viewer. I > created a vector with global size one, wrote the number to the vector, and > then viewed the vector. In the latest version of Petsc, this does not seem to > work anymore. What do you suggest as a work around? Hereby an example code > which crashes on more than 1 processor: > > static char help[] = "Just a simple test"; > #undef __FUNCT__ > #define __FUNCT__ "main" > int main(int argc,char **args) > { > Vec Onevec; > PetscInt size,ierr; > PetscScalar var = 3.0; > > PetscInitialize(&argc,&args,(char *)0,help); > ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size);CHKERRQ(ierr); > ierr = VecCreateMPI(PETSC_COMM_WORLD, PETSC_DECIDE, 1, > &Onevec);CHKERRQ(ierr); > ierr = VecSetValue(Onevec, 0, var, INSERT_VALUES); > CHKERRQ(ierr); > ierr = VecAssemblyBegin(Onevec); > CHKERRQ(ierr); > ierr = VecAssemblyEnd(Onevec); > CHKERRQ(ierr); > > VecDestroy(Onevec); > ierr = PetscFinalize();CHKERRQ(ierr); > return 0; > } > > Thanks, > > Berend. > > -- One trouble is that despite this system, anyone who reads journals widely and critically is forced to realize that there are scarcely any bars to eventual publication. There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print. -- Drummond Rennie From zonexo at gmail.com Thu Jan 18 21:45:28 2007 From: zonexo at gmail.com (Ben Tay) Date: Fri, 19 Jan 2007 11:45:28 +0800 Subject: mpi not working In-Reply-To: References: <804ab5d40701150653r1e522500s551eaad8fea116d5@mail.gmail.com> <25804.18538.qm@web36802.mail.mud.yahoo.com> <804ab5d40701152356u6bf38de1u438fb28b19b0922c@mail.gmail.com> <804ab5d40701161634u58b77c1cr16daa8d72e28f8a7@mail.gmail.com> <804ab5d40701161848s13015c43vb553eae7ff3d0a51@mail.gmail.com> Message-ID: <804ab5d40701181945k29a451c7l95910006d559a83f@mail.gmail.com> Thanks Satish! I'm really careless. My school has got the myrinet and intel mpi and I'm using the wrong one. I've got it working now. Thanks again! On 1/18/07, Satish Balay wrote: > > You've built PETSc with the following. > > > > > > > > > --with-mpi-dir=/opt/mpich/intel/ --with-x=0 > > However you are comparing simple MPI test with a different MPI [from > /opt/mpich/myrinet/intel] > > > > > > > /usr/lsf6/bin/mpijob_gm /opt/mpich/myrinet/intel/bin/mpirun > a.out > > You are using different MPI imps for each of these cases - hence the > results are different. > > I guess you should be using --with-mpi-dir=/opt/mpich/myrinet/intel > with PETSc configure. > > If you still encounter problems - please send us the COMPLETE info wrt > the 2 tests. > > i.e compile, run , output , with the location of compilers used [which > mpif90] > > Satish > > On Wed, 17 Jan 2007, Ben Tay wrote: > > > Hi, > > > > My school's server limit to minimum 4 processors. The output is > > > > Vector length 20 > > Vector length 20 > > Vector length 20 40 60 > > All other values should be near zero > > VecScale 0 > > VecCopy 0 > > VecAXPY 0 > > VecAYPX 0 > > VecSwap 0 > > VecSwap 0 > > VecWAXPY 0 > > VecPointwiseMult 0 > > VecPointwiseDivide 0 > > VecMAXPY 0 0 0 > > Vector length 20 40 60 > > All other values should be near zero > > VecScale 0 > > VecCopy 0 > > VecAXPY 0 > > VecAYPX 0 > > VecSwap 0 > > VecSwap 0 > > VecWAXPY 0 > > VecPointwiseMult 0 > > VecPointwiseDivide 0 > > VecMAXPY 0 0 0 > > Vector length 20 > > Vector length 20 40 60 > > All other values should be near zero > > Vector length 20 > > Vector length 20 40 60 > > All other values should be near zero > > VecScale 0 > > VecCopy 0 > > VecAXPY 0 > > VecAYPX 0 > > VecSwap 0 > > VecSwap 0 > > VecWAXPY 0 > > VecPointwiseMult 0 > > VecPointwiseDivide 0 > > VecMAXPY 0 0 0 > > VecScale 0 > > VecCopy 0 > > VecAXPY 0 > > VecAYPX 0 > > VecSwap 0 > > VecSwap 0 > > VecWAXPY 0 > > VecPointwiseMult 0 > > VecPointwiseDivide 0 > > VecMAXPY 0 0 0 > > > > So what's the verdict? > > > > Thank you. > > > > > > On 1/17/07, Barry Smith wrote: > > > > > > > > > Ben, > > > > > > I don't know what to say; what you report is inherently contradictory. > > > What happens when you run src/vec/vec/examples/tutorials/ex1.c on 2 > > > processors? > > > > > > Barry > > > > > > > > > On Wed, 17 Jan 2007, Ben Tay wrote: > > > > > > > Thanks Shaman. But the problem is that I get > > > > > > > > 0,1 > > > > 0,1 > > > > 0,1 > > > > 0,1 > > > > > > > > instead of > > > > > > > > 0,4 > > > > 1,4 > > > > 2,4 > > > > 3,4 which means there are 4 processors instead of the above, whereby > it > > > > seems that 4 serial jobs are running. > > > > > > > > Barry: > > > > > > > > The script was given by my school when parallel jobs are to be > > > submitted. I > > > > use the same script when submitting pure mpi job and it works. On > the > > > other > > > > hand, the PETSc parallel code ran successfully on another of my > school's > > > > server. It was also submitted using a script, but a slightly > different > > > one > > > > since it's another system. However, that server is very busy hence I > > > usually > > > > use the current server. > > > > > > > > Do you have other other solution? Or should I try other ways of > > > compilation? > > > > Btw, I am using ifc 7.0 and icc 7.0. The codes are written in > fortran. > > > > > > > > Thank you. > > > > > > > > > > > > On 1/16/07, Barry Smith wrote: > > > > > > > > > > > > > > > Ben, > > > > > > > > > > You definitely have to submit a PETSc job just like > > > > > any other MPI job. So please try using the script. > > > > > > > > > > Barry > > > > > > > > > > > > > > > On Tue, 16 Jan 2007, Ben Tay wrote: > > > > > > > > > > > Hi Pan, > > > > > > > > > > > > I also got very big library files if I use PETSc with mpich2. > > > > > > > > > > > > Btw, I have tried several options but I still don't understand > why I > > > > > can't > > > > > > get mpi to work with PETSc. > > > > > > > > > > > > The 4 processors are running together but each running its own > code. > > > > > > > > > > > > I just use > > > > > > > > > > > > > > > > > > integer :: nprocs,rank,ierr > > > > > > > > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > > > > > > > > > > > call MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr) > > > > > > > > > > > > call MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr) > > > > > > > > > > > > print *, rank, nprocs > > > > > > > > > > > > call PetscFinalize(ierr) > > > > > > > > > > > > > > > > > > > > > > > > The answers I get is 0,1 repeated 4 times instead of 0,4 1,4 2,4 > > > 3,4. > > > > > > > > > > > > I'm using my school's server's mpich and it work if I just > compile > > > in > > > > > pure > > > > > > mpi. > > > > > > > > > > > > Btw, if I need to send the job to 4 processors, I need to use a > > > script > > > > > file: > > > > > > > > > > > > #BSUB -o std-output > > > > > > #BSUB -q linux_parallel_test > > > > > > #BSUB -n 4 > > > > > > /usr/lsf6/bin/mpijob_gm /opt/mpich/myrinet/intel/bin/mpirun > a.out > > > > > > > > > > > > I wonder if the problem lies here... > > > > > > > > > > > > > > > > > > > > > > > > Thank you. > > > > > > > > > > > > > > > > > > > > > > > > On 1/16/07, li pan wrote: > > > > > > > > > > > > > > I did try to download and install petsc2.3.2, but end > > > > > > > up with error: mpich can not be download & installed, > > > > > > > please install mpich for windows manually. > > > > > > > In the homepage of mpich2, I didn't choose the version > > > > > > > for windows but the source code version. And compiled > > > > > > > it by myself. Then, I gave the --with-mpi-dir="install > > > > > > > dir". Petsc was configured, and now it's doing "make". > > > > > > > One interesting thing is, I installed petsc in linux > > > > > > > before. The mpi libraries were very large > > > > > > > (libmpich.a==60 mb). But this time in cygwin it was > > > > > > > only several mbs. > > > > > > > > > > > > > > best > > > > > > > > > > > > > > pan > > > > > > > > > > > > > > > > > > > > > --- Ben Tay wrote: > > > > > > > > > > > > > > > hi, > > > > > > > > > > > > > > > > i install PETSc using the following command: > > > > > > > > > > > > > > > > ./config/configure.py --with-vendor-compilers=intel > > > > > > > > --with-gnu-compilers=0 > > > > > > > > > > > > > > > --with-blas-lapack-dir=/lsftmp/g0306332/inter/mkl/lib/32 > > > > > > > > --with-mpi-dir=/opt/mpich/intel/ --with-x=0 > > > > > > > > --with-shared > > > > > > > > > > > > > > > > then i got: > > > > > > > > > > > > > > > > Compilers: > > > > > > > > > > > > > > > > C Compiler: /opt/mpich/intel/bin/mpicc > > > > > > > > -fPIC -g > > > > > > > > Fortran Compiler: /opt/mpich/intel/bin/mpif90 > > > > > > > > -I. -fPIC -g -w90 -w > > > > > > > > Linkers: > > > > > > > > Shared linker: /opt/mpich/intel/bin/mpicc > > > > > > > > -shared -fPIC -g > > > > > > > > Dynamic linker: /opt/mpich/intel/bin/mpicc > > > > > > > > -shared -fPIC -g > > > > > > > > PETSc: > > > > > > > > PETSC_ARCH: linux-mpif90 > > > > > > > > PETSC_DIR: /nas/lsftmp/g0306332/petsc-2.3.2-p8 > > > > > > > > ** > > > > > > > > ** Now build and test the libraries with "make all > > > > > > > > test" > > > > > > > > ** > > > > > > > > Clanguage: C > > > > > > > > Scalar type:real > > > > > > > > MPI: > > > > > > > > Includes: ['/opt/mpich/intel/include'] > > > > > > > > PETSc shared libraries: enabled > > > > > > > > PETSc dynamic libraries: disabled > > > > > > > > BLAS/LAPACK: > > > > > > > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > > > > > > > > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack > > > > > > > > -lmkl_ia32 -lguide > > > > > > > > > > > > > > > > i ran "make all test" and everything seems fine > > > > > > > > > > > > > > > > /opt/mpich/intel/bin/mpicc -c -fPIC -g > > > > > > > > > > > > > > > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8-I > /nas/lsftmp/g0306332/petsc- > > > > > > > > 2.3.2-p8/bmake/linux-mpif90 > > > > > > > > -I/nas/lsftmp/g0306332/petsc-2.3.2-p8/include > > > > > > > > -I/opt/mpich/intel/include > > > > > > > > -D__SDIR__="src/snes/examples/tutorials/" ex19.c > > > > > > > > /opt/mpich/intel/bin/mpicc -fPIC -g -o ex19 > > > > > > > > ex19.o-Wl,-rpath,/nas/lsftmp/g0306332/petsc- > > > > > > > > 2.3.2-p8/lib/linux-mpif90 > > > > > > > > > > > > > > > -L/nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-mpif90 > > > > > > > > -lpetscsnes -lpetscksp -lpetscdm -lpetscmat > > > > > > > > -lpetscvec -lpetsc > > > > > > > > -Wl,-rpath,/lsftmp/g0306332/inter/mkl/lib/32 > > > > > > > > -L/lsftmp/g0306332/inter/mkl/lib/32 -lmkl_lapack > > > > > > > > -lmkl_ia32 -lguide > > > > > > > > -lPEPCF90 -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > > > > -Wl,-rpath,/opt/mpich/intel/lib > > > > > > > > -L/opt/mpich/intel/lib -Wl,-rpath,-rpath > > > > > > > > -Wl,-rpath,-ldl -L-ldl -lmpich > > > > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > > > > -L/opt/intel/compiler70/ia32/lib > > > > > > > > -Wl,-rpath,/usr/lib -Wl,-rpath,/usr/lib -L/usr/lib > > > > > > > > -limf -lirc -lcprts -lcxa > > > > > > > > -lunwind -ldl -lmpichf90 -lPEPCF90 > > > > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > > > > -L/opt/intel/compiler70/ia32/lib -Wl,-rpath,/usr/lib > > > > > > > > -L/usr/lib -lintrins > > > > > > > > -lIEPCF90 -lF90 -lm -Wl,-rpath,\ -Wl,-rpath,\ -L\ > > > > > > > > -ldl -lmpich > > > > > > > > -Wl,-rpath,/opt/intel/compiler70/ia32/lib > > > > > > > > -L/opt/intel/compiler70/ia32/lib > > > > > > > > -Wl,-rpath,/usr/lib -L/usr/lib -limf -lirc -lcprts > > > > > > > > -lcxa -lunwind -ldl > > > > > > > > /bin/rm -f ex19.o > > > > > > > > C/C++ example src/snes/examples/tutorials/ex19 run > > > > > > > > successfully with 1 MPI > > > > > > > > process > > > > > > > > C/C++ example src/snes/examples/tutorials/ex19 run > > > > > > > > successfully with 2 MPI > > > > > > > > processes > > > > > > > > Fortran example src/snes/examples/tutorials/ex5f run > > > > > > > > successfully with 1 MPI > > > > > > > > process > > > > > > > > Completed test examples > > > > > > > > > > > > > > > > I then tried to run my own parallel code. It's a > > > > > > > > simple code which prints > > > > > > > > the rank of each processor. > > > > > > > > > > > > > > > > If I compile the code using just mpif90 test.F(using > > > > > > > > just mpif.h) > > > > > > > > > > > > > > > > I get 0,1,2,3 (4 processors). > > > > > > > > > > > > > > > > however, if i change the code to use petsc.h etc > > > > > > > > ie. > > > > > > > > > > > > > > > > > > > > > > > > program ns2d_c > > > > > > > > > > > > > > > > implicit none > > > > > > > > > > > > > > > > > > > > > > > > #include "include/finclude/petsc.h" > > > > > > > > #include "include/finclude/petscvec.h" > > > > > > > > #include "include/finclude/petscmat.h" > > > > > > > > #include "include/finclude/petscksp.h" > > > > > > > > #include "include/finclude/petscpc.h" > > > > > > > > #include "include/finclude/petscmat.h90" > > > > > > > > > > > > > > > > integer,parameter :: size_x=8,size_y=4 > > > > > > > > > > > > > > > > integer :: > > > > > > > > ierr,Istart_p,Iend_p,Ntot,Istart_m,Iend_m,k > > > > > > > > > > > > > > > > PetscMPIInt nprocs,rank > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > > > > > > > > > > > > > > > call > > > > > > > > MPI_Comm_rank(PETSC_COMM_WORLD,rank,ierr) > > > > > > > > > > > > > > > > call > > > > > > > > MPI_Comm_size(PETSC_COMM_WORLD,nprocs,ierr) > > > > > > > > > > > > > > > > end program ns2d_c > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > i then rename the filename to ex2f.F and use "make > > > > > > > > ex2f" > > > > > > > > > > > > > > > > the result I get is something like 0,0,0,0. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Why is this so? > > > > > > > > > > > > > > > > Thank you. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ____________________________________________________________________________________ > > > > > > > Never Miss an Email > > > > > > > Stay connected with Yahoo! Mail on your mobile. Get started! > > > > > > > http://mobile.yahoo.com/services?promote=mail > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From berend at chalmers.se Fri Jan 19 02:59:59 2007 From: berend at chalmers.se (Berend van Wachem) Date: Fri, 19 Jan 2007 09:59:59 +0100 Subject: Vector containing one number In-Reply-To: References: <200701182228.45229.berend@chalmers.se> Message-ID: <200701190959.59457.berend@chalmers.se> Hi, I have problems running it if np>1. I am using open-mpi, is that the problem, you think? Other PETSc examples seem to be working fine on multiple processors. Thanks Berend. $ mpirun -np 2 ex1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC ERROR: or try http://valgrind.org on linux or man libgmalloc on Apple to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 2.3.2, Patch 3, Fri Sep 29 17:09:34 CDT 2006 HG revision: 9215af156a9cbcdc1ec666e2b5c7934688ddc526 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ > Runs fine for me: > > knepley at knepley-laptop:/PETSc3/petsc/petsc-dev/src/vec/vec/examples/ >tutorials$ $MPIRUN -np 4 ./ex1001 -vec_view > Process [0] > 3 > Process [1] > Process [2] > Process [3] > > Matt > > On 1/18/07, Berend van Wachem wrote: > > Hi, > > > > In one of the input/output routines, I want to write one number to > > a viewer. I created a vector with global size one, wrote the > > number to the vector, and then viewed the vector. In the latest > > version of Petsc, this does not seem to work anymore. What do you > > suggest as a work around? Hereby an example code which crashes on > > more than 1 processor: > > > > static char help[] = "Just a simple test"; > > #undef __FUNCT__ > > #define __FUNCT__ "main" > > int main(int argc,char **args) > > { > > Vec Onevec; > > PetscInt size,ierr; > > PetscScalar var = 3.0; > > > > PetscInitialize(&argc,&args,(char *)0,help); > > ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size);CHKERRQ(ierr); > > ierr = VecCreateMPI(PETSC_COMM_WORLD, PETSC_DECIDE, 1, > > &Onevec);CHKERRQ(ierr); > > ierr = VecSetValue(Onevec, 0, var, INSERT_VALUES); > > CHKERRQ(ierr); > > ierr = VecAssemblyBegin(Onevec); > > CHKERRQ(ierr); > > ierr = VecAssemblyEnd(Onevec); > > CHKERRQ(ierr); > > > > VecDestroy(Onevec); > > ierr = PetscFinalize();CHKERRQ(ierr); > > return 0; > > } > > > > Thanks, > > > > Berend. From berend at chalmers.se Fri Jan 19 06:42:23 2007 From: berend at chalmers.se (Berend van Wachem) Date: Fri, 19 Jan 2007 13:42:23 +0100 Subject: Vector containing one number In-Reply-To: References: <200701182228.45229.berend@chalmers.se> Message-ID: <200701191342.23206.berend@chalmers.se> Hi, valgrind shows something related to open-mpi, see below. I'll try mpich to see if the problem goes away. Thanks, Berend. ==3683== Syscall param writev(vector[...]) points to uninitialised byte(s) ==3683== at 0x543A607: writev (in /lib64/tls/libc-2.3.6.so) ==3683== by 0x5BE0AEA: mca_oob_tcp_msg_send_handler (in /usr/local/lib/openmp i/mca_oob_tcp.so) ==3683== by 0x5BE1827: mca_oob_tcp_peer_send (in /usr/local/lib/openmpi/mca_o ob_tcp.so) ==3683== by 0x5BE5B93: mca_oob_tcp_send_nb (in /usr/local/lib/openmpi/mca_oob _tcp.so) ==3683== by 0x4B47DBD: _orte_iof_base_frag_ack (in /usr/local/lib/liborte.so. 0.0.0) ==3683== by 0x4B47639: orte_iof_base_endpoint_forward (in /usr/local/lib/libo rte.so.0.0.0) ==3683== by 0x6D45538: orte_iof_svc_proxy_recv (in /usr/local/lib/openmpi/mca _iof_svc.so) ==3683== by 0x5BE137A: mca_oob_tcp_msg_recv_complete (in /usr/local/lib/openm pi/mca_oob_tcp.so) ==3683== by 0x5BE2D18: mca_oob_tcp_peer_recv_handler (in /usr/local/lib/openm pi/mca_oob_tcp.so) ==3683== by 0x4C7E466: opal_event_loop (in /usr/local/lib/libopal.so.0.0.0) ==3683== by 0x4C78A55: opal_progress (in /usr/local/lib/libopal.so.0.0.0) ==3683== by 0x4030AF: opal_condition_wait (condition.h:81) ==3683== Address 0x41CF348 is not stack'd, malloc'd or (recently) free'd From DOMI0002 at ntu.edu.sg Fri Jan 19 07:06:26 2007 From: DOMI0002 at ntu.edu.sg (#DOMINIC DENVER JOHN CHANDAR#) Date: Fri, 19 Jan 2007 21:06:26 +0800 Subject: Serial Petsc with SuperKU_dist Message-ID: Hi Is it possible to build the serial version of PETSc with SuperLU_DIST_2.0 ? or should i use the parallel version only ? Thanks, Dominic -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jan 19 07:48:10 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 19 Jan 2007 07:48:10 -0600 (CST) Subject: Serial Petsc with SuperKU_dist In-Reply-To: References: Message-ID: Not, Superlu DIST requires a real MPI implementation ( but, of course, can be be run on just one process if you want). But for sequential solves you can just use Superlu. Barry On Fri, 19 Jan 2007, #DOMINIC DENVER JOHN CHANDAR# wrote: > Hi > > Is it possible to build the serial version of PETSc with > SuperLU_DIST_2.0 ? or should i use the parallel version only ? > > Thanks, > Dominic > From bsmith at mcs.anl.gov Fri Jan 19 07:53:52 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 19 Jan 2007 07:53:52 -0600 (CST) Subject: Vector containing one number In-Reply-To: <200701191342.23206.berend@chalmers.se> References: <200701182228.45229.berend@chalmers.se> <200701191342.23206.berend@chalmers.se> Message-ID: This routine does make sends and receives of length zero; this is legal in MPI but possibly Open-MPI has bugs dealing with this. Barry On Fri, 19 Jan 2007, Berend van Wachem wrote: > Hi, > > valgrind shows something related to open-mpi, see below. I'll try mpich > to see if the problem goes away. > > Thanks, > Berend. > > > ==3683== Syscall param writev(vector[...]) points to uninitialised > byte(s) > ==3683== at 0x543A607: writev (in /lib64/tls/libc-2.3.6.so) > ==3683== by 0x5BE0AEA: mca_oob_tcp_msg_send_handler > (in /usr/local/lib/openmp > i/mca_oob_tcp.so) > ==3683== by 0x5BE1827: mca_oob_tcp_peer_send > (in /usr/local/lib/openmpi/mca_o > ob_tcp.so) > ==3683== by 0x5BE5B93: mca_oob_tcp_send_nb > (in /usr/local/lib/openmpi/mca_oob > _tcp.so) > ==3683== by 0x4B47DBD: _orte_iof_base_frag_ack > (in /usr/local/lib/liborte.so. > 0.0.0) > ==3683== by 0x4B47639: orte_iof_base_endpoint_forward > (in /usr/local/lib/libo > rte.so.0.0.0) > ==3683== by 0x6D45538: orte_iof_svc_proxy_recv > (in /usr/local/lib/openmpi/mca > _iof_svc.so) > ==3683== by 0x5BE137A: mca_oob_tcp_msg_recv_complete > (in /usr/local/lib/openm > pi/mca_oob_tcp.so) > ==3683== by 0x5BE2D18: mca_oob_tcp_peer_recv_handler > (in /usr/local/lib/openm > pi/mca_oob_tcp.so) > ==3683== by 0x4C7E466: opal_event_loop > (in /usr/local/lib/libopal.so.0.0.0) > ==3683== by 0x4C78A55: opal_progress > (in /usr/local/lib/libopal.so.0.0.0) > ==3683== by 0x4030AF: opal_condition_wait (condition.h:81) > ==3683== Address 0x41CF348 is not stack'd, malloc'd or (recently) > free'd > > From berend at chalmers.se Fri Jan 19 10:26:44 2007 From: berend at chalmers.se (Berend van Wachem) Date: Fri, 19 Jan 2007 17:26:44 +0100 Subject: Vector containing one number In-Reply-To: References: <200701182228.45229.berend@chalmers.se> <200701191342.23206.berend@chalmers.se> Message-ID: <200701191726.44613.berend@chalmers.se> Hi Barry, A small test here confirms your suspicion. I switched back to MPICH, which works without errors, but which has no support under eclipse unfortunately. Thanks, Berend. > This routine does make sends and receives of length zero; > this is legal in MPI but possibly Open-MPI has bugs dealing with > this. > > Barry > > On Fri, 19 Jan 2007, Berend van Wachem wrote: > > Hi, > > > > valgrind shows something related to open-mpi, see below. I'll try > > mpich to see if the problem goes away. > > > > Thanks, > > Berend. > > > > > > ==3683== Syscall param writev(vector[...]) points to uninitialised > > byte(s) > > ==3683== at 0x543A607: writev (in /lib64/tls/libc-2.3.6.so) > > ==3683== by 0x5BE0AEA: mca_oob_tcp_msg_send_handler > > (in /usr/local/lib/openmp > > i/mca_oob_tcp.so) > > ==3683== by 0x5BE1827: mca_oob_tcp_peer_send > > (in /usr/local/lib/openmpi/mca_o > > ob_tcp.so) > > ==3683== by 0x5BE5B93: mca_oob_tcp_send_nb > > (in /usr/local/lib/openmpi/mca_oob > > _tcp.so) > > ==3683== by 0x4B47DBD: _orte_iof_base_frag_ack > > (in /usr/local/lib/liborte.so. > > 0.0.0) > > ==3683== by 0x4B47639: orte_iof_base_endpoint_forward > > (in /usr/local/lib/libo > > rte.so.0.0.0) > > ==3683== by 0x6D45538: orte_iof_svc_proxy_recv > > (in /usr/local/lib/openmpi/mca > > _iof_svc.so) > > ==3683== by 0x5BE137A: mca_oob_tcp_msg_recv_complete > > (in /usr/local/lib/openm > > pi/mca_oob_tcp.so) > > ==3683== by 0x5BE2D18: mca_oob_tcp_peer_recv_handler > > (in /usr/local/lib/openm > > pi/mca_oob_tcp.so) > > ==3683== by 0x4C7E466: opal_event_loop > > (in /usr/local/lib/libopal.so.0.0.0) > > ==3683== by 0x4C78A55: opal_progress > > (in /usr/local/lib/libopal.so.0.0.0) > > ==3683== by 0x4030AF: opal_condition_wait (condition.h:81) > > ==3683== Address 0x41CF348 is not stack'd, malloc'd or (recently) > > free'd From DOMI0002 at ntu.edu.sg Fri Jan 19 10:49:54 2007 From: DOMI0002 at ntu.edu.sg (#DOMINIC DENVER JOHN CHANDAR#) Date: Sat, 20 Jan 2007 00:49:54 +0800 Subject: Serial Petsc .... But undefined reference to MPI_Init()? Message-ID: Hi, Ive installed the serial version of petsc couple of months back sucessfully. I did it again just now on Itanium-64 with the compiler options mpi=0. Just wondering, although libmpiuni.a is getting linked, i still get undefined ref to MPI_Init()? Any clues ? gcc -c -Wall -g3 -I/home/pdominic/petsc-2.2.1 -I/home/pdominic/petsc-2.2.1/bmake/linux-gnu-ia64-intel -I/home/pdominic/petsc-2.2.1/include -I/home/pdominic/petsc-2.2.1/include/mpiuni -DPETSC_USE_BOPT_g -DPETSC_USE_EXTERN_CXX -D__SDIR__="src/ksp/examples/tests/" ex2.c gcc -Wall -g3 -Wl,-rpath,/home/pdominic/petsc-2.2.1/lib/libg/linux-gnu-ia64-intel -o ex2 ex2.o -L/home/pdominic/petsc-2.2.1/lib/libg/linux-gnu-ia64-intel -lpetscksp -lpetscdm -lpetscmat -lpetscvec -lpetsc -L/usr/X11R6/lib -lX11 -Wl,-rpath,/usr/lib -L/usr/lib -llapack -Wl,-rpath,/usr/lib -L/usr/lib -lblas -Wl,-rpath,/usr/src/gcc-3.0.4-build2/lib/gcc-lib/ia64-unknown-linux/3.0. 4 -L/usr/src/gcc-3.0.4-build2/lib/gcc-lib/ia64-unknown-linux/3.0.4 -lg2c -L/home/pdominic/petsc-2.2.1/lib/libg/linux-gnu-ia64-intel -lmpiuni -ldl -ldl -ldl -ldl -lm -lg2c -lm -L/usr/local/lib/gcc-lib/ia64-unknown-linux/3.0.4 -L/usr/local/lib/gcc-lib/ia64-unknown-linux/3.0.4/../../.. -lm ex2.o: In function `main': /home/pdominic/petsc-2.2.1/src/ksp/examples/tests/ex2.c:31: undefined reference to `MPI_Init' /home/pdominic/petsc-2.2.1/src/ksp/examples/tests/ex2.c:33: undefined reference to `MPI_Finalize' collect2: ld returned 1 exit status make: [ex2] Error 1 (ignored) rm -f ex2.o Regards, Dominic -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jan 19 10:56:57 2007 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 19 Jan 2007 10:56:57 -0600 Subject: Serial Petsc .... But undefined reference to MPI_Init()? In-Reply-To: References: Message-ID: Some other MPI installation is interfering with MPIUNI. In $PETC_DIR/include/mpiuni/mpi.h, we #define MPI_Init to Petsc_MPI_Init. Thus, in the compile of ex2.c, some other mpi.h is being picked up. You can see which one using gcc -E. Matt On 1/19/07, #DOMINIC DENVER JOHN CHANDAR# wrote: > > > Hi, > > Ive installed the serial version of petsc couple of months back > sucessfully. I did it again just now on Itanium-64 with the compiler options > mpi=0. Just wondering, although libmpiuni.a is getting linked, i still get > undefined ref to MPI_Init()? Any clues ? > > gcc -c -Wall -g3 -I/home/pdominic/petsc-2.2.1 > -I/home/pdominic/petsc-2.2.1/bmake/linux-gnu-ia64-intel > -I/home/pdominic/petsc-2.2.1/include > -I/home/pdominic/petsc-2.2.1/include/mpiuni -DPETSC_USE_BOPT_g > -DPETSC_USE_EXTERN_CXX -D__SDIR__="src/ksp/examples/tests/" > ex2.c > > gcc -Wall -g3 > -Wl,-rpath,/home/pdominic/petsc-2.2.1/lib/libg/linux-gnu-ia64-intel > -o ex2 ex2.o -L/home/pdominic/petsc-2.2.1/lib/libg/linux-gnu-ia64-intel > -lpetscksp -lpetscdm -lpetscmat -lpetscvec -lpetsc > > -L/usr/X11R6/lib -lX11 > > > > > -Wl,-rpath,/usr/lib -L/usr/lib -llapack -Wl,-rpath,/usr/lib -L/usr/lib > -lblas > -Wl,-rpath,/usr/src/gcc-3.0.4-build2/lib/gcc-lib/ia64-unknown-linux/3.0.4 > -L/usr/src/gcc-3.0.4-build2/lib/gcc-lib/ia64-unknown-linux/3.0.4 > -lg2c -L/home/pdominic/petsc-2.2.1/lib/libg/linux-gnu-ia64-intel -lmpiuni > -ldl -ldl -ldl -ldl -lm -lg2c -lm > -L/usr/local/lib/gcc-lib/ia64-unknown-linux/3.0.4 > -L/usr/local/lib/gcc-lib/ia64-unknown-linux/3.0.4/../../.. > -lm > ex2.o: In function `main': > > /home/pdominic/petsc-2.2.1/src/ksp/examples/tests/ex2.c:31: > undefined reference to `MPI_Init' > /home/pdominic/petsc-2.2.1/src/ksp/examples/tests/ex2.c:33: > undefined reference to `MPI_Finalize' > collect2: ld returned 1 exit status > make: [ex2] Error 1 (ignored) > rm -f ex2.o > > > Regards, > Dominic -- One trouble is that despite this system, anyone who reads journals widely and critically is forced to realize that there are scarcely any bars to eventual publication. There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print. -- Drummond Rennie From DOMI0002 at ntu.edu.sg Fri Jan 19 12:20:03 2007 From: DOMI0002 at ntu.edu.sg (#DOMINIC DENVER JOHN CHANDAR#) Date: Sat, 20 Jan 2007 02:20:03 +0800 Subject: Serial Petsc .... But undefined reference to MPI_Init()? In-Reply-To: Message-ID: Hi, Thanks, As you said, I found that mpi.h was being obtained from /usr/src/mpich-1.2.5 and this call was from petsc.h ..no wonder how that came ?Anyways, I made a clean build replacing it by the mpi.h from mpiuni/mpi.h and all went fine. Regs, Dominic -----Original Message----- From: owner-petsc-users at mcs.anl.gov [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Saturday, January 20, 2007 12:57 AM To: petsc-users at mcs.anl.gov Subject: Re: Serial Petsc .... But undefined reference to MPI_Init()? Some other MPI installation is interfering with MPIUNI. In $PETC_DIR/include/mpiuni/mpi.h, we #define MPI_Init to Petsc_MPI_Init. Thus, in the compile of ex2.c, some other mpi.h is being picked up. You can see which one using gcc -E. Matt On 1/19/07, #DOMINIC DENVER JOHN CHANDAR# wrote: > > > Hi, > > Ive installed the serial version of petsc couple of months back > sucessfully. I did it again just now on Itanium-64 with the compiler > options mpi=0. Just wondering, although libmpiuni.a is getting linked, > i still get undefined ref to MPI_Init()? Any clues ? > > gcc -c -Wall -g3 -I/home/pdominic/petsc-2.2.1 > -I/home/pdominic/petsc-2.2.1/bmake/linux-gnu-ia64-intel > -I/home/pdominic/petsc-2.2.1/include > -I/home/pdominic/petsc-2.2.1/include/mpiuni -DPETSC_USE_BOPT_g > -DPETSC_USE_EXTERN_CXX -D__SDIR__="src/ksp/examples/tests/" > ex2.c > > gcc -Wall -g3 > -Wl,-rpath,/home/pdominic/petsc-2.2.1/lib/libg/linux-gnu-ia64-intel > -o ex2 ex2.o > -L/home/pdominic/petsc-2.2.1/lib/libg/linux-gnu-ia64-intel > -lpetscksp -lpetscdm -lpetscmat -lpetscvec -lpetsc > > -L/usr/X11R6/lib -lX11 > > > > > -Wl,-rpath,/usr/lib -L/usr/lib -llapack -Wl,-rpath,/usr/lib -L/usr/lib > -lblas > -Wl,-rpath,/usr/src/gcc-3.0.4-build2/lib/gcc-lib/ia64-unknown-linux/3. > 0.4 > -L/usr/src/gcc-3.0.4-build2/lib/gcc-lib/ia64-unknown-linux/3.0.4 > -lg2c -L/home/pdominic/petsc-2.2.1/lib/libg/linux-gnu-ia64-intel > -lmpiuni -ldl -ldl -ldl -ldl -lm -lg2c -lm > -L/usr/local/lib/gcc-lib/ia64-unknown-linux/3.0.4 > -L/usr/local/lib/gcc-lib/ia64-unknown-linux/3.0.4/../../.. > -lm > ex2.o: In function `main': > > /home/pdominic/petsc-2.2.1/src/ksp/examples/tests/ex2.c:31: > undefined reference to `MPI_Init' > /home/pdominic/petsc-2.2.1/src/ksp/examples/tests/ex2.c:33: > undefined reference to `MPI_Finalize' > collect2: ld returned 1 exit status > make: [ex2] Error 1 (ignored) > rm -f ex2.o > > > Regards, > Dominic -- One trouble is that despite this system, anyone who reads journals widely and critically is forced to realize that there are scarcely any bars to eventual publication. There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print. -- Drummond Rennie From zonexo at gmail.com Fri Jan 19 19:56:38 2007 From: zonexo at gmail.com (Ben Tay) Date: Sat, 20 Jan 2007 09:56:38 +0800 Subject: MPI forum similar to PETSc for MPI problems? Message-ID: <804ab5d40701191756x4f3827e3sfa47068641f6663f@mail.gmail.com> Hi, I've encountered some MPI programming problems. Does anyone know of a mailing list sort of discussion forum similar to this PETSc one which deals with MPI? Btw, my question is as follow, it will be great if someone can help too ;-) It's in fortran but I guess it should be similar in C. Put it simply, how do I broadcast part of a derived type variable to other processors ie type particle real x,y,z integer number end type particle type(particle) part(10,10) So I want to broadcast part(:,:)%x to other processors. How can I do that? I tried to use: call MPI_BCAST(part(i,j)%x, 1, MPI_Real,1, MPI_COMM_WORLD, ierr) but it can't work. Thank you very much -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jan 19 20:39:43 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 19 Jan 2007 20:39:43 -0600 (CST) Subject: MPI forum similar to PETSc for MPI problems? In-Reply-To: <804ab5d40701191756x4f3827e3sfa47068641f6663f@mail.gmail.com> References: <804ab5d40701191756x4f3827e3sfa47068641f6663f@mail.gmail.com> Message-ID: On Sat, 20 Jan 2007, Ben Tay wrote: > Hi, > > I've encountered some MPI programming problems. Does anyone know of a > mailing list sort of discussion forum similar to this PETSc one which deals > with MPI? I don't know. Open MPI has a users list, maybe that is appropriate http://www.open-mpi.org/community/lists/users/ > > Btw, my question is as follow, it will be great if someone can help too ;-) > > It's in fortran but I guess it should be similar in C. Put it simply, how do > I broadcast part of a derived type variable to other processors ie > > type particle > > real x,y,z > > integer number > > end type particle > > type(particle) part(10,10) > > So I want to broadcast part(:,:)%x to other processors. How can I do that? I > tried to use: > > call MPI_BCAST(part(i,j)%x, 1, MPI_Real,1, MPI_COMM_WORLD, ierr) but it > can't work. What happens? The value on the other processes is not changed to what it should be? > > Thank you very much > From fuentes at ices.utexas.edu Sat Jan 20 12:15:21 2007 From: fuentes at ices.utexas.edu (David Fuentes) Date: Sat, 20 Jan 2007 12:15:21 -0600 (CST) Subject: MPI forum similar to PETSc for MPI problems? In-Reply-To: <804ab5d40701191756x4f3827e3sfa47068641f6663f@mail.gmail.com> References: <804ab5d40701191756x4f3827e3sfa47068641f6663f@mail.gmail.com> Message-ID: Hello, I had a similiar problem, I used the "sequence" statement in the type definition and it worked for me. type particle sequence real x,y,z integer number end type particle good luck, david On Sat, 20 Jan 2007, Ben Tay wrote: > Hi, > > I've encountered some MPI programming problems. Does anyone know of a > mailing list sort of discussion forum similar to this PETSc one which deals > with MPI? > <> Btw, my question is as follow, it will be great if someone can help too ;-) > > It's in fortran but I guess it should be similar in C. Put it simply, how do > I broadcast part of a derived type variable to other processors ie > > type particle > > real x,y,z > > integer number > > end type particle > > type(particle) part(10,10) > > So I want to broadcast part(:,:)%x to other processors. How can I do that? I > tried to use: > > call MPI_BCAST(part(i,j)%x, 1, MPI_Real,1, MPI_COMM_WORLD, ierr) but it > can't work. > > Thank you very much > From zonexo at gmail.com Tue Jan 23 05:42:34 2007 From: zonexo at gmail.com (Ben Tay) Date: Tue, 23 Jan 2007 19:42:34 +0800 Subject: Running squential and parallel linear solver in the same code? Message-ID: <804ab5d40701230342i10500249v921137c1e5081ac3@mail.gmail.com> Hi, In my MPI CFD code, I have to solve 2 linear eqns - momentum and poisson eqn. There's some error in the code and I don't know which one causes the error. May I know if it is possible to solve 1 eqn in sequential mode while the other in MPI to pinpoint the error since my code works ok in a sequential mode. I tried to use this for one of the eqn: call MatCreateSeqAIJ(PETSC_COMM_SELF,size_x*size_y,size_x*size_y,9,PETSC_NULL_INTEGER,A_mat,ierr) call VecCreateSeq(PETSC_COMM_SELF,size_x*size_y,b_rhs,ierr) but there's error. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jan 23 07:58:02 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 23 Jan 2007 07:58:02 -0600 (CST) Subject: Running squential and parallel linear solver in the same code? In-Reply-To: <804ab5d40701230342i10500249v921137c1e5081ac3@mail.gmail.com> References: <804ab5d40701230342i10500249v921137c1e5081ac3@mail.gmail.com> Message-ID: Ben, You can use -pc_type redundant -redundant_pc_type lu to solve the linear system with a sequential LU to test if that improves the situation. Likely your generation of the matrix on 2 processes is wrong, you can run with -mat_view on a small problem with both 1 and 2 processes and see if they are the same in both cases. I don't think what you propose to do below makes sense, Barry On Tue, 23 Jan 2007, Ben Tay wrote: > Hi, > > In my MPI CFD code, I have to solve 2 linear eqns - momentum and poisson > eqn. There's some error in the code and I don't know which one causes the > error. May I know if it is possible to solve 1 eqn in sequential mode while > the other in MPI to pinpoint the error since my code works ok in a > sequential mode. > > I tried to use this for one of the eqn: > > > call > MatCreateSeqAIJ(PETSC_COMM_SELF,size_x*size_y,size_x*size_y,9,PETSC_NULL_INTEGER,A_mat,ierr) > > call VecCreateSeq(PETSC_COMM_SELF,size_x*size_y,b_rhs,ierr) > > but there's error. > > Thank you. > From hzhang at mcs.anl.gov Tue Jan 23 09:15:08 2007 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 23 Jan 2007 09:15:08 -0600 (CST) Subject: minisymposium for PETSc users at ICIAM07 Message-ID: Dear PETSc users, We are proposing to organize a minisymposium for petesc users during SIAM ICIAM07 (http://www.iciam07.ch/index). If you consider presenting your work in the symposium, please contact me (hzhang at mcs.anl.gov) as soon as possible. The Abstract for the symposium is attached below. Thanks, Hong > Abstract: > Minisymposium: > Parallel Numerical Computation using PETSc > Portable, Extensible Toolkit for Scientific Computation (PETSc) is a suite > of data structures and routines for the scalable (parallel) solution of > scientific applications modeled by partial differential equations. This > minisymposium aims at the people engaged in algorithmic research and > software development on large-scale numerical simulations. It provides > opportunity for the current PETSc users to present their applications and > their requirements for the future, as well as introduces potential users > on how the software tools can be used to ease the difficulty and reduce > the development time on their complex mathematical problems. This is also > a good occasion for informal exchanges and discussions between users and > developers of PETSc. > > From zonexo at gmail.com Tue Jan 23 20:18:45 2007 From: zonexo at gmail.com (Ben Tay) Date: Wed, 24 Jan 2007 10:18:45 +0800 Subject: Running squential and parallel linear solver in the same code? In-Reply-To: References: <804ab5d40701230342i10500249v921137c1e5081ac3@mail.gmail.com> Message-ID: <804ab5d40701231818m87de46euec97ab7f8bf76417@mail.gmail.com> Hi Barry, Thanks for your recommendation. I've managed to use mat_view to view the matrix. However, there's something which I don't understand. My linear eqns consist of 36 pts - so I see 36 rows - 0 to 35. For the uniprocessor, I saw 0-35rows. These show the coefficients of my matrix. Then there's a repeat of 0-35 rows again. But what are these values? I don't understand. For the 4 processors, I saw exactly the same 0-35rows. After that I saw 0-8 rows repeated 4 times. The values are now different, even for the same 1st 0 row. Some values are missing. I guess there's where my mistakes are. Thank you. On 1/23/07, Barry Smith wrote: > > > Ben, > > You can use -pc_type redundant -redundant_pc_type lu to solve the > linear system with a sequential LU to test if that improves the situation. > > Likely your generation of the matrix on 2 processes is wrong, you can run > with -mat_view on a small problem with both 1 and 2 processes and see if > they are the same in both cases. > > I don't think what you propose to do below makes sense, > > Barry > > > On Tue, 23 Jan 2007, Ben Tay wrote: > > > Hi, > > > > In my MPI CFD code, I have to solve 2 linear eqns - momentum and poisson > > eqn. There's some error in the code and I don't know which one causes > the > > error. May I know if it is possible to solve 1 eqn in sequential mode > while > > the other in MPI to pinpoint the error since my code works ok in a > > sequential mode. > > > > I tried to use this for one of the eqn: > > > > > > call > > > MatCreateSeqAIJ(PETSC_COMM_SELF,size_x*size_y,size_x*size_y,9,PETSC_NULL_INTEGER,A_mat,ierr) > > > > call VecCreateSeq(PETSC_COMM_SELF,size_x*size_y,b_rhs,ierr) > > > > but there's error. > > > > Thank you. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Tue Jan 23 20:39:53 2007 From: zonexo at gmail.com (Ben Tay) Date: Wed, 24 Jan 2007 10:39:53 +0800 Subject: PETSc performance on Intel core 2 duo Message-ID: <804ab5d40701231839t375ac338if0191e87e53aec23@mail.gmail.com> Hi, Intel's new processor core 2 duo (c2d) have been getting pretty good reviews. How well is PETSc performance on the c2d? I wonder how much speed improvement will I get if I upgrade from my current Athlon XP 2400+ at 2200mhz. Also the speed improvement over my school's xeon 3.06ghz server. Is there any place where I can get or run some benchmark to get a rough comparison? Btw, how important is L1/L2 cache impact on PETSc's performance? e.g. 2Mb vs 4 Mb L2 cache different. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jan 23 20:46:08 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 23 Jan 2007 20:46:08 -0600 (CST) Subject: Running squential and parallel linear solver in the same code? In-Reply-To: <804ab5d40701231818m87de46euec97ab7f8bf76417@mail.gmail.com> References: <804ab5d40701230342i10500249v921137c1e5081ac3@mail.gmail.com> <804ab5d40701231818m87de46euec97ab7f8bf76417@mail.gmail.com> Message-ID: The "second" matrices in each run are the ILU factors on each process. So long as the 0-35 row matrices are the same on each run you are all set. BTW: KSPSolve() does not generate an error if it does not converge. Run with -ksp_converged_reason to see if it converged or not and why. Barry On Wed, 24 Jan 2007, Ben Tay wrote: > Hi Barry, > > Thanks for your recommendation. I've managed to use mat_view to view the > matrix. However, there's something which I don't understand. My linear eqns > consist of 36 pts - so I see 36 rows - 0 to 35. > > For the uniprocessor, I saw 0-35rows. These show the coefficients of my > matrix. Then there's a repeat of 0-35 rows again. But what are these values? > I don't understand. > > For the 4 processors, I saw exactly the same 0-35rows. After that I saw 0-8 > rows repeated 4 times. The values are now different, even for the same 1st 0 > row. Some values are missing. I guess there's where my mistakes are. > > Thank you. > > > On 1/23/07, Barry Smith wrote: > > > > > > Ben, > > > > You can use -pc_type redundant -redundant_pc_type lu to solve the > > linear system with a sequential LU to test if that improves the situation. > > > > Likely your generation of the matrix on 2 processes is wrong, you can run > > with -mat_view on a small problem with both 1 and 2 processes and see if > > they are the same in both cases. > > > > I don't think what you propose to do below makes sense, > > > > Barry > > > > > > On Tue, 23 Jan 2007, Ben Tay wrote: > > > > > Hi, > > > > > > In my MPI CFD code, I have to solve 2 linear eqns - momentum and poisson > > > eqn. There's some error in the code and I don't know which one causes > > the > > > error. May I know if it is possible to solve 1 eqn in sequential mode > > while > > > the other in MPI to pinpoint the error since my code works ok in a > > > sequential mode. > > > > > > I tried to use this for one of the eqn: > > > > > > > > > call > > > > > MatCreateSeqAIJ(PETSC_COMM_SELF,size_x*size_y,size_x*size_y,9,PETSC_NULL_INTEGER,A_mat,ierr) > > > > > > call VecCreateSeq(PETSC_COMM_SELF,size_x*size_y,b_rhs,ierr) > > > > > > but there's error. > > > > > > Thank you. > > > > > > > > From sofonea at acad-tim.tm.edu.ro Wed Jan 24 00:11:39 2007 From: sofonea at acad-tim.tm.edu.ro (Victor Sofonea) Date: Wed, 24 Jan 2007 08:11:39 +0200 (EET) Subject: Scattering DA Vectors In-Reply-To: References: <005401c6fe58$b1d0b4e0$3364a8c0@dmrc6700512> <005a01c6fe85$5a336160$3364a8c0@dmrc6700512> <006601c6fe8a$7cb73040$3364a8c0@dmrc6700512> <001701c6fee5$fede5f70$3364a8c0@dmrc6700512> <002801c6fee9$c5bfbc30$3364a8c0@dmrc6700512> Message-ID: <20070124074729.E16867@waals.acad-tim.tm.edu.ro> Hi, I'm using two-dimensional DAs in my code to create global and local vectors DA da; Vec nloc, lnloc, zero_nloc; PetscInt imax = 102, jmax=3; DACreate2d(PETSC_COMM_WORLD, DA_YPERIODIC, DA_STENCIL_BOX, imax,jmax,PETSC_DECIDE,PETSC_DECIDE, 1,1,PETSC_NULL,PETSC_NULL,&da); ierr = DACreateLocalVector(da,&lnloc); CHKERRQ(ierr); ierr = DACreateGlobalVector(da,&nloc); CHKERRQ(ierr); and at some moment I need copy all values of the global vector nloc on the zeroth processor. For this purpose, I tried to create a scatter context and use VecScatterBegin...End: ierr = VecScatterCreateToZero(nloc,&zero_ctx,&zero_nloc);CHKERRQ(ierr); ierr = VecScatterBegin(nloc,zero_nloc,INSERT_VALUES,SCATTER_FORWARD,zero_ctx); CHKERRQ(ierr); ierr = VecScatterEnd(nloc,zero_nloc,INSERT_VALUES,SCATTER_FORWARD,zero_ctx); CHKERRQ(ierr); This scatter procedure does not work for global vectors created using DACreateGlobalVector(...) since (on processor 0, when using 2 processors, but I plan to use more processors in the future): PetscReal **aznloc ierr = DAVecGetArray(da,zero_nloc,&aznloc); CHKERRQ(ierr); gives the error +++++++++++++ BEGIN ERROR MESSAGE [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Arguments are incompatible! [0]PETSC ERROR: Vector local size 306 is not compatible with DA local sizes 153 260 ! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 2.3.2, Patch 7, Fri Dec 1 11:22:44 CST 2006 HG revision: 4fc87ad4c9d9855d212e1fa6135ed04b299cdc92 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./dachannel on a freebsd named smoluchowski.acad-tim.tm.edu.ro by sofonea Wed Jan 24 08:08:34 2007 [0]PETSC ERROR: Libraries linked from /usr/local/petsc-2.3.2-p7-mpi//lib/freebsd [0]PETSC ERROR: Configure run at Tue Jan 2 18:08:24 2007 [0]PETSC ERROR: Configure options --with-deb[1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Arguments are incompatible! [1]PETSC ERROR: Vector local size 0 is not compatible with DA local sizes 153 260 ! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 2.3.2, Patch 7, Fri Dec 1 11:22:44 CST 2006 HG revision: 4fc87ad4c9d9855d212e1fa6135ed04b299cdc92 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ./dachannel on a freebsd named smoluchowski.acad-tim.tm.edu.ro by sofonea Wed Jan 24 08:08:34 2007 [1]PETSC ERROR: Libraries linked from /usr/local/petsc-2.3.2-p7-mpi//lib/freebsd [1]PETSC ERROR: Configure run at Tue Jan 2 18:08:24 2007 [1]PETSC ERROR: Configure options --with-debugging=0ugging=0 --download-f-blas-lapack=ifneeded --download-mpich=ifneeded --with-shared=0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: DAVecGetArray() line 50 in src/dm/da/src/dagetarray.c [0]PETSC ERROR: User provided function() line 343 in dainout.c [0]PETSC ERROR: User provided function() line 1130 in damain.c --download-f-blas-lapack=ifneeded --download-mpich=ifneeded --with-shared=0 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: DAVecGetArray() line 50 in src/dm/da/src/dagetarray.c [1]PETSC ERROR: User provided function() line 343 in dainout.c [1]PETSC ERROR: User provided function() line 1130 in damain.c ++++++++++++ END ERROR MESSAGE I tried also PetscReal *aznloc ierr = VecGetArray(zero_nloc,&aznloc); CHKERRQ(ierr); Although this works, I need the natural ordering for further processing the elements of aznloc, as follows: PetscReal nn[imax]; for(i=0; i References: <005401c6fe58$b1d0b4e0$3364a8c0@dmrc6700512> <005a01c6fe85$5a336160$3364a8c0@dmrc6700512> <006601c6fe8a$7cb73040$3364a8c0@dmrc6700512> <001701c6fee5$fede5f70$3364a8c0@dmrc6700512> <002801c6fee9$c5bfbc30$3364a8c0@dmrc6700512> <20070124074729.E16867@waals.acad-tim.tm.edu.ro> Message-ID: On Wed, 24 Jan 2007, Victor Sofonea wrote: > > Hi, > > I'm using two-dimensional DAs in my code to create global and local > vectors > > DA da; > Vec nloc, lnloc, zero_nloc; > PetscInt imax = 102, jmax=3; > > DACreate2d(PETSC_COMM_WORLD, DA_YPERIODIC, DA_STENCIL_BOX, > imax,jmax,PETSC_DECIDE,PETSC_DECIDE, > 1,1,PETSC_NULL,PETSC_NULL,&da); > > ierr = DACreateLocalVector(da,&lnloc); CHKERRQ(ierr); > > ierr = DACreateGlobalVector(da,&nloc); CHKERRQ(ierr); > > > and at some moment I need copy all values of the global vector nloc > on the zeroth processor. For this purpose, I tried to create a scatter > context and use VecScatterBegin...End: > Before creating the VecScatter to zero you need to do DACreateNaturalVector() and DAGlobalToNaturalBegin/End() then do the scatter below. > ierr = VecScatterCreateToZero(nloc,&zero_ctx,&zero_nloc);CHKERRQ(ierr); > > ierr = VecScatterBegin(nloc,zero_nloc,INSERT_VALUES,SCATTER_FORWARD,zero_ctx); > CHKERRQ(ierr); > ierr = VecScatterEnd(nloc,zero_nloc,INSERT_VALUES,SCATTER_FORWARD,zero_ctx); > CHKERRQ(ierr); > > This scatter procedure does not work for global vectors created using > DACreateGlobalVector(...) since (on processor 0, when using 2 > processors, but I plan to use more processors in the future): > > PetscReal **aznloc > > ierr = DAVecGetArray(da,zero_nloc,&aznloc); CHKERRQ(ierr); > Use VecGetArray2d() which allows you to "shape" the vector as you need. Good luck, Barry > gives the error > > +++++++++++++ BEGIN ERROR MESSAGE > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Arguments are incompatible! > [0]PETSC ERROR: Vector local size 306 is not compatible with DA local > sizes 153 260 > ! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 2.3.2, Patch 7, Fri Dec 1 11:22:44 > CST 2006 HG revision: 4fc87ad4c9d9855d212e1fa6135ed04b299cdc92 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./dachannel on a freebsd named > smoluchowski.acad-tim.tm.edu.ro by sofonea Wed Jan 24 08:08:34 2007 > [0]PETSC ERROR: Libraries linked from > /usr/local/petsc-2.3.2-p7-mpi//lib/freebsd > [0]PETSC ERROR: Configure run at Tue Jan 2 18:08:24 2007 > [0]PETSC ERROR: Configure options --with-deb[1]PETSC ERROR: > --------------------- Error Message ------------------------------------ > [1]PETSC ERROR: Arguments are incompatible! > [1]PETSC ERROR: Vector local size 0 is not compatible with DA local sizes > 153 260 > ! > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 2.3.2, Patch 7, Fri Dec 1 11:22:44 > CST 2006 HG revision: 4fc87ad4c9d9855d212e1fa6135ed04b299cdc92 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: ./dachannel on a freebsd named > smoluchowski.acad-tim.tm.edu.ro by sofonea Wed Jan 24 08:08:34 2007 > [1]PETSC ERROR: Libraries linked from > /usr/local/petsc-2.3.2-p7-mpi//lib/freebsd > [1]PETSC ERROR: Configure run at Tue Jan 2 18:08:24 2007 > [1]PETSC ERROR: Configure options --with-debugging=0ugging=0 > --download-f-blas-lapack=ifneeded --download-mpich=ifneeded > --with-shared=0 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: DAVecGetArray() line 50 in src/dm/da/src/dagetarray.c > [0]PETSC ERROR: User provided function() line 343 in dainout.c > [0]PETSC ERROR: User provided function() line 1130 in damain.c > --download-f-blas-lapack=ifneeded --download-mpich=ifneeded > --with-shared=0 > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: DAVecGetArray() line 50 in src/dm/da/src/dagetarray.c > [1]PETSC ERROR: User provided function() line 343 in dainout.c > [1]PETSC ERROR: User provided function() line 1130 in damain.c > > ++++++++++++ END ERROR MESSAGE > > I tried also > > PetscReal *aznloc > > ierr = VecGetArray(zero_nloc,&aznloc); CHKERRQ(ierr); > > Although this works, I need the natural ordering for further processing > the elements of aznloc, as follows: > > PetscReal nn[imax]; > > for(i=0; i { > nn[i] = 0.00 > for(i=0; j { > nn[i] += aznloc[j][i]; > } > } > > > In other words, I need aznloc to be a two-dimensional array to preserve > the same idex ranges 0...imax and 0...jmax used for the global vector > nloc). > > Do you have a solution ? > > Thank you, > > Victor Sofonea > > From aja2111 at columbia.edu Wed Jan 24 10:12:00 2007 From: aja2111 at columbia.edu (Aron Ahmadia) Date: Wed, 24 Jan 2007 11:12:00 -0500 Subject: PETSc performance on Intel core 2 duo In-Reply-To: <804ab5d40701231839t375ac338if0191e87e53aec23@mail.gmail.com> References: <804ab5d40701231839t375ac338if0191e87e53aec23@mail.gmail.com> Message-ID: <37604ab40701240812y3af72ac1vb63cae031b2eb515@mail.gmail.com> Hi Ben, These are some really difficult questions to answer quickly. PETSc performance is dependent on a large variety of factors, from the architecture of your computer to the layout of your network to how much RAM you have, most importantly the problem you are trying to solve. I have an Intel Core Duo laptop as well as a dual-processor Intel Core 2 Duo workstation here at Columbia, I'd be happy to benchmark some of your code for you if it will help you decide on the upgrade. Thanks, ~Aron On 1/23/07, Ben Tay wrote: > > Hi, > > Intel's new processor core 2 duo (c2d) have been getting pretty good > reviews. How well is PETSc performance on the c2d? > > I wonder how much speed improvement will I get if I upgrade from my > current Athlon XP 2400+ at 2200mhz. Also the speed improvement over my > school's xeon 3.06ghz server. > > Is there any place where I can get or run some benchmark to get a rough > comparison? > > Btw, how important is L1/L2 cache impact on PETSc's performance? e.g. 2Mb > vs 4 Mb L2 cache different. > > Thank you. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From js2615 at columbia.edu Wed Jan 24 15:49:49 2007 From: js2615 at columbia.edu (js2615 at columbia.edu) Date: Wed, 24 Jan 2007 16:49:49 -0500 Subject: graphic dump on cluster In-Reply-To: References: <45A298A6.3020400@free.fr> Message-ID: <1169675389.45b7d47d63200@cubmail.cc.columbia.edu> Hi Petsc experts, I have a question about the graphic output using multiple processors. Suppose I am using X11 to ssh into the cluster, and would like to view the graphical output at each iteration using ierr = VecView(x,PETSC_VIEWER_DRAW_WORLD);CHKERRQ(ierr); I got the following error message, turns out, when I use multiple processors (not a problem using one processor though): Unable to open display on localhost:10.0 . Make sure your COMPUTE NODES are authorized to connect to this X server and either your DISPLAY variable is set or you use the -display name option ! [3]PETSC ERROR: [1]PETSC ERROR: Libraries linked from /opt/petsc-2.3.2/lib/linux-gnu-c-debug [2]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Configure run at Fri Oct 27 11:40:04 2006 Petsc Release Version 2.3.2, Patch 3, Fri Sep 29 17:09:34 CDT 2006 HG revision: 9215af156a9cbcdc1ec666e2b5c7934688ddc526 [3]PETSC ERROR: Libraries linked from /opt/petsc-2.3.2/lib/linux-gnu-c-debug See docs/changes/index.html for recent updates. [2]PETSC ERROR: [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [3]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: [3]PETSC ERROR: ------------------------------------------------------------------------ Configure options --with-mpich=/opt/mpich2 --with-blas-lapack=lib=/usr/lib/libatlas.so --with-shared=0 [3]PETSC ERROR: ./ex19 on a linux-gnu named n0010 by jianing Wed Jan 24 16:36:03 2007 [3]PETSC ERROR: Libraries linked from /opt/petsc-2.3.2/lib/linux-gnu-c-debug Configure run at Fri Oct 27 11:40:04 2006 [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: PetscDrawXGetDisplaySize_Private() line 618 in src/sys/draw/impls/x/xops.c [2]PETSC ERROR: [3]PETSC ERROR: PETSc unable to use X windows proceeding without graphics Configure run at Fri Oct 27 11:40:04 2006 [1]PETSC ERROR: Configure options --with-mpich=/opt/mpich2 --with-blas-lapack=lib=/usr/lib/libatlas.so --with-shared=0 [3]PETSC ERROR: [1]PETSC ERROR: Configure options --with-mpich=/opt/mpich2 --with-blas-lapack=lib=/usr/lib/libatlas.so --with-shared=0 ------------------------------------------------------------------------ [3]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: PetscDrawXGetDisplaySize_Private() line 618 in src/sys/draw/impls/x/xops.c [3]PETSC ERROR: PETSc unable to use X windows proceeding without graphics PetscDrawXGetDisplaySize_Private() line 618 in src/sys/draw/impls/x/xops.c [1]PETSC ERROR: PETSc unable to use X windows proceeding without graphics Is there any permission issue I need to set up? Thanks, Jianing From knepley at gmail.com Wed Jan 24 16:44:28 2007 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 24 Jan 2007 16:44:28 -0600 Subject: graphic dump on cluster In-Reply-To: <1169675389.45b7d47d63200@cubmail.cc.columbia.edu> References: <45A298A6.3020400@free.fr> <1169675389.45b7d47d63200@cubmail.cc.columbia.edu> Message-ID: On 1/24/07, js2615 at columbia.edu wrote: > Hi Petsc experts, > > I have a question about the graphic output using multiple > processors. Suppose I am using X11 to ssh into the cluster, and > would like to view the graphical output at each iteration using > > ierr = VecView(x,PETSC_VIEWER_DRAW_WORLD);CHKERRQ(ierr); > > I got the following error message, turns out, when I use multiple > processors (not a problem using one processor though): > > Unable to open display on localhost:10.0 > . Make sure your COMPUTE NODES are authorized to connect > to this X server and either your DISPLAY variable > is set or you use the -display name option The error message is exactly right. You either need to set the DISPLAY env var or give the -display option to PETSc. I think giving -Y to ssh forwards your Xconnection automatically (setting DISPLAY). Matt From bsmith at mcs.anl.gov Wed Jan 24 16:48:05 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 24 Jan 2007 16:48:05 -0600 (CST) Subject: graphic dump on cluster In-Reply-To: References: <45A298A6.3020400@free.fr> <1169675389.45b7d47d63200@cubmail.cc.columbia.edu> Message-ID: Note that the PETSc X model has ALL nodes connect to the X server (not just where rank == 0). Thus all of them need 1) the correct display value (-display mymachine:0.0 is usually the best way to provide it) 2) permission to access the display (X permission stuff, read up on it) and 3) a route to the X server (via TCP/IP). Barry On Wed, 24 Jan 2007, Matthew Knepley wrote: > On 1/24/07, js2615 at columbia.edu wrote: > > Hi Petsc experts, > > > > I have a question about the graphic output using multiple > > processors. Suppose I am using X11 to ssh into the cluster, and > > would like to view the graphical output at each iteration using > > > > ierr = VecView(x,PETSC_VIEWER_DRAW_WORLD);CHKERRQ(ierr); > > > > I got the following error message, turns out, when I use multiple > > processors (not a problem using one processor though): > > > > Unable to open display on localhost:10.0 > > . Make sure your COMPUTE NODES are authorized to connect > > to this X server and either your DISPLAY variable > > is set or you use the -display name option > > The error message is exactly right. You either need to set the DISPLAY env > var or give the -display option to PETSc. I think giving -Y to ssh forwards > your > Xconnection automatically (setting DISPLAY). > > Matt > > From balay at mcs.anl.gov Wed Jan 24 17:00:34 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 24 Jan 2007 17:00:34 -0600 (CST) Subject: graphic dump on cluster In-Reply-To: References: <45A298A6.3020400@free.fr> <1169675389.45b7d47d63200@cubmail.cc.columbia.edu> Message-ID: Its hard to get this working properly on a cluster. One way to do this is: - ssh to each node [where the job will be run] so that there exists a X11 tunnel from all nodes - make sure all of them have the same ssh-x11 display. i.e once you ssh - check with 'echo $DISPLAY' - now run the PETSc binary with this display value.. Satish On Wed, 24 Jan 2007, Barry Smith wrote: > > Note that the PETSc X model has ALL nodes connect to the X server (not > just where rank == 0). Thus all of them need > 1) the correct display value (-display mymachine:0.0 is usually the best way to > provide it) > 2) permission to access the display (X permission stuff, read up on it) and > 3) a route to the X server (via TCP/IP). > > Barry > > > On Wed, 24 Jan 2007, Matthew Knepley wrote: > > > On 1/24/07, js2615 at columbia.edu wrote: > > > Hi Petsc experts, > > > > > > I have a question about the graphic output using multiple > > > processors. Suppose I am using X11 to ssh into the cluster, and > > > would like to view the graphical output at each iteration using > > > > > > ierr = VecView(x,PETSC_VIEWER_DRAW_WORLD);CHKERRQ(ierr); > > > > > > I got the following error message, turns out, when I use multiple > > > processors (not a problem using one processor though): > > > > > > Unable to open display on localhost:10.0 > > > . Make sure your COMPUTE NODES are authorized to connect > > > to this X server and either your DISPLAY variable > > > is set or you use the -display name option > > > > The error message is exactly right. You either need to set the DISPLAY env > > var or give the -display option to PETSc. I think giving -Y to ssh forwards > > your > > Xconnection automatically (setting DISPLAY). > > > > Matt > > > > > > From aja2111 at columbia.edu Wed Jan 24 17:03:17 2007 From: aja2111 at columbia.edu (Aron Ahmadia) Date: Wed, 24 Jan 2007 18:03:17 -0500 Subject: graphic dump on cluster In-Reply-To: <1169675389.45b7d47d63200@cubmail.cc.columbia.edu> References: <45A298A6.3020400@free.fr> <1169675389.45b7d47d63200@cubmail.cc.columbia.edu> Message-ID: <37604ab40701241503h4a7a2ff7uc3260f0df6184e6d@mail.gmail.com> Hi Jianing, I don't think the orion cluster is set up for the worker nodes to be able to connect back to a remote workstation. I would collect the data locally on the master node, and use PETSC_VIEWER_DRAW_SELF. Please double-check that the display code works on single-processor jobs, that error looks like an X11 setup error as Matt notes. ~A On 1/24/07, js2615 at columbia.edu wrote: > > Hi Petsc experts, > > I have a question about the graphic output using multiple > processors. Suppose I am using X11 to ssh into the cluster, and > would like to view the graphical output at each iteration using > > ierr = VecView(x,PETSC_VIEWER_DRAW_WORLD);CHKERRQ(ierr); > > I got the following error message, turns out, when I use multiple > processors (not a problem using one processor though): > > Unable to open display on localhost:10.0 > . Make sure your COMPUTE NODES are authorized to connect > to this X server and either your DISPLAY variable > is set or you use the -display name option > ! > [3]PETSC ERROR: [1]PETSC ERROR: Libraries linked from > /opt/petsc-2.3.2/lib/linux-gnu-c-debug > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [3]PETSC ERROR: Configure run at Fri Oct 27 11:40:04 2006 > Petsc Release Version 2.3.2, Patch 3, Fri Sep 29 17:09:34 CDT 2006 > HG revision: 9215af156a9cbcdc1ec666e2b5c7934688ddc526 > [3]PETSC ERROR: Libraries linked from > /opt/petsc-2.3.2/lib/linux-gnu-c-debug > See docs/changes/index.html for recent updates. > [2]PETSC ERROR: [3]PETSC ERROR: See docs/faq.html for hints about > trouble shooting. > [3]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: [3]PETSC ERROR: > ------------------------------------------------------------------------ > Configure options --with-mpich=/opt/mpich2 > --with-blas-lapack=lib=/usr/lib/libatlas.so --with-shared=0 > [3]PETSC ERROR: ./ex19 on a linux-gnu named n0010 by jianing Wed Jan > 24 16:36:03 2007 > [3]PETSC ERROR: Libraries linked from > /opt/petsc-2.3.2/lib/linux-gnu-c-debug > Configure run at Fri Oct 27 11:40:04 2006 > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: PetscDrawXGetDisplaySize_Private() line 618 in > src/sys/draw/impls/x/xops.c > [2]PETSC ERROR: [3]PETSC ERROR: PETSc unable to use X windows > proceeding without graphics > Configure run at Fri Oct 27 11:40:04 2006 > [1]PETSC ERROR: Configure options --with-mpich=/opt/mpich2 > --with-blas-lapack=lib=/usr/lib/libatlas.so --with-shared=0 > [3]PETSC ERROR: [1]PETSC ERROR: Configure options > --with-mpich=/opt/mpich2 > --with-blas-lapack=lib=/usr/lib/libatlas.so --with-shared=0 > ------------------------------------------------------------------------ > [3]PETSC ERROR: [1]PETSC ERROR: > ------------------------------------------------------------------------ > [3]PETSC ERROR: PetscDrawXGetDisplaySize_Private() line 618 in > src/sys/draw/impls/x/xops.c > [3]PETSC ERROR: PETSc unable to use X windows > proceeding without graphics > PetscDrawXGetDisplaySize_Private() line 618 in > src/sys/draw/impls/x/xops.c > [1]PETSC ERROR: PETSc unable to use X windows > proceeding without graphics > > Is there any permission issue I need to set up? > > Thanks, > > Jianing > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jan 24 17:06:21 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 24 Jan 2007 17:06:21 -0600 (CST) Subject: graphic dump on cluster In-Reply-To: References: <45A298A6.3020400@free.fr> <1169675389.45b7d47d63200@cubmail.cc.columbia.edu> Message-ID: The X graphics in PETSc are intended for debugging and simple stuff; this should be done on a laptop or workstation. Except in limited cases I would not use it on a cluster. Barry On Wed, 24 Jan 2007, Satish Balay wrote: > Its hard to get this working properly on a cluster. > > One way to do this is: > > - ssh to each node [where the job will be run] so that > there exists a X11 tunnel from all nodes > - make sure all of them have the same ssh-x11 display. i.e > once you ssh - check with 'echo $DISPLAY' > - now run the PETSc binary with this display value.. > > Satish > > On Wed, 24 Jan 2007, Barry Smith wrote: > > > > > Note that the PETSc X model has ALL nodes connect to the X server (not > > just where rank == 0). Thus all of them need > > 1) the correct display value (-display mymachine:0.0 is usually the best way to > > provide it) > > 2) permission to access the display (X permission stuff, read up on it) and > > 3) a route to the X server (via TCP/IP). > > > > Barry > > > > > > On Wed, 24 Jan 2007, Matthew Knepley wrote: > > > > > On 1/24/07, js2615 at columbia.edu wrote: > > > > Hi Petsc experts, > > > > > > > > I have a question about the graphic output using multiple > > > > processors. Suppose I am using X11 to ssh into the cluster, and > > > > would like to view the graphical output at each iteration using > > > > > > > > ierr = VecView(x,PETSC_VIEWER_DRAW_WORLD);CHKERRQ(ierr); > > > > > > > > I got the following error message, turns out, when I use multiple > > > > processors (not a problem using one processor though): > > > > > > > > Unable to open display on localhost:10.0 > > > > . Make sure your COMPUTE NODES are authorized to connect > > > > to this X server and either your DISPLAY variable > > > > is set or you use the -display name option > > > > > > The error message is exactly right. You either need to set the DISPLAY env > > > var or give the -display option to PETSc. I think giving -Y to ssh forwards > > > your > > > Xconnection automatically (setting DISPLAY). > > > > > > Matt > > > > > > > > > > > > From aja2111 at columbia.edu Wed Jan 24 17:08:38 2007 From: aja2111 at columbia.edu (Aron Ahmadia) Date: Wed, 24 Jan 2007 18:08:38 -0500 Subject: graphic dump on cluster In-Reply-To: References: <45A298A6.3020400@free.fr> <1169675389.45b7d47d63200@cubmail.cc.columbia.edu> Message-ID: <37604ab40701241508t7ae21712x94e4fca20369c59a@mail.gmail.com> Is there a good script lying around somewhere for setting the X11 connections up from the master/interactive node? This seems like it could be a huge pain if you've got a bunch of worker nodes sitting in a private network behind the master in classic Beowulf style and you don't have a systems administrator to set it up for you. ~A On 1/24/07, Barry Smith wrote: > > > Note that the PETSc X model has ALL nodes connect to the X server (not > just where rank == 0). Thus all of them need > 1) the correct display value (-display mymachine:0.0 is usually the best > way to > provide it) > 2) permission to access the display (X permission stuff, read up on it) > and > 3) a route to the X server (via TCP/IP). > > Barry > > > On Wed, 24 Jan 2007, Matthew Knepley wrote: > > > On 1/24/07, js2615 at columbia.edu wrote: > > > Hi Petsc experts, > > > > > > I have a question about the graphic output using multiple > > > processors. Suppose I am using X11 to ssh into the cluster, and > > > would like to view the graphical output at each iteration using > > > > > > ierr = VecView(x,PETSC_VIEWER_DRAW_WORLD);CHKERRQ(ierr); > > > > > > I got the following error message, turns out, when I use multiple > > > processors (not a problem using one processor though): > > > > > > Unable to open display on localhost:10.0 > > > . Make sure your COMPUTE NODES are authorized to connect > > > to this X server and either your DISPLAY variable > > > is set or you use the -display name option > > > > The error message is exactly right. You either need to set the DISPLAY > env > > var or give the -display option to PETSc. I think giving -Y to ssh > forwards > > your > > Xconnection automatically (setting DISPLAY). > > > > Matt > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Jan 24 17:51:15 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 24 Jan 2007 17:51:15 -0600 (CST) Subject: graphic dump on cluster In-Reply-To: <37604ab40701241508t7ae21712x94e4fca20369c59a@mail.gmail.com> References: <45A298A6.3020400@free.fr> <1169675389.45b7d47d63200@cubmail.cc.columbia.edu> <37604ab40701241508t7ae21712x94e4fca20369c59a@mail.gmail.com> Message-ID: On Wed, 24 Jan 2007, Aron Ahmadia wrote: > Is there a good script lying around somewhere for setting the X11 > connections up from the master/interactive node? This seems like it could > be a huge pain if you've got a bunch of worker nodes sitting in a private > network behind the master in classic Beowulf style and you don't have a > systems administrator to set it up for you. Different clusters have different 'scripts' for job submission. So you'll have to figureout how to 'sneak' in the ssh-x11 connections in here during the job startup. [i.e this not easy to automate] And then there are issues of x11 authentication & ssh authentication to worry about [if we are opening sockets connectins across machines] However there is a simple alternative to get this working - it the following are true for your cluster: - all compute-nodes share home filesystem with the frontend-node [i.e everyone can read the same ~/.Xauthority file for x11 permissions] - a willing sys-admin who can change sshd config on the front-end. The change is to add the following to /etc/ssh/sshd_config and restart sshd >>> X11UseLocalhost no << With the above config - one can get X11 working as follows: - compute nodes directly talk via x11 to the front-end - the frontend forwards this x11 communication to the users desktop via ssh. i.e you would do the following: - login to frontend node from your desktop with ssh-x11 [ssh -Y frontend]. - Check what the display is [echo $DISPLAY]. It should be frontend:10.0 or something equivalent. - Now run the PETSc executable with the option [-display frontend:10.0] There might be some firewall issues that need to be taken canre of [the x11 connections from compute-nodes to front-end should not be blocked by the firewall] Satish From balay at mcs.anl.gov Wed Jan 24 18:31:42 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 24 Jan 2007 18:31:42 -0600 (CST) Subject: PETSc performance on Intel core 2 duo In-Reply-To: <37604ab40701240812y3af72ac1vb63cae031b2eb515@mail.gmail.com> References: <804ab5d40701231839t375ac338if0191e87e53aec23@mail.gmail.com> <37604ab40701240812y3af72ac1vb63cae031b2eb515@mail.gmail.com> Message-ID: Ideally you'll have to benchmark the correct code to determine the differences in performance between different architectures. We have some 'sequential' performance numbers for a sample test case in src/benchmarks/results/performance_cfd_2_10. Also anyone can run this benchmark on the hardware they are evaluating - with the options mentioned in this file [the datafiles are at ftp://ftp.mcs.anl.gov/pub/petsc/matrices] With sparse computations, we generally see that the limiting factor is memory bandwidth. So a core-2-duo desktop could have 21GB/s shared [first generation has lower value?] between 4 procs. If this is correct than I would expect it to provide pretty good performance.. Satish On Wed, 24 Jan 2007, Aron Ahmadia wrote: > Hi Ben, > > These are some really difficult questions to answer quickly. PETSc > performance is dependent on a large variety of factors, from the > architecture of your computer to the layout of your network to how much RAM > you have, most importantly the problem you are trying to solve. > > I have an Intel Core Duo laptop as well as a dual-processor Intel Core 2 Duo > workstation here at Columbia, I'd be happy to benchmark some of your code > for you if it will help you decide on the upgrade. > > Thanks, > ~Aron > > On 1/23/07, Ben Tay wrote: > > > > Hi, > > > > Intel's new processor core 2 duo (c2d) have been getting pretty good > > reviews. How well is PETSc performance on the c2d? > > > > I wonder how much speed improvement will I get if I upgrade from my > > current Athlon XP 2400+ at 2200mhz. Also the speed improvement over my > > school's xeon 3.06ghz server. > > > > Is there any place where I can get or run some benchmark to get a rough > > comparison? > > > > Btw, how important is L1/L2 cache impact on PETSc's performance? e.g. 2Mb > > vs 4 Mb L2 cache different. > > > > Thank you. > > > From js2615 at columbia.edu Thu Jan 25 00:09:19 2007 From: js2615 at columbia.edu (js2615 at columbia.edu) Date: Thu, 25 Jan 2007 01:09:19 -0500 Subject: graphic dump on cluster In-Reply-To: References: <45A298A6.3020400@free.fr> <1169675389.45b7d47d63200@cubmail.cc.columbia.edu> <37604ab40701241508t7ae21712x94e4fca20369c59a@mail.gmail.com> Message-ID: <1169705359.45b8498fc927c@cubmail.cc.columbia.edu> Thanks for the prompt reply. This is truly a great alternative. I have followed the instructions, and it worked for me by using -display frontend:10.0 option. I understand that the Petsc graphic output is mainly for debugging (which should probably be done just on my laptop), but I guess it is helpful to have the functionale of dumping output on multiple processors, even just for a fun demo. Aron: In the original setup, my DISPLAY variable is set to $DISPLAY=localhost:0.0. This works indeed when I only use -np 1 (I guess it only start the work on the master node n0000). Cheers, Jianing > Different clusters have different 'scripts' for job submission. > So > you'll have to figureout how to 'sneak' in the ssh-x11 > connections in > here during the job startup. [i.e this not easy to automate] > > And then there are issues of x11 authentication & ssh > authentication > to worry about [if we are opening sockets connectins across > machines] > > However there is a simple alternative to get this working - it > the > following are true for your cluster: > - all compute-nodes share home filesystem with the frontend-node > [i.e > everyone can read the same ~/.Xauthority file for x11 > permissions] > - a willing sys-admin who can change sshd config on the > front-end. The > change is to add the following to /etc/ssh/sshd_config and > restart sshd > > >>> > X11UseLocalhost no > << > > With the above config - one can get X11 working as follows: > - compute nodes directly talk via x11 to the front-end > - the frontend forwards this x11 communication to the users > desktop > via ssh. > > i.e you would do the following: > - login to frontend node from your desktop with ssh-x11 [ssh -Y > frontend]. > - Check what the display is [echo $DISPLAY]. It should be > frontend:10.0 or something equivalent. > - Now run the PETSc executable with the option [-display > frontend:10.0] > > There might be some firewall issues that need to be taken canre > of > [the x11 connections from compute-nodes to front-end should not > be > blocked by the firewall] > > Satish > > From nicolas.bathfield at chalmers.se Thu Jan 25 04:15:45 2007 From: nicolas.bathfield at chalmers.se (Nicolas Bathfield) Date: Thu, 25 Jan 2007 11:15:45 +0100 (CET) Subject: PETSC-Hypre: number of level with boomeramg Message-ID: <24467.193.183.3.2.1169720145.squirrel@webmail.chalmers.se> Hi Petsc users, I have a question regarding the use of Hypre and especially Boomeramg. How can we control the number of levels in the multigrid algortihm? Can we chose it, or if not, can we display it in some way? Best regards, Nicolas. -- Nicolas BATHFIELD Chalmers University of Technology Shipping and Marine Technology phone: +46 (0)31 772 1476 fax: +46 (0)31 772 3699 From asdfgf at ig.com.br Thu Jan 25 05:33:52 2007 From: asdfgf at ig.com.br (Fernando Campos) Date: Thu, 25 Jan 2007 09:33:52 -0200 Subject: PETSC-Hypre: number of level with boomeramg Message-ID: <20070125_113352_038707.asdfgf@ig.com.br> Nicolas, You should try the option -pc_hypre_boomeramg_max_levels in the line command. In order to see the level's details you can use the PESTc option -ksp_view. Best Regards Fernando. Em (11:15:45), petsc-users at mcs.anl.gov escreveu: >Hi Petsc users, > >I have a question regarding the use of Hypre and especially Boomeramg. >How can we control the number of levels in the multigrid algortihm? Can we >chose it, or if not, can we display it in some way? > >Best regards, > >Nicolas. > >-- >Nicolas BATHFIELD >Chalmers University of Technology >Shipping and Marine Technology >phone: +46 (0)31 772 1476 >fax: +46 (0)31 772 3699 > >---------- From nicolas.bathfield at chalmers.se Thu Jan 25 08:31:40 2007 From: nicolas.bathfield at chalmers.se (Nicolas Bathfield) Date: Thu, 25 Jan 2007 15:31:40 +0100 (CET) Subject: PETSC-Hypre: number of level with boomeramg In-Reply-To: <20070125_113352_038707.asdfgf@ig.com.br> References: <20070125_113352_038707.asdfgf@ig.com.br> Message-ID: <5281.193.183.3.2.1169735500.squirrel@webmail.chalmers.se> Dear Fernando, Thanks for your reply. I had notice the value maxlevels given by -ksp_view, but I did not know how to change this value. Still, as the default is 25 levels I think, I have no way to know exactly how many levels are used for a given case. I can reformulate my question in this way: does boomeramg coarsen the "grid" to the coarsest level possible, i.e. 1 cell over the domain, or does it stop before that? (grid is not the right term when it comes to algebraic multilevel methods, but I can't find any other clear way to express it). How can I control the coarsening? Thinking about it, perhaps boomeramd actually refines the grid up to the maximum number of levels...Is it so? Thanks for your help. Nicolas > Nicolas, > > You should try the option -pc_hypre_boomeramg_max_levels in the line > command. In order to see the level's details you can use the PESTc option > -ksp_view. > > Best Regards > > Fernando. > > > Em (11:15:45), petsc-users at mcs.anl.gov escreveu: > > >>Hi Petsc users, >> >>I have a question regarding the use of Hypre and especially Boomeramg. >>How can we control the number of levels in the multigrid algortihm? Can >> we >>chose it, or if not, can we display it in some way? >> >>Best regards, >> >>Nicolas. >> >>-- >>Nicolas BATHFIELD >>Chalmers University of Technology >>Shipping and Marine Technology >>phone: +46 (0)31 772 1476 >>fax: +46 (0)31 772 3699 >> >>---------- > > > -- Nicolas BATHFIELD Chalmers University of Technology Shipping and Marine Technology phone: +46 (0)31 772 1476 fax: +46 (0)31 772 3699 From shma7099 at student.uu.se Thu Jan 25 09:02:14 2007 From: shma7099 at student.uu.se (Shaman Mahmoudi) Date: Thu, 25 Jan 2007 16:02:14 +0100 Subject: PETSC-Hypre: number of level with boomeramg In-Reply-To: <5281.193.183.3.2.1169735500.squirrel@webmail.chalmers.se> References: <20070125_113352_038707.asdfgf@ig.com.br> <5281.193.183.3.2.1169735500.squirrel@webmail.chalmers.se> Message-ID: Try -pc_hypre_boomeramg_print_statistics in the console for more information regarding your AMG. I believe that is the right syntax. I am not 100% though! It is a long time ago I touched it. With best regards, Shaman Mahmoudi On Jan 25, 2007, at 3:31 PM, Nicolas Bathfield wrote: > Dear Fernando, > > Thanks for your reply. > > I had notice the value maxlevels given by -ksp_view, but I did not > know > how to change this value. > > Still, as the default is 25 levels I think, I have no way to know > exactly > how many levels are used for a given case. > I can reformulate my question in this way: does boomeramg coarsen the > "grid" to the coarsest level possible, i.e. 1 cell over the domain, or > does it stop before that? (grid is not the right term when it comes to > algebraic multilevel methods, but I can't find any other clear way to > express it). How can I control the coarsening? > > Thinking about it, perhaps boomeramd actually refines the grid up > to the > maximum number of levels...Is it so? > > Thanks for your help. > > Nicolas > >> Nicolas, >> >> You should try the option -pc_hypre_boomeramg_max_levels in the line >> command. In order to see the level's details you can use the PESTc >> option >> -ksp_view. >> >> Best Regards >> >> Fernando. >> >> >> Em (11:15:45), petsc-users at mcs.anl.gov escreveu: >> >> >>> Hi Petsc users, >>> >>> I have a question regarding the use of Hypre and especially >>> Boomeramg. >>> How can we control the number of levels in the multigrid >>> algortihm? Can >>> we >>> chose it, or if not, can we display it in some way? >>> >>> Best regards, >>> >>> Nicolas. >>> >>> -- >>> Nicolas BATHFIELD >>> Chalmers University of Technology >>> Shipping and Marine Technology >>> phone: +46 (0)31 772 1476 >>> fax: +46 (0)31 772 3699 >>> >>> ---------- >> >> >> > > > -- > Nicolas BATHFIELD > Chalmers University of Technology > Shipping and Marine Technology > phone: +46 (0)31 772 1476 > fax: +46 (0)31 772 3699 > From nicolas.bathfield at chalmers.se Thu Jan 25 09:42:05 2007 From: nicolas.bathfield at chalmers.se (Nicolas Bathfield) Date: Thu, 25 Jan 2007 16:42:05 +0100 (CET) Subject: PETSC-Hypre: number of level with boomeramg In-Reply-To: References: <20070125_113352_038707.asdfgf@ig.com.br> <5281.193.183.3.2.1169735500.squirrel@webmail.chalmers.se> Message-ID: <5589.193.183.3.2.1169739725.squirrel@webmail.chalmers.se> Thanks, the syntax is correct and it shows everything I need! Best regards, Nicolas > Try -pc_hypre_boomeramg_print_statistics in the console for more > information regarding your AMG. > > I believe that is the right syntax. I am not 100% though! It is a > long time ago I touched it. > > With best regards, Shaman Mahmoudi > > On Jan 25, 2007, at 3:31 PM, Nicolas Bathfield wrote: > >> Dear Fernando, >> >> Thanks for your reply. >> >> I had notice the value maxlevels given by -ksp_view, but I did not >> know >> how to change this value. >> >> Still, as the default is 25 levels I think, I have no way to know >> exactly >> how many levels are used for a given case. >> I can reformulate my question in this way: does boomeramg coarsen the >> "grid" to the coarsest level possible, i.e. 1 cell over the domain, or >> does it stop before that? (grid is not the right term when it comes to >> algebraic multilevel methods, but I can't find any other clear way to >> express it). How can I control the coarsening? >> >> Thinking about it, perhaps boomeramd actually refines the grid up >> to the >> maximum number of levels...Is it so? >> >> Thanks for your help. >> >> Nicolas >> >>> Nicolas, >>> >>> You should try the option -pc_hypre_boomeramg_max_levels in the line >>> command. In order to see the level's details you can use the PESTc >>> option >>> -ksp_view. >>> >>> Best Regards >>> >>> Fernando. >>> >>> >>> Em (11:15:45), petsc-users at mcs.anl.gov escreveu: >>> >>> >>>> Hi Petsc users, >>>> >>>> I have a question regarding the use of Hypre and especially >>>> Boomeramg. >>>> How can we control the number of levels in the multigrid >>>> algortihm? Can >>>> we >>>> chose it, or if not, can we display it in some way? >>>> >>>> Best regards, >>>> >>>> Nicolas. >>>> >>>> -- >>>> Nicolas BATHFIELD >>>> Chalmers University of Technology >>>> Shipping and Marine Technology >>>> phone: +46 (0)31 772 1476 >>>> fax: +46 (0)31 772 3699 >>>> >>>> ---------- >>> >>> >>> >> >> >> -- >> Nicolas BATHFIELD >> Chalmers University of Technology >> Shipping and Marine Technology >> phone: +46 (0)31 772 1476 >> fax: +46 (0)31 772 3699 >> > > -- Nicolas BATHFIELD Chalmers University of Technology Shipping and Marine Technology phone: +46 (0)31 772 1476 fax: +46 (0)31 772 3699 From asdfgf at ig.com.br Thu Jan 25 10:26:10 2007 From: asdfgf at ig.com.br (Fernando Campos) Date: Thu, 25 Jan 2007 14:26:10 -0200 Subject: PETSC-Hypre: number of level with boomeramg Message-ID: <20070125_162610_096123.asdfgf@ig.com.br> Dear Nicolas, It is a good question, but I don't know the answer... Why don't you try to set the maximum levels to a large numbers in order to see how many levels boomerAMG creates? I believe that it refines the grid up to the maximum number of levels as you sad. I am very interested in both geometric and algebraic multigrids and I also have a question: "Is it possible to obtain the coarse matrices of each level using boomerAMG?" Best Regards. Em (15:31:40), petsc-users at mcs.anl.gov escreveu: >Dear Fernando, > >Thanks for your reply. > >I had notice the value maxlevels given by -ksp_view, but I did not know >how to change this value. > >Still, as the default is 25 levels I think, I have no way to know exactly >how many levels are used for a given case. >I can reformulate my question in this way: does boomeramg coarsen the >"grid" to the coarsest level possible, i.e. 1 cell over the domain, or >does it stop before that? (grid is not the right term when it comes to >algebraic multilevel methods, but I can't find any other clear way to >express it). How can I control the coarsening? > >Thinking about it, perhaps boomeramd actually refines the grid up to the >maximum number of levels...Is it so? > >Thanks for your help. > >Nicolas > >> Nicolas, >> >> You should try the option -pc_hypre_boomeramg_max_levels in the line >> command. In order to see the level's details you can use the PESTc option >> -ksp_view. >> >> Best Regards >> >> Fernando. >> >> >> Em (11:15:45), petsc-users at mcs.anl.gov escreveu: >> >> >>>Hi Petsc users, >>> >>>I have a question regarding the use of Hypre and especially Boomeramg. >>>How can we control the number of levels in the multigrid algortihm? Can >>> we >>>chose it, or if not, can we display it in some way? >>> >>>Best regards, >>> >>>Nicolas. >>> >>>-- >>>Nicolas BATHFIELD >>>Chalmers University of Technology >>>Shipping and Marine Technology >>>phone: +46 (0)31 772 1476 >>>fax: +46 (0)31 772 3699 >>> >>>---------- >> >> >> > >-- >Nicolas BATHFIELD >Chalmers University of Technology >Shipping and Marine Technology >phone: +46 (0)31 772 1476 >fax: +46 (0)31 772 3699 > >---------- From yaron at oak-research.com Thu Jan 25 10:47:50 2007 From: yaron at oak-research.com (yaron at oak-research.com) Date: Thu, 25 Jan 2007 08:47:50 -0800 Subject: PETSC-Hypre: number of level with boomeramg Message-ID: <20070125164751.14146.qmail@s402.sureserver.com> Here's a thought- why not ask the Hypre guys? http://www.llnl.gov/CASC/linear_solvers/team.html Cheers Yaron -------Original Message------- From: Fernando Campos Subject: Re: PETSC-Hypre: number of level with boomeramg Sent: 25 Jan '07 08:26 Dear Nicolas, It is a good question, but I don't know the answer... Why don't you try to set the maximum levels to a large numbers in order to see how many levels boomerAMG creates? I believe that it refines the grid up to the maximum number of levels as you sad. I am very interested in both geometric and algebraic multigrids and I also have a question: "Is it possible to obtain the coarse matrices of each level using boomerAMG?" Best Regards. Em (15:31:40), [LINK: http://webmail.oak-research.com/compose.php?to=petsc-users at mcs.anl.gov] petsc-users at mcs.anl.gov escreveu: >Dear Fernando, > >Thanks for your reply. > >I had notice the value maxlevels given by -ksp_view, but I did not know >how to change this value. > >Still, as the default is 25 levels I think, I have no way to know exactly >how many levels are used for a given case. >I can reformulate my question in this way: does boomeramg coarsen the >"grid" to the coarsest level possible, i.e. 1 cell over the domain, or >does it stop before that? (grid is not the right term when it comes to >algebraic multilevel methods, but I can't find any other clear way to >express it). How can I control the coarsening? > >Thinking about it, perhaps boomeramd actually refines the grid up to the >maximum number of levels...Is it so? > >Thanks for your help. > >Nicolas > >> Nicolas, >> >> You should try the option -pc_hypre_boomeramg_max_levels in the line >> command. In order to see the level's details you can use the PESTc option >> -ksp_view. >> >> Best Regards >> >> Fernando. >> >> >> Em (11:15:45), [LINK: http://webmail.oak-research.com/compose.php?to=petsc-users at mcs.anl.gov] petsc-users at mcs.anl.gov escreveu: >> >> >>>Hi Petsc users, >>> >>>I have a question regarding the use of Hypre and especially Boomeramg. >>>How can we control the number of levels in the multigrid algortihm? Can >>> we >>>chose it, or if not, can we display it in some way? >>> >>>Best regards, >>> >>>Nicolas. >>> >>>-- >>>Nicolas BATHFIELD >>>Chalmers University of Technology >>>Shipping and Marine Technology >>>phone: +46 (0)31 772 1476 >>>fax: +46 (0)31 772 3699 >>> >>>---------- >> >> >> > >-- >Nicolas BATHFIELD >Chalmers University of Technology >Shipping and Marine Technology >phone: +46 (0)31 772 1476 >fax: +46 (0)31 772 3699 > >---------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jan 25 15:34:01 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 25 Jan 2007 15:34:01 -0600 (CST) Subject: PETSC-Hypre: number of level with boomeramg In-Reply-To: <20070125_162610_096123.asdfgf@ig.com.br> References: <20070125_162610_096123.asdfgf@ig.com.br> Message-ID: On Thu, 25 Jan 2007, Fernando Campos wrote: > Dear Nicolas, > > It is a good question, but I don't know the answer... > Why don't you try to set the maximum levels to a large numbers in order to > see how many levels boomerAMG creates? I believe that it refines the grid up > to the maximum number of levels as you sad. > > I am very interested in both geometric and algebraic multigrids and I also > have a question: "Is it possible to obtain the coarse matrices of each level > using boomerAMG?" In the PETSc the coarse matrices are left in hypre format so we do not have direct access to print them out. You would have to futz around in the hypre data structures to fine them (if you did this we'd love if you could contribute the code to print them out to PETSc.). Barry > > Best Regards. > > > Em (15:31:40), petsc-users at mcs.anl.gov escreveu: > > > >Dear Fernando, > > > >Thanks for your reply. > > > >I had notice the value maxlevels given by -ksp_view, but I did not know > >how to change this value. > > > >Still, as the default is 25 levels I think, I have no way to know exactly > >how many levels are used for a given case. > >I can reformulate my question in this way: does boomeramg coarsen the > >"grid" to the coarsest level possible, i.e. 1 cell over the domain, or > >does it stop before that? (grid is not the right term when it comes to > >algebraic multilevel methods, but I can't find any other clear way to > >express it). How can I control the coarsening? > > > >Thinking about it, perhaps boomeramd actually refines the grid up to the > >maximum number of levels...Is it so? > > > >Thanks for your help. > > > >Nicolas > > > >> Nicolas, > >> > >> You should try the option -pc_hypre_boomeramg_max_levels in the line > >> command. In order to see the level's details you can use the PESTc option > >> -ksp_view. > >> > >> Best Regards > >> > >> Fernando. > >> > >> > >> Em (11:15:45), petsc-users at mcs.anl.gov escreveu: > >> > >> > >>>Hi Petsc users, > >>> > >>>I have a question regarding the use of Hypre and especially Boomeramg. > >>>How can we control the number of levels in the multigrid algortihm? Can > >>> we > >>>chose it, or if not, can we display it in some way? > >>> > >>>Best regards, > >>> > >>>Nicolas. > >>> > >>>-- > >>>Nicolas BATHFIELD > >>>Chalmers University of Technology > >>>Shipping and Marine Technology > >>>phone: +46 (0)31 772 1476 > >>>fax: +46 (0)31 772 3699 > >>> > >>>---------- > >> > >> > >> > > > >-- > >Nicolas BATHFIELD > >Chalmers University of Technology > >Shipping and Marine Technology > >phone: +46 (0)31 772 1476 > >fax: +46 (0)31 772 3699 > > > >---------- > > > From zonexo at gmail.com Thu Jan 25 21:02:19 2007 From: zonexo at gmail.com (Ben Tay) Date: Fri, 26 Jan 2007 11:02:19 +0800 Subject: Using PETSc in structured c-grid for CFD and multigrid Message-ID: <804ab5d40701251902x31cd3d29ye97ce5c0b2924e4d@mail.gmail.com> Hi, I was discussing with another user in another forum (cfd-online.com) about using PETSc in my cfd code. I am now using KSP to solve my momentum and poisson eqn by inserting values into the matrix. I was told that using PETSc this way is only for unstructured grids. It is very inefficient and much slower if I'm using it for my structured grid because I am not exploiting the regular structure of my grid. Is that true? I'm solving flow around airfoil using c-grid. So how can I improve? Is it by using DA? I took a glance and it seems quite complicated. Also, is multigrid available in PETSc? Chapter 7 discusses about it but it seems very brief. Is there a more elaborate tutorial besides that c examples? Hope someone can give me some ideas. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Thu Jan 25 21:12:03 2007 From: zonexo at gmail.com (Ben Tay) Date: Fri, 26 Jan 2007 11:12:03 +0800 Subject: Is MKL linked statically or shared? Message-ID: <804ab5d40701251912y18a8c60ayfcd7a13d2849122a@mail.gmail.com> Hi, My school's server only has MKL installed on some nodes. Hence I was told that if I need to use it, I'll have to link them statically. So how does PETSc use MKL? Btw, I'm currently using shared PETSc library. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Jan 26 00:17:51 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 26 Jan 2007 00:17:51 -0600 (CST) Subject: Is MKL linked statically or shared? In-Reply-To: <804ab5d40701251912y18a8c60ayfcd7a13d2849122a@mail.gmail.com> References: <804ab5d40701251912y18a8c60ayfcd7a13d2849122a@mail.gmail.com> Message-ID: On Fri, 26 Jan 2007, Ben Tay wrote: > My school's server only has MKL installed on some nodes. Hence I > was told that if I need to use it, I'll have to link them > statically. So how does PETSc use MKL? I guess the question is - how do you work arround the problem [of badly installed MKL] when using PETSc. For this you need to understand a little bit about how compilers work. [wrt linking shared vs static libraries] When you specify: '-L/foo -lbar' the compiler first looks for The library 'bar' in /foo. Here it first looks for the shared version [i.e libbar.so]. If this version is not found - then it looks for the static version. [i.e libbar.a] However if you wish to force using the linker to use static version of library [when the shared version is also present] - you have the following options: - specifiy the library as /foo/libbar.a in the link command. - force linker to only use static libraries [for eg: 'gcc -static']. But then - if you don't have static version for any library [in the list of dependent libraries] - this link fails. - specify linker options to mix static & sharedlibs. For ex: 'gcc -Wl,-Bstatic -lfoo -Wl,-Bdynamic -lbar'. Neither of these approaches are supported by configure [as configure assumes '-L/foo -lbar' always works]. So you'll have to edit bmake/$PETSC_ARCH/petscconf - and make this change to the required libraries manually. For eg: I have the following in petscconf.h BLASLAPACK_LIB = -llapack -lblas The change would be either of the following [assumging gcc as the compiler]: BLASLAPACK_LIB = /usr/lib/liblapack.a /usr/lib/libblas.a or BLASLAPACK_LIB = -Wl,-Bstatic -llapack -lblas -Wl,-Bdynamic > Btw, I'm currently using shared PETSc library. If you make changes to the link options [as above] then you'll have to rebuild the PETSc shared libraries [with the following command] make deleteshared shared BTW: If PETSc sharedlibs are working across all machines [i.e PETSc install is on common file system across all of them] - then you can simply copy all the missing .so files into $PETSC_DIR/lib/$PETSC_ARCH - and everything should just work. cp /opt/intel/mkl/lib/ia32/*.so ${PETSC_DIR}/lib/${PETSC_ARCH] Satish From knepley at gmail.com Fri Jan 26 09:20:07 2007 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 26 Jan 2007 09:20:07 -0600 Subject: Using PETSc in structured c-grid for CFD and multigrid In-Reply-To: <804ab5d40701251902x31cd3d29ye97ce5c0b2924e4d@mail.gmail.com> References: <804ab5d40701251902x31cd3d29ye97ce5c0b2924e4d@mail.gmail.com> Message-ID: On 1/25/07, Ben Tay wrote: > > Hi, > > I was discussing with another user in another forum (cfd-online.com) about > using PETSc in my cfd code. I am now using KSP to solve my momentum and > poisson eqn by inserting values into the matrix. I was told that using PETSc > this way is only for unstructured grids. It is very inefficient and much > slower if I'm using it for my structured grid because I am not exploiting > the regular structure of my grid. > > Is that true? I'm solving flow around airfoil using c-grid. > If you are using a Cartesian grid, the DA formulation is better. However, this nonsense from people about "massive inefficiency" is just crap. There is a small difference in time which is almost trivial. However, there is a big difference in ease of programming. I am always fascinated how people can magnify small problems in order to preserve their job. So how can I improve? Is it by using DA? I took a glance and it seems quite > complicated. > Wrong glance. Much simpler. Also, is multigrid available in PETSc? Chapter 7 discusses about it but it > seems very brief. Is there a more elaborate tutorial besides that c > examples? > Yes, use DMMG. Much easier with DA. Matt Hope someone can give me some ideas. > > Thank you. > > > -- One trouble is that despite this system, anyone who reads journals widely and critically is forced to realize that there are scarcely any bars to eventual publication. There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print. -- Drummond Rennie -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Fri Jan 26 09:58:45 2007 From: zonexo at gmail.com (Ben Tay) Date: Fri, 26 Jan 2007 23:58:45 +0800 Subject: Using PETSc in structured c-grid for CFD and multigrid In-Reply-To: References: <804ab5d40701251902x31cd3d29ye97ce5c0b2924e4d@mail.gmail.com> Message-ID: <804ab5d40701260758n558d3d2dsc8e5a197e3fefd9d@mail.gmail.com> Hi, just to clarify. I'm using structured grid but not cartesian ie strictly horizontal/vertical. So I can't DA, can I? But can I use DMMG? So PETSc comes with a multigrid preconditioner and solver? Thank you On 1/26/07, Matthew Knepley wrote: > > On 1/25/07, Ben Tay wrote: > > > > Hi, > > > > I was discussing with another user in another forum (cfd-online.com) > > about using PETSc in my cfd code. I am now using KSP to solve my momentum > > and poisson eqn by inserting values into the matrix. I was told that using > > PETSc this way is only for unstructured grids. It is very inefficient and > > much slower if I'm using it for my structured grid because I am not > > exploiting the regular structure of my grid. > > > > Is that true? I'm solving flow around airfoil using c-grid. > > > > If you are using a Cartesian grid, the DA formulation is better. > However, this nonsense from people about > "massive inefficiency" is just crap. There is a small difference in time > which is almost trivial. However, there > is a big difference in ease of programming. I am always fascinated how > people can magnify small problems > in order to preserve their job. > > > So how can I improve? Is it by using DA? I took a glance and it seems > > quite complicated. > > > > Wrong glance. Much simpler. > > > Also, is multigrid available in PETSc? Chapter 7 discusses about it but > > it seems very brief. Is there a more elaborate tutorial besides that c > > examples? > > > > Yes, use DMMG. Much easier with DA. > > Matt > > > Hope someone can give me some ideas. > > > > Thank you. > > > > > > > > > > -- > One trouble is that despite this system, anyone who reads journals widely > and critically is forced to realize that there are scarcely any bars to > eventual > publication. There seems to be no study too fragmented, no hypothesis too > trivial, no literature citation too biased or too egotistical, no design > too > warped, no methodology too bungled, no presentation of results too > inaccurate, too obscure, and too contradictory, no analysis too > self-serving, > no argument too circular, no conclusions too trifling or too unjustified, > and > no grammar and syntax too offensive for a paper to end up in print. -- > Drummond Rennie -------------- next part -------------- An HTML attachment was scrubbed... URL: From berend at chalmers.se Fri Jan 26 10:01:00 2007 From: berend at chalmers.se (Berend van Wachem) Date: Fri, 26 Jan 2007 17:01:00 +0100 Subject: Using PETSc in structured c-grid for CFD and multigrid In-Reply-To: <804ab5d40701260758n558d3d2dsc8e5a197e3fefd9d@mail.gmail.com> References: <804ab5d40701251902x31cd3d29ye97ce5c0b2924e4d@mail.gmail.com> <804ab5d40701260758n558d3d2dsc8e5a197e3fefd9d@mail.gmail.com> Message-ID: <200701261701.00939.berend@chalmers.se> Hi Ben, Sure you can - it just means you will need to save the X,Y,Z locations of the grid points as well. All a DA requires is a regular I,J,K numbering over your grid. Berend. > Hi, > > just to clarify. I'm using structured grid but not cartesian ie > strictly horizontal/vertical. So I can't DA, can I? > > But can I use DMMG? So PETSc comes with a multigrid preconditioner > and solver? > > Thank you > > On 1/26/07, Matthew Knepley wrote: > > On 1/25/07, Ben Tay wrote: > > > Hi, > > > > > > I was discussing with another user in another forum > > > (cfd-online.com) about using PETSc in my cfd code. I am now > > > using KSP to solve my momentum and poisson eqn by inserting > > > values into the matrix. I was told that using PETSc this way is > > > only for unstructured grids. It is very inefficient and much > > > slower if I'm using it for my structured grid because I am not > > > exploiting the regular structure of my grid. > > > > > > Is that true? I'm solving flow around airfoil using c-grid. > > > > If you are using a Cartesian grid, the DA formulation is better. > > However, this nonsense from people about > > "massive inefficiency" is just crap. There is a small difference > > in time which is almost trivial. However, there > > is a big difference in ease of programming. I am always fascinated > > how people can magnify small problems > > in order to preserve their job. > > > > > > So how can I improve? Is it by using DA? I took a glance and it > > seems > > > > > quite complicated. > > > > Wrong glance. Much simpler. > > > > > > Also, is multigrid available in PETSc? Chapter 7 discusses about > > it but > > > > > it seems very brief. Is there a more elaborate tutorial besides > > > that c examples? > > > > Yes, use DMMG. Much easier with DA. > > > > Matt > > > > > > Hope someone can give me some ideas. > > > > > Thank you. > > > > -- > > One trouble is that despite this system, anyone who reads journals > > widely and critically is forced to realize that there are scarcely > > any bars to eventual > > publication. There seems to be no study too fragmented, no > > hypothesis too trivial, no literature citation too biased or too > > egotistical, no design too > > warped, no methodology too bungled, no presentation of results too > > inaccurate, too obscure, and too contradictory, no analysis too > > self-serving, > > no argument too circular, no conclusions too trifling or too > > unjustified, and > > no grammar and syntax too offensive for a paper to end up in > > print. -- Drummond Rennie From berend at tfd.chalmers.se Fri Jan 26 01:53:19 2007 From: berend at tfd.chalmers.se (Berend van Wachem) Date: Fri, 26 Jan 2007 08:53:19 +0100 (CET) Subject: Using PETSc in structured c-grid for CFD and multigrid In-Reply-To: <804ab5d40701251902x31cd3d29ye97ce5c0b2924e4d@mail.gmail.com> References: <804ab5d40701251902x31cd3d29ye97ce5c0b2924e4d@mail.gmail.com> Message-ID: <64437.85.224.167.125.1169797999.squirrel@webmail.chalmers.se> Hi, I am not an expert - but have used PETSc for both structured and unstructured grids. When you use an unstructured code for a structured grid, there is additional overhead (addressing, connectivity) which is redundant; this information is not required for solving on a structured grid. I would say this is maximum a 10% efficiency loss for bigger problems - it does not affect solving the matrix, only in gathering your coefficients. I would not rewrite my CFD code for this. If you only deal with structured grids, using the PETSc DA framework should work for you - you are not saving all connectivity. The DA framework is not difficult at all, according to my opinion. Look at a few examples that come with PETSc. I use a block structured solver - using multiple DA's within one problem. Let me know if you are interested in this, and I can send you parts of code. Multigrid is certainly possible (I would reccomend through HYPRE, discussed on the mailinglist, although I still have problems with it), but the question is how efficient it will be for your CFD problem. For an efficient multigrid in CFD, it is important to consider the coefficient structure arising from the momentum equations - the grouping of cells should occur following the advection term. Only then will you achieve linear scaling with the problem size. For instance, consider a rotating flow in a square box. Most multigrid algorithms will group cells in "squares" which will not lead to a significant improvement, as the flow (advection, pressure grad) does not move in these squares. In fact, to have an efficient multigrid algorithm, the cels should be grouped along the circular flow. As this cannot be seen directly from the pressure coefficients, I doubt any "automatic" multigrid algorithm (in Hypre or Petsc) would be able to capture this, but don't quote me on it - I am not 100% sure. So concluding, if you want to do efficient multigridding for CFD, you will need to point out which cells are grouped into which structure, based upon the upwind advection coefficients. Good luck, Berend. > Hi, > > I was discussing with another user in another forum (cfd-online.com) about > using PETSc in my cfd code. I am now using KSP to solve my momentum and > poisson eqn by inserting values into the matrix. I was told that using > PETSc > this way is only for unstructured grids. It is very inefficient and much > slower if I'm using it for my structured grid because I am not exploiting > the regular structure of my grid. > > Is that true? I'm solving flow around airfoil using c-grid. > > So how can I improve? Is it by using DA? I took a glance and it seems > quite > complicated. > > Also, is multigrid available in PETSc? Chapter 7 discusses about it but it > seems very brief. Is there a more elaborate tutorial besides that c > examples? > > Hope someone can give me some ideas. > > Thank you. > From zonexo at gmail.com Fri Jan 26 19:54:20 2007 From: zonexo at gmail.com (Ben Tay) Date: Sat, 27 Jan 2007 09:54:20 +0800 Subject: Using PETSc in structured c-grid for CFD and multigrid In-Reply-To: <64437.85.224.167.125.1169797999.squirrel@webmail.chalmers.se> References: <804ab5d40701251902x31cd3d29ye97ce5c0b2924e4d@mail.gmail.com> <64437.85.224.167.125.1169797999.squirrel@webmail.chalmers.se> Message-ID: <804ab5d40701261754p240a652brc7b112b04b0963b0@mail.gmail.com> Thank you Berend! I'll go through DA again. I'm also looking at HYPRE. Its way of creating grids and linking them seems intuitive. Btw, is there a mailing list for HYPRE similar to PETSc to discuss problems? I find that their explanation are quite brief. I tried to install HYPRE 2.0 on windows using cygwin but it failed. I then install it as an external software thru PETSc. I think it's installing HYPRE 1.0 or something. But similarly, there's illegal operation. Installing HYPRE 2.0 on my school's linux worked, though there's seems to be some minor error. So what's the best way to employ multigird? Is it to install as an external software thru PETSc or just use HYPRE on its own? Btw, it will be great if you can send me parts of your code regarding DA. Thank you very much! On 1/26/07, Berend van Wachem wrote: > > Hi, > > I am not an expert - but have used PETSc for both structured and > unstructured grids. > > When you use an unstructured code for a structured grid, there is > additional overhead (addressing, connectivity) which is redundant; this > information is not required for solving on a structured grid. I would say > this is maximum a 10% efficiency loss for bigger problems - it does not > affect solving the matrix, only in gathering your coefficients. I would > not rewrite my CFD code for this. > > If you only deal with structured grids, using the PETSc DA framework > should work for you - you are not saving all connectivity. The DA > framework is not difficult at all, according to my opinion. Look at a few > examples that come with PETSc. I use a block structured solver - using > multiple DA's within one problem. Let me know if you are interested in > this, and I can send you parts of code. > > Multigrid is certainly possible (I would reccomend through HYPRE, > discussed on the mailinglist, although I still have problems with it), but > the question is how efficient it will be for your CFD problem. For an > efficient multigrid in CFD, it is important to consider the coefficient > structure arising from the momentum equations - the grouping of cells > should occur following the advection term. Only then will you achieve > linear scaling with the problem size. For instance, consider a rotating > flow in a square box. Most multigrid algorithms will group cells in > "squares" which will not lead to a significant improvement, as the flow > (advection, pressure grad) does not move in these squares. In fact, to > have an efficient multigrid algorithm, the cels should be grouped along > the circular flow. As this cannot be seen directly from the pressure > coefficients, I doubt any "automatic" multigrid algorithm (in Hypre or > Petsc) would be able to capture this, but don't quote me on it - I am not > 100% sure. So concluding, if you want to do efficient multigridding for > CFD, you will need to point out which cells are grouped into which > structure, based upon the upwind advection coefficients. > > Good luck, > > Berend. > > > > Hi, > > > > I was discussing with another user in another forum (cfd-online.com) > about > > using PETSc in my cfd code. I am now using KSP to solve my momentum and > > poisson eqn by inserting values into the matrix. I was told that using > > PETSc > > this way is only for unstructured grids. It is very inefficient and much > > slower if I'm using it for my structured grid because I am not > exploiting > > the regular structure of my grid. > > > > Is that true? I'm solving flow around airfoil using c-grid. > > > > So how can I improve? Is it by using DA? I took a glance and it seems > > quite > > complicated. > > > > Also, is multigrid available in PETSc? Chapter 7 discusses about it but > it > > seems very brief. Is there a more elaborate tutorial besides that c > > examples? > > > > Hope someone can give me some ideas. > > > > Thank you. > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Fri Jan 26 21:11:26 2007 From: zonexo at gmail.com (Ben Tay) Date: Sat, 27 Jan 2007 11:11:26 +0800 Subject: Lapack/blas and Gotoblas Message-ID: <804ab5d40701261911p32c963b7w7ee02bd471e6bd70@mail.gmail.com> Hi, My school's server has lapack and blas installed. However, I can't get it to be automatically detected by using ./configure. what is the filename that I should used? Is it liblapack.a and libblas.a? I can't find libblas.a in the /usr/lib directory though. Is there another filename for it? I found that if I specify just the dir (/usr/lib) or liblapack.a for both the lapack and blas lib, it won't work Btw, can I use GotoBlas? I tried to specify libgoto.a or libgoto_northwoodp- r1.10.so and I managed to compile. However, during testing, the error msg are for libgoto.a : /nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-nompi-goto/libpetscksp.so: undefined reference to `pthread_create' /nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-nompi-goto/libpetscksp.so: undefined reference to `dpotrs_' /nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-nompi-goto/libpetscksp.so: undefined reference to `pthread_join' for libgoto_northwoodp-r1.10.so : /nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-nompi-goto/libpetscksp.so: undefined reference to `dpotrs_' That was linked for shared library. If it is for static library, the error is: /nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-nompi-goto/libpetscksp.a( bcgsl.o)(.text+0x103a): In function `KSPSolve_BCGSL': : undefined reference to `dpotrs_' /nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-nompi-goto/libpetscksp.a( bcgsl.o)(.text+0x10f3): In function `KSPSolve_BCGSL': : undefined reference to `dpotrs_' /nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-nompi-goto/libpetscksp.a( bcgsl.o)(.text+0x116f): In function `KSPSolve_BCGSL': : undefined reference to `dpotrs_' /lsftmp/g0306332/GotoBLAS/libgoto.a(blas_server.o)(.text+0x370): In function `blas_thread_init': : undefined reference to `pthread_create' /lsftmp/g0306332/GotoBLAS/libgoto.a(blas_server.o)(.text+0x709): In function `blas_thread_shutdown_': So can gotoblas be used? Thank you very much -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jan 26 21:27:17 2007 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 26 Jan 2007 21:27:17 -0600 Subject: Lapack/blas and Gotoblas In-Reply-To: <804ab5d40701261911p32c963b7w7ee02bd471e6bd70@mail.gmail.com> References: <804ab5d40701261911p32c963b7w7ee02bd471e6bd70@mail.gmail.com> Message-ID: This mail belongs on petsc-maint. On 1/26/07, Ben Tay wrote: > > Hi, > > My school's server has lapack and blas installed. However, I can't get it > to be automatically detected by using ./configure. what is the filename that > I should used? Is it liblapack.a and libblas.a? I can't find libblas.a in > the /usr/lib directory though. Is there another filename for it? I found > that if I specify just the dir (/usr/lib) or liblapack.a for both the > lapack and blas lib, it won't work > It looks like you do not have BLAS installed. Btw, can I use GotoBlas? I tried to specify libgoto.a or libgoto_northwoodp- > r1.10.so and I managed to compile. However, during testing, the error msg > are > > for libgoto.a : > > /nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-nompi-goto/libpetscksp.so: > undefined reference to `pthread_create' > /nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-nompi-goto/libpetscksp.so: > undefined reference to `dpotrs_' > /nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-nompi-goto/libpetscksp.so: > undefined reference to `pthread_join' > This library depends on pthreads (at least), so you have to specify it as well: --with-blas-lib=[libgoto.a,libpthread.a] for libgoto_northwoodp-r1.10.so : > > /nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-nompi-goto/libpetscksp.so: > undefined reference to `dpotrs_' > This is not a full BLAS. That was linked for shared library. If it is for static library, the error > is: > > /nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-nompi-goto/libpetscksp.a( > bcgsl.o)(.text+0x103a): In function `KSPSolve_BCGSL': > : undefined reference to `dpotrs_' > /nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-nompi-goto/libpetscksp.a( > bcgsl.o)(.text+0x10f3): In function `KSPSolve_BCGSL': > : undefined reference to `dpotrs_' > /nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-nompi-goto/libpetscksp.a( > bcgsl.o)(.text+0x116f): In function `KSPSolve_BCGSL': > : undefined reference to `dpotrs_' > /lsftmp/g0306332/GotoBLAS/libgoto.a(blas_server.o)(.text+0x370): In > function `blas_thread_init': > : undefined reference to `pthread_create' > /lsftmp/g0306332/GotoBLAS/libgoto.a(blas_server.o)(.text+0x709): In > function `blas_thread_shutdown_': > > So can gotoblas be used? > > Thank you very much > -- One trouble is that despite this system, anyone who reads journals widely and critically is forced to realize that there are scarcely any bars to eventual publication. There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print. -- Drummond Rennie -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Jan 26 21:30:45 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 26 Jan 2007 21:30:45 -0600 (CST) Subject: Using PETSc in structured c-grid for CFD and multigrid In-Reply-To: <804ab5d40701261754p240a652brc7b112b04b0963b0@mail.gmail.com> References: <804ab5d40701251902x31cd3d29ye97ce5c0b2924e4d@mail.gmail.com> <64437.85.224.167.125.1169797999.squirrel@webmail.chalmers.se> <804ab5d40701261754p240a652brc7b112b04b0963b0@mail.gmail.com> Message-ID: On Sat, 27 Jan 2007, Ben Tay wrote: > I tried to install HYPRE 2.0 on windows using cygwin but it failed. I then > install it as an external software thru PETSc. I think it's installing HYPRE > 1.0 or something. But similarly, there's illegal operation. If you are using hypre from PETSc - you'll have to install the compatible sources i.e use --download-hypre=1, otherwise you are likely to encounter problems. However I can't say that installer works on windows with MS compilers. If you wish to try installing hypre manually - you can grab the sources from the url documented in python/PETSc/hypre.py and try.. Satish From balay at mcs.anl.gov Fri Jan 26 21:33:52 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 26 Jan 2007 21:33:52 -0600 (CST) Subject: Lapack/blas and Gotoblas In-Reply-To: <804ab5d40701261911p32c963b7w7ee02bd471e6bd70@mail.gmail.com> References: <804ab5d40701261911p32c963b7w7ee02bd471e6bd70@mail.gmail.com> Message-ID: On Sat, 27 Jan 2007, Ben Tay wrote: > My school's server has lapack and blas installed. However, I can't > get it to be automatically detected by using ./configure. what is > the filename that I should used? what do you have for: ls /usr/lib/libblas* /usr/lib/liblapack* > Btw, can I use GotoBlas? I tried to specify libgoto.a or > libgoto_northwoodp- r1.10.so and I managed to compile. However, > during testing, the error msg are yeah - you can use gotoblas - but you'll still need lapack. > /nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-nompi-goto/libpetscksp.so: > undefined reference to `pthread_join' Lools like goto requires -lpthread Satish From zonexo at gmail.com Fri Jan 26 21:56:11 2007 From: zonexo at gmail.com (Ben Tay) Date: Sat, 27 Jan 2007 11:56:11 +0800 Subject: Lapack/blas and Gotoblas In-Reply-To: References: <804ab5d40701261911p32c963b7w7ee02bd471e6bd70@mail.gmail.com> Message-ID: <804ab5d40701261956o49ffd95dh791c0f517e9eedb6@mail.gmail.com> Hi, The result I get for lapack is /usr/lib/liblapack.a /usr/lib/liblapack.so /usr/lib/liblapack.so.3 /usr/lib/liblapack.so.3.0 /usr/lib/liblapack.so.3.0.3 but for blas, there's no match. However, my school's tutorial said lapack and blas can be used with the option -llapack -lblas. Is there some other directory where they are usually install? I'm wondering if I can use lapack + gotoblas together? I tried to specify the libraries but ./configure failed. Thank you. On 1/27/07, Satish Balay wrote: > > > On Sat, 27 Jan 2007, Ben Tay wrote: > > > My school's server has lapack and blas installed. However, I can't > > get it to be automatically detected by using ./configure. what is > > the filename that I should used? > > what do you have for: > > ls /usr/lib/libblas* /usr/lib/liblapack* > > > Btw, can I use GotoBlas? I tried to specify libgoto.a or > > libgoto_northwoodp- r1.10.so and I managed to compile. However, > > during testing, the error msg are > > yeah - you can use gotoblas - but you'll still need lapack. > > > /nas/lsftmp/g0306332/petsc-2.3.2-p8/lib/linux-nompi-goto/libpetscksp.so: > > undefined reference to `pthread_join' > > Lools like goto requires -lpthread > > Satish > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Jan 26 22:02:47 2007 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 26 Jan 2007 22:02:47 -0600 (CST) Subject: Lapack/blas and Gotoblas In-Reply-To: <804ab5d40701261956o49ffd95dh791c0f517e9eedb6@mail.gmail.com> References: <804ab5d40701261911p32c963b7w7ee02bd471e6bd70@mail.gmail.com> <804ab5d40701261956o49ffd95dh791c0f517e9eedb6@mail.gmail.com> Message-ID: On Sat, 27 Jan 2007, Ben Tay wrote: > The result I get for lapack is > > /usr/lib/liblapack.a /usr/lib/liblapack.so /usr/lib/liblapack.so.3 > /usr/lib/liblapack.so.3.0 /usr/lib/liblapack.so.3.0.3 > > but for blas, there's no match. However, my school's tutorial said lapack > and blas can be used with the option -llapack -lblas. Looks like they messed up. > > Is there some other directory where they are usually install? > > I'm wondering if I can use lapack + gotoblas together? I tried to specify > the libraries but ./configure failed. Did you try Matt's sugestion on how to use gotoblas? --with-blas-lib=[libgoto.a,libpthread.a] If you continue to have problems - send configure.log to petsc-maint at mcs.anl.gov Satish From dalcinl at gmail.com Sat Jan 27 15:33:12 2007 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 27 Jan 2007 18:33:12 -0300 Subject: Using PETSc in structured c-grid for CFD and multigrid In-Reply-To: <64437.85.224.167.125.1169797999.squirrel@webmail.chalmers.se> References: <804ab5d40701251902x31cd3d29ye97ce5c0b2924e4d@mail.gmail.com> <64437.85.224.167.125.1169797999.squirrel@webmail.chalmers.se> Message-ID: On 1/26/07, Berend van Wachem wrote: > Multigrid is certainly possible (I would reccomend through HYPRE, > discussed on the mailinglist, although I still have problems with it), but > the question is how efficient it will be for your CFD problem. >I doubt any "automatic" multigrid algorithm (in Hypre or > Petsc) would be able to capture this, but don't quote me on it - I am not > 100% sure. Indeed. In my recent experience trying to use BoomerAMG (with fully coupled, implicit FEM SUPG/PSPG formulation for incompressible NS eqs.), the results were quite disappointing. I am now trying to implement preconditioners based on approximate the Schur complement on the pressure. This decouples the full problem, but it need to solve an advection-diffusion-reaction problem and Poisson problems (such that multigrid can work well oin them) each time the preconditioner is applied. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From aja2111 at columbia.edu Sat Jan 27 17:41:02 2007 From: aja2111 at columbia.edu (Aron Ahmadia) Date: Sat, 27 Jan 2007 18:41:02 -0500 Subject: PETSc In-Reply-To: References: Message-ID: <37604ab40701271541yf0a0e7at9b9bf0fb01f1ac09@mail.gmail.com> Hi Brian, I took a quick peek at the source for DAGetGlobalVector in the current release: " for (i=0; iglobalin[i]) { *g = da->globalin[i]; da->globalin[i] = PETSC_NULL; goto alldone; } } ierr = DACreateGlobalVector(da,g);CHKERRQ(ierr); alldone: for (i=0; iglobalout[i]) { da->globalout[i] = *g; break; } " The two appear to be very similar, with this exception. Get/Restore work with one copy of the DA's global vector, but will make a copy if one has already been checked out or none exist. DACreateGlobalVector will always make a copy of the global vector, regardless of the circumstances. I think you're better off using Get and Restore (less data copying) unless you'd like to work with multiple copies for some reason. ~A On 1/27/07, Brian Grierson wrote: > > Could you please tell me the difference between > > DACreateGlobalVector / VecDestroy > and > DAGetGlobalVector / DARestoreGlobalVector > > > > > > > > > > > > Brian A. Grierson > > > > > Department of Applied Physics > > Columbia University > > Mobile: 646.259.1038 > > Work: 212-854-4839 > > bag2107 at columbia.edu > > bgrierson21 at gmail.com > > http://www.columbia.edu/~bag2107/http://www.apam.columbia.edu/ctx/ctx.html > > From aja2111 at columbia.edu Sat Jan 27 18:09:49 2007 From: aja2111 at columbia.edu (Aron Ahmadia) Date: Sat, 27 Jan 2007 19:09:49 -0500 Subject: PETSc In-Reply-To: <869FBD90-4026-47B5-B4DD-939B97912115@columbia.edu> References: <37604ab40701271541yf0a0e7at9b9bf0fb01f1ac09@mail.gmail.com> <869FBD90-4026-47B5-B4DD-939B97912115@columbia.edu> Message-ID: <37604ab40701271609y4e9e5d0by9f02a5996bafe24b@mail.gmail.com> Sounds like a documentation bug to me, are you referring to this? http://www-unix.mcs.anl.gov/petsc/petsc-2/snapshots/petsc-current/docs/manualpages/DA/DARestoreGlobalVector.html#DARestoreGlobalVector Perhaps I'm misunderstanding how Get/Restore are being used. I'll let one of the developers chime in on this when they've got a moment. ~A On 1/27/07, Brian Grierson wrote: > Another quick question. PETSc online help says DAGetGlobal/DARestoreGlobal > is a Seq vector operation, while > DACreateGlobal/VecDesroy is parallel. > > I don't understand how you could create a global vector in a Seq manner on a > DA??? > > > > Brian A. Grierson > > > > > Department of Applied Physics > > Columbia University > > Mobile: 646.259.1038 > > Work: 212-854-4839 > > bag2107 at columbia.edu > > bgrierson21 at gmail.com > > http://www.columbia.edu/~bag2107/http://www.apam.columbia.edu/ctx/ctx.html > > > On Jan 27, 2007, at 6:41 PM, Aron Ahmadia wrote: > > Hi Brian, > > I took a quick peek at the source for DAGetGlobalVector in the current > release: > > " > for (i=0; i if (da->globalin[i]) { > *g = da->globalin[i]; > da->globalin[i] = PETSC_NULL; > goto alldone; > } > } > ierr = DACreateGlobalVector(da,g);CHKERRQ(ierr); > > alldone: > for (i=0; i if (!da->globalout[i]) { > da->globalout[i] = *g; > break; > } > " > > The two appear to be very similar, with this exception. Get/Restore > work with one copy of the DA's global vector, but will make a copy if > one has already been checked out or none exist. > > DACreateGlobalVector will always make a copy of the global vector, > regardless of the circumstances. I think you're better off using Get > and Restore (less data copying) unless you'd like to work with > multiple copies for some reason. > > ~A > > On 1/27/07, Brian Grierson wrote: > > Could you please tell me the difference between > > DACreateGlobalVector / VecDestroy > and > DAGetGlobalVector / DARestoreGlobalVector > > > > > > > > > > > > Brian A. Grierson > > > > > Department of Applied Physics > > Columbia University > > Mobile: 646.259.1038 > > Work: 212-854-4839 > > bag2107 at columbia.edu > > bgrierson21 at gmail.com > > http://www.columbia.edu/~bag2107/http://www.apam.columbia.edu/ctx/ctx.html > > > > From knepley at gmail.com Sat Jan 27 18:24:56 2007 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 27 Jan 2007 18:24:56 -0600 Subject: PETSc In-Reply-To: <37604ab40701271609y4e9e5d0by9f02a5996bafe24b@mail.gmail.com> References: <37604ab40701271541yf0a0e7at9b9bf0fb01f1ac09@mail.gmail.com> <869FBD90-4026-47B5-B4DD-939B97912115@columbia.edu> <37604ab40701271609y4e9e5d0by9f02a5996bafe24b@mail.gmail.com> Message-ID: On 1/27/07, Aron Ahmadia wrote: > > Sounds like a documentation bug to me, are you referring to this? > > http://www-unix.mcs.anl.gov/petsc/petsc-2/snapshots/petsc-current/docs/manualpages/DA/DARestoreGlobalVector.html#DARestoreGlobalVector > > Perhaps I'm misunderstanding how Get/Restore are being used. I'll let > one of the developers chime in on this when they've got a moment. This is a cut&paste error from GetLocalVector(). Matt ~A > > On 1/27/07, Brian Grierson wrote: > > Another quick question. PETSc online help says > DAGetGlobal/DARestoreGlobal > > is a Seq vector operation, while > > DACreateGlobal/VecDesroy is parallel. > > > > I don't understand how you could create a global vector in a Seq manner > on a > > DA??? > > > > > > > > Brian A. Grierson > > > > > > > > > > Department of Applied Physics > > > > Columbia University > > > > Mobile: 646.259.1038 > > > > Work: 212-854-4839 > > > > bag2107 at columbia.edu > > > > bgrierson21 at gmail.com > > > > > http://www.columbia.edu/~bag2107/http://www.apam.columbia.edu/ctx/ctx.html > > > > > > On Jan 27, 2007, at 6:41 PM, Aron Ahmadia wrote: > > > > Hi Brian, > > > > I took a quick peek at the source for DAGetGlobalVector in the current > > release: > > > > " > > for (i=0; i > if (da->globalin[i]) { > > *g = da->globalin[i]; > > da->globalin[i] = PETSC_NULL; > > goto alldone; > > } > > } > > ierr = DACreateGlobalVector(da,g);CHKERRQ(ierr); > > > > alldone: > > for (i=0; i > if (!da->globalout[i]) { > > da->globalout[i] = *g; > > break; > > } > > " > > > > The two appear to be very similar, with this exception. Get/Restore > > work with one copy of the DA's global vector, but will make a copy if > > one has already been checked out or none exist. > > > > DACreateGlobalVector will always make a copy of the global vector, > > regardless of the circumstances. I think you're better off using Get > > and Restore (less data copying) unless you'd like to work with > > multiple copies for some reason. > > > > ~A > > > > On 1/27/07, Brian Grierson wrote: > > > > Could you please tell me the difference between > > > > DACreateGlobalVector / VecDestroy > > and > > DAGetGlobalVector / DARestoreGlobalVector > > > > > > > > > > > > > > > > > > > > > > > > Brian A. Grierson > > > > > > > > > > Department of Applied Physics > > > > Columbia University > > > > Mobile: 646.259.1038 > > > > Work: 212-854-4839 > > > > bag2107 at columbia.edu > > > > bgrierson21 at gmail.com > > > > > http://www.columbia.edu/~bag2107/http://www.apam.columbia.edu/ctx/ctx.html > > > > > > > > > > -- One trouble is that despite this system, anyone who reads journals widely and critically is forced to realize that there are scarcely any bars to eventual publication. There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print. -- Drummond Rennie -------------- next part -------------- An HTML attachment was scrubbed... URL: From aja2111 at columbia.edu Sat Jan 27 18:43:18 2007 From: aja2111 at columbia.edu (Aron Ahmadia) Date: Sat, 27 Jan 2007 19:43:18 -0500 Subject: PETSc In-Reply-To: References: <37604ab40701271541yf0a0e7at9b9bf0fb01f1ac09@mail.gmail.com> <869FBD90-4026-47B5-B4DD-939B97912115@columbia.edu> <37604ab40701271609y4e9e5d0by9f02a5996bafe24b@mail.gmail.com> Message-ID: <37604ab40701271643m6b92d32jb3d5498e95ce90b8@mail.gmail.com> Brian, Work vectors should never be created/deleted every time an application is run, they should be generated at the beginning and then accessed. In this case calling get/restore should be fine. ~A On 1/27/07, Matthew Knepley wrote: > On 1/27/07, Aron Ahmadia wrote: > > Sounds like a documentation bug to me, are you referring to this? > > > http://www-unix.mcs.anl.gov/petsc/petsc-2/snapshots/petsc-current/docs/manualpages/DA/DARestoreGlobalVector.html#DARestoreGlobalVector > > > > Perhaps I'm misunderstanding how Get/Restore are being used. I'll let > > one of the developers chime in on this when they've got a moment. > > This is a cut&paste error from GetLocalVector(). > > Matt > > > ~A > > > > On 1/27/07, Brian Grierson < bag2107 at columbia.edu> wrote: > > > Another quick question. PETSc online help says > DAGetGlobal/DARestoreGlobal > > > is a Seq vector operation, while > > > DACreateGlobal/VecDesroy is parallel. > > > > > > I don't understand how you could create a global vector in a Seq manner > on a > > > DA??? > > > > > > > > > > > > Brian A. Grierson > > > > > > > > > > > > > > > Department of Applied Physics > > > > > > Columbia University > > > > > > Mobile: 646.259.1038 > > > > > > Work: 212-854-4839 > > > > > > bag2107 at columbia.edu > > > > > > bgrierson21 at gmail.com > > > > > > > http://www.columbia.edu/~bag2107/http://www.apam.columbia.edu/ctx/ctx.html > > > > > > > > > On Jan 27, 2007, at 6:41 PM, Aron Ahmadia wrote: > > > > > > Hi Brian, > > > > > > I took a quick peek at the source for DAGetGlobalVector in the current > > > release: > > > > > > " > > > for (i=0; i > > if (da->globalin[i]) { > > > *g = da->globalin[i]; > > > da->globalin[i] = PETSC_NULL; > > > goto alldone; > > > } > > > } > > > ierr = DACreateGlobalVector(da,g);CHKERRQ(ierr); > > > > > > alldone: > > > for (i=0; i > > if (!da->globalout[i]) { > > > da->globalout[i] = *g; > > > break; > > > } > > > " > > > > > > The two appear to be very similar, with this exception. Get/Restore > > > work with one copy of the DA's global vector, but will make a copy if > > > one has already been checked out or none exist. > > > > > > DACreateGlobalVector will always make a copy of the global vector, > > > regardless of the circumstances. I think you're better off using Get > > > and Restore (less data copying) unless you'd like to work with > > > multiple copies for some reason. > > > > > > ~A > > > > > > On 1/27/07, Brian Grierson < bag2107 at columbia.edu> wrote: > > > > > > Could you please tell me the difference between > > > > > > DACreateGlobalVector / VecDestroy > > > and > > > DAGetGlobalVector / DARestoreGlobalVector > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Brian A. Grierson > > > > > > > > > > > > > > > Department of Applied Physics > > > > > > Columbia University > > > > > > Mobile: 646.259.1038 > > > > > > Work: 212-854-4839 > > > > > > bag2107 at columbia.edu > > > > > > bgrierson21 at gmail.com > > > > > > > http://www.columbia.edu/~bag2107/http://www.apam.columbia.edu/ctx/ctx.html > > > > > > > > > > > > > > > > > > > > -- > One trouble is that despite this system, anyone who reads journals widely > and critically is forced to realize that there are scarcely any bars to > eventual > publication. There seems to be no study too fragmented, no hypothesis too > trivial, no literature citation too biased or too egotistical, no design too > warped, no methodology too bungled, no presentation of results too > inaccurate, too obscure, and too contradictory, no analysis too > self-serving, > no argument too circular, no conclusions too trifling or too unjustified, > and > no grammar and syntax too offensive for a paper to end up in print. -- > Drummond Rennie From aja2111 at columbia.edu Sat Jan 27 18:43:45 2007 From: aja2111 at columbia.edu (Aron Ahmadia) Date: Sat, 27 Jan 2007 19:43:45 -0500 Subject: PETSc In-Reply-To: <37604ab40701271643m6b92d32jb3d5498e95ce90b8@mail.gmail.com> References: <37604ab40701271541yf0a0e7at9b9bf0fb01f1ac09@mail.gmail.com> <869FBD90-4026-47B5-B4DD-939B97912115@columbia.edu> <37604ab40701271609y4e9e5d0by9f02a5996bafe24b@mail.gmail.com> <37604ab40701271643m6b92d32jb3d5498e95ce90b8@mail.gmail.com> Message-ID: <37604ab40701271643q6b74f7b0t82cef51f61479a02@mail.gmail.com> Sorry, that should have been subroutine, not application. ~A On 1/27/07, Aron Ahmadia wrote: > Brian, > > Work vectors should never be created/deleted every time an application > is run, they should be generated at the beginning and then accessed. > > In this case calling get/restore should be fine. > > ~A > > On 1/27/07, Matthew Knepley wrote: > > On 1/27/07, Aron Ahmadia wrote: > > > Sounds like a documentation bug to me, are you referring to this? > > > > > http://www-unix.mcs.anl.gov/petsc/petsc-2/snapshots/petsc-current/docs/manualpages/DA/DARestoreGlobalVector.html#DARestoreGlobalVector > > > > > > Perhaps I'm misunderstanding how Get/Restore are being used. I'll let > > > one of the developers chime in on this when they've got a moment. > > > > This is a cut&paste error from GetLocalVector(). > > > > Matt > > > > > ~A > > > > > > On 1/27/07, Brian Grierson < bag2107 at columbia.edu> wrote: > > > > Another quick question. PETSc online help says > > DAGetGlobal/DARestoreGlobal > > > > is a Seq vector operation, while > > > > DACreateGlobal/VecDesroy is parallel. > > > > > > > > I don't understand how you could create a global vector in a Seq manner > > on a > > > > DA??? > > > > > > > > > > > > > > > > Brian A. Grierson > > > > > > > > > > > > > > > > > > > > Department of Applied Physics > > > > > > > > Columbia University > > > > > > > > Mobile: 646.259.1038 > > > > > > > > Work: 212-854-4839 > > > > > > > > bag2107 at columbia.edu > > > > > > > > bgrierson21 at gmail.com > > > > > > > > > > http://www.columbia.edu/~bag2107/http://www.apam.columbia.edu/ctx/ctx.html > > > > > > > > > > > > On Jan 27, 2007, at 6:41 PM, Aron Ahmadia wrote: > > > > > > > > Hi Brian, > > > > > > > > I took a quick peek at the source for DAGetGlobalVector in the current > > > > release: > > > > > > > > " > > > > for (i=0; i > > > if (da->globalin[i]) { > > > > *g = da->globalin[i]; > > > > da->globalin[i] = PETSC_NULL; > > > > goto alldone; > > > > } > > > > } > > > > ierr = DACreateGlobalVector(da,g);CHKERRQ(ierr); > > > > > > > > alldone: > > > > for (i=0; i > > > if (!da->globalout[i]) { > > > > da->globalout[i] = *g; > > > > break; > > > > } > > > > " > > > > > > > > The two appear to be very similar, with this exception. Get/Restore > > > > work with one copy of the DA's global vector, but will make a copy if > > > > one has already been checked out or none exist. > > > > > > > > DACreateGlobalVector will always make a copy of the global vector, > > > > regardless of the circumstances. I think you're better off using Get > > > > and Restore (less data copying) unless you'd like to work with > > > > multiple copies for some reason. > > > > > > > > ~A > > > > > > > > On 1/27/07, Brian Grierson < bag2107 at columbia.edu> wrote: > > > > > > > > Could you please tell me the difference between > > > > > > > > DACreateGlobalVector / VecDestroy > > > > and > > > > DAGetGlobalVector / DARestoreGlobalVector > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Brian A. Grierson > > > > > > > > > > > > > > > > > > > > Department of Applied Physics > > > > > > > > Columbia University > > > > > > > > Mobile: 646.259.1038 > > > > > > > > Work: 212-854-4839 > > > > > > > > bag2107 at columbia.edu > > > > > > > > bgrierson21 at gmail.com > > > > > > > > > > http://www.columbia.edu/~bag2107/http://www.apam.columbia.edu/ctx/ctx.html > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > One trouble is that despite this system, anyone who reads journals widely > > and critically is forced to realize that there are scarcely any bars to > > eventual > > publication. There seems to be no study too fragmented, no hypothesis too > > trivial, no literature citation too biased or too egotistical, no design too > > warped, no methodology too bungled, no presentation of results too > > inaccurate, too obscure, and too contradictory, no analysis too > > self-serving, > > no argument too circular, no conclusions too trifling or too unjustified, > > and > > no grammar and syntax too offensive for a paper to end up in print. -- > > Drummond Rennie > From jiaxun_hou at yahoo.com.cn Sat Jan 27 21:07:42 2007 From: jiaxun_hou at yahoo.com.cn (jiaxun hou) Date: Sun, 28 Jan 2007 11:07:42 +0800 (CST) Subject: about combining the GMRES and smooth methods Message-ID: <591517.47355.qm@web15809.mail.cnb.yahoo.com> Hi, I want to add a smooth method in the GMRES routine. I do it in the function KSPSolve_GMRES of /src/ksp/ksp/impls/gmres/gmres.c. The code likes that: PetscErrorCode KSPSolve_GMRES(KSP ksp) { ..... ierr = GMREScycle(&its,ksp);CHKERRQ(ierr); ierr = MatRelax(ksp->pc->mat, ksp->vec_rhs,1,SOR_FORWARD_SWEEP,0,1,1,ksp->vec_sol);CHKERRQ(ierr); ..... } But compile fail. Is it the right way? Thanks. ___________________________________________________________ ??????-3.5G???20M?? http://cn.mail.yahoo.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Jan 27 21:28:35 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 27 Jan 2007 21:28:35 -0600 (CST) Subject: about combining the GMRES and smooth methods In-Reply-To: <591517.47355.qm@web15809.mail.cnb.yahoo.com> References: <591517.47355.qm@web15809.mail.cnb.yahoo.com> Message-ID: It seems extremely likely that you do not want to do this. If you want to use SOR or SSOR as a preconditioner for GMRES simply use -pc_type sor; see the manual page for PCSOR for other options. Barry On Sun, 28 Jan 2007, jiaxun hou wrote: > Hi, I want to add a smooth method in the GMRES routine. I do it in the function KSPSolve_GMRES of /src/ksp/ksp/impls/gmres/gmres.c. The code likes that: PetscErrorCode KSPSolve_GMRES(KSP ksp) { ..... ierr = GMREScycle(&its,ksp);CHKERRQ(ierr); ierr = MatRelax(ksp->pc->mat, ksp->vec_rhs,1,SOR_FORWARD_SWEEP,0,1,1,ksp->vec_sol);CHKERRQ(ierr); ..... } But compile fail. Is it the right way? Thanks. ___________________________________________________________ ????????????-3.5G??????20M???? http://cn.mail.yahoo.com/ From jiaxun_hou at yahoo.com.cn Sat Jan 27 22:29:27 2007 From: jiaxun_hou at yahoo.com.cn (jiaxun hou) Date: Sun, 28 Jan 2007 12:29:27 +0800 (CST) Subject: =?gb2312?B?u9i4tKO6IGFib3V0IGNvbWJpbmluZyB0aGUgR01SRVMgYW5kIHNtb290aCBt?= =?gb2312?B?ZXRob2Rz?= Message-ID: <175564.13934.qm@web15810.mail.cnb.yahoo.com> But I already had another preconditioner. I want to study the behave of the smooth method in the GMRES with my preconditioner. How can I do? Thanks! ------------------------------------------------------------- ----- ???? ---- ???? Barry Smith ???? petsc-users at mcs.anl.gov ???? 2007/1/28(??), ??11:28:35 ??? Re: about combining the GMRES and smooth methods It seems extremely likely that you do not want to do this. If you want to use SOR or SSOR as a preconditioner for GMRES simply use -pc_type sor; see the manual page for PCSOR for other options. Barry On Sun, 28 Jan 2007, jiaxun hou wrote: > Hi, I want to add a smooth method in the GMRES routine. I do it in the function KSPSolve_GMRES of /src/ksp/ksp/impls/gmres/gmres.c. The code likes that: PetscErrorCode KSPSolve_GMRES(KSP ksp) { ..... ierr = GMREScycle(&its,ksp);CHKERRQ(ierr); ierr = MatRelax(ksp->pc->mat, ksp->vec_rhs,1,SOR_FORWARD_SWEEP,0,1,1,ksp->vec_sol);CHKERRQ(ierr); ..... } But compile fail. Is it the right way? Thanks. ___________________________________________________________ ??????-3.5G???20M?? http://cn.mail.yahoo.com/ ___________________________________________________________ ????????-3.5G???20M??? http://cn.mail.yahoo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Jan 27 22:43:28 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 27 Jan 2007 22:43:28 -0600 (CST) Subject: =?gb2312?B?u9i4tKO6IGFib3V0IGNvbWJpbmluZyB0aGUgR01SRVMgYW5kIHNtb290aCBt?= =?gb2312?B?ZXRob2Rz?= In-Reply-To: <175564.13934.qm@web15810.mail.cnb.yahoo.com> References: <175564.13934.qm@web15810.mail.cnb.yahoo.com> Message-ID: If you want help with why a compile fails you need to at least send the entire output when the compile fails. Barry On Sun, 28 Jan 2007, jiaxun hou wrote: > But I already had another preconditioner. I want to study the behave of the smooth method in the GMRES with my preconditioner. How can I do? Thanks! ------------------------------------------------------------- ----- ???????? ---- ???????? Barry Smith ???????? petsc-users at mcs.anl.gov ???????? 2007/1/28(????), ????11:28:35 ?????? Re: about combining the GMRES and smooth methods It seems extremely likely that you do not want to do this. If you want to use SOR or SSOR as a preconditioner for GMRES simply use -pc_type sor; see the manual page for PCSOR for other options. Barry On Sun, 28 Jan 2007, jiaxun hou wrote: > Hi, I want to add a smooth method in the GMRES routine. I do it in the function KSPSolve_GMRES of /src/ksp/ksp/impls/gmres/gmres.c. The code likes that: PetscErrorCode KSPSolve_GMRES(KSP ksp) { ..... ierr = GMREScycle(&its,ksp);CHKERRQ(ierr); ierr = MatRelax(ksp->pc->mat, ksp->vec_rhs,1,SOR_FORWARD_SWEEP,0,1,1,ksp->vec_sol);CHKERRQ(ierr); ..... } But compile fail. Is it the right way? Thanks. ___________________________________________________________ ????????????-3.5G??????20M???? http://cn.mail.yahoo.com/ ___________________________________________________________ ????????????????-3.5G??????20M?????? http://cn.mail.yahoo.com From jiaxun_hou at yahoo.com.cn Sat Jan 27 23:03:55 2007 From: jiaxun_hou at yahoo.com.cn (jiaxun hou) Date: Sun, 28 Jan 2007 13:03:55 +0800 (CST) Subject: =?gb2312?B?u9i4tKO6ILvYuLSjuiBhYm91dCBjb21iaW5pbmcgdGhlIEdNUkVTIGFuZCBz?= =?gb2312?B?bW9vdGggbWV0aG9kcw==?= Message-ID: <20070128050355.21326.qmail@web15806.mail.cnb.yahoo.com> I am sorry. This is the error message: g++ -c -Wall -Wshadow -Wwrite-strings -fomit-frame-pointer -Wno-strict-aliasing -g3 -PIC -I/home/software/petsc-2.3.1-p10 -I/home/software/petsc-2.3.1-p10/bmake/linux-gnu-cxx-complex-debug -I/home/software/petsc-2.3.1-p10/include -I/home/software/mpich2/include -D__SDIR__='"src/ksp/ksp/impls/gmres/"' gmres.c g++: unrecognized option '-PIC' gmres.c: In function ?PetscErrorCode KSPSolve_GMRES(_p_KSP*)?: gmres.c:259: error: invalid use of undefined type ?struct _p_PC? /home/software/petsc-2.3.1-p10/include/petscpc.h:27: error: forward declaration of ?struct _p_PC? make[1]: [/home/software/petsc-2.3.1-p10/lib/linux-gnu-cxx-complex-debug/libpetscksp.a(gmres.o)] Error 1 (ignored) /usr/bin/ar cr /home/software/petsc-2.3.1-p10/lib/linux-gnu-cxx-complex-debug/libpetscksp.a gmres.o/usr/bin/ar: gmres.o: No such file or directory make[1]: [/home/software/petsc-2.3.1-p10/lib/linux-gnu-cxx-complex-debug/libpetscksp.a(gmres.o)] Error 1 (ignored) if test -n ""; then /usr/bin/ar cr gmres.lo; fi /bin/rm -f gmres.o gmres.lo It said that the struct _p_PC was undefined. But it seems impossible. I dont know where is wrong? Thanks! ----- ???? ---- ???? Barry Smith ???? petsc-users at mcs.anl.gov ???? 2007/1/28(??), ??12:43:28 ??? Re: ??? about combining the GMRES and smooth methods If you want help with why a compile fails you need to at least send the entire output when the compile fails. Barry On Sun, 28 Jan 2007, jiaxun hou wrote: > But I already had another preconditioner. I want to study the behave of the smooth method in the GMRES with my preconditioner. How can I do? Thanks! ------------------------------------------------------------- ----- ???? ---- ???? Barry Smith ???? petsc-users at mcs.anl.gov ???? 2007/1/28(??), ??11:28:35 ??? Re: about combining the GMRES and smooth methods It seems extremely likely that you do not want to do this. If you want to use SOR or SSOR as a preconditioner for GMRES simply use -pc_type sor; see the manual page for PCSOR for other options. Barry On Sun, 28 Jan 2007, jiaxun hou wrote: > Hi, I want to add a smooth method in the GMRES routine. I do it in the function KSPSolve_GMRES of /src/ksp/ksp/impls/gmres/gmres.c. The code likes that: PetscErrorCode KSPSolve_GMRES(KSP ksp) { ..... ierr = GMREScycle(&its,ksp);CHKERRQ(ierr); ierr = MatRelax(ksp->pc->mat, ksp->vec_rhs,1,SOR_FORWARD_SWEEP,0,1,1,ksp->vec_sol);CHKERRQ(ierr); ..... } But compile fail. Is it the right way? Thanks. ___________________________________________________________ ??????-3.5G???20M?? http://cn.mail.yahoo.com/ ___________________________________________________________ ????????-3.5G???20M??? http://cn.mail.yahoo.com ___________________________________________________________ Mp3???-??????? http://music.yahoo.com.cn/?source=mail_mailbox_footer -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Sun Jan 28 15:52:52 2007 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sun, 28 Jan 2007 18:52:52 -0300 Subject: =?GB2312?B?UmU6ILvYuLSjuiBhYm91dCBjb21iaW5pbmcgdGg=?= =?GB2312?B?ZSBHTVJFUyBhbmQgc21vb3RoIG1ldGhvZHM=?= In-Reply-To: <175564.13934.qm@web15810.mail.cnb.yahoo.com> References: <175564.13934.qm@web15810.mail.cnb.yahoo.com> Message-ID: On 1/28/07, jiaxun hou wrote: > > But I already had another preconditioner. I want to study the behave of the > smooth method in the GMRES with my preconditioner. How can I do? > Thanks! So perhaps it is better to use a composite preconditioner, calling PCCreate(comm,&pc) and next PCSetType(pc, PCCOMPOSITE), and finally add your preconditioner followed by the smoother... I think your original approach was wrong, if you look at GMREScycle(), you will see that this routine applies the PC inside each cycle. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From bsmith at mcs.anl.gov Sun Jan 28 18:37:18 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 28 Jan 2007 18:37:18 -0600 (CST) Subject: =?GB2312?B?UmU6ILvYuLSjuiBhYm91dCBjb21iaW5pbmcgdGg=?= =?GB2312?B?ZSBHTVJFUyBhbmQgc21vb3RoIG1ldGhvZHM=?= In-Reply-To: References: <175564.13934.qm@web15810.mail.cnb.yahoo.com> Message-ID: Lisandro, Good point. One could use (always run with -ksp_view to insure it is doing what you think). -pc_type composite -ksp_type richardson -pc_composite_pcs ksp,sor -sub_0_ksp_ksp_max_it 5 This would run 5 iterations of the default KSP and PC followed by a sweep of SSOR followed by 5 iterations of the KSP etc. etc. Using -ksp_type fgmres will use the above two steps (KSP of 5 iterations plus SOR) as a preconditioner for flexible GMRES. Adding -sub_1_pc_sor_its 2 would have it do two sweeps of SSOR instead of 1. etc. etc. There are unlimited possibilities, one can nest the solvers as much as they like, no coding needed. Have fun, Barry On Sun, 28 Jan 2007, Lisandro Dalcin wrote: > On 1/28/07, jiaxun hou wrote: > > > > But I already had another preconditioner. I want to study the behave of the > > smooth method in the GMRES with my preconditioner. How can I do? > > Thanks! > > So perhaps it is better to use a composite preconditioner, calling > PCCreate(comm,&pc) and next PCSetType(pc, PCCOMPOSITE), and finally > add your preconditioner followed by the smoother... > > I think your original approach was wrong, if you look at GMREScycle(), > you will see that this routine applies the PC inside each cycle. > > > From zonexo at gmail.com Mon Jan 29 20:33:44 2007 From: zonexo at gmail.com (Ben Tay) Date: Tue, 30 Jan 2007 10:33:44 +0800 Subject: Difference between debugging and optimized version of PETSc Message-ID: <804ab5d40701291833r22722504ia1c0fd3c27bca46d@mail.gmail.com> Hi, May I know what's the difference between the optimized and debugging version? Have the optimization flags such as -O3 been used in the compilcation of the libraries for the optimized one? Whereas for the debugging version, there is no optimization by the compiler? If my program works fine in the debugging version, does it mean that I will not get errors in the optimized version? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jan 29 20:44:52 2007 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 29 Jan 2007 20:44:52 -0600 Subject: Difference between debugging and optimized version of PETSc In-Reply-To: <804ab5d40701291833r22722504ia1c0fd3c27bca46d@mail.gmail.com> References: <804ab5d40701291833r22722504ia1c0fd3c27bca46d@mail.gmail.com> Message-ID: On 1/29/07, Ben Tay wrote: > > Hi, > > May I know what's the difference between the optimized and debugging > version? > > Have the optimization flags such as -O3 been used in the compilcation of > the libraries for the optimized one? Whereas for the debugging version, > there is no optimization by the compiler? > It is just the difference in compiler flags. You would have to look at the compiler documentation. If my program works fine in the debugging version, does it mean that I will > not get errors in the optimized version? > Most likely. Matt Thanks > -- One trouble is that despite this system, anyone who reads journals widely and critically is forced to realize that there are scarcely any bars to eventual publication. There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print. -- Drummond Rennie -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jan 29 21:12:54 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 29 Jan 2007 21:12:54 -0600 (CST) Subject: Difference between debugging and optimized version of PETSc In-Reply-To: References: <804ab5d40701291833r22722504ia1c0fd3c27bca46d@mail.gmail.com> Message-ID: PETSc also does extensive additional "assert type" testing of input arguments to functions etc in the debug version. Barry On Mon, 29 Jan 2007, Matthew Knepley wrote: > On 1/29/07, Ben Tay wrote: > > > > Hi, > > > > May I know what's the difference between the optimized and debugging > > version? > > > > Have the optimization flags such as -O3 been used in the compilcation of > > the libraries for the optimized one? Whereas for the debugging version, > > there is no optimization by the compiler? > > > > It is just the difference in compiler flags. You would have to look at the > compiler documentation. > > If my program works fine in the debugging version, does it mean that I will > > not get errors in the optimized version? > > > > Most likely. > > Matt > > Thanks > > > From yaron at oak-research.com Mon Jan 29 21:34:52 2007 From: yaron at oak-research.com (yaron at oak-research.com) Date: Mon, 29 Jan 2007 19:34:52 -0800 Subject: Non-uniform 2D mesh questions Message-ID: <20070130033452.24961.qmail@s402.sureserver.com> Hi all I have a laplace-type problem that's physically built from repeating instances of the same block. I'm creaing matrices for the individual blocks, and I'd like to reuse the individual block matrices in order to compose the complete problem. (i.e if there 10K instances of 20 blocks, I'd like to build 20 matrices, then use them to compose the large complete matrix) Is a 2D DA the right object to do that? And if so, where can I find a small example of building the DA object in parallel, then using the different (for every instance) mappings of local nodes to global nodes in order to build the complete matrix? Thanks Yaron -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jan 29 21:40:04 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 29 Jan 2007 21:40:04 -0600 (CST) Subject: Non-uniform 2D mesh questions In-Reply-To: <20070130033452.24961.qmail@s402.sureserver.com> References: <20070130033452.24961.qmail@s402.sureserver.com> Message-ID: Yaron, Is each one of these "blocks" a small rectangular part of the domain (like a 4 by 5 set of nodes)? I don't understand what you want to do. Barry On Mon, 29 Jan 2007, yaron at oak-research.com wrote: > Hi all > I have a laplace-type problem that's physically built from repeating > instances of the same block. > I'm creaing matrices for the individual blocks, and I'd like to reuse > the individual block matrices in order to compose the complete problem. > (i.e if there 10K instances of 20 blocks, I'd like to build 20 matrices, > then use them to compose the large complete matrix) > Is a 2D DA the right object to do that? And if so, where can I find a > small example of building the DA object in parallel, then using the > different (for every instance) mappings of local nodes to global nodes in > order to build the complete matrix? > > > Thanks > Yaron > From zonexo at gmail.com Mon Jan 29 21:59:26 2007 From: zonexo at gmail.com (Ben Tay) Date: Tue, 30 Jan 2007 11:59:26 +0800 Subject: Difference between debugging and optimized version of PETSc In-Reply-To: References: <804ab5d40701291833r22722504ia1c0fd3c27bca46d@mail.gmail.com> Message-ID: <804ab5d40701291959t612acab8r2fb37640b746dfe6@mail.gmail.com> Sorry I don't really understand. So is there any difference in optimization between the 2 libraries? Or is it just extensive additional "assert type"? Thank you. On 1/30/07, Barry Smith wrote: > > > PETSc also does extensive additional "assert type" testing of input > arguments > to functions etc in the debug version. > > Barry > > > On Mon, 29 Jan 2007, Matthew Knepley wrote: > > > On 1/29/07, Ben Tay wrote: > > > > > > Hi, > > > > > > May I know what's the difference between the optimized and debugging > > > version? > > > > > > Have the optimization flags such as -O3 been used in the compilcation > of > > > the libraries for the optimized one? Whereas for the debugging > version, > > > there is no optimization by the compiler? > > > > > > > It is just the difference in compiler flags. You would have to look at > the > > compiler documentation. > > > > If my program works fine in the debugging version, does it mean that I > will > > > not get errors in the optimized version? > > > > > > > Most likely. > > > > Matt > > > > Thanks > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jan 29 22:02:41 2007 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 29 Jan 2007 22:02:41 -0600 Subject: Difference between debugging and optimized version of PETSc In-Reply-To: <804ab5d40701291959t612acab8r2fb37640b746dfe6@mail.gmail.com> References: <804ab5d40701291833r22722504ia1c0fd3c27bca46d@mail.gmail.com> <804ab5d40701291959t612acab8r2fb37640b746dfe6@mail.gmail.com> Message-ID: On 1/29/07, Ben Tay wrote: > > Sorry I don't really understand. > > So is there any difference in optimization between the 2 libraries? Or is > it just extensive additional "assert type"? > As I pointed out the the previous mail, you should read the compiler documentation. Matt Thank you. > > > On 1/30/07, Barry Smith wrote: > > > > > > PETSc also does extensive additional "assert type" testing of input > > arguments > > to functions etc in the debug version. > > > > Barry > > > > > > On Mon, 29 Jan 2007, Matthew Knepley wrote: > > > > > On 1/29/07, Ben Tay wrote: > > > > > > > > Hi, > > > > > > > > May I know what's the difference between the optimized and debugging > > > > > > version? > > > > > > > > Have the optimization flags such as -O3 been used in the > > compilcation of > > > > the libraries for the optimized one? Whereas for the debugging > > version, > > > > there is no optimization by the compiler? > > > > > > > > > > It is just the difference in compiler flags. You would have to look at > > the > > > compiler documentation. > > > > > > If my program works fine in the debugging version, does it mean that I > > will > > > > not get errors in the optimized version? > > > > > > > > > > Most likely. > > > > > > Matt > > > > > > Thanks > > > > > > > > > > > > -- One trouble is that despite this system, anyone who reads journals widely and critically is forced to realize that there are scarcely any bars to eventual publication. There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print. -- Drummond Rennie -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jan 29 22:24:26 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 29 Jan 2007 22:24:26 -0600 (CST) Subject: Difference between debugging and optimized version of PETSc In-Reply-To: References: <804ab5d40701291833r22722504ia1c0fd3c27bca46d@mail.gmail.com> <804ab5d40701291959t612acab8r2fb37640b746dfe6@mail.gmail.com> Message-ID: debugging version is compiled with -g so you can see the symbols in the debugger; optimized is compiled with -Osomething to have the compiler work harder to generate efficient code. Barry Optimized version can easily be 3 times faster, or more or less. On Mon, 29 Jan 2007, Matthew Knepley wrote: > On 1/29/07, Ben Tay wrote: > > > > Sorry I don't really understand. > > > > So is there any difference in optimization between the 2 libraries? Or is > > it just extensive additional "assert type"? > > > > As I pointed out the the previous mail, you should read the compiler > documentation. > > Matt > > Thank you. > > > > > > On 1/30/07, Barry Smith wrote: > > > > > > > > > PETSc also does extensive additional "assert type" testing of input > > > arguments > > > to functions etc in the debug version. > > > > > > Barry > > > > > > > > > On Mon, 29 Jan 2007, Matthew Knepley wrote: > > > > > > > On 1/29/07, Ben Tay wrote: > > > > > > > > > > Hi, > > > > > > > > > > May I know what's the difference between the optimized and debugging > > > > > > > > version? > > > > > > > > > > Have the optimization flags such as -O3 been used in the > > > compilcation of > > > > > the libraries for the optimized one? Whereas for the debugging > > > version, > > > > > there is no optimization by the compiler? > > > > > > > > > > > > > It is just the difference in compiler flags. You would have to look at > > > the > > > > compiler documentation. > > > > > > > > If my program works fine in the debugging version, does it mean that I > > > will > > > > > not get errors in the optimized version? > > > > > > > > > > > > > Most likely. > > > > > > > > Matt > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > From yaron at oak-research.com Mon Jan 29 22:35:35 2007 From: yaron at oak-research.com (yaron at oak-research.com) Date: Mon, 29 Jan 2007 20:35:35 -0800 Subject: Non-uniform 2D mesh questions Message-ID: <20070130043535.6967.qmail@s402.sureserver.com> Barry- Yes, each block is a rectangular portion of the domain. Not so small though (more like 100 x 100 nodes) Yaron -------Original Message------- From: Barry Smith Subject: Re: Non-uniform 2D mesh questions Sent: 29 Jan '07 19:40 Yaron, Is each one of these "blocks" a small rectangular part of the domain (like a 4 by 5 set of nodes)? I don't understand what you want to do. Barry On Mon, 29 Jan 2007, [LINK: http://webmail.oak-research.com/compose.php?to=yaron at oak-research.com] yaron at oak-research.com wrote: > Hi all > I have a laplace-type problem that's physically built from repeating > instances of the same block. > I'm creaing matrices for the individual blocks, and I'd like to reuse > the individual block matrices in order to compose the complete problem. > (i.e if there 10K instances of 20 blocks, I'd like to build 20 matrices, > then use them to compose the large complete matrix) > Is a 2D DA the right object to do that? And if so, where can I find a > small example of building the DA object in parallel, then using the > different (for every instance) mappings of local nodes to global nodes in > order to build the complete matrix? > > > Thanks > Yaron > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yaron at oak-research.com Mon Jan 29 22:35:35 2007 From: yaron at oak-research.com (yaron at oak-research.com) Date: Mon, 29 Jan 2007 20:35:35 -0800 Subject: Non-uniform 2D mesh questions Message-ID: <20070130043535.6967.qmail@s402.sureserver.com> Barry- Yes, each block is a rectangular portion of the domain. Not so small though (more like 100 x 100 nodes) Yaron -------Original Message------- From: Barry Smith Subject: Re: Non-uniform 2D mesh questions Sent: 29 Jan '07 19:40 Yaron, Is each one of these "blocks" a small rectangular part of the domain (like a 4 by 5 set of nodes)? I don't understand what you want to do. Barry On Mon, 29 Jan 2007, [LINK: http://webmail.oak-research.com/compose.php?to=yaron at oak-research.com] yaron at oak-research.com wrote: > Hi all > I have a laplace-type problem that's physically built from repeating > instances of the same block. > I'm creaing matrices for the individual blocks, and I'd like to reuse > the individual block matrices in order to compose the complete problem. > (i.e if there 10K instances of 20 blocks, I'd like to build 20 matrices, > then use them to compose the large complete matrix) > Is a 2D DA the right object to do that? And if so, where can I find a > small example of building the DA object in parallel, then using the > different (for every instance) mappings of local nodes to global nodes in > order to build the complete matrix? > > > Thanks > Yaron > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jan 30 12:58:51 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 30 Jan 2007 12:58:51 -0600 (CST) Subject: Non-uniform 2D mesh questions In-Reply-To: <20070130043535.6967.qmail@s402.sureserver.com> References: <20070130043535.6967.qmail@s402.sureserver.com> Message-ID: Yaron, Do you want to end up generating a single large sparse matrix? Like a MPIAIJ matrix? Or do you want to somehow not store the entire huge matrix but still be able to solve with the composed matrix? Or both? Barry On Mon, 29 Jan 2007, yaron at oak-research.com wrote: > Barry- > Yes, each block is a rectangular portion of the domain. Not so small > though (more like 100 x 100 nodes) > > Yaron > > > -------Original Message------- > From: Barry Smith > Subject: Re: Non-uniform 2D mesh questions > Sent: 29 Jan '07 19:40 > > > Yaron, > > Is each one of these "blocks" a small rectangular part of the > domain (like a 4 by 5 set of nodes)? I don't understand what you > want to do. > > Barry > > > On Mon, 29 Jan 2007, [LINK: > http://webmail.oak-research.com/compose.php?to=yaron at oak-research.com] > yaron at oak-research.com wrote: > > > Hi all > > I have a laplace-type problem that's physically built from repeating > > instances of the same block. > > I'm creaing matrices for the individual blocks, and I'd like to reuse > > the individual block matrices in order to compose the complete > problem. > > (i.e if there 10K instances of 20 blocks, I'd like to build 20 > matrices, > > then use them to compose the large complete matrix) > > Is a 2D DA the right object to do that? And if so, where can I find a > > small example of building the DA object in parallel, then using the > > different (for every instance) mappings of local nodes to global nodes > in > > order to build the complete matrix? > > > > > > Thanks > > Yaron > > > From yaron at oak-research.com Tue Jan 30 17:07:18 2007 From: yaron at oak-research.com (yaron at oak-research.com) Date: Tue, 30 Jan 2007 15:07:18 -0800 Subject: Non-uniform 2D mesh questions Message-ID: <20070130230718.29179.qmail@s402.sureserver.com> Barry- So far I only thought of having a single large sparse matrix. Yaron -------Original Message------- From: Barry Smith Subject: Re: Non-uniform 2D mesh questions Sent: 30 Jan '07 10:58 Yaron, Do you want to end up generating a single large sparse matrix? Like a MPIAIJ matrix? Or do you want to somehow not store the entire huge matrix but still be able to solve with the composed matrix? Or both? Barry On Mon, 29 Jan 2007, [LINK: http://webmail.oak-research.com/compose.php?to=yaron at oak-research.com] yaron at oak-research.com wrote: > Barry- > Yes, each block is a rectangular portion of the domain. Not so small > though (more like 100 x 100 nodes) > > Yaron > > > -------Original Message------- > From: Barry Smith > Subject: Re: Non-uniform 2D mesh questions > Sent: 29 Jan '07 19:40 > > > Yaron, > > Is each one of these "blocks" a small rectangular part of the > domain (like a 4 by 5 set of nodes)? I don't understand what you > want to do. > > Barry > > > On Mon, 29 Jan 2007, [LINK: > [LINK: http://webmail.oak-research.com/compose.php?to=yaron at oak-research.com] http://webmail.oak-research.com/compose.php?to=yaron at oak-research.com] > [LINK: http://webmail.oak-research.com/compose.php?to=yaron at oak-research.com] yaron at oak-research.com wrote: > > > Hi all > > I have a laplace-type problem that's physically built from repeating > > instances of the same block. > > I'm creaing matrices for the individual blocks, and I'd like to reuse > > the individual block matrices in order to compose the complete > problem. > > (i.e if there 10K instances of 20 blocks, I'd like to build 20 > matrices, > > then use them to compose the large complete matrix) > > Is a 2D DA the right object to do that? And if so, where can I find a > > small example of building the DA object in parallel, then using the > > different (for every instance) mappings of local nodes to global nodes > in > > order to build the complete matrix? > > > > > > Thanks > > Yaron > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yaron at oak-research.com Tue Jan 30 17:07:18 2007 From: yaron at oak-research.com (yaron at oak-research.com) Date: Tue, 30 Jan 2007 15:07:18 -0800 Subject: Non-uniform 2D mesh questions Message-ID: <20070130230718.29179.qmail@s402.sureserver.com> Barry- So far I only thought of having a single large sparse matrix. Yaron -------Original Message------- From: Barry Smith Subject: Re: Non-uniform 2D mesh questions Sent: 30 Jan '07 10:58 Yaron, Do you want to end up generating a single large sparse matrix? Like a MPIAIJ matrix? Or do you want to somehow not store the entire huge matrix but still be able to solve with the composed matrix? Or both? Barry On Mon, 29 Jan 2007, [LINK: http://webmail.oak-research.com/compose.php?to=yaron at oak-research.com] yaron at oak-research.com wrote: > Barry- > Yes, each block is a rectangular portion of the domain. Not so small > though (more like 100 x 100 nodes) > > Yaron > > > -------Original Message------- > From: Barry Smith > Subject: Re: Non-uniform 2D mesh questions > Sent: 29 Jan '07 19:40 > > > Yaron, > > Is each one of these "blocks" a small rectangular part of the > domain (like a 4 by 5 set of nodes)? I don't understand what you > want to do. > > Barry > > > On Mon, 29 Jan 2007, [LINK: > [LINK: http://webmail.oak-research.com/compose.php?to=yaron at oak-research.com] http://webmail.oak-research.com/compose.php?to=yaron at oak-research.com] > [LINK: http://webmail.oak-research.com/compose.php?to=yaron at oak-research.com] yaron at oak-research.com wrote: > > > Hi all > > I have a laplace-type problem that's physically built from repeating > > instances of the same block. > > I'm creaing matrices for the individual blocks, and I'd like to reuse > > the individual block matrices in order to compose the complete > problem. > > (i.e if there 10K instances of 20 blocks, I'd like to build 20 > matrices, > > then use them to compose the large complete matrix) > > Is a 2D DA the right object to do that? And if so, where can I find a > > small example of building the DA object in parallel, then using the > > different (for every instance) mappings of local nodes to global nodes > in > > order to build the complete matrix? > > > > > > Thanks > > Yaron > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From DOMI0002 at ntu.edu.sg Wed Jan 31 00:43:45 2007 From: DOMI0002 at ntu.edu.sg (#DOMINIC DENVER JOHN CHANDAR#) Date: Wed, 31 Jan 2007 14:43:45 +0800 Subject: Reading a Sparse matrix in CRS format from a function to be used in parallel linear solvers Message-ID: Hi, I have a function which creates a sparse matrix following a certain eqn and returns a pointer to the row, column index and the value of a sparse matrix (CRS- form). i j val 1 1 1.0 2 2 1.0 2 3 -1.5.. so on... The matrix may not have a definite pattern like the one we see for tridiagonal or pentadiagonal matrices or the test cases (linear solvers) cited in the petsc documentation. Now, i would like to use the linear solvers in petsc to solve the system of equations. I do not want to write that sparse matrix into a file and read it into petsc since it may be huge.I would like to call that function from petsc and would return the pointer to i j and val and insert values of this sparse matrix using MatSetValues in parallel. How can I achieve this ? Cheers, Dominic -------------- next part -------------- An HTML attachment was scrubbed... URL: From berend at chalmers.se Wed Jan 31 01:30:35 2007 From: berend at chalmers.se (Berend van Wachem) Date: Wed, 31 Jan 2007 08:30:35 +0100 Subject: Non-uniform 2D mesh questions In-Reply-To: <20070130230718.29179.qmail@s402.sureserver.com> References: <20070130230718.29179.qmail@s402.sureserver.com> Message-ID: <200701310830.36055.berend@chalmers.se> Hi, I guess my question might be superfluous - but I've read a number of contradictory things. You would like multiple DA vectors to share one matrix? If so, this is exactly what I am doing and if you want, I can send you parts of code how to do so. Berend. > Barry- > So far I only thought of having a single large sparse matrix. > > Yaron > > > -------Original Message------- > From: Barry Smith > Subject: Re: Non-uniform 2D mesh questions > Sent: 30 Jan '07 10:58 > > > Yaron, > > Do you want to end up generating a single large sparse matrix? > Like a MPIAIJ > matrix? Or do you want to somehow not store the entire huge matrix > but still > be able to solve with the composed matrix? Or both? > > Barry > > > On Mon, 29 Jan 2007, [LINK: > http://webmail.oak-research.com/compose.php?to=yaron at oak-research.co >m] > > yaron at oak-research.com wrote: > > Barry- > > Yes, each block is a rectangular portion of the domain. Not so > > small though (more like 100 x 100 nodes) > > > > Yaron > > > > > > -------Original Message------- > > From: Barry Smith > > Subject: Re: Non-uniform 2D mesh questions > > Sent: 29 Jan '07 19:40 > > > > > > Yaron, > > > > Is each one of these "blocks" a small rectangular part of the > > domain (like a 4 by 5 set of nodes)? I don't understand what you > > want to do. > > > > Barry > > > > > > On Mon, 29 Jan 2007, [LINK: > > [LINK: > > http://webmail.oak-research.com/compose.php?to=yaron at oak-research.co >m] > http://webmail.oak-research.com/compose.php?to=yaron at oak-research.co >m] > > > [LINK: > > http://webmail.oak-research.com/compose.php?to=yaron at oak-research.co >m] > > yaron at oak-research.com wrote: > > > Hi all > > > I have a laplace-type problem that's physically built from > > > repeating instances of the same block. > > > I'm creaing matrices for the individual blocks, and I'd like > > > to > > reuse > > > > the individual block matrices in order to compose the complete > > > > problem. > > > > > (i.e if there 10K instances of 20 blocks, I'd like to build 20 > > > > matrices, > > > > > then use them to compose the large complete matrix) > > > Is a 2D DA the right object to do that? And if so, where can I > > > find > > a > > > > small example of building the DA object in parallel, then > > > using the different (for every instance) mappings of local > > > nodes to global > > nodes > > > in > > > > > order to build the complete matrix? > > > > > > > > > Thanks > > > Yaron From berend at chalmers.se Wed Jan 31 01:32:23 2007 From: berend at chalmers.se (Berend van Wachem) Date: Wed, 31 Jan 2007 08:32:23 +0100 Subject: Using PETSc in structured c-grid for CFD and multigrid In-Reply-To: <804ab5d40701261754p240a652brc7b112b04b0963b0@mail.gmail.com> References: <804ab5d40701251902x31cd3d29ye97ce5c0b2924e4d@mail.gmail.com> <64437.85.224.167.125.1169797999.squirrel@webmail.chalmers.se> <804ab5d40701261754p240a652brc7b112b04b0963b0@mail.gmail.com> Message-ID: <200701310832.24823.berend@chalmers.se> Hi Ben, The challenge in your problem is how you "glue" the C grid in the back; there you will need to do some additional scattering. I would set-up the IS for this, and then use that to scatter the values into "ghostcells" which will be present on the block(s). Berend. > Thank you Berend! I'll go through DA again. I'm also looking at > HYPRE. Its way of creating grids and linking them seems intuitive. > Btw, is there a mailing list for HYPRE similar to PETSc to discuss > problems? I find that their explanation are quite brief. > > I tried to install HYPRE 2.0 on windows using cygwin but it failed. > I then install it as an external software thru PETSc. I think it's > installing HYPRE 1.0 or something. But similarly, there's illegal > operation. > > Installing HYPRE 2.0 on my school's linux worked, though there's > seems to be some minor error. So what's the best way to employ > multigird? Is it to install as an external software thru PETSc or > just use HYPRE on its own? > > Btw, it will be great if you can send me parts of your code > regarding DA. > > Thank you very much! > > On 1/26/07, Berend van Wachem wrote: > > Hi, > > > > I am not an expert - but have used PETSc for both structured and > > unstructured grids. > > > > When you use an unstructured code for a structured grid, there is > > additional overhead (addressing, connectivity) which is redundant; > > this information is not required for solving on a structured grid. > > I would say this is maximum a 10% efficiency loss for bigger > > problems - it does not affect solving the matrix, only in > > gathering your coefficients. I would not rewrite my CFD code for > > this. > > > > If you only deal with structured grids, using the PETSc DA > > framework should work for you - you are not saving all > > connectivity. The DA framework is not difficult at all, according > > to my opinion. Look at a few examples that come with PETSc. I use > > a block structured solver - using multiple DA's within one > > problem. Let me know if you are interested in this, and I can send > > you parts of code. > > > > Multigrid is certainly possible (I would reccomend through HYPRE, > > discussed on the mailinglist, although I still have problems with > > it), but the question is how efficient it will be for your CFD > > problem. For an efficient multigrid in CFD, it is important to > > consider the coefficient structure arising from the momentum > > equations - the grouping of cells should occur following the > > advection term. Only then will you achieve linear scaling with the > > problem size. For instance, consider a rotating flow in a square > > box. Most multigrid algorithms will group cells in "squares" which > > will not lead to a significant improvement, as the flow > > (advection, pressure grad) does not move in these squares. In > > fact, to have an efficient multigrid algorithm, the cels should be > > grouped along the circular flow. As this cannot be seen directly > > from the pressure coefficients, I doubt any "automatic" multigrid > > algorithm (in Hypre or Petsc) would be able to capture this, but > > don't quote me on it - I am not 100% sure. So concluding, if you > > want to do efficient multigridding for CFD, you will need to point > > out which cells are grouped into which structure, based upon the > > upwind advection coefficients. > > > > Good luck, > > > > Berend. > > > > > Hi, > > > > > > I was discussing with another user in another forum > > > (cfd-online.com) > > > > about > > > > > using PETSc in my cfd code. I am now using KSP to solve my > > > momentum and poisson eqn by inserting values into the matrix. I > > > was told that using PETSc > > > this way is only for unstructured grids. It is very inefficient > > > and much slower if I'm using it for my structured grid because I > > > am not > > > > exploiting > > > > > the regular structure of my grid. > > > > > > Is that true? I'm solving flow around airfoil using c-grid. > > > > > > So how can I improve? Is it by using DA? I took a glance and it > > > seems quite > > > complicated. > > > > > > Also, is multigrid available in PETSc? Chapter 7 discusses about > > > it but > > > > it > > > > > seems very brief. Is there a more elaborate tutorial besides > > > that c examples? > > > > > > Hope someone can give me some ideas. > > > > > > Thank you. From knepley at gmail.com Wed Jan 31 08:17:11 2007 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 31 Jan 2007 08:17:11 -0600 Subject: Reading a Sparse matrix in CRS format from a function to be used in parallel linear solvers In-Reply-To: References: Message-ID: On 1/31/07, #DOMINIC DENVER JOHN CHANDAR# wrote: > > Hi, > > I have a function which creates a sparse matrix following a certain > eqn and returns a pointer to the row, column index and the value of a sparse > matrix (CRS- form). > i j val > 1 1 1.0 > 2 2 1.0 > 2 3 -1.5.. > so on... > > The matrix *may not **have a definite pattern* like the one we see for > tridiagonal or pentadiagonal matrices or the test cases (linear solvers) > cited in the petsc documentation. Now, i would like to use the linear > solvers in petsc to solve the system of equations. I do not want to write > that sparse matrix into a file and read it into petsc since it may be > huge.I would like to call that function from petsc and would return the > pointer to i j and val and insert values of this sparse matrix using > MatSetValues in parallel. How can I achieve this ? > We do not have CRS format as it is not a very space efficient format. Most of our formats are row-oriented. I suggest looking at the example for MatSetValues() and trying to rework the discretization function to directly call this. Most discretizations naturally generate row-by-row values or even blocks like FEM. Even totally nustructured stuff from economics is row-by-row. Thanks, Matt Cheers, > Dominic > -- One trouble is that despite this system, anyone who reads journals widely and critically is forced to realize that there are scarcely any bars to eventual publication. There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print. -- Drummond Rennie -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Jan 31 09:57:07 2007 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 31 Jan 2007 09:57:07 -0600 (CST) Subject: Reading a Sparse matrix in CRS format from a function to be used in parallel linear solvers In-Reply-To: References: Message-ID: Dominic, You may use MatCreateMPIIJWithArrays() to achieve this. Hong On Wed, 31 Jan 2007, Matthew Knepley wrote: > On 1/31/07, #DOMINIC DENVER JOHN CHANDAR# wrote: > > > > Hi, > > > > I have a function which creates a sparse matrix following a certain > > eqn and returns a pointer to the row, column index and the value of a sparse > > matrix (CRS- form). > > i j val > > 1 1 1.0 > > 2 2 1.0 > > 2 3 -1.5.. > > so on... > > > > The matrix *may not **have a definite pattern* like the one we see for > > tridiagonal or pentadiagonal matrices or the test cases (linear solvers) > > cited in the petsc documentation. Now, i would like to use the linear > > solvers in petsc to solve the system of equations. I do not want to write > > that sparse matrix into a file and read it into petsc since it may be > > huge.I would like to call that function from petsc and would return the > > pointer to i j and val and insert values of this sparse matrix using > > MatSetValues in parallel. How can I achieve this ? > > > > We do not have CRS format as it is not a very space efficient format. Most > of our formats are row-oriented. I suggest > looking at the example for MatSetValues() and trying to rework the > discretization function to directly call this. Most > discretizations naturally generate row-by-row values or even blocks like > FEM. Even totally nustructured stuff from > economics is row-by-row. > > Thanks, > > Matt > > Cheers, > > Dominic > > > -- > One trouble is that despite this system, anyone who reads journals widely > and critically is forced to realize that there are scarcely any bars to > eventual > publication. There seems to be no study too fragmented, no hypothesis too > trivial, no literature citation too biased or too egotistical, no design too > warped, no methodology too bungled, no presentation of results too > inaccurate, too obscure, and too contradictory, no analysis too > self-serving, > no argument too circular, no conclusions too trifling or too unjustified, > and > no grammar and syntax too offensive for a paper to end up in print. -- > Drummond Rennie > From DOMI0002 at ntu.edu.sg Wed Jan 31 10:45:34 2007 From: DOMI0002 at ntu.edu.sg (#DOMINIC DENVER JOHN CHANDAR#) Date: Thu, 1 Feb 2007 00:45:34 +0800 Subject: Reading a Sparse matrix in CRS format from a function to be used in parallel linear solvers In-Reply-To: Message-ID: Hi, Thanks Hong and Matt. So looks like I don't have to call MatSetValues() since I specify the row, column indices and the values corresponding to it when I use MatCreateMPIIJWithArrays()? Cheers, Dominic -----Original Message----- From: owner-petsc-users at mcs.anl.gov [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Hong Zhang Sent: Wednesday, January 31, 2007 11:57 PM To: petsc-users at mcs.anl.gov Subject: Re: Reading a Sparse matrix in CRS format from a function to be used in parallel linear solvers Dominic, You may use MatCreateMPIIJWithArrays() to achieve this. Hong On Wed, 31 Jan 2007, Matthew Knepley wrote: > On 1/31/07, #DOMINIC DENVER JOHN CHANDAR# wrote: > > > > Hi, > > > > I have a function which creates a sparse matrix following a > > certain eqn and returns a pointer to the row, column index and the > > value of a sparse matrix (CRS- form). > > i j val > > 1 1 1.0 > > 2 2 1.0 > > 2 3 -1.5.. > > so on... > > > > The matrix *may not **have a definite pattern* like the one we see > > for tridiagonal or pentadiagonal matrices or the test cases (linear > > solvers) cited in the petsc documentation. Now, i would like to use > > the linear solvers in petsc to solve the system of equations. I do > > not want to write that sparse matrix into a file and read it into > > petsc since it may be huge.I would like to call that function from > > petsc and would return the pointer to i j and val and insert values > > of this sparse matrix using MatSetValues in parallel. How can I achieve this ? > > > > We do not have CRS format as it is not a very space efficient format. > Most of our formats are row-oriented. I suggest looking at the example > for MatSetValues() and trying to rework the discretization function to > directly call this. Most discretizations naturally generate row-by-row > values or even blocks like FEM. Even totally nustructured stuff from > economics is row-by-row. > > Thanks, > > Matt > > Cheers, > > Dominic > > > -- > One trouble is that despite this system, anyone who reads journals > widely and critically is forced to realize that there are scarcely any > bars to eventual publication. There seems to be no study too > fragmented, no hypothesis too trivial, no literature citation too > biased or too egotistical, no design too warped, no methodology too > bungled, no presentation of results too inaccurate, too obscure, and > too contradictory, no analysis too self-serving, no argument too > circular, no conclusions too trifling or too unjustified, and no > grammar and syntax too offensive for a paper to end up in print. -- > Drummond Rennie > From hzhang at mcs.anl.gov Wed Jan 31 11:18:24 2007 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 31 Jan 2007 11:18:24 -0600 (CST) Subject: Reading a Sparse matrix in CRS format from a function to be used in parallel linear solvers In-Reply-To: References: Message-ID: On Thu, 1 Feb 2007, #DOMINIC DENVER JOHN CHANDAR# wrote: > Hi, > > Thanks Hong and Matt. So looks like I don't have to call > MatSetValues() since I specify the row, column indices and the values > corresponding to it when I use MatCreateMPIIJWithArrays()? No, petsc actually does MatSetValues() inside MatCreateMPIAIJWithArrays(). Note, it sould be MatCreateMPIAIJWithArrays() ^^^^^^ I copied wrong name from the web :-( Please read http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJWithArrays.html You must input local i,j,a arrays in standard CSR format, not exactly your CRS- form. Hong > > Cheers, > Dominic > > -----Original Message----- > From: owner-petsc-users at mcs.anl.gov > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Hong Zhang > Sent: Wednesday, January 31, 2007 11:57 PM > To: petsc-users at mcs.anl.gov > Subject: Re: Reading a Sparse matrix in CRS format from a function to be > used in parallel linear solvers > > > Dominic, > > You may use MatCreateMPIIJWithArrays() to achieve this. > > Hong > > On Wed, 31 Jan 2007, Matthew Knepley wrote: > > > On 1/31/07, #DOMINIC DENVER JOHN CHANDAR# wrote: > > > > > > Hi, > > > > > > I have a function which creates a sparse matrix following a > > > certain eqn and returns a pointer to the row, column index and the > > > value of a sparse matrix (CRS- form). > > > i j val > > > 1 1 1.0 > > > 2 2 1.0 > > > 2 3 -1.5.. > > > so on... > > > > > > The matrix *may not **have a definite pattern* like the one we see > > > for tridiagonal or pentadiagonal matrices or the test cases (linear > > > solvers) cited in the petsc documentation. Now, i would like to use > > > the linear solvers in petsc to solve the system of equations. I do > > > not want to write that sparse matrix into a file and read it into > > > petsc since it may be huge.I would like to call that function from > > > petsc and would return the pointer to i j and val and insert values > > > of this sparse matrix using MatSetValues in parallel. How can I > achieve this ? > > > > > > > We do not have CRS format as it is not a very space efficient format. > > Most of our formats are row-oriented. I suggest looking at the example > > > for MatSetValues() and trying to rework the discretization function to > > > directly call this. Most discretizations naturally generate row-by-row > > > values or even blocks like FEM. Even totally nustructured stuff from > > economics is row-by-row. > > > > Thanks, > > > > Matt > > > > Cheers, > > > Dominic > > > > > -- > > One trouble is that despite this system, anyone who reads journals > > widely and critically is forced to realize that there are scarcely any > > > bars to eventual publication. There seems to be no study too > > fragmented, no hypothesis too trivial, no literature citation too > > biased or too egotistical, no design too warped, no methodology too > > bungled, no presentation of results too inaccurate, too obscure, and > > too contradictory, no analysis too self-serving, no argument too > > circular, no conclusions too trifling or too unjustified, and no > > grammar and syntax too offensive for a paper to end up in print. -- > > Drummond Rennie > > > > From bsmith at mcs.anl.gov Wed Jan 31 11:47:31 2007 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 31 Jan 2007 11:47:31 -0600 (CST) Subject: Using PETSc in structured c-grid for CFD and multigrid In-Reply-To: <200701310832.24823.berend@chalmers.se> References: <804ab5d40701251902x31cd3d29ye97ce5c0b2924e4d@mail.gmail.com> <64437.85.224.167.125.1169797999.squirrel@webmail.chalmers.se> <804ab5d40701261754p240a652brc7b112b04b0963b0@mail.gmail.com> <200701310832.24823.berend@chalmers.se> Message-ID: The glueing might be able to be handled by using periodic for that dimension of the DA you create. But this gets tricky if you have any nodes that have "an extra degree of freedom". Barry On Wed, 31 Jan 2007, Berend van Wachem wrote: > Hi Ben, > > The challenge in your problem is how you "glue" the C grid in the back; > there you will need to do some additional scattering. I would set-up > the IS for this, and then use that to scatter the values > into "ghostcells" which will be present on the block(s). > > Berend. > > > Thank you Berend! I'll go through DA again. I'm also looking at > > HYPRE. Its way of creating grids and linking them seems intuitive. > > Btw, is there a mailing list for HYPRE similar to PETSc to discuss > > problems? I find that their explanation are quite brief. > > > > I tried to install HYPRE 2.0 on windows using cygwin but it failed. > > I then install it as an external software thru PETSc. I think it's > > installing HYPRE 1.0 or something. But similarly, there's illegal > > operation. > > > > Installing HYPRE 2.0 on my school's linux worked, though there's > > seems to be some minor error. So what's the best way to employ > > multigird? Is it to install as an external software thru PETSc or > > just use HYPRE on its own? > > > > Btw, it will be great if you can send me parts of your code > > regarding DA. > > > > Thank you very much! > > > > On 1/26/07, Berend van Wachem wrote: > > > Hi, > > > > > > I am not an expert - but have used PETSc for both structured and > > > unstructured grids. > > > > > > When you use an unstructured code for a structured grid, there is > > > additional overhead (addressing, connectivity) which is redundant; > > > this information is not required for solving on a structured grid. > > > I would say this is maximum a 10% efficiency loss for bigger > > > problems - it does not affect solving the matrix, only in > > > gathering your coefficients. I would not rewrite my CFD code for > > > this. > > > > > > If you only deal with structured grids, using the PETSc DA > > > framework should work for you - you are not saving all > > > connectivity. The DA framework is not difficult at all, according > > > to my opinion. Look at a few examples that come with PETSc. I use > > > a block structured solver - using multiple DA's within one > > > problem. Let me know if you are interested in this, and I can send > > > you parts of code. > > > > > > Multigrid is certainly possible (I would reccomend through HYPRE, > > > discussed on the mailinglist, although I still have problems with > > > it), but the question is how efficient it will be for your CFD > > > problem. For an efficient multigrid in CFD, it is important to > > > consider the coefficient structure arising from the momentum > > > equations - the grouping of cells should occur following the > > > advection term. Only then will you achieve linear scaling with the > > > problem size. For instance, consider a rotating flow in a square > > > box. Most multigrid algorithms will group cells in "squares" which > > > will not lead to a significant improvement, as the flow > > > (advection, pressure grad) does not move in these squares. In > > > fact, to have an efficient multigrid algorithm, the cels should be > > > grouped along the circular flow. As this cannot be seen directly > > > from the pressure coefficients, I doubt any "automatic" multigrid > > > algorithm (in Hypre or Petsc) would be able to capture this, but > > > don't quote me on it - I am not 100% sure. So concluding, if you > > > want to do efficient multigridding for CFD, you will need to point > > > out which cells are grouped into which structure, based upon the > > > upwind advection coefficients. > > > > > > Good luck, > > > > > > Berend. > > > > > > > Hi, > > > > > > > > I was discussing with another user in another forum > > > > (cfd-online.com) > > > > > > about > > > > > > > using PETSc in my cfd code. I am now using KSP to solve my > > > > momentum and poisson eqn by inserting values into the matrix. I > > > > was told that using PETSc > > > > this way is only for unstructured grids. It is very inefficient > > > > and much slower if I'm using it for my structured grid because I > > > > am not > > > > > > exploiting > > > > > > > the regular structure of my grid. > > > > > > > > Is that true? I'm solving flow around airfoil using c-grid. > > > > > > > > So how can I improve? Is it by using DA? I took a glance and it > > > > seems quite > > > > complicated. > > > > > > > > Also, is multigrid available in PETSc? Chapter 7 discusses about > > > > it but > > > > > > it > > > > > > > seems very brief. Is there a more elaborate tutorial besides > > > > that c examples? > > > > > > > > Hope someone can give me some ideas. > > > > > > > > Thank you. > > From zonexo at gmail.com Wed Jan 31 21:32:40 2007 From: zonexo at gmail.com (Ben Tay) Date: Thu, 1 Feb 2007 11:32:40 +0800 Subject: Using PETSc in structured c-grid for CFD and multigrid In-Reply-To: References: <804ab5d40701251902x31cd3d29ye97ce5c0b2924e4d@mail.gmail.com> <64437.85.224.167.125.1169797999.squirrel@webmail.chalmers.se> <804ab5d40701261754p240a652brc7b112b04b0963b0@mail.gmail.com> <200701310832.24823.berend@chalmers.se> Message-ID: <804ab5d40701311932o43d85cc2q9dbf7d6189a47cdd@mail.gmail.com> Hi, somone suggested that I treat that face as a dirichlet boundary condition. after 1 or a few iterations, the face value will be updated and it will be repeated until covergerence. I wonder if that is possible as well? It'll make the job much easier, although the iteration may take longer... On 2/1/07, Barry Smith wrote: > > > The glueing might be able to be handled by using periodic for that > dimension of the DA you create. But this gets tricky if you have any > nodes that have "an extra degree of freedom". > > Barry > > > On Wed, 31 Jan 2007, Berend van Wachem wrote: > > > Hi Ben, > > > > The challenge in your problem is how you "glue" the C grid in the back; > > there you will need to do some additional scattering. I would set-up > > the IS for this, and then use that to scatter the values > > into "ghostcells" which will be present on the block(s). > > > > Berend. > > > > > Thank you Berend! I'll go through DA again. I'm also looking at > > > HYPRE. Its way of creating grids and linking them seems intuitive. > > > Btw, is there a mailing list for HYPRE similar to PETSc to discuss > > > problems? I find that their explanation are quite brief. > > > > > > I tried to install HYPRE 2.0 on windows using cygwin but it failed. > > > I then install it as an external software thru PETSc. I think it's > > > installing HYPRE 1.0 or something. But similarly, there's illegal > > > operation. > > > > > > Installing HYPRE 2.0 on my school's linux worked, though there's > > > seems to be some minor error. So what's the best way to employ > > > multigird? Is it to install as an external software thru PETSc or > > > just use HYPRE on its own? > > > > > > Btw, it will be great if you can send me parts of your code > > > regarding DA. > > > > > > Thank you very much! > > > > > > On 1/26/07, Berend van Wachem wrote: > > > > Hi, > > > > > > > > I am not an expert - but have used PETSc for both structured and > > > > unstructured grids. > > > > > > > > When you use an unstructured code for a structured grid, there is > > > > additional overhead (addressing, connectivity) which is redundant; > > > > this information is not required for solving on a structured grid. > > > > I would say this is maximum a 10% efficiency loss for bigger > > > > problems - it does not affect solving the matrix, only in > > > > gathering your coefficients. I would not rewrite my CFD code for > > > > this. > > > > > > > > If you only deal with structured grids, using the PETSc DA > > > > framework should work for you - you are not saving all > > > > connectivity. The DA framework is not difficult at all, according > > > > to my opinion. Look at a few examples that come with PETSc. I use > > > > a block structured solver - using multiple DA's within one > > > > problem. Let me know if you are interested in this, and I can send > > > > you parts of code. > > > > > > > > Multigrid is certainly possible (I would reccomend through HYPRE, > > > > discussed on the mailinglist, although I still have problems with > > > > it), but the question is how efficient it will be for your CFD > > > > problem. For an efficient multigrid in CFD, it is important to > > > > consider the coefficient structure arising from the momentum > > > > equations - the grouping of cells should occur following the > > > > advection term. Only then will you achieve linear scaling with the > > > > problem size. For instance, consider a rotating flow in a square > > > > box. Most multigrid algorithms will group cells in "squares" which > > > > will not lead to a significant improvement, as the flow > > > > (advection, pressure grad) does not move in these squares. In > > > > fact, to have an efficient multigrid algorithm, the cels should be > > > > grouped along the circular flow. As this cannot be seen directly > > > > from the pressure coefficients, I doubt any "automatic" multigrid > > > > algorithm (in Hypre or Petsc) would be able to capture this, but > > > > don't quote me on it - I am not 100% sure. So concluding, if you > > > > want to do efficient multigridding for CFD, you will need to point > > > > out which cells are grouped into which structure, based upon the > > > > upwind advection coefficients. > > > > > > > > Good luck, > > > > > > > > Berend. > > > > > > > > > Hi, > > > > > > > > > > I was discussing with another user in another forum > > > > > (cfd-online.com) > > > > > > > > about > > > > > > > > > using PETSc in my cfd code. I am now using KSP to solve my > > > > > momentum and poisson eqn by inserting values into the matrix. I > > > > > was told that using PETSc > > > > > this way is only for unstructured grids. It is very inefficient > > > > > and much slower if I'm using it for my structured grid because I > > > > > am not > > > > > > > > exploiting > > > > > > > > > the regular structure of my grid. > > > > > > > > > > Is that true? I'm solving flow around airfoil using c-grid. > > > > > > > > > > So how can I improve? Is it by using DA? I took a glance and it > > > > > seems quite > > > > > complicated. > > > > > > > > > > Also, is multigrid available in PETSc? Chapter 7 discusses about > > > > > it but > > > > > > > > it > > > > > > > > > seems very brief. Is there a more elaborate tutorial besides > > > > > that c examples? > > > > > > > > > > Hope someone can give me some ideas. > > > > > > > > > > Thank you. > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: