From xiwang at dragon.rutgers.edu Mon Oct 2 22:47:23 2006 From: xiwang at dragon.rutgers.edu (Xiaoxu Wang) Date: Mon, 02 Oct 2006 23:47:23 -0400 Subject: MatCreateMPIAIJ Message-ID: <4521DD4B.2080701@dragon.rutgers.edu> Hi, I am a beginner on PETSC. This is my question about MatCreateMPIAIJ. I have a sparse matrix and only know the total number of nonzeros each row. The number of local rows and the number of local columns has been set to PETSC_DECIDE. Therefore I don't know the size of each portion and it is hard to tell d_nnz and o_nnz. I use the total number of nunzeros as d_nnz and get an allocation error. How to deal with this problem? Should I explicitly specify the number of local rows and the number of local columns and calculate d_nnz and o_nnz? Or let PETSC decide the number of local rows and the number of local columns, and then use another function to get these values? Any other better solution? Thanks a lot, Xiaoxu From bsmith at mcs.anl.gov Tue Oct 3 07:32:19 2006 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 3 Oct 2006 07:32:19 -0500 (CDT) Subject: MatCreateMPIAIJ In-Reply-To: <4521DD4B.2080701@dragon.rutgers.edu> References: <4521DD4B.2080701@dragon.rutgers.edu> Message-ID: On Mon, 2 Oct 2006, Xiaoxu Wang wrote: > Hi, > > I am a beginner on PETSC. This is my question about MatCreateMPIAIJ. I > have a sparse matrix and only know the total number of nonzeros each row. The > number of local rows and the number of local columns has been set to > PETSC_DECIDE. Therefore I don't know the size of each portion and it is hard > to tell d_nnz and o_nnz. I use the total number of nunzeros as d_nnz and get > an allocation error. It would be helpful to know the exact error message: > How to deal with this problem? Should I explicitly > specify the number of local rows and the number of local columns and calculate > d_nnz and o_nnz? Or let PETSC decide the number of local rows and the number > of local columns, and then use another function to get these values? Any > other better solution? You should specifically set the number of local rows and columns and then compute the values of d_nnz and o_nnz. How you determine d_nnz and o_nnz is problem dependent and depends on what type of discretization you are using: finite differences, elements etc. Barry > > Thanks a lot, > Xiaoxu > > From xiwang at dragon.rutgers.edu Tue Oct 3 13:03:53 2006 From: xiwang at dragon.rutgers.edu (Xiaoxu Wang) Date: Tue, 03 Oct 2006 14:03:53 -0400 Subject: MatCreateMPIAIJ Message-ID: <4522A609.8050106@dragon.rutgers.edu> The error is at line 1023 of "plog.c". --------------- An unhandled exception of type 'System.NullReferenceException' occurred in Unknown Module. Additional information: Object reference not set to an instance of an object. ---------------- The matrix I am trying to initialize is the stiffness matrix K of finite element methods. Later the dynamical system is solved as (I + Dt*K) U(t+1) = U(t) + Dt.F(t) by calling KSPSolve. Any idea how to set the number of local rows? I am running the code on PC now, but want to keep MatCreateMPIAIJ in it. Xiaoxu ----- Original Message ----- From: "Barry Smith" To: Sent: Tuesday, October 03, 2006 8:32 AM Subject: Re: MatCreateMPIAIJ > > > On Mon, 2 Oct 2006, Xiaoxu Wang wrote: > >> Hi, >> >> I am a beginner on PETSC. This is my question about >> MatCreateMPIAIJ. I >> have a sparse matrix and only know the total number of nonzeros each >> row. The >> number of local rows and the number of local columns has been set to >> PETSC_DECIDE. Therefore I don't know the size of each portion and it >> is hard >> to tell d_nnz and o_nnz. I use the total number of nunzeros as d_nnz >> and get >> an allocation error. > > > It would be helpful to know the exact error message: > >> How to deal with this problem? Should I explicitly >> specify the number of local rows and the number of local columns and >> calculate >> d_nnz and o_nnz? Or let PETSC decide the number of local rows and the >> number >> of local columns, and then use another function to get these values? >> Any >> other better solution? > > > You should specifically set the number of local rows and columns and > then > compute the values of d_nnz and o_nnz. How you determine d_nnz and > o_nnz is > problem dependent and depends on what type of discretization you are > using: > finite differences, elements etc. > > > Barry > >> >> Thanks a lot, >> Xiaoxu >> >> > > From mafunk at nmsu.edu Tue Oct 3 23:14:15 2006 From: mafunk at nmsu.edu (Matt Funk) Date: Tue, 3 Oct 2006 22:14:15 -0600 Subject: using hypre through petsc In-Reply-To: References: <215294220609190330n630747f7p9c1996c4d38f23ae@mail.gmail.com> <200609291719.22969.mafunk@nmsu.edu> Message-ID: <200610032214.17023.mafunk@nmsu.edu> Hi guys, first of all thanks for all the help. I was wondering another thing now though. The hypre options that i PETSC prints are: -pc_type Preconditioner:(one of) none jacobi pbjacobi bjacobi sor lu shell mg eisenstat ilu icc cholesky asm ksp composite redundant nn mat fieldsplit hypre tfs (PCSetType) HYPRE preconditioner options -pc_hypre_type (choose one of) pilut parasails boomeramg euclid I was wondering how come the other hypre preconditioners such as SMG or PFMG are not listed. (Or did i miss something)? thanks mat On Friday 29 September 2006 19:56, Barry Smith wrote: > Yes, you must call SNESSetFromOptions, KSPSetFromOptions or > PCSetFromOptions. It is at this point in the code that it checks the > provided options and prints the help messages (as it checks them). > > You do not need to hardwire the options in the code to have the proper > help printed. > > Barry > > On Fri, 29 Sep 2006, Matt Funk wrote: > > Hi, > > > > i don't know what i am doing wrong, but it still doesn't give me the any > > hypre output. > > > > So i need to call KSPSetFromOptions? or PCSetFromOptions? > > > > Also, at what point in the program should it print those options out? > > After i call XSetFromOptions? > > > > Do i need to set the PCType to hypre first and then > > PCHYPRESetType(m_pc,"boomeramg") and then it prints it out? > > > > mat > > > > On Friday 29 September 2006 15:35, Barry Smith wrote: > > > On Fri, 29 Sep 2006, Yaron Kretchmer wrote: > > > > If the latter (Petsc not configured to use Hypre), wouldn't Matt get > > > > an error when specifying hypre as preconditioner? > > > > > > Yes he would, but it could get lost among all the help messages > > > printed. > > > > > > Barry > > > > > > > Yaron > > > > > > > > On 9/29/06, Barry Smith wrote: > > > > > Either you are not calling KSPSetFromOptions() in your > > > > > code (or SNESSetFromOptions() OR PETSc was not configured > > > > > to use hypre. To configure PETSc to use hypre you > > > > > must use the additional config/configure.py option > > > > > --download-hypre > > > > > > > > > > Good luck, > > > > > > > > > > Barry > > > > > > > > > > On Fri, 29 Sep 2006, Matt Funk wrote: > > > > > > This might be a pretty stupid question, especially since i did > > > > > > this a > > > > > > > > > > while > > > > > > > > > > > back and it worked. > > > > > > > > > > > > However, how do i specify thsis? Do i run my executable do and > > > > > > then > > > > > > > > > > simply > > > > > > > > > > > append -pc_type hypre -pc_hypre_type boomeramg -help and it will > > > > > > print > > > > > > > > > > out > > > > > > > > > > > the options? > > > > > > > > > > > > > > > > > > Because when i do that i get: > > > > > > Options for all PETSc programs: > > > > > > -on_error_abort: cause an abort when an error is detected. > > > > > > Useful only when run in the debugger > > > > > > -on_error_attach_debugger [gdb,dbx,xxgdb,ups,noxterm] > > > > > > start the debugger in new xterm > > > > > > unless noxterm is given > > > > > > -start_in_debugger [gdb,dbx,xxgdb,ups,noxterm] > > > > > > start all processes in the debugger > > > > > > -on_error_emacs > > > > > > emacs jumps to error file > > > > > > -debugger_nodes [n1,n2,..] Nodes to start in debugger > > > > > > -debugger_pause [m] : delay (in seconds) to attach debugger > > > > > > -stop_for_debugger : prints message on how to attach debugger > > > > > > manually waits the delay for you to attach > > > > > > -display display: Location where graphics and debuggers are > > > > > > displayed -no_signal_handler: do not trap error signals > > > > > > -mpi_return_on_error: MPI returns error code, rather than abort > > > > > > on > > > > > > > > > > internal > > > > > > > > > > > error > > > > > > -fp_trap: stop on floating point exceptions > > > > > > note on IBM RS6000 this slows run greatly > > > > > > -malloc_dump : dump list of unfreed memory at > > > > > > > > > > conclusion > > > > > > > > > > > -malloc: use our error checking malloc > > > > > > -malloc no: don't use error checking malloc > > > > > > -mallocinfo: prints total memory usage > > > > > > -malloc_debug: enables extended checking for memory corruption > > > > > > -options_table: dump list of options inputted > > > > > > -options_left: dump list of unused options > > > > > > -options_left no: don't dump list of unused options > > > > > > -tmp tmpdir: alternative /tmp directory > > > > > > -shared_tmp: tmp directory is shared by all processors > > > > > > -not_shared_tmp: each processor has separate tmp directory > > > > > > -memory_info: print memory usage at end of run > > > > > > -get_total_flops: total flops over all processors > > > > > > -log[_all _summary]: logging objects and events > > > > > > -log_trace [filename]: prints trace of all PETSc calls > > > > > > -info : print informative messages about the > > > > > > > > > > calculations > > > > > > > > > > > -v: prints PETSc version number and release date > > > > > > -options_file : reads options from file > > > > > > -petsc_sleep n: sleeps n seconds before running program > > > > > > > > > > > > > > > > > > but nothing about hypre .... ? > > > > > > > > > > > > > > > > > > mat > > > > > > > > > > > > On Tuesday 19 September 2006 17:05, Yaron Kretchmer wrote: > > > > > > > Hi Matt > > > > > > > You can start experimenting by using > > > > > > > -pc_type hypre -pc_hypre_type boomeramg -help > > > > > > > which will give you all the boomeramg options. > > > > > > > > > > > > > > When I use it I get intermittent segfaults, which I couldn't > > > > > > > resolve > > > > > > > > > > with > > > > > > > > > > > > the Hypre guys, so good luck. > > > > > > > > > > > > > > Yaron > > > > > > > > > > > > > > On 9/19/06, Matt Funk wrote: > > > > > > > > Hi, > > > > > > > > > > > > > > > > i am trying to use boomeramg from hypre in petsc. What i am > > > > > > > > doing > > > > > > > > > > right > > > > > > > > > > > > > now is > > > > > > > > simply do: > > > > > > > > > > > > > > > > m_ierr = PCSetType(m_pc,PCHYPRE);//CHKERRQ(ierr) > > > > > > > > m_ierr = PCHYPRESetType(m_pc,"boomeramg"); > > > > > > > > > > > > > > > > then i do my kspsetup etc... > > > > > > > > > > > > > > > > i was wondering what other things i need to configure/set to > > > > > > > > > > properly use > > > > > > > > > > > > > the > > > > > > > > hypre preconditioner with petsc. This is new to me, so i am > > > > > > > > not > > > > > > > > > > really > > > > > > > > > > > > > experienced with this (that is using external packages > > > > > > > > through > > > > > > > > > > petsc). > > > > > > > > > > > > > Also, > > > > > > > > these are the only hypre calls i found in the manual pages. > > > > > > > > > > > > > > > > Maybe somebody with more experience can share some of their > > > > > > > > wisdom > > > > > > > > > > ..., > > > > > > > > > > > > > or point to a place where i can read up on it? > > > > > > > > > > > > > > > > > > > > > > > > thanks > > > > > > > > mat From bsmith at mcs.anl.gov Wed Oct 4 07:17:13 2006 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 4 Oct 2006 07:17:13 -0500 (CDT) Subject: using hypre through petsc In-Reply-To: <200610032214.17023.mafunk@nmsu.edu> References: <215294220609190330n630747f7p9c1996c4d38f23ae@mail.gmail.com> <200609291719.22969.mafunk@nmsu.edu> <200610032214.17023.mafunk@nmsu.edu> Message-ID: The listed ones are the only ones accessible through PETSc. Note that the listed ones are general purpose, you give them a matrix and off they go. The ones you mention SMG and PFMG are not general purpose they require more information than just a matrix so they could not possibly be used simply by switching a command line option. Barry On Tue, 3 Oct 2006, Matt Funk wrote: > Hi guys, > > first of all thanks for all the help. I was wondering another thing now > though. The hypre options that i PETSC prints are: > > -pc_type Preconditioner:(one of) none jacobi pbjacobi bjacobi sor lu shell > mg > eisenstat ilu icc cholesky asm ksp composite redundant nn mat fieldsplit > hypre tfs (PCSetType) > HYPRE preconditioner options > -pc_hypre_type (choose one of) pilut parasails boomeramg euclid > > > I was wondering how come the other hypre preconditioners such as SMG or PFMG > are not listed. (Or did i miss something)? > > thanks > mat > > On Friday 29 September 2006 19:56, Barry Smith wrote: > > Yes, you must call SNESSetFromOptions, KSPSetFromOptions or > > PCSetFromOptions. It is at this point in the code that it checks the > > provided options and prints the help messages (as it checks them). > > > > You do not need to hardwire the options in the code to have the proper > > help printed. > > > > Barry > > > > On Fri, 29 Sep 2006, Matt Funk wrote: > > > Hi, > > > > > > i don't know what i am doing wrong, but it still doesn't give me the any > > > hypre output. > > > > > > So i need to call KSPSetFromOptions? or PCSetFromOptions? > > > > > > Also, at what point in the program should it print those options out? > > > After i call XSetFromOptions? > > > > > > Do i need to set the PCType to hypre first and then > > > PCHYPRESetType(m_pc,"boomeramg") and then it prints it out? > > > > > > mat > > > > > > On Friday 29 September 2006 15:35, Barry Smith wrote: > > > > On Fri, 29 Sep 2006, Yaron Kretchmer wrote: > > > > > If the latter (Petsc not configured to use Hypre), wouldn't Matt get > > > > > an error when specifying hypre as preconditioner? > > > > > > > > Yes he would, but it could get lost among all the help messages > > > > printed. > > > > > > > > Barry > > > > > > > > > Yaron > > > > > > > > > > On 9/29/06, Barry Smith wrote: > > > > > > Either you are not calling KSPSetFromOptions() in your > > > > > > code (or SNESSetFromOptions() OR PETSc was not configured > > > > > > to use hypre. To configure PETSc to use hypre you > > > > > > must use the additional config/configure.py option > > > > > > --download-hypre > > > > > > > > > > > > Good luck, > > > > > > > > > > > > Barry > > > > > > > > > > > > On Fri, 29 Sep 2006, Matt Funk wrote: > > > > > > > This might be a pretty stupid question, especially since i did > > > > > > > this a > > > > > > > > > > > > while > > > > > > > > > > > > > back and it worked. > > > > > > > > > > > > > > However, how do i specify thsis? Do i run my executable do and > > > > > > > then > > > > > > > > > > > > simply > > > > > > > > > > > > > append -pc_type hypre -pc_hypre_type boomeramg -help and it will > > > > > > > print > > > > > > > > > > > > out > > > > > > > > > > > > > the options? > > > > > > > > > > > > > > > > > > > > > Because when i do that i get: > > > > > > > Options for all PETSc programs: > > > > > > > -on_error_abort: cause an abort when an error is detected. > > > > > > > Useful only when run in the debugger > > > > > > > -on_error_attach_debugger [gdb,dbx,xxgdb,ups,noxterm] > > > > > > > start the debugger in new xterm > > > > > > > unless noxterm is given > > > > > > > -start_in_debugger [gdb,dbx,xxgdb,ups,noxterm] > > > > > > > start all processes in the debugger > > > > > > > -on_error_emacs > > > > > > > emacs jumps to error file > > > > > > > -debugger_nodes [n1,n2,..] Nodes to start in debugger > > > > > > > -debugger_pause [m] : delay (in seconds) to attach debugger > > > > > > > -stop_for_debugger : prints message on how to attach debugger > > > > > > > manually waits the delay for you to attach > > > > > > > -display display: Location where graphics and debuggers are > > > > > > > displayed -no_signal_handler: do not trap error signals > > > > > > > -mpi_return_on_error: MPI returns error code, rather than abort > > > > > > > on > > > > > > > > > > > > internal > > > > > > > > > > > > > error > > > > > > > -fp_trap: stop on floating point exceptions > > > > > > > note on IBM RS6000 this slows run greatly > > > > > > > -malloc_dump : dump list of unfreed memory at > > > > > > > > > > > > conclusion > > > > > > > > > > > > > -malloc: use our error checking malloc > > > > > > > -malloc no: don't use error checking malloc > > > > > > > -mallocinfo: prints total memory usage > > > > > > > -malloc_debug: enables extended checking for memory corruption > > > > > > > -options_table: dump list of options inputted > > > > > > > -options_left: dump list of unused options > > > > > > > -options_left no: don't dump list of unused options > > > > > > > -tmp tmpdir: alternative /tmp directory > > > > > > > -shared_tmp: tmp directory is shared by all processors > > > > > > > -not_shared_tmp: each processor has separate tmp directory > > > > > > > -memory_info: print memory usage at end of run > > > > > > > -get_total_flops: total flops over all processors > > > > > > > -log[_all _summary]: logging objects and events > > > > > > > -log_trace [filename]: prints trace of all PETSc calls > > > > > > > -info : print informative messages about the > > > > > > > > > > > > calculations > > > > > > > > > > > > > -v: prints PETSc version number and release date > > > > > > > -options_file : reads options from file > > > > > > > -petsc_sleep n: sleeps n seconds before running program > > > > > > > > > > > > > > > > > > > > > but nothing about hypre .... ? > > > > > > > > > > > > > > > > > > > > > mat > > > > > > > > > > > > > > On Tuesday 19 September 2006 17:05, Yaron Kretchmer wrote: > > > > > > > > Hi Matt > > > > > > > > You can start experimenting by using > > > > > > > > -pc_type hypre -pc_hypre_type boomeramg -help > > > > > > > > which will give you all the boomeramg options. > > > > > > > > > > > > > > > > When I use it I get intermittent segfaults, which I couldn't > > > > > > > > resolve > > > > > > > > > > > > with > > > > > > > > > > > > > > the Hypre guys, so good luck. > > > > > > > > > > > > > > > > Yaron > > > > > > > > > > > > > > > > On 9/19/06, Matt Funk wrote: > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > i am trying to use boomeramg from hypre in petsc. What i am > > > > > > > > > doing > > > > > > > > > > > > right > > > > > > > > > > > > > > > now is > > > > > > > > > simply do: > > > > > > > > > > > > > > > > > > m_ierr = PCSetType(m_pc,PCHYPRE);//CHKERRQ(ierr) > > > > > > > > > m_ierr = PCHYPRESetType(m_pc,"boomeramg"); > > > > > > > > > > > > > > > > > > then i do my kspsetup etc... > > > > > > > > > > > > > > > > > > i was wondering what other things i need to configure/set to > > > > > > > > > > > > properly use > > > > > > > > > > > > > > > the > > > > > > > > > hypre preconditioner with petsc. This is new to me, so i am > > > > > > > > > not > > > > > > > > > > > > really > > > > > > > > > > > > > > > experienced with this (that is using external packages > > > > > > > > > through > > > > > > > > > > > > petsc). > > > > > > > > > > > > > > > Also, > > > > > > > > > these are the only hypre calls i found in the manual pages. > > > > > > > > > > > > > > > > > > Maybe somebody with more experience can share some of their > > > > > > > > > wisdom > > > > > > > > > > > > ..., > > > > > > > > > > > > > > > or point to a place where i can read up on it? > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks > > > > > > > > > mat > > From mafunk at nmsu.edu Wed Oct 4 10:47:27 2006 From: mafunk at nmsu.edu (Matt Funk) Date: Wed, 4 Oct 2006 09:47:27 -0600 Subject: using hypre through petsc In-Reply-To: References: <215294220609190330n630747f7p9c1996c4d38f23ae@mail.gmail.com> <200610032214.17023.mafunk@nmsu.edu> Message-ID: <200610040947.29271.mafunk@nmsu.edu> Ok, thanks mat On Wednesday 04 October 2006 06:17, Barry Smith wrote: > The listed ones are the only ones accessible through PETSc. > Note that the listed ones are general purpose, you give them a matrix > and off they go. The ones you mention SMG and PFMG are not > general purpose they require more information than just a matrix > so they could not possibly be used simply by switching a command line > option. > > Barry > > On Tue, 3 Oct 2006, Matt Funk wrote: > > Hi guys, > > > > first of all thanks for all the help. I was wondering another thing now > > though. The hypre options that i PETSC prints are: > > > > -pc_type Preconditioner:(one of) none jacobi pbjacobi bjacobi sor lu > > shell mg > > eisenstat ilu icc cholesky asm ksp composite redundant nn mat > > fieldsplit hypre tfs (PCSetType) > > HYPRE preconditioner options > > -pc_hypre_type (choose one of) pilut parasails boomeramg euclid > > > > > > I was wondering how come the other hypre preconditioners such as SMG or > > PFMG are not listed. (Or did i miss something)? > > > > thanks > > mat > > > > On Friday 29 September 2006 19:56, Barry Smith wrote: > > > Yes, you must call SNESSetFromOptions, KSPSetFromOptions or > > > PCSetFromOptions. It is at this point in the code that it checks the > > > provided options and prints the help messages (as it checks them). > > > > > > You do not need to hardwire the options in the code to have the proper > > > help printed. > > > > > > Barry > > > > > > On Fri, 29 Sep 2006, Matt Funk wrote: > > > > Hi, > > > > > > > > i don't know what i am doing wrong, but it still doesn't give me the > > > > any hypre output. > > > > > > > > So i need to call KSPSetFromOptions? or PCSetFromOptions? > > > > > > > > Also, at what point in the program should it print those options out? > > > > After i call XSetFromOptions? > > > > > > > > Do i need to set the PCType to hypre first and then > > > > PCHYPRESetType(m_pc,"boomeramg") and then it prints it out? > > > > > > > > mat > > > > > > > > On Friday 29 September 2006 15:35, Barry Smith wrote: > > > > > On Fri, 29 Sep 2006, Yaron Kretchmer wrote: > > > > > > If the latter (Petsc not configured to use Hypre), wouldn't Matt > > > > > > get an error when specifying hypre as preconditioner? > > > > > > > > > > Yes he would, but it could get lost among all the help messages > > > > > printed. > > > > > > > > > > Barry > > > > > > > > > > > Yaron > > > > > > > > > > > > On 9/29/06, Barry Smith wrote: > > > > > > > Either you are not calling KSPSetFromOptions() in your > > > > > > > code (or SNESSetFromOptions() OR PETSc was not configured > > > > > > > to use hypre. To configure PETSc to use hypre you > > > > > > > must use the additional config/configure.py option > > > > > > > --download-hypre > > > > > > > > > > > > > > Good luck, > > > > > > > > > > > > > > Barry > > > > > > > > > > > > > > On Fri, 29 Sep 2006, Matt Funk wrote: > > > > > > > > This might be a pretty stupid question, especially since i > > > > > > > > did this a > > > > > > > > > > > > > > while > > > > > > > > > > > > > > > back and it worked. > > > > > > > > > > > > > > > > However, how do i specify thsis? Do i run my executable do > > > > > > > > and then > > > > > > > > > > > > > > simply > > > > > > > > > > > > > > > append -pc_type hypre -pc_hypre_type boomeramg -help and it > > > > > > > > will print > > > > > > > > > > > > > > out > > > > > > > > > > > > > > > the options? > > > > > > > > > > > > > > > > > > > > > > > > Because when i do that i get: > > > > > > > > Options for all PETSc programs: > > > > > > > > -on_error_abort: cause an abort when an error is detected. > > > > > > > > Useful only when run in the debugger > > > > > > > > -on_error_attach_debugger [gdb,dbx,xxgdb,ups,noxterm] > > > > > > > > start the debugger in new xterm > > > > > > > > unless noxterm is given > > > > > > > > -start_in_debugger [gdb,dbx,xxgdb,ups,noxterm] > > > > > > > > start all processes in the debugger > > > > > > > > -on_error_emacs > > > > > > > > emacs jumps to error file > > > > > > > > -debugger_nodes [n1,n2,..] Nodes to start in debugger > > > > > > > > -debugger_pause [m] : delay (in seconds) to attach debugger > > > > > > > > -stop_for_debugger : prints message on how to attach > > > > > > > > debugger manually waits the delay for you to attach > > > > > > > > -display display: Location where graphics and debuggers are > > > > > > > > displayed -no_signal_handler: do not trap error signals > > > > > > > > -mpi_return_on_error: MPI returns error code, rather than > > > > > > > > abort on > > > > > > > > > > > > > > internal > > > > > > > > > > > > > > > error > > > > > > > > -fp_trap: stop on floating point exceptions > > > > > > > > note on IBM RS6000 this slows run greatly > > > > > > > > -malloc_dump : dump list of unfreed > > > > > > > > memory at > > > > > > > > > > > > > > conclusion > > > > > > > > > > > > > > > -malloc: use our error checking malloc > > > > > > > > -malloc no: don't use error checking malloc > > > > > > > > -mallocinfo: prints total memory usage > > > > > > > > -malloc_debug: enables extended checking for memory > > > > > > > > corruption -options_table: dump list of options inputted > > > > > > > > -options_left: dump list of unused options > > > > > > > > -options_left no: don't dump list of unused options > > > > > > > > -tmp tmpdir: alternative /tmp directory > > > > > > > > -shared_tmp: tmp directory is shared by all processors > > > > > > > > -not_shared_tmp: each processor has separate tmp directory > > > > > > > > -memory_info: print memory usage at end of run > > > > > > > > -get_total_flops: total flops over all processors > > > > > > > > -log[_all _summary]: logging objects and events > > > > > > > > -log_trace [filename]: prints trace of all PETSc calls > > > > > > > > -info : print informative messages about > > > > > > > > the > > > > > > > > > > > > > > calculations > > > > > > > > > > > > > > > -v: prints PETSc version number and release date > > > > > > > > -options_file : reads options from file > > > > > > > > -petsc_sleep n: sleeps n seconds before running program > > > > > > > > > > > > > > > > > > > > > > > > but nothing about hypre .... ? > > > > > > > > > > > > > > > > > > > > > > > > mat > > > > > > > > > > > > > > > > On Tuesday 19 September 2006 17:05, Yaron Kretchmer wrote: > > > > > > > > > Hi Matt > > > > > > > > > You can start experimenting by using > > > > > > > > > -pc_type hypre -pc_hypre_type boomeramg -help > > > > > > > > > which will give you all the boomeramg options. > > > > > > > > > > > > > > > > > > When I use it I get intermittent segfaults, which I > > > > > > > > > couldn't resolve > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > the Hypre guys, so good luck. > > > > > > > > > > > > > > > > > > Yaron > > > > > > > > > > > > > > > > > > On 9/19/06, Matt Funk wrote: > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > i am trying to use boomeramg from hypre in petsc. What i > > > > > > > > > > am doing > > > > > > > > > > > > > > right > > > > > > > > > > > > > > > > > now is > > > > > > > > > > simply do: > > > > > > > > > > > > > > > > > > > > m_ierr = PCSetType(m_pc,PCHYPRE);//CHKERRQ(ierr) > > > > > > > > > > m_ierr = PCHYPRESetType(m_pc,"boomeramg"); > > > > > > > > > > > > > > > > > > > > then i do my kspsetup etc... > > > > > > > > > > > > > > > > > > > > i was wondering what other things i need to configure/set > > > > > > > > > > to > > > > > > > > > > > > > > properly use > > > > > > > > > > > > > > > > > the > > > > > > > > > > hypre preconditioner with petsc. This is new to me, so i > > > > > > > > > > am not > > > > > > > > > > > > > > really > > > > > > > > > > > > > > > > > experienced with this (that is using external packages > > > > > > > > > > through > > > > > > > > > > > > > > petsc). > > > > > > > > > > > > > > > > > Also, > > > > > > > > > > these are the only hypre calls i found in the manual > > > > > > > > > > pages. > > > > > > > > > > > > > > > > > > > > Maybe somebody with more experience can share some of > > > > > > > > > > their wisdom > > > > > > > > > > > > > > ..., > > > > > > > > > > > > > > > > > or point to a place where i can read up on it? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks > > > > > > > > > > mat From xiwang at dragon.rutgers.edu Wed Oct 4 13:18:09 2006 From: xiwang at dragon.rutgers.edu (Xiaoxu Wang) Date: Wed, 04 Oct 2006 14:18:09 -0400 Subject: MatCreateMPIAIJ In-Reply-To: <4522A609.8050106@dragon.rutgers.edu> References: <4522A609.8050106@dragon.rutgers.edu> Message-ID: <4523FAE1.4040302@dragon.rutgers.edu> Hi, Thank you for your help already. Can anybody tell me how to set the size of local matrix? Can I set it arbitrarily? Xiaoxu Xiaoxu Wang wrote: > > The error is at line 1023 of "plog.c". > --------------- > An unhandled exception of type 'System.NullReferenceException' > occurred in Unknown Module. > > Additional information: Object reference not set to an instance of an > object. > ---------------- > > The matrix I am trying to initialize is the stiffness matrix K of > finite element methods. Later the dynamical system is solved as (I + > Dt*K) U(t+1) = U(t) + Dt.F(t) by calling KSPSolve. Any idea how to set > the number of local rows? I am running the code on PC now, but want > to keep MatCreateMPIAIJ in it. > Xiaoxu > ----- Original Message ----- From: "Barry Smith" > To: > Sent: Tuesday, October 03, 2006 8:32 AM > Subject: Re: MatCreateMPIAIJ > > >> >> >> On Mon, 2 Oct 2006, Xiaoxu Wang wrote: >> >>> Hi, >>> >>> I am a beginner on PETSC. This is my question about >>> MatCreateMPIAIJ. I >>> have a sparse matrix and only know the total number of nonzeros each >>> row. The >>> number of local rows and the number of local columns has been set to >>> PETSC_DECIDE. Therefore I don't know the size of each portion and it >>> is hard >>> to tell d_nnz and o_nnz. I use the total number of nunzeros as d_nnz >>> and get >>> an allocation error. >> >> >> >> It would be helpful to know the exact error message: >> >>> How to deal with this problem? Should I explicitly >>> specify the number of local rows and the number of local columns and >>> calculate >>> d_nnz and o_nnz? Or let PETSC decide the number of local rows and >>> the number >>> of local columns, and then use another function to get these >>> values? Any >>> other better solution? >> >> >> >> You should specifically set the number of local rows and columns >> and then >> compute the values of d_nnz and o_nnz. How you determine d_nnz and >> o_nnz is >> problem dependent and depends on what type of discretization you are >> using: >> finite differences, elements etc. >> >> >> Barry >> >>> >>> Thanks a lot, >>> Xiaoxu >>> >>> >> >> > From knepley at gmail.com Wed Oct 4 13:28:55 2006 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 4 Oct 2006 13:28:55 -0500 Subject: MatCreateMPIAIJ In-Reply-To: <4523FAE1.4040302@dragon.rutgers.edu> References: <4522A609.8050106@dragon.rutgers.edu> <4523FAE1.4040302@dragon.rutgers.edu> Message-ID: On 10/4/06, Xiaoxu Wang wrote: > > Hi, Thank you for your help already. Can anybody tell me how to set the > size of local matrix? Can I set it arbitrarily? Yes, you can. If you also set the global size it must be consistent. Matt Xiaoxu > > Xiaoxu Wang wrote: > > > > > The error is at line 1023 of "plog.c". > > --------------- > > An unhandled exception of type 'System.NullReferenceException' > > occurred in Unknown Module. > > > > Additional information: Object reference not set to an instance of an > > object. > > ---------------- > > > > The matrix I am trying to initialize is the stiffness matrix K of > > finite element methods. Later the dynamical system is solved as (I + > > Dt*K) U(t+1) = U(t) + Dt.F(t) by calling KSPSolve. Any idea how to set > > the number of local rows? I am running the code on PC now, but want > > to keep MatCreateMPIAIJ in it. > > Xiaoxu > > ----- Original Message ----- From: "Barry Smith" > > To: > > Sent: Tuesday, October 03, 2006 8:32 AM > > Subject: Re: MatCreateMPIAIJ > > > > > >> > >> > >> On Mon, 2 Oct 2006, Xiaoxu Wang wrote: > >> > >>> Hi, > >>> > >>> I am a beginner on PETSC. This is my question about > >>> MatCreateMPIAIJ. I > >>> have a sparse matrix and only know the total number of nonzeros each > >>> row. The > >>> number of local rows and the number of local columns has been set to > >>> PETSC_DECIDE. Therefore I don't know the size of each portion and it > >>> is hard > >>> to tell d_nnz and o_nnz. I use the total number of nunzeros as d_nnz > >>> and get > >>> an allocation error. > >> > >> > >> > >> It would be helpful to know the exact error message: > >> > >>> How to deal with this problem? Should I explicitly > >>> specify the number of local rows and the number of local columns and > >>> calculate > >>> d_nnz and o_nnz? Or let PETSC decide the number of local rows and > >>> the number > >>> of local columns, and then use another function to get these > >>> values? Any > >>> other better solution? > >> > >> > >> > >> You should specifically set the number of local rows and columns > >> and then > >> compute the values of d_nnz and o_nnz. How you determine d_nnz and > >> o_nnz is > >> problem dependent and depends on what type of discretization you are > >> using: > >> finite differences, elements etc. > >> > >> > >> Barry > >> > >>> > >>> Thanks a lot, > >>> Xiaoxu > >>> > >>> > >> > >> > > > > -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness -------------- next part -------------- An HTML attachment was scrubbed... URL: From xiwang at dragon.rutgers.edu Wed Oct 4 14:00:57 2006 From: xiwang at dragon.rutgers.edu (Xiaoxu Wang) Date: Wed, 04 Oct 2006 15:00:57 -0400 Subject: MatCreateMPIAIJ In-Reply-To: References: <4522A609.8050106@dragon.rutgers.edu> <4523FAE1.4040302@dragon.rutgers.edu> Message-ID: <452404E9.2030007@dragon.rutgers.edu> Does it work as long as the global length is several times of the local length? Thanks Xiaoxu Matthew Knepley wrote: > On 10/4/06, *Xiaoxu Wang* > wrote: > > Hi, Thank you for your help already. Can anybody tell me how to > set the > size of local matrix? Can I set it arbitrarily? > > > Yes, you can. If you also set the global size it must be consistent. > > Matt > > Xiaoxu > > Xiaoxu Wang wrote: > > > > > The error is at line 1023 of " plog.c". > > --------------- > > An unhandled exception of type 'System.NullReferenceException' > > occurred in Unknown Module. > > > > Additional information: Object reference not set to an instance > of an > > object. > > ---------------- > > > > The matrix I am trying to initialize is the stiffness matrix K of > > finite element methods. Later the dynamical system is solved > as (I + > > Dt*K) U(t+1) = U(t) + Dt.F(t) by calling KSPSolve. Any idea how > to set > > the number of local rows? I am running the code on PC now, but want > > to keep MatCreateMPIAIJ in it. > > Xiaoxu > > ----- Original Message ----- From: "Barry Smith" < > bsmith at mcs.anl.gov > > > To: > > > Sent: Tuesday, October 03, 2006 8:32 AM > > Subject: Re: MatCreateMPIAIJ > > > > > >> > >> > >> On Mon, 2 Oct 2006, Xiaoxu Wang wrote: > >> > >>> Hi, > >>> > >>> I am a beginner on PETSC. This is my question about > >>> MatCreateMPIAIJ. I > >>> have a sparse matrix and only know the total number of > nonzeros each > >>> row. The > >>> number of local rows and the number of local columns has been > set to > >>> PETSC_DECIDE. Therefore I don't know the size of each portion > and it > >>> is hard > >>> to tell d_nnz and o_nnz. I use the total number of nunzeros as > d_nnz > >>> and get > >>> an allocation error. > >> > >> > >> > >> It would be helpful to know the exact error message: > >> > >>> How to deal with this problem? Should I explicitly > >>> specify the number of local rows and the number of local > columns and > >>> calculate > >>> d_nnz and o_nnz? Or let PETSC decide the number of local rows and > >>> the number > >>> of local columns, and then use another function to get these > >>> values? Any > >>> other better solution? > >> > >> > >> > >> You should specifically set the number of local rows and columns > >> and then > >> compute the values of d_nnz and o_nnz. How you determine d_nnz and > >> o_nnz is > >> problem dependent and depends on what type of discretization > you are > >> using: > >> finite differences, elements etc. > >> > >> > >> Barry > >> > >>> > >>> Thanks a lot, > >>> Xiaoxu > >>> > >>> > >> > >> > > > > > > > -- > "Failure has a thousand explanations. Success doesn't need one" -- Sir > Alec Guiness From knepley at gmail.com Wed Oct 4 14:13:52 2006 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 4 Oct 2006 14:13:52 -0500 Subject: MatCreateMPIAIJ In-Reply-To: <452404E9.2030007@dragon.rutgers.edu> References: <4522A609.8050106@dragon.rutgers.edu> <4523FAE1.4040302@dragon.rutgers.edu> <452404E9.2030007@dragon.rutgers.edu> Message-ID: On 10/4/06, Xiaoxu Wang wrote: > > Does it work as long as the global length is several times of the local > length? I think you are misunderstanding these terms. The local length is the number of matrix rows assigned to the given process. The global length is the sum of the local lengths. Matt Thanks > Xiaoxu > > Matthew Knepley wrote: > > > On 10/4/06, *Xiaoxu Wang* > > wrote: > > > > Hi, Thank you for your help already. Can anybody tell me how to > > set the > > size of local matrix? Can I set it arbitrarily? > > > > > > Yes, you can. If you also set the global size it must be consistent. > > > > Matt > > > > Xiaoxu > > > > Xiaoxu Wang wrote: > > > > > > > > The error is at line 1023 of " plog.c". > > > --------------- > > > An unhandled exception of type 'System.NullReferenceException' > > > occurred in Unknown Module. > > > > > > Additional information: Object reference not set to an instance > > of an > > > object. > > > ---------------- > > > > > > The matrix I am trying to initialize is the stiffness matrix K of > > > finite element methods. Later the dynamical system is solved > > as (I + > > > Dt*K) U(t+1) = U(t) + Dt.F(t) by calling KSPSolve. Any idea how > > to set > > > the number of local rows? I am running the code on PC now, but > want > > > to keep MatCreateMPIAIJ in it. > > > Xiaoxu > > > ----- Original Message ----- From: "Barry Smith" < > > bsmith at mcs.anl.gov > > > > To: > > > > Sent: Tuesday, October 03, 2006 8:32 AM > > > Subject: Re: MatCreateMPIAIJ > > > > > > > > >> > > >> > > >> On Mon, 2 Oct 2006, Xiaoxu Wang wrote: > > >> > > >>> Hi, > > >>> > > >>> I am a beginner on PETSC. This is my question about > > >>> MatCreateMPIAIJ. I > > >>> have a sparse matrix and only know the total number of > > nonzeros each > > >>> row. The > > >>> number of local rows and the number of local columns has been > > set to > > >>> PETSC_DECIDE. Therefore I don't know the size of each portion > > and it > > >>> is hard > > >>> to tell d_nnz and o_nnz. I use the total number of nunzeros as > > d_nnz > > >>> and get > > >>> an allocation error. > > >> > > >> > > >> > > >> It would be helpful to know the exact error message: > > >> > > >>> How to deal with this problem? Should I explicitly > > >>> specify the number of local rows and the number of local > > columns and > > >>> calculate > > >>> d_nnz and o_nnz? Or let PETSC decide the number of local rows > and > > >>> the number > > >>> of local columns, and then use another function to get these > > >>> values? Any > > >>> other better solution? > > >> > > >> > > >> > > >> You should specifically set the number of local rows and > columns > > >> and then > > >> compute the values of d_nnz and o_nnz. How you determine d_nnz > and > > >> o_nnz is > > >> problem dependent and depends on what type of discretization > > you are > > >> using: > > >> finite differences, elements etc. > > >> > > >> > > >> Barry > > >> > > >>> > > >>> Thanks a lot, > > >>> Xiaoxu > > >>> > > >>> > > >> > > >> > > > > > > > > > > > > > -- > > "Failure has a thousand explanations. Success doesn't need one" -- Sir > > Alec Guiness > > > -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness -------------- next part -------------- An HTML attachment was scrubbed... URL: From paulcon at stanford.edu Wed Oct 4 17:36:31 2006 From: paulcon at stanford.edu (Paul Constantine) Date: Wed, 04 Oct 2006 15:36:31 -0700 Subject: Newbie seeking MatSetValuesBlocked example Message-ID: <4524376F.2040309@stanford.edu> i just started using PETSc and i'm looking for a code example using MatSetValuesBlocked. thanks, paul From balay at mcs.anl.gov Fri Oct 6 10:30:17 2006 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 6 Oct 2006 10:30:17 -0500 (CDT) Subject: Newbie seeking MatSetValuesBlocked example In-Reply-To: <4524376F.2040309@stanford.edu> References: <4524376F.2040309@stanford.edu> Message-ID: They are not real applications - but ex52.c, ex56.c, ex64.c in src/mat/examples/tests show the usage of MatSetValuesBlocked(). Satish On Wed, 4 Oct 2006, Paul Constantine wrote: > i just started using PETSc and i'm looking for a code example using > MatSetValuesBlocked. > thanks, > paul > > From g0306037 at nus.edu.sg Fri Oct 6 02:28:31 2006 From: g0306037 at nus.edu.sg (He Xuefei) Date: Fri, 06 Oct 2006 15:28:31 +0800 Subject: Cannot install petsc on my Ubuntu Message-ID: <1160119711.5126.10.camel@localhost> Hi, I am a beginner with Linux. I just want to use petsc with C++ and do not need to use MPI. There is only petsc-dev in Ubuntu Synaptic Package Manager. After I installed it and followed your installation instructions, I always have "./config/configure.py No such file or directory". I have tried any methods on your website, but cannot fix that. I need to do this ./config/configure.py --with-clanguage=cxx --with-mpi=0, am I right? Could you please help me out? Tell me how to do step by step? Thank you very much. Xuefei From balay at mcs.anl.gov Fri Oct 6 10:38:25 2006 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 6 Oct 2006 10:38:25 -0500 (CDT) Subject: Cannot install petsc on my Ubuntu In-Reply-To: <1160119711.5126.10.camel@localhost> References: <1160119711.5126.10.camel@localhost> Message-ID: If you are using PETSc from 'Ubuntu Synaptic Package' - then you should follow their instructions on installing that package. I presume this package already includes precompiled libraries. However - if you download petsc-2.3.2 from our website - you'll need instructions from our website. I sugest downloading PETSc from http://www-unix.mcs.anl.gov/petsc/petsc-as/download/index.html and following the instructions at: http://www-unix.mcs.anl.gov/petsc/petsc-as/documentation/installation.html > ./config/configure.py --with-clanguage=cxx --with-mpi=0 Yes - this should work with our source distribution - provided you have python,gcc,g++,g77,blas,blas-dev,lapack,lapack-dev [or equivalent packages] already installed on your machine. Satish On Fri, 6 Oct 2006, He Xuefei wrote: > Hi, > > I am a beginner with Linux. I just want to use petsc with C++ and do not > need to use MPI. There is only petsc-dev in Ubuntu Synaptic Package > Manager. After I installed it and followed your installation > instructions, I always have "./config/configure.py No such file or > directory". I have tried any methods on your website, but cannot fix > that. > > I need to do this ./config/configure.py --with-clanguage=cxx > --with-mpi=0, am I right? > > Could you please help me out? Tell me how to do step by step? > > Thank you very much. > > Xuefei > > From knepley at gmail.com Fri Oct 6 10:39:53 2006 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 6 Oct 2006 10:39:53 -0500 Subject: Cannot install petsc on my Ubuntu In-Reply-To: <1160119711.5126.10.camel@localhost> References: <1160119711.5126.10.camel@localhost> Message-ID: On 10/6/06, He Xuefei wrote: > > Hi, > > I am a beginner with Linux. I just want to use petsc with C++ and do not > need to use MPI. There is only petsc-dev in Ubuntu Synaptic Package > Manager. After I installed it and followed your installation > instructions, I always have "./config/configure.py No such file or > directory". I have tried any methods on your website, but cannot fix > that. > > I need to do this ./config/configure.py --with-clanguage=cxx > --with-mpi=0, am I right? > > Could you please help me out? Tell me how to do step by step? The Ubuntu version uses the Debian build framework, and you should not need to build anything, jsut run the examples I believe. If you want to build, I suggest downloading the tarball from our website, and then following the installation instructions. Thanks, Matt Thank you very much. > > Xuefei > > -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness -------------- next part -------------- An HTML attachment was scrubbed... URL: From xiwang at dragon.rutgers.edu Sun Oct 8 17:08:52 2006 From: xiwang at dragon.rutgers.edu (Xiaoxu Wang) Date: Sun, 08 Oct 2006 18:08:52 -0400 Subject: MatCreateSeqAIJ Message-ID: <452976F4.308@dragon.rutgers.edu> Hi, Dear all, I am compiling the following code at Microsoft .net2003 and get the exception error. #include int main(int argc, char* argv[]) { Mat testMatrix; MatCreateSeqAIJ(PETSC_COMM_WORLD, 5, 5, 3, 0, &testMatrix); return 0; } -------------------------------- First-chance exception at 0x00419b1a in PetscTest.exe: 0xC0000005: Access violation reading location 0x00000018. Unhandled exception at 0x00419b1a in PetscTest.exe: 0xC0000005: Access violation reading location 0x00000018. First-chance exception at 0x00419b1a in PetscTest.exe: 0xC0000005: Access violation reading location 0x00000018. Unhandled exception at 0x00419b1a in PetscTest.exe: 0xC0000005: Access violation reading location 0x00000018. -------------------------------------------------------- The Petsc library is compiled using Cygwin. Is it because that different C compiler is used? Thanks a lot Xiaoxu From xiwang at dragon.rutgers.edu Sun Oct 8 17:15:42 2006 From: xiwang at dragon.rutgers.edu (Xiaoxu Wang) Date: Sun, 08 Oct 2006 18:15:42 -0400 Subject: please ignore the last message Message-ID: <4529788E.7090908@dragon.rutgers.edu> I forgot to initialize the system. Xiaoxu From mappol at gmail.com Mon Oct 9 10:22:37 2006 From: mappol at gmail.com (Patrick Lechner) Date: Mon, 9 Oct 2006 16:22:37 +0100 Subject: PETSc and OpenMP Message-ID: <215294220610090822t2fa3dd69ud4e0ffff815a128f@mail.gmail.com> Dear all, I am currently writing a code and try to combine PETSc and OpenMP (where PETSc will be used inside a parallel environment). My questions now are: 1. Is it safe to combine these libraries? 2. What will I have to consider to make my code work reliably? Thanks a lot for your help. Best wishes, Patrick -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Oct 9 10:30:28 2006 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 9 Oct 2006 10:30:28 -0500 Subject: PETSc and OpenMP In-Reply-To: <215294220610090822t2fa3dd69ud4e0ffff815a128f@mail.gmail.com> References: <215294220610090822t2fa3dd69ud4e0ffff815a128f@mail.gmail.com> Message-ID: On 10/9/06, Patrick Lechner wrote: > > Dear all, > I am currently writing a code and try to combine PETSc and OpenMP (where > PETSc will be used inside a parallel environment). > My questions now are: > 1. Is it safe to combine these libraries? > 2. What will I have to consider to make my code work reliably? The main problem will be thread safety. We do not claim to be thread-safe, as we occasionally use static variables. However, we try to minimize this. I think most of the uses are in output routines. You will also need a thread safe MPI (or use MPIUNI). We have tried this in the past, and never really gotten any benefit from doing it. What is your target application? Thanks, Matt Thanks a lot for your help. > Best wishes, > Patrick > -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Oct 9 10:50:47 2006 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 9 Oct 2006 10:50:47 -0500 (CDT) Subject: PETSc and OpenMP In-Reply-To: <215294220610090822t2fa3dd69ud4e0ffff815a128f@mail.gmail.com> References: <215294220610090822t2fa3dd69ud4e0ffff815a128f@mail.gmail.com> Message-ID: Patrick, Are you using "user level threads", that is your code stops and starts additional threads directly or are you using loop level threads, that is OpenMPI manages all the threads for you. If the later you are fine, if the former make sure only one thread makes PETSc calls. Barry On Mon, 9 Oct 2006, Patrick Lechner wrote: > Dear all, > I am currently writing a code and try to combine PETSc and OpenMP (where > PETSc will be used inside a parallel environment). > My questions now are: > 1. Is it safe to combine these libraries? > 2. What will I have to consider to make my code work reliably? > Thanks a lot for your help. > Best wishes, > Patrick > From mappol at gmail.com Mon Oct 9 11:12:01 2006 From: mappol at gmail.com (Patrick Lechner) Date: Mon, 9 Oct 2006 17:12:01 +0100 Subject: PETSc and OpenMP In-Reply-To: References: <215294220610090822t2fa3dd69ud4e0ffff815a128f@mail.gmail.com> Message-ID: <215294220610090912k7b292d5fq9d4b90a91416928@mail.gmail.com> Dear all, I basically use PETSc inside an existing OpenMP application. All PETSc solves are done in serial in this application. I therefore do not need MPI (at the moment...). The code solves finite element problems, where the OpenMP environment is used to use various parameters for these systems. All private variables are allocated on the stack, the OpenMP parallel loop makes calls to PETSc. Thanks for your replies, Patrick On 10/9/06, Barry Smith wrote: > > > Patrick, > > Are you using "user level threads", that is your code stops > and starts additional threads directly or are you using loop level > threads, that is OpenMPI manages all the threads for you. If the > later you are fine, if the former make sure only one thread > makes PETSc calls. > > Barry > > > On Mon, 9 Oct 2006, Patrick Lechner wrote: > > > Dear all, > > I am currently writing a code and try to combine PETSc and OpenMP (where > > PETSc will be used inside a parallel environment). > > My questions now are: > > 1. Is it safe to combine these libraries? > > 2. What will I have to consider to make my code work reliably? > > Thanks a lot for your help. > > Best wishes, > > Patrick > > > > -- ================================= Patrick Lechner Numerical Analysist / Numerical Modeller Flat 1 159 Hardgate Aberdeen, AB11 6XQ Phone: 07815 927333 E-mail: patrick at lechner.com Homepage: http://www.patrick.lechner.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Oct 9 11:42:35 2006 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 9 Oct 2006 11:42:35 -0500 (CDT) Subject: PETSc and OpenMP In-Reply-To: <215294220610090912k7b292d5fq9d4b90a91416928@mail.gmail.com> References: <215294220610090822t2fa3dd69ud4e0ffff815a128f@mail.gmail.com> <215294220610090912k7b292d5fq9d4b90a91416928@mail.gmail.com> Message-ID: On Mon, 9 Oct 2006, Patrick Lechner wrote: > Dear all, > > I basically use PETSc inside an existing OpenMP application. All PETSc > solves are done in serial in this application. I therefore do not need MPI > (at the moment...). The code solves finite element problems, where the > OpenMP environment is used to use various parameters for these systems. All > private variables are allocated on the stack, the OpenMP parallel loop makes > calls to PETSc. This is a problem. If you have do i=1,100000 call SomeCodeThatCallsPETSc() enddo and the OpenMPI uses threads to parallelize the loop; you will have multiple calls to PETSc "at the same time". PETSc may not handle this, wierd stuff is likely to happen since several threads will access/change PETSc global variables at the "same time": at least profiling and logging information maybe other stuff. So you will need, at least to not do any PETSc profiling. Let us know how it goes, Barry > Thanks for your replies, > Patrick > > > > On 10/9/06, Barry Smith wrote: > > > > > > Patrick, > > > > Are you using "user level threads", that is your code stops > > and starts additional threads directly or are you using loop level > > threads, that is OpenMPI manages all the threads for you. If the > > later you are fine, if the former make sure only one thread > > makes PETSc calls. > > > > Barry > > > > > > On Mon, 9 Oct 2006, Patrick Lechner wrote: > > > > > Dear all, > > > I am currently writing a code and try to combine PETSc and OpenMP (where > > > PETSc will be used inside a parallel environment). > > > My questions now are: > > > 1. Is it safe to combine these libraries? > > > 2. What will I have to consider to make my code work reliably? > > > Thanks a lot for your help. > > > Best wishes, > > > Patrick > > > > > > > > > > From randy at geosystem.us Tue Oct 10 13:57:23 2006 From: randy at geosystem.us (Randall Mackie) Date: Tue, 10 Oct 2006 11:57:23 -0700 Subject: question on DMMG Message-ID: <452BED13.3070408@geosystem.us> I'm interested in trying to use multi-grid techniques to solve my problem, but it's unclear to me how to define the right-hand side and system matrix for my case where the grids are not uniform (ie, the grid lengths x,y,z are variable, and the grid properties are variable). Right now I'm solving my equations using a DA. So, on the finest scale I know how to compute the RHS and the Jacobian. It's unclear to me how PETSc does the multi-gridding, and therefore how I would write my code. Any pointers would be appreciated. Thanks, Randy -- Randall Mackie GSY-USA, Inc. PMB# 643 2261 Market St., San Francisco, CA 94114-1600 Tel (415) 469-8649 Fax (415) 469-5044 California Registered Geophysicist License No. GP 1034 From bsmith at mcs.anl.gov Tue Oct 10 14:53:04 2006 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 10 Oct 2006 14:53:04 -0500 (CDT) Subject: question on DMMG In-Reply-To: <452BED13.3070408@geosystem.us> References: <452BED13.3070408@geosystem.us> Message-ID: Randy, Take a look at src/ksp/ksp/examples/tutorials and DASetCoordinates(). You should do it all with a DMMG and then it should be pretty easy. Barry Note also: if you do not want to compute coarse matrices you can use DMMGSetGalerkin() or -dmmg_galerkin and have PETSc compute them. On Tue, 10 Oct 2006, Randall Mackie wrote: > I'm interested in trying to use multi-grid techniques to solve my problem, > but it's unclear to me how to define the right-hand side and system matrix > for my case where the grids are not uniform (ie, the grid lengths x,y,z are > variable, and the grid properties are variable). > > Right now I'm solving my equations using a DA. So, on the finest scale I know > how to compute the RHS and the Jacobian. It's unclear to me how PETSc > does the multi-gridding, and therefore how I would write my code. > > Any pointers would be appreciated. > > Thanks, Randy > > From randy at geosystem.us Tue Oct 10 19:28:34 2006 From: randy at geosystem.us (Randall Mackie) Date: Tue, 10 Oct 2006 17:28:34 -0700 Subject: question on DMMG In-Reply-To: References: <452BED13.3070408@geosystem.us> Message-ID: <452C3AB2.5090905@geosystem.us> Barry, If I use DMMGSetUseGalerkinCoarse, then do I need DMMGSetKsp, and where do you set up the finest level grid matrix and right hand side? Randy Barry Smith wrote: > Randy, > > Take a look at src/ksp/ksp/examples/tutorials and > DASetCoordinates(). > > You should do it all with a DMMG and then it should be > pretty easy. > > Barry > > Note also: if you do not want to compute coarse matrices > you can use DMMGSetGalerkin() or -dmmg_galerkin and have > PETSc compute them. > > > On Tue, 10 Oct 2006, Randall Mackie wrote: > >> I'm interested in trying to use multi-grid techniques to solve my problem, >> but it's unclear to me how to define the right-hand side and system matrix >> for my case where the grids are not uniform (ie, the grid lengths x,y,z are >> variable, and the grid properties are variable). >> >> Right now I'm solving my equations using a DA. So, on the finest scale I know >> how to compute the RHS and the Jacobian. It's unclear to me how PETSc >> does the multi-gridding, and therefore how I would write my code. >> >> Any pointers would be appreciated. >> >> Thanks, Randy >> >> > -- Randall Mackie GSY-USA, Inc. PMB# 643 2261 Market St., San Francisco, CA 94114-1600 Tel (415) 469-8649 Fax (415) 469-5044 California Registered Geophysicist License No. GP 1034 From bsmith at mcs.anl.gov Tue Oct 10 20:59:46 2006 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 10 Oct 2006 20:59:46 -0500 (CDT) Subject: question on DMMG In-Reply-To: <452C3AB2.5090905@geosystem.us> References: <452BED13.3070408@geosystem.us> <452C3AB2.5090905@geosystem.us> Message-ID: Randy, You use DMMGSetKSP() and set the functions to compute the matrix and right hand side. With galerkin these will only be called once on the finest grid. (Without the galerkin option it will call your matrix function on each level). You never "set" a matrix; PETSc calls your function to fill in the matrix. Barry On Tue, 10 Oct 2006, Randall Mackie wrote: > Barry, > > If I use DMMGSetUseGalerkinCoarse, then do I need DMMGSetKsp, and > where do you set up the finest level grid matrix and right hand > side? > > Randy > > > Barry Smith wrote: > > Randy, > > > > Take a look at src/ksp/ksp/examples/tutorials and > > DASetCoordinates(). > > You should do it all with a DMMG and then it should be > > pretty easy. > > > > Barry > > > > Note also: if you do not want to compute coarse matrices > > you can use DMMGSetGalerkin() or -dmmg_galerkin and have > > PETSc compute them. > > > > > > On Tue, 10 Oct 2006, Randall Mackie wrote: > > > > > I'm interested in trying to use multi-grid techniques to solve my problem, > > > but it's unclear to me how to define the right-hand side and system matrix > > > for my case where the grids are not uniform (ie, the grid lengths x,y,z > > > are > > > variable, and the grid properties are variable). > > > > > > Right now I'm solving my equations using a DA. So, on the finest scale I > > > know > > > how to compute the RHS and the Jacobian. It's unclear to me how PETSc > > > does the multi-gridding, and therefore how I would write my code. > > > > > > Any pointers would be appreciated. > > > > > > Thanks, Randy > > > > > > > > > > From julvar at tamu.edu Wed Oct 11 09:55:25 2006 From: julvar at tamu.edu (Julian) Date: Wed, 11 Oct 2006 09:55:25 -0500 Subject: General matrix interative solver Message-ID: <001901c6ed45$50794910$24b75ba5@aero.ad.tamu.edu> Hello, I implemented the iterative sparse matrix solver in PetSc into my FEM code recently. I compared the results from a problem with 1317 unknowns. I used a direct solver to obtain the reference solution. I have another in-house sparse iterative solver that I have been using so far. It was written by someone else but I have access to the source for that solver. I find the 'error norm' in the solution by taking the square root of the sum of the squares of the absolute differences between the solution from the direct solver and the iterative solver. I am ignoring the numerical zeros in the solutions when doing this. I find that in order to get same order of the error norm (1e-13) as the in-house iterative solver, the petsc solver takes a much longer time and larger number of iterations. While the inhouse solver took less than one second, the petsc solver took 13 seconds. The inhouse solver took 476 iterations whereas the petsc solver took 4738 iterations. I'm guessing this has to do with different setting of the solver in petsc such as the preconditioner etc. Can you tell me what the different settings are? And how to tweak them so that I can atleast get as good as a performance as the inhouse code ? Given below is how I have implemented the petsc solver: /////initialization PetscInitializeNoArguments(); Assert( mat = (double*)malloc(sizeof(Mat)) ); MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, PETSC_DEFAULT, PETSC_NULL, (Mat*)mat); ////// this is the function I use to populate the matrix MatSetValue(*(Mat*)mat, ii, jj, value, ADD_VALUES); ////// this is how I actaully solve the matrix MatAssemblyBegin(*(Mat*)mat, MAT_FINAL_ASSEMBLY); MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); double iter_error = 1e-10; int max_iter_num = 10000; int num_of_iter; Vec rhs, x; VecCreateSeqWithArray(PETSC_COMM_SELF, L, b, &rhs); VecDuplicate(rhs, &x); KSP ksp; KSPCreate(PETSC_COMM_SELF, &ksp); KSPSetTolerances(ksp, iter_error, PETSC_DEFAULT, PETSC_DEFAULT, max_iter_num); KSPSetFromOptions(ksp); KSPSetOperators(ksp, *(Mat*)mat, *(Mat*)mat, SAME_PRECONDITIONER); KSPSolve(ksp,rhs,x); PetscReal r_norm; KSPGetResidualNorm(ksp, &r_norm); KSPGetIterationNumber(ksp, &num_of_iter); cout << "max_iter_num\t" << max_iter_num << endl; cout << "iter_error\t" << iter_error << endl; cout << "Matrix solver step " << num_of_iter << ", residual " << r_norm << ".\n"; PetscScalar *p; VecGetArray(x, &p); for(int i=0; i cout << "iter_error\t" << iter_error << endl; > > cout << "Matrix solver step " << num_of_iter << ", residual " << > r_norm << ".\n"; > > PetscScalar *p; > VecGetArray(x, &p); > for(int i=0; i b[i] = p[i]; > } > VecRestoreArray(x, &p); > > KSPDestroy(ksp); > VecDestroy(rhs); > VecDestroy(x); > > cout <<"Iterations for convergence="<< num_of_iter << " - Residual Norm = " > << r_norm << endl; > > > > If this is not the typical method to be used to solve this kind of problem, > please let me know what functions I should use. > I should mention that the inhouse code is for symmetric matrices and from > what I understand, the petsc solver works for general unsymmetric matrices. > But I think for iterative solvers, it should still give around the same > performance. > I tested the solvers against some other problems as well, and I got the same > performance.. In some cases, no matter how many iterations it goes through, > the petsc solver would not go below a certain error norm whereas the inhouse > solver would get almost exactly the same answer as the direct solver > solution. I'm thinking the petsc solver should be able to solve this problem > just as easily. It would be great if anyone could help me figure out the > appropriate settings I should use in the petsc solver. > > Thanks, > Julian. > > From hzhang at mcs.anl.gov Wed Oct 11 10:23:52 2006 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 11 Oct 2006 10:23:52 -0500 (CDT) Subject: General matrix interative solver In-Reply-To: <001901c6ed45$50794910$24b75ba5@aero.ad.tamu.edu> References: <001901c6ed45$50794910$24b75ba5@aero.ad.tamu.edu> Message-ID: Julian, When comparing the inhouse solver and petsc solver, you need make sure 1. Timings are collected for solution process. The matrix and vector assembly should be excluded. 2. They should use same iterative algorithm. By default, petsc uses gmres with restart=30 and ilu(0) preconditioner. Petsc supports symmetric matrices, e.g., runtime option '-ksp_type cg -pc_type icc' might give better performance 3. They should start from same intial guess. By default, petsc initial guess is zero. You can use '-ksp_view' to see what algorithm and options are used in petsc. Hong On Wed, 11 Oct 2006, Julian wrote: > Hello, > > I implemented the iterative sparse matrix solver in PetSc into my FEM code > recently. I compared the results from a problem with 1317 unknowns. I used a > direct solver to obtain the reference solution. I have another in-house > sparse iterative solver that I have been using so far. It was written by > someone else but I have access to the source for that solver. > I find the 'error norm' in the solution by taking the square root of the sum > of the squares of the absolute differences between the solution from the > direct solver and the iterative solver. I am ignoring the numerical zeros in > the solutions when doing this. > I find that in order to get same order of the error norm (1e-13) as the > in-house iterative solver, the petsc solver takes a much longer time and > larger number of iterations. While the inhouse solver took less than one > second, the petsc solver took 13 seconds. The inhouse solver took 476 > iterations whereas the petsc solver took 4738 iterations. > I'm guessing this has to do with different setting of the solver in petsc > such as the preconditioner etc. > Can you tell me what the different settings are? And how to tweak them so > that I can atleast get as good as a performance as the inhouse code ? > Given below is how I have implemented the petsc solver: > > /////initialization > PetscInitializeNoArguments(); > Assert( mat = (double*)malloc(sizeof(Mat)) ); > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, > PETSC_DEFAULT, PETSC_NULL, (Mat*)mat); > > ////// this is the function I use to populate the matrix > MatSetValue(*(Mat*)mat, ii, jj, value, ADD_VALUES); > > ////// this is how I actaully solve the matrix > MatAssemblyBegin(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > double iter_error = 1e-10; > int max_iter_num = 10000; > int num_of_iter; > > Vec rhs, x; > VecCreateSeqWithArray(PETSC_COMM_SELF, L, b, &rhs); > VecDuplicate(rhs, &x); > > KSP ksp; > KSPCreate(PETSC_COMM_SELF, &ksp); > KSPSetTolerances(ksp, iter_error, PETSC_DEFAULT, > PETSC_DEFAULT, max_iter_num); > KSPSetFromOptions(ksp); > KSPSetOperators(ksp, *(Mat*)mat, *(Mat*)mat, > SAME_PRECONDITIONER); > KSPSolve(ksp,rhs,x); > > PetscReal r_norm; > KSPGetResidualNorm(ksp, &r_norm); > KSPGetIterationNumber(ksp, &num_of_iter); > > cout << "max_iter_num\t" << max_iter_num << endl; > cout << "iter_error\t" << iter_error << endl; > > cout << "Matrix solver step " << num_of_iter << ", residual " << > r_norm << ".\n"; > > PetscScalar *p; > VecGetArray(x, &p); > for(int i=0; i b[i] = p[i]; > } > VecRestoreArray(x, &p); > > KSPDestroy(ksp); > VecDestroy(rhs); > VecDestroy(x); > > cout <<"Iterations for convergence="<< num_of_iter << " - Residual Norm = " > << r_norm << endl; > > > > If this is not the typical method to be used to solve this kind of problem, > please let me know what functions I should use. > I should mention that the inhouse code is for symmetric matrices and from > what I understand, the petsc solver works for general unsymmetric matrices. > But I think for iterative solvers, it should still give around the same > performance. > I tested the solvers against some other problems as well, and I got the same > performance.. In some cases, no matter how many iterations it goes through, > the petsc solver would not go below a certain error norm whereas the inhouse > solver would get almost exactly the same answer as the direct solver > solution. I'm thinking the petsc solver should be able to solve this problem > just as easily. It would be great if anyone could help me figure out the > appropriate settings I should use in the petsc solver. > > Thanks, > Julian. > > From julvar at tamu.edu Wed Oct 11 11:22:53 2006 From: julvar at tamu.edu (Julian) Date: Wed, 11 Oct 2006 11:22:53 -0500 Subject: General matrix interative solver In-Reply-To: Message-ID: <005701c6ed51$81118220$24b75ba5@aero.ad.tamu.edu> Hong, The times are for the solution process alone and the initial guess is the same, i.e. zero. As for the algorithm, they are probably different. Can you tell me how to pass 'runtime options' like -help and -ksp_view from within the code... I mean, at compile time. thanks > > Julian, > > When comparing the inhouse solver and petsc solver, you need make sure > > 1. Timings are collected for solution process. The matrix and vector > assembly should be excluded. > 2. They should use same iterative algorithm. By default, petsc uses > gmres with restart=30 and ilu(0) preconditioner. Petsc supports > symmetric matrices, e.g., runtime option '-ksp_type cg > -pc_type icc' > might give better performance > 3. They should start from same intial guess. By default, petsc > initial guess is zero. > > You can use '-ksp_view' to see what algorithm and options are > used in petsc. > > Hong > > On Wed, 11 Oct 2006, Julian wrote: > > > Hello, > > > > I implemented the iterative sparse matrix solver in PetSc > into my FEM > > code recently. I compared the results from a problem with 1317 > > unknowns. I used a direct solver to obtain the reference > solution. I > > have another in-house sparse iterative solver that I have > been using > > so far. It was written by someone else but I have access to > the source for that solver. > > I find the 'error norm' in the solution by taking the > square root of > > the sum of the squares of the absolute differences between the > > solution from the direct solver and the iterative solver. I am > > ignoring the numerical zeros in the solutions when doing this. > > I find that in order to get same order of the error norm (1e-13) as > > the in-house iterative solver, the petsc solver takes a much longer > > time and larger number of iterations. While the inhouse solver took > > less than one second, the petsc solver took 13 seconds. The inhouse > > solver took 476 iterations whereas the petsc solver took > 4738 iterations. > > I'm guessing this has to do with different setting of the solver in > > petsc such as the preconditioner etc. > > Can you tell me what the different settings are? And how to > tweak them > > so that I can atleast get as good as a performance as the > inhouse code ? > > Given below is how I have implemented the petsc solver: > > > > /////initialization > > PetscInitializeNoArguments(); > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, > > PETSC_DEFAULT, PETSC_NULL, (Mat*)mat); > > > > ////// this is the function I use to populate the matrix > > MatSetValue(*(Mat*)mat, ii, jj, value, ADD_VALUES); > > > > ////// this is how I actaully solve the matrix > > MatAssemblyBegin(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > double iter_error = 1e-10; > > int max_iter_num = 10000; > > int num_of_iter; > > > > Vec rhs, x; > > VecCreateSeqWithArray(PETSC_COMM_SELF, L, b, &rhs); > > VecDuplicate(rhs, &x); > > > > KSP ksp; > > KSPCreate(PETSC_COMM_SELF, &ksp); > > KSPSetTolerances(ksp, iter_error, PETSC_DEFAULT, > > PETSC_DEFAULT, max_iter_num); > > KSPSetFromOptions(ksp); > > KSPSetOperators(ksp, *(Mat*)mat, *(Mat*)mat, > > SAME_PRECONDITIONER); > > KSPSolve(ksp,rhs,x); > > > > PetscReal r_norm; > > KSPGetResidualNorm(ksp, &r_norm); > > KSPGetIterationNumber(ksp, &num_of_iter); > > > > cout << "max_iter_num\t" << max_iter_num << endl; > > cout << "iter_error\t" << iter_error << endl; > > > > cout << "Matrix solver step " << num_of_iter << ", > residual " > > << r_norm << ".\n"; > > > > PetscScalar *p; > > VecGetArray(x, &p); > > for(int i=0; i > b[i] = p[i]; > > } > > VecRestoreArray(x, &p); > > > > KSPDestroy(ksp); > > VecDestroy(rhs); > > VecDestroy(x); > > > > cout <<"Iterations for convergence="<< num_of_iter << " - > Residual Norm = " > > << r_norm << endl; > > > > > > > > If this is not the typical method to be used to solve this kind of > > problem, please let me know what functions I should use. > > I should mention that the inhouse code is for symmetric > matrices and > > from what I understand, the petsc solver works for general > unsymmetric matrices. > > But I think for iterative solvers, it should still give around the > > same performance. > > I tested the solvers against some other problems as well, and I got > > the same performance.. In some cases, no matter how many > iterations it > > goes through, the petsc solver would not go below a certain > error norm > > whereas the inhouse solver would get almost exactly the > same answer as > > the direct solver solution. I'm thinking the petsc solver should be > > able to solve this problem just as easily. It would be > great if anyone > > could help me figure out the appropriate settings I should > use in the petsc solver. > > > > Thanks, > > Julian. > > > > > > From bsmith at mcs.anl.gov Wed Oct 11 11:25:00 2006 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 11 Oct 2006 11:25:00 -0500 (CDT) Subject: General matrix interative solver In-Reply-To: <005701c6ed51$81118220$24b75ba5@aero.ad.tamu.edu> References: <005701c6ed51$81118220$24b75ba5@aero.ad.tamu.edu> Message-ID: PetscOptionsSetValue() but recommend putting them on the command line or in a file (pass the name of the file into PetscInitialize()). Having to recompile for every option is too painful. Barry On Wed, 11 Oct 2006, Julian wrote: > Hong, > > The times are for the solution process alone and the initial guess is the > same, i.e. zero. > > As for the algorithm, they are probably different. > Can you tell me how to pass 'runtime options' like -help and -ksp_view from > within the code... I mean, at compile time. > > thanks > > > > > Julian, > > > > When comparing the inhouse solver and petsc solver, you need make sure > > > > 1. Timings are collected for solution process. The matrix and vector > > assembly should be excluded. > > 2. They should use same iterative algorithm. By default, petsc uses > > gmres with restart=30 and ilu(0) preconditioner. Petsc supports > > symmetric matrices, e.g., runtime option '-ksp_type cg > > -pc_type icc' > > might give better performance > > 3. They should start from same intial guess. By default, petsc > > initial guess is zero. > > > > You can use '-ksp_view' to see what algorithm and options are > > used in petsc. > > > > Hong > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > Hello, > > > > > > I implemented the iterative sparse matrix solver in PetSc > > into my FEM > > > code recently. I compared the results from a problem with 1317 > > > unknowns. I used a direct solver to obtain the reference > > solution. I > > > have another in-house sparse iterative solver that I have > > been using > > > so far. It was written by someone else but I have access to > > the source for that solver. > > > I find the 'error norm' in the solution by taking the > > square root of > > > the sum of the squares of the absolute differences between the > > > solution from the direct solver and the iterative solver. I am > > > ignoring the numerical zeros in the solutions when doing this. > > > I find that in order to get same order of the error norm (1e-13) as > > > the in-house iterative solver, the petsc solver takes a much longer > > > time and larger number of iterations. While the inhouse solver took > > > less than one second, the petsc solver took 13 seconds. The inhouse > > > solver took 476 iterations whereas the petsc solver took > > 4738 iterations. > > > I'm guessing this has to do with different setting of the solver in > > > petsc such as the preconditioner etc. > > > Can you tell me what the different settings are? And how to > > tweak them > > > so that I can atleast get as good as a performance as the > > inhouse code ? > > > Given below is how I have implemented the petsc solver: > > > > > > /////initialization > > > PetscInitializeNoArguments(); > > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, > > > PETSC_DEFAULT, PETSC_NULL, (Mat*)mat); > > > > > > ////// this is the function I use to populate the matrix > > > MatSetValue(*(Mat*)mat, ii, jj, value, ADD_VALUES); > > > > > > ////// this is how I actaully solve the matrix > > > MatAssemblyBegin(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > > > double iter_error = 1e-10; > > > int max_iter_num = 10000; > > > int num_of_iter; > > > > > > Vec rhs, x; > > > VecCreateSeqWithArray(PETSC_COMM_SELF, L, b, &rhs); > > > VecDuplicate(rhs, &x); > > > > > > KSP ksp; > > > KSPCreate(PETSC_COMM_SELF, &ksp); > > > KSPSetTolerances(ksp, iter_error, PETSC_DEFAULT, > > > PETSC_DEFAULT, max_iter_num); > > > KSPSetFromOptions(ksp); > > > KSPSetOperators(ksp, *(Mat*)mat, *(Mat*)mat, > > > SAME_PRECONDITIONER); > > > KSPSolve(ksp,rhs,x); > > > > > > PetscReal r_norm; > > > KSPGetResidualNorm(ksp, &r_norm); > > > KSPGetIterationNumber(ksp, &num_of_iter); > > > > > > cout << "max_iter_num\t" << max_iter_num << endl; > > > cout << "iter_error\t" << iter_error << endl; > > > > > > cout << "Matrix solver step " << num_of_iter << ", > > residual " > > > << r_norm << ".\n"; > > > > > > PetscScalar *p; > > > VecGetArray(x, &p); > > > for(int i=0; i > > b[i] = p[i]; > > > } > > > VecRestoreArray(x, &p); > > > > > > KSPDestroy(ksp); > > > VecDestroy(rhs); > > > VecDestroy(x); > > > > > > cout <<"Iterations for convergence="<< num_of_iter << " - > > Residual Norm = " > > > << r_norm << endl; > > > > > > > > > > > > If this is not the typical method to be used to solve this kind of > > > problem, please let me know what functions I should use. > > > I should mention that the inhouse code is for symmetric > > matrices and > > > from what I understand, the petsc solver works for general > > unsymmetric matrices. > > > But I think for iterative solvers, it should still give around the > > > same performance. > > > I tested the solvers against some other problems as well, and I got > > > the same performance.. In some cases, no matter how many > > iterations it > > > goes through, the petsc solver would not go below a certain > > error norm > > > whereas the inhouse solver would get almost exactly the > > same answer as > > > the direct solver solution. I'm thinking the petsc solver should be > > > able to solve this problem just as easily. It would be > > great if anyone > > > could help me figure out the appropriate settings I should > > use in the petsc solver. > > > > > > Thanks, > > > Julian. > > > > > > > > > > > > From julvar at tamu.edu Wed Oct 11 11:40:21 2006 From: julvar at tamu.edu (Julian) Date: Wed, 11 Oct 2006 11:40:21 -0500 Subject: General matrix interative solver In-Reply-To: Message-ID: <005b01c6ed53$f1df75a0$24b75ba5@aero.ad.tamu.edu> You are right.. This is a very small problem... I eventually plan on using it for very large problems. But for now, I wanted to confirm that I am using the right settings and I get an accurate enough solution by comparing it with the direct solver solution. Currently, with the way I have integrated it with my fea code, I am unable to send the command line arguments to the solver. Is there some easy way to send the arguments to the solver during command time? Julian. > -----Original Message----- > From: owner-petsc-users at mcs.anl.gov > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry Smith > Sent: Wednesday, October 11, 2006 10:22 AM > To: petsc-users at mcs.anl.gov > Subject: Re: General matrix interative solver > > > Julian, > > If your matrix is symmetric, positive definite then you > should ALWAYS use -ksp_type cg to use the conjugate gradient > method instead of the more expensive (and unneeded) GMRES. > > Next, in testing you can/should run with -ksp_view to have > the exact solver options printed out to the screen. > > Running with -help will have it print the current solver options. > > Run with -ksp_converged_reason to see if the solver > converged or if it failed and why. > > Next you need to determine what the inhouse solver is doing > and try to set PETSc to use the same solver. > > This problem is very, very small for an iterative solver; > recommend iterative solvers only when working with, say > 20,000+ unknowns (depends on matrix conditioning, number of > nonzero etcs also). For small problems direct solver is the > way to go. You can use the PETSc direct solver with -ksp_type > preonly -pc_type lu (or for symmetric positive definite use > -pc_type cholesky). > > Let us know what progress you make, it is imperative you > determine what the in house solver is using. > > Barry > > > > On Wed, 11 Oct 2006, Julian wrote: > > > Hello, > > > > I implemented the iterative sparse matrix solver in PetSc > into my FEM > > code recently. I compared the results from a problem with 1317 > > unknowns. I used a direct solver to obtain the reference > solution. I > > have another in-house sparse iterative solver that I have > been using > > so far. It was written by someone else but I have access to > the source for that solver. > > I find the 'error norm' in the solution by taking the > square root of > > the sum of the squares of the absolute differences between the > > solution from the direct solver and the iterative solver. I am > > ignoring the numerical zeros in the solutions when doing this. > > I find that in order to get same order of the error norm (1e-13) as > > the in-house iterative solver, the petsc solver takes a much longer > > time and larger number of iterations. While the inhouse solver took > > less than one second, the petsc solver took 13 seconds. The inhouse > > solver took 476 iterations whereas the petsc solver took > 4738 iterations. > > I'm guessing this has to do with different setting of the solver in > > petsc such as the preconditioner etc. > > Can you tell me what the different settings are? And how to > tweak them > > so that I can atleast get as good as a performance as the > inhouse code ? > > Given below is how I have implemented the petsc solver: > > > > /////initialization > > PetscInitializeNoArguments(); > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, > > PETSC_DEFAULT, PETSC_NULL, (Mat*)mat); > > > > ////// this is the function I use to populate the matrix > > MatSetValue(*(Mat*)mat, ii, jj, value, ADD_VALUES); > > > > ////// this is how I actaully solve the matrix > > MatAssemblyBegin(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > double iter_error = 1e-10; > > int max_iter_num = 10000; > > int num_of_iter; > > > > Vec rhs, x; > > VecCreateSeqWithArray(PETSC_COMM_SELF, L, b, &rhs); > > VecDuplicate(rhs, &x); > > > > KSP ksp; > > KSPCreate(PETSC_COMM_SELF, &ksp); > > KSPSetTolerances(ksp, iter_error, PETSC_DEFAULT, > > PETSC_DEFAULT, max_iter_num); > > KSPSetFromOptions(ksp); > > KSPSetOperators(ksp, *(Mat*)mat, *(Mat*)mat, > > SAME_PRECONDITIONER); > > KSPSolve(ksp,rhs,x); > > > > PetscReal r_norm; > > KSPGetResidualNorm(ksp, &r_norm); > > KSPGetIterationNumber(ksp, &num_of_iter); > > > > cout << "max_iter_num\t" << max_iter_num << endl; > > cout << "iter_error\t" << iter_error << endl; > > > > cout << "Matrix solver step " << num_of_iter << ", > residual " > > << r_norm << ".\n"; > > > > PetscScalar *p; > > VecGetArray(x, &p); > > for(int i=0; i > b[i] = p[i]; > > } > > VecRestoreArray(x, &p); > > > > KSPDestroy(ksp); > > VecDestroy(rhs); > > VecDestroy(x); > > > > cout <<"Iterations for convergence="<< num_of_iter << " - > Residual Norm = " > > << r_norm << endl; > > > > > > > > If this is not the typical method to be used to solve this kind of > > problem, please let me know what functions I should use. > > I should mention that the inhouse code is for symmetric > matrices and > > from what I understand, the petsc solver works for general > unsymmetric matrices. > > But I think for iterative solvers, it should still give around the > > same performance. > > I tested the solvers against some other problems as well, and I got > > the same performance.. In some cases, no matter how many > iterations it > > goes through, the petsc solver would not go below a certain > error norm > > whereas the inhouse solver would get almost exactly the > same answer as > > the direct solver solution. I'm thinking the petsc solver should be > > able to solve this problem just as easily. It would be > great if anyone > > could help me figure out the appropriate settings I should > use in the petsc solver. > > > > Thanks, > > Julian. > > > > > > From julvar at tamu.edu Wed Oct 11 12:36:50 2006 From: julvar at tamu.edu (Julian) Date: Wed, 11 Oct 2006 12:36:50 -0500 Subject: General matrix interative solver In-Reply-To: Message-ID: <005c01c6ed5b$d5469f60$24b75ba5@aero.ad.tamu.edu> Barry, I tried sending the commands from a file for now... And once I used '-ksp_type cg', I got pretty much the same performance as that of the inhouse code ! Now, I'll try the performance with larger cases... Thanks for your help! Julian. > -----Original Message----- > From: owner-petsc-users at mcs.anl.gov > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry Smith > Sent: Wednesday, October 11, 2006 11:25 AM > To: petsc-users at mcs.anl.gov > Subject: RE: General matrix interative solver > > > PetscOptionsSetValue() but recommend putting them on the > command line or in a file (pass the name of the file into > PetscInitialize()). Having to recompile for every option is > too painful. > > Barry > > > On Wed, 11 Oct 2006, Julian wrote: > > > Hong, > > > > The times are for the solution process alone and the > initial guess is > > the same, i.e. zero. > > > > As for the algorithm, they are probably different. > > Can you tell me how to pass 'runtime options' like -help > and -ksp_view > > from within the code... I mean, at compile time. > > > > thanks > > > > > > > > Julian, > > > > > > When comparing the inhouse solver and petsc solver, you need make > > > sure > > > > > > 1. Timings are collected for solution process. The matrix > and vector > > > assembly should be excluded. > > > 2. They should use same iterative algorithm. By default, > petsc uses > > > gmres with restart=30 and ilu(0) preconditioner. Petsc supports > > > symmetric matrices, e.g., runtime option '-ksp_type cg > -pc_type > > > icc' > > > might give better performance > > > 3. They should start from same intial guess. By default, petsc > > > initial guess is zero. > > > > > > You can use '-ksp_view' to see what algorithm and options > are used > > > in petsc. > > > > > > Hong > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > Hello, > > > > > > > > I implemented the iterative sparse matrix solver in PetSc > > > into my FEM > > > > code recently. I compared the results from a problem with 1317 > > > > unknowns. I used a direct solver to obtain the reference > > > solution. I > > > > have another in-house sparse iterative solver that I have > > > been using > > > > so far. It was written by someone else but I have access to > > > the source for that solver. > > > > I find the 'error norm' in the solution by taking the > > > square root of > > > > the sum of the squares of the absolute differences between the > > > > solution from the direct solver and the iterative solver. I am > > > > ignoring the numerical zeros in the solutions when doing this. > > > > I find that in order to get same order of the error > norm (1e-13) > > > > as the in-house iterative solver, the petsc solver takes a much > > > > longer time and larger number of iterations. While the inhouse > > > > solver took less than one second, the petsc solver took 13 > > > > seconds. The inhouse solver took 476 iterations whereas > the petsc > > > > solver took > > > 4738 iterations. > > > > I'm guessing this has to do with different setting of > the solver > > > > in petsc such as the preconditioner etc. > > > > Can you tell me what the different settings are? And how to > > > tweak them > > > > so that I can atleast get as good as a performance as the > > > inhouse code ? > > > > Given below is how I have implemented the petsc solver: > > > > > > > > /////initialization > > > > PetscInitializeNoArguments(); > > > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > > > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, > > > > PETSC_DEFAULT, PETSC_NULL, (Mat*)mat); > > > > > > > > ////// this is the function I use to populate the matrix > > > > MatSetValue(*(Mat*)mat, ii, jj, value, ADD_VALUES); > > > > > > > > ////// this is how I actaully solve the matrix > > > > MatAssemblyBegin(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > > > > > double iter_error = 1e-10; > > > > int max_iter_num = 10000; > > > > int num_of_iter; > > > > > > > > Vec rhs, x; > > > > VecCreateSeqWithArray(PETSC_COMM_SELF, L, b, &rhs); > > > > VecDuplicate(rhs, &x); > > > > > > > > KSP ksp; > > > > KSPCreate(PETSC_COMM_SELF, &ksp); > > > > KSPSetTolerances(ksp, iter_error, PETSC_DEFAULT, > > > > PETSC_DEFAULT, max_iter_num); > > > > KSPSetFromOptions(ksp); > > > > KSPSetOperators(ksp, *(Mat*)mat, *(Mat*)mat, > > > > SAME_PRECONDITIONER); > > > > KSPSolve(ksp,rhs,x); > > > > > > > > PetscReal r_norm; > > > > KSPGetResidualNorm(ksp, &r_norm); > > > > KSPGetIterationNumber(ksp, &num_of_iter); > > > > > > > > cout << "max_iter_num\t" << max_iter_num << endl; > > > > cout << "iter_error\t" << iter_error << endl; > > > > > > > > cout << "Matrix solver step " << num_of_iter << ", > > > residual " > > > > << r_norm << ".\n"; > > > > > > > > PetscScalar *p; > > > > VecGetArray(x, &p); > > > > for(int i=0; i > > > b[i] = p[i]; > > > > } > > > > VecRestoreArray(x, &p); > > > > > > > > KSPDestroy(ksp); > > > > VecDestroy(rhs); > > > > VecDestroy(x); > > > > > > > > cout <<"Iterations for convergence="<< num_of_iter << " - > > > Residual Norm = " > > > > << r_norm << endl; > > > > > > > > > > > > > > > > If this is not the typical method to be used to solve > this kind of > > > > problem, please let me know what functions I should use. > > > > I should mention that the inhouse code is for symmetric > > > matrices and > > > > from what I understand, the petsc solver works for general > > > unsymmetric matrices. > > > > But I think for iterative solvers, it should still give > around the > > > > same performance. > > > > I tested the solvers against some other problems as well, and I > > > > got the same performance.. In some cases, no matter how many > > > iterations it > > > > goes through, the petsc solver would not go below a certain > > > error norm > > > > whereas the inhouse solver would get almost exactly the > > > same answer as > > > > the direct solver solution. I'm thinking the petsc > solver should > > > > be able to solve this problem just as easily. It would be > > > great if anyone > > > > could help me figure out the appropriate settings I should > > > use in the petsc solver. > > > > > > > > Thanks, > > > > Julian. > > > > > > > > > > > > > > > > > > > From julvar at tamu.edu Wed Oct 11 14:07:21 2006 From: julvar at tamu.edu (Julian) Date: Wed, 11 Oct 2006 14:07:21 -0500 Subject: General matrix interative solver In-Reply-To: <005c01c6ed5b$d5469f60$24b75ba5@aero.ad.tamu.edu> Message-ID: <005f01c6ed68$7b1ce230$24b75ba5@aero.ad.tamu.edu> I'm using '-ksp_type cg' now to solve the symmetric matric...But then do I need to store all the non-zero elements of a symmetric matrix ? i.e. A[i,j] as well as A[j,i] ? Shouldn't I be able to store just the upper or lower triangle or the matrix ? Is that possible with petsc ? Julian. > -----Original Message----- > From: owner-petsc-users at mcs.anl.gov > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Julian > Sent: Wednesday, October 11, 2006 12:37 PM > To: petsc-users at mcs.anl.gov > Subject: RE: General matrix interative solver > > Barry, > > I tried sending the commands from a file for now... And once > I used '-ksp_type cg', I got pretty much the same performance > as that of the inhouse code ! > Now, I'll try the performance with larger cases... > > Thanks for your help! > > Julian. > > > > -----Original Message----- > > From: owner-petsc-users at mcs.anl.gov > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry Smith > > Sent: Wednesday, October 11, 2006 11:25 AM > > To: petsc-users at mcs.anl.gov > > Subject: RE: General matrix interative solver > > > > > > PetscOptionsSetValue() but recommend putting them on the command > > line or in a file (pass the name of the file into > PetscInitialize()). > > Having to recompile for every option is too painful. > > > > Barry > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > Hong, > > > > > > The times are for the solution process alone and the > > initial guess is > > > the same, i.e. zero. > > > > > > As for the algorithm, they are probably different. > > > Can you tell me how to pass 'runtime options' like -help > > and -ksp_view > > > from within the code... I mean, at compile time. > > > > > > thanks > > > > > > > > > > > Julian, > > > > > > > > When comparing the inhouse solver and petsc solver, you > need make > > > > sure > > > > > > > > 1. Timings are collected for solution process. The matrix > > and vector > > > > assembly should be excluded. > > > > 2. They should use same iterative algorithm. By default, > > petsc uses > > > > gmres with restart=30 and ilu(0) preconditioner. > Petsc supports > > > > symmetric matrices, e.g., runtime option '-ksp_type cg > > -pc_type > > > > icc' > > > > might give better performance > > > > 3. They should start from same intial guess. By default, petsc > > > > initial guess is zero. > > > > > > > > You can use '-ksp_view' to see what algorithm and options > > are used > > > > in petsc. > > > > > > > > Hong > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > Hello, > > > > > > > > > > I implemented the iterative sparse matrix solver in PetSc > > > > into my FEM > > > > > code recently. I compared the results from a problem > with 1317 > > > > > unknowns. I used a direct solver to obtain the reference > > > > solution. I > > > > > have another in-house sparse iterative solver that I have > > > > been using > > > > > so far. It was written by someone else but I have access to > > > > the source for that solver. > > > > > I find the 'error norm' in the solution by taking the > > > > square root of > > > > > the sum of the squares of the absolute differences > between the > > > > > solution from the direct solver and the iterative > solver. I am > > > > > ignoring the numerical zeros in the solutions when doing this. > > > > > I find that in order to get same order of the error > > norm (1e-13) > > > > > as the in-house iterative solver, the petsc solver > takes a much > > > > > longer time and larger number of iterations. While > the inhouse > > > > > solver took less than one second, the petsc solver took 13 > > > > > seconds. The inhouse solver took 476 iterations whereas > > the petsc > > > > > solver took > > > > 4738 iterations. > > > > > I'm guessing this has to do with different setting of > > the solver > > > > > in petsc such as the preconditioner etc. > > > > > Can you tell me what the different settings are? And how to > > > > tweak them > > > > > so that I can atleast get as good as a performance as the > > > > inhouse code ? > > > > > Given below is how I have implemented the petsc solver: > > > > > > > > > > /////initialization > > > > > PetscInitializeNoArguments(); > > > > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > > > > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, > > > > > PETSC_DEFAULT, PETSC_NULL, (Mat*)mat); > > > > > > > > > > ////// this is the function I use to populate the matrix > > > > > MatSetValue(*(Mat*)mat, ii, jj, value, ADD_VALUES); > > > > > > > > > > ////// this is how I actaully solve the matrix > > > > > MatAssemblyBegin(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > > MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > > > > > > > double iter_error = 1e-10; > > > > > int max_iter_num = 10000; > > > > > int num_of_iter; > > > > > > > > > > Vec rhs, x; > > > > > VecCreateSeqWithArray(PETSC_COMM_SELF, L, b, &rhs); > > > > > VecDuplicate(rhs, &x); > > > > > > > > > > KSP ksp; > > > > > KSPCreate(PETSC_COMM_SELF, &ksp); > > > > > KSPSetTolerances(ksp, iter_error, PETSC_DEFAULT, > > > > > PETSC_DEFAULT, max_iter_num); > > > > > KSPSetFromOptions(ksp); > > > > > KSPSetOperators(ksp, *(Mat*)mat, *(Mat*)mat, > > > > > SAME_PRECONDITIONER); > > > > > KSPSolve(ksp,rhs,x); > > > > > > > > > > PetscReal r_norm; > > > > > KSPGetResidualNorm(ksp, &r_norm); > > > > > KSPGetIterationNumber(ksp, &num_of_iter); > > > > > > > > > > cout << "max_iter_num\t" << max_iter_num << endl; > > > > > cout << "iter_error\t" << iter_error << endl; > > > > > > > > > > cout << "Matrix solver step " << num_of_iter << ", > > > > residual " > > > > > << r_norm << ".\n"; > > > > > > > > > > PetscScalar *p; > > > > > VecGetArray(x, &p); > > > > > for(int i=0; i > > > > b[i] = p[i]; > > > > > } > > > > > VecRestoreArray(x, &p); > > > > > > > > > > KSPDestroy(ksp); > > > > > VecDestroy(rhs); > > > > > VecDestroy(x); > > > > > > > > > > cout <<"Iterations for convergence="<< num_of_iter << " - > > > > Residual Norm = " > > > > > << r_norm << endl; > > > > > > > > > > > > > > > > > > > > If this is not the typical method to be used to solve > > this kind of > > > > > problem, please let me know what functions I should use. > > > > > I should mention that the inhouse code is for symmetric > > > > matrices and > > > > > from what I understand, the petsc solver works for general > > > > unsymmetric matrices. > > > > > But I think for iterative solvers, it should still give > > around the > > > > > same performance. > > > > > I tested the solvers against some other problems as > well, and I > > > > > got the same performance.. In some cases, no matter how many > > > > iterations it > > > > > goes through, the petsc solver would not go below a certain > > > > error norm > > > > > whereas the inhouse solver would get almost exactly the > > > > same answer as > > > > > the direct solver solution. I'm thinking the petsc > > solver should > > > > > be able to solve this problem just as easily. It would be > > > > great if anyone > > > > > could help me figure out the appropriate settings I should > > > > use in the petsc solver. > > > > > > > > > > Thanks, > > > > > Julian. > > > > > > > > > > > > > > > > > > > > > > > > > > > > From hzhang at mcs.anl.gov Wed Oct 11 14:13:40 2006 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 11 Oct 2006 14:13:40 -0500 (CDT) Subject: General matrix interative solver In-Reply-To: <005f01c6ed68$7b1ce230$24b75ba5@aero.ad.tamu.edu> References: <005f01c6ed68$7b1ce230$24b75ba5@aero.ad.tamu.edu> Message-ID: Julian, If you do not perform matrix reordering, you can use petsc sbaij format, which only stores the upper triangular entries. aij format has more algorithmic options and is more efficient and supports matrix reordering. Hong On Wed, 11 Oct 2006, Julian wrote: > I'm using '-ksp_type cg' now to solve the symmetric matric...But then do I > need to store all the non-zero elements of a symmetric matrix ? i.e. A[i,j] > as well as A[j,i] ? > > Shouldn't I be able to store just the upper or lower triangle or the matrix > ? Is that possible with petsc ? > > Julian. > > > > -----Original Message----- > > From: owner-petsc-users at mcs.anl.gov > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Julian > > Sent: Wednesday, October 11, 2006 12:37 PM > > To: petsc-users at mcs.anl.gov > > Subject: RE: General matrix interative solver > > > > Barry, > > > > I tried sending the commands from a file for now... And once > > I used '-ksp_type cg', I got pretty much the same performance > > as that of the inhouse code ! > > Now, I'll try the performance with larger cases... > > > > Thanks for your help! > > > > Julian. > > > > > > > -----Original Message----- > > > From: owner-petsc-users at mcs.anl.gov > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry Smith > > > Sent: Wednesday, October 11, 2006 11:25 AM > > > To: petsc-users at mcs.anl.gov > > > Subject: RE: General matrix interative solver > > > > > > > > > PetscOptionsSetValue() but recommend putting them on the command > > > line or in a file (pass the name of the file into > > PetscInitialize()). > > > Having to recompile for every option is too painful. > > > > > > Barry > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > Hong, > > > > > > > > The times are for the solution process alone and the > > > initial guess is > > > > the same, i.e. zero. > > > > > > > > As for the algorithm, they are probably different. > > > > Can you tell me how to pass 'runtime options' like -help > > > and -ksp_view > > > > from within the code... I mean, at compile time. > > > > > > > > thanks > > > > > > > > > > > > > > Julian, > > > > > > > > > > When comparing the inhouse solver and petsc solver, you > > need make > > > > > sure > > > > > > > > > > 1. Timings are collected for solution process. The matrix > > > and vector > > > > > assembly should be excluded. > > > > > 2. They should use same iterative algorithm. By default, > > > petsc uses > > > > > gmres with restart=30 and ilu(0) preconditioner. > > Petsc supports > > > > > symmetric matrices, e.g., runtime option '-ksp_type cg > > > -pc_type > > > > > icc' > > > > > might give better performance > > > > > 3. They should start from same intial guess. By default, petsc > > > > > initial guess is zero. > > > > > > > > > > You can use '-ksp_view' to see what algorithm and options > > > are used > > > > > in petsc. > > > > > > > > > > Hong > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > Hello, > > > > > > > > > > > > I implemented the iterative sparse matrix solver in PetSc > > > > > into my FEM > > > > > > code recently. I compared the results from a problem > > with 1317 > > > > > > unknowns. I used a direct solver to obtain the reference > > > > > solution. I > > > > > > have another in-house sparse iterative solver that I have > > > > > been using > > > > > > so far. It was written by someone else but I have access to > > > > > the source for that solver. > > > > > > I find the 'error norm' in the solution by taking the > > > > > square root of > > > > > > the sum of the squares of the absolute differences > > between the > > > > > > solution from the direct solver and the iterative > > solver. I am > > > > > > ignoring the numerical zeros in the solutions when doing this. > > > > > > I find that in order to get same order of the error > > > norm (1e-13) > > > > > > as the in-house iterative solver, the petsc solver > > takes a much > > > > > > longer time and larger number of iterations. While > > the inhouse > > > > > > solver took less than one second, the petsc solver took 13 > > > > > > seconds. The inhouse solver took 476 iterations whereas > > > the petsc > > > > > > solver took > > > > > 4738 iterations. > > > > > > I'm guessing this has to do with different setting of > > > the solver > > > > > > in petsc such as the preconditioner etc. > > > > > > Can you tell me what the different settings are? And how to > > > > > tweak them > > > > > > so that I can atleast get as good as a performance as the > > > > > inhouse code ? > > > > > > Given below is how I have implemented the petsc solver: > > > > > > > > > > > > /////initialization > > > > > > PetscInitializeNoArguments(); > > > > > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > > > > > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, > > > > > > PETSC_DEFAULT, PETSC_NULL, (Mat*)mat); > > > > > > > > > > > > ////// this is the function I use to populate the matrix > > > > > > MatSetValue(*(Mat*)mat, ii, jj, value, ADD_VALUES); > > > > > > > > > > > > ////// this is how I actaully solve the matrix > > > > > > MatAssemblyBegin(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > > > MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > > > > > > > > > double iter_error = 1e-10; > > > > > > int max_iter_num = 10000; > > > > > > int num_of_iter; > > > > > > > > > > > > Vec rhs, x; > > > > > > VecCreateSeqWithArray(PETSC_COMM_SELF, L, b, &rhs); > > > > > > VecDuplicate(rhs, &x); > > > > > > > > > > > > KSP ksp; > > > > > > KSPCreate(PETSC_COMM_SELF, &ksp); > > > > > > KSPSetTolerances(ksp, iter_error, PETSC_DEFAULT, > > > > > > PETSC_DEFAULT, max_iter_num); > > > > > > KSPSetFromOptions(ksp); > > > > > > KSPSetOperators(ksp, *(Mat*)mat, *(Mat*)mat, > > > > > > SAME_PRECONDITIONER); > > > > > > KSPSolve(ksp,rhs,x); > > > > > > > > > > > > PetscReal r_norm; > > > > > > KSPGetResidualNorm(ksp, &r_norm); > > > > > > KSPGetIterationNumber(ksp, &num_of_iter); > > > > > > > > > > > > cout << "max_iter_num\t" << max_iter_num << endl; > > > > > > cout << "iter_error\t" << iter_error << endl; > > > > > > > > > > > > cout << "Matrix solver step " << num_of_iter << ", > > > > > residual " > > > > > > << r_norm << ".\n"; > > > > > > > > > > > > PetscScalar *p; > > > > > > VecGetArray(x, &p); > > > > > > for(int i=0; i > > > > > b[i] = p[i]; > > > > > > } > > > > > > VecRestoreArray(x, &p); > > > > > > > > > > > > KSPDestroy(ksp); > > > > > > VecDestroy(rhs); > > > > > > VecDestroy(x); > > > > > > > > > > > > cout <<"Iterations for convergence="<< num_of_iter << " - > > > > > Residual Norm = " > > > > > > << r_norm << endl; > > > > > > > > > > > > > > > > > > > > > > > > If this is not the typical method to be used to solve > > > this kind of > > > > > > problem, please let me know what functions I should use. > > > > > > I should mention that the inhouse code is for symmetric > > > > > matrices and > > > > > > from what I understand, the petsc solver works for general > > > > > unsymmetric matrices. > > > > > > But I think for iterative solvers, it should still give > > > around the > > > > > > same performance. > > > > > > I tested the solvers against some other problems as > > well, and I > > > > > > got the same performance.. In some cases, no matter how many > > > > > iterations it > > > > > > goes through, the petsc solver would not go below a certain > > > > > error norm > > > > > > whereas the inhouse solver would get almost exactly the > > > > > same answer as > > > > > > the direct solver solution. I'm thinking the petsc > > > solver should > > > > > > be able to solve this problem just as easily. It would be > > > > > great if anyone > > > > > > could help me figure out the appropriate settings I should > > > > > use in the petsc solver. > > > > > > > > > > > > Thanks, > > > > > > Julian. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From julvar at tamu.edu Wed Oct 11 14:54:54 2006 From: julvar at tamu.edu (Julian) Date: Wed, 11 Oct 2006 14:54:54 -0500 Subject: General matrix interative solver In-Reply-To: Message-ID: <006001c6ed6f$1f59f620$24b75ba5@aero.ad.tamu.edu> Hong, I made the following change to specify it to be in sbaij format... And then proceeded to store only the upper triangle. Assert( mat = (double*)malloc(sizeof(Mat)) ); MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, PETSC_DEFAULT, PETSC_NULL, (Mat*)mat); MatSetType(*(Mat*)mat,MATSBAIJ); But I get the following error when I try to solve: ------------------------------------------------------------------------ Petsc Release Version 2.3.1, Patch 16, Fri Jul 21 15:09:15 CDT 2006 HG revision: 5bc424fae3770001f0d316695a9956fde3bc58b0 See docs/changes/index.html for recent updates. See docs/faq.html for hints about trouble shooting. See docs/index.html for manual pages. ------------------------------------------------------------------------ Unknown Name on a cygwin-cx named ALPHA2 by j0v1008 Wed Oct 11 14:51:00 2006 Libraries linked from /cygdrive/e/cygwin/home/petsc-2.3.1-p16/lib/cygwin-cxx-rea l-debug Configure run at Mon Aug 14 15:35:57 2006 Configure options --with-cc="win32fe cl --nodetect" --with-cxx="win32fe cl --nod etect" --COPTFLAGS="-MDd -Z7" --with-fc=0 --with-clanguage=cxx --download-c-blas -lapack=1 --with-mpi=0 --useThreads=0 --with-shared=0 ------------------------------------------------------------------------ [0]PETSC ERROR: MatILUFactorSymbolic() line 4573 in src/mat/interface/e:\cygwin\ home\PETSC-~1.1-P\src\mat\INTERF~1\matrix.c [0]PETSC ERROR: No support for this operation for this object type! [0]PETSC ERROR: Matrix type sbaij symbolic ILU! [0]PETSC ERROR: PCSetUp_ILU() line 532 in src/ksp/pc/impls/factor/ilu/e:\cygwin\ home\PETSC-~1.1-P\src\ksp\pc\impls\factor\ilu\ilu.c [0]PETSC ERROR: PCSetUp() line 798 in src/ksp/pc/interface/e:\cygwin\home\PETSC- ~1.1-P\src\ksp\pc\INTERF~1\precon.c [0]PETSC ERROR: KSPSetUp() line 234 in src/ksp/ksp/interface/e:\cygwin\home\PETS C-~1.1-P\src\ksp\ksp\INTERF~1\itfunc.c [0]PETSC ERROR: KSPSolve() line 334 in src/ksp/ksp/interface/e:\cygwin\home\PETS C-~1.1-P\src\ksp\ksp\INTERF~1\itfunc.c > -----Original Message----- > From: owner-petsc-users at mcs.anl.gov > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Hong Zhang > Sent: Wednesday, October 11, 2006 2:14 PM > To: petsc-users at mcs.anl.gov > Subject: RE: General matrix interative solver > > > Julian, > > If you do not perform matrix reordering, you can use petsc > sbaij format, which only stores the upper triangular entries. > aij format has more algorithmic options and is more efficient > and supports matrix reordering. > > Hong > > On Wed, 11 Oct 2006, Julian wrote: > > > I'm using '-ksp_type cg' now to solve the symmetric > matric...But then > > do I need to store all the non-zero elements of a symmetric > matrix ? > > i.e. A[i,j] as well as A[j,i] ? > > > > Shouldn't I be able to store just the upper or lower > triangle or the > > matrix ? Is that possible with petsc ? > > > > Julian. > > > > > > > -----Original Message----- > > > From: owner-petsc-users at mcs.anl.gov > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Julian > > > Sent: Wednesday, October 11, 2006 12:37 PM > > > To: petsc-users at mcs.anl.gov > > > Subject: RE: General matrix interative solver > > > > > > Barry, > > > > > > I tried sending the commands from a file for now... And > once I used > > > '-ksp_type cg', I got pretty much the same performance as that of > > > the inhouse code ! > > > Now, I'll try the performance with larger cases... > > > > > > Thanks for your help! > > > > > > Julian. > > > > > > > > > > -----Original Message----- > > > > From: owner-petsc-users at mcs.anl.gov > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry Smith > > > > Sent: Wednesday, October 11, 2006 11:25 AM > > > > To: petsc-users at mcs.anl.gov > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > PetscOptionsSetValue() but recommend putting them on > the command > > > > line or in a file (pass the name of the file into > > > PetscInitialize()). > > > > Having to recompile for every option is too painful. > > > > > > > > Barry > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > Hong, > > > > > > > > > > The times are for the solution process alone and the > > > > initial guess is > > > > > the same, i.e. zero. > > > > > > > > > > As for the algorithm, they are probably different. > > > > > Can you tell me how to pass 'runtime options' like -help > > > > and -ksp_view > > > > > from within the code... I mean, at compile time. > > > > > > > > > > thanks > > > > > > > > > > > > > > > > > Julian, > > > > > > > > > > > > When comparing the inhouse solver and petsc solver, you > > > need make > > > > > > sure > > > > > > > > > > > > 1. Timings are collected for solution process. The matrix > > > > and vector > > > > > > assembly should be excluded. > > > > > > 2. They should use same iterative algorithm. By default, > > > > petsc uses > > > > > > gmres with restart=30 and ilu(0) preconditioner. > > > Petsc supports > > > > > > symmetric matrices, e.g., runtime option '-ksp_type cg > > > > -pc_type > > > > > > icc' > > > > > > might give better performance 3. They should start from > > > > > > same intial guess. By default, petsc > > > > > > initial guess is zero. > > > > > > > > > > > > You can use '-ksp_view' to see what algorithm and options > > > > are used > > > > > > in petsc. > > > > > > > > > > > > Hong > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > I implemented the iterative sparse matrix solver in PetSc > > > > > > into my FEM > > > > > > > code recently. I compared the results from a problem > > > with 1317 > > > > > > > unknowns. I used a direct solver to obtain the reference > > > > > > solution. I > > > > > > > have another in-house sparse iterative solver that I have > > > > > > been using > > > > > > > so far. It was written by someone else but I have > access to > > > > > > the source for that solver. > > > > > > > I find the 'error norm' in the solution by taking the > > > > > > square root of > > > > > > > the sum of the squares of the absolute differences > > > between the > > > > > > > solution from the direct solver and the iterative > > > solver. I am > > > > > > > ignoring the numerical zeros in the solutions > when doing this. > > > > > > > I find that in order to get same order of the error > > > > norm (1e-13) > > > > > > > as the in-house iterative solver, the petsc solver > > > takes a much > > > > > > > longer time and larger number of iterations. While > > > the inhouse > > > > > > > solver took less than one second, the petsc > solver took 13 > > > > > > > seconds. The inhouse solver took 476 iterations whereas > > > > the petsc > > > > > > > solver took > > > > > > 4738 iterations. > > > > > > > I'm guessing this has to do with different setting of > > > > the solver > > > > > > > in petsc such as the preconditioner etc. > > > > > > > Can you tell me what the different settings are? > And how to > > > > > > tweak them > > > > > > > so that I can atleast get as good as a performance as the > > > > > > inhouse code ? > > > > > > > Given below is how I have implemented the petsc solver: > > > > > > > > > > > > > > /////initialization > > > > > > > PetscInitializeNoArguments(); > > > > > > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > > > > > > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, > > > > > > > PETSC_DEFAULT, PETSC_NULL, (Mat*)mat); > > > > > > > > > > > > > > ////// this is the function I use to populate the matrix > > > > > > > MatSetValue(*(Mat*)mat, ii, jj, value, ADD_VALUES); > > > > > > > > > > > > > > ////// this is how I actaully solve the matrix > > > > > > > MatAssemblyBegin(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > > > > MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > > > > > > > > > > > double iter_error = 1e-10; > > > > > > > int max_iter_num = 10000; > > > > > > > int num_of_iter; > > > > > > > > > > > > > > Vec rhs, x; > > > > > > > VecCreateSeqWithArray(PETSC_COMM_SELF, L, b, &rhs); > > > > > > > VecDuplicate(rhs, &x); > > > > > > > > > > > > > > KSP ksp; > > > > > > > KSPCreate(PETSC_COMM_SELF, &ksp); > > > > > > > KSPSetTolerances(ksp, iter_error, PETSC_DEFAULT, > > > > > > > PETSC_DEFAULT, max_iter_num); > > > > > > > KSPSetFromOptions(ksp); > > > > > > > KSPSetOperators(ksp, *(Mat*)mat, *(Mat*)mat, > > > > > > > SAME_PRECONDITIONER); > > > > > > > KSPSolve(ksp,rhs,x); > > > > > > > > > > > > > > PetscReal r_norm; > > > > > > > KSPGetResidualNorm(ksp, &r_norm); > > > > > > > KSPGetIterationNumber(ksp, &num_of_iter); > > > > > > > > > > > > > > cout << "max_iter_num\t" << max_iter_num << endl; > > > > > > > cout << "iter_error\t" << iter_error << endl; > > > > > > > > > > > > > > cout << "Matrix solver step " << num_of_iter << ", > > > > > > residual " > > > > > > > << r_norm << ".\n"; > > > > > > > > > > > > > > PetscScalar *p; > > > > > > > VecGetArray(x, &p); > > > > > > > for(int i=0; i > > > > > > b[i] = p[i]; > > > > > > > } > > > > > > > VecRestoreArray(x, &p); > > > > > > > > > > > > > > KSPDestroy(ksp); > > > > > > > VecDestroy(rhs); > > > > > > > VecDestroy(x); > > > > > > > > > > > > > > cout <<"Iterations for convergence="<< num_of_iter << " - > > > > > > Residual Norm = " > > > > > > > << r_norm << endl; > > > > > > > > > > > > > > > > > > > > > > > > > > > > If this is not the typical method to be used to solve > > > > this kind of > > > > > > > problem, please let me know what functions I should use. > > > > > > > I should mention that the inhouse code is for symmetric > > > > > > matrices and > > > > > > > from what I understand, the petsc solver works for general > > > > > > unsymmetric matrices. > > > > > > > But I think for iterative solvers, it should still give > > > > around the > > > > > > > same performance. > > > > > > > I tested the solvers against some other problems as > > > well, and I > > > > > > > got the same performance.. In some cases, no > matter how many > > > > > > iterations it > > > > > > > goes through, the petsc solver would not go below > a certain > > > > > > error norm > > > > > > > whereas the inhouse solver would get almost exactly the > > > > > > same answer as > > > > > > > the direct solver solution. I'm thinking the petsc > > > > solver should > > > > > > > be able to solve this problem just as easily. It would be > > > > > > great if anyone > > > > > > > could help me figure out the appropriate settings I should > > > > > > use in the petsc solver. > > > > > > > > > > > > > > Thanks, > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From bsmith at mcs.anl.gov Wed Oct 11 14:58:22 2006 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 11 Oct 2006 14:58:22 -0500 (CDT) Subject: General matrix interative solver In-Reply-To: <006001c6ed6f$1f59f620$24b75ba5@aero.ad.tamu.edu> References: <006001c6ed6f$1f59f620$24b75ba5@aero.ad.tamu.edu> Message-ID: ILU is for general matrices; for symmetric you need to use PCSetType(pc,PCICC); or -pc_type icc Barry On Wed, 11 Oct 2006, Julian wrote: > Hong, > I made the following change to specify it to be in sbaij format... And then > proceeded to store only the upper triangle. > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, PETSC_DEFAULT, PETSC_NULL, > (Mat*)mat); > MatSetType(*(Mat*)mat,MATSBAIJ); > > But I get the following error when I try to solve: > > ------------------------------------------------------------------------ > Petsc Release Version 2.3.1, Patch 16, Fri Jul 21 15:09:15 CDT 2006 HG > revision: > 5bc424fae3770001f0d316695a9956fde3bc58b0 > See docs/changes/index.html for recent updates. > See docs/faq.html for hints about trouble shooting. > See docs/index.html for manual pages. > ------------------------------------------------------------------------ > Unknown Name on a cygwin-cx named ALPHA2 by j0v1008 Wed Oct 11 14:51:00 2006 > Libraries linked from > /cygdrive/e/cygwin/home/petsc-2.3.1-p16/lib/cygwin-cxx-rea > l-debug > Configure run at Mon Aug 14 15:35:57 2006 > Configure options --with-cc="win32fe cl --nodetect" --with-cxx="win32fe cl > --nod > etect" --COPTFLAGS="-MDd -Z7" --with-fc=0 --with-clanguage=cxx > --download-c-blas > -lapack=1 --with-mpi=0 --useThreads=0 --with-shared=0 > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatILUFactorSymbolic() line 4573 in > src/mat/interface/e:\cygwin\ > home\PETSC-~1.1-P\src\mat\INTERF~1\matrix.c > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: Matrix type sbaij symbolic ILU! > [0]PETSC ERROR: PCSetUp_ILU() line 532 in > src/ksp/pc/impls/factor/ilu/e:\cygwin\ > home\PETSC-~1.1-P\src\ksp\pc\impls\factor\ilu\ilu.c > [0]PETSC ERROR: PCSetUp() line 798 in > src/ksp/pc/interface/e:\cygwin\home\PETSC- > ~1.1-P\src\ksp\pc\INTERF~1\precon.c > [0]PETSC ERROR: KSPSetUp() line 234 in > src/ksp/ksp/interface/e:\cygwin\home\PETS > C-~1.1-P\src\ksp\ksp\INTERF~1\itfunc.c > [0]PETSC ERROR: KSPSolve() line 334 in > src/ksp/ksp/interface/e:\cygwin\home\PETS > C-~1.1-P\src\ksp\ksp\INTERF~1\itfunc.c > > > -----Original Message----- > > From: owner-petsc-users at mcs.anl.gov > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Hong Zhang > > Sent: Wednesday, October 11, 2006 2:14 PM > > To: petsc-users at mcs.anl.gov > > Subject: RE: General matrix interative solver > > > > > > Julian, > > > > If you do not perform matrix reordering, you can use petsc > > sbaij format, which only stores the upper triangular entries. > > aij format has more algorithmic options and is more efficient > > and supports matrix reordering. > > > > Hong > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > I'm using '-ksp_type cg' now to solve the symmetric > > matric...But then > > > do I need to store all the non-zero elements of a symmetric > > matrix ? > > > i.e. A[i,j] as well as A[j,i] ? > > > > > > Shouldn't I be able to store just the upper or lower > > triangle or the > > > matrix ? Is that possible with petsc ? > > > > > > Julian. > > > > > > > > > > -----Original Message----- > > > > From: owner-petsc-users at mcs.anl.gov > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Julian > > > > Sent: Wednesday, October 11, 2006 12:37 PM > > > > To: petsc-users at mcs.anl.gov > > > > Subject: RE: General matrix interative solver > > > > > > > > Barry, > > > > > > > > I tried sending the commands from a file for now... And > > once I used > > > > '-ksp_type cg', I got pretty much the same performance as that of > > > > the inhouse code ! > > > > Now, I'll try the performance with larger cases... > > > > > > > > Thanks for your help! > > > > > > > > Julian. > > > > > > > > > > > > > -----Original Message----- > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry Smith > > > > > Sent: Wednesday, October 11, 2006 11:25 AM > > > > > To: petsc-users at mcs.anl.gov > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > > PetscOptionsSetValue() but recommend putting them on > > the command > > > > > line or in a file (pass the name of the file into > > > > PetscInitialize()). > > > > > Having to recompile for every option is too painful. > > > > > > > > > > Barry > > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > Hong, > > > > > > > > > > > > The times are for the solution process alone and the > > > > > initial guess is > > > > > > the same, i.e. zero. > > > > > > > > > > > > As for the algorithm, they are probably different. > > > > > > Can you tell me how to pass 'runtime options' like -help > > > > > and -ksp_view > > > > > > from within the code... I mean, at compile time. > > > > > > > > > > > > thanks > > > > > > > > > > > > > > > > > > > > Julian, > > > > > > > > > > > > > > When comparing the inhouse solver and petsc solver, you > > > > need make > > > > > > > sure > > > > > > > > > > > > > > 1. Timings are collected for solution process. The matrix > > > > > and vector > > > > > > > assembly should be excluded. > > > > > > > 2. They should use same iterative algorithm. By default, > > > > > petsc uses > > > > > > > gmres with restart=30 and ilu(0) preconditioner. > > > > Petsc supports > > > > > > > symmetric matrices, e.g., runtime option '-ksp_type cg > > > > > -pc_type > > > > > > > icc' > > > > > > > might give better performance 3. They should start from > > > > > > > same intial guess. By default, petsc > > > > > > > initial guess is zero. > > > > > > > > > > > > > > You can use '-ksp_view' to see what algorithm and options > > > > > are used > > > > > > > in petsc. > > > > > > > > > > > > > > Hong > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > I implemented the iterative sparse matrix solver in PetSc > > > > > > > into my FEM > > > > > > > > code recently. I compared the results from a problem > > > > with 1317 > > > > > > > > unknowns. I used a direct solver to obtain the reference > > > > > > > solution. I > > > > > > > > have another in-house sparse iterative solver that I have > > > > > > > been using > > > > > > > > so far. It was written by someone else but I have > > access to > > > > > > > the source for that solver. > > > > > > > > I find the 'error norm' in the solution by taking the > > > > > > > square root of > > > > > > > > the sum of the squares of the absolute differences > > > > between the > > > > > > > > solution from the direct solver and the iterative > > > > solver. I am > > > > > > > > ignoring the numerical zeros in the solutions > > when doing this. > > > > > > > > I find that in order to get same order of the error > > > > > norm (1e-13) > > > > > > > > as the in-house iterative solver, the petsc solver > > > > takes a much > > > > > > > > longer time and larger number of iterations. While > > > > the inhouse > > > > > > > > solver took less than one second, the petsc > > solver took 13 > > > > > > > > seconds. The inhouse solver took 476 iterations whereas > > > > > the petsc > > > > > > > > solver took > > > > > > > 4738 iterations. > > > > > > > > I'm guessing this has to do with different setting of > > > > > the solver > > > > > > > > in petsc such as the preconditioner etc. > > > > > > > > Can you tell me what the different settings are? > > And how to > > > > > > > tweak them > > > > > > > > so that I can atleast get as good as a performance as the > > > > > > > inhouse code ? > > > > > > > > Given below is how I have implemented the petsc solver: > > > > > > > > > > > > > > > > /////initialization > > > > > > > > PetscInitializeNoArguments(); > > > > > > > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > > > > > > > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, > > > > > > > > PETSC_DEFAULT, PETSC_NULL, (Mat*)mat); > > > > > > > > > > > > > > > > ////// this is the function I use to populate the matrix > > > > > > > > MatSetValue(*(Mat*)mat, ii, jj, value, ADD_VALUES); > > > > > > > > > > > > > > > > ////// this is how I actaully solve the matrix > > > > > > > > MatAssemblyBegin(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > > > > > MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > > > > > > > > > > > > > double iter_error = 1e-10; > > > > > > > > int max_iter_num = 10000; > > > > > > > > int num_of_iter; > > > > > > > > > > > > > > > > Vec rhs, x; > > > > > > > > VecCreateSeqWithArray(PETSC_COMM_SELF, L, b, &rhs); > > > > > > > > VecDuplicate(rhs, &x); > > > > > > > > > > > > > > > > KSP ksp; > > > > > > > > KSPCreate(PETSC_COMM_SELF, &ksp); > > > > > > > > KSPSetTolerances(ksp, iter_error, PETSC_DEFAULT, > > > > > > > > PETSC_DEFAULT, max_iter_num); > > > > > > > > KSPSetFromOptions(ksp); > > > > > > > > KSPSetOperators(ksp, *(Mat*)mat, *(Mat*)mat, > > > > > > > > SAME_PRECONDITIONER); > > > > > > > > KSPSolve(ksp,rhs,x); > > > > > > > > > > > > > > > > PetscReal r_norm; > > > > > > > > KSPGetResidualNorm(ksp, &r_norm); > > > > > > > > KSPGetIterationNumber(ksp, &num_of_iter); > > > > > > > > > > > > > > > > cout << "max_iter_num\t" << max_iter_num << endl; > > > > > > > > cout << "iter_error\t" << iter_error << endl; > > > > > > > > > > > > > > > > cout << "Matrix solver step " << num_of_iter << ", > > > > > > > residual " > > > > > > > > << r_norm << ".\n"; > > > > > > > > > > > > > > > > PetscScalar *p; > > > > > > > > VecGetArray(x, &p); > > > > > > > > for(int i=0; i > > > > > > > b[i] = p[i]; > > > > > > > > } > > > > > > > > VecRestoreArray(x, &p); > > > > > > > > > > > > > > > > KSPDestroy(ksp); > > > > > > > > VecDestroy(rhs); > > > > > > > > VecDestroy(x); > > > > > > > > > > > > > > > > cout <<"Iterations for convergence="<< num_of_iter << " - > > > > > > > Residual Norm = " > > > > > > > > << r_norm << endl; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If this is not the typical method to be used to solve > > > > > this kind of > > > > > > > > problem, please let me know what functions I should use. > > > > > > > > I should mention that the inhouse code is for symmetric > > > > > > > matrices and > > > > > > > > from what I understand, the petsc solver works for general > > > > > > > unsymmetric matrices. > > > > > > > > But I think for iterative solvers, it should still give > > > > > around the > > > > > > > > same performance. > > > > > > > > I tested the solvers against some other problems as > > > > well, and I > > > > > > > > got the same performance.. In some cases, no > > matter how many > > > > > > > iterations it > > > > > > > > goes through, the petsc solver would not go below > > a certain > > > > > > > error norm > > > > > > > > whereas the inhouse solver would get almost exactly the > > > > > > > same answer as > > > > > > > > the direct solver solution. I'm thinking the petsc > > > > > solver should > > > > > > > > be able to solve this problem just as easily. It would be > > > > > > > great if anyone > > > > > > > > could help me figure out the appropriate settings I should > > > > > > > use in the petsc solver. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From hzhang at mcs.anl.gov Thu Oct 12 09:02:34 2006 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Thu, 12 Oct 2006 09:02:34 -0500 (CDT) Subject: General matrix interative solver In-Reply-To: <006001c6ed6f$1f59f620$24b75ba5@aero.ad.tamu.edu> References: <006001c6ed6f$1f59f620$24b75ba5@aero.ad.tamu.edu> Message-ID: > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatILUFactorSymbolic() line 4573 in > src/mat/interface/e:\cygwin\ > home\PETSC-~1.1-P\src\mat\INTERF~1\matrix.c > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: Matrix type sbaij symbolic ILU! Using sbaij format, you must use preconditioner icc, not ilu. Hong > [0]PETSC ERROR: PCSetUp_ILU() line 532 in > src/ksp/pc/impls/factor/ilu/e:\cygwin\ > home\PETSC-~1.1-P\src\ksp\pc\impls\factor\ilu\ilu.c > [0]PETSC ERROR: PCSetUp() line 798 in > src/ksp/pc/interface/e:\cygwin\home\PETSC- > ~1.1-P\src\ksp\pc\INTERF~1\precon.c > [0]PETSC ERROR: KSPSetUp() line 234 in > src/ksp/ksp/interface/e:\cygwin\home\PETS > C-~1.1-P\src\ksp\ksp\INTERF~1\itfunc.c > [0]PETSC ERROR: KSPSolve() line 334 in > src/ksp/ksp/interface/e:\cygwin\home\PETS > C-~1.1-P\src\ksp\ksp\INTERF~1\itfunc.c > > > -----Original Message----- > > From: owner-petsc-users at mcs.anl.gov > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Hong Zhang > > Sent: Wednesday, October 11, 2006 2:14 PM > > To: petsc-users at mcs.anl.gov > > Subject: RE: General matrix interative solver > > > > > > Julian, > > > > If you do not perform matrix reordering, you can use petsc > > sbaij format, which only stores the upper triangular entries. > > aij format has more algorithmic options and is more efficient > > and supports matrix reordering. > > > > Hong > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > I'm using '-ksp_type cg' now to solve the symmetric > > matric...But then > > > do I need to store all the non-zero elements of a symmetric > > matrix ? > > > i.e. A[i,j] as well as A[j,i] ? > > > > > > Shouldn't I be able to store just the upper or lower > > triangle or the > > > matrix ? Is that possible with petsc ? > > > > > > Julian. > > > > > > > > > > -----Original Message----- > > > > From: owner-petsc-users at mcs.anl.gov > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Julian > > > > Sent: Wednesday, October 11, 2006 12:37 PM > > > > To: petsc-users at mcs.anl.gov > > > > Subject: RE: General matrix interative solver > > > > > > > > Barry, > > > > > > > > I tried sending the commands from a file for now... And > > once I used > > > > '-ksp_type cg', I got pretty much the same performance as that of > > > > the inhouse code ! > > > > Now, I'll try the performance with larger cases... > > > > > > > > Thanks for your help! > > > > > > > > Julian. > > > > > > > > > > > > > -----Original Message----- > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry Smith > > > > > Sent: Wednesday, October 11, 2006 11:25 AM > > > > > To: petsc-users at mcs.anl.gov > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > > PetscOptionsSetValue() but recommend putting them on > > the command > > > > > line or in a file (pass the name of the file into > > > > PetscInitialize()). > > > > > Having to recompile for every option is too painful. > > > > > > > > > > Barry > > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > Hong, > > > > > > > > > > > > The times are for the solution process alone and the > > > > > initial guess is > > > > > > the same, i.e. zero. > > > > > > > > > > > > As for the algorithm, they are probably different. > > > > > > Can you tell me how to pass 'runtime options' like -help > > > > > and -ksp_view > > > > > > from within the code... I mean, at compile time. > > > > > > > > > > > > thanks > > > > > > > > > > > > > > > > > > > > Julian, > > > > > > > > > > > > > > When comparing the inhouse solver and petsc solver, you > > > > need make > > > > > > > sure > > > > > > > > > > > > > > 1. Timings are collected for solution process. The matrix > > > > > and vector > > > > > > > assembly should be excluded. > > > > > > > 2. They should use same iterative algorithm. By default, > > > > > petsc uses > > > > > > > gmres with restart=30 and ilu(0) preconditioner. > > > > Petsc supports > > > > > > > symmetric matrices, e.g., runtime option '-ksp_type cg > > > > > -pc_type > > > > > > > icc' > > > > > > > might give better performance 3. They should start from > > > > > > > same intial guess. By default, petsc > > > > > > > initial guess is zero. > > > > > > > > > > > > > > You can use '-ksp_view' to see what algorithm and options > > > > > are used > > > > > > > in petsc. > > > > > > > > > > > > > > Hong > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > I implemented the iterative sparse matrix solver in PetSc > > > > > > > into my FEM > > > > > > > > code recently. I compared the results from a problem > > > > with 1317 > > > > > > > > unknowns. I used a direct solver to obtain the reference > > > > > > > solution. I > > > > > > > > have another in-house sparse iterative solver that I have > > > > > > > been using > > > > > > > > so far. It was written by someone else but I have > > access to > > > > > > > the source for that solver. > > > > > > > > I find the 'error norm' in the solution by taking the > > > > > > > square root of > > > > > > > > the sum of the squares of the absolute differences > > > > between the > > > > > > > > solution from the direct solver and the iterative > > > > solver. I am > > > > > > > > ignoring the numerical zeros in the solutions > > when doing this. > > > > > > > > I find that in order to get same order of the error > > > > > norm (1e-13) > > > > > > > > as the in-house iterative solver, the petsc solver > > > > takes a much > > > > > > > > longer time and larger number of iterations. While > > > > the inhouse > > > > > > > > solver took less than one second, the petsc > > solver took 13 > > > > > > > > seconds. The inhouse solver took 476 iterations whereas > > > > > the petsc > > > > > > > > solver took > > > > > > > 4738 iterations. > > > > > > > > I'm guessing this has to do with different setting of > > > > > the solver > > > > > > > > in petsc such as the preconditioner etc. > > > > > > > > Can you tell me what the different settings are? > > And how to > > > > > > > tweak them > > > > > > > > so that I can atleast get as good as a performance as the > > > > > > > inhouse code ? > > > > > > > > Given below is how I have implemented the petsc solver: > > > > > > > > > > > > > > > > /////initialization > > > > > > > > PetscInitializeNoArguments(); > > > > > > > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > > > > > > > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, > > > > > > > > PETSC_DEFAULT, PETSC_NULL, (Mat*)mat); > > > > > > > > > > > > > > > > ////// this is the function I use to populate the matrix > > > > > > > > MatSetValue(*(Mat*)mat, ii, jj, value, ADD_VALUES); > > > > > > > > > > > > > > > > ////// this is how I actaully solve the matrix > > > > > > > > MatAssemblyBegin(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > > > > > MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > > > > > > > > > > > > > double iter_error = 1e-10; > > > > > > > > int max_iter_num = 10000; > > > > > > > > int num_of_iter; > > > > > > > > > > > > > > > > Vec rhs, x; > > > > > > > > VecCreateSeqWithArray(PETSC_COMM_SELF, L, b, &rhs); > > > > > > > > VecDuplicate(rhs, &x); > > > > > > > > > > > > > > > > KSP ksp; > > > > > > > > KSPCreate(PETSC_COMM_SELF, &ksp); > > > > > > > > KSPSetTolerances(ksp, iter_error, PETSC_DEFAULT, > > > > > > > > PETSC_DEFAULT, max_iter_num); > > > > > > > > KSPSetFromOptions(ksp); > > > > > > > > KSPSetOperators(ksp, *(Mat*)mat, *(Mat*)mat, > > > > > > > > SAME_PRECONDITIONER); > > > > > > > > KSPSolve(ksp,rhs,x); > > > > > > > > > > > > > > > > PetscReal r_norm; > > > > > > > > KSPGetResidualNorm(ksp, &r_norm); > > > > > > > > KSPGetIterationNumber(ksp, &num_of_iter); > > > > > > > > > > > > > > > > cout << "max_iter_num\t" << max_iter_num << endl; > > > > > > > > cout << "iter_error\t" << iter_error << endl; > > > > > > > > > > > > > > > > cout << "Matrix solver step " << num_of_iter << ", > > > > > > > residual " > > > > > > > > << r_norm << ".\n"; > > > > > > > > > > > > > > > > PetscScalar *p; > > > > > > > > VecGetArray(x, &p); > > > > > > > > for(int i=0; i > > > > > > > b[i] = p[i]; > > > > > > > > } > > > > > > > > VecRestoreArray(x, &p); > > > > > > > > > > > > > > > > KSPDestroy(ksp); > > > > > > > > VecDestroy(rhs); > > > > > > > > VecDestroy(x); > > > > > > > > > > > > > > > > cout <<"Iterations for convergence="<< num_of_iter << " - > > > > > > > Residual Norm = " > > > > > > > > << r_norm << endl; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If this is not the typical method to be used to solve > > > > > this kind of > > > > > > > > problem, please let me know what functions I should use. > > > > > > > > I should mention that the inhouse code is for symmetric > > > > > > > matrices and > > > > > > > > from what I understand, the petsc solver works for general > > > > > > > unsymmetric matrices. > > > > > > > > But I think for iterative solvers, it should still give > > > > > around the > > > > > > > > same performance. > > > > > > > > I tested the solvers against some other problems as > > > > well, and I > > > > > > > > got the same performance.. In some cases, no > > matter how many > > > > > > > iterations it > > > > > > > > goes through, the petsc solver would not go below > > a certain > > > > > > > error norm > > > > > > > > whereas the inhouse solver would get almost exactly the > > > > > > > same answer as > > > > > > > > the direct solver solution. I'm thinking the petsc > > > > > solver should > > > > > > > > be able to solve this problem just as easily. It would be > > > > > > > great if anyone > > > > > > > > could help me figure out the appropriate settings I should > > > > > > > use in the petsc solver. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From julvar at tamu.edu Thu Oct 12 13:46:32 2006 From: julvar at tamu.edu (Julian) Date: Thu, 12 Oct 2006 13:46:32 -0500 Subject: General matrix interative solver In-Reply-To: Message-ID: <00cf01c6ee2e$bc8b7440$24b75ba5@aero.ad.tamu.edu> Thanks, I got it working after I switched to icc All this time I was running test cases where I just read in an already assembled global stiffness matrix and load vector and solved the problem using petsc. After I got that working, I tried to fully integrate the solver with the FEA program. One problem I had was with populating the matrix. The interfaces to the other existing solvers in our code were written so that we can assign a value directly to a memory space assigned for an element in the matrix. Something like mat(I,J)=value; Where mat(I,J) gives the reference to the location. But petsc does not work that way (I had already asked this on the mailing list a couple of months back) and I understand maybe its wise to do that, considering it handles parallel implementation as well. So, I changed our code to work with petsc. Now, when assembling the global matrix, sometimes I need to use ADD_VALUES and other times INSERT_VALUES (depending on constraints) At first, I used MatAssemblyBegin(*(Mat*)mat, MAT_FINAL_ASSEMBLY); MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); But then it took too long and I realized I should be using MAT_FLUSH_ASSEMBLY. But even then it takes considerably more time in assembling than my other solvers. So, I used a flag to call the assembly only when I am switching between ADD_VALUES and INSERT_VALUES. That reduced the calls to MAT_FLUSH_ASSEMBLY but it still was a slower than the other solvers.. And this was for a small problem (1317 unknowns) I tried it out with a larger problem ~15000 unknowns and the program just got stuck during the assembly process ! It wouldn't crash but looks like its doing something inside of petsc... probably memory allocations. So, I think I need to start preallocating memory. I'm working on that now... So hopefully, after that the assembly process won't slow down bcause of memory allocation. But my question for now... is using the flag method to call assembly only when needed the most efficient way to do this? Or do you have any better ideas? One other possiblity is to have an array of flags, so as to avoid the call to INSERT_VALUES altogether... And thereby eliminating the need to call MAT_FLUSH_ASSEMBLY. I'll get back to you after I get the preallocation right. Maybe that will fix everything. Thanks, Julian. > -----Original Message----- > From: owner-petsc-users at mcs.anl.gov > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry Smith > Sent: Wednesday, October 11, 2006 2:58 PM > To: petsc-users at mcs.anl.gov > Subject: RE: General matrix interative solver > > > ILU is for general matrices; for symmetric you need to use > PCSetType(pc,PCICC); or -pc_type icc > > Barry > > > On Wed, 11 Oct 2006, Julian wrote: > > > Hong, > > I made the following change to specify it to be in sbaij > format... And > > then proceeded to store only the upper triangle. > > > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, PETSC_DEFAULT, > PETSC_NULL, > > (Mat*)mat); > > MatSetType(*(Mat*)mat,MATSBAIJ); > > > > But I get the following error when I try to solve: > > > > > ---------------------------------------------------------------------- > > -- Petsc Release Version 2.3.1, Patch 16, Fri Jul 21 > 15:09:15 CDT 2006 > > HG > > revision: > > 5bc424fae3770001f0d316695a9956fde3bc58b0 > > See docs/changes/index.html for recent updates. > > See docs/faq.html for hints about trouble shooting. > > See docs/index.html for manual pages. > > > ---------------------------------------------------------------------- > > -- Unknown Name on a cygwin-cx named ALPHA2 by j0v1008 Wed Oct 11 > > 14:51:00 2006 Libraries linked from > > /cygdrive/e/cygwin/home/petsc-2.3.1-p16/lib/cygwin-cxx-rea > > l-debug > > Configure run at Mon Aug 14 15:35:57 2006 Configure options > > --with-cc="win32fe cl --nodetect" --with-cxx="win32fe cl > --nod etect" > > --COPTFLAGS="-MDd -Z7" --with-fc=0 --with-clanguage=cxx > > --download-c-blas > > -lapack=1 --with-mpi=0 --useThreads=0 --with-shared=0 > > > ---------------------------------------------------------------------- > > -- [0]PETSC ERROR: MatILUFactorSymbolic() line 4573 in > > src/mat/interface/e:\cygwin\ > > home\PETSC-~1.1-P\src\mat\INTERF~1\matrix.c > > [0]PETSC ERROR: No support for this operation for this object type! > > [0]PETSC ERROR: Matrix type sbaij symbolic ILU! > > [0]PETSC ERROR: PCSetUp_ILU() line 532 in > > src/ksp/pc/impls/factor/ilu/e:\cygwin\ > > home\PETSC-~1.1-P\src\ksp\pc\impls\factor\ilu\ilu.c > > [0]PETSC ERROR: PCSetUp() line 798 in > > src/ksp/pc/interface/e:\cygwin\home\PETSC- > > ~1.1-P\src\ksp\pc\INTERF~1\precon.c > > [0]PETSC ERROR: KSPSetUp() line 234 in > > src/ksp/ksp/interface/e:\cygwin\home\PETS > > C-~1.1-P\src\ksp\ksp\INTERF~1\itfunc.c > > [0]PETSC ERROR: KSPSolve() line 334 in > > src/ksp/ksp/interface/e:\cygwin\home\PETS > > C-~1.1-P\src\ksp\ksp\INTERF~1\itfunc.c > > > > > -----Original Message----- > > > From: owner-petsc-users at mcs.anl.gov > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Hong Zhang > > > Sent: Wednesday, October 11, 2006 2:14 PM > > > To: petsc-users at mcs.anl.gov > > > Subject: RE: General matrix interative solver > > > > > > > > > Julian, > > > > > > If you do not perform matrix reordering, you can use petsc sbaij > > > format, which only stores the upper triangular entries. > > > aij format has more algorithmic options and is more efficient and > > > supports matrix reordering. > > > > > > Hong > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > I'm using '-ksp_type cg' now to solve the symmetric > > > matric...But then > > > > do I need to store all the non-zero elements of a symmetric > > > matrix ? > > > > i.e. A[i,j] as well as A[j,i] ? > > > > > > > > Shouldn't I be able to store just the upper or lower > > > triangle or the > > > > matrix ? Is that possible with petsc ? > > > > > > > > Julian. > > > > > > > > > > > > > -----Original Message----- > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Julian > > > > > Sent: Wednesday, October 11, 2006 12:37 PM > > > > > To: petsc-users at mcs.anl.gov > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > Barry, > > > > > > > > > > I tried sending the commands from a file for now... And > > > once I used > > > > > '-ksp_type cg', I got pretty much the same > performance as that > > > > > of the inhouse code ! > > > > > Now, I'll try the performance with larger cases... > > > > > > > > > > Thanks for your help! > > > > > > > > > > Julian. > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry > > > > > > Smith > > > > > > Sent: Wednesday, October 11, 2006 11:25 AM > > > > > > To: petsc-users at mcs.anl.gov > > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > > > > > PetscOptionsSetValue() but recommend putting them on > > > the command > > > > > > line or in a file (pass the name of the file into > > > > > PetscInitialize()). > > > > > > Having to recompile for every option is too painful. > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > Hong, > > > > > > > > > > > > > > The times are for the solution process alone and the > > > > > > initial guess is > > > > > > > the same, i.e. zero. > > > > > > > > > > > > > > As for the algorithm, they are probably different. > > > > > > > Can you tell me how to pass 'runtime options' like -help > > > > > > and -ksp_view > > > > > > > from within the code... I mean, at compile time. > > > > > > > > > > > > > > thanks > > > > > > > > > > > > > > > > > > > > > > > Julian, > > > > > > > > > > > > > > > > When comparing the inhouse solver and petsc solver, you > > > > > need make > > > > > > > > sure > > > > > > > > > > > > > > > > 1. Timings are collected for solution process. > The matrix > > > > > > and vector > > > > > > > > assembly should be excluded. > > > > > > > > 2. They should use same iterative algorithm. By default, > > > > > > petsc uses > > > > > > > > gmres with restart=30 and ilu(0) preconditioner. > > > > > Petsc supports > > > > > > > > symmetric matrices, e.g., runtime option > '-ksp_type cg > > > > > > -pc_type > > > > > > > > icc' > > > > > > > > might give better performance 3. They should > start from > > > > > > > > same intial guess. By default, petsc > > > > > > > > initial guess is zero. > > > > > > > > > > > > > > > > You can use '-ksp_view' to see what algorithm > and options > > > > > > are used > > > > > > > > in petsc. > > > > > > > > > > > > > > > > Hong > > > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > I implemented the iterative sparse matrix solver in > > > > > > > > > PetSc > > > > > > > > into my FEM > > > > > > > > > code recently. I compared the results from a problem > > > > > with 1317 > > > > > > > > > unknowns. I used a direct solver to obtain > the reference > > > > > > > > solution. I > > > > > > > > > have another in-house sparse iterative solver that I > > > > > > > > > have > > > > > > > > been using > > > > > > > > > so far. It was written by someone else but I have > > > access to > > > > > > > > the source for that solver. > > > > > > > > > I find the 'error norm' in the solution by taking the > > > > > > > > square root of > > > > > > > > > the sum of the squares of the absolute differences > > > > > between the > > > > > > > > > solution from the direct solver and the iterative > > > > > solver. I am > > > > > > > > > ignoring the numerical zeros in the solutions > > > when doing this. > > > > > > > > > I find that in order to get same order of the error > > > > > > norm (1e-13) > > > > > > > > > as the in-house iterative solver, the petsc solver > > > > > takes a much > > > > > > > > > longer time and larger number of iterations. While > > > > > the inhouse > > > > > > > > > solver took less than one second, the petsc > > > solver took 13 > > > > > > > > > seconds. The inhouse solver took 476 > iterations whereas > > > > > > the petsc > > > > > > > > > solver took > > > > > > > > 4738 iterations. > > > > > > > > > I'm guessing this has to do with different setting of > > > > > > the solver > > > > > > > > > in petsc such as the preconditioner etc. > > > > > > > > > Can you tell me what the different settings are? > > > And how to > > > > > > > > tweak them > > > > > > > > > so that I can atleast get as good as a performance as > > > > > > > > > the > > > > > > > > inhouse code ? > > > > > > > > > Given below is how I have implemented the > petsc solver: > > > > > > > > > > > > > > > > > > /////initialization > > > > > > > > > PetscInitializeNoArguments(); > > > > > > > > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > > > > > > > > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, > > > > > > > > > PETSC_DEFAULT, PETSC_NULL, (Mat*)mat); > > > > > > > > > > > > > > > > > > ////// this is the function I use to populate > the matrix > > > > > > > > > MatSetValue(*(Mat*)mat, ii, jj, value, ADD_VALUES); > > > > > > > > > > > > > > > > > > ////// this is how I actaully solve the matrix > > > > > > > > > MatAssemblyBegin(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > > > > > > MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > > > > > > > > > > > > > > > double iter_error = 1e-10; > > > > > > > > > int max_iter_num = 10000; > > > > > > > > > int num_of_iter; > > > > > > > > > > > > > > > > > > Vec rhs, x; > > > > > > > > > VecCreateSeqWithArray(PETSC_COMM_SELF, L, > b, &rhs); > > > > > > > > > VecDuplicate(rhs, &x); > > > > > > > > > > > > > > > > > > KSP ksp; > > > > > > > > > KSPCreate(PETSC_COMM_SELF, &ksp); > > > > > > > > > KSPSetTolerances(ksp, iter_error, PETSC_DEFAULT, > > > > > > > > > PETSC_DEFAULT, max_iter_num); > > > > > > > > > KSPSetFromOptions(ksp); > > > > > > > > > KSPSetOperators(ksp, *(Mat*)mat, *(Mat*)mat, > > > > > > > > > SAME_PRECONDITIONER); > > > > > > > > > KSPSolve(ksp,rhs,x); > > > > > > > > > > > > > > > > > > PetscReal r_norm; > > > > > > > > > KSPGetResidualNorm(ksp, &r_norm); > > > > > > > > > KSPGetIterationNumber(ksp, &num_of_iter); > > > > > > > > > > > > > > > > > > cout << "max_iter_num\t" << > max_iter_num << endl; > > > > > > > > > cout << "iter_error\t" << iter_error << endl; > > > > > > > > > > > > > > > > > > cout << "Matrix solver step " << > num_of_iter << > > > > > > > > > ", > > > > > > > > residual " > > > > > > > > > << r_norm << ".\n"; > > > > > > > > > > > > > > > > > > PetscScalar *p; > > > > > > > > > VecGetArray(x, &p); > > > > > > > > > for(int i=0; i > > > > > > > > b[i] = p[i]; > > > > > > > > > } > > > > > > > > > VecRestoreArray(x, &p); > > > > > > > > > > > > > > > > > > KSPDestroy(ksp); > > > > > > > > > VecDestroy(rhs); > > > > > > > > > VecDestroy(x); > > > > > > > > > > > > > > > > > > cout <<"Iterations for convergence="<< > num_of_iter << " > > > > > > > > > - > > > > > > > > Residual Norm = " > > > > > > > > > << r_norm << endl; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If this is not the typical method to be used to solve > > > > > > this kind of > > > > > > > > > problem, please let me know what functions I > should use. > > > > > > > > > I should mention that the inhouse code is for > symmetric > > > > > > > > matrices and > > > > > > > > > from what I understand, the petsc solver works for > > > > > > > > > general > > > > > > > > unsymmetric matrices. > > > > > > > > > But I think for iterative solvers, it should > still give > > > > > > around the > > > > > > > > > same performance. > > > > > > > > > I tested the solvers against some other problems as > > > > > well, and I > > > > > > > > > got the same performance.. In some cases, no > > > matter how many > > > > > > > > iterations it > > > > > > > > > goes through, the petsc solver would not go below > > > a certain > > > > > > > > error norm > > > > > > > > > whereas the inhouse solver would get almost > exactly the > > > > > > > > same answer as > > > > > > > > > the direct solver solution. I'm thinking the petsc > > > > > > solver should > > > > > > > > > be able to solve this problem just as easily. > It would > > > > > > > > > be > > > > > > > > great if anyone > > > > > > > > > could help me figure out the appropriate settings I > > > > > > > > > should > > > > > > > > use in the petsc solver. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From julvar at tamu.edu Thu Oct 12 14:31:09 2006 From: julvar at tamu.edu (Julian) Date: Thu, 12 Oct 2006 14:31:09 -0500 Subject: General matrix interative solver In-Reply-To: <00cf01c6ee2e$bc8b7440$24b75ba5@aero.ad.tamu.edu> Message-ID: <00d001c6ee34$f87ab820$24b75ba5@aero.ad.tamu.edu> I think I got the preallocation right.. But it still gets stuck at pretty much the same spot. When it slowed to almost pretty much a full stop, I did a break all in debug mode and it looks like its stuck in a memset function which according to the call stack is called down the line from the following petsc function: PetscErrorCode PETSC_DLLEXPORT PetscMallocAlign(size_t mem,int line,const char func[],const char file[],const char dir[],void** result) I am using the symmetric matrix (MATSBAIJ) ..and I'm not sure how I should specify the nnz when I create the matrix Is it the number of non-zeroes in each row in the upper triangle (cos that is all I'm storing, right?) or is it the non zeroes in each row for the whole matrix? Julian. > -----Original Message----- > From: owner-petsc-users at mcs.anl.gov > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Julian > Sent: Thursday, October 12, 2006 1:47 PM > To: petsc-users at mcs.anl.gov > Subject: RE: General matrix interative solver > > Thanks, I got it working after I switched to icc > > All this time I was running test cases where I just read in > an already assembled global stiffness matrix and load vector > and solved the problem using petsc. > > After I got that working, I tried to fully integrate the > solver with the FEA program. > One problem I had was with populating the matrix. The > interfaces to the other existing solvers in our code were > written so that we can assign a value directly to a memory > space assigned for an element in the matrix. > Something like mat(I,J)=value; Where mat(I,J) gives the > reference to the location. But petsc does not work that way > (I had already asked this on the mailing list a couple of > months back) and I understand maybe its wise to do that, > considering it handles parallel implementation as well. So, I > changed our code to work with petsc. > > Now, when assembling the global matrix, sometimes I need to > use ADD_VALUES and other times INSERT_VALUES (depending on > constraints) > At first, I used MatAssemblyBegin(*(Mat*)mat, > MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); But then it > took too long and I realized I should be using > MAT_FLUSH_ASSEMBLY. But even then it takes considerably more > time in assembling than my other solvers. So, I used a flag > to call the assembly only when I am switching between > ADD_VALUES and INSERT_VALUES. That reduced the calls to > MAT_FLUSH_ASSEMBLY but it still was a slower than the other > solvers.. And this was for a small problem (1317 unknowns) I > tried it out with a larger problem ~15000 unknowns and the > program just got stuck during the assembly process ! It > wouldn't crash but looks like its doing something inside of > petsc... probably memory allocations. So, I think I need to > start preallocating memory. I'm working on that now... So > hopefully, after that the assembly process won't slow down > bcause of memory allocation. But my question for now... is > using the flag method to call assembly only when needed the > most efficient way to do this? Or do you have any better > ideas? One other possiblity is to have an array of flags, so > as to avoid the call to INSERT_VALUES altogether... And > thereby eliminating the need to call MAT_FLUSH_ASSEMBLY. > I'll get back to you after I get the preallocation right. > Maybe that will fix everything. > > Thanks, > Julian. > > > > -----Original Message----- > > From: owner-petsc-users at mcs.anl.gov > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry Smith > > Sent: Wednesday, October 11, 2006 2:58 PM > > To: petsc-users at mcs.anl.gov > > Subject: RE: General matrix interative solver > > > > > > ILU is for general matrices; for symmetric you need to use > > PCSetType(pc,PCICC); or -pc_type icc > > > > Barry > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > Hong, > > > I made the following change to specify it to be in sbaij > > format... And > > > then proceeded to store only the upper triangle. > > > > > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, PETSC_DEFAULT, > > PETSC_NULL, > > > (Mat*)mat); > > > MatSetType(*(Mat*)mat,MATSBAIJ); > > > > > > But I get the following error when I try to solve: > > > > > > > > > ---------------------------------------------------------------------- > > > -- Petsc Release Version 2.3.1, Patch 16, Fri Jul 21 > > 15:09:15 CDT 2006 > > > HG > > > revision: > > > 5bc424fae3770001f0d316695a9956fde3bc58b0 > > > See docs/changes/index.html for recent updates. > > > See docs/faq.html for hints about trouble shooting. > > > See docs/index.html for manual pages. > > > > > > ---------------------------------------------------------------------- > > > -- Unknown Name on a cygwin-cx named ALPHA2 by j0v1008 Wed Oct 11 > > > 14:51:00 2006 Libraries linked from > > > /cygdrive/e/cygwin/home/petsc-2.3.1-p16/lib/cygwin-cxx-rea > > > l-debug > > > Configure run at Mon Aug 14 15:35:57 2006 Configure options > > > --with-cc="win32fe cl --nodetect" --with-cxx="win32fe cl > > --nod etect" > > > --COPTFLAGS="-MDd -Z7" --with-fc=0 --with-clanguage=cxx > > > --download-c-blas > > > -lapack=1 --with-mpi=0 --useThreads=0 --with-shared=0 > > > > > > ---------------------------------------------------------------------- > > > -- [0]PETSC ERROR: MatILUFactorSymbolic() line 4573 in > > > src/mat/interface/e:\cygwin\ > > > home\PETSC-~1.1-P\src\mat\INTERF~1\matrix.c > > > [0]PETSC ERROR: No support for this operation for this > object type! > > > [0]PETSC ERROR: Matrix type sbaij symbolic ILU! > > > [0]PETSC ERROR: PCSetUp_ILU() line 532 in > > > src/ksp/pc/impls/factor/ilu/e:\cygwin\ > > > home\PETSC-~1.1-P\src\ksp\pc\impls\factor\ilu\ilu.c > > > [0]PETSC ERROR: PCSetUp() line 798 in > > > src/ksp/pc/interface/e:\cygwin\home\PETSC- > > > ~1.1-P\src\ksp\pc\INTERF~1\precon.c > > > [0]PETSC ERROR: KSPSetUp() line 234 in > > > src/ksp/ksp/interface/e:\cygwin\home\PETS > > > C-~1.1-P\src\ksp\ksp\INTERF~1\itfunc.c > > > [0]PETSC ERROR: KSPSolve() line 334 in > > > src/ksp/ksp/interface/e:\cygwin\home\PETS > > > C-~1.1-P\src\ksp\ksp\INTERF~1\itfunc.c > > > > > > > -----Original Message----- > > > > From: owner-petsc-users at mcs.anl.gov > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Hong Zhang > > > > Sent: Wednesday, October 11, 2006 2:14 PM > > > > To: petsc-users at mcs.anl.gov > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > Julian, > > > > > > > > If you do not perform matrix reordering, you can use > petsc sbaij > > > > format, which only stores the upper triangular entries. > > > > aij format has more algorithmic options and is more > efficient and > > > > supports matrix reordering. > > > > > > > > Hong > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > I'm using '-ksp_type cg' now to solve the symmetric > > > > matric...But then > > > > > do I need to store all the non-zero elements of a symmetric > > > > matrix ? > > > > > i.e. A[i,j] as well as A[j,i] ? > > > > > > > > > > Shouldn't I be able to store just the upper or lower > > > > triangle or the > > > > > matrix ? Is that possible with petsc ? > > > > > > > > > > Julian. > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Julian > > > > > > Sent: Wednesday, October 11, 2006 12:37 PM > > > > > > To: petsc-users at mcs.anl.gov > > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > Barry, > > > > > > > > > > > > I tried sending the commands from a file for now... And > > > > once I used > > > > > > '-ksp_type cg', I got pretty much the same > > performance as that > > > > > > of the inhouse code ! > > > > > > Now, I'll try the performance with larger cases... > > > > > > > > > > > > Thanks for your help! > > > > > > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry > > > > > > > Smith > > > > > > > Sent: Wednesday, October 11, 2006 11:25 AM > > > > > > > To: petsc-users at mcs.anl.gov > > > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > > > > > > > > PetscOptionsSetValue() but recommend putting them on > > > > the command > > > > > > > line or in a file (pass the name of the file into > > > > > > PetscInitialize()). > > > > > > > Having to recompile for every option is too painful. > > > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > > > Hong, > > > > > > > > > > > > > > > > The times are for the solution process alone and the > > > > > > > initial guess is > > > > > > > > the same, i.e. zero. > > > > > > > > > > > > > > > > As for the algorithm, they are probably different. > > > > > > > > Can you tell me how to pass 'runtime options' like -help > > > > > > > and -ksp_view > > > > > > > > from within the code... I mean, at compile time. > > > > > > > > > > > > > > > > thanks > > > > > > > > > > > > > > > > > > > > > > > > > > Julian, > > > > > > > > > > > > > > > > > > When comparing the inhouse solver and petsc > solver, you > > > > > > need make > > > > > > > > > sure > > > > > > > > > > > > > > > > > > 1. Timings are collected for solution process. > > The matrix > > > > > > > and vector > > > > > > > > > assembly should be excluded. > > > > > > > > > 2. They should use same iterative algorithm. > By default, > > > > > > > petsc uses > > > > > > > > > gmres with restart=30 and ilu(0) preconditioner. > > > > > > Petsc supports > > > > > > > > > symmetric matrices, e.g., runtime option > > '-ksp_type cg > > > > > > > -pc_type > > > > > > > > > icc' > > > > > > > > > might give better performance 3. They should > > start from > > > > > > > > > same intial guess. By default, petsc > > > > > > > > > initial guess is zero. > > > > > > > > > > > > > > > > > > You can use '-ksp_view' to see what algorithm > > and options > > > > > > > are used > > > > > > > > > in petsc. > > > > > > > > > > > > > > > > > > Hong > > > > > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > I implemented the iterative sparse matrix solver in > > > > > > > > > > PetSc > > > > > > > > > into my FEM > > > > > > > > > > code recently. I compared the results from a problem > > > > > > with 1317 > > > > > > > > > > unknowns. I used a direct solver to obtain > > the reference > > > > > > > > > solution. I > > > > > > > > > > have another in-house sparse iterative > solver that I > > > > > > > > > > have > > > > > > > > > been using > > > > > > > > > > so far. It was written by someone else but I have > > > > access to > > > > > > > > > the source for that solver. > > > > > > > > > > I find the 'error norm' in the solution by > taking the > > > > > > > > > square root of > > > > > > > > > > the sum of the squares of the absolute differences > > > > > > between the > > > > > > > > > > solution from the direct solver and the iterative > > > > > > solver. I am > > > > > > > > > > ignoring the numerical zeros in the solutions > > > > when doing this. > > > > > > > > > > I find that in order to get same order of the error > > > > > > > norm (1e-13) > > > > > > > > > > as the in-house iterative solver, the petsc solver > > > > > > takes a much > > > > > > > > > > longer time and larger number of iterations. While > > > > > > the inhouse > > > > > > > > > > solver took less than one second, the petsc > > > > solver took 13 > > > > > > > > > > seconds. The inhouse solver took 476 > > iterations whereas > > > > > > > the petsc > > > > > > > > > > solver took > > > > > > > > > 4738 iterations. > > > > > > > > > > I'm guessing this has to do with different > setting of > > > > > > > the solver > > > > > > > > > > in petsc such as the preconditioner etc. > > > > > > > > > > Can you tell me what the different settings are? > > > > And how to > > > > > > > > > tweak them > > > > > > > > > > so that I can atleast get as good as a > performance as > > > > > > > > > > the > > > > > > > > > inhouse code ? > > > > > > > > > > Given below is how I have implemented the > > petsc solver: > > > > > > > > > > > > > > > > > > > > /////initialization > > > > > > > > > > PetscInitializeNoArguments(); > > > > > > > > > > Assert( mat = > (double*)malloc(sizeof(Mat)) ); > > > > > > > > > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, > > > > > > > > > > PETSC_DEFAULT, PETSC_NULL, (Mat*)mat); > > > > > > > > > > > > > > > > > > > > ////// this is the function I use to populate > > the matrix > > > > > > > > > > MatSetValue(*(Mat*)mat, ii, jj, value, ADD_VALUES); > > > > > > > > > > > > > > > > > > > > ////// this is how I actaully solve the matrix > > > > > > > > > > MatAssemblyBegin(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > > > > > > > MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > > > > > > > > > > > > > > > > > double iter_error = 1e-10; > > > > > > > > > > int max_iter_num = 10000; > > > > > > > > > > int num_of_iter; > > > > > > > > > > > > > > > > > > > > Vec rhs, x; > > > > > > > > > > VecCreateSeqWithArray(PETSC_COMM_SELF, L, > > b, &rhs); > > > > > > > > > > VecDuplicate(rhs, &x); > > > > > > > > > > > > > > > > > > > > KSP ksp; > > > > > > > > > > KSPCreate(PETSC_COMM_SELF, &ksp); > > > > > > > > > > KSPSetTolerances(ksp, iter_error, PETSC_DEFAULT, > > > > > > > > > > PETSC_DEFAULT, max_iter_num); > > > > > > > > > > KSPSetFromOptions(ksp); > > > > > > > > > > KSPSetOperators(ksp, *(Mat*)mat, *(Mat*)mat, > > > > > > > > > > SAME_PRECONDITIONER); > > > > > > > > > > KSPSolve(ksp,rhs,x); > > > > > > > > > > > > > > > > > > > > PetscReal r_norm; > > > > > > > > > > KSPGetResidualNorm(ksp, &r_norm); > > > > > > > > > > KSPGetIterationNumber(ksp, &num_of_iter); > > > > > > > > > > > > > > > > > > > > cout << "max_iter_num\t" << > > max_iter_num << endl; > > > > > > > > > > cout << "iter_error\t" << > iter_error << endl; > > > > > > > > > > > > > > > > > > > > cout << "Matrix solver step " << > > num_of_iter << > > > > > > > > > > ", > > > > > > > > > residual " > > > > > > > > > > << r_norm << ".\n"; > > > > > > > > > > > > > > > > > > > > PetscScalar *p; > > > > > > > > > > VecGetArray(x, &p); > > > > > > > > > > for(int i=0; i > > > > > > > > > b[i] = p[i]; > > > > > > > > > > } > > > > > > > > > > VecRestoreArray(x, &p); > > > > > > > > > > > > > > > > > > > > KSPDestroy(ksp); > > > > > > > > > > VecDestroy(rhs); > > > > > > > > > > VecDestroy(x); > > > > > > > > > > > > > > > > > > > > cout <<"Iterations for convergence="<< > > num_of_iter << " > > > > > > > > > > - > > > > > > > > > Residual Norm = " > > > > > > > > > > << r_norm << endl; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If this is not the typical method to be > used to solve > > > > > > > this kind of > > > > > > > > > > problem, please let me know what functions I > > should use. > > > > > > > > > > I should mention that the inhouse code is for > > symmetric > > > > > > > > > matrices and > > > > > > > > > > from what I understand, the petsc solver works for > > > > > > > > > > general > > > > > > > > > unsymmetric matrices. > > > > > > > > > > But I think for iterative solvers, it should > > still give > > > > > > > around the > > > > > > > > > > same performance. > > > > > > > > > > I tested the solvers against some other problems as > > > > > > well, and I > > > > > > > > > > got the same performance.. In some cases, no > > > > matter how many > > > > > > > > > iterations it > > > > > > > > > > goes through, the petsc solver would not go below > > > > a certain > > > > > > > > > error norm > > > > > > > > > > whereas the inhouse solver would get almost > > exactly the > > > > > > > > > same answer as > > > > > > > > > > the direct solver solution. I'm thinking the petsc > > > > > > > solver should > > > > > > > > > > be able to solve this problem just as easily. > > It would > > > > > > > > > > be > > > > > > > > > great if anyone > > > > > > > > > > could help me figure out the appropriate settings I > > > > > > > > > > should > > > > > > > > > use in the petsc solver. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From balay at mcs.anl.gov Thu Oct 12 14:32:52 2006 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 12 Oct 2006 14:32:52 -0500 (CDT) Subject: General matrix interative solver In-Reply-To: <00d001c6ee34$f87ab820$24b75ba5@aero.ad.tamu.edu> References: <00d001c6ee34$f87ab820$24b75ba5@aero.ad.tamu.edu> Message-ID: On Thu, 12 Oct 2006, Julian wrote: > I think I got the preallocation right.. But it still gets stuck at pretty > much the same spot. Can you veify with '-info' - and look for number of mallocs in the verbose output. Satish > When it slowed to almost pretty much a full stop, I did a break all in debug > mode and it looks like its stuck in a memset function which according to the > call stack is called down the line from the following petsc function: > PetscErrorCode PETSC_DLLEXPORT PetscMallocAlign(size_t mem,int line,const > char func[],const char file[],const char dir[],void** result) > > I am using the symmetric matrix (MATSBAIJ) ..and I'm not sure how I should > specify the nnz when I create the matrix > Is it the number of non-zeroes in each row in the upper triangle (cos that > is all I'm storing, right?) or is it the non zeroes in each row for the > whole matrix? > > Julian. > > > -----Original Message----- > > From: owner-petsc-users at mcs.anl.gov > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Julian > > Sent: Thursday, October 12, 2006 1:47 PM > > To: petsc-users at mcs.anl.gov > > Subject: RE: General matrix interative solver > > > > Thanks, I got it working after I switched to icc > > > > All this time I was running test cases where I just read in > > an already assembled global stiffness matrix and load vector > > and solved the problem using petsc. > > > > After I got that working, I tried to fully integrate the > > solver with the FEA program. > > One problem I had was with populating the matrix. The > > interfaces to the other existing solvers in our code were > > written so that we can assign a value directly to a memory > > space assigned for an element in the matrix. > > Something like mat(I,J)=value; Where mat(I,J) gives the > > reference to the location. But petsc does not work that way > > (I had already asked this on the mailing list a couple of > > months back) and I understand maybe its wise to do that, > > considering it handles parallel implementation as well. So, I > > changed our code to work with petsc. > > > > Now, when assembling the global matrix, sometimes I need to > > use ADD_VALUES and other times INSERT_VALUES (depending on > > constraints) > > At first, I used MatAssemblyBegin(*(Mat*)mat, > > MAT_FINAL_ASSEMBLY); > > MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); But then it > > took too long and I realized I should be using > > MAT_FLUSH_ASSEMBLY. But even then it takes considerably more > > time in assembling than my other solvers. So, I used a flag > > to call the assembly only when I am switching between > > ADD_VALUES and INSERT_VALUES. That reduced the calls to > > MAT_FLUSH_ASSEMBLY but it still was a slower than the other > > solvers.. And this was for a small problem (1317 unknowns) I > > tried it out with a larger problem ~15000 unknowns and the > > program just got stuck during the assembly process ! It > > wouldn't crash but looks like its doing something inside of > > petsc... probably memory allocations. So, I think I need to > > start preallocating memory. I'm working on that now... So > > hopefully, after that the assembly process won't slow down > > bcause of memory allocation. But my question for now... is > > using the flag method to call assembly only when needed the > > most efficient way to do this? Or do you have any better > > ideas? One other possiblity is to have an array of flags, so > > as to avoid the call to INSERT_VALUES altogether... And > > thereby eliminating the need to call MAT_FLUSH_ASSEMBLY. > > I'll get back to you after I get the preallocation right. > > Maybe that will fix everything. > > > > Thanks, > > Julian. > > > > > > > -----Original Message----- > > > From: owner-petsc-users at mcs.anl.gov > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry Smith > > > Sent: Wednesday, October 11, 2006 2:58 PM > > > To: petsc-users at mcs.anl.gov > > > Subject: RE: General matrix interative solver > > > > > > > > > ILU is for general matrices; for symmetric you need to use > > > PCSetType(pc,PCICC); or -pc_type icc > > > > > > Barry > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > Hong, > > > > I made the following change to specify it to be in sbaij > > > format... And > > > > then proceeded to store only the upper triangle. > > > > > > > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > > > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, PETSC_DEFAULT, > > > PETSC_NULL, > > > > (Mat*)mat); > > > > MatSetType(*(Mat*)mat,MATSBAIJ); > > > > > > > > But I get the following error when I try to solve: > > > > > > > > > > > > > ---------------------------------------------------------------------- > > > > -- Petsc Release Version 2.3.1, Patch 16, Fri Jul 21 > > > 15:09:15 CDT 2006 > > > > HG > > > > revision: > > > > 5bc424fae3770001f0d316695a9956fde3bc58b0 > > > > See docs/changes/index.html for recent updates. > > > > See docs/faq.html for hints about trouble shooting. > > > > See docs/index.html for manual pages. > > > > > > > > > ---------------------------------------------------------------------- > > > > -- Unknown Name on a cygwin-cx named ALPHA2 by j0v1008 Wed Oct 11 > > > > 14:51:00 2006 Libraries linked from > > > > /cygdrive/e/cygwin/home/petsc-2.3.1-p16/lib/cygwin-cxx-rea > > > > l-debug > > > > Configure run at Mon Aug 14 15:35:57 2006 Configure options > > > > --with-cc="win32fe cl --nodetect" --with-cxx="win32fe cl > > > --nod etect" > > > > --COPTFLAGS="-MDd -Z7" --with-fc=0 --with-clanguage=cxx > > > > --download-c-blas > > > > -lapack=1 --with-mpi=0 --useThreads=0 --with-shared=0 > > > > > > > > > ---------------------------------------------------------------------- > > > > -- [0]PETSC ERROR: MatILUFactorSymbolic() line 4573 in > > > > src/mat/interface/e:\cygwin\ > > > > home\PETSC-~1.1-P\src\mat\INTERF~1\matrix.c > > > > [0]PETSC ERROR: No support for this operation for this > > object type! > > > > [0]PETSC ERROR: Matrix type sbaij symbolic ILU! > > > > [0]PETSC ERROR: PCSetUp_ILU() line 532 in > > > > src/ksp/pc/impls/factor/ilu/e:\cygwin\ > > > > home\PETSC-~1.1-P\src\ksp\pc\impls\factor\ilu\ilu.c > > > > [0]PETSC ERROR: PCSetUp() line 798 in > > > > src/ksp/pc/interface/e:\cygwin\home\PETSC- > > > > ~1.1-P\src\ksp\pc\INTERF~1\precon.c > > > > [0]PETSC ERROR: KSPSetUp() line 234 in > > > > src/ksp/ksp/interface/e:\cygwin\home\PETS > > > > C-~1.1-P\src\ksp\ksp\INTERF~1\itfunc.c > > > > [0]PETSC ERROR: KSPSolve() line 334 in > > > > src/ksp/ksp/interface/e:\cygwin\home\PETS > > > > C-~1.1-P\src\ksp\ksp\INTERF~1\itfunc.c > > > > > > > > > -----Original Message----- > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Hong Zhang > > > > > Sent: Wednesday, October 11, 2006 2:14 PM > > > > > To: petsc-users at mcs.anl.gov > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > > Julian, > > > > > > > > > > If you do not perform matrix reordering, you can use > > petsc sbaij > > > > > format, which only stores the upper triangular entries. > > > > > aij format has more algorithmic options and is more > > efficient and > > > > > supports matrix reordering. > > > > > > > > > > Hong > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > I'm using '-ksp_type cg' now to solve the symmetric > > > > > matric...But then > > > > > > do I need to store all the non-zero elements of a symmetric > > > > > matrix ? > > > > > > i.e. A[i,j] as well as A[j,i] ? > > > > > > > > > > > > Shouldn't I be able to store just the upper or lower > > > > > triangle or the > > > > > > matrix ? Is that possible with petsc ? > > > > > > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Julian > > > > > > > Sent: Wednesday, October 11, 2006 12:37 PM > > > > > > > To: petsc-users at mcs.anl.gov > > > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > Barry, > > > > > > > > > > > > > > I tried sending the commands from a file for now... And > > > > > once I used > > > > > > > '-ksp_type cg', I got pretty much the same > > > performance as that > > > > > > > of the inhouse code ! > > > > > > > Now, I'll try the performance with larger cases... > > > > > > > > > > > > > > Thanks for your help! > > > > > > > > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry > > > > > > > > Smith > > > > > > > > Sent: Wednesday, October 11, 2006 11:25 AM > > > > > > > > To: petsc-users at mcs.anl.gov > > > > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > > > > > > > > > > > PetscOptionsSetValue() but recommend putting them on > > > > > the command > > > > > > > > line or in a file (pass the name of the file into > > > > > > > PetscInitialize()). > > > > > > > > Having to recompile for every option is too painful. > > > > > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > > > > > Hong, > > > > > > > > > > > > > > > > > > The times are for the solution process alone and the > > > > > > > > initial guess is > > > > > > > > > the same, i.e. zero. > > > > > > > > > > > > > > > > > > As for the algorithm, they are probably different. > > > > > > > > > Can you tell me how to pass 'runtime options' like -help > > > > > > > > and -ksp_view > > > > > > > > > from within the code... I mean, at compile time. > > > > > > > > > > > > > > > > > > thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Julian, > > > > > > > > > > > > > > > > > > > > When comparing the inhouse solver and petsc > > solver, you > > > > > > > need make > > > > > > > > > > sure > > > > > > > > > > > > > > > > > > > > 1. Timings are collected for solution process. > > > The matrix > > > > > > > > and vector > > > > > > > > > > assembly should be excluded. > > > > > > > > > > 2. They should use same iterative algorithm. > > By default, > > > > > > > > petsc uses > > > > > > > > > > gmres with restart=30 and ilu(0) preconditioner. > > > > > > > Petsc supports > > > > > > > > > > symmetric matrices, e.g., runtime option > > > '-ksp_type cg > > > > > > > > -pc_type > > > > > > > > > > icc' > > > > > > > > > > might give better performance 3. They should > > > start from > > > > > > > > > > same intial guess. By default, petsc > > > > > > > > > > initial guess is zero. > > > > > > > > > > > > > > > > > > > > You can use '-ksp_view' to see what algorithm > > > and options > > > > > > > > are used > > > > > > > > > > in petsc. > > > > > > > > > > > > > > > > > > > > Hong > > > > > > > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > I implemented the iterative sparse matrix solver in > > > > > > > > > > > PetSc > > > > > > > > > > into my FEM > > > > > > > > > > > code recently. I compared the results from a problem > > > > > > > with 1317 > > > > > > > > > > > unknowns. I used a direct solver to obtain > > > the reference > > > > > > > > > > solution. I > > > > > > > > > > > have another in-house sparse iterative > > solver that I > > > > > > > > > > > have > > > > > > > > > > been using > > > > > > > > > > > so far. It was written by someone else but I have > > > > > access to > > > > > > > > > > the source for that solver. > > > > > > > > > > > I find the 'error norm' in the solution by > > taking the > > > > > > > > > > square root of > > > > > > > > > > > the sum of the squares of the absolute differences > > > > > > > between the > > > > > > > > > > > solution from the direct solver and the iterative > > > > > > > solver. I am > > > > > > > > > > > ignoring the numerical zeros in the solutions > > > > > when doing this. > > > > > > > > > > > I find that in order to get same order of the error > > > > > > > > norm (1e-13) > > > > > > > > > > > as the in-house iterative solver, the petsc solver > > > > > > > takes a much > > > > > > > > > > > longer time and larger number of iterations. While > > > > > > > the inhouse > > > > > > > > > > > solver took less than one second, the petsc > > > > > solver took 13 > > > > > > > > > > > seconds. The inhouse solver took 476 > > > iterations whereas > > > > > > > > the petsc > > > > > > > > > > > solver took > > > > > > > > > > 4738 iterations. > > > > > > > > > > > I'm guessing this has to do with different > > setting of > > > > > > > > the solver > > > > > > > > > > > in petsc such as the preconditioner etc. > > > > > > > > > > > Can you tell me what the different settings are? > > > > > And how to > > > > > > > > > > tweak them > > > > > > > > > > > so that I can atleast get as good as a > > performance as > > > > > > > > > > > the > > > > > > > > > > inhouse code ? > > > > > > > > > > > Given below is how I have implemented the > > > petsc solver: > > > > > > > > > > > > > > > > > > > > > > /////initialization > > > > > > > > > > > PetscInitializeNoArguments(); > > > > > > > > > > > Assert( mat = > > (double*)malloc(sizeof(Mat)) ); > > > > > > > > > > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, > > > > > > > > > > > PETSC_DEFAULT, PETSC_NULL, (Mat*)mat); > > > > > > > > > > > > > > > > > > > > > > ////// this is the function I use to populate > > > the matrix > > > > > > > > > > > MatSetValue(*(Mat*)mat, ii, jj, value, ADD_VALUES); > > > > > > > > > > > > > > > > > > > > > > ////// this is how I actaully solve the matrix > > > > > > > > > > > MatAssemblyBegin(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > > > > > > > > MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > > > > > > > > > > > > > > > > > > > double iter_error = 1e-10; > > > > > > > > > > > int max_iter_num = 10000; > > > > > > > > > > > int num_of_iter; > > > > > > > > > > > > > > > > > > > > > > Vec rhs, x; > > > > > > > > > > > VecCreateSeqWithArray(PETSC_COMM_SELF, L, > > > b, &rhs); > > > > > > > > > > > VecDuplicate(rhs, &x); > > > > > > > > > > > > > > > > > > > > > > KSP ksp; > > > > > > > > > > > KSPCreate(PETSC_COMM_SELF, &ksp); > > > > > > > > > > > KSPSetTolerances(ksp, iter_error, PETSC_DEFAULT, > > > > > > > > > > > PETSC_DEFAULT, max_iter_num); > > > > > > > > > > > KSPSetFromOptions(ksp); > > > > > > > > > > > KSPSetOperators(ksp, *(Mat*)mat, *(Mat*)mat, > > > > > > > > > > > SAME_PRECONDITIONER); > > > > > > > > > > > KSPSolve(ksp,rhs,x); > > > > > > > > > > > > > > > > > > > > > > PetscReal r_norm; > > > > > > > > > > > KSPGetResidualNorm(ksp, &r_norm); > > > > > > > > > > > KSPGetIterationNumber(ksp, &num_of_iter); > > > > > > > > > > > > > > > > > > > > > > cout << "max_iter_num\t" << > > > max_iter_num << endl; > > > > > > > > > > > cout << "iter_error\t" << > > iter_error << endl; > > > > > > > > > > > > > > > > > > > > > > cout << "Matrix solver step " << > > > num_of_iter << > > > > > > > > > > > ", > > > > > > > > > > residual " > > > > > > > > > > > << r_norm << ".\n"; > > > > > > > > > > > > > > > > > > > > > > PetscScalar *p; > > > > > > > > > > > VecGetArray(x, &p); > > > > > > > > > > > for(int i=0; i > > > > > > > > > > b[i] = p[i]; > > > > > > > > > > > } > > > > > > > > > > > VecRestoreArray(x, &p); > > > > > > > > > > > > > > > > > > > > > > KSPDestroy(ksp); > > > > > > > > > > > VecDestroy(rhs); > > > > > > > > > > > VecDestroy(x); > > > > > > > > > > > > > > > > > > > > > > cout <<"Iterations for convergence="<< > > > num_of_iter << " > > > > > > > > > > > - > > > > > > > > > > Residual Norm = " > > > > > > > > > > > << r_norm << endl; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If this is not the typical method to be > > used to solve > > > > > > > > this kind of > > > > > > > > > > > problem, please let me know what functions I > > > should use. > > > > > > > > > > > I should mention that the inhouse code is for > > > symmetric > > > > > > > > > > matrices and > > > > > > > > > > > from what I understand, the petsc solver works for > > > > > > > > > > > general > > > > > > > > > > unsymmetric matrices. > > > > > > > > > > > But I think for iterative solvers, it should > > > still give > > > > > > > > around the > > > > > > > > > > > same performance. > > > > > > > > > > > I tested the solvers against some other problems as > > > > > > > well, and I > > > > > > > > > > > got the same performance.. In some cases, no > > > > > matter how many > > > > > > > > > > iterations it > > > > > > > > > > > goes through, the petsc solver would not go below > > > > > a certain > > > > > > > > > > error norm > > > > > > > > > > > whereas the inhouse solver would get almost > > > exactly the > > > > > > > > > > same answer as > > > > > > > > > > > the direct solver solution. I'm thinking the petsc > > > > > > > > solver should > > > > > > > > > > > be able to solve this problem just as easily. > > > It would > > > > > > > > > > > be > > > > > > > > > > great if anyone > > > > > > > > > > > could help me figure out the appropriate settings I > > > > > > > > > > > should > > > > > > > > > > use in the petsc solver. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From bsmith at mcs.anl.gov Thu Oct 12 14:33:45 2006 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 12 Oct 2006 14:33:45 -0500 (CDT) Subject: General matrix interative solver In-Reply-To: <00d001c6ee34$f87ab820$24b75ba5@aero.ad.tamu.edu> References: <00d001c6ee34$f87ab820$24b75ba5@aero.ad.tamu.edu> Message-ID: On Thu, 12 Oct 2006, Julian wrote: > I think I got the preallocation right.. But it still gets stuck at pretty > much the same spot. > When it slowed to almost pretty much a full stop, I did a break all in debug > mode and it looks like its stuck in a memset function which according to the > call stack is called down the line from the following petsc function: > PetscErrorCode PETSC_DLLEXPORT PetscMallocAlign(size_t mem,int line,const > char func[],const char file[],const char dir[],void** result) > The preallocation is not being done correctly. > I am using the symmetric matrix (MATSBAIJ) ..and I'm not sure how I should > specify the nnz when I create the matrix > Is it the number of non-zeroes in each row in the upper triangle (cos that > is all I'm storing, right?) or is it the non zeroes in each row for the > whole matrix? Nonzeros in the upper triangular plus diagonal. > > Julian. > > > -----Original Message----- > > From: owner-petsc-users at mcs.anl.gov > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Julian > > Sent: Thursday, October 12, 2006 1:47 PM > > To: petsc-users at mcs.anl.gov > > Subject: RE: General matrix interative solver > > > > Thanks, I got it working after I switched to icc > > > > All this time I was running test cases where I just read in > > an already assembled global stiffness matrix and load vector > > and solved the problem using petsc. > > > > After I got that working, I tried to fully integrate the > > solver with the FEA program. > > One problem I had was with populating the matrix. The > > interfaces to the other existing solvers in our code were > > written so that we can assign a value directly to a memory > > space assigned for an element in the matrix. > > Something like mat(I,J)=value; Where mat(I,J) gives the > > reference to the location. But petsc does not work that way > > (I had already asked this on the mailing list a couple of > > months back) and I understand maybe its wise to do that, > > considering it handles parallel implementation as well. So, I > > changed our code to work with petsc. > > > > Now, when assembling the global matrix, sometimes I need to > > use ADD_VALUES and other times INSERT_VALUES (depending on > > constraints) > > At first, I used MatAssemblyBegin(*(Mat*)mat, > > MAT_FINAL_ASSEMBLY); > > MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); But then it > > took too long and I realized I should be using > > MAT_FLUSH_ASSEMBLY. But even then it takes considerably more > > time in assembling than my other solvers. So, I used a flag > > to call the assembly only when I am switching between > > ADD_VALUES and INSERT_VALUES. That reduced the calls to > > MAT_FLUSH_ASSEMBLY but it still was a slower than the other > > solvers.. And this was for a small problem (1317 unknowns) I > > tried it out with a larger problem ~15000 unknowns and the > > program just got stuck during the assembly process ! It > > wouldn't crash but looks like its doing something inside of > > petsc... probably memory allocations. So, I think I need to > > start preallocating memory. I'm working on that now... So > > hopefully, after that the assembly process won't slow down > > bcause of memory allocation. But my question for now... is > > using the flag method to call assembly only when needed the > > most efficient way to do this? Or do you have any better > > ideas? One other possiblity is to have an array of flags, so > > as to avoid the call to INSERT_VALUES altogether... And > > thereby eliminating the need to call MAT_FLUSH_ASSEMBLY. > > I'll get back to you after I get the preallocation right. > > Maybe that will fix everything. > > > > Thanks, > > Julian. > > > > > > > -----Original Message----- > > > From: owner-petsc-users at mcs.anl.gov > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry Smith > > > Sent: Wednesday, October 11, 2006 2:58 PM > > > To: petsc-users at mcs.anl.gov > > > Subject: RE: General matrix interative solver > > > > > > > > > ILU is for general matrices; for symmetric you need to use > > > PCSetType(pc,PCICC); or -pc_type icc > > > > > > Barry > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > Hong, > > > > I made the following change to specify it to be in sbaij > > > format... And > > > > then proceeded to store only the upper triangle. > > > > > > > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > > > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, PETSC_DEFAULT, > > > PETSC_NULL, > > > > (Mat*)mat); > > > > MatSetType(*(Mat*)mat,MATSBAIJ); > > > > > > > > But I get the following error when I try to solve: > > > > > > > > > > > > > ---------------------------------------------------------------------- > > > > -- Petsc Release Version 2.3.1, Patch 16, Fri Jul 21 > > > 15:09:15 CDT 2006 > > > > HG > > > > revision: > > > > 5bc424fae3770001f0d316695a9956fde3bc58b0 > > > > See docs/changes/index.html for recent updates. > > > > See docs/faq.html for hints about trouble shooting. > > > > See docs/index.html for manual pages. > > > > > > > > > ---------------------------------------------------------------------- > > > > -- Unknown Name on a cygwin-cx named ALPHA2 by j0v1008 Wed Oct 11 > > > > 14:51:00 2006 Libraries linked from > > > > /cygdrive/e/cygwin/home/petsc-2.3.1-p16/lib/cygwin-cxx-rea > > > > l-debug > > > > Configure run at Mon Aug 14 15:35:57 2006 Configure options > > > > --with-cc="win32fe cl --nodetect" --with-cxx="win32fe cl > > > --nod etect" > > > > --COPTFLAGS="-MDd -Z7" --with-fc=0 --with-clanguage=cxx > > > > --download-c-blas > > > > -lapack=1 --with-mpi=0 --useThreads=0 --with-shared=0 > > > > > > > > > ---------------------------------------------------------------------- > > > > -- [0]PETSC ERROR: MatILUFactorSymbolic() line 4573 in > > > > src/mat/interface/e:\cygwin\ > > > > home\PETSC-~1.1-P\src\mat\INTERF~1\matrix.c > > > > [0]PETSC ERROR: No support for this operation for this > > object type! > > > > [0]PETSC ERROR: Matrix type sbaij symbolic ILU! > > > > [0]PETSC ERROR: PCSetUp_ILU() line 532 in > > > > src/ksp/pc/impls/factor/ilu/e:\cygwin\ > > > > home\PETSC-~1.1-P\src\ksp\pc\impls\factor\ilu\ilu.c > > > > [0]PETSC ERROR: PCSetUp() line 798 in > > > > src/ksp/pc/interface/e:\cygwin\home\PETSC- > > > > ~1.1-P\src\ksp\pc\INTERF~1\precon.c > > > > [0]PETSC ERROR: KSPSetUp() line 234 in > > > > src/ksp/ksp/interface/e:\cygwin\home\PETS > > > > C-~1.1-P\src\ksp\ksp\INTERF~1\itfunc.c > > > > [0]PETSC ERROR: KSPSolve() line 334 in > > > > src/ksp/ksp/interface/e:\cygwin\home\PETS > > > > C-~1.1-P\src\ksp\ksp\INTERF~1\itfunc.c > > > > > > > > > -----Original Message----- > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Hong Zhang > > > > > Sent: Wednesday, October 11, 2006 2:14 PM > > > > > To: petsc-users at mcs.anl.gov > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > > Julian, > > > > > > > > > > If you do not perform matrix reordering, you can use > > petsc sbaij > > > > > format, which only stores the upper triangular entries. > > > > > aij format has more algorithmic options and is more > > efficient and > > > > > supports matrix reordering. > > > > > > > > > > Hong > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > I'm using '-ksp_type cg' now to solve the symmetric > > > > > matric...But then > > > > > > do I need to store all the non-zero elements of a symmetric > > > > > matrix ? > > > > > > i.e. A[i,j] as well as A[j,i] ? > > > > > > > > > > > > Shouldn't I be able to store just the upper or lower > > > > > triangle or the > > > > > > matrix ? Is that possible with petsc ? > > > > > > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Julian > > > > > > > Sent: Wednesday, October 11, 2006 12:37 PM > > > > > > > To: petsc-users at mcs.anl.gov > > > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > Barry, > > > > > > > > > > > > > > I tried sending the commands from a file for now... And > > > > > once I used > > > > > > > '-ksp_type cg', I got pretty much the same > > > performance as that > > > > > > > of the inhouse code ! > > > > > > > Now, I'll try the performance with larger cases... > > > > > > > > > > > > > > Thanks for your help! > > > > > > > > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry > > > > > > > > Smith > > > > > > > > Sent: Wednesday, October 11, 2006 11:25 AM > > > > > > > > To: petsc-users at mcs.anl.gov > > > > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > > > > > > > > > > > PetscOptionsSetValue() but recommend putting them on > > > > > the command > > > > > > > > line or in a file (pass the name of the file into > > > > > > > PetscInitialize()). > > > > > > > > Having to recompile for every option is too painful. > > > > > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > > > > > Hong, > > > > > > > > > > > > > > > > > > The times are for the solution process alone and the > > > > > > > > initial guess is > > > > > > > > > the same, i.e. zero. > > > > > > > > > > > > > > > > > > As for the algorithm, they are probably different. > > > > > > > > > Can you tell me how to pass 'runtime options' like -help > > > > > > > > and -ksp_view > > > > > > > > > from within the code... I mean, at compile time. > > > > > > > > > > > > > > > > > > thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Julian, > > > > > > > > > > > > > > > > > > > > When comparing the inhouse solver and petsc > > solver, you > > > > > > > need make > > > > > > > > > > sure > > > > > > > > > > > > > > > > > > > > 1. Timings are collected for solution process. > > > The matrix > > > > > > > > and vector > > > > > > > > > > assembly should be excluded. > > > > > > > > > > 2. They should use same iterative algorithm. > > By default, > > > > > > > > petsc uses > > > > > > > > > > gmres with restart=30 and ilu(0) preconditioner. > > > > > > > Petsc supports > > > > > > > > > > symmetric matrices, e.g., runtime option > > > '-ksp_type cg > > > > > > > > -pc_type > > > > > > > > > > icc' > > > > > > > > > > might give better performance 3. They should > > > start from > > > > > > > > > > same intial guess. By default, petsc > > > > > > > > > > initial guess is zero. > > > > > > > > > > > > > > > > > > > > You can use '-ksp_view' to see what algorithm > > > and options > > > > > > > > are used > > > > > > > > > > in petsc. > > > > > > > > > > > > > > > > > > > > Hong > > > > > > > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > I implemented the iterative sparse matrix solver in > > > > > > > > > > > PetSc > > > > > > > > > > into my FEM > > > > > > > > > > > code recently. I compared the results from a problem > > > > > > > with 1317 > > > > > > > > > > > unknowns. I used a direct solver to obtain > > > the reference > > > > > > > > > > solution. I > > > > > > > > > > > have another in-house sparse iterative > > solver that I > > > > > > > > > > > have > > > > > > > > > > been using > > > > > > > > > > > so far. It was written by someone else but I have > > > > > access to > > > > > > > > > > the source for that solver. > > > > > > > > > > > I find the 'error norm' in the solution by > > taking the > > > > > > > > > > square root of > > > > > > > > > > > the sum of the squares of the absolute differences > > > > > > > between the > > > > > > > > > > > solution from the direct solver and the iterative > > > > > > > solver. I am > > > > > > > > > > > ignoring the numerical zeros in the solutions > > > > > when doing this. > > > > > > > > > > > I find that in order to get same order of the error > > > > > > > > norm (1e-13) > > > > > > > > > > > as the in-house iterative solver, the petsc solver > > > > > > > takes a much > > > > > > > > > > > longer time and larger number of iterations. While > > > > > > > the inhouse > > > > > > > > > > > solver took less than one second, the petsc > > > > > solver took 13 > > > > > > > > > > > seconds. The inhouse solver took 476 > > > iterations whereas > > > > > > > > the petsc > > > > > > > > > > > solver took > > > > > > > > > > 4738 iterations. > > > > > > > > > > > I'm guessing this has to do with different > > setting of > > > > > > > > the solver > > > > > > > > > > > in petsc such as the preconditioner etc. > > > > > > > > > > > Can you tell me what the different settings are? > > > > > And how to > > > > > > > > > > tweak them > > > > > > > > > > > so that I can atleast get as good as a > > performance as > > > > > > > > > > > the > > > > > > > > > > inhouse code ? > > > > > > > > > > > Given below is how I have implemented the > > > petsc solver: > > > > > > > > > > > > > > > > > > > > > > /////initialization > > > > > > > > > > > PetscInitializeNoArguments(); > > > > > > > > > > > Assert( mat = > > (double*)malloc(sizeof(Mat)) ); > > > > > > > > > > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, > > > > > > > > > > > PETSC_DEFAULT, PETSC_NULL, (Mat*)mat); > > > > > > > > > > > > > > > > > > > > > > ////// this is the function I use to populate > > > the matrix > > > > > > > > > > > MatSetValue(*(Mat*)mat, ii, jj, value, ADD_VALUES); > > > > > > > > > > > > > > > > > > > > > > ////// this is how I actaully solve the matrix > > > > > > > > > > > MatAssemblyBegin(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > > > > > > > > MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); > > > > > > > > > > > > > > > > > > > > > > double iter_error = 1e-10; > > > > > > > > > > > int max_iter_num = 10000; > > > > > > > > > > > int num_of_iter; > > > > > > > > > > > > > > > > > > > > > > Vec rhs, x; > > > > > > > > > > > VecCreateSeqWithArray(PETSC_COMM_SELF, L, > > > b, &rhs); > > > > > > > > > > > VecDuplicate(rhs, &x); > > > > > > > > > > > > > > > > > > > > > > KSP ksp; > > > > > > > > > > > KSPCreate(PETSC_COMM_SELF, &ksp); > > > > > > > > > > > KSPSetTolerances(ksp, iter_error, PETSC_DEFAULT, > > > > > > > > > > > PETSC_DEFAULT, max_iter_num); > > > > > > > > > > > KSPSetFromOptions(ksp); > > > > > > > > > > > KSPSetOperators(ksp, *(Mat*)mat, *(Mat*)mat, > > > > > > > > > > > SAME_PRECONDITIONER); > > > > > > > > > > > KSPSolve(ksp,rhs,x); > > > > > > > > > > > > > > > > > > > > > > PetscReal r_norm; > > > > > > > > > > > KSPGetResidualNorm(ksp, &r_norm); > > > > > > > > > > > KSPGetIterationNumber(ksp, &num_of_iter); > > > > > > > > > > > > > > > > > > > > > > cout << "max_iter_num\t" << > > > max_iter_num << endl; > > > > > > > > > > > cout << "iter_error\t" << > > iter_error << endl; > > > > > > > > > > > > > > > > > > > > > > cout << "Matrix solver step " << > > > num_of_iter << > > > > > > > > > > > ", > > > > > > > > > > residual " > > > > > > > > > > > << r_norm << ".\n"; > > > > > > > > > > > > > > > > > > > > > > PetscScalar *p; > > > > > > > > > > > VecGetArray(x, &p); > > > > > > > > > > > for(int i=0; i > > > > > > > > > > b[i] = p[i]; > > > > > > > > > > > } > > > > > > > > > > > VecRestoreArray(x, &p); > > > > > > > > > > > > > > > > > > > > > > KSPDestroy(ksp); > > > > > > > > > > > VecDestroy(rhs); > > > > > > > > > > > VecDestroy(x); > > > > > > > > > > > > > > > > > > > > > > cout <<"Iterations for convergence="<< > > > num_of_iter << " > > > > > > > > > > > - > > > > > > > > > > Residual Norm = " > > > > > > > > > > > << r_norm << endl; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If this is not the typical method to be > > used to solve > > > > > > > > this kind of > > > > > > > > > > > problem, please let me know what functions I > > > should use. > > > > > > > > > > > I should mention that the inhouse code is for > > > symmetric > > > > > > > > > > matrices and > > > > > > > > > > > from what I understand, the petsc solver works for > > > > > > > > > > > general > > > > > > > > > > unsymmetric matrices. > > > > > > > > > > > But I think for iterative solvers, it should > > > still give > > > > > > > > around the > > > > > > > > > > > same performance. > > > > > > > > > > > I tested the solvers against some other problems as > > > > > > > well, and I > > > > > > > > > > > got the same performance.. In some cases, no > > > > > matter how many > > > > > > > > > > iterations it > > > > > > > > > > > goes through, the petsc solver would not go below > > > > > a certain > > > > > > > > > > error norm > > > > > > > > > > > whereas the inhouse solver would get almost > > > exactly the > > > > > > > > > > same answer as > > > > > > > > > > > the direct solver solution. I'm thinking the petsc > > > > > > > > solver should > > > > > > > > > > > be able to solve this problem just as easily. > > > It would > > > > > > > > > > > be > > > > > > > > > > great if anyone > > > > > > > > > > > could help me figure out the appropriate settings I > > > > > > > > > > > should > > > > > > > > > > use in the petsc solver. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From julvar at tamu.edu Thu Oct 12 17:03:04 2006 From: julvar at tamu.edu (Julian) Date: Thu, 12 Oct 2006 17:03:04 -0500 Subject: General matrix interative solver In-Reply-To: Message-ID: <00d801c6ee4a$30ef5660$24b75ba5@aero.ad.tamu.edu> Ok, so it looks like it is not preallocating.... This is what is get when I do a '-info': [0] PetscInitializePETSc successfully started: number of processors = 1 [0] PetscGetHostNameRejecting domainname, likely is NIS ALPHA2.????????????????? ???????????????????????????????????????????????????????????????????????????? ???? ????????????????????????????????????????????????M [0] PetscInitializeRunning on machine: ALPHA2 [0] PetscCommDuplicateDuplicating a communicator 1 1 max tags = 100000000 [0] MatSetUpPreallocationWarning not preallocating matrix storage This is what I have been doing: PetscInitialize(0,0,"petscOptions.txt",0); int i; int *nnz; nnz=new int[L]; for(i=0;i -----Original Message----- > From: owner-petsc-users at mcs.anl.gov > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Satish Balay > Sent: Thursday, October 12, 2006 2:33 PM > To: petsc-users at mcs.anl.gov > Subject: RE: General matrix interative solver > > On Thu, 12 Oct 2006, Julian wrote: > > > I think I got the preallocation right.. But it still gets stuck at > > pretty much the same spot. > > Can you veify with '-info' - and look for number of mallocs > in the verbose output. > > Satish > > When it slowed to almost pretty much a full stop, I did a > break all in > > debug mode and it looks like its stuck in a memset function which > > according to the call stack is called down the line from > the following petsc function: > > PetscErrorCode PETSC_DLLEXPORT PetscMallocAlign(size_t mem,int > > line,const char func[],const char file[],const char dir[],void** > > result) > > > > I am using the symmetric matrix (MATSBAIJ) ..and I'm not sure how I > > should specify the nnz when I create the matrix Is it the number of > > non-zeroes in each row in the upper triangle (cos that is all I'm > > storing, right?) or is it the non zeroes in each row for the whole > > matrix? > > > > Julian. > > > > > -----Original Message----- > > > From: owner-petsc-users at mcs.anl.gov > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Julian > > > Sent: Thursday, October 12, 2006 1:47 PM > > > To: petsc-users at mcs.anl.gov > > > Subject: RE: General matrix interative solver > > > > > > Thanks, I got it working after I switched to icc > > > > > > All this time I was running test cases where I just read in an > > > already assembled global stiffness matrix and load vector > and solved > > > the problem using petsc. > > > > > > After I got that working, I tried to fully integrate the solver > > > with the FEA program. > > > One problem I had was with populating the matrix. The > interfaces to > > > the other existing solvers in our code were written so > that we can > > > assign a value directly to a memory space assigned for an > element in > > > the matrix. > > > Something like mat(I,J)=value; Where mat(I,J) gives the > reference to > > > the location. But petsc does not work that way (I had > already asked > > > this on the mailing list a couple of months back) and I > understand > > > maybe its wise to do that, considering it handles parallel > > > implementation as well. So, I changed our code to work with petsc. > > > > > > Now, when assembling the global matrix, sometimes I need to use > > > ADD_VALUES and other times INSERT_VALUES (depending on > > > constraints) > > > At first, I used MatAssemblyBegin(*(Mat*)mat, > > > MAT_FINAL_ASSEMBLY); > > > MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); But then > it took too > > > long and I realized I should be using > MAT_FLUSH_ASSEMBLY. But even > > > then it takes considerably more time in assembling than my other > > > solvers. So, I used a flag to call the assembly only when I am > > > switching between ADD_VALUES and INSERT_VALUES. That reduced the > > > calls to MAT_FLUSH_ASSEMBLY but it still was a slower > than the other > > > solvers.. And this was for a small problem (1317 > unknowns) I tried > > > it out with a larger problem ~15000 unknowns and the program just > > > got stuck during the assembly process ! It wouldn't crash > but looks > > > like its doing something inside of petsc... probably memory > > > allocations. So, I think I need to start preallocating > memory. I'm > > > working on that now... So hopefully, after that the > assembly process > > > won't slow down bcause of memory allocation. But my question for > > > now... is using the flag method to call assembly only when needed > > > the most efficient way to do this? Or do you have any > better ideas? > > > One other possiblity is to have an array of flags, so as to avoid > > > the call to INSERT_VALUES altogether... And thereby > eliminating the > > > need to call MAT_FLUSH_ASSEMBLY. > > > I'll get back to you after I get the preallocation right. > > > Maybe that will fix everything. > > > > > > Thanks, > > > Julian. > > > > > > > > > > -----Original Message----- > > > > From: owner-petsc-users at mcs.anl.gov > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry Smith > > > > Sent: Wednesday, October 11, 2006 2:58 PM > > > > To: petsc-users at mcs.anl.gov > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > ILU is for general matrices; for symmetric you need to use > > > > PCSetType(pc,PCICC); or -pc_type icc > > > > > > > > Barry > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > Hong, > > > > > I made the following change to specify it to be in sbaij > > > > format... And > > > > > then proceeded to store only the upper triangle. > > > > > > > > > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > > > > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, PETSC_DEFAULT, > > > > PETSC_NULL, > > > > > (Mat*)mat); > > > > > MatSetType(*(Mat*)mat,MATSBAIJ); > > > > > > > > > > But I get the following error when I try to solve: > > > > > > > > > > > > > > > > > > -------------------------------------------------------------------- > > > -- > > > > > -- Petsc Release Version 2.3.1, Patch 16, Fri Jul 21 > > > > 15:09:15 CDT 2006 > > > > > HG > > > > > revision: > > > > > 5bc424fae3770001f0d316695a9956fde3bc58b0 > > > > > See docs/changes/index.html for recent updates. > > > > > See docs/faq.html for hints about trouble shooting. > > > > > See docs/index.html for manual pages. > > > > > > > > > > > > > -------------------------------------------------------------------- > > > -- > > > > > -- Unknown Name on a cygwin-cx named ALPHA2 by > j0v1008 Wed Oct > > > > > 11 14:51:00 2006 Libraries linked from > > > > > /cygdrive/e/cygwin/home/petsc-2.3.1-p16/lib/cygwin-cxx-rea > > > > > l-debug > > > > > Configure run at Mon Aug 14 15:35:57 2006 Configure options > > > > > --with-cc="win32fe cl --nodetect" --with-cxx="win32fe cl > > > > --nod etect" > > > > > --COPTFLAGS="-MDd -Z7" --with-fc=0 --with-clanguage=cxx > > > > > --download-c-blas > > > > > -lapack=1 --with-mpi=0 --useThreads=0 --with-shared=0 > > > > > > > > > > > > > -------------------------------------------------------------------- > > > -- > > > > > -- [0]PETSC ERROR: MatILUFactorSymbolic() line 4573 in > > > > > src/mat/interface/e:\cygwin\ > > > > > home\PETSC-~1.1-P\src\mat\INTERF~1\matrix.c > > > > > [0]PETSC ERROR: No support for this operation for this > > > object type! > > > > > [0]PETSC ERROR: Matrix type sbaij symbolic ILU! > > > > > [0]PETSC ERROR: PCSetUp_ILU() line 532 in > > > > > src/ksp/pc/impls/factor/ilu/e:\cygwin\ > > > > > home\PETSC-~1.1-P\src\ksp\pc\impls\factor\ilu\ilu.c > > > > > [0]PETSC ERROR: PCSetUp() line 798 in > > > > > src/ksp/pc/interface/e:\cygwin\home\PETSC- > > > > > ~1.1-P\src\ksp\pc\INTERF~1\precon.c > > > > > [0]PETSC ERROR: KSPSetUp() line 234 in > > > > > src/ksp/ksp/interface/e:\cygwin\home\PETS > > > > > C-~1.1-P\src\ksp\ksp\INTERF~1\itfunc.c > > > > > [0]PETSC ERROR: KSPSolve() line 334 in > > > > > src/ksp/ksp/interface/e:\cygwin\home\PETS > > > > > C-~1.1-P\src\ksp\ksp\INTERF~1\itfunc.c > > > > > > > > > > > -----Original Message----- > > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of > Hong Zhang > > > > > > Sent: Wednesday, October 11, 2006 2:14 PM > > > > > > To: petsc-users at mcs.anl.gov > > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > > > > > Julian, > > > > > > > > > > > > If you do not perform matrix reordering, you can use > > > petsc sbaij > > > > > > format, which only stores the upper triangular entries. > > > > > > aij format has more algorithmic options and is more > > > efficient and > > > > > > supports matrix reordering. > > > > > > > > > > > > Hong > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > I'm using '-ksp_type cg' now to solve the symmetric > > > > > > matric...But then > > > > > > > do I need to store all the non-zero elements of a > symmetric > > > > > > matrix ? > > > > > > > i.e. A[i,j] as well as A[j,i] ? > > > > > > > > > > > > > > Shouldn't I be able to store just the upper or lower > > > > > > triangle or the > > > > > > > matrix ? Is that possible with petsc ? > > > > > > > > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On > Behalf Of Julian > > > > > > > > Sent: Wednesday, October 11, 2006 12:37 PM > > > > > > > > To: petsc-users at mcs.anl.gov > > > > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > > > Barry, > > > > > > > > > > > > > > > > I tried sending the commands from a file for now... And > > > > > > once I used > > > > > > > > '-ksp_type cg', I got pretty much the same > > > > performance as that > > > > > > > > of the inhouse code ! > > > > > > > > Now, I'll try the performance with larger cases... > > > > > > > > > > > > > > > > Thanks for your help! > > > > > > > > > > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of > > > > > > > > > Barry Smith > > > > > > > > > Sent: Wednesday, October 11, 2006 11:25 AM > > > > > > > > > To: petsc-users at mcs.anl.gov > > > > > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > > > > > > > > > > > > > > PetscOptionsSetValue() but recommend putting them on > > > > > > the command > > > > > > > > > line or in a file (pass the name of the file into > > > > > > > > PetscInitialize()). > > > > > > > > > Having to recompile for every option is too painful. > > > > > > > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > > > > > > > Hong, > > > > > > > > > > > > > > > > > > > > The times are for the solution process alone and the > > > > > > > > > initial guess is > > > > > > > > > > the same, i.e. zero. > > > > > > > > > > > > > > > > > > > > As for the algorithm, they are probably different. > > > > > > > > > > Can you tell me how to pass 'runtime options' like > > > > > > > > > > -help > > > > > > > > > and -ksp_view > > > > > > > > > > from within the code... I mean, at compile time. > > > > > > > > > > > > > > > > > > > > thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Julian, > > > > > > > > > > > > > > > > > > > > > > When comparing the inhouse solver and petsc > > > solver, you > > > > > > > > need make > > > > > > > > > > > sure > > > > > > > > > > > > > > > > > > > > > > 1. Timings are collected for solution process. > > > > The matrix > > > > > > > > > and vector > > > > > > > > > > > assembly should be excluded. > > > > > > > > > > > 2. They should use same iterative algorithm. > > > By default, > > > > > > > > > petsc uses > > > > > > > > > > > gmres with restart=30 and ilu(0) > preconditioner. > > > > > > > > Petsc supports > > > > > > > > > > > symmetric matrices, e.g., runtime option > > > > '-ksp_type cg > > > > > > > > > -pc_type > > > > > > > > > > > icc' > > > > > > > > > > > might give better performance 3. They should > > > > start from > > > > > > > > > > > same intial guess. By default, petsc > > > > > > > > > > > initial guess is zero. > > > > > > > > > > > > > > > > > > > > > > You can use '-ksp_view' to see what algorithm > > > > and options > > > > > > > > > are used > > > > > > > > > > > in petsc. > > > > > > > > > > > > > > > > > > > > > > Hong > > > > > > > > > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > > > I implemented the iterative sparse > matrix solver > > > > > > > > > > > > in PetSc > > > > > > > > > > > into my FEM > > > > > > > > > > > > code recently. I compared the results from a > > > > > > > > > > > > problem > > > > > > > > with 1317 > > > > > > > > > > > > unknowns. I used a direct solver to obtain > > > > the reference > > > > > > > > > > > solution. I > > > > > > > > > > > > have another in-house sparse iterative > > > solver that I > > > > > > > > > > > > have > > > > > > > > > > > been using > > > > > > > > > > > > so far. It was written by someone else > but I have > > > > > > access to > > > > > > > > > > > the source for that solver. > > > > > > > > > > > > I find the 'error norm' in the solution by > > > taking the > > > > > > > > > > > square root of > > > > > > > > > > > > the sum of the squares of the absolute > differences > > > > > > > > between the > > > > > > > > > > > > solution from the direct solver and the > iterative > > > > > > > > solver. I am > > > > > > > > > > > > ignoring the numerical zeros in the solutions > > > > > > when doing this. > > > > > > > > > > > > I find that in order to get same order of the > > > > > > > > > > > > error > > > > > > > > > norm (1e-13) > > > > > > > > > > > > as the in-house iterative solver, the > petsc solver > > > > > > > > takes a much > > > > > > > > > > > > longer time and larger number of > iterations. While > > > > > > > > the inhouse > > > > > > > > > > > > solver took less than one second, the petsc > > > > > > solver took 13 > > > > > > > > > > > > seconds. The inhouse solver took 476 > > > > iterations whereas > > > > > > > > > the petsc > > > > > > > > > > > > solver took > > > > > > > > > > > 4738 iterations. > > > > > > > > > > > > I'm guessing this has to do with different > > > setting of > > > > > > > > > the solver > > > > > > > > > > > > in petsc such as the preconditioner etc. > > > > > > > > > > > > Can you tell me what the different > settings are? > > > > > > And how to > > > > > > > > > > > tweak them > > > > > > > > > > > > so that I can atleast get as good as a > > > performance as > > > > > > > > > > > > the > > > > > > > > > > > inhouse code ? > > > > > > > > > > > > Given below is how I have implemented the > > > > petsc solver: > > > > > > > > > > > > > > > > > > > > > > > > /////initialization > > > > > > > > > > > > PetscInitializeNoArguments(); > > > > > > > > > > > > Assert( mat = > > > (double*)malloc(sizeof(Mat)) ); > > > > > > > > > > > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, > > > > > > > > > > > > PETSC_DEFAULT, PETSC_NULL, (Mat*)mat); > > > > > > > > > > > > > > > > > > > > > > > > ////// this is the function I use to populate > > > > the matrix > > > > > > > > > > > > MatSetValue(*(Mat*)mat, ii, jj, value, > > > > > > > > > > > > ADD_VALUES); > > > > > > > > > > > > > > > > > > > > > > > > ////// this is how I actaully solve the matrix > > > > > > > > > > > > MatAssemblyBegin(*(Mat*)mat, > MAT_FINAL_ASSEMBLY); > > > > > > > > > > > > MatAssemblyEnd(*(Mat*)mat, > MAT_FINAL_ASSEMBLY); > > > > > > > > > > > > > > > > > > > > > > > > double iter_error = 1e-10; > > > > > > > > > > > > int max_iter_num = 10000; > > > > > > > > > > > > int num_of_iter; > > > > > > > > > > > > > > > > > > > > > > > > Vec rhs, x; > > > > > > > > > > > > VecCreateSeqWithArray(PETSC_COMM_SELF, L, > > > > b, &rhs); > > > > > > > > > > > > VecDuplicate(rhs, &x); > > > > > > > > > > > > > > > > > > > > > > > > KSP ksp; > > > > > > > > > > > > KSPCreate(PETSC_COMM_SELF, &ksp); > > > > > > > > > > > > KSPSetTolerances(ksp, iter_error, > PETSC_DEFAULT, > > > > > > > > > > > > PETSC_DEFAULT, max_iter_num); > > > > > > > > > > > > KSPSetFromOptions(ksp); > > > > > > > > > > > > KSPSetOperators(ksp, *(Mat*)mat, *(Mat*)mat, > > > > > > > > > > > > SAME_PRECONDITIONER); > > > > > > > > > > > > KSPSolve(ksp,rhs,x); > > > > > > > > > > > > > > > > > > > > > > > > PetscReal r_norm; > > > > > > > > > > > > KSPGetResidualNorm(ksp, &r_norm); > > > > > > > > > > > > KSPGetIterationNumber(ksp, &num_of_iter); > > > > > > > > > > > > > > > > > > > > > > > > cout << "max_iter_num\t" << > > > > max_iter_num << endl; > > > > > > > > > > > > cout << "iter_error\t" << > > > iter_error << endl; > > > > > > > > > > > > > > > > > > > > > > > > cout << "Matrix solver step " << > > > > num_of_iter << > > > > > > > > > > > > ", > > > > > > > > > > > residual " > > > > > > > > > > > > << r_norm << ".\n"; > > > > > > > > > > > > > > > > > > > > > > > > PetscScalar *p; > > > > > > > > > > > > VecGetArray(x, &p); > > > > > > > > > > > > for(int i=0; i > > > > > > > > > > > b[i] = p[i]; > > > > > > > > > > > > } > > > > > > > > > > > > VecRestoreArray(x, &p); > > > > > > > > > > > > > > > > > > > > > > > > KSPDestroy(ksp); > > > > > > > > > > > > VecDestroy(rhs); > > > > > > > > > > > > VecDestroy(x); > > > > > > > > > > > > > > > > > > > > > > > > cout <<"Iterations for convergence="<< > > > > num_of_iter << " > > > > > > > > > > > > - > > > > > > > > > > > Residual Norm = " > > > > > > > > > > > > << r_norm << endl; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If this is not the typical method to be > > > used to solve > > > > > > > > > this kind of > > > > > > > > > > > > problem, please let me know what functions I > > > > should use. > > > > > > > > > > > > I should mention that the inhouse code is for > > > > symmetric > > > > > > > > > > > matrices and > > > > > > > > > > > > from what I understand, the petsc > solver works for > > > > > > > > > > > > general > > > > > > > > > > > unsymmetric matrices. > > > > > > > > > > > > But I think for iterative solvers, it should > > > > still give > > > > > > > > > around the > > > > > > > > > > > > same performance. > > > > > > > > > > > > I tested the solvers against some other > problems > > > > > > > > > > > > as > > > > > > > > well, and I > > > > > > > > > > > > got the same performance.. In some cases, no > > > > > > matter how many > > > > > > > > > > > iterations it > > > > > > > > > > > > goes through, the petsc solver would > not go below > > > > > > a certain > > > > > > > > > > > error norm > > > > > > > > > > > > whereas the inhouse solver would get almost > > > > exactly the > > > > > > > > > > > same answer as > > > > > > > > > > > > the direct solver solution. I'm > thinking the petsc > > > > > > > > > solver should > > > > > > > > > > > > be able to solve this problem just as easily. > > > > It would > > > > > > > > > > > > be > > > > > > > > > > > great if anyone > > > > > > > > > > > > could help me figure out the > appropriate settings > > > > > > > > > > > > I should > > > > > > > > > > > use in the petsc solver. > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From balay at mcs.anl.gov Thu Oct 12 17:29:10 2006 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 12 Oct 2006 17:29:10 -0500 (CDT) Subject: General matrix interative solver In-Reply-To: <00d801c6ee4a$30ef5660$24b75ba5@aero.ad.tamu.edu> References: <00d801c6ee4a$30ef5660$24b75ba5@aero.ad.tamu.edu> Message-ID: MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, PETSC_DEFAULT, nnz,(Mat*)mat); MatSetType(*(Mat*)mat,MATSBAIJ); The above is a fuzzy way of creating a SBAIJ matrix. Use: either: MatCreateSeqSBAIJ() or MatCreate() MatSetType(SBAIJ) MatSeqSBAIJSetPreallocation() What you've done is: Create SeqAIJ set Preallocation for SeqAIJ change MatType [SeqAIJ preallocation is of no use for SeqSBAIJ] Satish On Thu, 12 Oct 2006, Julian wrote: > Ok, so it looks like it is not preallocating.... This is what is get when I > do a '-info': > > [0] PetscInitializePETSc successfully started: number of processors = 1 > [0] PetscGetHostNameRejecting domainname, likely is NIS > ALPHA2.????????????????? > ???????????????????????????????????????????????????????????????????????????? > ???? > ????????????????????????????????????????????????M > [0] PetscInitializeRunning on machine: ALPHA2 > [0] PetscCommDuplicateDuplicating a communicator 1 1 max tags = 100000000 > [0] MatSetUpPreallocationWarning not preallocating matrix storage > > > This is what I have been doing: > > PetscInitialize(0,0,"petscOptions.txt",0); > > int i; > int *nnz; > nnz=new int[L]; > for(i=0;i nnz[i]=row[i].getNum(); > } > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, PETSC_DEFAULT, nnz, > (Mat*)mat); > > MatSetType(*(Mat*)mat,MATSBAIJ); > delete [] nnz; > > Is this not the right method to do preallocate? > > Julian. > > > -----Original Message----- > > From: owner-petsc-users at mcs.anl.gov > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Satish Balay > > Sent: Thursday, October 12, 2006 2:33 PM > > To: petsc-users at mcs.anl.gov > > Subject: RE: General matrix interative solver > > > > On Thu, 12 Oct 2006, Julian wrote: > > > > > I think I got the preallocation right.. But it still gets stuck at > > > pretty much the same spot. > > > > Can you veify with '-info' - and look for number of mallocs > > in the verbose output. > > > > Satish > > > When it slowed to almost pretty much a full stop, I did a > > break all in > > > debug mode and it looks like its stuck in a memset function which > > > according to the call stack is called down the line from > > the following petsc function: > > > PetscErrorCode PETSC_DLLEXPORT PetscMallocAlign(size_t mem,int > > > line,const char func[],const char file[],const char dir[],void** > > > result) > > > > > > I am using the symmetric matrix (MATSBAIJ) ..and I'm not sure how I > > > should specify the nnz when I create the matrix Is it the number of > > > non-zeroes in each row in the upper triangle (cos that is all I'm > > > storing, right?) or is it the non zeroes in each row for the whole > > > matrix? > > > > > > Julian. > > > > > > > -----Original Message----- > > > > From: owner-petsc-users at mcs.anl.gov > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Julian > > > > Sent: Thursday, October 12, 2006 1:47 PM > > > > To: petsc-users at mcs.anl.gov > > > > Subject: RE: General matrix interative solver > > > > > > > > Thanks, I got it working after I switched to icc > > > > > > > > All this time I was running test cases where I just read in an > > > > already assembled global stiffness matrix and load vector > > and solved > > > > the problem using petsc. > > > > > > > > After I got that working, I tried to fully integrate the solver > > > > with the FEA program. > > > > One problem I had was with populating the matrix. The > > interfaces to > > > > the other existing solvers in our code were written so > > that we can > > > > assign a value directly to a memory space assigned for an > > element in > > > > the matrix. > > > > Something like mat(I,J)=value; Where mat(I,J) gives the > > reference to > > > > the location. But petsc does not work that way (I had > > already asked > > > > this on the mailing list a couple of months back) and I > > understand > > > > maybe its wise to do that, considering it handles parallel > > > > implementation as well. So, I changed our code to work with petsc. > > > > > > > > Now, when assembling the global matrix, sometimes I need to use > > > > ADD_VALUES and other times INSERT_VALUES (depending on > > > > constraints) > > > > At first, I used MatAssemblyBegin(*(Mat*)mat, > > > > MAT_FINAL_ASSEMBLY); > > > > MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); But then > > it took too > > > > long and I realized I should be using > > MAT_FLUSH_ASSEMBLY. But even > > > > then it takes considerably more time in assembling than my other > > > > solvers. So, I used a flag to call the assembly only when I am > > > > switching between ADD_VALUES and INSERT_VALUES. That reduced the > > > > calls to MAT_FLUSH_ASSEMBLY but it still was a slower > > than the other > > > > solvers.. And this was for a small problem (1317 > > unknowns) I tried > > > > it out with a larger problem ~15000 unknowns and the program just > > > > got stuck during the assembly process ! It wouldn't crash > > but looks > > > > like its doing something inside of petsc... probably memory > > > > allocations. So, I think I need to start preallocating > > memory. I'm > > > > working on that now... So hopefully, after that the > > assembly process > > > > won't slow down bcause of memory allocation. But my question for > > > > now... is using the flag method to call assembly only when needed > > > > the most efficient way to do this? Or do you have any > > better ideas? > > > > One other possiblity is to have an array of flags, so as to avoid > > > > the call to INSERT_VALUES altogether... And thereby > > eliminating the > > > > need to call MAT_FLUSH_ASSEMBLY. > > > > I'll get back to you after I get the preallocation right. > > > > Maybe that will fix everything. > > > > > > > > Thanks, > > > > Julian. > > > > > > > > > > > > > -----Original Message----- > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry Smith > > > > > Sent: Wednesday, October 11, 2006 2:58 PM > > > > > To: petsc-users at mcs.anl.gov > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > > ILU is for general matrices; for symmetric you need to use > > > > > PCSetType(pc,PCICC); or -pc_type icc > > > > > > > > > > Barry > > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > Hong, > > > > > > I made the following change to specify it to be in sbaij > > > > > format... And > > > > > > then proceeded to store only the upper triangle. > > > > > > > > > > > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > > > > > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, PETSC_DEFAULT, > > > > > PETSC_NULL, > > > > > > (Mat*)mat); > > > > > > MatSetType(*(Mat*)mat,MATSBAIJ); > > > > > > > > > > > > But I get the following error when I try to solve: > > > > > > > > > > > > > > > > > > > > > > > -------------------------------------------------------------------- > > > > -- > > > > > > -- Petsc Release Version 2.3.1, Patch 16, Fri Jul 21 > > > > > 15:09:15 CDT 2006 > > > > > > HG > > > > > > revision: > > > > > > 5bc424fae3770001f0d316695a9956fde3bc58b0 > > > > > > See docs/changes/index.html for recent updates. > > > > > > See docs/faq.html for hints about trouble shooting. > > > > > > See docs/index.html for manual pages. > > > > > > > > > > > > > > > > > -------------------------------------------------------------------- > > > > -- > > > > > > -- Unknown Name on a cygwin-cx named ALPHA2 by > > j0v1008 Wed Oct > > > > > > 11 14:51:00 2006 Libraries linked from > > > > > > /cygdrive/e/cygwin/home/petsc-2.3.1-p16/lib/cygwin-cxx-rea > > > > > > l-debug > > > > > > Configure run at Mon Aug 14 15:35:57 2006 Configure options > > > > > > --with-cc="win32fe cl --nodetect" --with-cxx="win32fe cl > > > > > --nod etect" > > > > > > --COPTFLAGS="-MDd -Z7" --with-fc=0 --with-clanguage=cxx > > > > > > --download-c-blas > > > > > > -lapack=1 --with-mpi=0 --useThreads=0 --with-shared=0 > > > > > > > > > > > > > > > > > -------------------------------------------------------------------- > > > > -- > > > > > > -- [0]PETSC ERROR: MatILUFactorSymbolic() line 4573 in > > > > > > src/mat/interface/e:\cygwin\ > > > > > > home\PETSC-~1.1-P\src\mat\INTERF~1\matrix.c > > > > > > [0]PETSC ERROR: No support for this operation for this > > > > object type! > > > > > > [0]PETSC ERROR: Matrix type sbaij symbolic ILU! > > > > > > [0]PETSC ERROR: PCSetUp_ILU() line 532 in > > > > > > src/ksp/pc/impls/factor/ilu/e:\cygwin\ > > > > > > home\PETSC-~1.1-P\src\ksp\pc\impls\factor\ilu\ilu.c > > > > > > [0]PETSC ERROR: PCSetUp() line 798 in > > > > > > src/ksp/pc/interface/e:\cygwin\home\PETSC- > > > > > > ~1.1-P\src\ksp\pc\INTERF~1\precon.c > > > > > > [0]PETSC ERROR: KSPSetUp() line 234 in > > > > > > src/ksp/ksp/interface/e:\cygwin\home\PETS > > > > > > C-~1.1-P\src\ksp\ksp\INTERF~1\itfunc.c > > > > > > [0]PETSC ERROR: KSPSolve() line 334 in > > > > > > src/ksp/ksp/interface/e:\cygwin\home\PETS > > > > > > C-~1.1-P\src\ksp\ksp\INTERF~1\itfunc.c > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of > > Hong Zhang > > > > > > > Sent: Wednesday, October 11, 2006 2:14 PM > > > > > > > To: petsc-users at mcs.anl.gov > > > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > > > > > > > > Julian, > > > > > > > > > > > > > > If you do not perform matrix reordering, you can use > > > > petsc sbaij > > > > > > > format, which only stores the upper triangular entries. > > > > > > > aij format has more algorithmic options and is more > > > > efficient and > > > > > > > supports matrix reordering. > > > > > > > > > > > > > > Hong > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > > > I'm using '-ksp_type cg' now to solve the symmetric > > > > > > > matric...But then > > > > > > > > do I need to store all the non-zero elements of a > > symmetric > > > > > > > matrix ? > > > > > > > > i.e. A[i,j] as well as A[j,i] ? > > > > > > > > > > > > > > > > Shouldn't I be able to store just the upper or lower > > > > > > > triangle or the > > > > > > > > matrix ? Is that possible with petsc ? > > > > > > > > > > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On > > Behalf Of Julian > > > > > > > > > Sent: Wednesday, October 11, 2006 12:37 PM > > > > > > > > > To: petsc-users at mcs.anl.gov > > > > > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > > > > > Barry, > > > > > > > > > > > > > > > > > > I tried sending the commands from a file for now... And > > > > > > > once I used > > > > > > > > > '-ksp_type cg', I got pretty much the same > > > > > performance as that > > > > > > > > > of the inhouse code ! > > > > > > > > > Now, I'll try the performance with larger cases... > > > > > > > > > > > > > > > > > > Thanks for your help! > > > > > > > > > > > > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of > > > > > > > > > > Barry Smith > > > > > > > > > > Sent: Wednesday, October 11, 2006 11:25 AM > > > > > > > > > > To: petsc-users at mcs.anl.gov > > > > > > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > PetscOptionsSetValue() but recommend putting them on > > > > > > > the command > > > > > > > > > > line or in a file (pass the name of the file into > > > > > > > > > PetscInitialize()). > > > > > > > > > > Having to recompile for every option is too painful. > > > > > > > > > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > > > > > > > > > Hong, > > > > > > > > > > > > > > > > > > > > > > The times are for the solution process alone and the > > > > > > > > > > initial guess is > > > > > > > > > > > the same, i.e. zero. > > > > > > > > > > > > > > > > > > > > > > As for the algorithm, they are probably different. > > > > > > > > > > > Can you tell me how to pass 'runtime options' like > > > > > > > > > > > -help > > > > > > > > > > and -ksp_view > > > > > > > > > > > from within the code... I mean, at compile time. > > > > > > > > > > > > > > > > > > > > > > thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Julian, > > > > > > > > > > > > > > > > > > > > > > > > When comparing the inhouse solver and petsc > > > > solver, you > > > > > > > > > need make > > > > > > > > > > > > sure > > > > > > > > > > > > > > > > > > > > > > > > 1. Timings are collected for solution process. > > > > > The matrix > > > > > > > > > > and vector > > > > > > > > > > > > assembly should be excluded. > > > > > > > > > > > > 2. They should use same iterative algorithm. > > > > By default, > > > > > > > > > > petsc uses > > > > > > > > > > > > gmres with restart=30 and ilu(0) > > preconditioner. > > > > > > > > > Petsc supports > > > > > > > > > > > > symmetric matrices, e.g., runtime option > > > > > '-ksp_type cg > > > > > > > > > > -pc_type > > > > > > > > > > > > icc' > > > > > > > > > > > > might give better performance 3. They should > > > > > start from > > > > > > > > > > > > same intial guess. By default, petsc > > > > > > > > > > > > initial guess is zero. > > > > > > > > > > > > > > > > > > > > > > > > You can use '-ksp_view' to see what algorithm > > > > > and options > > > > > > > > > > are used > > > > > > > > > > > > in petsc. > > > > > > > > > > > > > > > > > > > > > > > > Hong > > > > > > > > > > > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > > > > > I implemented the iterative sparse > > matrix solver > > > > > > > > > > > > > in PetSc > > > > > > > > > > > > into my FEM > > > > > > > > > > > > > code recently. I compared the results from a > > > > > > > > > > > > > problem > > > > > > > > > with 1317 > > > > > > > > > > > > > unknowns. I used a direct solver to obtain > > > > > the reference > > > > > > > > > > > > solution. I > > > > > > > > > > > > > have another in-house sparse iterative > > > > solver that I > > > > > > > > > > > > > have > > > > > > > > > > > > been using > > > > > > > > > > > > > so far. It was written by someone else > > but I have > > > > > > > access to > > > > > > > > > > > > the source for that solver. > > > > > > > > > > > > > I find the 'error norm' in the solution by > > > > taking the > > > > > > > > > > > > square root of > > > > > > > > > > > > > the sum of the squares of the absolute > > differences > > > > > > > > > between the > > > > > > > > > > > > > solution from the direct solver and the > > iterative > > > > > > > > > solver. I am > > > > > > > > > > > > > ignoring the numerical zeros in the solutions > > > > > > > when doing this. > > > > > > > > > > > > > I find that in order to get same order of the > > > > > > > > > > > > > error > > > > > > > > > > norm (1e-13) > > > > > > > > > > > > > as the in-house iterative solver, the > > petsc solver > > > > > > > > > takes a much > > > > > > > > > > > > > longer time and larger number of > > iterations. While > > > > > > > > > the inhouse > > > > > > > > > > > > > solver took less than one second, the petsc > > > > > > > solver took 13 > > > > > > > > > > > > > seconds. The inhouse solver took 476 > > > > > iterations whereas > > > > > > > > > > the petsc > > > > > > > > > > > > > solver took > > > > > > > > > > > > 4738 iterations. > > > > > > > > > > > > > I'm guessing this has to do with different > > > > setting of > > > > > > > > > > the solver > > > > > > > > > > > > > in petsc such as the preconditioner etc. > > > > > > > > > > > > > Can you tell me what the different > > settings are? > > > > > > > And how to > > > > > > > > > > > > tweak them > > > > > > > > > > > > > so that I can atleast get as good as a > > > > performance as > > > > > > > > > > > > > the > > > > > > > > > > > > inhouse code ? > > > > > > > > > > > > > Given below is how I have implemented the > > > > > petsc solver: > > > > > > > > > > > > > > > > > > > > > > > > > > /////initialization > > > > > > > > > > > > > PetscInitializeNoArguments(); > > > > > > > > > > > > > Assert( mat = > > > > (double*)malloc(sizeof(Mat)) ); > > > > > > > > > > > > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, > > > > > > > > > > > > > PETSC_DEFAULT, PETSC_NULL, (Mat*)mat); > > > > > > > > > > > > > > > > > > > > > > > > > > ////// this is the function I use to populate > > > > > the matrix > > > > > > > > > > > > > MatSetValue(*(Mat*)mat, ii, jj, value, > > > > > > > > > > > > > ADD_VALUES); > > > > > > > > > > > > > > > > > > > > > > > > > > ////// this is how I actaully solve the matrix > > > > > > > > > > > > > MatAssemblyBegin(*(Mat*)mat, > > MAT_FINAL_ASSEMBLY); > > > > > > > > > > > > > MatAssemblyEnd(*(Mat*)mat, > > MAT_FINAL_ASSEMBLY); > > > > > > > > > > > > > > > > > > > > > > > > > > double iter_error = 1e-10; > > > > > > > > > > > > > int max_iter_num = 10000; > > > > > > > > > > > > > int num_of_iter; > > > > > > > > > > > > > > > > > > > > > > > > > > Vec rhs, x; > > > > > > > > > > > > > VecCreateSeqWithArray(PETSC_COMM_SELF, L, > > > > > b, &rhs); > > > > > > > > > > > > > VecDuplicate(rhs, &x); > > > > > > > > > > > > > > > > > > > > > > > > > > KSP ksp; > > > > > > > > > > > > > KSPCreate(PETSC_COMM_SELF, &ksp); > > > > > > > > > > > > > KSPSetTolerances(ksp, iter_error, > > PETSC_DEFAULT, > > > > > > > > > > > > > PETSC_DEFAULT, max_iter_num); > > > > > > > > > > > > > KSPSetFromOptions(ksp); > > > > > > > > > > > > > KSPSetOperators(ksp, *(Mat*)mat, *(Mat*)mat, > > > > > > > > > > > > > SAME_PRECONDITIONER); > > > > > > > > > > > > > KSPSolve(ksp,rhs,x); > > > > > > > > > > > > > > > > > > > > > > > > > > PetscReal r_norm; > > > > > > > > > > > > > KSPGetResidualNorm(ksp, &r_norm); > > > > > > > > > > > > > KSPGetIterationNumber(ksp, &num_of_iter); > > > > > > > > > > > > > > > > > > > > > > > > > > cout << "max_iter_num\t" << > > > > > max_iter_num << endl; > > > > > > > > > > > > > cout << "iter_error\t" << > > > > iter_error << endl; > > > > > > > > > > > > > > > > > > > > > > > > > > cout << "Matrix solver step " << > > > > > num_of_iter << > > > > > > > > > > > > > ", > > > > > > > > > > > > residual " > > > > > > > > > > > > > << r_norm << ".\n"; > > > > > > > > > > > > > > > > > > > > > > > > > > PetscScalar *p; > > > > > > > > > > > > > VecGetArray(x, &p); > > > > > > > > > > > > > for(int i=0; i > > > > > > > > > > > > b[i] = p[i]; > > > > > > > > > > > > > } > > > > > > > > > > > > > VecRestoreArray(x, &p); > > > > > > > > > > > > > > > > > > > > > > > > > > KSPDestroy(ksp); > > > > > > > > > > > > > VecDestroy(rhs); > > > > > > > > > > > > > VecDestroy(x); > > > > > > > > > > > > > > > > > > > > > > > > > > cout <<"Iterations for convergence="<< > > > > > num_of_iter << " > > > > > > > > > > > > > - > > > > > > > > > > > > Residual Norm = " > > > > > > > > > > > > > << r_norm << endl; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If this is not the typical method to be > > > > used to solve > > > > > > > > > > this kind of > > > > > > > > > > > > > problem, please let me know what functions I > > > > > should use. > > > > > > > > > > > > > I should mention that the inhouse code is for > > > > > symmetric > > > > > > > > > > > > matrices and > > > > > > > > > > > > > from what I understand, the petsc > > solver works for > > > > > > > > > > > > > general > > > > > > > > > > > > unsymmetric matrices. > > > > > > > > > > > > > But I think for iterative solvers, it should > > > > > still give > > > > > > > > > > around the > > > > > > > > > > > > > same performance. > > > > > > > > > > > > > I tested the solvers against some other > > problems > > > > > > > > > > > > > as > > > > > > > > > well, and I > > > > > > > > > > > > > got the same performance.. In some cases, no > > > > > > > matter how many > > > > > > > > > > > > iterations it > > > > > > > > > > > > > goes through, the petsc solver would > > not go below > > > > > > > a certain > > > > > > > > > > > > error norm > > > > > > > > > > > > > whereas the inhouse solver would get almost > > > > > exactly the > > > > > > > > > > > > same answer as > > > > > > > > > > > > > the direct solver solution. I'm > > thinking the petsc > > > > > > > > > > solver should > > > > > > > > > > > > > be able to solve this problem just as easily. > > > > > It would > > > > > > > > > > > > > be > > > > > > > > > > > > great if anyone > > > > > > > > > > > > > could help me figure out the > > appropriate settings > > > > > > > > > > > > > I should > > > > > > > > > > > > use in the petsc solver. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From julvar at tamu.edu Thu Oct 12 18:25:21 2006 From: julvar at tamu.edu (Julian) Date: Thu, 12 Oct 2006 18:25:21 -0500 Subject: General matrix interative solver In-Reply-To: Message-ID: <00d901c6ee55$affad500$24b75ba5@aero.ad.tamu.edu> I am now using MatCreateSeqSBAIJ() - I am assuming the second parameter bs=1 in my case because I am using one processor. Is that correct ? The method of creating a matrix I used was taken from Pavel Solin's book 'Partial Differential Equations and the finite Element Method'. Is there any particular reason why is should follow this method: mat = (double*)malloc(sizeof(Mat)) MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, PETSC_DEFAULT, nnz, (Mat*)mat); In the examples that I have seen in the petsc website I don't see this statement where memory is allocated. Anyway, I changed my code to the following: PetscInitialize(0,0,"petscOptions.txt",0); int i; int *nnz; nnz=new int[L]; for(i=0;i -----Original Message----- > From: owner-petsc-users at mcs.anl.gov > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Satish Balay > Sent: Thursday, October 12, 2006 5:29 PM > To: petsc-users at mcs.anl.gov > Subject: RE: General matrix interative solver > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, PETSC_DEFAULT, > nnz,(Mat*)mat); > MatSetType(*(Mat*)mat,MATSBAIJ); > > The above is a fuzzy way of creating a SBAIJ matrix. Use: either: > > MatCreateSeqSBAIJ() > > or > MatCreate() > MatSetType(SBAIJ) > MatSeqSBAIJSetPreallocation() > > What you've done is: > Create SeqAIJ > set Preallocation for SeqAIJ > change MatType [SeqAIJ preallocation is of no use for SeqSBAIJ] > > Satish > > On Thu, 12 Oct 2006, Julian wrote: > > > Ok, so it looks like it is not preallocating.... This is > what is get > > when I do a '-info': > > > > [0] PetscInitializePETSc successfully started: number of > processors = > > 1 [0] PetscGetHostNameRejecting domainname, likely is NIS > > ALPHA2.????????????????? > > > ?????????????????????????????????????????????????????????????? > ?????????????? > > ???? > > ????????????????????????????????????????????????M > > [0] PetscInitializeRunning on machine: ALPHA2 [0] > > PetscCommDuplicateDuplicating a communicator 1 1 max tags = > 100000000 > > [0] MatSetUpPreallocationWarning not preallocating matrix storage > > > > > > This is what I have been doing: > > > > PetscInitialize(0,0,"petscOptions.txt",0); > > > > int i; > > int *nnz; > > nnz=new int[L]; > > for(i=0;i > nnz[i]=row[i].getNum(); > > } > > > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, PETSC_DEFAULT, nnz, > > (Mat*)mat); > > > > MatSetType(*(Mat*)mat,MATSBAIJ); > > delete [] nnz; > > > > Is this not the right method to do preallocate? > > > > Julian. > > > > > -----Original Message----- > > > From: owner-petsc-users at mcs.anl.gov > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Satish Balay > > > Sent: Thursday, October 12, 2006 2:33 PM > > > To: petsc-users at mcs.anl.gov > > > Subject: RE: General matrix interative solver > > > > > > On Thu, 12 Oct 2006, Julian wrote: > > > > > > > I think I got the preallocation right.. But it still > gets stuck at > > > > pretty much the same spot. > > > > > > Can you veify with '-info' - and look for number of > mallocs in the > > > verbose output. > > > > > > Satish > > > > When it slowed to almost pretty much a full stop, I did a > > > break all in > > > > debug mode and it looks like its stuck in a memset > function which > > > > according to the call stack is called down the line from > > > the following petsc function: > > > > PetscErrorCode PETSC_DLLEXPORT PetscMallocAlign(size_t mem,int > > > > line,const char func[],const char file[],const char dir[],void** > > > > result) > > > > > > > > I am using the symmetric matrix (MATSBAIJ) ..and I'm > not sure how > > > > I should specify the nnz when I create the matrix Is it > the number > > > > of non-zeroes in each row in the upper triangle (cos > that is all > > > > I'm storing, right?) or is it the non zeroes in each > row for the > > > > whole matrix? > > > > > > > > Julian. > > > > > > > > > -----Original Message----- > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Julian > > > > > Sent: Thursday, October 12, 2006 1:47 PM > > > > > To: petsc-users at mcs.anl.gov > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > Thanks, I got it working after I switched to icc > > > > > > > > > > All this time I was running test cases where I just > read in an > > > > > already assembled global stiffness matrix and load vector > > > and solved > > > > > the problem using petsc. > > > > > > > > > > After I got that working, I tried to fully integrate > the solver > > > > > with the FEA program. > > > > > One problem I had was with populating the matrix. The > > > interfaces to > > > > > the other existing solvers in our code were written so > > > that we can > > > > > assign a value directly to a memory space assigned for an > > > element in > > > > > the matrix. > > > > > Something like mat(I,J)=value; Where mat(I,J) gives the > > > reference to > > > > > the location. But petsc does not work that way (I had > > > already asked > > > > > this on the mailing list a couple of months back) and I > > > understand > > > > > maybe its wise to do that, considering it handles parallel > > > > > implementation as well. So, I changed our code to > work with petsc. > > > > > > > > > > Now, when assembling the global matrix, sometimes I > need to use > > > > > ADD_VALUES and other times INSERT_VALUES (depending on > > > > > constraints) > > > > > At first, I used MatAssemblyBegin(*(Mat*)mat, > > > > > MAT_FINAL_ASSEMBLY); > > > > > MatAssemblyEnd(*(Mat*)mat, MAT_FINAL_ASSEMBLY); But then > > > it took too > > > > > long and I realized I should be using > > > MAT_FLUSH_ASSEMBLY. But even > > > > > then it takes considerably more time in assembling > than my other > > > > > solvers. So, I used a flag to call the assembly only > when I am > > > > > switching between ADD_VALUES and INSERT_VALUES. That > reduced the > > > > > calls to MAT_FLUSH_ASSEMBLY but it still was a slower > > > than the other > > > > > solvers.. And this was for a small problem (1317 > > > unknowns) I tried > > > > > it out with a larger problem ~15000 unknowns and the program > > > > > just got stuck during the assembly process ! It wouldn't crash > > > but looks > > > > > like its doing something inside of petsc... probably memory > > > > > allocations. So, I think I need to start preallocating > > > memory. I'm > > > > > working on that now... So hopefully, after that the > > > assembly process > > > > > won't slow down bcause of memory allocation. But my > question for > > > > > now... is using the flag method to call assembly only when > > > > > needed the most efficient way to do this? Or do you have any > > > better ideas? > > > > > One other possiblity is to have an array of flags, so as to > > > > > avoid the call to INSERT_VALUES altogether... And thereby > > > eliminating the > > > > > need to call MAT_FLUSH_ASSEMBLY. > > > > > I'll get back to you after I get the preallocation right. > > > > > Maybe that will fix everything. > > > > > > > > > > Thanks, > > > > > Julian. > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry > > > > > > Smith > > > > > > Sent: Wednesday, October 11, 2006 2:58 PM > > > > > > To: petsc-users at mcs.anl.gov > > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > > > > > ILU is for general matrices; for symmetric you > need to use > > > > > > PCSetType(pc,PCICC); or -pc_type icc > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > Hong, > > > > > > > I made the following change to specify it to be in sbaij > > > > > > format... And > > > > > > > then proceeded to store only the upper triangle. > > > > > > > > > > > > > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > > > > > > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, PETSC_DEFAULT, > > > > > > PETSC_NULL, > > > > > > > (Mat*)mat); > > > > > > > MatSetType(*(Mat*)mat,MATSBAIJ); > > > > > > > > > > > > > > But I get the following error when I try to solve: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------------------------------------------------------------- > > > > > -- > > > > > > > -- Petsc Release Version 2.3.1, Patch 16, Fri Jul 21 > > > > > > 15:09:15 CDT 2006 > > > > > > > HG > > > > > > > revision: > > > > > > > 5bc424fae3770001f0d316695a9956fde3bc58b0 > > > > > > > See docs/changes/index.html for recent updates. > > > > > > > See docs/faq.html for hints about trouble shooting. > > > > > > > See docs/index.html for manual pages. > > > > > > > > > > > > > > > > > > > > > > -------------------------------------------------------------------- > > > > > -- > > > > > > > -- Unknown Name on a cygwin-cx named ALPHA2 by > > > j0v1008 Wed Oct > > > > > > > 11 14:51:00 2006 Libraries linked from > > > > > > > /cygdrive/e/cygwin/home/petsc-2.3.1-p16/lib/cygwin-cxx-rea > > > > > > > l-debug > > > > > > > Configure run at Mon Aug 14 15:35:57 2006 > Configure options > > > > > > > --with-cc="win32fe cl --nodetect" --with-cxx="win32fe cl > > > > > > --nod etect" > > > > > > > --COPTFLAGS="-MDd -Z7" --with-fc=0 --with-clanguage=cxx > > > > > > > --download-c-blas > > > > > > > -lapack=1 --with-mpi=0 --useThreads=0 --with-shared=0 > > > > > > > > > > > > > > > > > > > > > > -------------------------------------------------------------------- > > > > > -- > > > > > > > -- [0]PETSC ERROR: MatILUFactorSymbolic() line 4573 in > > > > > > > src/mat/interface/e:\cygwin\ > > > > > > > home\PETSC-~1.1-P\src\mat\INTERF~1\matrix.c > > > > > > > [0]PETSC ERROR: No support for this operation for this > > > > > object type! > > > > > > > [0]PETSC ERROR: Matrix type sbaij symbolic ILU! > > > > > > > [0]PETSC ERROR: PCSetUp_ILU() line 532 in > > > > > > > src/ksp/pc/impls/factor/ilu/e:\cygwin\ > > > > > > > home\PETSC-~1.1-P\src\ksp\pc\impls\factor\ilu\ilu.c > > > > > > > [0]PETSC ERROR: PCSetUp() line 798 in > > > > > > > src/ksp/pc/interface/e:\cygwin\home\PETSC- > > > > > > > ~1.1-P\src\ksp\pc\INTERF~1\precon.c > > > > > > > [0]PETSC ERROR: KSPSetUp() line 234 in > > > > > > > src/ksp/ksp/interface/e:\cygwin\home\PETS > > > > > > > C-~1.1-P\src\ksp\ksp\INTERF~1\itfunc.c > > > > > > > [0]PETSC ERROR: KSPSolve() line 334 in > > > > > > > src/ksp/ksp/interface/e:\cygwin\home\PETS > > > > > > > C-~1.1-P\src\ksp\ksp\INTERF~1\itfunc.c > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of > > > Hong Zhang > > > > > > > > Sent: Wednesday, October 11, 2006 2:14 PM > > > > > > > > To: petsc-users at mcs.anl.gov > > > > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > > > > > > > > > > > Julian, > > > > > > > > > > > > > > > > If you do not perform matrix reordering, you can use > > > > > petsc sbaij > > > > > > > > format, which only stores the upper triangular entries. > > > > > > > > aij format has more algorithmic options and is more > > > > > efficient and > > > > > > > > supports matrix reordering. > > > > > > > > > > > > > > > > Hong > > > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > > > > > I'm using '-ksp_type cg' now to solve the symmetric > > > > > > > > matric...But then > > > > > > > > > do I need to store all the non-zero elements of a > > > symmetric > > > > > > > > matrix ? > > > > > > > > > i.e. A[i,j] as well as A[j,i] ? > > > > > > > > > > > > > > > > > > Shouldn't I be able to store just the upper or lower > > > > > > > > triangle or the > > > > > > > > > matrix ? Is that possible with petsc ? > > > > > > > > > > > > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On > > > Behalf Of Julian > > > > > > > > > > Sent: Wednesday, October 11, 2006 12:37 PM > > > > > > > > > > To: petsc-users at mcs.anl.gov > > > > > > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > > > > > > > Barry, > > > > > > > > > > > > > > > > > > > > I tried sending the commands from a file for now... > > > > > > > > > > And > > > > > > > > once I used > > > > > > > > > > '-ksp_type cg', I got pretty much the same > > > > > > performance as that > > > > > > > > > > of the inhouse code ! > > > > > > > > > > Now, I'll try the performance with larger cases... > > > > > > > > > > > > > > > > > > > > Thanks for your help! > > > > > > > > > > > > > > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > From: owner-petsc-users at mcs.anl.gov > > > > > > > > > > > [mailto:owner-petsc-users at mcs.anl.gov] On > Behalf Of > > > > > > > > > > > Barry Smith > > > > > > > > > > > Sent: Wednesday, October 11, 2006 11:25 AM > > > > > > > > > > > To: petsc-users at mcs.anl.gov > > > > > > > > > > > Subject: RE: General matrix interative solver > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > PetscOptionsSetValue() but recommend > putting them > > > > > > > > > > > on > > > > > > > > the command > > > > > > > > > > > line or in a file (pass the name of the file into > > > > > > > > > > PetscInitialize()). > > > > > > > > > > > Having to recompile for every option is > too painful. > > > > > > > > > > > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > > > > > > > > > > > Hong, > > > > > > > > > > > > > > > > > > > > > > > > The times are for the solution process > alone and > > > > > > > > > > > > the > > > > > > > > > > > initial guess is > > > > > > > > > > > > the same, i.e. zero. > > > > > > > > > > > > > > > > > > > > > > > > As for the algorithm, they are probably > different. > > > > > > > > > > > > Can you tell me how to pass 'runtime > options' like > > > > > > > > > > > > -help > > > > > > > > > > > and -ksp_view > > > > > > > > > > > > from within the code... I mean, at compile time. > > > > > > > > > > > > > > > > > > > > > > > > thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Julian, > > > > > > > > > > > > > > > > > > > > > > > > > > When comparing the inhouse solver and petsc > > > > > solver, you > > > > > > > > > > need make > > > > > > > > > > > > > sure > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Timings are collected for solution > process. > > > > > > The matrix > > > > > > > > > > > and vector > > > > > > > > > > > > > assembly should be excluded. > > > > > > > > > > > > > 2. They should use same iterative algorithm. > > > > > By default, > > > > > > > > > > > petsc uses > > > > > > > > > > > > > gmres with restart=30 and ilu(0) > > > preconditioner. > > > > > > > > > > Petsc supports > > > > > > > > > > > > > symmetric matrices, e.g., runtime option > > > > > > '-ksp_type cg > > > > > > > > > > > -pc_type > > > > > > > > > > > > > icc' > > > > > > > > > > > > > might give better performance 3. > They should > > > > > > start from > > > > > > > > > > > > > same intial guess. By default, petsc > > > > > > > > > > > > > initial guess is zero. > > > > > > > > > > > > > > > > > > > > > > > > > > You can use '-ksp_view' to see what algorithm > > > > > > and options > > > > > > > > > > > are used > > > > > > > > > > > > > in petsc. > > > > > > > > > > > > > > > > > > > > > > > > > > Hong > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, 11 Oct 2006, Julian wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > > > > > > > I implemented the iterative sparse > > > matrix solver > > > > > > > > > > > > > > in PetSc > > > > > > > > > > > > > into my FEM > > > > > > > > > > > > > > code recently. I compared the > results from a > > > > > > > > > > > > > > problem > > > > > > > > > > with 1317 > > > > > > > > > > > > > > unknowns. I used a direct solver to obtain > > > > > > the reference > > > > > > > > > > > > > solution. I > > > > > > > > > > > > > > have another in-house sparse iterative > > > > > solver that I > > > > > > > > > > > > > > have > > > > > > > > > > > > > been using > > > > > > > > > > > > > > so far. It was written by someone else > > > but I have > > > > > > > > access to > > > > > > > > > > > > > the source for that solver. > > > > > > > > > > > > > > I find the 'error norm' in the solution by > > > > > taking the > > > > > > > > > > > > > square root of > > > > > > > > > > > > > > the sum of the squares of the absolute > > > differences > > > > > > > > > > between the > > > > > > > > > > > > > > solution from the direct solver and the > > > iterative > > > > > > > > > > solver. I am > > > > > > > > > > > > > > ignoring the numerical zeros in the > solutions > > > > > > > > when doing this. > > > > > > > > > > > > > > I find that in order to get same > order of the > > > > > > > > > > > > > > error > > > > > > > > > > > norm (1e-13) > > > > > > > > > > > > > > as the in-house iterative solver, the > > > petsc solver > > > > > > > > > > takes a much > > > > > > > > > > > > > > longer time and larger number of > > > iterations. While > > > > > > > > > > the inhouse > > > > > > > > > > > > > > solver took less than one second, the petsc > > > > > > > > solver took 13 > > > > > > > > > > > > > > seconds. The inhouse solver took 476 > > > > > > iterations whereas > > > > > > > > > > > the petsc > > > > > > > > > > > > > > solver took > > > > > > > > > > > > > 4738 iterations. > > > > > > > > > > > > > > I'm guessing this has to do with different > > > > > setting of > > > > > > > > > > > the solver > > > > > > > > > > > > > > in petsc such as the preconditioner etc. > > > > > > > > > > > > > > Can you tell me what the different > > > settings are? > > > > > > > > And how to > > > > > > > > > > > > > tweak them > > > > > > > > > > > > > > so that I can atleast get as good as a > > > > > performance as > > > > > > > > > > > > > > the > > > > > > > > > > > > > inhouse code ? > > > > > > > > > > > > > > Given below is how I have implemented the > > > > > > petsc solver: > > > > > > > > > > > > > > > > > > > > > > > > > > > > /////initialization > > > > > > > > > > > > > > PetscInitializeNoArguments(); > > > > > > > > > > > > > > Assert( mat = > > > > > (double*)malloc(sizeof(Mat)) ); > > > > > > > > > > > > > > MatCreateSeqAIJ(PETSC_COMM_SELF, L, L, > > > > > > > > > > > > > > PETSC_DEFAULT, PETSC_NULL, (Mat*)mat); > > > > > > > > > > > > > > > > > > > > > > > > > > > > ////// this is the function I use > to populate > > > > > > the matrix > > > > > > > > > > > > > > MatSetValue(*(Mat*)mat, ii, jj, value, > > > > > > > > > > > > > > ADD_VALUES); > > > > > > > > > > > > > > > > > > > > > > > > > > > > ////// this is how I actaully solve > the matrix > > > > > > > > > > > > > > MatAssemblyBegin(*(Mat*)mat, > > > MAT_FINAL_ASSEMBLY); > > > > > > > > > > > > > > MatAssemblyEnd(*(Mat*)mat, > > > MAT_FINAL_ASSEMBLY); > > > > > > > > > > > > > > > > > > > > > > > > > > > > double iter_error = 1e-10; > > > > > > > > > > > > > > int max_iter_num = 10000; > > > > > > > > > > > > > > int num_of_iter; > > > > > > > > > > > > > > > > > > > > > > > > > > > > Vec rhs, x; > > > > > > > > > > > > > > > VecCreateSeqWithArray(PETSC_COMM_SELF, L, > > > > > > b, &rhs); > > > > > > > > > > > > > > VecDuplicate(rhs, &x); > > > > > > > > > > > > > > > > > > > > > > > > > > > > KSP ksp; > > > > > > > > > > > > > > KSPCreate(PETSC_COMM_SELF, &ksp); > > > > > > > > > > > > > > KSPSetTolerances(ksp, iter_error, > > > PETSC_DEFAULT, > > > > > > > > > > > > > > PETSC_DEFAULT, max_iter_num); > > > > > > > > > > > > > > KSPSetFromOptions(ksp); > > > > > > > > > > > > > > KSPSetOperators(ksp, > *(Mat*)mat, *(Mat*)mat, > > > > > > > > > > > > > > SAME_PRECONDITIONER); > > > > > > > > > > > > > > KSPSolve(ksp,rhs,x); > > > > > > > > > > > > > > > > > > > > > > > > > > > > PetscReal r_norm; > > > > > > > > > > > > > > KSPGetResidualNorm(ksp, &r_norm); > > > > > > > > > > > > > > KSPGetIterationNumber(ksp, > &num_of_iter); > > > > > > > > > > > > > > > > > > > > > > > > > > > > cout << "max_iter_num\t" << > > > > > > max_iter_num << endl; > > > > > > > > > > > > > > cout << "iter_error\t" << > > > > > iter_error << endl; > > > > > > > > > > > > > > > > > > > > > > > > > > > > cout << "Matrix solver step " << > > > > > > num_of_iter << > > > > > > > > > > > > > > ", > > > > > > > > > > > > > residual " > > > > > > > > > > > > > > << r_norm << ".\n"; > > > > > > > > > > > > > > > > > > > > > > > > > > > > PetscScalar *p; > > > > > > > > > > > > > > VecGetArray(x, &p); > > > > > > > > > > > > > > for(int i=0; i > > > > > > > > > > > > > b[i] = p[i]; > > > > > > > > > > > > > > } > > > > > > > > > > > > > > VecRestoreArray(x, &p); > > > > > > > > > > > > > > > > > > > > > > > > > > > > KSPDestroy(ksp); > > > > > > > > > > > > > > VecDestroy(rhs); > > > > > > > > > > > > > > VecDestroy(x); > > > > > > > > > > > > > > > > > > > > > > > > > > > > cout <<"Iterations for convergence="<< > > > > > > num_of_iter << " > > > > > > > > > > > > > > - > > > > > > > > > > > > > Residual Norm = " > > > > > > > > > > > > > > << r_norm << endl; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If this is not the typical method to be > > > > > used to solve > > > > > > > > > > > this kind of > > > > > > > > > > > > > > problem, please let me know what functions I > > > > > > should use. > > > > > > > > > > > > > > I should mention that the inhouse > code is for > > > > > > symmetric > > > > > > > > > > > > > matrices and > > > > > > > > > > > > > > from what I understand, the petsc > > > solver works for > > > > > > > > > > > > > > general > > > > > > > > > > > > > unsymmetric matrices. > > > > > > > > > > > > > > But I think for iterative solvers, it should > > > > > > still give > > > > > > > > > > > around the > > > > > > > > > > > > > > same performance. > > > > > > > > > > > > > > I tested the solvers against some other > > > problems > > > > > > > > > > > > > > as > > > > > > > > > > well, and I > > > > > > > > > > > > > > got the same performance.. In some cases, no > > > > > > > > matter how many > > > > > > > > > > > > > iterations it > > > > > > > > > > > > > > goes through, the petsc solver would > > > not go below > > > > > > > > a certain > > > > > > > > > > > > > error norm > > > > > > > > > > > > > > whereas the inhouse solver would get almost > > > > > > exactly the > > > > > > > > > > > > > same answer as > > > > > > > > > > > > > > the direct solver solution. I'm > > > thinking the petsc > > > > > > > > > > > solver should > > > > > > > > > > > > > > be able to solve this problem just > as easily. > > > > > > It would > > > > > > > > > > > > > > be > > > > > > > > > > > > > great if anyone > > > > > > > > > > > > > > could help me figure out the > > > appropriate settings > > > > > > > > > > > > > > I should > > > > > > > > > > > > > use in the petsc solver. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Julian. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From balay at mcs.anl.gov Thu Oct 12 18:45:46 2006 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 12 Oct 2006 18:45:46 -0500 (CDT) Subject: General matrix interative solver In-Reply-To: <00d901c6ee55$affad500$24b75ba5@aero.ad.tamu.edu> References: <00d901c6ee55$affad500$24b75ba5@aero.ad.tamu.edu> Message-ID: On Thu, 12 Oct 2006, Julian wrote: > I am now using MatCreateSeqSBAIJ() - I am assuming the second > parameter bs=1 in my case because I am using one processor. Is that > correct ? No - it refers to a blocksize. The matrix could have a block structure. In this case - the correct blocksize can be specified. The matrix operations will exploit this block structure for better performance. bs=1 is a fine [This is the equivalent of AIJ type in the block notation] > The method of creating a matrix I used was taken from Pavel Solin's > book 'Partial Differential Equations and the finite Element > Method'. Is there any particular reason why is should follow this > method: > mat = (double*)malloc(sizeof(Mat)) > MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, PETSC_DEFAULT, nnz, (Mat*)mat); > In the examples that I have seen in the petsc website I don't see > this statement where memory is allocated. No - you should just use PETSc example codes a guidelines. i.e Mat mat; MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, PETSC_DEFAULT, nnz, &mat); Mat is just a pointer - and all the memory allocation required is done inside XXXCreate() routines. > Anyway, I changed my code to the following: > PetscInitialize(0,0,"petscOptions.txt",0); > > int i; > int *nnz; > nnz=new int[L]; > for(i=0;i nnz[i]=row[i].getNum(); > } > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, PETSC_DEFAULT, nnz, (Mat*)mat); > delete [] nnz; > And I'm still getting stuck at the same place... There is a > difference though.. Now the assembly process is even slower than > before (when there was no preallocation) I noticed that it is > getting stuck around the 200th MAT_FLUSH_ASSEMBLY call.. I don't > know if that is significant.Is there some limit on how many times > MAT_FLUSH_ASSEMBLY can be called ? MAT_FLUSH_ASSEMBLY is cheap - in sequential mode - but expensive in parallel mode [becasue of parallel syncronization]. However ... > This is the petsc output I get: > [0] PetscInitializePETSc successfully started: number of processors = 1 > [0] PetscGetHostNameRejecting domainname, likely is NIS ALPHA2.??0?? > [0] PetscInitializeRunning on machine: ALPHA2 > [0] PetscCommDuplicateDuplicating a communicator 1 1 max tags = 100000000 The above log is incomplete. You should be looking for lines with the string 'malloc' Satish > > I also tried this : > mat = new Mat; > MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, PETSC_DEFAULT, nnz, mat); > > And got the same performance. From julvar at tamu.edu Thu Oct 12 19:16:24 2006 From: julvar at tamu.edu (Julian) Date: Thu, 12 Oct 2006 19:16:24 -0500 Subject: General matrix interative solver In-Reply-To: Message-ID: <00e101c6ee5c$d19c2130$24b75ba5@aero.ad.tamu.edu> Ok, so this is what I am doing now... And I still get the same result. It goes very slowly till the 200th MAT_FLUSH_ASSEMBLY call and then just stays there... The output that I gave u earlier is the complete output its been giving for the following code: Mat mat; PetscInitialize(0,0,"petscOptions.txt",0); int i; int *nnz; nnz=new int[L]; for(i=0;i -----Original Message----- > From: owner-petsc-users at mcs.anl.gov > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Satish Balay > Sent: Thursday, October 12, 2006 6:46 PM > To: petsc-users at mcs.anl.gov > Subject: RE: General matrix interative solver > > On Thu, 12 Oct 2006, Julian wrote: > > > I am now using MatCreateSeqSBAIJ() - I am assuming the second > > parameter bs=1 in my case because I am using one processor. Is that > > correct ? > > No - it refers to a blocksize. The matrix could have a block > structure. In this case - the correct blocksize can be > specified. The matrix operations will exploit this block > structure for better performance. > > bs=1 is a fine [This is the equivalent of AIJ type in the > block notation] > > > The method of creating a matrix I used was taken from Pavel Solin's > > book 'Partial Differential Equations and the finite Element > Method'. > > Is there any particular reason why is should follow this > > method: > > mat = (double*)malloc(sizeof(Mat)) > > MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, PETSC_DEFAULT, nnz, > > (Mat*)mat); In the examples that I have seen in the petsc website I > > don't see this statement where memory is allocated. > > No - you should just use PETSc example codes a guidelines. i.e > > Mat mat; > MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, PETSC_DEFAULT, > nnz, &mat); > > Mat is just a pointer - and all the memory allocation > required is done inside XXXCreate() routines. > > > Anyway, I changed my code to the following: > > PetscInitialize(0,0,"petscOptions.txt",0); > > > > int i; > > int *nnz; > > nnz=new int[L]; > > for(i=0;i > nnz[i]=row[i].getNum(); > > } > > > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > > MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, > PETSC_DEFAULT, nnz, (Mat*)mat); > > delete [] nnz; > > > And I'm still getting stuck at the same place... There is a > difference > > though.. Now the assembly process is even slower than before (when > > there was no preallocation) I noticed that it is getting > stuck around > > the 200th MAT_FLUSH_ASSEMBLY call.. I don't know if that is > > significant.Is there some limit on how many times > MAT_FLUSH_ASSEMBLY > > can be called ? > > MAT_FLUSH_ASSEMBLY is cheap - in sequential mode - but > expensive in parallel mode [becasue of parallel > syncronization]. However ... > > > This is the petsc output I get: > > [0] PetscInitializePETSc successfully started: number of > processors = > > 1 [0] PetscGetHostNameRejecting domainname, likely is NIS > ALPHA2.??0?? > > [0] PetscInitializeRunning on machine: ALPHA2 [0] > > PetscCommDuplicateDuplicating a communicator 1 1 max tags = > 100000000 > > The above log is incomplete. You should be looking for lines > with the string 'malloc' > > Satish > > > > > I also tried this : > > mat = new Mat; > > MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, > PETSC_DEFAULT, nnz, mat); > > > > And got the same performance. > From balay at mcs.anl.gov Thu Oct 12 19:32:59 2006 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 12 Oct 2006 19:32:59 -0500 (CDT) Subject: General matrix interative solver In-Reply-To: <00e101c6ee5c$d19c2130$24b75ba5@aero.ad.tamu.edu> References: <00e101c6ee5c$d19c2130$24b75ba5@aero.ad.tamu.edu> Message-ID: - calling MAT_FINAL_ASSEMBLY before inserting the values is of no use. - if the code get slow after 200 calls => your preallocation ran out at that time. i.e incorrect preallocation. - You can call MatSetOption(mat,MAT_NEW_NONZERO_ALLOCATION_ERR) to verify [If an insertion causes a malloc - the above will flag it as error]. You can then run in a debugger [By placing a breakpint in PetscError() - and then checking - why the specified preallocation info is not sufficient] - You might want to rethink your MatSetValues code. Eventhough its ok right now to have that many calls to MAT_FLUSH_ASSEMBLY, the code could probably be organized to avoid this. - This thread is getting pretty long on the petsc-users mailing list. If you still have issues - please continue this discussion at petsc-maint at mcs.anl.gov Satish On Thu, 12 Oct 2006, Julian wrote: > Ok, so this is what I am doing now... And I still get the same result. It goes very slowly till the 200th MAT_FLUSH_ASSEMBLY call and then just stays there... The output that I gave u earlier is the complete output its been giving for the following code: > > Mat mat; > PetscInitialize(0,0,"petscOptions.txt",0); > int i; > int *nnz; > nnz=new int[L]; > for(i=0;i nnz[i]=row[i].getNum(); > } > > MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, PETSC_DEFAULT, nnz, &mat); > delete [] nnz; > > Maybe it is not giving the output you are requiring because the program never gets to the point where MAT_FINAL_ASSEMBLY is called. > > Now, when I do a MAT_FINAL_ASSEMBLY call right after the preallocation, then I get a different output > > Mat mat; > PetscInitialize(0,0,"petscOptions.txt",0); > int i; > int *nnz; > nnz=new int[L]; > for(i=0;i nnz[i]=row[i].getNum(); > } > > MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, PETSC_DEFAULT, nnz, &mat); > > MatAssemblyBegin(mat, MAT_FINAL_ASSEMBLY); // <----------------- > MatAssemblyEnd(mat, MAT_FINAL_ASSEMBLY); // <----------------- > > delete [] nnz; > > This is what I get : > [0] PetscInitializePETSc successfully started: number of processors = 1 > [0] PetscGetHostNameRejecting domainname, likely is NIS ALPHA2.??0?? > [0] PetscInitializeRunning on machine: ALPHA2 > [0] PetscCommDuplicateDuplicating a communicator 1 1 max tags = 100000000 > [0] MatAssemblyEnd_SeqSBAIJMatrix size: 14883 X 14883, block size 1; storage spa > ce: 1040055 unneeded, 0 used > [0] MatAssemblyEnd_SeqSBAIJNumber of mallocs during MatSetValues is 0 > [0] MatAssemblyEnd_SeqSBAIJMost nonzeros blocks in any row is 0 > > In this case, the assembly goes very fast TILL the 200th MAT_FLUSH_ASSEMBLY call and THEN gets stuck! > > > > > -----Original Message----- > > From: owner-petsc-users at mcs.anl.gov > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Satish Balay > > Sent: Thursday, October 12, 2006 6:46 PM > > To: petsc-users at mcs.anl.gov > > Subject: RE: General matrix interative solver > > > > On Thu, 12 Oct 2006, Julian wrote: > > > > > I am now using MatCreateSeqSBAIJ() - I am assuming the second > > > parameter bs=1 in my case because I am using one processor. Is that > > > correct ? > > > > No - it refers to a blocksize. The matrix could have a block > > structure. In this case - the correct blocksize can be > > specified. The matrix operations will exploit this block > > structure for better performance. > > > > bs=1 is a fine [This is the equivalent of AIJ type in the > > block notation] > > > > > The method of creating a matrix I used was taken from Pavel Solin's > > > book 'Partial Differential Equations and the finite Element > > Method'. > > > Is there any particular reason why is should follow this > > > method: > > > mat = (double*)malloc(sizeof(Mat)) > > > MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, PETSC_DEFAULT, nnz, > > > (Mat*)mat); In the examples that I have seen in the petsc website I > > > don't see this statement where memory is allocated. > > > > No - you should just use PETSc example codes a guidelines. i.e > > > > Mat mat; > > MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, PETSC_DEFAULT, > > nnz, &mat); > > > > Mat is just a pointer - and all the memory allocation > > required is done inside XXXCreate() routines. > > > > > Anyway, I changed my code to the following: > > > PetscInitialize(0,0,"petscOptions.txt",0); > > > > > > int i; > > > int *nnz; > > > nnz=new int[L]; > > > for(i=0;i > > nnz[i]=row[i].getNum(); > > > } > > > > > > Assert( mat = (double*)malloc(sizeof(Mat)) ); > > > MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, > > PETSC_DEFAULT, nnz, (Mat*)mat); > > > delete [] nnz; > > > > > And I'm still getting stuck at the same place... There is a > > difference > > > though.. Now the assembly process is even slower than before (when > > > there was no preallocation) I noticed that it is getting > > stuck around > > > the 200th MAT_FLUSH_ASSEMBLY call.. I don't know if that is > > > significant.Is there some limit on how many times > > MAT_FLUSH_ASSEMBLY > > > can be called ? > > > > MAT_FLUSH_ASSEMBLY is cheap - in sequential mode - but > > expensive in parallel mode [becasue of parallel > > syncronization]. However ... > > > > > This is the petsc output I get: > > > [0] PetscInitializePETSc successfully started: number of > > processors = > > > 1 [0] PetscGetHostNameRejecting domainname, likely is NIS > > ALPHA2.??0?? > > > [0] PetscInitializeRunning on machine: ALPHA2 [0] > > > PetscCommDuplicateDuplicating a communicator 1 1 max tags = > > 100000000 > > > > The above log is incomplete. You should be looking for lines > > with the string 'malloc' > > > > Satish > > > > > > > > I also tried this : > > > mat = new Mat; > > > MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, > > PETSC_DEFAULT, nnz, mat); > > > > > > And got the same performance. > > > > From petsc-maint at mcs.anl.gov Fri Oct 13 11:16:29 2006 From: petsc-maint at mcs.anl.gov (Satish Balay) Date: Fri, 13 Oct 2006 11:16:29 -0500 (CDT) Subject: [PETSC #15433] RE: General matrix interative solver In-Reply-To: <00fa01c6eee0$d6125b10$24b75ba5@aero.ad.tamu.edu> References: <00fa01c6eee0$d6125b10$24b75ba5@aero.ad.tamu.edu> Message-ID: On Fri, 13 Oct 2006, Julian wrote: > > > > > I am now using MatCreateSeqSBAIJ() - I am assuming the second > > > parameter bs=1 in my case because I am using one processor. Is that > > > correct ? > > > > No - it refers to a blocksize. The matrix could have a block > > structure. In this case - the correct blocksize can be > > specified. The matrix operations will exploit this block > > structure for better performance. > > > > bs=1 is a fine [This is the equivalent of AIJ type in the > > block notation] > > I don't fully understand the thing about the block structure.. Can you point > me to some documentation that explains this ? Block matrices will occur in multicomponent problems [where each node of a grid has multiple unknowns - like pressure,density,velocity etc..] For eg: - if you have the following 6x6 matrix: 1 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 2 2 0 0 0 0 2 2 3 3 4 4 0 0 3 3 4 4 0 0 It has a block structure of 2. In the block notation could be: a 0 0 0 0 b c d 0 where a,b,c,d are 2x2 blocks a = 1 1 1 1 b = 2 2 2 2 c = 3 3 3 3 d = 4 4 4 4 So the regular sparse notation would be (i,j,val): [Which can be used with MatSetValues()] (0,0,1) (0,1,1) (1,0,1) (1,1,1) (2,4,2) (2,5,2) ...... The block noatation would be (block-i,block-j,block-val) [Which can be used with MatSetValuesBlocked()] (0,0,1 1 1 1) (1,2,2 2 2 2) (2,0,3,3,3) (2,1,4 4 4 4) I'm cc:ing petsc-users - as others also may find this info useful. Hope this helps, Satish From balay at mcs.anl.gov Fri Oct 13 15:10:39 2006 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 13 Oct 2006 15:10:39 -0500 (CDT) Subject: suppressing some command line options? In-Reply-To: References: <450EDA26.4040907@cims.nyu.edu> Message-ID: I'm attaching a patch that can be applied to petsc-2.3.2 for this to work. cd petsc-2.3.2-p3 patch -Np1 < opt.patch Alternatively petsc-dev should have it http://www-unix.mcs.anl.gov/petsc/petsc-as/developers/index.html Let us know if this works. Satish On Mon, 18 Sep 2006, Barry Smith wrote: > > Boyce, > > Satish is working on this right now. It is a problem that we > never properly resolved in that past and it needs to be solved. > Unfortunately we won't be able to get it into the current petsc release, > it will have to go in petsc-dev. > > Thanks for reminding us of this problem, > > Barry > > > On Mon, 18 Sep 2006, Boyce Griffith wrote: > > > Hi, Folks -- > > > > I have an application in which, for better or for worse, I generate a huge > > number of sequential sparse AIJ matrices. I was wondering if it was possible > > to suppress the command line options affiliated with those objects. (I am > > using MatCreateSeqAIJ, and I am not using MatSetFromOptions.) > > > > The basic issue is that when a user runs the program with the '-help' flag, > > the options associated with these matrices tend to overwhelm all of the other > > command line options, e.g., KSP settings. > > > > Thanks in advance for any suggestions, > > > > -- Boyce Griffith > > > > > > From balay at mcs.anl.gov Fri Oct 13 15:12:06 2006 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 13 Oct 2006 15:12:06 -0500 (CDT) Subject: suppressing some command line options? In-Reply-To: References: <450EDA26.4040907@cims.nyu.edu> Message-ID: ops - forgot the attachment. [attached now] Satish On Fri, 13 Oct 2006, Satish Balay wrote: > I'm attaching a patch that can be applied to petsc-2.3.2 for this to > work. > > cd petsc-2.3.2-p3 > patch -Np1 < opt.patch > > Alternatively petsc-dev should have it > http://www-unix.mcs.anl.gov/petsc/petsc-as/developers/index.html > > Let us know if this works. > Satish > > On Mon, 18 Sep 2006, Barry Smith wrote: > > > > > Boyce, > > > > Satish is working on this right now. It is a problem that we > > never properly resolved in that past and it needs to be solved. > > Unfortunately we won't be able to get it into the current petsc release, > > it will have to go in petsc-dev. > > > > Thanks for reminding us of this problem, > > > > Barry > > > > > > On Mon, 18 Sep 2006, Boyce Griffith wrote: > > > > > Hi, Folks -- > > > > > > I have an application in which, for better or for worse, I generate a huge > > > number of sequential sparse AIJ matrices. I was wondering if it was possible > > > to suppress the command line options affiliated with those objects. (I am > > > using MatCreateSeqAIJ, and I am not using MatSetFromOptions.) > > > > > > The basic issue is that when a user runs the program with the '-help' flag, > > > the options associated with these matrices tend to overwhelm all of the other > > > command line options, e.g., KSP settings. > > > > > > Thanks in advance for any suggestions, > > > > > > -- Boyce Griffith > > > > > > > > > > > > -------------- next part -------------- diff -r f5f14c217157875403b3b6ad5183d34fdf2a2a69 src/sys/objects/aoptions.c --- a/src/sys/objects/aoptions.c Tue Oct 10 11:01:54 2006 -0500 +++ b/src/sys/objects/aoptions.c Fri Oct 13 00:29:31 2006 -0500 @@ -33,14 +33,98 @@ struct _p_Options { PetscOptions next; }; +typedef struct _p_OptionsHelp* OptionsHelp; +struct _p_OptionsHelp { + char *prefix; + char *title; + char *mansec; + OptionsHelp next; +}; + static struct { PetscOptions next; char *prefix,*mprefix; char *title; MPI_Comm comm; - PetscTruth printhelp,changedmethod; + PetscTruth printhelp,changedmethod,alreadyprinted; + OptionsHelp help; } PetscOptionsObject; PetscInt PetscOptionsPublishCount = 0; + + +#undef __FUNCT__ +#define __FUNCT__ "PetscOptionsHelpAddList" +PetscErrorCode PetscOptionsHelpAddList(const char prefix[],const char title[],const char mansec[]) +{ + int ierr; + OptionsHelp newhelp; + PetscFunctionBegin; + ierr = PetscNew(struct _p_OptionsHelp,&newhelp);CHKERRQ(ierr); + ierr = PetscStrallocpy(prefix,&newhelp->prefix);CHKERRQ(ierr); + ierr = PetscStrallocpy(title,&newhelp->title);CHKERRQ(ierr); + ierr = PetscStrallocpy(mansec,&newhelp->mansec);CHKERRQ(ierr); + newhelp->next = 0; + + if (!PetscOptionsObject.help) { + PetscOptionsObject.help = newhelp; + } else { + newhelp->next = PetscOptionsObject.help; + PetscOptionsObject.help = newhelp; + } + PetscFunctionReturn(0); +} + +#undef __FUNCT__ +#define __FUNCT__ "PetscOptionsHelpDestroyList" +PetscErrorCode PetscOptionsHelpDestroyList(void) +{ + PetscErrorCode ierr; + OptionsHelp help = PetscOptionsObject.help, next; + + PetscFunctionBegin; + while (help) { + next = help->next; + ierr = PetscStrfree(help->prefix);CHKERRQ(ierr); + ierr = PetscStrfree(help->title);CHKERRQ(ierr); + ierr = PetscStrfree(help->mansec);CHKERRQ(ierr); + ierr = PetscFree(help);CHKERRQ(ierr); + help = next; + } + PetscFunctionReturn(0); +} + + +#undef __FUNCT__ +#define __FUNCT__ "PetscOptionsHelpFindList" +PetscErrorCode PetscOptionsHelpFindList(const char prefix[],const char title[],const char mansec[],PetscTruth *flg) +{ + PetscErrorCode ierr; + PetscTruth flg1,flg2,flg3; + PetscFunctionBegin; + OptionsHelp help = PetscOptionsObject.help; + while (help) { + ierr = PetscStrcmp(help->prefix,prefix,&flg1);CHKERRQ(ierr); + ierr = PetscStrcmp(help->title,title,&flg2);CHKERRQ(ierr); + ierr = PetscStrcmp(help->mansec,mansec,&flg3);CHKERRQ(ierr); + if (flg1 && flg2 && flg3) { + *flg = 1; + break; + } + help = help->next; + } + PetscFunctionReturn(0); + +} + +#undef __FUNCT__ +#define __FUNCT__ "PetscOptionsHelpCheckAddList" +PetscErrorCode PetscOptionsHelpCheckAddList(const char prefix[],const char title[],const char mansec[],PetscTruth *flg) +{ + PetscFunctionBegin; + PetscOptionsHelpFindList(prefix,title,mansec,flg); + if (!(*flg)) PetscOptionsHelpAddList(prefix,title,mansec); + PetscFunctionReturn(0); +} #undef __FUNCT__ #define __FUNCT__ "PetscOptionsBegin_Private" @@ -60,7 +144,10 @@ PetscErrorCode PetscOptionsBegin_Private ierr = PetscOptionsHasName(PETSC_NULL,"-help",&PetscOptionsObject.printhelp);CHKERRQ(ierr); if (PetscOptionsObject.printhelp && PetscOptionsPublishCount == 1) { - ierr = (*PetscHelpPrintf)(comm,"%s -------------------------------------------------\n",title);CHKERRQ(ierr); + ierr = PetscOptionsHelpCheckAddList(prefix,title,mansec,&PetscOptionsObject.alreadyprinted); + if (!PetscOptionsObject.alreadyprinted) { + ierr = (*PetscHelpPrintf)(comm,"%s -------------------------------------------------\n",title);CHKERRQ(ierr); + } } PetscFunctionReturn(0); } @@ -82,6 +169,7 @@ static int PetscOptionsCreate_Private(co (*amsopt)->type = t; (*amsopt)->data = 0; (*amsopt)->edata = 0; + ierr = PetscStrallocpy(text,&(*amsopt)->text);CHKERRQ(ierr); ierr = PetscStrallocpy(opt,&(*amsopt)->option);CHKERRQ(ierr); ierr = PetscStrallocpy(man,&(*amsopt)->man);CHKERRQ(ierr); @@ -144,6 +232,8 @@ PetscErrorCode PetscOptionsEnd_Private(v /* reset counter to -2; this updates the screen with the new options for the selected method */ if (PetscOptionsObject.changedmethod) PetscOptionsPublishCount = -2; + /* reset alreadyprinted flag */ + PetscOptionsObject.alreadyprinted = PETSC_FALSE; while (PetscOptionsObject.next) { if (PetscOptionsObject.next->set) { @@ -469,7 +559,7 @@ PetscErrorCode PETSC_DLLEXPORT PetscOpti PetscFunctionBegin; ierr = PetscOptionsHasName(PetscOptionsObject.prefix,opt,flg);CHKERRQ(ierr); - if (PetscOptionsObject.printhelp && PetscOptionsPublishCount == 1) { + if (PetscOptionsObject.printhelp && PetscOptionsPublishCount == 1 && !PetscOptionsObject.alreadyprinted) { ierr = (*PetscHelpPrintf)(PetscOptionsObject.comm," -%s%s: %s (%s)\n",PetscOptionsObject.prefix?PetscOptionsObject.prefix:"",opt+1,text,man);CHKERRQ(ierr); } PetscFunctionReturn(0); diff -r f5f14c217157875403b3b6ad5183d34fdf2a2a69 src/sys/objects/pinit.c --- a/src/sys/objects/pinit.c Tue Oct 10 11:01:54 2006 -0500 +++ b/src/sys/objects/pinit.c Fri Oct 13 00:24:06 2006 -0500 @@ -20,6 +20,7 @@ EXTERN PetscErrorCode PetscSequentialPha EXTERN PetscErrorCode PetscSequentialPhaseBegin_Private(MPI_Comm,int); EXTERN PetscErrorCode PetscSequentialPhaseEnd_Private(MPI_Comm,int); EXTERN PetscErrorCode PetscLogCloseHistoryFile(FILE **); +EXTERN PetscErrorCode PetscOptionsHelpDestroyList(void); /* this is used by the _, __, and ___ macros (see include/petscerror.h) */ PetscErrorCode __gierr = 0; @@ -622,6 +623,7 @@ PetscErrorCode PETSC_DLLEXPORT PetscFina PETSC_VIEWER_XXX_(). */ ierr = PetscObjectRegisterDestroyAll();CHKERRQ(ierr); + ierr = PetscOptionsHelpDestroyList();CHKERRQ(ierr); #if defined(PETSC_USE_DEBUG) if (PetscStackActive) { diff --git a/src/sys/objects/aoptions.c b/src/sys/objects/aoptions.c --- a/src/sys/objects/aoptions.c Fri Oct 13 14:52:55 2006 +++ b/src/sys/objects/aoptions.c Fri Oct 13 19:32:04 2006 @@ -107,7 +107,7 @@ ierr = PetscStrcmp(help->title,title,&flg2);CHKERRQ(ierr); ierr = PetscStrcmp(help->mansec,mansec,&flg3);CHKERRQ(ierr); if (flg1 && flg2 && flg3) { - *flg = 1; + *flg = PETSC_TRUE; break; } help = help->next; From tmyue at westnet.com.au Sat Oct 14 16:48:11 2006 From: tmyue at westnet.com.au (TAN MENG YUE) Date: Sun, 15 Oct 2006 07:48:11 +1000 Subject: NetCDF viewer/data-output Message-ID: <8ADECA82-9F07-4F9B-9F44-4F6F346473BD@westnet.com.au> Hi, I noticed that there is an option for a NetCDF viewier/data-output. How do I use it? I was looking at the the SNES Bratu ex14 as a 3D example but it seems to only be able to output Matlab and ASCII data. I don't have access to Matlab. I am after some time vary NetCDF data set for visualizating evolving/ animating isosurfaces+volume rendering with OpenDX. Cheers -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2423 bytes Desc: not available URL: From julvar at tamu.edu Sun Oct 15 21:15:14 2006 From: julvar at tamu.edu (Julian) Date: Sun, 15 Oct 2006 21:15:14 -0500 Subject: MPI question In-Reply-To: Message-ID: <000a01c6f0c8$eabae480$24b75ba5@aero.ad.tamu.edu> I am using petsc in my fem program on the supercomputer we have here at texas a&m univ. I configured petsc to build the libraries with mpi. So, I ran 'mpirun -np 2 /home/j0v1008/bin/alphampi a.alpha' in order to use 2 processors. Till the point where petsc is initialized, everything looks normal. From the point petsc in initialized, I get duplicates of everything that is written to cout (at the very least, maybe it's doing duplicates of other things as well). I understand that all the petsc functions might be called twice or so depending on how it parallelizes, but I don't understand why it executes lines that are outside of the petsc world, that have nothing to do with mpi. Maybe this would clear things up: void main() { Cout << "line1" //insert code to initialize mpi Cout << "line2" MyfunctionusingMPI() Cout << "line3" //insert code to finalizeMPI } Say I run this program with 2 processors, will "line2" be printed twice ? If yes, can you explain why it does that when I didn't ask for that line to be parallelized? If no, then why is my fea program behaving so? Thanks, Julian. From knepley at gmail.com Sun Oct 15 21:35:08 2006 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 15 Oct 2006 21:35:08 -0500 Subject: MPI question In-Reply-To: <000a01c6f0c8$eabae480$24b75ba5@aero.ad.tamu.edu> References: <000a01c6f0c8$eabae480$24b75ba5@aero.ad.tamu.edu> Message-ID: On 10/15/06, Julian wrote: > > I am using petsc in my fem program on the supercomputer we have here at > texas a&m univ. I configured petsc to build the libraries with mpi. > So, I ran 'mpirun -np 2 /home/j0v1008/bin/alphampi a.alpha' in order to > use > 2 processors. > Till the point where petsc is initialized, everything looks normal. From > the > point petsc in initialized, I get duplicates of everything that is written > to cout (at the very least, maybe it's doing duplicates of other things as > well). > I understand that all the petsc functions might be called twice or so > depending on how it parallelizes, but I don't understand why it executes > lines that are outside of the petsc world, that have nothing to do with > mpi. > > Maybe this would clear things up: > > void main() > { > Cout << "line1" > //insert code to initialize mpi > Cout << "line2" > MyfunctionusingMPI() > Cout << "line3" > //insert code to finalizeMPI > } > > Say I run this program with 2 processors, will "line2" be printed twice ? > If > yes, can you explain why it does that when I didn't ask for that line to > be > parallelized? Yes. MPI is an SPMD model. There are multiple running copies of the same program. I suggest "Using MPI" by Gropp, et. al. for a better understading of parallel code. Matt If no, then why is my fea program behaving so? > > Thanks, > Julian. > > -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness -------------- next part -------------- An HTML attachment was scrubbed... URL: From berend at chalmers.se Mon Oct 16 01:34:33 2006 From: berend at chalmers.se (Berend van Wachem) Date: Mon, 16 Oct 2006 08:34:33 +0200 Subject: MPI question In-Reply-To: <000a01c6f0c8$eabae480$24b75ba5@aero.ad.tamu.edu> References: <000a01c6f0c8$eabae480$24b75ba5@aero.ad.tamu.edu> Message-ID: <453327F9.7090605@chalmers.se> Hi Julian, I am not affiliated to the petsc-team, but have been using petsc a number of years. The idea of MPI is that your whole program get executed on each processor. So getting output twice on each processor is quite normal. So not only PETSC functions are called by each processor, every command you write will be called by each processor. Because variables have different values upon each processor, you can imply slightly different behaviour, or have each processor work on an own data set. If you want output, I recommend you to use the PETSC functions for IO: PetscPrintf, or PetscSynchronizedPrintf. I found it helpful to read an introduction to MPI, "Using MPI" by Gropp, Lusk, Skjellum. Good luck, Berend. > I am using petsc in my fem program on the supercomputer we have here at > texas a&m univ. I configured petsc to build the libraries with mpi. > So, I ran 'mpirun -np 2 /home/j0v1008/bin/alphampi a.alpha' in order to use > 2 processors. > Till the point where petsc is initialized, everything looks normal. From the > point petsc in initialized, I get duplicates of everything that is written > to cout (at the very least, maybe it's doing duplicates of other things as > well). > I understand that all the petsc functions might be called twice or so > depending on how it parallelizes, but I don't understand why it executes > lines that are outside of the petsc world, that have nothing to do with mpi. > > Maybe this would clear things up: > > void main() > { > Cout << "line1" > //insert code to initialize mpi > Cout << "line2" > MyfunctionusingMPI() > Cout << "line3" > //insert code to finalizeMPI > } > > Say I run this program with 2 processors, will "line2" be printed twice ? If > yes, can you explain why it does that when I didn't ask for that line to be > parallelized? > If no, then why is my fea program behaving so? > > Thanks, > Julian. > From yxliu at fudan.edu.cn Tue Oct 17 03:07:32 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Tue, 17 Oct 2006 16:07:32 +0800 Subject: about parallel execution Message-ID: <005e01c6f1c3$4c327610$3364a8c0@dmrc6700512> Hi, I am a new user of PETSC. I want to know how to execute of the PETSC application. For example, my code are like below, PETSCInitialize(); code1; code2; ..... PETSCFinalize(); Does PETSC copy the above codes, assuming there n processors, to each processor? Will code1 be execute on each processor? Best regards, Yixun -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Tue Oct 17 08:21:14 2006 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 17 Oct 2006 08:21:14 -0500 (CDT) Subject: about parallel execution In-Reply-To: <005e01c6f1c3$4c327610$3364a8c0@dmrc6700512> References: <005e01c6f1c3$4c327610$3364a8c0@dmrc6700512> Message-ID: Yixun, Chapter 1 of the Petsc users manual gives detailed instruction on 1. set petsc environment 2. write petsc program 3. compile and run program You may start running petsc examples as described in the manual. Hong On Tue, 17 Oct 2006, Yixun Liu wrote: > Hi, > I am a new user of PETSC. I want to know how to execute of the PETSC application. > For example, my code are like below, > > PETSCInitialize(); > code1; > code2; > ..... > PETSCFinalize(); > > Does PETSC copy the above codes, assuming there n processors, to each processor? Will code1 be execute on each processor? > > Best regards, > > Yixun > From balay at mcs.anl.gov Tue Oct 17 09:09:32 2006 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 17 Oct 2006 09:09:32 -0500 (CDT) Subject: about parallel execution In-Reply-To: References: <005e01c6f1c3$4c327610$3364a8c0@dmrc6700512> Message-ID: Note: PETSc uses MPI - and it follws the SPMD [Single Program Multiple Data] model. You might want to check the replies to the same question posted earlier at: http://www-unix.mcs.anl.gov/web-mail-archive/lists/petsc-users/2006/10/msg00052.html Satish On Tue, 17 Oct 2006, Hong Zhang wrote: > > Yixun, > > Chapter 1 of the > Petsc users manual gives detailed instruction > on > 1. set petsc environment > 2. write petsc program > 3. compile and run program > > You may start running petsc examples as described > in the manual. > > Hong > > On Tue, 17 Oct 2006, Yixun Liu wrote: > > > Hi, > > I am a new user of PETSC. I want to know how to execute of the PETSC application. > > For example, my code are like below, > > > > PETSCInitialize(); > > code1; > > code2; > > ..... > > PETSCFinalize(); > > > > Does PETSC copy the above codes, assuming there n processors, to each processor? Will code1 be execute on each processor? > > > > Best regards, > > > > Yixun > > > > From yxliu at fudan.edu.cn Tue Oct 17 09:33:38 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Tue, 17 Oct 2006 22:33:38 +0800 Subject: about parallel execution References: <005e01c6f1c3$4c327610$3364a8c0@dmrc6700512> Message-ID: <007a01c6f1f9$3c99c920$3364a8c0@dmrc6700512> Hi, Thanks for your reply. I read the message you provide and Berend said that "So not only PETSC functions are called by each processor, every command you write will be called by each processor". However,if each PETSC function is called by each processor, the function like VecSet(F_global, zero), in which F_global is a parallel vector, will be called many times. In my opinion, to be called once is enough. Regards, Yixun ----- Original Message ----- From: "Satish Balay" To: Sent: Tuesday, October 17, 2006 10:09 PM Subject: Re: about parallel execution > Note: PETSc uses MPI - and it follws the SPMD [Single Program Multiple > Data] model. You might want to check the replies to the same question > posted earlier at: > > http://www-unix.mcs.anl.gov/web-mail-archive/lists/petsc-users/2006/10/msg00052.html > > Satish > > On Tue, 17 Oct 2006, Hong Zhang wrote: > >> >> Yixun, >> >> Chapter 1 of the >> Petsc users manual gives detailed instruction >> on >> 1. set petsc environment >> 2. write petsc program >> 3. compile and run program >> >> You may start running petsc examples as described >> in the manual. >> >> Hong >> >> On Tue, 17 Oct 2006, Yixun Liu wrote: >> >> > Hi, >> > I am a new user of PETSC. I want to know how to execute of the PETSC application. >> > For example, my code are like below, >> > >> > PETSCInitialize(); >> > code1; >> > code2; >> > ..... >> > PETSCFinalize(); >> > >> > Does PETSC copy the above codes, assuming there n processors, to each processor? Will code1 be execute on each processor? >> > >> > Best regards, >> > >> > Yixun >> > >> >> > From hzhang at mcs.anl.gov Tue Oct 17 10:03:08 2006 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 17 Oct 2006 10:03:08 -0500 (CDT) Subject: about parallel execution In-Reply-To: <007a01c6f1f9$3c99c920$3364a8c0@dmrc6700512> References: <005e01c6f1c3$4c327610$3364a8c0@dmrc6700512> <007a01c6f1f9$3c99c920$3364a8c0@dmrc6700512> Message-ID: Yixun, Please read the Petsc users manual, at least Chapter 1. The ex2 describes how to write a parallel code, such that each processor sets part of vector or matrix values. When possible, it sets its local values to reduce communication. Hong On Tue, 17 Oct 2006, Yixun Liu wrote: > Hi, > > Thanks for your reply. I read the message you provide and Berend said that "So not only PETSC functions are called by each processor, every command you write will be called by each processor". However,if each PETSC function is called by each processor, the function like > VecSet(F_global, zero), in which F_global is a parallel vector, will be called many times. In my opinion, to be called once is enough. > > Regards, > > Yixun > > ----- Original Message ----- > From: "Satish Balay" > To: > Sent: Tuesday, October 17, 2006 10:09 PM > Subject: Re: about parallel execution > > > > Note: PETSc uses MPI - and it follws the SPMD [Single Program Multiple > > Data] model. You might want to check the replies to the same question > > posted earlier at: > > > > http://www-unix.mcs.anl.gov/web-mail-archive/lists/petsc-users/2006/10/msg00052.html > > > > Satish > > > > On Tue, 17 Oct 2006, Hong Zhang wrote: > > > >> > >> Yixun, > >> > >> Chapter 1 of the > >> Petsc users manual gives detailed instruction > >> on > >> 1. set petsc environment > >> 2. write petsc program > >> 3. compile and run program > >> > >> You may start running petsc examples as described > >> in the manual. > >> > >> Hong > >> > >> On Tue, 17 Oct 2006, Yixun Liu wrote: > >> > >> > Hi, > >> > I am a new user of PETSC. I want to know how to execute of the PETSC application. > >> > For example, my code are like below, > >> > > >> > PETSCInitialize(); > >> > code1; > >> > code2; > >> > ..... > >> > PETSCFinalize(); > >> > > >> > Does PETSC copy the above codes, assuming there n processors, to each processor? Will code1 be execute on each processor? > >> > > >> > Best regards, > >> > > >> > Yixun > >> > > >> > >> > > > > From balay at mcs.anl.gov Tue Oct 17 10:10:50 2006 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 17 Oct 2006 10:10:50 -0500 (CDT) Subject: about parallel execution In-Reply-To: <007a01c6f1f9$3c99c920$3364a8c0@dmrc6700512> References: <005e01c6f1c3$4c327610$3364a8c0@dmrc6700512> <007a01c6f1f9$3c99c920$3364a8c0@dmrc6700512> Message-ID: Note: the porgram/code executed on each processor is the same. However the data on each processor could be different. This ammounts to distributing work across all processors, and noprocessor is doing duplicate work. [This is the meaning of single program multiple data] Sounds like you need a good introduction to parallel computing. Satish On Tue, 17 Oct 2006, Yixun Liu wrote: > Hi, > > Thanks for your reply. I read the message you provide and Berend said that "So not only PETSC functions are called by each processor, every command you write will be called by each processor". However,if each PETSC function is called by each processor, the function like > VecSet(F_global, zero), in which F_global is a parallel vector, will be called many times. In my opinion, to be called once is enough. > > Regards, > > Yixun > > ----- Original Message ----- > From: "Satish Balay" > To: > Sent: Tuesday, October 17, 2006 10:09 PM > Subject: Re: about parallel execution > > > > Note: PETSc uses MPI - and it follws the SPMD [Single Program Multiple > > Data] model. You might want to check the replies to the same question > > posted earlier at: > > > > http://www-unix.mcs.anl.gov/web-mail-archive/lists/petsc-users/2006/10/msg00052.html > > > > Satish > > > > On Tue, 17 Oct 2006, Hong Zhang wrote: > > > >> > >> Yixun, > >> > >> Chapter 1 of the > >> Petsc users manual gives detailed instruction > >> on > >> 1. set petsc environment > >> 2. write petsc program > >> 3. compile and run program > >> > >> You may start running petsc examples as described > >> in the manual. > >> > >> Hong > >> > >> On Tue, 17 Oct 2006, Yixun Liu wrote: > >> > >> > Hi, > >> > I am a new user of PETSC. I want to know how to execute of the PETSC application. > >> > For example, my code are like below, > >> > > >> > PETSCInitialize(); > >> > code1; > >> > code2; > >> > ..... > >> > PETSCFinalize(); > >> > > >> > Does PETSC copy the above codes, assuming there n processors, to each processor? Will code1 be execute on each processor? > >> > > >> > Best regards, > >> > > >> > Yixun > >> > > >> > >> > > > > From yxliu at fudan.edu.cn Wed Oct 18 03:03:09 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Wed, 18 Oct 2006 16:03:09 +0800 Subject: about MatSetValues() Message-ID: <000801c6f28b$da04a030$3364a8c0@dmrc6700512> Hi, In this function I need to provide the number of rows and their global indices and the number of columns and their global indices. What I want to know is if I only give these indices regardless which porcessor ownes its indices? In another words, do I need to know the indices, which this process ownes before I call this function? Regards, Yixun Liu -------------- next part -------------- An HTML attachment was scrubbed... URL: From nilserik at gmail.com Wed Oct 18 03:47:03 2006 From: nilserik at gmail.com (=?ISO-8859-1?Q?Nils_Erik_Svang=E5rd?=) Date: Wed, 18 Oct 2006 10:47:03 +0200 Subject: PETSc CFD advice Message-ID: <82f936c20610180147x1fdae75fue781399161308c86@mail.gmail.com> Hello! Last year I did my master thesis which consisted of converting and old fortran based CFD application using block structured explicit solver to use the PETSc framework. Now my mentor/supervisor has contacted me again, they are thinking about doing a unstructured solver and are looking at PETSc. I said that I think using PETSc is a good idea, and if they are going to write new code I think they should look at using C++ instead of Fortran (because I think C++ is better suported by PETSc). Was my advice any good? Any other advices they should look at before beginning? best regards /nisse -- Nils-Erik Svang?rd E-Mail: nilserik at gmail.com MSN: nilserik at gmail.com Skype: schweingaard Mobil: +46-(0)70-3612178 From knepley at gmail.com Wed Oct 18 07:26:31 2006 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 18 Oct 2006 07:26:31 -0500 Subject: PETSc CFD advice In-Reply-To: <82f936c20610180147x1fdae75fue781399161308c86@mail.gmail.com> References: <82f936c20610180147x1fdae75fue781399161308c86@mail.gmail.com> Message-ID: On 10/18/06, Nils Erik Svang?rd wrote: > > Hello! > Last year I did my master thesis which consisted of converting and old > fortran based CFD application using block structured explicit solver > to use the PETSc framework. > Now my mentor/supervisor has contacted me again, they are thinking > about doing a unstructured solver and are looking at PETSc. > I said that I think using PETSc is a good idea, and if they are going > to write new code I think they should look at using C++ instead of > Fortran (because I think C++ is better suported by PETSc). > Was my advice any good? Any other advices they should look at before > beginning? While we do fully support Fortran, we have new more advanced features in C++, so I think yor advice was good as long as they can be comfortable in C++ :) There is new support for unstructured meshes in PETSc. It is still in alpha release, so it is not in the same category as the other PETSc structures, but I would encourage them to look at it. I recently gave a tutorial at Columbia (which I will post on the web tonight), which demonstrates some of this. Thanks, Matt best regards > /nisse > > -- > Nils-Erik Svang?rd > E-Mail: nilserik at gmail.com > MSN: nilserik at gmail.com > Skype: schweingaard > Mobil: +46-(0)70-3612178 > > -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Oct 18 09:22:58 2006 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 18 Oct 2006 09:22:58 -0500 (CDT) Subject: about MatSetValues() In-Reply-To: <000801c6f28b$da04a030$3364a8c0@dmrc6700512> References: <000801c6f28b$da04a030$3364a8c0@dmrc6700512> Message-ID: Yixun, > In this function I need to provide the number of rows and their global indices and >the number of columns and their global indices. >What I want to know is if I only give these >indices regardless which porcessor ownes its indices? Yes. User does not need to know the ownership of rows/columns. Petsc hides low-level communications to enable fast code development. When user knows the distribution of data, he is able to set values into the owned processor for better performance. Take a look at the petsc tutorial sliedes posted at http://www-unix.mcs.anl.gov/petsc/petsc-as/documentation/tutorials/index.html We have detailed introduction on this subject. >In another words, >do I need to know the indices, which this process ownes before I call this function? > You don't have to know. But if you know, you can write a better application code. Hong From julvar at tamu.edu Wed Oct 18 10:53:02 2006 From: julvar at tamu.edu (Julian) Date: Wed, 18 Oct 2006 10:53:02 -0500 Subject: MPI question In-Reply-To: <453327F9.7090605@chalmers.se> Message-ID: <00ca01c6f2cd$7e4c1290$24b75ba5@aero.ad.tamu.edu> Thanks Berend and Matt, I'm gonna take a look at that book... Luckily, they have an e-book version through our library! Julian. > -----Original Message----- > From: owner-petsc-users at mcs.anl.gov > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Berend van Wachem > Sent: Monday, October 16, 2006 1:35 AM > To: petsc-users at mcs.anl.gov > Subject: Re: MPI question > > Hi Julian, > > I am not affiliated to the petsc-team, but have been using > petsc a number of years. > The idea of MPI is that your whole program get executed on > each processor. So getting output twice on each processor is > quite normal. So not only PETSC functions are called by each > processor, every command you write will be called by each > processor. Because variables have different values upon each > processor, you can imply slightly different behaviour, or > have each processor work on an own data set. > > If you want output, I recommend you to use the PETSC > functions for IO: > PetscPrintf, or PetscSynchronizedPrintf. > > I found it helpful to read an introduction to MPI, "Using > MPI" by Gropp, Lusk, Skjellum. > > Good luck, > > Berend. > > > I am using petsc in my fem program on the supercomputer we > have here > > at texas a&m univ. I configured petsc to build the > libraries with mpi. > > So, I ran 'mpirun -np 2 /home/j0v1008/bin/alphampi > a.alpha' in order > > to use > > 2 processors. > > Till the point where petsc is initialized, everything looks normal. > > From the point petsc in initialized, I get duplicates of everything > > that is written to cout (at the very least, maybe it's doing > > duplicates of other things as well). > > I understand that all the petsc functions might be called > twice or so > > depending on how it parallelizes, but I don't understand why it > > executes lines that are outside of the petsc world, that > have nothing to do with mpi. > > > > Maybe this would clear things up: > > > > void main() > > { > > Cout << "line1" > > //insert code to initialize mpi > > Cout << "line2" > > MyfunctionusingMPI() > > Cout << "line3" > > //insert code to finalizeMPI > > } > > > > Say I run this program with 2 processors, will "line2" be printed > > twice ? If yes, can you explain why it does that when I > didn't ask for > > that line to be parallelized? > > If no, then why is my fea program behaving so? > > > > Thanks, > > Julian. > > > From yxliu at fudan.edu.cn Fri Oct 20 00:35:26 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Fri, 20 Oct 2006 13:35:26 +0800 Subject: about PETSC Message-ID: <002901c6f409$9201eac0$3364a8c0@dmrc6700512> Hi, Could I use PETSC on WinXP, MSVC6.0? Regards, Yixun -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Oct 20 07:06:42 2006 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 20 Oct 2006 07:06:42 -0500 Subject: about PETSC In-Reply-To: <002901c6f409$9201eac0$3364a8c0@dmrc6700512> References: <002901c6f409$9201eac0$3364a8c0@dmrc6700512> Message-ID: Yes, but you will need cygwin as well. See the instructions on the installation page: http://www-unix.mcs.anl.gov/petsc/petsc-as/documentation/installation.html Matt On 10/20/06, Yixun Liu wrote: > > Hi, > > Could I use PETSC on WinXP, MSVC6.0? > > Regards, > > Yixun > -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness -------------- next part -------------- An HTML attachment was scrubbed... URL: From yxliu at fudan.edu.cn Sat Oct 21 02:04:04 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Sat, 21 Oct 2006 15:04:04 +0800 Subject: about PETSC installation Message-ID: <002301c6f4df$186bf350$3364a8c0@dmrc6700512> Hi, I install PETSC on WindowsXP, MS VC++6.0, Cygwin, Dell standard dual processors PC, I do as the PETSc installation page says, $ ./config/configure.py --with-mpi-dir=d:/MPICH2 However, the error is "the Fortran error! mpi_init() could not be located!" Then I try this, $ ./config/configure.py --with-mpi-dir=d:/MPICH2 --with-cc=c++ The error is "Unknown Fortran name mangling" Hope your help! Best regards, Yixun Liu -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sat Oct 21 09:17:06 2006 From: balay at mcs.anl.gov (Satish Balay) Date: Sat, 21 Oct 2006 09:17:06 -0500 (CDT) Subject: about PETSC installation In-Reply-To: <002301c6f4df$186bf350$3364a8c0@dmrc6700512> References: <002301c6f4df$186bf350$3364a8c0@dmrc6700512> Message-ID: On Sat, 21 Oct 2006, Yixun Liu wrote: > Hi, > I install PETSC on WindowsXP, MS VC++6.0, Cygwin, Dell standard dual processors PC, > I do as the PETSc installation page says, > > $ ./config/configure.py --with-mpi-dir=d:/MPICH2 > However, the error is "the Fortran error! mpi_init() could not be located!" > > Then I try this, > $ ./config/configure.py --with-mpi-dir=d:/MPICH2 --with-cc=c++ > The error is "Unknown Fortran name mangling" You probabably want to do: ./config/configure.py --with-mpi-dir=/cygdrive/d/MPICH2 --with-cc='win32fe cl' --with-fc=0 --with-cxx='win32fe cl' --with-clanguage=cxx --download-c-blas-lapack=1 Check installation instructions for additional info. If you encounter further problems - send us the corresponding configure.log to the address petsc-maint at mcs.anl.gov [not to the users list] Satish > > Hope your help! > > Best regards, > > Yixun Liu > From yxliu at fudan.edu.cn Sat Oct 21 20:36:13 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Sun, 22 Oct 2006 09:36:13 +0800 Subject: about PETSC installation References: <002301c6f4df$186bf350$3364a8c0@dmrc6700512> Message-ID: <004b01c6f57a$75add6a0$3364a8c0@dmrc6700512> Hi, Thanks for your help. Firstly I want to know if the error is caused by no Fortran on my computer? BTW, could you tell me more about the configure option, which you suggest? Best regards, Yixun ----- Original Message ----- From: "Satish Balay" To: "PETSC" Sent: Saturday, October 21, 2006 10:17 PM Subject: Re: about PETSC installation > On Sat, 21 Oct 2006, Yixun Liu wrote: > >> Hi, >> I install PETSC on WindowsXP, MS VC++6.0, Cygwin, Dell standard dual processors PC, >> I do as the PETSc installation page says, >> >> $ ./config/configure.py --with-mpi-dir=d:/MPICH2 >> However, the error is "the Fortran error! mpi_init() could not be located!" >> >> Then I try this, >> $ ./config/configure.py --with-mpi-dir=d:/MPICH2 --with-cc=c++ >> The error is "Unknown Fortran name mangling" > > > You probabably want to do: > > ./config/configure.py --with-mpi-dir=/cygdrive/d/MPICH2 --with-cc='win32fe cl' --with-fc=0 --with-cxx='win32fe cl' --with-clanguage=cxx --download-c-blas-lapack=1 > > Check installation instructions for additional info. If you encounter > further problems - send us the corresponding configure.log to the > address petsc-maint at mcs.anl.gov [not to the users list] > > Satish > >> >> Hope your help! >> >> Best regards, >> >> Yixun Liu >> > From knepley at gmail.com Sat Oct 21 20:54:50 2006 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 21 Oct 2006 20:54:50 -0500 Subject: about PETSC installation In-Reply-To: <004b01c6f57a$75add6a0$3364a8c0@dmrc6700512> References: <002301c6f4df$186bf350$3364a8c0@dmrc6700512> <004b01c6f57a$75add6a0$3364a8c0@dmrc6700512> Message-ID: On 10/21/06, Yixun Liu wrote: > > Hi, > Thanks for your help. > > Firstly I want to know if the error is caused by no Fortran on my > computer? Yes, you can turn of Fortran using --with-fc=0. Matt BTW, could you tell me more about the configure option, which you suggest? > > Best regards, > > Yixun > > ----- Original Message ----- > From: "Satish Balay" > To: "PETSC" > Sent: Saturday, October 21, 2006 10:17 PM > Subject: Re: about PETSC installation > > > > On Sat, 21 Oct 2006, Yixun Liu wrote: > > > >> Hi, > >> I install PETSC on WindowsXP, MS VC++6.0, Cygwin, Dell standard dual > processors PC, > >> I do as the PETSc installation page says, > >> > >> $ ./config/configure.py --with-mpi-dir=d:/MPICH2 > >> However, the error is "the Fortran error! mpi_init() could not be > located!" > >> > >> Then I try this, > >> $ ./config/configure.py --with-mpi-dir=d:/MPICH2 --with-cc=c++ > >> The error is "Unknown Fortran name mangling" > > > > > > You probabably want to do: > > > > ./config/configure.py --with-mpi-dir=/cygdrive/d/MPICH2 > --with-cc='win32fe cl' --with-fc=0 --with-cxx='win32fe cl' > --with-clanguage=cxx --download-c-blas-lapack=1 > > > > Check installation instructions for additional info. If you encounter > > further problems - send us the corresponding configure.log to the > > address petsc-maint at mcs.anl.gov [not to the users list] > > > > Satish > > > >> > >> Hope your help! > >> > >> Best regards, > >> > >> Yixun Liu > >> > > > > -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness -------------- next part -------------- An HTML attachment was scrubbed... URL: From yxliu at fudan.edu.cn Sat Oct 21 21:43:57 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Sun, 22 Oct 2006 10:43:57 +0800 Subject: about PETSC installation References: <002301c6f4df$186bf350$3364a8c0@dmrc6700512> <004b01c6f57a$75add6a0$3364a8c0@dmrc6700512> Message-ID: <008501c6f583$ebe19bf0$3364a8c0@dmrc6700512> Thanks. If I turn off Fortran, does PETSC work well? I think PETSC depends on Fortran because it use LAPACK. Regards, Yixun ----- Original Message ----- From: Matthew Knepley To: petsc-users at mcs.anl.gov Sent: Sunday, October 22, 2006 9:54 AM Subject: Re: about PETSC installation On 10/21/06, Yixun Liu wrote: Hi, Thanks for your help. Firstly I want to know if the error is caused by no Fortran on my computer? Yes, you can turn of Fortran using --with-fc=0. Matt BTW, could you tell me more about the configure option, which you suggest? Best regards, Yixun ----- Original Message ----- From: "Satish Balay" To: "PETSC" Sent: Saturday, October 21, 2006 10:17 PM Subject: Re: about PETSC installation > On Sat, 21 Oct 2006, Yixun Liu wrote: > >> Hi, >> I install PETSC on WindowsXP, MS VC++6.0, Cygwin, Dell standard dual processors PC, >> I do as the PETSc installation page says, >> >> $ ./config/configure.py --with-mpi-dir=d:/MPICH2 >> However, the error is "the Fortran error! mpi_init() could not be located!" >> >> Then I try this, >> $ ./config/configure.py --with-mpi-dir=d:/MPICH2 --with-cc=c++ >> The error is "Unknown Fortran name mangling" > > > You probabably want to do: > > ./config/configure.py --with-mpi-dir=/cygdrive/d/MPICH2 --with-cc='win32fe cl' --with-fc=0 --with-cxx='win32fe cl' --with-clanguage=cxx --download-c-blas-lapack=1 > > Check installation instructions for additional info. If you encounter > further problems - send us the corresponding configure.log to the > address petsc-maint at mcs.anl.gov [not to the users list] > > Satish > >> >> Hope your help! >> >> Best regards, >> >> Yixun Liu >> > -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness -------------- next part -------------- An HTML attachment was scrubbed... URL: From yxliu at fudan.edu.cn Sat Oct 21 22:19:46 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Sun, 22 Oct 2006 11:19:46 +0800 Subject: about PETSC installation References: <002301c6f4df$186bf350$3364a8c0@dmrc6700512> Message-ID: <009901c6f588$ecc68b70$3364a8c0@dmrc6700512> Hi, I try it, the error is "Error downloading f2cblaslapack.tar.gz requested with -with-c-blas-lapack option". So I install fortran. I think fortran provide blas and lapack. If this what command should I use? Best, Yixun ----- Original Message ----- From: "Satish Balay" To: "PETSC" Sent: Saturday, October 21, 2006 10:17 PM Subject: Re: about PETSC installation > On Sat, 21 Oct 2006, Yixun Liu wrote: > >> Hi, >> I install PETSC on WindowsXP, MS VC++6.0, Cygwin, Dell standard dual processors PC, >> I do as the PETSc installation page says, >> >> $ ./config/configure.py --with-mpi-dir=d:/MPICH2 >> However, the error is "the Fortran error! mpi_init() could not be located!" >> >> Then I try this, >> $ ./config/configure.py --with-mpi-dir=d:/MPICH2 --with-cc=c++ >> The error is "Unknown Fortran name mangling" > > > You probabably want to do: > > ./config/configure.py --with-mpi-dir=/cygdrive/d/MPICH2 --with-cc='win32fe cl' --with-fc=0 --with-cxx='win32fe cl' --with-clanguage=cxx --download-c-blas-lapack=1 > > Check installation instructions for additional info. If you encounter > further problems - send us the corresponding configure.log to the > address petsc-maint at mcs.anl.gov [not to the users list] > > Satish > >> >> Hope your help! >> >> Best regards, >> >> Yixun Liu >> > From knepley at gmail.com Sat Oct 21 22:46:36 2006 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 21 Oct 2006 22:46:36 -0500 Subject: about PETSC installation In-Reply-To: <009901c6f588$ecc68b70$3364a8c0@dmrc6700512> References: <002301c6f4df$186bf350$3364a8c0@dmrc6700512> <009901c6f588$ecc68b70$3364a8c0@dmrc6700512> Message-ID: On 10/21/06, Yixun Liu wrote: > > Hi, > > I try it, the error is "Error downloading f2cblaslapack.tar.gz requested > with -with-c-blas-lapack option". So I install fortran. I think fortran > provide blas and lapack. If this what command should I use? Please refer installation questions to petsc-maint at mcs.anl.gov. Fortran does not provide those libraries. You would need something like --download-f-blas-lapack. However, it would likely for for the same reason as the last option. Every time your installation fails, you should send the configure.log to petsc-maint, as it advises you to do in the error messge. Matt Best, > > Yixun > -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness -------------- next part -------------- An HTML attachment was scrubbed... URL: From jiaxun_hou at yahoo.com.cn Sat Oct 21 22:40:51 2006 From: jiaxun_hou at yahoo.com.cn (jiaxun hou) Date: Sun, 22 Oct 2006 11:40:51 +0800 (CST) Subject: about Toeplitz type matrix Message-ID: <20061022034051.75362.qmail@web15813.mail.cnb.yahoo.com> Hi, I am using PETSc in my program to solve Toeplitz type systems. Are there any commands to construct the Toeplitz matrics in a convenient and efficient way? Regards, Jiaxun ___________________________________________________________ ????????-3.5G???20M??? http://cn.mail.yahoo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Oct 21 23:03:15 2006 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 21 Oct 2006 23:03:15 -0500 Subject: about Toeplitz type matrix In-Reply-To: <20061022034051.75362.qmail@web15813.mail.cnb.yahoo.com> References: <20061022034051.75362.qmail@web15813.mail.cnb.yahoo.com> Message-ID: There are no specific routines for constructing Toeplitz systems. Should be straightforward with MAtSetValues(). Thanks, Matt On 10/21/06, jiaxun hou wrote: > > Hi, > I am using PETSc in my program to solve Toeplitz type systems. Are there > any commands to construct the Toeplitz matrics in a convenient and efficient > way? > > Regards, > Jiaxun > > > ------------------------------ > ????????-3.5G???20M??? > -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sat Oct 21 23:30:55 2006 From: balay at mcs.anl.gov (Satish Balay) Date: Sat, 21 Oct 2006 23:30:55 -0500 (CDT) Subject: about PETSC installation In-Reply-To: <008501c6f583$ebe19bf0$3364a8c0@dmrc6700512> References: <002301c6f4df$186bf350$3364a8c0@dmrc6700512> <004b01c6f57a$75add6a0$3364a8c0@dmrc6700512> <008501c6f583$ebe19bf0$3364a8c0@dmrc6700512> Message-ID: We don't require fortran. Hence the option to turn it off. [--with-fc=0 --download-c-blas-lapack=1]. It appears that you want to use PETSc from C++ with Visual Studio 6.0. [i.e you should use --with-clanguage=cxx ]. In this case [of Visual Studio] its best to avoid fortran compiler completely [Compaq Fortran is the one that goes with VC6.0] [i.e use f2cblas - not fblas - otherwise you'll have to use fortran compiler as linker - even if you are compiling c++ code.] Hence my previous sugestion below: >>>>>>>>>>>>> ./config/configure.py --with-mpi-dir=/cygdrive/d/MPICH2 --with-cc='win32fe cl' --with-fc=0 --with-cxx='win32fe cl' --with-clanguage=cxx --download-c-blas-lapack=1 Check installation instructions for additional info. If you encounter further problems - send us the corresponding configure.log to the address petsc-maint at mcs.anl.gov [not to the users list] >>>>>>>> Satish On Sun, 22 Oct 2006, Yixun Liu wrote: > Thanks. > If I turn off Fortran, does PETSC work well? I think PETSC depends on Fortran because it use LAPACK. > > Regards, > > Yixun > ----- Original Message ----- > From: Matthew Knepley > To: petsc-users at mcs.anl.gov > Sent: Sunday, October 22, 2006 9:54 AM > Subject: Re: about PETSC installation > > > On 10/21/06, Yixun Liu wrote: > Hi, > Thanks for your help. > > Firstly I want to know if the error is caused by no Fortran on my computer? > > Yes, you can turn of Fortran using --with-fc=0. > > Matt > > > > BTW, could you tell me more about the configure option, which you suggest? > > Best regards, > > Yixun > > ----- Original Message ----- > From: "Satish Balay" > To: "PETSC" > Sent: Saturday, October 21, 2006 10:17 PM > Subject: Re: about PETSC installation > > > > On Sat, 21 Oct 2006, Yixun Liu wrote: > > > >> Hi, > >> I install PETSC on WindowsXP, MS VC++6.0, Cygwin, Dell standard dual processors PC, > >> I do as the PETSc installation page says, > >> > >> $ ./config/configure.py --with-mpi-dir=d:/MPICH2 > >> However, the error is "the Fortran error! mpi_init() could not be located!" > >> > >> Then I try this, > >> $ ./config/configure.py --with-mpi-dir=d:/MPICH2 --with-cc=c++ > >> The error is "Unknown Fortran name mangling" > > > > > > You probabably want to do: > > > > ./config/configure.py --with-mpi-dir=/cygdrive/d/MPICH2 --with-cc='win32fe cl' --with-fc=0 --with-cxx='win32fe cl' --with-clanguage=cxx --download-c-blas-lapack=1 > > > > Check installation instructions for additional info. If you encounter > > further problems - send us the corresponding configure.log to the > > address petsc-maint at mcs.anl.gov [not to the users list] > > > > Satish > > > >> > >> Hope your help! > >> > >> Best regards, > >> > >> Yixun Liu > >> > > > > > > > > -- > "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness > From bsmith at mcs.anl.gov Mon Oct 23 15:15:48 2006 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 23 Oct 2006 15:15:48 -0500 (CDT) Subject: about Toeplitz type matrix In-Reply-To: <20061022034051.75362.qmail@web15813.mail.cnb.yahoo.com> References: <20061022034051.75362.qmail@web15813.mail.cnb.yahoo.com> Message-ID: Jiaxun, We don't have any such routines. Often people used specialized Toeplitz solvers that take advantage of very specific matrix structure to perform very quickly. PETSc does not have any of these special solvers or storage formats since we try to provide general purpose solvers. Barry Of course, PETSc is extensible and such solvers could be added and we would gladly accept and maintain solvers people contribute. On Sun, 22 Oct 2006, jiaxun hou wrote: > Hi, I am using PETSc in my program to solve Toeplitz type systems. Are there any commands to construct the Toeplitz matrics in a convenient and efficient way? Regards, Jiaxun ___________________________________________________________ ????????????????-3.5G??????20M?????? http://cn.mail.yahoo.com From yxliu at fudan.edu.cn Mon Oct 23 21:49:20 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Tue, 24 Oct 2006 10:49:20 +0800 Subject: about MatCreateSeqAIJ and MatCreateMPIAIJ Message-ID: <00d801c6f717$015eb410$3364a8c0@dmrc6700512> Hi, I am confused with this two routines. MatCreateMPIAIJ(PETSC_COMM_WORLD,...) creates a parallel matrix,which will be distributed on all the processors. MatCreateSeqAIJ(PETSC_COMM_SELF, ...) creates a sequential matrix. But because this routine will be called by each processor, which processor create this matrix? Best, Yixun -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Oct 23 22:39:58 2006 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 23 Oct 2006 22:39:58 -0500 (CDT) Subject: about MatCreateSeqAIJ and MatCreateMPIAIJ In-Reply-To: <00d801c6f717$015eb410$3364a8c0@dmrc6700512> References: <00d801c6f717$015eb410$3364a8c0@dmrc6700512> Message-ID: On Tue, 24 Oct 2006, Yixun Liu wrote: > Hi, > I am confused with this two routines. > > MatCreateMPIAIJ(PETSC_COMM_WORLD,...) creates a parallel > matrix,which will be distributed on all the > processors. MatCreateSeqAIJ(PETSC_COMM_SELF, ...) creates a > sequential matrix. But because this routine will be called by each > processor, which processor create this matrix? If the routine is called on each processor - then each of them will have a sequential matrix. Generally one uses SeqAIJ when running the application on only one processor. Also one can think of a case where someone might want to store sequential matrices on each processor [the matrix sizes/values being different on each proc] For eg: We internally store MPIAIJ mat as a collection of SeqAIJ matrices on each proc. For additional info - check the 'Example usage' section at: http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJ.html Satish From yxliu at fudan.edu.cn Tue Oct 24 01:31:07 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Tue, 24 Oct 2006 14:31:07 +0800 Subject: about MatCreateSeqAIJ and MatCreateMPIAIJ References: <00d801c6f717$015eb410$3364a8c0@dmrc6700512> Message-ID: <00f801c6f735$fd14bd40$3364a8c0@dmrc6700512> Hi, Thanks. I need two things to be confirmed. For example, I have 4 processors and the codes are like, PetscInitialize(); ... MatCreateSeqAIJ(); ... PetscFinalize(); Can I draw the conclusion that each code between PetscInitialize() and PetscFinalize() will be called by each processor? So, each processor will have a sequential matrix. Best, Yixun ----- Original Message ----- From: "Satish Balay" To: "PETSC" Sent: Tuesday, October 24, 2006 11:39 AM Subject: Re: about MatCreateSeqAIJ and MatCreateMPIAIJ > On Tue, 24 Oct 2006, Yixun Liu wrote: > >> Hi, >> I am confused with this two routines. >> >> MatCreateMPIAIJ(PETSC_COMM_WORLD,...) creates a parallel >> matrix,which will be distributed on all the >> processors. MatCreateSeqAIJ(PETSC_COMM_SELF, ...) creates a >> sequential matrix. But because this routine will be called by each >> processor, which processor create this matrix? > > If the routine is called on each processor - then each of them will > have a sequential matrix. > > Generally one uses SeqAIJ when running the application on only one > processor. Also one can think of a case where someone might want to > store sequential matrices on each processor [the matrix sizes/values > being different on each proc] > > For eg: We internally store MPIAIJ mat as a collection of SeqAIJ > matrices on each proc. For additional info - check the 'Example usage' > section at: > > http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJ.html > > Satish > From jiaxun_hou at yahoo.com.cn Tue Oct 24 02:13:57 2006 From: jiaxun_hou at yahoo.com.cn (jiaxun hou) Date: Tue, 24 Oct 2006 15:13:57 +0800 (CST) Subject: =?big5?Q?=A6=5E=CE=60=A1G_about_Toeplitz_type_matrix?= Message-ID: <20061024071357.29707.qmail@web15803.mail.cnb.yahoo.com> Thanks for your replies. I will make these solvers by myself, and try to contribute them. Regards, Mason ------------------------------------------------------------------------- Jiaxun, We don't have any such routines. Often people used specialized Toeplitz solvers that take advantage of very specific matrix structure to perform very quickly. PETSc does not have any of these special solvers or storage formats since we try to provide general purpose solvers. Barry Of course, PETSc is extensible and such solvers could be added and we would gladly accept and maintain solvers people contribute. On Sun, 22 Oct 2006, jiaxun hou wrote: > Hi, I am using PETSc in my program to solve Toeplitz type systems. Are there any commands to construct the Toeplitz matrics in a convenient and efficient way? Regards, Jiaxun ___________________________________________________________ ????????-3.5G???20M??? http://cn.mail.yahoo.com ___________________________________________________________ ????????-3.5G???20M??? http://cn.mail.yahoo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Oct 24 08:29:44 2006 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 24 Oct 2006 08:29:44 -0500 Subject: about MatCreateSeqAIJ and MatCreateMPIAIJ In-Reply-To: <00f801c6f735$fd14bd40$3364a8c0@dmrc6700512> References: <00d801c6f717$015eb410$3364a8c0@dmrc6700512> <00f801c6f735$fd14bd40$3364a8c0@dmrc6700512> Message-ID: Yes. Matt On 10/24/06, Yixun Liu wrote: > > Hi, > Thanks. > > I need two things to be confirmed. For example, I have 4 processors and > the codes are like, > > PetscInitialize(); > ... > MatCreateSeqAIJ(); > ... > PetscFinalize(); > > Can I draw the conclusion that each code between PetscInitialize() > and PetscFinalize() will be called by each processor? So, each processor > will have a sequential matrix. > > Best, > > Yixun > > > ----- Original Message ----- > From: "Satish Balay" > To: "PETSC" > Sent: Tuesday, October 24, 2006 11:39 AM > Subject: Re: about MatCreateSeqAIJ and MatCreateMPIAIJ > > > > On Tue, 24 Oct 2006, Yixun Liu wrote: > > > >> Hi, > >> I am confused with this two routines. > >> > >> MatCreateMPIAIJ(PETSC_COMM_WORLD,...) creates a parallel > >> matrix,which will be distributed on all the > >> processors. MatCreateSeqAIJ(PETSC_COMM_SELF, ...) creates a > >> sequential matrix. But because this routine will be called by each > >> processor, which processor create this matrix? > > > > If the routine is called on each processor - then each of them will > > have a sequential matrix. > > > > Generally one uses SeqAIJ when running the application on only one > > processor. Also one can think of a case where someone might want to > > store sequential matrices on each processor [the matrix sizes/values > > being different on each proc] > > > > For eg: We internally store MPIAIJ mat as a collection of SeqAIJ > > matrices on each proc. For additional info - check the 'Example usage' > > section at: > > > > > http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJ.html > > > > Satish > > > > -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness -------------- next part -------------- An HTML attachment was scrubbed... URL: From julvar at tamu.edu Tue Oct 24 13:07:15 2006 From: julvar at tamu.edu (Julian) Date: Tue, 24 Oct 2006 13:07:15 -0500 Subject: SeqSBAIJ with cholesky In-Reply-To: Message-ID: <003501c6f797$3c87e860$24b75ba5@aero.ad.tamu.edu> Hi, I am trying to use -pc_type cholesky with a SBAIJ (MatCreateSeqSBAIJ) Unless I specifically ask for it, its not going to reorder the matrix, right? I've been used to calling it 'renumbering' the matrix but I'm guessing its the same thing - rearranging the rows and columns so the bandwidth of the matrix is reduced, am I correct? >From this earlier email, I'm understanding that the sbaij format does not support reordering? Btw, I tried it anyway.. And it gave some segmentation violation error...maybe it can catch the incompatibility earlier on, so that the real problem is more obvious. Do you plan on providing the reordering feature for sbaij in the future? Almost all structural fem codes deal with symmetric positive semi definite matrices. So, that would be a very useful feature... Also why do you say that the aij format is more efficient than the sbaij format? I would have thought utilizing the symmetry would make the sbaij format more efficient. is the sbaij format not supported in parallel? With how it stands right now, is there any other solver supported by petsc (external or otherwise) that lets me use reordering with the symmetric matrix ? Thanks, Julian. > -----Original Message----- > From: owner-petsc-users at mcs.anl.gov > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Hong Zhang > Sent: Wednesday, October 11, 2006 2:14 PM > To: petsc-users at mcs.anl.gov > Subject: RE: General matrix interative solver > > > Julian, > > If you do not perform matrix reordering, you can use petsc > sbaij format, which only stores the upper triangular entries. > aij format has more algorithmic options and is more efficient > and supports matrix reordering. > > Hong From misabel at litec.csic.es Tue Oct 24 12:57:22 2006 From: misabel at litec.csic.es (Isabel Gil) Date: Tue, 24 Oct 2006 19:57:22 +0200 Subject: Undefined symbol in executing time Message-ID: <453E5402.1080005@litec.csic.es> Hello PETSc's users! I am trying to use PETSc for the first time along with another library. I compile my program without any problem but when I try to execute it, in a parallel way, I obtain the following error: "symbol lookup error: /path_where_PETSc_is_installed/petsc-2.2.1/lib/libg_c++/linux-gnu/libpetscmat.so: undefined symbol: PETSC_COMM_SELF" I have used the command "nm" and ,effectively, this symbol is undefined. I don't really know where the problem is. I am using version 2.2.1 Any help and advice will be welcome Thanks in advance Isa From balay at mcs.anl.gov Tue Oct 24 13:38:03 2006 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 24 Oct 2006 13:38:03 -0500 (CDT) Subject: Undefined symbol in executing time In-Reply-To: <453E5402.1080005@litec.csic.es> References: <453E5402.1080005@litec.csic.es> Message-ID: I have the symbol at the following location.. asterix:/home/balay/soft/petsc-2.2.1/lib/libg/asterix>nm -Ao *.a |grep PETSC_COMM_SELF |grep -v ' U ' libpetsc.a:init.o:0000000c B PETSC_COMM_SELF Perhaps there were errors when the libraries were built? Satish On Tue, 24 Oct 2006, Isabel Gil wrote: > Hello PETSc's users! > > I am trying to use PETSc for the first time along with another library. I > compile my program without any problem but when I try to execute it, in a > parallel way, I obtain the following error: > "symbol lookup error: > /path_where_PETSc_is_installed/petsc-2.2.1/lib/libg_c++/linux-gnu/libpetscmat.so: > undefined symbol: PETSC_COMM_SELF" > > I have used the command "nm" and ,effectively, this symbol is undefined. I > don't really know where the problem is. I am using version 2.2.1 > > Any help and advice will be welcome > Thanks in advance > Isa > > From knepley at gmail.com Tue Oct 24 14:17:04 2006 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 24 Oct 2006 14:17:04 -0500 Subject: Undefined symbol in executing time In-Reply-To: <453E5402.1080005@litec.csic.es> References: <453E5402.1080005@litec.csic.es> Message-ID: To look for build errors, consult the file $PETSC_DIR/make_log_*, or send it to us. Thanks, Matt On 10/24/06, Isabel Gil wrote: > > Hello PETSc's users! > > I am trying to use PETSc for the first time along with another library. > I compile my program without any problem but when I try to execute it, > in a parallel way, I obtain the following error: > "symbol lookup error: > /path_where_PETSc_is_installed/petsc-2.2.1 > /lib/libg_c++/linux-gnu/libpetscmat.so: > undefined symbol: PETSC_COMM_SELF" > > I have used the command "nm" and ,effectively, this symbol is undefined. > I don't really know where the problem is. I am using version 2.2.1 > > Any help and advice will be welcome > Thanks in advance > Isa > > -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness -------------- next part -------------- An HTML attachment was scrubbed... URL: From yxliu at fudan.edu.cn Tue Oct 24 21:21:52 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Wed, 25 Oct 2006 10:21:52 +0800 Subject: about option database Message-ID: <002501c6f7dc$572cd6e0$3364a8c0@dmrc6700512> Hi, I am reading the source code of $PETSC_DIR\src\ksp\ksp\examples\tutorials\ex2.c. Some codes are, ... PetscInitialize(&argc,&args,(char *)0,help); ierr = PetscOptionsGetInt(PETSC_NULL,"-m",&m,PETSC_NULL);CHKERRQ(ierr); ... However, I cannot find the default option database file .petscrc in user home directory. Regards, Yixun -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Oct 24 21:31:19 2006 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 24 Oct 2006 21:31:19 -0500 Subject: about option database In-Reply-To: <002501c6f7dc$572cd6e0$3364a8c0@dmrc6700512> References: <002501c6f7dc$572cd6e0$3364a8c0@dmrc6700512> Message-ID: Options can be read from: 1) The command line 2) a ~/.petscrc file 3) the PETSC_OPTIONS environment variable Matt On 10/24/06, Yixun Liu wrote: > > Hi, > > I am reading the source code of > $PETSC_DIR\src\ksp\ksp\examples\tutorials\ex2.c. Some codes are, > ... > PetscInitialize(&argc,&args,(char *)0,help); > ierr = PetscOptionsGetInt(PETSC_NULL,"-m",&m,PETSC_NULL);CHKERRQ(ierr); > ... > However, I cannot find the default option database file .petscrc in user > home directory. > > Regards, > > Yixun > > -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Oct 25 03:57:13 2006 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 25 Oct 2006 03:57:13 -0500 (CDT) Subject: SeqSBAIJ with cholesky In-Reply-To: <003501c6f797$3c87e860$24b75ba5@aero.ad.tamu.edu> References: <003501c6f797$3c87e860$24b75ba5@aero.ad.tamu.edu> Message-ID: Julian, > > I am trying to use -pc_type cholesky with a SBAIJ (MatCreateSeqSBAIJ) > Unless I specifically ask for it, its not going to reorder the matrix, > right? I've been used to calling it 'renumbering' the matrix but I'm > guessing its the same thing - rearranging the rows and columns so the > bandwidth of the matrix is reduced, am I correct? > > From this earlier email, I'm understanding that the sbaij format does not > support reordering? Btw, I tried it anyway.. And it gave some segmentation > violation error...maybe it can catch the incompatibility earlier on, so that > the real problem is more obvious. I wrote matrix reordering for seqsbaij matrix format, but found it has bug when apply cholesky factorization to it. Considering its performance, I feel that the matrix reordering for sbaij matrix should be avoided, and replaced by using aij matrix. The matrix reordering for sbaij matrix requires reordering of entire matrix data structure (rows and columns!), causing wild data accessing and communication, and only gains saving half of the original sparse matrix entries. ^^^^^^^^^^^^^^^^^^^^^ Using aij matrix format, the matrix reordering can be done eficiently. We provide icc and cholesky matrix factorizations for aij format, e.g., preconditioners are stored in sbaij format (half of matrix entries). Note, matrix factors are in general denser than the original sparse matrix. > Do you plan on providing the reordering feature for sbaij in the future? > Almost all structural fem codes deal with symmetric positive semi definite > matrices. So, that would be a very useful feature... > Also why do you say that the aij format is more efficient than the sbaij > format? I would have thought utilizing the symmetry would make the sbaij > format more efficient. > is the sbaij format not supported in parallel? > icc/cholesky preconditioners save half of storage than ilu/lu preconditioner and might be faster. If your have symmetric matrix, use aij format for your matrix, and apply icc/cholesky preconditioner. You can apply any symmetric ordering efficiently to the aij matrix to reduce the fillings in matrix factorization. We don't have plan to support reordering for sbaij matrix - at least it is not our priority. Let us know if you think differently, Hong > With how it stands right now, is there any other solver supported by petsc > (external or otherwise) that lets me use reordering with the symmetric > matrix ? > > Thanks, > Julian. > > > -----Original Message----- > > From: owner-petsc-users at mcs.anl.gov > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Hong Zhang > > Sent: Wednesday, October 11, 2006 2:14 PM > > To: petsc-users at mcs.anl.gov > > Subject: RE: General matrix interative solver > > > > > > Julian, > > > > If you do not perform matrix reordering, you can use petsc > > sbaij format, which only stores the upper triangular entries. > > aij format has more algorithmic options and is more efficient > > and supports matrix reordering. > > > > Hong > > From julvar at tamu.edu Wed Oct 25 11:47:44 2006 From: julvar at tamu.edu (Julian) Date: Wed, 25 Oct 2006 11:47:44 -0500 Subject: SeqSBAIJ with cholesky In-Reply-To: Message-ID: <000001c6f855$4c929020$24b75ba5@aero.ad.tamu.edu> Hong, Thanks for the reply. Can you please clarify some things: I used MatCreateSeqSBAIJ to create the sbaij matrix. Does this mean that the entire matrix is stored under one processor? Does the 'seq' in MatCreateSeqSBAIJ stand for sequential, meaning "not parallel"? If that is the case, can you tell me how I should write it so that, it creates a parallel matrix if I'm using more than one processor(that would be more efficient, right?), and create a sequential matrix if I am using only one processor? > I wrote matrix reordering for seqsbaij matrix format, but > found it has bug when apply cholesky factorization to it. > Considering its performance, I feel that the matrix > reordering for sbaij matrix should be avoided, and replaced > by using aij matrix. > > The matrix reordering for sbaij matrix requires reordering of > entire matrix data structure (rows and columns!), causing > wild data accessing and communication, and only gains saving > half of the original sparse matrix entries. > ^^^^^^^^^^^^^^^^^^^^^ > Using aij matrix format, the matrix reordering can be done eficiently. > We provide icc and cholesky matrix factorizations for aij > format, e.g., preconditioners are stored in sbaij format > (half of matrix entries). > Note, matrix factors are in general denser than the original > sparse matrix. I will check out using the aij matrix but having to store almost double the matrix entries seem inefficient to me. I just took a quick look at the MUMPS solver. It seems like they do have support for distributed direct solving of symmetric matrices with ordering. So, this kind of thing has been done before.. But I don't know about the efficiency because I haven't use MUMPS before.. But I do have a symmetric direct solver (using 1 processor) in my fem code that uses ordering and right now it is a LOT faster that the symmetric solver in petsc. I'm guessing that is because petsc does not support reordering. > icc/cholesky preconditioners save half of storage than ilu/lu > preconditioner and might be faster. If your have symmetric > matrix, use aij format for your matrix, and apply > icc/cholesky preconditioner. > You can apply any symmetric ordering efficiently to the aij > matrix to reduce the fillings in matrix factorization. Can you please clarify that? What do you mean by symmetric ordering? > Let us know if you think differently, I would certainly prefer symmetric solvers that can do reordering. But I'll let you know what I come up with when I use aij with cholesky. Thanks, Julian. From yxliu at fudan.edu.cn Wed Oct 25 20:44:44 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Thu, 26 Oct 2006 09:44:44 +0800 Subject: about MPI_Comm_size(PETSC_COMM_WORLD,&size) Message-ID: <001101c6f8a0$503cb750$3364a8c0@dmrc6700512> Hi, My computer is a standare dual processors PC of Dell corp., but after calling ierr = PI_Comm_size(PETSC_COMM_WORLD,&size), I find the size is one. Is it correct? Best, Yixun -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Oct 25 20:47:00 2006 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 25 Oct 2006 20:47:00 -0500 Subject: about MPI_Comm_size(PETSC_COMM_WORLD,&size) In-Reply-To: <001101c6f8a0$503cb750$3364a8c0@dmrc6700512> References: <001101c6f8a0$503cb750$3364a8c0@dmrc6700512> Message-ID: On 10/25/06, Yixun Liu wrote: > > Hi, > > My computer is a standare dual processors PC of Dell corp., but after > calling ierr = PI_Comm_size(PETSC_COMM_WORLD,&size), I find the size is > one. Is it correct? > MPI only knows what mpiruns tells it. You ran the exampe with -np 1. If you use -np 2, then it will have size 2. Matt Best, > > Yixun > -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness -------------- next part -------------- An HTML attachment was scrubbed... URL: From yxliu at fudan.edu.cn Wed Oct 25 21:18:43 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Thu, 26 Oct 2006 10:18:43 +0800 Subject: about MPI_Comm_size(PETSC_COMM_WORLD,&size) References: <001101c6f8a0$503cb750$3364a8c0@dmrc6700512> Message-ID: <001f01c6f8a5$0f792e10$3364a8c0@dmrc6700512> Hi, Can I think that if I set -np 100, but only have 2 actually, the size will be 100? Another question,I run the progarm from VC++6.0 instead of command line. How to make the program run on 2 processors? Best, Yixun ----- Original Message ----- From: Matthew Knepley To: petsc-users at mcs.anl.gov Sent: Thursday, October 26, 2006 9:47 AM Subject: Re: about MPI_Comm_size(PETSC_COMM_WORLD,&size) On 10/25/06, Yixun Liu wrote: Hi, My computer is a standare dual processors PC of Dell corp., but after calling ierr = PI_Comm_size(PETSC_COMM_WORLD,&size), I find the size is one. Is it correct? MPI only knows what mpiruns tells it. You ran the exampe with -np 1. If you use -np 2, then it will have size 2. Matt Best, Yixun -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Oct 25 21:24:11 2006 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 25 Oct 2006 21:24:11 -0500 Subject: about MPI_Comm_size(PETSC_COMM_WORLD,&size) In-Reply-To: <001f01c6f8a5$0f792e10$3364a8c0@dmrc6700512> References: <001101c6f8a0$503cb750$3364a8c0@dmrc6700512> <001f01c6f8a5$0f792e10$3364a8c0@dmrc6700512> Message-ID: On 10/25/06, Yixun Liu wrote: > > Hi, > > Can I think that if I set -np 100, but only have 2 actually, the size will > be 100? > Yes. This counts processes, not processors. Another question,I run the progarm from VC++6.0 instead of command line. How > to make the program run on 2 processors? > You use the mpirun for the MPI you are using. I think you should get a book on MPI. I recommend the one by Gropp, et. al. It is had to use PETSc without these basic MPI notions, such as communicator, process, etc. Matt Best, > > Yixun > > ----- Original Message ----- > *From:* Matthew Knepley > *To:* petsc-users at mcs.anl.gov > *Sent:* Thursday, October 26, 2006 9:47 AM > *Subject:* Re: about MPI_Comm_size(PETSC_COMM_WORLD,&size) > > On 10/25/06, Yixun Liu wrote: > > > > Hi, > > > > My computer is a standare dual processors PC of Dell corp., but after > > calling ierr = PI_Comm_size(PETSC_COMM_WORLD,&size), I find the size is > > one. Is it correct? > > > > MPI only knows what mpiruns tells it. You ran the exampe with -np 1. If > you use -np 2, then it will have size 2. > > Matt > > Best, > > > > Yixun > > > -- > "Failure has a thousand explanations. Success doesn't need one" -- Sir > Alec Guiness > > -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness -------------- next part -------------- An HTML attachment was scrubbed... URL: From charden at scs.fsu.edu Wed Oct 25 21:37:00 2006 From: charden at scs.fsu.edu (Chris Harden) Date: Wed, 25 Oct 2006 22:37:00 -0400 Subject: about MPI_Comm_size(PETSC_COMM_WORLD,&size) In-Reply-To: <001f01c6f8a5$0f792e10$3364a8c0@dmrc6700512> References: <001101c6f8a0$503cb750$3364a8c0@dmrc6700512> <001f01c6f8a5$0f792e10$3364a8c0@dmrc6700512> Message-ID: <78F1621B-478B-4FA8-9E1D-CE5B5148A537@scs.fsu.edu> Hello, I think VC is ok for editing (maybe), but PETSc and mpich are easier to work with in the command line. You might save yourself some headache. Just paste your file into a plain text.cpp and put it in a directory such as your desktop. Then cd to your desktop and compile from the command line. I promise your life will be simpler in the long term. best, Chris On Oct 25, 2006, at 10:18 PM, Yixun Liu wrote: > Hi, > > Can I think that if I set -np 100, but only have 2 actually, the > size will be 100? > Another question,I run the progarm from VC++6.0 instead of command > line. How to make the program run on 2 processors? > > Best, > > Yixun > ----- Original Message ----- > From: Matthew Knepley > To: petsc-users at mcs.anl.gov > Sent: Thursday, October 26, 2006 9:47 AM > Subject: Re: about MPI_Comm_size(PETSC_COMM_WORLD,&size) > > On 10/25/06, Yixun Liu wrote: > Hi, > > My computer is a standare dual processors PC of Dell corp., but > after calling ierr = PI_Comm_size(PETSC_COMM_WORLD,&size), I find > the size is one. Is it correct? > > MPI only knows what mpiruns tells it. You ran the exampe with -np > 1. If you use -np 2, then it will have size 2. > > Matt > > Best, > > Yixun > -- > "Failure has a thousand explanations. Success doesn't need one" -- > Sir Alec Guiness > Chris Harden Graduate Assistant School of Computational Science Florida State University 446-B Dirac Science Library Tallahassee, Florida 32306-4120 Voice: 850.644.3442 E-mail: charden at scs.fsu.edu http://www.scs.fsu.edu/~charden/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From yxliu at fudan.edu.cn Thu Oct 26 00:09:15 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Thu, 26 Oct 2006 13:09:15 +0800 Subject: about MPI_Comm_size(PETSC_COMM_WORLD,&size) References: <001101c6f8a0$503cb750$3364a8c0@dmrc6700512> <001f01c6f8a5$0f792e10$3364a8c0@dmrc6700512> <78F1621B-478B-4FA8-9E1D-CE5B5148A537@scs.fsu.edu> Message-ID: <004501c6f8bc$e2387a10$3364a8c0@dmrc6700512> Hi, Can I compiler and link it in VC enviroment and then copy the .exe to a directory. Then, at command line run mpiexec -n 2 xxx.exe? I try it, but the error is "Credentials for yxliu rejected connecting to dmrc-6700512 Aborting: Unable to connect to dmrc-6700512". I have used wmpiregister to register the accout correctly. Best, Yixun ----- Original Message ----- From: Chris Harden To: petsc-users at mcs.anl.gov Sent: Thursday, October 26, 2006 10:37 AM Subject: Re: about MPI_Comm_size(PETSC_COMM_WORLD,&size) Hello, I think VC is ok for editing (maybe), but PETSc and mpich are easier to work with in the command line. You might save yourself some headache. Just paste your file into a plain text.cpp and put it in a directory such as your desktop. Then cd to your desktop and compile from the command line. I promise your life will be simpler in the long term. best, Chris On Oct 25, 2006, at 10:18 PM, Yixun Liu wrote: Hi, ? Can I think that if I set -np 100, but only have 2 actually, the size will be 100? Another question,I run? the progarm? from VC++6.0 instead of command line. How to make the program run on 2 processors? ? Best, ? Yixun ----- Original Message ----- From: Matthew Knepley To: petsc-users at mcs.anl.gov Sent: Thursday, October 26, 2006 9:47 AM Subject: Re: about MPI_Comm_size(PETSC_COMM_WORLD,&size) On 10/25/06,? Yixun Liu? wrote: Hi, ? My computer is a standare dual processors PC of Dell corp., but after calling ? ierr = PI_Comm_size(PETSC_COMM_WORLD,&size), I find the size is one. Is it correct? MPI only knows what mpiruns tells it. You ran the exampe with -np 1. If you use -np 2, then it will have size 2. ? Matt Best, ? Yixun --? "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness Chris Harden Graduate Assistant School of Computational Science Florida State University 446-B Dirac Science Library Tallahassee, Florida 32306-4120 Voice: 850.644.3442 E-mail: charden at scs.fsu.edu http://www.scs.fsu.edu/~charden/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Oct 26 00:21:38 2006 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 26 Oct 2006 00:21:38 -0500 (CDT) Subject: about MPI_Comm_size(PETSC_COMM_WORLD,&size) In-Reply-To: <004501c6f8bc$e2387a10$3364a8c0@dmrc6700512> References: <001101c6f8a0$503cb750$3364a8c0@dmrc6700512> <001f01c6f8a5$0f792e10$3364a8c0@dmrc6700512> <78F1621B-478B-4FA8-9E1D-CE5B5148A537@scs.fsu.edu> <004501c6f8bc$e2387a10$3364a8c0@dmrc6700512> Message-ID: On Thu, 26 Oct 2006, Yixun Liu wrote: > Hi, > Can I compiler and link it in VC enviroment and then copy the .exe to a directory. you can just go to the dir which has the executable. [copy is not necessary] > Then, at command line run mpiexec -n 2 xxx.exe? yes > I try it, but the error is "Credentials for yxliu rejected > connecting to dmrc-6700512 Aborting: Unable to connect to > dmrc-6700512". I have used wmpiregister to register the accout > correctly. Sounds like your mpi install is not setup properly. I'm guessing you are using mpich2. I don't know enough about this version. Can you check if any of the following work? executable mpiexec -n 1 executable mpiexec -n 2 executable mpiexec -n 2 -localonly executable Satish From yxliu at fudan.edu.cn Thu Oct 26 00:54:58 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Thu, 26 Oct 2006 13:54:58 +0800 Subject: about MPI_Comm_size(PETSC_COMM_WORLD,&size) References: <001101c6f8a0$503cb750$3364a8c0@dmrc6700512> <001f01c6f8a5$0f792e10$3364a8c0@dmrc6700512> <78F1621B-478B-4FA8-9E1D-CE5B5148A537@scs.fsu.edu> <004501c6f8bc$e2387a10$3364a8c0@dmrc6700512> Message-ID: <004d01c6f8c3$44e7f040$3364a8c0@dmrc6700512> Hi, I tried what you said and mpiexec - n 2 localonly is ok. Do I need to reinstall the MPICH2? Best, Yixun ----- Original Message ----- From: "Satish Balay" To: Sent: Thursday, October 26, 2006 1:21 PM Subject: Re: about MPI_Comm_size(PETSC_COMM_WORLD,&size) > > > On Thu, 26 Oct 2006, Yixun Liu wrote: > >> Hi, >> Can I compiler and link it in VC enviroment and then copy the .exe to a directory. > > you can just go to the dir which has the executable. [copy is not necessary] > >> Then, at command line run mpiexec -n 2 xxx.exe? > > yes > >> I try it, but the error is "Credentials for yxliu rejected >> connecting to dmrc-6700512 Aborting: Unable to connect to >> dmrc-6700512". I have used wmpiregister to register the accout >> correctly. > > > Sounds like your mpi install is not setup properly. I'm guessing > you are using mpich2. I don't know enough about this version. > > Can you check if any of the following work? > executable > mpiexec -n 1 executable > mpiexec -n 2 executable > mpiexec -n 2 -localonly executable > > Satish > From balay at mcs.anl.gov Thu Oct 26 01:19:37 2006 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 26 Oct 2006 01:19:37 -0500 (CDT) Subject: about MPI_Comm_size(PETSC_COMM_WORLD,&size) In-Reply-To: <004d01c6f8c3$44e7f040$3364a8c0@dmrc6700512> References: <001101c6f8a0$503cb750$3364a8c0@dmrc6700512> <001f01c6f8a5$0f792e10$3364a8c0@dmrc6700512> <78F1621B-478B-4FA8-9E1D-CE5B5148A537@scs.fsu.edu> <004501c6f8bc$e2387a10$3364a8c0@dmrc6700512> <004d01c6f8c3$44e7f040$3364a8c0@dmrc6700512> Message-ID: As long as you plan to run multiple MPI procs on the same machine, 'localonly' option is appropriate and should be fine. If you wish to avoid using this option - then you probably need to run the mpd configuration tool and set things up correctly [don't know exactly what that would be]. Its certainly not something which requires a reinstall of mpich. Satish On Thu, 26 Oct 2006, Yixun Liu wrote: > Hi, > > I tried what you said and mpiexec - n 2 localonly is ok. Do I need to reinstall the MPICH2? > > Best, > > Yixun > ----- Original Message ----- > From: "Satish Balay" > To: > Sent: Thursday, October 26, 2006 1:21 PM > Subject: Re: about MPI_Comm_size(PETSC_COMM_WORLD,&size) > > > > > > > > On Thu, 26 Oct 2006, Yixun Liu wrote: > > > >> Hi, > >> Can I compiler and link it in VC enviroment and then copy the .exe to a directory. > > > > you can just go to the dir which has the executable. [copy is not necessary] > > > >> Then, at command line run mpiexec -n 2 xxx.exe? > > > > yes > > > >> I try it, but the error is "Credentials for yxliu rejected > >> connecting to dmrc-6700512 Aborting: Unable to connect to > >> dmrc-6700512". I have used wmpiregister to register the accout > >> correctly. > > > > > > Sounds like your mpi install is not setup properly. I'm guessing > > you are using mpich2. I don't know enough about this version. > > > > Can you check if any of the following work? > > executable > > mpiexec -n 1 executable > > mpiexec -n 2 executable > > mpiexec -n 2 -localonly executable > > > > Satish > > > > From yxliu at fudan.edu.cn Thu Oct 26 04:19:36 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Thu, 26 Oct 2006 17:19:36 +0800 Subject: how many processes is better? Message-ID: <000a01c6f8df$db717880$3364a8c0@dmrc6700512> Hi, I want to know how many processes is better. My computer is dual processors PC with superthread. According to the exampe ex2 I set the problem domain to 800*200. I set the number of processes to 1,2,4,10 respecively and the runtime is 26, 19, 18, 15 respectively. How many processes is better. In my opinion 4(2processors * 2superthreads) is better based on my computer, but why 10 ran more quickly? Best, Yixun -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Oct 26 07:53:16 2006 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 26 Oct 2006 07:53:16 -0500 Subject: how many processes is better? In-Reply-To: <000a01c6f8df$db717880$3364a8c0@dmrc6700512> References: <000a01c6f8df$db717880$3364a8c0@dmrc6700512> Message-ID: On 10/26/06, Yixun Liu wrote: > > Hi, > I want to know how many processes is better. My computer is dual > processors PC with superthread. According to the exampe ex2 I set the > problem domain to 800*200. I set the number of processes to 1,2,4,10 > respecively and the runtime is 26, 19, 18, 15 respectively. How many > processes is better. In my opinion 4(2processors * 2superthreads) is better > based on my computer, but why 10 ran more quickly? > Again, this question is about gneric theory of parallel processing, and really has not answer as you pose it. I suggest the book "Parallel Computing" by Grama, et. al. Matt Best, > > Yixun > -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness -------------- next part -------------- An HTML attachment was scrubbed... URL: From yxliu at fudan.edu.cn Thu Oct 26 19:17:33 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Fri, 27 Oct 2006 08:17:33 +0800 Subject: how many processes is better? References: <000a01c6f8df$db717880$3364a8c0@dmrc6700512> Message-ID: <000b01c6f95d$4c7ad170$3364a8c0@dmrc6700512> Thanks. I will read it:-) ----- Original Message ----- From: Matthew Knepley To: petsc-users at mcs.anl.gov Sent: Thursday, October 26, 2006 8:53 PM Subject: Re: how many processes is better? On 10/26/06, Yixun Liu wrote: Hi, I want to know how many processes is better. My computer is dual processors PC with superthread. According to the exampe ex2 I set the problem domain to 800*200. I set the number of processes to 1,2,4,10 respecively and the runtime is 26, 19, 18, 15 respectively. How many processes is better. In my opinion 4(2processors * 2superthreads) is better based on my computer, but why 10 ran more quickly? Again, this question is about gneric theory of parallel processing, and really has not answer as you pose it. I suggest the book "Parallel Computing" by Grama, et. al. Matt Best, Yixun -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness -------------- next part -------------- An HTML attachment was scrubbed... URL: From yxliu at fudan.edu.cn Sat Oct 28 02:54:16 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Sat, 28 Oct 2006 15:54:16 +0800 Subject: does PETSC use template? Message-ID: <000801c6fa66$44856980$3364a8c0@dmrc6700512> Hi, Does PETSC use template? I think it does not use it because it implements with c. But, why vc++ compiler generate the error "error C2894: templates cannot be declared to have 'C' linkage" when I write the code as, #ifdef __cplusplus extern "C"{ #endif #include #include #include #include #ifdef __cplusplus } #endif Best, Yixun -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Oct 28 08:40:22 2006 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 28 Oct 2006 08:40:22 -0500 Subject: does PETSC use template? In-Reply-To: <000801c6fa66$44856980$3364a8c0@dmrc6700512> References: <000801c6fa66$44856980$3364a8c0@dmrc6700512> Message-ID: On 10/28/06, Yixun Liu wrote: > > > Hi, > > Does PETSC use template? I think it does not use it because it implements > with c. But, why vc++ compiler generate the error "error C2894: templates > cannot be declared to have 'C' linkage" when I write the code as, PETSc has C and C++. You should not use this construction to get C linkage for PETSc. If you configured using --with-clanguage=c it already is, and if you used --with-clanguage=c++ you need to add --with-c-support. > #ifdef __cplusplus > extern "C"{ > #endif > > #include > #include > #include > #include > > > #ifdef __cplusplus > } > #endif Matt > Best, > > Yixun > -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness From yxliu at fudan.edu.cn Sat Oct 28 20:12:37 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Sun, 29 Oct 2006 09:12:37 +0800 Subject: why i cannot find PetscMapCreateMPI() Message-ID: <002601c6fb06$0fb84d30$3364a8c0@dmrc6700512> Hi, I read a source code,which is compilered successfully on Linux, Gcc. When I migrate it from Linux to Windows compiler cannot find the routine PetscMapCreateMPI(). I search from the manulpages and cannot find it. Does this routine is only a Linux version? Regards, Yixun -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Oct 28 22:05:12 2006 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 28 Oct 2006 22:05:12 -0500 (CDT) Subject: why i cannot find PetscMapCreateMPI() In-Reply-To: <002601c6fb06$0fb84d30$3364a8c0@dmrc6700512> References: <002601c6fb06$0fb84d30$3364a8c0@dmrc6700512> Message-ID: Parts of the PETSc API changes between releases; this is fundamental to PETSc development and is clearly stated in the users manual. Thus when you link against PETSc libraries you have to make sure you are consistent. In http://www-unix.mcs.anl.gov/petsc/petsc-as/documentation/changes/231.html you will find the note: PetscMap is no longer a public PETSc object See the manual page for PetscMapInitialize() for its new usage http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Vec/PetscMapInitialize.html Barry On Sun, 29 Oct 2006, Yixun Liu wrote: > Hi, > > I read a source code,which is compilered successfully on Linux, Gcc. When I migrate it from Linux to Windows compiler cannot find the routine PetscMapCreateMPI(). I search from the manulpages and cannot find it. Does this routine is only a Linux version? > > Regards, > > Yixun > From yxliu at fudan.edu.cn Sat Oct 28 22:09:55 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Sun, 29 Oct 2006 11:09:55 +0800 Subject: about runtime lib conflict Message-ID: <002f01c6fb07$b5bd4220$3364a8c0@dmrc6700512> Hi, I use PETSC with other libs, but they use different runtime lib. I hope to config PETSC with Debug multithread dll runtime lib. How to do it? Best, Yixun -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sun Oct 29 01:42:39 2006 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 29 Oct 2006 01:42:39 -0500 (CDT) Subject: about runtime lib conflict In-Reply-To: <002f01c6fb07$b5bd4220$3364a8c0@dmrc6700512> References: <002f01c6fb07$b5bd4220$3364a8c0@dmrc6700512> Message-ID: On Sun, 29 Oct 2006, Yixun Liu wrote: > Hi, > > I use PETSC with other libs, but they use different runtime lib. I don't understand this statement. > I hope to config PETSC with Debug multithread dll runtime lib. How > to do it? 1. Build PETSc libraries with the default configure options [The defaults enable Debug, multithread, dll options for MS compilers.] 2. Build the libraries 3. Verify 'make test' works 4. If you need to use these libraries from MS Project File, follow instructions in the section 'Project Files' at http://www-unix.mcs.anl.gov/petsc/petsc-as/documentation/installation.html From yxliu at fudan.edu.cn Sun Oct 29 01:28:07 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Sun, 29 Oct 2006 15:28:07 +0800 Subject: about runtime lib conflict References: <002f01c6fb07$b5bd4220$3364a8c0@dmrc6700512> Message-ID: <003b01c6fb2b$c7942c60$3364a8c0@dmrc6700512> Hi, In my application I use two libs. One uses run time lib MSVCPRTD and PETSC uses run time lib LIBCPMTD. The run time lib is determined by the compiler option. The two libs will use different run time libs when they use different compiler option. But this will cause lib conflict. So I need to config the PETSC to let it use the same run time lib with the other. Best, Yixun ----- Original Message ----- From: "Satish Balay" To: "PETSC" Sent: Sunday, October 29, 2006 2:42 PM Subject: Re: about runtime lib conflict > On Sun, 29 Oct 2006, Yixun Liu wrote: > >> Hi, >> > >> I use PETSC with other libs, but they use different runtime lib. > > I don't understand this statement. > >> I hope to config PETSC with Debug multithread dll runtime lib. How >> to do it? > > 1. Build PETSc libraries with the default configure options > [The defaults enable Debug, multithread, dll options for MS compilers.] > > 2. Build the libraries > > 3. Verify 'make test' works > > 4. If you need to use these libraries from MS Project File, follow > instructions in the section 'Project Files' at > http://www-unix.mcs.anl.gov/petsc/petsc-as/documentation/installation.html > From balay at mcs.anl.gov Sun Oct 29 14:58:08 2006 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 29 Oct 2006 14:58:08 -0600 (CST) Subject: about runtime lib conflict In-Reply-To: <003b01c6fb2b$c7942c60$3364a8c0@dmrc6700512> References: <002f01c6fb07$b5bd4220$3364a8c0@dmrc6700512> <003b01c6fb2b$c7942c60$3364a8c0@dmrc6700512> Message-ID: >From http://support.microsoft.com/kb/154419/ MSVCRTD.lib: Debug multithreaded DLL (MDd) Basic C Runtime library LIBCPMTD.lib: Debug multithreaded (MTd) Standard C++ library We default to using ' -MT -Z7' compiler options. But looks like you need '-MD -Z7' You can configure PETSc with the additional options. COPTFLAGS='-MD -Z7' CXXOPTFLAGS='-MD -Z7' Satish On Sun, 29 Oct 2006, Yixun Liu wrote: > Hi, > > In my application I use two libs. One uses run time lib MSVCPRTD and PETSC uses run time lib LIBCPMTD. The run time lib is determined by the compiler option. The two libs will use different run time libs when they use different compiler option. But this will cause lib conflict. So I need to config the PETSC to let it use the same run time lib with the other. > > Best, > > Yixun > > ----- Original Message ----- > From: "Satish Balay" > To: "PETSC" > Sent: Sunday, October 29, 2006 2:42 PM > Subject: Re: about runtime lib conflict > > > > On Sun, 29 Oct 2006, Yixun Liu wrote: > > > >> Hi, > >> > > > >> I use PETSC with other libs, but they use different runtime lib. > > > > I don't understand this statement. > > > >> I hope to config PETSC with Debug multithread dll runtime lib. How > >> to do it? > > > > 1. Build PETSc libraries with the default configure options > > [The defaults enable Debug, multithread, dll options for MS compilers.] > > > > 2. Build the libraries > > > > 3. Verify 'make test' works > > > > 4. If you need to use these libraries from MS Project File, follow > > instructions in the section 'Project Files' at > > http://www-unix.mcs.anl.gov/petsc/petsc-as/documentation/installation.html > > > > From billy at dem.uminho.pt Mon Oct 30 22:39:03 2006 From: billy at dem.uminho.pt (billy at dem.uminho.pt) Date: Tue, 31 Oct 2006 04:39:03 +0000 Subject: Matrix convergence - part 2 Message-ID: <1162269543.4546d3677a181@serv-g1.ccom.uminho.pt> Hi, When I use the option -sub_pc_type ilu or no option, I can't lower the residual of the pressure which is derived from a Laplace equation to the specified tolerance of 1E-8: Matrix p Number of iterations: 8 Residual: +3.971771E-01 On the other hand if I use option -sub_pc_type lu, I get: Matrix p Number of iterations: 1 Residual: +6.174315E-24 Since this only happens in some cases, I would like to switch between ILU and LU preconditioning. I tried that by adding the following code: PCASMGetSubKSP(pc, &nlocal, &first, &subksp); KSPGetPC(subksp, &subpc); PCSetType(subpc, SubPrecondProc); Being SubPrecondProc = PCILU or SubPrecondProc = PCLU. However I get the following error: 0]PETSC ERROR: PCASMGetSubKSP_ASM() line 488 in src/ksp/pc/impls/asm/asm.c [0]PETSC ERROR: Operation done in wrong order! [0]PETSC ERROR: Need to call PCSetUP() on PC (or KSPSetUp() on the outer KSP object) before calling here! [0]PETSC ERROR: PCASMGetSubKSP() line 786 in src/ksp/pc/impls/asm/asm.c [0]PETSC ERROR: KSPGetPC() line 1135 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: Corrupt argument: see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Corrupt! [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1! [0]PETSC ERROR: PCSetType() line 61 in src/ksp/pc/interface/pcset.c [0]PETSC ERROR: Invalid argument! [0]PETSC ERROR: Wrong type of object: Parameter # 1! Matrix s Number of iterations: 0 Residual: +0.000000E+00 [0]PETSC ERROR: PCASMGetSubKSP_ASM() line 488 in src/ksp/pc/impls/asm/asm.c [0]PETSC ERROR: Operation done in wrong order! [0]PETSC ERROR: Need to call PCSetUP() on PC (or KSPSetUp() on the outer KSP object) before calling here! [0]PETSC ERROR: PCASMGetSubKSP() line 786 in src/ksp/pc/impls/asm/asm.c [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or try http://valgrind.org on linux to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------- Stack Frames --------------- [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] KSPGetPC line 1134 src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] PCASMGetSubKSP_ASM line 486 src/ksp/pc/impls/asm/asm.c [0]PETSC ERROR: [0] PCASMGetSubKSP line 782 src/ksp/pc/impls/asm/asm.c [0]PETSC ERROR: -------------------------------------------- [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ! Anyone knows why? Thanks, Billy. From yxliu at fudan.edu.cn Tue Oct 31 02:37:08 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Tue, 31 Oct 2006 16:37:08 +0800 Subject: about runtime lib conflict References: <002f01c6fb07$b5bd4220$3364a8c0@dmrc6700512> <003b01c6fb2b$c7942c60$3364a8c0@dmrc6700512> Message-ID: <000701c6fcc7$c0774050$3364a8c0@dmrc6700512> Hi, Should be COPTFLAGS='-MDd -Z7' CXXOPTFLAGS='-MDd -Z7' due to debug version? Best, Yixun ----- Original Message ----- From: "Satish Balay" To: Sent: Monday, October 30, 2006 4:58 AM Subject: Re: about runtime lib conflict > From http://support.microsoft.com/kb/154419/ > > MSVCRTD.lib: Debug multithreaded DLL (MDd) Basic C Runtime library > LIBCPMTD.lib: Debug multithreaded (MTd) Standard C++ library > > We default to using ' -MT -Z7' compiler options. But looks like > you need '-MD -Z7' > > You can configure PETSc with the additional options. > > COPTFLAGS='-MD -Z7' CXXOPTFLAGS='-MD -Z7' > > Satish > > On Sun, 29 Oct 2006, Yixun Liu wrote: > >> Hi, >> >> In my application I use two libs. One uses run time lib MSVCPRTD and PETSC uses run time lib LIBCPMTD. The run time lib is determined by the compiler option. The two libs will use different run time libs when they use different compiler option. But this will cause lib conflict. So I need to config the PETSC to let it use the same run time lib with the other. >> >> Best, >> >> Yixun >> >> ----- Original Message ----- >> From: "Satish Balay" >> To: "PETSC" >> Sent: Sunday, October 29, 2006 2:42 PM >> Subject: Re: about runtime lib conflict >> >> >> > On Sun, 29 Oct 2006, Yixun Liu wrote: >> > >> >> Hi, >> >> >> > >> >> I use PETSC with other libs, but they use different runtime lib. >> > >> > I don't understand this statement. >> > >> >> I hope to config PETSC with Debug multithread dll runtime lib. How >> >> to do it? >> > >> > 1. Build PETSc libraries with the default configure options >> > [The defaults enable Debug, multithread, dll options for MS compilers.] >> > >> > 2. Build the libraries >> > >> > 3. Verify 'make test' works >> > >> > 4. If you need to use these libraries from MS Project File, follow >> > instructions in the section 'Project Files' at >> > http://www-unix.mcs.anl.gov/petsc/petsc-as/documentation/installation.html >> > >> >> > From bsmith at mcs.anl.gov Tue Oct 31 07:56:54 2006 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 31 Oct 2006 07:56:54 -0600 (CST) Subject: Matrix convergence - part 2 In-Reply-To: <1162269543.4546d3677a181@serv-g1.ccom.uminho.pt> References: <1162269543.4546d3677a181@serv-g1.ccom.uminho.pt> Message-ID: Billy, Just do what it says, Call PCSetUp(pc) before PCASMGetSubKSP(). Barry On Tue, 31 Oct 2006, billy at dem.uminho.pt wrote: > > Hi, > > When I use the option -sub_pc_type ilu or no option, I can't lower the residual > of the pressure which is derived from a Laplace equation to the specified > tolerance of 1E-8: > > Matrix p Number of iterations: 8 Residual: +3.971771E-01 > > On the other hand if I use option -sub_pc_type lu, I get: > > Matrix p Number of iterations: 1 Residual: +6.174315E-24 > > Since this only happens in some cases, I would like to switch between ILU and LU > preconditioning. > > I tried that by adding the following code: > > PCASMGetSubKSP(pc, &nlocal, &first, &subksp); > > KSPGetPC(subksp, &subpc); > > PCSetType(subpc, SubPrecondProc); > > > Being SubPrecondProc = PCILU or SubPrecondProc = PCLU. However I get the > following error: > > 0]PETSC ERROR: PCASMGetSubKSP_ASM() line 488 in src/ksp/pc/impls/asm/asm.c > [0]PETSC ERROR: Operation done in wrong order! > [0]PETSC ERROR: Need to call PCSetUP() on PC (or KSPSetUp() on the outer KSP > object) before calling here! > [0]PETSC ERROR: PCASMGetSubKSP() line 786 in src/ksp/pc/impls/asm/asm.c > [0]PETSC ERROR: KSPGetPC() line 1135 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: Corrupt argument: see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Corrupt! > [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1! > [0]PETSC ERROR: PCSetType() line 61 in src/ksp/pc/interface/pcset.c > [0]PETSC ERROR: Invalid argument! > [0]PETSC ERROR: Wrong type of object: Parameter # 1! > > Matrix s Number of iterations: 0 Residual: +0.000000E+00 > [0]PETSC ERROR: PCASMGetSubKSP_ASM() line 488 in src/ksp/pc/impls/asm/asm.c > [0]PETSC ERROR: Operation done in wrong order! > [0]PETSC ERROR: Need to call PCSetUP() on PC (or KSPSetUp() on the outer KSP > object) before calling here! > [0]PETSC ERROR: PCASMGetSubKSP() line 786 in src/ksp/pc/impls/asm/asm.c > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably > memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or try http://valgrind.org on linux to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------- Stack Frames --------------- > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] KSPGetPC line 1134 src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] PCASMGetSubKSP_ASM line 486 src/ksp/pc/impls/asm/asm.c > [0]PETSC ERROR: [0] PCASMGetSubKSP line 782 src/ksp/pc/impls/asm/asm.c > [0]PETSC ERROR: -------------------------------------------- > [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: ! > > Anyone knows why? > > Thanks, > > Billy. > > > From balay at mcs.anl.gov Tue Oct 31 09:03:24 2006 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 31 Oct 2006 09:03:24 -0600 (CST) Subject: about runtime lib conflict In-Reply-To: <000701c6fcc7$c0774050$3364a8c0@dmrc6700512> References: <002f01c6fb07$b5bd4220$3364a8c0@dmrc6700512> <003b01c6fb2b$c7942c60$3364a8c0@dmrc6700512> <000701c6fcc7$c0774050$3364a8c0@dmrc6700512> Message-ID: On Tue, 31 Oct 2006, Yixun Liu wrote: > Hi, > > Should be COPTFLAGS='-MDd -Z7' CXXOPTFLAGS='-MDd -Z7' due to debug version? -Z7 already indecates debugging - so both should be fine.. Satish From hzhang at mcs.anl.gov Tue Oct 31 10:16:39 2006 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 31 Oct 2006 10:16:39 -0600 (CST) Subject: SeqSBAIJ with cholesky In-Reply-To: <000001c6f855$4c929020$24b75ba5@aero.ad.tamu.edu> References: <000001c6f855$4c929020$24b75ba5@aero.ad.tamu.edu> Message-ID: Julian, > > I used MatCreateSeqSBAIJ to create the sbaij matrix. Does this mean that the > entire matrix is stored under one processor? Does the 'seq' in Yes. If you use more than one processors, each processor creates its own entire matrix, which can have different sizes and values. > MatCreateSeqSBAIJ stand for sequential, meaning "not parallel"? Yes. > If that is the case, can you tell me how I should write it so that, it > creates a parallel matrix if I'm using more than one processor(that would be > more efficient, right?), and create a sequential matrix if I am using only > one processor? You may look at example ~petsc/src/ksp/ksp/examples/tutorials/ex5.c on how to create a mpi matrix. The default matrix format is aij. You can use runtime option '-mat_type sbaij -mat_ignore_lower_triangular' or calling MatSetType(sA,MATSBAIJ) in your code to set sbaij format. In this case, all processors jointly create a single parallel matrix with block rows distributed into each processor. > > > > The matrix reordering for sbaij matrix requires reordering of > > entire matrix data structure (rows and columns!), causing > > wild data accessing and communication, and only gains saving > > half of the original sparse matrix entries. > > ^^^^^^^^^^^^^^^^^^^^^ > > Using aij matrix format, the matrix reordering can be done eficiently. > > We provide icc and cholesky matrix factorizations for aij > > format, e.g., preconditioners are stored in sbaij format > > (half of matrix entries). > > Note, matrix factors are in general denser than the original > > sparse matrix. > > I will check out using the aij matrix but having to store almost double the > matrix entries seem inefficient to me. Again, you only store entire original sparse matrix, its nonzeros is much less than nozeros in the matrix factorizations or preconditioners in general. Using icc or cholesky matrix factorization, you save half of storage and efficient data retrieval. > I just took a quick look at the MUMPS solver. It seems like they do have > support for distributed direct solving of symmetric matrices with ordering. > So, this kind of thing has been done before.. But I don't know about the > efficiency because I haven't use MUMPS before.. But I do have a symmetric > direct solver (using 1 processor) in my fem code that uses ordering and > right now it is a LOT faster that the symmetric solver in petsc. I'm > guessing that is because petsc does not support reordering. Matrix reordering can be done for symmetric matrix. We took out its support because the better and efficient way is using aij matrix + matrix ordering + symmetric preconditioner (icc or cholesky). I just came back from visiting MUMPS group in France (this is why I delayed this email). Although mumps allow user to input half of symmetric matrix distributively, it first collects matrix structure (row and column indices) into centralized host process, does analysis and reordering, then reorganizes user matrix values into its own data structure - lots of internal operations! Saving half of your matrix storage doesn't lead to overall saving. You may contact MUMPs developer on details of their symmetric solver. If you want use parallel direct solver, mumps is a good choice. Note, petsc is interfaced with mumps. > > > icc/cholesky preconditioners save half of storage than ilu/lu > > preconditioner and might be faster. If your have symmetric > > matrix, use aij format for your matrix, and apply > > icc/cholesky preconditioner. > > You can apply any symmetric ordering efficiently to the aij > > matrix to reduce the fillings in matrix factorization. I've repeated this view. > > Can you please clarify that? What do you mean by symmetric ordering? Let Pr and Pc be row and column orderings. We solve (Pr* A * Pc^{-1}) * Pc * x = Pr* rhs in general. If you have a symmetric matrix and want to use symmetric preconditioner/solver, you must maintain a symmetrix matrix after matrix ordering, i.e., applying permutation matrix P to PAP^{-1} P * x = P * rhs. Since P is a permutation matrix, it satisfies P^{-1} = P^T, thus, PAP^{-1} remains symmetric. >I would certainly prefer symmetric solvers that can do reordering. But >I'll >let you know what I come up with when I use aij with cholesky. Matrix reordering reduces fill-ins in icc/cholesky factorization and facilitates efficient data accessing - very important for sparse matrix computation. Using aij matrix enables user to have effient matrix ordering, while retain the space savings and fast data accessing during preconditioning and solve. We have investigated matrx ordering schemes (including well-established scheme) on symmtric matrices stored in half. Saving half of user's input doesn't lead to overal saving of spaces. Hong From yxliu at fudan.edu.cn Tue Oct 31 19:48:15 2006 From: yxliu at fudan.edu.cn (Yixun Liu) Date: Wed, 01 Nov 2006 09:48:15 +0800 Subject: about setting PETSC_DIR Message-ID: <000801c6fd57$cc1702c0$3364a8c0@dmrc6700512> Hi, I use Cygwin on windowsXP. I want to know how to set system enviroment varialble PETSC_DIR. PETSC_DIR = /cygdrive/d/myvc/petsc-2.3.2-p3 or PETSC_DIR = d:\myvc\petsc-2.3.2-p3? The former can be recongized by cygwin but not by windows. The latter otherwise. Best, Yixun -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Oct 31 20:13:25 2006 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 31 Oct 2006 20:13:25 -0600 (CST) Subject: about setting PETSC_DIR In-Reply-To: <000801c6fd57$cc1702c0$3364a8c0@dmrc6700512> References: <000801c6fd57$cc1702c0$3364a8c0@dmrc6700512> Message-ID: On Wed, 1 Nov 2006, Yixun Liu wrote: > Hi, > > I use Cygwin on windowsXP. I want to know how to set system enviroment varialble PETSC_DIR. PETSC_DIR = /cygdrive/d/myvc/petsc-2.3.2-p3 or PETSC_DIR = d:\myvc\petsc-2.3.2-p3? The former can be recongized by cygwin but not by windows. The latter otherwise. > Depending on where you use PETSC_DIR - it should be set in the correct notation. If you are constantly switching between cygwin & MSDev project files - then its painful to depend upon the env variable. I guess I should change the variable name in the projectfiles to PETSC_DIR_WIN - so both variables can be set/used. Since MS project files are not really portable - you might as well remove $PETSC_DIR from them - and use the correct windows version of the path in them.. Satish