From thomas.witkowski at tu-dresden.de Mon Oct 5 02:39:20 2009 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Mon, 05 Oct 2009 09:39:20 +0200 Subject: Mixed finite element discretization with petsc? Message-ID: <4AC9A2A8.5060208@tu-dresden.de> Hi, does any of you has experience with the discretization of mixed finite elements using petsc? From my point the main problem is that I cannot dived the overall matrix row by row to the processes. So what's the best way to use petsc for this kind of discretizations? Thomas From jed at 59A2.org Mon Oct 5 02:47:51 2009 From: jed at 59A2.org (Jed Brown) Date: Mon, 5 Oct 2009 09:47:51 +0200 Subject: Mixed finite element discretization with petsc? In-Reply-To: <4AC9A2A8.5060208@tu-dresden.de> References: <4AC9A2A8.5060208@tu-dresden.de> Message-ID: <383ade90910050047j7d0cedbeu841dabfbd0d77f78@mail.gmail.com> On Mon, Oct 5, 2009 at 09:39, Thomas Witkowski wrote: >?From my point the main problem is that I cannot dived > the overall matrix row by row to the processes. Why not? The usual thing is to partition your mesh and then resolve ownership of the interface nodes. You will insert element matrices where some of the nodes are not owned (i.e. set values for rows that are now owned) but PETSc handles this so it is transparent to you. Jed From thomas.witkowski at tu-dresden.de Mon Oct 5 04:09:44 2009 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Mon, 05 Oct 2009 11:09:44 +0200 Subject: Mixed finite element discretization with petsc? In-Reply-To: <383ade90910050047j7d0cedbeu841dabfbd0d77f78@mail.gmail.com> References: <4AC9A2A8.5060208@tu-dresden.de> <383ade90910050047j7d0cedbeu841dabfbd0d77f78@mail.gmail.com> Message-ID: <4AC9B7D8.9080400@tu-dresden.de> On Mon, Oct 5, 2009 at 09:39, Thomas Witkowski > wrote: > >> From my point the main problem is that I cannot dived >> the overall matrix row by row to the processes. >> > > Why not? > > The usual thing is to partition your mesh and then resolve ownership > of the interface nodes. You will insert element matrices where some > of the nodes are not owned (i.e. set values for rows that are now > owned) but PETSc handles this so it is transparent to you. > Okay, I thought a little bit to complicated :) So it's clear for me that there is no problem in defining each row of the overall system to one processor. But how to make continues row indices? In my sequential code I'm assembling the matrices block wise, so the overall matrix looks as follows: {A & B \\ B^T & 0} * {u & p) = {0 & 0} So when I've partitioned my mesh, and say I've 1000 nodes in the mesh, the first row and the row 1001 are owned by the same process, because they come from the discretization of the same node. So is it right to bring all these matrix rows together by using different row indices? Thomas > Jed > > > From jed at 59A2.org Mon Oct 5 04:49:39 2009 From: jed at 59A2.org (Jed Brown) Date: Mon, 5 Oct 2009 11:49:39 +0200 Subject: Mixed finite element discretization with petsc? In-Reply-To: <4AC9B7D8.9080400@tu-dresden.de> References: <4AC9A2A8.5060208@tu-dresden.de> <383ade90910050047j7d0cedbeu841dabfbd0d77f78@mail.gmail.com> <4AC9B7D8.9080400@tu-dresden.de> Message-ID: <383ade90910050249j1a1880fby9898dca10d6de35f@mail.gmail.com> On Mon, Oct 5, 2009 at 11:09, Thomas Witkowski wrote: > Okay, I thought a little bit to complicated :) So it's clear for me that > there is no problem in defining each row of the overall system to one > processor. But how to make continues row indices? > > In my sequential code I'm assembling the matrices block wise, so the overall > matrix looks as follows: > > {A & B \\ B^T & 0} * {u & p) = {0 & 0} This representation is purely symbolic. There is a permutation of the dofs that would look like this, but it's not the ordering that you actually want to use > So when I've partitioned my mesh, and say I've 1000 nodes in the mesh, the > first row and the row 1001 are owned by the same process, because they come > from the discretization of the same node. So is it right to bring all these > matrix rows together by using different row indices? 1. Partition the mesh and resolve ownership 2. Each process counts the number of owned dofs for velocity + pressure. 3. MPI_Scan so that every process knows it's starting offset. 4. Each process numbers owned dofs starting with this offset. 5. Scatter this global numbering so that every process knows the global index of its unowned interface dofs 6. Preallocate matrix Note that some PETSc calls, such as creating a vector, do the MPI_Scan logic for you. The procedure above will work fine with a direct solver, but most preconditioners do a terrible job if you just give them an assembled indefinite matrix. To do better, you may want to explore Schur complement or special domain decomposition preconditioners. The former can be done using PCFieldSplit, but usually you need some physical insight to precondition the Schur complement. These PCs sometimes prefer a different matrix representation, so if you have a good idea of what will work for your problem, let us know and we can give suggestions. Jed From Andreas.Grassl at student.uibk.ac.at Mon Oct 5 11:09:02 2009 From: Andreas.Grassl at student.uibk.ac.at (Andreas Grassl) Date: Mon, 05 Oct 2009 18:09:02 +0200 Subject: convergence monitoring Message-ID: <4ACA1A1E.3060401@student.uibk.ac.at> Hello, I want to monitor the convergence of the KSP-solver, i.e. plot the number of iterations vs. the error/residual norm. I discovered the options -ksp_monitor_draw_true_residual and -ksp_monitor_true_residual. Now the questions: What does the grafical output of -ksp_monitor_draw_true_residual represent? I see the iteration count at the x-axis and expected the norm of the residual at the y-axis, and some scaled value at the y-axis. Is this the logarithm of the residual/which residual? If I output to .ps, I get overlayed the steps and don't see anything useful at the end. Is there a way to extract only the last picture? Is there an easy (without hacking the PETSc sources) way to output a customized convergence monitoring line like -ksp_monitor_true_residual at every step? Cheers, ando -- /"\ Grassl Andreas \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik X against HTML email Technikerstr. 13 Zi 709 / \ +43 (0)512 507 6091 From bsmith at mcs.anl.gov Mon Oct 5 11:25:43 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 5 Oct 2009 11:25:43 -0500 Subject: convergence monitoring In-Reply-To: <4ACA1A1E.3060401@student.uibk.ac.at> References: <4ACA1A1E.3060401@student.uibk.ac.at> Message-ID: <813AEFB3-586B-4B05-B5D2-C24AD4EE00C6@mcs.anl.gov> On Oct 5, 2009, at 11:09 AM, Andreas Grassl wrote: > Hello, > > I want to monitor the convergence of the KSP-solver, i.e. plot the > number of > iterations vs. the error/residual norm. > > I discovered the options -ksp_monitor_draw_true_residual and > -ksp_monitor_true_residual. > > Now the questions: What does the grafical output of > -ksp_monitor_draw_true_residual represent? I see the iteration count > at the > x-axis and expected the norm of the residual at the y-axis, and some > scaled > value at the y-axis. Is this the logarithm of the residual/which > residual? It is the logarithm of || b - A x || because of the "true" in the option name. > > If I output to .ps, I get overlayed the steps and don't see anything > useful at > the end. Is there a way to extract only the last picture? The postscript viewer is pretty much a mess, ugly and limited; it is too much work for us to maintain. > > Is there an easy (without hacking the PETSc sources) way to output a > customized > convergence monitoring line like -ksp_monitor_true_residual at every > step? What do you want to do? I cannot understand your question. You can use KSPMonitorSet() to provide any function you want to display residuals anyway you want. Barry > > Cheers, > > ando > > -- > /"\ Grassl Andreas > \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik > X against HTML email Technikerstr. 13 Zi 709 > / \ +43 (0)512 507 6091 From Andreas.Grassl at student.uibk.ac.at Tue Oct 6 01:47:20 2009 From: Andreas.Grassl at student.uibk.ac.at (Andreas Grassl) Date: Tue, 06 Oct 2009 08:47:20 +0200 Subject: convergence monitoring In-Reply-To: <813AEFB3-586B-4B05-B5D2-C24AD4EE00C6@mcs.anl.gov> References: <4ACA1A1E.3060401@student.uibk.ac.at> <813AEFB3-586B-4B05-B5D2-C24AD4EE00C6@mcs.anl.gov> Message-ID: <4ACAE7F8.1090002@student.uibk.ac.at> Barry Smith schrieb: > On Oct 5, 2009, at 11:09 AM, Andreas Grassl wrote: >> Is there an easy (without hacking the PETSc sources) way to output a >> customized >> convergence monitoring line like -ksp_monitor_true_residual at every >> step? > > What do you want to do? I cannot understand your question. You can > use KSPMonitorSet() to > provide any function you want to display residuals anyway you want. This is the function I was searching. Surfing through the various routines I was overlooking the short description and concentrated here only on the option keys :) Cheers, ando -- /"\ Grassl Andreas \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik X against HTML email Technikerstr. 13 Zi 709 / \ +43 (0)512 507 6091 From yelkhamra at gmail.com Tue Oct 6 20:51:49 2009 From: yelkhamra at gmail.com (Yaakoub El Khamra) Date: Tue, 6 Oct 2009 20:51:49 -0500 Subject: Question about Coloring Message-ID: <47a831090910061851m484f21dh7a70fc6e3cff3a1b@mail.gmail.com> Greetings I have a system of equations that I am solving with SNES on a DA, and the coloring is almost exactly as the one I would get with DAGetColoring but different in a very small and identifiable number of locations. Is there a way to modify the coloring I get from DAGetColoring directly as opposed to running MatGetColoring? Or should I register my own coloring using MatColoringRegisterDynamic? Any ideas/suggestions/examples are greatly appreciated. Many thanks in advance. Regards Yaakoub El Khamra From yye00 at cct.lsu.edu Tue Oct 6 20:53:30 2009 From: yye00 at cct.lsu.edu (Yaakoub El Khamra) Date: Tue, 6 Oct 2009 20:53:30 -0500 Subject: Question about Coloring Message-ID: <47a831090910061853g4594f937hc11172fdf8156a38@mail.gmail.com> Greetings I have a system of equations that I am solving with SNES on a DA, and the coloring is almost exactly as the one I would get with DAGetColoring but different in a very small and identifiable number of locations. Is there a way to modify the coloring I get from DAGetColoring directly as opposed to running MatGetColoring? Or should I register my own coloring using MatColoringRegisterDynamic? Any ideas/suggestions/examples are greatly appreciated. Many thanks in advance. Regards Yaakoub El Khamra From bsmith at mcs.anl.gov Tue Oct 6 21:02:39 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 6 Oct 2009 21:02:39 -0500 Subject: Question about Coloring In-Reply-To: <47a831090910061851m484f21dh7a70fc6e3cff3a1b@mail.gmail.com> References: <47a831090910061851m484f21dh7a70fc6e3cff3a1b@mail.gmail.com> Message-ID: There are any number of valid colorings for same matrix nonzero pattern. The only reason to like one over the other is if it requires fewer colors (then it requires fewer function evaluations). There is no easy way to modify a coloring once it is created with DAGetColoring(). If you have code that efficiently colors the matrix then yes you can use MatColoringRegisterDynamic() to provide it. I take the coloring DAGetColoring() gives you is not a correct coloring for your problem? You could also copy the code from DAGetColoring() and modify it for your problem. Barry On Oct 6, 2009, at 8:51 PM, Yaakoub El Khamra wrote: > Greetings > I have a system of equations that I am solving with SNES on a DA, and > the coloring is almost exactly as the one I would get with > DAGetColoring but different in a very small and identifiable number of > locations. Is there a way to modify the coloring I get from > DAGetColoring directly as opposed to running MatGetColoring? Or should > I register my own coloring using MatColoringRegisterDynamic? Any > ideas/suggestions/examples are greatly appreciated. > > > Many thanks in advance. > Regards > Yaakoub El Khamra From yelkhamra at gmail.com Tue Oct 6 21:04:33 2009 From: yelkhamra at gmail.com (Yaakoub El Khamra) Date: Tue, 6 Oct 2009 21:04:33 -0500 Subject: Question about Coloring In-Reply-To: References: <47a831090910061851m484f21dh7a70fc6e3cff3a1b@mail.gmail.com> Message-ID: <47a831090910061904w1aca3753hd01970b12ef0fab0@mail.gmail.com> Greetings Great idea! Modify the DAGetColoring code and register it. Thank you very much! Regards Yaakoub El Khamra On Tue, Oct 6, 2009 at 9:02 PM, Barry Smith wrote: > > ? ?There are any number of valid colorings for same matrix nonzero pattern. > The only reason to like one over the other is if it requires fewer colors > (then it requires fewer function evaluations). > > ? ?There is no easy way to modify a coloring once it is created with > DAGetColoring(). > > ? ?If you have code that efficiently colors the matrix then yes you can use > MatColoringRegisterDynamic() to provide it. > > ? ?I take the coloring DAGetColoring() gives you is not a correct coloring > for your problem? > > ? ?You could also copy the code from DAGetColoring() and modify it for your > problem. > > ? Barry > > On Oct 6, 2009, at 8:51 PM, Yaakoub El Khamra wrote: > >> Greetings >> I have a system of equations that I am solving with SNES on a DA, and >> the coloring is almost exactly as the one I would get with >> DAGetColoring but different in a very small and identifiable number of >> locations. Is there a way to modify the coloring I get from >> DAGetColoring directly as opposed to running MatGetColoring? Or should >> I register my own coloring using MatColoringRegisterDynamic? Any >> ideas/suggestions/examples are greatly appreciated. >> >> >> Many thanks in advance. >> Regards >> Yaakoub El Khamra > > From mafunk at nmsu.edu Wed Oct 7 17:26:23 2009 From: mafunk at nmsu.edu (Matt Funk) Date: Wed, 7 Oct 2009 16:26:23 -0600 Subject: parallel solve question Message-ID: <200910071626.24382.mafunk@nmsu.edu> Hi, i have a problem for which i am not exaclty sure about what to do. I set up a simple 2D rectangular domain and decompose it into four equal boxes. I then build the petsc matrix based on this layout as well as the corresponsing RHS vector. I print out the matrix and RHS vector right before my KSPSolve call, and right after that call i print out the solution vector 'x'. I do this for 2 runs. 1) 1 processor 2) 4 processors. For both runs i do a difference (i.e. on the output files using diff) on all 3 quantities (the matrix, the RHS vector and the solution vector). The 'diff' command reports no difference between the files for the matrix and RHS vector. However, the soltution vector is different between the 2 runs. How different depends a little on what precond/solver combination i use and the tolerances. However, for example for BJacobi/GMRES with reltol=abstol=1e-12 the vector element with the maximum difference is on the order 1e-05. This is only after the first timestep. My problem has some nonlinearlity to it such that this will become a problem later on. The worst difference i have seen is if i use hypre's euclid. It was on the order of 1e-02. So my question is whether someone has an idea why this is happening (i suppose it is related to the parallel communication) and if there is way to fix it. thanks matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at mcs.anl.gov Wed Oct 7 18:29:52 2009 From: knepley at mcs.anl.gov (Matthew Knepley) Date: Wed, 7 Oct 2009 18:29:52 -0500 Subject: parallel solve question In-Reply-To: <200910071626.24382.mafunk@nmsu.edu> References: <200910071626.24382.mafunk@nmsu.edu> Message-ID: This sounds like it has to do with the condition of your system, not any parallel problem. Errors in the solution can only be reduced to about (condition number) * (residual norm). Matt On Wed, Oct 7, 2009 at 5:26 PM, Matt Funk wrote: > Hi, > > > i have a problem for which i am not exaclty sure about what to do. > I set up a simple 2D rectangular domain and decompose it into four equal > boxes. I then build the petsc matrix based on this layout as well as the > corresponsing RHS vector. > > > I print out the matrix and RHS vector right before my KSPSolve call, and > right after that call i print out the solution vector 'x'. > > > I do this for 2 runs. > 1) 1 processor > 2) 4 processors. > > > For both runs i do a difference (i.e. on the output files using diff) on > all 3 quantities (the matrix, the RHS vector and the solution vector). > > > The 'diff' command reports no difference between the files for the matrix > and RHS vector. > > > However, the soltution vector is different between the 2 runs. How > different depends a little on what precond/solver combination i use and the > tolerances. > > > However, for example for BJacobi/GMRES with reltol=abstol=1e-12 the vector > element with the maximum difference is on the order 1e-05. This is only > after the first timestep. My problem has some nonlinearlity to it such that > this will become a problem later on. > > > The worst difference i have seen is if i use hypre's euclid. It was on the > order of 1e-02. > > > > So my question is whether someone has an idea why this is happening (i > suppose it is related to the parallel communication) and if there is way to > fix it. > > > > thanks > matt > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From peyser.alex at gmail.com Wed Oct 7 19:15:35 2009 From: peyser.alex at gmail.com (Alex Peyser) Date: Wed, 7 Oct 2009 20:15:35 -0400 Subject: stdout and KSPMonitor Message-ID: <200910072015.35914.peyser.alex@gmail.com> I'm just sending this for anybody who ever runs into this problem: If you kill stderr or stdout somewhere in your code (particularly if you have your own mpi launcher that does the spawn), KSPMonitor functions will simply kill the process on output. Not surprising -- it doesn't have many recovery options, other than I guess propagating an error up. This is definitely the case for petsc-2.3.3 -- I doubt that such code would have been changed in more modern versions. It's entirely possible that it isn't even in the petsc code this occurs, but that some mpi implementations are handling the output to stdout when it's closed by killing the process. It just creates a bit of chasing, since such code would probably be quite remote from the petsc calls themselves, and at least mpich2 makes it impossible to localize in the running code. Regards, Alex Peyser -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: From tim.kroeger at cevis.uni-bremen.de Thu Oct 8 03:47:05 2009 From: tim.kroeger at cevis.uni-bremen.de (Tim Kroeger) Date: Thu, 8 Oct 2009 10:47:05 +0200 (CEST) Subject: --with-pic Message-ID: Dear all, On a 64-bit system, I need to compile with the option "-fPIC" to be able to link with my application. I configured PETSC with "--with-pic=fPIC", but still it seems to call the gfortran compiler with "-PIC" rather than "-fPIC" for some or all fortran source files. The compiler does not understand the "-PIC" option, hence prints an error message and exits. How can I tell PETSc to use "-fPIC" instead? Best Regards, Tim -- Dr. Tim Kroeger tim.kroeger at mevis.fraunhofer.de Phone +49-421-218-7710 tim.kroeger at cevis.uni-bremen.de Fax +49-421-218-4236 Fraunhofer MEVIS, Institute for Medical Image Computing Universitaetsallee 29, 28359 Bremen, Germany From hakan.jakobsson at math.umu.se Thu Oct 8 03:55:32 2009 From: hakan.jakobsson at math.umu.se (=?ISO-8859-1?Q?H=E5kan_Jakobsson?=) Date: Thu, 8 Oct 2009 10:55:32 +0200 Subject: problem with MatPtAP Message-ID: <687bb3e80910080155q16fa7224g4dbe7c0bf1bc3453@mail.gmail.com> Hi, I have run into some problems with using MatPtAP. I have two similar matrix/projection matrix pairs (K1,V1) and (K2,V2). Dimensions are K1 - 10227x10227, V1 - 10227x255 K2 - 10146x10146, V2 - 10146x336 For the first pair MatPtAP returns ok, but for the second one ierr=MatPtAP(K, V, MAT_INITIAL_MATRIX, 1.0, &VtKV); CHKERRQ(ierr); fails with [0]PETSC ERROR: MatPtAPNumeric_SeqAIJ_SeqAIJ() line 245 in src/mat/impls/aij/seq/matptap.c [0]PETSC ERROR: MatPtAPNumeric_SeqAIJ() line 36 in src/mat/impls/aij/seq/matptap.c [0]PETSC ERROR: MatPtAPNumeric() line 7279 in src/mat/interface/matrix.c [0]PETSC ERROR: MatPtAP_Basic() line 15 in src/mat/utils/ptap.c [0]PETSC ERROR: MatPtAP() line 7223 in src/mat/interface/matrix.c The same also applies when using MatPtAPNumeric. For Mat VtKV; ierr=MatPtAPSymbolic(K, V, 1.0, &VtKV); CHKERRQ(ierr); ierr=MatPtAPNumeric(K, V, VtKV); CHKERRQ(ierr); MatPtAPSymbolic returns ok, but MatPtAPNumeric fails with error message [0]PETSC ERROR: MatPtAPNumeric_SeqAIJ_SeqAIJ() line 245 in src/mat/impls/aij/seq/matptap.c [0]PETSC ERROR: MatPtAPNumeric_SeqAIJ() line 36 in src/mat/impls/aij/seq/matptap.c [0]PETSC ERROR: MatPtAPNumeric() line 7279 in src/mat/interface/matrix.c When splitting the operation into a MatMatMultTranspose and a MatMatMult I don't get any errors. Any ideas on what might be the going on here? PETSc version is 3.0.0-p8. /H?kan From knepley at gmail.com Thu Oct 8 04:39:30 2009 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 8 Oct 2009 04:39:30 -0500 Subject: --with-pic In-Reply-To: References: Message-ID: On Thu, Oct 8, 2009 at 3:47 AM, Tim Kroeger wrote: > Dear all, > > On a 64-bit system, I need to compile with the option "-fPIC" to be able to > link with my application. I configured PETSC with "--with-pic=fPIC", but > still it seems to call the gfortran compiler with "-PIC" rather than "-fPIC" > for some or all fortran source files. The compiler does not understand the > "-PIC" option, hence prints an error message and exits. How can I tell > PETSc to use "-fPIC" instead? > 1) --with-pic only turns this check on and off. It does not take a flag 2) You must send the configure.log or we have no way of knowing what is going on. Send it to petsc-maint Matt > Best Regards, > > Tim > > -- > Dr. Tim Kroeger > tim.kroeger at mevis.fraunhofer.de Phone +49-421-218-7710 > tim.kroeger at cevis.uni-bremen.de Fax +49-421-218-4236 > > Fraunhofer MEVIS, Institute for Medical Image Computing > Universitaetsallee 29, 28359 Bremen, Germany > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Thu Oct 8 07:20:41 2009 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Thu, 08 Oct 2009 14:20:41 +0200 Subject: Mixed finite element discretization with petsc? In-Reply-To: <383ade90910050249j1a1880fby9898dca10d6de35f@mail.gmail.com> References: <4AC9A2A8.5060208@tu-dresden.de> <383ade90910050047j7d0cedbeu841dabfbd0d77f78@mail.gmail.com> <4AC9B7D8.9080400@tu-dresden.de> <383ade90910050249j1a1880fby9898dca10d6de35f@mail.gmail.com> Message-ID: <4ACDD919.30408@tu-dresden.de> Jed Brown wrote: > On Mon, Oct 5, 2009 at 11:09, Thomas Witkowski > wrote: > > >> Okay, I thought a little bit to complicated :) So it's clear for me that >> there is no problem in defining each row of the overall system to one >> processor. But how to make continues row indices? >> >> In my sequential code I'm assembling the matrices block wise, so the overall >> matrix looks as follows: >> >> {A & B \\ B^T & 0} * {u & p) = {0 & 0} >> > > This representation is purely symbolic. There is a permutation of the > dofs that would look like this, but it's not the ordering that you > actually want to use > okay, but with a different ordering I'll lose symmetry of the system. > >> So when I've partitioned my mesh, and say I've 1000 nodes in the mesh, the >> first row and the row 1001 are owned by the same process, because they come >> from the discretization of the same node. So is it right to bring all these >> matrix rows together by using different row indices? >> > > 1. Partition the mesh and resolve ownership > 2. Each process counts the number of owned dofs for velocity + pressure. > 3. MPI_Scan so that every process knows it's starting offset. > 4. Each process numbers owned dofs starting with this offset. > 5. Scatter this global numbering so that every process knows the > global index of its unowned interface dofs > 6. Preallocate matrix > Most of the points are already done in my fem code (it's a general fem library, and I've solved complicated higher order pdes with up to 512 cores using petsc), but it's not generalized to mixed finite elements. > Note that some PETSc calls, such as creating a vector, do the MPI_Scan > logic for you. > > The procedure above will work fine with a direct solver, but most > preconditioners do a terrible job if you just give them an assembled > indefinite matrix. To do better, you may want to explore Schur > complement or special domain decomposition preconditioners. The > former can be done using PCFieldSplit, but usually you need some > physical insight to precondition the Schur complement. These PCs > sometimes prefer a different matrix representation, so if you have a > good idea of what will work for your problem, let us know and we can > give suggestions. > Yes, preconditioning is really a problem at the moment. I see the effect, that with increasing domain size the number of iterations is also growing. I think, it's only possible to solve the problem with an pde specific preconditioner eliminating the element size from the spectrum of the linear system. Thanks alot to make some things more clear to me! Thomas From jed at 59A2.org Thu Oct 8 07:43:16 2009 From: jed at 59A2.org (Jed Brown) Date: Thu, 8 Oct 2009 14:43:16 +0200 Subject: Mixed finite element discretization with petsc? In-Reply-To: <4ACDD919.30408@tu-dresden.de> References: <4AC9A2A8.5060208@tu-dresden.de> <383ade90910050047j7d0cedbeu841dabfbd0d77f78@mail.gmail.com> <4AC9B7D8.9080400@tu-dresden.de> <383ade90910050249j1a1880fby9898dca10d6de35f@mail.gmail.com> <4ACDD919.30408@tu-dresden.de> Message-ID: <383ade90910080543m730a2f83m95492d51f4be4bb2@mail.gmail.com> On Thu, Oct 8, 2009 at 14:20, Thomas Witkowski wrote: >> This representation is purely symbolic. ?There is a permutation of the >> dofs that would look like this, but it's not the ordering that you >> actually want to use >> > > okay, but with a different ordering I'll lose symmetry of the system. No, it is a symmetric permutation. For indefinite problems, I implement the preconditioning matrix using MatShell where I provide MatGetSubMatrix (which either returns the cached submatrix or assembles it on demand, the former is sometimes more efficient and more convenient for profiling). You should have a reasonable idea about how to precondition the subsystems before you go this route. Jed From hzhang at mcs.anl.gov Thu Oct 8 09:03:27 2009 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Thu, 8 Oct 2009 09:03:27 -0500 (CDT) Subject: problem with MatPtAP In-Reply-To: <687bb3e80910080155q16fa7224g4dbe7c0bf1bc3453@mail.gmail.com> References: <687bb3e80910080155q16fa7224g4dbe7c0bf1bc3453@mail.gmail.com> Message-ID: Please try followings: 1) only run the second pairs and 2) use a larger fill value, e.g., fill=5.0 or 10.0 MatPtAP(K2, V2, MAT_INITIAL_MATRIX, fill, &VtKV); let us know what happens. If it still crashes, send us your matrices K2 and V2 in petsc binary format. I'll test it. Hong On Thu, 8 Oct 2009, H??kan Jakobsson wrote: > Hi, > > I have run into some problems with using MatPtAP. I have two similar > matrix/projection matrix pairs (K1,V1) and (K2,V2). Dimensions are > > K1 - 10227x10227, V1 - 10227x255 > K2 - 10146x10146, V2 - 10146x336 > > For the first pair MatPtAP returns ok, but for the second one > > ierr=MatPtAP(K, V, MAT_INITIAL_MATRIX, 1.0, &VtKV); CHKERRQ(ierr); > > fails with > > [0]PETSC ERROR: MatPtAPNumeric_SeqAIJ_SeqAIJ() line 245 in > src/mat/impls/aij/seq/matptap.c > [0]PETSC ERROR: MatPtAPNumeric_SeqAIJ() line 36 in > src/mat/impls/aij/seq/matptap.c > [0]PETSC ERROR: MatPtAPNumeric() line 7279 in src/mat/interface/matrix.c > [0]PETSC ERROR: MatPtAP_Basic() line 15 in src/mat/utils/ptap.c > [0]PETSC ERROR: MatPtAP() line 7223 in src/mat/interface/matrix.c > > The same also applies when using MatPtAPNumeric. For > > Mat VtKV; > ierr=MatPtAPSymbolic(K, V, 1.0, &VtKV); CHKERRQ(ierr); > ierr=MatPtAPNumeric(K, V, VtKV); CHKERRQ(ierr); > > MatPtAPSymbolic returns ok, but MatPtAPNumeric fails with error message > > [0]PETSC ERROR: MatPtAPNumeric_SeqAIJ_SeqAIJ() line 245 in > src/mat/impls/aij/seq/matptap.c > [0]PETSC ERROR: MatPtAPNumeric_SeqAIJ() line 36 in > src/mat/impls/aij/seq/matptap.c > [0]PETSC ERROR: MatPtAPNumeric() line 7279 in src/mat/interface/matrix.c > > When splitting the operation into a MatMatMultTranspose and a > MatMatMult I don't get any errors. > > Any ideas on what might be the going on here? PETSc version is 3.0.0-p8. > > /H?kan > From mafunk at nmsu.edu Thu Oct 8 10:14:05 2009 From: mafunk at nmsu.edu (Matt Funk) Date: Thu, 8 Oct 2009 09:14:05 -0600 Subject: parallel solve question In-Reply-To: References: <200910071626.24382.mafunk@nmsu.edu> Message-ID: <200910080914.05908.mafunk@nmsu.edu> Hi Matt, i don't understand how it can be the condition of the system. After all, the matrix as well as the RHS vector is EXACTLY the same between the 2 runs. This is what puzzles me so much. The only difference is whether i solve it using 1 core vs 4 cores. Of course i could be missing something. matt On Wednesday 07 October 2009, Matthew Knepley wrote: > This sounds like it has to do with the condition of your system, not any > parallel problem. Errors > in the solution can only be reduced to about (condition number) * (residual > norm). > > Matt > > On Wed, Oct 7, 2009 at 5:26 PM, Matt Funk wrote: > > Hi, > > > > > > i have a problem for which i am not exaclty sure about what to do. > > I set up a simple 2D rectangular domain and decompose it into four equal > > boxes. I then build the petsc matrix based on this layout as well as the > > corresponsing RHS vector. > > > > > > I print out the matrix and RHS vector right before my KSPSolve call, and > > right after that call i print out the solution vector 'x'. > > > > > > I do this for 2 runs. > > 1) 1 processor > > 2) 4 processors. > > > > > > For both runs i do a difference (i.e. on the output files using diff) on > > all 3 quantities (the matrix, the RHS vector and the solution vector). > > > > > > The 'diff' command reports no difference between the files for the matrix > > and RHS vector. > > > > > > However, the soltution vector is different between the 2 runs. How > > different depends a little on what precond/solver combination i use and > > the tolerances. > > > > > > However, for example for BJacobi/GMRES with reltol=abstol=1e-12 the > > vector element with the maximum difference is on the order 1e-05. This is > > only after the first timestep. My problem has some nonlinearlity to it > > such that this will become a problem later on. > > > > > > The worst difference i have seen is if i use hypre's euclid. It was on > > the order of 1e-02. > > > > > > > > So my question is whether someone has an idea why this is happening (i > > suppose it is related to the parallel communication) and if there is way > > to fix it. > > > > > > > > thanks > > matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Oct 8 10:18:30 2009 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 8 Oct 2009 10:18:30 -0500 Subject: parallel solve question In-Reply-To: <200910080914.05908.mafunk@nmsu.edu> References: <200910071626.24382.mafunk@nmsu.edu> <200910080914.05908.mafunk@nmsu.edu> Message-ID: On Thu, Oct 8, 2009 at 10:14 AM, Matt Funk wrote: > Hi Matt, > > > i don't understand how it can be the condition of the system. > After all, the matrix as well as the RHS vector is EXACTLY the same between > the 2 runs. This is what puzzles me so much. > The order of operations is different in the serial and parallel cases. With a very ill-conditioned matrix, which it sounds like you have, the reordering produces noticeably different residuals, even though they both satisfy the error bound. Matt > The only difference is whether i solve it using 1 core vs 4 cores. > > > Of course i could be missing something. > > > > matt > > > > > On Wednesday 07 October 2009, Matthew Knepley wrote: > > This sounds like it has to do with the condition of your system, not any > > parallel problem. Errors > > in the solution can only be reduced to about (condition number) * > (residual > > norm). > > > > Matt > > > > On Wed, Oct 7, 2009 at 5:26 PM, Matt Funk wrote: > > > Hi, > > > > > > > > > i have a problem for which i am not exaclty sure about what to do. > > > I set up a simple 2D rectangular domain and decompose it into four > equal > > > boxes. I then build the petsc matrix based on this layout as well as > the > > > corresponsing RHS vector. > > > > > > > > > I print out the matrix and RHS vector right before my KSPSolve call, > and > > > right after that call i print out the solution vector 'x'. > > > > > > > > > I do this for 2 runs. > > > 1) 1 processor > > > 2) 4 processors. > > > > > > > > > For both runs i do a difference (i.e. on the output files using diff) > on > > > all 3 quantities (the matrix, the RHS vector and the solution vector). > > > > > > > > > The 'diff' command reports no difference between the files for the > matrix > > > and RHS vector. > > > > > > > > > However, the soltution vector is different between the 2 runs. How > > > different depends a little on what precond/solver combination i use and > > > the tolerances. > > > > > > > > > However, for example for BJacobi/GMRES with reltol=abstol=1e-12 the > > > vector element with the maximum difference is on the order 1e-05. This > is > > > only after the first timestep. My problem has some nonlinearlity to it > > > such that this will become a problem later on. > > > > > > > > > The worst difference i have seen is if i use hypre's euclid. It was on > > > the order of 1e-02. > > > > > > > > > > > > So my question is whether someone has an idea why this is happening (i > > > suppose it is related to the parallel communication) and if there is > way > > > to fix it. > > > > > > > > > > > > thanks > > > matt > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hakan.jakobsson at math.umu.se Thu Oct 8 10:33:29 2009 From: hakan.jakobsson at math.umu.se (=?ISO-8859-1?Q?H=E5kan_Jakobsson?=) Date: Thu, 8 Oct 2009 17:33:29 +0200 Subject: problem with MatPtAP In-Reply-To: References: <687bb3e80910080155q16fa7224g4dbe7c0bf1bc3453@mail.gmail.com> Message-ID: <687bb3e80910080833j532ccbceg50c7df1a33c19c97@mail.gmail.com> Hi, I increased the fill value but the problem remains. I'm sending the matrices to petsc-maint at mcs.anl.gov. Thanks alot /H?kan On Thu, Oct 8, 2009 at 4:03 PM, Hong Zhang wrote: > > Please try followings: > > 1) only run the second pairs > ? and > 2) use a larger fill value, e.g., fill=5.0 or 10.0 > ? MatPtAP(K2, V2, MAT_INITIAL_MATRIX, fill, &VtKV); let us know what > happens. > > If it still crashes, send us your matrices > K2 and V2 in petsc binary format. > I'll test it. > > Hong > > On Thu, 8 Oct 2009, H??kan Jakobsson wrote: > >> Hi, >> >> I have run into some problems with using MatPtAP. I have two similar >> matrix/projection matrix pairs (K1,V1) and (K2,V2). Dimensions are >> >> K1 - 10227x10227, V1 - 10227x255 >> K2 - 10146x10146, V2 - 10146x336 >> >> For the first pair MatPtAP returns ok, but for the second one >> >> ierr=MatPtAP(K, V, MAT_INITIAL_MATRIX, 1.0, &VtKV); CHKERRQ(ierr); >> >> fails with >> >> [0]PETSC ERROR: MatPtAPNumeric_SeqAIJ_SeqAIJ() line 245 in >> src/mat/impls/aij/seq/matptap.c >> [0]PETSC ERROR: MatPtAPNumeric_SeqAIJ() line 36 in >> src/mat/impls/aij/seq/matptap.c >> [0]PETSC ERROR: MatPtAPNumeric() line 7279 in src/mat/interface/matrix.c >> [0]PETSC ERROR: MatPtAP_Basic() line 15 in src/mat/utils/ptap.c >> [0]PETSC ERROR: MatPtAP() line 7223 in src/mat/interface/matrix.c >> >> The same also applies when using MatPtAPNumeric. For >> >> Mat VtKV; >> ierr=MatPtAPSymbolic(K, V, 1.0, &VtKV); CHKERRQ(ierr); >> ierr=MatPtAPNumeric(K, V, VtKV); CHKERRQ(ierr); >> >> MatPtAPSymbolic returns ok, but MatPtAPNumeric fails with error message >> >> [0]PETSC ERROR: MatPtAPNumeric_SeqAIJ_SeqAIJ() line 245 in >> src/mat/impls/aij/seq/matptap.c >> [0]PETSC ERROR: MatPtAPNumeric_SeqAIJ() line 36 in >> src/mat/impls/aij/seq/matptap.c >> [0]PETSC ERROR: MatPtAPNumeric() line 7279 in src/mat/interface/matrix.c >> >> When splitting the operation into a MatMatMultTranspose and a >> MatMatMult I don't get any errors. >> >> Any ideas on what might be the going on here? PETSc version is 3.0.0-p8. >> >> /H?kan > From mafunk at nmsu.edu Thu Oct 8 15:21:44 2009 From: mafunk at nmsu.edu (Matt Funk) Date: Thu, 8 Oct 2009 14:21:44 -0600 Subject: Question about usage of DA object Message-ID: <200910081421.45276.mafunk@nmsu.edu> Hi, i was wondering if i can get some advice on how to use the DA objec tin PETSC. Is it possible to create a DA object by inserting a set of boxes into it? Or can the DA object only be used based on a single rectangular box? say my domain has 2 boxes: 1) (0,0) - (64,64) 2) (64,64) - (128,128) Can i "register" those 2 boxes with the object? If that is possible, then is it valid if certain cells/nodes of these boxes overlap? say they both contain cell/node (64,64)? If that is possible, and i run in parallel, will the solution on cell/node (64,64) be returned to both processes (the first owning box1 and the second box2). Right now i am building my matrix 'by hand' and i believe the mentioned "overlap" is causing an ill-conditioned matrix (as addressed earlier in this list), even through the values on the overlapping nodes are the same. thanks matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Oct 8 18:04:40 2009 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 8 Oct 2009 18:04:40 -0500 Subject: Question about usage of DA object In-Reply-To: <200910081421.45276.mafunk@nmsu.edu> References: <200910081421.45276.mafunk@nmsu.edu> Message-ID: On Thu, Oct 8, 2009 at 3:21 PM, Matt Funk wrote: > Hi, > > > i was wondering if i can get some advice on how to use the DA objec tin > PETSC. > > > Is it possible to create a DA object by inserting a set of boxes into it? > Or can the DA object only be used based on a single rectangular box? > A DA is a single rectangular prism. You could choose not to assign equations to some vertices, or to put the identity there. Matt > say my domain has 2 boxes: > 1) (0,0) - (64,64) > 2) (64,64) - (128,128) > > > Can i "register" those 2 boxes with the object? > > > If that is possible, then is it valid if certain cells/nodes of these boxes > overlap? say they both contain cell/node (64,64)? > > > > If that is possible, and i run in parallel, will the solution on cell/node > (64,64) be returned to both processes (the first owning box1 and the second > box2). > > > > Right now i am building my matrix 'by hand' and i believe the mentioned > "overlap" is causing an ill-conditioned matrix (as addressed earlier in this > list), even through the values on the overlapping nodes are the same. > > > > thanks > matt > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From brockp at umich.edu Thu Oct 8 20:10:28 2009 From: brockp at umich.edu (Brock Palen) Date: Thu, 8 Oct 2009 21:10:28 -0400 Subject: petsc on HPC Podcast Message-ID: <5F4B6A99-367D-4629-9FD7-3C951035B823@umich.edu> I (Brock Palen) host an HPC podcast (www.rce-cast.com) with Jeff Squyres of Open-MPI fame. We would like to invite Pet-sc to be on the show. The interview is fun, lasts about an hour and is done online so requires very little commitment. Please contact me off list. We normally have one or two devs from a project. Brock Palen www.umich.edu/~brockp Center for Advanced Computing brockp at umich.edu (734)936-1985 From jarunan at ascomp.ch Fri Oct 9 06:05:25 2009 From: jarunan at ascomp.ch (jarunan at ascomp.ch) Date: Fri, 09 Oct 2009 13:05:25 +0200 Subject: About makefile Message-ID: <20091009130525.06a38igak0ss0cks@webmail.ascomp.ch> Hello After the compilation, is it compulsory to delete .o file? Actually, once I found that .o file do not update unless I delete it before recompiling. I might misunderstand. if not, I would like to know some reason behind. Thank you Jarunan -- Jarunan Panyasantisuk Development Engineer ASCOMP GmbH, Technoparkstr. 1 CH-8005 Zurich, Switzerland Phone : +41 44 445 4072 Fax : +41 44 445 4075 E-mail: jarunan at ascomp.ch www.ascomp.ch From brockp at umich.edu Fri Oct 9 07:33:31 2009 From: brockp at umich.edu (Brock Palen) Date: Fri, 9 Oct 2009 08:33:31 -0400 Subject: About makefile In-Reply-To: <20091009130525.06a38igak0ss0cks@webmail.ascomp.ch> References: <20091009130525.06a38igak0ss0cks@webmail.ascomp.ch> Message-ID: <38DCA36D-20B8-468E-8AE3-F67ABF59DF9E@umich.edu> On Oct 9, 2009, at 7:05 AM, jarunan at ascomp.ch wrote: > Hello > > After the compilation, is it compulsory to delete .o file? > Actually, once I found that .o file do not update unless I delete it > before recompiling. I might misunderstand. if not, I would like to > know some reason behind. Make only recompiles files where the source file is newer than the resulting file (the .o file). Thus if you have not modified the source file for a .o file, it will not be recompiled and will be relinked as is. If you wish to force everything to recompile (maybe new compiler options etc) then you need to delete all your old .o .a .so and .mod files. So they are recreated with the new options. This is a feature of make, that is really helpful, it keeps you from having to wait to recompile code that has already been compiled once, if it will not change. Brock Palen www.umich.edu/~brockp Center for Advanced Computing brockp at umich.edu (734)936-1985 > > > Thank you > Jarunan > > > -- > Jarunan Panyasantisuk > Development Engineer > ASCOMP GmbH, Technoparkstr. 1 > CH-8005 Zurich, Switzerland > Phone : +41 44 445 4072 > Fax : +41 44 445 4075 > E-mail: jarunan at ascomp.ch > www.ascomp.ch > > > From balay at mcs.anl.gov Fri Oct 9 08:01:19 2009 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 9 Oct 2009 08:01:19 -0500 (CDT) Subject: About makefile In-Reply-To: <38DCA36D-20B8-468E-8AE3-F67ABF59DF9E@umich.edu> References: <20091009130525.06a38igak0ss0cks@webmail.ascomp.ch> <38DCA36D-20B8-468E-8AE3-F67ABF59DF9E@umich.edu> Message-ID: The reason we delete the .o files and force a recompile each time for examples is: not all dependencies are captured in our makefiles. For eg: when switching PETSC_ARCH, or [as developers] - if PETSc sources are changed for the same PETSC_ARCH. [examples don't have dependencies that would force a recompile] So its easier for us just to do recompile of examples each time - than keeping track of the above changes [which happen constantly for us developers] As users the above issue might not matter [as most use a single PETSC_ARCH, and the PETSC install hardly changes]. So - you can save this time by not deleting .o files - and doing 'make clean' - when you really need to reset. Satish On Fri, 9 Oct 2009, Brock Palen wrote: > > > > On Oct 9, 2009, at 7:05 AM, jarunan at ascomp.ch wrote: > > > Hello > > > > After the compilation, is it compulsory to delete .o file? > > Actually, once I found that .o file do not update unless I delete it before > > recompiling. I might misunderstand. if not, I would like to know some reason > > behind. > > Make only recompiles files where the source file is newer than the resulting > file (the .o file). Thus if you have not modified the source file for a .o > file, it will not be recompiled and will be relinked as is. If you wish to > force everything to recompile (maybe new compiler options etc) then you need > to delete all your old .o .a .so and .mod files. So they are recreated with > the new options. > > This is a feature of make, that is really helpful, it keeps you from having to > wait to recompile code that has already been compiled once, if it will not > change. > > Brock Palen > www.umich.edu/~brockp > Center for Advanced Computing > brockp at umich.edu > (734)936-1985 > > > > > > > Thank you > > Jarunan > > > > > > -- > > Jarunan Panyasantisuk > > Development Engineer > > ASCOMP GmbH, Technoparkstr. 1 > > CH-8005 Zurich, Switzerland > > Phone : +41 44 445 4072 > > Fax : +41 44 445 4075 > > E-mail: jarunan at ascomp.ch > > www.ascomp.ch > > > > > > > From jarunan at ascomp.ch Fri Oct 9 10:14:32 2009 From: jarunan at ascomp.ch (jarunan at ascomp.ch) Date: Fri, 09 Oct 2009 17:14:32 +0200 Subject: About makefile In-Reply-To: References: <20091009130525.06a38igak0ss0cks@webmail.ascomp.ch> <38DCA36D-20B8-468E-8AE3-F67ABF59DF9E@umich.edu> Message-ID: <20091009171432.yb3ciuguo0c08ok0@webmail.ascomp.ch> Thank you very much, Satish. Thank you very much, Brock Palen. I am clear about it now. Jarunan Quoting Satish Balay : > The reason we delete the .o files and force a recompile each time for > examples is: not all dependencies are captured in our makefiles. For > eg: when switching PETSC_ARCH, or [as developers] - if PETSc sources > are changed for the same PETSC_ARCH. [examples don't have dependencies > that would force a recompile] > > So its easier for us just to do recompile of examples each time - than > keeping track of the above changes [which happen constantly for us > developers] > > As users the above issue might not matter [as most use a single > PETSC_ARCH, and the PETSC install hardly changes]. So - you can save > this time by not deleting .o files - and doing 'make clean' - when > you really need to reset. > > Satish > > > On Fri, 9 Oct 2009, Brock Palen wrote: > >> >> >> >> On Oct 9, 2009, at 7:05 AM, jarunan at ascomp.ch wrote: >> >> > Hello >> > >> > After the compilation, is it compulsory to delete .o file? >> > Actually, once I found that .o file do not update unless I delete >> it before >> > recompiling. I might misunderstand. if not, I would like to know >> some reason >> > behind. >> >> Make only recompiles files where the source file is newer than the resulting >> file (the .o file). Thus if you have not modified the source file for a .o >> file, it will not be recompiled and will be relinked as is. If you wish to >> force everything to recompile (maybe new compiler options etc) >> then you need >> to delete all your old .o .a .so and .mod files. So they are recreated with >> the new options. >> >> This is a feature of make, that is really helpful, it keeps you >> from having to >> wait to recompile code that has already been compiled once, if it will not >> change. >> >> Brock Palen >> www.umich.edu/~brockp >> Center for Advanced Computing >> brockp at umich.edu >> (734)936-1985 >> >> > >> > >> > Thank you >> > Jarunan >> > >> > >> > -- >> > Jarunan Panyasantisuk >> > Development Engineer >> > ASCOMP GmbH, Technoparkstr. 1 >> > CH-8005 Zurich, Switzerland >> > Phone : +41 44 445 4072 >> > Fax : +41 44 445 4075 >> > E-mail: jarunan at ascomp.ch >> > www.ascomp.ch >> > >> > >> > >> > > -- Jarunan Panyasantisuk Development Engineer ASCOMP GmbH, Technoparkstr. 1 CH-8005 Zurich, Switzerland Phone : +41 44 445 4072 Fax : +41 44 445 4075 E-mail: jarunan at ascomp.ch www.ascomp.ch From u.tabak at tudelft.nl Fri Oct 9 14:19:10 2009 From: u.tabak at tudelft.nl (Umut Tabak) Date: Fri, 09 Oct 2009 21:19:10 +0200 Subject: Binary Matrix read problem Message-ID: <4ACF8CAE.6040508@tudelft.nl> Dear all, I upgraded to petsc-3.0.0-p8 a couple of minutes ago, but I had a problem with reading binary matrices written from matlab. I get [0]PETSC ERROR: Read from file failed! [0]PETSC ERROR: Read past end of file! I have a 64 bit system, did I miss sth in the configuration, I checked the installation page again, but could not see the reason of the problem. My codes worked fine before with patch level 4. In ddd I get the error from MatLoad below: ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD, "K.bin", FILE_MODE_READ, &view ); CHKERRQ(ierr); ierr = MatLoad(view, MATSEQAIJ, &K ); CHKERRQ(ierr); (as expected). Thanks for the pointers in advance. Umut From u.tabak at tudelft.nl Fri Oct 9 17:24:40 2009 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sat, 10 Oct 2009 00:24:40 +0200 Subject: Matlab matrix writer Message-ID: <4ACFB828.8000508@tudelft.nl> Dear developers/users, As a follow up for my previous question, if I use the matlab functions that come with the distribution, namely, PetscBinaryWrite m file, and try to read this matrix, I end up with [0]PETSC ERROR: Unexpected data in file! [0]PETSC ERROR: not matrix object in file! I am also checking further what is going wrong in the m file. Someone perhaps could answer faster. BR, Umut From bsmith at mcs.anl.gov Fri Oct 9 18:10:22 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 9 Oct 2009 18:10:22 -0500 Subject: Matlab matrix writer In-Reply-To: <4ACFB828.8000508@tudelft.nl> References: <4ACFB828.8000508@tudelft.nl> Message-ID: <72F8E25E-5AC0-4F07-BCEE-65EACA1A54CF@mcs.anl.gov> The most likely cause is you are passing a non-sparse Matlab matrix to the writer; it is then saving it as a dense array (vector) so PETSc cannot read it. Run the array through the Matlab sparse() function and make sure it is sparse before you pass it to the save problem. Also make sure you have the latest patched PETSc. If this does not resolve the problem then email to petsc-maint at mcs.anl.gov the Matlab code that produces the "bad" binary file. Barry On Oct 9, 2009, at 5:24 PM, Umut Tabak wrote: > Dear developers/users, > > As a follow up for my previous question, if I use the matlab > functions that come with the distribution, namely, PetscBinaryWrite > m file, and try to read this matrix, I end up with > > [0]PETSC ERROR: Unexpected data in file! > [0]PETSC ERROR: not matrix object in file! > > I am also checking further what is going wrong in the m file. > Someone perhaps could answer faster. > > BR, > Umut > From bsmith at mcs.anl.gov Fri Oct 9 18:14:32 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 9 Oct 2009 18:14:32 -0500 Subject: Binary Matrix read problem In-Reply-To: <4ACF8CAE.6040508@tudelft.nl> References: <4ACF8CAE.6040508@tudelft.nl> Message-ID: <09BA3489-89E5-41B9-A852-E6670AD98A16@mcs.anl.gov> If you are using ./configure with --with-64-bit-indices this could happen since Matlab has its ints as 32 bit. To load the matrix from Matlab you need to not using --with-64-bit-indices with configure. This could also happen if you built PETSc with complex numbers. It can only read binary files from Matlab with real numbers. Barry On Oct 9, 2009, at 2:19 PM, Umut Tabak wrote: > Dear all, > > I upgraded to petsc-3.0.0-p8 a couple of minutes ago, but I had a > problem with reading binary matrices written from matlab. I get > > [0]PETSC ERROR: Read from file failed! > [0]PETSC ERROR: Read past end of file! > > I have a 64 bit system, did I miss sth in the configuration, I > checked the installation page again, but could not see the reason of > the problem. My codes worked fine before with patch level 4. In ddd > I get the error from MatLoad below: > > ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD, "K.bin", > FILE_MODE_READ, &view ); CHKERRQ(ierr); > ierr = MatLoad(view, MATSEQAIJ, &K ); CHKERRQ(ierr); > > (as expected). > > Thanks for the pointers in advance. > Umut From yfeng1 at tigers.lsu.edu Sun Oct 11 21:38:44 2009 From: yfeng1 at tigers.lsu.edu (Yin Feng) Date: Sun, 11 Oct 2009 21:38:44 -0500 Subject: a problem with error of "An error occurred in MPI_Comm_rank after MPI was finalized" Message-ID: <1e8c69dc0910111938s6358a174o5c03ad98c0e24b9d@mail.gmail.com> I put PETSc solver in one function and use another function to call that.This problem only appears when I put the function with PETSc solver in a loop, it works well at first step, and reports error "An error occurred in MPI_Comm_rank after MPI was finalized" at second time. The program is designed to support only one processor like: Vec x,b; Mat A; KSP ksp; PC pc; PetscInt i,j,col[N]; PetscScalar value[N]; PetscScalar val; PetscInitialize(PETSC_NULL,PETSC_NULL,PETSC_NULL,PETSC_NULL); VecCreate(PETSC_COMM_WORLD,&x); VecSetSizes(x,PETSC_DECIDE,N); VecSetFromOptions(x); VecDuplicate(x,&b); MatCreate(PETSC_COMM_WORLD,&A); MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N); MatSetFromOptions(A); ........ ........ KSPCreate(PETSC_COMM_WORLD,&ksp); KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); KSPGetPC(ksp,&pc); ................ ............... KSPSetTolerances(ksp,tol,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); KSPSetFromOptions(ksp); KSPSolve(ksp,b,x); VecDestroy(x); VecDestroy(b); MatDestroy(A); KSPDestroy(ksp); PetscFinalize(); Any one has ideal about this? The detailed error description is: An error occurred in MPI_Comm_rank *** after MPI was finalized *** MPI_ERRORS_ARE_FATAL (goodbye) Abort before MPI_INIT completed successfully; not able to guarantee that all other processes were killed! Thank you so much in advance! -------------- next part -------------- An HTML attachment was scrubbed... URL: From yfeng1 at tigers.lsu.edu Sun Oct 11 21:59:44 2009 From: yfeng1 at tigers.lsu.edu (Yin Feng) Date: Sun, 11 Oct 2009 21:59:44 -0500 Subject: a problem with error of "An error occurred in MPI_Comm_rank after MPI was finalized" In-Reply-To: <1e8c69dc0910111938s6358a174o5c03ad98c0e24b9d@mail.gmail.com> References: <1e8c69dc0910111938s6358a174o5c03ad98c0e24b9d@mail.gmail.com> Message-ID: <1e8c69dc0910111959n3eab3efeg2c62e1b04bb7e5b5@mail.gmail.com> The completed code is: void PETSc(double mat[],double vec[],double sol[],long N,double tol) { Vec x,b; Mat A; KSP ksp; PC pc; PetscInt i,j,col[N]; PetscScalar value[N]; PetscScalar val; PetscInitialize(PETSC_NULL,PETSC_NULL,PETSC_NULL,PETSC_NULL); VecCreate(PETSC_COMM_WORLD,&x); VecSetSizes(x,PETSC_DECIDE,N); VecSetFromOptions(x); VecDuplicate(x,&b); MatCreate(PETSC_COMM_WORLD,&A); MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N); MatSetFromOptions(A); for (i=0; i wrote: > I put PETSc solver in one function and use another function to call that.This > problem only appears when I put the function with PETSc solver in a loop, > it works well at first step, and reports error "An error occurred in > MPI_Comm_rank after MPI was finalized" > at second time. The program is designed to support only one processor like: > > Vec x,b; > > Mat A; > > KSP ksp; > > PC pc; > > PetscInt i,j,col[N]; > > PetscScalar value[N]; > > PetscScalar val; > > > PetscInitialize(PETSC_NULL,PETSC_NULL,PETSC_NULL,PETSC_NULL); > > VecCreate(PETSC_COMM_WORLD,&x); > > VecSetSizes(x,PETSC_DECIDE,N); > > VecSetFromOptions(x); > > VecDuplicate(x,&b); > > MatCreate(PETSC_COMM_WORLD,&A); > > MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N); > > MatSetFromOptions(A); > > ........ > > ........ > > KSPCreate(PETSC_COMM_WORLD,&ksp); > > KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); > > KSPGetPC(ksp,&pc); > > ................ > ............... > > > KSPSetTolerances(ksp,tol,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); > > KSPSetFromOptions(ksp); > > KSPSolve(ksp,b,x); > > > VecDestroy(x); > > VecDestroy(b); > > MatDestroy(A); > > KSPDestroy(ksp); > > PetscFinalize(); > > > Any one has ideal about this? > > The detailed error description is: > > An error occurred in MPI_Comm_rank > > *** after MPI was finalized > > *** MPI_ERRORS_ARE_FATAL (goodbye) > > Abort before MPI_INIT completed successfully; not able to guarantee that > all other processes were killed! > > Thank you so much in advance! > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yelkhamra at gmail.com Sun Oct 11 22:04:54 2009 From: yelkhamra at gmail.com (Yaakoub El Khamra) Date: Sun, 11 Oct 2009 22:04:54 -0500 Subject: a problem with error of "An error occurred in MPI_Comm_rank after MPI was finalized" In-Reply-To: <1e8c69dc0910111959n3eab3efeg2c62e1b04bb7e5b5@mail.gmail.com> References: <1e8c69dc0910111938s6358a174o5c03ad98c0e24b9d@mail.gmail.com> <1e8c69dc0910111959n3eab3efeg2c62e1b04bb7e5b5@mail.gmail.com> Message-ID: <47a831090910112004w150cd6a2s843f22465a6ea2cb@mail.gmail.com> This is for Mayank's project isn't it Regards Yaakoub El Khamra On Sun, Oct 11, 2009 at 9:59 PM, Yin Feng wrote: > The completed code is: > > void PETSc(double mat[],double vec[],double sol[],long N,double tol) { > > Vec? ? ? ? ? ? x,b; > > Mat? ? ? ? ? ? A; > > KSP? ? ? ? ? ? ksp; > > PC ? ? ? ? ? ? pc; > > PetscInt ? ? ? i,j,col[N]; > > PetscScalar? ? value[N]; > > PetscScalar? ? val; > > PetscInitialize(PETSC_NULL,PETSC_NULL,PETSC_NULL,PETSC_NULL); > > VecCreate(PETSC_COMM_WORLD,&x); > > VecSetSizes(x,PETSC_DECIDE,N); > > VecSetFromOptions(x); > > VecDuplicate(x,&b); > > MatCreate(PETSC_COMM_WORLD,&A); > > MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N); > > MatSetFromOptions(A); > > for (i=0; i > for (j=0;j > value[j]=mat[i*N+j]; > > col[j]=j; > > } > > MatSetValues(A,1,&i,N,col,value,INSERT_VALUES); > > } > > MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); > > MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); > > > > for (i=0; i > val=vec[i]; > > VecSetValues(b,1,&i,&val,INSERT_VALUES); > > val=sol[i]; > > VecSetValues(x,1,&i,&val,INSERT_VALUES); > > } > > VecAssemblyBegin(b); > > VecAssemblyEnd(b); > > KSPCreate(PETSC_COMM_WORLD,&ksp); > > KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); > > KSPGetPC(ksp,&pc); > > ?? ? ? ?KSPSetType(ksp, KSPGMRES); > > ?? ? ? ?PCSetType(pc,? PCBJACOBI); > > ?KSPSetTolerances(ksp,tol,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); > > ? KSPSetFromOptions(ksp); > > ? KSPSolve(ksp,b,x); > > ? for (i=0;i > ? ? ? VecGetValues(x,1,&i,&sol[i]); > > ? } > > > > ? VecDestroy(x); > > ? VecDestroy(b); > > ? MatDestroy(A); > > ? KSPDestroy(ksp); > > ? PetscFinalize(); > > } > > Once this function is call in a loop, it reports error. > On Sun, Oct 11, 2009 at 9:38 PM, Yin Feng wrote: >> >> I put PETSc solver in one function and use another function to call that. >> This problem only appears when I put the function with PETSc solver in a >> loop, >> it works well at first step, and reports error "An error occurred in >> MPI_Comm_rank after MPI was finalized" >> at second time. The program is designed to support only one processor >> like: >> >> Vec? ? ? ? ? ? x,b; >> >> Mat? ? ? ? ? ? A; >> >> KSP? ? ? ? ? ? ksp; >> >> PC ? ? ? ? ? ? pc; >> >> PetscInt ? ? ? i,j,col[N]; >> >> PetscScalar? ? value[N]; >> >> PetscScalar? ? val; >> >> PetscInitialize(PETSC_NULL,PETSC_NULL,PETSC_NULL,PETSC_NULL); >> >> VecCreate(PETSC_COMM_WORLD,&x); >> >> VecSetSizes(x,PETSC_DECIDE,N); >> >> VecSetFromOptions(x); >> >> VecDuplicate(x,&b); >> >> MatCreate(PETSC_COMM_WORLD,&A); >> >> MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N); >> >> MatSetFromOptions(A); >> >> ?? ? ? ?........ >> >> ?? ? ? ?........ >> >> KSPCreate(PETSC_COMM_WORLD,&ksp); >> >> KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); >> >> KSPGetPC(ksp,&pc); >> >> ?? ? ? ?................ >> ?? ? ? ?............... >> >> >> ?KSPSetTolerances(ksp,tol,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); >> >> ?? ? ? ?KSPSetFromOptions(ksp); >> >> ?? ? ? ?KSPSolve(ksp,b,x); >> >> ?? ? ? ?VecDestroy(x); >> >> ?? ? ? ?VecDestroy(b); >> >> ?? ? ? ?MatDestroy(A); >> >> ?? ? ? ?KSPDestroy(ksp); >> >> ?? ? ? ?PetscFinalize(); >> >> Any one has ideal about this? >> >> The detailed error description is: >> >> An error occurred in MPI_Comm_rank >> >> *** after MPI was finalized >> >> *** MPI_ERRORS_ARE_FATAL (goodbye) >> >> Abort before MPI_INIT completed successfully; not able to guarantee that >> all other processes were killed! >> >> Thank you so much in advance! > > From bsmith at mcs.anl.gov Sun Oct 11 22:06:49 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 11 Oct 2009 22:06:49 -0500 Subject: a problem with error of "An error occurred in MPI_Comm_rank after MPI was finalized" In-Reply-To: <1e8c69dc0910111959n3eab3efeg2c62e1b04bb7e5b5@mail.gmail.com> References: <1e8c69dc0910111938s6358a174o5c03ad98c0e24b9d@mail.gmail.com> <1e8c69dc0910111959n3eab3efeg2c62e1b04bb7e5b5@mail.gmail.com> Message-ID: MPI does NOT allow calling MPI_Init() twice. (By default PetscInitialize() calls MPI_Init().) You need to take the PetscInitialize() out of the subroutine and instead call it once at the beginning of your program. Barry On Oct 11, 2009, at 9:59 PM, Yin Feng wrote: > The completed code is: > > void PETSc(double mat[],double vec[],double sol[],long N,double tol) { > Vec x,b; > Mat A; > KSP ksp; > PC pc; > PetscInt i,j,col[N]; > PetscScalar value[N]; > PetscScalar val; > > PetscInitialize(PETSC_NULL,PETSC_NULL,PETSC_NULL,PETSC_NULL); > VecCreate(PETSC_COMM_WORLD,&x); > VecSetSizes(x,PETSC_DECIDE,N); > VecSetFromOptions(x); > VecDuplicate(x,&b); > > MatCreate(PETSC_COMM_WORLD,&A); > MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N); > MatSetFromOptions(A); > > for (i=0; i for (j=0;j value[j]=mat[i*N+j]; > col[j]=j; > } > MatSetValues(A,1,&i,N,col,value,INSERT_VALUES); > } > > MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); > > for (i=0; i val=vec[i]; > VecSetValues(b,1,&i,&val,INSERT_VALUES); > val=sol[i]; > VecSetValues(x,1,&i,&val,INSERT_VALUES); > } > > VecAssemblyBegin(b); > VecAssemblyEnd(b); > > KSPCreate(PETSC_COMM_WORLD,&ksp); > KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); > KSPGetPC(ksp,&pc); > KSPSetType(ksp, KSPGMRES); > PCSetType(pc, PCBJACOBI); > > KSPSetTolerances(ksp,tol,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); > KSPSetFromOptions(ksp); > KSPSolve(ksp,b,x); > > for (i=0;i VecGetValues(x,1,&i,&sol[i]); > } > > VecDestroy(x); > VecDestroy(b); > MatDestroy(A); > KSPDestroy(ksp); > PetscFinalize(); > } > > Once this function is call in a loop, it reports error. > > On Sun, Oct 11, 2009 at 9:38 PM, Yin Feng > wrote: > I put PETSc solver in one function and use another function to call > that. > This problem only appears when I put the function with PETSc solver > in a loop, > it works well at first step, and reports error "An error occurred in > MPI_Comm_rank after MPI was finalized" > at second time. The program is designed to support only one > processor like: > > Vec x,b; > Mat A; > KSP ksp; > PC pc; > PetscInt i,j,col[N]; > PetscScalar value[N]; > PetscScalar val; > > PetscInitialize(PETSC_NULL,PETSC_NULL,PETSC_NULL,PETSC_NULL); > VecCreate(PETSC_COMM_WORLD,&x); > VecSetSizes(x,PETSC_DECIDE,N); > VecSetFromOptions(x); > VecDuplicate(x,&b); > MatCreate(PETSC_COMM_WORLD,&A); > MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N); > MatSetFromOptions(A); > ........ > ........ > KSPCreate(PETSC_COMM_WORLD,&ksp); > KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); > KSPGetPC(ksp,&pc); > > ................ > ............... > > KSPSetTolerances(ksp,tol,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); > KSPSetFromOptions(ksp); > KSPSolve(ksp,b,x); > > VecDestroy(x); > VecDestroy(b); > MatDestroy(A); > KSPDestroy(ksp); > PetscFinalize(); > > Any one has ideal about this? > The detailed error description is: > An error occurred in MPI_Comm_rank > *** after MPI was finalized > *** MPI_ERRORS_ARE_FATAL (goodbye) > Abort before MPI_INIT completed successfully; not able to guarantee > that all other processes were killed! > > Thank you so much in advance! > > > From yfeng1 at tigers.lsu.edu Sun Oct 11 22:07:02 2009 From: yfeng1 at tigers.lsu.edu (Yin Feng) Date: Sun, 11 Oct 2009 22:07:02 -0500 Subject: a problem with error of "An error occurred in MPI_Comm_rank after MPI was finalized" In-Reply-To: <47a831090910112004w150cd6a2s843f22465a6ea2cb@mail.gmail.com> References: <1e8c69dc0910111938s6358a174o5c03ad98c0e24b9d@mail.gmail.com> <1e8c69dc0910111959n3eab3efeg2c62e1b04bb7e5b5@mail.gmail.com> <47a831090910112004w150cd6a2s843f22465a6ea2cb@mail.gmail.com> Message-ID: <1e8c69dc0910112007q678f401apb8c0877bf5900cd6@mail.gmail.com> yes, exactly On Sun, Oct 11, 2009 at 10:04 PM, Yaakoub El Khamra wrote: > This is for Mayank's project isn't it > > Regards > Yaakoub El Khamra > > > > > On Sun, Oct 11, 2009 at 9:59 PM, Yin Feng wrote: > > The completed code is: > > > > void PETSc(double mat[],double vec[],double sol[],long N,double tol) { > > > > Vec x,b; > > > > Mat A; > > > > KSP ksp; > > > > PC pc; > > > > PetscInt i,j,col[N]; > > > > PetscScalar value[N]; > > > > PetscScalar val; > > > > PetscInitialize(PETSC_NULL,PETSC_NULL,PETSC_NULL,PETSC_NULL); > > > > VecCreate(PETSC_COMM_WORLD,&x); > > > > VecSetSizes(x,PETSC_DECIDE,N); > > > > VecSetFromOptions(x); > > > > VecDuplicate(x,&b); > > > > MatCreate(PETSC_COMM_WORLD,&A); > > > > MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N); > > > > MatSetFromOptions(A); > > > > for (i=0; i > > > for (j=0;j > > > value[j]=mat[i*N+j]; > > > > col[j]=j; > > > > } > > > > MatSetValues(A,1,&i,N,col,value,INSERT_VALUES); > > > > } > > > > MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); > > > > MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); > > > > > > > > for (i=0; i > > > val=vec[i]; > > > > VecSetValues(b,1,&i,&val,INSERT_VALUES); > > > > val=sol[i]; > > > > VecSetValues(x,1,&i,&val,INSERT_VALUES); > > > > } > > > > VecAssemblyBegin(b); > > > > VecAssemblyEnd(b); > > > > KSPCreate(PETSC_COMM_WORLD,&ksp); > > > > KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); > > > > KSPGetPC(ksp,&pc); > > > > KSPSetType(ksp, KSPGMRES); > > > > PCSetType(pc, PCBJACOBI); > > > > KSPSetTolerances(ksp,tol,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); > > > > KSPSetFromOptions(ksp); > > > > KSPSolve(ksp,b,x); > > > > for (i=0;i > > > VecGetValues(x,1,&i,&sol[i]); > > > > } > > > > > > > > VecDestroy(x); > > > > VecDestroy(b); > > > > MatDestroy(A); > > > > KSPDestroy(ksp); > > > > PetscFinalize(); > > > > } > > > > Once this function is call in a loop, it reports error. > > On Sun, Oct 11, 2009 at 9:38 PM, Yin Feng wrote: > >> > >> I put PETSc solver in one function and use another function to call > that. > >> This problem only appears when I put the function with PETSc solver in a > >> loop, > >> it works well at first step, and reports error "An error occurred in > >> MPI_Comm_rank after MPI was finalized" > >> at second time. The program is designed to support only one processor > >> like: > >> > >> Vec x,b; > >> > >> Mat A; > >> > >> KSP ksp; > >> > >> PC pc; > >> > >> PetscInt i,j,col[N]; > >> > >> PetscScalar value[N]; > >> > >> PetscScalar val; > >> > >> PetscInitialize(PETSC_NULL,PETSC_NULL,PETSC_NULL,PETSC_NULL); > >> > >> VecCreate(PETSC_COMM_WORLD,&x); > >> > >> VecSetSizes(x,PETSC_DECIDE,N); > >> > >> VecSetFromOptions(x); > >> > >> VecDuplicate(x,&b); > >> > >> MatCreate(PETSC_COMM_WORLD,&A); > >> > >> MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N); > >> > >> MatSetFromOptions(A); > >> > >> ........ > >> > >> ........ > >> > >> KSPCreate(PETSC_COMM_WORLD,&ksp); > >> > >> KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); > >> > >> KSPGetPC(ksp,&pc); > >> > >> ................ > >> ............... > >> > >> > >> KSPSetTolerances(ksp,tol,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); > >> > >> KSPSetFromOptions(ksp); > >> > >> KSPSolve(ksp,b,x); > >> > >> VecDestroy(x); > >> > >> VecDestroy(b); > >> > >> MatDestroy(A); > >> > >> KSPDestroy(ksp); > >> > >> PetscFinalize(); > >> > >> Any one has ideal about this? > >> > >> The detailed error description is: > >> > >> An error occurred in MPI_Comm_rank > >> > >> *** after MPI was finalized > >> > >> *** MPI_ERRORS_ARE_FATAL (goodbye) > >> > >> Abort before MPI_INIT completed successfully; not able to guarantee that > >> all other processes were killed! > >> > >> Thank you so much in advance! > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yelkhamra at gmail.com Sun Oct 11 22:06:48 2009 From: yelkhamra at gmail.com (Yaakoub El Khamra) Date: Sun, 11 Oct 2009 22:06:48 -0500 Subject: a problem with error of "An error occurred in MPI_Comm_rank after MPI was finalized" In-Reply-To: <1e8c69dc0910111959n3eab3efeg2c62e1b04bb7e5b5@mail.gmail.com> References: <1e8c69dc0910111938s6358a174o5c03ad98c0e24b9d@mail.gmail.com> <1e8c69dc0910111959n3eab3efeg2c62e1b04bb7e5b5@mail.gmail.com> Message-ID: <47a831090910112006r374d23dv8260ad537c905618@mail.gmail.com> you need to set types for matrices. You also want to check the errors from the petsc functions. The function is CHKERRQ you also might want to do a CHKMEMQ. Regards Yaakoub El Khamra On Sun, Oct 11, 2009 at 9:59 PM, Yin Feng wrote: > The completed code is: > > void PETSc(double mat[],double vec[],double sol[],long N,double tol) { > > Vec? ? ? ? ? ? x,b; > > Mat? ? ? ? ? ? A; > > KSP? ? ? ? ? ? ksp; > > PC ? ? ? ? ? ? pc; > > PetscInt ? ? ? i,j,col[N]; > > PetscScalar? ? value[N]; > > PetscScalar? ? val; > > PetscInitialize(PETSC_NULL,PETSC_NULL,PETSC_NULL,PETSC_NULL); > > VecCreate(PETSC_COMM_WORLD,&x); > > VecSetSizes(x,PETSC_DECIDE,N); > > VecSetFromOptions(x); > > VecDuplicate(x,&b); > > MatCreate(PETSC_COMM_WORLD,&A); > > MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N); > > MatSetFromOptions(A); > > for (i=0; i > for (j=0;j > value[j]=mat[i*N+j]; > > col[j]=j; > > } > > MatSetValues(A,1,&i,N,col,value,INSERT_VALUES); > > } > > MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); > > MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); > > > > for (i=0; i > val=vec[i]; > > VecSetValues(b,1,&i,&val,INSERT_VALUES); > > val=sol[i]; > > VecSetValues(x,1,&i,&val,INSERT_VALUES); > > } > > VecAssemblyBegin(b); > > VecAssemblyEnd(b); > > KSPCreate(PETSC_COMM_WORLD,&ksp); > > KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); > > KSPGetPC(ksp,&pc); > > ?? ? ? ?KSPSetType(ksp, KSPGMRES); > > ?? ? ? ?PCSetType(pc,? PCBJACOBI); > > ?KSPSetTolerances(ksp,tol,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); > > ? KSPSetFromOptions(ksp); > > ? KSPSolve(ksp,b,x); > > ? for (i=0;i > ? ? ? VecGetValues(x,1,&i,&sol[i]); > > ? } > > > > ? VecDestroy(x); > > ? VecDestroy(b); > > ? MatDestroy(A); > > ? KSPDestroy(ksp); > > ? PetscFinalize(); > > } > > Once this function is call in a loop, it reports error. > On Sun, Oct 11, 2009 at 9:38 PM, Yin Feng wrote: >> >> I put PETSc solver in one function and use another function to call that. >> This problem only appears when I put the function with PETSc solver in a >> loop, >> it works well at first step, and reports error "An error occurred in >> MPI_Comm_rank after MPI was finalized" >> at second time. The program is designed to support only one processor >> like: >> >> Vec? ? ? ? ? ? x,b; >> >> Mat? ? ? ? ? ? A; >> >> KSP? ? ? ? ? ? ksp; >> >> PC ? ? ? ? ? ? pc; >> >> PetscInt ? ? ? i,j,col[N]; >> >> PetscScalar? ? value[N]; >> >> PetscScalar? ? val; >> >> PetscInitialize(PETSC_NULL,PETSC_NULL,PETSC_NULL,PETSC_NULL); >> >> VecCreate(PETSC_COMM_WORLD,&x); >> >> VecSetSizes(x,PETSC_DECIDE,N); >> >> VecSetFromOptions(x); >> >> VecDuplicate(x,&b); >> >> MatCreate(PETSC_COMM_WORLD,&A); >> >> MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N); >> >> MatSetFromOptions(A); >> >> ?? ? ? ?........ >> >> ?? ? ? ?........ >> >> KSPCreate(PETSC_COMM_WORLD,&ksp); >> >> KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); >> >> KSPGetPC(ksp,&pc); >> >> ?? ? ? ?................ >> ?? ? ? ?............... >> >> >> ?KSPSetTolerances(ksp,tol,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); >> >> ?? ? ? ?KSPSetFromOptions(ksp); >> >> ?? ? ? ?KSPSolve(ksp,b,x); >> >> ?? ? ? ?VecDestroy(x); >> >> ?? ? ? ?VecDestroy(b); >> >> ?? ? ? ?MatDestroy(A); >> >> ?? ? ? ?KSPDestroy(ksp); >> >> ?? ? ? ?PetscFinalize(); >> >> Any one has ideal about this? >> >> The detailed error description is: >> >> An error occurred in MPI_Comm_rank >> >> *** after MPI was finalized >> >> *** MPI_ERRORS_ARE_FATAL (goodbye) >> >> Abort before MPI_INIT completed successfully; not able to guarantee that >> all other processes were killed! >> >> Thank you so much in advance! > > From yfeng1 at tigers.lsu.edu Sun Oct 11 22:12:41 2009 From: yfeng1 at tigers.lsu.edu (Yin Feng) Date: Sun, 11 Oct 2009 22:12:41 -0500 Subject: a problem with error of "An error occurred in MPI_Comm_rank after MPI was finalized" In-Reply-To: <47a831090910112006r374d23dv8260ad537c905618@mail.gmail.com> References: <1e8c69dc0910111938s6358a174o5c03ad98c0e24b9d@mail.gmail.com> <1e8c69dc0910111959n3eab3efeg2c62e1b04bb7e5b5@mail.gmail.com> <47a831090910112006r374d23dv8260ad537c905618@mail.gmail.com> Message-ID: <1e8c69dc0910112012n220ac07n40870f384be5b188@mail.gmail.com> I know the function called CHKERRQ, but what does CHKMEMQ mean? On Sun, Oct 11, 2009 at 10:06 PM, Yaakoub El Khamra wrote: > you need to set types for matrices. You also want to check the errors > from the petsc functions. The function is CHKERRQ you also might want > to do a CHKMEMQ. > > > Regards > Yaakoub El Khamra > > > > > On Sun, Oct 11, 2009 at 9:59 PM, Yin Feng wrote: > > The completed code is: > > > > void PETSc(double mat[],double vec[],double sol[],long N,double tol) { > > > > Vec x,b; > > > > Mat A; > > > > KSP ksp; > > > > PC pc; > > > > PetscInt i,j,col[N]; > > > > PetscScalar value[N]; > > > > PetscScalar val; > > > > PetscInitialize(PETSC_NULL,PETSC_NULL,PETSC_NULL,PETSC_NULL); > > > > VecCreate(PETSC_COMM_WORLD,&x); > > > > VecSetSizes(x,PETSC_DECIDE,N); > > > > VecSetFromOptions(x); > > > > VecDuplicate(x,&b); > > > > MatCreate(PETSC_COMM_WORLD,&A); > > > > MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N); > > > > MatSetFromOptions(A); > > > > for (i=0; i > > > for (j=0;j > > > value[j]=mat[i*N+j]; > > > > col[j]=j; > > > > } > > > > MatSetValues(A,1,&i,N,col,value,INSERT_VALUES); > > > > } > > > > MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); > > > > MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); > > > > > > > > for (i=0; i > > > val=vec[i]; > > > > VecSetValues(b,1,&i,&val,INSERT_VALUES); > > > > val=sol[i]; > > > > VecSetValues(x,1,&i,&val,INSERT_VALUES); > > > > } > > > > VecAssemblyBegin(b); > > > > VecAssemblyEnd(b); > > > > KSPCreate(PETSC_COMM_WORLD,&ksp); > > > > KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); > > > > KSPGetPC(ksp,&pc); > > > > KSPSetType(ksp, KSPGMRES); > > > > PCSetType(pc, PCBJACOBI); > > > > KSPSetTolerances(ksp,tol,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); > > > > KSPSetFromOptions(ksp); > > > > KSPSolve(ksp,b,x); > > > > for (i=0;i > > > VecGetValues(x,1,&i,&sol[i]); > > > > } > > > > > > > > VecDestroy(x); > > > > VecDestroy(b); > > > > MatDestroy(A); > > > > KSPDestroy(ksp); > > > > PetscFinalize(); > > > > } > > > > Once this function is call in a loop, it reports error. > > On Sun, Oct 11, 2009 at 9:38 PM, Yin Feng wrote: > >> > >> I put PETSc solver in one function and use another function to call > that. > >> This problem only appears when I put the function with PETSc solver in a > >> loop, > >> it works well at first step, and reports error "An error occurred in > >> MPI_Comm_rank after MPI was finalized" > >> at second time. The program is designed to support only one processor > >> like: > >> > >> Vec x,b; > >> > >> Mat A; > >> > >> KSP ksp; > >> > >> PC pc; > >> > >> PetscInt i,j,col[N]; > >> > >> PetscScalar value[N]; > >> > >> PetscScalar val; > >> > >> PetscInitialize(PETSC_NULL,PETSC_NULL,PETSC_NULL,PETSC_NULL); > >> > >> VecCreate(PETSC_COMM_WORLD,&x); > >> > >> VecSetSizes(x,PETSC_DECIDE,N); > >> > >> VecSetFromOptions(x); > >> > >> VecDuplicate(x,&b); > >> > >> MatCreate(PETSC_COMM_WORLD,&A); > >> > >> MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N); > >> > >> MatSetFromOptions(A); > >> > >> ........ > >> > >> ........ > >> > >> KSPCreate(PETSC_COMM_WORLD,&ksp); > >> > >> KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); > >> > >> KSPGetPC(ksp,&pc); > >> > >> ................ > >> ............... > >> > >> > >> KSPSetTolerances(ksp,tol,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); > >> > >> KSPSetFromOptions(ksp); > >> > >> KSPSolve(ksp,b,x); > >> > >> VecDestroy(x); > >> > >> VecDestroy(b); > >> > >> MatDestroy(A); > >> > >> KSPDestroy(ksp); > >> > >> PetscFinalize(); > >> > >> Any one has ideal about this? > >> > >> The detailed error description is: > >> > >> An error occurred in MPI_Comm_rank > >> > >> *** after MPI was finalized > >> > >> *** MPI_ERRORS_ARE_FATAL (goodbye) > >> > >> Abort before MPI_INIT completed successfully; not able to guarantee that > >> all other processes were killed! > >> > >> Thank you so much in advance! > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yfeng1 at tigers.lsu.edu Sun Oct 11 23:01:37 2009 From: yfeng1 at tigers.lsu.edu (Yin Feng) Date: Sun, 11 Oct 2009 23:01:37 -0500 Subject: a problem with error of "An error occurred in MPI_Comm_rank after MPI was finalized" In-Reply-To: References: <1e8c69dc0910111938s6358a174o5c03ad98c0e24b9d@mail.gmail.com> <1e8c69dc0910111959n3eab3efeg2c62e1b04bb7e5b5@mail.gmail.com> Message-ID: <1e8c69dc0910112101m59190dck774daba2e9371dd9@mail.gmail.com> I move PetscInitialize and PetscFinalize out of loop and it works now. Thank you, Barry! While, I found it runs very slow, do you know the reason of it? On Sun, Oct 11, 2009 at 10:06 PM, Barry Smith wrote: > > MPI does NOT allow calling MPI_Init() twice. (By default PetscInitialize() > calls MPI_Init().) > > You need to take the PetscInitialize() out of the subroutine and instead > call it once at the beginning of your program. > > Barry > > > On Oct 11, 2009, at 9:59 PM, Yin Feng wrote: > > The completed code is: >> >> void PETSc(double mat[],double vec[],double sol[],long N,double tol) { >> Vec x,b; >> Mat A; >> KSP ksp; >> PC pc; >> PetscInt i,j,col[N]; >> PetscScalar value[N]; >> PetscScalar val; >> >> PetscInitialize(PETSC_NULL,PETSC_NULL,PETSC_NULL,PETSC_NULL); >> VecCreate(PETSC_COMM_WORLD,&x); >> VecSetSizes(x,PETSC_DECIDE,N); >> VecSetFromOptions(x); >> VecDuplicate(x,&b); >> >> MatCreate(PETSC_COMM_WORLD,&A); >> MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N); >> MatSetFromOptions(A); >> >> for (i=0; i> for (j=0;j> value[j]=mat[i*N+j]; >> col[j]=j; >> } >> MatSetValues(A,1,&i,N,col,value,INSERT_VALUES); >> } >> >> MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); >> MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); >> >> for (i=0; i> val=vec[i]; >> VecSetValues(b,1,&i,&val,INSERT_VALUES); >> val=sol[i]; >> VecSetValues(x,1,&i,&val,INSERT_VALUES); >> } >> >> VecAssemblyBegin(b); >> VecAssemblyEnd(b); >> >> KSPCreate(PETSC_COMM_WORLD,&ksp); >> KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); >> KSPGetPC(ksp,&pc); >> KSPSetType(ksp, KSPGMRES); >> PCSetType(pc, PCBJACOBI); >> >> KSPSetTolerances(ksp,tol,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); >> KSPSetFromOptions(ksp); >> KSPSolve(ksp,b,x); >> >> for (i=0;i> VecGetValues(x,1,&i,&sol[i]); >> } >> >> VecDestroy(x); >> VecDestroy(b); >> MatDestroy(A); >> KSPDestroy(ksp); >> PetscFinalize(); >> } >> >> Once this function is call in a loop, it reports error. >> >> On Sun, Oct 11, 2009 at 9:38 PM, Yin Feng wrote: >> I put PETSc solver in one function and use another function to call that. >> This problem only appears when I put the function with PETSc solver in a >> loop, >> it works well at first step, and reports error "An error occurred in >> MPI_Comm_rank after MPI was finalized" >> at second time. The program is designed to support only one processor >> like: >> >> Vec x,b; >> Mat A; >> KSP ksp; >> PC pc; >> PetscInt i,j,col[N]; >> PetscScalar value[N]; >> PetscScalar val; >> >> PetscInitialize(PETSC_NULL,PETSC_NULL,PETSC_NULL,PETSC_NULL); >> VecCreate(PETSC_COMM_WORLD,&x); >> VecSetSizes(x,PETSC_DECIDE,N); >> VecSetFromOptions(x); >> VecDuplicate(x,&b); >> MatCreate(PETSC_COMM_WORLD,&A); >> MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N); >> MatSetFromOptions(A); >> ........ >> ........ >> KSPCreate(PETSC_COMM_WORLD,&ksp); >> KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); >> KSPGetPC(ksp,&pc); >> >> ................ >> ............... >> >> KSPSetTolerances(ksp,tol,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); >> KSPSetFromOptions(ksp); >> KSPSolve(ksp,b,x); >> >> VecDestroy(x); >> VecDestroy(b); >> MatDestroy(A); >> KSPDestroy(ksp); >> PetscFinalize(); >> >> Any one has ideal about this? >> The detailed error description is: >> An error occurred in MPI_Comm_rank >> *** after MPI was finalized >> *** MPI_ERRORS_ARE_FATAL (goodbye) >> Abort before MPI_INIT completed successfully; not able to guarantee that >> all other processes were killed! >> >> Thank you so much in advance! >> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Oct 11 23:20:55 2009 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 11 Oct 2009 23:20:55 -0500 Subject: a problem with error of "An error occurred in MPI_Comm_rank after MPI was finalized" In-Reply-To: <1e8c69dc0910112101m59190dck774daba2e9371dd9@mail.gmail.com> References: <1e8c69dc0910111938s6358a174o5c03ad98c0e24b9d@mail.gmail.com> <1e8c69dc0910111959n3eab3efeg2c62e1b04bb7e5b5@mail.gmail.com> <1e8c69dc0910112101m59190dck774daba2e9371dd9@mail.gmail.com> Message-ID: Quick guess is that you do not preallocate a matrix correctly. Matt On Sun, Oct 11, 2009 at 11:01 PM, Yin Feng wrote: > I move PetscInitialize and PetscFinalize out of loop and it works now. > Thank you, Barry! > > While, I found it runs very slow, do you know the reason of it? > > > On Sun, Oct 11, 2009 at 10:06 PM, Barry Smith wrote: > >> >> MPI does NOT allow calling MPI_Init() twice. (By default >> PetscInitialize() calls MPI_Init().) >> >> You need to take the PetscInitialize() out of the subroutine and instead >> call it once at the beginning of your program. >> >> Barry >> >> >> On Oct 11, 2009, at 9:59 PM, Yin Feng wrote: >> >> The completed code is: >>> >>> void PETSc(double mat[],double vec[],double sol[],long N,double tol) { >>> Vec x,b; >>> Mat A; >>> KSP ksp; >>> PC pc; >>> PetscInt i,j,col[N]; >>> PetscScalar value[N]; >>> PetscScalar val; >>> >>> PetscInitialize(PETSC_NULL,PETSC_NULL,PETSC_NULL,PETSC_NULL); >>> VecCreate(PETSC_COMM_WORLD,&x); >>> VecSetSizes(x,PETSC_DECIDE,N); >>> VecSetFromOptions(x); >>> VecDuplicate(x,&b); >>> >>> MatCreate(PETSC_COMM_WORLD,&A); >>> MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N); >>> MatSetFromOptions(A); >>> >>> for (i=0; i>> for (j=0;j>> value[j]=mat[i*N+j]; >>> col[j]=j; >>> } >>> MatSetValues(A,1,&i,N,col,value,INSERT_VALUES); >>> } >>> >>> MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); >>> MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); >>> >>> for (i=0; i>> val=vec[i]; >>> VecSetValues(b,1,&i,&val,INSERT_VALUES); >>> val=sol[i]; >>> VecSetValues(x,1,&i,&val,INSERT_VALUES); >>> } >>> >>> VecAssemblyBegin(b); >>> VecAssemblyEnd(b); >>> >>> KSPCreate(PETSC_COMM_WORLD,&ksp); >>> KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); >>> KSPGetPC(ksp,&pc); >>> KSPSetType(ksp, KSPGMRES); >>> PCSetType(pc, PCBJACOBI); >>> >>> KSPSetTolerances(ksp,tol,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); >>> KSPSetFromOptions(ksp); >>> KSPSolve(ksp,b,x); >>> >>> for (i=0;i>> VecGetValues(x,1,&i,&sol[i]); >>> } >>> >>> VecDestroy(x); >>> VecDestroy(b); >>> MatDestroy(A); >>> KSPDestroy(ksp); >>> PetscFinalize(); >>> } >>> >>> Once this function is call in a loop, it reports error. >>> >>> On Sun, Oct 11, 2009 at 9:38 PM, Yin Feng wrote: >>> I put PETSc solver in one function and use another function to call that. >>> This problem only appears when I put the function with PETSc solver in a >>> loop, >>> it works well at first step, and reports error "An error occurred in >>> MPI_Comm_rank after MPI was finalized" >>> at second time. The program is designed to support only one processor >>> like: >>> >>> Vec x,b; >>> Mat A; >>> KSP ksp; >>> PC pc; >>> PetscInt i,j,col[N]; >>> PetscScalar value[N]; >>> PetscScalar val; >>> >>> PetscInitialize(PETSC_NULL,PETSC_NULL,PETSC_NULL,PETSC_NULL); >>> VecCreate(PETSC_COMM_WORLD,&x); >>> VecSetSizes(x,PETSC_DECIDE,N); >>> VecSetFromOptions(x); >>> VecDuplicate(x,&b); >>> MatCreate(PETSC_COMM_WORLD,&A); >>> MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N); >>> MatSetFromOptions(A); >>> ........ >>> ........ >>> KSPCreate(PETSC_COMM_WORLD,&ksp); >>> KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); >>> KSPGetPC(ksp,&pc); >>> >>> ................ >>> ............... >>> >>> KSPSetTolerances(ksp,tol,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); >>> KSPSetFromOptions(ksp); >>> KSPSolve(ksp,b,x); >>> >>> VecDestroy(x); >>> VecDestroy(b); >>> MatDestroy(A); >>> KSPDestroy(ksp); >>> PetscFinalize(); >>> >>> Any one has ideal about this? >>> The detailed error description is: >>> An error occurred in MPI_Comm_rank >>> *** after MPI was finalized >>> *** MPI_ERRORS_ARE_FATAL (goodbye) >>> Abort before MPI_INIT completed successfully; not able to guarantee that >>> all other processes were killed! >>> >>> Thank you so much in advance! >>> >>> >>> >>> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Mon Oct 12 01:34:49 2009 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Mon, 12 Oct 2009 08:34:49 +0200 Subject: a problem with error of "An error occurred in MPI_Comm_rank after MPI was finalized" In-Reply-To: <1e8c69dc0910111938s6358a174o5c03ad98c0e24b9d@mail.gmail.com> References: <1e8c69dc0910111938s6358a174o5c03ad98c0e24b9d@mail.gmail.com> Message-ID: <4AD2CE09.7090904@tu-dresden.de> Put the functions PetscInitialize(...) and PetscFinalize() out of the function when using it in a loop. Both functions have to be called only once in your program. Thomas Yin Feng wrote: > I put PETSc solver in one function and use another function to call that. > This problem only appears when I put the function with PETSc solver in > a loop, > it works well at first step, and reports error "An error occurred in > MPI_Comm_rank after MPI was finalized" > at second time. The program is designed to support only one processor > like: > > Vec x,b; > > Mat A; > > KSP ksp; > > PC pc; > > PetscInt i,j,col[N]; > > PetscScalar value[N]; > > PetscScalar val; > > > PetscInitialize(PETSC_NULL,PETSC_NULL,PETSC_NULL,PETSC_NULL); > > VecCreate(PETSC_COMM_WORLD,&x); > > VecSetSizes(x,PETSC_DECIDE,N); > > VecSetFromOptions(x); > > VecDuplicate(x,&b); > > MatCreate(PETSC_COMM_WORLD,&A); > > MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N); > > MatSetFromOptions(A); > > ........ > > ........ > > KSPCreate(PETSC_COMM_WORLD,&ksp); > > KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); > > KSPGetPC(ksp,&pc); > > ................ > ............... > > > KSPSetTolerances(ksp,tol,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT); > > KSPSetFromOptions(ksp); > > KSPSolve(ksp,b,x); > > > VecDestroy(x); > > VecDestroy(b); > > MatDestroy(A); > > KSPDestroy(ksp); > > PetscFinalize(); > > > Any one has ideal about this? > > The detailed error description is: > > An error occurred in MPI_Comm_rank > > *** after MPI was finalized > > *** MPI_ERRORS_ARE_FATAL (goodbye) > > Abort before MPI_INIT completed successfully; not able to guarantee > that all other processes were killed! > > > Thank you so much in advance! > From jarunan at ascomp.ch Mon Oct 12 07:21:52 2009 From: jarunan at ascomp.ch (jarunan at ascomp.ch) Date: Mon, 12 Oct 2009 14:21:52 +0200 Subject: MatView In-Reply-To: <4AD2CE09.7090904@tu-dresden.de> References: <1e8c69dc0910111938s6358a174o5c03ad98c0e24b9d@mail.gmail.com> <4AD2CE09.7090904@tu-dresden.de> Message-ID: <20091012142152.nq0m26lmo08g40wo@webmail.ascomp.ch> Hello, I have a matrix with global sizes 1176x1176 and I would like to see its values after setting value. Anyway with ASCII viewer allow to see not more than 512 rows. If I write to binary file, how can I inspect the value? Regards, Jarunan -- Jarunan Panyasantisuk Development Engineer ASCOMP GmbH, Technoparkstr. 1 CH-8005 Zurich, Switzerland Phone : +41 44 445 4072 Fax : +41 44 445 4075 E-mail: jarunan at ascomp.ch www.ascomp.ch From Andreas.Grassl at student.uibk.ac.at Mon Oct 12 08:03:04 2009 From: Andreas.Grassl at student.uibk.ac.at (Andreas Grassl) Date: Mon, 12 Oct 2009 15:03:04 +0200 Subject: SBAIJ issue Message-ID: <4AD32908.2050702@student.uibk.ac.at> Hello, I'm trying to work with MUMPS-Cholesky as direct solver and CG as iterative solver and have problems with SBAIJ-Matrices. I have a MatSetValues-line not regarding the symmetry but force it by the command line option -mat_ignore_lower_triangular. The problems show up at "bigger" problems (~450000 DOF) with "more" processes (1 and 2 work fine, 4 gives a hangup) and/or trying to save the matrix. The machine has 4 cores and 12 GB memory. NNZ per row is ~80. Attached you find the output of the program run with -info. Any hints where to search? Cheers, ando -- /"\ Grassl Andreas \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik X against HTML email Technikerstr. 13 Zi 709 / \ +43 (0)512 507 6091 -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 2procinfo URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 2procinfokspviewbinary URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 4procinfo URL: From hzhang at mcs.anl.gov Mon Oct 12 11:32:04 2009 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Mon, 12 Oct 2009 11:32:04 -0500 Subject: SBAIJ issue In-Reply-To: <4AD32908.2050702@student.uibk.ac.at> References: <4AD32908.2050702@student.uibk.ac.at> Message-ID: Ando, I do not see any error message from attached info below. Even '-log_summary' gives correct display. I guess you sent us the working output (np=2). I would suggest you run your code with debugger, e.g., '-start_in_debugger'. When it hangs, type Control-C, and type 'where' to check where it hangs. Hong On Oct 12, 2009, at 8:03 AM, Andreas Grassl wrote: > Hello, > > I'm trying to work with MUMPS-Cholesky as direct solver and CG as > iterative > solver and have problems with SBAIJ-Matrices. > > I have a MatSetValues-line not regarding the symmetry but force it > by the > command line option -mat_ignore_lower_triangular. > > The problems show up at "bigger" problems (~450000 DOF) with "more" > processes (1 > and 2 work fine, 4 gives a hangup) and/or trying to save the matrix. > The machine > has 4 cores and 12 GB memory. NNZ per row is ~80. > > Attached you find the output of the program run with -info. > > Any hints where to search? > > Cheers, > > ando > > -- > /"\ Grassl Andreas > \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik > X against HTML email Technikerstr. 13 Zi 709 > / \ +43 (0)512 507 6091 > same call as 2procinfo without -ksp_info_binary > > mpiexec -np 2 externalsolver > [0] PetscInitialize(): PETSc successfully started: number of > processors = 2 > [1] PetscInitialize(): PETSc successfully started: number of > processors = 2 > [1] PetscInitialize(): Running on machine: mat1.uibk.ac.at > [0] PetscInitialize(): Running on machine: mat1.uibk.ac.at > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > [0] PetscCommDuplicate(): returning tag 2147483647 > [1] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > [1] PetscCommDuplicate(): returning tag 2147483647 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): returning tag 2147483646 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [0] PetscCommDuplicate(): returning tag 2147483646 > [0] MatSetUpPreallocation(): Warning not preallocating matrix storage > [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374783 max tags = 2147483647 > [0] PetscCommDuplicate(): returning tag 2147483647 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374783 max tags = 2147483647 > [1] PetscCommDuplicate(): returning tag 2147483647 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483646 > [0] PetscCommDuplicate(): returning tag 2147483646 > [0] MatSetOption_MPISBAIJ(): Option SYMMETRIC ignored > [0] MatStashScatterBegin_Private(): No of messages: 1 > [0] MatStashScatterBegin_Private(): Mesg_to: 1: size: 30544064 > [0] MatStashScatterBegin_Private(): No of messages: 0 > [0] MatAssemblyBegin_MPISBAIJ(): Stash has 3818007 entries,uses 8 > mallocs. > [0] MatAssemblyBegin_MPISBAIJ(): Block-Stash has 3818007 entries, > uses 8 mallocs. > [0] MatAssemblyEnd_SeqSBAIJ(): Matrix size: 221545 X 221545, block > size 1; storage space: 16676676 unneeded, 5477824 used > [0] MatAssemblyEnd_SeqSBAIJ(): Number of mallocs during MatSetValues > is 0 > [0] MatAssemblyEnd_SeqSBAIJ(): Most nonzeros blocks in any row is 62 > [1] MatAssemblyEnd_SeqSBAIJ(): Matrix size: 225367 X 225367, block > size 1; storage space: 16793278 unneeded, 5743422 used > [1] MatAssemblyEnd_SeqSBAIJ(): Number of mallocs during MatSetValues > is 0 > [1] MatAssemblyEnd_SeqSBAIJ(): Most nonzeros blocks in any row is 65 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483645 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483644 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483643 > [1] PetscCommDuplicate(): returning tag 2147483639 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483645 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483644 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483643 > [0] PetscCommDuplicate(): returning tag 2147483639 > [0] PetscCommDuplicate(): returning tag 2147483634 > [1] PetscCommDuplicate(): returning tag 2147483634 > [1] PetscCommDuplicate(): returning tag 2147483630 > [1] PetscCommDuplicate(): returning tag 2147483625 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483642 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483641 > [1] PetscCommDuplicate(): returning tag 2147483620 > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > [0] VecScatterCreate(): General case: MPI to Seq > [0] PetscCommDuplicate(): returning tag 2147483630 > [0] PetscCommDuplicate(): returning tag 2147483625 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483642 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483641 > [0] PetscCommDuplicate(): returning tag 2147483620 > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > [0] VecScatterCreate(): General case: MPI to MPI > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483640 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483639 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDupli[1] PetscCommDuplicate(): Using internal PETSc > communicator 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483640 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483639 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483638 > [1] MatAssemblyEnd_SeqBAIJ(): Matrix size: 225367 X 0, block size 1; > storage space: 22536700 unneeded, 0 used > [1] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues > is 0 > [1] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 0 > [1] Mat_CheckCompressedRow(): Found the ratio (num_zerorows 225367)/ > (num_localrows 225367) > 0.6. Use CompressedRow routines. > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): returning tag 2147483614 > [1] PetscCommDuplicate(): returning tag 2147483613 > [1]before KSPsetup > [1] PetscCommDuplicate(): Using cate(): returning tag 2147483638 > [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 221545 X 6275, block size > 1; storage space: 22060506 unneeded, 93994 used > [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues > is 0 > [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 32 > [0] Mat_CheckCompressedRow(): Found the ratio (num_zerorows 216534)/ > (num_localrows 221545) > 0.6. Use CompressedRow routines. > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [0] PetscCommDuplicate(): returning tag 2147483614 > [0] PetscCommDuplicate(): returning tag 2147483613 > [0]before KSPsetup > [0] PCSetUp(): Setting up new PC > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483637 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483636 > [0] PetscCommDuplicate(): returning tag 2147483612 > [0] PetscCommDuplicate(): Using internainternal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483637 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483636 > [1] PetscCommDuplicate(): returning tag 2147483612 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483635 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483634 > l PETSc communicator 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483635 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483634 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483633 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483632 > [1] PetscCommDuplicate(): returning tag 2147483607 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483633 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483632 > [0] PetscCommDuplicate(): returning tag 2147483607 > [0] PetscCommDuplicate(): returning tag 2147483602 > [0] VecScatterCreate(): Special case: processor zero gets entire > parallel vector, rest get none > [1] PetscCommDuplicate(): returning tag 2147483602 > > DMUMPS 4.8.4 > L D L^T Solver for symmetric positive definite matrices > Type of parallelism: Working host > > ****** ANALYSIS STEP ******** > > Density: NBdense, Average, Median = 0 49 50 > Ordering based on METIS > ** Peak of sequential stack size (number of real entries) : > 16763905. > A root of estimated size 2965 has been selected for > Scalapack. > > Leaving analysis phase with ... > INFOG(1) = 0 > INFOG(2) = 0 > -- (20) Number of entries in factors (estim.) = 120657071 > -- (3) Storage of factors (REAL, estimated) = 137364268 > -- (4) Storage of factors (INT , estimated) = 6138804 > -- (5) Maximum frontal size (estimated) = 3705 > -- (6) Number of nodes in the tree = 21863 > -- (7) Ordering option effectively used = 5 > ICNTL(6) Maximum transversal option = 0 > ICNTL(7) Pivot order option = 7 > Percentage of memory relaxation (effective) = 200 > Number of level 2 nodes = 1 > Number of split nodes = 0 > RINFO(1) Operations during elimination (estim) = 1.114D+11 > Distributed matrix entry format (ICNTL(18)) = 3 > ** Rank of proc needing largest memory in IC facto : 0 > ** Estimated corresponding MBYTES for IC facto : 2085 > ** Estimated avg. MBYTES per work. proc at facto (IC) : 2044 > ** TOTAL space in MBYTES for IC factorization : 4088 > ** Rank of proc needing largest memory for OOC facto : 1 > ** Estimated corresponding MBYTES for OOC facto : 644 > ** Estimated avg. MBYTES per work. proc at facto (OOC) : 630 > ** TOTAL space in MBYTES for OOC factorization : 1261 > > ****** FACTORIZATION STEP ******** > > > GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ... > NUMBER OF WORKING PROCESSES = 2 > REAL SPACE FOR FACTORS = 137364268 > INTEGER SPACE FOR FACTORS = 6138804 > MAXIMUM FRONTAL SIZE (ESTIMATED) = 3705 > NUMBER OF NODES IN THE TREE = 21863 > Maximum effective relaxed size of S = 239060000 > Average effective relaxed size of S = 234725641 > > REDISTRIB: TOTAL DATA LOCAL/SENT = 9086109 2238491 > GLOBAL TIME FOR MATRIX DISTRIBUTION = 0.7322 > ** Memory relaxation parameter ( ICNTL(14) ) : 200 > ** Rank of processor needing largest memory in facto : 0 > ** Space in MBYTES used by this processor for facto : 2085 > ** Avg. Space in MBYTES per working proc during facto : 2044 > > ELAPSED TIME FOR FACTORIZATION = 28.1724 > Maximum effective space used in S (KEEP(67) = 77516871 > Average effective space used in S (KEEP(67) = 75395654 > ** EFF Min: Rank of processor needing largest memory : 0 > ** EFF Min: Space in MBYTES used by this processor : 741 > ** EFF Min: Avg. Space in MBYTES per working proc : 720 > > GLOBAL STATISTICS > RINFOG(2) OPERATIONS DURING NODE ASSEMBLY = 2.565D+08 > ------(3) OPERATIONS DURING NODE ELIMINATION = 1.114D+11 > INFOG (9) REAL SPACE FOR FACTORS = 137364268 > INFOG(10) INTEGER SPACE FOR FACTORS = 6138809 > INFOG(11) MAXIMUM FRONT SIZE = 3705 > INFOG(29) NUMBER OF ENTRIES IN FACTORS = 116259976 > INFOG(12) NB OF NEGATIVE PIVOTS = 0 > INFOG(14) NUMBER OF MEMORY COMPRESS = 0 > [0]after KSPsetup > [0]RHS setup > [1]after KSPsetup > [1]RHS setup > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): returning tag 2147483601 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [0] PetscCommDuplicate(): returning tag 2147483601 > [0] PetscCommDuplicate(): returning tag 2147483596 > [1] PetscCommDuplicate(): returning tag 2147483596 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [0] PetscCommDuplicate(): returning tag 2147483591 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): returning tag 2147483591 > [0] VecAssemblyBegin_MPI(): Stash has 225367 entries, uses 12 mallocs. > [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 mallocs. > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483631 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483631 > > > ****** SOLVE & CHECK STEP ******** > > > STATISTICS PRIOR SOLVE PHASE ........... > NUMBER OF RIGHT-HAND-SIDES = 1 > BLOCKING FACTOR FOR MULTIPLE RHS = 1 > ICNTL (9) = 1 > --- (10) = 0 > --- (11) = 0 > --- (20) = 0 > --- (21) = 1 > ** Rank of processor needing largest memory in solve : 0 > ** Space in MBYTES used by this processor for solve : 2013 > ** Avg. Space in MBYTES per working proc during solve : 1975 > > > LEAVING SOLVER WITH: INFOG(1) ............ = 0 > INFOG(2) ............ = 0 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483630 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483629 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483630 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483629 > [0] PetscCommDuplicate(): returning tag 2147483590 > [1] PetscCommDuplicate(): returning tag 2147483590 > [1] VecScatterCreate(): General case: Seq to MPI > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > [0] VecScatterCreate(): General case: Seq to MPI > [0] PetscCommDuplicate(): returning tag 2147483586 > KSP Object: > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > PC Object: > type: cholesky > Cholesky: out-of-place factorization > matrix ordering: natural > Cholesky: factor fill ratio needed 0 > Factored matrix follows > Matrix Object: > type=mpisbaij, rows=446912, cols=446912 > package used to perform factorization: mumps > [1] PetscCommDuplicate(): returning tag 2147483586 > [1] PetscCommDuplicate(): returning tag 2147483585 > [1] PetscCommDuplicate(): returning tag 2147483584 > [1] PetscCommDuplicate(): returning tag 2147483583 > [1] PetscCommDuplicate(): returning tag 2147483582 > [1] PetscCommDuplicate(): returning tag 2147483581 > [1] PetscCommDuplicate(): returning tag 2147483580 > total: nonzeros=0, allocated nonzeros=893824 > MUMPS run parameters: > SYM (matrix type): 1 > PAR (host participation): 1 > ICNTL(1) (output for error): 6 > ICNTL(2) (output of diagnostic msg):0 > ICNTL(3) (output for global info): 6 > ICNTL(4) (level of printing): -1 > ICNTL(5) (input mat struct): 0 > ICNTL(6) (matrix prescaling): 0 > ICNTL(7) (matrix ordering): 7 > ICNTL(8) (scalling strategy): 77 > ICNTL(9) (A/A^T x=b is solved): 1 > ICNTL(10) (max num of refinements): 0 > ICNTL(11) (error analysis): 0 > ICNTL(12) (efficiency control): 1 > ICNTL(13) (efficiency control): 0 > ICNTL(14) (percentage of estimated workspace increase): > 200 > ICNTL(18)[1] PetscCommDuplicate(): returning tag > 2147483579 > (input mat struct): 3 > ICNTL(19) (Shur complement info): 0 > ICNTL(20) (rhs sparse pattern): 0 > ICNTL(21) (solution struct): 1 > ICNTL(22) (in-core/out-of-core facility): 0 > ICNTL(23) (max size of memory can be allocated locally):0 > ICNTL(24) (detection of null pivot rows): 0 > ICNTL(25) (computation of a null space basis): 0 > ICNTL(26) (Schur options for rhs or solution): 0 > ICNTL(27) (experimental parameter): > -8 > CNTL(1) (relative pivoting threshold): 0 > CNTL(2) (stopping criterion of refinement): 1.49012e-08 > CNTL(3) (absolute pivoting threshold): 0 > CNTL(4) (value of static pivoting): -1 > CNTL(5) (fixation for null pivots): 0 > RINFO(1) (local estimated flops for the elimination after > analysis): > [0] 5.16132e+10 > [0] PetscCommDuplicate(): returning tag 2147483585 > [1] 5.97957e+10 > RINFO(2) (local estimated flops for the assembly after > factorization): > [0] 1.33366e+08 > [0] PetscCommDuplicate(): returning tag 2147483584 > [1] 1.23105e+08 > RINFO(3) (local estimated flops for the elimination after > factorization): > [0] 5.16124e+10 > [0] PetscCommDuplicate(): returning tag 2147483583 > [1] 5.9795e+10 > INFO(15) (estimated size of (in MB) MUMPS internal data for > running numerical factorization): > [0] 2085 > [0] PetscCommDuplicate(): returning tag 2147483582 > [1] 2003 > INFO(16) (size of (in MB) MUMPS internal data used during > numerical factorization): > [0] 2085 > [0] PetscCommDuplicate(): returning tag 2147483581 > [1] 2003 > INFO(23) (num of pivots eliminated on this processor after > factorization): > [0] 236610 > [0] PetscCommDuplicate(): returning tag 2147483580 > [1] 210302 > RINFOG(1) (global estimated flops for the elimination > after analysis): 1.11409e+11 > RINFOG(2) (global estimated flops for the assembly > after factorization): 2.56471e+08 > RINFOG(3) (global estimated flops for the elimination > after factorization): 1.11407e+11 > INFOG(3) (estimated real workspace for factors on all > processors after analysis): 137364268 > INFOG(4) (estimated integer workspace for factors on > all processors after analysis): 6138804 > INFOG(5) (estimated maximum front size in the complete > tree): 3705 > INFOG(6) (number of nodes in the complete tree): 21863 > INFOG(7) (ordering option effectively uese after > analysis): 5 > INFOG(8) (structural symmetry in percent of the > permuted matrix after analysis): 100 > INFOG(9) (total real/complex workspace to store the > matrix factors after factorization): 137364268 > INFOG(10) (total integer space store the matrix factors > after factorization): 6138809 > INFOG(11) (order of largest frontal matrix after > factorization): 3705 > INFOG(12) (number of off-diagonal pivots): 0 > INFOG(13) (number of delayed pivots after > factorization): 0 > INFOG(14) (number of memory compress after > factorization): 0 > INFOG(15) (number of steps of iterative refinement > after solution): 0 > INFOG(16) (estimated size (in MB) of all MUMPS internal > data for factorization after analysis: value on the most memory > consuming processor): 2085 > INFOG(17) (estimated size of all MUMPS internal data > for factorization after analysis: sum over all processors): 4088 > INFOG(18) (size of all MUMPS internal data allocated > during factorization: value on the most memory consuming processor): > 2085 > INFOG(19) (size of all MUMPS internal data allocated > during factorization: sum over all processors): 4088 > INFOG(20) (estimated number of entries in the factors): > 120657071 > INFOG(21) (size in MB of memory effectively used during > factorization - value on the most memory consuming processor): 741 > INFOG(22) (size in MB of memory effectively used during > factorization - sum over all processors): 1441 > INFOG(23) (after analysis: value of ICNTL(6) > effectively used): 0 > INFOG(24) (after analysis: value of ICNTL(12) > effectively used): 1 > INFOG(25) (after factorization: number of pivots > modified by static pivoting): 0 > linear system matrix = precond matrix: > Matrix Object: > type=mpisbaij, rows=446912, cols=446912 > total: nonzeros=11315240, allocated nonzeros=89382400 > block size is 1 > [0] PetscCommDuplicate(): returning tag 2147483579 > [1] PetscCommDuplicate(): returning tag 2147483578 > [1] PetscCommDuplicate(): returning tag 2147483577 > [1] Petsc_DelViewer(): Deleting viewer data in an MPI_Comm -2080374784 > [0] PetscCommDuplicate(): returning tag 2147483578 > [0] PetscCommDuplicate(): returning tag 2147483577 > [0] Petsc_DelViewer(): Deleting viewer data in an MPI_Comm -2080374784 > [0] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm 1140850689 > [0] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374783 > [0] Petsc_DelTag(): Deleting tag data in an MPI_Comm -2080374783 > [0] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483576 > [1] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm 1140850689 > [1] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374783 > [1] Petsc_DelTag(): Deleting tag data in an MPI_Comm -2080374783 > [0] Petsc_DelViewer(): Deleting viewer data in an MPI_Comm -2080374784 > [0] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm 1140850688 > [0] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374784 > [0] Petsc_DelTag(): Deleting tag data in an MPI_Comm -2080374784 > [0] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm -2080374784 > [0] Petsc_DelViewer(): Deleting viewer data in an MPI_Comm -2080374784 > ************************************************************************************************************************ > *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript - > r -fCourier9' to print this document *** > ************************************************************************************************************************ > > ---------------------------------------------- PETSc Performance > Summary: ---------------------------------------------- > > [1] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483576 > [1] Petsc_DelViewer(): Deleting viewer data in an MPI_Comm -2080374784 > [1] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm 1140850688 > [1] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374784 > [1] Petsc_DelTag(): Deleting tag data in an MPI_Comm -2080374784 > [1] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm -2080374784 > [1] Petsc_DelViewer(): Deleting viewer data in an MPI_Comm -2080374784 > externalsolver on a linux32-i named mat1.uibk.ac.at with 2 > processors, by csae1801 Mon Oct 12 12:08:38 2009 > Using Petsc Release Version 3.0.0, Patch 8, Fri Aug 21 14:02:12 CDT > 2009 > > Max Max/Min Avg Total > Time (sec): 3.989e+01 1.00000 3.989e+01 > Objects: 3.600e+01 1.00000 3.600e+01 > Flops: 0.000e+00 0.00000 0.000e+00 0.000e+00 > Flops/sec: 0.000e+00 0.00000 0.000e+00 0.000e+00 > MPI Messages: 2.550e+01 1.08511 2.450e+01 4.900e+01 > MPI Message Lengths: 6.596e+07 1.00053 2.692e+06 1.319e+08 > MPI Reductions: 5.800e+01 1.00000 > > Flop counting convention: 1 flop = 1 real number operation of type > (multiply/divide/add/subtract) > e.g., VecAXPY() for real vectors of > length N --> 2N flops > and VecAXPY() for complex vectors of > length N --> 8N flops > > Summary of Stages: ----- Time ------ ----- Flops ----- --- > Messages --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total counts > %Total Avg %Total counts %Total > 0: Main Stage: 2.8402e-01 0.7% 0.0000e+00 0.0% 1.000e > +00 2.0% 8.163e-02 0.0% 0.000e+00 0.0% > 1: decomposition: 4.7451e-01 1.2% 0.0000e+00 0.0% 0.000e > +00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% > 2: Matrix Create: 7.2766e-01 1.8% 0.0000e+00 0.0% 0.000e > +00 0.0% 0.000e+00 0.0% 2.000e+00 3.4% > 3: Matrix fill: 2.5720e+00 6.4% 0.0000e+00 0.0% 1.600e+01 > 32.7% 2.486e+06 92.4% 1.500e+01 25.9% > 4: Solver setup: 3.2841e+01 82.3% 0.0000e+00 0.0% 0.000e > +00 0.0% 0.000e+00 0.0% 8.000e+00 13.8% > 5: RHS setup: 3.2125e-02 0.1% 0.0000e+00 0.0% 4.000e > +00 8.2% 1.094e+05 4.1% 4.000e+00 6.9% > 6: Solve: 4.9751e-01 1.2% 0.0000e+00 0.0% 2.500e+01 > 51.0% 5.944e+04 2.2% 6.000e+00 10.3% > 7: Postproc: 2.4653e+00 6.2% 0.0000e+00 0.0% 3.000e > +00 6.1% 3.679e+04 1.4% 0.000e+00 0.0% > > ------------------------------------------------------------------------------------------------------------------------ > See the 'Profiling' chapter of the users' manual for details on > interpreting output. > Phase summary info: > Count: number of times phase was executed > Time and Flops: Max - maximum over all processors > Ratio - ratio of maximum to minimum over all > processors > Mess: number of messages sent > Avg. len: average message length > Reduct: number of global reductions > Global: entire computation > Stage: stages of a computation. Set stages with > PetscLogStagePush() and PetscLogStagePop(). > %T - percent time in this phase %F - percent flops in > this phase > %M - percent messages in this phase %L - percent message > lengths in this phase > %R - percent reductions in this phase > Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max > time over all processors) > ------------------------------------------------------------------------------------------------------------------------ > Event Count Time (sec) > Flops --- Global --- --- Stage --- > Total > Max Ratio Max Ratio Max Ratio Mess Avg > len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > PetscBarrier 1 1.0 3.1269e-0328.4 0.00e+00 0.0 0.0e+00 0.0e > +00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 > > --- Event Stage 1: decomposition > > > --- Event Stage 2: Matrix Create > > > --- Event Stage 3: Matrix fill > > MatAssemblyBegin 1 1.0 3.3235e-01 1.9 0.00e+00 0.0 6.0e+00 2.0e > +07 3.0e+00 1 0 12 92 5 10 0 38100 20 0 > MatAssemblyEnd 1 1.0 1.0536e+00 1.0 0.00e+00 0.0 1.0e+01 1.8e > +04 1.2e+01 3 0 20 0 21 41 0 62 0 80 0 > > --- Event Stage 4: Solver setup > > MatCholFctrSym 1 1.0 4.0531e-06 2.1 0.00e+00 0.0 0.0e+00 0.0e > +00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatCholFctrNum 1 1.0 3.2830e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e > +00 5.0e+00 82 0 0 0 9 100 0 0 0 62 0 > KSPSetup 1 1.0 6.9141e-06 1.4 0.00e+00 0.0 0.0e+00 0.0e > +00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > PCSetUp 1 1.0 3.2841e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e > +00 8.0e+00 82 0 0 0 14 100 0 0 0100 0 > > --- Event Stage 5: RHS setup > > VecAssemblyBegin 1 1.0 6.2160e-03 1.6 0.00e+00 0.0 4.0e+00 1.3e > +06 3.0e+00 0 0 8 4 5 16 0100100 75 0 > VecAssemblyEnd 1 1.0 8.4589e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e > +00 0.0e+00 0 0 0 0 0 26 0 0 0 0 0 > > --- Event Stage 6: Solve > > MatSolve 1 1.0 4.9525e-01 1.0 0.00e+00 0.0 6.0e+00 4.8e > +05 4.0e+00 1 0 12 2 7 100 0 24 98 67 0 > MatView 2 1.0 1.4269e-03 1.0 0.00e+00 0.0 1.8e+01 2.7e > +03 2.0e+00 0 0 37 0 3 0 0 72 2 33 0 > VecSet 1 1.0 8.1515e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e > +00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecScatterBegin 2 1.0 6.7840e-03 1.3 0.00e+00 0.0 2.0e+00 1.3e > +06 0.0e+00 0 0 4 2 0 1 0 8 86 0 0 > VecScatterEnd 1 1.0 2.1610e-03 1.7 0.00e+00 0.0 0.0e+00 0.0e > +00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > KSPSolve 1 1.0 4.9603e-01 1.0 0.00e+00 0.0 6.0e+00 4.8e > +05 4.0e+00 1 0 12 2 7 100 0 24 98 67 0 > PCApply 1 1.0 4.9526e-01 1.0 0.00e+00 0.0 6.0e+00 4.8e > +05 4.0e+00 1 0 12 2 7 100 0 24 98 67 0 > > --- Event Stage 7: Postproc > > VecView 1 1.0 2.1809e+00 2.0 0.00e+00 0.0 2.0e+00 9.0e > +05 0.0e+00 4 0 4 1 0 66 0 67100 0 0 > ------------------------------------------------------------------------------------------------------------------------ > > Memory usage is given in bytes: > > Object Type Creations Destructions Memory Descendants' > Mem. > > --- Event Stage 0: Main Stage > > IS L to G Mapping 0 1 889756 0 > Matrix 0 3 547690752 0 > Vec 0 6 3702888 0 > Vec Scatter 0 2 1736 0 > Viewer 0 1 544 0 > > --- Event Stage 1: decomposition > > > --- Event Stage 2: Matrix Create > > IS L to G Mapping 1 0 0 0 > Matrix 3 0 0 0 > > --- Event Stage 3: Matrix fill > > Index Set 4 4 152680 0 > Vec 7 1 1304 0 > Vec Scatter 2 0 0 0 > > --- Event Stage 4: Solver setup > > Index Set 3 2 1008 0 > Matrix 3 0 0 0 > Vec 2 1 1773664 0 > Vec Scatter 1 0 0 0 > Krylov Solver 1 0 0 0 > Preconditioner 1 0 0 0 > > --- Event Stage 5: RHS setup > > Vec 2 0 0 0 > Viewer 1 0 0 0 > > --- Event Stage 6: Solve > > Index Set 2 2 947456 0 > Vec 1 0 0 0 > Vec Scatter 1 0 0 0 > Viewer 1 0 0 0 > > --- Event Stage 7: Postproc > > Index Set 0 1 504 0 > Matrix 0 3 8870136 0 > Vec 0 4 7125104 0 > Vec Scatter 0 2 1320 0 > Krylov Solver 0 1 832 0 > Preconditioner 0 1 728 0 > Viewer 0 1 544 0 > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > =================================================================[1] > PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > [1] PetscCommDuplicate(): returning tag 2147483647 > [1] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm 1140850688 > [1] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374784 > [1] Petsc_DelTag(): Deleting tag data in an MPI_Comm -2080374784 > [1] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm -2080374784 > ======================================== > Average time to get PetscTime(): 1.3113e-06 > Average time for MPI_Barrier(): 0.000144005 > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > [0] PetscCommDuplicate(): returning tag 2147483647 > Average time for zero size MPI_Send(): 6.54459e-05 > [0] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm 1140850688 > [0] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374784 > [0] Petsc_DelTag(): Deleting tag data in an MPI_Comm -2080374784 > [0] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm -2080374784 > #PETSc Option Table entries: > -dump_data false > -info > -ksp_compute_singularvalues > -ksp_rtol 1e-6 > -ksp_type preonly > -ksp_view > -log_summary > -mat_ignore_lower_triangular > -mat_mumps_icntl_14 200 > -mat_mumps_sym 1 > -mat_type sbaij > -monitorname errormonitor.dat > -nz 100 > -options_left > -pc_factor_mat_solver_package mumps > -pc_type cholesky > -preload false > -verb false > #End o PETSc Option Table entries > Compiled without FORTRAN kernels > Compiled with full precision matrices (default) > sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 > sizeof(PetscScalar) 8 > Configure run at: Wed Oct 7 18:16:40 2009 > Configure options: --with-scalar-type=real --with-debugging=0 --with- > precision=double --with-cc=icc --with-fc=ifort --with-cxx=icpc -- > with-shared=0 --with-mpi=1 --with-external-packages-dir=/home/lux/ > csae1801/petsc/externalpackages --download-mpich=ifneeded --with- > scalapack=1 --download-scalapack=ifneeded --download-f-blas- > lapack=ifneeded --with-blacs=1 --download-blacs=ifneeded --with- > parmetis=1 --download-parmetis=ifneeded --with-mumps=1 --download- > mumps=ifneeded --with-hypre=1 --with-hypre-dir=/home/lux/csae1801/ > petsc/externalpackages/hypre-2.4.0b/src/hypre --with-spooles=1 -- > download-spooles=ifneeded --with-superlu_dist=1 --download- > superlu_dist=ifneeded PETSC_ARCH=linux32-intel-c-prod > ----------------------------------------- > Libraries compiled on Mi 7. Okt 18:17:41 CEST 2009 on mat1.uibk.ac.at > Machine characteristics: Linux mat1.uibk.ac.at 2.6.18-53.el5xen #1 > SMP Wed Oct 10 16:48:44 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux > Using PETSc directory: /home/lux/csae1801/petsc/petsc-3.0.0-p8 > Using PETSc arch: linux32-intel-c-prod > ----------------------------------------- > Using C compiler: /home/lux/csae1801/petsc/petsc-3.0.0-p8/linux32- > intel-c-prod/bin/mpicc -O > Using Fortran compiler: /home/lux/csae1801/petsc/petsc-3.0.0-p8/ > linux32-intel-c-prod/bin/mpif90 -O > ----------------------------------------- > Using include paths: -I/home/lux/csae1801/petsc/petsc-3.0.0-p8/ > linux32-intel-c-prod/include -I/home/lux/csae1801/petsc/petsc-3.0.0- > p8/include -I/home/lux/csae1801/petsc/petsc-3.0.0-p8/linux32-intel-c- > prod/include -I/home/lux/csae1801/petsc/externalpackages/ > hypre-2.4.0b/src/hypre/include > ------------------------------------------ > Using C linker: /home/lux/csae1801/petsc/petsc-3.0.0-p8/linux32- > intel-c-prod/bin/mpicc -O > Using Fortran linker: /home/lux/csae1801/petsc/petsc-3.0.0-p8/ > linux32-intel-c-prod/bin/mpif90 -O > Using libraries: -Wl,-rpath,/home/lux/csae1801/petsc/petsc-3.0.0-p8/ > linux32-intel-c-prod/lib -L/home/lux/csae1801/petsc/petsc-3.0.0-p8/ > linux32-intel-c-prod/lib -lpetscts -lpetscsnes -lpetscksp -lpetscdm - > lpetscmat -lpetscvec -lpetsc -lX11 -Wl,-rpath,/home/lux/ > csae1801/petsc/petsc-3.0.0-p8/linux32-intel-c-prod/lib -L/home/lux/ > csae1801/petsc/petsc-3.0.0-p8/linux32-intel-c-prod/lib - > lsuperlu_dist_2.3 -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common > -lpord -lparmetis -lmetis -lscalapack -lblacs -Wl,-rpath,/home/lux/ > csae1801/petsc/externalpackages/hypre-2.4.0b/src/hypre/lib -L/home/ > lux/csae1801/petsc/externalpackages/hypre-2.4.0b/src/hypre/lib - > lHYPRE -lmpichcxx -lstdc++ -lspooles -lflapack -lfblas -lnsl -laio - > lrt -lPEPCF90 -ldl -L/home/lux/csae1801/petsc/petsc-3.0.0-p8/linux32- > intel-c-prod/lib -lmpich -lpthread -lrt -L/opt/intel/cmkl/10.1.1.019/ > lib/em64t -L/opt/intel/Compiler/11.0/074/ipp/em64t/lib -L/opt/intel/ > Compiler/11.0/074/mkl/lib/em64t -L/opt/intel/Compiler/11.0/074/tbb/ > em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib -L/opt/intel/Compiler/ > 11.0/074/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -limf - > lsvml -lipgo -ldecimal -lirc -lgcc_s -lirc_s -lmpichf90 -lifport - > lifcore -lm -lm -lmpichcxx -lstdc++ -lmpichcxx -lstdc++ -ldl -lmpich > -lpthread -lrt -limf -lsvml -lipgo -ldecimal -lirc -lgcc_s -lirc_s - > ldl > ------------------------------------------ > #PETSc Option Table entries: > -dump_data false > -info > -ksp_compute_singularvalues > -ksp_rtol 1e-6 > -ksp_type preonly > -ksp_view > -log_summary > -mat_ignore_lower_triangular > -mat_mumps_icntl_14 200 > -mat_mumps_sym 1 > -mat_type sbaij > -monitorname errormonitor.dat > -nz 100 > -options_left > -pc_factor_mat_solver_package mumps > -pc_type cholesky > -preload false > -verb false > #End o PETSc Option Table entries > There are no unused options. > > 1olarr > > before sol put > sol put succes > before fullsol > /DIANA/AP/LS41 12:08:39 35.66-CPU 11.16-IO SOLVE > /DIANA/AP/LS41 12:08:39 35.66-CPU 11.18-IO FILLUP > /DIANA/AP/LS41 12:08:39 35.66-CPU 11.18-IO FILLUP > /DIANA/AP/LS41 12:08:39 35.66-CPU 11.18-IO POST > /DIANA/AP/LS41 12:08:41 36.79-CPU 12.27-IO POST > /DIANA/AP/LS41 12:08:41 36.79-CPU 12.27-IO LINSTA > /DIANA/DC/END 12:08:41 36.79-CPU 12.27-IO STOP > DIANA JOB 3446 finished > > 1 proc externalsolver stuck at 100% with 12%memory out of 12GB > 1 proc externalsolver at 0% > > mpiexec -np 2 externalsolver > [0] PetscInitialize(): PETSc successfully started: number of > processors = 2 > [0] PetscInitialize(): Running on machine: mat1.uibk.ac.at > [1] PetscInitialize(): PETSc successfully started: number of > processors = 2 > [1] PetscInitialize(): Running on machine: mat1.uibk.ac.at > [1] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > [1] PetscCommDuplicate(): returning tag 2147483647 > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > [0] PetscCommDuplicate(): returning tag 2147483647 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): returning tag 2147483646 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [0] PetscCommDuplicate(): returning tag 2147483646 > [0] MatSetUpPreallocation(): Warning not preallocating matrix storage > [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374783 max tags = 2147483647 > [1] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374783 max tags = 2147483647 > [1] PetscCommDuplicate(): returning tag 2147483647 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483646 > [0] PetscCommDuplicate(): returning tag 2147483647 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483646 > [0] MatSetOption_MPISBAIJ(): Option SYMMETRIC ignored > [0] MatStashScatterBegin_Private(): No of messages: 1 > [0] MatStashScatterBegin_Private(): Mesg_to: 1: size: 30544064 > [0] MatStashScatterBegin_Private(): No of messages: 0 > [0] MatAssemblyBegin_MPISBAIJ(): Stash has 3818007 entries,uses 8 > mallocs. > [0] MatAssemblyBegin_MPISBAIJ(): Block-Stash has 3818007 entries, > uses 8 mallocs. > [0] MatAssemblyEnd_SeqSBAIJ(): Matrix size: 221545 X 221545, block > size 1; storage space: 16676676 unneeded, 5477824 used > [0] MatAssemblyEnd_SeqSBAIJ(): Number of mallocs during MatSetValues > is 0 > [0] MatAssemblyEnd_SeqSBAIJ(): Most nonzeros blocks in any row is 62 > [1] MatAssemblyEnd_SeqSBAIJ(): Matrix size: 225367 X 225367, block > size 1; storage space: 16793278 unneeded, 5743422 used > [1] MatAssemblyEnd_SeqSBAIJ(): Number of mallocs during MatSetValues > is 0 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483645 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483644 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483643 > [0] PetscCommDuplicate(): returning tag 2147483639 > [0] PetscCommDuplicate(): returning tag 2147483634 > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > [0] VecScatterCreate(): General case: MPI to Seq > [0] PetscCommDuplicate(): returning tag 2147483630 > [0] PetscCommDuplicate(): returning tag 2147483625 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483642 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483641 > [0] PetscCommDuplicate[1] MatAssemblyEnd_SeqSBAIJ(): Most nonzeros > blocks in any row is 65 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483645 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483644 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483643 > [1] PetscCommDuplicate(): returning tag 2147483639 > [1] PetscCommDuplicate(): returning tag 2147483634 > [1] PetscCommDuplicate(): returning tag 2147483630 > [1] PetscCommDuplicate(): returning tag 2147483625 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483642 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483641 > [1] PetscCommDuplicate(): returning tag 2147483620 > [1] PetscC(): returning tag 2147483620 > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > [0] VecScatterCreate(): General case: MPI to MPI > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483640 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483639 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483638 > [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 221545 X 6275, block size > 1; storage space: 22060506 unneeded, 93994 used > [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues > is 0 > [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 32 > [0] Mat_CheckCompressedRow(): Found the ratio (num_zerorows 216534)/ > (num_localrows 221545) > 0.6. Use CompressedRow routines. > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [0] PetsommDuplicate(): Using internal PETSc communicator 1140850689 > -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483640 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483639 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483638 > [1] MatAssemblyEnd_SeqBAIJ(): Matrix size: 225367 X 0, block size 1; > storage space: 22536700 unneeded, 0 used > [1] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues > is 0 > [1] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 0 > [1] Mat_CheckCompressedRow(): Found the ratio (num_zerorows 225367)/ > (num_localrows 225367) > 0.6. Use CompressedRow routines. > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): returning tag 2147483614 > [1] PetscCommDuplicate(): returning tag 2147483613 > [1]before KSPsetup > [1] PetscCommDuplicate(): Using internal PcCommDuplicate(): > returning tag 2147483614 > [0] PetscCommDuplicate(): returning tag 2147483613 > [0]before KSPsetup > [0] PCSetUp(): Setting up new PC > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483637 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483636 > [0] PetscCommDuplicate(): returning tag 2147483612 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483635 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483634 > ETSc communicator 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483637 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483636 > [1] PetscCommDuplicate(): returning tag 2147483612 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483635 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483634 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483633 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483632 > [0] PetscCommDuplicate(): returning tag 2147483607 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483633 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483632 > [1] PetscCommDuplicate(): returning tag 2147483607 > [0] PetscCommDuplicate(): returning tag 2147483602 > [0] VecScatterCreate(): Special case: processor zero gets entire > parallel vector, rest get none > [1] PetscCommDuplicate(): returning tag 2147483602 > > DMUMPS 4.8.4 > L D L^T Solver for symmetric positive definite matrices > Type of parallelism: Working host > > ****** ANALYSIS STEP ******** > > Density: NBdense, Average, Median = 0 49 50 > Ordering based on METIS > ** Peak of sequential stack size (number of real entries) : > 16763905. > A root of estimated size 2965 has been selected for > Scalapack. > > Leaving analysis phase with ... > INFOG(1) = 0 > INFOG(2) = 0 > -- (20) Number of entries in factors (estim.) = 120657071 > -- (3) Storage of factors (REAL, estimated) = 137364268 > -- (4) Storage of factors (INT , estimated) = 6138804 > -- (5) Maximum frontal size (estimated) = 3705 > -- (6) Number of nodes in the tree = 21863 > -- (7) Ordering option effectively used = 5 > ICNTL(6) Maximum transversal option = 0 > ICNTL(7) Pivot order option = 7 > Percentage of memory relaxation (effective) = 200 > Number of level 2 nodes = 1 > Number of split nodes = 0 > RINFO(1) Operations during elimination (estim) = 1.114D+11 > Distributed matrix entry format (ICNTL(18)) = 3 > ** Rank of proc needing largest memory in IC facto : 0 > ** Estimated corresponding MBYTES for IC facto : 2085 > ** Estimated avg. MBYTES per work. proc at facto (IC) : 2044 > ** TOTAL space in MBYTES for IC factorization : 4088 > ** Rank of proc needing largest memory for OOC facto : 1 > ** Estimated corresponding MBYTES for OOC facto : 644 > ** Estimated avg. MBYTES per work. proc at facto (OOC) : 630 > ** TOTAL space in MBYTES for OOC factorization : 1261 > > ****** FACTORIZATION STEP ******** > > > GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ... > NUMBER OF WORKING PROCESSES = 2 > REAL SPACE FOR FACTORS = 137364268 > INTEGER SPACE FOR FACTORS = 6138804 > MAXIMUM FRONTAL SIZE (ESTIMATED) = 3705 > NUMBER OF NODES IN THE TREE = 21863 > Maximum effective relaxed size of S = 239060000 > Average effective relaxed size of S = 234725641 > > REDISTRIB: TOTAL DATA LOCAL/SENT = 9086109 2238491 > GLOBAL TIME FOR MATRIX DISTRIBUTION = 0.8155 > ** Memory relaxation parameter ( ICNTL(14) ) : 200 > ** Rank of processor needing largest memory in facto : 0 > ** Space in MBYTES used by this processor for facto : 2085 > ** Avg. Space in MBYTES per working proc during facto : 2044 > > ELAPSED TIME FOR FACTORIZATION = 28.2595 > Maximum effective space used in S (KEEP(67) = 77516871 > Average effective space used in S (KEEP(67) = 75395654 > ** EFF Min: Rank of processor needing largest memory : 0 > ** EFF Min: Space in MBYTES used by this processor : 741 > ** EFF Min: Avg. Space in MBYTES per working proc : 720 > > GLOBAL STATISTICS > RINFOG(2) OPERATIONS DURING NODE ASSEMBLY = 2.565D+08 > ------(3) OPERATIONS DURING NODE ELIMINATION = 1.114D+11 > INFOG (9) REAL SPACE FOR FACTORS = 137364268 > INFOG(10) INTEGER SPACE FOR FACTORS = 6138809 > INFOG(11) MAXIMUM FRONT SIZE = 3705 > INFOG(29) NUMBER OF ENTRIES IN FACTORS = 116259976 > INFOG(12) NB OF NEGATIVE PIVOTS = 0 > INFOG(14) NUMBER OF MEMORY COMPRESS = 0 > [0]after KSPsetup > [0]RHS setup > [1]after KSPsetup > [1]RHS setup > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): returning tag 2147483601 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [0] PetscCommDuplicate(): returning tag 2147483601 > [1] PetscCommDuplicate(): returning tag 2147483596 > [0] PetscCommDuplicate(): returning tag 2147483596 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): returning tag 2147483591 > [0] PetscCommDuplicate(): returning tag 2147483591 > [0] VecAssemblyBegin_MPI(): Stash has 225367 entries, uses 12 mallocs. > [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 mallocs. > [0] PetscCommDuplicate(): returning tag 2147483590 > [1] PetscCommDuplicate(): returning tag 2147483590 > [1] PetscCommDuplicate(): returning tag 2147483589 > [0] PetscCommDuplicate(): returning tag 2147483589 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483631 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483630 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483631 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483630 > > 4 processes of externalsolver stuck with 0% cpu > > > mpiexec -np 4 externalsolver > [1] PetscInitialize(): PETSc successfully started: number of > processors = 4 > [1] PetscInitialize(): Running on machine: mat1.uibk.ac.at > [3] PetscInitialize(): PETSc successfully started: number of > processors = 4 > [3] PetscInitialize(): Running on machine: mat1.uibk.ac.at > [0] PetscInitialize(): PETSc successfully started: number of > processors = 4 > [0] PetscInitialize(): Running on machine: mat1.uibk.ac.at > [2] PetscInitialize(): PETSc successfully started: number of > processors = 4 > [2] PetscInitialize(): Running on machine: mat1.uibk.ac.at > [3] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > [3] PetscCommDuplicate(): returning tag 2147483647 > [1] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > [1] PetscCommDuplicate(): returning tag 2147483647 > [2] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > [2] PetscCommDuplicate(): returning tag 2147483647 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [2] PetscCommDuplicate(): returning tag 2147483646 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [3] PetscCommDuplicate(): returning tag 2147483646 > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > [0] PetscCommDuplicate(): returning tag 2147483647 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [0] PetscCommDuplicate(): returning tag 2147483646 > [0] MatSetUpPreallocation(): Warning not preallocating matrix storage > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): returning tag 2147483646 > [1] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374783 max tags = 2147483647 > [3] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374783 max tags = 2147483647 > [3] PetscCommDuplicate(): returning tag 2147483647 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483646 > [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374783 max tags = 2147483647 > [0] PetscCommDuplicate(): returning tag 2147483647 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483646 > [1] PetscCommDuplicate(): returning tag 2147483647 > [2] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374783 max tags = 2147483647 > [2] PetscCommDuplicate(): returning tag 2147483647 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483646 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483646 > [0] MatSetOption_MPISBAIJ(): Option SYMMETRIC ignored > [0] MatStashScatterBegin_Private(): No of messages: 3 > [0] MatStashScatterBegin_Private(): Mesg_to: 1: size: 11261544 > [0] MatStashScatterBegin_Private(): Mesg_to: 2: size: 10384840 > [0] MatStashScatterBegin_Private(): Mesg_to: 3: size: 393344 > [0] MatStashScatterBegin_Private(): No of messages: 0 > [0] MatAssemblyBegin_MPISBAIJ(): Stash has 2754963 entries,uses 8 > mallocs. > [0] MatAssemblyBegin_MPISBAIJ(): Block-Stash has 2754963 entries, > uses 8 mallocs. > [3] MatAssemblyEnd_SeqSBAIJ(): Matrix size: 112413 X 112413, block > size 1; storage space: 8404901 unneeded, 2836399 used > [3] MatAssemblyEnd_SeqSBAIJ(): Number of mallocs during MatSetValues > is 0 > [3] MatAssemblyEnd_SeqSBAIJ(): Most nonzeros blocks in any row is 65 > [1] MatAssemblyEnd_SeqSBAIJ(): Matrix size: 113299 X 113299, block > size 1; storage space: 8647559 unneeded, 2682341 used > [1] MatAssemblyEnd_SeqSBAIJ(): Number of mallocs during MatSetValues > is 0 > [1] MatAssemblyEnd_SeqSBAIJ(): Most nonzeros blocks in any row is 62 > [0] MatAssemblyEnd_SeqSBAIJ(): Matrix size: 112871 X 112871, block > size 1; storage space: 8670798 unneeded, 2616302 used > [0] MatAssemblyEnd_SeqSBAIJ(): Number of mallocs during MatSetValues > is 0 > [0] MatAssemblyEnd_SeqSBAIJ(): Most nonzeros blocks in any row is 52 > [2] MatAssemblyEnd_SeqSBAIJ(): Matrix size: 108329 X 108329, block > size 1; storage space: 8247737 unneeded, 2585163 used > [2] MatAssemblyEnd_SeqSBAIJ(): Number of mallocs during MatSetValues > is 0 > [2] MatAssemblyEnd_SeqSBAIJ(): Most nonzeros blocks in any row is 52 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483645 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483644 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483643 > [1] PetscCommDuplicate(): returning tag 2147483639 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483645 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483644 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483643 > [3] PetscCommDuplicate(): returning tag 2147483639 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483645 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483644 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483643 > [2] PetscCommDuplicate(): returning tag 2147483639 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483645 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483644 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483643 > [0] PetscCommDuplicate(): returning tag 2147483639 > [0] PetscCommDuplicate(): returning tag 2147483634 > [2] PetscCommDuplicate(): returning tag 2147483634 > [1] PetscCommDuplicate(): returning tag 2147483634 > [3] PetscCommDuplicate(): returning tag 2147483634 > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > [0] VecScatterCreate(): General case: MPI to Seq > [0] PetscCommDuplicate(): returning tag 2147483630 > [0] PetscCommDuplicate(): returning tag 2147483625 > [1] PetscCommDuplicate(): returning tag 2147483630 > [1] PetscCommDuplicate(): returning tag 2147483625 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483642 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483641 > [1] PetscCommDuplicate(): returning tag 2147483620 > [2] PetscCommDuplicate(): returning tag 2147483630 > [2] PetscCommDuplicate(): returning tag 2147483625 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483642 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483641 > [2] PetscCommDuplicate(): returning tag 2147483620 > [3] PetscCommDuplicate(): returning tag 2147483630 > [3] PetscCommDuplicate(): returning tag 2147483625 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483642 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483641 > [3] PetscCommDuplicate(): returning tag 2147483620 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483640 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483639 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483638 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483642 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483641 > [0] PetscCommDuplicate(): returning tag 2147483620 > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > [0] VecScatterCreate(): General case: MPI to MPI > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483640 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483639 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483638 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483640 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483639 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483638 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483640 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483639 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483638 > [3] MatAssemblyEnd_SeqBAIJ(): Matrix size: 112413 X 0, block size 1; > storage space: 11241300 unneeded, 0 used > [3] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues > is 0 > [3] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 0 > [3] Mat_CheckCompressedRow(): Found the ratio (num_zerorows 112413)/ > (num_localrows 112413) > 0.6. Use CompressedRow routines. > [1] MatAssemblyEnd_SeqBAIJ(): Matrix size: 113299 X 2952, block size > 1; storage space: 11286236 unneeded, 43664 used > [1] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues > is 0 > [1] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 29 > [1] Mat_CheckCompressedRow(): Found the ratio (num_zerorows 110902)/ > (num_localrows 113299) > 0.6. Use CompressedRow routines. > [2] MatAssemblyEnd_SeqBAIJ(): Matrix size: 108329 X 9258, block size > 1; storage space: 10624060 unneeded, 208840 used > [2] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues > is 0 > [2] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 43 > [2] Mat_CheckCompressedRow(): Found the ratio (num_zerorows 92865)/ > (num_localrows 108329) > 0.6. Use CompressedRow routines. > [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 112871 X 17849, block > size 1; storage space: 10944569 unneeded, 342531 used > [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues > is 0 > [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 43 > [0] Mat_CheckCompressedRow(): Found the ratio (num_zerorows 92260)/ > (num_localrows 112871) > 0.6. Use CompressedRow routines. > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [0] PetscCommDuplicate(): returning tag 2147483614 > [0] PetscCommDuplicate(): returning tag 2147483613 > [0]before KSPsetup > [0] PCSetUp(): Setting up new PC > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483637 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483636 > [0] PetscCommDuplicate(): returning tag 2147483612 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): returning tag 2147483614 > [1] PetscCommDuplicate(): returning tag 2147483613 > [1]before KSPsetup > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483637 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483636 > [1] PetscCommDuplicate(): returning tag 2147483612 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [2] PetscCommDuplicate(): returning tag 2147483614 > [2] PetscCommDuplicate(): returning tag 2147483613 > [2]before KSPsetup > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483637 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483636 > [2] PetscCommDuplicate(): returning tag 2147483612 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483635 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483634 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483635 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483634 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [3] PetscCommDuplicate(): returning tag 2147483614 > [3] PetscCommDuplicate(): returning tag 2147483613 > [3]before KSPsetup > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483637 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483636 > [3] PetscCommDuplicate(): returning tag 2147483612 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483635 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483634 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483635 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483634 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483633 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483632 > [1] PetscCommDuplicate(): returning tag 2147483607 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483633 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483632 > [2] PetscCommDuplicate(): returning tag 2147483607 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483633 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483632 > [0] PetscCommDuplicate(): returning tag 2147483607 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483633 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483632 > [3] PetscCommDuplicate(): returning tag 2147483607 > [0] PetscCommDuplicate(): returning tag 2147483602 > [1] PetscCommDuplicate(): returning tag 2147483602 > [2] PetscCommDuplicate(): returning tag 2147483602 > [3] PetscCommDuplicate(): returning tag 2147483602 > [0] VecScatterCreate(): Special case: processor zero gets entire > parallel vector, rest get none > > DMUMPS 4.8.4 > L D L^T Solver for symmetric positive definite matrices > Type of parallelism: Working host > > ****** ANALYSIS STEP ******** > > Density: NBdense, Average, Median = 0 49 50 > Ordering based on METIS > ** Peak of sequential stack size (number of real entries) : > 16763905. > A root of estimated size 2965 has been selected for > Scalapack. > > Leaving analysis phase with ... > INFOG(1) = 0 > INFOG(2) = 0 > -- (20) Number of entries in factors (estim.) = 120657071 > -- (3) Storage of factors (REAL, estimated) = 137362626 > -- (4) Storage of factors (INT , estimated) = 6167135 > -- (5) Maximum frontal size (estimated) = 3705 > -- (6) Number of nodes in the tree = 21863 > -- (7) Ordering option effectively used = 5 > ICNTL(6) Maximum transversal option = 0 > ICNTL(7) Pivot order option = 7 > Percentage of memory relaxation (effective) = 200 > Number of level 2 nodes = 5 > Number of split nodes = 0 > RINFO(1) Operations during elimination (estim) = 1.114D+11 > Distributed matrix entry format (ICNTL(18)) = 3 > ** Rank of proc needing largest memory in IC facto : 0 > ** Estimated corresponding MBYTES for IC facto : 1112 > ** Estimated avg. MBYTES per work. proc at facto (IC) : 1083 > ** TOTAL space in MBYTES for IC factorization : 4333 > ** Rank of proc needing largest memory for OOC facto : 1 > ** Estimated corresponding MBYTES for OOC facto : 465 > ** Estimated avg. MBYTES per work. proc at facto (OOC) : 421 > ** TOTAL space in MBYTES for OOC factorization : 1684 > > ****** FACTORIZATION STEP ******** > > > GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ... > NUMBER OF WORKING PROCESSES = 4 > REAL SPACE FOR FACTORS = 137362626 > INTEGER SPACE FOR FACTORS = 6167135 > MAXIMUM FRONTAL SIZE (ESTIMATED) = 3705 > NUMBER OF NODES IN THE TREE = 21863 > Maximum effective relaxed size of S = 125678965 > Average effective relaxed size of S = 122625031 > > REDISTRIB: TOTAL DATA LOCAL/SENT = 2497290 8939630 > GLOBAL TIME FOR MATRIX DISTRIBUTION = 0.6324 > ** Memory relaxation parameter ( ICNTL(14) ) : 200 > ** Rank of processor needing largest memory in facto : 0 > ** Space in MBYTES used by this processor for facto : 1112 > ** Avg. Space in MBYTES per working proc during facto : 1083 > From bsmith at mcs.anl.gov Mon Oct 12 17:30:16 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 12 Oct 2009 17:30:16 -0500 Subject: SBAIJ issue In-Reply-To: <4AD32908.2050702@student.uibk.ac.at> References: <4AD32908.2050702@student.uibk.ac.at> Message-ID: Perhaps you have done enough memory preallocation for the multiprocessors and it is going very--very slowly with memory allocations? Barry On Oct 12, 2009, at 8:03 AM, Andreas Grassl wrote: > Hello, > > I'm trying to work with MUMPS-Cholesky as direct solver and CG as > iterative > solver and have problems with SBAIJ-Matrices. > > I have a MatSetValues-line not regarding the symmetry but force it > by the > command line option -mat_ignore_lower_triangular. > > The problems show up at "bigger" problems (~450000 DOF) with "more" > processes (1 > and 2 work fine, 4 gives a hangup) and/or trying to save the matrix. > The machine > has 4 cores and 12 GB memory. NNZ per row is ~80. > > Attached you find the output of the program run with -info. > > Any hints where to search? > > Cheers, > > ando > > -- > /"\ Grassl Andreas > \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik > X against HTML email Technikerstr. 13 Zi 709 > / \ +43 (0)512 507 6091 > same call as 2procinfo without -ksp_info_binary > > mpiexec -np 2 externalsolver > [0] PetscInitialize(): PETSc successfully started: number of > processors = 2 > [1] PetscInitialize(): PETSc successfully started: number of > processors = 2 > [1] PetscInitialize(): Running on machine: mat1.uibk.ac.at > [0] PetscInitialize(): Running on machine: mat1.uibk.ac.at > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > [0] PetscCommDuplicate(): returning tag 2147483647 > [1] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > [1] PetscCommDuplicate(): returning tag 2147483647 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): returning tag 2147483646 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [0] PetscCommDuplicate(): returning tag 2147483646 > [0] MatSetUpPreallocation(): Warning not preallocating matrix storage > [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374783 max tags = 2147483647 > [0] PetscCommDuplicate(): returning tag 2147483647 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374783 max tags = 2147483647 > [1] PetscCommDuplicate(): returning tag 2147483647 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483646 > [0] PetscCommDuplicate(): returning tag 2147483646 > [0] MatSetOption_MPISBAIJ(): Option SYMMETRIC ignored > [0] MatStashScatterBegin_Private(): No of messages: 1 > [0] MatStashScatterBegin_Private(): Mesg_to: 1: size: 30544064 > [0] MatStashScatterBegin_Private(): No of messages: 0 > [0] MatAssemblyBegin_MPISBAIJ(): Stash has 3818007 entries,uses 8 > mallocs. > [0] MatAssemblyBegin_MPISBAIJ(): Block-Stash has 3818007 entries, > uses 8 mallocs. > [0] MatAssemblyEnd_SeqSBAIJ(): Matrix size: 221545 X 221545, block > size 1; storage space: 16676676 unneeded, 5477824 used > [0] MatAssemblyEnd_SeqSBAIJ(): Number of mallocs during MatSetValues > is 0 > [0] MatAssemblyEnd_SeqSBAIJ(): Most nonzeros blocks in any row is 62 > [1] MatAssemblyEnd_SeqSBAIJ(): Matrix size: 225367 X 225367, block > size 1; storage space: 16793278 unneeded, 5743422 used > [1] MatAssemblyEnd_SeqSBAIJ(): Number of mallocs during MatSetValues > is 0 > [1] MatAssemblyEnd_SeqSBAIJ(): Most nonzeros blocks in any row is 65 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483645 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483644 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483643 > [1] PetscCommDuplicate(): returning tag 2147483639 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483645 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483644 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483643 > [0] PetscCommDuplicate(): returning tag 2147483639 > [0] PetscCommDuplicate(): returning tag 2147483634 > [1] PetscCommDuplicate(): returning tag 2147483634 > [1] PetscCommDuplicate(): returning tag 2147483630 > [1] PetscCommDuplicate(): returning tag 2147483625 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483642 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483641 > [1] PetscCommDuplicate(): returning tag 2147483620 > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > [0] VecScatterCreate(): General case: MPI to Seq > [0] PetscCommDuplicate(): returning tag 2147483630 > [0] PetscCommDuplicate(): returning tag 2147483625 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483642 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483641 > [0] PetscCommDuplicate(): returning tag 2147483620 > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > [0] VecScatterCreate(): General case: MPI to MPI > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483640 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483639 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDupli[1] PetscCommDuplicate(): Using internal PETSc > communicator 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483640 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483639 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483638 > [1] MatAssemblyEnd_SeqBAIJ(): Matrix size: 225367 X 0, block size 1; > storage space: 22536700 unneeded, 0 used > [1] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues > is 0 > [1] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 0 > [1] Mat_CheckCompressedRow(): Found the ratio (num_zerorows 225367)/ > (num_localrows 225367) > 0.6. Use CompressedRow routines. > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): returning tag 2147483614 > [1] PetscCommDuplicate(): returning tag 2147483613 > [1]before KSPsetup > [1] PetscCommDuplicate(): Using cate(): returning tag 2147483638 > [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 221545 X 6275, block size > 1; storage space: 22060506 unneeded, 93994 used > [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues > is 0 > [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 32 > [0] Mat_CheckCompressedRow(): Found the ratio (num_zerorows 216534)/ > (num_localrows 221545) > 0.6. Use CompressedRow routines. > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [0] PetscCommDuplicate(): returning tag 2147483614 > [0] PetscCommDuplicate(): returning tag 2147483613 > [0]before KSPsetup > [0] PCSetUp(): Setting up new PC > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483637 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483636 > [0] PetscCommDuplicate(): returning tag 2147483612 > [0] PetscCommDuplicate(): Using internainternal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483637 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483636 > [1] PetscCommDuplicate(): returning tag 2147483612 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483635 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483634 > l PETSc communicator 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483635 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483634 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483633 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483632 > [1] PetscCommDuplicate(): returning tag 2147483607 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483633 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483632 > [0] PetscCommDuplicate(): returning tag 2147483607 > [0] PetscCommDuplicate(): returning tag 2147483602 > [0] VecScatterCreate(): Special case: processor zero gets entire > parallel vector, rest get none > [1] PetscCommDuplicate(): returning tag 2147483602 > > DMUMPS 4.8.4 > L D L^T Solver for symmetric positive definite matrices > Type of parallelism: Working host > > ****** ANALYSIS STEP ******** > > Density: NBdense, Average, Median = 0 49 50 > Ordering based on METIS > ** Peak of sequential stack size (number of real entries) : > 16763905. > A root of estimated size 2965 has been selected for > Scalapack. > > Leaving analysis phase with ... > INFOG(1) = 0 > INFOG(2) = 0 > -- (20) Number of entries in factors (estim.) = 120657071 > -- (3) Storage of factors (REAL, estimated) = 137364268 > -- (4) Storage of factors (INT , estimated) = 6138804 > -- (5) Maximum frontal size (estimated) = 3705 > -- (6) Number of nodes in the tree = 21863 > -- (7) Ordering option effectively used = 5 > ICNTL(6) Maximum transversal option = 0 > ICNTL(7) Pivot order option = 7 > Percentage of memory relaxation (effective) = 200 > Number of level 2 nodes = 1 > Number of split nodes = 0 > RINFO(1) Operations during elimination (estim) = 1.114D+11 > Distributed matrix entry format (ICNTL(18)) = 3 > ** Rank of proc needing largest memory in IC facto : 0 > ** Estimated corresponding MBYTES for IC facto : 2085 > ** Estimated avg. MBYTES per work. proc at facto (IC) : 2044 > ** TOTAL space in MBYTES for IC factorization : 4088 > ** Rank of proc needing largest memory for OOC facto : 1 > ** Estimated corresponding MBYTES for OOC facto : 644 > ** Estimated avg. MBYTES per work. proc at facto (OOC) : 630 > ** TOTAL space in MBYTES for OOC factorization : 1261 > > ****** FACTORIZATION STEP ******** > > > GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ... > NUMBER OF WORKING PROCESSES = 2 > REAL SPACE FOR FACTORS = 137364268 > INTEGER SPACE FOR FACTORS = 6138804 > MAXIMUM FRONTAL SIZE (ESTIMATED) = 3705 > NUMBER OF NODES IN THE TREE = 21863 > Maximum effective relaxed size of S = 239060000 > Average effective relaxed size of S = 234725641 > > REDISTRIB: TOTAL DATA LOCAL/SENT = 9086109 2238491 > GLOBAL TIME FOR MATRIX DISTRIBUTION = 0.7322 > ** Memory relaxation parameter ( ICNTL(14) ) : 200 > ** Rank of processor needing largest memory in facto : 0 > ** Space in MBYTES used by this processor for facto : 2085 > ** Avg. Space in MBYTES per working proc during facto : 2044 > > ELAPSED TIME FOR FACTORIZATION = 28.1724 > Maximum effective space used in S (KEEP(67) = 77516871 > Average effective space used in S (KEEP(67) = 75395654 > ** EFF Min: Rank of processor needing largest memory : 0 > ** EFF Min: Space in MBYTES used by this processor : 741 > ** EFF Min: Avg. Space in MBYTES per working proc : 720 > > GLOBAL STATISTICS > RINFOG(2) OPERATIONS DURING NODE ASSEMBLY = 2.565D+08 > ------(3) OPERATIONS DURING NODE ELIMINATION = 1.114D+11 > INFOG (9) REAL SPACE FOR FACTORS = 137364268 > INFOG(10) INTEGER SPACE FOR FACTORS = 6138809 > INFOG(11) MAXIMUM FRONT SIZE = 3705 > INFOG(29) NUMBER OF ENTRIES IN FACTORS = 116259976 > INFOG(12) NB OF NEGATIVE PIVOTS = 0 > INFOG(14) NUMBER OF MEMORY COMPRESS = 0 > [0]after KSPsetup > [0]RHS setup > [1]after KSPsetup > [1]RHS setup > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): returning tag 2147483601 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [0] PetscCommDuplicate(): returning tag 2147483601 > [0] PetscCommDuplicate(): returning tag 2147483596 > [1] PetscCommDuplicate(): returning tag 2147483596 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [0] PetscCommDuplicate(): returning tag 2147483591 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): returning tag 2147483591 > [0] VecAssemblyBegin_MPI(): Stash has 225367 entries, uses 12 mallocs. > [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 mallocs. > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483631 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483631 > > > ****** SOLVE & CHECK STEP ******** > > > STATISTICS PRIOR SOLVE PHASE ........... > NUMBER OF RIGHT-HAND-SIDES = 1 > BLOCKING FACTOR FOR MULTIPLE RHS = 1 > ICNTL (9) = 1 > --- (10) = 0 > --- (11) = 0 > --- (20) = 0 > --- (21) = 1 > ** Rank of processor needing largest memory in solve : 0 > ** Space in MBYTES used by this processor for solve : 2013 > ** Avg. Space in MBYTES per working proc during solve : 1975 > > > LEAVING SOLVER WITH: INFOG(1) ............ = 0 > INFOG(2) ............ = 0 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483630 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483629 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483630 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483629 > [0] PetscCommDuplicate(): returning tag 2147483590 > [1] PetscCommDuplicate(): returning tag 2147483590 > [1] VecScatterCreate(): General case: Seq to MPI > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > [0] VecScatterCreate(): General case: Seq to MPI > [0] PetscCommDuplicate(): returning tag 2147483586 > KSP Object: > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > PC Object: > type: cholesky > Cholesky: out-of-place factorization > matrix ordering: natural > Cholesky: factor fill ratio needed 0 > Factored matrix follows > Matrix Object: > type=mpisbaij, rows=446912, cols=446912 > package used to perform factorization: mumps > [1] PetscCommDuplicate(): returning tag 2147483586 > [1] PetscCommDuplicate(): returning tag 2147483585 > [1] PetscCommDuplicate(): returning tag 2147483584 > [1] PetscCommDuplicate(): returning tag 2147483583 > [1] PetscCommDuplicate(): returning tag 2147483582 > [1] PetscCommDuplicate(): returning tag 2147483581 > [1] PetscCommDuplicate(): returning tag 2147483580 > total: nonzeros=0, allocated nonzeros=893824 > MUMPS run parameters: > SYM (matrix type): 1 > PAR (host participation): 1 > ICNTL(1) (output for error): 6 > ICNTL(2) (output of diagnostic msg):0 > ICNTL(3) (output for global info): 6 > ICNTL(4) (level of printing): -1 > ICNTL(5) (input mat struct): 0 > ICNTL(6) (matrix prescaling): 0 > ICNTL(7) (matrix ordering): 7 > ICNTL(8) (scalling strategy): 77 > ICNTL(9) (A/A^T x=b is solved): 1 > ICNTL(10) (max num of refinements): 0 > ICNTL(11) (error analysis): 0 > ICNTL(12) (efficiency control): 1 > ICNTL(13) (efficiency control): 0 > ICNTL(14) (percentage of estimated workspace increase): > 200 > ICNTL(18)[1] PetscCommDuplicate(): returning tag > 2147483579 > (input mat struct): 3 > ICNTL(19) (Shur complement info): 0 > ICNTL(20) (rhs sparse pattern): 0 > ICNTL(21) (solution struct): 1 > ICNTL(22) (in-core/out-of-core facility): 0 > ICNTL(23) (max size of memory can be allocated locally):0 > ICNTL(24) (detection of null pivot rows): 0 > ICNTL(25) (computation of a null space basis): 0 > ICNTL(26) (Schur options for rhs or solution): 0 > ICNTL(27) (experimental parameter): > -8 > CNTL(1) (relative pivoting threshold): 0 > CNTL(2) (stopping criterion of refinement): 1.49012e-08 > CNTL(3) (absolute pivoting threshold): 0 > CNTL(4) (value of static pivoting): -1 > CNTL(5) (fixation for null pivots): 0 > RINFO(1) (local estimated flops for the elimination after > analysis): > [0] 5.16132e+10 > [0] PetscCommDuplicate(): returning tag 2147483585 > [1] 5.97957e+10 > RINFO(2) (local estimated flops for the assembly after > factorization): > [0] 1.33366e+08 > [0] PetscCommDuplicate(): returning tag 2147483584 > [1] 1.23105e+08 > RINFO(3) (local estimated flops for the elimination after > factorization): > [0] 5.16124e+10 > [0] PetscCommDuplicate(): returning tag 2147483583 > [1] 5.9795e+10 > INFO(15) (estimated size of (in MB) MUMPS internal data for > running numerical factorization): > [0] 2085 > [0] PetscCommDuplicate(): returning tag 2147483582 > [1] 2003 > INFO(16) (size of (in MB) MUMPS internal data used during > numerical factorization): > [0] 2085 > [0] PetscCommDuplicate(): returning tag 2147483581 > [1] 2003 > INFO(23) (num of pivots eliminated on this processor after > factorization): > [0] 236610 > [0] PetscCommDuplicate(): returning tag 2147483580 > [1] 210302 > RINFOG(1) (global estimated flops for the elimination > after analysis): 1.11409e+11 > RINFOG(2) (global estimated flops for the assembly > after factorization): 2.56471e+08 > RINFOG(3) (global estimated flops for the elimination > after factorization): 1.11407e+11 > INFOG(3) (estimated real workspace for factors on all > processors after analysis): 137364268 > INFOG(4) (estimated integer workspace for factors on > all processors after analysis): 6138804 > INFOG(5) (estimated maximum front size in the complete > tree): 3705 > INFOG(6) (number of nodes in the complete tree): 21863 > INFOG(7) (ordering option effectively uese after > analysis): 5 > INFOG(8) (structural symmetry in percent of the > permuted matrix after analysis): 100 > INFOG(9) (total real/complex workspace to store the > matrix factors after factorization): 137364268 > INFOG(10) (total integer space store the matrix factors > after factorization): 6138809 > INFOG(11) (order of largest frontal matrix after > factorization): 3705 > INFOG(12) (number of off-diagonal pivots): 0 > INFOG(13) (number of delayed pivots after > factorization): 0 > INFOG(14) (number of memory compress after > factorization): 0 > INFOG(15) (number of steps of iterative refinement > after solution): 0 > INFOG(16) (estimated size (in MB) of all MUMPS internal > data for factorization after analysis: value on the most memory > consuming processor): 2085 > INFOG(17) (estimated size of all MUMPS internal data > for factorization after analysis: sum over all processors): 4088 > INFOG(18) (size of all MUMPS internal data allocated > during factorization: value on the most memory consuming processor): > 2085 > INFOG(19) (size of all MUMPS internal data allocated > during factorization: sum over all processors): 4088 > INFOG(20) (estimated number of entries in the factors): > 120657071 > INFOG(21) (size in MB of memory effectively used during > factorization - value on the most memory consuming processor): 741 > INFOG(22) (size in MB of memory effectively used during > factorization - sum over all processors): 1441 > INFOG(23) (after analysis: value of ICNTL(6) > effectively used): 0 > INFOG(24) (after analysis: value of ICNTL(12) > effectively used): 1 > INFOG(25) (after factorization: number of pivots > modified by static pivoting): 0 > linear system matrix = precond matrix: > Matrix Object: > type=mpisbaij, rows=446912, cols=446912 > total: nonzeros=11315240, allocated nonzeros=89382400 > block size is 1 > [0] PetscCommDuplicate(): returning tag 2147483579 > [1] PetscCommDuplicate(): returning tag 2147483578 > [1] PetscCommDuplicate(): returning tag 2147483577 > [1] Petsc_DelViewer(): Deleting viewer data in an MPI_Comm -2080374784 > [0] PetscCommDuplicate(): returning tag 2147483578 > [0] PetscCommDuplicate(): returning tag 2147483577 > [0] Petsc_DelViewer(): Deleting viewer data in an MPI_Comm -2080374784 > [0] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm 1140850689 > [0] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374783 > [0] Petsc_DelTag(): Deleting tag data in an MPI_Comm -2080374783 > [0] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483576 > [1] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm 1140850689 > [1] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374783 > [1] Petsc_DelTag(): Deleting tag data in an MPI_Comm -2080374783 > [0] Petsc_DelViewer(): Deleting viewer data in an MPI_Comm -2080374784 > [0] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm 1140850688 > [0] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374784 > [0] Petsc_DelTag(): Deleting tag data in an MPI_Comm -2080374784 > [0] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm -2080374784 > [0] Petsc_DelViewer(): Deleting viewer data in an MPI_Comm -2080374784 > ************************************************************************************************************************ > *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript - > r -fCourier9' to print this document *** > ************************************************************************************************************************ > > ---------------------------------------------- PETSc Performance > Summary: ---------------------------------------------- > > [1] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483576 > [1] Petsc_DelViewer(): Deleting viewer data in an MPI_Comm -2080374784 > [1] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm 1140850688 > [1] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374784 > [1] Petsc_DelTag(): Deleting tag data in an MPI_Comm -2080374784 > [1] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm -2080374784 > [1] Petsc_DelViewer(): Deleting viewer data in an MPI_Comm -2080374784 > externalsolver on a linux32-i named mat1.uibk.ac.at with 2 > processors, by csae1801 Mon Oct 12 12:08:38 2009 > Using Petsc Release Version 3.0.0, Patch 8, Fri Aug 21 14:02:12 CDT > 2009 > > Max Max/Min Avg Total > Time (sec): 3.989e+01 1.00000 3.989e+01 > Objects: 3.600e+01 1.00000 3.600e+01 > Flops: 0.000e+00 0.00000 0.000e+00 0.000e+00 > Flops/sec: 0.000e+00 0.00000 0.000e+00 0.000e+00 > MPI Messages: 2.550e+01 1.08511 2.450e+01 4.900e+01 > MPI Message Lengths: 6.596e+07 1.00053 2.692e+06 1.319e+08 > MPI Reductions: 5.800e+01 1.00000 > > Flop counting convention: 1 flop = 1 real number operation of type > (multiply/divide/add/subtract) > e.g., VecAXPY() for real vectors of > length N --> 2N flops > and VecAXPY() for complex vectors of > length N --> 8N flops > > Summary of Stages: ----- Time ------ ----- Flops ----- --- > Messages --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total counts > %Total Avg %Total counts %Total > 0: Main Stage: 2.8402e-01 0.7% 0.0000e+00 0.0% 1.000e > +00 2.0% 8.163e-02 0.0% 0.000e+00 0.0% > 1: decomposition: 4.7451e-01 1.2% 0.0000e+00 0.0% 0.000e > +00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% > 2: Matrix Create: 7.2766e-01 1.8% 0.0000e+00 0.0% 0.000e > +00 0.0% 0.000e+00 0.0% 2.000e+00 3.4% > 3: Matrix fill: 2.5720e+00 6.4% 0.0000e+00 0.0% 1.600e+01 > 32.7% 2.486e+06 92.4% 1.500e+01 25.9% > 4: Solver setup: 3.2841e+01 82.3% 0.0000e+00 0.0% 0.000e > +00 0.0% 0.000e+00 0.0% 8.000e+00 13.8% > 5: RHS setup: 3.2125e-02 0.1% 0.0000e+00 0.0% 4.000e > +00 8.2% 1.094e+05 4.1% 4.000e+00 6.9% > 6: Solve: 4.9751e-01 1.2% 0.0000e+00 0.0% 2.500e+01 > 51.0% 5.944e+04 2.2% 6.000e+00 10.3% > 7: Postproc: 2.4653e+00 6.2% 0.0000e+00 0.0% 3.000e > +00 6.1% 3.679e+04 1.4% 0.000e+00 0.0% > > ------------------------------------------------------------------------------------------------------------------------ > See the 'Profiling' chapter of the users' manual for details on > interpreting output. > Phase summary info: > Count: number of times phase was executed > Time and Flops: Max - maximum over all processors > Ratio - ratio of maximum to minimum over all > processors > Mess: number of messages sent > Avg. len: average message length > Reduct: number of global reductions > Global: entire computation > Stage: stages of a computation. Set stages with > PetscLogStagePush() and PetscLogStagePop(). > %T - percent time in this phase %F - percent flops in > this phase > %M - percent messages in this phase %L - percent message > lengths in this phase > %R - percent reductions in this phase > Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max > time over all processors) > ------------------------------------------------------------------------------------------------------------------------ > Event Count Time (sec) > Flops --- Global --- --- Stage --- > Total > Max Ratio Max Ratio Max Ratio Mess Avg > len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > PetscBarrier 1 1.0 3.1269e-0328.4 0.00e+00 0.0 0.0e+00 0.0e > +00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 > > --- Event Stage 1: decomposition > > > --- Event Stage 2: Matrix Create > > > --- Event Stage 3: Matrix fill > > MatAssemblyBegin 1 1.0 3.3235e-01 1.9 0.00e+00 0.0 6.0e+00 2.0e > +07 3.0e+00 1 0 12 92 5 10 0 38100 20 0 > MatAssemblyEnd 1 1.0 1.0536e+00 1.0 0.00e+00 0.0 1.0e+01 1.8e > +04 1.2e+01 3 0 20 0 21 41 0 62 0 80 0 > > --- Event Stage 4: Solver setup > > MatCholFctrSym 1 1.0 4.0531e-06 2.1 0.00e+00 0.0 0.0e+00 0.0e > +00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatCholFctrNum 1 1.0 3.2830e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e > +00 5.0e+00 82 0 0 0 9 100 0 0 0 62 0 > KSPSetup 1 1.0 6.9141e-06 1.4 0.00e+00 0.0 0.0e+00 0.0e > +00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > PCSetUp 1 1.0 3.2841e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e > +00 8.0e+00 82 0 0 0 14 100 0 0 0100 0 > > --- Event Stage 5: RHS setup > > VecAssemblyBegin 1 1.0 6.2160e-03 1.6 0.00e+00 0.0 4.0e+00 1.3e > +06 3.0e+00 0 0 8 4 5 16 0100100 75 0 > VecAssemblyEnd 1 1.0 8.4589e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e > +00 0.0e+00 0 0 0 0 0 26 0 0 0 0 0 > > --- Event Stage 6: Solve > > MatSolve 1 1.0 4.9525e-01 1.0 0.00e+00 0.0 6.0e+00 4.8e > +05 4.0e+00 1 0 12 2 7 100 0 24 98 67 0 > MatView 2 1.0 1.4269e-03 1.0 0.00e+00 0.0 1.8e+01 2.7e > +03 2.0e+00 0 0 37 0 3 0 0 72 2 33 0 > VecSet 1 1.0 8.1515e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e > +00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecScatterBegin 2 1.0 6.7840e-03 1.3 0.00e+00 0.0 2.0e+00 1.3e > +06 0.0e+00 0 0 4 2 0 1 0 8 86 0 0 > VecScatterEnd 1 1.0 2.1610e-03 1.7 0.00e+00 0.0 0.0e+00 0.0e > +00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > KSPSolve 1 1.0 4.9603e-01 1.0 0.00e+00 0.0 6.0e+00 4.8e > +05 4.0e+00 1 0 12 2 7 100 0 24 98 67 0 > PCApply 1 1.0 4.9526e-01 1.0 0.00e+00 0.0 6.0e+00 4.8e > +05 4.0e+00 1 0 12 2 7 100 0 24 98 67 0 > > --- Event Stage 7: Postproc > > VecView 1 1.0 2.1809e+00 2.0 0.00e+00 0.0 2.0e+00 9.0e > +05 0.0e+00 4 0 4 1 0 66 0 67100 0 0 > ------------------------------------------------------------------------------------------------------------------------ > > Memory usage is given in bytes: > > Object Type Creations Destructions Memory Descendants' > Mem. > > --- Event Stage 0: Main Stage > > IS L to G Mapping 0 1 889756 0 > Matrix 0 3 547690752 0 > Vec 0 6 3702888 0 > Vec Scatter 0 2 1736 0 > Viewer 0 1 544 0 > > --- Event Stage 1: decomposition > > > --- Event Stage 2: Matrix Create > > IS L to G Mapping 1 0 0 0 > Matrix 3 0 0 0 > > --- Event Stage 3: Matrix fill > > Index Set 4 4 152680 0 > Vec 7 1 1304 0 > Vec Scatter 2 0 0 0 > > --- Event Stage 4: Solver setup > > Index Set 3 2 1008 0 > Matrix 3 0 0 0 > Vec 2 1 1773664 0 > Vec Scatter 1 0 0 0 > Krylov Solver 1 0 0 0 > Preconditioner 1 0 0 0 > > --- Event Stage 5: RHS setup > > Vec 2 0 0 0 > Viewer 1 0 0 0 > > --- Event Stage 6: Solve > > Index Set 2 2 947456 0 > Vec 1 0 0 0 > Vec Scatter 1 0 0 0 > Viewer 1 0 0 0 > > --- Event Stage 7: Postproc > > Index Set 0 1 504 0 > Matrix 0 3 8870136 0 > Vec 0 4 7125104 0 > Vec Scatter 0 2 1320 0 > Krylov Solver 0 1 832 0 > Preconditioner 0 1 728 0 > Viewer 0 1 544 0 > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > =================================================================[1] > PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > [1] PetscCommDuplicate(): returning tag 2147483647 > [1] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm 1140850688 > [1] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374784 > [1] Petsc_DelTag(): Deleting tag data in an MPI_Comm -2080374784 > [1] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm -2080374784 > ======================================== > Average time to get PetscTime(): 1.3113e-06 > Average time for MPI_Barrier(): 0.000144005 > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > [0] PetscCommDuplicate(): returning tag 2147483647 > Average time for zero size MPI_Send(): 6.54459e-05 > [0] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm 1140850688 > [0] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374784 > [0] Petsc_DelTag(): Deleting tag data in an MPI_Comm -2080374784 > [0] Petsc_DelComm(): Deleting PETSc communicator imbedded in a user > MPI_Comm -2080374784 > #PETSc Option Table entries: > -dump_data false > -info > -ksp_compute_singularvalues > -ksp_rtol 1e-6 > -ksp_type preonly > -ksp_view > -log_summary > -mat_ignore_lower_triangular > -mat_mumps_icntl_14 200 > -mat_mumps_sym 1 > -mat_type sbaij > -monitorname errormonitor.dat > -nz 100 > -options_left > -pc_factor_mat_solver_package mumps > -pc_type cholesky > -preload false > -verb false > #End o PETSc Option Table entries > Compiled without FORTRAN kernels > Compiled with full precision matrices (default) > sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 > sizeof(PetscScalar) 8 > Configure run at: Wed Oct 7 18:16:40 2009 > Configure options: --with-scalar-type=real --with-debugging=0 --with- > precision=double --with-cc=icc --with-fc=ifort --with-cxx=icpc -- > with-shared=0 --with-mpi=1 --with-external-packages-dir=/home/lux/ > csae1801/petsc/externalpackages --download-mpich=ifneeded --with- > scalapack=1 --download-scalapack=ifneeded --download-f-blas- > lapack=ifneeded --with-blacs=1 --download-blacs=ifneeded --with- > parmetis=1 --download-parmetis=ifneeded --with-mumps=1 --download- > mumps=ifneeded --with-hypre=1 --with-hypre-dir=/home/lux/csae1801/ > petsc/externalpackages/hypre-2.4.0b/src/hypre --with-spooles=1 -- > download-spooles=ifneeded --with-superlu_dist=1 --download- > superlu_dist=ifneeded PETSC_ARCH=linux32-intel-c-prod > ----------------------------------------- > Libraries compiled on Mi 7. Okt 18:17:41 CEST 2009 on mat1.uibk.ac.at > Machine characteristics: Linux mat1.uibk.ac.at 2.6.18-53.el5xen #1 > SMP Wed Oct 10 16:48:44 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux > Using PETSc directory: /home/lux/csae1801/petsc/petsc-3.0.0-p8 > Using PETSc arch: linux32-intel-c-prod > ----------------------------------------- > Using C compiler: /home/lux/csae1801/petsc/petsc-3.0.0-p8/linux32- > intel-c-prod/bin/mpicc -O > Using Fortran compiler: /home/lux/csae1801/petsc/petsc-3.0.0-p8/ > linux32-intel-c-prod/bin/mpif90 -O > ----------------------------------------- > Using include paths: -I/home/lux/csae1801/petsc/petsc-3.0.0-p8/ > linux32-intel-c-prod/include -I/home/lux/csae1801/petsc/petsc-3.0.0- > p8/include -I/home/lux/csae1801/petsc/petsc-3.0.0-p8/linux32-intel-c- > prod/include -I/home/lux/csae1801/petsc/externalpackages/ > hypre-2.4.0b/src/hypre/include > ------------------------------------------ > Using C linker: /home/lux/csae1801/petsc/petsc-3.0.0-p8/linux32- > intel-c-prod/bin/mpicc -O > Using Fortran linker: /home/lux/csae1801/petsc/petsc-3.0.0-p8/ > linux32-intel-c-prod/bin/mpif90 -O > Using libraries: -Wl,-rpath,/home/lux/csae1801/petsc/petsc-3.0.0-p8/ > linux32-intel-c-prod/lib -L/home/lux/csae1801/petsc/petsc-3.0.0-p8/ > linux32-intel-c-prod/lib -lpetscts -lpetscsnes -lpetscksp -lpetscdm - > lpetscmat -lpetscvec -lpetsc -lX11 -Wl,-rpath,/home/lux/ > csae1801/petsc/petsc-3.0.0-p8/linux32-intel-c-prod/lib -L/home/lux/ > csae1801/petsc/petsc-3.0.0-p8/linux32-intel-c-prod/lib - > lsuperlu_dist_2.3 -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common > -lpord -lparmetis -lmetis -lscalapack -lblacs -Wl,-rpath,/home/lux/ > csae1801/petsc/externalpackages/hypre-2.4.0b/src/hypre/lib -L/home/ > lux/csae1801/petsc/externalpackages/hypre-2.4.0b/src/hypre/lib - > lHYPRE -lmpichcxx -lstdc++ -lspooles -lflapack -lfblas -lnsl -laio - > lrt -lPEPCF90 -ldl -L/home/lux/csae1801/petsc/petsc-3.0.0-p8/linux32- > intel-c-prod/lib -lmpich -lpthread -lrt -L/opt/intel/cmkl/10.1.1.019/ > lib/em64t -L/opt/intel/Compiler/11.0/074/ipp/em64t/lib -L/opt/intel/ > Compiler/11.0/074/mkl/lib/em64t -L/opt/intel/Compiler/11.0/074/tbb/ > em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib -L/opt/intel/Compiler/ > 11.0/074/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -limf - > lsvml -lipgo -ldecimal -lirc -lgcc_s -lirc_s -lmpichf90 -lifport - > lifcore -lm -lm -lmpichcxx -lstdc++ -lmpichcxx -lstdc++ -ldl -lmpich > -lpthread -lrt -limf -lsvml -lipgo -ldecimal -lirc -lgcc_s -lirc_s - > ldl > ------------------------------------------ > #PETSc Option Table entries: > -dump_data false > -info > -ksp_compute_singularvalues > -ksp_rtol 1e-6 > -ksp_type preonly > -ksp_view > -log_summary > -mat_ignore_lower_triangular > -mat_mumps_icntl_14 200 > -mat_mumps_sym 1 > -mat_type sbaij > -monitorname errormonitor.dat > -nz 100 > -options_left > -pc_factor_mat_solver_package mumps > -pc_type cholesky > -preload false > -verb false > #End o PETSc Option Table entries > There are no unused options. > > 1olarr > > before sol put > sol put succes > before fullsol > /DIANA/AP/LS41 12:08:39 35.66-CPU 11.16-IO SOLVE > /DIANA/AP/LS41 12:08:39 35.66-CPU 11.18-IO FILLUP > /DIANA/AP/LS41 12:08:39 35.66-CPU 11.18-IO FILLUP > /DIANA/AP/LS41 12:08:39 35.66-CPU 11.18-IO POST > /DIANA/AP/LS41 12:08:41 36.79-CPU 12.27-IO POST > /DIANA/AP/LS41 12:08:41 36.79-CPU 12.27-IO LINSTA > /DIANA/DC/END 12:08:41 36.79-CPU 12.27-IO STOP > DIANA JOB 3446 finished > > 1 proc externalsolver stuck at 100% with 12%memory out of 12GB > 1 proc externalsolver at 0% > > mpiexec -np 2 externalsolver > [0] PetscInitialize(): PETSc successfully started: number of > processors = 2 > [0] PetscInitialize(): Running on machine: mat1.uibk.ac.at > [1] PetscInitialize(): PETSc successfully started: number of > processors = 2 > [1] PetscInitialize(): Running on machine: mat1.uibk.ac.at > [1] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > [1] PetscCommDuplicate(): returning tag 2147483647 > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > [0] PetscCommDuplicate(): returning tag 2147483647 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): returning tag 2147483646 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [0] PetscCommDuplicate(): returning tag 2147483646 > [0] MatSetUpPreallocation(): Warning not preallocating matrix storage > [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374783 max tags = 2147483647 > [1] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374783 max tags = 2147483647 > [1] PetscCommDuplicate(): returning tag 2147483647 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483646 > [0] PetscCommDuplicate(): returning tag 2147483647 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483646 > [0] MatSetOption_MPISBAIJ(): Option SYMMETRIC ignored > [0] MatStashScatterBegin_Private(): No of messages: 1 > [0] MatStashScatterBegin_Private(): Mesg_to: 1: size: 30544064 > [0] MatStashScatterBegin_Private(): No of messages: 0 > [0] MatAssemblyBegin_MPISBAIJ(): Stash has 3818007 entries,uses 8 > mallocs. > [0] MatAssemblyBegin_MPISBAIJ(): Block-Stash has 3818007 entries, > uses 8 mallocs. > [0] MatAssemblyEnd_SeqSBAIJ(): Matrix size: 221545 X 221545, block > size 1; storage space: 16676676 unneeded, 5477824 used > [0] MatAssemblyEnd_SeqSBAIJ(): Number of mallocs during MatSetValues > is 0 > [0] MatAssemblyEnd_SeqSBAIJ(): Most nonzeros blocks in any row is 62 > [1] MatAssemblyEnd_SeqSBAIJ(): Matrix size: 225367 X 225367, block > size 1; storage space: 16793278 unneeded, 5743422 used > [1] MatAssemblyEnd_SeqSBAIJ(): Number of mallocs during MatSetValues > is 0 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483645 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483644 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483643 > [0] PetscCommDuplicate(): returning tag 2147483639 > [0] PetscCommDuplicate(): returning tag 2147483634 > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > [0] VecScatterCreate(): General case: MPI to Seq > [0] PetscCommDuplicate(): returning tag 2147483630 > [0] PetscCommDuplicate(): returning tag 2147483625 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483642 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483641 > [0] PetscCommDuplicate[1] MatAssemblyEnd_SeqSBAIJ(): Most nonzeros > blocks in any row is 65 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483645 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483644 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483643 > [1] PetscCommDuplicate(): returning tag 2147483639 > [1] PetscCommDuplicate(): returning tag 2147483634 > [1] PetscCommDuplicate(): returning tag 2147483630 > [1] PetscCommDuplicate(): returning tag 2147483625 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483642 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483641 > [1] PetscCommDuplicate(): returning tag 2147483620 > [1] PetscC(): returning tag 2147483620 > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > [0] VecScatterCreate(): General case: MPI to MPI > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483640 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483639 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483638 > [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 221545 X 6275, block size > 1; storage space: 22060506 unneeded, 93994 used > [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues > is 0 > [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 32 > [0] Mat_CheckCompressedRow(): Found the ratio (num_zerorows 216534)/ > (num_localrows 221545) > 0.6. Use CompressedRow routines. > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [0] PetsommDuplicate(): Using internal PETSc communicator 1140850689 > -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483640 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483639 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483638 > [1] MatAssemblyEnd_SeqBAIJ(): Matrix size: 225367 X 0, block size 1; > storage space: 22536700 unneeded, 0 used > [1] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues > is 0 > [1] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 0 > [1] Mat_CheckCompressedRow(): Found the ratio (num_zerorows 225367)/ > (num_localrows 225367) > 0.6. Use CompressedRow routines. > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): returning tag 2147483614 > [1] PetscCommDuplicate(): returning tag 2147483613 > [1]before KSPsetup > [1] PetscCommDuplicate(): Using internal PcCommDuplicate(): > returning tag 2147483614 > [0] PetscCommDuplicate(): returning tag 2147483613 > [0]before KSPsetup > [0] PCSetUp(): Setting up new PC > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483637 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483636 > [0] PetscCommDuplicate(): returning tag 2147483612 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483635 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483634 > ETSc communicator 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483637 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483636 > [1] PetscCommDuplicate(): returning tag 2147483612 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483635 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483634 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483633 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483632 > [0] PetscCommDuplicate(): returning tag 2147483607 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483633 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483632 > [1] PetscCommDuplicate(): returning tag 2147483607 > [0] PetscCommDuplicate(): returning tag 2147483602 > [0] VecScatterCreate(): Special case: processor zero gets entire > parallel vector, rest get none > [1] PetscCommDuplicate(): returning tag 2147483602 > > DMUMPS 4.8.4 > L D L^T Solver for symmetric positive definite matrices > Type of parallelism: Working host > > ****** ANALYSIS STEP ******** > > Density: NBdense, Average, Median = 0 49 50 > Ordering based on METIS > ** Peak of sequential stack size (number of real entries) : > 16763905. > A root of estimated size 2965 has been selected for > Scalapack. > > Leaving analysis phase with ... > INFOG(1) = 0 > INFOG(2) = 0 > -- (20) Number of entries in factors (estim.) = 120657071 > -- (3) Storage of factors (REAL, estimated) = 137364268 > -- (4) Storage of factors (INT , estimated) = 6138804 > -- (5) Maximum frontal size (estimated) = 3705 > -- (6) Number of nodes in the tree = 21863 > -- (7) Ordering option effectively used = 5 > ICNTL(6) Maximum transversal option = 0 > ICNTL(7) Pivot order option = 7 > Percentage of memory relaxation (effective) = 200 > Number of level 2 nodes = 1 > Number of split nodes = 0 > RINFO(1) Operations during elimination (estim) = 1.114D+11 > Distributed matrix entry format (ICNTL(18)) = 3 > ** Rank of proc needing largest memory in IC facto : 0 > ** Estimated corresponding MBYTES for IC facto : 2085 > ** Estimated avg. MBYTES per work. proc at facto (IC) : 2044 > ** TOTAL space in MBYTES for IC factorization : 4088 > ** Rank of proc needing largest memory for OOC facto : 1 > ** Estimated corresponding MBYTES for OOC facto : 644 > ** Estimated avg. MBYTES per work. proc at facto (OOC) : 630 > ** TOTAL space in MBYTES for OOC factorization : 1261 > > ****** FACTORIZATION STEP ******** > > > GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ... > NUMBER OF WORKING PROCESSES = 2 > REAL SPACE FOR FACTORS = 137364268 > INTEGER SPACE FOR FACTORS = 6138804 > MAXIMUM FRONTAL SIZE (ESTIMATED) = 3705 > NUMBER OF NODES IN THE TREE = 21863 > Maximum effective relaxed size of S = 239060000 > Average effective relaxed size of S = 234725641 > > REDISTRIB: TOTAL DATA LOCAL/SENT = 9086109 2238491 > GLOBAL TIME FOR MATRIX DISTRIBUTION = 0.8155 > ** Memory relaxation parameter ( ICNTL(14) ) : 200 > ** Rank of processor needing largest memory in facto : 0 > ** Space in MBYTES used by this processor for facto : 2085 > ** Avg. Space in MBYTES per working proc during facto : 2044 > > ELAPSED TIME FOR FACTORIZATION = 28.2595 > Maximum effective space used in S (KEEP(67) = 77516871 > Average effective space used in S (KEEP(67) = 75395654 > ** EFF Min: Rank of processor needing largest memory : 0 > ** EFF Min: Space in MBYTES used by this processor : 741 > ** EFF Min: Avg. Space in MBYTES per working proc : 720 > > GLOBAL STATISTICS > RINFOG(2) OPERATIONS DURING NODE ASSEMBLY = 2.565D+08 > ------(3) OPERATIONS DURING NODE ELIMINATION = 1.114D+11 > INFOG (9) REAL SPACE FOR FACTORS = 137364268 > INFOG(10) INTEGER SPACE FOR FACTORS = 6138809 > INFOG(11) MAXIMUM FRONT SIZE = 3705 > INFOG(29) NUMBER OF ENTRIES IN FACTORS = 116259976 > INFOG(12) NB OF NEGATIVE PIVOTS = 0 > INFOG(14) NUMBER OF MEMORY COMPRESS = 0 > [0]after KSPsetup > [0]RHS setup > [1]after KSPsetup > [1]RHS setup > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): returning tag 2147483601 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [0] PetscCommDuplicate(): returning tag 2147483601 > [1] PetscCommDuplicate(): returning tag 2147483596 > [0] PetscCommDuplicate(): returning tag 2147483596 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): returning tag 2147483591 > [0] PetscCommDuplicate(): returning tag 2147483591 > [0] VecAssemblyBegin_MPI(): Stash has 225367 entries, uses 12 mallocs. > [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 mallocs. > [0] PetscCommDuplicate(): returning tag 2147483590 > [1] PetscCommDuplicate(): returning tag 2147483590 > [1] PetscCommDuplicate(): returning tag 2147483589 > [0] PetscCommDuplicate(): returning tag 2147483589 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483631 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483630 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483631 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483630 > > 4 processes of externalsolver stuck with 0% cpu > > > mpiexec -np 4 externalsolver > [1] PetscInitialize(): PETSc successfully started: number of > processors = 4 > [1] PetscInitialize(): Running on machine: mat1.uibk.ac.at > [3] PetscInitialize(): PETSc successfully started: number of > processors = 4 > [3] PetscInitialize(): Running on machine: mat1.uibk.ac.at > [0] PetscInitialize(): PETSc successfully started: number of > processors = 4 > [0] PetscInitialize(): Running on machine: mat1.uibk.ac.at > [2] PetscInitialize(): PETSc successfully started: number of > processors = 4 > [2] PetscInitialize(): Running on machine: mat1.uibk.ac.at > [3] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > [3] PetscCommDuplicate(): returning tag 2147483647 > [1] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > [1] PetscCommDuplicate(): returning tag 2147483647 > [2] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > [2] PetscCommDuplicate(): returning tag 2147483647 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [2] PetscCommDuplicate(): returning tag 2147483646 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [3] PetscCommDuplicate(): returning tag 2147483646 > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 > -2080374784 max tags = 2147483647 > [0] PetscCommDuplicate(): returning tag 2147483647 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [0] PetscCommDuplicate(): returning tag 2147483646 > [0] MatSetUpPreallocation(): Warning not preallocating matrix storage > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): returning tag 2147483646 > [1] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374783 max tags = 2147483647 > [3] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374783 max tags = 2147483647 > [3] PetscCommDuplicate(): returning tag 2147483647 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483646 > [0] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374783 max tags = 2147483647 > [0] PetscCommDuplicate(): returning tag 2147483647 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483646 > [1] PetscCommDuplicate(): returning tag 2147483647 > [2] PetscCommDuplicate(): Duplicating a communicator 1140850689 > -2080374783 max tags = 2147483647 > [2] PetscCommDuplicate(): returning tag 2147483647 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483646 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483646 > [0] MatSetOption_MPISBAIJ(): Option SYMMETRIC ignored > [0] MatStashScatterBegin_Private(): No of messages: 3 > [0] MatStashScatterBegin_Private(): Mesg_to: 1: size: 11261544 > [0] MatStashScatterBegin_Private(): Mesg_to: 2: size: 10384840 > [0] MatStashScatterBegin_Private(): Mesg_to: 3: size: 393344 > [0] MatStashScatterBegin_Private(): No of messages: 0 > [0] MatAssemblyBegin_MPISBAIJ(): Stash has 2754963 entries,uses 8 > mallocs. > [0] MatAssemblyBegin_MPISBAIJ(): Block-Stash has 2754963 entries, > uses 8 mallocs. > [3] MatAssemblyEnd_SeqSBAIJ(): Matrix size: 112413 X 112413, block > size 1; storage space: 8404901 unneeded, 2836399 used > [3] MatAssemblyEnd_SeqSBAIJ(): Number of mallocs during MatSetValues > is 0 > [3] MatAssemblyEnd_SeqSBAIJ(): Most nonzeros blocks in any row is 65 > [1] MatAssemblyEnd_SeqSBAIJ(): Matrix size: 113299 X 113299, block > size 1; storage space: 8647559 unneeded, 2682341 used > [1] MatAssemblyEnd_SeqSBAIJ(): Number of mallocs during MatSetValues > is 0 > [1] MatAssemblyEnd_SeqSBAIJ(): Most nonzeros blocks in any row is 62 > [0] MatAssemblyEnd_SeqSBAIJ(): Matrix size: 112871 X 112871, block > size 1; storage space: 8670798 unneeded, 2616302 used > [0] MatAssemblyEnd_SeqSBAIJ(): Number of mallocs during MatSetValues > is 0 > [0] MatAssemblyEnd_SeqSBAIJ(): Most nonzeros blocks in any row is 52 > [2] MatAssemblyEnd_SeqSBAIJ(): Matrix size: 108329 X 108329, block > size 1; storage space: 8247737 unneeded, 2585163 used > [2] MatAssemblyEnd_SeqSBAIJ(): Number of mallocs during MatSetValues > is 0 > [2] MatAssemblyEnd_SeqSBAIJ(): Most nonzeros blocks in any row is 52 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483645 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483644 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483643 > [1] PetscCommDuplicate(): returning tag 2147483639 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483645 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483644 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483643 > [3] PetscCommDuplicate(): returning tag 2147483639 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483645 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483644 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483643 > [2] PetscCommDuplicate(): returning tag 2147483639 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483645 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483644 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483643 > [0] PetscCommDuplicate(): returning tag 2147483639 > [0] PetscCommDuplicate(): returning tag 2147483634 > [2] PetscCommDuplicate(): returning tag 2147483634 > [1] PetscCommDuplicate(): returning tag 2147483634 > [3] PetscCommDuplicate(): returning tag 2147483634 > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > [0] VecScatterCreate(): General case: MPI to Seq > [0] PetscCommDuplicate(): returning tag 2147483630 > [0] PetscCommDuplicate(): returning tag 2147483625 > [1] PetscCommDuplicate(): returning tag 2147483630 > [1] PetscCommDuplicate(): returning tag 2147483625 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483642 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483641 > [1] PetscCommDuplicate(): returning tag 2147483620 > [2] PetscCommDuplicate(): returning tag 2147483630 > [2] PetscCommDuplicate(): returning tag 2147483625 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483642 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483641 > [2] PetscCommDuplicate(): returning tag 2147483620 > [3] PetscCommDuplicate(): returning tag 2147483630 > [3] PetscCommDuplicate(): returning tag 2147483625 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483642 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483641 > [3] PetscCommDuplicate(): returning tag 2147483620 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483640 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483639 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483638 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483642 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483641 > [0] PetscCommDuplicate(): returning tag 2147483620 > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > [0] VecScatterCreate(): General case: MPI to MPI > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483640 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483639 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483638 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483640 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483639 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483638 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483640 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483639 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483638 > [3] MatAssemblyEnd_SeqBAIJ(): Matrix size: 112413 X 0, block size 1; > storage space: 11241300 unneeded, 0 used > [3] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues > is 0 > [3] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 0 > [3] Mat_CheckCompressedRow(): Found the ratio (num_zerorows 112413)/ > (num_localrows 112413) > 0.6. Use CompressedRow routines. > [1] MatAssemblyEnd_SeqBAIJ(): Matrix size: 113299 X 2952, block size > 1; storage space: 11286236 unneeded, 43664 used > [1] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues > is 0 > [1] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 29 > [1] Mat_CheckCompressedRow(): Found the ratio (num_zerorows 110902)/ > (num_localrows 113299) > 0.6. Use CompressedRow routines. > [2] MatAssemblyEnd_SeqBAIJ(): Matrix size: 108329 X 9258, block size > 1; storage space: 10624060 unneeded, 208840 used > [2] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues > is 0 > [2] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 43 > [2] Mat_CheckCompressedRow(): Found the ratio (num_zerorows 92865)/ > (num_localrows 108329) > 0.6. Use CompressedRow routines. > [0] MatAssemblyEnd_SeqBAIJ(): Matrix size: 112871 X 17849, block > size 1; storage space: 10944569 unneeded, 342531 used > [0] MatAssemblyEnd_SeqBAIJ(): Number of mallocs during MatSetValues > is 0 > [0] MatAssemblyEnd_SeqBAIJ(): Most nonzeros blocks in any row is 43 > [0] Mat_CheckCompressedRow(): Found the ratio (num_zerorows 92260)/ > (num_localrows 112871) > 0.6. Use CompressedRow routines. > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [0] PetscCommDuplicate(): returning tag 2147483614 > [0] PetscCommDuplicate(): returning tag 2147483613 > [0]before KSPsetup > [0] PCSetUp(): Setting up new PC > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483637 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483636 > [0] PetscCommDuplicate(): returning tag 2147483612 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [1] PetscCommDuplicate(): returning tag 2147483614 > [1] PetscCommDuplicate(): returning tag 2147483613 > [1]before KSPsetup > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483637 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483636 > [1] PetscCommDuplicate(): returning tag 2147483612 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [2] PetscCommDuplicate(): returning tag 2147483614 > [2] PetscCommDuplicate(): returning tag 2147483613 > [2]before KSPsetup > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483637 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483636 > [2] PetscCommDuplicate(): returning tag 2147483612 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483635 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483634 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483635 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483634 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850688 -2080374784 > [3] PetscCommDuplicate(): returning tag 2147483614 > [3] PetscCommDuplicate(): returning tag 2147483613 > [3]before KSPsetup > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483637 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483636 > [3] PetscCommDuplicate(): returning tag 2147483612 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483635 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483634 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483635 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483634 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483633 > [1] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [1] PetscCommDuplicate(): returning tag 2147483632 > [1] PetscCommDuplicate(): returning tag 2147483607 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483633 > [2] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [2] PetscCommDuplicate(): returning tag 2147483632 > [2] PetscCommDuplicate(): returning tag 2147483607 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483633 > [0] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [0] PetscCommDuplicate(): returning tag 2147483632 > [0] PetscCommDuplicate(): returning tag 2147483607 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483633 > [3] PetscCommDuplicate(): Using internal PETSc communicator > 1140850689 -2080374783 > [3] PetscCommDuplicate(): returning tag 2147483632 > [3] PetscCommDuplicate(): returning tag 2147483607 > [0] PetscCommDuplicate(): returning tag 2147483602 > [1] PetscCommDuplicate(): returning tag 2147483602 > [2] PetscCommDuplicate(): returning tag 2147483602 > [3] PetscCommDuplicate(): returning tag 2147483602 > [0] VecScatterCreate(): Special case: processor zero gets entire > parallel vector, rest get none > > DMUMPS 4.8.4 > L D L^T Solver for symmetric positive definite matrices > Type of parallelism: Working host > > ****** ANALYSIS STEP ******** > > Density: NBdense, Average, Median = 0 49 50 > Ordering based on METIS > ** Peak of sequential stack size (number of real entries) : > 16763905. > A root of estimated size 2965 has been selected for > Scalapack. > > Leaving analysis phase with ... > INFOG(1) = 0 > INFOG(2) = 0 > -- (20) Number of entries in factors (estim.) = 120657071 > -- (3) Storage of factors (REAL, estimated) = 137362626 > -- (4) Storage of factors (INT , estimated) = 6167135 > -- (5) Maximum frontal size (estimated) = 3705 > -- (6) Number of nodes in the tree = 21863 > -- (7) Ordering option effectively used = 5 > ICNTL(6) Maximum transversal option = 0 > ICNTL(7) Pivot order option = 7 > Percentage of memory relaxation (effective) = 200 > Number of level 2 nodes = 5 > Number of split nodes = 0 > RINFO(1) Operations during elimination (estim) = 1.114D+11 > Distributed matrix entry format (ICNTL(18)) = 3 > ** Rank of proc needing largest memory in IC facto : 0 > ** Estimated corresponding MBYTES for IC facto : 1112 > ** Estimated avg. MBYTES per work. proc at facto (IC) : 1083 > ** TOTAL space in MBYTES for IC factorization : 4333 > ** Rank of proc needing largest memory for OOC facto : 1 > ** Estimated corresponding MBYTES for OOC facto : 465 > ** Estimated avg. MBYTES per work. proc at facto (OOC) : 421 > ** TOTAL space in MBYTES for OOC factorization : 1684 > > ****** FACTORIZATION STEP ******** > > > GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ... > NUMBER OF WORKING PROCESSES = 4 > REAL SPACE FOR FACTORS = 137362626 > INTEGER SPACE FOR FACTORS = 6167135 > MAXIMUM FRONTAL SIZE (ESTIMATED) = 3705 > NUMBER OF NODES IN THE TREE = 21863 > Maximum effective relaxed size of S = 125678965 > Average effective relaxed size of S = 122625031 > > REDISTRIB: TOTAL DATA LOCAL/SENT = 2497290 8939630 > GLOBAL TIME FOR MATRIX DISTRIBUTION = 0.6324 > ** Memory relaxation parameter ( ICNTL(14) ) : 200 > ** Rank of processor needing largest memory in facto : 0 > ** Space in MBYTES used by this processor for facto : 1112 > ** Avg. Space in MBYTES per working proc during facto : 1083 > From Andreas.Grassl at student.uibk.ac.at Tue Oct 13 03:05:50 2009 From: Andreas.Grassl at student.uibk.ac.at (Andreas Grassl) Date: Tue, 13 Oct 2009 10:05:50 +0200 Subject: SBAIJ issue In-Reply-To: References: <4AD32908.2050702@student.uibk.ac.at> Message-ID: <4AD434DE.8050804@student.uibk.ac.at> Hong Zhang schrieb: > Ando, > > I do not see any error message from attached info below. > Even '-log_summary' gives correct display. > I guess you sent us the working output (np=2). > I have attached 3 files. The one you found with -log_summary printed is indeed the working scenario. The other 2 are hanging. Output of top for np=4 when still "running": 8466 csae1801 25 0 1442m 704m 5708 R 100 5.9 1:20.87 externalsolver 8468 csae1801 25 0 1413m 697m 5052 R 100 5.8 1:13.45 externalsolver 8469 csae1801 25 0 1359m 614m 5148 R 100 5.1 1:12.75 externalsolver 8467 csae1801 25 0 1415m 702m 5096 R 96 5.9 1:13.01 externalsolver Output of top for np=4 when hanging: 8466 csae1801 18 0 1443m 769m 6120 S 0 6.4 2:09.47 externalsolver 8468 csae1801 15 0 1413m 759m 5420 S 0 6.3 2:00.87 externalsolver 8467 csae1801 15 0 1415m 748m 5396 S 0 6.2 2:01.21 externalsolver 8469 csae1801 18 0 1359m 688m 5460 S 0 5.7 2:01.39 externalsolver other processes use about 12% memory in sum. > I would suggest you run your code with debugger, > e.g., '-start_in_debugger'. > When it hangs, type Control-C, > and type 'where' to check where it hangs. > I guess it is hanging somewhere after the numerical factorization because the extrapolated time would match. Using debug-version or nondebug doesn't change the behaviour Output from where (using gdb): #0 0x0000003a0ccc5cdf in poll () from /lib64/libc.so.6 #1 0x00000000011d1024 in MPIDU_Sock_wait (sock_set=0x4464890, millisecond_timeout=4, eventp=0xffffffffffffffff) at sock_wait.i:124 #2 0x00000000011a3203 in MPIDI_CH3I_Progress (blocking=71714960, state=0x4) at ch3_progress.c:1038 #3 0x00000000011843ce in PMPI_Recv (buf=0x4464890, count=4, datatype=-1, source=-1, tag=108517088, comm=168072704, status=0x4f503b0) at recv.c:156 #4 0x0000000000ea9926 in BI_Srecv (ctxt=0x4f522d0, src=-2, msgid=2, bp=0x1813ad8) at BI_Srecv.c:8 #5 0x0000000000ea9414 in BI_SringBR (ctxt=0x4f522d0, bp=0x1813ad8, send=0xea9800 , src=1) at BI_SringBR.c:16 #6 0x0000000000ea22b1 in igebr2d_ (ConTxt=0x7fff0afeb110, scope=0x12a57f8 "Rowwise", top=0x17b9094 "S", m=0x12a57b8, n=0x12a57b8, A=0x7fff0afeb560, lda=0x12a57b8, rsrc=0x7fff0afeb118, csrc=0x7fff0afeb090) at igebr2d_.c:198 #7 0x0000000000e3b0f5 in pdpotf2 (uplo=Invalid C/C++ type code 13 in symbol table. ) at pdpotf2.f:340 #8 0x0000000000e2c818 in pdpotrf (uplo=Invalid C/C++ type code 13 in symbol table. ) at pdpotrf.f:327 #9 0x0000000000c5daf6 in dmumps_146 (myid=0, root= {mblock = 48, nblock = 48, nprow = 2, npcol = 2, myrow = 0, mycol = 0, root_size = 2965, tot_root_size = 2965, cntxt_blacs = 0, rg2l_row = 0x676f0bf, rg2l_col = 0x676f107, ipiv = 0x676f14f, descriptor = {1, 0, 2965, 2965, 48, 48, 0, 0, 1488}, descb = {0, 0, 0, 0, 0, 0, 0, 0, 0}, yes = 4294967295, gridinit_done = 4294967295, lpiv = 1, schur_pointer = 0x676f1eb, schur_mloc = 0, schur_nloc = 0, schur_lld = 0, qr_tau = 0x676f23f, qr_rcond = 0, maxg = 0, gind = 0, grow = 0x676f297, gcos = 0x676f2df, gsin = 0x676f327, elg_max = 0, null_max = 0, elind = 0, euind = 0, nlupdate = 0, nuupdate = 0, perm_row = 0x676f387, perm_col = 0x676f3cf, elrow = 0x676f417, eurow = 0x676f45f, ptrel = 0x676f4a7, ptreu = 0x676f4ef, elelg = 0x676f537, euelg = 0x676f57f, dl = 0x676f5c7}, n=446912, iroot=266997, comm=-2080374780, iw=0x2aaaf5c49010, liw=8275423, ifree=1646107, a=0x2aaab9d6e010, la=125678965, ptrast=0xb09d2fc, ptlust_s=0xb05d200, ptrfac=0xb0727b0, step=0xb4aca20, info={0, 0}, ldlt=1, qr=0, wk=0x2aaacab98ba0, lwk=90267651, keep= {8, 2571, 96, 24, 16, 48, 150, 120, 400, 6875958, 2147483646, 200, 3015153, 3259551, 1655023, 0, 0, 0, 0, 0, 0, 0, 0, 18, 0, 1646982, 3705, 21863, 8275423, 0, 0, 0, 0, 4, 8, 1, 800, 266997, 160000, -456788, 8, 0, 190998, 190998, 0, 1, 2, 5, 12663, 1, 48, 0, 0, 3, 0, 5, 500, 250, 0, 0, 0, 100, 60, 10, 120, 28139, 84754429, 0, 1, 0, 21863, 0, 0, 0, 1, 2, 30, 0, 2147483647, 1, 0, 5, 4, -8, 100, 1, 70, 70, 0, 1, 4, 0, 0, 0, 1, 0, 0, 0, 4, 12000000, 8791225, 150, 0, 16, 0, 1, 0, 1370, 0, 0, 0, 0, 11315240, 12209064, 0 , 6167135, 3705, 0 , 2214144, 0, 0, 0, 0, 0, 0, -1, 2, 2, 2214144, 201, 2, 0, 1, 0, 50, 1, 0, 0, 5, 2291986, 1670494, 1678547, 142320, 32, 0, 0, 0, 1, 3, 0, 1, 0, 0, 0, 12, 1, 10, 0 }, keep8= {0, 407769668, 177587312, 0, 0, 0, 0, 0, 31341437, 30351541, 35301388, 41892965, 125678965, 12496233, 574564, 0, 37488833, 0 , 120657071, 0, 137362626, 0 }) at dmumps_part7.F:286 #10 0x0000000000c17921 in dmumps_251 (n=446912, iw=0x2aaaf5c49010, liw=8275423, a=0x2aaab9d6e010, la=125678965, nstk_steps=0xb0dd3d0, nbprocfils=0xb0f296c, iflag=0, nd=0x4dbe8f0, fils=0xb661130, step=0xb4aca20, frere=0x4dd3ea0, dad=0x4de9450, cand=0x6a24830, istep_to_iniv2=0x4dfea00, tab_pos_in_pere=0x67bbff0, maxfrt=0, ntotpv=0, ptrist=0xb087d60, ptrast=0xb09d2fc, pimaster=0xb0b2898, pamaster=0xb0c7e34, ptrarw=0xb9c9f40, ptraiw=0xb815840, itloc=0xb107f08, ierror=0, ipool=0xb2bc608, lpool=21867, rinfo={28139655699.833332, 0 }, posfac=35411315, iwpos=1646106, lrlu=90267651, iptrlu=125678965, lrlus=90267651, leaf=1865, nbroot=1, nbrtot=4, uu=0, icntl= {6, 0, 6, -1, 0, 0, 7, 77, 1, 0, 0, 1, 0, 200, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, -8, 0 , 1, 0}, ptlust_s=0xb05d200, ptrfac=0xb0727b0, nsteps=5877, info= {0, 0, 35301388, 1646982, 3705, 0, 8275423, 125678965, 0, 0, 0, 0, 0, 0, 1112, 1112, 421, 0, 8392947, 37488833, 0, 0, 0, 31341437, 0 }, keep= {8, 2571, 96, 24, 16, 48, 150, 120, 400, 6875958, 2147483646, 200, 3015153, 3259551, 1655023, 0, 0, 0, 0, 0, 0, 0, 0, 18, 0, 1646982, 3705, 21863, 8275423, 0, 0, 0, 0, 4, 8, 1, 800, 266997, 160000, -456788, 8, 0, 190998, 190998, 0, 1, 2, 5, 12663, 1, 48, 0, 0, 3, 0, 5, 500, 250, 0, 0, 0, 100, 60, 10, 120, 28139, 84754429, 0, 1, 0, 21863, 0, 0, 0, 1, 2, 30, 0, 2147483647, 1, 0, 5, 4, -8, 100, 1, 70, 70, 0, 1, 4, 0, 0, 0, 1, 0, 0, 0, 4, 12000000, 8791225, 150, 0, 16, 0, 1, 0, 1370, 0, 0, 0, 0, 11315240, 12209064, 0 , 6167135, 3705, 0 , 2214144, 0, 0, 0, 0, 0, 0, -1, 2, 2, 2214144, 201, 2, 0, 1, 0, 50, 1, 0, 0, 5, 2291986, 1670494, 1678547, 142320, 32, 0, 0, 0, 1, 3, 0, 1, 0, 0, 0, 12, 1, 10, 0 }, keep8= {0, 407769668, 177587312, 0, 0, 0, 0, 0, 31341437, 30351541, 35301388, 41892965, 12567896---Type to continue, or q to quit--- -- /"\ Grassl Andreas \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik X against HTML email Technikerstr. 13 Zi 709 / \ +43 (0)512 507 6091 From Andreas.Grassl at student.uibk.ac.at Tue Oct 13 05:00:55 2009 From: Andreas.Grassl at student.uibk.ac.at (Andreas Grassl) Date: Tue, 13 Oct 2009 12:00:55 +0200 Subject: SBAIJ issue In-Reply-To: References: <4AD32908.2050702@student.uibk.ac.at> Message-ID: <4AD44FD7.6000800@student.uibk.ac.at> Barry Smith schrieb: > > Perhaps you have done enough memory preallocation for the > multiprocessors and it is going very--very slowly with memory allocations? do you mean this line MatMPISBAIJSetPreallocation(A, bs,nz, PETSC_NULL,nz, PETSC_NULL) I set bs to 1 and nz to 77 or do you mean the icntl(14) option of MUMPS? Increasing to 2000 allows to complete the run with np=4. on a even larger problem (~1800000 DOF) I now get by setting the icntl(23) option to a reasonable value a full run. So it seems it is a MUMPS-finetuning-problem!? The problem with -ksp_view_binary is persisting. Is it possible that MUMPS is working much faster and load balancing better if it has a vaste amount of memory available? Any general advices beside switching to a larger machine? Cheers, ando -- /"\ Grassl Andreas \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik X against HTML email Technikerstr. 13 Zi 709 / \ +43 (0)512 507 6091 From Andreas.Grassl at student.uibk.ac.at Tue Oct 13 07:44:13 2009 From: Andreas.Grassl at student.uibk.ac.at (Andreas Grassl) Date: Tue, 13 Oct 2009 14:44:13 +0200 Subject: SBAIJ issue In-Reply-To: References: <4AD32908.2050702@student.uibk.ac.at> Message-ID: <4AD4761D.2070800@student.uibk.ac.at> Hong Zhang schrieb: > I would suggest you run your code with debugger, > e.g., '-start_in_debugger'. > When it hangs, type Control-C, > and type 'where' to check where it hangs. The debugger output of the ongoing process when given option -ksp_view_binary with mumps_cholesky. It is hanging after solving. Explanations and suggestions? Cheers, ando Program received signal SIGINT, Interrupt. [Switching to Thread 46912507935936 (LWP 11965)] 0x00002aaaab00eb8a in __intel_new_memset () from /opt/intel/Compiler/11.0/074/lib/intel64/libirc.so (gdb) bt #0 0x00002aaaab00eb8a in __intel_new_memset () from /opt/intel/Compiler/11.0/074/lib/intel64/libirc.so #1 0x00002aaaaafecb66 in _intel_fast_memset.J () from /opt/intel/Compiler/11.0/074/lib/intel64/libirc.so #2 0x0000000000aefefe in PetscMemzero (a=0x14b206c0, n=8051408) at memc.c:205 #3 0x0000000000ab41d0 in PetscTrFreeDefault (aa=0x14b206c0, line=89, function=0x1260b10 "MatSeqXAIJFreeAIJ", file=0x1260840 "/home/lux/csae1801/petsc/petsc-3.0.0-p8/include/../src/mat/impls/aij/seq/aij.h", dir=0x1260ad4 "src/mat/impls/sbaij/mpi/") at mtr.c:318 #4 0x00000000008f606e in MatSeqXAIJFreeAIJ (AA=0x143112a0, a=0x143124c8, j=0x143124b8, i=0x143124b0) at /home/lux/csae1801/petsc/petsc-3.0.0-p8/include/../src/mat/impls/aij/seq/aij.h:89 #5 0x00000000008f773e in MatSetValues_MPISBAIJ (mat=0x133aaf30, m=1, im=0x15cc2f70, n=43, in=0x15f6fa10, v=0x16cfb938, addv=NOT_SET_VALUES) at mpisbaij.c:202 #6 0x00000000008fc0a9 in MatAssemblyEnd_MPISBAIJ (mat=0x133aaf30, mode=MAT_FINAL_ASSEMBLY) at mpisbaij.c:539 #7 0x0000000000633e5e in MatAssemblyEnd (mat=0x133aaf30, type=MAT_FINAL_ASSEMBLY) at matrix.c:4561 #8 0x00000000008fe302 in MatView_MPISBAIJ_ASCIIorDraworSocket (mat=0x11df00e0, viewer=0x133a4070) at mpisbaij.c:704 #9 0x00000000008fe95c in MatView_MPISBAIJ (mat=0x11df00e0, viewer=0x133a4070) at mpisbaij.c:733 #10 0x0000000000603570 in MatView (mat=0x11df00e0, viewer=0x133a4070) at matrix.c:643 #11 0x00000000004c9962 in KSPSolve (ksp=0x11f3ed80, b=0x116650a0, x=0x115fe9b0) at itfunc.c:328 #12 0x000000000040a5ff in main (argc=1, argv=0x7fff3baade68) at externalsolver.c:590 (gdb) c Continuing. Program received signal SIGINT, Interrupt. 0x00002aaaab00adf3 in __intel_new_memcpy () from /opt/intel/Compiler/11.0/074/lib/intel64/libirc.so (gdb) bt #0 0x00002aaaab00adf3 in __intel_new_memcpy () from /opt/intel/Compiler/11.0/074/lib/intel64/libirc.so #1 0x00002aaaaafecb16 in _intel_fast_memcpy.J () from /opt/intel/Compiler/11.0/074/lib/intel64/libirc.so #2 0x0000000000aef6f5 in PetscMemcpy (a=0x14b2abc0, b=0x156bed20, n=3721504) at memc.c:102 #3 0x00000000008f74b7 in MatSetValues_MPISBAIJ (mat=0x133aaf30, m=1, im=0x15cc7964, n=44, in=0x15f74404, v=0x16d04d20, addv=NOT_SET_VALUES) at mpisbaij.c:202 #4 0x00000000008fc0a9 in MatAssemblyEnd_MPISBAIJ (mat=0x133aaf30, mode=MAT_FINAL_ASSEMBLY) at mpisbaij.c:539 #5 0x0000000000633e5e in MatAssemblyEnd (mat=0x133aaf30, type=MAT_FINAL_ASSEMBLY) at matrix.c:4561 #6 0x00000000008fe302 in MatView_MPISBAIJ_ASCIIorDraworSocket (mat=0x11df00e0, viewer=0x133a4070) at mpisbaij.c:704 #7 0x00000000008fe95c in MatView_MPISBAIJ (mat=0x11df00e0, viewer=0x133a4070) at mpisbaij.c:733 #8 0x0000000000603570 in MatView (mat=0x11df00e0, viewer=0x133a4070) at matrix.c:643 #9 0x00000000004c9962 in KSPSolve (ksp=0x11f3ed80, b=0x116650a0, x=0x115fe9b0) at itfunc.c:328 #10 0x000000000040a5ff in main (argc=1, argv=0x7fff3baade68) at externalsolver.c:590 (gdb) -- /"\ Grassl Andreas \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik X against HTML email Technikerstr. 13 Zi 709 / \ +43 (0)512 507 6091 From nmoran at thphys.nuim.ie Tue Oct 13 08:28:06 2009 From: nmoran at thphys.nuim.ie (Niall Moran) Date: Tue, 13 Oct 2009 14:28:06 +0100 Subject: Building Issues on BGL Message-ID: Hi, I am having issues compiling the latest version of PETSc on a Bluegene/ L machine. I am using the IBM compilers version V1R3M3_420_2007-071023. The error I am getting during compile is ...../petsc-3.0.0-p8/src/sys/viewer/impls/socket "send.c", line 105.12: 1540-0274 (S) The name lookup for "gethostbyname" did not find a declaration. "send.c", line 110.37: 1540-0251 (S) The "->" operator cannot be applied to the undefined class "struct hostent". "send.c", line 110.48: 1540-0251 (S) The "->" operator cannot be applied to the undefined class "struct hostent". "send.c", line 112.21: 1540-0251 (S) The "->" operator cannot be applied to the undefined class "struct hostent". "send.c", line 115.21: 1540-0251 (S) The "->" operator cannot be applied to the undefined class "struct hostent". "send.c", line 188.8: 1540-0274 (S) The name lookup for "gethostbyname" did not find a declaration. "send.c", line 191.21: 1540-0251 (S) The "->" operator cannot be applied to the undefined class "struct hostent". "send.c", line 197.34: 1540-0274 (S) The name lookup for "SO_REUSEADDR" did not find a declaration. /usr/bin/ar: send.o: No such file or directory The config file I am using is below. I have tried commenting out some of the variables defined in the petscconf.h but to no avail. The variables I tried were PETSC_USE_SOCKET_VIEWER PETSC_HAVE_SYS_SOCKET_H PETSC_HAVE_GETHOSTBYNAME PETSC_HAVE_SOCKET I never encountered such issues with PETSc pre release 3. Any help is greatly appreciated. Regards, Niall. configure_options = [ '--with-cc=mpixlc', '--with-cxx=mpixlcxx', '--with-fc=mpixlf90 -qnosave -qfixed=72', '--with-mpi-dir=/bgl/BlueLight/ppcfloor/bglsys', # required by BLACS to get mpif.h '--with-clanguage=cxx', '--with-is-color-value-type=short', '--with-shared=0', '-COPTFLAGS=-O3 -qbgl -qarch=440d -qtune=440 -qmaxmem=-1', '-CXXOPTFLAGS=-O3 -qbgl -qarch=440d -qtune=440 -qmaxmem=-1', '-FOPTFLAGS=-O3 -qbgl -qarch=440d -qtune=440 -qmaxmem=-1', '-LIBS=-lc -lnss_files -lnss_dns -lresolv', '--with-debugging=0', '--with-scalar-type=complex', # the following option gets automatically enabled on BGL/with IBM compilers. # '--with-fortran-kernels=bgl' '--with-batch=1', '--with-memcmp-ok', '--sizeof_char=1', '--sizeof_void_p=4', '--sizeof_short=2', '--sizeof_int=4', '--sizeof_long=4', '--sizeof_size_t=4', '--sizeof_long_long=8', '--sizeof_float=4', '--sizeof_double=8', '--bits_per_byte=8', '--sizeof_MPI_Comm=4', '--sizeof_MPI_Fint=4', '--have-mpi-long-double=1', '--download-f-blas-lapack=1', '--download-hypre=0', '--download-spooles=0', '--download-superlu=0', '--download-parmetis=0', '--download-superlu_dist=0', '--download-blacs=1', '--download-scalapack=1', '--download-mumps=0' ] From knepley at mcs.anl.gov Tue Oct 13 08:48:28 2009 From: knepley at mcs.anl.gov (Matthew Knepley) Date: Tue, 13 Oct 2009 08:48:28 -0500 Subject: Building Issues on BGL In-Reply-To: References: Message-ID: 1) We always need configure.log 2) If you remove PETSC_USE_SOCKET_VIEWER, nothing in that directory will be built Matt On Tue, Oct 13, 2009 at 8:28 AM, Niall Moran wrote: > Hi, > > I am having issues compiling the latest version of PETSc on a Bluegene/L > machine. I am using the IBM compilers version > V1R3M3_420_2007-071023. > > The error I am getting during compile is > > ...../petsc-3.0.0-p8/src/sys/viewer/impls/socket > "send.c", line 105.12: 1540-0274 (S) The name lookup for "gethostbyname" > did not find a declaration. > "send.c", line 110.37: 1540-0251 (S) The "->" operator cannot be applied to > the undefined class "struct hostent". > "send.c", line 110.48: 1540-0251 (S) The "->" operator cannot be applied to > the undefined class "struct hostent". > "send.c", line 112.21: 1540-0251 (S) The "->" operator cannot be applied to > the undefined class "struct hostent". > "send.c", line 115.21: 1540-0251 (S) The "->" operator cannot be applied to > the undefined class "struct hostent". > "send.c", line 188.8: 1540-0274 (S) The name lookup for "gethostbyname" did > not find a declaration. > "send.c", line 191.21: 1540-0251 (S) The "->" operator cannot be applied to > the undefined class "struct hostent". > "send.c", line 197.34: 1540-0274 (S) The name lookup for "SO_REUSEADDR" did > not find a declaration. > /usr/bin/ar: send.o: No such file or directory > > The config file I am using is below. I have tried commenting out some of > the variables defined in the petscconf.h but to no avail. The variables I > tried were > PETSC_USE_SOCKET_VIEWER > PETSC_HAVE_SYS_SOCKET_H > PETSC_HAVE_GETHOSTBYNAME > PETSC_HAVE_SOCKET > > I never encountered such issues with PETSc pre release 3. Any help is > greatly appreciated. > > Regards, > > Niall. > > configure_options = [ > '--with-cc=mpixlc', > '--with-cxx=mpixlcxx', > '--with-fc=mpixlf90 -qnosave -qfixed=72', > '--with-mpi-dir=/bgl/BlueLight/ppcfloor/bglsys', # required by BLACS to > get mpif.h > '--with-clanguage=cxx', > > '--with-is-color-value-type=short', > '--with-shared=0', > '-COPTFLAGS=-O3 -qbgl -qarch=440d -qtune=440 -qmaxmem=-1', > '-CXXOPTFLAGS=-O3 -qbgl -qarch=440d -qtune=440 -qmaxmem=-1', > '-FOPTFLAGS=-O3 -qbgl -qarch=440d -qtune=440 -qmaxmem=-1', > '-LIBS=-lc -lnss_files -lnss_dns -lresolv', > '--with-debugging=0', > '--with-scalar-type=complex', > > # the following option gets automatically enabled on BGL/with IBM > compilers. > # '--with-fortran-kernels=bgl' > > '--with-batch=1', > '--with-memcmp-ok', > '--sizeof_char=1', > '--sizeof_void_p=4', > '--sizeof_short=2', > '--sizeof_int=4', > '--sizeof_long=4', > '--sizeof_size_t=4', > '--sizeof_long_long=8', > '--sizeof_float=4', > '--sizeof_double=8', > '--bits_per_byte=8', > '--sizeof_MPI_Comm=4', > '--sizeof_MPI_Fint=4', > '--have-mpi-long-double=1', > > '--download-f-blas-lapack=1', > '--download-hypre=0', > '--download-spooles=0', > '--download-superlu=0', > '--download-parmetis=0', > '--download-superlu_dist=0', > '--download-blacs=1', > '--download-scalapack=1', > '--download-mumps=0' > ] > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Tue Oct 13 08:56:27 2009 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 13 Oct 2009 08:56:27 -0500 Subject: SBAIJ issue In-Reply-To: <4AD4761D.2070800@student.uibk.ac.at> References: <4AD32908.2050702@student.uibk.ac.at> <4AD4761D.2070800@student.uibk.ac.at> Message-ID: <991C1FAD-48A9-435F-942C-BE8DF27968C7@mcs.anl.gov> It seems the code hangs in MatView_MPISBAIJ_ASCIIorDraworSocket (), a not well-tested routine. Can you send us your matrix in petsc binary format? We can test it and find a fix. My understanding is: when given the option -ksp_view_binary on np=4, the code hangs. What happens without the option -ksp_view_binary? If this is the case, the hang is not related to mumps. Hong On Oct 13, 2009, at 7:44 AM, Andreas Grassl wrote: > Hong Zhang schrieb: >> I would suggest you run your code with debugger, >> e.g., '-start_in_debugger'. >> When it hangs, type Control-C, >> and type 'where' to check where it hangs. > > The debugger output of the ongoing process when given option - > ksp_view_binary > with mumps_cholesky. It is hanging after solving. > > Explanations and suggestions? > > Cheers, > > ando > > Program received signal SIGINT, Interrupt. > [Switching to Thread 46912507935936 (LWP 11965)] > 0x00002aaaab00eb8a in __intel_new_memset () > from /opt/intel/Compiler/11.0/074/lib/intel64/libirc.so > (gdb) bt > #0 0x00002aaaab00eb8a in __intel_new_memset () > from /opt/intel/Compiler/11.0/074/lib/intel64/libirc.so > #1 0x00002aaaaafecb66 in _intel_fast_memset.J () > from /opt/intel/Compiler/11.0/074/lib/intel64/libirc.so > #2 0x0000000000aefefe in PetscMemzero (a=0x14b206c0, n=8051408) at > memc.c:205 > #3 0x0000000000ab41d0 in PetscTrFreeDefault (aa=0x14b206c0, line=89, > function=0x1260b10 "MatSeqXAIJFreeAIJ", > file=0x1260840 > "/home/lux/csae1801/petsc/petsc-3.0.0-p8/include/../src/mat/impls/ > aij/seq/aij.h", > dir=0x1260ad4 "src/mat/impls/sbaij/mpi/") at mtr.c:318 > #4 0x00000000008f606e in MatSeqXAIJFreeAIJ (AA=0x143112a0, > a=0x143124c8, > j=0x143124b8, > i=0x143124b0) > at > /home/lux/csae1801/petsc/petsc-3.0.0-p8/include/../src/mat/impls/aij/ > seq/aij.h:89 > #5 0x00000000008f773e in MatSetValues_MPISBAIJ (mat=0x133aaf30, m=1, > im=0x15cc2f70, n=43, > in=0x15f6fa10, v=0x16cfb938, addv=NOT_SET_VALUES) at mpisbaij.c:202 > #6 0x00000000008fc0a9 in MatAssemblyEnd_MPISBAIJ (mat=0x133aaf30, > mode=MAT_FINAL_ASSEMBLY) > at mpisbaij.c:539 > #7 0x0000000000633e5e in MatAssemblyEnd (mat=0x133aaf30, > type=MAT_FINAL_ASSEMBLY) > at matrix.c:4561 > #8 0x00000000008fe302 in MatView_MPISBAIJ_ASCIIorDraworSocket > (mat=0x11df00e0, > viewer=0x133a4070) at mpisbaij.c:704 > #9 0x00000000008fe95c in MatView_MPISBAIJ (mat=0x11df00e0, > viewer=0x133a4070) > at mpisbaij.c:733 > #10 0x0000000000603570 in MatView (mat=0x11df00e0, > viewer=0x133a4070) at > matrix.c:643 > #11 0x00000000004c9962 in KSPSolve (ksp=0x11f3ed80, b=0x116650a0, > x=0x115fe9b0) > at itfunc.c:328 > #12 0x000000000040a5ff in main (argc=1, argv=0x7fff3baade68) at > externalsolver.c:590 > (gdb) c > Continuing. > > Program received signal SIGINT, Interrupt. > 0x00002aaaab00adf3 in __intel_new_memcpy () > from /opt/intel/Compiler/11.0/074/lib/intel64/libirc.so > (gdb) bt > #0 0x00002aaaab00adf3 in __intel_new_memcpy () > from /opt/intel/Compiler/11.0/074/lib/intel64/libirc.so > #1 0x00002aaaaafecb16 in _intel_fast_memcpy.J () > from /opt/intel/Compiler/11.0/074/lib/intel64/libirc.so > #2 0x0000000000aef6f5 in PetscMemcpy (a=0x14b2abc0, b=0x156bed20, > n=3721504) at > memc.c:102 > #3 0x00000000008f74b7 in MatSetValues_MPISBAIJ (mat=0x133aaf30, m=1, > im=0x15cc7964, n=44, > in=0x15f74404, v=0x16d04d20, addv=NOT_SET_VALUES) at mpisbaij.c:202 > #4 0x00000000008fc0a9 in MatAssemblyEnd_MPISBAIJ (mat=0x133aaf30, > mode=MAT_FINAL_ASSEMBLY) > at mpisbaij.c:539 > #5 0x0000000000633e5e in MatAssemblyEnd (mat=0x133aaf30, > type=MAT_FINAL_ASSEMBLY) > at matrix.c:4561 > #6 0x00000000008fe302 in MatView_MPISBAIJ_ASCIIorDraworSocket > (mat=0x11df00e0, > viewer=0x133a4070) at mpisbaij.c:704 > #7 0x00000000008fe95c in MatView_MPISBAIJ (mat=0x11df00e0, > viewer=0x133a4070) > at mpisbaij.c:733 > #8 0x0000000000603570 in MatView (mat=0x11df00e0, > viewer=0x133a4070) at > matrix.c:643 > #9 0x00000000004c9962 in KSPSolve (ksp=0x11f3ed80, b=0x116650a0, > x=0x115fe9b0) > at itfunc.c:328 > #10 0x000000000040a5ff in main (argc=1, argv=0x7fff3baade68) at > externalsolver.c:590 > (gdb) > > -- > /"\ Grassl Andreas > \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik > X against HTML email Technikerstr. 13 Zi 709 > / \ +43 (0)512 507 6091 From balay at mcs.anl.gov Tue Oct 13 13:24:31 2009 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 13 Oct 2009 13:24:31 -0500 (CDT) Subject: Building Issues on BGL In-Reply-To: <7F2B79A8-847A-4C78-8807-78254183C335@thphys.nuim.ie> References: <7F2B79A8-847A-4C78-8807-78254183C335@thphys.nuim.ie> Message-ID: You have to remove the defines or rename them to something else in petscconf.h. Just commenting or undefining doesn't work. [this is because our build system does a grep on this file - so its doesn't exactly follow preprocessing directives] Regarding the original issue - I'm not sure why it doesn't work. We have V1R3M0_460_2008-081112P on our BGP - and I don't remember seeing this issue. Also - note; logfiles configure.log etc. should be sent to petsc-maint - and not the mailing list [we don't want to flood huge log files to all folks on this list] Satish On Tue, 13 Oct 2009, Niall Moran wrote: > 1) Please find attached the configure.log. > > 2) It seems just commenting out the definition of PETSC_USE_SOCKET_VIEWER did > not work but by undefining no attempt was made to build anything in the > directory. > > #if defined(PETSC_USE_SOCKET_VIEWER) > #undefine PETSC_USE_SOCKET_VIEWER > #endif > > Many thanks, > > Niall. > > On 13 Oct 2009, at 14:48, Matthew Knepley wrote: > > > 1) We always need configure.log > > > > 2) If you remove PETSC_USE_SOCKET_VIEWER, nothing in that directory will be > > built > > > > Matt > > > > On Tue, Oct 13, 2009 at 8:28 AM, Niall Moran wrote: > > Hi, > > > > I am having issues compiling the latest version of PETSc on a Bluegene/L > > machine. I am using the IBM compilers version > > V1R3M3_420_2007-071023. > > > > The error I am getting during compile is > > > > ...../petsc-3.0.0-p8/src/sys/viewer/impls/socket > > "send.c", line 105.12: 1540-0274 (S) The name lookup for "gethostbyname" did > > not find a declaration. > > "send.c", line 110.37: 1540-0251 (S) The "->" operator cannot be applied to > > the undefined class "struct hostent". > > "send.c", line 110.48: 1540-0251 (S) The "->" operator cannot be applied to > > the undefined class "struct hostent". > > "send.c", line 112.21: 1540-0251 (S) The "->" operator cannot be applied to > > the undefined class "struct hostent". > > "send.c", line 115.21: 1540-0251 (S) The "->" operator cannot be applied to > > the undefined class "struct hostent". > > "send.c", line 188.8: 1540-0274 (S) The name lookup for "gethostbyname" did > > not find a declaration. > > "send.c", line 191.21: 1540-0251 (S) The "->" operator cannot be applied to > > the undefined class "struct hostent". > > "send.c", line 197.34: 1540-0274 (S) The name lookup for "SO_REUSEADDR" did > > not find a declaration. > > /usr/bin/ar: send.o: No such file or directory > > > > The config file I am using is below. I have tried commenting out some of the > > variables defined in the petscconf.h but to no avail. The variables I tried > > were > > PETSC_USE_SOCKET_VIEWER > > PETSC_HAVE_SYS_SOCKET_H > > PETSC_HAVE_GETHOSTBYNAME > > PETSC_HAVE_SOCKET > > > > I never encountered such issues with PETSc pre release 3. Any help is > > greatly appreciated. > > > > Regards, > > > > Niall. > > > > configure_options = [ > > '--with-cc=mpixlc', > > '--with-cxx=mpixlcxx', > > '--with-fc=mpixlf90 -qnosave -qfixed=72', > > '--with-mpi-dir=/bgl/BlueLight/ppcfloor/bglsys', # required by BLACS to get > > mpif.h > > '--with-clanguage=cxx', > > > > '--with-is-color-value-type=short', > > '--with-shared=0', > > '-COPTFLAGS=-O3 -qbgl -qarch=440d -qtune=440 -qmaxmem=-1', > > '-CXXOPTFLAGS=-O3 -qbgl -qarch=440d -qtune=440 -qmaxmem=-1', > > '-FOPTFLAGS=-O3 -qbgl -qarch=440d -qtune=440 -qmaxmem=-1', > > '-LIBS=-lc -lnss_files -lnss_dns -lresolv', > > '--with-debugging=0', > > '--with-scalar-type=complex', > > > > # the following option gets automatically enabled on BGL/with IBM compilers. > > # '--with-fortran-kernels=bgl' > > > > '--with-batch=1', > > '--with-memcmp-ok', > > '--sizeof_char=1', > > '--sizeof_void_p=4', > > '--sizeof_short=2', > > '--sizeof_int=4', > > '--sizeof_long=4', > > '--sizeof_size_t=4', > > '--sizeof_long_long=8', > > '--sizeof_float=4', > > '--sizeof_double=8', > > '--bits_per_byte=8', > > '--sizeof_MPI_Comm=4', > > '--sizeof_MPI_Fint=4', > > '--have-mpi-long-double=1', > > > > '--download-f-blas-lapack=1', > > '--download-hypre=0', > > '--download-spooles=0', > > '--download-superlu=0', > > '--download-parmetis=0', > > '--download-superlu_dist=0', > > '--download-blacs=1', > > '--download-scalapack=1', > > '--download-mumps=0' > > ] > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments > > is infinitely more interesting than any results to which their experiments > > lead. > > -- Norbert Wiener > From zonexo at gmail.com Wed Oct 14 10:43:17 2009 From: zonexo at gmail.com (Wee-Beng TAY) Date: Wed, 14 Oct 2009 23:43:17 +0800 Subject: Is solving a Nx * Ny system equivalent to a Nx * Ny * 1 system in PETSc? Message-ID: <4AD5F195.7050400@gmail.com> Hi, I'm not sure if my question is trivial but I'm wondering if it makes any difference in solving a Nx * Ny system and a Nx * Ny * 1 system in PETSc? For e.g., 10x10 vs. 10x10x1. Will there be any problem or difference in answer? What about the speed? -- Thank you very much and have a nice day! Yours sincerely, Wee-Beng Tay From bsmith at mcs.anl.gov Wed Oct 14 12:41:33 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 14 Oct 2009 12:41:33 -0500 Subject: Is solving a Nx * Ny system equivalent to a Nx * Ny * 1 system in PETSc? In-Reply-To: <4AD5F195.7050400@gmail.com> References: <4AD5F195.7050400@gmail.com> Message-ID: Speed should be pretty much the same. I could interpret your question as: If one as a 3d PDE solver is it worth writing a totally separate 2d code or can one just run the 3d code with 2 dimensions? My answer to this question is, since code development and maintenance is very time-consuming it makes sense to just have one code. Barry On Oct 14, 2009, at 10:43 AM, Wee-Beng TAY wrote: > Hi, > > I'm not sure if my question is trivial but I'm wondering if it makes > any difference in solving a Nx * Ny system and a Nx * Ny * 1 system > in PETSc? For e.g., 10x10 vs. 10x10x1. Will there be any problem or > difference in answer? What about the speed? > > -- > Thank you very much and have a nice day! > > Yours sincerely, > > Wee-Beng Tay > From zonexo at gmail.com Thu Oct 15 09:21:24 2009 From: zonexo at gmail.com (Wee-Beng TAY) Date: Thu, 15 Oct 2009 22:21:24 +0800 Subject: Is solving a Nx * Ny system equivalent to a Nx * Ny * 1 system in PETSc? In-Reply-To: References: <4AD5F195.7050400@gmail.com> Message-ID: <4AD72FE4.9090602@gmail.com> Hi Barry, That's just what I'm thinking! Thank you very much and have a nice day! Yours sincerely, Wee-Beng Tay Barry Smith wrote: > > Speed should be pretty much the same. > > I could interpret your question as: If one as a 3d PDE solver is it > worth writing a totally separate 2d code or can one just run the 3d > code with 2 dimensions? My answer to this question is, since code > development and maintenance is very time-consuming it makes sense to > just have one code. > > Barry > > On Oct 14, 2009, at 10:43 AM, Wee-Beng TAY wrote: > >> Hi, >> >> I'm not sure if my question is trivial but I'm wondering if it makes >> any difference in solving a Nx * Ny system and a Nx * Ny * 1 system >> in PETSc? For e.g., 10x10 vs. 10x10x1. Will there be any problem or >> difference in answer? What about the speed? >> >> -- >> Thank you very much and have a nice day! >> >> Yours sincerely, >> >> Wee-Beng Tay >> > > From Harun.BAYRAKTAR at 3ds.com Thu Oct 15 12:52:33 2009 From: Harun.BAYRAKTAR at 3ds.com (BAYRAKTAR Harun) Date: Thu, 15 Oct 2009 13:52:33 -0400 Subject: Repeatability of results Message-ID: Hi, Is there a way in PETSc 2.3.3-p15 to force the order of operations in parallel so results do not vary from run to run on more than two processors? I did look in the archives and saw -vecscatter_reproduce but it was not clear to me if this is the only place where out of order operations can occur due to communications. For example can it happen during matrix assembly if 16 processors add to the same rows in the matrix? Thanks for your help, Harun Bayraktar -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Oct 15 14:27:38 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 15 Oct 2009 14:27:38 -0500 Subject: Repeatability of results In-Reply-To: References: Message-ID: Yes, it can happen in the matrix assembly with addition. Also in can happen in vector assembly. But the most common place it happens is in any reduction, like dot products. Reproduceability of all digits is nor worth trying to get, you cannot get it without PETSc having a lot more code. (and being much slower in that case). Barry On Oct 15, 2009, at 12:52 PM, BAYRAKTAR Harun wrote: > Hi, > > Is there a way in PETSc 2.3.3-p15 to force the order of operations > in parallel so results do not vary from run to run on more than two > processors? I did look in the archives and saw -vecscatter_reproduce > but it was not clear to me if this is the only place where out of > order operations can occur due to communications. For example can it > happen during matrix assembly if 16 processors add to the same rows > in the matrix? > > Thanks for your help, > Harun Bayraktar > From Guy_Antoine.Atenekeng_Kahou at lri.fr Fri Oct 16 10:00:49 2009 From: Guy_Antoine.Atenekeng_Kahou at lri.fr (Guy_Antoine.Atenekeng_Kahou at lri.fr) Date: Fri, 16 Oct 2009 17:00:49 +0200 Subject: condition number Message-ID: <6fc4bae61ce97e7355804a352894d86a.squirrel@www2.lri.fr> Hello, What is options to get conditionner number with superlu in petsc? Regards From hzhang at mcs.anl.gov Fri Oct 16 11:19:59 2009 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Fri, 16 Oct 2009 11:19:59 -0500 (CDT) Subject: condition number In-Reply-To: <6fc4bae61ce97e7355804a352894d86a.squirrel@www2.lri.fr> References: <6fc4bae61ce97e7355804a352894d86a.squirrel@www2.lri.fr> Message-ID: -mat_superlu_conditionnumber Run your code with -help |grep -i superlu SuperLU Options ------------------------------------------------- -mat_superlu_equil: Equil (None) -mat_superlu_colperm (choose one of) NATURAL MMD_ATA MMD_AT_PLUS_A COLAMD -mat_superlu_iterrefine (choose one of) NOREFINE SINGLE DOUBLE EXTRA -mat_superlu_symmetricmode: SymmetricMode (None) -mat_superlu_diagpivotthresh <1>: DiagPivotThresh (None) -mat_superlu_pivotgrowth: PivotGrowth (None) -mat_superlu_conditionnumber: ConditionNumber (None) -mat_superlu_rowperm (choose one of) NOROWPERM LargeDiag -mat_superlu_replacetinypivot: ReplaceTinyPivot (None) -mat_superlu_printstat: PrintStat (None) -mat_superlu_lwork <0>: size of work array in bytes used by factorization (None) -mat_superlu_ilu_droptol <0>: ILU_DropTol (None) -mat_superlu_ilu_filltol <0>: ILU_FillTol (None) -mat_superlu_ilu_fillfactor <0>: ILU_FillFactor (None) -mat_superlu_ilu_droprull <0>: ILU_DropRule (None) -mat_superlu_ilu_norm <0>: ILU_Norm (None) -mat_superlu_ilu_milu <0>: ILU_MILU (None) Hong On Fri, 16 Oct 2009, Guy_Antoine.Atenekeng_Kahou at lri.fr wrote: > Hello, > What is options to get conditionner number with superlu in petsc? > Regards > > From mossaiby at yahoo.com Sat Oct 17 04:40:34 2009 From: mossaiby at yahoo.com (Farshid Mossaiby) Date: Sat, 17 Oct 2009 02:40:34 -0700 (PDT) Subject: Off-topic: One dimensional incompressible Navier Stokes equation In-Reply-To: <6fc4bae61ce97e7355804a352894d86a.squirrel@www2.lri.fr> Message-ID: <364959.97266.qm@web52203.mail.re2.yahoo.com> Hi all, Sorry for another off-topic question! Can somebody please let me know what is the incompressible NS equation in 1D? And, has there been any attempt to solve this non-linear equation (in 1D, 2D or 3D) using Newton-Raphson linearization? Thanks, Farshid Mossaiby From Andreas.Grassl at student.uibk.ac.at Tue Oct 20 09:37:28 2009 From: Andreas.Grassl at student.uibk.ac.at (Andreas Grassl) Date: Tue, 20 Oct 2009 16:37:28 +0200 Subject: PaStiX crash Message-ID: <4ADDCB28.9070100@student.uibk.ac.at> Hello, I wanted to use PaStix and have the problem, that the debugging version works and PETSc compiled with option --with-debugging=0 gives following error: what could be wrong? +--------------------------------------------------------------------+ + PaStiX : Parallel Sparse matriX package + +--------------------------------------------------------------------+ Matrix size 7166 x 7166 Number of nonzeros 177831 +--------------------------------------------------------------------+ + Options + +--------------------------------------------------------------------+ Version : exported SMP_SOPALIN : Defined VERSION MPI : Defined PASTIX_BUBBLE : Not defined STATS_SOPALIN : Not defined NAPA_SOPALIN : Defined TEST_IRECV : Not defined TEST_ISEND : Defined THREAD_COMM : Not defined THREAD_FUNNELED : Not defined TAG : Exact Thread FORCE_CONSO : Not defined RECV_FANIN_OR_BLOCK : Not defined OUT_OF_CORE : Not defined DISTRIBUTED : Not defined FLUIDBOX : Not defined METIS : Not defined INTEGER TYPE : int32_t FLOAT TYPE : double +--------------------------------------------------------------------+ Check : ordering OK Check : Sort CSC OK [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC ERROR: or try http://valgrind.org on linux or man libgmalloc on Apple to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 8, Fri Aug 21 14:02:12 CDT 2009 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./standalonesolver on a linux32-i named login.leo1 by c702174 Tue Oct 20 11:55:24 2009 [0]PETSC ERROR: Libraries linked from /mnt/x4540/hpc-scratch/c702174/leo1/petsc/petsc-3.0.0-p8/linux32-intel-c-leo1/lib [0]PETSC ERROR: Configure run at Tue Oct 20 00:39:27 2009 [0]PETSC ERROR: Configure options --with-scalar-type=real --with-debugging=0 --with-precision=double --with-shared=0 --with-mpi=1 --with-mpi-dir=/usr/site/hpc/x86_64/glibc-2.5/italy/openmpi/1.3.3/intel-11.0 --with-scalapack=1 --download-scalapack=ifneeded --download-f-blas-lapack=ifneeded --with-blacs=1 --download-blacs=ifneeded --with-parmetis=1 --download-parmetis=ifneeded --with-mumps=1 --download-mumps=ifneeded --with-spooles=1 --download-spooles=ifneeded --with-superlu_dist=1 --download-superlu_dist=ifneeded --with-scotch=1 --download-scotch=ifneeded --with-pastix=1 --download-pastix=ifneeded --with-umfpack=1 --download-umfpack=ifneeded PETSC_ARCH=linux32-intel-c-leo1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file Cheers, ando -- /"\ Grassl Andreas \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik X against HTML email Technikerstr. 13 Zi 709 / \ +43 (0)512 507 6091 From balay at mcs.anl.gov Tue Oct 20 09:41:07 2009 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 20 Oct 2009 09:41:07 -0500 (CDT) Subject: PaStiX crash In-Reply-To: <4ADDCB28.9070100@student.uibk.ac.at> References: <4ADDCB28.9070100@student.uibk.ac.at> Message-ID: Perhaps you can try running in valgrind to see here the problem is. You can also try --with-debugging=0 COPTFLAGS='-g -O' - and see if it crashes. If so - run in a debugger to determine the problem. Satish On Tue, 20 Oct 2009, Andreas Grassl wrote: > Hello, > > I wanted to use PaStix and have the problem, that the debugging version works > and PETSc compiled with option --with-debugging=0 gives following error: > > what could be wrong? > > > +--------------------------------------------------------------------+ > + PaStiX : Parallel Sparse matriX package + > +--------------------------------------------------------------------+ > Matrix size 7166 x 7166 > Number of nonzeros 177831 > +--------------------------------------------------------------------+ > + Options + > +--------------------------------------------------------------------+ > Version : exported > SMP_SOPALIN : Defined > VERSION MPI : Defined > PASTIX_BUBBLE : Not defined > STATS_SOPALIN : Not defined > NAPA_SOPALIN : Defined > TEST_IRECV : Not defined > TEST_ISEND : Defined > THREAD_COMM : Not defined > THREAD_FUNNELED : Not defined > TAG : Exact Thread > FORCE_CONSO : Not defined > RECV_FANIN_OR_BLOCK : Not defined > OUT_OF_CORE : Not defined > DISTRIBUTED : Not defined > FLUIDBOX : Not defined > METIS : Not defined > INTEGER TYPE : int32_t > FLOAT TYPE : double > +--------------------------------------------------------------------+ > Check : ordering OK > Check : Sort CSC OK > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably > memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC > ERROR: or try http://valgrind.org on linux or man libgmalloc on Apple to find > memory corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [0]PETSC ERROR: to get more information on the crash. > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 8, Fri Aug 21 14:02:12 CDT 2009 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./standalonesolver on a linux32-i named login.leo1 by c702174 > Tue Oct 20 11:55:24 2009 > [0]PETSC ERROR: Libraries linked from > /mnt/x4540/hpc-scratch/c702174/leo1/petsc/petsc-3.0.0-p8/linux32-intel-c-leo1/lib > [0]PETSC ERROR: Configure run at Tue Oct 20 00:39:27 2009 > [0]PETSC ERROR: Configure options --with-scalar-type=real --with-debugging=0 > --with-precision=double --with-shared=0 --with-mpi=1 > --with-mpi-dir=/usr/site/hpc/x86_64/glibc-2.5/italy/openmpi/1.3.3/intel-11.0 > --with-scalapack=1 --download-scalapack=ifneeded > --download-f-blas-lapack=ifneeded --with-blacs=1 --download-blacs=ifneeded > --with-parmetis=1 --download-parmetis=ifneeded --with-mumps=1 > --download-mumps=ifneeded --with-spooles=1 --download-spooles=ifneeded > --with-superlu_dist=1 --download-superlu_dist=ifneeded --with-scotch=1 > --download-scotch=ifneeded --with-pastix=1 --download-pastix=ifneeded > --with-umfpack=1 --download-umfpack=ifneeded PETSC_ARCH=linux32-intel-c-leo1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > > > Cheers, > > ando > From jarunan at ascomp.ch Tue Oct 20 09:52:26 2009 From: jarunan at ascomp.ch (jarunan at ascomp.ch) Date: Tue, 20 Oct 2009 16:52:26 +0200 Subject: GMRES performance In-Reply-To: References: <4ADDCB28.9070100@student.uibk.ac.at> Message-ID: <20091020165226.eaqgcy9zokoww48w@webmail.ascomp.ch> Hello, I would like to know some information about GMRES performance in PETSc if you have any experience. I am running a parallel test case(9300 cells) comparing cpu time using by solvers in Petsc. While BICGSTAB was doing 0.9 sec, GMRES 15 sec with the same preconditioner(Additive Schwarz). I did not expect that GMRES would be that much slower. Everything is default. Please share your experiences, how the performance of solvers are in your test cases. Which option I should set to improve GMRES performance? Is there the best combination of Preconditioner and solver? Thank you very much Jarunan -- Jarunan Panyasantisuk Development Engineer ASCOMP GmbH, Technoparkstr. 1 CH-8005 Zurich, Switzerland Phone : +41 44 445 4072 Fax : +41 44 445 4075 E-mail: jarunan at ascomp.ch www.ascomp.ch From knepley at gmail.com Tue Oct 20 11:08:17 2009 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 20 Oct 2009 11:08:17 -0500 Subject: GMRES performance In-Reply-To: <20091020165226.eaqgcy9zokoww48w@webmail.ascomp.ch> References: <4ADDCB28.9070100@student.uibk.ac.at> <20091020165226.eaqgcy9zokoww48w@webmail.ascomp.ch> Message-ID: On Tue, Oct 20, 2009 at 9:52 AM, wrote: > Hello, > > I would like to know some information about GMRES performance in PETSc if > you have any experience. > > I am running a parallel test case(9300 cells) comparing cpu time using by > solvers in Petsc. While BICGSTAB was doing 0.9 sec, GMRES 15 sec with the > same preconditioner(Additive Schwarz). I did not expect that GMRES would be > that much slower. Everything is default. > I ssume you mean it takes many more iterates. This is certainly possible. > Please share your experiences, how the performance of solvers are in your > test cases. Which option I should set to improve GMRES performance? Is there > the best combination of Preconditioner and solver? > Solvers are highly problem dependent. There is no black box solution. Matt > Thank you very much > Jarunan > > -- > Jarunan Panyasantisuk > Development Engineer > ASCOMP GmbH, Technoparkstr. 1 > CH-8005 Zurich, Switzerland > Phone : +41 44 445 4072 > Fax : +41 44 445 4075 > E-mail: jarunan at ascomp.ch > www.ascomp.ch > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Oct 20 12:44:24 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 20 Oct 2009 12:44:24 -0500 Subject: PaStiX crash In-Reply-To: <4ADDCB28.9070100@student.uibk.ac.at> References: <4ADDCB28.9070100@student.uibk.ac.at> Message-ID: Run with valgrind (www.valgrind.org). Valgrind is simply the MOST important software tool for C/C++ developers available in the past 10+ years. First run the debug version in valgrind, because often there can be memory errors that don't crash the program and valgrind will find them. Barry On Oct 20, 2009, at 9:37 AM, Andreas Grassl wrote: > Hello, > > I wanted to use PaStix and have the problem, that the debugging > version works > and PETSc compiled with option --with-debugging=0 gives following > error: > > what could be wrong? > > > +--------------------------------------------------------------------+ > + PaStiX : Parallel Sparse matriX package + > +--------------------------------------------------------------------+ > Matrix size 7166 x 7166 > Number of nonzeros 177831 > +--------------------------------------------------------------------+ > + Options + > +--------------------------------------------------------------------+ > Version : exported > SMP_SOPALIN : Defined > VERSION MPI : Defined > PASTIX_BUBBLE : Not defined > STATS_SOPALIN : Not defined > NAPA_SOPALIN : Defined > TEST_IRECV : Not defined > TEST_ISEND : Defined > THREAD_COMM : Not defined > THREAD_FUNNELED : Not defined > TAG : Exact Thread > FORCE_CONSO : Not defined > RECV_FANIN_OR_BLOCK : Not defined > OUT_OF_CORE : Not defined > DISTRIBUTED : Not defined > FLUIDBOX : Not defined > METIS : Not defined > INTEGER TYPE : int32_t > FLOAT TYPE : double > +--------------------------------------------------------------------+ > Check : ordering OK > Check : Sort CSC OK > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation > Violation, probably > memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or - > on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal > [0]PETSC > ERROR: or try http://valgrind.org on linux or man libgmalloc on > Apple to find > memory corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, > link, and run > [0]PETSC ERROR: to get more information on the crash. > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 8, Fri Aug 21 > 14:02:12 CDT 2009 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./standalonesolver on a linux32-i named login.leo1 > by c702174 > Tue Oct 20 11:55:24 2009 > [0]PETSC ERROR: Libraries linked from > /mnt/x4540/hpc-scratch/c702174/leo1/petsc/petsc-3.0.0-p8/linux32- > intel-c-leo1/lib > [0]PETSC ERROR: Configure run at Tue Oct 20 00:39:27 2009 > [0]PETSC ERROR: Configure options --with-scalar-type=real --with- > debugging=0 > --with-precision=double --with-shared=0 --with-mpi=1 > --with-mpi-dir=/usr/site/hpc/x86_64/glibc-2.5/italy/openmpi/1.3.3/ > intel-11.0 > --with-scalapack=1 --download-scalapack=ifneeded > --download-f-blas-lapack=ifneeded --with-blacs=1 --download- > blacs=ifneeded > --with-parmetis=1 --download-parmetis=ifneeded --with-mumps=1 > --download-mumps=ifneeded --with-spooles=1 --download-spooles=ifneeded > --with-superlu_dist=1 --download-superlu_dist=ifneeded --with-scotch=1 > --download-scotch=ifneeded --with-pastix=1 --download-pastix=ifneeded > --with-umfpack=1 --download-umfpack=ifneeded PETSC_ARCH=linux32- > intel-c-leo1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > > > Cheers, > > ando > -- > /"\ Grassl Andreas > \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik > X against HTML email Technikerstr. 13 Zi 709 > / \ +43 (0)512 507 6091 From bsmith at mcs.anl.gov Tue Oct 20 12:46:51 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 20 Oct 2009 12:46:51 -0500 Subject: GMRES performance In-Reply-To: <20091020165226.eaqgcy9zokoww48w@webmail.ascomp.ch> References: <4ADDCB28.9070100@student.uibk.ac.at> <20091020165226.eaqgcy9zokoww48w@webmail.ascomp.ch> Message-ID: <2E445285-0CA7-4B8E-990E-13C5562D7EAF@mcs.anl.gov> On Oct 20, 2009, at 9:52 AM, jarunan at ascomp.ch wrote: > Hello, > > I would like to know some information about GMRES performance in > PETSc if you have any experience. > > I am running a parallel test case(9300 cells) comparing cpu time > using by solvers in Petsc. While BICGSTAB was doing 0.9 sec, GMRES > 15 sec with the same preconditioner(Additive Schwarz). I did not > expect that GMRES would be that much slower. Everything is default. You need much more information than runtimes of the two cases to understand why one is faster than the other. > > Please share your experiences, how the performance of solvers are in > your test cases. Which option I should set to improve GMRES > performance? Is there the best combination of Preconditioner and > solver? Run both (with debugging turned off, hence optimized) using the option -log_summary and look at where each is code is spending its time. You can also see how many iterations (MatMult) each solver is requiring. Feel free to send the -log_summary output to petsc-maint at mcs.anl.gov if you do not understand it. Barry > > Thank you very much > Jarunan > > -- > Jarunan Panyasantisuk > Development Engineer > ASCOMP GmbH, Technoparkstr. 1 > CH-8005 Zurich, Switzerland > Phone : +41 44 445 4072 > Fax : +41 44 445 4075 > E-mail: jarunan at ascomp.ch > www.ascomp.ch From keita at cray.com Tue Oct 20 16:37:08 2009 From: keita at cray.com (Keita Teranishi) Date: Tue, 20 Oct 2009 16:37:08 -0500 Subject: How to get more detailed output in make.log Message-ID: <5D6E0DF460ACF34C88644E1EA91DCD0D021DFE17@CFEXMBX.americas.cray.com> Hi, In PETSc, makefile in the top directory produces make.log, which shows only file names that have been compiled, but we'd like to see more detailed output like interaction with compiler commands such as "cc -O3 -c zplog.c." Is there any way to do that? Thanks, ================================ Keita Teranishi Scientific Library Group Cray, Inc. keita at cray.com ================================ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Oct 20 16:41:39 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 20 Oct 2009 16:41:39 -0500 Subject: How to get more detailed output in make.log In-Reply-To: <5D6E0DF460ACF34C88644E1EA91DCD0D021DFE17@CFEXMBX.americas.cray.com> References: <5D6E0DF460ACF34C88644E1EA91DCD0D021DFE17@CFEXMBX.americas.cray.com> Message-ID: <60EEAF01-AEC6-4170-BA18-87C5D0CB1010@mcs.anl.gov> make ACTION=lib tree Barry On Oct 20, 2009, at 4:37 PM, Keita Teranishi wrote: > Hi, > > In PETSc, makefile in the top directory produces make.log, which > shows only file names that have been compiled, but we?d like to see > more detailed output like interaction with compiler commands such as > ?cc ?O3 -c zplog.c.? Is there any way to do that? > > Thanks, > ================================ > Keita Teranishi > Scientific Library Group > Cray, Inc. > keita at cray.com > ================================ > From balay at mcs.anl.gov Tue Oct 20 16:41:59 2009 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 20 Oct 2009 16:41:59 -0500 (CDT) Subject: How to get more detailed output in make.log In-Reply-To: <5D6E0DF460ACF34C88644E1EA91DCD0D021DFE17@CFEXMBX.americas.cray.com> References: <5D6E0DF460ACF34C88644E1EA91DCD0D021DFE17@CFEXMBX.americas.cray.com> Message-ID: Usually the header at the begining lists these compiler options separately. But if you want the extra info - you can do: make ACTION=lib tree Note: this is not exactly the same build as the default build ['libfast' attempts to compile multiple files with a sigle command - whereas 'lib' does one source file at a time]. There is also some shell magic which is hidden [this shell stuff tries to figureout if compiles in the current dir should be done or skipped] Satish On Tue, 20 Oct 2009, Keita Teranishi wrote: > Hi, > > > > In PETSc, makefile in the top directory produces make.log, which shows only file names that have been compiled, but we'd like to see more detailed output like interaction with compiler commands such as "cc -O3 -c zplog.c." Is there any way to do that? > > > Thanks, > ================================ > Keita Teranishi > Scientific Library Group > Cray, Inc. > keita at cray.com > ================================ > > From u.tabak at tudelft.nl Tue Oct 20 19:27:16 2009 From: u.tabak at tudelft.nl (Umut Tabak) Date: Wed, 21 Oct 2009 02:27:16 +0200 Subject: C++ matrix/vector initialization in constructor Message-ID: <4ADE5564.7070701@tudelft.nl> Dear all, I tried to write a very primitive wrapper class for a linear equation solver, Ax=B. I am puzzled at a point and turned to the list. In my constructors, I use solveKxF::solveKxF(Mat &K, Vec &F, Vec &x) { // assign A = K; B = F; VecDuplicate(x, &y); // set up the solver setUpSolver(); } K and F are allocated before creating the object, however I do no allocation for A and B and PETSc automatically does the trick for me, right? (At least, it seems that way.) I just have the data members Mat A and Vec B which seems to work as above where the allocation is somewhere going behind the scenes in this assignment/initialization(which one is right term I am not sure). Best regards, Umut From knepley at gmail.com Wed Oct 21 04:25:16 2009 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 21 Oct 2009 04:25:16 -0500 Subject: C++ matrix/vector initialization in constructor In-Reply-To: <4ADE5564.7070701@tudelft.nl> References: <4ADE5564.7070701@tudelft.nl> Message-ID: The types Mat and Vec are actually pointers. The assignments are just pointer copies. Matt On Tue, Oct 20, 2009 at 7:27 PM, Umut Tabak wrote: > Dear all, > > I tried to write a very primitive wrapper class for a linear equation > solver, Ax=B. I am puzzled at a point and turned to the list. In my > constructors, I use > > solveKxF::solveKxF(Mat &K, Vec &F, Vec &x) > { // assign > A = K; B = F; > VecDuplicate(x, &y); > // set up the solver > setUpSolver(); > } > > K and F are allocated before creating the object, however I do no > allocation for A and B and PETSc automatically does the trick for me, right? > (At least, it seems that way.) I just have the data members Mat A and Vec B > which seems to work as above where the allocation is somewhere going behind > the scenes in this assignment/initialization(which one is right term I am > not sure). > > Best regards, > Umut > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Wed Oct 21 06:23:13 2009 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Wed, 21 Oct 2009 13:23:13 +0200 Subject: C++ error! MPI_Finalize() could not be located! Message-ID: <4ADEEF21.4090904@itis.ethz.ch> I get the following error building petsc-3.0.0-p8 on 32 bit linux (Ubuntu 9.04): ================================================================================= Running make on MPICH; this may take several minutes ================================================================================= TESTING: CxxMPICheck from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:599) ********************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): --------------------------------------------------------------------------------------- C++ error! MPI_Finalize() could not be located! ********************************************************************************* Tail of configure.log: ********************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): --------------------------------------------------------------------------------------- C++ error! MPI_Finalize() could not be located! ********************************************************************************* File "./config/configure.py", line 217, in petsc_configure framework.configure(out = sys.stdout) File "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/framework.py", line 902, in configure child.configure() File "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/package.py", line 445, in configure self.executeTest(self.configureLibrary) File "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/base.py", line 97, in executeTest ret = apply(test, args,kargs) File "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/packages/MPI.py", line 695, in configureLibrary self.executeTest(self.CxxMPICheck) File "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/base.py", line 97, in executeTest ret = apply(test, args,kargs) File "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/packages/MPI.py", line 613, in CxxMPICheck raise RuntimeError('C++ error! MPI_Finalize() could not be located!') The configure line: ./config/configure.py --with-cc=icc --with-fc=ifort --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 --with-x=0 with the Intel compiler version 11.0.083 (with -long_double and -no-multibyte-chars in icc.cfg). The same configuration (OS, compilers, versions) but on 64 bit system compiles fine. Any ideas? Google is silent. with regards, Dominik From knepley at gmail.com Wed Oct 21 06:31:10 2009 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 21 Oct 2009 06:31:10 -0500 Subject: C++ error! MPI_Finalize() could not be located! In-Reply-To: <4ADEEF21.4090904@itis.ethz.ch> References: <4ADEEF21.4090904@itis.ethz.ch> Message-ID: Send configure.log to petsc-maint at mcs.anl.gov Matt On Wed, Oct 21, 2009 at 6:23 AM, Dominik Szczerba wrote: > I get the following error building petsc-3.0.0-p8 on 32 bit linux (Ubuntu > 9.04): > > ================================================================================= > Running make on MPICH; this > may take several minutes > ================================================================================= > TESTING: CxxMPICheck from > config.packages.MPI(config/BuildSystem/config/packages/MPI.py:599) > > ********************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > > --------------------------------------------------------------------------------------- > C++ error! MPI_Finalize() could not be located! > > ********************************************************************************* > > Tail of configure.log: > > > ********************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > > --------------------------------------------------------------------------------------- > C++ error! MPI_Finalize() could not be located! > > ********************************************************************************* > File "./config/configure.py", line 217, in petsc_configure > framework.configure(out = sys.stdout) > File > "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/framework.py", > line 902, in configure > child.configure() > File > "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/package.py", line > 445, in configure > self.executeTest(self.configureLibrary) > File "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/base.py", > line 97, in executeTest > ret = apply(test, args,kargs) > File > "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/packages/MPI.py", > line 695, in configureLibrary > self.executeTest(self.CxxMPICheck) > File "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/base.py", > line 97, in executeTest > ret = apply(test, args,kargs) > File > "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/packages/MPI.py", > line 613, in CxxMPICheck > raise RuntimeError('C++ error! MPI_Finalize() could not be located!') > > > > The configure line: > > ./config/configure.py --with-cc=icc --with-fc=ifort > --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 --with-x=0 > > with the Intel compiler version 11.0.083 (with -long_double > and -no-multibyte-chars in icc.cfg). > > The same configuration (OS, compilers, versions) but on 64 bit system > compiles fine. Any ideas? Google is silent. > > with regards, > Dominik > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From petsc-maint at mcs.anl.gov Wed Oct 21 10:28:51 2009 From: petsc-maint at mcs.anl.gov (Satish Balay) Date: Wed, 21 Oct 2009 10:28:51 -0500 (CDT) Subject: [petsc-maint #36400] Re: C++ error! MPI_Finalize() could not be located! In-Reply-To: References: <4ADEEF21.4090904@itis.ethz.ch> <4ADF27D7.3030404@itis.ethz.ch> Message-ID: Since you are using hypre - also use the additional option -with-cxx=icpc [by default its picking up g++ - which is causing this incompatibility with icc etc..] Satish On Wed, 21 Oct 2009, Dominik Szczerba wrote: > configure.log is attached. Looking forward your feedback. > Dominik > > > > Matthew Knepley wrote: > > Send configure.log to petsc-maint at mcs.anl.gov > > > > > > Matt > > > > On Wed, Oct 21, 2009 at 6:23 AM, Dominik Szczerba > > wrote: > > > > I get the following error building petsc-3.0.0-p8 on 32 bit linux > > (Ubuntu 9.04): > > > > ================================================================================= > > Running make on > > MPICH; this may take several minutes > > ================================================================================= > > TESTING: CxxMPICheck from > > config.packages.MPI(config/BuildSystem/config/packages/MPI.py:599) > > ********************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > > for details): > > --------------------------------------------------------------------------------------- > > C++ error! MPI_Finalize() could not be located! > > ********************************************************************************* > > > > Tail of configure.log: > > > > ********************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > > for details): > > --------------------------------------------------------------------------------------- > > C++ error! MPI_Finalize() could not be located! > > ********************************************************************************* > > File "./config/configure.py", line 217, in petsc_configure > > framework.configure(out = sys.stdout) > > File > > "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/framework.py", > > line 902, in configure > > child.configure() > > File > > "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/package.py", > > line 445, in configure > > self.executeTest(self.configureLibrary) > > File > > "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/base.py", > > line 97, in executeTest > > ret = apply(test, args,kargs) > > File > > "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/packages/MPI.py", > > line 695, in configureLibrary > > self.executeTest(self.CxxMPICheck) > > File > > "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/base.py", > > line 97, in executeTest > > ret = apply(test, args,kargs) > > File > > "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/packages/MPI.py", > > line 613, in CxxMPICheck > > raise RuntimeError('C++ error! MPI_Finalize() could not be located!') > > > > > > > > The configure line: > > > > ./config/configure.py --with-cc=icc --with-fc=ifort > > --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 > > --with-x=0 > > > > with the Intel compiler version 11.0.083 (with -long_double > > and -no-multibyte-chars in icc.cfg). > > > > The same configuration (OS, compilers, versions) but on 64 bit > > system compiles fine. Any ideas? Google is silent. > > > > with regards, > > Dominik > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > > their experiments lead. > > -- Norbert Wiener > > > From Andreas.Grassl at student.uibk.ac.at Wed Oct 21 10:49:39 2009 From: Andreas.Grassl at student.uibk.ac.at (Andreas Grassl) Date: Wed, 21 Oct 2009 17:49:39 +0200 Subject: PaStiX crash In-Reply-To: References: <4ADDCB28.9070100@student.uibk.ac.at> Message-ID: <4ADF2D93.6090405@student.uibk.ac.at> Satish Balay schrieb: > Perhaps you can try running in valgrind to see here the problem is. > > You can also try --with-debugging=0 COPTFLAGS='-g -O' - and see if it > crashes. If so - run in a debugger to determine the problem. here you find the output of valgrind: http://pastebin.com/m16478dcf It seems the problem is around the scotch library. Trying to substitute the library with the working version from the debugging-branch did not work and I found no options to change the ordering algorithm to e.g. (par)metis installed for mumps Any ideas? Cheers, ando > > Satish > > On Tue, 20 Oct 2009, Andreas Grassl wrote: > >> Hello, >> >> I wanted to use PaStix and have the problem, that the debugging version works >> and PETSc compiled with option --with-debugging=0 gives following error: >> >> what could be wrong? >> >> >> +--------------------------------------------------------------------+ >> + PaStiX : Parallel Sparse matriX package + >> +--------------------------------------------------------------------+ >> Matrix size 7166 x 7166 >> Number of nonzeros 177831 >> +--------------------------------------------------------------------+ >> + Options + >> +--------------------------------------------------------------------+ >> Version : exported >> SMP_SOPALIN : Defined >> VERSION MPI : Defined >> PASTIX_BUBBLE : Not defined >> STATS_SOPALIN : Not defined >> NAPA_SOPALIN : Defined >> TEST_IRECV : Not defined >> TEST_ISEND : Defined >> THREAD_COMM : Not defined >> THREAD_FUNNELED : Not defined >> TAG : Exact Thread >> FORCE_CONSO : Not defined >> RECV_FANIN_OR_BLOCK : Not defined >> OUT_OF_CORE : Not defined >> DISTRIBUTED : Not defined >> FLUIDBOX : Not defined >> METIS : Not defined >> INTEGER TYPE : int32_t >> FLOAT TYPE : double >> +--------------------------------------------------------------------+ >> Check : ordering OK >> Check : Sort CSC OK >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably >> memory access out of range >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC >> ERROR: or try http://valgrind.org on linux or man libgmalloc on Apple to find >> memory corruption errors >> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >> [0]PETSC ERROR: to get more information on the crash. >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Signal received! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 8, Fri Aug 21 14:02:12 CDT 2009 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./standalonesolver on a linux32-i named login.leo1 by c702174 >> Tue Oct 20 11:55:24 2009 >> [0]PETSC ERROR: Libraries linked from >> /mnt/x4540/hpc-scratch/c702174/leo1/petsc/petsc-3.0.0-p8/linux32-intel-c-leo1/lib >> [0]PETSC ERROR: Configure run at Tue Oct 20 00:39:27 2009 >> [0]PETSC ERROR: Configure options --with-scalar-type=real --with-debugging=0 >> --with-precision=double --with-shared=0 --with-mpi=1 >> --with-mpi-dir=/usr/site/hpc/x86_64/glibc-2.5/italy/openmpi/1.3.3/intel-11.0 >> --with-scalapack=1 --download-scalapack=ifneeded >> --download-f-blas-lapack=ifneeded --with-blacs=1 --download-blacs=ifneeded >> --with-parmetis=1 --download-parmetis=ifneeded --with-mumps=1 >> --download-mumps=ifneeded --with-spooles=1 --download-spooles=ifneeded >> --with-superlu_dist=1 --download-superlu_dist=ifneeded --with-scotch=1 >> --download-scotch=ifneeded --with-pastix=1 --download-pastix=ifneeded >> --with-umfpack=1 --download-umfpack=ifneeded PETSC_ARCH=linux32-intel-c-leo1 >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >> >> >> Cheers, >> >> ando >> > -- /"\ Grassl Andreas \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik X against HTML email Technikerstr. 13 Zi 709 / \ +43 (0)512 507 6091 From bsmith at mcs.anl.gov Wed Oct 21 12:48:31 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 21 Oct 2009 12:48:31 -0500 Subject: PaStiX crash In-Reply-To: <4ADF2D93.6090405@student.uibk.ac.at> References: <4ADDCB28.9070100@student.uibk.ac.at> <4ADF2D93.6090405@student.uibk.ac.at> Message-ID: <5814DFD5-1EE0-48C3-9452-039AA7961BA0@mcs.anl.gov> I think if you compile all the code (including Scotch) with the -g option as Satish suggested then it should show exact line numbers in the source code where the corruption occurs and you could report it to the Scotch developers. As it is without the line numbers it may be difficult for the Scotch developers to determine the problem. Barry On Oct 21, 2009, at 10:49 AM, Andreas Grassl wrote: > Satish Balay schrieb: >> Perhaps you can try running in valgrind to see here the problem is. >> >> You can also try --with-debugging=0 COPTFLAGS='-g -O' - and see if it >> crashes. If so - run in a debugger to determine the problem. > > here you find the output of valgrind: > > http://pastebin.com/m16478dcf > > It seems the problem is around the scotch library. Trying to > substitute the > library with the working version from the debugging-branch did not > work and I > found no options to change the ordering algorithm to e.g. (par)metis > installed > for mumps > > Any ideas? > > Cheers, > > ando > >> >> Satish >> >> On Tue, 20 Oct 2009, Andreas Grassl wrote: >> >>> Hello, >>> >>> I wanted to use PaStix and have the problem, that the debugging >>> version works >>> and PETSc compiled with option --with-debugging=0 gives following >>> error: >>> >>> what could be wrong? >>> >>> >>> + >>> --------------------------------------------------------------------+ >>> + PaStiX : Parallel Sparse matriX >>> package + >>> + >>> --------------------------------------------------------------------+ >>> Matrix size 7166 x 7166 >>> Number of nonzeros 177831 >>> + >>> --------------------------------------------------------------------+ >>> + >>> Options + >>> + >>> --------------------------------------------------------------------+ >>> Version : exported >>> SMP_SOPALIN : Defined >>> VERSION MPI : Defined >>> PASTIX_BUBBLE : Not defined >>> STATS_SOPALIN : Not defined >>> NAPA_SOPALIN : Defined >>> TEST_IRECV : Not defined >>> TEST_ISEND : Defined >>> THREAD_COMM : Not defined >>> THREAD_FUNNELED : Not defined >>> TAG : Exact Thread >>> FORCE_CONSO : Not defined >>> RECV_FANIN_OR_BLOCK : Not defined >>> OUT_OF_CORE : Not defined >>> DISTRIBUTED : Not defined >>> FLUIDBOX : Not defined >>> METIS : Not defined >>> INTEGER TYPE : int32_t >>> FLOAT TYPE : double >>> + >>> --------------------------------------------------------------------+ >>> Check : ordering OK >>> Check : Sort CSC OK >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation >>> Violation, probably >>> memory access out of range >>> [0]PETSC ERROR: Try option -start_in_debugger or - >>> on_error_attach_debugger >>> [0]PETSC ERROR: or see >>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal >>> [0]PETSC >>> ERROR: or try http://valgrind.org on linux or man libgmalloc on >>> Apple to find >>> memory corruption errors >>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, >>> link, and run >>> [0]PETSC ERROR: to get more information on the crash. >>> [0]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [0]PETSC ERROR: Signal received! >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 8, Fri Aug 21 >>> 14:02:12 CDT 2009 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: ./standalonesolver on a linux32-i named login.leo1 >>> by c702174 >>> Tue Oct 20 11:55:24 2009 >>> [0]PETSC ERROR: Libraries linked from >>> /mnt/x4540/hpc-scratch/c702174/leo1/petsc/petsc-3.0.0-p8/linux32- >>> intel-c-leo1/lib >>> [0]PETSC ERROR: Configure run at Tue Oct 20 00:39:27 2009 >>> [0]PETSC ERROR: Configure options --with-scalar-type=real --with- >>> debugging=0 >>> --with-precision=double --with-shared=0 --with-mpi=1 >>> --with-mpi-dir=/usr/site/hpc/x86_64/glibc-2.5/italy/openmpi/1.3.3/ >>> intel-11.0 >>> --with-scalapack=1 --download-scalapack=ifneeded >>> --download-f-blas-lapack=ifneeded --with-blacs=1 --download- >>> blacs=ifneeded >>> --with-parmetis=1 --download-parmetis=ifneeded --with-mumps=1 >>> --download-mumps=ifneeded --with-spooles=1 --download- >>> spooles=ifneeded >>> --with-superlu_dist=1 --download-superlu_dist=ifneeded --with- >>> scotch=1 >>> --download-scotch=ifneeded --with-pastix=1 --download- >>> pastix=ifneeded >>> --with-umfpack=1 --download-umfpack=ifneeded PETSC_ARCH=linux32- >>> intel-c-leo1 >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: User provided function() line 0 in unknown >>> directory unknown file >>> >>> >>> Cheers, >>> >>> ando >>> >> > > > -- > /"\ Grassl Andreas > \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik > X against HTML email Technikerstr. 13 Zi 709 > / \ +43 (0)512 507 6091 From dominik at itis.ethz.ch Wed Oct 21 16:09:55 2009 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Wed, 21 Oct 2009 23:09:55 +0200 Subject: [petsc-maint #36400] Re: C++ error! MPI_Finalize() could not be located! In-Reply-To: References: <4ADEEF21.4090904@itis.ethz.ch> <4ADF27D7.3030404@itis.ethz.ch> Message-ID: <4ADF78A3.3010403@itis.ethz.ch> Thanks, the error is gone. That really saved my day! But you have to agree the connection between --with-cxx=icpc and "MPI_Finalize() could not be located!" is not trivially deduced. with regards, Dominik Satish Balay wrote: > Since you are using hypre - also use the additional option -with-cxx=icpc > > [by default its picking up g++ - which is causing this incompatibility with icc etc..] > > Satish > > > On Wed, 21 Oct 2009, Dominik Szczerba wrote: > >> configure.log is attached. Looking forward your feedback. >> Dominik >> >> >> >> Matthew Knepley wrote: >>> Send configure.log to petsc-maint at mcs.anl.gov >>> >>> >>> Matt >>> >>> On Wed, Oct 21, 2009 at 6:23 AM, Dominik Szczerba >> > wrote: >>> >>> I get the following error building petsc-3.0.0-p8 on 32 bit linux >>> (Ubuntu 9.04): >>> >>> ================================================================================= >>> Running make on >>> MPICH; this may take several minutes >>> ================================================================================= >>> TESTING: CxxMPICheck from >>> config.packages.MPI(config/BuildSystem/config/packages/MPI.py:599) >>> ********************************************************************************* >>> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log >>> for details): >>> --------------------------------------------------------------------------------------- >>> C++ error! MPI_Finalize() could not be located! >>> ********************************************************************************* >>> >>> Tail of configure.log: >>> >>> ********************************************************************************* >>> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log >>> for details): >>> --------------------------------------------------------------------------------------- >>> C++ error! MPI_Finalize() could not be located! >>> ********************************************************************************* >>> File "./config/configure.py", line 217, in petsc_configure >>> framework.configure(out = sys.stdout) >>> File >>> "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/framework.py", >>> line 902, in configure >>> child.configure() >>> File >>> "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/package.py", >>> line 445, in configure >>> self.executeTest(self.configureLibrary) >>> File >>> "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/base.py", >>> line 97, in executeTest >>> ret = apply(test, args,kargs) >>> File >>> "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/packages/MPI.py", >>> line 695, in configureLibrary >>> self.executeTest(self.CxxMPICheck) >>> File >>> "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/base.py", >>> line 97, in executeTest >>> ret = apply(test, args,kargs) >>> File >>> "/home/domel/pack/petsc-3.0.0-p8/config/BuildSystem/config/packages/MPI.py", >>> line 613, in CxxMPICheck >>> raise RuntimeError('C++ error! MPI_Finalize() could not be located!') >>> >>> >>> >>> The configure line: >>> >>> ./config/configure.py --with-cc=icc --with-fc=ifort >>> --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 >>> --with-x=0 >>> >>> with the Intel compiler version 11.0.083 (with -long_double >>> and -no-multibyte-chars in icc.cfg). >>> >>> The same configuration (OS, compilers, versions) but on 64 bit >>> system compiles fine. Any ideas? Google is silent. >>> >>> with regards, >>> Dominik >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which >>> their experiments lead. >>> -- Norbert Wiener >> >> > From Andreas.Grassl at student.uibk.ac.at Thu Oct 22 03:32:15 2009 From: Andreas.Grassl at student.uibk.ac.at (Andreas Grassl) Date: Thu, 22 Oct 2009 10:32:15 +0200 Subject: PaStiX crash In-Reply-To: <5814DFD5-1EE0-48C3-9452-039AA7961BA0@mcs.anl.gov> References: <4ADDCB28.9070100@student.uibk.ac.at> <4ADF2D93.6090405@student.uibk.ac.at> <5814DFD5-1EE0-48C3-9452-039AA7961BA0@mcs.anl.gov> Message-ID: <4AE0188F.3050700@student.uibk.ac.at> Hi Barry, here again with line numbers: http://pastebin.com/m630324e I noticed, the Pastix-version with only '-g' gives no errors. Hope this output now helps for debugging. Cheers, ando Barry Smith schrieb: > > I think if you compile all the code (including Scotch) with the -g > option as Satish suggested then it should show exact line numbers in the > source code where the corruption occurs and you could report it to the > Scotch developers. As it is without the line numbers it may be difficult > for the Scotch developers to determine the problem. > > > Barry > > On Oct 21, 2009, at 10:49 AM, Andreas Grassl wrote: > >> Satish Balay schrieb: >>> Perhaps you can try running in valgrind to see here the problem is. >>> >>> You can also try --with-debugging=0 COPTFLAGS='-g -O' - and see if it >>> crashes. If so - run in a debugger to determine the problem. >> >> here you find the output of valgrind: >> >> http://pastebin.com/m16478dcf >> >> It seems the problem is around the scotch library. Trying to >> substitute the >> library with the working version from the debugging-branch did not >> work and I >> found no options to change the ordering algorithm to e.g. (par)metis >> installed >> for mumps >> >> Any ideas? >> >> Cheers, >> >> ando >> >>> >>> Satish >>> >>> On Tue, 20 Oct 2009, Andreas Grassl wrote: >>> >>>> Hello, >>>> >>>> I wanted to use PaStix and have the problem, that the debugging >>>> version works >>>> and PETSc compiled with option --with-debugging=0 gives following >>>> error: >>>> >>>> what could be wrong? >>>> >>>> >>>> +--------------------------------------------------------------------+ >>>> + PaStiX : Parallel Sparse matriX package + >>>> +--------------------------------------------------------------------+ >>>> Matrix size 7166 x 7166 >>>> Number of nonzeros 177831 >>>> +--------------------------------------------------------------------+ >>>> + Options + >>>> +--------------------------------------------------------------------+ >>>> Version : exported >>>> SMP_SOPALIN : Defined >>>> VERSION MPI : Defined >>>> PASTIX_BUBBLE : Not defined >>>> STATS_SOPALIN : Not defined >>>> NAPA_SOPALIN : Defined >>>> TEST_IRECV : Not defined >>>> TEST_ISEND : Defined >>>> THREAD_COMM : Not defined >>>> THREAD_FUNNELED : Not defined >>>> TAG : Exact Thread >>>> FORCE_CONSO : Not defined >>>> RECV_FANIN_OR_BLOCK : Not defined >>>> OUT_OF_CORE : Not defined >>>> DISTRIBUTED : Not defined >>>> FLUIDBOX : Not defined >>>> METIS : Not defined >>>> INTEGER TYPE : int32_t >>>> FLOAT TYPE : double >>>> +--------------------------------------------------------------------+ >>>> Check : ordering OK >>>> Check : Sort CSC OK >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> >>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation >>>> Violation, probably >>>> memory access out of range >>>> [0]PETSC ERROR: Try option -start_in_debugger or >>>> -on_error_attach_debugger >>>> [0]PETSC ERROR: or see >>>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC >>>> >>>> ERROR: or try http://valgrind.org on linux or man libgmalloc on >>>> Apple to find >>>> memory corruption errors >>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, >>>> link, and run >>>> [0]PETSC ERROR: to get more information on the crash. >>>> [0]PETSC ERROR: --------------------- Error Message >>>> ------------------------------------ >>>> [0]PETSC ERROR: Signal received! >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> >>>> [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 8, Fri Aug 21 >>>> 14:02:12 CDT 2009 >>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> >>>> [0]PETSC ERROR: ./standalonesolver on a linux32-i named login.leo1 >>>> by c702174 >>>> Tue Oct 20 11:55:24 2009 >>>> [0]PETSC ERROR: Libraries linked from >>>> /mnt/x4540/hpc-scratch/c702174/leo1/petsc/petsc-3.0.0-p8/linux32-intel-c-leo1/lib >>>> >>>> [0]PETSC ERROR: Configure run at Tue Oct 20 00:39:27 2009 >>>> [0]PETSC ERROR: Configure options --with-scalar-type=real >>>> --with-debugging=0 >>>> --with-precision=double --with-shared=0 --with-mpi=1 >>>> --with-mpi-dir=/usr/site/hpc/x86_64/glibc-2.5/italy/openmpi/1.3.3/intel-11.0 >>>> >>>> --with-scalapack=1 --download-scalapack=ifneeded >>>> --download-f-blas-lapack=ifneeded --with-blacs=1 >>>> --download-blacs=ifneeded >>>> --with-parmetis=1 --download-parmetis=ifneeded --with-mumps=1 >>>> --download-mumps=ifneeded --with-spooles=1 --download-spooles=ifneeded >>>> --with-superlu_dist=1 --download-superlu_dist=ifneeded --with-scotch=1 >>>> --download-scotch=ifneeded --with-pastix=1 --download-pastix=ifneeded >>>> --with-umfpack=1 --download-umfpack=ifneeded >>>> PETSC_ARCH=linux32-intel-c-leo1 >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> >>>> [0]PETSC ERROR: User provided function() line 0 in unknown directory >>>> unknown file >>>> >>>> >>>> Cheers, >>>> >>>> ando >>>> >>> >> >> >> -- >> /"\ Grassl Andreas >> \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik >> X against HTML email Technikerstr. 13 Zi 709 >> / \ +43 (0)512 507 6091 > -- /"\ Grassl Andreas \ / ASCII Ribbon Campaign Uni Innsbruck Institut f. Mathematik X against HTML email Technikerstr. 13 Zi 709 / \ +43 (0)512 507 6091 From jarunan at ascomp.ch Thu Oct 22 03:37:21 2009 From: jarunan at ascomp.ch (jarunan at ascomp.ch) Date: Thu, 22 Oct 2009 10:37:21 +0200 Subject: GMRES performance In-Reply-To: <2E445285-0CA7-4B8E-990E-13C5562D7EAF@mcs.anl.gov> References: <4ADDCB28.9070100@student.uibk.ac.at> <20091020165226.eaqgcy9zokoww48w@webmail.ascomp.ch> <2E445285-0CA7-4B8E-990E-13C5562D7EAF@mcs.anl.gov> Message-ID: <20091022103721.xt9qtrj320wsw4c4@webmail.ascomp.ch> Hello, I followed the suggestion in PETSc previous threads by adding options -pc_type asm -sub_pc_type lu Now the solver is really fast!!!!! Thanks you Jarunan Quoting Barry Smith : > > On Oct 20, 2009, at 9:52 AM, jarunan at ascomp.ch wrote: > >> Hello, >> >> I would like to know some information about GMRES performance in >> PETSc if you have any experience. >> >> I am running a parallel test case(9300 cells) comparing cpu time >> using by solvers in Petsc. While BICGSTAB was doing 0.9 sec, GMRES >> 15 sec with the same preconditioner(Additive Schwarz). I did not >> expect that GMRES would be that much slower. Everything is default. > > You need much more information than runtimes of the two cases to > understand why one is faster than the other. >> >> Please share your experiences, how the performance of solvers are >> in your test cases. Which option I should set to improve GMRES >> performance? Is there the best combination of Preconditioner and >> solver? > > Run both (with debugging turned off, hence optimized) using the > option -log_summary and look at where each is code is spending its > time. You can also see how many iterations (MatMult) each solver is > requiring. Feel free to send the -log_summary output to > petsc-maint at mcs.anl.gov if you do not understand it. > > Barry > >> >> Thank you very much >> Jarunan >> >> -- >> Jarunan Panyasantisuk >> Development Engineer >> ASCOMP GmbH, Technoparkstr. 1 >> CH-8005 Zurich, Switzerland >> Phone : +41 44 445 4072 >> Fax : +41 44 445 4075 >> E-mail: jarunan at ascomp.ch >> www.ascomp.ch -- Jarunan Panyasantisuk Development Engineer ASCOMP GmbH, Technoparkstr. 1 CH-8005 Zurich, Switzerland Phone : +41 44 445 4072 Fax : +41 44 445 4075 E-mail: jarunan at ascomp.ch www.ascomp.ch From u.tabak at tudelft.nl Thu Oct 22 06:08:19 2009 From: u.tabak at tudelft.nl (Umut Tabak) Date: Thu, 22 Oct 2009 13:08:19 +0200 Subject: KSP different results with the default and direct LU Message-ID: <4AE03D23.5090603@tudelft.nl> Dear all, I am trying get myself more acquainted with PETSc. I am trying to solve a linear equation system which is conditioned badly. If I use the approach as given in ex1.c of the KSP section(by default, I read from the manual that it uses restarted GMRES , preconditioned with JACOBI in ex1.c which I am not very aware of these numerical procedures that in deep). I get some wrong results for the solution. I checked my results with Matlab, If I use the command line options as suggested on page 68 of the manual, to solve it directly, -ksp_type preonly -pc_type lu, I can both get the right solution and it is faster than the iterative solution which is used by default. So as far as I can follow from the mailing list, iterative methods and preconditioners are problem dependent, so my question would be, should I find the right approach/solver combination by some trial and error by using different combinations that are outlined in the manual?(This can also be dependent on the problem types that I would like to solve as well I suppose, for the moment, quite badly conditioned.) Any pointers are appreciated. Best regards, Umut From bsmith at mcs.anl.gov Thu Oct 22 08:36:11 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 22 Oct 2009 08:36:11 -0500 Subject: KSP different results with the default and direct LU In-Reply-To: <4AE03D23.5090603@tudelft.nl> References: <4AE03D23.5090603@tudelft.nl> Message-ID: <7A5F8E00-2A92-478C-965D-6C4BE1354A98@mcs.anl.gov> On Oct 22, 2009, at 6:08 AM, Umut Tabak wrote: > Dear all, > > I am trying get myself more acquainted with PETSc. > > I am trying to solve a linear equation system which is conditioned > badly. If I use the approach as given in ex1.c of the KSP section(by > default, I read from the manual that it uses restarted GMRES , > preconditioned with JACOBI in ex1.c which I am not very aware of > these numerical procedures that in deep). I get some wrong results > for the solution. When testing always run with -ksp_converged_reason or call KSPGetConvergedReason() after KSP solve to determine if PETSc thinks it has actually solved the system. Also since iterative solvers only solve to some tolerance the answer may not be "wrong" it may just be accurate up to the tolerance and with ill-conditioned matrices a tight tolerance on the residual norm may still be a loose tolerance on the norm of the error. > > I checked my results with Matlab, If I use the command line options > as suggested on page 68 of the manual, to solve it directly, - > ksp_type preonly -pc_type lu, I can both get the right solution and > it is faster than the iterative solution which is used by default. > So as far as I can follow from the mailing list, iterative methods > and preconditioners are problem dependent, so my question would be, > should I find the right approach/solver combination by some trial > and error by using different combinations that are outlined in the > manual?(This can also be dependent on the problem types that I would > like to solve as well I suppose, for the moment, quite badly > conditioned.) > General purpose iterative solvers, like in PETSc, used willy- nilly for very ill-conditioned linear systems, are basically worthless. You need to either stick to direct solvers or understand the types of iterative solvers that are used in the field of expertise for your class of problems. For example, if your problems come from semi-conductor simulations then you need to understand the literature of iterative solvers for semi-conductors before proceeding. For reasonably well-conditioned solvers where a variety of iterative methods "just work", that is converge ok, you can try them all to see what is fastest on your machine, but for nasty matrices this "trial and error" is a waste of time, because almost none of the iterative solvers will even converge, those that converge will not always converge (they are not reliable) and they will be slower than direct solvers. We've had good success with the MUMPS parallel direct solver and recommend trying the other ones as well. If your goal is to run some simulation (and not do "research" on iterative solvers for nasty matrices) I would just determine the best direct solver and use it (and live with the memory usage and time it requires). Barry > Any pointers are appreciated. > > Best regards, > Umut > > From u.tabak at tudelft.nl Thu Oct 22 08:52:08 2009 From: u.tabak at tudelft.nl (Umut Tabak) Date: Thu, 22 Oct 2009 15:52:08 +0200 Subject: KSP different results with the default and direct LU In-Reply-To: <7A5F8E00-2A92-478C-965D-6C4BE1354A98@mcs.anl.gov> References: <4AE03D23.5090603@tudelft.nl> <7A5F8E00-2A92-478C-965D-6C4BE1354A98@mcs.anl.gov> Message-ID: <4AE06388.8070602@tudelft.nl> Barry Smith wrote: > > When testing always run with -ksp_converged_reason or call > KSPGetConvergedReason() after KSP solve to determine if PETSc thinks > it has actually solved the system. Also since iterative solvers only > solve to some tolerance the answer may not be "wrong" it may just be > accurate up to the tolerance and with ill-conditioned matrices a tight > tolerance on the residual norm may still be a loose tolerance on the > norm of the error. Dear Barry, Thanks for the advice and tolerance reminder, indeed I missed that. > General purpose iterative solvers, like in PETSc, used willy-nilly > for very ill-conditioned linear systems, are basically worthless. You > need to either stick to direct solvers or understand the types of > iterative solvers that are used in the field of expertise for your > class of problems. For example, if your problems come from > semi-conductor simulations then you need to understand the literature > of iterative solvers for semi-conductors before proceeding. For > reasonably well-conditioned solvers where a variety of iterative > methods "just work", that is converge ok, you can try them all to see > what is fastest on your machine, but for nasty matrices this "trial > and error" is a waste of time, because almost none of the iterative > solvers will even converge, those that converge will not always > converge (they are not reliable) and they will be slower than direct > solvers. We've had good success with the MUMPS parallel direct solver > and recommend trying the other ones as well. If your goal is to run > some simulation (and not do "research" on iterative solvers for nasty > matrices) I would just determine the best direct solver and use it > (and live with the memory usage and time it requires). Thanks for this explanation, it is really helpful. My matrices seem quite nasty(and unsymmetric from the point of view of eigenvalues/vectors), so my goal is not conducting research in this field but rather to solve my linear systems and nasty eigenvalue problems as an engineer to get the results reliably. So as a conclusion I should go 'direct' as long as I am not pretty sure about reliability of my iterative methods. Thanks again. Umut From icksa1 at gmail.com Thu Oct 22 12:24:36 2009 From: icksa1 at gmail.com (Milad Fatenejad) Date: Thu, 22 Oct 2009 12:24:36 -0500 Subject: Manually setting DA min and max indexes Message-ID: Hello: Is it possible to set the min/max m,n,p indexes when creating a 3D distributed array? I'm writing a 3D code on a structured mesh, and the mesh is partitioned among the processes elsewhere. Assuming that each process already knows what chunk of the mesh it owns, is it possible to give Petsc this information when you create a DA so that the partitioning is consistent between PETSc and the rest of the code? Thank You Milad From jed at 59A2.org Thu Oct 22 12:40:46 2009 From: jed at 59A2.org (Jed Brown) Date: Thu, 22 Oct 2009 19:40:46 +0200 Subject: Manually setting DA min and max indexes In-Reply-To: References: Message-ID: <4AE0991E.1060300@59A2.org> Milad Fatenejad wrote: > Hello: > > Is it possible to set the min/max m,n,p indexes when creating a 3D > distributed array? I'm writing a 3D code on a structured mesh, and the > mesh is partitioned among the processes elsewhere. Assuming that each > process already knows what chunk of the mesh it owns, is it possible > to give Petsc this information > when you create a DA so that the partitioning is consistent between > PETSc and the rest of the code? For a simple decomposition, you just set the local sizes (m,n,p instead of M,N,P in DACreate3d). For irregular cases, use the lx,ly,lz inputs (or DASetVertexDivision). If you want to number the subdomains differently than PETSc, you should be able to make a communicator with your preferred numbering (see MPI_Group_incl) and create the DA on this communicator. Jed -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 261 bytes Desc: OpenPGP digital signature URL: From recrusader at gmail.com Thu Oct 22 14:24:47 2009 From: recrusader at gmail.com (Yujie) Date: Thu, 22 Oct 2009 14:24:47 -0500 Subject: One simple question about complex number Message-ID: <7ff0ee010910221224y1969ab5bnb994b434d8741d6e@mail.gmail.com> Hi, PETSc Developers, What is the difference between PETSc complex number and stl::complex. If I define a variable "var" using stl::complex, whether is it ok to do "var=2.0+PETSC_i*3.0"? Thanks a lot. Regards, Yujie -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Oct 22 15:05:56 2009 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 22 Oct 2009 15:05:56 -0500 (CDT) Subject: One simple question about complex number In-Reply-To: <7ff0ee010910221224y1969ab5bnb994b434d8741d6e@mail.gmail.com> References: <7ff0ee010910221224y1969ab5bnb994b434d8741d6e@mail.gmail.com> Message-ID: we use 'std:complex' with --with-clanguage=cxx [and with c - we use c99 complex support]. I'm not sure what stl::complex is. Satish On Thu, 22 Oct 2009, Yujie wrote: > Hi, PETSc Developers, > > What is the difference between PETSc complex number and stl::complex. If I > define a variable "var" using stl::complex, whether is it ok to do > "var=2.0+PETSC_i*3.0"? Thanks a lot. > > Regards, > Yujie > From recrusader at gmail.com Thu Oct 22 15:11:59 2009 From: recrusader at gmail.com (Yujie) Date: Thu, 22 Oct 2009 15:11:59 -0500 Subject: One simple question about complex number In-Reply-To: References: <7ff0ee010910221224y1969ab5bnb994b434d8741d6e@mail.gmail.com> Message-ID: <7ff0ee010910221311m3f253030ua093e80b7b3a7a7f@mail.gmail.com> Thank you for you reply, Satish. STL is Standard Template Library. I think stl::complex should be the same with std::complex. Regards, Yujie On Thu, Oct 22, 2009 at 3:05 PM, Satish Balay wrote: > we use 'std:complex' with --with-clanguage=cxx [and with c - we use > c99 complex support]. I'm not sure what stl::complex is. > > Satish > > On Thu, 22 Oct 2009, Yujie wrote: > > > Hi, PETSc Developers, > > > > What is the difference between PETSc complex number and stl::complex. If > I > > define a variable "var" using stl::complex, whether is it ok to do > > "var=2.0+PETSC_i*3.0"? Thanks a lot. > > > > Regards, > > Yujie > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Thu Oct 22 15:32:32 2009 From: jed at 59A2.org (Jed Brown) Date: Thu, 22 Oct 2009 22:32:32 +0200 Subject: One simple question about complex number In-Reply-To: <7ff0ee010910221311m3f253030ua093e80b7b3a7a7f@mail.gmail.com> References: <7ff0ee010910221224y1969ab5bnb994b434d8741d6e@mail.gmail.com> <7ff0ee010910221311m3f253030ua093e80b7b3a7a7f@mail.gmail.com> Message-ID: <4AE0C160.3080400@59A2.org> Yujie wrote: > Thank you for you reply, Satish. STL is Standard Template Library. I > think stl::complex should be the same with std::complex. The STL complex type is std::complex (STL uses the namespace std::), are you sure that you have an stl::complex? Jed -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 261 bytes Desc: OpenPGP digital signature URL: From torres.pedrozpk at gmail.com Thu Oct 22 17:38:28 2009 From: torres.pedrozpk at gmail.com (Pedro Juan Torres Lopez) Date: Thu, 22 Oct 2009 20:38:28 -0200 Subject: MatZeroRows ignore negative indices? Message-ID: Hello, In MatZeroRows() if I put negative value in the 3rd argument (rows[ ]) it will be ignored?. I'm sorry if I'm being repetitive with this issue. Regards Pedro -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Oct 22 19:13:43 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 22 Oct 2009 19:13:43 -0500 Subject: MatZeroRows ignore negative indices? In-Reply-To: References: Message-ID: No, there is no support for this. You should pass in an array that only contains the rows you want zeroed. You cannot pass in negative entries for row numbers. Barry On Oct 22, 2009, at 5:38 PM, Pedro Juan Torres Lopez wrote: > Hello, > > In MatZeroRows() if I put negative value in the 3rd argument > (rows[ ]) it will be ignored?. I'm sorry if I'm being repetitive > with this issue. > > Regards > > Pedro From dominik at itis.ethz.ch Fri Oct 23 14:32:02 2009 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 23 Oct 2009 21:32:02 +0200 Subject: [petsc-users] different results with 1 and more CPUs Message-ID: <4AE204B2.3050906@itis.ethz.ch> After several months of code stability I experience now the famous infamous dependence of my results on the # of CPUs (and more precisely, divergence with -np > 4). I upgraded Petsc version meanwhile, but I want to start by assuming I have a problem. Any pointers to how to set up a trap for this bug are highly appreciated. From jed at 59A2.org Fri Oct 23 14:51:30 2009 From: jed at 59A2.org (Jed Brown) Date: Fri, 23 Oct 2009 21:51:30 +0200 Subject: [petsc-users] different results with 1 and more CPUs In-Reply-To: <4AE204B2.3050906@itis.ethz.ch> References: <4AE204B2.3050906@itis.ethz.ch> Message-ID: <4AE20942.1020503@59A2.org> Dominik Szczerba wrote: > After several months of code stability I experience now the famous > infamous dependence of my results on the # of CPUs (and more precisely, > divergence with -np > 4). I upgraded Petsc version meanwhile, but I want > to start by assuming I have a problem. Any pointers to how to set up a > trap for this bug are highly appreciated. It's hopeless to get bit-for-bit identical answers from any parallel program, but the situation is much trickier with iterative solvers because almost every preconditioner is a different algorithm when you change the distribution. That said, you should be able to get the differences to converge by tightening tolerances (until you run out of floating point precision with very ill-conditioned systems). When you have a convergence problem in the linear solver, try a direct solver, -pc_type lu -pc_factor_mat_solver_package mumps (or even -pc_type redundant if you don't have a parallel direct solver). You can also write out a matrix in parallel (see MatView()) and compare it to the serial matrix. If you use SNES, I recommend running a small problem size in parallel and using -snes_type test which compares your Jacobian to one computed by finite differences. If you are still having problems after confirming that your function evaluation and matrix assembly is correct in parallel, the preconditioner is almost certainly the culprit. You have 3 options: 1. Use a parallel direct solver like MUMPS 2. Try to brute-force a standard preconditioner, e.g. -pc_type asm -pc_asm_overlap 3 -sub_pc_type lu 3. Read up on/design a problem-specific preconditioner that actually works and implement it. As Barry said here the other day, (3) is probably only a good choice if you want to do research in this rather than get some problems solved. Jed -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 261 bytes Desc: OpenPGP digital signature URL: From billy at dem.uminho.pt Sun Oct 25 10:04:52 2009 From: billy at dem.uminho.pt (=?iso-8859-1?Q?Billy_Ara=FAjo?=) Date: Sun, 25 Oct 2009 15:04:52 -0000 Subject: Memory leaks?? Message-ID: <1200D8BEDB3DD54DBA528E210F372BF3D944E1@BEFUNCIONARIOS.uminho.pt> I ran my application with PETSc and verified that memory consumption increased over time. I am not sure what is happening. Running my application with valgrind I get several leaks such as: ==25943== 16 bytes in 1 blocks are indirectly lost in loss record 9 of 11 ==25943== at 0x4024C1C: malloc (vg_replace_malloc.c:195) ==25943== by 0x4380C60: tsearch (tsearch.c:281) ==25943== by 0x4393D2D: __nss_lookup_function (nsswitch.c:342) ==25943== by 0x4831F25: ??? ==25943== by 0x4832B5C: ??? ==25943== by 0x434CCE4: getpwuid_r@@GLIBC_2.1.2 (getXXbyYY_r.c:253) ==25943== by 0x434C64E: getpwuid (getXXbyYY.c:117) ==25943== by 0x85DF39E: PetscGetUserName (fuser.c:68) ==25943== by 0x859E6D9: PetscErrorPrintfInitialize (errtrace.c:68) ==25943== by 0x85D0F78: PetscInitialize (pinit.c:521) ==25943== by 0x8050E37: main (main.c:516) In main.c:516, I just have: PetscInitialize (&argc, &argv, (char *) 0, help); which I thing is the normal way of calling PetscInitialize. Regards, Billy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vasoula_emp at hotmail.com Sun Oct 25 12:37:49 2009 From: vasoula_emp at hotmail.com (VASIA KALAVRI) Date: Sun, 25 Oct 2009 19:37:49 +0200 Subject: Parallel Cholesky solve Message-ID: Hello everyone. I have created a matrix of type MATMPIAIJ and I want to perform Cholesky factorization on it in order to solve a linear system. I have downloaded the Pastix package and I use MAT_SOLVER_PASTIX as an argument to MatGetFactor() but I get the following error message: "No support for this operation for this object type! Cannot use PETSc AIJ matrices with PaStiX Cholesky, use SBAIJ matrix!" Is there a way to use parralel Cholesky factorization with MATMPIAIJ type instead of SBAIJ? Thanks. _________________________________________________________________ Windows Live: Keep your friends up to date with what you do online. http://www.microsoft.com/middleeast/windows/windowslive/see-it-in-action/social-network-basics.aspx?ocid=PID23461::T:WLMTAGL:ON:WL:en-xm:SI_SB_1:092010 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Oct 25 12:48:48 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 25 Oct 2009 12:48:48 -0500 Subject: Parallel Cholesky solve In-Reply-To: References: Message-ID: <16AE532E-0A1F-4CFD-895F-AF17B37C50D4@mcs.anl.gov> On Oct 25, 2009, at 12:37 PM, VASIA KALAVRI wrote: > Hello everyone. > I have created a matrix of type MATMPIAIJ and I want to perform > Cholesky factorization on it in order to solve a linear system. > I have downloaded the Pastix package and I use MAT_SOLVER_PASTIX as > an argument to MatGetFactor() but I get the following error message: > > "No support for this operation for this object type! > Cannot use PETSc AIJ matrices with PaStiX Cholesky, use SBAIJ matrix!" > > Is there a way to use parralel Cholesky factorization with MATMPIAIJ > type instead of SBAIJ? No. Why not just create a MATMPISBAIJ or use MatConvert() to take the AIJ to SBAIJ. Barry > Thanks. > > Windows Live: Keep your friends up to date with what you do online. From bsmith at mcs.anl.gov Sun Oct 25 12:51:13 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 25 Oct 2009 12:51:13 -0500 Subject: Memory leaks?? In-Reply-To: <1200D8BEDB3DD54DBA528E210F372BF3D944E1@BEFUNCIONARIOS.uminho.pt> References: <1200D8BEDB3DD54DBA528E210F372BF3D944E1@BEFUNCIONARIOS.uminho.pt> Message-ID: This is a leak caused by the OS, there is nothing one can do about it. It also doesn't matter since it only happens once at the beginning of the run. It will not cause memory consumption to increase over time. Suggest running with -malloc_dump to see if some object is not freed. Barry On Oct 25, 2009, at 10:04 AM, Billy Ara?jo wrote: > > I ran my application with PETSc and verified that memory consumption > increased over time. I am not sure what is happening. > > Running my application with valgrind I get several leaks such as: > > ==25943== 16 bytes in 1 blocks are indirectly lost in loss record 9 > of 11 > ==25943== at 0x4024C1C: malloc (vg_replace_malloc.c:195) > ==25943== by 0x4380C60: tsearch (tsearch.c:281) > ==25943== by 0x4393D2D: __nss_lookup_function (nsswitch.c:342) > ==25943== by 0x4831F25: ??? > ==25943== by 0x4832B5C: ??? > ==25943== by 0x434CCE4: getpwuid_r@@GLIBC_2.1.2 (getXXbyYY_r.c:253) > ==25943== by 0x434C64E: getpwuid (getXXbyYY.c:117) > ==25943== by 0x85DF39E: PetscGetUserName (fuser.c:68) > ==25943== by 0x859E6D9: PetscErrorPrintfInitialize (errtrace.c:68) > ==25943== by 0x85D0F78: PetscInitialize (pinit.c:521) > ==25943== by 0x8050E37: main (main.c:516) > > > In main.c:516, I just have: > > PetscInitialize (&argc, &argv, (char *) 0, help); > > which I thing is the normal way of calling PetscInitialize. > > > Regards, > > Billy. > From fpacull at fluorem.com Tue Oct 27 10:12:21 2009 From: fpacull at fluorem.com (francois pacull) Date: Tue, 27 Oct 2009 16:12:21 +0100 Subject: UMFPACK out of memory Message-ID: <4AE70DD5.6030309@fluorem.com> Dear PETSc team, I have a few questions... During the resolution of a linear system in parallel, I am trying to apply a local LU solver to each of the SEQaij diagonal blocks of a MPIaij matrix partitioned with PARMETIS. - I started with MUMPS but it seems that it only works with unpartitioned aij matrices, is it really the case? Or could we use MUMPS to build an additive Schwarz preconditioner for example? - Then I tried UMFPACK. This works fine when the diagonal blocks (and the memory required to store the factors) are small but crashes when they are a little bit larger. For example with a "numeric final size" of 4203.7 MBytes, I got the following message "ERROR: out of memory" while there was plenty of memory left in the computer. I tried either with the UMFPACK version 5.2, downloaded by PETSc, or with a manually installed version 5.4, linked to PETSc. Is this a behavior from UMFPACK that you already experienced? - Since UMFPACK seemed to have a memory limit around 4096 MB, I tried to install a PETSc version with the option "--with-64-bit-indices", however none of the partitioning packages could be compiled with this option (parmetis,chaco,jostle,party,scotch). Is there a way to compile PETSc with 64 bit indices AND a partitioning package? - Finally, I tried to modify the PETSc source code umfpack.c so that it would deal with 64 bit indices, but I only ended up so far with a segmentation violation message at the execution... Is it the only way I could use UMPACK with large sparse matrices? Thank you, Regards, francois pacull. From knepley at gmail.com Tue Oct 27 10:17:57 2009 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 27 Oct 2009 10:17:57 -0500 Subject: UMFPACK out of memory In-Reply-To: <4AE70DD5.6030309@fluorem.com> References: <4AE70DD5.6030309@fluorem.com> Message-ID: On Tue, Oct 27, 2009 at 10:12 AM, francois pacull wrote: > Dear PETSc team, > > I have a few questions... During the resolution of a linear system in > parallel, I am trying to apply a local LU solver to each of the SEQaij > diagonal blocks of a MPIaij matrix partitioned with PARMETIS. > > - I started with MUMPS but it seems that it only works with unpartitioned > aij matrices, is it really the case? Or could we use MUMPS to build an > additive Schwarz preconditioner for example? > You can use MUMPS for the subproblem solver. > - Then I tried UMFPACK. This works fine when the diagonal blocks (and the > memory required to store the factors) are small but crashes when they are a > little bit larger. For example with a "numeric final size" of 4203.7 MBytes, > I got the following message "ERROR: out of memory" while there was plenty > of memory left in the computer. I tried either with the UMFPACK version 5.2, > downloaded by PETSc, or with a manually installed version 5.4, linked to > PETSc. Is this a behavior from UMFPACK that you already experienced? > Send all the error output. However, in oder to address more than 4G, you will need 64-bit pointers. - Since UMFPACK seemed to have a memory limit around 4096 MB, I tried to > install a PETSc version with the option "--with-64-bit-indices", however > none of the partitioning packages could be compiled with this option > (parmetis,chaco,jostle,party,scotch). Is there a way to compile PETSc with > 64 bit indices AND a partitioning package? > Not that I know of. > - Finally, I tried to modify the PETSc source code umfpack.c so that it > would deal with 64 bit indices, but I only ended up so far with a > segmentation violation message at the execution... Is it the only way I > could use UMPACK with large sparse matrices? > Why not just upgrade to a 64-bit OS if you want to address so much memory on a single machine? Matt > Thank you, > Regards, > francois pacull. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From fpacull at fluorem.com Tue Oct 27 10:56:06 2009 From: fpacull at fluorem.com (francois pacull) Date: Tue, 27 Oct 2009 16:56:06 +0100 Subject: UMFPACK out of memory In-Reply-To: References: <4AE70DD5.6030309@fluorem.com> Message-ID: <4AE71816.4000505@fluorem.com> Thank you Matt for your quick reply. I will try to use MUMPS soon. About UMFPACK, the OS used is a 64-bit Linux one. Also, we do have 64-bit pointers from what I understand; the following lines are from the PETSc configure.log file: #ifndef PETSC_SIZEOF_VOID_P #define PETSC_SIZEOF_VOID_P 8 #endif And here is the umfpack_UMF_report_info from the subdomain that crashed: UMFPACK V5.4.0 (May 20, 2009), Info: matrix entry defined as: double Int (generic integer) defined as: int BLAS library used: Fortran BLAS. size of BLAS integer: 4 MATLAB: no. CPU timer: POSIX times ( ) routine. number of rows in matrix A: 79002 number of columns in matrix A: 79002 entries in matrix A: 12030970 memory usage reported in: 8-byte Units size of int: 4 bytes size of UF_long: 8 bytes size of pointer: 8 bytes size of numerical entry: 8 bytes strategy used: unsymmetric ordering used: colamd on A modify Q during factorization: yes prefer diagonal pivoting: no pivots with zero Markowitz cost: 0 submatrix S after removing zero-cost pivots: number of "dense" rows: 0 number of "dense" columns: 0 number of empty rows: 0 number of empty columns 0 submatrix S square and diagonal preserved symbolic factorization defragmentations: 0 symbolic memory usage (Units): 27435792 symbolic memory usage (MBytes): 209.3 Symbolic size (Units): 177636 Symbolic size (MBytes): 1 symbolic factorization CPU time (sec): 4.95 symbolic factorization wallclock time(sec): 4.95 symbolic/numeric factorization: upper bound actual % variable-sized part of Numeric object: initial size (Units) 31236744 - - peak size (Units) 597607658 - - final size (Units) 550474688 - - Numeric final size (Units) 550988250 - - Numeric final size (MBytes) 4203.7 - - peak memory usage (Units) 598718594 - - peak memory usage (MBytes) 4567.9 - - numeric factorization flops 1.63141e+12 - - nz in L (incl diagonal) 171352664 - - nz in U (incl diagonal) 346187947 - - nz in L+U (incl diagonal) 517461609 - - largest front (# entries) 15783705 - - largest # rows in front 2815 - - largest # columns in front 5607 - - UMFPACK V5.4.0 (May 20, 2009): ERROR: out of memory Thanks again, francois. Matthew Knepley a ?crit : > On Tue, Oct 27, 2009 at 10:12 AM, francois pacull > wrote: > > Dear PETSc team, > > I have a few questions... During the resolution of a linear system > in parallel, I am trying to apply a local LU solver to each of the > SEQaij diagonal blocks of a MPIaij matrix partitioned with PARMETIS. > > - I started with MUMPS but it seems that it only works with > unpartitioned aij matrices, is it really the case? Or could we use > MUMPS to build an additive Schwarz preconditioner for example? > > > You can use MUMPS for the subproblem solver. > > > - Then I tried UMFPACK. This works fine when the diagonal blocks > (and the memory required to store the factors) are small but > crashes when they are a little bit larger. For example with a > "numeric final size" of 4203.7 MBytes, I got the following > message "ERROR: out of memory" while there was plenty of memory > left in the computer. I tried either with the UMFPACK version 5.2, > downloaded by PETSc, or with a manually installed version 5.4, > linked to PETSc. Is this a behavior from UMFPACK that you already > experienced? > > > Send all the error output. However, in oder to address more than 4G, > you will need 64-bit pointers. > > - Since UMFPACK seemed to have a memory limit around 4096 MB, I > tried to install a PETSc version with the option > "--with-64-bit-indices", however none of the partitioning packages > could be compiled with this option > (parmetis,chaco,jostle,party,scotch). Is there a way to compile > PETSc with 64 bit indices AND a partitioning package? > > > Not that I know of. > > > - Finally, I tried to modify the PETSc source code umfpack.c so > that it would deal with 64 bit indices, but I only ended up so far > with a segmentation violation message at the execution... Is it > the only way I could use UMPACK with large sparse matrices? > > > Why not just upgrade to a 64-bit OS if you want to address so much > memory on a single machine? > > Matt > > > Thank you, > Regards, > francois pacull. > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener From knepley at gmail.com Tue Oct 27 11:00:41 2009 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 27 Oct 2009 11:00:41 -0500 Subject: UMFPACK out of memory In-Reply-To: <4AE71816.4000505@fluorem.com> References: <4AE70DD5.6030309@fluorem.com> <4AE71816.4000505@fluorem.com> Message-ID: On Tue, Oct 27, 2009 at 10:56 AM, francois pacull wrote: > Thank you Matt for your quick reply. > > I will try to use MUMPS soon. About UMFPACK, the OS used is a 64-bit Linux > one. Also, we do have 64-bit pointers from what I understand; the following > lines are from the PETSc configure.log file: > > #ifndef PETSC_SIZEOF_VOID_P > #define PETSC_SIZEOF_VOID_P 8 > #endif > Yes, then it appears that UMFPACK cannot be used for large memory. Did you try MUMPS? Matt > And here is the umfpack_UMF_report_info from the subdomain that crashed: > > > UMFPACK V5.4.0 (May 20, 2009), Info: > matrix entry defined as: double > Int (generic integer) defined as: int > BLAS library used: Fortran BLAS. size of BLAS integer: 4 > MATLAB: no. > CPU timer: POSIX times ( ) routine. > number of rows in matrix A: 79002 > number of columns in matrix A: 79002 > entries in matrix A: 12030970 > memory usage reported in: 8-byte Units > size of int: 4 bytes > size of UF_long: 8 bytes > size of pointer: 8 bytes > size of numerical entry: 8 bytes > > strategy used: unsymmetric > ordering used: colamd on A > modify Q during factorization: yes > prefer diagonal pivoting: no > pivots with zero Markowitz cost: 0 > submatrix S after removing zero-cost pivots: > number of "dense" rows: 0 > number of "dense" columns: 0 > number of empty rows: 0 > number of empty columns 0 > submatrix S square and diagonal preserved > symbolic factorization defragmentations: 0 > symbolic memory usage (Units): 27435792 > symbolic memory usage (MBytes): 209.3 > Symbolic size (Units): 177636 > Symbolic size (MBytes): 1 > symbolic factorization CPU time (sec): 4.95 > symbolic factorization wallclock time(sec): 4.95 > > symbolic/numeric factorization: upper bound actual > % > variable-sized part of Numeric object: > initial size (Units) 31236744 - > - > peak size (Units) 597607658 - > - > final size (Units) 550474688 - > - > Numeric final size (Units) 550988250 - > - > Numeric final size (MBytes) 4203.7 - > - > peak memory usage (Units) 598718594 - > - > peak memory usage (MBytes) 4567.9 - > - > numeric factorization flops 1.63141e+12 - > - > nz in L (incl diagonal) 171352664 - > - > nz in U (incl diagonal) 346187947 - > - > nz in L+U (incl diagonal) 517461609 - > - > largest front (# entries) 15783705 - > - > largest # rows in front 2815 - > - > largest # columns in front 5607 - > - > > > UMFPACK V5.4.0 (May 20, 2009): ERROR: out of memory > > Thanks again, > francois. > > > > Matthew Knepley a ?crit : > > On Tue, Oct 27, 2009 at 10:12 AM, francois pacull > fpacull at fluorem.com>> wrote: >> >> Dear PETSc team, >> >> I have a few questions... During the resolution of a linear system >> in parallel, I am trying to apply a local LU solver to each of the >> SEQaij diagonal blocks of a MPIaij matrix partitioned with PARMETIS. >> >> - I started with MUMPS but it seems that it only works with >> unpartitioned aij matrices, is it really the case? Or could we use >> MUMPS to build an additive Schwarz preconditioner for example? >> >> >> You can use MUMPS for the subproblem solver. >> >> - Then I tried UMFPACK. This works fine when the diagonal blocks >> (and the memory required to store the factors) are small but >> crashes when they are a little bit larger. For example with a >> "numeric final size" of 4203.7 MBytes, I got the following >> message "ERROR: out of memory" while there was plenty of memory >> left in the computer. I tried either with the UMFPACK version 5.2, >> downloaded by PETSc, or with a manually installed version 5.4, >> linked to PETSc. Is this a behavior from UMFPACK that you already >> experienced? >> >> >> Send all the error output. However, in oder to address more than 4G, you >> will need 64-bit pointers. >> >> - Since UMFPACK seemed to have a memory limit around 4096 MB, I >> tried to install a PETSc version with the option >> "--with-64-bit-indices", however none of the partitioning packages >> could be compiled with this option >> (parmetis,chaco,jostle,party,scotch). Is there a way to compile >> PETSc with 64 bit indices AND a partitioning package? >> >> >> Not that I know of. >> >> - Finally, I tried to modify the PETSc source code umfpack.c so >> that it would deal with 64 bit indices, but I only ended up so far >> with a segmentation violation message at the execution... Is it >> the only way I could use UMPACK with large sparse matrices? >> >> >> Why not just upgrade to a 64-bit OS if you want to address so much memory >> on a single machine? >> >> Matt >> >> Thank you, >> Regards, >> francois pacull. >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From fpacull at fluorem.com Tue Oct 27 14:06:35 2009 From: fpacull at fluorem.com (francois pacull) Date: Tue, 27 Oct 2009 20:06:35 +0100 Subject: UMFPACK out of memory In-Reply-To: References: <4AE70DD5.6030309@fluorem.com> <4AE71816.4000505@fluorem.com> Message-ID: <4AE744BB.5030305@fluorem.com> Yes Matt, MUMPS as the subproblem solver works well. I think that when I tried it earlier, I forgot to convert my seqbaij submatrices into seqaij ones... Thanks, francois. ps: for information to the community, I had at first a similar problem as when using UMFPACK (crashed when a submatrix is large) because of the default MUMPS reordering mat_mumps_icntl_7 option that was set to METIS. After switching to PORD, the problem was solved. Matthew Knepley a ?crit : > On Tue, Oct 27, 2009 at 10:56 AM, francois pacull > wrote: > > Thank you Matt for your quick reply. > > I will try to use MUMPS soon. About UMFPACK, the OS used is a > 64-bit Linux one. Also, we do have 64-bit pointers from what I > understand; the following lines are from the PETSc configure.log file: > > #ifndef PETSC_SIZEOF_VOID_P > #define PETSC_SIZEOF_VOID_P 8 > #endif > > > Yes, then it appears that UMFPACK cannot be used for large memory. Did > you try MUMPS? > > Matt > > > And here is the umfpack_UMF_report_info from the subdomain that > crashed: > > > UMFPACK V5.4.0 (May 20, 2009), Info: > matrix entry defined as: double > Int (generic integer) defined as: int > BLAS library used: Fortran BLAS. size of BLAS integer: 4 > MATLAB: no. > CPU timer: POSIX times ( ) routine. > number of rows in matrix A: 79002 > number of columns in matrix A: 79002 > entries in matrix A: 12030970 > memory usage reported in: 8-byte Units > size of int: 4 bytes > size of UF_long: 8 bytes > size of pointer: 8 bytes > size of numerical entry: 8 bytes > > strategy used: unsymmetric > ordering used: colamd on A > modify Q during factorization: yes > prefer diagonal pivoting: no > pivots with zero Markowitz cost: 0 > submatrix S after removing zero-cost pivots: > number of "dense" rows: 0 > number of "dense" columns: 0 > number of empty rows: 0 > number of empty columns 0 > submatrix S square and diagonal preserved > symbolic factorization defragmentations: 0 > symbolic memory usage (Units): 27435792 > symbolic memory usage (MBytes): 209.3 > Symbolic size (Units): 177636 > Symbolic size (MBytes): 1 > symbolic factorization CPU time (sec): 4.95 > symbolic factorization wallclock time(sec): 4.95 > > symbolic/numeric factorization: upper bound > actual % > variable-sized part of Numeric object: > initial size (Units) 31236744 > - - > peak size (Units) 597607658 > - - > final size (Units) 550474688 > - - > Numeric final size (Units) 550988250 > - - > Numeric final size (MBytes) 4203.7 > - - > peak memory usage (Units) 598718594 > - - > peak memory usage (MBytes) 4567.9 > - - > numeric factorization flops 1.63141e+12 > - - > nz in L (incl diagonal) 171352664 > - - > nz in U (incl diagonal) 346187947 > - - > nz in L+U (incl diagonal) 517461609 > - - > largest front (# entries) 15783705 > - - > largest # rows in front 2815 > - - > largest # columns in front 5607 > - - > > > UMFPACK V5.4.0 (May 20, 2009): ERROR: out of memory > > Thanks again, > francois. > > > > Matthew Knepley a ?crit : > > On Tue, Oct 27, 2009 at 10:12 AM, francois pacull > > >> wrote: > > Dear PETSc team, > > I have a few questions... During the resolution of a linear > system > in parallel, I am trying to apply a local LU solver to each > of the > SEQaij diagonal blocks of a MPIaij matrix partitioned with > PARMETIS. > > - I started with MUMPS but it seems that it only works with > unpartitioned aij matrices, is it really the case? Or could > we use > MUMPS to build an additive Schwarz preconditioner for example? > > > You can use MUMPS for the subproblem solver. > > - Then I tried UMFPACK. This works fine when the diagonal > blocks > (and the memory required to store the factors) are small but > crashes when they are a little bit larger. For example with a > "numeric final size" of 4203.7 MBytes, I got the following > message "ERROR: out of memory" while there was plenty of memory > left in the computer. I tried either with the UMFPACK > version 5.2, > downloaded by PETSc, or with a manually installed version 5.4, > linked to PETSc. Is this a behavior from UMFPACK that you > already > experienced? > > > Send all the error output. However, in oder to address more > than 4G, you will need 64-bit pointers. > > - Since UMFPACK seemed to have a memory limit around 4096 > MB, I > tried to install a PETSc version with the option > "--with-64-bit-indices", however none of the partitioning > packages > could be compiled with this option > (parmetis,chaco,jostle,party,scotch). Is there a way to compile > PETSc with 64 bit indices AND a partitioning package? > > > Not that I know of. > > - Finally, I tried to modify the PETSc source code umfpack.c so > that it would deal with 64 bit indices, but I only ended up > so far > with a segmentation violation message at the execution... Is it > the only way I could use UMPACK with large sparse matrices? > > > Why not just upgrade to a 64-bit OS if you want to address so > much memory on a single machine? > > Matt > > Thank you, > Regards, > francois pacull. > > > > > -- > What most experimenters take for granted before they begin > their experiments is infinitely more interesting than any > results to which their experiments lead. > -- Norbert Wiener > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener From dominik at itis.ethz.ch Tue Oct 27 15:06:18 2009 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Tue, 27 Oct 2009 21:06:18 +0100 Subject: [petsc-users] Problems with the intel compilers Message-ID: <4AE752BA.7030404@itis.ethz.ch> I have noticed already a longer while ago: setting up PETSc with the Intel compilers (--with-cc=icc --with-cxx=icpc) takes an order of magnitude longer than with the native GNU. The step taking most of the time is configuring mpich. I have tried to configure mpich separately and indeed, with gnu it is a breeze, with intel 10x slower. In both cases, linux both 32 and 64 bit, Ubuntu 9.04 and debian testing. Intel compilers 10.x and 11.x. I just wanted to ask opinion if anybody has similar observations and/or finds using intel worthwhile at all. Dominik From s.kramer at imperial.ac.uk Tue Oct 27 15:28:48 2009 From: s.kramer at imperial.ac.uk (Stephan Kramer) Date: Tue, 27 Oct 2009 20:28:48 +0000 Subject: MPIBAIJ and sor Message-ID: <4AE75800.8010700@imperial.ac.uk> Hi all, I'm slightly confused by the fact that if I use my BAIJ matrix in parallel and select the sor preconditioner. it complains about mat->ops->relax and mat->ops->pbrelax not being defined. Also I can't find a MatRelax_MPIBAIJ in src/mat/impls/baij/mpi/mpibaij.c as I would have expected. Is there no sor type relaxation for parallel block matrices at all? Cheers Stephan From bsmith at mcs.anl.gov Tue Oct 27 15:35:50 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 27 Oct 2009 15:35:50 -0500 Subject: MPIBAIJ and sor In-Reply-To: <4AE75800.8010700@imperial.ac.uk> References: <4AE75800.8010700@imperial.ac.uk> Message-ID: <31D3563A-AAF1-400E-AAB3-B21CE1FC72E9@mcs.anl.gov> Stephan, There is no MatRelax_ for BAIJ matrices and it is only written for SBAIJ (symmetric storage) for block size 1. Most people end up using ILU(0) or ICC(0) for relaxing with block matrices. It converges much better (generally) and takes the same amount of time. The only drawback is that it requires another copy of the matrix (uses more memory) for the "factored" version. Most people are time constrained instead of memory constrained so this is not an issue. It would not be terribly difficult to add relaxation for BAIJ; it has to be done one block size at a time. What block size do you need, are you interested in writing it? Barry On Oct 27, 2009, at 3:28 PM, Stephan Kramer wrote: > Hi all, > > I'm slightly confused by the fact that if I use my BAIJ matrix in > parallel and select the sor preconditioner. it complains about mat- > >ops->relax and mat->ops->pbrelax not being defined. Also I can't > find a MatRelax_MPIBAIJ in src/mat/impls/baij/mpi/mpibaij.c as I > would have expected. Is there no sor type relaxation for parallel > block matrices at all? > > Cheers > Stephan From jed at 59A2.org Tue Oct 27 15:39:42 2009 From: jed at 59A2.org (Jed Brown) Date: Tue, 27 Oct 2009 21:39:42 +0100 Subject: MPIBAIJ and sor In-Reply-To: <4AE75800.8010700@imperial.ac.uk> References: <4AE75800.8010700@imperial.ac.uk> Message-ID: <4AE75A8E.6040606@59A2.org> Stephan Kramer wrote: > I'm slightly confused by the fact that if I use my BAIJ matrix in > parallel and select the sor preconditioner. it complains about > mat->ops->relax and mat->ops->pbrelax not being defined. There is no parallel SOR, it would make the algorithm nearly serial. Use -sub_pc_type sor with -ksp_type asm or bjacobi. > Also I can't find a MatRelax_MPIBAIJ in > src/mat/impls/baij/mpi/mpibaij.c as I would have expected. Is there no > sor type relaxation for parallel block matrices at all? Since the small dense blocks are available at once, it would be silly to relax each component separately (your CPU spends most of its time waiting on memory). Instead, BAIJ relaxes all components simultaneously by solving with these small blocks. This is "point-block" relaxation, look for MatPBRelax_SeqBAIJ_N for N in 1..7. Jed -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 261 bytes Desc: OpenPGP digital signature URL: From s.kramer at imperial.ac.uk Tue Oct 27 16:00:36 2009 From: s.kramer at imperial.ac.uk (Stephan Kramer) Date: Tue, 27 Oct 2009 21:00:36 +0000 Subject: MPIBAIJ and sor In-Reply-To: <31D3563A-AAF1-400E-AAB3-B21CE1FC72E9@mcs.anl.gov> References: <4AE75800.8010700@imperial.ac.uk> <31D3563A-AAF1-400E-AAB3-B21CE1FC72E9@mcs.anl.gov> Message-ID: <4AE75F74.6060806@imperial.ac.uk> Barry Smith wrote: > Stephan, > > There is no MatRelax_ for BAIJ matrices and it is only written for > SBAIJ (symmetric storage) for block size 1. > Ah ok > Most people end up using ILU(0) or ICC(0) for relaxing with block > matrices. It converges much better (generally) and takes the same > amount of time. The only drawback is that it requires another copy of > the matrix (uses more memory) for the "factored" version. Most people > are time constrained instead of memory constrained so this is not an > issue. Will give ILU a try then. I'm using this to solve an implicit dg momentum equation in an ocean model: so basically advection and diffusion terms (tensor form) and some coupling between the momentum components due to coriolis. In serial I found a significant improvement in convergence using the block structure (corresponding to dg elements). We have used both SOR and ILU before for our momentum equation, their performance usually didn't differ much (depending on problem type), sor overall gave a more robust performance which is why it's our default now. Not sure what the best choice is in terms of parallel scalability to be honest. Memory usage is a big concern, dg gives a lot of degrees of freedom (most people just use it explicitly in fact), so we have a huge matrix. > > It would not be terribly difficult to add relaxation for BAIJ; it > has to be done one block size at a time. What block size do you need, > are you interested in writing it? Typically 3 (triangles in 2 dimensions) or 4 (tets in 3D). Wouldn't mind having a look at implementing, will do some experiments with ILU first though. Thanks Stephan > > > Barry > > On Oct 27, 2009, at 3:28 PM, Stephan Kramer wrote: > >> Hi all, >> >> I'm slightly confused by the fact that if I use my BAIJ matrix in >> parallel and select the sor preconditioner. it complains about mat- >>> ops->relax and mat->ops->pbrelax not being defined. Also I can't >> find a MatRelax_MPIBAIJ in src/mat/impls/baij/mpi/mpibaij.c as I >> would have expected. Is there no sor type relaxation for parallel >> block matrices at all? >> >> Cheers >> Stephan > > -- Stephan Kramer Applied Modelling and Computation Group, Department of Earth Science and Engineering, Imperial College London From s.kramer at imperial.ac.uk Tue Oct 27 16:00:36 2009 From: s.kramer at imperial.ac.uk (Stephan Kramer) Date: Tue, 27 Oct 2009 21:00:36 +0000 Subject: MPIBAIJ and sor In-Reply-To: <31D3563A-AAF1-400E-AAB3-B21CE1FC72E9@mcs.anl.gov> References: <4AE75800.8010700@imperial.ac.uk> <31D3563A-AAF1-400E-AAB3-B21CE1FC72E9@mcs.anl.gov> Message-ID: <4AE75F74.6060806@imperial.ac.uk> Barry Smith wrote: > Stephan, > > There is no MatRelax_ for BAIJ matrices and it is only written for > SBAIJ (symmetric storage) for block size 1. > Ah ok > Most people end up using ILU(0) or ICC(0) for relaxing with block > matrices. It converges much better (generally) and takes the same > amount of time. The only drawback is that it requires another copy of > the matrix (uses more memory) for the "factored" version. Most people > are time constrained instead of memory constrained so this is not an > issue. Will give ILU a try then. I'm using this to solve an implicit dg momentum equation in an ocean model: so basically advection and diffusion terms (tensor form) and some coupling between the momentum components due to coriolis. In serial I found a significant improvement in convergence using the block structure (corresponding to dg elements). We have used both SOR and ILU before for our momentum equation, their performance usually didn't differ much (depending on problem type), sor overall gave a more robust performance which is why it's our default now. Not sure what the best choice is in terms of parallel scalability to be honest. Memory usage is a big concern, dg gives a lot of degrees of freedom (most people just use it explicitly in fact), so we have a huge matrix. > > It would not be terribly difficult to add relaxation for BAIJ; it > has to be done one block size at a time. What block size do you need, > are you interested in writing it? Typically 3 (triangles in 2 dimensions) or 4 (tets in 3D). Wouldn't mind having a look at implementing, will do some experiments with ILU first though. Thanks Stephan > > > Barry > > On Oct 27, 2009, at 3:28 PM, Stephan Kramer wrote: > >> Hi all, >> >> I'm slightly confused by the fact that if I use my BAIJ matrix in >> parallel and select the sor preconditioner. it complains about mat- >>> ops->relax and mat->ops->pbrelax not being defined. Also I can't >> find a MatRelax_MPIBAIJ in src/mat/impls/baij/mpi/mpibaij.c as I >> would have expected. Is there no sor type relaxation for parallel >> block matrices at all? >> >> Cheers >> Stephan > > -- Stephan Kramer Applied Modelling and Computation Group, Department of Earth Science and Engineering, Imperial College London From bsmith at mcs.anl.gov Tue Oct 27 16:03:46 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 27 Oct 2009 16:03:46 -0500 Subject: MPIBAIJ and sor In-Reply-To: <4AE75F74.6060806@imperial.ac.uk> References: <4AE75800.8010700@imperial.ac.uk> <31D3563A-AAF1-400E-AAB3-B21CE1FC72E9@mcs.anl.gov> <4AE75F74.6060806@imperial.ac.uk> Message-ID: As you will already know. Jed corrected me, I was wrong and there is a relax for BAIJ. Barry On Oct 27, 2009, at 4:00 PM, Stephan Kramer wrote: > Barry Smith wrote: >> Stephan, >> There is no MatRelax_ for BAIJ matrices and it is only written >> for SBAIJ (symmetric storage) for block size 1. > > Ah ok > >> Most people end up using ILU(0) or ICC(0) for relaxing with >> block matrices. It converges much better (generally) and takes the >> same amount of time. The only drawback is that it requires another >> copy of the matrix (uses more memory) for the "factored" version. >> Most people are time constrained instead of memory constrained so >> this is not an issue. > > Will give ILU a try then. I'm using this to solve an implicit dg > momentum > equation in an ocean model: so basically advection and diffusion terms > (tensor form) and some coupling between the momentum components due > to coriolis. In serial I found a significant improvement in > convergence using the block structure (corresponding to dg > elements). We have used both SOR and ILU before for our momentum > equation, their performance usually didn't differ > much (depending on problem type), sor overall gave a more robust > performance which is why it's our default now. Not sure what the best > choice is in terms of parallel scalability to be honest. Memory > usage is a > big concern, dg gives a lot of degrees of freedom (most people just > use it > explicitly in fact), so we have a huge matrix. > >> It would not be terribly difficult to add relaxation for BAIJ; >> it has to be done one block size at a time. What block size do you >> need, are you interested in writing it? > > Typically 3 (triangles in 2 dimensions) or 4 (tets in 3D). Wouldn't > mind having a look at implementing, will do some experiments with > ILU first though. > > Thanks > Stephan > >> Barry >> On Oct 27, 2009, at 3:28 PM, Stephan Kramer wrote: >>> Hi all, >>> >>> I'm slightly confused by the fact that if I use my BAIJ matrix in >>> parallel and select the sor preconditioner. it complains about mat- >>>> ops->relax and mat->ops->pbrelax not being defined. Also I can't >>> find a MatRelax_MPIBAIJ in src/mat/impls/baij/mpi/mpibaij.c as I >>> would have expected. Is there no sor type relaxation for parallel >>> block matrices at all? >>> >>> Cheers >>> Stephan > > > -- > Stephan Kramer > Applied Modelling and Computation Group, > Department of Earth Science and Engineering, > Imperial College London From bsmith at mcs.anl.gov Tue Oct 27 16:04:24 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 27 Oct 2009 16:04:24 -0500 Subject: MPIBAIJ and sor In-Reply-To: <4AE75A8E.6040606@59A2.org> References: <4AE75800.8010700@imperial.ac.uk> <4AE75A8E.6040606@59A2.org> Message-ID: I completely forgot about the MatPBRelax_ routines. MatRelax() which is used by PCSOR automatically tries MatRelax_xxx() for the matrix and then MatPBRelax_xxx() if the MatRelax_xxx() does not exist. MatRelax_MPIAIJ() (and MatRelax_MPISBAIJ()) handles the block Jacobi and relax on each block automatically and allows for any number of local smoothing before updating the ghost points. If you need MatPBRelax_SeqBAIJ_N() for N > 7 that is very easily added by taking the 7 case and extending it. Barry This discussion brings up a few issues 1) having the MatRelax() and MatPBRelax() both is a bit confusing. As Jed points out it is possible to write a MatRelax() for BAIJ, but that is almost always silly since essentially the same number of floating point operations gives you the better point block relaxation. I originally thought we might support all point-block and point relaxation for all matrix types and hence have both the MatRelax() and MatPBRelax(). Perhaps that was unneededly general and we should merge them. Then I wouldn't have searched for MatRelax_SeqBAIJ and given you the wrong answer initially. 2) Using SOR as the name of the preconditioner but relax as the name of the matrix operation is questionable. Better to use the same name everywhere? 3) One could very easily modify the MatRelax_MPIAIJ() to get a MatRelax_MPIBAIJ() and we should do that (and test it). But as Jed points out it is redundant (as is MatRelax_MPIAIJ()). On Oct 27, 2009, at 3:39 PM, Jed Brown wrote: > Stephan Kramer wrote: >> I'm slightly confused by the fact that if I use my BAIJ matrix in >> parallel and select the sor preconditioner. it complains about >> mat->ops->relax and mat->ops->pbrelax not being defined. > > There is no parallel SOR, it would make the algorithm nearly serial. > Use -sub_pc_type sor with -ksp_type asm or bjacobi. > >> Also I can't find a MatRelax_MPIBAIJ in >> src/mat/impls/baij/mpi/mpibaij.c as I would have expected. Is there >> no >> sor type relaxation for parallel block matrices at all? > > Since the small dense blocks are available at once, it would be > silly to > relax each component separately (your CPU spends most of its time > waiting on memory). Instead, BAIJ relaxes all components > simultaneously > by solving with these small blocks. This is "point-block" relaxation, > look for MatPBRelax_SeqBAIJ_N for N in 1..7. > > Jed > From s.kramer at imperial.ac.uk Tue Oct 27 16:09:16 2009 From: s.kramer at imperial.ac.uk (Stephan Kramer) Date: Tue, 27 Oct 2009 21:09:16 +0000 Subject: MPIBAIJ and sor In-Reply-To: <4AE75A8E.6040606@59A2.org> References: <4AE75800.8010700@imperial.ac.uk> <4AE75A8E.6040606@59A2.org> Message-ID: <4AE7617C.2000501@imperial.ac.uk> Jed Brown wrote: > Stephan Kramer wrote: >> I'm slightly confused by the fact that if I use my BAIJ matrix in >> parallel and select the sor preconditioner. it complains about >> mat->ops->relax and mat->ops->pbrelax not being defined. > > There is no parallel SOR, it would make the algorithm nearly serial. > Use -sub_pc_type sor with -ksp_type asm or bjacobi. > Ah I see. Yes, what I was looking for, was the equivalent of what happens with MPIAIJ if you select PCSOR and set SOR_LOCAL_SYMMETRIC_SWEEP (but then with point block relaxation) Thanks, that makes sense... >> Also I can't find a MatRelax_MPIBAIJ in >> src/mat/impls/baij/mpi/mpibaij.c as I would have expected. Is there no >> sor type relaxation for parallel block matrices at all? > > Since the small dense blocks are available at once, it would be silly to > relax each component separately (your CPU spends most of its time > waiting on memory). Instead, BAIJ relaxes all components simultaneously > by solving with these small blocks. This is "point-block" relaxation, > look for MatPBRelax_SeqBAIJ_N for N in 1..7. Yes, sorry I meant MatPBRelax_MPIBAIJ indeed > > Jed > -- Stephan Kramer Applied Modelling and Computation Group, Department of Earth Science and Engineering, Imperial College London From Andreas.Grassl at student.uibk.ac.at Tue Oct 27 17:16:54 2009 From: Andreas.Grassl at student.uibk.ac.at (Andreas Grassl) Date: Tue, 27 Oct 2009 23:16:54 +0100 Subject: [petsc-users] Problems with the intel compilers In-Reply-To: <4AE752BA.7030404@itis.ethz.ch> References: <4AE752BA.7030404@itis.ethz.ch> Message-ID: <4AE77156.6070406@student.uibk.ac.at> Hi Dominik, Dominik Szczerba wrote: > I have noticed already a longer while ago: setting up PETSc with the > Intel compilers (--with-cc=icc --with-cxx=icpc) takes an order of > magnitude longer than with the native GNU. The step taking most of the > time is configuring mpich. I have tried to configure mpich separately > and indeed, with gnu it is a breeze, with intel 10x slower. In both > cases, linux both 32 and 64 bit, Ubuntu 9.04 and debian testing. Intel > compilers 10.x and 11.x. > > I just wanted to ask opinion if anybody has similar observations and/or > finds using intel worthwhile at all. On itanium machines intel compilers produce much faster code compared to gnu-compilers. On other machines I did not notice a big speed difference, but I think intel compilers produce slightly faster code as well. Cheers, ando -- /"\ \ / ASCII Ribbon X against HTML email / \ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 315 bytes Desc: OpenPGP digital signature URL: From dominik at itis.ethz.ch Tue Oct 27 17:54:55 2009 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Tue, 27 Oct 2009 23:54:55 +0100 Subject: [petsc-users] Problems with the intel compilers In-Reply-To: <4AE77156.6070406@student.uibk.ac.at> References: <4AE752BA.7030404@itis.ethz.ch> <4AE77156.6070406@student.uibk.ac.at> Message-ID: <4AE77A3F.6040505@itis.ethz.ch> Thanks Ando, I know of the slight speed advantage in some cases on the Intel side, but the question was more why the compilation takes so much longer? Dominik Andreas Grassl wrote: > Hi Dominik, > > Dominik Szczerba wrote: >> I have noticed already a longer while ago: setting up PETSc with the >> Intel compilers (--with-cc=icc --with-cxx=icpc) takes an order of >> magnitude longer than with the native GNU. The step taking most of the >> time is configuring mpich. I have tried to configure mpich separately >> and indeed, with gnu it is a breeze, with intel 10x slower. In both >> cases, linux both 32 and 64 bit, Ubuntu 9.04 and debian testing. Intel >> compilers 10.x and 11.x. >> >> I just wanted to ask opinion if anybody has similar observations and/or >> finds using intel worthwhile at all. > > On itanium machines intel compilers produce much faster code compared to > gnu-compilers. On other machines I did not notice a big speed difference, but I > think intel compilers produce slightly faster code as well. > > Cheers, > > ando > From likask at civil.gla.ac.uk Tue Oct 27 18:19:16 2009 From: likask at civil.gla.ac.uk (Lukasz Kaczmarczyk) Date: Tue, 27 Oct 2009 23:19:16 +0000 Subject: KSPMonitorTrueResidualNorm and SNES Message-ID: Hello, I have noticed minor problem, potentially bug. For nonlinear analysis with SNES and option -ksp_monitor_true_residual I get 0 Norm Res_strain 9.36e-12 Norm Res_stress 1.43e-05 Internal its 1 0 SNES Function norm 1.428489691212e-05 0 KSP preconditioned resid norm 7.213823259362e-04 true resid norm 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.213823259362e-04 true resid norm 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.903423697449e-04 true resid norm 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 1 KSP preconditioned resid norm 4.903423697449e-04 true resid norm 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 It look that function KSPMonitorTrueResidualNorm is evaluated twice, whereas for default KSP monitor only once. I use petsc-3.0.0-p7. Kind regards, Lukasz From sdettrick at gmail.com Tue Oct 27 18:22:52 2009 From: sdettrick at gmail.com (Sean Dettrick) Date: Tue, 27 Oct 2009 16:22:52 -0700 Subject: [petsc-users] Problems with the intel compilers In-Reply-To: <4AE77A3F.6040505@itis.ethz.ch> References: <4AE752BA.7030404@itis.ethz.ch> <4AE77156.6070406@student.uibk.ac.at> <4AE77A3F.6040505@itis.ethz.ch> Message-ID: Intel optimization is slow, and is turned on by default. If you turn off intel optimization completely with -O0, then you'll probably find compilation speeds up to be similar to gnu. Is the increased compilation time of intel worth it? We find intel programs are usually faster than gnu ones. And the intel debugger idb is in many ways more powerful than gdb. But on the other hand we have a fortran based CG solver where gfortran compiled code is much faster than the intel one. Cheers, Sean On Oct 27, 2009, at 3:54 PM, Dominik Szczerba wrote: > Thanks Ando, I know of the slight speed advantage in some cases on > the Intel side, but the question was more why the compilation takes > so much longer? > > Dominik > > Andreas Grassl wrote: >> Hi Dominik, >> Dominik Szczerba wrote: >>> I have noticed already a longer while ago: setting up PETSc with the >>> Intel compilers (--with-cc=icc --with-cxx=icpc) takes an order of >>> magnitude longer than with the native GNU. The step taking most of >>> the >>> time is configuring mpich. I have tried to configure mpich >>> separately >>> and indeed, with gnu it is a breeze, with intel 10x slower. In both >>> cases, linux both 32 and 64 bit, Ubuntu 9.04 and debian testing. >>> Intel >>> compilers 10.x and 11.x. >>> >>> I just wanted to ask opinion if anybody has similar observations >>> and/or >>> finds using intel worthwhile at all. >> On itanium machines intel compilers produce much faster code >> compared to >> gnu-compilers. On other machines I did not notice a big speed >> difference, but I >> think intel compilers produce slightly faster code as well. >> Cheers, >> ando > From knepley at gmail.com Tue Oct 27 18:23:53 2009 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 27 Oct 2009 18:23:53 -0500 Subject: KSPMonitorTrueResidualNorm and SNES In-Reply-To: References: Message-ID: It is possible to add the monitor twice. Are you sure this did not happen? Thanks, Matt On Tue, Oct 27, 2009 at 6:19 PM, Lukasz Kaczmarczyk wrote: > Hello, > > I have noticed minor problem, potentially bug. For nonlinear analysis with > SNES and option -ksp_monitor_true_residual I get > 0 Norm Res_strain 9.36e-12 Norm Res_stress 1.43e-05 Internal its 1 > 0 SNES Function norm 1.428489691212e-05 > 0 KSP preconditioned resid norm 7.213823259362e-04 true resid norm > 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 7.213823259362e-04 true resid norm > 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.903423697449e-04 true resid norm > 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 > 1 KSP preconditioned resid norm 4.903423697449e-04 true resid norm > 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 > > It look that function KSPMonitorTrueResidualNorm is evaluated twice, > whereas for default KSP monitor only once. I use petsc-3.0.0-p7. > > Kind regards, > Lukasz > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Oct 27 20:30:15 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 27 Oct 2009 20:30:15 -0500 Subject: KSPMonitorTrueResidualNorm and SNES In-Reply-To: References: Message-ID: What prehistoric version of PETSc are you using? Barry On Oct 27, 2009, at 6:19 PM, Lukasz Kaczmarczyk wrote: > Hello, > > I have noticed minor problem, potentially bug. For nonlinear > analysis with SNES and option -ksp_monitor_true_residual I get > 0 Norm Res_strain 9.36e-12 Norm Res_stress 1.43e-05 Internal its 1 > 0 SNES Function norm 1.428489691212e-05 > 0 KSP preconditioned resid norm 7.213823259362e-04 true resid > norm 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 7.213823259362e-04 true resid > norm 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.903423697449e-04 true resid > norm 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 > 1 KSP preconditioned resid norm 4.903423697449e-04 true resid > norm 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 > > It look that function KSPMonitorTrueResidualNorm is evaluated twice, > whereas for default KSP monitor only once. I use petsc-3.0.0-p7. > > Kind regards, > Lukasz From likask at civil.gla.ac.uk Wed Oct 28 02:28:10 2009 From: likask at civil.gla.ac.uk (Lukasz Kaczmarczyk) Date: Wed, 28 Oct 2009 07:28:10 +0000 Subject: KSPMonitorTrueResidualNorm and SNES In-Reply-To: References: Message-ID: <7C4CBAC2-0E3F-4A1B-AC0B-90B15C34EDDF@civil.gla.ac.uk> Hi Barry I am using petsc-3.0.0-p7. Regards, Lukasz On 28 Oct 2009, at 01:30, Barry Smith wrote: > > What prehistoric version of PETSc are you using? > > > Barry > On Oct 27, 2009, at 6:19 PM, Lukasz Kaczmarczyk wrote: > >> Hello, >> >> I have noticed minor problem, potentially bug. For nonlinear >> analysis with SNES and option -ksp_monitor_true_residual I get >> 0 Norm Res_strain 9.36e-12 Norm Res_stress 1.43e-05 Internal its 1 >> 0 SNES Function norm 1.428489691212e-05 >> 0 KSP preconditioned resid norm 7.213823259362e-04 true resid >> norm 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 7.213823259362e-04 true resid >> norm 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 4.903423697449e-04 true resid >> norm 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 >> 1 KSP preconditioned resid norm 4.903423697449e-04 true resid >> norm 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 >> >> It look that function KSPMonitorTrueResidualNorm is evaluated >> twice, whereas for default KSP monitor only once. I use petsc-3.0.0- >> p7. >> >> Kind regards, >> Lukasz > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From likask at civil.gla.ac.uk Wed Oct 28 02:28:10 2009 From: likask at civil.gla.ac.uk (Lukasz Kaczmarczyk) Date: Wed, 28 Oct 2009 07:28:10 +0000 Subject: KSPMonitorTrueResidualNorm and SNES In-Reply-To: References: Message-ID: <7C4CBAC2-0E3F-4A1B-AC0B-90B15C34EDDF@civil.gla.ac.uk> Hi Barry I am using petsc-3.0.0-p7. Regards, Lukasz On 28 Oct 2009, at 01:30, Barry Smith wrote: > > What prehistoric version of PETSc are you using? > > > Barry > On Oct 27, 2009, at 6:19 PM, Lukasz Kaczmarczyk wrote: > >> Hello, >> >> I have noticed minor problem, potentially bug. For nonlinear >> analysis with SNES and option -ksp_monitor_true_residual I get >> 0 Norm Res_strain 9.36e-12 Norm Res_stress 1.43e-05 Internal its 1 >> 0 SNES Function norm 1.428489691212e-05 >> 0 KSP preconditioned resid norm 7.213823259362e-04 true resid >> norm 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 7.213823259362e-04 true resid >> norm 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 4.903423697449e-04 true resid >> norm 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 >> 1 KSP preconditioned resid norm 4.903423697449e-04 true resid >> norm 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 >> >> It look that function KSPMonitorTrueResidualNorm is evaluated >> twice, whereas for default KSP monitor only once. I use petsc-3.0.0- >> p7. >> >> Kind regards, >> Lukasz > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From likask at civil.gla.ac.uk Wed Oct 28 02:39:53 2009 From: likask at civil.gla.ac.uk (Lukasz Kaczmarczyk) Date: Wed, 28 Oct 2009 07:39:53 +0000 Subject: KSPMonitorTrueResidualNorm and SNES In-Reply-To: References: Message-ID: Hi Matt I add monitor from options only. Pleas note that for default KSP monitor a norm of preconditioned residual is printed only once. Problem exists when I monitor true residuals. Regards, Lukasz On 27 Oct 2009, at 23:23, Matthew Knepley wrote: > It is possible to add the monitor twice. Are you sure this did not > happen? > > Thanks, > > Matt > > On Tue, Oct 27, 2009 at 6:19 PM, Lukasz Kaczmarczyk > wrote: > Hello, > > I have noticed minor problem, potentially bug. For nonlinear > analysis with SNES and option -ksp_monitor_true_residual I get > 0 Norm Res_strain 9.36e-12 Norm Res_stress 1.43e-05 Internal its 1 > 0 SNES Function norm 1.428489691212e-05 > 0 KSP preconditioned resid norm 7.213823259362e-04 true resid > norm 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 7.213823259362e-04 true resid > norm 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.903423697449e-04 true resid > norm 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 > 1 KSP preconditioned resid norm 4.903423697449e-04 true resid > norm 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 > > It look that function KSPMonitorTrueResidualNorm is evaluated twice, > whereas for default KSP monitor only once. I use petsc-3.0.0-p7. > > Kind regards, > Lukasz > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From recrusader at gmail.com Wed Oct 28 10:21:34 2009 From: recrusader at gmail.com (Yujie) Date: Wed, 28 Oct 2009 10:21:34 -0500 Subject: Does PETSc have some functions like A.*B, A./B in Matlab? Message-ID: <7ff0ee010910280821g3e695bc8rf7fe56cb12775b46@mail.gmail.com> Dear PETSc Developers, Does PETSc provide some functions, like A.*B, A./B in Matlab? Thanks a lot. Regards, Yujie -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhyshr at mcs.anl.gov Wed Oct 28 11:08:32 2009 From: abhyshr at mcs.anl.gov (abhyshr at mcs.anl.gov) Date: Wed, 28 Oct 2009 11:08:32 -0500 (CDT) Subject: Does PETSc have some functions like A.*B, A./B in Matlab? In-Reply-To: <5070102.40691256745929847.JavaMail.root@zimbra> Message-ID: <27590708.40941256746112955.JavaMail.root@zimbra> If A and B are vectors, then you can use VecPointwiseMult for A.*B and VecPointwiseDivide for A./B. For matrices,pointwise multiplication and division functions are not provided in Petsc. One possible way to do A.*B and A./B for matrices is to loop through the columns of the matrices,call MatGetColumnVector for each column of matrix A,B and then call VecPointwiseMult/VecPointwiseDivide. Shri ----- Original Message ----- From: "Yujie" To: "PETSc users list" Sent: Wednesday, October 28, 2009 10:21:34 AM GMT -06:00 US/Canada Central Subject: Does PETSc have some functions like A.*B, A./B in Matlab? Dear PETSc Developers, Does PETSc provide some functions, like A.*B, A./B in Matlab? Thanks a lot. Regards, Yujie From jed at 59A2.org Wed Oct 28 11:39:44 2009 From: jed at 59A2.org (Jed Brown) Date: Wed, 28 Oct 2009 17:39:44 +0100 Subject: Does PETSc have some functions like A.*B, A./B in Matlab? In-Reply-To: <27590708.40941256746112955.JavaMail.root@zimbra> References: <27590708.40941256746112955.JavaMail.root@zimbra> Message-ID: <4AE873D0.9030401@59A2.org> abhyshr at mcs.anl.gov wrote: > If A and B are vectors, then you can use VecPointwiseMult for A.*B and VecPointwiseDivide for A./B. > For matrices,pointwise multiplication and division functions are not provided in Petsc. One possible way to do > A.*B and A./B for matrices is to loop through the columns of the matrices,call MatGetColumnVector for each column of matrix A,B > and then call VecPointwiseMult/VecPointwiseDivide. This would be very inefficient because it produces dense output and PETSc matrices are row-aligned. I would suggest using MatGetRow which is much faster and preserves sparsity. Jed > > Shri > > ----- Original Message ----- > From: "Yujie" > To: "PETSc users list" > Sent: Wednesday, October 28, 2009 10:21:34 AM GMT -06:00 US/Canada Central > Subject: Does PETSc have some functions like A.*B, A./B in Matlab? > > Dear PETSc Developers, > > Does PETSc provide some functions, like A.*B, A./B in Matlab? Thanks a lot. > > Regards, > Yujie -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 261 bytes Desc: OpenPGP digital signature URL: From bsmith at mcs.anl.gov Wed Oct 28 11:40:30 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 28 Oct 2009 11:40:30 -0500 Subject: KSPMonitorTrueResidualNorm and SNES In-Reply-To: References: Message-ID: You are calling SNESSetFromOptions() or KSPSetFromOptions() twice. If you call it once the problem won't appear. There is an extremely ugly hack that is implemented for the default monitor, but not other monitors, to prevent this double printing. Whoever wrote this ugly hack should have done it properly. Barry On Oct 28, 2009, at 2:39 AM, Lukasz Kaczmarczyk wrote: > Hi Matt > > I add monitor from options only. > > Pleas note that for default KSP monitor a norm of preconditioned > residual is printed only once. Problem exists when I monitor true > residuals. > > Regards, > Lukasz > > > > On 27 Oct 2009, at 23:23, Matthew Knepley wrote: > >> It is possible to add the monitor twice. Are you sure this did not >> happen? >> >> Thanks, >> >> Matt >> >> On Tue, Oct 27, 2009 at 6:19 PM, Lukasz Kaczmarczyk > > wrote: >> Hello, >> >> I have noticed minor problem, potentially bug. For nonlinear >> analysis with SNES and option -ksp_monitor_true_residual I get >> 0 Norm Res_strain 9.36e-12 Norm Res_stress 1.43e-05 Internal its 1 >> 0 SNES Function norm 1.428489691212e-05 >> 0 KSP preconditioned resid norm 7.213823259362e-04 true resid >> norm 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 7.213823259362e-04 true resid >> norm 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 4.903423697449e-04 true resid >> norm 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 >> 1 KSP preconditioned resid norm 4.903423697449e-04 true resid >> norm 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 >> >> It look that function KSPMonitorTrueResidualNorm is evaluated >> twice, whereas for default KSP monitor only once. I use petsc-3.0.0- >> p7. >> >> Kind regards, >> Lukasz >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener From knepley at gmail.com Wed Oct 28 12:46:13 2009 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 28 Oct 2009 12:46:13 -0500 Subject: Does PETSc have some functions like A.*B, A./B in Matlab? In-Reply-To: <4AE873D0.9030401@59A2.org> References: <27590708.40941256746112955.JavaMail.root@zimbra> <4AE873D0.9030401@59A2.org> Message-ID: There is MatAXPY(). Matt On Wed, Oct 28, 2009 at 11:39 AM, Jed Brown wrote: > abhyshr at mcs.anl.gov wrote: > > If A and B are vectors, then you can use VecPointwiseMult for A.*B and > VecPointwiseDivide for A./B. > > For matrices,pointwise multiplication and division functions are not > provided in Petsc. One possible way to do > > A.*B and A./B for matrices is to loop through the columns of the > matrices,call MatGetColumnVector for each column of matrix A,B > > and then call VecPointwiseMult/VecPointwiseDivide. > > This would be very inefficient because it produces dense output and > PETSc matrices are row-aligned. I would suggest using MatGetRow which > is much faster and preserves sparsity. > > Jed > > > > > Shri > > > > ----- Original Message ----- > > From: "Yujie" > > To: "PETSc users list" > > Sent: Wednesday, October 28, 2009 10:21:34 AM GMT -06:00 US/Canada > Central > > Subject: Does PETSc have some functions like A.*B, A./B in Matlab? > > > > Dear PETSc Developers, > > > > Does PETSc provide some functions, like A.*B, A./B in Matlab? Thanks a > lot. > > > > Regards, > > Yujie > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From vasoula_emp at hotmail.com Wed Oct 28 13:41:51 2009 From: vasoula_emp at hotmail.com (VASIA KALAVRI) Date: Wed, 28 Oct 2009 20:41:51 +0200 Subject: Parallel Cholesky solve In-Reply-To: <16AE532E-0A1F-4CFD-895F-AF17B37C50D4@mcs.anl.gov> References: Message-ID: Thanks for the immediate response. The thing is when I use MATMPISBAIJ format I get the message: "Lower triangular value cannot be set for sbaij format" but I want to set these values. Vasia. _________________________________________________________________ Keep your friends updated?even when you?re not signed in. http://www.microsoft.com/middleeast/windows/windowslive/see-it-in-action/social-network-basics.aspx?ocid=PID23461::T:WLMTAGL:ON:WL:en-xm:SI_SB_5:092010 -------------- next part -------------- An HTML attachment was scrubbed... URL: From likask at civil.gla.ac.uk Wed Oct 28 13:51:18 2009 From: likask at civil.gla.ac.uk (Lukasz Kaczmarczyk) Date: Wed, 28 Oct 2009 18:51:18 +0000 Subject: KSPMonitorTrueResidualNorm and SNES In-Reply-To: References: Message-ID: <49BC9AED-2AF8-4A68-BD99-C33E12B13978@civil.gla.ac.uk> Thank you for answer. What you described bellow is my case. I initially solve linear problem using KSP only. To set up solver the KSPSetFromOptions is used. Having initial data I solve nonlinear time dependent problem using SNES set up by SNESSetFromOptions. I will switch off monitor after first linear solution phase using KSPMonitorCancel. SNESSetFromOptions should set up it once again. Kind regards Lukasz On 28 Oct 2009, at 16:40, Barry Smith wrote: > > You are calling SNESSetFromOptions() or KSPSetFromOptions() twice. > If you call it once the problem won't appear. > > There is an extremely ugly hack that is implemented for the > default monitor, but not other monitors, to prevent this double > printing. Whoever wrote this ugly hack should have done it properly. > > Barry > > On Oct 28, 2009, at 2:39 AM, Lukasz Kaczmarczyk wrote: > >> Hi Matt >> >> I add monitor from options only. >> >> Pleas note that for default KSP monitor a norm of preconditioned >> residual is printed only once. Problem exists when I monitor true >> residuals. >> >> Regards, >> Lukasz >> >> >> >> On 27 Oct 2009, at 23:23, Matthew Knepley wrote: >> >>> It is possible to add the monitor twice. Are you sure this did not >>> happen? >>> >>> Thanks, >>> >>> Matt >>> >>> On Tue, Oct 27, 2009 at 6:19 PM, Lukasz Kaczmarczyk >> > wrote: >>> Hello, >>> >>> I have noticed minor problem, potentially bug. For nonlinear >>> analysis with SNES and option -ksp_monitor_true_residual I get >>> 0 Norm Res_strain 9.36e-12 Norm Res_stress 1.43e-05 Internal its 1 >>> 0 SNES Function norm 1.428489691212e-05 >>> 0 KSP preconditioned resid norm 7.213823259362e-04 true resid >>> norm 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 >>> 0 KSP preconditioned resid norm 7.213823259362e-04 true resid >>> norm 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 >>> 1 KSP preconditioned resid norm 4.903423697449e-04 true resid >>> norm 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 >>> 1 KSP preconditioned resid norm 4.903423697449e-04 true resid >>> norm 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 >>> >>> It look that function KSPMonitorTrueResidualNorm is evaluated >>> twice, whereas for default KSP monitor only once. I use >>> petsc-3.0.0-p7. >>> >>> Kind regards, >>> Lukasz >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to >>> which their experiments lead. >>> -- Norbert Wiener > > From Hung.V.Nguyen at usace.army.mil Wed Oct 28 14:12:45 2009 From: Hung.V.Nguyen at usace.army.mil (Nguyen, Hung V ERDC-ITL-MS) Date: Wed, 28 Oct 2009 14:12:45 -0500 Subject: Question about -log_summary option Message-ID: Hello, When running with -log_summary option, I got the information about Flops/sec as below. My question is whether Petsc does counting floating-point operations for the whole application codes or only part with petsc? Thanks, -hung u/hvnguyen/adh_dev/ADH_petsc_Anod/bin/adh on a cray-xt-p named nid00525 with 32 processors, by Unknown Thu Oct 8 02:59:49 2009 Using Petsc Release Version 3.0.0, Patch 2, Wed Jan 14 22:57:05 CST 2009 Max Max/Min Avg Total Time (sec): 4.609e+03 1.00002 4.609e+03 Objects: 4.770e+02 1.00000 4.770e+02 Flops: 1.691e+12 1.38465 1.524e+12 4.875e+13 Flops/sec: 3.669e+08 1.38464 3.305e+08 1.058e+10 MPI Messages: 8.104e+06 15.98588 3.103e+06 9.930e+07 MPI Message Lengths: 1.667e+10 12.62135 2.618e+03 2.600e+11 MPI Reductions: 2.008e+04 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops From jed at 59A2.org Wed Oct 28 14:15:45 2009 From: jed at 59A2.org (Jed Brown) Date: Wed, 28 Oct 2009 20:15:45 +0100 Subject: Question about -log_summary option In-Reply-To: References: Message-ID: <4AE89861.3040406@59A2.org> Nguyen, Hung V ERDC-ITL-MS wrote: > Hello, > > When running with -log_summary option, I got the information about Flops/sec > as below. My question is whether Petsc does counting floating-point > operations for the whole application codes or only part with petsc? PETSc doesn't magically know what your code is doing. You can register events and call PetscLogFlops() if you want to profile your own code. Jed -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 261 bytes Desc: OpenPGP digital signature URL: From hzhang at mcs.anl.gov Wed Oct 28 14:20:14 2009 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 28 Oct 2009 14:20:14 -0500 Subject: Parallel Cholesky solve In-Reply-To: References: Message-ID: <73234AEF-B085-482E-B56A-D147BEB79339@mcs.anl.gov> On Oct 28, 2009, at 1:41 PM, VASIA KALAVRI wrote: > Thanks for the immediate response. > The thing is when I use MATMPISBAIJ format I get the message: "Lower > triangular value cannot be set for sbaij format" > but I want to set these values. You can add option '-mat_ignore_lower_triangular' or call MatSetOption(mat,MAT_IGNORE_LOWER_TRIANGULAR,PETSC_TRUE) Hong > > Vasia. > > Keep your friends updated? even when you?re not signed in. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Hung.V.Nguyen at usace.army.mil Wed Oct 28 14:21:41 2009 From: Hung.V.Nguyen at usace.army.mil (Nguyen, Hung V ERDC-ITL-MS) Date: Wed, 28 Oct 2009 14:21:41 -0500 Subject: Question about -log_summary option In-Reply-To: <4AE89861.3040406@59A2.org> References: <4AE89861.3040406@59A2.org> Message-ID: Thank you. That's what I guess. I don't plan to profile my own codes. I am just wondering if I can use it or not. -hung -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Wednesday, October 28, 2009 2:16 PM To: petsc-users at mcs.anl.gov Subject: Re: Question about -log_summary option Nguyen, Hung V ERDC-ITL-MS wrote: > Hello, > > When running with -log_summary option, I got the information about > Flops/sec as below. My question is whether Petsc does counting > floating-point operations for the whole application codes or only part with petsc? PETSc doesn't magically know what your code is doing. You can register events and call PetscLogFlops() if you want to profile your own code. Jed From likask at civil.gla.ac.uk Wed Oct 28 15:32:42 2009 From: likask at civil.gla.ac.uk (Lukasz Kaczmarczyk) Date: Wed, 28 Oct 2009 20:32:42 +0000 Subject: KSPMonitorTrueResidualNorm and SNES In-Reply-To: <49BC9AED-2AF8-4A68-BD99-C33E12B13978@civil.gla.ac.uk> References: <49BC9AED-2AF8-4A68-BD99-C33E12B13978@civil.gla.ac.uk> Message-ID: <12DF3014-8B63-48DC-A11C-8A3BB057AF03@civil.gla.ac.uk> > I will switch off monitor after first linear solution phase using > KSPMonitorCancel. SNESSetFromOptions should set up it once again Unfortunately this is not working. > On 28 Oct 2009, at 16:40, Barry Smith wrote: > >> >> You are calling SNESSetFromOptions() or KSPSetFromOptions() twice. >> If you call it once the problem won't appear. >> >> There is an extremely ugly hack that is implemented for the >> default monitor, but not other monitors, to prevent this double >> printing. Whoever wrote this ugly hack should have done it properly. >> >> Barry >> >> On Oct 28, 2009, at 2:39 AM, Lukasz Kaczmarczyk wrote: >> >>> Hi Matt >>> >>> I add monitor from options only. >>> >>> Pleas note that for default KSP monitor a norm of preconditioned >>> residual is printed only once. Problem exists when I monitor true >>> residuals. >>> >>> Regards, >>> Lukasz >>> >>> >>> >>> On 27 Oct 2009, at 23:23, Matthew Knepley wrote: >>> >>>> It is possible to add the monitor twice. Are you sure this did >>>> not happen? >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> On Tue, Oct 27, 2009 at 6:19 PM, Lukasz Kaczmarczyk >>> > wrote: >>>> Hello, >>>> >>>> I have noticed minor problem, potentially bug. For nonlinear >>>> analysis with SNES and option -ksp_monitor_true_residual I get >>>> 0 Norm Res_strain 9.36e-12 Norm Res_stress 1.43e-05 Internal >>>> its 1 >>>> 0 SNES Function norm 1.428489691212e-05 >>>> 0 KSP preconditioned resid norm 7.213823259362e-04 true resid >>>> norm 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 >>>> 0 KSP preconditioned resid norm 7.213823259362e-04 true resid >>>> norm 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 >>>> 1 KSP preconditioned resid norm 4.903423697449e-04 true resid >>>> norm 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 >>>> 1 KSP preconditioned resid norm 4.903423697449e-04 true resid >>>> norm 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 >>>> >>>> It look that function KSPMonitorTrueResidualNorm is evaluated >>>> twice, whereas for default KSP monitor only once. I use >>>> petsc-3.0.0-p7. >>>> >>>> Kind regards, >>>> Lukasz >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to >>>> which their experiments lead. >>>> -- Norbert Wiener >> >> > Lukasz Kaczmarczyk Lecturer Department of Civil Engineering, University of Glasgow, GLASGOW, G12 8LT Tel: +44 141 3305348 email: likask at civil.gla.ac.uk web: http://www.civil.gla.ac.uk/~kaczmarczyk/ web: http://code.google.com/p/yaffems/ From bsmith at mcs.anl.gov Wed Oct 28 15:53:49 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 28 Oct 2009 15:53:49 -0500 Subject: Parallel Cholesky solve In-Reply-To: References: Message-ID: <70DFBC3E-8BE1-4481-8FC5-4D5FEC033A06@mcs.anl.gov> On Oct 28, 2009, at 1:41 PM, VASIA KALAVRI wrote: > Thanks for the immediate response. > The thing is when I use MATMPISBAIJ format I get the message: "Lower > triangular value cannot be set for sbaij format" > but I want to set these values. The SBAIJ interface only accepts the diagonal and upper diagonal entries. If you are computing the lower triangular entries (and not the upper) you can simply pass the columns as rows and the rows as columns. If you are providing both the lower and upper triangular you can either (1) do as Hong says and have it ignore the lower or (2) never set the lower. Barry > > Vasia. > > Keep your friends updated? even when you?re not signed in. From knepley at gmail.com Wed Oct 28 16:07:35 2009 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 28 Oct 2009 16:07:35 -0500 Subject: KSPMonitorTrueResidualNorm and SNES In-Reply-To: <12DF3014-8B63-48DC-A11C-8A3BB057AF03@civil.gla.ac.uk> References: <49BC9AED-2AF8-4A68-BD99-C33E12B13978@civil.gla.ac.uk> <12DF3014-8B63-48DC-A11C-8A3BB057AF03@civil.gla.ac.uk> Message-ID: On Wed, Oct 28, 2009 at 3:32 PM, Lukasz Kaczmarczyk wrote: > > I will switch off monitor after first linear solution phase using >> KSPMonitorCancel. SNESSetFromOptions should set up it once again >> > Unfortunately this is not working. > Can you reproduce this with an example, like SNES ex5? Matt > On 28 Oct 2009, at 16:40, Barry Smith wrote: >> >> >>> You are calling SNESSetFromOptions() or KSPSetFromOptions() twice. If >>> you call it once the problem won't appear. >>> >>> There is an extremely ugly hack that is implemented for the default >>> monitor, but not other monitors, to prevent this double printing. Whoever >>> wrote this ugly hack should have done it properly. >>> >>> Barry >>> >>> On Oct 28, 2009, at 2:39 AM, Lukasz Kaczmarczyk wrote: >>> >>> Hi Matt >>>> >>>> I add monitor from options only. >>>> >>>> Pleas note that for default KSP monitor a norm of preconditioned >>>> residual is printed only once. Problem exists when I monitor true residuals. >>>> >>>> Regards, >>>> Lukasz >>>> >>>> >>>> >>>> On 27 Oct 2009, at 23:23, Matthew Knepley wrote: >>>> >>>> It is possible to add the monitor twice. Are you sure this did not >>>>> happen? >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> On Tue, Oct 27, 2009 at 6:19 PM, Lukasz Kaczmarczyk < >>>>> likask at civil.gla.ac.uk> wrote: >>>>> Hello, >>>>> >>>>> I have noticed minor problem, potentially bug. For nonlinear analysis >>>>> with SNES and option -ksp_monitor_true_residual I get >>>>> 0 Norm Res_strain 9.36e-12 Norm Res_stress 1.43e-05 Internal its 1 >>>>> 0 SNES Function norm 1.428489691212e-05 >>>>> 0 KSP preconditioned resid norm 7.213823259362e-04 true resid norm >>>>> 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 >>>>> 0 KSP preconditioned resid norm 7.213823259362e-04 true resid norm >>>>> 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 >>>>> 1 KSP preconditioned resid norm 4.903423697449e-04 true resid norm >>>>> 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 >>>>> 1 KSP preconditioned resid norm 4.903423697449e-04 true resid norm >>>>> 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 >>>>> >>>>> It look that function KSPMonitorTrueResidualNorm is evaluated twice, >>>>> whereas for default KSP monitor only once. I use petsc-3.0.0-p7. >>>>> >>>>> Kind regards, >>>>> Lukasz >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>> >>> >> > Lukasz Kaczmarczyk > Lecturer > Department of Civil Engineering, > University of Glasgow, > GLASGOW, G12 8LT > Tel: +44 141 3305348 > email: likask at civil.gla.ac.uk > web: http://www.civil.gla.ac.uk/~kaczmarczyk/ > web: http://code.google.com/p/yaffems/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From likask at civil.gla.ac.uk Wed Oct 28 17:28:22 2009 From: likask at civil.gla.ac.uk (Lukasz Kaczmarczyk) Date: Wed, 28 Oct 2009 22:28:22 +0000 Subject: KSPMonitorTrueResidualNorm and SNES In-Reply-To: References: <49BC9AED-2AF8-4A68-BD99-C33E12B13978@civil.gla.ac.uk> <12DF3014-8B63-48DC-A11C-8A3BB057AF03@civil.gla.ac.uk> Message-ID: Hello No for example: ./ex5 -ksp_monitor_true_residual -snes_monitor This problem is not reproduced. My apologies for taking your time for stupid error, I called explicitly ierr = SNESGetKSP(snes,&ksp); CHKERRQ(ierr); ierr = KSPSetFromOptions(ksp); CHKERRQ(ierr); ierr = SNESSetFromOptions(snes);CHKERRQ(ierr); As an excuse I tell that reading manual I take impression that KSP should be set up independently, and it was ok until I start to look at true residual norm. I should not call KSPSetFromOptions(). Regards, Lukasz On 28 Oct 2009, at 21:07, Matthew Knepley wrote: > On Wed, Oct 28, 2009 at 3:32 PM, Lukasz Kaczmarczyk > wrote: > > I will switch off monitor after first linear solution phase using > KSPMonitorCancel. SNESSetFromOptions should set up it once again > Unfortunately this is not working. > > Can you reproduce this with an example, like SNES ex5? > > Matt > > On 28 Oct 2009, at 16:40, Barry Smith wrote: > > > You are calling SNESSetFromOptions() or KSPSetFromOptions() twice. > If you call it once the problem won't appear. > > There is an extremely ugly hack that is implemented for the default > monitor, but not other monitors, to prevent this double printing. > Whoever wrote this ugly hack should have done it properly. > > Barry > > On Oct 28, 2009, at 2:39 AM, Lukasz Kaczmarczyk wrote: > > Hi Matt > > I add monitor from options only. > > Pleas note that for default KSP monitor a norm of preconditioned > residual is printed only once. Problem exists when I monitor true > residuals. > > Regards, > Lukasz > > On 27 Oct 2009, at 23:23, Matthew Knepley wrote: > > It is possible to add the monitor twice. Are you sure this did not > happen? > > Thanks, > > Matt > > On Tue, Oct 27, 2009 at 6:19 PM, Lukasz Kaczmarczyk > wrote: > Hello, > > I have noticed minor problem, potentially bug. For nonlinear > analysis with SNES and option -ksp_monitor_true_residual I get > 0 Norm Res_strain 9.36e-12 Norm Res_stress 1.43e-05 Internal its 1 > 0 SNES Function norm 1.428489691212e-05 > 0 KSP preconditioned resid norm 7.213823259362e-04 true resid norm > 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 7.213823259362e-04 true resid norm > 1.428489691212e-05 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.903423697449e-04 true resid norm > 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 > 1 KSP preconditioned resid norm 4.903423697449e-04 true resid norm > 9.339595309685e-06 ||Ae||/||Ax|| 6.538090801174e-01 > > It look that function KSPMonitorTrueResidualNorm is evaluated twice, > whereas for default KSP monitor only once. I use petsc-3.0.0-p7. > > Kind regards, > Lukasz > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener From vasoula_emp at hotmail.com Thu Oct 29 02:27:35 2009 From: vasoula_emp at hotmail.com (VASIA KALAVRI) Date: Thu, 29 Oct 2009 09:27:35 +0200 Subject: Parallel Cholesky solve In-Reply-To: <70DFBC3E-8BE1-4481-8FC5-4D5FEC033A06@mcs.anl.gov> References: Message-ID: Thanks a lot! Vasia. > From: bsmith at mcs.anl.gov > To: petsc-users at mcs.anl.gov > Subject: Re: Parallel Cholesky solve > Date: Wed, 28 Oct 2009 15:53:49 -0500 > > > On Oct 28, 2009, at 1:41 PM, VASIA KALAVRI wrote: > > > Thanks for the immediate response. > > The thing is when I use MATMPISBAIJ format I get the message: "Lower > > triangular value cannot be set for sbaij format" > > but I want to set these values. > > The SBAIJ interface only accepts the diagonal and upper diagonal > entries. If you are computing the lower triangular entries (and not > the upper) you can simply pass the columns as rows and the rows as > columns. If you are providing both the lower and upper triangular you > can either (1) do as Hong says and have it ignore the lower or (2) > never set the lower. > > Barry > > > > > Vasia. > > > > Keep your friends updated? even when you?re not signed in. > _________________________________________________________________ Windows Live: Friends get your Flickr, Yelp, and Digg updates when they e-mail you. http://www.microsoft.com/middleeast/windows/windowslive/see-it-in-action/social-network-basics.aspx?ocid=PID23461::T:WLMTAGL:ON:WL:en-xm:SI_SB_3:092010 -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhyshr at mcs.anl.gov Thu Oct 29 11:53:07 2009 From: abhyshr at mcs.anl.gov (abhyshr at mcs.anl.gov) Date: Thu, 29 Oct 2009 11:53:07 -0500 (CDT) Subject: Does PETSc have some functions like A.*B, A./B in Matlab? In-Reply-To: <25365497.74771256834912227.JavaMail.root@zimbra> Message-ID: <3808503.74861256835187098.JavaMail.root@zimbra> Jed is right.MatGetRow will be efficient than MatGetColumnVector.Don't know about MatAXPY() though.From the description it looks like it does Matrix + scalar*Matrix operation. Shri ----- Original Message ----- From: "Matthew Knepley" To: "PETSc users list" Sent: Wednesday, October 28, 2009 12:46:13 PM GMT -06:00 US/Canada Central Subject: Re: Does PETSc have some functions like A.*B, A./B in Matlab? There is MatAXPY(). ? Matt On Wed, Oct 28, 2009 at 11:39 AM, Jed Brown < jed at 59a2.org > wrote: abhyshr at mcs.anl.gov wrote: > If A and B are vectors, then you can use VecPointwiseMult for A.*B and VecPointwiseDivide for A./B. > For matrices,pointwise multiplication and division functions are not provided in Petsc. One possible way to do > A.*B and A./B for matrices is to loop through the columns of the matrices,call MatGetColumnVector for each column of matrix A,B > and then call VecPointwiseMult/VecPointwiseDivide. This would be very inefficient because it produces dense output and PETSc matrices are row-aligned. ?I would suggest using MatGetRow which is much faster and preserves sparsity. Jed > > Shri > > ----- Original Message ----- > From: "Yujie" < recrusader at gmail.com > > To: "PETSc users list" < petsc-users at mcs.anl.gov > > Sent: Wednesday, October 28, 2009 10:21:34 AM GMT -06:00 US/Canada Central > Subject: Does PETSc have some functions like A.*B, A./B in Matlab? > > Dear PETSc Developers, > > Does PETSc provide some functions, like A.*B, A./B in Matlab? Thanks a lot. > > Regards, > Yujie -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From bsmith at mcs.anl.gov Fri Oct 30 17:14:40 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 30 Oct 2009 17:14:40 -0500 Subject: MPIBAIJ and sor In-Reply-To: References: <4AE75800.8010700@imperial.ac.uk> <4AE75A8E.6040606@59A2.org> Message-ID: I have added the support for SOR with MPIBAIJ matrices (MatSOR_MPIBAIJ()) to petsc-dev. (Note as in the discussion this is actually block Jacobi with SOR on each process). I have changed the name of MatRelax() to MatSOR() in petsc-dev so that a consistent name is used, instead of sometimes using relax. I have also merged the MatPBRelax() into MatRelax() in petsc-dev (now MatSOR()) to simplify and clarify the code. Barry On Oct 27, 2009, at 4:04 PM, Barry Smith wrote: > > I completely forgot about the MatPBRelax_ routines. > > MatRelax() which is used by PCSOR automatically tries > MatRelax_xxx() for the matrix and then MatPBRelax_xxx() if the > MatRelax_xxx() does not exist. > > MatRelax_MPIAIJ() (and MatRelax_MPISBAIJ()) handles the block > Jacobi and relax on each block automatically and allows for any > number of local smoothing before updating the ghost points. > > If you need MatPBRelax_SeqBAIJ_N() for N > 7 that is very easily > added by taking the 7 case and extending it. > > Barry > > This discussion brings up a few issues > > 1) having the MatRelax() and MatPBRelax() both is a bit confusing. > As Jed points out it is possible to write a MatRelax() for BAIJ, but > that is almost always silly since essentially the same number > of floating point operations gives you the better point block > relaxation. I originally thought we might support all point-block > and point relaxation for all matrix types and hence have both the > MatRelax() and MatPBRelax(). Perhaps that was unneededly general > and we should merge them. Then I wouldn't have searched for > MatRelax_SeqBAIJ and given you the wrong answer initially. > > 2) Using SOR as the name of the preconditioner but relax as the name > of the matrix operation is questionable. Better to use the same name > everywhere? > > 3) One could very easily modify the MatRelax_MPIAIJ() to get a > MatRelax_MPIBAIJ() and we should do that (and test it). But as Jed > points out it is redundant (as is MatRelax_MPIAIJ()). > > On Oct 27, 2009, at 3:39 PM, Jed Brown wrote: > >> Stephan Kramer wrote: >>> I'm slightly confused by the fact that if I use my BAIJ matrix in >>> parallel and select the sor preconditioner. it complains about >>> mat->ops->relax and mat->ops->pbrelax not being defined. >> >> There is no parallel SOR, it would make the algorithm nearly serial. >> Use -sub_pc_type sor with -ksp_type asm or bjacobi. >> >>> Also I can't find a MatRelax_MPIBAIJ in >>> src/mat/impls/baij/mpi/mpibaij.c as I would have expected. Is >>> there no >>> sor type relaxation for parallel block matrices at all? >> >> Since the small dense blocks are available at once, it would be >> silly to >> relax each component separately (your CPU spends most of its time >> waiting on memory). Instead, BAIJ relaxes all components >> simultaneously >> by solving with these small blocks. This is "point-block" >> relaxation, >> look for MatPBRelax_SeqBAIJ_N for N in 1..7. >> >> Jed >> >