From timothee.nicolas at gmail.com Tue Dec 1 01:57:54 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Tue, 1 Dec 2015 16:57:54 +0900 Subject: [petsc-users] Weighted Jacobi Message-ID: Hi all, Is weighted Jacobi available as a preconditioner ? I can't find it in the list of preconditioners. If not, what is the rationale between this choice ? It is pretty straightforward to code, so if it is not available I can do it without problem I guess, but I am just wondering. In the matrix-free case where SOR is not available by default, it may be better than pure Jacobi, and much easier to parallelize than SOR. Best Timothee -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Tue Dec 1 03:02:59 2015 From: dave.mayhem23 at gmail.com (Dave May) Date: Tue, 1 Dec 2015 10:02:59 +0100 Subject: [petsc-users] Weighted Jacobi In-Reply-To: References: Message-ID: I believe what you are looking for is defined by the following options -ksp_type richardson -ksp_richardson_scale -pc_type jacobi Thanks, Dave On 1 December 2015 at 08:57, Timoth?e Nicolas wrote: > Hi all, > > Is weighted Jacobi available as a preconditioner ? I can't find it in the > list of preconditioners. If not, what is the rationale between this choice > ? It is pretty straightforward to code, so if it is not available I can do > it without problem I guess, but I am just wondering. In the matrix-free > case where SOR is not available by default, it may be better than pure > Jacobi, and much easier to parallelize than SOR. > > Best > > Timothee > -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothee.nicolas at gmail.com Tue Dec 1 03:26:57 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Tue, 1 Dec 2015 18:26:57 +0900 Subject: [petsc-users] Weighted Jacobi In-Reply-To: References: Message-ID: OK, I'll look at that, thanks Timothee 2015-12-01 18:02 GMT+09:00 Dave May : > I believe what you are looking for is defined by the following options > -ksp_type richardson > -ksp_richardson_scale > -pc_type jacobi > > Thanks, > Dave > > On 1 December 2015 at 08:57, Timoth?e Nicolas > wrote: > >> Hi all, >> >> Is weighted Jacobi available as a preconditioner ? I can't find it in the >> list of preconditioners. If not, what is the rationale between this choice >> ? It is pretty straightforward to code, so if it is not available I can do >> it without problem I guess, but I am just wondering. In the matrix-free >> case where SOR is not available by default, it may be better than pure >> Jacobi, and much easier to parallelize than SOR. >> >> Best >> >> Timothee >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mono at mek.dtu.dk Tue Dec 1 04:23:37 2015 From: mono at mek.dtu.dk (=?utf-8?B?TW9ydGVuIE5vYmVsLUrDuHJnZW5zZW4=?=) Date: Tue, 1 Dec 2015 10:23:37 +0000 Subject: [petsc-users] DMPlex: Ghost points after DMRefine In-Reply-To: <619D3787-40B8-4E37-B143-22A49C595D28@dtu.dk> References: <619D3787-40B8-4E37-B143-22A49C595D28@dtu.dk> Message-ID: I found a solution to my problem by using the global section instead. I still don?t quite understand what my problem ISLocalToGlobalMapping was. // New code bool isGhost(DM dm, PetscInt point){ PetscSection globalSection; DMGetDefaultGlobalSection(dm, &globalSection); PetscInt offset; PetscSectionGetOffset(globalSection,point, &offset); return offset < 0; } // Old code bool isGhost(DM dm, PetscInt point){ const PetscInt* array; ISLocalToGlobalMapping ltogm; DMGetLocalToGlobalMapping(dm,<ogm); ISLocalToGlobalMappingGetIndices(ltogm, &array); return array[point]<0; } From: Morten Nobel-J?rgensen > Date: Monday 30 November 2015 at 17:24 To: Matthew Knepley > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] DMPlex: Ghost points after DMRefine Hi Matt I don?t think the problem is within Petsc - rather somewhere in my code. When I dump the DMPlex using DMView (ascii-info?detail) the ghost mapping seems to be setup correctly. Is there a better way to determine if a local point is a ghost point? The way I iterate the DMPlex is like this: void iterateDMPlex(DM dm){ Vec coordinates; DMGetCoordinatesLocal(dm, &coordinates); PetscSection defaultSection; DMGetDefaultSection(dm, &defaultSection); PetscSection coordSection; DMGetCoordinateSection(dm, &coordSection); PetscScalar *coords; VecGetArray(coordinates, &coords); DM cdm; DMGetCoordinateDM(dm, &cdm); // iterate (local) mesh PetscInt cellsFrom, cellsTo; std::string s = ""; DMPlexGetHeightStratum(dm, 0, &cellsFrom, &cellsTo); for (PetscInt i=cellsFrom;i> Date: Monday 30 November 2015 at 14:08 To: Morten Nobel-J?rgensen > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] DMPlex: Ghost points after DMRefine On Mon, Nov 30, 2015 at 7:01 AM, Morten Nobel-J?rgensen > wrote: I have a very simple unstructured mesh composed of two triangles (four vertices) with one shared edge using a DMPlex: /|\ / | \ \ | / \|/ After distributing this mesh to two processes, each process owns a triangle. However one process owns tree vertices, while the last vertex is owned by the other process. The problem occurs after uniformly refining the dm. The mesh now looks like this: /|\ /\|/\ \/|\/ \|/ The new center vertex is now not listed as a ghost vertex but instead exists as two individual points. Is there any way that this new center vertex could be created as a ghost vertex during refinement? This could be a bug with the l2g mapping. I do not recreate it when refining, only the SF defining the mapping. Here is an experiment: do not retrieve the mapping until after the refinement. Do you get what you want? If so, I can easily fix this by destroying the map when I refine. Thanks, Matt Kind regards, Morten Ps. Here are some code snippets for getting global point index and test of point is a ghost point: int localToGlobal(DM dm, PetscInt point){ const PetscInt* array; ISLocalToGlobalMapping ltogm; DMGetLocalToGlobalMapping(dm,<ogm); ISLocalToGlobalMappingGetIndices(ltogm, &array); PetscInt res = array[point]; if (res < 0){ // if ghost res = -res +1; } return res; } bool isGhost(DM dm, PetscInt point){ const PetscInt* array; ISLocalToGlobalMapping ltogm; DMGetLocalToGlobalMapping(dm,<ogm); ISLocalToGlobalMappingGetIndices(ltogm, &array); return array[point]<0; } -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Dec 1 08:10:53 2015 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 1 Dec 2015 08:10:53 -0600 Subject: [petsc-users] Weighted Jacobi In-Reply-To: References: Message-ID: Or do you mean *-pc_jacobi_type rowsum* * Thanks,* * Matt* On Tue, Dec 1, 2015 at 3:26 AM, Timoth?e Nicolas wrote: > OK, I'll look at that, thanks > > Timothee > > 2015-12-01 18:02 GMT+09:00 Dave May : > >> I believe what you are looking for is defined by the following options >> -ksp_type richardson >> -ksp_richardson_scale >> -pc_type jacobi >> >> Thanks, >> Dave >> >> On 1 December 2015 at 08:57, Timoth?e Nicolas > > wrote: >> >>> Hi all, >>> >>> Is weighted Jacobi available as a preconditioner ? I can't find it in >>> the list of preconditioners. If not, what is the rationale between this >>> choice ? It is pretty straightforward to code, so if it is not available I >>> can do it without problem I guess, but I am just wondering. In the >>> matrix-free case where SOR is not available by default, it may be better >>> than pure Jacobi, and much easier to parallelize than SOR. >>> >>> Best >>> >>> Timothee >>> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothee.nicolas at gmail.com Tue Dec 1 08:27:14 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Tue, 1 Dec 2015 23:27:14 +0900 Subject: [petsc-users] Weighted Jacobi In-Reply-To: References: Message-ID: I meant X^{k+1} = omega*D^{-1}*(b - (A-D)*X^k) + (1-omega)*X^k This is what the wikipedia page says about weighted Jacobi : https://en.wikipedia.org/wiki/Jacobi_method#Weighted_Jacobi_method. It is easy to hard code anyway. Thx Timoth?e 2015-12-01 23:10 GMT+09:00 Matthew Knepley : > Or do you mean > > *-pc_jacobi_type rowsum* > > * Thanks,* > > * Matt* > > On Tue, Dec 1, 2015 at 3:26 AM, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > >> OK, I'll look at that, thanks >> >> Timothee >> >> 2015-12-01 18:02 GMT+09:00 Dave May : >> >>> I believe what you are looking for is defined by the following options >>> -ksp_type richardson >>> -ksp_richardson_scale >>> -pc_type jacobi >>> >>> Thanks, >>> Dave >>> >>> On 1 December 2015 at 08:57, Timoth?e Nicolas < >>> timothee.nicolas at gmail.com> wrote: >>> >>>> Hi all, >>>> >>>> Is weighted Jacobi available as a preconditioner ? I can't find it in >>>> the list of preconditioners. If not, what is the rationale between this >>>> choice ? It is pretty straightforward to code, so if it is not available I >>>> can do it without problem I guess, but I am just wondering. In the >>>> matrix-free case where SOR is not available by default, it may be better >>>> than pure Jacobi, and much easier to parallelize than SOR. >>>> >>>> Best >>>> >>>> Timothee >>>> >>> >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Dec 1 09:35:39 2015 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 1 Dec 2015 09:35:39 -0600 Subject: [petsc-users] DMPlex: Ghost points after DMRefine In-Reply-To: References: <619D3787-40B8-4E37-B143-22A49C595D28@dtu.dk> Message-ID: On Tue, Dec 1, 2015 at 4:23 AM, Morten Nobel-J?rgensen wrote: > I found a solution to my problem by using the global section instead. I > still don?t quite understand what my problem ISLocalToGlobalMapping was. > Yes, that is the solution I use. > // New code > > bool isGhost(DM dm, PetscInt point){ > PetscSection globalSection; > DMGetDefaultGlobalSection(dm, &globalSection); > PetscInt offset; > PetscSectionGetOffset(globalSection,point, &offset); > return offset < 0; > } > > I think I ignore nonlocal indices in the l2g mapping rather than making them negative. This can of course be changed. Matt > // Old code > > bool isGhost(DM dm, PetscInt point){ > const PetscInt* array; > ISLocalToGlobalMapping ltogm; > DMGetLocalToGlobalMapping(dm,<ogm); > ISLocalToGlobalMappingGetIndices(ltogm, &array); > return array[point]<0; > } > > > From: Morten Nobel-J?rgensen > Date: Monday 30 November 2015 at 17:24 > To: Matthew Knepley > > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] DMPlex: Ghost points after DMRefine > > Hi Matt > > I don?t think the problem is within Petsc - rather somewhere in my code. > When I dump the DMPlex using DMView (ascii-info?detail) the ghost mapping > seems to be setup correctly. > > Is there a better way to determine if a local point is a ghost point? > > The way I iterate the DMPlex is like this: > > void iterateDMPlex(DM dm){ > Vec coordinates; > DMGetCoordinatesLocal(dm, &coordinates); > PetscSection defaultSection; > DMGetDefaultSection(dm, &defaultSection); > PetscSection coordSection; > DMGetCoordinateSection(dm, &coordSection); > PetscScalar *coords; > VecGetArray(coordinates, &coords); > DM cdm; > DMGetCoordinateDM(dm, &cdm); > > // iterate (local) mesh > PetscInt cellsFrom, cellsTo; > std::string s = ""; > DMPlexGetHeightStratum(dm, 0, &cellsFrom, &cellsTo); > for (PetscInt i=cellsFrom;i PetscInt edgesSize; > const PetscInt *edgeIndices; > DMPlexGetConeSize(dm, i, &edgesSize); > DMPlexGetCone(dm, i, &edgeIndices); > s = s + "Element: "+std::to_string(i)+"\n"; > > for (int edgeId = 0;edgeId PetscInt points = edgeIndices[edgeId]; > PetscInt edgePoint = edgeIndices[edgeId]; > s = s + "\tEdge: "+std::to_string(edgePoint)+"\n"; > PetscInt vertexSize; > const PetscInt *vertexIndices; > DMPlexGetConeSize(dm, edgePoint, &vertexSize); > DMPlexGetCone(dm, edgePoint, &vertexIndices); > > for (int j=0;j s = s + "\t\tVertex: "+std::to_string(vertexIndices[j]); > > s = s + " coords "; > PetscScalar* values; > VecGetValuesSection(coordinates, coordSection,vertexIndices[j],&values); > > int dim = 2; > for (int k=0;k double coordinate = values[k]; > > s = s +std::to_string(coordinate)+" "; > } > > s = s + (isGhost(cdm, vertexIndices[j])?" ghost":""); > > s = s + "\n"; > > } > } > } > > VecRestoreArray(coordinates, &coords); > > int rank; > MPI_Comm_rank (PETSC_COMM_WORLD, &rank); // get current process id > > PetscSynchronizedPrintf(PETSC_COMM_WORLD,*?*dmplex iteration rank %d \n %s\n",rank, s.c_str()); > PetscSynchronizedFlush(PETSC_COMM_WORLD,PETSC_STDOUT); > } > > > From: Matthew Knepley > Date: Monday 30 November 2015 at 14:08 > To: Morten Nobel-J?rgensen > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] DMPlex: Ghost points after DMRefine > > On Mon, Nov 30, 2015 at 7:01 AM, Morten Nobel-J?rgensen > wrote: > >> I have a very simple unstructured mesh composed of two triangles (four >> vertices) with one shared edge using a DMPlex: >> >> /|\ >> / | \ >> \ | / >> \|/ >> >> After distributing this mesh to two processes, each process owns a >> triangle. However one process owns tree vertices, while the last vertex is >> owned by the other process. >> >> The problem occurs after uniformly refining the dm. The mesh now looks >> like this: >> >> /|\ >> /\|/\ >> \/|\/ >> \|/ >> >> The new center vertex is now not listed as a ghost vertex but instead >> exists as two individual points. >> >> Is there any way that this new center vertex could be created as a ghost >> vertex during refinement? >> > > This could be a bug with the l2g mapping. I do not recreate it when > refining, only the SF defining the mapping. > Here is an experiment: do not retrieve the mapping until after the > refinement. Do you get what you want? If so, > I can easily fix this by destroying the map when I refine. > > Thanks, > > Matt > > >> Kind regards, >> Morten >> >> Ps. Here are some code snippets for getting global point index and test >> of point is a ghost point: >> >> int localToGlobal(DM dm, PetscInt point){ >> const PetscInt* array; >> ISLocalToGlobalMapping ltogm; >> DMGetLocalToGlobalMapping(dm,<ogm); >> ISLocalToGlobalMappingGetIndices(ltogm, &array); >> PetscInt res = array[point]; >> if (res < 0){ // if ghost >> res = -res +1; >> } >> return res; >> } >> >> bool isGhost(DM dm, PetscInt point){ >> const PetscInt* array; >> ISLocalToGlobalMapping ltogm; >> DMGetLocalToGlobalMapping(dm,<ogm); >> ISLocalToGlobalMappingGetIndices(ltogm, &array); >> return array[point]<0; >> } >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From security-noreply at linkedin.com Tue Dec 1 09:37:36 2015 From: security-noreply at linkedin.com (LinkedIn Security) Date: Tue, 1 Dec 2015 15:37:36 +0000 (UTC) Subject: [petsc-users] Matt, a new email address was added to your account Message-ID: <1221741485.7436379.1448984256104.JavaMail.app@lva1-app3887.prod.linkedin.com> Hi Matt, The email address matthias.funk at sap.com was recently added to your LinkedIn account. If you don't want that email associated with your account after all, here's how to remove it: Type https://www.linkedin.com/e/v2?e=2557fb-ihnjsp8t-1p&t=nas&midToken=AQEKWfzqDe1kYg&ek=security_add_email_notification into your browser Sign in with your email address and password Find Primary email in the upper left and click Change/Add Click Remove next to the email address you're removing. Thanks for using LinkedIn and for helping us to keep your account secure. The LinkedIn Team This email was intended for Matt Funk (Software Engineer at SAP). Learn why we included this: https://www.linkedin.com/e/v2?e=2557fb-ihnjsp8t-1p&a=customerServiceUrl&midToken=AQEKWfzqDe1kYg&ek=security_add_email_notification&articleId=4788 If you need assistance or have questions, please contact LinkedIn Customer Service: https://www.linkedin.com/e/v2?e=2557fb-ihnjsp8t-1p&a=customerServiceUrl&midToken=AQEKWfzqDe1kYg&ek=security_add_email_notification © 2015 LinkedIn Corporation, 2029 Stierlin Court, Mountain View CA 94043. LinkedIn and the LinkedIn logo are registered trademarks of LinkedIn. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mono at mek.dtu.dk Tue Dec 1 10:51:10 2015 From: mono at mek.dtu.dk (=?utf-8?B?TW9ydGVuIE5vYmVsLUrDuHJnZW5zZW4=?=) Date: Tue, 1 Dec 2015 16:51:10 +0000 Subject: [petsc-users] DMPlex: Ghost points after DMRefine In-Reply-To: References: <619D3787-40B8-4E37-B143-22A49C595D28@dtu.dk> Message-ID: I found a solution to my problem by using the global section instead. I still don?t quite understand what my problem ISLocalToGlobalMapping was. > Yes, that is the solution I use. Thanks ? good to hear that I?m on the right track :) > I think I ignore nonlocal indices in the l2g mapping rather than making them negative. This can of course be changed. Ok - Morten -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Dec 1 12:13:38 2015 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 1 Dec 2015 12:13:38 -0600 Subject: [petsc-users] PETSC error: Caught signal number 8 FPE In-Reply-To: References: <44C0070A-C473-4218-85E1-A4451218C250@dsic.upv.es> Message-ID: On Mon, Nov 30, 2015 at 2:43 PM, Soumya Mukherjee wrote: > Thanks for the reply. > > The error message shows > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Scalar value must be same on all processes, argument # 3 > This is a common error when you calculate a NaN somehow. Thanks, Matt > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.1, Jul, 22, 2015 > [0]PETSC ERROR: ./main on a arch-linux2-cxx-debug named > soumya-OptiPlex-9010 by soumya Mon Nov 30 12:30:28 2015 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran > --with-cxx=g++ --with-clanguage=cxx --download-fblaslapack --download-mpich > --with-scalar-type=complex > [0]PETSC ERROR: #1016 BVScaleColumn() line 380 in > /home/soumya/slepc-3.6.0/src/sys/classes/bv/interface/bvops.c > [0]PETSC ERROR: #1017 EPSBasicArnoldi() line 65 in > /home/soumya/slepc-3.6.0/src/eps/impls/krylov/epskrylov.c > [0]PETSC ERROR: #1018 EPSSolve_KrylovSchur_Default() line 201 in > /home/soumya/slepc-3.6.0/src/eps/impls/krylov/krylovschur/krylovschur.c > [0]PETSC ERROR: #1019 EPSSolve() line 101 in > /home/soumya/slepc-3.6.0/src/eps/interface/epssolve.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Must call EPSSolve() first: Parameter #1 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.1, Jul, 22, 2015 > [0]PETSC ERROR: ./main on a arch-linux2-cxx-debug named > soumya-OptiPlex-9010 by soumya Mon Nov 30 12:30:28 2015 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran > --with-cxx=g++ --with-clanguage=cxx --download-fblaslapack --download-mpich > --with-scalar-type=complex > [0]PETSC ERROR: #1020 EPSGetConverged() line 236 in > /home/soumya/slepc-3.6.0/src/eps/interface/epssolve.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Must call EPSSolve() first: Parameter #1 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.1, Jul, 22, 2015 > [0]PETSC ERROR: ./main on a arch-linux2-cxx-debug named > soumya-OptiPlex-9010 by soumya Mon Nov 30 12:30:28 2015 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran > --with-cxx=g++ --with-clanguage=cxx --download-fblaslapack --download-mpich > --with-scalar-type=complex > [0]PETSC ERROR: #1021 EPSGetEigenpair() line 378 in > /home/soumya/slepc-3.6.0/src/eps/interface/epssolve.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Must call EPSSolve() first: Parameter #1 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.1, Jul, 22, 2015 > [0]PETSC ERROR: ./main on a arch-linux2-cxx-debug named > soumya-OptiPlex-9010 by soumya Mon Nov 30 12:30:28 2015 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran > --with-cxx=g++ --with-clanguage=cxx --download-fblaslapack --download-mpich > --with-scalar-type=complex > [0]PETSC ERROR: #1022 EPSGetEigenpair() line 378 in > /home/soumya/slepc-3.6.0/src/eps/interface/epssolve.c > > On Mon, Nov 30, 2015 at 9:08 AM, Matthew Knepley > wrote: > >> On Mon, Nov 30, 2015 at 7:59 AM, Soumya Mukherjee < >> soumyamechanics at gmail.com> wrote: >> >>> It is a PETSc error. And I just wanted to know if runs without an error >>> in your machine. >>> >> This is not a PETSc error, as such. PETSc installs a signal handler so >> that we can try and get more >> information about signals. However, it is likely that you have a Floating >> Point Exception, like a divide >> by zero, in your user code. >> >> Thanks, >> >> Matt >> >>> On Nov 30, 2015 4:34 AM, "Jose E. Roman" wrote: >>> > >>> > I am not going to run your code. We are not a free debugging service. >>> You have to debug the code yourself, and let us know only if the issue is >>> related to the SLEPc library. Start adding error checking code with the >>> CHKERRQ macro to all PETSc/SLEPc calls. This will catch most errors. It no >>> errors are detected, then run with a debugger such as gdb or valgrind to >>> determine the exact point where the program fails. >>> > >>> > Jose >>> > >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From adlinds3 at ncsu.edu Tue Dec 1 13:22:54 2015 From: adlinds3 at ncsu.edu (Alex Lindsay) Date: Tue, 1 Dec 2015 14:22:54 -0500 Subject: [petsc-users] Output newton step In-Reply-To: <7F088E0C-413B-4311-8CF1-87E33643BE42@mcs.anl.gov> References: <565CAF6B.50209@ncsu.edu> <7F088E0C-413B-4311-8CF1-87E33643BE42@mcs.anl.gov> Message-ID: <565DF38E.7060005@ncsu.edu> Thanks Barry that helps a lot. It looks like the defaults for me (PetSc 3.6.0 release) are stdout for -ksp_view_solution and draw for -snes_monitor_solution_update. I don't see any binary files generated with the defaults. On 11/30/2015 05:56 PM, Barry Smith wrote: >> On Nov 30, 2015, at 2:19 PM, Alex Lindsay wrote: >> >> Is there an option for outputting the Newton step after my linear solve? >> >> Alex > Do you want the solution of the linear system before the line search (line search may shrink the vector) use -ksp_view_solution or the actual update selected by Newton -snes_monitor_solution_update > > If you use the master branch of PETSc then both of these flags take the option > > [ascii or binary or draw][:filename][:viewer format] > > allowing printing as ascii, binary or drawing the solution in a window (Drawing only works for DMDA 1d or 2d). > > Barry > > For the release version of PETSc it saves the vectors in a binary file called binaryoutput > > > > > From bsmith at mcs.anl.gov Tue Dec 1 13:29:14 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 1 Dec 2015 13:29:14 -0600 Subject: [petsc-users] Output newton step In-Reply-To: <565DF38E.7060005@ncsu.edu> References: <565CAF6B.50209@ncsu.edu> <7F088E0C-413B-4311-8CF1-87E33643BE42@mcs.anl.gov> <565DF38E.7060005@ncsu.edu> Message-ID: <08A16193-1D02-4E0F-A8FA-B27273432276@mcs.anl.gov> > On Dec 1, 2015, at 1:22 PM, Alex Lindsay wrote: > > Thanks Barry that helps a lot. It looks like the defaults for me (PetSc 3.6.0 release) are stdout for -ksp_view_solution and draw for -snes_monitor_solution_update. I don't see any binary files generated with the defaults. You can use -ksp_view_solution binary:filename to save each in the file To get anything more than the draw for -snes_monitor_solution_update you have to upgrade to master to be able save as binary Barry > > On 11/30/2015 05:56 PM, Barry Smith wrote: >>> On Nov 30, 2015, at 2:19 PM, Alex Lindsay wrote: >>> >>> Is there an option for outputting the Newton step after my linear solve? >>> >>> Alex >> Do you want the solution of the linear system before the line search (line search may shrink the vector) use -ksp_view_solution or the actual update selected by Newton -snes_monitor_solution_update >> >> If you use the master branch of PETSc then both of these flags take the option >> >> [ascii or binary or draw][:filename][:viewer format] >> >> allowing printing as ascii, binary or drawing the solution in a window (Drawing only works for DMDA 1d or 2d). >> >> Barry >> >> For the release version of PETSc it saves the vectors in a binary file called binaryoutput >> >> >> >> >> > From jed at jedbrown.org Tue Dec 1 13:46:34 2015 From: jed at jedbrown.org (Jed Brown) Date: Tue, 01 Dec 2015 12:46:34 -0700 Subject: [petsc-users] Weighted Jacobi In-Reply-To: References: Message-ID: <878u5ex885.fsf@jedbrown.org> Timoth?e Nicolas writes: > I meant > > X^{k+1} = omega*D^{-1}*(b - (A-D)*X^k) + (1-omega)*X^k Let's rearrange: X^{k+1} = omega*D^{-1}*(b - A*X^k) + omega*D^{-1}*D*X^k + (1-omega)*X^k = omega*D^{-1}*(b - A*X^k) + omega*X^k - omega*X^k + X^k = X^k + omega*D^{-1}*(b - A*X^k) There, isn't that simpler? This is Richardson preconditioned by Jacobi, as Dave says. Matt's question is about how to define D, which is available through options to Jacobi. The diagonal is the default. But you'll likely be happier using Chebyshev, which is strictly more general than damped Jacobi. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From danyang.su at gmail.com Tue Dec 1 17:02:08 2015 From: danyang.su at gmail.com (Danyang Su) Date: Tue, 1 Dec 2015 15:02:08 -0800 Subject: [petsc-users] Error reported by MUMPS in numerical factorization phase Message-ID: <565E26F0.6050109@gmail.com> Hi All, My code fails due to the error in external library. It works fine for the previous 2000+ timesteps but then crashes. [4]PETSC ERROR: Error in external library [4]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFO(1)=-1, INFO(2)=0 The full error message is attached. Then I tried the same simulation on another machine using the same number of processors, it does not fail. Is there anything wrong with the configuration? or would it be possible if anything is wrong in my code? Thanks, Danyang -------------- next part -------------- [4]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [4]PETSC ERROR: Error in external library [4]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFO(1)=-1, INFO(2)=0 [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [4]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 [4]PETSC ERROR: ../min3p_thcm on a arch-linux2-c-debug named pod26b15 by danyangs Tue Dec 1 11:29:00 2015 [4]PETSC ERROR: Configure options --prefix=/global/software/lib64/intel/petsc-3.6.2 --with-64-bit-pointers=0 --with-pthread=0 --with-pthreadclasses=0 --with-cc=/global/software/openmpi-1.6.5/intel/bin/mpicc --CFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-cxx=/global/software/openmpi-1.6.5/intel/bin/mpicxx --CXXFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-fc=/global/software/openmpi-1.6.5/intel/bin/mpif90 --FFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-cxx-dialect=C++11 --with-single-library=1 --with-shared-libraries --with-shared-ld=/global/software/openmpi-1.6.5/intel-2011/bin/mpicc --sharedLibraryFlags="-fpic -fPIC " --with-large-file-io=1 --with-mpi=1 --with-mpi-shared=1 --with-mpirun=/global/software/openmpi-1.6.5/intel/bin/mpiexec --with-mpi-compilers=1 --with-x=yes --with-blas-lapack-dir=/global/software/intel/composerxe/mkl/lib/intel64 --with-ptscotch=0 --with-x=1 --with-hdf5=1 --with-hdf5-dir=/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1 --with-netcdf=0 --with-fftw=1 --with-fftw-dir=/global/software/lib64/intel/fftw-3.3.3 --download-blacs=yes --download-scalapack=yes --download-superlu_dist=yes --download-mumps=yes --download-metis=yes --download-parmetis=yes --download-spooles=yes --download-cproto=yes --download-suitesparse=yes --download-hypre=yes --download-amd=yes --download-adifor=yes --download-euclid=yes --download-spai=yes --download-sprng=yes --download-ml=yes --download-boost=yes --download-triangle=yes --download-generator=yes --with-boost=1 --with-petsc4py=0 --with-numpy=1 exit 0 [4]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1172 in /tmp/petsc-3.6.2/src/mat/impls/aij/mpi/mumps/mumps.c [5]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [5]PETSC ERROR: Error in external library [5]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFO(1)=-1, INFO(2)=0 [5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [5]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 [5]PETSC ERROR: ../min3p_thcm on a arch-linux2-c-debug named pod26b15 by danyangs Tue Dec 1 11:29:00 2015 [5]PETSC ERROR: Configure options --prefix=/global/software/lib64/intel/petsc-3.6.2 --with-64-bit-pointers=0 --with-pthread=0 --with-pthreadclasses=0 --with-cc=/global/software/openmpi-1.6.5/intel/bin/mpicc --CFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-cxx=/global/software/openmpi-1.6.5/intel/bin/mpicxx --CXXFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-fc=/global/software/openmpi-1.6.5/intel/bin/mpif90 --FFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-cxx-dialect=C++11 --with-single-library=1 --with-shared-libraries --with-shared-ld=/global/software/openmpi-1.6.5/intel-2011/bin/mpicc --sharedLibraryFlags="-fpic -fPIC " --with-large-file-io=1 --with-mpi=1 --with-mpi-shared=1 --with-mpirun=/global/software/openmpi-1.6.5/intel/bin/mpiexec --with-mpi-compilers=1 --with-x=yes --with-blas-lapack-dir=/global/software/intel/composerxe/mkl/lib/intel64 --with-ptscotch=0 --with-x=1 --with-hdf5=1 --with-hdf5-dir=/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1 --with-netcdf=0 --with-fftw=1 --with-fftw-dir=/global/software/lib64/intel/fftw-3.3.3 --download-blacs=yes --download-scalapack=yes --download-superlu_dist=yes --download-mumps=yes --download-metis=yes --download-parmetis=yes --download-spooles=yes --download-cproto=yes --download-suitesparse=yes --download-hypre=yes --download-amd=yes --download-adifor=yes --download-euclid=yes --download-spai=yes --download-sprng=yes --download-ml=yes --download-boost=yes --download-triangle=yes --download-generator=yes --with-boost=1 --with-petsc4py=0 --with-numpy=1 exit 0 [5]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1172 in /tmp/petsc-3.6.2/src/mat/impls/aij/mpi/mumps/mumps.c [6]PETSC ERROR: [7]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [7]PETSC ERROR: Error in external library [7]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFO(1)=-1, INFO(2)=0 [7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [7]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 [7]PETSC ERROR: ../min3p_thcm on a arch-linux2-c-debug named pod26b15 by danyangs Tue Dec 1 11:29:00 2015 [7]PETSC ERROR: Configure options --prefix=/global/software/lib64/intel/petsc-3.6.2 --with-64-bit-pointers=0 --with-pthread=0 --with-pthreadclasses=0 --with-cc=/global/software/openmpi-1.6.5/intel/bin/mpicc --CFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-cxx=/global/software/openmpi-1.6.5/intel/bin/mpicxx --CXXFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-fc=/global/software/openmpi-1.6.5/intel/bin/mpif90 --FFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-cxx-dialect=C++11 --with-single-library=1 --with-shared-libraries --with-shared-ld=/global/software/openmpi-1.6.5/intel-2011/bin/mpicc --sharedLibraryFlags="-fpic -fPIC " --with-large-file-io=1 --with-mpi=1 --with-mpi-shared=1 --with-mpirun=/global/software/openmpi-1.6.5/intel/bin/mpiexec --with-mpi-compilers=1 --with-x=yes --with-blas-lapack-dir=/global/software/intel/composerxe/mkl/lib/intel64 --with-ptscotch=0 --with-x=1 --with-hdf5=1 --with-hdf5-dir=/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1 --with-netcdf=0 --with-fftw=1 --with-fftw-dir=/global/software/lib64/intel/fftw-3.3.3 --download-blacs=yes --download-scalapack=yes --download-superlu_dist=yes --download-mumps=yes --download-metis=yes --download-parmetis=yes --download-spooles=yes --download-cproto=yes --download-suitesparse=yes --download-hypre=yes --download-amd=yes --download-adifor=yes --download-euclid=yes --download-spai=yes --download-sprng=yes --download-ml=yes --download-boost=yes --download-triangle=yes --download-generator=yes --with-boost=1 --with-petsc4py=0 --with-numpy=1 exit 0 [7]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1172 in /tmp/petsc-3.6.2/src/mat/impls/aij/mpi/mumps/mumps.c [7]PETSC ERROR: #2 MatLUFactorNumeric() line 2946 in /tmp/petsc-3.6.2/src/mat/interface/matrix.c [7]PETSC ERROR: #3 PCSetUp_LU() line 152 in /tmp/petsc-3.6.2/src/ksp/pc/impls/factor/lu/lu.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Error in external library [0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFO(1)=-9, INFO(2)=113254 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 [0]PETSC ERROR: ../min3p_thcm on a arch-linux2-c-debug named pod26b15 by danyangs Tue Dec 1 11:29:00 2015 [0]PETSC ERROR: Configure options --prefix=/global/software/lib64/intel/petsc-3.6.2 --with-64-bit-pointers=0 --with-pthread=0 --with-pthreadclasses=0 --with-cc=/global/software/openmpi-1.6.5/intel/bin/mpicc --CFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-cxx=/global/software/openmpi-1.6.5/intel/bin/mpicxx --CXXFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-fc=/global/software/openmpi-1.6.5/intel/bin/mpif90 --FFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-cxx-dialect=C++11 --with-single-library=1 --with-shared-libraries --with-shared-ld=/global/software/openmpi-1.6.5/intel-2011/bin/mpicc --sharedLibraryFlags="-fpic -fPIC " --with-large-file-io=1 --with-mpi=1 --with-mpi-shared=1 --with-mpirun=/global/software/openmpi-1.6.5/intel/bin/mpiexec --with-mpi-compilers=1 --with-x=yes --with-blas-lapack-dir=/global/software/intel/composerxe/mkl/lib/intel64 --with-ptscotch=0 --with-x=1 --with-hdf5=1 --with-hdf5-dir=/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1 --with-netcdf=0 --with-fftw=1 --with-fftw-dir=/global/software/lib64/intel/fftw-3.3.3 --download-blacs=yes --download-scalapack=yes --download-superlu_dist=yes --download-mumps=yes --download-metis=yes --download-parmetis=yes --download-spooles=yes --download-cproto=yes --download-suitesparse=yes --download-hypre=yes --download-amd=yes --download-adifor=yes --download-euclid=yes --download-spai=yes --download-sprng=yes --download-ml=yes --download-boost=yes --download-triangle=yes --download-generator=yes --with-boost=1 --with-petsc4py=0 --with-numpy=1 exit 0 [0]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1172 in /tmp/petsc-3.6.2/src/mat/impls/aij/mpi/mumps/mumps.c [0]PETSC ERROR: #2 MatLUFactorNumeric() line 2946 in /tmp/petsc-3.6.2/src/mat/interface/matrix.c [0]PETSC ERROR: #3 PCSetUp_LU() line 152 in /tmp/petsc-3.6.2/src/ksp/pc/impls/factor/lu/lu.c [0]PETSC ERROR: #4 PCSetUp() line 983 in /tmp/petsc-3.6.2/src/ksp/pc/interface/precon.c [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Error in external library [1]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFO(1)=-1, INFO(2)=0 [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 [1]PETSC ERROR: ../min3p_thcm on a arch-linux2-c-debug named pod26b15 by danyangs Tue Dec 1 11:29:00 2015 [1]PETSC ERROR: Configure options --prefix=/global/software/lib64/intel/petsc-3.6.2 --with-64-bit-pointers=0 --with-pthread=0 --with-pthreadclasses=0 --with-cc=/global/software/openmpi-1.6.5/intel/bin/mpicc --CFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-cxx=/global/software/openmpi-1.6.5/intel/bin/mpicxx --CXXFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-fc=/global/software/openmpi-1.6.5/intel/bin/mpif90 --FFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-cxx-dialect=C++11 --with-single-library=1 --with-shared-libraries --with-shared-ld=/global/software/openmpi-1.6.5/intel-2011/bin/mpicc --sharedLibraryFlags="-fpic -fPIC " --with-large-file-io=1 --with-mpi=1 --with-mpi-shared=1 --with-mpirun=/global/software/openmpi-1.6.5/intel/bin/mpiexec --with-mpi-compilers=1 --with-x=yes --with-blas-lapack-dir=/global/software/intel/composerxe/mkl/lib/intel64 --with-ptscotch=0 --with-x=1 --with-hdf5=1 --with-hdf5-dir=/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1 --with-netcdf=0 --with-fftw=1 --with-fftw-dir=/global/software/lib64/intel/fftw-3.3.3 --download-blacs=yes --download-scalapack=yes --download-superlu_dist=yes --download-mumps=yes --download-metis=yes --download-parmetis=yes --download-spooles=yes --download-cproto=yes --download-suitesparse=yes --download-hypre=yes --download-amd=yes --download-adifor=yes --download-euclid=yes --download-spai=yes --download-sprng=yes --download-ml=yes --download-boost=yes --download-triangle=yes --download-generator=yes --with-boost=1 --with-petsc4py=0 --with-numpy=1 exit 0 [1]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1172 in /tmp/petsc-3.6.2/src/mat/impls/aij/mpi/mumps/mumps.c [1]PETSC ERROR: #2 MatLUFactorNumeric() line 2946 in /tmp/petsc-3.6.2/src/mat/interface/matrix.c [1]PETSC ERROR: #3 PCSetUp_LU() line 152 in /tmp/petsc-3.6.2/src/ksp/pc/impls/factor/lu/lu.c [1]PETSC ERROR: #4 PCSetUp() line 983 in /tmp/petsc-3.6.2/src/ksp/pc/interface/precon.c [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [2]PETSC ERROR: Error in external library [2]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFO(1)=-1, INFO(2)=0 [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [2]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 [2]PETSC ERROR: ../min3p_thcm on a arch-linux2-c-debug named pod26b15 by danyangs Tue Dec 1 11:29:00 2015 [2]PETSC ERROR: Configure options --prefix=/global/software/lib64/intel/petsc-3.6.2 --with-64-bit-pointers=0 --with-pthread=0 --with-pthreadclasses=0 --with-cc=/global/software/openmpi-1.6.5/intel/bin/mpicc --CFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-cxx=/global/software/openmpi-1.6.5/intel/bin/mpicxx --CXXFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-fc=/global/software/openmpi-1.6.5/intel/bin/mpif90 --FFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-cxx-dialect=C++11 --with-single-library=1 --with-shared-libraries --with-shared-ld=/global/software/openmpi-1.6.5/intel-2011/bin/mpicc --sharedLibraryFlags="-fpic -fPIC " --with-large-file-io=1 --with-mpi=1 --with-mpi-shared=1 --with-mpirun=/global/software/openmpi-1.6.5/intel/bin/mpiexec --with-mpi-compilers=1 --with-x=yes --with-blas-lapack-dir=/global/software/intel/composerxe/mkl/lib/intel64 --with-ptscotch=0 --with-x=1 --with-hdf5=1 --with-hdf5-dir=/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1 --with-netcdf=0 --with-fftw=1 --with-fftw-dir=/global/software/lib64/intel/fftw-3.3.3 --download-blacs=yes --download-scalapack=yes --download-superlu_dist=yes --download-mumps=yes --download-metis=yes --download-parmetis=yes --download-spooles=yes --download-cproto=yes --download-suitesparse=yes --download-hypre=yes --download-amd=yes --download-adifor=yes --download-euclid=yes --download-spai=yes --download-sprng=yes --download-ml=yes --download-boost=yes --download-triangle=yes --download-generator=yes --with-boost=1 --with-petsc4py=0 --with-numpy=1 exit 0 [2]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1172 in /tmp/petsc-3.6.2/src/mat/impls/aij/mpi/mumps/mumps.c [2]PETSC ERROR: #2 MatLUFactorNumeric() line 2946 in /tmp/petsc-3.6.2/src/mat/interface/matrix.c [2]PETSC ERROR: #3 PCSetUp_LU() line 152 in /tmp/petsc-3.6.2/src/ksp/pc/impls/factor/lu/lu.c [2]PETSC ERROR: #4 PCSetUp() line 983 in /tmp/petsc-3.6.2/src/ksp/pc/interface/precon.c [2]PETSC ERROR: #5 KSPSetUp() line 332 in /tmp/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c [2]PETSC ERROR: #6 KSPSolve() line 546 in /tmp/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [3]PETSC ERROR: Error in external library [3]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFO(1)=-1, INFO(2)=0 [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [3]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 [3]PETSC ERROR: ../min3p_thcm on a arch-linux2-c-debug named pod26b15 by danyangs Tue Dec 1 11:29:00 2015 [3]PETSC ERROR: Configure options --prefix=/global/software/lib64/intel/petsc-3.6.2 --with-64-bit-pointers=0 --with-pthread=0 --with-pthreadclasses=0 --with-cc=/global/software/openmpi-1.6.5/intel/bin/mpicc --CFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-cxx=/global/software/openmpi-1.6.5/intel/bin/mpicxx --CXXFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-fc=/global/software/openmpi-1.6.5/intel/bin/mpif90 --FFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-cxx-dialect=C++11 --with-single-library=1 --with-shared-libraries --with-shared-ld=/global/software/openmpi-1.6.5/intel-2011/bin/mpicc --sharedLibraryFlags="-fpic -fPIC " --with-large-file-io=1 --with-mpi=1 --with-mpi-shared=1 --with-mpirun=/global/software/openmpi-1.6.5/intel/bin/mpiexec --with-mpi-compilers=1 --with-x=yes --with-blas-lapack-dir=/global/software/intel/composerxe/mkl/lib/intel64 --with-ptscotch=0 --with-x=1 --with-hdf5=1 --with-hdf5-dir=/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1 --with-netcdf=0 --with-fftw=1 --with-fftw-dir=/global/software/lib64/intel/fftw-3.3.3 --download-blacs=yes --download-scalapack=yes --download-superlu_dist=yes --download-mumps=yes --download-metis=yes --download-parmetis=yes --download-spooles=yes --download-cproto=yes --download-suitesparse=yes --download-hypre=yes --download-amd=yes --download-adifor=yes --download-euclid=yes --download-spai=yes --download-sprng=yes --download-ml=yes --download-boost=yes --download-triangle=yes --download-generator=yes --with-boost=1 --with-petsc4py=0 --with-numpy=1 exit 0 [3]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1172 in /tmp/petsc-3.6.2/src/mat/impls/aij/mpi/mumps/mumps.c [3]PETSC ERROR: #2 MatLUFactorNumeric() line 2946 in /tmp/petsc-3.6.2/src/mat/interface/matrix.c [3]PETSC ERROR: #3 PCSetUp_LU() line 152 in /tmp/petsc-3.6.2/src/ksp/pc/impls/factor/lu/lu.c [3]PETSC ERROR: #4 PCSetUp() line 983 in /tmp/petsc-3.6.2/src/ksp/pc/interface/precon.c [3]PETSC ERROR: #5 KSPSetUp() line 332 in /tmp/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c [3]PETSC ERROR: #6 KSPSolve() line 546 in /tmp/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c [4]PETSC ERROR: #2 MatLUFactorNumeric() line 2946 in /tmp/petsc-3.6.2/src/mat/interface/matrix.c [4]PETSC ERROR: #3 PCSetUp_LU() line 152 in /tmp/petsc-3.6.2/src/ksp/pc/impls/factor/lu/lu.c [4]PETSC ERROR: #4 PCSetUp() line 983 in /tmp/petsc-3.6.2/src/ksp/pc/interface/precon.c [4]PETSC ERROR: #5 KSPSetUp() line 332 in /tmp/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c [4]PETSC ERROR: #6 KSPSolve() line 546 in /tmp/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c [5]PETSC ERROR: #2 MatLUFactorNumeric() line 2946 in /tmp/petsc-3.6.2/src/mat/interface/matrix.c [5]PETSC ERROR: #3 PCSetUp_LU() line 152 in /tmp/petsc-3.6.2/src/ksp/pc/impls/factor/lu/lu.c [5]PETSC ERROR: #4 PCSetUp() line 983 in /tmp/petsc-3.6.2/src/ksp/pc/interface/precon.c [5]PETSC ERROR: #5 KSPSetUp() line 332 in /tmp/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c [5]PETSC ERROR: #6 KSPSolve() line 546 in /tmp/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c [7]PETSC ERROR: #4 PCSetUp() line 983 in /tmp/petsc-3.6.2/src/ksp/pc/interface/precon.c [7]PETSC ERROR: #5 KSPSetUp() line 332 in /tmp/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c [7]PETSC ERROR: #6 KSPSolve() line 546 in /tmp/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #5 KSPSetUp() line 332 in /tmp/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #6 KSPSolve() line 546 in /tmp/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: #5 KSPSetUp() line 332 in /tmp/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: #6 KSPSolve() line 546 in /tmp/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c --------------------- Error Message -------------------------------------------------------------- [6]PETSC ERROR: Error in external library [6]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFO(1)=-1, INFO(2)=0 [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [6]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 [6]PETSC ERROR: ../min3p_thcm on a arch-linux2-c-debug named pod26b15 by danyangs Tue Dec 1 11:29:00 2015 [6]PETSC ERROR: Configure options --prefix=/global/software/lib64/intel/petsc-3.6.2 --with-64-bit-pointers=0 --with-pthread=0 --with-pthreadclasses=0 --with-cc=/global/software/openmpi-1.6.5/intel/bin/mpicc --CFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-cxx=/global/software/openmpi-1.6.5/intel/bin/mpicxx --CXXFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-fc=/global/software/openmpi-1.6.5/intel/bin/mpif90 --FFLAGS="-O3 -axSSE4.2,SSE4.1 -xSSSE3 -I/global/software/intel/composerxe/mkl/include -I/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/include -I/global/software/lib64/intel/petsc-3.6.2/include" --with-cxx-dialect=C++11 --with-single-library=1 --with-shared-libraries --with-shared-ld=/global/software/openmpi-1.6.5/intel-2011/bin/mpicc --sharedLibraryFlags="-fpic -fPIC " --with-large-file-io=1 --with-mpi=1 --with-mpi-shared=1 --with-mpirun=/global/software/openmpi-1.6.5/intel/bin/mpiexec --with-mpi-compilers=1 --with-x=yes --with-blas-lapack-dir=/global/software/intel/composerxe/mkl/lib/intel64 --with-ptscotch=0 --with-x=1 --with-hdf5=1 --with-hdf5-dir=/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1 --with-netcdf=0 --with-fftw=1 --with-fftw-dir=/global/software/lib64/intel/fftw-3.3.3 --download-blacs=yes --download-scalapack=yes --download-superlu_dist=yes --download-mumps=yes --download-metis=yes --download-parmetis=yes --download-spooles=yes --download-cproto=yes --download-suitesparse=yes --download-hypre=yes --download-amd=yes --download-adifor=yes --download-euclid=yes --download-spai=yes --download-sprng=yes --download-ml=yes --download-boost=yes --download-triangle=yes --download-generator=yes --with-boost=1 --with-petsc4py=0 --with-numpy=1 exit 0 [6]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1172 in /tmp/petsc-3.6.2/src/mat/impls/aij/mpi/mumps/mumps.c [6]PETSC ERROR: #2 MatLUFactorNumeric() line 2946 in /tmp/petsc-3.6.2/src/mat/interface/matrix.c [6]PETSC ERROR: #3 PCSetUp_LU() line 152 in /tmp/petsc-3.6.2/src/ksp/pc/impls/factor/lu/lu.c [6]PETSC ERROR: #4 PCSetUp() line 983 in /tmp/petsc-3.6.2/src/ksp/pc/interface/precon.c [6]PETSC ERROR: #5 KSPSetUp() line 332 in /tmp/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c [6]PETSC ERROR: #6 KSPSolve() line 546 in /tmp/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 7 in communicator MPI_COMM_WORLD with errorcode 76. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpiexec has exited due to process rank 6 with PID 17428 on node pod26b15 exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpiexec (as reported here). -------------------------------------------------------------------------- [pod26b15:17421] 7 more processes have sent help message help-mpi-api.txt / mpi-abort [pod26b15:17421] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages From jychang48 at gmail.com Tue Dec 1 17:13:10 2015 From: jychang48 at gmail.com (Justin Chang) Date: Tue, 1 Dec 2015 16:13:10 -0700 Subject: [petsc-users] Issues with the Variational Inequality solver Message-ID: Hi all, So I am running into some issues with the SNESVINEWTONRSLS solver. I originally had this done in firedrake, but have ported it to petsc4py. Attached is the code as well as the two required input files. First issue, when I run the program like this: python testVI.py psiA_1 psiB_1 2 1 I get this error: Traceback (most recent call last): File "testVI.py", line 103, in snes.solve(None,X) File "PETSc/SNES.pyx", line 520, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:169677) petsc4py.PETSc.Error: error code 60 [0] SNESSolve() line 3992 in /Users/justin/Software/petsc/src/snes/interface/snes.c [0] SNESSolve_VINEWTONRSLS() line 507 in /Users/justin/Software/petsc/src/snes/impls/vi/rs/virs.c [0] KSPSetOperators() line 544 in /Users/justin/Software/petsc/src/ksp/ksp/interface/itcreate.c [0] PCSetOperators() line 1170 in /Users/justin/Software/petsc/src/ksp/pc/interface/precon.c [0] Nonconforming object sizes [0] Cannot change local size of Amat after use old sizes 2 2 new sizes 3 3 No such error occurred when this was ran in firedrake, though I did notice that -snes_view did indicate that some of the iterations had 2x2 instead of 3x3 matrices. Why is this happening? I thought I should be solving a 3x3 system each time so why would a 2x2 ever arise? More issues to come :) Thanks, Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: testVI.py Type: text/x-python-script Size: 3088 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: psiA_1 Type: application/octet-stream Size: 69448 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: psiB_1 Type: application/octet-stream Size: 68176 bytes Desc: not available URL: From bikash at umich.edu Tue Dec 1 17:18:51 2015 From: bikash at umich.edu (Bikash Kanungo) Date: Tue, 1 Dec 2015 18:18:51 -0500 Subject: [petsc-users] MatCreateShell with VecCreateGhost Message-ID: Hi, I want to implement a matrix-free KSP solve using user-defined MATOP_MULT and MATOP_MULT_TRANSPOSE. Each of these user-defined functions take in Mat A, Vec x and Vec y as arguments and store A*x in y. To perform A*x in the user defined fashion, each processor requires access to some non-local nodes of x. So, is there a way I can create these two vectors (x and y) using VecCreateGhost? Thanks, Bikash -- Bikash S. Kanungo PhD Student Computational Materials Physics Group Mechanical Engineering University of Michigan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Dec 1 18:47:07 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 1 Dec 2015 18:47:07 -0600 Subject: [petsc-users] MatCreateShell with VecCreateGhost In-Reply-To: References: Message-ID: <3FD057D2-8717-438C-856F-72D002B4AC41@mcs.anl.gov> > On Dec 1, 2015, at 5:18 PM, Bikash Kanungo wrote: > > Hi, > > I want to implement a matrix-free KSP solve using user-defined MATOP_MULT and MATOP_MULT_TRANSPOSE. Each of these user-defined functions take in Mat A, Vec x and Vec y as arguments and store A*x in y. To perform A*x in the user defined fashion, each processor requires access to some non-local nodes of x. So, is there a way I can create these two vectors (x and y) using VecCreateGhost? Yes, use VecCreateGhost(). Barry > > Thanks, > Bikash > > -- > Bikash S. Kanungo > PhD Student > Computational Materials Physics Group > Mechanical Engineering > University of Michigan > From bsmith at mcs.anl.gov Tue Dec 1 18:58:10 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 1 Dec 2015 18:58:10 -0600 Subject: [petsc-users] Issues with the Variational Inequality solver In-Reply-To: References: Message-ID: <521DF698-ACC4-4F5B-BDEE-558B2142DF57@mcs.anl.gov> > On Dec 1, 2015, at 5:13 PM, Justin Chang wrote: > > Hi all, > > So I am running into some issues with the SNESVINEWTONRSLS solver. I originally had this done in firedrake, but have ported it to petsc4py. Attached is the code as well as the two required input files. > > First issue, when I run the program like this: > > python testVI.py psiA_1 psiB_1 2 1 > > I get this error: > > Traceback (most recent call last): > File "testVI.py", line 103, in > snes.solve(None,X) > File "PETSc/SNES.pyx", line 520, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:169677) > petsc4py.PETSc.Error: error code 60 > [0] SNESSolve() line 3992 in /Users/justin/Software/petsc/src/snes/interface/snes.c > [0] SNESSolve_VINEWTONRSLS() line 507 in /Users/justin/Software/petsc/src/snes/impls/vi/rs/virs.c > [0] KSPSetOperators() line 544 in /Users/justin/Software/petsc/src/ksp/ksp/interface/itcreate.c > [0] PCSetOperators() line 1170 in /Users/justin/Software/petsc/src/ksp/pc/interface/precon.c > [0] Nonconforming object sizes > [0] Cannot change local size of Amat after use old sizes 2 2 new sizes 3 3 > > No such error occurred when this was ran in firedrake, though I did notice that -snes_view did indicate that some of the iterations had 2x2 instead of 3x3 matrices. Why is this happening? I thought I should be solving a 3x3 system each time so why would a 2x2 ever arise? Justin, For VI's the rs (reduced space solver) solves the linearized problem on a subset of the variables, thus the size of the linear system may change from iteration of Newton to the next iteration of Newton. In the rs solver we have ierr = ISEqual(vi->IS_inact_prev,vi->IS_inact,&isequal);CHKERRQ(ierr); if (!isequal) { ierr = SNESVIResetPCandKSP(snes,jac_inact_inact,prejac_inact_inact);CHKERRQ(ierr); } so what __should__ happen is that if the size of the reduced space changes KSPReset() is called on the KSP allowing the KSP/PC to handle a different sized system then before. But why doesn't it work? Why isn't KSPReset() called? Somehow the ISEqual is not working. Can you run the C debugger and put a breakpoint in the ISEqual() then check the inputs and see if it is correctly setting the isequal to false when it should? Each time the vi->IS_inact changes the KSPReset() should get called. You can check this in the debugger. Barry > > More issues to come :) > > Thanks, > Justin > From bikash at umich.edu Tue Dec 1 18:59:02 2015 From: bikash at umich.edu (Bikash Kanungo) Date: Tue, 1 Dec 2015 19:59:02 -0500 Subject: [petsc-users] MatCreateShell with VecCreateGhost In-Reply-To: <3FD057D2-8717-438C-856F-72D002B4AC41@mcs.anl.gov> References: <3FD057D2-8717-438C-856F-72D002B4AC41@mcs.anl.gov> Message-ID: Hi Barry, These vectors are internal to the KSP solve. How do I access the creation of these vectors? Regards, Bikash On Dec 1, 2015 7:47 PM, "Barry Smith" wrote: > > > On Dec 1, 2015, at 5:18 PM, Bikash Kanungo wrote: > > > > Hi, > > > > I want to implement a matrix-free KSP solve using user-defined > MATOP_MULT and MATOP_MULT_TRANSPOSE. Each of these user-defined functions > take in Mat A, Vec x and Vec y as arguments and store A*x in y. To perform > A*x in the user defined fashion, each processor requires access to some > non-local nodes of x. So, is there a way I can create these two vectors (x > and y) using VecCreateGhost? > > Yes, use VecCreateGhost(). > > Barry > > > > > > Thanks, > > Bikash > > > > -- > > Bikash S. Kanungo > > PhD Student > > Computational Materials Physics Group > > Mechanical Engineering > > University of Michigan > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Dec 1 19:00:46 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 1 Dec 2015 19:00:46 -0600 Subject: [petsc-users] MatCreateShell with VecCreateGhost In-Reply-To: <3FD057D2-8717-438C-856F-72D002B4AC41@mcs.anl.gov> References: <3FD057D2-8717-438C-856F-72D002B4AC41@mcs.anl.gov> Message-ID: <3DE5E5C5-C31B-44CE-8B22-E3B453EAE48B@mcs.anl.gov> Sorry, I could not control myself. So long as you pass vectors created with VecCreateGhost() to KSPSolve() the vectors passed to the multiply inside the solver will also be ghosted vectors. Send all output messages etc if it doesn't work for you Barry > On Dec 1, 2015, at 6:47 PM, Barry Smith wrote: > > >> On Dec 1, 2015, at 5:18 PM, Bikash Kanungo wrote: >> >> Hi, >> >> I want to implement a matrix-free KSP solve using user-defined MATOP_MULT and MATOP_MULT_TRANSPOSE. Each of these user-defined functions take in Mat A, Vec x and Vec y as arguments and store A*x in y. To perform A*x in the user defined fashion, each processor requires access to some non-local nodes of x. So, is there a way I can create these two vectors (x and y) using VecCreateGhost? > > Yes, use VecCreateGhost(). > > Barry > > >> >> Thanks, >> Bikash >> >> -- >> Bikash S. Kanungo >> PhD Student >> Computational Materials Physics Group >> Mechanical Engineering >> University of Michigan >> > From timothee.nicolas at gmail.com Tue Dec 1 19:56:39 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Wed, 2 Dec 2015 10:56:39 +0900 Subject: [petsc-users] MatCreateShell with VecCreateGhost In-Reply-To: <3DE5E5C5-C31B-44CE-8B22-E3B453EAE48B@mcs.anl.gov> References: <3FD057D2-8717-438C-856F-72D002B4AC41@mcs.anl.gov> <3DE5E5C5-C31B-44CE-8B22-E3B453EAE48B@mcs.anl.gov> Message-ID: Btw, If the vector X is created via a DMDA, then the vector localX and the pointer lx obtained via call DMGetLocalVector(da,localX,ierr) call DMGlobalToLocalBegin(da,X,INSERT_VALUES,localX,ierr) call DMGlobalToLocalEnd(da,X,INSERT_VALUES,localX,ierr) call VecGetArrayReadF90(localX,lx,ierr) are automatically ghosted, right ? Thx Timothee 2015-12-02 10:00 GMT+09:00 Barry Smith : > > Sorry, I could not control myself. > > So long as you pass vectors created with VecCreateGhost() to KSPSolve() > the vectors passed to the multiply inside the solver will also be ghosted > vectors. > > Send all output messages etc if it doesn't work for you > > Barry > > > > > On Dec 1, 2015, at 6:47 PM, Barry Smith wrote: > > > > > >> On Dec 1, 2015, at 5:18 PM, Bikash Kanungo wrote: > >> > >> Hi, > >> > >> I want to implement a matrix-free KSP solve using user-defined > MATOP_MULT and MATOP_MULT_TRANSPOSE. Each of these user-defined functions > take in Mat A, Vec x and Vec y as arguments and store A*x in y. To perform > A*x in the user defined fashion, each processor requires access to some > non-local nodes of x. So, is there a way I can create these two vectors (x > and y) using VecCreateGhost? > > > > Yes, use VecCreateGhost(). > > > > Barry > > > > > >> > >> Thanks, > >> Bikash > >> > >> -- > >> Bikash S. Kanungo > >> PhD Student > >> Computational Materials Physics Group > >> Mechanical Engineering > >> University of Michigan > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Dec 1 20:04:03 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 1 Dec 2015 20:04:03 -0600 Subject: [petsc-users] MatCreateShell with VecCreateGhost In-Reply-To: References: <3FD057D2-8717-438C-856F-72D002B4AC41@mcs.anl.gov> <3DE5E5C5-C31B-44CE-8B22-E3B453EAE48B@mcs.anl.gov> Message-ID: <16ECEE4D-F851-400F-8DE0-A966BAE3D63F@mcs.anl.gov> This is a pretty different thing. DMGetLocalVector(da,localX,ierr) gives a vector back with the correct amount of space for local ghosts and if you use DMDA it this the correct way to go. VecCreateGhost() creates a special vector that actually stores both the ghosted and nonghosted values in overlapping regions, but is totally inappropriate for DMDA. Barry > On Dec 1, 2015, at 7:56 PM, Timoth?e Nicolas wrote: > > Btw, > > If the vector X is created via a DMDA, then the vector localX and the pointer lx obtained via > > call DMGetLocalVector(da,localX,ierr) > call DMGlobalToLocalBegin(da,X,INSERT_VALUES,localX,ierr) > call DMGlobalToLocalEnd(da,X,INSERT_VALUES,localX,ierr) > call VecGetArrayReadF90(localX,lx,ierr) > > are automatically ghosted, right ? > > Thx > > Timothee > > > 2015-12-02 10:00 GMT+09:00 Barry Smith : > > Sorry, I could not control myself. > > So long as you pass vectors created with VecCreateGhost() to KSPSolve() the vectors passed to the multiply inside the solver will also be ghosted vectors. > > Send all output messages etc if it doesn't work for you > > Barry > > > > > On Dec 1, 2015, at 6:47 PM, Barry Smith wrote: > > > > > >> On Dec 1, 2015, at 5:18 PM, Bikash Kanungo wrote: > >> > >> Hi, > >> > >> I want to implement a matrix-free KSP solve using user-defined MATOP_MULT and MATOP_MULT_TRANSPOSE. Each of these user-defined functions take in Mat A, Vec x and Vec y as arguments and store A*x in y. To perform A*x in the user defined fashion, each processor requires access to some non-local nodes of x. So, is there a way I can create these two vectors (x and y) using VecCreateGhost? > > > > Yes, use VecCreateGhost(). > > > > Barry > > > > > >> > >> Thanks, > >> Bikash > >> > >> -- > >> Bikash S. Kanungo > >> PhD Student > >> Computational Materials Physics Group > >> Mechanical Engineering > >> University of Michigan > >> > > > > From bikash at umich.edu Wed Dec 2 08:43:31 2015 From: bikash at umich.edu (Bikash Kanungo) Date: Wed, 2 Dec 2015 09:43:31 -0500 Subject: [petsc-users] MatCreateShell with VecCreateGhost In-Reply-To: <16ECEE4D-F851-400F-8DE0-A966BAE3D63F@mcs.anl.gov> References: <3FD057D2-8717-438C-856F-72D002B4AC41@mcs.anl.gov> <3DE5E5C5-C31B-44CE-8B22-E3B453EAE48B@mcs.anl.gov> <16ECEE4D-F851-400F-8DE0-A966BAE3D63F@mcs.anl.gov> Message-ID: Thanks for your suggestion Barry. Regards, Bikash On Tue, Dec 1, 2015 at 9:04 PM, Barry Smith wrote: > > This is a pretty different thing. DMGetLocalVector(da,localX,ierr) gives > a vector back with the correct amount of space for local ghosts and if you > use DMDA it this the correct way to go. VecCreateGhost() creates a special > vector that actually stores both the ghosted and nonghosted values in > overlapping regions, but is totally inappropriate for DMDA. > > Barry > > > On Dec 1, 2015, at 7:56 PM, Timoth?e Nicolas > wrote: > > > > Btw, > > > > If the vector X is created via a DMDA, then the vector localX and the > pointer lx obtained via > > > > call DMGetLocalVector(da,localX,ierr) > > call DMGlobalToLocalBegin(da,X,INSERT_VALUES,localX,ierr) > > call DMGlobalToLocalEnd(da,X,INSERT_VALUES,localX,ierr) > > call VecGetArrayReadF90(localX,lx,ierr) > > > > are automatically ghosted, right ? > > > > Thx > > > > Timothee > > > > > > 2015-12-02 10:00 GMT+09:00 Barry Smith : > > > > Sorry, I could not control myself. > > > > So long as you pass vectors created with VecCreateGhost() to > KSPSolve() the vectors passed to the multiply inside the solver will also > be ghosted vectors. > > > > Send all output messages etc if it doesn't work for you > > > > Barry > > > > > > > > > On Dec 1, 2015, at 6:47 PM, Barry Smith wrote: > > > > > > > > >> On Dec 1, 2015, at 5:18 PM, Bikash Kanungo wrote: > > >> > > >> Hi, > > >> > > >> I want to implement a matrix-free KSP solve using user-defined > MATOP_MULT and MATOP_MULT_TRANSPOSE. Each of these user-defined functions > take in Mat A, Vec x and Vec y as arguments and store A*x in y. To perform > A*x in the user defined fashion, each processor requires access to some > non-local nodes of x. So, is there a way I can create these two vectors (x > and y) using VecCreateGhost? > > > > > > Yes, use VecCreateGhost(). > > > > > > Barry > > > > > > > > >> > > >> Thanks, > > >> Bikash > > >> > > >> -- > > >> Bikash S. Kanungo > > >> PhD Student > > >> Computational Materials Physics Group > > >> Mechanical Engineering > > >> University of Michigan > > >> > > > > > > > > > -- Bikash S. Kanungo PhD Student Computational Materials Physics Group Mechanical Engineering University of Michigan -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Dec 2 10:39:52 2015 From: hzhang at mcs.anl.gov (Hong) Date: Wed, 2 Dec 2015 10:39:52 -0600 Subject: [petsc-users] Error reported by MUMPS in numerical factorization phase In-Reply-To: <565E26F0.6050109@gmail.com> References: <565E26F0.6050109@gmail.com> Message-ID: Danyang : > > My code fails due to the error in external library. It works fine for the > previous 2000+ timesteps but then crashes. > > [4]PETSC ERROR: Error in external library > [4]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: > INFO(1)=-1, INFO(2)=0 > This simply says an error occurred in proc[0] during numerical factorization, which usually either encounter a zeropivot or run out of memory. Since it is at a later timesteps, which I guess you reuse matrix factor, zeropivot might be the problem. Is possible to run it in debugging mode? In this way, mumps would dump out more information. > > Then I tried the same simulation on another machine using the same number > of processors, it does not fail. > Does this machine have larger memory? Hong -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Wed Dec 2 11:17:53 2015 From: danyang.su at gmail.com (Danyang Su) Date: Wed, 2 Dec 2015 09:17:53 -0800 Subject: [petsc-users] Error reported by MUMPS in numerical factorization phase In-Reply-To: References: <565E26F0.6050109@gmail.com> Message-ID: <565F27C1.1060309@gmail.com> Hi Hong, It's not easy to run in debugging mode as the cluster does not have petsc installed using debug mode. Restart the case from the crashing time does not has the problem. So if I want to detect this error, I need to start the simulation from beginning which takes hours in the cluster. Do you mean I need to redo symbolic factorization? For now, I only do factorization once at the first timestep and then reuse it. Some of the code is shown below. if (timestep == 1) then call PCFactorSetMatSolverPackage(pc_flow,MATSOLVERMUMPS,ierr) CHKERRQ(ierr) call PCFactorSetUpMatSolverPackage(pc_flow,ierr) CHKERRQ(ierr) call PCFactorGetMatrix(pc_flow,a_flow_j,ierr) CHKERRQ(ierr) end if call KSPSolve(ksp_flow,b_flow,x_flow,ierr) CHKERRQ(ierr) Thanks, Danyang On 15-12-02 08:39 AM, Hong wrote: > Danyang : > > My code fails due to the error in external library. It works fine > for the previous 2000+ timesteps but then crashes. > > [4]PETSC ERROR: Error in external library > [4]PETSC ERROR: Error reported by MUMPS in numerical factorization > phase: INFO(1)=-1, INFO(2)=0 > > This simply says an error occurred in proc[0] during numerical > factorization, which usually either encounter a zeropivot or run out > of memory. Since it is at a later timesteps, which I guess you reuse > matrix factor, zeropivot might be the problem. > Is possible to run it in debugging mode? In this way, mumps would dump > out more information. > > > Then I tried the same simulation on another machine using the same > number of processors, it does not fail. > > Does this machine have larger memory? > > Hong -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Dec 2 12:26:56 2015 From: hzhang at mcs.anl.gov (Hong) Date: Wed, 2 Dec 2015 12:26:56 -0600 Subject: [petsc-users] Error reported by MUMPS in numerical factorization phase In-Reply-To: <565F27C1.1060309@gmail.com> References: <565E26F0.6050109@gmail.com> <565F27C1.1060309@gmail.com> Message-ID: Danyang: It is likely a zero pivot. I'm adding a feature to petsc. When matrix factorization fails, computation continues with error information stored in ksp->reason=DIVERGED_PCSETUP_FAILED. For your timestepping code, you may able to automatically reduce timestep and continue your simulation. Do you want to test it? If so, you need install petsc-dev with my branch hzhang/matpackage-erroriffpe on your cluster. We may merge this branch to petsc-master soon. > > It's not easy to run in debugging mode as the cluster does not have petsc > installed using debug mode. Restart the case from the crashing time does > not has the problem. So if I want to detect this error, I need to start the > simulation from beginning which takes hours in the cluster. > This is why we are adding this new feature. > > Do you mean I need to redo symbolic factorization? For now, I only do > factorization once at the first timestep and then reuse it. Some of the > code is shown below. > > if (timestep == 1) then > call PCFactorSetMatSolverPackage(pc_flow,MATSOLVERMUMPS,ierr) > CHKERRQ(ierr) > > call PCFactorSetUpMatSolverPackage(pc_flow,ierr) > CHKERRQ(ierr) > > call PCFactorGetMatrix(pc_flow,a_flow_j,ierr) > CHKERRQ(ierr) > end if > > call KSPSolve(ksp_flow,b_flow,x_flow,ierr) > CHKERRQ(ierr) > I do not think you need to change this part of code. Does you code check convergence at each time step? Hong > > > On 15-12-02 08:39 AM, Hong wrote: > > Danyang : >> >> My code fails due to the error in external library. It works fine for the >> previous 2000+ timesteps but then crashes. >> >> [4]PETSC ERROR: Error in external library >> [4]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: >> INFO(1)=-1, INFO(2)=0 >> > > This simply says an error occurred in proc[0] during numerical > factorization, which usually either encounter a zeropivot or run out of > memory. Since it is at a later timesteps, which I guess you reuse matrix > factor, zeropivot might be the problem. > Is possible to run it in debugging mode? In this way, mumps would dump out > more information. > >> >> Then I tried the same simulation on another machine using the same number >> of processors, it does not fail. >> > Does this machine have larger memory? > > Hong > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adlinds3 at ncsu.edu Wed Dec 2 12:45:21 2015 From: adlinds3 at ncsu.edu (Alex Lindsay) Date: Wed, 2 Dec 2015 13:45:21 -0500 Subject: [petsc-users] Recommended DOF/processor for direct solver Message-ID: <565F3C41.3090502@ncsu.edu> Hi, I've read on your FAQ page that a good rule of thumb for parallelization is 20k DOF/processor. Does this apply equally to direct solves as well as iterative? May be a dumb question... Alex From danyang.su at gmail.com Wed Dec 2 13:36:07 2015 From: danyang.su at gmail.com (Danyang Su) Date: Wed, 2 Dec 2015 11:36:07 -0800 Subject: [petsc-users] Error reported by MUMPS in numerical factorization phase In-Reply-To: References: <565E26F0.6050109@gmail.com> <565F27C1.1060309@gmail.com> Message-ID: <565F4827.50506@gmail.com> Hi Hong, Thank. I can test it but it may takes some time to install petsc-dev on the cluster. I will try more cases to see if I can get this error on my local machine which is much more convenient for me to test in debug mode. So far, the error does not occur on my local machine using the same code, the same petsc-3.6.2 version, the same case and the same number of processors. The system and petsc configuration is different. Regards, Danyang On 15-12-02 10:26 AM, Hong wrote: > Danyang: > It is likely a zero pivot. I'm adding a feature to petsc. When matrix > factorization fails, computation continues with error information > stored in > ksp->reason=DIVERGED_PCSETUP_FAILED. > For your timestepping code, you may able to automatically reduce > timestep and continue your simulation. > > Do you want to test it? If so, you need install petsc-dev with my > branch hzhang/matpackage-erroriffpe on your cluster. We may merge this > branch to petsc-master soon. > > > It's not easy to run in debugging mode as the cluster does not > have petsc installed using debug mode. Restart the case from the > crashing time does not has the problem. So if I want to detect > this error, I need to start the simulation from beginning which > takes hours in the cluster. > > > This is why we are adding this new feature. > > > Do you mean I need to redo symbolic factorization? For now, I only > do factorization once at the first timestep and then reuse it. > Some of the code is shown below. > > if (timestep == 1) then > call > PCFactorSetMatSolverPackage(pc_flow,MATSOLVERMUMPS,ierr) > CHKERRQ(ierr) > > call PCFactorSetUpMatSolverPackage(pc_flow,ierr) > CHKERRQ(ierr) > > call PCFactorGetMatrix(pc_flow,a_flow_j,ierr) > CHKERRQ(ierr) > end if > > call KSPSolve(ksp_flow,b_flow,x_flow,ierr) > CHKERRQ(ierr) > > > I do not think you need to change this part of code. > Does you code check convergence at each time step? > > Hong > > > > On 15-12-02 08:39 AM, Hong wrote: >> Danyang : >> >> My code fails due to the error in external library. It works >> fine for the previous 2000+ timesteps but then crashes. >> >> [4]PETSC ERROR: Error in external library >> [4]PETSC ERROR: Error reported by MUMPS in numerical >> factorization phase: INFO(1)=-1, INFO(2)=0 >> >> This simply says an error occurred in proc[0] during numerical >> factorization, which usually either encounter a zeropivot or run >> out of memory. Since it is at a later timesteps, which I guess >> you reuse matrix factor, zeropivot might be the problem. >> Is possible to run it in debugging mode? In this way, mumps would >> dump out more information. >> >> >> Then I tried the same simulation on another machine using the >> same number of processors, it does not fail. >> >> Does this machine have larger memory? >> >> Hong > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Dec 2 13:47:32 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 2 Dec 2015 13:47:32 -0600 Subject: [petsc-users] Recommended DOF/processor for direct solver In-Reply-To: <565F3C41.3090502@ncsu.edu> References: <565F3C41.3090502@ncsu.edu> Message-ID: <8689E354-A4AD-41DB-BCA5-ACCC4B41B815@mcs.anl.gov> Not a dumb question but maybe there is no smart answer. I have never seen direct solver packages speak in these terms. IMHO for direct solvers it is more a question of wanting to make sure the part of the factored matrix on each process fits in the memory of that process. That is with direct solvers you generally use more processes (and hence more memory) to do larger problems, not to do smaller problems faster. Since the amount of fill in a direct solver depends very strongly on the specific nonzero structure of the matrix there is no formula that goes from "number of unknowns per process" to "how much memory is needed to store the local factor". Barry > On Dec 2, 2015, at 12:45 PM, Alex Lindsay wrote: > > Hi, > > I've read on your FAQ page that a good rule of thumb for parallelization is 20k DOF/processor. Does this apply equally to direct solves as well as iterative? May be a dumb question... > > Alex From jed at jedbrown.org Wed Dec 2 13:51:32 2015 From: jed at jedbrown.org (Jed Brown) Date: Wed, 02 Dec 2015 12:51:32 -0700 Subject: [petsc-users] Recommended DOF/processor for direct solver In-Reply-To: <565F3C41.3090502@ncsu.edu> References: <565F3C41.3090502@ncsu.edu> Message-ID: <87h9k0vdbv.fsf@jedbrown.org> Alex Lindsay writes: > I've read on your FAQ page that a good rule of thumb for parallelization > is 20k DOF/processor. Does this apply equally to direct solves as well > as iterative? May be a dumb question... Depends on dimension and discretization, but no simple rule of thumb works because direct solvers are not scalable (in terms of memory or time). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jychang48 at gmail.com Wed Dec 2 13:56:46 2015 From: jychang48 at gmail.com (Justin Chang) Date: Wed, 2 Dec 2015 12:56:46 -0700 Subject: [petsc-users] Issues with the Variational Inequality solver In-Reply-To: <521DF698-ACC4-4F5B-BDEE-558B2142DF57@mcs.anl.gov> References: <521DF698-ACC4-4F5B-BDEE-558B2142DF57@mcs.anl.gov> Message-ID: Barry, So I do not believe there is any problem with the ISEqual() per se, because what I am doing is I am solving 5151 different VI problem using the same SNES/KSP/PC/Mat. That is, I am not "resetting" these data structures once I am done with one problem and move on to the next. If I ran each case individually, I get no error; the error comes when I run these problems sequentially without resetting the SNES after each problem. I haven't run the C debugger or done any of that yet, but could this be a plausible explanation for my error? Originally I was thinking about creating/destroying SNES for each problem but I was wondering if that might affect performance. Thanks, Justin On Tue, Dec 1, 2015 at 5:58 PM, Barry Smith wrote: > > > On Dec 1, 2015, at 5:13 PM, Justin Chang wrote: > > > > Hi all, > > > > So I am running into some issues with the SNESVINEWTONRSLS solver. I > originally had this done in firedrake, but have ported it to petsc4py. > Attached is the code as well as the two required input files. > > > > First issue, when I run the program like this: > > > > python testVI.py psiA_1 psiB_1 2 1 > > > > I get this error: > > > > Traceback (most recent call last): > > File "testVI.py", line 103, in > > snes.solve(None,X) > > File "PETSc/SNES.pyx", line 520, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:169677) > > petsc4py.PETSc.Error: error code 60 > > [0] SNESSolve() line 3992 in > /Users/justin/Software/petsc/src/snes/interface/snes.c > > [0] SNESSolve_VINEWTONRSLS() line 507 in > /Users/justin/Software/petsc/src/snes/impls/vi/rs/virs.c > > [0] KSPSetOperators() line 544 in > /Users/justin/Software/petsc/src/ksp/ksp/interface/itcreate.c > > [0] PCSetOperators() line 1170 in > /Users/justin/Software/petsc/src/ksp/pc/interface/precon.c > > [0] Nonconforming object sizes > > [0] Cannot change local size of Amat after use old sizes 2 2 new sizes 3 > 3 > > > > No such error occurred when this was ran in firedrake, though I did > notice that -snes_view did indicate that some of the iterations had 2x2 > instead of 3x3 matrices. Why is this happening? I thought I should be > solving a 3x3 system each time so why would a 2x2 ever arise? > > Justin, > > For VI's the rs (reduced space solver) solves the linearized problem on > a subset of the variables, thus the size of the linear system may change > from iteration of Newton to the next iteration of Newton. > > In the rs solver we have > > ierr = ISEqual(vi->IS_inact_prev,vi->IS_inact,&isequal);CHKERRQ(ierr); > if (!isequal) { > ierr = > SNESVIResetPCandKSP(snes,jac_inact_inact,prejac_inact_inact);CHKERRQ(ierr); > } > > so what __should__ happen is that if the size of the reduced space changes > KSPReset() is called on the KSP allowing the KSP/PC to handle a different > sized system then before. But why doesn't it work? Why isn't KSPReset() > called? Somehow the ISEqual is not working. > > Can you run the C debugger and put a breakpoint in the ISEqual() then > check the inputs and see if it is correctly setting the isequal to false > when it should? Each time the vi->IS_inact changes the KSPReset() should > get called. You can check this in the debugger. > > Barry > > > > > More issues to come :) > > > > Thanks, > > Justin > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Dec 2 14:36:54 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 2 Dec 2015 14:36:54 -0600 Subject: [petsc-users] Issues with the Variational Inequality solver In-Reply-To: References: <521DF698-ACC4-4F5B-BDEE-558B2142DF57@mcs.anl.gov> Message-ID: <1F85349D-7AAE-4FC3-A31E-B9232BBDC8FA@mcs.anl.gov> > On Dec 2, 2015, at 1:56 PM, Justin Chang wrote: > > Barry, > > So I do not believe there is any problem with the ISEqual() per se, because what I am doing is I am solving 5151 different VI problem using the same SNES/KSP/PC/Mat. That is, I am not "resetting" these data structures once I am done with one problem and move on to the next. If I ran each case individually, I get no error; the error comes when I run these problems sequentially without resetting the SNES after each problem. > > I haven't run the C debugger or done any of that yet, but could this be a plausible explanation for my error? This is exactly the issue. Independent of VIs if you change the size of a problem you pass to SNES you need to call SNESReset() between each call of a different sizes. > Originally I was thinking about creating/destroying SNES for each problem but I was wondering if that might affect performance. Using SNESReset() is the way you should do it. Creating and destroying it each time should only be a tiny, immeasurably slower. Barry > > Thanks, > Justin > > On Tue, Dec 1, 2015 at 5:58 PM, Barry Smith wrote: > > > On Dec 1, 2015, at 5:13 PM, Justin Chang wrote: > > > > Hi all, > > > > So I am running into some issues with the SNESVINEWTONRSLS solver. I originally had this done in firedrake, but have ported it to petsc4py. Attached is the code as well as the two required input files. > > > > First issue, when I run the program like this: > > > > python testVI.py psiA_1 psiB_1 2 1 > > > > I get this error: > > > > Traceback (most recent call last): > > File "testVI.py", line 103, in > > snes.solve(None,X) > > File "PETSc/SNES.pyx", line 520, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:169677) > > petsc4py.PETSc.Error: error code 60 > > [0] SNESSolve() line 3992 in /Users/justin/Software/petsc/src/snes/interface/snes.c > > [0] SNESSolve_VINEWTONRSLS() line 507 in /Users/justin/Software/petsc/src/snes/impls/vi/rs/virs.c > > [0] KSPSetOperators() line 544 in /Users/justin/Software/petsc/src/ksp/ksp/interface/itcreate.c > > [0] PCSetOperators() line 1170 in /Users/justin/Software/petsc/src/ksp/pc/interface/precon.c > > [0] Nonconforming object sizes > > [0] Cannot change local size of Amat after use old sizes 2 2 new sizes 3 3 > > > > No such error occurred when this was ran in firedrake, though I did notice that -snes_view did indicate that some of the iterations had 2x2 instead of 3x3 matrices. Why is this happening? I thought I should be solving a 3x3 system each time so why would a 2x2 ever arise? > > Justin, > > For VI's the rs (reduced space solver) solves the linearized problem on a subset of the variables, thus the size of the linear system may change from iteration of Newton to the next iteration of Newton. > > In the rs solver we have > > ierr = ISEqual(vi->IS_inact_prev,vi->IS_inact,&isequal);CHKERRQ(ierr); > if (!isequal) { > ierr = SNESVIResetPCandKSP(snes,jac_inact_inact,prejac_inact_inact);CHKERRQ(ierr); > } > > so what __should__ happen is that if the size of the reduced space changes KSPReset() is called on the KSP allowing the KSP/PC to handle a different sized system then before. But why doesn't it work? Why isn't KSPReset() called? Somehow the ISEqual is not working. > > Can you run the C debugger and put a breakpoint in the ISEqual() then check the inputs and see if it is correctly setting the isequal to false when it should? Each time the vi->IS_inact changes the KSPReset() should get called. You can check this in the debugger. > > Barry > > > > > More issues to come :) > > > > Thanks, > > Justin > > > > From sdettrick at trialphaenergy.com Wed Dec 2 15:39:53 2015 From: sdettrick at trialphaenergy.com (Sean Dettrick) Date: Wed, 2 Dec 2015 13:39:53 -0800 Subject: [petsc-users] Petsc Draw to pixmap Message-ID: <10862A58-63D0-4CB3-B194-64A1DF1E08DD@trialphaenergy.com> Hi, I?d like to have petsc draw output to a pixmap, but have been unable to install afterstep on my Mac. Direct download of the tar files from http://www.afterstep.org/afterimage/getcode.php failed, as did cvs checkout. It seems the target sites may not exist at present. Is there another way to install afterstep? Alternatively, is there a way to manually redirect the X11 output using something like XCreatePixmap? Thanks! Sean Dettrick -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Dec 2 15:45:00 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 2 Dec 2015 15:45:00 -0600 Subject: [petsc-users] Petsc Draw to pixmap In-Reply-To: <10862A58-63D0-4CB3-B194-64A1DF1E08DD@trialphaenergy.com> References: <10862A58-63D0-4CB3-B194-64A1DF1E08DD@trialphaenergy.com> Message-ID: <47E26DAB-01EB-4CF0-83F9-75AAC04BDDC7@mcs.anl.gov> Sean, I have never been able to build from that source either. I used homebrew to install it and its worked for several years. I also tried to see if one could pull the image from the X pixel map and it looked like a huge project (the pixmap stuff is not as trivial as one would think it would be) so I gave up on that and started using afterimage (which essentially does all the for you). Barry > On Dec 2, 2015, at 3:39 PM, Sean Dettrick wrote: > > Hi, > > I?d like to have petsc draw output to a pixmap, but have been unable to install afterstep on my Mac. Direct download of the tar files from http://www.afterstep.org/afterimage/getcode.php failed, as did cvs checkout. It seems the target sites may not exist at present. > > Is there another way to install afterstep? > > Alternatively, is there a way to manually redirect the X11 output using something like XCreatePixmap? > > Thanks! > Sean Dettrick From dsu at eos.ubc.ca Wed Dec 2 16:30:00 2015 From: dsu at eos.ubc.ca (Danyang Su) Date: Wed, 2 Dec 2015 14:30:00 -0800 Subject: [petsc-users] SuperLU convergence problem Message-ID: <565F70E8.6030300@eos.ubc.ca> Dear All, I got some strange solutions when using SuperLU for a complex reactive transport simulation. Before the timestep 28, everything works fine, for both sequential version and parallel version. Starting from timestep 29, the timestep cannot increase for SuperLU solver, for both sequential version and parallel version. To check this problem, I dumped all the solutions using the sequential iterative solver (Step 1). Then I rerun this case using iterative solver (external solver, not from PETSc), SuperLU (from PETSc) and MUMPS (from PETSc) (Step 2). At every timestep and newton iteration, the solution is exported, and then the solution is replaced with the one from the sequential iterative solver from Step 1. In this case, all the matrices and rhs are identical for the three different solvers. To my surprising, solutions from SuperLU at timestep 29 seems not correct for the first 4 Newton iterations, but the solutions from iteration solver and MUMPS are correct. Please find all the matrices, rhs and solutions at timestep 29 via the link below. The data is a bit large so that I just share it through Dropbox. A piece of matlab code to read these data and then computer the norm has also been attached. _https://www.dropbox.com/s/rr8ueysgflmxs7h/results-check.tar.gz?dl=0_ Below is a summary of the norm from the three solvers at timestep 29, newton iteration 1 to 5. Timestep 29 Norm of residual seq 1.661321e-09, superlu 1.657103e+04, mumps 3.731225e-11 Norm of residual seq 1.753079e-09, superlu 6.675467e+02, mumps 1.509919e-13 Norm of residual seq 4.914971e-10, superlu 1.236362e-01, mumps 2.139303e-17 Norm of residual seq 3.532769e-10, superlu 1.304670e-04, mumps 5.387000e-20 Norm of residual seq 3.885629e-10, superlu 2.754876e-07, mumps 4.108675e-21 Would anybody please check if SuperLU can solve these matrices? Another possibility is that something is wrong in my own code. But so far, I cannot find any problem in my code since the same code works fine if I using iterative solver or direct solver MUMPS. But for other cases I have tested, all these solvers work fine. Please let me know if I did not write down the problem clearly. Thanks, Danyang -------------- next part -------------- An HTML attachment was scrubbed... URL: From yongle.du at gmail.com Wed Dec 2 17:09:13 2015 From: yongle.du at gmail.com (DU Yongle) Date: Wed, 2 Dec 2015 18:09:13 -0500 Subject: [petsc-users] Matrix-free functions for SNES Message-ID: Dear Sir/Madam: Sorry to bother, but I didn't find any information I need to implement matrix-free functions for SNES after a long time search of the manual and sample codes. Suppose I am solving nonlinear equation systems F(x) = 0, I have two choices: (1) MatCreateMFFD(snes, matJ): This calculate matJ * x with a finite difference scheme. I can create a function to calculate matJ * x, and link it to SNES by SNESSetJacobian. However, I do not prefer this feature because I am calculating the mat-vec product in another way. (2) MatCreateMF(snes, matJ): Here, matJ is not defined explicitly. It is some operations (like mat-vec product). must be linked to SNES, and must be linked to SNES through SNESSetJacobian. For KSP solvers, I know that a matshell type can be created and a mat-vec product can be assigned to matJ in this way: MatCreateShell( ....) MatShellSetOperation(....) KSPSetOperators(ksp, matJ, matJ) How to do the similar operations for SNES if matJ is already created via MatCreateMF? How to define the mat size and mat-vec product? What other operations are required in a customized function in order to link it to SNES via SNESSetJacobian? Thanks a for your help. Yongle Du -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Dec 2 17:14:09 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 2 Dec 2015 17:14:09 -0600 Subject: [petsc-users] Matrix-free functions for SNES In-Reply-To: References: Message-ID: <59EAEF15-FFA6-428C-86F4-A5FA49C134A8@mcs.anl.gov> Do "what you would do for KSP" but with "SNES" so use > MatCreateShell( ....) > MatShellSetOperation(....) SNESSetJacobian() and pass in your shell matrix as the Jacobian. > On Dec 2, 2015, at 5:09 PM, DU Yongle wrote: > > Dear Sir/Madam: > > Sorry to bother, but I didn't find any information I need to implement matrix-free functions for SNES after a long time search of the manual and sample codes. > > Suppose I am solving nonlinear equation systems F(x) = 0, I have two choices: > > (1) MatCreateMFFD(snes, matJ): > This calculate matJ * x with a finite difference scheme. I can create a function to calculate matJ * x, and link it to SNES by SNESSetJacobian. However, I do not prefer this feature because I am calculating the mat-vec product in another way. > > (2) MatCreateMF(snes, matJ): > Here, matJ is not defined explicitly. It is some operations (like mat-vec product). must be linked to SNES, and must be linked to SNES through SNESSetJacobian. > > For KSP solvers, I know that a matshell type can be created and a mat-vec product can be assigned to matJ in this way: > MatCreateShell( ....) > MatShellSetOperation(....) > KSPSetOperators(ksp, matJ, matJ) > > How to do the similar operations for SNES if matJ is already created via MatCreateMF? You would not use MatCreateMF() in your case. It only does finite difference scheme which is not what you want so why would you even consider it? > How to define the mat size and mat-vec product? What other operations are required in a customized function in order to link it to SNES via SNESSetJacobian? > > Thanks a for your help. > > Yongle Du > > > > From a.croucher at auckland.ac.nz Wed Dec 2 22:09:34 2015 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Thu, 03 Dec 2015 17:09:34 +1300 Subject: [petsc-users] SNES function domain error Message-ID: <565FC07E.4000805@auckland.ac.nz> hi I am trying to use SNESSetFunctionDomainError() to make SNES abort when my solution vector goes out of the function domain. However, it's not always working. Specifically, I check the solution in a function called after the linesearch (set using SNESLineSearchSetPostCheck()). If it has gone out of the function domain I call SNESSetFunctionDomainError(). (I also check for phase changes at this point and do variable switching if necessary, which seems to work fine as long as I only use SNESLINESEARCHBASIC.) This causes the linesearch reason to return SNES_LINESEARCH_FAILED_DOMAIN. In SNESSolve_NEWTONLS() (ls.c:251) it checks the linesearch reason, but then also checks if (snes->stol*xnorm > ynorm). If that is true then it resets the snes->reason to SNES_CONVERGED_SNORM_RELATIVE and returns, i.e. it overrides the function domain error. So it thinks everything is fine, even though the 'converged' solution is out of the function domain. Is this how it's meant to work? If so, what can I do to ensure the SNES always aborts if the solution goes out of the function domain? At present it's only working if (snes->stol*xnorm <= ynorm) as well. Cheers, Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher at auckland.ac.nz tel: +64 (0)9 923 84611 From bsmith at mcs.anl.gov Wed Dec 2 23:43:41 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 2 Dec 2015 23:43:41 -0600 Subject: [petsc-users] SNES function domain error In-Reply-To: <565FC07E.4000805@auckland.ac.nz> References: <565FC07E.4000805@auckland.ac.nz> Message-ID: <5DD4C98F-F57E-46FF-917B-E315C9BC13D7@mcs.anl.gov> Set the -snes_stol to be so small that it doesn't trigger this reason? We are working on improving the "failure" modes of KSP/SNES so hopefully we'll have a better handle this type of thing in a few months. > On Dec 2, 2015, at 10:09 PM, Adrian Croucher wrote: > > hi > > I am trying to use SNESSetFunctionDomainError() to make SNES abort when my solution vector goes out of the function domain. However, it's not always working. > > Specifically, I check the solution in a function called after the linesearch (set using SNESLineSearchSetPostCheck()). If it has gone out of the function domain I call SNESSetFunctionDomainError(). (I also check for phase changes at this point and do variable switching if necessary, which seems to work fine as long as I only use SNESLINESEARCHBASIC.) I absolutely do NOT recommend doing variable switching with SNES. Our Newton solvers are not designed with variable switching in mind so you get these absurd limitations like needing SNESLINESEARCHBASIC It would be great if someone who understood variable switching would either 1) contribute a Newton that supports variable switching to PETSc or 2) enhance the current SNESSolve_NewtonLS to support variable switching (I don't know which is easier). Barry > > This causes the linesearch reason to return SNES_LINESEARCH_FAILED_DOMAIN. > > In SNESSolve_NEWTONLS() (ls.c:251) it checks the linesearch reason, but then also checks if (snes->stol*xnorm > ynorm). If that is true then it resets the snes->reason to SNES_CONVERGED_SNORM_RELATIVE and returns, i.e. it overrides the function domain error. So it thinks everything is fine, even though the 'converged' solution is out of the function domain. > > Is this how it's meant to work? If so, what can I do to ensure the SNES always aborts if the solution goes out of the function domain? At present it's only working if (snes->stol*xnorm <= ynorm) as well. > > Cheers, Adrian > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 84611 > From sdettrick at trialphaenergy.com Thu Dec 3 01:00:47 2015 From: sdettrick at trialphaenergy.com (Sean Dettrick) Date: Wed, 2 Dec 2015 23:00:47 -0800 Subject: [petsc-users] Petsc Draw to pixmap In-Reply-To: <47E26DAB-01EB-4CF0-83F9-75AAC04BDDC7@mcs.anl.gov> References: <10862A58-63D0-4CB3-B194-64A1DF1E08DD@trialphaenergy.com> <47E26DAB-01EB-4CF0-83F9-75AAC04BDDC7@mcs.anl.gov> Message-ID: <049242DC-BD76-454A-B16F-13C781281370@trialphaenergy.com> Hi Barry, Thanks for the amazingly fast reply. I installed home-brew but then was unable to find afterstep available as a package. The closest thing I could find was asterm (after step terminal emulator. Do you remember if you installed it from a canonical repo? I did find afterstep on github, but had trouble installing it on the mac. Meanwhile, a colleague has given me a python script, so I?ll probably use that instead for now. Thanks again, Sean > On Dec 2, 2015, at 1:45 PM, Barry Smith wrote: > > > Sean, > > I have never been able to build from that source either. > > I used homebrew to install it and its worked for several years. > > I also tried to see if one could pull the image from the X pixel map and it looked like a huge project (the pixmap stuff is not as trivial as one would think it would be) so I gave up on that and started using afterimage (which essentially does all the for you). > > Barry > >> On Dec 2, 2015, at 3:39 PM, Sean Dettrick wrote: >> >> Hi, >> >> I?d like to have petsc draw output to a pixmap, but have been unable to install afterstep on my Mac. Direct download of the tar files from http://www.afterstep.org/afterimage/getcode.php failed, as did cvs checkout. It seems the target sites may not exist at present. >> >> Is there another way to install afterstep? >> >> Alternatively, is there a way to manually redirect the X11 output using something like XCreatePixmap? >> >> Thanks! >> Sean Dettrick > From bsmith at mcs.anl.gov Thu Dec 3 01:10:31 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 3 Dec 2015 01:10:31 -0600 Subject: [petsc-users] Petsc Draw to pixmap In-Reply-To: <049242DC-BD76-454A-B16F-13C781281370@trialphaenergy.com> References: <10862A58-63D0-4CB3-B194-64A1DF1E08DD@trialphaenergy.com> <47E26DAB-01EB-4CF0-83F9-75AAC04BDDC7@mcs.anl.gov> <049242DC-BD76-454A-B16F-13C781281370@trialphaenergy.com> Message-ID: I am such an idiot. No I did build it from source I forgot the comment in afterimage.py After image is available from http://www.afterstep.org/afterimage/getcode.php # # It is used by the PetscDrawSetSave() routine to save X windows graphics to files # # If installing on an Apple make sure to read the details on PetscDrawSetSave manual # page before installing # from that page I found If X windows generates an error message about X_CreateWindow() failing then Afterimage was installed without X windows. Reinstall Afterimage using the ./configure flags --x-includes=/pathtoXincludes --x-libraries=/pathtoXlibraries For example under Mac OS X Mountain Lion --x-includes=/opt/X11/include -x-libraries=/opt/X11/lib Did you try all this stuff? Barry > On Dec 3, 2015, at 1:00 AM, Sean Dettrick wrote: > > Hi Barry, > > Thanks for the amazingly fast reply. I installed home-brew but then was unable to find afterstep available as a package. The closest thing I could find was asterm (after step terminal emulator. Do you remember if you installed it from a canonical repo? > > I did find afterstep on github, but had trouble installing it on the mac. > > Meanwhile, a colleague has given me a python script, so I?ll probably use that instead for now. > > Thanks again, > Sean > >> On Dec 2, 2015, at 1:45 PM, Barry Smith wrote: >> >> >> Sean, >> >> I have never been able to build from that source either. >> >> I used homebrew to install it and its worked for several years. >> >> I also tried to see if one could pull the image from the X pixel map and it looked like a huge project (the pixmap stuff is not as trivial as one would think it would be) so I gave up on that and started using afterimage (which essentially does all the for you). >> >> Barry >> >>> On Dec 2, 2015, at 3:39 PM, Sean Dettrick wrote: >>> >>> Hi, >>> >>> I?d like to have petsc draw output to a pixmap, but have been unable to install afterstep on my Mac. Direct download of the tar files from http://www.afterstep.org/afterimage/getcode.php failed, as did cvs checkout. It seems the target sites may not exist at present. >>> >>> Is there another way to install afterstep? >>> >>> Alternatively, is there a way to manually redirect the X11 output using something like XCreatePixmap? >>> >>> Thanks! >>> Sean Dettrick >> > From sdettrick at trialphaenergy.com Thu Dec 3 02:06:38 2015 From: sdettrick at trialphaenergy.com (Sean Dettrick) Date: Thu, 3 Dec 2015 00:06:38 -0800 Subject: [petsc-users] Petsc Draw to pixmap In-Reply-To: References: <10862A58-63D0-4CB3-B194-64A1DF1E08DD@trialphaenergy.com> <47E26DAB-01EB-4CF0-83F9-75AAC04BDDC7@mcs.anl.gov> <049242DC-BD76-454A-B16F-13C781281370@trialphaenergy.com> Message-ID: <7F5CF937-6C73-4EC2-A0BB-C11A3D8CEB40@trialphaenergy.com> No, it is my fault, when I was unable to download from afterstep.org earlier I didn?t realize that it was probably my work firewall blocking me. Thus sending me on a wild goose chase with macports and home-brew and github version, which didn?t compile. But now I?ve tried to follow your advice, and have a new problem. I installed Afterimage from your suggested site, with suggested X lib flags: ./configure --x-includes=/opt/X11/include -x-libraries=/opt/X11/lib make make install No errors. Then re-configured and built petsc: export PETSC_DIR=/Users/sdettrick/libs/petsc-3.6.1 export PETSC_ARCH=mac-gcc-hdf5-mpich-afterimage ./configure --with-afterimage --download-hdf5 --download-mpich --PETSC_ARCH=mac-gcc-hdf5-mpich-afterimage --PETSC_DIR=/Users/sdettrick/libs/petsc-3.6.1 make PETSC_DIR=/Users/sdettrick/libs/petsc-3.6.1 PETSC_ARCH=mac-gcc-hdf5-mpich-afterimage all No errors. Then try to run an example with image output: cd src/dm/examples/tutorials/ make ex5 ./ex5 ./ex5 -draw_save test.png The example generates the error you mentioned, which it didn?t before installing afterimage: X Error of failed request: BadValue (integer parameter out of range for operation) Major opcode of failed request: 1 (X_CreateWindow) Value in failed request: 0x40 Serial number of failed request: 7 Current serial number in output stream: 14 Browsing through configure.log, it seems that petsc is using the same X11 library. Did I miss something? Thanks! Sean On Dec 2, 2015, at 11:10 PM, Barry Smith > wrote: I am such an idiot. No I did build it from source I forgot the comment in afterimage.py After image is available from http://www.afterstep.org/afterimage/getcode.php # # It is used by the PetscDrawSetSave() routine to save X windows graphics to files # # If installing on an Apple make sure to read the details on PetscDrawSetSave manual # page before installing # from that page I found If X windows generates an error message about X_CreateWindow() failing then Afterimage was installed without X windows. Reinstall Afterimage using the ./configure flags --x-includes=/pathtoXincludes --x-libraries=/pathtoXlibraries For example under Mac OS X Mountain Lion --x-includes=/opt/X11/include -x-libraries=/opt/X11/lib Did you try all this stuff? Barry On Dec 3, 2015, at 1:00 AM, Sean Dettrick > wrote: Hi Barry, Thanks for the amazingly fast reply. I installed home-brew but then was unable to find afterstep available as a package. The closest thing I could find was asterm (after step terminal emulator. Do you remember if you installed it from a canonical repo? I did find afterstep on github, but had trouble installing it on the mac. Meanwhile, a colleague has given me a python script, so I?ll probably use that instead for now. Thanks again, Sean On Dec 2, 2015, at 1:45 PM, Barry Smith > wrote: Sean, I have never been able to build from that source either. I used homebrew to install it and its worked for several years. I also tried to see if one could pull the image from the X pixel map and it looked like a huge project (the pixmap stuff is not as trivial as one would think it would be) so I gave up on that and started using afterimage (which essentially does all the for you). Barry On Dec 2, 2015, at 3:39 PM, Sean Dettrick > wrote: Hi, I?d like to have petsc draw output to a pixmap, but have been unable to install afterstep on my Mac. Direct download of the tar files from http://www.afterstep.org/afterimage/getcode.php failed, as did cvs checkout. It seems the target sites may not exist at present. Is there another way to install afterstep? Alternatively, is there a way to manually redirect the X11 output using something like XCreatePixmap? Thanks! Sean Dettrick -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Dec 3 05:54:47 2015 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 3 Dec 2015 05:54:47 -0600 Subject: [petsc-users] Petsc Draw to pixmap In-Reply-To: <7F5CF937-6C73-4EC2-A0BB-C11A3D8CEB40@trialphaenergy.com> References: <10862A58-63D0-4CB3-B194-64A1DF1E08DD@trialphaenergy.com> <47E26DAB-01EB-4CF0-83F9-75AAC04BDDC7@mcs.anl.gov> <049242DC-BD76-454A-B16F-13C781281370@trialphaenergy.com> <7F5CF937-6C73-4EC2-A0BB-C11A3D8CEB40@trialphaenergy.com> Message-ID: On Thu, Dec 3, 2015 at 2:06 AM, Sean Dettrick wrote: > > No, it is my fault, when I was unable to download from afterstep.org > earlier I didn?t realize that it was probably my work firewall blocking > me. Thus sending me on a wild goose chase with macports and home-brew and > github version, which didn?t compile. > > But now I?ve tried to follow your advice, and have a new problem. > > I installed Afterimage from your suggested site, with suggested X lib > flags: > > ./configure --x-includes=/opt/X11/include -x-libraries=/opt/X11/lib > Did you check that it worked? Configure can (semi-)silently turn things off. Thanks, Matt > make > make install > > No errors. Then re-configured and built petsc: > > export PETSC_DIR=/Users/sdettrick/libs/petsc-3.6.1 > export PETSC_ARCH=mac-gcc-hdf5-mpich-afterimage > ./configure --with-afterimage --download-hdf5 --download-mpich > --PETSC_ARCH=mac-gcc-hdf5-mpich-afterimage > --PETSC_DIR=/Users/sdettrick/libs/petsc-3.6.1 > make PETSC_DIR=/Users/sdettrick/libs/petsc-3.6.1 > PETSC_ARCH=mac-gcc-hdf5-mpich-afterimage all > > No errors. Then try to run an example with image output: > > cd src/dm/examples/tutorials/ > make ex5 > ./ex5 > ./ex5 -draw_save test.png > > The example generates the error you mentioned, which it didn?t before > installing afterimage: > > X Error of failed request: BadValue (integer parameter out of range for > operation) > Major opcode of failed request: 1 (X_CreateWindow) > Value in failed request: 0x40 > Serial number of failed request: 7 > Current serial number in output stream: 14 > > Browsing through configure.log, it seems that petsc is using the same X11 > library. > > Did I miss something? > > Thanks! > Sean > > > On Dec 2, 2015, at 11:10 PM, Barry Smith wrote: > > > I am such an idiot. No I did build it from source I forgot the comment in > afterimage.py > > After image is available from > http://www.afterstep.org/afterimage/getcode.php > # > # It is used by the PetscDrawSetSave() routine to save X windows graphics > to files > # > # If installing on an Apple make sure to read the details on > PetscDrawSetSave manual > # page before installing > # > > from that page I found > > If X windows generates an error message about X_CreateWindow() failing > then Afterimage was installed without X windows. Reinstall Afterimage using > the > ./configure flags --x-includes=/pathtoXincludes > --x-libraries=/pathtoXlibraries For example under Mac OS X Mountain Lion > --x-includes=/opt/X11/include -x-libraries=/opt/X11/lib > > Did you try all this stuff? > > Barry > > > On Dec 3, 2015, at 1:00 AM, Sean Dettrick > wrote: > > Hi Barry, > > Thanks for the amazingly fast reply. I installed home-brew but then was > unable to find afterstep available as a package. The closest thing I could > find was asterm (after step terminal emulator. Do you remember if you > installed it from a canonical repo? > > I did find afterstep on github, but had trouble installing it on the mac. > > Meanwhile, a colleague has given me a python script, so I?ll probably use > that instead for now. > > Thanks again, > Sean > > On Dec 2, 2015, at 1:45 PM, Barry Smith wrote: > > > Sean, > > I have never been able to build from that source either. > > I used homebrew to install it and its worked for several years. > > I also tried to see if one could pull the image from the X pixel map and > it looked like a huge project (the pixmap stuff is not as trivial as one > would think it would be) so I gave up on that and started using afterimage > (which essentially does all the for you). > > Barry > > On Dec 2, 2015, at 3:39 PM, Sean Dettrick > wrote: > > Hi, > > I?d like to have petsc draw output to a pixmap, but have been unable to > install afterstep on my Mac. Direct download of the tar files from > http://www.afterstep.org/afterimage/getcode.php failed, as did cvs > checkout. It seems the target sites may not exist at present. > > Is there another way to install afterstep? > > Alternatively, is there a way to manually redirect the X11 output using > something like XCreatePixmap? > > Thanks! > Sean Dettrick > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Dec 3 07:01:09 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 3 Dec 2015 07:01:09 -0600 Subject: [petsc-users] Petsc Draw to pixmap In-Reply-To: References: <10862A58-63D0-4CB3-B194-64A1DF1E08DD@trialphaenergy.com> <47E26DAB-01EB-4CF0-83F9-75AAC04BDDC7@mcs.anl.gov> <049242DC-BD76-454A-B16F-13C781281370@trialphaenergy.com> <7F5CF937-6C73-4EC2-A0BB-C11A3D8CEB40@trialphaenergy.com> Message-ID: <6F1F5E87-586D-4FA6-81FE-24241A139423@mcs.anl.gov> > On Dec 3, 2015, at 5:54 AM, Matthew Knepley wrote: > > On Thu, Dec 3, 2015 at 2:06 AM, Sean Dettrick wrote: > > No, it is my fault, when I was unable to download from afterstep.org earlier I didn?t realize that it was probably my work firewall blocking me. Thus sending me on a wild goose chase with macports and home-brew and github version, which didn?t compile. > > But now I?ve tried to follow your advice, and have a new problem. > > I installed Afterimage from your suggested site, with suggested X lib flags: > > ./configure --x-includes=/opt/X11/include -x-libraries=/opt/X11/lib > > Did you check that it worked? Configure can (semi-)silently turn things off. Yes, I think Matt is right. This is the error message you get if afterimage was installed without X. Check through all the logs and build stuff of afterimage. Barry > > Thanks, > > Matt > > make > make install > > No errors. Then re-configured and built petsc: > > export PETSC_DIR=/Users/sdettrick/libs/petsc-3.6.1 > export PETSC_ARCH=mac-gcc-hdf5-mpich-afterimage > ./configure --with-afterimage --download-hdf5 --download-mpich --PETSC_ARCH=mac-gcc-hdf5-mpich-afterimage --PETSC_DIR=/Users/sdettrick/libs/petsc-3.6.1 > make PETSC_DIR=/Users/sdettrick/libs/petsc-3.6.1 PETSC_ARCH=mac-gcc-hdf5-mpich-afterimage all > > No errors. Then try to run an example with image output: > > cd src/dm/examples/tutorials/ > make ex5 > ./ex5 > ./ex5 -draw_save test.png > > The example generates the error you mentioned, which it didn?t before installing afterimage: > > X Error of failed request: BadValue (integer parameter out of range for operation) > Major opcode of failed request: 1 (X_CreateWindow) > Value in failed request: 0x40 > Serial number of failed request: 7 > Current serial number in output stream: 14 > > Browsing through configure.log, it seems that petsc is using the same X11 library. > > Did I miss something? > > Thanks! > Sean > > >> On Dec 2, 2015, at 11:10 PM, Barry Smith wrote: >> >> >> I am such an idiot. No I did build it from source I forgot the comment in afterimage.py >> >> After image is available from http://www.afterstep.org/afterimage/getcode.php >> # >> # It is used by the PetscDrawSetSave() routine to save X windows graphics to files >> # >> # If installing on an Apple make sure to read the details on PetscDrawSetSave manual >> # page before installing >> # >> >> from that page I found >> >> If X windows generates an error message about X_CreateWindow() failing then Afterimage was installed without X windows. Reinstall Afterimage using the >> ./configure flags --x-includes=/pathtoXincludes --x-libraries=/pathtoXlibraries For example under Mac OS X Mountain Lion --x-includes=/opt/X11/include -x-libraries=/opt/X11/lib >> >> Did you try all this stuff? >> >> Barry >> >> >>> On Dec 3, 2015, at 1:00 AM, Sean Dettrick wrote: >>> >>> Hi Barry, >>> >>> Thanks for the amazingly fast reply. I installed home-brew but then was unable to find afterstep available as a package. The closest thing I could find was asterm (after step terminal emulator. Do you remember if you installed it from a canonical repo? >>> >>> I did find afterstep on github, but had trouble installing it on the mac. >>> >>> Meanwhile, a colleague has given me a python script, so I?ll probably use that instead for now. >>> >>> Thanks again, >>> Sean >>> >>>> On Dec 2, 2015, at 1:45 PM, Barry Smith wrote: >>>> >>>> >>>> Sean, >>>> >>>> I have never been able to build from that source either. >>>> >>>> I used homebrew to install it and its worked for several years. >>>> >>>> I also tried to see if one could pull the image from the X pixel map and it looked like a huge project (the pixmap stuff is not as trivial as one would think it would be) so I gave up on that and started using afterimage (which essentially does all the for you). >>>> >>>> Barry >>>> >>>>> On Dec 2, 2015, at 3:39 PM, Sean Dettrick wrote: >>>>> >>>>> Hi, >>>>> >>>>> I?d like to have petsc draw output to a pixmap, but have been unable to install afterstep on my Mac. Direct download of the tar files from http://www.afterstep.org/afterimage/getcode.php failed, as did cvs checkout. It seems the target sites may not exist at present. >>>>> >>>>> Is there another way to install afterstep? >>>>> >>>>> Alternatively, is there a way to manually redirect the X11 output using something like XCreatePixmap? >>>>> >>>>> Thanks! >>>>> Sean Dettrick >>>> >>> >> > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From hzhang at mcs.anl.gov Thu Dec 3 12:50:09 2015 From: hzhang at mcs.anl.gov (Hong) Date: Thu, 3 Dec 2015 12:50:09 -0600 Subject: [petsc-users] SuperLU convergence problem In-Reply-To: <565F70E8.6030300@eos.ubc.ca> References: <565F70E8.6030300@eos.ubc.ca> Message-ID: Danyang: > > > To my surprising, solutions from SuperLU at timestep 29 seems not correct > for the first 4 Newton iterations, but the solutions from iteration solver > and MUMPS are correct. > > Please find all the matrices, rhs and solutions at timestep 29 via the > link below. The data is a bit large so that I just share it through > Dropbox. A piece of matlab code to read these data and then computer the > norm has also been attached. > *https://www.dropbox.com/s/rr8ueysgflmxs7h/results-check.tar.gz?dl=0 > * > Can you send us matrix in petsc binary format? e.g., call MatView(M, PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD)) or '-ksp_view_mat binary' Hong > > > Below is a summary of the norm from the three solvers at timestep 29, > newton iteration 1 to 5. > > Timestep 29 > Norm of residual seq 1.661321e-09, superlu 1.657103e+04, mumps > 3.731225e-11 > Norm of residual seq 1.753079e-09, superlu 6.675467e+02, mumps > 1.509919e-13 > Norm of residual seq 4.914971e-10, superlu 1.236362e-01, mumps > 2.139303e-17 > Norm of residual seq 3.532769e-10, superlu 1.304670e-04, mumps > 5.387000e-20 > Norm of residual seq 3.885629e-10, superlu 2.754876e-07, mumps > 4.108675e-21 > > Would anybody please check if SuperLU can solve these matrices? Another > possibility is that something is wrong in my own code. But so far, I cannot > find any problem in my code since the same code works fine if I using > iterative solver or direct solver MUMPS. But for other cases I have > tested, all these solvers work fine. > > Please let me know if I did not write down the problem clearly. > > Thanks, > > Danyang > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dsu at eos.ubc.ca Thu Dec 3 14:45:03 2015 From: dsu at eos.ubc.ca (Danyang Su) Date: Thu, 3 Dec 2015 12:45:03 -0800 Subject: [petsc-users] SuperLU convergence problem In-Reply-To: References: <565F70E8.6030300@eos.ubc.ca> Message-ID: <5660A9CF.1030809@eos.ubc.ca> Hi Hong, The binary format of matrix, rhs and solution can be downloaded via the link below. https://www.dropbox.com/s/cl3gfi0s0kjlktf/matrix_and_rhs_bin.tar.gz?dl=0 Thanks, Danyang On 15-12-03 10:50 AM, Hong wrote: > Danyang: > > > > To my surprising, solutions from SuperLU at timestep 29 seems not > correct for the first 4 Newton iterations, but the solutions from > iteration solver and MUMPS are correct. > > Please find all the matrices, rhs and solutions at timestep 29 via > the link below. The data is a bit large so that I just share it > through Dropbox. A piece of matlab code to read these data and > then computer the norm has also been attached. > _https://www.dropbox.com/s/rr8ueysgflmxs7h/results-check.tar.gz?dl=0_ > > > Can you send us matrix in petsc binary format? > > e.g., call MatView(M, PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD)) > or '-ksp_view_mat binary' > > Hong > > > > Below is a summary of the norm from the three solvers at timestep > 29, newton iteration 1 to 5. > > Timestep 29 > Norm of residual seq 1.661321e-09, superlu 1.657103e+04, mumps > 3.731225e-11 > Norm of residual seq 1.753079e-09, superlu 6.675467e+02, mumps > 1.509919e-13 > Norm of residual seq 4.914971e-10, superlu 1.236362e-01, mumps > 2.139303e-17 > Norm of residual seq 3.532769e-10, superlu 1.304670e-04, mumps > 5.387000e-20 > Norm of residual seq 3.885629e-10, superlu 2.754876e-07, mumps > 4.108675e-21 > > Would anybody please check if SuperLU can solve these matrices? > Another possibility is that something is wrong in my own code. But > so far, I cannot find any problem in my code since the same code > works fine if I using iterative solver or direct solver MUMPS. But > for other cases I have tested, all these solvers work fine. > > Please let me know if I did not write down the problem clearly. > > Thanks, > > Danyang > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jychang48 at gmail.com Thu Dec 3 15:00:07 2015 From: jychang48 at gmail.com (Justin Chang) Date: Thu, 3 Dec 2015 14:00:07 -0700 Subject: [petsc-users] Issues with the Variational Inequality solver In-Reply-To: <1F85349D-7AAE-4FC3-A31E-B9232BBDC8FA@mcs.anl.gov> References: <521DF698-ACC4-4F5B-BDEE-558B2142DF57@mcs.anl.gov> <1F85349D-7AAE-4FC3-A31E-B9232BBDC8FA@mcs.anl.gov> Message-ID: Thanks Barry, Next issue: Consider the following problem: psiA = cA - 2*cB (1a) psiC = cC + 2*cB (1b) cC^2 = k*cA^2*cB (1c) where psiA, psiC, and k are given. If I solve these five problems using the standard Newton method: Case 0: psiA: 9.400282e-01 psiC: 6.045961e-09 k: 2.000000 cA: 9.400282e-01 cB: 1.555428e-16 cC: 6.045961e-09 Case 1: psiA: 8.472901e-01 psiC: 7.425271e-09 k: 2.000000 cA: 8.472901e-01 cB: 2.602870e-16 cC: 7.425270e-09 Case 2: psiA: 8.337199e-01 psiC: 7.831614e-09 k: 2.000000 cA: 8.337199e-01 cB: 2.942675e-16 cC: 7.831613e-09 Case 3: psiA: 8.268140e-01 psiC: 7.912911e-09 k: 2.000000 cA: 8.268140e-01 cB: 3.029178e-16 cC: 7.912910e-09 Case 4: psiA: 9.852477e-01 psiC: 7.992223e-09 k: 2.000000 cA: 9.852477e-01 cB: 2.593282e-16 cC: 7.992222e-09 These solutions are "correct", that is if I plug the c's back into equations (1a)-(1b), i get the original psi's. Now suppose I use the Variational Inequality such that cA/B/C >= 0, I would expect to get the exact same solution (since the c's would be non-negative regardless). But instead I get these solutions: Case 0: psiA: 9.400282e-01 psiC: 6.045961e-09 k: 2.000000 cA: 1.318866e-16 cB: 3.097570e-08 cC: 6.045961e-09 Case 1: psiA: 8.472901e-01 psiC: 7.425271e-09 k: 2.000000 cA: 4.624944e-17 cB: 3.015792e-08 cC: 7.425271e-09 Case 2: psiA: 8.337199e-01 psiC: 7.831614e-09 k: 2.000000 cA: 1.733276e-16 cB: 3.079856e-08 cC: 7.831614e-09 Case 3: psiA: 8.268140e-01 psiC: 7.912911e-09 k: 2.000000 cA: 1.721078e-16 cB: 3.061785e-08 cC: 7.912911e-09 Case 4: psiA: 9.852477e-01 psiC: 7.992223e-09 k: 2.000000 cA: 2.666770e-16 cB: 4.610822e-08 cC: 7.992223e-09 Basically, the cA's shoot down to zero and the cB's are slightly greater than zero. When I plug the c's into equations (1a) - (1b), I do not get the correct solution at all. I would expect discrepancies if my c's were originally negative, but for these five problems, shouldn't VI give the same answer as the Newton's method? Attached is the petsc4py code, the two input files containing psiA and psiC, and the outputs from '-snes_monitor -snes_view -ksp_monitor_true_residual'. Run as: python testVI.py testA testC 2 <0/1> where 0 is Newton, 1 is VI. Know what's going on here? Thanks, Justin On Wed, Dec 2, 2015 at 1:36 PM, Barry Smith wrote: > > > On Dec 2, 2015, at 1:56 PM, Justin Chang wrote: > > > > Barry, > > > > So I do not believe there is any problem with the ISEqual() per se, > because what I am doing is I am solving 5151 different VI problem using the > same SNES/KSP/PC/Mat. That is, I am not "resetting" these data structures > once I am done with one problem and move on to the next. If I ran each case > individually, I get no error; the error comes when I run these problems > sequentially without resetting the SNES after each problem. > > > > I haven't run the C debugger or done any of that yet, but could this be > a plausible explanation for my error? > > This is exactly the issue. Independent of VIs if you change the size of > a problem you pass to SNES you need to call SNESReset() between each call > of a different sizes. > > > Originally I was thinking about creating/destroying SNES for each > problem but I was wondering if that might affect performance. > > Using SNESReset() is the way you should do it. Creating and destroying > it each time should only be a tiny, immeasurably slower. > > Barry > > > > > Thanks, > > Justin > > > > On Tue, Dec 1, 2015 at 5:58 PM, Barry Smith wrote: > > > > > On Dec 1, 2015, at 5:13 PM, Justin Chang wrote: > > > > > > Hi all, > > > > > > So I am running into some issues with the SNESVINEWTONRSLS solver. I > originally had this done in firedrake, but have ported it to petsc4py. > Attached is the code as well as the two required input files. > > > > > > First issue, when I run the program like this: > > > > > > python testVI.py psiA_1 psiB_1 2 1 > > > > > > I get this error: > > > > > > Traceback (most recent call last): > > > File "testVI.py", line 103, in > > > snes.solve(None,X) > > > File "PETSc/SNES.pyx", line 520, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:169677) > > > petsc4py.PETSc.Error: error code 60 > > > [0] SNESSolve() line 3992 in > /Users/justin/Software/petsc/src/snes/interface/snes.c > > > [0] SNESSolve_VINEWTONRSLS() line 507 in > /Users/justin/Software/petsc/src/snes/impls/vi/rs/virs.c > > > [0] KSPSetOperators() line 544 in > /Users/justin/Software/petsc/src/ksp/ksp/interface/itcreate.c > > > [0] PCSetOperators() line 1170 in > /Users/justin/Software/petsc/src/ksp/pc/interface/precon.c > > > [0] Nonconforming object sizes > > > [0] Cannot change local size of Amat after use old sizes 2 2 new sizes > 3 3 > > > > > > No such error occurred when this was ran in firedrake, though I did > notice that -snes_view did indicate that some of the iterations had 2x2 > instead of 3x3 matrices. Why is this happening? I thought I should be > solving a 3x3 system each time so why would a 2x2 ever arise? > > > > Justin, > > > > For VI's the rs (reduced space solver) solves the linearized problem > on a subset of the variables, thus the size of the linear system may change > from iteration of Newton to the next iteration of Newton. > > > > In the rs solver we have > > > > ierr = > ISEqual(vi->IS_inact_prev,vi->IS_inact,&isequal);CHKERRQ(ierr); > > if (!isequal) { > > ierr = > SNESVIResetPCandKSP(snes,jac_inact_inact,prejac_inact_inact);CHKERRQ(ierr); > > } > > > > so what __should__ happen is that if the size of the reduced space > changes KSPReset() is called on the KSP allowing the KSP/PC to handle a > different sized system then before. But why doesn't it work? Why isn't > KSPReset() called? Somehow the ISEqual is not working. > > > > Can you run the C debugger and put a breakpoint in the ISEqual() then > check the inputs and see if it is correctly setting the isequal to false > when it should? Each time the vi->IS_inact changes the KSPReset() should > get called. You can check this in the debugger. > > > > Barry > > > > > > > > More issues to come :) > > > > > > Thanks, > > > Justin > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: testVI.py Type: text/x-python-script Size: 3071 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: testA Type: application/octet-stream Size: 64 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: testC Type: application/octet-stream Size: 64 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: newton_output Type: application/octet-stream Size: 11308 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: vi_output Type: application/octet-stream Size: 43977 bytes Desc: not available URL: From knepley at gmail.com Thu Dec 3 15:26:44 2015 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 3 Dec 2015 15:26:44 -0600 Subject: [petsc-users] Issues with the Variational Inequality solver In-Reply-To: References: <521DF698-ACC4-4F5B-BDEE-558B2142DF57@mcs.anl.gov> <1F85349D-7AAE-4FC3-A31E-B9232BBDC8FA@mcs.anl.gov> Message-ID: On Thu, Dec 3, 2015 at 3:00 PM, Justin Chang wrote: > Thanks Barry, > > Next issue: > Barry, when Justin described this problem to me, it sounded like there might be a bug in the active set method which put in the lower value when it was supposed to be the upper value. Matt > Consider the following problem: > > psiA = cA - 2*cB (1a) > psiC = cC + 2*cB (1b) > cC^2 = k*cA^2*cB (1c) > > where psiA, psiC, and k are given. If I solve these five problems using > the standard Newton method: > > Case 0: > > psiA: 9.400282e-01 > > psiC: 6.045961e-09 > > k: 2.000000 > > cA: 9.400282e-01 > > cB: 1.555428e-16 > > cC: 6.045961e-09 > > Case 1: > > psiA: 8.472901e-01 > > psiC: 7.425271e-09 > > k: 2.000000 > > cA: 8.472901e-01 > > cB: 2.602870e-16 > > cC: 7.425270e-09 > > Case 2: > > psiA: 8.337199e-01 > > psiC: 7.831614e-09 > > k: 2.000000 > > cA: 8.337199e-01 > > cB: 2.942675e-16 > > cC: 7.831613e-09 > > Case 3: > > psiA: 8.268140e-01 > > psiC: 7.912911e-09 > > k: 2.000000 > > cA: 8.268140e-01 > > cB: 3.029178e-16 > > cC: 7.912910e-09 > > Case 4: > > psiA: 9.852477e-01 > > psiC: 7.992223e-09 > > k: 2.000000 > > cA: 9.852477e-01 > > cB: 2.593282e-16 > > cC: 7.992222e-09 > > These solutions are "correct", that is if I plug the c's back into > equations (1a)-(1b), i get the original psi's. > > Now suppose I use the Variational Inequality such that cA/B/C >= 0, I > would expect to get the exact same solution (since the c's would be > non-negative regardless). But instead I get these solutions: > > Case 0: > > psiA: 9.400282e-01 > > psiC: 6.045961e-09 > > k: 2.000000 > > cA: 1.318866e-16 > > cB: 3.097570e-08 > > cC: 6.045961e-09 > > Case 1: > > psiA: 8.472901e-01 > > psiC: 7.425271e-09 > > k: 2.000000 > > cA: 4.624944e-17 > > cB: 3.015792e-08 > > cC: 7.425271e-09 > > Case 2: > > psiA: 8.337199e-01 > > psiC: 7.831614e-09 > > k: 2.000000 > > cA: 1.733276e-16 > > cB: 3.079856e-08 > > cC: 7.831614e-09 > > Case 3: > > psiA: 8.268140e-01 > > psiC: 7.912911e-09 > > k: 2.000000 > > cA: 1.721078e-16 > > cB: 3.061785e-08 > > cC: 7.912911e-09 > > Case 4: > > psiA: 9.852477e-01 > > psiC: 7.992223e-09 > > k: 2.000000 > > cA: 2.666770e-16 > > cB: 4.610822e-08 > > cC: 7.992223e-09 > > Basically, the cA's shoot down to zero and the cB's are slightly greater > than zero. When I plug the c's into equations (1a) - (1b), I do not get the > correct solution at all. I would expect discrepancies if my c's were > originally negative, but for these five problems, shouldn't VI give the > same answer as the Newton's method? > > Attached is the petsc4py code, the two input files containing psiA and > psiC, and the outputs from '-snes_monitor -snes_view > -ksp_monitor_true_residual'. Run as: > > python testVI.py testA testC 2 <0/1> > > where 0 is Newton, 1 is VI. > > Know what's going on here? > > Thanks, > Justin > > On Wed, Dec 2, 2015 at 1:36 PM, Barry Smith wrote: > >> >> > On Dec 2, 2015, at 1:56 PM, Justin Chang wrote: >> > >> > Barry, >> > >> > So I do not believe there is any problem with the ISEqual() per se, >> because what I am doing is I am solving 5151 different VI problem using the >> same SNES/KSP/PC/Mat. That is, I am not "resetting" these data structures >> once I am done with one problem and move on to the next. If I ran each case >> individually, I get no error; the error comes when I run these problems >> sequentially without resetting the SNES after each problem. >> > >> > I haven't run the C debugger or done any of that yet, but could this be >> a plausible explanation for my error? >> >> This is exactly the issue. Independent of VIs if you change the size >> of a problem you pass to SNES you need to call SNESReset() between each >> call of a different sizes. >> >> > Originally I was thinking about creating/destroying SNES for each >> problem but I was wondering if that might affect performance. >> >> Using SNESReset() is the way you should do it. Creating and destroying >> it each time should only be a tiny, immeasurably slower. >> >> Barry >> >> > >> > Thanks, >> > Justin >> > >> > On Tue, Dec 1, 2015 at 5:58 PM, Barry Smith wrote: >> > >> > > On Dec 1, 2015, at 5:13 PM, Justin Chang wrote: >> > > >> > > Hi all, >> > > >> > > So I am running into some issues with the SNESVINEWTONRSLS solver. I >> originally had this done in firedrake, but have ported it to petsc4py. >> Attached is the code as well as the two required input files. >> > > >> > > First issue, when I run the program like this: >> > > >> > > python testVI.py psiA_1 psiB_1 2 1 >> > > >> > > I get this error: >> > > >> > > Traceback (most recent call last): >> > > File "testVI.py", line 103, in >> > > snes.solve(None,X) >> > > File "PETSc/SNES.pyx", line 520, in petsc4py.PETSc.SNES.solve >> (src/petsc4py.PETSc.c:169677) >> > > petsc4py.PETSc.Error: error code 60 >> > > [0] SNESSolve() line 3992 in >> /Users/justin/Software/petsc/src/snes/interface/snes.c >> > > [0] SNESSolve_VINEWTONRSLS() line 507 in >> /Users/justin/Software/petsc/src/snes/impls/vi/rs/virs.c >> > > [0] KSPSetOperators() line 544 in >> /Users/justin/Software/petsc/src/ksp/ksp/interface/itcreate.c >> > > [0] PCSetOperators() line 1170 in >> /Users/justin/Software/petsc/src/ksp/pc/interface/precon.c >> > > [0] Nonconforming object sizes >> > > [0] Cannot change local size of Amat after use old sizes 2 2 new >> sizes 3 3 >> > > >> > > No such error occurred when this was ran in firedrake, though I did >> notice that -snes_view did indicate that some of the iterations had 2x2 >> instead of 3x3 matrices. Why is this happening? I thought I should be >> solving a 3x3 system each time so why would a 2x2 ever arise? >> > >> > Justin, >> > >> > For VI's the rs (reduced space solver) solves the linearized problem >> on a subset of the variables, thus the size of the linear system may change >> from iteration of Newton to the next iteration of Newton. >> > >> > In the rs solver we have >> > >> > ierr = >> ISEqual(vi->IS_inact_prev,vi->IS_inact,&isequal);CHKERRQ(ierr); >> > if (!isequal) { >> > ierr = >> SNESVIResetPCandKSP(snes,jac_inact_inact,prejac_inact_inact);CHKERRQ(ierr); >> > } >> > >> > so what __should__ happen is that if the size of the reduced space >> changes KSPReset() is called on the KSP allowing the KSP/PC to handle a >> different sized system then before. But why doesn't it work? Why isn't >> KSPReset() called? Somehow the ISEqual is not working. >> > >> > Can you run the C debugger and put a breakpoint in the ISEqual() >> then check the inputs and see if it is correctly setting the isequal to >> false when it should? Each time the vi->IS_inact changes the KSPReset() >> should get called. You can check this in the debugger. >> > >> > Barry >> > >> > > >> > > More issues to come :) >> > > >> > > Thanks, >> > > Justin >> > > >> > >> > >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Thu Dec 3 15:44:57 2015 From: hzhang at mcs.anl.gov (Hong) Date: Thu, 3 Dec 2015 15:44:57 -0600 Subject: [petsc-users] SuperLU convergence problem In-Reply-To: <5660A9CF.1030809@eos.ubc.ca> References: <565F70E8.6030300@eos.ubc.ca> <5660A9CF.1030809@eos.ubc.ca> Message-ID: Danyang: Using petsc/src/ksp/ksp/examples/tutorials/ex10.c, I tested a_flow_check_168.bin mpiexec -n 4 ./ex10 -f0 /Users/Hong/Downloads/matrix_and_rhs_bin/a_flow_check_168.bin -rhs /Users/Hong/Downloads/matrix_and_rhs_bin/x_flow_check_168.bin -pc_type lu -pc_factor_mat_solver_package superlu_dist Number of iterations = 1 Residual norm 1.38302e-05 petsc (sequential) and mumps all give residual norm = 1.e-5. For 169, mpiexec -n 4 ./ex10 -f0 /Users/Hong/Downloads/matrix_and_rhs_bin/a_flow_check_169.bin -rhs /Users/Hong/Downloads/matrix_and_rhs_bin/x_flow_check_169.bin -pc_type lu -pc_factor_mat_solver_package superlu_dist Number of iterations = 1 Residual norm 6.83854e-10 For 170, 171, 172, I get mpiexec -n 4 ./ex10 -f0 /Users/Hong/Downloads/matrix_and_rhs_bin/a_flow_check_172.bin -rhs /Users/Hong/Downloads/matrix_and_rhs_bin/x_flow_check_172.bin -pc_type lu -pc_factor_mat_solver_package superlu_dist Number of iterations = 1 Residual norm < 1.e-12 Can you use this example test these matrices? I use superlu_dist-v4.1. Hong Hi Hong, > > The binary format of matrix, rhs and solution can be downloaded via the > link below. > > https://www.dropbox.com/s/cl3gfi0s0kjlktf/matrix_and_rhs_bin.tar.gz?dl=0 > > Thanks, > > Danyang > > > On 15-12-03 10:50 AM, Hong wrote: > > Danyang: > >> >> >> To my surprising, solutions from SuperLU at timestep 29 seems not correct >> for the first 4 Newton iterations, but the solutions from iteration solver >> and MUMPS are correct. >> >> Please find all the matrices, rhs and solutions at timestep 29 via the >> link below. The data is a bit large so that I just share it through >> Dropbox. A piece of matlab code to read these data and then computer the >> norm has also been attached. >> *https://www.dropbox.com/s/rr8ueysgflmxs7h/results-check.tar.gz?dl=0 >> * >> > > Can you send us matrix in petsc binary format? > > e.g., call MatView(M, PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD)) > or '-ksp_view_mat binary' > > Hong > >> >> >> Below is a summary of the norm from the three solvers at timestep 29, >> newton iteration 1 to 5. >> >> Timestep 29 >> Norm of residual seq 1.661321e-09, superlu 1.657103e+04, mumps >> 3.731225e-11 >> Norm of residual seq 1.753079e-09, superlu 6.675467e+02, mumps >> 1.509919e-13 >> Norm of residual seq 4.914971e-10, superlu 1.236362e-01, mumps >> 2.139303e-17 >> Norm of residual seq 3.532769e-10, superlu 1.304670e-04, mumps >> 5.387000e-20 >> Norm of residual seq 3.885629e-10, superlu 2.754876e-07, mumps >> 4.108675e-21 >> >> Would anybody please check if SuperLU can solve these matrices? Another >> possibility is that something is wrong in my own code. But so far, I cannot >> find any problem in my code since the same code works fine if I using >> iterative solver or direct solver MUMPS. But for other cases I have >> tested, all these solvers work fine. >> >> Please let me know if I did not write down the problem clearly. >> >> Thanks, >> >> Danyang >> >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jychang48 at gmail.com Thu Dec 3 15:59:21 2015 From: jychang48 at gmail.com (Justin Chang) Date: Thu, 3 Dec 2015 14:59:21 -0700 Subject: [petsc-users] Fwd: Issues with the Variational Inequality solver In-Reply-To: References: <521DF698-ACC4-4F5B-BDEE-558B2142DF57@mcs.anl.gov> <1F85349D-7AAE-4FC3-A31E-B9232BBDC8FA@mcs.anl.gov> Message-ID: Sorry forgot to hit reply all ---------- Forwarded message ---------- From: Justin Chang Date: Thu, Dec 3, 2015 at 2:40 PM Subject: Re: [petsc-users] Issues with the Variational Inequality solver To: Matthew Knepley Matt, for these problems I set an upper bound of 10, so I am not sure that's related to the problem at hand. However, that is the *next* issue I will bring up if this current one is resolved ;) Another thing I found out... The five case studies I listed used no preconditioner. When I use the jacobi preconditioner, the Newton and VI solutions are identical. Whereas these problems now fail: with Newton: Case 0: psiA: 6.297272e-01 psiC: 1.705043e-08 k: 2.000000 cA: 6.297272e-01 cB: 4.616557e-16 cC: 1.705043e-08 Case 1: psiA: 7.570078e-01 psiC: 1.754758e-08 k: 2.000000 cA: 7.570078e-01 cB: 4.067561e-16 cC: 1.754758e-08 with VI: Case 0: psiA: 6.297272e-01 psiC: 1.705043e-08 k: 2.000000 cA: -7.684746e-17 cB: 5.362599e-09 cC: 4.616557e-16 Case 1: psiA: 7.570078e-01 psiC: 1.754758e-08 k: 2.000000 cA: 3.781787e-16 cB: 1.152521e-08 cC: 4.067561e-16 For the first set of problems, Jacobi works but no preconditioner fails. For the second set of problems, Jacobi fails but no preconditioner works. ?? Justin On Thu, Dec 3, 2015 at 2:26 PM, Matthew Knepley wrote: > On Thu, Dec 3, 2015 at 3:00 PM, Justin Chang wrote: > >> Thanks Barry, >> >> Next issue: >> > > Barry, when Justin described this problem to me, it sounded like there > might be a bug in the active set method which > put in the lower value when it was supposed to be the upper value. > > Matt > > >> Consider the following problem: >> >> psiA = cA - 2*cB (1a) >> psiC = cC + 2*cB (1b) >> cC^2 = k*cA^2*cB (1c) >> >> where psiA, psiC, and k are given. If I solve these five problems using >> the standard Newton method: >> >> Case 0: >> >> psiA: 9.400282e-01 >> >> psiC: 6.045961e-09 >> >> k: 2.000000 >> >> cA: 9.400282e-01 >> >> cB: 1.555428e-16 >> >> cC: 6.045961e-09 >> >> Case 1: >> >> psiA: 8.472901e-01 >> >> psiC: 7.425271e-09 >> >> k: 2.000000 >> >> cA: 8.472901e-01 >> >> cB: 2.602870e-16 >> >> cC: 7.425270e-09 >> >> Case 2: >> >> psiA: 8.337199e-01 >> >> psiC: 7.831614e-09 >> >> k: 2.000000 >> >> cA: 8.337199e-01 >> >> cB: 2.942675e-16 >> >> cC: 7.831613e-09 >> >> Case 3: >> >> psiA: 8.268140e-01 >> >> psiC: 7.912911e-09 >> >> k: 2.000000 >> >> cA: 8.268140e-01 >> >> cB: 3.029178e-16 >> >> cC: 7.912910e-09 >> >> Case 4: >> >> psiA: 9.852477e-01 >> >> psiC: 7.992223e-09 >> >> k: 2.000000 >> >> cA: 9.852477e-01 >> >> cB: 2.593282e-16 >> >> cC: 7.992222e-09 >> >> These solutions are "correct", that is if I plug the c's back into >> equations (1a)-(1b), i get the original psi's. >> >> Now suppose I use the Variational Inequality such that cA/B/C >= 0, I >> would expect to get the exact same solution (since the c's would be >> non-negative regardless). But instead I get these solutions: >> >> Case 0: >> >> psiA: 9.400282e-01 >> >> psiC: 6.045961e-09 >> >> k: 2.000000 >> >> cA: 1.318866e-16 >> >> cB: 3.097570e-08 >> >> cC: 6.045961e-09 >> >> Case 1: >> >> psiA: 8.472901e-01 >> >> psiC: 7.425271e-09 >> >> k: 2.000000 >> >> cA: 4.624944e-17 >> >> cB: 3.015792e-08 >> >> cC: 7.425271e-09 >> >> Case 2: >> >> psiA: 8.337199e-01 >> >> psiC: 7.831614e-09 >> >> k: 2.000000 >> >> cA: 1.733276e-16 >> >> cB: 3.079856e-08 >> >> cC: 7.831614e-09 >> >> Case 3: >> >> psiA: 8.268140e-01 >> >> psiC: 7.912911e-09 >> >> k: 2.000000 >> >> cA: 1.721078e-16 >> >> cB: 3.061785e-08 >> >> cC: 7.912911e-09 >> >> Case 4: >> >> psiA: 9.852477e-01 >> >> psiC: 7.992223e-09 >> >> k: 2.000000 >> >> cA: 2.666770e-16 >> >> cB: 4.610822e-08 >> >> cC: 7.992223e-09 >> >> Basically, the cA's shoot down to zero and the cB's are slightly greater >> than zero. When I plug the c's into equations (1a) - (1b), I do not get the >> correct solution at all. I would expect discrepancies if my c's were >> originally negative, but for these five problems, shouldn't VI give the >> same answer as the Newton's method? >> >> Attached is the petsc4py code, the two input files containing psiA and >> psiC, and the outputs from '-snes_monitor -snes_view >> -ksp_monitor_true_residual'. Run as: >> >> python testVI.py testA testC 2 <0/1> >> >> where 0 is Newton, 1 is VI. >> >> Know what's going on here? >> >> Thanks, >> Justin >> >> On Wed, Dec 2, 2015 at 1:36 PM, Barry Smith wrote: >> >>> >>> > On Dec 2, 2015, at 1:56 PM, Justin Chang wrote: >>> > >>> > Barry, >>> > >>> > So I do not believe there is any problem with the ISEqual() per se, >>> because what I am doing is I am solving 5151 different VI problem using the >>> same SNES/KSP/PC/Mat. That is, I am not "resetting" these data structures >>> once I am done with one problem and move on to the next. If I ran each case >>> individually, I get no error; the error comes when I run these problems >>> sequentially without resetting the SNES after each problem. >>> > >>> > I haven't run the C debugger or done any of that yet, but could this >>> be a plausible explanation for my error? >>> >>> This is exactly the issue. Independent of VIs if you change the size >>> of a problem you pass to SNES you need to call SNESReset() between each >>> call of a different sizes. >>> >>> > Originally I was thinking about creating/destroying SNES for each >>> problem but I was wondering if that might affect performance. >>> >>> Using SNESReset() is the way you should do it. Creating and >>> destroying it each time should only be a tiny, immeasurably slower. >>> >>> Barry >>> >>> > >>> > Thanks, >>> > Justin >>> > >>> > On Tue, Dec 1, 2015 at 5:58 PM, Barry Smith >>> wrote: >>> > >>> > > On Dec 1, 2015, at 5:13 PM, Justin Chang >>> wrote: >>> > > >>> > > Hi all, >>> > > >>> > > So I am running into some issues with the SNESVINEWTONRSLS solver. I >>> originally had this done in firedrake, but have ported it to petsc4py. >>> Attached is the code as well as the two required input files. >>> > > >>> > > First issue, when I run the program like this: >>> > > >>> > > python testVI.py psiA_1 psiB_1 2 1 >>> > > >>> > > I get this error: >>> > > >>> > > Traceback (most recent call last): >>> > > File "testVI.py", line 103, in >>> > > snes.solve(None,X) >>> > > File "PETSc/SNES.pyx", line 520, in petsc4py.PETSc.SNES.solve >>> (src/petsc4py.PETSc.c:169677) >>> > > petsc4py.PETSc.Error: error code 60 >>> > > [0] SNESSolve() line 3992 in >>> /Users/justin/Software/petsc/src/snes/interface/snes.c >>> > > [0] SNESSolve_VINEWTONRSLS() line 507 in >>> /Users/justin/Software/petsc/src/snes/impls/vi/rs/virs.c >>> > > [0] KSPSetOperators() line 544 in >>> /Users/justin/Software/petsc/src/ksp/ksp/interface/itcreate.c >>> > > [0] PCSetOperators() line 1170 in >>> /Users/justin/Software/petsc/src/ksp/pc/interface/precon.c >>> > > [0] Nonconforming object sizes >>> > > [0] Cannot change local size of Amat after use old sizes 2 2 new >>> sizes 3 3 >>> > > >>> > > No such error occurred when this was ran in firedrake, though I did >>> notice that -snes_view did indicate that some of the iterations had 2x2 >>> instead of 3x3 matrices. Why is this happening? I thought I should be >>> solving a 3x3 system each time so why would a 2x2 ever arise? >>> > >>> > Justin, >>> > >>> > For VI's the rs (reduced space solver) solves the linearized >>> problem on a subset of the variables, thus the size of the linear system >>> may change from iteration of Newton to the next iteration of Newton. >>> > >>> > In the rs solver we have >>> > >>> > ierr = >>> ISEqual(vi->IS_inact_prev,vi->IS_inact,&isequal);CHKERRQ(ierr); >>> > if (!isequal) { >>> > ierr = >>> SNESVIResetPCandKSP(snes,jac_inact_inact,prejac_inact_inact);CHKERRQ(ierr); >>> > } >>> > >>> > so what __should__ happen is that if the size of the reduced space >>> changes KSPReset() is called on the KSP allowing the KSP/PC to handle a >>> different sized system then before. But why doesn't it work? Why isn't >>> KSPReset() called? Somehow the ISEqual is not working. >>> > >>> > Can you run the C debugger and put a breakpoint in the ISEqual() >>> then check the inputs and see if it is correctly setting the isequal to >>> false when it should? Each time the vi->IS_inact changes the KSPReset() >>> should get called. You can check this in the debugger. >>> > >>> > Barry >>> > >>> > > >>> > > More issues to come :) >>> > > >>> > > Thanks, >>> > > Justin >>> > > >>> > >>> > >>> >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Thu Dec 3 15:59:58 2015 From: hzhang at mcs.anl.gov (Hong) Date: Thu, 3 Dec 2015 15:59:58 -0600 Subject: [petsc-users] SuperLU convergence problem In-Reply-To: <5660A9CF.1030809@eos.ubc.ca> References: <565F70E8.6030300@eos.ubc.ca> <5660A9CF.1030809@eos.ubc.ca> Message-ID: Danyang : Further testing a_flow_check_168.bin, ./ex10 -f0 /Users/Hong/Downloads/matrix_and_rhs_bin/a_flow_check_168.bin -rhs /Users/Hong/Downloads/matrix_and_rhs_bin/x_flow_check_168.bin -pc_type lu -pc_factor_mat_solver_package superlu -ksp_monitor_true_residual -mat_superlu_conditionnumber Recip. condition number = 1.610480e-12 0 KSP preconditioned resid norm 6.873340313547e+09 true resid norm 7.295020990196e+03 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.051833296449e-02 true resid norm 2.976859070118e-02 ||r(i)||/||b|| 4.080672384793e-06 Number of iterations = 1 Residual norm 0.0297686 condition number of this matrix = 1/1.610480e-12 = 1.e+12, i.e., this matrix is ill-conditioned. Hong Hi Hong, > > The binary format of matrix, rhs and solution can be downloaded via the > link below. > > https://www.dropbox.com/s/cl3gfi0s0kjlktf/matrix_and_rhs_bin.tar.gz?dl=0 > > Thanks, > > Danyang > > > On 15-12-03 10:50 AM, Hong wrote: > > Danyang: > >> >> >> To my surprising, solutions from SuperLU at timestep 29 seems not correct >> for the first 4 Newton iterations, but the solutions from iteration solver >> and MUMPS are correct. >> >> Please find all the matrices, rhs and solutions at timestep 29 via the >> link below. The data is a bit large so that I just share it through >> Dropbox. A piece of matlab code to read these data and then computer the >> norm has also been attached. >> *https://www.dropbox.com/s/rr8ueysgflmxs7h/results-check.tar.gz?dl=0 >> * >> > > Can you send us matrix in petsc binary format? > > e.g., call MatView(M, PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD)) > or '-ksp_view_mat binary' > > Hong > >> >> >> Below is a summary of the norm from the three solvers at timestep 29, >> newton iteration 1 to 5. >> >> Timestep 29 >> Norm of residual seq 1.661321e-09, superlu 1.657103e+04, mumps >> 3.731225e-11 >> Norm of residual seq 1.753079e-09, superlu 6.675467e+02, mumps >> 1.509919e-13 >> Norm of residual seq 4.914971e-10, superlu 1.236362e-01, mumps >> 2.139303e-17 >> Norm of residual seq 3.532769e-10, superlu 1.304670e-04, mumps >> 5.387000e-20 >> Norm of residual seq 3.885629e-10, superlu 2.754876e-07, mumps >> 4.108675e-21 >> >> Would anybody please check if SuperLU can solve these matrices? Another >> possibility is that something is wrong in my own code. But so far, I cannot >> find any problem in my code since the same code works fine if I using >> iterative solver or direct solver MUMPS. But for other cases I have >> tested, all these solvers work fine. >> >> Please let me know if I did not write down the problem clearly. >> >> Thanks, >> >> Danyang >> >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Dec 3 16:26:13 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 3 Dec 2015 16:26:13 -0600 Subject: [petsc-users] Issues with the Variational Inequality solver In-Reply-To: References: <521DF698-ACC4-4F5B-BDEE-558B2142DF57@mcs.anl.gov> <1F85349D-7AAE-4FC3-A31E-B9232BBDC8FA@mcs.anl.gov> Message-ID: > On Dec 3, 2015, at 3:59 PM, Justin Chang wrote: > > Sorry forgot to hit reply all > > ---------- Forwarded message ---------- > From: Justin Chang > Date: Thu, Dec 3, 2015 at 2:40 PM > Subject: Re: [petsc-users] Issues with the Variational Inequality solver > To: Matthew Knepley > > > Matt, for these problems I set an upper bound of 10, so I am not sure that's related to the problem at hand. However, that is the *next* issue I will bring up if this current one is resolved ;) > > Another thing I found out... > > The five case studies I listed used no preconditioner. When I use the jacobi preconditioner, the Newton and VI solutions are identical. > > Whereas these problems now fail: > > with Newton: > Case 0: > psiA: 6.297272e-01 > psiC: 1.705043e-08 > k: 2.000000 > cA: 6.297272e-01 > cB: 4.616557e-16 > cC: 1.705043e-08 > Case 1: > psiA: 7.570078e-01 > psiC: 1.754758e-08 > k: 2.000000 > cA: 7.570078e-01 > cB: 4.067561e-16 > cC: 1.754758e-08 > > with VI: > > Case 0: > psiA: 6.297272e-01 > psiC: 1.705043e-08 > k: 2.000000 > cA: -7.684746e-17 > cB: 5.362599e-09 > cC: 4.616557e-16 > Case 1: > psiA: 7.570078e-01 > psiC: 1.754758e-08 > k: 2.000000 > cA: 3.781787e-16 > cB: 1.152521e-08 > cC: 4.067561e-16 > > For the first set of problems, Jacobi works but no preconditioner fails. > For the second set of problems, Jacobi fails but no preconditioner works. "Fails" is not very informative. What happens if you use LU? When debugging a nonlinear solver always use a direct solver for the linear solver (and make sure the direct solver actually worked). > > ?? > > Justin > > On Thu, Dec 3, 2015 at 2:26 PM, Matthew Knepley wrote: > On Thu, Dec 3, 2015 at 3:00 PM, Justin Chang wrote: > Thanks Barry, > > Next issue: > > Barry, when Justin described this problem to me, it sounded like there might be a bug in the active set method which > put in the lower value when it was supposed to be the upper value. > > Matt > > Consider the following problem: > > psiA = cA - 2*cB (1a) > psiC = cC + 2*cB (1b) > cC^2 = k*cA^2*cB (1c) > > where psiA, psiC, and k are given. If I solve these five problems using the standard Newton method: > > Case 0: > psiA: 9.400282e-01 > psiC: 6.045961e-09 > k: 2.000000 > cA: 9.400282e-01 > cB: 1.555428e-16 > cC: 6.045961e-09 > Case 1: > psiA: 8.472901e-01 > psiC: 7.425271e-09 > k: 2.000000 > cA: 8.472901e-01 > cB: 2.602870e-16 > cC: 7.425270e-09 > Case 2: > psiA: 8.337199e-01 > psiC: 7.831614e-09 > k: 2.000000 > cA: 8.337199e-01 > cB: 2.942675e-16 > cC: 7.831613e-09 > Case 3: > psiA: 8.268140e-01 > psiC: 7.912911e-09 > k: 2.000000 > cA: 8.268140e-01 > cB: 3.029178e-16 > cC: 7.912910e-09 > Case 4: > psiA: 9.852477e-01 > psiC: 7.992223e-09 > k: 2.000000 > cA: 9.852477e-01 > cB: 2.593282e-16 > cC: 7.992222e-09 > > These solutions are "correct", that is if I plug the c's back into equations (1a)-(1b), i get the original psi's. > > Now suppose I use the Variational Inequality such that cA/B/C >= 0, I would expect to get the exact same solution (since the c's would be non-negative regardless). But instead I get these solutions: > > Case 0: > psiA: 9.400282e-01 > psiC: 6.045961e-09 > k: 2.000000 > cA: 1.318866e-16 > cB: 3.097570e-08 > cC: 6.045961e-09 > Case 1: > psiA: 8.472901e-01 > psiC: 7.425271e-09 > k: 2.000000 > cA: 4.624944e-17 > cB: 3.015792e-08 > cC: 7.425271e-09 > Case 2: > psiA: 8.337199e-01 > psiC: 7.831614e-09 > k: 2.000000 > cA: 1.733276e-16 > cB: 3.079856e-08 > cC: 7.831614e-09 > Case 3: > psiA: 8.268140e-01 > psiC: 7.912911e-09 > k: 2.000000 > cA: 1.721078e-16 > cB: 3.061785e-08 > cC: 7.912911e-09 > Case 4: > psiA: 9.852477e-01 > psiC: 7.992223e-09 > k: 2.000000 > cA: 2.666770e-16 > cB: 4.610822e-08 > cC: 7.992223e-09 > > Basically, the cA's shoot down to zero and the cB's are slightly greater than zero. When I plug the c's into equations (1a) - (1b), I do not get the correct solution at all. I would expect discrepancies if my c's were originally negative, but for these five problems, shouldn't VI give the same answer as the Newton's method? > > Attached is the petsc4py code, the two input files containing psiA and psiC, and the outputs from '-snes_monitor -snes_view -ksp_monitor_true_residual'. Run as: > > python testVI.py testA testC 2 <0/1> > > where 0 is Newton, 1 is VI. > > Know what's going on here? > > Thanks, > Justin > > On Wed, Dec 2, 2015 at 1:36 PM, Barry Smith wrote: > > > On Dec 2, 2015, at 1:56 PM, Justin Chang wrote: > > > > Barry, > > > > So I do not believe there is any problem with the ISEqual() per se, because what I am doing is I am solving 5151 different VI problem using the same SNES/KSP/PC/Mat. That is, I am not "resetting" these data structures once I am done with one problem and move on to the next. If I ran each case individually, I get no error; the error comes when I run these problems sequentially without resetting the SNES after each problem. > > > > I haven't run the C debugger or done any of that yet, but could this be a plausible explanation for my error? > > This is exactly the issue. Independent of VIs if you change the size of a problem you pass to SNES you need to call SNESReset() between each call of a different sizes. > > > Originally I was thinking about creating/destroying SNES for each problem but I was wondering if that might affect performance. > > Using SNESReset() is the way you should do it. Creating and destroying it each time should only be a tiny, immeasurably slower. > > Barry > > > > > Thanks, > > Justin > > > > On Tue, Dec 1, 2015 at 5:58 PM, Barry Smith wrote: > > > > > On Dec 1, 2015, at 5:13 PM, Justin Chang wrote: > > > > > > Hi all, > > > > > > So I am running into some issues with the SNESVINEWTONRSLS solver. I originally had this done in firedrake, but have ported it to petsc4py. Attached is the code as well as the two required input files. > > > > > > First issue, when I run the program like this: > > > > > > python testVI.py psiA_1 psiB_1 2 1 > > > > > > I get this error: > > > > > > Traceback (most recent call last): > > > File "testVI.py", line 103, in > > > snes.solve(None,X) > > > File "PETSc/SNES.pyx", line 520, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:169677) > > > petsc4py.PETSc.Error: error code 60 > > > [0] SNESSolve() line 3992 in /Users/justin/Software/petsc/src/snes/interface/snes.c > > > [0] SNESSolve_VINEWTONRSLS() line 507 in /Users/justin/Software/petsc/src/snes/impls/vi/rs/virs.c > > > [0] KSPSetOperators() line 544 in /Users/justin/Software/petsc/src/ksp/ksp/interface/itcreate.c > > > [0] PCSetOperators() line 1170 in /Users/justin/Software/petsc/src/ksp/pc/interface/precon.c > > > [0] Nonconforming object sizes > > > [0] Cannot change local size of Amat after use old sizes 2 2 new sizes 3 3 > > > > > > No such error occurred when this was ran in firedrake, though I did notice that -snes_view did indicate that some of the iterations had 2x2 instead of 3x3 matrices. Why is this happening? I thought I should be solving a 3x3 system each time so why would a 2x2 ever arise? > > > > Justin, > > > > For VI's the rs (reduced space solver) solves the linearized problem on a subset of the variables, thus the size of the linear system may change from iteration of Newton to the next iteration of Newton. > > > > In the rs solver we have > > > > ierr = ISEqual(vi->IS_inact_prev,vi->IS_inact,&isequal);CHKERRQ(ierr); > > if (!isequal) { > > ierr = SNESVIResetPCandKSP(snes,jac_inact_inact,prejac_inact_inact);CHKERRQ(ierr); > > } > > > > so what __should__ happen is that if the size of the reduced space changes KSPReset() is called on the KSP allowing the KSP/PC to handle a different sized system then before. But why doesn't it work? Why isn't KSPReset() called? Somehow the ISEqual is not working. > > > > Can you run the C debugger and put a breakpoint in the ISEqual() then check the inputs and see if it is correctly setting the isequal to false when it should? Each time the vi->IS_inact changes the KSPReset() should get called. You can check this in the debugger. > > > > Barry > > > > > > > > More issues to come :) > > > > > > Thanks, > > > Justin > > > > > > > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > From sdettrick at trialphaenergy.com Thu Dec 3 17:03:28 2015 From: sdettrick at trialphaenergy.com (Sean Dettrick) Date: Thu, 3 Dec 2015 15:03:28 -0800 Subject: [petsc-users] Petsc Draw to pixmap In-Reply-To: <6F1F5E87-586D-4FA6-81FE-24241A139423@mcs.anl.gov> References: <10862A58-63D0-4CB3-B194-64A1DF1E08DD@trialphaenergy.com> <47E26DAB-01EB-4CF0-83F9-75AAC04BDDC7@mcs.anl.gov> <049242DC-BD76-454A-B16F-13C781281370@trialphaenergy.com> <7F5CF937-6C73-4EC2-A0BB-C11A3D8CEB40@trialphaenergy.com> <6F1F5E87-586D-4FA6-81FE-24241A139423@mcs.anl.gov> Message-ID: <73318954-EB89-426D-B6B8-F738FFC1B259@trialphaenergy.com> On Dec 3, 2015, at 5:01 AM, Barry Smith > wrote: On Dec 3, 2015, at 5:54 AM, Matthew Knepley > wrote: On Thu, Dec 3, 2015 at 2:06 AM, Sean Dettrick > wrote: No, it is my fault, when I was unable to download from afterstep.org earlier I didn?t realize that it was probably my work firewall blocking me. Thus sending me on a wild goose chase with macports and home-brew and github version, which didn?t compile. But now I?ve tried to follow your advice, and have a new problem. I installed Afterimage from your suggested site, with suggested X lib flags: ./configure --x-includes=/opt/X11/include -x-libraries=/opt/X11/lib Did you check that it worked? Configure can (semi-)silently turn things off. Yes, I think Matt is right. This is the error message you get if afterimage was installed without X. Check through all the logs and build stuff of afterimage. Thanks! The thing finally works. I had to clean everything and do fresh installs, but now it is fine. Summarizing my experience of afterimage on a mac in case anybody else wants to do it: Afterimage on the mac is in theory available via macports as part of the afterstep package, but it failed to build on my Mac (Yosemite). It is not available in home-brew. The current github version of afterimage failed to build on my mac. The version from http://www.afterstep.org/afterimage/getcode.php DOES work, configured as: ./configure --x-includes=/opt/X11/include --x-libraries=/opt/X11/lib --with-x make make install Then petsc is configured as: ./configure --with-afterimage --download-hdf5 --download-mpich The example src/dm/examples/tutorials/ex5.c then works: ./ex5 -draw_save Cheers, Sean Barry Thanks, Matt make make install No errors. Then re-configured and built petsc: export PETSC_DIR=/Users/sdettrick/libs/petsc-3.6.1 export PETSC_ARCH=mac-gcc-hdf5-mpich-afterimage ./configure --with-afterimage --download-hdf5 --download-mpich --PETSC_ARCH=mac-gcc-hdf5-mpich-afterimage --PETSC_DIR=/Users/sdettrick/libs/petsc-3.6.1 make PETSC_DIR=/Users/sdettrick/libs/petsc-3.6.1 PETSC_ARCH=mac-gcc-hdf5-mpich-afterimage all No errors. Then try to run an example with image output: cd src/dm/examples/tutorials/ make ex5 ./ex5 ./ex5 -draw_save test.png The example generates the error you mentioned, which it didn?t before installing afterimage: X Error of failed request: BadValue (integer parameter out of range for operation) Major opcode of failed request: 1 (X_CreateWindow) Value in failed request: 0x40 Serial number of failed request: 7 Current serial number in output stream: 14 Browsing through configure.log, it seems that petsc is using the same X11 library. Did I miss something? Thanks! Sean On Dec 2, 2015, at 11:10 PM, Barry Smith > wrote: I am such an idiot. No I did build it from source I forgot the comment in afterimage.py After image is available from http://www.afterstep.org/afterimage/getcode.php # # It is used by the PetscDrawSetSave() routine to save X windows graphics to files # # If installing on an Apple make sure to read the details on PetscDrawSetSave manual # page before installing # from that page I found If X windows generates an error message about X_CreateWindow() failing then Afterimage was installed without X windows. Reinstall Afterimage using the ./configure flags --x-includes=/pathtoXincludes --x-libraries=/pathtoXlibraries For example under Mac OS X Mountain Lion --x-includes=/opt/X11/include -x-libraries=/opt/X11/lib Did you try all this stuff? Barry On Dec 3, 2015, at 1:00 AM, Sean Dettrick > wrote: Hi Barry, Thanks for the amazingly fast reply. I installed home-brew but then was unable to find afterstep available as a package. The closest thing I could find was asterm (after step terminal emulator. Do you remember if you installed it from a canonical repo? I did find afterstep on github, but had trouble installing it on the mac. Meanwhile, a colleague has given me a python script, so I?ll probably use that instead for now. Thanks again, Sean On Dec 2, 2015, at 1:45 PM, Barry Smith > wrote: Sean, I have never been able to build from that source either. I used homebrew to install it and its worked for several years. I also tried to see if one could pull the image from the X pixel map and it looked like a huge project (the pixmap stuff is not as trivial as one would think it would be) so I gave up on that and started using afterimage (which essentially does all the for you). Barry On Dec 2, 2015, at 3:39 PM, Sean Dettrick > wrote: Hi, I?d like to have petsc draw output to a pixmap, but have been unable to install afterstep on my Mac. Direct download of the tar files from http://www.afterstep.org/afterimage/getcode.php failed, as did cvs checkout. It seems the target sites may not exist at present. Is there another way to install afterstep? Alternatively, is there a way to manually redirect the X11 output using something like XCreatePixmap? Thanks! Sean Dettrick -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dsu at eos.ubc.ca Thu Dec 3 17:09:29 2015 From: dsu at eos.ubc.ca (Danyang Su) Date: Thu, 3 Dec 2015 15:09:29 -0800 Subject: [petsc-users] SuperLU convergence problem In-Reply-To: References: <565F70E8.6030300@eos.ubc.ca> <5660A9CF.1030809@eos.ubc.ca> Message-ID: <5660CBA9.4020502@eos.ubc.ca> Hi Hong, I just checked using ex10 for these matrices and rhs, they all work fine. I found something is wrong in my code when using direct solver. The second parameter mat in PCFactorGetMatrix(PC pc,Mat *mat) is not initialized in my code for SUPERLU or MUMPS. I will fix this bug, rerun the tests and get back to you later. Thanks very much, Danyang On 15-12-03 01:59 PM, Hong wrote: > Danyang : > Further testing a_flow_check_168.bin, > ./ex10 -f0 > /Users/Hong/Downloads/matrix_and_rhs_bin/a_flow_check_168.bin -rhs > /Users/Hong/Downloads/matrix_and_rhs_bin/x_flow_check_168.bin -pc_type > lu -pc_factor_mat_solver_package superlu -ksp_monitor_true_residual > -mat_superlu_conditionnumber > Recip. condition number = 1.610480e-12 > 0 KSP preconditioned resid norm 6.873340313547e+09 true resid norm > 7.295020990196e+03 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.051833296449e-02 true resid norm > 2.976859070118e-02 ||r(i)||/||b|| 4.080672384793e-06 > Number of iterations = 1 > Residual norm 0.0297686 > > condition number of this matrix = 1/1.610480e-12 = 1.e+12, > i.e., this matrix is ill-conditioned. > > Hong > > > Hi Hong, > > The binary format of matrix, rhs and solution can be downloaded > via the link below. > > https://www.dropbox.com/s/cl3gfi0s0kjlktf/matrix_and_rhs_bin.tar.gz?dl=0 > > Thanks, > > Danyang > > > On 15-12-03 10:50 AM, Hong wrote: >> Danyang: >> >> >> >> To my surprising, solutions from SuperLU at timestep 29 seems >> not correct for the first 4 Newton iterations, but the >> solutions from iteration solver and MUMPS are correct. >> >> Please find all the matrices, rhs and solutions at timestep >> 29 via the link below. The data is a bit large so that I just >> share it through Dropbox. A piece of matlab code to read >> these data and then computer the norm has also been attached. >> _https://www.dropbox.com/s/rr8ueysgflmxs7h/results-check.tar.gz?dl=0_ >> >> >> Can you send us matrix in petsc binary format? >> >> e.g., call MatView(M, PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD)) >> or '-ksp_view_mat binary' >> >> Hong >> >> >> >> Below is a summary of the norm from the three solvers at >> timestep 29, newton iteration 1 to 5. >> >> Timestep 29 >> Norm of residual seq 1.661321e-09, superlu 1.657103e+04, >> mumps 3.731225e-11 >> Norm of residual seq 1.753079e-09, superlu 6.675467e+02, >> mumps 1.509919e-13 >> Norm of residual seq 4.914971e-10, superlu 1.236362e-01, >> mumps 2.139303e-17 >> Norm of residual seq 3.532769e-10, superlu 1.304670e-04, >> mumps 5.387000e-20 >> Norm of residual seq 3.885629e-10, superlu 2.754876e-07, >> mumps 4.108675e-21 >> >> Would anybody please check if SuperLU can solve these >> matrices? Another possibility is that something is wrong in >> my own code. But so far, I cannot find any problem in my code >> since the same code works fine if I using iterative solver or >> direct solver MUMPS. But for other cases I have tested, all >> these solvers work fine. >> >> Please let me know if I did not write down the problem clearly. >> >> Thanks, >> >> Danyang >> >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.croucher at auckland.ac.nz Thu Dec 3 18:22:12 2015 From: a.croucher at auckland.ac.nz (Adrian Croucher) Date: Fri, 04 Dec 2015 13:22:12 +1300 Subject: [petsc-users] SNES function domain error In-Reply-To: <5DD4C98F-F57E-46FF-917B-E315C9BC13D7@mcs.anl.gov> References: <565FC07E.4000805@auckland.ac.nz> <5DD4C98F-F57E-46FF-917B-E315C9BC13D7@mcs.anl.gov> Message-ID: <5660DCB4.6000803@auckland.ac.nz> hi On 03/12/15 18:43, Barry Smith wrote: > I absolutely do NOT recommend doing variable switching with SNES. Our Newton solvers are not designed with variable switching in mind so you get these absurd limitations like needing SNESLINESEARCHBASIC > > It would be great if someone who understood variable switching would either 1) contribute a Newton that supports variable switching to PETSc or 2) enhance the current SNESSolve_NewtonLS to support variable switching (I don't know which is easier). I had resigned myself to not being able to use a line search, partly because the nonlinear solver in the TOUGH2 simulator (which we've used for many years to solve geothermal flow problems with phase changes and variable switching) doesn't use one either, and it generally works fine without one. Without the line search, SNES is doing much the same thing and doesn't seem to be upset by the variable switching either. It's only the error handling that's causing trouble. But also I couldn't really see how a line search could be made to work with variable switching. When you switch variables, you're effectively doing a discontinuous leap into another part of the solution space. How can searching along a line between the old phase conditions and the new ones be made to make sense? Are there literature examples where people have done it? Cheers, Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher at auckland.ac.nz tel: +64 (0)9 923 84611 From bsmith at mcs.anl.gov Thu Dec 3 18:48:40 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 3 Dec 2015 18:48:40 -0600 Subject: [petsc-users] SNES function domain error In-Reply-To: <5660DCB4.6000803@auckland.ac.nz> References: <565FC07E.4000805@auckland.ac.nz> <5DD4C98F-F57E-46FF-917B-E315C9BC13D7@mcs.anl.gov> <5660DCB4.6000803@auckland.ac.nz> Message-ID: <747E517F-1D73-4D25-80A1-0066F5E37078@mcs.anl.gov> > On Dec 3, 2015, at 6:22 PM, Adrian Croucher wrote: > > hi > > On 03/12/15 18:43, Barry Smith wrote: >> I absolutely do NOT recommend doing variable switching with SNES. Our Newton solvers are not designed with variable switching in mind so you get these absurd limitations like needing SNESLINESEARCHBASIC >> >> It would be great if someone who understood variable switching would either 1) contribute a Newton that supports variable switching to PETSc or 2) enhance the current SNESSolve_NewtonLS to support variable switching (I don't know which is easier). > > I had resigned myself to not being able to use a line search, partly because the nonlinear solver in the TOUGH2 simulator (which we've used for many years to solve geothermal flow problems with phase changes and variable switching) doesn't use one either, and it generally works fine without one. Without the line search, SNES is doing much the same thing and doesn't seem to be upset by the variable switching either. It's only the error handling that's causing trouble. > > But also I couldn't really see how a line search could be made to work with variable switching. When you switch variables, you're effectively doing a discontinuous leap into another part of the solution space. How can searching along a line between the old phase conditions and the new ones be made to make sense? Are there literature examples where people have done it? These are hard questions (at least to me) which is why I've never been able to implement a full featured Newton with variable switching. Switching variables between function evaluations in a line search is completely nuts which is why I envision a "full featured" Newton with variable switching to be somewhat (or a lot) like active set (also called reduced space) Newton methods. Basically freeze the variables (not allow switching them) and build the Newton correction including a full line search in a subspace that wouldn't violate any "bounds" on the variables. Then after this Newton step you decide if certain variables should be switched, and then you do the next full featured Newton step. So instead of freely switching variables at each function evaluation you only switch them at each "Jacobian" evaluation. There is some PETSc code SNESSolve_VINEWTONRSLS that solves nonlinear equations with bounds on some or all variables; but in this case there are not all the complications of variable switching, basically when variables are on the bound you just "turned off" the related equation. So it is possible that it is a very special case of what is needed for variable switching. Actually if you could provide a simple case of where variable switching is using I'd like to try to develop some algorithms, so far when I ask people they give me a big messy things with many variables that I could never understand. Barry > > Cheers, Adrian > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > University of Auckland, New Zealand > email: a.croucher at auckland.ac.nz > tel: +64 (0)9 923 84611 > From timothee.nicolas at gmail.com Sat Dec 5 04:15:44 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Sat, 5 Dec 2015 19:15:44 +0900 Subject: [petsc-users] DMCreateMatrix from one DMDA to an another Message-ID: Hi, I need to create a rectangular matrix which takes vector structured by a DMDA and transforms them to vectors structured with an other DMDA. It does not look like there is a routine to do this since apparently DMCreateMatrix assumes square matrix. What is the clean way to do this ? The problem is I have a DMDA with 8 degrees of freedom (dof) and my matrix sends vectors living in a subspace with 3 dof to the other 5 dof subspace. I can define a square matrix living in the large, 8 dof subspace, but this is of course very inefficient in terms of memory and time. Thanks Best Timoth?e -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Sat Dec 5 05:41:14 2015 From: dave.mayhem23 at gmail.com (Dave May) Date: Sat, 5 Dec 2015 12:41:14 +0100 Subject: [petsc-users] DMCreateMatrix from one DMDA to an another In-Reply-To: References: Message-ID: On 5 December 2015 at 11:15, Timoth?e Nicolas wrote: > Hi, > > I need to create a rectangular matrix which takes vector structured by a > DMDA and transforms them to vectors structured with an other DMDA. It does > not look like there is a routine to do this since apparently DMCreateMatrix > assumes square matrix. What is the clean way to do this ? > Do the DMDA's have exactly the same parallel layout and only differ by the number of DOFs? If yes, and you only want to move entries between vectors associated with the two DMDA's, you can perform the transfer between vectors locally (rank-wise) in a very simple manner using the local number of points from DMDA and the known block sizes (DOFs) associated with your DMDA and the size of the subspace you ultimately want to construct. Thanks, Dave > The problem is I have a DMDA with 8 degrees of freedom (dof) and my matrix > sends vectors living in a subspace with 3 dof to the other 5 dof subspace. > I can define a square matrix living in the large, 8 dof subspace, but this > is of course very inefficient in terms of memory and time. > > Thanks > > Best > > Timoth?e > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dsu at eos.ubc.ca Sat Dec 5 18:11:18 2015 From: dsu at eos.ubc.ca (Danyang Su) Date: Sat, 05 Dec 2015 16:11:18 -0800 Subject: [petsc-users] SuperLU convergence problem (More test) In-Reply-To: References: <565F70E8.6030300@eos.ubc.ca> <5660A9CF.1030809@eos.ubc.ca> Message-ID: <56637D26.2080907@eos.ubc.ca> Hi Hong, I did more test today and finally found that the solution accuracy depends on the initial (first) matrix quality. I modified the ex52f.F to do the test. There are 6 matrices and right-hand-side vectors. All these matrices and rhs are from my reactive transport simulation. Results will be quite different depending on which one you use to do factorization. Results will also be different if you run with different options. My code is similar to the First or the Second test below. When the matrix is well conditioned, it works fine. But if the initial matrix is well conditioned, it likely to crash when the matrix become ill-conditioned. Since most of my case are well conditioned so I didn't detect the problem before. This case is a special one. How can I avoid this problem? Shall I redo factorization? Can PETSc automatically detect this prolbem or is there any option available to do this? All the data and test code (modified ex52f) can be found via the dropbox link below. _ __https://www.dropbox.com/s/4al1a60creogd8m/petsc-superlu-test.tar.gz?dl=0_ Summary of my test is shown below. First, use the Matrix 1 to setup KSP solver and factorization, then solve 168 to 172 mpiexec.hydra -n 1 ./ex52f -f0 /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin -rhs /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin -loop_matrices flow_check -loop_folder /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package superlu_dist Norm of error 3.8815E-11 iterations 1 -->Test for matrix 168 Norm of error 4.2307E-01 iterations 32 -->Test for matrix 169 Norm of error 3.0528E-01 iterations 32 -->Test for matrix 170 Norm of error 3.1177E-01 iterations 32 -->Test for matrix 171 Norm of error 3.2793E-01 iterations 32 -->Test for matrix 172 Norm of error 3.1251E-01 iterations 31 Second, use the Matrix 1 to setup KSP solver and factorization using the implemented SuperLU relative codes. I thought this will generate the same results as the First test, but it actually not. mpiexec.hydra -n 1 ./ex52f -f0 /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin -rhs /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin -loop_matrices flow_check -loop_folder /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -superlu_default Norm of error 2.2632E-12 iterations 1 -->Test for matrix 168 Norm of error 1.0817E+04 iterations 1 -->Test for matrix 169 Norm of error 1.0786E+04 iterations 1 -->Test for matrix 170 Norm of error 1.0792E+04 iterations 1 -->Test for matrix 171 Norm of error 1.0792E+04 iterations 1 -->Test for matrix 172 Norm of error 1.0792E+04 iterations 1 Third, use the Matrix 168 to setup KSP solver and factorization, then solve 168 to 172 mpiexec.hydra -n 1 ./ex52f -f0 /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin -rhs /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check -loop_folder /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package superlu_dist Norm of error 9.5528E-10 iterations 1 -->Test for matrix 168 Norm of error 9.4945E-10 iterations 1 -->Test for matrix 169 Norm of error 6.4279E-10 iterations 1 -->Test for matrix 170 Norm of error 7.4633E-10 iterations 1 -->Test for matrix 171 Norm of error 7.4863E-10 iterations 1 -->Test for matrix 172 Norm of error 8.9701E-10 iterations 1 Fourth, use the Matrix 168 to setup KSP solver and factorization using the implemented SuperLU relative codes. I thought this will generate the same results as the Third test, but it actually not. mpiexec.hydra -n 1 ./ex52f -f0 /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin -rhs /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check -loop_folder /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -superlu_default Norm of error 3.7017E-11 iterations 1 -->Test for matrix 168 Norm of error 3.6420E-11 iterations 1 -->Test for matrix 169 Norm of error 3.7184E-11 iterations 1 -->Test for matrix 170 Norm of error 3.6847E-11 iterations 1 -->Test for matrix 171 Norm of error 3.7883E-11 iterations 1 -->Test for matrix 172 Norm of error 3.8805E-11 iterations 1 Thanks very much, Danyang On 15-12-03 01:59 PM, Hong wrote: > Danyang : > Further testing a_flow_check_168.bin, > ./ex10 -f0 > /Users/Hong/Downloads/matrix_and_rhs_bin/a_flow_check_168.bin -rhs > /Users/Hong/Downloads/matrix_and_rhs_bin/x_flow_check_168.bin -pc_type > lu -pc_factor_mat_solver_package superlu -ksp_monitor_true_residual > -mat_superlu_conditionnumber > Recip. condition number = 1.610480e-12 > 0 KSP preconditioned resid norm 6.873340313547e+09 true resid norm > 7.295020990196e+03 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.051833296449e-02 true resid norm > 2.976859070118e-02 ||r(i)||/||b|| 4.080672384793e-06 > Number of iterations = 1 > Residual norm 0.0297686 > > condition number of this matrix = 1/1.610480e-12 = 1.e+12, > i.e., this matrix is ill-conditioned. > > Hong > > > Hi Hong, > > The binary format of matrix, rhs and solution can be downloaded > via the link below. > > https://www.dropbox.com/s/cl3gfi0s0kjlktf/matrix_and_rhs_bin.tar.gz?dl=0 > > Thanks, > > Danyang > > > On 15-12-03 10:50 AM, Hong wrote: >> Danyang: >> >> >> >> To my surprising, solutions from SuperLU at timestep 29 seems >> not correct for the first 4 Newton iterations, but the >> solutions from iteration solver and MUMPS are correct. >> >> Please find all the matrices, rhs and solutions at timestep >> 29 via the link below. The data is a bit large so that I just >> share it through Dropbox. A piece of matlab code to read >> these data and then computer the norm has also been attached. >> _https://www.dropbox.com/s/rr8ueysgflmxs7h/results-check.tar.gz?dl=0_ >> >> >> Can you send us matrix in petsc binary format? >> >> e.g., call MatView(M, PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD)) >> or '-ksp_view_mat binary' >> >> Hong >> >> >> >> Below is a summary of the norm from the three solvers at >> timestep 29, newton iteration 1 to 5. >> >> Timestep 29 >> Norm of residual seq 1.661321e-09, superlu 1.657103e+04, >> mumps 3.731225e-11 >> Norm of residual seq 1.753079e-09, superlu 6.675467e+02, >> mumps 1.509919e-13 >> Norm of residual seq 4.914971e-10, superlu 1.236362e-01, >> mumps 2.139303e-17 >> Norm of residual seq 3.532769e-10, superlu 1.304670e-04, >> mumps 5.387000e-20 >> Norm of residual seq 3.885629e-10, superlu 2.754876e-07, >> mumps 4.108675e-21 >> >> Would anybody please check if SuperLU can solve these >> matrices? Another possibility is that something is wrong in >> my own code. But so far, I cannot find any problem in my code >> since the same code works fine if I using iterative solver or >> direct solver MUMPS. But for other cases I have tested, all >> these solvers work fine. >> >> Please let me know if I did not write down the problem clearly. >> >> Thanks, >> >> Danyang >> >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothee.nicolas at gmail.com Sun Dec 6 04:19:41 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Sun, 6 Dec 2015 19:19:41 +0900 Subject: [petsc-users] DMCreateMatrix from one DMDA to an another In-Reply-To: References: Message-ID: Hi, The answer to the first question is yes. I was puzzled about the second part of the answer, but I realize I was not clear enough. I don't want to just transfer information between 2 DMDAs, I want to perform an operation on vectors with a rectangular matrix. To be more explicit, the input vector of the operation is a vector with 3 components (hence dof=3), and the results of the operator is put in one vector equation + 2 scalar equation, which makes it dof = 5. So, either I cannot fill my matrix (if the number of rows exceeds what is allowed by the DMDA), or I can fill my matrix but then the matrix and vectors are not compatible, as in this error : [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Nonconforming object sizes [0]PETSC ERROR: Mat mat,Vec y: global dim 4900 2940 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.1, Jul, 22, 2015 [0]PETSC ERROR: ./miips on a arch-linux2-c-debug named Carl-9000 by timothee Sun Dec 6 19:17:20 2015 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich [0]PETSC ERROR: #1 MatMult() line 2215 in /home/timothee/Documents/petsc-3.6.1/src/mat/interface/matrix.c So I assume it means the matrix is assumed to be square. Is there a way to change this ? Best Timoth?e 2015-12-05 20:41 GMT+09:00 Dave May : > > > On 5 December 2015 at 11:15, Timoth?e Nicolas > wrote: > >> Hi, >> >> I need to create a rectangular matrix which takes vector structured by a >> DMDA and transforms them to vectors structured with an other DMDA. It does >> not look like there is a routine to do this since apparently DMCreateMatrix >> assumes square matrix. What is the clean way to do this ? >> > > Do the DMDA's have exactly the same parallel layout and only differ by the > number of DOFs? > > If yes, and you only want to move entries between vectors associated with > the two DMDA's, you can perform the transfer between vectors locally > (rank-wise) in a very simple manner using the local number of points from > DMDA and the known block sizes (DOFs) associated with your DMDA and the > size of the subspace you ultimately want to construct. > > Thanks, > Dave > > >> The problem is I have a DMDA with 8 degrees of freedom (dof) and my >> matrix sends vectors living in a subspace with 3 dof to the other 5 dof >> subspace. I can define a square matrix living in the large, 8 dof subspace, >> but this is of course very inefficient in terms of memory and time. >> >> Thanks >> >> Best >> >> Timoth?e >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Sun Dec 6 07:29:15 2015 From: dave.mayhem23 at gmail.com (Dave May) Date: Sun, 6 Dec 2015 14:29:15 +0100 Subject: [petsc-users] DMCreateMatrix from one DMDA to an another In-Reply-To: References: Message-ID: On 6 December 2015 at 11:19, Timoth?e Nicolas wrote: > Hi, > > The answer to the first question is yes. I was puzzled about the second > part of the answer, but I realize I was not clear enough. I don't want to > just transfer information between 2 DMDAs, I want to perform an operation > on vectors with a rectangular matrix. > Okay - I misunderstood. > To be more explicit, the input vector of the operation is a vector with 3 > components (hence dof=3), and the results of the operator is put in one > vector equation + 2 scalar equation, which makes it dof = 5. > > So, either I cannot fill my matrix (if the number of rows exceeds what is > allowed by the DMDA), or I can fill my matrix but then the matrix and > vectors are not compatible, as in this error : > Do you need to explicitly assemble this rectangular operator? Expressing this operator via a matrix-free representation would be relatively straight forward. However, since the DMDA's have the same parallel layout you are in good shape to define the rectangular matrix if you really want the assembled thing. Your row space would be defined by the DMDA with block size 5 and the column space would be defined by the DMDA with block size 3. For example the matrix would be created like in the following way (assuming a 2D DMDA) PetscInt m,n,M,N; DMDAGetInfo(dm3,NULL,&M,&N,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL); nodes = M *N; DMDAGetCorners(dm3,NULL,NULL,NULL,&m,&n,NULL); lnodes = m * n; MatCreate(PETSC_COMM_WORLD,&A); MatSetSizes(A,lnodes*5,lnodes*3,nodes*5,nodes*3); The sparsity pattern is defined by the connectivity of the DMDA points AND the different block sizes. To define the non-zero structure, you need to traverse the part of the dmda defined by the natural indices given by si <= i < si+m, sj <= j < sj+n and check whether the the 5x3 block associated with each i,j is living within the diagonal or off-diagonal block of the matrix. For a given i,j the off-diagonal is defined by any i',j' associated your stencil which is not in the range si <= i' <= si + m, sj <= j' < sj + n. (e.g. it is a ghost node). Your arrays counting the non-zero entries on the diagonal (nnz[]) and off diag (nnz_od[]) can be indexed using i + j * m. This corresponds to the indexing used by the DMDA. Your non-zero counting function would look something like the pseudo code below PetscInt *nnz,*nnz_od; PetscMalloc(sizeof(PetscInt)*m*n*5,&nnz); PetscMalloc(sizeof(PetscInt)*m*n*5,&nnz_od); PetscMemzero(nnz,sizeof(PetscInt)*m*n*5); PetscMemzero(nnz_od,sizeof(PetscInt)*m*n*5); for (j=0; j > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Nonconforming object sizes > [0]PETSC ERROR: Mat mat,Vec y: global dim 4900 2940 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.1, Jul, 22, 2015 > [0]PETSC ERROR: ./miips on a arch-linux2-c-debug named Carl-9000 by > timothee Sun Dec 6 19:17:20 2015 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-fblaslapack --download-mpich > [0]PETSC ERROR: #1 MatMult() line 2215 in > /home/timothee/Documents/petsc-3.6.1/src/mat/interface/matrix.c > > So I assume it means the matrix is assumed to be square. Is there a way to > change this ? > > Best > > Timoth?e > > > > 2015-12-05 20:41 GMT+09:00 Dave May : > >> >> >> On 5 December 2015 at 11:15, Timoth?e Nicolas > > wrote: >> >>> Hi, >>> >>> I need to create a rectangular matrix which takes vector structured by a >>> DMDA and transforms them to vectors structured with an other DMDA. It does >>> not look like there is a routine to do this since apparently DMCreateMatrix >>> assumes square matrix. What is the clean way to do this ? >>> >> >> Do the DMDA's have exactly the same parallel layout and only differ by >> the number of DOFs? >> >> If yes, and you only want to move entries between vectors associated with >> the two DMDA's, you can perform the transfer between vectors locally >> (rank-wise) in a very simple manner using the local number of points from >> DMDA and the known block sizes (DOFs) associated with your DMDA and the >> size of the subspace you ultimately want to construct. >> >> Thanks, >> Dave >> >> >>> The problem is I have a DMDA with 8 degrees of freedom (dof) and my >>> matrix sends vectors living in a subspace with 3 dof to the other 5 dof >>> subspace. I can define a square matrix living in the large, 8 dof subspace, >>> but this is of course very inefficient in terms of memory and time. >>> >>> Thanks >>> >>> Best >>> >>> Timoth?e >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothee.nicolas at gmail.com Sun Dec 6 08:16:03 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Sun, 6 Dec 2015 23:16:03 +0900 Subject: [petsc-users] DMCreateMatrix from one DMDA to an another In-Reply-To: References: Message-ID: Wow, Great answer, thanks, I should probably be able to do it like this, I'll try my best ! I actually do want the assembled matrix now. I have coded the matrix-free version already and it works fine, but I want to use multigrid and I figured that in order to be be most efficient at the coarse level it is best to use -ksp_type preonly -pc_type lu, which requires a true matrix (the matrix I am talking about is the product between a matrix 3dof --> 5dof with a matrix 5dof --> 3dof, which ends up square). But I want to retain a memory light implementation so I use matrix-free formulation at all the levels but the last, which is also light by definition, so I should not suffer much in terms of memory. Thanks Timoth?e 2015-12-06 22:29 GMT+09:00 Dave May : > > > On 6 December 2015 at 11:19, Timoth?e Nicolas > wrote: > >> Hi, >> >> The answer to the first question is yes. I was puzzled about the second >> part of the answer, but I realize I was not clear enough. I don't want to >> just transfer information between 2 DMDAs, I want to perform an operation >> on vectors with a rectangular matrix. >> > > Okay - I misunderstood. > > > >> To be more explicit, the input vector of the operation is a vector with 3 >> components (hence dof=3), and the results of the operator is put in one >> vector equation + 2 scalar equation, which makes it dof = 5. >> >> So, either I cannot fill my matrix (if the number of rows exceeds what is >> allowed by the DMDA), or I can fill my matrix but then the matrix and >> vectors are not compatible, as in this error : >> > > > Do you need to explicitly assemble this rectangular operator? > Expressing this operator via a matrix-free representation would be > relatively straight forward. > > However, since the DMDA's have the same parallel layout you are in good > shape to define the rectangular matrix if you really want the assembled > thing. Your row space would be defined by the DMDA with block size 5 and > the column space would be defined by the DMDA with block size 3. For > example the matrix would be created like in the following way (assuming a > 2D DMDA) > > PetscInt m,n,M,N; > > > DMDAGetInfo(dm3,NULL,&M,&N,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL); > nodes = M *N; > DMDAGetCorners(dm3,NULL,NULL,NULL,&m,&n,NULL); > lnodes = m * n; > MatCreate(PETSC_COMM_WORLD,&A); > MatSetSizes(A,lnodes*5,lnodes*3,nodes*5,nodes*3); > > The sparsity pattern is defined by the connectivity of the DMDA points AND > the different block sizes. > To define the non-zero structure, you need to traverse the part of the > dmda defined by the natural indices given by si <= i < si+m, sj <= j < sj+n > and check whether the the 5x3 block associated with each i,j is living > within the diagonal or off-diagonal block of the matrix. For a given i,j > the off-diagonal is defined by any i',j' associated your stencil which is > not in the range si <= i' <= si + m, sj <= j' < sj + n. (e.g. it is a ghost > node). > > Your arrays counting the non-zero entries on the diagonal (nnz[]) and off > diag (nnz_od[]) can be indexed using i + j * m. This corresponds to the > indexing used by the DMDA. Your non-zero counting function would look > something like the pseudo code below > > PetscInt *nnz,*nnz_od; > > PetscMalloc(sizeof(PetscInt)*m*n*5,&nnz); > PetscMalloc(sizeof(PetscInt)*m*n*5,&nnz_od); > > PetscMemzero(nnz,sizeof(PetscInt)*m*n*5); > PetscMemzero(nnz_od,sizeof(PetscInt)*m*n*5); > > for (j=0; j for (i=0; i PetscInt idx,d,cnti,cntg; > PetscBool is_ghost; > > idx = i + j * m; > > cnti = 0; > cntg = 0; > for (all_points_in_stencil) { /* User logic to define this loop */ > > is_ghost = PETSC_FALSE; > /*User logic goes here to test if stencil point is a ghost point */ > > /* accumulate counts for the stencil */ > if (is_ghost) { cntg++; } /* values living on a neighbour rank */ > else { cnti++; } /* values local to current rank */ > } > > /* assume all dofs in row and col space in block are coupled to each > other */ > for (d=0; d<5; d++) { > nnz[5*idx+d] = 3 * cnti; > nnz_od[5*idx+d] = 3 * cntg; > } > > } > } > > Thanks, > Dave > > >> >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Nonconforming object sizes >> [0]PETSC ERROR: Mat mat,Vec y: global dim 4900 2940 >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.6.1, Jul, 22, 2015 >> [0]PETSC ERROR: ./miips on a arch-linux2-c-debug named Carl-9000 by >> timothee Sun Dec 6 19:17:20 2015 >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --with-fc=gfortran --download-fblaslapack --download-mpich >> [0]PETSC ERROR: #1 MatMult() line 2215 in >> /home/timothee/Documents/petsc-3.6.1/src/mat/interface/matrix.c >> >> So I assume it means the matrix is assumed to be square. Is there a way >> to change this ? >> >> Best >> >> Timoth?e >> >> >> >> 2015-12-05 20:41 GMT+09:00 Dave May : >> >>> >>> >>> On 5 December 2015 at 11:15, Timoth?e Nicolas < >>> timothee.nicolas at gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I need to create a rectangular matrix which takes vector structured by >>>> a DMDA and transforms them to vectors structured with an other DMDA. It >>>> does not look like there is a routine to do this since apparently >>>> DMCreateMatrix assumes square matrix. What is the clean way to do this ? >>>> >>> >>> Do the DMDA's have exactly the same parallel layout and only differ by >>> the number of DOFs? >>> >>> If yes, and you only want to move entries between vectors associated >>> with the two DMDA's, you can perform the transfer between vectors locally >>> (rank-wise) in a very simple manner using the local number of points from >>> DMDA and the known block sizes (DOFs) associated with your DMDA and the >>> size of the subspace you ultimately want to construct. >>> >>> Thanks, >>> Dave >>> >>> >>>> The problem is I have a DMDA with 8 degrees of freedom (dof) and my >>>> matrix sends vectors living in a subspace with 3 dof to the other 5 dof >>>> subspace. I can define a square matrix living in the large, 8 dof subspace, >>>> but this is of course very inefficient in terms of memory and time. >>>> >>>> Thanks >>>> >>>> Best >>>> >>>> Timoth?e >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bibrakc at gmail.com Mon Dec 7 00:11:14 2015 From: bibrakc at gmail.com (Bibrak Qamar) Date: Mon, 7 Dec 2015 01:11:14 -0500 Subject: [petsc-users] How to get values from a matrix and set again? Message-ID: Hi, I am trying to apply a function on each element of a matrix. The function is the sigmoid function which transforms each element of the matrix. I am trying to approach it like this but get error. What could be the problem? Or is there a better way to do this? int sigmoid(Mat x){ printf("in sigmoid\n"); PetscErrorCode ierr; PetscInt Istart, Iend, Ii, Ji, rows, cols; PetscScalar v = 0.0; ierr = MatGetOwnershipRange(x,&Istart,&Iend);CHKERRQ(ierr); MatGetSize(x,&rows,&cols); for (Ii=Istart; Ii From timothee.nicolas at gmail.com Mon Dec 7 00:29:51 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Mon, 7 Dec 2015 15:29:51 +0900 Subject: [petsc-users] How to get values from a matrix and set again? In-Reply-To: References: Message-ID: Hi, The error message seems to tell you that your matrix is not assembled. Have you used the calls MatAssemblyBegin and MatAssemblyEnd When you build your matrix ? Timothee 2015-12-07 15:11 GMT+09:00 Bibrak Qamar : > Hi, > > I am trying to apply a function on each element of a matrix. The function > is the sigmoid function which transforms each element of the matrix. > > I am trying to approach it like this but get error. What could be the > problem? Or is there a better way to do this? > > int sigmoid(Mat x){ > printf("in sigmoid\n"); > > PetscErrorCode ierr; > PetscInt Istart, Iend, Ii, Ji, rows, cols; > > PetscScalar v = 0.0; > ierr = MatGetOwnershipRange(x,&Istart,&Iend);CHKERRQ(ierr); > MatGetSize(x,&rows,&cols); > > for (Ii=Istart; Ii for (Ji=0; Ji ierr = MatGetValues(x,1,&Ii,1,&Ji,&v);CHKERRQ(ierr); > v = mysigmoid(v); > ierr = MatSetValues(x,1,&Ii,1,&Ji,&v,INSERT_VALUES);CHKERRQ(ierr); > } > } > > MatAssemblyBegin(x,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(x,MAT_FINAL_ASSEMBLY); > > > MatView(x,PETSC_VIEWER_STDOUT_WORLD); > } > > > error msg is: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Not for unassembled matrix > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [ > [0]PETSC ERROR: #1 MatGetValues() line 1780 in > /gpfs/home/petsc-3.6.3/src/mat/interface/matrix.c > [0]PETSC ERROR: #2 sigmoid() line 21 in neural.c > > > Thanks > Bibrak Qamar > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bibrakc at gmail.com Mon Dec 7 00:37:28 2015 From: bibrakc at gmail.com (Bibrak Qamar) Date: Mon, 7 Dec 2015 01:37:28 -0500 Subject: [petsc-users] How to get values from a matrix and set again? In-Reply-To: References: Message-ID: Yes, I called MatAssemblyBegin and MatAssemblyEnd. But I figured out that you need to call it every time you use MatGetValues() Here is the updated code: PetscInt sigmoidMat(Mat x){ printf("in sigmoid\n"); PetscErrorCode ierr; PetscInt Istart, Iend, Ii, Ji, rows, cols; PetscScalar v = 0.0; ierr = MatGetOwnershipRange(x,&Istart,&Iend);CHKERRQ(ierr); MatGetSize(x,&rows,&cols); for (Ii=Istart; Ii wrote: > Hi, > > The error message seems to tell you that your matrix is not assembled. > Have you used the calls > > MatAssemblyBegin > > and > > MatAssemblyEnd > > When you build your matrix ? > > Timothee > > > 2015-12-07 15:11 GMT+09:00 Bibrak Qamar : > >> Hi, >> >> I am trying to apply a function on each element of a matrix. The function >> is the sigmoid function which transforms each element of the matrix. >> >> I am trying to approach it like this but get error. What could be the >> problem? Or is there a better way to do this? >> >> int sigmoid(Mat x){ >> printf("in sigmoid\n"); >> >> PetscErrorCode ierr; >> PetscInt Istart, Iend, Ii, Ji, rows, cols; >> >> PetscScalar v = 0.0; >> ierr = MatGetOwnershipRange(x,&Istart,&Iend);CHKERRQ(ierr); >> MatGetSize(x,&rows,&cols); >> >> for (Ii=Istart; Ii> for (Ji=0; Ji> ierr = MatGetValues(x,1,&Ii,1,&Ji,&v);CHKERRQ(ierr); >> v = mysigmoid(v); >> ierr = MatSetValues(x,1,&Ii,1,&Ji,&v,INSERT_VALUES);CHKERRQ(ierr); >> } >> } >> >> MatAssemblyBegin(x,MAT_FINAL_ASSEMBLY); >> MatAssemblyEnd(x,MAT_FINAL_ASSEMBLY); >> >> >> MatView(x,PETSC_VIEWER_STDOUT_WORLD); >> } >> >> >> error msg is: >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Object is in wrong state >> [0]PETSC ERROR: Not for unassembled matrix >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >> [ >> [0]PETSC ERROR: #1 MatGetValues() line 1780 in >> /gpfs/home/petsc-3.6.3/src/mat/interface/matrix.c >> [0]PETSC ERROR: #2 sigmoid() line 21 in neural.c >> >> >> Thanks >> Bibrak Qamar >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothee.nicolas at gmail.com Mon Dec 7 00:43:29 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Mon, 7 Dec 2015 15:43:29 +0900 Subject: [petsc-users] How to get values from a matrix and set again? In-Reply-To: References: Message-ID: Indeed the manual says ( http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetValues.html ) MatGetValues () requires that the matrix has been assembled with MatAssemblyBegin ()/MatAssemblyEnd (). Thus, calls to MatSetValues () and MatGetValues () CANNOT be made in succession without intermediate matrix assembly. However in your case, I don't think you want to Assemble the matrix every time you retrieve a value, since you want to update all the matrix it seems highly unefficient. I think you should first put all the values you're interested in (presumably all the values in your matrix) in an array, transform the values in the array with your sigmoid function, and then put them back in the matrix, and finally assemble it. Timothee 2015-12-07 15:37 GMT+09:00 Bibrak Qamar : > Yes, I called MatAssemblyBegin and MatAssemblyEnd. > > But I figured out that you need to call it every time you use > MatGetValues() > > Here is the updated code: > > > PetscInt sigmoidMat(Mat x){ > printf("in sigmoid\n"); > > PetscErrorCode ierr; > PetscInt Istart, Iend, Ii, Ji, rows, cols; > > > PetscScalar v = 0.0; > ierr = MatGetOwnershipRange(x,&Istart,&Iend);CHKERRQ(ierr); > MatGetSize(x,&rows,&cols); > > > for (Ii=Istart; Ii for (Ji=0; Ji MatAssemblyBegin(x,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(x,MAT_FINAL_ASSEMBLY); > ierr = MatGetValues(x,1,&Ii,1,&Ji,&v);CHKERRQ(ierr); > v = 1.0/(1+exp(-1*v)); > ierr = MatSetValues(x,1,&Ii,1,&Ji,&v,INSERT_VALUES);CHKERRQ(ierr); > } > } > > MatAssemblyBegin(x,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(x,MAT_FINAL_ASSEMBLY); > > > MatView(x,PETSC_VIEWER_STDOUT_WORLD); > return 0; > } > > > > Bibrak Qamar > > > On Mon, Dec 7, 2015 at 1:29 AM, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > >> Hi, >> >> The error message seems to tell you that your matrix is not assembled. >> Have you used the calls >> >> MatAssemblyBegin >> >> and >> >> MatAssemblyEnd >> >> When you build your matrix ? >> >> Timothee >> >> >> 2015-12-07 15:11 GMT+09:00 Bibrak Qamar : >> >>> Hi, >>> >>> I am trying to apply a function on each element of a matrix. The >>> function is the sigmoid function which transforms each element of the >>> matrix. >>> >>> I am trying to approach it like this but get error. What could be the >>> problem? Or is there a better way to do this? >>> >>> int sigmoid(Mat x){ >>> printf("in sigmoid\n"); >>> >>> PetscErrorCode ierr; >>> PetscInt Istart, Iend, Ii, Ji, rows, cols; >>> >>> PetscScalar v = 0.0; >>> ierr = MatGetOwnershipRange(x,&Istart,&Iend);CHKERRQ(ierr); >>> MatGetSize(x,&rows,&cols); >>> >>> for (Ii=Istart; Ii>> for (Ji=0; Ji>> ierr = MatGetValues(x,1,&Ii,1,&Ji,&v);CHKERRQ(ierr); >>> v = mysigmoid(v); >>> ierr = MatSetValues(x,1,&Ii,1,&Ji,&v,INSERT_VALUES);CHKERRQ(ierr); >>> } >>> } >>> >>> MatAssemblyBegin(x,MAT_FINAL_ASSEMBLY); >>> MatAssemblyEnd(x,MAT_FINAL_ASSEMBLY); >>> >>> >>> MatView(x,PETSC_VIEWER_STDOUT_WORLD); >>> } >>> >>> >>> error msg is: >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Object is in wrong state >>> [0]PETSC ERROR: Not for unassembled matrix >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>> [ >>> [0]PETSC ERROR: #1 MatGetValues() line 1780 in >>> /gpfs/home/petsc-3.6.3/src/mat/interface/matrix.c >>> [0]PETSC ERROR: #2 sigmoid() line 21 in neural.c >>> >>> >>> Thanks >>> Bibrak Qamar >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Dec 7 05:23:35 2015 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 7 Dec 2015 05:23:35 -0600 Subject: [petsc-users] How to get values from a matrix and set again? In-Reply-To: References: Message-ID: On Mon, Dec 7, 2015 at 12:11 AM, Bibrak Qamar wrote: > Hi, > > I am trying to apply a function on each element of a matrix. The function > is the sigmoid function which transforms each element of the matrix. > This pattern is problematic because we have no good way to indicate to PETSc that only 1 process intends to change a value, so after each SetValues() you need an assembly. You want something like MatChop(), http://www.mcs.anl.gov/petsc/petsc-current/src/mat/utils/axpy.c.html#MatChop Thanks, Matt > I am trying to approach it like this but get error. What could be the > problem? Or is there a better way to do this? > > int sigmoid(Mat x){ > printf("in sigmoid\n"); > > PetscErrorCode ierr; > PetscInt Istart, Iend, Ii, Ji, rows, cols; > > PetscScalar v = 0.0; > ierr = MatGetOwnershipRange(x,&Istart,&Iend);CHKERRQ(ierr); > MatGetSize(x,&rows,&cols); > > for (Ii=Istart; Ii for (Ji=0; Ji ierr = MatGetValues(x,1,&Ii,1,&Ji,&v);CHKERRQ(ierr); > v = mysigmoid(v); > ierr = MatSetValues(x,1,&Ii,1,&Ji,&v,INSERT_VALUES);CHKERRQ(ierr); > } > } > > MatAssemblyBegin(x,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(x,MAT_FINAL_ASSEMBLY); > > > MatView(x,PETSC_VIEWER_STDOUT_WORLD); > } > > > error msg is: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Not for unassembled matrix > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [ > [0]PETSC ERROR: #1 MatGetValues() line 1780 in > /gpfs/home/petsc-3.6.3/src/mat/interface/matrix.c > [0]PETSC ERROR: #2 sigmoid() line 21 in neural.c > > > Thanks > Bibrak Qamar > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhmerchant at gmail.com Mon Dec 7 11:21:36 2015 From: bhmerchant at gmail.com (Brian Merchant) Date: Mon, 7 Dec 2015 09:21:36 -0800 Subject: [petsc-users] Is it still worth switching to PETSc if I can't write a Jacobian for my problem? Message-ID: Hi all, I am considering using petsc4py instead of scipy.integrate.odeint (which is a wrapper for Fortran solvers) for a problem involving the solution of a system of ODEs. The problem has the potential to be stiff. Writing down its Jacobian is very hard. So far, I have been able to produce reasonable speed gains by writing the RHS functions in "something like C" (using either numba or Cython). I'd like to get even more performance out, hence my consideration of PETSc. Due to the large number of equations involved, it is already tedious to think about writing down a Jacobian. Even worse though, is that some of the functions governing a particular interaction do not have neat analytical forms (let alone whether or not their derivatives have neat analytical forms), so we might have a mess of piecewise functions needed to approximate them if we were to go about still trying to produce a Jacobian... All the toy examples I see of PETSc time stepping problems have Jacobians defined, so I wonder if I would even get a speed gain going from switching to it, if perhaps one of the reasons why I have a high computational cost is due to not being able to provide a Jacobian function? I described the sort of problem I am working with in more detail in this scicomp.stackexchange question, which is where most of this question is copied from, except it also comes with a toy version of the problem I am dealing with: http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a All your advice would be most helpful :) Kind regards,Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Mon Dec 7 12:42:07 2015 From: hzhang at mcs.anl.gov (Hong) Date: Mon, 7 Dec 2015 12:42:07 -0600 Subject: [petsc-users] SuperLU convergence problem (More test) In-Reply-To: <56637D26.2080907@eos.ubc.ca> References: <565F70E8.6030300@eos.ubc.ca> <5660A9CF.1030809@eos.ubc.ca> <56637D26.2080907@eos.ubc.ca> Message-ID: Danyang : Adding '-mat_superlu_dist_fact SamePattern' fixed the problem. Below is how I figured it out. 1. Reading ex52f.F, I see '-superlu_default' = '-pc_factor_mat_solver_package superlu_dist', the later enables runtime options for other packages. I use superlu_dist-4.2 and superlu-4.1 for the tests below. 2. Use the Matrix 168 to setup KSP solver and factorization, all packages, petsc, superlu_dist and mumps give same correct results: ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package petsc -->loac matrix a -->load rhs b size l,m,n,mm 90000 90000 90000 90000 Norm of error 7.7308E-11 iterations 1 -->Test for matrix 168 .. -->Test for matrix 172 Norm of error 3.8461E-11 iterations 1 ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package superlu_dist Norm of error 9.4073E-11 iterations 1 -->Test for matrix 168 ... -->Test for matrix 172 Norm of error 3.8187E-11 iterations 1 3. Use superlu, I get ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package superlu Norm of error 1.0191E-06 iterations 1 -->Test for matrix 168 ... -->Test for matrix 172 Norm of error 9.7858E-07 iterations 1 Replacing default DiagPivotThresh: 1. to 0.0, I get same solutions as other packages: ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package superlu -mat_superlu_diagpivotthresh 0.0 Norm of error 8.3614E-11 iterations 1 -->Test for matrix 168 ... -->Test for matrix 172 Norm of error 3.7098E-11 iterations 1 4. using '-mat_view ascii::ascii_info', I found that a_flow_check_1.bin and a_flow_check_168.bin seem have same structure: -->loac matrix a Mat Object: 1 MPI processes type: seqaij rows=90000, cols=90000 total: nonzeros=895600, allocated nonzeros=895600 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 45000 nodes, limit used is 5 5. Using a_flow_check_1.bin, I am able to reproduce the error you reported: all packages give correct results except superlu_dist: ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_1.bin -rhs matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package superlu_dist Norm of error 2.5970E-12 iterations 1 -->Test for matrix 168 Norm of error 1.3936E-01 iterations 34 -->Test for matrix 169 I guess the error might come from reuse of matrix factor. Replacing default -mat_superlu_dist_fact with -mat_superlu_dist_fact SamePattern, I get ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_1.bin -rhs matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package superlu_dist -mat_superlu_dist_fact SamePattern Norm of error 2.5970E-12 iterations 1 -->Test for matrix 168 Norm of error 9.4073E-11 iterations 1 -->Test for matrix 169 Norm of error 6.4303E-11 iterations 1 -->Test for matrix 170 Norm of error 7.4327E-11 iterations 1 -->Test for matrix 171 Norm of error 5.4162E-11 iterations 1 -->Test for matrix 172 Norm of error 3.4440E-11 iterations 1 --> End of test, bye Sherry may tell you why SamePattern_SameRowPerm cause the difference here. Best on the above experiments, I would set following as default '-mat_superlu_diagpivotthresh 0.0' in petsc/superlu interface. '-mat_superlu_dist_fact SamePattern' in petsc/superlu_dist interface. Hong Hi Hong, > > I did more test today and finally found that the solution accuracy depends > on the initial (first) matrix quality. I modified the ex52f.F to do the > test. There are 6 matrices and right-hand-side vectors. All these matrices > and rhs are from my reactive transport simulation. Results will be quite > different depending on which one you use to do factorization. Results will > also be different if you run with different options. My code is similar to > the First or the Second test below. When the matrix is well conditioned, it > works fine. But if the initial matrix is well conditioned, it likely to > crash when the matrix become ill-conditioned. Since most of my case are > well conditioned so I didn't detect the problem before. This case is a > special one. > > > How can I avoid this problem? Shall I redo factorization? Can PETSc > automatically detect this prolbem or is there any option available to do > this? > > All the data and test code (modified ex52f) can be found via the dropbox > link below. > > *https://www.dropbox.com/s/4al1a60creogd8m/petsc-superlu-test.tar.gz?dl=0 > * > > > Summary of my test is shown below. > > First, use the Matrix 1 to setup KSP solver and factorization, then solve > 168 to 172 > > mpiexec.hydra -n 1 ./ex52f -f0 > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin > -rhs > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin > -loop_matrices flow_check -loop_folder > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -pc_type lu > -pc_factor_mat_solver_package superlu_dist > > Norm of error 3.8815E-11 iterations 1 > -->Test for matrix 168 > Norm of error 4.2307E-01 iterations 32 > -->Test for matrix 169 > Norm of error 3.0528E-01 iterations 32 > -->Test for matrix 170 > Norm of error 3.1177E-01 iterations 32 > -->Test for matrix 171 > Norm of error 3.2793E-01 iterations 32 > -->Test for matrix 172 > Norm of error 3.1251E-01 iterations 31 > > Second, use the Matrix 1 to setup KSP solver and factorization using the > implemented SuperLU relative codes. I thought this will generate the same > results as the First test, but it actually not. > > mpiexec.hydra -n 1 ./ex52f -f0 > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin > -rhs > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin > -loop_matrices flow_check -loop_folder > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -superlu_default > > Norm of error 2.2632E-12 iterations 1 > -->Test for matrix 168 > Norm of error 1.0817E+04 iterations 1 > -->Test for matrix 169 > Norm of error 1.0786E+04 iterations 1 > -->Test for matrix 170 > Norm of error 1.0792E+04 iterations 1 > -->Test for matrix 171 > Norm of error 1.0792E+04 iterations 1 > -->Test for matrix 172 > Norm of error 1.0792E+04 iterations 1 > > > Third, use the Matrix 168 to setup KSP solver and factorization, then > solve 168 to 172 > > mpiexec.hydra -n 1 ./ex52f -f0 > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin > -rhs > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin > -loop_matrices flow_check -loop_folder > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -pc_type lu > -pc_factor_mat_solver_package superlu_dist > > Norm of error 9.5528E-10 iterations 1 > -->Test for matrix 168 > Norm of error 9.4945E-10 iterations 1 > -->Test for matrix 169 > Norm of error 6.4279E-10 iterations 1 > -->Test for matrix 170 > Norm of error 7.4633E-10 iterations 1 > -->Test for matrix 171 > Norm of error 7.4863E-10 iterations 1 > -->Test for matrix 172 > Norm of error 8.9701E-10 iterations 1 > > Fourth, use the Matrix 168 to setup KSP solver and factorization using the > implemented SuperLU relative codes. I thought this will generate the same > results as the Third test, but it actually not. > > mpiexec.hydra -n 1 ./ex52f -f0 > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin > -rhs > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin > -loop_matrices flow_check -loop_folder > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -superlu_default > > Norm of error 3.7017E-11 iterations 1 > -->Test for matrix 168 > Norm of error 3.6420E-11 iterations 1 > -->Test for matrix 169 > Norm of error 3.7184E-11 iterations 1 > -->Test for matrix 170 > Norm of error 3.6847E-11 iterations 1 > -->Test for matrix 171 > Norm of error 3.7883E-11 iterations 1 > -->Test for matrix 172 > Norm of error 3.8805E-11 iterations 1 > > Thanks very much, > > Danyang > > On 15-12-03 01:59 PM, Hong wrote: > > Danyang : > Further testing a_flow_check_168.bin, > ./ex10 -f0 /Users/Hong/Downloads/matrix_and_rhs_bin/a_flow_check_168.bin > -rhs /Users/Hong/Downloads/matrix_and_rhs_bin/x_flow_check_168.bin -pc_type > lu -pc_factor_mat_solver_package superlu -ksp_monitor_true_residual > -mat_superlu_conditionnumber > Recip. condition number = 1.610480e-12 > 0 KSP preconditioned resid norm 6.873340313547e+09 true resid norm > 7.295020990196e+03 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.051833296449e-02 true resid norm > 2.976859070118e-02 ||r(i)||/||b|| 4.080672384793e-06 > Number of iterations = 1 > Residual norm 0.0297686 > > condition number of this matrix = 1/1.610480e-12 = 1.e+12, > i.e., this matrix is ill-conditioned. > > Hong > > > Hi Hong, >> >> The binary format of matrix, rhs and solution can be downloaded via the >> link below. >> >> https://www.dropbox.com/s/cl3gfi0s0kjlktf/matrix_and_rhs_bin.tar.gz?dl=0 >> >> Thanks, >> >> Danyang >> >> >> On 15-12-03 10:50 AM, Hong wrote: >> >> Danyang: >> >>> >>> >>> To my surprising, solutions from SuperLU at timestep 29 seems not >>> correct for the first 4 Newton iterations, but the solutions from iteration >>> solver and MUMPS are correct. >>> >>> Please find all the matrices, rhs and solutions at timestep 29 via the >>> link below. The data is a bit large so that I just share it through >>> Dropbox. A piece of matlab code to read these data and then computer the >>> norm has also been attached. >>> *https://www.dropbox.com/s/rr8ueysgflmxs7h/results-check.tar.gz?dl=0 >>> * >>> >> >> Can you send us matrix in petsc binary format? >> >> e.g., call MatView(M, PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD)) >> or '-ksp_view_mat binary' >> >> Hong >> >>> >>> >>> Below is a summary of the norm from the three solvers at timestep 29, >>> newton iteration 1 to 5. >>> >>> Timestep 29 >>> Norm of residual seq 1.661321e-09, superlu 1.657103e+04, mumps >>> 3.731225e-11 >>> Norm of residual seq 1.753079e-09, superlu 6.675467e+02, mumps >>> 1.509919e-13 >>> Norm of residual seq 4.914971e-10, superlu 1.236362e-01, mumps >>> 2.139303e-17 >>> Norm of residual seq 3.532769e-10, superlu 1.304670e-04, mumps >>> 5.387000e-20 >>> Norm of residual seq 3.885629e-10, superlu 2.754876e-07, mumps >>> 4.108675e-21 >>> >>> Would anybody please check if SuperLU can solve these matrices? Another >>> possibility is that something is wrong in my own code. But so far, I cannot >>> find any problem in my code since the same code works fine if I using >>> iterative solver or direct solver MUMPS. But for other cases I have >>> tested, all these solvers work fine. >>> >>> Please let me know if I did not write down the problem clearly. >>> >>> Thanks, >>> >>> Danyang >>> >>> >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Dec 7 13:17:18 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 7 Dec 2015 13:17:18 -0600 Subject: [petsc-users] How to get values from a matrix and set again? In-Reply-To: References: Message-ID: <04347E95-0E68-4867-A496-B4CC4B3912F6@mcs.anl.gov> Presumably you are using a MPIAIJ matrix. You can use Mat x,A,B; MatMPIAIJGetSeqAIJ(x,&A,&B,NULL); PetscScalar *a; MatSeqAIJGetArray(A,&a); MatInfo info; MatGetInfo(A,MAT_LOCAL,&info); for (i=0; i On Dec 7, 2015, at 12:11 AM, Bibrak Qamar wrote: > > Hi, > > I am trying to apply a function on each element of a matrix. The function is the sigmoid function which transforms each element of the matrix. > > I am trying to approach it like this but get error. What could be the problem? Or is there a better way to do this? > > int sigmoid(Mat x){ > printf("in sigmoid\n"); > > PetscErrorCode ierr; > PetscInt Istart, Iend, Ii, Ji, rows, cols; > > PetscScalar v = 0.0; > ierr = MatGetOwnershipRange(x,&Istart,&Iend);CHKERRQ(ierr); > MatGetSize(x,&rows,&cols); > > for (Ii=Istart; Ii for (Ji=0; Ji ierr = MatGetValues(x,1,&Ii,1,&Ji,&v);CHKERRQ(ierr); > v = mysigmoid(v); > ierr = MatSetValues(x,1,&Ii,1,&Ji,&v,INSERT_VALUES);CHKERRQ(ierr); > } > } > > MatAssemblyBegin(x,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(x,MAT_FINAL_ASSEMBLY); > > > MatView(x,PETSC_VIEWER_STDOUT_WORLD); > } > > > error msg is: > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Not for unassembled matrix > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [ > [0]PETSC ERROR: #1 MatGetValues() line 1780 in /gpfs/home/petsc-3.6.3/src/mat/interface/matrix.c > [0]PETSC ERROR: #2 sigmoid() line 21 in neural.c > > > Thanks > Bibrak Qamar > From dsu at eos.ubc.ca Mon Dec 7 13:28:26 2015 From: dsu at eos.ubc.ca (Danyang Su) Date: Mon, 7 Dec 2015 11:28:26 -0800 Subject: [petsc-users] SuperLU convergence problem (More test) In-Reply-To: References: <565F70E8.6030300@eos.ubc.ca> <5660A9CF.1030809@eos.ubc.ca> <56637D26.2080907@eos.ubc.ca> Message-ID: <5665DDDA.405@eos.ubc.ca> Hello Hong, Thanks for the quick reply and the option "-mat_superlu_dist_fact SamePattern" works like a charm, if I use this option from the command line. How can I add this option as the default. I tried using PetscOptionsInsertString("-mat_superlu_dist_fact SamePattern",ierr) in my code but this does not work. Thanks, Danyang On 15-12-07 10:42 AM, Hong wrote: > Danyang : > > Adding '-mat_superlu_dist_fact SamePattern' fixed the problem. Below > is how I figured it out. > > 1. Reading ex52f.F, I see '-superlu_default' = > '-pc_factor_mat_solver_package superlu_dist', the later enables > runtime options for other packages. I use superlu_dist-4.2 and > superlu-4.1 for the tests below. > > 2. Use the Matrix 168 to setup KSP solver and factorization, all > packages, petsc, superlu_dist and mumps give same correct results: > > ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs > matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check > -loop_folder matrix_and_rhs_bin -pc_type lu > -pc_factor_mat_solver_package petsc > -->loac matrix a > -->load rhs b > size l,m,n,mm 90000 90000 90000 90000 > Norm of error 7.7308E-11 iterations 1 > -->Test for matrix 168 > .. > -->Test for matrix 172 > Norm of error 3.8461E-11 iterations 1 > > ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs > matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check > -loop_folder matrix_and_rhs_bin -pc_type lu > -pc_factor_mat_solver_package superlu_dist > Norm of error 9.4073E-11 iterations 1 > -->Test for matrix 168 > ... > -->Test for matrix 172 > Norm of error 3.8187E-11 iterations 1 > > 3. Use superlu, I get > ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs > matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check > -loop_folder matrix_and_rhs_bin -pc_type lu > -pc_factor_mat_solver_package superlu > Norm of error 1.0191E-06 iterations 1 > -->Test for matrix 168 > ... > -->Test for matrix 172 > Norm of error 9.7858E-07 iterations 1 > > Replacing default DiagPivotThresh: 1. to 0.0, I get same solutions as > other packages: > > ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs > matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check > -loop_folder matrix_and_rhs_bin -pc_type lu > -pc_factor_mat_solver_package superlu -mat_superlu_diagpivotthresh 0.0 > > Norm of error 8.3614E-11 iterations 1 > -->Test for matrix 168 > ... > -->Test for matrix 172 > Norm of error 3.7098E-11 iterations 1 > > 4. > using '-mat_view ascii::ascii_info', I found that a_flow_check_1.bin > and a_flow_check_168.bin seem have same structure: > > -->loac matrix a > Mat Object: 1 MPI processes > type: seqaij > rows=90000, cols=90000 > total: nonzeros=895600, allocated nonzeros=895600 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 45000 nodes, limit used is 5 > > 5. > Using a_flow_check_1.bin, I am able to reproduce the error you > reported: all packages give correct results except superlu_dist: > ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_1.bin -rhs > matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check > -loop_folder matrix_and_rhs_bin -pc_type lu > -pc_factor_mat_solver_package superlu_dist > Norm of error 2.5970E-12 iterations 1 > -->Test for matrix 168 > Norm of error 1.3936E-01 iterations 34 > -->Test for matrix 169 > > I guess the error might come from reuse of matrix factor. Replacing > default > -mat_superlu_dist_fact with > -mat_superlu_dist_fact SamePattern, I get > > ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_1.bin -rhs > matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check > -loop_folder matrix_and_rhs_bin -pc_type lu > -pc_factor_mat_solver_package superlu_dist -mat_superlu_dist_fact > SamePattern > > Norm of error 2.5970E-12 iterations 1 > -->Test for matrix 168 > Norm of error 9.4073E-11 iterations 1 > -->Test for matrix 169 > Norm of error 6.4303E-11 iterations 1 > -->Test for matrix 170 > Norm of error 7.4327E-11 iterations 1 > -->Test for matrix 171 > Norm of error 5.4162E-11 iterations 1 > -->Test for matrix 172 > Norm of error 3.4440E-11 iterations 1 > --> End of test, bye > > Sherry may tell you why SamePattern_SameRowPerm cause the difference here. > Best on the above experiments, I would set following as default > '-mat_superlu_diagpivotthresh 0.0' in petsc/superlu interface. > '-mat_superlu_dist_fact SamePattern' in petsc/superlu_dist interface. > > Hong > > Hi Hong, > > I did more test today and finally found that the solution accuracy > depends on the initial (first) matrix quality. I modified the > ex52f.F to do the test. There are 6 matrices and right-hand-side > vectors. All these matrices and rhs are from my reactive transport > simulation. Results will be quite different depending on which one > you use to do factorization. Results will also be different if you > run with different options. My code is similar to the First or the > Second test below. When the matrix is well conditioned, it works > fine. But if the initial matrix is well conditioned, it likely to > crash when the matrix become ill-conditioned. Since most of my > case are well conditioned so I didn't detect the problem before. > This case is a special one. > > > How can I avoid this problem? Shall I redo factorization? Can > PETSc automatically detect this prolbem or is there any option > available to do this? > > All the data and test code (modified ex52f) can be found via the > dropbox link below. > _ > __https://www.dropbox.com/s/4al1a60creogd8m/petsc-superlu-test.tar.gz?dl=0_ > > > Summary of my test is shown below. > > First, use the Matrix 1 to setup KSP solver and factorization, > then solve 168 to 172 > > mpiexec.hydra -n 1 ./ex52f -f0 > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin > -rhs > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin > -loop_matrices flow_check -loop_folder > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -pc_type lu > -pc_factor_mat_solver_package superlu_dist > > Norm of error 3.8815E-11 iterations 1 > -->Test for matrix 168 > Norm of error 4.2307E-01 iterations 32 > -->Test for matrix 169 > Norm of error 3.0528E-01 iterations 32 > -->Test for matrix 170 > Norm of error 3.1177E-01 iterations 32 > -->Test for matrix 171 > Norm of error 3.2793E-01 iterations 32 > -->Test for matrix 172 > Norm of error 3.1251E-01 iterations 31 > > Second, use the Matrix 1 to setup KSP solver and factorization > using the implemented SuperLU relative codes. I thought this will > generate the same results as the First test, but it actually not. > > mpiexec.hydra -n 1 ./ex52f -f0 > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin > -rhs > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin > -loop_matrices flow_check -loop_folder > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -superlu_default > > Norm of error 2.2632E-12 iterations 1 > -->Test for matrix 168 > Norm of error 1.0817E+04 iterations 1 > -->Test for matrix 169 > Norm of error 1.0786E+04 iterations 1 > -->Test for matrix 170 > Norm of error 1.0792E+04 iterations 1 > -->Test for matrix 171 > Norm of error 1.0792E+04 iterations 1 > -->Test for matrix 172 > Norm of error 1.0792E+04 iterations 1 > > > Third, use the Matrix 168 to setup KSP solver and factorization, > then solve 168 to 172 > > mpiexec.hydra -n 1 ./ex52f -f0 > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin > -rhs > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin > -loop_matrices flow_check -loop_folder > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -pc_type lu > -pc_factor_mat_solver_package superlu_dist > > Norm of error 9.5528E-10 iterations 1 > -->Test for matrix 168 > Norm of error 9.4945E-10 iterations 1 > -->Test for matrix 169 > Norm of error 6.4279E-10 iterations 1 > -->Test for matrix 170 > Norm of error 7.4633E-10 iterations 1 > -->Test for matrix 171 > Norm of error 7.4863E-10 iterations 1 > -->Test for matrix 172 > Norm of error 8.9701E-10 iterations 1 > > Fourth, use the Matrix 168 to setup KSP solver and factorization > using the implemented SuperLU relative codes. I thought this will > generate the same results as the Third test, but it actually not. > > mpiexec.hydra -n 1 ./ex52f -f0 > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin > -rhs > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin > -loop_matrices flow_check -loop_folder > /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -superlu_default > > Norm of error 3.7017E-11 iterations 1 > -->Test for matrix 168 > Norm of error 3.6420E-11 iterations 1 > -->Test for matrix 169 > Norm of error 3.7184E-11 iterations 1 > -->Test for matrix 170 > Norm of error 3.6847E-11 iterations 1 > -->Test for matrix 171 > Norm of error 3.7883E-11 iterations 1 > -->Test for matrix 172 > Norm of error 3.8805E-11 iterations 1 > > Thanks very much, > > Danyang > > On 15-12-03 01:59 PM, Hong wrote: >> Danyang : >> Further testing a_flow_check_168.bin, >> ./ex10 -f0 >> /Users/Hong/Downloads/matrix_and_rhs_bin/a_flow_check_168.bin >> -rhs >> /Users/Hong/Downloads/matrix_and_rhs_bin/x_flow_check_168.bin >> -pc_type lu -pc_factor_mat_solver_package superlu >> -ksp_monitor_true_residual -mat_superlu_conditionnumber >> Recip. condition number = 1.610480e-12 >> 0 KSP preconditioned resid norm 6.873340313547e+09 true resid >> norm 7.295020990196e+03 ||r(i)||/||b|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.051833296449e-02 true resid >> norm 2.976859070118e-02 ||r(i)||/||b|| 4.080672384793e-06 >> Number of iterations = 1 >> Residual norm 0.0297686 >> >> condition number of this matrix = 1/1.610480e-12 = 1.e+12, >> i.e., this matrix is ill-conditioned. >> >> Hong >> >> >> Hi Hong, >> >> The binary format of matrix, rhs and solution can be >> downloaded via the link below. >> >> https://www.dropbox.com/s/cl3gfi0s0kjlktf/matrix_and_rhs_bin.tar.gz?dl=0 >> >> Thanks, >> >> Danyang >> >> >> On 15-12-03 10:50 AM, Hong wrote: >>> Danyang: >>> >>> >>> >>> To my surprising, solutions from SuperLU at timestep 29 >>> seems not correct for the first 4 Newton iterations, but >>> the solutions from iteration solver and MUMPS are correct. >>> >>> Please find all the matrices, rhs and solutions at >>> timestep 29 via the link below. The data is a bit large >>> so that I just share it through Dropbox. A piece of >>> matlab code to read these data and then computer the >>> norm has also been attached. >>> _https://www.dropbox.com/s/rr8ueysgflmxs7h/results-check.tar.gz?dl=0_ >>> >>> >>> Can you send us matrix in petsc binary format? >>> >>> e.g., call MatView(M, PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD)) >>> or '-ksp_view_mat binary' >>> >>> Hong >>> >>> >>> >>> Below is a summary of the norm from the three solvers at >>> timestep 29, newton iteration 1 to 5. >>> >>> Timestep 29 >>> Norm of residual seq 1.661321e-09, superlu 1.657103e+04, >>> mumps 3.731225e-11 >>> Norm of residual seq 1.753079e-09, superlu 6.675467e+02, >>> mumps 1.509919e-13 >>> Norm of residual seq 4.914971e-10, superlu 1.236362e-01, >>> mumps 2.139303e-17 >>> Norm of residual seq 3.532769e-10, superlu 1.304670e-04, >>> mumps 5.387000e-20 >>> Norm of residual seq 3.885629e-10, superlu 2.754876e-07, >>> mumps 4.108675e-21 >>> >>> Would anybody please check if SuperLU can solve these >>> matrices? Another possibility is that something is wrong >>> in my own code. But so far, I cannot find any problem in >>> my code since the same code works fine if I using >>> iterative solver or direct solver MUMPS. But for other >>> cases I have tested, all these solvers work fine. >>> >>> Please let me know if I did not write down the problem >>> clearly. >>> >>> Thanks, >>> >>> Danyang >>> >>> >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Mon Dec 7 14:01:20 2015 From: hzhang at mcs.anl.gov (Hong) Date: Mon, 7 Dec 2015 14:01:20 -0600 Subject: [petsc-users] SuperLU convergence problem (More test) In-Reply-To: <5665DDDA.405@eos.ubc.ca> References: <565F70E8.6030300@eos.ubc.ca> <5660A9CF.1030809@eos.ubc.ca> <56637D26.2080907@eos.ubc.ca> <5665DDDA.405@eos.ubc.ca> Message-ID: Danyang: Add 'call MatSetFromOptions(A,ierr)' to your code. Attached below is ex52f.F modified from your ex52f.F to be compatible with petsc-dev. Hong Hello Hong, > > Thanks for the quick reply and the option "-mat_superlu_dist_fact > SamePattern" works like a charm, if I use this option from the command > line. > > How can I add this option as the default. I tried using > PetscOptionsInsertString("-mat_superlu_dist_fact SamePattern",ierr) in my > code but this does not work. > > Thanks, > > Danyang > > > On 15-12-07 10:42 AM, Hong wrote: > > Danyang : > > Adding '-mat_superlu_dist_fact SamePattern' fixed the problem. Below is > how I figured it out. > > 1. Reading ex52f.F, I see '-superlu_default' = > '-pc_factor_mat_solver_package superlu_dist', the later enables runtime > options for other packages. I use superlu_dist-4.2 and superlu-4.1 for the > tests below. > > 2. Use the Matrix 168 to setup KSP solver and factorization, all packages, > petsc, superlu_dist and mumps give same correct results: > > ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs > matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check > -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package > petsc > -->loac matrix a > -->load rhs b > size l,m,n,mm 90000 90000 90000 90000 > Norm of error 7.7308E-11 iterations 1 > -->Test for matrix 168 > .. > -->Test for matrix 172 > Norm of error 3.8461E-11 iterations 1 > > ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs > matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check > -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package > superlu_dist > Norm of error 9.4073E-11 iterations 1 > -->Test for matrix 168 > ... > -->Test for matrix 172 > Norm of error 3.8187E-11 iterations 1 > > 3. Use superlu, I get > ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs > matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check > -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package > superlu > Norm of error 1.0191E-06 iterations 1 > -->Test for matrix 168 > ... > -->Test for matrix 172 > Norm of error 9.7858E-07 iterations 1 > > Replacing default DiagPivotThresh: 1. to 0.0, I get same solutions as > other packages: > > ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs > matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check > -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package > superlu -mat_superlu_diagpivotthresh 0.0 > > Norm of error 8.3614E-11 iterations 1 > -->Test for matrix 168 > ... > -->Test for matrix 172 > Norm of error 3.7098E-11 iterations 1 > > 4. > using '-mat_view ascii::ascii_info', I found that a_flow_check_1.bin and > a_flow_check_168.bin seem have same structure: > > -->loac matrix a > Mat Object: 1 MPI processes > type: seqaij > rows=90000, cols=90000 > total: nonzeros=895600, allocated nonzeros=895600 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 45000 nodes, limit used is 5 > > 5. > Using a_flow_check_1.bin, I am able to reproduce the error you reported: > all packages give correct results except superlu_dist: > ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_1.bin -rhs > matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check > -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package > superlu_dist > Norm of error 2.5970E-12 iterations 1 > -->Test for matrix 168 > Norm of error 1.3936E-01 iterations 34 > -->Test for matrix 169 > > I guess the error might come from reuse of matrix factor. Replacing default > -mat_superlu_dist_fact with > -mat_superlu_dist_fact SamePattern, I get > > ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_1.bin -rhs > matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check > -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package > superlu_dist -mat_superlu_dist_fact SamePattern > > Norm of error 2.5970E-12 iterations 1 > -->Test for matrix 168 > Norm of error 9.4073E-11 iterations 1 > -->Test for matrix 169 > Norm of error 6.4303E-11 iterations 1 > -->Test for matrix 170 > Norm of error 7.4327E-11 iterations 1 > -->Test for matrix 171 > Norm of error 5.4162E-11 iterations 1 > -->Test for matrix 172 > Norm of error 3.4440E-11 iterations 1 > --> End of test, bye > > Sherry may tell you why SamePattern_SameRowPerm cause the difference here. > Best on the above experiments, I would set following as default > '-mat_superlu_diagpivotthresh 0.0' in petsc/superlu interface. > '-mat_superlu_dist_fact SamePattern' in petsc/superlu_dist interface. > > Hong > > Hi Hong, >> >> I did more test today and finally found that the solution accuracy >> depends on the initial (first) matrix quality. I modified the ex52f.F to do >> the test. There are 6 matrices and right-hand-side vectors. All these >> matrices and rhs are from my reactive transport simulation. Results will be >> quite different depending on which one you use to do factorization. Results >> will also be different if you run with different options. My code is >> similar to the First or the Second test below. When the matrix is well >> conditioned, it works fine. But if the initial matrix is well conditioned, >> it likely to crash when the matrix become ill-conditioned. Since most of my >> case are well conditioned so I didn't detect the problem before. This case >> is a special one. >> >> >> How can I avoid this problem? Shall I redo factorization? Can PETSc >> automatically detect this prolbem or is there any option available to do >> this? >> >> All the data and test code (modified ex52f) can be found via the dropbox >> link below. >> >> *https://www.dropbox.com/s/4al1a60creogd8m/petsc-superlu-test.tar.gz?dl=0 >> * >> >> >> Summary of my test is shown below. >> >> First, use the Matrix 1 to setup KSP solver and factorization, then solve >> 168 to 172 >> >> mpiexec.hydra -n 1 ./ex52f -f0 >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin >> -rhs >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin >> -loop_matrices flow_check -loop_folder >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -pc_type lu >> -pc_factor_mat_solver_package superlu_dist >> >> Norm of error 3.8815E-11 iterations 1 >> -->Test for matrix 168 >> Norm of error 4.2307E-01 iterations 32 >> -->Test for matrix 169 >> Norm of error 3.0528E-01 iterations 32 >> -->Test for matrix 170 >> Norm of error 3.1177E-01 iterations 32 >> -->Test for matrix 171 >> Norm of error 3.2793E-01 iterations 32 >> -->Test for matrix 172 >> Norm of error 3.1251E-01 iterations 31 >> >> Second, use the Matrix 1 to setup KSP solver and factorization using the >> implemented SuperLU relative codes. I thought this will generate the same >> results as the First test, but it actually not. >> >> mpiexec.hydra -n 1 ./ex52f -f0 >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin >> -rhs >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin >> -loop_matrices flow_check -loop_folder >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -superlu_default >> >> Norm of error 2.2632E-12 iterations 1 >> -->Test for matrix 168 >> Norm of error 1.0817E+04 iterations 1 >> -->Test for matrix 169 >> Norm of error 1.0786E+04 iterations 1 >> -->Test for matrix 170 >> Norm of error 1.0792E+04 iterations 1 >> -->Test for matrix 171 >> Norm of error 1.0792E+04 iterations 1 >> -->Test for matrix 172 >> Norm of error 1.0792E+04 iterations 1 >> >> >> Third, use the Matrix 168 to setup KSP solver and factorization, then >> solve 168 to 172 >> >> mpiexec.hydra -n 1 ./ex52f -f0 >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin >> -rhs >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin >> -loop_matrices flow_check -loop_folder >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -pc_type lu >> -pc_factor_mat_solver_package superlu_dist >> >> Norm of error 9.5528E-10 iterations 1 >> -->Test for matrix 168 >> Norm of error 9.4945E-10 iterations 1 >> -->Test for matrix 169 >> Norm of error 6.4279E-10 iterations 1 >> -->Test for matrix 170 >> Norm of error 7.4633E-10 iterations 1 >> -->Test for matrix 171 >> Norm of error 7.4863E-10 iterations 1 >> -->Test for matrix 172 >> Norm of error 8.9701E-10 iterations 1 >> >> Fourth, use the Matrix 168 to setup KSP solver and factorization using >> the implemented SuperLU relative codes. I thought this will generate the >> same results as the Third test, but it actually not. >> >> mpiexec.hydra -n 1 ./ex52f -f0 >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin >> -rhs >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin >> -loop_matrices flow_check -loop_folder >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -superlu_default >> >> Norm of error 3.7017E-11 iterations 1 >> -->Test for matrix 168 >> Norm of error 3.6420E-11 iterations 1 >> -->Test for matrix 169 >> Norm of error 3.7184E-11 iterations 1 >> -->Test for matrix 170 >> Norm of error 3.6847E-11 iterations 1 >> -->Test for matrix 171 >> Norm of error 3.7883E-11 iterations 1 >> -->Test for matrix 172 >> Norm of error 3.8805E-11 iterations 1 >> >> Thanks very much, >> >> Danyang >> >> On 15-12-03 01:59 PM, Hong wrote: >> >> Danyang : >> Further testing a_flow_check_168.bin, >> ./ex10 -f0 /Users/Hong/Downloads/matrix_and_rhs_bin/a_flow_check_168.bin >> -rhs /Users/Hong/Downloads/matrix_and_rhs_bin/x_flow_check_168.bin -pc_type >> lu -pc_factor_mat_solver_package superlu -ksp_monitor_true_residual >> -mat_superlu_conditionnumber >> Recip. condition number = 1.610480e-12 >> 0 KSP preconditioned resid norm 6.873340313547e+09 true resid norm >> 7.295020990196e+03 ||r(i)||/||b|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.051833296449e-02 true resid norm >> 2.976859070118e-02 ||r(i)||/||b|| 4.080672384793e-06 >> Number of iterations = 1 >> Residual norm 0.0297686 >> >> condition number of this matrix = 1/1.610480e-12 = 1.e+12, >> i.e., this matrix is ill-conditioned. >> >> Hong >> >> >> Hi Hong, >>> >>> The binary format of matrix, rhs and solution can be downloaded via the >>> link below. >>> >>> https://www.dropbox.com/s/cl3gfi0s0kjlktf/matrix_and_rhs_bin.tar.gz?dl=0 >>> >>> Thanks, >>> >>> Danyang >>> >>> >>> On 15-12-03 10:50 AM, Hong wrote: >>> >>> Danyang: >>> >>>> >>>> >>>> To my surprising, solutions from SuperLU at timestep 29 seems not >>>> correct for the first 4 Newton iterations, but the solutions from iteration >>>> solver and MUMPS are correct. >>>> >>>> Please find all the matrices, rhs and solutions at timestep 29 via the >>>> link below. The data is a bit large so that I just share it through >>>> Dropbox. A piece of matlab code to read these data and then computer the >>>> norm has also been attached. >>>> *https://www.dropbox.com/s/rr8ueysgflmxs7h/results-check.tar.gz?dl=0 >>>> * >>>> >>> >>> Can you send us matrix in petsc binary format? >>> >>> e.g., call MatView(M, PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD)) >>> or '-ksp_view_mat binary' >>> >>> Hong >>> >>>> >>>> >>>> Below is a summary of the norm from the three solvers at timestep 29, >>>> newton iteration 1 to 5. >>>> >>>> Timestep 29 >>>> Norm of residual seq 1.661321e-09, superlu 1.657103e+04, mumps >>>> 3.731225e-11 >>>> Norm of residual seq 1.753079e-09, superlu 6.675467e+02, mumps >>>> 1.509919e-13 >>>> Norm of residual seq 4.914971e-10, superlu 1.236362e-01, mumps >>>> 2.139303e-17 >>>> Norm of residual seq 3.532769e-10, superlu 1.304670e-04, mumps >>>> 5.387000e-20 >>>> Norm of residual seq 3.885629e-10, superlu 2.754876e-07, mumps >>>> 4.108675e-21 >>>> >>>> Would anybody please check if SuperLU can solve these matrices? Another >>>> possibility is that something is wrong in my own code. But so far, I cannot >>>> find any problem in my code since the same code works fine if I using >>>> iterative solver or direct solver MUMPS. But for other cases I have >>>> tested, all these solvers work fine. >>>> >>>> Please let me know if I did not write down the problem clearly. >>>> >>>> Thanks, >>>> >>>> Danyang >>>> >>>> >>>> >>>> >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex52f.F Type: application/octet-stream Size: 10333 bytes Desc: not available URL: From dsu at eos.ubc.ca Mon Dec 7 14:14:26 2015 From: dsu at eos.ubc.ca (Danyang Su) Date: Mon, 7 Dec 2015 12:14:26 -0800 Subject: [petsc-users] SuperLU convergence problem (More test) In-Reply-To: References: <565F70E8.6030300@eos.ubc.ca> <5660A9CF.1030809@eos.ubc.ca> <56637D26.2080907@eos.ubc.ca> <5665DDDA.405@eos.ubc.ca> Message-ID: <5665E8A2.8060803@eos.ubc.ca> Thank. The inserted options works now. I didn't put PetscOptionsInsertString in the right place before. Danyang On 15-12-07 12:01 PM, Hong wrote: > Danyang: > Add 'call MatSetFromOptions(A,ierr)' to your code. > Attached below is ex52f.F modified from your ex52f.F to be compatible > with petsc-dev. > > Hong > > Hello Hong, > > Thanks for the quick reply and the option "-mat_superlu_dist_fact > SamePattern" works like a charm, if I use this option from the > command line. > > How can I add this option as the default. I tried using > PetscOptionsInsertString("-mat_superlu_dist_fact > SamePattern",ierr) in my code but this does not work. > > Thanks, > > Danyang > > > On 15-12-07 10:42 AM, Hong wrote: >> Danyang : >> >> Adding '-mat_superlu_dist_fact SamePattern' fixed the problem. >> Below is how I figured it out. >> >> 1. Reading ex52f.F, I see '-superlu_default' = >> '-pc_factor_mat_solver_package superlu_dist', the later enables >> runtime options for other packages. I use superlu_dist-4.2 and >> superlu-4.1 for the tests below. >> >> 2. Use the Matrix 168 to setup KSP solver and factorization, all >> packages, petsc, superlu_dist and mumps give same correct results: >> >> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs >> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check >> -loop_folder matrix_and_rhs_bin -pc_type lu >> -pc_factor_mat_solver_package petsc >> -->loac matrix a >> -->load rhs b >> size l,m,n,mm 90000 90000 90000 90000 >> Norm of error 7.7308E-11 iterations 1 >> -->Test for matrix 168 >> .. >> -->Test for matrix 172 >> Norm of error 3.8461E-11 iterations 1 >> >> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs >> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check >> -loop_folder matrix_and_rhs_bin -pc_type lu >> -pc_factor_mat_solver_package superlu_dist >> Norm of error 9.4073E-11 iterations 1 >> -->Test for matrix 168 >> ... >> -->Test for matrix 172 >> Norm of error 3.8187E-11 iterations 1 >> >> 3. Use superlu, I get >> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs >> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check >> -loop_folder matrix_and_rhs_bin -pc_type lu >> -pc_factor_mat_solver_package superlu >> Norm of error 1.0191E-06 iterations 1 >> -->Test for matrix 168 >> ... >> -->Test for matrix 172 >> Norm of error 9.7858E-07 iterations 1 >> >> Replacing default DiagPivotThresh: 1. to 0.0, I get same >> solutions as other packages: >> >> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs >> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check >> -loop_folder matrix_and_rhs_bin -pc_type lu >> -pc_factor_mat_solver_package superlu >> -mat_superlu_diagpivotthresh 0.0 >> >> Norm of error 8.3614E-11 iterations 1 >> -->Test for matrix 168 >> ... >> -->Test for matrix 172 >> Norm of error 3.7098E-11 iterations 1 >> >> 4. >> using '-mat_view ascii::ascii_info', I found that >> a_flow_check_1.bin and a_flow_check_168.bin seem have same structure: >> >> -->loac matrix a >> Mat Object: 1 MPI processes >> type: seqaij >> rows=90000, cols=90000 >> total: nonzeros=895600, allocated nonzeros=895600 >> total number of mallocs used during MatSetValues calls =0 >> using I-node routines: found 45000 nodes, limit used is 5 >> >> 5. >> Using a_flow_check_1.bin, I am able to reproduce the error you >> reported: all packages give correct results except superlu_dist: >> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_1.bin -rhs >> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check >> -loop_folder matrix_and_rhs_bin -pc_type lu >> -pc_factor_mat_solver_package superlu_dist >> Norm of error 2.5970E-12 iterations 1 >> -->Test for matrix 168 >> Norm of error 1.3936E-01 iterations 34 >> -->Test for matrix 169 >> >> I guess the error might come from reuse of matrix factor. >> Replacing default >> -mat_superlu_dist_fact with >> -mat_superlu_dist_fact SamePattern, I get >> >> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_1.bin -rhs >> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check >> -loop_folder matrix_and_rhs_bin -pc_type lu >> -pc_factor_mat_solver_package superlu_dist -mat_superlu_dist_fact >> SamePattern >> >> Norm of error 2.5970E-12 iterations 1 >> -->Test for matrix 168 >> Norm of error 9.4073E-11 iterations 1 >> -->Test for matrix 169 >> Norm of error 6.4303E-11 iterations 1 >> -->Test for matrix 170 >> Norm of error 7.4327E-11 iterations 1 >> -->Test for matrix 171 >> Norm of error 5.4162E-11 iterations 1 >> -->Test for matrix 172 >> Norm of error 3.4440E-11 iterations 1 >> --> End of test, bye >> >> Sherry may tell you why SamePattern_SameRowPerm cause the >> difference here. >> Best on the above experiments, I would set following as default >> '-mat_superlu_diagpivotthresh 0.0' in petsc/superlu interface. >> '-mat_superlu_dist_fact SamePattern' in petsc/superlu_dist interface. >> >> Hong >> >> Hi Hong, >> >> I did more test today and finally found that the solution >> accuracy depends on the initial (first) matrix quality. I >> modified the ex52f.F to do the test. There are 6 matrices and >> right-hand-side vectors. All these matrices and rhs are from >> my reactive transport simulation. Results will be quite >> different depending on which one you use to do factorization. >> Results will also be different if you run with different >> options. My code is similar to the First or the Second test >> below. When the matrix is well conditioned, it works fine. >> But if the initial matrix is well conditioned, it likely to >> crash when the matrix become ill-conditioned. Since most of >> my case are well conditioned so I didn't detect the problem >> before. This case is a special one. >> >> >> How can I avoid this problem? Shall I redo factorization? Can >> PETSc automatically detect this prolbem or is there any >> option available to do this? >> >> All the data and test code (modified ex52f) can be found via >> the dropbox link below. >> _ >> __https://www.dropbox.com/s/4al1a60creogd8m/petsc-superlu-test.tar.gz?dl=0_ >> >> >> Summary of my test is shown below. >> >> First, use the Matrix 1 to setup KSP solver and >> factorization, then solve 168 to 172 >> >> mpiexec.hydra -n 1 ./ex52f -f0 >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin >> -rhs >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin >> -loop_matrices flow_check -loop_folder >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -pc_type >> lu -pc_factor_mat_solver_package superlu_dist >> >> Norm of error 3.8815E-11 iterations 1 >> -->Test for matrix 168 >> Norm of error 4.2307E-01 iterations 32 >> -->Test for matrix 169 >> Norm of error 3.0528E-01 iterations 32 >> -->Test for matrix 170 >> Norm of error 3.1177E-01 iterations 32 >> -->Test for matrix 171 >> Norm of error 3.2793E-01 iterations 32 >> -->Test for matrix 172 >> Norm of error 3.1251E-01 iterations 31 >> >> Second, use the Matrix 1 to setup KSP solver and >> factorization using the implemented SuperLU relative codes. I >> thought this will generate the same results as the First >> test, but it actually not. >> >> mpiexec.hydra -n 1 ./ex52f -f0 >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin >> -rhs >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin >> -loop_matrices flow_check -loop_folder >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin >> -superlu_default >> >> Norm of error 2.2632E-12 iterations 1 >> -->Test for matrix 168 >> Norm of error 1.0817E+04 iterations 1 >> -->Test for matrix 169 >> Norm of error 1.0786E+04 iterations 1 >> -->Test for matrix 170 >> Norm of error 1.0792E+04 iterations 1 >> -->Test for matrix 171 >> Norm of error 1.0792E+04 iterations 1 >> -->Test for matrix 172 >> Norm of error 1.0792E+04 iterations 1 >> >> >> Third, use the Matrix 168 to setup KSP solver and >> factorization, then solve 168 to 172 >> >> mpiexec.hydra -n 1 ./ex52f -f0 >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin >> -rhs >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin >> -loop_matrices flow_check -loop_folder >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -pc_type >> lu -pc_factor_mat_solver_package superlu_dist >> >> Norm of error 9.5528E-10 iterations 1 >> -->Test for matrix 168 >> Norm of error 9.4945E-10 iterations 1 >> -->Test for matrix 169 >> Norm of error 6.4279E-10 iterations 1 >> -->Test for matrix 170 >> Norm of error 7.4633E-10 iterations 1 >> -->Test for matrix 171 >> Norm of error 7.4863E-10 iterations 1 >> -->Test for matrix 172 >> Norm of error 8.9701E-10 iterations 1 >> >> Fourth, use the Matrix 168 to setup KSP solver and >> factorization using the implemented SuperLU relative codes. I >> thought this will generate the same results as the Third >> test, but it actually not. >> >> mpiexec.hydra -n 1 ./ex52f -f0 >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin >> -rhs >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin >> -loop_matrices flow_check -loop_folder >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin >> -superlu_default >> >> Norm of error 3.7017E-11 iterations 1 >> -->Test for matrix 168 >> Norm of error 3.6420E-11 iterations 1 >> -->Test for matrix 169 >> Norm of error 3.7184E-11 iterations 1 >> -->Test for matrix 170 >> Norm of error 3.6847E-11 iterations 1 >> -->Test for matrix 171 >> Norm of error 3.7883E-11 iterations 1 >> -->Test for matrix 172 >> Norm of error 3.8805E-11 iterations 1 >> >> Thanks very much, >> >> Danyang >> >> On 15-12-03 01:59 PM, Hong wrote: >>> Danyang : >>> Further testing a_flow_check_168.bin, >>> ./ex10 -f0 >>> /Users/Hong/Downloads/matrix_and_rhs_bin/a_flow_check_168.bin -rhs >>> /Users/Hong/Downloads/matrix_and_rhs_bin/x_flow_check_168.bin -pc_type >>> lu -pc_factor_mat_solver_package superlu >>> -ksp_monitor_true_residual -mat_superlu_conditionnumber >>> Recip. condition number = 1.610480e-12 >>> 0 KSP preconditioned resid norm 6.873340313547e+09 true >>> resid norm 7.295020990196e+03 ||r(i)||/||b|| 1.000000000000e+00 >>> 1 KSP preconditioned resid norm 2.051833296449e-02 true >>> resid norm 2.976859070118e-02 ||r(i)||/||b|| 4.080672384793e-06 >>> Number of iterations = 1 >>> Residual norm 0.0297686 >>> >>> condition number of this matrix = 1/1.610480e-12 = 1.e+12, >>> i.e., this matrix is ill-conditioned. >>> >>> Hong >>> >>> >>> Hi Hong, >>> >>> The binary format of matrix, rhs and solution can be >>> downloaded via the link below. >>> >>> https://www.dropbox.com/s/cl3gfi0s0kjlktf/matrix_and_rhs_bin.tar.gz?dl=0 >>> >>> Thanks, >>> >>> Danyang >>> >>> >>> On 15-12-03 10:50 AM, Hong wrote: >>>> Danyang: >>>> >>>> >>>> >>>> To my surprising, solutions from SuperLU at >>>> timestep 29 seems not correct for the first 4 >>>> Newton iterations, but the solutions from iteration >>>> solver and MUMPS are correct. >>>> >>>> Please find all the matrices, rhs and solutions at >>>> timestep 29 via the link below. The data is a bit >>>> large so that I just share it through Dropbox. A >>>> piece of matlab code to read these data and then >>>> computer the norm has also been attached. >>>> _https://www.dropbox.com/s/rr8ueysgflmxs7h/results-check.tar.gz?dl=0_ >>>> >>>> >>>> Can you send us matrix in petsc binary format? >>>> >>>> e.g., call MatView(M, >>>> PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD)) >>>> or '-ksp_view_mat binary' >>>> >>>> Hong >>>> >>>> >>>> >>>> Below is a summary of the norm from the three >>>> solvers at timestep 29, newton iteration 1 to 5. >>>> >>>> Timestep 29 >>>> Norm of residual seq 1.661321e-09, superlu >>>> 1.657103e+04, mumps 3.731225e-11 >>>> Norm of residual seq 1.753079e-09, superlu >>>> 6.675467e+02, mumps 1.509919e-13 >>>> Norm of residual seq 4.914971e-10, superlu >>>> 1.236362e-01, mumps 2.139303e-17 >>>> Norm of residual seq 3.532769e-10, superlu >>>> 1.304670e-04, mumps 5.387000e-20 >>>> Norm of residual seq 3.885629e-10, superlu >>>> 2.754876e-07, mumps 4.108675e-21 >>>> >>>> Would anybody please check if SuperLU can solve >>>> these matrices? Another possibility is that >>>> something is wrong in my own code. But so far, I >>>> cannot find any problem in my code since the same >>>> code works fine if I using iterative solver or >>>> direct solver MUMPS. But for other cases I have >>>> tested, all these solvers work fine. >>>> >>>> Please let me know if I did not write down the >>>> problem clearly. >>>> >>>> Thanks, >>>> >>>> Danyang >>>> >>>> >>>> >>>> >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Dec 7 14:45:50 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 7 Dec 2015 14:45:50 -0600 Subject: [petsc-users] Is it still worth switching to PETSc if I can't write a Jacobian for my problem? In-Reply-To: References: Message-ID: <7A5CAB4B-2812-4AD6-B0C9-B5ECD703AD3B@mcs.anl.gov> Brian, Could send an example of your "rhs" function; not a totally trivial example Barry > On Dec 7, 2015, at 11:21 AM, Brian Merchant wrote: > > Hi all, > > I am considering using petsc4py instead of scipy.integrate.odeint (which is a wrapper for Fortran solvers) for a problem involving the solution of a system of ODEs. The problem has the potential to be stiff. Writing down its Jacobian is very hard. > > So far, I have been able to produce reasonable speed gains by writing the RHS functions in "something like C" (using either numba or Cython). I'd like to get even more performance out, hence my consideration of PETSc. > > Due to the large number of equations involved, it is already tedious to think about writing down a Jacobian. Even worse though, is that some of the functions governing a particular interaction do not have neat analytical forms (let alone whether or not their derivatives have neat analytical forms), so we might have a mess of piecewise functions needed to approximate them if we were to go about still trying to produce a Jacobian... > > All the toy examples I see of PETSc time stepping problems have Jacobians defined, so I wonder if I would even get a speed gain going from switching to it, if perhaps one of the reasons why I have a high computational cost is due to not being able to provide a Jacobian function? > > I described the sort of problem I am working with in more detail in this scicomp.stackexchange question, which is where most of this question is copied from, except it also comes with a toy version of the problem I am dealing with: http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a > > All your advice would be most helpful :) > > Kind regards,Brian > From bibrakc at gmail.com Mon Dec 7 16:27:45 2015 From: bibrakc at gmail.com (Bibrak Qamar) Date: Mon, 7 Dec 2015 17:27:45 -0500 Subject: [petsc-users] Vector Vector Multiply to get a Matrix Message-ID: I am in a situation where I have to multiply two vectors of size nx1 to get a matrix of size nxn. Is there a better way to achieve this without manually copying vectors into a matrix form? Thanks Bibrak -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Dec 7 16:35:43 2015 From: jed at jedbrown.org (Jed Brown) Date: Mon, 07 Dec 2015 15:35:43 -0700 Subject: [petsc-users] Vector Vector Multiply to get a Matrix In-Reply-To: References: Message-ID: <87si3drio0.fsf@jedbrown.org> Bibrak Qamar writes: > I am in a situation where I have to multiply two vectors of size nx1 to get > a matrix of size nxn. Well that's just silly. What do you want to do with this rank-1 matrix? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bisheshkh at gmail.com Mon Dec 7 17:35:16 2015 From: bisheshkh at gmail.com (Bishesh Khanal) Date: Tue, 8 Dec 2015 00:35:16 +0100 Subject: [petsc-users] zero pivot in LU factorization when using -fieldsplit_0_pc_type gamg in schur fieldsplit PC Message-ID: Hello, I'm revisiting my code for solving a system of eqs similar to Stokes to get better convergence for big jumps in viscosity (mu) and wanted to experiment with petsc's gamg: mu lap(u) + grad(p) = f1 div(u) = f2 Dirichlet boundary condition of zero velocity. The implementation is a finite difference with a staggered grid discretization. Using Schur FieldSplit PC type, and for constant viscosity, using hypre PC for A00 block had been serving well. Now when trying gamg, I get zero pivot error. From some of the past discussions in the archives of the mailing list, it seems I've to play around with ..._sub_pc_factor_shift_type but I could not figure out the exact set of options. Setting mu = 1 everywhere; Using the following options: -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_precondition self -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type gamg I get an error. When I use -fieldsplit_0_pc_type lu , it converges in a single iteration so I guess it should mean that A00 is not singular. Below are the results of using: 1. -fieldsplit_0_pc_type hypre (converges) 2. -ksp_view -fieldsplit_0_pc_type lu (converges) 3. -fieldsplit_0_pc_type gamg (all error output) 1. ---------------------------------------- With -fieldsplit_0_pc_type hypre ---------------------------------------- Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 7 Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 7 Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 17 Linear solve converged due to CONVERGED_RTOL iterations 2 2. -------------------------------------------- With -fieldsplit_0_pc_type lu -ksp_view -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_precondition self -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type lu -fieldsplit_1_ksp_rtol 1.0e-10 -fieldsplit_1_ksp_converged_reason -ksp_converged_reason -ksp_view Linear fieldsplit_0_ solve converged due to CONVERGED_ATOL iterations 0 Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 ... Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 22 Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 ... Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 22 Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 Linear solve converged due to CONVERGED_RTOL iterations 1 KSP Object: 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: fieldsplit FieldSplit with Schur preconditioner, blocksize = 4, factorization FULL Preconditioner for the Schur complement formed from S itself Split info: Split number 0 Fields 0, 1, 2 Split number 1 Fields 3 KSP solver for A00 block KSP Object: (fieldsplit_0_) 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 5, needed 20.772 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=54243, cols=54243, bs=3 package used to perform factorization: petsc total: nonzeros=8.41718e+07, allocated nonzeros=8.41718e+07 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 17801 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaij rows=54243, cols=54243, bs=3 total: nonzeros=4.05217e+06, allocated nonzeros=4.05217e+06 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 18081 nodes, limit used is 5 KSP solver for S = A11 - A10 inv(A00) A01 KSP Object: (fieldsplit_1_) 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-10, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: (fieldsplit_1_) 1 MPI processes type: none linear system matrix = precond matrix: Mat Object: (fieldsplit_1_) 1 MPI processes type: schurcomplement rows=18081, cols=18081 has attached null space Schur complement A11 - A10 inv(A00) A01 A11 Mat Object: (fieldsplit_1_) 1 MPI processes type: seqaij rows=18081, cols=18081 total: nonzeros=450241, allocated nonzeros=450241 total number of mallocs used during MatSetValues calls =0 not using I-node routines A10 Mat Object: 1 MPI processes type: seqaij rows=18081, cols=54243 total: nonzeros=1.35072e+06, allocated nonzeros=1.35072e+06 total number of mallocs used during MatSetValues calls =0 not using I-node routines KSP of A00 KSP Object: (fieldsplit_0_) 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 5, needed 20.772 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=54243, cols=54243, bs=3 package used to perform factorization: petsc total: nonzeros=8.41718e+07, allocated nonzeros=8.41718e+07 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 17801 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaij rows=54243, cols=54243, bs=3 total: nonzeros=4.05217e+06, allocated nonzeros=4.05217e+06 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 18081 nodes, limit used is 5 A01 Mat Object: 1 MPI processes type: seqaij rows=54243, cols=18081, rbs=3, cbs = 1 total: nonzeros=1.35072e+06, allocated nonzeros=1.35072e+06 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 18081 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=72324, cols=72324, bs=4 total: nonzeros=7.20386e+06, allocated nonzeros=7.20386e+06 total number of mallocs used during MatSetValues calls =0 has attached null space using I-node routines: found 18081 nodes, limit used is 5 3. ---------------------------------------- when using gamg: -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_precondition self -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type gamg -fieldsplit_0_ksp_converged_reason -fieldsplit_1_ksp_converged_reason -ksp_converged_reason -ksp_view ---------------------------------------- [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot [0]PETSC ERROR: Zero pivot, row 3 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC ERROR: /user/bkhanal/home/works/AdLemModel/build/src/AdLemMain on a arch-linux2-cxx-debug named delorme by bkhanal Mon Dec 7 23:13:00 2015 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-clanguage=cxx --download-fblaslapack --download-scalapack --download-mpich --download-superlu_dist --download-mumps --download-hypre --download-metis --download-parmetis [0]PETSC ERROR: #1 PetscKernel_A_gets_inverse_A_5() line 48 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/baij/seq/dgefa5.c [0]PETSC ERROR: #2 MatSOR_SeqAIJ_Inode() line 2808 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [0]PETSC ERROR: #3 MatSOR() line 3697 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c [0]PETSC ERROR: #4 PCApply_SOR() line 37 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/sor/sor.c [0]PETSC ERROR: #5 PCApply() line 483 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #6 KSP_PCApply() line 242 in /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h [0]PETSC ERROR: #7 KSPInitialResidual() line 63 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c [0]PETSC ERROR: #8 KSPSolve_GMRES() line 235 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: #9 KSPSolve() line 604 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #10 KSPSolve_Chebyshev() line 381 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/cheby/cheby.c [0]PETSC ERROR: #11 KSPSolve() line 604 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #12 PCMGMCycle_Private() line 19 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: #13 PCMGMCycle_Private() line 48 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: #14 PCApply_MG() line 340 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: #15 PCApply() line 483 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #16 KSP_PCApply() line 242 in /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h [0]PETSC ERROR: #17 KSPInitialResidual() line 63 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c [0]PETSC ERROR: #18 KSPSolve_GMRES() line 235 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: #19 KSPSolve() line 604 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #20 MatMult_SchurComplement() line 105 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/utils/schurm.c [0]PETSC ERROR: #21 MatNullSpaceTest() line 431 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matnull.c WARNING: not a valid pressure null space [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Vec is locked read only, argument # 1 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC ERROR: /user/bkhanal/home/works/AdLemModel/build/src/AdLemMain on a arch-linux2-cxx-debug named delorme by bkhanal Mon Dec 7 23:13:00 2015 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-clanguage=cxx --download-fblaslapack --download-scalapack --download-mpich --download-superlu_dist --download-mumps --download-hypre --download-metis --download-parmetis [0]PETSC ERROR: #22 VecSet() line 573 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/vec/vec/interface/rvector.c [0]PETSC ERROR: #23 MatMultTranspose_SeqAIJ() line 1263 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: #24 MatMultTranspose() line 2282 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c [0]PETSC ERROR: #25 MatRestrict() line 7829 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c [0]PETSC ERROR: #26 PCMGMCycle_Private() line 44 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: #27 PCApply_MG() line 340 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: #28 PCApply() line 483 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #29 KSP_PCApply() line 242 in /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h [0]PETSC ERROR: #30 KSPInitialResidual() line 63 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c [0]PETSC ERROR: #31 KSPSolve_GMRES() line 235 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: #32 KSPSolve() line 604 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #33 PCApply_FieldSplit_Schur() line 898 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #34 PCApply() line 483 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #35 KSP_PCApply() line 242 in /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h [0]PETSC ERROR: #36 KSPInitialResidual() line 63 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c [0]PETSC ERROR: #37 KSPSolve_GMRES() line 235 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: #38 KSPSolve() line 604 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #39 solveModel() line 313 in /home/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] getSolVelocityAt line 137 /home/bkhanal/works/AdLemModel/src/PetscAdLem3D.cxx [0]PETSC ERROR: [0] VecSet line 568 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/vec/vec/interface/rvector.c [0]PETSC ERROR: [0] MatMultTranspose_SeqAIJ line 1262 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: [0] MatMultTranspose line 2263 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c [0]PETSC ERROR: [0] MatRestrict line 7817 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c [0]PETSC ERROR: [0] PCMGMCycle_Private line 17 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: [0] PCApply_MG line 325 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: [0] PCApply line 463 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c [0]PETSC ERROR: [0] KSP_PCApply line 240 /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h [0]PETSC ERROR: [0] KSPInitialResidual line 44 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c [0]PETSC ERROR: [0] KSPSolve_GMRES line 225 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: [0] KSPSolve line 510 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] PCApply_FieldSplit_Schur line 848 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: [0] PCApply line 463 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c [0]PETSC ERROR: [0] KSP_PCApply line 240 /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h [0]PETSC ERROR: [0] KSPInitialResidual line 44 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c [0]PETSC ERROR: [0] KSPSolve_GMRES line 225 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: [0] KSPSolve line 510 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] PetscKernel_A_gets_inverse_A_5 line 25 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/baij/seq/dgefa5.c [0]PETSC ERROR: [0] MatSOR_SeqAIJ_Inode line 2764 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [0]PETSC ERROR: [0] MatSOR line 3678 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c [0]PETSC ERROR: [0] PCApply_SOR line 35 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/sor/sor.c [0]PETSC ERROR: [0] PCApply line 463 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c [0]PETSC ERROR: [0] KSP_PCApply line 240 /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h [0]PETSC ERROR: [0] KSPInitialResidual line 44 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c [0]PETSC ERROR: [0] KSPSolve_GMRES line 225 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: [0] KSPSolve line 510 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] KSPSolve_Chebyshev line 358 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/cheby/cheby.c [0]PETSC ERROR: [0] KSPSolve line 510 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] PCMGMCycle_Private line 17 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: [0] PCMGMCycle_Private line 17 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: [0] PCApply_MG line 325 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: [0] PCApply line 463 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c [0]PETSC ERROR: [0] KSP_PCApply line 240 /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h [0]PETSC ERROR: [0] KSPInitialResidual line 44 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c [0]PETSC ERROR: [0] KSPSolve_GMRES line 225 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC ERROR: /user/bkhanal/home/works/AdLemModel/build/src/AdLemMain on a arch-linux2-cxx-debug named delorme by bkhanal Mon Dec 7 23:13:00 2015 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-clanguage=cxx --download-fblaslapack --download-scalapack --download-mpich --download-superlu_dist --download-mumps --download-hypre --download-metis --download-parmetis [0]PETSC ERROR: #40 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 [unset]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Dec 7 17:57:20 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 7 Dec 2015 17:57:20 -0600 Subject: [petsc-users] zero pivot in LU factorization when using -fieldsplit_0_pc_type gamg in schur fieldsplit PC In-Reply-To: References: Message-ID: It is getting a zero pivot when factoring the little dense diagonal blocks of the smoother matrix (to do SOR). What happens if you run with the additional option -mat_no_inode? Run with the additional option -start_in_debugger noxterm In the debugger type cont when it crashes type bt then use up as many times as needed to get to the routine MatSOR() then type call MatView(mat,0) you should see that diagonal subblock from the 3rd row to 7th row is singular. Normally we would expect this from the operator mu lap(u) unless the mu has zero entries? What happens if you run -fieldsplit_0_pc_type sor ? Does it also give a zero factorization? > On Dec 7, 2015, at 5:35 PM, Bishesh Khanal wrote: > > Hello, > I'm revisiting my code for solving a system of eqs similar to Stokes to get better convergence for big jumps in viscosity (mu) and wanted to experiment with petsc's gamg: > mu lap(u) + grad(p) = f1 > div(u) = f2 > Dirichlet boundary condition of zero velocity. > The implementation is a finite difference with a staggered grid discretization. > > Using Schur FieldSplit PC type, and for constant viscosity, using hypre PC for A00 block had been serving well. Now when trying gamg, I get zero pivot error. From some of the past discussions in the archives of the mailing list, it seems I've to play around with ..._sub_pc_factor_shift_type but I could not figure out the exact set of options. > > Setting mu = 1 everywhere; Using the following options: > -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_precondition self -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type gamg > I get an error. > When I use -fieldsplit_0_pc_type lu , it converges in a single iteration so I guess it should mean that A00 is not singular. > Below are the results of using: > 1. -fieldsplit_0_pc_type hypre (converges) > 2. -ksp_view -fieldsplit_0_pc_type lu (converges) > 3. -fieldsplit_0_pc_type gamg (all error output) > > > 1. ---------------------------------------- > With -fieldsplit_0_pc_type hypre > ---------------------------------------- > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 7 > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 7 > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 17 > Linear solve converged due to CONVERGED_RTOL iterations 2 > > 2. -------------------------------------------- > With -fieldsplit_0_pc_type lu -ksp_view > -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_precondition self -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type lu -fieldsplit_1_ksp_rtol 1.0e-10 -fieldsplit_1_ksp_converged_reason -ksp_converged_reason -ksp_view > > Linear fieldsplit_0_ solve converged due to CONVERGED_ATOL iterations 0 > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > ... > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 22 > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > ... > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 22 > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > Linear solve converged due to CONVERGED_RTOL iterations 1 > > KSP Object: 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: fieldsplit > FieldSplit with Schur preconditioner, blocksize = 4, factorization FULL > Preconditioner for the Schur complement formed from S itself > Split info: > Split number 0 Fields 0, 1, 2 > Split number 1 Fields 3 > KSP solver for A00 block > KSP Object: (fieldsplit_0_) 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: (fieldsplit_0_) 1 MPI processes > type: lu > LU: out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5, needed 20.772 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=54243, cols=54243, bs=3 > package used to perform factorization: petsc > total: nonzeros=8.41718e+07, allocated nonzeros=8.41718e+07 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 17801 nodes, limit used is 5 > linear system matrix = precond matrix: > Mat Object: (fieldsplit_0_) 1 MPI processes > type: seqaij > rows=54243, cols=54243, bs=3 > total: nonzeros=4.05217e+06, allocated nonzeros=4.05217e+06 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 18081 nodes, limit used is 5 > KSP solver for S = A11 - A10 inv(A00) A01 > KSP Object: (fieldsplit_1_) 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-10, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: (fieldsplit_1_) 1 MPI processes > type: none > linear system matrix = precond matrix: > Mat Object: (fieldsplit_1_) 1 MPI processes > type: schurcomplement > rows=18081, cols=18081 > has attached null space > Schur complement A11 - A10 inv(A00) A01 > A11 > Mat Object: (fieldsplit_1_) 1 MPI processes > type: seqaij > rows=18081, cols=18081 > total: nonzeros=450241, allocated nonzeros=450241 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > A10 > Mat Object: 1 MPI processes > type: seqaij > rows=18081, cols=54243 > total: nonzeros=1.35072e+06, allocated nonzeros=1.35072e+06 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > KSP of A00 > KSP Object: (fieldsplit_0_) 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: (fieldsplit_0_) 1 MPI processes > type: lu > LU: out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5, needed 20.772 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=54243, cols=54243, bs=3 > package used to perform factorization: petsc > total: nonzeros=8.41718e+07, allocated nonzeros=8.41718e+07 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 17801 nodes, limit used is 5 > linear system matrix = precond matrix: > Mat Object: (fieldsplit_0_) 1 MPI processes > type: seqaij > rows=54243, cols=54243, bs=3 > total: nonzeros=4.05217e+06, allocated nonzeros=4.05217e+06 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 18081 nodes, limit used is 5 > A01 > Mat Object: 1 MPI processes > type: seqaij > rows=54243, cols=18081, rbs=3, cbs = 1 > total: nonzeros=1.35072e+06, allocated nonzeros=1.35072e+06 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 18081 nodes, limit used is 5 > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=72324, cols=72324, bs=4 > total: nonzeros=7.20386e+06, allocated nonzeros=7.20386e+06 > total number of mallocs used during MatSetValues calls =0 > has attached null space > using I-node routines: found 18081 nodes, limit used is 5 > > > 3. ---------------------------------------- > when using gamg: -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_precondition self -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type gamg -fieldsplit_0_ksp_converged_reason -fieldsplit_1_ksp_converged_reason -ksp_converged_reason -ksp_view > ---------------------------------------- > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot > [0]PETSC ERROR: Zero pivot, row 3 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR: /user/bkhanal/home/works/AdLemModel/build/src/AdLemMain on a arch-linux2-cxx-debug named delorme by bkhanal Mon Dec 7 23:13:00 2015 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-clanguage=cxx --download-fblaslapack --download-scalapack --download-mpich --download-superlu_dist --download-mumps --download-hypre --download-metis --download-parmetis > [0]PETSC ERROR: #1 PetscKernel_A_gets_inverse_A_5() line 48 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/baij/seq/dgefa5.c > [0]PETSC ERROR: #2 MatSOR_SeqAIJ_Inode() line 2808 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/aij/seq/inode.c > [0]PETSC ERROR: #3 MatSOR() line 3697 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c > [0]PETSC ERROR: #4 PCApply_SOR() line 37 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/sor/sor.c > [0]PETSC ERROR: #5 PCApply() line 483 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #6 KSP_PCApply() line 242 in /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > [0]PETSC ERROR: #7 KSPInitialResidual() line 63 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: #8 KSPSolve_GMRES() line 235 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #9 KSPSolve() line 604 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #10 KSPSolve_Chebyshev() line 381 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/cheby/cheby.c > [0]PETSC ERROR: #11 KSPSolve() line 604 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #12 PCMGMCycle_Private() line 19 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: #13 PCMGMCycle_Private() line 48 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: #14 PCApply_MG() line 340 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: #15 PCApply() line 483 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #16 KSP_PCApply() line 242 in /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > [0]PETSC ERROR: #17 KSPInitialResidual() line 63 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: #18 KSPSolve_GMRES() line 235 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #19 KSPSolve() line 604 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #20 MatMult_SchurComplement() line 105 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/utils/schurm.c > [0]PETSC ERROR: #21 MatNullSpaceTest() line 431 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matnull.c > > WARNING: not a valid pressure null space > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Vec is locked read only, argument # 1 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR: /user/bkhanal/home/works/AdLemModel/build/src/AdLemMain on a arch-linux2-cxx-debug named delorme by bkhanal Mon Dec 7 23:13:00 2015 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-clanguage=cxx --download-fblaslapack --download-scalapack --download-mpich --download-superlu_dist --download-mumps --download-hypre --download-metis --download-parmetis > [0]PETSC ERROR: #22 VecSet() line 573 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/vec/vec/interface/rvector.c > [0]PETSC ERROR: #23 MatMultTranspose_SeqAIJ() line 1263 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/aij/seq/aij.c > [0]PETSC ERROR: #24 MatMultTranspose() line 2282 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c > [0]PETSC ERROR: #25 MatRestrict() line 7829 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c > [0]PETSC ERROR: #26 PCMGMCycle_Private() line 44 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: #27 PCApply_MG() line 340 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: #28 PCApply() line 483 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #29 KSP_PCApply() line 242 in /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > [0]PETSC ERROR: #30 KSPInitialResidual() line 63 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: #31 KSPSolve_GMRES() line 235 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #32 KSPSolve() line 604 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #33 PCApply_FieldSplit_Schur() line 898 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: #34 PCApply() line 483 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #35 KSP_PCApply() line 242 in /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > [0]PETSC ERROR: #36 KSPInitialResidual() line 63 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: #37 KSPSolve_GMRES() line 235 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #38 KSPSolve() line 604 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #39 solveModel() line 313 in /home/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] getSolVelocityAt line 137 /home/bkhanal/works/AdLemModel/src/PetscAdLem3D.cxx > [0]PETSC ERROR: [0] VecSet line 568 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/vec/vec/interface/rvector.c > [0]PETSC ERROR: [0] MatMultTranspose_SeqAIJ line 1262 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/aij/seq/aij.c > [0]PETSC ERROR: [0] MatMultTranspose line 2263 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c > [0]PETSC ERROR: [0] MatRestrict line 7817 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c > [0]PETSC ERROR: [0] PCMGMCycle_Private line 17 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: [0] PCApply_MG line 325 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: [0] PCApply line 463 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: [0] KSP_PCApply line 240 /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > [0]PETSC ERROR: [0] KSPInitialResidual line 44 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: [0] KSPSolve_GMRES line 225 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: [0] KSPSolve line 510 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] PCApply_FieldSplit_Schur line 848 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: [0] PCApply line 463 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: [0] KSP_PCApply line 240 /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > [0]PETSC ERROR: [0] KSPInitialResidual line 44 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: [0] KSPSolve_GMRES line 225 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: [0] KSPSolve line 510 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] PetscKernel_A_gets_inverse_A_5 line 25 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/baij/seq/dgefa5.c > [0]PETSC ERROR: [0] MatSOR_SeqAIJ_Inode line 2764 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/aij/seq/inode.c > [0]PETSC ERROR: [0] MatSOR line 3678 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c > [0]PETSC ERROR: [0] PCApply_SOR line 35 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/sor/sor.c > [0]PETSC ERROR: [0] PCApply line 463 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: [0] KSP_PCApply line 240 /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > [0]PETSC ERROR: [0] KSPInitialResidual line 44 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: [0] KSPSolve_GMRES line 225 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: [0] KSPSolve line 510 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] KSPSolve_Chebyshev line 358 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/cheby/cheby.c > [0]PETSC ERROR: [0] KSPSolve line 510 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] PCMGMCycle_Private line 17 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: [0] PCMGMCycle_Private line 17 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: [0] PCApply_MG line 325 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: [0] PCApply line 463 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: [0] KSP_PCApply line 240 /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > [0]PETSC ERROR: [0] KSPInitialResidual line 44 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: [0] KSPSolve_GMRES line 225 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR: /user/bkhanal/home/works/AdLemModel/build/src/AdLemMain on a arch-linux2-cxx-debug named delorme by bkhanal Mon Dec 7 23:13:00 2015 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-clanguage=cxx --download-fblaslapack --download-scalapack --download-mpich --download-superlu_dist --download-mumps --download-hypre --download-metis --download-parmetis > [0]PETSC ERROR: #40 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > [unset]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > From bisheshkh at gmail.com Tue Dec 8 03:44:23 2015 From: bisheshkh at gmail.com (Bishesh Khanal) Date: Tue, 8 Dec 2015 10:44:23 +0100 Subject: [petsc-users] zero pivot in LU factorization when using -fieldsplit_0_pc_type gamg in schur fieldsplit PC In-Reply-To: References: Message-ID: On Tue, Dec 8, 2015 at 12:57 AM, Barry Smith wrote: > > It is getting a zero pivot when factoring the little dense diagonal > blocks of the smoother matrix (to do SOR). > > What happens if you run with the additional option -mat_no_inode? > With -mat_no_inode I see that the fieldsplit_0_solve has preconditioned norm converging but not the true resid norm. And therefore fieldsplit_1_solve not converge. Residual norms for fieldsplit_0_ solve. 0 KSP preconditioned resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| -nan Residual norms for fieldsplit_0_ solve. 0 KSP preconditioned resid norm 5.366616065799e-01 true resid norm 1.763251719324e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.051153486997e-15 true resid norm 1.763377056173e+01 ||r(i)||/||b|| 1.000071082789e+00 Residual norms for fieldsplit_1_ solve. Residual norms for fieldsplit_0_ solve. 0 KSP preconditioned resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| -nan 0 KSP preconditioned resid norm 7.379043098663e+00 true resid norm 7.379043147181e+00 ||r(i)||/||b|| 1.000000000000e+00 Residual norms for fieldsplit_0_ solve. 0 KSP preconditioned resid norm 1.013832784140e-01 true resid norm 1.194932744284e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.651850464015e-16 true resid norm 1.195413534155e+00 ||r(i)||/||b|| 1.000402357265e+00 Residual norms for fieldsplit_0_ solve. 0 KSP preconditioned resid norm 9.468489343208e-01 true resid norm 1.115983634787e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.573576948730e-15 true resid norm 1.116432658910e+01 ||r(i)||/||b|| 1.000402357265e+00 1 KSP preconditioned resid norm 7.376179991979e+00 true resid norm 7.376180019146e+00 ||r(i)||/||b|| 9.996119919645e-01 Residual norms for fieldsplit_0_ solve. 0 KSP preconditioned resid norm 4.141204698273e+00 true resid norm 1.010004734003e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.094717134282e-14 true resid norm 1.612661239170e+00 ||r(i)||/||b|| 1.596686812327e+00 Residual norms for fieldsplit_0_ solve. 0 KSP preconditioned resid norm 9.488116584954e-01 true resid norm 2.657897132073e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.486948812256e-15 true resid norm 2.658134372650e+01 ||r(i)||/||b|| 1.000089258750e+00 2 KSP preconditioned resid norm 7.376160898815e+00 true resid norm 7.376160928269e+00 ||r(i)||/||b|| 9.996094047893e-01 ....................... ....................... Residual norms for fieldsplit_0_ solve. 0 KSP preconditioned resid norm 3.198579793302e+00 true resid norm 1.508640813109e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.839267393927e-14 true resid norm 1.825155061421e+00 ||r(i)||/||b|| 1.209800931781e+00 Residual norms for fieldsplit_0_ solve. 0 KSP preconditioned resid norm 1.497257742332e+01 true resid norm 4.292799680684e+14 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.138874539002e-13 true resid norm 4.292799680684e+14 ||r(i)||/||b|| 1.000000000000e+00 43 KSP preconditioned resid norm 6.372901415746e+00 true resid norm 7.990778566794e+00 ||r(i)||/||b|| 1.082901726878e+00 .... And continues > Run with the additional option -start_in_debugger noxterm > > In the debugger type > > cont > > when it crashes type > > bt > > then use > > up > > as many times as needed to get to the routine MatSOR() then type > > call MatView(mat,0) > > you should see that diagonal subblock from the 3rd row to 7th row is > singular. Normally we would expect this from the operator > mu lap(u) unless the mu has zero entries? > Sorry I didn't understand what exactly the diagonal sublock mean here? mat is of a big size, so it was not easy to see with MatView. Looking at its output, what I see is that every now and then there are 3 or 4 contiguous rows (i, i+1, i+2) having mostly 0 values. No mu does not have zero entries, I set it to 1 everywhere except at Dirichlet boundaries where mu is not used. > What happens if you run -fieldsplit_0_pc_type sor ? Does it also give a > zero factorization? > No it does not give zero factorization, the system converges in 2 iterations. The -fieldsplit_0_ solves mostly in around 27 iterations, and for -fieldsplit_1_: Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 7 Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 7 Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 17 > > > > On Dec 7, 2015, at 5:35 PM, Bishesh Khanal wrote: > > > > Hello, > > I'm revisiting my code for solving a system of eqs similar to Stokes to > get better convergence for big jumps in viscosity (mu) and wanted to > experiment with petsc's gamg: > > mu lap(u) + grad(p) = f1 > > div(u) = f2 > > Dirichlet boundary condition of zero velocity. > > The implementation is a finite difference with a staggered grid > discretization. > > > > Using Schur FieldSplit PC type, and for constant viscosity, using hypre > PC for A00 block had been serving well. Now when trying gamg, I get zero > pivot error. From some of the past discussions in the archives of the > mailing list, it seems I've to play around with > ..._sub_pc_factor_shift_type but I could not figure out the exact set of > options. > > > > Setting mu = 1 everywhere; Using the following options: > > -pc_type fieldsplit -pc_fieldsplit_type schur > -pc_fieldsplit_schur_precondition self -pc_fieldsplit_dm_splits 0 > -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3 > -fieldsplit_0_pc_type gamg > > I get an error. > > When I use -fieldsplit_0_pc_type lu , it converges in a single iteration > so I guess it should mean that A00 is not singular. > > Below are the results of using: > > 1. -fieldsplit_0_pc_type hypre (converges) > > 2. -ksp_view -fieldsplit_0_pc_type lu (converges) > > 3. -fieldsplit_0_pc_type gamg (all error output) > > > > > > 1. ---------------------------------------- > > With -fieldsplit_0_pc_type hypre > > ---------------------------------------- > > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 7 > > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 7 > > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations > 17 > > Linear solve converged due to CONVERGED_RTOL iterations 2 > > > > 2. -------------------------------------------- > > With -fieldsplit_0_pc_type lu -ksp_view > > -pc_type fieldsplit -pc_fieldsplit_type schur > -pc_fieldsplit_schur_precondition self -pc_fieldsplit_dm_splits 0 > -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3 > -fieldsplit_0_pc_type lu -fieldsplit_1_ksp_rtol 1.0e-10 > -fieldsplit_1_ksp_converged_reason -ksp_converged_reason -ksp_view > > > > Linear fieldsplit_0_ solve converged due to CONVERGED_ATOL iterations 0 > > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > > ... > > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations > 22 > > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > > ... > > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations > 22 > > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > > > KSP Object: 1 MPI processes > > type: gmres > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > > GMRES: happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > > left preconditioning > > using PRECONDITIONED norm type for convergence test > > PC Object: 1 MPI processes > > type: fieldsplit > > FieldSplit with Schur preconditioner, blocksize = 4, factorization > FULL > > Preconditioner for the Schur complement formed from S itself > > Split info: > > Split number 0 Fields 0, 1, 2 > > Split number 1 Fields 3 > > KSP solver for A00 block > > KSP Object: (fieldsplit_0_) 1 MPI processes > > type: gmres > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > > GMRES: happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > > left preconditioning > > using PRECONDITIONED norm type for convergence test > > PC Object: (fieldsplit_0_) 1 MPI processes > > type: lu > > LU: out-of-place factorization > > tolerance for zero pivot 2.22045e-14 > > matrix ordering: nd > > factor fill ratio given 5, needed 20.772 > > Factored matrix follows: > > Mat Object: 1 MPI processes > > type: seqaij > > rows=54243, cols=54243, bs=3 > > package used to perform factorization: petsc > > total: nonzeros=8.41718e+07, allocated > nonzeros=8.41718e+07 > > total number of mallocs used during MatSetValues calls =0 > > using I-node routines: found 17801 nodes, limit used > is 5 > > linear system matrix = precond matrix: > > Mat Object: (fieldsplit_0_) 1 MPI processes > > type: seqaij > > rows=54243, cols=54243, bs=3 > > total: nonzeros=4.05217e+06, allocated nonzeros=4.05217e+06 > > total number of mallocs used during MatSetValues calls =0 > > using I-node routines: found 18081 nodes, limit used is 5 > > KSP solver for S = A11 - A10 inv(A00) A01 > > KSP Object: (fieldsplit_1_) 1 MPI processes > > type: gmres > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > > GMRES: happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-10, absolute=1e-50, divergence=10000 > > left preconditioning > > using PRECONDITIONED norm type for convergence test > > PC Object: (fieldsplit_1_) 1 MPI processes > > type: none > > linear system matrix = precond matrix: > > Mat Object: (fieldsplit_1_) 1 MPI processes > > type: schurcomplement > > rows=18081, cols=18081 > > has attached null space > > Schur complement A11 - A10 inv(A00) A01 > > A11 > > Mat Object: (fieldsplit_1_) 1 > MPI processes > > type: seqaij > > rows=18081, cols=18081 > > total: nonzeros=450241, allocated nonzeros=450241 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node routines > > A10 > > Mat Object: 1 MPI processes > > type: seqaij > > rows=18081, cols=54243 > > total: nonzeros=1.35072e+06, allocated > nonzeros=1.35072e+06 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node routines > > KSP of A00 > > KSP Object: (fieldsplit_0_) 1 > MPI processes > > type: gmres > > GMRES: restart=30, using Classical (unmodified) > Gram-Schmidt Orthogonalization with no iterative refinement > > GMRES: happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, > divergence=10000 > > left preconditioning > > using PRECONDITIONED norm type for convergence test > > PC Object: (fieldsplit_0_) 1 > MPI processes > > type: lu > > LU: out-of-place factorization > > tolerance for zero pivot 2.22045e-14 > > matrix ordering: nd > > factor fill ratio given 5, needed 20.772 > > Factored matrix follows: > > Mat Object: 1 MPI processes > > type: seqaij > > rows=54243, cols=54243, bs=3 > > package used to perform factorization: petsc > > total: nonzeros=8.41718e+07, allocated > nonzeros=8.41718e+07 > > total number of mallocs used during MatSetValues > calls =0 > > using I-node routines: found 17801 nodes, > limit used is 5 > > linear system matrix = precond matrix: > > Mat Object: (fieldsplit_0_) > 1 MPI processes > > type: seqaij > > rows=54243, cols=54243, bs=3 > > total: nonzeros=4.05217e+06, allocated > nonzeros=4.05217e+06 > > total number of mallocs used during MatSetValues calls > =0 > > using I-node routines: found 18081 nodes, limit used > is 5 > > A01 > > Mat Object: 1 MPI processes > > type: seqaij > > rows=54243, cols=18081, rbs=3, cbs = 1 > > total: nonzeros=1.35072e+06, allocated > nonzeros=1.35072e+06 > > total number of mallocs used during MatSetValues calls =0 > > using I-node routines: found 18081 nodes, limit used > is 5 > > linear system matrix = precond matrix: > > Mat Object: 1 MPI processes > > type: seqaij > > rows=72324, cols=72324, bs=4 > > total: nonzeros=7.20386e+06, allocated nonzeros=7.20386e+06 > > total number of mallocs used during MatSetValues calls =0 > > has attached null space > > using I-node routines: found 18081 nodes, limit used is 5 > > > > > > 3. ---------------------------------------- > > when using gamg: -pc_type fieldsplit -pc_fieldsplit_type schur > -pc_fieldsplit_schur_precondition self -pc_fieldsplit_dm_splits 0 > -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3 > -fieldsplit_0_pc_type gamg -fieldsplit_0_ksp_converged_reason > -fieldsplit_1_ksp_converged_reason -ksp_converged_reason -ksp_view > > ---------------------------------------- > > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Zero pivot in LU factorization: > http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot > > [0]PETSC ERROR: Zero pivot, row 3 > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > > [0]PETSC ERROR: /user/bkhanal/home/works/AdLemModel/build/src/AdLemMain > on a arch-linux2-cxx-debug named delorme by bkhanal Mon Dec 7 23:13:00 2015 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --with-clanguage=cxx --download-fblaslapack > --download-scalapack --download-mpich --download-superlu_dist > --download-mumps --download-hypre --download-metis --download-parmetis > > [0]PETSC ERROR: #1 PetscKernel_A_gets_inverse_A_5() line 48 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/baij/seq/dgefa5.c > > [0]PETSC ERROR: #2 MatSOR_SeqAIJ_Inode() line 2808 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/aij/seq/inode.c > > [0]PETSC ERROR: #3 MatSOR() line 3697 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c > > [0]PETSC ERROR: #4 PCApply_SOR() line 37 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/sor/sor.c > > [0]PETSC ERROR: #5 PCApply() line 483 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #6 KSP_PCApply() line 242 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: #7 KSPInitialResidual() line 63 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: #8 KSPSolve_GMRES() line 235 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: #9 KSPSolve() line 604 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #10 KSPSolve_Chebyshev() line 381 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/cheby/cheby.c > > [0]PETSC ERROR: #11 KSPSolve() line 604 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #12 PCMGMCycle_Private() line 19 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: #13 PCMGMCycle_Private() line 48 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: #14 PCApply_MG() line 340 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: #15 PCApply() line 483 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #16 KSP_PCApply() line 242 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: #17 KSPInitialResidual() line 63 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: #18 KSPSolve_GMRES() line 235 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: #19 KSPSolve() line 604 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #20 MatMult_SchurComplement() line 105 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/utils/schurm.c > > [0]PETSC ERROR: #21 MatNullSpaceTest() line 431 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matnull.c > > > > WARNING: not a valid pressure null space > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Object is in wrong state > > [0]PETSC ERROR: Vec is locked read only, argument # 1 > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > > [0]PETSC ERROR: /user/bkhanal/home/works/AdLemModel/build/src/AdLemMain > on a arch-linux2-cxx-debug named delorme by bkhanal Mon Dec 7 23:13:00 2015 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --with-clanguage=cxx --download-fblaslapack > --download-scalapack --download-mpich --download-superlu_dist > --download-mumps --download-hypre --download-metis --download-parmetis > > [0]PETSC ERROR: #22 VecSet() line 573 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/vec/vec/interface/rvector.c > > [0]PETSC ERROR: #23 MatMultTranspose_SeqAIJ() line 1263 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/aij/seq/aij.c > > [0]PETSC ERROR: #24 MatMultTranspose() line 2282 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c > > [0]PETSC ERROR: #25 MatRestrict() line 7829 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c > > [0]PETSC ERROR: #26 PCMGMCycle_Private() line 44 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: #27 PCApply_MG() line 340 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: #28 PCApply() line 483 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #29 KSP_PCApply() line 242 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: #30 KSPInitialResidual() line 63 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: #31 KSPSolve_GMRES() line 235 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: #32 KSPSolve() line 604 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #33 PCApply_FieldSplit_Schur() line 898 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/fieldsplit/fieldsplit.c > > [0]PETSC ERROR: #34 PCApply() line 483 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #35 KSP_PCApply() line 242 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: #36 KSPInitialResidual() line 63 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: #37 KSPSolve_GMRES() line 235 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: #38 KSPSolve() line 604 in > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #39 solveModel() line 313 in > /home/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > > [0]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac > OS X to find memory corruption errors > > [0]PETSC ERROR: likely location of problem given in stack below > > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > > [0]PETSC ERROR: INSTEAD the line number of the start of the > function > > [0]PETSC ERROR: is given. > > [0]PETSC ERROR: [0] getSolVelocityAt line 137 > /home/bkhanal/works/AdLemModel/src/PetscAdLem3D.cxx > > [0]PETSC ERROR: [0] VecSet line 568 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/vec/vec/interface/rvector.c > > [0]PETSC ERROR: [0] MatMultTranspose_SeqAIJ line 1262 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/aij/seq/aij.c > > [0]PETSC ERROR: [0] MatMultTranspose line 2263 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c > > [0]PETSC ERROR: [0] MatRestrict line 7817 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c > > [0]PETSC ERROR: [0] PCMGMCycle_Private line 17 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: [0] PCApply_MG line 325 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: [0] PCApply line 463 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: [0] KSP_PCApply line 240 > /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: [0] KSPInitialResidual line 44 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: [0] KSPSolve_GMRES line 225 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: [0] KSPSolve line 510 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: [0] PCApply_FieldSplit_Schur line 848 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/fieldsplit/fieldsplit.c > > [0]PETSC ERROR: [0] PCApply line 463 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: [0] KSP_PCApply line 240 > /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: [0] KSPInitialResidual line 44 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: [0] KSPSolve_GMRES line 225 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: [0] KSPSolve line 510 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: [0] PetscKernel_A_gets_inverse_A_5 line 25 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/baij/seq/dgefa5.c > > [0]PETSC ERROR: [0] MatSOR_SeqAIJ_Inode line 2764 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/aij/seq/inode.c > > [0]PETSC ERROR: [0] MatSOR line 3678 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c > > [0]PETSC ERROR: [0] PCApply_SOR line 35 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/sor/sor.c > > [0]PETSC ERROR: [0] PCApply line 463 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: [0] KSP_PCApply line 240 > /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: [0] KSPInitialResidual line 44 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: [0] KSPSolve_GMRES line 225 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: [0] KSPSolve line 510 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: [0] KSPSolve_Chebyshev line 358 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/cheby/cheby.c > > [0]PETSC ERROR: [0] KSPSolve line 510 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: [0] PCMGMCycle_Private line 17 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: [0] PCMGMCycle_Private line 17 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: [0] PCApply_MG line 325 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: [0] PCApply line 463 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: [0] KSP_PCApply line 240 > /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: [0] KSPInitialResidual line 44 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: [0] KSPSolve_GMRES line 225 > /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Signal received > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > > [0]PETSC ERROR: /user/bkhanal/home/works/AdLemModel/build/src/AdLemMain > on a arch-linux2-cxx-debug named delorme by bkhanal Mon Dec 7 23:13:00 2015 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --with-clanguage=cxx --download-fblaslapack > --download-scalapack --download-mpich --download-superlu_dist > --download-mumps --download-hypre --download-metis --download-parmetis > > [0]PETSC ERROR: #40 User provided function() line 0 in unknown file > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > [unset]: aborting job: > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xsli at lbl.gov Tue Dec 8 11:08:29 2015 From: xsli at lbl.gov (Xiaoye S. Li) Date: Tue, 8 Dec 2015 09:08:29 -0800 Subject: [petsc-users] SuperLU convergence problem (More test) In-Reply-To: References: <565F70E8.6030300@eos.ubc.ca> <5660A9CF.1030809@eos.ubc.ca> <56637D26.2080907@eos.ubc.ca> Message-ID: Hi Hong, Thank you very much for keeping tracking down the problem. I apologize for slow response... some administrative duties consumed most of my cycles for a few days. SamePattern can be used for a sequence of linear systems with the same sparsity pattern. Then the sparsity ordering is done only once, save some work. SamePattern_SameRowPerm is more aggressive, it assumes both pattern and numerical values are the same (similar), the numerical pivoting (row permutation) is also reused. This option should be used with caution, because when the values change a lot, the row permutation is not good any more. In SuperLU_DIST/SRC/pdgssvx.c, the leading comment in the code describes details. I am copying the relevant part below. Thanks, Sherry * * 3. The second value of options->Fact assumes that a matrix with the same * * * sparsity pattern as A has already been factored: * * * * * * - options->Fact = SamePattern: A is factored, assuming that it has * * * the same nonzero pattern as a previously factored matrix. In * * * this case the algorithm saves time by reusing the previously * * * computed column permutation vector stored in * * * ScalePermstruct->perm_c and the "elimination tree" of A * * * stored in LUstruct->etree * * * * * * In this case the user must still specify the following options * * * as before: * * * * * * o options->Equil * * * o options->RowPerm * * * o options->ReplaceTinyPivot * * * * * * but not options->ColPerm, whose value is ignored. This is because the * * * previous column permutation from ScalePermstruct->perm_c is used as * * * input. The user must also supply * * * * * * o A, the input matrix * * * o ScalePermstruct->perm_c, the column permutation * * * o LUstruct->etree, the elimination tree * * * The outputs returned include * * * * * * o A, the input matrix A overwritten by the scaled and permuted * * * matrix as described above * * * o ScalePermstruct, modified to describe how the input matrix A was * * * equilibrated and row permuted * * * o LUstruct, modified to contain the new L and U factors * * * * * * 4. The third value of options->Fact assumes that a matrix B with the same * * * sparsity pattern as A has already been factored, and where the * * * row permutation of B can be reused for A. This is useful when A and B * * * have similar numerical values, so that the same row permutation * * * will make both factorizations numerically stable. This lets us reuse * * * all of the previously computed structure of L and U. * * * * * * - options->Fact = SamePattern_SameRowPerm: A is factored, * * * assuming not only the same nonzero pattern as the previously * * * factored matrix B, but reusing B's row permutation. * * * * * * In this case the user must still specify the following options * * * as before: * * * * * * o options->Equil * * * o options->ReplaceTinyPivot * * * * * * but not options->RowPerm or options->ColPerm, whose values are * * * ignored. This is because the permutations from ScalePermstruct->perm_r * * * and ScalePermstruct->perm_c are used as input. * * * * * * The user must also supply * * * * * * o A, the input matrix * * * o ScalePermstruct->DiagScale, how the previous matrix was row * * * and/or column scaled * * * o ScalePermstruct->R, the row scalings of the previous matrix, * * * if any * * * o ScalePermstruct->C, the columns scalings of the previous matrix, * * * if any * * * o ScalePermstruct->perm_r, the row permutation of the previous * * * matrix * * * o ScalePermstruct->perm_c, the column permutation of the previous * * * matrix * * * o all of LUstruct, the previously computed information about * * * L and U (the actual numerical values of L and U * * * stored in LUstruct->Llu are ignored) * * * * * * The outputs returned include * * * * * * o A, the input matrix A overwritten by the scaled and permuted * * * matrix as described above * * * o ScalePermstruct, modified to describe how the input matrix A was * * * equilibrated (thus ScalePermstruct->DiagScale, * * * R and C may be modified) * * * o LUstruct, modified to contain the new L and U factors * On Mon, Dec 7, 2015 at 10:42 AM, Hong wrote: > Danyang : > > Adding '-mat_superlu_dist_fact SamePattern' fixed the problem. Below is > how I figured it out. > > 1. Reading ex52f.F, I see '-superlu_default' = > '-pc_factor_mat_solver_package superlu_dist', the later enables runtime > options for other packages. I use superlu_dist-4.2 and superlu-4.1 for the > tests below. > > 2. Use the Matrix 168 to setup KSP solver and factorization, all packages, > petsc, superlu_dist and mumps give same correct results: > > ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs > matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check > -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package > petsc > -->loac matrix a > -->load rhs b > size l,m,n,mm 90000 90000 90000 90000 > Norm of error 7.7308E-11 iterations 1 > -->Test for matrix 168 > .. > -->Test for matrix 172 > Norm of error 3.8461E-11 iterations 1 > > ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs > matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check > -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package > superlu_dist > Norm of error 9.4073E-11 iterations 1 > -->Test for matrix 168 > ... > -->Test for matrix 172 > Norm of error 3.8187E-11 iterations 1 > > 3. Use superlu, I get > ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs > matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check > -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package > superlu > Norm of error 1.0191E-06 iterations 1 > -->Test for matrix 168 > ... > -->Test for matrix 172 > Norm of error 9.7858E-07 iterations 1 > > Replacing default DiagPivotThresh: 1. to 0.0, I get same solutions as > other packages: > > ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs > matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check > -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package > superlu -mat_superlu_diagpivotthresh 0.0 > > Norm of error 8.3614E-11 iterations 1 > -->Test for matrix 168 > ... > -->Test for matrix 172 > Norm of error 3.7098E-11 iterations 1 > > 4. > using '-mat_view ascii::ascii_info', I found that a_flow_check_1.bin and > a_flow_check_168.bin seem have same structure: > > -->loac matrix a > Mat Object: 1 MPI processes > type: seqaij > rows=90000, cols=90000 > total: nonzeros=895600, allocated nonzeros=895600 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 45000 nodes, limit used is 5 > > 5. > Using a_flow_check_1.bin, I am able to reproduce the error you reported: > all packages give correct results except superlu_dist: > ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_1.bin -rhs > matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check > -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package > superlu_dist > Norm of error 2.5970E-12 iterations 1 > -->Test for matrix 168 > Norm of error 1.3936E-01 iterations 34 > -->Test for matrix 169 > > I guess the error might come from reuse of matrix factor. Replacing default > -mat_superlu_dist_fact with > -mat_superlu_dist_fact SamePattern, I get > > ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_1.bin -rhs > matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check > -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package > superlu_dist -mat_superlu_dist_fact SamePattern > > Norm of error 2.5970E-12 iterations 1 > -->Test for matrix 168 > Norm of error 9.4073E-11 iterations 1 > -->Test for matrix 169 > Norm of error 6.4303E-11 iterations 1 > -->Test for matrix 170 > Norm of error 7.4327E-11 iterations 1 > -->Test for matrix 171 > Norm of error 5.4162E-11 iterations 1 > -->Test for matrix 172 > Norm of error 3.4440E-11 iterations 1 > --> End of test, bye > > Sherry may tell you why SamePattern_SameRowPerm cause the difference here. > Best on the above experiments, I would set following as default > '-mat_superlu_diagpivotthresh 0.0' in petsc/superlu interface. > '-mat_superlu_dist_fact SamePattern' in petsc/superlu_dist interface. > > Hong > > Hi Hong, >> >> I did more test today and finally found that the solution accuracy >> depends on the initial (first) matrix quality. I modified the ex52f.F to do >> the test. There are 6 matrices and right-hand-side vectors. All these >> matrices and rhs are from my reactive transport simulation. Results will be >> quite different depending on which one you use to do factorization. Results >> will also be different if you run with different options. My code is >> similar to the First or the Second test below. When the matrix is well >> conditioned, it works fine. But if the initial matrix is well conditioned, >> it likely to crash when the matrix become ill-conditioned. Since most of my >> case are well conditioned so I didn't detect the problem before. This case >> is a special one. >> >> >> How can I avoid this problem? Shall I redo factorization? Can PETSc >> automatically detect this prolbem or is there any option available to do >> this? >> >> All the data and test code (modified ex52f) can be found via the dropbox >> link below. >> >> *https://www.dropbox.com/s/4al1a60creogd8m/petsc-superlu-test.tar.gz?dl=0 >> * >> >> >> Summary of my test is shown below. >> >> First, use the Matrix 1 to setup KSP solver and factorization, then solve >> 168 to 172 >> >> mpiexec.hydra -n 1 ./ex52f -f0 >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin >> -rhs >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin >> -loop_matrices flow_check -loop_folder >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -pc_type lu >> -pc_factor_mat_solver_package superlu_dist >> >> Norm of error 3.8815E-11 iterations 1 >> -->Test for matrix 168 >> Norm of error 4.2307E-01 iterations 32 >> -->Test for matrix 169 >> Norm of error 3.0528E-01 iterations 32 >> -->Test for matrix 170 >> Norm of error 3.1177E-01 iterations 32 >> -->Test for matrix 171 >> Norm of error 3.2793E-01 iterations 32 >> -->Test for matrix 172 >> Norm of error 3.1251E-01 iterations 31 >> >> Second, use the Matrix 1 to setup KSP solver and factorization using the >> implemented SuperLU relative codes. I thought this will generate the same >> results as the First test, but it actually not. >> >> mpiexec.hydra -n 1 ./ex52f -f0 >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin >> -rhs >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin >> -loop_matrices flow_check -loop_folder >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -superlu_default >> >> Norm of error 2.2632E-12 iterations 1 >> -->Test for matrix 168 >> Norm of error 1.0817E+04 iterations 1 >> -->Test for matrix 169 >> Norm of error 1.0786E+04 iterations 1 >> -->Test for matrix 170 >> Norm of error 1.0792E+04 iterations 1 >> -->Test for matrix 171 >> Norm of error 1.0792E+04 iterations 1 >> -->Test for matrix 172 >> Norm of error 1.0792E+04 iterations 1 >> >> >> Third, use the Matrix 168 to setup KSP solver and factorization, then >> solve 168 to 172 >> >> mpiexec.hydra -n 1 ./ex52f -f0 >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin >> -rhs >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin >> -loop_matrices flow_check -loop_folder >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -pc_type lu >> -pc_factor_mat_solver_package superlu_dist >> >> Norm of error 9.5528E-10 iterations 1 >> -->Test for matrix 168 >> Norm of error 9.4945E-10 iterations 1 >> -->Test for matrix 169 >> Norm of error 6.4279E-10 iterations 1 >> -->Test for matrix 170 >> Norm of error 7.4633E-10 iterations 1 >> -->Test for matrix 171 >> Norm of error 7.4863E-10 iterations 1 >> -->Test for matrix 172 >> Norm of error 8.9701E-10 iterations 1 >> >> Fourth, use the Matrix 168 to setup KSP solver and factorization using >> the implemented SuperLU relative codes. I thought this will generate the >> same results as the Third test, but it actually not. >> >> mpiexec.hydra -n 1 ./ex52f -f0 >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin >> -rhs >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin >> -loop_matrices flow_check -loop_folder >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -superlu_default >> >> Norm of error 3.7017E-11 iterations 1 >> -->Test for matrix 168 >> Norm of error 3.6420E-11 iterations 1 >> -->Test for matrix 169 >> Norm of error 3.7184E-11 iterations 1 >> -->Test for matrix 170 >> Norm of error 3.6847E-11 iterations 1 >> -->Test for matrix 171 >> Norm of error 3.7883E-11 iterations 1 >> -->Test for matrix 172 >> Norm of error 3.8805E-11 iterations 1 >> >> Thanks very much, >> >> Danyang >> >> On 15-12-03 01:59 PM, Hong wrote: >> >> Danyang : >> Further testing a_flow_check_168.bin, >> ./ex10 -f0 /Users/Hong/Downloads/matrix_and_rhs_bin/a_flow_check_168.bin >> -rhs /Users/Hong/Downloads/matrix_and_rhs_bin/x_flow_check_168.bin -pc_type >> lu -pc_factor_mat_solver_package superlu -ksp_monitor_true_residual >> -mat_superlu_conditionnumber >> Recip. condition number = 1.610480e-12 >> 0 KSP preconditioned resid norm 6.873340313547e+09 true resid norm >> 7.295020990196e+03 ||r(i)||/||b|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.051833296449e-02 true resid norm >> 2.976859070118e-02 ||r(i)||/||b|| 4.080672384793e-06 >> Number of iterations = 1 >> Residual norm 0.0297686 >> >> condition number of this matrix = 1/1.610480e-12 = 1.e+12, >> i.e., this matrix is ill-conditioned. >> >> Hong >> >> >> Hi Hong, >>> >>> The binary format of matrix, rhs and solution can be downloaded via the >>> link below. >>> >>> https://www.dropbox.com/s/cl3gfi0s0kjlktf/matrix_and_rhs_bin.tar.gz?dl=0 >>> >>> Thanks, >>> >>> Danyang >>> >>> >>> On 15-12-03 10:50 AM, Hong wrote: >>> >>> Danyang: >>> >>>> >>>> >>>> To my surprising, solutions from SuperLU at timestep 29 seems not >>>> correct for the first 4 Newton iterations, but the solutions from iteration >>>> solver and MUMPS are correct. >>>> >>>> Please find all the matrices, rhs and solutions at timestep 29 via the >>>> link below. The data is a bit large so that I just share it through >>>> Dropbox. A piece of matlab code to read these data and then computer the >>>> norm has also been attached. >>>> *https://www.dropbox.com/s/rr8ueysgflmxs7h/results-check.tar.gz?dl=0 >>>> * >>>> >>> >>> Can you send us matrix in petsc binary format? >>> >>> e.g., call MatView(M, PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD)) >>> or '-ksp_view_mat binary' >>> >>> Hong >>> >>>> >>>> >>>> Below is a summary of the norm from the three solvers at timestep 29, >>>> newton iteration 1 to 5. >>>> >>>> Timestep 29 >>>> Norm of residual seq 1.661321e-09, superlu 1.657103e+04, mumps >>>> 3.731225e-11 >>>> Norm of residual seq 1.753079e-09, superlu 6.675467e+02, mumps >>>> 1.509919e-13 >>>> Norm of residual seq 4.914971e-10, superlu 1.236362e-01, mumps >>>> 2.139303e-17 >>>> Norm of residual seq 3.532769e-10, superlu 1.304670e-04, mumps >>>> 5.387000e-20 >>>> Norm of residual seq 3.885629e-10, superlu 2.754876e-07, mumps >>>> 4.108675e-21 >>>> >>>> Would anybody please check if SuperLU can solve these matrices? Another >>>> possibility is that something is wrong in my own code. But so far, I cannot >>>> find any problem in my code since the same code works fine if I using >>>> iterative solver or direct solver MUMPS. But for other cases I have >>>> tested, all these solvers work fine. >>>> >>>> Please let me know if I did not write down the problem clearly. >>>> >>>> Thanks, >>>> >>>> Danyang >>>> >>>> >>>> >>>> >>> >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bibrakc at gmail.com Tue Dec 8 11:46:58 2015 From: bibrakc at gmail.com (Bibrak Qamar) Date: Tue, 8 Dec 2015 12:46:58 -0500 Subject: [petsc-users] How to get values from a matrix and set again? In-Reply-To: <04347E95-0E68-4867-A496-B4CC4B3912F6@mcs.anl.gov> References: <04347E95-0E68-4867-A496-B4CC4B3912F6@mcs.anl.gov> Message-ID: Thanks Bibrak Qamar On Mon, Dec 7, 2015 at 2:17 PM, Barry Smith wrote: > > Presumably you are using a MPIAIJ matrix. You can use > > Mat x,A,B; > MatMPIAIJGetSeqAIJ(x,&A,&B,NULL); > > PetscScalar *a; > MatSeqAIJGetArray(A,&a); > MatInfo info; > MatGetInfo(A,MAT_LOCAL,&info); > for (i=0; i a[i] = yourfunction(a[i]) > } > MatSeqAIJRestoreArray(A,&a); > /* do same thing for B matrix */ > > > On Dec 7, 2015, at 12:11 AM, Bibrak Qamar wrote: > > > > Hi, > > > > I am trying to apply a function on each element of a matrix. The > function is the sigmoid function which transforms each element of the > matrix. > > > > I am trying to approach it like this but get error. What could be the > problem? Or is there a better way to do this? > > > > int sigmoid(Mat x){ > > printf("in sigmoid\n"); > > > > PetscErrorCode ierr; > > PetscInt Istart, Iend, Ii, Ji, rows, cols; > > > > PetscScalar v = 0.0; > > ierr = MatGetOwnershipRange(x,&Istart,&Iend);CHKERRQ(ierr); > > MatGetSize(x,&rows,&cols); > > > > for (Ii=Istart; Ii > for (Ji=0; Ji > ierr = MatGetValues(x,1,&Ii,1,&Ji,&v);CHKERRQ(ierr); > > v = mysigmoid(v); > > ierr = MatSetValues(x,1,&Ii,1,&Ji,&v,INSERT_VALUES);CHKERRQ(ierr); > > } > > } > > > > MatAssemblyBegin(x,MAT_FINAL_ASSEMBLY); > > MatAssemblyEnd(x,MAT_FINAL_ASSEMBLY); > > > > > > MatView(x,PETSC_VIEWER_STDOUT_WORLD); > > } > > > > > > error msg is: > > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Object is in wrong state > > [0]PETSC ERROR: Not for unassembled matrix > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > > [ > > [0]PETSC ERROR: #1 MatGetValues() line 1780 in > /gpfs/home/petsc-3.6.3/src/mat/interface/matrix.c > > [0]PETSC ERROR: #2 sigmoid() line 21 in neural.c > > > > > > Thanks > > Bibrak Qamar > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Dec 8 13:14:23 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 8 Dec 2015 13:14:23 -0600 Subject: [petsc-users] zero pivot in LU factorization when using -fieldsplit_0_pc_type gamg in schur fieldsplit PC In-Reply-To: References: Message-ID: > On Dec 8, 2015, at 3:44 AM, Bishesh Khanal wrote: > > > > On Tue, Dec 8, 2015 at 12:57 AM, Barry Smith wrote: > > It is getting a zero pivot when factoring the little dense diagonal blocks of the smoother matrix (to do SOR). > > What happens if you run with the additional option -mat_no_inode? > > With -mat_no_inode I see that the fieldsplit_0_solve has preconditioned norm converging but not the true resid norm. And therefore fieldsplit_1_solve not converge. > > Residual norms for fieldsplit_0_ solve. > 0 KSP preconditioned resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| -nan > Residual norms for fieldsplit_0_ solve. > 0 KSP preconditioned resid norm 5.366616065799e-01 true resid norm 1.763251719324e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.051153486997e-15 true resid norm 1.763377056173e+01 ||r(i)||/||b|| 1.000071082789e+00 > Residual norms for fieldsplit_1_ solve. > Residual norms for fieldsplit_0_ solve. > 0 KSP preconditioned resid norm 0.000000000000e+00 true resid norm 0.000000000000e+00 ||r(i)||/||b|| -nan > 0 KSP preconditioned resid norm 7.379043098663e+00 true resid norm 7.379043147181e+00 ||r(i)||/||b|| 1.000000000000e+00 > Residual norms for fieldsplit_0_ solve. > 0 KSP preconditioned resid norm 1.013832784140e-01 true resid norm 1.194932744284e+00 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.651850464015e-16 true resid norm 1.195413534155e+00 ||r(i)||/||b|| 1.000402357265e+00 > Residual norms for fieldsplit_0_ solve. > 0 KSP preconditioned resid norm 9.468489343208e-01 true resid norm 1.115983634787e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.573576948730e-15 true resid norm 1.116432658910e+01 ||r(i)||/||b|| 1.000402357265e+00 > 1 KSP preconditioned resid norm 7.376179991979e+00 true resid norm 7.376180019146e+00 ||r(i)||/||b|| 9.996119919645e-01 > Residual norms for fieldsplit_0_ solve. > 0 KSP preconditioned resid norm 4.141204698273e+00 true resid norm 1.010004734003e+00 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.094717134282e-14 true resid norm 1.612661239170e+00 ||r(i)||/||b|| 1.596686812327e+00 > Residual norms for fieldsplit_0_ solve. > 0 KSP preconditioned resid norm 9.488116584954e-01 true resid norm 2.657897132073e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.486948812256e-15 true resid norm 2.658134372650e+01 ||r(i)||/||b|| 1.000089258750e+00 > 2 KSP preconditioned resid norm 7.376160898815e+00 true resid norm 7.376160928269e+00 ||r(i)||/||b|| 9.996094047893e-01 > ....................... > ....................... > Residual norms for fieldsplit_0_ solve. > 0 KSP preconditioned resid norm 3.198579793302e+00 true resid norm 1.508640813109e+00 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.839267393927e-14 true resid norm 1.825155061421e+00 ||r(i)||/||b|| 1.209800931781e+00 > Residual norms for fieldsplit_0_ solve. > 0 KSP preconditioned resid norm 1.497257742332e+01 true resid norm 4.292799680684e+14 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.138874539002e-13 true resid norm 4.292799680684e+14 ||r(i)||/||b|| 1.000000000000e+00 > 43 KSP preconditioned resid norm 6.372901415746e+00 true resid norm 7.990778566794e+00 ||r(i)||/||b|| 1.082901726878e+00 > .... And continues > > > Run with the additional option -start_in_debugger noxterm > > In the debugger type > > cont > > when it crashes type > > bt > > then use > > up > > as many times as needed to get to the routine MatSOR() then type > > call MatView(mat,0) > > you should see that diagonal subblock from the 3rd row to 7th row is singular. Normally we would expect this from the operator > mu lap(u) unless the mu has zero entries? > > Sorry I didn't understand what exactly the diagonal sublock mean here? > mat is of a big size, so it was not easy to see with MatView. Looking at its output, what I see is that every now and then there are 3 or 4 contiguous rows (i, i+1, i+2) having mostly 0 values. No mu does not have zero entries, I set it to 1 everywhere except at Dirichlet boundaries where mu is not used. > > > What happens if you run -fieldsplit_0_pc_type sor ? Does it also give a zero factorization? > > No it does not give zero factorization, the system converges in 2 iterations. > The -fieldsplit_0_ solves mostly in around 27 iterations, and for -fieldsplit_1_: > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 7 > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 7 > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 17 Ok, so how does it work for larger problems? Maybe you can use SOR for your first solver instead of GAMG if SOR works well enough? Barry > > > > > > > > On Dec 7, 2015, at 5:35 PM, Bishesh Khanal wrote: > > > > Hello, > > I'm revisiting my code for solving a system of eqs similar to Stokes to get better convergence for big jumps in viscosity (mu) and wanted to experiment with petsc's gamg: > > mu lap(u) + grad(p) = f1 > > div(u) = f2 > > Dirichlet boundary condition of zero velocity. > > The implementation is a finite difference with a staggered grid discretization. > > > > Using Schur FieldSplit PC type, and for constant viscosity, using hypre PC for A00 block had been serving well. Now when trying gamg, I get zero pivot error. From some of the past discussions in the archives of the mailing list, it seems I've to play around with ..._sub_pc_factor_shift_type but I could not figure out the exact set of options. > > > > Setting mu = 1 everywhere; Using the following options: > > -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_precondition self -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type gamg > > I get an error. > > When I use -fieldsplit_0_pc_type lu , it converges in a single iteration so I guess it should mean that A00 is not singular. > > Below are the results of using: > > 1. -fieldsplit_0_pc_type hypre (converges) > > 2. -ksp_view -fieldsplit_0_pc_type lu (converges) > > 3. -fieldsplit_0_pc_type gamg (all error output) > > > > > > 1. ---------------------------------------- > > With -fieldsplit_0_pc_type hypre > > ---------------------------------------- > > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 7 > > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 7 > > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 17 > > Linear solve converged due to CONVERGED_RTOL iterations 2 > > > > 2. -------------------------------------------- > > With -fieldsplit_0_pc_type lu -ksp_view > > -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_precondition self -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type lu -fieldsplit_1_ksp_rtol 1.0e-10 -fieldsplit_1_ksp_converged_reason -ksp_converged_reason -ksp_view > > > > Linear fieldsplit_0_ solve converged due to CONVERGED_ATOL iterations 0 > > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > > ... > > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 22 > > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > > ... > > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL iterations 22 > > Linear fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1 > > Linear solve converged due to CONVERGED_RTOL iterations 1 > > > > KSP Object: 1 MPI processes > > type: gmres > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > GMRES: happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > > left preconditioning > > using PRECONDITIONED norm type for convergence test > > PC Object: 1 MPI processes > > type: fieldsplit > > FieldSplit with Schur preconditioner, blocksize = 4, factorization FULL > > Preconditioner for the Schur complement formed from S itself > > Split info: > > Split number 0 Fields 0, 1, 2 > > Split number 1 Fields 3 > > KSP solver for A00 block > > KSP Object: (fieldsplit_0_) 1 MPI processes > > type: gmres > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > GMRES: happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > > left preconditioning > > using PRECONDITIONED norm type for convergence test > > PC Object: (fieldsplit_0_) 1 MPI processes > > type: lu > > LU: out-of-place factorization > > tolerance for zero pivot 2.22045e-14 > > matrix ordering: nd > > factor fill ratio given 5, needed 20.772 > > Factored matrix follows: > > Mat Object: 1 MPI processes > > type: seqaij > > rows=54243, cols=54243, bs=3 > > package used to perform factorization: petsc > > total: nonzeros=8.41718e+07, allocated nonzeros=8.41718e+07 > > total number of mallocs used during MatSetValues calls =0 > > using I-node routines: found 17801 nodes, limit used is 5 > > linear system matrix = precond matrix: > > Mat Object: (fieldsplit_0_) 1 MPI processes > > type: seqaij > > rows=54243, cols=54243, bs=3 > > total: nonzeros=4.05217e+06, allocated nonzeros=4.05217e+06 > > total number of mallocs used during MatSetValues calls =0 > > using I-node routines: found 18081 nodes, limit used is 5 > > KSP solver for S = A11 - A10 inv(A00) A01 > > KSP Object: (fieldsplit_1_) 1 MPI processes > > type: gmres > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > GMRES: happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-10, absolute=1e-50, divergence=10000 > > left preconditioning > > using PRECONDITIONED norm type for convergence test > > PC Object: (fieldsplit_1_) 1 MPI processes > > type: none > > linear system matrix = precond matrix: > > Mat Object: (fieldsplit_1_) 1 MPI processes > > type: schurcomplement > > rows=18081, cols=18081 > > has attached null space > > Schur complement A11 - A10 inv(A00) A01 > > A11 > > Mat Object: (fieldsplit_1_) 1 MPI processes > > type: seqaij > > rows=18081, cols=18081 > > total: nonzeros=450241, allocated nonzeros=450241 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node routines > > A10 > > Mat Object: 1 MPI processes > > type: seqaij > > rows=18081, cols=54243 > > total: nonzeros=1.35072e+06, allocated nonzeros=1.35072e+06 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node routines > > KSP of A00 > > KSP Object: (fieldsplit_0_) 1 MPI processes > > type: gmres > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > GMRES: happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > > left preconditioning > > using PRECONDITIONED norm type for convergence test > > PC Object: (fieldsplit_0_) 1 MPI processes > > type: lu > > LU: out-of-place factorization > > tolerance for zero pivot 2.22045e-14 > > matrix ordering: nd > > factor fill ratio given 5, needed 20.772 > > Factored matrix follows: > > Mat Object: 1 MPI processes > > type: seqaij > > rows=54243, cols=54243, bs=3 > > package used to perform factorization: petsc > > total: nonzeros=8.41718e+07, allocated nonzeros=8.41718e+07 > > total number of mallocs used during MatSetValues calls =0 > > using I-node routines: found 17801 nodes, limit used is 5 > > linear system matrix = precond matrix: > > Mat Object: (fieldsplit_0_) 1 MPI processes > > type: seqaij > > rows=54243, cols=54243, bs=3 > > total: nonzeros=4.05217e+06, allocated nonzeros=4.05217e+06 > > total number of mallocs used during MatSetValues calls =0 > > using I-node routines: found 18081 nodes, limit used is 5 > > A01 > > Mat Object: 1 MPI processes > > type: seqaij > > rows=54243, cols=18081, rbs=3, cbs = 1 > > total: nonzeros=1.35072e+06, allocated nonzeros=1.35072e+06 > > total number of mallocs used during MatSetValues calls =0 > > using I-node routines: found 18081 nodes, limit used is 5 > > linear system matrix = precond matrix: > > Mat Object: 1 MPI processes > > type: seqaij > > rows=72324, cols=72324, bs=4 > > total: nonzeros=7.20386e+06, allocated nonzeros=7.20386e+06 > > total number of mallocs used during MatSetValues calls =0 > > has attached null space > > using I-node routines: found 18081 nodes, limit used is 5 > > > > > > 3. ---------------------------------------- > > when using gamg: -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_precondition self -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type gamg -fieldsplit_0_ksp_converged_reason -fieldsplit_1_ksp_converged_reason -ksp_converged_reason -ksp_view > > ---------------------------------------- > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot > > [0]PETSC ERROR: Zero pivot, row 3 > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > > [0]PETSC ERROR: /user/bkhanal/home/works/AdLemModel/build/src/AdLemMain on a arch-linux2-cxx-debug named delorme by bkhanal Mon Dec 7 23:13:00 2015 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-clanguage=cxx --download-fblaslapack --download-scalapack --download-mpich --download-superlu_dist --download-mumps --download-hypre --download-metis --download-parmetis > > [0]PETSC ERROR: #1 PetscKernel_A_gets_inverse_A_5() line 48 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/baij/seq/dgefa5.c > > [0]PETSC ERROR: #2 MatSOR_SeqAIJ_Inode() line 2808 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/aij/seq/inode.c > > [0]PETSC ERROR: #3 MatSOR() line 3697 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c > > [0]PETSC ERROR: #4 PCApply_SOR() line 37 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/sor/sor.c > > [0]PETSC ERROR: #5 PCApply() line 483 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #6 KSP_PCApply() line 242 in /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: #7 KSPInitialResidual() line 63 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: #8 KSPSolve_GMRES() line 235 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: #9 KSPSolve() line 604 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #10 KSPSolve_Chebyshev() line 381 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/cheby/cheby.c > > [0]PETSC ERROR: #11 KSPSolve() line 604 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #12 PCMGMCycle_Private() line 19 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: #13 PCMGMCycle_Private() line 48 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: #14 PCApply_MG() line 340 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: #15 PCApply() line 483 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #16 KSP_PCApply() line 242 in /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: #17 KSPInitialResidual() line 63 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: #18 KSPSolve_GMRES() line 235 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: #19 KSPSolve() line 604 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #20 MatMult_SchurComplement() line 105 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/utils/schurm.c > > [0]PETSC ERROR: #21 MatNullSpaceTest() line 431 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matnull.c > > > > WARNING: not a valid pressure null space > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Object is in wrong state > > [0]PETSC ERROR: Vec is locked read only, argument # 1 > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > > [0]PETSC ERROR: /user/bkhanal/home/works/AdLemModel/build/src/AdLemMain on a arch-linux2-cxx-debug named delorme by bkhanal Mon Dec 7 23:13:00 2015 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-clanguage=cxx --download-fblaslapack --download-scalapack --download-mpich --download-superlu_dist --download-mumps --download-hypre --download-metis --download-parmetis > > [0]PETSC ERROR: #22 VecSet() line 573 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/vec/vec/interface/rvector.c > > [0]PETSC ERROR: #23 MatMultTranspose_SeqAIJ() line 1263 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/aij/seq/aij.c > > [0]PETSC ERROR: #24 MatMultTranspose() line 2282 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c > > [0]PETSC ERROR: #25 MatRestrict() line 7829 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c > > [0]PETSC ERROR: #26 PCMGMCycle_Private() line 44 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: #27 PCApply_MG() line 340 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: #28 PCApply() line 483 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #29 KSP_PCApply() line 242 in /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: #30 KSPInitialResidual() line 63 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: #31 KSPSolve_GMRES() line 235 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: #32 KSPSolve() line 604 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #33 PCApply_FieldSplit_Schur() line 898 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/fieldsplit/fieldsplit.c > > [0]PETSC ERROR: #34 PCApply() line 483 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #35 KSP_PCApply() line 242 in /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: #36 KSPInitialResidual() line 63 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: #37 KSPSolve_GMRES() line 235 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: #38 KSPSolve() line 604 in /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #39 solveModel() line 313 in /home/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > > [0]PETSC ERROR: likely location of problem given in stack below > > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > > [0]PETSC ERROR: INSTEAD the line number of the start of the function > > [0]PETSC ERROR: is given. > > [0]PETSC ERROR: [0] getSolVelocityAt line 137 /home/bkhanal/works/AdLemModel/src/PetscAdLem3D.cxx > > [0]PETSC ERROR: [0] VecSet line 568 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/vec/vec/interface/rvector.c > > [0]PETSC ERROR: [0] MatMultTranspose_SeqAIJ line 1262 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/aij/seq/aij.c > > [0]PETSC ERROR: [0] MatMultTranspose line 2263 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c > > [0]PETSC ERROR: [0] MatRestrict line 7817 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c > > [0]PETSC ERROR: [0] PCMGMCycle_Private line 17 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: [0] PCApply_MG line 325 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: [0] PCApply line 463 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: [0] KSP_PCApply line 240 /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: [0] KSPInitialResidual line 44 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: [0] KSPSolve_GMRES line 225 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: [0] KSPSolve line 510 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: [0] PCApply_FieldSplit_Schur line 848 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/fieldsplit/fieldsplit.c > > [0]PETSC ERROR: [0] PCApply line 463 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: [0] KSP_PCApply line 240 /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: [0] KSPInitialResidual line 44 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: [0] KSPSolve_GMRES line 225 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: [0] KSPSolve line 510 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: [0] PetscKernel_A_gets_inverse_A_5 line 25 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/baij/seq/dgefa5.c > > [0]PETSC ERROR: [0] MatSOR_SeqAIJ_Inode line 2764 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/impls/aij/seq/inode.c > > [0]PETSC ERROR: [0] MatSOR line 3678 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/mat/interface/matrix.c > > [0]PETSC ERROR: [0] PCApply_SOR line 35 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/sor/sor.c > > [0]PETSC ERROR: [0] PCApply line 463 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: [0] KSP_PCApply line 240 /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: [0] KSPInitialResidual line 44 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: [0] KSPSolve_GMRES line 225 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: [0] KSPSolve line 510 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: [0] KSPSolve_Chebyshev line 358 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/cheby/cheby.c > > [0]PETSC ERROR: [0] KSPSolve line 510 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: [0] PCMGMCycle_Private line 17 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: [0] PCMGMCycle_Private line 17 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: [0] PCApply_MG line 325 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: [0] PCApply line 463 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: [0] KSP_PCApply line 240 /home/bkhanal/Documents/softwares/petsc-3.6.3/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: [0] KSPInitialResidual line 44 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: [0] KSPSolve_GMRES line 225 /home/bkhanal/Documents/softwares/petsc-3.6.3/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Signal received > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > > [0]PETSC ERROR: /user/bkhanal/home/works/AdLemModel/build/src/AdLemMain on a arch-linux2-cxx-debug named delorme by bkhanal Mon Dec 7 23:13:00 2015 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-clanguage=cxx --download-fblaslapack --download-scalapack --download-mpich --download-superlu_dist --download-mumps --download-hypre --download-metis --download-parmetis > > [0]PETSC ERROR: #40 User provided function() line 0 in unknown file > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > [unset]: aborting job: > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > > > From dsu at eos.ubc.ca Tue Dec 8 20:37:20 2015 From: dsu at eos.ubc.ca (Danyang Su) Date: Tue, 8 Dec 2015 18:37:20 -0800 Subject: [petsc-users] SuperLU convergence problem (More test) In-Reply-To: References: <565F70E8.6030300@eos.ubc.ca> <5660A9CF.1030809@eos.ubc.ca> <56637D26.2080907@eos.ubc.ca> <5665DDDA.405@eos.ubc.ca> Message-ID: <566793E0.6060907@eos.ubc.ca> Hi Hong, Sorry to bother you again. The modified code works much better than before using both superlu or mumps. However, it still encounters failure. The case is similar with the previous one, ill-conditioned matrices. The code crashed after a long time simulation if I use superlu_dist, but will not fail if use superlu. I restart the simulation before the time it crashes and can reproduce the following error timestep: 22 time: 1.750E+04 years delt: 2.500E+00 years iter: 1 max.sia: 5.053E-03 tol.sia: 1.000E-01 Newton Iteration Convergence Summary: Newton maximum maximum solver iteration updatePa updateTemp residual iterations maxvolpa maxvoltemp nexvolpa nexvoltemp 1 0.1531E+08 0.1755E+04 0.6920E-05 1 5585 4402 5814 5814 *** Error in `../program_test': malloc(): memory corruption: 0x0000000003a70d50 *** Program received signal SIGABRT: Process abort signal. Backtrace for this error: The solver failed at timestep 22, Newton iteration 2. I exported the matrices at timestep 1 (matrix 1) and timestep 22 (matrix 140 and 141). Matrix 141 is where it failed. The three matrices here are not ill-conditioned form the estimated value. I did the same using the new modified ex52f code and found pretty different results for matrix 141. The norm by superlu is much acceptable than superlu_dist. In this test, memory corruption was not detected. The codes and example data can be download from the link below. https://www.dropbox.com/s/i1ls0bg0vt7gu0v/petsc-superlu-test2.tar.gz?dl=0 ****************More test on matrix_and_rhs_bin2******************* mpiexec.hydra -n 1 ./ex52f -f0 ./matrix_and_rhs_bin2/a_flow_check_1.bin -rhs ./matrix_and_rhs_bin2/b_flow_check_1.bin -loop_matrices flow_check -loop_folder ./matrix_and_rhs_bin2 -matrix_index_start 140 -matrix_index_end 141 -pc_type lu -pc_factor_mat_solver_package superlu -ksp_monitor_true_residual -mat_superlu_conditionnumber -->loac matrix a -->load rhs b size l,m,n,mm 90000 90000 90000 90000 Recip. condition number = 6.000846e-16 0 KSP preconditioned resid norm 1.146871454377e+08 true resid norm 4.711091037809e+03 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.071118508260e-06 true resid norm 3.363767171515e-08 ||r(i)||/||b|| 7.140102249181e-12 Norm of error 3.3638E-08 iterations 1 -->Test for matrix 140 Recip. condition number = 2.256434e-27 0 KSP preconditioned resid norm 2.084372893355e+14 true resid norm 4.711091037809e+03 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.689629276419e+00 true resid norm 1.037236635337e-01 ||r(i)||/||b|| 2.201690918330e-05 Norm of error 1.0372E-01 iterations 1 -->Test for matrix 141 Recip. condition number = 1.256452e-18 0 KSP preconditioned resid norm 1.055488964519e+08 true resid norm 4.711091037809e+03 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.998827511681e-04 true resid norm 4.805214542776e-04 ||r(i)||/||b|| 1.019979130994e-07 Norm of error 4.8052E-04 iterations 1 --> End of test, bye mpiexec.hydra -n 1 ./ex52f -f0 ./matrix_and_rhs_bin2/a_flow_check_1.bin -rhs ./matrix_and_rhs_bin2/b_flow_check_1.bin -loop_matrices flow_check -loop_folder ./matrix_and_rhs_bin2 -matrix_index_start 140 -matrix_index_end 141 -pc_type lu -pc_factor_mat_solver_package superlu_dist -->loac matrix a -->load rhs b size l,m,n,mm 90000 90000 90000 90000 Norm of error 3.6752E-08 iterations 1 -->Test for matrix 140 Norm of error 1.6335E-01 iterations 1 -->Test for matrix 141 Norm of error 3.4345E+01 iterations 1 --> End of test, bye Thanks, Danyang On 15-12-07 12:01 PM, Hong wrote: > Danyang: > Add 'call MatSetFromOptions(A,ierr)' to your code. > Attached below is ex52f.F modified from your ex52f.F to be compatible > with petsc-dev. > > Hong > > Hello Hong, > > Thanks for the quick reply and the option "-mat_superlu_dist_fact > SamePattern" works like a charm, if I use this option from the > command line. > > How can I add this option as the default. I tried using > PetscOptionsInsertString("-mat_superlu_dist_fact > SamePattern",ierr) in my code but this does not work. > > Thanks, > > Danyang > > > On 15-12-07 10:42 AM, Hong wrote: >> Danyang : >> >> Adding '-mat_superlu_dist_fact SamePattern' fixed the problem. >> Below is how I figured it out. >> >> 1. Reading ex52f.F, I see '-superlu_default' = >> '-pc_factor_mat_solver_package superlu_dist', the later enables >> runtime options for other packages. I use superlu_dist-4.2 and >> superlu-4.1 for the tests below. >> >> 2. Use the Matrix 168 to setup KSP solver and factorization, all >> packages, petsc, superlu_dist and mumps give same correct results: >> >> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs >> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check >> -loop_folder matrix_and_rhs_bin -pc_type lu >> -pc_factor_mat_solver_package petsc >> -->loac matrix a >> -->load rhs b >> size l,m,n,mm 90000 90000 90000 90000 >> Norm of error 7.7308E-11 iterations 1 >> -->Test for matrix 168 >> .. >> -->Test for matrix 172 >> Norm of error 3.8461E-11 iterations 1 >> >> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs >> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check >> -loop_folder matrix_and_rhs_bin -pc_type lu >> -pc_factor_mat_solver_package superlu_dist >> Norm of error 9.4073E-11 iterations 1 >> -->Test for matrix 168 >> ... >> -->Test for matrix 172 >> Norm of error 3.8187E-11 iterations 1 >> >> 3. Use superlu, I get >> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs >> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check >> -loop_folder matrix_and_rhs_bin -pc_type lu >> -pc_factor_mat_solver_package superlu >> Norm of error 1.0191E-06 iterations 1 >> -->Test for matrix 168 >> ... >> -->Test for matrix 172 >> Norm of error 9.7858E-07 iterations 1 >> >> Replacing default DiagPivotThresh: 1. to 0.0, I get same >> solutions as other packages: >> >> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs >> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check >> -loop_folder matrix_and_rhs_bin -pc_type lu >> -pc_factor_mat_solver_package superlu >> -mat_superlu_diagpivotthresh 0.0 >> >> Norm of error 8.3614E-11 iterations 1 >> -->Test for matrix 168 >> ... >> -->Test for matrix 172 >> Norm of error 3.7098E-11 iterations 1 >> >> 4. >> using '-mat_view ascii::ascii_info', I found that >> a_flow_check_1.bin and a_flow_check_168.bin seem have same structure: >> >> -->loac matrix a >> Mat Object: 1 MPI processes >> type: seqaij >> rows=90000, cols=90000 >> total: nonzeros=895600, allocated nonzeros=895600 >> total number of mallocs used during MatSetValues calls =0 >> using I-node routines: found 45000 nodes, limit used is 5 >> >> 5. >> Using a_flow_check_1.bin, I am able to reproduce the error you >> reported: all packages give correct results except superlu_dist: >> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_1.bin -rhs >> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check >> -loop_folder matrix_and_rhs_bin -pc_type lu >> -pc_factor_mat_solver_package superlu_dist >> Norm of error 2.5970E-12 iterations 1 >> -->Test for matrix 168 >> Norm of error 1.3936E-01 iterations 34 >> -->Test for matrix 169 >> >> I guess the error might come from reuse of matrix factor. >> Replacing default >> -mat_superlu_dist_fact with >> -mat_superlu_dist_fact SamePattern, I get >> >> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_1.bin -rhs >> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check >> -loop_folder matrix_and_rhs_bin -pc_type lu >> -pc_factor_mat_solver_package superlu_dist -mat_superlu_dist_fact >> SamePattern >> >> Norm of error 2.5970E-12 iterations 1 >> -->Test for matrix 168 >> Norm of error 9.4073E-11 iterations 1 >> -->Test for matrix 169 >> Norm of error 6.4303E-11 iterations 1 >> -->Test for matrix 170 >> Norm of error 7.4327E-11 iterations 1 >> -->Test for matrix 171 >> Norm of error 5.4162E-11 iterations 1 >> -->Test for matrix 172 >> Norm of error 3.4440E-11 iterations 1 >> --> End of test, bye >> >> Sherry may tell you why SamePattern_SameRowPerm cause the >> difference here. >> Best on the above experiments, I would set following as default >> '-mat_superlu_diagpivotthresh 0.0' in petsc/superlu interface. >> '-mat_superlu_dist_fact SamePattern' in petsc/superlu_dist interface. >> >> Hong >> >> Hi Hong, >> >> I did more test today and finally found that the solution >> accuracy depends on the initial (first) matrix quality. I >> modified the ex52f.F to do the test. There are 6 matrices and >> right-hand-side vectors. All these matrices and rhs are from >> my reactive transport simulation. Results will be quite >> different depending on which one you use to do factorization. >> Results will also be different if you run with different >> options. My code is similar to the First or the Second test >> below. When the matrix is well conditioned, it works fine. >> But if the initial matrix is well conditioned, it likely to >> crash when the matrix become ill-conditioned. Since most of >> my case are well conditioned so I didn't detect the problem >> before. This case is a special one. >> >> >> How can I avoid this problem? Shall I redo factorization? Can >> PETSc automatically detect this prolbem or is there any >> option available to do this? >> >> All the data and test code (modified ex52f) can be found via >> the dropbox link below. >> _ >> __https://www.dropbox.com/s/4al1a60creogd8m/petsc-superlu-test.tar.gz?dl=0_ >> >> >> Summary of my test is shown below. >> >> First, use the Matrix 1 to setup KSP solver and >> factorization, then solve 168 to 172 >> >> mpiexec.hydra -n 1 ./ex52f -f0 >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin >> -rhs >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin >> -loop_matrices flow_check -loop_folder >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -pc_type >> lu -pc_factor_mat_solver_package superlu_dist >> >> Norm of error 3.8815E-11 iterations 1 >> -->Test for matrix 168 >> Norm of error 4.2307E-01 iterations 32 >> -->Test for matrix 169 >> Norm of error 3.0528E-01 iterations 32 >> -->Test for matrix 170 >> Norm of error 3.1177E-01 iterations 32 >> -->Test for matrix 171 >> Norm of error 3.2793E-01 iterations 32 >> -->Test for matrix 172 >> Norm of error 3.1251E-01 iterations 31 >> >> Second, use the Matrix 1 to setup KSP solver and >> factorization using the implemented SuperLU relative codes. I >> thought this will generate the same results as the First >> test, but it actually not. >> >> mpiexec.hydra -n 1 ./ex52f -f0 >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin >> -rhs >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin >> -loop_matrices flow_check -loop_folder >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin >> -superlu_default >> >> Norm of error 2.2632E-12 iterations 1 >> -->Test for matrix 168 >> Norm of error 1.0817E+04 iterations 1 >> -->Test for matrix 169 >> Norm of error 1.0786E+04 iterations 1 >> -->Test for matrix 170 >> Norm of error 1.0792E+04 iterations 1 >> -->Test for matrix 171 >> Norm of error 1.0792E+04 iterations 1 >> -->Test for matrix 172 >> Norm of error 1.0792E+04 iterations 1 >> >> >> Third, use the Matrix 168 to setup KSP solver and >> factorization, then solve 168 to 172 >> >> mpiexec.hydra -n 1 ./ex52f -f0 >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin >> -rhs >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin >> -loop_matrices flow_check -loop_folder >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -pc_type >> lu -pc_factor_mat_solver_package superlu_dist >> >> Norm of error 9.5528E-10 iterations 1 >> -->Test for matrix 168 >> Norm of error 9.4945E-10 iterations 1 >> -->Test for matrix 169 >> Norm of error 6.4279E-10 iterations 1 >> -->Test for matrix 170 >> Norm of error 7.4633E-10 iterations 1 >> -->Test for matrix 171 >> Norm of error 7.4863E-10 iterations 1 >> -->Test for matrix 172 >> Norm of error 8.9701E-10 iterations 1 >> >> Fourth, use the Matrix 168 to setup KSP solver and >> factorization using the implemented SuperLU relative codes. I >> thought this will generate the same results as the Third >> test, but it actually not. >> >> mpiexec.hydra -n 1 ./ex52f -f0 >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin >> -rhs >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin >> -loop_matrices flow_check -loop_folder >> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin >> -superlu_default >> >> Norm of error 3.7017E-11 iterations 1 >> -->Test for matrix 168 >> Norm of error 3.6420E-11 iterations 1 >> -->Test for matrix 169 >> Norm of error 3.7184E-11 iterations 1 >> -->Test for matrix 170 >> Norm of error 3.6847E-11 iterations 1 >> -->Test for matrix 171 >> Norm of error 3.7883E-11 iterations 1 >> -->Test for matrix 172 >> Norm of error 3.8805E-11 iterations 1 >> >> Thanks very much, >> >> Danyang >> >> On 15-12-03 01:59 PM, Hong wrote: >>> Danyang : >>> Further testing a_flow_check_168.bin, >>> ./ex10 -f0 >>> /Users/Hong/Downloads/matrix_and_rhs_bin/a_flow_check_168.bin -rhs >>> /Users/Hong/Downloads/matrix_and_rhs_bin/x_flow_check_168.bin -pc_type >>> lu -pc_factor_mat_solver_package superlu >>> -ksp_monitor_true_residual -mat_superlu_conditionnumber >>> Recip. condition number = 1.610480e-12 >>> 0 KSP preconditioned resid norm 6.873340313547e+09 true >>> resid norm 7.295020990196e+03 ||r(i)||/||b|| 1.000000000000e+00 >>> 1 KSP preconditioned resid norm 2.051833296449e-02 true >>> resid norm 2.976859070118e-02 ||r(i)||/||b|| 4.080672384793e-06 >>> Number of iterations = 1 >>> Residual norm 0.0297686 >>> >>> condition number of this matrix = 1/1.610480e-12 = 1.e+12, >>> i.e., this matrix is ill-conditioned. >>> >>> Hong >>> >>> >>> Hi Hong, >>> >>> The binary format of matrix, rhs and solution can be >>> downloaded via the link below. >>> >>> https://www.dropbox.com/s/cl3gfi0s0kjlktf/matrix_and_rhs_bin.tar.gz?dl=0 >>> >>> Thanks, >>> >>> Danyang >>> >>> >>> On 15-12-03 10:50 AM, Hong wrote: >>>> Danyang: >>>> >>>> >>>> >>>> To my surprising, solutions from SuperLU at >>>> timestep 29 seems not correct for the first 4 >>>> Newton iterations, but the solutions from iteration >>>> solver and MUMPS are correct. >>>> >>>> Please find all the matrices, rhs and solutions at >>>> timestep 29 via the link below. The data is a bit >>>> large so that I just share it through Dropbox. A >>>> piece of matlab code to read these data and then >>>> computer the norm has also been attached. >>>> _https://www.dropbox.com/s/rr8ueysgflmxs7h/results-check.tar.gz?dl=0_ >>>> >>>> >>>> Can you send us matrix in petsc binary format? >>>> >>>> e.g., call MatView(M, >>>> PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD)) >>>> or '-ksp_view_mat binary' >>>> >>>> Hong >>>> >>>> >>>> >>>> Below is a summary of the norm from the three >>>> solvers at timestep 29, newton iteration 1 to 5. >>>> >>>> Timestep 29 >>>> Norm of residual seq 1.661321e-09, superlu >>>> 1.657103e+04, mumps 3.731225e-11 >>>> Norm of residual seq 1.753079e-09, superlu >>>> 6.675467e+02, mumps 1.509919e-13 >>>> Norm of residual seq 4.914971e-10, superlu >>>> 1.236362e-01, mumps 2.139303e-17 >>>> Norm of residual seq 3.532769e-10, superlu >>>> 1.304670e-04, mumps 5.387000e-20 >>>> Norm of residual seq 3.885629e-10, superlu >>>> 2.754876e-07, mumps 4.108675e-21 >>>> >>>> Would anybody please check if SuperLU can solve >>>> these matrices? Another possibility is that >>>> something is wrong in my own code. But so far, I >>>> cannot find any problem in my code since the same >>>> code works fine if I using iterative solver or >>>> direct solver MUMPS. But for other cases I have >>>> tested, all these solvers work fine. >>>> >>>> Please let me know if I did not write down the >>>> problem clearly. >>>> >>>> Thanks, >>>> >>>> Danyang >>>> >>>> >>>> >>>> >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Tue Dec 8 22:10:19 2015 From: hzhang at mcs.anl.gov (Hong) Date: Tue, 8 Dec 2015 22:10:19 -0600 Subject: [petsc-users] SuperLU convergence problem (More test) In-Reply-To: <566793E0.6060907@eos.ubc.ca> References: <565F70E8.6030300@eos.ubc.ca> <5660A9CF.1030809@eos.ubc.ca> <56637D26.2080907@eos.ubc.ca> <5665DDDA.405@eos.ubc.ca> <566793E0.6060907@eos.ubc.ca> Message-ID: Danyang : Your matrices are ill-conditioned, numerically singular with Recip. condition number = 6.000846e-16 Recip. condition number = 2.256434e-27 Recip. condition number = 1.256452e-18 i.e., condition numbers = O(1.e16 - 1.e27), there is no accuracy in computed solution. I checked your matrix 168 - 172, got Recip. condition number = 1.548816e-12. You need check your model to understand why the matrices are so ill-conditioned. Hong Hi Hong, > > Sorry to bother you again. The modified code works much better than before > using both superlu or mumps. However, it still encounters failure. The case > is similar with the previous one, ill-conditioned matrices. > > The code crashed after a long time simulation if I use superlu_dist, but > will not fail if use superlu. I restart the simulation before the time it > crashes and can reproduce the following error > > timestep: 22 time: 1.750E+04 years delt: 2.500E+00 years iter: > 1 max.sia: 5.053E-03 tol.sia: 1.000E-01 > Newton Iteration Convergence Summary: > Newton maximum maximum solver > iteration updatePa updateTemp residual iterations maxvolpa > maxvoltemp nexvolpa nexvoltemp > 1 0.1531E+08 0.1755E+04 0.6920E-05 1 5585 > 4402 5814 5814 > > *** Error in `../program_test': malloc(): memory corruption: > 0x0000000003a70d50 *** > Program received signal SIGABRT: Process abort signal. > Backtrace for this error: > > The solver failed at timestep 22, Newton iteration 2. I exported the > matrices at timestep 1 (matrix 1) and timestep 22 (matrix 140 and 141). > Matrix 141 is where it failed. The three matrices here are not > ill-conditioned form the estimated value. > > I did the same using the new modified ex52f code and found pretty > different results for matrix 141. The norm by superlu is much acceptable > than superlu_dist. In this test, memory corruption was not detected. The > codes and example data can be download from the link below. > > https://www.dropbox.com/s/i1ls0bg0vt7gu0v/petsc-superlu-test2.tar.gz?dl=0 > > > ****************More test on matrix_and_rhs_bin2******************* > mpiexec.hydra -n 1 ./ex52f -f0 ./matrix_and_rhs_bin2/a_flow_check_1.bin > -rhs ./matrix_and_rhs_bin2/b_flow_check_1.bin -loop_matrices flow_check > -loop_folder ./matrix_and_rhs_bin2 -matrix_index_start 140 > -matrix_index_end 141 -pc_type lu -pc_factor_mat_solver_package superlu > -ksp_monitor_true_residual -mat_superlu_conditionnumber > -->loac matrix a > -->load rhs b > size l,m,n,mm 90000 90000 90000 90000 > Recip. condition number = 6.000846e-16 > 0 KSP preconditioned resid norm 1.146871454377e+08 true resid norm > 4.711091037809e+03 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.071118508260e-06 true resid norm > 3.363767171515e-08 ||r(i)||/||b|| 7.140102249181e-12 > Norm of error 3.3638E-08 iterations 1 > -->Test for matrix 140 > Recip. condition number = 2.256434e-27 > 0 KSP preconditioned resid norm 2.084372893355e+14 true resid norm > 4.711091037809e+03 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.689629276419e+00 true resid norm > 1.037236635337e-01 ||r(i)||/||b|| 2.201690918330e-05 > Norm of error 1.0372E-01 iterations 1 > -->Test for matrix 141 > Recip. condition number = 1.256452e-18 > 0 KSP preconditioned resid norm 1.055488964519e+08 true resid norm > 4.711091037809e+03 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.998827511681e-04 true resid norm > 4.805214542776e-04 ||r(i)||/||b|| 1.019979130994e-07 > Norm of error 4.8052E-04 iterations 1 > --> End of test, bye > > > mpiexec.hydra -n 1 ./ex52f -f0 ./matrix_and_rhs_bin2/a_flow_check_1.bin > -rhs ./matrix_and_rhs_bin2/b_flow_check_1.bin -loop_matrices flow_check > -loop_folder ./matrix_and_rhs_bin2 -matrix_index_start 140 > -matrix_index_end 141 -pc_type lu -pc_factor_mat_solver_package > superlu_dist > -->loac matrix a > -->load rhs b > size l,m,n,mm 90000 90000 90000 90000 > Norm of error 3.6752E-08 iterations 1 > -->Test for matrix 140 > Norm of error 1.6335E-01 iterations 1 > -->Test for matrix 141 > Norm of error 3.4345E+01 iterations 1 > --> End of test, bye > > Thanks, > > Danyang > > On 15-12-07 12:01 PM, Hong wrote: > > Danyang: > Add 'call MatSetFromOptions(A,ierr)' to your code. > Attached below is ex52f.F modified from your ex52f.F to be compatible with > petsc-dev. > > Hong > > Hello Hong, >> >> Thanks for the quick reply and the option "-mat_superlu_dist_fact >> SamePattern" works like a charm, if I use this option from the command >> line. >> >> How can I add this option as the default. I tried using >> PetscOptionsInsertString("-mat_superlu_dist_fact SamePattern",ierr) in my >> code but this does not work. >> >> Thanks, >> >> Danyang >> >> >> On 15-12-07 10:42 AM, Hong wrote: >> >> Danyang : >> >> Adding '-mat_superlu_dist_fact SamePattern' fixed the problem. Below is >> how I figured it out. >> >> 1. Reading ex52f.F, I see '-superlu_default' = >> '-pc_factor_mat_solver_package superlu_dist', the later enables runtime >> options for other packages. I use superlu_dist-4.2 and superlu-4.1 for the >> tests below. >> >> 2. Use the Matrix 168 to setup KSP solver and factorization, all >> packages, petsc, superlu_dist and mumps give same correct results: >> >> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs >> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check >> -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package >> petsc >> -->loac matrix a >> -->load rhs b >> size l,m,n,mm 90000 90000 90000 90000 >> Norm of error 7.7308E-11 iterations 1 >> -->Test for matrix 168 >> .. >> -->Test for matrix 172 >> Norm of error 3.8461E-11 iterations 1 >> >> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs >> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check >> -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package >> superlu_dist >> Norm of error 9.4073E-11 iterations 1 >> -->Test for matrix 168 >> ... >> -->Test for matrix 172 >> Norm of error 3.8187E-11 iterations 1 >> >> 3. Use superlu, I get >> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs >> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check >> -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package >> superlu >> Norm of error 1.0191E-06 iterations 1 >> -->Test for matrix 168 >> ... >> -->Test for matrix 172 >> Norm of error 9.7858E-07 iterations 1 >> >> Replacing default DiagPivotThresh: 1. to 0.0, I get same solutions as >> other packages: >> >> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs >> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check >> -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package >> superlu -mat_superlu_diagpivotthresh 0.0 >> >> Norm of error 8.3614E-11 iterations 1 >> -->Test for matrix 168 >> ... >> -->Test for matrix 172 >> Norm of error 3.7098E-11 iterations 1 >> >> 4. >> using '-mat_view ascii::ascii_info', I found that a_flow_check_1.bin and >> a_flow_check_168.bin seem have same structure: >> >> -->loac matrix a >> Mat Object: 1 MPI processes >> type: seqaij >> rows=90000, cols=90000 >> total: nonzeros=895600, allocated nonzeros=895600 >> total number of mallocs used during MatSetValues calls =0 >> using I-node routines: found 45000 nodes, limit used is 5 >> >> 5. >> Using a_flow_check_1.bin, I am able to reproduce the error you reported: >> all packages give correct results except superlu_dist: >> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_1.bin -rhs >> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check >> -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package >> superlu_dist >> Norm of error 2.5970E-12 iterations 1 >> -->Test for matrix 168 >> Norm of error 1.3936E-01 iterations 34 >> -->Test for matrix 169 >> >> I guess the error might come from reuse of matrix factor. Replacing >> default >> -mat_superlu_dist_fact with >> -mat_superlu_dist_fact SamePattern, I get >> >> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_1.bin -rhs >> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices flow_check >> -loop_folder matrix_and_rhs_bin -pc_type lu -pc_factor_mat_solver_package >> superlu_dist -mat_superlu_dist_fact SamePattern >> >> Norm of error 2.5970E-12 iterations 1 >> -->Test for matrix 168 >> Norm of error 9.4073E-11 iterations 1 >> -->Test for matrix 169 >> Norm of error 6.4303E-11 iterations 1 >> -->Test for matrix 170 >> Norm of error 7.4327E-11 iterations 1 >> -->Test for matrix 171 >> Norm of error 5.4162E-11 iterations 1 >> -->Test for matrix 172 >> Norm of error 3.4440E-11 iterations 1 >> --> End of test, bye >> >> Sherry may tell you why SamePattern_SameRowPerm cause the difference here. >> Best on the above experiments, I would set following as default >> '-mat_superlu_diagpivotthresh 0.0' in petsc/superlu interface. >> '-mat_superlu_dist_fact SamePattern' in petsc/superlu_dist interface. >> >> Hong >> >> Hi Hong, >>> >>> I did more test today and finally found that the solution accuracy >>> depends on the initial (first) matrix quality. I modified the ex52f.F to do >>> the test. There are 6 matrices and right-hand-side vectors. All these >>> matrices and rhs are from my reactive transport simulation. Results will be >>> quite different depending on which one you use to do factorization. Results >>> will also be different if you run with different options. My code is >>> similar to the First or the Second test below. When the matrix is well >>> conditioned, it works fine. But if the initial matrix is well conditioned, >>> it likely to crash when the matrix become ill-conditioned. Since most of my >>> case are well conditioned so I didn't detect the problem before. This case >>> is a special one. >>> >>> >>> How can I avoid this problem? Shall I redo factorization? Can PETSc >>> automatically detect this prolbem or is there any option available to do >>> this? >>> >>> All the data and test code (modified ex52f) can be found via the dropbox >>> link below. >>> >>> *https://www.dropbox.com/s/4al1a60creogd8m/petsc-superlu-test.tar.gz?dl=0 >>> * >>> >>> >>> Summary of my test is shown below. >>> >>> First, use the Matrix 1 to setup KSP solver and factorization, then >>> solve 168 to 172 >>> >>> mpiexec.hydra -n 1 ./ex52f -f0 >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin >>> -rhs >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin >>> -loop_matrices flow_check -loop_folder >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -pc_type lu >>> -pc_factor_mat_solver_package superlu_dist >>> >>> Norm of error 3.8815E-11 iterations 1 >>> -->Test for matrix 168 >>> Norm of error 4.2307E-01 iterations 32 >>> -->Test for matrix 169 >>> Norm of error 3.0528E-01 iterations 32 >>> -->Test for matrix 170 >>> Norm of error 3.1177E-01 iterations 32 >>> -->Test for matrix 171 >>> Norm of error 3.2793E-01 iterations 32 >>> -->Test for matrix 172 >>> Norm of error 3.1251E-01 iterations 31 >>> >>> Second, use the Matrix 1 to setup KSP solver and factorization using the >>> implemented SuperLU relative codes. I thought this will generate the same >>> results as the First test, but it actually not. >>> >>> mpiexec.hydra -n 1 ./ex52f -f0 >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin >>> -rhs >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin >>> -loop_matrices flow_check -loop_folder >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -superlu_default >>> >>> Norm of error 2.2632E-12 iterations 1 >>> -->Test for matrix 168 >>> Norm of error 1.0817E+04 iterations 1 >>> -->Test for matrix 169 >>> Norm of error 1.0786E+04 iterations 1 >>> -->Test for matrix 170 >>> Norm of error 1.0792E+04 iterations 1 >>> -->Test for matrix 171 >>> Norm of error 1.0792E+04 iterations 1 >>> -->Test for matrix 172 >>> Norm of error 1.0792E+04 iterations 1 >>> >>> >>> Third, use the Matrix 168 to setup KSP solver and factorization, then >>> solve 168 to 172 >>> >>> mpiexec.hydra -n 1 ./ex52f -f0 >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin >>> -rhs >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin >>> -loop_matrices flow_check -loop_folder >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -pc_type lu >>> -pc_factor_mat_solver_package superlu_dist >>> >>> Norm of error 9.5528E-10 iterations 1 >>> -->Test for matrix 168 >>> Norm of error 9.4945E-10 iterations 1 >>> -->Test for matrix 169 >>> Norm of error 6.4279E-10 iterations 1 >>> -->Test for matrix 170 >>> Norm of error 7.4633E-10 iterations 1 >>> -->Test for matrix 171 >>> Norm of error 7.4863E-10 iterations 1 >>> -->Test for matrix 172 >>> Norm of error 8.9701E-10 iterations 1 >>> >>> Fourth, use the Matrix 168 to setup KSP solver and factorization using >>> the implemented SuperLU relative codes. I thought this will generate the >>> same results as the Third test, but it actually not. >>> >>> mpiexec.hydra -n 1 ./ex52f -f0 >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin >>> -rhs >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin >>> -loop_matrices flow_check -loop_folder >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin -superlu_default >>> >>> Norm of error 3.7017E-11 iterations 1 >>> -->Test for matrix 168 >>> Norm of error 3.6420E-11 iterations 1 >>> -->Test for matrix 169 >>> Norm of error 3.7184E-11 iterations 1 >>> -->Test for matrix 170 >>> Norm of error 3.6847E-11 iterations 1 >>> -->Test for matrix 171 >>> Norm of error 3.7883E-11 iterations 1 >>> -->Test for matrix 172 >>> Norm of error 3.8805E-11 iterations 1 >>> >>> Thanks very much, >>> >>> Danyang >>> >>> On 15-12-03 01:59 PM, Hong wrote: >>> >>> Danyang : >>> Further testing a_flow_check_168.bin, >>> ./ex10 -f0 /Users/Hong/Downloads/matrix_and_rhs_bin/a_flow_check_168.bin >>> -rhs /Users/Hong/Downloads/matrix_and_rhs_bin/x_flow_check_168.bin -pc_type >>> lu -pc_factor_mat_solver_package superlu -ksp_monitor_true_residual >>> -mat_superlu_conditionnumber >>> Recip. condition number = 1.610480e-12 >>> 0 KSP preconditioned resid norm 6.873340313547e+09 true resid norm >>> 7.295020990196e+03 ||r(i)||/||b|| 1.000000000000e+00 >>> 1 KSP preconditioned resid norm 2.051833296449e-02 true resid norm >>> 2.976859070118e-02 ||r(i)||/||b|| 4.080672384793e-06 >>> Number of iterations = 1 >>> Residual norm 0.0297686 >>> >>> condition number of this matrix = 1/1.610480e-12 = 1.e+12, >>> i.e., this matrix is ill-conditioned. >>> >>> Hong >>> >>> >>> Hi Hong, >>>> >>>> The binary format of matrix, rhs and solution can be downloaded via the >>>> link below. >>>> >>>> https://www.dropbox.com/s/cl3gfi0s0kjlktf/matrix_and_rhs_bin.tar.gz?dl=0 >>>> >>>> Thanks, >>>> >>>> Danyang >>>> >>>> >>>> On 15-12-03 10:50 AM, Hong wrote: >>>> >>>> Danyang: >>>> >>>>> >>>>> >>>>> To my surprising, solutions from SuperLU at timestep 29 seems not >>>>> correct for the first 4 Newton iterations, but the solutions from iteration >>>>> solver and MUMPS are correct. >>>>> >>>>> Please find all the matrices, rhs and solutions at timestep 29 via the >>>>> link below. The data is a bit large so that I just share it through >>>>> Dropbox. A piece of matlab code to read these data and then computer the >>>>> norm has also been attached. >>>>> * >>>>> https://www.dropbox.com/s/rr8ueysgflmxs7h/results-check.tar.gz?dl=0 >>>>> * >>>>> >>>> >>>> Can you send us matrix in petsc binary format? >>>> >>>> e.g., call MatView(M, PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD)) >>>> or '-ksp_view_mat binary' >>>> >>>> Hong >>>> >>>>> >>>>> >>>>> Below is a summary of the norm from the three solvers at timestep 29, >>>>> newton iteration 1 to 5. >>>>> >>>>> Timestep 29 >>>>> Norm of residual seq 1.661321e-09, superlu 1.657103e+04, mumps >>>>> 3.731225e-11 >>>>> Norm of residual seq 1.753079e-09, superlu 6.675467e+02, mumps >>>>> 1.509919e-13 >>>>> Norm of residual seq 4.914971e-10, superlu 1.236362e-01, mumps >>>>> 2.139303e-17 >>>>> Norm of residual seq 3.532769e-10, superlu 1.304670e-04, mumps >>>>> 5.387000e-20 >>>>> Norm of residual seq 3.885629e-10, superlu 2.754876e-07, mumps >>>>> 4.108675e-21 >>>>> >>>>> Would anybody please check if SuperLU can solve these matrices? >>>>> Another possibility is that something is wrong in my own code. But so far, >>>>> I cannot find any problem in my code since the same code works fine if I >>>>> using iterative solver or direct solver MUMPS. But for other cases I have >>>>> tested, all these solvers work fine. >>>>> >>>>> Please let me know if I did not write down the problem clearly. >>>>> >>>>> Thanks, >>>>> >>>>> Danyang >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dsu at eos.ubc.ca Tue Dec 8 22:27:13 2015 From: dsu at eos.ubc.ca (Danyang Su) Date: Tue, 08 Dec 2015 20:27:13 -0800 Subject: [petsc-users] SuperLU convergence problem (More test) In-Reply-To: References: <565F70E8.6030300@eos.ubc.ca> <5660A9CF.1030809@eos.ubc.ca> <56637D26.2080907@eos.ubc.ca> <5665DDDA.405@eos.ubc.ca> <566793E0.6060907@eos.ubc.ca> Message-ID: <5667ADA1.1020709@eos.ubc.ca> Hi Hong, Thanks for checking this. A mechanical model was added at the time when the solver failed, causing some problem. We need to improve this part in the code. Thanks again and best wishes, Danyang On 15-12-08 08:10 PM, Hong wrote: > Danyang : > Your matrices are ill-conditioned, numerically singular with > Recip. condition number = 6.000846e-16 > Recip. condition number = 2.256434e-27 > Recip. condition number = 1.256452e-18 > i.e., condition numbers = O(1.e16 - 1.e27), there is no accuracy in > computed solution. > > I checked your matrix 168 - 172, got Recip. condition number > = 1.548816e-12. > > You need check your model to understand why the matrices are so > ill-conditioned. > > Hong > > Hi Hong, > > Sorry to bother you again. The modified code works much better > than before using both superlu or mumps. However, it still > encounters failure. The case is similar with the previous one, > ill-conditioned matrices. > > The code crashed after a long time simulation if I use > superlu_dist, but will not fail if use superlu. I restart the > simulation before the time it crashes and can reproduce the > following error > > timestep: 22 time: 1.750E+04 years delt: 2.500E+00 years > iter: 1 max.sia: 5.053E-03 tol.sia: 1.000E-01 > Newton Iteration Convergence Summary: > Newton maximum maximum solver > iteration updatePa updateTemp residual iterations > maxvolpa maxvoltemp nexvolpa nexvoltemp > 1 0.1531E+08 0.1755E+04 0.6920E-05 1 > 5585 4402 5814 5814 > > *** Error in `../program_test': malloc(): memory corruption: > 0x0000000003a70d50 *** > Program received signal SIGABRT: Process abort signal. > Backtrace for this error: > > The solver failed at timestep 22, Newton iteration 2. I exported > the matrices at timestep 1 (matrix 1) and timestep 22 (matrix 140 > and 141). Matrix 141 is where it failed. The three matrices here > are not ill-conditioned form the estimated value. > > I did the same using the new modified ex52f code and found pretty > different results for matrix 141. The norm by superlu is much > acceptable than superlu_dist. In this test, memory corruption was > not detected. The codes and example data can be download from the > link below. > > https://www.dropbox.com/s/i1ls0bg0vt7gu0v/petsc-superlu-test2.tar.gz?dl=0 > > > ****************More test on matrix_and_rhs_bin2******************* > mpiexec.hydra -n 1 ./ex52f -f0 > ./matrix_and_rhs_bin2/a_flow_check_1.bin -rhs > ./matrix_and_rhs_bin2/b_flow_check_1.bin -loop_matrices flow_check > -loop_folder ./matrix_and_rhs_bin2 -matrix_index_start 140 > -matrix_index_end 141 -pc_type lu -pc_factor_mat_solver_package > superlu -ksp_monitor_true_residual -mat_superlu_conditionnumber > -->loac matrix a > -->load rhs b > size l,m,n,mm 90000 90000 90000 90000 > Recip. condition number = 6.000846e-16 > 0 KSP preconditioned resid norm 1.146871454377e+08 true resid > norm 4.711091037809e+03 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.071118508260e-06 true resid > norm 3.363767171515e-08 ||r(i)||/||b|| 7.140102249181e-12 > Norm of error 3.3638E-08 iterations 1 > -->Test for matrix 140 > Recip. condition number = 2.256434e-27 > 0 KSP preconditioned resid norm 2.084372893355e+14 true resid > norm 4.711091037809e+03 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.689629276419e+00 true resid > norm 1.037236635337e-01 ||r(i)||/||b|| 2.201690918330e-05 > Norm of error 1.0372E-01 iterations 1 > -->Test for matrix 141 > Recip. condition number = 1.256452e-18 > 0 KSP preconditioned resid norm 1.055488964519e+08 true resid > norm 4.711091037809e+03 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.998827511681e-04 true resid > norm 4.805214542776e-04 ||r(i)||/||b|| 1.019979130994e-07 > Norm of error 4.8052E-04 iterations 1 > --> End of test, bye > > > mpiexec.hydra -n 1 ./ex52f -f0 > ./matrix_and_rhs_bin2/a_flow_check_1.bin -rhs > ./matrix_and_rhs_bin2/b_flow_check_1.bin -loop_matrices flow_check > -loop_folder ./matrix_and_rhs_bin2 -matrix_index_start 140 > -matrix_index_end 141 -pc_type lu -pc_factor_mat_solver_package > superlu_dist > -->loac matrix a > -->load rhs b > size l,m,n,mm 90000 90000 90000 90000 > Norm of error 3.6752E-08 iterations 1 > -->Test for matrix 140 > Norm of error 1.6335E-01 iterations 1 > -->Test for matrix 141 > Norm of error 3.4345E+01 iterations 1 > --> End of test, bye > > Thanks, > > Danyang > > On 15-12-07 12:01 PM, Hong wrote: >> Danyang: >> Add 'call MatSetFromOptions(A,ierr)' to your code. >> Attached below is ex52f.F modified from your ex52f.F to be >> compatible with petsc-dev. >> >> Hong >> >> Hello Hong, >> >> Thanks for the quick reply and the option >> "-mat_superlu_dist_fact SamePattern" works like a charm, if I >> use this option from the command line. >> >> How can I add this option as the default. I tried using >> PetscOptionsInsertString("-mat_superlu_dist_fact >> SamePattern",ierr) in my code but this does not work. >> >> Thanks, >> >> Danyang >> >> >> On 15-12-07 10:42 AM, Hong wrote: >>> Danyang : >>> >>> Adding '-mat_superlu_dist_fact SamePattern' fixed the >>> problem. Below is how I figured it out. >>> >>> 1. Reading ex52f.F, I see '-superlu_default' = >>> '-pc_factor_mat_solver_package superlu_dist', the later >>> enables runtime options for other packages. I use >>> superlu_dist-4.2 and superlu-4.1 for the tests below. >>> >>> 2. Use the Matrix 168 to setup KSP solver and factorization, >>> all packages, petsc, superlu_dist and mumps give same >>> correct results: >>> >>> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs >>> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices >>> flow_check -loop_folder matrix_and_rhs_bin -pc_type lu >>> -pc_factor_mat_solver_package petsc >>> -->loac matrix a >>> -->load rhs b >>> size l,m,n,mm 90000 90000 90000 90000 >>> Norm of error 7.7308E-11 iterations 1 >>> -->Test for matrix 168 >>> .. >>> -->Test for matrix 172 >>> Norm of error 3.8461E-11 iterations 1 >>> >>> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs >>> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices >>> flow_check -loop_folder matrix_and_rhs_bin -pc_type lu >>> -pc_factor_mat_solver_package superlu_dist >>> Norm of error 9.4073E-11 iterations 1 >>> -->Test for matrix 168 >>> ... >>> -->Test for matrix 172 >>> Norm of error 3.8187E-11 iterations 1 >>> >>> 3. Use superlu, I get >>> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs >>> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices >>> flow_check -loop_folder matrix_and_rhs_bin -pc_type lu >>> -pc_factor_mat_solver_package superlu >>> Norm of error 1.0191E-06 iterations 1 >>> -->Test for matrix 168 >>> ... >>> -->Test for matrix 172 >>> Norm of error 9.7858E-07 iterations 1 >>> >>> Replacing default DiagPivotThresh: 1. to 0.0, I get same >>> solutions as other packages: >>> >>> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_168.bin -rhs >>> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices >>> flow_check -loop_folder matrix_and_rhs_bin -pc_type lu >>> -pc_factor_mat_solver_package superlu >>> -mat_superlu_diagpivotthresh 0.0 >>> >>> Norm of error 8.3614E-11 iterations 1 >>> -->Test for matrix 168 >>> ... >>> -->Test for matrix 172 >>> Norm of error 3.7098E-11 iterations 1 >>> >>> 4. >>> using '-mat_view ascii::ascii_info', I found that >>> a_flow_check_1.bin and a_flow_check_168.bin seem have same >>> structure: >>> >>> -->loac matrix a >>> Mat Object: 1 MPI processes >>> type: seqaij >>> rows=90000, cols=90000 >>> total: nonzeros=895600, allocated nonzeros=895600 >>> total number of mallocs used during MatSetValues calls =0 >>> using I-node routines: found 45000 nodes, limit used is 5 >>> >>> 5. >>> Using a_flow_check_1.bin, I am able to reproduce the error >>> you reported: all packages give correct results except >>> superlu_dist: >>> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_1.bin -rhs >>> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices >>> flow_check -loop_folder matrix_and_rhs_bin -pc_type lu >>> -pc_factor_mat_solver_package superlu_dist >>> Norm of error 2.5970E-12 iterations 1 >>> -->Test for matrix 168 >>> Norm of error 1.3936E-01 iterations 34 >>> -->Test for matrix 169 >>> >>> I guess the error might come from reuse of matrix factor. >>> Replacing default >>> -mat_superlu_dist_fact with >>> -mat_superlu_dist_fact SamePattern, I get >>> >>> ./ex52f -f0 matrix_and_rhs_bin/a_flow_check_1.bin -rhs >>> matrix_and_rhs_bin/b_flow_check_168.bin -loop_matrices >>> flow_check -loop_folder matrix_and_rhs_bin -pc_type lu >>> -pc_factor_mat_solver_package superlu_dist >>> -mat_superlu_dist_fact SamePattern >>> >>> Norm of error 2.5970E-12 iterations 1 >>> -->Test for matrix 168 >>> Norm of error 9.4073E-11 iterations 1 >>> -->Test for matrix 169 >>> Norm of error 6.4303E-11 iterations 1 >>> -->Test for matrix 170 >>> Norm of error 7.4327E-11 iterations 1 >>> -->Test for matrix 171 >>> Norm of error 5.4162E-11 iterations 1 >>> -->Test for matrix 172 >>> Norm of error 3.4440E-11 iterations 1 >>> --> End of test, bye >>> >>> Sherry may tell you why SamePattern_SameRowPerm cause the >>> difference here. >>> Best on the above experiments, I would set following as default >>> '-mat_superlu_diagpivotthresh 0.0' in petsc/superlu interface. >>> '-mat_superlu_dist_fact SamePattern' in petsc/superlu_dist >>> interface. >>> >>> Hong >>> >>> Hi Hong, >>> >>> I did more test today and finally found that the >>> solution accuracy depends on the initial (first) matrix >>> quality. I modified the ex52f.F to do the test. There >>> are 6 matrices and right-hand-side vectors. All these >>> matrices and rhs are from my reactive transport >>> simulation. Results will be quite different depending on >>> which one you use to do factorization. Results will also >>> be different if you run with different options. My code >>> is similar to the First or the Second test below. When >>> the matrix is well conditioned, it works fine. But if >>> the initial matrix is well conditioned, it likely to >>> crash when the matrix become ill-conditioned. Since most >>> of my case are well conditioned so I didn't detect the >>> problem before. This case is a special one. >>> >>> >>> How can I avoid this problem? Shall I redo >>> factorization? Can PETSc automatically detect this >>> prolbem or is there any option available to do this? >>> >>> All the data and test code (modified ex52f) can be found >>> via the dropbox link below. >>> _ >>> __https://www.dropbox.com/s/4al1a60creogd8m/petsc-superlu-test.tar.gz?dl=0_ >>> >>> >>> Summary of my test is shown below. >>> >>> First, use the Matrix 1 to setup KSP solver and >>> factorization, then solve 168 to 172 >>> >>> mpiexec.hydra -n 1 ./ex52f -f0 >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin >>> -rhs >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin >>> -loop_matrices flow_check -loop_folder >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin >>> -pc_type lu -pc_factor_mat_solver_package superlu_dist >>> >>> Norm of error 3.8815E-11 iterations 1 >>> -->Test for matrix 168 >>> Norm of error 4.2307E-01 iterations 32 >>> -->Test for matrix 169 >>> Norm of error 3.0528E-01 iterations 32 >>> -->Test for matrix 170 >>> Norm of error 3.1177E-01 iterations 32 >>> -->Test for matrix 171 >>> Norm of error 3.2793E-01 iterations 32 >>> -->Test for matrix 172 >>> Norm of error 3.1251E-01 iterations 31 >>> >>> Second, use the Matrix 1 to setup KSP solver and >>> factorization using the implemented SuperLU relative >>> codes. I thought this will generate the same results as >>> the First test, but it actually not. >>> >>> mpiexec.hydra -n 1 ./ex52f -f0 >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_1.bin >>> -rhs >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_1.bin >>> -loop_matrices flow_check -loop_folder >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin >>> -superlu_default >>> >>> Norm of error 2.2632E-12 iterations 1 >>> -->Test for matrix 168 >>> Norm of error 1.0817E+04 iterations 1 >>> -->Test for matrix 169 >>> Norm of error 1.0786E+04 iterations 1 >>> -->Test for matrix 170 >>> Norm of error 1.0792E+04 iterations 1 >>> -->Test for matrix 171 >>> Norm of error 1.0792E+04 iterations 1 >>> -->Test for matrix 172 >>> Norm of error 1.0792E+04 iterations 1 >>> >>> >>> Third, use the Matrix 168 to setup KSP solver and >>> factorization, then solve 168 to 172 >>> >>> mpiexec.hydra -n 1 ./ex52f -f0 >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin >>> -rhs >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin >>> -loop_matrices flow_check -loop_folder >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin >>> -pc_type lu -pc_factor_mat_solver_package superlu_dist >>> >>> Norm of error 9.5528E-10 iterations 1 >>> -->Test for matrix 168 >>> Norm of error 9.4945E-10 iterations 1 >>> -->Test for matrix 169 >>> Norm of error 6.4279E-10 iterations 1 >>> -->Test for matrix 170 >>> Norm of error 7.4633E-10 iterations 1 >>> -->Test for matrix 171 >>> Norm of error 7.4863E-10 iterations 1 >>> -->Test for matrix 172 >>> Norm of error 8.9701E-10 iterations 1 >>> >>> Fourth, use the Matrix 168 to setup KSP solver and >>> factorization using the implemented SuperLU relative >>> codes. I thought this will generate the same results as >>> the Third test, but it actually not. >>> >>> mpiexec.hydra -n 1 ./ex52f -f0 >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/a_flow_check_168.bin >>> -rhs >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin/b_flow_check_168.bin >>> -loop_matrices flow_check -loop_folder >>> /home/dsu/work/petsc-superlu-test/matrix_and_rhs_bin >>> -superlu_default >>> >>> Norm of error 3.7017E-11 iterations 1 >>> -->Test for matrix 168 >>> Norm of error 3.6420E-11 iterations 1 >>> -->Test for matrix 169 >>> Norm of error 3.7184E-11 iterations 1 >>> -->Test for matrix 170 >>> Norm of error 3.6847E-11 iterations 1 >>> -->Test for matrix 171 >>> Norm of error 3.7883E-11 iterations 1 >>> -->Test for matrix 172 >>> Norm of error 3.8805E-11 iterations 1 >>> >>> Thanks very much, >>> >>> Danyang >>> >>> On 15-12-03 01:59 PM, Hong wrote: >>>> Danyang : >>>> Further testing a_flow_check_168.bin, >>>> ./ex10 -f0 >>>> /Users/Hong/Downloads/matrix_and_rhs_bin/a_flow_check_168.bin >>>> -rhs >>>> /Users/Hong/Downloads/matrix_and_rhs_bin/x_flow_check_168.bin >>>> -pc_type lu -pc_factor_mat_solver_package superlu >>>> -ksp_monitor_true_residual -mat_superlu_conditionnumber >>>> Recip. condition number = 1.610480e-12 >>>> 0 KSP preconditioned resid norm 6.873340313547e+09 true >>>> resid norm 7.295020990196e+03 ||r(i)||/||b|| >>>> 1.000000000000e+00 >>>> 1 KSP preconditioned resid norm 2.051833296449e-02 true >>>> resid norm 2.976859070118e-02 ||r(i)||/||b|| >>>> 4.080672384793e-06 >>>> Number of iterations = 1 >>>> Residual norm 0.0297686 >>>> >>>> condition number of this matrix = 1/1.610480e-12 = 1.e+12, >>>> i.e., this matrix is ill-conditioned. >>>> >>>> Hong >>>> >>>> >>>> Hi Hong, >>>> >>>> The binary format of matrix, rhs and solution can >>>> be downloaded via the link below. >>>> >>>> https://www.dropbox.com/s/cl3gfi0s0kjlktf/matrix_and_rhs_bin.tar.gz?dl=0 >>>> >>>> Thanks, >>>> >>>> Danyang >>>> >>>> >>>> On 15-12-03 10:50 AM, Hong wrote: >>>>> Danyang: >>>>> >>>>> >>>>> >>>>> To my surprising, solutions from SuperLU at >>>>> timestep 29 seems not correct for the first 4 >>>>> Newton iterations, but the solutions from >>>>> iteration solver and MUMPS are correct. >>>>> >>>>> Please find all the matrices, rhs and >>>>> solutions at timestep 29 via the link below. >>>>> The data is a bit large so that I just share >>>>> it through Dropbox. A piece of matlab code to >>>>> read these data and then computer the norm has >>>>> also been attached. >>>>> _https://www.dropbox.com/s/rr8ueysgflmxs7h/results-check.tar.gz?dl=0_ >>>>> >>>>> >>>>> Can you send us matrix in petsc binary format? >>>>> >>>>> e.g., call MatView(M, >>>>> PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD)) >>>>> or '-ksp_view_mat binary' >>>>> >>>>> Hong >>>>> >>>>> >>>>> >>>>> Below is a summary of the norm from the three >>>>> solvers at timestep 29, newton iteration 1 to 5. >>>>> >>>>> Timestep 29 >>>>> Norm of residual seq 1.661321e-09, superlu >>>>> 1.657103e+04, mumps 3.731225e-11 >>>>> Norm of residual seq 1.753079e-09, superlu >>>>> 6.675467e+02, mumps 1.509919e-13 >>>>> Norm of residual seq 4.914971e-10, superlu >>>>> 1.236362e-01, mumps 2.139303e-17 >>>>> Norm of residual seq 3.532769e-10, superlu >>>>> 1.304670e-04, mumps 5.387000e-20 >>>>> Norm of residual seq 3.885629e-10, superlu >>>>> 2.754876e-07, mumps 4.108675e-21 >>>>> >>>>> Would anybody please check if SuperLU can >>>>> solve these matrices? Another possibility is >>>>> that something is wrong in my own code. But so >>>>> far, I cannot find any problem in my code >>>>> since the same code works fine if I using >>>>> iterative solver or direct solver MUMPS. But >>>>> for other cases I have tested, all these >>>>> solvers work fine. >>>>> >>>>> Please let me know if I did not write down the >>>>> problem clearly. >>>>> >>>>> Thanks, >>>>> >>>>> Danyang >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From orxan.shibli at gmail.com Wed Dec 9 13:00:22 2015 From: orxan.shibli at gmail.com (Orxan Shibliyev) Date: Wed, 9 Dec 2015 21:00:22 +0200 Subject: [petsc-users] Conflicting KSPSetOperators functions Message-ID: Hi I uploaded my code to cluster which has Petsc 3.4.4. While compiling the code it gave error: too few arguments to function ?PetscErrorCode KSPSetOperators(KSP, Mat, Mat, MatStructure) On my computer I use latest Petsc version and I use only three arguments for the function and it works well. Is the problem due to version difference? Thanks, Orhan Shibliyev -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Dec 9 13:11:40 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 9 Dec 2015 13:11:40 -0600 Subject: [petsc-users] Conflicting KSPSetOperators functions In-Reply-To: References: Message-ID: <14C0862D-D82F-4123-81BA-790B45DA5FF9@mcs.anl.gov> Install the latest version of PETSc yourself and do not use that ancient and out-dated version that is installed there (that would be going backwards). PETSc is designed to be installable by any user. Barry > On Dec 9, 2015, at 1:00 PM, Orxan Shibliyev wrote: > > Hi > > I uploaded my code to cluster which has Petsc 3.4.4. While compiling the code it gave error: > > too few arguments to function ?PetscErrorCode KSPSetOperators(KSP, Mat, Mat, MatStructure) > > On my computer I use latest Petsc version and I use only three arguments for the function and it works well. Is the problem due to version difference? > > Thanks, > Orhan Shibliyev From bsmith at mcs.anl.gov Wed Dec 9 16:18:52 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 9 Dec 2015 16:18:52 -0600 Subject: [petsc-users] Is it still worth switching to PETSc if I can't write a Jacobian for my problem? In-Reply-To: References: <7A5CAB4B-2812-4AD6-B0C9-B5ECD703AD3B@mcs.anl.gov> Message-ID: <533442B3-DDC3-4C5F-9BC3-A21CE78A9BC0@mcs.anl.gov> I prefer the actual code, not the mathematics or the explanation > On Dec 9, 2015, at 3:42 PM, Brian Merchant wrote: > > Hi Barry, > > > Could send an example of your "rhs" function; not a totally trivial example > > Sure thing! Although, did you check out the exam I tried to build up in this stackexchange question, along with a picture: http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a > > I ask because that's probably the best I can do without using as little math as possible. > > Otherwise, what I'll do is take a couple of days to carefully look at my work, and write up a non-trivial example of a difficult-to-differentiate RHS, that still is a simplification of the whole mess -- expect a one or two page PDF? > > Kind regards, > Brian > > On Mon, Dec 7, 2015 at 12:45 PM, Barry Smith wrote: > > Brian, > > Could send an example of your "rhs" function; not a totally trivial example > > Barry > > > On Dec 7, 2015, at 11:21 AM, Brian Merchant wrote: > > > > Hi all, > > > > I am considering using petsc4py instead of scipy.integrate.odeint (which is a wrapper for Fortran solvers) for a problem involving the solution of a system of ODEs. The problem has the potential to be stiff. Writing down its Jacobian is very hard. > > > > So far, I have been able to produce reasonable speed gains by writing the RHS functions in "something like C" (using either numba or Cython). I'd like to get even more performance out, hence my consideration of PETSc. > > > > Due to the large number of equations involved, it is already tedious to think about writing down a Jacobian. Even worse though, is that some of the functions governing a particular interaction do not have neat analytical forms (let alone whether or not their derivatives have neat analytical forms), so we might have a mess of piecewise functions needed to approximate them if we were to go about still trying to produce a Jacobian... > > > > All the toy examples I see of PETSc time stepping problems have Jacobians defined, so I wonder if I would even get a speed gain going from switching to it, if perhaps one of the reasons why I have a high computational cost is due to not being able to provide a Jacobian function? > > > > I described the sort of problem I am working with in more detail in this scicomp.stackexchange question, which is where most of this question is copied from, except it also comes with a toy version of the problem I am dealing with: http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a > > > > All your advice would be most helpful :) > > > > Kind regards,Brian > > > > From bhmerchant at gmail.com Wed Dec 9 16:50:09 2015 From: bhmerchant at gmail.com (Brian Merchant) Date: Wed, 9 Dec 2015 14:50:09 -0800 Subject: [petsc-users] Is it still worth switching to PETSc if I can't write a Jacobian for my problem? In-Reply-To: <533442B3-DDC3-4C5F-9BC3-A21CE78A9BC0@mcs.anl.gov> References: <7A5CAB4B-2812-4AD6-B0C9-B5ECD703AD3B@mcs.anl.gov> <533442B3-DDC3-4C5F-9BC3-A21CE78A9BC0@mcs.anl.gov> Message-ID: Alright, I can get that for you! Thank you very much for your time! On Wed, Dec 9, 2015 at 2:18 PM, Barry Smith wrote: > > I prefer the actual code, not the mathematics or the explanation > > > On Dec 9, 2015, at 3:42 PM, Brian Merchant wrote: > > > > Hi Barry, > > > > > Could send an example of your "rhs" function; not a totally trivial > example > > > > Sure thing! Although, did you check out the exam I tried to build up in > this stackexchange question, along with a picture: > http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a > > > > I ask because that's probably the best I can do without using as little > math as possible. > > > > Otherwise, what I'll do is take a couple of days to carefully look at my > work, and write up a non-trivial example of a difficult-to-differentiate > RHS, that still is a simplification of the whole mess -- expect a one or > two page PDF? > > > > Kind regards, > > Brian > > > > On Mon, Dec 7, 2015 at 12:45 PM, Barry Smith wrote: > > > > Brian, > > > > Could send an example of your "rhs" function; not a totally trivial > example > > > > Barry > > > > > On Dec 7, 2015, at 11:21 AM, Brian Merchant > wrote: > > > > > > Hi all, > > > > > > I am considering using petsc4py instead of scipy.integrate.odeint > (which is a wrapper for Fortran solvers) for a problem involving the > solution of a system of ODEs. The problem has the potential to be stiff. > Writing down its Jacobian is very hard. > > > > > > So far, I have been able to produce reasonable speed gains by writing > the RHS functions in "something like C" (using either numba or Cython). I'd > like to get even more performance out, hence my consideration of PETSc. > > > > > > Due to the large number of equations involved, it is already tedious > to think about writing down a Jacobian. Even worse though, is that some of > the functions governing a particular interaction do not have neat > analytical forms (let alone whether or not their derivatives have neat > analytical forms), so we might have a mess of piecewise functions needed to > approximate them if we were to go about still trying to produce a > Jacobian... > > > > > > All the toy examples I see of PETSc time stepping problems have > Jacobians defined, so I wonder if I would even get a speed gain going from > switching to it, if perhaps one of the reasons why I have a high > computational cost is due to not being able to provide a Jacobian function? > > > > > > I described the sort of problem I am working with in more detail in > this scicomp.stackexchange question, which is where most of this question > is copied from, except it also comes with a toy version of the problem I am > dealing with: > http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a > > > > > > All your advice would be most helpful :) > > > > > > Kind regards,Brian > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlmackie862 at gmail.com Wed Dec 9 19:28:13 2015 From: rlmackie862 at gmail.com (Randall Mackie) Date: Wed, 9 Dec 2015 17:28:13 -0800 Subject: [petsc-users] possible to save direct solver factorization for later use Message-ID: <1622E0B5-EEA1-4005-9C24-7E2956CBC20C@gmail.com> Is it possible to save a direct solver factorization (like Mumps) for use in later parts of a code? In general, I?m thinking of a scenario like this: mat A - do lots of solves using Mumps mat B - do lots of solves using Mumps do other stuff mat A - do lots of solves using Mumps mat B - do lots of solves using Mumps If possible, I?d rather only do the numerical factorizations once, and then reuse as required. Do KSPView and KSPLoad to/from binary files save this information? Thanks, Randy From jed at jedbrown.org Wed Dec 9 20:17:11 2015 From: jed at jedbrown.org (Jed Brown) Date: Wed, 09 Dec 2015 19:17:11 -0700 Subject: [petsc-users] possible to save direct solver factorization for later use In-Reply-To: <1622E0B5-EEA1-4005-9C24-7E2956CBC20C@gmail.com> References: <1622E0B5-EEA1-4005-9C24-7E2956CBC20C@gmail.com> Message-ID: <87oadznj2w.fsf@jedbrown.org> Randall Mackie writes: > Is it possible to save a direct solver factorization (like Mumps) for use in later parts of a code? > > In general, I?m thinking of a scenario like this: > > mat A - do lots of solves using Mumps > > mat B - do lots of solves using Mumps > > > do other stuff > > mat A - do lots of solves using Mumps > > mat B - do lots of solves using Mumps > > > If possible, I?d rather only do the numerical factorizations once, and then reuse as required. Use different KSPs for each solve. > Do KSPView and KSPLoad to/from binary files save this information? No, and saving factors to a file likely wouldn't even be useful unless factorization takes on the order of an hour or more. If they are too big to fit in DRAM simultaneously (and you can't just use more nodes), your best bet is to have some local memory and let the OS swap them out (it'll automatically do the right thing in this instance). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From tobystone at binvestmenttld.net Thu Dec 10 02:16:51 2015 From: tobystone at binvestmenttld.net (Toby Stone) Date: Thu, 10 Dec 2015 09:16:51 +0100 Subject: [petsc-users] Proposal Message-ID: <1494363773064803083615559@s18796407> Good day. I am Toby Stone. I am a wealth manager from Barah Investment Limited, Middle East. I have enclosed a business proposal to this email that I think will be of interest to you. Please kindly go through the proposal and get back to me. If for any reason you are unable to view the pdf file, let me know and I will resend either in plain text or in a different format. Regards, Toby Stone. Barah Investments Ltd Telefax: +2033991374 Email: tobystone at binvestmentltd.net -------------- next part -------------- A non-text attachment was scrubbed... Name: Proposal.pdf Type: application/pdf Size: 410175 bytes Desc: not available URL: From myriam.peyrounette at imft.fr Thu Dec 10 11:06:24 2015 From: myriam.peyrounette at imft.fr ('PEYROUNETTE Myriam') Date: Thu, 10 Dec 2015 18:06:24 +0100 Subject: [petsc-users] Build a PETSc matrix with "ghost lines" ? Message-ID: <5669B110.4010604@imft.fr> Hi all, I am partitioning an unstructured network in order to parallelize my code. To communicate some node data between processors, I have to deal with ghost values. I know PETSc allows its vectors to carry ghost values thanks to VecCreateGhost() or VecMPISetGhost() functions. In the same spirit, are there similar functions to create matrix with ghost lines ? To give an example : I have a matrix containing the coordinates of each node and I would like to access the coordinates of a neighboring node, even if it is located on a different processor. Thanks, Myriam From bsmith at mcs.anl.gov Thu Dec 10 11:14:22 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 10 Dec 2015 11:14:22 -0600 Subject: [petsc-users] Build a PETSc matrix with "ghost lines" ? In-Reply-To: <5669B110.4010604@imft.fr> References: <5669B110.4010604@imft.fr> Message-ID: <4974C243-50AC-4645-877C-56059A8B1CF8@mcs.anl.gov> > On Dec 10, 2015, at 11:06 AM, PEYROUNETTE Myriam wrote: > > Hi all, > > I am partitioning an unstructured network in order to parallelize my code. To communicate some node data between processors, I have to deal with ghost values. I know PETSc allows its vectors to carry ghost values thanks to VecCreateGhost() or VecMPISetGhost() functions. In the same spirit, are there similar functions to create matrix with ghost lines ? > > To give an example : I have a matrix containing the coordinates of each node and I would like to access the coordinates of a neighboring node, even if it is located on a different processor. You would never use a Mat to store coordinates. Use a Vec (with a bs of 2 or 3 if in 2 or 3 dimensions) Barry > > Thanks, > Myriam From bhmerchant at gmail.com Thu Dec 10 12:52:49 2015 From: bhmerchant at gmail.com (Brian Merchant) Date: Thu, 10 Dec 2015 10:52:49 -0800 Subject: [petsc-users] Is it still worth switching to PETSc if I can't write a Jacobian for my problem? In-Reply-To: <533442B3-DDC3-4C5F-9BC3-A21CE78A9BC0@mcs.anl.gov> References: <7A5CAB4B-2812-4AD6-B0C9-B5ECD703AD3B@mcs.anl.gov> <533442B3-DDC3-4C5F-9BC3-A21CE78A9BC0@mcs.anl.gov> Message-ID: Hi Barry, Here's some non-trivial example code: https://gist.github.com/bmer/2af429f88b0b696648a8 I have still made some simplifications by removing some phase variables, expanding on variable names in general, and so on. The rhs function itself is defined on line 578. The functions referred to within it should be all defined above, so you can have a peek at them as necessary. Starting from line 634 I show how I use the rhs function. In particular, note the "disjointed" evaluation of the integral -- I don't just evaluate from 0 to t all at one go, but rather evaluate the integral in pieces (let's call the time spent between the end of one integral evaluation, and the start of the next integral evaluation a "pause"). This is so that if there were multiple amoebas, during the "pause", I can take into account changes in some of the parameters due to contact between one amoeba and another -- poor man's simplification. Please let me know if this is what you were looking for. I wouldn't be surprised if it wasn't, but instead would be happy to try to rework what I've got so it's more in line with what would be meaningful to you. Kind regards, Brian On Wed, Dec 9, 2015 at 2:18 PM, Barry Smith wrote: > > I prefer the actual code, not the mathematics or the explanation > > > On Dec 9, 2015, at 3:42 PM, Brian Merchant wrote: > > > > Hi Barry, > > > > > Could send an example of your "rhs" function; not a totally trivial > example > > > > Sure thing! Although, did you check out the exam I tried to build up in > this stackexchange question, along with a picture: > http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a > > > > I ask because that's probably the best I can do without using as little > math as possible. > > > > Otherwise, what I'll do is take a couple of days to carefully look at my > work, and write up a non-trivial example of a difficult-to-differentiate > RHS, that still is a simplification of the whole mess -- expect a one or > two page PDF? > > > > Kind regards, > > Brian > > > > On Mon, Dec 7, 2015 at 12:45 PM, Barry Smith wrote: > > > > Brian, > > > > Could send an example of your "rhs" function; not a totally trivial > example > > > > Barry > > > > > On Dec 7, 2015, at 11:21 AM, Brian Merchant > wrote: > > > > > > Hi all, > > > > > > I am considering using petsc4py instead of scipy.integrate.odeint > (which is a wrapper for Fortran solvers) for a problem involving the > solution of a system of ODEs. The problem has the potential to be stiff. > Writing down its Jacobian is very hard. > > > > > > So far, I have been able to produce reasonable speed gains by writing > the RHS functions in "something like C" (using either numba or Cython). I'd > like to get even more performance out, hence my consideration of PETSc. > > > > > > Due to the large number of equations involved, it is already tedious > to think about writing down a Jacobian. Even worse though, is that some of > the functions governing a particular interaction do not have neat > analytical forms (let alone whether or not their derivatives have neat > analytical forms), so we might have a mess of piecewise functions needed to > approximate them if we were to go about still trying to produce a > Jacobian... > > > > > > All the toy examples I see of PETSc time stepping problems have > Jacobians defined, so I wonder if I would even get a speed gain going from > switching to it, if perhaps one of the reasons why I have a high > computational cost is due to not being able to provide a Jacobian function? > > > > > > I described the sort of problem I am working with in more detail in > this scicomp.stackexchange question, which is where most of this question > is copied from, except it also comes with a toy version of the problem I am > dealing with: > http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a > > > > > > All your advice would be most helpful :) > > > > > > Kind regards,Brian > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Dec 10 13:51:37 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 10 Dec 2015 13:51:37 -0600 Subject: [petsc-users] Is it still worth switching to PETSc if I can't write a Jacobian for my problem? In-Reply-To: References: <7A5CAB4B-2812-4AD6-B0C9-B5ECD703AD3B@mcs.anl.gov> <533442B3-DDC3-4C5F-9BC3-A21CE78A9BC0@mcs.anl.gov> Message-ID: <90AACE87-C68E-423E-AE7B-7C866D14B93A@mcs.anl.gov> Brian, I see two distinct issues here 1) being apply to apply your right hand side efficiently and 2) what type of ODE integrators, if any, can work well for your problem with its funky, possibly discontinuous right hand side? 1) Looking at the simplicity of your data structure and function evaluation I think you should just write your right hand side functions in C. The code would be no more complicated than it is now, from what you showed me. Even with "just in time" compilation likely you lose at least a factor of 10 by coding in python, maybe 100? I don't know. It looks to me like an easy translation of your current routines to C. 2) This is tricky. You a) cannot compute derivatives? and b) the function and derivatives may not be smooth? If the function was well behaved you could use finite differences to compute the Jacobian (reasonably efficiently, the cost is just some number of function evaluations) but with a flaky right hand side function the finite differences can produce garbage. You could use explicit schemes with a small enough timestep to satisfy any stability condition and forget about using implicit schemes. Then it becomes crucial to have a very fast right hand side function (since you will need many time steps). If you are lucky you can use something like a 4th order RK scheme (but again maybe with a non smooth right hand side may you can't). I am no expert. Perhaps Emil who is far more knowledgable about these things has questions? Barry > On Dec 10, 2015, at 12:52 PM, Brian Merchant wrote: > > Hi Barry, > > Here's some non-trivial example code: https://gist.github.com/bmer/2af429f88b0b696648a8 > > I have still made some simplifications by removing some phase variables, expanding on variable names in general, and so on. > > The rhs function itself is defined on line 578. The functions referred to within it should be all defined above, so you can have a peek at them as necessary. > > Starting from line 634 I show how I use the rhs function. In particular, note the "disjointed" evaluation of the integral -- I don't just evaluate from 0 to t all at one go, but rather evaluate the integral in pieces (let's call the time spent between the end of one integral evaluation, and the start of the next integral evaluation a "pause"). This is so that if there were multiple amoebas, during the "pause", I can take into account changes in some of the parameters due to contact between one amoeba and another -- poor man's simplification. > > Please let me know if this is what you were looking for. I wouldn't be surprised if it wasn't, but instead would be happy to try to rework what I've got so it's more in line with what would be meaningful to you. > > Kind regards, > Brian > > On Wed, Dec 9, 2015 at 2:18 PM, Barry Smith wrote: > > I prefer the actual code, not the mathematics or the explanation > > > On Dec 9, 2015, at 3:42 PM, Brian Merchant wrote: > > > > Hi Barry, > > > > > Could send an example of your "rhs" function; not a totally trivial example > > > > Sure thing! Although, did you check out the exam I tried to build up in this stackexchange question, along with a picture: http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a > > > > I ask because that's probably the best I can do without using as little math as possible. > > > > Otherwise, what I'll do is take a couple of days to carefully look at my work, and write up a non-trivial example of a difficult-to-differentiate RHS, that still is a simplification of the whole mess -- expect a one or two page PDF? > > > > Kind regards, > > Brian > > > > On Mon, Dec 7, 2015 at 12:45 PM, Barry Smith wrote: > > > > Brian, > > > > Could send an example of your "rhs" function; not a totally trivial example > > > > Barry > > > > > On Dec 7, 2015, at 11:21 AM, Brian Merchant wrote: > > > > > > Hi all, > > > > > > I am considering using petsc4py instead of scipy.integrate.odeint (which is a wrapper for Fortran solvers) for a problem involving the solution of a system of ODEs. The problem has the potential to be stiff. Writing down its Jacobian is very hard. > > > > > > So far, I have been able to produce reasonable speed gains by writing the RHS functions in "something like C" (using either numba or Cython). I'd like to get even more performance out, hence my consideration of PETSc. > > > > > > Due to the large number of equations involved, it is already tedious to think about writing down a Jacobian. Even worse though, is that some of the functions governing a particular interaction do not have neat analytical forms (let alone whether or not their derivatives have neat analytical forms), so we might have a mess of piecewise functions needed to approximate them if we were to go about still trying to produce a Jacobian... > > > > > > All the toy examples I see of PETSc time stepping problems have Jacobians defined, so I wonder if I would even get a speed gain going from switching to it, if perhaps one of the reasons why I have a high computational cost is due to not being able to provide a Jacobian function? > > > > > > I described the sort of problem I am working with in more detail in this scicomp.stackexchange question, which is where most of this question is copied from, except it also comes with a toy version of the problem I am dealing with: http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a > > > > > > All your advice would be most helpful :) > > > > > > Kind regards,Brian > > > > > > > > > From emconsta at mcs.anl.gov Thu Dec 10 14:55:34 2015 From: emconsta at mcs.anl.gov (Emil Constantinescu) Date: Thu, 10 Dec 2015 14:55:34 -0600 Subject: [petsc-users] Is it still worth switching to PETSc if I can't write a Jacobian for my problem? In-Reply-To: <90AACE87-C68E-423E-AE7B-7C866D14B93A@mcs.anl.gov> References: <7A5CAB4B-2812-4AD6-B0C9-B5ECD703AD3B@mcs.anl.gov> <533442B3-DDC3-4C5F-9BC3-A21CE78A9BC0@mcs.anl.gov> <90AACE87-C68E-423E-AE7B-7C866D14B93A@mcs.anl.gov> Message-ID: <5669E6C6.8090001@mcs.anl.gov> Brian, Can you take a look at what odeint returns? Specifically, at: ?mused? a vector of method indicators for each successful time step: 1: adams (nonstiff), 2: bdf (stiff) I suspect it's using Adams all the way, which means it's doesn't need a Jacobian. Emil On 12/10/15 1:51 PM, Barry Smith wrote: > > Brian, > > I see two distinct issues here > > 1) being apply to apply your right hand side efficiently and > > 2) what type of ODE integrators, if any, can work well for your problem with its funky, possibly discontinuous right hand side? > > 1) Looking at the simplicity of your data structure and function evaluation I think you should just write your right hand side functions in C. The code would be no more complicated than it is now, from what you showed me. Even with "just in time" compilation likely you lose at least a factor of 10 by coding in python, maybe 100? I don't know. It looks to me like an easy translation of your current routines to C. > > 2) This is tricky. You a) cannot compute derivatives? and b) the function and derivatives may not be smooth? > > If the function was well behaved you could use finite differences to compute the Jacobian (reasonably efficiently, the cost is just some number of function evaluations) but with a flaky right hand side function the finite differences can produce garbage. > > You could use explicit schemes with a small enough timestep to satisfy any stability condition and forget about using implicit schemes. Then it becomes crucial to have a very fast right hand side function (since you will need many time steps). If you are lucky you can use something like a 4th order RK scheme (but again maybe with a non smooth right hand side may you can't). > > I am no expert. Perhaps Emil who is far more knowledgable about these things has questions? > > Barry > > > >> On Dec 10, 2015, at 12:52 PM, Brian Merchant wrote: >> >> Hi Barry, >> >> Here's some non-trivial example code: https://gist.github.com/bmer/2af429f88b0b696648a8 >> >> I have still made some simplifications by removing some phase variables, expanding on variable names in general, and so on. >> >> The rhs function itself is defined on line 578. The functions referred to within it should be all defined above, so you can have a peek at them as necessary. >> >> Starting from line 634 I show how I use the rhs function. In particular, note the "disjointed" evaluation of the integral -- I don't just evaluate from 0 to t all at one go, but rather evaluate the integral in pieces (let's call the time spent between the end of one integral evaluation, and the start of the next integral evaluation a "pause"). This is so that if there were multiple amoebas, during the "pause", I can take into account changes in some of the parameters due to contact between one amoeba and another -- poor man's simplification. >> >> Please let me know if this is what you were looking for. I wouldn't be surprised if it wasn't, but instead would be happy to try to rework what I've got so it's more in line with what would be meaningful to you. >> >> Kind regards, >> Brian >> >> On Wed, Dec 9, 2015 at 2:18 PM, Barry Smith wrote: >> >> I prefer the actual code, not the mathematics or the explanation >> >>> On Dec 9, 2015, at 3:42 PM, Brian Merchant wrote: >>> >>> Hi Barry, >>> >>>> Could send an example of your "rhs" function; not a totally trivial example >>> >>> Sure thing! Although, did you check out the exam I tried to build up in this stackexchange question, along with a picture: http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a >>> >>> I ask because that's probably the best I can do without using as little math as possible. >>> >>> Otherwise, what I'll do is take a couple of days to carefully look at my work, and write up a non-trivial example of a difficult-to-differentiate RHS, that still is a simplification of the whole mess -- expect a one or two page PDF? >>> >>> Kind regards, >>> Brian >>> >>> On Mon, Dec 7, 2015 at 12:45 PM, Barry Smith wrote: >>> >>> Brian, >>> >>> Could send an example of your "rhs" function; not a totally trivial example >>> >>> Barry >>> >>>> On Dec 7, 2015, at 11:21 AM, Brian Merchant wrote: >>>> >>>> Hi all, >>>> >>>> I am considering using petsc4py instead of scipy.integrate.odeint (which is a wrapper for Fortran solvers) for a problem involving the solution of a system of ODEs. The problem has the potential to be stiff. Writing down its Jacobian is very hard. >>>> >>>> So far, I have been able to produce reasonable speed gains by writing the RHS functions in "something like C" (using either numba or Cython). I'd like to get even more performance out, hence my consideration of PETSc. >>>> >>>> Due to the large number of equations involved, it is already tedious to think about writing down a Jacobian. Even worse though, is that some of the functions governing a particular interaction do not have neat analytical forms (let alone whether or not their derivatives have neat analytical forms), so we might have a mess of piecewise functions needed to approximate them if we were to go about still trying to produce a Jacobian... >>>> >>>> All the toy examples I see of PETSc time stepping problems have Jacobians defined, so I wonder if I would even get a speed gain going from switching to it, if perhaps one of the reasons why I have a high computational cost is due to not being able to provide a Jacobian function? >>>> >>>> I described the sort of problem I am working with in more detail in this scicomp.stackexchange question, which is where most of this question is copied from, except it also comes with a toy version of the problem I am dealing with: http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a >>>> >>>> All your advice would be most helpful :) >>>> >>>> Kind regards,Brian >>>> >>> >>> >> >> > From bhmerchant at gmail.com Thu Dec 10 14:56:31 2015 From: bhmerchant at gmail.com (Brian Merchant) Date: Thu, 10 Dec 2015 12:56:31 -0800 Subject: [petsc-users] Is it still worth switching to PETSc if I can't write a Jacobian for my problem? In-Reply-To: <90AACE87-C68E-423E-AE7B-7C866D14B93A@mcs.anl.gov> References: <7A5CAB4B-2812-4AD6-B0C9-B5ECD703AD3B@mcs.anl.gov> <533442B3-DDC3-4C5F-9BC3-A21CE78A9BC0@mcs.anl.gov> <90AACE87-C68E-423E-AE7B-7C866D14B93A@mcs.anl.gov> Message-ID: Hi Barry, Points 1) and 2) are bang on -- you understand the situation very well. 1) with regards to being able to apply the RHS efficiently in C -- I would be happy to do this, but I worry about how to cleanly get C arrays back into NumPy arrays -- NumPy arrays are easy to manipulate for animation, and general data analysis and visualization. Still, this is not so bad, because all it requires is some learning from my part, since I know that NumPy does have a C API. 2) b) Yes, the functions are discontinuous, in particular because some parameters can jump in value from 0 to non-zero when certain conditions are met ("is point outside of of boundary polygon? then turn on extra inhibition"). a) I think if I ignored such discontinuities and just considered a simpler problem where I had to write the derivative of r_n (one of the chemistry variables at node n, whose inhibition is related to the magnitude of strain in the edges connecting to the node in question) with respect to x_n (the x-direction position variable of node n), things begin to get tricky and complicated, manual differentiation wise. Then, we also have to consider how r_n changes as x_(n+k) changes (how does r at node n change, with respect to the change in position of another node, n+k, which would affect the strain at this node), and things get even more cumbersome and annoying -- at least in my head. Someone mentioned "automatic differentiation", but I don't know enough about that to say anything more -- is that the same as computing the Jacobian using finite differences? Some quick Wikipedia reading seems to say "no". > You could use explicit schemes with a small enough timestep to satisfy any stability condition and forget about using implicit schemes. Then it becomes crucial to have a very fast right hand side function (since you will need many time steps). If you are lucky you can use something like a 4th order RK scheme (but again maybe with a non smooth right hand side may you can't). In the literature, this seems to be common approach for those who are only modelling space variables at node n, and not any chemistry. I have a feeling (although this is probably because I am a novice at numerical analysis) that calculating some bound on the timesteps below which stability is guaranteed would be very tough for the full problem including space variables and chemistry variables. I suppose I could skip the math and just try to reduce the timestep until things seem to work in comparison with the results from an implicit scheme, and then decide "good enough!"? How robust would that be though? If I have to repeat the comparison with results from an implicit method quite often, then it may not be worth it? Kind regards, Brian On Thu, Dec 10, 2015 at 11:51 AM, Barry Smith wrote: > > Brian, > > I see two distinct issues here > > 1) being apply to apply your right hand side efficiently and > > 2) what type of ODE integrators, if any, can work well for your problem > with its funky, possibly discontinuous right hand side? > > 1) Looking at the simplicity of your data structure and function > evaluation I think you should just write your right hand side functions in > C. The code would be no more complicated than it is now, from what you > showed me. Even with "just in time" compilation likely you lose at least a > factor of 10 by coding in python, maybe 100? I don't know. It looks to me > like an easy translation of your current routines to C. > > 2) This is tricky. You a) cannot compute derivatives? and b) the function > and derivatives may not be smooth? > > If the function was well behaved you could use finite differences to > compute the Jacobian (reasonably efficiently, the cost is just some number > of function evaluations) but with a flaky right hand side function the > finite differences can produce garbage. > > You could use explicit schemes with a small enough timestep to satisfy > any stability condition and forget about using implicit schemes. Then it > becomes crucial to have a very fast right hand side function (since you > will need many time steps). If you are lucky you can use something like a > 4th order RK scheme (but again maybe with a non smooth right hand side may > you can't). > > I am no expert. Perhaps Emil who is far more knowledgable about these > things has questions? > > Barry > > > > > On Dec 10, 2015, at 12:52 PM, Brian Merchant > wrote: > > > > Hi Barry, > > > > Here's some non-trivial example code: > https://gist.github.com/bmer/2af429f88b0b696648a8 > > > > I have still made some simplifications by removing some phase variables, > expanding on variable names in general, and so on. > > > > The rhs function itself is defined on line 578. The functions referred > to within it should be all defined above, so you can have a peek at them as > necessary. > > > > Starting from line 634 I show how I use the rhs function. In particular, > note the "disjointed" evaluation of the integral -- I don't just evaluate > from 0 to t all at one go, but rather evaluate the integral in pieces > (let's call the time spent between the end of one integral evaluation, and > the start of the next integral evaluation a "pause"). This is so that if > there were multiple amoebas, during the "pause", I can take into account > changes in some of the parameters due to contact between one amoeba and > another -- poor man's simplification. > > > > Please let me know if this is what you were looking for. I wouldn't be > surprised if it wasn't, but instead would be happy to try to rework what > I've got so it's more in line with what would be meaningful to you. > > > > Kind regards, > > Brian > > > > On Wed, Dec 9, 2015 at 2:18 PM, Barry Smith wrote: > > > > I prefer the actual code, not the mathematics or the explanation > > > > > On Dec 9, 2015, at 3:42 PM, Brian Merchant > wrote: > > > > > > Hi Barry, > > > > > > > Could send an example of your "rhs" function; not a totally trivial > example > > > > > > Sure thing! Although, did you check out the exam I tried to build up > in this stackexchange question, along with a picture: > http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a > > > > > > I ask because that's probably the best I can do without using as > little math as possible. > > > > > > Otherwise, what I'll do is take a couple of days to carefully look at > my work, and write up a non-trivial example of a difficult-to-differentiate > RHS, that still is a simplification of the whole mess -- expect a one or > two page PDF? > > > > > > Kind regards, > > > Brian > > > > > > On Mon, Dec 7, 2015 at 12:45 PM, Barry Smith > wrote: > > > > > > Brian, > > > > > > Could send an example of your "rhs" function; not a totally > trivial example > > > > > > Barry > > > > > > > On Dec 7, 2015, at 11:21 AM, Brian Merchant > wrote: > > > > > > > > Hi all, > > > > > > > > I am considering using petsc4py instead of scipy.integrate.odeint > (which is a wrapper for Fortran solvers) for a problem involving the > solution of a system of ODEs. The problem has the potential to be stiff. > Writing down its Jacobian is very hard. > > > > > > > > So far, I have been able to produce reasonable speed gains by > writing the RHS functions in "something like C" (using either numba or > Cython). I'd like to get even more performance out, hence my consideration > of PETSc. > > > > > > > > Due to the large number of equations involved, it is already tedious > to think about writing down a Jacobian. Even worse though, is that some of > the functions governing a particular interaction do not have neat > analytical forms (let alone whether or not their derivatives have neat > analytical forms), so we might have a mess of piecewise functions needed to > approximate them if we were to go about still trying to produce a > Jacobian... > > > > > > > > All the toy examples I see of PETSc time stepping problems have > Jacobians defined, so I wonder if I would even get a speed gain going from > switching to it, if perhaps one of the reasons why I have a high > computational cost is due to not being able to provide a Jacobian function? > > > > > > > > I described the sort of problem I am working with in more detail in > this scicomp.stackexchange question, which is where most of this question > is copied from, except it also comes with a toy version of the problem I am > dealing with: > http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a > > > > > > > > All your advice would be most helpful :) > > > > > > > > Kind regards,Brian > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Dec 10 15:12:38 2015 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 10 Dec 2015 15:12:38 -0600 Subject: [petsc-users] Is it still worth switching to PETSc if I can't write a Jacobian for my problem? In-Reply-To: References: <7A5CAB4B-2812-4AD6-B0C9-B5ECD703AD3B@mcs.anl.gov> <533442B3-DDC3-4C5F-9BC3-A21CE78A9BC0@mcs.anl.gov> <90AACE87-C68E-423E-AE7B-7C866D14B93A@mcs.anl.gov> Message-ID: On Thu, Dec 10, 2015 at 2:56 PM, Brian Merchant wrote: > Hi Barry, > > Points 1) and 2) are bang on -- you understand the situation very well. > > 1) with regards to being able to apply the RHS efficiently in C -- I would > be happy to do this, but I worry about how to cleanly get C arrays back > into NumPy arrays -- NumPy arrays are easy to manipulate for animation, and > general data analysis and visualization. Still, this is not so bad, because > all it requires is some learning from my part, since I know that NumPy does > have a C API. > v.getArray() gives back a NumPy array. Thanks, Matt > 2) b) Yes, the functions are discontinuous, in particular because some > parameters can jump in value from 0 to non-zero when certain conditions are > met ("is point outside of of boundary polygon? then turn on extra > inhibition"). > > a) I think if I ignored such discontinuities and just considered a simpler > problem where I had to write the derivative of r_n (one of the chemistry > variables at node n, whose inhibition is related to the magnitude of strain > in the edges connecting to the node in question) with respect to x_n (the > x-direction position variable of node n), things begin to get tricky and > complicated, manual differentiation wise. > > Then, we also have to consider how r_n changes as x_(n+k) changes (how > does r at node n change, with respect to the change in position of another > node, n+k, which would affect the strain at this node), and things get even > more cumbersome and annoying -- at least in my head. Someone mentioned > "automatic differentiation", but I don't know enough about that to say > anything more -- is that the same as computing the Jacobian using finite > differences? Some quick Wikipedia reading seems to say "no". > > > > You could use explicit schemes with a small enough timestep to satisfy > any stability condition and forget about using implicit schemes. Then it > becomes crucial to have a very fast right hand side function (since you > will need many time steps). If you are lucky you can use something like a > 4th order RK scheme (but again maybe with a non smooth right hand side may > you can't). > > In the literature, this seems to be common approach for those who are only > modelling space variables at node n, and not any chemistry. I have a > feeling (although this is probably because I am a novice at numerical > analysis) that calculating some bound on the timesteps below which > stability is guaranteed would be very tough for the full problem including > space variables and chemistry variables. I suppose I could skip the math > and just try to reduce the timestep until things seem to work in comparison > with the results from an implicit scheme, and then decide "good enough!"? > > How robust would that be though? If I have to repeat the comparison with > results from an implicit method quite often, then it may not be worth it? > > > Kind regards, > Brian > > > > > > > > > On Thu, Dec 10, 2015 at 11:51 AM, Barry Smith wrote: > >> >> Brian, >> >> I see two distinct issues here >> >> 1) being apply to apply your right hand side efficiently and >> >> 2) what type of ODE integrators, if any, can work well for your problem >> with its funky, possibly discontinuous right hand side? >> >> 1) Looking at the simplicity of your data structure and function >> evaluation I think you should just write your right hand side functions in >> C. The code would be no more complicated than it is now, from what you >> showed me. Even with "just in time" compilation likely you lose at least a >> factor of 10 by coding in python, maybe 100? I don't know. It looks to me >> like an easy translation of your current routines to C. >> >> 2) This is tricky. You a) cannot compute derivatives? and b) the function >> and derivatives may not be smooth? >> >> If the function was well behaved you could use finite differences to >> compute the Jacobian (reasonably efficiently, the cost is just some number >> of function evaluations) but with a flaky right hand side function the >> finite differences can produce garbage. >> >> You could use explicit schemes with a small enough timestep to >> satisfy any stability condition and forget about using implicit schemes. >> Then it becomes crucial to have a very fast right hand side function (since >> you will need many time steps). If you are lucky you can use something like >> a 4th order RK scheme (but again maybe with a non smooth right hand side >> may you can't). >> >> I am no expert. Perhaps Emil who is far more knowledgable about these >> things has questions? >> >> Barry >> >> >> >> > On Dec 10, 2015, at 12:52 PM, Brian Merchant >> wrote: >> > >> > Hi Barry, >> > >> > Here's some non-trivial example code: >> https://gist.github.com/bmer/2af429f88b0b696648a8 >> > >> > I have still made some simplifications by removing some phase >> variables, expanding on variable names in general, and so on. >> > >> > The rhs function itself is defined on line 578. The functions referred >> to within it should be all defined above, so you can have a peek at them as >> necessary. >> > >> > Starting from line 634 I show how I use the rhs function. In >> particular, note the "disjointed" evaluation of the integral -- I don't >> just evaluate from 0 to t all at one go, but rather evaluate the integral >> in pieces (let's call the time spent between the end of one integral >> evaluation, and the start of the next integral evaluation a "pause"). This >> is so that if there were multiple amoebas, during the "pause", I can take >> into account changes in some of the parameters due to contact between one >> amoeba and another -- poor man's simplification. >> > >> > Please let me know if this is what you were looking for. I wouldn't be >> surprised if it wasn't, but instead would be happy to try to rework what >> I've got so it's more in line with what would be meaningful to you. >> > >> > Kind regards, >> > Brian >> > >> > On Wed, Dec 9, 2015 at 2:18 PM, Barry Smith wrote: >> > >> > I prefer the actual code, not the mathematics or the explanation >> > >> > > On Dec 9, 2015, at 3:42 PM, Brian Merchant >> wrote: >> > > >> > > Hi Barry, >> > > >> > > > Could send an example of your "rhs" function; not a totally trivial >> example >> > > >> > > Sure thing! Although, did you check out the exam I tried to build up >> in this stackexchange question, along with a picture: >> http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a >> > > >> > > I ask because that's probably the best I can do without using as >> little math as possible. >> > > >> > > Otherwise, what I'll do is take a couple of days to carefully look at >> my work, and write up a non-trivial example of a difficult-to-differentiate >> RHS, that still is a simplification of the whole mess -- expect a one or >> two page PDF? >> > > >> > > Kind regards, >> > > Brian >> > > >> > > On Mon, Dec 7, 2015 at 12:45 PM, Barry Smith >> wrote: >> > > >> > > Brian, >> > > >> > > Could send an example of your "rhs" function; not a totally >> trivial example >> > > >> > > Barry >> > > >> > > > On Dec 7, 2015, at 11:21 AM, Brian Merchant >> wrote: >> > > > >> > > > Hi all, >> > > > >> > > > I am considering using petsc4py instead of scipy.integrate.odeint >> (which is a wrapper for Fortran solvers) for a problem involving the >> solution of a system of ODEs. The problem has the potential to be stiff. >> Writing down its Jacobian is very hard. >> > > > >> > > > So far, I have been able to produce reasonable speed gains by >> writing the RHS functions in "something like C" (using either numba or >> Cython). I'd like to get even more performance out, hence my consideration >> of PETSc. >> > > > >> > > > Due to the large number of equations involved, it is already >> tedious to think about writing down a Jacobian. Even worse though, is that >> some of the functions governing a particular interaction do not have neat >> analytical forms (let alone whether or not their derivatives have neat >> analytical forms), so we might have a mess of piecewise functions needed to >> approximate them if we were to go about still trying to produce a >> Jacobian... >> > > > >> > > > All the toy examples I see of PETSc time stepping problems have >> Jacobians defined, so I wonder if I would even get a speed gain going from >> switching to it, if perhaps one of the reasons why I have a high >> computational cost is due to not being able to provide a Jacobian function? >> > > > >> > > > I described the sort of problem I am working with in more detail in >> this scicomp.stackexchange question, which is where most of this question >> is copied from, except it also comes with a toy version of the problem I am >> dealing with: >> http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a >> > > > >> > > > All your advice would be most helpful :) >> > > > >> > > > Kind regards,Brian >> > > > >> > > >> > > >> > >> > >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhmerchant at gmail.com Thu Dec 10 15:47:22 2015 From: bhmerchant at gmail.com (Brian Merchant) Date: Thu, 10 Dec 2015 13:47:22 -0800 Subject: [petsc-users] Is it still worth switching to PETSc if I can't write a Jacobian for my problem? In-Reply-To: <5669E6C6.8090001@mcs.anl.gov> References: <7A5CAB4B-2812-4AD6-B0C9-B5ECD703AD3B@mcs.anl.gov> <533442B3-DDC3-4C5F-9BC3-A21CE78A9BC0@mcs.anl.gov> <90AACE87-C68E-423E-AE7B-7C866D14B93A@mcs.anl.gov> <5669E6C6.8090001@mcs.anl.gov> Message-ID: Hi Emil: I had a look at odeint's output -- it seems that while Adams is used initially, bdf is used most often overall (especially in later timesteps), some 80% of the timesteps in total. This is likely because I ran the test on the full problem, which includes interactions between multiple amoebas -- contact between amoebas turns certain parameters on temporarily. Matt: is this the getArray method of petsc4py? It converts a PETSc array to a NumPy array? How did the PETSc array (let's say from a C program) get loaded by petsc4py in the first place -- is there some sort of a "load" function? I am sure there are straightforward answers to this question, but these are the sorts of things I don't know right now. Kind regards, Brian On Thu, Dec 10, 2015 at 12:55 PM, Emil Constantinescu wrote: > Brian, > > Can you take a look at what odeint returns? Specifically, at: > > ?mused? a vector of method indicators for each successful time > step: 1: adams (nonstiff), 2: bdf (stiff) > > I suspect it's using Adams all the way, which means it's doesn't need a > Jacobian. > > Emil > > > > On 12/10/15 1:51 PM, Barry Smith wrote: > >> >> Brian, >> >> I see two distinct issues here >> >> 1) being apply to apply your right hand side efficiently and >> >> 2) what type of ODE integrators, if any, can work well for your problem >> with its funky, possibly discontinuous right hand side? >> >> 1) Looking at the simplicity of your data structure and function >> evaluation I think you should just write your right hand side functions in >> C. The code would be no more complicated than it is now, from what you >> showed me. Even with "just in time" compilation likely you lose at least a >> factor of 10 by coding in python, maybe 100? I don't know. It looks to me >> like an easy translation of your current routines to C. >> >> 2) This is tricky. You a) cannot compute derivatives? and b) the function >> and derivatives may not be smooth? >> >> If the function was well behaved you could use finite differences to >> compute the Jacobian (reasonably efficiently, the cost is just some number >> of function evaluations) but with a flaky right hand side function the >> finite differences can produce garbage. >> >> You could use explicit schemes with a small enough timestep to >> satisfy any stability condition and forget about using implicit schemes. >> Then it becomes crucial to have a very fast right hand side function (since >> you will need many time steps). If you are lucky you can use something like >> a 4th order RK scheme (but again maybe with a non smooth right hand side >> may you can't). >> >> I am no expert. Perhaps Emil who is far more knowledgable about these >> things has questions? >> >> Barry >> >> >> >> On Dec 10, 2015, at 12:52 PM, Brian Merchant >>> wrote: >>> >>> Hi Barry, >>> >>> Here's some non-trivial example code: >>> https://gist.github.com/bmer/2af429f88b0b696648a8 >>> >>> I have still made some simplifications by removing some phase variables, >>> expanding on variable names in general, and so on. >>> >>> The rhs function itself is defined on line 578. The functions referred >>> to within it should be all defined above, so you can have a peek at them as >>> necessary. >>> >>> Starting from line 634 I show how I use the rhs function. In particular, >>> note the "disjointed" evaluation of the integral -- I don't just evaluate >>> from 0 to t all at one go, but rather evaluate the integral in pieces >>> (let's call the time spent between the end of one integral evaluation, and >>> the start of the next integral evaluation a "pause"). This is so that if >>> there were multiple amoebas, during the "pause", I can take into account >>> changes in some of the parameters due to contact between one amoeba and >>> another -- poor man's simplification. >>> >>> Please let me know if this is what you were looking for. I wouldn't be >>> surprised if it wasn't, but instead would be happy to try to rework what >>> I've got so it's more in line with what would be meaningful to you. >>> >>> Kind regards, >>> Brian >>> >>> On Wed, Dec 9, 2015 at 2:18 PM, Barry Smith wrote: >>> >>> I prefer the actual code, not the mathematics or the explanation >>> >>> On Dec 9, 2015, at 3:42 PM, Brian Merchant wrote: >>>> >>>> Hi Barry, >>>> >>>> Could send an example of your "rhs" function; not a totally trivial >>>>> example >>>>> >>>> >>>> Sure thing! Although, did you check out the exam I tried to build up in >>>> this stackexchange question, along with a picture: >>>> http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a >>>> >>>> I ask because that's probably the best I can do without using as little >>>> math as possible. >>>> >>>> Otherwise, what I'll do is take a couple of days to carefully look at >>>> my work, and write up a non-trivial example of a difficult-to-differentiate >>>> RHS, that still is a simplification of the whole mess -- expect a one or >>>> two page PDF? >>>> >>>> Kind regards, >>>> Brian >>>> >>>> On Mon, Dec 7, 2015 at 12:45 PM, Barry Smith >>>> wrote: >>>> >>>> Brian, >>>> >>>> Could send an example of your "rhs" function; not a totally >>>> trivial example >>>> >>>> Barry >>>> >>>> On Dec 7, 2015, at 11:21 AM, Brian Merchant >>>>> wrote: >>>>> >>>>> Hi all, >>>>> >>>>> I am considering using petsc4py instead of scipy.integrate.odeint >>>>> (which is a wrapper for Fortran solvers) for a problem involving the >>>>> solution of a system of ODEs. The problem has the potential to be stiff. >>>>> Writing down its Jacobian is very hard. >>>>> >>>>> So far, I have been able to produce reasonable speed gains by writing >>>>> the RHS functions in "something like C" (using either numba or Cython). I'd >>>>> like to get even more performance out, hence my consideration of PETSc. >>>>> >>>>> Due to the large number of equations involved, it is already tedious >>>>> to think about writing down a Jacobian. Even worse though, is that some of >>>>> the functions governing a particular interaction do not have neat >>>>> analytical forms (let alone whether or not their derivatives have neat >>>>> analytical forms), so we might have a mess of piecewise functions needed to >>>>> approximate them if we were to go about still trying to produce a >>>>> Jacobian... >>>>> >>>>> All the toy examples I see of PETSc time stepping problems have >>>>> Jacobians defined, so I wonder if I would even get a speed gain going from >>>>> switching to it, if perhaps one of the reasons why I have a high >>>>> computational cost is due to not being able to provide a Jacobian function? >>>>> >>>>> I described the sort of problem I am working with in more detail in >>>>> this scicomp.stackexchange question, which is where most of this question >>>>> is copied from, except it also comes with a toy version of the problem I am >>>>> dealing with: >>>>> http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a >>>>> >>>>> All your advice would be most helpful :) >>>>> >>>>> Kind regards,Brian >>>>> >>>>> >>>> >>>> >>> >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From hongzhang at anl.gov Thu Dec 10 15:55:18 2015 From: hongzhang at anl.gov (Hong Zhang) Date: Thu, 10 Dec 2015 15:55:18 -0600 Subject: [petsc-users] Is it still worth switching to PETSc if I can't write a Jacobian for my problem? In-Reply-To: <90AACE87-C68E-423E-AE7B-7C866D14B93A@mcs.anl.gov> References: <7A5CAB4B-2812-4AD6-B0C9-B5ECD703AD3B@mcs.anl.gov> <533442B3-DDC3-4C5F-9BC3-A21CE78A9BC0@mcs.anl.gov> <90AACE87-C68E-423E-AE7B-7C866D14B93A@mcs.anl.gov> Message-ID: <867FA79D-3728-48CB-A849-7B6A5ECD76F0@anl.gov> Hi Brian, If you can?t provide Jacobian and do not need to run the code in parallel, I do not think using PETSc would make too much difference compared to using scipy.integrate.odeint. With both of them, you can either use explicit time integration methods with small time steps, or use implicit methods with internally approximated Jacobians (PETSc can compute a finite difference Jacobian for you). However, scipy.integrate.odeint implements multistep methods, while PETSc provides a rich set of multistage methods (e.g. Runge-Kutta type methods) which may have better stability properties. An explicit integration method with better stability may allow for larger time steps in your problem, thus leading to possible performance gain. But the speedup may not be significant. If you are using implicit methods with internally approximated Jacobians, high computational cost can be expected. In this case, providing a correct Jacobian might be the best way to improve the performance. And if you decide to do so, PETSc is definitely worth a try because it can help you check if the Jacobian is correct and give you diagnostic information about the convergence of the linear solves, the nonlinear solves and the time stepping. Apart from that, you may also try the IMEX time integrators in PETSc. When using IMEX, you need to manually split your right hand side function into two parts ? stiff and non-stiff parts, and only provide the Jacobian for the stiff part (usually linear). This requires the users know the source of stiffness in their problems. Some IMEX examples for advection-diffusion-rection problems and several simple ODEs can be found at http://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/ and http://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/advection-diffusion-reaction/ex3.c.html Hong On Dec 10, 2015, at 1:51 PM, Barry Smith wrote: > > Brian, > > I see two distinct issues here > > 1) being apply to apply your right hand side efficiently and > > 2) what type of ODE integrators, if any, can work well for your problem with its funky, possibly discontinuous right hand side? > > 1) Looking at the simplicity of your data structure and function evaluation I think you should just write your right hand side functions in C. The code would be no more complicated than it is now, from what you showed me. Even with "just in time" compilation likely you lose at least a factor of 10 by coding in python, maybe 100? I don't know. It looks to me like an easy translation of your current routines to C. > > 2) This is tricky. You a) cannot compute derivatives? and b) the function and derivatives may not be smooth? > > If the function was well behaved you could use finite differences to compute the Jacobian (reasonably efficiently, the cost is just some number of function evaluations) but with a flaky right hand side function the finite differences can produce garbage. > > You could use explicit schemes with a small enough timestep to satisfy any stability condition and forget about using implicit schemes. Then it becomes crucial to have a very fast right hand side function (since you will need many time steps). If you are lucky you can use something like a 4th order RK scheme (but again maybe with a non smooth right hand side may you can't). > > I am no expert. Perhaps Emil who is far more knowledgable about these things has questions? > > Barry > > > >> On Dec 10, 2015, at 12:52 PM, Brian Merchant wrote: >> >> Hi Barry, >> >> Here's some non-trivial example code: https://gist.github.com/bmer/2af429f88b0b696648a8 >> >> I have still made some simplifications by removing some phase variables, expanding on variable names in general, and so on. >> >> The rhs function itself is defined on line 578. The functions referred to within it should be all defined above, so you can have a peek at them as necessary. >> >> Starting from line 634 I show how I use the rhs function. In particular, note the "disjointed" evaluation of the integral -- I don't just evaluate from 0 to t all at one go, but rather evaluate the integral in pieces (let's call the time spent between the end of one integral evaluation, and the start of the next integral evaluation a "pause"). This is so that if there were multiple amoebas, during the "pause", I can take into account changes in some of the parameters due to contact between one amoeba and another -- poor man's simplification. >> >> Please let me know if this is what you were looking for. I wouldn't be surprised if it wasn't, but instead would be happy to try to rework what I've got so it's more in line with what would be meaningful to you. >> >> Kind regards, >> Brian >> >> On Wed, Dec 9, 2015 at 2:18 PM, Barry Smith wrote: >> >> I prefer the actual code, not the mathematics or the explanation >> >>> On Dec 9, 2015, at 3:42 PM, Brian Merchant wrote: >>> >>> Hi Barry, >>> >>>> Could send an example of your "rhs" function; not a totally trivial example >>> >>> Sure thing! Although, did you check out the exam I tried to build up in this stackexchange question, along with a picture: http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a >>> >>> I ask because that's probably the best I can do without using as little math as possible. >>> >>> Otherwise, what I'll do is take a couple of days to carefully look at my work, and write up a non-trivial example of a difficult-to-differentiate RHS, that still is a simplification of the whole mess -- expect a one or two page PDF? >>> >>> Kind regards, >>> Brian >>> >>> On Mon, Dec 7, 2015 at 12:45 PM, Barry Smith wrote: >>> >>> Brian, >>> >>> Could send an example of your "rhs" function; not a totally trivial example >>> >>> Barry >>> >>>> On Dec 7, 2015, at 11:21 AM, Brian Merchant wrote: >>>> >>>> Hi all, >>>> >>>> I am considering using petsc4py instead of scipy.integrate.odeint (which is a wrapper for Fortran solvers) for a problem involving the solution of a system of ODEs. The problem has the potential to be stiff. Writing down its Jacobian is very hard. >>>> >>>> So far, I have been able to produce reasonable speed gains by writing the RHS functions in "something like C" (using either numba or Cython). I'd like to get even more performance out, hence my consideration of PETSc. >>>> >>>> Due to the large number of equations involved, it is already tedious to think about writing down a Jacobian. Even worse though, is that some of the functions governing a particular interaction do not have neat analytical forms (let alone whether or not their derivatives have neat analytical forms), so we might have a mess of piecewise functions needed to approximate them if we were to go about still trying to produce a Jacobian... >>>> >>>> All the toy examples I see of PETSc time stepping problems have Jacobians defined, so I wonder if I would even get a speed gain going from switching to it, if perhaps one of the reasons why I have a high computational cost is due to not being able to provide a Jacobian function? >>>> >>>> I described the sort of problem I am working with in more detail in this scicomp.stackexchange question, which is where most of this question is copied from, except it also comes with a toy version of the problem I am dealing with: http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a >>>> >>>> All your advice would be most helpful :) >>>> >>>> Kind regards,Brian >>>> >>> >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Dec 10 15:59:10 2015 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 10 Dec 2015 15:59:10 -0600 Subject: [petsc-users] Is it still worth switching to PETSc if I can't write a Jacobian for my problem? In-Reply-To: References: <7A5CAB4B-2812-4AD6-B0C9-B5ECD703AD3B@mcs.anl.gov> <533442B3-DDC3-4C5F-9BC3-A21CE78A9BC0@mcs.anl.gov> <90AACE87-C68E-423E-AE7B-7C866D14B93A@mcs.anl.gov> <5669E6C6.8090001@mcs.anl.gov> Message-ID: On Thu, Dec 10, 2015 at 3:47 PM, Brian Merchant wrote: > Hi Emil: > > I had a look at odeint's output -- it seems that while Adams is used > initially, bdf is used most often overall (especially in later timesteps), > some 80% of the timesteps in total. > > This is likely because I ran the test on the full problem, which includes > interactions between multiple amoebas -- contact between amoebas turns > certain parameters on temporarily. > > Matt: is this the getArray method of petsc4py? It converts a PETSc array > to a NumPy array? How did the PETSc array (let's say from a C program) get > loaded by petsc4py in the first place -- is there some sort of a "load" > function? I am sure there are straightforward answers to this question, but > these are the sorts of things I don't know right now. > The memory is always in one place. The PETSc Vec and NumPy array are just two ways of looking into it. Matt > Kind regards, > Brian > > On Thu, Dec 10, 2015 at 12:55 PM, Emil Constantinescu < > emconsta at mcs.anl.gov> wrote: > >> Brian, >> >> Can you take a look at what odeint returns? Specifically, at: >> >> ?mused? a vector of method indicators for each successful time >> step: 1: adams (nonstiff), 2: bdf (stiff) >> >> I suspect it's using Adams all the way, which means it's doesn't need a >> Jacobian. >> >> Emil >> >> >> >> On 12/10/15 1:51 PM, Barry Smith wrote: >> >>> >>> Brian, >>> >>> I see two distinct issues here >>> >>> 1) being apply to apply your right hand side efficiently and >>> >>> 2) what type of ODE integrators, if any, can work well for your problem >>> with its funky, possibly discontinuous right hand side? >>> >>> 1) Looking at the simplicity of your data structure and function >>> evaluation I think you should just write your right hand side functions in >>> C. The code would be no more complicated than it is now, from what you >>> showed me. Even with "just in time" compilation likely you lose at least a >>> factor of 10 by coding in python, maybe 100? I don't know. It looks to me >>> like an easy translation of your current routines to C. >>> >>> 2) This is tricky. You a) cannot compute derivatives? and b) the >>> function and derivatives may not be smooth? >>> >>> If the function was well behaved you could use finite differences >>> to compute the Jacobian (reasonably efficiently, the cost is just some >>> number of function evaluations) but with a flaky right hand side function >>> the finite differences can produce garbage. >>> >>> You could use explicit schemes with a small enough timestep to >>> satisfy any stability condition and forget about using implicit schemes. >>> Then it becomes crucial to have a very fast right hand side function (since >>> you will need many time steps). If you are lucky you can use something like >>> a 4th order RK scheme (but again maybe with a non smooth right hand side >>> may you can't). >>> >>> I am no expert. Perhaps Emil who is far more knowledgable about >>> these things has questions? >>> >>> Barry >>> >>> >>> >>> On Dec 10, 2015, at 12:52 PM, Brian Merchant >>>> wrote: >>>> >>>> Hi Barry, >>>> >>>> Here's some non-trivial example code: >>>> https://gist.github.com/bmer/2af429f88b0b696648a8 >>>> >>>> I have still made some simplifications by removing some phase >>>> variables, expanding on variable names in general, and so on. >>>> >>>> The rhs function itself is defined on line 578. The functions referred >>>> to within it should be all defined above, so you can have a peek at them as >>>> necessary. >>>> >>>> Starting from line 634 I show how I use the rhs function. In >>>> particular, note the "disjointed" evaluation of the integral -- I don't >>>> just evaluate from 0 to t all at one go, but rather evaluate the integral >>>> in pieces (let's call the time spent between the end of one integral >>>> evaluation, and the start of the next integral evaluation a "pause"). This >>>> is so that if there were multiple amoebas, during the "pause", I can take >>>> into account changes in some of the parameters due to contact between one >>>> amoeba and another -- poor man's simplification. >>>> >>>> Please let me know if this is what you were looking for. I wouldn't be >>>> surprised if it wasn't, but instead would be happy to try to rework what >>>> I've got so it's more in line with what would be meaningful to you. >>>> >>>> Kind regards, >>>> Brian >>>> >>>> On Wed, Dec 9, 2015 at 2:18 PM, Barry Smith wrote: >>>> >>>> I prefer the actual code, not the mathematics or the explanation >>>> >>>> On Dec 9, 2015, at 3:42 PM, Brian Merchant >>>>> wrote: >>>>> >>>>> Hi Barry, >>>>> >>>>> Could send an example of your "rhs" function; not a totally trivial >>>>>> example >>>>>> >>>>> >>>>> Sure thing! Although, did you check out the exam I tried to build up >>>>> in this stackexchange question, along with a picture: >>>>> http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a >>>>> >>>>> I ask because that's probably the best I can do without using as >>>>> little math as possible. >>>>> >>>>> Otherwise, what I'll do is take a couple of days to carefully look at >>>>> my work, and write up a non-trivial example of a difficult-to-differentiate >>>>> RHS, that still is a simplification of the whole mess -- expect a one or >>>>> two page PDF? >>>>> >>>>> Kind regards, >>>>> Brian >>>>> >>>>> On Mon, Dec 7, 2015 at 12:45 PM, Barry Smith >>>>> wrote: >>>>> >>>>> Brian, >>>>> >>>>> Could send an example of your "rhs" function; not a totally >>>>> trivial example >>>>> >>>>> Barry >>>>> >>>>> On Dec 7, 2015, at 11:21 AM, Brian Merchant >>>>>> wrote: >>>>>> >>>>>> Hi all, >>>>>> >>>>>> I am considering using petsc4py instead of scipy.integrate.odeint >>>>>> (which is a wrapper for Fortran solvers) for a problem involving the >>>>>> solution of a system of ODEs. The problem has the potential to be stiff. >>>>>> Writing down its Jacobian is very hard. >>>>>> >>>>>> So far, I have been able to produce reasonable speed gains by writing >>>>>> the RHS functions in "something like C" (using either numba or Cython). I'd >>>>>> like to get even more performance out, hence my consideration of PETSc. >>>>>> >>>>>> Due to the large number of equations involved, it is already tedious >>>>>> to think about writing down a Jacobian. Even worse though, is that some of >>>>>> the functions governing a particular interaction do not have neat >>>>>> analytical forms (let alone whether or not their derivatives have neat >>>>>> analytical forms), so we might have a mess of piecewise functions needed to >>>>>> approximate them if we were to go about still trying to produce a >>>>>> Jacobian... >>>>>> >>>>>> All the toy examples I see of PETSc time stepping problems have >>>>>> Jacobians defined, so I wonder if I would even get a speed gain going from >>>>>> switching to it, if perhaps one of the reasons why I have a high >>>>>> computational cost is due to not being able to provide a Jacobian function? >>>>>> >>>>>> I described the sort of problem I am working with in more detail in >>>>>> this scicomp.stackexchange question, which is where most of this question >>>>>> is copied from, except it also comes with a toy version of the problem I am >>>>>> dealing with: >>>>>> http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a >>>>>> >>>>>> All your advice would be most helpful :) >>>>>> >>>>>> Kind regards,Brian >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From emconsta at mcs.anl.gov Thu Dec 10 16:22:24 2015 From: emconsta at mcs.anl.gov (Emil Constantinescu) Date: Thu, 10 Dec 2015 16:22:24 -0600 Subject: [petsc-users] Is it still worth switching to PETSc if I can't write a Jacobian for my problem? In-Reply-To: References: <7A5CAB4B-2812-4AD6-B0C9-B5ECD703AD3B@mcs.anl.gov> <533442B3-DDC3-4C5F-9BC3-A21CE78A9BC0@mcs.anl.gov> <90AACE87-C68E-423E-AE7B-7C866D14B93A@mcs.anl.gov> <5669E6C6.8090001@mcs.anl.gov> Message-ID: <5669FB20.5080700@mcs.anl.gov> On 12/10/15 3:47 PM, Brian Merchant wrote: > Hi Emil: > > I had a look at odeint's output -- it seems that while Adams is used > initially, bdf is used most often overall (especially in later > timesteps), some 80% of the timesteps in total. > > This is likely because I ran the test on the full problem, which > includes interactions between multiple amoebas -- contact between > amoebas turns certain parameters on temporarily. Hmm, so it may be that the problem is not stiff, but stepping over discontinuities makes it appear stiff and hence the switch to BDF or it may be that the problem has periods when it's really stiff. Like Hong said, by using PETSc as the time stepper you may be able to diagnose better what's going on b/c you have a choice of using many integrators and even events should you try to address those discontinuities directly. Emil > Matt: is this the getArray method of petsc4py? It converts a PETSc array > to a NumPy array? How did the PETSc array (let's say from a C program) > get loaded by petsc4py in the first place -- is there some sort of a > "load" function? I am sure there are straightforward answers to this > question, but these are the sorts of things I don't know right now. > > > Kind regards, > Brian > > On Thu, Dec 10, 2015 at 12:55 PM, Emil Constantinescu > > wrote: > > Brian, > > Can you take a look at what odeint returns? Specifically, at: > > ?mused? a vector of method indicators for each successful > time step: 1: adams (nonstiff), 2: bdf (stiff) > > I suspect it's using Adams all the way, which means it's doesn't > need a Jacobian. > > Emil > > > > On 12/10/15 1:51 PM, Barry Smith wrote: > > > Brian, > > I see two distinct issues here > > 1) being apply to apply your right hand side efficiently and > > 2) what type of ODE integrators, if any, can work well for your > problem with its funky, possibly discontinuous right hand side? > > 1) Looking at the simplicity of your data structure and > function evaluation I think you should just write your right > hand side functions in C. The code would be no more complicated > than it is now, from what you showed me. Even with "just in > time" compilation likely you lose at least a factor of 10 by > coding in python, maybe 100? I don't know. It looks to me like > an easy translation of your current routines to C. > > 2) This is tricky. You a) cannot compute derivatives? and b) the > function and derivatives may not be smooth? > > If the function was well behaved you could use finite > differences to compute the Jacobian (reasonably efficiently, the > cost is just some number of function evaluations) but with a > flaky right hand side function the finite differences can > produce garbage. > > You could use explicit schemes with a small enough > timestep to satisfy any stability condition and forget about > using implicit schemes. Then it becomes crucial to have a very > fast right hand side function (since you will need many time > steps). If you are lucky you can use something like a 4th order > RK scheme (but again maybe with a non smooth right hand side may > you can't). > > I am no expert. Perhaps Emil who is far more knowledgable > about these things has questions? > > Barry > > > > On Dec 10, 2015, at 12:52 PM, Brian Merchant > > wrote: > > Hi Barry, > > Here's some non-trivial example code: > https://gist.github.com/bmer/2af429f88b0b696648a8 > > I have still made some simplifications by removing some > phase variables, expanding on variable names in general, and > so on. > > The rhs function itself is defined on line 578. The > functions referred to within it should be all defined above, > so you can have a peek at them as necessary. > > Starting from line 634 I show how I use the rhs function. In > particular, note the "disjointed" evaluation of the integral > -- I don't just evaluate from 0 to t all at one go, but > rather evaluate the integral in pieces (let's call the time > spent between the end of one integral evaluation, and the > start of the next integral evaluation a "pause"). This is so > that if there were multiple amoebas, during the "pause", I > can take into account changes in some of the parameters due > to contact between one amoeba and another -- poor man's > simplification. > > Please let me know if this is what you were looking for. I > wouldn't be surprised if it wasn't, but instead would be > happy to try to rework what I've got so it's more in line > with what would be meaningful to you. > > Kind regards, > Brian > > On Wed, Dec 9, 2015 at 2:18 PM, Barry Smith > > wrote: > > I prefer the actual code, not the mathematics or the > explanation > > On Dec 9, 2015, at 3:42 PM, Brian Merchant > > wrote: > > Hi Barry, > > Could send an example of your "rhs" function; not a > totally trivial example > > > Sure thing! Although, did you check out the exam I tried > to build up in this stackexchange question, along with a > picture: > http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a > > I ask because that's probably the best I can do without > using as little math as possible. > > Otherwise, what I'll do is take a couple of days to > carefully look at my work, and write up a non-trivial > example of a difficult-to-differentiate RHS, that still > is a simplification of the whole mess -- expect a one or > two page PDF? > > Kind regards, > Brian > > On Mon, Dec 7, 2015 at 12:45 PM, Barry Smith > > wrote: > > Brian, > > Could send an example of your "rhs" function; not > a totally trivial example > > Barry > > On Dec 7, 2015, at 11:21 AM, Brian Merchant > > > wrote: > > Hi all, > > I am considering using petsc4py instead of > scipy.integrate.odeint (which is a wrapper for > Fortran solvers) for a problem involving the > solution of a system of ODEs. The problem has the > potential to be stiff. Writing down its Jacobian is > very hard. > > So far, I have been able to produce reasonable speed > gains by writing the RHS functions in "something > like C" (using either numba or Cython). I'd like to > get even more performance out, hence my > consideration of PETSc. > > Due to the large number of equations involved, it is > already tedious to think about writing down a > Jacobian. Even worse though, is that some of the > functions governing a particular interaction do not > have neat analytical forms (let alone whether or not > their derivatives have neat analytical forms), so we > might have a mess of piecewise functions needed to > approximate them if we were to go about still trying > to produce a Jacobian... > > All the toy examples I see of PETSc time stepping > problems have Jacobians defined, so I wonder if I > would even get a speed gain going from switching to > it, if perhaps one of the reasons why I have a high > computational cost is due to not being able to > provide a Jacobian function? > > I described the sort of problem I am working with in > more detail in this scicomp.stackexchange question, > which is where most of this question is copied from, > except it also comes with a toy version of the > problem I am dealing with: > http://scicomp.stackexchange.com/questions/21501/is-it-worth-switching-to-timesteppers-provided-by-petsc-if-i-cant-write-down-a > > All your advice would be most helpful :) > > Kind regards,Brian > > > > > > > From gideon.simpson at gmail.com Fri Dec 11 08:19:13 2015 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Fri, 11 Dec 2015 09:19:13 -0500 Subject: [petsc-users] SPRNG package Message-ID: <32444DF9-350A-43BC-9400-55CB6FAC84ED@gmail.com> I was looking at the source files and noticed in the comments that when petsc is built with sprng, the petsc interface isn?t to the parallel sprng RNG. But is the package built with MPI, so that I could manually use the parallel RNG? -gideon -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Fri Dec 11 09:29:38 2015 From: epscodes at gmail.com (Xiangdong) Date: Fri, 11 Dec 2015 10:29:38 -0500 Subject: [petsc-users] user-defined snes stopping criteria Message-ID: Hello everyone, Instead of using the standard atol, rtol, stol etc, can user define a new stopping criterion for snes? For example, if I want to use a specialized norm as my snes stopping criteria, which functions should I call to pass this information? Thank you. Best, Xiangdong -------------- next part -------------- An HTML attachment was scrubbed... URL: From popov at uni-mainz.de Fri Dec 11 09:44:35 2015 From: popov at uni-mainz.de (anton) Date: Fri, 11 Dec 2015 16:44:35 +0100 Subject: [petsc-users] user-defined snes stopping criteria In-Reply-To: References: Message-ID: <566AEF63.80804@uni-mainz.de> SNESSetConvergenceTest Best, Anton On 12/11/2015 04:29 PM, Xiangdong wrote: > Hello everyone, > > Instead of using the standard atol, rtol, stol etc, can user define a > new stopping criterion for snes? For example, if I want to use a > specialized norm as my snes stopping criteria, which functions should > I call to pass this information? > > Thank you. > > Best, > Xiangdong -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Fri Dec 11 10:30:46 2015 From: hzhang at mcs.anl.gov (Hong) Date: Fri, 11 Dec 2015 10:30:46 -0600 Subject: [petsc-users] SPRNG package In-Reply-To: <32444DF9-350A-43BC-9400-55CB6FAC84ED@gmail.com> References: <32444DF9-350A-43BC-9400-55CB6FAC84ED@gmail.com> Message-ID: Gideon: > I was looking at the source files and noticed in the comments that when > petsc is built with sprng, the petsc interface isn?t to the parallel sprng > RNG. But is the package built with MPI, so that I could manually use the > parallel RNG? > Petsc- sprng-1.0 interface was written many years ago. It is for parallel computation. Students contributed an example at petsc/src/sys/classes/random/examples/tutorials/ex2.c Very few users have ever used this interface. If you encounter any problem, please report to us. Hong -------------- next part -------------- An HTML attachment was scrubbed... URL: From gideon.simpson at gmail.com Fri Dec 11 10:47:27 2015 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Fri, 11 Dec 2015 11:47:27 -0500 Subject: [petsc-users] SPRNG package In-Reply-To: References: <32444DF9-350A-43BC-9400-55CB6FAC84ED@gmail.com> Message-ID: <1689D723-AE19-4118-B33B-48DFABDDBBD1@gmail.com> Ok, but if I look at http://www.mcs.anl.gov/petsc/petsc-dev/src/sys/classes/random/impls/sprng/sprng.c there is a comment: This is NOT currently using a parallel random number generator. Sprng does have an MPI version we should investigate. -gideon > On Dec 11, 2015, at 11:30 AM, Hong wrote: > > Gideon: > I was looking at the source files and noticed in the comments that when petsc is built with sprng, the petsc interface isn?t to the parallel sprng RNG. But is the package built with MPI, so that I could manually use the parallel RNG? > Petsc- sprng-1.0 interface was written many years ago. > It is for parallel computation. Students contributed an example at > petsc/src/sys/classes/random/examples/tutorials/ex2.c > > Very few users have ever used this interface. > If you encounter any problem, please report to us. > > Hong > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Dec 10 22:13:39 2015 From: jed at jedbrown.org (Jed Brown) Date: Thu, 10 Dec 2015 21:13:39 -0700 Subject: [petsc-users] Solving/creating SPD systems In-Reply-To: References: <20151128163526.GA4917@Patricks-MBP-4.railnet.train> Message-ID: <878u51mxl8.fsf@jedbrown.org> Justin Chang writes: > So I am wanting to compare the performance of various FEM discretization > with their respective "best" possible solver/pre conditioner. There > are saddle-point systems which HDiv formulations like RT0 work, but then > there are others like LSFEM that are naturally SPD and so the CG solver can > be used (though finding a good preconditioner is still an open problem). LSFEM ensures an SPD system because it's effectively solving the normal equations. You can use CGNE if you want, but it's not likely to work well. Note that LS methods give up local conservation, which was the reason to choose a H(div) formulation in the first place. You can use compatible spaces/Lagrange multiplies with LS techniques to maintain conservation, but you'll return to a saddle point problem (with more degrees of freedom). There's no free lunch. > I have read and learned that the advantage of LSFEM is that it will always > give you an SPD system, even for non-linear problems (because what you do > is linearize the problem first and then minimize/take the Gateaux > derivative to get the weak form). But after talking to some people and > reading some stuff online, it seems one could also make non SPD systems SPD > (hence eliminating what may be the only advantage of LSFEM). (Symmetric) saddle point problems have an SPD Schur complement. Schur complements are dense in general, but some discretization choices (e.g., quadrature in BDM) can make the primal block diagonal or block-diagonal, resulting in a sparse Schur complement. If the Schur complement is dense, you might be able to approximate it by a sparse matrix. The quality of such an approximation depends on the physics and the discretization. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jychang48 at gmail.com Fri Dec 11 12:58:58 2015 From: jychang48 at gmail.com (Justin Chang) Date: Fri, 11 Dec 2015 11:58:58 -0700 Subject: [petsc-users] Solving/creating SPD systems In-Reply-To: <878u51mxl8.fsf@jedbrown.org> References: <20151128163526.GA4917@Patricks-MBP-4.railnet.train> <878u51mxl8.fsf@jedbrown.org> Message-ID: Jed, 1) What exactly are the PETSc options for CGNE? Also, would LSQR be worth trying? I am doing all of this through Firedrake, so I hope these things can be done directly through simply providing command line PETSc options :) 2) So i spoke with Matt the other day, and the primary issue I am having with LSFEM is finding a suitable preconditioner for the problematic penalty term in Darcy's equation (i.e., the div-div term). So if I had this: (u, v) + (u, grad(q)) + (grad(p), v) + (grad(p), grad(q)) + (div(u), (div(v)) = (rho*b, v + grad(q)) If I remove the div-div term, I have a very nice SPD system which could simply be solved with CG/HYPRE. Do you know of any good preconditioning strategies for this type of problem? Thanks, Justin On Thu, Dec 10, 2015 at 9:13 PM, Jed Brown wrote: > Justin Chang writes: > > > So I am wanting to compare the performance of various FEM discretization > > with their respective "best" possible solver/pre conditioner. There > > are saddle-point systems which HDiv formulations like RT0 work, but then > > there are others like LSFEM that are naturally SPD and so the CG solver > can > > be used (though finding a good preconditioner is still an open problem). > > LSFEM ensures an SPD system because it's effectively solving the normal > equations. You can use CGNE if you want, but it's not likely to work > well. Note that LS methods give up local conservation, which was the > reason to choose a H(div) formulation in the first place. You can use > compatible spaces/Lagrange multiplies with LS techniques to maintain > conservation, but you'll return to a saddle point problem (with more > degrees of freedom). There's no free lunch. > > > I have read and learned that the advantage of LSFEM is that it will > always > > give you an SPD system, even for non-linear problems (because what you do > > is linearize the problem first and then minimize/take the Gateaux > > derivative to get the weak form). But after talking to some people and > > reading some stuff online, it seems one could also make non SPD systems > SPD > > (hence eliminating what may be the only advantage of LSFEM). > > (Symmetric) saddle point problems have an SPD Schur complement. Schur > complements are dense in general, but some discretization choices (e.g., > quadrature in BDM) can make the primal block diagonal or block-diagonal, > resulting in a sparse Schur complement. If the Schur complement is > dense, you might be able to approximate it by a sparse matrix. The > quality of such an approximation depends on the physics and the > discretization. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Dec 11 13:15:52 2015 From: jed at jedbrown.org (Jed Brown) Date: Fri, 11 Dec 2015 12:15:52 -0700 Subject: [petsc-users] Solving/creating SPD systems In-Reply-To: References: <20151128163526.GA4917@Patricks-MBP-4.railnet.train> <878u51mxl8.fsf@jedbrown.org> Message-ID: <87zixg23vb.fsf@jedbrown.org> Justin Chang writes: > Jed, > > 1) What exactly are the PETSc options for CGNE? -ksp_type cgne (Conjugate Gradients on the Normal Equations) > Also, would LSQR be worth trying? I am doing all of this through > Firedrake, so I hope these things can be done directly through simply > providing command line PETSc options :) You can try, but I think this line of thinking is getting off in the weeds. > 2) So i spoke with Matt the other day, and the primary issue I am having > with LSFEM is finding a suitable preconditioner for the problematic penalty > term in Darcy's equation (i.e., the div-div term). So if I had this: > > (u, v) + (u, grad(q)) + (grad(p), v) + (grad(p), grad(q)) + (div(u), > (div(v)) = (rho*b, v + grad(q)) > > If I remove the div-div term, I have a very nice SPD system which could > simply be solved with CG/HYPRE. Do you know of any good preconditioning > strategies for this type of problem? That term is singular, so if the penalty is strong, it will be a bear to solve. Penalties suck. Sometimes you can add more variables to get better compatibility. See FOSLL*, for example. My opinion is that least squares methods are riddled with lame compromises and tradeoffs that you shouldn't have to make. If you want something robust, use compatible spaces and (possibly) deal with the fact that you are solving a saddle point problem. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Fri Dec 11 13:32:15 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 11 Dec 2015 13:32:15 -0600 Subject: [petsc-users] SPRNG package In-Reply-To: <1689D723-AE19-4118-B33B-48DFABDDBBD1@gmail.com> References: <32444DF9-350A-43BC-9400-55CB6FAC84ED@gmail.com> <1689D723-AE19-4118-B33B-48DFABDDBBD1@gmail.com> Message-ID: <179E5948-DF03-40DF-B6C9-3A113DD4C0E3@mcs.anl.gov> To answer your question If you look in config/BuildSystem/config/packages/sprng.py you'll see cflags += ' ' + '-DSPRNG_MPI' # either using MPI or MPIUNI so it looks like --download-sprng does install the MPI version of sprng that you should be able to use the parallel sprng functionality from your code. Barry > On Dec 11, 2015, at 10:47 AM, Gideon Simpson wrote: > > Ok, but if I look at > > http://www.mcs.anl.gov/petsc/petsc-dev/src/sys/classes/random/impls/sprng/sprng.c > > there is a comment: > > This is NOT currently using a parallel random number generator. Sprng does have > an MPI version we should investigate. > > > > -gideon > >> On Dec 11, 2015, at 11:30 AM, Hong wrote: >> >> Gideon: >> I was looking at the source files and noticed in the comments that when petsc is built with sprng, the petsc interface isn?t to the parallel sprng RNG. But is the package built with MPI, so that I could manually use the parallel RNG? >> Petsc- sprng-1.0 interface was written many years ago. >> It is for parallel computation. Students contributed an example at >> petsc/src/sys/classes/random/examples/tutorials/ex2.c >> >> Very few users have ever used this interface. >> If you encounter any problem, please report to us. >> >> Hong >> > From hzhang at mcs.anl.gov Fri Dec 11 13:40:20 2015 From: hzhang at mcs.anl.gov (Hong) Date: Fri, 11 Dec 2015 13:40:20 -0600 Subject: [petsc-users] SPRNG package In-Reply-To: <179E5948-DF03-40DF-B6C9-3A113DD4C0E3@mcs.anl.gov> References: <32444DF9-350A-43BC-9400-55CB6FAC84ED@gmail.com> <1689D723-AE19-4118-B33B-48DFABDDBBD1@gmail.com> <179E5948-DF03-40DF-B6C9-3A113DD4C0E3@mcs.anl.gov> Message-ID: Barry : > > > there is a comment: > > > > This is NOT currently using a parallel random number generator. Sprng > does have > > an MPI version we should investigate. > Shall we remove this comment? Hong > > >> On Dec 11, 2015, at 11:30 AM, Hong wrote: > >> > >> Gideon: > >> I was looking at the source files and noticed in the comments that when > petsc is built with sprng, the petsc interface isn?t to the parallel sprng > RNG. But is the package built with MPI, so that I could manually use the > parallel RNG? > >> Petsc- sprng-1.0 interface was written many years ago. > >> It is for parallel computation. Students contributed an example at > >> petsc/src/sys/classes/random/examples/tutorials/ex2.c > >> > >> Very few users have ever used this interface. > >> If you encounter any problem, please report to us. > >> > >> Hong > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gideon.simpson at gmail.com Fri Dec 11 13:51:29 2015 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Fri, 11 Dec 2015 14:51:29 -0500 Subject: [petsc-users] SPRNG package In-Reply-To: References: <32444DF9-350A-43BC-9400-55CB6FAC84ED@gmail.com> <1689D723-AE19-4118-B33B-48DFABDDBBD1@gmail.com> <179E5948-DF03-40DF-B6C9-3A113DD4C0E3@mcs.anl.gov> Message-ID: It seems confused. You can compile SPRNG with/without MPI. However, even if you compile it with MPI support, it can still be run as a serial RNG. -gideon > On Dec 11, 2015, at 2:40 PM, Hong wrote: > > Barry : > > there is a comment: > > > > This is NOT currently using a parallel random number generator. Sprng does have > > an MPI version we should investigate. > > Shall we remove this comment? > Hong > > >> On Dec 11, 2015, at 11:30 AM, Hong > wrote: > >> > >> Gideon: > >> I was looking at the source files and noticed in the comments that when petsc is built with sprng, the petsc interface isn?t to the parallel sprng RNG. But is the package built with MPI, so that I could manually use the parallel RNG? > >> Petsc- sprng-1.0 interface was written many years ago. > >> It is for parallel computation. Students contributed an example at > >> petsc/src/sys/classes/random/examples/tutorials/ex2.c > >> > >> Very few users have ever used this interface. > >> If you encounter any problem, please report to us. > >> > >> Hong > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jychang48 at gmail.com Fri Dec 11 15:47:05 2015 From: jychang48 at gmail.com (Justin Chang) Date: Fri, 11 Dec 2015 14:47:05 -0700 Subject: [petsc-users] Solving/creating SPD systems In-Reply-To: <87zixg23vb.fsf@jedbrown.org> References: <20151128163526.GA4917@Patricks-MBP-4.railnet.train> <878u51mxl8.fsf@jedbrown.org> <87zixg23vb.fsf@jedbrown.org> Message-ID: Pardon me for my apparent lack of understanding over what may be simple concepts, but why is div[u]*div[v] singular in the context of LSFEM? On Fri, Dec 11, 2015 at 12:15 PM, Jed Brown wrote: > Justin Chang writes: > > > Jed, > > > > 1) What exactly are the PETSc options for CGNE? > > -ksp_type cgne > > (Conjugate Gradients on the Normal Equations) > > > Also, would LSQR be worth trying? I am doing all of this through > > Firedrake, so I hope these things can be done directly through simply > > providing command line PETSc options :) > > You can try, but I think this line of thinking is getting off in the weeds. > > > 2) So i spoke with Matt the other day, and the primary issue I am having > > with LSFEM is finding a suitable preconditioner for the problematic > penalty > > term in Darcy's equation (i.e., the div-div term). So if I had this: > > > > (u, v) + (u, grad(q)) + (grad(p), v) + (grad(p), grad(q)) + (div(u), > > (div(v)) = (rho*b, v + grad(q)) > > > > If I remove the div-div term, I have a very nice SPD system which could > > simply be solved with CG/HYPRE. Do you know of any good preconditioning > > strategies for this type of problem? > > That term is singular, so if the penalty is strong, it will be a bear to > solve. > > Penalties suck. > > Sometimes you can add more variables to get better compatibility. See > FOSLL*, for example. > > My opinion is that least squares methods are riddled with lame > compromises and tradeoffs that you shouldn't have to make. If you want > something robust, use compatible spaces and (possibly) deal with the > fact that you are solving a saddle point problem. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Dec 11 16:02:49 2015 From: jed at jedbrown.org (Jed Brown) Date: Fri, 11 Dec 2015 14:02:49 -0800 Subject: [petsc-users] Solving/creating SPD systems In-Reply-To: References: <20151128163526.GA4917@Patricks-MBP-4.railnet.train> <878u51mxl8.fsf@jedbrown.org> <87zixg23vb.fsf@jedbrown.org> Message-ID: <874mfo1w52.fsf@jedbrown.org> Justin Chang writes: > Pardon me for my apparent lack of understanding over what may be simple > concepts, but why is div[u]*div[v] singular in the context of LSFEM? The corresponding strong form is grad(div(u)). If u has n*d entries (in d dimensions -- it is a vector), div(u) has only n entries. That means grad(div(u)) must have a null space of dimension at least n*(d-1). If you a matrix A and a singular matrix B, then A + c*B cannot be well-conditioned for sufficiently large c. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Fri Dec 11 16:31:15 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 11 Dec 2015 16:31:15 -0600 Subject: [petsc-users] SPRNG package In-Reply-To: References: <32444DF9-350A-43BC-9400-55CB6FAC84ED@gmail.com> <1689D723-AE19-4118-B33B-48DFABDDBBD1@gmail.com> <179E5948-DF03-40DF-B6C9-3A113DD4C0E3@mcs.anl.gov> Message-ID: <121C3E17-9D36-4383-AA91-A1B4BD5C9E70@mcs.anl.gov> I believe that we compile sprng with MPI but PETSc uses sprng only as a sequential random number generator hence I believe the comment is correct and should be left. Barry > On Dec 11, 2015, at 1:51 PM, Gideon Simpson wrote: > > It seems confused. You can compile SPRNG with/without MPI. However, even if you compile it with MPI support, it can still be run as a serial RNG. > > -gideon > >> On Dec 11, 2015, at 2:40 PM, Hong wrote: >> >> Barry : >> > there is a comment: >> > >> > This is NOT currently using a parallel random number generator. Sprng does have >> > an MPI version we should investigate. >> >> Shall we remove this comment? >> Hong >> >> >> On Dec 11, 2015, at 11:30 AM, Hong wrote: >> >> >> >> Gideon: >> >> I was looking at the source files and noticed in the comments that when petsc is built with sprng, the petsc interface isn?t to the parallel sprng RNG. But is the package built with MPI, so that I could manually use the parallel RNG? >> >> Petsc- sprng-1.0 interface was written many years ago. >> >> It is for parallel computation. Students contributed an example at >> >> petsc/src/sys/classes/random/examples/tutorials/ex2.c >> >> >> >> Very few users have ever used this interface. >> >> If you encounter any problem, please report to us. >> >> >> >> Hong >> >> >> > >> >> > From fdkong.jd at gmail.com Fri Dec 11 21:40:47 2015 From: fdkong.jd at gmail.com (Fande Kong) Date: Fri, 11 Dec 2015 20:40:47 -0700 Subject: [petsc-users] How to specify how many digits we could print using PetscPrintf Message-ID: Hi all, By default, PetscPrintf prints 6 digits. How to specify how many digits we could print using PetscPrintf? For example, I want to print 16 digits for a double value (PetscReal). Thanks, Fande, -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Dec 11 21:49:55 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 11 Dec 2015 21:49:55 -0600 Subject: [petsc-users] How to specify how many digits we could print using PetscPrintf In-Reply-To: References: Message-ID: <36D54217-4261-48EE-915A-3E8FFC7105CB@mcs.anl.gov> > On Dec 11, 2015, at 9:40 PM, Fande Kong wrote: > > Hi all, > > By default, PetscPrintf prints 6 digits. Not really, it uses the standard printf format commands. > How to specify how many digits we could print using PetscPrintf? For example, I want to print 16 digits for a double value (PetscReal). See for example http://pubs.opengroup.org/onlinepubs/009695399/functions/fprintf.html Barry > > > Thanks, > > Fande, From Eric.Chamberland at giref.ulaval.ca Fri Dec 11 22:18:32 2015 From: Eric.Chamberland at giref.ulaval.ca (=?UTF-8?Q?=c3=89ric_Chamberland?=) Date: Fri, 11 Dec 2015 23:18:32 -0500 Subject: [petsc-users] Release canditate? Message-ID: <566BA018.3060506@giref.ulaval.ca> Hi, we just waited since petsc 3.5.4 until 3.6.3 to retrieve a working petsc for our usages (got bugs with 3.6.1, 3.6.2) Now, just discovered 3.6.3 have a bug with mkl_pardiso... :/ We will have to wait until 3.6.4 (or patch the source with 3f7bb31 ... I was thinking that it could have been otherwise if release candidate packages would have been made available for anyone to test... and to give you feedback... and then a better official release... We can easily install it here and use the RC with our non-regression tests. Maybe other users would be interested, like me, to test the release candidates? Does that sounds interesting to you? Thanks, Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Dec 11 22:22:58 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 11 Dec 2015 22:22:58 -0600 Subject: [petsc-users] Release canditate? In-Reply-To: <566BA018.3060506@giref.ulaval.ca> References: <566BA018.3060506@giref.ulaval.ca> Message-ID: <6574689A-666B-4415-8735-478F400CBB60@mcs.anl.gov> Eric, Would it be possible for the release candidates to be a git repository branch or does it have to be a tarball? Generating and regenerating the tarball is a time consuming process which is why we don't use release candidates, but if you are able to test off the git repository we could certainly "pre announce" releases and allow testing before the actual release. Barry > On Dec 11, 2015, at 10:18 PM, ?ric Chamberland wrote: > > Hi, > > we just waited since petsc 3.5.4 until 3.6.3 to retrieve a working petsc for our usages (got bugs with 3.6.1, 3.6.2) > > Now, just discovered 3.6.3 have a bug with mkl_pardiso... :/ > > We will have to wait until 3.6.4 (or patch the source with 3f7bb31... > > I was thinking that it could have been otherwise if release candidate packages would have been made available for anyone to test... > and to give you feedback... and then a better official release... > > We can easily install it here and use the RC with our non-regression tests. > > Maybe other users would be interested, like me, to test the release candidates? > > Does that sounds interesting to you? > > Thanks, > > Eric > From Eric.Chamberland at giref.ulaval.ca Fri Dec 11 22:35:42 2015 From: Eric.Chamberland at giref.ulaval.ca (=?UTF-8?Q?=c3=89ric_Chamberland?=) Date: Fri, 11 Dec 2015 23:35:42 -0500 Subject: [petsc-users] Release canditate? In-Reply-To: <6574689A-666B-4415-8735-478F400CBB60@mcs.anl.gov> References: <566BA018.3060506@giref.ulaval.ca> <6574689A-666B-4415-8735-478F400CBB60@mcs.anl.gov> Message-ID: <566BA41E.3000802@giref.ulaval.ca> Le 2015-12-11 23:22, Barry Smith a ?crit : > Eric, > > Would it be possible for the release candidates to be a git repository branch or does it have to be a tarball? Generating and regenerating the for sure, if you want a maximum of RC testers, I think it should be a tarball... In my book, a RC is eventually the final release so everything must be the same... > tarball is a time consuming process which is why we don't use release candidates, but if you are able to test off the git repository we could ok, but hmmm, why isn't it possible to automate the making of the tarball? > certainly "pre announce" releases and allow testing before the actual release. If tarballs are impossible, I would certainly take the time to test a pre-announce release.. btw, the petsc-announce mailing list is somewhat silent... Thanks anyway to have evaluated the idea! :) Eric > > Barry > >> On Dec 11, 2015, at 10:18 PM, ?ric Chamberland wrote: >> >> Hi, >> >> we just waited since petsc 3.5.4 until 3.6.3 to retrieve a working petsc for our usages (got bugs with 3.6.1, 3.6.2) >> >> Now, just discovered 3.6.3 have a bug with mkl_pardiso... :/ >> >> We will have to wait until 3.6.4 (or patch the source with 3f7bb31... >> >> I was thinking that it could have been otherwise if release candidate packages would have been made available for anyone to test... >> and to give you feedback... and then a better official release... >> >> We can easily install it here and use the RC with our non-regression tests. >> >> Maybe other users would be interested, like me, to test the release candidates? >> >> Does that sounds interesting to you? >> >> Thanks, >> >> Eric >> From bsmith at mcs.anl.gov Fri Dec 11 22:40:45 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 11 Dec 2015 22:40:45 -0600 Subject: [petsc-users] Release canditate? In-Reply-To: <566BA41E.3000802@giref.ulaval.ca> References: <566BA018.3060506@giref.ulaval.ca> <6574689A-666B-4415-8735-478F400CBB60@mcs.anl.gov> <566BA41E.3000802@giref.ulaval.ca> Message-ID: > On Dec 11, 2015, at 10:35 PM, ?ric Chamberland wrote: > > > > Le 2015-12-11 23:22, Barry Smith a ?crit : >> Eric, >> >> Would it be possible for the release candidates to be a git repository branch or does it have to be a tarball? Generating and regenerating the > for sure, if you want a maximum of RC testers, I think it should be a tarball... In my book, a RC is eventually the final release so everything must be the same... >> tarball is a time consuming process which is why we don't use release candidates, but if you are able to test off the git repository we could > ok, but hmmm, why isn't it possible to automate the making of the tarball? > >> certainly "pre announce" releases and allow testing before the actual release. > If tarballs are impossible, I would certainly take the time to test a pre-announce release. Ok, we'll try to make rc tarballs and announce them on petsc-announce before the real release. Thanks for the suggestion. > > btw, the petsc-announce mailing list is somewhat silent... We try to use it only for releases. Barry > > Thanks anyway to have evaluated the idea! :) > > Eric > >> >> Barry >> >>> On Dec 11, 2015, at 10:18 PM, ?ric Chamberland wrote: >>> >>> Hi, >>> >>> we just waited since petsc 3.5.4 until 3.6.3 to retrieve a working petsc for our usages (got bugs with 3.6.1, 3.6.2) >>> >>> Now, just discovered 3.6.3 have a bug with mkl_pardiso... :/ >>> >>> We will have to wait until 3.6.4 (or patch the source with 3f7bb31... >>> >>> I was thinking that it could have been otherwise if release candidate packages would have been made available for anyone to test... >>> and to give you feedback... and then a better official release... >>> >>> We can easily install it here and use the RC with our non-regression tests. >>> >>> Maybe other users would be interested, like me, to test the release candidates? >>> >>> Does that sounds interesting to you? >>> >>> Thanks, >>> >>> Eric >>> > From balay at mcs.anl.gov Fri Dec 11 22:43:31 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 11 Dec 2015 22:43:31 -0600 Subject: [petsc-users] Release canditate? In-Reply-To: <566BA41E.3000802@giref.ulaval.ca> References: <566BA018.3060506@giref.ulaval.ca> <6574689A-666B-4415-8735-478F400CBB60@mcs.anl.gov> <566BA41E.3000802@giref.ulaval.ca> Message-ID: We do autmoatically generate tarballs everynight - its avaliable at: http://ftp.mcs.anl.gov/pub/petsc/petsc-master.tar.gz [but we don't maintain snapshots - i.e old tarballs for this] However adding in 'release' strings is a manual process - so we do this at the release time. [so we would have to do this stuff for rc] Here is a counter argument against this proposal. For folks interested in RC - could consider [for eg] 3.6.0 as RC - and test it out - and have all the issues discovered by 3.6.1. But this usually doesn't hapeen [and we have 3.6.1, 3.6.2, 3.6.3, 3.6.4 etc..] Also testing against master would catch issues early on - and not wait until the rc/release.. Satish On Fri, 11 Dec 2015, ?ric Chamberland wrote: > > > Le 2015-12-11 23:22, Barry Smith a ?crit : > > Eric, > > > > Would it be possible for the release candidates to be a git repository > > branch or does it have to be a tarball? Generating and regenerating the > for sure, if you want a maximum of RC testers, I think it should be a > tarball... In my book, a RC is eventually the final release so everything > must be the same... > > tarball is a time consuming process which is why we don't use release > > candidates, but if you are able to test off the git repository we could > ok, but hmmm, why isn't it possible to automate the making of the tarball? > > > certainly "pre announce" releases and allow testing before the actual > > release. > If tarballs are impossible, I would certainly take the time to test a > pre-announce release.. > > btw, the petsc-announce mailing list is somewhat silent... > > Thanks anyway to have evaluated the idea! :) > > Eric > > > > > Barry > > > > > On Dec 11, 2015, at 10:18 PM, ?ric Chamberland > > > wrote: > > > > > > Hi, > > > > > > we just waited since petsc 3.5.4 until 3.6.3 to retrieve a working petsc > > > for our usages (got bugs with 3.6.1, 3.6.2) > > > > > > Now, just discovered 3.6.3 have a bug with mkl_pardiso... :/ > > > > > > We will have to wait until 3.6.4 (or patch the source with 3f7bb31... > > > > > > I was thinking that it could have been otherwise if release candidate > > > packages would have been made available for anyone to test... > > > and to give you feedback... and then a better official release... > > > > > > We can easily install it here and use the RC with our non-regression > > > tests. > > > > > > Maybe other users would be interested, like me, to test the release > > > candidates? > > > > > > Does that sounds interesting to you? > > > > > > Thanks, > > > > > > Eric > > > > > From jed at jedbrown.org Sat Dec 12 01:27:13 2015 From: jed at jedbrown.org (Jed Brown) Date: Fri, 11 Dec 2015 23:27:13 -0800 Subject: [petsc-users] Release canditate? In-Reply-To: <566BA018.3060506@giref.ulaval.ca> References: <566BA018.3060506@giref.ulaval.ca> Message-ID: <87oadwyvn2.fsf@jedbrown.org> ?ric Chamberland writes: > Hi, > > we just waited since petsc 3.5.4 until 3.6.3 to retrieve a working petsc > for our usages (got bugs with 3.6.1, 3.6.2) > > Now, just discovered 3.6.3 have a bug with mkl_pardiso... :/ v3.6.0 is now 6 months old, but the mkl_pardiso stuff was just fixed recently. Fewer people actually try release candidates than real releases, so if we had spent the last 6 months spinning release candidates, it might still not have been noticed. It seems to me the problem is that we have untested code such that a bug like this can persist so long without being noticed. I don't see how release candidates help in this regard. Note that running 'master' after a feature freeze is like running a release candidate except that bugs are fixed in a rolling fashion instead of in batches at the next release candidate. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From mfadams at lbl.gov Sat Dec 12 07:43:10 2015 From: mfadams at lbl.gov (Mark Adams) Date: Sat, 12 Dec 2015 08:43:10 -0500 Subject: [petsc-users] TSSetIFunction Message-ID: I use TSSetIFunction and put RHS stuff in here, and don't use TSSetRHSFunction. This works fine with beuler, but euler gives a zero solution, so I assume explicit solvers do not call TSSetIFunction. I assume "I" is for implicit, but I also have a DAE and "When solving DAEs you must use this [TSSetIFunction] function." I infer that I can not do DAEs explicitly, but I just wanted to check. Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Dec 12 12:08:24 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 12 Dec 2015 12:08:24 -0600 Subject: [petsc-users] TSSetIFunction In-Reply-To: References: Message-ID: <58A1829B-820A-419C-B403-92DFA7311DC8@mcs.anl.gov> > On Dec 12, 2015, at 7:43 AM, Mark Adams wrote: > > I use TSSetIFunction and put RHS stuff in here, and don't use TSSetRHSFunction. > > This works fine with beuler, but euler gives a zero solution, so I assume explicit solvers do not call TSSetIFunction. Correct > I assume "I" is for implicit, but I also have a DAE and "When solving DAEs you must use this [TSSetIFunction] function." I infer that I can not do DAEs explicitly, but I just wanted to check. Correct > > Thanks, > Mark From Eric.Chamberland at giref.ulaval.ca Sat Dec 12 12:18:57 2015 From: Eric.Chamberland at giref.ulaval.ca (Eric Chamberland) Date: Sat, 12 Dec 2015 13:18:57 -0500 Subject: [petsc-users] Release canditate? In-Reply-To: References: <566BA018.3060506@giref.ulaval.ca> <6574689A-666B-4415-8735-478F400CBB60@mcs.anl.gov> <566BA41E.3000802@giref.ulaval.ca> Message-ID: <566C6511.9090006@giref.ulaval.ca> Le 2015-12-11 23:43, Satish Balay a ?crit : > We do autmoatically generate tarballs everynight - its avaliable at: > http://ftp.mcs.anl.gov/pub/petsc/petsc-master.tar.gz [but we don't > maintain snapshots - i.e old tarballs for this] interesting! I didn't knew that... Then maybe creating a tarball for the "maint" branch and naming it with the date (or the sha), and then making it available on the "Download" page would be a more easy thing to do? How "far" or different is the master branch from the maint branch right now? I see 3f7bb31 in maint but it is not clear (in bitbucket) if it is in master...? > However adding in 'release' strings is a manual process - so we do > this at the release time. [so we would have to do this stuff for rc] > > Here is a counter argument against this proposal. For folks interested > in RC - could consider [for eg] 3.6.0 as RC - and test it out - and > have all the issues discovered by 3.6.1. But this usually doesn't > hapeen [and we have 3.6.1, 3.6.2, 3.6.3, 3.6.4 etc..] > > Also testing against master would catch issues early on - and not wait > until the rc/release.. Yep, but isn't Petsc master far away from maint? Thanks! Eric ps: there is a typo here at the bottom (http://www.mcs.anl.gov/petsc/developers/): "The nightly tarball ... the doumentation built. ..." -------------- next part -------------- An HTML attachment was scrubbed... URL: From gideon.simpson at gmail.com Sat Dec 12 14:09:40 2015 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Sat, 12 Dec 2015 15:09:40 -0500 Subject: [petsc-users] SPRNG package In-Reply-To: <121C3E17-9D36-4383-AA91-A1B4BD5C9E70@mcs.anl.gov> References: <32444DF9-350A-43BC-9400-55CB6FAC84ED@gmail.com> <1689D723-AE19-4118-B33B-48DFABDDBBD1@gmail.com> <179E5948-DF03-40DF-B6C9-3A113DD4C0E3@mcs.anl.gov> <121C3E17-9D36-4383-AA91-A1B4BD5C9E70@mcs.anl.gov> Message-ID: <8582DA66-EF59-488D-AC61-67FCDAE4CAB0@gmail.com> Something about this continues to confuse me. I?m attaching two simple examples using SPRNG+PETSc, which generate the same results. Indeed, when run with mpi with more than one process, they generate different sequences of random numbers on reach process, but how is it handling this, if it?s not using SPRNG?s parallel RNG? -gideon > On Dec 11, 2015, at 5:31 PM, Barry Smith wrote: > > > I believe that we compile sprng with MPI but PETSc uses sprng only as a sequential random number generator hence I believe the comment is correct and should be left. > > Barry > >> On Dec 11, 2015, at 1:51 PM, Gideon Simpson wrote: >> >> It seems confused. You can compile SPRNG with/without MPI. However, even if you compile it with MPI support, it can still be run as a serial RNG. >> >> -gideon >> >>> On Dec 11, 2015, at 2:40 PM, Hong wrote: >>> >>> Barry : >>>> there is a comment: >>>> >>>> This is NOT currently using a parallel random number generator. Sprng does have >>>> an MPI version we should investigate. >>> >>> Shall we remove this comment? >>> Hong >>> >>>>> On Dec 11, 2015, at 11:30 AM, Hong wrote: >>>>> >>>>> Gideon: >>>>> I was looking at the source files and noticed in the comments that when petsc is built with sprng, the petsc interface isn?t to the parallel sprng RNG. But is the package built with MPI, so that I could manually use the parallel RNG? >>>>> Petsc- sprng-1.0 interface was written many years ago. >>>>> It is for parallel computation. Students contributed an example at >>>>> petsc/src/sys/classes/random/examples/tutorials/ex2.c >>>>> >>>>> Very few users have ever used this interface. >>>>> If you encounter any problem, please report to us. >>>>> >>>>> Hong >>>>> >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: petsc_sprng1.c Type: application/octet-stream Size: 633 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: petsc_sprng2.c Type: application/octet-stream Size: 797 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Dec 12 14:17:14 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 12 Dec 2015 14:17:14 -0600 Subject: [petsc-users] SPRNG package In-Reply-To: <8582DA66-EF59-488D-AC61-67FCDAE4CAB0@gmail.com> References: <32444DF9-350A-43BC-9400-55CB6FAC84ED@gmail.com> <1689D723-AE19-4118-B33B-48DFABDDBBD1@gmail.com> <179E5948-DF03-40DF-B6C9-3A113DD4C0E3@mcs.anl.gov> <121C3E17-9D36-4383-AA91-A1B4BD5C9E70@mcs.anl.gov> <8582DA66-EF59-488D-AC61-67FCDAE4CAB0@gmail.com> Message-ID: > On Dec 12, 2015, at 2:09 PM, Gideon Simpson wrote: > > Something about this continues to confuse me. I?m attaching two simple examples using SPRNG+PETSc, which generate the same results. Indeed, when run with mpi with more than one process, they generate different sequences of random numbers on reach process, but how is it handling this, if it?s not using SPRNG?s parallel RNG? We just put a different seed based on the MPI rank for each process; we do this for all the random number generators we use*. Note the PETSc code that calls sprng is very short and simple and easily understood /src/sys/classes/random/impls/sprng/sprng.c Barry * I am not claiming this is a good idea, it is just what we do. > > -gideon > > >> On Dec 11, 2015, at 5:31 PM, Barry Smith wrote: >> >> >> I believe that we compile sprng with MPI but PETSc uses sprng only as a sequential random number generator hence I believe the comment is correct and should be left. >> >> Barry >> >>> On Dec 11, 2015, at 1:51 PM, Gideon Simpson wrote: >>> >>> It seems confused. You can compile SPRNG with/without MPI. However, even if you compile it with MPI support, it can still be run as a serial RNG. >>> >>> -gideon >>> >>>> On Dec 11, 2015, at 2:40 PM, Hong wrote: >>>> >>>> Barry : >>>>> there is a comment: >>>>> >>>>> This is NOT currently using a parallel random number generator. Sprng does have >>>>> an MPI version we should investigate. >>>> >>>> Shall we remove this comment? >>>> Hong >>>> >>>>>> On Dec 11, 2015, at 11:30 AM, Hong wrote: >>>>>> >>>>>> Gideon: >>>>>> I was looking at the source files and noticed in the comments that when petsc is built with sprng, the petsc interface isn?t to the parallel sprng RNG. But is the package built with MPI, so that I could manually use the parallel RNG? >>>>>> Petsc- sprng-1.0 interface was written many years ago. >>>>>> It is for parallel computation. Students contributed an example at >>>>>> petsc/src/sys/classes/random/examples/tutorials/ex2.c >>>>>> >>>>>> Very few users have ever used this interface. >>>>>> If you encounter any problem, please report to us. >>>>>> >>>>>> Hong >>>>>> >>>>> >>>> >>>> >>> >> > > From gideon.simpson at gmail.com Sat Dec 12 14:22:10 2015 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Sat, 12 Dec 2015 15:22:10 -0500 Subject: [petsc-users] SPRNG package In-Reply-To: References: <32444DF9-350A-43BC-9400-55CB6FAC84ED@gmail.com> <1689D723-AE19-4118-B33B-48DFABDDBBD1@gmail.com> <179E5948-DF03-40DF-B6C9-3A113DD4C0E3@mcs.anl.gov> <121C3E17-9D36-4383-AA91-A1B4BD5C9E70@mcs.anl.gov> <8582DA66-EF59-488D-AC61-67FCDAE4CAB0@gmail.com> Message-ID: Thanks for clarifying. Yes, I?d argue the distinct seeds on the processes is an unsafe way to handle parallel RNG. -gideon > On Dec 12, 2015, at 3:17 PM, Barry Smith wrote: > > >> On Dec 12, 2015, at 2:09 PM, Gideon Simpson wrote: >> >> Something about this continues to confuse me. I?m attaching two simple examples using SPRNG+PETSc, which generate the same results. Indeed, when run with mpi with more than one process, they generate different sequences of random numbers on reach process, but how is it handling this, if it?s not using SPRNG?s parallel RNG? > > We just put a different seed based on the MPI rank for each process; we do this for all the random number generators we use*. Note the PETSc code that calls sprng is very short and simple and easily understood /src/sys/classes/random/impls/sprng/sprng.c > > Barry > > * I am not claiming this is a good idea, it is just what we do. >> >> -gideon >> >> >>> On Dec 11, 2015, at 5:31 PM, Barry Smith wrote: >>> >>> >>> I believe that we compile sprng with MPI but PETSc uses sprng only as a sequential random number generator hence I believe the comment is correct and should be left. >>> >>> Barry >>> >>>> On Dec 11, 2015, at 1:51 PM, Gideon Simpson wrote: >>>> >>>> It seems confused. You can compile SPRNG with/without MPI. However, even if you compile it with MPI support, it can still be run as a serial RNG. >>>> >>>> -gideon >>>> >>>>> On Dec 11, 2015, at 2:40 PM, Hong wrote: >>>>> >>>>> Barry : >>>>>> there is a comment: >>>>>> >>>>>> This is NOT currently using a parallel random number generator. Sprng does have >>>>>> an MPI version we should investigate. >>>>> >>>>> Shall we remove this comment? >>>>> Hong >>>>> >>>>>>> On Dec 11, 2015, at 11:30 AM, Hong wrote: >>>>>>> >>>>>>> Gideon: >>>>>>> I was looking at the source files and noticed in the comments that when petsc is built with sprng, the petsc interface isn?t to the parallel sprng RNG. But is the package built with MPI, so that I could manually use the parallel RNG? >>>>>>> Petsc- sprng-1.0 interface was written many years ago. >>>>>>> It is for parallel computation. Students contributed an example at >>>>>>> petsc/src/sys/classes/random/examples/tutorials/ex2.c >>>>>>> >>>>>>> Very few users have ever used this interface. >>>>>>> If you encounter any problem, please report to us. >>>>>>> >>>>>>> Hong >>>>>>> >>>>>> >>>>> >>>>> >>>> >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Dec 12 18:12:15 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 12 Dec 2015 18:12:15 -0600 Subject: [petsc-users] SPRNG package In-Reply-To: References: <32444DF9-350A-43BC-9400-55CB6FAC84ED@gmail.com> <1689D723-AE19-4118-B33B-48DFABDDBBD1@gmail.com> <179E5948-DF03-40DF-B6C9-3A113DD4C0E3@mcs.anl.gov> <121C3E17-9D36-4383-AA91-A1B4BD5C9E70@mcs.anl.gov> <8582DA66-EF59-488D-AC61-67FCDAE4CAB0@gmail.com> Message-ID: Yes, from the manual page: This is only a primative "parallel" random number generator, it should NOT be used for sophisticated parallel Monte Carlo methods since it will very likely not have the correct statistics across processors. You can provide your own parallel generator using PetscRandomRegister(); > On Dec 12, 2015, at 2:22 PM, Gideon Simpson wrote: > > Thanks for clarifying. Yes, I?d argue the distinct seeds on the processes is an unsafe way to handle parallel RNG. > > -gideon > >> On Dec 12, 2015, at 3:17 PM, Barry Smith wrote: >> >> >>> On Dec 12, 2015, at 2:09 PM, Gideon Simpson wrote: >>> >>> Something about this continues to confuse me. I?m attaching two simple examples using SPRNG+PETSc, which generate the same results. Indeed, when run with mpi with more than one process, they generate different sequences of random numbers on reach process, but how is it handling this, if it?s not using SPRNG?s parallel RNG? >> >> We just put a different seed based on the MPI rank for each process; we do this for all the random number generators we use*. Note the PETSc code that calls sprng is very short and simple and easily understood /src/sys/classes/random/impls/sprng/sprng.c >> >> Barry >> >> * I am not claiming this is a good idea, it is just what we do. >>> >>> -gideon >>> >>> >>>> On Dec 11, 2015, at 5:31 PM, Barry Smith wrote: >>>> >>>> >>>> I believe that we compile sprng with MPI but PETSc uses sprng only as a sequential random number generator hence I believe the comment is correct and should be left. >>>> >>>> Barry >>>> >>>>> On Dec 11, 2015, at 1:51 PM, Gideon Simpson wrote: >>>>> >>>>> It seems confused. You can compile SPRNG with/without MPI. However, even if you compile it with MPI support, it can still be run as a serial RNG. >>>>> >>>>> -gideon >>>>> >>>>>> On Dec 11, 2015, at 2:40 PM, Hong wrote: >>>>>> >>>>>> Barry : >>>>>>> there is a comment: >>>>>>> >>>>>>> This is NOT currently using a parallel random number generator. Sprng does have >>>>>>> an MPI version we should investigate. >>>>>> >>>>>> Shall we remove this comment? >>>>>> Hong >>>>>> >>>>>>>> On Dec 11, 2015, at 11:30 AM, Hong wrote: >>>>>>>> >>>>>>>> Gideon: >>>>>>>> I was looking at the source files and noticed in the comments that when petsc is built with sprng, the petsc interface isn?t to the parallel sprng RNG. But is the package built with MPI, so that I could manually use the parallel RNG? >>>>>>>> Petsc- sprng-1.0 interface was written many years ago. >>>>>>>> It is for parallel computation. Students contributed an example at >>>>>>>> petsc/src/sys/classes/random/examples/tutorials/ex2.c >>>>>>>> >>>>>>>> Very few users have ever used this interface. >>>>>>>> If you encounter any problem, please report to us. >>>>>>>> >>>>>>>> Hong >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>> >> > From gideon.simpson at gmail.com Sat Dec 12 21:38:05 2015 From: gideon.simpson at gmail.com (gideon.simpson at gmail.com) Date: Sat, 12 Dec 2015 22:38:05 -0500 Subject: [petsc-users] SPRNG package In-Reply-To: References: <32444DF9-350A-43BC-9400-55CB6FAC84ED@gmail.com> <1689D723-AE19-4118-B33B-48DFABDDBBD1@gmail.com> <179E5948-DF03-40DF-B6C9-3A113DD4C0E3@mcs.anl.gov> <121C3E17-9D36-4383-AA91-A1B4BD5C9E70@mcs.anl.gov> <8582DA66-EF59-488D-AC61-67FCDAE4CAB0@gmail.com> Message-ID: <3059E1B7-B7B7-4D39-84DF-8596675FD40E@gmail.com> Maybe the online documentation should be better labeled. When I saw that PETSc had SPRN support, I assumed that meant it had access to the parallel rng. > On Dec 12, 2015, at 7:12 PM, Barry Smith wrote: > > > Yes, from the manual page: This is only a primative "parallel" random number generator, it should NOT be used for sophisticated parallel Monte Carlo methods since it will very likely not have the correct statistics across processors. You can provide your own parallel generator using PetscRandomRegister(); > > >> On Dec 12, 2015, at 2:22 PM, Gideon Simpson wrote: >> >> Thanks for clarifying. Yes, I?d argue the distinct seeds on the processes is an unsafe way to handle parallel RNG. >> >> -gideon >> >>> On Dec 12, 2015, at 3:17 PM, Barry Smith wrote: >>> >>> >>>> On Dec 12, 2015, at 2:09 PM, Gideon Simpson wrote: >>>> >>>> Something about this continues to confuse me. I?m attaching two simple examples using SPRNG+PETSc, which generate the same results. Indeed, when run with mpi with more than one process, they generate different sequences of random numbers on reach process, but how is it handling this, if it?s not using SPRNG?s parallel RNG? >>> >>> We just put a different seed based on the MPI rank for each process; we do this for all the random number generators we use*. Note the PETSc code that calls sprng is very short and simple and easily understood /src/sys/classes/random/impls/sprng/sprng.c >>> >>> Barry >>> >>> * I am not claiming this is a good idea, it is just what we do. >>>> >>>> -gideon >>>> >>>> >>>>> On Dec 11, 2015, at 5:31 PM, Barry Smith wrote: >>>>> >>>>> >>>>> I believe that we compile sprng with MPI but PETSc uses sprng only as a sequential random number generator hence I believe the comment is correct and should be left. >>>>> >>>>> Barry >>>>> >>>>>> On Dec 11, 2015, at 1:51 PM, Gideon Simpson wrote: >>>>>> >>>>>> It seems confused. You can compile SPRNG with/without MPI. However, even if you compile it with MPI support, it can still be run as a serial RNG. >>>>>> >>>>>> -gideon >>>>>> >>>>>>> On Dec 11, 2015, at 2:40 PM, Hong wrote: >>>>>>> >>>>>>> Barry : >>>>>>>> there is a comment: >>>>>>>> >>>>>>>> This is NOT currently using a parallel random number generator. Sprng does have >>>>>>>> an MPI version we should investigate. >>>>>>> >>>>>>> Shall we remove this comment? >>>>>>> Hong >>>>>>> >>>>>>>>> On Dec 11, 2015, at 11:30 AM, Hong wrote: >>>>>>>>> >>>>>>>>> Gideon: >>>>>>>>> I was looking at the source files and noticed in the comments that when petsc is built with sprng, the petsc interface isn?t to the parallel sprng RNG. But is the package built with MPI, so that I could manually use the parallel RNG? >>>>>>>>> Petsc- sprng-1.0 interface was written many years ago. >>>>>>>>> It is for parallel computation. Students contributed an example at >>>>>>>>> petsc/src/sys/classes/random/examples/tutorials/ex2.c >>>>>>>>> >>>>>>>>> Very few users have ever used this interface. >>>>>>>>> If you encounter any problem, please report to us. >>>>>>>>> >>>>>>>>> Hong >>>> >>>> > From wadud.miah at gmail.com Sun Dec 13 15:38:01 2015 From: wadud.miah at gmail.com (W. Miah) Date: Sun, 13 Dec 2015 21:38:01 +0000 Subject: [petsc-users] PETSc runtime error Message-ID: Hello, I am getting the following error: [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message ------------------------------------ [3]PETSC ERROR: Invalid argument! [3]PETSC ERROR: Invalid object classid 0 This often happens if you compile with PETSC_USE_DYNAMIC_LIBRARIES, but link with static libraries.! [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------ I can't see why using static libraries should make any difference than using shared libraries. The configure options used were: [1]PETSC ERROR: Configure options --prefix=/home/wm/petsc-3.3.0-omp --with-blas-lapack-dir=/usr/local/intel_6/xe_2013_sp1.2.144/mkl/lib/intel64 --with-debugging=0 --with-fortran-interfaces=1 --with-openmp=1 --CC=mpiicc --CXX=mpiicpc --FC=mpiifort --with-x=0 It was built using intelmpi/intel/4.0.3 and intel/14.0.Any help will be greatly appreciated. Thanks in advance, Wadud. -- web: http://miahw.wordpress.com mobile: 07905 755604 gnupg: 2E29 B22F -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Dec 13 15:52:02 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 13 Dec 2015 15:52:02 -0600 Subject: [petsc-users] PETSc runtime error In-Reply-To: References: Message-ID: <9B9C2AB6-3983-4971-90B4-ACE9211BF561@mcs.anl.gov> Often this is a result of a mis-match between the mpi used and the mpiexec being used. Did make test work after you compiled PETSc? Make sure the mpiexec in your path corresponds to the mpiicc you are using. Send the configure.log and full error message Barry > On Dec 13, 2015, at 3:38 PM, W. Miah wrote: > > Hello, > > I am getting the following error: > > [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message ------------------------------------ > [3]PETSC ERROR: Invalid argument! > [3]PETSC ERROR: Invalid object classid 0 > This often happens if you compile with PETSC_USE_DYNAMIC_LIBRARIES, but link with static libraries.! > [3]PETSC ERROR: ------------------------------------------------------------------------ > [3]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------ > > I can't see why using static libraries should make any difference than using shared libraries. The configure options used were: > > [1]PETSC ERROR: Configure options --prefix=/home/wm/petsc-3.3.0-omp --with-blas-lapack-dir=/usr/local/intel_6/xe_2013_sp1.2.144/mkl/lib/intel64 --with-debugging=0 --with-fortran-interfaces=1 --with-openmp=1 --CC=mpiicc --CXX=mpiicpc --FC=mpiifort --with-x=0 > > It was built using intelmpi/intel/4.0.3 and intel/14.0.Any help will be greatly appreciated. > > Thanks in advance, > Wadud. > > -- > web: http://miahw.wordpress.com > mobile: 07905 755604 > gnupg: 2E29 B22F From balay at mcs.anl.gov Sun Dec 13 21:12:33 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 13 Dec 2015 21:12:33 -0600 Subject: [petsc-users] Release canditate? In-Reply-To: <566C6511.9090006@giref.ulaval.ca> References: <566BA018.3060506@giref.ulaval.ca> <6574689A-666B-4415-8735-478F400CBB60@mcs.anl.gov> <566BA41E.3000802@giref.ulaval.ca> <566C6511.9090006@giref.ulaval.ca> Message-ID: On Sat, 12 Dec 2015, Eric Chamberland wrote: > Le 2015-12-11 23:43, Satish Balay a ?crit : > > We do autmoatically generate tarballs everynight - its avaliable at: > > http://ftp.mcs.anl.gov/pub/petsc/petsc-master.tar.gz [but we don't > > maintain snapshots - i.e old tarballs for this] > interesting! I didn't knew that... Then maybe creating a tarball for the > "maint" branch and naming it with the date (or the sha), and then making it > available on the "Download" page would be a more easy thing to do? Not sure I understand. Why snapshots of 'maint' branch? But if its refering 'master' [which petsc-master.tar.gz is from] - archive all the snapshots - but its not clear if its worth it.. [maintain snapshots for 1 month? 1year? forever?] - esp when git is more convinent to bisect and to go back in history to find the buggy commit.. > > How "far" or different is the master branch from the maint branch right now? > I see 3f7bb31 > > in maint but it is not clear (in bitbucket) if it is in master...? The commit of interest is 7b37fee - and its already in master - but then - there were additional chagnes to mkl_pardiso in master [so this part of code is reorganized] But wrt your question - all commits in maint will be merged into master [immediately - or after sometime]. Ideally - fixes to 'maint' get tested first in 'next', then in 'master' - and then merged to 'maint'. > > However adding in 'release' strings is a manual process - so we do > > this at the release time. [so we would have to do this stuff for rc] > > > > Here is a counter argument against this proposal. For folks interested > > in RC - could consider [for eg] 3.6.0 as RC - and test it out - and > > have all the issues discovered by 3.6.1. But this usually doesn't > > hapeen [and we have 3.6.1, 3.6.2, 3.6.3, 3.6.4 etc..] > > > > Also testing against master would catch issues early on - and not wait > > until the rc/release.. > Yep, but isn't Petsc master far away from maint? Just to be clear: maint - maintainance release branch (primarily for patches to current release) i.e for 3.6.1, 3.6.2, 3.6.3 etc.. master - current stable development for future release 3.7 [i.e if interested in RC - you would be looking at master branch] Assuming you are thinking off 3.7.rc1, 3.7.rc2, 3.7.0, 3.7.1 etc as the versions [not 3.6.4.rc] > > Thanks! > > Eric > > ps: there is a typo here at the bottom > (http://www.mcs.anl.gov/petsc/developers/): > > "The nightly tarball ... the doumentation built. ..." Fixed now. Thanks, Satish From timothee.nicolas at gmail.com Mon Dec 14 01:09:46 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Mon, 14 Dec 2015 16:09:46 +0900 Subject: [petsc-users] Big discrepancy between machines Message-ID: Hi, I have noticed I have a VERY big difference in behaviour between two machines in my problem, solved with SNES. I can't explain it, because I have tested my operators which give the same result. I also checked that the vectors fed to the SNES are the same. The problem happens only with my shell preconditioner. When I don't use it, and simply solve using -snes_mf, I don't see anymore than the usual 3-4 changing digits at the end of the residuals. However, when I use my pcshell, the results are completely different between the two machines. I have attached output_SuperComputer.txt and output_DesktopComputer.txt, which correspond to the output from the exact same code and options (and of course same input data file !). More precisely output_SuperComputer.txt : output on a supercomputer called Helios, sorry I don't know the exact specs. In this case, the SNES norms are reduced successively: 0 SNES Function norm 4.867111712420e-03 1 SNES Function norm 5.632325929998e-08 2 SNES Function norm 7.427800084502e-15 output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 GHz Intel Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop with Mac OS X Mavericks). In this case, I obtain the following for the SNES norms, while in the other, I obtain 0 SNES Function norm 4.867111713544e-03 1 SNES Function norm 1.560094052222e-03 2 SNES Function norm 1.552118650943e-03 3 SNES Function norm 1.552106297094e-03 4 SNES Function norm 1.552106277949e-03 which I can't explain, because otherwise the KSP residual (with the same operator, which I checked) behave well. As you can see, the first time the preconditioner is applied (DB_, DP_, Drho_ and PS_ solves), the two outputs coincide (except for the few last digits, up to 9 actually, which is more than I would expect), and everything starts to diverge at the first print of the main KSP (the one stemming from the SNES) residual norms. Do you have an idea what may cause such a strange behaviour ? Best Timothee -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- 0 SNES Function norm 4.867111713544e-03 Residual norms for DB_ solve. 0 KSP Residual norm 2.366556734874e-07 1 KSP Residual norm 1.599134162118e-11 2 KSP Residual norm 1.379636472597e-15 Residual norms for DP_ solve. 0 KSP Residual norm 4.268082421881e-09 1 KSP Residual norm 3.147853422474e-14 Residual norms for Drho_ solve. 0 KSP Residual norm 1.548251260747e-08 1 KSP Residual norm 5.430867597212e-18 Residual norms for PS_ solve. 0 KSP Residual norm 4.867111515741e-03 1 KSP Residual norm 1.907710093690e-04 2 KSP Residual norm 7.658106671754e-06 3 KSP Residual norm 3.357005929454e-07 4 KSP Residual norm 1.454080306567e-08 0 KSP Residual norm 5.139313706047e-03 Residual norms for DB_ solve. 0 KSP Residual norm 5.440635448768e-05 1 KSP Residual norm 5.050541808202e-09 2 KSP Residual norm 4.288718765754e-13 Residual norms for DP_ solve. 0 KSP Residual norm 8.017413031372e-03 1 KSP Residual norm 1.517037052583e-08 Residual norms for Drho_ solve. 0 KSP Residual norm 3.019014439771e-01 1 KSP Residual norm 1.698740458365e-10 Residual norms for PS_ solve. 0 KSP Residual norm 9.470339285118e-01 1 KSP Residual norm 3.711941719696e-02 2 KSP Residual norm 1.490076291921e-03 3 KSP Residual norm 6.531888110917e-05 4 KSP Residual norm 2.829271710054e-06 1 KSP Residual norm 1.076619979170e-05 Residual norms for DB_ solve. 0 KSP Residual norm 1.455733053736e-02 1 KSP Residual norm 8.558119750766e-07 2 KSP Residual norm 6.221252391682e-11 Residual norms for DP_ solve. 0 KSP Residual norm 1.904297391721e-02 1 KSP Residual norm 8.633819203365e-08 Residual norms for Drho_ solve. 0 KSP Residual norm 9.771004397981e-01 1 KSP Residual norm 3.797547005550e-10 Residual norms for PS_ solve. 0 KSP Residual norm 2.100138029120e-01 1 KSP Residual norm 8.008499816230e-03 2 KSP Residual norm 3.188390248422e-04 3 KSP Residual norm 1.392814641696e-05 4 KSP Residual norm 6.017324204570e-07 2 KSP Residual norm 1.378553492814e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 1 SNES Function norm 1.560094052222e-03 Residual norms for DB_ solve. 0 KSP Residual norm 1.305901856652e-09 1 KSP Residual norm 1.016323286480e-13 2 KSP Residual norm 8.697452307336e-18 Residual norms for DP_ solve. 0 KSP Residual norm 4.141600765836e-05 1 KSP Residual norm 7.836080325484e-11 Residual norms for Drho_ solve. 0 KSP Residual norm 1.559532533569e-03 1 KSP Residual norm 8.775290484669e-13 Residual norms for PS_ solve. 0 KSP Residual norm 7.939953978965e-08 1 KSP Residual norm 3.091105675950e-09 2 KSP Residual norm 1.281590573300e-10 3 KSP Residual norm 5.678754805485e-12 4 KSP Residual norm 2.357983561332e-13 0 KSP Residual norm 7.983926898450e-06 Residual norms for DB_ solve. 0 KSP Residual norm 1.635766726710e-04 1 KSP Residual norm 1.273119897835e-08 2 KSP Residual norm 1.089516309227e-12 Residual norms for DP_ solve. 0 KSP Residual norm 2.656200473798e-02 1 KSP Residual norm 5.025904537481e-08 Residual norms for Drho_ solve. 0 KSP Residual norm 1.000150398902e+00 1 KSP Residual norm 5.627979304770e-10 Residual norms for PS_ solve. 0 KSP Residual norm 9.944957775471e-03 1 KSP Residual norm 3.871513600798e-04 2 KSP Residual norm 1.605196431847e-05 3 KSP Residual norm 7.112462681111e-07 4 KSP Residual norm 2.953388825860e-08 1 KSP Residual norm 2.200468803859e-10 Residual norms for DB_ solve. 0 KSP Residual norm 1.307412819177e-02 1 KSP Residual norm 9.578697048783e-07 2 KSP Residual norm 8.340922260300e-11 Residual norms for DP_ solve. 0 KSP Residual norm 2.721965031182e-02 1 KSP Residual norm 1.401958230926e-07 Residual norms for Drho_ solve. 0 KSP Residual norm 8.302667215672e-01 1 KSP Residual norm 1.116675867061e-09 Residual norms for PS_ solve. 0 KSP Residual norm 5.598451080918e-01 1 KSP Residual norm 2.202634919872e-02 2 KSP Residual norm 9.217499213474e-04 3 KSP Residual norm 4.080246263216e-05 4 KSP Residual norm 1.686626638957e-06 2 KSP Residual norm 1.603543671001e-13 Linear solve converged due to CONVERGED_RTOL iterations 2 2 SNES Function norm 1.552118650943e-03 Residual norms for DB_ solve. 0 KSP Residual norm 2.793650509227e-15 1 KSP Residual norm 2.146724819226e-19 2 KSP Residual norm 1.937772738260e-23 Residual norms for DP_ solve. 0 KSP Residual norm 4.120426859540e-05 1 KSP Residual norm 7.796021317362e-11 Residual norms for Drho_ solve. 0 KSP Residual norm 1.551559786739e-03 1 KSP Residual norm 8.730428515318e-13 Residual norms for PS_ solve. 0 KSP Residual norm 1.428033305389e-12 1 KSP Residual norm 5.807655876136e-14 2 KSP Residual norm 2.674763604785e-15 3 KSP Residual norm 1.196747820651e-16 4 KSP Residual norm 5.334942027622e-18 0 KSP Residual norm 1.237286805551e-08 Residual norms for DB_ solve. 0 KSP Residual norm 2.271987261067e-07 1 KSP Residual norm 1.742498472575e-11 2 KSP Residual norm 1.573543263716e-15 Residual norms for DP_ solve. 0 KSP Residual norm 2.654741283900e-02 1 KSP Residual norm 5.022943319228e-08 Residual norms for Drho_ solve. 0 KSP Residual norm 9.996528036365e-01 1 KSP Residual norm 5.624851138340e-10 Residual norms for PS_ solve. 0 KSP Residual norm 1.154281723143e-04 1 KSP Residual norm 4.693593763434e-06 2 KSP Residual norm 2.161231018351e-07 3 KSP Residual norm 9.671206291572e-09 4 KSP Residual norm 4.311554921541e-10 1 KSP Residual norm 3.016343311594e-14 Linear solve converged due to CONVERGED_RTOL iterations 1 3 SNES Function norm 1.552106297094e-03 Residual norms for DB_ solve. 0 KSP Residual norm 6.025886869455e-17 1 KSP Residual norm 3.683671084034e-21 2 KSP Residual norm 2.756059908370e-25 Residual norms for DP_ solve. 0 KSP Residual norm 4.120394063681e-05 1 KSP Residual norm 7.795959265451e-11 Residual norms for Drho_ solve. 0 KSP Residual norm 1.551547437335e-03 1 KSP Residual norm 8.730359027070e-13 Residual norms for PS_ solve. 0 KSP Residual norm 2.415967494271e-14 1 KSP Residual norm 1.029996000145e-15 2 KSP Residual norm 4.850784439943e-17 3 KSP Residual norm 2.034713843148e-18 4 KSP Residual norm 8.770156385840e-20 0 KSP Residual norm 1.917411615790e-11 Residual norms for DB_ solve. 0 KSP Residual norm 3.137430130807e-06 1 KSP Residual norm 1.918603503891e-10 2 KSP Residual norm 1.435073680190e-14 Residual norms for DP_ solve. 0 KSP Residual norm 2.654319443621e-02 1 KSP Residual norm 5.002189211387e-08 Residual norms for Drho_ solve. 0 KSP Residual norm 9.996525441184e-01 1 KSP Residual norm 5.623748967611e-10 Residual norms for PS_ solve. 0 KSP Residual norm 1.258077372494e-03 1 KSP Residual norm 5.363662191604e-05 2 KSP Residual norm 2.525980612687e-06 3 KSP Residual norm 1.059553063015e-07 4 KSP Residual norm 4.567015737839e-09 1 KSP Residual norm 1.130508373028e-16 Linear solve converged due to CONVERGED_RTOL iterations 1 4 SNES Function norm 1.552106277949e-03 Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 4 Total CPU time since PetscInitialize: 1.3750E+01 CPU time used for SNESSolve: 1.3354E+01 Number of linear iterations : 6 Number of function evaluations : 15 Kinetic Energy = 8.283578E-11 Magnetic Energy = 2.278864E-12 Exiting the main MHD Loop Deallocating remaining arrays Destroying remaining Petsc elements ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./miips on a arch-darwin-c-debug named iMac27Nicolas.nifs.ac.jp with 1 processor, by timotheenicolas Mon Dec 14 15:55:59 2015 Using Petsc Release Version 3.6.1, Jul, 22, 2015 Max Max/Min Avg Total Time (sec): 1.382e+01 1.00000 1.382e+01 Objects: 2.590e+02 1.00000 2.590e+02 Flops: 7.271e+08 1.00000 7.271e+08 7.271e+08 Flops/sec: 5.262e+07 1.00000 5.262e+07 5.262e+07 MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 MPI Reductions: 0.000e+00 0.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.3818e+01 100.0% 7.2707e+08 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage SNESSolve 1 1.0 1.3345e+01 1.0 7.06e+08 1.0 0.0e+00 0.0e+00 0.0e+00 97 97 0 0 0 97 97 0 0 0 53 SNESFunctionEval 15 1.0 2.5123e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 18 0 0 0 0 0 SNESJacobianEval 4 1.0 3.0715e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SNESLineSearch 4 1.0 1.3934e+00 1.0 7.14e+07 1.0 0.0e+00 0.0e+00 0.0e+00 10 10 0 0 0 10 10 0 0 0 51 VecView 2 1.0 8.8236e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecDot 4 1.0 5.5161e-03 1.0 8.92e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 1617 VecMDot 86 1.0 5.5805e-02 1.0 1.32e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 18 0 0 0 0 18 0 0 0 2368 VecNorm 141 1.0 5.4996e-02 1.0 1.25e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 17 0 0 0 0 17 0 0 0 2271 VecScale 200 1.0 5.6164e-02 1.0 8.92e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 12 0 0 0 0 12 0 0 0 1588 VecCopy 180 1.0 6.0994e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 242 1.0 1.5345e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecAXPY 119 1.0 7.6371e-02 1.0 1.20e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 17 0 0 0 1 17 0 0 0 1577 VecWAXPY 17 1.0 3.4016e-02 1.0 3.01e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 4 0 0 0 0 4 0 0 0 885 VecMAXPY 130 1.0 8.4107e-02 1.0 2.01e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 28 0 0 0 1 28 0 0 0 2393 VecAssemblyBegin 245 1.0 7.4085e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 245 1.0 2.5799e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 2 1.0 3.4161e-03 1.0 2.23e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 653 VecScatterBegin 260 1.0 2.5876e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 VecReduceArith 8 1.0 8.0670e-03 1.0 1.78e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 2212 VecReduceComm 4 1.0 2.7520e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 130 1.0 7.2848e-02 1.0 1.51e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 21 0 0 0 1 21 0 0 0 2067 MatMult MF 10 1.0 1.7229e+00 1.0 5.58e+07 1.0 0.0e+00 0.0e+00 0.0e+00 12 8 0 0 0 12 8 0 0 0 32 MatMult 110 1.0 1.1690e+01 1.0 1.06e+08 1.0 0.0e+00 0.0e+00 0.0e+00 85 15 0 0 0 85 15 0 0 0 9 MatMultAdd 40 1.0 2.7459e+00 1.0 3.35e+07 1.0 0.0e+00 0.0e+00 0.0e+00 20 5 0 0 0 20 5 0 0 0 12 MatAssemblyBegin 4 1.0 2.9000e-07 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 4 1.0 3.0516e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 86 1.0 1.0825e-01 1.0 2.64e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 36 0 0 0 1 36 0 0 0 2441 KSPSetUp 8 1.0 2.7075e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 4 1.0 1.1773e+01 1.0 6.32e+08 1.0 0.0e+00 0.0e+00 0.0e+00 85 87 0 0 0 85 87 0 0 0 54 PCSetUp 8 1.0 1.6327e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCApply 10 1.0 1.0665e+01 1.0 5.07e+08 1.0 0.0e+00 0.0e+00 0.0e+00 77 70 0 0 0 77 70 0 0 0 48 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage SNES 1 1 1324 0 SNESLineSearch 1 1 856 0 DMSNES 2 2 1312 0 Vector 169 169 710843024 0 Vector Scatter 8 8 5184 0 MatMFFD 1 1 768 0 Matrix 8 8 19376 0 Distributed Mesh 8 8 39152 0 Star Forest Bipartite Graph 16 16 13328 0 Discrete System 8 8 6720 0 Index Set 16 16 4045944 0 IS L to G Mapping 7 7 4037968 0 Krylov Solver 5 5 91760 0 DMKSP interface 1 1 640 0 Preconditioner 5 5 4312 0 Viewer 3 2 1520 0 ======================================================================================================================== Average time to get PetscTime(): 2.82002e-08 #PETSc Option Table entries: -DB_ksp_monitor -DB_pc_type none -DP_ksp_monitor -DP_pc_type none -Drho_ksp_monitor -Drho_pc_type none -PS_ksp_monitor -PS_pc_type none -ksp_converged_reason -ksp_monitor -log_summary -mult_lu -nts 1 -snes_converged_reason -snes_mf -snes_monitor #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --with-debugging=no ----------------------------------------- Libraries compiled on Mon Dec 14 14:57:32 2015 on iMac27Nicolas.nifs.ac.jp Machine characteristics: Darwin-14.5.0-x86_64-i386-64bit Using PETSc directory: /Users/timotheenicolas/PETSC/petsc-3.6.1 Using PETSc arch: arch-darwin-c-debug ----------------------------------------- Using C compiler: /Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/bin/mpif90 -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/include -I/Users/timotheenicolas/PETSC/petsc-3.6.1/include -I/Users/timotheenicolas/PETSC/petsc-3.6.1/include -I/Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/include -I/opt/X11/include ----------------------------------------- Using C linker: /Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/bin/mpicc Using Fortran linker: /Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/bin/mpif90 Using libraries: -Wl,-rpath,/Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/lib -L/Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/lib -lpetsc -Wl,-rpath,/Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/lib -L/Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/lib -lflapack -lfblas -Wl,-rpath,/opt/X11/lib -L/opt/X11/lib -lX11 -lssl -lcrypto -Wl,-rpath,/Library/Developer/CommandLineTools/usr/lib/clang/7.0.0/lib/darwin -L/Library/Developer/CommandLineTools/usr/lib/clang/7.0.0/lib/darwin -lmpifort -lgfortran -Wl,-rpath,/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0 -L/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0 -Wl,-rpath,/usr/local/lib -L/usr/local/lib -lgfortran -lgcc_ext.10.5 -lquadmath -lm -lclang_rt.osx -lmpicxx -lc++ -Wl,-rpath,/Library/Developer/CommandLineTools/usr/bin/../lib/clang/7.0.0/lib/darwin -L/Library/Developer/CommandLineTools/usr/bin/../lib/clang/7.0.0/lib/darwin -lclang_rt.osx -Wl,-rpath,/Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/lib -L/Users/timotheenicolas/PETSC/petsc-3.6.1/arch-darwin-c-debug/lib -ldl -lmpi -lpmpi -lSystem -Wl,-rpath,/Library/Developer/CommandLineTools/usr/bin/../lib/clang/7.0.0/lib/darwin -L/Library/Developer/CommandLineTools/usr/bin/../lib/clang/7.0.0/lib/darwin -lclang_rt.osx -ldl ----------------------------------------- -------------- next part -------------- 0 SNES Function norm 4.867111712420e-03 Residual norms for DB_ solve. 0 KSP Residual norm 2.366556734390e-07 1 KSP Residual norm 1.599134161658e-11 2 KSP Residual norm 1.379636472384e-15 Residual norms for DP_ solve. 0 KSP Residual norm 4.268082420930e-09 1 KSP Residual norm 3.147853421751e-14 Residual norms for Drho_ solve. 0 KSP Residual norm 1.548251260747e-08 1 KSP Residual norm 5.430867597213e-18 Residual norms for PS_ solve. 0 KSP Residual norm 4.867111514618e-03 1 KSP Residual norm 1.907717649118e-04 2 KSP Residual norm 7.658144026212e-06 3 KSP Residual norm 3.357066761239e-07 4 KSP Residual norm 1.454205241131e-08 0 KSP Residual norm 4.899341386700e-03 Residual norms for DB_ solve. 0 KSP Residual norm 5.707119370325e-05 1 KSP Residual norm 5.297917198531e-09 2 KSP Residual norm 4.498782273058e-13 Residual norms for DP_ solve. 0 KSP Residual norm 8.711633299940e-07 1 KSP Residual norm 6.424911624338e-12 Residual norms for Drho_ solve. 0 KSP Residual norm 3.160121194373e-06 1 KSP Residual norm 1.108489319887e-15 Residual norms for PS_ solve. 0 KSP Residual norm 9.934191997238e-01 1 KSP Residual norm 3.893765235821e-02 2 KSP Residual norm 1.563067404860e-03 3 KSP Residual norm 6.851945301781e-05 4 KSP Residual norm 2.968105385402e-06 1 KSP Residual norm 1.520860261048e-07 Residual norms for DB_ solve. 0 KSP Residual norm 1.029574455279e+00 1 KSP Residual norm 6.052311862765e-05 2 KSP Residual norm 4.397928200182e-09 Residual norms for DP_ solve. 0 KSP Residual norm 7.778726792599e-05 1 KSP Residual norm 4.659723031268e-10 Residual norms for Drho_ solve. 0 KSP Residual norm 3.464866626155e-07 1 KSP Residual norm 4.466560238078e-16 Residual norms for PS_ solve. 0 KSP Residual norm 2.922206828301e-01 1 KSP Residual norm 1.405945327578e-02 2 KSP Residual norm 7.875288422188e-04 3 KSP Residual norm 3.296403312814e-05 4 KSP Residual norm 1.378743485812e-06 2 KSP Residual norm 1.553474427381e-12 Linear solve converged due to CONVERGED_RTOL iterations 2 1 SNES Function norm 5.632325929998e-08 Residual norms for DB_ solve. 0 KSP Residual norm 1.302167182121e-09 1 KSP Residual norm 1.016337630022e-13 2 KSP Residual norm 8.676582604839e-18 Residual norms for DP_ solve. 0 KSP Residual norm 6.913957320690e-11 1 KSP Residual norm 4.966469692746e-16 Residual norms for Drho_ solve. 0 KSP Residual norm 1.132396259485e-09 1 KSP Residual norm 3.945452134707e-20 Residual norms for PS_ solve. 0 KSP Residual norm 5.629471166161e-08 1 KSP Residual norm 2.221140747995e-09 2 KSP Residual norm 9.103520624251e-11 3 KSP Residual norm 3.881846850794e-12 4 KSP Residual norm 1.617926366497e-13 0 KSP Residual norm 5.656401015719e-08 Residual norms for DB_ solve. 0 KSP Residual norm 2.302174239526e-02 1 KSP Residual norm 1.796923989546e-06 2 KSP Residual norm 1.534054142323e-10 Residual norms for DP_ solve. 0 KSP Residual norm 1.222421046956e-03 1 KSP Residual norm 8.780879980160e-09 Residual norms for Drho_ solve. 0 KSP Residual norm 2.001976946159e-02 1 KSP Residual norm 6.975153185817e-13 Residual norms for PS_ solve. 0 KSP Residual norm 9.952378186456e-01 1 KSP Residual norm 3.926701064580e-02 2 KSP Residual norm 1.609382459345e-03 3 KSP Residual norm 6.862589195784e-05 4 KSP Residual norm 2.860278802804e-06 1 KSP Residual norm 2.921998658056e-12 Residual norms for DB_ solve. 0 KSP Residual norm 6.653463668565e-01 1 KSP Residual norm 3.822008488207e-05 2 KSP Residual norm 2.781508132254e-09 Residual norms for DP_ solve. 0 KSP Residual norm 3.871253757817e-03 1 KSP Residual norm 2.691108885444e-08 Residual norms for Drho_ solve. 0 KSP Residual norm 9.846149611801e-03 1 KSP Residual norm 8.090481654281e-13 Residual norms for PS_ solve. 0 KSP Residual norm 7.827483124782e-01 1 KSP Residual norm 3.575117584186e-02 2 KSP Residual norm 1.584099327764e-03 3 KSP Residual norm 6.603898369937e-05 4 KSP Residual norm 2.829819432667e-06 2 KSP Residual norm 1.006906168481e-16 Linear solve converged due to CONVERGED_RTOL iterations 2 2 SNES Function norm 7.427800084502e-15 Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 2 Total CPU time since PetscInitialize: 5.2341E+00 CPU time used for SNESSolve: 4.7887E+00 Number of linear iterations : 4 Number of function evaluations : 9 Kinetic Energy = 8.283578E-11 Magnetic Energy = 2.278864E-12 Exiting the main MHD Loop Deallocating remaining arrays Destroying remaining Petsc elements ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./miips on a arch-linux2-c-opt named helios87 with 1 processor, by tnicolas Mon Dec 14 15:56:00 2015 Using Petsc Release Version 3.6.0, Jun, 09, 2015 Max Max/Min Avg Total Time (sec): 5.319e+00 1.00000 5.319e+00 Objects: 2.270e+02 1.00000 2.270e+02 Flops: 4.461e+08 1.00000 4.461e+08 4.461e+08 Flops/sec: 8.387e+07 1.00000 8.387e+07 8.387e+07 MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 MPI Reductions: 0.000e+00 0.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 5.3187e+00 100.0% 4.4605e+08 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage SNESSolve 1 1.0 4.7775e+00 1.0 4.25e+08 1.0 0.0e+00 0.0e+00 0.0e+00 90 95 0 0 0 90 95 0 0 0 89 SNESFunctionEval 9 1.0 6.5902e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 12 0 0 0 0 12 0 0 0 0 0 SNESJacobianEval 2 1.0 4.2868e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SNESLineSearch 2 1.0 3.1556e-01 1.0 3.57e+07 1.0 0.0e+00 0.0e+00 0.0e+00 6 8 0 0 0 6 8 0 0 0 113 VecView 2 1.0 2.0043e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecDot 2 1.0 1.6801e-03 1.0 4.46e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 2655 VecMDot 52 1.0 3.4153e-02 1.0 8.20e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 18 0 0 0 1 18 0 0 0 2400 VecNorm 85 1.0 1.8394e-02 1.0 7.58e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 17 0 0 0 0 17 0 0 0 4122 VecScale 120 1.0 2.6920e-02 1.0 5.35e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 12 0 0 0 1 12 0 0 0 1988 VecCopy 110 1.0 5.4078e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecSet 174 1.0 2.7492e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 5 0 0 0 0 5 0 0 0 0 0 VecAXPY 73 1.0 5.1098e-02 1.0 7.58e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 17 0 0 0 1 17 0 0 0 1484 VecWAXPY 11 1.0 2.1958e-02 1.0 1.90e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 4 0 0 0 0 4 0 0 0 863 VecMAXPY 78 1.0 4.1620e-02 1.0 1.24e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 28 0 0 0 1 28 0 0 0 2987 VecAssemblyBegin 149 1.0 6.8188e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 149 1.0 4.5538e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 2 1.0 3.2470e-03 1.0 2.23e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 687 VecScatterBegin 158 1.0 1.5226e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 3 0 0 0 0 3 0 0 0 0 0 VecReduceArith 4 1.0 3.0560e-03 1.0 8.92e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 2919 VecReduceComm 2 1.0 9.0599e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 78 1.0 2.8533e-02 1.0 9.03e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 20 0 0 0 1 20 0 0 0 3166 MatMult MF 6 1.0 4.6223e-01 1.0 3.35e+07 1.0 0.0e+00 0.0e+00 0.0e+00 9 8 0 0 0 9 8 0 0 0 72 MatMult 66 1.0 4.0039e+00 1.0 6.36e+07 1.0 0.0e+00 0.0e+00 0.0e+00 75 14 0 0 0 75 14 0 0 0 16 MatMultAdd 24 1.0 7.7939e-01 1.0 2.01e+07 1.0 0.0e+00 0.0e+00 0.0e+00 15 5 0 0 0 15 5 0 0 0 26 MatAssemblyBegin 2 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 2 1.0 4.2679e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 52 1.0 6.1561e-02 1.0 1.64e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 37 0 0 0 1 37 0 0 0 2663 KSPSetUp 6 1.0 4.1640e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 KSPSolve 2 1.0 4.3720e+00 1.0 3.87e+08 1.0 0.0e+00 0.0e+00 0.0e+00 82 87 0 0 0 82 87 0 0 0 89 PCSetUp 6 1.0 1.6212e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCApply 6 1.0 4.0072e+00 1.0 3.04e+08 1.0 0.0e+00 0.0e+00 0.0e+00 75 68 0 0 0 75 68 0 0 0 76 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage SNES 1 1 1332 0 SNESLineSearch 1 1 864 0 DMSNES 2 2 1328 0 Vector 137 137 568058072 0 Vector Scatter 8 8 5248 0 MatMFFD 1 1 784 0 Matrix 8 8 19440 0 Distributed Mesh 8 8 39216 0 Star Forest Bipartite Graph 16 16 13584 0 Discrete System 8 8 6784 0 Index Set 16 16 4046072 0 IS L to G Mapping 7 7 4038024 0 Krylov Solver 5 5 91800 0 DMKSP interface 1 1 648 0 Preconditioner 5 5 4352 0 Viewer 3 2 1536 0 ======================================================================================================================== Average time to get PetscTime(): 0 #PETSc Option Table entries: -DB_ksp_monitor -DB_pc_type none -DP_ksp_monitor -DP_pc_type none -Drho_ksp_monitor -Drho_pc_type none -PS_ksp_monitor -PS_pc_type none -ksp_converged_reason -ksp_monitor -log_summary -mult_lu -nts 1 -snes_converged_reason -snes_mf -snes_monitor #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --prefix=/csc/softs/anl/petsc-3.6.0/intel-15.0.0.090/bullxmpi-1.2.8.2/real --with-debugging=0 --with-x=0 --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpicxx --with-fortran --known-mpi-shared-libraries=1 --with-scalar-type=real --with-precision=double --CFLAGS="-g -O3 -mavx -mkl" --CXXFLAGS="-g -O3 -mavx -mkl" --FFLAGS="-g -O3 -mavx -mkl" ----------------------------------------- Libraries compiled on Mon Sep 28 20:22:47 2015 on helios85 Machine characteristics: Linux-2.6.32-573.1.1.el6.Bull.80.x86_64-x86_64-with-redhat-6.4-Santiago Using PETSc directory: /csc/releases/buildlog/anl/petsc-3.6.0/intel-15.0.0.090/bullxmpi-1.2.8.2/real/petsc-3.6.0 Using PETSc arch: arch-linux2-c-opt ----------------------------------------- Using C compiler: mpicc -g -O3 -mavx -mkl -fPIC ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: mpif90 -g -O3 -mavx -mkl -fPIC ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/csc/releases/buildlog/anl/petsc-3.6.0/intel-15.0.0.090/bullxmpi-1.2.8.2/real/petsc-3.6.0/arch-linux2-c-opt/include -I/csc/releases/buildlog/anl/petsc-3.6.0/intel-15.0.0.090/bullxmpi-1.2.8.2/real/petsc-3.6.0/include -I/csc/releases/buildlog/anl/petsc-3.6.0/intel-15.0.0.090/bullxmpi-1.2.8.2/real/petsc-3.6.0/include -I/csc/releases/buildlog/anl/petsc-3.6.0/intel-15.0.0.090/bullxmpi-1.2.8.2/real/petsc-3.6.0/arch-linux2-c-opt/include -I/opt/mpi/bullxmpi/1.2.8.2/include ----------------------------------------- Using C linker: mpicc Using Fortran linker: mpif90 Using libraries: -Wl,-rpath,/csc/releases/buildlog/anl/petsc-3.6.0/intel-15.0.0.090/bullxmpi-1.2.8.2/real/petsc-3.6.0/arch-linux2-c-opt/lib -L/csc/releases/buildlog/anl/petsc-3.6.0/intel-15.0.0.090/bullxmpi-1.2.8.2/real/petsc-3.6.0/arch-linux2-c-opt/lib -lpetsc -lhwloc -lxml2 -lssl -lcrypto -Wl,-rpath,/opt/mpi/bullxmpi/1.2.8.2/lib -L/opt/mpi/bullxmpi/1.2.8.2/lib -Wl,-rpath,/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64 -L/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -lmpi_f90 -lmpi_f77 -lm -lifport -lifcore -lm -lmpi_cxx -ldl -Wl,-rpath,/opt/mpi/bullxmpi/1.2.8.2/lib -L/opt/mpi/bullxmpi/1.2.8.2/lib -lmpi -lnuma -lrt -lnsl -lutil -Wl,-rpath,/opt/mpi/bullxmpi/1.2.8.2/lib -L/opt/mpi/bullxmpi/1.2.8.2/lib -Wl,-rpath,/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64 -L/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -Wl,-rpath,/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64 -L/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -Wl,-rpath,/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -limf -Wl,-rpath,/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64 -L/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64 -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -Wl,-rpath,/opt/mpi/bullxmpi/1.2.8.2/lib -L/opt/mpi/bullxmpi/1.2.8.2/lib -Wl,-rpath,/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64 -L/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -Wl,-rpath,/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64 -L/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.0.090/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64 -L/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64 -ldl ----------------------------------------- From knepley at gmail.com Mon Dec 14 01:20:43 2015 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 14 Dec 2015 01:20:43 -0600 Subject: [petsc-users] Big discrepancy between machines In-Reply-To: References: Message-ID: On Mon, Dec 14, 2015 at 1:09 AM, Timoth?e Nicolas < timothee.nicolas at gmail.com> wrote: > Hi, > > I have noticed I have a VERY big difference in behaviour between two > machines in my problem, solved with SNES. I can't explain it, because I > have tested my operators which give the same result. I also checked that > the vectors fed to the SNES are the same. The problem happens only with my > shell preconditioner. When I don't use it, and simply solve using -snes_mf, > I don't see anymore than the usual 3-4 changing digits at the end of the > residuals. However, when I use my pcshell, the results are completely > different between the two machines. > I don't think its possible from this info to tell exactly what is happening. However, if your shell preconditioner had a null space, you could imagine that you initially have a consistent approximation, but on one machine you get a perturbation in the null direction which never goes away. I have no other ideas. Matt > I have attached output_SuperComputer.txt and output_DesktopComputer.txt, > which correspond to the output from the exact same code and options (and of > course same input data file !). More precisely > > output_SuperComputer.txt : output on a supercomputer called Helios, sorry > I don't know the exact specs. > In this case, the SNES norms are reduced successively: > 0 SNES Function norm 4.867111712420e-03 > 1 SNES Function norm 5.632325929998e-08 > 2 SNES Function norm 7.427800084502e-15 > > output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 GHz Intel > Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop with Mac > OS X Mavericks). > In this case, I obtain the following for the SNES norms, > while in the other, I obtain > 0 SNES Function norm 4.867111713544e-03 > 1 SNES Function norm 1.560094052222e-03 > 2 SNES Function norm 1.552118650943e-03 > 3 SNES Function norm 1.552106297094e-03 > 4 SNES Function norm 1.552106277949e-03 > which I can't explain, because otherwise the KSP residual (with the same > operator, which I checked) behave well. > > As you can see, the first time the preconditioner is applied (DB_, DP_, > Drho_ and PS_ solves), the two outputs coincide (except for the few last > digits, up to 9 actually, which is more than I would expect), and > everything starts to diverge at the first print of the main KSP (the one > stemming from the SNES) residual norms. > > Do you have an idea what may cause such a strange behaviour ? > > Best > > Timothee > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothee.nicolas at gmail.com Mon Dec 14 01:30:44 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Mon, 14 Dec 2015 16:30:44 +0900 Subject: [petsc-users] Big discrepancy between machines In-Reply-To: References: Message-ID: I see, To check whether I have a null space or not, in principle, I should see it by computing the determinant of my matrices, right ? In principle I should not have a null space, so that would be a good method to check if I have any coding error, wouldn't it ? Thx Timothee 2015-12-14 16:20 GMT+09:00 Matthew Knepley : > On Mon, Dec 14, 2015 at 1:09 AM, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > >> Hi, >> >> I have noticed I have a VERY big difference in behaviour between two >> machines in my problem, solved with SNES. I can't explain it, because I >> have tested my operators which give the same result. I also checked that >> the vectors fed to the SNES are the same. The problem happens only with my >> shell preconditioner. When I don't use it, and simply solve using -snes_mf, >> I don't see anymore than the usual 3-4 changing digits at the end of the >> residuals. However, when I use my pcshell, the results are completely >> different between the two machines. >> > > I don't think its possible from this info to tell exactly what is > happening. However, if your shell preconditioner had a null space, you > could imagine > that you initially have a consistent approximation, but on one machine you > get a perturbation in the null direction which never goes away. I have > no other ideas. > > Matt > > >> I have attached output_SuperComputer.txt and output_DesktopComputer.txt, >> which correspond to the output from the exact same code and options (and of >> course same input data file !). More precisely >> >> output_SuperComputer.txt : output on a supercomputer called Helios, sorry >> I don't know the exact specs. >> In this case, the SNES norms are reduced successively: >> 0 SNES Function norm 4.867111712420e-03 >> 1 SNES Function norm 5.632325929998e-08 >> 2 SNES Function norm 7.427800084502e-15 >> >> output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 GHz Intel >> Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop with Mac >> OS X Mavericks). >> In this case, I obtain the following for the SNES norms, >> while in the other, I obtain >> 0 SNES Function norm 4.867111713544e-03 >> 1 SNES Function norm 1.560094052222e-03 >> 2 SNES Function norm 1.552118650943e-03 >> 3 SNES Function norm 1.552106297094e-03 >> 4 SNES Function norm 1.552106277949e-03 >> which I can't explain, because otherwise the KSP residual (with the same >> operator, which I checked) behave well. >> >> As you can see, the first time the preconditioner is applied (DB_, DP_, >> Drho_ and PS_ solves), the two outputs coincide (except for the few last >> digits, up to 9 actually, which is more than I would expect), and >> everything starts to diverge at the first print of the main KSP (the one >> stemming from the SNES) residual norms. >> >> Do you have an idea what may cause such a strange behaviour ? >> >> Best >> >> Timothee >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Mon Dec 14 02:04:52 2015 From: dave.mayhem23 at gmail.com (Dave May) Date: Mon, 14 Dec 2015 09:04:52 +0100 Subject: [petsc-users] Big discrepancy between machines In-Reply-To: References: Message-ID: One suggestion is you have some uninitialized variables in your pcshell. Despite your arch being called "debug", your configure options indicate you have turned debugging off. C standard doesn't prescribe how uninit variables should be treated - the behavior is labelled as undefined. As a result, different compilers on different archs with the same optimization flags can and will treat uninit variables differently. I find OSX c compilers tend to set them to zero. I suggest compiling a debug build on both machines and trying your test again. Also, consider running the debug builds through valgrind. Thanks, Dave On Monday, 14 December 2015, Timoth?e Nicolas wrote: > Hi, > > I have noticed I have a VERY big difference in behaviour between two > machines in my problem, solved with SNES. I can't explain it, because I > have tested my operators which give the same result. I also checked that > the vectors fed to the SNES are the same. The problem happens only with my > shell preconditioner. When I don't use it, and simply solve using -snes_mf, > I don't see anymore than the usual 3-4 changing digits at the end of the > residuals. However, when I use my pcshell, the results are completely > different between the two machines. > > I have attached output_SuperComputer.txt and output_DesktopComputer.txt, > which correspond to the output from the exact same code and options (and of > course same input data file !). More precisely > > output_SuperComputer.txt : output on a supercomputer called Helios, sorry > I don't know the exact specs. > In this case, the SNES norms are reduced successively: > 0 SNES Function norm 4.867111712420e-03 > 1 SNES Function norm 5.632325929998e-08 > 2 SNES Function norm 7.427800084502e-15 > > output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 GHz Intel > Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop with Mac > OS X Mavericks). > In this case, I obtain the following for the SNES norms, > while in the other, I obtain > 0 SNES Function norm 4.867111713544e-03 > 1 SNES Function norm 1.560094052222e-03 > 2 SNES Function norm 1.552118650943e-03 > 3 SNES Function norm 1.552106297094e-03 > 4 SNES Function norm 1.552106277949e-03 > which I can't explain, because otherwise the KSP residual (with the same > operator, which I checked) behave well. > > As you can see, the first time the preconditioner is applied (DB_, DP_, > Drho_ and PS_ solves), the two outputs coincide (except for the few last > digits, up to 9 actually, which is more than I would expect), and > everything starts to diverge at the first print of the main KSP (the one > stemming from the SNES) residual norms. > > Do you have an idea what may cause such a strange behaviour ? > > Best > > Timothee > -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothee.nicolas at gmail.com Mon Dec 14 02:07:53 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Mon, 14 Dec 2015 17:07:53 +0900 Subject: [petsc-users] Big discrepancy between machines In-Reply-To: References: Message-ID: Hum, OK. I use FORTRAN by the way. Is your comment still valid ? I'll check anyway, but I thought I had been careful about this sort of things. Also, I thought the problem on Mac OS X may have been due to the fact I used the version with debugging on, so I rerun configure with --with-debugging=no, which did not change anything. Thx Timothee 2015-12-14 17:04 GMT+09:00 Dave May : > One suggestion is you have some uninitialized variables in your pcshell. > Despite your arch being called "debug", your configure options indicate you > have turned debugging off. > > C standard doesn't prescribe how uninit variables should be treated - the > behavior is labelled as undefined. As a result, different compilers on > different archs with the same optimization flags can and will treat uninit > variables differently. I find OSX c compilers tend to set them to zero. > > I suggest compiling a debug build on both machines and trying your > test again. Also, consider running the debug builds through valgrind. > > Thanks, > Dave > > On Monday, 14 December 2015, Timoth?e Nicolas > wrote: > >> Hi, >> >> I have noticed I have a VERY big difference in behaviour between two >> machines in my problem, solved with SNES. I can't explain it, because I >> have tested my operators which give the same result. I also checked that >> the vectors fed to the SNES are the same. The problem happens only with my >> shell preconditioner. When I don't use it, and simply solve using -snes_mf, >> I don't see anymore than the usual 3-4 changing digits at the end of the >> residuals. However, when I use my pcshell, the results are completely >> different between the two machines. >> >> I have attached output_SuperComputer.txt and output_DesktopComputer.txt, >> which correspond to the output from the exact same code and options (and of >> course same input data file !). More precisely >> >> output_SuperComputer.txt : output on a supercomputer called Helios, sorry >> I don't know the exact specs. >> In this case, the SNES norms are reduced successively: >> 0 SNES Function norm 4.867111712420e-03 >> 1 SNES Function norm 5.632325929998e-08 >> 2 SNES Function norm 7.427800084502e-15 >> >> output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 GHz Intel >> Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop with Mac >> OS X Mavericks). >> In this case, I obtain the following for the SNES norms, >> while in the other, I obtain >> 0 SNES Function norm 4.867111713544e-03 >> 1 SNES Function norm 1.560094052222e-03 >> 2 SNES Function norm 1.552118650943e-03 >> 3 SNES Function norm 1.552106297094e-03 >> 4 SNES Function norm 1.552106277949e-03 >> which I can't explain, because otherwise the KSP residual (with the same >> operator, which I checked) behave well. >> >> As you can see, the first time the preconditioner is applied (DB_, DP_, >> Drho_ and PS_ solves), the two outputs coincide (except for the few last >> digits, up to 9 actually, which is more than I would expect), and >> everything starts to diverge at the first print of the main KSP (the one >> stemming from the SNES) residual norms. >> >> Do you have an idea what may cause such a strange behaviour ? >> >> Best >> >> Timothee >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Mon Dec 14 02:26:01 2015 From: dave.mayhem23 at gmail.com (Dave May) Date: Mon, 14 Dec 2015 09:26:01 +0100 Subject: [petsc-users] Big discrepancy between machines In-Reply-To: References: Message-ID: On Monday, 14 December 2015, Timoth?e Nicolas > wrote: > Hum, OK. I use FORTRAN by the way. Is your comment still valid ? > No. Fortran compilers init variables to zero. In this case, I would run a debug build on your OSX machine through valgrind and make sure it is clean. Other obvious thing to check what happens if use exactly the same petsc builds on both machines. I see 3.6.1 and 3.6.0 are being used. For all this type of checking, I would definitely use debug builds on both machines. Your cluster build is using the highest level of optimization... > I'll check anyway, but I thought I had been careful about this sort of > things. > > Also, I thought the problem on Mac OS X may have been due to the fact I > used the version with debugging on, so I rerun configure with > --with-debugging=no, which did not change anything. > > Thx > > Timothee > > > 2015-12-14 17:04 GMT+09:00 Dave May : > >> One suggestion is you have some uninitialized variables in your pcshell. >> Despite your arch being called "debug", your configure options indicate you >> have turned debugging off. >> >> C standard doesn't prescribe how uninit variables should be treated - the >> behavior is labelled as undefined. As a result, different compilers on >> different archs with the same optimization flags can and will treat uninit >> variables differently. I find OSX c compilers tend to set them to zero. >> >> I suggest compiling a debug build on both machines and trying your >> test again. Also, consider running the debug builds through valgrind. >> >> Thanks, >> Dave >> >> On Monday, 14 December 2015, Timoth?e Nicolas >> wrote: >> >>> Hi, >>> >>> I have noticed I have a VERY big difference in behaviour between two >>> machines in my problem, solved with SNES. I can't explain it, because I >>> have tested my operators which give the same result. I also checked that >>> the vectors fed to the SNES are the same. The problem happens only with my >>> shell preconditioner. When I don't use it, and simply solve using -snes_mf, >>> I don't see anymore than the usual 3-4 changing digits at the end of the >>> residuals. However, when I use my pcshell, the results are completely >>> different between the two machines. >>> >>> I have attached output_SuperComputer.txt and output_DesktopComputer.txt, >>> which correspond to the output from the exact same code and options (and of >>> course same input data file !). More precisely >>> >>> output_SuperComputer.txt : output on a supercomputer called Helios, >>> sorry I don't know the exact specs. >>> In this case, the SNES norms are reduced successively: >>> 0 SNES Function norm 4.867111712420e-03 >>> 1 SNES Function norm 5.632325929998e-08 >>> 2 SNES Function norm 7.427800084502e-15 >>> >>> output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 GHz Intel >>> Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop with Mac >>> OS X Mavericks). >>> In this case, I obtain the following for the SNES norms, >>> while in the other, I obtain >>> 0 SNES Function norm 4.867111713544e-03 >>> 1 SNES Function norm 1.560094052222e-03 >>> 2 SNES Function norm 1.552118650943e-03 >>> 3 SNES Function norm 1.552106297094e-03 >>> 4 SNES Function norm 1.552106277949e-03 >>> which I can't explain, because otherwise the KSP residual (with the same >>> operator, which I checked) behave well. >>> >>> As you can see, the first time the preconditioner is applied (DB_, DP_, >>> Drho_ and PS_ solves), the two outputs coincide (except for the few last >>> digits, up to 9 actually, which is more than I would expect), and >>> everything starts to diverge at the first print of the main KSP (the one >>> stemming from the SNES) residual norms. >>> >>> Do you have an idea what may cause such a strange behaviour ? >>> >>> Best >>> >>> Timothee >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothee.nicolas at gmail.com Mon Dec 14 02:34:50 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Mon, 14 Dec 2015 17:34:50 +0900 Subject: [petsc-users] Big discrepancy between machines In-Reply-To: References: Message-ID: OK, The problem is that I don't think I can change this easily as far as the cluster is concerned. I obtain access to petsc by loading the petsc module, and even if I have a few choices, I don't see any debug builds... 2015-12-14 17:26 GMT+09:00 Dave May : > > > On Monday, 14 December 2015, Timoth?e Nicolas > wrote: > >> Hum, OK. I use FORTRAN by the way. Is your comment still valid ? >> > > No. Fortran compilers init variables to zero. > In this case, I would run a debug build on your OSX machine through > valgrind and make sure it is clean. > > Other obvious thing to check what happens if use exactly the same petsc > builds on both machines. I see 3.6.1 and 3.6.0 are being used. > > For all this type of checking, I would definitely use debug builds on both > machines. Your cluster build is using the highest level of optimization... > > > > >> I'll check anyway, but I thought I had been careful about this sort of >> things. >> >> Also, I thought the problem on Mac OS X may have been due to the fact I >> used the version with debugging on, so I rerun configure with >> --with-debugging=no, which did not change anything. >> >> Thx >> >> Timothee >> >> >> 2015-12-14 17:04 GMT+09:00 Dave May : >> >>> One suggestion is you have some uninitialized variables in your pcshell. >>> Despite your arch being called "debug", your configure options indicate you >>> have turned debugging off. >>> >>> C standard doesn't prescribe how uninit variables should be treated - >>> the behavior is labelled as undefined. As a result, different compilers on >>> different archs with the same optimization flags can and will treat uninit >>> variables differently. I find OSX c compilers tend to set them to zero. >>> >>> I suggest compiling a debug build on both machines and trying your >>> test again. Also, consider running the debug builds through valgrind. >>> >>> Thanks, >>> Dave >>> >>> On Monday, 14 December 2015, Timoth?e Nicolas < >>> timothee.nicolas at gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I have noticed I have a VERY big difference in behaviour between two >>>> machines in my problem, solved with SNES. I can't explain it, because I >>>> have tested my operators which give the same result. I also checked that >>>> the vectors fed to the SNES are the same. The problem happens only with my >>>> shell preconditioner. When I don't use it, and simply solve using -snes_mf, >>>> I don't see anymore than the usual 3-4 changing digits at the end of the >>>> residuals. However, when I use my pcshell, the results are completely >>>> different between the two machines. >>>> >>>> I have attached output_SuperComputer.txt and >>>> output_DesktopComputer.txt, which correspond to the output from the exact >>>> same code and options (and of course same input data file !). More precisely >>>> >>>> output_SuperComputer.txt : output on a supercomputer called Helios, >>>> sorry I don't know the exact specs. >>>> In this case, the SNES norms are reduced successively: >>>> 0 SNES Function norm 4.867111712420e-03 >>>> 1 SNES Function norm 5.632325929998e-08 >>>> 2 SNES Function norm 7.427800084502e-15 >>>> >>>> output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 GHz >>>> Intel Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop with >>>> Mac OS X Mavericks). >>>> In this case, I obtain the following for the SNES norms, >>>> while in the other, I obtain >>>> 0 SNES Function norm 4.867111713544e-03 >>>> 1 SNES Function norm 1.560094052222e-03 >>>> 2 SNES Function norm 1.552118650943e-03 >>>> 3 SNES Function norm 1.552106297094e-03 >>>> 4 SNES Function norm 1.552106277949e-03 >>>> which I can't explain, because otherwise the KSP residual (with the >>>> same operator, which I checked) behave well. >>>> >>>> As you can see, the first time the preconditioner is applied (DB_, DP_, >>>> Drho_ and PS_ solves), the two outputs coincide (except for the few last >>>> digits, up to 9 actually, which is more than I would expect), and >>>> everything starts to diverge at the first print of the main KSP (the one >>>> stemming from the SNES) residual norms. >>>> >>>> Do you have an idea what may cause such a strange behaviour ? >>>> >>>> Best >>>> >>>> Timothee >>>> >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Mon Dec 14 02:38:18 2015 From: dave.mayhem23 at gmail.com (Dave May) Date: Mon, 14 Dec 2015 09:38:18 +0100 Subject: [petsc-users] Big discrepancy between machines In-Reply-To: References: Message-ID: You have the configure line, so it should be relatively straight forward to configure / build petsc in your home directory. On 14 December 2015 at 09:34, Timoth?e Nicolas wrote: > OK, The problem is that I don't think I can change this easily as far as > the cluster is concerned. I obtain access to petsc by loading the petsc > module, and even if I have a few choices, I don't see any debug builds... > > 2015-12-14 17:26 GMT+09:00 Dave May : > >> >> >> On Monday, 14 December 2015, Timoth?e Nicolas >> wrote: >> >>> Hum, OK. I use FORTRAN by the way. Is your comment still valid ? >>> >> >> No. Fortran compilers init variables to zero. >> In this case, I would run a debug build on your OSX machine through >> valgrind and make sure it is clean. >> >> Other obvious thing to check what happens if use exactly the same petsc >> builds on both machines. I see 3.6.1 and 3.6.0 are being used. >> >> For all this type of checking, I would definitely use debug builds on >> both machines. Your cluster build is using the highest level of >> optimization... >> >> >> >> >>> I'll check anyway, but I thought I had been careful about this sort of >>> things. >>> >>> Also, I thought the problem on Mac OS X may have been due to the fact I >>> used the version with debugging on, so I rerun configure with >>> --with-debugging=no, which did not change anything. >>> >>> Thx >>> >>> Timothee >>> >>> >>> 2015-12-14 17:04 GMT+09:00 Dave May : >>> >>>> One suggestion is you have some uninitialized variables in your >>>> pcshell. Despite your arch being called "debug", your configure options >>>> indicate you have turned debugging off. >>>> >>>> C standard doesn't prescribe how uninit variables should be treated - >>>> the behavior is labelled as undefined. As a result, different compilers on >>>> different archs with the same optimization flags can and will treat uninit >>>> variables differently. I find OSX c compilers tend to set them to zero. >>>> >>>> I suggest compiling a debug build on both machines and trying your >>>> test again. Also, consider running the debug builds through valgrind. >>>> >>>> Thanks, >>>> Dave >>>> >>>> On Monday, 14 December 2015, Timoth?e Nicolas < >>>> timothee.nicolas at gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I have noticed I have a VERY big difference in behaviour between two >>>>> machines in my problem, solved with SNES. I can't explain it, because I >>>>> have tested my operators which give the same result. I also checked that >>>>> the vectors fed to the SNES are the same. The problem happens only with my >>>>> shell preconditioner. When I don't use it, and simply solve using -snes_mf, >>>>> I don't see anymore than the usual 3-4 changing digits at the end of the >>>>> residuals. However, when I use my pcshell, the results are completely >>>>> different between the two machines. >>>>> >>>>> I have attached output_SuperComputer.txt and >>>>> output_DesktopComputer.txt, which correspond to the output from the exact >>>>> same code and options (and of course same input data file !). More precisely >>>>> >>>>> output_SuperComputer.txt : output on a supercomputer called Helios, >>>>> sorry I don't know the exact specs. >>>>> In this case, the SNES norms are reduced successively: >>>>> 0 SNES Function norm 4.867111712420e-03 >>>>> 1 SNES Function norm 5.632325929998e-08 >>>>> 2 SNES Function norm 7.427800084502e-15 >>>>> >>>>> output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 GHz >>>>> Intel Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop with >>>>> Mac OS X Mavericks). >>>>> In this case, I obtain the following for the SNES norms, >>>>> while in the other, I obtain >>>>> 0 SNES Function norm 4.867111713544e-03 >>>>> 1 SNES Function norm 1.560094052222e-03 >>>>> 2 SNES Function norm 1.552118650943e-03 >>>>> 3 SNES Function norm 1.552106297094e-03 >>>>> 4 SNES Function norm 1.552106277949e-03 >>>>> which I can't explain, because otherwise the KSP residual (with the >>>>> same operator, which I checked) behave well. >>>>> >>>>> As you can see, the first time the preconditioner is applied (DB_, >>>>> DP_, Drho_ and PS_ solves), the two outputs coincide (except for the few >>>>> last digits, up to 9 actually, which is more than I would expect), and >>>>> everything starts to diverge at the first print of the main KSP (the one >>>>> stemming from the SNES) residual norms. >>>>> >>>>> Do you have an idea what may cause such a strange behaviour ? >>>>> >>>>> Best >>>>> >>>>> Timothee >>>>> >>>> >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothee.nicolas at gmail.com Mon Dec 14 02:38:43 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Mon, 14 Dec 2015 17:38:43 +0900 Subject: [petsc-users] Big discrepancy between machines In-Reply-To: References: Message-ID: OK, I'll try that, thx 2015-12-14 17:38 GMT+09:00 Dave May : > You have the configure line, so it should be relatively straight forward > to configure / build petsc in your home directory. > > > On 14 December 2015 at 09:34, Timoth?e Nicolas > wrote: > >> OK, The problem is that I don't think I can change this easily as far as >> the cluster is concerned. I obtain access to petsc by loading the petsc >> module, and even if I have a few choices, I don't see any debug builds... >> >> 2015-12-14 17:26 GMT+09:00 Dave May : >> >>> >>> >>> On Monday, 14 December 2015, Timoth?e Nicolas < >>> timothee.nicolas at gmail.com> wrote: >>> >>>> Hum, OK. I use FORTRAN by the way. Is your comment still valid ? >>>> >>> >>> No. Fortran compilers init variables to zero. >>> In this case, I would run a debug build on your OSX machine through >>> valgrind and make sure it is clean. >>> >>> Other obvious thing to check what happens if use exactly the same petsc >>> builds on both machines. I see 3.6.1 and 3.6.0 are being used. >>> >>> For all this type of checking, I would definitely use debug builds on >>> both machines. Your cluster build is using the highest level of >>> optimization... >>> >>> >>> >>> >>>> I'll check anyway, but I thought I had been careful about this sort of >>>> things. >>>> >>>> Also, I thought the problem on Mac OS X may have been due to the fact I >>>> used the version with debugging on, so I rerun configure with >>>> --with-debugging=no, which did not change anything. >>>> >>>> Thx >>>> >>>> Timothee >>>> >>>> >>>> 2015-12-14 17:04 GMT+09:00 Dave May : >>>> >>>>> One suggestion is you have some uninitialized variables in your >>>>> pcshell. Despite your arch being called "debug", your configure options >>>>> indicate you have turned debugging off. >>>>> >>>>> C standard doesn't prescribe how uninit variables should be treated - >>>>> the behavior is labelled as undefined. As a result, different compilers on >>>>> different archs with the same optimization flags can and will treat uninit >>>>> variables differently. I find OSX c compilers tend to set them to zero. >>>>> >>>>> I suggest compiling a debug build on both machines and trying your >>>>> test again. Also, consider running the debug builds through valgrind. >>>>> >>>>> Thanks, >>>>> Dave >>>>> >>>>> On Monday, 14 December 2015, Timoth?e Nicolas < >>>>> timothee.nicolas at gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I have noticed I have a VERY big difference in behaviour between two >>>>>> machines in my problem, solved with SNES. I can't explain it, because I >>>>>> have tested my operators which give the same result. I also checked that >>>>>> the vectors fed to the SNES are the same. The problem happens only with my >>>>>> shell preconditioner. When I don't use it, and simply solve using -snes_mf, >>>>>> I don't see anymore than the usual 3-4 changing digits at the end of the >>>>>> residuals. However, when I use my pcshell, the results are completely >>>>>> different between the two machines. >>>>>> >>>>>> I have attached output_SuperComputer.txt and >>>>>> output_DesktopComputer.txt, which correspond to the output from the exact >>>>>> same code and options (and of course same input data file !). More precisely >>>>>> >>>>>> output_SuperComputer.txt : output on a supercomputer called Helios, >>>>>> sorry I don't know the exact specs. >>>>>> In this case, the SNES norms are reduced successively: >>>>>> 0 SNES Function norm 4.867111712420e-03 >>>>>> 1 SNES Function norm 5.632325929998e-08 >>>>>> 2 SNES Function norm 7.427800084502e-15 >>>>>> >>>>>> output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 GHz >>>>>> Intel Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop with >>>>>> Mac OS X Mavericks). >>>>>> In this case, I obtain the following for the SNES norms, >>>>>> while in the other, I obtain >>>>>> 0 SNES Function norm 4.867111713544e-03 >>>>>> 1 SNES Function norm 1.560094052222e-03 >>>>>> 2 SNES Function norm 1.552118650943e-03 >>>>>> 3 SNES Function norm 1.552106297094e-03 >>>>>> 4 SNES Function norm 1.552106277949e-03 >>>>>> which I can't explain, because otherwise the KSP residual (with the >>>>>> same operator, which I checked) behave well. >>>>>> >>>>>> As you can see, the first time the preconditioner is applied (DB_, >>>>>> DP_, Drho_ and PS_ solves), the two outputs coincide (except for the few >>>>>> last digits, up to 9 actually, which is more than I would expect), and >>>>>> everything starts to diverge at the first print of the main KSP (the one >>>>>> stemming from the SNES) residual norms. >>>>>> >>>>>> Do you have an idea what may cause such a strange behaviour ? >>>>>> >>>>>> Best >>>>>> >>>>>> Timothee >>>>>> >>>>> >>>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Eric.Chamberland at giref.ulaval.ca Mon Dec 14 09:56:04 2015 From: Eric.Chamberland at giref.ulaval.ca (Eric Chamberland) Date: Mon, 14 Dec 2015 10:56:04 -0500 Subject: [petsc-users] Release canditate? In-Reply-To: References: <566BA018.3060506@giref.ulaval.ca> <6574689A-666B-4415-8735-478F400CBB60@mcs.anl.gov> <566BA41E.3000802@giref.ulaval.ca> <566C6511.9090006@giref.ulaval.ca> Message-ID: <566EE694.9030801@giref.ulaval.ca> Le 2015-12-13 22:12, Satish Balay a ?crit : > On Sat, 12 Dec 2015, Eric Chamberland wrote: > >> Le 2015-12-11 23:43, Satish Balay a ?crit : >>> We do autmoatically generate tarballs everynight - its avaliable at: >>> http://ftp.mcs.anl.gov/pub/petsc/petsc-master.tar.gz [but we don't >>> maintain snapshots - i.e old tarballs for this] >> interesting! I didn't knew that... Then maybe creating a tarball for the >> "maint" branch and naming it with the date (or the sha), and then making it >> available on the "Download" page would be a more easy thing to do? > > Not sure I understand. Why snapshots of 'maint' branch? > > But if its refering 'master' [which petsc-master.tar.gz is from] - > archive all the snapshots - but its not clear if its worth it.. > [maintain snapshots for 1 month? 1year? forever?] - esp when git is > more convinent to bisect and to go back in history to find the buggy > commit.. > >> >> How "far" or different is the master branch from the maint branch right now? >> I see 3f7bb31 >> >> in maint but it is not clear (in bitbucket) if it is in master...? > > The commit of interest is 7b37fee - and its already in master - but > then - there were additional chagnes to mkl_pardiso in master [so this > part of code is reorganized] > > But wrt your question - all commits in maint will be merged into > master [immediately - or after sometime]. Ideally - fixes to 'maint' > get tested first in 'next', then in 'master' - and then merged to > 'maint'. > >>> However adding in 'release' strings is a manual process - so we do >>> this at the release time. [so we would have to do this stuff for rc] >>> >>> Here is a counter argument against this proposal. For folks interested >>> in RC - could consider [for eg] 3.6.0 as RC - and test it out - and >>> have all the issues discovered by 3.6.1. But this usually doesn't >>> hapeen [and we have 3.6.1, 3.6.2, 3.6.3, 3.6.4 etc..] >>> >>> Also testing against master would catch issues early on - and not wait >>> until the rc/release.. >> Yep, but isn't Petsc master far away from maint? > > Just to be clear: > > maint - maintainance release branch (primarily for patches to current > release) i.e for 3.6.1, 3.6.2, 3.6.3 etc.. > > master - current stable development for future release 3.7 [i.e if > interested in RC - you would be looking at master branch] > > Assuming you are thinking off 3.7.rc1, 3.7.rc2, 3.7.0, 3.7.1 etc as > the versions [not 3.6.4.rc] I see. In fact, I was thinking of 3.6.4.rc1, 3.6.4.rc2, then 3.6.4, followed by 3.6.5.rc1, 3.6.5.rc2, then 3.6.5, etc... But I am also interested into testing the "3.7.rc_nightly" which is somewhat the petsc-master.tar.gz available right now... But it is noted as follow: "This usage is for experts only." (http://www.mcs.anl.gov/petsc/download/), and I am not an expert... so I thought 3.6.4.rc1 would be something not reserved for experts but only friendly end-users like me... ;) Anyway, I'll just try the petsc-master.tar.gz right now just to see... but would be somewhat interested into 3.6.4.rc1... Maybe some chosen revisions of petsc-master.tar.gz might be pointed as "feature closed" and to be released so it is time to test them...? I think I might script something here to wget the tarball, compile it and run tests automatically with the petsc-master.tar.gz... so as soon as some problem appears with our usage or the forthcoming petsc, we will be able to give feedback... maybe I could wget it only if your nightly tests are ok? Hmmm, maybe your nightly tests could update a symlink or something "friendly" to wget only if all tests are ok? For example, if all tests are sufficiently ok, maybe you could update something like a "latest-good-petsc-master.tar.gz" which I could wget unconditionally? Just an idea... Thanks! Eric From jed at jedbrown.org Mon Dec 14 10:09:10 2015 From: jed at jedbrown.org (Jed Brown) Date: Mon, 14 Dec 2015 08:09:10 -0800 Subject: [petsc-users] Release canditate? In-Reply-To: <566EE694.9030801@giref.ulaval.ca> References: <566BA018.3060506@giref.ulaval.ca> <6574689A-666B-4415-8735-478F400CBB60@mcs.anl.gov> <566BA41E.3000802@giref.ulaval.ca> <566C6511.9090006@giref.ulaval.ca> <566EE694.9030801@giref.ulaval.ca> Message-ID: <87io41ypuh.fsf@jedbrown.org> Eric Chamberland writes: > I see. In fact, I was thinking of 3.6.4.rc1, 3.6.4.rc2, then 3.6.4, > followed by 3.6.5.rc1, 3.6.5.rc2, then 3.6.5, etc... Those patch releases are only bug fixes, not new features. I'm not aware of PETSc having a problem of introducing bugs in these patch releases and I think vanishingly few users have the patience or interest in testing them. I recommend using 'maint' (via Git or tarball) if you want the latest bug fixes. > I think I might script something here to wget the tarball, compile it > and run tests automatically with the petsc-master.tar.gz... so as soon > as some problem appears with our usage or the forthcoming petsc, we will > be able to give feedback... maybe I could wget it only if your nightly > tests are ok? Hmmm, maybe your nightly tests could update a symlink or > something "friendly" to wget only if all tests are ok? For example, if > all tests are sufficiently ok, maybe you could update something like a > "latest-good-petsc-master.tar.gz" which I could wget unconditionally? > Just an idea... It is a technical challenge to have a proof positive "everything is okay". -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From Eric.Chamberland at giref.ulaval.ca Mon Dec 14 10:24:36 2015 From: Eric.Chamberland at giref.ulaval.ca (Eric Chamberland) Date: Mon, 14 Dec 2015 11:24:36 -0500 Subject: [petsc-users] Release canditate? In-Reply-To: <87io41ypuh.fsf@jedbrown.org> References: <566BA018.3060506@giref.ulaval.ca> <6574689A-666B-4415-8735-478F400CBB60@mcs.anl.gov> <566BA41E.3000802@giref.ulaval.ca> <566C6511.9090006@giref.ulaval.ca> <566EE694.9030801@giref.ulaval.ca> <87io41ypuh.fsf@jedbrown.org> Message-ID: <566EED44.9070907@giref.ulaval.ca> Le 2015-12-14 11:09, Jed Brown a ?crit : > Eric Chamberland writes: >> I see. In fact, I was thinking of 3.6.4.rc1, 3.6.4.rc2, then 3.6.4, >> followed by 3.6.5.rc1, 3.6.5.rc2, then 3.6.5, etc... > > Those patch releases are only bug fixes, not new features. I'm not > aware of PETSc having a problem of introducing bugs in these patch > releases and I think vanishingly few users have the patience or interest > in testing them. I recommend using 'maint' (via Git or tarball) if you > want the latest bug fixes. > >> I think I might script something here to wget the tarball, compile it >> and run tests automatically with the petsc-master.tar.gz... so as soon >> as some problem appears with our usage or the forthcoming petsc, we will >> be able to give feedback... maybe I could wget it only if your nightly >> tests are ok? Hmmm, maybe your nightly tests could update a symlink or >> something "friendly" to wget only if all tests are ok? For example, if >> all tests are sufficiently ok, maybe you could update something like a >> "latest-good-petsc-master.tar.gz" which I could wget unconditionally? >> Just an idea... > > It is a technical challenge to have a proof positive "everything is > okay". > I know, but you have all the ingredients: tests and nightly builds with a summary. You would be able to state a "everything is ok" on that basis. Then, you will always want to have a better code coverage so the "everything is ok" will be stronger... Btw, I saw your code coverage is very good: about 83% ! Fwiw, we do this here with home made scripts and we automatically deliver new versions when our test base is "all green" over 13 different builds (including a valgrind one, btw). Other funny thing we do: when the code is 100% green, we tag it. We keep the history of "100% green" tags for latter use (finding a regression). Also, some users like to be "on the edge" with all latest features, but only if they are working: so they "pull" the new commits and merge/rebase only with "100% green" tags... for which we created a git aliase which would be equivalent to "git pulltogreen". Thanks, Eric From balay at mcs.anl.gov Mon Dec 14 12:00:22 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 14 Dec 2015 12:00:22 -0600 Subject: [petsc-users] Release canditate? In-Reply-To: <566EED44.9070907@giref.ulaval.ca> References: <566BA018.3060506@giref.ulaval.ca> <6574689A-666B-4415-8735-478F400CBB60@mcs.anl.gov> <566BA41E.3000802@giref.ulaval.ca> <566C6511.9090006@giref.ulaval.ca> <566EE694.9030801@giref.ulaval.ca> <87io41ypuh.fsf@jedbrown.org> <566EED44.9070907@giref.ulaval.ca> Message-ID: On Mon, 14 Dec 2015, Eric Chamberland wrote: > Le 2015-12-14 11:09, Jed Brown a ?crit : > > Eric Chamberland writes: > > > I see. In fact, I was thinking of 3.6.4.rc1, 3.6.4.rc2, then 3.6.4, > > > followed by 3.6.5.rc1, 3.6.5.rc2, then 3.6.5, etc... > > > > Those patch releases are only bug fixes, not new features. I'm not > > aware of PETSc having a problem of introducing bugs in these patch > > releases and I think vanishingly few users have the patience or interest > > in testing them. I recommend using 'maint' (via Git or tarball) if you > > want the latest bug fixes. > > > > > I think I might script something here to wget the tarball, compile it > > > and run tests automatically with the petsc-master.tar.gz... so as soon > > > as some problem appears with our usage or the forthcoming petsc, we will > > > be able to give feedback... maybe I could wget it only if your nightly > > > tests are ok? Hmmm, maybe your nightly tests could update a symlink or > > > something "friendly" to wget only if all tests are ok? For example, if > > > all tests are sufficiently ok, maybe you could update something like a > > > "latest-good-petsc-master.tar.gz" which I could wget unconditionally? > > > Just an idea... > > > > It is a technical challenge to have a proof positive "everything is > > okay". > > > > I know, but you have all the ingredients: tests and nightly builds with a > summary. > > You would be able to state a "everything is ok" on that basis. Yeah Barry has been pushing to get to that stage - [its improved considerable over the past few months] but not everything is automated yet. Its still a manual process to go through the remaining logs and decide 'ok'/'not-ok' Satish > Then, you will > always want to have a better code coverage so the "everything is ok" will be > stronger... Btw, I saw your code coverage is very good: about 83% ! > > Fwiw, we do this here with home made scripts and we automatically deliver new > versions when our test base is "all green" over 13 different builds (including > a valgrind one, btw). > > Other funny thing we do: when the code is 100% green, we tag it. We keep the > history of "100% green" tags for latter use (finding a regression). Also, > some users like to be "on the edge" with all latest features, but only if they > are working: so they "pull" the new commits and merge/rebase only with "100% > green" tags... for which we created a git aliase which would be equivalent to > "git pulltogreen". > > Thanks, > > Eric > > > From timothee.nicolas at gmail.com Mon Dec 14 23:06:50 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Tue, 15 Dec 2015 14:06:50 +0900 Subject: [petsc-users] Big discrepancy between machines In-Reply-To: References: Message-ID: There is a diference in valgrind indeed between the two. It seems to be clean on my desktop Mac OS X but not on the cluster. I'll try to see what's causing this. I still don't understand well what's causing memory leaks in the case where all PETSc objects are freed correctly (as can pbe checked with -log_summary). Also, I have tried running either valgrind ./my_code -option1 -option2... or valgrind mpiexec -n 1 ./my_code -option1 -option2... It seems the second is the correct way to proceed right ? This gives very different behaviour for valgrind. Timothee 2015-12-14 17:38 GMT+09:00 Timoth?e Nicolas : > OK, I'll try that, thx > > 2015-12-14 17:38 GMT+09:00 Dave May : > >> You have the configure line, so it should be relatively straight forward >> to configure / build petsc in your home directory. >> >> >> On 14 December 2015 at 09:34, Timoth?e Nicolas < >> timothee.nicolas at gmail.com> wrote: >> >>> OK, The problem is that I don't think I can change this easily as far as >>> the cluster is concerned. I obtain access to petsc by loading the petsc >>> module, and even if I have a few choices, I don't see any debug builds... >>> >>> 2015-12-14 17:26 GMT+09:00 Dave May : >>> >>>> >>>> >>>> On Monday, 14 December 2015, Timoth?e Nicolas < >>>> timothee.nicolas at gmail.com> wrote: >>>> >>>>> Hum, OK. I use FORTRAN by the way. Is your comment still valid ? >>>>> >>>> >>>> No. Fortran compilers init variables to zero. >>>> In this case, I would run a debug build on your OSX machine through >>>> valgrind and make sure it is clean. >>>> >>>> Other obvious thing to check what happens if use exactly the same >>>> petsc builds on both machines. I see 3.6.1 and 3.6.0 are being used. >>>> >>>> For all this type of checking, I would definitely use debug builds on >>>> both machines. Your cluster build is using the highest level of >>>> optimization... >>>> >>>> >>>> >>>> >>>>> I'll check anyway, but I thought I had been careful about this sort of >>>>> things. >>>>> >>>>> Also, I thought the problem on Mac OS X may have been due to the fact >>>>> I used the version with debugging on, so I rerun configure with >>>>> --with-debugging=no, which did not change anything. >>>>> >>>>> Thx >>>>> >>>>> Timothee >>>>> >>>>> >>>>> 2015-12-14 17:04 GMT+09:00 Dave May : >>>>> >>>>>> One suggestion is you have some uninitialized variables in your >>>>>> pcshell. Despite your arch being called "debug", your configure options >>>>>> indicate you have turned debugging off. >>>>>> >>>>>> C standard doesn't prescribe how uninit variables should be treated - >>>>>> the behavior is labelled as undefined. As a result, different compilers on >>>>>> different archs with the same optimization flags can and will treat uninit >>>>>> variables differently. I find OSX c compilers tend to set them to zero. >>>>>> >>>>>> I suggest compiling a debug build on both machines and trying your >>>>>> test again. Also, consider running the debug builds through valgrind. >>>>>> >>>>>> Thanks, >>>>>> Dave >>>>>> >>>>>> On Monday, 14 December 2015, Timoth?e Nicolas < >>>>>> timothee.nicolas at gmail.com> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I have noticed I have a VERY big difference in behaviour between two >>>>>>> machines in my problem, solved with SNES. I can't explain it, because I >>>>>>> have tested my operators which give the same result. I also checked that >>>>>>> the vectors fed to the SNES are the same. The problem happens only with my >>>>>>> shell preconditioner. When I don't use it, and simply solve using -snes_mf, >>>>>>> I don't see anymore than the usual 3-4 changing digits at the end of the >>>>>>> residuals. However, when I use my pcshell, the results are completely >>>>>>> different between the two machines. >>>>>>> >>>>>>> I have attached output_SuperComputer.txt and >>>>>>> output_DesktopComputer.txt, which correspond to the output from the exact >>>>>>> same code and options (and of course same input data file !). More precisely >>>>>>> >>>>>>> output_SuperComputer.txt : output on a supercomputer called Helios, >>>>>>> sorry I don't know the exact specs. >>>>>>> In this case, the SNES norms are reduced successively: >>>>>>> 0 SNES Function norm 4.867111712420e-03 >>>>>>> 1 SNES Function norm 5.632325929998e-08 >>>>>>> 2 SNES Function norm 7.427800084502e-15 >>>>>>> >>>>>>> output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 GHz >>>>>>> Intel Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop with >>>>>>> Mac OS X Mavericks). >>>>>>> In this case, I obtain the following for the SNES norms, >>>>>>> while in the other, I obtain >>>>>>> 0 SNES Function norm 4.867111713544e-03 >>>>>>> 1 SNES Function norm 1.560094052222e-03 >>>>>>> 2 SNES Function norm 1.552118650943e-03 >>>>>>> 3 SNES Function norm 1.552106297094e-03 >>>>>>> 4 SNES Function norm 1.552106277949e-03 >>>>>>> which I can't explain, because otherwise the KSP residual (with the >>>>>>> same operator, which I checked) behave well. >>>>>>> >>>>>>> As you can see, the first time the preconditioner is applied (DB_, >>>>>>> DP_, Drho_ and PS_ solves), the two outputs coincide (except for the few >>>>>>> last digits, up to 9 actually, which is more than I would expect), and >>>>>>> everything starts to diverge at the first print of the main KSP (the one >>>>>>> stemming from the SNES) residual norms. >>>>>>> >>>>>>> Do you have an idea what may cause such a strange behaviour ? >>>>>>> >>>>>>> Best >>>>>>> >>>>>>> Timothee >>>>>>> >>>>>> >>>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Dec 14 23:20:08 2015 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 14 Dec 2015 23:20:08 -0600 Subject: [petsc-users] Big discrepancy between machines In-Reply-To: References: Message-ID: On Mon, Dec 14, 2015 at 11:06 PM, Timoth?e Nicolas < timothee.nicolas at gmail.com> wrote: > There is a diference in valgrind indeed between the two. It seems to be > clean on my desktop Mac OS X but not on the cluster. I'll try to see what's > causing this. I still don't understand well what's causing memory leaks in > the case where all PETSc objects are freed correctly (as can pbe checked > with -log_summary). > > Also, I have tried running either > > valgrind ./my_code -option1 -option2... > > or > > valgrind mpiexec -n 1 ./my_code -option1 -option2... > Note here you would need --trace-children=yes for valgrind. Matt > It seems the second is the correct way to proceed right ? This gives very > different behaviour for valgrind. > > Timothee > > > > 2015-12-14 17:38 GMT+09:00 Timoth?e Nicolas : > >> OK, I'll try that, thx >> >> 2015-12-14 17:38 GMT+09:00 Dave May : >> >>> You have the configure line, so it should be relatively straight forward >>> to configure / build petsc in your home directory. >>> >>> >>> On 14 December 2015 at 09:34, Timoth?e Nicolas < >>> timothee.nicolas at gmail.com> wrote: >>> >>>> OK, The problem is that I don't think I can change this easily as far >>>> as the cluster is concerned. I obtain access to petsc by loading the petsc >>>> module, and even if I have a few choices, I don't see any debug builds... >>>> >>>> 2015-12-14 17:26 GMT+09:00 Dave May : >>>> >>>>> >>>>> >>>>> On Monday, 14 December 2015, Timoth?e Nicolas < >>>>> timothee.nicolas at gmail.com> wrote: >>>>> >>>>>> Hum, OK. I use FORTRAN by the way. Is your comment still valid ? >>>>>> >>>>> >>>>> No. Fortran compilers init variables to zero. >>>>> In this case, I would run a debug build on your OSX machine through >>>>> valgrind and make sure it is clean. >>>>> >>>>> Other obvious thing to check what happens if use exactly the same >>>>> petsc builds on both machines. I see 3.6.1 and 3.6.0 are being used. >>>>> >>>>> For all this type of checking, I would definitely use debug builds on >>>>> both machines. Your cluster build is using the highest level of >>>>> optimization... >>>>> >>>>> >>>>> >>>>> >>>>>> I'll check anyway, but I thought I had been careful about this sort >>>>>> of things. >>>>>> >>>>>> Also, I thought the problem on Mac OS X may have been due to the fact >>>>>> I used the version with debugging on, so I rerun configure with >>>>>> --with-debugging=no, which did not change anything. >>>>>> >>>>>> Thx >>>>>> >>>>>> Timothee >>>>>> >>>>>> >>>>>> 2015-12-14 17:04 GMT+09:00 Dave May : >>>>>> >>>>>>> One suggestion is you have some uninitialized variables in your >>>>>>> pcshell. Despite your arch being called "debug", your configure options >>>>>>> indicate you have turned debugging off. >>>>>>> >>>>>>> C standard doesn't prescribe how uninit variables should be treated >>>>>>> - the behavior is labelled as undefined. As a result, different compilers >>>>>>> on different archs with the same optimization flags can and will treat >>>>>>> uninit variables differently. I find OSX c compilers tend to set them to >>>>>>> zero. >>>>>>> >>>>>>> I suggest compiling a debug build on both machines and trying your >>>>>>> test again. Also, consider running the debug builds through valgrind. >>>>>>> >>>>>>> Thanks, >>>>>>> Dave >>>>>>> >>>>>>> On Monday, 14 December 2015, Timoth?e Nicolas < >>>>>>> timothee.nicolas at gmail.com> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I have noticed I have a VERY big difference in behaviour between >>>>>>>> two machines in my problem, solved with SNES. I can't explain it, because I >>>>>>>> have tested my operators which give the same result. I also checked that >>>>>>>> the vectors fed to the SNES are the same. The problem happens only with my >>>>>>>> shell preconditioner. When I don't use it, and simply solve using -snes_mf, >>>>>>>> I don't see anymore than the usual 3-4 changing digits at the end of the >>>>>>>> residuals. However, when I use my pcshell, the results are completely >>>>>>>> different between the two machines. >>>>>>>> >>>>>>>> I have attached output_SuperComputer.txt and >>>>>>>> output_DesktopComputer.txt, which correspond to the output from the exact >>>>>>>> same code and options (and of course same input data file !). More precisely >>>>>>>> >>>>>>>> output_SuperComputer.txt : output on a supercomputer called Helios, >>>>>>>> sorry I don't know the exact specs. >>>>>>>> In this case, the SNES norms are reduced successively: >>>>>>>> 0 SNES Function norm 4.867111712420e-03 >>>>>>>> 1 SNES Function norm 5.632325929998e-08 >>>>>>>> 2 SNES Function norm 7.427800084502e-15 >>>>>>>> >>>>>>>> output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 GHz >>>>>>>> Intel Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop with >>>>>>>> Mac OS X Mavericks). >>>>>>>> In this case, I obtain the following for the SNES norms, >>>>>>>> while in the other, I obtain >>>>>>>> 0 SNES Function norm 4.867111713544e-03 >>>>>>>> 1 SNES Function norm 1.560094052222e-03 >>>>>>>> 2 SNES Function norm 1.552118650943e-03 >>>>>>>> 3 SNES Function norm 1.552106297094e-03 >>>>>>>> 4 SNES Function norm 1.552106277949e-03 >>>>>>>> which I can't explain, because otherwise the KSP residual (with the >>>>>>>> same operator, which I checked) behave well. >>>>>>>> >>>>>>>> As you can see, the first time the preconditioner is applied (DB_, >>>>>>>> DP_, Drho_ and PS_ solves), the two outputs coincide (except for the few >>>>>>>> last digits, up to 9 actually, which is more than I would expect), and >>>>>>>> everything starts to diverge at the first print of the main KSP (the one >>>>>>>> stemming from the SNES) residual norms. >>>>>>>> >>>>>>>> Do you have an idea what may cause such a strange behaviour ? >>>>>>>> >>>>>>>> Best >>>>>>>> >>>>>>>> Timothee >>>>>>>> >>>>>>> >>>>>> >>>> >>> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.costa.civ at mail.mil Wed Dec 16 08:21:09 2015 From: luis.costa.civ at mail.mil (Costa, Luis CIV (US)) Date: Wed, 16 Dec 2015 14:21:09 +0000 Subject: [petsc-users] HYPRE mpif77 -g error Message-ID: I am having trouble building HYPRE on an IBM iDataPlex cluster running RHEL with Intel MPI compilers. I receive the following error: 2015/12/16 02:29:44 - INFO: [package:run_job] checking whether /site/intel/impi/4.1.3.048/intel64/bin/mpif77 accepts -g... no 2015/12/16 02:29:44 - INFO: [package:run_job] configure: WARNING: compilation failed I've attached my config.log also. Any help is appreciated. Thank you! Luis Costa Research Scientist US ARMY RDECOM-ARDEC Explosive Technology & Prototyping Division Energetics, Warheads & Manufacturing Technologies Directorate RDAR-MEE-W, Building 3022, Room 44 Picatinny, NJ 07806-5000 COM: 973.724.4236 DSN: 880.4236 CELL: 973.668.0044 FAX: 973.724.4059 luis.costa.civ at mail.mil -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: config.log Type: application/octet-stream Size: 21252 bytes Desc: config.log URL: From bsmith at mcs.anl.gov Wed Dec 16 10:52:10 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 16 Dec 2015 10:52:10 -0600 Subject: [petsc-users] HYPRE mpif77 -g error In-Reply-To: References: Message-ID: It found /site/intel/impi/4.1.3.048/intel64/bin/mpif77 to use as the Fortran compiler but that script does not work since it cannot find the g77. configure:5671: /site/intel/impi/4.1.3.048/intel64/bin/mpif77 -c conftest.f >&5 /site/intel/impi/4.1.3.048/intel64/bin/mpif77: line 643: g77: command not found configure:5678: $? = 127 configure: failed program was: | subroutine foobar() | return | end | subroutine foo_bar() | return | end configure:5880: error: in `/u/home/lcosta/.hashdist/tmp/hypre-7ywg3ao4uvlv/src': configure:5883: error: cannot compile a simple Fortran program Regardless in 2015 it is insane to use g77, the gfortran compiler is what you should use. You should talk to whoever maintains the MPI etc on your system. Barry > On Dec 16, 2015, at 8:21 AM, Costa, Luis CIV (US) wrote: > > I am having trouble building HYPRE on an IBM iDataPlex cluster running RHEL with Intel MPI compilers. I receive the following error: > > 2015/12/16 02:29:44 - INFO: [package:run_job] checking whether /site/intel/impi/4.1.3.048/intel64/bin/mpif77 accepts -g... no > 2015/12/16 02:29:44 - INFO: [package:run_job] configure: WARNING: compilation failed > I've attached my config.log also. Any help is appreciated. > > Thank you! > > > Luis Costa > Research Scientist > US ARMY RDECOM-ARDEC > Explosive Technology & Prototyping Division > Energetics, Warheads & Manufacturing Technologies Directorate > RDAR-MEE-W, Building 3022, Room 44 > Picatinny, NJ 07806-5000 > COM: 973.724.4236 > DSN: 880.4236 > CELL: 973.668.0044 > FAX: 973.724.4059 > luis.costa.civ at mail.mil > From balay at mcs.anl.gov Wed Dec 16 10:59:14 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 16 Dec 2015 10:59:14 -0600 Subject: [petsc-users] HYPRE mpif77 -g error In-Reply-To: References: Message-ID: Looks like this hypre is not built via petsc. If so - it would have used 'mpif90' $ ./configure --prefix=/scratch/balay/petsc/arch-linux2-c-debug MAKE=/usr/bin/make --libdir=/scratch/balay/petsc/arch-linux2-c-debug/lib CC="mpicc" CFLAGS="-fPIC -g3" CXX="mpicxx" CXXFLAGS="-g -fPIC" F90="mpif90" F90FLAGS="-fPIC -ffree-line-length-0 -g" F77="mpif90" FFLAGS="-fPIC -ffree-line-length-0 -g" FC="mpif90" FCFLAGS="-fPIC -ffree-line-length-0 -g" --with-MPI-include="/usr/include/mpich2" --with-MPI-lib-dirs="" --with-MPI-libs="" --with-blas-libs= --with-blas-lib-dir= --with-lapack-libs= --with-lapack-lib-dir= --with-blas=yes --with-lapack=yes --with-fmangle-blas=one-underscore --with-fmangle-lapack=one-underscore --without-babel --without-mli --without-fei --without-superlu AR="/usr/bin/ar cr" Satish On Wed, 16 Dec 2015, Barry Smith wrote: > > It found /site/intel/impi/4.1.3.048/intel64/bin/mpif77 to use as the Fortran compiler but that script does not work since it cannot find the g77. > > > configure:5671: /site/intel/impi/4.1.3.048/intel64/bin/mpif77 -c conftest.f >&5 > /site/intel/impi/4.1.3.048/intel64/bin/mpif77: line 643: g77: command not found > configure:5678: $? = 127 > configure: failed program was: > | subroutine foobar() > | return > | end > | subroutine foo_bar() > | return > | end > configure:5880: error: in `/u/home/lcosta/.hashdist/tmp/hypre-7ywg3ao4uvlv/src': > configure:5883: error: cannot compile a simple Fortran program > > Regardless in 2015 it is insane to use g77, the gfortran compiler is what you should use. You should talk to whoever maintains the MPI etc on your system. > > Barry > > > > > On Dec 16, 2015, at 8:21 AM, Costa, Luis CIV (US) wrote: > > > > I am having trouble building HYPRE on an IBM iDataPlex cluster running RHEL with Intel MPI compilers. I receive the following error: > > > > 2015/12/16 02:29:44 - INFO: [package:run_job] checking whether /site/intel/impi/4.1.3.048/intel64/bin/mpif77 accepts -g... no > > 2015/12/16 02:29:44 - INFO: [package:run_job] configure: WARNING: compilation failed > > I've attached my config.log also. Any help is appreciated. > > > > Thank you! > > > > > > Luis Costa > > Research Scientist > > US ARMY RDECOM-ARDEC > > Explosive Technology & Prototyping Division > > Energetics, Warheads & Manufacturing Technologies Directorate > > RDAR-MEE-W, Building 3022, Room 44 > > Picatinny, NJ 07806-5000 > > COM: 973.724.4236 > > DSN: 880.4236 > > CELL: 973.668.0044 > > FAX: 973.724.4059 > > luis.costa.civ at mail.mil > > > > From balay at mcs.anl.gov Wed Dec 16 11:05:51 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 16 Dec 2015 11:05:51 -0600 Subject: [petsc-users] HYPRE mpif77 -g error In-Reply-To: References: Message-ID: $ ./configure --prefix=/u/home/lcosta/.hashdist/bld/hypre/7ywg3ao4uvlv --with-blas=yes --with-blas-libs=blas --with-blas-lib-dirs=/u/home/lcosta/.hashdist/bld/blas/ga3yotglkobk/lib --with-lapack=yes --with-lapack-libs=lapack --with-lapack-lib-dirs=/u/home/lcosta/.hashdist/bld/lapack/bing2onlchtu/lib --with-MPI --with-MPI-include=/u/home/lcosta/.hashdist/bld/mpi/jrk6uiczudla/include --with-MPI-lib-dirs=/u/home/lcosta/.hashdist/bld/mpi/jrk6uiczudla/lib --with-MPI-lib=mpi --without-babel --without-mli --without-fei --without-superlu --enable-shared I see hashdist in there - so you should probably report to hashdist mailing list https://hashdist.github.io/ cc:ing Aron. Satish On Wed, 16 Dec 2015, Satish Balay wrote: > Looks like this hypre is not built via petsc. If so - it would have used 'mpif90' > > $ ./configure --prefix=/scratch/balay/petsc/arch-linux2-c-debug MAKE=/usr/bin/make --libdir=/scratch/balay/petsc/arch-linux2-c-debug/lib CC="mpicc" CFLAGS="-fPIC -g3" CXX="mpicxx" CXXFLAGS="-g -fPIC" F90="mpif90" F90FLAGS="-fPIC -ffree-line-length-0 -g" F77="mpif90" FFLAGS="-fPIC -ffree-line-length-0 -g" FC="mpif90" FCFLAGS="-fPIC -ffree-line-length-0 -g" --with-MPI-include="/usr/include/mpich2" --with-MPI-lib-dirs="" --with-MPI-libs="" --with-blas-libs= --with-blas-lib-dir= --with-lapack-libs= --with-lapack-lib-dir= --with-blas=yes --with-lapack=yes --with-fmangle-blas=one-underscore --with-fmangle-lapack=one-underscore --without-babel --without-mli --without-fei --without-superlu AR="/usr/bin/ar cr" > > Satish > > On Wed, 16 Dec 2015, Barry Smith wrote: > > > > > It found /site/intel/impi/4.1.3.048/intel64/bin/mpif77 to use as the Fortran compiler but that script does not work since it cannot find the g77. > > > > > > configure:5671: /site/intel/impi/4.1.3.048/intel64/bin/mpif77 -c conftest.f >&5 > > /site/intel/impi/4.1.3.048/intel64/bin/mpif77: line 643: g77: command not found > > configure:5678: $? = 127 > > configure: failed program was: > > | subroutine foobar() > > | return > > | end > > | subroutine foo_bar() > > | return > > | end > > configure:5880: error: in `/u/home/lcosta/.hashdist/tmp/hypre-7ywg3ao4uvlv/src': > > configure:5883: error: cannot compile a simple Fortran program > > > > Regardless in 2015 it is insane to use g77, the gfortran compiler is what you should use. You should talk to whoever maintains the MPI etc on your system. > > > > Barry > > > > > > > > > On Dec 16, 2015, at 8:21 AM, Costa, Luis CIV (US) wrote: > > > > > > I am having trouble building HYPRE on an IBM iDataPlex cluster running RHEL with Intel MPI compilers. I receive the following error: > > > > > > 2015/12/16 02:29:44 - INFO: [package:run_job] checking whether /site/intel/impi/4.1.3.048/intel64/bin/mpif77 accepts -g... no > > > 2015/12/16 02:29:44 - INFO: [package:run_job] configure: WARNING: compilation failed > > > I've attached my config.log also. Any help is appreciated. > > > > > > Thank you! > > > > > > > > > Luis Costa > > > Research Scientist > > > US ARMY RDECOM-ARDEC > > > Explosive Technology & Prototyping Division > > > Energetics, Warheads & Manufacturing Technologies Directorate > > > RDAR-MEE-W, Building 3022, Room 44 > > > Picatinny, NJ 07806-5000 > > > COM: 973.724.4236 > > > DSN: 880.4236 > > > CELL: 973.668.0044 > > > FAX: 973.724.4059 > > > luis.costa.civ at mail.mil > > > > > > > > > From aotero at fi.uba.ar Wed Dec 16 14:39:32 2015 From: aotero at fi.uba.ar (Alejandro D Otero) Date: Wed, 16 Dec 2015 17:39:32 -0300 Subject: [petsc-users] Distributing data along with DMPlex Message-ID: Hi, I need some help understanding how to distribute data together with a dmplex representing a FE mesh. At the beginning I define the structure of the dmplex assigning certain number of DoF to cells, edges and vertexes, in one process (the dmplex in the rest is empty) I create a petscsecton and I create an associated global vector with the quantities I want to store. Then I distribute the dmplex over all the processes. * Although this does not perform well it is just a starting point. I know it has to be improved. I would like to have the global vector distributed accordingly so that each process has access to the corresponding local part with its DoF (possibly adding some ghost values corresponding to the shared DoF not taken care by it). Is there any 'correct' way to do that in PETSc? Thanks in advance, Alejandro -------------- next part -------------- An HTML attachment was scrubbed... URL: From jychang48 at gmail.com Wed Dec 16 16:01:47 2015 From: jychang48 at gmail.com (Justin Chang) Date: Wed, 16 Dec 2015 14:01:47 -0800 Subject: [petsc-users] Distributing data along with DMPlex In-Reply-To: References: Message-ID: I think you would follow this order: 1*) create a DMPlex (depth, chart, etc) on rank 0. Other ranks have an empty DM 2) DMPlexDistribute() 3*) Create the PetscSection 4) DMCreateGlobalVector() 5) DMCreateLocalVector() Now you have a global vector and a local vector for your distributed DMPlex. The mapping/ghosting/etc of dofs is already taken care of. * if you're using standard Galerkin FE then in SNES examples 12 and 62 (and maybe others?) the first step is handled through the mesh generation functions and step 3 is handled through step 4 Thanks, Justin On Wednesday, December 16, 2015, Alejandro D Otero wrote: > Hi, I need some help understanding how to distribute data together with a > dmplex representing a FE mesh. > At the beginning I define the structure of the dmplex assigning certain > number of DoF to cells, edges and vertexes, in one process (the dmplex in > the rest is empty) > I create a petscsecton and I create an associated global vector with the > quantities I want to store. > Then I distribute the dmplex over all the processes. > * Although this does not perform well it is just a starting point. I know > it has to be improved. > > I would like to have the global vector distributed accordingly so that > each process has access to the corresponding local part with its DoF > (possibly adding some ghost values corresponding to the shared DoF not > taken care by it). > > Is there any 'correct' way to do that in PETSc? > > Thanks in advance, > > Alejandro > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaabell at ucdavis.edu Thu Dec 17 01:06:26 2015 From: jaabell at ucdavis.edu (Jose A. Abell M.) Date: Wed, 16 Dec 2015 23:06:26 -0800 Subject: [petsc-users] Slow MatAssemblyBegin MAT_FINAL_ASSEMBLY Message-ID: Hello dear PETSc users, This is a problem that pops up often, from what I see, in the mailing list. My program takes a long time assembling the matrix. What I know: - Matrix Size is (MatMPIAIJ) 2670402 - Number of processes running PETSc: 95 - Not going to virtual memory (no swapping, used mem well withing each node's capacity) - System is partitioned with ParMETIS for load balancing - I see memory moving around in each node (total used memory changes a bit, grows and then frees) - Matrix is filled in blocks of size 81x81 (FEM code, so this ends up being a sparse matrix) - I don't do flushes at all. Only MAT_FINAL_ASSEMBLY when all the MatSetValues are done. Should I do MAT_FLUSH_ASSEMBLY even though I have enough memory to store the buffers? If so, how often? Every 100 blocks? What else could it be? Its taking several hours to asseble this matrix. I re-use the sparsity pattern, so subsequent assemblies are fast. Does this mean that my preallocation is wrong? Regards, -- Jos? Abell *PhD Candidate* Computational Geomechanics Group Dept. of Civil and Environmental Engineering UC Davis www.joseabell.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Thu Dec 17 02:58:17 2015 From: dave.mayhem23 at gmail.com (Dave May) Date: Thu, 17 Dec 2015 09:58:17 +0100 Subject: [petsc-users] Slow MatAssemblyBegin MAT_FINAL_ASSEMBLY In-Reply-To: References: Message-ID: On 17 December 2015 at 08:06, Jose A. Abell M. wrote: > Hello dear PETSc users, > > This is a problem that pops up often, from what I see, in the mailing > list. My program takes a long time assembling the matrix. > > What I know: > > > - Matrix Size is (MatMPIAIJ) 2670402 > - Number of processes running PETSc: 95 > - Not going to virtual memory (no swapping, used mem well withing each > node's capacity) > - System is partitioned with ParMETIS for load balancing > - I see memory moving around in each node (total used memory changes a > bit, grows and then frees) > - Matrix is filled in blocks of size 81x81 (FEM code, so this ends up > being a sparse matrix) > - I don't do flushes at all. Only MAT_FINAL_ASSEMBLY when all the > MatSetValues are done. > > Should I do MAT_FLUSH_ASSEMBLY even though I have enough memory to store > the buffers? If so, how often? Every 100 blocks? > > What else could it be? > > Its taking several hours to asseble this matrix. I re-use the sparsity > pattern, so subsequent assemblies are fast. Does this mean that my > preallocation is wrong? > The preallocation could be wrong. That is the usual cause of very slow matrix assembly. To confirm this hypothesis, run your code with the command line option -info. You will get an enormous amount of information in stdout. You might consider using -info with a smallish problem size / core count. Inspect the output generated by -info and look for lines like this: [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 If the number of mallocs during MatSetValues() is not zero, then your preallocation is not exactly correct. A small number of mallocs, say less than 10, might be accepted (performance wise). However if the number of mallocs is > 100, then assembly time will be terribly slow. Thanks, Dave > > Regards, > > > > > -- > > Jos? Abell > *PhD Candidate* > Computational Geomechanics Group > Dept. of Civil and Environmental Engineering > UC Davis > www.joseabell.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothee.nicolas at gmail.com Thu Dec 17 04:00:21 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Thu, 17 Dec 2015 19:00:21 +0900 Subject: [petsc-users] Big discrepancy between machines In-Reply-To: References: Message-ID: Hi, So, valgrind is OK (at least on the local machine. Actually on the cluster helios, it produces strange results even for the simplest petsc program PetscInitialize followed by PetscFinalize, I will try to figure this out with their technical team), and I have also tried with exactly the same versions (3.6.0) and it does not change the behavior. So now I would like to now how to have a grip on what comes in and out of the SNES and the KSP internal to the SNES. That is, I would like to inspect manually the vector which enters the SNES in the first place (should be zero I believe), what is being fed to the KSP, and the vector which comes out of it, etc. if possible at each iteration of the SNES. I want to actually *see* these vectors, and compute there norm by hand. The trouble is, it is really hard to understand why the newton residuals are not reduced since the KSP converges so nicely. This does not make any sense to me, so I want to know what happens to the vectors. But on the SNES list of routines, I did not find the tools that would allow me to do that (and messing around with the C code is too hard for me, it would take me weeks). Does someone have a hint ? Thx Timothee 2015-12-15 14:20 GMT+09:00 Matthew Knepley : > On Mon, Dec 14, 2015 at 11:06 PM, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > >> There is a diference in valgrind indeed between the two. It seems to be >> clean on my desktop Mac OS X but not on the cluster. I'll try to see what's >> causing this. I still don't understand well what's causing memory leaks in >> the case where all PETSc objects are freed correctly (as can pbe checked >> with -log_summary). >> >> Also, I have tried running either >> >> valgrind ./my_code -option1 -option2... >> >> or >> >> valgrind mpiexec -n 1 ./my_code -option1 -option2... >> > > Note here you would need --trace-children=yes for valgrind. > > Matt > > >> It seems the second is the correct way to proceed right ? This gives very >> different behaviour for valgrind. >> >> Timothee >> >> >> >> 2015-12-14 17:38 GMT+09:00 Timoth?e Nicolas : >> >>> OK, I'll try that, thx >>> >>> 2015-12-14 17:38 GMT+09:00 Dave May : >>> >>>> You have the configure line, so it should be relatively straight >>>> forward to configure / build petsc in your home directory. >>>> >>>> >>>> On 14 December 2015 at 09:34, Timoth?e Nicolas < >>>> timothee.nicolas at gmail.com> wrote: >>>> >>>>> OK, The problem is that I don't think I can change this easily as far >>>>> as the cluster is concerned. I obtain access to petsc by loading the petsc >>>>> module, and even if I have a few choices, I don't see any debug builds... >>>>> >>>>> 2015-12-14 17:26 GMT+09:00 Dave May : >>>>> >>>>>> >>>>>> >>>>>> On Monday, 14 December 2015, Timoth?e Nicolas < >>>>>> timothee.nicolas at gmail.com> wrote: >>>>>> >>>>>>> Hum, OK. I use FORTRAN by the way. Is your comment still valid ? >>>>>>> >>>>>> >>>>>> No. Fortran compilers init variables to zero. >>>>>> In this case, I would run a debug build on your OSX machine through >>>>>> valgrind and make sure it is clean. >>>>>> >>>>>> Other obvious thing to check what happens if use exactly the same >>>>>> petsc builds on both machines. I see 3.6.1 and 3.6.0 are being used. >>>>>> >>>>>> For all this type of checking, I would definitely use debug builds on >>>>>> both machines. Your cluster build is using the highest level of >>>>>> optimization... >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> I'll check anyway, but I thought I had been careful about this sort >>>>>>> of things. >>>>>>> >>>>>>> Also, I thought the problem on Mac OS X may have been due to the >>>>>>> fact I used the version with debugging on, so I rerun configure with >>>>>>> --with-debugging=no, which did not change anything. >>>>>>> >>>>>>> Thx >>>>>>> >>>>>>> Timothee >>>>>>> >>>>>>> >>>>>>> 2015-12-14 17:04 GMT+09:00 Dave May : >>>>>>> >>>>>>>> One suggestion is you have some uninitialized variables in your >>>>>>>> pcshell. Despite your arch being called "debug", your configure options >>>>>>>> indicate you have turned debugging off. >>>>>>>> >>>>>>>> C standard doesn't prescribe how uninit variables should be treated >>>>>>>> - the behavior is labelled as undefined. As a result, different compilers >>>>>>>> on different archs with the same optimization flags can and will treat >>>>>>>> uninit variables differently. I find OSX c compilers tend to set them to >>>>>>>> zero. >>>>>>>> >>>>>>>> I suggest compiling a debug build on both machines and trying your >>>>>>>> test again. Also, consider running the debug builds through valgrind. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Dave >>>>>>>> >>>>>>>> On Monday, 14 December 2015, Timoth?e Nicolas < >>>>>>>> timothee.nicolas at gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I have noticed I have a VERY big difference in behaviour between >>>>>>>>> two machines in my problem, solved with SNES. I can't explain it, because I >>>>>>>>> have tested my operators which give the same result. I also checked that >>>>>>>>> the vectors fed to the SNES are the same. The problem happens only with my >>>>>>>>> shell preconditioner. When I don't use it, and simply solve using -snes_mf, >>>>>>>>> I don't see anymore than the usual 3-4 changing digits at the end of the >>>>>>>>> residuals. However, when I use my pcshell, the results are completely >>>>>>>>> different between the two machines. >>>>>>>>> >>>>>>>>> I have attached output_SuperComputer.txt and >>>>>>>>> output_DesktopComputer.txt, which correspond to the output from the exact >>>>>>>>> same code and options (and of course same input data file !). More precisely >>>>>>>>> >>>>>>>>> output_SuperComputer.txt : output on a supercomputer called >>>>>>>>> Helios, sorry I don't know the exact specs. >>>>>>>>> In this case, the SNES norms are reduced successively: >>>>>>>>> 0 SNES Function norm 4.867111712420e-03 >>>>>>>>> 1 SNES Function norm 5.632325929998e-08 >>>>>>>>> 2 SNES Function norm 7.427800084502e-15 >>>>>>>>> >>>>>>>>> output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 GHz >>>>>>>>> Intel Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop with >>>>>>>>> Mac OS X Mavericks). >>>>>>>>> In this case, I obtain the following for the SNES norms, >>>>>>>>> while in the other, I obtain >>>>>>>>> 0 SNES Function norm 4.867111713544e-03 >>>>>>>>> 1 SNES Function norm 1.560094052222e-03 >>>>>>>>> 2 SNES Function norm 1.552118650943e-03 >>>>>>>>> 3 SNES Function norm 1.552106297094e-03 >>>>>>>>> 4 SNES Function norm 1.552106277949e-03 >>>>>>>>> which I can't explain, because otherwise the KSP residual (with >>>>>>>>> the same operator, which I checked) behave well. >>>>>>>>> >>>>>>>>> As you can see, the first time the preconditioner is applied (DB_, >>>>>>>>> DP_, Drho_ and PS_ solves), the two outputs coincide (except for the few >>>>>>>>> last digits, up to 9 actually, which is more than I would expect), and >>>>>>>>> everything starts to diverge at the first print of the main KSP (the one >>>>>>>>> stemming from the SNES) residual norms. >>>>>>>>> >>>>>>>>> Do you have an idea what may cause such a strange behaviour ? >>>>>>>>> >>>>>>>>> Best >>>>>>>>> >>>>>>>>> Timothee >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Thu Dec 17 04:19:29 2015 From: dave.mayhem23 at gmail.com (Dave May) Date: Thu, 17 Dec 2015 11:19:29 +0100 Subject: [petsc-users] Big discrepancy between machines In-Reply-To: References: Message-ID: On 17 December 2015 at 11:00, Timoth?e Nicolas wrote: > Hi, > > So, valgrind is OK (at least on the local machine. Actually on the cluster > helios, it produces strange results even for the simplest petsc program > PetscInitialize followed by PetscFinalize, I will try to figure this out > with their technical team), and I have also tried with exactly the same > versions (3.6.0) and it does not change the behavior. > > So now I would like to now how to have a grip on what comes in and out of > the SNES and the KSP internal to the SNES. That is, I would like to inspect > manually the vector which enters the SNES in the first place (should be > zero I believe), what is being fed to the KSP, and the vector which comes > out of it, etc. if possible at each iteration of the SNES. I want to > actually *see* these vectors, and compute there norm by hand. The trouble > is, it is really hard to understand why the newton residuals are not > reduced since the KSP converges so nicely. This does not make any sense to > me, so I want to know what happens to the vectors. But on the SNES list of > routines, I did not find the tools that would allow me to do that (and > messing around with the C code is too hard for me, it would take me weeks). > Does someone have a hint ? > The only sane way to do this is to write a custom monitor for your SNES object. http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESMonitorSet.html Inside your monitor, you have access the SNES, and everything it defines, e.g. the current solution, non-linear residual, KSP etc. See these pages http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetSolution.html http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetFunction.html http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetKSP.html Then you can pull apart the residual and compute specific norms (or plot the residual). Hopefully you can access everything you need to perform your analysis. Cheers, Dave > > Thx > > Timothee > > > > > 2015-12-15 14:20 GMT+09:00 Matthew Knepley : > >> On Mon, Dec 14, 2015 at 11:06 PM, Timoth?e Nicolas < >> timothee.nicolas at gmail.com> wrote: >> >>> There is a diference in valgrind indeed between the two. It seems to be >>> clean on my desktop Mac OS X but not on the cluster. I'll try to see what's >>> causing this. I still don't understand well what's causing memory leaks in >>> the case where all PETSc objects are freed correctly (as can pbe checked >>> with -log_summary). >>> >>> Also, I have tried running either >>> >>> valgrind ./my_code -option1 -option2... >>> >>> or >>> >>> valgrind mpiexec -n 1 ./my_code -option1 -option2... >>> >> >> Note here you would need --trace-children=yes for valgrind. >> >> Matt >> >> >>> It seems the second is the correct way to proceed right ? This gives >>> very different behaviour for valgrind. >>> >>> Timothee >>> >>> >>> >>> 2015-12-14 17:38 GMT+09:00 Timoth?e Nicolas >>> : >>> >>>> OK, I'll try that, thx >>>> >>>> 2015-12-14 17:38 GMT+09:00 Dave May : >>>> >>>>> You have the configure line, so it should be relatively straight >>>>> forward to configure / build petsc in your home directory. >>>>> >>>>> >>>>> On 14 December 2015 at 09:34, Timoth?e Nicolas < >>>>> timothee.nicolas at gmail.com> wrote: >>>>> >>>>>> OK, The problem is that I don't think I can change this easily as far >>>>>> as the cluster is concerned. I obtain access to petsc by loading the petsc >>>>>> module, and even if I have a few choices, I don't see any debug builds... >>>>>> >>>>>> 2015-12-14 17:26 GMT+09:00 Dave May : >>>>>> >>>>>>> >>>>>>> >>>>>>> On Monday, 14 December 2015, Timoth?e Nicolas < >>>>>>> timothee.nicolas at gmail.com> wrote: >>>>>>> >>>>>>>> Hum, OK. I use FORTRAN by the way. Is your comment still valid ? >>>>>>>> >>>>>>> >>>>>>> No. Fortran compilers init variables to zero. >>>>>>> In this case, I would run a debug build on your OSX machine through >>>>>>> valgrind and make sure it is clean. >>>>>>> >>>>>>> Other obvious thing to check what happens if use exactly the same >>>>>>> petsc builds on both machines. I see 3.6.1 and 3.6.0 are being used. >>>>>>> >>>>>>> For all this type of checking, I would definitely use debug builds >>>>>>> on both machines. Your cluster build is using the highest level of >>>>>>> optimization... >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> I'll check anyway, but I thought I had been careful about this sort >>>>>>>> of things. >>>>>>>> >>>>>>>> Also, I thought the problem on Mac OS X may have been due to the >>>>>>>> fact I used the version with debugging on, so I rerun configure with >>>>>>>> --with-debugging=no, which did not change anything. >>>>>>>> >>>>>>>> Thx >>>>>>>> >>>>>>>> Timothee >>>>>>>> >>>>>>>> >>>>>>>> 2015-12-14 17:04 GMT+09:00 Dave May : >>>>>>>> >>>>>>>>> One suggestion is you have some uninitialized variables in your >>>>>>>>> pcshell. Despite your arch being called "debug", your configure options >>>>>>>>> indicate you have turned debugging off. >>>>>>>>> >>>>>>>>> C standard doesn't prescribe how uninit variables should be >>>>>>>>> treated - the behavior is labelled as undefined. As a result, different >>>>>>>>> compilers on different archs with the same optimization flags can and will >>>>>>>>> treat uninit variables differently. I find OSX c compilers tend to set them >>>>>>>>> to zero. >>>>>>>>> >>>>>>>>> I suggest compiling a debug build on both machines and trying your >>>>>>>>> test again. Also, consider running the debug builds through valgrind. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Dave >>>>>>>>> >>>>>>>>> On Monday, 14 December 2015, Timoth?e Nicolas < >>>>>>>>> timothee.nicolas at gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I have noticed I have a VERY big difference in behaviour between >>>>>>>>>> two machines in my problem, solved with SNES. I can't explain it, because I >>>>>>>>>> have tested my operators which give the same result. I also checked that >>>>>>>>>> the vectors fed to the SNES are the same. The problem happens only with my >>>>>>>>>> shell preconditioner. When I don't use it, and simply solve using -snes_mf, >>>>>>>>>> I don't see anymore than the usual 3-4 changing digits at the end of the >>>>>>>>>> residuals. However, when I use my pcshell, the results are completely >>>>>>>>>> different between the two machines. >>>>>>>>>> >>>>>>>>>> I have attached output_SuperComputer.txt and >>>>>>>>>> output_DesktopComputer.txt, which correspond to the output from the exact >>>>>>>>>> same code and options (and of course same input data file !). More precisely >>>>>>>>>> >>>>>>>>>> output_SuperComputer.txt : output on a supercomputer called >>>>>>>>>> Helios, sorry I don't know the exact specs. >>>>>>>>>> In this case, the SNES norms are reduced successively: >>>>>>>>>> 0 SNES Function norm 4.867111712420e-03 >>>>>>>>>> 1 SNES Function norm 5.632325929998e-08 >>>>>>>>>> 2 SNES Function norm 7.427800084502e-15 >>>>>>>>>> >>>>>>>>>> output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 >>>>>>>>>> GHz Intel Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop >>>>>>>>>> with Mac OS X Mavericks). >>>>>>>>>> In this case, I obtain the following for the SNES norms, >>>>>>>>>> while in the other, I obtain >>>>>>>>>> 0 SNES Function norm 4.867111713544e-03 >>>>>>>>>> 1 SNES Function norm 1.560094052222e-03 >>>>>>>>>> 2 SNES Function norm 1.552118650943e-03 >>>>>>>>>> 3 SNES Function norm 1.552106297094e-03 >>>>>>>>>> 4 SNES Function norm 1.552106277949e-03 >>>>>>>>>> which I can't explain, because otherwise the KSP residual (with >>>>>>>>>> the same operator, which I checked) behave well. >>>>>>>>>> >>>>>>>>>> As you can see, the first time the preconditioner is applied >>>>>>>>>> (DB_, DP_, Drho_ and PS_ solves), the two outputs coincide (except for the >>>>>>>>>> few last digits, up to 9 actually, which is more than I would expect), and >>>>>>>>>> everything starts to diverge at the first print of the main KSP (the one >>>>>>>>>> stemming from the SNES) residual norms. >>>>>>>>>> >>>>>>>>>> Do you have an idea what may cause such a strange behaviour ? >>>>>>>>>> >>>>>>>>>> Best >>>>>>>>>> >>>>>>>>>> Timothee >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>> >>>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothee.nicolas at gmail.com Thu Dec 17 07:47:12 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Thu, 17 Dec 2015 22:47:12 +0900 Subject: [petsc-users] Big discrepancy between machines In-Reply-To: References: Message-ID: It works very smoothly for the SNES :-), but KSPGetSolution keeps returning a zero vector... KSPGetResidualNorm gives me the norm alright, but I would like to actually see the vectors. Is KSPGetSolution the wrong routine ? Thx Timoth?e 2015-12-17 19:19 GMT+09:00 Dave May : > > > On 17 December 2015 at 11:00, Timoth?e Nicolas > wrote: > >> Hi, >> >> So, valgrind is OK (at least on the local machine. Actually on the >> cluster helios, it produces strange results even for the simplest petsc >> program PetscInitialize followed by PetscFinalize, I will try to figure >> this out with their technical team), and I have also tried with exactly the >> same versions (3.6.0) and it does not change the behavior. >> >> So now I would like to now how to have a grip on what comes in and out of >> the SNES and the KSP internal to the SNES. That is, I would like to inspect >> manually the vector which enters the SNES in the first place (should be >> zero I believe), what is being fed to the KSP, and the vector which comes >> out of it, etc. if possible at each iteration of the SNES. I want to >> actually *see* these vectors, and compute there norm by hand. The >> trouble is, it is really hard to understand why the newton residuals are >> not reduced since the KSP converges so nicely. This does not make any sense >> to me, so I want to know what happens to the vectors. But on the SNES list >> of routines, I did not find the tools that would allow me to do that (and >> messing around with the C code is too hard for me, it would take me weeks). >> Does someone have a hint ? >> > > The only sane way to do this is to write a custom monitor for your SNES > object. > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESMonitorSet.html > > Inside your monitor, you have access the SNES, and everything it defines, > e.g. the current solution, non-linear residual, KSP etc. See these pages > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetSolution.html > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetFunction.html > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetKSP.html > > Then you can pull apart the residual and compute specific norms (or plot > the residual). > > Hopefully you can access everything you need to perform your analysis. > > Cheers, > Dave > > >> >> Thx >> >> Timothee >> >> >> >> >> 2015-12-15 14:20 GMT+09:00 Matthew Knepley : >> >>> On Mon, Dec 14, 2015 at 11:06 PM, Timoth?e Nicolas < >>> timothee.nicolas at gmail.com> wrote: >>> >>>> There is a diference in valgrind indeed between the two. It seems to be >>>> clean on my desktop Mac OS X but not on the cluster. I'll try to see what's >>>> causing this. I still don't understand well what's causing memory leaks in >>>> the case where all PETSc objects are freed correctly (as can pbe checked >>>> with -log_summary). >>>> >>>> Also, I have tried running either >>>> >>>> valgrind ./my_code -option1 -option2... >>>> >>>> or >>>> >>>> valgrind mpiexec -n 1 ./my_code -option1 -option2... >>>> >>> >>> Note here you would need --trace-children=yes for valgrind. >>> >>> Matt >>> >>> >>>> It seems the second is the correct way to proceed right ? This gives >>>> very different behaviour for valgrind. >>>> >>>> Timothee >>>> >>>> >>>> >>>> 2015-12-14 17:38 GMT+09:00 Timoth?e Nicolas >>> >: >>>> >>>>> OK, I'll try that, thx >>>>> >>>>> 2015-12-14 17:38 GMT+09:00 Dave May : >>>>> >>>>>> You have the configure line, so it should be relatively straight >>>>>> forward to configure / build petsc in your home directory. >>>>>> >>>>>> >>>>>> On 14 December 2015 at 09:34, Timoth?e Nicolas < >>>>>> timothee.nicolas at gmail.com> wrote: >>>>>> >>>>>>> OK, The problem is that I don't think I can change this easily as >>>>>>> far as the cluster is concerned. I obtain access to petsc by loading the >>>>>>> petsc module, and even if I have a few choices, I don't see any debug >>>>>>> builds... >>>>>>> >>>>>>> 2015-12-14 17:26 GMT+09:00 Dave May : >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Monday, 14 December 2015, Timoth?e Nicolas < >>>>>>>> timothee.nicolas at gmail.com> wrote: >>>>>>>> >>>>>>>>> Hum, OK. I use FORTRAN by the way. Is your comment still valid ? >>>>>>>>> >>>>>>>> >>>>>>>> No. Fortran compilers init variables to zero. >>>>>>>> In this case, I would run a debug build on your OSX machine through >>>>>>>> valgrind and make sure it is clean. >>>>>>>> >>>>>>>> Other obvious thing to check what happens if use exactly the same >>>>>>>> petsc builds on both machines. I see 3.6.1 and 3.6.0 are being used. >>>>>>>> >>>>>>>> For all this type of checking, I would definitely use debug builds >>>>>>>> on both machines. Your cluster build is using the highest level of >>>>>>>> optimization... >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> I'll check anyway, but I thought I had been careful about this >>>>>>>>> sort of things. >>>>>>>>> >>>>>>>>> Also, I thought the problem on Mac OS X may have been due to the >>>>>>>>> fact I used the version with debugging on, so I rerun configure with >>>>>>>>> --with-debugging=no, which did not change anything. >>>>>>>>> >>>>>>>>> Thx >>>>>>>>> >>>>>>>>> Timothee >>>>>>>>> >>>>>>>>> >>>>>>>>> 2015-12-14 17:04 GMT+09:00 Dave May : >>>>>>>>> >>>>>>>>>> One suggestion is you have some uninitialized variables in your >>>>>>>>>> pcshell. Despite your arch being called "debug", your configure options >>>>>>>>>> indicate you have turned debugging off. >>>>>>>>>> >>>>>>>>>> C standard doesn't prescribe how uninit variables should be >>>>>>>>>> treated - the behavior is labelled as undefined. As a result, different >>>>>>>>>> compilers on different archs with the same optimization flags can and will >>>>>>>>>> treat uninit variables differently. I find OSX c compilers tend to set them >>>>>>>>>> to zero. >>>>>>>>>> >>>>>>>>>> I suggest compiling a debug build on both machines and trying >>>>>>>>>> your test again. Also, consider running the debug builds through valgrind. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Dave >>>>>>>>>> >>>>>>>>>> On Monday, 14 December 2015, Timoth?e Nicolas < >>>>>>>>>> timothee.nicolas at gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I have noticed I have a VERY big difference in behaviour between >>>>>>>>>>> two machines in my problem, solved with SNES. I can't explain it, because I >>>>>>>>>>> have tested my operators which give the same result. I also checked that >>>>>>>>>>> the vectors fed to the SNES are the same. The problem happens only with my >>>>>>>>>>> shell preconditioner. When I don't use it, and simply solve using -snes_mf, >>>>>>>>>>> I don't see anymore than the usual 3-4 changing digits at the end of the >>>>>>>>>>> residuals. However, when I use my pcshell, the results are completely >>>>>>>>>>> different between the two machines. >>>>>>>>>>> >>>>>>>>>>> I have attached output_SuperComputer.txt and >>>>>>>>>>> output_DesktopComputer.txt, which correspond to the output from the exact >>>>>>>>>>> same code and options (and of course same input data file !). More precisely >>>>>>>>>>> >>>>>>>>>>> output_SuperComputer.txt : output on a supercomputer called >>>>>>>>>>> Helios, sorry I don't know the exact specs. >>>>>>>>>>> In this case, the SNES norms are reduced successively: >>>>>>>>>>> 0 SNES Function norm 4.867111712420e-03 >>>>>>>>>>> 1 SNES Function norm 5.632325929998e-08 >>>>>>>>>>> 2 SNES Function norm 7.427800084502e-15 >>>>>>>>>>> >>>>>>>>>>> output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 >>>>>>>>>>> GHz Intel Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop >>>>>>>>>>> with Mac OS X Mavericks). >>>>>>>>>>> In this case, I obtain the following for the SNES norms, >>>>>>>>>>> while in the other, I obtain >>>>>>>>>>> 0 SNES Function norm 4.867111713544e-03 >>>>>>>>>>> 1 SNES Function norm 1.560094052222e-03 >>>>>>>>>>> 2 SNES Function norm 1.552118650943e-03 >>>>>>>>>>> 3 SNES Function norm 1.552106297094e-03 >>>>>>>>>>> 4 SNES Function norm 1.552106277949e-03 >>>>>>>>>>> which I can't explain, because otherwise the KSP residual (with >>>>>>>>>>> the same operator, which I checked) behave well. >>>>>>>>>>> >>>>>>>>>>> As you can see, the first time the preconditioner is applied >>>>>>>>>>> (DB_, DP_, Drho_ and PS_ solves), the two outputs coincide (except for the >>>>>>>>>>> few last digits, up to 9 actually, which is more than I would expect), and >>>>>>>>>>> everything starts to diverge at the first print of the main KSP (the one >>>>>>>>>>> stemming from the SNES) residual norms. >>>>>>>>>>> >>>>>>>>>>> Do you have an idea what may cause such a strange behaviour ? >>>>>>>>>>> >>>>>>>>>>> Best >>>>>>>>>>> >>>>>>>>>>> Timothee >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaabell at ucdavis.edu Thu Dec 17 09:10:30 2015 From: jaabell at ucdavis.edu (Jose A. Abell M.) Date: Thu, 17 Dec 2015 07:10:30 -0800 Subject: [petsc-users] Slow MatAssemblyBegin MAT_FINAL_ASSEMBLY In-Reply-To: References: Message-ID: Thank you Dave! Do you have a rough idea of how long a matrix like that should take to assemble? Not hours. Right? Regards, Jose -- Jos? Abell *PhD Candidate* Computational Geomechanics Group Dept. of Civil and Environmental Engineering UC Davis www.joseabell.com On Thu, Dec 17, 2015 at 12:58 AM, Dave May wrote: > > > On 17 December 2015 at 08:06, Jose A. Abell M. > wrote: > >> Hello dear PETSc users, >> >> This is a problem that pops up often, from what I see, in the mailing >> list. My program takes a long time assembling the matrix. >> >> What I know: >> >> >> - Matrix Size is (MatMPIAIJ) 2670402 >> - Number of processes running PETSc: 95 >> - Not going to virtual memory (no swapping, used mem well withing >> each node's capacity) >> - System is partitioned with ParMETIS for load balancing >> - I see memory moving around in each node (total used memory changes >> a bit, grows and then frees) >> - Matrix is filled in blocks of size 81x81 (FEM code, so this ends up >> being a sparse matrix) >> - I don't do flushes at all. Only MAT_FINAL_ASSEMBLY when all the >> MatSetValues are done. >> >> Should I do MAT_FLUSH_ASSEMBLY even though I have enough memory to store >> the buffers? If so, how often? Every 100 blocks? >> >> What else could it be? >> >> Its taking several hours to asseble this matrix. I re-use the sparsity >> pattern, so subsequent assemblies are fast. Does this mean that my >> preallocation is wrong? >> > > The preallocation could be wrong. That is the usual cause of very slow > matrix assembly. To confirm this hypothesis, run your code with the command > line option -info. You will get an enormous amount of information in > stdout. You might consider using -info with a smallish problem size / core > count. > > Inspect the output generated by -info and look for lines like this: > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > If the number of mallocs during MatSetValues() is not zero, then your > preallocation is not exactly correct. A small number of mallocs, say less > than 10, might be accepted (performance wise). However if the number of > mallocs is > 100, then assembly time will be terribly slow. > > Thanks, > Dave > > > > >> >> Regards, >> >> >> >> >> -- >> >> Jos? Abell >> *PhD Candidate* >> Computational Geomechanics Group >> Dept. of Civil and Environmental Engineering >> UC Davis >> www.joseabell.com >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Dec 17 10:45:48 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 17 Dec 2015 10:45:48 -0600 Subject: [petsc-users] Big discrepancy between machines In-Reply-To: References: Message-ID: <219A5E6E-ADD7-4647-8487-C30D29F05344@mcs.anl.gov> > On Dec 17, 2015, at 7:47 AM, Timoth?e Nicolas wrote: > > It works very smoothly for the SNES :-), but KSPGetSolution keeps returning a zero vector... KSPGetResidualNorm gives me the norm alright, but I would like to actually see the vectors. Is KSPGetSolution the wrong routine ? If you are using GMRES, which is the default, it actually DOES NOT have a representation of the solution at each step. Yes that seems odd but it only computes the solution vector at a restart or when the iteration ends. This is why KSPGetSolution doesn't provide anything useful. You can use KSPBuildSolution() to have it construct the "current" solution whenever you need it. Barry > > Thx > > Timoth?e > > 2015-12-17 19:19 GMT+09:00 Dave May : > > > On 17 December 2015 at 11:00, Timoth?e Nicolas wrote: > Hi, > > So, valgrind is OK (at least on the local machine. Actually on the cluster helios, it produces strange results even for the simplest petsc program PetscInitialize followed by PetscFinalize, I will try to figure this out with their technical team), and I have also tried with exactly the same versions (3.6.0) and it does not change the behavior. > > So now I would like to now how to have a grip on what comes in and out of the SNES and the KSP internal to the SNES. That is, I would like to inspect manually the vector which enters the SNES in the first place (should be zero I believe), what is being fed to the KSP, and the vector which comes out of it, etc. if possible at each iteration of the SNES. I want to actually see these vectors, and compute there norm by hand. The trouble is, it is really hard to understand why the newton residuals are not reduced since the KSP converges so nicely. This does not make any sense to me, so I want to know what happens to the vectors. But on the SNES list of routines, I did not find the tools that would allow me to do that (and messing around with the C code is too hard for me, it would take me weeks). Does someone have a hint ? > > The only sane way to do this is to write a custom monitor for your SNES object. > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESMonitorSet.html > > Inside your monitor, you have access the SNES, and everything it defines, e.g. the current solution, non-linear residual, KSP etc. See these pages > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetSolution.html > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetFunction.html > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetKSP.html > > Then you can pull apart the residual and compute specific norms (or plot the residual). > > Hopefully you can access everything you need to perform your analysis. > > Cheers, > Dave > > > Thx > > Timothee > > > > > 2015-12-15 14:20 GMT+09:00 Matthew Knepley : > On Mon, Dec 14, 2015 at 11:06 PM, Timoth?e Nicolas wrote: > There is a diference in valgrind indeed between the two. It seems to be clean on my desktop Mac OS X but not on the cluster. I'll try to see what's causing this. I still don't understand well what's causing memory leaks in the case where all PETSc objects are freed correctly (as can pbe checked with -log_summary). > > Also, I have tried running either > > valgrind ./my_code -option1 -option2... > > or > > valgrind mpiexec -n 1 ./my_code -option1 -option2... > > Note here you would need --trace-children=yes for valgrind. > > Matt > > It seems the second is the correct way to proceed right ? This gives very different behaviour for valgrind. > > Timothee > > > > 2015-12-14 17:38 GMT+09:00 Timoth?e Nicolas : > OK, I'll try that, thx > > 2015-12-14 17:38 GMT+09:00 Dave May : > You have the configure line, so it should be relatively straight forward to configure / build petsc in your home directory. > > > On 14 December 2015 at 09:34, Timoth?e Nicolas wrote: > OK, The problem is that I don't think I can change this easily as far as the cluster is concerned. I obtain access to petsc by loading the petsc module, and even if I have a few choices, I don't see any debug builds... > > 2015-12-14 17:26 GMT+09:00 Dave May : > > > On Monday, 14 December 2015, Timoth?e Nicolas wrote: > Hum, OK. I use FORTRAN by the way. Is your comment still valid ? > > No. Fortran compilers init variables to zero. > In this case, I would run a debug build on your OSX machine through valgrind and make sure it is clean. > > Other obvious thing to check what happens if use exactly the same petsc builds on both machines. I see 3.6.1 and 3.6.0 are being used. > > For all this type of checking, I would definitely use debug builds on both machines. Your cluster build is using the highest level of optimization... > > > > I'll check anyway, but I thought I had been careful about this sort of things. > > Also, I thought the problem on Mac OS X may have been due to the fact I used the version with debugging on, so I rerun configure with --with-debugging=no, which did not change anything. > > Thx > > Timothee > > > 2015-12-14 17:04 GMT+09:00 Dave May : > One suggestion is you have some uninitialized variables in your pcshell. Despite your arch being called "debug", your configure options indicate you have turned debugging off. > > C standard doesn't prescribe how uninit variables should be treated - the behavior is labelled as undefined. As a result, different compilers on different archs with the same optimization flags can and will treat uninit variables differently. I find OSX c compilers tend to set them to zero. > > I suggest compiling a debug build on both machines and trying your test again. Also, consider running the debug builds through valgrind. > > Thanks, > Dave > > On Monday, 14 December 2015, Timoth?e Nicolas wrote: > Hi, > > I have noticed I have a VERY big difference in behaviour between two machines in my problem, solved with SNES. I can't explain it, because I have tested my operators which give the same result. I also checked that the vectors fed to the SNES are the same. The problem happens only with my shell preconditioner. When I don't use it, and simply solve using -snes_mf, I don't see anymore than the usual 3-4 changing digits at the end of the residuals. However, when I use my pcshell, the results are completely different between the two machines. > > I have attached output_SuperComputer.txt and output_DesktopComputer.txt, which correspond to the output from the exact same code and options (and of course same input data file !). More precisely > > output_SuperComputer.txt : output on a supercomputer called Helios, sorry I don't know the exact specs. > In this case, the SNES norms are reduced successively: > 0 SNES Function norm 4.867111712420e-03 > 1 SNES Function norm 5.632325929998e-08 > 2 SNES Function norm 7.427800084502e-15 > > output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 GHz Intel Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop with Mac OS X Mavericks). > In this case, I obtain the following for the SNES norms, > while in the other, I obtain > 0 SNES Function norm 4.867111713544e-03 > 1 SNES Function norm 1.560094052222e-03 > 2 SNES Function norm 1.552118650943e-03 > 3 SNES Function norm 1.552106297094e-03 > 4 SNES Function norm 1.552106277949e-03 > which I can't explain, because otherwise the KSP residual (with the same operator, which I checked) behave well. > > As you can see, the first time the preconditioner is applied (DB_, DP_, Drho_ and PS_ solves), the two outputs coincide (except for the few last digits, up to 9 actually, which is more than I would expect), and everything starts to diverge at the first print of the main KSP (the one stemming from the SNES) residual norms. > > Do you have an idea what may cause such a strange behaviour ? > > Best > > Timothee > > > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > > From dave.mayhem23 at gmail.com Thu Dec 17 11:12:38 2015 From: dave.mayhem23 at gmail.com (Dave May) Date: Thu, 17 Dec 2015 18:12:38 +0100 Subject: [petsc-users] Slow MatAssemblyBegin MAT_FINAL_ASSEMBLY In-Reply-To: References: Message-ID: On Thursday, 17 December 2015, Jose A. Abell M. wrote: > Thank you Dave! > > Do you have a rough idea of how long a matrix like that should take to > assemble? > Not hours. Right? > If the preallocation is correct, and most of the entries to be inserted live locally (and don't need to be scattered to another rank), it should definitely not take hours. > > Regards, > Jose > > -- > > Jos? Abell > *PhD Candidate* > Computational Geomechanics Group > Dept. of Civil and Environmental Engineering > UC Davis > www.joseabell.com > > > On Thu, Dec 17, 2015 at 12:58 AM, Dave May > wrote: > >> >> >> On 17 December 2015 at 08:06, Jose A. Abell M. > > wrote: >> >>> Hello dear PETSc users, >>> >>> This is a problem that pops up often, from what I see, in the mailing >>> list. My program takes a long time assembling the matrix. >>> >>> What I know: >>> >>> >>> - Matrix Size is (MatMPIAIJ) 2670402 >>> - Number of processes running PETSc: 95 >>> - Not going to virtual memory (no swapping, used mem well withing >>> each node's capacity) >>> - System is partitioned with ParMETIS for load balancing >>> - I see memory moving around in each node (total used memory changes >>> a bit, grows and then frees) >>> - Matrix is filled in blocks of size 81x81 (FEM code, so this ends >>> up being a sparse matrix) >>> - I don't do flushes at all. Only MAT_FINAL_ASSEMBLY when all the >>> MatSetValues are done. >>> >>> Should I do MAT_FLUSH_ASSEMBLY even though I have enough memory to >>> store the buffers? If so, how often? Every 100 blocks? >>> >>> What else could it be? >>> >>> Its taking several hours to asseble this matrix. I re-use the sparsity >>> pattern, so subsequent assemblies are fast. Does this mean that my >>> preallocation is wrong? >>> >> >> The preallocation could be wrong. That is the usual cause of very slow >> matrix assembly. To confirm this hypothesis, run your code with the command >> line option -info. You will get an enormous amount of information in >> stdout. You might consider using -info with a smallish problem size / core >> count. >> >> Inspect the output generated by -info and look for lines like this: >> >> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >> >> If the number of mallocs during MatSetValues() is not zero, then your >> preallocation is not exactly correct. A small number of mallocs, say less >> than 10, might be accepted (performance wise). However if the number of >> mallocs is > 100, then assembly time will be terribly slow. >> >> Thanks, >> Dave >> >> >> >> >>> >>> Regards, >>> >>> >>> >>> >>> -- >>> >>> Jos? Abell >>> *PhD Candidate* >>> Computational Geomechanics Group >>> Dept. of Civil and Environmental Engineering >>> UC Davis >>> www.joseabell.com >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Dec 17 12:59:09 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 17 Dec 2015 12:59:09 -0600 Subject: [petsc-users] Slow MatAssemblyBegin MAT_FINAL_ASSEMBLY In-Reply-To: References: Message-ID: > On Dec 17, 2015, at 11:12 AM, Dave May wrote: > > > > On Thursday, 17 December 2015, Jose A. Abell M. wrote: > Thank you Dave! > > Do you have a rough idea of how long a matrix like that should take to assemble? > Not hours. Right? > > If the preallocation is correct, and most of the entries to be inserted live locally (and don't need to be scattered to another rank), it should definitely not take hours. If the preallocation is correct then the time to do the first assembly may take maybe twice as long as subsequent assemblies but it should not be much more than that. Barry > > > > Regards, > Jose > > -- > > Jos? Abell > PhD Candidate > Computational Geomechanics Group > Dept. of Civil and Environmental Engineering > UC Davis > www.joseabell.com > > > On Thu, Dec 17, 2015 at 12:58 AM, Dave May wrote: > > > On 17 December 2015 at 08:06, Jose A. Abell M. wrote: > Hello dear PETSc users, > > This is a problem that pops up often, from what I see, in the mailing list. My program takes a long time assembling the matrix. > > What I know: > > ? Matrix Size is (MatMPIAIJ) 2670402 > ? Number of processes running PETSc: 95 > ? Not going to virtual memory (no swapping, used mem well withing each node's capacity) > ? System is partitioned with ParMETIS for load balancing > ? I see memory moving around in each node (total used memory changes a bit, grows and then frees) > ? Matrix is filled in blocks of size 81x81 (FEM code, so this ends up being a sparse matrix) > ? I don't do flushes at all. Only MAT_FINAL_ASSEMBLY when all the MatSetValues are done. > Should I do MAT_FLUSH_ASSEMBLY even though I have enough memory to store the buffers? If so, how often? Every 100 blocks? > > What else could it be? > > Its taking several hours to asseble this matrix. I re-use the sparsity pattern, so subsequent assemblies are fast. Does this mean that my preallocation is wrong? > > The preallocation could be wrong. That is the usual cause of very slow matrix assembly. To confirm this hypothesis, run your code with the command line option -info. You will get an enormous amount of information in stdout. You might consider using -info with a smallish problem size / core count. > > Inspect the output generated by -info and look for lines like this: > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > If the number of mallocs during MatSetValues() is not zero, then your preallocation is not exactly correct. A small number of mallocs, say less than 10, might be accepted (performance wise). However if the number of mallocs is > 100, then assembly time will be terribly slow. > > Thanks, > Dave > > > > > Regards, > > > > > -- > > Jos? Abell > PhD Candidate > Computational Geomechanics Group > Dept. of Civil and Environmental Engineering > UC Davis > www.joseabell.com > > > From jaabell at ucdavis.edu Thu Dec 17 13:04:50 2015 From: jaabell at ucdavis.edu (Jose A. Abell M.) Date: Thu, 17 Dec 2015 11:04:50 -0800 Subject: [petsc-users] Slow MatAssemblyBegin MAT_FINAL_ASSEMBLY In-Reply-To: References: Message-ID: Thank you all! I will check with a smaller example to see how my allocation is doing. -- Jos? Abell *PhD Candidate* Computational Geomechanics Group Dept. of Civil and Environmental Engineering UC Davis www.joseabell.com On Thu, Dec 17, 2015 at 10:59 AM, Barry Smith wrote: > > > On Dec 17, 2015, at 11:12 AM, Dave May wrote: > > > > > > > > On Thursday, 17 December 2015, Jose A. Abell M. > wrote: > > Thank you Dave! > > > > Do you have a rough idea of how long a matrix like that should take to > assemble? > > Not hours. Right? > > > > If the preallocation is correct, and most of the entries to be inserted > live locally (and don't need to be scattered to another rank), it should > definitely not take hours. > > If the preallocation is correct then the time to do the first assembly > may take maybe twice as long as subsequent assemblies but it should not be > much more than that. > > Barry > > > > > > > > > Regards, > > Jose > > > > -- > > > > Jos? Abell > > PhD Candidate > > Computational Geomechanics Group > > Dept. of Civil and Environmental Engineering > > UC Davis > > www.joseabell.com > > > > > > On Thu, Dec 17, 2015 at 12:58 AM, Dave May > wrote: > > > > > > On 17 December 2015 at 08:06, Jose A. Abell M. > wrote: > > Hello dear PETSc users, > > > > This is a problem that pops up often, from what I see, in the mailing > list. My program takes a long time assembling the matrix. > > > > What I know: > > > > ? Matrix Size is (MatMPIAIJ) 2670402 > > ? Number of processes running PETSc: 95 > > ? Not going to virtual memory (no swapping, used mem well withing > each node's capacity) > > ? System is partitioned with ParMETIS for load balancing > > ? I see memory moving around in each node (total used memory > changes a bit, grows and then frees) > > ? Matrix is filled in blocks of size 81x81 (FEM code, so this ends > up being a sparse matrix) > > ? I don't do flushes at all. Only MAT_FINAL_ASSEMBLY when all the > MatSetValues are done. > > Should I do MAT_FLUSH_ASSEMBLY even though I have enough memory to store > the buffers? If so, how often? Every 100 blocks? > > > > What else could it be? > > > > Its taking several hours to asseble this matrix. I re-use the sparsity > pattern, so subsequent assemblies are fast. Does this mean that my > preallocation is wrong? > > > > The preallocation could be wrong. That is the usual cause of very slow > matrix assembly. To confirm this hypothesis, run your code with the command > line option -info. You will get an enormous amount of information in > stdout. You might consider using -info with a smallish problem size / core > count. > > > > Inspect the output generated by -info and look for lines like this: > > > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > > > If the number of mallocs during MatSetValues() is not zero, then your > preallocation is not exactly correct. A small number of mallocs, say less > than 10, might be accepted (performance wise). However if the number of > mallocs is > 100, then assembly time will be terribly slow. > > > > Thanks, > > Dave > > > > > > > > > > Regards, > > > > > > > > > > -- > > > > Jos? Abell > > PhD Candidate > > Computational Geomechanics Group > > Dept. of Civil and Environmental Engineering > > UC Davis > > www.joseabell.com > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aotero at fi.uba.ar Thu Dec 17 13:21:34 2015 From: aotero at fi.uba.ar (Alejandro D Otero) Date: Thu, 17 Dec 2015 16:21:34 -0300 Subject: [petsc-users] Distributing data along with DMPlex In-Reply-To: References: Message-ID: Thanks, The problem then is that after DMPlexDistribute the DMPlex 'points' are renumbered. So if the values are related to each point in the original numbering how do I set the values after the distribution. I know the property stored in the vector related to the entities with the numbering of the original mesh which I use to create the first DMPlex. Ideally for me, I would like to set the values in the vector before DMPlexDistribute and get the vector components renumbered and redistributed accordingly in a global vector. And then, get the local vector. Hope it could be more clear now. Regards, Alejandro On Wed, Dec 16, 2015 at 7:01 PM, Justin Chang wrote: > I think you would follow this order: > > 1*) create a DMPlex (depth, chart, etc) on rank 0. Other ranks have an > empty DM > > 2) DMPlexDistribute() > > 3*) Create the PetscSection > > 4) DMCreateGlobalVector() > > 5) DMCreateLocalVector() > > Now you have a global vector and a local vector for your distributed > DMPlex. The mapping/ghosting/etc of dofs is already taken care of. > > * if you're using standard Galerkin FE then in SNES examples 12 and 62 > (and maybe others?) the first step is handled through the mesh generation > functions and step 3 is handled through step 4 > > Thanks, > Justin > > On Wednesday, December 16, 2015, Alejandro D Otero > wrote: > >> Hi, I need some help understanding how to distribute data together with a >> dmplex representing a FE mesh. >> At the beginning I define the structure of the dmplex assigning certain >> number of DoF to cells, edges and vertexes, in one process (the dmplex in >> the rest is empty) >> I create a petscsecton and I create an associated global vector with the >> quantities I want to store. >> Then I distribute the dmplex over all the processes. >> * Although this does not perform well it is just a starting point. I know >> it has to be improved. >> >> I would like to have the global vector distributed accordingly so that >> each process has access to the corresponding local part with its DoF >> (possibly adding some ghost values corresponding to the shared DoF not >> taken care by it). >> >> Is there any 'correct' way to do that in PETSc? >> >> Thanks in advance, >> >> Alejandro >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jychang48 at gmail.com Thu Dec 17 13:27:06 2015 From: jychang48 at gmail.com (Justin Chang) Date: Thu, 17 Dec 2015 11:27:06 -0800 Subject: [petsc-users] Distributing data along with DMPlex In-Reply-To: References: Message-ID: So you would use something like DMPlexDistributeField() in that case. You have your original/global DM, PetscSection, and Vec of values. Then using the PetscSF that was created during DMPlexDistribute, you map the global stuff into the local stuff. On Thu, Dec 17, 2015 at 11:21 AM, Alejandro D Otero wrote: > Thanks, > The problem then is that after DMPlexDistribute the DMPlex 'points' are > renumbered. So if the values are related to each point in the original > numbering how do I set the values after the distribution. I know the > property stored in the vector related to the entities with the numbering of > the original mesh which I use to create the first DMPlex. > > Ideally for me, I would like to set the values in the vector before > DMPlexDistribute and get the vector components renumbered and redistributed > accordingly in a global vector. And then, get the local vector. > > Hope it could be more clear now. > Regards, > > Alejandro > > > > On Wed, Dec 16, 2015 at 7:01 PM, Justin Chang wrote: > >> I think you would follow this order: >> >> 1*) create a DMPlex (depth, chart, etc) on rank 0. Other ranks have an >> empty DM >> >> 2) DMPlexDistribute() >> >> 3*) Create the PetscSection >> >> 4) DMCreateGlobalVector() >> >> 5) DMCreateLocalVector() >> >> Now you have a global vector and a local vector for your distributed >> DMPlex. The mapping/ghosting/etc of dofs is already taken care of. >> >> * if you're using standard Galerkin FE then in SNES examples 12 and 62 >> (and maybe others?) the first step is handled through the mesh generation >> functions and step 3 is handled through step 4 >> >> Thanks, >> Justin >> >> On Wednesday, December 16, 2015, Alejandro D Otero >> wrote: >> >>> Hi, I need some help understanding how to distribute data together with >>> a dmplex representing a FE mesh. >>> At the beginning I define the structure of the dmplex assigning certain >>> number of DoF to cells, edges and vertexes, in one process (the dmplex in >>> the rest is empty) >>> I create a petscsecton and I create an associated global vector with the >>> quantities I want to store. >>> Then I distribute the dmplex over all the processes. >>> * Although this does not perform well it is just a starting point. I >>> know it has to be improved. >>> >>> I would like to have the global vector distributed accordingly so that >>> each process has access to the corresponding local part with its DoF >>> (possibly adding some ghost values corresponding to the shared DoF not >>> taken care by it). >>> >>> Is there any 'correct' way to do that in PETSc? >>> >>> Thanks in advance, >>> >>> Alejandro >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothee.nicolas at gmail.com Thu Dec 17 20:16:53 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Fri, 18 Dec 2015 11:16:53 +0900 Subject: [petsc-users] Big discrepancy between machines In-Reply-To: <219A5E6E-ADD7-4647-8487-C30D29F05344@mcs.anl.gov> References: <219A5E6E-ADD7-4647-8487-C30D29F05344@mcs.anl.gov> Message-ID: OK, I can build the residuals with this routine. It allowed me to see that the residuals computed by PETSc (with KSPGetResidualNorm), which do converge rapidly, are totally inconsistent with what I compute from call KSPBuildSolution(ksp,PETSC_NULL_OBJECT,x,ierr) call VecDuplicate(x,y,ierr) call KSPGetRHS(ksp,b,ierr) call KSPGetOperators(ksp,A,PETSC_NULL_OBJECT,ierr) call MatMult(A,x,y,ierr) call VecAXPY(y,-one,b,ierr) call VecNorm(y,NORM_2,norm,ierr) call PrintReal(' KSP Residual norm',norm,itwelve,ierr) call VecDestroy(y,ierr) Now, the residuals computed with this method *are* consistent with the next residual computed by SNES. In other words, I get the following output (see below, the first KSP residual is the internal PETSc one, the second is mine). As you can see, the one I compute is consistent (up to a few digits) with what goes inside the updated solution of SNES. How is that possible ? I tried to see what KSPGetResidualNorm does but I could only find the instruction *rnorm = ksp->rnorm and then I don't know where to look for... Here's what the output looks like 0 SNES Function norm 4.867111713544e-03 0 KSP Residual norm 4.958714442097e-03 KSP Residual norm 4.867111713544E-03 1 KSP Residual norm 3.549907385578e-04 KSP Residual norm 1.651154147130E-03 2 KSP Residual norm 3.522862963778e-05 KSP Residual norm 1.557509645650E-03 3 KSP Residual norm 3.541384239147e-06 KSP Residual norm 1.561088378958E-03 4 KSP Residual norm 3.641326695942e-07 KSP Residual norm 1.560783284631E-03 5 KSP Residual norm 3.850512634373e-08 KSP Residual norm 1.560806669239E-03 1 SNES Function norm 1.560806759605e-03 etc.. etc... On the cluster, I don't find exactly the behavior above, the two norms agree rather well up to a few digits, at least at the beginning, but they start to be very different at the end of the iterations (up to 2 orders of magnitude, which also gets me quite worried). Thx Timothee 2015-12-18 1:45 GMT+09:00 Barry Smith : > > > On Dec 17, 2015, at 7:47 AM, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > > > It works very smoothly for the SNES :-), but KSPGetSolution keeps > returning a zero vector... KSPGetResidualNorm gives me the norm alright, > but I would like to actually see the vectors. Is KSPGetSolution the wrong > routine ? > > If you are using GMRES, which is the default, it actually DOES NOT have > a representation of the solution at each step. Yes that seems odd but it > only computes the solution vector at a restart or when the iteration ends. > This is why KSPGetSolution doesn't provide anything useful. You can use > KSPBuildSolution() to have it construct the "current" solution whenever you > need it. > > Barry > > > > > > > Thx > > > > Timoth?e > > > > 2015-12-17 19:19 GMT+09:00 Dave May : > > > > > > On 17 December 2015 at 11:00, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > Hi, > > > > So, valgrind is OK (at least on the local machine. Actually on the > cluster helios, it produces strange results even for the simplest petsc > program PetscInitialize followed by PetscFinalize, I will try to figure > this out with their technical team), and I have also tried with exactly the > same versions (3.6.0) and it does not change the behavior. > > > > So now I would like to now how to have a grip on what comes in and out > of the SNES and the KSP internal to the SNES. That is, I would like to > inspect manually the vector which enters the SNES in the first place > (should be zero I believe), what is being fed to the KSP, and the vector > which comes out of it, etc. if possible at each iteration of the SNES. I > want to actually see these vectors, and compute there norm by hand. The > trouble is, it is really hard to understand why the newton residuals are > not reduced since the KSP converges so nicely. This does not make any sense > to me, so I want to know what happens to the vectors. But on the SNES list > of routines, I did not find the tools that would allow me to do that (and > messing around with the C code is too hard for me, it would take me weeks). > Does someone have a hint ? > > > > The only sane way to do this is to write a custom monitor for your SNES > object. > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESMonitorSet.html > > > > Inside your monitor, you have access the SNES, and everything it > defines, e.g. the current solution, non-linear residual, KSP etc. See these > pages > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetSolution.html > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetFunction.html > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetKSP.html > > > > Then you can pull apart the residual and compute specific norms (or plot > the residual). > > > > Hopefully you can access everything you need to perform your analysis. > > > > Cheers, > > Dave > > > > > > Thx > > > > Timothee > > > > > > > > > > 2015-12-15 14:20 GMT+09:00 Matthew Knepley : > > On Mon, Dec 14, 2015 at 11:06 PM, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > There is a diference in valgrind indeed between the two. It seems to be > clean on my desktop Mac OS X but not on the cluster. I'll try to see what's > causing this. I still don't understand well what's causing memory leaks in > the case where all PETSc objects are freed correctly (as can pbe checked > with -log_summary). > > > > Also, I have tried running either > > > > valgrind ./my_code -option1 -option2... > > > > or > > > > valgrind mpiexec -n 1 ./my_code -option1 -option2... > > > > Note here you would need --trace-children=yes for valgrind. > > > > Matt > > > > It seems the second is the correct way to proceed right ? This gives > very different behaviour for valgrind. > > > > Timothee > > > > > > > > 2015-12-14 17:38 GMT+09:00 Timoth?e Nicolas >: > > OK, I'll try that, thx > > > > 2015-12-14 17:38 GMT+09:00 Dave May : > > You have the configure line, so it should be relatively straight forward > to configure / build petsc in your home directory. > > > > > > On 14 December 2015 at 09:34, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > OK, The problem is that I don't think I can change this easily as far as > the cluster is concerned. I obtain access to petsc by loading the petsc > module, and even if I have a few choices, I don't see any debug builds... > > > > 2015-12-14 17:26 GMT+09:00 Dave May : > > > > > > On Monday, 14 December 2015, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > Hum, OK. I use FORTRAN by the way. Is your comment still valid ? > > > > No. Fortran compilers init variables to zero. > > In this case, I would run a debug build on your OSX machine through > valgrind and make sure it is clean. > > > > Other obvious thing to check what happens if use exactly the same petsc > builds on both machines. I see 3.6.1 and 3.6.0 are being used. > > > > For all this type of checking, I would definitely use debug builds on > both machines. Your cluster build is using the highest level of > optimization... > > > > > > > > I'll check anyway, but I thought I had been careful about this sort of > things. > > > > Also, I thought the problem on Mac OS X may have been due to the fact I > used the version with debugging on, so I rerun configure with > --with-debugging=no, which did not change anything. > > > > Thx > > > > Timothee > > > > > > 2015-12-14 17:04 GMT+09:00 Dave May : > > One suggestion is you have some uninitialized variables in your pcshell. > Despite your arch being called "debug", your configure options indicate you > have turned debugging off. > > > > C standard doesn't prescribe how uninit variables should be treated - > the behavior is labelled as undefined. As a result, different compilers on > different archs with the same optimization flags can and will treat uninit > variables differently. I find OSX c compilers tend to set them to zero. > > > > I suggest compiling a debug build on both machines and trying your test > again. Also, consider running the debug builds through valgrind. > > > > Thanks, > > Dave > > > > On Monday, 14 December 2015, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > Hi, > > > > I have noticed I have a VERY big difference in behaviour between two > machines in my problem, solved with SNES. I can't explain it, because I > have tested my operators which give the same result. I also checked that > the vectors fed to the SNES are the same. The problem happens only with my > shell preconditioner. When I don't use it, and simply solve using -snes_mf, > I don't see anymore than the usual 3-4 changing digits at the end of the > residuals. However, when I use my pcshell, the results are completely > different between the two machines. > > > > I have attached output_SuperComputer.txt and output_DesktopComputer.txt, > which correspond to the output from the exact same code and options (and of > course same input data file !). More precisely > > > > output_SuperComputer.txt : output on a supercomputer called Helios, > sorry I don't know the exact specs. > > In this case, the SNES norms are reduced successively: > > 0 SNES Function norm 4.867111712420e-03 > > 1 SNES Function norm 5.632325929998e-08 > > 2 SNES Function norm 7.427800084502e-15 > > > > output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 GHz Intel > Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop with Mac > OS X Mavericks). > > In this case, I obtain the following for the SNES norms, > > while in the other, I obtain > > 0 SNES Function norm 4.867111713544e-03 > > 1 SNES Function norm 1.560094052222e-03 > > 2 SNES Function norm 1.552118650943e-03 > > 3 SNES Function norm 1.552106297094e-03 > > 4 SNES Function norm 1.552106277949e-03 > > which I can't explain, because otherwise the KSP residual (with the same > operator, which I checked) behave well. > > > > As you can see, the first time the preconditioner is applied (DB_, DP_, > Drho_ and PS_ solves), the two outputs coincide (except for the few last > digits, up to 9 actually, which is more than I would expect), and > everything starts to diverge at the first print of the main KSP (the one > stemming from the SNES) residual norms. > > > > Do you have an idea what may cause such a strange behaviour ? > > > > Best > > > > Timothee > > > > > > > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Dec 17 20:51:45 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 17 Dec 2015 20:51:45 -0600 Subject: [petsc-users] Big discrepancy between machines In-Reply-To: References: <219A5E6E-ADD7-4647-8487-C30D29F05344@mcs.anl.gov> Message-ID: <08DC562C-1083-42AC-87E9-4C7D0BEDF11B@mcs.anl.gov> There is something inconsistent with the preconditioners or the matrix vector product. Do you have a gmres outside a gmres? That definitely won't work and would produce this problem. If you have a gmres somewhere inside the preconditioner then you need to have a fgmres outside of that (and not a gmres). Run with -ksp_view and send the output showing what your solver is Barry > On Dec 17, 2015, at 8:16 PM, Timoth?e Nicolas wrote: > > OK, I can build the residuals with this routine. It allowed me to see that the residuals computed by PETSc (with KSPGetResidualNorm), which do converge rapidly, are totally inconsistent with what I compute from > > call KSPBuildSolution(ksp,PETSC_NULL_OBJECT,x,ierr) > call VecDuplicate(x,y,ierr) > call KSPGetRHS(ksp,b,ierr) > call KSPGetOperators(ksp,A,PETSC_NULL_OBJECT,ierr) > call MatMult(A,x,y,ierr) > call VecAXPY(y,-one,b,ierr) > call VecNorm(y,NORM_2,norm,ierr) > call PrintReal(' KSP Residual norm',norm,itwelve,ierr) > call VecDestroy(y,ierr) > > Now, the residuals computed with this method are consistent with the next residual computed by SNES. In other words, I get the following output (see below, the first KSP residual is the internal PETSc one, the second is mine). As you can see, the one I compute is consistent (up to a few digits) with what goes inside the updated solution of SNES. How is that possible ? I tried to see what KSPGetResidualNorm does but I could only find the instruction > > *rnorm = ksp->rnorm > > and then I don't know where to look for... > > Here's what the output looks like > > 0 SNES Function norm 4.867111713544e-03 > > 0 KSP Residual norm 4.958714442097e-03 > KSP Residual norm 4.867111713544E-03 > > 1 KSP Residual norm 3.549907385578e-04 > KSP Residual norm 1.651154147130E-03 > > 2 KSP Residual norm 3.522862963778e-05 > KSP Residual norm 1.557509645650E-03 > > 3 KSP Residual norm 3.541384239147e-06 > KSP Residual norm 1.561088378958E-03 > > 4 KSP Residual norm 3.641326695942e-07 > KSP Residual norm 1.560783284631E-03 > > 5 KSP Residual norm 3.850512634373e-08 > KSP Residual norm 1.560806669239E-03 > > 1 SNES Function norm 1.560806759605e-03 > etc.. etc... > > On the cluster, I don't find exactly the behavior above, the two norms agree rather well up to a few digits, at least at the beginning, but they start to be very different at the end of the iterations (up to 2 orders of magnitude, which also gets me quite worried). > > Thx > > Timothee > > > > 2015-12-18 1:45 GMT+09:00 Barry Smith : > > > On Dec 17, 2015, at 7:47 AM, Timoth?e Nicolas wrote: > > > > It works very smoothly for the SNES :-), but KSPGetSolution keeps returning a zero vector... KSPGetResidualNorm gives me the norm alright, but I would like to actually see the vectors. Is KSPGetSolution the wrong routine ? > > If you are using GMRES, which is the default, it actually DOES NOT have a representation of the solution at each step. Yes that seems odd but it only computes the solution vector at a restart or when the iteration ends. This is why KSPGetSolution doesn't provide anything useful. You can use KSPBuildSolution() to have it construct the "current" solution whenever you need it. > > Barry > > > > > > > Thx > > > > Timoth?e > > > > 2015-12-17 19:19 GMT+09:00 Dave May : > > > > > > On 17 December 2015 at 11:00, Timoth?e Nicolas wrote: > > Hi, > > > > So, valgrind is OK (at least on the local machine. Actually on the cluster helios, it produces strange results even for the simplest petsc program PetscInitialize followed by PetscFinalize, I will try to figure this out with their technical team), and I have also tried with exactly the same versions (3.6.0) and it does not change the behavior. > > > > So now I would like to now how to have a grip on what comes in and out of the SNES and the KSP internal to the SNES. That is, I would like to inspect manually the vector which enters the SNES in the first place (should be zero I believe), what is being fed to the KSP, and the vector which comes out of it, etc. if possible at each iteration of the SNES. I want to actually see these vectors, and compute there norm by hand. The trouble is, it is really hard to understand why the newton residuals are not reduced since the KSP converges so nicely. This does not make any sense to me, so I want to know what happens to the vectors. But on the SNES list of routines, I did not find the tools that would allow me to do that (and messing around with the C code is too hard for me, it would take me weeks). Does someone have a hint ? > > > > The only sane way to do this is to write a custom monitor for your SNES object. > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESMonitorSet.html > > > > Inside your monitor, you have access the SNES, and everything it defines, e.g. the current solution, non-linear residual, KSP etc. See these pages > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetSolution.html > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetFunction.html > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetKSP.html > > > > Then you can pull apart the residual and compute specific norms (or plot the residual). > > > > Hopefully you can access everything you need to perform your analysis. > > > > Cheers, > > Dave > > > > > > Thx > > > > Timothee > > > > > > > > > > 2015-12-15 14:20 GMT+09:00 Matthew Knepley : > > On Mon, Dec 14, 2015 at 11:06 PM, Timoth?e Nicolas wrote: > > There is a diference in valgrind indeed between the two. It seems to be clean on my desktop Mac OS X but not on the cluster. I'll try to see what's causing this. I still don't understand well what's causing memory leaks in the case where all PETSc objects are freed correctly (as can pbe checked with -log_summary). > > > > Also, I have tried running either > > > > valgrind ./my_code -option1 -option2... > > > > or > > > > valgrind mpiexec -n 1 ./my_code -option1 -option2... > > > > Note here you would need --trace-children=yes for valgrind. > > > > Matt > > > > It seems the second is the correct way to proceed right ? This gives very different behaviour for valgrind. > > > > Timothee > > > > > > > > 2015-12-14 17:38 GMT+09:00 Timoth?e Nicolas : > > OK, I'll try that, thx > > > > 2015-12-14 17:38 GMT+09:00 Dave May : > > You have the configure line, so it should be relatively straight forward to configure / build petsc in your home directory. > > > > > > On 14 December 2015 at 09:34, Timoth?e Nicolas wrote: > > OK, The problem is that I don't think I can change this easily as far as the cluster is concerned. I obtain access to petsc by loading the petsc module, and even if I have a few choices, I don't see any debug builds... > > > > 2015-12-14 17:26 GMT+09:00 Dave May : > > > > > > On Monday, 14 December 2015, Timoth?e Nicolas wrote: > > Hum, OK. I use FORTRAN by the way. Is your comment still valid ? > > > > No. Fortran compilers init variables to zero. > > In this case, I would run a debug build on your OSX machine through valgrind and make sure it is clean. > > > > Other obvious thing to check what happens if use exactly the same petsc builds on both machines. I see 3.6.1 and 3.6.0 are being used. > > > > For all this type of checking, I would definitely use debug builds on both machines. Your cluster build is using the highest level of optimization... > > > > > > > > I'll check anyway, but I thought I had been careful about this sort of things. > > > > Also, I thought the problem on Mac OS X may have been due to the fact I used the version with debugging on, so I rerun configure with --with-debugging=no, which did not change anything. > > > > Thx > > > > Timothee > > > > > > 2015-12-14 17:04 GMT+09:00 Dave May : > > One suggestion is you have some uninitialized variables in your pcshell. Despite your arch being called "debug", your configure options indicate you have turned debugging off. > > > > C standard doesn't prescribe how uninit variables should be treated - the behavior is labelled as undefined. As a result, different compilers on different archs with the same optimization flags can and will treat uninit variables differently. I find OSX c compilers tend to set them to zero. > > > > I suggest compiling a debug build on both machines and trying your test again. Also, consider running the debug builds through valgrind. > > > > Thanks, > > Dave > > > > On Monday, 14 December 2015, Timoth?e Nicolas wrote: > > Hi, > > > > I have noticed I have a VERY big difference in behaviour between two machines in my problem, solved with SNES. I can't explain it, because I have tested my operators which give the same result. I also checked that the vectors fed to the SNES are the same. The problem happens only with my shell preconditioner. When I don't use it, and simply solve using -snes_mf, I don't see anymore than the usual 3-4 changing digits at the end of the residuals. However, when I use my pcshell, the results are completely different between the two machines. > > > > I have attached output_SuperComputer.txt and output_DesktopComputer.txt, which correspond to the output from the exact same code and options (and of course same input data file !). More precisely > > > > output_SuperComputer.txt : output on a supercomputer called Helios, sorry I don't know the exact specs. > > In this case, the SNES norms are reduced successively: > > 0 SNES Function norm 4.867111712420e-03 > > 1 SNES Function norm 5.632325929998e-08 > > 2 SNES Function norm 7.427800084502e-15 > > > > output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 GHz Intel Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop with Mac OS X Mavericks). > > In this case, I obtain the following for the SNES norms, > > while in the other, I obtain > > 0 SNES Function norm 4.867111713544e-03 > > 1 SNES Function norm 1.560094052222e-03 > > 2 SNES Function norm 1.552118650943e-03 > > 3 SNES Function norm 1.552106297094e-03 > > 4 SNES Function norm 1.552106277949e-03 > > which I can't explain, because otherwise the KSP residual (with the same operator, which I checked) behave well. > > > > As you can see, the first time the preconditioner is applied (DB_, DP_, Drho_ and PS_ solves), the two outputs coincide (except for the few last digits, up to 9 actually, which is more than I would expect), and everything starts to diverge at the first print of the main KSP (the one stemming from the SNES) residual norms. > > > > Do you have an idea what may cause such a strange behaviour ? > > > > Best > > > > Timothee > > > > > > > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > > > > > From timothee.nicolas at gmail.com Thu Dec 17 21:41:27 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Fri, 18 Dec 2015 12:41:27 +0900 Subject: [petsc-users] Big discrepancy between machines In-Reply-To: <08DC562C-1083-42AC-87E9-4C7D0BEDF11B@mcs.anl.gov> References: <219A5E6E-ADD7-4647-8487-C30D29F05344@mcs.anl.gov> <08DC562C-1083-42AC-87E9-4C7D0BEDF11B@mcs.anl.gov> Message-ID: Ah ! That was the reason ! I do have actually four GMRES inside the outside main ksp GMRES. I do a lot of shenanigans in my preconditioning. Namely, I take the input vector x, and extract the different degrees of freedom, putting them inside 4 vectors (2 vectors with 3 dof, and 2 vectors of scalars). Then I run a GMRES on each of these vectors (preconditioned with multigrid for at least one of them). When this is all done I combine all the vectors again (there are some matrix multiplication involved, with the different vectors coupled) to put them in a vector with same format (8 dof) as the initial input vector, and the preconditioning is done. So, simply adding -ksp_type fgmres seems to fix my problem, I just checked. You have just saved me several days/weeks of scratching my head, if not more ! I had no clue standard GMRES was picky about preconditioners. So now, after telling you what I do in the preconditioner, you confirm indeed that simply using FGMRES is OK right ? Thx ! Best Timothee 2015-12-18 11:51 GMT+09:00 Barry Smith : > > There is something inconsistent with the preconditioners or the matrix > vector product. Do you have a gmres outside a gmres? That definitely won't > work and would produce this problem. If you have a gmres somewhere inside > the preconditioner then you need to have a fgmres outside of that (and not > a gmres). Run with -ksp_view and send the output showing what your solver is > > Barry > > > On Dec 17, 2015, at 8:16 PM, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > > > OK, I can build the residuals with this routine. It allowed me to see > that the residuals computed by PETSc (with KSPGetResidualNorm), which do > converge rapidly, are totally inconsistent with what I compute from > > > > call KSPBuildSolution(ksp,PETSC_NULL_OBJECT,x,ierr) > > call VecDuplicate(x,y,ierr) > > call KSPGetRHS(ksp,b,ierr) > > call KSPGetOperators(ksp,A,PETSC_NULL_OBJECT,ierr) > > call MatMult(A,x,y,ierr) > > call VecAXPY(y,-one,b,ierr) > > call VecNorm(y,NORM_2,norm,ierr) > > call PrintReal(' KSP Residual norm',norm,itwelve,ierr) > > call VecDestroy(y,ierr) > > > > Now, the residuals computed with this method are consistent with the > next residual computed by SNES. In other words, I get the following output > (see below, the first KSP residual is the internal PETSc one, the second is > mine). As you can see, the one I compute is consistent (up to a few digits) > with what goes inside the updated solution of SNES. How is that possible ? > I tried to see what KSPGetResidualNorm does but I could only find the > instruction > > > > *rnorm = ksp->rnorm > > > > and then I don't know where to look for... > > > > Here's what the output looks like > > > > 0 SNES Function norm 4.867111713544e-03 > > > > 0 KSP Residual norm 4.958714442097e-03 > > KSP Residual norm 4.867111713544E-03 > > > > 1 KSP Residual norm 3.549907385578e-04 > > KSP Residual norm 1.651154147130E-03 > > > > 2 KSP Residual norm 3.522862963778e-05 > > KSP Residual norm 1.557509645650E-03 > > > > 3 KSP Residual norm 3.541384239147e-06 > > KSP Residual norm 1.561088378958E-03 > > > > 4 KSP Residual norm 3.641326695942e-07 > > KSP Residual norm 1.560783284631E-03 > > > > 5 KSP Residual norm 3.850512634373e-08 > > KSP Residual norm 1.560806669239E-03 > > > > 1 SNES Function norm 1.560806759605e-03 > > etc.. etc... > > > > On the cluster, I don't find exactly the behavior above, the two norms > agree rather well up to a few digits, at least at the beginning, but they > start to be very different at the end of the iterations (up to 2 orders of > magnitude, which also gets me quite worried). > > > > Thx > > > > Timothee > > > > > > > > 2015-12-18 1:45 GMT+09:00 Barry Smith : > > > > > On Dec 17, 2015, at 7:47 AM, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > > > > > It works very smoothly for the SNES :-), but KSPGetSolution keeps > returning a zero vector... KSPGetResidualNorm gives me the norm alright, > but I would like to actually see the vectors. Is KSPGetSolution the wrong > routine ? > > > > If you are using GMRES, which is the default, it actually DOES NOT > have a representation of the solution at each step. Yes that seems odd but > it only computes the solution vector at a restart or when the iteration > ends. This is why KSPGetSolution doesn't provide anything useful. You can > use KSPBuildSolution() to have it construct the "current" solution whenever > you need it. > > > > Barry > > > > > > > > > > > > Thx > > > > > > Timoth?e > > > > > > 2015-12-17 19:19 GMT+09:00 Dave May : > > > > > > > > > On 17 December 2015 at 11:00, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > > Hi, > > > > > > So, valgrind is OK (at least on the local machine. Actually on the > cluster helios, it produces strange results even for the simplest petsc > program PetscInitialize followed by PetscFinalize, I will try to figure > this out with their technical team), and I have also tried with exactly the > same versions (3.6.0) and it does not change the behavior. > > > > > > So now I would like to now how to have a grip on what comes in and out > of the SNES and the KSP internal to the SNES. That is, I would like to > inspect manually the vector which enters the SNES in the first place > (should be zero I believe), what is being fed to the KSP, and the vector > which comes out of it, etc. if possible at each iteration of the SNES. I > want to actually see these vectors, and compute there norm by hand. The > trouble is, it is really hard to understand why the newton residuals are > not reduced since the KSP converges so nicely. This does not make any sense > to me, so I want to know what happens to the vectors. But on the SNES list > of routines, I did not find the tools that would allow me to do that (and > messing around with the C code is too hard for me, it would take me weeks). > Does someone have a hint ? > > > > > > The only sane way to do this is to write a custom monitor for your > SNES object. > > > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESMonitorSet.html > > > > > > Inside your monitor, you have access the SNES, and everything it > defines, e.g. the current solution, non-linear residual, KSP etc. See these > pages > > > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetSolution.html > > > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetFunction.html > > > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetKSP.html > > > > > > Then you can pull apart the residual and compute specific norms (or > plot the residual). > > > > > > Hopefully you can access everything you need to perform your analysis. > > > > > > Cheers, > > > Dave > > > > > > > > > Thx > > > > > > Timothee > > > > > > > > > > > > > > > 2015-12-15 14:20 GMT+09:00 Matthew Knepley : > > > On Mon, Dec 14, 2015 at 11:06 PM, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > > There is a diference in valgrind indeed between the two. It seems to > be clean on my desktop Mac OS X but not on the cluster. I'll try to see > what's causing this. I still don't understand well what's causing memory > leaks in the case where all PETSc objects are freed correctly (as can pbe > checked with -log_summary). > > > > > > Also, I have tried running either > > > > > > valgrind ./my_code -option1 -option2... > > > > > > or > > > > > > valgrind mpiexec -n 1 ./my_code -option1 -option2... > > > > > > Note here you would need --trace-children=yes for valgrind. > > > > > > Matt > > > > > > It seems the second is the correct way to proceed right ? This gives > very different behaviour for valgrind. > > > > > > Timothee > > > > > > > > > > > > 2015-12-14 17:38 GMT+09:00 Timoth?e Nicolas < > timothee.nicolas at gmail.com>: > > > OK, I'll try that, thx > > > > > > 2015-12-14 17:38 GMT+09:00 Dave May : > > > You have the configure line, so it should be relatively straight > forward to configure / build petsc in your home directory. > > > > > > > > > On 14 December 2015 at 09:34, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > > OK, The problem is that I don't think I can change this easily as far > as the cluster is concerned. I obtain access to petsc by loading the petsc > module, and even if I have a few choices, I don't see any debug builds... > > > > > > 2015-12-14 17:26 GMT+09:00 Dave May : > > > > > > > > > On Monday, 14 December 2015, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > > Hum, OK. I use FORTRAN by the way. Is your comment still valid ? > > > > > > No. Fortran compilers init variables to zero. > > > In this case, I would run a debug build on your OSX machine through > valgrind and make sure it is clean. > > > > > > Other obvious thing to check what happens if use exactly the same > petsc builds on both machines. I see 3.6.1 and 3.6.0 are being used. > > > > > > For all this type of checking, I would definitely use debug builds on > both machines. Your cluster build is using the highest level of > optimization... > > > > > > > > > > > > I'll check anyway, but I thought I had been careful about this sort of > things. > > > > > > Also, I thought the problem on Mac OS X may have been due to the fact > I used the version with debugging on, so I rerun configure with > --with-debugging=no, which did not change anything. > > > > > > Thx > > > > > > Timothee > > > > > > > > > 2015-12-14 17:04 GMT+09:00 Dave May : > > > One suggestion is you have some uninitialized variables in your > pcshell. Despite your arch being called "debug", your configure options > indicate you have turned debugging off. > > > > > > C standard doesn't prescribe how uninit variables should be treated - > the behavior is labelled as undefined. As a result, different compilers on > different archs with the same optimization flags can and will treat uninit > variables differently. I find OSX c compilers tend to set them to zero. > > > > > > I suggest compiling a debug build on both machines and trying your > test again. Also, consider running the debug builds through valgrind. > > > > > > Thanks, > > > Dave > > > > > > On Monday, 14 December 2015, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > > Hi, > > > > > > I have noticed I have a VERY big difference in behaviour between two > machines in my problem, solved with SNES. I can't explain it, because I > have tested my operators which give the same result. I also checked that > the vectors fed to the SNES are the same. The problem happens only with my > shell preconditioner. When I don't use it, and simply solve using -snes_mf, > I don't see anymore than the usual 3-4 changing digits at the end of the > residuals. However, when I use my pcshell, the results are completely > different between the two machines. > > > > > > I have attached output_SuperComputer.txt and > output_DesktopComputer.txt, which correspond to the output from the exact > same code and options (and of course same input data file !). More precisely > > > > > > output_SuperComputer.txt : output on a supercomputer called Helios, > sorry I don't know the exact specs. > > > In this case, the SNES norms are reduced successively: > > > 0 SNES Function norm 4.867111712420e-03 > > > 1 SNES Function norm 5.632325929998e-08 > > > 2 SNES Function norm 7.427800084502e-15 > > > > > > output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 GHz > Intel Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop with > Mac OS X Mavericks). > > > In this case, I obtain the following for the SNES norms, > > > while in the other, I obtain > > > 0 SNES Function norm 4.867111713544e-03 > > > 1 SNES Function norm 1.560094052222e-03 > > > 2 SNES Function norm 1.552118650943e-03 > > > 3 SNES Function norm 1.552106297094e-03 > > > 4 SNES Function norm 1.552106277949e-03 > > > which I can't explain, because otherwise the KSP residual (with the > same operator, which I checked) behave well. > > > > > > As you can see, the first time the preconditioner is applied (DB_, > DP_, Drho_ and PS_ solves), the two outputs coincide (except for the few > last digits, up to 9 actually, which is more than I would expect), and > everything starts to diverge at the first print of the main KSP (the one > stemming from the SNES) residual norms. > > > > > > Do you have an idea what may cause such a strange behaviour ? > > > > > > Best > > > > > > Timothee > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > > -- Norbert Wiener > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Dec 17 21:53:22 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 17 Dec 2015 21:53:22 -0600 Subject: [petsc-users] Big discrepancy between machines In-Reply-To: References: <219A5E6E-ADD7-4647-8487-C30D29F05344@mcs.anl.gov> <08DC562C-1083-42AC-87E9-4C7D0BEDF11B@mcs.anl.gov> Message-ID: Except for the "flexible" Krylov methods like FGMRES, you cannot put a Krylov method inside a Krylov method because (non-flexible) Krylov methods require the preconditioner be a linear operator and Krylov methods are not linear operators. > On Dec 17, 2015, at 9:41 PM, Timoth?e Nicolas wrote: > > Ah ! That was the reason ! > > I do have actually four GMRES inside the outside main ksp GMRES. I do a lot of shenanigans in my preconditioning. Namely, I take the input vector x, and extract the different degrees of freedom, putting them inside 4 vectors (2 vectors with 3 dof, and 2 vectors of scalars). Then I run a GMRES on each of these vectors (preconditioned with multigrid for at least one of them). When this is all done I combine all the vectors again (there are some matrix multiplication involved, with the different vectors coupled) to put them in a vector with same format (8 dof) as the initial input vector, and the preconditioning is done. > > So, simply adding -ksp_type fgmres seems to fix my problem, I just checked. You have just saved me several days/weeks of scratching my head, if not more ! > > I had no clue standard GMRES was picky about preconditioners. So now, after telling you what I do in the preconditioner, you confirm indeed that simply using FGMRES is OK right ? > > Thx ! > > Best > > Timothee > > > > 2015-12-18 11:51 GMT+09:00 Barry Smith : > > There is something inconsistent with the preconditioners or the matrix vector product. Do you have a gmres outside a gmres? That definitely won't work and would produce this problem. If you have a gmres somewhere inside the preconditioner then you need to have a fgmres outside of that (and not a gmres). Run with -ksp_view and send the output showing what your solver is > > Barry > > > On Dec 17, 2015, at 8:16 PM, Timoth?e Nicolas wrote: > > > > OK, I can build the residuals with this routine. It allowed me to see that the residuals computed by PETSc (with KSPGetResidualNorm), which do converge rapidly, are totally inconsistent with what I compute from > > > > call KSPBuildSolution(ksp,PETSC_NULL_OBJECT,x,ierr) > > call VecDuplicate(x,y,ierr) > > call KSPGetRHS(ksp,b,ierr) > > call KSPGetOperators(ksp,A,PETSC_NULL_OBJECT,ierr) > > call MatMult(A,x,y,ierr) > > call VecAXPY(y,-one,b,ierr) > > call VecNorm(y,NORM_2,norm,ierr) > > call PrintReal(' KSP Residual norm',norm,itwelve,ierr) > > call VecDestroy(y,ierr) > > > > Now, the residuals computed with this method are consistent with the next residual computed by SNES. In other words, I get the following output (see below, the first KSP residual is the internal PETSc one, the second is mine). As you can see, the one I compute is consistent (up to a few digits) with what goes inside the updated solution of SNES. How is that possible ? I tried to see what KSPGetResidualNorm does but I could only find the instruction > > > > *rnorm = ksp->rnorm > > > > and then I don't know where to look for... > > > > Here's what the output looks like > > > > 0 SNES Function norm 4.867111713544e-03 > > > > 0 KSP Residual norm 4.958714442097e-03 > > KSP Residual norm 4.867111713544E-03 > > > > 1 KSP Residual norm 3.549907385578e-04 > > KSP Residual norm 1.651154147130E-03 > > > > 2 KSP Residual norm 3.522862963778e-05 > > KSP Residual norm 1.557509645650E-03 > > > > 3 KSP Residual norm 3.541384239147e-06 > > KSP Residual norm 1.561088378958E-03 > > > > 4 KSP Residual norm 3.641326695942e-07 > > KSP Residual norm 1.560783284631E-03 > > > > 5 KSP Residual norm 3.850512634373e-08 > > KSP Residual norm 1.560806669239E-03 > > > > 1 SNES Function norm 1.560806759605e-03 > > etc.. etc... > > > > On the cluster, I don't find exactly the behavior above, the two norms agree rather well up to a few digits, at least at the beginning, but they start to be very different at the end of the iterations (up to 2 orders of magnitude, which also gets me quite worried). > > > > Thx > > > > Timothee > > > > > > > > 2015-12-18 1:45 GMT+09:00 Barry Smith : > > > > > On Dec 17, 2015, at 7:47 AM, Timoth?e Nicolas wrote: > > > > > > It works very smoothly for the SNES :-), but KSPGetSolution keeps returning a zero vector... KSPGetResidualNorm gives me the norm alright, but I would like to actually see the vectors. Is KSPGetSolution the wrong routine ? > > > > If you are using GMRES, which is the default, it actually DOES NOT have a representation of the solution at each step. Yes that seems odd but it only computes the solution vector at a restart or when the iteration ends. This is why KSPGetSolution doesn't provide anything useful. You can use KSPBuildSolution() to have it construct the "current" solution whenever you need it. > > > > Barry > > > > > > > > > > > > Thx > > > > > > Timoth?e > > > > > > 2015-12-17 19:19 GMT+09:00 Dave May : > > > > > > > > > On 17 December 2015 at 11:00, Timoth?e Nicolas wrote: > > > Hi, > > > > > > So, valgrind is OK (at least on the local machine. Actually on the cluster helios, it produces strange results even for the simplest petsc program PetscInitialize followed by PetscFinalize, I will try to figure this out with their technical team), and I have also tried with exactly the same versions (3.6.0) and it does not change the behavior. > > > > > > So now I would like to now how to have a grip on what comes in and out of the SNES and the KSP internal to the SNES. That is, I would like to inspect manually the vector which enters the SNES in the first place (should be zero I believe), what is being fed to the KSP, and the vector which comes out of it, etc. if possible at each iteration of the SNES. I want to actually see these vectors, and compute there norm by hand. The trouble is, it is really hard to understand why the newton residuals are not reduced since the KSP converges so nicely. This does not make any sense to me, so I want to know what happens to the vectors. But on the SNES list of routines, I did not find the tools that would allow me to do that (and messing around with the C code is too hard for me, it would take me weeks). Does someone have a hint ? > > > > > > The only sane way to do this is to write a custom monitor for your SNES object. > > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESMonitorSet.html > > > > > > Inside your monitor, you have access the SNES, and everything it defines, e.g. the current solution, non-linear residual, KSP etc. See these pages > > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetSolution.html > > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetFunction.html > > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetKSP.html > > > > > > Then you can pull apart the residual and compute specific norms (or plot the residual). > > > > > > Hopefully you can access everything you need to perform your analysis. > > > > > > Cheers, > > > Dave > > > > > > > > > Thx > > > > > > Timothee > > > > > > > > > > > > > > > 2015-12-15 14:20 GMT+09:00 Matthew Knepley : > > > On Mon, Dec 14, 2015 at 11:06 PM, Timoth?e Nicolas wrote: > > > There is a diference in valgrind indeed between the two. It seems to be clean on my desktop Mac OS X but not on the cluster. I'll try to see what's causing this. I still don't understand well what's causing memory leaks in the case where all PETSc objects are freed correctly (as can pbe checked with -log_summary). > > > > > > Also, I have tried running either > > > > > > valgrind ./my_code -option1 -option2... > > > > > > or > > > > > > valgrind mpiexec -n 1 ./my_code -option1 -option2... > > > > > > Note here you would need --trace-children=yes for valgrind. > > > > > > Matt > > > > > > It seems the second is the correct way to proceed right ? This gives very different behaviour for valgrind. > > > > > > Timothee > > > > > > > > > > > > 2015-12-14 17:38 GMT+09:00 Timoth?e Nicolas : > > > OK, I'll try that, thx > > > > > > 2015-12-14 17:38 GMT+09:00 Dave May : > > > You have the configure line, so it should be relatively straight forward to configure / build petsc in your home directory. > > > > > > > > > On 14 December 2015 at 09:34, Timoth?e Nicolas wrote: > > > OK, The problem is that I don't think I can change this easily as far as the cluster is concerned. I obtain access to petsc by loading the petsc module, and even if I have a few choices, I don't see any debug builds... > > > > > > 2015-12-14 17:26 GMT+09:00 Dave May : > > > > > > > > > On Monday, 14 December 2015, Timoth?e Nicolas wrote: > > > Hum, OK. I use FORTRAN by the way. Is your comment still valid ? > > > > > > No. Fortran compilers init variables to zero. > > > In this case, I would run a debug build on your OSX machine through valgrind and make sure it is clean. > > > > > > Other obvious thing to check what happens if use exactly the same petsc builds on both machines. I see 3.6.1 and 3.6.0 are being used. > > > > > > For all this type of checking, I would definitely use debug builds on both machines. Your cluster build is using the highest level of optimization... > > > > > > > > > > > > I'll check anyway, but I thought I had been careful about this sort of things. > > > > > > Also, I thought the problem on Mac OS X may have been due to the fact I used the version with debugging on, so I rerun configure with --with-debugging=no, which did not change anything. > > > > > > Thx > > > > > > Timothee > > > > > > > > > 2015-12-14 17:04 GMT+09:00 Dave May : > > > One suggestion is you have some uninitialized variables in your pcshell. Despite your arch being called "debug", your configure options indicate you have turned debugging off. > > > > > > C standard doesn't prescribe how uninit variables should be treated - the behavior is labelled as undefined. As a result, different compilers on different archs with the same optimization flags can and will treat uninit variables differently. I find OSX c compilers tend to set them to zero. > > > > > > I suggest compiling a debug build on both machines and trying your test again. Also, consider running the debug builds through valgrind. > > > > > > Thanks, > > > Dave > > > > > > On Monday, 14 December 2015, Timoth?e Nicolas wrote: > > > Hi, > > > > > > I have noticed I have a VERY big difference in behaviour between two machines in my problem, solved with SNES. I can't explain it, because I have tested my operators which give the same result. I also checked that the vectors fed to the SNES are the same. The problem happens only with my shell preconditioner. When I don't use it, and simply solve using -snes_mf, I don't see anymore than the usual 3-4 changing digits at the end of the residuals. However, when I use my pcshell, the results are completely different between the two machines. > > > > > > I have attached output_SuperComputer.txt and output_DesktopComputer.txt, which correspond to the output from the exact same code and options (and of course same input data file !). More precisely > > > > > > output_SuperComputer.txt : output on a supercomputer called Helios, sorry I don't know the exact specs. > > > In this case, the SNES norms are reduced successively: > > > 0 SNES Function norm 4.867111712420e-03 > > > 1 SNES Function norm 5.632325929998e-08 > > > 2 SNES Function norm 7.427800084502e-15 > > > > > > output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 GHz Intel Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop with Mac OS X Mavericks). > > > In this case, I obtain the following for the SNES norms, > > > while in the other, I obtain > > > 0 SNES Function norm 4.867111713544e-03 > > > 1 SNES Function norm 1.560094052222e-03 > > > 2 SNES Function norm 1.552118650943e-03 > > > 3 SNES Function norm 1.552106297094e-03 > > > 4 SNES Function norm 1.552106277949e-03 > > > which I can't explain, because otherwise the KSP residual (with the same operator, which I checked) behave well. > > > > > > As you can see, the first time the preconditioner is applied (DB_, DP_, Drho_ and PS_ solves), the two outputs coincide (except for the few last digits, up to 9 actually, which is more than I would expect), and everything starts to diverge at the first print of the main KSP (the one stemming from the SNES) residual norms. > > > > > > Do you have an idea what may cause such a strange behaviour ? > > > > > > Best > > > > > > Timothee > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > -- Norbert Wiener > > > > > > > > > > > > > > > From timothee.nicolas at gmail.com Thu Dec 17 22:04:40 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Fri, 18 Dec 2015 13:04:40 +0900 Subject: [petsc-users] Big discrepancy between machines In-Reply-To: References: <219A5E6E-ADD7-4647-8487-C30D29F05344@mcs.anl.gov> <08DC562C-1083-42AC-87E9-4C7D0BEDF11B@mcs.anl.gov> Message-ID: Aw, really ? I thought that the Krylov method found the approximation by a linear combination of {b,Ab,A^2b,...} where b is the right hand side. Where is it not linear ? 2015-12-18 12:53 GMT+09:00 Barry Smith : > > Except for the "flexible" Krylov methods like FGMRES, you cannot put a > Krylov method inside a Krylov method because (non-flexible) Krylov methods > require the preconditioner be a linear operator and Krylov methods are not > linear operators. > > > > On Dec 17, 2015, at 9:41 PM, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > > > Ah ! That was the reason ! > > > > I do have actually four GMRES inside the outside main ksp GMRES. I do a > lot of shenanigans in my preconditioning. Namely, I take the input vector > x, and extract the different degrees of freedom, putting them inside 4 > vectors (2 vectors with 3 dof, and 2 vectors of scalars). Then I run a > GMRES on each of these vectors (preconditioned with multigrid for at least > one of them). When this is all done I combine all the vectors again (there > are some matrix multiplication involved, with the different vectors > coupled) to put them in a vector with same format (8 dof) as the initial > input vector, and the preconditioning is done. > > > > So, simply adding -ksp_type fgmres seems to fix my problem, I just > checked. You have just saved me several days/weeks of scratching my head, > if not more ! > > > > I had no clue standard GMRES was picky about preconditioners. So now, > after telling you what I do in the preconditioner, you confirm indeed that > simply using FGMRES is OK right ? > > > > Thx ! > > > > Best > > > > Timothee > > > > > > > > 2015-12-18 11:51 GMT+09:00 Barry Smith : > > > > There is something inconsistent with the preconditioners or the > matrix vector product. Do you have a gmres outside a gmres? That definitely > won't work and would produce this problem. If you have a gmres somewhere > inside the preconditioner then you need to have a fgmres outside of that > (and not a gmres). Run with -ksp_view and send the output showing what your > solver is > > > > Barry > > > > > On Dec 17, 2015, at 8:16 PM, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > > > > > OK, I can build the residuals with this routine. It allowed me to see > that the residuals computed by PETSc (with KSPGetResidualNorm), which do > converge rapidly, are totally inconsistent with what I compute from > > > > > > call KSPBuildSolution(ksp,PETSC_NULL_OBJECT,x,ierr) > > > call VecDuplicate(x,y,ierr) > > > call KSPGetRHS(ksp,b,ierr) > > > call KSPGetOperators(ksp,A,PETSC_NULL_OBJECT,ierr) > > > call MatMult(A,x,y,ierr) > > > call VecAXPY(y,-one,b,ierr) > > > call VecNorm(y,NORM_2,norm,ierr) > > > call PrintReal(' KSP Residual norm',norm,itwelve,ierr) > > > call VecDestroy(y,ierr) > > > > > > Now, the residuals computed with this method are consistent with the > next residual computed by SNES. In other words, I get the following output > (see below, the first KSP residual is the internal PETSc one, the second is > mine). As you can see, the one I compute is consistent (up to a few digits) > with what goes inside the updated solution of SNES. How is that possible ? > I tried to see what KSPGetResidualNorm does but I could only find the > instruction > > > > > > *rnorm = ksp->rnorm > > > > > > and then I don't know where to look for... > > > > > > Here's what the output looks like > > > > > > 0 SNES Function norm 4.867111713544e-03 > > > > > > 0 KSP Residual norm 4.958714442097e-03 > > > KSP Residual norm 4.867111713544E-03 > > > > > > 1 KSP Residual norm 3.549907385578e-04 > > > KSP Residual norm 1.651154147130E-03 > > > > > > 2 KSP Residual norm 3.522862963778e-05 > > > KSP Residual norm 1.557509645650E-03 > > > > > > 3 KSP Residual norm 3.541384239147e-06 > > > KSP Residual norm 1.561088378958E-03 > > > > > > 4 KSP Residual norm 3.641326695942e-07 > > > KSP Residual norm 1.560783284631E-03 > > > > > > 5 KSP Residual norm 3.850512634373e-08 > > > KSP Residual norm 1.560806669239E-03 > > > > > > 1 SNES Function norm 1.560806759605e-03 > > > etc.. etc... > > > > > > On the cluster, I don't find exactly the behavior above, the two norms > agree rather well up to a few digits, at least at the beginning, but they > start to be very different at the end of the iterations (up to 2 orders of > magnitude, which also gets me quite worried). > > > > > > Thx > > > > > > Timothee > > > > > > > > > > > > 2015-12-18 1:45 GMT+09:00 Barry Smith : > > > > > > > On Dec 17, 2015, at 7:47 AM, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > > > > > > > It works very smoothly for the SNES :-), but KSPGetSolution keeps > returning a zero vector... KSPGetResidualNorm gives me the norm alright, > but I would like to actually see the vectors. Is KSPGetSolution the wrong > routine ? > > > > > > If you are using GMRES, which is the default, it actually DOES NOT > have a representation of the solution at each step. Yes that seems odd but > it only computes the solution vector at a restart or when the iteration > ends. This is why KSPGetSolution doesn't provide anything useful. You can > use KSPBuildSolution() to have it construct the "current" solution whenever > you need it. > > > > > > Barry > > > > > > > > > > > > > > > > > Thx > > > > > > > > Timoth?e > > > > > > > > 2015-12-17 19:19 GMT+09:00 Dave May : > > > > > > > > > > > > On 17 December 2015 at 11:00, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > > > Hi, > > > > > > > > So, valgrind is OK (at least on the local machine. Actually on the > cluster helios, it produces strange results even for the simplest petsc > program PetscInitialize followed by PetscFinalize, I will try to figure > this out with their technical team), and I have also tried with exactly the > same versions (3.6.0) and it does not change the behavior. > > > > > > > > So now I would like to now how to have a grip on what comes in and > out of the SNES and the KSP internal to the SNES. That is, I would like to > inspect manually the vector which enters the SNES in the first place > (should be zero I believe), what is being fed to the KSP, and the vector > which comes out of it, etc. if possible at each iteration of the SNES. I > want to actually see these vectors, and compute there norm by hand. The > trouble is, it is really hard to understand why the newton residuals are > not reduced since the KSP converges so nicely. This does not make any sense > to me, so I want to know what happens to the vectors. But on the SNES list > of routines, I did not find the tools that would allow me to do that (and > messing around with the C code is too hard for me, it would take me weeks). > Does someone have a hint ? > > > > > > > > The only sane way to do this is to write a custom monitor for your > SNES object. > > > > > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESMonitorSet.html > > > > > > > > Inside your monitor, you have access the SNES, and everything it > defines, e.g. the current solution, non-linear residual, KSP etc. See these > pages > > > > > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetSolution.html > > > > > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetFunction.html > > > > > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetKSP.html > > > > > > > > Then you can pull apart the residual and compute specific norms (or > plot the residual). > > > > > > > > Hopefully you can access everything you need to perform your > analysis. > > > > > > > > Cheers, > > > > Dave > > > > > > > > > > > > Thx > > > > > > > > Timothee > > > > > > > > > > > > > > > > > > > > 2015-12-15 14:20 GMT+09:00 Matthew Knepley : > > > > On Mon, Dec 14, 2015 at 11:06 PM, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > > > There is a diference in valgrind indeed between the two. It seems to > be clean on my desktop Mac OS X but not on the cluster. I'll try to see > what's causing this. I still don't understand well what's causing memory > leaks in the case where all PETSc objects are freed correctly (as can pbe > checked with -log_summary). > > > > > > > > Also, I have tried running either > > > > > > > > valgrind ./my_code -option1 -option2... > > > > > > > > or > > > > > > > > valgrind mpiexec -n 1 ./my_code -option1 -option2... > > > > > > > > Note here you would need --trace-children=yes for valgrind. > > > > > > > > Matt > > > > > > > > It seems the second is the correct way to proceed right ? This gives > very different behaviour for valgrind. > > > > > > > > Timothee > > > > > > > > > > > > > > > > 2015-12-14 17:38 GMT+09:00 Timoth?e Nicolas < > timothee.nicolas at gmail.com>: > > > > OK, I'll try that, thx > > > > > > > > 2015-12-14 17:38 GMT+09:00 Dave May : > > > > You have the configure line, so it should be relatively straight > forward to configure / build petsc in your home directory. > > > > > > > > > > > > On 14 December 2015 at 09:34, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > > > OK, The problem is that I don't think I can change this easily as > far as the cluster is concerned. I obtain access to petsc by loading the > petsc module, and even if I have a few choices, I don't see any debug > builds... > > > > > > > > 2015-12-14 17:26 GMT+09:00 Dave May : > > > > > > > > > > > > On Monday, 14 December 2015, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > > > Hum, OK. I use FORTRAN by the way. Is your comment still valid ? > > > > > > > > No. Fortran compilers init variables to zero. > > > > In this case, I would run a debug build on your OSX machine through > valgrind and make sure it is clean. > > > > > > > > Other obvious thing to check what happens if use exactly the same > petsc builds on both machines. I see 3.6.1 and 3.6.0 are being used. > > > > > > > > For all this type of checking, I would definitely use debug builds > on both machines. Your cluster build is using the highest level of > optimization... > > > > > > > > > > > > > > > > I'll check anyway, but I thought I had been careful about this sort > of things. > > > > > > > > Also, I thought the problem on Mac OS X may have been due to the > fact I used the version with debugging on, so I rerun configure with > --with-debugging=no, which did not change anything. > > > > > > > > Thx > > > > > > > > Timothee > > > > > > > > > > > > 2015-12-14 17:04 GMT+09:00 Dave May : > > > > One suggestion is you have some uninitialized variables in your > pcshell. Despite your arch being called "debug", your configure options > indicate you have turned debugging off. > > > > > > > > C standard doesn't prescribe how uninit variables should be treated > - the behavior is labelled as undefined. As a result, different compilers > on different archs with the same optimization flags can and will treat > uninit variables differently. I find OSX c compilers tend to set them to > zero. > > > > > > > > I suggest compiling a debug build on both machines and trying your > test again. Also, consider running the debug builds through valgrind. > > > > > > > > Thanks, > > > > Dave > > > > > > > > On Monday, 14 December 2015, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > > > Hi, > > > > > > > > I have noticed I have a VERY big difference in behaviour between two > machines in my problem, solved with SNES. I can't explain it, because I > have tested my operators which give the same result. I also checked that > the vectors fed to the SNES are the same. The problem happens only with my > shell preconditioner. When I don't use it, and simply solve using -snes_mf, > I don't see anymore than the usual 3-4 changing digits at the end of the > residuals. However, when I use my pcshell, the results are completely > different between the two machines. > > > > > > > > I have attached output_SuperComputer.txt and > output_DesktopComputer.txt, which correspond to the output from the exact > same code and options (and of course same input data file !). More precisely > > > > > > > > output_SuperComputer.txt : output on a supercomputer called Helios, > sorry I don't know the exact specs. > > > > In this case, the SNES norms are reduced successively: > > > > 0 SNES Function norm 4.867111712420e-03 > > > > 1 SNES Function norm 5.632325929998e-08 > > > > 2 SNES Function norm 7.427800084502e-15 > > > > > > > > output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 GHz > Intel Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop with > Mac OS X Mavericks). > > > > In this case, I obtain the following for the SNES norms, > > > > while in the other, I obtain > > > > 0 SNES Function norm 4.867111713544e-03 > > > > 1 SNES Function norm 1.560094052222e-03 > > > > 2 SNES Function norm 1.552118650943e-03 > > > > 3 SNES Function norm 1.552106297094e-03 > > > > 4 SNES Function norm 1.552106277949e-03 > > > > which I can't explain, because otherwise the KSP residual (with the > same operator, which I checked) behave well. > > > > > > > > As you can see, the first time the preconditioner is applied (DB_, > DP_, Drho_ and PS_ solves), the two outputs coincide (except for the few > last digits, up to 9 actually, which is more than I would expect), and > everything starts to diverge at the first print of the main KSP (the one > stemming from the SNES) residual norms. > > > > > > > > Do you have an idea what may cause such a strange behaviour ? > > > > > > > > Best > > > > > > > > Timothee > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > > > -- Norbert Wiener > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Dec 17 22:27:02 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 17 Dec 2015 22:27:02 -0600 Subject: [petsc-users] Big discrepancy between machines In-Reply-To: References: <219A5E6E-ADD7-4647-8487-C30D29F05344@mcs.anl.gov> <08DC562C-1083-42AC-87E9-4C7D0BEDF11B@mcs.anl.gov> Message-ID: > On Dec 17, 2015, at 10:04 PM, Timoth?e Nicolas wrote: > > Aw, really ? I thought that the Krylov method found the approximation by a linear combination of {b,Ab,A^2b,...} where b is the right hand side. It is > Where is it not linear ? GMRES finds min_alpha || b - A (\sum_i=0,p alpha_i A^ib) ||_2 which is the same thing as saying minimize the 2 norm of the residual with x in the Krylov space. For simplicity consider just 2 terms {b, Ab}. Plug these 2 terms and and then take the derivative of norm with respect to ALPHA to get the minimum solution, this gives you a little two by two system to solve for the alpha. Note that the solution is NOT linear in the b vector (it has some dot products with the b vectors and Ab vectors. Hence the minimization in general is not linear in b so the solution found by a Krylov method is not linear in the input b. An equivalent way of looking at this is to look at all the Krylov methods codes (except for Chebychev which is a linear operator) they all depend on inner products on vectors depending on the input vector b; inner products of vectors from this one input vector are not linear in that vector. Barry Note this is one reason why our default smoother KSP method for GAMG is Chebyshev. > > 2015-12-18 12:53 GMT+09:00 Barry Smith : > > Except for the "flexible" Krylov methods like FGMRES, you cannot put a Krylov method inside a Krylov method because (non-flexible) Krylov methods require the preconditioner be a linear operator and Krylov methods are not linear operators. > > > > On Dec 17, 2015, at 9:41 PM, Timoth?e Nicolas wrote: > > > > Ah ! That was the reason ! > > > > I do have actually four GMRES inside the outside main ksp GMRES. I do a lot of shenanigans in my preconditioning. Namely, I take the input vector x, and extract the different degrees of freedom, putting them inside 4 vectors (2 vectors with 3 dof, and 2 vectors of scalars). Then I run a GMRES on each of these vectors (preconditioned with multigrid for at least one of them). When this is all done I combine all the vectors again (there are some matrix multiplication involved, with the different vectors coupled) to put them in a vector with same format (8 dof) as the initial input vector, and the preconditioning is done. > > > > So, simply adding -ksp_type fgmres seems to fix my problem, I just checked. You have just saved me several days/weeks of scratching my head, if not more ! > > > > I had no clue standard GMRES was picky about preconditioners. So now, after telling you what I do in the preconditioner, you confirm indeed that simply using FGMRES is OK right ? > > > > Thx ! > > > > Best > > > > Timothee > > > > > > > > 2015-12-18 11:51 GMT+09:00 Barry Smith : > > > > There is something inconsistent with the preconditioners or the matrix vector product. Do you have a gmres outside a gmres? That definitely won't work and would produce this problem. If you have a gmres somewhere inside the preconditioner then you need to have a fgmres outside of that (and not a gmres). Run with -ksp_view and send the output showing what your solver is > > > > Barry > > > > > On Dec 17, 2015, at 8:16 PM, Timoth?e Nicolas wrote: > > > > > > OK, I can build the residuals with this routine. It allowed me to see that the residuals computed by PETSc (with KSPGetResidualNorm), which do converge rapidly, are totally inconsistent with what I compute from > > > > > > call KSPBuildSolution(ksp,PETSC_NULL_OBJECT,x,ierr) > > > call VecDuplicate(x,y,ierr) > > > call KSPGetRHS(ksp,b,ierr) > > > call KSPGetOperators(ksp,A,PETSC_NULL_OBJECT,ierr) > > > call MatMult(A,x,y,ierr) > > > call VecAXPY(y,-one,b,ierr) > > > call VecNorm(y,NORM_2,norm,ierr) > > > call PrintReal(' KSP Residual norm',norm,itwelve,ierr) > > > call VecDestroy(y,ierr) > > > > > > Now, the residuals computed with this method are consistent with the next residual computed by SNES. In other words, I get the following output (see below, the first KSP residual is the internal PETSc one, the second is mine). As you can see, the one I compute is consistent (up to a few digits) with what goes inside the updated solution of SNES. How is that possible ? I tried to see what KSPGetResidualNorm does but I could only find the instruction > > > > > > *rnorm = ksp->rnorm > > > > > > and then I don't know where to look for... > > > > > > Here's what the output looks like > > > > > > 0 SNES Function norm 4.867111713544e-03 > > > > > > 0 KSP Residual norm 4.958714442097e-03 > > > KSP Residual norm 4.867111713544E-03 > > > > > > 1 KSP Residual norm 3.549907385578e-04 > > > KSP Residual norm 1.651154147130E-03 > > > > > > 2 KSP Residual norm 3.522862963778e-05 > > > KSP Residual norm 1.557509645650E-03 > > > > > > 3 KSP Residual norm 3.541384239147e-06 > > > KSP Residual norm 1.561088378958E-03 > > > > > > 4 KSP Residual norm 3.641326695942e-07 > > > KSP Residual norm 1.560783284631E-03 > > > > > > 5 KSP Residual norm 3.850512634373e-08 > > > KSP Residual norm 1.560806669239E-03 > > > > > > 1 SNES Function norm 1.560806759605e-03 > > > etc.. etc... > > > > > > On the cluster, I don't find exactly the behavior above, the two norms agree rather well up to a few digits, at least at the beginning, but they start to be very different at the end of the iterations (up to 2 orders of magnitude, which also gets me quite worried). > > > > > > Thx > > > > > > Timothee > > > > > > > > > > > > 2015-12-18 1:45 GMT+09:00 Barry Smith : > > > > > > > On Dec 17, 2015, at 7:47 AM, Timoth?e Nicolas wrote: > > > > > > > > It works very smoothly for the SNES :-), but KSPGetSolution keeps returning a zero vector... KSPGetResidualNorm gives me the norm alright, but I would like to actually see the vectors. Is KSPGetSolution the wrong routine ? > > > > > > If you are using GMRES, which is the default, it actually DOES NOT have a representation of the solution at each step. Yes that seems odd but it only computes the solution vector at a restart or when the iteration ends. This is why KSPGetSolution doesn't provide anything useful. You can use KSPBuildSolution() to have it construct the "current" solution whenever you need it. > > > > > > Barry > > > > > > > > > > > > > > > > > Thx > > > > > > > > Timoth?e > > > > > > > > 2015-12-17 19:19 GMT+09:00 Dave May : > > > > > > > > > > > > On 17 December 2015 at 11:00, Timoth?e Nicolas wrote: > > > > Hi, > > > > > > > > So, valgrind is OK (at least on the local machine. Actually on the cluster helios, it produces strange results even for the simplest petsc program PetscInitialize followed by PetscFinalize, I will try to figure this out with their technical team), and I have also tried with exactly the same versions (3.6.0) and it does not change the behavior. > > > > > > > > So now I would like to now how to have a grip on what comes in and out of the SNES and the KSP internal to the SNES. That is, I would like to inspect manually the vector which enters the SNES in the first place (should be zero I believe), what is being fed to the KSP, and the vector which comes out of it, etc. if possible at each iteration of the SNES. I want to actually see these vectors, and compute there norm by hand. The trouble is, it is really hard to understand why the newton residuals are not reduced since the KSP converges so nicely. This does not make any sense to me, so I want to know what happens to the vectors. But on the SNES list of routines, I did not find the tools that would allow me to do that (and messing around with the C code is too hard for me, it would take me weeks). Does someone have a hint ? > > > > > > > > The only sane way to do this is to write a custom monitor for your SNES object. > > > > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESMonitorSet.html > > > > > > > > Inside your monitor, you have access the SNES, and everything it defines, e.g. the current solution, non-linear residual, KSP etc. See these pages > > > > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetSolution.html > > > > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetFunction.html > > > > > > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESGetKSP.html > > > > > > > > Then you can pull apart the residual and compute specific norms (or plot the residual). > > > > > > > > Hopefully you can access everything you need to perform your analysis. > > > > > > > > Cheers, > > > > Dave > > > > > > > > > > > > Thx > > > > > > > > Timothee > > > > > > > > > > > > > > > > > > > > 2015-12-15 14:20 GMT+09:00 Matthew Knepley : > > > > On Mon, Dec 14, 2015 at 11:06 PM, Timoth?e Nicolas wrote: > > > > There is a diference in valgrind indeed between the two. It seems to be clean on my desktop Mac OS X but not on the cluster. I'll try to see what's causing this. I still don't understand well what's causing memory leaks in the case where all PETSc objects are freed correctly (as can pbe checked with -log_summary). > > > > > > > > Also, I have tried running either > > > > > > > > valgrind ./my_code -option1 -option2... > > > > > > > > or > > > > > > > > valgrind mpiexec -n 1 ./my_code -option1 -option2... > > > > > > > > Note here you would need --trace-children=yes for valgrind. > > > > > > > > Matt > > > > > > > > It seems the second is the correct way to proceed right ? This gives very different behaviour for valgrind. > > > > > > > > Timothee > > > > > > > > > > > > > > > > 2015-12-14 17:38 GMT+09:00 Timoth?e Nicolas : > > > > OK, I'll try that, thx > > > > > > > > 2015-12-14 17:38 GMT+09:00 Dave May : > > > > You have the configure line, so it should be relatively straight forward to configure / build petsc in your home directory. > > > > > > > > > > > > On 14 December 2015 at 09:34, Timoth?e Nicolas wrote: > > > > OK, The problem is that I don't think I can change this easily as far as the cluster is concerned. I obtain access to petsc by loading the petsc module, and even if I have a few choices, I don't see any debug builds... > > > > > > > > 2015-12-14 17:26 GMT+09:00 Dave May : > > > > > > > > > > > > On Monday, 14 December 2015, Timoth?e Nicolas wrote: > > > > Hum, OK. I use FORTRAN by the way. Is your comment still valid ? > > > > > > > > No. Fortran compilers init variables to zero. > > > > In this case, I would run a debug build on your OSX machine through valgrind and make sure it is clean. > > > > > > > > Other obvious thing to check what happens if use exactly the same petsc builds on both machines. I see 3.6.1 and 3.6.0 are being used. > > > > > > > > For all this type of checking, I would definitely use debug builds on both machines. Your cluster build is using the highest level of optimization... > > > > > > > > > > > > > > > > I'll check anyway, but I thought I had been careful about this sort of things. > > > > > > > > Also, I thought the problem on Mac OS X may have been due to the fact I used the version with debugging on, so I rerun configure with --with-debugging=no, which did not change anything. > > > > > > > > Thx > > > > > > > > Timothee > > > > > > > > > > > > 2015-12-14 17:04 GMT+09:00 Dave May : > > > > One suggestion is you have some uninitialized variables in your pcshell. Despite your arch being called "debug", your configure options indicate you have turned debugging off. > > > > > > > > C standard doesn't prescribe how uninit variables should be treated - the behavior is labelled as undefined. As a result, different compilers on different archs with the same optimization flags can and will treat uninit variables differently. I find OSX c compilers tend to set them to zero. > > > > > > > > I suggest compiling a debug build on both machines and trying your test again. Also, consider running the debug builds through valgrind. > > > > > > > > Thanks, > > > > Dave > > > > > > > > On Monday, 14 December 2015, Timoth?e Nicolas wrote: > > > > Hi, > > > > > > > > I have noticed I have a VERY big difference in behaviour between two machines in my problem, solved with SNES. I can't explain it, because I have tested my operators which give the same result. I also checked that the vectors fed to the SNES are the same. The problem happens only with my shell preconditioner. When I don't use it, and simply solve using -snes_mf, I don't see anymore than the usual 3-4 changing digits at the end of the residuals. However, when I use my pcshell, the results are completely different between the two machines. > > > > > > > > I have attached output_SuperComputer.txt and output_DesktopComputer.txt, which correspond to the output from the exact same code and options (and of course same input data file !). More precisely > > > > > > > > output_SuperComputer.txt : output on a supercomputer called Helios, sorry I don't know the exact specs. > > > > In this case, the SNES norms are reduced successively: > > > > 0 SNES Function norm 4.867111712420e-03 > > > > 1 SNES Function norm 5.632325929998e-08 > > > > 2 SNES Function norm 7.427800084502e-15 > > > > > > > > output_DesktopComputer.txt : output on a Mac OS X Yosemite 3.4 GHz Intel Core i5 16GB 1600 MHz DDr3. (the same happens on an other laptop with Mac OS X Mavericks). > > > > In this case, I obtain the following for the SNES norms, > > > > while in the other, I obtain > > > > 0 SNES Function norm 4.867111713544e-03 > > > > 1 SNES Function norm 1.560094052222e-03 > > > > 2 SNES Function norm 1.552118650943e-03 > > > > 3 SNES Function norm 1.552106297094e-03 > > > > 4 SNES Function norm 1.552106277949e-03 > > > > which I can't explain, because otherwise the KSP residual (with the same operator, which I checked) behave well. > > > > > > > > As you can see, the first time the preconditioner is applied (DB_, DP_, Drho_ and PS_ solves), the two outputs coincide (except for the few last digits, up to 9 actually, which is more than I would expect), and everything starts to diverge at the first print of the main KSP (the one stemming from the SNES) residual norms. > > > > > > > > Do you have an idea what may cause such a strange behaviour ? > > > > > > > > Best > > > > > > > > Timothee > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > > -- Norbert Wiener > > > > > > > > > > > > > > > > > > > > > > > > From venkateshgk.j at gmail.com Fri Dec 18 01:46:31 2015 From: venkateshgk.j at gmail.com (venkatesh g) Date: Fri, 18 Dec 2015 13:16:31 +0530 Subject: [petsc-users] Eigenvalue problem in batch Message-ID: Hi all, I am using Petsc and Slepc to solve my eigenvalue problem. I have 1000 A and B matrices saved as binary files. So I use in my script : *aprun -n 1 -N 1 ./ex7 -f1 A -f2 B -st_type sinvert -eps_target 0.01 * This is for one A and one B. So, how to do it for each A and B in multiple cores ? Pls. let me know. Thank you Venkatesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothee.nicolas at gmail.com Fri Dec 18 01:58:09 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Fri, 18 Dec 2015 16:58:09 +0900 Subject: [petsc-users] Eigenvalue problem in batch In-Reply-To: References: Message-ID: Hi, Can you be a bit more specific ? You have 1000 different files for A and B ? Like, you have A_0001.bin, A_0002.bin, ... A_1000.bin and B_0001.bin, B_0002.bin, ... B_1000.bin ? In that case it's a bash problem and not a petsc problem, you just need to make a loop on an integer i, use something like printf -v i2 "%04d" ${i} so that i2 is written as 0001, 0002, ... 1000, and not as 1, 2, ... , 1000, concatenate i2 with 'A_' or 'B_' and put that in your launch string. I expect it should work, you can put variables as petsc arguments. But what do you mean with multiple cores ? You want to do it in parallel at the same time ? Then I don't know how to do it. Best Timothee 2015-12-18 16:46 GMT+09:00 venkatesh g : > Hi all, > I am using Petsc and Slepc to solve my eigenvalue problem. > > I have 1000 A and B matrices saved as binary files. So I use in my script : > > *aprun -n 1 -N 1 ./ex7 -f1 A -f2 B -st_type sinvert -eps_target 0.01 * > > This is for one A and one B. So, how to do it for each A and B in multiple > cores ? > > Pls. let me know. > > Thank you > Venkatesh > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From venkateshgk.j at gmail.com Fri Dec 18 02:40:42 2015 From: venkateshgk.j at gmail.com (venkatesh g) Date: Fri, 18 Dec 2015 14:10:42 +0530 Subject: [petsc-users] Eigenvalue problem in batch In-Reply-To: References: Message-ID: Hi Timothee, Yes I have A_0001.bin ... A_1000.bin and B_0001.bin... B1000.bin. Yes is it possible to submit at same time ? Venkatesh On Fri, Dec 18, 2015 at 1:28 PM, Timoth?e Nicolas < timothee.nicolas at gmail.com> wrote: > Hi, > > Can you be a bit more specific ? You have 1000 different files for A and B > ? Like, you have A_0001.bin, A_0002.bin, ... A_1000.bin and B_0001.bin, > B_0002.bin, ... B_1000.bin ? > > In that case it's a bash problem and not a petsc problem, you just need to > make a loop on an integer i, use something like > > printf -v i2 "%04d" ${i} > > so that i2 is written as 0001, 0002, ... 1000, and not as 1, 2, ... , > 1000, concatenate i2 with 'A_' or 'B_' and put that in your launch string. > I expect it should work, you can put variables as petsc arguments. > > But what do you mean with multiple cores ? You want to do it in parallel > at the same time ? Then I don't know how to do it. > > Best > > Timothee > > > > 2015-12-18 16:46 GMT+09:00 venkatesh g : > >> Hi all, >> I am using Petsc and Slepc to solve my eigenvalue problem. >> >> I have 1000 A and B matrices saved as binary files. So I use in my script >> : >> >> *aprun -n 1 -N 1 ./ex7 -f1 A -f2 B -st_type sinvert -eps_target 0.01 * >> >> This is for one A and one B. So, how to do it for each A and B in >> multiple cores ? >> >> Pls. let me know. >> >> Thank you >> Venkatesh >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothee.nicolas at gmail.com Fri Dec 18 03:02:55 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Fri, 18 Dec 2015 18:02:55 +0900 Subject: [petsc-users] Eigenvalue problem in batch In-Reply-To: References: Message-ID: You need to be more specific, what kind of machine do you have ? If you have very few cores, it will not help much to launch them all at the same time, but at least you can make the thing automatic using the loop as I told you. Your computer will solve the first problem and then automatically go to the second etc etc. How much time does it take do perform your task on one set of matrices A and B ? If you are on a cluster with many many cores, then you can probably launch your task in parallel, but I don't know how to do this exactly. But if it's a cluster, you can ask the administrators. Best Timothee 2015-12-18 17:40 GMT+09:00 venkatesh g : > Hi Timothee, > > Yes I have A_0001.bin ... A_1000.bin and B_0001.bin... B1000.bin. Yes is > it possible to submit at same time ? > > Venkatesh > > On Fri, Dec 18, 2015 at 1:28 PM, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > >> Hi, >> >> Can you be a bit more specific ? You have 1000 different files for A and >> B ? Like, you have A_0001.bin, A_0002.bin, ... A_1000.bin and B_0001.bin, >> B_0002.bin, ... B_1000.bin ? >> >> In that case it's a bash problem and not a petsc problem, you just need >> to make a loop on an integer i, use something like >> >> printf -v i2 "%04d" ${i} >> >> so that i2 is written as 0001, 0002, ... 1000, and not as 1, 2, ... , >> 1000, concatenate i2 with 'A_' or 'B_' and put that in your launch string. >> I expect it should work, you can put variables as petsc arguments. >> >> But what do you mean with multiple cores ? You want to do it in parallel >> at the same time ? Then I don't know how to do it. >> >> Best >> >> Timothee >> >> >> >> 2015-12-18 16:46 GMT+09:00 venkatesh g : >> >>> Hi all, >>> I am using Petsc and Slepc to solve my eigenvalue problem. >>> >>> I have 1000 A and B matrices saved as binary files. So I use in my >>> script : >>> >>> *aprun -n 1 -N 1 ./ex7 -f1 A -f2 B -st_type sinvert -eps_target 0.01 * >>> >>> This is for one A and one B. So, how to do it for each A and B in >>> multiple cores ? >>> >>> Pls. let me know. >>> >>> Thank you >>> Venkatesh >>> >>> >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From venkateshgk.j at gmail.com Fri Dec 18 03:07:37 2015 From: venkateshgk.j at gmail.com (venkatesh g) Date: Fri, 18 Dec 2015 14:37:37 +0530 Subject: [petsc-users] Eigenvalue problem in batch In-Reply-To: References: Message-ID: Hi, I have access to Cray with many cores. It takes about 100 secs to solve in a single core. If I require to solve it sequentially, it will take a lot of time. Ok I can ask the admins, but in case you have an idea, pls. let me know. Best Regards, Venkatesh On Fri, Dec 18, 2015 at 2:32 PM, Timoth?e Nicolas < timothee.nicolas at gmail.com> wrote: > You need to be more specific, what kind of machine do you have ? If you > have very few cores, it will not help much to launch them all at the same > time, but at least you can make the thing automatic using the loop as I > told you. Your computer will solve the first problem and then automatically > go to the second etc etc. How much time does it take do perform your task > on one set of matrices A and B ? > > If you are on a cluster with many many cores, then you can probably launch > your task in parallel, but I don't know how to do this exactly. But if it's > a cluster, you can ask the administrators. > > Best > > Timothee > > > > 2015-12-18 17:40 GMT+09:00 venkatesh g : > >> Hi Timothee, >> >> Yes I have A_0001.bin ... A_1000.bin and B_0001.bin... B1000.bin. Yes is >> it possible to submit at same time ? >> >> Venkatesh >> >> On Fri, Dec 18, 2015 at 1:28 PM, Timoth?e Nicolas < >> timothee.nicolas at gmail.com> wrote: >> >>> Hi, >>> >>> Can you be a bit more specific ? You have 1000 different files for A and >>> B ? Like, you have A_0001.bin, A_0002.bin, ... A_1000.bin and B_0001.bin, >>> B_0002.bin, ... B_1000.bin ? >>> >>> In that case it's a bash problem and not a petsc problem, you just need >>> to make a loop on an integer i, use something like >>> >>> printf -v i2 "%04d" ${i} >>> >>> so that i2 is written as 0001, 0002, ... 1000, and not as 1, 2, ... , >>> 1000, concatenate i2 with 'A_' or 'B_' and put that in your launch string. >>> I expect it should work, you can put variables as petsc arguments. >>> >>> But what do you mean with multiple cores ? You want to do it in parallel >>> at the same time ? Then I don't know how to do it. >>> >>> Best >>> >>> Timothee >>> >>> >>> >>> 2015-12-18 16:46 GMT+09:00 venkatesh g : >>> >>>> Hi all, >>>> I am using Petsc and Slepc to solve my eigenvalue problem. >>>> >>>> I have 1000 A and B matrices saved as binary files. So I use in my >>>> script : >>>> >>>> *aprun -n 1 -N 1 ./ex7 -f1 A -f2 B -st_type sinvert -eps_target 0.01 * >>>> >>>> This is for one A and one B. So, how to do it for each A and B in >>>> multiple cores ? >>>> >>>> Pls. let me know. >>>> >>>> Thank you >>>> Venkatesh >>>> >>>> >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zocca.marco at gmail.com Fri Dec 18 04:31:34 2015 From: zocca.marco at gmail.com (Marco Zocca) Date: Fri, 18 Dec 2015 11:31:34 +0100 Subject: [petsc-users] Eigenvalue problem in batch petsc-users Digest, Vol 84, Issue 59 Message-ID: Hi Venkatesh, it sounds like a job for the batch scheduler; again, a "bash" problem, with some additional variables: On a RHEL cluster, I was using Torque/PBS, which gives the `qsub`, `qstat` commands. I would guess that Cray has analogous ones . PETSc and SLEPc are instead toolkits for splitting the individual solutions in bands or blocks that can be solved in parallel. These are two distinct levels of parallelization for the total computational task. Best regards, Marco > > Hi all, > I am using Petsc and Slepc to solve my eigenvalue problem. > > I have 1000 A and B matrices saved as binary files. So I use in my script : > > *aprun -n 1 -N 1 ./ex7 -f1 A -f2 B -st_type sinvert -eps_target 0.01 * > > This is for one A and one B. So, how to do it for each A and B in multiple > cores ? > > Pls. let me know. > > Thank you > Venkatesh > -------------- next part -------------- > > > Hi, > I have access to Cray with many cores. > It takes about 100 secs to solve in a single core. If I require to solve it > sequentially, it will take a lot of time. > Ok I can ask the admins, but in case you have an idea, pls. let me know. > > Best Regards, > Venkatesh > > On Fri, Dec 18, 2015 at 2:32 PM, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > > You need to be more specific, what kind of machine do you have ? If you > > have very few cores, it will not help much to launch them all at the same > > time, but at least you can make the thing automatic using the loop as I > > told you. Your computer will solve the first problem and then > automatically > > go to the second etc etc. How much time does it take do perform your task > > on one set of matrices A and B ? > > > > If you are on a cluster with many many cores, then you can probably > launch > > your task in parallel, but I don't know how to do this exactly. But if > it's > > a cluster, you can ask the administrators. > > > > Best > > > > Timothee > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jychang48 at gmail.com Fri Dec 18 10:21:04 2015 From: jychang48 at gmail.com (Justin Chang) Date: Fri, 18 Dec 2015 08:21:04 -0800 Subject: [petsc-users] Status on parallel mesh reader in DMPlex Message-ID: Hi all, What's the status on the implementation of the parallel mesh reader/generator for DMPlex meshes? Is anyone actively working on this? If so is there a branch that I can peek into? Thanks, Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Fri Dec 18 12:20:20 2015 From: fdkong.jd at gmail.com (Fande Kong) Date: Fri, 18 Dec 2015 11:20:20 -0700 Subject: [petsc-users] Status on parallel mesh reader in DMPlex Message-ID: > Message: 2 >Date: Fri, 18 Dec 2015 08:21:04 -0800 >From: Justin Chang >To: petsc-users >Subject: [petsc-users] Status on parallel mesh reader in DMPlex >Message-ID: > > Content-Type: text/plain; charset="utf-8" > Hi all, >What's the status on the implementation of the parallel I am actually developing a parallel loader. The loader has two steps: (1) Use 1 core or several cores to read a sequential mesh, and partition it into $np$ parts using a partitioner, $np$ is the number of cores you want to use to do a simulation. Usually, $np$ is large, for example, 10,000 cores. And then we write the partitioned data into the file system as a HDF5 file. (2) Load the partitioned data (HDF5 file) into $np$ cores in parallel. This idea works pretty well for me at this point. But, this loader is not compatible with DMPlex now. However, I could change the code somehow, If this idea is acceptable. You could also implement it by yourself. It is not too bad. >mesh reader/generator for DMPlex meshes? Is anyone actively working on > this? If so is there a branch that I can peek into? Parallel generator is highly nontrivial, I think. It is a very hard topic. > Thanks, > Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From wadud.miah at gmail.com Fri Dec 18 12:49:47 2015 From: wadud.miah at gmail.com (W. Miah) Date: Fri, 18 Dec 2015 18:49:47 +0000 Subject: [petsc-users] Intel MKL Sparse blas Message-ID: Hi, Do you have any plans for a PETSc port that uses the optimised Intel MKL sparse BLAS: https://software.intel.com/en-us/node/520797 I think this will be an interesting optimisation exercise. Best regards, -- web: http://miahw.wordpress.com mobile: +447905 755604 gpg: 4CC2 1A75 BDFB 2E29 B22F -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Dec 18 13:33:49 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 18 Dec 2015 13:33:49 -0600 Subject: [petsc-users] Intel MKL Sparse blas In-Reply-To: References: Message-ID: <9B858EE3-73A2-4C94-9CCF-0A5443D97232@mcs.anl.gov> We're always happy to accept contributions. Intel unfortunately made some terrible choices for APIs for these things so it is unlikely without a specific funding source that we would ourselves add code to PETSc to utilize them. Barry > On Dec 18, 2015, at 12:49 PM, W. Miah wrote: > > Hi, > > Do you have any plans for a PETSc port that uses the optimised Intel MKL sparse BLAS: > > https://software.intel.com/en-us/node/520797 > > I think this will be an interesting optimisation exercise. > > Best regards, > > -- > web: http://miahw.wordpress.com > mobile: +447905 755604 > gpg: 4CC2 1A75 BDFB 2E29 B22F From richardtmills at gmail.com Fri Dec 18 13:50:30 2015 From: richardtmills at gmail.com (Richard Mills) Date: Fri, 18 Dec 2015 11:50:30 -0800 Subject: [petsc-users] Intel MKL Sparse blas In-Reply-To: References: Message-ID: Hi Wadud, I work at Intel and plan to add support for the inspector-executor sparse BLAS routines (https://software.intel.com/en-us/node/590105) soon. (I think this should be straightforward but it's been a question of simply finding the time.) I will add a subclass of MATAIJ that uses these. Do you ask because you have anything in particular that you are hoping to get out of this? --Richard On Fri, Dec 18, 2015 at 10:49 AM, W. Miah wrote: > Hi, > > Do you have any plans for a PETSc port that uses the optimised Intel MKL > sparse BLAS: > > https://software.intel.com/en-us/node/520797 > > I think this will be an interesting optimisation exercise. > > Best regards, > > -- > web: http://miahw.wordpress.com > mobile: +447905 755604 > gpg: 4CC2 1A75 BDFB 2E29 B22F > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wadud.miah at gmail.com Fri Dec 18 14:07:07 2015 From: wadud.miah at gmail.com (W. Miah) Date: Fri, 18 Dec 2015 20:07:07 +0000 Subject: [petsc-users] Intel MKL Sparse blas In-Reply-To: References: Message-ID: Hi Richard/Barry, I don't have any particular need just yet, but since most HPC clusters have Intel CPUs, I think an Intel Sparse MKL PETSc branch would bring immediate optimisations. I do realise that it does require quite a bit of work as Intel probably didn't design the API that well and which will require some re-factoring. I'm CC'ing Gennady from Intel whom I spoke to about this. Perhaps the MKL branch work can be coordinated with him. Regards, Wadud. On 18 December 2015 at 19:50, Richard Mills wrote: > Hi Wadud, > > I work at Intel and plan to add support for the inspector-executor sparse > BLAS routines (https://software.intel.com/en-us/node/590105) soon. (I > think this should be straightforward but it's been a question of simply > finding the time.) I will add a subclass of MATAIJ that uses these. Do > you ask because you have anything in particular that you are hoping to get > out of this? > > --Richard > > On Fri, Dec 18, 2015 at 10:49 AM, W. Miah wrote: > >> Hi, >> >> Do you have any plans for a PETSc port that uses the optimised Intel MKL >> sparse BLAS: >> >> https://software.intel.com/en-us/node/520797 >> >> I think this will be an interesting optimisation exercise. >> >> Best regards, >> >> -- >> web: http://miahw.wordpress.com >> mobile: +447905 755604 >> gpg: 4CC2 1A75 BDFB 2E29 B22F >> > > -- web: http://miahw.wordpress.com mobile: +447905 755604 gpg: 4CC2 1A75 BDFB 2E29 B22F -------------- next part -------------- An HTML attachment was scrubbed... URL: From richardtmills at gmail.com Fri Dec 18 15:44:31 2015 From: richardtmills at gmail.com (Richard Mills) Date: Fri, 18 Dec 2015 13:44:31 -0800 Subject: [petsc-users] Intel MKL Sparse blas In-Reply-To: <9B858EE3-73A2-4C94-9CCF-0A5443D97232@mcs.anl.gov> References: <9B858EE3-73A2-4C94-9CCF-0A5443D97232@mcs.anl.gov> Message-ID: Hi Barry, Just curious: Do you have any particular recommendations for how you would change the API? I am not a fan simply because the API is very "BLAS like". I'm wondering if that is your primary objection, or if there is anything that specifically sticks out as an annoyance. (I have yet to do any serious work with these, so I don't know what I would find needs improvement.) --Richard On Fri, Dec 18, 2015 at 11:33 AM, Barry Smith wrote: > > We're always happy to accept contributions. Intel unfortunately made > some terrible choices for APIs for these things so it is unlikely without a > specific funding source that we would ourselves add code to PETSc to > utilize them. > > Barry > > > On Dec 18, 2015, at 12:49 PM, W. Miah wrote: > > > > Hi, > > > > Do you have any plans for a PETSc port that uses the optimised Intel MKL > sparse BLAS: > > > > https://software.intel.com/en-us/node/520797 > > > > I think this will be an interesting optimisation exercise. > > > > Best regards, > > > > -- > > web: http://miahw.wordpress.com > > mobile: +447905 755604 > > gpg: 4CC2 1A75 BDFB 2E29 B22F > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Dec 18 15:49:54 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 18 Dec 2015 15:49:54 -0600 Subject: [petsc-users] Intel MKL Sparse blas In-Reply-To: References: <9B858EE3-73A2-4C94-9CCF-0A5443D97232@mcs.anl.gov> Message-ID: <20B8AED1-A3A2-416E-B4F2-8CF4184B4323@mcs.anl.gov> Richard, I haven't looked at it enough to have specific complaints :-) Barry > On Dec 18, 2015, at 3:44 PM, Richard Mills wrote: > > Hi Barry, > > Just curious: Do you have any particular recommendations for how you would change the API? I am not a fan simply because the API is very "BLAS like". I'm wondering if that is your primary objection, or if there is anything that specifically sticks out as an annoyance. (I have yet to do any serious work with these, so I don't know what I would find needs improvement.) > > --Richard > > On Fri, Dec 18, 2015 at 11:33 AM, Barry Smith wrote: > > We're always happy to accept contributions. Intel unfortunately made some terrible choices for APIs for these things so it is unlikely without a specific funding source that we would ourselves add code to PETSc to utilize them. > > Barry > > > On Dec 18, 2015, at 12:49 PM, W. Miah wrote: > > > > Hi, > > > > Do you have any plans for a PETSc port that uses the optimised Intel MKL sparse BLAS: > > > > https://software.intel.com/en-us/node/520797 > > > > I think this will be an interesting optimisation exercise. > > > > Best regards, > > > > -- > > web: http://miahw.wordpress.com > > mobile: +447905 755604 > > gpg: 4CC2 1A75 BDFB 2E29 B22F > > From knepley at gmail.com Fri Dec 18 18:05:24 2015 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 18 Dec 2015 16:05:24 -0800 Subject: [petsc-users] Status on parallel mesh reader in DMPlex In-Reply-To: References: Message-ID: I am doing it with Michael Lange. We plan to have it done by the summer. There is not much left to do. Thanks, Matt On Fri, Dec 18, 2015 at 8:21 AM, Justin Chang wrote: > Hi all, > > What's the status on the implementation of the parallel > mesh reader/generator for DMPlex meshes? Is anyone actively working on > this? If so is there a branch that I can peek into? > > Thanks, > Justin > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Dec 21 22:11:49 2015 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 21 Dec 2015 22:11:49 -0600 Subject: [petsc-users] Distributing data along with DMPlex In-Reply-To: References: Message-ID: On Thu, Dec 17, 2015 at 1:27 PM, Justin Chang wrote: > So you would use something like DMPlexDistributeField() > in > that case. You have your original/global DM, PetscSection, and Vec of > values. Then using the PetscSF that was created during DMPlexDistribute, > you map the global stuff into the local stuff. > Justin is correct. However, I will note that this strategy is not scalable. Usually the data is much larger than the mesh, so this runs out of memory much sooner. Are you sure that this is what you want? Thanks, Matt > On Thu, Dec 17, 2015 at 11:21 AM, Alejandro D Otero > wrote: > >> Thanks, >> The problem then is that after DMPlexDistribute the DMPlex 'points' are >> renumbered. So if the values are related to each point in the original >> numbering how do I set the values after the distribution. I know the >> property stored in the vector related to the entities with the numbering of >> the original mesh which I use to create the first DMPlex. >> >> Ideally for me, I would like to set the values in the vector before >> DMPlexDistribute and get the vector components renumbered and redistributed >> accordingly in a global vector. And then, get the local vector. >> >> Hope it could be more clear now. >> Regards, >> >> Alejandro >> >> >> >> On Wed, Dec 16, 2015 at 7:01 PM, Justin Chang >> wrote: >> >>> I think you would follow this order: >>> >>> 1*) create a DMPlex (depth, chart, etc) on rank 0. Other ranks have an >>> empty DM >>> >>> 2) DMPlexDistribute() >>> >>> 3*) Create the PetscSection >>> >>> 4) DMCreateGlobalVector() >>> >>> 5) DMCreateLocalVector() >>> >>> Now you have a global vector and a local vector for your distributed >>> DMPlex. The mapping/ghosting/etc of dofs is already taken care of. >>> >>> * if you're using standard Galerkin FE then in SNES examples 12 and 62 >>> (and maybe others?) the first step is handled through the mesh generation >>> functions and step 3 is handled through step 4 >>> >>> Thanks, >>> Justin >>> >>> On Wednesday, December 16, 2015, Alejandro D Otero >>> wrote: >>> >>>> Hi, I need some help understanding how to distribute data together with >>>> a dmplex representing a FE mesh. >>>> At the beginning I define the structure of the dmplex assigning certain >>>> number of DoF to cells, edges and vertexes, in one process (the dmplex in >>>> the rest is empty) >>>> I create a petscsecton and I create an associated global vector with >>>> the quantities I want to store. >>>> Then I distribute the dmplex over all the processes. >>>> * Although this does not perform well it is just a starting point. I >>>> know it has to be improved. >>>> >>>> I would like to have the global vector distributed accordingly so that >>>> each process has access to the corresponding local part with its DoF >>>> (possibly adding some ghost values corresponding to the shared DoF not >>>> taken care by it). >>>> >>>> Is there any 'correct' way to do that in PETSc? >>>> >>>> Thanks in advance, >>>> >>>> Alejandro >>>> >>> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From kshyatt at physics.ucsb.edu Mon Dec 21 23:27:35 2015 From: kshyatt at physics.ucsb.edu (Katharine Hyatt) Date: Tue, 22 Dec 2015 00:27:35 -0500 Subject: [petsc-users] SEGV in PCSetUp_LU/MatFactorNumeric_MUMPS Message-ID: <00B3AEAD-5534-4312-9251-FFC91ACA332C@physics.ucsb.edu> Hello PETSc Users, I hope this is the right mailing list. I am hitting a SEGV in my code which is running SLEPc + PETSc to do a shift-and-invert eigensolution. I have run the code through valgrind and nothing pops up before (about 20 minutes in): [1]PETSC ERROR: #1 User provided function() line 0 in unknown file [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [2]PETSC ERROR: INSTEAD the line number of the start of the function [2]PETSC ERROR: is given. [2]PETSC ERROR: [2] MatFactorNumeric_MUMPS line 1151 /home/kshyatt/software/petsc-3.6.0/src/mat/impls/aij/mpi/mumps/mumps.c [2]PETSC ERROR: [2] MatLUFactorNumeric line 2937 /home/kshyatt/software/petsc-3.6.0/src/mat/interface/matrix.c [2]PETSC ERROR: [2] PCSetUp_LU line 99 /home/kshyatt/software/petsc-3.6.0/src/ksp/pc/impls/factor/lu/lu.c [4]PETSC ERROR: [4] PCSetUp_LU line 99 /home/kshyatt/software/petsc-3.6.0/src/ksp/pc/impls/factor/lu/lu.c [4]PETSC ERROR: [4] PCSetUp line 944 /home/kshyatt/software/petsc-3.6.0/src/ksp/pc/interface/precon.c [4]PETSC ERROR: [4] KSPSetUp line 247 /home/kshyatt/software/petsc-3.6.0/src/ksp/ksp/interface/itfunc.c [4]PETSC ERROR: [4] STSetUp_Sinvert line 116 /home/kshyatt/sources/slepc-3.6.0/src/sys/classes/st/impls/sinvert/sinvert.c [4]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [8]PETSC ERROR: Signal received [8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [2]PETSC ERROR: [2] PCSetUp line 944 /home/kshyatt/software/petsc-3.6.0/src/ksp/pc/interface/precon.c [2]PETSC ERROR: [2] KSPSetUp line 247 /home/kshyatt/software/petsc-3.6.0/src/ksp/ksp/interface/itfunc.c [2]PETSC ERROR: [2] STSetUp_Sinvert line 116 /home/kshyatt/sources/slepc-3.6.0/src/sys/classes/st/impls/sinvert/sinvert.c [2]PETSC ERROR: [2] STSetUp line 273 /home/kshyatt/sources/slepc-3.6.0/src/sys/classes/st/interface/stsolve.c [2]PETSC ERROR: [4] STSetUp line 273 /home/kshyatt/sources/slepc-3.6.0/src/sys/classes/st/interface/stsolve.c [4]PETSC ERROR: [4] EPSSetUp line 58 /home/kshyatt/sources/slepc-3.6.0/src/eps/interface/epssetup.c [4]PETSC ERROR: [4] EPSSolve line 83 /home/kshyatt/sources/slepc-3.6.0/src/eps/interface/epssolve.c [8]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015 [8]PETSC ERROR: ./diagonalize on a arch-linux2-c-debug named node22 by kshyatt Mon Dec 21 15:54:32 2015 [8]PETSC ERROR: Configure options --download-hdf5 --download-parmetis --download-superlu_dist --download-superlu --download-cmake --download-metis --download-mumps --download-scalapack --with-blas-lapack-dir=/opt/intel/composer_xe_2015/mkl/lib/intel64/ I have tried SuperLU_dist as well, and the error has not happened over 6 hours so far. I?ve attached the code I am using. I have tried this code successfully on smaller matrices ( dimension 63,503 x 63,503, with about 698,540 nonzero elements) and I haven?t attached the bigger matrix I?m trying now because the HDF5 file is very large. The larger matrix is size 853,775 x 853,775 and has 11,099,080 nonzero elements. If you need it to reproduce the problem, I can copy it somewhere you can access, or I can send you a Julia script to generate an equivalent one. Running with valgrind and SuperLU_dist doesn?t show anything. I?m running the script across 10 nodes, with 12 CPUs each, and about 47GB of RAM + swap. Even when I run with just 1 MPI process per node, I see the segfault. I?m running the application with: mpirun -npernode 1 -machinefile $PBS_NODEFILE ./diagonalize 1 bilayer-24-01-8-4.h5 1-bilayer-24-01-8-4.h5 4 24 -eps_nev 250 -st_ksp_type preonly -st_pc_type lu -st_pc_factor_mat_solver_package mumps -st_ksp_rtol 1e-7 -eps_monitor I don?t see any output from SLEPc?s EPS monitoring before the segfault (which makes sense to me, because it?s happening in the inversion step, right?). The segfault has also happened to me with version 3.6.3 of PETSc on SDSC?s Comet. Is this a problem with my code? With PETSc? or with MUMPS? Thanks, Katharine -------------- next part -------------- A non-text attachment was scrubbed... Name: diagonalize.cpp Type: application/octet-stream Size: 8790 bytes Desc: not available URL: -------------- next part -------------- From timothee.nicolas at gmail.com Tue Dec 22 00:02:48 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Tue, 22 Dec 2015 15:02:48 +0900 Subject: [petsc-users] DMCreateMatrix from one DMDA to an another In-Reply-To: References: Message-ID: Hi, I'd like to take advantage of the stencil structure of the DMDA to write into my matrix. I have already written all my routines so that I can use MatSetValuesStencil so it's most convenient for me, otherwise I will have to rethink all the structure. Since I don't create my rectangular matrix with DMCreateMatrix but with MatCreate and MatSetSizes instead, the manual tells me I have to call first MatSetStencil and MatSetLocalToGlobalMapping before being allowed to call MatSetValuesStencil. So before going on the full example, I tried to do this on a simple square matrix but I can't get the thing to work. It must be an obvious mistake but can't find it. My code is PetscErrorCode :: ierr DM :: da Mat :: A PetscInt :: xm ISLocalToGlobalMapping :: ltog PetscInt :: dims(1), starts(1) PetscInt :: dim MatStencil :: row(4,1), col(4,1) PetscScalar :: v call DMDACreate1d (PETSC_COMM_WORLD,DM_BOUNDARY_NONE,iten,ione,ione,PETSC_NULL_INTEGER,da,ierr) call DMDAGetCorners(da,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, & & xm,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr) call MatCreate(PETSC_COMM_WORLD,A,ierr) call MatSetSizes(A,xm,xm,PETSC_DETERMINE,PETSC_DETERMINE,ierr) call DMGetLocalToGlobalMapping(da,ltog,ierr) call MatSetLocalToGlobalMapping(A,ltog,ltog,ierr) call DMDAGetGhostCorners(da,starts(1),PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, & & dims(1),PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr) dim=ione call MatSetStencil(A,dim,dims,starts,ione,ierr) row(MatStencil_i,1) = 0 col(MatStencil_i,1) = 0 call MatSetValuesStencil(A,ione,row,ione,col,zero,INSERT_VALUES,ierr) But I keep getting a segmentation fault error. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.1, Jul, 22, 2015 [0]PETSC ERROR: ./miips on a arch-darwin-c-debug named iMac27Nicolas.nifs.ac.jp by timotheenicolas Tue Dec 22 14:57:46 2015 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --with-debugging=no [0]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 [cli_0]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 9812 RUNNING AT iMac27Nicolas.nifs.ac.jp = EXIT CODE: 59 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== If I replace everything with call DMCreateMatrix(da,A,ierr) it works fine. What information did I miss from the manual ? What am I doing wrong ? Best Timothee 2015-12-06 23:16 GMT+09:00 Timoth?e Nicolas : > Wow, Great answer, thanks, I should probably be able to do it like this, > I'll try my best ! > > I actually do want the assembled matrix now. I have coded the matrix-free > version already and it works fine, but I want to use multigrid and I > figured that in order to be be most efficient at the coarse level it is > best to use -ksp_type preonly -pc_type lu, which requires a true matrix > (the matrix I am talking about is the product between a matrix 3dof --> > 5dof with a matrix 5dof --> 3dof, which ends up square). But I want to > retain a memory light implementation so I use matrix-free formulation at > all the levels but the last, which is also light by definition, so I should > not suffer much in terms of memory. > > Thanks > > Timoth?e > > > > 2015-12-06 22:29 GMT+09:00 Dave May : > >> >> >> On 6 December 2015 at 11:19, Timoth?e Nicolas > > wrote: >> >>> Hi, >>> >>> The answer to the first question is yes. I was puzzled about the second >>> part of the answer, but I realize I was not clear enough. I don't want to >>> just transfer information between 2 DMDAs, I want to perform an operation >>> on vectors with a rectangular matrix. >>> >> >> Okay - I misunderstood. >> >> >> >>> To be more explicit, the input vector of the operation is a vector with >>> 3 components (hence dof=3), and the results of the operator is put in one >>> vector equation + 2 scalar equation, which makes it dof = 5. >>> >>> So, either I cannot fill my matrix (if the number of rows exceeds what >>> is allowed by the DMDA), or I can fill my matrix but then the matrix and >>> vectors are not compatible, as in this error : >>> >> >> >> Do you need to explicitly assemble this rectangular operator? >> Expressing this operator via a matrix-free representation would be >> relatively straight forward. >> >> However, since the DMDA's have the same parallel layout you are in good >> shape to define the rectangular matrix if you really want the assembled >> thing. Your row space would be defined by the DMDA with block size 5 and >> the column space would be defined by the DMDA with block size 3. For >> example the matrix would be created like in the following way (assuming a >> 2D DMDA) >> >> PetscInt m,n,M,N; >> >> >> DMDAGetInfo(dm3,NULL,&M,&N,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL); >> nodes = M *N; >> DMDAGetCorners(dm3,NULL,NULL,NULL,&m,&n,NULL); >> lnodes = m * n; >> MatCreate(PETSC_COMM_WORLD,&A); >> MatSetSizes(A,lnodes*5,lnodes*3,nodes*5,nodes*3); >> >> The sparsity pattern is defined by the connectivity of the DMDA points >> AND the different block sizes. >> To define the non-zero structure, you need to traverse the part of the >> dmda defined by the natural indices given by si <= i < si+m, sj <= j < sj+n >> and check whether the the 5x3 block associated with each i,j is living >> within the diagonal or off-diagonal block of the matrix. For a given i,j >> the off-diagonal is defined by any i',j' associated your stencil which is >> not in the range si <= i' <= si + m, sj <= j' < sj + n. (e.g. it is a ghost >> node). >> >> Your arrays counting the non-zero entries on the diagonal (nnz[]) and off >> diag (nnz_od[]) can be indexed using i + j * m. This corresponds to the >> indexing used by the DMDA. Your non-zero counting function would look >> something like the pseudo code below >> >> PetscInt *nnz,*nnz_od; >> >> PetscMalloc(sizeof(PetscInt)*m*n*5,&nnz); >> PetscMalloc(sizeof(PetscInt)*m*n*5,&nnz_od); >> >> PetscMemzero(nnz,sizeof(PetscInt)*m*n*5); >> PetscMemzero(nnz_od,sizeof(PetscInt)*m*n*5); >> >> for (j=0; j> for (i=0; i> PetscInt idx,d,cnti,cntg; >> PetscBool is_ghost; >> >> idx = i + j * m; >> >> cnti = 0; >> cntg = 0; >> for (all_points_in_stencil) { /* User logic to define this loop */ >> >> is_ghost = PETSC_FALSE; >> /*User logic goes here to test if stencil point is a ghost point */ >> >> /* accumulate counts for the stencil */ >> if (is_ghost) { cntg++; } /* values living on a neighbour rank */ >> else { cnti++; } /* values local to current rank */ >> } >> >> /* assume all dofs in row and col space in block are coupled to each >> other */ >> for (d=0; d<5; d++) { >> nnz[5*idx+d] = 3 * cnti; >> nnz_od[5*idx+d] = 3 * cntg; >> } >> >> } >> } >> >> Thanks, >> Dave >> >> >>> >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Nonconforming object sizes >>> [0]PETSC ERROR: Mat mat,Vec y: global dim 4900 2940 >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.6.1, Jul, 22, 2015 >>> [0]PETSC ERROR: ./miips on a arch-linux2-c-debug named Carl-9000 by >>> timothee Sun Dec 6 19:17:20 2015 >>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >>> --with-fc=gfortran --download-fblaslapack --download-mpich >>> [0]PETSC ERROR: #1 MatMult() line 2215 in >>> /home/timothee/Documents/petsc-3.6.1/src/mat/interface/matrix.c >>> >>> So I assume it means the matrix is assumed to be square. Is there a way >>> to change this ? >>> >>> Best >>> >>> Timoth?e >>> >>> >>> >>> 2015-12-05 20:41 GMT+09:00 Dave May : >>> >>>> >>>> >>>> On 5 December 2015 at 11:15, Timoth?e Nicolas < >>>> timothee.nicolas at gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I need to create a rectangular matrix which takes vector structured by >>>>> a DMDA and transforms them to vectors structured with an other DMDA. It >>>>> does not look like there is a routine to do this since apparently >>>>> DMCreateMatrix assumes square matrix. What is the clean way to do this ? >>>>> >>>> >>>> Do the DMDA's have exactly the same parallel layout and only differ by >>>> the number of DOFs? >>>> >>>> If yes, and you only want to move entries between vectors associated >>>> with the two DMDA's, you can perform the transfer between vectors locally >>>> (rank-wise) in a very simple manner using the local number of points from >>>> DMDA and the known block sizes (DOFs) associated with your DMDA and the >>>> size of the subspace you ultimately want to construct. >>>> >>>> Thanks, >>>> Dave >>>> >>>> >>>>> The problem is I have a DMDA with 8 degrees of freedom (dof) and my >>>>> matrix sends vectors living in a subspace with 3 dof to the other 5 dof >>>>> subspace. I can define a square matrix living in the large, 8 dof subspace, >>>>> but this is of course very inefficient in terms of memory and time. >>>>> >>>>> Thanks >>>>> >>>>> Best >>>>> >>>>> Timoth?e >>>>> >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Tue Dec 22 04:14:00 2015 From: jroman at dsic.upv.es (Jose E. Roman) Date: Tue, 22 Dec 2015 11:14:00 +0100 Subject: [petsc-users] SEGV in PCSetUp_LU/MatFactorNumeric_MUMPS In-Reply-To: <00B3AEAD-5534-4312-9251-FFC91ACA332C@physics.ucsb.edu> References: <00B3AEAD-5534-4312-9251-FFC91ACA332C@physics.ucsb.edu> Message-ID: <7E31CA0B-5B69-4C84-917E-A890A4FBC4D2@dsic.upv.es> > El 22 dic 2015, a las 6:27, Katharine Hyatt escribi?: > > Hello PETSc Users, > > I hope this is the right mailing list. I am hitting a SEGV in my code which is running SLEPc + PETSc to do a shift-and-invert eigensolution. I have run the code through valgrind and nothing pops up before (about 20 minutes in): > > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [2]PETSC ERROR: INSTEAD the line number of the start of the function > [2]PETSC ERROR: is given. > [2]PETSC ERROR: [2] MatFactorNumeric_MUMPS line 1151 /home/kshyatt/software/petsc-3.6.0/src/mat/impls/aij/mpi/mumps/mumps.c > [2]PETSC ERROR: [2] MatLUFactorNumeric line 2937 /home/kshyatt/software/petsc-3.6.0/src/mat/interface/matrix.c > [2]PETSC ERROR: [2] PCSetUp_LU line 99 /home/kshyatt/software/petsc-3.6.0/src/ksp/pc/impls/factor/lu/lu.c > [4]PETSC ERROR: [4] PCSetUp_LU line 99 /home/kshyatt/software/petsc-3.6.0/src/ksp/pc/impls/factor/lu/lu.c > [4]PETSC ERROR: [4] PCSetUp line 944 /home/kshyatt/software/petsc-3.6.0/src/ksp/pc/interface/precon.c > [4]PETSC ERROR: [4] KSPSetUp line 247 /home/kshyatt/software/petsc-3.6.0/src/ksp/ksp/interface/itfunc.c > [4]PETSC ERROR: [4] STSetUp_Sinvert line 116 /home/kshyatt/sources/slepc-3.6.0/src/sys/classes/st/impls/sinvert/sinvert.c > [4]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [8]PETSC ERROR: Signal received > [8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [2]PETSC ERROR: [2] PCSetUp line 944 /home/kshyatt/software/petsc-3.6.0/src/ksp/pc/interface/precon.c > [2]PETSC ERROR: [2] KSPSetUp line 247 /home/kshyatt/software/petsc-3.6.0/src/ksp/ksp/interface/itfunc.c > [2]PETSC ERROR: [2] STSetUp_Sinvert line 116 /home/kshyatt/sources/slepc-3.6.0/src/sys/classes/st/impls/sinvert/sinvert.c > [2]PETSC ERROR: [2] STSetUp line 273 /home/kshyatt/sources/slepc-3.6.0/src/sys/classes/st/interface/stsolve.c > [2]PETSC ERROR: [4] STSetUp line 273 /home/kshyatt/sources/slepc-3.6.0/src/sys/classes/st/interface/stsolve.c > [4]PETSC ERROR: [4] EPSSetUp line 58 /home/kshyatt/sources/slepc-3.6.0/src/eps/interface/epssetup.c > [4]PETSC ERROR: [4] EPSSolve line 83 /home/kshyatt/sources/slepc-3.6.0/src/eps/interface/epssolve.c > [8]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015 > [8]PETSC ERROR: ./diagonalize on a arch-linux2-c-debug named node22 by kshyatt Mon Dec 21 15:54:32 2015 > [8]PETSC ERROR: Configure options --download-hdf5 --download-parmetis --download-superlu_dist --download-superlu --download-cmake --download-metis --download-mumps --download-scalapack --with-blas-lapack-dir=/opt/intel/composer_xe_2015/mkl/lib/intel64/ > > I have tried SuperLU_dist as well, and the error has not happened over 6 hours so far. I?ve attached the code I am using. I have tried this code successfully on smaller matrices ( dimension 63,503 x 63,503, with about 698,540 nonzero elements) and I haven?t attached the bigger matrix I?m trying now because the HDF5 file is very large. The larger matrix is size 853,775 x 853,775 and has 11,099,080 nonzero elements. If you need it to reproduce the problem, I can copy it somewhere you can access, or I can send you a Julia script to generate an equivalent one. Running with valgrind and SuperLU_dist doesn?t show anything. > > I?m running the script across 10 nodes, with 12 CPUs each, and about 47GB of RAM + swap. Even when I run with just 1 MPI process per node, I see the segfault. I?m running the application with: > > mpirun -npernode 1 -machinefile $PBS_NODEFILE ./diagonalize 1 bilayer-24-01-8-4.h5 1-bilayer-24-01-8-4.h5 4 24 -eps_nev 250 -st_ksp_type preonly -st_pc_type lu -st_pc_factor_mat_solver_package mumps -st_ksp_rtol 1e-7 -eps_monitor > > I don?t see any output from SLEPc?s EPS monitoring before the segfault (which makes sense to me, because it?s happening in the inversion step, right?). The segfault has also happened to me with version 3.6.3 of PETSc on SDSC?s Comet. > > Is this a problem with my code? With PETSc? or with MUMPS? > > Thanks, > Katharine > > SLEPc code seems correct. We strongly recommend EPSKRYLOVSCHUR instead of EPSLANCZOS, although this has nothing to do with the error you are reporting. I would bet it is an out-of-memory problem. In the SLEPc side, you can reduce the memory requirements by using the mpd parameter, see section 2.6.5 of the users manual, for instance -eps_mpd 30 However, most of the memory is being consumed by MUMPS. Maybe using Scotch instead of Parmetis reduces fill-in. You can also try running on more nodes. Jose From aotero at fi.uba.ar Tue Dec 22 07:17:13 2015 From: aotero at fi.uba.ar (Alejandro D Otero) Date: Tue, 22 Dec 2015 10:17:13 -0300 Subject: [petsc-users] Distributing data along with DMPlex In-Reply-To: References: Message-ID: Thanks for your answer. In principle, this is what I want, although I know that this doesn't scale. At this point I am reading the mesh in only one proc so I have all the properties there, that is why after that I need to distribute the mesh along with the data. Is there a better solution within this scheme (only one proc reads mesh data)? Regards, Alejandro On Tue, Dec 22, 2015 at 1:11 AM, Matthew Knepley wrote: > On Thu, Dec 17, 2015 at 1:27 PM, Justin Chang wrote: > >> So you would use something like DMPlexDistributeField() >> in >> that case. You have your original/global DM, PetscSection, and Vec of >> values. Then using the PetscSF that was created during DMPlexDistribute, >> you map the global stuff into the local stuff. >> > > Justin is correct. However, I will note that this strategy is not > scalable. Usually the data is much larger than the mesh, > so this runs out of memory much sooner. Are you sure that this is what you > want? > > Thanks, > > Matt > > >> On Thu, Dec 17, 2015 at 11:21 AM, Alejandro D Otero >> wrote: >> >>> Thanks, >>> The problem then is that after DMPlexDistribute the DMPlex 'points' are >>> renumbered. So if the values are related to each point in the original >>> numbering how do I set the values after the distribution. I know the >>> property stored in the vector related to the entities with the numbering of >>> the original mesh which I use to create the first DMPlex. >>> >>> Ideally for me, I would like to set the values in the vector before >>> DMPlexDistribute and get the vector components renumbered and redistributed >>> accordingly in a global vector. And then, get the local vector. >>> >>> Hope it could be more clear now. >>> Regards, >>> >>> Alejandro >>> >>> >>> >>> On Wed, Dec 16, 2015 at 7:01 PM, Justin Chang >>> wrote: >>> >>>> I think you would follow this order: >>>> >>>> 1*) create a DMPlex (depth, chart, etc) on rank 0. Other ranks have an >>>> empty DM >>>> >>>> 2) DMPlexDistribute() >>>> >>>> 3*) Create the PetscSection >>>> >>>> 4) DMCreateGlobalVector() >>>> >>>> 5) DMCreateLocalVector() >>>> >>>> Now you have a global vector and a local vector for your distributed >>>> DMPlex. The mapping/ghosting/etc of dofs is already taken care of. >>>> >>>> * if you're using standard Galerkin FE then in SNES examples 12 and 62 >>>> (and maybe others?) the first step is handled through the mesh generation >>>> functions and step 3 is handled through step 4 >>>> >>>> Thanks, >>>> Justin >>>> >>>> On Wednesday, December 16, 2015, Alejandro D Otero >>>> wrote: >>>> >>>>> Hi, I need some help understanding how to distribute data together >>>>> with a dmplex representing a FE mesh. >>>>> At the beginning I define the structure of the dmplex assigning >>>>> certain number of DoF to cells, edges and vertexes, in one process (the >>>>> dmplex in the rest is empty) >>>>> I create a petscsecton and I create an associated global vector with >>>>> the quantities I want to store. >>>>> Then I distribute the dmplex over all the processes. >>>>> * Although this does not perform well it is just a starting point. I >>>>> know it has to be improved. >>>>> >>>>> I would like to have the global vector distributed accordingly so that >>>>> each process has access to the corresponding local part with its DoF >>>>> (possibly adding some ghost values corresponding to the shared DoF not >>>>> taken care by it). >>>>> >>>>> Is there any 'correct' way to do that in PETSc? >>>>> >>>>> Thanks in advance, >>>>> >>>>> Alejandro >>>>> >>>> >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From snakexf at gmail.com Tue Dec 22 07:51:05 2015 From: snakexf at gmail.com (Feng Xing) Date: Tue, 22 Dec 2015 14:51:05 +0100 Subject: [petsc-users] Reuse direct solver Message-ID: Hello everyone, I want to solve Ax=b with a direct LU method many times where A is a parallel aij matrix. I found two ways in the doc and in some slides (maybe not updated). 1. KSPSetType(ksp,KSPPREONLY); PCSetType(pc,PCLU); PCFactorSetMatSolverPackage(pc,MATSOLVERSUPERLU_DIST); KSPSetReusePreconditioner(ksp, PETSC_TRUE); KSPSolve(A,b,x); 2. MatGetOrdering(A, MATORDERINGNATURAL, &isr, &isc); MatGetFactor(A, MATSOLVERMUMPS, MAT_FACTOR_LU, &A_fact); MatLUFactorSymbolic(A_fact, A, isr, isc, &info); MatLUFactorNumeric(A_fact, A, &info); MatSolve(A_fact,b,x); I would like to ask if both of them allow to do one time LU factorisation, then implement backward/forward LU each KSPSolve or MatSolve? Besides, does the ways support superlu_dist and mumps? Thank you very much. Feng Xing Postdoc in France From knepley at gmail.com Tue Dec 22 08:51:20 2015 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 22 Dec 2015 08:51:20 -0600 Subject: [petsc-users] Distributing data along with DMPlex In-Reply-To: References: Message-ID: On Tue, Dec 22, 2015 at 7:17 AM, Alejandro D Otero wrote: > Thanks for your answer. > In principle, this is what I want, although I know that this doesn't > scale. At this point I am reading the mesh in only one proc so I have all > the properties there, that is why after that I need to distribute the mesh > along with the data. > Is there a better solution within this scheme (only one proc reads mesh > data)? > Most physical problems that I deal with decouple properties from the mesh. This seems reasonable since the mesh is just a crutch to solve the continuum problem. So, for example, in PyLith ( https://geodynamics.org/cig/software/pylith/), we read in/generate a mesh in serial, manipulate it to add faults etc., mark the volumes/boundaries, distribute it, and then in parallel for each local marked section we queries a spatial database to get the properties and boundary data. By summer, we will have parallel read for the initial mesh. Thanks, Matt > Regards, > > Alejandro > > On Tue, Dec 22, 2015 at 1:11 AM, Matthew Knepley > wrote: > >> On Thu, Dec 17, 2015 at 1:27 PM, Justin Chang >> wrote: >> >>> So you would use something like DMPlexDistributeField() >>> in >>> that case. You have your original/global DM, PetscSection, and Vec of >>> values. Then using the PetscSF that was created during DMPlexDistribute, >>> you map the global stuff into the local stuff. >>> >> >> Justin is correct. However, I will note that this strategy is not >> scalable. Usually the data is much larger than the mesh, >> so this runs out of memory much sooner. Are you sure that this is what >> you want? >> >> Thanks, >> >> Matt >> >> >>> On Thu, Dec 17, 2015 at 11:21 AM, Alejandro D Otero >>> wrote: >>> >>>> Thanks, >>>> The problem then is that after DMPlexDistribute the DMPlex 'points' are >>>> renumbered. So if the values are related to each point in the original >>>> numbering how do I set the values after the distribution. I know the >>>> property stored in the vector related to the entities with the numbering of >>>> the original mesh which I use to create the first DMPlex. >>>> >>>> Ideally for me, I would like to set the values in the vector before >>>> DMPlexDistribute and get the vector components renumbered and redistributed >>>> accordingly in a global vector. And then, get the local vector. >>>> >>>> Hope it could be more clear now. >>>> Regards, >>>> >>>> Alejandro >>>> >>>> >>>> >>>> On Wed, Dec 16, 2015 at 7:01 PM, Justin Chang >>>> wrote: >>>> >>>>> I think you would follow this order: >>>>> >>>>> 1*) create a DMPlex (depth, chart, etc) on rank 0. Other ranks have an >>>>> empty DM >>>>> >>>>> 2) DMPlexDistribute() >>>>> >>>>> 3*) Create the PetscSection >>>>> >>>>> 4) DMCreateGlobalVector() >>>>> >>>>> 5) DMCreateLocalVector() >>>>> >>>>> Now you have a global vector and a local vector for your distributed >>>>> DMPlex. The mapping/ghosting/etc of dofs is already taken care of. >>>>> >>>>> * if you're using standard Galerkin FE then in SNES examples 12 and 62 >>>>> (and maybe others?) the first step is handled through the mesh generation >>>>> functions and step 3 is handled through step 4 >>>>> >>>>> Thanks, >>>>> Justin >>>>> >>>>> On Wednesday, December 16, 2015, Alejandro D Otero >>>>> wrote: >>>>> >>>>>> Hi, I need some help understanding how to distribute data together >>>>>> with a dmplex representing a FE mesh. >>>>>> At the beginning I define the structure of the dmplex assigning >>>>>> certain number of DoF to cells, edges and vertexes, in one process (the >>>>>> dmplex in the rest is empty) >>>>>> I create a petscsecton and I create an associated global vector with >>>>>> the quantities I want to store. >>>>>> Then I distribute the dmplex over all the processes. >>>>>> * Although this does not perform well it is just a starting point. I >>>>>> know it has to be improved. >>>>>> >>>>>> I would like to have the global vector distributed accordingly so >>>>>> that each process has access to the corresponding local part with its DoF >>>>>> (possibly adding some ghost values corresponding to the shared DoF not >>>>>> taken care by it). >>>>>> >>>>>> Is there any 'correct' way to do that in PETSc? >>>>>> >>>>>> Thanks in advance, >>>>>> >>>>>> Alejandro >>>>>> >>>>> >>>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Dec 22 08:53:41 2015 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 22 Dec 2015 08:53:41 -0600 Subject: [petsc-users] Reuse direct solver In-Reply-To: References: Message-ID: On Tue, Dec 22, 2015 at 7:51 AM, Feng Xing wrote: > Hello everyone, > > I want to solve Ax=b with a direct LU method many times where A is a > parallel aij matrix. I found two ways in the doc and in some slides (maybe > not updated). > > 1. KSPSetType(ksp,KSPPREONLY); > PCSetType(pc,PCLU); > PCFactorSetMatSolverPackage(pc,MATSOLVERSUPERLU_DIST); > KSPSetReusePreconditioner(ksp, PETSC_TRUE); > KSPSolve(A,b,x); > Use the first method. In fact, if your matrix does not change, you do not even need the Reuse flag. PETSc will automatically reuse the PC. The flag allows you to reuse the PC for a changing matrix. Thanks, Matt > 2. MatGetOrdering(A, MATORDERINGNATURAL, &isr, &isc); > MatGetFactor(A, MATSOLVERMUMPS, MAT_FACTOR_LU, &A_fact); > MatLUFactorSymbolic(A_fact, A, isr, isc, &info); > MatLUFactorNumeric(A_fact, A, &info); > MatSolve(A_fact,b,x); > > I would like to ask if both of them allow to do one time LU factorisation, > then implement backward/forward LU each KSPSolve or MatSolve? > Besides, does the ways support superlu_dist and mumps? > > Thank you very much. > > Feng Xing > Postdoc in France > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Dec 22 15:05:51 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 22 Dec 2015 15:05:51 -0600 Subject: [petsc-users] DMCreateMatrix from one DMDA to an another In-Reply-To: References: Message-ID: First run with valgrind and if that doesn't find anything then in the debugger and check the data structures where it crashes. There is nothing obviously wrong, some subtle mistake that can only be found with valgrind. Barry > On Dec 22, 2015, at 12:02 AM, Timoth?e Nicolas wrote: > > Hi, > > I'd like to take advantage of the stencil structure of the DMDA to write into my matrix. I have already written all my routines so that I can use MatSetValuesStencil so it's most convenient for me, otherwise I will have to rethink all the structure. > > Since I don't create my rectangular matrix with DMCreateMatrix but with MatCreate and MatSetSizes instead, the manual tells me I have to call first MatSetStencil and MatSetLocalToGlobalMapping before being allowed to call MatSetValuesStencil. So before going on the full example, I tried to do this on a simple square matrix but I can't get the thing to work. It must be an obvious mistake but can't find it. My code is > > PetscErrorCode :: ierr > DM :: da > Mat :: A > PetscInt :: xm > ISLocalToGlobalMapping :: ltog > PetscInt :: dims(1), starts(1) > PetscInt :: dim > MatStencil :: row(4,1), col(4,1) > PetscScalar :: v > > call DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,iten,ione,ione,PETSC_NULL_INTEGER,da,ierr) > call DMDAGetCorners(da,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, & > & xm,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr) > > call MatCreate(PETSC_COMM_WORLD,A,ierr) > > call MatSetSizes(A,xm,xm,PETSC_DETERMINE,PETSC_DETERMINE,ierr) > > call DMGetLocalToGlobalMapping(da,ltog,ierr) > call MatSetLocalToGlobalMapping(A,ltog,ltog,ierr) > call DMDAGetGhostCorners(da,starts(1),PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, & > & dims(1),PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr) > dim=ione > call MatSetStencil(A,dim,dims,starts,ione,ierr) > > row(MatStencil_i,1) = 0 > col(MatStencil_i,1) = 0 > > call MatSetValuesStencil(A,ione,row,ione,col,zero,INSERT_VALUES,ierr) > > But I keep getting a segmentation fault error. > > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [0]PETSC ERROR: to get more information on the crash. > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.1, Jul, 22, 2015 > [0]PETSC ERROR: ./miips on a arch-darwin-c-debug named iMac27Nicolas.nifs.ac.jp by timotheenicolas Tue Dec 22 14:57:46 2015 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --with-debugging=no > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > [cli_0]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 9812 RUNNING AT iMac27Nicolas.nifs.ac.jp > = EXIT CODE: 59 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > > If I replace everything with call DMCreateMatrix(da,A,ierr) it works fine. > > What information did I miss from the manual ? What am I doing wrong ? > > Best > > Timothee > > > > > > 2015-12-06 23:16 GMT+09:00 Timoth?e Nicolas : > Wow, Great answer, thanks, I should probably be able to do it like this, I'll try my best ! > > I actually do want the assembled matrix now. I have coded the matrix-free version already and it works fine, but I want to use multigrid and I figured that in order to be be most efficient at the coarse level it is best to use -ksp_type preonly -pc_type lu, which requires a true matrix (the matrix I am talking about is the product between a matrix 3dof --> 5dof with a matrix 5dof --> 3dof, which ends up square). But I want to retain a memory light implementation so I use matrix-free formulation at all the levels but the last, which is also light by definition, so I should not suffer much in terms of memory. > > Thanks > > Timoth?e > > > > 2015-12-06 22:29 GMT+09:00 Dave May : > > > On 6 December 2015 at 11:19, Timoth?e Nicolas wrote: > Hi, > > The answer to the first question is yes. I was puzzled about the second part of the answer, but I realize I was not clear enough. I don't want to just transfer information between 2 DMDAs, I want to perform an operation on vectors with a rectangular matrix. > > Okay - I misunderstood. > > > To be more explicit, the input vector of the operation is a vector with 3 components (hence dof=3), and the results of the operator is put in one vector equation + 2 scalar equation, which makes it dof = 5. > > So, either I cannot fill my matrix (if the number of rows exceeds what is allowed by the DMDA), or I can fill my matrix but then the matrix and vectors are not compatible, as in this error : > > > Do you need to explicitly assemble this rectangular operator? > Expressing this operator via a matrix-free representation would be relatively straight forward. > > However, since the DMDA's have the same parallel layout you are in good shape to define the rectangular matrix if you really want the assembled thing. Your row space would be defined by the DMDA with block size 5 and the column space would be defined by the DMDA with block size 3. For example the matrix would be created like in the following way (assuming a 2D DMDA) > > PetscInt m,n,M,N; > > DMDAGetInfo(dm3,NULL,&M,&N,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL); > nodes = M *N; > DMDAGetCorners(dm3,NULL,NULL,NULL,&m,&n,NULL); > lnodes = m * n; > MatCreate(PETSC_COMM_WORLD,&A); > MatSetSizes(A,lnodes*5,lnodes*3,nodes*5,nodes*3); > > The sparsity pattern is defined by the connectivity of the DMDA points AND the different block sizes. > To define the non-zero structure, you need to traverse the part of the dmda defined by the natural indices given by si <= i < si+m, sj <= j < sj+n and check whether the the 5x3 block associated with each i,j is living within the diagonal or off-diagonal block of the matrix. For a given i,j the off-diagonal is defined by any i',j' associated your stencil which is not in the range si <= i' <= si + m, sj <= j' < sj + n. (e.g. it is a ghost node). > > Your arrays counting the non-zero entries on the diagonal (nnz[]) and off diag (nnz_od[]) can be indexed using i + j * m. This corresponds to the indexing used by the DMDA. Your non-zero counting function would look something like the pseudo code below > > PetscInt *nnz,*nnz_od; > > PetscMalloc(sizeof(PetscInt)*m*n*5,&nnz); > PetscMalloc(sizeof(PetscInt)*m*n*5,&nnz_od); > > PetscMemzero(nnz,sizeof(PetscInt)*m*n*5); > PetscMemzero(nnz_od,sizeof(PetscInt)*m*n*5); > > for (j=0; j for (i=0; i PetscInt idx,d,cnti,cntg; > PetscBool is_ghost; > > idx = i + j * m; > > cnti = 0; > cntg = 0; > for (all_points_in_stencil) { /* User logic to define this loop */ > > is_ghost = PETSC_FALSE; > /*User logic goes here to test if stencil point is a ghost point */ > > /* accumulate counts for the stencil */ > if (is_ghost) { cntg++; } /* values living on a neighbour rank */ > else { cnti++; } /* values local to current rank */ > } > > /* assume all dofs in row and col space in block are coupled to each other */ > for (d=0; d<5; d++) { > nnz[5*idx+d] = 3 * cnti; > nnz_od[5*idx+d] = 3 * cntg; > } > > } > } > > Thanks, > Dave > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Nonconforming object sizes > [0]PETSC ERROR: Mat mat,Vec y: global dim 4900 2940 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.1, Jul, 22, 2015 > [0]PETSC ERROR: ./miips on a arch-linux2-c-debug named Carl-9000 by timothee Sun Dec 6 19:17:20 2015 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich > [0]PETSC ERROR: #1 MatMult() line 2215 in /home/timothee/Documents/petsc-3.6.1/src/mat/interface/matrix.c > > So I assume it means the matrix is assumed to be square. Is there a way to change this ? > > Best > > Timoth?e > > > > 2015-12-05 20:41 GMT+09:00 Dave May : > > > On 5 December 2015 at 11:15, Timoth?e Nicolas wrote: > Hi, > > I need to create a rectangular matrix which takes vector structured by a DMDA and transforms them to vectors structured with an other DMDA. It does not look like there is a routine to do this since apparently DMCreateMatrix assumes square matrix. What is the clean way to do this ? > > Do the DMDA's have exactly the same parallel layout and only differ by the number of DOFs? > > If yes, and you only want to move entries between vectors associated with the two DMDA's, you can perform the transfer between vectors locally (rank-wise) in a very simple manner using the local number of points from DMDA and the known block sizes (DOFs) associated with your DMDA and the size of the subspace you ultimately want to construct. > > Thanks, > Dave > > > The problem is I have a DMDA with 8 degrees of freedom (dof) and my matrix sends vectors living in a subspace with 3 dof to the other 5 dof subspace. I can define a square matrix living in the large, 8 dof subspace, but this is of course very inefficient in terms of memory and time. > > Thanks > > Best > > Timoth?e > > > > > From kshyatt at physics.ucsb.edu Tue Dec 22 16:32:16 2015 From: kshyatt at physics.ucsb.edu (Katharine Hyatt) Date: Tue, 22 Dec 2015 17:32:16 -0500 Subject: [petsc-users] SEGV in PCSetUp_LU/MatFactorNumeric_MUMPS In-Reply-To: <7E31CA0B-5B69-4C84-917E-A890A4FBC4D2@dsic.upv.es> References: <00B3AEAD-5534-4312-9251-FFC91ACA332C@physics.ucsb.edu> <7E31CA0B-5B69-4C84-917E-A890A4FBC4D2@dsic.upv.es> Message-ID: <269773DD-F861-4727-9F6C-F300A97045D4@physics.ucsb.edu> Setting -eps_mdp and going up to 20 nodes seems to have fixed the problem. Thanks! > On Dec 22, 2015, at 05:14, Jose E. Roman wrote: > > >> El 22 dic 2015, a las 6:27, Katharine Hyatt escribi?: >> >> Hello PETSc Users, >> >> I hope this is the right mailing list. I am hitting a SEGV in my code which is running SLEPc + PETSc to do a shift-and-invert eigensolution. I have run the code through valgrind and nothing pops up before (about 20 minutes in): >> >> [1]PETSC ERROR: #1 User provided function() line 0 in unknown file >> [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> [2]PETSC ERROR: INSTEAD the line number of the start of the function >> [2]PETSC ERROR: is given. >> [2]PETSC ERROR: [2] MatFactorNumeric_MUMPS line 1151 /home/kshyatt/software/petsc-3.6.0/src/mat/impls/aij/mpi/mumps/mumps.c >> [2]PETSC ERROR: [2] MatLUFactorNumeric line 2937 /home/kshyatt/software/petsc-3.6.0/src/mat/interface/matrix.c >> [2]PETSC ERROR: [2] PCSetUp_LU line 99 /home/kshyatt/software/petsc-3.6.0/src/ksp/pc/impls/factor/lu/lu.c >> [4]PETSC ERROR: [4] PCSetUp_LU line 99 /home/kshyatt/software/petsc-3.6.0/src/ksp/pc/impls/factor/lu/lu.c >> [4]PETSC ERROR: [4] PCSetUp line 944 /home/kshyatt/software/petsc-3.6.0/src/ksp/pc/interface/precon.c >> [4]PETSC ERROR: [4] KSPSetUp line 247 /home/kshyatt/software/petsc-3.6.0/src/ksp/ksp/interface/itfunc.c >> [4]PETSC ERROR: [4] STSetUp_Sinvert line 116 /home/kshyatt/sources/slepc-3.6.0/src/sys/classes/st/impls/sinvert/sinvert.c >> [4]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [8]PETSC ERROR: Signal received >> [8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [2]PETSC ERROR: [2] PCSetUp line 944 /home/kshyatt/software/petsc-3.6.0/src/ksp/pc/interface/precon.c >> [2]PETSC ERROR: [2] KSPSetUp line 247 /home/kshyatt/software/petsc-3.6.0/src/ksp/ksp/interface/itfunc.c >> [2]PETSC ERROR: [2] STSetUp_Sinvert line 116 /home/kshyatt/sources/slepc-3.6.0/src/sys/classes/st/impls/sinvert/sinvert.c >> [2]PETSC ERROR: [2] STSetUp line 273 /home/kshyatt/sources/slepc-3.6.0/src/sys/classes/st/interface/stsolve.c >> [2]PETSC ERROR: [4] STSetUp line 273 /home/kshyatt/sources/slepc-3.6.0/src/sys/classes/st/interface/stsolve.c >> [4]PETSC ERROR: [4] EPSSetUp line 58 /home/kshyatt/sources/slepc-3.6.0/src/eps/interface/epssetup.c >> [4]PETSC ERROR: [4] EPSSolve line 83 /home/kshyatt/sources/slepc-3.6.0/src/eps/interface/epssolve.c >> [8]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015 >> [8]PETSC ERROR: ./diagonalize on a arch-linux2-c-debug named node22 by kshyatt Mon Dec 21 15:54:32 2015 >> [8]PETSC ERROR: Configure options --download-hdf5 --download-parmetis --download-superlu_dist --download-superlu --download-cmake --download-metis --download-mumps --download-scalapack --with-blas-lapack-dir=/opt/intel/composer_xe_2015/mkl/lib/intel64/ >> >> I have tried SuperLU_dist as well, and the error has not happened over 6 hours so far. I?ve attached the code I am using. I have tried this code successfully on smaller matrices ( dimension 63,503 x 63,503, with about 698,540 nonzero elements) and I haven?t attached the bigger matrix I?m trying now because the HDF5 file is very large. The larger matrix is size 853,775 x 853,775 and has 11,099,080 nonzero elements. If you need it to reproduce the problem, I can copy it somewhere you can access, or I can send you a Julia script to generate an equivalent one. Running with valgrind and SuperLU_dist doesn?t show anything. >> >> I?m running the script across 10 nodes, with 12 CPUs each, and about 47GB of RAM + swap. Even when I run with just 1 MPI process per node, I see the segfault. I?m running the application with: >> >> mpirun -npernode 1 -machinefile $PBS_NODEFILE ./diagonalize 1 bilayer-24-01-8-4.h5 1-bilayer-24-01-8-4.h5 4 24 -eps_nev 250 -st_ksp_type preonly -st_pc_type lu -st_pc_factor_mat_solver_package mumps -st_ksp_rtol 1e-7 -eps_monitor >> >> I don?t see any output from SLEPc?s EPS monitoring before the segfault (which makes sense to me, because it?s happening in the inversion step, right?). The segfault has also happened to me with version 3.6.3 of PETSc on SDSC?s Comet. >> >> Is this a problem with my code? With PETSc? or with MUMPS? >> >> Thanks, >> Katharine >> >> > > SLEPc code seems correct. We strongly recommend EPSKRYLOVSCHUR instead of EPSLANCZOS, although this has nothing to do with the error you are reporting. > > I would bet it is an out-of-memory problem. In the SLEPc side, you can reduce the memory requirements by using the mpd parameter, see section 2.6.5 of the users manual, for instance -eps_mpd 30 > However, most of the memory is being consumed by MUMPS. Maybe using Scotch instead of Parmetis reduces fill-in. You can also try running on more nodes. > > Jose > > From zonexo at gmail.com Tue Dec 22 19:48:30 2015 From: zonexo at gmail.com (TAY wee-beng) Date: Wed, 23 Dec 2015 09:48:30 +0800 Subject: [petsc-users] Removing unused warning during gnu compile Message-ID: <5679FD6E.4070706@gmail.com> Hi, When I compile using gfortran, I need it to warn me of any unused parameter in my code. However, I found that it also warns of unused parameter in PETSc code, as shown below. How can I let it only display warning for my own code, instead of PETSc? Thanks. wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petscsnes.h:116.54: Included at /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petsc.h:15: Included at fractional_initial.F90:28: integer(kind=selected_int_kind(5)) SNES_QN_LBFGS 1 Warning: Unused parameter 'snes_qn_lbfgs' declared at (1) /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petscsnes.h:144.64: Included at /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petsc.h:15: Included at fractional_initial.F90:28: integer(kind=selected_int_kind(5)) SNES_QN_RESTART_DEFAULT 1 Warning: Unused parameter 'snes_qn_restart_default' declared at (1) /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petscsnes.h:145.61: Included at /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petsc.h:15: Included at fractional_initial.F90:28: integer(kind=selected_int_kind(5)) SNES_QN_RESTART_NONE 1 Warning: Unused parameter 'snes_qn_restart_none' declared at (1) /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petscsnes.h:147.65: Included at /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petsc.h:15: Included at fractional_initial.F90:28: integer(kind=selected_int_kind(5)) SNES_QN_RESTART_PERIODIC 1 -- Thank you Yours sincerely, TAY wee-beng From bsmith at mcs.anl.gov Tue Dec 22 22:33:25 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 22 Dec 2015 22:33:25 -0600 Subject: [petsc-users] Removing unused warning during gnu compile In-Reply-To: <5679FD6E.4070706@gmail.com> References: <5679FD6E.4070706@gmail.com> Message-ID: <8B21BC6F-8ED0-4756-BA5A-4B382167BAE2@mcs.anl.gov> Are the warning messages below the only ones you get or do you get many more? Barry > On Dec 22, 2015, at 7:48 PM, TAY wee-beng wrote: > > Hi, > > When I compile using gfortran, I need it to warn me of any unused parameter in my code. However, I found that it also warns of unused parameter in PETSc code, as shown below. How can I let it only display warning for my own code, instead of PETSc? Thanks. > > wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petscsnes.h:116.54: > Included at /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petsc.h:15: > Included at fractional_initial.F90:28: > > integer(kind=selected_int_kind(5)) SNES_QN_LBFGS > 1 > Warning: Unused parameter 'snes_qn_lbfgs' declared at (1) > /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petscsnes.h:144.64: > Included at /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petsc.h:15: > Included at fractional_initial.F90:28: > > integer(kind=selected_int_kind(5)) SNES_QN_RESTART_DEFAULT > 1 > Warning: Unused parameter 'snes_qn_restart_default' declared at (1) > /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petscsnes.h:145.61: > Included at /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petsc.h:15: > Included at fractional_initial.F90:28: > > integer(kind=selected_int_kind(5)) SNES_QN_RESTART_NONE > 1 > Warning: Unused parameter 'snes_qn_restart_none' declared at (1) > /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petscsnes.h:147.65: > Included at /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petsc.h:15: > Included at fractional_initial.F90:28: > > integer(kind=selected_int_kind(5)) SNES_QN_RESTART_PERIODIC > 1 > > -- > Thank you > > Yours sincerely, > > TAY wee-beng > From timothee.nicolas at gmail.com Wed Dec 23 00:34:22 2015 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Wed, 23 Dec 2015 15:34:22 +0900 Subject: [petsc-users] DMCreateMatrix from one DMDA to an another In-Reply-To: References: Message-ID: Ho, I found that MatSetUp or something like MatMPIAIJSetPreallocation have to be used beofre the MatSetValuesStencil. And if MatMPIAIJSetPreallocation then accordingly MatSetType must also be set to MPIAIJ. That solves the above problem. With this, and using my own routine to set the preallocation, as suggested by Dave, I can recover the same behavior for my matrices as when I set them with DMCreateMatrix, but in the *square *case only (and by the way the whole allocation business seems to be faster in this case than using DMCreateMatrix). However I still have problems in the rectangular case. The parallel case does not work and in one processor it works and gives the expected result only if I use ADD_VALUES in MatSetValuesStencil, even though the entries are supposed to be empty ! In the multiprocessor case, I get the "argument out of range" error like this: 1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Argument out of range [1]PETSC ERROR: New nonzero at (3833,342512) caused a malloc Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015 [1]PETSC ERROR: ./miips on a arch-linux2-c-opt named helios91 by tnicolas Wed Dec 23 15:15:52 2015 [1]PETSC ERROR: Configure options --prefix=/csc/softs/anl/petsc-3.6.0/intel-15.0.0.090/bullxmpi-1.2.8.2/real --with-debugging=0 --with-x=0 --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpicxx --with-fortran --known-mpi-shared-libraries=1 --with-scalar-type=real --with-precision=double --CFLAGS="-g -O3 -mavx -mkl" --CXXFLAGS="-g -O3 -mavx -mkl" --FFLAGS="-g -O3 -mavx -mkl" [1]PETSC ERROR: #2265 MatSetValues_MPIAIJ() line 616 in /csc/releases/buildlog/anl/petsc-3.6.0/intel-15.0.0.090/bullxmpi-1.2.8.2/real/petsc-3.6.0/src/mat/impls/aij/mpi/mpiaij.c [1]PETSC ERROR: #2266 MatSetValues() line 1175 in /csc/releases/buildlog/anl/petsc-3.6.0/intel-15.0.0.090/bullxmpi-1.2.8.2/real/petsc-3.6.0/src/mat/interface/matrix.c [1]PETSC ERROR: #2267 MatSetValuesLocal() line 2023 in /csc/releases/buildlog/anl/petsc-3.6.0/intel-15.0.0.090/bullxmpi-1.2.8.2/real/petsc-3.6.0/src/mat/interface/matrix.c [1]PETSC ERROR: #2268 MatSetValuesStencil() line 1423 in /csc/releases/buildlog/anl/petsc-3.6.0/intel-15.0.0.090/bullxmpi-1.2.8.2/real/petsc-3.6.0/src/mat/interface/matrix.c This suggests my allocation is incorrect, however using the -info option, I see that [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 209088 X 348480; storage space: 20645544 unneeded,6170106 used [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 31 Which seems to be in contradiction with the above error. So I am thinking it is a different problem. I am wondering if the problem does not come from the fact that ideally the matrix should be constituted from 3x5 blocks, but this does not seem to be allowed yet : http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatMPIBAIJSetPreallocation.html : "*bs* - size of block, the blocks are ALWAYS square. One can use MatSetBlockSizes () to set a different row and column blocksize but the row blocksize always defines the size of the blocks. The column blocksize sets the blocksize of the vectors obtained with MatCreateVecs () " So I cannot call MatSetBlockSizes(A,3,5,ierr) and then MatMPIBAIJSetPreallocation because it would still assume square blocks (size 3) if I understand well. So I still don't see how to define a rectangular matrix which handily takes a vector from a DMDA and upon applying the operation, turns it to an other DMDA with same layout (except for number of DOF). Best Timothee 2015-12-23 6:05 GMT+09:00 Barry Smith : > > First run with valgrind and if that doesn't find anything then in the > debugger and check the data structures where it crashes. There is nothing > obviously wrong, some subtle mistake that can only be found with valgrind. > > Barry > > > On Dec 22, 2015, at 12:02 AM, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > > > Hi, > > > > I'd like to take advantage of the stencil structure of the DMDA to write > into my matrix. I have already written all my routines so that I can use > MatSetValuesStencil so it's most convenient for me, otherwise I will have > to rethink all the structure. > > > > Since I don't create my rectangular matrix with DMCreateMatrix but with > MatCreate and MatSetSizes instead, the manual tells me I have to call first > MatSetStencil and MatSetLocalToGlobalMapping before being allowed to call > MatSetValuesStencil. So before going on the full example, I tried to do > this on a simple square matrix but I can't get the thing to work. It must > be an obvious mistake but can't find it. My code is > > > > PetscErrorCode :: ierr > > DM :: da > > Mat :: A > > PetscInt :: xm > > ISLocalToGlobalMapping :: ltog > > PetscInt :: dims(1), starts(1) > > PetscInt :: dim > > MatStencil :: row(4,1), col(4,1) > > PetscScalar :: v > > > > call > DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,iten,ione,ione,PETSC_NULL_INTEGER,da,ierr) > > call > DMDAGetCorners(da,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, > & > > & xm,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr) > > > > call MatCreate(PETSC_COMM_WORLD,A,ierr) > > > > call MatSetSizes(A,xm,xm,PETSC_DETERMINE,PETSC_DETERMINE,ierr) > > > > call DMGetLocalToGlobalMapping(da,ltog,ierr) > > call MatSetLocalToGlobalMapping(A,ltog,ltog,ierr) > > call > DMDAGetGhostCorners(da,starts(1),PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, > & > > & > dims(1),PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr) > > dim=ione > > call MatSetStencil(A,dim,dims,starts,ione,ierr) > > > > row(MatStencil_i,1) = 0 > > col(MatStencil_i,1) = 0 > > > > call MatSetValuesStencil(A,ione,row,ione,col,zero,INSERT_VALUES,ierr) > > > > But I keep getting a segmentation fault error. > > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > > [0]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac > OS X to find memory corruption errors > > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [0]PETSC ERROR: to get more information on the crash. > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Signal received > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.6.1, Jul, 22, 2015 > > [0]PETSC ERROR: ./miips on a arch-darwin-c-debug named > iMac27Nicolas.nifs.ac.jp by timotheenicolas Tue Dec 22 14:57:46 2015 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-fblaslapack --download-mpich > --with-debugging=no > > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > [cli_0]: aborting job: > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > > > > =================================================================================== > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > = PID 9812 RUNNING AT iMac27Nicolas.nifs.ac.jp > > = EXIT CODE: 59 > > = CLEANING UP REMAINING PROCESSES > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > > =================================================================================== > > > > If I replace everything with call DMCreateMatrix(da,A,ierr) it works > fine. > > > > What information did I miss from the manual ? What am I doing wrong ? > > > > Best > > > > Timothee > > > > > > > > > > > > 2015-12-06 23:16 GMT+09:00 Timoth?e Nicolas >: > > Wow, Great answer, thanks, I should probably be able to do it like this, > I'll try my best ! > > > > I actually do want the assembled matrix now. I have coded the > matrix-free version already and it works fine, but I want to use multigrid > and I figured that in order to be be most efficient at the coarse level it > is best to use -ksp_type preonly -pc_type lu, which requires a true matrix > (the matrix I am talking about is the product between a matrix 3dof --> > 5dof with a matrix 5dof --> 3dof, which ends up square). But I want to > retain a memory light implementation so I use matrix-free formulation at > all the levels but the last, which is also light by definition, so I should > not suffer much in terms of memory. > > > > Thanks > > > > Timoth?e > > > > > > > > 2015-12-06 22:29 GMT+09:00 Dave May : > > > > > > On 6 December 2015 at 11:19, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > Hi, > > > > The answer to the first question is yes. I was puzzled about the second > part of the answer, but I realize I was not clear enough. I don't want to > just transfer information between 2 DMDAs, I want to perform an operation > on vectors with a rectangular matrix. > > > > Okay - I misunderstood. > > > > > > To be more explicit, the input vector of the operation is a vector with > 3 components (hence dof=3), and the results of the operator is put in one > vector equation + 2 scalar equation, which makes it dof = 5. > > > > So, either I cannot fill my matrix (if the number of rows exceeds what > is allowed by the DMDA), or I can fill my matrix but then the matrix and > vectors are not compatible, as in this error : > > > > > > Do you need to explicitly assemble this rectangular operator? > > Expressing this operator via a matrix-free representation would be > relatively straight forward. > > > > However, since the DMDA's have the same parallel layout you are in good > shape to define the rectangular matrix if you really want the assembled > thing. Your row space would be defined by the DMDA with block size 5 and > the column space would be defined by the DMDA with block size 3. For > example the matrix would be created like in the following way (assuming a > 2D DMDA) > > > > PetscInt m,n,M,N; > > > > > DMDAGetInfo(dm3,NULL,&M,&N,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL); > > nodes = M *N; > > DMDAGetCorners(dm3,NULL,NULL,NULL,&m,&n,NULL); > > lnodes = m * n; > > MatCreate(PETSC_COMM_WORLD,&A); > > MatSetSizes(A,lnodes*5,lnodes*3,nodes*5,nodes*3); > > > > The sparsity pattern is defined by the connectivity of the DMDA points > AND the different block sizes. > > To define the non-zero structure, you need to traverse the part of the > dmda defined by the natural indices given by si <= i < si+m, sj <= j < sj+n > and check whether the the 5x3 block associated with each i,j is living > within the diagonal or off-diagonal block of the matrix. For a given i,j > the off-diagonal is defined by any i',j' associated your stencil which is > not in the range si <= i' <= si + m, sj <= j' < sj + n. (e.g. it is a ghost > node). > > > > Your arrays counting the non-zero entries on the diagonal (nnz[]) and > off diag (nnz_od[]) can be indexed using i + j * m. This corresponds to the > indexing used by the DMDA. Your non-zero counting function would look > something like the pseudo code below > > > > PetscInt *nnz,*nnz_od; > > > > PetscMalloc(sizeof(PetscInt)*m*n*5,&nnz); > > PetscMalloc(sizeof(PetscInt)*m*n*5,&nnz_od); > > > > PetscMemzero(nnz,sizeof(PetscInt)*m*n*5); > > PetscMemzero(nnz_od,sizeof(PetscInt)*m*n*5); > > > > for (j=0; j > for (i=0; i > PetscInt idx,d,cnti,cntg; > > PetscBool is_ghost; > > > > idx = i + j * m; > > > > cnti = 0; > > cntg = 0; > > for (all_points_in_stencil) { /* User logic to define this loop */ > > > > is_ghost = PETSC_FALSE; > > /*User logic goes here to test if stencil point is a ghost point */ > > > > /* accumulate counts for the stencil */ > > if (is_ghost) { cntg++; } /* values living on a neighbour rank */ > > else { cnti++; } /* values local to current rank */ > > } > > > > /* assume all dofs in row and col space in block are coupled to each > other */ > > for (d=0; d<5; d++) { > > nnz[5*idx+d] = 3 * cnti; > > nnz_od[5*idx+d] = 3 * cntg; > > } > > > > } > > } > > > > Thanks, > > Dave > > > > > > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Nonconforming object sizes > > [0]PETSC ERROR: Mat mat,Vec y: global dim 4900 2940 > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.6.1, Jul, 22, 2015 > > [0]PETSC ERROR: ./miips on a arch-linux2-c-debug named Carl-9000 by > timothee Sun Dec 6 19:17:20 2015 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-fblaslapack --download-mpich > > [0]PETSC ERROR: #1 MatMult() line 2215 in > /home/timothee/Documents/petsc-3.6.1/src/mat/interface/matrix.c > > > > So I assume it means the matrix is assumed to be square. Is there a way > to change this ? > > > > Best > > > > Timoth?e > > > > > > > > 2015-12-05 20:41 GMT+09:00 Dave May : > > > > > > On 5 December 2015 at 11:15, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > Hi, > > > > I need to create a rectangular matrix which takes vector structured by a > DMDA and transforms them to vectors structured with an other DMDA. It does > not look like there is a routine to do this since apparently DMCreateMatrix > assumes square matrix. What is the clean way to do this ? > > > > Do the DMDA's have exactly the same parallel layout and only differ by > the number of DOFs? > > > > If yes, and you only want to move entries between vectors associated > with the two DMDA's, you can perform the transfer between vectors locally > (rank-wise) in a very simple manner using the local number of points from > DMDA and the known block sizes (DOFs) associated with your DMDA and the > size of the subspace you ultimately want to construct. > > > > Thanks, > > Dave > > > > > > The problem is I have a DMDA with 8 degrees of freedom (dof) and my > matrix sends vectors living in a subspace with 3 dof to the other 5 dof > subspace. I can define a square matrix living in the large, 8 dof subspace, > but this is of course very inefficient in terms of memory and time. > > > > Thanks > > > > Best > > > > Timoth?e > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Dec 23 09:35:44 2015 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 23 Dec 2015 09:35:44 -0600 Subject: [petsc-users] DMCreateMatrix from one DMDA to an another In-Reply-To: References: Message-ID: On Wed, Dec 23, 2015 at 12:34 AM, Timoth?e Nicolas < timothee.nicolas at gmail.com> wrote: > Ho, > > I found that MatSetUp or something like MatMPIAIJSetPreallocation have to > be used beofre the MatSetValuesStencil. And if MatMPIAIJSetPreallocation > then accordingly MatSetType must also be set to MPIAIJ. That solves the > above problem. With this, and using my own routine to set the > preallocation, as suggested by Dave, I can recover the same behavior for my > matrices as when I set them with DMCreateMatrix, but in the *square *case > only (and by the way the whole allocation business seems to be faster in > this case than using DMCreateMatrix). > > However I still have problems in the rectangular case. The parallel case > does not work and in one processor it works and gives the expected result > only if I use ADD_VALUES in MatSetValuesStencil, even though the entries > are supposed to be empty ! > > In the multiprocessor case, I get the "argument out of range" error like > this: > > 1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: Argument out of range > [1]PETSC ERROR: New nonzero at (3833,342512) caused a malloc > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn > off this check > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [1]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015 > [1]PETSC ERROR: ./miips on a arch-linux2-c-opt named helios91 by tnicolas > Wed Dec 23 15:15:52 2015 > [1]PETSC ERROR: Configure options > --prefix=/csc/softs/anl/petsc-3.6.0/intel-15.0.0.090/bullxmpi-1.2.8.2/real > --with-debugging=0 --with-x=0 --with-cc=mpicc --with-fc=mpif90 > --with-cxx=mpicxx --with-fortran --known-mpi-shared-libraries=1 > --with-scalar-type=real --with-precision=double --CFLAGS="-g -O3 -mavx > -mkl" --CXXFLAGS="-g -O3 -mavx -mkl" --FFLAGS="-g -O3 -mavx -mkl" > [1]PETSC ERROR: #2265 MatSetValues_MPIAIJ() line 616 in > /csc/releases/buildlog/anl/petsc-3.6.0/intel-15.0.0.090/bullxmpi-1.2.8.2/real/petsc-3.6.0/src/mat/impls/aij/mpi/mpiaij.c > [1]PETSC ERROR: #2266 MatSetValues() line 1175 in > /csc/releases/buildlog/anl/petsc-3.6.0/intel-15.0.0.090/bullxmpi-1.2.8.2/real/petsc-3.6.0/src/mat/interface/matrix.c > [1]PETSC ERROR: #2267 MatSetValuesLocal() line 2023 in > /csc/releases/buildlog/anl/petsc-3.6.0/intel-15.0.0.090/bullxmpi-1.2.8.2/real/petsc-3.6.0/src/mat/interface/matrix.c > [1]PETSC ERROR: #2268 MatSetValuesStencil() line 1423 in > /csc/releases/buildlog/anl/petsc-3.6.0/intel-15.0.0.090/bullxmpi-1.2.8.2/real/petsc-3.6.0/src/mat/interface/matrix.c > > This suggests my allocation is incorrect, however using the -info option, > I see that > > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 209088 X 348480; storage space: > 20645544 unneeded,6170106 used > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 31 > > Which seems to be in contradiction with the above error. So I am thinking > it is a different problem. > Make this problem simpler. That is the only way to tell what is going on. Do a 4x4 grid, so you have maybe 16 rows and can print everything out. Matt > I am wondering if the problem does not come from the fact that ideally the > matrix should be constituted from 3x5 blocks, but this does not seem to be > allowed yet : > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatMPIBAIJSetPreallocation.html > : > > "*bs* - size of block, the blocks are ALWAYS square. One can use > MatSetBlockSizes > () > to set a different row and column blocksize but the row blocksize always > defines the size of the blocks. The column blocksize sets the blocksize of > the vectors obtained with MatCreateVecs > () > " > > So I cannot call MatSetBlockSizes(A,3,5,ierr) > > and then MatMPIBAIJSetPreallocation because it would still assume square > blocks (size 3) if I understand well. So I still don't see how to define a > rectangular matrix which handily takes a vector from a DMDA and upon > applying the operation, turns it to an other DMDA with same layout (except > for number of DOF). > > Best > > Timothee > > > > 2015-12-23 6:05 GMT+09:00 Barry Smith : > >> >> First run with valgrind and if that doesn't find anything then in the >> debugger and check the data structures where it crashes. There is nothing >> obviously wrong, some subtle mistake that can only be found with valgrind. >> >> Barry >> >> > On Dec 22, 2015, at 12:02 AM, Timoth?e Nicolas < >> timothee.nicolas at gmail.com> wrote: >> > >> > Hi, >> > >> > I'd like to take advantage of the stencil structure of the DMDA to >> write into my matrix. I have already written all my routines so that I can >> use MatSetValuesStencil so it's most convenient for me, otherwise I will >> have to rethink all the structure. >> > >> > Since I don't create my rectangular matrix with DMCreateMatrix but with >> MatCreate and MatSetSizes instead, the manual tells me I have to call first >> MatSetStencil and MatSetLocalToGlobalMapping before being allowed to call >> MatSetValuesStencil. So before going on the full example, I tried to do >> this on a simple square matrix but I can't get the thing to work. It must >> be an obvious mistake but can't find it. My code is >> > >> > PetscErrorCode :: ierr >> > DM :: da >> > Mat :: A >> > PetscInt :: xm >> > ISLocalToGlobalMapping :: ltog >> > PetscInt :: dims(1), starts(1) >> > PetscInt :: dim >> > MatStencil :: row(4,1), col(4,1) >> > PetscScalar :: v >> > >> > call >> DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,iten,ione,ione,PETSC_NULL_INTEGER,da,ierr) >> > call >> DMDAGetCorners(da,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, >> & >> > & xm,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr) >> > >> > call MatCreate(PETSC_COMM_WORLD,A,ierr) >> > >> > call MatSetSizes(A,xm,xm,PETSC_DETERMINE,PETSC_DETERMINE,ierr) >> > >> > call DMGetLocalToGlobalMapping(da,ltog,ierr) >> > call MatSetLocalToGlobalMapping(A,ltog,ltog,ierr) >> > call >> DMDAGetGhostCorners(da,starts(1),PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, >> & >> > & >> dims(1),PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr) >> > dim=ione >> > call MatSetStencil(A,dim,dims,starts,ione,ierr) >> > >> > row(MatStencil_i,1) = 0 >> > col(MatStencil_i,1) = 0 >> > >> > call MatSetValuesStencil(A,ione,row,ione,col,zero,INSERT_VALUES,ierr) >> > >> > But I keep getting a segmentation fault error. >> > >> > [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> > [0]PETSC ERROR: Try option -start_in_debugger or >> -on_error_attach_debugger >> > [0]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >> OS X to find memory corruption errors >> > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, >> and run >> > [0]PETSC ERROR: to get more information on the crash. >> > [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> > [0]PETSC ERROR: Signal received >> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> > [0]PETSC ERROR: Petsc Release Version 3.6.1, Jul, 22, 2015 >> > [0]PETSC ERROR: ./miips on a arch-darwin-c-debug named >> iMac27Nicolas.nifs.ac.jp by timotheenicolas Tue Dec 22 14:57:46 2015 >> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --with-fc=gfortran --download-fblaslapack --download-mpich >> --with-debugging=no >> > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file >> > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >> > [cli_0]: aborting job: >> > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >> > >> > >> =================================================================================== >> > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> > = PID 9812 RUNNING AT iMac27Nicolas.nifs.ac.jp >> > = EXIT CODE: 59 >> > = CLEANING UP REMAINING PROCESSES >> > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> > >> =================================================================================== >> > >> > If I replace everything with call DMCreateMatrix(da,A,ierr) it works >> fine. >> > >> > What information did I miss from the manual ? What am I doing wrong ? >> > >> > Best >> > >> > Timothee >> > >> > >> > >> > >> > >> > 2015-12-06 23:16 GMT+09:00 Timoth?e Nicolas > >: >> > Wow, Great answer, thanks, I should probably be able to do it like >> this, I'll try my best ! >> > >> > I actually do want the assembled matrix now. I have coded the >> matrix-free version already and it works fine, but I want to use multigrid >> and I figured that in order to be be most efficient at the coarse level it >> is best to use -ksp_type preonly -pc_type lu, which requires a true matrix >> (the matrix I am talking about is the product between a matrix 3dof --> >> 5dof with a matrix 5dof --> 3dof, which ends up square). But I want to >> retain a memory light implementation so I use matrix-free formulation at >> all the levels but the last, which is also light by definition, so I should >> not suffer much in terms of memory. >> > >> > Thanks >> > >> > Timoth?e >> > >> > >> > >> > 2015-12-06 22:29 GMT+09:00 Dave May : >> > >> > >> > On 6 December 2015 at 11:19, Timoth?e Nicolas < >> timothee.nicolas at gmail.com> wrote: >> > Hi, >> > >> > The answer to the first question is yes. I was puzzled about the second >> part of the answer, but I realize I was not clear enough. I don't want to >> just transfer information between 2 DMDAs, I want to perform an operation >> on vectors with a rectangular matrix. >> > >> > Okay - I misunderstood. >> > >> > >> > To be more explicit, the input vector of the operation is a vector with >> 3 components (hence dof=3), and the results of the operator is put in one >> vector equation + 2 scalar equation, which makes it dof = 5. >> > >> > So, either I cannot fill my matrix (if the number of rows exceeds what >> is allowed by the DMDA), or I can fill my matrix but then the matrix and >> vectors are not compatible, as in this error : >> > >> > >> > Do you need to explicitly assemble this rectangular operator? >> > Expressing this operator via a matrix-free representation would be >> relatively straight forward. >> > >> > However, since the DMDA's have the same parallel layout you are in good >> shape to define the rectangular matrix if you really want the assembled >> thing. Your row space would be defined by the DMDA with block size 5 and >> the column space would be defined by the DMDA with block size 3. For >> example the matrix would be created like in the following way (assuming a >> 2D DMDA) >> > >> > PetscInt m,n,M,N; >> > >> > >> DMDAGetInfo(dm3,NULL,&M,&N,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL); >> > nodes = M *N; >> > DMDAGetCorners(dm3,NULL,NULL,NULL,&m,&n,NULL); >> > lnodes = m * n; >> > MatCreate(PETSC_COMM_WORLD,&A); >> > MatSetSizes(A,lnodes*5,lnodes*3,nodes*5,nodes*3); >> > >> > The sparsity pattern is defined by the connectivity of the DMDA points >> AND the different block sizes. >> > To define the non-zero structure, you need to traverse the part of the >> dmda defined by the natural indices given by si <= i < si+m, sj <= j < sj+n >> and check whether the the 5x3 block associated with each i,j is living >> within the diagonal or off-diagonal block of the matrix. For a given i,j >> the off-diagonal is defined by any i',j' associated your stencil which is >> not in the range si <= i' <= si + m, sj <= j' < sj + n. (e.g. it is a ghost >> node). >> > >> > Your arrays counting the non-zero entries on the diagonal (nnz[]) and >> off diag (nnz_od[]) can be indexed using i + j * m. This corresponds to the >> indexing used by the DMDA. Your non-zero counting function would look >> something like the pseudo code below >> > >> > PetscInt *nnz,*nnz_od; >> > >> > PetscMalloc(sizeof(PetscInt)*m*n*5,&nnz); >> > PetscMalloc(sizeof(PetscInt)*m*n*5,&nnz_od); >> > >> > PetscMemzero(nnz,sizeof(PetscInt)*m*n*5); >> > PetscMemzero(nnz_od,sizeof(PetscInt)*m*n*5); >> > >> > for (j=0; j> > for (i=0; i> > PetscInt idx,d,cnti,cntg; >> > PetscBool is_ghost; >> > >> > idx = i + j * m; >> > >> > cnti = 0; >> > cntg = 0; >> > for (all_points_in_stencil) { /* User logic to define this loop */ >> > >> > is_ghost = PETSC_FALSE; >> > /*User logic goes here to test if stencil point is a ghost point */ >> > >> > /* accumulate counts for the stencil */ >> > if (is_ghost) { cntg++; } /* values living on a neighbour rank */ >> > else { cnti++; } /* values local to current rank */ >> > } >> > >> > /* assume all dofs in row and col space in block are coupled to each >> other */ >> > for (d=0; d<5; d++) { >> > nnz[5*idx+d] = 3 * cnti; >> > nnz_od[5*idx+d] = 3 * cntg; >> > } >> > >> > } >> > } >> > >> > Thanks, >> > Dave >> > >> > >> > >> > [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> > [0]PETSC ERROR: Nonconforming object sizes >> > [0]PETSC ERROR: Mat mat,Vec y: global dim 4900 2940 >> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> > [0]PETSC ERROR: Petsc Release Version 3.6.1, Jul, 22, 2015 >> > [0]PETSC ERROR: ./miips on a arch-linux2-c-debug named Carl-9000 by >> timothee Sun Dec 6 19:17:20 2015 >> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --with-fc=gfortran --download-fblaslapack --download-mpich >> > [0]PETSC ERROR: #1 MatMult() line 2215 in >> /home/timothee/Documents/petsc-3.6.1/src/mat/interface/matrix.c >> > >> > So I assume it means the matrix is assumed to be square. Is there a way >> to change this ? >> > >> > Best >> > >> > Timoth?e >> > >> > >> > >> > 2015-12-05 20:41 GMT+09:00 Dave May : >> > >> > >> > On 5 December 2015 at 11:15, Timoth?e Nicolas < >> timothee.nicolas at gmail.com> wrote: >> > Hi, >> > >> > I need to create a rectangular matrix which takes vector structured by >> a DMDA and transforms them to vectors structured with an other DMDA. It >> does not look like there is a routine to do this since apparently >> DMCreateMatrix assumes square matrix. What is the clean way to do this ? >> > >> > Do the DMDA's have exactly the same parallel layout and only differ by >> the number of DOFs? >> > >> > If yes, and you only want to move entries between vectors associated >> with the two DMDA's, you can perform the transfer between vectors locally >> (rank-wise) in a very simple manner using the local number of points from >> DMDA and the known block sizes (DOFs) associated with your DMDA and the >> size of the subspace you ultimately want to construct. >> > >> > Thanks, >> > Dave >> > >> > >> > The problem is I have a DMDA with 8 degrees of freedom (dof) and my >> matrix sends vectors living in a subspace with 3 dof to the other 5 dof >> subspace. I can define a square matrix living in the large, 8 dof subspace, >> but this is of course very inefficient in terms of memory and time. >> > >> > Thanks >> > >> > Best >> > >> > Timoth?e >> > >> > >> > >> > >> > >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Wed Dec 23 20:45:17 2015 From: zonexo at gmail.com (TAY wee-beng) Date: Thu, 24 Dec 2015 10:45:17 +0800 Subject: [petsc-users] Removing unused warning during gnu compile In-Reply-To: <8B21BC6F-8ED0-4756-BA5A-4B382167BAE2@mcs.anl.gov> References: <5679FD6E.4070706@gmail.com> <8B21BC6F-8ED0-4756-BA5A-4B382167BAE2@mcs.anl.gov> Message-ID: <567B5C3D.5090003@gmail.com> Hi Barry, I got more but I only show some. Or is it because I use petsc.h instead of the individual *.h? Thank you Yours sincerely, TAY wee-beng On 23/12/2015 12:33 PM, Barry Smith wrote: > Are the warning messages below the only ones you get or do you get many more? > > Barry > >> On Dec 22, 2015, at 7:48 PM, TAY wee-beng wrote: >> >> Hi, >> >> When I compile using gfortran, I need it to warn me of any unused parameter in my code. However, I found that it also warns of unused parameter in PETSc code, as shown below. How can I let it only display warning for my own code, instead of PETSc? Thanks. >> >> wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petscsnes.h:116.54: >> Included at /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petsc.h:15: >> Included at fractional_initial.F90:28: >> >> integer(kind=selected_int_kind(5)) SNES_QN_LBFGS >> 1 >> Warning: Unused parameter 'snes_qn_lbfgs' declared at (1) >> /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petscsnes.h:144.64: >> Included at /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petsc.h:15: >> Included at fractional_initial.F90:28: >> >> integer(kind=selected_int_kind(5)) SNES_QN_RESTART_DEFAULT >> 1 >> Warning: Unused parameter 'snes_qn_restart_default' declared at (1) >> /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petscsnes.h:145.61: >> Included at /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petsc.h:15: >> Included at fractional_initial.F90:28: >> >> integer(kind=selected_int_kind(5)) SNES_QN_RESTART_NONE >> 1 >> Warning: Unused parameter 'snes_qn_restart_none' declared at (1) >> /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petscsnes.h:147.65: >> Included at /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petsc.h:15: >> Included at fractional_initial.F90:28: >> >> integer(kind=selected_int_kind(5)) SNES_QN_RESTART_PERIODIC >> 1 >> >> -- >> Thank you >> >> Yours sincerely, >> >> TAY wee-beng >> From bsmith at mcs.anl.gov Wed Dec 23 22:14:45 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 23 Dec 2015 22:14:45 -0600 Subject: [petsc-users] Removing unused warning during gnu compile In-Reply-To: <567B5C3D.5090003@gmail.com> References: <5679FD6E.4070706@gmail.com> <8B21BC6F-8ED0-4756-BA5A-4B382167BAE2@mcs.anl.gov> <567B5C3D.5090003@gmail.com> Message-ID: Maybe try the gfortran option listed here http://stackoverflow.com/questions/24493754/suppress-unused-variable-warning-in-gfortran-using-wextra-and-wall-fortran try passing --FFLAGS=-Wno-unused-parameter as an additional argument to ./configure Barry > On Dec 23, 2015, at 8:45 PM, TAY wee-beng wrote: > > Hi Barry, > > I got more but I only show some. Or is it because I use petsc.h instead of the individual *.h? > > Thank you > > Yours sincerely, > > TAY wee-beng > > On 23/12/2015 12:33 PM, Barry Smith wrote: >> Are the warning messages below the only ones you get or do you get many more? >> >> Barry >> >>> On Dec 22, 2015, at 7:48 PM, TAY wee-beng wrote: >>> >>> Hi, >>> >>> When I compile using gfortran, I need it to warn me of any unused parameter in my code. However, I found that it also warns of unused parameter in PETSc code, as shown below. How can I let it only display warning for my own code, instead of PETSc? Thanks. >>> >>> wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petscsnes.h:116.54: >>> Included at /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petsc.h:15: >>> Included at fractional_initial.F90:28: >>> >>> integer(kind=selected_int_kind(5)) SNES_QN_LBFGS >>> 1 >>> Warning: Unused parameter 'snes_qn_lbfgs' declared at (1) >>> /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petscsnes.h:144.64: >>> Included at /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petsc.h:15: >>> Included at fractional_initial.F90:28: >>> >>> integer(kind=selected_int_kind(5)) SNES_QN_RESTART_DEFAULT >>> 1 >>> Warning: Unused parameter 'snes_qn_restart_default' declared at (1) >>> /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petscsnes.h:145.61: >>> Included at /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petsc.h:15: >>> Included at fractional_initial.F90:28: >>> >>> integer(kind=selected_int_kind(5)) SNES_QN_RESTART_NONE >>> 1 >>> Warning: Unused parameter 'snes_qn_restart_none' declared at (1) >>> /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petscsnes.h:147.65: >>> Included at /home/wtay/Lib/petsc-3.6.2_shared_gnu_debug/include/petsc/finclude/petsc.h:15: >>> Included at fractional_initial.F90:28: >>> >>> integer(kind=selected_int_kind(5)) SNES_QN_RESTART_PERIODIC >>> 1 >>> >>> -- >>> Thank you >>> >>> Yours sincerely, >>> >>> TAY wee-beng >>> > From zonexo at gmail.com Thu Dec 24 10:14:41 2015 From: zonexo at gmail.com (TAY wee-beng) Date: Fri, 25 Dec 2015 00:14:41 +0800 Subject: [petsc-users] Code sometimes work, sometimes hang when increase cpu usage Message-ID: <567C19F1.6040307@gmail.com> Hi, I have this strange error. I converted my CFD code from a z directon only partition to the yz direction partition. The code works fine but when I increase the cpu no, strange things happen when solving the Poisson eqn. I increase cpu no from 24 to 40. Sometimes it works, sometimes it doesn't. When it doesn't, it just hangs there with no output, or it gives the error below: Using MPI_Barrier during debug shows that it hangs at call KSPSolve(ksp,b_rhs,xx,ierr). I use hypre BoomerAMG and GAMG (-poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg) Why is this so random? Also how do I debug this type of problem. [32]PETSC ERROR: ------------------------------------------------------------------------ [32]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [32]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [32]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [32]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [32]PETSC ERROR: likely location of problem given in stack below [32]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [32]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [32]PETSC ERROR: INSTEAD the line number of the start of the function [32]PETSC ERROR: is given. [32]PETSC ERROR: [32] HYPRE_SetupXXX line 174 /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/impls/hypre/hypre.c [32]PETSC ERROR: [32] PCSetUp_HYPRE line 122 /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/impls/hypre/hypre.c [32]PETSC ERROR: [32] PCSetUp line 945 /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/interface/precon.c [32]PETSC ERROR: [32] KSPSetUp line 247 /home/wtay/Codes/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c [32]PETSC ERROR: [32] KSPSolve line 510 /home/wtay/Codes/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c [32]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [32]PETSC ERROR: Signal received [32]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [32]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 [32]PETSC ERROR: ./a.out on a petsc-3.6.2_shared_gnu_debug named n12-40 by wtay Thu Dec 24 17:01:51 2015 [32]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --download-fblaslapack=1 --with-debugging=1 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.2_shared_gnu_debug --known-mpi-shared=1 --with-shared-libraries --with-fortran-interfaces=1 [32]PETSC ERROR: #1 User provided function() line 0 in unknown file -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 32 in communicator MPI_COMM_WORLD with errorcode 59. -- Thank you. Yours sincerely, TAY wee-beng From ajit.ndesai at gmail.com Thu Dec 24 10:16:40 2015 From: ajit.ndesai at gmail.com (Ajit Desai) Date: Thu, 24 Dec 2015 11:16:40 -0500 Subject: [petsc-users] Issue when calling multiple processors from PETSc Message-ID: Hello everyone, New user of PETSc and trying to get familiar with it for parallel computing applications. I wrote simple hello world code using PETSc environment and found a issue with the output. For instance : When I compile and execute "PETSc_helloWorld.F90" code with multiple processor. It prints following output. MacUser$ *mpiexec -np 4 ./PETSc_helloWorld* * Hello from PETSc World, rank: 0 of total 1 processes.* * Hello from PETSc World, rank: 0 of total 1 processes.* * Hello from PETSc World, rank: 0 of total 1 processes.* * Hello from PETSc World, rank: 0 of total 1 processes.* Similar code with MPI prints following output: ?MacUser? $ *mpiexec -np 4 ./a.out * * Hello from MPI World, rank: 0 * *? ?* * of total 4 processes.* * Hello from MPI World, rank: 1 * *? ?* * of total ? ?4 processes.* * Hello from MPI World, rank: 2 * *? ?* * of total 4 processes.* * Hello from MPI World, rank: 3 of total 4 processes. * ?I am not sure is this the issue with the PETSc installation ?in my machine or am I missing something here? I have attached both codes and *log_summary* of PETSc for your convenience. Thanks & Regards, *Ajit Desai* *--* * Carleton University * * Ottawa, Canada* -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./PETSc_helloWorld on a arch-darwin-c-opt named Ajits-MacBook-Pro.local with 1 processor, by ajit Thu Dec 24 11:13:35 2015 Using Petsc Release Version 3.6.3, Dec, 03, 2015 Max Max/Min Avg Total Time (sec): 1.687e-04 1.00000 1.687e-04 Objects: 1.000e+00 1.00000 1.000e+00 Flops: 0.000e+00 0.00000 0.000e+00 0.000e+00 Flops/sec: 0.000e+00 0.00000 0.000e+00 0.000e+00 MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 MPI Reductions: 0.000e+00 0.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.5859e-04 94.0% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Viewer 1 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 3.11e-08 #PETSc Option Table entries: -log_summary #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --with-debugging=no ----------------------------------------- Libraries compiled on Mon Dec 14 13:03:33 2015 on Ajits-MacBook-Pro.local Machine characteristics: Darwin-15.2.0-x86_64-i386-64bit Using PETSc directory: /Users/ajit/Downloads/SOFTWAREs/petsc-3.6.3 Using PETSc arch: arch-darwin-c-opt ----------------------------------------- Using C compiler: /Users/ajit/Downloads/SOFTWAREs/petsc-3.6.3/arch-darwin-c-opt/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /Users/ajit/Downloads/SOFTWAREs/petsc-3.6.3/arch-darwin-c-opt/bin/mpif90 -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/Users/ajit/Downloads/SOFTWAREs/petsc-3.6.3/arch-darwin-c-opt/include -I/Users/ajit/Downloads/SOFTWAREs/petsc-3.6.3/include -I/Users/ajit/Downloads/SOFTWAREs/petsc-3.6.3/include -I/Users/ajit/Downloads/SOFTWAREs/petsc-3.6.3/arch-darwin-c-opt/include -I/opt/X11/include -I/opt/local/include ----------------------------------------- Using C linker: /Users/ajit/Downloads/SOFTWAREs/petsc-3.6.3/arch-darwin-c-opt/bin/mpicc Using Fortran linker: /Users/ajit/Downloads/SOFTWAREs/petsc-3.6.3/arch-darwin-c-opt/bin/mpif90 Using libraries: -Wl,-rpath,/Users/ajit/Downloads/SOFTWAREs/petsc-3.6.3/arch-darwin-c-opt/lib -L/Users/ajit/Downloads/SOFTWAREs/petsc-3.6.3/arch-darwin-c-opt/lib -lpetsc -Wl,-rpath,/Users/ajit/Downloads/SOFTWAREs/petsc-3.6.3/arch-darwin-c-opt/lib -L/Users/ajit/Downloads/SOFTWAREs/petsc-3.6.3/arch-darwin-c-opt/lib -lflapack -lfblas -Wl,-rpath,/opt/X11/lib -L/opt/X11/lib -lX11 -Wl,-rpath,/opt/local/lib -L/opt/local/lib -lhwloc -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/7.0.2/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/7.0.2/lib/darwin -lmpifort -lgfortran -Wl,-rpath,/usr/local/gfortran/lib/gcc/x86_64-apple-darwin14/4.9.2 -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin14/4.9.2 -Wl,-rpath,/usr/local/gfortran/lib -L/usr/local/gfortran/lib -lgfortran -lgcc_ext.10.5 -lquadmath -lm -lclang_rt.osx -lmpicxx -lc++ -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.0.2/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.0.2/lib/darwin -lclang_rt.osx -Wl,-rpath,/Users/ajit/Downloads/SOFTWAREs/petsc-3.6.3/arch-darwin-c-opt/lib -L/Users/ajit/Downloads/SOFTWAREs/petsc-3.6.3/arch-darwin-c-opt/lib -ldl -lmpi -lpmpi -lSystem -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.0.2/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.0.2/lib/darwin -lclang_rt.osx -ldl ----------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: makefile Type: application/octet-stream Size: 434 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PETSc_helloWorld.F90 Type: application/octet-stream Size: 621 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: MPI_helloWorld.f90 Type: application/octet-stream Size: 571 bytes Desc: not available URL: From knepley at gmail.com Thu Dec 24 10:33:42 2015 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 24 Dec 2015 10:33:42 -0600 Subject: [petsc-users] Code sometimes work, sometimes hang when increase cpu usage In-Reply-To: <567C19F1.6040307@gmail.com> References: <567C19F1.6040307@gmail.com> Message-ID: It sounds like you have memory corruption in a different part of the code. Run in valgrind. Matt On Thu, Dec 24, 2015 at 10:14 AM, TAY wee-beng wrote: > Hi, > > I have this strange error. I converted my CFD code from a z directon only > partition to the yz direction partition. The code works fine but when I > increase the cpu no, strange things happen when solving the Poisson eqn. > > I increase cpu no from 24 to 40. > > Sometimes it works, sometimes it doesn't. When it doesn't, it just hangs > there with no output, or it gives the error below: > > Using MPI_Barrier during debug shows that it hangs at > > call KSPSolve(ksp,b_rhs,xx,ierr). > > I use hypre BoomerAMG and GAMG (-poisson_pc_gamg_agg_nsmooths 1 > -poisson_pc_type gamg) > > > Why is this so random? Also how do I debug this type of problem. > > > [32]PETSC ERROR: > ------------------------------------------------------------------------ > [32]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [32]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [32]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [32]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [32]PETSC ERROR: likely location of problem given in stack below > [32]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [32]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [32]PETSC ERROR: INSTEAD the line number of the start of the function > [32]PETSC ERROR: is given. > [32]PETSC ERROR: [32] HYPRE_SetupXXX line 174 > /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/impls/hypre/hypre.c > [32]PETSC ERROR: [32] PCSetUp_HYPRE line 122 > /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/impls/hypre/hypre.c > [32]PETSC ERROR: [32] PCSetUp line 945 > /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/interface/precon.c > [32]PETSC ERROR: [32] KSPSetUp line 247 > /home/wtay/Codes/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c > [32]PETSC ERROR: [32] KSPSolve line 510 > /home/wtay/Codes/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c > [32]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [32]PETSC ERROR: Signal received > [32]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [32]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 > [32]PETSC ERROR: ./a.out on a petsc-3.6.2_shared_gnu_debug named n12-40 by > wtay Thu Dec 24 17:01:51 2015 > [32]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ > --download-fblaslapack=1 --with-debugging=1 --download-hypre=1 > --prefix=/home/wtay/Lib/petsc-3.6.2_shared_gnu_debug --known-mpi-shared=1 > --with-shared-libraries --with-fortran-interfaces=1 > [32]PETSC ERROR: #1 User provided function() line 0 in unknown file > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 32 in communicator MPI_COMM_WORLD > with errorcode 59. > > -- > Thank you. > > Yours sincerely, > > TAY wee-beng > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Dec 24 10:37:43 2015 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 24 Dec 2015 10:37:43 -0600 Subject: [petsc-users] Issue when calling multiple processors from PETSc In-Reply-To: References: Message-ID: On Thu, Dec 24, 2015 at 10:16 AM, Ajit Desai wrote: > Hello everyone, > > New user of PETSc and trying to get familiar with it for parallel > computing applications. > I wrote simple hello world code using PETSc environment and found a issue > with the output. For instance : > > When I compile and execute "PETSc_helloWorld.F90" code with multiple > processor. It prints following output. > Almost certainly you are using an 'mpiexec' that does not match the MPI library that you built PETSc with. Did you use --download-mpich? Try this cd $PETSC_DIR make check That will test it with the correct mpiexec. Matt > MacUser$ *mpiexec -np 4 ./PETSc_helloWorld* > > * Hello from PETSc World, rank: 0 of total 1 processes.* > > * Hello from PETSc World, rank: 0 of total 1 processes.* > > * Hello from PETSc World, rank: 0 of total 1 processes.* > > * Hello from PETSc World, rank: 0 of total 1 processes.* > > > Similar code with MPI prints following output: > > ?MacUser? > $ > *mpiexec -np 4 ./a.out * > > * Hello from MPI World, rank: 0 * > *? ?* > * of total 4 processes.* > > * Hello from MPI World, rank: 1 * > *? ?* > * of total ? ?4 processes.* > > * Hello from MPI World, rank: 2 * > *? ?* > * of total 4 processes.* > > * Hello from MPI World, rank: 3 of total 4 processes. * > > ?I am not sure is this the issue with the PETSc installation ?in my > machine or am I missing something here? > I have attached both codes and *log_summary* of PETSc for your > convenience. > > Thanks & Regards, > > *Ajit Desai* > *--* > * Carleton University * > * Ottawa, Canada* > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From ajit.ndesai at gmail.com Thu Dec 24 10:47:17 2015 From: ajit.ndesai at gmail.com (Ajit Desai) Date: Thu, 24 Dec 2015 11:47:17 -0500 Subject: [petsc-users] Issue when calling multiple processors from PETSc In-Reply-To: References: Message-ID: Thanks for your quick reply Matt. I was using different MPIexec. The issue is now resolved. Regards, Ajit *Ajit Desai* *--* * PhD Scholar, Carleton University * * Ottawa, Canada* On Thu, Dec 24, 2015 at 11:37 AM, Matthew Knepley wrote: > On Thu, Dec 24, 2015 at 10:16 AM, Ajit Desai > wrote: > >> Hello everyone, >> >> New user of PETSc and trying to get familiar with it for parallel >> computing applications. >> I wrote simple hello world code using PETSc environment and found a issue >> with the output. For instance : >> >> When I compile and execute "PETSc_helloWorld.F90" code with multiple >> processor. It prints following output. >> > > Almost certainly you are using an 'mpiexec' that does not match the MPI > library that you built PETSc with. Did you > use --download-mpich? Try this > > cd $PETSC_DIR > make check > > That will test it with the correct mpiexec. > > Matt > > >> MacUser$ *mpiexec -np 4 ./PETSc_helloWorld* >> >> * Hello from PETSc World, rank: 0 of total 1 processes.* >> >> * Hello from PETSc World, rank: 0 of total 1 processes.* >> >> * Hello from PETSc World, rank: 0 of total 1 processes.* >> >> * Hello from PETSc World, rank: 0 of total 1 processes.* >> >> >> Similar code with MPI prints following output: >> >> ?MacUser? >> $ >> *mpiexec -np 4 ./a.out * >> >> * Hello from MPI World, rank: 0 * >> *? ?* >> * of total 4 processes.* >> >> * Hello from MPI World, rank: 1 * >> *? ?* >> * of total ? ?4 processes.* >> >> * Hello from MPI World, rank: 2 * >> *? ?* >> * of total 4 processes.* >> >> * Hello from MPI World, rank: 3 of total 4 processes. * >> >> ?I am not sure is this the issue with the PETSc installation ?in my >> machine or am I missing something here? >> I have attached both codes and *log_summary* of PETSc for your >> convenience. >> >> Thanks & Regards, >> >> *Ajit Desai* >> *--* >> * Carleton University * >> * Ottawa, Canada* >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Thu Dec 24 22:37:02 2015 From: zonexo at gmail.com (TAY wee-beng) Date: Fri, 25 Dec 2015 12:37:02 +0800 Subject: [petsc-users] Code sometimes work, sometimes hang when increase cpu usage In-Reply-To: References: <567C19F1.6040307@gmail.com> Message-ID: <567CC7EE.7070500@gmail.com> Hi, I tried valgrind in MPI but it aborts very early, with the error msg regarding PETSc initialize. I retry again, using a lower resolution. GAMG works, but BoomerAMG and hypre doesn't. Increasing cpu too high (80) also cause it to hang. 60 works fine. My grid size is 98x169x169 But when I increase the resolution, GAMG can't work again. I tried to increase the cpu no but it still doesn't work. Previously, using single z direction partition, it work using GAMG and hypre. So what could be the problem? Thank you. Yours sincerely, TAY wee-beng On 25/12/2015 12:33 AM, Matthew Knepley wrote: > It sounds like you have memory corruption in a different part of the > code. Run in valgrind. > > Matt > > On Thu, Dec 24, 2015 at 10:14 AM, TAY wee-beng > wrote: > > Hi, > > I have this strange error. I converted my CFD code from a z > directon only partition to the yz direction partition. The code > works fine but when I increase the cpu no, strange things happen > when solving the Poisson eqn. > > I increase cpu no from 24 to 40. > > Sometimes it works, sometimes it doesn't. When it doesn't, it just > hangs there with no output, or it gives the error below: > > Using MPI_Barrier during debug shows that it hangs at > > call KSPSolve(ksp,b_rhs,xx,ierr). > > I use hypre BoomerAMG and GAMG (-poisson_pc_gamg_agg_nsmooths 1 > -poisson_pc_type gamg) > > > Why is this so random? Also how do I debug this type of problem. > > > [32]PETSC ERROR: > ------------------------------------------------------------------------ > [32]PETSC ERROR: Caught signal number 11 SEGV: Segmentation > Violation, probably memory access out of range > [32]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > [32]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [32]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple > Mac OS X to find memory corruption errors > [32]PETSC ERROR: likely location of problem given in stack below > [32]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [32]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [32]PETSC ERROR: INSTEAD the line number of the start of the > function > [32]PETSC ERROR: is given. > [32]PETSC ERROR: [32] HYPRE_SetupXXX line 174 > /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/impls/hypre/hypre.c > [32]PETSC ERROR: [32] PCSetUp_HYPRE line 122 > /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/impls/hypre/hypre.c > [32]PETSC ERROR: [32] PCSetUp line 945 > /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/interface/precon.c > [32]PETSC ERROR: [32] KSPSetUp line 247 > /home/wtay/Codes/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c > [32]PETSC ERROR: [32] KSPSolve line 510 > /home/wtay/Codes/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c > [32]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [32]PETSC ERROR: Signal received > [32]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [32]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 > [32]PETSC ERROR: ./a.out on a petsc-3.6.2_shared_gnu_debug named > n12-40 by wtay Thu Dec 24 17:01:51 2015 > [32]PETSC ERROR: Configure options > --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --download-fblaslapack=1 > --with-debugging=1 --download-hypre=1 > --prefix=/home/wtay/Lib/petsc-3.6.2_shared_gnu_debug > --known-mpi-shared=1 --with-shared-libraries > --with-fortran-interfaces=1 > [32]PETSC ERROR: #1 User provided function() line 0 in unknown file > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 32 in communicator MPI_COMM_WORLD > with errorcode 59. > > -- > Thank you. > > Yours sincerely, > > TAY wee-beng > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Dec 24 22:42:36 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 24 Dec 2015 22:42:36 -0600 Subject: [petsc-users] Code sometimes work, sometimes hang when increase cpu usage In-Reply-To: <567CC7EE.7070500@gmail.com> References: <567C19F1.6040307@gmail.com> <567CC7EE.7070500@gmail.com> Message-ID: <6C14CF4D-029E-46B3-99FF-41DE39454F83@mcs.anl.gov> > On Dec 24, 2015, at 10:37 PM, TAY wee-beng wrote: > > Hi, > > I tried valgrind in MPI but it aborts very early, with the error msg regarding PETSc initialize. It shouldn't "abort" it should print some error message and continue. Please send all the output when running with valgrind. It is possible you are solving large enough problem that require configure --with-64-bit-indices . Does that resolve the problem? Barry > > I retry again, using a lower resolution. > > GAMG works, but BoomerAMG and hypre doesn't. Increasing cpu too high (80) also cause it to hang. 60 works fine. > > My grid size is 98x169x169 > > But when I increase the resolution, GAMG can't work again. > > I tried to increase the cpu no but it still doesn't work. > > Previously, using single z direction partition, it work using GAMG and hypre. So what could be the problem? > Thank you. > > Yours sincerely, > > TAY wee-beng > > On 25/12/2015 12:33 AM, Matthew Knepley wrote: >> It sounds like you have memory corruption in a different part of the code. Run in valgrind. >> >> Matt >> >> On Thu, Dec 24, 2015 at 10:14 AM, TAY wee-beng wrote: >> Hi, >> >> I have this strange error. I converted my CFD code from a z directon only partition to the yz direction partition. The code works fine but when I increase the cpu no, strange things happen when solving the Poisson eqn. >> >> I increase cpu no from 24 to 40. >> >> Sometimes it works, sometimes it doesn't. When it doesn't, it just hangs there with no output, or it gives the error below: >> >> Using MPI_Barrier during debug shows that it hangs at >> >> call KSPSolve(ksp,b_rhs,xx,ierr). >> >> I use hypre BoomerAMG and GAMG (-poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg) >> >> >> Why is this so random? Also how do I debug this type of problem. >> >> >> [32]PETSC ERROR: ------------------------------------------------------------------------ >> [32]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >> [32]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [32]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> [32]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >> [32]PETSC ERROR: likely location of problem given in stack below >> [32]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >> [32]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> [32]PETSC ERROR: INSTEAD the line number of the start of the function >> [32]PETSC ERROR: is given. >> [32]PETSC ERROR: [32] HYPRE_SetupXXX line 174 /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/impls/hypre/hypre.c >> [32]PETSC ERROR: [32] PCSetUp_HYPRE line 122 /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/impls/hypre/hypre.c >> [32]PETSC ERROR: [32] PCSetUp line 945 /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/interface/precon.c >> [32]PETSC ERROR: [32] KSPSetUp line 247 /home/wtay/Codes/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c >> [32]PETSC ERROR: [32] KSPSolve line 510 /home/wtay/Codes/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c >> [32]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [32]PETSC ERROR: Signal received >> [32]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [32]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 >> [32]PETSC ERROR: ./a.out on a petsc-3.6.2_shared_gnu_debug named n12-40 by wtay Thu Dec 24 17:01:51 2015 >> [32]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --download-fblaslapack=1 --with-debugging=1 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.2_shared_gnu_debug --known-mpi-shared=1 --with-shared-libraries --with-fortran-interfaces=1 >> [32]PETSC ERROR: #1 User provided function() line 0 in unknown file >> -------------------------------------------------------------------------- >> MPI_ABORT was invoked on rank 32 in communicator MPI_COMM_WORLD >> with errorcode 59. >> >> -- >> Thank you. >> >> Yours sincerely, >> >> TAY wee-beng >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > From zonexo at gmail.com Fri Dec 25 03:55:06 2015 From: zonexo at gmail.com (TAY wee-beng) Date: Fri, 25 Dec 2015 17:55:06 +0800 Subject: [petsc-users] Code sometimes work, sometimes hang when increase cpu usage In-Reply-To: <6C14CF4D-029E-46B3-99FF-41DE39454F83@mcs.anl.gov> References: <567C19F1.6040307@gmail.com> <567CC7EE.7070500@gmail.com> <6C14CF4D-029E-46B3-99FF-41DE39454F83@mcs.anl.gov> Message-ID: <567D127A.20307@gmail.com> Hi, I just realised that the nodes which I tested on may have some problems, as it has just been setup. So there's problem in the MPI communication. I now do my test on the old nodes. I have reduced my problem to a serial one. The code works fine with the debug version. But for the optimized version, there's different response for different compilers. For the intel built, it works pass time step 1, solving the poisson eqn. But it gives segmentation fault in the next time step. For gfortran, it hangs right at the start. I have attached the valgrind output. But there's some problems running valgrind with my code. My normal code runs like this: inal initial IIB_cell_no 24000 min I_cell_no 0 max I_cell_no 900 final initial I_cell_no 45000 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u) 24000 45000 24000 45000 IIB_I_cell_no_uvw_total1 8297 8251 8332 13965 13830 14089 ... solve Poisson eqn 1 0.01445783 0.26942225 0.33036381 1.15264402 -0.29464833E+03 -0.28723563E+02 0.27972784E+07 2 0.01445783 0.26942225 0.33036381 1.15264402 -0.29464833E+03 -0.28723563E+02 0.27972784E+07 ... For the ifort, it stops bet 1 and 2. For the gfortran, it hangs right at the start. Thank you. Yours sincerely, TAY wee-beng On 25/12/2015 12:42 PM, Barry Smith wrote: >> On Dec 24, 2015, at 10:37 PM, TAY wee-beng wrote: >> >> Hi, >> >> I tried valgrind in MPI but it aborts very early, with the error msg regarding PETSc initialize. > It shouldn't "abort" it should print some error message and continue. Please send all the output when running with valgrind. > > It is possible you are solving large enough problem that require configure --with-64-bit-indices . Does that resolve the problem? > > Barry > >> I retry again, using a lower resolution. >> >> GAMG works, but BoomerAMG and hypre doesn't. Increasing cpu too high (80) also cause it to hang. 60 works fine. >> >> My grid size is 98x169x169 >> >> But when I increase the resolution, GAMG can't work again. >> >> I tried to increase the cpu no but it still doesn't work. >> >> Previously, using single z direction partition, it work using GAMG and hypre. So what could be the problem? >> Thank you. >> >> Yours sincerely, >> >> TAY wee-beng >> >> On 25/12/2015 12:33 AM, Matthew Knepley wrote: >>> It sounds like you have memory corruption in a different part of the code. Run in valgrind. >>> >>> Matt >>> >>> On Thu, Dec 24, 2015 at 10:14 AM, TAY wee-beng wrote: >>> Hi, >>> >>> I have this strange error. I converted my CFD code from a z directon only partition to the yz direction partition. The code works fine but when I increase the cpu no, strange things happen when solving the Poisson eqn. >>> >>> I increase cpu no from 24 to 40. >>> >>> Sometimes it works, sometimes it doesn't. When it doesn't, it just hangs there with no output, or it gives the error below: >>> >>> Using MPI_Barrier during debug shows that it hangs at >>> >>> call KSPSolve(ksp,b_rhs,xx,ierr). >>> >>> I use hypre BoomerAMG and GAMG (-poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg) >>> >>> >>> Why is this so random? Also how do I debug this type of problem. >>> >>> >>> [32]PETSC ERROR: ------------------------------------------------------------------------ >>> [32]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>> [32]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>> [32]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>> [32]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>> [32]PETSC ERROR: likely location of problem given in stack below >>> [32]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>> [32]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>> [32]PETSC ERROR: INSTEAD the line number of the start of the function >>> [32]PETSC ERROR: is given. >>> [32]PETSC ERROR: [32] HYPRE_SetupXXX line 174 /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/impls/hypre/hypre.c >>> [32]PETSC ERROR: [32] PCSetUp_HYPRE line 122 /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/impls/hypre/hypre.c >>> [32]PETSC ERROR: [32] PCSetUp line 945 /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/interface/precon.c >>> [32]PETSC ERROR: [32] KSPSetUp line 247 /home/wtay/Codes/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c >>> [32]PETSC ERROR: [32] KSPSolve line 510 /home/wtay/Codes/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c >>> [32]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [32]PETSC ERROR: Signal received >>> [32]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [32]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 >>> [32]PETSC ERROR: ./a.out on a petsc-3.6.2_shared_gnu_debug named n12-40 by wtay Thu Dec 24 17:01:51 2015 >>> [32]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --download-fblaslapack=1 --with-debugging=1 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.2_shared_gnu_debug --known-mpi-shared=1 --with-shared-libraries --with-fortran-interfaces=1 >>> [32]PETSC ERROR: #1 User provided function() line 0 in unknown file >>> -------------------------------------------------------------------------- >>> MPI_ABORT was invoked on rank 32 in communicator MPI_COMM_WORLD >>> with errorcode 59. >>> >>> -- >>> Thank you. >>> >>> Yours sincerely, >>> >>> TAY wee-beng >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener -------------- next part -------------- [wtay at n12-72:1]$ valgrind ./a.out ==39723== Memcheck, a memory error detector ==39723== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al. ==39723== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info ==39723== Command: ./a.out ==39723== ==39723== Conditional jump or move depends on uninitialised value(s) ==39723== at 0x4018B16: index (in /usr/lib64/ld-2.17.so) ==39723== by 0x40079D3: expand_dynamic_string_token (in /usr/lib64/ld-2.17.so) ==39723== by 0x40084F5: _dl_map_object (in /usr/lib64/ld-2.17.so) ==39723== by 0x400160D: map_doit (in /usr/lib64/ld-2.17.so) ==39723== by 0x400F2F3: _dl_catch_error (in /usr/lib64/ld-2.17.so) ==39723== by 0x4000E8D: do_preload (in /usr/lib64/ld-2.17.so) ==39723== by 0x4004296: dl_main (in /usr/lib64/ld-2.17.so) ==39723== by 0x4016804: _dl_sysdep_start (in /usr/lib64/ld-2.17.so) ==39723== by 0x4004DA3: _dl_start (in /usr/lib64/ld-2.17.so) ==39723== by 0x4001427: ??? (in /usr/lib64/ld-2.17.so) ==39723== ==39723== Conditional jump or move depends on uninitialised value(s) ==39723== at 0x4018B1B: index (in /usr/lib64/ld-2.17.so) ==39723== by 0x40079D3: expand_dynamic_string_token (in /usr/lib64/ld-2.17.so) ==39723== by 0x40084F5: _dl_map_object (in /usr/lib64/ld-2.17.so) ==39723== by 0x400160D: map_doit (in /usr/lib64/ld-2.17.so) ==39723== by 0x400F2F3: _dl_catch_error (in /usr/lib64/ld-2.17.so) ==39723== by 0x4000E8D: do_preload (in /usr/lib64/ld-2.17.so) ==39723== by 0x4004296: dl_main (in /usr/lib64/ld-2.17.so) ==39723== by 0x4016804: _dl_sysdep_start (in /usr/lib64/ld-2.17.so) ==39723== by 0x4004DA3: _dl_start (in /usr/lib64/ld-2.17.so) ==39723== by 0x4001427: ??? (in /usr/lib64/ld-2.17.so) ==39723== ==39723== Conditional jump or move depends on uninitialised value(s) ==39723== at 0x4018B27: index (in /usr/lib64/ld-2.17.so) ==39723== by 0x40079D3: expand_dynamic_string_token (in /usr/lib64/ld-2.17.so) ==39723== by 0x40084F5: _dl_map_object (in /usr/lib64/ld-2.17.so) ==39723== by 0x400160D: map_doit (in /usr/lib64/ld-2.17.so) ==39723== by 0x400F2F3: _dl_catch_error (in /usr/lib64/ld-2.17.so) ==39723== by 0x4000E8D: do_preload (in /usr/lib64/ld-2.17.so) ==39723== by 0x4004296: dl_main (in /usr/lib64/ld-2.17.so) ==39723== by 0x4016804: _dl_sysdep_start (in /usr/lib64/ld-2.17.so) ==39723== by 0x4004DA3: _dl_start (in /usr/lib64/ld-2.17.so) ==39723== by 0x4001427: ??? (in /usr/lib64/ld-2.17.so) ==39723== vex amd64->IR: unhandled instruction bytes: 0x66 0xF 0x1B 0x4 0x24 0x66 0xF 0x1B ==39723== valgrind: Unrecognised instruction at address 0x40152b7. ==39723== at 0x40152B7: _dl_runtime_resolve (in /usr/lib64/ld-2.17.so) ==39723== by 0x7FF82F8: __exp_finite (in /usr/lib64/libm-2.17.so) ==39723== by 0x400B8BA: _dl_relocate_object (in /usr/lib64/ld-2.17.so) ==39723== by 0x4003AC9: dl_main (in /usr/lib64/ld-2.17.so) ==39723== by 0x4016804: _dl_sysdep_start (in /usr/lib64/ld-2.17.so) ==39723== by 0x4004DA3: _dl_start (in /usr/lib64/ld-2.17.so) ==39723== by 0x4001427: ??? (in /usr/lib64/ld-2.17.so) ==39723== Your program just tried to execute an instruction that Valgrind ==39723== did not recognise. There are two possible reasons for this. ==39723== 1. Your program has a bug and erroneously jumped to a non-code ==39723== location. If you are running Memcheck and you just saw a ==39723== warning about a bad jump, it's probably your program's fault. ==39723== 2. The instruction is legitimate but Valgrind doesn't handle it, ==39723== i.e. it's Valgrind's fault. If you think this is the case or ==39723== you are not sure, please let us know and we'll try to fix it. ==39723== Either way, Valgrind will now raise a SIGILL signal which will ==39723== probably kill your program. ==39723== ==39723== Process terminating with default action of signal 4 (SIGILL) ==39723== Illegal opcode at address 0x40152B7 ==39723== at 0x40152B7: _dl_runtime_resolve (in /usr/lib64/ld-2.17.so) ==39723== by 0x7FF82F8: __exp_finite (in /usr/lib64/libm-2.17.so) ==39723== by 0x400B8BA: _dl_relocate_object (in /usr/lib64/ld-2.17.so) ==39723== by 0x4003AC9: dl_main (in /usr/lib64/ld-2.17.so) ==39723== by 0x4016804: _dl_sysdep_start (in /usr/lib64/ld-2.17.so) ==39723== by 0x4004DA3: _dl_start (in /usr/lib64/ld-2.17.so) ==39723== by 0x4001427: ??? (in /usr/lib64/ld-2.17.so) ==39723== Jump to the invalid address stated on the next line ==39723== at 0x486: ??? ==39723== by 0x92968EF: ??? (in /usr/lib64/libc-2.17.so) ==39723== by 0x4003AC9: dl_main (in /usr/lib64/ld-2.17.so) ==39723== by 0x4016804: _dl_sysdep_start (in /usr/lib64/ld-2.17.so) ==39723== by 0x4004DA3: _dl_start (in /usr/lib64/ld-2.17.so) ==39723== by 0x4001427: ??? (in /usr/lib64/ld-2.17.so) ==39723== Address 0x486 is not stack'd, malloc'd or (recently) free'd ==39723== ==39723== ==39723== Process terminating with default action of signal 11 (SIGSEGV) ==39723== Bad permissions for mapped region at address 0x486 ==39723== at 0x486: ??? ==39723== by 0x92968EF: ??? (in /usr/lib64/libc-2.17.so) ==39723== by 0x4003AC9: dl_main (in /usr/lib64/ld-2.17.so) ==39723== by 0x4016804: _dl_sysdep_start (in /usr/lib64/ld-2.17.so) ==39723== by 0x4004DA3: _dl_start (in /usr/lib64/ld-2.17.so) ==39723== by 0x4001427: ??? (in /usr/lib64/ld-2.17.so) ==39723== ==39723== HEAP SUMMARY: ==39723== in use at exit: 0 bytes in 0 blocks ==39723== total heap usage: 0 allocs, 0 frees, 0 bytes allocated ==39723== ==39723== All heap blocks were freed -- no leaks are possible ==39723== ==39723== For counts of detected and suppressed errors, rerun with: -v ==39723== Use --track-origins=yes to see where uninitialised values come from ==39723== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 0 from 0) Segmentation fault (core dumped) From zonexo at gmail.com Fri Dec 25 05:41:24 2015 From: zonexo at gmail.com (TAY wee-beng) Date: Fri, 25 Dec 2015 19:41:24 +0800 Subject: [petsc-users] Code sometimes work, sometimes hang when increase cpu usage In-Reply-To: <6C14CF4D-029E-46B3-99FF-41DE39454F83@mcs.anl.gov> References: <567C19F1.6040307@gmail.com> <567CC7EE.7070500@gmail.com> <6C14CF4D-029E-46B3-99FF-41DE39454F83@mcs.anl.gov> Message-ID: <567D2B64.1030307@gmail.com> Hi, Sorry, there seems to be some problems with my valgrind. I have repeated it again, with the optimized and debug version Thank you. Yours sincerely, TAY wee-beng On 25/12/2015 12:42 PM, Barry Smith wrote: >> On Dec 24, 2015, at 10:37 PM, TAY wee-beng wrote: >> >> Hi, >> >> I tried valgrind in MPI but it aborts very early, with the error msg regarding PETSc initialize. > It shouldn't "abort" it should print some error message and continue. Please send all the output when running with valgrind. > > It is possible you are solving large enough problem that require configure --with-64-bit-indices . Does that resolve the problem? > > Barry > >> I retry again, using a lower resolution. >> >> GAMG works, but BoomerAMG and hypre doesn't. Increasing cpu too high (80) also cause it to hang. 60 works fine. >> >> My grid size is 98x169x169 >> >> But when I increase the resolution, GAMG can't work again. >> >> I tried to increase the cpu no but it still doesn't work. >> >> Previously, using single z direction partition, it work using GAMG and hypre. So what could be the problem? >> Thank you. >> >> Yours sincerely, >> >> TAY wee-beng >> >> On 25/12/2015 12:33 AM, Matthew Knepley wrote: >>> It sounds like you have memory corruption in a different part of the code. Run in valgrind. >>> >>> Matt >>> >>> On Thu, Dec 24, 2015 at 10:14 AM, TAY wee-beng wrote: >>> Hi, >>> >>> I have this strange error. I converted my CFD code from a z directon only partition to the yz direction partition. The code works fine but when I increase the cpu no, strange things happen when solving the Poisson eqn. >>> >>> I increase cpu no from 24 to 40. >>> >>> Sometimes it works, sometimes it doesn't. When it doesn't, it just hangs there with no output, or it gives the error below: >>> >>> Using MPI_Barrier during debug shows that it hangs at >>> >>> call KSPSolve(ksp,b_rhs,xx,ierr). >>> >>> I use hypre BoomerAMG and GAMG (-poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg) >>> >>> >>> Why is this so random? Also how do I debug this type of problem. >>> >>> >>> [32]PETSC ERROR: ------------------------------------------------------------------------ >>> [32]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>> [32]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>> [32]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>> [32]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>> [32]PETSC ERROR: likely location of problem given in stack below >>> [32]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>> [32]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>> [32]PETSC ERROR: INSTEAD the line number of the start of the function >>> [32]PETSC ERROR: is given. >>> [32]PETSC ERROR: [32] HYPRE_SetupXXX line 174 /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/impls/hypre/hypre.c >>> [32]PETSC ERROR: [32] PCSetUp_HYPRE line 122 /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/impls/hypre/hypre.c >>> [32]PETSC ERROR: [32] PCSetUp line 945 /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/interface/precon.c >>> [32]PETSC ERROR: [32] KSPSetUp line 247 /home/wtay/Codes/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c >>> [32]PETSC ERROR: [32] KSPSolve line 510 /home/wtay/Codes/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c >>> [32]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [32]PETSC ERROR: Signal received >>> [32]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [32]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 >>> [32]PETSC ERROR: ./a.out on a petsc-3.6.2_shared_gnu_debug named n12-40 by wtay Thu Dec 24 17:01:51 2015 >>> [32]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --download-fblaslapack=1 --with-debugging=1 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.2_shared_gnu_debug --known-mpi-shared=1 --with-shared-libraries --with-fortran-interfaces=1 >>> [32]PETSC ERROR: #1 User provided function() line 0 in unknown file >>> -------------------------------------------------------------------------- >>> MPI_ABORT was invoked on rank 32 in communicator MPI_COMM_WORLD >>> with errorcode 59. >>> >>> -- >>> Thank you. >>> >>> Yours sincerely, >>> >>> TAY wee-beng >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener -------------- next part -------------- [wtay at n12-04:4]$ !1005 -bash: !1005: event not found [wtay at n12-04:4]$ ~/Lib/valgrind-3.11.0/bin/valgrind --leak-check=yes ./a.out ==26133== Memcheck, a memory error detector ==26133== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==26133== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info ==26133== Command: ./a.out ==26133== **************************************************************************** * hwloc has encountered what looks like an error from the operating system. * * L3 (cpuset 0x000003f0) intersects with NUMANode (P#0 cpuset 0x0000003f) without inclusion! * Error occurred in topology.c line 942 * * The following FAQ entry in a recent hwloc documentation may help: * What should I do when hwloc reports "operating system" warnings? * Otherwise please report this error message to the hwloc user's mailing list, * along with the output+tarball generated by the hwloc-gather-topology script. **************************************************************************** 0.0000000000000000 0.59999999999999998 17.500000000000000 120.00000000000000 0.0000000000000000 1.0000000000000000 1.0000000000000000 0.40000000000000002 0 -400000 0 AB,AA,BB -2.4770000226562843 2.4110000195214525 2.4770000226562843 2.3460000164341182 size_x,size_y,size_z 76 130 136 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 0 1 130 1 136 1 1343680 ==26133== Warning: set address range perms: large range [0x4c96e040, 0x9693d704) (undefined) ==26133== Warning: set address range perms: large range [0x9693e040, 0xbb925ec4) (undefined) body_cg_ini 0.85000099999999779 9.9999999982738459E-007 6.9577187502060447 3104 surfaces with wrong vertex ordering Warning - length difference between element and cell max_element_length,min_element_length,min_delta 7.8475401769960568E-002 3.3499956100000006E-002 5.0000000000000003E-002 maximum ngh_surfaces and ngh_vertics are 77 35 minimum ngh_surfaces and ngh_vertics are 24 10 min IIB_cell_no 1104 max IIB_cell_no 1104 final initial IIB_cell_no 55200 min I_cell_no 677 max I_cell_no 677 final initial I_cell_no 33850 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u) 55200 33850 55200 33850 IIB_I_cell_no_uvw_total1 1104 1096 1080 677 659 637 1 0.02000000 0.29477817 0.35064783 1.20310939 -0.12558960E+03 -0.11164008E+02 0.13429373E+07 ==26133== Warning: set address range perms: large range [0x10dfa8040, 0x157f77714) (undefined) ==26133== Warning: set address range perms: large range [0x157f78040, 0x17cf5fed4) (undefined) 2 0.01334598 0.39731658 0.43604838 1.36260008 -0.14490188E+03 -0.76621036E+01 0.13428419E+07 ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x7DB89CE: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7BEA99: MAIN__ (ibm3d_high_Re.F90:4122) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x7DB6D27: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB7CCD: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB8F6D: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7BEA99: MAIN__ (ibm3d_high_Re.F90:4122) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x7DB7D83: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB8F6D: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7BEA99: MAIN__ (ibm3d_high_Re.F90:4122) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x7DB89CE: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7BEAAA: MAIN__ (ibm3d_high_Re.F90:4122) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x7DB7D83: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB8F6D: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7BEAAA: MAIN__ (ibm3d_high_Re.F90:4122) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x7DB89CE: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7BEABB: MAIN__ (ibm3d_high_Re.F90:4122) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x7DB7D83: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB8F6D: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7BEABB: MAIN__ (ibm3d_high_Re.F90:4122) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x7DB89CE: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7B97CA: MAIN__ (ibm3d_high_Re.F90:4445) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x7DB6D27: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB7CCD: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB8F6D: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7B97CA: MAIN__ (ibm3d_high_Re.F90:4445) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x7DB7D83: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB8F6D: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7B97CA: MAIN__ (ibm3d_high_Re.F90:4445) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x7DB89CE: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x7DB89D7: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92DED97: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92DEEBF: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D8B72: __mpn_extract_double (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DF441: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D8B77: __mpn_extract_double (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DF441: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92DF9B1: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92DFA08: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92DFA35: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92DFA40: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92DFA6F: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92DFA7E: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92DFE6C: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92DFBD4: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92DFCB5: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92DFE41: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92DFE23: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x7DB6D27: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB6DC9: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB870D: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x7DB710F: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB870D: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x7DB6F57: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB870D: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x7DB6F3B: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB870D: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x7DB6D27: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB6F9B: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB870D: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3F8E: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x7DB89CE: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x7DB89D7: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92DEFDB: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92DFC73: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92E0479: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7918: __mpn_lshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E047D: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D791B: __mpn_lshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E047D: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92E04B3: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92E04D8: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92E063F: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7B67: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7B6E: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7F60: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7FC2: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7B91: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7BDD: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7BE5: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7BEC: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7203: __mpn_add_n (in /usr/lib64/libc-2.17.so) ==26133== by 0x92D7BFF: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7BC7: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7BD7: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92E0543: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92E056A: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92E0577: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92E058C: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92E05AB: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92E05C3: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92E05DF: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92E05E5: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92E0785: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D79D5: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D79FC: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7A26: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7A79: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7A88: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7ABA: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92E04F8: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7EC4: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7EDA: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7EDF: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7EE1: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7F95: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7F9D: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7FC0: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7F60: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7F67: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7FC2: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7B91: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D72B8: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0xFFEFFC54F: ??? ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D72BB: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0xFFEFFC54F: ??? ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D72CE: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0xFFEFFC54F: ??? ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D72F8: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0xFFEFFC54F: ??? ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7301: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0xFFEFFC54F: ??? ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7365: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0xFFEFFC54F: ??? ==26133== by 0xFFEFFC63F: ??? ==26133== by 0x92D7BC6: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x50DBEF2: PetscMemzero (petscsys.h:2008) ==26133== by 0x13FFFFFF: ??? (in /opt/ud/openmpi-1.8.8/lib/openmpi/mca_osc_sm.so) ==26133== by 0x908512F: ??? (in /usr/lib64/libpthread-2.17.so) ==26133== by 0x50DBEF2: PetscMemzero (petscsys.h:2008) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7329: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0xFFEFFC54F: ??? ==26133== by 0xFFEFFC63F: ??? ==26133== by 0x92D7BC6: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x50DBEF2: PetscMemzero (petscsys.h:2008) ==26133== by 0x13FFFFFF: ??? (in /opt/ud/openmpi-1.8.8/lib/openmpi/mca_osc_sm.so) ==26133== by 0x908512F: ??? (in /usr/lib64/libpthread-2.17.so) ==26133== by 0x50DBEF2: PetscMemzero (petscsys.h:2008) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7330: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0xFFEFFC54F: ??? ==26133== by 0xFFEFFC63F: ??? ==26133== by 0x92D7BC6: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x50DBEF2: PetscMemzero (petscsys.h:2008) ==26133== by 0x13FFFFFF: ??? (in /opt/ud/openmpi-1.8.8/lib/openmpi/mca_osc_sm.so) ==26133== by 0x908512F: ??? (in /usr/lib64/libpthread-2.17.so) ==26133== by 0x50DBEF2: PetscMemzero (petscsys.h:2008) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7370: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0xFFEFFC54F: ??? ==26133== by 0xFFEFFC63F: ??? ==26133== by 0x92D7BC6: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x50DBEF2: PetscMemzero (petscsys.h:2008) ==26133== by 0x13FFFFFF: ??? (in /opt/ud/openmpi-1.8.8/lib/openmpi/mca_osc_sm.so) ==26133== by 0x908512F: ??? (in /usr/lib64/libpthread-2.17.so) ==26133== by 0x50DBEF2: PetscMemzero (petscsys.h:2008) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7BC7: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7BD7: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92E0603: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D79D7: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7AAA: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7F82: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7F8A: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D78CF: __mpn_lshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0BC1: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D78D1: __mpn_lshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0BC1: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D799A: __mpn_lshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0BC1: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D79AA: __mpn_lshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0BC1: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== ==26133== More than 100 errors detected. Subsequent errors ==26133== will still be recorded, but in less detail than before. ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7977: __mpn_lshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0BC1: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92E0BCE: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7EE6: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7F05: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7F77: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D79FE: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7A0B: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7A0D: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7A9A: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7A9E: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7A84: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92E06B0: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92E0667: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92E0671: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92E0676: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92E067C: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92E0803: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7EC4: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7EDA: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7EDF: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7EE1: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7EE6: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7F05: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7F77: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7FC0: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7F60: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7F67: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7FC2: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92E082C: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92E083B: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D79D5: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D79FC: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D79FE: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7A0B: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7A0D: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7A9A: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7A9E: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D7ABA: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7A84: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7A88: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92DFA4A: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92DFA63: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92DFA69: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C4061: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Syscall param write(buf) points to uninitialised byte(s) ==26133== at 0x90841FD: ??? (in /usr/lib64/libpthread-2.17.so) ==26133== by 0x7DB51C7: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB54BC: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB5597: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DBC03E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB0DDE: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB39B8: _gfortran_st_write_done (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C413B: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== Address 0xcca3d50 is 432 bytes inside a block of size 8,192 alloc'd ==26133== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==26133== by 0x7CE8914: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB516D: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB5B1C: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DAC6EF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DACFCB: _gfortran_st_open (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C3929: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D72D3: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0xFFEFFC54F: ??? ==26133== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==26133== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x4C41E6: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/4/a.out) ==26133== by 0x7B9800: MAIN__ (ibm3d_high_Re.F90:4447) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== Conditional jump or move depends on uninitialised value(s) ==26133== at 0x92D72E2: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0xFFEFFC54F: ??? ==26133== by 0xFFEFFC63F: ??? ==26133== by 0x92D7BC6: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x50DBEF2: PetscMemzero (petscsys.h:2008) ==26133== by 0x13FFFFFF: ??? (in /opt/ud/openmpi-1.8.8/lib/openmpi/mca_osc_sm.so) ==26133== by 0x908512F: ??? (in /usr/lib64/libpthread-2.17.so) ==26133== by 0x50DBEF2: PetscMemzero (petscsys.h:2008) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D72EB: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0xFFEFFC54F: ??? ==26133== by 0xFFEFFC63F: ??? ==26133== by 0x92D7BC6: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x50DBEF2: PetscMemzero (petscsys.h:2008) ==26133== by 0x13FFFFFF: ??? (in /opt/ud/openmpi-1.8.8/lib/openmpi/mca_osc_sm.so) ==26133== by 0x908512F: ??? (in /usr/lib64/libpthread-2.17.so) ==26133== by 0x50DBEF2: PetscMemzero (petscsys.h:2008) ==26133== ==26133== Use of uninitialised value of size 8 ==26133== at 0x92D7351: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==26133== by 0xFFEFFC54F: ??? ==26133== by 0xFFEFFC63F: ??? ==26133== by 0x92D7BC6: __mpn_mul (in /usr/lib64/libc-2.17.so) ==26133== by 0x50DBEF2: PetscMemzero (petscsys.h:2008) ==26133== by 0x13FFFFFF: ??? (in /opt/ud/openmpi-1.8.8/lib/openmpi/mca_osc_sm.so) ==26133== by 0x908512F: ??? (in /usr/lib64/libpthread-2.17.so) ==26133== by 0x50DBEF2: PetscMemzero (petscsys.h:2008) ==26133== ==26133== Warning: set address range perms: large range [0x157f78028, 0x17cf5feec) (noaccess) ==26133== Warning: set address range perms: large range [0x10dfa8028, 0x157f7772c) (noaccess) ==26133== Warning: set address range perms: large range [0x9693e028, 0xbb925edc) (noaccess) ==26133== Warning: set address range perms: large range [0x4c96e028, 0x9693d71c) (noaccess) ==26133== ==26133== HEAP SUMMARY: ==26133== in use at exit: 728,297 bytes in 9,515 blocks ==26133== total heap usage: 673,080 allocs, 663,565 frees, 7,638,652,437 bytes allocated ==26133== ==26133== 1 bytes in 1 blocks are definitely lost in loss record 1 of 124 ==26133== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==26133== by 0xE3DFA90: ??? ==26133== by 0x9929736: opal_db_base_store (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==26133== by 0x8BD02B2: ompi_modex_send_string (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==26133== by 0x8BCC74A: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==26133== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==26133== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==26133== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==26133== by 0x7A7582: MAIN__ (ibm3d_high_Re.F90:47) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== 7 bytes in 1 blocks are definitely lost in loss record 2 of 124 ==26133== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==26133== by 0x9318529: strdup (in /usr/lib64/libc-2.17.so) ==26133== by 0xE3DF9D5: ??? ==26133== by 0x9929736: opal_db_base_store (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==26133== by 0xD7BF47D: ??? ==26133== by 0x96692D7: orte_init (in /opt/ud/openmpi-1.8.8/lib/libopen-rte.so.7.0.6) ==26133== by 0x8BCC6BC: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==26133== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==26133== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==26133== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==26133== by 0x7A7582: MAIN__ (ibm3d_high_Re.F90:47) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== 24 bytes in 1 blocks are definitely lost in loss record 11 of 124 ==26133== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==26133== by 0xE3DFA90: ??? ==26133== by 0x9929736: opal_db_base_store (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==26133== by 0x8BD0189: ompi_modex_send (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==26133== by 0x122E2527: ??? ==26133== by 0x8C07C52: mca_btl_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==26133== by 0x1121E870: ??? ==26133== by 0x8C0741D: mca_bml_base_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==26133== by 0x12B9E9FE: ??? ==26133== by 0x8C17D08: mca_pml_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==26133== by 0x8BCC9BE: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==26133== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==26133== ==26133== 376 (232 direct, 144 indirect) bytes in 1 blocks are definitely lost in loss record 107 of 124 ==26133== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==26133== by 0x1121D984: ??? ==26133== by 0x12B9D648: ??? ==26133== by 0x8BCCD7D: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==26133== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==26133== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==26133== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==26133== by 0x7A7582: MAIN__ (ibm3d_high_Re.F90:47) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== 536 (24 direct, 512 indirect) bytes in 1 blocks are definitely lost in loss record 111 of 124 ==26133== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==26133== by 0x7CE8914: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DBBECA: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB46B9: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7CE5E87: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x400F4E2: _dl_init (in /usr/lib64/ld-2.17.so) ==26133== by 0x4001459: ??? (in /usr/lib64/ld-2.17.so) ==26133== ==26133== 536 (24 direct, 512 indirect) bytes in 1 blocks are definitely lost in loss record 112 of 124 ==26133== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==26133== by 0x7CE8914: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DBBECA: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7DB4791: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x7CE5E87: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==26133== by 0x400F4E2: _dl_init (in /usr/lib64/ld-2.17.so) ==26133== by 0x4001459: ??? (in /usr/lib64/ld-2.17.so) ==26133== ==26133== 1,021 bytes in 21 blocks are definitely lost in loss record 116 of 124 ==26133== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==26133== by 0x9306E07: vasprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3DD6: asprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9929F77: dlopen_foreachfile (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==26133== by 0x990DAFC: mca_base_component_find (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==26133== by 0x9917362: mca_base_framework_components_register (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==26133== by 0x99177EB: mca_base_framework_register (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==26133== by 0x9917841: mca_base_framework_open (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==26133== by 0x98F7B48: opal_init (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==26133== by 0x96691A4: orte_init (in /opt/ud/openmpi-1.8.8/lib/libopen-rte.so.7.0.6) ==26133== by 0x8BCC6BC: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==26133== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==26133== ==26133== 4,224 (1,152 direct, 3,072 indirect) bytes in 48 blocks are definitely lost in loss record 119 of 124 ==26133== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==26133== by 0x994078A: opal_hwloc191_hwloc_bitmap_alloc (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==26133== by 0x9952649: look_sysfscpu (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==26133== by 0x9954569: hwloc_look_linuxfs (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==26133== by 0x99693BA: opal_hwloc191_hwloc_topology_load (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==26133== by 0x993CA9D: opal_hwloc_base_get_topology (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==26133== by 0x8BCCABC: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==26133== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==26133== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==26133== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==26133== by 0x7A7582: MAIN__ (ibm3d_high_Re.F90:47) ==26133== by 0x408F2E: main (ibm3d_high_Re.F90:3) ==26133== ==26133== 5,004 bytes in 93 blocks are definitely lost in loss record 120 of 124 ==26133== at 0x4C2BB8A: realloc (vg_replace_malloc.c:785) ==26133== by 0x9306DDA: vasprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x92E3DD6: asprintf (in /usr/lib64/libc-2.17.so) ==26133== by 0x9929F77: dlopen_foreachfile (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==26133== by 0x990DAFC: mca_base_component_find (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==26133== by 0x9917362: mca_base_framework_components_register (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==26133== by 0x99177EB: mca_base_framework_register (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==26133== by 0x9917841: mca_base_framework_open (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==26133== by 0x98F7B48: opal_init (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==26133== by 0x96691A4: orte_init (in /opt/ud/openmpi-1.8.8/lib/libopen-rte.so.7.0.6) ==26133== by 0x8BCC6BC: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==26133== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==26133== ==26133== LEAK SUMMARY: ==26133== definitely lost: 7,489 bytes in 168 blocks ==26133== indirectly lost: 4,240 bytes in 53 blocks ==26133== possibly lost: 0 bytes in 0 blocks ==26133== still reachable: 716,568 bytes in 9,294 blocks ==26133== suppressed: 0 bytes in 0 blocks ==26133== Reachable blocks (those to which a pointer was found) are not shown. ==26133== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==26133== ==26133== For counts of detected and suppressed errors, rerun with: -v ==26133== Use --track-origins=yes to see where uninitialised values come from ==26133== ERROR SUMMARY: 240758 errors from 157 contexts (suppressed: 2 from 2) [wtay at n12-04:4]$ -------------- next part -------------- [wtay at hpc12:1]$ ~/Lib/valgrind-3.11.0/bin/valgrind --leak-check=yes ./a.out ==20772== Memcheck, a memory error detector ==20772== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==20772== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info ==20772== Command: ./a.out ==20772== 0.0000000000000000 0.59999999999999998 17.500000000000000 120.00000000000000 0.0000000000000000 1.0000000000000000 1.0000000000000000 0.40000000000000002 0 -400000 0 AB,AA,BB -2.4770000226562843 2.4110000195214525 2.4770000226562843 2.3460000164341182 size_x,size_y,size_z 76 130 136 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 0 1 130 1 136 1 1343680 ==20772== Warning: set address range perms: large range [0x4ddee040, 0x97dbd704) (undefined) ==20772== Warning: set address range perms: large range [0x97dbe040, 0xbcda5ec4) (undefined) ^C==20772== ==20772== Process terminating with default action of signal 2 (SIGINT) ==20772== at 0x937DB7D: ??? (in /usr/lib64/libc-2.17.so) ==20772== by 0x9936905: poll_dispatch (poll.c:165) ==20772== by 0x992E723: opal_libevent2021_event_base_loop (event.c:1633) ==20772== by 0x968B24D: orte_progress_thread_engine (in /opt/ud/openmpi-1.8.8/lib/libopen-rte.so.7.0.6) ==20772== by 0x907DDF4: start_thread (in /usr/lib64/libpthread-2.17.so) ==20772== by 0x93881AC: clone (in /usr/lib64/libc-2.17.so) ==20772== ==20772== HEAP SUMMARY: ==20772== in use at exit: 2,771,247,738 bytes in 17,476 blocks ==20772== total heap usage: 35,685 allocs, 18,209 frees, 3,723,954,797 bytes allocated ==20772== ^C^C^C^C^C^C^C^C^C^C==20772== 336 bytes in 1 blocks are possibly lost in loss record 6,282 of 7,216 ==20772== at 0x4C2B9B4: calloc (vg_replace_malloc.c:711) ==20772== by 0x4011E84: _dl_allocate_tls (in /usr/lib64/ld-2.17.so) ==20772== by 0x907E990: pthread_create@@GLIBC_2.2.5 (in /usr/lib64/libpthread-2.17.so) ==20772== by 0x98FB7D5: opal_thread_start (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==20772== by 0x968B719: orte_ess_base_app_setup (in /opt/ud/openmpi-1.8.8/lib/libopen-rte.so.7.0.6) ==20772== by 0xD7BF378: rte_init (in /opt/ud/openmpi-1.8.8/lib/openmpi/mca_ess_singleton.so) ==20772== by 0x96692D7: orte_init (in /opt/ud/openmpi-1.8.8/lib/libopen-rte.so.7.0.6) ==20772== by 0x8BCC6BC: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==20772== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==20772== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==20772== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==20772== by 0x8A5CBF: MAIN__ (ibm3d_high_Re.F90:47) ==20772== ==20772== 1,021 bytes in 21 blocks are definitely lost in loss record 6,395 of 7,216 ==20772== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==20772== by 0x9306E07: vasprintf (in /usr/lib64/libc-2.17.so) ==20772== by 0x92E3DD6: asprintf (in /usr/lib64/libc-2.17.so) ==20772== by 0x9929F77: dlopen_foreachfile (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==20772== by 0x990DAFC: mca_base_component_find (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==20772== by 0x9917362: mca_base_framework_components_register (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==20772== by 0x99177EB: mca_base_framework_register (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==20772== by 0x9917841: mca_base_framework_open (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==20772== by 0x98F7B48: opal_init (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==20772== by 0x96691A4: orte_init (in /opt/ud/openmpi-1.8.8/lib/libopen-rte.so.7.0.6) ==20772== by 0x8BCC6BC: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==20772== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==20772== ==20772== 5,004 bytes in 93 blocks are definitely lost in loss record 7,112 of 7,216 ==20772== at 0x4C2BB8A: realloc (vg_replace_malloc.c:785) ==20772== by 0x9306DDA: vasprintf (in /usr/lib64/libc-2.17.so) ==20772== by 0x92E3DD6: asprintf (in /usr/lib64/libc-2.17.so) ==20772== by 0x9929F77: dlopen_foreachfile (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==20772== by 0x990DAFC: mca_base_component_find (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==20772== by 0x9917362: mca_base_framework_components_register (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==20772== by 0x99177EB: mca_base_framework_register (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==20772== by 0x9917841: mca_base_framework_open (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==20772== by 0x98F7B48: opal_init (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==20772== by 0x96691A4: orte_init (in /opt/ud/openmpi-1.8.8/lib/libopen-rte.so.7.0.6) ==20772== by 0x8BCC6BC: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==20772== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==20772== ==20772== LEAK SUMMARY: ==20772== definitely lost: 6,025 bytes in 114 blocks ==20772== indirectly lost: 0 bytes in 0 blocks ==20772== possibly lost: 336 bytes in 1 blocks ==20772== still reachable: 2,771,241,377 bytes in 17,361 blocks ==20772== suppressed: 0 bytes in 0 blocks ==20772== Reachable blocks (those to which a pointer was found) are not shown. ==20772== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==20772== ==20772== For counts of detected and suppressed errors, rerun with: -v ==20772== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 2 from 2) Killed [wtay at hpc12:1]$ qsub -I -X -l nodes=1:ppn=32:typef qsub: waiting for job 7354.hpc12.hpc to start qsub: job 7354.hpc12.hpc ready Welcome ------------------------------------------------------------------------ - For support contact beheer-o-linux-ict at tudelft.nl - Documentation at /opt/ud/LOCAL/doc/LOCALdoc.txt ------------------------------------------------------------------------ System state: 11:50:53 up 56 days, 21:37, 0 users, load average: 0.00, 0.01, 0.19 Filesystem Size Used Avail Use% Mounted on hpc12:/export/disk1/home1/wtay 30T 28T 2.5T 92% /home/wtay WARNING: Total disk usage > 90% [wtay at n12-72:~]$ cd Results/1 [wtay at n12-72:1]$ LD_PRELOAD=/home/wtay/Lib/valgrind-3.11.0/lib/valgrind/libmpiwrap-amd64-linux.so [wtay at n12-72:1]$ ~/Lib/valgrind-3.11.0/bin/valgrind --leak-check=yes ./a.out ==41023== Memcheck, a memory error detector ==41023== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==41023== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info ==41023== Command: ./a.out ==41023== ==41023== Syscall param write(buf) points to uninitialised byte(s) ==41023== at 0x90841FD: ??? (in /usr/lib64/libpthread-2.17.so) ==41023== by 0x1142C51A: mca_btl_openib_add_procs (in /opt/ud/openmpi-1.8.8/lib/openmpi/mca_btl_openib.so) ==41023== by 0x1121D484: mca_bml_r2_add_procs (in /opt/ud/openmpi-1.8.8/lib/openmpi/mca_bml_r2.so) ==41023== by 0x12B9D648: mca_pml_ob1_add_procs (in /opt/ud/openmpi-1.8.8/lib/openmpi/mca_pml_ob1.so) ==41023== by 0x8BCCD7D: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==41023== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==41023== by 0x8A5CBF: MAIN__ (ibm3d_high_Re.F90:47) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== Address 0xffeffea48 is on thread 1's stack ==41023== in frame #1, created by mca_btl_openib_add_procs (???:) ==41023== 0.0000000000000000 0.59999999999999998 17.500000000000000 120.00000000000000 0.0000000000000000 1.0000000000000000 1.0000000000000000 0.40000000000000002 0 -400000 0 AB,AA,BB -2.4770000226562843 2.4110000195214525 2.4770000226562843 2.3460000164341182 size_x,size_y,size_z 76 130 136 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 0 1 130 1 136 1 1343680 ==41023== Warning: set address range perms: large range [0x4ddee040, 0x97dbd704) (undefined) ==41023== Warning: set address range perms: large range [0x97dbe040, 0xbcda5ec4) (undefined) body_cg_ini 0.85000099999999779 9.9999999982738459E-007 6.9577187502060447 3104 surfaces with wrong vertex ordering Warning - length difference between element and cell max_element_length,min_element_length,min_delta 7.8475401769960568E-002 3.3499956100000006E-002 5.0000000000000003E-002 maximum ngh_surfaces and ngh_vertics are 77 35 minimum ngh_surfaces and ngh_vertics are 24 10 min IIB_cell_no 1104 max IIB_cell_no 1104 final initial IIB_cell_no 55200 min I_cell_no 677 max I_cell_no 677 final initial I_cell_no 33850 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u) 55200 33850 55200 33850 IIB_I_cell_no_uvw_total1 1104 1096 1080 677 659 637 1 0.02000000 0.29477817 0.35064783 1.20310939 -0.12558960E+03 -0.11164008E+02 0.13429373E+07 ==41023== Warning: set address range perms: large range [0x10f828040, 0x1597f7714) (undefined) ==41023== Warning: set address range perms: large range [0x1597f8040, 0x17e7dfed4) (undefined) 2 0.01334598 0.39731658 0.43604838 1.36260008 -0.14490188E+03 -0.76621036E+01 0.13428419E+07 ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB89CE: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB89D7: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DED97: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DEEBF: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D8B72: __mpn_extract_double (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DF441: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D8B77: __mpn_extract_double (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DF441: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DEFDB: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DFC73: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92E0479: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7918: __mpn_lshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E047D: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D791B: __mpn_lshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E047D: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92E04B3: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92E04D8: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92E063F: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7B67: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7B6E: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7F60: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7FC2: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7B91: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7BDD: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7BE5: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7BEC: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7203: __mpn_add_n (in /usr/lib64/libc-2.17.so) ==41023== by 0x92D7BFF: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7BC7: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7BD7: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0512: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92E0543: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92E056A: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92E0577: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92E058C: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92E05AB: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92E05C3: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92E05DF: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92E05E5: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92E0785: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D79D5: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D79FC: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7A26: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7A79: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7A88: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7ABA: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92E04F8: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7EC4: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7EDA: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7EDF: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7EE1: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7F95: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7F9D: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7FC0: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7F60: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7F67: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7FC2: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7B91: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D72B8: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0xFFEFFC38F: ??? ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D72BB: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0xFFEFFC38F: ??? ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D72CE: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0xFFEFFC38F: ??? ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D72F8: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0xFFEFFC38F: ??? ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7301: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0xFFEFFC38F: ??? ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7365: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0xFFEFFC38F: ??? ==41023== by 0xFFEFFC47F: ??? ==41023== by 0x92D7BC6: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x908512F: ??? (in /usr/lib64/libpthread-2.17.so) ==41023== by 0x50DBEF2: PetscMemzero (petscsys.h:2008) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7329: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0xFFEFFC38F: ??? ==41023== by 0xFFEFFC47F: ??? ==41023== by 0x92D7BC6: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x908512F: ??? (in /usr/lib64/libpthread-2.17.so) ==41023== by 0x50DBEF2: PetscMemzero (petscsys.h:2008) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7330: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0xFFEFFC38F: ??? ==41023== by 0xFFEFFC47F: ??? ==41023== by 0x92D7BC6: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x908512F: ??? (in /usr/lib64/libpthread-2.17.so) ==41023== by 0x50DBEF2: PetscMemzero (petscsys.h:2008) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7370: __mpn_addmul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0xFFEFFC38F: ??? ==41023== by 0xFFEFFC47F: ??? ==41023== by 0x92D7BC6: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x908512F: ??? (in /usr/lib64/libpthread-2.17.so) ==41023== by 0x50DBEF2: PetscMemzero (petscsys.h:2008) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7BC7: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7BD7: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92E0603: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D78CF: __mpn_lshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0BC1: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D78D1: __mpn_lshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0BC1: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D799A: __mpn_lshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0BC1: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D79AA: __mpn_lshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0BC1: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7977: __mpn_lshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0BC1: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92E0BCE: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7EE6: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7F05: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7F77: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D79FE: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7A0B: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7A0D: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7A9A: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7A9E: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7A84: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DFA4A: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DFA63: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92DFA69: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DFA6F: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DFBD4: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DFCB5: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB6D27: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB6DC9: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB870D: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB710F: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB870D: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB6D27: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB6F9B: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB870D: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54A4: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB89CE: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54C3: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB89D7: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54C3: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB710F: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB870D: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54C3: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB89CE: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB89D7: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D79D7: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7AAA: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7F82: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7F8A: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92E0803: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7EC4: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== ==41023== More than 100 errors detected. Subsequent errors ==41023== will still be recorded, but in less detail than before. ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7EDA: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7EDF: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7EE1: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7EE6: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7F05: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7F77: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7FC0: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7F60: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7F67: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7FC2: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92E082C: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92E083B: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D79D5: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D79FC: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D79FE: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7A0B: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7A0D: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7A9A: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7A9E: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7ABA: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7A84: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7A88: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E54E2: MAIN__ (ibm3d_high_Re.F90:4106) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB89CE: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB89D7: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB8AC3: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB8AD1: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7F4F: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92D7B80: __mpn_mul (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E06A7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7A28: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7A39: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7A4D: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0619: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92DEAD9: hack_digit.13549 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DF8D7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7EC4: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DEAE0: hack_digit.13549 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DF8D7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7EDA: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DEAE0: hack_digit.13549 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DF8D7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7EDF: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DEAE0: hack_digit.13549 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DF8D7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7EE1: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DEAE0: hack_digit.13549 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DF8D7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D7EE6: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DEAE0: hack_digit.13549 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DF8D7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7EE8: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DEAE0: hack_digit.13549 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DF8D7: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92DEAD9: hack_digit.13549 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DF9AA: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DF9B1: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DFA08: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92DEAD9: hack_digit.13549 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DFA25: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DFA35: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DFA40: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DFE6C: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DFE9E: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DFEBE: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DFEE5: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DFEFC: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DFE41: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DFE23: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB9DC8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB710F: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB9DFB: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB6EE2: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB9DFB: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB6F57: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB9DFB: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB6F3B: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB9DFB: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DB7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB75CC: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB9DFB: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D928B: _itoa_word (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DBCA0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB75CC: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB9DFB: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92D9295: _itoa_word (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DBCA0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB75CC: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB9DFB: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DBCE8: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB75CC: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB9DFB: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DB8AB: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB75CC: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB9DFB: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DB92E: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB75CC: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB9DFB: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB8ADF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB8AE8: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB8B7B: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB8BCD: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7F4F: __mpn_mul_1 (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E081E: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB8C3B: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7A28: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB8C3B: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Use of uninitialised value of size 8 ==41023== at 0x92D7A4D: __mpn_rshift (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E0BA4: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB8C3B: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB710F: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB8C72: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB6F57: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB8C72: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB6F3B: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB8C72: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBB1A4: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBC8E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAFEE6: _gfortran_transfer_array (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x8E866D: MAIN__ (ibm3d_high_Re.F90:4445) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x92DFA7E: __printf_fp (in /usr/lib64/libc-2.17.so) ==41023== by 0x92DD7E0: vfprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9307078: vsnprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3CB1: snprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x7DB86DF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x4EE643: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/1/a.out) ==41023== by 0x8E86B6: MAIN__ (ibm3d_high_Re.F90:4447) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB6F57: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB870D: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x4EE643: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/1/a.out) ==41023== by 0x8E86B6: MAIN__ (ibm3d_high_Re.F90:4447) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Conditional jump or move depends on uninitialised value(s) ==41023== at 0x7DB6F3B: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB870D: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB1B3E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x4EE643: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/1/a.out) ==41023== by 0x8E86B6: MAIN__ (ibm3d_high_Re.F90:4447) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Syscall param write(buf) points to uninitialised byte(s) ==41023== at 0x90841FD: ??? (in /usr/lib64/libpthread-2.17.so) ==41023== by 0x7DB51C7: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB54BC: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB5597: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBC03E: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB0DDE: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB39B8: _gfortran_st_write_done (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x4EE652: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/1/a.out) ==41023== by 0x8E86B6: MAIN__ (ibm3d_high_Re.F90:4447) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== Address 0xcba09b0 is 432 bytes inside a block of size 8,192 alloc'd ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x7CE8914: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB516D: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB5B1C: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DAC6EF: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DACFCB: _gfortran_st_open (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x4EDEF9: __body_MOD_write_body_tecplot_pressure (in /home/wtay/Results/1/a.out) ==41023== by 0x8E86B6: MAIN__ (ibm3d_high_Re.F90:4447) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== Warning: set address range perms: large range [0x1597f8028, 0x17e7dfeec) (noaccess) ==41023== Warning: set address range perms: large range [0x10f828028, 0x1597f772c) (noaccess) ==41023== Warning: set address range perms: large range [0x97dbe028, 0xbcda5edc) (noaccess) ==41023== Warning: set address range perms: large range [0x4ddee028, 0x97dbd71c) (noaccess) ==41023== Syscall param write(buf) points to uninitialised byte(s) ==41023== at 0x90841FD: ??? (in /usr/lib64/libpthread-2.17.so) ==41023== by 0x1144030B: udcm_module_finalize (in /opt/ud/openmpi-1.8.8/lib/openmpi/mca_btl_openib.so) ==41023== by 0x11429EAB: mca_btl_openib_finalize (in /opt/ud/openmpi-1.8.8/lib/openmpi/mca_btl_openib.so) ==41023== by 0x8C0764F: mca_btl_base_close (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x9917974: mca_base_framework_close (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x9917974: mca_base_framework_close (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x8BCD7BD: ompi_mpi_finalize (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x4F9DF45: PetscFinalize (pinit.c:1367) ==41023== by 0x4FB752A: petscfinalize_ (zstart.c:458) ==41023== by 0x41A618: __global_data_MOD_de_ini_var (global.F90:2991) ==41023== by 0x8E98C4: MAIN__ (ibm3d_high_Re.F90:4545) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== Address 0xffeffb6f4 is on thread 1's stack ==41023== in frame #1, created by udcm_module_finalize (???:) ==41023== ==41023== Syscall param write(buf) points to uninitialised byte(s) ==41023== at 0x90841FD: ??? (in /usr/lib64/libpthread-2.17.so) ==41023== by 0x1142DE2F: device_destruct (in /opt/ud/openmpi-1.8.8/lib/openmpi/mca_btl_openib.so) ==41023== by 0x11429F98: mca_btl_openib_finalize (in /opt/ud/openmpi-1.8.8/lib/openmpi/mca_btl_openib.so) ==41023== by 0x8C0764F: mca_btl_base_close (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x9917974: mca_base_framework_close (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x9917974: mca_base_framework_close (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x8BCD7BD: ompi_mpi_finalize (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x4F9DF45: PetscFinalize (pinit.c:1367) ==41023== by 0x4FB752A: petscfinalize_ (zstart.c:458) ==41023== by 0x41A618: __global_data_MOD_de_ini_var (global.F90:2991) ==41023== by 0x8E98C4: MAIN__ (ibm3d_high_Re.F90:4545) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== Address 0xffeffb768 is on thread 1's stack ==41023== in frame #1, created by device_destruct (???:) ==41023== ==41023== Syscall param write(buf) points to uninitialised byte(s) ==41023== at 0x90841FD: ??? (in /usr/lib64/libpthread-2.17.so) ==41023== by 0x1142DB71: btl_openib_component_close (in /opt/ud/openmpi-1.8.8/lib/openmpi/mca_btl_openib.so) ==41023== by 0x990E9F8: mca_base_component_close (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x990EA71: mca_base_components_close (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x8C07694: mca_btl_base_close (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x9917974: mca_base_framework_close (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x9917974: mca_base_framework_close (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x8BCD7BD: ompi_mpi_finalize (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x4F9DF45: PetscFinalize (pinit.c:1367) ==41023== by 0x4FB752A: petscfinalize_ (zstart.c:458) ==41023== by 0x41A618: __global_data_MOD_de_ini_var (global.F90:2991) ==41023== by 0x8E98C4: MAIN__ (ibm3d_high_Re.F90:4545) ==41023== Address 0xffeffb754 is on thread 1's stack ==41023== in frame #1, created by btl_openib_component_close (???:) ==41023== ==41023== ==41023== HEAP SUMMARY: ==41023== in use at exit: 753,369 bytes in 9,581 blocks ==41023== total heap usage: 672,580 allocs, 662,999 frees, 7,639,468,235 bytes allocated ==41023== ==41023== 1 bytes in 1 blocks are definitely lost in loss record 1 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0xE3DFA90: ??? ==41023== by 0x9929736: opal_db_base_store (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x8BD02B2: ompi_modex_send_string (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BCC74A: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==41023== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==41023== by 0x8A5CBF: MAIN__ (ibm3d_high_Re.F90:47) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== 7 bytes in 1 blocks are definitely lost in loss record 2 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x9318529: strdup (in /usr/lib64/libc-2.17.so) ==41023== by 0xE3DF9D5: ??? ==41023== by 0x9929736: opal_db_base_store (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0xD7BF47D: ??? ==41023== by 0x96692D7: orte_init (in /opt/ud/openmpi-1.8.8/lib/libopen-rte.so.7.0.6) ==41023== by 0x8BCC6BC: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==41023== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==41023== by 0x8A5CBF: MAIN__ (ibm3d_high_Re.F90:47) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== 8 bytes in 1 blocks are definitely lost in loss record 4 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x1142F9AA: ??? ==41023== by 0x8C07C52: mca_btl_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x1121E870: ??? ==41023== by 0x8C0741D: mca_bml_base_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x12B9E9FE: ??? ==41023== by 0x8C17D08: mca_pml_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BCC9BE: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==41023== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==41023== by 0x8A5CBF: MAIN__ (ibm3d_high_Re.F90:47) ==41023== ==41023== 8 bytes in 1 blocks are definitely lost in loss record 5 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x4C2BB0B: realloc (vg_replace_malloc.c:785) ==41023== by 0x98F6177: opal_pointer_array_add (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x1142BEA5: ??? ==41023== by 0x1121D484: ??? ==41023== by 0x12B9D648: ??? ==41023== by 0x8BCCD7D: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==41023== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==41023== by 0x8A5CBF: MAIN__ (ibm3d_high_Re.F90:47) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== 16 bytes in 1 blocks are definitely lost in loss record 9 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x1142E3DA: ??? ==41023== by 0x8C07C52: mca_btl_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x1121E870: ??? ==41023== by 0x8C0741D: mca_bml_base_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x12B9E9FE: ??? ==41023== by 0x8C17D08: mca_pml_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BCC9BE: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==41023== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==41023== by 0x8A5CBF: MAIN__ (ibm3d_high_Re.F90:47) ==41023== ==41023== 16 bytes in 1 blocks are definitely lost in loss record 10 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x1142E45F: ??? ==41023== by 0x8C07C52: mca_btl_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x1121E870: ??? ==41023== by 0x8C0741D: mca_bml_base_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x12B9E9FE: ??? ==41023== by 0x8C17D08: mca_pml_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BCC9BE: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==41023== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==41023== by 0x8A5CBF: MAIN__ (ibm3d_high_Re.F90:47) ==41023== ==41023== 16 bytes in 1 blocks are definitely lost in loss record 11 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x1142E4DD: ??? ==41023== by 0x8C07C52: mca_btl_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x1121E870: ??? ==41023== by 0x8C0741D: mca_bml_base_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x12B9E9FE: ??? ==41023== by 0x8C17D08: mca_pml_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BCC9BE: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==41023== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==41023== by 0x8A5CBF: MAIN__ (ibm3d_high_Re.F90:47) ==41023== ==41023== 16 bytes in 1 blocks are definitely lost in loss record 12 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x1142BECD: ??? ==41023== by 0x1121D484: ??? ==41023== by 0x12B9D648: ??? ==41023== by 0x8BCCD7D: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==41023== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==41023== by 0x8A5CBF: MAIN__ (ibm3d_high_Re.F90:47) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== 24 bytes in 1 blocks are definitely lost in loss record 19 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0xE3DFA90: ??? ==41023== by 0x9929736: opal_db_base_store (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x8BD0189: ompi_modex_send (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x122E2527: ??? ==41023== by 0x8C07C52: mca_btl_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x1121E870: ??? ==41023== by 0x8C0741D: mca_bml_base_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x12B9E9FE: ??? ==41023== by 0x8C17D08: mca_pml_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BCC9BE: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== ==41023== 27 bytes in 1 blocks are definitely lost in loss record 21 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x9306E07: vasprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3DD6: asprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x114302B6: ??? ==41023== by 0x8C07C52: mca_btl_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x1121E870: ??? ==41023== by 0x8C0741D: mca_bml_base_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x12B9E9FE: ??? ==41023== by 0x8C17D08: mca_pml_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BCC9BE: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==41023== ==41023== 38 bytes in 1 blocks are definitely lost in loss record 77 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0xE3DFA90: ??? ==41023== by 0x9929736: opal_db_base_store (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x8BD0189: ompi_modex_send (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x1142D9DE: ??? ==41023== by 0x1142FB03: ??? ==41023== by 0x8C07C52: mca_btl_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x1121E870: ??? ==41023== by 0x8C0741D: mca_bml_base_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x12B9E9FE: ??? ==41023== by 0x8C17D08: mca_pml_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BCC9BE: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== ==41023== 64 bytes in 4 blocks are definitely lost in loss record 114 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x1142C0C4: ??? ==41023== by 0x1121D484: ??? ==41023== by 0x12B9D648: ??? ==41023== by 0x8BCCD7D: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==41023== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==41023== by 0x8A5CBF: MAIN__ (ibm3d_high_Re.F90:47) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== 64 bytes in 4 blocks are definitely lost in loss record 115 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x1142C008: ??? ==41023== by 0x1121D484: ??? ==41023== by 0x12B9D648: ??? ==41023== by 0x8BCCD7D: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==41023== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==41023== by 0x8A5CBF: MAIN__ (ibm3d_high_Re.F90:47) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== 128 (112 direct, 16 indirect) bytes in 1 blocks are definitely lost in loss record 131 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x1142882E: ??? ==41023== by 0x1142ECE7: ??? ==41023== by 0x8C07C52: mca_btl_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x1121E870: ??? ==41023== by 0x8C0741D: mca_bml_base_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x12B9E9FE: ??? ==41023== by 0x8C17D08: mca_pml_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BCC9BE: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==41023== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==41023== ==41023== 160 bytes in 1 blocks are definitely lost in loss record 136 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x1142F718: ??? ==41023== by 0x8C07C52: mca_btl_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x1121E870: ??? ==41023== by 0x8C0741D: mca_bml_base_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x12B9E9FE: ??? ==41023== by 0x8C17D08: mca_pml_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BCC9BE: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==41023== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==41023== by 0x8A5CBF: MAIN__ (ibm3d_high_Re.F90:47) ==41023== ==41023== 169 bytes in 8 blocks are definitely lost in loss record 137 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x9318529: strdup (in /usr/lib64/libc-2.17.so) ==41023== by 0x1143B258: ??? ==41023== by 0x1143B53A: ??? ==41023== by 0x1143B987: ??? ==41023== by 0x1142E258: ??? ==41023== by 0x8C07C52: mca_btl_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x1121E870: ??? ==41023== by 0x8C0741D: mca_bml_base_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x12B9E9FE: ??? ==41023== by 0x8C17D08: mca_pml_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BCC9BE: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== ==41023== 192 (112 direct, 80 indirect) bytes in 1 blocks are definitely lost in loss record 140 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x1142BE2D: ??? ==41023== by 0x1121D484: ??? ==41023== by 0x12B9D648: ??? ==41023== by 0x8BCCD7D: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==41023== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==41023== by 0x8A5CBF: MAIN__ (ibm3d_high_Re.F90:47) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== 211 (184 direct, 27 indirect) bytes in 1 blocks are definitely lost in loss record 141 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x10C1362C: ??? ==41023== by 0x8C0EDC1: mca_mpool_base_module_create (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x114302FA: ??? ==41023== by 0x8C07C52: mca_btl_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x1121E870: ??? ==41023== by 0x8C0741D: mca_bml_base_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x12B9E9FE: ??? ==41023== by 0x8C17D08: mca_pml_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BCC9BE: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==41023== ==41023== 448 (232 direct, 216 indirect) bytes in 1 blocks are definitely lost in loss record 148 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x1121D984: ??? ==41023== by 0x12B9D648: ??? ==41023== by 0x8BCCD7D: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==41023== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==41023== by 0x8A5CBF: MAIN__ (ibm3d_high_Re.F90:47) ==41023== by 0x8E9916: main (ibm3d_high_Re.F90:3) ==41023== ==41023== 536 (24 direct, 512 indirect) bytes in 1 blocks are definitely lost in loss record 151 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x7CE8914: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBECA: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB46B9: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7CE5E87: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x400F4E2: _dl_init (in /usr/lib64/ld-2.17.so) ==41023== by 0x4001459: ??? (in /usr/lib64/ld-2.17.so) ==41023== ==41023== 536 (24 direct, 512 indirect) bytes in 1 blocks are definitely lost in loss record 152 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x7CE8914: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DBBECA: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7DB4791: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x7CE5E87: ??? (in /usr/lib64/libgfortran.so.3.0.0) ==41023== by 0x400F4E2: _dl_init (in /usr/lib64/ld-2.17.so) ==41023== by 0x4001459: ??? (in /usr/lib64/ld-2.17.so) ==41023== ==41023== 1,021 bytes in 21 blocks are definitely lost in loss record 159 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x9306E07: vasprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3DD6: asprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9929F77: dlopen_foreachfile (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x990DAFC: mca_base_component_find (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x9917362: mca_base_framework_components_register (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x99177EB: mca_base_framework_register (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x9917841: mca_base_framework_open (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x98F7B48: opal_init (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x96691A4: orte_init (in /opt/ud/openmpi-1.8.8/lib/libopen-rte.so.7.0.6) ==41023== by 0x8BCC6BC: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== ==41023== 1,472 bytes in 1 blocks are definitely lost in loss record 161 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x8BB1EA0: ompi_free_list_grow (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x1142E532: ??? ==41023== by 0x8C07C52: mca_btl_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x1121E870: ??? ==41023== by 0x8C0741D: mca_bml_base_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x12B9E9FE: ??? ==41023== by 0x8C17D08: mca_pml_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BCC9BE: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==41023== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==41023== ==41023== 2,304 bytes in 1 blocks are definitely lost in loss record 163 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x8BB1EA0: ompi_free_list_grow (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x1142E44D: ??? ==41023== by 0x8C07C52: mca_btl_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x1121E870: ??? ==41023== by 0x8C0741D: mca_bml_base_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x12B9E9FE: ??? ==41023== by 0x8C17D08: mca_pml_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BCC9BE: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==41023== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==41023== ==41023== 2,304 bytes in 1 blocks are definitely lost in loss record 164 of 171 ==41023== at 0x4C29C3D: malloc (vg_replace_malloc.c:299) ==41023== by 0x8BB1EA0: ompi_free_list_grow (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x1142E4CB: ??? ==41023== by 0x8C07C52: mca_btl_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x1121E870: ??? ==41023== by 0x8C0741D: mca_bml_base_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x12B9E9FE: ??? ==41023== by 0x8C17D08: mca_pml_base_select (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BCC9BE: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8536594: mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi_mpifh.so.2.5.2) ==41023== by 0x4FB6CF1: petscinitialize_ (zstart.c:295) ==41023== ==41023== 5,004 bytes in 93 blocks are definitely lost in loss record 166 of 171 ==41023== at 0x4C2BB8A: realloc (vg_replace_malloc.c:785) ==41023== by 0x9306DDA: vasprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x92E3DD6: asprintf (in /usr/lib64/libc-2.17.so) ==41023== by 0x9929F77: dlopen_foreachfile (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x990DAFC: mca_base_component_find (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x9917362: mca_base_framework_components_register (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x99177EB: mca_base_framework_register (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x9917841: mca_base_framework_open (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x98F7B48: opal_init (in /opt/ud/openmpi-1.8.8/lib/libopen-pal.so.6.2.2) ==41023== by 0x96691A4: orte_init (in /opt/ud/openmpi-1.8.8/lib/libopen-rte.so.7.0.6) ==41023== by 0x8BCC6BC: ompi_mpi_init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== by 0x8BEC2BA: PMPI_Init (in /opt/ud/openmpi-1.8.8/lib/libmpi.so.1.6.3) ==41023== ==41023== LEAK SUMMARY: ==41023== definitely lost: 13,427 bytes in 151 blocks ==41023== indirectly lost: 1,363 bytes in 8 blocks ==41023== possibly lost: 0 bytes in 0 blocks ==41023== still reachable: 738,579 bytes in 9,422 blocks ==41023== suppressed: 0 bytes in 0 blocks ==41023== Reachable blocks (those to which a pointer was found) are not shown. ==41023== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==41023== ==41023== For counts of detected and suppressed errors, rerun with: -v ==41023== Use --track-origins=yes to see where uninitialised values come from ==41023== ERROR SUMMARY: 89912 errors from 203 contexts (suppressed: 2 from 2) [wtay at n12-72:1]$ From knepley at gmail.com Fri Dec 25 08:29:14 2015 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 25 Dec 2015 08:29:14 -0600 Subject: [petsc-users] Code sometimes work, sometimes hang when increase cpu usage In-Reply-To: <567D2B64.1030307@gmail.com> References: <567C19F1.6040307@gmail.com> <567CC7EE.7070500@gmail.com> <6C14CF4D-029E-46B3-99FF-41DE39454F83@mcs.anl.gov> <567D2B64.1030307@gmail.com> Message-ID: It appears that you have an uninitialized variable (or more than one). When compiled with debugging, variables are normally initialized to zero. Thanks, Matt On Fri, Dec 25, 2015 at 5:41 AM, TAY wee-beng wrote: > Hi, > > Sorry, there seems to be some problems with my valgrind. I have repeated > it again, with the optimized and debug version > > > > > > Thank you. > > Yours sincerely, > > TAY wee-beng > > On 25/12/2015 12:42 PM, Barry Smith wrote: > >> On Dec 24, 2015, at 10:37 PM, TAY wee-beng wrote: >>> >>> Hi, >>> >>> I tried valgrind in MPI but it aborts very early, with the error msg >>> regarding PETSc initialize. >>> >> It shouldn't "abort" it should print some error message and continue. >> Please send all the output when running with valgrind. >> >> It is possible you are solving large enough problem that require >> configure --with-64-bit-indices . Does that resolve the problem? >> >> Barry >> >> I retry again, using a lower resolution. >>> >>> GAMG works, but BoomerAMG and hypre doesn't. Increasing cpu too high >>> (80) also cause it to hang. 60 works fine. >>> >>> My grid size is 98x169x169 >>> >>> But when I increase the resolution, GAMG can't work again. >>> >>> I tried to increase the cpu no but it still doesn't work. >>> >>> Previously, using single z direction partition, it work using GAMG and >>> hypre. So what could be the problem? >>> Thank you. >>> >>> Yours sincerely, >>> >>> TAY wee-beng >>> >>> On 25/12/2015 12:33 AM, Matthew Knepley wrote: >>> >>>> It sounds like you have memory corruption in a different part of the >>>> code. Run in valgrind. >>>> >>>> Matt >>>> >>>> On Thu, Dec 24, 2015 at 10:14 AM, TAY wee-beng >>>> wrote: >>>> Hi, >>>> >>>> I have this strange error. I converted my CFD code from a z directon >>>> only partition to the yz direction partition. The code works fine but when >>>> I increase the cpu no, strange things happen when solving the Poisson eqn. >>>> >>>> I increase cpu no from 24 to 40. >>>> >>>> Sometimes it works, sometimes it doesn't. When it doesn't, it just >>>> hangs there with no output, or it gives the error below: >>>> >>>> Using MPI_Barrier during debug shows that it hangs at >>>> >>>> call KSPSolve(ksp,b_rhs,xx,ierr). >>>> >>>> I use hypre BoomerAMG and GAMG (-poisson_pc_gamg_agg_nsmooths 1 >>>> -poisson_pc_type gamg) >>>> >>>> >>>> Why is this so random? Also how do I debug this type of problem. >>>> >>>> >>>> [32]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [32]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >>>> probably memory access out of range >>>> [32]PETSC ERROR: Try option -start_in_debugger or >>>> -on_error_attach_debugger >>>> [32]PETSC ERROR: or see >>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>>> [32]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >>>> OS X to find memory corruption errors >>>> [32]PETSC ERROR: likely location of problem given in stack below >>>> [32]PETSC ERROR: --------------------- Stack Frames >>>> ------------------------------------ >>>> [32]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>> available, >>>> [32]PETSC ERROR: INSTEAD the line number of the start of the >>>> function >>>> [32]PETSC ERROR: is given. >>>> [32]PETSC ERROR: [32] HYPRE_SetupXXX line 174 >>>> /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/impls/hypre/hypre.c >>>> [32]PETSC ERROR: [32] PCSetUp_HYPRE line 122 >>>> /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/impls/hypre/hypre.c >>>> [32]PETSC ERROR: [32] PCSetUp line 945 >>>> /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/interface/precon.c >>>> [32]PETSC ERROR: [32] KSPSetUp line 247 >>>> /home/wtay/Codes/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c >>>> [32]PETSC ERROR: [32] KSPSolve line 510 >>>> /home/wtay/Codes/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c >>>> [32]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [32]PETSC ERROR: Signal received >>>> [32]PETSC ERROR: See >>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>> shooting. >>>> [32]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 >>>> [32]PETSC ERROR: ./a.out on a petsc-3.6.2_shared_gnu_debug named n12-40 >>>> by wtay Thu Dec 24 17:01:51 2015 >>>> [32]PETSC ERROR: Configure options >>>> --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --download-fblaslapack=1 >>>> --with-debugging=1 --download-hypre=1 >>>> --prefix=/home/wtay/Lib/petsc-3.6.2_shared_gnu_debug --known-mpi-shared=1 >>>> --with-shared-libraries --with-fortran-interfaces=1 >>>> [32]PETSC ERROR: #1 User provided function() line 0 in unknown file >>>> >>>> -------------------------------------------------------------------------- >>>> MPI_ABORT was invoked on rank 32 in communicator MPI_COMM_WORLD >>>> with errorcode 59. >>>> >>>> -- >>>> Thank you. >>>> >>>> Yours sincerely, >>>> >>>> TAY wee-beng >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrosso at uci.edu Sun Dec 27 11:56:45 2015 From: mrosso at uci.edu (Michele Rosso) Date: Sun, 27 Dec 2015 18:56:45 +0100 Subject: [petsc-users] PetscOptionsBegin in Fortran Message-ID: <1451239005.3529.4.camel@enterprise-A> Hi, is there a Fortran version for PetscOptionsBegin? I tried it to use it in my fortran code but the compiler complains about an undefined reference at link time. I am using petsc-master. Michele -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Dec 27 12:05:06 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 27 Dec 2015 12:05:06 -0600 Subject: [petsc-users] PetscOptionsBegin in Fortran In-Reply-To: <1451239005.3529.4.camel@enterprise-A> References: <1451239005.3529.4.camel@enterprise-A> Message-ID: No, in Fortran you can only use the PetscOptionsGet[Int,...] routines not the PetscOptions[Int,...] routines. Barry > On Dec 27, 2015, at 11:56 AM, Michele Rosso wrote: > > Hi, > > is there a Fortran version for PetscOptionsBegin? > I tried it to use it in my fortran code but the compiler complains about an undefined reference at link time. > I am using petsc-master. > > Michele From yangchao at iscas.ac.cn Sun Dec 27 21:17:07 2015 From: yangchao at iscas.ac.cn (Chao Yang) Date: Mon, 28 Dec 2015 11:17:07 +0800 Subject: [petsc-users] Support of NetCDF Message-ID: <656D5E79-BD12-4189-A495-5B9006F941E5@iscas.ac.cn> Hi, A quick question. Does PETSc support NetCDF format for input/output? Any suggestion is highly appreciated. Thanks! Best regards, Chao From fdkong.jd at gmail.com Mon Dec 28 12:10:08 2015 From: fdkong.jd at gmail.com (Fande Kong) Date: Mon, 28 Dec 2015 11:10:08 -0700 Subject: [petsc-users] Support of NetCDF Message-ID: >Message: 2 > Date: Mon, 28 Dec 2015 11:17:07 +0800 >From: Chao Yang >To: petsc-users at mcs.anl.gov >Subject: [petsc-users] Support of NetCDF >Message-ID: <656D5E79-BD12-4189-A495-5B9006F941E5 at iscas.ac.cn> >Content-Type: text/plain; charset=us-ascii >Hi, >A quick question. Does PETSc support NetCDF format for input/output? DMPlex supports input only through exodus. http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexCreateExodus.html >Any suggestion is highly appreciated. Thanks! If you want to do parallel IO, it is better to use HDF5 and has both input and output for vector and IS. >Best regards, >Chao Fande, -------------- next part -------------- An HTML attachment was scrubbed... URL: From yangchao at iscas.ac.cn Mon Dec 28 19:41:51 2015 From: yangchao at iscas.ac.cn (Chao Yang) Date: Tue, 29 Dec 2015 09:41:51 +0800 Subject: [petsc-users] Support of NetCDF In-Reply-To: References: Message-ID: <73AF2EF1-421B-43E1-B249-F11946F224DC@iscas.ac.cn> Thanks, Fande. But it is NetCDF that I need, not HDF5, for some specific reason, though they are somehow connected. > ? 2015?12?29????2:10?Fande Kong ??? > > >Message: 2 > > Date: Mon, 28 Dec 2015 11:17:07 +0800 > >From: Chao Yang > > >To: petsc-users at mcs.anl.gov > >Subject: [petsc-users] Support of NetCDF > >Message-ID: <656D5E79-BD12-4189-A495-5B9006F941E5 at iscas.ac.cn > > >Content-Type: text/plain; charset=us-ascii > > >Hi, > > >A quick question. Does PETSc support NetCDF format for input/output? > > DMPlex supports input only through exodus. http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexCreateExodus.html > > >Any suggestion is highly appreciated. Thanks! > > If you want to do parallel IO, it is better to use HDF5 and has both input and output for vector and IS. > > >Best regards, > >Chao > > > Fande, -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Dec 28 22:00:21 2015 From: jed at jedbrown.org (Jed Brown) Date: Mon, 28 Dec 2015 21:00:21 -0700 Subject: [petsc-users] Support of NetCDF In-Reply-To: <73AF2EF1-421B-43E1-B249-F11946F224DC@iscas.ac.cn> References: <73AF2EF1-421B-43E1-B249-F11946F224DC@iscas.ac.cn> Message-ID: <87ege5ucoa.fsf@jedbrown.org> Chao Yang writes: > Thanks, Fande. But it is NetCDF that I need, not HDF5, for some specific reason, though they are somehow connected. There is no raw NetCDF, but Exodus is a NetCDF-based format. >> ? 2015?12?29????2:10?Fande Kong ??? >> >> >Message: 2 >> > Date: Mon, 28 Dec 2015 11:17:07 +0800 >> >From: Chao Yang > >> >To: petsc-users at mcs.anl.gov >> >Subject: [petsc-users] Support of NetCDF >> >Message-ID: <656D5E79-BD12-4189-A495-5B9006F941E5 at iscas.ac.cn > >> >Content-Type: text/plain; charset=us-ascii >> >> >Hi, >> >> >A quick question. Does PETSc support NetCDF format for input/output? >> >> DMPlex supports input only through exodus. http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexCreateExodus.html >> >> >Any suggestion is highly appreciated. Thanks! >> >> If you want to do parallel IO, it is better to use HDF5 and has both input and output for vector and IS. >> >> >Best regards, >> >Chao >> >> >> Fande, -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From zocca.marco at gmail.com Tue Dec 29 04:49:08 2015 From: zocca.marco at gmail.com (Marco Zocca) Date: Tue, 29 Dec 2015 11:49:08 +0100 Subject: [petsc-users] LP solvers based on PETSc ? Message-ID: Dear all, Is anyone here aware of linear program solvers, of any form, being implemented on top of PETSc ? Thank you in advance, Marco From yangchao at iscas.ac.cn Tue Dec 29 10:18:12 2015 From: yangchao at iscas.ac.cn (Chao Yang) Date: Wed, 30 Dec 2015 00:18:12 +0800 Subject: [petsc-users] Support of NetCDF In-Reply-To: <87ege5ucoa.fsf@jedbrown.org> References: <73AF2EF1-421B-43E1-B249-F11946F224DC@iscas.ac.cn> <87ege5ucoa.fsf@jedbrown.org> Message-ID: <7CC87BCF-2FA8-4EAF-B09F-EC822657BE82@iscas.ac.cn> Thanks, Jed. > ? 2015?12?29????12:00?Jed Brown ??? > > Chao Yang writes: > >> Thanks, Fande. But it is NetCDF that I need, not HDF5, for some specific reason, though they are somehow connected. > > There is no raw NetCDF, but Exodus is a NetCDF-based format. > >>> ? 2015?12?29????2:10?Fande Kong ??? >>> >>>> Message: 2 >>>> Date: Mon, 28 Dec 2015 11:17:07 +0800 >>>> From: Chao Yang > >>>> To: petsc-users at mcs.anl.gov >>>> Subject: [petsc-users] Support of NetCDF >>>> Message-ID: <656D5E79-BD12-4189-A495-5B9006F941E5 at iscas.ac.cn > >>>> Content-Type: text/plain; charset=us-ascii >>> >>>> Hi, >>> >>>> A quick question. Does PETSc support NetCDF format for input/output? >>> >>> DMPlex supports input only through exodus. http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexCreateExodus.html >>> >>>> Any suggestion is highly appreciated. Thanks! >>> >>> If you want to do parallel IO, it is better to use HDF5 and has both input and output for vector and IS. >>> >>>> Best regards, >>>> Chao >>> >>> >>> Fande, From jed at jedbrown.org Tue Dec 29 13:05:05 2015 From: jed at jedbrown.org (Jed Brown) Date: Tue, 29 Dec 2015 12:05:05 -0700 Subject: [petsc-users] LP solvers based on PETSc ? In-Reply-To: References: Message-ID: <87bn99rs7y.fsf@jedbrown.org> Marco Zocca writes: > Is anyone here aware of linear program solvers, of any form, being > implemented on top of PETSc ? I'm not aware of dedicated LP solvers, but you could use TAO's Interior Point methods. See src/tao/constrained/examples/tutorials/toy.c for example. Note that the implementation is immature. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From zonexo at gmail.com Wed Dec 30 19:20:09 2015 From: zonexo at gmail.com (TAY wee-beng) Date: Thu, 31 Dec 2015 09:20:09 +0800 Subject: [petsc-users] Code sometimes work, sometimes hang when increase cpu usage In-Reply-To: References: <567C19F1.6040307@gmail.com> <567CC7EE.7070500@gmail.com> <6C14CF4D-029E-46B3-99FF-41DE39454F83@mcs.anl.gov> <567D2B64.1030307@gmail.com> Message-ID: <568482C9.2090201@gmail.com> Hi, I've been debugging and removing some errors. Now the code works on most cluster nodes but fails on 2 of them. The strange thing is that I'm using the same gnu compiler but only deploying to the newer setup nodes. The newer nodes work when using my old code, which is similar except that its domain partition is only in the z direction. The new code partitions in y and z direction. It fails at the Poisson eqn solving part. Is there anyway I can find out why this is happening? Thank you. Yours sincerely, TAY wee-beng On 25/12/2015 10:29 PM, Matthew Knepley wrote: > It appears that you have an uninitialized variable (or more than one). > When compiled with debugging, variables > are normally initialized to zero. > > Thanks, > > Matt > > On Fri, Dec 25, 2015 at 5:41 AM, TAY wee-beng > wrote: > > Hi, > > Sorry, there seems to be some problems with my valgrind. I have > repeated it again, with the optimized and debug version > > > > > > Thank you. > > Yours sincerely, > > TAY wee-beng > > On 25/12/2015 12:42 PM, Barry Smith wrote: > > On Dec 24, 2015, at 10:37 PM, TAY wee-beng > > wrote: > > Hi, > > I tried valgrind in MPI but it aborts very early, with the > error msg regarding PETSc initialize. > > It shouldn't "abort" it should print some error message and > continue. Please send all the output when running with valgrind. > > It is possible you are solving large enough problem that > require configure --with-64-bit-indices . Does that resolve > the problem? > > Barry > > I retry again, using a lower resolution. > > GAMG works, but BoomerAMG and hypre doesn't. Increasing > cpu too high (80) also cause it to hang. 60 works fine. > > My grid size is 98x169x169 > > But when I increase the resolution, GAMG can't work again. > > I tried to increase the cpu no but it still doesn't work. > > Previously, using single z direction partition, it work > using GAMG and hypre. So what could be the problem? > Thank you. > > Yours sincerely, > > TAY wee-beng > > On 25/12/2015 12:33 AM, Matthew Knepley wrote: > > It sounds like you have memory corruption in a > different part of the code. Run in valgrind. > > Matt > > On Thu, Dec 24, 2015 at 10:14 AM, TAY wee-beng > > wrote: > Hi, > > I have this strange error. I converted my CFD code > from a z directon only partition to the yz direction > partition. The code works fine but when I increase the > cpu no, strange things happen when solving the Poisson > eqn. > > I increase cpu no from 24 to 40. > > Sometimes it works, sometimes it doesn't. When it > doesn't, it just hangs there with no output, or it > gives the error below: > > Using MPI_Barrier during debug shows that it hangs at > > call KSPSolve(ksp,b_rhs,xx,ierr). > > I use hypre BoomerAMG and GAMG > (-poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg) > > > Why is this so random? Also how do I debug this type > of problem. > > > [32]PETSC ERROR: > ------------------------------------------------------------------------ > [32]PETSC ERROR: Caught signal number 11 SEGV: > Segmentation Violation, probably memory access out of > range > [32]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > [32]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [32]PETSC ERROR: or try http://valgrind.org on > GNU/linux and Apple Mac OS X to find memory corruption > errors > [32]PETSC ERROR: likely location of problem given in > stack below > [32]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [32]PETSC ERROR: Note: The EXACT line numbers in the > stack are not available, > [32]PETSC ERROR: INSTEAD the line number of the > start of the function > [32]PETSC ERROR: is given. > [32]PETSC ERROR: [32] HYPRE_SetupXXX line 174 > /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/impls/hypre/hypre.c > [32]PETSC ERROR: [32] PCSetUp_HYPRE line 122 > /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/impls/hypre/hypre.c > [32]PETSC ERROR: [32] PCSetUp line 945 > /home/wtay/Codes/petsc-3.6.2/src/ksp/pc/interface/precon.c > [32]PETSC ERROR: [32] KSPSetUp line 247 > /home/wtay/Codes/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c > [32]PETSC ERROR: [32] KSPSolve line 510 > /home/wtay/Codes/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c > [32]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [32]PETSC ERROR: Signal received > [32]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [32]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, > 2015 > [32]PETSC ERROR: ./a.out on a > petsc-3.6.2_shared_gnu_debug named n12-40 by wtay Thu > Dec 24 17:01:51 2015 > [32]PETSC ERROR: Configure options > --with-mpi-dir=/opt/ud/openmpi-1.8.8/ > --download-fblaslapack=1 --with-debugging=1 > --download-hypre=1 > --prefix=/home/wtay/Lib/petsc-3.6.2_shared_gnu_debug > --known-mpi-shared=1 --with-shared-libraries > --with-fortran-interfaces=1 > [32]PETSC ERROR: #1 User provided function() line 0 > in unknown file > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 32 in communicator > MPI_COMM_WORLD > with errorcode 59. > > -- > Thank you. > > Yours sincerely, > > TAY wee-beng > > > > > -- > What most experimenters take for granted before they > begin their experiments is infinitely more interesting > than any results to which their experiments lead. > -- Norbert Wiener > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From amneetb at live.unc.edu Thu Dec 31 00:02:18 2015 From: amneetb at live.unc.edu (Bhalla, Amneet Pal S) Date: Thu, 31 Dec 2015 06:02:18 +0000 Subject: [petsc-users] VecGet{Restore}SubVector behavior Message-ID: <3AD53862-5D8B-44F3-8202-F9033875B2AD@ad.unc.edu> Hi Folks, Following upon this thread http://lists.mcs.anl.gov/pipermail/petsc-users/2013-October/018989.html is the behavior of VecRestoreSubVector() changed for non-contiguous enteries of original Vec to what is posted here? For example, if I extract a noncontiguous sub vector b from a Vec A, modify enteries of b, and restore b, would I see corresponding changes in A? Thanks, ? Amneet ===================================================== Amneet Bhalla Postdoctoral Research Associate Department of Mathematics and McAllister Heart Institute University of North Carolina at Chapel Hill Email: amneet at unc.edu Web: https://abhalla.web.unc.edu ===================================================== -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Dec 31 07:56:26 2015 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 31 Dec 2015 07:56:26 -0600 Subject: [petsc-users] VecGet{Restore}SubVector behavior In-Reply-To: <3AD53862-5D8B-44F3-8202-F9033875B2AD@ad.unc.edu> References: <3AD53862-5D8B-44F3-8202-F9033875B2AD@ad.unc.edu> Message-ID: On Thu, Dec 31, 2015 at 12:02 AM, Bhalla, Amneet Pal S wrote: > Hi Folks, > > Following upon this thread > > http://lists.mcs.anl.gov/pipermail/petsc-users/2013-October/018989.html > > is the behavior of VecRestoreSubVector() changed for non-contiguous > enteries of original Vec to what is posted here? For example, if I > extract a noncontiguous sub vector b from a Vec A, modify enteries of b, > and restore b, would I see corresponding changes in > A? > No. Now we automatically create a VecScatter for non-contiguous access. Thanks, Matt > Thanks, > > ? Amneet > ===================================================== > Amneet Bhalla > Postdoctoral Research Associate > Department of Mathematics and McAllister Heart Institute > University of North Carolina at Chapel Hill > Email: amneet at unc.edu > Web: https://abhalla.web.unc.edu > ===================================================== > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: