From knepley at gmail.com Wed Apr 1 06:42:44 2015 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 1 Apr 2015 06:42:44 -0500 Subject: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult In-Reply-To: <1358222418.2626161.1427861106397.JavaMail.yahoo@mail.yahoo.com> References: <1358222418.2626161.1427861106397.JavaMail.yahoo@mail.yahoo.com> Message-ID: On Tue, Mar 31, 2015 at 11:05 PM, Steena M wrote: > Thanks Matt. I'm still getting the malloc error > If you are loading a matrix, its just CHKERRQ( MatCreate(comm, &A)); CHKERRQ( MatLoad(A,fd)); Matt > [0]PETSC ERROR: Argument out of range! > [0]PETSC ERROR: New nonzero at (2,18) caused a malloc! > > and > > a new incorrect matrix file error: > > [0]PETSC ERROR: Unexpected data in file! > [0]PETSC ERROR: not matrix object! > > Maybe the order of calls is mixed up. This is the code snippet: > > if (rank ==0) > { > PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); > > CHKERRQ(MatSetSizes(A, 15, PETSC_DETERMINE, 20, 20)); > CHKERRQ(MatSetType(A, MATMPIBAIJ)); > CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); > CHKERRQ( MatLoad(A,fd)); > CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); > } > else > { > PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); > CHKERRQ( MatSetSizes(A, 5, PETSC_DETERMINE, 20, 20) ); > CHKERRQ(MatSetType(A, MATMPIBAIJ)); > CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); > CHKERRQ(MatLoad(A,fd)); > CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); > } > > Is there something I'm missing? > > Thanks, > Steena > > > > > On Tuesday, March 31, 2015 6:10 PM, Matthew Knepley > wrote: > > > On Tue, Mar 31, 2015 at 6:51 PM, Steena M wrote: > > Thanks Barry. I'm still getting the malloc error with NULL. Is there a way > to distribute the matrix without explicit preallocation? Different matrices > will be loaded during runtime and assigning preallocation parameters would > mean an additional preprocessing step. > > > 1) MatSetOption(MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) > > 2) Note that this is never ever ever more efficient than making another > pass and preallocating > > Thanks, > > Matt > > -------------------------------------------- > On Sun, 3/29/15, Barry Smith wrote: > > Subject: Re: [petsc-users] Unequal sparse matrix row distribution for MPI > MatMult > To: "Steena M" > Cc: "Matthew Knepley" , petsc-users at mcs.anl.gov > Date: Sunday, March 29, 2015, 9:26 PM > > > > On > Mar 29, 2015, at 11:05 PM, Steena M > wrote: > > > > Thanks > Matt. I used PETSC_DETERMINE but I'm now getting an > allocation-based error: > > > > [0]PETSC ERROR: --------------------- > Error Message ------------------------------------ > > [0]PETSC ERROR: Argument out of range! > > [0]PETSC ERROR: New nonzero at (2,18) > caused a malloc! > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > > I tried > preallocating on each rank for the diagonal and off diagonal > section of the matrix as the next step My current > approximations for preallocation > > > > CHKERRQ( > MatMPIBAIJSetPreallocation(A,1,5,PETSC_DEFAULT,5,PETSC_DEFAULT)); > > > These > arguments where you pass PETSC_DEFAULT are expecting a > pointer not an integer. You can pass NULL in those > locations. Though it is better to provide the correct > preallocation rather than some defaults. > > Barry > > > > > are throwing segmentation errors. > > > > [0]PETSC ERROR: > Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > > > > Any insights into what I'm doing > wrong? > > > > Thanks, > > Steena > > > > > > > > On Sun, 3/29/15, Matthew Knepley > wrote: > > > > Subject: > Re: [petsc-users] Unequal sparse matrix row distribution for > MPI MatMult > > To: "Steena M" > > > Cc: "Barry Smith" , > petsc-users at mcs.anl.gov > > Date: Sunday, March 29, 2015, 10:02 PM > > > > On Sun, Mar 29, 2015 > at > > 9:56 PM, Steena M > > wrote: > > Hi > > Barry, > > > > > > > > I am trying to partition a 20 row and 20 > col sparse matrix > > between two procs > such that proc 0 has 15 rows and 20 cols > > and proc 1 has 5 rows and 20 cols. The > code snippet: > > > > > > > > > > > > > > > > > CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A)); > // > > at runtime: -matload_block_size 1 > > > > > > > > > > > > > if (rank ==0) > > > > { > > > > > CHKERRQ( MatSetSizes(A, 15, 20, 20, > > 20) ); //rank 0 gets 75% of the rows > > > > > CHKERRQ( MatSetType(A, MATMPIBAIJ) > > ); > > > > CHKERRQ( > MatLoad(A,fd) ); > > > > } > > > > > > > > else > > > > > { > > > > > CHKERRQ( MatSetSizes(A, 5, > 20, 20, > > 20) ); //rank 1 gets 25% of the > rows > > > > > CHKERRQ( MatSetType(A, MATMPIBAIJ) > > ); > > > > CHKERRQ( > MatLoad(A,fd) ); > > > > } > > > > > > > > > This throws the following error (probably > from psplit.c): > > > > > [1]PETSC ERROR: --------------------- Error Message > > ------------------------------------ > > > > [1]PETSC ERROR: > Nonconforming object sizes! > > > > [1]PETSC ERROR: Sum of local lengths 40 > does not equal > > global length 20, my > local length 20 > > > > likely a call to > VecSetSizes() or MatSetSizes() is > > > wrong. > > > > See > http://www.mcs.anl.gov/petsc/documentation/faq.html#split! > > > > > > > > This error printout > doesn't quite make sense to me. > > > I'm trying to specify a total matrix size of 20x20... > I > > haven't yet figured out where the > '40' comes > > from in the error > message. > > > > > > > > Any thoughts on what > might be going wrong? > > > > Its the column specification. Just > > use PETSC_DETERMINE for the local columns > since all our > > sparse matrixformats are > row divisions > > anyway. > > > > Thanks, > > > > Matt > > Thanks in advance, > > > > > Steena > > > > > > > > > > > > > > > > > -------------------------------------------- > > > > On Sun, 3/22/15, > Barry Smith > > wrote: > > > > > > > > Subject: Re: [petsc-users] Unequal > sparse matrix row > > distribution for MPI > MatMult > > > > To: > "Steena M" > > > > Cc: petsc-users at mcs.anl.gov > > > > Date: Sunday, > March 22, 2015, 3:58 PM > > > > > > > > > > > > > > > > Steena, > > > > > > > > I am > > > > > a little unsure of your question. > > > > > > > > > 1) > you can create a MPIBAIJ > > > > matrix with any distribution of block > rows per process > > you > > > > want, just set the > local row size for each process to > > > be > > > > what you > like. Use MatCreateVecs() to get > > > correspondingly > > > > > laid out vectors. > > > > > > > > > > or 2) if you have a MPIBAIJ > > > > matrix with > "equal" row layout and you want a > > new > > > > one with uneven row layout you can > simply use > > > > > MatGetSubMatrix() to create that new matrix. > > > > > > > > Barry > > > > > > > > Unless you have > another reason to have the > > > > matrix with an equal number row layout I > would just > > generate > > > > the matrix with > the layout you want. > > > > > > > > > > > > > On Mar 22, 2015, at 5:50 PM, Steena > M > > > > > > > > wrote: > > > > > > > > > > Hello, > > > > > > > > > > I need to > distribute > > > > a > sparse matrix such that each proc owns an unequal > > number > > > > of blocked rows before I proceed with > MPI MatMult. My > > > > > initial thoughts on doing this: > > > > > > > > > > 1) Use MatGetSubMatrices() on the > test > > > > MATMPIBAIJ > matrix to produce a new matrix where each > > proc > > > > has an unequal number of rows. > > > > > > > > > > 2) Provide > scatter context for vector X > > > > (for MatMult )using IS iscol from > MatGetSubMatrices() > > while > > > > creating the > vector X. > > > > > > > > > > > 3) > Call MatMult() > > > > > > > > > > > Will > MatMult_MPIBAIJ continue to scatter > > > > this matrix and vector such that each > proc will own an > > equal > > > > number of matrix > rows and corresponding diagonal vector > > > > > elements? Should I write my own > MPIMatMult function to > > > > retain my redistribution of the matrix > and vector? > > > > > > > > > > > > Thanks in > > > > > advance, > > > > > > Steena > > > > > > > > > > > > > > -- > > What most > experimenters > > take for granted before > they begin their experiments is > > > infinitely more interesting than any results to which > their > > experiments lead. > > -- Norbert > > Wiener > > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Apr 1 07:20:50 2015 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 1 Apr 2015 07:20:50 -0500 Subject: [petsc-users] Obtaining constraint values in DM vector solution In-Reply-To: References: Message-ID: On Tue, Mar 31, 2015 at 6:08 PM, Justin Chang wrote: > In my first program I have already set the name for the solution. Vec U > has the name "pressure". > > In my second program, how do I obtain and use/load "pressure"? Would I > have to use PetscObjectQuery somehow? > You must name auxVec before calling VecLoad(). It then looks for that name in the HDF5 file. Thanks, Matt > On Tue, Mar 31, 2015 at 6:00 PM, Matthew Knepley > wrote: > >> On Tue, Mar 31, 2015 at 5:54 PM, Justin Chang wrote: >> >>> Matt thanks for the responses. I plan on loading the solution sol.h5 >>> obtained from one program into another (as the auxiliary). So in my other >>> program, I am solving a different problem but on the same mesh. I have the >>> following lines in my other code: >>> >> >> Yes, if you want to do this, you have to give these vectors proper names: >> >> PetscObjectSetName((PetscObject), U, "solution"); >> >> Thanks, >> >> Matt >> >> >>> ierr = >>> PetscViewerHDF5Open(PETSC_COMM_SELF,"sol.h5",FILE_MODE_READ,&viewer);CHKERRQ(ierr); >>> ierr = VecLoad(auxVec,&viewer);CHKERRQ(ierr); >>> ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr); >>> >>> But when this program reaches VecLoad i get the following errors: >>> >>> HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 0: >>> #000: H5D.c line 334 in H5Dopen2(): not found >>> major: Dataset >>> minor: Object not found >>> #001: H5Gloc.c line 430 in H5G_loc_find(): can't find object >>> major: Symbol table >>> minor: Object not found >>> #002: H5Gtraverse.c line 861 in H5G_traverse(): internal path >>> traversal failed >>> major: Symbol table >>> minor: Object not found >>> #003: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal >>> operator failed >>> major: Symbol table >>> minor: Callback failed >>> #004: H5Gloc.c line 385 in H5G_loc_find_cb(): object >>> 'Vec_0x7f9831516fb0_0' doesn't exist >>> major: Symbol table >>> minor: Object not found >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Error in external library >>> [0]PETSC ERROR: Error in HDF5 call H5Dopen2() Status -1 >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2528-gbee642f >>> GIT Date: 2015-03-29 20:36:38 -0500 >>> [0]PETSC ERROR: ./trans on a arch-darwin-c-debug named >>> CAML-03.CIVE.UH.EDU by jychang48 Tue Mar 31 17:45:38 2015 >>> [0]PETSC ERROR: Configure options --download-chaco --download-exodusii >>> --download-fblaslapack --download-hdf5 --download-metis --download-mumps >>> --download-netcdf --download-openmpi --download-parmetis >>> --download-scalapack --download-triangle --with-cc=gcc --with-cmake=cmake >>> --with-cxx=g++ --with-debugging=1 --with-fc=gfortran --with-valgrind=1 >>> CFLAGS= CXXFLAGS= PETSC_ARCH=arch-darwin-c-debug >>> [0]PETSC ERROR: #1 VecLoad_HDF5() line 268 in >>> /Users/jychang48/Software/petsc-dev/src/vec/vec/utils/vecio.c >>> [0]PETSC ERROR: #2 VecLoad_Default() line 391 in >>> /Users/jychang48/Software/petsc-dev/src/vec/vec/utils/vecio.c >>> [0]PETSC ERROR: #3 VecLoad_Plex_Local() line 158 in >>> /Users/jychang48/Software/petsc-dev/src/dm/impls/plex/plex.c >>> [0]PETSC ERROR: #4 VecLoad() line 969 in >>> /Users/jychang48/Software/petsc-dev/src/vec/vec/interface/vector.c >>> [0]PETSC ERROR: #5 CreateMesh() line 265 in >>> /Users/jychang48/Dropbox/Research_Topics/Code_PETSc/Nonneg/trans.c >>> [0]PETSC ERROR: #6 main() line 381 in >>> /Users/jychang48/Dropbox/Research_Topics/Code_PETSc/Nonneg/trans.c >>> >>> Any idea what's going on here? >>> >>> On Tue, Mar 31, 2015 at 5:46 PM, Matthew Knepley >>> wrote: >>> >>>> On Tue, Mar 31, 2015 at 5:29 PM, Justin Chang wrote: >>>> >>>>> How do I invoke this command into my program? When I add "-vec_view >>>>> hdf5:sol.h5" my program says this has been left out. >>>>> >>>> >>>> You can call VecView() directly with an HDF5 Viewer, or I think its >>>> easier to call >>>> >>>> VecViewFromOptions(U, NULL, "-vec_view"); >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> On Tue, Mar 31, 2015 at 5:13 PM, Matthew Knepley >>>>> wrote: >>>>> >>>>>> On Tue, Mar 31, 2015 at 5:06 PM, Justin Chang >>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> So in my DMPlex FEM (serial) simulations, I want to output the >>>>>>> solution U to a binary file. I have the following: >>>>>>> >>>>>>> ierr = >>>>>>> PetscViewerBinaryOpen(PETSC_COMM_WORLD,"sol.dat",FILE_MODE_WRITE,&viewer);CHKERRQ(ierr); >>>>>>> ierr = VecView(U,viewer);CHKERRQ(ierr); >>>>>>> ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr); >>>>>>> >>>>>>> The mesh I am solving has 13368 nodes, and 404 of them are >>>>>>> constrained using the DMPlexAddBoundary(...) function. However, when I >>>>>>> output the vector I only get 12964 values, which I believe is leaving out >>>>>>> the 404 constrained ones. How do I get all dofs in the vector U (both free >>>>>>> and constrained) printed? >>>>>>> >>>>>> >>>>>> I would use HDF5 which will do this automatically: >>>>>> >>>>>> -vec_view hdf5:sol.h5 >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Justin Chang >>>>>>> PhD Candidate, Civil Engineering - Computational Sciences >>>>>>> University of Houston, Department of Civil and Environmental >>>>>>> Engineering >>>>>>> Houston, TX 77004 >>>>>>> (512) 963-3262 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Justin Chang >>>>> PhD Candidate, Civil Engineering - Computational Sciences >>>>> University of Houston, Department of Civil and Environmental >>>>> Engineering >>>>> Houston, TX 77004 >>>>> (512) 963-3262 >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >>> >>> -- >>> Justin Chang >>> PhD Candidate, Civil Engineering - Computational Sciences >>> University of Houston, Department of Civil and Environmental Engineering >>> Houston, TX 77004 >>> (512) 963-3262 >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Carol.Brickley at awe.co.uk Wed Apr 1 08:43:34 2015 From: Carol.Brickley at awe.co.uk (Carol.Brickley at awe.co.uk) Date: Wed, 1 Apr 2015 13:43:34 +0000 Subject: [petsc-users] EXTERNAL: Re: Problems trying to use -log_summary In-Reply-To: References: <201503311710.t2VHAnpw002827@msw1.awe.co.uk> Message-ID: <201504011343.t31Dhe4d009917@msw2.awe.co.uk> Thanks Couldn't get it to recognise the flag, -log_summary, but wrote to file using PetscLogPrintDetailed(PETSC_COMM_WORLD, 'petsc_log_summary.txt', petsc_err). Thanks Carol Dr Carol Brickley BSc,PhD,ARCS,DIC,MBCS Senior Software Engineer Applied Computer Science DS+T, AWE Aldermaston Reading Berkshire RG7 4PR Direct: 0118 9855035 -----Original Message----- From: Satish Balay [mailto:balay at mcs.anl.gov] Sent: 31 March 2015 18:22 To: Matthew Knepley Cc: Brickley Carol AWE; petsc-users at mcs.anl.gov Subject: EXTERNAL: Re: [petsc-users] Problems trying to use -log_summary Also check if petsc examples work [with -log_summary]. This will isolate the issue [petsc build vs application code] Satish On Tue, 31 Mar 2015, Matthew Knepley wrote: > On Tue, Mar 31, 2015 at 12:10 PM, wrote: > > > Hi, > > > > I am having problems trying to use -log_summary. I have added > > PetscLogBegin,PetscLogInitSummary and compiled with PETSC_USE_LOG. I then > > run with -log_summary but the code will not accept this flag, > > > > Use the debugger and find out why its not executing the routine. It could > be that you did not rebuild everything correctly, > or that the option is not being passed correctly, etc. There are so many > variables that we cannot really be helpful yet. Also, > since this is used by almost every PETSc run done, it is very unlikely to > be a bug in the code. > > Thanks, > > Matt > > > > > > Any ideas? > > > > Carol > > > > > > > > *Dr Carol Brickley * > > > > *BSc,PhD,ARCS,DIC,MBCS* > > > > > > > > *Senior Software Engineer* > > > > *Applied Computer Science* > > > > *DS+T,* > > > > *AWE* > > > > *Aldermaston* > > > > *Reading* > > > > *Berkshire* > > > > *RG7 4PR* > > > > > > > > *Direct: 0118 9855035* > > > > > > > > > > ___________________________________________________ > > ____________________________ The information in this email and in any > > attachment(s) is commercial in confidence. If you are not the named > > addressee(s) or if you receive this email in error then any distribution, > > copying or use of this communication or the information in it is strictly > > prohibited. Please notify us immediately by email at admin.internet(at) > > awe.co.uk, and then delete this message from your computer. While > > attachments are virus checked, AWE plc does not accept any liability in > > respect of any virus which is not detected. AWE Plc Registered in England > > and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR > > > > > > ___________________________________________________ ____________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR From bichinhoverde at spwinternet.com.br Wed Apr 1 10:51:23 2015 From: bichinhoverde at spwinternet.com.br (bichinhoverde) Date: Wed, 1 Apr 2015 12:51:23 -0300 Subject: [petsc-users] Linear solver diverging Message-ID: I have a linear system which I can easily solve using the Pardiso library, which is a direct solver. But when I try Petsc, the solver does not converge. It reduces the norm a little but then gets stuck. I tried several combinations of pc_type and ksp_type. It seems that pc=jacobi is better, since my linear system does not have the diagonal element in some rows. Is there some advice you can give me? Some way to analyse the matrix and find out why it does not work? -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Apr 1 10:58:21 2015 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 1 Apr 2015 10:58:21 -0500 Subject: [petsc-users] Linear solver diverging In-Reply-To: References: Message-ID: On Wed, Apr 1, 2015 at 10:51 AM, bichinhoverde < bichinhoverde at spwinternet.com.br> wrote: > I have a linear system which I can easily solve using the Pardiso library, > which is a direct solver. But when I try Petsc, the solver does not > converge. It reduces the norm a little but then gets stuck. > > I tried several combinations of pc_type and ksp_type. It seems that > pc=jacobi is better, since my linear system does not have the diagonal > element in some rows. > > Is there some advice you can give me? Some way to analyse the matrix and > find out why it does not work? > Without knowing anything about the system, there is nothing you can say. Why do you want to change from using a direct solver? Have you looked in the literature for iterative solvers that work on this problem? Thanks, Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bichinhoverde at spwinternet.com.br Wed Apr 1 11:07:42 2015 From: bichinhoverde at spwinternet.com.br (bichinhoverde) Date: Wed, 1 Apr 2015 13:07:42 -0300 Subject: [petsc-users] Linear solver diverging In-Reply-To: References: Message-ID: The system I am trying to solve is a 2D Poisson equation, but split into 5 1D equations. Something like this: p_xx + p_yy = f p_xx - d(p_x)/dx = 0 p_yy - d(p_y)/dy = 0 p_x - d(p)/dx = 0 p_y - d(p)/dy = 0 If I use standard finite differences, it works fine with gmres and jacobi. But when I switch to compact finite differences, only the direct solver works. It is on a staggered mesh, so p, p_xx and p_yy are on the cell center and p_x and p_y are on the edges. I recently submitted a paper and the reviewer complained about the direct linear solver. So I wanted to do a performance comparison. On Wed, Apr 1, 2015 at 12:58 PM, Matthew Knepley wrote: > On Wed, Apr 1, 2015 at 10:51 AM, bichinhoverde < > bichinhoverde at spwinternet.com.br> wrote: > >> I have a linear system which I can easily solve using the Pardiso >> library, which is a direct solver. But when I try Petsc, the solver does >> not converge. It reduces the norm a little but then gets stuck. >> >> I tried several combinations of pc_type and ksp_type. It seems that >> pc=jacobi is better, since my linear system does not have the diagonal >> element in some rows. >> >> Is there some advice you can give me? Some way to analyse the matrix and >> find out why it does not work? >> > > Without knowing anything about the system, there is nothing you can say. > > Why do you want to change from using a direct solver? > > Have you looked in the literature for iterative solvers that work on this > problem? > > Thanks, > > Matt > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Apr 1 11:24:27 2015 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 1 Apr 2015 11:24:27 -0500 Subject: [petsc-users] Linear solver diverging In-Reply-To: References: Message-ID: On Wed, Apr 1, 2015 at 11:07 AM, bichinhoverde < bichinhoverde at spwinternet.com.br> wrote: > The system I am trying to solve is a 2D Poisson equation, but split into 5 > 1D equations. Something like this: > > p_xx + p_yy = f > p_xx - d(p_x)/dx = 0 > p_yy - d(p_y)/dy = 0 > p_x - d(p)/dx = 0 > p_y - d(p)/dy = 0 > > If I use standard finite differences, it works fine with gmres and jacobi. > But when I switch to compact finite differences, only the direct solver > works. > > It is on a staggered mesh, so p, p_xx and p_yy are on the cell center and > p_x and p_y are on the edges. > > I recently submitted a paper and the reviewer complained about the direct > linear solver. So I wanted to do a performance comparison. > I am not sure why the direct solver is a problem, other than unthinking prejudice. If you want to use an iterative solver here, you will likely need to use PCFIELDSPLIT since you have saddle-point structure from the constraints (I think). Thanks, Matt > On Wed, Apr 1, 2015 at 12:58 PM, Matthew Knepley > wrote: > >> On Wed, Apr 1, 2015 at 10:51 AM, bichinhoverde < >> bichinhoverde at spwinternet.com.br> wrote: >> >>> I have a linear system which I can easily solve using the Pardiso >>> library, which is a direct solver. But when I try Petsc, the solver does >>> not converge. It reduces the norm a little but then gets stuck. >>> >>> I tried several combinations of pc_type and ksp_type. It seems that >>> pc=jacobi is better, since my linear system does not have the diagonal >>> element in some rows. >>> >>> Is there some advice you can give me? Some way to analyse the matrix and >>> find out why it does not work? >>> >> >> Without knowing anything about the system, there is nothing you can say. >> >> Why do you want to change from using a direct solver? >> >> Have you looked in the literature for iterative solvers that work on this >> problem? >> >> Thanks, >> >> Matt >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bichinhoverde at spwinternet.com.br Wed Apr 1 11:34:38 2015 From: bichinhoverde at spwinternet.com.br (bichinhoverde) Date: Wed, 1 Apr 2015 13:34:38 -0300 Subject: [petsc-users] Linear solver diverging In-Reply-To: References: Message-ID: The reviewer said: "direct methods are useless for realistic problems". ?Unthinking prejudice? Maybe I will just compare the performance of the non-compact iterative version to the compact direct one and show that it is not that bad. Thanks for the help. On Wed, Apr 1, 2015 at 1:24 PM, Matthew Knepley wrote: > On Wed, Apr 1, 2015 at 11:07 AM, bichinhoverde < > bichinhoverde at spwinternet.com.br> wrote: > >> The system I am trying to solve is a 2D Poisson equation, but split into >> 5 1D equations. Something like this: >> >> p_xx + p_yy = f >> p_xx - d(p_x)/dx = 0 >> p_yy - d(p_y)/dy = 0 >> p_x - d(p)/dx = 0 >> p_y - d(p)/dy = 0 >> >> If I use standard finite differences, it works fine with gmres and >> jacobi. But when I switch to compact finite differences, only the direct >> solver works. >> >> It is on a staggered mesh, so p, p_xx and p_yy are on the cell center and >> p_x and p_y are on the edges. >> >> I recently submitted a paper and the reviewer complained about the direct >> linear solver. So I wanted to do a performance comparison. >> > > I am not sure why the direct solver is a problem, other than > ?? > unthinking prejudice. > > If you want to use an iterative solver here, you will likely need to use > PCFIELDSPLIT since you have saddle-point > structure from the constraints (I think). > > Thanks, > > Matt > > >> On Wed, Apr 1, 2015 at 12:58 PM, Matthew Knepley >> wrote: >> >>> On Wed, Apr 1, 2015 at 10:51 AM, bichinhoverde < >>> bichinhoverde at spwinternet.com.br> wrote: >>> >>>> I have a linear system which I can easily solve using the Pardiso >>>> library, which is a direct solver. But when I try Petsc, the solver does >>>> not converge. It reduces the norm a little but then gets stuck. >>>> >>>> I tried several combinations of pc_type and ksp_type. It seems that >>>> pc=jacobi is better, since my linear system does not have the diagonal >>>> element in some rows. >>>> >>>> Is there some advice you can give me? Some way to analyse the matrix >>>> and find out why it does not work? >>>> >>> >>> Without knowing anything about the system, there is nothing you can say. >>> >>> Why do you want to change from using a direct solver? >>> >>> Have you looked in the literature for iterative solvers that work on >>> this problem? >>> >>> Thanks, >>> >>> Matt >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Apr 1 11:39:21 2015 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 1 Apr 2015 11:39:21 -0500 Subject: [petsc-users] Linear solver diverging In-Reply-To: References: Message-ID: On Wed, Apr 1, 2015 at 11:34 AM, bichinhoverde < bichinhoverde at spwinternet.com.br> wrote: > The reviewer said: "direct methods are useless for realistic problems". > ?Unthinking prejudice? > The phrase realistic problems is stupid on its face. What I can say is that a direct solver applied to the 2D constant coefficient Poisson equation is unlikely to beat geometric multigrid on anything but small problems. Thanks, Matt > Maybe I will just compare the performance of the non-compact iterative > version to the compact direct one and show that it is not that bad. > > Thanks for the help. > > > > > On Wed, Apr 1, 2015 at 1:24 PM, Matthew Knepley wrote: > >> On Wed, Apr 1, 2015 at 11:07 AM, bichinhoverde < >> bichinhoverde at spwinternet.com.br> wrote: >> >>> The system I am trying to solve is a 2D Poisson equation, but split into >>> 5 1D equations. Something like this: >>> >>> p_xx + p_yy = f >>> p_xx - d(p_x)/dx = 0 >>> p_yy - d(p_y)/dy = 0 >>> p_x - d(p)/dx = 0 >>> p_y - d(p)/dy = 0 >>> >>> If I use standard finite differences, it works fine with gmres and >>> jacobi. But when I switch to compact finite differences, only the direct >>> solver works. >>> >>> It is on a staggered mesh, so p, p_xx and p_yy are on the cell center >>> and p_x and p_y are on the edges. >>> >>> I recently submitted a paper and the reviewer complained about the >>> direct linear solver. So I wanted to do a performance comparison. >>> >> >> I am not sure why the direct solver is a problem, other than >> ?? >> unthinking prejudice. >> >> If you want to use an iterative solver here, you will likely need to use >> PCFIELDSPLIT since you have saddle-point >> structure from the constraints (I think). >> >> Thanks, >> >> Matt >> >> >>> On Wed, Apr 1, 2015 at 12:58 PM, Matthew Knepley >>> wrote: >>> >>>> On Wed, Apr 1, 2015 at 10:51 AM, bichinhoverde < >>>> bichinhoverde at spwinternet.com.br> wrote: >>>> >>>>> I have a linear system which I can easily solve using the Pardiso >>>>> library, which is a direct solver. But when I try Petsc, the solver does >>>>> not converge. It reduces the norm a little but then gets stuck. >>>>> >>>>> I tried several combinations of pc_type and ksp_type. It seems that >>>>> pc=jacobi is better, since my linear system does not have the diagonal >>>>> element in some rows. >>>>> >>>>> Is there some advice you can give me? Some way to analyse the matrix >>>>> and find out why it does not work? >>>>> >>>> >>>> Without knowing anything about the system, there is nothing you can say. >>>> >>>> Why do you want to change from using a direct solver? >>>> >>>> Have you looked in the literature for iterative solvers that work on >>>> this problem? >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bourdin at lsu.edu Wed Apr 1 12:04:24 2015 From: bourdin at lsu.edu (Blaise A Bourdin) Date: Wed, 1 Apr 2015 17:04:24 +0000 Subject: [petsc-users] MatStencil / MatSetValuesStencil in petsc4py Message-ID: Hi, I notice that there is not MatSetValuesStencil in petsc4py (only MatSetValueStencil), and that the python name for MatStencil is _Mat_Stencil, so that it must be an internal type. Is there an alternative to MatSetValuesStencil for setting values in a Mat obtained from a DMDA in petsc4py? Regards, Blaise -- Department of Mathematics and Center for Computation & Technology Louisiana State University, Baton Rouge, LA 70803, USA Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin From stm8086 at yahoo.com Wed Apr 1 14:18:29 2015 From: stm8086 at yahoo.com (Steena M) Date: Wed, 1 Apr 2015 19:18:29 +0000 (UTC) Subject: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult In-Reply-To: <656F0F36-03B5-43C2-9FDC-82F1B1337E42@mcs.anl.gov> References: <656F0F36-03B5-43C2-9FDC-82F1B1337E42@mcs.anl.gov> Message-ID: <264546423.1720212.1427915909241.JavaMail.yahoo@mail.yahoo.com> Thanks Barry. I removed the Preallocation calls.?It is still complaining about the malloc and incorrect data in the matrix file. I generate binary matrix files using PETSc's pythonscript to loop through a set of UFL sparse matrices. For this use case: mtx_mat = scipy.io.mmread('trefethen.mtx') PetscBinaryIO.PetscBinaryIO().writeMatSciPy(open('trefnew.dat','w'), mtx_mat) ? On Tuesday, March 31, 2015 9:15 PM, Barry Smith wrote: ? You should not need to call any preallocation routines when using MatLoad() ? How did you generate the file? Are you sure it has the correct information for the matrix? ? Barry > On Mar 31, 2015, at 11:05 PM, Steena M wrote: > > Thanks Matt. I'm still getting the malloc error > > [0]PETSC ERROR: Argument out of range! > [0]PETSC ERROR: New nonzero at (2,18) caused a malloc! > > and > > a new incorrect matrix file error: > > [0]PETSC ERROR: Unexpected data in file! > [0]PETSC ERROR: not matrix object! > > Maybe the order of calls is mixed up. This is the code snippet: > > ??? if (rank ==0) > ??? { > ??? PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); > ??? ? > ??? ??? CHKERRQ(MatSetSizes(A, 15, PETSC_DETERMINE, 20, 20)); > ??? ??? CHKERRQ(MatSetType(A, MATMPIBAIJ)); > ??? ??? CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); > ??? ??? CHKERRQ( MatLoad(A,fd)); > ??? CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); > ??? } > ??? > ??? else > ??? { > ??? ??? PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); > ??? ??? > ??? ??? CHKERRQ( MatSetSizes(A, 5, PETSC_DETERMINE, 20, 20) ); > ??? ??? CHKERRQ(MatSetType(A, MATMPIBAIJ)); > ??? ??? CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); > ??? ??? CHKERRQ(MatLoad(A,fd)); > ??? CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); > ??? ??? } > > Is there something I'm missing? > > Thanks, > Steena > ??? > > > > > On Tuesday, March 31, 2015 6:10 PM, Matthew Knepley wrote: > > > On Tue, Mar 31, 2015 at 6:51 PM, Steena M wrote: > Thanks Barry. I'm still getting the malloc error with NULL. Is there a way to distribute the matrix without explicit preallocation? Different matrices will be loaded during runtime and assigning preallocation parameters would mean an additional preprocessing step. > > 1) MatSetOption(MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) >? > 2) Note that this is never ever ever more efficient than making another pass and preallocating > >? Thanks, > >? ? ? Matt > -------------------------------------------- > On Sun, 3/29/15, Barry Smith wrote: > >? Subject: Re: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult >? To: "Steena M" >? Cc: "Matthew Knepley" , petsc-users at mcs.anl.gov >? Date: Sunday, March 29, 2015, 9:26 PM > > >? > On >? Mar 29, 2015, at 11:05 PM, Steena M >? wrote: >? > >? > Thanks >? Matt. I used PETSC_DETERMINE but I'm now getting an >? allocation-based error: >? > >? > [0]PETSC ERROR: --------------------- >? Error Message ------------------------------------ >? > [0]PETSC ERROR: Argument out of range! >? > [0]PETSC ERROR: New nonzero at (2,18) >? caused a malloc! >? > [0]PETSC ERROR: >? ------------------------------------------------------------------------ >? > >? > I tried >? preallocating on each rank for the diagonal and off diagonal >? section of the matrix as the next step? My current >? approximations for preallocation >? > >? > CHKERRQ( >? MatMPIBAIJSetPreallocation(A,1,5,PETSC_DEFAULT,5,PETSC_DEFAULT)); > > >? ? These >? arguments where you pass PETSC_DEFAULT are expecting a >? pointer not an integer. You can pass NULL in those >? locations. Though it is better to provide the correct >? preallocation rather than some defaults. > >? ? Barry > >? > >? > are throwing segmentation errors. >? > >? > [0]PETSC ERROR: >? Caught signal number 11 SEGV: Segmentation Violation, >? probably memory access out of range >? > >? > Any insights into what I'm doing >? wrong? >? > >? > Thanks, >? > Steena >? > >? > >? > >? > On Sun, 3/29/15, Matthew Knepley >? wrote: >? > >? > Subject: >? Re: [petsc-users] Unequal sparse matrix row distribution for >? MPI MatMult >? > To: "Steena M" >? >? > Cc: "Barry Smith" , >? petsc-users at mcs.anl.gov >? > Date: Sunday, March 29, 2015, 10:02 PM >? > >? > On Sun, Mar 29, 2015 >? at >? > 9:56 PM, Steena M >? > wrote: >? > Hi >? > Barry, >? > >? > >? > >? > I am trying to partition a 20 row and 20 >? col sparse matrix >? > between two procs >? such that proc 0 has 15 rows and 20 cols >? > and proc 1 has 5 rows and 20 cols. The >? code snippet: >? > >? > >? > >? > >? > >? > >? > >? >? ? >? ? CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A)); >? // >? > at runtime: -matload_block_size 1 >? > >? > >? > >? > >? > >? >? ? >? ? if (rank ==0) >? > >? >? ? ? ? { >? > >? >? ? ? ? ? ? >? ? CHKERRQ( MatSetSizes(A, 15, 20, 20, >? > 20) ); //rank 0 gets 75% of the rows >? > >? >? ? ? ? ? ? >? ? CHKERRQ( MatSetType(A, MATMPIBAIJ) >? > ); >? > >? >? ? ? ? ? ? ? ? CHKERRQ( >? MatLoad(A,fd) ); >? > >? >? ? ? ? ? } >? > >? > >? > >? >? ? ? ? else >? > >? >? ? >? ? { >? > >? > >? ? ? ? ? ? ? ? CHKERRQ( MatSetSizes(A, 5, >? 20, 20, >? > 20) ); //rank 1 gets 25% of the >? rows >? > >? >? ? ? >? ? ? ? ? CHKERRQ( MatSetType(A, MATMPIBAIJ) >? > ); >? > >? >? ? ? ? ? ? ? ? CHKERRQ( >? MatLoad(A,fd) ); >? > >? >? ? ? } >? > > >? > >? > >? > This throws the following error (probably >? from psplit.c): >? > >? > >? [1]PETSC ERROR: --------------------- Error Message >? > ------------------------------------ >? > >? > [1]PETSC ERROR: >? Nonconforming object sizes! >? > >? > [1]PETSC ERROR: Sum of local lengths 40 >? does not equal >? > global length 20, my >? local length 20 >? > >? >? likely a call to >? VecSetSizes() or MatSetSizes() is >? > >? wrong. >? > >? > See >? http://www.mcs.anl.gov/petsc/documentation/faq.html#split! >? > >? > >? > >? > This error printout >? doesn't quite make sense to me. >? > >? I'm trying to specify a total matrix size of 20x20... >? I >? > haven't yet figured out where the >? '40' comes >? > from in the error >? message. >? > >? > >? > >? > Any thoughts on what >? might be going wrong? >? > >? > Its the column specification. Just >? > use PETSC_DETERMINE for the local columns >? since all our >? > sparse matrixformats are >? row divisions >? > anyway. >? > >? > Thanks, >? >? >? >? Matt >? > Thanks in advance, >? > > >? > Steena >? > >? > >? > >? > >? > >? > >? > >? > >? -------------------------------------------- >? > >? > On Sun, 3/22/15, >? Barry Smith >? > wrote: >? > >? > >? > >? >? Subject: Re: [petsc-users] Unequal >? sparse matrix row >? > distribution for MPI >? MatMult >? > >? >? To: >? "Steena M" >? > >? >? Cc: petsc-users at mcs.anl.gov >? > >? >? Date: Sunday, >? March 22, 2015, 3:58 PM >? > >? > >? > >? > >? > >? >? >? > >? >? Steena, >? > >? > >? > >? >? ? I am >? > > >? >? a little unsure of your question. > >? > >? > >? > >? >? ? 1) >? you can create a MPIBAIJ >? > >? >? matrix with any distribution of block >? rows per process >? > you >? > >? >? want, just set the >? local row size for each process to >? > >? be >? > >? >? what you >? like.? Use MatCreateVecs() to get >? > >? correspondingly >? > >? > >? laid out vectors. >? > >? > > >? > >? > >? ? or 2) if you have a MPIBAIJ >? > >? >? matrix with >? "equal" row layout and you want a >? > new >? > >? >? one with uneven row layout you can >? simply use >? > >? > >? MatGetSubMatrix() to create that new matrix. >? > >? > >? > >? >? ? Barry >? > >? > >? > >? >? Unless you have >? another reason to have the >? > >? >? matrix with an equal number row layout I >? would just >? > generate >? > >? >? the matrix with >? the layout you want. >? > >? > >? > >? > >? > >? >? > On Mar 22, 2015, at 5:50 PM, Steena >? M >? > >? >? >? > >? >? wrote: >? > >? >? > >? > >? >? > Hello, >? > >? >? > >? > >? >? > I need to >? distribute >? > >? >? a >? sparse matrix such that each proc owns an unequal >? > number >? > >? >? of blocked rows before I proceed with >? MPI MatMult. My >? > >? > >? initial thoughts on doing this: >? > >? >? > >? > >? >? > 1) Use? MatGetSubMatrices() on the >? test >? > >? >? MATMPIBAIJ >? matrix to produce a new matrix where each >? > proc >? > >? >? has an unequal number of rows. >? > >? >? > >? > >? >? > 2) Provide >? scatter context for vector X >? > >? >? (for MatMult )using IS iscol from >? MatGetSubMatrices() >? > while >? > >? >? creating the >? vector X. >? > >? > >? > >? > >? >? > 3) >? Call MatMult() >? > >? > >? > >? > >? >? > Will >? MatMult_MPIBAIJ continue to scatter >? > >? >? this matrix and vector such that each >? proc will own an >? > equal >? > >? >? number of matrix >? rows and corresponding diagonal vector >? > > >? >? elements? Should I write my own >? MPIMatMult function to >? > >? >? retain my redistribution of the matrix >? and vector? >? > >? > >? > >? > >? >? > >? Thanks in >? > >? > >? advance, >? > >? >? > >? Steena >? > >? > >? > >? > >? > >? > >? > -- >? > What most >? experimenters >? > take for granted before >? they begin their experiments is >? > >? infinitely more interesting than any results to which >? their >? > experiments lead. >? > -- Norbert >? > Wiener > >? > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Apr 1 14:24:01 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 1 Apr 2015 14:24:01 -0500 Subject: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult In-Reply-To: <264546423.1720212.1427915909241.JavaMail.yahoo@mail.yahoo.com> References: <656F0F36-03B5-43C2-9FDC-82F1B1337E42@mcs.anl.gov> <264546423.1720212.1427915909241.JavaMail.yahoo@mail.yahoo.com> Message-ID: <4E540218-7EFC-438C-B48B-F7D746929E38@mcs.anl.gov> Send a data file you generated and your reader program and we'll debug it. Barry > On Apr 1, 2015, at 2:18 PM, Steena M wrote: > > Thanks Barry. I removed the Preallocation calls. It is still complaining about the malloc and incorrect data in the matrix file. I generate binary matrix files using PETSc's pythonscript to loop through a set of UFL sparse matrices. For this use case: > > mtx_mat = scipy.io.mmread('trefethen.mtx') > PetscBinaryIO.PetscBinaryIO().writeMatSciPy(open('trefnew.dat','w'), mtx_mat) > > > > > > > On Tuesday, March 31, 2015 9:15 PM, Barry Smith wrote: > > > > You should not need to call any preallocation routines when using MatLoad() > > > How did you generate the file? Are you sure it has the correct information for the matrix? > > Barry > > > > > On Mar 31, 2015, at 11:05 PM, Steena M wrote: > > > > Thanks Matt. I'm still getting the malloc error > > > > [0]PETSC ERROR: Argument out of range! > > [0]PETSC ERROR: New nonzero at (2,18) caused a malloc! > > > > and > > > > a new incorrect matrix file error: > > > > [0]PETSC ERROR: Unexpected data in file! > > [0]PETSC ERROR: not matrix object! > > > > Maybe the order of calls is mixed up. This is the code snippet: > > > > if (rank ==0) > > { > > PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); > > > > CHKERRQ(MatSetSizes(A, 15, PETSC_DETERMINE, 20, 20)); > > CHKERRQ(MatSetType(A, MATMPIBAIJ)); > > CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); > > CHKERRQ( MatLoad(A,fd)); > > CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); > > } > > > > else > > { > > PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); > > > > CHKERRQ( MatSetSizes(A, 5, PETSC_DETERMINE, 20, 20) ); > > CHKERRQ(MatSetType(A, MATMPIBAIJ)); > > CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); > > CHKERRQ(MatLoad(A,fd)); > > CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); > > } > > > > Is there something I'm missing? > > > > Thanks, > > Steena > > > > > > > > > > > > On Tuesday, March 31, 2015 6:10 PM, Matthew Knepley wrote: > > > > > > On Tue, Mar 31, 2015 at 6:51 PM, Steena M wrote: > > Thanks Barry. I'm still getting the malloc error with NULL. Is there a way to distribute the matrix without explicit preallocation? Different matrices will be loaded during runtime and assigning preallocation parameters would mean an additional preprocessing step. > > > > 1) MatSetOption(MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) > > > > 2) Note that this is never ever ever more efficient than making another pass and preallocating > > > > Thanks, > > > > Matt > > -------------------------------------------- > > On Sun, 3/29/15, Barry Smith wrote: > > > > Subject: Re: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult > > To: "Steena M" > > Cc: "Matthew Knepley" , petsc-users at mcs.anl.gov > > Date: Sunday, March 29, 2015, 9:26 PM > > > > > > > On > > Mar 29, 2015, at 11:05 PM, Steena M > > wrote: > > > > > > Thanks > > Matt. I used PETSC_DETERMINE but I'm now getting an > > allocation-based error: > > > > > > [0]PETSC ERROR: --------------------- > > Error Message ------------------------------------ > > > [0]PETSC ERROR: Argument out of range! > > > [0]PETSC ERROR: New nonzero at (2,18) > > caused a malloc! > > > [0]PETSC ERROR: > > ------------------------------------------------------------------------ > > > > > > I tried > > preallocating on each rank for the diagonal and off diagonal > > section of the matrix as the next step My current > > approximations for preallocation > > > > > > CHKERRQ( > > MatMPIBAIJSetPreallocation(A,1,5,PETSC_DEFAULT,5,PETSC_DEFAULT)); > > > > > > These > > arguments where you pass PETSC_DEFAULT are expecting a > > pointer not an integer. You can pass NULL in those > > locations. Though it is better to provide the correct > > preallocation rather than some defaults. > > > > Barry > > > > > > > > are throwing segmentation errors. > > > > > > [0]PETSC ERROR: > > Caught signal number 11 SEGV: Segmentation Violation, > > probably memory access out of range > > > > > > Any insights into what I'm doing > > wrong? > > > > > > Thanks, > > > Steena > > > > > > > > > > > > On Sun, 3/29/15, Matthew Knepley > > wrote: > > > > > > Subject: > > Re: [petsc-users] Unequal sparse matrix row distribution for > > MPI MatMult > > > To: "Steena M" > > > > > Cc: "Barry Smith" , > > petsc-users at mcs.anl.gov > > > Date: Sunday, March 29, 2015, 10:02 PM > > > > > > On Sun, Mar 29, 2015 > > at > > > 9:56 PM, Steena M > > > wrote: > > > Hi > > > Barry, > > > > > > > > > > > > I am trying to partition a 20 row and 20 > > col sparse matrix > > > between two procs > > such that proc 0 has 15 rows and 20 cols > > > and proc 1 has 5 rows and 20 cols. The > > code snippet: > > > > > > > > > > > > > > > > > > > > > > > > > > CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A)); > > // > > > at runtime: -matload_block_size 1 > > > > > > > > > > > > > > > > > > > > if (rank ==0) > > > > > > { > > > > > > > > CHKERRQ( MatSetSizes(A, 15, 20, 20, > > > 20) ); //rank 0 gets 75% of the rows > > > > > > > > CHKERRQ( MatSetType(A, MATMPIBAIJ) > > > ); > > > > > > CHKERRQ( > > MatLoad(A,fd) ); > > > > > > } > > > > > > > > > > > > else > > > > > > > > { > > > > > > > > CHKERRQ( MatSetSizes(A, 5, > > 20, 20, > > > 20) ); //rank 1 gets 25% of the > > rows > > > > > > > > CHKERRQ( MatSetType(A, MATMPIBAIJ) > > > ); > > > > > > CHKERRQ( > > MatLoad(A,fd) ); > > > > > > } > > > > > > > > > > > > > > This throws the following error (probably > > from psplit.c): > > > > > > > > [1]PETSC ERROR: --------------------- Error Message > > > ------------------------------------ > > > > > > [1]PETSC ERROR: > > Nonconforming object sizes! > > > > > > [1]PETSC ERROR: Sum of local lengths 40 > > does not equal > > > global length 20, my > > local length 20 > > > > > > likely a call to > > VecSetSizes() or MatSetSizes() is > > > > > wrong. > > > > > > See > > http://www.mcs.anl.gov/petsc/documentation/faq.html#split! > > > > > > > > > > > > This error printout > > doesn't quite make sense to me. > > > > > I'm trying to specify a total matrix size of 20x20... > > I > > > haven't yet figured out where the > > '40' comes > > > from in the error > > message. > > > > > > > > > > > > Any thoughts on what > > might be going wrong? > > > > > > Its the column specification. Just > > > use PETSC_DETERMINE for the local columns > > since all our > > > sparse matrixformats are > > row divisions > > > anyway. > > > > > > Thanks, > > > > > > Matt > > > Thanks in advance, > > > > > > > > Steena > > > > > > > > > > > > > > > > > > > > > > > > > > -------------------------------------------- > > > > > > On Sun, 3/22/15, > > Barry Smith > > > wrote: > > > > > > > > > > > > Subject: Re: [petsc-users] Unequal > > sparse matrix row > > > distribution for MPI > > MatMult > > > > > > To: > > "Steena M" > > > > > > Cc: petsc-users at mcs.anl.gov > > > > > > Date: Sunday, > > March 22, 2015, 3:58 PM > > > > > > > > > > > > > > > > > > > > > > > > Steena, > > > > > > > > > > > > I am > > > > > > > > a little unsure of your question. > > > > > > > > > > > > > > 1) > > you can create a MPIBAIJ > > > > > > matrix with any distribution of block > > rows per process > > > you > > > > > > want, just set the > > local row size for each process to > > > > > be > > > > > > what you > > like. Use MatCreateVecs() to get > > > > > correspondingly > > > > > > > > laid out vectors. > > > > > > > > > > > > > > > > or 2) if you have a MPIBAIJ > > > > > > matrix with > > "equal" row layout and you want a > > > new > > > > > > one with uneven row layout you can > > simply use > > > > > > > > MatGetSubMatrix() to create that new matrix. > > > > > > > > > > > > Barry > > > > > > > > > > > > Unless you have > > another reason to have the > > > > > > matrix with an equal number row layout I > > would just > > > generate > > > > > > the matrix with > > the layout you want. > > > > > > > > > > > > > > > > > > > On Mar 22, 2015, at 5:50 PM, Steena > > M > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > I need to > > distribute > > > > > > a > > sparse matrix such that each proc owns an unequal > > > number > > > > > > of blocked rows before I proceed with > > MPI MatMult. My > > > > > > > > initial thoughts on doing this: > > > > > > > > > > > > > > 1) Use MatGetSubMatrices() on the > > test > > > > > > MATMPIBAIJ > > matrix to produce a new matrix where each > > > proc > > > > > > has an unequal number of rows. > > > > > > > > > > > > > > 2) Provide > > scatter context for vector X > > > > > > (for MatMult )using IS iscol from > > MatGetSubMatrices() > > > while > > > > > > creating the > > vector X. > > > > > > > > > > > > > > > > 3) > > Call MatMult() > > > > > > > > > > > > > > > > Will > > MatMult_MPIBAIJ continue to scatter > > > > > > this matrix and vector such that each > > proc will own an > > > equal > > > > > > number of matrix > > rows and corresponding diagonal vector > > > > > > > > elements? Should I write my own > > MPIMatMult function to > > > > > > retain my redistribution of the matrix > > and vector? > > > > > > > > > > > > > > > > > > Thanks in > > > > > > > > advance, > > > > > > > > > Steena > > > > > > > > > > > > > > > > > > > > > -- > > > What most > > experimenters > > > take for granted before > > they begin their experiments is > > > > > infinitely more interesting than any results to which > > their > > > experiments lead. > > > -- Norbert > > > Wiener > > > > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > > > From bsmith at mcs.anl.gov Wed Apr 1 14:31:04 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 1 Apr 2015 14:31:04 -0500 Subject: [petsc-users] DMDA halo exchange In-Reply-To: <551AD7A3.8030002@txcorp.com> References: <551AB963.8060003@txcorp.com> <551AD7A3.8030002@txcorp.com> Message-ID: <090DB926-40DF-4E55-8C7D-994C9AA0573F@mcs.anl.gov> Dominic, I have turned the optimization on to skip the unneeded copy with the DMLocalToLocal in the branches barry/optimize-dmlocaltolocal_da and next. It passes all the PETSc tests; let me know if you see any problems. Barry > On Mar 31, 2015, at 12:21 PM, Dominic Meiser wrote: > > Great, thanks very much. > Cheers, > Dominic > > On 03/31/2015 10:24 AM, Barry Smith wrote: >> Dominic, >> >> DMLocalToLocal is the intended usage. There is code inside the VecScatter to detect unneeded copies, which would result in only touching the ghost cells, but for some reason that is not currently turned on, I'll take a look at fixing it in master. >> >> Barry >> >>> On Mar 31, 2015, at 10:12 AM, Dominic Meiser wrote: >>> >>> Hi, >>> >>> I have a DMDA for a structured grid in a 3D FDTD calculation. What is the most efficient way to update the ghost regions after each time step? Currently I update a global vector and then I scatter to a local vector using DMGlobalToLocalBegin/End for the next time step. I also tried DMLocalToLocal with the same source and target vector and that appears to be slightly faster, perhaps due to better cache usage in the copy of the owned region. Ideally I would like to update just the ghost cells from the owned regions. Is there a good way to do that? Something similar to VecGhostUpdateBegin/End? >>> >>> Thanks, >>> Dominic >>> >>> -- >>> Dominic Meiser >>> Tech-X Corporation >>> 5621 Arapahoe Avenue >>> Boulder, CO 80303 >>> USA >>> Telephone: 303-996-2036 >>> Fax: 303-448-7756 >>> www.txcorp.com >>> >> >> > > > -- > Dominic Meiser > Tech-X Corporation > 5621 Arapahoe Avenue > Boulder, CO 80303 > USA > Telephone: 303-996-2036 > Fax: 303-448-7756 > www.txcorp.com > From dmeiser at txcorp.com Wed Apr 1 14:54:35 2015 From: dmeiser at txcorp.com (Dominic Meiser) Date: Wed, 01 Apr 2015 13:54:35 -0600 Subject: [petsc-users] DMDA halo exchange In-Reply-To: <090DB926-40DF-4E55-8C7D-994C9AA0573F@mcs.anl.gov> References: <551AB963.8060003@txcorp.com> <551AD7A3.8030002@txcorp.com> <090DB926-40DF-4E55-8C7D-994C9AA0573F@mcs.anl.gov> Message-ID: <551C4CFB.6080801@txcorp.com> On 04/01/2015 01:31 PM, Barry Smith wrote: > Dominic, > > I have turned the optimization on to skip the unneeded copy with the DMLocalToLocal in the branches barry/optimize-dmlocaltolocal_da and next. It passes all the PETSc tests; let me know if you see any problems. Great, thanks very much. I'll give it a shot. Cheers, Dominic > > Barry > >> On Mar 31, 2015, at 12:21 PM, Dominic Meiser wrote: >> >> Great, thanks very much. >> Cheers, >> Dominic >> >> On 03/31/2015 10:24 AM, Barry Smith wrote: >>> Dominic, >>> >>> DMLocalToLocal is the intended usage. There is code inside the VecScatter to detect unneeded copies, which would result in only touching the ghost cells, but for some reason that is not currently turned on, I'll take a look at fixing it in master. >>> >>> Barry >>> >>>> On Mar 31, 2015, at 10:12 AM, Dominic Meiser wrote: >>>> >>>> Hi, >>>> >>>> I have a DMDA for a structured grid in a 3D FDTD calculation. What is the most efficient way to update the ghost regions after each time step? Currently I update a global vector and then I scatter to a local vector using DMGlobalToLocalBegin/End for the next time step. I also tried DMLocalToLocal with the same source and target vector and that appears to be slightly faster, perhaps due to better cache usage in the copy of the owned region. Ideally I would like to update just the ghost cells from the owned regions. Is there a good way to do that? Something similar to VecGhostUpdateBegin/End? >>>> >>>> Thanks, >>>> Dominic >>>> >>>> -- >>>> Dominic Meiser >>>> Tech-X Corporation >>>> 5621 Arapahoe Avenue >>>> Boulder, CO 80303 >>>> USA >>>> Telephone: 303-996-2036 >>>> Fax: 303-448-7756 >>>> www.txcorp.com >>>> >>> >> >> -- >> Dominic Meiser >> Tech-X Corporation >> 5621 Arapahoe Avenue >> Boulder, CO 80303 >> USA >> Telephone: 303-996-2036 >> Fax: 303-448-7756 >> www.txcorp.com >> > > -- Dominic Meiser Tech-X Corporation 5621 Arapahoe Avenue Boulder, CO 80303 USA Telephone: 303-996-2036 Fax: 303-448-7756 www.txcorp.com From stm8086 at yahoo.com Wed Apr 1 15:10:53 2015 From: stm8086 at yahoo.com (Steena M) Date: Wed, 1 Apr 2015 20:10:53 +0000 (UTC) Subject: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult In-Reply-To: <4E540218-7EFC-438C-B48B-F7D746929E38@mcs.anl.gov> References: <4E540218-7EFC-438C-B48B-F7D746929E38@mcs.anl.gov> Message-ID: <441091432.3174678.1427919054040.JavaMail.yahoo@mail.yahoo.com> Thanks Barry. Attached is the driver program, the binary mat file, and the corresponding mtx file. Runtime command used: sierra324 at monteiro:time srun -n 2 -ppdebug ./petsc-mpibaij-unequalrows -fin trefethen.dat -matload_block_size 1 On Wednesday, April 1, 2015 12:28 PM, Barry Smith wrote: ? Send a data file you generated and your reader program and we'll debug it. ? Barry > On Apr 1, 2015, at 2:18 PM, Steena M wrote: > > Thanks Barry. I removed the Preallocation calls. It is still complaining about the malloc and incorrect data in the matrix file. I generate binary matrix files using PETSc's pythonscript to loop through a set of UFL sparse matrices. For this use case: > > mtx_mat = scipy.io.mmread('trefethen.mtx') > PetscBinaryIO.PetscBinaryIO().writeMatSciPy(open('trefnew.dat','w'), mtx_mat) > > >? > > > > On Tuesday, March 31, 2015 9:15 PM, Barry Smith wrote: > > > >? You should not need to call any preallocation routines when using MatLoad() > > >? How did you generate the file? Are you sure it has the correct information for the matrix? > >? Barry > > > > > On Mar 31, 2015, at 11:05 PM, Steena M wrote: > > > > Thanks Matt. I'm still getting the malloc error > > > > [0]PETSC ERROR: Argument out of range! > > [0]PETSC ERROR: New nonzero at (2,18) caused a malloc! > > > > and > > > > a new incorrect matrix file error: > > > > [0]PETSC ERROR: Unexpected data in file! > > [0]PETSC ERROR: not matrix object! > > > > Maybe the order of calls is mixed up. This is the code snippet: > > > >? ? if (rank ==0) > >? ? { > >? ? PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); > >? ? ? > >? ? ? ? CHKERRQ(MatSetSizes(A, 15, PETSC_DETERMINE, 20, 20)); > >? ? ? ? CHKERRQ(MatSetType(A, MATMPIBAIJ)); > >? ? ? ? CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); > >? ? ? ? CHKERRQ( MatLoad(A,fd)); > >? ? CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); > >? ? } > >? ? > >? ? else > >? ? { > >? ? ? ? PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); > >? ? ? ? > >? ? ? ? CHKERRQ( MatSetSizes(A, 5, PETSC_DETERMINE, 20, 20) ); > >? ? ? ? CHKERRQ(MatSetType(A, MATMPIBAIJ)); > >? ? ? ? CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); > >? ? ? ? CHKERRQ(MatLoad(A,fd)); > >? ? CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); > >? ? ? ? } > > > > Is there something I'm missing? > > > > Thanks, > > Steena > >? ? > > > > > > > > > > On Tuesday, March 31, 2015 6:10 PM, Matthew Knepley wrote: > > > > > > On Tue, Mar 31, 2015 at 6:51 PM, Steena M wrote: > > Thanks Barry. I'm still getting the malloc error with NULL. Is there a way to distribute the matrix without explicit preallocation? Different matrices will be loaded during runtime and assigning preallocation parameters would mean an additional preprocessing step. > > > > 1) MatSetOption(MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) > >? > > 2) Note that this is never ever ever more efficient than making another pass and preallocating > > > >? Thanks, > > > >? ? ? Matt > > -------------------------------------------- > > On Sun, 3/29/15, Barry Smith wrote: > > > >? Subject: Re: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult > >? To: "Steena M" > >? Cc: "Matthew Knepley" , petsc-users at mcs.anl.gov > >? Date: Sunday, March 29, 2015, 9:26 PM > > > > > >? > On > >? Mar 29, 2015, at 11:05 PM, Steena M > >? wrote: > >? > > >? > Thanks > >? Matt. I used PETSC_DETERMINE but I'm now getting an > >? allocation-based error: > >? > > >? > [0]PETSC ERROR: --------------------- > >? Error Message ------------------------------------ > >? > [0]PETSC ERROR: Argument out of range! > >? > [0]PETSC ERROR: New nonzero at (2,18) > >? caused a malloc! > >? > [0]PETSC ERROR: > >? ------------------------------------------------------------------------ > >? > > >? > I tried > >? preallocating on each rank for the diagonal and off diagonal > >? section of the matrix as the next step? My current > >? approximations for preallocation > >? > > >? > CHKERRQ( > >? MatMPIBAIJSetPreallocation(A,1,5,PETSC_DEFAULT,5,PETSC_DEFAULT)); > > > > > >? ? These > >? arguments where you pass PETSC_DEFAULT are expecting a > >? pointer not an integer. You can pass NULL in those > >? locations. Though it is better to provide the correct > >? preallocation rather than some defaults. > > > >? ? Barry > > > >? > > >? > are throwing segmentation errors. > >? > > >? > [0]PETSC ERROR: > >? Caught signal number 11 SEGV: Segmentation Violation, > >? probably memory access out of range > >? > > >? > Any insights into what I'm doing > >? wrong? > >? > > >? > Thanks, > >? > Steena > >? > > >? > > >? > > >? > On Sun, 3/29/15, Matthew Knepley > >? wrote: > >? > > >? > Subject: > >? Re: [petsc-users] Unequal sparse matrix row distribution for > >? MPI MatMult > >? > To: "Steena M" > >? > >? > Cc: "Barry Smith" , > >? petsc-users at mcs.anl.gov > >? > Date: Sunday, March 29, 2015, 10:02 PM > >? > > >? > On Sun, Mar 29, 2015 > >? at > >? > 9:56 PM, Steena M > >? > wrote: > >? > Hi > >? > Barry, > >? > > >? > > >? > > >? > I am trying to partition a 20 row and 20 > >? col sparse matrix > >? > between two procs > >? such that proc 0 has 15 rows and 20 cols > >? > and proc 1 has 5 rows and 20 cols. The > >? code snippet: > >? > > >? > > >? > > >? > > >? > > >? > > >? > > >? >? ? > >? ? CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A)); > >? // > >? > at runtime: -matload_block_size 1 > >? > > >? > > >? > > >? > > >? > > >? >? ? > >? ? if (rank ==0) > >? > > >? >? ? ? ? { > >? > > >? >? ? ? ? ? ? > >? ? CHKERRQ( MatSetSizes(A, 15, 20, 20, > >? > 20) ); //rank 0 gets 75% of the rows > >? > > >? >? ? ? ? ? ? > >? ? CHKERRQ( MatSetType(A, MATMPIBAIJ) > >? > ); > >? > > >? >? ? ? ? ? ? ? ? CHKERRQ( > >? MatLoad(A,fd) ); > >? > > >? >? ? ? ? ? } > >? > > >? > > >? > > >? >? ? ? ? else > >? > > >? >? ? > >? ? { > >? > > >? > > >? ? ? ? ? ? ? ? CHKERRQ( MatSetSizes(A, 5, > >? 20, 20, > >? > 20) ); //rank 1 gets 25% of the > >? rows > >? > > >? >? ? ? > >? ? ? ? ? CHKERRQ( MatSetType(A, MATMPIBAIJ) > >? > ); > >? > > >? >? ? ? ? ? ? ? ? CHKERRQ( > >? MatLoad(A,fd) ); > >? > > >? >? ? ? } > >? > > > > >? > > >? > > >? > This throws the following error (probably > >? from psplit.c): > >? > > >? > > >? [1]PETSC ERROR: --------------------- Error Message > >? > ------------------------------------ > >? > > >? > [1]PETSC ERROR: > >? Nonconforming object sizes! > >? > > >? > [1]PETSC ERROR: Sum of local lengths 40 > >? does not equal > >? > global length 20, my > >? local length 20 > >? > > >? >? likely a call to > >? VecSetSizes() or MatSetSizes() is > >? > > >? wrong. > >? > > >? > See > >? http://www.mcs.anl.gov/petsc/documentation/faq.html#split! > >? > > >? > > >? > > >? > This error printout > >? doesn't quite make sense to me. > >? > > >? I'm trying to specify a total matrix size of 20x20... > >? I > >? > haven't yet figured out where the > >? '40' comes > >? > from in the error > >? message. > >? > > >? > > >? > > >? > Any thoughts on what > >? might be going wrong? > >? > > >? > Its the column specification. Just > >? > use PETSC_DETERMINE for the local columns > >? since all our > >? > sparse matrixformats are > >? row divisions > >? > anyway. > >? > > >? > Thanks, > >? >? > >? >? Matt > >? > Thanks in advance, > >? > > > > >? > Steena > >? > > >? > > >? > > >? > > >? > > >? > > >? > > >? > > >? -------------------------------------------- > >? > > >? > On Sun, 3/22/15, > >? Barry Smith > >? > wrote: > >? > > >? > > >? > > >? >? Subject: Re: [petsc-users] Unequal > >? sparse matrix row > >? > distribution for MPI > >? MatMult > >? > > >? >? To: > >? "Steena M" > >? > > >? >? Cc: petsc-users at mcs.anl.gov > >? > > >? >? Date: Sunday, > >? March 22, 2015, 3:58 PM > >? > > >? > > >? > > >? > > >? > > >? >? > >? > > >? >? Steena, > >? > > >? > > >? > > >? >? ? I am > >? > > > > >? >? a little unsure of your question. > > > >? > > >? > > >? > > >? >? ? 1) > >? you can create a MPIBAIJ > >? > > >? >? matrix with any distribution of block > >? rows per process > >? > you > >? > > >? >? want, just set the > >? local row size for each process to > >? > > >? be > >? > > >? >? what you > >? like.? Use MatCreateVecs() to get > >? > > >? correspondingly > >? > > >? > > >? laid out vectors. > >? > > >? > > > > >? > > >? > > >? ? or 2) if you have a MPIBAIJ > >? > > >? >? matrix with > >? "equal" row layout and you want a > >? > new > >? > > >? >? one with uneven row layout you can > >? simply use > >? > > >? > > >? MatGetSubMatrix() to create that new matrix. > >? > > >? > > >? > > >? >? ? Barry > >? > > >? > > >? > > >? >? Unless you have > >? another reason to have the > >? > > >? >? matrix with an equal number row layout I > >? would just > >? > generate > >? > > >? >? the matrix with > >? the layout you want. > >? > > >? > > >? > > >? > > >? > > >? >? > On Mar 22, 2015, at 5:50 PM, Steena > >? M > >? > > >? >? > >? > > >? >? wrote: > >? > > >? >? > > >? > > >? >? > Hello, > >? > > >? >? > > >? > > >? >? > I need to > >? distribute > >? > > >? >? a > >? sparse matrix such that each proc owns an unequal > >? > number > >? > > >? >? of blocked rows before I proceed with > >? MPI MatMult. My > >? > > >? > > >? initial thoughts on doing this: > >? > > >? >? > > >? > > >? >? > 1) Use? MatGetSubMatrices() on the > >? test > >? > > >? >? MATMPIBAIJ > >? matrix to produce a new matrix where each > >? > proc > >? > > >? >? has an unequal number of rows. > >? > > >? >? > > >? > > >? >? > 2) Provide > >? scatter context for vector X > >? > > >? >? (for MatMult )using IS iscol from > >? MatGetSubMatrices() > >? > while > >? > > >? >? creating the > >? vector X. > >? > > >? > > >? > > >? > > >? >? > 3) > >? Call MatMult() > >? > > >? > > >? > > >? > > >? >? > Will > >? MatMult_MPIBAIJ continue to scatter > >? > > >? >? this matrix and vector such that each > >? proc will own an > >? > equal > >? > > >? >? number of matrix > >? rows and corresponding diagonal vector > >? > > > > >? >? elements? Should I write my own > >? MPIMatMult function to > >? > > >? >? retain my redistribution of the matrix > >? and vector? > >? > > >? > > >? > > >? > > >? >? > > >? Thanks in > >? > > >? > > >? advance, > >? > > >? >? > > >? Steena > >? > > >? > > >? > > >? > > >? > > >? > > >? > -- > >? > What most > >? experimenters > >? > take for granted before > >? they begin their experiments is > >? > > >? infinitely more interesting than any results to which > >? their > >? > experiments lead. > >? > -- Norbert > >? > Wiener > > > >? > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: trefethen.mtx Type: application/octet-stream Size: 2405 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: trefethen.dat Type: application/octet-stream Size: 1992 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: petsc-unequalrows-mpibaij.c Type: application/octet-stream Size: 1697 bytes Desc: not available URL: From ansp6066 at colorado.edu Wed Apr 1 15:32:26 2015 From: ansp6066 at colorado.edu (Andrew Spott) Date: Wed, 01 Apr 2015 13:32:26 -0700 (PDT) Subject: [petsc-users] getting EPS to rebuild the BV context Message-ID: <1427920346090.79bc021@Nodemailer> When resizing an EPS (adjusting the nev, ncv and/or mpd values) after a deflation space has been entered, SLEPC complains that you can?t ?resize a BV with constraints?. Is there a way of forcing this issue by killing the BV and remaking it? ?(re-adding the constraints). -Andrew -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Wed Apr 1 17:00:17 2015 From: jroman at dsic.upv.es (Jose E. Roman) Date: Thu, 2 Apr 2015 00:00:17 +0200 Subject: [petsc-users] getting EPS to rebuild the BV context In-Reply-To: <1427920346090.79bc021@Nodemailer> References: <1427920346090.79bc021@Nodemailer> Message-ID: <53546267-34AE-4DB1-9E1F-95D494FB71D7@dsic.upv.es> El 01/04/2015, a las 22:32, Andrew Spott escribi?: > When resizing an EPS (adjusting the nev, ncv and/or mpd values) after a deflation space has been entered, SLEPC complains that you can?t ?resize a BV with constraints?. > > Is there a way of forcing this issue by killing the BV and remaking it? (re-adding the constraints). > > -Andrew > This case is not handled well. A workaround is to call EPSReset(), but probably there should be a better way to do this. Are you using master? Jose From ansp6066 at colorado.edu Wed Apr 1 17:03:24 2015 From: ansp6066 at colorado.edu (Andrew Spott) Date: Wed, 01 Apr 2015 15:03:24 -0700 (PDT) Subject: [petsc-users] getting EPS to rebuild the BV context In-Reply-To: <53546267-34AE-4DB1-9E1F-95D494FB71D7@dsic.upv.es> References: <53546267-34AE-4DB1-9E1F-95D494FB71D7@dsic.upv.es> Message-ID: <1427925803627.f361ecb4@Nodemailer> >Are you using master?This problem was run into on 3.5.3, but I?m working on trying to get the code to work on master. A cursory glance at the code for master seems to tell me that this problem hasn?t changed. -Andrew On Wed, Apr 1, 2015 at 4:00 PM, Jose E. Roman wrote: > El 01/04/2015, a las 22:32, Andrew Spott escribi?: >> When resizing an EPS (adjusting the nev, ncv and/or mpd values) after a deflation space has been entered, SLEPC complains that you can?t ?resize a BV with constraints?. >> >> Is there a way of forcing this issue by killing the BV and remaking it? (re-adding the constraints). >> >> -Andrew >> > This case is not handled well. A workaround is to call EPSReset(), but probably there should be a better way to do this. Are you using master? > Jose -------------- next part -------------- An HTML attachment was scrubbed... URL: From juris.vencels at gmail.com Wed Apr 1 17:09:28 2015 From: juris.vencels at gmail.com (Juris Vencels) Date: Wed, 01 Apr 2015 16:09:28 -0600 Subject: [petsc-users] Preconditioning Matrix-free SNES, preconditioner update, PCView In-Reply-To: <0D2B2AB5-D789-49F6-9EF8-81399BB7A100@mcs.anl.gov> References: <551B51E1.8060901@gmail.com> <0D2B2AB5-D789-49F6-9EF8-81399BB7A100@mcs.anl.gov> Message-ID: <551C6C98.3030605@gmail.com> On 03/31/2015 08:26 PM, Barry Smith wrote: > No, the model is that each time the matrix changes the PCSetUp is called automatically and so is a shell set up if you have provided it, so you need to change your PCShellSetUp() function so that it can be called repeatedly for each new matrix and call > >> >call PCShellSetApply(pc,PCShellSetUp,ierr) > when you create the Shell PC. > > Then make a test run in the debugger to make sure that your setup is being called each time the matrix changes. > It works, thank you! > Note: also if you are using ILU preconditioning, you do not need to use PCSHELL, you can just use PCSetType(pc,PCILU) or -pc_type ilu. I am confused about this, I am using matrix-free SNES and Manual Section 5.5. says: > The matrix-free variant is allowed only when the linear systems are > solved by an it- > erative method in combination with no preconditioning (PCNONE or > -pc_type none), a user-provided > preconditioner matrix, or a user-provided preconditioner shell > (PCSHELL, discussed in Section 4.4); ... PETSc has function "PCShellSetApply " which sets routine to use as preconditioner. How can I set a routine if I do not use PCSHELL? There is no such function as " PCSetApply" anymore. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Wed Apr 1 17:25:43 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 2 Apr 2015 01:25:43 +0300 Subject: [petsc-users] running Petsc programs without a connection. In-Reply-To: References: Message-ID: On 26 March 2015 at 10:15, Sanjay Kharche wrote: > Is there a way of configuring or otherwise to get Petsc programs to run when I do not have an internet connection? Make sure you have localhost defined in /etc/hosts, I'm had the same issue in Fedora with fresh installs. $ cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 kw2060 kw2060.kaust.edu.sa ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From bsmith at mcs.anl.gov Wed Apr 1 17:37:10 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 1 Apr 2015 17:37:10 -0500 Subject: [petsc-users] Preconditioning Matrix-free SNES, preconditioner update, PCView In-Reply-To: <551C6C98.3030605@gmail.com> References: <551B51E1.8060901@gmail.com> <0D2B2AB5-D789-49F6-9EF8-81399BB7A100@mcs.anl.gov> <551C6C98.3030605@gmail.com> Message-ID: <229AF327-BF2D-48BC-8FA1-C86CB8D06615@mcs.anl.gov> > On Apr 1, 2015, at 5:09 PM, Juris Vencels wrote: > > On 03/31/2015 08:26 PM, Barry Smith wrote: >> No, the model is that each time the matrix changes the PCSetUp is called automatically and so is a shell set up if you have provided it, so you need to change your PCShellSetUp() function so that it can be called repeatedly for each new matrix and call >> >> >>> > >>> call PCShellSetApply(pc,PCShellSetUp,ierr) >>> >> when you create the Shell PC. >> >> Then make a test run in the debugger to make sure that your setup is being called each time the matrix changes. >> >> > > It works, thank you! > >> Note: also if you are using ILU preconditioning, you do not need to use PCSHELL, you can just use PCSetType(pc,PCILU) or -pc_type ilu. > > I am confused about this, I am using matrix-free SNES and Manual Section 5.5. says: > >> The matrix-free variant is allowed only when the linear systems are solved by an it- >> erative method in combination with no preconditioning (PCNONE or -pc_type none), a user-provided >> preconditioner matrix, or a user-provided preconditioner shell (PCSHELL, discussed in Section 4.4); ... > > PETSc has function "PCShellSetApply " which sets routine to use as preconditioner. > How can I set a routine if I do not use PCSHELL? There is no such function as " PCSetApply" anymore. One can use -snes_mf_operator this causes the operator that DEFINES the linear system to be matrix free. Meanwhile one still builds an explicit approximation to the Jacobian as the SECOND matrix argument to SNESSetJacobian(). The preconditioner is then buildt from this second matrix. So if one does -pc_type ilu the preconditioner is built by doing ILU on that second matrix. If you use -snes_mf you need to use a PCSHELL but if you have an explicit matrix from which you are going to build the preconditioner then there is no reason to use PCSHELL (which is more complicated) because you can use -snes_mf_operator instead and have PETSc manage the preconditioner in the usual manner with, for example, -pc_type ilu etc Barry From david.knezevic at akselos.com Wed Apr 1 22:29:56 2015 From: david.knezevic at akselos.com (David Knezevic) Date: Wed, 1 Apr 2015 23:29:56 -0400 Subject: [petsc-users] Error configuring with --download-ml and Intel MKL In-Reply-To: References: Message-ID: On Thu, Mar 26, 2015 at 6:23 PM, Matthew Knepley wrote: > On Thu, Mar 26, 2015 at 5:10 PM, David Knezevic < > david.knezevic at akselos.com> wrote: > >> On Thu, Mar 26, 2015 at 4:21 PM, Matthew Knepley >> wrote: >> >>> On Thu, Mar 26, 2015 at 1:48 PM, David Knezevic < >>> david.knezevic at akselos.com> wrote: >>> >>>> Hi all, >>>> >>>> I'm trying to configure PETSc using Intel's MKL and with --download-ml. >>>> Here is my configure line: >>>> >>>> ./configure >>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015/mkl/lib/intel64 >>>> --download-ml >>>> >>>> I get an error when ML does "checking for dgemm". The configure.log is >>>> attached. I was wondering if anyone has any suggestions about how I can get >>>> this to work? >>>> >>> >>> We will need $PETSC_ARCH/externalpackages/ml-*/config.log to see where >>> their test failed. >>> >> >> >> I've attached this file. >> > > I do not understand the error: > > configure:10865: mpicxx -o conftest -Wall -Wwrite-strings > -Wno-strict-aliasing -Wno-unknown-pragmas -g -O0 -fPIC > -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX -I/usr/lib/openmpi/include > -I/usr/lib/openmpi/include/openmpi -g -O2 -I/usr/lib/openmpi/include > -I/usr/lib/openmpi/include/openmpi conftest.cpp > -Wl,-rpath,/opt/intel/composer_xe_2015/mkl/lib/intel64 > -L/opt/intel/composer_xe_2015/mkl/lib/intel64 -lmkl_intel_lp64 > -lmkl_sequential -lmkl_core -lpthread -lm -Wl,-rpath,/usr/lib/openmpi/lib > -L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 > -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu > -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm > -Wl,-rpath,/usr/lib/openmpi/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 > -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu > -lgfortran -lm -lquadmath -lm -L/usr//lib -L/usr/lib/openmpi/lib > -L/usr/lib/gcc/x86_64-linux-gnu/4.8 > -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu > -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib -L/lib/x86_64-linux-gnu > -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib > -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../.. -lmpi_f90 -lmpi_f77 -lmpi -ldl > -lhwloc -lgfortran -lm -lquadmath -lpthread >&5 > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined > reference to `logf' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined > reference to `atan2' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined > reference to `sin' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined > reference to `fabs' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined > reference to `exp' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined > reference to `cos' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined > reference to `sqrt' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_intel_lp64.so: > undefined reference to `log' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined > reference to `pow' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined > reference to `log10' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined > reference to `ceil' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined > reference to `expf' > collect2: error: ld returned 1 exit status > > Clearly these symbols should be in -lm, which is in the link line, and it > linked when we tried in our configure. Can > you run this link line manually and figure out what is wrong? > I didn't get to the bottom of this issue with ML+MKL yet, but I have another case that's more pressing right now, and I was wondering if you might be able to help me out with it. The configuration I'm trying is as follows: ./configure --with-scalar-type=complex --with-clanguage=cxx --download-metis --download-parmetis --download-superlu_dist --with-blas-lapack-dir=/path/to/mkl So I'm still trying to use MKL for the time being (though I certainly appreciate your input about the drawbacks of this!), along with SuperLU_dist in complex mode. This configuration fails for me. I don't see any configuration log file in $PETSC_ARCH/externalpackages/SuperLU_DIST, is it supposed to be there somewhere? The error in the PETSc configure.log is pasted below. Interestingly if I configure in real mode (by removing "--with-scalar-type=complex --with-clanguage=cxx") then it works fine. If anyone has any suggestions about what I might be able to do to fix this, that'd be most appreciated. Thanks! David ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Downloaded superlu_dist could not be used. Please check install in /home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug ******************************************************************************* File "./config/configure.py", line 272, in petsc_configure framework.configure(out = sys.stdout) File "/home/dknez/software/petsc-3.5.2/config/BuildSystem/config/framework.py", line 935, in configure child.configure() File "/home/dknez/software/petsc-3.5.2/config/BuildSystem/config/package.py", line 606, in configure self.executeTest(self.configureLibrary) File "/home/dknez/software/petsc-3.5.2/config/BuildSystem/config/base.py", line 126, in executeTest ret = test(*args,**kargs) File "/home/dknez/software/petsc-3.5.2/config/BuildSystem/config/package.py", line 536, in configureLibrary for location, directory, lib, incl in self.generateGuesses(): File "/home/dknez/software/petsc-3.5.2/config/BuildSystem/config/package.py", line 273, in generateGuesses raise RuntimeError('Downloaded '+self.package+' could not be used. Please check install in '+d+'\n') ================================================================================ Finishing Configure Run at Wed Apr 1 23:09:45 2015 ================================================================================ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Apr 1 22:46:34 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 1 Apr 2015 22:46:34 -0500 Subject: [petsc-users] Error configuring with --download-ml and Intel MKL In-Reply-To: References: Message-ID: <427013B3-7C56-438E-AB98-1EC0DF2B08B3@mcs.anl.gov> Send configure.log from the failed configure > On Apr 1, 2015, at 10:29 PM, David Knezevic wrote: > > On Thu, Mar 26, 2015 at 6:23 PM, Matthew Knepley wrote: > On Thu, Mar 26, 2015 at 5:10 PM, David Knezevic wrote: > On Thu, Mar 26, 2015 at 4:21 PM, Matthew Knepley wrote: > On Thu, Mar 26, 2015 at 1:48 PM, David Knezevic wrote: > Hi all, > > I'm trying to configure PETSc using Intel's MKL and with --download-ml. Here is my configure line: > > ./configure --with-blas-lapack-dir=/opt/intel/composer_xe_2015/mkl/lib/intel64 --download-ml > > I get an error when ML does "checking for dgemm". The configure.log is attached. I was wondering if anyone has any suggestions about how I can get this to work? > > We will need $PETSC_ARCH/externalpackages/ml-*/config.log to see where their test failed. > > > I've attached this file. > > I do not understand the error: > > configure:10865: mpicxx -o conftest -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O0 -fPIC -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -g -O2 -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi conftest.cpp -Wl,-rpath,/opt/intel/composer_xe_2015/mkl/lib/intel64 -L/opt/intel/composer_xe_2015/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm -Wl,-rpath,/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -lgfortran -lm -lquadmath -lm -L/usr//lib -L/usr/lib/openmpi/lib -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../.. -lmpi_f90 -lmpi_f77 -lmpi -ldl -lhwloc -lgfortran -lm -lquadmath -lpthread >&5 > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined reference to `logf' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined reference to `atan2' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined reference to `sin' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined reference to `fabs' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined reference to `exp' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined reference to `cos' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined reference to `sqrt' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_intel_lp64.so: undefined reference to `log' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined reference to `pow' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined reference to `log10' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined reference to `ceil' > /opt/intel/composer_xe_2015/mkl/lib/intel64/libmkl_core.so: undefined reference to `expf' > collect2: error: ld returned 1 exit status > > Clearly these symbols should be in -lm, which is in the link line, and it linked when we tried in our configure. Can > you run this link line manually and figure out what is wrong? > > > I didn't get to the bottom of this issue with ML+MKL yet, but I have another case that's more pressing right now, and I was wondering if you might be able to help me out with it. The configuration I'm trying is as follows: > > ./configure --with-scalar-type=complex --with-clanguage=cxx --download-metis --download-parmetis --download-superlu_dist --with-blas-lapack-dir=/path/to/mkl > > So I'm still trying to use MKL for the time being (though I certainly appreciate your input about the drawbacks of this!), along with SuperLU_dist in complex mode. This configuration fails for me. I don't see any configuration log file in $PETSC_ARCH/externalpackages/SuperLU_DIST, is it supposed to be there somewhere? > > The error in the PETSc configure.log is pasted below. Interestingly if I configure in real mode (by removing "--with-scalar-type=complex --with-clanguage=cxx") then it works fine. > > If anyone has any suggestions about what I might be able to do to fix this, that'd be most appreciated. > > Thanks! > David > > > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > Downloaded superlu_dist could not be used. Please check install in /home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug > ******************************************************************************* > File "./config/configure.py", line 272, in petsc_configure > framework.configure(out = sys.stdout) > File "/home/dknez/software/petsc-3.5.2/config/BuildSystem/config/framework.py", line 935, in configure > child.configure() > File "/home/dknez/software/petsc-3.5.2/config/BuildSystem/config/package.py", line 606, in configure > self.executeTest(self.configureLibrary) > File "/home/dknez/software/petsc-3.5.2/config/BuildSystem/config/base.py", line 126, in executeTest > ret = test(*args,**kargs) > File "/home/dknez/software/petsc-3.5.2/config/BuildSystem/config/package.py", line 536, in configureLibrary > for location, directory, lib, incl in self.generateGuesses(): > File "/home/dknez/software/petsc-3.5.2/config/BuildSystem/config/package.py", line 273, in generateGuesses > raise RuntimeError('Downloaded '+self.package+' could not be used. Please check install in '+d+'\n') > ================================================================================ > Finishing Configure Run at Wed Apr 1 23:09:45 2015 > ================================================================================ > > > From david.knezevic at akselos.com Wed Apr 1 22:58:11 2015 From: david.knezevic at akselos.com (David Knezevic) Date: Wed, 1 Apr 2015 23:58:11 -0400 Subject: [petsc-users] Error configuring with --download-ml and Intel MKL In-Reply-To: <427013B3-7C56-438E-AB98-1EC0DF2B08B3@mcs.anl.gov> References: <427013B3-7C56-438E-AB98-1EC0DF2B08B3@mcs.anl.gov> Message-ID: On Wed, Apr 1, 2015 at 11:46 PM, Barry Smith wrote: > > Send configure.log from the failed configure > It's attached. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 4467346 bytes Desc: not available URL: From balay at mcs.anl.gov Wed Apr 1 23:06:39 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 1 Apr 2015 23:06:39 -0500 Subject: [petsc-users] Error configuring with --download-ml and Intel MKL In-Reply-To: References: <427013B3-7C56-438E-AB98-1EC0DF2B08B3@mcs.anl.gov> Message-ID: On Wed, 1 Apr 2015, David Knezevic wrote: > On Wed, Apr 1, 2015 at 11:46 PM, Barry Smith wrote: > > > > > Send configure.log from the failed configure > > > > > It's attached. > Same issue as before - for some reason mpicxx is not finding math symbols in -lm Satish ----- Executing: mpicxx -o /tmp/petsc-rljk4x/config.libraries/conftest -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O0 /tmp/petsc-rljk4x/config.libraries/conftest.o -Wl,-rpath,/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib64 -L/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib64 -lsuperlu_dist_3.3 -Wl,-rpath,/opt/intel/system_studio_2015.2.050/mkl/lib/intel64 -L/opt/intel/system_studio_2015.2.050/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm -Wl,-rpath,/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -lgfortran -lm -lquadmath -lm -Wl,-rpath,/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib -L/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib -lparmetis -Wl,-rpath,/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib -L/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib -lmetis -lm -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s -lpthread -ldl Possible ERROR while running linker: exit code 256 stderr: /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `logf' /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `atan2' /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `sin' /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `fabs' /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `exp' /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `cos' /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `sqrt' /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_intel_lp64.so: undefined reference to `log' /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `pow' /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `log10' /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `ceil' /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `expf' From balay at mcs.anl.gov Wed Apr 1 23:13:30 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 1 Apr 2015 23:13:30 -0500 Subject: [petsc-users] Error configuring with --download-ml and Intel MKL In-Reply-To: References: <427013B3-7C56-438E-AB98-1EC0DF2B08B3@mcs.anl.gov> Message-ID: Perhaps you can try using MKL with intel compilers - and see if the problem persists. Satish On Wed, 1 Apr 2015, Satish Balay wrote: > On Wed, 1 Apr 2015, David Knezevic wrote: > > > On Wed, Apr 1, 2015 at 11:46 PM, Barry Smith wrote: > > > > > > > > Send configure.log from the failed configure > > > > > > > > > It's attached. > > > > Same issue as before - for some reason mpicxx is not finding math symbols in -lm > > Satish > > ----- > Executing: mpicxx -o /tmp/petsc-rljk4x/config.libraries/conftest -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O0 /tmp/petsc-rljk4x/config.libraries/conftest.o -Wl,-rpath,/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib64 -L/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib64 -lsuperlu_dist_3.3 -Wl,-rpath,/opt/intel/system_studio_2015.2.050/mkl/lib/intel64 -L/opt/intel/system_studio_2015.2.050/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm -Wl,-rpath,/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -lgfortran -lm -lquadmath -lm -Wl,-rpath,/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib -L/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib -lparmetis -Wl,-rpath,/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib -L/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib -lmetis -lm -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s -lpthread -ldl > Possible ERROR while running linker: exit code 256 > stderr: > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `logf' > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `atan2' > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `sin' > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `fabs' > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `exp' > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `cos' > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `sqrt' > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_intel_lp64.so: undefined reference to `log' > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `pow' > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `log10' > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `ceil' > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `expf' From balay at mcs.anl.gov Thu Apr 2 00:26:47 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 2 Apr 2015 00:26:47 -0500 Subject: [petsc-users] Error configuring with --download-ml and Intel MKL In-Reply-To: References: <427013B3-7C56-438E-AB98-1EC0DF2B08B3@mcs.anl.gov> Message-ID: Also - you can try not using '--with-clanguage=cxx'. Complex should work with C build aswell. Satish On Wed, 1 Apr 2015, Satish Balay wrote: > Perhaps you can try using MKL with intel compilers - and see if the problem persists. > > Satish > > On Wed, 1 Apr 2015, Satish Balay wrote: > > > On Wed, 1 Apr 2015, David Knezevic wrote: > > > > > On Wed, Apr 1, 2015 at 11:46 PM, Barry Smith wrote: > > > > > > > > > > > Send configure.log from the failed configure > > > > > > > > > > > > > It's attached. > > > > > > > Same issue as before - for some reason mpicxx is not finding math symbols in -lm > > > > Satish > > > > ----- > > Executing: mpicxx -o /tmp/petsc-rljk4x/config.libraries/conftest -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O0 /tmp/petsc-rljk4x/config.libraries/conftest.o -Wl,-rpath,/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib64 -L/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib64 -lsuperlu_dist_3.3 -Wl,-rpath,/opt/intel/system_studio_2015.2.050/mkl/lib/intel64 -L/opt/intel/system_studio_2015.2.050/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm -Wl,-rpath,/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -lgfortran -lm -lquadmath -lm -Wl,-rpath,/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib -L/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib -lparmetis -Wl,-rpath,/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib -L/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib -lmetis -lm -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s -lpthread -ldl > > Possible ERROR while running linker: exit code 256 > > stderr: > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `logf' > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `atan2' > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `sin' > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `fabs' > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `exp' > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `cos' > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `sqrt' > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_intel_lp64.so: undefined reference to `log' > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `pow' > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `log10' > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `ceil' > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `expf' > From balay at mcs.anl.gov Thu Apr 2 00:52:30 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 2 Apr 2015 00:52:30 -0500 Subject: [petsc-users] Error configuring with --download-ml and Intel MKL In-Reply-To: References: <427013B3-7C56-438E-AB98-1EC0DF2B08B3@mcs.anl.gov> Message-ID: And to get g++ to behave similar to gcc on 14.04 - you can add an extra configure option LIBS="-Wl,--no-as-needed -lc" [so --with-clanguage=cxx should work with it] ./configure --with-scalar-type=complex --with-clanguage=cxx --download-metis --download-parmetis --download-superlu_dist --with-blas-lapack-dir=/opt/intel/system_studio_2015.2.050/mkl/lib/intel64 LIBS="-Wl,--no-as-needed -lc" Satish On Thu, 2 Apr 2015, Satish Balay wrote: > Also - you can try not using '--with-clanguage=cxx'. Complex should > work with C build aswell. > > Satish > > On Wed, 1 Apr 2015, Satish Balay wrote: > > > Perhaps you can try using MKL with intel compilers - and see if the problem persists. > > > > Satish > > > > On Wed, 1 Apr 2015, Satish Balay wrote: > > > > > On Wed, 1 Apr 2015, David Knezevic wrote: > > > > > > > On Wed, Apr 1, 2015 at 11:46 PM, Barry Smith wrote: > > > > > > > > > > > > > > Send configure.log from the failed configure > > > > > > > > > > > > > > > > > It's attached. > > > > > > > > > > Same issue as before - for some reason mpicxx is not finding math symbols in -lm > > > > > > Satish > > > > > > ----- > > > Executing: mpicxx -o /tmp/petsc-rljk4x/config.libraries/conftest -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O0 /tmp/petsc-rljk4x/config.libraries/conftest.o -Wl,-rpath,/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib64 -L/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib64 -lsuperlu_dist_3.3 -Wl,-rpath,/opt/intel/system_studio_2015.2.050/mkl/lib/intel64 -L/opt/intel/system_studio_2015.2.050/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm -Wl,-rpath,/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -lgfortran -lm -lquadmath -lm -Wl,-rpath,/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib -L/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib -lparmetis -Wl,-rpath,/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib -L/home/dknez/software/petsc-3.5.2/arch-linux2-cxx-debug/lib -lmetis -lm -lm -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s -lpthread -ldl > > > Possible ERROR while running linker: exit code 256 > > > stderr: > > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `logf' > > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `atan2' > > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `sin' > > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `fabs' > > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `exp' > > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `cos' > > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `sqrt' > > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_intel_lp64.so: undefined reference to `log' > > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `pow' > > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `log10' > > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `ceil' > > > /opt/intel/system_studio_2015.2.050/mkl/lib/intel64/libmkl_core.so: undefined reference to `expf' > > From jroman at dsic.upv.es Thu Apr 2 08:37:14 2015 From: jroman at dsic.upv.es (Jose E. Roman) Date: Thu, 2 Apr 2015 15:37:14 +0200 Subject: [petsc-users] getting EPS to rebuild the BV context In-Reply-To: <1427925803627.f361ecb4@Nodemailer> References: <53546267-34AE-4DB1-9E1F-95D494FB71D7@dsic.upv.es> <1427925803627.f361ecb4@Nodemailer> Message-ID: I was able to fix this without changing the BV API, so the patch will go both to maint and master (after passing nightly tests). https://bitbucket.org/slepc/slepc/commits/5f17011 Thanks for reporting this. Jose El 02/04/2015, a las 00:03, Andrew Spott escribi?: > >Are you using master? > This problem was run into on 3.5.3, but I?m working on trying to get the code to work on master. A cursory glance at the code for master seems to tell me that this problem hasn?t changed. > > -Andrew > > > > On Wed, Apr 1, 2015 at 4:00 PM, Jose E. Roman wrote: > > > El 01/04/2015, a las 22:32, Andrew Spott escribi?: > > > When resizing an EPS (adjusting the nev, ncv and/or mpd values) after a deflation space has been entered, SLEPC complains that you can?t ?resize a BV with constraints?. > > > > Is there a way of forcing this issue by killing the BV and remaking it? (re-adding the constraints). > > > > -Andrew > > > > This case is not handled well. A workaround is to call EPSReset(), but probably there should be a better way to do this. Are you using master? > > Jose > > > From david.knezevic at akselos.com Thu Apr 2 09:49:52 2015 From: david.knezevic at akselos.com (David Knezevic) Date: Thu, 2 Apr 2015 10:49:52 -0400 Subject: [petsc-users] Error configuring with --download-ml and Intel MKL In-Reply-To: References: <427013B3-7C56-438E-AB98-1EC0DF2B08B3@mcs.anl.gov> Message-ID: On Thu, Apr 2, 2015 at 1:52 AM, Satish Balay wrote: > And to get g++ to behave similar to gcc on 14.04 - you can add an > extra configure option LIBS="-Wl,--no-as-needed -lc" [so > --with-clanguage=cxx should work with it] > > ./configure --with-scalar-type=complex --with-clanguage=cxx > --download-metis --download-parmetis --download-superlu_dist > --with-blas-lapack-dir=/opt/intel/system_studio_2015.2.050/mkl/lib/intel64 > LIBS="-Wl,--no-as-needed -lc" > Thanks! Adding the LIBS option fixed this for me. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Apr 2 10:58:26 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 2 Apr 2015 10:58:26 -0500 Subject: [petsc-users] Error configuring with --download-ml and Intel MKL In-Reply-To: References: <427013B3-7C56-438E-AB98-1EC0DF2B08B3@mcs.anl.gov> Message-ID: On Thu, 2 Apr 2015, David Knezevic wrote: > On Thu, Apr 2, 2015 at 1:52 AM, Satish Balay wrote: > > > And to get g++ to behave similar to gcc on 14.04 - you can add an > > extra configure option LIBS="-Wl,--no-as-needed -lc" [so > > --with-clanguage=cxx should work with it] > > > > ./configure --with-scalar-type=complex --with-clanguage=cxx > > --download-metis --download-parmetis --download-superlu_dist > > --with-blas-lapack-dir=/opt/intel/system_studio_2015.2.050/mkl/lib/intel64 > > LIBS="-Wl,--no-as-needed -lc" > > > > > Thanks! Adding the LIBS option fixed this for me. Looks like you can also use: CXXFLAGS=-Wl,--no-as-needed And --download-ml should work with this.. [as CXXFLAGS get passed down to ML's configure] Satish From david.knezevic at akselos.com Thu Apr 2 11:52:47 2015 From: david.knezevic at akselos.com (David Knezevic) Date: Thu, 2 Apr 2015 12:52:47 -0400 Subject: [petsc-users] Error configuring with --download-ml and Intel MKL In-Reply-To: References: <427013B3-7C56-438E-AB98-1EC0DF2B08B3@mcs.anl.gov> Message-ID: On Thu, Apr 2, 2015 at 11:58 AM, Satish Balay wrote: > On Thu, 2 Apr 2015, David Knezevic wrote: > > > On Thu, Apr 2, 2015 at 1:52 AM, Satish Balay wrote: > > > > > And to get g++ to behave similar to gcc on 14.04 - you can add an > > > extra configure option LIBS="-Wl,--no-as-needed -lc" [so > > > --with-clanguage=cxx should work with it] > > > > > > ./configure --with-scalar-type=complex --with-clanguage=cxx > > > --download-metis --download-parmetis --download-superlu_dist > > > > --with-blas-lapack-dir=/opt/intel/system_studio_2015.2.050/mkl/lib/intel64 > > > LIBS="-Wl,--no-as-needed -lc" > > > > > > > > > Thanks! Adding the LIBS option fixed this for me. > > Looks like you can also use: > > CXXFLAGS=-Wl,--no-as-needed > > And --download-ml should work with this.. [as CXXFLAGS get passed down to > ML's configure] > Yep, that works for me for both ML and SuperLU with MKL. Thanks so much for your help! David -------------- next part -------------- An HTML attachment was scrubbed... URL: From ansp6066 at colorado.edu Thu Apr 2 11:54:29 2015 From: ansp6066 at colorado.edu (Andrew Spott) Date: Thu, 02 Apr 2015 09:54:29 -0700 (PDT) Subject: [petsc-users] getting EPS to rebuild the BV context In-Reply-To: References: Message-ID: <1427993669438.9320e105@Nodemailer> Thanks for fixing it so quickly! -Andrew On Thu, Apr 2, 2015 at 7:37 AM, Jose E. Roman wrote: > I was able to fix this without changing the BV API, so the patch will go both to maint and master (after passing nightly tests). > https://bitbucket.org/slepc/slepc/commits/5f17011 > Thanks for reporting this. > Jose > El 02/04/2015, a las 00:03, Andrew Spott escribi?: >> >Are you using master? >> This problem was run into on 3.5.3, but I?m working on trying to get the code to work on master. A cursory glance at the code for master seems to tell me that this problem hasn?t changed. >> >> -Andrew >> >> >> >> On Wed, Apr 1, 2015 at 4:00 PM, Jose E. Roman wrote: >> >> >> El 01/04/2015, a las 22:32, Andrew Spott escribi?: >> >> > When resizing an EPS (adjusting the nev, ncv and/or mpd values) after a deflation space has been entered, SLEPC complains that you can?t ?resize a BV with constraints?. >> > >> > Is there a way of forcing this issue by killing the BV and remaking it? (re-adding the constraints). >> > >> > -Andrew >> > >> >> This case is not handled well. A workaround is to call EPSReset(), but probably there should be a better way to do this. Are you using master? >> >> Jose >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Apr 2 12:03:53 2015 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 2 Apr 2015 12:03:53 -0500 Subject: [petsc-users] DMView and DMLoad In-Reply-To: References: Message-ID: I have fixed this bug. The fix is merged to next, and will make it to master soon. Thanks, Matt On Tue, Mar 10, 2015 at 5:05 PM, Matthew Knepley wrote: > On Tue, Mar 10, 2015 at 5:02 PM, Ataollah Mesgarnejad < > amesga1 at tigers.lsu.edu> wrote: > >> Thanks Matt, >> >> I changed your file to reproduce the error. If you run it on more than >> one processor you will get something similar to: >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> >> [0]PETSC ERROR: Invalid argument >> >> [0]PETSC ERROR: Number of coordinates loaded 9 does not match number of >> vertices 13 >> >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> >> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2186-g052b31f GIT >> Date: 2015-03-04 11:15:59 -0600 >> >> [0]PETSC ERROR: /work/01624/amesga/Code/dmcomplex/ex5 on a >> sandybridge-cxx-dbg named c560-303.stampede.tacc.utexas.edu by amesga >> Tue Mar 10 16:57:01 2015 >> >> [0]PETSC ERROR: Configure options --COPTFLAGS= --CXXOPTFLAGS= >> --FOPTFLAGS= --download-chaco=1 --download-exodusii=1 --download-hdf5=1 >> --download-metis=1 --download-netcdf=1 --download-parmetis=1 --with-blacs=1 >> --with-blas-lapack-dir=/opt/apps/intel/13/composer_xe_2013_sp1.1.106/mkl/lib/intel64 >> --with-clanguage=C++ --with-debugging=yes --with-hdf5=1 --with-metis=1 >> --with-valgrind=1 --download-triangle=1 >> --with-valgrind-dir=/opt/apps/valgrind/3.8.1/ --with-mpi-compilers=1 >> --with-mpi-dir=/opt/apps/intel13/impi/4.1.3.049/intel64 >> --with-scalar-type=real --with-shared-libraries=1 >> --with-vendor-compilers=intel --with-x11=1 -with-pic >> PETSC_ARCH=sandybridge-cxx-dbg >> >> [0]PETSC ERROR: #1 DMPlexLoad_HDF5() line 741 in >> /work/01624/amesga/Software/petsc/src/dm/impls/plex/plexhdf5.c >> >> [0]PETSC ERROR: #2 DMLoad_Plex() line 605 in >> /work/01624/amesga/Software/petsc/src/dm/impls/plex/plex.c >> >> [0]PETSC ERROR: #3 DMLoad() line 2939 in >> /work/01624/amesga/Software/petsc/src/dm/interface/dm.c >> >> [0]PETSC ERROR: #4 main() line 31 in >> /work/01624/amesga/Code/dmcomplex/ex5.c >> >> [0]PETSC ERROR: No PETSc Option Table entries >> >> [0]PETSC ERROR: ----------------End of Error Message ------- >> >> >> It seems like HDF5 version of DMView and DMLoad work fine for a serial >> DMPlex but DMLoad breaks in parallel. >> > > Cool, will get it fixed. I have only been loading Vecs since DMLoad() > cannot recreate hierarchies, and thus is not > compatible with the FAS tests I am mostly running. > > Thanks, > > Matt > > >> Many thanks, >> >> Ata >> >> On Tue, Mar 10, 2015 at 3:15 PM, Matthew Knepley >> wrote: >> >>> On Mon, Mar 9, 2015 at 11:28 AM, Ataollah Mesgarnejad < >>> amesga1 at tigers.lsu.edu> wrote: >>> >>>> Dear all, >>>> >>>> I'm trying to save and load distributed DMPlex in HDF5 format with >>>> DMView and DMLoad. The problem is even though there are no errors while >>>> loading the DM when I look at the DM loaded from file is not equal to the >>>> DM that was saved. Also, when loading the DM if DMSetType is called before >>>> the DMLoad, DMLoad produces errors (like line 316 >>>> >>>> in src/snes/examples/tutorials/ex12.c ). >>>> >>>> I'm including a small code to replicate the problem here plus an input >>>> mesh file. >>>> >>> >>> Here is a simpler test, which I am going to make DMPlex ex5. The file >>> you sent does not make sense >>> to me since it has cells with only two vertices. Run it with >>> >>> -dm_view ::ascii_info_detail >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Best, >>>> Ata >>>> >>>> /** >>>> >>>> * @file >>>> >>>> * >>>> >>>> * Created by Ataollah Mesgarnejad on 3/6/15. >>>> >>>> * >>>> >>>> * This is a test to read and write a distributed DM. >>>> >>>> * >>>> >>>> */ >>>> >>>> >>>> >>>> static char help[] = "An example of the usage of PetscSF to scatter >>>> data back and forth between a \ >>>> >>>> a serial DM and its parallel counterpart.\n"; >>>> >>>> >>>> #include >>>> >>>> #include >>>> >>>> #include >>>> >>>> #include >>>> >>>> #include /*I "petscdm.h" I*/ >>>> >>>> #include >>>> >>>> #include >>>> >>>> >>>> #undef __FUNCT__ >>>> >>>> #define __FUNCT__ "main" >>>> >>>> int main(int argc,char **argv) >>>> >>>> { >>>> >>>> PetscErrorCode ierr; >>>> >>>> DM dm, distDM,distDMold; >>>> >>>> char ifilename[PETSC_MAX_PATH_LEN]; >>>> >>>> PetscBool flg; >>>> >>>> int CPU_word_size = 0,IO_word_size = 0,exoid; >>>> >>>> float version; >>>> >>>> PetscInt numproc,rank; >>>> >>>> PetscViewer stdoutViewer,DMPlexHDF5View; >>>> >>>> PetscReal *VArray; >>>> >>>> PetscInt dim; >>>> >>>> PetscInt numFields = 2; >>>> >>>> PetscInt numComp[2] = {1,1}; >>>> >>>> PetscInt numDof[6] = {1, 0, 0, >>>> >>>> 0, 0, 1}; /*{Vertex, Edge, Cell} */ >>>> >>>> PetscInt bcFields[1] = {0}, numBC=0; >>>> >>>> //PetscInt *remoteOffsets; >>>> >>>> char** namelist; >>>> >>>> PetscInt off; >>>> >>>> IS bcPoints[1] = {NULL}; >>>> >>>> IS *ISlist; >>>> >>>> PetscSection seqSection, distSection; >>>> >>>> Vec distV, V; >>>> >>>> PetscSF pointSF;//,pointSFold; >>>> >>>> PetscInt pStart, pEnd, dof; >>>> >>>> PetscBool load_files = PETSC_FALSE, save_files = >>>> PETSC_FALSE; >>>> >>>> MPI_Comm comm; >>>> >>>> >>>> >>>> ierr = PetscInitialize(&argc, &argv, (char*)0, help);CHKERRQ(ierr); >>>> >>>> ierr = PetscViewerASCIIGetStdout(PETSC_COMM_SELF,&stdoutViewer); >>>> CHKERRQ(ierr); >>>> >>>> comm = PETSC_COMM_WORLD; >>>> >>>> // PetscErrorCode DMCreateSubDM(DM dm, PetscInt numFields, >>>> PetscInt fields[], IS *is, DM *subdm) >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> ierr = PetscOptionsBegin(PETSC_COMM_WORLD,"","PetscSF Test Options" >>>> ,"none");CHKERRQ(ierr); >>>> >>>> ierr = PetscOptionsBool("-save_files","save the distributed vector >>>> in HDF5 format","",PETSC_FALSE,&save_files,NULL);CHKERRQ(ierr); >>>> >>>> ierr = PetscOptionsBool("-load_files","Load the distributed vector >>>> in HDF5 format","",PETSC_FALSE,&load_files,NULL);CHKERRQ(ierr); >>>> >>>> ierr = PetscOptionsEnd(); >>>> >>>> ierr = PetscOptionsGetString(PETSC_NULL,"-i",ifilename,sizeof >>>> ifilename,&flg);CHKERRQ(ierr); >>>> >>>> >>>> >>>> ierr = MPI_Comm_size(PETSC_COMM_WORLD,&numproc); >>>> >>>> ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank); >>>> >>>> if (!rank) >>>> >>>> { >>>> >>>> exoid = ex_open(ifilename,EX_READ >>>> ,&CPU_word_size,&IO_word_size,&version); >>>> >>>> if (exoid <= 0) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"ex_open(\"%s\",...) >>>> did not return a valid file ID",ifilename); >>>> >>>> } >>>> >>>> else >>>> >>>> { >>>> >>>> exoid = -1; /* Not used */ >>>> >>>> } >>>> >>>> ierr = DMPlexCreateExodus(PETSC_COMM_WORLD,exoid,PETSC_FALSE,&dm); >>>> CHKERRQ(ierr); >>>> >>>> ierr = PetscObjectSetName((PetscObject) dm,"sequential-DM"); >>>> >>>> ierr = DMGetDimension(dm, &dim); >>>> CHKERRQ(ierr); >>>> >>>> >>>> >>>> if (numproc < 2 ) distDM = dm; >>>> >>>> else ierr = DMPlexDistribute(dm, 0, &pointSF, &distDM); >>>> CHKERRQ(ierr); >>>> >>>> ierr = PetscObjectSetName((PetscObject) distDM,"Distributed-DM"); >>>> >>>> >>>> ierr = DMPlexCreateSection(dm, dim, numFields, numComp, >>>> numDof, >>>> >>>> numBC, bcFields, bcPoints,PETSC_NULL, >>>> &seqSection); CHKERRQ(ierr); >>>> >>>> ierr = DMPlexCreateSection(distDM, dim, numFields, numComp, >>>> numDof, >>>> >>>> numBC, bcFields, bcPoints,PETSC_NULL, >>>> &distSection); CHKERRQ(ierr); >>>> >>>> >>>> >>>> ierr = DMSetDefaultSection(dm, seqSection); >>>> CHKERRQ(ierr); >>>> >>>> ierr = DMSetDefaultSection(distDM, distSection); >>>> CHKERRQ(ierr); >>>> >>>> >>>> >>>> ierr = DMCreateGlobalVector(dm, &V); >>>> CHKERRQ(ierr); >>>> >>>> ierr = VecCreate(comm, &distV); >>>> CHKERRQ(ierr); >>>> >>>> ierr = PetscObjectSetName((PetscObject) distV,"Distributed-V"); >>>> >>>> >>>> ierr = VecGetArray(V, &VArray); >>>> CHKERRQ(ierr); >>>> >>>> >>>> >>>> // fill vertex data >>>> >>>> ierr = DMPlexGetDepthStratum(dm, 0, &pStart, &pEnd); >>>> CHKERRQ(ierr); >>>> >>>> for (int p = pStart; p < pEnd; p++) >>>> >>>> { >>>> >>>> ierr = PetscSectionGetDof(seqSection, p, &dof); >>>> CHKERRQ(ierr); >>>> >>>> ierr = PetscSectionGetOffset(seqSection, p, &off); >>>> CHKERRQ(ierr); >>>> >>>> for (int d = 0; d < dof; d++) >>>> >>>> { >>>> >>>> VArray[off + d] = p; >>>> >>>> } >>>> >>>> } >>>> >>>> >>>> >>>> // fill cell data >>>> >>>> ierr = DMPlexGetDepthStratum(dm, 1, &pStart, &pEnd); >>>> CHKERRQ(ierr); >>>> >>>> for (int p = pStart; p < pEnd; p++) >>>> >>>> { >>>> >>>> ierr = PetscSectionGetDof(seqSection, p, &dof); >>>> CHKERRQ(ierr); >>>> >>>> ierr = PetscSectionGetOffset(seqSection, p, &off); >>>> CHKERRQ(ierr); >>>> >>>> for (int d = 0; d < dof; d++) >>>> >>>> { >>>> >>>> VArray[off + d] = -p-1; >>>> >>>> } >>>> >>>> } >>>> >>>> >>>> >>>> ierr = VecRestoreArray(V, &VArray); CHKERRQ(ierr); >>>> >>>> ierr = DMCreateFieldIS(dm, &numFields, &namelist, &ISlist); >>>> CHKERRQ(ierr); >>>> >>>> ierr = DMPlexDistributeField(dm, pointSF, seqSection, V, >>>> distSection, distV); CHKERRQ >>>> (ierr); >>>> >>>> #if defined(PETSC_HAVE_HDF5) >>>> >>>> if (save_files) >>>> >>>> { >>>> >>>> // distribute fields >>>> >>>> >>>> >>>> ierr = PetscPrintf(PETSC_COMM_WORLD, "==== Distrubuted DM\n"); >>>> CHKERRQ(ierr); >>>> >>>> ierr = DMView(distDM, PETSC_VIEWER_STDOUT_WORLD); >>>> CHKERRQ(ierr); >>>> >>>> PetscSynchronizedFlush(PETSC_COMM_WORLD, PETSC_STDOUT); >>>> >>>> // Save distributed data >>>> >>>> >>>> >>>> ierr = PetscViewerHDF5Open(PetscObjectComm((PetscObject) >>>> distDM),"distDM.h5", FILE_MODE_WRITE, &DMPlexHDF5View);CHKERRQ(ierr); >>>> >>>> ierr = PetscPrintf(PETSC_COMM_WORLD,"Writing dist DM ...\n"); >>>> >>>> PetscSynchronizedFlush(PETSC_COMM_WORLD, PETSC_STDOUT); >>>> >>>> ierr = DMView(distDM, DMPlexHDF5View); >>>> CHKERRQ(ierr); >>>> >>>> >>>> >>>> PetscViewerDestroy(&DMPlexHDF5View); >>>> >>>> >>>> >>>> } >>>> >>>> else if (load_files) >>>> >>>> { >>>> >>>> ierr = PetscPrintf(PETSC_COMM_WORLD,"Loading dist vectors >>>> ...\n"); CHKERRQ(ierr); >>>> >>>> ierr = PetscViewerHDF5Open(PETSC_COMM_SELF,"distDM.h5", >>>> FILE_MODE_READ, &DMPlexHDF5View); CHKERRQ(ierr); >>>> >>>> ierr = DMCreate(comm,&distDMold); >>>> CHKERRQ(ierr); >>>> >>>> >>>> >>>> ierr = DMLoad(distDMold,DMPlexHDF5View); >>>> CHKERRQ(ierr); >>>> >>>> ierr = DMSetType(distDMold, DMPLEX); >>>> CHKERRQ(ierr); >>>> >>>> /*ierr = DMPlexCreateSection(distDMold, dim, numFields, >>>> numComp, numDof, >>>> >>>> numBC, bcFields, >>>> bcPoints,PETSC_NULL, &distSectionold); >>>> CHKERRQ(ierr); >>>> >>>> ierr = DMSetDefaultSection(distDMold, >>>> distSectionold);CHKERRQ(ierr); >>>> >>>> ierr = DMGetPointSF(distDMold,&pointSFold); >>>> CHKERRQ(ierr);*/ >>>> >>>> PetscViewerDestroy(&DMPlexHDF5View); >>>> >>>> ierr = PetscPrintf(PETSC_COMM_WORLD, "==== Read DM\n"); >>>> CHKERRQ(ierr); >>>> >>>> ierr = DMView(distDMold, PETSC_VIEWER_STDOUT_WORLD); >>>> CHKERRQ(ierr); >>>> >>>> PetscSynchronizedFlush(PETSC_COMM_WORLD, PETSC_STDOUT); >>>> CHKERRQ(ierr); >>>> >>>> DMPlexEqual(distDM, distDMold, &flg); >>>> >>>> if (flg) >>>> >>>> { >>>> >>>> PetscPrintf(PETSC_COMM_WORLD,"\n DMs equal\n"); >>>> >>>> } >>>> >>>> else >>>> >>>> { >>>> >>>> PetscPrintf(PETSC_COMM_WORLD,"\n DMs are not equal\n\n"); >>>> >>>> } >>>> >>>> } >>>> >>>> #endif >>>> >>>> >>>> >>>> ierr = VecDestroy(&V); CHKERRQ(ierr); >>>> >>>> ierr = VecDestroy(&distV); CHKERRQ(ierr); >>>> >>>> //ierr = VecDestroy(&seqV); CHKERRQ(ierr); >>>> >>>> >>>> >>>> ierr = DMDestroy(&dm);CHKERRQ(ierr); >>>> >>>> ierr = DMDestroy(&distDM);CHKERRQ(ierr); >>>> >>>> >>>> >>>> ierr = PetscFinalize(); >>>> >>>> return 0; >>>> >>>> } >>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jchang27 at uh.edu Thu Apr 2 18:39:51 2015 From: jchang27 at uh.edu (Justin Chang) Date: Thu, 2 Apr 2015 18:39:51 -0500 Subject: [petsc-users] Integrating TAO into SNES and TS Message-ID: Hi everyone, I have a two part question regarding the integration of the following optimization problem min 1/2 u^T*K*u + u^T*f S.T. u >= 0 into SNES and TS 1) For SNES, assuming I am working with a linear FE equation, I have the following algorithm/steps for solving my problem a) Set an initial guess x b) Obtain residual r and jacobian A through functions SNESComputeFunction() and SNESComputeJacobian() respectively c) Form vector b = r - A*x d) Set Hessian equal to A, gradient to A*x, objective function value to 1/2*x^T*A*x + x^T*b, and variable (lower) bounds to a zero vector e) Call TaoSolve This works well at the moment, but my question is there a more "efficient" way of doing this? Because with my current setup, I am making a rather bold assumption that my problem would converge in one SNES iteration without the bounded constraints and does not have any unexpected nonlinearities. 2) How would I go about doing the above for time-stepping problems? At each time step, I want to solve a convex optimization subject to the lower bounds constraint. I plan on using backward euler and my resulting jacobian should still be compatible with the above optimization problem. Thanks, -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Apr 2 18:53:04 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 2 Apr 2015 18:53:04 -0500 Subject: [petsc-users] Integrating TAO into SNES and TS In-Reply-To: References: Message-ID: <16064BCD-8B50-4E95-AF7F-386F6780E645@mcs.anl.gov> An alternative approach is for you to solve it as a (non)linear variational inequality. See src/snes/examples/tutorials/ex9.c How you should proceed depends on your long term goal. What problem do you really want to solve? Is it really a linear time dependent problem with 0 bounds on U? Can the problem always be represented as an optimization problem easily? What are and what will be the properties of K? For example if K is positive definite then likely the bounds will remain try without explicitly providing the constraints. Barry > On Apr 2, 2015, at 6:39 PM, Justin Chang wrote: > > Hi everyone, > > I have a two part question regarding the integration of the following optimization problem > > min 1/2 u^T*K*u + u^T*f > S.T. u >= 0 > > into SNES and TS > > 1) For SNES, assuming I am working with a linear FE equation, I have the following algorithm/steps for solving my problem > > a) Set an initial guess x > b) Obtain residual r and jacobian A through functions SNESComputeFunction() and SNESComputeJacobian() respectively > c) Form vector b = r - A*x > d) Set Hessian equal to A, gradient to A*x, objective function value to 1/2*x^T*A*x + x^T*b, and variable (lower) bounds to a zero vector > e) Call TaoSolve > > This works well at the moment, but my question is there a more "efficient" way of doing this? Because with my current setup, I am making a rather bold assumption that my problem would converge in one SNES iteration without the bounded constraints and does not have any unexpected nonlinearities. > > 2) How would I go about doing the above for time-stepping problems? At each time step, I want to solve a convex optimization subject to the lower bounds constraint. I plan on using backward euler and my resulting jacobian should still be compatible with the above optimization problem. > > Thanks, > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 From jchang27 at uh.edu Fri Apr 3 12:31:36 2015 From: jchang27 at uh.edu (Justin Chang) Date: Fri, 3 Apr 2015 12:31:36 -0500 Subject: [petsc-users] Integrating TAO into SNES and TS In-Reply-To: <16064BCD-8B50-4E95-AF7F-386F6780E645@mcs.anl.gov> References: <16064BCD-8B50-4E95-AF7F-386F6780E645@mcs.anl.gov> Message-ID: I am solving the following anisotropic transient diffusion equation subject to 0 bounds: du/dt = div[D*grad[u]] + f Where the dispersion tensor D(x) is symmetric and positive definite. This formulation violates the discrete maximum principles so one of the ways to ensure nonnegative concentrations is to employ convex optimization. I am following the procedures in Nakshatrala and Valocchi (2009) JCP and Nagarajan and Nakshatrala (2011) IJNMF. The Variational Inequality method works gives what I want for my transient case, but what if I want to implement the Tao methodology in TS? That is, what TS functions do I need to set up steps a) through e) for each time step (also the Jacobian remains the same for all time steps so I would only call this once). Normally I would just call TSSolve() and let the libraries and functions do everything, but I would like to incorporate TaoSolve into every time step. Thanks, -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 On Thu, Apr 2, 2015 at 6:53 PM, Barry Smith wrote: > > An alternative approach is for you to solve it as a (non)linear > variational inequality. See src/snes/examples/tutorials/ex9.c > > How you should proceed depends on your long term goal. What problem do > you really want to solve? Is it really a linear time dependent problem with > 0 bounds on U? Can the problem always be represented as an optimization > problem easily? What are and what will be the properties of K? For example > if K is positive definite then likely the bounds will remain try without > explicitly providing the constraints. > > Barry > > > On Apr 2, 2015, at 6:39 PM, Justin Chang wrote: > > > > Hi everyone, > > > > I have a two part question regarding the integration of the following > optimization problem > > > > min 1/2 u^T*K*u + u^T*f > > S.T. u >= 0 > > > > into SNES and TS > > > > 1) For SNES, assuming I am working with a linear FE equation, I have the > following algorithm/steps for solving my problem > > > > a) Set an initial guess x > > b) Obtain residual r and jacobian A through functions > SNESComputeFunction() and SNESComputeJacobian() respectively > > c) Form vector b = r - A*x > > d) Set Hessian equal to A, gradient to A*x, objective function value to > 1/2*x^T*A*x + x^T*b, and variable (lower) bounds to a zero vector > > e) Call TaoSolve > > > > This works well at the moment, but my question is there a more > "efficient" way of doing this? Because with my current setup, I am making a > rather bold assumption that my problem would converge in one SNES iteration > without the bounded constraints and does not have any unexpected > nonlinearities. > > > > 2) How would I go about doing the above for time-stepping problems? At > each time step, I want to solve a convex optimization subject to the lower > bounds constraint. I plan on using backward euler and my resulting jacobian > should still be compatible with the above optimization problem. > > > > Thanks, > > > > -- > > Justin Chang > > PhD Candidate, Civil Engineering - Computational Sciences > > University of Houston, Department of Civil and Environmental Engineering > > Houston, TX 77004 > > (512) 963-3262 > > -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Apr 3 18:17:29 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 3 Apr 2015 18:17:29 -0500 Subject: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult In-Reply-To: <441091432.3174678.1427919054040.JavaMail.yahoo@mail.yahoo.com> References: <4E540218-7EFC-438C-B48B-F7D746929E38@mcs.anl.gov> <441091432.3174678.1427919054040.JavaMail.yahoo@mail.yahoo.com> Message-ID: <86DE4DD8-55D6-49B7-A87D-75D2E3AA0D4E@mcs.anl.gov> Steena, Sorry for all the delays. Our code was just totally wrong for a user provided decomposition for two reasons 1) the logic in MatLoad_MPIXAIJ() was too convoluted to work in all cases and 2) we didn't have a single test case. I have attached a patch that fixes it for petsc-3.5.3 -------------- next part -------------- A non-text attachment was scrubbed... Name: fix-matload.patch Type: application/octet-stream Size: 15009 bytes Desc: not available URL: -------------- next part -------------- . The fix is also in the branch barry/fix-matload-uneven-rows and I'll put it into next then after testing it will got into maint and master and the next PETSc patch release. Please let us know if the patch doesn't work for you Thanks for your patience, Barry > On Apr 1, 2015, at 3:10 PM, Steena M wrote: > > Thanks Barry. Attached is the driver program, the binary mat file, and the corresponding mtx file. > > Runtime command used: > > sierra324 at monteiro:time srun -n 2 -ppdebug ./petsc-mpibaij-unequalrows -fin trefethen.dat -matload_block_size 1 > > > > > On Wednesday, April 1, 2015 12:28 PM, Barry Smith wrote: > > > > Send a data file you generated and your reader program and we'll debug it. > > Barry > > > On Apr 1, 2015, at 2:18 PM, Steena M wrote: > > > > Thanks Barry. I removed the Preallocation calls. It is still complaining about the malloc and incorrect data in the matrix file. I generate binary matrix files using PETSc's pythonscript to loop through a set of UFL sparse matrices. For this use case: > > > > mtx_mat = scipy.io.mmread('trefethen.mtx') > > PetscBinaryIO.PetscBinaryIO().writeMatSciPy(open('trefnew.dat','w'), mtx_mat) > > > > > > > > > > > > > > On Tuesday, March 31, 2015 9:15 PM, Barry Smith wrote: > > > > > > > > You should not need to call any preallocation routines when using MatLoad() > > > > > > How did you generate the file? Are you sure it has the correct information for the matrix? > > > > Barry > > > > > > > > > On Mar 31, 2015, at 11:05 PM, Steena M wrote: > > > > > > Thanks Matt. I'm still getting the malloc error > > > > > > [0]PETSC ERROR: Argument out of range! > > > [0]PETSC ERROR: New nonzero at (2,18) caused a malloc! > > > > > > and > > > > > > a new incorrect matrix file error: > > > > > > [0]PETSC ERROR: Unexpected data in file! > > > [0]PETSC ERROR: not matrix object! > > > > > > Maybe the order of calls is mixed up. This is the code snippet: > > > > > > if (rank ==0) > > > { > > > PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); > > > > > > CHKERRQ(MatSetSizes(A, 15, PETSC_DETERMINE, 20, 20)); > > > CHKERRQ(MatSetType(A, MATMPIBAIJ)); > > > CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); > > > CHKERRQ( MatLoad(A,fd)); > > > CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); > > > } > > > > > > else > > > { > > > PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); > > > > > > CHKERRQ( MatSetSizes(A, 5, PETSC_DETERMINE, 20, 20) ); > > > CHKERRQ(MatSetType(A, MATMPIBAIJ)); > > > CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); > > > CHKERRQ(MatLoad(A,fd)); > > > CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); > > > } > > > > > > Is there something I'm missing? > > > > > > Thanks, > > > Steena > > > > > > > > > > > > > > > > > > On Tuesday, March 31, 2015 6:10 PM, Matthew Knepley wrote: > > > > > > > > > On Tue, Mar 31, 2015 at 6:51 PM, Steena M wrote: > > > Thanks Barry. I'm still getting the malloc error with NULL. Is there a way to distribute the matrix without explicit preallocation? Different matrices will be loaded during runtime and assigning preallocation parameters would mean an additional preprocessing step. > > > > > > 1) MatSetOption(MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) > > > > > > 2) Note that this is never ever ever more efficient than making another pass and preallocating > > > > > > Thanks, > > > > > > Matt > > > -------------------------------------------- > > > On Sun, 3/29/15, Barry Smith wrote: > > > > > > Subject: Re: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult > > > To: "Steena M" > > > Cc: "Matthew Knepley" , petsc-users at mcs.anl.gov > > > Date: Sunday, March 29, 2015, 9:26 PM > > > > > > > > > > On > > > Mar 29, 2015, at 11:05 PM, Steena M > > > wrote: > > > > > > > > Thanks > > > Matt. I used PETSC_DETERMINE but I'm now getting an > > > allocation-based error: > > > > > > > > [0]PETSC ERROR: --------------------- > > > Error Message ------------------------------------ > > > > [0]PETSC ERROR: Argument out of range! > > > > [0]PETSC ERROR: New nonzero at (2,18) > > > caused a malloc! > > > > [0]PETSC ERROR: > > > ------------------------------------------------------------------------ > > > > > > > > I tried > > > preallocating on each rank for the diagonal and off diagonal > > > section of the matrix as the next step My current > > > approximations for preallocation > > > > > > > > CHKERRQ( > > > MatMPIBAIJSetPreallocation(A,1,5,PETSC_DEFAULT,5,PETSC_DEFAULT)); > > > > > > > > > These > > > arguments where you pass PETSC_DEFAULT are expecting a > > > pointer not an integer. You can pass NULL in those > > > locations. Though it is better to provide the correct > > > preallocation rather than some defaults. > > > > > > Barry > > > > > > > > > > > are throwing segmentation errors. > > > > > > > > [0]PETSC ERROR: > > > Caught signal number 11 SEGV: Segmentation Violation, > > > probably memory access out of range > > > > > > > > Any insights into what I'm doing > > > wrong? > > > > > > > > Thanks, > > > > Steena > > > > > > > > > > > > > > > > On Sun, 3/29/15, Matthew Knepley > > > wrote: > > > > > > > > Subject: > > > Re: [petsc-users] Unequal sparse matrix row distribution for > > > MPI MatMult > > > > To: "Steena M" > > > > > > > Cc: "Barry Smith" , > > > petsc-users at mcs.anl.gov > > > > Date: Sunday, March 29, 2015, 10:02 PM > > > > > > > > On Sun, Mar 29, 2015 > > > at > > > > 9:56 PM, Steena M > > > > wrote: > > > > Hi > > > > Barry, > > > > > > > > > > > > > > > > I am trying to partition a 20 row and 20 > > > col sparse matrix > > > > between two procs > > > such that proc 0 has 15 rows and 20 cols > > > > and proc 1 has 5 rows and 20 cols. The > > > code snippet: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A)); > > > // > > > > at runtime: -matload_block_size 1 > > > > > > > > > > > > > > > > > > > > > > > > > > > if (rank ==0) > > > > > > > > { > > > > > > > > > > > CHKERRQ( MatSetSizes(A, 15, 20, 20, > > > > 20) ); //rank 0 gets 75% of the rows > > > > > > > > > > > CHKERRQ( MatSetType(A, MATMPIBAIJ) > > > > ); > > > > > > > > CHKERRQ( > > > MatLoad(A,fd) ); > > > > > > > > } > > > > > > > > > > > > > > > > else > > > > > > > > > > > { > > > > > > > > > > > CHKERRQ( MatSetSizes(A, 5, > > > 20, 20, > > > > 20) ); //rank 1 gets 25% of the > > > rows > > > > > > > > > > > CHKERRQ( MatSetType(A, MATMPIBAIJ) > > > > ); > > > > > > > > CHKERRQ( > > > MatLoad(A,fd) ); > > > > > > > > } > > > > > > > > > > > > > > > > > > > This throws the following error (probably > > > from psplit.c): > > > > > > > > > > > [1]PETSC ERROR: --------------------- Error Message > > > > ------------------------------------ > > > > > > > > [1]PETSC ERROR: > > > Nonconforming object sizes! > > > > > > > > [1]PETSC ERROR: Sum of local lengths 40 > > > does not equal > > > > global length 20, my > > > local length 20 > > > > > > > > likely a call to > > > VecSetSizes() or MatSetSizes() is > > > > > > > wrong. > > > > > > > > See > > > http://www.mcs.anl.gov/petsc/documentation/faq.html#split! > > > > > > > > > > > > > > > > This error printout > > > doesn't quite make sense to me. > > > > > > > I'm trying to specify a total matrix size of 20x20... > > > I > > > > haven't yet figured out where the > > > '40' comes > > > > from in the error > > > message. > > > > > > > > > > > > > > > > Any thoughts on what > > > might be going wrong? > > > > > > > > Its the column specification. Just > > > > use PETSC_DETERMINE for the local columns > > > since all our > > > > sparse matrixformats are > > > row divisions > > > > anyway. > > > > > > > > Thanks, > > > > > > > > Matt > > > > Thanks in advance, > > > > > > > > > > > Steena > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------------------------------------- > > > > > > > > On Sun, 3/22/15, > > > Barry Smith > > > > wrote: > > > > > > > > > > > > > > > > Subject: Re: [petsc-users] Unequal > > > sparse matrix row > > > > distribution for MPI > > > MatMult > > > > > > > > To: > > > "Steena M" > > > > > > > > Cc: petsc-users at mcs.anl.gov > > > > > > > > Date: Sunday, > > > March 22, 2015, 3:58 PM > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Steena, > > > > > > > > > > > > > > > > I am > > > > > > > > > > > a little unsure of your question. > > > > > > > > > > > > > > > > > > > 1) > > > you can create a MPIBAIJ > > > > > > > > matrix with any distribution of block > > > rows per process > > > > you > > > > > > > > want, just set the > > > local row size for each process to > > > > > > > be > > > > > > > > what you > > > like. Use MatCreateVecs() to get > > > > > > > correspondingly > > > > > > > > > > > laid out vectors. > > > > > > > > > > > > > > > > > > > > > > or 2) if you have a MPIBAIJ > > > > > > > > matrix with > > > "equal" row layout and you want a > > > > new > > > > > > > > one with uneven row layout you can > > > simply use > > > > > > > > > > > MatGetSubMatrix() to create that new matrix. > > > > > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > Unless you have > > > another reason to have the > > > > > > > > matrix with an equal number row layout I > > > would just > > > > generate > > > > > > > > the matrix with > > > the layout you want. > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 22, 2015, at 5:50 PM, Steena > > > M > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > I need to > > > distribute > > > > > > > > a > > > sparse matrix such that each proc owns an unequal > > > > number > > > > > > > > of blocked rows before I proceed with > > > MPI MatMult. My > > > > > > > > > > > initial thoughts on doing this: > > > > > > > > > > > > > > > > > > 1) Use MatGetSubMatrices() on the > > > test > > > > > > > > MATMPIBAIJ > > > matrix to produce a new matrix where each > > > > proc > > > > > > > > has an unequal number of rows. > > > > > > > > > > > > > > > > > > 2) Provide > > > scatter context for vector X > > > > > > > > (for MatMult )using IS iscol from > > > MatGetSubMatrices() > > > > while > > > > > > > > creating the > > > vector X. > > > > > > > > > > > > > > > > > > > > > 3) > > > Call MatMult() > > > > > > > > > > > > > > > > > > > > > Will > > > MatMult_MPIBAIJ continue to scatter > > > > > > > > this matrix and vector such that each > > > proc will own an > > > > equal > > > > > > > > number of matrix > > > rows and corresponding diagonal vector > > > > > > > > > > > elements? Should I write my own > > > MPIMatMult function to > > > > > > > > retain my redistribution of the matrix > > > and vector? > > > > > > > > > > > > > > > > > > > > > > > > Thanks in > > > > > > > > > > > advance, > > > > > > > > > > > > Steena > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > What most > > > experimenters > > > > take for granted before > > > they begin their experiments is > > > > > > > infinitely more interesting than any results to which > > > their > > > > experiments lead. > > > > -- Norbert > > > > Wiener > > > > > > > > > > > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > -- Norbert Wiener > > > > > > > > > > > > > From bsmith at mcs.anl.gov Fri Apr 3 18:35:23 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 3 Apr 2015 18:35:23 -0500 Subject: [petsc-users] Integrating TAO into SNES and TS In-Reply-To: References: <16064BCD-8B50-4E95-AF7F-386F6780E645@mcs.anl.gov> Message-ID: <0759EFDD-F7BB-4826-B6E0-241159EF0D21@mcs.anl.gov> Justin, We haven't done anything with TS to handle variational inequalities. So you can either write your own backward Euler (outside of TS) that solves each time-step problem either as 1) an optimization problem using Tao or 2) as a variational inequality using SNES. More adventurously you could look at the TSTHETA code in TS (which is a general form that includes Euler, Backward Euler and Crank-Nicolson and see if you can add the constraints to the SNES problem that is solved; in theory this is straightforward but it would require understanding the current code (which Jed, of course, overwrote :-). I think you should do this. Barry > On Apr 3, 2015, at 12:31 PM, Justin Chang wrote: > > I am solving the following anisotropic transient diffusion equation subject to 0 bounds: > > du/dt = div[D*grad[u]] + f > > Where the dispersion tensor D(x) is symmetric and positive definite. This formulation violates the discrete maximum principles so one of the ways to ensure nonnegative concentrations is to employ convex optimization. I am following the procedures in Nakshatrala and Valocchi (2009) JCP and Nagarajan and Nakshatrala (2011) IJNMF. > > The Variational Inequality method works gives what I want for my transient case, but what if I want to implement the Tao methodology in TS? That is, what TS functions do I need to set up steps a) through e) for each time step (also the Jacobian remains the same for all time steps so I would only call this once). Normally I would just call TSSolve() and let the libraries and functions do everything, but I would like to incorporate TaoSolve into every time step. > > Thanks, > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > > On Thu, Apr 2, 2015 at 6:53 PM, Barry Smith wrote: > > An alternative approach is for you to solve it as a (non)linear variational inequality. See src/snes/examples/tutorials/ex9.c > > How you should proceed depends on your long term goal. What problem do you really want to solve? Is it really a linear time dependent problem with 0 bounds on U? Can the problem always be represented as an optimization problem easily? What are and what will be the properties of K? For example if K is positive definite then likely the bounds will remain try without explicitly providing the constraints. > > Barry > > > On Apr 2, 2015, at 6:39 PM, Justin Chang wrote: > > > > Hi everyone, > > > > I have a two part question regarding the integration of the following optimization problem > > > > min 1/2 u^T*K*u + u^T*f > > S.T. u >= 0 > > > > into SNES and TS > > > > 1) For SNES, assuming I am working with a linear FE equation, I have the following algorithm/steps for solving my problem > > > > a) Set an initial guess x > > b) Obtain residual r and jacobian A through functions SNESComputeFunction() and SNESComputeJacobian() respectively > > c) Form vector b = r - A*x > > d) Set Hessian equal to A, gradient to A*x, objective function value to 1/2*x^T*A*x + x^T*b, and variable (lower) bounds to a zero vector > > e) Call TaoSolve > > > > This works well at the moment, but my question is there a more "efficient" way of doing this? Because with my current setup, I am making a rather bold assumption that my problem would converge in one SNES iteration without the bounded constraints and does not have any unexpected nonlinearities. > > > > 2) How would I go about doing the above for time-stepping problems? At each time step, I want to solve a convex optimization subject to the lower bounds constraint. I plan on using backward euler and my resulting jacobian should still be compatible with the above optimization problem. > > > > Thanks, > > > > -- > > Justin Chang > > PhD Candidate, Civil Engineering - Computational Sciences > > University of Houston, Department of Civil and Environmental Engineering > > Houston, TX 77004 > > (512) 963-3262 > > > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 From jchang27 at uh.edu Fri Apr 3 19:35:08 2015 From: jchang27 at uh.edu (Justin Chang) Date: Fri, 3 Apr 2015 19:35:08 -0500 Subject: [petsc-users] Integrating TAO into SNES and TS In-Reply-To: <0759EFDD-F7BB-4826-B6E0-241159EF0D21@mcs.anl.gov> References: <16064BCD-8B50-4E95-AF7F-386F6780E645@mcs.anl.gov> <0759EFDD-F7BB-4826-B6E0-241159EF0D21@mcs.anl.gov> Message-ID: I guess I will have to write my own code then :) I am not all that familiar with Variational Inequalities at the moment, but if my Jacobian is symmetric and positive definite and I only have lower and upper bounds, doesn't the problem simply reduce to that of a convex optimization? That is, with SNES act as if it were Tao? On Fri, Apr 3, 2015 at 6:35 PM, Barry Smith wrote: > > Justin, > > We haven't done anything with TS to handle variational inequalities. So > you can either write your own backward Euler (outside of TS) that solves > each time-step problem either as 1) an optimization problem using Tao or 2) > as a variational inequality using SNES. > > More adventurously you could look at the TSTHETA code in TS (which is a > general form that includes Euler, Backward Euler and Crank-Nicolson and see > if you can add the constraints to the SNES problem that is solved; in > theory this is straightforward but it would require understanding the > current code (which Jed, of course, overwrote :-). I think you should do > this. > > Barry > > > > On Apr 3, 2015, at 12:31 PM, Justin Chang wrote: > > > > I am solving the following anisotropic transient diffusion equation > subject to 0 bounds: > > > > du/dt = div[D*grad[u]] + f > > > > Where the dispersion tensor D(x) is symmetric and positive definite. > This formulation violates the discrete maximum principles so one of the > ways to ensure nonnegative concentrations is to employ convex optimization. > I am following the procedures in Nakshatrala and Valocchi (2009) JCP and > Nagarajan and Nakshatrala (2011) IJNMF. > > > > The Variational Inequality method works gives what I want for my > transient case, but what if I want to implement the Tao methodology in TS? > That is, what TS functions do I need to set up steps a) through e) for each > time step (also the Jacobian remains the same for all time steps so I would > only call this once). Normally I would just call TSSolve() and let the > libraries and functions do everything, but I would like to incorporate > TaoSolve into every time step. > > > > Thanks, > > > > -- > > Justin Chang > > PhD Candidate, Civil Engineering - Computational Sciences > > University of Houston, Department of Civil and Environmental Engineering > > Houston, TX 77004 > > (512) 963-3262 > > > > On Thu, Apr 2, 2015 at 6:53 PM, Barry Smith wrote: > > > > An alternative approach is for you to solve it as a (non)linear > variational inequality. See src/snes/examples/tutorials/ex9.c > > > > How you should proceed depends on your long term goal. What problem do > you really want to solve? Is it really a linear time dependent problem with > 0 bounds on U? Can the problem always be represented as an optimization > problem easily? What are and what will be the properties of K? For example > if K is positive definite then likely the bounds will remain try without > explicitly providing the constraints. > > > > Barry > > > > > On Apr 2, 2015, at 6:39 PM, Justin Chang wrote: > > > > > > Hi everyone, > > > > > > I have a two part question regarding the integration of the following > optimization problem > > > > > > min 1/2 u^T*K*u + u^T*f > > > S.T. u >= 0 > > > > > > into SNES and TS > > > > > > 1) For SNES, assuming I am working with a linear FE equation, I have > the following algorithm/steps for solving my problem > > > > > > a) Set an initial guess x > > > b) Obtain residual r and jacobian A through functions > SNESComputeFunction() and SNESComputeJacobian() respectively > > > c) Form vector b = r - A*x > > > d) Set Hessian equal to A, gradient to A*x, objective function value > to 1/2*x^T*A*x + x^T*b, and variable (lower) bounds to a zero vector > > > e) Call TaoSolve > > > > > > This works well at the moment, but my question is there a more > "efficient" way of doing this? Because with my current setup, I am making a > rather bold assumption that my problem would converge in one SNES iteration > without the bounded constraints and does not have any unexpected > nonlinearities. > > > > > > 2) How would I go about doing the above for time-stepping problems? At > each time step, I want to solve a convex optimization subject to the lower > bounds constraint. I plan on using backward euler and my resulting jacobian > should still be compatible with the above optimization problem. > > > > > > Thanks, > > > > > > -- > > > Justin Chang > > > PhD Candidate, Civil Engineering - Computational Sciences > > > University of Houston, Department of Civil and Environmental > Engineering > > > Houston, TX 77004 > > > (512) 963-3262 > > > > > > > > > > -- > > Justin Chang > > PhD Candidate, Civil Engineering - Computational Sciences > > University of Houston, Department of Civil and Environmental Engineering > > Houston, TX 77004 > > (512) 963-3262 > > -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Apr 3 19:38:02 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 3 Apr 2015 19:38:02 -0500 Subject: [petsc-users] Integrating TAO into SNES and TS In-Reply-To: References: <16064BCD-8B50-4E95-AF7F-386F6780E645@mcs.anl.gov> <0759EFDD-F7BB-4826-B6E0-241159EF0D21@mcs.anl.gov> Message-ID: <6F8EDD4E-4AC5-4AEB-B593-6F0E682ABA1B@mcs.anl.gov> > On Apr 3, 2015, at 7:35 PM, Justin Chang wrote: > > I guess I will have to write my own code then :) > > I am not all that familiar with Variational Inequalities at the moment, but if my Jacobian is symmetric and positive definite and I only have lower and upper bounds, doesn't the problem simply reduce to that of a convex optimization? That is, with SNES act as if it were Tao? Yes, I think that is essentially correctly. Barry > > On Fri, Apr 3, 2015 at 6:35 PM, Barry Smith wrote: > > Justin, > > We haven't done anything with TS to handle variational inequalities. So you can either write your own backward Euler (outside of TS) that solves each time-step problem either as 1) an optimization problem using Tao or 2) as a variational inequality using SNES. > > More adventurously you could look at the TSTHETA code in TS (which is a general form that includes Euler, Backward Euler and Crank-Nicolson and see if you can add the constraints to the SNES problem that is solved; in theory this is straightforward but it would require understanding the current code (which Jed, of course, overwrote :-). I think you should do this. > > Barry > > > > On Apr 3, 2015, at 12:31 PM, Justin Chang wrote: > > > > I am solving the following anisotropic transient diffusion equation subject to 0 bounds: > > > > du/dt = div[D*grad[u]] + f > > > > Where the dispersion tensor D(x) is symmetric and positive definite. This formulation violates the discrete maximum principles so one of the ways to ensure nonnegative concentrations is to employ convex optimization. I am following the procedures in Nakshatrala and Valocchi (2009) JCP and Nagarajan and Nakshatrala (2011) IJNMF. > > > > The Variational Inequality method works gives what I want for my transient case, but what if I want to implement the Tao methodology in TS? That is, what TS functions do I need to set up steps a) through e) for each time step (also the Jacobian remains the same for all time steps so I would only call this once). Normally I would just call TSSolve() and let the libraries and functions do everything, but I would like to incorporate TaoSolve into every time step. > > > > Thanks, > > > > -- > > Justin Chang > > PhD Candidate, Civil Engineering - Computational Sciences > > University of Houston, Department of Civil and Environmental Engineering > > Houston, TX 77004 > > (512) 963-3262 > > > > On Thu, Apr 2, 2015 at 6:53 PM, Barry Smith wrote: > > > > An alternative approach is for you to solve it as a (non)linear variational inequality. See src/snes/examples/tutorials/ex9.c > > > > How you should proceed depends on your long term goal. What problem do you really want to solve? Is it really a linear time dependent problem with 0 bounds on U? Can the problem always be represented as an optimization problem easily? What are and what will be the properties of K? For example if K is positive definite then likely the bounds will remain try without explicitly providing the constraints. > > > > Barry > > > > > On Apr 2, 2015, at 6:39 PM, Justin Chang wrote: > > > > > > Hi everyone, > > > > > > I have a two part question regarding the integration of the following optimization problem > > > > > > min 1/2 u^T*K*u + u^T*f > > > S.T. u >= 0 > > > > > > into SNES and TS > > > > > > 1) For SNES, assuming I am working with a linear FE equation, I have the following algorithm/steps for solving my problem > > > > > > a) Set an initial guess x > > > b) Obtain residual r and jacobian A through functions SNESComputeFunction() and SNESComputeJacobian() respectively > > > c) Form vector b = r - A*x > > > d) Set Hessian equal to A, gradient to A*x, objective function value to 1/2*x^T*A*x + x^T*b, and variable (lower) bounds to a zero vector > > > e) Call TaoSolve > > > > > > This works well at the moment, but my question is there a more "efficient" way of doing this? Because with my current setup, I am making a rather bold assumption that my problem would converge in one SNES iteration without the bounded constraints and does not have any unexpected nonlinearities. > > > > > > 2) How would I go about doing the above for time-stepping problems? At each time step, I want to solve a convex optimization subject to the lower bounds constraint. I plan on using backward euler and my resulting jacobian should still be compatible with the above optimization problem. > > > > > > Thanks, > > > > > > -- > > > Justin Chang > > > PhD Candidate, Civil Engineering - Computational Sciences > > > University of Houston, Department of Civil and Environmental Engineering > > > Houston, TX 77004 > > > (512) 963-3262 > > > > > > > > > > -- > > Justin Chang > > PhD Candidate, Civil Engineering - Computational Sciences > > University of Houston, Department of Civil and Environmental Engineering > > Houston, TX 77004 > > (512) 963-3262 > > > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 From jchang27 at uh.edu Sat Apr 4 11:54:03 2015 From: jchang27 at uh.edu (Justin Chang) Date: Sat, 4 Apr 2015 11:54:03 -0500 Subject: [petsc-users] advection-diffusion pointwise function for FEM Message-ID: Hi everyone, I want to include advection into my diffusion FEM code (I am assuming a small Peclet number so stability isn't an issue ATM). That is I want to incorporate the second term as a pointwise function: du/dt + v * grad[c] - div[grad[c]] = f Where v is the velocity (obtained from the auxiliary term a_x[]). For the residual, would it be of the following form: f0_u(const PetscScalar u[], const PetscScalar u_t[], const PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const PetscScalar a_x[], const PetscReal x[], PetscScalar f0[]) { PetscInt d; f[0] = u_t[0]; for (d = 0; d < spatialDim; ++d) f0[0] += a_x[d]*u_x[d]; } What about the jacobian? My guess would be to use g1(...) but what would the inside of this function be? Thanks, -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesj at purdue.edu Sat Apr 4 19:33:14 2015 From: charlesj at purdue.edu (James A Charles) Date: Sat, 4 Apr 2015 20:33:14 -0400 (EDT) Subject: [petsc-users] MatMatMult with dense matrices. In-Reply-To: <608364017.275789.1428192602232.JavaMail.root@mailhub029.itcs.purdue.edu> Message-ID: <1198645843.275836.1428193994265.JavaMail.root@mailhub029.itcs.purdue.edu> Hello, I'm multiplying two dense matrices. I compared the results against MATLAB and I am getting different results. Attached you will see a zip that contains a .mat file that has the A, B, and C. C_MATLAB is the multiplication of A*B in MATLAB and C is the PETSc 3.4.3 result. I also attached the .m files A, B, and C that were saved from PETSc. Is there something I'm doing wrong or any reason due to the shape of the matrices that this would fail? If it gives any more information we convert the previous matrix that A is formed of A2 (A = A1*A2) to dense prior to the multiplication using MatConvert. Thanks for your help, James Charles -------------- next part -------------- A non-text attachment was scrubbed... Name: matmatmult_comparison.zip Type: application/zip Size: 1804789 bytes Desc: not available URL: From Sanjay.Kharche at manchester.ac.uk Sun Apr 5 07:39:49 2015 From: Sanjay.Kharche at manchester.ac.uk (Sanjay Kharche) Date: Sun, 5 Apr 2015 12:39:49 +0000 Subject: [petsc-users] compiling code that uses Petsc and Sundials Message-ID: Dear All I would like to compile my code that uses Petsc and Sundials using a morph of the petsc makefile on my supercomputer. There are a few technical issues that I will appreciate your feedback on. My code uses Petsc and Sundials. On my local machine, I installed Petsc with Sundials. The makefile examples I got allow me to compile my code which is a single file. I can see that -lsundials_cvode -lsundials_nvecserial -lsundials_nvecparallel are all there, as well as the blas lapack. Pasted below is my simple makefile and what happens when I make sk2d Now I want to port my code to our local supercomputer, where I am not admin. The admin have installed petsc. I managed to compile and run the examples of that install - so it works on its own. This copy of petsc was not installed by connecting to Sundials. Therefore, I have installed sundials in my home directory. I tried to add the sundials paths in the makefile below at various locations (in fact, I could not work out where exactly to put the includes and libs!), but I could not get my code to compile. Any suggestions? cheers Sanjay Local computer makefile: cat makefile include ${PETSC_DIR}/conf/variables include ${PETSC_DIR}/conf/rules sk2d: sk2d.o chkopts -${CLINKER} -o sk2d sk2d.o ${PETSC_TS_LIB} -lm -inline-calloc veryclean: rm -rf *.dat *.vtk *.vts ex15 *.o sk2d *~ output of make sk2d: make sk2d /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/bin/mpicc -o sk2d.o -c -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 -I/export2/sanjay/petsc/include -I/export2/sanjay/petsc/linux-gnu-c-debug/include -I/media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/include `pwd`/sk2d.c /export2/sanjay/mypetscProgs/Work/twod/sk2d.c: In function ?main?: /export2/sanjay/mypetscProgs/Work/twod/sk2d.c:112:24: warning: variable ?usr_time? set but not used [-Wunused-but-set-variable] /export2/sanjay/mypetscProgs/Work/twod/sk2d.c:111:7: warning: unused variable ?output? [-Wunused-variable] /export2/sanjay/mypetscProgs/Work/twod/sk2d.c:101:72: warning: unused variable ?usr_nbs? [-Wunused-variable] /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 -o sk2d sk2d.o -Wl,-rpath,/export2/sanjay/petsc/linux-gnu-c-debug/lib -L/export2/sanjay/petsc/linux-gnu-c-debug/lib -lpetsc -Wl,-rpath,/media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/lib -L/media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/lib -lscalapack -lsundials_cvode -lsundials_nvecserial -lsundials_nvecparallel -lflapack -lfblas -lX11 -lssl -lcrypto -lpthread -lm -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.6.3 -L/usr/lib/gcc/x86_64-redhat-linux/4.6.3 -lmpichf90 -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpichcxx -lstdc++ -Wl,-rpath,/media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/lib -L/media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.6.3 -L/usr/lib/gcc/x86_64-redhat-linux/4.6.3 -ldl -Wl,-rpath,/media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/lib -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -ldl -lm -inline-calloc From balay at mcs.anl.gov Sun Apr 5 13:32:48 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 5 Apr 2015 13:32:48 -0500 Subject: [petsc-users] compiling code that uses Petsc and Sundials In-Reply-To: References: Message-ID: 1. you can always rebuild PETSc on the "local supercomputer" Just grab the reconfigure_arch.py script from the current install [to capture the recommended mpi, blas etc..] - modify it as you need - and rerun it with your copy of PETSc sources. Wrt your install of sundials [assuming its built with the same compilers, mpi etc.. the following might work.. Satish ---------- $ cat makefile MYSUNDIALS_INCLUDE = -I/path/to/my-sundials/include MYSUNDIALS_LIB = -L/path/to/my-sundials -sundials_cvode -lsundials_nvecserial -lsundials_nvecparallel CPPFLAGS = ${MYSUNDIALS_INCLUDE} CLEANFILES = *.dat *.vtk *.vts ex15 *.o sk2d *~ include ${PETSC_DIR}/conf/variables include ${PETSC_DIR}/conf/rules sk2d: sk2d.o chkopts -${CLINKER} -o sk2d sk2d.o ${PETSC_TS_LIB} ${MYSUNDIALS_LIB} ${BLASLAPACK_LIB} ${MPI_LIB} -lm -inline-calloc On Sun, 5 Apr 2015, Sanjay Kharche wrote: > > Dear All > > I would like to compile my code that uses Petsc and Sundials using a morph of the petsc makefile on my supercomputer. There are a few technical issues that I will appreciate your feedback on. > > My code uses Petsc and Sundials. On my local machine, I installed Petsc with Sundials. The makefile examples I got allow me to compile my code which is a single file. I can see that -lsundials_cvode -lsundials_nvecserial -lsundials_nvecparallel are all there, as well as the blas lapack. Pasted below is my simple makefile and what happens when I make sk2d > > Now I want to port my code to our local supercomputer, where I am not admin. The admin have installed petsc. I managed to compile and run the examples of that install - so it works on its own. This copy of petsc was not installed by connecting to Sundials. Therefore, I have installed sundials in my home directory. I tried to add the sundials paths in the makefile below at various locations (in fact, I could not work out where exactly to put the includes and libs!), but I could not get my code to compile. Any suggestions? > > cheers > Sanjay > > Local computer makefile: > cat makefile > > include ${PETSC_DIR}/conf/variables > include ${PETSC_DIR}/conf/rules > > sk2d: sk2d.o chkopts > -${CLINKER} -o sk2d sk2d.o ${PETSC_TS_LIB} -lm -inline-calloc > > veryclean: > rm -rf *.dat *.vtk *.vts ex15 *.o sk2d *~ > > output of make sk2d: > > make sk2d > /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/bin/mpicc -o sk2d.o -c -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 -I/export2/sanjay/petsc/include -I/export2/sanjay/petsc/linux-gnu-c-debug/include -I/media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/include `pwd`/sk2d.c > /export2/sanjay/mypetscProgs/Work/twod/sk2d.c: In function ?main?: > /export2/sanjay/mypetscProgs/Work/twod/sk2d.c:112:24: warning: variable ?usr_time? set but not used [-Wunused-but-set-variable] > /export2/sanjay/mypetscProgs/Work/twod/sk2d.c:111:7: warning: unused variable ?output? [-Wunused-variable] > /export2/sanjay/mypetscProgs/Work/twod/sk2d.c:101:72: warning: unused variable ?usr_nbs? [-Wunused-variable] > /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 -o sk2d sk2d.o -Wl,-rpath,/export2/sanjay/petsc/linux-gnu-c-debug/lib -L/export2/sanjay/petsc/linux-gnu-c-debug/lib -lpetsc -Wl,-rpath,/media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/lib -L/media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/lib -lscalapack -lsundials_cvode -lsundials_nvecserial -lsundials_nvecparallel -lflapack -lfblas -lX11 -lssl -lcrypto -lpthread -lm -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.6.3 -L/usr/lib/gcc/x86_64-redhat-linux/4.6.3 -lmpichf90 -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpichcxx -lstdc++ -Wl,-rpath,/media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/lib -L/media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.6.3 -L/usr/lib/gcc/x86_64-redhat-linux/4.6.3 -ldl -Wl,-rpath,/media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/lib -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -ldl -lm -inline-calloc > From S.R.Kharche at exeter.ac.uk Sun Apr 5 14:37:34 2015 From: S.R.Kharche at exeter.ac.uk (Kharche, Sanjay) Date: Sun, 5 Apr 2015 19:37:34 +0000 Subject: [petsc-users] compiling code that uses Petsc and Sundials In-Reply-To: References: , Message-ID: I just tried the makefile trick, I think that will serve my purpose for now. thanks. Sanjay ________________________________________ From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Satish Balay [balay at mcs.anl.gov] Sent: 05 April 2015 19:32 To: Sanjay Kharche Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] compiling code that uses Petsc and Sundials 1. you can always rebuild PETSc on the "local supercomputer" Just grab the reconfigure_arch.py script from the current install [to capture the recommended mpi, blas etc..] - modify it as you need - and rerun it with your copy of PETSc sources. Wrt your install of sundials [assuming its built with the same compilers, mpi etc.. the following might work.. Satish ---------- $ cat makefile MYSUNDIALS_INCLUDE = -I/path/to/my-sundials/include MYSUNDIALS_LIB = -L/path/to/my-sundials -sundials_cvode -lsundials_nvecserial -lsundials_nvecparallel CPPFLAGS = ${MYSUNDIALS_INCLUDE} CLEANFILES = *.dat *.vtk *.vts ex15 *.o sk2d *~ include ${PETSC_DIR}/conf/variables include ${PETSC_DIR}/conf/rules sk2d: sk2d.o chkopts -${CLINKER} -o sk2d sk2d.o ${PETSC_TS_LIB} ${MYSUNDIALS_LIB} ${BLASLAPACK_LIB} ${MPI_LIB} -lm -inline-calloc On Sun, 5 Apr 2015, Sanjay Kharche wrote: > > Dear All > > I would like to compile my code that uses Petsc and Sundials using a morph of the petsc makefile on my supercomputer. There are a few technical issues that I will appreciate your feedback on. > > My code uses Petsc and Sundials. On my local machine, I installed Petsc with Sundials. The makefile examples I got allow me to compile my code which is a single file. I can see that -lsundials_cvode -lsundials_nvecserial -lsundials_nvecparallel are all there, as well as the blas lapack. Pasted below is my simple makefile and what happens when I make sk2d > > Now I want to port my code to our local supercomputer, where I am not admin. The admin have installed petsc. I managed to compile and run the examples of that install - so it works on its own. This copy of petsc was not installed by connecting to Sundials. Therefore, I have installed sundials in my home directory. I tried to add the sundials paths in the makefile below at various locations (in fact, I could not work out where exactly to put the includes and libs!), but I could not get my code to compile. Any suggestions? > > cheers > Sanjay > > Local computer makefile: > cat makefile > > include ${PETSC_DIR}/conf/variables > include ${PETSC_DIR}/conf/rules > > sk2d: sk2d.o chkopts > -${CLINKER} -o sk2d sk2d.o ${PETSC_TS_LIB} -lm -inline-calloc > > veryclean: > rm -rf *.dat *.vtk *.vts ex15 *.o sk2d *~ > > output of make sk2d: > > make sk2d > /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/bin/mpicc -o sk2d.o -c -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 -I/export2/sanjay/petsc/include -I/export2/sanjay/petsc/linux-gnu-c-debug/include -I/media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/include `pwd`/sk2d.c > /export2/sanjay/mypetscProgs/Work/twod/sk2d.c: In function ?main?: > /export2/sanjay/mypetscProgs/Work/twod/sk2d.c:112:24: warning: variable ?usr_time? set but not used [-Wunused-but-set-variable] > /export2/sanjay/mypetscProgs/Work/twod/sk2d.c:111:7: warning: unused variable ?output? [-Wunused-variable] > /export2/sanjay/mypetscProgs/Work/twod/sk2d.c:101:72: warning: unused variable ?usr_nbs? [-Wunused-variable] > /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 -o sk2d sk2d.o -Wl,-rpath,/export2/sanjay/petsc/linux-gnu-c-debug/lib -L/export2/sanjay/petsc/linux-gnu-c-debug/lib -lpetsc -Wl,-rpath,/media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/lib -L/media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/lib -lscalapack -lsundials_cvode -lsundials_nvecserial -lsundials_nvecparallel -lflapack -lfblas -lX11 -lssl -lcrypto -lpthread -lm -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.6.3 -L/usr/lib/gcc/x86_64-redhat-linux/4.6.3 -lmpichf90 -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpichcxx -lstdc++ -Wl,-rpath,/media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/lib -L/media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.6.3 -L/usr/lib/gcc/x86_64-redhat-linux/4.6.3 -ldl -Wl,-rpath,/media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/linux-gnu-c-debug/lib -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -ldl -lm -inline-calloc > From hzhang at mcs.anl.gov Sun Apr 5 15:25:39 2015 From: hzhang at mcs.anl.gov (Hong) Date: Sun, 5 Apr 2015 15:25:39 -0500 Subject: [petsc-users] MatMatMult with dense matrices. In-Reply-To: <1198645843.275836.1428193994265.JavaMail.root@mailhub029.itcs.purdue.edu> References: <608364017.275789.1428192602232.JavaMail.root@mailhub029.itcs.purdue.edu> <1198645843.275836.1428193994265.JavaMail.root@mailhub029.itcs.purdue.edu> Message-ID: James: > > > I'm multiplying two dense matrices. I compared the results against MATLAB > and I am getting different results. Attached you will see a zip that > contains a .mat file that has the A, B, and C. C_MATLAB is the > multiplication of A*B in MATLAB and C is the PETSc 3.4.3 result. I also > attached the .m files A, B, and C that were saved from PETSc. Is there > something I'm doing wrong or What is the file type of matmatmult_comparison.mat? How to read it? I need to see how it defers from C.m. any reason due to the shape of the matrices that this would fail? If it > gives any more information we convert the previous matrix that A is formed > of A2 (A = A1*A2) to dense prior to the multiplication using MatConvert. > It seems both A and B are dense, complex square matrices. Did you call MatMatMult() in sequential or parallel? What matrix format did you use? Hong -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesj at purdue.edu Sun Apr 5 15:35:38 2015 From: charlesj at purdue.edu (James A Charles) Date: Sun, 5 Apr 2015 16:35:38 -0400 (EDT) Subject: [petsc-users] MatMatMult with dense matrices. In-Reply-To: Message-ID: <1216207222.276864.1428266138309.JavaMail.root@mailhub029.itcs.purdue.edu> Hi Hong, You can open up matmatmult_comparison.mat in MATLAB ( load matmatmult_comparison.mat ). That .mat contains A, B, C, and C_MATLAB. I called matmatmult in sequential. We use AIJ when A2 was sparse but at the time of this multiplication everything was dense. Thanks, James ----- Original Message ----- From: "Hong" To: "James A Charles" Cc: "PETSc users list" , dlemus at purdue.edu, "Daniel Mejia" , "Tillmann Kubis" Sent: Sunday, April 5, 2015 4:25:39 PM Subject: Re: [petsc-users] MatMatMult with dense matrices. James: I'm multiplying two dense matrices. I compared the results against MATLAB and I am getting different results. Attached you will see a zip that contains a .mat file that has the A, B, and C. C_MATLAB is the multiplication of A*B in MATLAB and C is the PETSc 3.4.3 result. I also attached the .m files A, B, and C that were saved from PETSc. Is there something I'm doing wrong or What is the file type of matmatmult_comparison.mat? How to read it? I need to see how it defers from C.m. any reason due to the shape of the matrices that this would fail? If it gives any more information we convert the previous matrix that A is formed of A2 (A = A1*A2) to dense prior to the multiplication using MatConvert. It seems both A and B are dense, complex square matrices. Did you call MatMatMult() in sequential or parallel? What matrix format did you use? Hong From bsmith at mcs.anl.gov Sun Apr 5 15:44:13 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 5 Apr 2015 15:44:13 -0500 Subject: [petsc-users] MatMatMult with dense matrices. In-Reply-To: <1216207222.276864.1428266138309.JavaMail.root@mailhub029.itcs.purdue.edu> References: <1216207222.276864.1428266138309.JavaMail.root@mailhub029.itcs.purdue.edu> Message-ID: We would need to see the PETSc side of the code to see if there is anything wrong there. > On Apr 5, 2015, at 3:35 PM, James A Charles wrote: > > Hi Hong, > > You can open up matmatmult_comparison.mat in MATLAB ( load matmatmult_comparison.mat ). That .mat contains A, B, C, and C_MATLAB. I called matmatmult in sequential. We use AIJ when A2 was sparse but at the time of this multiplication everything was dense. > > Thanks, > James > > > ----- Original Message ----- > From: "Hong" > To: "James A Charles" > Cc: "PETSc users list" , dlemus at purdue.edu, "Daniel Mejia" , "Tillmann Kubis" > Sent: Sunday, April 5, 2015 4:25:39 PM > Subject: Re: [petsc-users] MatMatMult with dense matrices. > > > > > James: > > > I'm multiplying two dense matrices. I compared the results against MATLAB and I am getting different results. Attached you will see a zip that contains a .mat file that has the A, B, and C. C_MATLAB is the multiplication of A*B in MATLAB and C is the PETSc 3.4.3 result. I also attached the .m files A, B, and C that were saved from PETSc. Is there something I'm doing wrong or > > > What is the file type of matmatmult_comparison.mat? > How to read it? > I need to see how it defers from C.m. > > > > any reason due to the shape of the matrices that this would fail? If it gives any more information we convert the previous matrix that A is formed of A2 (A = A1*A2) to dense prior to the multiplication using MatConvert. > > > > It seems both A and B are dense, complex square matrices. > Did you call MatMatMult() in sequential or parallel? What matrix format did you use? > > > Hong From hzhang at mcs.anl.gov Sun Apr 5 17:13:42 2015 From: hzhang at mcs.anl.gov (Hong) Date: Sun, 5 Apr 2015 17:13:42 -0500 Subject: [petsc-users] MatMatMult with dense matrices. In-Reply-To: References: <1216207222.276864.1428266138309.JavaMail.root@mailhub029.itcs.purdue.edu> Message-ID: James, My understanding is that you call MatMatMult in sequential for computing C = A*B, in which A is in aij format, and B is dense, and the product C is dense, but gives different values than what you got from Matlab. Do you have a stand-alone petsc code to reproduce this so I can investigate what goes wrong? Hong On Sun, Apr 5, 2015 at 3:44 PM, Barry Smith wrote: > > We would need to see the PETSc side of the code to see if there is > anything wrong there. > > > On Apr 5, 2015, at 3:35 PM, James A Charles wrote: > > > > Hi Hong, > > > > You can open up matmatmult_comparison.mat in MATLAB ( load > matmatmult_comparison.mat ). That .mat contains A, B, C, and C_MATLAB. I > called matmatmult in sequential. We use AIJ when A2 was sparse but at the > time of this multiplication everything was dense. > > > > Thanks, > > James > > > > > > ----- Original Message ----- > > From: "Hong" > > To: "James A Charles" > > Cc: "PETSc users list" , dlemus at purdue.edu, > "Daniel Mejia" , "Tillmann Kubis" > > Sent: Sunday, April 5, 2015 4:25:39 PM > > Subject: Re: [petsc-users] MatMatMult with dense matrices. > > > > > > > > > > James: > > > > > > I'm multiplying two dense matrices. I compared the results against > MATLAB and I am getting different results. Attached you will see a zip that > contains a .mat file that has the A, B, and C. C_MATLAB is the > multiplication of A*B in MATLAB and C is the PETSc 3.4.3 result. I also > attached the .m files A, B, and C that were saved from PETSc. Is there > something I'm doing wrong or > > > > > > What is the file type of matmatmult_comparison.mat? > > How to read it? > > I need to see how it defers from C.m. > > > > > > > > any reason due to the shape of the matrices that this would fail? If it > gives any more information we convert the previous matrix that A is formed > of A2 (A = A1*A2) to dense prior to the multiplication using MatConvert. > > > > > > > > It seems both A and B are dense, complex square matrices. > > Did you call MatMatMult() in sequential or parallel? What matrix format > did you use? > > > > > > Hong > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesj at purdue.edu Sun Apr 5 17:15:50 2015 From: charlesj at purdue.edu (James A Charles) Date: Sun, 5 Apr 2015 18:15:50 -0400 (EDT) Subject: [petsc-users] MatMatMult with dense matrices. In-Reply-To: Message-ID: <1385051439.277105.1428272150210.JavaMail.root@mailhub029.itcs.purdue.edu> Hi Hong, A is also dense. It was converted to dense in an intermediate calculation before this particular multiplication. So far I have only been doing this in our code. Let me write a stand alone PETSc cdoe and I will get back to you. Thanks, James ----- Original Message ----- From: "Hong" To: "Barry Smith" Cc: "James A Charles" , "Hong" , dlemus at purdue.edu, "PETSc users list" , "Daniel Mejia" , "Tillmann Kubis" Sent: Sunday, April 5, 2015 6:13:42 PM Subject: Re: [petsc-users] MatMatMult with dense matrices. James, My understanding is that you call MatMatMult in sequential for computing C = A*B, in which A is in aij format, and B is dense, and the product C is dense, but gives different values than what you got from Matlab. Do you have a stand-alone petsc code to reproduce this so I can investigate what goes wrong? Hong On Sun, Apr 5, 2015 at 3:44 PM, Barry Smith < bsmith at mcs.anl.gov > wrote: We would need to see the PETSc side of the code to see if there is anything wrong there. > On Apr 5, 2015, at 3:35 PM, James A Charles < charlesj at purdue.edu > wrote: > > Hi Hong, > > You can open up matmatmult_comparison.mat in MATLAB ( load matmatmult_comparison.mat ). That .mat contains A, B, C, and C_MATLAB. I called matmatmult in sequential. We use AIJ when A2 was sparse but at the time of this multiplication everything was dense. > > Thanks, > James > > > ----- Original Message ----- > From: "Hong" < hzhang at mcs.anl.gov > > To: "James A Charles" < charlesj at purdue.edu > > Cc: "PETSc users list" < petsc-users at mcs.anl.gov >, dlemus at purdue.edu , "Daniel Mejia" < denphi at denphi.com >, "Tillmann Kubis" < tkubis at purdue.edu > > Sent: Sunday, April 5, 2015 4:25:39 PM > Subject: Re: [petsc-users] MatMatMult with dense matrices. > > > > > James: > > > I'm multiplying two dense matrices. I compared the results against MATLAB and I am getting different results. Attached you will see a zip that contains a .mat file that has the A, B, and C. C_MATLAB is the multiplication of A*B in MATLAB and C is the PETSc 3.4.3 result. I also attached the .m files A, B, and C that were saved from PETSc. Is there something I'm doing wrong or > > > What is the file type of matmatmult_comparison.mat? > How to read it? > I need to see how it defers from C.m. > > > > any reason due to the shape of the matrices that this would fail? If it gives any more information we convert the previous matrix that A is formed of A2 (A = A1*A2) to dense prior to the multiplication using MatConvert. > > > > It seems both A and B are dense, complex square matrices. > Did you call MatMatMult() in sequential or parallel? What matrix format did you use? > > > Hong From emittelstaedt at uidaho.edu Mon Apr 6 00:16:51 2015 From: emittelstaedt at uidaho.edu (Eric Mittelstaedt) Date: Sun, 5 Apr 2015 22:16:51 -0700 Subject: [petsc-users] Assembling Restriction and Prolongation Operators for Parallel Calculations Message-ID: <552216C3.1090000@uidaho.edu> Hello all, We are working on implementing restriction and prolongation operators for algebraic multigrid in our finite difference (staggered grid) code that solves the incompressible Stokes problem. So far, the restriction and prolongation operators we have assembled in serial can successfully solve on a simple 2D grid with multiple-level v-cycles. However, we are running into problems when we attempt to progress to parallel solves. One of the primary errors we run into is an incompatibility in matrix types for MatMatMatMult and similar calls: [0]PETSC ERROR: [1]PETSC ERROR: MatMatMatMult() line 9173 in /opt/petsc/src/mat/interface/matrix.c MatMatMatMult requires A, seqaij, to be compatible with B, mpiaij, C, seqaij MatMatMatMult() line 9173 in /opt/petsc/src/mat/interface/matrix.c MatMatMatMult requires A, seqaij, to be compatible with B, mpiaij, C, seqaij The root cause of this is likely the matrix types we chose for restriction and prolongation. To ease calculation of our restriction (R) and prolongation (P) matrices, we have created them as sparse, sequential matrices (SEQAIJ) that are identical on all procs. Will this matrix type actually work for R and P operators in parallel? It seem that setting this matrix type correctly is the heart of our problem. Originally, we chose SEQAIJ because we want to let Petsc deal with partitioning of the grid on multiple processors. If we do this, however, we don't know a priori whether Petsc will place a processor boundary on an even or odd nodal point. This makes determining the processor bounds on the subsequent coarse levels difficult. One method for resolving this may be to handle the partitioning ourselves, but it would be nice to avoid this if possible. Is there a better way to set up these operators? Thank you, Eric and Arthur From david.knezevic at akselos.com Mon Apr 6 08:30:23 2015 From: david.knezevic at akselos.com (David Knezevic) Date: Mon, 6 Apr 2015 09:30:23 -0400 Subject: [petsc-users] Status of mkl_cpardiso configuration option? Message-ID: Hi all, I'm interested in trying out mkl_cpardiso with PETSc (since I gather that mkl_pardiso doesn't work with MPI, and mkl_cpardiso is included in MKL). I saw a thread on the petsc-users list about this from 2013 (subject: "mkl-cpardiso configuration"), so I was wondering what the status of this is? I see that there is a configuration option for mkl_pardiso (I tried this and it works well for me). But I gather that there is no configuration option for mkl_cpardiso at this stage, is that correct? Is this configuration option something that you're interested in adding? If so I'd be happy to help out if possible. Thanks, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Apr 6 08:49:05 2015 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 6 Apr 2015 08:49:05 -0500 Subject: [petsc-users] Assembling Restriction and Prolongation Operators for Parallel Calculations In-Reply-To: <552216C3.1090000@uidaho.edu> References: <552216C3.1090000@uidaho.edu> Message-ID: On Mon, Apr 6, 2015 at 12:16 AM, Eric Mittelstaedt wrote: > Hello all, > We are working on implementing restriction and prolongation operators > for algebraic multigrid in our finite difference (staggered grid) code that > solves the incompressible Stokes problem. So far, the restriction and > prolongation operators we have assembled in serial can successfully solve > on a simple 2D grid with multiple-level v-cycles. However, we are running > into problems when we attempt to progress to parallel solves. One of the > primary errors we run into is an incompatibility in matrix types for > MatMatMatMult and similar calls: > > [0]PETSC ERROR: [1]PETSC ERROR: MatMatMatMult() line 9173 in > /opt/petsc/src/mat/interface/matrix.c MatMatMatMult requires A, seqaij, > to be compatible with B, mpiaij, C, seqaij > MatMatMatMult() line 9173 in /opt/petsc/src/mat/interface/matrix.c > MatMatMatMult requires A, seqaij, to be compatible with B, mpiaij, C, seqaij > > > The root cause of this is likely the matrix types we chose for restriction > and prolongation. To ease calculation of our restriction (R) and > prolongation (P) matrices, we have created them as sparse, sequential > matrices (SEQAIJ) that are identical on all procs. Will this matrix type > actually work for R and P operators in parallel? > You need to create them as parallel matrices, MPIAIJ for example, that have the same layout as the system matrix. > It seem that setting this matrix type correctly is the heart of our > problem. Originally, we chose SEQAIJ because we want to let Petsc deal > with partitioning of the grid on multiple processors. If we do this, > however, we don't know a priori whether Petsc will place a processor > boundary on an even or odd nodal point. This makes determining the > processor bounds on the subsequent coarse levels difficult. One method for > resolving this may be to handle the partitioning ourselves, but it would be > nice to avoid this if possible. Is there a better way to set up these > operators? > You use the same partitioning as the system matrices on the two levels. Does this makes sense? Thanks, Matt > Thank you, > Eric and Arthur > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Apr 6 11:19:24 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 6 Apr 2015 11:19:24 -0500 Subject: [petsc-users] Status of mkl_cpardiso configuration option? In-Reply-To: References: Message-ID: If you use petsc development branch 'master' from git - you should be able to use mkl cPardiso. https://bitbucket.org/petsc/petsc/commits/d305a81be0fbb2fe8c02e3da7510a1143eafa263 Satish On Mon, 6 Apr 2015, David Knezevic wrote: > Hi all, > > I'm interested in trying out mkl_cpardiso with PETSc (since I gather that > mkl_pardiso doesn't work with MPI, and mkl_cpardiso is included in MKL). I > saw a thread on the petsc-users list about this from 2013 (subject: > "mkl-cpardiso configuration"), so I was wondering what the status of this > is? > > I see that there is a configuration option for mkl_pardiso (I tried this > and it works well for me). But I gather that there is no configuration > option for mkl_cpardiso at this stage, is that correct? Is this > configuration option something that you're interested in adding? If so I'd > be happy to help out if possible. > > Thanks, > David > From david.knezevic at akselos.com Mon Apr 6 11:29:06 2015 From: david.knezevic at akselos.com (David Knezevic) Date: Mon, 6 Apr 2015 12:29:06 -0400 Subject: [petsc-users] Status of mkl_cpardiso configuration option? In-Reply-To: References: Message-ID: On Mon, Apr 6, 2015 at 12:19 PM, Satish Balay wrote: > If you use petsc development branch 'master' from git - you should be > able to use mkl cPardiso. > > > https://bitbucket.org/petsc/petsc/commits/d305a81be0fbb2fe8c02e3da7510a1143eafa263 > > Satish > OK, thanks! David -------------- next part -------------- An HTML attachment was scrubbed... URL: From emittelstaedt at uidaho.edu Mon Apr 6 15:59:36 2015 From: emittelstaedt at uidaho.edu (Eric Mittelstaedt) Date: Mon, 6 Apr 2015 13:59:36 -0700 Subject: [petsc-users] Assembling Restriction and Prolongation Operators for Parallel Calculations In-Reply-To: References: <552216C3.1090000@uidaho.edu> Message-ID: <5522F3B8.7090409@uidaho.edu> Hi Matt On 4/6/2015 6:49 AM, Matthew Knepley wrote: > On Mon, Apr 6, 2015 at 12:16 AM, Eric Mittelstaedt > > wrote: > > Hello all, > We are working on implementing restriction and prolongation > operators for algebraic multigrid in our finite difference > (staggered grid) code that solves the incompressible Stokes > problem. So far, the restriction and prolongation operators we > have assembled in serial can successfully solve on a simple 2D > grid with multiple-level v-cycles. However, we are running into > problems when we attempt to progress to parallel solves. One of > the primary errors we run into is an incompatibility in matrix > types for MatMatMatMult and similar calls: > > [0]PETSC ERROR: [1]PETSC ERROR: MatMatMatMult() line 9173 in > /opt/petsc/src/mat/interface/matrix.c MatMatMatMult requires A, > seqaij, to be compatible with B, mpiaij, C, seqaij > MatMatMatMult() line 9173 in /opt/petsc/src/mat/interface/matrix.c > MatMatMatMult requires A, seqaij, to be compatible with B, mpiaij, > C, seqaij > > > The root cause of this is likely the matrix types we chose for > restriction and prolongation. To ease calculation of our > restriction (R) and prolongation (P) matrices, we have created > them as sparse, sequential matrices (SEQAIJ) that are identical on > all procs. Will this matrix type actually work for R and P > operators in parallel? > > > You need to create them as parallel matrices, MPIAIJ for example, that > have the same layout as the system matrix. Okay, this is what I thought was the problem. It is really a matter of making sure we have the correct information on each processor. > > It seem that setting this matrix type correctly is the heart of > our problem. Originally, we chose SEQAIJ because we want to let > Petsc deal with partitioning of the grid on multiple processors. > If we do this, however, we don't know a priori whether Petsc will > place a processor boundary on an even or odd nodal point. This > makes determining the processor bounds on the subsequent coarse > levels difficult. One method for resolving this may be to handle > the partitioning ourselves, but it would be nice to avoid this if > possible. Is there a better way to set up these operators? > > > You use the same partitioning as the system matrices on the two > levels. Does this makes sense? > It does, but I am unsure how to get the partitioning of the coarser grids since we don't deal with them directly. Is it safe to assume that the partitioning on the coarser grid will always overlap with the same part of the domain as the fine level? In other words, the current processor will receive the coarse grid that is formed by the portion of the restriction matrix that it owns, right? Thanks! Eric > Thanks, > > Matt > > Thank you, > Eric and Arthur > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -- ************************************* Eric Mittelstaedt, Ph.D. Assistant Professor Dept. of Geological Sciences University of Idaho email:emittelstaedt at uidaho.edu phone: (208) 885 2045 http://scholar.google.com/citations?user=DvRzEqkAAAAJ ************************************* -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Apr 6 16:12:44 2015 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 6 Apr 2015 16:12:44 -0500 Subject: [petsc-users] Assembling Restriction and Prolongation Operators for Parallel Calculations In-Reply-To: <5522F3B8.7090409@uidaho.edu> References: <552216C3.1090000@uidaho.edu> <5522F3B8.7090409@uidaho.edu> Message-ID: On Mon, Apr 6, 2015 at 3:59 PM, Eric Mittelstaedt wrote: > Hi Matt > > On 4/6/2015 6:49 AM, Matthew Knepley wrote: > > On Mon, Apr 6, 2015 at 12:16 AM, Eric Mittelstaedt < > emittelstaedt at uidaho.edu> wrote: > >> Hello all, >> We are working on implementing restriction and prolongation operators >> for algebraic multigrid in our finite difference (staggered grid) code that >> solves the incompressible Stokes problem. So far, the restriction and >> prolongation operators we have assembled in serial can successfully solve >> on a simple 2D grid with multiple-level v-cycles. However, we are running >> into problems when we attempt to progress to parallel solves. One of the >> primary errors we run into is an incompatibility in matrix types for >> MatMatMatMult and similar calls: >> >> [0]PETSC ERROR: [1]PETSC ERROR: MatMatMatMult() line 9173 in >> /opt/petsc/src/mat/interface/matrix.c MatMatMatMult requires A, seqaij, to >> be compatible with B, mpiaij, C, seqaij >> MatMatMatMult() line 9173 in /opt/petsc/src/mat/interface/matrix.c >> MatMatMatMult requires A, seqaij, to be compatible with B, mpiaij, C, seqaij >> >> >> The root cause of this is likely the matrix types we chose for >> restriction and prolongation. To ease calculation of our restriction (R) >> and prolongation (P) matrices, we have created them as sparse, sequential >> matrices (SEQAIJ) that are identical on all procs. Will this matrix type >> actually work for R and P operators in parallel? >> > > You need to create them as parallel matrices, MPIAIJ for example, that > have the same layout as the system matrix. > > > Okay, this is what I thought was the problem. It is really a matter of > making sure we have the correct information on each processor. > > > >> It seem that setting this matrix type correctly is the heart of our >> problem. Originally, we chose SEQAIJ because we want to let Petsc deal >> with partitioning of the grid on multiple processors. If we do this, >> however, we don't know a priori whether Petsc will place a processor >> boundary on an even or odd nodal point. This makes determining the >> processor bounds on the subsequent coarse levels difficult. One method for >> resolving this may be to handle the partitioning ourselves, but it would be >> nice to avoid this if possible. Is there a better way to set up these >> operators? >> > > You use the same partitioning as the system matrices on the two levels. > Does this makes sense? > > It does, but I am unsure how to get the partitioning of the coarser > grids since we don't deal with them directly. Is it safe to assume that > the partitioning on the coarser grid will always overlap with the same part > of the domain as the fine level? In other words, the current processor will > receive the coarse grid that is formed by the portion of the restriction > matrix that it owns, right? > You can get the coarse grid by calling DMCoarsen(), just like the AMG will do. Then you can ask the coverage questions directly to the DMDA. I don't remember exactly what we guarantee with regard to coverage of the fine grid by the coarse. Maybe Jed does since he jsut rewrote this for the HPGMG benchmark. Thanks, MAtt > Thanks! > Eric > > > Thanks, > > Matt > > > >> Thank you, >> Eric and Arthur >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- > > ************************************* > Eric Mittelstaedt, Ph.D. > Assistant Professor > Dept. of Geological Sciences > University of Idaho > email: emittelstaedt at uidaho.edu > phone: (208) 885 2045http://scholar.google.com/citations?user=DvRzEqkAAAAJ > ************************************* > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Lukasz.Kaczmarczyk at glasgow.ac.uk Tue Apr 7 06:38:01 2015 From: Lukasz.Kaczmarczyk at glasgow.ac.uk (Lukasz Kaczmarczyk) Date: Tue, 7 Apr 2015 11:38:01 +0000 Subject: [petsc-users] Problem with diagonal MPIAIJ matrix Message-ID: Hello, Pleas see attached following code, http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/_p_c_m_g_set_up_via_approx_orders_8cpp_source.html I encounter following error, if restriction and interpolation operator is the same for multigrid solver. This happens when I use hierarchical approx. basis and my R matrix is has only values on diagonal. If I set PCMGSetRestriction it works great, but I believe that this is not optimal solution. Could you advise some solution for this? 176 // Petsc Development GIT revision: v3.5.3-1524-gee900cc GIT Date: 2015-01-31 17:44:15 -0600 178 // [0]PETSC ERROR: [0] MatPtAPSymbolic_MPIAIJ_MPIAIJ line 124 /opt/petsc/src/mat/impls/aij/mpi/mpiptap.c 179 // [0]PETSC ERROR: [0] MatPtAP_MPIAIJ_MPIAIJ line 80 /opt/petsc/src/mat/impls/aij/mpi/mpiptap.c 180 // [0]PETSC ERROR: [0] MatPtAP line 8458 /opt/petsc/src/mat/interface/matrix.c 181 // [0]PETSC ERROR: [0] PCSetUp_MG line 552 /opt/petsc/src/ksp/pc/impls/mg/mg.c 182 // [0]PETSC ERROR: [0] KSPSetUp line 220 /opt/petsc/src/ksp/ksp/interface/itfunc.c 183 184 // ==4284== Invalid read of size 8 185 // ==4284== at 0x5CAC873: MatPtAPSymbolic_MPIAIJ_MPIAIJ (mpiptap.c:154) 186 // ==4284== by 0x5CABA74: MatPtAP_MPIAIJ_MPIAIJ (mpiptap.c:83) 187 // ==4284== by 0x5D566D1: MatPtAP (matrix.c:8537) 188 // ==4284== by 0x61F27D6: PCSetUp_MG (mg.c:642) 189 // ==4284== by 0x612BD9C: PCSetUp (precon.c:909) 190 // ==4284== by 0x62C1A51: KSPSetUp (itfunc.c:306) 191 // ==4284== by 0xB98326: main (elasticity.cpp:403) 192 // ==4284== Address 0x208 is not stack'd, malloc'd or (recently) free'd Kind regards, Lukasz From bhatiamanav at gmail.com Tue Apr 7 07:06:56 2015 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Tue, 7 Apr 2015 07:06:56 -0500 Subject: [petsc-users] iterative solvers Message-ID: Hi, I am solving a transonic Euler flow problem discretized with SUPG. The mesh is made of Tet4 elements and there are about 7M dofs, which I am trying to solve over 192 cores. I had earlier written about the linear solver returning with Inf, and have since moved beyond that such that I am able to get a solution with the following command line parameters: -ksp_gmres_restart 100 -pc_type asm -sub_pc_type ilu -sub_pc_factor_levels 4 -sub_ksp_type preonly The problem now is that I am limited to very small time-steps (~10e-6), outside of which the solver starts to choke and the solution diverges. I have tried other command line options and this above combination is what seems to provide the best possible solution strategy, albeit constrained in time steps. I would appreciate any recommendations on solver configurations that may allow me to take larger time-steps. Would coding up AMG be a possible alternative (more robust and dependable)? Please let me know if you need additional information. Regards, Manav -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesj at purdue.edu Tue Apr 7 13:58:35 2015 From: charlesj at purdue.edu (James A Charles) Date: Tue, 7 Apr 2015 14:58:35 -0400 (EDT) Subject: [petsc-users] MatMatMult with dense matrices. In-Reply-To: Message-ID: <1631590191.289162.1428433115075.JavaMail.root@mailhub029.itcs.purdue.edu> Hello, Thanks a lot for looking into this. I'm still working on debugging this on our side. It might be an issue with us. I will keep you updated. Thanks, James ----- Original Message ----- From: "Hong" To: "Barry Smith" Cc: "James A Charles" , "Hong" , dlemus at purdue.edu, "PETSc users list" , "Daniel Mejia" , "Tillmann Kubis" Sent: Sunday, April 5, 2015 6:13:42 PM Subject: Re: [petsc-users] MatMatMult with dense matrices. James, My understanding is that you call MatMatMult in sequential for computing C = A*B, in which A is in aij format, and B is dense, and the product C is dense, but gives different values than what you got from Matlab. Do you have a stand-alone petsc code to reproduce this so I can investigate what goes wrong? Hong On Sun, Apr 5, 2015 at 3:44 PM, Barry Smith < bsmith at mcs.anl.gov > wrote: We would need to see the PETSc side of the code to see if there is anything wrong there. > On Apr 5, 2015, at 3:35 PM, James A Charles < charlesj at purdue.edu > wrote: > > Hi Hong, > > You can open up matmatmult_comparison.mat in MATLAB ( load matmatmult_comparison.mat ). That .mat contains A, B, C, and C_MATLAB. I called matmatmult in sequential. We use AIJ when A2 was sparse but at the time of this multiplication everything was dense. > > Thanks, > James > > > ----- Original Message ----- > From: "Hong" < hzhang at mcs.anl.gov > > To: "James A Charles" < charlesj at purdue.edu > > Cc: "PETSc users list" < petsc-users at mcs.anl.gov >, dlemus at purdue.edu , "Daniel Mejia" < denphi at denphi.com >, "Tillmann Kubis" < tkubis at purdue.edu > > Sent: Sunday, April 5, 2015 4:25:39 PM > Subject: Re: [petsc-users] MatMatMult with dense matrices. > > > > > James: > > > I'm multiplying two dense matrices. I compared the results against MATLAB and I am getting different results. Attached you will see a zip that contains a .mat file that has the A, B, and C. C_MATLAB is the multiplication of A*B in MATLAB and C is the PETSc 3.4.3 result. I also attached the .m files A, B, and C that were saved from PETSc. Is there something I'm doing wrong or > > > What is the file type of matmatmult_comparison.mat? > How to read it? > I need to see how it defers from C.m. > > > > any reason due to the shape of the matrices that this would fail? If it gives any more information we convert the previous matrix that A is formed of A2 (A = A1*A2) to dense prior to the multiplication using MatConvert. > > > > It seems both A and B are dense, complex square matrices. > Did you call MatMatMult() in sequential or parallel? What matrix format did you use? > > > Hong From hzhang at mcs.anl.gov Tue Apr 7 14:13:36 2015 From: hzhang at mcs.anl.gov (Hong) Date: Tue, 7 Apr 2015 14:13:36 -0500 Subject: [petsc-users] MatMatMult with dense matrices. In-Reply-To: <1631590191.289162.1428433115075.JavaMail.root@mailhub029.itcs.purdue.edu> References: <1631590191.289162.1428433115075.JavaMail.root@mailhub029.itcs.purdue.edu> Message-ID: James: > > > Thanks a lot for looking into this. I'm still working on debugging this on > our side. It might be an issue with us. I will keep you updated. > Take your time. Hong > > ----- Original Message ----- > From: "Hong" > To: "Barry Smith" > Cc: "James A Charles" , "Hong" , > dlemus at purdue.edu, "PETSc users list" , "Daniel > Mejia" , "Tillmann Kubis" > Sent: Sunday, April 5, 2015 6:13:42 PM > Subject: Re: [petsc-users] MatMatMult with dense matrices. > > > James, > > > My understanding is that > you call MatMatMult in sequential for computing C = A*B, in which > A is in aij format, and B is dense, and the product C is dense, but gives > different values than what you got from Matlab. > > > Do you have a stand-alone petsc code to reproduce this so I can > investigate what goes wrong? > > > Hong > > > > On Sun, Apr 5, 2015 at 3:44 PM, Barry Smith < bsmith at mcs.anl.gov > wrote: > > > > We would need to see the PETSc side of the code to see if there is > anything wrong there. > > > > > On Apr 5, 2015, at 3:35 PM, James A Charles < charlesj at purdue.edu > > wrote: > > > > Hi Hong, > > > > You can open up matmatmult_comparison.mat in MATLAB ( load > matmatmult_comparison.mat ). That .mat contains A, B, C, and C_MATLAB. I > called matmatmult in sequential. We use AIJ when A2 was sparse but at the > time of this multiplication everything was dense. > > > > Thanks, > > James > > > > > > ----- Original Message ----- > > From: "Hong" < hzhang at mcs.anl.gov > > > To: "James A Charles" < charlesj at purdue.edu > > > Cc: "PETSc users list" < petsc-users at mcs.anl.gov >, dlemus at purdue.edu , > "Daniel Mejia" < denphi at denphi.com >, "Tillmann Kubis" < tkubis at purdue.edu > > > > Sent: Sunday, April 5, 2015 4:25:39 PM > > Subject: Re: [petsc-users] MatMatMult with dense matrices. > > > > > > > > > > James: > > > > > > I'm multiplying two dense matrices. I compared the results against > MATLAB and I am getting different results. Attached you will see a zip that > contains a .mat file that has the A, B, and C. C_MATLAB is the > multiplication of A*B in MATLAB and C is the PETSc 3.4.3 result. I also > attached the .m files A, B, and C that were saved from PETSc. Is there > something I'm doing wrong or > > > > > > What is the file type of matmatmult_comparison.mat? > > How to read it? > > I need to see how it defers from C.m. > > > > > > > > any reason due to the shape of the matrices that this would fail? If it > gives any more information we convert the previous matrix that A is formed > of A2 (A = A1*A2) to dense prior to the multiplication using MatConvert. > > > > > > > > It seems both A and B are dense, complex square matrices. > > Did you call MatMatMult() in sequential or parallel? What matrix format > did you use? > > > > > > Hong > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dlemus at purdue.edu Tue Apr 7 14:29:38 2015 From: dlemus at purdue.edu (Daniel A Lemus) Date: Tue, 7 Apr 2015 15:29:38 -0400 (EDT) Subject: [petsc-users] SLEPc function SlepcSortEigenvalues In-Reply-To: <929789701.248786.1428434172891.JavaMail.root@mailhub030.itcs.purdue.edu> Message-ID: <1987819114.248869.1428434978073.JavaMail.root@mailhub030.itcs.purdue.edu> Hi, I'm using SLEPc version 3.5.3 and it seems that EPSSortEigenvalues was replaced with SlepcSortEigenvalues in a recent release. I can find very little documentation on this SlepcSortEigenvalues function and the SlepcSC argument it takes. There is also no mention of these changes in the SLEPc changes list. Are there any examples on how to use this function and how to set the SlepcSC sorting criterion? Thanks, Daniel From jroman at dsic.upv.es Tue Apr 7 14:42:30 2015 From: jroman at dsic.upv.es (Jose E. Roman) Date: Tue, 7 Apr 2015 21:42:30 +0200 Subject: [petsc-users] SLEPc function SlepcSortEigenvalues In-Reply-To: <1987819114.248869.1428434978073.JavaMail.root@mailhub030.itcs.purdue.edu> References: <1987819114.248869.1428434978073.JavaMail.root@mailhub030.itcs.purdue.edu> Message-ID: <20C6083F-A945-4038-BC7C-94D0F63C3567@dsic.upv.es> El 07/04/2015, a las 21:29, Daniel A Lemus escribi?: > Hi, > > I'm using SLEPc version 3.5.3 and it seems that EPSSortEigenvalues was replaced with SlepcSortEigenvalues in a recent release. > > I can find very little documentation on this SlepcSortEigenvalues function and the SlepcSC argument it takes. There is also no mention of these changes in the SLEPc changes list. > > Are there any examples on how to use this function and how to set the SlepcSC sorting criterion? > > Thanks, > Daniel This function is intended for internal use only, not for application code, that is why it is not well documented. Here is the manpage: http://slepc.upv.es/documentation/current/docs/manualpages/sys/SlepcSortEigenvalues.html The 'sorting criterion' argument SlepcSC is used in some of the test examples of the DS auxiliary class, e.g. http://slepc.upv.es/documentation/current/src/sys/classes/ds/examples/tests/test1.c.html Basically it is a struct containing a sorting function and a mapping function, possibly with data contexts. This functionality is not available from Fortran. Jose From mpovolot at purdue.edu Tue Apr 7 14:59:15 2015 From: mpovolot at purdue.edu (Michael Povolotskyi) Date: Tue, 07 Apr 2015 15:59:15 -0400 Subject: [petsc-users] SLEPc function SlepcSortEigenvalues In-Reply-To: <20C6083F-A945-4038-BC7C-94D0F63C3567@dsic.upv.es> References: <1987819114.248869.1428434978073.JavaMail.root@mailhub030.itcs.purdue.edu> <20C6083F-A945-4038-BC7C-94D0F63C3567@dsic.upv.es> Message-ID: <55243713.6020400@purdue.edu> Hi, I have a related question. I need to sort eigenvectors after the eigenvalue problem is solved. For PETSC/SLEPC 3.4 I do the following EPSSetEigenvalueComparison(eps, &my_sort_function, NULL); eps->which = EPS_WHICH_USER; ierr = EPSSortEigenvalues(eps,eps->nconv,eps->eigr,eps->eigi, eps->perm); How should I modify the code for PETSc/SLEPc 3.5? Thank you, Michael. On 04/07/2015 03:42 PM, Jose E. Roman wrote: > El 07/04/2015, a las 21:29, Daniel A Lemus escribi?: > >> Hi, >> >> I'm using SLEPc version 3.5.3 and it seems that EPSSortEigenvalues was replaced with SlepcSortEigenvalues in a recent release. >> >> I can find very little documentation on this SlepcSortEigenvalues function and the SlepcSC argument it takes. There is also no mention of these changes in the SLEPc changes list. >> >> Are there any examples on how to use this function and how to set the SlepcSC sorting criterion? >> >> Thanks, >> Daniel > This function is intended for internal use only, not for application code, that is why it is not well documented. Here is the manpage: > http://slepc.upv.es/documentation/current/docs/manualpages/sys/SlepcSortEigenvalues.html > > The 'sorting criterion' argument SlepcSC is used in some of the test examples of the DS auxiliary class, e.g. > http://slepc.upv.es/documentation/current/src/sys/classes/ds/examples/tests/test1.c.html > > Basically it is a struct containing a sorting function and a mapping function, possibly with data contexts. > > This functionality is not available from Fortran. > > Jose > From bsmith at mcs.anl.gov Tue Apr 7 15:00:30 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 7 Apr 2015 15:00:30 -0500 Subject: [petsc-users] iterative solvers In-Reply-To: References: Message-ID: <5B531653-097B-4CD1-8F3E-2EEC23BFCE68@mcs.anl.gov> You cannot just use algebraic multigrid directly on the transonic Euler flow problem discretized with SUPG you must break the problem into pieces (probably using PCFIELDSPLIT) and then use appropriate preconditioners for each piece. Barry Note that ILU is a bottom feeder preconditioner, it is only working because you are using a huge fill factor (so it is a lot like a direct solver) and even then it is working really poorly. You should google for good preconditioners for transonic Euler flow and SUPG (and ignore anything that mentions ILU) to get a handle on how PCFIELDSPLIT could be used for your problem. > On Apr 7, 2015, at 7:06 AM, Manav Bhatia wrote: > > Hi, > > I am solving a transonic Euler flow problem discretized with SUPG. The mesh is made of Tet4 elements and there are about 7M dofs, which I am trying to solve over 192 cores. > > I had earlier written about the linear solver returning with Inf, and have since moved beyond that such that I am able to get a solution with the following command line parameters: > > -ksp_gmres_restart 100 -pc_type asm -sub_pc_type ilu -sub_pc_factor_levels 4 -sub_ksp_type preonly > > The problem now is that I am limited to very small time-steps (~10e-6), outside of which the solver starts to choke and the solution diverges. I have tried other command line options and this above combination is what seems to provide the best possible solution strategy, albeit constrained in time steps. > > I would appreciate any recommendations on solver configurations that may allow me to take larger time-steps. > > Would coding up AMG be a possible alternative (more robust and dependable)? > > Please let me know if you need additional information. > > Regards, > Manav > From bhatiamanav at gmail.com Tue Apr 7 15:05:15 2015 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Tue, 7 Apr 2015 15:05:15 -0500 Subject: [petsc-users] iterative solvers In-Reply-To: <5B531653-097B-4CD1-8F3E-2EEC23BFCE68@mcs.anl.gov> References: <5B531653-097B-4CD1-8F3E-2EEC23BFCE68@mcs.anl.gov> Message-ID: Thanks Barry. Your message provides a good starting point. I will need to do more reading to understand my options. Thanks, Manav > On Apr 7, 2015, at 3:00 PM, Barry Smith wrote: > > > You cannot just use algebraic multigrid directly on the transonic Euler flow problem discretized with SUPG you must break the problem into pieces (probably using PCFIELDSPLIT) and then use appropriate preconditioners for each piece. > > Barry > > Note that ILU is a bottom feeder preconditioner, it is only working because you are using a huge fill factor (so it is a lot like a direct solver) and even then it is working really poorly. You should google for good preconditioners for transonic Euler flow and SUPG (and ignore anything that mentions ILU) to get a handle on how PCFIELDSPLIT could be used for your problem. > > >> On Apr 7, 2015, at 7:06 AM, Manav Bhatia wrote: >> >> Hi, >> >> I am solving a transonic Euler flow problem discretized with SUPG. The mesh is made of Tet4 elements and there are about 7M dofs, which I am trying to solve over 192 cores. >> >> I had earlier written about the linear solver returning with Inf, and have since moved beyond that such that I am able to get a solution with the following command line parameters: >> >> -ksp_gmres_restart 100 -pc_type asm -sub_pc_type ilu -sub_pc_factor_levels 4 -sub_ksp_type preonly >> >> The problem now is that I am limited to very small time-steps (~10e-6), outside of which the solver starts to choke and the solution diverges. I have tried other command line options and this above combination is what seems to provide the best possible solution strategy, albeit constrained in time steps. >> >> I would appreciate any recommendations on solver configurations that may allow me to take larger time-steps. >> >> Would coding up AMG be a possible alternative (more robust and dependable)? >> >> Please let me know if you need additional information. >> >> Regards, >> Manav >> > From jroman at dsic.upv.es Tue Apr 7 15:32:47 2015 From: jroman at dsic.upv.es (Jose E. Roman) Date: Tue, 7 Apr 2015 22:32:47 +0200 Subject: [petsc-users] SLEPc function SlepcSortEigenvalues In-Reply-To: <55243713.6020400@purdue.edu> References: <1987819114.248869.1428434978073.JavaMail.root@mailhub030.itcs.purdue.edu> <20C6083F-A945-4038-BC7C-94D0F63C3567@dsic.upv.es> <55243713.6020400@purdue.edu> Message-ID: Something like this: #include SlepcSC sc; ierr = PetscNew(&sc);CHKERRQ(ierr); sc->comparison = my_sort_function; sc->comparisonctx = NULL; sc->map = NULL; sc->mapobj = NULL; ierr = SlepcSortEigenvalues(sc,eps->nconv,eps->eigr,eps->eigi,eps->perm);CHKERRQ(ierr); where the signature of my_sort_function should be: PetscErrorCode (*comparison)(PetscScalar,PetscScalar,PetscScalar,PetscScalar,PetscInt*,void*); Jose El 07/04/2015, a las 21:59, Michael Povolotskyi escribi?: > Hi, I have a related question. > I need to sort eigenvectors after the eigenvalue problem is solved. > For PETSC/SLEPC 3.4 I do the following > > EPSSetEigenvalueComparison(eps, &my_sort_function, NULL); > eps->which = EPS_WHICH_USER; > ierr = EPSSortEigenvalues(eps,eps->nconv,eps->eigr,eps->eigi, eps->perm); > > > How should I modify the code for PETSc/SLEPc 3.5? > Thank you, > Michael. > > On 04/07/2015 03:42 PM, Jose E. Roman wrote: >> El 07/04/2015, a las 21:29, Daniel A Lemus escribi?: >> >>> Hi, >>> >>> I'm using SLEPc version 3.5.3 and it seems that EPSSortEigenvalues was replaced with SlepcSortEigenvalues in a recent release. >>> >>> I can find very little documentation on this SlepcSortEigenvalues function and the SlepcSC argument it takes. There is also no mention of these changes in the SLEPc changes list. >>> >>> Are there any examples on how to use this function and how to set the SlepcSC sorting criterion? >>> >>> Thanks, >>> Daniel >> This function is intended for internal use only, not for application code, that is why it is not well documented. Here is the manpage: >> http://slepc.upv.es/documentation/current/docs/manualpages/sys/SlepcSortEigenvalues.html >> >> The 'sorting criterion' argument SlepcSC is used in some of the test examples of the DS auxiliary class, e.g. >> http://slepc.upv.es/documentation/current/src/sys/classes/ds/examples/tests/test1.c.html >> >> Basically it is a struct containing a sorting function and a mapping function, possibly with data contexts. >> >> This functionality is not available from Fortran. >> >> Jose >> > From bsmith at mcs.anl.gov Tue Apr 7 15:51:43 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 7 Apr 2015 15:51:43 -0500 Subject: [petsc-users] Problem with diagonal MPIAIJ matrix In-Reply-To: References: Message-ID: <83E7EF06-4320-4D51-ABFB-0E31A7D29547@mcs.anl.gov> I think this problem (or a related problem) was fixed in master recently. Could you please try with the master branch and see if you still have a problem? http://www.mcs.anl.gov/petsc/developers/index.html Barry > On Apr 7, 2015, at 6:38 AM, Lukasz Kaczmarczyk wrote: > > Hello, > > Pleas see attached following code, > http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/_p_c_m_g_set_up_via_approx_orders_8cpp_source.html > > I encounter following error, if restriction and interpolation operator is the same for multigrid solver. This happens when I use hierarchical approx. basis and my R matrix is has only values on diagonal. If I set PCMGSetRestriction it works great, but I believe that this is not optimal solution. Could you advise some solution for this? > > > 176 // Petsc Development GIT revision: v3.5.3-1524-gee900cc GIT Date: 2015-01-31 17:44:15 -0600 > 178 // [0]PETSC ERROR: [0] MatPtAPSymbolic_MPIAIJ_MPIAIJ line 124 /opt/petsc/src/mat/impls/aij/mpi/mpiptap.c > 179 // [0]PETSC ERROR: [0] MatPtAP_MPIAIJ_MPIAIJ line 80 /opt/petsc/src/mat/impls/aij/mpi/mpiptap.c > 180 // [0]PETSC ERROR: [0] MatPtAP line 8458 /opt/petsc/src/mat/interface/matrix.c > 181 // [0]PETSC ERROR: [0] PCSetUp_MG line 552 /opt/petsc/src/ksp/pc/impls/mg/mg.c > 182 // [0]PETSC ERROR: [0] KSPSetUp line 220 /opt/petsc/src/ksp/ksp/interface/itfunc.c > 183 > 184 // ==4284== Invalid read of size 8 > 185 // ==4284== at 0x5CAC873: MatPtAPSymbolic_MPIAIJ_MPIAIJ (mpiptap.c:154) > 186 // ==4284== by 0x5CABA74: MatPtAP_MPIAIJ_MPIAIJ (mpiptap.c:83) > 187 // ==4284== by 0x5D566D1: MatPtAP (matrix.c:8537) > 188 // ==4284== by 0x61F27D6: PCSetUp_MG (mg.c:642) > 189 // ==4284== by 0x612BD9C: PCSetUp (precon.c:909) > 190 // ==4284== by 0x62C1A51: KSPSetUp (itfunc.c:306) > 191 // ==4284== by 0xB98326: main (elasticity.cpp:403) > 192 // ==4284== Address 0x208 is not stack'd, malloc'd or (recently) free'd > > > Kind regards, > Lukasz From jchang27 at uh.edu Tue Apr 7 19:18:04 2015 From: jchang27 at uh.edu (Justin Chang) Date: Tue, 7 Apr 2015 19:18:04 -0500 Subject: [petsc-users] Uninterpolating DMLabels? Message-ID: Hi all, So I am interpolating/uninterpolating my DMPlex mesh. This is the mesh information for a cell-vertex mesh only: DM Object: 1 MPI processes type: plex DM_0x84000000_0 in 3 dimensions: 0-cells: 159 3-cells: 592 Labels: marker: 1 strata of sizes (100) depth: 2 strata of sizes (159, 592) I am interpolating the mesh with the following lines: ierr = DMPlexInterpolate(*dm, &idm);CHKERRQ(ierr); ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); ierr = DMPlexGetLabel(idm, "marker", &label);CHKERRQ(ierr); ierr = DMPlexMarkBoundaryFaces(idm,label);CHKERRQ(ierr); ierr = DMDestroy(dm);CHKERRQ(ierr); *dm = idm; And the resulting mesh: DM Object: 1 MPI processes type: plex DM_0x84000000_1 in 3 dimensions: 0-cells: 159 1-cells: 845 2-cells: 1280 3-cells: 592 Labels: marker: 1 strata of sizes (292) depth: 4 strata of sizes (159, 845, 1280, 592) The reason I did this is so that incase I decide to refine the mesh with "-dm_refine " the DM will be sure to include the "marker" points in the refinement process. Now, if I decide to uninterpolate the mesh with the following lines: ierr = DMPlexUninterpolate(*dm, &idm);CHKERRQ(ierr); ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); ierr = DMDestroy(dm);CHKERRQ(ierr); *dm = idm; I end up with this mesh: DM Object: 1 MPI processes type: plex DM_0x84000000_2 in 3 dimensions: 0-cells: 159 3-cells: 592 Labels: marker: 1 strata of sizes (292) depth: 2 strata of sizes (159, 592) The problem is that although the cells and vertices have gone back to the original number, the "marker" label still includes face/edge points. This gives me an error once I invoke DMCreateGobalVector(...). [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Argument out of range [0]PETSC ERROR: Section point 755 should be in [0, 751) [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2528-gbee642f GIT Date: 2015-03-29 20:36:38 -0500 [0]PETSC ERROR: ./cube_with_hole on a arch-linux2-c-debug named pacotaco by justin Tue Apr 7 19:09:12 2015 [0]PETSC ERROR: Configure options --download-chaco --download-exodusii --download-fblaslapack --download-hdf5 --download-metis --download-mpich --download-mumps --download-netcdf --download-parmetis --download-scalapack --download-triangle --with-cc=gcc --with-cmake=cmake --with-cxx=g++ --with-debugging=1 --with-fc=gfortran --with-valgrind=1 PETSC_ARCH=arch-linux2-c-debug [0]PETSC ERROR: #1 PetscSectionGetDof() line 499 in /home/justin/petsc-dev/src/vec/is/utils/vsectionis.c [0]PETSC ERROR: #2 DMPlexGetConeSize() line 841 in /home/justin/petsc-dev/src/dm/impls/plex/plex.c [0]PETSC ERROR: #3 DMPlexGetTransitiveClosure() line 1342 in /home/justin/petsc-dev/src/dm/impls/plex/plex.c [0]PETSC ERROR: #4 DMPlexLabelComplete() line 88 in /home/justin/petsc-dev/src/dm/impls/plex/plexsubmesh.c [0]PETSC ERROR: #5 DMCreateDefaultSection_Plex() line 5605 in /home/justin/petsc-dev/src/dm/impls/plex/plex.c [0]PETSC ERROR: #6 DMGetDefaultSection() line 3035 in /home/justin/petsc-dev/src/dm/interface/dm.c [0]PETSC ERROR: #7 DMGetDefaultGlobalSection() line 3266 in /home/justin/petsc-dev/src/dm/interface/dm.c [0]PETSC ERROR: #8 DMCreateGlobalVector_Section_Private() line 13 in /home/justin/petsc-dev/src/dm/interface/dmi.c [0]PETSC ERROR: #9 DMCreateGlobalVector_Plex() line 1170 in /home/justin/petsc-dev/src/dm/impls/plex/plexcreate.c [0]PETSC ERROR: #10 DMCreateGlobalVector() line 698 in /home/justin/petsc-dev/src/dm/interface/dm.c [0]PETSC ERROR: #11 main() line 363 in /home/justin/Dropbox/Research_Topics/Code_PETSc/Nonneg/cube_with_hole.c >From this error, it seems it's trying to access sieve points that non longer exist. And I think it has to do with the label "marker" still contains data from the interpolated mesh. Is there a way to "uninterpolate" or remove such points? Or is there a better way of approaching this whole thing? Thanks, -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Apr 7 20:37:25 2015 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 7 Apr 2015 20:37:25 -0500 Subject: [petsc-users] Uninterpolating DMLabels? In-Reply-To: References: Message-ID: On Tue, Apr 7, 2015 at 7:18 PM, Justin Chang wrote: > Hi all, > > So I am interpolating/uninterpolating my DMPlex mesh. This is the mesh > information for a cell-vertex mesh only: > > DM Object: 1 MPI processes > type: plex > DM_0x84000000_0 in 3 dimensions: > 0-cells: 159 > 3-cells: 592 > Labels: > marker: 1 strata of sizes (100) > depth: 2 strata of sizes (159, 592) > > I am interpolating the mesh with the following lines: > > ierr = DMPlexInterpolate(*dm, &idm);CHKERRQ(ierr); > ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); > ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); > ierr = DMPlexGetLabel(idm, "marker", &label);CHKERRQ(ierr); > ierr = DMPlexMarkBoundaryFaces(idm,label);CHKERRQ(ierr); > ierr = DMDestroy(dm);CHKERRQ(ierr); > *dm = idm; > > And the resulting mesh: > > DM Object: 1 MPI processes > type: plex > DM_0x84000000_1 in 3 dimensions: > 0-cells: 159 > 1-cells: 845 > 2-cells: 1280 > 3-cells: 592 > Labels: > marker: 1 strata of sizes (292) > depth: 4 strata of sizes (159, 845, 1280, 592) > > The reason I did this is so that incase I decide to refine the mesh with > "-dm_refine " the DM will be sure to include the "marker" points in > the refinement process. Now, if I decide to uninterpolate the mesh with the > following lines: > > ierr = DMPlexUninterpolate(*dm, &idm);CHKERRQ(ierr); > ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); > ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); > ierr = DMDestroy(dm);CHKERRQ(ierr); > *dm = idm; > > I end up with this mesh: > > DM Object: 1 MPI processes > type: plex > DM_0x84000000_2 in 3 dimensions: > 0-cells: 159 > 3-cells: 592 > Labels: > marker: 1 strata of sizes (292) > depth: 2 strata of sizes (159, 592) > > The problem is that although the cells and vertices have gone back to the > original number, the "marker" label still includes face/edge points. This > gives me an error once I invoke DMCreateGobalVector(...). > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Argument out of range > [0]PETSC ERROR: Section point 755 should be in [0, 751) > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2528-gbee642f GIT > Date: 2015-03-29 20:36:38 -0500 > [0]PETSC ERROR: ./cube_with_hole on a arch-linux2-c-debug named pacotaco > by justin Tue Apr 7 19:09:12 2015 > [0]PETSC ERROR: Configure options --download-chaco --download-exodusii > --download-fblaslapack --download-hdf5 --download-metis --download-mpich > --download-mumps --download-netcdf --download-parmetis --download-scalapack > --download-triangle --with-cc=gcc --with-cmake=cmake --with-cxx=g++ > --with-debugging=1 --with-fc=gfortran --with-valgrind=1 > PETSC_ARCH=arch-linux2-c-debug > [0]PETSC ERROR: #1 PetscSectionGetDof() line 499 in > /home/justin/petsc-dev/src/vec/is/utils/vsectionis.c > [0]PETSC ERROR: #2 DMPlexGetConeSize() line 841 in > /home/justin/petsc-dev/src/dm/impls/plex/plex.c > [0]PETSC ERROR: #3 DMPlexGetTransitiveClosure() line 1342 in > /home/justin/petsc-dev/src/dm/impls/plex/plex.c > [0]PETSC ERROR: #4 DMPlexLabelComplete() line 88 in > /home/justin/petsc-dev/src/dm/impls/plex/plexsubmesh.c > [0]PETSC ERROR: #5 DMCreateDefaultSection_Plex() line 5605 in > /home/justin/petsc-dev/src/dm/impls/plex/plex.c > [0]PETSC ERROR: #6 DMGetDefaultSection() line 3035 in > /home/justin/petsc-dev/src/dm/interface/dm.c > [0]PETSC ERROR: #7 DMGetDefaultGlobalSection() line 3266 in > /home/justin/petsc-dev/src/dm/interface/dm.c > [0]PETSC ERROR: #8 DMCreateGlobalVector_Section_Private() line 13 in > /home/justin/petsc-dev/src/dm/interface/dmi.c > [0]PETSC ERROR: #9 DMCreateGlobalVector_Plex() line 1170 in > /home/justin/petsc-dev/src/dm/impls/plex/plexcreate.c > [0]PETSC ERROR: #10 DMCreateGlobalVector() line 698 in > /home/justin/petsc-dev/src/dm/interface/dm.c > [0]PETSC ERROR: #11 main() line 363 in > /home/justin/Dropbox/Research_Topics/Code_PETSc/Nonneg/cube_with_hole.c > > > From this error, it seems it's trying to access sieve points that non > longer exist. And I think it has to do with the label "marker" still > contains data from the interpolated mesh. Is there a way to "uninterpolate" > or remove such points? Or is there a better way of approaching this whole > thing? > 1) It would be easy to write a small function to throw out the points in the label that do not exist in the DM. You would just extract each stratumIS and then Clear each invalid point. 2) However, I do not understand the rationale for this above. You could just call MarkBoundaryFaces on the final mesh. If you are really trying to preserve a label across regular refinement AND you really do not want an interpolated mesh, then code up 1). I have yet to see a case where the extra memory for edges and faces makes a difference, but it may exist. Thanks, Matt > Thanks, > > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From siddhesh4godbole at gmail.com Tue Apr 7 23:40:23 2015 From: siddhesh4godbole at gmail.com (siddhesh godbole) Date: Wed, 8 Apr 2015 10:10:23 +0530 Subject: [petsc-users] eps eigenvectors error Message-ID: hello i am trying to retrieve the solution after EPSSolve but i am unable to get the eigenvectors when i call EPSGetEigenpair or EPSGetEigenvector i am getting following error [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Null argument, when expecting valid pointer [0]PETSC ERROR: Null Object: Parameter # 3 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 [0]PETSC ERROR: ./eps13D on a arch-linux2-c-debug named iitm by iitm Wed Apr 8 10:04:57 2015 [0]PETSC ERROR: Configure options --download-mpich --download-f2cblaslapack=1 [0]PETSC ERROR: #1 EPSGetEigenvector() line 488 in /home/iitm/Downloads/slepc-3.5.2/src/eps/interface/epssolve.c [0]PETSC ERROR: #2 main() line 266 in /home/iitm/Desktop/project/eps13D.c [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- what could be the reason? sincerely *Siddhesh M Godbole* 5th year Dual Degree, Civil Eng & Applied Mech. IIT Madras -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Wed Apr 8 01:43:42 2015 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 8 Apr 2015 08:43:42 +0200 Subject: [petsc-users] eps eigenvectors error In-Reply-To: References: Message-ID: <24FB565C-F461-437C-B9C1-6BD253C8E446@dsic.upv.es> El 08/04/2015, a las 06:40, siddhesh godbole escribi?: > hello > > i am trying to retrieve the solution after EPSSolve but i am unable to get the eigenvectors when i call EPSGetEigenpair or EPSGetEigenvector > > i am getting following error > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Null argument, when expecting valid pointer > [0]PETSC ERROR: Null Object: Parameter # 3 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 > [0]PETSC ERROR: ./eps13D on a arch-linux2-c-debug named iitm by iitm Wed Apr 8 10:04:57 2015 > [0]PETSC ERROR: Configure options --download-mpich --download-f2cblaslapack=1 > [0]PETSC ERROR: #1 EPSGetEigenvector() line 488 in /home/iitm/Downloads/slepc-3.5.2/src/eps/interface/epssolve.c > [0]PETSC ERROR: #2 main() line 266 in /home/iitm/Desktop/project/eps13D.c > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- > > > what could be the reason? > EPSGetEigenvector() needs to receive one (or two) valid vector object, where it will place the requested eigenvector. In other words, you must create the vector yourself and pass it. See for instance ex7.c Jose From kaiser at igpm.rwth-aachen.de Wed Apr 8 01:46:40 2015 From: kaiser at igpm.rwth-aachen.de (Klaus Kaiser) Date: Wed, 08 Apr 2015 08:46:40 +0200 Subject: [petsc-users] Problem with MPI, MatAXPY and SAME_NONZERO_PATTERN In-Reply-To: References: <55156DA7.6010602@igpm.rwth-aachen.de> <55157460.4080200@igpm.rwth-aachen.de> <5515805B.3060406@igpm.rwth-aachen.de> <551922FF.2020802@igpm.rwth-aachen.de> Message-ID: <5524CED0.7060805@igpm.rwth-aachen.de> Hallo Metthew, you were albe to reconstruct this problem on your petsc installation? Best Klaus On 03/30/2015 01:43 PM, Matthew Knepley wrote: > On Mon, Mar 30, 2015 at 5:18 AM, Klaus Kaiser > > wrote: > > Hallo Metthew, > > at the weekend I updated petsc to the newest release version 3.5.2 > and I still got this strange behavior. > > > Okay, I will check it out. The first thing I am going to do is try > with AIJ matrices instead of BAIJ, which > you can probably do before me. If you get the right result, then its a > bug in the BAIJ implementation, > which I am confident has not been tested as thoroughly. > > Thanks, > > Matt > > Best > > Klaus > > > On 03/27/2015 05:07 PM, Klaus Kaiser wrote: >> Hallo Matthew, >> >> here is a short example I used for 8 cores: >> >> int d_nnz[3]; >> int o_nnz[3]; >> d_nnz[0] = 3; d_nnz[1] = 3; d_nnz[2] = 3; >> o_nnz[0] = 6; o_nnz[1] = 6; o_nnz[2] = 6; >> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, >> d_nnz, 0, o_nnz,&A); >> ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); >> ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); >> ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); >> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, >> d_nnz, 0, o_nnz,&Ah); >> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, >> d_nnz, 0, o_nnz,&At); >> >> std::vector insert(3*3*3*3, 1.0); >> for(int i=0;i<8;++i) >> { >> int rows[3] = {i,i+1,i+3}; >> int cols[3] = {i,i+1,i+3}; >> >> MatSetValuesBlocked(A, 3, rows, 3, cols, &insert[0], >> ADD_VALUES); >> } >> >> MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); >> MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); >> MatDuplicate(A,MAT_COPY_VALUES,&Ah); >> MatDuplicate(A,MAT_COPY_VALUES,&At); >> MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY); >> MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY); >> MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY); >> MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY); >> >> MatAXPY(Ah,1.,At,SAME_NONZERO_PATTERN); >> MatAXPY(A,1.,Ah,SAME_NONZERO_PATTERN); >> >> MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); >> MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); >> >> The result is, that only some values are computed correctly. >> >> Best and Thank you a lot >> >> Klaus >> >> >> >> On 03/27/2015 04:26 PM, Matthew Knepley wrote: >>> On Fri, Mar 27, 2015 at 10:16 AM, Klaus Kaiser >>> > >>> wrote: >>> >>> Hallo Matthew, >>> >>> thanks for your fast response. With "no beside the nonzero >>> structure" I meant, that I do not create a different >>> non-zero structure while I'm adding values to my matrices. >>> >>> I also tried MAT_COPY_VALUES and the result was the same. >>> >>> >>> Please send the small example and we will figure out what is >>> going wrong. >>> >>> Thanks, >>> >>> Matt >>> >>> I do not get a error message but the resulting matrix is >>> wrong. Here I have a short example. The first 9 rows and >>> first 3 colomn of the matrices Ah, At and Ah+At on the first >>> processor (of 8): >>> >>> Ah: >>> 0: 60.3553 -0.249975 2.77556e-17 >>> 1: 0 60.3553 0 >>> 2: 0 0 60.3553 >>> 3: 17.6777 0.374962 0.124987 >>> 4: 0 17.6777 0 >>> 5: 0 0 17.6777 >>> 6: -7.32233 -0.124987 0.374962 >>> 7: 0 -7.32233 0 >>> 8: 0 0 -7.32233 >>> >>> >>> At: >>> 0: 0 0 0 >>> 1: 2500 0 0 >>> 2: -4.54747e-13 0 0 >>> 3: 0 0 0 >>> 4: 1250 0 0 >>> 5: 1250 0 0 >>> 6: 0 0 0 >>> 7: 1250 0 0 >>> 8: 3750 0 0 >>> >>> Ah+At >>> 0: 60.3553 -0.249975 2.77556e-17 >>> 1: 2500 60.3553 0 >>> 2: -4.54747e-13 0 60.3553 >>> 3: 17.6777 0.374962 0.124987 >>> 4: 0 17.6777 0 >>> 5: 0 0 17.6777 >>> 6: -7.32233 -0.124987 0.374962 >>> 7: 0 -7.32233 0 >>> 8: 0 0 -7.32233 >>> >>> you can see the first 3 rows of the resulting matrix looks >>> exactly like what I would expect, but the last 6 rows only >>> consists of the values of Ah. When you would also look on >>> the matrix A and A+Ah+At you would also see, that values >>> where both matrices have an nonzero entry are not sum correctly. >>> >>> Best >>> >>> Klaus >>> >>> On 03/27/2015 03:59 PM, Matthew Knepley wrote: >>>> On Fri, Mar 27, 2015 at 9:48 AM, Klaus Kaiser >>>> >>> > wrote: >>>> >>>> Hallo, >>>> >>>> I have a strange behavior in my code concerning the >>>> function MatAXPY. I create 3 different Matrices >>>> >>>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, >>>> local_size, local_size, system_size, system_size, 0, >>>> d_nnz, 0, o_nnz,&A); >>>> ierr = >>>> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); >>>> ierr = >>>> MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); >>>> ierr = >>>> MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); >>>> >>>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, >>>> local_size, local_size, system_size, system_size, 0, >>>> d_nnz, 0, o_nnz,&At); >>>> ierr = >>>> MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); >>>> ierr = >>>> MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); >>>> ierr = >>>> MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); >>>> >>>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, >>>> local_size, local_size, system_size, system_size, 0, >>>> d_nnz, 0, o_nnz,&Ah); >>>> ierr = >>>> MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); >>>> ierr = >>>> MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); >>>> ierr = >>>> MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); >>>> >>>> >>>> These creations are superfluous since you use >>>> MatDuplicate() below. >>>> >>>> and want to sum these three matrixes with different >>>> factors. First I fill the Matrix A with some values, >>>> and duplicate the structure of A to the other two matrices: >>>> >>>> MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); >>>> MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); >>>> MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah); >>>> MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At); >>>> MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY); >>>> MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY); >>>> MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY); >>>> MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY); >>>> >>>> >>>> After this I fill the matrices At and Ah with some >>>> other values, which are not beside the non zero >>>> structure (I also tried with just copying the Matrix >>>> A). Now after another MatAssembly >>>> >>>> >>>> I do not understand "no beside the nonzero structure". Do >>>> you mean that the nonzero structure is the same? >>>> >>>> Can you first test with MAT_COPY_VALUES? >>>> >>>> I want to add these Matrices in the form A+c*(Ah+d*At): >>>> >>>> >>>> MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN); >>>> MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN); >>>> >>>> When I run the method with mpi and one core everything >>>> works fine. Starting the same method with more cores, >>>> the sum of the matrices fails. It seems like some >>>> values are added >>>> >>>> >>>> Please send the full output of a failure when you use >>>> MAT_COPY_VALUES on 2 procs. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> correctly and many values are missed. Using >>>> DIFFERENT_NONZERO_STRUCTURE leads to the right behavior >>>> in the multi-core case, but is very slow. I checked >>>> with a viewer if all matrices have the same nonzero >>>> structure and this is the case. >>>> >>>> Does anyone know why this fails, or do I have made any >>>> wrong thoughts? >>>> >>>> I'm corrently working with a petsc version (Petsc >>>> Release Version 3.3.0, Patch 5, Sat Dec 1 15:10:41 CST >>>> 2012), I looked into the changelogs up to the current >>>> version and did not find any note about MatAXPY or MatAYPX. >>>> >>>> >>>> Best and Thanks a lot for your help >>>> >>>> Klaus >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin >>>> their experiments is infinitely more interesting than any >>>> results to which their experiments lead. >>>> -- Norbert Wiener >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to >>> which their experiments lead. >>> -- Norbert Wiener >> > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliver.browne at upm.es Tue Apr 7 10:35:34 2015 From: oliver.browne at upm.es (Oliver Browne) Date: Tue, 07 Apr 2015 17:35:34 +0200 Subject: [petsc-users] MPI stdin Message-ID: <0fb69621f4c0737aaf2f73cb7c2bbd58@upm.es> Hi, Does MPICH have an option similar to the below (from open_MPI) -stdin|--stdin Specify procs to receive stdin [rank, all, none] (default: 0, indicating rank 0) Thanks, Ollie From elbueler at alaska.edu Wed Apr 8 11:08:48 2015 From: elbueler at alaska.edu (Ed Bueler) Date: Wed, 8 Apr 2015 10:08:48 -0600 Subject: [petsc-users] how to catch zero pivot errors underneath SNES Message-ID: Dear PETSc -- Context: I am solving a not very smooth and very nonlinear problem using SNES. Robustness is an issue but I think there is no theory to help me, and I am stuck mollifying the problem a bit when divergence happens. Thus I deal with the returned divergence reason (i.e. SNESConvergedReason) by making the problem slightly easier. This deals with DIVERGED_LINE_SEARCH and DIVERGED_LINEAR_SOLVE the way I want, for example. The problem is that sometimes zero pivots occur, either in default (GMRES+ILU or GMRES+BJACOBI+sub_ILU) or in domain decomposition (GMRES+ASM+sub_LU). I want to catch and deal with these zero pivots in the same way, ideally. Is there a way to make zero pivot propagate through to generate a new negative SNESConvergedReason? Note that DIVERGED_LINEAR_SOLVE would tell me enough; it seems to me a zero pivot during application of the preconditioner *is* a DIVERGED_LINEAR_SOLVE error anyway. If not, can I at least catch the zero pivot error so it is not an error (i.e. so the program continues), without ignoring all errors? I think I don't want to push PetscIgnoreErrorHandler, because I am happy with how other errors are handled. Thanks, Ed -- Ed Bueler Dept of Math and Stat and Geophysical Institute University of Alaska Fairbanks Fairbanks, AK 99775-6660 301C Chapman and 410D Elvey 907 474-7693 and 907 474-7199 (fax 907 474-5394) -------------- next part -------------- An HTML attachment was scrubbed... URL: From elbueler at alaska.edu Wed Apr 8 12:26:02 2015 From: elbueler at alaska.edu (Ed Bueler) Date: Wed, 8 Apr 2015 11:26:02 -0600 Subject: [petsc-users] how to catch zero pivot errors underneath SNES In-Reply-To: References: Message-ID: PS: I *did* indeed write a kludge for this, namely catching the ierr output from "ierr = SNESSolve(...)", and then if it equals 71 convert it to a new negative value of SNESConvergedReason and adding a new string "DIVERGED_LU_ZERO_PIVOT", because I have other scripts that diagnose parameter study results and whether they diverged. But the kludge still spews traceback at me. To eliminate that I suppose I'd need to also wrap the default petsc error handler so that it is silent on 71. So I wonder if a SNES API addition could be helpful here if my usage case is common? Ed On Wed, Apr 8, 2015 at 10:08 AM, Ed Bueler wrote: > Dear PETSc -- > > Context: I am solving a not very smooth and very nonlinear problem using > SNES. Robustness is an issue but I think there is no theory to help me, > and I am stuck mollifying the problem a bit when divergence happens. Thus I > deal with the returned divergence reason (i.e. SNESConvergedReason) by > making the problem slightly easier. This deals with DIVERGED_LINE_SEARCH > and DIVERGED_LINEAR_SOLVE the way I want, for example. > > The problem is that sometimes zero pivots occur, either in default > (GMRES+ILU or GMRES+BJACOBI+sub_ILU) or in domain decomposition > (GMRES+ASM+sub_LU). > > I want to catch and deal with these zero pivots in the same way, ideally. > Is there a way to make zero pivot propagate through to generate a new > negative SNESConvergedReason? Note that DIVERGED_LINEAR_SOLVE would tell > me enough; it seems to me a zero pivot during application of the > preconditioner *is* a DIVERGED_LINEAR_SOLVE error anyway. > > If not, can I at least catch the zero pivot error so it is not an error > (i.e. so the program continues), without ignoring all errors? I think I > don't want to push PetscIgnoreErrorHandler, because I am happy with how > other errors are handled. > > Thanks, > > Ed > > -- > Ed Bueler > Dept of Math and Stat and Geophysical Institute > University of Alaska Fairbanks > Fairbanks, AK 99775-6660 > 301C Chapman and 410D Elvey > 907 474-7693 and 907 474-7199 (fax 907 474-5394) > -- Ed Bueler Dept of Math and Stat and Geophysical Institute University of Alaska Fairbanks Fairbanks, AK 99775-6660 301C Chapman and 410D Elvey 907 474-7693 and 907 474-7199 (fax 907 474-5394) -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Apr 8 13:21:38 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 8 Apr 2015 13:21:38 -0500 Subject: [petsc-users] how to catch zero pivot errors underneath SNES In-Reply-To: References: Message-ID: <73CC6C00-91FA-4090-BE78-64057EB8DB18@mcs.anl.gov> Ed, This is something that requires some code reorganization. We've always treated the "inability to construct a preconditioner" as a hard error generating a SETERRQ() regardless of whether the problem came from a numerical issue (such as zero pivot) or bad user input etc. We need to change the code to allow treating it as a soft error that can be recovered from but this means being able to return the information about a failed factorization up from the matrix implementation level to the matrix interface level to the PC and then to the KSP. It will take some thought on how to do this well. Can you add this as an issue on the bitbucket site. Barry I tried at one point to generalize the PETSc error management to allow recovering from errors, much like exceptions in C++, but decided it was too difficult to organize properly in C so we currently have the model that the error management is only for hard errors and soft errors need to have regular code paths to handle them. Unfortunately adding the needed code paths for some things like zero pivots requires a good amount of thought and new coding. Another example where lots of plumbing needs to be put in is software recovery from cases where user function evaluations generate domain errors in, for example, a matrix-free matrix-vector product. > On Apr 8, 2015, at 11:08 AM, Ed Bueler wrote: > > Dear PETSc -- > > Context: I am solving a not very smooth and very nonlinear problem using SNES. Robustness is an issue but I think there is no theory to help me, and I am stuck mollifying the problem a bit when divergence happens. Thus I deal with the returned divergence reason (i.e. SNESConvergedReason) by making the problem slightly easier. This deals with DIVERGED_LINE_SEARCH and DIVERGED_LINEAR_SOLVE the way I want, for example. > > The problem is that sometimes zero pivots occur, either in default (GMRES+ILU or GMRES+BJACOBI+sub_ILU) or in domain decomposition (GMRES+ASM+sub_LU). > > I want to catch and deal with these zero pivots in the same way, ideally. Is there a way to make zero pivot propagate through to generate a new negative SNESConvergedReason? Note that DIVERGED_LINEAR_SOLVE would tell me enough; it seems to me a zero pivot during application of the preconditioner *is* a DIVERGED_LINEAR_SOLVE error anyway. > > If not, can I at least catch the zero pivot error so it is not an error (i.e. so the program continues), without ignoring all errors? I think I don't want to push PetscIgnoreErrorHandler, because I am happy with how other errors are handled. > > Thanks, > > Ed > > -- > Ed Bueler > Dept of Math and Stat and Geophysical Institute > University of Alaska Fairbanks > Fairbanks, AK 99775-6660 > 301C Chapman and 410D Elvey > 907 474-7693 and 907 474-7199 (fax 907 474-5394) From baronlaw at purdue.edu Wed Apr 8 13:39:59 2015 From: baronlaw at purdue.edu (Baron Law) Date: Wed, 8 Apr 2015 14:39:59 -0400 Subject: [petsc-users] Reuse Matrix Message-ID: <001501d0722b$6b29b500$417d1f00$@purdue.edu> Hi, I need to compute the solution of least square regression for many times (> 10^6) for large matrix A (>10^9x10^3). I use a hypercube basis, so each row of the matrix A has exactly one non-zero entry with value 1. On different runs, the positions of the 1 are different. I would like to reuse the sparse structure of A, what is the best way to do in this scenario? I have heard of MatCreateMPIAIJWithSplitArrays, but it is kind of difficult to use. I try it a bit and seems like after creation if I need to change the column index, I need to skip the diagonal block. Also, I am not 100% sure, how to find which columns belong to the diagonal block for each process. But as mentioned, I need to run the regression million times, even it is difficult to use, I will try if it is fastest method. Best, Baron -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Apr 8 14:22:40 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 8 Apr 2015 14:22:40 -0500 Subject: [petsc-users] Reuse Matrix In-Reply-To: <001501d0722b$6b29b500$417d1f00$@purdue.edu> References: <001501d0722b$6b29b500$417d1f00$@purdue.edu> Message-ID: <07471D86-9D3B-4B07-BA1B-4948BBCB068B@mcs.anl.gov> Baron, This is tricky. when the 1 moves between the "diagonal part" and the "off-diagonal" part it will trigger a new allocation and thus slow things down a great deal. I think the traditional PETSc matrix formats are not good for your situation. I think it is likely much better to use a MATSHELL for this matrix. What do you use the matrix for? You do matrix-vector product with it, anything else? You can write your own matrix vector product with MATSHELL customized for your problem. Barry I cheat would be to have a single nonzero entry always in the diagonal part AND the off-diagonal part of each row and just set one or the other to 0.0 each time so you always use the exact same matrix layout. But this will induce unneeded communication since PETSc will need to set up the scatter for that off diagonal entry even though it is zero. > On Apr 8, 2015, at 1:39 PM, Baron Law wrote: > > Hi, > > I need to compute the solution of least square regression for many times (> 10^6) for large matrix A (>10^9x10^3). I use a hypercube basis, so each row of the matrix A has exactly one non-zero entry with value 1. On different runs, the positions of the 1 are different. I would like to reuse the sparse structure of A, what is the best way to do in this scenario? > > I have heard of MatCreateMPIAIJWithSplitArrays, but it is kind of difficult to use. I try it a bit and seems like after creation if I need to change the column index, I need to skip the diagonal block. Also, I am not 100% sure, how to find which columns belong to the diagonal block for each process. But as mentioned, I need to run the regression million times, even it is difficult to use, I will try if it is fastest method. > > Best, > Baron From baronlaw at purdue.edu Wed Apr 8 14:29:43 2015 From: baronlaw at purdue.edu (Baron Law) Date: Wed, 8 Apr 2015 15:29:43 -0400 Subject: [petsc-users] Reuse Matrix In-Reply-To: <07471D86-9D3B-4B07-BA1B-4948BBCB068B@mcs.anl.gov> References: <001501d0722b$6b29b500$417d1f00$@purdue.edu> <07471D86-9D3B-4B07-BA1B-4948BBCB068B@mcs.anl.gov> Message-ID: <002201d07232$5f1705e0$1d4511a0$@purdue.edu> Thanks Barry, I originally plan to use the matrix for KSP with LSQR solver. Now what I try to do is implement my own sparse matrix using Vec as I just need to store the column index. Instead of using KSP, I try to implement my own regression solver as the result beta[i], is simply the mean of all RHS y where the non-zero column = i. Best, Baron -----Original Message----- From: Barry Smith [mailto:bsmith at mcs.anl.gov] Sent: Wednesday, April 08, 2015 3:23 PM To: Baron Law Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Reuse Matrix Baron, This is tricky. when the 1 moves between the "diagonal part" and the "off-diagonal" part it will trigger a new allocation and thus slow things down a great deal. I think the traditional PETSc matrix formats are not good for your situation. I think it is likely much better to use a MATSHELL for this matrix. What do you use the matrix for? You do matrix-vector product with it, anything else? You can write your own matrix vector product with MATSHELL customized for your problem. Barry I cheat would be to have a single nonzero entry always in the diagonal part AND the off-diagonal part of each row and just set one or the other to 0.0 each time so you always use the exact same matrix layout. But this will induce unneeded communication since PETSc will need to set up the scatter for that off diagonal entry even though it is zero. > On Apr 8, 2015, at 1:39 PM, Baron Law wrote: > > Hi, > > I need to compute the solution of least square regression for many times (> 10^6) for large matrix A (>10^9x10^3). I use a hypercube basis, so each row of the matrix A has exactly one non-zero entry with value 1. On different runs, the positions of the 1 are different. I would like to reuse the sparse structure of A, what is the best way to do in this scenario? > > I have heard of MatCreateMPIAIJWithSplitArrays, but it is kind of difficult to use. I try it a bit and seems like after creation if I need to change the column index, I need to skip the diagonal block. Also, I am not 100% sure, how to find which columns belong to the diagonal block for each process. But as mentioned, I need to run the regression million times, even it is difficult to use, I will try if it is fastest method. > > Best, > Baron From amesga1 at tigers.lsu.edu Wed Apr 8 17:47:49 2015 From: amesga1 at tigers.lsu.edu (Ataollah Mesgarnejad) Date: Wed, 8 Apr 2015 17:47:49 -0500 Subject: [petsc-users] Seg fault when trying to access the graph of pointSF from DMPlexDistribute Message-ID: Dear all, I get a weird segmentation fault when I try to access the arrays of PetscInt and PetscSFNode from are the graph of the pointSF that DMPlexDistribute returns. I'm attaching a small test that can show the problem. You can run with: testPointSF -i TwoTri.gen As always thank you for your help, Ata -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: testPointSF.cpp Type: text/x-c++src Size: 4000 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: TwoTri.gen Type: application/octet-stream Size: 1532 bytes Desc: not available URL: From knepley at gmail.com Wed Apr 8 18:05:00 2015 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 8 Apr 2015 18:05:00 -0500 Subject: [petsc-users] Seg fault when trying to access the graph of pointSF from DMPlexDistribute In-Reply-To: References: Message-ID: On Wed, Apr 8, 2015 at 5:47 PM, Ataollah Mesgarnejad wrote: > Dear all, > > I get a weird segmentation fault when I try to access the arrays of > PetscInt and PetscSFNode from are the graph of the pointSF that > DMPlexDistribute returns. I'm attaching a small test that can show the > problem. You can run with: > The leaves can be NULL if it is a contiguous array covering all indices. You can see that the Plex code checks for this. Thanks, Matt > testPointSF -i TwoTri.gen > > As always thank you for your help, > Ata > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.knezevic at akselos.com Wed Apr 8 20:04:09 2015 From: david.knezevic at akselos.com (David Knezevic) Date: Wed, 8 Apr 2015 21:04:09 -0400 Subject: [petsc-users] Exploiting symmetry with direct solvers Message-ID: I'm curious about how to do a symmetric LDL^T factorization (instead of LU) with MUMPS and SuperLU. Based on this example: http://www.mcs.anl.gov/petsc/petsc-3.4/src/ksp/ksp/examples/tutorials/ex52.c.html my understanding is as follows: - With MUMPS I gather that we need to specify: MatSetOption(A,MAT_SPD,PETSC_TRUE); PCSetType(pc,PCCHOLESKY); I guess "-pc_type cholesky" on the command line is equivalent to the PCSetType call, right? Is specifying MAT_SPD required in order for MUMPS to do an LDL^T factorization? - I gather that SuperLU doesn't provide a symmetric factorization. Is this correct? Thanks! David -------------- next part -------------- An HTML attachment was scrubbed... URL: From amesga1 at tigers.lsu.edu Wed Apr 8 20:22:09 2015 From: amesga1 at tigers.lsu.edu (Ataollah Mesgarnejad) Date: Wed, 8 Apr 2015 20:22:09 -0500 Subject: [petsc-users] Seg fault when trying to access the graph of pointSF from DMPlexDistribute In-Reply-To: References: Message-ID: <38AEEB11-DDF2-4734-BD3A-935B369309D2@tigers.lsu.edu> Thanks Matt, I knew you can pass NULL for a contiguous array but I didn?t suspect that you can get NULL from PetscSFGetGraph. Thank you for your time, Ata > On Apr 8, 2015, at 6:05 PM, Matthew Knepley wrote: > > On Wed, Apr 8, 2015 at 5:47 PM, Ataollah Mesgarnejad > wrote: > Dear all, > > I get a weird segmentation fault when I try to access the arrays of PetscInt and PetscSFNode from are the graph of the pointSF that DMPlexDistribute returns. I'm attaching a small test that can show the problem. You can run with: > > The leaves can be NULL if it is a contiguous array covering all indices. You can see that the Plex code checks for this. > > Thanks, > > Matt > > testPointSF -i TwoTri.gen > > As always thank you for your help, > Ata > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Apr 8 20:28:29 2015 From: hzhang at mcs.anl.gov (Hong) Date: Wed, 8 Apr 2015 20:28:29 -0500 Subject: [petsc-users] Exploiting symmetry with direct solvers In-Reply-To: References: Message-ID: David , > I'm curious about how to do a symmetric LDL^T factorization (instead of > LU) with MUMPS and SuperLU. Based on this example: > > > http://www.mcs.anl.gov/petsc/petsc-3.4/src/ksp/ksp/examples/tutorials/ex52.c.html > > my understanding is as follows: > > - With MUMPS I gather that we need to specify: > > MatSetOption(A,MAT_SPD,PETSC_TRUE); > PCSetType(pc,PCCHOLESKY); > > I guess "-pc_type cholesky" on the command line is equivalent to the > PCSetType call, right? Is specifying MAT_SPD required in order for MUMPS to > do an LDL^T factorization? > Mumps supports Cholesky factorization for symmetric, and symmetric+spd matrices. You may consult mumps user manual. > > - I gather that SuperLU doesn't provide a symmetric factorization. > SuperLU does not support Cholesky factorization. Hong -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.knezevic at akselos.com Wed Apr 8 21:16:29 2015 From: david.knezevic at akselos.com (David Knezevic) Date: Wed, 8 Apr 2015 22:16:29 -0400 Subject: [petsc-users] Exploiting symmetry with direct solvers In-Reply-To: References: Message-ID: On Wed, Apr 8, 2015 at 9:28 PM, Hong wrote: > David , > >> I'm curious about how to do a symmetric LDL^T factorization (instead of >> LU) with MUMPS and SuperLU. Based on this example: >> >> >> http://www.mcs.anl.gov/petsc/petsc-3.4/src/ksp/ksp/examples/tutorials/ex52.c.html >> >> my understanding is as follows: >> >> - With MUMPS I gather that we need to specify: >> >> MatSetOption(A,MAT_SPD,PETSC_TRUE); >> PCSetType(pc,PCCHOLESKY); >> >> I guess "-pc_type cholesky" on the command line is equivalent to the >> PCSetType call, right? Is specifying MAT_SPD required in order for MUMPS to >> do an LDL^T factorization? >> > Mumps supports Cholesky factorization for symmetric, and symmetric+spd > matrices. You may consult mumps user manual. > OK, thanks. But I was also wondering about the PETSc interface to MUMPS. I consulted the MUMPS manual, but it just says that you need to specify "SYM=1" in order to get an LDL^T factorization. I gather that if we set MatSetOption(A,MAT_SPD,PETSC_TRUE); in PETSc then that will tell MUMPS to use SYM=1, right? Thanks, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From dharmareddy84 at gmail.com Wed Apr 8 21:39:02 2015 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Wed, 8 Apr 2015 21:39:02 -0500 Subject: [petsc-users] DMNetwork usage example Message-ID: Hello, Is there a Fortran or C code example illustrating the usage of DMNetwork ? Thanks Reddy -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Apr 8 22:06:23 2015 From: hzhang at mcs.anl.gov (Hong) Date: Wed, 8 Apr 2015 22:06:23 -0500 Subject: [petsc-users] Exploiting symmetry with direct solvers In-Reply-To: References: Message-ID: David: > >>> Mumps supports Cholesky factorization for symmetric, and symmetric+spd >> matrices. You may consult mumps user manual. >> > > > OK, thanks. But I was also wondering about the PETSc interface to MUMPS. > > I consulted the MUMPS manual, but it just says that you need to specify > "SYM=1" in order to get an LDL^T factorization. I gather that if we set > MatSetOption(A,MAT_SPD,PETSC_TRUE); in PETSc then that will tell MUMPS to > use SYM=1, right? > When user requests Cholesky factorization, PETSc sets "SYM=2" as default; If user sets flag of spd, then the interface uses "SYM=1" in order to get an LDL^T factorization. See MatGetFactor_xxx_mumps() in petsc/src/mat/impls/aij/mpi/mumps/mumps.c: ... B->factortype = MAT_FACTOR_CHOLESKY; if (A->spd_set && A->spd) mumps->sym = 1; else mumps->sym = 2; ... Hong -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Apr 8 22:07:53 2015 From: hzhang at mcs.anl.gov (Hong) Date: Wed, 8 Apr 2015 22:07:53 -0500 Subject: [petsc-users] DMNetwork usage example In-Reply-To: References: Message-ID: See *petsc/src/snes/examples/tutorials/network/pflow* *Hong* On Wed, Apr 8, 2015 at 9:39 PM, Dharmendar Reddy wrote: > Hello, > Is there a Fortran or C code example illustrating the usage of > DMNetwork ? > > Thanks > Reddy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.knezevic at akselos.com Wed Apr 8 22:08:15 2015 From: david.knezevic at akselos.com (David Knezevic) Date: Wed, 8 Apr 2015 23:08:15 -0400 Subject: [petsc-users] Exploiting symmetry with direct solvers In-Reply-To: References: Message-ID: On Wed, Apr 8, 2015 at 11:06 PM, Hong wrote: > David: > >> >>>> Mumps supports Cholesky factorization for symmetric, and symmetric+spd >>> matrices. You may consult mumps user manual. >>> >> >> >> OK, thanks. But I was also wondering about the PETSc interface to MUMPS. >> >> I consulted the MUMPS manual, but it just says that you need to specify >> "SYM=1" in order to get an LDL^T factorization. I gather that if we set >> MatSetOption(A,MAT_SPD,PETSC_TRUE); in PETSc then that will tell MUMPS to >> use SYM=1, right? >> > > When user requests Cholesky factorization, PETSc sets "SYM=2" as default; > If user sets flag of spd, then the interface uses "SYM=1" in order to get > an LDL^T factorization. See MatGetFactor_xxx_mumps() in > petsc/src/mat/impls/aij/mpi/mumps/mumps.c: > > ... > B->factortype = MAT_FACTOR_CHOLESKY; > if (A->spd_set && A->spd) mumps->sym = 1; > else mumps->sym = 2; > ... > OK, got it, thanks! David -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Wed Apr 8 22:18:41 2015 From: zonexo at gmail.com (TAY wee-beng) Date: Thu, 09 Apr 2015 11:18:41 +0800 Subject: [petsc-users] Recommended examples for using KSP and multigrid with DM Message-ID: <5525EF91.3020709@gmail.com> Hi, I was following example ex29 in the past to write the code to solve my Poisson eqn using DM. Now it seems that ex29 is no longer valid. Which example should I follow now? Also, I am using KSPBCGS to solve my momentum eqn in the past. Do I need to change much if I'm now using DM? If so, what is a good example? -- Thank you Yours sincerely, TAY wee-beng From dharmareddy84 at gmail.com Wed Apr 8 23:04:57 2015 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Wed, 8 Apr 2015 23:04:57 -0500 Subject: [petsc-users] DMNetwork usage example In-Reply-To: References: Message-ID: Hello, I have a few question regarding the usage. In DMNetworkRegisterComponent(DM dm,const char *name,PetscInt size,PetscInt *key) * size *- the storage size in bytes for this component data If i call it from fortran, is size then the number of components ? For example, if i have a network of nodes, I want to store the location of the node ( double precision :: x,y,z) in the component named "position", then is the call: call DMNetworkRegisterComponent(DM dm,"position", 3, positionkey, ierr) I need to create a network. I know the number of nodes on network and the maximum number of neighbors any node can have. I can only add the node to node connections as few at a time. Is this possible to do with DMNetowrk ? Reddy On Wed, Apr 8, 2015 at 10:07 PM, Hong wrote: > See > *petsc/src/snes/examples/tutorials/network/pflow* > > *Hong* > > On Wed, Apr 8, 2015 at 9:39 PM, Dharmendar Reddy > wrote: > >> Hello, >> Is there a Fortran or C code example illustrating the usage of >> DMNetwork ? >> >> Thanks >> Reddy >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhyshr at mcs.anl.gov Wed Apr 8 23:40:55 2015 From: abhyshr at mcs.anl.gov (Abhyankar, Shrirang G.) Date: Thu, 9 Apr 2015 04:40:55 +0000 Subject: [petsc-users] DMNetwork usage example In-Reply-To: References: Message-ID: From: Dharmendar Reddy Date: Wednesday, April 8, 2015 at 11:04 PM To: hong zhang Cc: PETSc users list Subject: Re: [petsc-users] DMNetwork usage example >Hello, > > I have a few question regarding the usage. > >In DMNetworkRegisterComponent(DM dm,const char *name,PetscInt >size,PetscInt *key) >size - the storage size in bytes for this component data > > >If i call it from fortran, is size then the number of components ? > > I don?t think the Fortran interface for DMNetwork works and neither have I tested it. > > >For example, if i have a network of nodes, I want to store the location >of the node ( double precision :: x,y,z) in the component named >"position", then is the call: > > > >call DMNetworkRegisterComponent(DM dm,"position", 3, positionkey, ierr) > > > The size parameter is the storage size component data. So if you have a component position described by struct position { double x,y,z; }; then the call to DMNetworkRegisterComponent (in C) would be DMNetworkRegisterComponent(dm, ,sizeof(position),&positionkey); I don?t think I can get to making the Fortran interface work for DMNetwork in the next couple of months. You can try to use DMPlex instead if you are in urgent need of a Fortran interface for network problems. DMNetwork is built on top of DMPlex and uses a lot of its elements to manage the network layout and communication. Please read Chapter 17 in the manual. > > >I need to create a network. I know the number of nodes on network and the >maximum number of neighbors any node can have. I can only add the node to >node connections as few at a time. I am not sure I understand. If you provide the number of nodes and a list of node connectivity, DMNetwork will set up the network layout. Shri > > > >Is this possible to do with DMNetowrk ? > > > >Reddy > > > >On Wed, Apr 8, 2015 at 10:07 PM, Hong wrote: > >See >petsc/src/snes/examples/tutorials/network/pflow > > >Hong > > >On Wed, Apr 8, 2015 at 9:39 PM, Dharmendar Reddy > wrote: > >Hello, > > Is there a Fortran or C code example illustrating the usage of >DMNetwork ? > > > >Thanks > >Reddy > > > > > > > > > > > From pascal.tremblay at fr.michelin.com Thu Apr 9 02:06:13 2015 From: pascal.tremblay at fr.michelin.com (pascal.tremblay at fr.michelin.com) Date: Thu, 9 Apr 2015 09:06:13 +0200 Subject: [petsc-users] Exploiting symmetry with direct solvers In-Reply-To: References: Message-ID: Hi, There is also Intel MKL Pardiso that I have been using lately (at Michelin) and I have extended the mkl_pardiso.c interface to PETSc to support PCCHOLESKY (with an SEQSBAIJ matrix and a block size of one) and not only PCLU (with a SEQAIJ matrix). We have been testing it internally and so fare it is robust and very fast for a direct solver. I plan to submit my modifications to the main trunk (pull request) in the coming days ... I have seen the extension to Cluster Pardiso in the main trunk and it also done only for LU and MPIAIJ matrices. It would be nice to extended it to symetric matrices as well. I wanted to do the same extension for the Cluster Pardiso, but I didn't have time yet (and it does not work with with OpenMPI, but works with Intel MPI). Regards, Pascal Tremblay --------------------------------------------------------------------------------------------- M?canique Num?rique Michelin, Centre de technologie de Ladoux -DGRD/CTEU/PM/SIM/ER 63040 Clermont-Ferrand Cedex 09 - FRANCE T?l. (ext.): 04 73 10 91 63 T?l. (int.): 61 163 --------------------------------------------------------------------------------------------- Confidentialit? D3- Conservation <90j De : David Knezevic A : Hong , Cc : PETSc users list Date : 09/04/2015 04:17 Objet : Re: [petsc-users] Exploiting symmetry with direct solvers Envoy? par : On Wed, Apr 8, 2015 at 9:28 PM, Hong wrote: David , I'm curious about how to do a symmetric LDL^T factorization (instead of LU) with MUMPS and SuperLU. Based on this example: http://www.mcs.anl.gov/petsc/petsc-3.4/src/ksp/ksp/examples/tutorials/ex52.c.html my understanding is as follows: - With MUMPS I gather that we need to specify: MatSetOption(A,MAT_SPD,PETSC_TRUE); PCSetType(pc,PCCHOLESKY); I guess "-pc_type cholesky" on the command line is equivalent to the PCSetType call, right? Is specifying MAT_SPD required in order for MUMPS to do an LDL^T factorization? Mumps supports Cholesky factorization for symmetric, and symmetric+spd matrices. You may consult mumps user manual. OK, thanks. But I was also wondering about the PETSc interface to MUMPS. I consulted the MUMPS manual, but it just says that you need to specify "SYM=1" in order to get an LDL^T factorization. I gather that if we set MatSetOption(A,MAT_SPD,PETSC_TRUE); in PETSc then that will tell MUMPS to use SYM=1, right? Thanks, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From matteo.aletti at inria.fr Thu Apr 9 06:02:41 2015 From: matteo.aletti at inria.fr (Matteo Aletti) Date: Thu, 09 Apr 2015 13:02:41 +0200 Subject: [petsc-users] Problem when using MatLoad Message-ID: <55265C51.3050802@inria.fr> Hello, I was trying to use the function MatLoad to read an mpi dense (square) matrix from a binary file. I got an error related to a memory problem (in parallel). I tried to locate the problem using gdb and I think it is in the function /MatLoad_MPIDense_DenseInFile//. /The master cannot execute the line / / / ierr = PetscFree(vals);CHKERRQ(ierr);/ without an error. The other processors can. For me, the error is in the allocation of the /vals/ array: each processor allocates it with m*N number of elements, where N is the same for all of the procs and m is the local number of rows, which in my case is already given in the matrix. The master uses the array vals to read its own data, but also the other processors' data. The problem is that, in my case, one processor has a higher number of rows than the master and therefore vals is too short to store those values. For me the master should allocate it with size m_max*N , where m_max is the maximum number of local rows between all procs. Thanks, Best, Matteo -------------- next part -------------- An HTML attachment was scrubbed... URL: From siddhesh4godbole at gmail.com Thu Apr 9 06:07:32 2015 From: siddhesh4godbole at gmail.com (siddhesh godbole) Date: Thu, 9 Apr 2015 16:37:32 +0530 Subject: [petsc-users] petscmat.h missing definition Message-ID: Hello i have petsc-3.5.2 and i can not find MatGetArray and MatRestoreArray in my petscmat.h which is supposed to contain the definitions of it as per online help module. i am getting undefined reference to these routines errors. what should i do? kindly help *Siddhesh M Godbole* 5th year Dual Degree, Civil Eng & Applied Mech. IIT Madras -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Apr 9 06:53:34 2015 From: jed at jedbrown.org (Jed Brown) Date: Thu, 09 Apr 2015 05:53:34 -0600 Subject: [petsc-users] petscmat.h missing definition In-Reply-To: References: Message-ID: <874mopldq9.fsf@jedbrown.org> siddhesh godbole writes: > Hello > > i have petsc-3.5.2 and i can not find MatGetArray and MatRestoreArray MatGetArray()/MatRestoreArray() have been removed. Use MatDenseGetArray()/MatDenseRestoreArray() or MatSeqAIJGetArray()/MatSeqAIJRestoreArray() http://www.mcs.anl.gov/petsc/documentation/changes/34.html > in my petscmat.h which is supposed to contain the definitions of it as > per online help module. i am getting undefined reference to these > routines errors. what should i do? > > kindly help > > *Siddhesh M Godbole* > > 5th year Dual Degree, > Civil Eng & Applied Mech. > IIT Madras -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From georgesamarasdit at gmail.com Thu Apr 9 08:06:31 2015 From: georgesamarasdit at gmail.com (Georgios Samaras) Date: Thu, 9 Apr 2015 16:06:31 +0300 Subject: [petsc-users] PETSc for distributed computing? Message-ID: Dear all, I am looking for a C/C++ linear algebra package for performing inversion of a large dense matrix in a distributed environment. Is your package suitable for this goal? Thanks in advance, Georgios Samaras -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrick.sanan at gmail.com Thu Apr 9 08:31:37 2015 From: patrick.sanan at gmail.com (Patrick Sanan) Date: Thu, 9 Apr 2015 15:31:37 +0200 Subject: [petsc-users] PETSc for distributed computing? In-Reply-To: References: Message-ID: <382DADCD-513A-4591-9730-425DD55FFB43@gmail.com> Do you want an explicit inverse, or just to solve linear systems involving a dense matrix? If so, how many right hand sides? PETSc itself focuses on sparse linear algebra, but it includes an interface to the dense linear algebra library Elemental. Il giorno 09/apr/2015, alle ore 03:06 PM, Georgios Samaras ha scritto: > Dear all, > > I am looking for a C/C++ linear algebra package for performing inversion of a large dense matrix in a distributed environment. Is your package suitable for this goal? > > Thanks in advance, > Georgios Samaras From juris.vencels at gmail.com Thu Apr 9 12:55:59 2015 From: juris.vencels at gmail.com (Juris Vencels) Date: Thu, 09 Apr 2015 11:55:59 -0600 Subject: [petsc-users] finclude (web) vs petsc-finclude (git) Message-ID: <5526BD2F.4000109@gmail.com> Hi, I am using PETSc Fortran interface and FindPETSc.cmake script to find PETSc. https://github.com/jedbrown/cmake-modules/blob/master/FindPETSc.cmake PETSc from WEB has folder include folder: include/finclude while PETSc from git has folder: include/petsc-finclude Is there a way to make it compatible between different versions? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Apr 9 13:49:39 2015 From: jed at jedbrown.org (Jed Brown) Date: Thu, 09 Apr 2015 12:49:39 -0600 Subject: [petsc-users] finclude (web) vs petsc-finclude (git) In-Reply-To: <5526BD2F.4000109@gmail.com> References: <5526BD2F.4000109@gmail.com> Message-ID: <87oamxi1bw.fsf@jedbrown.org> Juris Vencels writes: > Hi, > > I am using PETSc Fortran interface and FindPETSc.cmake script to find PETSc. > https://github.com/jedbrown/cmake-modules/blob/master/FindPETSc.cmake Correct me if I'm wrong, but it doesn't sound like your question has anything to do with this cmake module. > PETSc from WEB has folder include folder: > include/finclude > > while PETSc from git has folder: > include/petsc-finclude > > Is there a way to make it compatible between different versions? You could create a symlink for that directory (but that's a terrible hack if you send your code to other people). You can guard it with the preprocessor: #if PETSC_VERSION_LT(3,6,0) #include #else #include #endif Note that you could define a path splicing macro so that you could write #include PETSC_FORTRAN_INCLUDE(petscxyz.h) but CMake's dependency analysis would fail (I think). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From haakon at hakostra.net Thu Apr 9 14:11:29 2015 From: haakon at hakostra.net (=?UTF-8?B?SMOla29uIFN0cmFuZGVuZXM=?=) Date: Thu, 09 Apr 2015 21:11:29 +0200 Subject: [petsc-users] Load Vec from 1D HDF5 dataset In-Reply-To: References: <550A94B9.90407@hakostra.net> <90B98E36-72F6-43D2-A157-5A816FE09746@mcs.anl.gov> <550B07D8.9060903@hakostra.net> <6317A2EC-0891-4C1D-8D54-22C71BCE48DA@mcs.anl.gov> <550BCBD9.80608@hakostra.net> <87zj77ivyg.fsf@jedbrown.org> <550C3642.5000208@hakostra.net> <87h9tfaedt.fsf@jedbrown.org> <550C40A7.5090303@hakostra.net> <878uerachd.fsf@jedbrown.org> <550C4B47.60903@hakostra.net> <01CA320F-4B9E-438D-9F9F-B1763CD9253E@mcs.anl.gov> <1CC4C38E-91A4-48D5-8840-2D224F30FE6E@mcs.anl.gov> <6DE84421-03A9-402E-801C-624D36A84C1B@mcs.anl.gov> <550C8ACC.3030102@hakostra.net> <5512D5FC.7090108@hakostra.net> Message-ID: <5526CEE1.5070901@hakostra.net> I have now had the opportunity to look into this again, and have made a pull request with a suggested change: https://bitbucket.org/petsc/petsc/pull-request/296/vecload_hdf5-can-without-setting-any/diff In short terms, for loading Vecs, I suggest to do a "best effort guess" on the contents of the dataset in the HDF5 file, by doing the following: 1) Assume no extra dimension for bs = 1 2) If the dataset happens to have one dimension more than expected, check the size of the dimension corresponding to the bs. 3) If the size of the dimension corresponding to the bs is == 1, accept the dataset with one extra dimension. Since it can accept Vecs of "both kind" (with and without one extra dimension) it is 100% backwards compatible for *loading* Vecs. The Vec output is not touched, and Barry's "basedimension2"-property is left for the output/write part. The same logic could/should(!) be applied when reading DMDA's with dof=1 as well, I just thought you might come with some critics on this first try. Regards, H?kon On 29. mars 2015 04:59, Barry Smith wrote: > > Hakon, > > I have pushed to the branch barry/feature-hdf5-flexible-initial-dimension and next the change so that Vecs and Vecs obtained from DMCreateGlobalVector() with a DMDA will NOT have the extra dimension if BS == 1. To add that extra dimension even when bs == 1 with VecView() or to handle a file with that extra dimension with VecLoad() one must call PetscViewerHDF5SetBaseDimension2() or -viewer_hdf5_base_dimension2 true > > Please try it out and let me know if you have any trouble > > Thanks > > Barry > > Matt, > > Scream as much as you want but adding that extra dimension automatically is not intuitive so it cannot be the default. > >> On Mar 25, 2015, at 10:36 AM, H?kon Strandenes wrote: >> >> Did you come to any conclusion on this issue? >> >> Regards, >> H?kon >> >> On 20. mars 2015 22:02, H?kon Strandenes wrote: >>> On 20. mars 2015 20:48, Barry Smith wrote: >>>> Why is 1 dimension a special case that is not worthy of its own >>>> format? The same thing would hold for 2d and 3d. One could then argue >>>> that we should have a single six dimensional format for the files for >>>> all vectors that PETSc produces. Then a 1d problem has five of the >>>> dimensions being 1. >>> >>> This is a very good point, and support my view. >>> >>> Let me come with two very simple example cases: >>> >>> >>> Case 1: >>> Create a list of grid points in an external preprocessor for the purpose >>> of loading this into a Vec later: >>> >>> x = np.linspace(0.0, 1.0, num=21) >>> f.create_dataset('/MESH/nodes/x', data=x) >>> >>> vs. >>> >>> x = np.linspace(0.0, 1.0, num=21) >>> x = x.reshape((21,1)) >>> f.create_dataset('/MESH/nodes/x', data=x) >>> >>> >>> Case 2: >>> Read three Vecs written to disk by PETSc, and calculate total "bounding >>> box volume" of the grid: >>> >>> g = h5py.File('grid.h5', 'r') >>> x = g['/MESH/nodes/x'] >>> y = g['/MESH/nodes/y'] >>> z = g['/MESH/nodes/z'] >>> Vol = (xp[-1] - xp[0])*(yp[-1] - yp[0])*(zp[-1] - zp[0]) >>> >>> vs. >>> >>> g = h5py.File('grid.h5', 'r') >>> x = g['/MESH/nodes/x'][:,0] >>> y = g['/MESH/nodes/y'][:,0] >>> z = g['/MESH/nodes/z'][:,0] >>> Vol = (x[-1] - x[0])*(y[-1] - y[0])*(z[-1] - z[0]) >>> >>> >>> In both cases I think handling this extra, unnecessary dimension makes >>> the code less attractive. It's not that either way is difficult, >>> problematic or impossible, but it's just that 1D Vecs should intuitively >>> be 1D datasets, and not 2D, 3D or 6D. This seriously confused me for >>> quite a while until I figured this out, even after having written an >>> entire Navier-Stokes DNS solver using the PETSc library for everything >>> except time integration and filling these simple 1D coordinate arrays! >>> >>> Regards, >>> H?kon > > From knepley at gmail.com Thu Apr 9 14:11:53 2015 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Apr 2015 14:11:53 -0500 Subject: [petsc-users] Problem with MPI, MatAXPY and SAME_NONZERO_PATTERN In-Reply-To: <5515805B.3060406@igpm.rwth-aachen.de> References: <55156DA7.6010602@igpm.rwth-aachen.de> <55157460.4080200@igpm.rwth-aachen.de> <5515805B.3060406@igpm.rwth-aachen.de> Message-ID: On Fri, Mar 27, 2015 at 11:07 AM, Klaus Kaiser wrote: > Hallo Matthew, > > here is a short example I used for 8 cores: > > int d_nnz[3]; > int o_nnz[3]; > d_nnz[0] = 3; d_nnz[1] = 3; d_nnz[2] = 3; > o_nnz[0] = 6; o_nnz[1] = 6; o_nnz[2] = 6; > ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, d_nnz, 0, > o_nnz,&A); > ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); > ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); > ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); > ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, d_nnz, 0, > o_nnz,&Ah); > ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, d_nnz, 0, > o_nnz,&At); > > std::vector insert(3*3*3*3, 1.0); > for(int i=0;i<8;++i) > { > int rows[3] = {i,i+1,i+3}; > int cols[3] = {i,i+1,i+3}; > > MatSetValuesBlocked(A, 3, rows, 3, cols, &insert[0], ADD_VALUES); > } > I do not understand this example. All the procs (8) are filling up only the first 32 rows. Is that what you intended? Can you tell me what output you expect? Thanks, Matt > MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); > MatDuplicate(A,MAT_COPY_VALUES,&Ah); > MatDuplicate(A,MAT_COPY_VALUES,&At); > MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY); > MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY); > > MatAXPY(Ah,1.,At,SAME_NONZERO_PATTERN); > MatAXPY(A,1.,Ah,SAME_NONZERO_PATTERN); > > MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); > > The result is, that only some values are computed correctly. > > Best and Thank you a lot > > Klaus > > > > > On 03/27/2015 04:26 PM, Matthew Knepley wrote: > > On Fri, Mar 27, 2015 at 10:16 AM, Klaus Kaiser < > kaiser at igpm.rwth-aachen.de> wrote: > >> Hallo Matthew, >> >> thanks for your fast response. With "no beside the nonzero structure" I >> meant, that I do not create a different non-zero structure while I'm adding >> values to my matrices. >> >> I also tried MAT_COPY_VALUES and the result was the same. >> > > Please send the small example and we will figure out what is going wrong. > > Thanks, > > Matt > > >> I do not get a error message but the resulting matrix is wrong. Here I >> have a short example. The first 9 rows and first 3 colomn of the matrices >> Ah, At and Ah+At on the first processor (of 8): >> >> Ah: >> 0: 60.3553 -0.249975 >> 2.77556e-17 >> 1: 0 >> 60.3553 0 >> 2: 0 0 >> 60.3553 >> 3: 17.6777 0.374962 >> 0.124987 >> 4: 0 >> 17.6777 0 >> 5: 0 0 >> 17.6777 >> 6: -7.32233 -0.124987 >> 0.374962 >> 7: 0 >> -7.32233 0 >> 8: 0 0 >> -7.32233 >> >> >> At: >> 0: 0 >> 0 0 >> 1: 2500 >> 0 0 >> 2: -4.54747e-13 >> 0 0 >> 3: 0 >> 0 0 >> 4: 1250 >> 0 0 >> 5: 1250 >> 0 0 >> 6: 0 >> 0 0 >> 7: 1250 >> 0 0 >> 8: 3750 >> 0 0 >> >> Ah+At >> 0: 60.3553 -0.249975 >> 2.77556e-17 >> 1: 2500 >> 60.3553 0 >> 2: -4.54747e-13 0 >> 60.3553 >> 3: 17.6777 0.374962 >> 0.124987 >> 4: 0 >> 17.6777 0 >> 5: 0 0 >> 17.6777 >> 6: -7.32233 -0.124987 >> 0.374962 >> 7: 0 >> -7.32233 0 >> 8: 0 0 >> -7.32233 >> >> you can see the first 3 rows of the resulting matrix looks exactly like >> what I would expect, but the last 6 rows only consists of the values of Ah. >> When you would also look on the matrix A and A+Ah+At you would also see, >> that values where both matrices have an nonzero entry are not sum correctly. >> >> Best >> >> Klaus >> >> On 03/27/2015 03:59 PM, Matthew Knepley wrote: >> >> On Fri, Mar 27, 2015 at 9:48 AM, Klaus Kaiser < >> kaiser at igpm.rwth-aachen.de> wrote: >> >>> Hallo, >>> >>> I have a strange behavior in my code concerning the function MatAXPY. I >>> create 3 different Matrices >>> >>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, >>> local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&A); >>> ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); >>> ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); >>> ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); >>> >>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, >>> local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&At); >>> ierr = MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); >>> ierr = MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); >>> ierr = MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); >>> >>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size, >>> local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&Ah); >>> ierr = MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); >>> ierr = MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); >>> ierr = MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); >>> >> >> These creations are superfluous since you use MatDuplicate() below. >> >> >>> and want to sum these three matrixes with different factors. First I >>> fill the Matrix A with some values, and duplicate the structure of A to the >>> other two matrices: >>> >>> MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); >>> MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); >>> MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah); >>> MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At); >>> MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY); >>> MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY); >>> MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY); >>> MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY); >>> >> >> >>> After this I fill the matrices At and Ah with some other values, which >>> are not beside the non zero structure (I also tried with just copying the >>> Matrix A). Now after another MatAssembly >> >> >> I do not understand "no beside the nonzero structure". Do you mean that >> the nonzero structure is the same? >> >> Can you first test with MAT_COPY_VALUES? >> >> >>> I want to add these Matrices in the form A+c*(Ah+d*At): >> >> >>> MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN); >>> MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN); >>> >>> When I run the method with mpi and one core everything works fine. >>> Starting the same method with more cores, the sum of the matrices fails. It >>> seems like some values are added >> >> >> Please send the full output of a failure when you use MAT_COPY_VALUES >> on 2 procs. >> >> Thanks, >> >> Matt >> >> >>> correctly and many values are missed. Using DIFFERENT_NONZERO_STRUCTURE >>> leads to the right behavior in the multi-core case, but is very slow. I >>> checked with a viewer if all matrices have the same nonzero structure and >>> this is the case. >>> >>> Does anyone know why this fails, or do I have made any wrong thoughts? >>> >>> I'm corrently working with a petsc version (Petsc Release Version 3.3.0, >>> Patch 5, Sat Dec 1 15:10:41 CST 2012), I looked into the changelogs up to >>> the current version and did not find any note about MatAXPY or MatAYPX. >>> >>> >>> Best and Thanks a lot for your help >>> >>> Klaus >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From stm8086 at yahoo.com Thu Apr 9 14:12:41 2015 From: stm8086 at yahoo.com (Steena M) Date: Thu, 9 Apr 2015 19:12:41 +0000 (UTC) Subject: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult In-Reply-To: <725037222.2337645.1428534563192.JavaMail.yahoo@mail.yahoo.com> References: <86DE4DD8-55D6-49B7-A87D-75D2E3AA0D4E@mcs.anl.gov> <725037222.2337645.1428534563192.JavaMail.yahoo@mail.yahoo.com> Message-ID: <2029028656.2993672.1428606761232.JavaMail.yahoo@mail.yahoo.com> Thanks, Barry! I patched the master, modified src/mat/examples/tests/ex190.c to suit my data, and the fix works. I need to execute MatMult and I am assigning ?vectors on each rank using MatCreateVecs(A, &x, &y). However, I don't think multiplication is happening at all. (I inserted a few printf statements inside MatMult_MPIBAIJ and inside MatMult_SeqBAIJ1 to check). ? VecSum(&ysum) is always zero. Maybe I'm assigning data incorrectly?? Code snippet in ex190.c after partitioning the matrix unequally ? ierr = MatLoad(A,fd);CHKERRQ(ierr);? ierr = MatCreateVecs(A, &x, &y);CHKERRQ(ierr);??? ierr = VecSet(x,one);CHKERRQ(ierr); //PetscScalar one = 1.0;? ierr = VecSet(y,zero); CHKERRQ(ierr);?//PetscScalar zero = 0.0;? ierr = MatMult(A,x,y); CHKERRQ(ierr);? ierr = ?VecSum(y,&ysum);CHKERRQ(ierr); //ysum is always zero.? Data is a 20x20 matrix and block size is 1. Any thoughts? Thanks,Steena???.? ? On Friday, April 3, 2015 4:17 PM, Barry Smith wrote: ? Steena, ? Sorry for all the delays. Our code was just totally wrong for a user provided decomposition for two reasons 1) the logic in MatLoad_MPIXAIJ() was too convoluted to work in all cases and 2) we didn't have a single test case. ? I have attached a patch that fixes it for petsc-3.5.3 .? The fix is also in the branch barry/fix-matload-uneven-rows and I'll put it into next then after testing it will got into maint and master and the next PETSc patch release. ? Please let us know if the patch doesn't work for you ? Thanks for your patience, ? Barry > On Apr 1, 2015, at 3:10 PM, Steena M wrote: > > Thanks Barry. Attached is the driver program, the binary mat file, and the corresponding mtx file. > > Runtime command used: > > sierra324 at monteiro:time srun -n 2 -ppdebug ./petsc-mpibaij-unequalrows -fin trefethen.dat -matload_block_size 1 > > > > > On Wednesday, April 1, 2015 12:28 PM, Barry Smith wrote: > > > >? Send a data file you generated and your reader program and we'll debug it. > >? Barry > > > On Apr 1, 2015, at 2:18 PM, Steena M wrote: > > > > Thanks Barry. I removed the Preallocation calls. It is still complaining about the malloc and incorrect data in the matrix file. I generate binary matrix files using PETSc's pythonscript to loop through a set of UFL sparse matrices. For this use case: > > > > mtx_mat = scipy.io.mmread('trefethen.mtx') > > PetscBinaryIO.PetscBinaryIO().writeMatSciPy(open('trefnew.dat','w'), mtx_mat) > > > > > >? > > > > > > > > On Tuesday, March 31, 2015 9:15 PM, Barry Smith wrote: > > > > > > > >? You should not need to call any preallocation routines when using MatLoad() > > > > > >? How did you generate the file? Are you sure it has the correct information for the matrix? > > > >? Barry > > > > > > > > > On Mar 31, 2015, at 11:05 PM, Steena M wrote: > > > > > > Thanks Matt. I'm still getting the malloc error > > > > > > [0]PETSC ERROR: Argument out of range! > > > [0]PETSC ERROR: New nonzero at (2,18) caused a malloc! > > > > > > and > > > > > > a new incorrect matrix file error: > > > > > > [0]PETSC ERROR: Unexpected data in file! > > > [0]PETSC ERROR: not matrix object! > > > > > > Maybe the order of calls is mixed up. This is the code snippet: > > > > > >? ? if (rank ==0) > > >? ? { > > >? ? PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); > > >? ? ? > > >? ? ? ? CHKERRQ(MatSetSizes(A, 15, PETSC_DETERMINE, 20, 20)); > > >? ? ? ? CHKERRQ(MatSetType(A, MATMPIBAIJ)); > > >? ? ? ? CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); > > >? ? ? ? CHKERRQ( MatLoad(A,fd)); > > >? ? CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); > > >? ? } > > >? ? > > >? ? else > > >? ? { > > >? ? ? ? PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); > > >? ? ? ? > > >? ? ? ? CHKERRQ( MatSetSizes(A, 5, PETSC_DETERMINE, 20, 20) ); > > >? ? ? ? CHKERRQ(MatSetType(A, MATMPIBAIJ)); > > >? ? ? ? CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); > > >? ? ? ? CHKERRQ(MatLoad(A,fd)); > > >? ? CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); > > >? ? ? ? } > > > > > > Is there something I'm missing? > > > > > > Thanks, > > > Steena > > >? ? > > > > > > > > > > > > > > > On Tuesday, March 31, 2015 6:10 PM, Matthew Knepley wrote: > > > > > > > > > On Tue, Mar 31, 2015 at 6:51 PM, Steena M wrote: > > > Thanks Barry. I'm still getting the malloc error with NULL. Is there a way to distribute the matrix without explicit preallocation? Different matrices will be loaded during runtime and assigning preallocation parameters would mean an additional preprocessing step. > > > > > > 1) MatSetOption(MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) > > >? > > > 2) Note that this is never ever ever more efficient than making another pass and preallocating > > > > > >? Thanks, > > > > > >? ? ? Matt > > > -------------------------------------------- > > > On Sun, 3/29/15, Barry Smith wrote: > > > > > >? Subject: Re: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult > > >? To: "Steena M" > > >? Cc: "Matthew Knepley" , petsc-users at mcs.anl.gov > > >? Date: Sunday, March 29, 2015, 9:26 PM > > > > > > > > >? > On > > >? Mar 29, 2015, at 11:05 PM, Steena M > > >? wrote: > > >? > > > >? > Thanks > > >? Matt. I used PETSC_DETERMINE but I'm now getting an > > >? allocation-based error: > > >? > > > >? > [0]PETSC ERROR: --------------------- > > >? Error Message ------------------------------------ > > >? > [0]PETSC ERROR: Argument out of range! > > >? > [0]PETSC ERROR: New nonzero at (2,18) > > >? caused a malloc! > > >? > [0]PETSC ERROR: > > >? ------------------------------------------------------------------------ > > >? > > > >? > I tried > > >? preallocating on each rank for the diagonal and off diagonal > > >? section of the matrix as the next step? My current > > >? approximations for preallocation > > >? > > > >? > CHKERRQ( > > >? MatMPIBAIJSetPreallocation(A,1,5,PETSC_DEFAULT,5,PETSC_DEFAULT)); > > > > > > > > >? ? These > > >? arguments where you pass PETSC_DEFAULT are expecting a > > >? pointer not an integer. You can pass NULL in those > > >? locations. Though it is better to provide the correct > > >? preallocation rather than some defaults. > > > > > >? ? Barry > > > > > >? > > > >? > are throwing segmentation errors. > > >? > > > >? > [0]PETSC ERROR: > > >? Caught signal number 11 SEGV: Segmentation Violation, > > >? probably memory access out of range > > >? > > > >? > Any insights into what I'm doing > > >? wrong? > > >? > > > >? > Thanks, > > >? > Steena > > >? > > > >? > > > >? > > > >? > On Sun, 3/29/15, Matthew Knepley > > >? wrote: > > >? > > > >? > Subject: > > >? Re: [petsc-users] Unequal sparse matrix row distribution for > > >? MPI MatMult > > >? > To: "Steena M" > > >? > > >? > Cc: "Barry Smith" , > > >? petsc-users at mcs.anl.gov > > >? > Date: Sunday, March 29, 2015, 10:02 PM > > >? > > > >? > On Sun, Mar 29, 2015 > > >? at > > >? > 9:56 PM, Steena M > > >? > wrote: > > >? > Hi > > >? > Barry, > > >? > > > >? > > > >? > > > >? > I am trying to partition a 20 row and 20 > > >? col sparse matrix > > >? > between two procs > > >? such that proc 0 has 15 rows and 20 cols > > >? > and proc 1 has 5 rows and 20 cols. The > > >? code snippet: > > >? > > > >? > > > >? > > > >? > > > >? > > > >? > > > >? > > > >? >? ? > > >? ? CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A)); > > >? // > > >? > at runtime: -matload_block_size 1 > > >? > > > >? > > > >? > > > >? > > > >? > > > >? >? ? > > >? ? if (rank ==0) > > >? > > > >? >? ? ? ? { > > >? > > > >? >? ? ? ? ? ? > > >? ? CHKERRQ( MatSetSizes(A, 15, 20, 20, > > >? > 20) ); //rank 0 gets 75% of the rows > > >? > > > >? >? ? ? ? ? ? > > >? ? CHKERRQ( MatSetType(A, MATMPIBAIJ) > > >? > ); > > >? > > > >? >? ? ? ? ? ? ? ? CHKERRQ( > > >? MatLoad(A,fd) ); > > >? > > > >? >? ? ? ? ? } > > >? > > > >? > > > >? > > > >? >? ? ? ? else > > >? > > > >? >? ? > > >? ? { > > >? > > > >? > > > >? ? ? ? ? ? ? ? CHKERRQ( MatSetSizes(A, 5, > > >? 20, 20, > > >? > 20) ); //rank 1 gets 25% of the > > >? rows > > >? > > > >? >? ? ? > > >? ? ? ? ? CHKERRQ( MatSetType(A, MATMPIBAIJ) > > >? > ); > > >? > > > >? >? ? ? ? ? ? ? ? CHKERRQ( > > >? MatLoad(A,fd) ); > > >? > > > >? >? ? ? } > > >? > > > > > > >? > > > >? > > > >? > This throws the following error (probably > > >? from psplit.c): > > >? > > > >? > > > >? [1]PETSC ERROR: --------------------- Error Message > > >? > ------------------------------------ > > >? > > > >? > [1]PETSC ERROR: > > >? Nonconforming object sizes! > > >? > > > >? > [1]PETSC ERROR: Sum of local lengths 40 > > >? does not equal > > >? > global length 20, my > > >? local length 20 > > >? > > > >? >? likely a call to > > >? VecSetSizes() or MatSetSizes() is > > >? > > > >? wrong. > > >? > > > >? > See > > >? http://www.mcs.anl.gov/petsc/documentation/faq.html#split! > > >? > > > >? > > > >? > > > >? > This error printout > > >? doesn't quite make sense to me. > > >? > > > >? I'm trying to specify a total matrix size of 20x20... > > >? I > > >? > haven't yet figured out where the > > >? '40' comes > > >? > from in the error > > >? message. > > >? > > > >? > > > >? > > > >? > Any thoughts on what > > >? might be going wrong? > > >? > > > >? > Its the column specification. Just > > >? > use PETSC_DETERMINE for the local columns > > >? since all our > > >? > sparse matrixformats are > > >? row divisions > > >? > anyway. > > >? > > > >? > Thanks, > > >? >? > > >? >? Matt > > >? > Thanks in advance, > > >? > > > > > > >? > Steena > > >? > > > >? > > > >? > > > >? > > > >? > > > >? > > > >? > > > >? > > > >? -------------------------------------------- > > >? > > > >? > On Sun, 3/22/15, > > >? Barry Smith > > >? > wrote: > > >? > > > >? > > > >? > > > >? >? Subject: Re: [petsc-users] Unequal > > >? sparse matrix row > > >? > distribution for MPI > > >? MatMult > > >? > > > >? >? To: > > >? "Steena M" > > >? > > > >? >? Cc: petsc-users at mcs.anl.gov > > >? > > > >? >? Date: Sunday, > > >? March 22, 2015, 3:58 PM > > >? > > > >? > > > >? > > > >? > > > >? > > > >? >? > > >? > > > >? >? Steena, > > >? > > > >? > > > >? > > > >? >? ? I am > > >? > > > > > > >? >? a little unsure of your question. > > > > > >? > > > >? > > > >? > > > >? >? ? 1) > > >? you can create a MPIBAIJ > > >? > > > >? >? matrix with any distribution of block > > >? rows per process > > >? > you > > >? > > > >? >? want, just set the > > >? local row size for each process to > > >? > > > >? be > > >? > > > >? >? what you > > >? like.? Use MatCreateVecs() to get > > >? > > > >? correspondingly > > >? > > > >? > > > >? laid out vectors. > > >? > > > >? > > > > > > >? > > > >? > > > >? ? or 2) if you have a MPIBAIJ > > >? > > > >? >? matrix with > > >? "equal" row layout and you want a > > >? > new > > >? > > > >? >? one with uneven row layout you can > > >? simply use > > >? > > > >? > > > >? MatGetSubMatrix() to create that new matrix. > > >? > > > >? > > > >? > > > >? >? ? Barry > > >? > > > >? > > > >? > > > >? >? Unless you have > > >? another reason to have the > > >? > > > >? >? matrix with an equal number row layout I > > >? would just > > >? > generate > > >? > > > >? >? the matrix with > > >? the layout you want. > > >? > > > >? > > > >? > > > >? > > > >? > > > >? >? > On Mar 22, 2015, at 5:50 PM, Steena > > >? M > > >? > > > >? >? > > >? > > > >? >? wrote: > > >? > > > >? >? > > > >? > > > >? >? > Hello, > > >? > > > >? >? > > > >? > > > >? >? > I need to > > >? distribute > > >? > > > >? >? a > > >? sparse matrix such that each proc owns an unequal > > >? > number > > >? > > > >? >? of blocked rows before I proceed with > > >? MPI MatMult. My > > >? > > > >? > > > >? initial thoughts on doing this: > > >? > > > >? >? > > > >? > > > >? >? > 1) Use? MatGetSubMatrices() on the > > >? test > > >? > > > >? >? MATMPIBAIJ > > >? matrix to produce a new matrix where each > > >? > proc > > >? > > > >? >? has an unequal number of rows. > > >? > > > >? >? > > > >? > > > >? >? > 2) Provide > > >? scatter context for vector X > > >? > > > >? >? (for MatMult )using IS iscol from > > >? MatGetSubMatrices() > > >? > while > > >? > > > >? >? creating the > > >? vector X. > > >? > > > >? > > > >? > > > >? > > > >? >? > 3) > > >? Call MatMult() > > >? > > > >? > > > >? > > > >? > > > >? >? > Will > > >? MatMult_MPIBAIJ continue to scatter > > >? > > > >? >? this matrix and vector such that each > > >? proc will own an > > >? > equal > > >? > > > >? >? number of matrix > > >? rows and corresponding diagonal vector > > >? > > > > > > >? >? elements? Should I write my own > > >? MPIMatMult function to > > >? > > > >? >? retain my redistribution of the matrix > > >? and vector? > > >? > > > >? > > > >? > > > >? > > > >? >? > > > >? Thanks in > > >? > > > >? > > > >? advance, > > >? > > > >? >? > > > >? Steena > > >? > > > >? > > > >? > > > >? > > > >? > > > >? > > > >? > -- > > >? > What most > > >? experimenters > > >? > take for granted before > > >? they begin their experiments is > > >? > > > >? infinitely more interesting than any results to which > > >? their > > >? > experiments lead. > > >? > -- Norbert > > >? > Wiener > > > > > >? > > > > > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > -- Norbert Wiener > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: fix-matload.patch Type: application/octet-stream Size: 15009 bytes Desc: not available URL: From knepley at gmail.com Thu Apr 9 14:15:01 2015 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Apr 2015 14:15:01 -0500 Subject: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult In-Reply-To: <2029028656.2993672.1428606761232.JavaMail.yahoo@mail.yahoo.com> References: <86DE4DD8-55D6-49B7-A87D-75D2E3AA0D4E@mcs.anl.gov> <725037222.2337645.1428534563192.JavaMail.yahoo@mail.yahoo.com> <2029028656.2993672.1428606761232.JavaMail.yahoo@mail.yahoo.com> Message-ID: On Thu, Apr 9, 2015 at 2:12 PM, Steena M wrote: > Thanks, Barry! I patched the master, modified > src/mat/examples/tests/ex190.c to suit my data, and the fix works. > > I need to execute MatMult and I am assigning vectors on each rank using > MatCreateVecs(A, &x, &y). However, I don't think multiplication is > happening at all. (I inserted a few printf statements inside > MatMult_MPIBAIJ and inside MatMult_SeqBAIJ1 to check). VecSum(&ysum) is > always zero. Maybe I'm assigning data incorrectly? > > Code snippet in ex190.c after partitioning the matrix unequally > > ierr = MatLoad(A,fd);CHKERRQ(ierr); > ierr = MatCreateVecs(A, &x, &y);CHKERRQ(ierr); > > ierr = VecSet(x,one);CHKERRQ(ierr); //PetscScalar one = 1.0; > ierr = VecSet(y,zero); CHKERRQ(ierr); //PetscScalar zero = 0.0; > MatView(A, 0) gives you the matrix, so you can see what Ax should be. > ierr = MatMult(A,x,y); CHKERRQ(ierr); > VecView(y, 0) gives you the output Matt > ierr = VecSum(y,&ysum);CHKERRQ(ierr); //ysum is always zero. > > Data is a 20x20 matrix and block size is 1. > > Any thoughts? > > Thanks, > Steena > > > . > > > > > On Friday, April 3, 2015 4:17 PM, Barry Smith > wrote: > > > > Steena, > > Sorry for all the delays. Our code was just totally wrong for a user > provided decomposition for two reasons 1) the logic in MatLoad_MPIXAIJ() > was too convoluted to work in all cases and 2) we didn't have a single test > case. > > I have attached a patch that fixes it for petsc-3.5.3 > . The fix is also in the branch barry/fix-matload-uneven-rows > and I'll put it into next then after testing it will got into maint and > master and the next PETSc patch release. > > Please let us know if the patch doesn't work for you > > Thanks for your patience, > > Barry > > > On Apr 1, 2015, at 3:10 PM, Steena M wrote: > > > > Thanks Barry. Attached is the driver program, the binary mat file, and > the corresponding mtx file. > > > > Runtime command used: > > > > sierra324 at monteiro:time srun -n 2 -ppdebug ./petsc-mpibaij-unequalrows > -fin trefethen.dat -matload_block_size 1 > > > > > > > > > > On Wednesday, April 1, 2015 12:28 PM, Barry Smith > wrote: > > > > > > > > Send a data file you generated and your reader program and we'll debug > it. > > > > Barry > > > > > On Apr 1, 2015, at 2:18 PM, Steena M wrote: > > > > > > Thanks Barry. I removed the Preallocation calls. It is still > complaining about the malloc and incorrect data in the matrix file. I > generate binary matrix files using PETSc's pythonscript to loop through a > set of UFL sparse matrices. For this use case: > > > > > > mtx_mat = scipy.io.mmread('trefethen.mtx') > > > PetscBinaryIO.PetscBinaryIO().writeMatSciPy(open('trefnew.dat','w'), > mtx_mat) > > > > > > > > > > > > > > > > > > > > > On Tuesday, March 31, 2015 9:15 PM, Barry Smith > wrote: > > > > > > > > > > > > You should not need to call any preallocation routines when using > MatLoad() > > > > > > > > > How did you generate the file? Are you sure it has the correct > information for the matrix? > > > > > > Barry > > > > > > > > > > > > > On Mar 31, 2015, at 11:05 PM, Steena M wrote: > > > > > > > > Thanks Matt. I'm still getting the malloc error > > > > > > > > [0]PETSC ERROR: Argument out of range! > > > > [0]PETSC ERROR: New nonzero at (2,18) caused a malloc! > > > > > > > > and > > > > > > > > a new incorrect matrix file error: > > > > > > > > [0]PETSC ERROR: Unexpected data in file! > > > > [0]PETSC ERROR: not matrix object! > > > > > > > > Maybe the order of calls is mixed up. This is the code snippet: > > > > > > > > if (rank ==0) > > > > { > > > > PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); > > > > > > > > CHKERRQ(MatSetSizes(A, 15, PETSC_DETERMINE, 20, 20)); > > > > CHKERRQ(MatSetType(A, MATMPIBAIJ)); > > > > CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); > > > > CHKERRQ( MatLoad(A,fd)); > > > > > CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); > > > > } > > > > > > > > else > > > > { > > > > PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); > > > > > > > > CHKERRQ( MatSetSizes(A, 5, PETSC_DETERMINE, 20, 20) ); > > > > CHKERRQ(MatSetType(A, MATMPIBAIJ)); > > > > CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); > > > > CHKERRQ(MatLoad(A,fd)); > > > > > CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); > > > > } > > > > > > > > Is there something I'm missing? > > > > > > > > Thanks, > > > > Steena > > > > > > > > > > > > > > > > > > > > > > > > On Tuesday, March 31, 2015 6:10 PM, Matthew Knepley < > knepley at gmail.com> wrote: > > > > > > > > > > > > On Tue, Mar 31, 2015 at 6:51 PM, Steena M wrote: > > > > Thanks Barry. I'm still getting the malloc error with NULL. Is there > a way to distribute the matrix without explicit preallocation? Different > matrices will be loaded during runtime and assigning preallocation > parameters would mean an additional preprocessing step. > > > > > > > > 1) MatSetOption(MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) > > > > > > > > 2) Note that this is never ever ever more efficient than making > another pass and preallocating > > > > > > > > Thanks, > > > > > > > > Matt > > > > -------------------------------------------- > > > > On Sun, 3/29/15, Barry Smith wrote: > > > > > > > > Subject: Re: [petsc-users] Unequal sparse matrix row distribution > for MPI MatMult > > > > To: "Steena M" > > > > Cc: "Matthew Knepley" , petsc-users at mcs.anl.gov > > > > Date: Sunday, March 29, 2015, 9:26 PM > > > > > > > > > > > > > On > > > > Mar 29, 2015, at 11:05 PM, Steena M > > > > wrote: > > > > > > > > > > Thanks > > > > Matt. I used PETSC_DETERMINE but I'm now getting an > > > > allocation-based error: > > > > > > > > > > [0]PETSC ERROR: --------------------- > > > > Error Message ------------------------------------ > > > > > [0]PETSC ERROR: Argument out of range! > > > > > [0]PETSC ERROR: New nonzero at (2,18) > > > > caused a malloc! > > > > > [0]PETSC ERROR: > > > > > ------------------------------------------------------------------------ > > > > > > > > > > I tried > > > > preallocating on each rank for the diagonal and off diagonal > > > > section of the matrix as the next step My current > > > > approximations for preallocation > > > > > > > > > > CHKERRQ( > > > > MatMPIBAIJSetPreallocation(A,1,5,PETSC_DEFAULT,5,PETSC_DEFAULT)); > > > > > > > > > > > > These > > > > arguments where you pass PETSC_DEFAULT are expecting a > > > > pointer not an integer. You can pass NULL in those > > > > locations. Though it is better to provide the correct > > > > preallocation rather than some defaults. > > > > > > > > Barry > > > > > > > > > > > > > > are throwing segmentation errors. > > > > > > > > > > [0]PETSC ERROR: > > > > Caught signal number 11 SEGV: Segmentation Violation, > > > > probably memory access out of range > > > > > > > > > > Any insights into what I'm doing > > > > wrong? > > > > > > > > > > Thanks, > > > > > Steena > > > > > > > > > > > > > > > > > > > > On Sun, 3/29/15, Matthew Knepley > > > > wrote: > > > > > > > > > > Subject: > > > > Re: [petsc-users] Unequal sparse matrix row distribution for > > > > MPI MatMult > > > > > To: "Steena M" > > > > > > > > > Cc: "Barry Smith" , > > > > petsc-users at mcs.anl.gov > > > > > Date: Sunday, March 29, 2015, 10:02 PM > > > > > > > > > > On Sun, Mar 29, 2015 > > > > at > > > > > 9:56 PM, Steena M > > > > > wrote: > > > > > Hi > > > > > Barry, > > > > > > > > > > > > > > > > > > > > I am trying to partition a 20 row and 20 > > > > col sparse matrix > > > > > between two procs > > > > such that proc 0 has 15 rows and 20 cols > > > > > and proc 1 has 5 rows and 20 cols. The > > > > code snippet: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A)); > > > > // > > > > > at runtime: -matload_block_size 1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > if (rank ==0) > > > > > > > > > > { > > > > > > > > > > > > > > CHKERRQ( MatSetSizes(A, 15, 20, 20, > > > > > 20) ); //rank 0 gets 75% of the rows > > > > > > > > > > > > > > CHKERRQ( MatSetType(A, MATMPIBAIJ) > > > > > ); > > > > > > > > > > CHKERRQ( > > > > MatLoad(A,fd) ); > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > else > > > > > > > > > > > > > > { > > > > > > > > > > > > > > CHKERRQ( MatSetSizes(A, 5, > > > > 20, 20, > > > > > 20) ); //rank 1 gets 25% of the > > > > rows > > > > > > > > > > > > > > CHKERRQ( MatSetType(A, MATMPIBAIJ) > > > > > ); > > > > > > > > > > CHKERRQ( > > > > MatLoad(A,fd) ); > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > This throws the following error (probably > > > > from psplit.c): > > > > > > > > > > > > > > [1]PETSC ERROR: --------------------- Error Message > > > > > ------------------------------------ > > > > > > > > > > [1]PETSC ERROR: > > > > Nonconforming object sizes! > > > > > > > > > > [1]PETSC ERROR: Sum of local lengths 40 > > > > does not equal > > > > > global length 20, my > > > > local length 20 > > > > > > > > > > likely a call to > > > > VecSetSizes() or MatSetSizes() is > > > > > > > > > wrong. > > > > > > > > > > See > > > > http://www.mcs.anl.gov/petsc/documentation/faq.html#split! > > > > > > > > > > > > > > > > > > > > This error printout > > > > doesn't quite make sense to me. > > > > > > > > > I'm trying to specify a total matrix size of 20x20... > > > > I > > > > > haven't yet figured out where the > > > > '40' comes > > > > > from in the error > > > > message. > > > > > > > > > > > > > > > > > > > > Any thoughts on what > > > > might be going wrong? > > > > > > > > > > Its the column specification. Just > > > > > use PETSC_DETERMINE for the local columns > > > > since all our > > > > > sparse matrixformats are > > > > row divisions > > > > > anyway. > > > > > > > > > > Thanks, > > > > > > > > > > Matt > > > > > Thanks in advance, > > > > > > > > > > > > > > Steena > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------------------------------------- > > > > > > > > > > On Sun, 3/22/15, > > > > Barry Smith > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Subject: Re: [petsc-users] Unequal > > > > sparse matrix row > > > > > distribution for MPI > > > > MatMult > > > > > > > > > > To: > > > > "Steena M" > > > > > > > > > > Cc: petsc-users at mcs.anl.gov > > > > > > > > > > Date: Sunday, > > > > March 22, 2015, 3:58 PM > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Steena, > > > > > > > > > > > > > > > > > > > > I am > > > > > > > > > > > > > > a little unsure of your question. > > > > > > > > > > > > > > > > > > > > > > > > 1) > > > > you can create a MPIBAIJ > > > > > > > > > > matrix with any distribution of block > > > > rows per process > > > > > you > > > > > > > > > > want, just set the > > > > local row size for each process to > > > > > > > > > be > > > > > > > > > > what you > > > > like. Use MatCreateVecs() to get > > > > > > > > > correspondingly > > > > > > > > > > > > > > laid out vectors. > > > > > > > > > > > > > > > > > > > > > > > > > > > > or 2) if you have a MPIBAIJ > > > > > > > > > > matrix with > > > > "equal" row layout and you want a > > > > > new > > > > > > > > > > one with uneven row layout you can > > > > simply use > > > > > > > > > > > > > > MatGetSubMatrix() to create that new matrix. > > > > > > > > > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > Unless you have > > > > another reason to have the > > > > > > > > > > matrix with an equal number row layout I > > > > would just > > > > > generate > > > > > > > > > > the matrix with > > > > the layout you want. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 22, 2015, at 5:50 PM, Steena > > > > M > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > I need to > > > > distribute > > > > > > > > > > a > > > > sparse matrix such that each proc owns an unequal > > > > > number > > > > > > > > > > of blocked rows before I proceed with > > > > MPI MatMult. My > > > > > > > > > > > > > > initial thoughts on doing this: > > > > > > > > > > > > > > > > > > > > > > 1) Use MatGetSubMatrices() on the > > > > test > > > > > > > > > > MATMPIBAIJ > > > > matrix to produce a new matrix where each > > > > > proc > > > > > > > > > > has an unequal number of rows. > > > > > > > > > > > > > > > > > > > > > > 2) Provide > > > > scatter context for vector X > > > > > > > > > > (for MatMult )using IS iscol from > > > > MatGetSubMatrices() > > > > > while > > > > > > > > > > creating the > > > > vector X. > > > > > > > > > > > > > > > > > > > > > > > > > > 3) > > > > Call MatMult() > > > > > > > > > > > > > > > > > > > > > > > > > > Will > > > > MatMult_MPIBAIJ continue to scatter > > > > > > > > > > this matrix and vector such that each > > > > proc will own an > > > > > equal > > > > > > > > > > number of matrix > > > > rows and corresponding diagonal vector > > > > > > > > > > > > > > elements? Should I write my own > > > > MPIMatMult function to > > > > > > > > > > retain my redistribution of the matrix > > > > and vector? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks in > > > > > > > > > > > > > > advance, > > > > > > > > > > > > > > > Steena > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > What most > > > > experimenters > > > > > take for granted before > > > > they begin their experiments is > > > > > > > > > infinitely more interesting than any results to which > > > > their > > > > > experiments lead. > > > > > -- Norbert > > > > > Wiener > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > > > -- Norbert Wiener > > > > > > > > > > > > > > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From stali at geology.wisc.edu Thu Apr 9 14:31:36 2015 From: stali at geology.wisc.edu (Tabrez Ali) Date: Thu, 09 Apr 2015 14:31:36 -0500 Subject: [petsc-users] finclude (web) vs petsc-finclude (git) In-Reply-To: <5526BD2F.4000109@gmail.com> References: <5526BD2F.4000109@gmail.com> Message-ID: <5526D398.90703@geology.wisc.edu> You can use your own petsc.h #include #if (PETSC_VERSION_MAJOR==3 && PETSC_VERSION_MINOR<=5) #include #else #include #endif which you can then include in your code as ... program foo #include "petsc.h" ... end program foo Tabrez On 04/09/2015 12:55 PM, Juris Vencels wrote: > Hi, > > I am using PETSc Fortran interface and FindPETSc.cmake script to find > PETSc. > https://github.com/jedbrown/cmake-modules/blob/master/FindPETSc.cmake > > PETSc from WEB has folder include folder: > include/finclude > > while PETSc from git has folder: > include/petsc-finclude > > Is there a way to make it compatible between different versions? > > Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Apr 9 14:39:36 2015 From: jed at jedbrown.org (Jed Brown) Date: Thu, 09 Apr 2015 13:39:36 -0600 Subject: [petsc-users] finclude (web) vs petsc-finclude (git) In-Reply-To: <5526D398.90703@geology.wisc.edu> References: <5526BD2F.4000109@gmail.com> <5526D398.90703@geology.wisc.edu> Message-ID: <87fv89hz0n.fsf@jedbrown.org> Tabrez Ali writes: > You can use your own petsc.h > > #include > #if (PETSC_VERSION_MAJOR==3 && PETSC_VERSION_MINOR<=5) It's more readable as #if PETSC_VERSION_LT(3,6,0) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From lvella at gmail.com Thu Apr 9 16:30:51 2015 From: lvella at gmail.com (Lucas Clemente Vella) Date: Thu, 9 Apr 2015 18:30:51 -0300 Subject: [petsc-users] Using OpenMP threads with PETSc Message-ID: As I understand, from reading a paper form Weiland, Mitchell, Parsons, Gorman and Kramer, entitled "Mixed-mode implementation of PETSc for scalable linear algebra on multi-core processors", they have implemented thread level parallelism inside Mat and Vec operations of PETSc, and it is available on master branch of the PETSc development version. I have a small benchmark program that loads a matrix into PETSc and solves it with MPI. I was hoping to be able to easily use the PETSc OpenMP threaded parallelism, preferably without having to change how I use PETSc from my program, but I was unable to do so. I have downloaded and built master branch from GIT with OpenMP enabled, configured like this (and it was a pain to have the have it to complete the configure with the right libraries, compilers and flags): ./configure --PETSC_DIR=/home/lvella/src/petsc --with-clanguage=C --FFLAGS="-Ofast -march=ivybridge -mmmx -mno-3dnow -msse -msse2 -msse3 -mssse3 -mno-sse4a -mcx16 -msahf -mno-movbe -maes -mno-sha -mpclmul -mpopcnt -mno-abm -mno-lwp -mno-fma -mno-fma4 -mno-xop -mno-bmi -mno-bmi2 -mno-tbm -mavx -mno-avx2 -msse4.2 -msse4.1 -mno-lzcnt -mno-rtm -mno-hle -mrdrnd -mf16c -mfsgsbase -mno-rdseed -mno-prfchw -mno-adx -mfxsr -mxsave -mxsaveopt -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf -mno-prefetchwt1 --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=25600 -mtune=ivybridge -I/opt/sw/openmpi/1.8.4-gcc4.9.1/include/" --CFLAGS="-Ofast -march=ivybridge -mmmx -mno-3dnow -msse -msse2 -msse3 -mssse3 -mno-sse4a -mcx16 -msahf -mno-movbe -maes -mno-sha -mpclmul -mpopcnt -mno-abm -mno-lwp -mno-fma -mno-fma4 -mno-xop -mno-bmi -mno-bmi2 -mno-tbm -mavx -mno-avx2 -msse4.2 -msse4.1 -mno-lzcnt -mno-rtm -mno-hle -mrdrnd -mf16c -mfsgsbase -mno-rdseed -mno-prfchw -mno-adx -mfxsr -mxsave -mxsaveopt -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf -mno-prefetchwt1 --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=25600 -mtune=ivybridge" --CXXFLAGS="-Ofast -march=ivybridge -mmmx -mno-3dnow -msse -msse2 -msse3 -mssse3 -mno-sse4a -mcx16 -msahf -mno-movbe -maes -mno-sha -mpclmul -mpopcnt -mno-abm -mno-lwp -mno-fma -mno-fma4 -mno-xop -mno-bmi -mno-bmi2 -mno-tbm -mavx -mno-avx2 -msse4.2 -msse4.1 -mno-lzcnt -mno-rtm -mno-hle -mrdrnd -mf16c -mfsgsbase -mno-rdseed -mno-prfchw -mno-adx -mfxsr -mxsave -mxsaveopt -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf -mno-prefetchwt1 --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=25600 -mtune=ivybridge" --with-etags=0 --with-visibility=1 --with-hypre=1 --with-hypre-include=/home/lvella/src/hypre-2.10.0b/src/hypre/include/ --with-hypre-lib=/home/lvella/src/hypre-2.10.0b/src/hypre/lib/libHYPRE.a --with-openmp=1 --with-pthread=1 --download-fblaslapack -with-cc=/opt/sw/gcc/4.9/bin/gcc-4.9 -with-cxx=0 -with-fc=/opt/sw/gcc/4.9/bin/gfortran-4.9 --with-mpi-compilers=0 --known-mpi-shared-libraries=1 --with-mpiuni-fortran-binding=0 --FC_LINKER_FLAGS=-lmpi_mpifh --with-debugging=0 --with-pthreadclasses=1 --with-threadcomm=1 It is hard to find in there, but all these relevant flags are there: --with-openmp=1 --with-pthread=1 --with-pthreadclasses=1 --with-threadcomm=1 For the later two, I was not too certain of why they were needed, but I couldn't use pthreads without them. Now, to run my program, I had several attempts (with just two cores to see if everything was right). The first one: export OMP_NUM_THREADS=2 ./solve-only -ksp_type bcgs -pc_type asm -pc_asm_overlap 12 -pc_asm_type restrict -sub_pc_type ilu -sub_pc_factor_levels 1 -sub_ksp_type preonly With this first attempt, the program runs, but there is no indication of any use of threads. The threads are there, but CPU usage on top is never above 100%. The second attempt was: export OMP_NUM_THREADS=2 ./solve-only -ksp_type bcgs -pc_type asm -pc_asm_overlap 12 -pc_asm_type restrict -sub_pc_type ilu -sub_pc_factor_levels 1 -sub_ksp_type preonly -threadcomm_nthreads $OMP_NUM_THREADS Now, the program is killed with the message: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: Cannot have more than 1 thread for the nonthread communicator,threads requested = 2 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2611-g88a78ad GIT Date: 2015-04-08 11:48:47 -0500 [0]PETSC ERROR: ../../solve-only on a arch-linux2-c-opt named r1i0n0 by lvella Thu Apr 9 18:13:18 2015 [0]PETSC ERROR: Configure options [...] [0]PETSC ERROR: #1 PetscThreadCommCreate_NoThread() line 10 in /home/lvella/src/petsc/src/sys/threadcomm/impls/nothread/nothread.c [0]PETSC ERROR: #2 PetscThreadCommSetType() line 507 in /home/lvella/src/petsc/src/sys/threadcomm/interface/threadcomm.c [0]PETSC ERROR: #3 PetscThreadCommWorldInitialize() line 1242 in /home/lvella/src/petsc/src/sys/threadcomm/interface/threadcomm.c [0]PETSC ERROR: #4 PetscGetThreadCommWorld() line 82 in /home/lvella/src/petsc/src/sys/threadcomm/interface/threadcomm.c [0]PETSC ERROR: #5 PetscCommGetThreadComm() line 117 in /home/lvella/src/petsc/src/sys/threadcomm/interface/threadcomm.c [0]PETSC ERROR: #6 PetscCommDuplicate() line 195 in /home/lvella/src/petsc/src/sys/objects/tagm.c [0]PETSC ERROR: #7 PetscHeaderCreate_Private() line 60 in /home/lvella/src/petsc/src/sys/objects/inherit.c [0]PETSC ERROR: #8 MatCreate() line 84 in /home/lvella/src/petsc/src/mat/utils/gcreate.c [0]PETSC ERROR: ------------------------------------------------------------------------ [...] For the next attempt, I've added the option "-threadcomm_type openmp", as the page http://www.mcs.anl.gov/petsc/features/threads.html says... export OMP_NUM_THREADS=2 ./solve-only -ksp_type bcgs -pc_type asm -pc_asm_overlap 12 -pc_asm_type restrict -sub_pc_type ilu -sub_pc_factor_levels 1 -sub_ksp_type preonly -threadcomm_nthreads $OMP_NUM_THREADS -threadcomm_type openmp But the program dies with this error: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing package: http://www.mcs.anl.gov/petsc/documentation/installation.html#external [0]PETSC ERROR: Unable to find requested PetscThreadComm type openmp [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2611-g88a78ad GIT Date: 2015-04-08 11:48:47 -0500 [0]PETSC ERROR: ../../solve-only on a arch-linux2-c-opt named r1i0n0 by lvella Thu Apr 9 18:16:26 2015 [0]PETSC ERROR: Configure options [...] [0]PETSC ERROR: #1 PetscThreadCommSetType() line 506 in /home/lvella/src/petsc/src/sys/threadcomm/interface/threadcomm.c [0]PETSC ERROR: #2 PetscThreadCommWorldInitialize() line 1242 in /home/lvella/src/petsc/src/sys/threadcomm/interface/threadcomm.c [0]PETSC ERROR: #3 PetscGetThreadCommWorld() line 82 in /home/lvella/src/petsc/src/sys/threadcomm/interface/threadcomm.c [0]PETSC ERROR: #4 PetscCommGetThreadComm() line 117 in /home/lvella/src/petsc/src/sys/threadcomm/interface/threadcomm.c [0]PETSC ERROR: #5 PetscCommDuplicate() line 195 in /home/lvella/src/petsc/src/sys/objects/tagm.c [0]PETSC ERROR: #6 PetscHeaderCreate_Private() line 60 in /home/lvella/src/petsc/src/sys/objects/inherit.c [0]PETSC ERROR: #7 MatCreate() line 84 in /home/lvella/src/petsc/src/mat/utils/gcreate.c [0]PETSC ERROR: ------------------------------------------------------------------------ [...] Lastly, I have tried with pthread: export OMP_NUM_THREADS=2 ./solve-only -ksp_type bcgs -pc_type asm -pc_asm_overlap 12 -pc_asm_type restrict -sub_pc_type ilu -sub_pc_factor_levels 1 -sub_ksp_type preonly -threadcomm_nthreads $OMP_NUM_THREADS -threadcomm_type pthread Now the program runs, but the CPU usage is never higher than 133%, which is much lower than the expected (if I solve the same matrix with Hypre + OpenMP, CPU usage peaks 200% most of the time). So, what threading model gives best results in PETSc, OpenMP or pthreads? And what I need to do use OpenMP? All my attempts were single-threaded, but I will need to work with MPI, is it straightforward to spawn many MPI processes (i.e. just put mpirun before the command?) Lastly, how to prevent PETSc to mess with CPU affinities, because I implement affinity control myself inside my program. -- Lucas Clemente Vella lvella at gmail.com From jed at jedbrown.org Thu Apr 9 16:50:21 2015 From: jed at jedbrown.org (Jed Brown) Date: Thu, 09 Apr 2015 15:50:21 -0600 Subject: [petsc-users] Using OpenMP threads with PETSc In-Reply-To: References: Message-ID: <876195geea.fsf@jedbrown.org> Lucas Clemente Vella writes: > For the next attempt, I've added the option "-threadcomm_type openmp", > as the page http://www.mcs.anl.gov/petsc/features/threads.html says... Need to configure --with-openmpclasses with petsc-dev to use -threadcomm_type openmp. > Now the program runs, but the CPU usage is never higher than 133%, > which is much lower than the expected (if I solve the same matrix with > Hypre + OpenMP, CPU usage peaks 200% most of the time). Before getting carried away, run with flat MPI (no threads ever) and compare the time. In many ways, this mode has fewer synchronizations and thus should perform better. That's what we see on most machines with HPGMG and lots of other people have found similar behavior. If it's slower, send -log_summary for both versions. Not all operations are currently threaded. > So, what threading model gives best results in PETSc, OpenMP or > pthreads? And what I need to do use OpenMP? All my attempts were > single-threaded, You mean single-process. > but I will need to work with MPI, is it straightforward to spawn many > MPI processes (i.e. just put mpirun before the command?) Yes. > Lastly, how to prevent PETSc to mess with CPU affinities, because I > implement affinity control myself inside my program. Affinity fighting is a recurring disaster for library interoperability with threads. I think what is currently done in PETSc is actively harmful and should be taken out. But I think not using threads is a much better solution for almost everyone. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From lvella at gmail.com Thu Apr 9 17:07:39 2015 From: lvella at gmail.com (Lucas Clemente Vella) Date: Thu, 9 Apr 2015 19:07:39 -0300 Subject: [petsc-users] Using OpenMP threads with PETSc In-Reply-To: <876195geea.fsf@jedbrown.org> References: <876195geea.fsf@jedbrown.org> Message-ID: 2015-04-09 18:50 GMT-03:00 Jed Brown : > Lucas Clemente Vella writes: > >> For the next attempt, I've added the option "-threadcomm_type openmp", >> as the page http://www.mcs.anl.gov/petsc/features/threads.html says... > > Need to configure --with-openmpclasses with petsc-dev to use > -threadcomm_type openmp. > >> Now the program runs, but the CPU usage is never higher than 133%, >> which is much lower than the expected (if I solve the same matrix with >> Hypre + OpenMP, CPU usage peaks 200% most of the time). > > Before getting carried away, run with flat MPI (no threads ever) and > compare the time. In many ways, this mode has fewer synchronizations > and thus should perform better. That's what we see on most machines > with HPGMG and lots of other people have found similar behavior. > > If it's slower, send -log_summary for both versions. Not all operations > are currently threaded. > >> So, what threading model gives best results in PETSc, OpenMP or >> pthreads? And what I need to do use OpenMP? All my attempts were >> single-threaded, > > You mean single-process. > >> but I will need to work with MPI, is it straightforward to spawn many >> MPI processes (i.e. just put mpirun before the command?) > > Yes. > >> Lastly, how to prevent PETSc to mess with CPU affinities, because I >> implement affinity control myself inside my program. > > Affinity fighting is a recurring disaster for library interoperability > with threads. I think what is currently done in PETSc is actively > harmful and should be taken out. But I think not using threads is a > much better solution for almost everyone. I just found that the 133% issue was because of affinity clash with other batch jobs running in the cluster's node. If I manually change it to allow any CPU (let the kernel pick), it runs at 200%. I'll comment out all CPU affinity changing calls from the code, and rebuild PETSc with --with-openmpclasses. I suspect the optimal setup is to have one process for each NUMA node, one thread for logical core, and affinity lock each process (all threads) to the cores corresponding to its NUMA node. The mentioned article showed improvements on almost all cases they compared OpenMP threads with MPI processes. -- Lucas Clemente Vella lvella at gmail.com From jed at jedbrown.org Thu Apr 9 17:33:11 2015 From: jed at jedbrown.org (Jed Brown) Date: Thu, 09 Apr 2015 16:33:11 -0600 Subject: [petsc-users] Using OpenMP threads with PETSc In-Reply-To: References: <876195geea.fsf@jedbrown.org> Message-ID: <87wq1lexug.fsf@jedbrown.org> Lucas Clemente Vella writes: > I suspect the optimal setup is to have one process for each NUMA node, > one thread for logical core, Why? Are you packing buffers in parallel (extra openmp overhead) or serial (Amdahl's law limitations)? The NIC most likely supports as many hardware contexts as cores, so there is a shorter critical path when using flat MPI. > and affinity lock each process (all threads) to the cores > corresponding to its NUMA node. The mentioned article showed > improvements on almost all cases they compared OpenMP threads with MPI > processes. This varies a lot by machine, network hardware and tuning, software stack, and problem sizes/configuration. It is not universal and I can show you lots of examples where flat MPI is the way to go. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From lvella at gmail.com Thu Apr 9 17:52:02 2015 From: lvella at gmail.com (Lucas Clemente Vella) Date: Thu, 9 Apr 2015 19:52:02 -0300 Subject: [petsc-users] Using OpenMP threads with PETSc In-Reply-To: <87wq1lexug.fsf@jedbrown.org> References: <876195geea.fsf@jedbrown.org> <87wq1lexug.fsf@jedbrown.org> Message-ID: 2015-04-09 19:33 GMT-03:00 Jed Brown : > Lucas Clemente Vella writes: >> I suspect the optimal setup is to have one process for each NUMA node, >> one thread for logical core, > > Why? Are you packing buffers in parallel (extra openmp overhead) or > serial (Amdahl's law limitations)? The NIC most likely supports as many > hardware contexts as cores, so there is a shorter critical path when > using flat MPI. Mainly because there is no need to copy buffers between processes (with MPI calls) when it is already fast to use them shared, inside the same NUMA node. As for processes running in other NUMA nodes, the extra copy incurred by MPI pays off with faster memory access. I believed MPI implementations didn't use the NIC to communicate when inside the same computer node, but instead used shared memory. By the way, there is already a precompiler macro to disable CPU affinity: PETSC_HAVE_SCHED_CPU_SET_T. How to disable it at configure? -- Lucas Clemente Vella lvella at gmail.com From jed at jedbrown.org Thu Apr 9 18:01:14 2015 From: jed at jedbrown.org (Jed Brown) Date: Thu, 09 Apr 2015 17:01:14 -0600 Subject: [petsc-users] Using OpenMP threads with PETSc In-Reply-To: References: <876195geea.fsf@jedbrown.org> <87wq1lexug.fsf@jedbrown.org> Message-ID: <87oamwgb45.fsf@jedbrown.org> Lucas Clemente Vella writes: > Mainly because there is no need to copy buffers between processes > (with MPI calls) when it is already fast to use them shared, inside > the same NUMA node. What about cache efficiency when your working set is not contiguous or is poorly shaped? > As for processes running in other NUMA nodes, the extra copy incurred > by MPI pays off with faster memory access. I believed MPI > implementations didn't use the NIC to communicate when inside the same > computer node, but instead used shared memory. Yes. > By the way, there is already a precompiler macro to disable CPU > affinity: PETSC_HAVE_SCHED_CPU_SET_T. > How to disable it at configure? I don't think there is an option, but it should be made a run-time rather than configure-time option anyway. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From lvella at gmail.com Thu Apr 9 18:23:06 2015 From: lvella at gmail.com (Lucas Clemente Vella) Date: Thu, 9 Apr 2015 20:23:06 -0300 Subject: [petsc-users] Using OpenMP threads with PETSc In-Reply-To: <87oamwgb45.fsf@jedbrown.org> References: <876195geea.fsf@jedbrown.org> <87wq1lexug.fsf@jedbrown.org> <87oamwgb45.fsf@jedbrown.org> Message-ID: >> Mainly because there is no need to copy buffers between processes >> (with MPI calls) when it is already fast to use them shared, inside >> the same NUMA node. > > What about cache efficiency when your working set is not contiguous or > is poorly shaped? Will be no worse than separated MPI processes competing for the same cache slot. At least with threads there is a chance that different tasks will hit the same cached memory. I do believe there should be a smart way to control and optimize thread work proximity on OMP for loops, i.e.: #pragma omp parallel for for(size_t i = 0; i < size; ++size) { // something not dependent on previous steps } Each two threads working on ranges closely together should run on the two hyperthreads of the same core, to maximize cache reuse. Because this usage pattern of OpenMP, it seems to me that it is already unlikely that two threads will be working to far off each other, but if I wanted this level of control, I should now be hacking some OpenMP implementation and/or the kernel. On the other hand, pthread interface favors more loosely coupled tasks, that may yield worse cache reuse, but I confess I didn't take the time to look inside PETSc how each of the threading libraries was used. -- Lucas Clemente Vella lvella at gmail.com From stm8086 at yahoo.com Thu Apr 9 18:49:17 2015 From: stm8086 at yahoo.com (Steena M) Date: Thu, 9 Apr 2015 23:49:17 +0000 (UTC) Subject: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult In-Reply-To: References: Message-ID: <1014905468.3172094.1428623357646.JavaMail.yahoo@mail.yahoo.com> Thanks, Matt. From *View(), vectors x and y are being created and initialized correctly and their layout matches that of the matrix. ?There is a segmentation fault happening on one of the ranks, after all elements of y have been computed. The error (pasted below) is printed and visible only when using MatView(A,0) ?Besides changing the block size, local row sizes, and setting up for MatMult() ?I have not changed the code in ex190.c? Process [0]12..Process [1]20..srun: error: sierra12: tasks 0-1: Segmentation fault (core dumped) Thanks in advance,Steena On Thursday, April 9, 2015 12:15 PM, Matthew Knepley wrote: On Thu, Apr 9, 2015 at 2:12 PM, Steena M wrote: Thanks, Barry! I patched the master, modified src/mat/examples/tests/ex190.c to suit my data, and the fix works. I need to execute MatMult and I am assigning ?vectors on each rank using MatCreateVecs(A, &x, &y). However, I don't think multiplication is happening at all. (I inserted a few printf statements inside MatMult_MPIBAIJ and inside MatMult_SeqBAIJ1 to check). ? VecSum(&ysum) is always zero. Maybe I'm assigning data incorrectly?? Code snippet in ex190.c after partitioning the matrix unequally ? ierr = MatLoad(A,fd);CHKERRQ(ierr);? ierr = MatCreateVecs(A, &x, &y);CHKERRQ(ierr);??? ierr = VecSet(x,one);CHKERRQ(ierr); //PetscScalar one = 1.0;? ierr = VecSet(y,zero); CHKERRQ(ierr);?//PetscScalar zero = 0.0; MatView(A, 0) gives you the matrix, so you can see what Ax should be.? ? ierr = MatMult(A,x,y); CHKERRQ(ierr); VecView(y, 0) gives you the output ? Matt? ? ierr = ?VecSum(y,&ysum);CHKERRQ(ierr); //ysum is always zero.? Data is a 20x20 matrix and block size is 1. Any thoughts? Thanks,Steena???.? ? On Friday, April 3, 2015 4:17 PM, Barry Smith wrote: ? Steena, ? Sorry for all the delays. Our code was just totally wrong for a user provided decomposition for two reasons 1) the logic in MatLoad_MPIXAIJ() was too convoluted to work in all cases and 2) we didn't have a single test case. ? I have attached a patch that fixes it for petsc-3.5.3 .? The fix is also in the branch barry/fix-matload-uneven-rows and I'll put it into next then after testing it will got into maint and master and the next PETSc patch release. ? Please let us know if the patch doesn't work for you ? Thanks for your patience, ? Barry > On Apr 1, 2015, at 3:10 PM, Steena M wrote: > > Thanks Barry. Attached is the driver program, the binary mat file, and the corresponding mtx file. > > Runtime command used: > > sierra324 at monteiro:time srun -n 2 -ppdebug ./petsc-mpibaij-unequalrows -fin trefethen.dat -matload_block_size 1 > > > > > On Wednesday, April 1, 2015 12:28 PM, Barry Smith wrote: > > > >? Send a data file you generated and your reader program and we'll debug it. > >? Barry > > > On Apr 1, 2015, at 2:18 PM, Steena M wrote: > > > > Thanks Barry. I removed the Preallocation calls. It is still complaining about the malloc and incorrect data in the matrix file. I generate binary matrix files using PETSc's pythonscript to loop through a set of UFL sparse matrices. For this use case: > > > > mtx_mat = scipy.io.mmread('trefethen.mtx') > > PetscBinaryIO.PetscBinaryIO().writeMatSciPy(open('trefnew.dat','w'), mtx_mat) > > > > > >? > > > > > > > > On Tuesday, March 31, 2015 9:15 PM, Barry Smith wrote: > > > > > > > >? You should not need to call any preallocation routines when using MatLoad() > > > > > >? How did you generate the file? Are you sure it has the correct information for the matrix? > > > >? Barry > > > > > > > > > On Mar 31, 2015, at 11:05 PM, Steena M wrote: > > > > > > Thanks Matt. I'm still getting the malloc error > > > > > > [0]PETSC ERROR: Argument out of range! > > > [0]PETSC ERROR: New nonzero at (2,18) caused a malloc! > > > > > > and > > > > > > a new incorrect matrix file error: > > > > > > [0]PETSC ERROR: Unexpected data in file! > > > [0]PETSC ERROR: not matrix object! > > > > > > Maybe the order of calls is mixed up. This is the code snippet: > > > > > >? ? if (rank ==0) > > >? ? { > > >? ? PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); > > >? ? ? > > >? ? ? ? CHKERRQ(MatSetSizes(A, 15, PETSC_DETERMINE, 20, 20)); > > >? ? ? ? CHKERRQ(MatSetType(A, MATMPIBAIJ)); > > >? ? ? ? CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); > > >? ? ? ? CHKERRQ( MatLoad(A,fd)); > > >? ? CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); > > >? ? } > > >? ? > > >? ? else > > >? ? { > > >? ? ? ? PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); > > >? ? ? ? > > >? ? ? ? CHKERRQ( MatSetSizes(A, 5, PETSC_DETERMINE, 20, 20) ); > > >? ? ? ? CHKERRQ(MatSetType(A, MATMPIBAIJ)); > > >? ? ? ? CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); > > >? ? ? ? CHKERRQ(MatLoad(A,fd)); > > >? ? CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); > > >? ? ? ? } > > > > > > Is there something I'm missing? > > > > > > Thanks, > > > Steena > > >? ? > > > > > > > > > > > > > > > On Tuesday, March 31, 2015 6:10 PM, Matthew Knepley wrote: > > > > > > > > > On Tue, Mar 31, 2015 at 6:51 PM, Steena M wrote: > > > Thanks Barry. I'm still getting the malloc error with NULL. Is there a way to distribute the matrix without explicit preallocation? Different matrices will be loaded during runtime and assigning preallocation parameters would mean an additional preprocessing step. > > > > > > 1) MatSetOption(MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) > > >? > > > 2) Note that this is never ever ever more efficient than making another pass and preallocating > > > > > >? Thanks, > > > > > >? ? ? Matt > > > -------------------------------------------- > > > On Sun, 3/29/15, Barry Smith wrote: > > > > > >? Subject: Re: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult > > >? To: "Steena M" > > >? Cc: "Matthew Knepley" , petsc-users at mcs.anl.gov > > >? Date: Sunday, March 29, 2015, 9:26 PM > > > > > > > > >? > On > > >? Mar 29, 2015, at 11:05 PM, Steena M > > >? wrote: > > >? > > > >? > Thanks > > >? Matt. I used PETSC_DETERMINE but I'm now getting an > > >? allocation-based error: > > >? > > > >? > [0]PETSC ERROR: --------------------- > > >? Error Message ------------------------------------ > > >? > [0]PETSC ERROR: Argument out of range! > > >? > [0]PETSC ERROR: New nonzero at (2,18) > > >? caused a malloc! > > >? > [0]PETSC ERROR: > > >? ------------------------------------------------------------------------ > > >? > > > >? > I tried > > >? preallocating on each rank for the diagonal and off diagonal > > >? section of the matrix as the next step? My current > > >? approximations for preallocation > > >? > > > >? > CHKERRQ( > > >? MatMPIBAIJSetPreallocation(A,1,5,PETSC_DEFAULT,5,PETSC_DEFAULT)); > > > > > > > > >? ? These > > >? arguments where you pass PETSC_DEFAULT are expecting a > > >? pointer not an integer. You can pass NULL in those > > >? locations. Though it is better to provide the correct > > >? preallocation rather than some defaults. > > > > > >? ? Barry > > > > > >? > > > >? > are throwing segmentation errors. > > >? > > > >? > [0]PETSC ERROR: > > >? Caught signal number 11 SEGV: Segmentation Violation, > > >? probably memory access out of range > > >? > > > >? > Any insights into what I'm doing > > >? wrong? > > >? > > > >? > Thanks, > > >? > Steena > > >? > > > >? > > > >? > > > >? > On Sun, 3/29/15, Matthew Knepley > > >? wrote: > > >? > > > >? > Subject: > > >? Re: [petsc-users] Unequal sparse matrix row distribution for > > >? MPI MatMult > > >? > To: "Steena M" > > >? > > >? > Cc: "Barry Smith" , > > >? petsc-users at mcs.anl.gov > > >? > Date: Sunday, March 29, 2015, 10:02 PM > > >? > > > >? > On Sun, Mar 29, 2015 > > >? at > > >? > 9:56 PM, Steena M > > >? > wrote: > > >? > Hi > > >? > Barry, > > >? > > > >? > > > >? > > > >? > I am trying to partition a 20 row and 20 > > >? col sparse matrix > > >? > between two procs > > >? such that proc 0 has 15 rows and 20 cols > > >? > and proc 1 has 5 rows and 20 cols. The > > >? code snippet: > > >? > > > >? > > > >? > > > >? > > > >? > > > >? > > > >? > > > >? >? ? > > >? ? CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A)); > > >? // > > >? > at runtime: -matload_block_size 1 > > >? > > > >? > > > >? > > > >? > > > >? > > > >? >? ? > > >? ? if (rank ==0) > > >? > > > >? >? ? ? ? { > > >? > > > >? >? ? ? ? ? ? > > >? ? CHKERRQ( MatSetSizes(A, 15, 20, 20, > > >? > 20) ); //rank 0 gets 75% of the rows > > >? > > > >? >? ? ? ? ? ? > > >? ? CHKERRQ( MatSetType(A, MATMPIBAIJ) > > >? > ); > > >? > > > >? >? ? ? ? ? ? ? ? CHKERRQ( > > >? MatLoad(A,fd) ); > > >? > > > >? >? ? ? ? ? } > > >? > > > >? > > > >? > > > >? >? ? ? ? else > > >? > > > >? >? ? > > >? ? { > > >? > > > >? > > > >? ? ? ? ? ? ? ? CHKERRQ( MatSetSizes(A, 5, > > >? 20, 20, > > >? > 20) ); //rank 1 gets 25% of the > > >? rows > > >? > > > >? >? ? ? > > >? ? ? ? ? CHKERRQ( MatSetType(A, MATMPIBAIJ) > > >? > ); > > >? > > > >? >? ? ? ? ? ? ? ? CHKERRQ( > > >? MatLoad(A,fd) ); > > >? > > > >? >? ? ? } > > >? > > > > > > >? > > > >? > > > >? > This throws the following error (probably > > >? from psplit.c): > > >? > > > >? > > > >? [1]PETSC ERROR: --------------------- Error Message > > >? > ------------------------------------ > > >? > > > >? > [1]PETSC ERROR: > > >? Nonconforming object sizes! > > >? > > > >? > [1]PETSC ERROR: Sum of local lengths 40 > > >? does not equal > > >? > global length 20, my > > >? local length 20 > > >? > > > >? >? likely a call to > > >? VecSetSizes() or MatSetSizes() is > > >? > > > >? wrong. > > >? > > > >? > See > > >? http://www.mcs.anl.gov/petsc/documentation/faq.html#split! > > >? > > > >? > > > >? > > > >? > This error printout > > >? doesn't quite make sense to me. > > >? > > > >? I'm trying to specify a total matrix size of 20x20... > > >? I > > >? > haven't yet figured out where the > > >? '40' comes > > >? > from in the error > > >? message. > > >? > > > >? > > > >? > > > >? > Any thoughts on what > > >? might be going wrong? > > >? > > > >? > Its the column specification. Just > > >? > use PETSC_DETERMINE for the local columns > > >? since all our > > >? > sparse matrixformats are > > >? row divisions > > >? > anyway. > > >? > > > >? > Thanks, > > >? >? > > >? >? Matt > > >? > Thanks in advance, > > >? > > > > > > >? > Steena > > >? > > > >? > > > >? > > > >? > > > >? > > > >? > > > >? > > > >? > > > >? -------------------------------------------- > > >? > > > >? > On Sun, 3/22/15, > > >? Barry Smith > > >? > wrote: > > >? > > > >? > > > >? > > > >? >? Subject: Re: [petsc-users] Unequal > > >? sparse matrix row > > >? > distribution for MPI > > >? MatMult > > >? > > > >? >? To: > > >? "Steena M" > > >? > > > >? >? Cc: petsc-users at mcs.anl.gov > > >? > > > >? >? Date: Sunday, > > >? March 22, 2015, 3:58 PM > > >? > > > >? > > > >? > > > >? > > > >? > > > >? >? > > >? > > > >? >? Steena, > > >? > > > >? > > > >? > > > >? >? ? I am > > >? > > > > > > >? >? a little unsure of your question. > > > > > >? > > > >? > > > >? > > > >? >? ? 1) > > >? you can create a MPIBAIJ > > >? > > > >? >? matrix with any distribution of block > > >? rows per process > > >? > you > > >? > > > >? >? want, just set the > > >? local row size for each process to > > >? > > > >? be > > >? > > > >? >? what you > > >? like.? Use MatCreateVecs() to get > > >? > > > >? correspondingly > > >? > > > >? > > > >? laid out vectors. > > >? > > > >? > > > > > > >? > > > >? > > > >? ? or 2) if you have a MPIBAIJ > > >? > > > >? >? matrix with > > >? "equal" row layout and you want a > > >? > new > > >? > > > >? >? one with uneven row layout you can > > >? simply use > > >? > > > >? > > > >? MatGetSubMatrix() to create that new matrix. > > >? > > > >? > > > >? > > > >? >? ? Barry > > >? > > > >? > > > >? > > > >? >? Unless you have > > >? another reason to have the > > >? > > > >? >? matrix with an equal number row layout I > > >? would just > > >? > generate > > >? > > > >? >? the matrix with > > >? the layout you want. > > >? > > > >? > > > >? > > > >? > > > >? > > > >? >? > On Mar 22, 2015, at 5:50 PM, Steena > > >? M > > >? > > > >? >? > > >? > > > >? >? wrote: > > >? > > > >? >? > > > >? > > > >? >? > Hello, > > >? > > > >? >? > > > >? > > > >? >? > I need to > > >? distribute > > >? > > > >? >? a > > >? sparse matrix such that each proc owns an unequal > > >? > number > > >? > > > >? >? of blocked rows before I proceed with > > >? MPI MatMult. My > > >? > > > >? > > > >? initial thoughts on doing this: > > >? > > > >? >? > > > >? > > > >? >? > 1) Use? MatGetSubMatrices() on the > > >? test > > >? > > > >? >? MATMPIBAIJ > > >? matrix to produce a new matrix where each > > >? > proc > > >? > > > >? >? has an unequal number of rows. > > >? > > > >? >? > > > >? > > > >? >? > 2) Provide > > >? scatter context for vector X > > >? > > > >? >? (for MatMult )using IS iscol from > > >? MatGetSubMatrices() > > >? > while > > >? > > > >? >? creating the > > >? vector X. > > >? > > > >? > > > >? > > > >? > > > >? >? > 3) > > >? Call MatMult() > > >? > > > >? > > > >? > > > >? > > > >? >? > Will > > >? MatMult_MPIBAIJ continue to scatter > > >? > > > >? >? this matrix and vector such that each > > >? proc will own an > > >? > equal > > >? > > > >? >? number of matrix > > >? rows and corresponding diagonal vector > > >? > > > > > > >? >? elements? Should I write my own > > >? MPIMatMult function to > > >? > > > >? >? retain my redistribution of the matrix > > >? and vector? > > >? > > > >? > > > >? > > > >? > > > >? >? > > > >? Thanks in > > >? > > > >? > > > >? advance, > > >? > > > >? >? > > > >? Steena > > >? > > > >? > > > >? > > > >? > > > >? > > > >? > > > >? > -- > > >? > What most > > >? experimenters > > >? > take for granted before > > >? they begin their experiments is > > >? > > > >? infinitely more interesting than any results to which > > >? their > > >? > experiments lead. > > >? > -- Norbert > > >? > Wiener > > > > > >? > > > > > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > -- Norbert Wiener > > > > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Apr 9 19:06:08 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 9 Apr 2015 19:06:08 -0500 Subject: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult In-Reply-To: <1014905468.3172094.1428623357646.JavaMail.yahoo@mail.yahoo.com> References: <1014905468.3172094.1428623357646.JavaMail.yahoo@mail.yahoo.com> Message-ID: Run with valgrind or in the debugger > On Apr 9, 2015, at 6:49 PM, Steena M wrote: > > Thanks, Matt. From *View(), vectors x and y are being created and initialized correctly and their layout matches that of the matrix. There is a segmentation fault happening on one of the ranks, after all elements of y have been computed. The error (pasted below) is printed and visible only when using MatView(A,0) Besides changing the block size, local row sizes, and setting up for MatMult() I have not changed the code in ex190.c > > Process [0] > 12 > . > . > Process [1] > 20 > . > . > srun: error: sierra12: tasks 0-1: Segmentation fault (core dumped) > > Thanks in advance, > Steena > > > > > On Thursday, April 9, 2015 12:15 PM, Matthew Knepley wrote: > > > On Thu, Apr 9, 2015 at 2:12 PM, Steena M wrote: > Thanks, Barry! I patched the master, modified src/mat/examples/tests/ex190.c to suit my data, and the fix works. > > I need to execute MatMult and I am assigning vectors on each rank using MatCreateVecs(A, &x, &y). However, I don't think multiplication is happening at all. (I inserted a few printf statements inside MatMult_MPIBAIJ and inside MatMult_SeqBAIJ1 to check). VecSum(&ysum) is always zero. Maybe I'm assigning data incorrectly? > > Code snippet in ex190.c after partitioning the matrix unequally > > ierr = MatLoad(A,fd);CHKERRQ(ierr); > ierr = MatCreateVecs(A, &x, &y);CHKERRQ(ierr); > > ierr = VecSet(x,one);CHKERRQ(ierr); //PetscScalar one = 1.0; > ierr = VecSet(y,zero); CHKERRQ(ierr); //PetscScalar zero = 0.0; > > MatView(A, 0) gives you the matrix, so you can see what Ax should be. > > ierr = MatMult(A,x,y); CHKERRQ(ierr); > > VecView(y, 0) gives you the output > > Matt > > ierr = VecSum(y,&ysum);CHKERRQ(ierr); //ysum is always zero. > > Data is a 20x20 matrix and block size is 1. > > Any thoughts? > > Thanks, > Steena > > > . > > > > > On Friday, April 3, 2015 4:17 PM, Barry Smith wrote: > > > > Steena, > > Sorry for all the delays. Our code was just totally wrong for a user provided decomposition for two reasons 1) the logic in MatLoad_MPIXAIJ() was too convoluted to work in all cases and 2) we didn't have a single test case. > > I have attached a patch that fixes it for petsc-3.5.3 > . The fix is also in the branch barry/fix-matload-uneven-rows > and I'll put it into next then after testing it will got into maint and master and the next PETSc patch release. > > Please let us know if the patch doesn't work for you > > Thanks for your patience, > > Barry > > > On Apr 1, 2015, at 3:10 PM, Steena M wrote: > > > > Thanks Barry. Attached is the driver program, the binary mat file, and the corresponding mtx file. > > > > Runtime command used: > > > > sierra324 at monteiro:time srun -n 2 -ppdebug ./petsc-mpibaij-unequalrows -fin trefethen.dat -matload_block_size 1 > > > > > > > > > > On Wednesday, April 1, 2015 12:28 PM, Barry Smith wrote: > > > > > > > > Send a data file you generated and your reader program and we'll debug it. > > > > Barry > > > > > On Apr 1, 2015, at 2:18 PM, Steena M wrote: > > > > > > Thanks Barry. I removed the Preallocation calls. It is still complaining about the malloc and incorrect data in the matrix file. I generate binary matrix files using PETSc's pythonscript to loop through a set of UFL sparse matrices. For this use case: > > > > > > mtx_mat = scipy.io.mmread('trefethen.mtx') > > > PetscBinaryIO.PetscBinaryIO().writeMatSciPy(open('trefnew.dat','w'), mtx_mat) > > > > > > > > > > > > > > > > > > > > > On Tuesday, March 31, 2015 9:15 PM, Barry Smith wrote: > > > > > > > > > > > > You should not need to call any preallocation routines when using MatLoad() > > > > > > > > > How did you generate the file? Are you sure it has the correct information for the matrix? > > > > > > Barry > > > > > > > > > > > > > On Mar 31, 2015, at 11:05 PM, Steena M wrote: > > > > > > > > Thanks Matt. I'm still getting the malloc error > > > > > > > > [0]PETSC ERROR: Argument out of range! > > > > [0]PETSC ERROR: New nonzero at (2,18) caused a malloc! > > > > > > > > and > > > > > > > > a new incorrect matrix file error: > > > > > > > > [0]PETSC ERROR: Unexpected data in file! > > > > [0]PETSC ERROR: not matrix object! > > > > > > > > Maybe the order of calls is mixed up. This is the code snippet: > > > > > > > > if (rank ==0) > > > > { > > > > PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); > > > > > > > > CHKERRQ(MatSetSizes(A, 15, PETSC_DETERMINE, 20, 20)); > > > > CHKERRQ(MatSetType(A, MATMPIBAIJ)); > > > > CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); > > > > CHKERRQ( MatLoad(A,fd)); > > > > CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); > > > > } > > > > > > > > else > > > > { > > > > PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); > > > > > > > > CHKERRQ( MatSetSizes(A, 5, PETSC_DETERMINE, 20, 20) ); > > > > CHKERRQ(MatSetType(A, MATMPIBAIJ)); > > > > CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); > > > > CHKERRQ(MatLoad(A,fd)); > > > > CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); > > > > } > > > > > > > > Is there something I'm missing? > > > > > > > > Thanks, > > > > Steena > > > > > > > > > > > > > > > > > > > > > > > > On Tuesday, March 31, 2015 6:10 PM, Matthew Knepley wrote: > > > > > > > > > > > > On Tue, Mar 31, 2015 at 6:51 PM, Steena M wrote: > > > > Thanks Barry. I'm still getting the malloc error with NULL. Is there a way to distribute the matrix without explicit preallocation? Different matrices will be loaded during runtime and assigning preallocation parameters would mean an additional preprocessing step. > > > > > > > > 1) MatSetOption(MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) > > > > > > > > 2) Note that this is never ever ever more efficient than making another pass and preallocating > > > > > > > > Thanks, > > > > > > > > Matt > > > > -------------------------------------------- > > > > On Sun, 3/29/15, Barry Smith wrote: > > > > > > > > Subject: Re: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult > > > > To: "Steena M" > > > > Cc: "Matthew Knepley" , petsc-users at mcs.anl.gov > > > > Date: Sunday, March 29, 2015, 9:26 PM > > > > > > > > > > > > > On > > > > Mar 29, 2015, at 11:05 PM, Steena M > > > > wrote: > > > > > > > > > > Thanks > > > > Matt. I used PETSC_DETERMINE but I'm now getting an > > > > allocation-based error: > > > > > > > > > > [0]PETSC ERROR: --------------------- > > > > Error Message ------------------------------------ > > > > > [0]PETSC ERROR: Argument out of range! > > > > > [0]PETSC ERROR: New nonzero at (2,18) > > > > caused a malloc! > > > > > [0]PETSC ERROR: > > > > ------------------------------------------------------------------------ > > > > > > > > > > I tried > > > > preallocating on each rank for the diagonal and off diagonal > > > > section of the matrix as the next step My current > > > > approximations for preallocation > > > > > > > > > > CHKERRQ( > > > > MatMPIBAIJSetPreallocation(A,1,5,PETSC_DEFAULT,5,PETSC_DEFAULT)); > > > > > > > > > > > > These > > > > arguments where you pass PETSC_DEFAULT are expecting a > > > > pointer not an integer. You can pass NULL in those > > > > locations. Though it is better to provide the correct > > > > preallocation rather than some defaults. > > > > > > > > Barry > > > > > > > > > > > > > > are throwing segmentation errors. > > > > > > > > > > [0]PETSC ERROR: > > > > Caught signal number 11 SEGV: Segmentation Violation, > > > > probably memory access out of range > > > > > > > > > > Any insights into what I'm doing > > > > wrong? > > > > > > > > > > Thanks, > > > > > Steena > > > > > > > > > > > > > > > > > > > > On Sun, 3/29/15, Matthew Knepley > > > > wrote: > > > > > > > > > > Subject: > > > > Re: [petsc-users] Unequal sparse matrix row distribution for > > > > MPI MatMult > > > > > To: "Steena M" > > > > > > > > > Cc: "Barry Smith" , > > > > petsc-users at mcs.anl.gov > > > > > Date: Sunday, March 29, 2015, 10:02 PM > > > > > > > > > > On Sun, Mar 29, 2015 > > > > at > > > > > 9:56 PM, Steena M > > > > > wrote: > > > > > Hi > > > > > Barry, > > > > > > > > > > > > > > > > > > > > I am trying to partition a 20 row and 20 > > > > col sparse matrix > > > > > between two procs > > > > such that proc 0 has 15 rows and 20 cols > > > > > and proc 1 has 5 rows and 20 cols. The > > > > code snippet: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A)); > > > > // > > > > > at runtime: -matload_block_size 1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > if (rank ==0) > > > > > > > > > > { > > > > > > > > > > > > > > CHKERRQ( MatSetSizes(A, 15, 20, 20, > > > > > 20) ); //rank 0 gets 75% of the rows > > > > > > > > > > > > > > CHKERRQ( MatSetType(A, MATMPIBAIJ) > > > > > ); > > > > > > > > > > CHKERRQ( > > > > MatLoad(A,fd) ); > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > else > > > > > > > > > > > > > > { > > > > > > > > > > > > > > CHKERRQ( MatSetSizes(A, 5, > > > > 20, 20, > > > > > 20) ); //rank 1 gets 25% of the > > > > rows > > > > > > > > > > > > > > CHKERRQ( MatSetType(A, MATMPIBAIJ) > > > > > ); > > > > > > > > > > CHKERRQ( > > > > MatLoad(A,fd) ); > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > This throws the following error (probably > > > > from psplit.c): > > > > > > > > > > > > > > [1]PETSC ERROR: --------------------- Error Message > > > > > ------------------------------------ > > > > > > > > > > [1]PETSC ERROR: > > > > Nonconforming object sizes! > > > > > > > > > > [1]PETSC ERROR: Sum of local lengths 40 > > > > does not equal > > > > > global length 20, my > > > > local length 20 > > > > > > > > > > likely a call to > > > > VecSetSizes() or MatSetSizes() is > > > > > > > > > wrong. > > > > > > > > > > See > > > > http://www.mcs.anl.gov/petsc/documentation/faq.html#split! > > > > > > > > > > > > > > > > > > > > This error printout > > > > doesn't quite make sense to me. > > > > > > > > > I'm trying to specify a total matrix size of 20x20... > > > > I > > > > > haven't yet figured out where the > > > > '40' comes > > > > > from in the error > > > > message. > > > > > > > > > > > > > > > > > > > > Any thoughts on what > > > > might be going wrong? > > > > > > > > > > Its the column specification. Just > > > > > use PETSC_DETERMINE for the local columns > > > > since all our > > > > > sparse matrixformats are > > > > row divisions > > > > > anyway. > > > > > > > > > > Thanks, > > > > > > > > > > Matt > > > > > Thanks in advance, > > > > > > > > > > > > > > Steena > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------------------------------------- > > > > > > > > > > On Sun, 3/22/15, > > > > Barry Smith > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Subject: Re: [petsc-users] Unequal > > > > sparse matrix row > > > > > distribution for MPI > > > > MatMult > > > > > > > > > > To: > > > > "Steena M" > > > > > > > > > > Cc: petsc-users at mcs.anl.gov > > > > > > > > > > Date: Sunday, > > > > March 22, 2015, 3:58 PM > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Steena, > > > > > > > > > > > > > > > > > > > > I am > > > > > > > > > > > > > > a little unsure of your question. > > > > > > > > > > > > > > > > > > > > > > > > 1) > > > > you can create a MPIBAIJ > > > > > > > > > > matrix with any distribution of block > > > > rows per process > > > > > you > > > > > > > > > > want, just set the > > > > local row size for each process to > > > > > > > > > be > > > > > > > > > > what you > > > > like. Use MatCreateVecs() to get > > > > > > > > > correspondingly > > > > > > > > > > > > > > laid out vectors. > > > > > > > > > > > > > > > > > > > > > > > > > > > > or 2) if you have a MPIBAIJ > > > > > > > > > > matrix with > > > > "equal" row layout and you want a > > > > > new > > > > > > > > > > one with uneven row layout you can > > > > simply use > > > > > > > > > > > > > > MatGetSubMatrix() to create that new matrix. > > > > > > > > > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > Unless you have > > > > another reason to have the > > > > > > > > > > matrix with an equal number row layout I > > > > would just > > > > > generate > > > > > > > > > > the matrix with > > > > the layout you want. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 22, 2015, at 5:50 PM, Steena > > > > M > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > I need to > > > > distribute > > > > > > > > > > a > > > > sparse matrix such that each proc owns an unequal > > > > > number > > > > > > > > > > of blocked rows before I proceed with > > > > MPI MatMult. My > > > > > > > > > > > > > > initial thoughts on doing this: > > > > > > > > > > > > > > > > > > > > > > 1) Use MatGetSubMatrices() on the > > > > test > > > > > > > > > > MATMPIBAIJ > > > > matrix to produce a new matrix where each > > > > > proc > > > > > > > > > > has an unequal number of rows. > > > > > > > > > > > > > > > > > > > > > > 2) Provide > > > > scatter context for vector X > > > > > > > > > > (for MatMult )using IS iscol from > > > > MatGetSubMatrices() > > > > > while > > > > > > > > > > creating the > > > > vector X. > > > > > > > > > > > > > > > > > > > > > > > > > > 3) > > > > Call MatMult() > > > > > > > > > > > > > > > > > > > > > > > > > > Will > > > > MatMult_MPIBAIJ continue to scatter > > > > > > > > > > this matrix and vector such that each > > > > proc will own an > > > > > equal > > > > > > > > > > number of matrix > > > > rows and corresponding diagonal vector > > > > > > > > > > > > > > elements? Should I write my own > > > > MPIMatMult function to > > > > > > > > > > retain my redistribution of the matrix > > > > and vector? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks in > > > > > > > > > > > > > > advance, > > > > > > > > > > > > > > > Steena > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > What most > > > > experimenters > > > > > take for granted before > > > > they begin their experiments is > > > > > > > > > infinitely more interesting than any results to which > > > > their > > > > > experiments lead. > > > > > -- Norbert > > > > > Wiener > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > > -- Norbert Wiener > > > > > > > > > > > > > > > > > > > > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > From jchang27 at uh.edu Thu Apr 9 19:21:28 2015 From: jchang27 at uh.edu (Justin Chang) Date: Thu, 9 Apr 2015 19:21:28 -0500 Subject: [petsc-users] Uninterpolating DMLabels? In-Reply-To: References: Message-ID: 1) I did what you had suggested and it works very well thanks. 2) In my case, calling uninterpolate does make a difference, just not necessarily in the solver step (I am using Tao's BLMVM). I attached two output files, one containing the timings and summaries with uninterpolating and one without uninterpolating. And the source code if you wanted to verify the "correctness" of my implementation. Namely, the biggest difference I see is in setting up the PetscSection. When uninterpolating the data this steps takes 8.64039 seconds but when I do not uninterpolate the data it takes 61.2977 seconds. Is this "significant" enough, or is it unimportant given the fact that this step occurs only once during any simulation? 3) Also, about DMPlexDistribute(...). Right now it still seems extremely slow, hence why I am distributing a coarse mesh and having each processor refine its local portion of the mesh. In my current implementation, I have rank 0 distribute the mesh via ParMETIS. Will there eventually be a methodology to do parallel I/O then redistribution for load balancing or does it already exist? Thanks, On Tue, Apr 7, 2015 at 8:37 PM, Matthew Knepley wrote: > On Tue, Apr 7, 2015 at 7:18 PM, Justin Chang wrote: > >> Hi all, >> >> So I am interpolating/uninterpolating my DMPlex mesh. This is the mesh >> information for a cell-vertex mesh only: >> >> DM Object: 1 MPI processes >> type: plex >> DM_0x84000000_0 in 3 dimensions: >> 0-cells: 159 >> 3-cells: 592 >> Labels: >> marker: 1 strata of sizes (100) >> depth: 2 strata of sizes (159, 592) >> >> I am interpolating the mesh with the following lines: >> >> ierr = DMPlexInterpolate(*dm, &idm);CHKERRQ(ierr); >> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); >> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); >> ierr = DMPlexGetLabel(idm, "marker", &label);CHKERRQ(ierr); >> ierr = DMPlexMarkBoundaryFaces(idm,label);CHKERRQ(ierr); >> ierr = DMDestroy(dm);CHKERRQ(ierr); >> *dm = idm; >> >> And the resulting mesh: >> >> DM Object: 1 MPI processes >> type: plex >> DM_0x84000000_1 in 3 dimensions: >> 0-cells: 159 >> 1-cells: 845 >> 2-cells: 1280 >> 3-cells: 592 >> Labels: >> marker: 1 strata of sizes (292) >> depth: 4 strata of sizes (159, 845, 1280, 592) >> >> The reason I did this is so that incase I decide to refine the mesh with >> "-dm_refine " the DM will be sure to include the "marker" points in >> the refinement process. Now, if I decide to uninterpolate the mesh with the >> following lines: >> >> ierr = DMPlexUninterpolate(*dm, &idm);CHKERRQ(ierr); >> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); >> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); >> ierr = DMDestroy(dm);CHKERRQ(ierr); >> *dm = idm; >> >> I end up with this mesh: >> >> DM Object: 1 MPI processes >> type: plex >> DM_0x84000000_2 in 3 dimensions: >> 0-cells: 159 >> 3-cells: 592 >> Labels: >> marker: 1 strata of sizes (292) >> depth: 2 strata of sizes (159, 592) >> >> The problem is that although the cells and vertices have gone back to the >> original number, the "marker" label still includes face/edge points. This >> gives me an error once I invoke DMCreateGobalVector(...). >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Argument out of range >> [0]PETSC ERROR: Section point 755 should be in [0, 751) >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2528-gbee642f GIT >> Date: 2015-03-29 20:36:38 -0500 >> [0]PETSC ERROR: ./cube_with_hole on a arch-linux2-c-debug named pacotaco >> by justin Tue Apr 7 19:09:12 2015 >> [0]PETSC ERROR: Configure options --download-chaco --download-exodusii >> --download-fblaslapack --download-hdf5 --download-metis --download-mpich >> --download-mumps --download-netcdf --download-parmetis --download-scalapack >> --download-triangle --with-cc=gcc --with-cmake=cmake --with-cxx=g++ >> --with-debugging=1 --with-fc=gfortran --with-valgrind=1 >> PETSC_ARCH=arch-linux2-c-debug >> [0]PETSC ERROR: #1 PetscSectionGetDof() line 499 in >> /home/justin/petsc-dev/src/vec/is/utils/vsectionis.c >> [0]PETSC ERROR: #2 DMPlexGetConeSize() line 841 in >> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >> [0]PETSC ERROR: #3 DMPlexGetTransitiveClosure() line 1342 in >> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >> [0]PETSC ERROR: #4 DMPlexLabelComplete() line 88 in >> /home/justin/petsc-dev/src/dm/impls/plex/plexsubmesh.c >> [0]PETSC ERROR: #5 DMCreateDefaultSection_Plex() line 5605 in >> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >> [0]PETSC ERROR: #6 DMGetDefaultSection() line 3035 in >> /home/justin/petsc-dev/src/dm/interface/dm.c >> [0]PETSC ERROR: #7 DMGetDefaultGlobalSection() line 3266 in >> /home/justin/petsc-dev/src/dm/interface/dm.c >> [0]PETSC ERROR: #8 DMCreateGlobalVector_Section_Private() line 13 in >> /home/justin/petsc-dev/src/dm/interface/dmi.c >> [0]PETSC ERROR: #9 DMCreateGlobalVector_Plex() line 1170 in >> /home/justin/petsc-dev/src/dm/impls/plex/plexcreate.c >> [0]PETSC ERROR: #10 DMCreateGlobalVector() line 698 in >> /home/justin/petsc-dev/src/dm/interface/dm.c >> [0]PETSC ERROR: #11 main() line 363 in >> /home/justin/Dropbox/Research_Topics/Code_PETSc/Nonneg/cube_with_hole.c >> >> >> From this error, it seems it's trying to access sieve points that non >> longer exist. And I think it has to do with the label "marker" still >> contains data from the interpolated mesh. Is there a way to "uninterpolate" >> or remove such points? Or is there a better way of approaching this whole >> thing? >> > > 1) It would be easy to write a small function to throw out the points in > the label that do not exist in the DM. You > would just extract each stratumIS and then Clear each invalid point. > > 2) However, I do not understand the rationale for this above. You could > just call MarkBoundaryFaces on the final mesh. > If you are really trying to preserve a label across regular refinement > AND you really do not want an interpolated mesh, > then code up 1). I have yet to see a case where the extra memory for > edges and faces makes a difference, but it may exist. > > Thanks, > > Matt > > >> Thanks, >> >> >> >> -- >> Justin Chang >> PhD Candidate, Civil Engineering - Computational Sciences >> University of Houston, Department of Civil and Environmental Engineering >> Houston, TX 77004 >> (512) 963-3262 >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- ======================== 1 processors: ======================== Creating DMPlex... done DM before interpolating: DM Object: 1 MPI processes type: plex DM_0x7fa87381c0a0_0 in 3 dimensions: 0-cells: 159 3-cells: 592 Labels: marker: 1 strata of sizes (100) depth: 2 strata of sizes (159, 592) DM after interpolation: DM Object: 1 MPI processes type: plex DM_0x7fa87381c0a0_1 in 3 dimensions: 0-cells: 159 1-cells: 845 2-cells: 1280 3-cells: 592 Labels: marker: 1 strata of sizes (580) depth: 4 strata of sizes (159, 845, 1280, 592) Distributing mesh... done Refining mesh... done DM Object:DM_0x7fa87381c0a0_1 1 MPI processes type: plex DM_0x7fa87381c0a0_1 in 3 dimensions: 0-cells: 416594 1-cells: 2866000 2-cells: 4874240 3-cells: 2424832 Labels: marker: 1 strata of sizes (147460) depth: 4 strata of sizes (416594, 2866000, 4874240, 2424832) DM after uninterpolation: DM Object: 1 MPI processes type: plex DM_0x7fa87381c0a0_2 in 3 dimensions: 0-cells: 416594 3-cells: 2424832 Labels: marker: 1 strata of sizes (24580) depth: 2 strata of sizes (416594, 2424832) Setting up and initializing problem... done ======================== Solving non-negative... ======================== Tao converged with reason 2 ======================== Time summary: ======================== Create DMPlex: 0.00444102s Distribute and refine DMPlex: 8.95675s Set up problem: 8.64039s Solver: 30.9317s Postprocess into VTU: 0.662271s ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./cube_with_hole on a arch-darwin-c-debug named CAML-03.CIVE.UH.EDU with 1 processor, by jychang48 Thu Apr 9 18:22:03 2015 Using Petsc Development GIT revision: v3.5.3-2528-gbee642f GIT Date: 2015-03-29 20:36:38 -0500 Max Max/Min Avg Total Time (sec): 4.934e+01 1.00000 4.934e+01 Objects: 2.950e+02 1.00000 2.950e+02 Flops: 2.389e+10 1.00000 2.389e+10 2.389e+10 Flops/sec: 4.841e+08 1.00000 4.841e+08 4.841e+08 MPI Messages: 5.500e+00 1.00000 5.500e+00 5.500e+00 MPI Message Lengths: 1.411e+07 1.00000 2.566e+06 1.411e+07 MPI Reductions: 2.100e+01 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 4.9336e+01 100.0% 2.3886e+10 100.0% 5.500e+00 100.0% 2.566e+06 100.0% 2.100e+01 100.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage ThreadCommRunKer 494 1.0 5.0792e+00 1.0 6.67e+09 1.0 0.0e+00 0.0e+00 0.0e+00 10 28 0 0 0 10 28 0 0 0 1314 CreateMesh 1 1.0 8.9612e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+01 18 0 0 0 95 18 0 0 0 95 0 VecView 1 1.0 2.3220e-02 1.0 6.47e+05 1.0 1.0e+00 3.1e+06 0.0e+00 0 0 18 22 0 0 0 18 22 0 28 VecDot 10740 1.0 4.3375e+00 1.0 8.42e+09 1.0 0.0e+00 0.0e+00 0.0e+00 9 35 0 0 0 9 35 0 0 0 1941 VecNorm 490 1.0 1.7819e-01 1.0 3.84e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 2156 VecScale 1955 1.0 5.8523e-01 1.0 7.66e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 3 0 0 0 1 3 0 0 0 1310 VecCopy 6356 1.0 2.6006e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 5 0 0 0 0 5 0 0 0 0 0 VecSet 47 1.0 2.1615e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 8287 1.0 3.3792e+00 1.0 6.69e+09 1.0 0.0e+00 0.0e+00 0.0e+00 7 28 0 0 0 7 28 0 0 0 1979 VecAYPX 976 1.0 4.0512e-01 1.0 3.83e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 944 VecWAXPY 1 1.0 5.2404e-04 1.0 3.92e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 748 VecPointwiseMult 2929 1.0 1.6945e+00 1.0 1.15e+09 1.0 0.0e+00 0.0e+00 0.0e+00 3 5 0 0 0 3 5 0 0 0 678 MatMult 495 1.0 4.4913e+00 1.0 5.52e+09 1.0 0.0e+00 0.0e+00 0.0e+00 9 23 0 0 0 9 23 0 0 0 1229 MatAssemblyBegin 2 1.0 2.1458e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 2 1.0 1.4330e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatZeroEntries 1 1.0 4.7700e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 DMPlexInterp 2 1.0 3.0179e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 DMPlexStratify 11 1.0 2.0543e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 4 0 0 0 0 0 DMPlexPrealloc 1 1.0 3.2820e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 7 0 0 0 0 7 0 0 0 0 0 DMPlexResidualFE 1 1.0 1.7055e+00 1.0 1.63e+08 1.0 0.0e+00 0.0e+00 0.0e+00 3 1 0 0 0 3 1 0 0 0 96 DMPlexJacobianFE 1 1.0 4.0298e+00 1.0 6.47e+05 1.0 0.0e+00 0.0e+00 0.0e+00 8 0 0 0 0 8 0 0 0 0 0 SFSetGraph 1 1.0 7.3969e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFBcastBegin 3 1.0 8.4453e-03 1.0 0.00e+00 0.0 4.5e+00 2.4e+06 0.0e+00 0 0 82 78 0 0 0 82 78 0 0 SFBcastEnd 3 1.0 3.5551e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFReduceBegin 1 1.0 1.4312e-03 1.0 0.00e+00 0.0 1.0e+00 3.1e+06 0.0e+00 0 0 18 22 0 0 0 18 22 0 0 SFReduceEnd 1 1.0 1.3900e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SNESFunctionEval 1 1.0 1.7262e+00 1.0 1.63e+08 1.0 3.5e+00 2.2e+06 0.0e+00 3 1 64 56 0 3 1 64 56 0 94 SNESJacobianEval 1 1.0 4.0329e+00 1.0 6.47e+05 1.0 1.0e+00 3.1e+06 0.0e+00 8 0 18 22 0 8 0 18 22 0 0 TaoSolve 1 1.0 2.5136e+01 1.0 2.35e+10 1.0 0.0e+00 0.0e+00 0.0e+00 51 98 0 0 0 51 98 0 0 0 935 TaoLineSearchApply 489 1.0 9.2746e+00 1.0 8.59e+09 1.0 0.0e+00 0.0e+00 0.0e+00 19 36 0 0 0 19 36 0 0 0 926 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Viewer 4 3 2312 0 Object 5 5 2980 0 Container 11 11 6380 0 Vector 42 42 626666400 0 Matrix 2 2 75555348 0 Distributed Mesh 23 23 107520 0 GraphPartitioner 11 11 6820 0 Star Forest Bipartite Graph 48 48 39336 0 Discrete System 23 23 19136 0 Index Set 59 59 50389936 0 IS L to G Mapping 1 1 1666972 0 Section 56 56 37856 0 SNES 1 1 1340 0 SNESLineSearch 1 1 880 0 DMSNES 1 1 680 0 Krylov Solver 1 1 1232 0 Preconditioner 1 1 864 0 Linear Space 1 1 656 0 Dual Space 1 1 672 0 FE Space 1 1 764 0 Tao 1 1 1720 0 TaoLineSearch 1 1 896 0 ======================================================================================================================== Average time to get PetscTime(): 9.53674e-08 #PETSc Option Table entries: -concentration 1,0 -dm_refine 4 -f_al 1e3 -f_at 1 -f_boundary cube_with_hole_test_bd.dat -f_mesh cube_with_hole_test_mesh.dat -f_out 1 -f_outname 3D_cube_orig.vtu -ksp_rtol 1.0e-9 -ksp_type cg -log_summary -nonnegative 1 -pc_type jacobi -petscpartitioner_type parmetis -petscspace_order 1 -snes_atol 1.0e-6 -tao_type blmvm #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --download-chaco --download-exodusii --download-fblaslapack --download-hdf5 --download-metis --download-mumps --download-netcdf --download-openmpi --download-parmetis --download-scalapack --download-triangle --with-cc=gcc --with-cmake=cmake --with-cxx=g++ --with-debugging=0 --with-fc=gfortran --with-valgrind=1 ----------------------------------------- Libraries compiled on Mon Apr 6 20:58:57 2015 on CAML-03.CIVE.UH.EDU Machine characteristics: Darwin-14.1.0-x86_64-i386-64bit Using PETSc directory: /Users/jychang48/Software/petsc-dev Using PETSc arch: arch-darwin-c-debug ----------------------------------------- Using C compiler: /Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/bin/mpif90 -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/include -I/Users/jychang48/Software/petsc-dev/include -I/Users/jychang48/Software/petsc-dev/include -I/Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/include -I/opt/local/include -I/opt/X11/include ----------------------------------------- Using C linker: /Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/bin/mpicc Using Fortran linker: /Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/bin/mpif90 Using libraries: -Wl,-rpath,/Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/lib -L/Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/lib -lpetsc -Wl,-rpath,/Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/lib -L/Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lexoIIv2for -lexodus -lflapack -lfblas -lparmetis -lnetcdf -ltriangle -lmetis -lssl -lcrypto -Wl,-rpath,/opt/local/lib -L/opt/local/lib -lhwloc -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lchaco -Wl,-rpath,/opt/X11/lib -L/opt/X11/lib -lX11 -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/6.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/6.0/lib/darwin -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lgfortran -Wl,-rpath,/usr/local/Cellar/gcc/4.9.2/lib/gcc/x86_64-apple-darwin14.0.0/4.9.2 -L/usr/local/Cellar/gcc/4.9.2/lib/gcc/x86_64-apple-darwin14.0.0/4.9.2 -Wl,-rpath,/usr/local/Cellar/gcc/4.9.2/lib -L/usr/local/Cellar/gcc/4.9.2/lib -lgfortran -lgcc_ext.10.5 -lquadmath -lm -lclang_rt.osx -lmpi_cxx -lc++ -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/lib/darwin -lclang_rt.osx -Wl,-rpath,/Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/lib -L/Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/lib -ldl -lmpi -lSystem -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/lib/darwin -lclang_rt.osx -ldl ----------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: cube_with_hole.c Type: text/x-csrc Size: 21809 bytes Desc: not available URL: -------------- next part -------------- ======================== 1 processors: ======================== Creating DMPlex... done DM before interpolating: DM Object: 1 MPI processes type: plex DM_0x7fe27ae3d3b0_0 in 3 dimensions: 0-cells: 159 3-cells: 592 Labels: marker: 1 strata of sizes (100) depth: 2 strata of sizes (159, 592) DM after interpolation: DM Object: 1 MPI processes type: plex DM_0x7fe27ae3d3b0_1 in 3 dimensions: 0-cells: 159 1-cells: 845 2-cells: 1280 3-cells: 592 Labels: marker: 1 strata of sizes (580) depth: 4 strata of sizes (159, 845, 1280, 592) Distributing mesh... done Refining mesh... done Setting up and initializing problem... done ======================== Solving non-negative... ======================== Tao converged with reason 2 ======================== Time summary: ======================== Create DMPlex: 0.00416207s Distribute and refine DMPlex: 4.0888s Set up problem: 61.2977s Solver: 42.2464s Postprocess into VTU: 3.73179s ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./cube_with_hole on a arch-darwin-c-debug named CAML-03.CIVE.UH.EDU with 1 processor, by jychang48 Thu Apr 9 18:26:21 2015 Using Petsc Development GIT revision: v3.5.3-2528-gbee642f GIT Date: 2015-03-29 20:36:38 -0500 Max Max/Min Avg Total Time (sec): 1.116e+02 1.00000 1.116e+02 Objects: 2.760e+02 1.00000 2.760e+02 Flops: 2.389e+10 1.00000 2.389e+10 2.389e+10 Flops/sec: 2.142e+08 1.00000 2.142e+08 2.142e+08 MPI Messages: 5.500e+00 1.00000 5.500e+00 5.500e+00 MPI Message Lengths: 1.411e+07 1.00000 2.566e+06 1.411e+07 MPI Reductions: 1.100e+01 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.1157e+02 100.0% 2.3895e+10 100.0% 5.500e+00 100.0% 2.566e+06 100.0% 1.100e+01 100.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage ThreadCommRunKer 494 1.0 5.0596e+00 1.0 6.67e+09 1.0 0.0e+00 0.0e+00 0.0e+00 5 28 0 0 0 5 28 0 0 0 1319 CreateMesh 1 1.0 4.0930e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 4 0 0 0 91 4 0 0 0 91 0 VecView 1 1.0 3.2374e-01 1.0 3.47e+06 1.0 1.0e+00 3.1e+06 0.0e+00 0 0 18 22 0 0 0 18 22 0 11 VecDot 10740 1.0 4.3332e+00 1.0 8.42e+09 1.0 0.0e+00 0.0e+00 0.0e+00 4 35 0 0 0 4 35 0 0 0 1943 VecNorm 490 1.0 1.7682e-01 1.0 3.84e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 2173 VecScale 1955 1.0 5.8404e-01 1.0 7.66e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 3 0 0 0 1 3 0 0 0 1312 VecCopy 6356 1.0 2.5955e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 VecSet 46 1.0 2.1418e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 8287 1.0 3.3791e+00 1.0 6.69e+09 1.0 0.0e+00 0.0e+00 0.0e+00 3 28 0 0 0 3 28 0 0 0 1979 VecAYPX 976 1.0 4.0522e-01 1.0 3.83e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 944 VecWAXPY 1 1.0 5.5599e-04 1.0 3.92e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 705 VecPointwiseMult 2929 1.0 1.6930e+00 1.0 1.15e+09 1.0 0.0e+00 0.0e+00 0.0e+00 2 5 0 0 0 2 5 0 0 0 678 MatMult 495 1.0 4.4760e+00 1.0 5.52e+09 1.0 0.0e+00 0.0e+00 0.0e+00 4 23 0 0 0 4 23 0 0 0 1234 MatAssemblyBegin 2 1.0 2.1458e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 2 1.0 1.4720e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatZeroEntries 1 1.0 4.8032e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 DMPlexInterp 2 1.0 2.8389e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 DMPlexStratify 10 1.0 1.8760e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 DMPlexPrealloc 1 1.0 4.1800e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 37 0 0 0 0 37 0 0 0 0 0 DMPlexResidualFE 1 1.0 8.6615e+00 1.0 1.66e+08 1.0 0.0e+00 0.0e+00 0.0e+00 8 1 0 0 0 8 1 0 0 0 19 DMPlexJacobianFE 1 1.0 8.3987e+00 1.0 3.47e+06 1.0 0.0e+00 0.0e+00 0.0e+00 8 0 0 0 0 8 0 0 0 0 0 SFSetGraph 1 1.0 6.8889e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFBcastBegin 3 1.0 8.0659e-03 1.0 0.00e+00 0.0 4.5e+00 2.4e+06 0.0e+00 0 0 82 78 0 0 0 82 78 0 0 SFBcastEnd 3 1.0 3.3667e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFReduceBegin 1 1.0 1.4300e-03 1.0 0.00e+00 0.0 1.0e+00 3.1e+06 0.0e+00 0 0 18 22 0 0 0 18 22 0 0 SFReduceEnd 1 1.0 1.2019e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SNESFunctionEval 1 1.0 8.7082e+00 1.0 1.66e+08 1.0 3.5e+00 2.2e+06 0.0e+00 8 1 64 56 0 8 1 64 56 0 19 SNESJacobianEval 1 1.0 8.4015e+00 1.0 3.47e+06 1.0 1.0e+00 3.1e+06 0.0e+00 8 0 18 22 0 8 0 18 22 0 0 TaoSolve 1 1.0 2.5100e+01 1.0 2.35e+10 1.0 0.0e+00 0.0e+00 0.0e+00 22 98 0 0 0 22 98 0 0 0 936 TaoLineSearchApply 489 1.0 9.2474e+00 1.0 8.59e+09 1.0 0.0e+00 0.0e+00 0.0e+00 8 36 0 0 0 8 36 0 0 0 928 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Viewer 4 3 2312 0 Object 5 5 2980 0 Container 11 11 6380 0 Vector 41 41 616666600 0 Matrix 2 2 75555348 0 Distributed Mesh 21 21 98176 0 GraphPartitioner 10 10 6200 0 Star Forest Bipartite Graph 44 44 36016 0 Discrete System 21 21 17472 0 Index Set 54 54 55120896 0 IS L to G Mapping 1 1 1666972 0 Section 52 52 35152 0 SNES 1 1 1340 0 SNESLineSearch 1 1 880 0 DMSNES 1 1 680 0 Krylov Solver 1 1 1232 0 Preconditioner 1 1 864 0 Linear Space 1 1 656 0 Dual Space 1 1 672 0 FE Space 1 1 764 0 Tao 1 1 1720 0 TaoLineSearch 1 1 896 0 ======================================================================================================================== Average time to get PetscTime(): 0 #PETSc Option Table entries: -concentration 1,0 -dm_refine 4 -f_al 1e3 -f_at 1 -f_boundary cube_with_hole_test_bd.dat -f_mesh cube_with_hole_test_mesh.dat -f_out 1 -f_outname 3D_cube_orig.vtu -ksp_rtol 1.0e-9 -ksp_type cg -log_summary -nonnegative 1 -pc_type jacobi -petscpartitioner_type parmetis -petscspace_order 1 -snes_atol 1.0e-6 -tao_type blmvm #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --download-chaco --download-exodusii --download-fblaslapack --download-hdf5 --download-metis --download-mumps --download-netcdf --download-openmpi --download-parmetis --download-scalapack --download-triangle --with-cc=gcc --with-cmake=cmake --with-cxx=g++ --with-debugging=0 --with-fc=gfortran --with-valgrind=1 ----------------------------------------- Libraries compiled on Mon Apr 6 20:58:57 2015 on CAML-03.CIVE.UH.EDU Machine characteristics: Darwin-14.1.0-x86_64-i386-64bit Using PETSc directory: /Users/jychang48/Software/petsc-dev Using PETSc arch: arch-darwin-c-debug ----------------------------------------- Using C compiler: /Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/bin/mpif90 -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/include -I/Users/jychang48/Software/petsc-dev/include -I/Users/jychang48/Software/petsc-dev/include -I/Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/include -I/opt/local/include -I/opt/X11/include ----------------------------------------- Using C linker: /Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/bin/mpicc Using Fortran linker: /Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/bin/mpif90 Using libraries: -Wl,-rpath,/Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/lib -L/Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/lib -lpetsc -Wl,-rpath,/Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/lib -L/Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lexoIIv2for -lexodus -lflapack -lfblas -lparmetis -lnetcdf -ltriangle -lmetis -lssl -lcrypto -Wl,-rpath,/opt/local/lib -L/opt/local/lib -lhwloc -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lchaco -Wl,-rpath,/opt/X11/lib -L/opt/X11/lib -lX11 -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/6.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/6.0/lib/darwin -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lgfortran -Wl,-rpath,/usr/local/Cellar/gcc/4.9.2/lib/gcc/x86_64-apple-darwin14.0.0/4.9.2 -L/usr/local/Cellar/gcc/4.9.2/lib/gcc/x86_64-apple-darwin14.0.0/4.9.2 -Wl,-rpath,/usr/local/Cellar/gcc/4.9.2/lib -L/usr/local/Cellar/gcc/4.9.2/lib -lgfortran -lgcc_ext.10.5 -lquadmath -lm -lclang_rt.osx -lmpi_cxx -lc++ -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/lib/darwin -lclang_rt.osx -Wl,-rpath,/Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/lib -L/Users/jychang48/Software/petsc-dev/arch-darwin-c-debug/lib -ldl -lmpi -lSystem -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/lib/darwin -lclang_rt.osx -ldl ----------------------------------------- From knepley at gmail.com Thu Apr 9 19:52:08 2015 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Apr 2015 19:52:08 -0500 Subject: [petsc-users] Uninterpolating DMLabels? In-Reply-To: References: Message-ID: On Thu, Apr 9, 2015 at 7:21 PM, Justin Chang wrote: > 1) I did what you had suggested and it works very well thanks. > Cool. > 2) In my case, calling uninterpolate does make a difference, just not > necessarily in the solver step (I am using Tao's BLMVM). I attached two > output files, one containing the timings and summaries with uninterpolating > and one without uninterpolating. And the source code if you wanted to > verify the "correctness" of my implementation. Namely, the biggest > difference I see is in setting up the PetscSection. When uninterpolating > the data this steps takes 8.64039 seconds but when I do not uninterpolate > the data it takes 61.2977 seconds. Is this "significant" enough, or is it > unimportant given the fact that this step occurs only once during any > simulation? > That is significant. Most of that time is in the operator preallocation routine. It was cleaner to first gather the point adjacency information, and then push the dof information on top. If I aggressively merged the dof data, I could have pruned a bunch of the work here and gotten much closer to the first time, at the expense of more complicated code. I guess this is the right pattern if you know that you will only have unknowns on cells and vertices. > 3) Also, about DMPlexDistribute(...). Right now it still seems extremely > slow, hence why I am distributing a coarse mesh and having each processor > refine its local portion of the mesh. In my current implementation, I have > rank 0 distribute the mesh via ParMETIS. Will there eventually be a > methodology to do parallel I/O then redistribution for load balancing or > does it already exist? > Redistribution for load balancing already exists. The stumbling block right now is parallel mesh import. However, the group at ICL (Michael Lange and Gerard Gorman) is working on this, and we expect to have it working by the end of summer. Until then, I recommend exactly what you are doing. Thanks, Matt > Thanks, > > On Tue, Apr 7, 2015 at 8:37 PM, Matthew Knepley wrote: > >> On Tue, Apr 7, 2015 at 7:18 PM, Justin Chang wrote: >> >>> Hi all, >>> >>> So I am interpolating/uninterpolating my DMPlex mesh. This is the mesh >>> information for a cell-vertex mesh only: >>> >>> DM Object: 1 MPI processes >>> type: plex >>> DM_0x84000000_0 in 3 dimensions: >>> 0-cells: 159 >>> 3-cells: 592 >>> Labels: >>> marker: 1 strata of sizes (100) >>> depth: 2 strata of sizes (159, 592) >>> >>> I am interpolating the mesh with the following lines: >>> >>> ierr = DMPlexInterpolate(*dm, &idm);CHKERRQ(ierr); >>> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); >>> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); >>> ierr = DMPlexGetLabel(idm, "marker", &label);CHKERRQ(ierr); >>> ierr = DMPlexMarkBoundaryFaces(idm,label);CHKERRQ(ierr); >>> ierr = DMDestroy(dm);CHKERRQ(ierr); >>> *dm = idm; >>> >>> And the resulting mesh: >>> >>> DM Object: 1 MPI processes >>> type: plex >>> DM_0x84000000_1 in 3 dimensions: >>> 0-cells: 159 >>> 1-cells: 845 >>> 2-cells: 1280 >>> 3-cells: 592 >>> Labels: >>> marker: 1 strata of sizes (292) >>> depth: 4 strata of sizes (159, 845, 1280, 592) >>> >>> The reason I did this is so that incase I decide to refine the mesh with >>> "-dm_refine " the DM will be sure to include the "marker" points in >>> the refinement process. Now, if I decide to uninterpolate the mesh with the >>> following lines: >>> >>> ierr = DMPlexUninterpolate(*dm, &idm);CHKERRQ(ierr); >>> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); >>> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); >>> ierr = DMDestroy(dm);CHKERRQ(ierr); >>> *dm = idm; >>> >>> I end up with this mesh: >>> >>> DM Object: 1 MPI processes >>> type: plex >>> DM_0x84000000_2 in 3 dimensions: >>> 0-cells: 159 >>> 3-cells: 592 >>> Labels: >>> marker: 1 strata of sizes (292) >>> depth: 2 strata of sizes (159, 592) >>> >>> The problem is that although the cells and vertices have gone back to >>> the original number, the "marker" label still includes face/edge points. >>> This gives me an error once I invoke DMCreateGobalVector(...). >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Argument out of range >>> [0]PETSC ERROR: Section point 755 should be in [0, 751) >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2528-gbee642f >>> GIT Date: 2015-03-29 20:36:38 -0500 >>> [0]PETSC ERROR: ./cube_with_hole on a arch-linux2-c-debug named pacotaco >>> by justin Tue Apr 7 19:09:12 2015 >>> [0]PETSC ERROR: Configure options --download-chaco --download-exodusii >>> --download-fblaslapack --download-hdf5 --download-metis --download-mpich >>> --download-mumps --download-netcdf --download-parmetis --download-scalapack >>> --download-triangle --with-cc=gcc --with-cmake=cmake --with-cxx=g++ >>> --with-debugging=1 --with-fc=gfortran --with-valgrind=1 >>> PETSC_ARCH=arch-linux2-c-debug >>> [0]PETSC ERROR: #1 PetscSectionGetDof() line 499 in >>> /home/justin/petsc-dev/src/vec/is/utils/vsectionis.c >>> [0]PETSC ERROR: #2 DMPlexGetConeSize() line 841 in >>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>> [0]PETSC ERROR: #3 DMPlexGetTransitiveClosure() line 1342 in >>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>> [0]PETSC ERROR: #4 DMPlexLabelComplete() line 88 in >>> /home/justin/petsc-dev/src/dm/impls/plex/plexsubmesh.c >>> [0]PETSC ERROR: #5 DMCreateDefaultSection_Plex() line 5605 in >>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>> [0]PETSC ERROR: #6 DMGetDefaultSection() line 3035 in >>> /home/justin/petsc-dev/src/dm/interface/dm.c >>> [0]PETSC ERROR: #7 DMGetDefaultGlobalSection() line 3266 in >>> /home/justin/petsc-dev/src/dm/interface/dm.c >>> [0]PETSC ERROR: #8 DMCreateGlobalVector_Section_Private() line 13 in >>> /home/justin/petsc-dev/src/dm/interface/dmi.c >>> [0]PETSC ERROR: #9 DMCreateGlobalVector_Plex() line 1170 in >>> /home/justin/petsc-dev/src/dm/impls/plex/plexcreate.c >>> [0]PETSC ERROR: #10 DMCreateGlobalVector() line 698 in >>> /home/justin/petsc-dev/src/dm/interface/dm.c >>> [0]PETSC ERROR: #11 main() line 363 in >>> /home/justin/Dropbox/Research_Topics/Code_PETSc/Nonneg/cube_with_hole.c >>> >>> >>> From this error, it seems it's trying to access sieve points that non >>> longer exist. And I think it has to do with the label "marker" still >>> contains data from the interpolated mesh. Is there a way to "uninterpolate" >>> or remove such points? Or is there a better way of approaching this whole >>> thing? >>> >> >> 1) It would be easy to write a small function to throw out the points in >> the label that do not exist in the DM. You >> would just extract each stratumIS and then Clear each invalid point. >> >> 2) However, I do not understand the rationale for this above. You could >> just call MarkBoundaryFaces on the final mesh. >> If you are really trying to preserve a label across regular >> refinement AND you really do not want an interpolated mesh, >> then code up 1). I have yet to see a case where the extra memory for >> edges and faces makes a difference, but it may exist. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> >>> >>> >>> -- >>> Justin Chang >>> PhD Candidate, Civil Engineering - Computational Sciences >>> University of Houston, Department of Civil and Environmental Engineering >>> Houston, TX 77004 >>> (512) 963-3262 >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jchang27 at uh.edu Thu Apr 9 20:14:51 2015 From: jchang27 at uh.edu (Justin Chang) Date: Thu, 9 Apr 2015 20:14:51 -0500 Subject: [petsc-users] Uninterpolating DMLabels? In-Reply-To: References: Message-ID: Matt, thank you for the clarification. One more question somewhat on a related note. I have auxiliary data corresponding to the permeability of a reservoir (for instance, the SPE10 benchmark problem). The data is cell-wise and read from a binary file. I want to refine both the DM and this data. Is there a way to obtain the mapping from the coarse mesh to the fine mesh? What I mean is, given a coarse cell, can I somehow get the transitive closure of the fine cells it created during the refinement process? This would allow me to copy the value of the coarse cell data onto it's children. Thanks, On Thu, Apr 9, 2015 at 7:52 PM, Matthew Knepley wrote: > On Thu, Apr 9, 2015 at 7:21 PM, Justin Chang wrote: > >> 1) I did what you had suggested and it works very well thanks. >> > > Cool. > > >> 2) In my case, calling uninterpolate does make a difference, just not >> necessarily in the solver step (I am using Tao's BLMVM). I attached two >> output files, one containing the timings and summaries with uninterpolating >> and one without uninterpolating. And the source code if you wanted to >> verify the "correctness" of my implementation. Namely, the biggest >> difference I see is in setting up the PetscSection. When uninterpolating >> the data this steps takes 8.64039 seconds but when I do not uninterpolate >> the data it takes 61.2977 seconds. Is this "significant" enough, or is it >> unimportant given the fact that this step occurs only once during any >> simulation? >> > > That is significant. Most of that time is in the operator preallocation > routine. It was cleaner to first gather the point > adjacency information, and then push the dof information on top. If I > aggressively merged the dof data, I could have > pruned a bunch of the work here and gotten much closer to the first time, > at the expense of more complicated code. > I guess this is the right pattern if you know that you will only have > unknowns on cells and vertices. > > >> 3) Also, about DMPlexDistribute(...). Right now it still seems extremely >> slow, hence why I am distributing a coarse mesh and having each processor >> refine its local portion of the mesh. In my current implementation, I have >> rank 0 distribute the mesh via ParMETIS. Will there eventually be a >> methodology to do parallel I/O then redistribution for load balancing or >> does it already exist? >> > > Redistribution for load balancing already exists. The stumbling block > right now is parallel mesh import. However, > the group at ICL (Michael Lange and Gerard Gorman) is working on this, and > we expect to have it working by > the end of summer. Until then, I recommend exactly what you are doing. > > Thanks, > > Matt > > >> Thanks, >> >> On Tue, Apr 7, 2015 at 8:37 PM, Matthew Knepley >> wrote: >> >>> On Tue, Apr 7, 2015 at 7:18 PM, Justin Chang wrote: >>> >>>> Hi all, >>>> >>>> So I am interpolating/uninterpolating my DMPlex mesh. This is the mesh >>>> information for a cell-vertex mesh only: >>>> >>>> DM Object: 1 MPI processes >>>> type: plex >>>> DM_0x84000000_0 in 3 dimensions: >>>> 0-cells: 159 >>>> 3-cells: 592 >>>> Labels: >>>> marker: 1 strata of sizes (100) >>>> depth: 2 strata of sizes (159, 592) >>>> >>>> I am interpolating the mesh with the following lines: >>>> >>>> ierr = DMPlexInterpolate(*dm, &idm);CHKERRQ(ierr); >>>> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); >>>> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); >>>> ierr = DMPlexGetLabel(idm, "marker", &label);CHKERRQ(ierr); >>>> ierr = DMPlexMarkBoundaryFaces(idm,label);CHKERRQ(ierr); >>>> ierr = DMDestroy(dm);CHKERRQ(ierr); >>>> *dm = idm; >>>> >>>> And the resulting mesh: >>>> >>>> DM Object: 1 MPI processes >>>> type: plex >>>> DM_0x84000000_1 in 3 dimensions: >>>> 0-cells: 159 >>>> 1-cells: 845 >>>> 2-cells: 1280 >>>> 3-cells: 592 >>>> Labels: >>>> marker: 1 strata of sizes (292) >>>> depth: 4 strata of sizes (159, 845, 1280, 592) >>>> >>>> The reason I did this is so that incase I decide to refine the mesh >>>> with "-dm_refine " the DM will be sure to include the "marker" >>>> points in the refinement process. Now, if I decide to uninterpolate the >>>> mesh with the following lines: >>>> >>>> ierr = DMPlexUninterpolate(*dm, &idm);CHKERRQ(ierr); >>>> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); >>>> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); >>>> ierr = DMDestroy(dm);CHKERRQ(ierr); >>>> *dm = idm; >>>> >>>> I end up with this mesh: >>>> >>>> DM Object: 1 MPI processes >>>> type: plex >>>> DM_0x84000000_2 in 3 dimensions: >>>> 0-cells: 159 >>>> 3-cells: 592 >>>> Labels: >>>> marker: 1 strata of sizes (292) >>>> depth: 2 strata of sizes (159, 592) >>>> >>>> The problem is that although the cells and vertices have gone back to >>>> the original number, the "marker" label still includes face/edge points. >>>> This gives me an error once I invoke DMCreateGobalVector(...). >>>> >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Argument out of range >>>> [0]PETSC ERROR: Section point 755 should be in [0, 751) >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2528-gbee642f >>>> GIT Date: 2015-03-29 20:36:38 -0500 >>>> [0]PETSC ERROR: ./cube_with_hole on a arch-linux2-c-debug named >>>> pacotaco by justin Tue Apr 7 19:09:12 2015 >>>> [0]PETSC ERROR: Configure options --download-chaco --download-exodusii >>>> --download-fblaslapack --download-hdf5 --download-metis --download-mpich >>>> --download-mumps --download-netcdf --download-parmetis --download-scalapack >>>> --download-triangle --with-cc=gcc --with-cmake=cmake --with-cxx=g++ >>>> --with-debugging=1 --with-fc=gfortran --with-valgrind=1 >>>> PETSC_ARCH=arch-linux2-c-debug >>>> [0]PETSC ERROR: #1 PetscSectionGetDof() line 499 in >>>> /home/justin/petsc-dev/src/vec/is/utils/vsectionis.c >>>> [0]PETSC ERROR: #2 DMPlexGetConeSize() line 841 in >>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>> [0]PETSC ERROR: #3 DMPlexGetTransitiveClosure() line 1342 in >>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>> [0]PETSC ERROR: #4 DMPlexLabelComplete() line 88 in >>>> /home/justin/petsc-dev/src/dm/impls/plex/plexsubmesh.c >>>> [0]PETSC ERROR: #5 DMCreateDefaultSection_Plex() line 5605 in >>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>> [0]PETSC ERROR: #6 DMGetDefaultSection() line 3035 in >>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>> [0]PETSC ERROR: #7 DMGetDefaultGlobalSection() line 3266 in >>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>> [0]PETSC ERROR: #8 DMCreateGlobalVector_Section_Private() line 13 in >>>> /home/justin/petsc-dev/src/dm/interface/dmi.c >>>> [0]PETSC ERROR: #9 DMCreateGlobalVector_Plex() line 1170 in >>>> /home/justin/petsc-dev/src/dm/impls/plex/plexcreate.c >>>> [0]PETSC ERROR: #10 DMCreateGlobalVector() line 698 in >>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>> [0]PETSC ERROR: #11 main() line 363 in >>>> /home/justin/Dropbox/Research_Topics/Code_PETSc/Nonneg/cube_with_hole.c >>>> >>>> >>>> From this error, it seems it's trying to access sieve points that non >>>> longer exist. And I think it has to do with the label "marker" still >>>> contains data from the interpolated mesh. Is there a way to "uninterpolate" >>>> or remove such points? Or is there a better way of approaching this whole >>>> thing? >>>> >>> >>> 1) It would be easy to write a small function to throw out the points in >>> the label that do not exist in the DM. You >>> would just extract each stratumIS and then Clear each invalid point. >>> >>> 2) However, I do not understand the rationale for this above. You could >>> just call MarkBoundaryFaces on the final mesh. >>> If you are really trying to preserve a label across regular >>> refinement AND you really do not want an interpolated mesh, >>> then code up 1). I have yet to see a case where the extra memory for >>> edges and faces makes a difference, but it may exist. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks, >>>> >>>> >>>> >>>> -- >>>> Justin Chang >>>> PhD Candidate, Civil Engineering - Computational Sciences >>>> University of Houston, Department of Civil and Environmental Engineering >>>> Houston, TX 77004 >>>> (512) 963-3262 >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> Justin Chang >> PhD Candidate, Civil Engineering - Computational Sciences >> University of Houston, Department of Civil and Environmental Engineering >> Houston, TX 77004 >> (512) 963-3262 >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Apr 9 20:20:45 2015 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Apr 2015 20:20:45 -0500 Subject: [petsc-users] Uninterpolating DMLabels? In-Reply-To: References: Message-ID: On Thu, Apr 9, 2015 at 8:14 PM, Justin Chang wrote: > Matt, thank you for the clarification. One more question somewhat on a > related note. > > I have auxiliary data corresponding to the permeability of a reservoir > (for instance, the SPE10 benchmark problem). The data is cell-wise and read > from a binary file. I want to refine both the DM and this data. Is there a > way to obtain the mapping from the coarse mesh to the fine mesh? What I > mean is, given a coarse cell, can I somehow get the transitive closure of > the fine cells it created during the refinement process? This would allow > me to copy the value of the coarse cell data onto it's children. > If you are doing regular refinement, there is a simple formula: c_f \in [ C c_c, C (c_c +1) ) where C is the number of cells created from each cell. You can get C from CellRefinerGetAffineTransforms_Internal() which I could promote to a real interface function. Thanks, Matt > Thanks, > > On Thu, Apr 9, 2015 at 7:52 PM, Matthew Knepley wrote: > >> On Thu, Apr 9, 2015 at 7:21 PM, Justin Chang wrote: >> >>> 1) I did what you had suggested and it works very well thanks. >>> >> >> Cool. >> >> >>> 2) In my case, calling uninterpolate does make a difference, just not >>> necessarily in the solver step (I am using Tao's BLMVM). I attached two >>> output files, one containing the timings and summaries with uninterpolating >>> and one without uninterpolating. And the source code if you wanted to >>> verify the "correctness" of my implementation. Namely, the biggest >>> difference I see is in setting up the PetscSection. When uninterpolating >>> the data this steps takes 8.64039 seconds but when I do not uninterpolate >>> the data it takes 61.2977 seconds. Is this "significant" enough, or is it >>> unimportant given the fact that this step occurs only once during any >>> simulation? >>> >> >> That is significant. Most of that time is in the operator preallocation >> routine. It was cleaner to first gather the point >> adjacency information, and then push the dof information on top. If I >> aggressively merged the dof data, I could have >> pruned a bunch of the work here and gotten much closer to the first time, >> at the expense of more complicated code. >> I guess this is the right pattern if you know that you will only have >> unknowns on cells and vertices. >> >> >>> 3) Also, about DMPlexDistribute(...). Right now it still seems extremely >>> slow, hence why I am distributing a coarse mesh and having each processor >>> refine its local portion of the mesh. In my current implementation, I have >>> rank 0 distribute the mesh via ParMETIS. Will there eventually be a >>> methodology to do parallel I/O then redistribution for load balancing or >>> does it already exist? >>> >> >> Redistribution for load balancing already exists. The stumbling block >> right now is parallel mesh import. However, >> the group at ICL (Michael Lange and Gerard Gorman) is working on this, >> and we expect to have it working by >> the end of summer. Until then, I recommend exactly what you are doing. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> >>> On Tue, Apr 7, 2015 at 8:37 PM, Matthew Knepley >>> wrote: >>> >>>> On Tue, Apr 7, 2015 at 7:18 PM, Justin Chang wrote: >>>> >>>>> Hi all, >>>>> >>>>> So I am interpolating/uninterpolating my DMPlex mesh. This is the mesh >>>>> information for a cell-vertex mesh only: >>>>> >>>>> DM Object: 1 MPI processes >>>>> type: plex >>>>> DM_0x84000000_0 in 3 dimensions: >>>>> 0-cells: 159 >>>>> 3-cells: 592 >>>>> Labels: >>>>> marker: 1 strata of sizes (100) >>>>> depth: 2 strata of sizes (159, 592) >>>>> >>>>> I am interpolating the mesh with the following lines: >>>>> >>>>> ierr = DMPlexInterpolate(*dm, &idm);CHKERRQ(ierr); >>>>> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); >>>>> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); >>>>> ierr = DMPlexGetLabel(idm, "marker", &label);CHKERRQ(ierr); >>>>> ierr = DMPlexMarkBoundaryFaces(idm,label);CHKERRQ(ierr); >>>>> ierr = DMDestroy(dm);CHKERRQ(ierr); >>>>> *dm = idm; >>>>> >>>>> And the resulting mesh: >>>>> >>>>> DM Object: 1 MPI processes >>>>> type: plex >>>>> DM_0x84000000_1 in 3 dimensions: >>>>> 0-cells: 159 >>>>> 1-cells: 845 >>>>> 2-cells: 1280 >>>>> 3-cells: 592 >>>>> Labels: >>>>> marker: 1 strata of sizes (292) >>>>> depth: 4 strata of sizes (159, 845, 1280, 592) >>>>> >>>>> The reason I did this is so that incase I decide to refine the mesh >>>>> with "-dm_refine " the DM will be sure to include the "marker" >>>>> points in the refinement process. Now, if I decide to uninterpolate the >>>>> mesh with the following lines: >>>>> >>>>> ierr = DMPlexUninterpolate(*dm, &idm);CHKERRQ(ierr); >>>>> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); >>>>> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); >>>>> ierr = DMDestroy(dm);CHKERRQ(ierr); >>>>> *dm = idm; >>>>> >>>>> I end up with this mesh: >>>>> >>>>> DM Object: 1 MPI processes >>>>> type: plex >>>>> DM_0x84000000_2 in 3 dimensions: >>>>> 0-cells: 159 >>>>> 3-cells: 592 >>>>> Labels: >>>>> marker: 1 strata of sizes (292) >>>>> depth: 2 strata of sizes (159, 592) >>>>> >>>>> The problem is that although the cells and vertices have gone back to >>>>> the original number, the "marker" label still includes face/edge points. >>>>> This gives me an error once I invoke DMCreateGobalVector(...). >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Argument out of range >>>>> [0]PETSC ERROR: Section point 755 should be in [0, 751) >>>>> [0]PETSC ERROR: See >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>>> shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2528-gbee642f >>>>> GIT Date: 2015-03-29 20:36:38 -0500 >>>>> [0]PETSC ERROR: ./cube_with_hole on a arch-linux2-c-debug named >>>>> pacotaco by justin Tue Apr 7 19:09:12 2015 >>>>> [0]PETSC ERROR: Configure options --download-chaco --download-exodusii >>>>> --download-fblaslapack --download-hdf5 --download-metis --download-mpich >>>>> --download-mumps --download-netcdf --download-parmetis --download-scalapack >>>>> --download-triangle --with-cc=gcc --with-cmake=cmake --with-cxx=g++ >>>>> --with-debugging=1 --with-fc=gfortran --with-valgrind=1 >>>>> PETSC_ARCH=arch-linux2-c-debug >>>>> [0]PETSC ERROR: #1 PetscSectionGetDof() line 499 in >>>>> /home/justin/petsc-dev/src/vec/is/utils/vsectionis.c >>>>> [0]PETSC ERROR: #2 DMPlexGetConeSize() line 841 in >>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>> [0]PETSC ERROR: #3 DMPlexGetTransitiveClosure() line 1342 in >>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>> [0]PETSC ERROR: #4 DMPlexLabelComplete() line 88 in >>>>> /home/justin/petsc-dev/src/dm/impls/plex/plexsubmesh.c >>>>> [0]PETSC ERROR: #5 DMCreateDefaultSection_Plex() line 5605 in >>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>> [0]PETSC ERROR: #6 DMGetDefaultSection() line 3035 in >>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>> [0]PETSC ERROR: #7 DMGetDefaultGlobalSection() line 3266 in >>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>> [0]PETSC ERROR: #8 DMCreateGlobalVector_Section_Private() line 13 in >>>>> /home/justin/petsc-dev/src/dm/interface/dmi.c >>>>> [0]PETSC ERROR: #9 DMCreateGlobalVector_Plex() line 1170 in >>>>> /home/justin/petsc-dev/src/dm/impls/plex/plexcreate.c >>>>> [0]PETSC ERROR: #10 DMCreateGlobalVector() line 698 in >>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>> [0]PETSC ERROR: #11 main() line 363 in >>>>> /home/justin/Dropbox/Research_Topics/Code_PETSc/Nonneg/cube_with_hole.c >>>>> >>>>> >>>>> From this error, it seems it's trying to access sieve points that non >>>>> longer exist. And I think it has to do with the label "marker" still >>>>> contains data from the interpolated mesh. Is there a way to "uninterpolate" >>>>> or remove such points? Or is there a better way of approaching this whole >>>>> thing? >>>>> >>>> >>>> 1) It would be easy to write a small function to throw out the points >>>> in the label that do not exist in the DM. You >>>> would just extract each stratumIS and then Clear each invalid point. >>>> >>>> 2) However, I do not understand the rationale for this above. You could >>>> just call MarkBoundaryFaces on the final mesh. >>>> If you are really trying to preserve a label across regular >>>> refinement AND you really do not want an interpolated mesh, >>>> then code up 1). I have yet to see a case where the extra memory >>>> for edges and faces makes a difference, but it may exist. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thanks, >>>>> >>>>> >>>>> >>>>> -- >>>>> Justin Chang >>>>> PhD Candidate, Civil Engineering - Computational Sciences >>>>> University of Houston, Department of Civil and Environmental >>>>> Engineering >>>>> Houston, TX 77004 >>>>> (512) 963-3262 >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >>> >>> -- >>> Justin Chang >>> PhD Candidate, Civil Engineering - Computational Sciences >>> University of Houston, Department of Civil and Environmental Engineering >>> Houston, TX 77004 >>> (512) 963-3262 >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Apr 9 20:50:17 2015 From: jed at jedbrown.org (Jed Brown) Date: Thu, 09 Apr 2015 19:50:17 -0600 Subject: [petsc-users] Using OpenMP threads with PETSc In-Reply-To: References: <876195geea.fsf@jedbrown.org> <87wq1lexug.fsf@jedbrown.org> <87oamwgb45.fsf@jedbrown.org> Message-ID: <87iod4g3ae.fsf@jedbrown.org> Lucas Clemente Vella writes: > Will be no worse than separated MPI processes competing for the same > cache slot. There can be LLC sharing, but L1 and usually L2 are private per core and are used less efficiently when each process has non-contiguous blocks. > At least with threads there is a chance that different tasks will hit > the same cached memory. I do believe there should be a smart way to > control and optimize thread work proximity on OMP for loops, i.e.: > > #pragma omp parallel for > for(size_t i = 0; i < size; ++size) { > // something not dependent on previous steps > } > > Each two threads working on ranges closely together should run on the > two hyperthreads of the same core, to maximize cache reuse. Because > this usage pattern of OpenMP, it seems to me that it is already > unlikely that two threads will be working to far off each other, but > if I wanted this level of control, I should now be hacking some OpenMP > implementation and/or the kernel. Test this hypothesis. > On the other hand, pthread interface favors more loosely coupled > tasks, that may yield worse cache reuse, but I confess I didn't take > the time to look inside PETSc how each of the threading libraries was > used. Actually, the pthread threadcomm is not that much different from this OpenMP strategy. Either could be used at a higher level, but the user interface would be different and something nobody says they want. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jchang27 at uh.edu Fri Apr 10 01:01:27 2015 From: jchang27 at uh.edu (Justin Chang) Date: Fri, 10 Apr 2015 01:01:27 -0500 Subject: [petsc-users] Uninterpolating DMLabels? In-Reply-To: References: Message-ID: Matt, I am not sure I follow what you're saying here. Currently I am working with 3D simplex elements so if the variable numSubcells is what you're referring to in CellRefinerGetAffineTransforms_Internal(), there's no example for REFINER_SIMPLEX_3D. What would it be in that case, 7? What I am looking for is a function and/or a set of algorithms where given a sieve point m from the original coarse mesh, I can obtain the corresponding set of (fine mesh) sieve points. I am assuming c_f \in [C c_c, C (c_c + 1) ) describes just that? Is there an existing DMPlex related function that implements this directly? Because I don't see how CellRefinerGetAffineTransforms_Internal() will help. Thanks, On Thu, Apr 9, 2015 at 8:20 PM, Matthew Knepley wrote: > On Thu, Apr 9, 2015 at 8:14 PM, Justin Chang wrote: > >> Matt, thank you for the clarification. One more question somewhat on a >> related note. >> >> I have auxiliary data corresponding to the permeability of a reservoir >> (for instance, the SPE10 benchmark problem). The data is cell-wise and read >> from a binary file. I want to refine both the DM and this data. Is there a >> way to obtain the mapping from the coarse mesh to the fine mesh? What I >> mean is, given a coarse cell, can I somehow get the transitive closure of >> the fine cells it created during the refinement process? This would allow >> me to copy the value of the coarse cell data onto it's children. >> > > If you are doing regular refinement, there is a simple formula: > > c_f \in [ C c_c, C (c_c +1) ) > > where C is the number of cells created from each cell. You can get C from > > CellRefinerGetAffineTransforms_Internal() > > which I could promote to a real interface function. > > Thanks, > > Matt > > >> Thanks, >> >> On Thu, Apr 9, 2015 at 7:52 PM, Matthew Knepley >> wrote: >> >>> On Thu, Apr 9, 2015 at 7:21 PM, Justin Chang wrote: >>> >>>> 1) I did what you had suggested and it works very well thanks. >>>> >>> >>> Cool. >>> >>> >>>> 2) In my case, calling uninterpolate does make a difference, just not >>>> necessarily in the solver step (I am using Tao's BLMVM). I attached two >>>> output files, one containing the timings and summaries with uninterpolating >>>> and one without uninterpolating. And the source code if you wanted to >>>> verify the "correctness" of my implementation. Namely, the biggest >>>> difference I see is in setting up the PetscSection. When uninterpolating >>>> the data this steps takes 8.64039 seconds but when I do not uninterpolate >>>> the data it takes 61.2977 seconds. Is this "significant" enough, or is it >>>> unimportant given the fact that this step occurs only once during any >>>> simulation? >>>> >>> >>> That is significant. Most of that time is in the operator preallocation >>> routine. It was cleaner to first gather the point >>> adjacency information, and then push the dof information on top. If I >>> aggressively merged the dof data, I could have >>> pruned a bunch of the work here and gotten much closer to the first >>> time, at the expense of more complicated code. >>> I guess this is the right pattern if you know that you will only have >>> unknowns on cells and vertices. >>> >>> >>>> 3) Also, about DMPlexDistribute(...). Right now it still seems >>>> extremely slow, hence why I am distributing a coarse mesh and having each >>>> processor refine its local portion of the mesh. In my current >>>> implementation, I have rank 0 distribute the mesh via ParMETIS. Will there >>>> eventually be a methodology to do parallel I/O then redistribution for load >>>> balancing or does it already exist? >>>> >>> >>> Redistribution for load balancing already exists. The stumbling block >>> right now is parallel mesh import. However, >>> the group at ICL (Michael Lange and Gerard Gorman) is working on this, >>> and we expect to have it working by >>> the end of summer. Until then, I recommend exactly what you are doing. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks, >>>> >>>> On Tue, Apr 7, 2015 at 8:37 PM, Matthew Knepley >>>> wrote: >>>> >>>>> On Tue, Apr 7, 2015 at 7:18 PM, Justin Chang wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> So I am interpolating/uninterpolating my DMPlex mesh. This is the >>>>>> mesh information for a cell-vertex mesh only: >>>>>> >>>>>> DM Object: 1 MPI processes >>>>>> type: plex >>>>>> DM_0x84000000_0 in 3 dimensions: >>>>>> 0-cells: 159 >>>>>> 3-cells: 592 >>>>>> Labels: >>>>>> marker: 1 strata of sizes (100) >>>>>> depth: 2 strata of sizes (159, 592) >>>>>> >>>>>> I am interpolating the mesh with the following lines: >>>>>> >>>>>> ierr = DMPlexInterpolate(*dm, &idm);CHKERRQ(ierr); >>>>>> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); >>>>>> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); >>>>>> ierr = DMPlexGetLabel(idm, "marker", &label);CHKERRQ(ierr); >>>>>> ierr = DMPlexMarkBoundaryFaces(idm,label);CHKERRQ(ierr); >>>>>> ierr = DMDestroy(dm);CHKERRQ(ierr); >>>>>> *dm = idm; >>>>>> >>>>>> And the resulting mesh: >>>>>> >>>>>> DM Object: 1 MPI processes >>>>>> type: plex >>>>>> DM_0x84000000_1 in 3 dimensions: >>>>>> 0-cells: 159 >>>>>> 1-cells: 845 >>>>>> 2-cells: 1280 >>>>>> 3-cells: 592 >>>>>> Labels: >>>>>> marker: 1 strata of sizes (292) >>>>>> depth: 4 strata of sizes (159, 845, 1280, 592) >>>>>> >>>>>> The reason I did this is so that incase I decide to refine the mesh >>>>>> with "-dm_refine " the DM will be sure to include the "marker" >>>>>> points in the refinement process. Now, if I decide to uninterpolate the >>>>>> mesh with the following lines: >>>>>> >>>>>> ierr = DMPlexUninterpolate(*dm, &idm);CHKERRQ(ierr); >>>>>> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); >>>>>> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); >>>>>> ierr = DMDestroy(dm);CHKERRQ(ierr); >>>>>> *dm = idm; >>>>>> >>>>>> I end up with this mesh: >>>>>> >>>>>> DM Object: 1 MPI processes >>>>>> type: plex >>>>>> DM_0x84000000_2 in 3 dimensions: >>>>>> 0-cells: 159 >>>>>> 3-cells: 592 >>>>>> Labels: >>>>>> marker: 1 strata of sizes (292) >>>>>> depth: 2 strata of sizes (159, 592) >>>>>> >>>>>> The problem is that although the cells and vertices have gone back to >>>>>> the original number, the "marker" label still includes face/edge points. >>>>>> This gives me an error once I invoke DMCreateGobalVector(...). >>>>>> >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: Argument out of range >>>>>> [0]PETSC ERROR: Section point 755 should be in [0, 751) >>>>>> [0]PETSC ERROR: See >>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>>>> shooting. >>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2528-gbee642f >>>>>> GIT Date: 2015-03-29 20:36:38 -0500 >>>>>> [0]PETSC ERROR: ./cube_with_hole on a arch-linux2-c-debug named >>>>>> pacotaco by justin Tue Apr 7 19:09:12 2015 >>>>>> [0]PETSC ERROR: Configure options --download-chaco >>>>>> --download-exodusii --download-fblaslapack --download-hdf5 --download-metis >>>>>> --download-mpich --download-mumps --download-netcdf --download-parmetis >>>>>> --download-scalapack --download-triangle --with-cc=gcc --with-cmake=cmake >>>>>> --with-cxx=g++ --with-debugging=1 --with-fc=gfortran --with-valgrind=1 >>>>>> PETSC_ARCH=arch-linux2-c-debug >>>>>> [0]PETSC ERROR: #1 PetscSectionGetDof() line 499 in >>>>>> /home/justin/petsc-dev/src/vec/is/utils/vsectionis.c >>>>>> [0]PETSC ERROR: #2 DMPlexGetConeSize() line 841 in >>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>>> [0]PETSC ERROR: #3 DMPlexGetTransitiveClosure() line 1342 in >>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>>> [0]PETSC ERROR: #4 DMPlexLabelComplete() line 88 in >>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plexsubmesh.c >>>>>> [0]PETSC ERROR: #5 DMCreateDefaultSection_Plex() line 5605 in >>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>>> [0]PETSC ERROR: #6 DMGetDefaultSection() line 3035 in >>>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>>> [0]PETSC ERROR: #7 DMGetDefaultGlobalSection() line 3266 in >>>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>>> [0]PETSC ERROR: #8 DMCreateGlobalVector_Section_Private() line 13 in >>>>>> /home/justin/petsc-dev/src/dm/interface/dmi.c >>>>>> [0]PETSC ERROR: #9 DMCreateGlobalVector_Plex() line 1170 in >>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plexcreate.c >>>>>> [0]PETSC ERROR: #10 DMCreateGlobalVector() line 698 in >>>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>>> [0]PETSC ERROR: #11 main() line 363 in >>>>>> /home/justin/Dropbox/Research_Topics/Code_PETSc/Nonneg/cube_with_hole.c >>>>>> >>>>>> >>>>>> From this error, it seems it's trying to access sieve points that non >>>>>> longer exist. And I think it has to do with the label "marker" still >>>>>> contains data from the interpolated mesh. Is there a way to "uninterpolate" >>>>>> or remove such points? Or is there a better way of approaching this whole >>>>>> thing? >>>>>> >>>>> >>>>> 1) It would be easy to write a small function to throw out the points >>>>> in the label that do not exist in the DM. You >>>>> would just extract each stratumIS and then Clear each invalid >>>>> point. >>>>> >>>>> 2) However, I do not understand the rationale for this above. You >>>>> could just call MarkBoundaryFaces on the final mesh. >>>>> If you are really trying to preserve a label across regular >>>>> refinement AND you really do not want an interpolated mesh, >>>>> then code up 1). I have yet to see a case where the extra memory >>>>> for edges and faces makes a difference, but it may exist. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thanks, >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Justin Chang >>>>>> PhD Candidate, Civil Engineering - Computational Sciences >>>>>> University of Houston, Department of Civil and Environmental >>>>>> Engineering >>>>>> Houston, TX 77004 >>>>>> (512) 963-3262 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>>> >>>> -- >>>> Justin Chang >>>> PhD Candidate, Civil Engineering - Computational Sciences >>>> University of Houston, Department of Civil and Environmental Engineering >>>> Houston, TX 77004 >>>> (512) 963-3262 >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> Justin Chang >> PhD Candidate, Civil Engineering - Computational Sciences >> University of Houston, Department of Civil and Environmental Engineering >> Houston, TX 77004 >> (512) 963-3262 >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaiser at igpm.rwth-aachen.de Fri Apr 10 02:02:08 2015 From: kaiser at igpm.rwth-aachen.de (Klaus Kaiser) Date: Fri, 10 Apr 2015 09:02:08 +0200 Subject: [petsc-users] Problem with MPI, MatAXPY and SAME_NONZERO_PATTERN In-Reply-To: References: <55156DA7.6010602@igpm.rwth-aachen.de> <55157460.4080200@igpm.rwth-aachen.de> <5515805B.3060406@igpm.rwth-aachen.de> Message-ID: <55277570.1050801@igpm.rwth-aachen.de> Of course this is not my real world example, and not the best way to fill a matrix. I just wanted to quickly fill the matrices and give an example for my problem. Therefore it does not matter how the matrices look like, the problem is that the resulting sum of these three matrices is wrong. (e.g. all matricies have values beside the diagonal block which have the value a, therefore the resulting sum should have the value 3*a, but in my computation the result is a. Some values are computed correctly, the entry (1,1) for example, but most are wrong.) Overall the interessting part is: MatAXPY(Ah,1.,At,SAME_NONZERO_PATTERN); MatAXPY(A,1.,Ah,SAME_NONZERO_PATTERN); I made an matrix output for the Matrix A before, after this calculation and at the end of the code. With the result, that the matrices after computing the sum are wrong. Best Klaus On 04/09/2015 09:11 PM, Matthew Knepley wrote: > On Fri, Mar 27, 2015 at 11:07 AM, Klaus Kaiser > > wrote: > > Hallo Matthew, > > here is a short example I used for 8 cores: > > int d_nnz[3]; > int o_nnz[3]; > d_nnz[0] = 3; d_nnz[1] = 3; d_nnz[2] = 3; > o_nnz[0] = 6; o_nnz[1] = 6; o_nnz[2] = 6; > ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, > d_nnz, 0, o_nnz,&A); > ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); > ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); > ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); > ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, > d_nnz, 0, o_nnz,&Ah); > ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0, > d_nnz, 0, o_nnz,&At); > > std::vector insert(3*3*3*3, 1.0); > for(int i=0;i<8;++i) > { > int rows[3] = {i,i+1,i+3}; > int cols[3] = {i,i+1,i+3}; > > MatSetValuesBlocked(A, 3, rows, 3, cols, &insert[0], > ADD_VALUES); > } > > > I do not understand this example. All the procs (8) are filling up > only the first 32 rows. Is that > what you intended? Can you tell me what output you expect? > > Thanks, > > Matt > > MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); > MatDuplicate(A,MAT_COPY_VALUES,&Ah); > MatDuplicate(A,MAT_COPY_VALUES,&At); > MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY); > MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY); > > MatAXPY(Ah,1.,At,SAME_NONZERO_PATTERN); > MatAXPY(A,1.,Ah,SAME_NONZERO_PATTERN); > > MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); > > The result is, that only some values are computed correctly. > > Best and Thank you a lot > > Klaus > > > > > On 03/27/2015 04:26 PM, Matthew Knepley wrote: >> On Fri, Mar 27, 2015 at 10:16 AM, Klaus Kaiser >> > >> wrote: >> >> Hallo Matthew, >> >> thanks for your fast response. With "no beside the nonzero >> structure" I meant, that I do not create a different non-zero >> structure while I'm adding values to my matrices. >> >> I also tried MAT_COPY_VALUES and the result was the same. >> >> >> Please send the small example and we will figure out what is >> going wrong. >> >> Thanks, >> >> Matt >> >> I do not get a error message but the resulting matrix is >> wrong. Here I have a short example. The first 9 rows and >> first 3 colomn of the matrices Ah, At and Ah+At on the first >> processor (of 8): >> >> Ah: >> 0: 60.3553 -0.249975 2.77556e-17 >> 1: 0 60.3553 0 >> 2: 0 0 60.3553 >> 3: 17.6777 0.374962 0.124987 >> 4: 0 17.6777 0 >> 5: 0 0 17.6777 >> 6: -7.32233 -0.124987 0.374962 >> 7: 0 -7.32233 0 >> 8: 0 0 -7.32233 >> >> >> At: >> 0: 0 0 0 >> 1: 2500 0 0 >> 2: -4.54747e-13 0 0 >> 3: 0 0 0 >> 4: 1250 0 0 >> 5: 1250 0 0 >> 6: 0 0 0 >> 7: 1250 0 0 >> 8: 3750 0 0 >> >> Ah+At >> 0: 60.3553 -0.249975 2.77556e-17 >> 1: 2500 60.3553 0 >> 2: -4.54747e-13 0 60.3553 >> 3: 17.6777 0.374962 0.124987 >> 4: 0 17.6777 0 >> 5: 0 0 17.6777 >> 6: -7.32233 -0.124987 0.374962 >> 7: 0 -7.32233 0 >> 8: 0 0 -7.32233 >> >> you can see the first 3 rows of the resulting matrix looks >> exactly like what I would expect, but the last 6 rows only >> consists of the values of Ah. When you would also look on the >> matrix A and A+Ah+At you would also see, that values where >> both matrices have an nonzero entry are not sum correctly. >> >> Best >> >> Klaus >> >> On 03/27/2015 03:59 PM, Matthew Knepley wrote: >>> On Fri, Mar 27, 2015 at 9:48 AM, Klaus Kaiser >>> >> > wrote: >>> >>> Hallo, >>> >>> I have a strange behavior in my code concerning the >>> function MatAXPY. I create 3 different Matrices >>> >>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, >>> local_size, local_size, system_size, system_size, 0, >>> d_nnz, 0, o_nnz,&A); >>> ierr = >>> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); >>> ierr = >>> MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); >>> ierr = >>> MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); >>> >>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, >>> local_size, local_size, system_size, system_size, 0, >>> d_nnz, 0, o_nnz,&At); >>> ierr = >>> MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); >>> ierr = >>> MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); >>> ierr = >>> MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); >>> >>> ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, >>> local_size, local_size, system_size, system_size, 0, >>> d_nnz, 0, o_nnz,&Ah); >>> ierr = >>> MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE); >>> ierr = >>> MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE); >>> ierr = >>> MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); >>> >>> >>> These creations are superfluous since you use MatDuplicate() >>> below. >>> >>> and want to sum these three matrixes with different >>> factors. First I fill the Matrix A with some values, and >>> duplicate the structure of A to the other two matrices: >>> >>> MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); >>> MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); >>> MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah); >>> MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At); >>> MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY); >>> MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY); >>> MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY); >>> MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY); >>> >>> >>> After this I fill the matrices At and Ah with some other >>> values, which are not beside the non zero structure (I >>> also tried with just copying the Matrix A). Now after >>> another MatAssembly >>> >>> >>> I do not understand "no beside the nonzero structure". Do >>> you mean that the nonzero structure is the same? >>> >>> Can you first test with MAT_COPY_VALUES? >>> >>> I want to add these Matrices in the form A+c*(Ah+d*At): >>> >>> >>> MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN); >>> MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN); >>> >>> When I run the method with mpi and one core everything >>> works fine. Starting the same method with more cores, >>> the sum of the matrices fails. It seems like some values >>> are added >>> >>> >>> Please send the full output of a failure when you use >>> MAT_COPY_VALUES on 2 procs. >>> >>> Thanks, >>> >>> Matt >>> >>> correctly and many values are missed. Using >>> DIFFERENT_NONZERO_STRUCTURE leads to the right behavior >>> in the multi-core case, but is very slow. I checked with >>> a viewer if all matrices have the same nonzero structure >>> and this is the case. >>> >>> Does anyone know why this fails, or do I have made any >>> wrong thoughts? >>> >>> I'm corrently working with a petsc version (Petsc >>> Release Version 3.3.0, Patch 5, Sat Dec 1 15:10:41 CST >>> 2012), I looked into the changelogs up to the current >>> version and did not find any note about MatAXPY or MatAYPX. >>> >>> >>> Best and Thanks a lot for your help >>> >>> Klaus >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin >>> their experiments is infinitely more interesting than any >>> results to which their experiments lead. >>> -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From rudraa at umich.edu Fri Apr 10 02:41:01 2015 From: rudraa at umich.edu (Shiva Rudraraju) Date: Fri, 10 Apr 2015 03:41:01 -0400 Subject: [petsc-users] Jacobian free solve and preconditioning Message-ID: Hi, My problem (finite strain mechanics with additional higher order gradient terms) involves a very complex jacobian. To avoid mistakes from hand coding the jacobian I use AD (Sacado/Adept). However, both the hand coded jacobian evaluation and AD computation of jacobian are computationally very expensive and compete-with/exceed the solve time. So I am looking at Jacobian free approchaes?.and would want to have some sort of matrix free preconditioning. I am thinking of using: -ksp_type fgmres -pc_type ksp -ksp_ksp_type gmres -ksp_pc_type jacobi -snes_mf But -snes_mf will not use any preconditioner, so I want to try something like -snes_mf_operator, but which uses GMRES/BiCG as a preconditioner without the need for explicit jacobian computation. However, snes_mf_operator requires the Jacobian function?.. Any way to obtain jacobian free preconditioning? Thanks, Shiva Shiva Rudraraju Asst. Research Scientist, University of Michigan Ann Arbor. From C.Klaij at marin.nl Fri Apr 10 06:48:33 2015 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Fri, 10 Apr 2015 11:48:33 +0000 Subject: [petsc-users] monitoring the convergence of fieldsplit 0 and 1 Message-ID: <1428666513941.72745@marin.nl> A question when using PCFieldSplit: for each linear iteration of the system, how many iterations for fielsplit 0 and 1? One way to find out is to run with -ksp_monitor, -fieldsplit_0_ksp_monitor and -fieldsplit_0_ksp_monitor. This gives the complete convergence history. Another way, suggested by Matt, is to use -ksp_monitor, -fieldsplit_0_ksp_converged_reason and -fieldsplit_1_ksp_converged_reason. This gives only the totals for fieldsplit 0 and 1 (but without saying for which one). Both ways require to somehow process the output, which is a bit inconvenient. Could KSPGetResidualHistory perhaps return (some) information on the subsystems' convergence for processing inside the code? Chris dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From siddhesh4godbole at gmail.com Fri Apr 10 06:46:36 2015 From: siddhesh4godbole at gmail.com (siddhesh godbole) Date: Fri, 10 Apr 2015 17:16:36 +0530 Subject: [petsc-users] Matlab socket connection Message-ID: Hello, I am having difficulty in launching petsc program from matlab. According to the error the problem is with the sopen function. error given in the matlab command window is that there are not enough arguments for sopen in the line *75*. i am attaching sopen.m file with mail. can you help me out with this as there is no one to solve my doubt in this area. i am also attaching my program file eps13D.m the sopen call is made by the function PetscOpenSocket in which line 10 subsequently also shoots an error. Thanks a lot *Siddhesh M Godbole* 5th year Dual Degree, Civil Eng & Applied Mech. IIT Madras -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: eps13D.m Type: text/x-objcsrc Size: 849 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: sopen.m Type: text/x-objcsrc Size: 504878 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PetscOpenSocket.m Type: text/x-objcsrc Size: 500 bytes Desc: not available URL: From bsmith at mcs.anl.gov Fri Apr 10 07:48:56 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 10 Apr 2015 07:48:56 -0500 Subject: [petsc-users] monitoring the convergence of fieldsplit 0 and 1 In-Reply-To: <1428666513941.72745@marin.nl> References: <1428666513941.72745@marin.nl> Message-ID: <75D79823-7AE0-47A7-BE9E-15AB81C3581E@mcs.anl.gov> Chris, It appears you should call PCFieldSplitGetSubKSP() and then get the information you want out of the individual KSPs. If this doesn't work please let us know. Barry > On Apr 10, 2015, at 6:48 AM, Klaij, Christiaan wrote: > > A question when using PCFieldSplit: for each linear iteration of > the system, how many iterations for fielsplit 0 and 1? > > One way to find out is to run with -ksp_monitor, > -fieldsplit_0_ksp_monitor and -fieldsplit_0_ksp_monitor. This > gives the complete convergence history. > > Another way, suggested by Matt, is to use -ksp_monitor, > -fieldsplit_0_ksp_converged_reason and > -fieldsplit_1_ksp_converged_reason. This gives only the totals > for fieldsplit 0 and 1 (but without saying for which one). > > Both ways require to somehow process the output, which is a bit > inconvenient. Could KSPGetResidualHistory perhaps return (some) > information on the subsystems' convergence for processing inside > the code? > > Chris > > > dr. ir. Christiaan Klaij > CFD Researcher > Research & Development > E mailto:C.Klaij at marin.nl > T +31 317 49 33 44 > > > MARIN > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > From bsmith at mcs.anl.gov Fri Apr 10 07:53:06 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 10 Apr 2015 07:53:06 -0500 Subject: [petsc-users] Matlab socket connection In-Reply-To: References: Message-ID: <38437ECC-F9DE-4577-B462-9D149942EC0A@mcs.anl.gov> Ahh, bad name spacing. PETSc has an sopen mex file for opening socket connections, some completely different other package has an sopen.m which is being found first. Make sure the PETSc sopen mex file is in the MATLABPATH before the other one or better yet make sure the other sopen.m is not in the MATLABPATH Barry > On Apr 10, 2015, at 6:46 AM, siddhesh godbole wrote: > > Hello, > > I am having difficulty in launching petsc program from matlab. According to the error the problem is with the sopen function. error given in the matlab command window is that there are not enough arguments for sopen in the line 75. i am attaching sopen.m file with mail. can you help me out with this as there is no one to solve my doubt in this area. i am also attaching my program file eps13D.m > the sopen call is made by the function PetscOpenSocket in which line 10 subsequently also shoots an error. > > Thanks a lot > > > Siddhesh M Godbole > > 5th year Dual Degree, > Civil Eng & Applied Mech. > IIT Madras > From C.Klaij at marin.nl Fri Apr 10 08:07:23 2015 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Fri, 10 Apr 2015 13:07:23 +0000 Subject: [petsc-users] monitoring the convergence of fieldsplit 0 and 1 In-Reply-To: <75D79823-7AE0-47A7-BE9E-15AB81C3581E@mcs.anl.gov> References: <1428666513941.72745@marin.nl>, <75D79823-7AE0-47A7-BE9E-15AB81C3581E@mcs.anl.gov> Message-ID: <1428671243078.94@marin.nl> Barry, Sure, I can call PCFieldSplitGetSubKSP() to get the fieldsplit_0 ksp and then KSPGetIterationNumber, but what does this number mean? It appears to be the number of iterations of the last time that the subsystem was solved, right? If so, this corresponds to the last iteration of the coupled system, how about all the previous iterations? Chris ________________________________________ From: Barry Smith Sent: Friday, April 10, 2015 2:48 PM To: Klaij, Christiaan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] monitoring the convergence of fieldsplit 0 and 1 Chris, It appears you should call PCFieldSplitGetSubKSP() and then get the information you want out of the individual KSPs. If this doesn't work please let us know. Barry > On Apr 10, 2015, at 6:48 AM, Klaij, Christiaan wrote: > > A question when using PCFieldSplit: for each linear iteration of > the system, how many iterations for fielsplit 0 and 1? > > One way to find out is to run with -ksp_monitor, > -fieldsplit_0_ksp_monitor and -fieldsplit_0_ksp_monitor. This > gives the complete convergence history. > > Another way, suggested by Matt, is to use -ksp_monitor, > -fieldsplit_0_ksp_converged_reason and > -fieldsplit_1_ksp_converged_reason. This gives only the totals > for fieldsplit 0 and 1 (but without saying for which one). > > Both ways require to somehow process the output, which is a bit > inconvenient. Could KSPGetResidualHistory perhaps return (some) > information on the subsystems' convergence for processing inside > the code? > > Chris > > > dr. ir. Christiaan Klaij > CFD Researcher > Research & Development > E mailto:C.Klaij at marin.nl > T +31 317 49 33 44 > > > MARIN > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > From georgesamarasdit at gmail.com Fri Apr 10 08:42:33 2015 From: georgesamarasdit at gmail.com (Georgios Samaras) Date: Fri, 10 Apr 2015 16:42:33 +0300 Subject: [petsc-users] PETSc for distributed computing? In-Reply-To: <382DADCD-513A-4591-9730-425DD55FFB43@gmail.com> References: <382DADCD-513A-4591-9730-425DD55FFB43@gmail.com> Message-ID: We (my professor (cc) and I) would like to solve a linear system involving a dense matrix, but we have not yet think of the number of right hand sides, but we are working on it. Thanks, Georgios Samaras On Thu, Apr 9, 2015 at 4:31 PM, Patrick Sanan wrote: > Do you want an explicit inverse, or just to solve linear systems involving > a dense matrix? If so, how many right hand sides? > > PETSc itself focuses on sparse linear algebra, but it includes an > interface to the dense linear algebra library Elemental. > > Il giorno 09/apr/2015, alle ore 03:06 PM, Georgios Samaras < > georgesamarasdit at gmail.com> ha scritto: > > > Dear all, > > > > I am looking for a C/C++ linear algebra package for performing > inversion of a large dense matrix in a distributed environment. Is your > package suitable for this goal? > > > > Thanks in advance, > > Georgios Samaras > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Fri Apr 10 10:22:41 2015 From: hzhang at mcs.anl.gov (Hong) Date: Fri, 10 Apr 2015 10:22:41 -0500 Subject: [petsc-users] PETSc for distributed computing? In-Reply-To: References: <382DADCD-513A-4591-9730-425DD55FFB43@gmail.com> Message-ID: Georgios Samaras : > We (my professor (cc) and I) would like to solve a linear system involving > a dense matrix, but we have not yet think of the number of right hand > sides, but we are working on it. > As Barry mentioned, you can either use elemental directly, or use petsc-elemental interface. An example can be found at ~petsc/src/mat/examples/tests/ex145.c You may use petsc KSP interface instead. I just modified ~petsc/src/ksp/ksp/examples/tutorials/ex2.c https://bitbucket.org/petsc/petsc/commits/1cc583424d1965a1a245571240baab92605aba58 so this example can be run with elemental with runtime options $ mpiexec -n 3 ./ex2 -pc_type lu -pc_factor_mat_solver_package elemental -mat_type elemental Norm of error 2.81086e-15 iterations 1 Suggest using petsc-dev (master branch) for petsc-elemental interface. Hong > Thanks, > Georgios Samaras > > On Thu, Apr 9, 2015 at 4:31 PM, Patrick Sanan > wrote: > >> Do you want an explicit inverse, or just to solve linear systems >> involving a dense matrix? If so, how many right hand sides? >> >> PETSc itself focuses on sparse linear algebra, but it includes an >> interface to the dense linear algebra library Elemental. >> >> Il giorno 09/apr/2015, alle ore 03:06 PM, Georgios Samaras < >> georgesamarasdit at gmail.com> ha scritto: >> >> > Dear all, >> > >> > I am looking for a C/C++ linear algebra package for performing >> inversion of a large dense matrix in a distributed environment. Is your >> package suitable for this goal? >> > >> > Thanks in advance, >> > Georgios Samaras >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 10 11:38:56 2015 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 10 Apr 2015 11:38:56 -0500 Subject: [petsc-users] Uninterpolating DMLabels? In-Reply-To: References: Message-ID: On Fri, Apr 10, 2015 at 1:01 AM, Justin Chang wrote: > Matt, > > I am not sure I follow what you're saying here. Currently I am working > with 3D simplex elements so if the variable numSubcells is what you're > referring to in CellRefinerGetAffineTransforms_Internal(), there's no > example for REFINER_SIMPLEX_3D. What would it be in that case, 7? > Its 8. There are 8 fine cells for every coarse cell. > What I am looking for is a function and/or a set of algorithms where given > a sieve point m from the original coarse mesh, I can obtain the > corresponding set of (fine mesh) sieve points. I am assuming c_f \in [C > c_c, C (c_c + 1) ) describes just that? > Yes, C = 8. > Is there an existing DMPlex related function that implements this > directly? Because I don't see how CellRefinerGetAffineTransforms_Internal() > will help. > I would just give you C. Yes, I would just write it. It should be 1 line. Thanks, Matt > > Thanks, > > > On Thu, Apr 9, 2015 at 8:20 PM, Matthew Knepley wrote: > >> On Thu, Apr 9, 2015 at 8:14 PM, Justin Chang wrote: >> >>> Matt, thank you for the clarification. One more question somewhat on a >>> related note. >>> >>> I have auxiliary data corresponding to the permeability of a reservoir >>> (for instance, the SPE10 benchmark problem). The data is cell-wise and read >>> from a binary file. I want to refine both the DM and this data. Is there a >>> way to obtain the mapping from the coarse mesh to the fine mesh? What I >>> mean is, given a coarse cell, can I somehow get the transitive closure of >>> the fine cells it created during the refinement process? This would allow >>> me to copy the value of the coarse cell data onto it's children. >>> >> >> If you are doing regular refinement, there is a simple formula: >> >> c_f \in [ C c_c, C (c_c +1) ) >> >> where C is the number of cells created from each cell. You can get C from >> >> CellRefinerGetAffineTransforms_Internal() >> >> which I could promote to a real interface function. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> >>> On Thu, Apr 9, 2015 at 7:52 PM, Matthew Knepley >>> wrote: >>> >>>> On Thu, Apr 9, 2015 at 7:21 PM, Justin Chang wrote: >>>> >>>>> 1) I did what you had suggested and it works very well thanks. >>>>> >>>> >>>> Cool. >>>> >>>> >>>>> 2) In my case, calling uninterpolate does make a difference, just not >>>>> necessarily in the solver step (I am using Tao's BLMVM). I attached two >>>>> output files, one containing the timings and summaries with uninterpolating >>>>> and one without uninterpolating. And the source code if you wanted to >>>>> verify the "correctness" of my implementation. Namely, the biggest >>>>> difference I see is in setting up the PetscSection. When uninterpolating >>>>> the data this steps takes 8.64039 seconds but when I do not uninterpolate >>>>> the data it takes 61.2977 seconds. Is this "significant" enough, or is it >>>>> unimportant given the fact that this step occurs only once during any >>>>> simulation? >>>>> >>>> >>>> That is significant. Most of that time is in the operator preallocation >>>> routine. It was cleaner to first gather the point >>>> adjacency information, and then push the dof information on top. If I >>>> aggressively merged the dof data, I could have >>>> pruned a bunch of the work here and gotten much closer to the first >>>> time, at the expense of more complicated code. >>>> I guess this is the right pattern if you know that you will only have >>>> unknowns on cells and vertices. >>>> >>>> >>>>> 3) Also, about DMPlexDistribute(...). Right now it still seems >>>>> extremely slow, hence why I am distributing a coarse mesh and having each >>>>> processor refine its local portion of the mesh. In my current >>>>> implementation, I have rank 0 distribute the mesh via ParMETIS. Will there >>>>> eventually be a methodology to do parallel I/O then redistribution for load >>>>> balancing or does it already exist? >>>>> >>>> >>>> Redistribution for load balancing already exists. The stumbling block >>>> right now is parallel mesh import. However, >>>> the group at ICL (Michael Lange and Gerard Gorman) is working on this, >>>> and we expect to have it working by >>>> the end of summer. Until then, I recommend exactly what you are doing. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thanks, >>>>> >>>>> On Tue, Apr 7, 2015 at 8:37 PM, Matthew Knepley >>>>> wrote: >>>>> >>>>>> On Tue, Apr 7, 2015 at 7:18 PM, Justin Chang wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> So I am interpolating/uninterpolating my DMPlex mesh. This is the >>>>>>> mesh information for a cell-vertex mesh only: >>>>>>> >>>>>>> DM Object: 1 MPI processes >>>>>>> type: plex >>>>>>> DM_0x84000000_0 in 3 dimensions: >>>>>>> 0-cells: 159 >>>>>>> 3-cells: 592 >>>>>>> Labels: >>>>>>> marker: 1 strata of sizes (100) >>>>>>> depth: 2 strata of sizes (159, 592) >>>>>>> >>>>>>> I am interpolating the mesh with the following lines: >>>>>>> >>>>>>> ierr = DMPlexInterpolate(*dm, &idm);CHKERRQ(ierr); >>>>>>> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); >>>>>>> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); >>>>>>> ierr = DMPlexGetLabel(idm, "marker", &label);CHKERRQ(ierr); >>>>>>> ierr = DMPlexMarkBoundaryFaces(idm,label);CHKERRQ(ierr); >>>>>>> ierr = DMDestroy(dm);CHKERRQ(ierr); >>>>>>> *dm = idm; >>>>>>> >>>>>>> And the resulting mesh: >>>>>>> >>>>>>> DM Object: 1 MPI processes >>>>>>> type: plex >>>>>>> DM_0x84000000_1 in 3 dimensions: >>>>>>> 0-cells: 159 >>>>>>> 1-cells: 845 >>>>>>> 2-cells: 1280 >>>>>>> 3-cells: 592 >>>>>>> Labels: >>>>>>> marker: 1 strata of sizes (292) >>>>>>> depth: 4 strata of sizes (159, 845, 1280, 592) >>>>>>> >>>>>>> The reason I did this is so that incase I decide to refine the mesh >>>>>>> with "-dm_refine " the DM will be sure to include the "marker" >>>>>>> points in the refinement process. Now, if I decide to uninterpolate the >>>>>>> mesh with the following lines: >>>>>>> >>>>>>> ierr = DMPlexUninterpolate(*dm, &idm);CHKERRQ(ierr); >>>>>>> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); >>>>>>> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); >>>>>>> ierr = DMDestroy(dm);CHKERRQ(ierr); >>>>>>> *dm = idm; >>>>>>> >>>>>>> I end up with this mesh: >>>>>>> >>>>>>> DM Object: 1 MPI processes >>>>>>> type: plex >>>>>>> DM_0x84000000_2 in 3 dimensions: >>>>>>> 0-cells: 159 >>>>>>> 3-cells: 592 >>>>>>> Labels: >>>>>>> marker: 1 strata of sizes (292) >>>>>>> depth: 2 strata of sizes (159, 592) >>>>>>> >>>>>>> The problem is that although the cells and vertices have gone back >>>>>>> to the original number, the "marker" label still includes face/edge points. >>>>>>> This gives me an error once I invoke DMCreateGobalVector(...). >>>>>>> >>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>> -------------------------------------------------------------- >>>>>>> [0]PETSC ERROR: Argument out of range >>>>>>> [0]PETSC ERROR: Section point 755 should be in [0, 751) >>>>>>> [0]PETSC ERROR: See >>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>>>>> shooting. >>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>> v3.5.3-2528-gbee642f GIT Date: 2015-03-29 20:36:38 -0500 >>>>>>> [0]PETSC ERROR: ./cube_with_hole on a arch-linux2-c-debug named >>>>>>> pacotaco by justin Tue Apr 7 19:09:12 2015 >>>>>>> [0]PETSC ERROR: Configure options --download-chaco >>>>>>> --download-exodusii --download-fblaslapack --download-hdf5 --download-metis >>>>>>> --download-mpich --download-mumps --download-netcdf --download-parmetis >>>>>>> --download-scalapack --download-triangle --with-cc=gcc --with-cmake=cmake >>>>>>> --with-cxx=g++ --with-debugging=1 --with-fc=gfortran --with-valgrind=1 >>>>>>> PETSC_ARCH=arch-linux2-c-debug >>>>>>> [0]PETSC ERROR: #1 PetscSectionGetDof() line 499 in >>>>>>> /home/justin/petsc-dev/src/vec/is/utils/vsectionis.c >>>>>>> [0]PETSC ERROR: #2 DMPlexGetConeSize() line 841 in >>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>>>> [0]PETSC ERROR: #3 DMPlexGetTransitiveClosure() line 1342 in >>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>>>> [0]PETSC ERROR: #4 DMPlexLabelComplete() line 88 in >>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plexsubmesh.c >>>>>>> [0]PETSC ERROR: #5 DMCreateDefaultSection_Plex() line 5605 in >>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>>>> [0]PETSC ERROR: #6 DMGetDefaultSection() line 3035 in >>>>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>>>> [0]PETSC ERROR: #7 DMGetDefaultGlobalSection() line 3266 in >>>>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>>>> [0]PETSC ERROR: #8 DMCreateGlobalVector_Section_Private() line 13 in >>>>>>> /home/justin/petsc-dev/src/dm/interface/dmi.c >>>>>>> [0]PETSC ERROR: #9 DMCreateGlobalVector_Plex() line 1170 in >>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plexcreate.c >>>>>>> [0]PETSC ERROR: #10 DMCreateGlobalVector() line 698 in >>>>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>>>> [0]PETSC ERROR: #11 main() line 363 in >>>>>>> /home/justin/Dropbox/Research_Topics/Code_PETSc/Nonneg/cube_with_hole.c >>>>>>> >>>>>>> >>>>>>> From this error, it seems it's trying to access sieve points that >>>>>>> non longer exist. And I think it has to do with the label "marker" still >>>>>>> contains data from the interpolated mesh. Is there a way to "uninterpolate" >>>>>>> or remove such points? Or is there a better way of approaching this whole >>>>>>> thing? >>>>>>> >>>>>> >>>>>> 1) It would be easy to write a small function to throw out the points >>>>>> in the label that do not exist in the DM. You >>>>>> would just extract each stratumIS and then Clear each invalid >>>>>> point. >>>>>> >>>>>> 2) However, I do not understand the rationale for this above. You >>>>>> could just call MarkBoundaryFaces on the final mesh. >>>>>> If you are really trying to preserve a label across regular >>>>>> refinement AND you really do not want an interpolated mesh, >>>>>> then code up 1). I have yet to see a case where the extra memory >>>>>> for edges and faces makes a difference, but it may exist. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Justin Chang >>>>>>> PhD Candidate, Civil Engineering - Computational Sciences >>>>>>> University of Houston, Department of Civil and Environmental >>>>>>> Engineering >>>>>>> Houston, TX 77004 >>>>>>> (512) 963-3262 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Justin Chang >>>>> PhD Candidate, Civil Engineering - Computational Sciences >>>>> University of Houston, Department of Civil and Environmental >>>>> Engineering >>>>> Houston, TX 77004 >>>>> (512) 963-3262 >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >>> >>> -- >>> Justin Chang >>> PhD Candidate, Civil Engineering - Computational Sciences >>> University of Houston, Department of Civil and Environmental Engineering >>> Houston, TX 77004 >>> (512) 963-3262 >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 10 12:04:27 2015 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 10 Apr 2015 12:04:27 -0500 Subject: [petsc-users] Jacobian free solve and preconditioning In-Reply-To: References: Message-ID: On Fri, Apr 10, 2015 at 2:41 AM, Shiva Rudraraju wrote: > Hi, > > My problem (finite strain mechanics with additional higher order gradient > terms) involves a very complex jacobian. To avoid mistakes from hand > coding the jacobian I use AD (Sacado/Adept). However, both the hand coded > jacobian evaluation and AD computation of jacobian are computationally very > expensive and compete-with/exceed the solve time. So I am looking at > Jacobian free approchaes?.and would want to have some sort of matrix free > preconditioning. I am thinking of using: > True AD costs 3-5 residual evaluations to apply the Jacobian. Thus, if Jacobian construction costs more than the solve, its likely that either a) the AD is far from optimal or b) your system is so well so could use a much less accurate solver So 1) How much does your solve cost? What is the method, how many iterates? 2) You should start by replicating your current setup using the default FD-coloring Jacobian construction and see how it compares in time > -ksp_type fgmres -pc_type ksp -ksp_ksp_type gmres -ksp_pc_type jacobi > -snes_mf > > But -snes_mf will not use any preconditioner, so I want to try something > like -snes_mf_operator, but which uses GMRES/BiCG as a preconditioner > without the need for explicit jacobian computation. However, > snes_mf_operator requires the Jacobian function?.. Any way to obtain > jacobian free preconditioning > 3) I have no idea what 'Jacobian free preconditioning' even means. You could use PCCHEBYSHEV with only a Jacobian action, but you need good eigenvalue bounds. You can use an approximate Jacobian to calculate the preconditioner (this is the intent of snes_mf_operator). Thanks, Matt > Thanks, > Shiva > > > Shiva Rudraraju > Asst. Research Scientist, > University of Michigan Ann Arbor. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Apr 10 12:57:24 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 10 Apr 2015 12:57:24 -0500 Subject: [petsc-users] monitoring the convergence of fieldsplit 0 and 1 In-Reply-To: <1428671243078.94@marin.nl> References: <1428666513941.72745@marin.nl> <75D79823-7AE0-47A7-BE9E-15AB81C3581E@mcs.anl.gov> <1428671243078.94@marin.nl> Message-ID: <6C4D3117-82CB-4486-8A39-E84083FD5014@mcs.anl.gov> > On Apr 10, 2015, at 8:07 AM, Klaij, Christiaan wrote: > > Barry, > > Sure, I can call PCFieldSplitGetSubKSP() to get the fieldsplit_0 > ksp and then KSPGetIterationNumber, but what does this number > mean? > > It appears to be the number of iterations of the last time that > the subsystem was solved, right? If so, this corresponds to the > last iteration of the coupled system, how about all the previous > iterations? If I added a KSPGetTotalIterations() would that help? Barry > > Chris > ________________________________________ > From: Barry Smith > Sent: Friday, April 10, 2015 2:48 PM > To: Klaij, Christiaan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] monitoring the convergence of fieldsplit 0 and 1 > > Chris, > > It appears you should call PCFieldSplitGetSubKSP() and then get the information you want out of the individual KSPs. If this doesn't work please let us know. > > Barry > >> On Apr 10, 2015, at 6:48 AM, Klaij, Christiaan wrote: >> >> A question when using PCFieldSplit: for each linear iteration of >> the system, how many iterations for fielsplit 0 and 1? >> >> One way to find out is to run with -ksp_monitor, >> -fieldsplit_0_ksp_monitor and -fieldsplit_0_ksp_monitor. This >> gives the complete convergence history. >> >> Another way, suggested by Matt, is to use -ksp_monitor, >> -fieldsplit_0_ksp_converged_reason and >> -fieldsplit_1_ksp_converged_reason. This gives only the totals >> for fieldsplit 0 and 1 (but without saying for which one). >> >> Both ways require to somehow process the output, which is a bit >> inconvenient. Could KSPGetResidualHistory perhaps return (some) >> information on the subsystems' convergence for processing inside >> the code? >> >> Chris >> >> >> dr. ir. Christiaan Klaij >> CFD Researcher >> Research & Development >> E mailto:C.Klaij at marin.nl >> T +31 317 49 33 44 >> >> >> MARIN >> 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands >> T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl >> > From jchang27 at uh.edu Fri Apr 10 12:58:25 2015 From: jchang27 at uh.edu (Justin Chang) Date: Fri, 10 Apr 2015 12:58:25 -0500 Subject: [petsc-users] Uninterpolating DMLabels? In-Reply-To: References: Message-ID: Sorry I am a little confused here. I understand that each refinement of a 3D simplex element yields 8 subcells, but give the expression c_f \in [C c_c, C (c_c + 1) ) what exactly are these and/or what do they mean: c_f \in [... ) C c_c (c_c + 1) Originally I was thinking the functions DMPlexGetTransitiveClosure(...) and/or DMPlexGetCoarseDM(...) need to be invoked somehow. Do they? Again thanks for your help -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 On Fri, Apr 10, 2015 at 11:38 AM, Matthew Knepley wrote: > On Fri, Apr 10, 2015 at 1:01 AM, Justin Chang wrote: > >> Matt, >> >> I am not sure I follow what you're saying here. Currently I am working >> with 3D simplex elements so if the variable numSubcells is what you're >> referring to in CellRefinerGetAffineTransforms_Internal(), there's no >> example for REFINER_SIMPLEX_3D. What would it be in that case, 7? >> > > Its 8. There are 8 fine cells for every coarse cell. > > >> What I am looking for is a function and/or a set of algorithms where >> given a sieve point m from the original coarse mesh, I can obtain the >> corresponding set of (fine mesh) sieve points. I am assuming c_f \in [C >> c_c, C (c_c + 1) ) describes just that? >> > > Yes, C = 8. > > >> Is there an existing DMPlex related function that implements this >> directly? Because I don't see how CellRefinerGetAffineTransforms_Internal() >> will help. >> > > I would just give you C. Yes, I would just write it. It should be 1 line. > > Thanks, > > Matt > > >> >> Thanks, >> >> >> On Thu, Apr 9, 2015 at 8:20 PM, Matthew Knepley >> wrote: >> >>> On Thu, Apr 9, 2015 at 8:14 PM, Justin Chang wrote: >>> >>>> Matt, thank you for the clarification. One more question somewhat on a >>>> related note. >>>> >>>> I have auxiliary data corresponding to the permeability of a reservoir >>>> (for instance, the SPE10 benchmark problem). The data is cell-wise and read >>>> from a binary file. I want to refine both the DM and this data. Is there a >>>> way to obtain the mapping from the coarse mesh to the fine mesh? What I >>>> mean is, given a coarse cell, can I somehow get the transitive closure of >>>> the fine cells it created during the refinement process? This would allow >>>> me to copy the value of the coarse cell data onto it's children. >>>> >>> >>> If you are doing regular refinement, there is a simple formula: >>> >>> c_f \in [ C c_c, C (c_c +1) ) >>> >>> where C is the number of cells created from each cell. You can get C from >>> >>> CellRefinerGetAffineTransforms_Internal() >>> >>> which I could promote to a real interface function. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks, >>>> >>>> On Thu, Apr 9, 2015 at 7:52 PM, Matthew Knepley >>>> wrote: >>>> >>>>> On Thu, Apr 9, 2015 at 7:21 PM, Justin Chang wrote: >>>>> >>>>>> 1) I did what you had suggested and it works very well thanks. >>>>>> >>>>> >>>>> Cool. >>>>> >>>>> >>>>>> 2) In my case, calling uninterpolate does make a difference, just not >>>>>> necessarily in the solver step (I am using Tao's BLMVM). I attached two >>>>>> output files, one containing the timings and summaries with uninterpolating >>>>>> and one without uninterpolating. And the source code if you wanted to >>>>>> verify the "correctness" of my implementation. Namely, the biggest >>>>>> difference I see is in setting up the PetscSection. When uninterpolating >>>>>> the data this steps takes 8.64039 seconds but when I do not uninterpolate >>>>>> the data it takes 61.2977 seconds. Is this "significant" enough, or is it >>>>>> unimportant given the fact that this step occurs only once during any >>>>>> simulation? >>>>>> >>>>> >>>>> That is significant. Most of that time is in the operator >>>>> preallocation routine. It was cleaner to first gather the point >>>>> adjacency information, and then push the dof information on top. If I >>>>> aggressively merged the dof data, I could have >>>>> pruned a bunch of the work here and gotten much closer to the first >>>>> time, at the expense of more complicated code. >>>>> I guess this is the right pattern if you know that you will only have >>>>> unknowns on cells and vertices. >>>>> >>>>> >>>>>> 3) Also, about DMPlexDistribute(...). Right now it still seems >>>>>> extremely slow, hence why I am distributing a coarse mesh and having each >>>>>> processor refine its local portion of the mesh. In my current >>>>>> implementation, I have rank 0 distribute the mesh via ParMETIS. Will there >>>>>> eventually be a methodology to do parallel I/O then redistribution for load >>>>>> balancing or does it already exist? >>>>>> >>>>> >>>>> Redistribution for load balancing already exists. The stumbling block >>>>> right now is parallel mesh import. However, >>>>> the group at ICL (Michael Lange and Gerard Gorman) is working on this, >>>>> and we expect to have it working by >>>>> the end of summer. Until then, I recommend exactly what you are doing. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thanks, >>>>>> >>>>>> On Tue, Apr 7, 2015 at 8:37 PM, Matthew Knepley >>>>>> wrote: >>>>>> >>>>>>> On Tue, Apr 7, 2015 at 7:18 PM, Justin Chang >>>>>>> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> So I am interpolating/uninterpolating my DMPlex mesh. This is the >>>>>>>> mesh information for a cell-vertex mesh only: >>>>>>>> >>>>>>>> DM Object: 1 MPI processes >>>>>>>> type: plex >>>>>>>> DM_0x84000000_0 in 3 dimensions: >>>>>>>> 0-cells: 159 >>>>>>>> 3-cells: 592 >>>>>>>> Labels: >>>>>>>> marker: 1 strata of sizes (100) >>>>>>>> depth: 2 strata of sizes (159, 592) >>>>>>>> >>>>>>>> I am interpolating the mesh with the following lines: >>>>>>>> >>>>>>>> ierr = DMPlexInterpolate(*dm, &idm);CHKERRQ(ierr); >>>>>>>> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); >>>>>>>> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); >>>>>>>> ierr = DMPlexGetLabel(idm, "marker", &label);CHKERRQ(ierr); >>>>>>>> ierr = DMPlexMarkBoundaryFaces(idm,label);CHKERRQ(ierr); >>>>>>>> ierr = DMDestroy(dm);CHKERRQ(ierr); >>>>>>>> *dm = idm; >>>>>>>> >>>>>>>> And the resulting mesh: >>>>>>>> >>>>>>>> DM Object: 1 MPI processes >>>>>>>> type: plex >>>>>>>> DM_0x84000000_1 in 3 dimensions: >>>>>>>> 0-cells: 159 >>>>>>>> 1-cells: 845 >>>>>>>> 2-cells: 1280 >>>>>>>> 3-cells: 592 >>>>>>>> Labels: >>>>>>>> marker: 1 strata of sizes (292) >>>>>>>> depth: 4 strata of sizes (159, 845, 1280, 592) >>>>>>>> >>>>>>>> The reason I did this is so that incase I decide to refine the mesh >>>>>>>> with "-dm_refine " the DM will be sure to include the "marker" >>>>>>>> points in the refinement process. Now, if I decide to uninterpolate the >>>>>>>> mesh with the following lines: >>>>>>>> >>>>>>>> ierr = DMPlexUninterpolate(*dm, &idm);CHKERRQ(ierr); >>>>>>>> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); >>>>>>>> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); >>>>>>>> ierr = DMDestroy(dm);CHKERRQ(ierr); >>>>>>>> *dm = idm; >>>>>>>> >>>>>>>> I end up with this mesh: >>>>>>>> >>>>>>>> DM Object: 1 MPI processes >>>>>>>> type: plex >>>>>>>> DM_0x84000000_2 in 3 dimensions: >>>>>>>> 0-cells: 159 >>>>>>>> 3-cells: 592 >>>>>>>> Labels: >>>>>>>> marker: 1 strata of sizes (292) >>>>>>>> depth: 2 strata of sizes (159, 592) >>>>>>>> >>>>>>>> The problem is that although the cells and vertices have gone back >>>>>>>> to the original number, the "marker" label still includes face/edge points. >>>>>>>> This gives me an error once I invoke DMCreateGobalVector(...). >>>>>>>> >>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>> -------------------------------------------------------------- >>>>>>>> [0]PETSC ERROR: Argument out of range >>>>>>>> [0]PETSC ERROR: Section point 755 should be in [0, 751) >>>>>>>> [0]PETSC ERROR: See >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>>>>>> shooting. >>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>>> v3.5.3-2528-gbee642f GIT Date: 2015-03-29 20:36:38 -0500 >>>>>>>> [0]PETSC ERROR: ./cube_with_hole on a arch-linux2-c-debug named >>>>>>>> pacotaco by justin Tue Apr 7 19:09:12 2015 >>>>>>>> [0]PETSC ERROR: Configure options --download-chaco >>>>>>>> --download-exodusii --download-fblaslapack --download-hdf5 --download-metis >>>>>>>> --download-mpich --download-mumps --download-netcdf --download-parmetis >>>>>>>> --download-scalapack --download-triangle --with-cc=gcc --with-cmake=cmake >>>>>>>> --with-cxx=g++ --with-debugging=1 --with-fc=gfortran --with-valgrind=1 >>>>>>>> PETSC_ARCH=arch-linux2-c-debug >>>>>>>> [0]PETSC ERROR: #1 PetscSectionGetDof() line 499 in >>>>>>>> /home/justin/petsc-dev/src/vec/is/utils/vsectionis.c >>>>>>>> [0]PETSC ERROR: #2 DMPlexGetConeSize() line 841 in >>>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>>>>> [0]PETSC ERROR: #3 DMPlexGetTransitiveClosure() line 1342 in >>>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>>>>> [0]PETSC ERROR: #4 DMPlexLabelComplete() line 88 in >>>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plexsubmesh.c >>>>>>>> [0]PETSC ERROR: #5 DMCreateDefaultSection_Plex() line 5605 in >>>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>>>>> [0]PETSC ERROR: #6 DMGetDefaultSection() line 3035 in >>>>>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>>>>> [0]PETSC ERROR: #7 DMGetDefaultGlobalSection() line 3266 in >>>>>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>>>>> [0]PETSC ERROR: #8 DMCreateGlobalVector_Section_Private() line 13 >>>>>>>> in /home/justin/petsc-dev/src/dm/interface/dmi.c >>>>>>>> [0]PETSC ERROR: #9 DMCreateGlobalVector_Plex() line 1170 in >>>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plexcreate.c >>>>>>>> [0]PETSC ERROR: #10 DMCreateGlobalVector() line 698 in >>>>>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>>>>> [0]PETSC ERROR: #11 main() line 363 in >>>>>>>> /home/justin/Dropbox/Research_Topics/Code_PETSc/Nonneg/cube_with_hole.c >>>>>>>> >>>>>>>> >>>>>>>> From this error, it seems it's trying to access sieve points that >>>>>>>> non longer exist. And I think it has to do with the label "marker" still >>>>>>>> contains data from the interpolated mesh. Is there a way to "uninterpolate" >>>>>>>> or remove such points? Or is there a better way of approaching this whole >>>>>>>> thing? >>>>>>>> >>>>>>> >>>>>>> 1) It would be easy to write a small function to throw out the >>>>>>> points in the label that do not exist in the DM. You >>>>>>> would just extract each stratumIS and then Clear each invalid >>>>>>> point. >>>>>>> >>>>>>> 2) However, I do not understand the rationale for this above. You >>>>>>> could just call MarkBoundaryFaces on the final mesh. >>>>>>> If you are really trying to preserve a label across regular >>>>>>> refinement AND you really do not want an interpolated mesh, >>>>>>> then code up 1). I have yet to see a case where the extra memory >>>>>>> for edges and faces makes a difference, but it may exist. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Justin Chang >>>>>>>> PhD Candidate, Civil Engineering - Computational Sciences >>>>>>>> University of Houston, Department of Civil and Environmental >>>>>>>> Engineering >>>>>>>> Houston, TX 77004 >>>>>>>> (512) 963-3262 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Justin Chang >>>>>> PhD Candidate, Civil Engineering - Computational Sciences >>>>>> University of Houston, Department of Civil and Environmental >>>>>> Engineering >>>>>> Houston, TX 77004 >>>>>> (512) 963-3262 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>>> >>>> -- >>>> Justin Chang >>>> PhD Candidate, Civil Engineering - Computational Sciences >>>> University of Houston, Department of Civil and Environmental Engineering >>>> Houston, TX 77004 >>>> (512) 963-3262 >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> Justin Chang >> PhD Candidate, Civil Engineering - Computational Sciences >> University of Houston, Department of Civil and Environmental Engineering >> Houston, TX 77004 >> (512) 963-3262 >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 10 12:58:52 2015 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 10 Apr 2015 12:58:52 -0500 Subject: [petsc-users] monitoring the convergence of fieldsplit 0 and 1 In-Reply-To: <6C4D3117-82CB-4486-8A39-E84083FD5014@mcs.anl.gov> References: <1428666513941.72745@marin.nl> <75D79823-7AE0-47A7-BE9E-15AB81C3581E@mcs.anl.gov> <1428671243078.94@marin.nl> <6C4D3117-82CB-4486-8A39-E84083FD5014@mcs.anl.gov> Message-ID: On Fri, Apr 10, 2015 at 12:57 PM, Barry Smith wrote: > > > On Apr 10, 2015, at 8:07 AM, Klaij, Christiaan wrote: > > > > Barry, > > > > Sure, I can call PCFieldSplitGetSubKSP() to get the fieldsplit_0 > > ksp and then KSPGetIterationNumber, but what does this number > > mean? > > > > It appears to be the number of iterations of the last time that > > the subsystem was solved, right? If so, this corresponds to the > > last iteration of the coupled system, how about all the previous > > iterations? > > If I added a KSPGetTotalIterations() would that help? Its hard to see how to define this consistently. I think this is better handled by custom monitors. We could provide a default monitor for FieldSplit which does this. Matt > > Barry > > > > > Chris > > ________________________________________ > > From: Barry Smith > > Sent: Friday, April 10, 2015 2:48 PM > > To: Klaij, Christiaan > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] monitoring the convergence of fieldsplit 0 > and 1 > > > > Chris, > > > > It appears you should call PCFieldSplitGetSubKSP() and then get the > information you want out of the individual KSPs. If this doesn't work > please let us know. > > > > Barry > > > >> On Apr 10, 2015, at 6:48 AM, Klaij, Christiaan > wrote: > >> > >> A question when using PCFieldSplit: for each linear iteration of > >> the system, how many iterations for fielsplit 0 and 1? > >> > >> One way to find out is to run with -ksp_monitor, > >> -fieldsplit_0_ksp_monitor and -fieldsplit_0_ksp_monitor. This > >> gives the complete convergence history. > >> > >> Another way, suggested by Matt, is to use -ksp_monitor, > >> -fieldsplit_0_ksp_converged_reason and > >> -fieldsplit_1_ksp_converged_reason. This gives only the totals > >> for fieldsplit 0 and 1 (but without saying for which one). > >> > >> Both ways require to somehow process the output, which is a bit > >> inconvenient. Could KSPGetResidualHistory perhaps return (some) > >> information on the subsystems' convergence for processing inside > >> the code? > >> > >> Chris > >> > >> > >> dr. ir. Christiaan Klaij > >> CFD Researcher > >> Research & Development > >> E mailto:C.Klaij at marin.nl > >> T +31 317 49 33 44 > >> > >> > >> MARIN > >> 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > >> T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > >> > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Apr 10 13:05:06 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 10 Apr 2015 13:05:06 -0500 Subject: [petsc-users] monitoring the convergence of fieldsplit 0 and 1 In-Reply-To: References: <1428666513941.72745@marin.nl> <75D79823-7AE0-47A7-BE9E-15AB81C3581E@mcs.anl.gov> <1428671243078.94@marin.nl> <6C4D3117-82CB-4486-8A39-E84083FD5014@mcs.anl.gov> Message-ID: <1CBE0420-A868-4ADB-A5D8-FE51A6BB6281@mcs.anl.gov> > On Apr 10, 2015, at 12:58 PM, Matthew Knepley wrote: > > On Fri, Apr 10, 2015 at 12:57 PM, Barry Smith wrote: > > > On Apr 10, 2015, at 8:07 AM, Klaij, Christiaan wrote: > > > > Barry, > > > > Sure, I can call PCFieldSplitGetSubKSP() to get the fieldsplit_0 > > ksp and then KSPGetIterationNumber, but what does this number > > mean? > > > > It appears to be the number of iterations of the last time that > > the subsystem was solved, right? If so, this corresponds to the > > last iteration of the coupled system, how about all the previous > > iterations? > > If I added a KSPGetTotalIterations() would that help? > > Its hard to see how to define this consistently. Just count the number of iterations that KSP object has done since it was created. How could that be inconsistent? > I think this is better handled by custom monitors. > We could provide a default monitor for FieldSplit which does this. > > Matt > > > Barry > > > > > Chris > > ________________________________________ > > From: Barry Smith > > Sent: Friday, April 10, 2015 2:48 PM > > To: Klaij, Christiaan > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] monitoring the convergence of fieldsplit 0 and 1 > > > > Chris, > > > > It appears you should call PCFieldSplitGetSubKSP() and then get the information you want out of the individual KSPs. If this doesn't work please let us know. > > > > Barry > > > >> On Apr 10, 2015, at 6:48 AM, Klaij, Christiaan wrote: > >> > >> A question when using PCFieldSplit: for each linear iteration of > >> the system, how many iterations for fielsplit 0 and 1? > >> > >> One way to find out is to run with -ksp_monitor, > >> -fieldsplit_0_ksp_monitor and -fieldsplit_0_ksp_monitor. This > >> gives the complete convergence history. > >> > >> Another way, suggested by Matt, is to use -ksp_monitor, > >> -fieldsplit_0_ksp_converged_reason and > >> -fieldsplit_1_ksp_converged_reason. This gives only the totals > >> for fieldsplit 0 and 1 (but without saying for which one). > >> > >> Both ways require to somehow process the output, which is a bit > >> inconvenient. Could KSPGetResidualHistory perhaps return (some) > >> information on the subsystems' convergence for processing inside > >> the code? > >> > >> Chris > >> > >> > >> dr. ir. Christiaan Klaij > >> CFD Researcher > >> Research & Development > >> E mailto:C.Klaij at marin.nl > >> T +31 317 49 33 44 > >> > >> > >> MARIN > >> 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > >> T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > >> > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From knepley at gmail.com Fri Apr 10 13:21:23 2015 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 10 Apr 2015 13:21:23 -0500 Subject: [petsc-users] Uninterpolating DMLabels? In-Reply-To: References: Message-ID: On Fri, Apr 10, 2015 at 12:58 PM, Justin Chang wrote: > Sorry I am a little confused here. I understand that each refinement of a > 3D simplex element yields 8 subcells, but give the expression c_f \in [C > c_c, C (c_c + 1) ) what exactly are these and/or what do they mean: > There are 8 fine cells created in each coarse cell. If the coarse cell number is c_c, then the fine cells numbers are C c_c, C c_c + 1, ..., C (c_c + 1) -1 So, if the coarse cell number is 3, then the fine cells inside are 24, 25, 26, 27, 28, 29, 30, 31 Thanks, Matt > c_f \in [... ) > C > c_c > (c_c + 1) > > Originally I was thinking the functions DMPlexGetTransitiveClosure(...) > and/or DMPlexGetCoarseDM(...) need to be invoked somehow. Do they? > > Again thanks for your help > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > > On Fri, Apr 10, 2015 at 11:38 AM, Matthew Knepley > wrote: > >> On Fri, Apr 10, 2015 at 1:01 AM, Justin Chang wrote: >> >>> Matt, >>> >>> I am not sure I follow what you're saying here. Currently I am working >>> with 3D simplex elements so if the variable numSubcells is what you're >>> referring to in CellRefinerGetAffineTransforms_Internal(), there's no >>> example for REFINER_SIMPLEX_3D. What would it be in that case, 7? >>> >> >> Its 8. There are 8 fine cells for every coarse cell. >> >> >>> What I am looking for is a function and/or a set of algorithms where >>> given a sieve point m from the original coarse mesh, I can obtain the >>> corresponding set of (fine mesh) sieve points. I am assuming c_f \in [C >>> c_c, C (c_c + 1) ) describes just that? >>> >> >> Yes, C = 8. >> >> >>> Is there an existing DMPlex related function that implements this >>> directly? Because I don't see how CellRefinerGetAffineTransforms_Internal() >>> will help. >>> >> >> I would just give you C. Yes, I would just write it. It should be 1 line. >> >> Thanks, >> >> Matt >> >> >>> >>> Thanks, >>> >>> >>> On Thu, Apr 9, 2015 at 8:20 PM, Matthew Knepley >>> wrote: >>> >>>> On Thu, Apr 9, 2015 at 8:14 PM, Justin Chang wrote: >>>> >>>>> Matt, thank you for the clarification. One more question somewhat on a >>>>> related note. >>>>> >>>>> I have auxiliary data corresponding to the permeability of a reservoir >>>>> (for instance, the SPE10 benchmark problem). The data is cell-wise and read >>>>> from a binary file. I want to refine both the DM and this data. Is there a >>>>> way to obtain the mapping from the coarse mesh to the fine mesh? What I >>>>> mean is, given a coarse cell, can I somehow get the transitive closure of >>>>> the fine cells it created during the refinement process? This would allow >>>>> me to copy the value of the coarse cell data onto it's children. >>>>> >>>> >>>> If you are doing regular refinement, there is a simple formula: >>>> >>>> c_f \in [ C c_c, C (c_c +1) ) >>>> >>>> where C is the number of cells created from each cell. You can get C >>>> from >>>> >>>> CellRefinerGetAffineTransforms_Internal() >>>> >>>> which I could promote to a real interface function. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thanks, >>>>> >>>>> On Thu, Apr 9, 2015 at 7:52 PM, Matthew Knepley >>>>> wrote: >>>>> >>>>>> On Thu, Apr 9, 2015 at 7:21 PM, Justin Chang wrote: >>>>>> >>>>>>> 1) I did what you had suggested and it works very well thanks. >>>>>>> >>>>>> >>>>>> Cool. >>>>>> >>>>>> >>>>>>> 2) In my case, calling uninterpolate does make a difference, just >>>>>>> not necessarily in the solver step (I am using Tao's BLMVM). I attached two >>>>>>> output files, one containing the timings and summaries with uninterpolating >>>>>>> and one without uninterpolating. And the source code if you wanted to >>>>>>> verify the "correctness" of my implementation. Namely, the biggest >>>>>>> difference I see is in setting up the PetscSection. When uninterpolating >>>>>>> the data this steps takes 8.64039 seconds but when I do not uninterpolate >>>>>>> the data it takes 61.2977 seconds. Is this "significant" enough, or is it >>>>>>> unimportant given the fact that this step occurs only once during any >>>>>>> simulation? >>>>>>> >>>>>> >>>>>> That is significant. Most of that time is in the operator >>>>>> preallocation routine. It was cleaner to first gather the point >>>>>> adjacency information, and then push the dof information on top. If I >>>>>> aggressively merged the dof data, I could have >>>>>> pruned a bunch of the work here and gotten much closer to the first >>>>>> time, at the expense of more complicated code. >>>>>> I guess this is the right pattern if you know that you will only have >>>>>> unknowns on cells and vertices. >>>>>> >>>>>> >>>>>>> 3) Also, about DMPlexDistribute(...). Right now it still seems >>>>>>> extremely slow, hence why I am distributing a coarse mesh and having each >>>>>>> processor refine its local portion of the mesh. In my current >>>>>>> implementation, I have rank 0 distribute the mesh via ParMETIS. Will there >>>>>>> eventually be a methodology to do parallel I/O then redistribution for load >>>>>>> balancing or does it already exist? >>>>>>> >>>>>> >>>>>> Redistribution for load balancing already exists. The stumbling block >>>>>> right now is parallel mesh import. However, >>>>>> the group at ICL (Michael Lange and Gerard Gorman) is working on >>>>>> this, and we expect to have it working by >>>>>> the end of summer. Until then, I recommend exactly what you are doing. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> On Tue, Apr 7, 2015 at 8:37 PM, Matthew Knepley >>>>>>> wrote: >>>>>>> >>>>>>>> On Tue, Apr 7, 2015 at 7:18 PM, Justin Chang >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> So I am interpolating/uninterpolating my DMPlex mesh. This is the >>>>>>>>> mesh information for a cell-vertex mesh only: >>>>>>>>> >>>>>>>>> DM Object: 1 MPI processes >>>>>>>>> type: plex >>>>>>>>> DM_0x84000000_0 in 3 dimensions: >>>>>>>>> 0-cells: 159 >>>>>>>>> 3-cells: 592 >>>>>>>>> Labels: >>>>>>>>> marker: 1 strata of sizes (100) >>>>>>>>> depth: 2 strata of sizes (159, 592) >>>>>>>>> >>>>>>>>> I am interpolating the mesh with the following lines: >>>>>>>>> >>>>>>>>> ierr = DMPlexInterpolate(*dm, &idm);CHKERRQ(ierr); >>>>>>>>> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); >>>>>>>>> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); >>>>>>>>> ierr = DMPlexGetLabel(idm, "marker", &label);CHKERRQ(ierr); >>>>>>>>> ierr = DMPlexMarkBoundaryFaces(idm,label);CHKERRQ(ierr); >>>>>>>>> ierr = DMDestroy(dm);CHKERRQ(ierr); >>>>>>>>> *dm = idm; >>>>>>>>> >>>>>>>>> And the resulting mesh: >>>>>>>>> >>>>>>>>> DM Object: 1 MPI processes >>>>>>>>> type: plex >>>>>>>>> DM_0x84000000_1 in 3 dimensions: >>>>>>>>> 0-cells: 159 >>>>>>>>> 1-cells: 845 >>>>>>>>> 2-cells: 1280 >>>>>>>>> 3-cells: 592 >>>>>>>>> Labels: >>>>>>>>> marker: 1 strata of sizes (292) >>>>>>>>> depth: 4 strata of sizes (159, 845, 1280, 592) >>>>>>>>> >>>>>>>>> The reason I did this is so that incase I decide to refine the >>>>>>>>> mesh with "-dm_refine " the DM will be sure to include the "marker" >>>>>>>>> points in the refinement process. Now, if I decide to uninterpolate the >>>>>>>>> mesh with the following lines: >>>>>>>>> >>>>>>>>> ierr = DMPlexUninterpolate(*dm, &idm);CHKERRQ(ierr); >>>>>>>>> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); >>>>>>>>> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); >>>>>>>>> ierr = DMDestroy(dm);CHKERRQ(ierr); >>>>>>>>> *dm = idm; >>>>>>>>> >>>>>>>>> I end up with this mesh: >>>>>>>>> >>>>>>>>> DM Object: 1 MPI processes >>>>>>>>> type: plex >>>>>>>>> DM_0x84000000_2 in 3 dimensions: >>>>>>>>> 0-cells: 159 >>>>>>>>> 3-cells: 592 >>>>>>>>> Labels: >>>>>>>>> marker: 1 strata of sizes (292) >>>>>>>>> depth: 2 strata of sizes (159, 592) >>>>>>>>> >>>>>>>>> The problem is that although the cells and vertices have gone back >>>>>>>>> to the original number, the "marker" label still includes face/edge points. >>>>>>>>> This gives me an error once I invoke DMCreateGobalVector(...). >>>>>>>>> >>>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>>> -------------------------------------------------------------- >>>>>>>>> [0]PETSC ERROR: Argument out of range >>>>>>>>> [0]PETSC ERROR: Section point 755 should be in [0, 751) >>>>>>>>> [0]PETSC ERROR: See >>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>>>>>>> shooting. >>>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>>>> v3.5.3-2528-gbee642f GIT Date: 2015-03-29 20:36:38 -0500 >>>>>>>>> [0]PETSC ERROR: ./cube_with_hole on a arch-linux2-c-debug named >>>>>>>>> pacotaco by justin Tue Apr 7 19:09:12 2015 >>>>>>>>> [0]PETSC ERROR: Configure options --download-chaco >>>>>>>>> --download-exodusii --download-fblaslapack --download-hdf5 --download-metis >>>>>>>>> --download-mpich --download-mumps --download-netcdf --download-parmetis >>>>>>>>> --download-scalapack --download-triangle --with-cc=gcc --with-cmake=cmake >>>>>>>>> --with-cxx=g++ --with-debugging=1 --with-fc=gfortran --with-valgrind=1 >>>>>>>>> PETSC_ARCH=arch-linux2-c-debug >>>>>>>>> [0]PETSC ERROR: #1 PetscSectionGetDof() line 499 in >>>>>>>>> /home/justin/petsc-dev/src/vec/is/utils/vsectionis.c >>>>>>>>> [0]PETSC ERROR: #2 DMPlexGetConeSize() line 841 in >>>>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>>>>>> [0]PETSC ERROR: #3 DMPlexGetTransitiveClosure() line 1342 in >>>>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>>>>>> [0]PETSC ERROR: #4 DMPlexLabelComplete() line 88 in >>>>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plexsubmesh.c >>>>>>>>> [0]PETSC ERROR: #5 DMCreateDefaultSection_Plex() line 5605 in >>>>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>>>>>> [0]PETSC ERROR: #6 DMGetDefaultSection() line 3035 in >>>>>>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>>>>>> [0]PETSC ERROR: #7 DMGetDefaultGlobalSection() line 3266 in >>>>>>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>>>>>> [0]PETSC ERROR: #8 DMCreateGlobalVector_Section_Private() line 13 >>>>>>>>> in /home/justin/petsc-dev/src/dm/interface/dmi.c >>>>>>>>> [0]PETSC ERROR: #9 DMCreateGlobalVector_Plex() line 1170 in >>>>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plexcreate.c >>>>>>>>> [0]PETSC ERROR: #10 DMCreateGlobalVector() line 698 in >>>>>>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>>>>>> [0]PETSC ERROR: #11 main() line 363 in >>>>>>>>> /home/justin/Dropbox/Research_Topics/Code_PETSc/Nonneg/cube_with_hole.c >>>>>>>>> >>>>>>>>> >>>>>>>>> From this error, it seems it's trying to access sieve points that >>>>>>>>> non longer exist. And I think it has to do with the label "marker" still >>>>>>>>> contains data from the interpolated mesh. Is there a way to "uninterpolate" >>>>>>>>> or remove such points? Or is there a better way of approaching this whole >>>>>>>>> thing? >>>>>>>>> >>>>>>>> >>>>>>>> 1) It would be easy to write a small function to throw out the >>>>>>>> points in the label that do not exist in the DM. You >>>>>>>> would just extract each stratumIS and then Clear each invalid >>>>>>>> point. >>>>>>>> >>>>>>>> 2) However, I do not understand the rationale for this above. You >>>>>>>> could just call MarkBoundaryFaces on the final mesh. >>>>>>>> If you are really trying to preserve a label across regular >>>>>>>> refinement AND you really do not want an interpolated mesh, >>>>>>>> then code up 1). I have yet to see a case where the extra >>>>>>>> memory for edges and faces makes a difference, but it may exist. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Justin Chang >>>>>>>>> PhD Candidate, Civil Engineering - Computational Sciences >>>>>>>>> University of Houston, Department of Civil and Environmental >>>>>>>>> Engineering >>>>>>>>> Houston, TX 77004 >>>>>>>>> (512) 963-3262 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Justin Chang >>>>>>> PhD Candidate, Civil Engineering - Computational Sciences >>>>>>> University of Houston, Department of Civil and Environmental >>>>>>> Engineering >>>>>>> Houston, TX 77004 >>>>>>> (512) 963-3262 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Justin Chang >>>>> PhD Candidate, Civil Engineering - Computational Sciences >>>>> University of Houston, Department of Civil and Environmental >>>>> Engineering >>>>> Houston, TX 77004 >>>>> (512) 963-3262 >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >>> >>> -- >>> Justin Chang >>> PhD Candidate, Civil Engineering - Computational Sciences >>> University of Houston, Department of Civil and Environmental Engineering >>> Houston, TX 77004 >>> (512) 963-3262 >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 10 13:26:12 2015 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 10 Apr 2015 13:26:12 -0500 Subject: [petsc-users] monitoring the convergence of fieldsplit 0 and 1 In-Reply-To: <1CBE0420-A868-4ADB-A5D8-FE51A6BB6281@mcs.anl.gov> References: <1428666513941.72745@marin.nl> <75D79823-7AE0-47A7-BE9E-15AB81C3581E@mcs.anl.gov> <1428671243078.94@marin.nl> <6C4D3117-82CB-4486-8A39-E84083FD5014@mcs.anl.gov> <1CBE0420-A868-4ADB-A5D8-FE51A6BB6281@mcs.anl.gov> Message-ID: On Fri, Apr 10, 2015 at 1:05 PM, Barry Smith wrote: > > > On Apr 10, 2015, at 12:58 PM, Matthew Knepley wrote: > > > > On Fri, Apr 10, 2015 at 12:57 PM, Barry Smith > wrote: > > > > > On Apr 10, 2015, at 8:07 AM, Klaij, Christiaan > wrote: > > > > > > Barry, > > > > > > Sure, I can call PCFieldSplitGetSubKSP() to get the fieldsplit_0 > > > ksp and then KSPGetIterationNumber, but what does this number > > > mean? > > > > > > It appears to be the number of iterations of the last time that > > > the subsystem was solved, right? If so, this corresponds to the > > > last iteration of the coupled system, how about all the previous > > > iterations? > > > > If I added a KSPGetTotalIterations() would that help? > > > > Its hard to see how to define this consistently. > > Just count the number of iterations that KSP object has done since it > was created. How could that be inconsistent? Oh, you want the counts to be localized in the subKSP. This not nearly as useful. For example, we could have left this count in the KSP, but we hoist it to the SNES. I am fine totaling the counts in the KSP, but I would propose custom monitors as a way of hoisting the info to the next level. Matt > > I think this is better handled by custom monitors. > > We could provide a default monitor for FieldSplit which does this. > > > > Matt > > > > > > Barry > > > > > > > > Chris > > > ________________________________________ > > > From: Barry Smith > > > Sent: Friday, April 10, 2015 2:48 PM > > > To: Klaij, Christiaan > > > Cc: petsc-users at mcs.anl.gov > > > Subject: Re: [petsc-users] monitoring the convergence of fieldsplit 0 > and 1 > > > > > > Chris, > > > > > > It appears you should call PCFieldSplitGetSubKSP() and then get > the information you want out of the individual KSPs. If this doesn't work > please let us know. > > > > > > Barry > > > > > >> On Apr 10, 2015, at 6:48 AM, Klaij, Christiaan > wrote: > > >> > > >> A question when using PCFieldSplit: for each linear iteration of > > >> the system, how many iterations for fielsplit 0 and 1? > > >> > > >> One way to find out is to run with -ksp_monitor, > > >> -fieldsplit_0_ksp_monitor and -fieldsplit_0_ksp_monitor. This > > >> gives the complete convergence history. > > >> > > >> Another way, suggested by Matt, is to use -ksp_monitor, > > >> -fieldsplit_0_ksp_converged_reason and > > >> -fieldsplit_1_ksp_converged_reason. This gives only the totals > > >> for fieldsplit 0 and 1 (but without saying for which one). > > >> > > >> Both ways require to somehow process the output, which is a bit > > >> inconvenient. Could KSPGetResidualHistory perhaps return (some) > > >> information on the subsystems' convergence for processing inside > > >> the code? > > >> > > >> Chris > > >> > > >> > > >> dr. ir. Christiaan Klaij > > >> CFD Researcher > > >> Research & Development > > >> E mailto:C.Klaij at marin.nl > > >> T +31 317 49 33 44 > > >> > > >> > > >> MARIN > > >> 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > > >> T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > > >> > > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Apr 10 13:30:54 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 10 Apr 2015 13:30:54 -0500 Subject: [petsc-users] monitoring the convergence of fieldsplit 0 and 1 In-Reply-To: References: <1428666513941.72745@marin.nl> <75D79823-7AE0-47A7-BE9E-15AB81C3581E@mcs.anl.gov> <1428671243078.94@marin.nl> <6C4D3117-82CB-4486-8A39-E84083FD5014@mcs.anl.gov> <1CBE0420-A868-4ADB-A5D8-FE51A6BB6281@mcs.anl.gov> Message-ID: <72FC4732-01D4-4B7D-8972-7BF0758AA5AB@mcs.anl.gov> > On Apr 10, 2015, at 1:26 PM, Matthew Knepley wrote: > > On Fri, Apr 10, 2015 at 1:05 PM, Barry Smith wrote: > > > On Apr 10, 2015, at 12:58 PM, Matthew Knepley wrote: > > > > On Fri, Apr 10, 2015 at 12:57 PM, Barry Smith wrote: > > > > > On Apr 10, 2015, at 8:07 AM, Klaij, Christiaan wrote: > > > > > > Barry, > > > > > > Sure, I can call PCFieldSplitGetSubKSP() to get the fieldsplit_0 > > > ksp and then KSPGetIterationNumber, but what does this number > > > mean? > > > > > > It appears to be the number of iterations of the last time that > > > the subsystem was solved, right? If so, this corresponds to the > > > last iteration of the coupled system, how about all the previous > > > iterations? > > > > If I added a KSPGetTotalIterations() would that help? > > > > Its hard to see how to define this consistently. > > Just count the number of iterations that KSP object has done since it was created. How could that be inconsistent? > > Oh, you want the counts to be localized in the subKSP. This not nearly as useful. Yes but it is the starting point for all the hoisting that may be needed. I'll add it. > For example, we could have left this > count in the KSP, but we hoist it to the SNES. I am fine totaling the counts in the KSP, but I would propose custom > monitors as a way of hoisting the info to the next level. > > Matt > > > I think this is better handled by custom monitors. > > We could provide a default monitor for FieldSplit which does this. > > > > Matt > > > > > > Barry > > > > > > > > Chris > > > ________________________________________ > > > From: Barry Smith > > > Sent: Friday, April 10, 2015 2:48 PM > > > To: Klaij, Christiaan > > > Cc: petsc-users at mcs.anl.gov > > > Subject: Re: [petsc-users] monitoring the convergence of fieldsplit 0 and 1 > > > > > > Chris, > > > > > > It appears you should call PCFieldSplitGetSubKSP() and then get the information you want out of the individual KSPs. If this doesn't work please let us know. > > > > > > Barry > > > > > >> On Apr 10, 2015, at 6:48 AM, Klaij, Christiaan wrote: > > >> > > >> A question when using PCFieldSplit: for each linear iteration of > > >> the system, how many iterations for fielsplit 0 and 1? > > >> > > >> One way to find out is to run with -ksp_monitor, > > >> -fieldsplit_0_ksp_monitor and -fieldsplit_0_ksp_monitor. This > > >> gives the complete convergence history. > > >> > > >> Another way, suggested by Matt, is to use -ksp_monitor, > > >> -fieldsplit_0_ksp_converged_reason and > > >> -fieldsplit_1_ksp_converged_reason. This gives only the totals > > >> for fieldsplit 0 and 1 (but without saying for which one). > > >> > > >> Both ways require to somehow process the output, which is a bit > > >> inconvenient. Could KSPGetResidualHistory perhaps return (some) > > >> information on the subsystems' convergence for processing inside > > >> the code? > > >> > > >> Chris > > >> > > >> > > >> dr. ir. Christiaan Klaij > > >> CFD Researcher > > >> Research & Development > > >> E mailto:C.Klaij at marin.nl > > >> T +31 317 49 33 44 > > >> > > >> > > >> MARIN > > >> 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > > >> T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > > >> > > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From fande.kong at colorado.edu Fri Apr 10 15:22:12 2015 From: fande.kong at colorado.edu (Fande Kong) Date: Fri, 10 Apr 2015 14:22:12 -0600 Subject: [petsc-users] timer in PETSc Message-ID: Hi all, I am running an application with more than 4,000 cores. I get a warning in the -log_summary: Warning -- total time of even greater than time of entire stage -- something is wrong with the timer. I have no any clues for this warning. Any suggestions to remove it? Thanks, Fande, -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Apr 10 15:27:27 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 10 Apr 2015 15:27:27 -0500 Subject: [petsc-users] Problem when using MatLoad In-Reply-To: <55265C51.3050802@inria.fr> References: <55265C51.3050802@inria.fr> Message-ID: <7DB87E4D-FECF-4205-82E8-4C527364037A@mcs.anl.gov> Fixed in the branch barry/fix-matload-mpidense-denseinfile and next will put into master after testing. I couldn't put my fix into maint (the release version), I recommend you upgrade to the development version for this feature or you can manually fix by editing MatLoad_MPIDense_DenseInFile() and using MPI_Reduce() to communicate the largest size needed to be passed to the malloc() instead of using the size from the first process. Thanks for letting us know about the bug. Barry > On Apr 9, 2015, at 6:02 AM, Matteo Aletti wrote: > > Hello, > > I was trying to use the function MatLoad to read an mpi dense (square) matrix from a binary file. > I got an error related to a memory problem (in parallel). > I tried to locate the problem using gdb and I think it is in the function MatLoad_MPIDense_DenseInFile. > The master cannot execute the line > ierr = PetscFree(vals);CHKERRQ(ierr); > without an error. The other processors can. > > For me, the error is in the allocation of the vals array: each processor allocates it with m*N number of elements, where N is the same for all of the procs and m is the local number of rows, which in my case is already given in the matrix. > > The master uses the array vals to read its own data, but also the other processors' data. The problem is that, in my case, one processor has a higher number of rows than the master and therefore vals is too short to store those values. For me the master should allocate it with size m_max*N , where m_max is the maximum number of local rows between all procs. > > Thanks, > Best, > Matteo From bsmith at mcs.anl.gov Fri Apr 10 15:30:52 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 10 Apr 2015 15:30:52 -0500 Subject: [petsc-users] timer in PETSc In-Reply-To: References: Message-ID: Are you adding your own events or stages or just running with -log_summary? If you added your own events or stages this likely happened because you changed between stages within an event so its start time is logged in one stage but the end time correct gets put into a different stage. Barry > On Apr 10, 2015, at 3:22 PM, Fande Kong wrote: > > Hi all, > > I am running an application with more than 4,000 cores. I get a warning in the -log_summary: Warning -- total time of even greater than time of entire stage -- something is wrong with the timer. > > I have no any clues for this warning. Any suggestions to remove it? > > Thanks, > > Fande, From jchang27 at uh.edu Fri Apr 10 15:39:32 2015 From: jchang27 at uh.edu (Justin Chang) Date: Fri, 10 Apr 2015 15:39:32 -0500 Subject: [petsc-users] Uninterpolating DMLabels? In-Reply-To: References: Message-ID: That makes a lot of sense now. So if I understand this correctly, with -dm_refine 2, all of the fine cells for coarse cell number 3 would be: 192 193 ... 255 -dm_refine 3 for the same cell would be: 1536 1537 .... 2047 and so on. And this would hold true even for distributed meshes? Thanks, On Fri, Apr 10, 2015 at 1:21 PM, Matthew Knepley wrote: > On Fri, Apr 10, 2015 at 12:58 PM, Justin Chang wrote: > >> Sorry I am a little confused here. I understand that each refinement of a >> 3D simplex element yields 8 subcells, but give the expression c_f \in [C >> c_c, C (c_c + 1) ) what exactly are these and/or what do they mean: >> > > There are 8 fine cells created in each coarse cell. If the coarse cell > number is c_c, then the fine cells numbers are > > C c_c, C c_c + 1, ..., C (c_c + 1) -1 > > So, if the coarse cell number is 3, then the fine cells inside are > > 24, 25, 26, 27, 28, 29, 30, 31 > > Thanks, > > Matt > > >> c_f \in [... ) >> C >> c_c >> (c_c + 1) >> >> Originally I was thinking the functions DMPlexGetTransitiveClosure(...) >> and/or DMPlexGetCoarseDM(...) need to be invoked somehow. Do they? >> >> Again thanks for your help >> >> -- >> Justin Chang >> PhD Candidate, Civil Engineering - Computational Sciences >> University of Houston, Department of Civil and Environmental Engineering >> Houston, TX 77004 >> (512) 963-3262 >> >> On Fri, Apr 10, 2015 at 11:38 AM, Matthew Knepley >> wrote: >> >>> On Fri, Apr 10, 2015 at 1:01 AM, Justin Chang wrote: >>> >>>> Matt, >>>> >>>> I am not sure I follow what you're saying here. Currently I am working >>>> with 3D simplex elements so if the variable numSubcells is what you're >>>> referring to in CellRefinerGetAffineTransforms_Internal(), there's no >>>> example for REFINER_SIMPLEX_3D. What would it be in that case, 7? >>>> >>> >>> Its 8. There are 8 fine cells for every coarse cell. >>> >>> >>>> What I am looking for is a function and/or a set of algorithms where >>>> given a sieve point m from the original coarse mesh, I can obtain the >>>> corresponding set of (fine mesh) sieve points. I am assuming c_f \in [C >>>> c_c, C (c_c + 1) ) describes just that? >>>> >>> >>> Yes, C = 8. >>> >>> >>>> Is there an existing DMPlex related function that implements this >>>> directly? Because I don't see how CellRefinerGetAffineTransforms_Internal() >>>> will help. >>>> >>> >>> I would just give you C. Yes, I would just write it. It should be 1 line. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> >>>> Thanks, >>>> >>>> >>>> On Thu, Apr 9, 2015 at 8:20 PM, Matthew Knepley >>>> wrote: >>>> >>>>> On Thu, Apr 9, 2015 at 8:14 PM, Justin Chang wrote: >>>>> >>>>>> Matt, thank you for the clarification. One more question somewhat on >>>>>> a related note. >>>>>> >>>>>> I have auxiliary data corresponding to the permeability of a >>>>>> reservoir (for instance, the SPE10 benchmark problem). The data is >>>>>> cell-wise and read from a binary file. I want to refine both the DM and >>>>>> this data. Is there a way to obtain the mapping from the coarse mesh to the >>>>>> fine mesh? What I mean is, given a coarse cell, can I somehow get the >>>>>> transitive closure of the fine cells it created during the refinement >>>>>> process? This would allow me to copy the value of the coarse cell data onto >>>>>> it's children. >>>>>> >>>>> >>>>> If you are doing regular refinement, there is a simple formula: >>>>> >>>>> c_f \in [ C c_c, C (c_c +1) ) >>>>> >>>>> where C is the number of cells created from each cell. You can get C >>>>> from >>>>> >>>>> CellRefinerGetAffineTransforms_Internal() >>>>> >>>>> which I could promote to a real interface function. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thanks, >>>>>> >>>>>> On Thu, Apr 9, 2015 at 7:52 PM, Matthew Knepley >>>>>> wrote: >>>>>> >>>>>>> On Thu, Apr 9, 2015 at 7:21 PM, Justin Chang >>>>>>> wrote: >>>>>>> >>>>>>>> 1) I did what you had suggested and it works very well thanks. >>>>>>>> >>>>>>> >>>>>>> Cool. >>>>>>> >>>>>>> >>>>>>>> 2) In my case, calling uninterpolate does make a difference, just >>>>>>>> not necessarily in the solver step (I am using Tao's BLMVM). I attached two >>>>>>>> output files, one containing the timings and summaries with uninterpolating >>>>>>>> and one without uninterpolating. And the source code if you wanted to >>>>>>>> verify the "correctness" of my implementation. Namely, the biggest >>>>>>>> difference I see is in setting up the PetscSection. When uninterpolating >>>>>>>> the data this steps takes 8.64039 seconds but when I do not uninterpolate >>>>>>>> the data it takes 61.2977 seconds. Is this "significant" enough, or is it >>>>>>>> unimportant given the fact that this step occurs only once during any >>>>>>>> simulation? >>>>>>>> >>>>>>> >>>>>>> That is significant. Most of that time is in the operator >>>>>>> preallocation routine. It was cleaner to first gather the point >>>>>>> adjacency information, and then push the dof information on top. If >>>>>>> I aggressively merged the dof data, I could have >>>>>>> pruned a bunch of the work here and gotten much closer to the first >>>>>>> time, at the expense of more complicated code. >>>>>>> I guess this is the right pattern if you know that you will only >>>>>>> have unknowns on cells and vertices. >>>>>>> >>>>>>> >>>>>>>> 3) Also, about DMPlexDistribute(...). Right now it still seems >>>>>>>> extremely slow, hence why I am distributing a coarse mesh and having each >>>>>>>> processor refine its local portion of the mesh. In my current >>>>>>>> implementation, I have rank 0 distribute the mesh via ParMETIS. Will there >>>>>>>> eventually be a methodology to do parallel I/O then redistribution for load >>>>>>>> balancing or does it already exist? >>>>>>>> >>>>>>> >>>>>>> Redistribution for load balancing already exists. The stumbling >>>>>>> block right now is parallel mesh import. However, >>>>>>> the group at ICL (Michael Lange and Gerard Gorman) is working on >>>>>>> this, and we expect to have it working by >>>>>>> the end of summer. Until then, I recommend exactly what you are >>>>>>> doing. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> On Tue, Apr 7, 2015 at 8:37 PM, Matthew Knepley >>>>>>>> wrote: >>>>>>>> >>>>>>>>> On Tue, Apr 7, 2015 at 7:18 PM, Justin Chang >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> So I am interpolating/uninterpolating my DMPlex mesh. This is the >>>>>>>>>> mesh information for a cell-vertex mesh only: >>>>>>>>>> >>>>>>>>>> DM Object: 1 MPI processes >>>>>>>>>> type: plex >>>>>>>>>> DM_0x84000000_0 in 3 dimensions: >>>>>>>>>> 0-cells: 159 >>>>>>>>>> 3-cells: 592 >>>>>>>>>> Labels: >>>>>>>>>> marker: 1 strata of sizes (100) >>>>>>>>>> depth: 2 strata of sizes (159, 592) >>>>>>>>>> >>>>>>>>>> I am interpolating the mesh with the following lines: >>>>>>>>>> >>>>>>>>>> ierr = DMPlexInterpolate(*dm, &idm);CHKERRQ(ierr); >>>>>>>>>> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); >>>>>>>>>> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); >>>>>>>>>> ierr = DMPlexGetLabel(idm, "marker", &label);CHKERRQ(ierr); >>>>>>>>>> ierr = DMPlexMarkBoundaryFaces(idm,label);CHKERRQ(ierr); >>>>>>>>>> ierr = DMDestroy(dm);CHKERRQ(ierr); >>>>>>>>>> *dm = idm; >>>>>>>>>> >>>>>>>>>> And the resulting mesh: >>>>>>>>>> >>>>>>>>>> DM Object: 1 MPI processes >>>>>>>>>> type: plex >>>>>>>>>> DM_0x84000000_1 in 3 dimensions: >>>>>>>>>> 0-cells: 159 >>>>>>>>>> 1-cells: 845 >>>>>>>>>> 2-cells: 1280 >>>>>>>>>> 3-cells: 592 >>>>>>>>>> Labels: >>>>>>>>>> marker: 1 strata of sizes (292) >>>>>>>>>> depth: 4 strata of sizes (159, 845, 1280, 592) >>>>>>>>>> >>>>>>>>>> The reason I did this is so that incase I decide to refine the >>>>>>>>>> mesh with "-dm_refine " the DM will be sure to include the "marker" >>>>>>>>>> points in the refinement process. Now, if I decide to uninterpolate the >>>>>>>>>> mesh with the following lines: >>>>>>>>>> >>>>>>>>>> ierr = DMPlexUninterpolate(*dm, &idm);CHKERRQ(ierr); >>>>>>>>>> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); >>>>>>>>>> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); >>>>>>>>>> ierr = DMDestroy(dm);CHKERRQ(ierr); >>>>>>>>>> *dm = idm; >>>>>>>>>> >>>>>>>>>> I end up with this mesh: >>>>>>>>>> >>>>>>>>>> DM Object: 1 MPI processes >>>>>>>>>> type: plex >>>>>>>>>> DM_0x84000000_2 in 3 dimensions: >>>>>>>>>> 0-cells: 159 >>>>>>>>>> 3-cells: 592 >>>>>>>>>> Labels: >>>>>>>>>> marker: 1 strata of sizes (292) >>>>>>>>>> depth: 2 strata of sizes (159, 592) >>>>>>>>>> >>>>>>>>>> The problem is that although the cells and vertices have gone >>>>>>>>>> back to the original number, the "marker" label still includes face/edge >>>>>>>>>> points. This gives me an error once I invoke DMCreateGobalVector(...). >>>>>>>>>> >>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>> [0]PETSC ERROR: Argument out of range >>>>>>>>>> [0]PETSC ERROR: Section point 755 should be in [0, 751) >>>>>>>>>> [0]PETSC ERROR: See >>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>>>>>>>> shooting. >>>>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>>>>> v3.5.3-2528-gbee642f GIT Date: 2015-03-29 20:36:38 -0500 >>>>>>>>>> [0]PETSC ERROR: ./cube_with_hole on a arch-linux2-c-debug named >>>>>>>>>> pacotaco by justin Tue Apr 7 19:09:12 2015 >>>>>>>>>> [0]PETSC ERROR: Configure options --download-chaco >>>>>>>>>> --download-exodusii --download-fblaslapack --download-hdf5 --download-metis >>>>>>>>>> --download-mpich --download-mumps --download-netcdf --download-parmetis >>>>>>>>>> --download-scalapack --download-triangle --with-cc=gcc --with-cmake=cmake >>>>>>>>>> --with-cxx=g++ --with-debugging=1 --with-fc=gfortran --with-valgrind=1 >>>>>>>>>> PETSC_ARCH=arch-linux2-c-debug >>>>>>>>>> [0]PETSC ERROR: #1 PetscSectionGetDof() line 499 in >>>>>>>>>> /home/justin/petsc-dev/src/vec/is/utils/vsectionis.c >>>>>>>>>> [0]PETSC ERROR: #2 DMPlexGetConeSize() line 841 in >>>>>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>>>>>>> [0]PETSC ERROR: #3 DMPlexGetTransitiveClosure() line 1342 in >>>>>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>>>>>>> [0]PETSC ERROR: #4 DMPlexLabelComplete() line 88 in >>>>>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plexsubmesh.c >>>>>>>>>> [0]PETSC ERROR: #5 DMCreateDefaultSection_Plex() line 5605 in >>>>>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>>>>>>> [0]PETSC ERROR: #6 DMGetDefaultSection() line 3035 in >>>>>>>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>>>>>>> [0]PETSC ERROR: #7 DMGetDefaultGlobalSection() line 3266 in >>>>>>>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>>>>>>> [0]PETSC ERROR: #8 DMCreateGlobalVector_Section_Private() line 13 >>>>>>>>>> in /home/justin/petsc-dev/src/dm/interface/dmi.c >>>>>>>>>> [0]PETSC ERROR: #9 DMCreateGlobalVector_Plex() line 1170 in >>>>>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plexcreate.c >>>>>>>>>> [0]PETSC ERROR: #10 DMCreateGlobalVector() line 698 in >>>>>>>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>>>>>>> [0]PETSC ERROR: #11 main() line 363 in >>>>>>>>>> /home/justin/Dropbox/Research_Topics/Code_PETSc/Nonneg/cube_with_hole.c >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> From this error, it seems it's trying to access sieve points that >>>>>>>>>> non longer exist. And I think it has to do with the label "marker" still >>>>>>>>>> contains data from the interpolated mesh. Is there a way to "uninterpolate" >>>>>>>>>> or remove such points? Or is there a better way of approaching this whole >>>>>>>>>> thing? >>>>>>>>>> >>>>>>>>> >>>>>>>>> 1) It would be easy to write a small function to throw out the >>>>>>>>> points in the label that do not exist in the DM. You >>>>>>>>> would just extract each stratumIS and then Clear each invalid >>>>>>>>> point. >>>>>>>>> >>>>>>>>> 2) However, I do not understand the rationale for this above. You >>>>>>>>> could just call MarkBoundaryFaces on the final mesh. >>>>>>>>> If you are really trying to preserve a label across regular >>>>>>>>> refinement AND you really do not want an interpolated mesh, >>>>>>>>> then code up 1). I have yet to see a case where the extra >>>>>>>>> memory for edges and faces makes a difference, but it may exist. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Justin Chang >>>>>>>>>> PhD Candidate, Civil Engineering - Computational Sciences >>>>>>>>>> University of Houston, Department of Civil and Environmental >>>>>>>>>> Engineering >>>>>>>>>> Houston, TX 77004 >>>>>>>>>> (512) 963-3262 >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Justin Chang >>>>>>>> PhD Candidate, Civil Engineering - Computational Sciences >>>>>>>> University of Houston, Department of Civil and Environmental >>>>>>>> Engineering >>>>>>>> Houston, TX 77004 >>>>>>>> (512) 963-3262 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Justin Chang >>>>>> PhD Candidate, Civil Engineering - Computational Sciences >>>>>> University of Houston, Department of Civil and Environmental >>>>>> Engineering >>>>>> Houston, TX 77004 >>>>>> (512) 963-3262 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>>> >>>> -- >>>> Justin Chang >>>> PhD Candidate, Civil Engineering - Computational Sciences >>>> University of Houston, Department of Civil and Environmental Engineering >>>> Houston, TX 77004 >>>> (512) 963-3262 >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> Justin Chang >> PhD Candidate, Civil Engineering - Computational Sciences >> University of Houston, Department of Civil and Environmental Engineering >> Houston, TX 77004 >> (512) 963-3262 >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 10 15:41:21 2015 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 10 Apr 2015 15:41:21 -0500 Subject: [petsc-users] Uninterpolating DMLabels? In-Reply-To: References: Message-ID: On Fri, Apr 10, 2015 at 3:39 PM, Justin Chang wrote: > That makes a lot of sense now. So if I understand this correctly, with > -dm_refine 2, all of the fine cells for coarse cell number 3 would be: > > 192 193 ... 255 > > -dm_refine 3 for the same cell would be: > > 1536 1537 .... 2047 > > and so on. And this would hold true even for distributed meshes? > Yes. The nice thing with regular refinement is that all the operations are completely local. Thanks, Matt > Thanks, > > On Fri, Apr 10, 2015 at 1:21 PM, Matthew Knepley > wrote: > >> On Fri, Apr 10, 2015 at 12:58 PM, Justin Chang wrote: >> >>> Sorry I am a little confused here. I understand that each refinement of >>> a 3D simplex element yields 8 subcells, but give the expression c_f \in >>> [C c_c, C (c_c + 1) ) what exactly are these and/or what do they mean: >>> >> >> There are 8 fine cells created in each coarse cell. If the coarse cell >> number is c_c, then the fine cells numbers are >> >> C c_c, C c_c + 1, ..., C (c_c + 1) -1 >> >> So, if the coarse cell number is 3, then the fine cells inside are >> >> 24, 25, 26, 27, 28, 29, 30, 31 >> >> Thanks, >> >> Matt >> >> >>> c_f \in [... ) >>> C >>> c_c >>> (c_c + 1) >>> >>> Originally I was thinking the functions DMPlexGetTransitiveClosure(...) >>> and/or DMPlexGetCoarseDM(...) need to be invoked somehow. Do they? >>> >>> Again thanks for your help >>> >>> -- >>> Justin Chang >>> PhD Candidate, Civil Engineering - Computational Sciences >>> University of Houston, Department of Civil and Environmental Engineering >>> Houston, TX 77004 >>> (512) 963-3262 >>> >>> On Fri, Apr 10, 2015 at 11:38 AM, Matthew Knepley >>> wrote: >>> >>>> On Fri, Apr 10, 2015 at 1:01 AM, Justin Chang wrote: >>>> >>>>> Matt, >>>>> >>>>> I am not sure I follow what you're saying here. Currently I am working >>>>> with 3D simplex elements so if the variable numSubcells is what you're >>>>> referring to in CellRefinerGetAffineTransforms_Internal(), there's no >>>>> example for REFINER_SIMPLEX_3D. What would it be in that case, 7? >>>>> >>>> >>>> Its 8. There are 8 fine cells for every coarse cell. >>>> >>>> >>>>> What I am looking for is a function and/or a set of algorithms where >>>>> given a sieve point m from the original coarse mesh, I can obtain the >>>>> corresponding set of (fine mesh) sieve points. I am assuming c_f \in [C >>>>> c_c, C (c_c + 1) ) describes just that? >>>>> >>>> >>>> Yes, C = 8. >>>> >>>> >>>>> Is there an existing DMPlex related function that implements this >>>>> directly? Because I don't see how CellRefinerGetAffineTransforms_Internal() >>>>> will help. >>>>> >>>> >>>> I would just give you C. Yes, I would just write it. It should be 1 >>>> line. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> >>>>> Thanks, >>>>> >>>>> >>>>> On Thu, Apr 9, 2015 at 8:20 PM, Matthew Knepley >>>>> wrote: >>>>> >>>>>> On Thu, Apr 9, 2015 at 8:14 PM, Justin Chang wrote: >>>>>> >>>>>>> Matt, thank you for the clarification. One more question somewhat on >>>>>>> a related note. >>>>>>> >>>>>>> I have auxiliary data corresponding to the permeability of a >>>>>>> reservoir (for instance, the SPE10 benchmark problem). The data is >>>>>>> cell-wise and read from a binary file. I want to refine both the DM and >>>>>>> this data. Is there a way to obtain the mapping from the coarse mesh to the >>>>>>> fine mesh? What I mean is, given a coarse cell, can I somehow get the >>>>>>> transitive closure of the fine cells it created during the refinement >>>>>>> process? This would allow me to copy the value of the coarse cell data onto >>>>>>> it's children. >>>>>>> >>>>>> >>>>>> If you are doing regular refinement, there is a simple formula: >>>>>> >>>>>> c_f \in [ C c_c, C (c_c +1) ) >>>>>> >>>>>> where C is the number of cells created from each cell. You can get C >>>>>> from >>>>>> >>>>>> CellRefinerGetAffineTransforms_Internal() >>>>>> >>>>>> which I could promote to a real interface function. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> On Thu, Apr 9, 2015 at 7:52 PM, Matthew Knepley >>>>>>> wrote: >>>>>>> >>>>>>>> On Thu, Apr 9, 2015 at 7:21 PM, Justin Chang >>>>>>>> wrote: >>>>>>>> >>>>>>>>> 1) I did what you had suggested and it works very well thanks. >>>>>>>>> >>>>>>>> >>>>>>>> Cool. >>>>>>>> >>>>>>>> >>>>>>>>> 2) In my case, calling uninterpolate does make a difference, just >>>>>>>>> not necessarily in the solver step (I am using Tao's BLMVM). I attached two >>>>>>>>> output files, one containing the timings and summaries with uninterpolating >>>>>>>>> and one without uninterpolating. And the source code if you wanted to >>>>>>>>> verify the "correctness" of my implementation. Namely, the biggest >>>>>>>>> difference I see is in setting up the PetscSection. When uninterpolating >>>>>>>>> the data this steps takes 8.64039 seconds but when I do not uninterpolate >>>>>>>>> the data it takes 61.2977 seconds. Is this "significant" enough, or is it >>>>>>>>> unimportant given the fact that this step occurs only once during any >>>>>>>>> simulation? >>>>>>>>> >>>>>>>> >>>>>>>> That is significant. Most of that time is in the operator >>>>>>>> preallocation routine. It was cleaner to first gather the point >>>>>>>> adjacency information, and then push the dof information on top. If >>>>>>>> I aggressively merged the dof data, I could have >>>>>>>> pruned a bunch of the work here and gotten much closer to the first >>>>>>>> time, at the expense of more complicated code. >>>>>>>> I guess this is the right pattern if you know that you will only >>>>>>>> have unknowns on cells and vertices. >>>>>>>> >>>>>>>> >>>>>>>>> 3) Also, about DMPlexDistribute(...). Right now it still seems >>>>>>>>> extremely slow, hence why I am distributing a coarse mesh and having each >>>>>>>>> processor refine its local portion of the mesh. In my current >>>>>>>>> implementation, I have rank 0 distribute the mesh via ParMETIS. Will there >>>>>>>>> eventually be a methodology to do parallel I/O then redistribution for load >>>>>>>>> balancing or does it already exist? >>>>>>>>> >>>>>>>> >>>>>>>> Redistribution for load balancing already exists. The stumbling >>>>>>>> block right now is parallel mesh import. However, >>>>>>>> the group at ICL (Michael Lange and Gerard Gorman) is working on >>>>>>>> this, and we expect to have it working by >>>>>>>> the end of summer. Until then, I recommend exactly what you are >>>>>>>> doing. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> On Tue, Apr 7, 2015 at 8:37 PM, Matthew Knepley >>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> On Tue, Apr 7, 2015 at 7:18 PM, Justin Chang >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> So I am interpolating/uninterpolating my DMPlex mesh. This is >>>>>>>>>>> the mesh information for a cell-vertex mesh only: >>>>>>>>>>> >>>>>>>>>>> DM Object: 1 MPI processes >>>>>>>>>>> type: plex >>>>>>>>>>> DM_0x84000000_0 in 3 dimensions: >>>>>>>>>>> 0-cells: 159 >>>>>>>>>>> 3-cells: 592 >>>>>>>>>>> Labels: >>>>>>>>>>> marker: 1 strata of sizes (100) >>>>>>>>>>> depth: 2 strata of sizes (159, 592) >>>>>>>>>>> >>>>>>>>>>> I am interpolating the mesh with the following lines: >>>>>>>>>>> >>>>>>>>>>> ierr = DMPlexInterpolate(*dm, &idm);CHKERRQ(ierr); >>>>>>>>>>> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); >>>>>>>>>>> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); >>>>>>>>>>> ierr = DMPlexGetLabel(idm, "marker", &label);CHKERRQ(ierr); >>>>>>>>>>> ierr = DMPlexMarkBoundaryFaces(idm,label);CHKERRQ(ierr); >>>>>>>>>>> ierr = DMDestroy(dm);CHKERRQ(ierr); >>>>>>>>>>> *dm = idm; >>>>>>>>>>> >>>>>>>>>>> And the resulting mesh: >>>>>>>>>>> >>>>>>>>>>> DM Object: 1 MPI processes >>>>>>>>>>> type: plex >>>>>>>>>>> DM_0x84000000_1 in 3 dimensions: >>>>>>>>>>> 0-cells: 159 >>>>>>>>>>> 1-cells: 845 >>>>>>>>>>> 2-cells: 1280 >>>>>>>>>>> 3-cells: 592 >>>>>>>>>>> Labels: >>>>>>>>>>> marker: 1 strata of sizes (292) >>>>>>>>>>> depth: 4 strata of sizes (159, 845, 1280, 592) >>>>>>>>>>> >>>>>>>>>>> The reason I did this is so that incase I decide to refine the >>>>>>>>>>> mesh with "-dm_refine " the DM will be sure to include the "marker" >>>>>>>>>>> points in the refinement process. Now, if I decide to uninterpolate the >>>>>>>>>>> mesh with the following lines: >>>>>>>>>>> >>>>>>>>>>> ierr = DMPlexUninterpolate(*dm, &idm);CHKERRQ(ierr); >>>>>>>>>>> ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); >>>>>>>>>>> ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); >>>>>>>>>>> ierr = DMDestroy(dm);CHKERRQ(ierr); >>>>>>>>>>> *dm = idm; >>>>>>>>>>> >>>>>>>>>>> I end up with this mesh: >>>>>>>>>>> >>>>>>>>>>> DM Object: 1 MPI processes >>>>>>>>>>> type: plex >>>>>>>>>>> DM_0x84000000_2 in 3 dimensions: >>>>>>>>>>> 0-cells: 159 >>>>>>>>>>> 3-cells: 592 >>>>>>>>>>> Labels: >>>>>>>>>>> marker: 1 strata of sizes (292) >>>>>>>>>>> depth: 2 strata of sizes (159, 592) >>>>>>>>>>> >>>>>>>>>>> The problem is that although the cells and vertices have gone >>>>>>>>>>> back to the original number, the "marker" label still includes face/edge >>>>>>>>>>> points. This gives me an error once I invoke DMCreateGobalVector(...). >>>>>>>>>>> >>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>> [0]PETSC ERROR: Argument out of range >>>>>>>>>>> [0]PETSC ERROR: Section point 755 should be in [0, 751) >>>>>>>>>>> [0]PETSC ERROR: See >>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>>>>>>>>> shooting. >>>>>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>>>>>> v3.5.3-2528-gbee642f GIT Date: 2015-03-29 20:36:38 -0500 >>>>>>>>>>> [0]PETSC ERROR: ./cube_with_hole on a arch-linux2-c-debug named >>>>>>>>>>> pacotaco by justin Tue Apr 7 19:09:12 2015 >>>>>>>>>>> [0]PETSC ERROR: Configure options --download-chaco >>>>>>>>>>> --download-exodusii --download-fblaslapack --download-hdf5 --download-metis >>>>>>>>>>> --download-mpich --download-mumps --download-netcdf --download-parmetis >>>>>>>>>>> --download-scalapack --download-triangle --with-cc=gcc --with-cmake=cmake >>>>>>>>>>> --with-cxx=g++ --with-debugging=1 --with-fc=gfortran --with-valgrind=1 >>>>>>>>>>> PETSC_ARCH=arch-linux2-c-debug >>>>>>>>>>> [0]PETSC ERROR: #1 PetscSectionGetDof() line 499 in >>>>>>>>>>> /home/justin/petsc-dev/src/vec/is/utils/vsectionis.c >>>>>>>>>>> [0]PETSC ERROR: #2 DMPlexGetConeSize() line 841 in >>>>>>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>>>>>>>> [0]PETSC ERROR: #3 DMPlexGetTransitiveClosure() line 1342 in >>>>>>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>>>>>>>> [0]PETSC ERROR: #4 DMPlexLabelComplete() line 88 in >>>>>>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plexsubmesh.c >>>>>>>>>>> [0]PETSC ERROR: #5 DMCreateDefaultSection_Plex() line 5605 in >>>>>>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plex.c >>>>>>>>>>> [0]PETSC ERROR: #6 DMGetDefaultSection() line 3035 in >>>>>>>>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>>>>>>>> [0]PETSC ERROR: #7 DMGetDefaultGlobalSection() line 3266 in >>>>>>>>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>>>>>>>> [0]PETSC ERROR: #8 DMCreateGlobalVector_Section_Private() line >>>>>>>>>>> 13 in /home/justin/petsc-dev/src/dm/interface/dmi.c >>>>>>>>>>> [0]PETSC ERROR: #9 DMCreateGlobalVector_Plex() line 1170 in >>>>>>>>>>> /home/justin/petsc-dev/src/dm/impls/plex/plexcreate.c >>>>>>>>>>> [0]PETSC ERROR: #10 DMCreateGlobalVector() line 698 in >>>>>>>>>>> /home/justin/petsc-dev/src/dm/interface/dm.c >>>>>>>>>>> [0]PETSC ERROR: #11 main() line 363 in >>>>>>>>>>> /home/justin/Dropbox/Research_Topics/Code_PETSc/Nonneg/cube_with_hole.c >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> From this error, it seems it's trying to access sieve points >>>>>>>>>>> that non longer exist. And I think it has to do with the label "marker" >>>>>>>>>>> still contains data from the interpolated mesh. Is there a way to >>>>>>>>>>> "uninterpolate" or remove such points? Or is there a better way of >>>>>>>>>>> approaching this whole thing? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 1) It would be easy to write a small function to throw out the >>>>>>>>>> points in the label that do not exist in the DM. You >>>>>>>>>> would just extract each stratumIS and then Clear each invalid >>>>>>>>>> point. >>>>>>>>>> >>>>>>>>>> 2) However, I do not understand the rationale for this above. You >>>>>>>>>> could just call MarkBoundaryFaces on the final mesh. >>>>>>>>>> If you are really trying to preserve a label across regular >>>>>>>>>> refinement AND you really do not want an interpolated mesh, >>>>>>>>>> then code up 1). I have yet to see a case where the extra >>>>>>>>>> memory for edges and faces makes a difference, but it may exist. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Justin Chang >>>>>>>>>>> PhD Candidate, Civil Engineering - Computational Sciences >>>>>>>>>>> University of Houston, Department of Civil and Environmental >>>>>>>>>>> Engineering >>>>>>>>>>> Houston, TX 77004 >>>>>>>>>>> (512) 963-3262 >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>>> experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Justin Chang >>>>>>>>> PhD Candidate, Civil Engineering - Computational Sciences >>>>>>>>> University of Houston, Department of Civil and Environmental >>>>>>>>> Engineering >>>>>>>>> Houston, TX 77004 >>>>>>>>> (512) 963-3262 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Justin Chang >>>>>>> PhD Candidate, Civil Engineering - Computational Sciences >>>>>>> University of Houston, Department of Civil and Environmental >>>>>>> Engineering >>>>>>> Houston, TX 77004 >>>>>>> (512) 963-3262 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Justin Chang >>>>> PhD Candidate, Civil Engineering - Computational Sciences >>>>> University of Houston, Department of Civil and Environmental >>>>> Engineering >>>>> Houston, TX 77004 >>>>> (512) 963-3262 >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >>> >>> -- >>> Justin Chang >>> PhD Candidate, Civil Engineering - Computational Sciences >>> University of Houston, Department of Civil and Environmental Engineering >>> Houston, TX 77004 >>> (512) 963-3262 >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From fande.kong at colorado.edu Fri Apr 10 15:58:15 2015 From: fande.kong at colorado.edu (Fande Kong) Date: Fri, 10 Apr 2015 14:58:15 -0600 Subject: [petsc-users] timer in PETSc In-Reply-To: References: Message-ID: Thanks. I did add my several events. From the following cases, we possibly can find an explanation. (1) Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.2115e+03 91.0% 8.6211e+12 2.3% 5.937e+07 3.2% 7.314e+03 77.2% 2.344e+03 10.1% 1: MG Apply: 1.1994e+02 9.0% 3.6819e+14 97.7% 1.780e+09 96.8% 2.160e+03 22.8% 2.080e+04 89.9% (2) Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 9.9736e+02 47.6% 2.4648e+13 0.8% 5.813e+07 1.0% 1.013e+03 32.5% 3.151e+03 2.7% 1: MG Apply: 1.0959e+03 52.4% 2.9005e+15 99.2% 5.734e+09 99.0% 2.102e+03 67.5% 1.114e+05 97.2% In the fist case, we do not have any warnings and the second case produces a warning. I am using MG as a preconditioner. When the MG is cheap like case 1, the Main Stage time is larger than the MG stage. We do not have any warnings. If the MG is expensive, the Main Stage time is smaller than MG. We have a warning now because SNESSolve time is larger than Main Stage. SNESSolve time includes outer solver and MG preconditioner. Why we do not add MG time to the Main Stage. Should the Main Stage be the total time? Fande, On Fri, Apr 10, 2015 at 2:30 PM, Barry Smith wrote: > > Are you adding your own events or stages or just running with > -log_summary? > > If you added your own events or stages this likely happened because you > changed between stages within an event so its start time is logged in one > stage but the end time correct gets put into a different stage. > > Barry > > > On Apr 10, 2015, at 3:22 PM, Fande Kong wrote: > > > > Hi all, > > > > I am running an application with more than 4,000 cores. I get a warning > in the -log_summary: Warning -- total time of even greater than time of > entire stage -- something is wrong with the timer. > > > > I have no any clues for this warning. Any suggestions to remove it? > > > > Thanks, > > > > Fande, > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jchang27 at uh.edu Fri Apr 10 16:16:14 2015 From: jchang27 at uh.edu (Justin Chang) Date: Fri, 10 Apr 2015 16:16:14 -0500 Subject: [petsc-users] Tao iterations Message-ID: Hi all, Is there a way to generically obtain the number of Tao iterations? I am looking through the -help options for Tao and I don't see any metric where you can output this quantity in the manner that you could for SNES or KSP solves. I am currently using blmvm and tron, and the only way I can see getting this metric is by outputting -tao_view and/or -tao_monitor and manually finding this number. I find this cumbersome especially for transient problems where I would like to simply have this number printed for each step instead of ending up with unnecessary info. Thanks, -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Apr 10 16:16:25 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 10 Apr 2015 16:16:25 -0500 Subject: [petsc-users] timer in PETSc In-Reply-To: References: Message-ID: <5E3C1995-1B82-4E46-AD23-AEA5F05342E1@mcs.anl.gov> Nothing is clear to me. One way to debug is put in just one stage at a time (comment out all others) and make sure each one generates consistent numbers by itself. Then add a second stage. You have to debug this just like any other bug... Barry > On Apr 10, 2015, at 3:58 PM, Fande Kong wrote: > > Thanks. I did add my several events. From the following cases, we possibly can find an explanation. > > (1) > > Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total counts %Total Avg %Total counts %Total > 0: Main Stage: 1.2115e+03 91.0% 8.6211e+12 2.3% 5.937e+07 3.2% 7.314e+03 77.2% 2.344e+03 10.1% > 1: MG Apply: 1.1994e+02 9.0% 3.6819e+14 97.7% 1.780e+09 96.8% 2.160e+03 22.8% 2.080e+04 89.9% > > (2) Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total counts %Total Avg %Total counts %Total > 0: Main Stage: 9.9736e+02 47.6% 2.4648e+13 0.8% 5.813e+07 1.0% 1.013e+03 32.5% 3.151e+03 2.7% > 1: MG Apply: 1.0959e+03 52.4% 2.9005e+15 99.2% 5.734e+09 99.0% 2.102e+03 67.5% 1.114e+05 97.2% > > > In the fist case, we do not have any warnings and the second case produces a warning. I am using MG as a preconditioner. When the MG is cheap like case 1, the Main Stage time is larger than the MG stage. We do not have any warnings. If the MG is expensive, the Main Stage time is smaller than MG. We have a warning now because SNESSolve time is larger than Main Stage. SNESSolve time includes outer solver and MG preconditioner. > > Why we do not add MG time to the Main Stage. Should the Main Stage be the total time? > > Fande, > > On Fri, Apr 10, 2015 at 2:30 PM, Barry Smith wrote: > > Are you adding your own events or stages or just running with -log_summary? > > If you added your own events or stages this likely happened because you changed between stages within an event so its start time is logged in one stage but the end time correct gets put into a different stage. > > Barry > > > On Apr 10, 2015, at 3:22 PM, Fande Kong wrote: > > > > Hi all, > > > > I am running an application with more than 4,000 cores. I get a warning in the -log_summary: Warning -- total time of even greater than time of entire stage -- something is wrong with the timer. > > > > I have no any clues for this warning. Any suggestions to remove it? > > > > Thanks, > > > > Fande, > > From elbueler at alaska.edu Fri Apr 10 16:25:01 2015 From: elbueler at alaska.edu (Ed Bueler) Date: Fri, 10 Apr 2015 15:25:01 -0600 Subject: [petsc-users] portability question for reduce operations Message-ID: Dear PETSc -- Now that PetscGlobalSum() and related are gone, I am exposed to MPI_Allreduce(), which is fine. PETSc defines some MPIU_ data types, apparently for portability. But runs with MPIU_INT (option 1 below) break for me on one machine; this seems to be about the petsc configuration. I have fixed this by using option 3 below. What is the most portable of these three, or is there yet a more portable way?: option 1. full use of MPIU_ types and ops: PetscReal x, xloc; PetscInt n, nloc; MPI_Allreduce(&xloc,&x,1,MPIU_REAL,MPIU_SUM,comm); MPI_Allreduce(&nloc,&n,1,MPIU_INT,MPIU_SUM,comm); option 2. use of MPIU_REAL but not MPIU_INT: PetscReal x, xloc; PetscInt n, nloc; MPI_Allreduce(&xloc,&x,1,MPIU_REAL,MPIU_SUM,comm); MPI_Allreduce(&nloc,&n,1,MPI_INT,MPI_SUM,comm); option 3. only use MPI's primitive MPI_ types: PetscReal x, xloc; PetscInt n, nloc; MPI_Allreduce(&xloc,&x,1,MPI_DOUBLE,MPI_SUM,comm); MPI_Allreduce(&nloc,&n,1,MPI_INT,MPI_SUM,comm); I couldn't find discussion of allreduce usage in the petsc manual. Thanks! Ed PS With option 1 on one machine I get "PETSC ERROR: Can only handle MPIU_REAL or MPIU_COMPLEX data types" in parallel but no error in serial. On my two other machines there is no error with option 1. All three machines work fine with option 3. I have not tested option 2. -- Ed Bueler Dept of Math and Stat and Geophysical Institute University of Alaska Fairbanks Fairbanks, AK 99775-6660 301C Chapman and 410D Elvey 907 474-7693 and 907 474-7199 (fax 907 474-5394) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Apr 10 16:31:52 2015 From: jed at jedbrown.org (Jed Brown) Date: Fri, 10 Apr 2015 15:31:52 -0600 Subject: [petsc-users] portability question for reduce operations In-Reply-To: References: Message-ID: <87twwnd60n.fsf@jedbrown.org> Ed Bueler writes: > option 1. full use of MPIU_ types and ops: > > PetscReal x, xloc; > PetscInt n, nloc; > MPI_Allreduce(&xloc,&x,1,MPIU_REAL,MPIU_SUM,comm); > MPI_Allreduce(&nloc,&n,1,MPIU_INT,MPIU_SUM,comm); Use MPI_SUM for integer types. MPIU_SUM is for real/scalar types. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Fri Apr 10 16:42:55 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 10 Apr 2015 16:42:55 -0500 Subject: [petsc-users] Tao iterations In-Reply-To: References: Message-ID: <1B558333-3422-4D80-97CC-4373BA2F0633@mcs.anl.gov> Justin, Sorry TAO simply doesn't even collect this information currently. But yes we should definitely make it available! Jason, Could you please add this; almost all the TaoSolve_xxx() have a local variable iter; change that to tao->niter (I'm guess this is suppose to capture this information) and add a TaoGetIterationNumber() and the uses can access this. Also modify at the end of TaoSolve() -tao_converged_reason to also print the iteration count. At the same time since you add this you can add a tao->totalits which would accumulate all iterations over all the solves for that Tao object and the routine TaoGetTotalIterations() to access this. Note that TaoSolve() would initialize tao->niter = 0 at the top. Thanks Barry > On Apr 10, 2015, at 4:16 PM, Justin Chang wrote: > > Hi all, > > Is there a way to generically obtain the number of Tao iterations? I am looking through the -help options for Tao and I don't see any metric where you can output this quantity in the manner that you could for SNES or KSP solves. I am currently using blmvm and tron, and the only way I can see getting this metric is by outputting -tao_view and/or -tao_monitor and manually finding this number. I find this cumbersome especially for transient problems where I would like to simply have this number printed for each step instead of ending up with unnecessary info. > > Thanks, > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 From bsmith at mcs.anl.gov Fri Apr 10 17:03:57 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 10 Apr 2015 17:03:57 -0500 Subject: [petsc-users] portability question for reduce operations In-Reply-To: <87twwnd60n.fsf@jedbrown.org> References: <87twwnd60n.fsf@jedbrown.org> Message-ID: <4DCA9F6C-989C-49B6-A544-746DDDB9910D@mcs.anl.gov> > On Apr 10, 2015, at 4:31 PM, Jed Brown wrote: > > Ed Bueler writes: >> option 1. full use of MPIU_ types and ops: >> >> PetscReal x, xloc; >> PetscInt n, nloc; >> MPI_Allreduce(&xloc,&x,1,MPIU_REAL,MPIU_SUM,comm); >> MPI_Allreduce(&nloc,&n,1,MPIU_INT,MPIU_SUM,comm); > > Use MPI_SUM for integer types. MPIU_SUM is for real/scalar types. With Satish's cleanup of basic type names and organization we should fix this so users don't get bitten by the inconsistency. It's a pity MPI didn't provide a way to incrementally add new types for already defined Op. Barry From bsmith at mcs.anl.gov Fri Apr 10 17:27:47 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 10 Apr 2015 17:27:47 -0500 Subject: [petsc-users] monitoring the convergence of fieldsplit 0 and 1 In-Reply-To: <1428671243078.94@marin.nl> References: <1428666513941.72745@marin.nl> <75D79823-7AE0-47A7-BE9E-15AB81C3581E@mcs.anl.gov> <1428671243078.94@marin.nl> Message-ID: <9EA3A2C1-5372-44A8-B0B7-4ADAF1D89819@mcs.anl.gov> Chris, I have added KSPGetTotalIterations() to the branch barry/add-ksp-total-iterations/master and next. After tests it will go into master Barry > On Apr 10, 2015, at 8:07 AM, Klaij, Christiaan wrote: > > Barry, > > Sure, I can call PCFieldSplitGetSubKSP() to get the fieldsplit_0 > ksp and then KSPGetIterationNumber, but what does this number > mean? > > It appears to be the number of iterations of the last time that > the subsystem was solved, right? If so, this corresponds to the > last iteration of the coupled system, how about all the previous > iterations? > > Chris > ________________________________________ > From: Barry Smith > Sent: Friday, April 10, 2015 2:48 PM > To: Klaij, Christiaan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] monitoring the convergence of fieldsplit 0 and 1 > > Chris, > > It appears you should call PCFieldSplitGetSubKSP() and then get the information you want out of the individual KSPs. If this doesn't work please let us know. > > Barry > >> On Apr 10, 2015, at 6:48 AM, Klaij, Christiaan wrote: >> >> A question when using PCFieldSplit: for each linear iteration of >> the system, how many iterations for fielsplit 0 and 1? >> >> One way to find out is to run with -ksp_monitor, >> -fieldsplit_0_ksp_monitor and -fieldsplit_0_ksp_monitor. This >> gives the complete convergence history. >> >> Another way, suggested by Matt, is to use -ksp_monitor, >> -fieldsplit_0_ksp_converged_reason and >> -fieldsplit_1_ksp_converged_reason. This gives only the totals >> for fieldsplit 0 and 1 (but without saying for which one). >> >> Both ways require to somehow process the output, which is a bit >> inconvenient. Could KSPGetResidualHistory perhaps return (some) >> information on the subsystems' convergence for processing inside >> the code? >> >> Chris >> >> >> dr. ir. Christiaan Klaij >> CFD Researcher >> Research & Development >> E mailto:C.Klaij at marin.nl >> T +31 317 49 33 44 >> >> >> MARIN >> 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands >> T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl >> > From bsmith at mcs.anl.gov Fri Apr 10 17:43:45 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 10 Apr 2015 17:43:45 -0500 Subject: [petsc-users] Recommended examples for using KSP and multigrid with DM In-Reply-To: <5525EF91.3020709@gmail.com> References: <5525EF91.3020709@gmail.com> Message-ID: <5F8DEB12-D4B5-4FA6-8F18-B0B667071CE9@mcs.anl.gov> Are you using DMDA or DMPLEX or some other DM? Does the Poisson equation change (variable coefficient) or is it a constant coefficient with the same values for all solves? ex29.c could still possibly serve as a model for the Poisson subproblem, why do you say it is no longer valid? Barry > On Apr 8, 2015, at 10:18 PM, TAY wee-beng wrote: > > Hi, > > I was following example ex29 in the past to write the code to solve my Poisson eqn using DM. Now it seems that ex29 is no longer valid. Which example should I follow now? > > Also, I am using KSPBCGS to solve my momentum eqn in the past. Do I need to change much if I'm now using DM? If so, what is a good example? > > -- > Thank you > > Yours sincerely, > > TAY wee-beng > From bsmith at mcs.anl.gov Fri Apr 10 17:44:12 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 10 Apr 2015 17:44:12 -0500 Subject: [petsc-users] MPI stdin In-Reply-To: <0fb69621f4c0737aaf2f73cb7c2bbd58@upm.es> References: <0fb69621f4c0737aaf2f73cb7c2bbd58@upm.es> Message-ID: I guess you should post this to the MPICH mailing list. Barry > On Apr 7, 2015, at 10:35 AM, Oliver Browne wrote: > > Hi, > > Does MPICH have an option similar to the below (from open_MPI) > > -stdin|--stdin > Specify procs to receive stdin [rank, all, none] > (default: 0, indicating rank 0) > > Thanks, > > Ollie From jason.sarich at gmail.com Fri Apr 10 17:53:35 2015 From: jason.sarich at gmail.com (Jason Sarich) Date: Fri, 10 Apr 2015 17:53:35 -0500 Subject: [petsc-users] Tao iterations In-Reply-To: References: Message-ID: Hi Justin, I'll get this in. I assume that displaying the number of iterations with tao_converged_reason is what you are asking for in particular? Or did you have something else in mind? Jason On Apr 10, 2015 16:42, "Smith, Barry F." wrote: > > Justin, > > Sorry TAO simply doesn't even collect this information currently. But > yes we should definitely make it available! > > Jason, > > Could you please add this; almost all the TaoSolve_xxx() have a local > variable iter; change that to tao->niter (I'm guess this is suppose to > capture this information) and add a TaoGetIterationNumber() and the uses > can access this. Also modify at the end of TaoSolve() -tao_converged_reason > to also print the iteration count. At the same time since you add this you > can add a tao->totalits which would accumulate all iterations over all the > solves for that Tao object and the routine TaoGetTotalIterations() to > access this. Note that TaoSolve() would initialize tao->niter = 0 at the > top. > > Thanks > > Barry > > > > > On Apr 10, 2015, at 4:16 PM, Justin Chang wrote: > > > > Hi all, > > > > Is there a way to generically obtain the number of Tao iterations? I am > looking through the -help options for Tao and I don't see any metric where > you can output this quantity in the manner that you could for SNES or KSP > solves. I am currently using blmvm and tron, and the only way I can see > getting this metric is by outputting -tao_view and/or -tao_monitor and > manually finding this number. I find this cumbersome especially for > transient problems where I would like to simply have this number printed > for each step instead of ending up with unnecessary info. > > > > Thanks, > > > > > > -- > > Justin Chang > > PhD Candidate, Civil Engineering - Computational Sciences > > University of Houston, Department of Civil and Environmental Engineering > > Houston, TX 77004 > > (512) 963-3262 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jychang48 at gmail.com Fri Apr 10 18:28:57 2015 From: jychang48 at gmail.com (Justin Chang) Date: Fri, 10 Apr 2015 18:28:57 -0500 Subject: [petsc-users] Tao iterations In-Reply-To: References: Message-ID: Whatever is convenient and/or follow the "PETSc" standards. Something similar to SNESGetIterationNumber() or KSPGetIterationNumber() would be nice. Ideally I want my code to look like this: ierr = TaoGetIterationNumber(tao,&its);CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD, "Number of Tao iterations = %D\n", its); Thanks :) On Fri, Apr 10, 2015 at 5:53 PM, Jason Sarich wrote: > Hi Justin, I'll get this in. I assume that displaying the number of > iterations with tao_converged_reason is what you are asking for in > particular? Or did you have something else in mind? > > Jason > On Apr 10, 2015 16:42, "Smith, Barry F." wrote: > >> >> Justin, >> >> Sorry TAO simply doesn't even collect this information currently. But >> yes we should definitely make it available! >> >> Jason, >> >> Could you please add this; almost all the TaoSolve_xxx() have a >> local variable iter; change that to tao->niter (I'm guess this is suppose >> to capture this information) and add a TaoGetIterationNumber() and the uses >> can access this. Also modify at the end of TaoSolve() -tao_converged_reason >> to also print the iteration count. At the same time since you add this you >> can add a tao->totalits which would accumulate all iterations over all the >> solves for that Tao object and the routine TaoGetTotalIterations() to >> access this. Note that TaoSolve() would initialize tao->niter = 0 at the >> top. >> >> Thanks >> >> Barry >> >> >> >> > On Apr 10, 2015, at 4:16 PM, Justin Chang wrote: >> > >> > Hi all, >> > >> > Is there a way to generically obtain the number of Tao iterations? I am >> looking through the -help options for Tao and I don't see any metric where >> you can output this quantity in the manner that you could for SNES or KSP >> solves. I am currently using blmvm and tron, and the only way I can see >> getting this metric is by outputting -tao_view and/or -tao_monitor and >> manually finding this number. I find this cumbersome especially for >> transient problems where I would like to simply have this number printed >> for each step instead of ending up with unnecessary info. >> > >> > Thanks, >> > >> > >> > -- >> > Justin Chang >> > PhD Candidate, Civil Engineering - Computational Sciences >> > University of Houston, Department of Civil and Environmental Engineering >> > Houston, TX 77004 >> > (512) 963-3262 >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Sat Apr 11 12:32:36 2015 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Sat, 11 Apr 2015 17:32:36 +0000 Subject: [petsc-users] monitoring the convergence of fieldsplit 0 and 1 In-Reply-To: <9EA3A2C1-5372-44A8-B0B7-4ADAF1D89819@mcs.anl.gov> References: <1428666513941.72745@marin.nl> <75D79823-7AE0-47A7-BE9E-15AB81C3581E@mcs.anl.gov> <1428671243078.94@marin.nl>, <9EA3A2C1-5372-44A8-B0B7-4ADAF1D89819@mcs.anl.gov> Message-ID: <1428773556766.53288@marin.nl> Barry, Matt Thanks, the total iteration count is already very useful. Looking forward to the custom monitors as well. Chris ________________________________________ From: Barry Smith Sent: Saturday, April 11, 2015 12:27 AM To: Klaij, Christiaan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] monitoring the convergence of fieldsplit 0 and 1 Chris, I have added KSPGetTotalIterations() to the branch barry/add-ksp-total-iterations/master and next. After tests it will go into master Barry > On Apr 10, 2015, at 8:07 AM, Klaij, Christiaan wrote: > > Barry, > > Sure, I can call PCFieldSplitGetSubKSP() to get the fieldsplit_0 > ksp and then KSPGetIterationNumber, but what does this number > mean? > > It appears to be the number of iterations of the last time that > the subsystem was solved, right? If so, this corresponds to the > last iteration of the coupled system, how about all the previous > iterations? > > Chris > ________________________________________ > From: Barry Smith > Sent: Friday, April 10, 2015 2:48 PM > To: Klaij, Christiaan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] monitoring the convergence of fieldsplit 0 and 1 > > Chris, > > It appears you should call PCFieldSplitGetSubKSP() and then get the information you want out of the individual KSPs. If this doesn't work please let us know. > > Barry > >> On Apr 10, 2015, at 6:48 AM, Klaij, Christiaan wrote: >> >> A question when using PCFieldSplit: for each linear iteration of >> the system, how many iterations for fielsplit 0 and 1? >> >> One way to find out is to run with -ksp_monitor, >> -fieldsplit_0_ksp_monitor and -fieldsplit_0_ksp_monitor. This >> gives the complete convergence history. >> >> Another way, suggested by Matt, is to use -ksp_monitor, >> -fieldsplit_0_ksp_converged_reason and >> -fieldsplit_1_ksp_converged_reason. This gives only the totals >> for fieldsplit 0 and 1 (but without saying for which one). >> >> Both ways require to somehow process the output, which is a bit >> inconvenient. Could KSPGetResidualHistory perhaps return (some) >> information on the subsystems' convergence for processing inside >> the code? >> >> Chris >> >> >> dr. ir. Christiaan Klaij >> CFD Researcher >> Research & Development >> E mailto:C.Klaij at marin.nl >> T +31 317 49 33 44 >> >> >> MARIN >> 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands >> T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl >> > From jchang27 at uh.edu Sat Apr 11 18:55:25 2015 From: jchang27 at uh.edu (Justin Chang) Date: Sat, 11 Apr 2015 18:55:25 -0500 Subject: [petsc-users] Uninterpolating a distributed mesh Message-ID: Hello, When I call DMPlexUninterpolate(...) on a distributed mesh (say 2 processors), it seems to overwrite the "ghost" points (i.e., the points not locally owned by the processor) and treats all points as if they are local to the processor. Say I have this 2D cell-vertex mesh: 14----15-----16 | \ 5 | \ 7 | | \ | \ | | 4 \ | 6 \ | 11----12-----13 | \ 1 | \ 3 | | \ | \ | | 0 \ | 2 \ | 8------9------10 Which results in the following DM: DM Object: 2 MPI processes type: plex DM_0x84000004_0 in 2 dimensions: 0-cells: 9 0 2-cells: 8 0 Labels: marker: 1 strata of sizes (8) depth: 2 strata of sizes (9, 8) I proceed by interpolating this DM: DM Object: 2 MPI processes type: plex DM_0x84000004_1 in 2 dimensions: 0-cells: 9 0 1-cells: 16 0 2-cells: 8 0 Labels: marker: 1 strata of sizes (16) depth: 3 strata of sizes (9, 16, 8) Then distributing across 2 processors: DM Object:Parallel Mesh 2 MPI processes type: plex Parallel Mesh in 2 dimensions: 0-cells: 6 6 1-cells: 9 9 2-cells: 4 4 Labels: marker: 1 strata of sizes (9) depth: 3 strata of sizes (6, 9, 4) I have the option of uniformly refining the mesh here but I choose not to for now. If my dofs are vertex based, then the global size of my DM vector is 9 and the local sizes for ranks 0 and 1 are 3 and 6 respectively. However, if I choose to uninterpolate the mesh by calling DMPlexUninterpolate(...), I get this: DM Object: 2 MPI processes type: plex DM_0x84000004_2 in 2 dimensions: 0-cells: 6 6 2-cells: 4 4 Labels: marker: 1 strata of sizes (5) depth: 2 strata of sizes (6, 4) And the global size of my DM vector becomes 12 and the local size for both ranks is 6. It looks like the ghost points in rank 0 have been duplicated, which is not suppose to happen. Is there a way to capture the ghost point information when uninterpolating the DM? Thanks, -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Sat Apr 11 19:03:25 2015 From: zonexo at gmail.com (TAY wee-beng) Date: Sun, 12 Apr 2015 08:03:25 +0800 Subject: [petsc-users] Recommended examples for using KSP and multigrid with DM In-Reply-To: <5F8DEB12-D4B5-4FA6-8F18-B0B667071CE9@mcs.anl.gov> References: <5525EF91.3020709@gmail.com> <5F8DEB12-D4B5-4FA6-8F18-B0B667071CE9@mcs.anl.gov> Message-ID: <5529B64D.8080304@gmail.com> Hi, I'm using DMDA. The LHS of the poisson eqn is fixed, only the RHS changes. I thought I can't find ex29.c in the current ver of PETSc, so I thought it's no longer valid. Do I always have to get the jacobian as in ex29 to solve the Poisson eqn? What about a good ex. using KSPBCGS to solve my momentum eqn with DMDA? Thank you. Yours sincerely, TAY wee-beng On 4/11/2015 6:43 AM, Barry Smith wrote: > Are you using DMDA or DMPLEX or some other DM? Does the Poisson equation change (variable coefficient) or is it a constant coefficient with the same values for all solves? ex29.c could still possibly serve as a model for the Poisson subproblem, why do you say it is no longer valid? > > Barry > > >> On Apr 8, 2015, at 10:18 PM, TAY wee-beng wrote: >> >> Hi, >> >> I was following example ex29 in the past to write the code to solve my Poisson eqn using DM. Now it seems that ex29 is no longer valid. Which example should I follow now? >> >> Also, I am using KSPBCGS to solve my momentum eqn in the past. Do I need to change much if I'm now using DM? If so, what is a good example? >> >> -- >> Thank you >> >> Yours sincerely, >> >> TAY wee-beng >> From knepley at gmail.com Sun Apr 12 04:38:52 2015 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 12 Apr 2015 04:38:52 -0500 Subject: [petsc-users] Recommended examples for using KSP and multigrid with DM In-Reply-To: <5529B64D.8080304@gmail.com> References: <5525EF91.3020709@gmail.com> <5F8DEB12-D4B5-4FA6-8F18-B0B667071CE9@mcs.anl.gov> <5529B64D.8080304@gmail.com> Message-ID: On Sat, Apr 11, 2015 at 7:03 PM, TAY wee-beng wrote: > Hi, > > I'm using DMDA. The LHS of the poisson eqn is fixed, only the RHS changes. > > I thought I can't find ex29.c in the current ver of PETSc, so I thought > it's no longer valid. Do I always have to get the jacobian as in ex29 to > solve the Poisson eqn? > It is there: https://bitbucket.org/petsc/petsc/src/0dbc604a2409e3ecaa129deb24062cf2be4d41a7/src/ksp/ksp/examples/tutorials/ex29.c?at=master > What about a good ex. using KSPBCGS to solve my momentum eqn with DMDA? > All examples are independent of the precise Krylov method. You can use that anywhere: cd src/snes/examples/tutorials make ex5 ./ex5 -ksp_type bcgs Thanks, Matt > Thank you. > > Yours sincerely, > > TAY wee-beng > > On 4/11/2015 6:43 AM, Barry Smith wrote: > >> Are you using DMDA or DMPLEX or some other DM? Does the Poisson >> equation change (variable coefficient) or is it a constant coefficient with >> the same values for all solves? ex29.c could still possibly serve as a >> model for the Poisson subproblem, why do you say it is no longer valid? >> >> Barry >> >> >> On Apr 8, 2015, at 10:18 PM, TAY wee-beng wrote: >>> >>> Hi, >>> >>> I was following example ex29 in the past to write the code to solve my >>> Poisson eqn using DM. Now it seems that ex29 is no longer valid. Which >>> example should I follow now? >>> >>> Also, I am using KSPBCGS to solve my momentum eqn in the past. Do I need >>> to change much if I'm now using DM? If so, what is a good example? >>> >>> -- >>> Thank you >>> >>> Yours sincerely, >>> >>> TAY wee-beng >>> >>> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Apr 12 12:26:58 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 12 Apr 2015 12:26:58 -0500 Subject: [petsc-users] For users of PETSc repository only Message-ID: <5936E396-EFAE-4D1A-A5FD-1E922DC3D230@mcs.anl.gov> If you use PETSc 3.5.3 or earlier releases you can ignore this message. It only affects those using the pre-release PETSc repository code. I have changed some of the directory structure of PETSc which will your makefiles and Fortran source code. The locations of the PETSc include makefiles has changed from lib/petsc-conf/ to lib/petsc/conf This means items such as include ${PETSC_DIR}/lib/petsc-conf/variables include ${PETSC_DIR}/lib/petsc-conf/rules need to be changed in your makefiles to include ${PETSC_DIR}/lib/petsc/conf/variables include ${PETSC_DIR}/lib/petsc/conf/rules The locations of the PETSc fortran include files has changed from petsc-finclude to petsc/finclude This means items such as #include #include in your Fortran source needs to be changed to #include #include The reason for the change is to match Linux packaging standards for locations of files and directories. I hope this will be the last change for this for a long while, thanks for your patience Barry From gideon.simpson at gmail.com Sun Apr 12 12:48:25 2015 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Sun, 12 Apr 2015 13:48:25 -0400 Subject: [petsc-users] pedagogical example Message-ID: I was hoping to demonstrate in my class the computational gain with petsc/mpi in solving a simple problem, like discretized poisson or heat, as the number of processes increases. Can anyone recommend any of the petsc examples for this purpose? Perhaps I?m just using poorly chosen KSP/PC pairs, but I haven?t been able to observe gain. I?m planning to demo this on a commodity intel cluster with infiniband. -gideon -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Apr 12 13:47:35 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 12 Apr 2015 13:47:35 -0500 Subject: [petsc-users] pedagogical example In-Reply-To: References: Message-ID: <88EBE1E5-E9C7-4F60-984F-4A50BFD58710@mcs.anl.gov> > On Apr 12, 2015, at 12:48 PM, Gideon Simpson wrote: > > I was hoping to demonstrate in my class the computational gain with petsc/mpi in solving a simple problem, like discretized poisson or heat, as the number of processes increases. Can anyone recommend any of the petsc examples for this purpose? Perhaps I?m just using poorly chosen KSP/PC pairs, but I haven?t been able to observe gain. I?m planning to demo this on a commodity intel cluster with infiniband. Gideon, I would use src/ksp/ksp/examples/tutorials/ex45 to get across three concepts 1) algorithmic complexity (1 process). Run it with several levels of refinement (say -da_refine 4 depending on how much memory you have) with a) -pc_type jacobi -ksp_type bcgs (algorithm with poor computational complexity, very parallel) b) -pc_type mg -ksp_type bcgs (algorithm with good computational complexity, good parallel but less than jacobi) then run it again with one more level of refinement (say -da_refine 5) and see how much more time each method takes 2) scaling (2 process) Run as with 1) but on two processes and note that the "poorer" algorithm Jacobi gives better "speedup" then mg 3) understanding the limitations of your machine (see http://www.mcs.anl.gov/petsc/documentation/faq.html#computers) how total memory bandwidth of all your available cores determines the performance of the PETSc solvers. So run the streams benchmark (included now with PETSc in the src/benchmarks/streams directory) to see its speedup when you use a different number of cores and combinations of cores on each node and across nodes and then run the PETSc example to see its speedups. Note that you likely have to do something smarter than mpiexec -n n ./ex45 ... to launch the program since you need to have control over what nodes the mpi puts each of the processes it starts up; for example does it spread the processes one on each node or first pack them on one node (check the documentation for your mpiexec and how to control this). You will find that different choices lead to very different performance and this can be related to the streams benchmark and available memory bandwidth. Barry > > -gideon > From shah.preyas at gmail.com Sun Apr 12 18:57:10 2015 From: shah.preyas at gmail.com (Preyas Shah) Date: Sun, 12 Apr 2015 16:57:10 -0700 Subject: [petsc-users] Simple question regarding dense matrix algebra using elemental Message-ID: Hi, I have been recently investigating the use of Petsc for solving a PDE related to my research and web search suggests that I should use petsc with elemental. So far, I was required to solve a matrix equation Ax=b where A was dense (with number of non zeros =0) but the size of the matrix was order 5000x5000. I employed the standard serial LU solver from Gnu Scientific Library and obtained a decent runtime that served my needs. Now I am investigating the same problem in a particularly singular limit of one parameters in my PDE. As a result, to obtain grid convergence on the physical domain, I am forced to go to sizes of A beyond 30000x30000. I am trying to find a good library that can solve such dense systems in **parallel**. Petsc says its capable of doing dense linear algebra but my web search hasn't shown me any examples where dense equations are solved in **parallel**. A webpage showing a minimum working example would be enough. Or any other advice :) Thanks for your time! ?~Preyas -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Sun Apr 12 21:47:15 2015 From: hzhang at mcs.anl.gov (Hong) Date: Sun, 12 Apr 2015 21:47:15 -0500 Subject: [petsc-users] Simple question regarding dense matrix algebra using elemental In-Reply-To: References: Message-ID: You can either use elemental directly, or use petsc-elemental interface. An example can be found at ~petsc/src/mat/examples/tests/ex145.c You may use petsc KSP interface instead. I just modified ~petsc/src/ksp/ksp/examples/tutorials/ex2.c so this example can be run with elemental with runtime options mpiexec -n 3 ./ex2 -pc_type lu -pc_factor_mat_solver_package elemental -mat_type elemental Norm of error 2.81086e-15 iterations 1 Please using petsc-dev (master branch) for petsc-elemental interface. Hong On Sun, Apr 12, 2015 at 6:57 PM, Preyas Shah wrote: > Hi, > > I have been recently investigating the use of Petsc for solving a PDE > related to my research and web search suggests that I should use petsc with > elemental. > > So far, I was required to solve a matrix equation Ax=b where A was dense > (with number of non zeros =0) but the size of the matrix was order > 5000x5000. I employed the standard serial LU solver from Gnu Scientific > Library and obtained a decent runtime that served my needs. > > Now I am investigating the same problem in a particularly singular limit > of one parameters in my PDE. As a result, to obtain grid convergence on the > physical domain, I am forced to go to sizes of A beyond 30000x30000. I am > trying to find a good library that can solve such dense systems in > **parallel**. Petsc says its capable of doing dense linear algebra but my > web search hasn't shown me any examples where dense equations are solved in > **parallel**. A webpage showing a minimum working example would be enough. > Or any other advice :) > > Thanks for your time! > ?~Preyas > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From salazardetroya at gmail.com Mon Apr 13 09:20:28 2015 From: salazardetroya at gmail.com (Miguel Angel Salazar de Troya) Date: Mon, 13 Apr 2015 09:20:28 -0500 Subject: [petsc-users] FSAL property in TS Message-ID: Hi I realized that even though there's a boolean named FSAL (First Same As Last) inside the TS object, it seems this property is not implemented yet. I just wanted to double check this is like this and also ask if it will be implemented soon. Thanks Miguel -- *Miguel Angel Salazar de Troya* Graduate Research Assistant Department of Mechanical Science and Engineering University of Illinois at Urbana-Champaign (217) 550-2360 salaza11 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Apr 13 09:29:29 2015 From: jed at jedbrown.org (Jed Brown) Date: Mon, 13 Apr 2015 08:29:29 -0600 Subject: [petsc-users] FSAL property in TS In-Reply-To: References: Message-ID: <87iod0aypi.fsf@jedbrown.org> Miguel Angel Salazar de Troya writes: > Hi > > I realized that even though there's a boolean named FSAL (First Same As > Last) inside the TS object, It's not in TS, but is in some implementations. > it seems this property is not implemented yet. What do you mean "implemented"? Some of the methods test their schemes for this property. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From salazardetroya at gmail.com Mon Apr 13 09:33:29 2015 From: salazardetroya at gmail.com (Miguel Angel Salazar de Troya) Date: Mon, 13 Apr 2015 09:33:29 -0500 Subject: [petsc-users] FSAL property in TS In-Reply-To: <87iod0aypi.fsf@jedbrown.org> References: <87iod0aypi.fsf@jedbrown.org> Message-ID: Sorry I should have been more specific. I was talking about the explicit Runge-Kutta method. Thanks Miguel On Mon, Apr 13, 2015 at 9:29 AM, Jed Brown wrote: > Miguel Angel Salazar de Troya writes: > > > Hi > > > > I realized that even though there's a boolean named FSAL (First Same As > > Last) inside the TS object, > > It's not in TS, but is in some implementations. > > > it seems this property is not implemented yet. > > What do you mean "implemented"? Some of the methods test their schemes > for this property. > -- *Miguel Angel Salazar de Troya* Graduate Research Assistant Department of Mechanical Science and Engineering University of Illinois at Urbana-Champaign (217) 550-2360 salaza11 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Apr 13 09:51:49 2015 From: jed at jedbrown.org (Jed Brown) Date: Mon, 13 Apr 2015 08:51:49 -0600 Subject: [petsc-users] FSAL property in TS In-Reply-To: References: <87iod0aypi.fsf@jedbrown.org> Message-ID: <87d238axoa.fsf@jedbrown.org> Miguel Angel Salazar de Troya writes: > Sorry I should have been more specific. I was talking about the explicit > Runge-Kutta method. It is currently only diagnostic. I think it applies to RK3BS and RK5DP and I suppose by "implemented", you mean "has the method been optimized to take advantage of the property?". The answer to that question is not yet, but it's subtle in case the user changes the state or function as part of a pre/post stage/step function. We have a similar problem with explicit stages in diagonally implicit RK methods for DAE. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From shah.preyas at gmail.com Mon Apr 13 12:20:17 2015 From: shah.preyas at gmail.com (Preyas Shah) Date: Mon, 13 Apr 2015 10:20:17 -0700 Subject: [petsc-users] Simple question regarding dense matrix algebra using elemental In-Reply-To: References: Message-ID: Another beginner's querry: ?I worked out an example that "combines" mat/examples/tests/ex145.c /ksp/ksp/examples/tutorials/ex2.c /ksp/ksp/examples/tutorials/ex30.c Please see attached code sample. The code compiles but does not solve the system correctly. I would like to know what changes I would need to ?implement to get it working right. Thanks! P.S. Feel free to include the (corrected) code sample in the test/tutorial folders of the development version. On Sun, Apr 12, 2015 at 7:47 PM, Hong wrote: > You can either use elemental directly, or use petsc-elemental interface. > An example can be found at > ~petsc/src/mat/examples/tests/ex145.c > > You may use petsc KSP interface instead. I just modified > ~petsc/src/ksp/ksp/examples/tutorials/ex2.c > so this example can be run with elemental with runtime options > mpiexec -n 3 ./ex2 -pc_type lu -pc_factor_mat_solver_package elemental > -mat_type elemental > Norm of error 2.81086e-15 iterations 1 > > Please using petsc-dev (master branch) for petsc-elemental interface. > > Hong > > On Sun, Apr 12, 2015 at 6:57 PM, Preyas Shah > wrote: > >> Hi, >> >> I have been recently investigating the use of Petsc for solving a PDE >> related to my research and web search suggests that I should use petsc with >> elemental. >> >> So far, I was required to solve a matrix equation Ax=b where A was dense >> (with number of non zeros =0) but the size of the matrix was order >> 5000x5000. I employed the standard serial LU solver from Gnu Scientific >> Library and obtained a decent runtime that served my needs. >> >> Now I am investigating the same problem in a particularly singular limit >> of one parameters in my PDE. As a result, to obtain grid convergence on the >> physical domain, I am forced to go to sizes of A beyond 30000x30000. I am >> trying to find a good library that can solve such dense systems in >> **parallel**. Petsc says its capable of doing dense linear algebra but my >> web search hasn't shown me any examples where dense equations are solved in >> **parallel**. A webpage showing a minimum working example would be enough. >> Or any other advice :) >> >> Thanks for your time! >> ?~Preyas >> >> > -- ~ Preyas Doctoral Researcher, MechE, Stanford Univ. *"Can a man still be brave if he's afraid?".**"That is the only time he can be brave."-- George R R Martin* *"To learn who rules over you, simply find out who you are not allowed to criticize." -- Voltaire* *"First they ignore you, then they laugh at you, then they fight you, then you win" -- Gandhi* -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Ksp.cpp Type: text/x-c++src Size: 5348 bytes Desc: not available URL: From knepley at gmail.com Mon Apr 13 12:26:02 2015 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 13 Apr 2015 12:26:02 -0500 Subject: [petsc-users] Simple question regarding dense matrix algebra using elemental In-Reply-To: References: Message-ID: On Mon, Apr 13, 2015 at 12:20 PM, Preyas Shah wrote: > Another beginner's querry: > > ?I worked out an example that "combines" > mat/examples/tests/ex145.c > /ksp/ksp/examples/tutorials/ex2.c > /ksp/ksp/examples/tutorials/ex30.c > > Please see attached code sample. The code compiles but does not solve the > system correctly. I would like to know what changes I would need to > ?implement to get it working right. Thanks! > Check the residual as well (rhs - MM sol). This should be small. Its possible you have not set the exact solution correctly, or the discretization error is large, or the condition number is large. Thanks, Matt > P.S. Feel free to include the (corrected) code sample in the > test/tutorial folders of the development version. > > On Sun, Apr 12, 2015 at 7:47 PM, Hong wrote: > >> You can either use elemental directly, or use petsc-elemental interface. >> An example can be found at >> ~petsc/src/mat/examples/tests/ex145.c >> >> You may use petsc KSP interface instead. I just modified >> ~petsc/src/ksp/ksp/examples/tutorials/ex2.c >> so this example can be run with elemental with runtime options >> mpiexec -n 3 ./ex2 -pc_type lu -pc_factor_mat_solver_package elemental >> -mat_type elemental >> Norm of error 2.81086e-15 iterations 1 >> >> Please using petsc-dev (master branch) for petsc-elemental interface. >> >> Hong >> >> On Sun, Apr 12, 2015 at 6:57 PM, Preyas Shah >> wrote: >> >>> Hi, >>> >>> I have been recently investigating the use of Petsc for solving a PDE >>> related to my research and web search suggests that I should use petsc with >>> elemental. >>> >>> So far, I was required to solve a matrix equation Ax=b where A was dense >>> (with number of non zeros =0) but the size of the matrix was order >>> 5000x5000. I employed the standard serial LU solver from Gnu Scientific >>> Library and obtained a decent runtime that served my needs. >>> >>> Now I am investigating the same problem in a particularly singular limit >>> of one parameters in my PDE. As a result, to obtain grid convergence on the >>> physical domain, I am forced to go to sizes of A beyond 30000x30000. I am >>> trying to find a good library that can solve such dense systems in >>> **parallel**. Petsc says its capable of doing dense linear algebra but my >>> web search hasn't shown me any examples where dense equations are solved in >>> **parallel**. A webpage showing a minimum working example would be enough. >>> Or any other advice :) >>> >>> Thanks for your time! >>> ?~Preyas >>> >>> >> > > > -- > ~ > Preyas > > Doctoral Researcher, MechE, Stanford Univ. > > *"Can a man still be brave if he's afraid?".**"That is the only time he > can be brave."-- George R R Martin* > *"To learn who rules over you, simply find out who you are not allowed to > criticize." -- Voltaire* > *"First they ignore you, then they laugh at you, then they fight you, then > you win" -- Gandhi* > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Apr 13 12:31:07 2015 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 13 Apr 2015 12:31:07 -0500 Subject: [petsc-users] Uninterpolating a distributed mesh In-Reply-To: References: Message-ID: On Sat, Apr 11, 2015 at 6:55 PM, Justin Chang wrote: > Hello, > > When I call DMPlexUninterpolate(...) on a distributed mesh (say 2 > processors), it seems to overwrite the "ghost" points (i.e., the points not > locally owned by the processor) and treats all points as if they are local > to the processor. > Yes, I wrote Uninterpolate() just for testing, and do not currently handle the SF. I put it on my TODO list. Its not hard if you want to try. You just filter out any points that are not cells and vertices from the SF, so PetscSFGetGraph() for (leaves) if leaf not a cell or vertex, skip PetscSFSetGraph() Thanks, Matt > Say I have this 2D cell-vertex mesh: > > 14----15-----16 > | \ 5 | \ 7 | > | \ | \ | > | 4 \ | 6 \ | > 11----12-----13 > | \ 1 | \ 3 | > | \ | \ | > | 0 \ | 2 \ | > 8------9------10 > > Which results in the following DM: > > DM Object: 2 MPI processes > type: plex > DM_0x84000004_0 in 2 dimensions: > 0-cells: 9 0 > 2-cells: 8 0 > Labels: > marker: 1 strata of sizes (8) > depth: 2 strata of sizes (9, 8) > > I proceed by interpolating this DM: > > DM Object: 2 MPI processes > type: plex > DM_0x84000004_1 in 2 dimensions: > 0-cells: 9 0 > 1-cells: 16 0 > 2-cells: 8 0 > Labels: > marker: 1 strata of sizes (16) > depth: 3 strata of sizes (9, 16, 8) > > Then distributing across 2 processors: > > DM Object:Parallel Mesh 2 MPI processes > type: plex > Parallel Mesh in 2 dimensions: > 0-cells: 6 6 > 1-cells: 9 9 > 2-cells: 4 4 > Labels: > marker: 1 strata of sizes (9) > depth: 3 strata of sizes (6, 9, 4) > > I have the option of uniformly refining the mesh here but I choose not to > for now. If my dofs are vertex based, then the global size of my DM vector > is 9 and the local sizes for ranks 0 and 1 are 3 and 6 respectively. > However, if I choose to uninterpolate the mesh by calling > DMPlexUninterpolate(...), I get this: > > DM Object: 2 MPI processes > type: plex > DM_0x84000004_2 in 2 dimensions: > 0-cells: 6 6 > 2-cells: 4 4 > Labels: > marker: 1 strata of sizes (5) > depth: 2 strata of sizes (6, 4) > > And the global size of my DM vector becomes 12 and the local size for both > ranks is 6. It looks like the ghost points in rank 0 have been duplicated, > which is not suppose to happen. > > Is there a way to capture the ghost point information when uninterpolating > the DM? > > Thanks, > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Mon Apr 13 13:42:15 2015 From: hzhang at mcs.anl.gov (Hong) Date: Mon, 13 Apr 2015 13:42:15 -0500 Subject: [petsc-users] Simple question regarding dense matrix algebra using elemental In-Reply-To: References: Message-ID: Preyas, I tested your code (renamed as src/ksp/ksp/examples/tests/ex46.cpp) and got mpiexec -n 1 ./ex46 ... 1 1 1 1 1 Norm of error 3.58644e-10 iterations 1 mpiexec -n 3 ./ex46 ... 1 1 1 1 1 Norm of error 5.81285e-11 iterations 1 option '-ksp_view' gives KSP Object: 3 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-16, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: 3 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 0, needed 0 Factored matrix follows: Mat Object: 3 MPI processes type: elemental rows=1000, cols=1000 package used to perform factorization: elemental total: nonzeros=334000, allocated nonzeros=334000 total number of mallocs used during MatSetValues calls =0 Elemental run parameters: allocated entries=334000 grid height=1, grid width=3 linear system matrix = precond matrix: ... Everything looks correct. Hong On Mon, Apr 13, 2015 at 12:26 PM, Matthew Knepley wrote: > On Mon, Apr 13, 2015 at 12:20 PM, Preyas Shah > wrote: > >> Another beginner's querry: >> >> ?I worked out an example that "combines" >> mat/examples/tests/ex145.c >> /ksp/ksp/examples/tutorials/ex2.c >> /ksp/ksp/examples/tutorials/ex30.c >> >> Please see attached code sample. The code compiles but does not solve the >> system correctly. I would like to know what changes I would need to >> ?implement to get it working right. Thanks! >> > > Check the residual as well (rhs - MM sol). This should be small. Its > possible you have not set > the exact solution correctly, or the discretization error is large, or the > condition number is large. > > Thanks, > > Matt > > >> P.S. Feel free to include the (corrected) code sample in the >> test/tutorial folders of the development version. >> >> On Sun, Apr 12, 2015 at 7:47 PM, Hong wrote: >> >>> You can either use elemental directly, or use petsc-elemental interface. >>> An example can be found at >>> ~petsc/src/mat/examples/tests/ex145.c >>> >>> You may use petsc KSP interface instead. I just modified >>> ~petsc/src/ksp/ksp/examples/tutorials/ex2.c >>> so this example can be run with elemental with runtime options >>> mpiexec -n 3 ./ex2 -pc_type lu -pc_factor_mat_solver_package elemental >>> -mat_type elemental >>> Norm of error 2.81086e-15 iterations 1 >>> >>> Please using petsc-dev (master branch) for petsc-elemental interface. >>> >>> Hong >>> >>> On Sun, Apr 12, 2015 at 6:57 PM, Preyas Shah >>> wrote: >>> >>>> Hi, >>>> >>>> I have been recently investigating the use of Petsc for solving a PDE >>>> related to my research and web search suggests that I should use petsc with >>>> elemental. >>>> >>>> So far, I was required to solve a matrix equation Ax=b where A was >>>> dense (with number of non zeros =0) but the size of the matrix was order >>>> 5000x5000. I employed the standard serial LU solver from Gnu Scientific >>>> Library and obtained a decent runtime that served my needs. >>>> >>>> Now I am investigating the same problem in a particularly singular >>>> limit of one parameters in my PDE. As a result, to obtain grid convergence >>>> on the physical domain, I am forced to go to sizes of A beyond 30000x30000. >>>> I am trying to find a good library that can solve such dense systems in >>>> **parallel**. Petsc says its capable of doing dense linear algebra but my >>>> web search hasn't shown me any examples where dense equations are solved in >>>> **parallel**. A webpage showing a minimum working example would be enough. >>>> Or any other advice :) >>>> >>>> Thanks for your time! >>>> ?~Preyas >>>> >>>> >>> >> >> >> -- >> ~ >> Preyas >> >> Doctoral Researcher, MechE, Stanford Univ. >> >> *"Can a man still be brave if he's afraid?".**"That is the only time he >> can be brave."-- George R R Martin* >> *"To learn who rules over you, simply find out who you are not allowed to >> criticize." -- Voltaire* >> *"First they ignore you, then they laugh at you, then they fight you, >> then you win" -- Gandhi* >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jchang27 at uh.edu Mon Apr 13 16:22:41 2015 From: jchang27 at uh.edu (Justin Chang) Date: Mon, 13 Apr 2015 16:22:41 -0500 Subject: [petsc-users] Uninterpolating a distributed mesh In-Reply-To: References: Message-ID: Is there an example somewhere that does something similar to this? Thanks, On Mon, Apr 13, 2015 at 12:31 PM, Matthew Knepley wrote: > On Sat, Apr 11, 2015 at 6:55 PM, Justin Chang wrote: > >> Hello, >> >> When I call DMPlexUninterpolate(...) on a distributed mesh (say 2 >> processors), it seems to overwrite the "ghost" points (i.e., the points not >> locally owned by the processor) and treats all points as if they are local >> to the processor. >> > > Yes, I wrote Uninterpolate() just for testing, and do not currently handle > the SF. I put it on my TODO list. > > Its not hard if you want to try. You just filter out any points that are > not cells and vertices from the SF, so > > PetscSFGetGraph() > for (leaves) > if leaf not a cell or vertex, skip > PetscSFSetGraph() > > Thanks, > > Matt > > >> Say I have this 2D cell-vertex mesh: >> >> 14----15-----16 >> | \ 5 | \ 7 | >> | \ | \ | >> | 4 \ | 6 \ | >> 11----12-----13 >> | \ 1 | \ 3 | >> | \ | \ | >> | 0 \ | 2 \ | >> 8------9------10 >> >> Which results in the following DM: >> >> DM Object: 2 MPI processes >> type: plex >> DM_0x84000004_0 in 2 dimensions: >> 0-cells: 9 0 >> 2-cells: 8 0 >> Labels: >> marker: 1 strata of sizes (8) >> depth: 2 strata of sizes (9, 8) >> >> I proceed by interpolating this DM: >> >> DM Object: 2 MPI processes >> type: plex >> DM_0x84000004_1 in 2 dimensions: >> 0-cells: 9 0 >> 1-cells: 16 0 >> 2-cells: 8 0 >> Labels: >> marker: 1 strata of sizes (16) >> depth: 3 strata of sizes (9, 16, 8) >> >> Then distributing across 2 processors: >> >> DM Object:Parallel Mesh 2 MPI processes >> type: plex >> Parallel Mesh in 2 dimensions: >> 0-cells: 6 6 >> 1-cells: 9 9 >> 2-cells: 4 4 >> Labels: >> marker: 1 strata of sizes (9) >> depth: 3 strata of sizes (6, 9, 4) >> >> I have the option of uniformly refining the mesh here but I choose not to >> for now. If my dofs are vertex based, then the global size of my DM vector >> is 9 and the local sizes for ranks 0 and 1 are 3 and 6 respectively. >> However, if I choose to uninterpolate the mesh by calling >> DMPlexUninterpolate(...), I get this: >> >> DM Object: 2 MPI processes >> type: plex >> DM_0x84000004_2 in 2 dimensions: >> 0-cells: 6 6 >> 2-cells: 4 4 >> Labels: >> marker: 1 strata of sizes (5) >> depth: 2 strata of sizes (6, 4) >> >> And the global size of my DM vector becomes 12 and the local size for >> both ranks is 6. It looks like the ghost points in rank 0 have been >> duplicated, which is not suppose to happen. >> >> Is there a way to capture the ghost point information when >> uninterpolating the DM? >> >> Thanks, >> >> -- >> Justin Chang >> PhD Candidate, Civil Engineering - Computational Sciences >> University of Houston, Department of Civil and Environmental Engineering >> Houston, TX 77004 >> (512) 963-3262 >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Apr 13 16:28:59 2015 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 13 Apr 2015 16:28:59 -0500 Subject: [petsc-users] Uninterpolating a distributed mesh In-Reply-To: References: Message-ID: On Mon, Apr 13, 2015 at 4:22 PM, Justin Chang wrote: > Is there an example somewhere that does something similar to this? > There is DMPlexShiftSF_Internal() in plexsubmesh.c Matt > Thanks, > > On Mon, Apr 13, 2015 at 12:31 PM, Matthew Knepley > wrote: > >> On Sat, Apr 11, 2015 at 6:55 PM, Justin Chang wrote: >> >>> Hello, >>> >>> When I call DMPlexUninterpolate(...) on a distributed mesh (say 2 >>> processors), it seems to overwrite the "ghost" points (i.e., the points not >>> locally owned by the processor) and treats all points as if they are local >>> to the processor. >>> >> >> Yes, I wrote Uninterpolate() just for testing, and do not currently >> handle the SF. I put it on my TODO list. >> >> Its not hard if you want to try. You just filter out any points that are >> not cells and vertices from the SF, so >> >> PetscSFGetGraph() >> for (leaves) >> if leaf not a cell or vertex, skip >> PetscSFSetGraph() >> >> Thanks, >> >> Matt >> >> >>> Say I have this 2D cell-vertex mesh: >>> >>> 14----15-----16 >>> | \ 5 | \ 7 | >>> | \ | \ | >>> | 4 \ | 6 \ | >>> 11----12-----13 >>> | \ 1 | \ 3 | >>> | \ | \ | >>> | 0 \ | 2 \ | >>> 8------9------10 >>> >>> Which results in the following DM: >>> >>> DM Object: 2 MPI processes >>> type: plex >>> DM_0x84000004_0 in 2 dimensions: >>> 0-cells: 9 0 >>> 2-cells: 8 0 >>> Labels: >>> marker: 1 strata of sizes (8) >>> depth: 2 strata of sizes (9, 8) >>> >>> I proceed by interpolating this DM: >>> >>> DM Object: 2 MPI processes >>> type: plex >>> DM_0x84000004_1 in 2 dimensions: >>> 0-cells: 9 0 >>> 1-cells: 16 0 >>> 2-cells: 8 0 >>> Labels: >>> marker: 1 strata of sizes (16) >>> depth: 3 strata of sizes (9, 16, 8) >>> >>> Then distributing across 2 processors: >>> >>> DM Object:Parallel Mesh 2 MPI processes >>> type: plex >>> Parallel Mesh in 2 dimensions: >>> 0-cells: 6 6 >>> 1-cells: 9 9 >>> 2-cells: 4 4 >>> Labels: >>> marker: 1 strata of sizes (9) >>> depth: 3 strata of sizes (6, 9, 4) >>> >>> I have the option of uniformly refining the mesh here but I choose not >>> to for now. If my dofs are vertex based, then the global size of my DM >>> vector is 9 and the local sizes for ranks 0 and 1 are 3 and 6 respectively. >>> However, if I choose to uninterpolate the mesh by calling >>> DMPlexUninterpolate(...), I get this: >>> >>> DM Object: 2 MPI processes >>> type: plex >>> DM_0x84000004_2 in 2 dimensions: >>> 0-cells: 6 6 >>> 2-cells: 4 4 >>> Labels: >>> marker: 1 strata of sizes (5) >>> depth: 2 strata of sizes (6, 4) >>> >>> And the global size of my DM vector becomes 12 and the local size for >>> both ranks is 6. It looks like the ghost points in rank 0 have been >>> duplicated, which is not suppose to happen. >>> >>> Is there a way to capture the ghost point information when >>> uninterpolating the DM? >>> >>> Thanks, >>> >>> -- >>> Justin Chang >>> PhD Candidate, Civil Engineering - Computational Sciences >>> University of Houston, Department of Civil and Environmental Engineering >>> Houston, TX 77004 >>> (512) 963-3262 >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From makhijad at colorado.edu Tue Apr 14 01:06:34 2015 From: makhijad at colorado.edu (Dave Makhija) Date: Tue, 14 Apr 2015 02:06:34 -0400 Subject: [petsc-users] Time integrated adjoints Message-ID: Hello, I'm evaluating my options for computing time dependent adjoints. I did not think PETSc supported this, but I see "TSAdjointSolve" in the development branch. That would be fantastic news if you plan to support time integrated adjoints! What features are envisioned and when is the targeted release date? Thanks! Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailinglists at xgm.de Tue Apr 14 04:11:47 2015 From: mailinglists at xgm.de (Florian Lindner) Date: Tue, 14 Apr 2015 11:11:47 +0200 Subject: [petsc-users] Distribution of columns on mutliple ranks Message-ID: <1523007.1fESZRiaYj@asaru> Hello, given I have this piece of python code: rank = MPI.COMM_WORLD.Get_rank() sizes = [ 4, 5] n = sizes[rank] A = PETSc.Mat() A.create() A.setSizes( ((n, PETSc.DETERMINE), (n, PETSc.DETERMINE)) ) # sets the local size, let petsc determine global size A.setFromOptions(); A.setUp() print("Global Size = ", A.getSize()) print("Local Size = ", A.getLocalSize()) and I run it with mpirun -n 2 and -mat_type dense (this, I think, should not be relevant), getSize() returns 9x9 for both ranks of couse. getLocalSize returns 4x4 and 5x5. I understand that the A is distributed like that: /4x4 ? \ \ ? 5x5 / When I would give ( (PETSC_DETERMINE, 9), (PETSC_DETERMINE, 9) ) to setSizes it would look like that: / 4 x 9 \ \ 5 x 9 / because PETSc uses row based partitioning. What happens when I set local rows and local cols, where do the parts that I marked with ? live? Thanks, ;-) Florian From nancy_polar at outlook.com Tue Apr 14 04:25:49 2015 From: nancy_polar at outlook.com (Nancy Polar) Date: Tue, 14 Apr 2015 19:25:49 +1000 Subject: [petsc-users] Finance Your Project Message-ID: <99746006.20150414192549@outlook.com> Att: Sir/Madam, We are private investors specializing in Joint Venture Partnership/Equity contributions/loans for project of various types to help individuals and organization reach their financial objectives. We customize structured financial solutions for a select group of clientele and investors. Our commitment to client services and innovative approach to financial problem solving. If you are interested in our services, kindly provide us with the following information: * Name of Applicant: .......... * Company name: ............... * Address/Location: ........... * Tel: ........................ * Amount Requested: ...........(If JV/Loan) * Finance Duration: ........... * Project name: ............... Regards, Nancy Polar Secretary to Adam Golding (Private Investor) ******************************************************** This email is intended for the addressee(s) only, and may contain confidential or privileged information. Unauthorised use of this information, or distribution, copying or other dissemination is strictly prohibited. If you have received this email in error, please delete it immediately. ******************************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Apr 14 07:30:11 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Apr 2015 07:30:11 -0500 Subject: [petsc-users] Time integrated adjoints In-Reply-To: References: Message-ID: <05299A3A-C45C-4F0A-8792-CEF561D34AEB@mcs.anl.gov> > On Apr 14, 2015, at 1:06 AM, Dave Makhija wrote: > > Hello, > > I'm evaluating my options for computing time dependent adjoints. I did not think PETSc supported this, but I see "TSAdjointSolve" in the development branch. That would be fantastic news if you plan to support time integrated adjoints! > > What features are envisioned and when is the targeted release date? Dave, We currently have discrete adjoint computations for explicit RK methods and for implicit theta methods (backward Euler and Crank-Nicholson). The development copy of the users manual has some discussion (http://www.mcs.anl.gov/petsc/petsc-master/docs/manual.pdf page 135) and there are several examples. I urge you to try them out now in master and stay in contact with us to insure we provide what you need as we are actively developing them. We plan a release for early June. Barry > > Thanks! > > Dave From knepley at gmail.com Tue Apr 14 08:58:37 2015 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 14 Apr 2015 08:58:37 -0500 Subject: [petsc-users] Distribution of columns on mutliple ranks In-Reply-To: <1523007.1fESZRiaYj@asaru> References: <1523007.1fESZRiaYj@asaru> Message-ID: On Tue, Apr 14, 2015 at 4:11 AM, Florian Lindner wrote: > Hello, > > given I have this piece of python code: > > rank = MPI.COMM_WORLD.Get_rank() > sizes = [ 4, 5] > n = sizes[rank] > > A = PETSc.Mat() > A.create() > A.setSizes( ((n, PETSc.DETERMINE), (n, PETSc.DETERMINE)) ) # sets the > local size, let petsc determine global size > A.setFromOptions(); A.setUp() > > print("Global Size = ", A.getSize()) > print("Local Size = ", A.getLocalSize()) > > > and I run it with mpirun -n 2 and -mat_type dense (this, I think, should > not be relevant), getSize() returns 9x9 for both ranks of couse. > > getLocalSize returns 4x4 and 5x5. > > I understand that the A is distributed like that: > > /4x4 ? \ > \ ? 5x5 / > > > When I would give ( (PETSC_DETERMINE, 9), (PETSC_DETERMINE, 9) ) to > setSizes it would look like that: > > / 4 x 9 \ > \ 5 x 9 / > > because PETSc uses row based partitioning. > > What happens when I set local rows and local cols, where do the parts that > I marked with ? live? > > Thanks, ;-) PETSc always uses row partitioning, no matter what arguments are set for local columns. Thanks, Matt > > Florian > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jason.sarich at gmail.com Tue Apr 14 09:35:28 2015 From: jason.sarich at gmail.com (Jason Sarich) Date: Tue, 14 Apr 2015 09:35:28 -0500 Subject: [petsc-users] Tao iterations In-Reply-To: References: Message-ID: Hi Justin, I have pushed these changes to the "next" branch, your code snippet should work fine there. Note that there is also available (since version 3.5.0) the routine TaoGetSolutionStatus(tao,&its,NULL,NULL,NULL,NULL,NULL) which will provide the same information Jason On Fri, Apr 10, 2015 at 6:28 PM, Justin Chang wrote: > Whatever is convenient and/or follow the "PETSc" standards. Something > similar to SNESGetIterationNumber() or KSPGetIterationNumber() would be > nice. Ideally I want my code to look like this: > > ierr = TaoGetIterationNumber(tao,&its);CHKERRQ(ierr); > ierr = PetscPrintf(PETSC_COMM_WORLD, "Number of Tao iterations = %D\n", > its); > > Thanks :) > > On Fri, Apr 10, 2015 at 5:53 PM, Jason Sarich > wrote: > >> Hi Justin, I'll get this in. I assume that displaying the number of >> iterations with tao_converged_reason is what you are asking for in >> particular? Or did you have something else in mind? >> >> Jason >> On Apr 10, 2015 16:42, "Smith, Barry F." wrote: >> >>> >>> Justin, >>> >>> Sorry TAO simply doesn't even collect this information currently. >>> But yes we should definitely make it available! >>> >>> Jason, >>> >>> Could you please add this; almost all the TaoSolve_xxx() have a >>> local variable iter; change that to tao->niter (I'm guess this is suppose >>> to capture this information) and add a TaoGetIterationNumber() and the uses >>> can access this. Also modify at the end of TaoSolve() -tao_converged_reason >>> to also print the iteration count. At the same time since you add this you >>> can add a tao->totalits which would accumulate all iterations over all the >>> solves for that Tao object and the routine TaoGetTotalIterations() to >>> access this. Note that TaoSolve() would initialize tao->niter = 0 at the >>> top. >>> >>> Thanks >>> >>> Barry >>> >>> >>> >>> > On Apr 10, 2015, at 4:16 PM, Justin Chang wrote: >>> > >>> > Hi all, >>> > >>> > Is there a way to generically obtain the number of Tao iterations? I >>> am looking through the -help options for Tao and I don't see any metric >>> where you can output this quantity in the manner that you could for SNES or >>> KSP solves. I am currently using blmvm and tron, and the only way I can see >>> getting this metric is by outputting -tao_view and/or -tao_monitor and >>> manually finding this number. I find this cumbersome especially for >>> transient problems where I would like to simply have this number printed >>> for each step instead of ending up with unnecessary info. >>> > >>> > Thanks, >>> > >>> > >>> > -- >>> > Justin Chang >>> > PhD Candidate, Civil Engineering - Computational Sciences >>> > University of Houston, Department of Civil and Environmental >>> Engineering >>> > Houston, TX 77004 >>> > (512) 963-3262 >>> >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fande.kong at colorado.edu Tue Apr 14 11:13:28 2015 From: fande.kong at colorado.edu (Fande Kong) Date: Tue, 14 Apr 2015 10:13:28 -0600 Subject: [petsc-users] DMPlexUniformRefine Message-ID: Hi all, I have several lines of code to read a mesh and uniformly refine it. DMPlex can not uniformly refine it. I am using PETSc-3.5.3. The code is very simple like follows: #undef __FUNCT__ #define __FUNCT__ "main" int main(int argc, char **argv) { DM dm,pdm, refinedMesh; /* Problem specification */ AppCtx user; /* user-defined work context */ PetscErrorCode ierr; ierr = PetscInitialize(&argc, &argv, NULL, help);CHKERRQ(ierr); /*ierr = CreateMesh(PETSC_COMM_WORLD, &user, &dm);CHKERRQ(ierr);*/ ierr = DMPlexCreateExodusFromFile(PETSC_COMM_WORLD, "./Tet4.exo", PETSC_FALSE, &dm);CHKERRQ(ierr); ierr = DMPlexDistribute(dm, "metis", 0, NULL, &pdm);CHKERRQ(ierr); ierr = DMDestroy(&dm);CHKERRQ(ierr); ierr = DMPlexSetRefinementUniform(pdm, PETSC_TRUE);CHKERRQ(ierr); ierr = DMRefine(pdm, PETSC_COMM_WORLD, &refinedMesh);CHKERRQ(ierr); ierr = DMDestroy(&pdm);CHKERRQ(ierr); ierr = DMDestroy(&refinedMesh);CHKERRQ(ierr); ierr = PetscFinalize(); return 0; } The mesh file is also attached. Fande, -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Tet4.exo Type: application/octet-stream Size: 7556 bytes Desc: not available URL: From knepley at gmail.com Tue Apr 14 11:27:35 2015 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 14 Apr 2015 11:27:35 -0500 Subject: [petsc-users] DMPlexUniformRefine In-Reply-To: References: Message-ID: On Tue, Apr 14, 2015 at 11:13 AM, Fande Kong wrote: > Hi all, > > I have several lines of code to read a mesh and uniformly refine it. > DMPlex can not uniformly refine it. I am using PETSc-3.5.3. The code is > very simple like follows: > > #undef __FUNCT__ > > #define __FUNCT__ "main" > > int main(int argc, char **argv) > > { > > DM dm,pdm, refinedMesh; /* Problem specification */ > > AppCtx user; /* user-defined work context */ > > PetscErrorCode ierr; > > > ierr = PetscInitialize(&argc, &argv, NULL, help);CHKERRQ(ierr); > > > /*ierr = CreateMesh(PETSC_COMM_WORLD, &user, &dm);CHKERRQ(ierr);*/ > > > ierr = DMPlexCreateExodusFromFile(PETSC_COMM_WORLD, "./Tet4.exo", > PETSC_FALSE, &dm);CHKERRQ(ierr); > ^^^^^^^^^^^^ Use PETSC_TRUE Matt > ierr = DMPlexDistribute(dm, "metis", 0, NULL, &pdm);CHKERRQ(ierr); > > ierr = DMDestroy(&dm);CHKERRQ(ierr); > > ierr = DMPlexSetRefinementUniform(pdm, PETSC_TRUE);CHKERRQ(ierr); > > ierr = DMRefine(pdm, PETSC_COMM_WORLD, &refinedMesh);CHKERRQ(ierr); > > ierr = DMDestroy(&pdm);CHKERRQ(ierr); > > ierr = DMDestroy(&refinedMesh);CHKERRQ(ierr); > > > ierr = PetscFinalize(); > > > return 0; > > } > > > The mesh file is also attached. > > Fande, > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From fande.kong at colorado.edu Tue Apr 14 11:29:08 2015 From: fande.kong at colorado.edu (Fande Kong) Date: Tue, 14 Apr 2015 10:29:08 -0600 Subject: [petsc-users] DMPlexUniformRefine In-Reply-To: References: Message-ID: Thanks. It does work now. On Tue, Apr 14, 2015 at 10:27 AM, Matthew Knepley wrote: > On Tue, Apr 14, 2015 at 11:13 AM, Fande Kong > wrote: > >> Hi all, >> >> I have several lines of code to read a mesh and uniformly refine it. >> DMPlex can not uniformly refine it. I am using PETSc-3.5.3. The code is >> very simple like follows: >> >> #undef __FUNCT__ >> >> #define __FUNCT__ "main" >> >> int main(int argc, char **argv) >> >> { >> >> DM dm,pdm, refinedMesh; /* Problem specification >> */ >> >> AppCtx user; /* user-defined work context */ >> >> PetscErrorCode ierr; >> >> >> ierr = PetscInitialize(&argc, &argv, NULL, help);CHKERRQ(ierr); >> >> >> /*ierr = CreateMesh(PETSC_COMM_WORLD, &user, &dm);CHKERRQ(ierr);*/ >> >> >> ierr = DMPlexCreateExodusFromFile(PETSC_COMM_WORLD, "./Tet4.exo", >> PETSC_FALSE, &dm);CHKERRQ(ierr); >> > > ^^^^^^^^^^^^ Use > PETSC_TRUE > > Matt > > >> ierr = DMPlexDistribute(dm, "metis", 0, NULL, &pdm);CHKERRQ(ierr); >> >> ierr = DMDestroy(&dm);CHKERRQ(ierr); >> >> ierr = DMPlexSetRefinementUniform(pdm, PETSC_TRUE);CHKERRQ(ierr); >> >> ierr = DMRefine(pdm, PETSC_COMM_WORLD, &refinedMesh);CHKERRQ(ierr); >> >> ierr = DMDestroy(&pdm);CHKERRQ(ierr); >> >> ierr = DMDestroy(&refinedMesh);CHKERRQ(ierr); >> >> >> ierr = PetscFinalize(); >> >> >> return 0; >> >> } >> >> >> The mesh file is also attached. >> >> Fande, >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From taem at mail.com Tue Apr 14 11:55:45 2015 From: taem at mail.com (store@apple) Date: Tue, 14 Apr 2015 09:55:45 -0700 Subject: [petsc-users] Apple Customer Team Message-ID: <20150414165543.567571A4148@smtp-a.tudelft.nl> ?This e-mail was sent to your mailbox and contains information directly related to your account with us Update Security notifications. Thanks, Apple Customer Team -------------- next part -------------- An HTML attachment was scrubbed... URL: From gideon.simpson at gmail.com Tue Apr 14 17:52:49 2015 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Tue, 14 Apr 2015 18:52:49 -0400 Subject: [petsc-users] ghost values Message-ID: <367D51C0-BC55-4C38-A86E-A48D3C6CFE87@gmail.com> If i use the DM_BOUNDARY_GHOSTED flag in the creation of a DMDA array, are the ghosted values automatically set to zero, or should they be manually set to zero if that?s the desired ghost value? -gideon -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Apr 14 18:02:47 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Apr 2015 18:02:47 -0500 Subject: [petsc-users] ghost values In-Reply-To: <367D51C0-BC55-4C38-A86E-A48D3C6CFE87@gmail.com> References: <367D51C0-BC55-4C38-A86E-A48D3C6CFE87@gmail.com> Message-ID: <02D6AC92-8709-46E9-9B64-5D562E7543CD@mcs.anl.gov> When PETSc vectors are created initially they always have 0 everywhere. So if you use DMCreateLocalVector() it will have zero in all those ghost places (as well as everywhere else). But if you use DMGetLocalVector() it returns vectors that maybe dirty so you need to fill in any locations you want to have a known value with that known value. Or call VecSet() to clear the entire vector. Barry > On Apr 14, 2015, at 5:52 PM, Gideon Simpson wrote: > > If i use the DM_BOUNDARY_GHOSTED flag in the creation of a DMDA array, are the ghosted values automatically set to zero, or should they be manually set to zero if that?s the desired ghost value? > > -gideon > From gideon.simpson at gmail.com Tue Apr 14 18:15:06 2015 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Tue, 14 Apr 2015 19:15:06 -0400 Subject: [petsc-users] ghost values In-Reply-To: <02D6AC92-8709-46E9-9B64-5D562E7543CD@mcs.anl.gov> References: <367D51C0-BC55-4C38-A86E-A48D3C6CFE87@gmail.com> <02D6AC92-8709-46E9-9B64-5D562E7543CD@mcs.anl.gov> Message-ID: <639BC594-A051-426D-82A4-39F79CE5E9F2@gmail.com> Other than zeroing things out, is there any substantive difference between DMCreateLocalVector and DMGetLocalVector? -gideon > On Apr 14, 2015, at 7:02 PM, Barry Smith wrote: > > > When PETSc vectors are created initially they always have 0 everywhere. So if you use DMCreateLocalVector() it will have zero in all those ghost places (as well as everywhere else). > > But if you use DMGetLocalVector() it returns vectors that maybe dirty so you need to fill in any locations you want to have a known value with that known value. Or call VecSet() to clear the entire vector. > > Barry > >> On Apr 14, 2015, at 5:52 PM, Gideon Simpson wrote: >> >> If i use the DM_BOUNDARY_GHOSTED flag in the creation of a DMDA array, are the ghosted values automatically set to zero, or should they be manually set to zero if that?s the desired ghost value? >> >> -gideon >> > From knepley at gmail.com Tue Apr 14 18:16:50 2015 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 14 Apr 2015 18:16:50 -0500 Subject: [petsc-users] ghost values In-Reply-To: <639BC594-A051-426D-82A4-39F79CE5E9F2@gmail.com> References: <367D51C0-BC55-4C38-A86E-A48D3C6CFE87@gmail.com> <02D6AC92-8709-46E9-9B64-5D562E7543CD@mcs.anl.gov> <639BC594-A051-426D-82A4-39F79CE5E9F2@gmail.com> Message-ID: On Tue, Apr 14, 2015 at 6:15 PM, Gideon Simpson wrote: > Other than zeroing things out, is there any substantive difference between > DMCreateLocalVector and DMGetLocalVector? > The Get version caches vectors, so you are not continually creating/destroying Matt > -gideon > > > On Apr 14, 2015, at 7:02 PM, Barry Smith wrote: > > > > > > When PETSc vectors are created initially they always have 0 everywhere. > So if you use DMCreateLocalVector() it will have zero in all those ghost > places (as well as everywhere else). > > > > But if you use DMGetLocalVector() it returns vectors that maybe dirty > so you need to fill in any locations you want to have a known value with > that known value. Or call VecSet() to clear the entire vector. > > > > Barry > > > >> On Apr 14, 2015, at 5:52 PM, Gideon Simpson > wrote: > >> > >> If i use the DM_BOUNDARY_GHOSTED flag in the creation of a DMDA array, > are the ghosted values automatically set to zero, or should they be > manually set to zero if that?s the desired ghost value? > >> > >> -gideon > >> > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gideon.simpson at gmail.com Tue Apr 14 18:25:46 2015 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Tue, 14 Apr 2015 19:25:46 -0400 Subject: [petsc-users] ghost values In-Reply-To: References: <367D51C0-BC55-4C38-A86E-A48D3C6CFE87@gmail.com> <02D6AC92-8709-46E9-9B64-5D562E7543CD@mcs.anl.gov> <639BC594-A051-426D-82A4-39F79CE5E9F2@gmail.com> Message-ID: <033A2CD4-1536-4435-B42D-A2D1EF5B3695@gmail.com> So other than the minor point of it zeroing out the data, is there any conceivable reason to use Create/Destroy over Get/Restore? -gideon > On Apr 14, 2015, at 7:16 PM, Matthew Knepley wrote: > > On Tue, Apr 14, 2015 at 6:15 PM, Gideon Simpson > wrote: > Other than zeroing things out, is there any substantive difference between DMCreateLocalVector and DMGetLocalVector? > > The Get version caches vectors, so you are not continually creating/destroying > > Matt > > -gideon > > > On Apr 14, 2015, at 7:02 PM, Barry Smith > wrote: > > > > > > When PETSc vectors are created initially they always have 0 everywhere. So if you use DMCreateLocalVector() it will have zero in all those ghost places (as well as everywhere else). > > > > But if you use DMGetLocalVector() it returns vectors that maybe dirty so you need to fill in any locations you want to have a known value with that known value. Or call VecSet() to clear the entire vector. > > > > Barry > > > >> On Apr 14, 2015, at 5:52 PM, Gideon Simpson > wrote: > >> > >> If i use the DM_BOUNDARY_GHOSTED flag in the creation of a DMDA array, are the ghosted values automatically set to zero, or should they be manually set to zero if that?s the desired ghost value? > >> > >> -gideon > >> > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Apr 14 18:26:31 2015 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 14 Apr 2015 18:26:31 -0500 Subject: [petsc-users] ghost values In-Reply-To: <033A2CD4-1536-4435-B42D-A2D1EF5B3695@gmail.com> References: <367D51C0-BC55-4C38-A86E-A48D3C6CFE87@gmail.com> <02D6AC92-8709-46E9-9B64-5D562E7543CD@mcs.anl.gov> <639BC594-A051-426D-82A4-39F79CE5E9F2@gmail.com> <033A2CD4-1536-4435-B42D-A2D1EF5B3695@gmail.com> Message-ID: On Tue, Apr 14, 2015 at 6:25 PM, Gideon Simpson wrote: > So other than the minor point of it zeroing out the data, is there any > conceivable reason to use Create/Destroy over Get/Restore? > Nope, and calling VecSet(v, 0.0) is really cheap. Matt > -gideon > > On Apr 14, 2015, at 7:16 PM, Matthew Knepley wrote: > > On Tue, Apr 14, 2015 at 6:15 PM, Gideon Simpson > wrote: > >> Other than zeroing things out, is there any substantive difference >> between DMCreateLocalVector and DMGetLocalVector? >> > > The Get version caches vectors, so you are not continually > creating/destroying > > Matt > > >> -gideon >> >> > On Apr 14, 2015, at 7:02 PM, Barry Smith wrote: >> > >> > >> > When PETSc vectors are created initially they always have 0 >> everywhere. So if you use DMCreateLocalVector() it will have zero in all >> those ghost places (as well as everywhere else). >> > >> > But if you use DMGetLocalVector() it returns vectors that maybe dirty >> so you need to fill in any locations you want to have a known value with >> that known value. Or call VecSet() to clear the entire vector. >> > >> > Barry >> > >> >> On Apr 14, 2015, at 5:52 PM, Gideon Simpson >> wrote: >> >> >> >> If i use the DM_BOUNDARY_GHOSTED flag in the creation of a DMDA array, >> are the ghosted values automatically set to zero, or should they be >> manually set to zero if that?s the desired ghost value? >> >> >> >> -gideon >> >> >> > >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Apr 14 19:23:51 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Apr 2015 19:23:51 -0500 Subject: [petsc-users] ghost values In-Reply-To: <033A2CD4-1536-4435-B42D-A2D1EF5B3695@gmail.com> References: <367D51C0-BC55-4C38-A86E-A48D3C6CFE87@gmail.com> <02D6AC92-8709-46E9-9B64-5D562E7543CD@mcs.anl.gov> <639BC594-A051-426D-82A4-39F79CE5E9F2@gmail.com> <033A2CD4-1536-4435-B42D-A2D1EF5B3695@gmail.com> Message-ID: <89A816E4-E873-4F74-821F-6ED290DA6258@mcs.anl.gov> > On Apr 14, 2015, at 6:25 PM, Gideon Simpson wrote: > > So other than the minor point of it zeroing out the data, is there any conceivable reason to use Create/Destroy over Get/Restore? The design intends that Create/Destroy are for long lived objects. For example in main you might call Create to make a vector you will store the solution for the entire simulation run in then just before you exit the program you destroy it. The Get/Restore are for work vectors that are used within one routine. The most classic example is in FormFunction where you often have something like DMGetLocalVector() ; DMGlobalToLocalBegin/End(global, local, ...) DMDAVecGetArray(local) ..../ DMRestoreLocalVector(). The idea is that rather creating work vectors in main and passing them around to use when needed, DM manages a "pool" of work vectors that you can grab whenever you need them and are low overhead to access. There is nothing that enforces you follow this style, but if you don't you need to be careful to know which vectors to call VecDestroy() on and which ones to call DMRestoreLocalVector() on plus it will be easier for other people to follow your code. Barry > > -gideon > >> On Apr 14, 2015, at 7:16 PM, Matthew Knepley wrote: >> >> On Tue, Apr 14, 2015 at 6:15 PM, Gideon Simpson wrote: >> Other than zeroing things out, is there any substantive difference between DMCreateLocalVector and DMGetLocalVector? >> >> The Get version caches vectors, so you are not continually creating/destroying >> >> Matt >> >> -gideon >> >> > On Apr 14, 2015, at 7:02 PM, Barry Smith wrote: >> > >> > >> > When PETSc vectors are created initially they always have 0 everywhere. So if you use DMCreateLocalVector() it will have zero in all those ghost places (as well as everywhere else). >> > >> > But if you use DMGetLocalVector() it returns vectors that maybe dirty so you need to fill in any locations you want to have a known value with that known value. Or call VecSet() to clear the entire vector. >> > >> > Barry >> > >> >> On Apr 14, 2015, at 5:52 PM, Gideon Simpson wrote: >> >> >> >> If i use the DM_BOUNDARY_GHOSTED flag in the creation of a DMDA array, are the ghosted values automatically set to zero, or should they be manually set to zero if that?s the desired ghost value? >> >> >> >> -gideon >> >> >> > >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > From gideon.simpson at gmail.com Tue Apr 14 19:43:42 2015 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Tue, 14 Apr 2015 20:43:42 -0400 Subject: [petsc-users] ghost values In-Reply-To: <89A816E4-E873-4F74-821F-6ED290DA6258@mcs.anl.gov> References: <367D51C0-BC55-4C38-A86E-A48D3C6CFE87@gmail.com> <02D6AC92-8709-46E9-9B64-5D562E7543CD@mcs.anl.gov> <639BC594-A051-426D-82A4-39F79CE5E9F2@gmail.com> <033A2CD4-1536-4435-B42D-A2D1EF5B3695@gmail.com> <89A816E4-E873-4F74-821F-6ED290DA6258@mcs.anl.gov> Message-ID: <5A4427EB-8380-479D-8A8B-D945D3AD42BD@gmail.com> So in your example of the form function, is something happening where, when I leave the scope of FormFunction, the invocation of DMGetLocalVector on the DM keeps a note that it had been called previously and has that memory already allocated? Does the design distinguish between different calls to DMGetLocalVector, like if I called it in one subroutine and then in another subroutine, or twice in a single subroutine? -gideon > On Apr 14, 2015, at 8:23 PM, Barry Smith wrote: > > >> On Apr 14, 2015, at 6:25 PM, Gideon Simpson wrote: >> >> So other than the minor point of it zeroing out the data, is there any conceivable reason to use Create/Destroy over Get/Restore? > > The design intends that Create/Destroy are for long lived objects. For example in main you might call Create to make a vector you will store the solution for the entire simulation run in then just before you exit the program you destroy it. > > The Get/Restore are for work vectors that are used within one routine. The most classic example is in FormFunction where you often have something like DMGetLocalVector() ; DMGlobalToLocalBegin/End(global, local, ...) DMDAVecGetArray(local) ..../ DMRestoreLocalVector(). The idea is that rather creating work vectors in main and passing them around to use when needed, DM manages a "pool" of work vectors that you can grab whenever you need them and are low overhead to access. > > There is nothing that enforces you follow this style, but if you don't you need to be careful to know which vectors to call VecDestroy() on and which ones to call DMRestoreLocalVector() on plus it will be easier for other people to follow your code. > > Barry > >> >> -gideon >> >>> On Apr 14, 2015, at 7:16 PM, Matthew Knepley wrote: >>> >>> On Tue, Apr 14, 2015 at 6:15 PM, Gideon Simpson wrote: >>> Other than zeroing things out, is there any substantive difference between DMCreateLocalVector and DMGetLocalVector? >>> >>> The Get version caches vectors, so you are not continually creating/destroying >>> >>> Matt >>> >>> -gideon >>> >>>> On Apr 14, 2015, at 7:02 PM, Barry Smith wrote: >>>> >>>> >>>> When PETSc vectors are created initially they always have 0 everywhere. So if you use DMCreateLocalVector() it will have zero in all those ghost places (as well as everywhere else). >>>> >>>> But if you use DMGetLocalVector() it returns vectors that maybe dirty so you need to fill in any locations you want to have a known value with that known value. Or call VecSet() to clear the entire vector. >>>> >>>> Barry >>>> >>>>> On Apr 14, 2015, at 5:52 PM, Gideon Simpson wrote: >>>>> >>>>> If i use the DM_BOUNDARY_GHOSTED flag in the creation of a DMDA array, are the ghosted values automatically set to zero, or should they be manually set to zero if that?s the desired ghost value? >>>>> >>>>> -gideon >>>>> >>>> >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >> > From bsmith at mcs.anl.gov Tue Apr 14 19:54:56 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Apr 2015 19:54:56 -0500 Subject: [petsc-users] ghost values In-Reply-To: <5A4427EB-8380-479D-8A8B-D945D3AD42BD@gmail.com> References: <367D51C0-BC55-4C38-A86E-A48D3C6CFE87@gmail.com> <02D6AC92-8709-46E9-9B64-5D562E7543CD@mcs.anl.gov> <639BC594-A051-426D-82A4-39F79CE5E9F2@gmail.com> <033A2CD4-1536-4435-B42D-A2D1EF5B3695@gmail.com> <89A816E4-E873-4F74-821F-6ED290DA6258@mcs.anl.gov> <5A4427EB-8380-479D-8A8B-D945D3AD42BD@gmail.com> Message-ID: > On Apr 14, 2015, at 7:43 PM, Gideon Simpson wrote: > > So in your example of the form function, is something happening where, when I leave the scope of FormFunction, the invocation of DMGetLocalVector on the DM keeps a note that it had been called previously and has that memory already allocated? Yes, it just keeps a lists of available vectors and gives you the next one on the list. When you call RestoreLocalVector() it returns the vector to the list. If it has handed out all the vectors on the list to you (and you have not restored them) it creates a new one and gives it to you. > Does the design distinguish between different calls to DMGetLocalVector, like if I called it in one subroutine and then in another subroutine, or twice in a single subroutine? > Not sure what you mean by this but if you call it twice it gives you two different vectors (unless of course you restored the first before requesting the second) It does't matter from what function your request the vector or restore it (but passing around a vector you obtained with GetVector through a complicated set of function calls is generally bad because you will forget to restore it). Hence it is better to Get; Use; Restore Here is the entire routine. PetscErrorCode DMGetLocalVector(DM dm,Vec *g) { PetscErrorCode ierr,i; PetscFunctionBegin; PetscValidHeaderSpecific(dm,DM_CLASSID,1); PetscValidPointer(g,2); for (i=0; ilocalin[i]) { *g = dm->localin[i]; dm->localin[i] = NULL; goto alldone; } } ierr = DMCreateLocalVector(dm,g);CHKERRQ(ierr); alldone: for (i=0; ilocalout[i]) { dm->localout[i] = *g; break; } } PetscFunctionReturn(0); } > -gideon > >> On Apr 14, 2015, at 8:23 PM, Barry Smith wrote: >> >> >>> On Apr 14, 2015, at 6:25 PM, Gideon Simpson wrote: >>> >>> So other than the minor point of it zeroing out the data, is there any conceivable reason to use Create/Destroy over Get/Restore? >> >> The design intends that Create/Destroy are for long lived objects. For example in main you might call Create to make a vector you will store the solution for the entire simulation run in then just before you exit the program you destroy it. >> >> The Get/Restore are for work vectors that are used within one routine. The most classic example is in FormFunction where you often have something like DMGetLocalVector() ; DMGlobalToLocalBegin/End(global, local, ...) DMDAVecGetArray(local) ..../ DMRestoreLocalVector(). The idea is that rather creating work vectors in main and passing them around to use when needed, DM manages a "pool" of work vectors that you can grab whenever you need them and are low overhead to access. >> >> There is nothing that enforces you follow this style, but if you don't you need to be careful to know which vectors to call VecDestroy() on and which ones to call DMRestoreLocalVector() on plus it will be easier for other people to follow your code. >> >> Barry >> >>> >>> -gideon >>> >>>> On Apr 14, 2015, at 7:16 PM, Matthew Knepley wrote: >>>> >>>> On Tue, Apr 14, 2015 at 6:15 PM, Gideon Simpson wrote: >>>> Other than zeroing things out, is there any substantive difference between DMCreateLocalVector and DMGetLocalVector? >>>> >>>> The Get version caches vectors, so you are not continually creating/destroying >>>> >>>> Matt >>>> >>>> -gideon >>>> >>>>> On Apr 14, 2015, at 7:02 PM, Barry Smith wrote: >>>>> >>>>> >>>>> When PETSc vectors are created initially they always have 0 everywhere. So if you use DMCreateLocalVector() it will have zero in all those ghost places (as well as everywhere else). >>>>> >>>>> But if you use DMGetLocalVector() it returns vectors that maybe dirty so you need to fill in any locations you want to have a known value with that known value. Or call VecSet() to clear the entire vector. >>>>> >>>>> Barry >>>>> >>>>>> On Apr 14, 2015, at 5:52 PM, Gideon Simpson wrote: >>>>>> >>>>>> If i use the DM_BOUNDARY_GHOSTED flag in the creation of a DMDA array, are the ghosted values automatically set to zero, or should they be manually set to zero if that?s the desired ghost value? >>>>>> >>>>>> -gideon >>>>>> >>>>> >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>> -- Norbert Wiener >>> >> > From jchang27 at uh.edu Tue Apr 14 21:20:31 2015 From: jchang27 at uh.edu (Justin Chang) Date: Tue, 14 Apr 2015 21:20:31 -0500 Subject: [petsc-users] Caught signal number 10 BUS: Bus Error Message-ID: Hello everyone, My program runs fine, with debug mode, on my Ubuntu laptop, but when I run the same code on my iMac (Yosemite) with no debug, I get this error: [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 10 BUS: Bus Error, possibly illegal memory access [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2687-g3beb275 GIT Date: 2015-04-14 12:56:30 -0500 [0]PETSC ERROR: ./main on a arch-darwin-c-debug named CAML-03.CIVE.UH.EDU by jychang48 Tue Apr 14 21:09:31 2015 [0]PETSC ERROR: Configure options --download-chaco --download-ctetgen --download-exodusii --download-fblaslapack --download-hdf5 --download-metis --download-mumps --download-netcdf --download-openmpi --download-parmetis --download-scalapack --download-triangle --with-cc=gcc --with-cmake=cmake --with-cxx=g++ --with-debugging=0 --with-fc=gfortran --with-valgrind=1 CFLAGS= CXXFLAGS= PETSC_ARCH=arch-darwin-c-debug [0]PETSC ERROR: #1 User provided function() line 0 in unknown file -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 59. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- ------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code.. Per user-direction, the job has been aborted. ------------------------------------------------------- -------------------------------------------------------------------------- mpiexec detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[11164,1],0] Exit code: 59 -------------------------------------------------------------------------- Can someone explain what this error means to me? Basically, I have an FEM code (using DMPlex), with auxiliary field coefficients, and this error occurs when I compute the Jacobian via SNESComputeJacobian(...). This error doesn't happen on my Ubuntu laptop. Thanks, -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Apr 14 21:28:12 2015 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 14 Apr 2015 21:28:12 -0500 Subject: [petsc-users] Caught signal number 10 BUS: Bus Error In-Reply-To: References: Message-ID: On Tue, Apr 14, 2015 at 9:20 PM, Justin Chang wrote: > Hello everyone, > > My program runs fine, with debug mode, on my Ubuntu laptop, but when I run > the same code on my iMac (Yosemite) with no debug, I get this error: > So it appears that this SEGV happens in your code, which likely means your pointwise function. You can confirm by getting a stack trace from gdb or lldb. Thanks, Matt > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 10 BUS: Bus Error, possibly illegal > memory access > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [0]PETSC ERROR: to get more information on the crash. > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2687-g3beb275 GIT > Date: 2015-04-14 12:56:30 -0500 > [0]PETSC ERROR: ./main on a arch-darwin-c-debug named CAML-03.CIVE.UH.EDU > by jychang48 Tue Apr 14 21:09:31 2015 > [0]PETSC ERROR: Configure options --download-chaco --download-ctetgen > --download-exodusii --download-fblaslapack --download-hdf5 --download-metis > --download-mumps --download-netcdf --download-openmpi --download-parmetis > --download-scalapack --download-triangle --with-cc=gcc --with-cmake=cmake > --with-cxx=g++ --with-debugging=0 --with-fc=gfortran --with-valgrind=1 > CFLAGS= CXXFLAGS= PETSC_ARCH=arch-darwin-c-debug > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 59. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > ------------------------------------------------------- > Primary job terminated normally, but 1 process returned > a non-zero exit code.. Per user-direction, the job has been aborted. > ------------------------------------------------------- > -------------------------------------------------------------------------- > mpiexec detected that one or more processes exited with non-zero status, > thus causing > the job to be terminated. The first process to do so was: > > Process name: [[11164,1],0] > Exit code: 59 > -------------------------------------------------------------------------- > > Can someone explain what this error means to me? Basically, I have an FEM > code (using DMPlex), with auxiliary field coefficients, and this error > occurs when I compute the Jacobian via SNESComputeJacobian(...). This error > doesn't happen on my Ubuntu laptop. > > Thanks, > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Apr 14 21:50:06 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 14 Apr 2015 21:50:06 -0500 Subject: [petsc-users] Caught signal number 10 BUS: Bus Error In-Reply-To: References: Message-ID: Also run with valgrind [perhaps on linux] to make sure valgrind doesn't flag any memory corruption. http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Satish On Tue, 14 Apr 2015, Matthew Knepley wrote: > On Tue, Apr 14, 2015 at 9:20 PM, Justin Chang wrote: > > > Hello everyone, > > > > My program runs fine, with debug mode, on my Ubuntu laptop, but when I run > > the same code on my iMac (Yosemite) with no debug, I get this error: > > > > So it appears that this SEGV happens in your code, which likely means your > pointwise function. You can confirm by > getting a stack trace from gdb or lldb. > > Thanks, > > Matt > > > > [0]PETSC ERROR: > > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Caught signal number 10 BUS: Bus Error, possibly illegal > > memory access > > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > [0]PETSC ERROR: or see > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > > X to find memory corruption errors > > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > > run > > [0]PETSC ERROR: to get more information on the crash. > > [0]PETSC ERROR: --------------------- Error Message > > -------------------------------------------------------------- > > [0]PETSC ERROR: Signal received > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > > for trouble shooting. > > [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2687-g3beb275 GIT > > Date: 2015-04-14 12:56:30 -0500 > > [0]PETSC ERROR: ./main on a arch-darwin-c-debug named CAML-03.CIVE.UH.EDU > > by jychang48 Tue Apr 14 21:09:31 2015 > > [0]PETSC ERROR: Configure options --download-chaco --download-ctetgen > > --download-exodusii --download-fblaslapack --download-hdf5 --download-metis > > --download-mumps --download-netcdf --download-openmpi --download-parmetis > > --download-scalapack --download-triangle --with-cc=gcc --with-cmake=cmake > > --with-cxx=g++ --with-debugging=0 --with-fc=gfortran --with-valgrind=1 > > CFLAGS= CXXFLAGS= PETSC_ARCH=arch-darwin-c-debug > > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > > -------------------------------------------------------------------------- > > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > > with errorcode 59. > > > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > > You may or may not see output from other processes, depending on > > exactly when Open MPI kills them. > > -------------------------------------------------------------------------- > > ------------------------------------------------------- > > Primary job terminated normally, but 1 process returned > > a non-zero exit code.. Per user-direction, the job has been aborted. > > ------------------------------------------------------- > > -------------------------------------------------------------------------- > > mpiexec detected that one or more processes exited with non-zero status, > > thus causing > > the job to be terminated. The first process to do so was: > > > > Process name: [[11164,1],0] > > Exit code: 59 > > -------------------------------------------------------------------------- > > > > Can someone explain what this error means to me? Basically, I have an FEM > > code (using DMPlex), with auxiliary field coefficients, and this error > > occurs when I compute the Jacobian via SNESComputeJacobian(...). This error > > doesn't happen on my Ubuntu laptop. > > > > Thanks, > > > > -- > > Justin Chang > > PhD Candidate, Civil Engineering - Computational Sciences > > University of Houston, Department of Civil and Environmental Engineering > > Houston, TX 77004 > > (512) 963-3262 > > > > > > From makhijad at colorado.edu Wed Apr 15 06:24:23 2015 From: makhijad at colorado.edu (Dave Makhija) Date: Wed, 15 Apr 2015 07:24:23 -0400 Subject: [petsc-users] Time integrated adjoints In-Reply-To: <05299A3A-C45C-4F0A-8792-CEF561D34AEB@mcs.anl.gov> References: <05299A3A-C45C-4F0A-8792-CEF561D34AEB@mcs.anl.gov> Message-ID: Barry, Is the method exact if I have a time dependent mass matrix (dF/dU_t changes each time step)? I'm not sure it's possible if TSSetIJacobian expects a function which computes the combination of dF/dU and a*dF/dU_t. But, perhaps there is something I'm missing. Otherwise, it looks very comprehensive. I like how you added the "cost integrand". If I recall correctly, the Trilinos package Rythmos is missing this. Thanks! Dave On Tue, Apr 14, 2015 at 8:30 AM, Barry Smith wrote: > > > On Apr 14, 2015, at 1:06 AM, Dave Makhija wrote: > > > > Hello, > > > > I'm evaluating my options for computing time dependent adjoints. I did > not think PETSc supported this, but I see "TSAdjointSolve" in the > development branch. That would be fantastic news if you plan to support > time integrated adjoints! > > > > What features are envisioned and when is the targeted release date? > > Dave, > > We currently have discrete adjoint computations for explicit RK methods > and for implicit theta methods (backward Euler and Crank-Nicholson). The > development copy of the users manual has some discussion ( > http://www.mcs.anl.gov/petsc/petsc-master/docs/manual.pdf page 135) and > there are several examples. I urge you to try them out now in master and > stay in contact with us to insure we provide what you need as we are > actively developing them. > We plan a release for early June. > > Barry > > > > > Thanks! > > > > Dave > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benoit.nennig at supmeca.fr Wed Apr 15 08:31:56 2015 From: benoit.nennig at supmeca.fr (NENNIG Benoit) Date: Wed, 15 Apr 2015 13:31:56 +0000 Subject: [petsc-users] petsc4py and mumps (or other direct sparse solver) Message-ID: <1429104736942.40745@supmeca.fr> Dear petsc-users, I am a beginner in petsc and I have some question with the python interface. I am trying to solve problem with mumps (or other direct sparse solver) I have written the following piece of code ksp = PETSc.KSP() ksp.create(PETSc.COMM_WORLD) ksp.setOperators(A) ksp.setType('preonly') pc=ksp.getPC() pc.setType('lu') pc.setFactorSolverPackage('mumps') It seems to works, however I have 2 questions 1/ how to set the Icntl parameters of mumps ? I didn't find the function MatMumpsSetIcntl. 2/ Is this piece of code right or written in good petsc style ? thanks a lot, Benoit From hzhang at mcs.anl.gov Wed Apr 15 10:15:17 2015 From: hzhang at mcs.anl.gov (Hong) Date: Wed, 15 Apr 2015 10:15:17 -0500 Subject: [petsc-users] petsc4py and mumps (or other direct sparse solver) In-Reply-To: <1429104736942.40745@supmeca.fr> References: <1429104736942.40745@supmeca.fr> Message-ID: Benoit : mumps options can be called from PETSc C-code as ierr = PetscOptionsInsertString("-mat_mumps_icntl_13 1 -mat_mumps_icntl_24 1 -mat_mumps_cntl_3 1e-12");CHKERRQ(ierr); or see petsc/src/ksp/ksp/examples/tutorials/ex52.c I do not know how to translate these calls into python API though. Hong Dear petsc-users, > > I am a beginner in petsc and I have some question with the python > interface. I am trying to solve problem with mumps (or other direct sparse > solver) > > I have written the following piece of code > > ksp = PETSc.KSP() > ksp.create(PETSc.COMM_WORLD) > ksp.setOperators(A) > ksp.setType('preonly') > pc=ksp.getPC() > pc.setType('lu') > pc.setFactorSolverPackage('mumps') > > It seems to works, however I have 2 questions > 1/ how to set the Icntl parameters of mumps ? I didn't find the function > MatMumpsSetIcntl. > 2/ Is this piece of code right or written in good petsc style ? > > thanks a lot, > > Benoit > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Wed Apr 15 11:38:11 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 15 Apr 2015 19:38:11 +0300 Subject: [petsc-users] petsc4py and mumps (or other direct sparse solver) In-Reply-To: References: <1429104736942.40745@supmeca.fr> Message-ID: On 15 April 2015 at 18:15, Hong wrote: > Benoit : > > mumps options can be called from PETSc C-code as > ierr = PetscOptionsInsertString("-mat_mumps_icntl_13 1 -mat_mumps_icntl_24 1 > -mat_mumps_cntl_3 1e-12");CHKERRQ(ierr); > > or see petsc/src/ksp/ksp/examples/tutorials/ex52.c > > I do not know how to translate these calls into python API though. > opts = PETSc.Options() # dict-like opts["mat_mumps_icntl_13"] = 1 opts["mat_mumps_icntl_24"] = 1 opts["mat_mumps_cntl_3"] = 1e-12 Or also: opts = PETSc.Options() opts.prefixPush("mat_mumps_") opts["icntl_13"] = 1 opts["icntl_24"] = 1 opts["cntl_3"] = 1e-12 opts.prefixPop() Or also: opts = PETSc.Options("mat_mumps_") opts["icntl_13"] = 1 opts["icntl_24"] = 1 opts["cntl_3"] = 1e-12 -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From bsmith at mcs.anl.gov Wed Apr 15 12:37:51 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 15 Apr 2015 12:37:51 -0500 Subject: [petsc-users] Time integrated adjoints In-Reply-To: References: <05299A3A-C45C-4F0A-8792-CEF561D34AEB@mcs.anl.gov> Message-ID: <69367D49-7876-4507-84D9-9E9DD03ECC0B@mcs.anl.gov> Hong and Emil, You will need to answer this. > On Apr 15, 2015, at 6:24 AM, Dave Makhija wrote: > > Barry, > > Is the method exact if I have a time dependent mass matrix (dF/dU_t changes each time step)? I'm not sure it's possible if TSSetIJacobian expects a function which computes the combination of dF/dU and a*dF/dU_t. But, perhaps there is something I'm missing. > > Otherwise, it looks very comprehensive. I like how you added the "cost integrand". If I recall correctly, the Trilinos package Rythmos is missing this. > > Thanks! > > Dave > > On Tue, Apr 14, 2015 at 8:30 AM, Barry Smith wrote: > > > On Apr 14, 2015, at 1:06 AM, Dave Makhija wrote: > > > > Hello, > > > > I'm evaluating my options for computing time dependent adjoints. I did not think PETSc supported this, but I see "TSAdjointSolve" in the development branch. That would be fantastic news if you plan to support time integrated adjoints! > > > > What features are envisioned and when is the targeted release date? > > Dave, > > We currently have discrete adjoint computations for explicit RK methods and for implicit theta methods (backward Euler and Crank-Nicholson). The development copy of the users manual has some discussion (http://www.mcs.anl.gov/petsc/petsc-master/docs/manual.pdf page 135) and there are several examples. I urge you to try them out now in master and stay in contact with us to insure we provide what you need as we are actively developing them. > We plan a release for early June. > > Barry > > > > > Thanks! > > > > Dave > > From emconsta at mcs.anl.gov Wed Apr 15 14:18:30 2015 From: emconsta at mcs.anl.gov (Emil Constantinescu) Date: Wed, 15 Apr 2015 14:18:30 -0500 Subject: [petsc-users] Time integrated adjoints In-Reply-To: <69367D49-7876-4507-84D9-9E9DD03ECC0B@mcs.anl.gov> References: <05299A3A-C45C-4F0A-8792-CEF561D34AEB@mcs.anl.gov> <69367D49-7876-4507-84D9-9E9DD03ECC0B@mcs.anl.gov> Message-ID: <552EB986.4040409@mcs.anl.gov> On 4/15/15 12:37 PM, Barry Smith wrote: > > Hong and Emil, > > You will need to answer this. > > >> On Apr 15, 2015, at 6:24 AM, Dave Makhija wrote: >> >> Barry, >> >> Is the method exact if I have a time dependent mass matrix (dF/dU_t changes each time step)? I'm not sure it's possible if TSSetIJacobian expects a function which computes the combination of dF/dU and a*dF/dU_t. But, perhaps there is something I'm missing. In the adjoint derivation, we assumed that we have F = M U_t - f(t,U); so a*dF/dU_t is a*M, which holds if M=M(t). So in principle it should work. However, because M(t_n) and M(t_{n+1}) are different, the adjoint implementation might be slightly inconsistent -- and I'm saying this out of caution. We did not test it on problems with M=M(t). Do you have such an example interfaced with PETSc we can try? Emil >> Otherwise, it looks very comprehensive. I like how you added the "cost integrand". If I recall correctly, the Trilinos package Rythmos is missing this. >> >> Thanks! >> >> Dave >> >> On Tue, Apr 14, 2015 at 8:30 AM, Barry Smith wrote: >> >>> On Apr 14, 2015, at 1:06 AM, Dave Makhija wrote: >>> >>> Hello, >>> >>> I'm evaluating my options for computing time dependent adjoints. I did not think PETSc supported this, but I see "TSAdjointSolve" in the development branch. That would be fantastic news if you plan to support time integrated adjoints! >>> >>> What features are envisioned and when is the targeted release date? >> >> Dave, >> >> We currently have discrete adjoint computations for explicit RK methods and for implicit theta methods (backward Euler and Crank-Nicholson). The development copy of the users manual has some discussion (http://www.mcs.anl.gov/petsc/petsc-master/docs/manual.pdf page 135) and there are several examples. I urge you to try them out now in master and stay in contact with us to insure we provide what you need as we are actively developing them. >> We plan a release for early June. >> >> Barry >> >>> >>> Thanks! >>> >>> Dave >> >> > From emconsta at mcs.anl.gov Wed Apr 15 20:51:13 2015 From: emconsta at mcs.anl.gov (Emil Constantinescu) Date: Wed, 15 Apr 2015 20:51:13 -0500 Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this In-Reply-To: References: <1426907935637.9e8be15c@Nodemailer> <1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov> <550D8101.6070006@mcs.anl.gov> <5510D6E6.2060700@mcs.anl.gov> Message-ID: <552F1591.7010004@mcs.anl.gov> On 3/24/15 5:31 AM, Lisandro Dalcin wrote: > Emil, is there any chance you can try my code with your problem? I > really need some feedback to push this to PETSc, otherwise Hi Lisandro - we checked ts_alpha_adapt and we tested it on a small system (mildly stiff van der Pol ODE). I enclosed a Figure generated by Debo that compares the error at the final time against ATOL - there alpha is the original one (with original adaptor). For this problem your adaptor works better [ error should be close to ATOL]. So from my perspective, your adaptor works well. However, we need to keep the old too so that we have a one step adaptor available ... since alpha is a one step method. Emil -------------- next part -------------- A non-text attachment was scrubbed... Name: Figure2.pdf Type: application/binary Size: 9630 bytes Desc: not available URL: From bsmith at mcs.anl.gov Wed Apr 15 20:53:29 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 15 Apr 2015 20:53:29 -0500 Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this In-Reply-To: <552F1591.7010004@mcs.anl.gov> References: <1426907935637.9e8be15c@Nodemailer> <1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov> <550D8101.6070006@mcs.anl.gov> <5510D6E6.2060700@mcs.anl.gov> <552F1591.7010004@mcs.anl.gov> Message-ID: <56377752-1D62-4A3B-B64C-08F86832DDED@mcs.anl.gov> The more adaptors the better > On Apr 15, 2015, at 8:51 PM, Emil Constantinescu wrote: > > > On 3/24/15 5:31 AM, Lisandro Dalcin wrote: >> Emil, is there any chance you can try my code with your problem? I >> really need some feedback to push this to PETSc, otherwise > > Hi Lisandro - we checked ts_alpha_adapt and we tested it on a small system (mildly stiff van der Pol ODE). I enclosed a Figure generated by Debo that compares the error at the final time against ATOL - there alpha is the original one (with original adaptor). For this problem your adaptor works better [ error should be close to ATOL]. So from my perspective, your adaptor works well. However, we need to keep the old too so that we have a one step adaptor available ... since alpha is a one step method. > > Emil > From dalcinl at gmail.com Thu Apr 16 02:13:09 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 16 Apr 2015 10:13:09 +0300 Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this In-Reply-To: <552F1591.7010004@mcs.anl.gov> References: <1426907935637.9e8be15c@Nodemailer> <1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov> <550D8101.6070006@mcs.anl.gov> <5510D6E6.2060700@mcs.anl.gov> <552F1591.7010004@mcs.anl.gov> Message-ID: On 16 April 2015 at 04:51, Emil Constantinescu wrote: > > On 3/24/15 5:31 AM, Lisandro Dalcin wrote: >> >> Emil, is there any chance you can try my code with your problem? I >> really need some feedback to push this to PETSc, otherwise > > > Hi Lisandro - we checked ts_alpha_adapt and we tested it on a small system > (mildly stiff van der Pol ODE). I enclosed a Figure generated by Debo that > compares the error at the final time against ATOL - there alpha is the > original one (with original adaptor). Original adaptor? Do you mean TSAlphaSetAdapt(ts,TSAlphaAdaptDefault,NULL)? In such a case, no surprises, TSAlphaAdaptDefault() is quite naive, it does not estimate the LTE, it is actually based in the change of the solution. Can you send/point me your code to take a look? > For this problem your adaptor works > better [ error should be close to ATOL]. > So from my perspective, your > adaptor works well. However, we need to keep the old too so that we have a > one step adaptor available ... since alpha is a one step method. > Sorry, I'm not following you. Generalized alpha is not truly a one-step method, because it retains Xdot from the previous time step. In the initial step, other codes out there assume that Xdot_prev=0. My "alpha1" implementation has a trick for the initial time step (using the same ideas, i.e. a BD formula) to both estimate Xdot_prev and the error. However, this is at the cost of performing two nonlinear solves in the initial step (IOW, two internal timesteps with backward Euler and dt/2). Please note this trick should be valid for any second-order method, at least to estimate the initial LTE in the first step. BTW, Have you compared my "-ts_type tsalpha1 -ts_alpha_adapt" with PETSc's "-ts_type cn -ts_theta adapt"? -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From bsmith at mcs.anl.gov Thu Apr 16 07:28:50 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 16 Apr 2015 07:28:50 -0500 Subject: [petsc-users] Matlab socket connection In-Reply-To: References: <38437ECC-F9DE-4577-B462-9D149942EC0A@mcs.anl.gov> Message-ID: <9C8BD777-E42B-4F1D-93A4-DC2806C6D38F@mcs.anl.gov> > On Apr 16, 2015, at 2:47 AM, siddhesh godbole wrote: > > Hello Barry > > sorry for the same doubt again but i have tried everything i could to send data across matlab from petsc via socket. matlab paths to ${PETSC_DIR}/share/petsc/matlab and ${PETSC_DIR}/${PETSC_ARCH}/lib/matlab are added properly, petsc was configured with --with matlab option, and it has successfully generated mex files corresponding to sopen etc too. i have a file named sopen.mexa64 in ${PETSC_ARCH}/lib/matlab. > > everytime the petsc program is lauched from matlab with Launch('./program',np,opt) it gives an error in the line p = PetscOpenSocket; in which the error is specifically in S = struct('fd', sopen()); Please always cut and paste the entire error message. Does src/ksp/ksp/examples/tutorials/ex41.m work? You can attempt to debug by manually opening Matlab and entering launch('./ex41 ',1,' '); p = PetscOpenSocket; etc. > i have made sure that previous sopen belonging to some other package as u said does not exist in the path. > can you tell me why i would be getting this error? > > extract from my code in c > PetscViewerSocketOpen(PETSC_COMM_WORLD,"server",PETSC_NULL,&fd); > ierr = MatCreate(PETSC_COMM_WORLD,&mmtop);CHKERRQ(ierr); > ierr = MatLoad(mmtop,fd);CHKERRQ(ierr); > > ierr = MatCreate(PETSC_COMM_WORLD,&kktop);CHKERRQ(ierr); > ierr = MatLoad(kktop,fd);CHKERRQ(ierr); > > ierr = MatView(Phi,fd);CHKERRQ(ierr); > > > extract from my matlab code interfacing with petsc > > mmtop=zeros(15,15); > kktop=zeros(15,15); > for i=1:15; > for j=1:15; > mmtop(i,j)=(i+j)+(i+1)*(j+2); > kktop(i,j)=(i+j+5)+(i+2)*(j+2); > end > end > if (nargin < 1) > np = 1; > end > if (nargin < 2) > opt = ' '; > end > launch('./eps13D ',np,opt); > > fd=PetscOpenSocket(); > > > PetscBinaryWrite(fd,mmtop); > PetscBinaryWrite(fd,kktop); > phi = PetscBinaryRead(fd); > > close(socket); > > > thank you > > > > > > > Siddhesh M Godbole > > 5th year Dual Degree, > Civil Eng & Applied Mech. > IIT Madras > > On Fri, Apr 10, 2015 at 10:39 PM, siddhesh godbole wrote: > Oh...I found that! Sorry for the silly doubt! > > Thank you so much Barry! > > On Apr 10, 2015 6:23 PM, "Barry Smith" wrote: > > Ahh, bad name spacing. PETSc has an sopen mex file for opening socket connections, some completely different other package has an sopen.m which is being found first. Make sure the PETSc sopen mex file is in the MATLABPATH before the other one or better yet make sure the other sopen.m is not in the MATLABPATH > > Barry > > > On Apr 10, 2015, at 6:46 AM, siddhesh godbole wrote: > > > > Hello, > > > > I am having difficulty in launching petsc program from matlab. According to the error the problem is with the sopen function. error given in the matlab command window is that there are not enough arguments for sopen in the line 75. i am attaching sopen.m file with mail. can you help me out with this as there is no one to solve my doubt in this area. i am also attaching my program file eps13D.m > > the sopen call is made by the function PetscOpenSocket in which line 10 subsequently also shoots an error. > > > > Thanks a lot > > > > > > Siddhesh M Godbole > > > > 5th year Dual Degree, > > Civil Eng & Applied Mech. > > IIT Madras > > > > From emconsta at mcs.anl.gov Thu Apr 16 08:44:54 2015 From: emconsta at mcs.anl.gov (Emil Constantinescu) Date: Thu, 16 Apr 2015 08:44:54 -0500 Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this In-Reply-To: References: <1426907935637.9e8be15c@Nodemailer> <1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov> <550D8101.6070006@mcs.anl.gov> <5510D6E6.2060700@mcs.anl.gov> <552F1591.7010004@mcs.anl.gov> Message-ID: <552FBCD6.4040702@mcs.anl.gov> On 4/16/15 2:13 AM, Lisandro Dalcin wrote: > On 16 April 2015 at 04:51, Emil Constantinescu wrote: >> >> On 3/24/15 5:31 AM, Lisandro Dalcin wrote: >>> >>> Emil, is there any chance you can try my code with your problem? I >>> really need some feedback to push this to PETSc, otherwise >> >> >> Hi Lisandro - we checked ts_alpha_adapt and we tested it on a small system >> (mildly stiff van der Pol ODE). I enclosed a Figure generated by Debo that >> compares the error at the final time against ATOL - there alpha is the >> original one (with original adaptor). > > Original adaptor? Do you mean TSAlphaSetAdapt(ts,TSAlphaAdaptDefault,NULL)? > > In such a case, no surprises, TSAlphaAdaptDefault() is quite naive, it > does not estimate the LTE, it is actually based in the change of the > solution. Yups, that's the one; but that still needs to be available unless a better one-step one is implemented. Note however that even estimating exactly LTE is not a guarantee that the error will be within ATOL. Emil > Can you send/point me your code to take a look? > >> For this problem your adaptor works >> better [ error should be close to ATOL]. > >> So from my perspective, your >> adaptor works well. However, we need to keep the old too so that we have a >> one step adaptor available ... since alpha is a one step method. >> > > Sorry, I'm not following you. Generalized alpha is not truly a > one-step method, because it retains Xdot from the previous time step. > In the initial step, other codes out there assume that Xdot_prev=0. > > My "alpha1" implementation has a trick for the initial time step > (using the same ideas, i.e. a BD formula) to both estimate Xdot_prev > and the error. However, this is at the cost of performing two > nonlinear solves in the initial step (IOW, two internal timesteps with > backward Euler and dt/2). Please note this trick should be valid for > any second-order method, at least to estimate the initial LTE in the > first step. > > BTW, Have you compared my "-ts_type tsalpha1 -ts_alpha_adapt" with > PETSc's "-ts_type cn -ts_theta adapt"? > From dalcinl at gmail.com Thu Apr 16 09:25:22 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 16 Apr 2015 17:25:22 +0300 Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this In-Reply-To: <552FBCD6.4040702@mcs.anl.gov> References: <1426907935637.9e8be15c@Nodemailer> <1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov> <550D8101.6070006@mcs.anl.gov> <5510D6E6.2060700@mcs.anl.gov> <552F1591.7010004@mcs.anl.gov> <552FBCD6.4040702@mcs.anl.gov> Message-ID: On 16 April 2015 at 16:44, Emil Constantinescu wrote: > On 4/16/15 2:13 AM, Lisandro Dalcin wrote: >> >> On 16 April 2015 at 04:51, Emil Constantinescu >> wrote: >>> >>> >>> On 3/24/15 5:31 AM, Lisandro Dalcin wrote: >>>> >>>> >>>> Emil, is there any chance you can try my code with your problem? I >>>> really need some feedback to push this to PETSc, otherwise >>> >>> >>> >>> Hi Lisandro - we checked ts_alpha_adapt and we tested it on a small >>> system >>> (mildly stiff van der Pol ODE). I enclosed a Figure generated by Debo >>> that >>> compares the error at the final time against ATOL - there alpha is the >>> original one (with original adaptor). >> >> >> Original adaptor? Do you mean >> TSAlphaSetAdapt(ts,TSAlphaAdaptDefault,NULL)? >> >> In such a case, no surprises, TSAlphaAdaptDefault() is quite naive, it >> does not estimate the LTE, it is actually based in the change of the >> solution. > > > Yups, that's the one; I was the author of TSAlphaAdaptDefault (the whole TSALPHA, in fact). I coded that beast "out of desperation" long time ago. This thing is not even implemented withing the current TSAdapt framework!. I think at least we should re-implement this beast within TSAdapt (I mean, similar as in -ts_theta_adapt). > but that still needs to be available unless a better > one-step one is implemented. I think you have that feeling simply because TSAlphaAdaptDefault() seems to work, but if you look carefully at the code, it does not make sense from the point of view of a LTE-based theory. OK, Then let's define a poor's man one-step adapt that at least have some sense. How would you implement adaptivity for backward Euler or the midpoint rule? Something based exclusively in some l2/inf norm of Xdot? IOW, we define LTE = X^{n+1} - X^n of O(\delta_t) and use the usual formula with exponent order=1 ? > Note however that even estimating exactly LTE > is not a guarantee that the error will be within ATOL. > Well, ATOL is to keep the LTE under control, of course there are no guarantees about the global error at the end step. That's what you meant? -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From emconsta at mcs.anl.gov Thu Apr 16 09:53:49 2015 From: emconsta at mcs.anl.gov (Emil Constantinescu) Date: Thu, 16 Apr 2015 09:53:49 -0500 Subject: [petsc-users] TimeStepper norm problems. EMIL Please read this In-Reply-To: References: <1426907935637.9e8be15c@Nodemailer> <1B630BCE-DDE5-40EE-9DD7-3EA68137215C@mcs.anl.gov> <550D8101.6070006@mcs.anl.gov> <5510D6E6.2060700@mcs.anl.gov> <552F1591.7010004@mcs.anl.gov> <552FBCD6.4040702@mcs.anl.gov> Message-ID: <552FCCFD.9000703@mcs.anl.gov> On 4/16/15 9:25 AM, Lisandro Dalcin wrote: > On 16 April 2015 at 16:44, Emil Constantinescu wrote: >> On 4/16/15 2:13 AM, Lisandro Dalcin wrote: >>> >>> On 16 April 2015 at 04:51, Emil Constantinescu >>> wrote: >>>> >>>> >>>> On 3/24/15 5:31 AM, Lisandro Dalcin wrote: >>>>> >>>>> >>>>> Emil, is there any chance you can try my code with your problem? I >>>>> really need some feedback to push this to PETSc, otherwise >>>> >>>> >>>> >>>> Hi Lisandro - we checked ts_alpha_adapt and we tested it on a small >>>> system >>>> (mildly stiff van der Pol ODE). I enclosed a Figure generated by Debo >>>> that >>>> compares the error at the final time against ATOL - there alpha is the >>>> original one (with original adaptor). >>> >>> >>> Original adaptor? Do you mean >>> TSAlphaSetAdapt(ts,TSAlphaAdaptDefault,NULL)? >>> >>> In such a case, no surprises, TSAlphaAdaptDefault() is quite naive, it >>> does not estimate the LTE, it is actually based in the change of the >>> solution. >> >> >> Yups, that's the one; > > I was the author of TSAlphaAdaptDefault (the whole TSALPHA, in fact). > I coded that beast "out of desperation" long time ago. This thing is > not even implemented withing the current TSAdapt framework!. I think > at least we should re-implement this beast within TSAdapt (I mean, > similar as in -ts_theta_adapt). I agree. The adaptor is not using TSAdapt and at some point it should be refactored to be consistent. I could not find a good reference detailing the generalized alpha method ... it's always described in cryptic way or in embedded in a given context. But I know it's second order, so we should be able to construct a first order embedded approximation (there appear to be significant degrees of freedom). >> but that still needs to be available unless a better >> one-step one is implemented. > > I think you have that feeling simply because TSAlphaAdaptDefault() > seems to work, but if you look carefully at the code, it does not make > sense from the point of view of a LTE-based theory. > > OK, Then let's define a poor's man one-step adapt that at least have > some sense. How would you implement adaptivity for backward Euler or > the midpoint rule? Something based exclusively in some l2/inf norm of > Xdot? IOW, we define LTE = X^{n+1} - X^n of O(\delta_t) and use the > usual formula with exponent order=1 ? I agree, we cannot do anything that's one step and computationally effective for BE of implicit midpoint; but I believe we can do something relevant for alpha. >> Note however that even estimating exactly LTE >> is not a guarantee that the error will be within ATOL. >> > > Well, ATOL is to keep the LTE under control, of course there are no > guarantees about the global error at the end step. That's what you > meant? Yes, I'm saying to emphasize that we cannot expect perfect results in that figure. > From song.gao2 at mail.mcgill.ca Thu Apr 16 13:35:02 2015 From: song.gao2 at mail.mcgill.ca (Song Gao) Date: Thu, 16 Apr 2015 14:35:02 -0400 Subject: [petsc-users] How to use PETSc in debug? Message-ID: Dear PETSc users, I'm debugging my code but cannot see those line informations in gdb I compiled PETSc with --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --with-debugging=yes I modified my code from KSP ex1f.F and use that makefile. Then compiled code make PETSC_DIR=/home/cfd/sgao/bin/petsc-3.5.3 PETSC_ARCH=FEM_project_DEBUG ex1f The log is attached, I see the -g flag there. But when I run code with gdb, I didn't see line information there. $gdb ./ex1f (gdb) b main Breakpoint 1 at 0x4080e0: file ../../../gcc-4.3.0/libgfortran/fmain.c, line 11. (gdb) r Starting program: /home/cfd/sgao/FEM_in_CFD/code/ex1f [Thread debugging using libthread_db enabled] Breakpoint 1, main (argc=1, argv=0x7fffffffcc58) at ../../../gcc-4.3.0/libgfortran/fmain.c:11 11 ../../../gcc-4.3.0/libgfortran/fmain.c: No such file or directory. in ../../../gcc-4.3.0/libgfortran/fmain.c (gdb) l 6 in ../../../gcc-4.3.0/libgfortran/fmain.c (gdb) What did I miss here? Thank you in advance. Song -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: make.log Type: text/x-log Size: 1909 bytes Desc: not available URL: From Sanjay.Kharche at manchester.ac.uk Thu Apr 16 13:50:45 2015 From: Sanjay.Kharche at manchester.ac.uk (Sanjay Kharche) Date: Thu, 16 Apr 2015 18:50:45 +0000 Subject: [petsc-users] Paraview 3.x Message-ID: Hi I have 3D output in vts format produced using PetscViewerVTKopen and VecView. I cannot volume render the data using paraview 3.x. The latest version of paraview (4.2.x) can visualise it. How can I make 3D vts output that is compatible with paraview 3.x? thanks Sanjay From knepley at gmail.com Thu Apr 16 13:55:42 2015 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 16 Apr 2015 13:55:42 -0500 Subject: [petsc-users] How to use PETSc in debug? In-Reply-To: References: Message-ID: On Thu, Apr 16, 2015 at 1:35 PM, Song Gao wrote: > Dear PETSc users, > > I'm debugging my code but cannot see those line informations in gdb > > I compiled PETSc with > --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack > --download-mpich --with-debugging=yes > > I modified my code from KSP ex1f.F and use that makefile. > > Then compiled code > make PETSC_DIR=/home/cfd/sgao/bin/petsc-3.5.3 PETSC_ARCH=FEM_project_DEBUG > ex1f > The log is attached, I see the -g flag there. But when I run code with > gdb, I didn't see line information there. > > > $gdb ./ex1f > (gdb) b main > Breakpoint 1 at 0x4080e0: file ../../../gcc-4.3.0/libgfortran/fmain.c, > line 11. > main() is not the Fortran entry point. You might be better off working in C. However, if you really want Fortran, try breaking at a line number in your file. Also make sure you did not delete the *.o file Matt > (gdb) r > Starting program: /home/cfd/sgao/FEM_in_CFD/code/ex1f > [Thread debugging using libthread_db enabled] > > Breakpoint 1, main (argc=1, argv=0x7fffffffcc58) at > ../../../gcc-4.3.0/libgfortran/fmain.c:11 > 11 ../../../gcc-4.3.0/libgfortran/fmain.c: No such file or directory. > in ../../../gcc-4.3.0/libgfortran/fmain.c > (gdb) l > 6 in ../../../gcc-4.3.0/libgfortran/fmain.c > (gdb) > > > What did I miss here? Thank you in advance. > > Song > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From song.gao2 at mail.mcgill.ca Thu Apr 16 14:00:14 2015 From: song.gao2 at mail.mcgill.ca (Song Gao) Date: Thu, 16 Apr 2015 15:00:14 -0400 Subject: [petsc-users] How to use PETSc in debug? In-Reply-To: References: Message-ID: Thanks! It solves this problem 2015-04-16 14:55 GMT-04:00 Matthew Knepley : > On Thu, Apr 16, 2015 at 1:35 PM, Song Gao > wrote: > >> Dear PETSc users, >> >> I'm debugging my code but cannot see those line informations in gdb >> >> I compiled PETSc with >> --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack >> --download-mpich --with-debugging=yes >> >> I modified my code from KSP ex1f.F and use that makefile. >> >> Then compiled code >> make PETSC_DIR=/home/cfd/sgao/bin/petsc-3.5.3 >> PETSC_ARCH=FEM_project_DEBUG ex1f >> The log is attached, I see the -g flag there. But when I run code with >> gdb, I didn't see line information there. >> >> >> $gdb ./ex1f >> (gdb) b main >> Breakpoint 1 at 0x4080e0: file ../../../gcc-4.3.0/libgfortran/fmain.c, >> line 11. >> > > main() is not the Fortran entry point. You might be better off working in > C. > > However, if you really want Fortran, try breaking at a line number in your > file. Also > make sure you did not delete the *.o file > > Matt > > >> (gdb) r >> Starting program: /home/cfd/sgao/FEM_in_CFD/code/ex1f >> [Thread debugging using libthread_db enabled] >> >> Breakpoint 1, main (argc=1, argv=0x7fffffffcc58) at >> ../../../gcc-4.3.0/libgfortran/fmain.c:11 >> 11 ../../../gcc-4.3.0/libgfortran/fmain.c: No such file or directory. >> in ../../../gcc-4.3.0/libgfortran/fmain.c >> (gdb) l >> 6 in ../../../gcc-4.3.0/libgfortran/fmain.c >> (gdb) >> >> >> What did I miss here? Thank you in advance. >> >> Song >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Apr 16 14:25:14 2015 From: jed at jedbrown.org (Jed Brown) Date: Thu, 16 Apr 2015 13:25:14 -0600 Subject: [petsc-users] Paraview 3.x In-Reply-To: References: Message-ID: <87mw2727vp.fsf@jedbrown.org> Sanjay Kharche writes: > Hi > > I have 3D output in vts format produced using PetscViewerVTKopen and > VecView. I cannot volume render the data using paraview 3.x. Can't open it at all or can't volume render it? Do you get an error message? > The latest version of paraview (4.2.x) can visualise it. Why not use current Paraview? > How can I make 3D vts output that is compatible with paraview 3.x? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From haakon at hakostra.net Thu Apr 16 14:54:17 2015 From: haakon at hakostra.net (=?UTF-8?Q?H=C3=A5kon_Strandenes?=) Date: Thu, 16 Apr 2015 21:54:17 +0200 Subject: [petsc-users] Paraview 3.x In-Reply-To: References: Message-ID: <55301272.107@hakostra.net> I does at least know that Paraview *used to* only support volume rendering for unstructured grid data and voxel datasets, i.e. datasets in which the grid spacing is constant along each direction (but different spacing in the different directions are allowed). The volume rendering capabilities did at least not previously support rectilinear grids with non-uniform spacing. See f.ex. http://public.kitware.com/pipermail/paraview/2012-May/024992.html I guess your VTS dataset is written as a rectilinear grid with non-uniform spacing? (I have never used this viewer) Maybe they simply have added this support (or a workaround) in the latest 4.2 series? And that volume rendering of VTS data is not possible in your 3.x version? H?kon On 16. april 2015 20:50, Sanjay Kharche wrote: > > Hi > > I have 3D output in vts format produced using PetscViewerVTKopen and > VecView. I cannot volume render the data using paraview 3.x. The latest > version of paraview (4.2.x) can visualise it. > > How can I make 3D vts output that is compatible with paraview 3.x? > > thanks > Sanjay > From huaibao.zhang at gmail.com Thu Apr 16 16:54:31 2015 From: huaibao.zhang at gmail.com (huaibao zhang) Date: Thu, 16 Apr 2015 17:54:31 -0400 Subject: [petsc-users] The parallel preconditioner Message-ID: Hello there, I have been confused of the extent of the available preconditioners in PETSc for a while. Here are the lists I got from one slide of a lecture notes saying Block Jacobi, Jacobi, Overlapping Additive Schwarz, ICC, ILU (sequential only) ILU(K), LU(direct solver, sequential only) sor etc. But in some other sources, only LU is highlighted with "sequential only". Which one is correct? I am using MATCreateAIJ to construct my A. Which preconditioner is available to me? By the way, I tested asm, ilu, icc, and they all failed. Thanks, Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Apr 16 16:58:34 2015 From: jed at jedbrown.org (Jed Brown) Date: Thu, 16 Apr 2015 15:58:34 -0600 Subject: [petsc-users] The parallel preconditioner In-Reply-To: References: Message-ID: <87mw27zqet.fsf@jedbrown.org> huaibao zhang writes: > Hello there, > > I have been confused of the extent of the available preconditioners in PETSc for a while. Here are the lists I got from one slide of a lecture notes saying > > Block Jacobi, > Jacobi, > Overlapping Additive Schwarz, > ICC, ILU (sequential only) > ILU(K), LU(direct solver, sequential only) > sor > etc. > > > But in some other sources, only LU is highlighted with "sequential > only". -pc_type lu -pc_factor_mat_solver_package superlu_dist (or "mumps") will give you a parallel solver. > Which one is correct? I am using MATCreateAIJ to construct my A. Which > preconditioner is available to me? By the way, I tested asm, ilu, icc, > and they all failed. "failed" is not sufficiently descriptive. It would also help to know what sort of problem you're solving. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From Sanjay.Kharche at manchester.ac.uk Thu Apr 16 22:57:41 2015 From: Sanjay.Kharche at manchester.ac.uk (Sanjay Kharche) Date: Fri, 17 Apr 2015 03:57:41 +0000 Subject: [petsc-users] Paraview 3.x In-Reply-To: <87mw2727vp.fsf@jedbrown.org> References: , <87mw2727vp.fsf@jedbrown.org> Message-ID: Hi My grid is rectilinear with equal spacing in all directions, it is a regular structured grid. As of now, I am working through the various versions of paraview on my Fedora/Linux as well as a apple mac (debian) to find a working balance version of paraview for my purpose. It seems this discussion may be more for another list than this one. I will still elaborate. When I load the 3D vts structured grid (output from my 3D petsc based program) into paraview 3.4, I do not get an option to render volume. There are no error messages. The best I can do is get an isosurface or threshold the data. When I used paraview 4.3.1 distro binary, I can viz it but the controls are different. I spend some time getting familiar with the controls yesterday. My pvbatch scripts are going to be remade if I am to go down the 4.3.1 route. Another fix I implemented is that to do a binary output, and then post-process so that my current set up of paraview/vtk still works. cheers Sanjay ________________________________________ From: Jed Brown [jed at jedbrown.org] Sent: 16 April 2015 20:25 To: Sanjay Kharche; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Paraview 3.x Sanjay Kharche writes: > Hi > > I have 3D output in vts format produced using PetscViewerVTKopen and > VecView. I cannot volume render the data using paraview 3.x. Can't open it at all or can't volume render it? Do you get an error message? > The latest version of paraview (4.2.x) can visualise it. Why not use current Paraview? > How can I make 3D vts output that is compatible with paraview 3.x? From jed at jedbrown.org Thu Apr 16 23:04:46 2015 From: jed at jedbrown.org (Jed Brown) Date: Thu, 16 Apr 2015 22:04:46 -0600 Subject: [petsc-users] Paraview 3.x In-Reply-To: References: <87mw2727vp.fsf@jedbrown.org> Message-ID: <874mofz9gh.fsf@jedbrown.org> Sanjay Kharche writes: > When I load the 3D vts structured grid (output from my 3D petsc based > program) into paraview 3.4, I do not get an option to render > volume. There are no error messages. The best I can do is get an > isosurface or threshold the data. Sounds like H?kon's diagnosis is correct. For regular spacing, you could use .vtr instead of .vts, though it might not change this limitation of Paraview-3.4. > When I used paraview 4.3.1 distro binary, I can viz it but the > controls are different. I spend some time getting familiar with the > controls yesterday. My pvbatch scripts are going to be remade if I am > to go down the 4.3.1 route. Another fix I implemented is that to do a > binary output, and then post-process so that my current set up of > paraview/vtk still works. I recommend upgrading. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From haakon at hakostra.net Fri Apr 17 01:57:19 2015 From: haakon at hakostra.net (=?windows-1252?Q?H=E5kon_Strandenes?=) Date: Fri, 17 Apr 2015 08:57:19 +0200 Subject: [petsc-users] Paraview 3.x In-Reply-To: References: , <87mw2727vp.fsf@jedbrown.org> Message-ID: <5530AECF.1050708@hakostra.net> On 17. april 2015 05:57, Sanjay Kharche wrote: > > Hi > > My grid is rectilinear with equal spacing in all directions, it is a regular structured grid. It doesn't matter if your dataset physically/numerically has constant spacing, because as long as your dataset is loaded as a .vts file, it is seen only as a structured grid by Paraview, but it does not see properties like orthogonality and uniform spacing. Your *.vts file can in general contain data on any structured mesh, even curved ones, and Paraview will treat it that way as well. > As of now, I am working through the various versions of paraview on my Fedora/Linux as well as a apple mac (debian) to find a working balance version of paraview for my purpose. It seems this discussion may be more for another list than this one. I will still elaborate. > > When I load the 3D vts structured grid (output from my 3D petsc based program) into paraview 3.4, I do not get an option to render volume. There are no error messages. That's true. No error message. Paraview, at least previously, could not, by design, do volume rendering of anything else than unstructured grids and "image data" (rectilinear grids with constant spacing), so the option are hidden. But no error has occurred, so no error message. > The best I can do is get an isosurface or threshold the data. Then Paraview has successfully loaded your data. No errors. > When I used paraview 4.3.1 distro binary, I can viz it but the controls are different. I spend some time getting familiar with the controls yesterday. My pvbatch scripts are going to be remade if I am to go down the 4.3.1 route. At some point you will have to do that anyway. > Another fix I implemented is that to do a binary output, and then post-process so that my current set up of paraview/vtk still works. I guess that when you load your binary dataset, Paraview suddenly sees this as a "image data" dataset, which you can use for volume rendering. For info on the VTK file formats, see http://ins.uni-bonn.de/teaching/alumni/bebendorf/Praktikum_SS12/vtk_file_formats.pdf Anyways, I absolutely support Jed Brown's advice: upgrade. Asap. Especially if that solves your problem. (However, I cannot get volume rendering working with PV 4.3.1 on non-equidistant meshes myself, but again, if upgrading helps for you, do it!) Regards, H?kon > > cheers > Sanjay > > > ________________________________________ > From: Jed Brown [jed at jedbrown.org] > Sent: 16 April 2015 20:25 > To: Sanjay Kharche; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Paraview 3.x > > Sanjay Kharche writes: > >> Hi >> >> I have 3D output in vts format produced using PetscViewerVTKopen and >> VecView. I cannot volume render the data using paraview 3.x. > > Can't open it at all or can't volume render it? Do you get an error message? > >> The latest version of paraview (4.2.x) can visualise it. > > Why not use current Paraview? > >> How can I make 3D vts output that is compatible with paraview 3.x? > From patrick.sanan at gmail.com Fri Apr 17 02:22:02 2015 From: patrick.sanan at gmail.com (Patrick Sanan) Date: Fri, 17 Apr 2015 09:22:02 +0200 Subject: [petsc-users] The parallel preconditioner In-Reply-To: References: Message-ID: <638097FB-5326-4BB2-ACF0-53B3763FB6AE@gmail.com> You can find the current list of preconditioners on line 40 here: http://www.mcs.anl.gov/petsc/petsc-current/include/petscpc.h.html You can click on the names to be taken to man pages. However, since preconditioners are by nature problem-dependent, it's a better idea to try to look up what might be an appropriate preconditioner for your system and then see how to use PETSc to apply it. There's a high probability that what you need is supported either natively by PETSc or as Jed's note demonstrates, through interfaces to external libraries (of which there are a large number - run ./configure -help to get an idea of the options available and the configure flags to use them) > Il giorno 16/apr/2015, alle ore 23:54, huaibao zhang ha scritto: > > Hello there, > > I have been confused of the extent of the available preconditioners in PETSc for a while. Here are the lists I got from one slide of a lecture notes saying > > Block Jacobi, > Jacobi, > Overlapping Additive Schwarz, > ICC, ILU (sequential only) > ILU(K), LU(direct solver, sequential only) > sor > etc. > > > But in some other sources, only LU is highlighted with "sequential only". Which one is correct? I am using MATCreateAIJ to construct my A. Which preconditioner is available to me? By the way, I tested asm, ilu, icc, and they all failed. > > Thanks, > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paulhuaizhang at gmail.com Fri Apr 17 08:28:08 2015 From: paulhuaizhang at gmail.com (paul zhang) Date: Fri, 17 Apr 2015 09:28:08 -0400 Subject: [petsc-users] The parallel preconditioner In-Reply-To: <638097FB-5326-4BB2-ACF0-53B3763FB6AE@gmail.com> References: <638097FB-5326-4BB2-ACF0-53B3763FB6AE@gmail.com> Message-ID: I agree with you. The performance indeed depends on the type of problem. Anyway, back to the questions I proposed, I am afraid ILU and LU are not sequential only. Am I right? Thanks, Paul Huaibao (Paul) Zhang *Gas Surface Interactions Lab* Department of Mechanical Engineering University of Kentucky, Lexington, KY, 40506-0503 *Office*: 216 Ralph G. Anderson Building *Web*:gsil.engineering.uky.edu On Fri, Apr 17, 2015 at 3:22 AM, Patrick Sanan wrote: > You can find the current list of preconditioners on line 40 here: > > http://www.mcs.anl.gov/petsc/petsc-current/include/petscpc.h.html > > You can click on the names to be taken to man pages. > > However, since preconditioners are by nature problem-dependent, it's a > better idea to try to look up what might be an appropriate preconditioner > for your system and then see how to use PETSc to apply it. There's a high > probability that what you need is supported either natively by PETSc or as > Jed's note demonstrates, through interfaces to external libraries (of which > there are a large number - run ./configure -help to get an idea of the > options available and the configure flags to use them) > > > Il giorno 16/apr/2015, alle ore 23:54, huaibao zhang < > huaibao.zhang at gmail.com> ha scritto: > > Hello there, > > I have been confused of the extent of the available preconditioners in > PETSc for a while. Here are the lists I got from one slide of a lecture > notes saying > > Block Jacobi, > Jacobi, > Overlapping Additive Schwarz, > ICC, ILU (sequential only) > ILU(K), LU(direct solver, sequential only) > sor > etc. > > > But in some other sources, only LU is highlighted with "sequential only". > Which one is correct? I am using MATCreateAIJ to construct my A. Which > preconditioner is available to me? By the way, I tested asm, ilu, icc, and > they all failed. > > Thanks, > Paul > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 17 08:31:25 2015 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 17 Apr 2015 08:31:25 -0500 Subject: [petsc-users] The parallel preconditioner In-Reply-To: References: <638097FB-5326-4BB2-ACF0-53B3763FB6AE@gmail.com> Message-ID: On Fri, Apr 17, 2015 at 8:28 AM, paul zhang wrote: > I agree with you. The performance indeed depends on the type of problem. > Anyway, back to the questions I proposed, I am afraid ILU and LU are > not sequential only. Am I right? > As it says in the table, the PETSc implementations are sequential, but there are parallel implementations available through other packages, such as SuperLU_dist, which can de downloaded through configure, e.g. --download-superlu_dist. Thanks, Matt > Thanks, > Paul > > Huaibao (Paul) Zhang > *Gas Surface Interactions Lab* > Department of Mechanical Engineering > University of Kentucky, > Lexington, > KY, 40506-0503 > *Office*: 216 Ralph G. Anderson Building > *Web*:gsil.engineering.uky.edu > > On Fri, Apr 17, 2015 at 3:22 AM, Patrick Sanan > wrote: > >> You can find the current list of preconditioners on line 40 here: >> >> http://www.mcs.anl.gov/petsc/petsc-current/include/petscpc.h.html >> >> You can click on the names to be taken to man pages. >> >> However, since preconditioners are by nature problem-dependent, it's a >> better idea to try to look up what might be an appropriate preconditioner >> for your system and then see how to use PETSc to apply it. There's a high >> probability that what you need is supported either natively by PETSc or as >> Jed's note demonstrates, through interfaces to external libraries (of which >> there are a large number - run ./configure -help to get an idea of the >> options available and the configure flags to use them) >> >> >> Il giorno 16/apr/2015, alle ore 23:54, huaibao zhang < >> huaibao.zhang at gmail.com> ha scritto: >> >> Hello there, >> >> I have been confused of the extent of the available preconditioners in >> PETSc for a while. Here are the lists I got from one slide of a lecture >> notes saying >> >> Block Jacobi, >> Jacobi, >> Overlapping Additive Schwarz, >> ICC, ILU (sequential only) >> ILU(K), LU(direct solver, sequential only) >> sor >> etc. >> >> >> But in some other sources, only LU is highlighted with "sequential only". >> Which one is correct? I am using MATCreateAIJ to construct my A. Which >> preconditioner is available to me? By the way, I tested asm, ilu, icc, and >> they all failed. >> >> Thanks, >> Paul >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From paulhuaizhang at gmail.com Fri Apr 17 08:48:47 2015 From: paulhuaizhang at gmail.com (paul zhang) Date: Fri, 17 Apr 2015 09:48:47 -0400 Subject: [petsc-users] The parallel preconditioner In-Reply-To: References: <638097FB-5326-4BB2-ACF0-53B3763FB6AE@gmail.com> Message-ID: Matt, That really makes sense to me. Thanks, Paul Huaibao (Paul) Zhang *Gas Surface Interactions Lab* Department of Mechanical Engineering University of Kentucky, Lexington, KY, 40506-0503 *Office*: 216 Ralph G. Anderson Building *Web*:gsil.engineering.uky.edu On Fri, Apr 17, 2015 at 9:31 AM, Matthew Knepley wrote: > On Fri, Apr 17, 2015 at 8:28 AM, paul zhang > wrote: > >> I agree with you. The performance indeed depends on the type of problem. >> Anyway, back to the questions I proposed, I am afraid ILU and LU are >> not sequential only. Am I right? >> > > As it says in the table, the PETSc implementations are sequential, but > there are parallel implementations available through > other packages, such as SuperLU_dist, which can de downloaded through > configure, e.g. --download-superlu_dist. > > Thanks, > > Matt > > >> Thanks, >> Paul >> >> Huaibao (Paul) Zhang >> *Gas Surface Interactions Lab* >> Department of Mechanical Engineering >> University of Kentucky, >> Lexington, >> KY, 40506-0503 >> *Office*: 216 Ralph G. Anderson Building >> *Web*:gsil.engineering.uky.edu >> >> On Fri, Apr 17, 2015 at 3:22 AM, Patrick Sanan >> wrote: >> >>> You can find the current list of preconditioners on line 40 here: >>> >>> http://www.mcs.anl.gov/petsc/petsc-current/include/petscpc.h.html >>> >>> You can click on the names to be taken to man pages. >>> >>> However, since preconditioners are by nature problem-dependent, it's a >>> better idea to try to look up what might be an appropriate preconditioner >>> for your system and then see how to use PETSc to apply it. There's a high >>> probability that what you need is supported either natively by PETSc or as >>> Jed's note demonstrates, through interfaces to external libraries (of which >>> there are a large number - run ./configure -help to get an idea of the >>> options available and the configure flags to use them) >>> >>> >>> Il giorno 16/apr/2015, alle ore 23:54, huaibao zhang < >>> huaibao.zhang at gmail.com> ha scritto: >>> >>> Hello there, >>> >>> I have been confused of the extent of the available preconditioners in >>> PETSc for a while. Here are the lists I got from one slide of a lecture >>> notes saying >>> >>> Block Jacobi, >>> Jacobi, >>> Overlapping Additive Schwarz, >>> ICC, ILU (sequential only) >>> ILU(K), LU(direct solver, sequential only) >>> sor >>> etc. >>> >>> >>> But in some other sources, only LU is highlighted with "sequential >>> only". Which one is correct? I am using MATCreateAIJ to construct my A. >>> Which preconditioner is available to me? By the way, I tested asm, ilu, >>> icc, and they all failed. >>> >>> Thanks, >>> Paul >>> >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Fri Apr 17 09:10:48 2015 From: mfadams at lbl.gov (Mark Adams) Date: Fri, 17 Apr 2015 10:10:48 -0400 Subject: [petsc-users] error using GetType in FORTRAN Message-ID: I declare PCCompositeType::pcctype But PCFieldSplitGetType does not seem to like it? Mark [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Invalid pointer [1]PETSC ERROR: Invalid Pointer to Int: Parameter # 2 [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2702-g1f714da GIT Date: 2015-04-16 11:32:06 -0500 [1]PETSC ERROR: ../../XGC1_3/xgc-em on a arch-macosx-gnu-g named MacBook-Pro.local by markadams Fri Apr 17 10:06:09 2015 [1]PETSC ERROR: Configure options --download-hypre=1 --download-p4est --download-superlu=1 --download-mumps --download-scalapack --download-superlu_dist=1 --download-parmetis=1 --download-metis=1 --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5 --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g [1]PETSC ERROR: #1 PCFieldSplitGetType() line 2046 in /Users/markadams/Codes/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c -------------- next part -------------- An HTML attachment was scrubbed... URL: From micketeer at gmail.com Fri Apr 17 14:11:09 2015 From: micketeer at gmail.com (=?UTF-8?Q?Mikael_=C3=96hman?=) Date: Fri, 17 Apr 2015 21:11:09 +0200 Subject: [petsc-users] Trying to use MatSeq*AIJSetColumnIndices Message-ID: In OOFEM, we do a "dry assembly" from which we compute the preallocation ( e.g. MatSeqSBAIJSetPreallocation ). Since I already compute the column index in order to compute the "nnz" input for preallocation, I figured I'd try to use the "MatSeq*AIJSetColumnIndices" functions as well. First, I'd like to make sure I'm using these in the correct way. I do the following order of calls: MatCreate(PETSC_COMM_SELF, & mtrx); MatSetSizes(mtrx, leqs, leqs, geqs, geqs); MatSetType(mtrx, MATSEQSBAIJ); MatSeqSBAIJSetPreallocation( mtrx, blocksize, 0, d_nnz_sym.givePointer() ); // Trying to add this for faster assembly: MatSeqSBAIJSetColumnIndices( mtrx, indices.givePointer() ); MatSetUp(mtrx); MatSetOption(mtrx, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE); I run a fairly large finite element analysis (600k DOFs) through VTUNE to profile. I have two issues: 1. There is no difference in the MatAssembly routine. I can't tell if I'm using the routine wrong, or if preallocation is really all you need to achieve maximum performance. MatAssembly isn't terribly slow for me, but it does take 1/5th of the time that KSPSolve takes (which seems a big high). 2. Setting the column indices for AIJ, BAIJ works without errors, but when I try with SBAIJ with a blocksize > 1, then it segfaults in "MatSeqSBAIJSetColumnIndices_SeqSBAIJ" at line 1067: for (i=0; iilen[i] = baij->imax[i]; where the loop (n = mat->cmap->n) goes up to the number of columns (as opposed to columns/blocksize). The corresponding BAIJ implementation uses "baij->mbs" as the loop limit. A bug perhaps? Regards, Mikael -------------- next part -------------- An HTML attachment was scrubbed... URL: From hus003 at ucsd.edu Fri Apr 17 14:38:29 2015 From: hus003 at ucsd.edu (Sun, Hui) Date: Fri, 17 Apr 2015 19:38:29 +0000 Subject: [petsc-users] install petsc with valgrind Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> Hello, I want to install petsc with valgrind. And I read some documentations, which suggests me to do ./configure --download-mpich --with-valgind-dir=? However, I also want to specify --prefix, but petsc doesn't allow me, because --download-mpich option does not work with --prefix install. Is there anyway that I can have petsc --prefix, together with valgrind? Best, Hui -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Apr 17 15:08:10 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 17 Apr 2015 15:08:10 -0500 Subject: [petsc-users] install petsc with valgrind In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> The development copy of PETSc allows using --prefix with MPICH http://www.mcs.anl.gov/petsc/developers/index.html > On Apr 17, 2015, at 2:38 PM, Sun, Hui wrote: > > Hello, > > I want to install petsc with valgrind. And I read some documentations, which suggests me to do > ./configure --download-mpich --with-valgind-dir=? > > However, I also want to specify --prefix, but petsc doesn't allow me, because --download-mpich option does not work with --prefix install. > > Is there anyway that I can have petsc --prefix, together with valgrind? > > Best, > Hui From knepley at gmail.com Fri Apr 17 15:11:19 2015 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 17 Apr 2015 15:11:19 -0500 Subject: [petsc-users] Trying to use MatSeq*AIJSetColumnIndices In-Reply-To: References: Message-ID: On Fri, Apr 17, 2015 at 2:11 PM, Mikael ?hman wrote: > In OOFEM, we do a "dry assembly" from which we compute the preallocation ( > e.g. MatSeqSBAIJSetPreallocation ). > > Since I already compute the column index in order to compute the "nnz" > input for preallocation, I figured I'd try to use the > "MatSeq*AIJSetColumnIndices" functions as well. > > First, I'd like to make sure I'm using these in the correct way. I do the > following order of calls: > > MatCreate(PETSC_COMM_SELF, & mtrx); > MatSetSizes(mtrx, leqs, leqs, geqs, geqs); > MatSetType(mtrx, MATSEQSBAIJ); > MatSeqSBAIJSetPreallocation( mtrx, blocksize, 0, > d_nnz_sym.givePointer() ); > // Trying to add this for faster assembly: > MatSeqSBAIJSetColumnIndices( mtrx, indices.givePointer() ); > MatSetUp(mtrx); > MatSetOption(mtrx, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_TRUE); > > I run a fairly large finite element analysis (600k DOFs) through VTUNE to > profile. > > > > I have two issues: > 1. There is no difference in the MatAssembly routine. I can't tell if I'm > using the routine wrong, or if preallocation is really all you need to > achieve maximum performance. > MatAssembly isn't terribly slow for me, but it does take 1/5th of the time > that KSPSolve takes (which seems a big high). > Preallocation is all you need. If you want to assure that the pattern is stable under assembly, the best thing to do is insert zeros. That is what we do by default for the DM matrices. > 2. Setting the column indices for AIJ, BAIJ works without errors, but when > I try with SBAIJ with a blocksize > 1, then it segfaults in > "MatSeqSBAIJSetColumnIndices_SeqSBAIJ" at line 1067: > > for (i=0; iilen[i] = baij->imax[i]; > > where the loop (n = mat->cmap->n) goes up to the number of columns (as > opposed to columns/blocksize). The corresponding BAIJ implementation uses > "baij->mbs" as the loop limit. > A bug perhaps? > Yes, that looks like a bug Matt > > Regards, Mikael > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Apr 17 15:12:34 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 17 Apr 2015 15:12:34 -0500 Subject: [petsc-users] error using GetType in FORTRAN In-Reply-To: References: Message-ID: <79ABF344-22F0-4FE8-9DF8-4793D8D0265F@mcs.anl.gov> Likely need to run in the debugger to see what generates the error. Perhaps a missing ierr? Barry > On Apr 17, 2015, at 9:10 AM, Mark Adams wrote: > > I declare > > PCCompositeType::pcctype > > But PCFieldSplitGetType does not seem to like it? > > Mark > > [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [1]PETSC ERROR: Invalid pointer > [1]PETSC ERROR: Invalid Pointer to Int: Parameter # 2 > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [1]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2702-g1f714da GIT Date: 2015-04-16 11:32:06 -0500 > [1]PETSC ERROR: ../../XGC1_3/xgc-em on a arch-macosx-gnu-g named MacBook-Pro.local by markadams Fri Apr 17 10:06:09 2015 > [1]PETSC ERROR: Configure options --download-hypre=1 --download-p4est --download-superlu=1 --download-mumps --download-scalapack --download-superlu_dist=1 --download-parmetis=1 --download-metis=1 --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5 --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g > [1]PETSC ERROR: #1 PCFieldSplitGetType() line 2046 in /Users/markadams/Codes/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c > From hus003 at ucsd.edu Fri Apr 17 15:58:42 2015 From: hus003 at ucsd.edu (Sun, Hui) Date: Fri, 17 Apr 2015 20:58:42 +0000 Subject: [petsc-users] install petsc with valgrind In-Reply-To: <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU>, <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> Thank you Barry. I just have it installed. And I was running a code, and got the following error message: HYDU_create_process (utils/launch/launch.c:75): execvp error on file valgrind: (No such file or directory) =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 66059 RUNNING AT math178.ucsd.edu = EXIT CODE: 255 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== What does this mean? Best, Hui ________________________________________ From: Barry Smith [bsmith at mcs.anl.gov] Sent: Friday, April 17, 2015 1:08 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] install petsc with valgrind The development copy of PETSc allows using --prefix with MPICH http://www.mcs.anl.gov/petsc/developers/index.html > On Apr 17, 2015, at 2:38 PM, Sun, Hui wrote: > > Hello, > > I want to install petsc with valgrind. And I read some documentations, which suggests me to do > ./configure --download-mpich --with-valgind-dir=? > > However, I also want to specify --prefix, but petsc doesn't allow me, because --download-mpich option does not work with --prefix install. > > Is there anyway that I can have petsc --prefix, together with valgrind? > > Best, > Hui From knepley at gmail.com Fri Apr 17 16:31:54 2015 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 17 Apr 2015 16:31:54 -0500 Subject: [petsc-users] install petsc with valgrind In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: On Fri, Apr 17, 2015 at 3:58 PM, Sun, Hui wrote: > Thank you Barry. I just have it installed. And I was running a code, and > got the following error message: > > HYDU_create_process (utils/launch/launch.c:75): execvp error on file > valgrind: (No such file or directory) > > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 66059 RUNNING AT math178.ucsd.edu > = EXIT CODE: 255 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > =================================================================================== > > What does this mean? > It appears that you are trying to run mpiexec valgrind and it cannot find the valgrind executable. Matt > Best, > Hui > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Friday, April 17, 2015 1:08 PM > To: Sun, Hui > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] install petsc with valgrind > > The development copy of PETSc allows using --prefix with MPICH > http://www.mcs.anl.gov/petsc/developers/index.html > > > > > On Apr 17, 2015, at 2:38 PM, Sun, Hui wrote: > > > > Hello, > > > > I want to install petsc with valgrind. And I read some documentations, > which suggests me to do > > ./configure --download-mpich --with-valgind-dir=? > > > > However, I also want to specify --prefix, but petsc doesn't allow me, > because --download-mpich option does not work with --prefix install. > > > > Is there anyway that I can have petsc --prefix, together with valgrind? > > > > Best, > > Hui > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Fri Apr 17 16:37:40 2015 From: mfadams at lbl.gov (Mark Adams) Date: Fri, 17 Apr 2015 17:37:40 -0400 Subject: [petsc-users] error using GetType in FORTRAN In-Reply-To: <79ABF344-22F0-4FE8-9DF8-4793D8D0265F@mcs.anl.gov> References: <79ABF344-22F0-4FE8-9DF8-4793D8D0265F@mcs.anl.gov> Message-ID: It looks like fieldsplitf.c has an extra PetscToPointer. type is already a pointer: PETSC_EXTERN void PETSC_STDCALL pcfieldsplitgettype_(PC pc,PCCompositeType *type, int *__ierr ){ *__ierr = PCFieldSplitGetType( (PC)PetscToPointer((pc) ), (PCCompositeType* )PetscToPointer((type) )); } On Fri, Apr 17, 2015 at 4:12 PM, Barry Smith wrote: > > Likely need to run in the debugger to see what generates the error. > Perhaps a missing ierr? > > Barry > > > > On Apr 17, 2015, at 9:10 AM, Mark Adams wrote: > > > > I declare > > > > PCCompositeType::pcctype > > > > But PCFieldSplitGetType does not seem to like it? > > > > Mark > > > > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [1]PETSC ERROR: Invalid pointer > > [1]PETSC ERROR: Invalid Pointer to Int: Parameter # 2 > > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [1]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2702-g1f714da > GIT Date: 2015-04-16 11:32:06 -0500 > > [1]PETSC ERROR: ../../XGC1_3/xgc-em on a arch-macosx-gnu-g named > MacBook-Pro.local by markadams Fri Apr 17 10:06:09 2015 > > [1]PETSC ERROR: Configure options --download-hypre=1 --download-p4est > --download-superlu=1 --download-mumps --download-scalapack > --download-superlu_dist=1 --download-parmetis=1 --download-metis=1 > --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5 > --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g > > [1]PETSC ERROR: #1 PCFieldSplitGetType() line 2046 in > /Users/markadams/Codes/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 17 16:43:05 2015 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 17 Apr 2015 16:43:05 -0500 Subject: [petsc-users] error using GetType in FORTRAN In-Reply-To: References: <79ABF344-22F0-4FE8-9DF8-4793D8D0265F@mcs.anl.gov> Message-ID: Does bfort have to be told that PCCompositeType is an enum? Matt "bfort's biggest fan" On Fri, Apr 17, 2015 at 4:37 PM, Mark Adams wrote: > It looks like fieldsplitf.c has an extra PetscToPointer. type is already > a pointer: > > PETSC_EXTERN void PETSC_STDCALL pcfieldsplitgettype_(PC > pc,PCCompositeType *type, int *__ierr ){ > *__ierr = PCFieldSplitGetType( > (PC)PetscToPointer((pc) ), > (PCCompositeType* )PetscToPointer((type) )); > } > > On Fri, Apr 17, 2015 at 4:12 PM, Barry Smith wrote: > >> >> Likely need to run in the debugger to see what generates the error. >> Perhaps a missing ierr? >> >> Barry >> >> >> > On Apr 17, 2015, at 9:10 AM, Mark Adams wrote: >> > >> > I declare >> > >> > PCCompositeType::pcctype >> > >> > But PCFieldSplitGetType does not seem to like it? >> > >> > Mark >> > >> > [1]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> > [1]PETSC ERROR: Invalid pointer >> > [1]PETSC ERROR: Invalid Pointer to Int: Parameter # 2 >> > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> > [1]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2702-g1f714da >> GIT Date: 2015-04-16 11:32:06 -0500 >> > [1]PETSC ERROR: ../../XGC1_3/xgc-em on a arch-macosx-gnu-g named >> MacBook-Pro.local by markadams Fri Apr 17 10:06:09 2015 >> > [1]PETSC ERROR: Configure options --download-hypre=1 --download-p4est >> --download-superlu=1 --download-mumps --download-scalapack >> --download-superlu_dist=1 --download-parmetis=1 --download-metis=1 >> --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5 >> --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g >> > [1]PETSC ERROR: #1 PCFieldSplitGetType() line 2046 in >> /Users/markadams/Codes/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> > >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Apr 17 16:49:05 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 17 Apr 2015 16:49:05 -0500 Subject: [petsc-users] error using GetType in FORTRAN In-Reply-To: References: <79ABF344-22F0-4FE8-9DF8-4793D8D0265F@mcs.anl.gov> Message-ID: <9F934B14-8082-49E5-AB36-53D4F07E5217@mcs.anl.gov> Looks like a bunch of these things are missing from lib/petsc/conf/bfort-petsc.txt it should be listed under native Barry > On Apr 17, 2015, at 4:43 PM, Matthew Knepley wrote: > > Does bfort have to be told that PCCompositeType is an enum? > > Matt "bfort's biggest fan" > > On Fri, Apr 17, 2015 at 4:37 PM, Mark Adams wrote: > It looks like fieldsplitf.c has an extra PetscToPointer. type is already a pointer: > > PETSC_EXTERN void PETSC_STDCALL pcfieldsplitgettype_(PC pc,PCCompositeType *type, int *__ierr ){ > *__ierr = PCFieldSplitGetType( > (PC)PetscToPointer((pc) ), > (PCCompositeType* )PetscToPointer((type) )); > } > > On Fri, Apr 17, 2015 at 4:12 PM, Barry Smith wrote: > > Likely need to run in the debugger to see what generates the error. Perhaps a missing ierr? > > Barry > > > > On Apr 17, 2015, at 9:10 AM, Mark Adams wrote: > > > > I declare > > > > PCCompositeType::pcctype > > > > But PCFieldSplitGetType does not seem to like it? > > > > Mark > > > > [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [1]PETSC ERROR: Invalid pointer > > [1]PETSC ERROR: Invalid Pointer to Int: Parameter # 2 > > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [1]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2702-g1f714da GIT Date: 2015-04-16 11:32:06 -0500 > > [1]PETSC ERROR: ../../XGC1_3/xgc-em on a arch-macosx-gnu-g named MacBook-Pro.local by markadams Fri Apr 17 10:06:09 2015 > > [1]PETSC ERROR: Configure options --download-hypre=1 --download-p4est --download-superlu=1 --download-mumps --download-scalapack --download-superlu_dist=1 --download-parmetis=1 --download-metis=1 --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5 --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g > > [1]PETSC ERROR: #1 PCFieldSplitGetType() line 2046 in /Users/markadams/Codes/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c > > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From bsmith at mcs.anl.gov Fri Apr 17 16:50:16 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 17 Apr 2015 16:50:16 -0500 Subject: [petsc-users] install petsc with valgrind In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: <498A9304-AF3F-42CF-800A-295FB9C0897D@mcs.anl.gov> Please please please always show WHAT YOU RAN with the error message. Otherwise we just must guess > On Apr 17, 2015, at 3:58 PM, Sun, Hui wrote: > > Thank you Barry. I just have it installed. And I was running a code, and got the following error message: > > HYDU_create_process (utils/launch/launch.c:75): execvp error on file valgrind: (No such file or directory) > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 66059 RUNNING AT math178.ucsd.edu > = EXIT CODE: 255 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > > What does this mean? > > Best, > Hui > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Friday, April 17, 2015 1:08 PM > To: Sun, Hui > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] install petsc with valgrind > > The development copy of PETSc allows using --prefix with MPICH http://www.mcs.anl.gov/petsc/developers/index.html > > > >> On Apr 17, 2015, at 2:38 PM, Sun, Hui wrote: >> >> Hello, >> >> I want to install petsc with valgrind. And I read some documentations, which suggests me to do >> ./configure --download-mpich --with-valgind-dir=? >> >> However, I also want to specify --prefix, but petsc doesn't allow me, because --download-mpich option does not work with --prefix install. >> >> Is there anyway that I can have petsc --prefix, together with valgrind? >> >> Best, >> Hui > From hus003 at ucsd.edu Fri Apr 17 16:55:35 2015 From: hus003 at ucsd.edu (Sun, Hui) Date: Fri, 17 Apr 2015 21:55:35 +0000 Subject: [petsc-users] install petsc with valgrind In-Reply-To: <498A9304-AF3F-42CF-800A-295FB9C0897D@mcs.anl.gov> References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU>, <498A9304-AF3F-42CF-800A-295FB9C0897D@mcs.anl.gov> Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010FE31E@XMAIL-MBX-BH1.AD.UCSD.EDU> Thank you, Matt and Barry. I was trying to run the following: -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm where NP=2, ND=32 And in the makefile I have specified the following: include ${PETSC_DIR}/lib/petsc/conf/variables include ${PETSC_DIR}/lib/petsc/conf/rules include ${PETSC_DIR}/lib/petsc/conf/test Best, Hui ________________________________________ From: Barry Smith [bsmith at mcs.anl.gov] Sent: Friday, April 17, 2015 2:50 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] install petsc with valgrind Please please please always show WHAT YOU RAN with the error message. Otherwise we just must guess > On Apr 17, 2015, at 3:58 PM, Sun, Hui wrote: > > Thank you Barry. I just have it installed. And I was running a code, and got the following error message: > > HYDU_create_process (utils/launch/launch.c:75): execvp error on file valgrind: (No such file or directory) > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 66059 RUNNING AT math178.ucsd.edu > = EXIT CODE: 255 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > > What does this mean? > > Best, > Hui > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Friday, April 17, 2015 1:08 PM > To: Sun, Hui > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] install petsc with valgrind > > The development copy of PETSc allows using --prefix with MPICH http://www.mcs.anl.gov/petsc/developers/index.html > > > >> On Apr 17, 2015, at 2:38 PM, Sun, Hui wrote: >> >> Hello, >> >> I want to install petsc with valgrind. And I read some documentations, which suggests me to do >> ./configure --download-mpich --with-valgind-dir=? >> >> However, I also want to specify --prefix, but petsc doesn't allow me, because --download-mpich option does not work with --prefix install. >> >> Is there anyway that I can have petsc --prefix, together with valgrind? >> >> Best, >> Hui > From knepley at gmail.com Fri Apr 17 16:56:39 2015 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 17 Apr 2015 16:56:39 -0500 Subject: [petsc-users] install petsc with valgrind In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010FE31E@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> <498A9304-AF3F-42CF-800A-295FB9C0897D@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE31E@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: As I said, it cannot find valgrind in your path. Remove it or install valgrind. Matt On Fri, Apr 17, 2015 at 4:55 PM, Sun, Hui wrote: > Thank you, Matt and Barry. I was trying to run the following: > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my > ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > where NP=2, ND=32 > > And in the makefile I have specified the following: > include ${PETSC_DIR}/lib/petsc/conf/variables > include ${PETSC_DIR}/lib/petsc/conf/rules > include ${PETSC_DIR}/lib/petsc/conf/test > > Best, > Hui > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Friday, April 17, 2015 2:50 PM > To: Sun, Hui > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] install petsc with valgrind > > Please please please always show WHAT YOU RAN with the error message. > Otherwise we just must guess > > > > On Apr 17, 2015, at 3:58 PM, Sun, Hui wrote: > > > > Thank you Barry. I just have it installed. And I was running a code, and > got the following error message: > > > > HYDU_create_process (utils/launch/launch.c:75): execvp error on file > valgrind: (No such file or directory) > > > > > =================================================================================== > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > = PID 66059 RUNNING AT math178.ucsd.edu > > = EXIT CODE: 255 > > = CLEANING UP REMAINING PROCESSES > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > > =================================================================================== > > > > What does this mean? > > > > Best, > > Hui > > > > ________________________________________ > > From: Barry Smith [bsmith at mcs.anl.gov] > > Sent: Friday, April 17, 2015 1:08 PM > > To: Sun, Hui > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] install petsc with valgrind > > > > The development copy of PETSc allows using --prefix with MPICH > http://www.mcs.anl.gov/petsc/developers/index.html > > > > > > > >> On Apr 17, 2015, at 2:38 PM, Sun, Hui wrote: > >> > >> Hello, > >> > >> I want to install petsc with valgrind. And I read some documentations, > which suggests me to do > >> ./configure --download-mpich --with-valgind-dir=? > >> > >> However, I also want to specify --prefix, but petsc doesn't allow me, > because --download-mpich option does not work with --prefix install. > >> > >> Is there anyway that I can have petsc --prefix, together with valgrind? > >> > >> Best, > >> Hui > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Apr 17 16:57:48 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 17 Apr 2015 16:57:48 -0500 Subject: [petsc-users] install petsc with valgrind In-Reply-To: References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> <498A9304-AF3F-42CF-800A-295FB9C0897D@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE31E@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: <902FF397-BB49-4647-9E91-E11D2C851446@mcs.anl.gov> Or make sure valgrind is in your path > On Apr 17, 2015, at 4:56 PM, Matthew Knepley wrote: > > As I said, it cannot find valgrind in your path. Remove it or install valgrind. > > Matt > > On Fri, Apr 17, 2015 at 4:55 PM, Sun, Hui wrote: > Thank you, Matt and Barry. I was trying to run the following: > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > where NP=2, ND=32 > > And in the makefile I have specified the following: > include ${PETSC_DIR}/lib/petsc/conf/variables > include ${PETSC_DIR}/lib/petsc/conf/rules > include ${PETSC_DIR}/lib/petsc/conf/test > > Best, > Hui > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Friday, April 17, 2015 2:50 PM > To: Sun, Hui > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] install petsc with valgrind > > Please please please always show WHAT YOU RAN with the error message. Otherwise we just must guess > > > > On Apr 17, 2015, at 3:58 PM, Sun, Hui wrote: > > > > Thank you Barry. I just have it installed. And I was running a code, and got the following error message: > > > > HYDU_create_process (utils/launch/launch.c:75): execvp error on file valgrind: (No such file or directory) > > > > =================================================================================== > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > = PID 66059 RUNNING AT math178.ucsd.edu > > = EXIT CODE: 255 > > = CLEANING UP REMAINING PROCESSES > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > =================================================================================== > > > > What does this mean? > > > > Best, > > Hui > > > > ________________________________________ > > From: Barry Smith [bsmith at mcs.anl.gov] > > Sent: Friday, April 17, 2015 1:08 PM > > To: Sun, Hui > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] install petsc with valgrind > > > > The development copy of PETSc allows using --prefix with MPICH http://www.mcs.anl.gov/petsc/developers/index.html > > > > > > > >> On Apr 17, 2015, at 2:38 PM, Sun, Hui wrote: > >> > >> Hello, > >> > >> I want to install petsc with valgrind. And I read some documentations, which suggests me to do > >> ./configure --download-mpich --with-valgind-dir=? > >> > >> However, I also want to specify --prefix, but petsc doesn't allow me, because --download-mpich option does not work with --prefix install. > >> > >> Is there anyway that I can have petsc --prefix, together with valgrind? > >> > >> Best, > >> Hui > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From hus003 at ucsd.edu Fri Apr 17 17:09:03 2015 From: hus003 at ucsd.edu (Sun, Hui) Date: Fri, 17 Apr 2015 22:09:03 +0000 Subject: [petsc-users] install petsc with valgrind In-Reply-To: <902FF397-BB49-4647-9E91-E11D2C851446@mcs.anl.gov> References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> <498A9304-AF3F-42CF-800A-295FB9C0897D@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE31E@XMAIL-MBX-BH1.AD.UCSD.EDU> , <902FF397-BB49-4647-9E91-E11D2C851446@mcs.anl.gov> Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010FE343@XMAIL-MBX-BH1.AD.UCSD.EDU> I'm not sure if I'm doing the right thing. But I have just export PATH=$PATH:dir_to_valgind_bin And I rerun the command: -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm Then it seems to output something, for example: ==66450== Invalid write of size 8 ==66450== at 0x100055225: Schur_SetMatrix (in ./test) ==66450== by 0x10004F218: StokesSetup (in ./test) ==66450== by 0x1000DC582: main (in ./test) ==66450== Address 0x7fff5f93fb28 is on thread 1's stack ==66450== in frame #0, created by Schur_SetMatrix (???) ==66450== And before those output, it has the following lines: --66450-- run: /usr/bin/dsymutil "./test" warning: (x86_64) /Users/hus003/projects/LS_BDRY/test/test.o unable to open object file warning: no debug symbols in executable (-arch x86_64) --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpicxx.12.dylib" --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpifort.12.dylib" --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpi.12.dylib" --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libpmpi.12.dylib" Does it mean that it's working? But what does these two warnings mean? Best, Hui ________________________________________ From: Barry Smith [bsmith at mcs.anl.gov] Sent: Friday, April 17, 2015 2:57 PM To: Matthew Knepley Cc: Sun, Hui; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] install petsc with valgrind Or make sure valgrind is in your path > On Apr 17, 2015, at 4:56 PM, Matthew Knepley wrote: > > As I said, it cannot find valgrind in your path. Remove it or install valgrind. > > Matt > > On Fri, Apr 17, 2015 at 4:55 PM, Sun, Hui wrote: > Thank you, Matt and Barry. I was trying to run the following: > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > where NP=2, ND=32 > > And in the makefile I have specified the following: > include ${PETSC_DIR}/lib/petsc/conf/variables > include ${PETSC_DIR}/lib/petsc/conf/rules > include ${PETSC_DIR}/lib/petsc/conf/test > > Best, > Hui > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Friday, April 17, 2015 2:50 PM > To: Sun, Hui > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] install petsc with valgrind > > Please please please always show WHAT YOU RAN with the error message. Otherwise we just must guess > > > > On Apr 17, 2015, at 3:58 PM, Sun, Hui wrote: > > > > Thank you Barry. I just have it installed. And I was running a code, and got the following error message: > > > > HYDU_create_process (utils/launch/launch.c:75): execvp error on file valgrind: (No such file or directory) > > > > =================================================================================== > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > = PID 66059 RUNNING AT math178.ucsd.edu > > = EXIT CODE: 255 > > = CLEANING UP REMAINING PROCESSES > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > =================================================================================== > > > > What does this mean? > > > > Best, > > Hui > > > > ________________________________________ > > From: Barry Smith [bsmith at mcs.anl.gov] > > Sent: Friday, April 17, 2015 1:08 PM > > To: Sun, Hui > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] install petsc with valgrind > > > > The development copy of PETSc allows using --prefix with MPICH http://www.mcs.anl.gov/petsc/developers/index.html > > > > > > > >> On Apr 17, 2015, at 2:38 PM, Sun, Hui wrote: > >> > >> Hello, > >> > >> I want to install petsc with valgrind. And I read some documentations, which suggests me to do > >> ./configure --download-mpich --with-valgind-dir=? > >> > >> However, I also want to specify --prefix, but petsc doesn't allow me, because --download-mpich option does not work with --prefix install. > >> > >> Is there anyway that I can have petsc --prefix, together with valgrind? > >> > >> Best, > >> Hui > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From knepley at gmail.com Fri Apr 17 17:10:45 2015 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 17 Apr 2015 17:10:45 -0500 Subject: [petsc-users] install petsc with valgrind In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010FE343@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> <498A9304-AF3F-42CF-800A-295FB9C0897D@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE31E@XMAIL-MBX-BH1.AD.UCSD.EDU> <902FF397-BB49-4647-9E91-E11D2C851446@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE343@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: On Fri, Apr 17, 2015 at 5:09 PM, Sun, Hui wrote: > I'm not sure if I'm doing the right thing. But I have just > export PATH=$PATH:dir_to_valgind_bin > > And I rerun the command: > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my > ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > Then it seems to output something, for example: > ==66450== Invalid write of size 8 > ==66450== at 0x100055225: Schur_SetMatrix (in ./test) > ==66450== by 0x10004F218: StokesSetup (in ./test) > ==66450== by 0x1000DC582: main (in ./test) > ==66450== Address 0x7fff5f93fb28 is on thread 1's stack > ==66450== in frame #0, created by Schur_SetMatrix (???) > ==66450== > Yes, this worked, and it shows you have a memory overwrite in your code. You have no line numbers because you deleted the *.o files without running dsymutil. > And before those output, it has the following lines: > --66450-- run: /usr/bin/dsymutil "./test" > warning: (x86_64) /Users/hus003/projects/LS_BDRY/test/test.o unable to > open object file > warning: no debug symbols in executable (-arch x86_64) > --66450-- run: /usr/bin/dsymutil > "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > --66450-- run: /usr/bin/dsymutil > "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpicxx.12.dylib" > --66450-- run: /usr/bin/dsymutil > "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpifort.12.dylib" > --66450-- run: /usr/bin/dsymutil > "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpi.12.dylib" > --66450-- run: /usr/bin/dsymutil > "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libpmpi.12.dylib" > > Does it mean that it's working? But what does these two warnings mean? > These warnings mean that symbols are missing from those libraries, which is no surprise. matt > Best, > Hui > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Friday, April 17, 2015 2:57 PM > To: Matthew Knepley > Cc: Sun, Hui; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] install petsc with valgrind > > Or make sure valgrind is in your path > > > On Apr 17, 2015, at 4:56 PM, Matthew Knepley wrote: > > > > As I said, it cannot find valgrind in your path. Remove it or install > valgrind. > > > > Matt > > > > On Fri, Apr 17, 2015 at 4:55 PM, Sun, Hui wrote: > > Thank you, Matt and Barry. I was trying to run the following: > > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my > ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > > > where NP=2, ND=32 > > > > And in the makefile I have specified the following: > > include ${PETSC_DIR}/lib/petsc/conf/variables > > include ${PETSC_DIR}/lib/petsc/conf/rules > > include ${PETSC_DIR}/lib/petsc/conf/test > > > > Best, > > Hui > > > > > > ________________________________________ > > From: Barry Smith [bsmith at mcs.anl.gov] > > Sent: Friday, April 17, 2015 2:50 PM > > To: Sun, Hui > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] install petsc with valgrind > > > > Please please please always show WHAT YOU RAN with the error message. > Otherwise we just must guess > > > > > > > On Apr 17, 2015, at 3:58 PM, Sun, Hui wrote: > > > > > > Thank you Barry. I just have it installed. And I was running a code, > and got the following error message: > > > > > > HYDU_create_process (utils/launch/launch.c:75): execvp error on file > valgrind: (No such file or directory) > > > > > > > =================================================================================== > > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > > = PID 66059 RUNNING AT math178.ucsd.edu > > > = EXIT CODE: 255 > > > = CLEANING UP REMAINING PROCESSES > > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > > > =================================================================================== > > > > > > What does this mean? > > > > > > Best, > > > Hui > > > > > > ________________________________________ > > > From: Barry Smith [bsmith at mcs.anl.gov] > > > Sent: Friday, April 17, 2015 1:08 PM > > > To: Sun, Hui > > > Cc: petsc-users at mcs.anl.gov > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > The development copy of PETSc allows using --prefix with MPICH > http://www.mcs.anl.gov/petsc/developers/index.html > > > > > > > > > > > >> On Apr 17, 2015, at 2:38 PM, Sun, Hui wrote: > > >> > > >> Hello, > > >> > > >> I want to install petsc with valgrind. And I read some > documentations, which suggests me to do > > >> ./configure --download-mpich --with-valgind-dir=? > > >> > > >> However, I also want to specify --prefix, but petsc doesn't allow me, > because --download-mpich option does not work with --prefix install. > > >> > > >> Is there anyway that I can have petsc --prefix, together with > valgrind? > > >> > > >> Best, > > >> Hui > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hus003 at ucsd.edu Fri Apr 17 17:17:02 2015 From: hus003 at ucsd.edu (Sun, Hui) Date: Fri, 17 Apr 2015 22:17:02 +0000 Subject: [petsc-users] install petsc with valgrind In-Reply-To: References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> <498A9304-AF3F-42CF-800A-295FB9C0897D@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE31E@XMAIL-MBX-BH1.AD.UCSD.EDU> <902FF397-BB49-4647-9E91-E11D2C851446@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE343@XMAIL-MBX-BH1.AD.UCSD.EDU>, Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010FE355@XMAIL-MBX-BH1.AD.UCSD.EDU> Thank you Matt! Hui ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: Friday, April 17, 2015 3:10 PM To: Sun, Hui Cc: Barry Smith; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] install petsc with valgrind On Fri, Apr 17, 2015 at 5:09 PM, Sun, Hui > wrote: I'm not sure if I'm doing the right thing. But I have just export PATH=$PATH:dir_to_valgind_bin And I rerun the command: -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm Then it seems to output something, for example: ==66450== Invalid write of size 8 ==66450== at 0x100055225: Schur_SetMatrix (in ./test) ==66450== by 0x10004F218: StokesSetup (in ./test) ==66450== by 0x1000DC582: main (in ./test) ==66450== Address 0x7fff5f93fb28 is on thread 1's stack ==66450== in frame #0, created by Schur_SetMatrix (???) ==66450== Yes, this worked, and it shows you have a memory overwrite in your code. You have no line numbers because you deleted the *.o files without running dsymutil. And before those output, it has the following lines: --66450-- run: /usr/bin/dsymutil "./test" warning: (x86_64) /Users/hus003/projects/LS_BDRY/test/test.o unable to open object file warning: no debug symbols in executable (-arch x86_64) --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpicxx.12.dylib" --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpifort.12.dylib" --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpi.12.dylib" --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libpmpi.12.dylib" Does it mean that it's working? But what does these two warnings mean? These warnings mean that symbols are missing from those libraries, which is no surprise. matt Best, Hui ________________________________________ From: Barry Smith [bsmith at mcs.anl.gov] Sent: Friday, April 17, 2015 2:57 PM To: Matthew Knepley Cc: Sun, Hui; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] install petsc with valgrind Or make sure valgrind is in your path > On Apr 17, 2015, at 4:56 PM, Matthew Knepley > wrote: > > As I said, it cannot find valgrind in your path. Remove it or install valgrind. > > Matt > > On Fri, Apr 17, 2015 at 4:55 PM, Sun, Hui > wrote: > Thank you, Matt and Barry. I was trying to run the following: > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > where NP=2, ND=32 > > And in the makefile I have specified the following: > include ${PETSC_DIR}/lib/petsc/conf/variables > include ${PETSC_DIR}/lib/petsc/conf/rules > include ${PETSC_DIR}/lib/petsc/conf/test > > Best, > Hui > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Friday, April 17, 2015 2:50 PM > To: Sun, Hui > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] install petsc with valgrind > > Please please please always show WHAT YOU RAN with the error message. Otherwise we just must guess > > > > On Apr 17, 2015, at 3:58 PM, Sun, Hui > wrote: > > > > Thank you Barry. I just have it installed. And I was running a code, and got the following error message: > > > > HYDU_create_process (utils/launch/launch.c:75): execvp error on file valgrind: (No such file or directory) > > > > =================================================================================== > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > = PID 66059 RUNNING AT math178.ucsd.edu > > = EXIT CODE: 255 > > = CLEANING UP REMAINING PROCESSES > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > =================================================================================== > > > > What does this mean? > > > > Best, > > Hui > > > > ________________________________________ > > From: Barry Smith [bsmith at mcs.anl.gov] > > Sent: Friday, April 17, 2015 1:08 PM > > To: Sun, Hui > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] install petsc with valgrind > > > > The development copy of PETSc allows using --prefix with MPICH http://www.mcs.anl.gov/petsc/developers/index.html > > > > > > > >> On Apr 17, 2015, at 2:38 PM, Sun, Hui > wrote: > >> > >> Hello, > >> > >> I want to install petsc with valgrind. And I read some documentations, which suggests me to do > >> ./configure --download-mpich --with-valgind-dir=? > >> > >> However, I also want to specify --prefix, but petsc doesn't allow me, because --download-mpich option does not work with --prefix install. > >> > >> Is there anyway that I can have petsc --prefix, together with valgrind? > >> > >> Best, > >> Hui > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hus003 at ucsd.edu Fri Apr 17 17:26:56 2015 From: hus003 at ucsd.edu (Sun, Hui) Date: Fri, 17 Apr 2015 22:26:56 +0000 Subject: [petsc-users] install petsc with valgrind In-Reply-To: References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> <498A9304-AF3F-42CF-800A-295FB9C0897D@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE31E@XMAIL-MBX-BH1.AD.UCSD.EDU> <902FF397-BB49-4647-9E91-E11D2C851446@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE343@XMAIL-MBX-BH1.AD.UCSD.EDU>, Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010FE366@XMAIL-MBX-BH1.AD.UCSD.EDU> How do I run dsymutil so as to get the line numbers? ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: Friday, April 17, 2015 3:10 PM To: Sun, Hui Cc: Barry Smith; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] install petsc with valgrind On Fri, Apr 17, 2015 at 5:09 PM, Sun, Hui > wrote: I'm not sure if I'm doing the right thing. But I have just export PATH=$PATH:dir_to_valgind_bin And I rerun the command: -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm Then it seems to output something, for example: ==66450== Invalid write of size 8 ==66450== at 0x100055225: Schur_SetMatrix (in ./test) ==66450== by 0x10004F218: StokesSetup (in ./test) ==66450== by 0x1000DC582: main (in ./test) ==66450== Address 0x7fff5f93fb28 is on thread 1's stack ==66450== in frame #0, created by Schur_SetMatrix (???) ==66450== Yes, this worked, and it shows you have a memory overwrite in your code. You have no line numbers because you deleted the *.o files without running dsymutil. And before those output, it has the following lines: --66450-- run: /usr/bin/dsymutil "./test" warning: (x86_64) /Users/hus003/projects/LS_BDRY/test/test.o unable to open object file warning: no debug symbols in executable (-arch x86_64) --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpicxx.12.dylib" --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpifort.12.dylib" --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpi.12.dylib" --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libpmpi.12.dylib" Does it mean that it's working? But what does these two warnings mean? These warnings mean that symbols are missing from those libraries, which is no surprise. matt Best, Hui ________________________________________ From: Barry Smith [bsmith at mcs.anl.gov] Sent: Friday, April 17, 2015 2:57 PM To: Matthew Knepley Cc: Sun, Hui; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] install petsc with valgrind Or make sure valgrind is in your path > On Apr 17, 2015, at 4:56 PM, Matthew Knepley > wrote: > > As I said, it cannot find valgrind in your path. Remove it or install valgrind. > > Matt > > On Fri, Apr 17, 2015 at 4:55 PM, Sun, Hui > wrote: > Thank you, Matt and Barry. I was trying to run the following: > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > where NP=2, ND=32 > > And in the makefile I have specified the following: > include ${PETSC_DIR}/lib/petsc/conf/variables > include ${PETSC_DIR}/lib/petsc/conf/rules > include ${PETSC_DIR}/lib/petsc/conf/test > > Best, > Hui > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Friday, April 17, 2015 2:50 PM > To: Sun, Hui > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] install petsc with valgrind > > Please please please always show WHAT YOU RAN with the error message. Otherwise we just must guess > > > > On Apr 17, 2015, at 3:58 PM, Sun, Hui > wrote: > > > > Thank you Barry. I just have it installed. And I was running a code, and got the following error message: > > > > HYDU_create_process (utils/launch/launch.c:75): execvp error on file valgrind: (No such file or directory) > > > > =================================================================================== > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > = PID 66059 RUNNING AT math178.ucsd.edu > > = EXIT CODE: 255 > > = CLEANING UP REMAINING PROCESSES > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > =================================================================================== > > > > What does this mean? > > > > Best, > > Hui > > > > ________________________________________ > > From: Barry Smith [bsmith at mcs.anl.gov] > > Sent: Friday, April 17, 2015 1:08 PM > > To: Sun, Hui > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] install petsc with valgrind > > > > The development copy of PETSc allows using --prefix with MPICH http://www.mcs.anl.gov/petsc/developers/index.html > > > > > > > >> On Apr 17, 2015, at 2:38 PM, Sun, Hui > wrote: > >> > >> Hello, > >> > >> I want to install petsc with valgrind. And I read some documentations, which suggests me to do > >> ./configure --download-mpich --with-valgind-dir=? > >> > >> However, I also want to specify --prefix, but petsc doesn't allow me, because --download-mpich option does not work with --prefix install. > >> > >> Is there anyway that I can have petsc --prefix, together with valgrind? > >> > >> Best, > >> Hui > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Apr 17 17:29:05 2015 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 17 Apr 2015 17:29:05 -0500 Subject: [petsc-users] install petsc with valgrind In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010FE366@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> <498A9304-AF3F-42CF-800A-295FB9C0897D@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE31E@XMAIL-MBX-BH1.AD.UCSD.EDU> <902FF397-BB49-4647-9E91-E11D2C851446@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE343@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB010FE366@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: On Fri, Apr 17, 2015 at 5:26 PM, Sun, Hui wrote: > How do I run dsymutil so as to get the line numbers? > dsymutil ./test Matt > ------------------------------ > *From:* Matthew Knepley [knepley at gmail.com] > *Sent:* Friday, April 17, 2015 3:10 PM > *To:* Sun, Hui > *Cc:* Barry Smith; petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] install petsc with valgrind > > On Fri, Apr 17, 2015 at 5:09 PM, Sun, Hui wrote: > >> I'm not sure if I'm doing the right thing. But I have just >> export PATH=$PATH:dir_to_valgind_bin >> >> And I rerun the command: >> -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my >> ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm >> >> Then it seems to output something, for example: >> ==66450== Invalid write of size 8 >> ==66450== at 0x100055225: Schur_SetMatrix (in ./test) >> ==66450== by 0x10004F218: StokesSetup (in ./test) >> ==66450== by 0x1000DC582: main (in ./test) >> ==66450== Address 0x7fff5f93fb28 is on thread 1's stack >> ==66450== in frame #0, created by Schur_SetMatrix (???) >> ==66450== >> > > Yes, this worked, and it shows you have a memory overwrite in your code. > You have no line > numbers because you deleted the *.o files without running dsymutil. > > >> And before those output, it has the following lines: >> --66450-- run: /usr/bin/dsymutil "./test" >> warning: (x86_64) /Users/hus003/projects/LS_BDRY/test/test.o unable to >> open object file >> warning: no debug symbols in executable (-arch x86_64) >> --66450-- run: /usr/bin/dsymutil >> "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" >> --66450-- run: /usr/bin/dsymutil >> "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpicxx.12.dylib" >> --66450-- run: /usr/bin/dsymutil >> "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpifort.12.dylib" >> --66450-- run: /usr/bin/dsymutil >> "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpi.12.dylib" >> --66450-- run: /usr/bin/dsymutil >> "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libpmpi.12.dylib" >> >> Does it mean that it's working? But what does these two warnings mean? >> > > These warnings mean that symbols are missing from those libraries, which > is no surprise. > > matt > > >> Best, >> Hui >> >> >> ________________________________________ >> From: Barry Smith [bsmith at mcs.anl.gov] >> Sent: Friday, April 17, 2015 2:57 PM >> To: Matthew Knepley >> Cc: Sun, Hui; petsc-users at mcs.anl.gov >> Subject: Re: [petsc-users] install petsc with valgrind >> >> Or make sure valgrind is in your path >> >> > On Apr 17, 2015, at 4:56 PM, Matthew Knepley wrote: >> > >> > As I said, it cannot find valgrind in your path. Remove it or install >> valgrind. >> > >> > Matt >> > >> > On Fri, Apr 17, 2015 at 4:55 PM, Sun, Hui wrote: >> > Thank you, Matt and Barry. I was trying to run the following: >> > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my >> ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm >> > >> > where NP=2, ND=32 >> > >> > And in the makefile I have specified the following: >> > include ${PETSC_DIR}/lib/petsc/conf/variables >> > include ${PETSC_DIR}/lib/petsc/conf/rules >> > include ${PETSC_DIR}/lib/petsc/conf/test >> > >> > Best, >> > Hui >> > >> > >> > ________________________________________ >> > From: Barry Smith [bsmith at mcs.anl.gov] >> > Sent: Friday, April 17, 2015 2:50 PM >> > To: Sun, Hui >> > Cc: petsc-users at mcs.anl.gov >> > Subject: Re: [petsc-users] install petsc with valgrind >> > >> > Please please please always show WHAT YOU RAN with the error message. >> Otherwise we just must guess >> > >> > >> > > On Apr 17, 2015, at 3:58 PM, Sun, Hui wrote: >> > > >> > > Thank you Barry. I just have it installed. And I was running a code, >> and got the following error message: >> > > >> > > HYDU_create_process (utils/launch/launch.c:75): execvp error on file >> valgrind: (No such file or directory) >> > > >> > > >> =================================================================================== >> > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> > > = PID 66059 RUNNING AT math178.ucsd.edu >> > > = EXIT CODE: 255 >> > > = CLEANING UP REMAINING PROCESSES >> > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> > > >> =================================================================================== >> > > >> > > What does this mean? >> > > >> > > Best, >> > > Hui >> > > >> > > ________________________________________ >> > > From: Barry Smith [bsmith at mcs.anl.gov] >> > > Sent: Friday, April 17, 2015 1:08 PM >> > > To: Sun, Hui >> > > Cc: petsc-users at mcs.anl.gov >> > > Subject: Re: [petsc-users] install petsc with valgrind >> > > >> > > The development copy of PETSc allows using --prefix with MPICH >> http://www.mcs.anl.gov/petsc/developers/index.html >> > > >> > > >> > > >> > >> On Apr 17, 2015, at 2:38 PM, Sun, Hui wrote: >> > >> >> > >> Hello, >> > >> >> > >> I want to install petsc with valgrind. And I read some >> documentations, which suggests me to do >> > >> ./configure --download-mpich --with-valgind-dir=? >> > >> >> > >> However, I also want to specify --prefix, but petsc doesn't allow >> me, because --download-mpich option does not work with --prefix install. >> > >> >> > >> Is there anyway that I can have petsc --prefix, together with >> valgrind? >> > >> >> > >> Best, >> > >> Hui >> > > >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > -- Norbert Wiener >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Apr 17 17:29:31 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 17 Apr 2015 17:29:31 -0500 Subject: [petsc-users] install petsc with valgrind In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010FE366@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> <498A9304-AF3F-42CF-800A-295FB9C0897D@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE31E@XMAIL-MBX-BH1.AD.UCSD.EDU> <902FF397-BB49-4647-9E91-E11D2C851446@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE343@XMAIL-MBX-BH1.AD.UCSD.EDU>, <7501CC2B7BBCC44A92ECEEC316170ECB010FE366@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: Well - its best if the .o files are not deleted.. [then you don't have to run dsymutil]. One way to do it [if using PETSc makefiles] is: make test RM=/bin/true To run dsymutil - I think you need the .o files anyway - and run it as instructed below. > --66450-- run: /usr/bin/dsymutil "./test" Satish On Fri, 17 Apr 2015, Sun, Hui wrote: > How do I run dsymutil so as to get the line numbers? > > > ________________________________ > From: Matthew Knepley [knepley at gmail.com] > Sent: Friday, April 17, 2015 3:10 PM > To: Sun, Hui > Cc: Barry Smith; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] install petsc with valgrind > > On Fri, Apr 17, 2015 at 5:09 PM, Sun, Hui > wrote: > I'm not sure if I'm doing the right thing. But I have just > export PATH=$PATH:dir_to_valgind_bin > > And I rerun the command: > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > Then it seems to output something, for example: > ==66450== Invalid write of size 8 > ==66450== at 0x100055225: Schur_SetMatrix (in ./test) > ==66450== by 0x10004F218: StokesSetup (in ./test) > ==66450== by 0x1000DC582: main (in ./test) > ==66450== Address 0x7fff5f93fb28 is on thread 1's stack > ==66450== in frame #0, created by Schur_SetMatrix (???) > ==66450== > > Yes, this worked, and it shows you have a memory overwrite in your code. You have no line > numbers because you deleted the *.o files without running dsymutil. > > And before those output, it has the following lines: > --66450-- run: /usr/bin/dsymutil "./test" > warning: (x86_64) /Users/hus003/projects/LS_BDRY/test/test.o unable to open object file > warning: no debug symbols in executable (-arch x86_64) > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpicxx.12.dylib" > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpifort.12.dylib" > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpi.12.dylib" > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libpmpi.12.dylib" > > Does it mean that it's working? But what does these two warnings mean? > > These warnings mean that symbols are missing from those libraries, which is no surprise. > > matt > > Best, > Hui > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Friday, April 17, 2015 2:57 PM > To: Matthew Knepley > Cc: Sun, Hui; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] install petsc with valgrind > > Or make sure valgrind is in your path > > > On Apr 17, 2015, at 4:56 PM, Matthew Knepley > wrote: > > > > As I said, it cannot find valgrind in your path. Remove it or install valgrind. > > > > Matt > > > > On Fri, Apr 17, 2015 at 4:55 PM, Sun, Hui > wrote: > > Thank you, Matt and Barry. I was trying to run the following: > > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > > > where NP=2, ND=32 > > > > And in the makefile I have specified the following: > > include ${PETSC_DIR}/lib/petsc/conf/variables > > include ${PETSC_DIR}/lib/petsc/conf/rules > > include ${PETSC_DIR}/lib/petsc/conf/test > > > > Best, > > Hui > > > > > > ________________________________________ > > From: Barry Smith [bsmith at mcs.anl.gov] > > Sent: Friday, April 17, 2015 2:50 PM > > To: Sun, Hui > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] install petsc with valgrind > > > > Please please please always show WHAT YOU RAN with the error message. Otherwise we just must guess > > > > > > > On Apr 17, 2015, at 3:58 PM, Sun, Hui > wrote: > > > > > > Thank you Barry. I just have it installed. And I was running a code, and got the following error message: > > > > > > HYDU_create_process (utils/launch/launch.c:75): execvp error on file valgrind: (No such file or directory) > > > > > > =================================================================================== > > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > > = PID 66059 RUNNING AT math178.ucsd.edu > > > = EXIT CODE: 255 > > > = CLEANING UP REMAINING PROCESSES > > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > > =================================================================================== > > > > > > What does this mean? > > > > > > Best, > > > Hui > > > > > > ________________________________________ > > > From: Barry Smith [bsmith at mcs.anl.gov] > > > Sent: Friday, April 17, 2015 1:08 PM > > > To: Sun, Hui > > > Cc: petsc-users at mcs.anl.gov > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > The development copy of PETSc allows using --prefix with MPICH http://www.mcs.anl.gov/petsc/developers/index.html > > > > > > > > > > > >> On Apr 17, 2015, at 2:38 PM, Sun, Hui > wrote: > > >> > > >> Hello, > > >> > > >> I want to install petsc with valgrind. And I read some documentations, which suggests me to do > > >> ./configure --download-mpich --with-valgind-dir=? > > >> > > >> However, I also want to specify --prefix, but petsc doesn't allow me, because --download-mpich option does not work with --prefix install. > > >> > > >> Is there anyway that I can have petsc --prefix, together with valgrind? > > >> > > >> Best, > > >> Hui > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > From hus003 at ucsd.edu Fri Apr 17 17:37:14 2015 From: hus003 at ucsd.edu (Sun, Hui) Date: Fri, 17 Apr 2015 22:37:14 +0000 Subject: [petsc-users] install petsc with valgrind In-Reply-To: References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> <498A9304-AF3F-42CF-800A-295FB9C0897D@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE31E@XMAIL-MBX-BH1.AD.UCSD.EDU> <902FF397-BB49-4647-9E91-E11D2C851446@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE343@XMAIL-MBX-BH1.AD.UCSD.EDU>, <7501CC2B7BBCC44A92ECEEC316170ECB010FE366@XMAIL-MBX-BH1.AD.UCSD.EDU>, Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010FE384@XMAIL-MBX-BH1.AD.UCSD.EDU> Thank you Satish! It works perfect. Hui ________________________________________ From: Satish Balay [balay at mcs.anl.gov] Sent: Friday, April 17, 2015 3:29 PM To: Sun, Hui Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] install petsc with valgrind Well - its best if the .o files are not deleted.. [then you don't have to run dsymutil]. One way to do it [if using PETSc makefiles] is: make test RM=/bin/true To run dsymutil - I think you need the .o files anyway - and run it as instructed below. > --66450-- run: /usr/bin/dsymutil "./test" Satish On Fri, 17 Apr 2015, Sun, Hui wrote: > How do I run dsymutil so as to get the line numbers? > > > ________________________________ > From: Matthew Knepley [knepley at gmail.com] > Sent: Friday, April 17, 2015 3:10 PM > To: Sun, Hui > Cc: Barry Smith; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] install petsc with valgrind > > On Fri, Apr 17, 2015 at 5:09 PM, Sun, Hui > wrote: > I'm not sure if I'm doing the right thing. But I have just > export PATH=$PATH:dir_to_valgind_bin > > And I rerun the command: > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > Then it seems to output something, for example: > ==66450== Invalid write of size 8 > ==66450== at 0x100055225: Schur_SetMatrix (in ./test) > ==66450== by 0x10004F218: StokesSetup (in ./test) > ==66450== by 0x1000DC582: main (in ./test) > ==66450== Address 0x7fff5f93fb28 is on thread 1's stack > ==66450== in frame #0, created by Schur_SetMatrix (???) > ==66450== > > Yes, this worked, and it shows you have a memory overwrite in your code. You have no line > numbers because you deleted the *.o files without running dsymutil. > > And before those output, it has the following lines: > --66450-- run: /usr/bin/dsymutil "./test" > warning: (x86_64) /Users/hus003/projects/LS_BDRY/test/test.o unable to open object file > warning: no debug symbols in executable (-arch x86_64) > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpicxx.12.dylib" > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpifort.12.dylib" > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpi.12.dylib" > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libpmpi.12.dylib" > > Does it mean that it's working? But what does these two warnings mean? > > These warnings mean that symbols are missing from those libraries, which is no surprise. > > matt > > Best, > Hui > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Friday, April 17, 2015 2:57 PM > To: Matthew Knepley > Cc: Sun, Hui; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] install petsc with valgrind > > Or make sure valgrind is in your path > > > On Apr 17, 2015, at 4:56 PM, Matthew Knepley > wrote: > > > > As I said, it cannot find valgrind in your path. Remove it or install valgrind. > > > > Matt > > > > On Fri, Apr 17, 2015 at 4:55 PM, Sun, Hui > wrote: > > Thank you, Matt and Barry. I was trying to run the following: > > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > > > where NP=2, ND=32 > > > > And in the makefile I have specified the following: > > include ${PETSC_DIR}/lib/petsc/conf/variables > > include ${PETSC_DIR}/lib/petsc/conf/rules > > include ${PETSC_DIR}/lib/petsc/conf/test > > > > Best, > > Hui > > > > > > ________________________________________ > > From: Barry Smith [bsmith at mcs.anl.gov] > > Sent: Friday, April 17, 2015 2:50 PM > > To: Sun, Hui > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] install petsc with valgrind > > > > Please please please always show WHAT YOU RAN with the error message. Otherwise we just must guess > > > > > > > On Apr 17, 2015, at 3:58 PM, Sun, Hui > wrote: > > > > > > Thank you Barry. I just have it installed. And I was running a code, and got the following error message: > > > > > > HYDU_create_process (utils/launch/launch.c:75): execvp error on file valgrind: (No such file or directory) > > > > > > =================================================================================== > > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > > = PID 66059 RUNNING AT math178.ucsd.edu > > > = EXIT CODE: 255 > > > = CLEANING UP REMAINING PROCESSES > > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > > =================================================================================== > > > > > > What does this mean? > > > > > > Best, > > > Hui > > > > > > ________________________________________ > > > From: Barry Smith [bsmith at mcs.anl.gov] > > > Sent: Friday, April 17, 2015 1:08 PM > > > To: Sun, Hui > > > Cc: petsc-users at mcs.anl.gov > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > The development copy of PETSc allows using --prefix with MPICH http://www.mcs.anl.gov/petsc/developers/index.html > > > > > > > > > > > >> On Apr 17, 2015, at 2:38 PM, Sun, Hui > wrote: > > >> > > >> Hello, > > >> > > >> I want to install petsc with valgrind. And I read some documentations, which suggests me to do > > >> ./configure --download-mpich --with-valgind-dir=? > > >> > > >> However, I also want to specify --prefix, but petsc doesn't allow me, because --download-mpich option does not work with --prefix install. > > >> > > >> Is there anyway that I can have petsc --prefix, together with valgrind? > > >> > > >> Best, > > >> Hui > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > From hus003 at ucsd.edu Sat Apr 18 10:21:48 2015 From: hus003 at ucsd.edu (Sun, Hui) Date: Sat, 18 Apr 2015 15:21:48 +0000 Subject: [petsc-users] Petsc MPI Vec of integers Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010FE3DB@XMAIL-MBX-BH1.AD.UCSD.EDU> I want to have some parallel array of integers. Is there an automatic way in petsc to do MPI Vec of PetscInt? -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Apr 18 10:43:49 2015 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 18 Apr 2015 10:43:49 -0500 Subject: [petsc-users] Petsc MPI Vec of integers In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010FE3DB@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE3DB@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: On Sat, Apr 18, 2015 at 10:21 AM, Sun, Hui wrote: > I want to have some parallel array of integers. Is there an automatic > way in petsc to do MPI Vec of PetscInt? > Use an IS. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hus003 at ucsd.edu Sun Apr 19 09:56:47 2015 From: hus003 at ucsd.edu (Sun, Hui) Date: Sun, 19 Apr 2015 14:56:47 +0000 Subject: [petsc-users] install petsc with valgrind In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010FE384@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> <498A9304-AF3F-42CF-800A-295FB9C0897D@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE31E@XMAIL-MBX-BH1.AD.UCSD.EDU> <902FF397-BB49-4647-9E91-E11D2C851446@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE343@XMAIL-MBX-BH1.AD.UCSD.EDU>, <7501CC2B7BBCC44A92ECEEC316170ECB010FE366@XMAIL-MBX-BH1.AD.UCSD.EDU>, , <7501CC2B7BBCC44A92ECEEC316170ECB010FE384@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010FE440@XMAIL-MBX-BH1.AD.UCSD.EDU> I try to run -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n 2 ./test -mx 32 -my 32 -mz 32 It seems to be working, except that it gives me an error message: --21019-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" --21020-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" error: No such file or directory - old dSYM file cannot be overwritten: old: '/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib' new:'/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib.x86_64'. Why do I get this message? Do I need to get rid of it? Best, Hui ________________________________________ From: Sun, Hui Sent: Friday, April 17, 2015 3:37 PM To: petsc-users Cc: Matthew Knepley Subject: RE: [petsc-users] install petsc with valgrind Thank you Satish! It works perfect. Hui ________________________________________ From: Satish Balay [balay at mcs.anl.gov] Sent: Friday, April 17, 2015 3:29 PM To: Sun, Hui Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] install petsc with valgrind Well - its best if the .o files are not deleted.. [then you don't have to run dsymutil]. One way to do it [if using PETSc makefiles] is: make test RM=/bin/true To run dsymutil - I think you need the .o files anyway - and run it as instructed below. > --66450-- run: /usr/bin/dsymutil "./test" Satish On Fri, 17 Apr 2015, Sun, Hui wrote: > How do I run dsymutil so as to get the line numbers? > > > ________________________________ > From: Matthew Knepley [knepley at gmail.com] > Sent: Friday, April 17, 2015 3:10 PM > To: Sun, Hui > Cc: Barry Smith; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] install petsc with valgrind > > On Fri, Apr 17, 2015 at 5:09 PM, Sun, Hui > wrote: > I'm not sure if I'm doing the right thing. But I have just > export PATH=$PATH:dir_to_valgind_bin > > And I rerun the command: > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > Then it seems to output something, for example: > ==66450== Invalid write of size 8 > ==66450== at 0x100055225: Schur_SetMatrix (in ./test) > ==66450== by 0x10004F218: StokesSetup (in ./test) > ==66450== by 0x1000DC582: main (in ./test) > ==66450== Address 0x7fff5f93fb28 is on thread 1's stack > ==66450== in frame #0, created by Schur_SetMatrix (???) > ==66450== > > Yes, this worked, and it shows you have a memory overwrite in your code. You have no line > numbers because you deleted the *.o files without running dsymutil. > > And before those output, it has the following lines: > --66450-- run: /usr/bin/dsymutil "./test" > warning: (x86_64) /Users/hus003/projects/LS_BDRY/test/test.o unable to open object file > warning: no debug symbols in executable (-arch x86_64) > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpicxx.12.dylib" > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpifort.12.dylib" > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpi.12.dylib" > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libpmpi.12.dylib" > > Does it mean that it's working? But what does these two warnings mean? > > These warnings mean that symbols are missing from those libraries, which is no surprise. > > matt > > Best, > Hui > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Friday, April 17, 2015 2:57 PM > To: Matthew Knepley > Cc: Sun, Hui; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] install petsc with valgrind > > Or make sure valgrind is in your path > > > On Apr 17, 2015, at 4:56 PM, Matthew Knepley > wrote: > > > > As I said, it cannot find valgrind in your path. Remove it or install valgrind. > > > > Matt > > > > On Fri, Apr 17, 2015 at 4:55 PM, Sun, Hui > wrote: > > Thank you, Matt and Barry. I was trying to run the following: > > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > > > where NP=2, ND=32 > > > > And in the makefile I have specified the following: > > include ${PETSC_DIR}/lib/petsc/conf/variables > > include ${PETSC_DIR}/lib/petsc/conf/rules > > include ${PETSC_DIR}/lib/petsc/conf/test > > > > Best, > > Hui > > > > > > ________________________________________ > > From: Barry Smith [bsmith at mcs.anl.gov] > > Sent: Friday, April 17, 2015 2:50 PM > > To: Sun, Hui > > Cc: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] install petsc with valgrind > > > > Please please please always show WHAT YOU RAN with the error message. Otherwise we just must guess > > > > > > > On Apr 17, 2015, at 3:58 PM, Sun, Hui > wrote: > > > > > > Thank you Barry. I just have it installed. And I was running a code, and got the following error message: > > > > > > HYDU_create_process (utils/launch/launch.c:75): execvp error on file valgrind: (No such file or directory) > > > > > > =================================================================================== > > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > > = PID 66059 RUNNING AT math178.ucsd.edu > > > = EXIT CODE: 255 > > > = CLEANING UP REMAINING PROCESSES > > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > > =================================================================================== > > > > > > What does this mean? > > > > > > Best, > > > Hui > > > > > > ________________________________________ > > > From: Barry Smith [bsmith at mcs.anl.gov] > > > Sent: Friday, April 17, 2015 1:08 PM > > > To: Sun, Hui > > > Cc: petsc-users at mcs.anl.gov > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > The development copy of PETSc allows using --prefix with MPICH http://www.mcs.anl.gov/petsc/developers/index.html > > > > > > > > > > > >> On Apr 17, 2015, at 2:38 PM, Sun, Hui > wrote: > > >> > > >> Hello, > > >> > > >> I want to install petsc with valgrind. And I read some documentations, which suggests me to do > > >> ./configure --download-mpich --with-valgind-dir=? > > >> > > >> However, I also want to specify --prefix, but petsc doesn't allow me, because --download-mpich option does not work with --prefix install. > > >> > > >> Is there anyway that I can have petsc --prefix, together with valgrind? > > >> > > >> Best, > > >> Hui > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > From knepley at gmail.com Sun Apr 19 10:15:53 2015 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 19 Apr 2015 10:15:53 -0500 Subject: [petsc-users] install petsc with valgrind In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010FE440@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> <498A9304-AF3F-42CF-800A-295FB9C0897D@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE31E@XMAIL-MBX-BH1.AD.UCSD.EDU> <902FF397-BB49-4647-9E91-E11D2C851446@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE343@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB010FE366@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB010FE384@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB010FE440@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: On Sun, Apr 19, 2015 at 9:56 AM, Sun, Hui wrote: > I try to run > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n 2 ./test -mx 32 -my 32 -mz 32 > > It seems to be working, except that it gives me an error message: > > --21019-- run: /usr/bin/dsymutil > "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > --21020-- run: /usr/bin/dsymutil > "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > error: No such file or directory - old dSYM file cannot be overwritten: > old: > '/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib' > It sounds like you created this as root or something. Matt > > new:'/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib.x86_64'. > > Why do I get this message? Do I need to get rid of it? > > Best, > Hui > > > ________________________________________ > From: Sun, Hui > Sent: Friday, April 17, 2015 3:37 PM > To: petsc-users > Cc: Matthew Knepley > Subject: RE: [petsc-users] install petsc with valgrind > > Thank you Satish! It works perfect. > > Hui > > > ________________________________________ > From: Satish Balay [balay at mcs.anl.gov] > Sent: Friday, April 17, 2015 3:29 PM > To: Sun, Hui > Cc: Matthew Knepley; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] install petsc with valgrind > > Well - its best if the .o files are not deleted.. [then you don't > have to run dsymutil]. One way to do it [if using PETSc makefiles] is: > > make test RM=/bin/true > > To run dsymutil - I think you need the .o files anyway - and run it as > instructed below. > > > --66450-- run: /usr/bin/dsymutil "./test" > > Satish > > On Fri, 17 Apr 2015, Sun, Hui wrote: > > > How do I run dsymutil so as to get the line numbers? > > > > > > ________________________________ > > From: Matthew Knepley [knepley at gmail.com] > > Sent: Friday, April 17, 2015 3:10 PM > > To: Sun, Hui > > Cc: Barry Smith; petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] install petsc with valgrind > > > > On Fri, Apr 17, 2015 at 5:09 PM, Sun, Hui hus003 at ucsd.edu>> wrote: > > I'm not sure if I'm doing the right thing. But I have just > > export PATH=$PATH:dir_to_valgind_bin > > > > And I rerun the command: > > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my > ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > > > Then it seems to output something, for example: > > ==66450== Invalid write of size 8 > > ==66450== at 0x100055225: Schur_SetMatrix (in ./test) > > ==66450== by 0x10004F218: StokesSetup (in ./test) > > ==66450== by 0x1000DC582: main (in ./test) > > ==66450== Address 0x7fff5f93fb28 is on thread 1's stack > > ==66450== in frame #0, created by Schur_SetMatrix (???) > > ==66450== > > > > Yes, this worked, and it shows you have a memory overwrite in your code. > You have no line > > numbers because you deleted the *.o files without running dsymutil. > > > > And before those output, it has the following lines: > > --66450-- run: /usr/bin/dsymutil "./test" > > warning: (x86_64) /Users/hus003/projects/LS_BDRY/test/test.o unable to > open object file > > warning: no debug symbols in executable (-arch x86_64) > > --66450-- run: /usr/bin/dsymutil > "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > > --66450-- run: /usr/bin/dsymutil > "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpicxx.12.dylib" > > --66450-- run: /usr/bin/dsymutil > "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpifort.12.dylib" > > --66450-- run: /usr/bin/dsymutil > "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpi.12.dylib" > > --66450-- run: /usr/bin/dsymutil > "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libpmpi.12.dylib" > > > > Does it mean that it's working? But what does these two warnings mean? > > > > These warnings mean that symbols are missing from those libraries, which > is no surprise. > > > > matt > > > > Best, > > Hui > > > > > > ________________________________________ > > From: Barry Smith [bsmith at mcs.anl.gov] > > Sent: Friday, April 17, 2015 2:57 PM > > To: Matthew Knepley > > Cc: Sun, Hui; petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] install petsc with valgrind > > > > Or make sure valgrind is in your path > > > > > On Apr 17, 2015, at 4:56 PM, Matthew Knepley > wrote: > > > > > > As I said, it cannot find valgrind in your path. Remove it or install > valgrind. > > > > > > Matt > > > > > > On Fri, Apr 17, 2015 at 4:55 PM, Sun, Hui hus003 at ucsd.edu>> wrote: > > > Thank you, Matt and Barry. I was trying to run the following: > > > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} > -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > > > > > where NP=2, ND=32 > > > > > > And in the makefile I have specified the following: > > > include ${PETSC_DIR}/lib/petsc/conf/variables > > > include ${PETSC_DIR}/lib/petsc/conf/rules > > > include ${PETSC_DIR}/lib/petsc/conf/test > > > > > > Best, > > > Hui > > > > > > > > > ________________________________________ > > > From: Barry Smith [bsmith at mcs.anl.gov] > > > Sent: Friday, April 17, 2015 2:50 PM > > > To: Sun, Hui > > > Cc: petsc-users at mcs.anl.gov > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > Please please please always show WHAT YOU RAN with the error > message. Otherwise we just must guess > > > > > > > > > > On Apr 17, 2015, at 3:58 PM, Sun, Hui hus003 at ucsd.edu>> wrote: > > > > > > > > Thank you Barry. I just have it installed. And I was running a code, > and got the following error message: > > > > > > > > HYDU_create_process (utils/launch/launch.c:75): execvp error on file > valgrind: (No such file or directory) > > > > > > > > > =================================================================================== > > > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > > > = PID 66059 RUNNING AT math178.ucsd.edu > > > > = EXIT CODE: 255 > > > > = CLEANING UP REMAINING PROCESSES > > > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > > > > =================================================================================== > > > > > > > > What does this mean? > > > > > > > > Best, > > > > Hui > > > > > > > > ________________________________________ > > > > From: Barry Smith [bsmith at mcs.anl.gov] > > > > Sent: Friday, April 17, 2015 1:08 PM > > > > To: Sun, Hui > > > > Cc: petsc-users at mcs.anl.gov > > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > > > The development copy of PETSc allows using --prefix with MPICH > http://www.mcs.anl.gov/petsc/developers/index.html > > > > > > > > > > > > > > > >> On Apr 17, 2015, at 2:38 PM, Sun, Hui hus003 at ucsd.edu>> wrote: > > > >> > > > >> Hello, > > > >> > > > >> I want to install petsc with valgrind. And I read some > documentations, which suggests me to do > > > >> ./configure --download-mpich --with-valgind-dir=? > > > >> > > > >> However, I also want to specify --prefix, but petsc doesn't allow > me, because --download-mpich option does not work with --prefix install. > > > >> > > > >> Is there anyway that I can have petsc --prefix, together with > valgrind? > > > >> > > > >> Best, > > > >> Hui > > > > > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > > -- Norbert Wiener > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hus003 at ucsd.edu Sun Apr 19 10:23:44 2015 From: hus003 at ucsd.edu (Sun, Hui) Date: Sun, 19 Apr 2015 15:23:44 +0000 Subject: [petsc-users] install petsc with valgrind In-Reply-To: References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> <498A9304-AF3F-42CF-800A-295FB9C0897D@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE31E@XMAIL-MBX-BH1.AD.UCSD.EDU> <902FF397-BB49-4647-9E91-E11D2C851446@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE343@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB010FE366@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB010FE384@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB010FE440@XMAIL-MBX-BH1.AD.UCSD.EDU>, Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010FE460@XMAIL-MBX-BH1.AD.UCSD.EDU> I didn't use a root account. ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: Sunday, April 19, 2015 8:15 AM To: Sun, Hui Cc: petsc-users Subject: Re: [petsc-users] install petsc with valgrind On Sun, Apr 19, 2015 at 9:56 AM, Sun, Hui > wrote: I try to run -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n 2 ./test -mx 32 -my 32 -mz 32 It seems to be working, except that it gives me an error message: --21019-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" --21020-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" error: No such file or directory - old dSYM file cannot be overwritten: old: '/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib' It sounds like you created this as root or something. Matt new:'/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib.x86_64'. Why do I get this message? Do I need to get rid of it? Best, Hui ________________________________________ From: Sun, Hui Sent: Friday, April 17, 2015 3:37 PM To: petsc-users Cc: Matthew Knepley Subject: RE: [petsc-users] install petsc with valgrind Thank you Satish! It works perfect. Hui ________________________________________ From: Satish Balay [balay at mcs.anl.gov] Sent: Friday, April 17, 2015 3:29 PM To: Sun, Hui Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] install petsc with valgrind Well - its best if the .o files are not deleted.. [then you don't have to run dsymutil]. One way to do it [if using PETSc makefiles] is: make test RM=/bin/true To run dsymutil - I think you need the .o files anyway - and run it as instructed below. > --66450-- run: /usr/bin/dsymutil "./test" Satish On Fri, 17 Apr 2015, Sun, Hui wrote: > How do I run dsymutil so as to get the line numbers? > > > ________________________________ > From: Matthew Knepley [knepley at gmail.com] > Sent: Friday, April 17, 2015 3:10 PM > To: Sun, Hui > Cc: Barry Smith; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] install petsc with valgrind > > On Fri, Apr 17, 2015 at 5:09 PM, Sun, Hui >> wrote: > I'm not sure if I'm doing the right thing. But I have just > export PATH=$PATH:dir_to_valgind_bin > > And I rerun the command: > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > Then it seems to output something, for example: > ==66450== Invalid write of size 8 > ==66450== at 0x100055225: Schur_SetMatrix (in ./test) > ==66450== by 0x10004F218: StokesSetup (in ./test) > ==66450== by 0x1000DC582: main (in ./test) > ==66450== Address 0x7fff5f93fb28 is on thread 1's stack > ==66450== in frame #0, created by Schur_SetMatrix (???) > ==66450== > > Yes, this worked, and it shows you have a memory overwrite in your code. You have no line > numbers because you deleted the *.o files without running dsymutil. > > And before those output, it has the following lines: > --66450-- run: /usr/bin/dsymutil "./test" > warning: (x86_64) /Users/hus003/projects/LS_BDRY/test/test.o unable to open object file > warning: no debug symbols in executable (-arch x86_64) > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpicxx.12.dylib" > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpifort.12.dylib" > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpi.12.dylib" > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libpmpi.12.dylib" > > Does it mean that it's working? But what does these two warnings mean? > > These warnings mean that symbols are missing from those libraries, which is no surprise. > > matt > > Best, > Hui > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov>] > Sent: Friday, April 17, 2015 2:57 PM > To: Matthew Knepley > Cc: Sun, Hui; petsc-users at mcs.anl.gov> > Subject: Re: [petsc-users] install petsc with valgrind > > Or make sure valgrind is in your path > > > On Apr 17, 2015, at 4:56 PM, Matthew Knepley >> wrote: > > > > As I said, it cannot find valgrind in your path. Remove it or install valgrind. > > > > Matt > > > > On Fri, Apr 17, 2015 at 4:55 PM, Sun, Hui >> wrote: > > Thank you, Matt and Barry. I was trying to run the following: > > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > > > where NP=2, ND=32 > > > > And in the makefile I have specified the following: > > include ${PETSC_DIR}/lib/petsc/conf/variables > > include ${PETSC_DIR}/lib/petsc/conf/rules > > include ${PETSC_DIR}/lib/petsc/conf/test > > > > Best, > > Hui > > > > > > ________________________________________ > > From: Barry Smith [bsmith at mcs.anl.gov>] > > Sent: Friday, April 17, 2015 2:50 PM > > To: Sun, Hui > > Cc: petsc-users at mcs.anl.gov> > > Subject: Re: [petsc-users] install petsc with valgrind > > > > Please please please always show WHAT YOU RAN with the error message. Otherwise we just must guess > > > > > > > On Apr 17, 2015, at 3:58 PM, Sun, Hui >> wrote: > > > > > > Thank you Barry. I just have it installed. And I was running a code, and got the following error message: > > > > > > HYDU_create_process (utils/launch/launch.c:75): execvp error on file valgrind: (No such file or directory) > > > > > > =================================================================================== > > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > > = PID 66059 RUNNING AT math178.ucsd.edu > > > = EXIT CODE: 255 > > > = CLEANING UP REMAINING PROCESSES > > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > > =================================================================================== > > > > > > What does this mean? > > > > > > Best, > > > Hui > > > > > > ________________________________________ > > > From: Barry Smith [bsmith at mcs.anl.gov>] > > > Sent: Friday, April 17, 2015 1:08 PM > > > To: Sun, Hui > > > Cc: petsc-users at mcs.anl.gov> > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > The development copy of PETSc allows using --prefix with MPICH http://www.mcs.anl.gov/petsc/developers/index.html > > > > > > > > > > > >> On Apr 17, 2015, at 2:38 PM, Sun, Hui >> wrote: > > >> > > >> Hello, > > >> > > >> I want to install petsc with valgrind. And I read some documentations, which suggests me to do > > >> ./configure --download-mpich --with-valgind-dir=? > > >> > > >> However, I also want to specify --prefix, but petsc doesn't allow me, because --download-mpich option does not work with --prefix install. > > >> > > >> Is there anyway that I can have petsc --prefix, together with valgrind? > > >> > > >> Best, > > >> Hui > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sun Apr 19 12:31:21 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 19 Apr 2015 12:31:21 -0500 Subject: [petsc-users] install petsc with valgrind In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010FE460@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> <498A9304-AF3F-42CF-800A-295FB9C0897D@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE31E@XMAIL-MBX-BH1.AD.UCSD.EDU> <902FF397-BB49-4647-9E91-E11D2C851446@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE343@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB010FE366@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB010FE384@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB010FE440@XMAIL-MBX-BH1.AD.UCSD.EDU>, <7501CC2B7BBCC44A92ECEEC316170ECB010FE460@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: What do you have for: ls -l /Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib ls -l /Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib.x86_64 ls -l /Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib Satish On Sun, 19 Apr 2015, Sun, Hui wrote: > I didn't use a root account. > > ________________________________ > From: Matthew Knepley [knepley at gmail.com] > Sent: Sunday, April 19, 2015 8:15 AM > To: Sun, Hui > Cc: petsc-users > Subject: Re: [petsc-users] install petsc with valgrind > > On Sun, Apr 19, 2015 at 9:56 AM, Sun, Hui > wrote: > I try to run > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n 2 ./test -mx 32 -my 32 -mz 32 > > It seems to be working, except that it gives me an error message: > > --21019-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > --21020-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > error: No such file or directory - old dSYM file cannot be overwritten: old: '/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib' > > It sounds like you created this as root or something. > > Matt > > new:'/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib.x86_64'. > > Why do I get this message? Do I need to get rid of it? > > Best, > Hui > > > ________________________________________ > From: Sun, Hui > Sent: Friday, April 17, 2015 3:37 PM > To: petsc-users > Cc: Matthew Knepley > Subject: RE: [petsc-users] install petsc with valgrind > > Thank you Satish! It works perfect. > > Hui > > > ________________________________________ > From: Satish Balay [balay at mcs.anl.gov] > Sent: Friday, April 17, 2015 3:29 PM > To: Sun, Hui > Cc: Matthew Knepley; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] install petsc with valgrind > > Well - its best if the .o files are not deleted.. [then you don't > have to run dsymutil]. One way to do it [if using PETSc makefiles] is: > > make test RM=/bin/true > > To run dsymutil - I think you need the .o files anyway - and run it as instructed below. > > > --66450-- run: /usr/bin/dsymutil "./test" > > Satish > > On Fri, 17 Apr 2015, Sun, Hui wrote: > > > How do I run dsymutil so as to get the line numbers? > > > > > > ________________________________ > > From: Matthew Knepley [knepley at gmail.com] > > Sent: Friday, April 17, 2015 3:10 PM > > To: Sun, Hui > > Cc: Barry Smith; petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] install petsc with valgrind > > > > On Fri, Apr 17, 2015 at 5:09 PM, Sun, Hui >> wrote: > > I'm not sure if I'm doing the right thing. But I have just > > export PATH=$PATH:dir_to_valgind_bin > > > > And I rerun the command: > > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > > > Then it seems to output something, for example: > > ==66450== Invalid write of size 8 > > ==66450== at 0x100055225: Schur_SetMatrix (in ./test) > > ==66450== by 0x10004F218: StokesSetup (in ./test) > > ==66450== by 0x1000DC582: main (in ./test) > > ==66450== Address 0x7fff5f93fb28 is on thread 1's stack > > ==66450== in frame #0, created by Schur_SetMatrix (???) > > ==66450== > > > > Yes, this worked, and it shows you have a memory overwrite in your code. You have no line > > numbers because you deleted the *.o files without running dsymutil. > > > > And before those output, it has the following lines: > > --66450-- run: /usr/bin/dsymutil "./test" > > warning: (x86_64) /Users/hus003/projects/LS_BDRY/test/test.o unable to open object file > > warning: no debug symbols in executable (-arch x86_64) > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpicxx.12.dylib" > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpifort.12.dylib" > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpi.12.dylib" > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libpmpi.12.dylib" > > > > Does it mean that it's working? But what does these two warnings mean? > > > > These warnings mean that symbols are missing from those libraries, which is no surprise. > > > > matt > > > > Best, > > Hui > > > > > > ________________________________________ > > From: Barry Smith [bsmith at mcs.anl.gov>] > > Sent: Friday, April 17, 2015 2:57 PM > > To: Matthew Knepley > > Cc: Sun, Hui; petsc-users at mcs.anl.gov> > > Subject: Re: [petsc-users] install petsc with valgrind > > > > Or make sure valgrind is in your path > > > > > On Apr 17, 2015, at 4:56 PM, Matthew Knepley >> wrote: > > > > > > As I said, it cannot find valgrind in your path. Remove it or install valgrind. > > > > > > Matt > > > > > > On Fri, Apr 17, 2015 at 4:55 PM, Sun, Hui >> wrote: > > > Thank you, Matt and Barry. I was trying to run the following: > > > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > > > > > where NP=2, ND=32 > > > > > > And in the makefile I have specified the following: > > > include ${PETSC_DIR}/lib/petsc/conf/variables > > > include ${PETSC_DIR}/lib/petsc/conf/rules > > > include ${PETSC_DIR}/lib/petsc/conf/test > > > > > > Best, > > > Hui > > > > > > > > > ________________________________________ > > > From: Barry Smith [bsmith at mcs.anl.gov>] > > > Sent: Friday, April 17, 2015 2:50 PM > > > To: Sun, Hui > > > Cc: petsc-users at mcs.anl.gov> > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > Please please please always show WHAT YOU RAN with the error message. Otherwise we just must guess > > > > > > > > > > On Apr 17, 2015, at 3:58 PM, Sun, Hui >> wrote: > > > > > > > > Thank you Barry. I just have it installed. And I was running a code, and got the following error message: > > > > > > > > HYDU_create_process (utils/launch/launch.c:75): execvp error on file valgrind: (No such file or directory) > > > > > > > > =================================================================================== > > > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > > > = PID 66059 RUNNING AT math178.ucsd.edu > > > > = EXIT CODE: 255 > > > > = CLEANING UP REMAINING PROCESSES > > > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > > > =================================================================================== > > > > > > > > What does this mean? > > > > > > > > Best, > > > > Hui > > > > > > > > ________________________________________ > > > > From: Barry Smith [bsmith at mcs.anl.gov>] > > > > Sent: Friday, April 17, 2015 1:08 PM > > > > To: Sun, Hui > > > > Cc: petsc-users at mcs.anl.gov> > > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > > > The development copy of PETSc allows using --prefix with MPICH http://www.mcs.anl.gov/petsc/developers/index.html > > > > > > > > > > > > > > > >> On Apr 17, 2015, at 2:38 PM, Sun, Hui >> wrote: > > > >> > > > >> Hello, > > > >> > > > >> I want to install petsc with valgrind. And I read some documentations, which suggests me to do > > > >> ./configure --download-mpich --with-valgind-dir=? > > > >> > > > >> However, I also want to specify --prefix, but petsc doesn't allow me, because --download-mpich option does not work with --prefix install. > > > >> > > > >> Is there anyway that I can have petsc --prefix, together with valgrind? > > > >> > > > >> Best, > > > >> Hui > > > > > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > -- Norbert Wiener > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > From balay at mcs.anl.gov Sun Apr 19 12:47:15 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 19 Apr 2015 12:47:15 -0500 Subject: [petsc-users] install petsc with valgrind In-Reply-To: References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> <498A9304-AF3F-42CF-800A-295FB9C0897D@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE31E@XMAIL-MBX-BH1.AD.UCSD.EDU> <902FF397-BB49-4647-9E91-E11D2C851446@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE343@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB010FE366@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB010FE384@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB010FE440@XMAIL-MBX-BH1.AD.UCSD.EDU>, <7501CC2B7BBCC44A92ECEEC316170ECB010FE460@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: BTW: can you also send your configure.log [perhaps compress it and attach] Satish On Sun, 19 Apr 2015, Satish Balay wrote: > What do you have for: > > ls -l /Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib > > ls -l /Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib.x86_64 > > ls -l /Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib > > Satish > > On Sun, 19 Apr 2015, Sun, Hui wrote: > > > I didn't use a root account. > > > > ________________________________ > > From: Matthew Knepley [knepley at gmail.com] > > Sent: Sunday, April 19, 2015 8:15 AM > > To: Sun, Hui > > Cc: petsc-users > > Subject: Re: [petsc-users] install petsc with valgrind > > > > On Sun, Apr 19, 2015 at 9:56 AM, Sun, Hui > wrote: > > I try to run > > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n 2 ./test -mx 32 -my 32 -mz 32 > > > > It seems to be working, except that it gives me an error message: > > > > --21019-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > > --21020-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > > error: No such file or directory - old dSYM file cannot be overwritten: old: '/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib' > > > > It sounds like you created this as root or something. > > > > Matt > > > > new:'/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib.x86_64'. > > > > Why do I get this message? Do I need to get rid of it? > > > > Best, > > Hui > > > > > > ________________________________________ > > From: Sun, Hui > > Sent: Friday, April 17, 2015 3:37 PM > > To: petsc-users > > Cc: Matthew Knepley > > Subject: RE: [petsc-users] install petsc with valgrind > > > > Thank you Satish! It works perfect. > > > > Hui > > > > > > ________________________________________ > > From: Satish Balay [balay at mcs.anl.gov] > > Sent: Friday, April 17, 2015 3:29 PM > > To: Sun, Hui > > Cc: Matthew Knepley; petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] install petsc with valgrind > > > > Well - its best if the .o files are not deleted.. [then you don't > > have to run dsymutil]. One way to do it [if using PETSc makefiles] is: > > > > make test RM=/bin/true > > > > To run dsymutil - I think you need the .o files anyway - and run it as instructed below. > > > > > --66450-- run: /usr/bin/dsymutil "./test" > > > > Satish > > > > On Fri, 17 Apr 2015, Sun, Hui wrote: > > > > > How do I run dsymutil so as to get the line numbers? > > > > > > > > > ________________________________ > > > From: Matthew Knepley [knepley at gmail.com] > > > Sent: Friday, April 17, 2015 3:10 PM > > > To: Sun, Hui > > > Cc: Barry Smith; petsc-users at mcs.anl.gov > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > On Fri, Apr 17, 2015 at 5:09 PM, Sun, Hui >> wrote: > > > I'm not sure if I'm doing the right thing. But I have just > > > export PATH=$PATH:dir_to_valgind_bin > > > > > > And I rerun the command: > > > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > > > > > Then it seems to output something, for example: > > > ==66450== Invalid write of size 8 > > > ==66450== at 0x100055225: Schur_SetMatrix (in ./test) > > > ==66450== by 0x10004F218: StokesSetup (in ./test) > > > ==66450== by 0x1000DC582: main (in ./test) > > > ==66450== Address 0x7fff5f93fb28 is on thread 1's stack > > > ==66450== in frame #0, created by Schur_SetMatrix (???) > > > ==66450== > > > > > > Yes, this worked, and it shows you have a memory overwrite in your code. You have no line > > > numbers because you deleted the *.o files without running dsymutil. > > > > > > And before those output, it has the following lines: > > > --66450-- run: /usr/bin/dsymutil "./test" > > > warning: (x86_64) /Users/hus003/projects/LS_BDRY/test/test.o unable to open object file > > > warning: no debug symbols in executable (-arch x86_64) > > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpicxx.12.dylib" > > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpifort.12.dylib" > > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpi.12.dylib" > > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libpmpi.12.dylib" > > > > > > Does it mean that it's working? But what does these two warnings mean? > > > > > > These warnings mean that symbols are missing from those libraries, which is no surprise. > > > > > > matt > > > > > > Best, > > > Hui > > > > > > > > > ________________________________________ > > > From: Barry Smith [bsmith at mcs.anl.gov>] > > > Sent: Friday, April 17, 2015 2:57 PM > > > To: Matthew Knepley > > > Cc: Sun, Hui; petsc-users at mcs.anl.gov> > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > Or make sure valgrind is in your path > > > > > > > On Apr 17, 2015, at 4:56 PM, Matthew Knepley >> wrote: > > > > > > > > As I said, it cannot find valgrind in your path. Remove it or install valgrind. > > > > > > > > Matt > > > > > > > > On Fri, Apr 17, 2015 at 4:55 PM, Sun, Hui >> wrote: > > > > Thank you, Matt and Barry. I was trying to run the following: > > > > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > > > > > > > where NP=2, ND=32 > > > > > > > > And in the makefile I have specified the following: > > > > include ${PETSC_DIR}/lib/petsc/conf/variables > > > > include ${PETSC_DIR}/lib/petsc/conf/rules > > > > include ${PETSC_DIR}/lib/petsc/conf/test > > > > > > > > Best, > > > > Hui > > > > > > > > > > > > ________________________________________ > > > > From: Barry Smith [bsmith at mcs.anl.gov>] > > > > Sent: Friday, April 17, 2015 2:50 PM > > > > To: Sun, Hui > > > > Cc: petsc-users at mcs.anl.gov> > > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > > > Please please please always show WHAT YOU RAN with the error message. Otherwise we just must guess > > > > > > > > > > > > > On Apr 17, 2015, at 3:58 PM, Sun, Hui >> wrote: > > > > > > > > > > Thank you Barry. I just have it installed. And I was running a code, and got the following error message: > > > > > > > > > > HYDU_create_process (utils/launch/launch.c:75): execvp error on file valgrind: (No such file or directory) > > > > > > > > > > =================================================================================== > > > > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > > > > = PID 66059 RUNNING AT math178.ucsd.edu > > > > > = EXIT CODE: 255 > > > > > = CLEANING UP REMAINING PROCESSES > > > > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > > > > =================================================================================== > > > > > > > > > > What does this mean? > > > > > > > > > > Best, > > > > > Hui > > > > > > > > > > ________________________________________ > > > > > From: Barry Smith [bsmith at mcs.anl.gov>] > > > > > Sent: Friday, April 17, 2015 1:08 PM > > > > > To: Sun, Hui > > > > > Cc: petsc-users at mcs.anl.gov> > > > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > > > > > The development copy of PETSc allows using --prefix with MPICH http://www.mcs.anl.gov/petsc/developers/index.html > > > > > > > > > > > > > > > > > > > >> On Apr 17, 2015, at 2:38 PM, Sun, Hui >> wrote: > > > > >> > > > > >> Hello, > > > > >> > > > > >> I want to install petsc with valgrind. And I read some documentations, which suggests me to do > > > > >> ./configure --download-mpich --with-valgind-dir=? > > > > >> > > > > >> However, I also want to specify --prefix, but petsc doesn't allow me, because --download-mpich option does not work with --prefix install. > > > > >> > > > > >> Is there anyway that I can have petsc --prefix, together with valgrind? > > > > >> > > > > >> Best, > > > > >> Hui > > > > > > > > > > > > > > > > > > > > > -- > > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > > -- Norbert Wiener > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > -- Norbert Wiener > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > From balay at mcs.anl.gov Sun Apr 19 14:29:46 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 19 Apr 2015 14:29:46 -0500 Subject: [petsc-users] install petsc with valgrind In-Reply-To: References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> <498A9304-AF3F-42CF-800A-295FB9C0897D@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE31E@XMAIL-MBX-BH1.AD.UCSD.EDU> <902FF397-BB49-4647-9E91-E11D2C851446@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE343@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB010FE366@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB010FE384@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB010FE440@XMAIL-MBX-BH1.AD.UCSD.EDU>, <7501CC2B7BBCC44A92ECEEC316170ECB010FE460@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: Ok - petscmpiexec does a bunch of things with valgrind - including --dsymutil=yes option [and then later deleting *.dSYM files]. I don't know why they --dsymutil=yes is required.. [and rm in ${PETSC_DIR}/${PETSC_ARCH} looks wrong.] Perhaps you can call valgrind directly [without the --dsymutil=yes option] .. valgrind -q --tool=memcheck --num-callers=20 --track-origins=yes -n2 ./exe Satish ----- if ( `uname` == 'Darwin') then set VALGRIND_OPTIONS="${VALGRIND_OPTIONS} --dsymutil=yes" endif $qrun -n $np ${VALGRIND_CMD} ${VALGRIND_OPTIONS} $* if ( `uname` == 'Darwin') then rm -rf *.dSYM rm -rf ${PETSC_DIR}/${PETSC_ARCH}/lib/lib*.dSYM endif On Sun, 19 Apr 2015, Satish Balay wrote: > BTW: can you also send your configure.log [perhaps compress it and attach] > > Satish > > On Sun, 19 Apr 2015, Satish Balay wrote: > > > What do you have for: > > > > ls -l /Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib > > > > ls -l /Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib.x86_64 > > > > ls -l /Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib > > > > Satish > > > > On Sun, 19 Apr 2015, Sun, Hui wrote: > > > > > I didn't use a root account. > > > > > > ________________________________ > > > From: Matthew Knepley [knepley at gmail.com] > > > Sent: Sunday, April 19, 2015 8:15 AM > > > To: Sun, Hui > > > Cc: petsc-users > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > On Sun, Apr 19, 2015 at 9:56 AM, Sun, Hui > wrote: > > > I try to run > > > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n 2 ./test -mx 32 -my 32 -mz 32 > > > > > > It seems to be working, except that it gives me an error message: > > > > > > --21019-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > > > --21020-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > > > error: No such file or directory - old dSYM file cannot be overwritten: old: '/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib' > > > > > > It sounds like you created this as root or something. > > > > > > Matt > > > > > > new:'/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib.x86_64'. > > > > > > Why do I get this message? Do I need to get rid of it? > > > > > > Best, > > > Hui > > > > > > > > > ________________________________________ > > > From: Sun, Hui > > > Sent: Friday, April 17, 2015 3:37 PM > > > To: petsc-users > > > Cc: Matthew Knepley > > > Subject: RE: [petsc-users] install petsc with valgrind > > > > > > Thank you Satish! It works perfect. > > > > > > Hui > > > > > > > > > ________________________________________ > > > From: Satish Balay [balay at mcs.anl.gov] > > > Sent: Friday, April 17, 2015 3:29 PM > > > To: Sun, Hui > > > Cc: Matthew Knepley; petsc-users at mcs.anl.gov > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > Well - its best if the .o files are not deleted.. [then you don't > > > have to run dsymutil]. One way to do it [if using PETSc makefiles] is: > > > > > > make test RM=/bin/true > > > > > > To run dsymutil - I think you need the .o files anyway - and run it as instructed below. > > > > > > > --66450-- run: /usr/bin/dsymutil "./test" > > > > > > Satish > > > > > > On Fri, 17 Apr 2015, Sun, Hui wrote: > > > > > > > How do I run dsymutil so as to get the line numbers? > > > > > > > > > > > > ________________________________ > > > > From: Matthew Knepley [knepley at gmail.com] > > > > Sent: Friday, April 17, 2015 3:10 PM > > > > To: Sun, Hui > > > > Cc: Barry Smith; petsc-users at mcs.anl.gov > > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > > > On Fri, Apr 17, 2015 at 5:09 PM, Sun, Hui >> wrote: > > > > I'm not sure if I'm doing the right thing. But I have just > > > > export PATH=$PATH:dir_to_valgind_bin > > > > > > > > And I rerun the command: > > > > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > > > > > > > Then it seems to output something, for example: > > > > ==66450== Invalid write of size 8 > > > > ==66450== at 0x100055225: Schur_SetMatrix (in ./test) > > > > ==66450== by 0x10004F218: StokesSetup (in ./test) > > > > ==66450== by 0x1000DC582: main (in ./test) > > > > ==66450== Address 0x7fff5f93fb28 is on thread 1's stack > > > > ==66450== in frame #0, created by Schur_SetMatrix (???) > > > > ==66450== > > > > > > > > Yes, this worked, and it shows you have a memory overwrite in your code. You have no line > > > > numbers because you deleted the *.o files without running dsymutil. > > > > > > > > And before those output, it has the following lines: > > > > --66450-- run: /usr/bin/dsymutil "./test" > > > > warning: (x86_64) /Users/hus003/projects/LS_BDRY/test/test.o unable to open object file > > > > warning: no debug symbols in executable (-arch x86_64) > > > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > > > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpicxx.12.dylib" > > > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpifort.12.dylib" > > > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpi.12.dylib" > > > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libpmpi.12.dylib" > > > > > > > > Does it mean that it's working? But what does these two warnings mean? > > > > > > > > These warnings mean that symbols are missing from those libraries, which is no surprise. > > > > > > > > matt > > > > > > > > Best, > > > > Hui > > > > > > > > > > > > ________________________________________ > > > > From: Barry Smith [bsmith at mcs.anl.gov>] > > > > Sent: Friday, April 17, 2015 2:57 PM > > > > To: Matthew Knepley > > > > Cc: Sun, Hui; petsc-users at mcs.anl.gov> > > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > > > Or make sure valgrind is in your path > > > > > > > > > On Apr 17, 2015, at 4:56 PM, Matthew Knepley >> wrote: > > > > > > > > > > As I said, it cannot find valgrind in your path. Remove it or install valgrind. > > > > > > > > > > Matt > > > > > > > > > > On Fri, Apr 17, 2015 at 4:55 PM, Sun, Hui >> wrote: > > > > > Thank you, Matt and Barry. I was trying to run the following: > > > > > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > > > > > > > > > where NP=2, ND=32 > > > > > > > > > > And in the makefile I have specified the following: > > > > > include ${PETSC_DIR}/lib/petsc/conf/variables > > > > > include ${PETSC_DIR}/lib/petsc/conf/rules > > > > > include ${PETSC_DIR}/lib/petsc/conf/test > > > > > > > > > > Best, > > > > > Hui > > > > > > > > > > > > > > > ________________________________________ > > > > > From: Barry Smith [bsmith at mcs.anl.gov>] > > > > > Sent: Friday, April 17, 2015 2:50 PM > > > > > To: Sun, Hui > > > > > Cc: petsc-users at mcs.anl.gov> > > > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > > > > > Please please please always show WHAT YOU RAN with the error message. Otherwise we just must guess > > > > > > > > > > > > > > > > On Apr 17, 2015, at 3:58 PM, Sun, Hui >> wrote: > > > > > > > > > > > > Thank you Barry. I just have it installed. And I was running a code, and got the following error message: > > > > > > > > > > > > HYDU_create_process (utils/launch/launch.c:75): execvp error on file valgrind: (No such file or directory) > > > > > > > > > > > > =================================================================================== > > > > > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > > > > > = PID 66059 RUNNING AT math178.ucsd.edu > > > > > > = EXIT CODE: 255 > > > > > > = CLEANING UP REMAINING PROCESSES > > > > > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > > > > > =================================================================================== > > > > > > > > > > > > What does this mean? > > > > > > > > > > > > Best, > > > > > > Hui > > > > > > > > > > > > ________________________________________ > > > > > > From: Barry Smith [bsmith at mcs.anl.gov>] > > > > > > Sent: Friday, April 17, 2015 1:08 PM > > > > > > To: Sun, Hui > > > > > > Cc: petsc-users at mcs.anl.gov> > > > > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > > > > > > > The development copy of PETSc allows using --prefix with MPICH http://www.mcs.anl.gov/petsc/developers/index.html > > > > > > > > > > > > > > > > > > > > > > > >> On Apr 17, 2015, at 2:38 PM, Sun, Hui >> wrote: > > > > > >> > > > > > >> Hello, > > > > > >> > > > > > >> I want to install petsc with valgrind. And I read some documentations, which suggests me to do > > > > > >> ./configure --download-mpich --with-valgind-dir=? > > > > > >> > > > > > >> However, I also want to specify --prefix, but petsc doesn't allow me, because --download-mpich option does not work with --prefix install. > > > > > >> > > > > > >> Is there anyway that I can have petsc --prefix, together with valgrind? > > > > > >> > > > > > >> Best, > > > > > >> Hui > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > > > -- Norbert Wiener > > > > > > > > > > > > > > > > -- > > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > > -- Norbert Wiener > > > > > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > -- Norbert Wiener > > > > > > From balay at mcs.anl.gov Sun Apr 19 15:30:28 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 19 Apr 2015 15:30:28 -0500 Subject: [petsc-users] install petsc with valgrind In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010FE440@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> <498A9304-AF3F-42CF-800A-295FB9C0897D@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE31E@XMAIL-MBX-BH1.AD.UCSD.EDU> <902FF397-BB49-4647-9E91-E11D2C851446@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE343@XMAIL-MBX-BH1.AD.UCSD.EDU>, <7501CC2B7BBCC44A92ECEEC316170ECB010FE366@XMAIL-MBX-BH1.AD.UCSD.EDU>, , <7501CC2B7BBCC44A92ECEEC316170ECB010FE384@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB010FE440@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: On Sun, 19 Apr 2015, Sun, Hui wrote: > I try to run > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n 2 ./test -mx 32 -my 32 -mz 32 > > It seems to be working, except that it gives me an error message: > > --21019-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" Looks like if you run the commands as suggested by valgrind - the erorr messages will go away. Satish > --21020-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > error: No such file or directory - old dSYM file cannot be overwritten: old: '/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib' > new:'/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib.x86_64'. > > Why do I get this message? Do I need to get rid of it? > > Best, > Hui > > > ________________________________________ > From: Sun, Hui > Sent: Friday, April 17, 2015 3:37 PM > To: petsc-users > Cc: Matthew Knepley > Subject: RE: [petsc-users] install petsc with valgrind > > Thank you Satish! It works perfect. > > Hui > > > ________________________________________ > From: Satish Balay [balay at mcs.anl.gov] > Sent: Friday, April 17, 2015 3:29 PM > To: Sun, Hui > Cc: Matthew Knepley; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] install petsc with valgrind > > Well - its best if the .o files are not deleted.. [then you don't > have to run dsymutil]. One way to do it [if using PETSc makefiles] is: > > make test RM=/bin/true > > To run dsymutil - I think you need the .o files anyway - and run it as instructed below. > > > --66450-- run: /usr/bin/dsymutil "./test" > > Satish > > On Fri, 17 Apr 2015, Sun, Hui wrote: > > > How do I run dsymutil so as to get the line numbers? > > > > > > ________________________________ > > From: Matthew Knepley [knepley at gmail.com] > > Sent: Friday, April 17, 2015 3:10 PM > > To: Sun, Hui > > Cc: Barry Smith; petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] install petsc with valgrind > > > > On Fri, Apr 17, 2015 at 5:09 PM, Sun, Hui > wrote: > > I'm not sure if I'm doing the right thing. But I have just > > export PATH=$PATH:dir_to_valgind_bin > > > > And I rerun the command: > > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > > > Then it seems to output something, for example: > > ==66450== Invalid write of size 8 > > ==66450== at 0x100055225: Schur_SetMatrix (in ./test) > > ==66450== by 0x10004F218: StokesSetup (in ./test) > > ==66450== by 0x1000DC582: main (in ./test) > > ==66450== Address 0x7fff5f93fb28 is on thread 1's stack > > ==66450== in frame #0, created by Schur_SetMatrix (???) > > ==66450== > > > > Yes, this worked, and it shows you have a memory overwrite in your code. You have no line > > numbers because you deleted the *.o files without running dsymutil. > > > > And before those output, it has the following lines: > > --66450-- run: /usr/bin/dsymutil "./test" > > warning: (x86_64) /Users/hus003/projects/LS_BDRY/test/test.o unable to open object file > > warning: no debug symbols in executable (-arch x86_64) > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpicxx.12.dylib" > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpifort.12.dylib" > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpi.12.dylib" > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libpmpi.12.dylib" > > > > Does it mean that it's working? But what does these two warnings mean? > > > > These warnings mean that symbols are missing from those libraries, which is no surprise. > > > > matt > > > > Best, > > Hui > > > > > > ________________________________________ > > From: Barry Smith [bsmith at mcs.anl.gov] > > Sent: Friday, April 17, 2015 2:57 PM > > To: Matthew Knepley > > Cc: Sun, Hui; petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] install petsc with valgrind > > > > Or make sure valgrind is in your path > > > > > On Apr 17, 2015, at 4:56 PM, Matthew Knepley > wrote: > > > > > > As I said, it cannot find valgrind in your path. Remove it or install valgrind. > > > > > > Matt > > > > > > On Fri, Apr 17, 2015 at 4:55 PM, Sun, Hui > wrote: > > > Thank you, Matt and Barry. I was trying to run the following: > > > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > > > > > where NP=2, ND=32 > > > > > > And in the makefile I have specified the following: > > > include ${PETSC_DIR}/lib/petsc/conf/variables > > > include ${PETSC_DIR}/lib/petsc/conf/rules > > > include ${PETSC_DIR}/lib/petsc/conf/test > > > > > > Best, > > > Hui > > > > > > > > > ________________________________________ > > > From: Barry Smith [bsmith at mcs.anl.gov] > > > Sent: Friday, April 17, 2015 2:50 PM > > > To: Sun, Hui > > > Cc: petsc-users at mcs.anl.gov > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > Please please please always show WHAT YOU RAN with the error message. Otherwise we just must guess > > > > > > > > > > On Apr 17, 2015, at 3:58 PM, Sun, Hui > wrote: > > > > > > > > Thank you Barry. I just have it installed. And I was running a code, and got the following error message: > > > > > > > > HYDU_create_process (utils/launch/launch.c:75): execvp error on file valgrind: (No such file or directory) > > > > > > > > =================================================================================== > > > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > > > = PID 66059 RUNNING AT math178.ucsd.edu > > > > = EXIT CODE: 255 > > > > = CLEANING UP REMAINING PROCESSES > > > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > > > =================================================================================== > > > > > > > > What does this mean? > > > > > > > > Best, > > > > Hui > > > > > > > > ________________________________________ > > > > From: Barry Smith [bsmith at mcs.anl.gov] > > > > Sent: Friday, April 17, 2015 1:08 PM > > > > To: Sun, Hui > > > > Cc: petsc-users at mcs.anl.gov > > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > > > The development copy of PETSc allows using --prefix with MPICH http://www.mcs.anl.gov/petsc/developers/index.html > > > > > > > > > > > > > > > >> On Apr 17, 2015, at 2:38 PM, Sun, Hui > wrote: > > > >> > > > >> Hello, > > > >> > > > >> I want to install petsc with valgrind. And I read some documentations, which suggests me to do > > > >> ./configure --download-mpich --with-valgind-dir=? > > > >> > > > >> However, I also want to specify --prefix, but petsc doesn't allow me, because --download-mpich option does not work with --prefix install. > > > >> > > > >> Is there anyway that I can have petsc --prefix, together with valgrind? > > > >> > > > >> Best, > > > >> Hui > > > > > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > -- Norbert Wiener > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > From jchang27 at uh.edu Sun Apr 19 17:20:23 2015 From: jchang27 at uh.edu (Justin Chang) Date: Sun, 19 Apr 2015 17:20:23 -0500 Subject: [petsc-users] Uninitialized values within mesh generation Message-ID: Hi all, I have the following lines for mesh generation: DM boundary; ierr = DMCreate(comm,&boundary);CHKERRQ(ierr); ierr = DMSetType(boundary, DMPLEX);CHKERRQ(ierr); ierr = DMSetDimension(boundary, spatialDim-1);CHKERRQ(ierr); switch (spatialDim) { case 2: { ierr = DMPlexCreateSquareBoundary(boundary,lower,upper,edges);CHKERRQ(ierr); break; } case 3: { ierr = DMPlexCreateCubeBoundary(boundary,lower,upper,edges);CHKERRQ(ierr); break; } default: SETERRQ1(comm,PETSC_ERR_SUP,"Dimension not supported: %d",spatialDim); } ierr = DMPlexGenerate(boundary,NULL,PETSC_TRUE,dm);CHKERRQ(ierr); ierr = DMDestroy(&boundary);CHKERRQ(ierr); where lower and upper are global PetscScalars * and edges is a global PetscInt *. These values are PetscCalloc'ed and initialized via commandline options (PetscOptionsGetScalar/IntArray). When I run my code with valgrind, I get the following messages: ==12706== Conditional jump or move depends on uninitialised value(s) ==12706== at 0x65E778C: TetGenMeshPointTraverse(TetGenMesh*, double**) (ctetgen.c:4107) ==12706== by 0x661C8F8: TetGenMeshDelaunizeVertices(TetGenMesh*) (ctetgen.c:12091) ==12706== by 0x6661A8E: TetGenTetrahedralize(TetGenOpts*, PLC*, PLC*) (ctetgen.c:21483) ==12706== by 0x5B655EB: DMPlexGenerate_CTetgen(_p_DM*, PetscBool, _p_DM**) (plexgenerate.c:834) ==12706== by 0x5B685F4: DMPlexGenerate (plexgenerate.c:1080) ==12706== by 0x409B6D: CreateMesh(int, AppCtx*, _p_DM**) (main.c:657) ==12706== by 0x40DBA8: main (main.c:1005) ==12706== ==12706== Conditional jump or move depends on uninitialised value(s) ==12706== at 0x661BF6C: TetGenMeshDelaunayIncrFlip(TetGenMesh*, triface*, double**, long, PetscBool, PetscBool, double, Queue*) (ctetgen.c:12011) ==12706== by 0x661CD0B: TetGenMeshDelaunizeVertices(TetGenMesh*) (ctetgen.c:12113) ==12706== by 0x6661A8E: TetGenTetrahedralize(TetGenOpts*, PLC*, PLC*) (ctetgen.c:21483) ==12706== by 0x5B655EB: DMPlexGenerate_CTetgen(_p_DM*, PetscBool, _p_DM**) (plexgenerate.c:834) ==12706== by 0x5B685F4: DMPlexGenerate (plexgenerate.c:1080) ==12706== by 0x409B6D: CreateMesh(int, AppCtx*, _p_DM**) (main.c:657) ==12706== by 0x40DBA8: main (main.c:1005) ==12706== ==12706== Conditional jump or move depends on uninitialised value(s) ==12706== at 0x661BFB6: TetGenMeshDelaunayIncrFlip(TetGenMesh*, triface*, double**, long, PetscBool, PetscBool, double, Queue*) (ctetgen.c:12014) ==12706== by 0x661CD0B: TetGenMeshDelaunizeVertices(TetGenMesh*) (ctetgen.c:12113) ==12706== by 0x6661A8E: TetGenTetrahedralize(TetGenOpts*, PLC*, PLC*) (ctetgen.c:21483) ==12706== by 0x5B655EB: DMPlexGenerate_CTetgen(_p_DM*, PetscBool, _p_DM**) (plexgenerate.c:834) ==12706== by 0x5B685F4: DMPlexGenerate (plexgenerate.c:1080) ==12706== by 0x409B6D: CreateMesh(int, AppCtx*, _p_DM**) (main.c:657) ==12706== by 0x40DBA8: main (main.c:1005) ==12706== ==12706== Conditional jump or move depends on uninitialised value(s) ==12706== at 0x661C004: TetGenMeshDelaunayIncrFlip(TetGenMesh*, triface*, double**, long, PetscBool, PetscBool, double, Queue*) (ctetgen.c:12017) ==12706== by 0x661CD0B: TetGenMeshDelaunizeVertices(TetGenMesh*) (ctetgen.c:12113) ==12706== by 0x6661A8E: TetGenTetrahedralize(TetGenOpts*, PLC*, PLC*) (ctetgen.c:21483) ==12706== by 0x5B655EB: DMPlexGenerate_CTetgen(_p_DM*, PetscBool, _p_DM**) (plexgenerate.c:834) ==12706== by 0x5B685F4: DMPlexGenerate (plexgenerate.c:1080) ==12706== by 0x409B6D: CreateMesh(int, AppCtx*, _p_DM**) (main.c:657) ==12706== by 0x40DBA8: main (main.c:1005) ==12706== ==12706== Conditional jump or move depends on uninitialised value(s) ==12706== at 0x661C048: TetGenMeshDelaunayIncrFlip(TetGenMesh*, triface*, double**, long, PetscBool, PetscBool, double, Queue*) (ctetgen.c:12020) ==12706== by 0x661CD0B: TetGenMeshDelaunizeVertices(TetGenMesh*) (ctetgen.c:12113) ==12706== by 0x6661A8E: TetGenTetrahedralize(TetGenOpts*, PLC*, PLC*) (ctetgen.c:21483) ==12706== by 0x5B655EB: DMPlexGenerate_CTetgen(_p_DM*, PetscBool, _p_DM**) (plexgenerate.c:834) ==12706== by 0x5B685F4: DMPlexGenerate (plexgenerate.c:1080) ==12706== by 0x409B6D: CreateMesh(int, AppCtx*, _p_DM**) (main.c:657) ==12706== by 0x40DBA8: main (main.c:1005) ==12706== ==12706== Conditional jump or move depends on uninitialised value(s) ==12706== at 0x661B232: TetGenMeshInsertVertexBW(TetGenMesh*, double*, triface*, PetscBool, PetscBool, PetscBool, PetscBool, locateresult*) (ctetgen.c:11905) ==12706== by 0x661C2AF: TetGenMeshDelaunayIncrFlip(TetGenMesh*, triface*, double**, long, PetscBool, PetscBool, double, Queue*) (ctetgen.c:12046) ==12706== by 0x661CD0B: TetGenMeshDelaunizeVertices(TetGenMesh*) (ctetgen.c:12113) ==12706== by 0x6661A8E: TetGenTetrahedralize(TetGenOpts*, PLC*, PLC*) (ctetgen.c:21483) ==12706== by 0x5B655EB: DMPlexGenerate_CTetgen(_p_DM*, PetscBool, _p_DM**) (plexgenerate.c:834) ==12706== by 0x5B685F4: DMPlexGenerate (plexgenerate.c:1080) ==12706== by 0x409B6D: CreateMesh(int, AppCtx*, _p_DM**) (main.c:657) ==12706== by 0x40DBA8: main (main.c:1005) ==12706== What does this mean? Is there something wrong/missing with the current implementation of the mesh generation? Thanks, Justin -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Apr 19 17:24:34 2015 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 19 Apr 2015 17:24:34 -0500 Subject: [petsc-users] Uninitialized values within mesh generation In-Reply-To: References: Message-ID: On Sun, Apr 19, 2015 at 5:20 PM, Justin Chang wrote: > Hi all, > > I have the following lines for mesh generation: > > DM boundary; > ierr = DMCreate(comm,&boundary);CHKERRQ(ierr); > ierr = DMSetType(boundary, DMPLEX);CHKERRQ(ierr); > ierr = DMSetDimension(boundary, spatialDim-1);CHKERRQ(ierr); > switch (spatialDim) { > case 2: > { > ierr = > DMPlexCreateSquareBoundary(boundary,lower,upper,edges);CHKERRQ(ierr); > break; > } > case 3: > { > ierr = > DMPlexCreateCubeBoundary(boundary,lower,upper,edges);CHKERRQ(ierr); > break; > } > default: > SETERRQ1(comm,PETSC_ERR_SUP,"Dimension not supported: > %d",spatialDim); > } > ierr = DMPlexGenerate(boundary,NULL,PETSC_TRUE,dm);CHKERRQ(ierr); > ierr = DMDestroy(&boundary);CHKERRQ(ierr); > > where lower and upper are global PetscScalars * and edges is a global > PetscInt *. These values are PetscCalloc'ed and initialized via commandline > options (PetscOptionsGetScalar/IntArray). When I run my code with valgrind, > I get the following messages: > > ==12706== Conditional jump or move depends on uninitialised value(s) > ==12706== at 0x65E778C: TetGenMeshPointTraverse(TetGenMesh*, double**) > (ctetgen.c:4107) > ==12706== by 0x661C8F8: TetGenMeshDelaunizeVertices(TetGenMesh*) > (ctetgen.c:12091) > ==12706== by 0x6661A8E: TetGenTetrahedralize(TetGenOpts*, PLC*, PLC*) > (ctetgen.c:21483) > ==12706== by 0x5B655EB: DMPlexGenerate_CTetgen(_p_DM*, PetscBool, > _p_DM**) (plexgenerate.c:834) > ==12706== by 0x5B685F4: DMPlexGenerate (plexgenerate.c:1080) > ==12706== by 0x409B6D: CreateMesh(int, AppCtx*, _p_DM**) (main.c:657) > ==12706== by 0x40DBA8: main (main.c:1005) > ==12706== > ==12706== Conditional jump or move depends on uninitialised value(s) > ==12706== at 0x661BF6C: TetGenMeshDelaunayIncrFlip(TetGenMesh*, > triface*, double**, long, PetscBool, PetscBool, double, Queue*) > (ctetgen.c:12011) > ==12706== by 0x661CD0B: TetGenMeshDelaunizeVertices(TetGenMesh*) > (ctetgen.c:12113) > ==12706== by 0x6661A8E: TetGenTetrahedralize(TetGenOpts*, PLC*, PLC*) > (ctetgen.c:21483) > ==12706== by 0x5B655EB: DMPlexGenerate_CTetgen(_p_DM*, PetscBool, > _p_DM**) (plexgenerate.c:834) > ==12706== by 0x5B685F4: DMPlexGenerate (plexgenerate.c:1080) > ==12706== by 0x409B6D: CreateMesh(int, AppCtx*, _p_DM**) (main.c:657) > ==12706== by 0x40DBA8: main (main.c:1005) > ==12706== > ==12706== Conditional jump or move depends on uninitialised value(s) > ==12706== at 0x661BFB6: TetGenMeshDelaunayIncrFlip(TetGenMesh*, > triface*, double**, long, PetscBool, PetscBool, double, Queue*) > (ctetgen.c:12014) > ==12706== by 0x661CD0B: TetGenMeshDelaunizeVertices(TetGenMesh*) > (ctetgen.c:12113) > ==12706== by 0x6661A8E: TetGenTetrahedralize(TetGenOpts*, PLC*, PLC*) > (ctetgen.c:21483) > ==12706== by 0x5B655EB: DMPlexGenerate_CTetgen(_p_DM*, PetscBool, > _p_DM**) (plexgenerate.c:834) > ==12706== by 0x5B685F4: DMPlexGenerate (plexgenerate.c:1080) > ==12706== by 0x409B6D: CreateMesh(int, AppCtx*, _p_DM**) (main.c:657) > ==12706== by 0x40DBA8: main (main.c:1005) > ==12706== > ==12706== Conditional jump or move depends on uninitialised value(s) > ==12706== at 0x661C004: TetGenMeshDelaunayIncrFlip(TetGenMesh*, > triface*, double**, long, PetscBool, PetscBool, double, Queue*) > (ctetgen.c:12017) > ==12706== by 0x661CD0B: TetGenMeshDelaunizeVertices(TetGenMesh*) > (ctetgen.c:12113) > ==12706== by 0x6661A8E: TetGenTetrahedralize(TetGenOpts*, PLC*, PLC*) > (ctetgen.c:21483) > ==12706== by 0x5B655EB: DMPlexGenerate_CTetgen(_p_DM*, PetscBool, > _p_DM**) (plexgenerate.c:834) > ==12706== by 0x5B685F4: DMPlexGenerate (plexgenerate.c:1080) > ==12706== by 0x409B6D: CreateMesh(int, AppCtx*, _p_DM**) (main.c:657) > ==12706== by 0x40DBA8: main (main.c:1005) > ==12706== > ==12706== Conditional jump or move depends on uninitialised value(s) > ==12706== at 0x661C048: TetGenMeshDelaunayIncrFlip(TetGenMesh*, > triface*, double**, long, PetscBool, PetscBool, double, Queue*) > (ctetgen.c:12020) > ==12706== by 0x661CD0B: TetGenMeshDelaunizeVertices(TetGenMesh*) > (ctetgen.c:12113) > ==12706== by 0x6661A8E: TetGenTetrahedralize(TetGenOpts*, PLC*, PLC*) > (ctetgen.c:21483) > ==12706== by 0x5B655EB: DMPlexGenerate_CTetgen(_p_DM*, PetscBool, > _p_DM**) (plexgenerate.c:834) > ==12706== by 0x5B685F4: DMPlexGenerate (plexgenerate.c:1080) > ==12706== by 0x409B6D: CreateMesh(int, AppCtx*, _p_DM**) (main.c:657) > ==12706== by 0x40DBA8: main (main.c:1005) > ==12706== > ==12706== Conditional jump or move depends on uninitialised value(s) > ==12706== at 0x661B232: TetGenMeshInsertVertexBW(TetGenMesh*, double*, > triface*, PetscBool, PetscBool, PetscBool, PetscBool, locateresult*) > (ctetgen.c:11905) > ==12706== by 0x661C2AF: TetGenMeshDelaunayIncrFlip(TetGenMesh*, > triface*, double**, long, PetscBool, PetscBool, double, Queue*) > (ctetgen.c:12046) > ==12706== by 0x661CD0B: TetGenMeshDelaunizeVertices(TetGenMesh*) > (ctetgen.c:12113) > ==12706== by 0x6661A8E: TetGenTetrahedralize(TetGenOpts*, PLC*, PLC*) > (ctetgen.c:21483) > ==12706== by 0x5B655EB: DMPlexGenerate_CTetgen(_p_DM*, PetscBool, > _p_DM**) (plexgenerate.c:834) > ==12706== by 0x5B685F4: DMPlexGenerate (plexgenerate.c:1080) > ==12706== by 0x409B6D: CreateMesh(int, AppCtx*, _p_DM**) (main.c:657) > ==12706== by 0x40DBA8: main (main.c:1005) > ==12706== > > What does this mean? Is there something wrong/missing with the current > implementation of the mesh generation? > Honestly, I have no idea. That is my mechanical translation of TetGen to C, which could have bugs but I could never find them. It seems to work on all problems I have tried. I did it so I could open up the interface beyond what he exposes in the command line interface. Also I made it use PetscMalloc, etc. Thanks, Matt > Thanks, > Justin > > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Apr 19 18:08:58 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 19 Apr 2015 18:08:58 -0500 Subject: [petsc-users] Uninitialized values within mesh generation In-Reply-To: References: Message-ID: You can use valgrind in conjunction with the debugger to tray to track down the problem. Or simply look at the ctetgen code at those source lines to see why values would be uninitialized. > On Apr 19, 2015, at 5:20 PM, Justin Chang wrote: > > Hi all, > > I have the following lines for mesh generation: > > DM boundary; > ierr = DMCreate(comm,&boundary);CHKERRQ(ierr); > ierr = DMSetType(boundary, DMPLEX);CHKERRQ(ierr); > ierr = DMSetDimension(boundary, spatialDim-1);CHKERRQ(ierr); > switch (spatialDim) { > case 2: > { > ierr = DMPlexCreateSquareBoundary(boundary,lower,upper,edges);CHKERRQ(ierr); > break; > } > case 3: > { > ierr = DMPlexCreateCubeBoundary(boundary,lower,upper,edges);CHKERRQ(ierr); > break; > } > default: > SETERRQ1(comm,PETSC_ERR_SUP,"Dimension not supported: %d",spatialDim); > } > ierr = DMPlexGenerate(boundary,NULL,PETSC_TRUE,dm);CHKERRQ(ierr); > ierr = DMDestroy(&boundary);CHKERRQ(ierr); > > where lower and upper are global PetscScalars * and edges is a global PetscInt *. These values are PetscCalloc'ed and initialized via commandline options (PetscOptionsGetScalar/IntArray). When I run my code with valgrind, I get the following messages: > > ==12706== Conditional jump or move depends on uninitialised value(s) > ==12706== at 0x65E778C: TetGenMeshPointTraverse(TetGenMesh*, double**) (ctetgen.c:4107) > ==12706== by 0x661C8F8: TetGenMeshDelaunizeVertices(TetGenMesh*) (ctetgen.c:12091) > ==12706== by 0x6661A8E: TetGenTetrahedralize(TetGenOpts*, PLC*, PLC*) (ctetgen.c:21483) > ==12706== by 0x5B655EB: DMPlexGenerate_CTetgen(_p_DM*, PetscBool, _p_DM**) (plexgenerate.c:834) > ==12706== by 0x5B685F4: DMPlexGenerate (plexgenerate.c:1080) > ==12706== by 0x409B6D: CreateMesh(int, AppCtx*, _p_DM**) (main.c:657) > ==12706== by 0x40DBA8: main (main.c:1005) > ==12706== > ==12706== Conditional jump or move depends on uninitialised value(s) > ==12706== at 0x661BF6C: TetGenMeshDelaunayIncrFlip(TetGenMesh*, triface*, double**, long, PetscBool, PetscBool, double, Queue*) (ctetgen.c:12011) > ==12706== by 0x661CD0B: TetGenMeshDelaunizeVertices(TetGenMesh*) (ctetgen.c:12113) > ==12706== by 0x6661A8E: TetGenTetrahedralize(TetGenOpts*, PLC*, PLC*) (ctetgen.c:21483) > ==12706== by 0x5B655EB: DMPlexGenerate_CTetgen(_p_DM*, PetscBool, _p_DM**) (plexgenerate.c:834) > ==12706== by 0x5B685F4: DMPlexGenerate (plexgenerate.c:1080) > ==12706== by 0x409B6D: CreateMesh(int, AppCtx*, _p_DM**) (main.c:657) > ==12706== by 0x40DBA8: main (main.c:1005) > ==12706== > ==12706== Conditional jump or move depends on uninitialised value(s) > ==12706== at 0x661BFB6: TetGenMeshDelaunayIncrFlip(TetGenMesh*, triface*, double**, long, PetscBool, PetscBool, double, Queue*) (ctetgen.c:12014) > ==12706== by 0x661CD0B: TetGenMeshDelaunizeVertices(TetGenMesh*) (ctetgen.c:12113) > ==12706== by 0x6661A8E: TetGenTetrahedralize(TetGenOpts*, PLC*, PLC*) (ctetgen.c:21483) > ==12706== by 0x5B655EB: DMPlexGenerate_CTetgen(_p_DM*, PetscBool, _p_DM**) (plexgenerate.c:834) > ==12706== by 0x5B685F4: DMPlexGenerate (plexgenerate.c:1080) > ==12706== by 0x409B6D: CreateMesh(int, AppCtx*, _p_DM**) (main.c:657) > ==12706== by 0x40DBA8: main (main.c:1005) > ==12706== > ==12706== Conditional jump or move depends on uninitialised value(s) > ==12706== at 0x661C004: TetGenMeshDelaunayIncrFlip(TetGenMesh*, triface*, double**, long, PetscBool, PetscBool, double, Queue*) (ctetgen.c:12017) > ==12706== by 0x661CD0B: TetGenMeshDelaunizeVertices(TetGenMesh*) (ctetgen.c:12113) > ==12706== by 0x6661A8E: TetGenTetrahedralize(TetGenOpts*, PLC*, PLC*) (ctetgen.c:21483) > ==12706== by 0x5B655EB: DMPlexGenerate_CTetgen(_p_DM*, PetscBool, _p_DM**) (plexgenerate.c:834) > ==12706== by 0x5B685F4: DMPlexGenerate (plexgenerate.c:1080) > ==12706== by 0x409B6D: CreateMesh(int, AppCtx*, _p_DM**) (main.c:657) > ==12706== by 0x40DBA8: main (main.c:1005) > ==12706== > ==12706== Conditional jump or move depends on uninitialised value(s) > ==12706== at 0x661C048: TetGenMeshDelaunayIncrFlip(TetGenMesh*, triface*, double**, long, PetscBool, PetscBool, double, Queue*) (ctetgen.c:12020) > ==12706== by 0x661CD0B: TetGenMeshDelaunizeVertices(TetGenMesh*) (ctetgen.c:12113) > ==12706== by 0x6661A8E: TetGenTetrahedralize(TetGenOpts*, PLC*, PLC*) (ctetgen.c:21483) > ==12706== by 0x5B655EB: DMPlexGenerate_CTetgen(_p_DM*, PetscBool, _p_DM**) (plexgenerate.c:834) > ==12706== by 0x5B685F4: DMPlexGenerate (plexgenerate.c:1080) > ==12706== by 0x409B6D: CreateMesh(int, AppCtx*, _p_DM**) (main.c:657) > ==12706== by 0x40DBA8: main (main.c:1005) > ==12706== > ==12706== Conditional jump or move depends on uninitialised value(s) > ==12706== at 0x661B232: TetGenMeshInsertVertexBW(TetGenMesh*, double*, triface*, PetscBool, PetscBool, PetscBool, PetscBool, locateresult*) (ctetgen.c:11905) > ==12706== by 0x661C2AF: TetGenMeshDelaunayIncrFlip(TetGenMesh*, triface*, double**, long, PetscBool, PetscBool, double, Queue*) (ctetgen.c:12046) > ==12706== by 0x661CD0B: TetGenMeshDelaunizeVertices(TetGenMesh*) (ctetgen.c:12113) > ==12706== by 0x6661A8E: TetGenTetrahedralize(TetGenOpts*, PLC*, PLC*) (ctetgen.c:21483) > ==12706== by 0x5B655EB: DMPlexGenerate_CTetgen(_p_DM*, PetscBool, _p_DM**) (plexgenerate.c:834) > ==12706== by 0x5B685F4: DMPlexGenerate (plexgenerate.c:1080) > ==12706== by 0x409B6D: CreateMesh(int, AppCtx*, _p_DM**) (main.c:657) > ==12706== by 0x40DBA8: main (main.c:1005) > ==12706== > > What does this mean? Is there something wrong/missing with the current implementation of the mesh generation? > > Thanks, > Justin > > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 From hus003 at ucsd.edu Sun Apr 19 18:14:10 2015 From: hus003 at ucsd.edu (Sun, Hui) Date: Sun, 19 Apr 2015 23:14:10 +0000 Subject: [petsc-users] install petsc with valgrind In-Reply-To: References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE2B0@XMAIL-MBX-BH1.AD.UCSD.EDU> <5301DF84-D1D4-430C-9365-7316FA65F104@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE2FF@XMAIL-MBX-BH1.AD.UCSD.EDU> <498A9304-AF3F-42CF-800A-295FB9C0897D@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE31E@XMAIL-MBX-BH1.AD.UCSD.EDU> <902FF397-BB49-4647-9E91-E11D2C851446@mcs.anl.gov> <7501CC2B7BBCC44A92ECEEC316170ECB010FE343@XMAIL-MBX-BH1.AD.UCSD.EDU>, <7501CC2B7BBCC44A92ECEEC316170ECB010FE366@XMAIL-MBX-BH1.AD.UCSD.EDU>, , <7501CC2B7BBCC44A92ECEEC316170ECB010FE384@XMAIL-MBX-BH1.AD.UCSD.EDU> <7501CC2B7BBCC44A92ECEEC316170ECB010FE440@XMAIL-MBX-BH1.AD.UCSD.EDU>, Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010FE4CF@XMAIL-MBX-BH1.AD.UCSD.EDU> Thank you for your help, Satish. I followed your suggestions to run the commands, and the error messages does go away. By the way, sorry for the late reply. I wasn't by my computer until now. Best, Hui ________________________________________ From: Satish Balay [balay at mcs.anl.gov] Sent: Sunday, April 19, 2015 1:30 PM To: Sun, Hui Cc: petsc-users Subject: Re: [petsc-users] install petsc with valgrind On Sun, 19 Apr 2015, Sun, Hui wrote: > I try to run > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n 2 ./test -mx 32 -my 32 -mz 32 > > It seems to be working, except that it gives me an error message: > > --21019-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" Looks like if you run the commands as suggested by valgrind - the erorr messages will go away. Satish > --21020-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > error: No such file or directory - old dSYM file cannot be overwritten: old: '/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib' > new:'/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib.dSYM/Contents/Resources/DWARF/libpetsc.3.05.3.dylib.x86_64'. > > Why do I get this message? Do I need to get rid of it? > > Best, > Hui > > > ________________________________________ > From: Sun, Hui > Sent: Friday, April 17, 2015 3:37 PM > To: petsc-users > Cc: Matthew Knepley > Subject: RE: [petsc-users] install petsc with valgrind > > Thank you Satish! It works perfect. > > Hui > > > ________________________________________ > From: Satish Balay [balay at mcs.anl.gov] > Sent: Friday, April 17, 2015 3:29 PM > To: Sun, Hui > Cc: Matthew Knepley; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] install petsc with valgrind > > Well - its best if the .o files are not deleted.. [then you don't > have to run dsymutil]. One way to do it [if using PETSc makefiles] is: > > make test RM=/bin/true > > To run dsymutil - I think you need the .o files anyway - and run it as instructed below. > > > --66450-- run: /usr/bin/dsymutil "./test" > > Satish > > On Fri, 17 Apr 2015, Sun, Hui wrote: > > > How do I run dsymutil so as to get the line numbers? > > > > > > ________________________________ > > From: Matthew Knepley [knepley at gmail.com] > > Sent: Friday, April 17, 2015 3:10 PM > > To: Sun, Hui > > Cc: Barry Smith; petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] install petsc with valgrind > > > > On Fri, Apr 17, 2015 at 5:09 PM, Sun, Hui > wrote: > > I'm not sure if I'm doing the right thing. But I have just > > export PATH=$PATH:dir_to_valgind_bin > > > > And I rerun the command: > > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > > > Then it seems to output something, for example: > > ==66450== Invalid write of size 8 > > ==66450== at 0x100055225: Schur_SetMatrix (in ./test) > > ==66450== by 0x10004F218: StokesSetup (in ./test) > > ==66450== by 0x1000DC582: main (in ./test) > > ==66450== Address 0x7fff5f93fb28 is on thread 1's stack > > ==66450== in frame #0, created by Schur_SetMatrix (???) > > ==66450== > > > > Yes, this worked, and it shows you have a memory overwrite in your code. You have no line > > numbers because you deleted the *.o files without running dsymutil. > > > > And before those output, it has the following lines: > > --66450-- run: /usr/bin/dsymutil "./test" > > warning: (x86_64) /Users/hus003/projects/LS_BDRY/test/test.o unable to open object file > > warning: no debug symbols in executable (-arch x86_64) > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind/petsc-valgrind-debug/lib/libpetsc.3.05.3.dylib" > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpicxx.12.dylib" > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpifort.12.dylib" > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libmpi.12.dylib" > > --66450-- run: /usr/bin/dsymutil "/Users/hus003/Documents/software/petsc-valgrind-install/lib/libpmpi.12.dylib" > > > > Does it mean that it's working? But what does these two warnings mean? > > > > These warnings mean that symbols are missing from those libraries, which is no surprise. > > > > matt > > > > Best, > > Hui > > > > > > ________________________________________ > > From: Barry Smith [bsmith at mcs.anl.gov] > > Sent: Friday, April 17, 2015 2:57 PM > > To: Matthew Knepley > > Cc: Sun, Hui; petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] install petsc with valgrind > > > > Or make sure valgrind is in your path > > > > > On Apr 17, 2015, at 4:56 PM, Matthew Knepley > wrote: > > > > > > As I said, it cannot find valgrind in your path. Remove it or install valgrind. > > > > > > Matt > > > > > > On Fri, Apr 17, 2015 at 4:55 PM, Sun, Hui > wrote: > > > Thank you, Matt and Barry. I was trying to run the following: > > > -@${PETSC_DIR}/bin/petscmpiexec -valgrind -n ${NP} ./test -mx ${ND} -my ${ND} -mz ${ND} -ksp_monitor_short -ksp_type fgmres -pc_type asm > > > > > > where NP=2, ND=32 > > > > > > And in the makefile I have specified the following: > > > include ${PETSC_DIR}/lib/petsc/conf/variables > > > include ${PETSC_DIR}/lib/petsc/conf/rules > > > include ${PETSC_DIR}/lib/petsc/conf/test > > > > > > Best, > > > Hui > > > > > > > > > ________________________________________ > > > From: Barry Smith [bsmith at mcs.anl.gov] > > > Sent: Friday, April 17, 2015 2:50 PM > > > To: Sun, Hui > > > Cc: petsc-users at mcs.anl.gov > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > Please please please always show WHAT YOU RAN with the error message. Otherwise we just must guess > > > > > > > > > > On Apr 17, 2015, at 3:58 PM, Sun, Hui > wrote: > > > > > > > > Thank you Barry. I just have it installed. And I was running a code, and got the following error message: > > > > > > > > HYDU_create_process (utils/launch/launch.c:75): execvp error on file valgrind: (No such file or directory) > > > > > > > > =================================================================================== > > > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > > > = PID 66059 RUNNING AT math178.ucsd.edu > > > > = EXIT CODE: 255 > > > > = CLEANING UP REMAINING PROCESSES > > > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > > > =================================================================================== > > > > > > > > What does this mean? > > > > > > > > Best, > > > > Hui > > > > > > > > ________________________________________ > > > > From: Barry Smith [bsmith at mcs.anl.gov] > > > > Sent: Friday, April 17, 2015 1:08 PM > > > > To: Sun, Hui > > > > Cc: petsc-users at mcs.anl.gov > > > > Subject: Re: [petsc-users] install petsc with valgrind > > > > > > > > The development copy of PETSc allows using --prefix with MPICH http://www.mcs.anl.gov/petsc/developers/index.html > > > > > > > > > > > > > > > >> On Apr 17, 2015, at 2:38 PM, Sun, Hui > wrote: > > > >> > > > >> Hello, > > > >> > > > >> I want to install petsc with valgrind. And I read some documentations, which suggests me to do > > > >> ./configure --download-mpich --with-valgind-dir=? > > > >> > > > >> However, I also want to specify --prefix, but petsc doesn't allow me, because --download-mpich option does not work with --prefix install. > > > >> > > > >> Is there anyway that I can have petsc --prefix, together with valgrind? > > > >> > > > >> Best, > > > >> Hui > > > > > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > -- Norbert Wiener > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > From hus003 at ucsd.edu Sun Apr 19 18:55:59 2015 From: hus003 at ucsd.edu (Sun, Hui) Date: Sun, 19 Apr 2015 23:55:59 +0000 Subject: [petsc-users] Could be a bug on hypre with petsc Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010FE4E5@XMAIL-MBX-BH1.AD.UCSD.EDU> I'm basically solving Poisson, and using valgrind to check memory issues. I set pctype to PCHYPRE. It works fine if I use one core. But if I use two cores, it gives me the following error messages about the memory, and then it computes results fine. It seems that it has to do with hypre, for if I set pctype to something else such as PCGAMG or PCJACOBI, it does not output any error message at all, regardless of one core or two cores. By the way, I use DM objects for the matrix and ksp. ==59097== Syscall param writev(vector[...]) points to uninitialised byte(s) ==59097== at 0x2F19FF2: writev (in /usr/lib/system/libsystem_kernel.dylib) ==59097== by 0x2AF470D: MPIDU_Sock_writev (sock_immed.i:610) ==59097== by 0x2ADE3E5: MPIDI_CH3_iStartMsgv (ch3_istartmsgv.c:110) ==59097== by 0x2ACEEF8: MPIDI_CH3_EagerContigSend (ch3u_eager.c:182) ==59097== by 0x2AD556F: MPID_Send (mpid_send.c:143) ==59097== by 0x2920057: MPI_Send (send.c:127) ==59097== by 0x1A712B1: hypre_MPI_Send (mpistubs.c:988) ==59097== by 0x1A6E77B: hypre_DataExchangeList (exchange_data.c:392) ==59097== by 0x194D7C1: hypre_NewCommPkgCreate_core (new_commpkg.c:267) ==59097== by 0x195108B: hypre_MatvecCommPkgCreate (par_csr_communication.c:478) ==59097== by 0x192020B: hypre_BoomerAMGCreateS (par_strength.c:190) ==59097== by 0x189D6AD: hypre_BoomerAMGSetup (par_amg_setup.c:883) ==59097== by 0x187BD7A: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:52) ==59097== by 0x11EFCE6: PCSetUp_HYPRE (hypre.c:157) ==59097== by 0x12766D4: PCSetUp (precon.c:918) ==59097== by 0x1385670: KSPSetUp (itfunc.c:330) ==59097== by 0x138798B: KSPSolve (itfunc.c:542) ==59097== by 0x10003C854: fourPoissonSolve (FourPoisson3D.h:383) ==59097== by 0x1000DEA42: main (test.c:473) ==59097== Address 0x100a295d8 is 8 bytes inside a block of size 28 alloc'd ==59097== at 0xC258: malloc (vg_replace_malloc.c:300) ==59097== by 0x1A6F501: hypre_MAlloc (hypre_memory.c:62) ==59097== by 0x1A6DDBA: hypre_DataExchangeList (exchange_data.c:212) ==59097== by 0x194D7C1: hypre_NewCommPkgCreate_core (new_commpkg.c:267) ==59097== by 0x195108B: hypre_MatvecCommPkgCreate (par_csr_communication.c:478) ==59097== by 0x192020B: hypre_BoomerAMGCreateS (par_strength.c:190) ==59097== by 0x189D6AD: hypre_BoomerAMGSetup (par_amg_setup.c:883) ==59097== by 0x187BD7A: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:52) ==59097== by 0x11EFCE6: PCSetUp_HYPRE (hypre.c:157) ==59097== by 0x12766D4: PCSetUp (precon.c:918) ==59097== by 0x1385670: KSPSetUp (itfunc.c:330) ==59097== by 0x138798B: KSPSolve (itfunc.c:542) ==59097== by 0x10003C854: fourPoissonSolve (FourPoisson3D.h:383) ==59097== by 0x1000DEA42: main (test.c:473) Best, Hui -------------- next part -------------- An HTML attachment was scrubbed... URL: From siddhesh4godbole at gmail.com Mon Apr 20 03:05:18 2015 From: siddhesh4godbole at gmail.com (siddhesh godbole) Date: Mon, 20 Apr 2015 13:35:18 +0530 Subject: [petsc-users] Matrix assebly Message-ID: Hello, I have created a matrix with initial values and i want to change the elements of the same matrix and in the iterations column wise. how do i insert new columns into the matrix when calls for assembly begin and end have already been made? I have used MatGetColumnVector to select which column to edit but how do i store it back into the same matrix? because it seems it does not happen otherwise. *Siddhesh M Godbole* 5th year Dual Degree, Civil Eng & Applied Mech. IIT Madras -------------- next part -------------- An HTML attachment was scrubbed... URL: From Carol.Brickley at awe.co.uk Mon Apr 20 04:06:14 2015 From: Carol.Brickley at awe.co.uk (Carol.Brickley at awe.co.uk) Date: Mon, 20 Apr 2015 09:06:14 +0000 Subject: [petsc-users] petsc 3.4.3 to 3.5.3 Message-ID: <201504200906.t3K96JBL018583@msw1.awe.co.uk> All, I am switching to petsc 3.5.3 and finding some differences with 3.4.3. KSPSetOperators no longer has the MatStructure argument. I see for SAME_PRECONDITIONER that you now use KSPSetReusePreconditioner with flag set to tru before the call to set to true KSPSetOperators. Is there any need to do anything for the other flags, eg; DIFFERENT_NONZERO_PATTERN, SAME_ NONZERO_PATTERN or is that now tracked by the Mat object as suggeste in the documentation? Thanks, Carol Dr Carol Brickley BSc,PhD,ARCS,DIC,MBCS Senior Software Engineer Applied Computer Science DS+T, AWE Aldermaston Reading Berkshire RG7 4PR Direct: 0118 9855035 ___________________________________________________ ____________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaus at uni-mainz.de Mon Apr 20 05:47:18 2015 From: kaus at uni-mainz.de (Boris Kaus) Date: Mon, 20 Apr 2015 12:47:18 +0200 Subject: [petsc-users] PhD position on developing software to model hydrofracturing in geothermal systems Message-ID: <3DAF4261-DBA9-443B-8C22-36083DE89712@uni-mainz.de> Dear PETSC-users, I have an open PhD position in my group to develop codes to simulate hydrofracturing in visco-elasto-plastic rocks using massively parallel computers. We will use PETSC for this, so in case you are interested or you know someone that might be interested please let them know. A detailed description of the project, together with application instructions is given here http://www.geowiss.uni-mainz.de/Illustrationen/Geophysics/CREEP-ESR12-Mainz_Bristol_final.pdf thanks! Boris _______________________________________________________ Boris J.P. Kaus Professor Institute of Geosciences, Center for Computational Sciences & Center for Volcanoes and Atmosphere in Magmatic Open Systems Johannes Gutenberg University of Mainz, Mainz, Germany J.J. Becherweg 21, 55099 Mainz, Germany. Office: 00-285 Tel: +49.6131.392.4527 http://www.geo-dynamics.eu _______________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jchang27 at uh.edu Mon Apr 20 06:09:14 2015 From: jchang27 at uh.edu (Justin Chang) Date: Mon, 20 Apr 2015 06:09:14 -0500 Subject: [petsc-users] Uninterpolating a distributed mesh In-Reply-To: References: Message-ID: Matt, I have attempted to implement this, called UninterpolateSF(), and attached is the code for the function. However, I suspect that there are bugs because when I refine the mesh via -dm_refine my program crashes. In my program I have the following lines for uninterpolating a distributed mesh: DM idm=NULL; ierr = DMPlexUninterpolate(*dm, &idm);CHKERRQ(ierr); ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); if (size>1) { ierr = UninterpolateSF(*dm,idm);CHKERRQ(ierr); } ierr = DMDestroy(dm);CHKERRQ(ierr); *dm = idm; Can you have a look into the source code I attached and see what may potentially be the issue? Thanks, On Mon, Apr 13, 2015 at 4:28 PM, Matthew Knepley wrote: > On Mon, Apr 13, 2015 at 4:22 PM, Justin Chang wrote: > >> Is there an example somewhere that does something similar to this? >> > > There is DMPlexShiftSF_Internal() in plexsubmesh.c > > Matt > > >> Thanks, >> >> On Mon, Apr 13, 2015 at 12:31 PM, Matthew Knepley >> wrote: >> >>> On Sat, Apr 11, 2015 at 6:55 PM, Justin Chang wrote: >>> >>>> Hello, >>>> >>>> When I call DMPlexUninterpolate(...) on a distributed mesh (say 2 >>>> processors), it seems to overwrite the "ghost" points (i.e., the points not >>>> locally owned by the processor) and treats all points as if they are local >>>> to the processor. >>>> >>> >>> Yes, I wrote Uninterpolate() just for testing, and do not currently >>> handle the SF. I put it on my TODO list. >>> >>> Its not hard if you want to try. You just filter out any points that are >>> not cells and vertices from the SF, so >>> >>> PetscSFGetGraph() >>> for (leaves) >>> if leaf not a cell or vertex, skip >>> PetscSFSetGraph() >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Say I have this 2D cell-vertex mesh: >>>> >>>> 14----15-----16 >>>> | \ 5 | \ 7 | >>>> | \ | \ | >>>> | 4 \ | 6 \ | >>>> 11----12-----13 >>>> | \ 1 | \ 3 | >>>> | \ | \ | >>>> | 0 \ | 2 \ | >>>> 8------9------10 >>>> >>>> Which results in the following DM: >>>> >>>> DM Object: 2 MPI processes >>>> type: plex >>>> DM_0x84000004_0 in 2 dimensions: >>>> 0-cells: 9 0 >>>> 2-cells: 8 0 >>>> Labels: >>>> marker: 1 strata of sizes (8) >>>> depth: 2 strata of sizes (9, 8) >>>> >>>> I proceed by interpolating this DM: >>>> >>>> DM Object: 2 MPI processes >>>> type: plex >>>> DM_0x84000004_1 in 2 dimensions: >>>> 0-cells: 9 0 >>>> 1-cells: 16 0 >>>> 2-cells: 8 0 >>>> Labels: >>>> marker: 1 strata of sizes (16) >>>> depth: 3 strata of sizes (9, 16, 8) >>>> >>>> Then distributing across 2 processors: >>>> >>>> DM Object:Parallel Mesh 2 MPI processes >>>> type: plex >>>> Parallel Mesh in 2 dimensions: >>>> 0-cells: 6 6 >>>> 1-cells: 9 9 >>>> 2-cells: 4 4 >>>> Labels: >>>> marker: 1 strata of sizes (9) >>>> depth: 3 strata of sizes (6, 9, 4) >>>> >>>> I have the option of uniformly refining the mesh here but I choose not >>>> to for now. If my dofs are vertex based, then the global size of my DM >>>> vector is 9 and the local sizes for ranks 0 and 1 are 3 and 6 respectively. >>>> However, if I choose to uninterpolate the mesh by calling >>>> DMPlexUninterpolate(...), I get this: >>>> >>>> DM Object: 2 MPI processes >>>> type: plex >>>> DM_0x84000004_2 in 2 dimensions: >>>> 0-cells: 6 6 >>>> 2-cells: 4 4 >>>> Labels: >>>> marker: 1 strata of sizes (5) >>>> depth: 2 strata of sizes (6, 4) >>>> >>>> And the global size of my DM vector becomes 12 and the local size for >>>> both ranks is 6. It looks like the ghost points in rank 0 have been >>>> duplicated, which is not suppose to happen. >>>> >>>> Is there a way to capture the ghost point information when >>>> uninterpolating the DM? >>>> >>>> Thanks, >>>> >>>> -- >>>> Justin Chang >>>> PhD Candidate, Civil Engineering - Computational Sciences >>>> University of Houston, Department of Civil and Environmental Engineering >>>> Houston, TX 77004 >>>> (512) 963-3262 >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> Justin Chang >> PhD Candidate, Civil Engineering - Computational Sciences >> University of Houston, Department of Civil and Environmental Engineering >> Houston, TX 77004 >> (512) 963-3262 >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: UninterpolateSF.c Type: text/x-csrc Size: 2050 bytes Desc: not available URL: From mailinglists at xgm.de Mon Apr 20 06:35:23 2015 From: mailinglists at xgm.de (Florian Lindner) Date: Mon, 20 Apr 2015 13:35:23 +0200 Subject: [petsc-users] Create identity IS Message-ID: <1496342.C5sGYTRWzh@asaru> Hello, I use an index set for a row mapping but don't want to use one for the column mapping. I try to create an idenitity index set to supply to ISLocalToGlobalMappingCreateIS and MatSetLocalToGlobalMapping: IS is; ISCreate(PETSC_COMM_WORLD, &is); ISSetIdentity(is); but I get an segmentation fault at the last line. What is the best way to create an identity index set? Thanks, Florian From dalcinl at gmail.com Mon Apr 20 06:57:58 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 20 Apr 2015 14:57:58 +0300 Subject: [petsc-users] Create identity IS In-Reply-To: <1496342.C5sGYTRWzh@asaru> References: <1496342.C5sGYTRWzh@asaru> Message-ID: On 20 April 2015 at 14:35, Florian Lindner wrote: > Hello, > > I use an index set for a row mapping but don't want to use one for the column mapping. > > I try to create an idenitity index set to supply to ISLocalToGlobalMappingCreateIS and MatSetLocalToGlobalMapping: > > IS is; > ISCreate(PETSC_COMM_WORLD, &is); At this point the index set does not have the type set. > ISSetIdentity(is); > > but I get an segmentation fault at the last line. > Bad error checking, you should get an error saying that the type is not yet set. > What is the best way to create an identity index set? > I would use ISCreateStride(), and perhaps next ISToGeneral() if you really want a ISGENERAL type. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From bsmith at mcs.anl.gov Mon Apr 20 07:17:39 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 20 Apr 2015 07:17:39 -0500 Subject: [petsc-users] Matrix assebly In-Reply-To: References: Message-ID: <08A92785-82C3-4FCF-8E86-4C50898B90EB@mcs.anl.gov> You would simply call MatSetValues() to change those values (or MatSetValuesBlocked, etc). There are no special routines for changing columns. Barry > On Apr 20, 2015, at 3:05 AM, siddhesh godbole wrote: > > Hello, > > I have created a matrix with initial values and i want to change the elements of the same matrix and in the iterations column wise. how do i insert new columns into the matrix when calls for assembly begin and end have already been made? > > I have used MatGetColumnVector to select which column to edit but how do i store it back into the same matrix? because it seems it does not happen otherwise. > > Siddhesh M Godbole > > 5th year Dual Degree, > Civil Eng & Applied Mech. > IIT Madras From bsmith at mcs.anl.gov Mon Apr 20 07:19:12 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 20 Apr 2015 07:19:12 -0500 Subject: [petsc-users] petsc 3.4.3 to 3.5.3 In-Reply-To: <201504200906.t3K96JBL018583@msw1.awe.co.uk> References: <201504200906.t3K96JBL018583@msw1.awe.co.uk> Message-ID: <4CCFD632-5A04-48B4-BB33-338BE9E93AA1@mcs.anl.gov> > On Apr 20, 2015, at 4:06 AM, wrote: > > All, > > I am switching to petsc 3.5.3 and finding some differences with 3.4.3. KSPSetOperators no longer has the MatStructure argument. I see for SAME_PRECONDITIONER that you > now use KSPSetReusePreconditioner with flag set to tru before the call to set to true KSPSetOperators. Is there any need to do anything for the other flags, eg; DIFFERENT_NONZERO_PATTERN, SAME_ NONZERO_PATTERN > or is that now tracked by the Mat object as suggeste in the documentation? You do not have to set any other options; the matrix now keeps track itself of wether the nonzero pattern has changed. Barry > > Thanks, > > Carol > > Dr Carol Brickley > BSc,PhD,ARCS,DIC,MBCS > > Senior Software Engineer > Applied Computer Science > DS+T, > AWE > Aldermaston > Reading > Berkshire > RG7 4PR > > Direct: 0118 9855035 > > ___________________________________________________ ____________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR > From mailinglists at xgm.de Mon Apr 20 07:23:55 2015 From: mailinglists at xgm.de (Florian Lindner) Date: Mon, 20 Apr 2015 14:23:55 +0200 Subject: [petsc-users] Create identity IS In-Reply-To: References: <1496342.C5sGYTRWzh@asaru> Message-ID: <1506248.CURpy2nGkv@asaru> Am Montag, 20. April 2015, 14:57:58 schrieb Lisandro Dalcin: > On 20 April 2015 at 14:35, Florian Lindner wrote: > > Hello, > > > > I use an index set for a row mapping but don't want to use one for the column mapping. > > > > I try to create an idenitity index set to supply to ISLocalToGlobalMappingCreateIS and MatSetLocalToGlobalMapping: > > > > IS is; > > ISCreate(PETSC_COMM_WORLD, &is); > > At this point the index set does not have the type set. > > > ISSetIdentity(is); > > > > but I get an segmentation fault at the last line. > > > > Bad error checking, you should get an error saying that the type is not yet set. > > > What is the best way to create an identity index set? > > > > I would use ISCreateStride(), and perhaps next ISToGeneral() if you > really want a ISGENERAL type. All I want is an identity index set, that I can use with ISLocalToGlobalMappingCreateIS and MatSetLocalToGlobalMapping. Problem is when I know only the local size, not the global size of the matrix ISCreateStride(PETSC_COMM_WORLD, nLocal, 0, 1, &is); followed by ISAllGather(is, &isg); gives an IS like that: 0 0 1 1 2 2 3 3 4 0 5 1 6 2 7 3 so 0 to 3 are mapped twice, while the rest is not mapped. Just using the local IS, without ISAllGather gives an Argument out of range at MatSetValuesLocal. EDIT: Got it now using: ISCreateStride(PETSC_COMM_WORLD, A.ownerRange().second - A.ownerRange().first, A.ownerRange().first, 1, &is); ISSetIdentity(is); ISView(is, PETSC_VIEWER_STDOUT_WORLD); ISAllGather(is, &isg); if (rank==0) ISView(isg, PETSC_VIEWER_STDOUT_SELF); ISLocalToGlobalMapping mapping; ISLocalToGlobalMappingCreateIS(isg, &mapping); MatSetLocalToGlobalMapping(A.matrix, mapping, mapping); based on the owner range. From song.gao2 at mail.mcgill.ca Mon Apr 20 10:03:12 2015 From: song.gao2 at mail.mcgill.ca (Song Gao) Date: Mon, 20 Apr 2015 11:03:12 -0400 Subject: [petsc-users] How to solve a matrix with zero diagonal entries? Message-ID: Dear Petsc-users, I modified the code based on KSP ex1f. I'm solving a 2D stream function vorticity finite element problem. The problem is relatively small, around 2500x2500. I choose to use single processor and LU preconditioner. But the algorithm set some zeros on the diagonals. By checking the PCLU manual page, I think I need shift, the code looks like this: call KSPGetPC(ksp,preconditioner,ierr) call PCSetType(preconditioner,PCLU,ierr) call PCFactorSetShiftAmount(preconditioner, 100.0D0,ierr) call KSPSetFromOptions(ksp,ierr) call KSPSolve(ksp, pet_rhs, pet_sol_update,ierr) But I still get errors here. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot [0]PETSC ERROR: Zero pivot row 4 value 0 tolerance 2.22045e-14 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 [0]PETSC ERROR: ./ex1f on a FEM_project named anakin by sgao Mon Apr 20 10:23:58 2015 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich [0]PETSC ERROR: #1 MatPivotCheck_none() line 622 in /home/cfd/sgao/public/for_FEM/petsc-3.5.3/include/petsc-private/matimpl.h [0]PETSC ERROR: #2 MatPivotCheck() line 641 in /home/cfd/sgao/public/for_FEM/petsc-3.5.3/include/petsc-private/matimpl.h [0]PETSC ERROR: #3 MatLUFactorNumeric_SeqAIJ() line 575 in /home/cfd/sgao/public/for_FEM/petsc-3.5.3/src/mat/impls/aij/seq/aijfact.c [0]PETSC ERROR: #4 MatLUFactorNumeric() line 2893 in /home/cfd/sgao/public/for_FEM/petsc-3.5.3/src/mat/interface/matrix.c [0]PETSC ERROR: #5 PCSetUp_LU() line 152 in /home/cfd/sgao/public/for_FEM/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c [0]PETSC ERROR: #6 PCSetUp() line 902 in /home/cfd/sgao/public/for_FEM/petsc-3.5.3/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #7 KSPSetUp() line 306 in /home/cfd/sgao/public/for_FEM/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #8 KSPSolve() line 418 in /home/cfd/sgao/public/for_FEM/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c I tried other run time options, but none of them works. Some gives Zero pivot errors, some doesn't converge. ./ex1f -pc_factor_shift_type NONZERO -ksp_monitor_true_residual -ksp_max_it 10 ./ex1f -pc_factor_shift_type NONZERO -pc_factor_shift_amount 1 -ksp_monitor_true_residual -ksp_max_it 10 ./ex1f -pc_factor_shift_type POSITIVE_DEFINITE -pc_factor_shift_amount 1 -ksp_monitor_true_residual -ksp_max_it 10 ./ex1f -pc_factor_nonzeros_along_diagonal -ksp_monitor_true_residual -ksp_max_it 10 The matrix on a small grid is attached. Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mat.data Type: application/octet-stream Size: 6197 bytes Desc: not available URL: From siddhesh4godbole at gmail.com Mon Apr 20 10:09:15 2015 From: siddhesh4godbole at gmail.com (siddhesh godbole) Date: Mon, 20 Apr 2015 20:39:15 +0530 Subject: [petsc-users] EPS performance Message-ID: hello i am trying to solve generalized eigenvalue problem using eps solver. i have a monte carlo loop in the program which has to run for about 1000 times. matrix sizes are 15 X 15 as a dummy problem and 23040 X23040 in the real problem. as i run the program i see that eps takes increasingly more amount of time for computation with respect to iterations and number of processors too. i have used all the options to the best of my knowledge and need your help to improve the program in any way possible. i am attaching .c file of mycode with this email. I would be extremely greatful to you for your valuable suggestions. *Siddhesh M Godbole* 5th year Dual Degree, Civil Eng & Applied Mech. IIT Madras -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mycode.c Type: text/x-csrc Size: 12921 bytes Desc: not available URL: From jroman at dsic.upv.es Mon Apr 20 10:17:15 2015 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 20 Apr 2015 17:17:15 +0200 Subject: [petsc-users] EPS performance In-Reply-To: References: Message-ID: <8CC4FEB2-D3BE-45FC-A805-3834C90F6245@dsic.upv.es> El 20/04/2015, a las 17:09, siddhesh godbole escribi?: > hello > > i am trying to solve generalized eigenvalue problem using eps solver. i have a monte carlo loop in the program which has to run for about 1000 times. matrix sizes are 15 X 15 as a dummy problem and 23040 X23040 in the real problem. > as i run the program i see that eps takes increasingly more amount of time for computation with respect to iterations and number of processors too. > i have used all the options to the best of my knowledge and need your help to improve the program in any way possible. > i am attaching .c file of mycode with this email. > > I would be extremely greatful to you for your valuable suggestions. > > > > Siddhesh M Godbole > > 5th year Dual Degree, > Civil Eng & Applied Mech. > IIT Madras > For the large problem, you should consider doing preallocation of matrices. A generalized eigenproblem needs a linear solver. By default EPS uses a sequential direct solver, see FAQ#6 http://slepc.upv.es/documentation/faq.htm and section 3.4.1 of the users manual. In the development version, we have changed this so that the user gets a run-time error when trying to use a sequential direct linear solver in parallel runs. Jose From bsmith at mcs.anl.gov Mon Apr 20 17:23:15 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 20 Apr 2015 17:23:15 -0500 Subject: [petsc-users] Could be a bug on hypre with petsc In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010FE4E5@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE4E5@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: <830A1052-FE13-4489-8453-AEEDA3385571@mcs.anl.gov> It could be a bug or it could just be sloppyness where they send uninitialized data that they never use at the other end. Either way you should submit it to the hypre team so they can decide if they want to fix it. Barry > On Apr 19, 2015, at 6:55 PM, Sun, Hui wrote: > > I'm basically solving Poisson, and using valgrind to check memory issues. I set pctype to PCHYPRE. It works fine if I use one core. But if I use two cores, it gives me the following error messages about the memory, and then it computes results fine. It seems that it has to do with hypre, for if I set pctype to something else such as PCGAMG or PCJACOBI, it does not output any error message at all, regardless of one core or two cores. By the way, I use DM objects for the matrix and ksp. > > > > ==59097== Syscall param writev(vector[...]) points to uninitialised byte(s) > > ==59097== at 0x2F19FF2: writev (in /usr/lib/system/libsystem_kernel.dylib) > > ==59097== by 0x2AF470D: MPIDU_Sock_writev (sock_immed.i:610) > > ==59097== by 0x2ADE3E5: MPIDI_CH3_iStartMsgv (ch3_istartmsgv.c:110) > > ==59097== by 0x2ACEEF8: MPIDI_CH3_EagerContigSend (ch3u_eager.c:182) > > ==59097== by 0x2AD556F: MPID_Send (mpid_send.c:143) > > ==59097== by 0x2920057: MPI_Send (send.c:127) > > ==59097== by 0x1A712B1: hypre_MPI_Send (mpistubs.c:988) > > ==59097== by 0x1A6E77B: hypre_DataExchangeList (exchange_data.c:392) > > ==59097== by 0x194D7C1: hypre_NewCommPkgCreate_core (new_commpkg.c:267) > > ==59097== by 0x195108B: hypre_MatvecCommPkgCreate (par_csr_communication.c:478) > > ==59097== by 0x192020B: hypre_BoomerAMGCreateS (par_strength.c:190) > > ==59097== by 0x189D6AD: hypre_BoomerAMGSetup (par_amg_setup.c:883) > > ==59097== by 0x187BD7A: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:52) > > ==59097== by 0x11EFCE6: PCSetUp_HYPRE (hypre.c:157) > > ==59097== by 0x12766D4: PCSetUp (precon.c:918) > > ==59097== by 0x1385670: KSPSetUp (itfunc.c:330) > > ==59097== by 0x138798B: KSPSolve (itfunc.c:542) > > ==59097== by 0x10003C854: fourPoissonSolve (FourPoisson3D.h:383) > > ==59097== by 0x1000DEA42: main (test.c:473) > > ==59097== Address 0x100a295d8 is 8 bytes inside a block of size 28 alloc'd > > ==59097== at 0xC258: malloc (vg_replace_malloc.c:300) > > ==59097== by 0x1A6F501: hypre_MAlloc (hypre_memory.c:62) > > ==59097== by 0x1A6DDBA: hypre_DataExchangeList (exchange_data.c:212) > > ==59097== by 0x194D7C1: hypre_NewCommPkgCreate_core (new_commpkg.c:267) > > ==59097== by 0x195108B: hypre_MatvecCommPkgCreate (par_csr_communication.c:478) > > ==59097== by 0x192020B: hypre_BoomerAMGCreateS (par_strength.c:190) > > ==59097== by 0x189D6AD: hypre_BoomerAMGSetup (par_amg_setup.c:883) > > ==59097== by 0x187BD7A: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:52) > > ==59097== by 0x11EFCE6: PCSetUp_HYPRE (hypre.c:157) > > ==59097== by 0x12766D4: PCSetUp (precon.c:918) > > ==59097== by 0x1385670: KSPSetUp (itfunc.c:330) > > ==59097== by 0x138798B: KSPSolve (itfunc.c:542) > > ==59097== by 0x10003C854: fourPoissonSolve (FourPoisson3D.h:383) > > ==59097== by 0x1000DEA42: main (test.c:473) > > > > Best, > > Hui > From jychang48 at gmail.com Mon Apr 20 18:32:17 2015 From: jychang48 at gmail.com (Justin Chang) Date: Mon, 20 Apr 2015 18:32:17 -0500 Subject: [petsc-users] Tao iterations In-Reply-To: References: Message-ID: Jason, I am using TaoGetSolutionStatus(tao,&its, ...) and it gives me exactly what I want. However, I seem to be having an issue with blmvm I wrote my own backward euler code for a transient linear diffusion problem with lower bounds >= 0 and upper bounds <= 1. For the first several time steps I am getting its > 0, and it decreases over time due to the nature of the discrete maximum principles. However, at some point my its become 0 and the solution does not "update", which seems to me that TaoSolve is not doing anything after that. This doesn't happen if I were to use tron (my KSP and PC are cg and jacobi respectively). Do you know why this behavior may occur? Thanks, On Tue, Apr 14, 2015 at 9:35 AM, Jason Sarich wrote: > Hi Justin, > > I have pushed these changes to the "next" branch, your code snippet should > work fine there. > > Note that there is also available (since version 3.5.0) the routine > TaoGetSolutionStatus(tao,&its,NULL,NULL,NULL,NULL,NULL) which will provide > the > same information > > Jason > > On Fri, Apr 10, 2015 at 6:28 PM, Justin Chang wrote: > >> Whatever is convenient and/or follow the "PETSc" standards. Something >> similar to SNESGetIterationNumber() or KSPGetIterationNumber() would be >> nice. Ideally I want my code to look like this: >> >> ierr = TaoGetIterationNumber(tao,&its);CHKERRQ(ierr); >> ierr = PetscPrintf(PETSC_COMM_WORLD, "Number of Tao iterations = %D\n", >> its); >> >> Thanks :) >> >> On Fri, Apr 10, 2015 at 5:53 PM, Jason Sarich >> wrote: >> >>> Hi Justin, I'll get this in. I assume that displaying the number of >>> iterations with tao_converged_reason is what you are asking for in >>> particular? Or did you have something else in mind? >>> >>> Jason >>> On Apr 10, 2015 16:42, "Smith, Barry F." wrote: >>> >>>> >>>> Justin, >>>> >>>> Sorry TAO simply doesn't even collect this information currently. >>>> But yes we should definitely make it available! >>>> >>>> Jason, >>>> >>>> Could you please add this; almost all the TaoSolve_xxx() have a >>>> local variable iter; change that to tao->niter (I'm guess this is suppose >>>> to capture this information) and add a TaoGetIterationNumber() and the uses >>>> can access this. Also modify at the end of TaoSolve() -tao_converged_reason >>>> to also print the iteration count. At the same time since you add this you >>>> can add a tao->totalits which would accumulate all iterations over all the >>>> solves for that Tao object and the routine TaoGetTotalIterations() to >>>> access this. Note that TaoSolve() would initialize tao->niter = 0 at the >>>> top. >>>> >>>> Thanks >>>> >>>> Barry >>>> >>>> >>>> >>>> > On Apr 10, 2015, at 4:16 PM, Justin Chang wrote: >>>> > >>>> > Hi all, >>>> > >>>> > Is there a way to generically obtain the number of Tao iterations? I >>>> am looking through the -help options for Tao and I don't see any metric >>>> where you can output this quantity in the manner that you could for SNES or >>>> KSP solves. I am currently using blmvm and tron, and the only way I can see >>>> getting this metric is by outputting -tao_view and/or -tao_monitor and >>>> manually finding this number. I find this cumbersome especially for >>>> transient problems where I would like to simply have this number printed >>>> for each step instead of ending up with unnecessary info. >>>> > >>>> > Thanks, >>>> > >>>> > >>>> > -- >>>> > Justin Chang >>>> > PhD Candidate, Civil Engineering - Computational Sciences >>>> > University of Houston, Department of Civil and Environmental >>>> Engineering >>>> > Houston, TX 77004 >>>> > (512) 963-3262 >>>> >>>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gideon.simpson at gmail.com Mon Apr 20 19:45:27 2015 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Mon, 20 Apr 2015 20:45:27 -0400 Subject: [petsc-users] design philosophy question Message-ID: <840CF8A7-1C54-49CF-B921-F9DFFD38D9D0@gmail.com> In cutting up a petsc code into subroutines, is there any design philosophy as to when to call global routines, like assemble? The problem I have in mind is that I want to build up a matrix, allowing for it to be of MPIAIJ type. Currently, I?m populating it row by row with a loop using MatSetValues. Since this is an uninteresting piece of the code, and it may be reused elsewhere, I was thinking of writing, in a separate file: PetscErrorCode ConstructMat(Mat A) But within ConstructMat, should I only populate the entries? Is it better design to put MatAssemblyBegin/End in that, or in the main program, after this has been called? -gideon From jchang27 at uh.edu Tue Apr 21 00:59:22 2015 From: jchang27 at uh.edu (Justin Chang) Date: Tue, 21 Apr 2015 00:59:22 -0500 Subject: [petsc-users] PetscLogFlop for a sqrt() Message-ID: Hi everyone, I want to manually count the number of flops that take place within my user-defined functions. Say I have to calculate the following: y = sqrt(pow(x[0],2)+pow(x[1],2)+pow(x[2],2)); What should the flop counter be for each call? Thanks, -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From siddhesh4godbole at gmail.com Tue Apr 21 03:10:28 2015 From: siddhesh4godbole at gmail.com (siddhesh godbole) Date: Tue, 21 Apr 2015 13:40:28 +0530 Subject: [petsc-users] new communicators Message-ID: Hello, can i create separate communicator for set of processes and run different programs in those communicator environment at a time? i would like to create _comm1 for 2 procs, _comm2 for 4 procs and so on. how do i do that? is there any reference? is creating comm cost effective? thanks *Siddhesh M Godbole* 5th year Dual Degree, Civil Eng & Applied Mech. IIT Madras -------------- next part -------------- An HTML attachment was scrubbed... URL: From siddhesh4godbole at gmail.com Tue Apr 21 03:21:24 2015 From: siddhesh4godbole at gmail.com (siddhesh godbole) Date: Tue, 21 Apr 2015 13:51:24 +0530 Subject: [petsc-users] MUMPS error Message-ID: Hello, i wanted to use mumps package for parallel direct linear solver in my spectral transformation call to ksp during EPSSolve. i ran this command as per section 3.4.1 in slepc guide. though this message tell me that i need to download mumps i still see the mumps package in my petsc-3.5.2/src/mat/impls/aij/mpi directory. how do i resolve this error? *iitm at iitm:~/Desktop/project$ ./eps13D -st_ksp_type preonly -st_pc_type lu -st_pc_factor_mat_solver_package mumps* [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Matrix format seqaij does not have a solver package mumps for LU. Perhaps you must ./configure with --download-mumps [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 [0]PETSC ERROR: ./eps13D on a arch-linux2-c-debug named iitm by iitm Tue Apr 21 13:45:05 2015 [0]PETSC ERROR: Configure options --with-matlab-dir=/home/iitm/Downloads/matlab14 [0]PETSC ERROR: #1 MatGetFactor() line 3962 in /home/iitm/Downloads/petsc-3.5.2/src/mat/interface/matrix.c [0]PETSC ERROR: #2 PCSetUp_LU() line 125 in /home/iitm/Downloads/petsc-3.5.2/src/ksp/pc/impls/factor/lu/lu.c [0]PETSC ERROR: #3 PCSetUp() line 902 in /home/iitm/Downloads/petsc-3.5.2/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #4 KSPSetUp() line 305 in /home/iitm/Downloads/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #5 STSetUp_Shift() line 132 in /home/iitm/Downloads/slepc-3.5.2/src/sys/classes/st/impls/shift/shift.c [0]PETSC ERROR: #6 STSetUp() line 301 in /home/iitm/Downloads/slepc-3.5.2/src/sys/classes/st/interface/stsolve.c [0]PETSC ERROR: #7 EPSSetUp() line 207 in /home/iitm/Downloads/slepc-3.5.2/src/eps/interface/epssetup.c [0]PETSC ERROR: #8 EPSSolve() line 88 in /home/iitm/Downloads/slepc-3.5.2/src/eps/interface/epssolve.c [0]PETSC ERROR: #9 main() line 252 in /home/iitm/Desktop/project/eps13D.c [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- *Siddhesh M Godbole* 5th year Dual Degree, Civil Eng & Applied Mech. IIT Madras -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Tue Apr 21 03:34:39 2015 From: jroman at dsic.upv.es (Jose E. Roman) Date: Tue, 21 Apr 2015 10:34:39 +0200 Subject: [petsc-users] MUMPS error In-Reply-To: References: Message-ID: <24653E9A-C3C7-493E-A79E-259CD4A0E014@dsic.upv.es> Proceed as the error message says: add --download-mumps in the configure line, together with additional dependency packages, e.g.: $ ./configure --download-mumps --download-scalapack --download-metis --download-parmetis Jose El 21/04/2015, a las 10:21, siddhesh godbole escribi?: > Hello, > > > i wanted to use mumps package for parallel direct linear solver in my spectral transformation call to ksp during EPSSolve. i ran this command as per section 3.4.1 in slepc guide. though this message tell me that i need to download mumps i still see the mumps package in my petsc-3.5.2/src/mat/impls/aij/mpi directory. > how do i resolve this error? > > > > iitm at iitm:~/Desktop/project$ ./eps13D -st_ksp_type preonly -st_pc_type lu -st_pc_factor_mat_solver_package mumps > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Matrix format seqaij does not have a solver package mumps for LU. Perhaps you must ./configure with --download-mumps > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 > [0]PETSC ERROR: ./eps13D on a arch-linux2-c-debug named iitm by iitm Tue Apr 21 13:45:05 2015 > [0]PETSC ERROR: Configure options --with-matlab-dir=/home/iitm/Downloads/matlab14 > [0]PETSC ERROR: #1 MatGetFactor() line 3962 in /home/iitm/Downloads/petsc-3.5.2/src/mat/interface/matrix.c > [0]PETSC ERROR: #2 PCSetUp_LU() line 125 in /home/iitm/Downloads/petsc-3.5.2/src/ksp/pc/impls/factor/lu/lu.c > [0]PETSC ERROR: #3 PCSetUp() line 902 in /home/iitm/Downloads/petsc-3.5.2/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #4 KSPSetUp() line 305 in /home/iitm/Downloads/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #5 STSetUp_Shift() line 132 in /home/iitm/Downloads/slepc-3.5.2/src/sys/classes/st/impls/shift/shift.c > [0]PETSC ERROR: #6 STSetUp() line 301 in /home/iitm/Downloads/slepc-3.5.2/src/sys/classes/st/interface/stsolve.c > [0]PETSC ERROR: #7 EPSSetUp() line 207 in /home/iitm/Downloads/slepc-3.5.2/src/eps/interface/epssetup.c > [0]PETSC ERROR: #8 EPSSolve() line 88 in /home/iitm/Downloads/slepc-3.5.2/src/eps/interface/epssolve.c > [0]PETSC ERROR: #9 main() line 252 in /home/iitm/Desktop/project/eps13D.c > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- > > Siddhesh M Godbole > > 5th year Dual Degree, > Civil Eng & Applied Mech. > IIT Madras From knepley at gmail.com Tue Apr 21 05:33:05 2015 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 21 Apr 2015 05:33:05 -0500 Subject: [petsc-users] PetscLogFlop for a sqrt() In-Reply-To: References: Message-ID: On Tue, Apr 21, 2015 at 12:59 AM, Justin Chang wrote: > Hi everyone, > > I want to manually count the number of flops that take place within my > user-defined functions. Say I have to calculate the following: > > y = sqrt(pow(x[0],2)+pow(x[1],2)+pow(x[2],2)); > > What should the flop counter be for each call? > 1) I would use PetscSqr() instead of pow(*, 2) and PetscSqrtReal/Scalar() 2) Sqrt is obviously more expensive than a multiply, but there is no good standard, so we just count it as 1 flop Thanks, Matt > Thanks, > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Apr 21 05:35:13 2015 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 21 Apr 2015 05:35:13 -0500 Subject: [petsc-users] new communicators In-Reply-To: References: Message-ID: On Tue, Apr 21, 2015 at 3:10 AM, siddhesh godbole < siddhesh4godbole at gmail.com> wrote: > Hello, > > > can i create separate communicator for set of processes and run different > programs in those communicator environment at a time? i would like to > create _comm1 for 2 procs, _comm2 for 4 procs and so on. how do i do that? > is there any reference? is creating comm cost effective? > Yes, you can use something like MPI_Comm_split() to get new subcommunicators, and then create objects with those that will just run on those processes. For example, you can have 2 SNESes on subcomm1 and subcomm2 to do two concurrent solves. Matt > thanks > > *Siddhesh M Godbole* > > 5th year Dual Degree, > Civil Eng & Applied Mech. > IIT Madras > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jason.sarich at gmail.com Tue Apr 21 09:28:02 2015 From: jason.sarich at gmail.com (Jason Sarich) Date: Tue, 21 Apr 2015 09:28:02 -0500 Subject: [petsc-users] Tao iterations In-Reply-To: References: Message-ID: Hi Justin, what reason is blmvm giving for stopping the solve? (you can use -tao_view or -tao_converged_reason to get this) Jason On Mon, Apr 20, 2015 at 6:32 PM, Justin Chang wrote: > Jason, > > I am using TaoGetSolutionStatus(tao,&its, ...) and it gives me exactly > what I want. However, I seem to be having an issue with blmvm > > I wrote my own backward euler code for a transient linear diffusion > problem with lower bounds >= 0 and upper bounds <= 1. For the first several > time steps I am getting its > 0, and it decreases over time due to the > nature of the discrete maximum principles. However, at some point my its > become 0 and the solution does not "update", which seems to me that > TaoSolve is not doing anything after that. This doesn't happen if I were to > use tron (my KSP and PC are cg and jacobi respectively). > > Do you know why this behavior may occur? > > Thanks, > > On Tue, Apr 14, 2015 at 9:35 AM, Jason Sarich > wrote: > >> Hi Justin, >> >> I have pushed these changes to the "next" branch, your code snippet >> should work fine there. >> >> Note that there is also available (since version 3.5.0) the routine >> TaoGetSolutionStatus(tao,&its,NULL,NULL,NULL,NULL,NULL) which will provide >> the >> same information >> >> Jason >> >> On Fri, Apr 10, 2015 at 6:28 PM, Justin Chang >> wrote: >> >>> Whatever is convenient and/or follow the "PETSc" standards. Something >>> similar to SNESGetIterationNumber() or KSPGetIterationNumber() would be >>> nice. Ideally I want my code to look like this: >>> >>> ierr = TaoGetIterationNumber(tao,&its);CHKERRQ(ierr); >>> ierr = PetscPrintf(PETSC_COMM_WORLD, "Number of Tao iterations = %D\n", >>> its); >>> >>> Thanks :) >>> >>> On Fri, Apr 10, 2015 at 5:53 PM, Jason Sarich >>> wrote: >>> >>>> Hi Justin, I'll get this in. I assume that displaying the number of >>>> iterations with tao_converged_reason is what you are asking for in >>>> particular? Or did you have something else in mind? >>>> >>>> Jason >>>> On Apr 10, 2015 16:42, "Smith, Barry F." wrote: >>>> >>>>> >>>>> Justin, >>>>> >>>>> Sorry TAO simply doesn't even collect this information currently. >>>>> But yes we should definitely make it available! >>>>> >>>>> Jason, >>>>> >>>>> Could you please add this; almost all the TaoSolve_xxx() have a >>>>> local variable iter; change that to tao->niter (I'm guess this is suppose >>>>> to capture this information) and add a TaoGetIterationNumber() and the uses >>>>> can access this. Also modify at the end of TaoSolve() -tao_converged_reason >>>>> to also print the iteration count. At the same time since you add this you >>>>> can add a tao->totalits which would accumulate all iterations over all the >>>>> solves for that Tao object and the routine TaoGetTotalIterations() to >>>>> access this. Note that TaoSolve() would initialize tao->niter = 0 at the >>>>> top. >>>>> >>>>> Thanks >>>>> >>>>> Barry >>>>> >>>>> >>>>> >>>>> > On Apr 10, 2015, at 4:16 PM, Justin Chang wrote: >>>>> > >>>>> > Hi all, >>>>> > >>>>> > Is there a way to generically obtain the number of Tao iterations? I >>>>> am looking through the -help options for Tao and I don't see any metric >>>>> where you can output this quantity in the manner that you could for SNES or >>>>> KSP solves. I am currently using blmvm and tron, and the only way I can see >>>>> getting this metric is by outputting -tao_view and/or -tao_monitor and >>>>> manually finding this number. I find this cumbersome especially for >>>>> transient problems where I would like to simply have this number printed >>>>> for each step instead of ending up with unnecessary info. >>>>> > >>>>> > Thanks, >>>>> > >>>>> > >>>>> > -- >>>>> > Justin Chang >>>>> > PhD Candidate, Civil Engineering - Computational Sciences >>>>> > University of Houston, Department of Civil and Environmental >>>>> Engineering >>>>> > Houston, TX 77004 >>>>> > (512) 963-3262 >>>>> >>>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jychang48 at gmail.com Tue Apr 21 09:48:43 2015 From: jychang48 at gmail.com (Justin Chang) Date: Tue, 21 Apr 2015 09:48:43 -0500 Subject: [petsc-users] Tao iterations In-Reply-To: References: Message-ID: Time step 1: Tao Object: 1 MPI processes type: blmvm Gradient steps: 0 TaoLineSearch Object: 1 MPI processes type: more-thuente Active Set subset type: subvec convergence tolerances: fatol=0.0001, frtol=0.0001 convergence tolerances: gatol=0, steptol=0, gttol=0 Residual in Function/Gradient:=0.0663148 Objective value=-55.5945 total number of iterations=35, (max: 2000) total number of function/gradient evaluations=37, (max: 4000) Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol Time step 2: Tao Object: 1 MPI processes type: blmvm Gradient steps: 0 TaoLineSearch Object: 1 MPI processes type: more-thuente Active Set subset type: subvec convergence tolerances: fatol=0.0001, frtol=0.0001 convergence tolerances: gatol=0, steptol=0, gttol=0 Residual in Function/Gradient:=0.0682307 Objective value=-66.9675 total number of iterations=23, (max: 2000) total number of function/gradient evaluations=25, (max: 4000) Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol Time step 3: Tao Object: 1 MPI processes type: blmvm Gradient steps: 0 TaoLineSearch Object: 1 MPI processes type: more-thuente Active Set subset type: subvec convergence tolerances: fatol=0.0001, frtol=0.0001 convergence tolerances: gatol=0, steptol=0, gttol=0 Residual in Function/Gradient:=0.0680522 Objective value=-71.8211 total number of iterations=19, (max: 2000) total number of function/gradient evaluations=22, (max: 4000) Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol Time step 4: Tao Object: 1 MPI processes type: blmvm Gradient steps: 0 TaoLineSearch Object: 1 MPI processes type: more-thuente Active Set subset type: subvec convergence tolerances: fatol=0.0001, frtol=0.0001 convergence tolerances: gatol=0, steptol=0, gttol=0 Residual in Function/Gradient:=0.0551556 Objective value=-75.1252 total number of iterations=18, (max: 2000) total number of function/gradient evaluations=20, (max: 4000) Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol Time step 5: Tao Object: 1 MPI processes type: blmvm Gradient steps: 0 TaoLineSearch Object: 1 MPI processes type: more-thuente Active Set subset type: subvec convergence tolerances: fatol=0.0001, frtol=0.0001 convergence tolerances: gatol=0, steptol=0, gttol=0 Residual in Function/Gradient:=0.0675667 Objective value=-77.4414 total number of iterations=6, (max: 2000) total number of function/gradient evaluations=8, (max: 4000) Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol Time step 6: Tao Object: 1 MPI processes type: blmvm Gradient steps: 0 TaoLineSearch Object: 1 MPI processes type: more-thuente Active Set subset type: subvec convergence tolerances: fatol=0.0001, frtol=0.0001 convergence tolerances: gatol=0, steptol=0, gttol=0 Residual in Function/Gradient:=0.059143 Objective value=-79.5007 total number of iterations=3, (max: 2000) total number of function/gradient evaluations=5, (max: 4000) Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol Time step 7: Tao Object: 1 MPI processes type: blmvm Gradient steps: 0 TaoLineSearch Object: 1 MPI processes type: more-thuente Active Set subset type: subvec convergence tolerances: fatol=0.0001, frtol=0.0001 convergence tolerances: gatol=0, steptol=0, gttol=0 Residual in Function/Gradient:=0.0433683 Objective value=-81.3546 total number of iterations=5, (max: 2000) total number of function/gradient evaluations=8, (max: 4000) Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol Time step 8: Tao Object: 1 MPI processes type: blmvm Gradient steps: 0 TaoLineSearch Object: 1 MPI processes type: more-thuente Active Set subset type: subvec convergence tolerances: fatol=0.0001, frtol=0.0001 convergence tolerances: gatol=0, steptol=0, gttol=0 Residual in Function/Gradient:=0.0840676 Objective value=-82.9382 total number of iterations=0, (max: 2000) total number of function/gradient evaluations=1, (max: 4000) Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol Time step 9: Tao Object: 1 MPI processes type: blmvm Gradient steps: 0 TaoLineSearch Object: 1 MPI processes type: more-thuente Active Set subset type: subvec convergence tolerances: fatol=0.0001, frtol=0.0001 convergence tolerances: gatol=0, steptol=0, gttol=0 Residual in Function/Gradient:=0.0840676 Objective value=-82.9382 total number of iterations=0, (max: 2000) total number of function/gradient evaluations=1, (max: 4000) Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol Time step 10: Tao Object: 1 MPI processes type: blmvm Gradient steps: 0 TaoLineSearch Object: 1 MPI processes type: more-thuente Active Set subset type: subvec convergence tolerances: fatol=0.0001, frtol=0.0001 convergence tolerances: gatol=0, steptol=0, gttol=0 Residual in Function/Gradient:=0.0840676 Objective value=-82.9382 total number of iterations=0, (max: 2000) total number of function/gradient evaluations=1, (max: 4000) Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol On Tue, Apr 21, 2015 at 9:28 AM, Jason Sarich wrote: > Hi Justin, > > what reason is blmvm giving for stopping the solve? (you can use -tao_view > or -tao_converged_reason to get this) > > Jason > > On Mon, Apr 20, 2015 at 6:32 PM, Justin Chang wrote: > >> Jason, >> >> I am using TaoGetSolutionStatus(tao,&its, ...) and it gives me exactly >> what I want. However, I seem to be having an issue with blmvm >> >> I wrote my own backward euler code for a transient linear diffusion >> problem with lower bounds >= 0 and upper bounds <= 1. For the first several >> time steps I am getting its > 0, and it decreases over time due to the >> nature of the discrete maximum principles. However, at some point my its >> become 0 and the solution does not "update", which seems to me that >> TaoSolve is not doing anything after that. This doesn't happen if I were to >> use tron (my KSP and PC are cg and jacobi respectively). >> >> Do you know why this behavior may occur? >> >> Thanks, >> >> On Tue, Apr 14, 2015 at 9:35 AM, Jason Sarich >> wrote: >> >>> Hi Justin, >>> >>> I have pushed these changes to the "next" branch, your code snippet >>> should work fine there. >>> >>> Note that there is also available (since version 3.5.0) the routine >>> TaoGetSolutionStatus(tao,&its,NULL,NULL,NULL,NULL,NULL) which will provide >>> the >>> same information >>> >>> Jason >>> >>> On Fri, Apr 10, 2015 at 6:28 PM, Justin Chang >>> wrote: >>> >>>> Whatever is convenient and/or follow the "PETSc" standards. >>>> Something similar to SNESGetIterationNumber() or KSPGetIterationNumber() >>>> would be nice. Ideally I want my code to look like this: >>>> >>>> ierr = TaoGetIterationNumber(tao,&its);CHKERRQ(ierr); >>>> ierr = PetscPrintf(PETSC_COMM_WORLD, "Number of Tao iterations = >>>> %D\n", its); >>>> >>>> Thanks :) >>>> >>>> On Fri, Apr 10, 2015 at 5:53 PM, Jason Sarich >>>> wrote: >>>> >>>>> Hi Justin, I'll get this in. I assume that displaying the number of >>>>> iterations with tao_converged_reason is what you are asking for in >>>>> particular? Or did you have something else in mind? >>>>> >>>>> Jason >>>>> On Apr 10, 2015 16:42, "Smith, Barry F." wrote: >>>>> >>>>>> >>>>>> Justin, >>>>>> >>>>>> Sorry TAO simply doesn't even collect this information currently. >>>>>> But yes we should definitely make it available! >>>>>> >>>>>> Jason, >>>>>> >>>>>> Could you please add this; almost all the TaoSolve_xxx() have a >>>>>> local variable iter; change that to tao->niter (I'm guess this is suppose >>>>>> to capture this information) and add a TaoGetIterationNumber() and the uses >>>>>> can access this. Also modify at the end of TaoSolve() -tao_converged_reason >>>>>> to also print the iteration count. At the same time since you add this you >>>>>> can add a tao->totalits which would accumulate all iterations over all the >>>>>> solves for that Tao object and the routine TaoGetTotalIterations() to >>>>>> access this. Note that TaoSolve() would initialize tao->niter = 0 at the >>>>>> top. >>>>>> >>>>>> Thanks >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>> >>>>>> > On Apr 10, 2015, at 4:16 PM, Justin Chang wrote: >>>>>> > >>>>>> > Hi all, >>>>>> > >>>>>> > Is there a way to generically obtain the number of Tao iterations? >>>>>> I am looking through the -help options for Tao and I don't see any metric >>>>>> where you can output this quantity in the manner that you could for SNES or >>>>>> KSP solves. I am currently using blmvm and tron, and the only way I can see >>>>>> getting this metric is by outputting -tao_view and/or -tao_monitor and >>>>>> manually finding this number. I find this cumbersome especially for >>>>>> transient problems where I would like to simply have this number printed >>>>>> for each step instead of ending up with unnecessary info. >>>>>> > >>>>>> > Thanks, >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > Justin Chang >>>>>> > PhD Candidate, Civil Engineering - Computational Sciences >>>>>> > University of Houston, Department of Civil and Environmental >>>>>> Engineering >>>>>> > Houston, TX 77004 >>>>>> > (512) 963-3262 >>>>>> >>>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jason.sarich at gmail.com Tue Apr 21 09:52:32 2015 From: jason.sarich at gmail.com (Jason Sarich) Date: Tue, 21 Apr 2015 09:52:32 -0500 Subject: [petsc-users] Tao iterations In-Reply-To: <3112142f40d64968ae157f4a5e819353@LUCKMAN.anl.gov> References: <3112142f40d64968ae157f4a5e819353@LUCKMAN.anl.gov> Message-ID: Hi Justin, blmvm believes that it is already sufficiently close to a minimum, so it doesn't do anything. You may need to tighten some of the tolerance to force an iteration. Jason On Tue, Apr 21, 2015 at 9:48 AM, Justin Chang wrote: > Time step 1: > > Tao Object: 1 MPI processes > type: blmvm > Gradient steps: 0 > TaoLineSearch Object: 1 MPI processes > type: more-thuente > Active Set subset type: subvec > convergence tolerances: fatol=0.0001, frtol=0.0001 > convergence tolerances: gatol=0, steptol=0, gttol=0 > Residual in Function/Gradient:=0.0663148 > Objective value=-55.5945 > total number of iterations=35, (max: 2000) > total number of function/gradient evaluations=37, (max: 4000) > Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol > > Time step 2: > > Tao Object: 1 MPI processes > type: blmvm > Gradient steps: 0 > TaoLineSearch Object: 1 MPI processes > type: more-thuente > Active Set subset type: subvec > convergence tolerances: fatol=0.0001, frtol=0.0001 > convergence tolerances: gatol=0, steptol=0, gttol=0 > Residual in Function/Gradient:=0.0682307 > Objective value=-66.9675 > total number of iterations=23, (max: 2000) > total number of function/gradient evaluations=25, (max: 4000) > Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol > > Time step 3: > > Tao Object: 1 MPI processes > type: blmvm > Gradient steps: 0 > TaoLineSearch Object: 1 MPI processes > type: more-thuente > Active Set subset type: subvec > convergence tolerances: fatol=0.0001, frtol=0.0001 > convergence tolerances: gatol=0, steptol=0, gttol=0 > Residual in Function/Gradient:=0.0680522 > Objective value=-71.8211 > total number of iterations=19, (max: 2000) > total number of function/gradient evaluations=22, (max: 4000) > Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol > > Time step 4: > > Tao Object: 1 MPI processes > type: blmvm > Gradient steps: 0 > TaoLineSearch Object: 1 MPI processes > type: more-thuente > Active Set subset type: subvec > convergence tolerances: fatol=0.0001, frtol=0.0001 > convergence tolerances: gatol=0, steptol=0, gttol=0 > Residual in Function/Gradient:=0.0551556 > Objective value=-75.1252 > total number of iterations=18, (max: 2000) > total number of function/gradient evaluations=20, (max: 4000) > Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol > > Time step 5: > > Tao Object: 1 MPI processes > type: blmvm > Gradient steps: 0 > TaoLineSearch Object: 1 MPI processes > type: more-thuente > Active Set subset type: subvec > convergence tolerances: fatol=0.0001, frtol=0.0001 > convergence tolerances: gatol=0, steptol=0, gttol=0 > Residual in Function/Gradient:=0.0675667 > Objective value=-77.4414 > total number of iterations=6, (max: 2000) > total number of function/gradient evaluations=8, (max: 4000) > Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol > > Time step 6: > > Tao Object: 1 MPI processes > type: blmvm > Gradient steps: 0 > TaoLineSearch Object: 1 MPI processes > type: more-thuente > Active Set subset type: subvec > convergence tolerances: fatol=0.0001, frtol=0.0001 > convergence tolerances: gatol=0, steptol=0, gttol=0 > Residual in Function/Gradient:=0.059143 > Objective value=-79.5007 > total number of iterations=3, (max: 2000) > total number of function/gradient evaluations=5, (max: 4000) > Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol > > Time step 7: > > Tao Object: 1 MPI processes > type: blmvm > Gradient steps: 0 > TaoLineSearch Object: 1 MPI processes > type: more-thuente > Active Set subset type: subvec > convergence tolerances: fatol=0.0001, frtol=0.0001 > convergence tolerances: gatol=0, steptol=0, gttol=0 > Residual in Function/Gradient:=0.0433683 > Objective value=-81.3546 > total number of iterations=5, (max: 2000) > total number of function/gradient evaluations=8, (max: 4000) > Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol > > Time step 8: > > Tao Object: 1 MPI processes > type: blmvm > Gradient steps: 0 > TaoLineSearch Object: 1 MPI processes > type: more-thuente > Active Set subset type: subvec > convergence tolerances: fatol=0.0001, frtol=0.0001 > convergence tolerances: gatol=0, steptol=0, gttol=0 > Residual in Function/Gradient:=0.0840676 > Objective value=-82.9382 > total number of iterations=0, (max: 2000) > total number of function/gradient evaluations=1, (max: 4000) > Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol > > Time step 9: > > Tao Object: 1 MPI processes > type: blmvm > Gradient steps: 0 > TaoLineSearch Object: 1 MPI processes > type: more-thuente > Active Set subset type: subvec > convergence tolerances: fatol=0.0001, frtol=0.0001 > convergence tolerances: gatol=0, steptol=0, gttol=0 > Residual in Function/Gradient:=0.0840676 > Objective value=-82.9382 > total number of iterations=0, (max: 2000) > total number of function/gradient evaluations=1, (max: 4000) > Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol > > Time step 10: > > Tao Object: 1 MPI processes > type: blmvm > Gradient steps: 0 > TaoLineSearch Object: 1 MPI processes > type: more-thuente > Active Set subset type: subvec > convergence tolerances: fatol=0.0001, frtol=0.0001 > convergence tolerances: gatol=0, steptol=0, gttol=0 > Residual in Function/Gradient:=0.0840676 > Objective value=-82.9382 > total number of iterations=0, (max: 2000) > total number of function/gradient evaluations=1, (max: 4000) > Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol > > > > > > On Tue, Apr 21, 2015 at 9:28 AM, Jason Sarich > wrote: > >> Hi Justin, >> >> what reason is blmvm giving for stopping the solve? (you can use >> -tao_view or -tao_converged_reason to get this) >> >> Jason >> >> On Mon, Apr 20, 2015 at 6:32 PM, Justin Chang >> wrote: >> >>> Jason, >>> >>> I am using TaoGetSolutionStatus(tao,&its, ...) and it gives me exactly >>> what I want. However, I seem to be having an issue with blmvm >>> >>> I wrote my own backward euler code for a transient linear diffusion >>> problem with lower bounds >= 0 and upper bounds <= 1. For the first several >>> time steps I am getting its > 0, and it decreases over time due to the >>> nature of the discrete maximum principles. However, at some point my its >>> become 0 and the solution does not "update", which seems to me that >>> TaoSolve is not doing anything after that. This doesn't happen if I were to >>> use tron (my KSP and PC are cg and jacobi respectively). >>> >>> Do you know why this behavior may occur? >>> >>> Thanks, >>> >>> On Tue, Apr 14, 2015 at 9:35 AM, Jason Sarich >>> wrote: >>> >>>> Hi Justin, >>>> >>>> I have pushed these changes to the "next" branch, your code snippet >>>> should work fine there. >>>> >>>> Note that there is also available (since version 3.5.0) the routine >>>> TaoGetSolutionStatus(tao,&its,NULL,NULL,NULL,NULL,NULL) which will provide >>>> the >>>> same information >>>> >>>> Jason >>>> >>>> On Fri, Apr 10, 2015 at 6:28 PM, Justin Chang >>>> wrote: >>>> >>>>> Whatever is convenient and/or follow the "PETSc" standards. >>>>> Something similar to SNESGetIterationNumber() or KSPGetIterationNumber() >>>>> would be nice. Ideally I want my code to look like this: >>>>> >>>>> ierr = TaoGetIterationNumber(tao,&its);CHKERRQ(ierr); >>>>> ierr = PetscPrintf(PETSC_COMM_WORLD, "Number of Tao iterations = >>>>> %D\n", its); >>>>> >>>>> Thanks :) >>>>> >>>>> On Fri, Apr 10, 2015 at 5:53 PM, Jason Sarich >>>>> wrote: >>>>> >>>>>> Hi Justin, I'll get this in. I assume that displaying the number of >>>>>> iterations with tao_converged_reason is what you are asking for in >>>>>> particular? Or did you have something else in mind? >>>>>> >>>>>> Jason >>>>>> On Apr 10, 2015 16:42, "Smith, Barry F." wrote: >>>>>> >>>>>>> >>>>>>> Justin, >>>>>>> >>>>>>> Sorry TAO simply doesn't even collect this information >>>>>>> currently. But yes we should definitely make it available! >>>>>>> >>>>>>> Jason, >>>>>>> >>>>>>> Could you please add this; almost all the TaoSolve_xxx() have a >>>>>>> local variable iter; change that to tao->niter (I'm guess this is suppose >>>>>>> to capture this information) and add a TaoGetIterationNumber() and the uses >>>>>>> can access this. Also modify at the end of TaoSolve() -tao_converged_reason >>>>>>> to also print the iteration count. At the same time since you add this you >>>>>>> can add a tao->totalits which would accumulate all iterations over all the >>>>>>> solves for that Tao object and the routine TaoGetTotalIterations() to >>>>>>> access this. Note that TaoSolve() would initialize tao->niter = 0 at the >>>>>>> top. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> >>>>>>> >>>>>>> > On Apr 10, 2015, at 4:16 PM, Justin Chang wrote: >>>>>>> > >>>>>>> > Hi all, >>>>>>> > >>>>>>> > Is there a way to generically obtain the number of Tao iterations? >>>>>>> I am looking through the -help options for Tao and I don't see any metric >>>>>>> where you can output this quantity in the manner that you could for SNES or >>>>>>> KSP solves. I am currently using blmvm and tron, and the only way I can see >>>>>>> getting this metric is by outputting -tao_view and/or -tao_monitor and >>>>>>> manually finding this number. I find this cumbersome especially for >>>>>>> transient problems where I would like to simply have this number printed >>>>>>> for each step instead of ending up with unnecessary info. >>>>>>> > >>>>>>> > Thanks, >>>>>>> > >>>>>>> > >>>>>>> > -- >>>>>>> > Justin Chang >>>>>>> > PhD Candidate, Civil Engineering - Computational Sciences >>>>>>> > University of Houston, Department of Civil and Environmental >>>>>>> Engineering >>>>>>> > Houston, TX 77004 >>>>>>> > (512) 963-3262 >>>>>>> >>>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Apr 21 10:00:26 2015 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 21 Apr 2015 10:00:26 -0500 Subject: [petsc-users] design philosophy question In-Reply-To: <840CF8A7-1C54-49CF-B921-F9DFFD38D9D0@gmail.com> References: <840CF8A7-1C54-49CF-B921-F9DFFD38D9D0@gmail.com> Message-ID: On Mon, Apr 20, 2015 at 7:45 PM, Gideon Simpson wrote: > In cutting up a petsc code into subroutines, is there any design > philosophy as to when to call global routines, like assemble? The problem > I have in mind is that I want to build up a matrix, allowing for it to be > of MPIAIJ type. Currently, I?m populating it row by row with a loop using > MatSetValues. Since this is an uninteresting piece of the code, and it may > be reused elsewhere, I was thinking of writing, in a separate file: > > PetscErrorCode ConstructMat(Mat A) > > But within ConstructMat, should I only populate the entries? Is it better > design to put MatAssemblyBegin/End in that, or in the main program, after > this has been called? I think if you plan to compose functions like these, then leave the Assembly() calls outside. However, if you plan to mix ADD and INSERT calls, then you have to at least call Assembly(A, FLUSH_ASSEMBLY) in the function. Thanks, Matt > -gideon > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jychang48 at gmail.com Tue Apr 21 10:33:44 2015 From: jychang48 at gmail.com (Justin Chang) Date: Tue, 21 Apr 2015 10:33:44 -0500 Subject: [petsc-users] Tao iterations In-Reply-To: References: <3112142f40d64968ae157f4a5e819353@LUCKMAN.anl.gov> Message-ID: Jason, Tightening the tolerances did the trick. Thanks. Though I do have a couple more related questions: 1) Is there a general guideline for choosing tron over blmvm or vice versa? Also is there another tao type that is also suitable given only bounded constraints? 2) Is it possible to obtain the total number of KSP and/or PG iterations from tron? Thanks, Justin On Tue, Apr 21, 2015 at 9:52 AM, Jason Sarich wrote: > Hi Justin, > > blmvm believes that it is already sufficiently close to a minimum, so it > doesn't do anything. You may need to tighten some of the tolerance to force > an iteration. > > Jason > > > On Tue, Apr 21, 2015 at 9:48 AM, Justin Chang wrote: > >> Time step 1: >> >> Tao Object: 1 MPI processes >> type: blmvm >> Gradient steps: 0 >> TaoLineSearch Object: 1 MPI processes >> type: more-thuente >> Active Set subset type: subvec >> convergence tolerances: fatol=0.0001, frtol=0.0001 >> convergence tolerances: gatol=0, steptol=0, gttol=0 >> Residual in Function/Gradient:=0.0663148 >> Objective value=-55.5945 >> total number of iterations=35, (max: 2000) >> total number of function/gradient evaluations=37, (max: 4000) >> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >> >> Time step 2: >> >> Tao Object: 1 MPI processes >> type: blmvm >> Gradient steps: 0 >> TaoLineSearch Object: 1 MPI processes >> type: more-thuente >> Active Set subset type: subvec >> convergence tolerances: fatol=0.0001, frtol=0.0001 >> convergence tolerances: gatol=0, steptol=0, gttol=0 >> Residual in Function/Gradient:=0.0682307 >> Objective value=-66.9675 >> total number of iterations=23, (max: 2000) >> total number of function/gradient evaluations=25, (max: 4000) >> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >> >> Time step 3: >> >> Tao Object: 1 MPI processes >> type: blmvm >> Gradient steps: 0 >> TaoLineSearch Object: 1 MPI processes >> type: more-thuente >> Active Set subset type: subvec >> convergence tolerances: fatol=0.0001, frtol=0.0001 >> convergence tolerances: gatol=0, steptol=0, gttol=0 >> Residual in Function/Gradient:=0.0680522 >> Objective value=-71.8211 >> total number of iterations=19, (max: 2000) >> total number of function/gradient evaluations=22, (max: 4000) >> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >> >> Time step 4: >> >> Tao Object: 1 MPI processes >> type: blmvm >> Gradient steps: 0 >> TaoLineSearch Object: 1 MPI processes >> type: more-thuente >> Active Set subset type: subvec >> convergence tolerances: fatol=0.0001, frtol=0.0001 >> convergence tolerances: gatol=0, steptol=0, gttol=0 >> Residual in Function/Gradient:=0.0551556 >> Objective value=-75.1252 >> total number of iterations=18, (max: 2000) >> total number of function/gradient evaluations=20, (max: 4000) >> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >> >> Time step 5: >> >> Tao Object: 1 MPI processes >> type: blmvm >> Gradient steps: 0 >> TaoLineSearch Object: 1 MPI processes >> type: more-thuente >> Active Set subset type: subvec >> convergence tolerances: fatol=0.0001, frtol=0.0001 >> convergence tolerances: gatol=0, steptol=0, gttol=0 >> Residual in Function/Gradient:=0.0675667 >> Objective value=-77.4414 >> total number of iterations=6, (max: 2000) >> total number of function/gradient evaluations=8, (max: 4000) >> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >> >> Time step 6: >> >> Tao Object: 1 MPI processes >> type: blmvm >> Gradient steps: 0 >> TaoLineSearch Object: 1 MPI processes >> type: more-thuente >> Active Set subset type: subvec >> convergence tolerances: fatol=0.0001, frtol=0.0001 >> convergence tolerances: gatol=0, steptol=0, gttol=0 >> Residual in Function/Gradient:=0.059143 >> Objective value=-79.5007 >> total number of iterations=3, (max: 2000) >> total number of function/gradient evaluations=5, (max: 4000) >> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >> >> Time step 7: >> >> Tao Object: 1 MPI processes >> type: blmvm >> Gradient steps: 0 >> TaoLineSearch Object: 1 MPI processes >> type: more-thuente >> Active Set subset type: subvec >> convergence tolerances: fatol=0.0001, frtol=0.0001 >> convergence tolerances: gatol=0, steptol=0, gttol=0 >> Residual in Function/Gradient:=0.0433683 >> Objective value=-81.3546 >> total number of iterations=5, (max: 2000) >> total number of function/gradient evaluations=8, (max: 4000) >> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >> >> Time step 8: >> >> Tao Object: 1 MPI processes >> type: blmvm >> Gradient steps: 0 >> TaoLineSearch Object: 1 MPI processes >> type: more-thuente >> Active Set subset type: subvec >> convergence tolerances: fatol=0.0001, frtol=0.0001 >> convergence tolerances: gatol=0, steptol=0, gttol=0 >> Residual in Function/Gradient:=0.0840676 >> Objective value=-82.9382 >> total number of iterations=0, (max: 2000) >> total number of function/gradient evaluations=1, (max: 4000) >> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >> >> Time step 9: >> >> Tao Object: 1 MPI processes >> type: blmvm >> Gradient steps: 0 >> TaoLineSearch Object: 1 MPI processes >> type: more-thuente >> Active Set subset type: subvec >> convergence tolerances: fatol=0.0001, frtol=0.0001 >> convergence tolerances: gatol=0, steptol=0, gttol=0 >> Residual in Function/Gradient:=0.0840676 >> Objective value=-82.9382 >> total number of iterations=0, (max: 2000) >> total number of function/gradient evaluations=1, (max: 4000) >> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >> >> Time step 10: >> >> Tao Object: 1 MPI processes >> type: blmvm >> Gradient steps: 0 >> TaoLineSearch Object: 1 MPI processes >> type: more-thuente >> Active Set subset type: subvec >> convergence tolerances: fatol=0.0001, frtol=0.0001 >> convergence tolerances: gatol=0, steptol=0, gttol=0 >> Residual in Function/Gradient:=0.0840676 >> Objective value=-82.9382 >> total number of iterations=0, (max: 2000) >> total number of function/gradient evaluations=1, (max: 4000) >> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >> >> >> >> >> >> On Tue, Apr 21, 2015 at 9:28 AM, Jason Sarich >> wrote: >> >>> Hi Justin, >>> >>> what reason is blmvm giving for stopping the solve? (you can use >>> -tao_view or -tao_converged_reason to get this) >>> >>> Jason >>> >>> On Mon, Apr 20, 2015 at 6:32 PM, Justin Chang >>> wrote: >>> >>>> Jason, >>>> >>>> I am using TaoGetSolutionStatus(tao,&its, ...) and it gives me exactly >>>> what I want. However, I seem to be having an issue with blmvm >>>> >>>> I wrote my own backward euler code for a transient linear diffusion >>>> problem with lower bounds >= 0 and upper bounds <= 1. For the first several >>>> time steps I am getting its > 0, and it decreases over time due to the >>>> nature of the discrete maximum principles. However, at some point my its >>>> become 0 and the solution does not "update", which seems to me that >>>> TaoSolve is not doing anything after that. This doesn't happen if I were to >>>> use tron (my KSP and PC are cg and jacobi respectively). >>>> >>>> Do you know why this behavior may occur? >>>> >>>> Thanks, >>>> >>>> On Tue, Apr 14, 2015 at 9:35 AM, Jason Sarich >>>> wrote: >>>> >>>>> Hi Justin, >>>>> >>>>> I have pushed these changes to the "next" branch, your code snippet >>>>> should work fine there. >>>>> >>>>> Note that there is also available (since version 3.5.0) the routine >>>>> TaoGetSolutionStatus(tao,&its,NULL,NULL,NULL,NULL,NULL) which will provide >>>>> the >>>>> same information >>>>> >>>>> Jason >>>>> >>>>> On Fri, Apr 10, 2015 at 6:28 PM, Justin Chang >>>>> wrote: >>>>> >>>>>> Whatever is convenient and/or follow the "PETSc" standards. >>>>>> Something similar to SNESGetIterationNumber() or KSPGetIterationNumber() >>>>>> would be nice. Ideally I want my code to look like this: >>>>>> >>>>>> ierr = TaoGetIterationNumber(tao,&its);CHKERRQ(ierr); >>>>>> ierr = PetscPrintf(PETSC_COMM_WORLD, "Number of Tao iterations = >>>>>> %D\n", its); >>>>>> >>>>>> Thanks :) >>>>>> >>>>>> On Fri, Apr 10, 2015 at 5:53 PM, Jason Sarich >>>>> > wrote: >>>>>> >>>>>>> Hi Justin, I'll get this in. I assume that displaying the number of >>>>>>> iterations with tao_converged_reason is what you are asking for in >>>>>>> particular? Or did you have something else in mind? >>>>>>> >>>>>>> Jason >>>>>>> On Apr 10, 2015 16:42, "Smith, Barry F." >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> Justin, >>>>>>>> >>>>>>>> Sorry TAO simply doesn't even collect this information >>>>>>>> currently. But yes we should definitely make it available! >>>>>>>> >>>>>>>> Jason, >>>>>>>> >>>>>>>> Could you please add this; almost all the TaoSolve_xxx() have >>>>>>>> a local variable iter; change that to tao->niter (I'm guess this is suppose >>>>>>>> to capture this information) and add a TaoGetIterationNumber() and the uses >>>>>>>> can access this. Also modify at the end of TaoSolve() -tao_converged_reason >>>>>>>> to also print the iteration count. At the same time since you add this you >>>>>>>> can add a tao->totalits which would accumulate all iterations over all the >>>>>>>> solves for that Tao object and the routine TaoGetTotalIterations() to >>>>>>>> access this. Note that TaoSolve() would initialize tao->niter = 0 at the >>>>>>>> top. >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> > On Apr 10, 2015, at 4:16 PM, Justin Chang >>>>>>>> wrote: >>>>>>>> > >>>>>>>> > Hi all, >>>>>>>> > >>>>>>>> > Is there a way to generically obtain the number of Tao >>>>>>>> iterations? I am looking through the -help options for Tao and I don't see >>>>>>>> any metric where you can output this quantity in the manner that you could >>>>>>>> for SNES or KSP solves. I am currently using blmvm and tron, and the only >>>>>>>> way I can see getting this metric is by outputting -tao_view and/or >>>>>>>> -tao_monitor and manually finding this number. I find this cumbersome >>>>>>>> especially for transient problems where I would like to simply have this >>>>>>>> number printed for each step instead of ending up with unnecessary info. >>>>>>>> > >>>>>>>> > Thanks, >>>>>>>> > >>>>>>>> > >>>>>>>> > -- >>>>>>>> > Justin Chang >>>>>>>> > PhD Candidate, Civil Engineering - Computational Sciences >>>>>>>> > University of Houston, Department of Civil and Environmental >>>>>>>> Engineering >>>>>>>> > Houston, TX 77004 >>>>>>>> > (512) 963-3262 >>>>>>>> >>>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jason.sarich at gmail.com Tue Apr 21 10:40:55 2015 From: jason.sarich at gmail.com (Jason Sarich) Date: Tue, 21 Apr 2015 10:40:55 -0500 Subject: [petsc-users] Tao iterations In-Reply-To: References: <3112142f40d64968ae157f4a5e819353@LUCKMAN.anl.gov> Message-ID: Justin, 1) The big difference between TRON and BLMVM is that TRON requires hessian information, BLMVM only uses gradient information. Thus TRON will usually converge faster, but requires more information, memory, and a KSP solver. GPCG (gradient projected conjugate gradient) is another gradient-only option, but usually performs worse than BLMVM. 2) TaoGetLinearSolveIterations() will get the total number of KSP iterations per solve Jason On Tue, Apr 21, 2015 at 10:33 AM, Justin Chang wrote: > Jason, > > Tightening the tolerances did the trick. Thanks. Though I do have a couple > more related questions: > > 1) Is there a general guideline for choosing tron over blmvm or vice > versa? Also is there another tao type that is also suitable given only > bounded constraints? > > 2) Is it possible to obtain the total number of KSP and/or PG iterations > from tron? > > Thanks, > Justin > > On Tue, Apr 21, 2015 at 9:52 AM, Jason Sarich > wrote: > >> Hi Justin, >> >> blmvm believes that it is already sufficiently close to a minimum, so it >> doesn't do anything. You may need to tighten some of the tolerance to force >> an iteration. >> >> Jason >> >> >> On Tue, Apr 21, 2015 at 9:48 AM, Justin Chang >> wrote: >> >>> Time step 1: >>> >>> Tao Object: 1 MPI processes >>> type: blmvm >>> Gradient steps: 0 >>> TaoLineSearch Object: 1 MPI processes >>> type: more-thuente >>> Active Set subset type: subvec >>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>> Residual in Function/Gradient:=0.0663148 >>> Objective value=-55.5945 >>> total number of iterations=35, (max: 2000) >>> total number of function/gradient evaluations=37, (max: 4000) >>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>> >>> Time step 2: >>> >>> Tao Object: 1 MPI processes >>> type: blmvm >>> Gradient steps: 0 >>> TaoLineSearch Object: 1 MPI processes >>> type: more-thuente >>> Active Set subset type: subvec >>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>> Residual in Function/Gradient:=0.0682307 >>> Objective value=-66.9675 >>> total number of iterations=23, (max: 2000) >>> total number of function/gradient evaluations=25, (max: 4000) >>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>> >>> Time step 3: >>> >>> Tao Object: 1 MPI processes >>> type: blmvm >>> Gradient steps: 0 >>> TaoLineSearch Object: 1 MPI processes >>> type: more-thuente >>> Active Set subset type: subvec >>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>> Residual in Function/Gradient:=0.0680522 >>> Objective value=-71.8211 >>> total number of iterations=19, (max: 2000) >>> total number of function/gradient evaluations=22, (max: 4000) >>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>> >>> Time step 4: >>> >>> Tao Object: 1 MPI processes >>> type: blmvm >>> Gradient steps: 0 >>> TaoLineSearch Object: 1 MPI processes >>> type: more-thuente >>> Active Set subset type: subvec >>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>> Residual in Function/Gradient:=0.0551556 >>> Objective value=-75.1252 >>> total number of iterations=18, (max: 2000) >>> total number of function/gradient evaluations=20, (max: 4000) >>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>> >>> Time step 5: >>> >>> Tao Object: 1 MPI processes >>> type: blmvm >>> Gradient steps: 0 >>> TaoLineSearch Object: 1 MPI processes >>> type: more-thuente >>> Active Set subset type: subvec >>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>> Residual in Function/Gradient:=0.0675667 >>> Objective value=-77.4414 >>> total number of iterations=6, (max: 2000) >>> total number of function/gradient evaluations=8, (max: 4000) >>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>> >>> Time step 6: >>> >>> Tao Object: 1 MPI processes >>> type: blmvm >>> Gradient steps: 0 >>> TaoLineSearch Object: 1 MPI processes >>> type: more-thuente >>> Active Set subset type: subvec >>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>> Residual in Function/Gradient:=0.059143 >>> Objective value=-79.5007 >>> total number of iterations=3, (max: 2000) >>> total number of function/gradient evaluations=5, (max: 4000) >>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>> >>> Time step 7: >>> >>> Tao Object: 1 MPI processes >>> type: blmvm >>> Gradient steps: 0 >>> TaoLineSearch Object: 1 MPI processes >>> type: more-thuente >>> Active Set subset type: subvec >>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>> Residual in Function/Gradient:=0.0433683 >>> Objective value=-81.3546 >>> total number of iterations=5, (max: 2000) >>> total number of function/gradient evaluations=8, (max: 4000) >>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>> >>> Time step 8: >>> >>> Tao Object: 1 MPI processes >>> type: blmvm >>> Gradient steps: 0 >>> TaoLineSearch Object: 1 MPI processes >>> type: more-thuente >>> Active Set subset type: subvec >>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>> Residual in Function/Gradient:=0.0840676 >>> Objective value=-82.9382 >>> total number of iterations=0, (max: 2000) >>> total number of function/gradient evaluations=1, (max: 4000) >>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>> >>> Time step 9: >>> >>> Tao Object: 1 MPI processes >>> type: blmvm >>> Gradient steps: 0 >>> TaoLineSearch Object: 1 MPI processes >>> type: more-thuente >>> Active Set subset type: subvec >>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>> Residual in Function/Gradient:=0.0840676 >>> Objective value=-82.9382 >>> total number of iterations=0, (max: 2000) >>> total number of function/gradient evaluations=1, (max: 4000) >>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>> >>> Time step 10: >>> >>> Tao Object: 1 MPI processes >>> type: blmvm >>> Gradient steps: 0 >>> TaoLineSearch Object: 1 MPI processes >>> type: more-thuente >>> Active Set subset type: subvec >>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>> Residual in Function/Gradient:=0.0840676 >>> Objective value=-82.9382 >>> total number of iterations=0, (max: 2000) >>> total number of function/gradient evaluations=1, (max: 4000) >>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>> >>> >>> >>> >>> >>> On Tue, Apr 21, 2015 at 9:28 AM, Jason Sarich >>> wrote: >>> >>>> Hi Justin, >>>> >>>> what reason is blmvm giving for stopping the solve? (you can use >>>> -tao_view or -tao_converged_reason to get this) >>>> >>>> Jason >>>> >>>> On Mon, Apr 20, 2015 at 6:32 PM, Justin Chang >>>> wrote: >>>> >>>>> Jason, >>>>> >>>>> I am using TaoGetSolutionStatus(tao,&its, ...) and it gives me >>>>> exactly what I want. However, I seem to be having an issue with blmvm >>>>> >>>>> I wrote my own backward euler code for a transient linear diffusion >>>>> problem with lower bounds >= 0 and upper bounds <= 1. For the first several >>>>> time steps I am getting its > 0, and it decreases over time due to the >>>>> nature of the discrete maximum principles. However, at some point my its >>>>> become 0 and the solution does not "update", which seems to me that >>>>> TaoSolve is not doing anything after that. This doesn't happen if I were to >>>>> use tron (my KSP and PC are cg and jacobi respectively). >>>>> >>>>> Do you know why this behavior may occur? >>>>> >>>>> Thanks, >>>>> >>>>> On Tue, Apr 14, 2015 at 9:35 AM, Jason Sarich >>>>> wrote: >>>>> >>>>>> Hi Justin, >>>>>> >>>>>> I have pushed these changes to the "next" branch, your code snippet >>>>>> should work fine there. >>>>>> >>>>>> Note that there is also available (since version 3.5.0) the routine >>>>>> TaoGetSolutionStatus(tao,&its,NULL,NULL,NULL,NULL,NULL) which will provide >>>>>> the >>>>>> same information >>>>>> >>>>>> Jason >>>>>> >>>>>> On Fri, Apr 10, 2015 at 6:28 PM, Justin Chang >>>>>> wrote: >>>>>> >>>>>>> Whatever is convenient and/or follow the "PETSc" standards. >>>>>>> Something similar to SNESGetIterationNumber() or KSPGetIterationNumber() >>>>>>> would be nice. Ideally I want my code to look like this: >>>>>>> >>>>>>> ierr = TaoGetIterationNumber(tao,&its);CHKERRQ(ierr); >>>>>>> ierr = PetscPrintf(PETSC_COMM_WORLD, "Number of Tao iterations = >>>>>>> %D\n", its); >>>>>>> >>>>>>> Thanks :) >>>>>>> >>>>>>> On Fri, Apr 10, 2015 at 5:53 PM, Jason Sarich < >>>>>>> jason.sarich at gmail.com> wrote: >>>>>>> >>>>>>>> Hi Justin, I'll get this in. I assume that displaying the number of >>>>>>>> iterations with tao_converged_reason is what you are asking for in >>>>>>>> particular? Or did you have something else in mind? >>>>>>>> >>>>>>>> Jason >>>>>>>> On Apr 10, 2015 16:42, "Smith, Barry F." >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> Justin, >>>>>>>>> >>>>>>>>> Sorry TAO simply doesn't even collect this information >>>>>>>>> currently. But yes we should definitely make it available! >>>>>>>>> >>>>>>>>> Jason, >>>>>>>>> >>>>>>>>> Could you please add this; almost all the TaoSolve_xxx() have >>>>>>>>> a local variable iter; change that to tao->niter (I'm guess this is suppose >>>>>>>>> to capture this information) and add a TaoGetIterationNumber() and the uses >>>>>>>>> can access this. Also modify at the end of TaoSolve() -tao_converged_reason >>>>>>>>> to also print the iteration count. At the same time since you add this you >>>>>>>>> can add a tao->totalits which would accumulate all iterations over all the >>>>>>>>> solves for that Tao object and the routine TaoGetTotalIterations() to >>>>>>>>> access this. Note that TaoSolve() would initialize tao->niter = 0 at the >>>>>>>>> top. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> > On Apr 10, 2015, at 4:16 PM, Justin Chang >>>>>>>>> wrote: >>>>>>>>> > >>>>>>>>> > Hi all, >>>>>>>>> > >>>>>>>>> > Is there a way to generically obtain the number of Tao >>>>>>>>> iterations? I am looking through the -help options for Tao and I don't see >>>>>>>>> any metric where you can output this quantity in the manner that you could >>>>>>>>> for SNES or KSP solves. I am currently using blmvm and tron, and the only >>>>>>>>> way I can see getting this metric is by outputting -tao_view and/or >>>>>>>>> -tao_monitor and manually finding this number. I find this cumbersome >>>>>>>>> especially for transient problems where I would like to simply have this >>>>>>>>> number printed for each step instead of ending up with unnecessary info. >>>>>>>>> > >>>>>>>>> > Thanks, >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > -- >>>>>>>>> > Justin Chang >>>>>>>>> > PhD Candidate, Civil Engineering - Computational Sciences >>>>>>>>> > University of Houston, Department of Civil and Environmental >>>>>>>>> Engineering >>>>>>>>> > Houston, TX 77004 >>>>>>>>> > (512) 963-3262 >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sanjay.Kharche at manchester.ac.uk Tue Apr 21 10:49:09 2015 From: Sanjay.Kharche at manchester.ac.uk (Sanjay Kharche) Date: Tue, 21 Apr 2015 15:49:09 +0000 Subject: [petsc-users] preconditioning Message-ID: Hi I am working through the KSP examples, but I will ask for any comments in the mean time. I have a TS code for 2D/3D heat equation, starting from ts/ex15. How can I easily modify this code to include a parallel preconditioner. As of now, I just need to add a preconditioner which is easy to program. The coefficients in my systems matrix are constant, but RHS changes at each iteration. cheers Sanjay From knepley at gmail.com Tue Apr 21 10:51:21 2015 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 21 Apr 2015 10:51:21 -0500 Subject: [petsc-users] preconditioning In-Reply-To: References: Message-ID: On Tue, Apr 21, 2015 at 10:49 AM, Sanjay Kharche < Sanjay.Kharche at manchester.ac.uk> wrote: > > Hi > > I am working through the KSP examples, but I will ask for any comments in > the mean time. > > I have a TS code for 2D/3D heat equation, starting from ts/ex15. How can I > easily modify this code to include a parallel preconditioner. As of now, I > just need to add a preconditioner which is easy to program. The > coefficients in my systems matrix are constant, but RHS changes at each > iteration. > Use -pc_type gamg. You have the Laplacian + Identity, so this should work fine. Thanks, Matt > cheers > Sanjay > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jychang48 at gmail.com Tue Apr 21 10:54:34 2015 From: jychang48 at gmail.com (Justin Chang) Date: Tue, 21 Apr 2015 10:54:34 -0500 Subject: [petsc-users] PetscLogFlop for a sqrt() In-Reply-To: References: Message-ID: Matt, Thank you for the clarification. Couple more questions: 1) Is there a difference between calling: y = ; and y = 0; y+=; that is, for the latter case it would require PetscLogFlop(1) and the former would not? 2) Looking through the source code of some of these PETSc functions, it seems the calculation of an index location for an array is not counted with PetscLogFlop(). For instance when I see: for (d = 0; d < dim; d++) { for (f = 0; f < dim; f++) { J[d*dim+f] = 0.0; for (g = 0; g < dim; g++) { J[d*dim+f] += R[d*dim+g]*J0[g*dim+f]; } } } PetscLogFlops(18.0); where R is a diagonal matrix and dim =3, the flops to document d*dim+f within J[...] is not accounted for within the 18 count (I think). If I had done something like pos = d*dim+f; J[pos] = ... wouldn't that be an extra 2 flops per loop iteration (excluding the zero off-diagonal parts)? Thanks, Justin On Tue, Apr 21, 2015 at 5:33 AM, Matthew Knepley wrote: > On Tue, Apr 21, 2015 at 12:59 AM, Justin Chang wrote: > >> Hi everyone, >> >> I want to manually count the number of flops that take place within my >> user-defined functions. Say I have to calculate the following: >> >> y = sqrt(pow(x[0],2)+pow(x[1],2)+pow(x[2],2)); >> >> What should the flop counter be for each call? >> > > 1) I would use PetscSqr() instead of pow(*, 2) and PetscSqrtReal/Scalar() > > 2) Sqrt is obviously more expensive than a multiply, but there is no good > standard, so we just count it as 1 flop > > Thanks, > > Matt > > >> Thanks, >> >> >> -- >> Justin Chang >> PhD Candidate, Civil Engineering - Computational Sciences >> University of Houston, Department of Civil and Environmental Engineering >> Houston, TX 77004 >> (512) 963-3262 >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Apr 21 10:58:33 2015 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 21 Apr 2015 10:58:33 -0500 Subject: [petsc-users] PetscLogFlop for a sqrt() In-Reply-To: References: Message-ID: On Tue, Apr 21, 2015 at 10:54 AM, Justin Chang wrote: > Matt, > > Thank you for the clarification. Couple more questions: > > 1) Is there a difference between calling: > > y = ; > > and > y = 0; > y+=; > > that is, for the latter case it would require PetscLogFlop(1) and the > former would not? > Yes. > 2) Looking through the source code of some of these PETSc functions, it > seems the calculation of an index location for an array is not counted with > PetscLogFlop(). For instance when I see: > > for (d = 0; d < dim; d++) { > for (f = 0; f < dim; f++) { > J[d*dim+f] = 0.0; > for (g = 0; g < dim; g++) { > J[d*dim+f] += R[d*dim+g]*J0[g*dim+f]; > } > } > } > PetscLogFlops(18.0); > Flop is Floating Point Operation. The index calculation is an Integer Operation. I agree that we could probably start counting those as well since in some sorts of applications its important, but right now we don't. Thanks, Matt > where R is a diagonal matrix and dim =3, the flops to document d*dim+f > within J[...] is not accounted for within the 18 count (I think). > > If I had done something like > > pos = d*dim+f; > J[pos] = ... > > wouldn't that be an extra 2 flops per loop iteration (excluding the zero > off-diagonal parts)? > > Thanks, > Justin > > > > On Tue, Apr 21, 2015 at 5:33 AM, Matthew Knepley > wrote: > >> On Tue, Apr 21, 2015 at 12:59 AM, Justin Chang wrote: >> >>> Hi everyone, >>> >>> I want to manually count the number of flops that take place within my >>> user-defined functions. Say I have to calculate the following: >>> >>> y = sqrt(pow(x[0],2)+pow(x[1],2)+pow(x[2],2)); >>> >>> What should the flop counter be for each call? >>> >> >> 1) I would use PetscSqr() instead of pow(*, 2) and PetscSqrtReal/Scalar() >> >> 2) Sqrt is obviously more expensive than a multiply, but there is no good >> standard, so we just count it as 1 flop >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> >>> >>> -- >>> Justin Chang >>> PhD Candidate, Civil Engineering - Computational Sciences >>> University of Houston, Department of Civil and Environmental Engineering >>> Houston, TX 77004 >>> (512) 963-3262 >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Apr 21 11:16:05 2015 From: jed at jedbrown.org (Jed Brown) Date: Tue, 21 Apr 2015 10:16:05 -0600 Subject: [petsc-users] PetscLogFlop for a sqrt() In-Reply-To: References: Message-ID: <874mo9foe2.fsf@jedbrown.org> Matthew Knepley writes: > Flop is Floating Point Operation. The index calculation is an Integer > Operation. I agree that we could probably start counting > those as well since in some sorts of applications its important, but right > now we don't. Index calculations often satisfy recurrences that the compiler folds into pointer increments and the like. Also, some architectures, like PowerPC, have floating point instructions that include mutating index operations in the true spirit of RISC. ;-) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From hus003 at ucsd.edu Tue Apr 21 11:59:31 2015 From: hus003 at ucsd.edu (Sun, Hui) Date: Tue, 21 Apr 2015 16:59:31 +0000 Subject: [petsc-users] Could be a bug on hypre with petsc In-Reply-To: <830A1052-FE13-4489-8453-AEEDA3385571@mcs.anl.gov> References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE4E5@XMAIL-MBX-BH1.AD.UCSD.EDU>, <830A1052-FE13-4489-8453-AEEDA3385571@mcs.anl.gov> Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010FE790@XMAIL-MBX-BH1.AD.UCSD.EDU> Thank you Barry. I've just reported the problem to Hypre. Hui ________________________________________ From: Barry Smith [bsmith at mcs.anl.gov] Sent: Monday, April 20, 2015 3:23 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Could be a bug on hypre with petsc It could be a bug or it could just be sloppyness where they send uninitialized data that they never use at the other end. Either way you should submit it to the hypre team so they can decide if they want to fix it. Barry > On Apr 19, 2015, at 6:55 PM, Sun, Hui wrote: > > I'm basically solving Poisson, and using valgrind to check memory issues. I set pctype to PCHYPRE. It works fine if I use one core. But if I use two cores, it gives me the following error messages about the memory, and then it computes results fine. It seems that it has to do with hypre, for if I set pctype to something else such as PCGAMG or PCJACOBI, it does not output any error message at all, regardless of one core or two cores. By the way, I use DM objects for the matrix and ksp. > > > > ==59097== Syscall param writev(vector[...]) points to uninitialised byte(s) > > ==59097== at 0x2F19FF2: writev (in /usr/lib/system/libsystem_kernel.dylib) > > ==59097== by 0x2AF470D: MPIDU_Sock_writev (sock_immed.i:610) > > ==59097== by 0x2ADE3E5: MPIDI_CH3_iStartMsgv (ch3_istartmsgv.c:110) > > ==59097== by 0x2ACEEF8: MPIDI_CH3_EagerContigSend (ch3u_eager.c:182) > > ==59097== by 0x2AD556F: MPID_Send (mpid_send.c:143) > > ==59097== by 0x2920057: MPI_Send (send.c:127) > > ==59097== by 0x1A712B1: hypre_MPI_Send (mpistubs.c:988) > > ==59097== by 0x1A6E77B: hypre_DataExchangeList (exchange_data.c:392) > > ==59097== by 0x194D7C1: hypre_NewCommPkgCreate_core (new_commpkg.c:267) > > ==59097== by 0x195108B: hypre_MatvecCommPkgCreate (par_csr_communication.c:478) > > ==59097== by 0x192020B: hypre_BoomerAMGCreateS (par_strength.c:190) > > ==59097== by 0x189D6AD: hypre_BoomerAMGSetup (par_amg_setup.c:883) > > ==59097== by 0x187BD7A: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:52) > > ==59097== by 0x11EFCE6: PCSetUp_HYPRE (hypre.c:157) > > ==59097== by 0x12766D4: PCSetUp (precon.c:918) > > ==59097== by 0x1385670: KSPSetUp (itfunc.c:330) > > ==59097== by 0x138798B: KSPSolve (itfunc.c:542) > > ==59097== by 0x10003C854: fourPoissonSolve (FourPoisson3D.h:383) > > ==59097== by 0x1000DEA42: main (test.c:473) > > ==59097== Address 0x100a295d8 is 8 bytes inside a block of size 28 alloc'd > > ==59097== at 0xC258: malloc (vg_replace_malloc.c:300) > > ==59097== by 0x1A6F501: hypre_MAlloc (hypre_memory.c:62) > > ==59097== by 0x1A6DDBA: hypre_DataExchangeList (exchange_data.c:212) > > ==59097== by 0x194D7C1: hypre_NewCommPkgCreate_core (new_commpkg.c:267) > > ==59097== by 0x195108B: hypre_MatvecCommPkgCreate (par_csr_communication.c:478) > > ==59097== by 0x192020B: hypre_BoomerAMGCreateS (par_strength.c:190) > > ==59097== by 0x189D6AD: hypre_BoomerAMGSetup (par_amg_setup.c:883) > > ==59097== by 0x187BD7A: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:52) > > ==59097== by 0x11EFCE6: PCSetUp_HYPRE (hypre.c:157) > > ==59097== by 0x12766D4: PCSetUp (precon.c:918) > > ==59097== by 0x1385670: KSPSetUp (itfunc.c:330) > > ==59097== by 0x138798B: KSPSolve (itfunc.c:542) > > ==59097== by 0x10003C854: fourPoissonSolve (FourPoisson3D.h:383) > > ==59097== by 0x1000DEA42: main (test.c:473) > > > > Best, > > Hui > From jychang48 at gmail.com Tue Apr 21 12:23:37 2015 From: jychang48 at gmail.com (Justin Chang) Date: Tue, 21 Apr 2015 12:23:37 -0500 Subject: [petsc-users] PetscLogFlop for a sqrt() In-Reply-To: <874mo9foe2.fsf@jedbrown.org> References: <874mo9foe2.fsf@jedbrown.org> Message-ID: Last question I would like to report the efficiency of my code. That is, flops/s over the theoretical peak performance (on n-cores). Where the TPP is clock * FLOPS/cycle * n. My current machine is a Intel? Core? i7-4790 CPU @ 3.60GHz and I am assuming that the FLOPS/cycle is 4. One of my serial test runs has achieved a FLOPS/s of 2.01e+09, which translates to an efficiency of almost 14%. I know these are crude measurements but would these manual flop counts be appropriate for this kind of measurement? Or would hardware counts from PAPI? Thanks, Justin On Tue, Apr 21, 2015 at 11:16 AM, Jed Brown wrote: > Matthew Knepley writes: > > Flop is Floating Point Operation. The index calculation is an Integer > > Operation. I agree that we could probably start counting > > those as well since in some sorts of applications its important, but > right > > now we don't. > > Index calculations often satisfy recurrences that the compiler folds > into pointer increments and the like. Also, some architectures, like > PowerPC, have floating point instructions that include mutating index > operations in the true spirit of RISC. ;-) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Apr 21 12:29:15 2015 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 21 Apr 2015 12:29:15 -0500 Subject: [petsc-users] PetscLogFlop for a sqrt() In-Reply-To: References: <874mo9foe2.fsf@jedbrown.org> Message-ID: On Tue, Apr 21, 2015 at 12:23 PM, Justin Chang wrote: > Last question > > I would like to report the efficiency of my code. That is, flops/s over > the theoretical peak performance (on n-cores). Where the TPP is clock * > FLOPS/cycle * n. My current machine is a Intel? Core? i7-4790 CPU @ 3.60GHz > and I am assuming that the FLOPS/cycle is 4. > > One of my serial test runs has achieved a FLOPS/s of 2.01e+09, which > translates to an efficiency of almost 14%. I know these are crude > measurements but would these manual flop counts be appropriate for this > kind of measurement? Or would hardware counts from PAPI? > 1) For this, I think the manual counts are good enough for the estimate 2) You really should not compare to TPP, which ignores memory bandwidth constraints: http://www.mcs.anl.gov/petsc/documentation/faq.html#computers You should run the STREAMS benchmark on your machine as the link says, and then use a "roofline" model to estimate the peak performance based on bandwidth. Here is a talk on that model: http://crd.lbl.gov/assets/pubs_presos/parlab08-roofline-talk.pdf and here is a paper which does exactly this for sparse MatVec (Krylov methods) http://www.cs.odu.edu/~keyes/papers/pcfd99_gkks.pdf Basically, you have some smaller multiplier to the bandwidth (arithmetic intensity), which gives you the real performance upper bound, not TPP. Thanks, Matt > Thanks, > Justin > > On Tue, Apr 21, 2015 at 11:16 AM, Jed Brown wrote: > >> Matthew Knepley writes: >> > Flop is Floating Point Operation. The index calculation is an Integer >> > Operation. I agree that we could probably start counting >> > those as well since in some sorts of applications its important, but >> right >> > now we don't. >> >> Index calculations often satisfy recurrences that the compiler folds >> into pointer increments and the like. Also, some architectures, like >> PowerPC, have floating point instructions that include mutating index >> operations in the true spirit of RISC. ;-) >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Apr 21 12:55:11 2015 From: jed at jedbrown.org (Jed Brown) Date: Tue, 21 Apr 2015 11:55:11 -0600 Subject: [petsc-users] PetscLogFlop for a sqrt() In-Reply-To: References: <874mo9foe2.fsf@jedbrown.org> Message-ID: <87vbgpe58g.fsf@jedbrown.org> Justin Chang writes: > Last question > > I would like to report the efficiency of my code. That is, flops/s over the > theoretical peak performance (on n-cores). Where the TPP is clock * > FLOPS/cycle * n. My current machine is a Intel? Core? i7-4790 CPU @ 3.60GHz > and I am assuming that the FLOPS/cycle is 4. This calculation is becoming obsolete because the vector clock rate is slower than the scalar clock rate. It is probably better to define peak flops as the best measured performance for tuned DGEMM. > One of my serial test runs has achieved a FLOPS/s of 2.01e+09, which > translates to an efficiency of almost 14%. I know these are crude > measurements but would these manual flop counts be appropriate for this > kind of measurement? Or would hardware counts from PAPI? Hardware counters are notoriously inaccurate since they may count speculative flops instead of useful flops. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From Sanjay.Kharche at manchester.ac.uk Tue Apr 21 13:35:04 2015 From: Sanjay.Kharche at manchester.ac.uk (Sanjay Kharche) Date: Tue, 21 Apr 2015 18:35:04 +0000 Subject: [petsc-users] preconditioning In-Reply-To: References: , Message-ID: Hi I just tried Matt's suggestion and tried to guess other preconditioning options as well. I didnt have the time to think about their appropriateness. All preconditioners worked with the actual ts/ex15. gmag took 13 times longer to finish than the case when I ran without any preconditioner. The fastest was the jacobi, followed by sor and bjacobi options to -pc_type I could not guess the names of the other listed preconditioners associated with ts and snes. In my code however, I found that sor and gamg did not work. I got error messages, and the first few lines are pasted below. My test was on 1 proc and 4 procs, both giving similar errors with differnent line numbers. The petsc part of my code is virtually identical to the example. Any comments on what is missing or can be changed? cheers Sanjay [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Arguments are incompatible [1]PETSC ERROR: Zero diagonal on row 39 [1]PETSC ERROR: Petsc Release Version 3.5.2, unknown [1]PETSC ERROR: ./sk2d on a linux-gnu-c-debug named atriumpc.ex.ac.uk by sanjay Tue Apr 21 19:28:53 2015 [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-sundials --download-scalapack --with-c2html=0 [1]PETSC ERROR: #1 MatInvertDiagonal_SeqAIJ() line 1724 in /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/mat/impls/aij/seq/aij.c [1]PETSC ERROR: #2 MatSOR_SeqAIJ() line 1756 in /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/mat/impls/aij/seq/aij.c [1]PETSC ERROR: #3 MatSOR_MPIAIJ() line 1483 in /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/mat/impls/aij/mpi/mpiaij.c [1]PETSC ERROR: #4 MatSOR() line 3643 in /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/mat/interface/matrix.c [1]PETSC ERROR: #5 PCApply_SOR() line 35 in /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/ksp/pc/impls/sor/sor.c [1]PETSC ERROR: #6 PCApply() line 440 in /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/ksp/pc/interface/precon.c ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: 21 April 2015 16:51 To: Sanjay Kharche Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] preconditioning On Tue, Apr 21, 2015 at 10:49 AM, Sanjay Kharche > wrote: Hi I am working through the KSP examples, but I will ask for any comments in the mean time. I have a TS code for 2D/3D heat equation, starting from ts/ex15. How can I easily modify this code to include a parallel preconditioner. As of now, I just need to add a preconditioner which is easy to program. The coefficients in my systems matrix are constant, but RHS changes at each iteration. Use -pc_type gamg. You have the Laplacian + Identity, so this should work fine. Thanks, Matt cheers Sanjay -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Apr 21 13:40:11 2015 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 21 Apr 2015 13:40:11 -0500 Subject: [petsc-users] preconditioning In-Reply-To: References: Message-ID: On Tue, Apr 21, 2015 at 1:35 PM, Sanjay Kharche < Sanjay.Kharche at manchester.ac.uk> wrote: > > Hi > > I just tried Matt's suggestion and tried to guess other preconditioning > options as well. I didnt have the time to think about their appropriateness. > > All preconditioners worked with the actual ts/ex15. gmag took 13 times > longer to finish than the case when I ran without any preconditioner. The > fastest was the jacobi, followed by sor and bjacobi options to -pc_type I > could not guess the names of the other listed preconditioners associated > with ts and snes. > With a small timestep, Jacobi is going to beat the crap out of anything else. However, in this case you should just be using an explicit method, not solving the system. With a large timestep, GAMG will win, although it will be beaten by GMG. > > In my code however, I found that sor and gamg did not work. I got error > messages, and the first few lines are pasted below. My test was on 1 proc > and 4 procs, both giving similar errors with differnent line numbers. The > petsc part of my code is virtually identical to the example. Any comments > on what is missing or can be changed? > You have a bug. There are not zeros on the diagonal of the heat oeprator. Matt > cheers > Sanjay > > > > > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: Arguments are incompatible > [1]PETSC ERROR: Zero diagonal on row 39 > [1]PETSC ERROR: Petsc Release Version 3.5.2, unknown > [1]PETSC ERROR: ./sk2d on a linux-gnu-c-debug named atriumpc.ex.ac.uk by > sanjay Tue Apr 21 19:28:53 2015 > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-fblaslapack --download-mpich > --download-sundials --download-scalapack --with-c2html=0 > [1]PETSC ERROR: #1 MatInvertDiagonal_SeqAIJ() line 1724 in > /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/mat/impls/aij/seq/aij.c > [1]PETSC ERROR: #2 MatSOR_SeqAIJ() line 1756 in > /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/mat/impls/aij/seq/aij.c > [1]PETSC ERROR: #3 MatSOR_MPIAIJ() line 1483 in > /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/mat/impls/aij/mpi/mpiaij.c > [1]PETSC ERROR: #4 MatSOR() line 3643 in > /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/mat/interface/matrix.c > [1]PETSC ERROR: #5 PCApply_SOR() line 35 in > /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/ksp/pc/impls/sor/sor.c > [1]PETSC ERROR: #6 PCApply() line 440 in > /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/ksp/pc/interface/precon.c > > > > ------------------------------ > *From:* Matthew Knepley [knepley at gmail.com] > *Sent:* 21 April 2015 16:51 > *To:* Sanjay Kharche > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] preconditioning > > On Tue, Apr 21, 2015 at 10:49 AM, Sanjay Kharche < > Sanjay.Kharche at manchester.ac.uk> wrote: > >> >> Hi >> >> I am working through the KSP examples, but I will ask for any comments in >> the mean time. >> >> I have a TS code for 2D/3D heat equation, starting from ts/ex15. How can >> I easily modify this code to include a parallel preconditioner. As of now, >> I just need to add a preconditioner which is easy to program. The >> coefficients in my systems matrix are constant, but RHS changes at each >> iteration. >> > > Use -pc_type gamg. You have the Laplacian + Identity, so this should > work fine. > > Thanks, > > Matt > > >> cheers >> Sanjay >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Apr 21 13:41:19 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 21 Apr 2015 13:41:19 -0500 Subject: [petsc-users] preconditioning In-Reply-To: References: Message-ID: <1A9B240F-6A4F-4BEC-955A-5C71997B8B31@mcs.anl.gov> Hmm, zeros on the diagonal that usually indicates either a bug in your code or a "Stokes-like" problem; for Stokes-like problems we recommend using PCFIELDSPLIT, this requires your understanding of the "block structure" of your matrix. Barry > On Apr 21, 2015, at 1:35 PM, Sanjay Kharche wrote: > > > Hi > > I just tried Matt's suggestion and tried to guess other preconditioning options as well. I didnt have the time to think about their appropriateness. > > All preconditioners worked with the actual ts/ex15. gmag took 13 times longer to finish than the case when I ran without any preconditioner. The fastest was the jacobi, followed by sor and bjacobi options to -pc_type I could not guess the names of the other listed preconditioners associated with ts and snes. > > In my code however, I found that sor and gamg did not work. I got error messages, and the first few lines are pasted below. My test was on 1 proc and 4 procs, both giving similar errors with differnent line numbers. The petsc part of my code is virtually identical to the example. Any comments on what is missing or can be changed? > > cheers > Sanjay > > > > > [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [1]PETSC ERROR: Arguments are incompatible > [1]PETSC ERROR: Zero diagonal on row 39 > [1]PETSC ERROR: Petsc Release Version 3.5.2, unknown > [1]PETSC ERROR: ./sk2d on a linux-gnu-c-debug named atriumpc.ex.ac.uk by sanjay Tue Apr 21 19:28:53 2015 > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-sundials --download-scalapack --with-c2html=0 > [1]PETSC ERROR: #1 MatInvertDiagonal_SeqAIJ() line 1724 in /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/mat/impls/aij/seq/aij.c > [1]PETSC ERROR: #2 MatSOR_SeqAIJ() line 1756 in /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/mat/impls/aij/seq/aij.c > [1]PETSC ERROR: #3 MatSOR_MPIAIJ() line 1483 in /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/mat/impls/aij/mpi/mpiaij.c > [1]PETSC ERROR: #4 MatSOR() line 3643 in /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/mat/interface/matrix.c > [1]PETSC ERROR: #5 PCApply_SOR() line 35 in /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/ksp/pc/impls/sor/sor.c > [1]PETSC ERROR: #6 PCApply() line 440 in /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/ksp/pc/interface/precon.c > > > > From: Matthew Knepley [knepley at gmail.com] > Sent: 21 April 2015 16:51 > To: Sanjay Kharche > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] preconditioning > > On Tue, Apr 21, 2015 at 10:49 AM, Sanjay Kharche wrote: > > Hi > > I am working through the KSP examples, but I will ask for any comments in the mean time. > > I have a TS code for 2D/3D heat equation, starting from ts/ex15. How can I easily modify this code to include a parallel preconditioner. As of now, I just need to add a preconditioner which is easy to program. The coefficients in my systems matrix are constant, but RHS changes at each iteration. > > Use -pc_type gamg. You have the Laplacian + Identity, so this should work fine. > > Thanks, > > Matt > > cheers > Sanjay > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From Sanjay.Kharche at manchester.ac.uk Tue Apr 21 14:04:07 2015 From: Sanjay.Kharche at manchester.ac.uk (Sanjay Kharche) Date: Tue, 21 Apr 2015 19:04:07 +0000 Subject: [petsc-users] preconditioning In-Reply-To: <1A9B240F-6A4F-4BEC-955A-5C71997B8B31@mcs.anl.gov> References: , <1A9B240F-6A4F-4BEC-955A-5C71997B8B31@mcs.anl.gov> Message-ID: I have no first derivatives. In any case, I will now revisit the systems matrix and RHS functions. ________________________________________ From: Barry Smith [bsmith at mcs.anl.gov] Sent: 21 April 2015 19:41 To: Sanjay Kharche Cc: Matthew Knepley; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] preconditioning Hmm, zeros on the diagonal that usually indicates either a bug in your code or a "Stokes-like" problem; for Stokes-like problems we recommend using PCFIELDSPLIT, this requires your understanding of the "block structure" of your matrix. Barry > On Apr 21, 2015, at 1:35 PM, Sanjay Kharche wrote: > > > Hi > > I just tried Matt's suggestion and tried to guess other preconditioning options as well. I didnt have the time to think about their appropriateness. > > All preconditioners worked with the actual ts/ex15. gmag took 13 times longer to finish than the case when I ran without any preconditioner. The fastest was the jacobi, followed by sor and bjacobi options to -pc_type I could not guess the names of the other listed preconditioners associated with ts and snes. > > In my code however, I found that sor and gamg did not work. I got error messages, and the first few lines are pasted below. My test was on 1 proc and 4 procs, both giving similar errors with differnent line numbers. The petsc part of my code is virtually identical to the example. Any comments on what is missing or can be changed? > > cheers > Sanjay > > > > > [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [1]PETSC ERROR: Arguments are incompatible > [1]PETSC ERROR: Zero diagonal on row 39 > [1]PETSC ERROR: Petsc Release Version 3.5.2, unknown > [1]PETSC ERROR: ./sk2d on a linux-gnu-c-debug named atriumpc.ex.ac.uk by sanjay Tue Apr 21 19:28:53 2015 > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-sundials --download-scalapack --with-c2html=0 > [1]PETSC ERROR: #1 MatInvertDiagonal_SeqAIJ() line 1724 in /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/mat/impls/aij/seq/aij.c > [1]PETSC ERROR: #2 MatSOR_SeqAIJ() line 1756 in /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/mat/impls/aij/seq/aij.c > [1]PETSC ERROR: #3 MatSOR_MPIAIJ() line 1483 in /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/mat/impls/aij/mpi/mpiaij.c > [1]PETSC ERROR: #4 MatSOR() line 3643 in /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/mat/interface/matrix.c > [1]PETSC ERROR: #5 PCApply_SOR() line 35 in /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/ksp/pc/impls/sor/sor.c > [1]PETSC ERROR: #6 PCApply() line 440 in /media/6c6201d2-4e2e-4b24-8d04-263b5bc06dfa/sanjay/petsc/src/ksp/pc/interface/precon.c > > > > From: Matthew Knepley [knepley at gmail.com] > Sent: 21 April 2015 16:51 > To: Sanjay Kharche > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] preconditioning > > On Tue, Apr 21, 2015 at 10:49 AM, Sanjay Kharche wrote: > > Hi > > I am working through the KSP examples, but I will ask for any comments in the mean time. > > I have a TS code for 2D/3D heat equation, starting from ts/ex15. How can I easily modify this code to include a parallel preconditioner. As of now, I just need to add a preconditioner which is easy to program. The coefficients in my systems matrix are constant, but RHS changes at each iteration. > > Use -pc_type gamg. You have the Laplacian + Identity, so this should work fine. > > Thanks, > > Matt > > cheers > Sanjay > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From bsmith at mcs.anl.gov Tue Apr 21 15:34:26 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 21 Apr 2015 15:34:26 -0500 Subject: [petsc-users] How to solve a matrix with zero diagonal entries? In-Reply-To: References: Message-ID: <8BF949BF-5BBD-4ED7-BEA9-E64B4863EDBA@mcs.anl.gov> Doing the zero pivot thing is a hack and is generally a very bad idea. Looks like you should be using a PCFIELDSPLIT preconditioner likely with a -pc_fieldsplit_type schur you can start with the option -pc_fieldsplit_detect_saddle_point but in the longer run you likely should provide programmatically in the code the IS that defines the "zero block" of the matrix. Barry > On Apr 20, 2015, at 10:03 AM, Song Gao wrote: > > Dear Petsc-users, > > I modified the code based on KSP ex1f. I'm solving a 2D stream function vorticity finite element problem. The problem is relatively small, around 2500x2500. I choose to use single processor and LU preconditioner. But the algorithm set some zeros on the diagonals. By checking the PCLU manual page, I think I need shift, the code looks like this: > > call KSPGetPC(ksp,preconditioner,ierr) > call PCSetType(preconditioner,PCLU,ierr) > call PCFactorSetShiftAmount(preconditioner, 100.0D0,ierr) > call KSPSetFromOptions(ksp,ierr) > call KSPSolve(ksp, pet_rhs, pet_sol_update,ierr) > > But I still get errors here. > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot > [0]PETSC ERROR: Zero pivot row 4 value 0 tolerance 2.22045e-14 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 > [0]PETSC ERROR: ./ex1f on a FEM_project named anakin by sgao Mon Apr 20 10:23:58 2015 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich > [0]PETSC ERROR: #1 MatPivotCheck_none() line 622 in /home/cfd/sgao/public/for_FEM/petsc-3.5.3/include/petsc-private/matimpl.h > [0]PETSC ERROR: #2 MatPivotCheck() line 641 in /home/cfd/sgao/public/for_FEM/petsc-3.5.3/include/petsc-private/matimpl.h > [0]PETSC ERROR: #3 MatLUFactorNumeric_SeqAIJ() line 575 in /home/cfd/sgao/public/for_FEM/petsc-3.5.3/src/mat/impls/aij/seq/aijfact.c > [0]PETSC ERROR: #4 MatLUFactorNumeric() line 2893 in /home/cfd/sgao/public/for_FEM/petsc-3.5.3/src/mat/interface/matrix.c > [0]PETSC ERROR: #5 PCSetUp_LU() line 152 in /home/cfd/sgao/public/for_FEM/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c > [0]PETSC ERROR: #6 PCSetUp() line 902 in /home/cfd/sgao/public/for_FEM/petsc-3.5.3/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #7 KSPSetUp() line 306 in /home/cfd/sgao/public/for_FEM/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #8 KSPSolve() line 418 in /home/cfd/sgao/public/for_FEM/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c > > > I tried other run time options, but none of them works. Some gives Zero pivot errors, some doesn't converge. > > ./ex1f -pc_factor_shift_type NONZERO -ksp_monitor_true_residual -ksp_max_it 10 > > ./ex1f -pc_factor_shift_type NONZERO -pc_factor_shift_amount 1 -ksp_monitor_true_residual -ksp_max_it 10 > > ./ex1f -pc_factor_shift_type POSITIVE_DEFINITE -pc_factor_shift_amount 1 -ksp_monitor_true_residual -ksp_max_it 10 > > ./ex1f -pc_factor_nonzeros_along_diagonal -ksp_monitor_true_residual -ksp_max_it 10 > > The matrix on a small grid is attached. Thanks > From david.knezevic at akselos.com Wed Apr 22 20:34:25 2015 From: david.knezevic at akselos.com (David Knezevic) Date: Wed, 22 Apr 2015 21:34:25 -0400 Subject: [petsc-users] TAO IPM Message-ID: I'm interested in trying out TAO IPM for constrained optimization with inequality, equality and bound constraints. I implemented a test case based on the "toy example" and it works well. But I noticed the following comment in the documentation : "This algorithm is more of a place-holder for future constrained optimization algorithms and should not yet be used for large problems or production code." So I was wondering what the status of this is? Is there anything that specifically doesn't work, or is it just not production-ready at this stage? (I didn't see any documentation about IPM in the TAO manual, so I figured I'd ask here). If IPM isn't recommended at this stage, is there anything else in TAO that works for generally constrained problems? Thanks, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From jchang27 at uh.edu Wed Apr 22 21:57:59 2015 From: jchang27 at uh.edu (Justin Chang) Date: Wed, 22 Apr 2015 21:57:59 -0500 Subject: [petsc-users] DMPlexProjectField not including BC constraints? Message-ID: Hello, I am using DMPlexProjectField(...) to postprocess my final solution u into another vector sol. I have the following pointwise functions: /* x-component of gradient */ void velx(const PetscScalar u[], const PetscScalar u_t[], const PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const PetscScalar a_x[], const PetscReal x[], PetscScalar v[]) { v[0] = -a[0]*mu*u_x[0]; } /* y-component of gradient */ void vely(const PetscScalar u[], const PetscScalar u_t[], const PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const PetscScalar a_x[], const PetscReal x[], PetscScalar v[]) { v[0] = -a[0]*mu*u_x[1]; } /* z-component of gradient */ void velz(const PetscScalar u[], const PetscScalar u_t[], const PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const PetscScalar a_x[], const PetscReal x[], PetscScalar v[]) { v[0] = -a[0]*mu*u_x[2]; } Where a[0] is the cell-wise permeability and mu is the (inverse of) viscosity. I am solving the diffusion equation (P1 elements). At the end of my simulation I have these lines: ierr = VecDuplicate(user.u,&user.sol);CHKERRQ(ierr); ierr = PetscObjectSetName((PetscObject) user.sol, "velocity");CHKERRQ(ierr); void (*vX[1])(const PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscReal *, PetscScalar *) = {velx}; ierr = DMPlexProjectField(dm,user.u,vX,INSERT_ALL_VALUES,user.sol);CHKERRQ(ierr); ierr = VecViewFromOptions(user.sol,NULL,"-vec_view_x");CHKERRQ(ierr); void (*vY[1])(const PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscReal *, PetscScalar *) = {vely}; ierr = DMPlexProjectField(dm,user.u,vY,INSERT_ALL_VALUES,user.sol);CHKERRQ(ierr); ierr = VecViewFromOptions(user.sol,NULL,"-vec_view_y");CHKERRQ(ierr); if (spatialDim == 3) { void (*vZ[1])(const PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscReal *, PetscScalar *) = {velz}; ierr = DMPlexProjectField(dm,user.u,vZ,INSERT_ALL_VALUES,user.sol);CHKERRQ(ierr); ierr = VecViewFromOptions(user.sol,NULL,"-vec_view_z");CHKERRQ(ierr); } ierr = VecDestroy(&user.sol);CHKERRQ(ierr); But when i look at the results, the dirichlet constrained values of u were projected into the respective locations in sol. I thought putting INSERT_ALL_VALUES would insert the pointwise functions into all locations including the constrained ones. How can I project the above pointwise functions into the constrained fields? Or what am I potentially missing here? Thanks, -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Apr 23 04:00:01 2015 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 23 Apr 2015 04:00:01 -0500 Subject: [petsc-users] DMPlexProjectField not including BC constraints? In-Reply-To: References: Message-ID: On Wed, Apr 22, 2015 at 9:57 PM, Justin Chang wrote: > Hello, > > I am using DMPlexProjectField(...) to postprocess my final solution u into > another vector sol. I have the following pointwise functions: > > /* x-component of gradient */ > void velx(const PetscScalar u[], const PetscScalar u_t[], const > PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const > PetscScalar a_x[], const PetscReal x[], PetscScalar v[]) > { > v[0] = -a[0]*mu*u_x[0]; > } > > /* y-component of gradient */ > void vely(const PetscScalar u[], const PetscScalar u_t[], const > PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const > PetscScalar a_x[], const PetscReal x[], PetscScalar v[]) > { > v[0] = -a[0]*mu*u_x[1]; > } > > /* z-component of gradient */ > void velz(const PetscScalar u[], const PetscScalar u_t[], const > PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const > PetscScalar a_x[], const PetscReal x[], PetscScalar v[]) > { > v[0] = -a[0]*mu*u_x[2]; > } > > Where a[0] is the cell-wise permeability and mu is the (inverse of) > viscosity. I am solving the diffusion equation (P1 elements). At the end of > my simulation I have these lines: > > ierr = VecDuplicate(user.u,&user.sol);CHKERRQ(ierr); > ierr = PetscObjectSetName((PetscObject) user.sol, > "velocity");CHKERRQ(ierr); > void (*vX[1])(const PetscScalar *, const PetscScalar *, const PetscScalar > *, const PetscScalar *, const PetscScalar *, const PetscScalar *, const > PetscReal *, PetscScalar *) = {velx}; > ierr = > DMPlexProjectField(dm,user.u,vX,INSERT_ALL_VALUES,user.sol);CHKERRQ(ierr); > ierr = VecViewFromOptions(user.sol,NULL,"-vec_view_x");CHKERRQ(ierr); > > void (*vY[1])(const PetscScalar *, const PetscScalar *, const PetscScalar > *, const PetscScalar *, const PetscScalar *, const PetscScalar *, const > PetscReal *, PetscScalar *) = {vely}; > ierr = > DMPlexProjectField(dm,user.u,vY,INSERT_ALL_VALUES,user.sol);CHKERRQ(ierr); > ierr = VecViewFromOptions(user.sol,NULL,"-vec_view_y");CHKERRQ(ierr); > > if (spatialDim == 3) { > void (*vZ[1])(const PetscScalar *, const PetscScalar *, const > PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscScalar > *, const PetscReal *, PetscScalar *) = {velz}; > ierr = > DMPlexProjectField(dm,user.u,vZ,INSERT_ALL_VALUES,user.sol);CHKERRQ(ierr); > ierr = VecViewFromOptions(user.sol,NULL,"-vec_view_z");CHKERRQ(ierr); > } > ierr = VecDestroy(&user.sol);CHKERRQ(ierr); > > But when i look at the results, the dirichlet constrained values of u were > projected into the respective locations in sol. I thought putting > INSERT_ALL_VALUES would insert the pointwise functions into all locations > including the constrained ones. How can I project the above pointwise > functions into the constrained fields? Or what am I potentially missing > here? > There are no Dirichlet constrained values in the global vector user.sol. When you call VecView() it gets projected to an enlarged vector including the BC and output. What you want is to project that vector into a space with no constraints, so you really need another DM/Section combination. I think this will work: 1) Get a local vector from dm 2) Use DMPlexProjectFieldLocal(INSERT_ALL_VALUES) to fill it up 3) DMClone(dm) and give it a Section with no constraints 4) Use DMLocalToGlobal() into the new DM 5) VecView() that new global vector Thanks, Matt > Thanks, > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From siddhesh4godbole at gmail.com Thu Apr 23 05:47:35 2015 From: siddhesh4godbole at gmail.com (siddhesh godbole) Date: Thu, 23 Apr 2015 16:17:35 +0530 Subject: [petsc-users] MPI speedup Message-ID: Hello, I want to know about the test which is conducted just after the PETSC is configured on the system to assess the possible speedup by MPI processes. I have saved the result file which says: *Number of MPI processes 10* *Process 0 iitm* *Process 1 iitm* *Process 2 iitm* *Process 3 iitm* *Process 4 iitm* *Process 5 iitm* *Process 6 iitm* *Process 7 iitm* *Process 8 iitm* *Process 9 iitm* *Function Rate (MB/s) * *Copy: 24186.8271* *Scale: 23914.0401* *Add: 27271.7149* *Triad: 27787.1630* *------------------------------------------------* *np speedup* *1 1.0* *2 1.75* *3 1.86* *4 1.84* *5 1.85* *6 1.83* *7 1.76* *8 1.79* *9 1.8* *10 1.8* *Estimation of possible speedup of MPI programs based on Streams benchmark.* 1) What parameters the speedup depends on? 2) what are the hardware requirements for higher speedup? ( i was expecting atleast 5 times speedup after generating 10 processes. 3) what could possibly be done to improve this ? i have intel? Core? i7-4930K CPU @ 3.40GHz ? 12 with 32 GB of RAM and 1 TB disk space. Thanks *Siddhesh M Godbole* 5th year Dual Degree, Civil Eng & Applied Mech. IIT Madras -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Apr 23 05:52:05 2015 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 23 Apr 2015 05:52:05 -0500 Subject: [petsc-users] MPI speedup In-Reply-To: References: Message-ID: On Thu, Apr 23, 2015 at 5:47 AM, siddhesh godbole < siddhesh4godbole at gmail.com> wrote: > Hello, > > I want to know about the test which is conducted just after the PETSC is > configured on the system to assess the possible speedup by MPI processes. I > have saved the result file which says: > *Number of MPI processes 10* > *Process 0 iitm* > *Process 1 iitm* > *Process 2 iitm* > *Process 3 iitm* > *Process 4 iitm* > *Process 5 iitm* > *Process 6 iitm* > *Process 7 iitm* > *Process 8 iitm* > *Process 9 iitm* > *Function Rate (MB/s) * > *Copy: 24186.8271* > *Scale: 23914.0401* > *Add: 27271.7149* > *Triad: 27787.1630* > *------------------------------------------------* > *np speedup* > *1 1.0* > *2 1.75* > *3 1.86* > *4 1.84* > *5 1.85* > *6 1.83* > *7 1.76* > *8 1.79* > *9 1.8* > *10 1.8* > *Estimation of possible speedup of MPI programs based on Streams > benchmark.* > > 1) What parameters the speedup depends on? > I am not sure what you are asking here. Speedup is defined as the time T on 1 process divided by the time T_p on p processes: S = T/T_p > 2) what are the hardware requirements for higher speedup? ( i was > expecting atleast 5 times speedup after generating 10 processes. > STREAMS measures the speedup of vectors operations, which are very similar to sparse matrix operations. Both are limited by memory bandwidth. > 3) what could possibly be done to improve this ? > 1) You could buy more nodes, since each node has a path to memory 2) You could change algorithms, but this has proven very difficult Thanks, Matt > i have intel? Core? i7-4930K CPU @ 3.40GHz ? 12 with 32 GB of RAM and 1 TB > disk space. > > > Thanks > *Siddhesh M Godbole* > > 5th year Dual Degree, > Civil Eng & Applied Mech. > IIT Madras > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From siddhesh4godbole at gmail.com Thu Apr 23 06:02:09 2015 From: siddhesh4godbole at gmail.com (siddhesh godbole) Date: Thu, 23 Apr 2015 16:32:09 +0530 Subject: [petsc-users] MPI speedup In-Reply-To: References: Message-ID: Matt So that means the time on 10 processes in merely 1.8 times the time on 1 process?? this is quite difficult to digest! Okay so if memory bandwidth a controlling factor here how will forming a cluster with same machines solve this problem? my cpu has max memory bandwidth of 59 GB/s . Apologies if the question are too silly! *Siddhesh M Godbole* 5th year Dual Degree, Civil Eng & Applied Mech. IIT Madras On Thu, Apr 23, 2015 at 4:22 PM, Matthew Knepley wrote: > On Thu, Apr 23, 2015 at 5:47 AM, siddhesh godbole < > siddhesh4godbole at gmail.com> wrote: > >> Hello, >> >> I want to know about the test which is conducted just after the PETSC is >> configured on the system to assess the possible speedup by MPI processes. I >> have saved the result file which says: >> *Number of MPI processes 10* >> *Process 0 iitm* >> *Process 1 iitm* >> *Process 2 iitm* >> *Process 3 iitm* >> *Process 4 iitm* >> *Process 5 iitm* >> *Process 6 iitm* >> *Process 7 iitm* >> *Process 8 iitm* >> *Process 9 iitm* >> *Function Rate (MB/s) * >> *Copy: 24186.8271* >> *Scale: 23914.0401* >> *Add: 27271.7149* >> *Triad: 27787.1630* >> *------------------------------------------------* >> *np speedup* >> *1 1.0* >> *2 1.75* >> *3 1.86* >> *4 1.84* >> *5 1.85* >> *6 1.83* >> *7 1.76* >> *8 1.79* >> *9 1.8* >> *10 1.8* >> *Estimation of possible speedup of MPI programs based on Streams >> benchmark.* >> >> 1) What parameters the speedup depends on? >> > > I am not sure what you are asking here. Speedup is defined as the time T > on 1 process divided > by the time T_p on p processes: > > S = T/T_p > > >> 2) what are the hardware requirements for higher speedup? ( i was >> expecting atleast 5 times speedup after generating 10 processes. >> > > STREAMS measures the speedup of vectors operations, which are very similar > to sparse matrix operations. Both > are limited by memory bandwidth. > > >> 3) what could possibly be done to improve this ? >> > > 1) You could buy more nodes, since each node has a path to memory > > 2) You could change algorithms, but this has proven very difficult > > Thanks, > > Matt > > >> i have intel? Core? i7-4930K CPU @ 3.40GHz ? 12 with 32 GB of RAM and 1 >> TB disk space. >> >> >> Thanks >> *Siddhesh M Godbole* >> >> 5th year Dual Degree, >> Civil Eng & Applied Mech. >> IIT Madras >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Apr 23 06:10:23 2015 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 23 Apr 2015 06:10:23 -0500 Subject: [petsc-users] MPI speedup In-Reply-To: References: Message-ID: On Thu, Apr 23, 2015 at 6:02 AM, siddhesh godbole < siddhesh4godbole at gmail.com> wrote: > Matt > > So that means the time on 10 processes in merely 1.8 times the time on 1 > process?? this is quite difficult to digest! Okay so if memory bandwidth > a controlling factor here how will forming a cluster with same machines > solve this problem? > my cpu has max memory bandwidth of 59 GB/s . > It is tempting to say that computer manufacturers are lying when they report performance. You can bring down 59 GB/s of memory, so that is less than 8B doubles/s. If you are adding two vectors, that means you do 1 flop for every 2 doubles you bring down, so you can do 4 GF/s. Your processor has 6 cores, each running at 3.4 GHz, and it can do 4flops/cycle using the vector instructions, so they report that it can do 81 GF/s, but you cannot get values to it fast enough to compute. Now, you are getting only 50% of peak bandwidth, which is bad. Maybe Jed knows why you are not getting 75-80% which is what we expect. Thanks, Matt > Apologies if the question are too silly! > > *Siddhesh M Godbole* > > 5th year Dual Degree, > Civil Eng & Applied Mech. > IIT Madras > > On Thu, Apr 23, 2015 at 4:22 PM, Matthew Knepley > wrote: > >> On Thu, Apr 23, 2015 at 5:47 AM, siddhesh godbole < >> siddhesh4godbole at gmail.com> wrote: >> >>> Hello, >>> >>> I want to know about the test which is conducted just after the PETSC is >>> configured on the system to assess the possible speedup by MPI processes. I >>> have saved the result file which says: >>> *Number of MPI processes 10* >>> *Process 0 iitm* >>> *Process 1 iitm* >>> *Process 2 iitm* >>> *Process 3 iitm* >>> *Process 4 iitm* >>> *Process 5 iitm* >>> *Process 6 iitm* >>> *Process 7 iitm* >>> *Process 8 iitm* >>> *Process 9 iitm* >>> *Function Rate (MB/s) * >>> *Copy: 24186.8271* >>> *Scale: 23914.0401* >>> *Add: 27271.7149* >>> *Triad: 27787.1630* >>> *------------------------------------------------* >>> *np speedup* >>> *1 1.0* >>> *2 1.75* >>> *3 1.86* >>> *4 1.84* >>> *5 1.85* >>> *6 1.83* >>> *7 1.76* >>> *8 1.79* >>> *9 1.8* >>> *10 1.8* >>> *Estimation of possible speedup of MPI programs based on Streams >>> benchmark.* >>> >>> 1) What parameters the speedup depends on? >>> >> >> I am not sure what you are asking here. Speedup is defined as the time T >> on 1 process divided >> by the time T_p on p processes: >> >> S = T/T_p >> >> >>> 2) what are the hardware requirements for higher speedup? ( i was >>> expecting atleast 5 times speedup after generating 10 processes. >>> >> >> STREAMS measures the speedup of vectors operations, which are very >> similar to sparse matrix operations. Both >> are limited by memory bandwidth. >> >> >>> 3) what could possibly be done to improve this ? >>> >> >> 1) You could buy more nodes, since each node has a path to memory >> >> 2) You could change algorithms, but this has proven very difficult >> >> Thanks, >> >> Matt >> >> >>> i have intel? Core? i7-4930K CPU @ 3.40GHz ? 12 with 32 GB of RAM and 1 >>> TB disk space. >>> >>> >>> Thanks >>> *Siddhesh M Godbole* >>> >>> 5th year Dual Degree, >>> Civil Eng & Applied Mech. >>> IIT Madras >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From popov at uni-mainz.de Thu Apr 23 06:28:48 2015 From: popov at uni-mainz.de (anton) Date: Thu, 23 Apr 2015 13:28:48 +0200 Subject: [petsc-users] DMDAVecGetArray Message-ID: <5538D770.2030807@uni-mainz.de> Hi, I'm currently calling DMDAVecGet(Restore)Array on a subset of processors. It seems to work correctly. So why does documentation say they are collective on Vec? Are they really? Unlike VecNorm, for example, it's not really clear why DMDAVecGetArray should be collective. Am I still on a safe side, or I'm seriously violating something that's not immediately obvious. Thanks, Anton From knepley at gmail.com Thu Apr 23 06:36:19 2015 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 23 Apr 2015 06:36:19 -0500 Subject: [petsc-users] DMDAVecGetArray In-Reply-To: <5538D770.2030807@uni-mainz.de> References: <5538D770.2030807@uni-mainz.de> Message-ID: On Thu, Apr 23, 2015 at 6:28 AM, anton wrote: > Hi, > > I'm currently calling DMDAVecGet(Restore)Array on a subset of processors. > It seems to work correctly. So why does documentation say they are > collective on Vec? Are they really? Unlike VecNorm, for example, it's not > really clear why DMDAVecGetArray should be collective. > The documentation should be fixed. Its "logically collective" just like VecGetArray(), and unlike VecGetArrayRead(). > Am I still on a safe side, or I'm seriously violating something that's not > immediately obvious. > You are violating something that is not obvious. GetArray() will increment the status marker on the Vec since you could have changed something, but then you will get inconsistency across processes in this marker, so you could make different decisions about whether to recompute norms, etc. Hard to debug errors. Thanks, Matt > Thanks, > Anton > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From popov at uni-mainz.de Thu Apr 23 08:41:37 2015 From: popov at uni-mainz.de (anton) Date: Thu, 23 Apr 2015 15:41:37 +0200 Subject: [petsc-users] DMDAVecGetArray In-Reply-To: References: <5538D770.2030807@uni-mainz.de> Message-ID: <5538F691.9080401@uni-mainz.de> On 04/23/2015 01:36 PM, Matthew Knepley wrote: > On Thu, Apr 23, 2015 at 6:28 AM, anton > wrote: > > Hi, > > I'm currently calling DMDAVecGet(Restore)Array on a subset of > processors. It seems to work correctly. So why does documentation > say they are collective on Vec? Are they really? Unlike VecNorm, > for example, it's not really clear why DMDAVecGetArray should be > collective. > > > The documentation should be fixed. Its "logically collective" just > like VecGetArray(), and unlike VecGetArrayRead(). > > Am I still on a safe side, or I'm seriously violating something > that's not immediately obvious. > > > You are violating something that is not obvious. GetArray() will > increment the status marker on the Vec since you > could have changed something, but then you will get inconsistency > across processes in this marker, so you could > make different decisions about whether to recompute norms, etc. Hard > to debug errors. > OK, thanks Matt, I'll simply add dummy calls on those processes where I don't need the data. Anton > Thanks, > > Matt > > Thanks, > Anton > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Apr 23 09:27:01 2015 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 23 Apr 2015 09:27:01 -0500 Subject: [petsc-users] Uninterpolating a distributed mesh In-Reply-To: References: Message-ID: On Mon, Apr 20, 2015 at 6:09 AM, Justin Chang wrote: > Matt, > > I have attempted to implement this, called UninterpolateSF(), and attached > is the code for the function. However, I suspect that there are bugs > because when I refine the mesh via -dm_refine my program crashes. > I have finally fixed this. It is in next. Thanks, Matt > In my program I have the following lines for uninterpolating a distributed > mesh: > > DM idm=NULL; > ierr = DMPlexUninterpolate(*dm, &idm);CHKERRQ(ierr); > ierr = DMPlexCopyCoordinates(*dm, idm);CHKERRQ(ierr); > ierr = DMPlexCopyLabels(*dm, idm);CHKERRQ(ierr); > if (size>1) { > ierr = UninterpolateSF(*dm,idm);CHKERRQ(ierr); > } > ierr = DMDestroy(dm);CHKERRQ(ierr); > *dm = idm; > > Can you have a look into the source code I attached and see what may > potentially be the issue? > > Thanks, > > > On Mon, Apr 13, 2015 at 4:28 PM, Matthew Knepley > wrote: > >> On Mon, Apr 13, 2015 at 4:22 PM, Justin Chang wrote: >> >>> Is there an example somewhere that does something similar to this? >>> >> >> There is DMPlexShiftSF_Internal() in plexsubmesh.c >> >> Matt >> >> >>> Thanks, >>> >>> On Mon, Apr 13, 2015 at 12:31 PM, Matthew Knepley >>> wrote: >>> >>>> On Sat, Apr 11, 2015 at 6:55 PM, Justin Chang wrote: >>>> >>>>> Hello, >>>>> >>>>> When I call DMPlexUninterpolate(...) on a distributed mesh (say 2 >>>>> processors), it seems to overwrite the "ghost" points (i.e., the points not >>>>> locally owned by the processor) and treats all points as if they are local >>>>> to the processor. >>>>> >>>> >>>> Yes, I wrote Uninterpolate() just for testing, and do not currently >>>> handle the SF. I put it on my TODO list. >>>> >>>> Its not hard if you want to try. You just filter out any points that >>>> are not cells and vertices from the SF, so >>>> >>>> PetscSFGetGraph() >>>> for (leaves) >>>> if leaf not a cell or vertex, skip >>>> PetscSFSetGraph() >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Say I have this 2D cell-vertex mesh: >>>>> >>>>> 14----15-----16 >>>>> | \ 5 | \ 7 | >>>>> | \ | \ | >>>>> | 4 \ | 6 \ | >>>>> 11----12-----13 >>>>> | \ 1 | \ 3 | >>>>> | \ | \ | >>>>> | 0 \ | 2 \ | >>>>> 8------9------10 >>>>> >>>>> Which results in the following DM: >>>>> >>>>> DM Object: 2 MPI processes >>>>> type: plex >>>>> DM_0x84000004_0 in 2 dimensions: >>>>> 0-cells: 9 0 >>>>> 2-cells: 8 0 >>>>> Labels: >>>>> marker: 1 strata of sizes (8) >>>>> depth: 2 strata of sizes (9, 8) >>>>> >>>>> I proceed by interpolating this DM: >>>>> >>>>> DM Object: 2 MPI processes >>>>> type: plex >>>>> DM_0x84000004_1 in 2 dimensions: >>>>> 0-cells: 9 0 >>>>> 1-cells: 16 0 >>>>> 2-cells: 8 0 >>>>> Labels: >>>>> marker: 1 strata of sizes (16) >>>>> depth: 3 strata of sizes (9, 16, 8) >>>>> >>>>> Then distributing across 2 processors: >>>>> >>>>> DM Object:Parallel Mesh 2 MPI processes >>>>> type: plex >>>>> Parallel Mesh in 2 dimensions: >>>>> 0-cells: 6 6 >>>>> 1-cells: 9 9 >>>>> 2-cells: 4 4 >>>>> Labels: >>>>> marker: 1 strata of sizes (9) >>>>> depth: 3 strata of sizes (6, 9, 4) >>>>> >>>>> I have the option of uniformly refining the mesh here but I choose not >>>>> to for now. If my dofs are vertex based, then the global size of my DM >>>>> vector is 9 and the local sizes for ranks 0 and 1 are 3 and 6 respectively. >>>>> However, if I choose to uninterpolate the mesh by calling >>>>> DMPlexUninterpolate(...), I get this: >>>>> >>>>> DM Object: 2 MPI processes >>>>> type: plex >>>>> DM_0x84000004_2 in 2 dimensions: >>>>> 0-cells: 6 6 >>>>> 2-cells: 4 4 >>>>> Labels: >>>>> marker: 1 strata of sizes (5) >>>>> depth: 2 strata of sizes (6, 4) >>>>> >>>>> And the global size of my DM vector becomes 12 and the local size for >>>>> both ranks is 6. It looks like the ghost points in rank 0 have been >>>>> duplicated, which is not suppose to happen. >>>>> >>>>> Is there a way to capture the ghost point information when >>>>> uninterpolating the DM? >>>>> >>>>> Thanks, >>>>> >>>>> -- >>>>> Justin Chang >>>>> PhD Candidate, Civil Engineering - Computational Sciences >>>>> University of Houston, Department of Civil and Environmental >>>>> Engineering >>>>> Houston, TX 77004 >>>>> (512) 963-3262 >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >>> >>> -- >>> Justin Chang >>> PhD Candidate, Civil Engineering - Computational Sciences >>> University of Houston, Department of Civil and Environmental Engineering >>> Houston, TX 77004 >>> (512) 963-3262 >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Apr 23 12:38:43 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 23 Apr 2015 12:38:43 -0500 Subject: [petsc-users] DMDAVecGetArray In-Reply-To: <5538F691.9080401@uni-mainz.de> References: <5538D770.2030807@uni-mainz.de> <5538F691.9080401@uni-mainz.de> Message-ID: <76B38D4E-6245-43A6-BA4E-0B8E66E55587@mcs.anl.gov> > On Apr 23, 2015, at 8:41 AM, anton wrote: > > > On 04/23/2015 01:36 PM, Matthew Knepley wrote: >> On Thu, Apr 23, 2015 at 6:28 AM, anton wrote: >> Hi, >> >> I'm currently calling DMDAVecGet(Restore)Array on a subset of processors. It seems to work correctly. So why does documentation say they are collective on Vec? Are they really? Unlike VecNorm, for example, it's not really clear why DMDAVecGetArray should be collective. >> >> The documentation should be fixed. Its "logically collective" just like VecGetArray(), and unlike VecGetArrayRead(). >> >> Am I still on a safe side, or I'm seriously violating something that's not immediately obvious. >> >> You are violating something that is not obvious. GetArray() will increment the status marker on the Vec since you >> could have changed something, but then you will get inconsistency across processes in this marker, so you could >> make different decisions about whether to recompute norms, etc. Hard to debug errors. >> > > OK, thanks Matt, I'll simply add dummy calls on those processes where I don't need the data. If you are only "reading" information from the array you should switch to the DMDAVecGetArrayRead() versions available in the development version of PETSc, then you do not need to call on all processes. Barry > > Anton > >> Thanks, >> >> Matt >> >> Thanks, >> Anton >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > From bsmith at mcs.anl.gov Thu Apr 23 12:42:33 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 23 Apr 2015 12:42:33 -0500 Subject: [petsc-users] MPI speedup In-Reply-To: References: Message-ID: Please see http://www.mcs.anl.gov/petsc/documentation/faq.html#computers and note the information about "binding" options for MPICH and OpenMPI that can sometimes improve the streams performance (and hence other algorithms performance) a good amount. Barry > On Apr 23, 2015, at 6:02 AM, siddhesh godbole wrote: > > Matt > > So that means the time on 10 processes in merely 1.8 times the time on 1 process?? this is quite difficult to digest! Okay so if memory bandwidth a controlling factor here how will forming a cluster with same machines solve this problem? > my cpu has max memory bandwidth of 59 GB/s . > > > Apologies if the question are too silly! > > Siddhesh M Godbole > > 5th year Dual Degree, > Civil Eng & Applied Mech. > IIT Madras > > On Thu, Apr 23, 2015 at 4:22 PM, Matthew Knepley wrote: > On Thu, Apr 23, 2015 at 5:47 AM, siddhesh godbole wrote: > Hello, > > I want to know about the test which is conducted just after the PETSC is configured on the system to assess the possible speedup by MPI processes. I have saved the result file which says: > Number of MPI processes 10 > Process 0 iitm > Process 1 iitm > Process 2 iitm > Process 3 iitm > Process 4 iitm > Process 5 iitm > Process 6 iitm > Process 7 iitm > Process 8 iitm > Process 9 iitm > Function Rate (MB/s) > Copy: 24186.8271 > Scale: 23914.0401 > Add: 27271.7149 > Triad: 27787.1630 > ------------------------------------------------ > np speedup > 1 1.0 > 2 1.75 > 3 1.86 > 4 1.84 > 5 1.85 > 6 1.83 > 7 1.76 > 8 1.79 > 9 1.8 > 10 1.8 > Estimation of possible speedup of MPI programs based on Streams benchmark. > > 1) What parameters the speedup depends on? > > I am not sure what you are asking here. Speedup is defined as the time T on 1 process divided > by the time T_p on p processes: > > S = T/T_p > > 2) what are the hardware requirements for higher speedup? ( i was expecting atleast 5 times speedup after generating 10 processes. > > STREAMS measures the speedup of vectors operations, which are very similar to sparse matrix operations. Both > are limited by memory bandwidth. > > 3) what could possibly be done to improve this ? > > 1) You could buy more nodes, since each node has a path to memory > > 2) You could change algorithms, but this has proven very difficult > > Thanks, > > Matt > > i have intel? Core? i7-4930K CPU @ 3.40GHz ? 12 with 32 GB of RAM and 1 TB disk space. > > > Thanks > Siddhesh M Godbole > > 5th year Dual Degree, > Civil Eng & Applied Mech. > IIT Madras > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > From jason.sarich at gmail.com Thu Apr 23 13:57:17 2015 From: jason.sarich at gmail.com (Jason Sarich) Date: Thu, 23 Apr 2015 13:57:17 -0500 Subject: [petsc-users] TAO IPM In-Reply-To: <9ff483acd60147a7b2a437bc9a8bc1af@LUCKMAN.anl.gov> References: <9ff483acd60147a7b2a437bc9a8bc1af@LUCKMAN.anl.gov> Message-ID: Hi David, The IPM code is a naive implementation and not very battle-tested. There is nothing specifically broken, but the linear solve operator becomes very ill-conditioned and for most cases will require a direct solver (e.g. superlu). This is the major problem, and if you're ok with that then it may work for you. Unfortunately, we don't have anything else to recommend for generally constrained problems. Jason Sarich On Wed, Apr 22, 2015 at 8:34 PM, David Knezevic wrote: > I'm interested in trying out TAO IPM for constrained optimization with > inequality, equality and bound constraints. I implemented a test case based > on the "toy example" > > and it works well. > > But I noticed the following comment in the documentation > > : > "This algorithm is more of a place-holder for future constrained > optimization algorithms and should not yet be used for large problems or > production code." > > So I was wondering what the status of this is? Is there anything that > specifically doesn't work, or is it just not production-ready at this > stage? (I didn't see any documentation about IPM in the TAO manual, so I > figured I'd ask here). > > If IPM isn't recommended at this stage, is there anything else in TAO > that works for generally constrained problems? > > Thanks, > David > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.knezevic at akselos.com Thu Apr 23 14:32:58 2015 From: david.knezevic at akselos.com (David Knezevic) Date: Thu, 23 Apr 2015 15:32:58 -0400 Subject: [petsc-users] TAO IPM In-Reply-To: References: <9ff483acd60147a7b2a437bc9a8bc1af@LUCKMAN.anl.gov> Message-ID: On Thu, Apr 23, 2015 at 2:57 PM, Jason Sarich wrote: > Hi David, > > The IPM code is a naive implementation and not very battle-tested. There > is nothing specifically broken, but the linear solve operator becomes very > ill-conditioned and for most cases will require a direct solver (e.g. > superlu). This is the major problem, and if you're ok with that then it may > work for you. > Unfortunately, we don't have anything else to recommend for generally > constrained problems. > Hi Jason, OK, understood, thanks very much for that info. Best regards, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Apr 23 15:11:10 2015 From: jed at jedbrown.org (Jed Brown) Date: Thu, 23 Apr 2015 14:11:10 -0600 Subject: [petsc-users] MPI speedup In-Reply-To: References: Message-ID: <87fv7qa9lt.fsf@jedbrown.org> Matthew Knepley writes: >Your processor has > 6 cores, each running at 3.4 GHz, and it can do 4flops/cycle using the > vector instructions, Actually, it has AVX, so it can do 8 flops/cycle (1 packed multiply and 1 packed add, each with 4-wide vector registers). >so they > report that it can do 81 GF/s, but you cannot get values to it fast enough > to compute. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From knepley at gmail.com Thu Apr 23 15:18:46 2015 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 23 Apr 2015 15:18:46 -0500 Subject: [petsc-users] MPI speedup In-Reply-To: <87fv7qa9lt.fsf@jedbrown.org> References: <87fv7qa9lt.fsf@jedbrown.org> Message-ID: On Thu, Apr 23, 2015 at 3:11 PM, Jed Brown wrote: > Matthew Knepley writes: > >Your processor has > > 6 cores, each running at 3.4 GHz, and it can do 4flops/cycle using the > > vector instructions, > > Actually, it has AVX, so it can do 8 flops/cycle (1 packed multiply and > 1 packed add, each with 4-wide vector registers). > I don't like counting FMA since its a special, not a generic, floating point instruction, but I guess everyone does it. Matt > >so they > > report that it can do 81 GF/s, but you cannot get values to it fast > enough > > to compute. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From alpkalpalp at gmail.com Thu Apr 23 15:19:10 2015 From: alpkalpalp at gmail.com (Alp Kalpalp) Date: Thu, 23 Apr 2015 23:19:10 +0300 Subject: [petsc-users] how to compute a projection space Message-ID: Hi, I need to compute the following in petsc; P = I -(Z * inv(Z'*A*Z) * Z' What is the best way to compute this? I am planing to use MatPtAP in order to compute the expression inside of the inverse. And then use MatMatSolve instead of inv and comtinue with MatMatMult and use MatShift finally. Any suggestions or comments are welcome.. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Apr 23 16:34:53 2015 From: jed at jedbrown.org (Jed Brown) Date: Thu, 23 Apr 2015 15:34:53 -0600 Subject: [petsc-users] MPI speedup In-Reply-To: References: <87fv7qa9lt.fsf@jedbrown.org> Message-ID: <87d22ua5qa.fsf@jedbrown.org> Matthew Knepley writes: > I don't like counting FMA since its a special, not a generic, floating > point instruction, but > I guess everyone does it. It's not an FMA, it's issuing an independent add and multiply each cycle. Haswell has dual-issue FMA (so "16 flops/cycle"). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From balay at mcs.anl.gov Thu Apr 23 16:53:29 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 23 Apr 2015 16:53:29 -0500 Subject: [petsc-users] MPI speedup In-Reply-To: References: Message-ID: http://ark.intel.com/products/77780/Intel-Core-i7-4930K-Processor-12M-Cache-up-to-3_90-GHz Looks like this CPU has 4 memory channels and supports DDR3-1866 memory. So to get max memory bandwidth - you should make sure you have: - DDR3-1866 [or PC3-14900] memory installed. - have 4 memory cards [or multiple of 4] installed. Satish On Thu, 23 Apr 2015, siddhesh godbole wrote: > Matt > > So that means the time on 10 processes in merely 1.8 times the time on 1 > process?? this is quite difficult to digest! Okay so if memory bandwidth > a controlling factor here how will forming a cluster with same machines > solve this problem? > my cpu has max memory bandwidth of 59 GB/s . > > > Apologies if the question are too silly! > > *Siddhesh M Godbole* > > 5th year Dual Degree, > Civil Eng & Applied Mech. > IIT Madras > > On Thu, Apr 23, 2015 at 4:22 PM, Matthew Knepley wrote: > > > On Thu, Apr 23, 2015 at 5:47 AM, siddhesh godbole < > > siddhesh4godbole at gmail.com> wrote: > > > >> Hello, > >> > >> I want to know about the test which is conducted just after the PETSC is > >> configured on the system to assess the possible speedup by MPI processes. I > >> have saved the result file which says: > >> *Number of MPI processes 10* > >> *Process 0 iitm* > >> *Process 1 iitm* > >> *Process 2 iitm* > >> *Process 3 iitm* > >> *Process 4 iitm* > >> *Process 5 iitm* > >> *Process 6 iitm* > >> *Process 7 iitm* > >> *Process 8 iitm* > >> *Process 9 iitm* > >> *Function Rate (MB/s) * > >> *Copy: 24186.8271* > >> *Scale: 23914.0401* > >> *Add: 27271.7149* > >> *Triad: 27787.1630* > >> *------------------------------------------------* > >> *np speedup* > >> *1 1.0* > >> *2 1.75* > >> *3 1.86* > >> *4 1.84* > >> *5 1.85* > >> *6 1.83* > >> *7 1.76* > >> *8 1.79* > >> *9 1.8* > >> *10 1.8* > >> *Estimation of possible speedup of MPI programs based on Streams > >> benchmark.* > >> > >> 1) What parameters the speedup depends on? > >> > > > > I am not sure what you are asking here. Speedup is defined as the time T > > on 1 process divided > > by the time T_p on p processes: > > > > S = T/T_p > > > > > >> 2) what are the hardware requirements for higher speedup? ( i was > >> expecting atleast 5 times speedup after generating 10 processes. > >> > > > > STREAMS measures the speedup of vectors operations, which are very similar > > to sparse matrix operations. Both > > are limited by memory bandwidth. > > > > > >> 3) what could possibly be done to improve this ? > >> > > > > 1) You could buy more nodes, since each node has a path to memory > > > > 2) You could change algorithms, but this has proven very difficult > > > > Thanks, > > > > Matt > > > > > >> i have intel? Core? i7-4930K CPU @ 3.40GHz ? 12 with 32 GB of RAM and 1 > >> TB disk space. > >> > >> > >> Thanks > >> *Siddhesh M Godbole* > >> > >> 5th year Dual Degree, > >> Civil Eng & Applied Mech. > >> IIT Madras > >> > > > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which their > > experiments lead. > > -- Norbert Wiener > > > From hus003 at ucsd.edu Thu Apr 23 18:48:43 2015 From: hus003 at ucsd.edu (Sun, Hui) Date: Thu, 23 Apr 2015 23:48:43 +0000 Subject: [petsc-users] How do I scatter a Vec from one processor to many? Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB010FE90D@XMAIL-MBX-BH1.AD.UCSD.EDU> I have a Sequential Vec v and a parallel Vec u, and I want to scatter v from processor rank 0 to Vec u. How shall I do it? If I directly call: VecScatterBegin(ctx,v,u,INSERT_VALUES,SCATTER_FORWARD); VecScatterEnd(ctx,v,u,INSERT_VALUES,SCATTER_FORWARD); Then how does the computer know that I want to scatter v from processor 0? If I call: if (rank==0) { VecScatterBegin(ctx,v,u,INSERT_VALUES,SCATTER_FORWARD); VecScatterEnd(ctx,v,u,INSERT_VALUES,SCATTER_FORWARD); } Then how can other processors know that a scatter has been called? So what shall I do here? Best, Hui -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Apr 23 19:57:37 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 23 Apr 2015 19:57:37 -0500 Subject: [petsc-users] How do I scatter a Vec from one processor to many? In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB010FE90D@XMAIL-MBX-BH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE90D@XMAIL-MBX-BH1.AD.UCSD.EDU> Message-ID: <7586A95F-749A-45FF-ACC8-F06278691E1B@mcs.anl.gov> All the information about where the data comes from and where it goes is imbedded in the two IS that are passed to the VecScatterCreate(). Those are in terms of the global indices into the vectors so indicate from which particular process the data comes from and where it goes. All processes that share the vector (the MPI comm that it was created with) need to call the VecScatterBegin/End in case they have some values to contribute. > On Apr 23, 2015, at 6:48 PM, Sun, Hui wrote: > > I have a Sequential Vec v and a parallel Vec u, and I want to scatter v from processor rank 0 to Vec u. How shall I do it? > > If I directly call: > VecScatterBegin(ctx,v,u,INSERT_VALUES,SCATTER_FORWARD); > VecScatterEnd(ctx,v,u,INSERT_VALUES,SCATTER_FORWARD); > > Then how does the computer know that I want to scatter v from processor 0? > > If I call: > if (rank==0) { > VecScatterBegin(ctx,v,u,INSERT_VALUES,SCATTER_FORWARD); > VecScatterEnd(ctx,v,u,INSERT_VALUES,SCATTER_FORWARD); > > } > > Then how can other processors know that a scatter has been called? > > So what shall I do here? > > Best, > Hui From jchang27 at uh.edu Thu Apr 23 22:20:51 2015 From: jchang27 at uh.edu (Justin Chang) Date: Thu, 23 Apr 2015 22:20:51 -0500 Subject: [petsc-users] DMPlexProjectField not including BC constraints? In-Reply-To: References: Message-ID: Matt, I have attempted this but I am getting an error when trying to output the vector into HDF5. This is my implementation: /* Initialize */ PetscDS prob; Vec vlocal, ulocal, vglobal; void (*vX[1])(....) = {velx}; /* Create DM for velocity */ ierr = DMClone(dm,&user.dmVel);CHKERRQ(ierr); ierr = DMPlexCopyCoordinates(dm, user.dmVel);CHKERRQ(ierr); ierr = PetscFECreateDefault(dm, spatialDim, 1, PETSC_TRUE, "solution_", -1, &user.feVel);CHKERRQ(ierr); ierr = DMGetDS(user.dmVel, &prob);CHKERRQ(ierr); ierr = PetscDSSetDiscretization(prob, 0, (PetscObject) user.feVel);CHKERRQ(ierr); ierr = DMCreateGlobalVector(user.dmVel,&vglobal);CHKERRQ(ierr); ierr = PetscObjectSetName((PetscObject) vglobal, "velocity");CHKERRQ(ierr); /* Obtain velocity */ ierr = DMGetLocalVector(dm,&ulocal);CHKERRQ(ierr); ierr = VecDuplicate(ulocal,&vlocal);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(dm,user.u,INSERT_ALL_VALUES,ulocal);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(dm,user.u,INSERT_ALL_VALUES,ulocal);CHKERRQ(ierr); ierr = DMPlexProjectFieldLocal(dm,ulocal,vX,INSERT_ALL_VALUES,vlocal);CHKERRQ(ierr); ierr = DMLocalToGlobalBegin(dm,vlocal,INSERT_ALL_VALUES,vglobal);CHKERRQ(ierr); ierr = DMLocalToGlobalEnd(dm,vlocal,INSERT_ALL_VALUES,vglobal);CHKERRQ(ierr); ierr = VecViewFromOptions(vglobal,NULL,"-vec_view_x");CHKERRQ(ierr); And as soon as I get to the last line, it gives me this error: [0]PETSC ERROR: PetscMallocValidate: error detected at PetscViewerFileSetName_HDF5() line 251 in /home/justin/petsc-dev/src/sys/classes/viewer/impls/hdf5/hdf5v.c [0]PETSC ERROR: Memory at address 0x3077ac0 is corrupted [0]PETSC ERROR: Probably write past beginning or end of array [0]PETSC ERROR: Last intact block allocated in VecCreate_Seq() line 38 in /home/justin/petsc-dev/src/vec/vec/impls/seq/bvec3.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Memory corruption: http://www.mcs.anl.gov/petsc/documentation/installation.html#valgrind [0]PETSC ERROR: [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2766-g0d12e5d GIT Date: 2015-04-21 21:32:23 -0500 [0]PETSC ERROR: ./main on a arch-linux2-c-debug named pacotaco by justin Thu Apr 23 22:14:21 2015 [0]PETSC ERROR: Configure options --download-chaco --download-exodusii --download-fblaslapack --download-hdf5 --download-metis --download-mpich --download-mumps --download-netcdf --download-parmetis --download-scalapack --download-tetgen --download-triangle --with-cc=gcc --with-clanguage=cxx --with-cmake=cmake --with-ctetgen --with-cxx=g++ --with-debugging=1 --with-fc=gfortran --with-valgrind=1 PETSC_ARCH=arch-linux2-c-debug [0]PETSC ERROR: #1 PetscMallocValidate() line 135 in /home/justin/petsc-dev/src/sys/memory/mtr.c [0]PETSC ERROR: #2 PetscViewerFileSetName_HDF5() line 251 in /home/justin/petsc-dev/src/sys/classes/viewer/impls/hdf5/hdf5v.c [0]PETSC ERROR: #3 PetscViewerFileSetName() line 623 in /home/justin/petsc-dev/src/sys/classes/viewer/impls/ascii/filev.c [0]PETSC ERROR: #4 PetscOptionsGetViewer() line 130 in /home/justin/petsc-dev/src/sys/classes/viewer/interface/viewreg.c [0]PETSC ERROR: #5 PetscObjectViewFromOptions() line 2622 in /home/justin/petsc-dev/src/sys/objects/options.c [0]PETSC ERROR: #6 main() line 1237 in /home/justin/Dropbox/dmplex-nonneg/main.c [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -al 100 [0]PETSC ERROR: -at 0.01 [0]PETSC ERROR: -bcloc 496000,496000,536000,542000,0,100,503000,503000,536000,542000,0,100,496000,503000,536000,536000,0,100,496000,503000,542000,542000,0,100 [0]PETSC ERROR: -bcnum 4 [0]PETSC ERROR: -bcval 1,1,1,1 [0]PETSC ERROR: -dim 3 [0]PETSC ERROR: -dm_refine 0 [0]PETSC ERROR: -dt 0.01 [0]PETSC ERROR: -edges 3,3,3 [0]PETSC ERROR: -floc 499550,499600,538950,539000,0,100 [0]PETSC ERROR: -flow datafiles/chrom_flow.dat [0]PETSC ERROR: -fnum 1 [0]PETSC ERROR: -ftime 0,99 [0]PETSC ERROR: -fval -0.1 [0]PETSC ERROR: -ksp_rtol 1.0e-9 [0]PETSC ERROR: -ksp_type cg [0]PETSC ERROR: -lower 0,0,0 [0]PETSC ERROR: -mat_petscspace_order 0 [0]PETSC ERROR: -mesh datafiles/chrom_mesh.dat [0]PETSC ERROR: -mu 3.95e-5 [0]PETSC ERROR: -nonneg 0 [0]PETSC ERROR: -numsteps 0 [0]PETSC ERROR: -options_left 0 [0]PETSC ERROR: -pc_type jacobi [0]PETSC ERROR: -progress 1 [0]PETSC ERROR: -solution_petscspace_order 1 [0]PETSC ERROR: -tao_type tron [0]PETSC ERROR: -upper 1,1,1 [0]PETSC ERROR: -vec_view_x hdf5:solx.h5 [0]PETSC ERROR: -vec_view_y hdf5:soly.h5 [0]PETSC ERROR: -vec_view_z hdf5:solz.h5 [0]PETSC ERROR: -vtuname pressure [0]PETSC ERROR: -vtuprintat 1,20,80,160,320,640,999 [0]PETSC ERROR: -vtusteps 7 [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- Any idea what's going on here? Also, if I replace the last line with VecView(vglobal,PETSC_VIEWER_STDOUT_WORLD) I get a completely wacko error: Vec Object:@????r???????b??j?r??. ~E?b??\2p}^r????"?.b?'xDs??q??|n??a?`K?Lwq??n??a??????p??g?u`?!? |? p?;?:2_?????5n??6/w $]???I!}?k?!c/?Z??s#li??=Y??X?I%?[?f?^?u=??T??Sx??Yc??F?u?Q?6 e??_?;6??a3L?z_?RX????l?D?X?(?(?P?=?/k?9?;z?IB? R??p$??j?-????7??-??H>?h?:????6 5?{??2@?L??]P?I#B? wrote: > On Wed, Apr 22, 2015 at 9:57 PM, Justin Chang wrote: > >> Hello, >> >> I am using DMPlexProjectField(...) to postprocess my final solution u >> into another vector sol. I have the following pointwise functions: >> >> /* x-component of gradient */ >> void velx(const PetscScalar u[], const PetscScalar u_t[], const >> PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const >> PetscScalar a_x[], const PetscReal x[], PetscScalar v[]) >> { >> v[0] = -a[0]*mu*u_x[0]; >> } >> >> /* y-component of gradient */ >> void vely(const PetscScalar u[], const PetscScalar u_t[], const >> PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const >> PetscScalar a_x[], const PetscReal x[], PetscScalar v[]) >> { >> v[0] = -a[0]*mu*u_x[1]; >> } >> >> /* z-component of gradient */ >> void velz(const PetscScalar u[], const PetscScalar u_t[], const >> PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const >> PetscScalar a_x[], const PetscReal x[], PetscScalar v[]) >> { >> v[0] = -a[0]*mu*u_x[2]; >> } >> >> Where a[0] is the cell-wise permeability and mu is the (inverse of) >> viscosity. I am solving the diffusion equation (P1 elements). At the end of >> my simulation I have these lines: >> >> ierr = VecDuplicate(user.u,&user.sol);CHKERRQ(ierr); >> ierr = PetscObjectSetName((PetscObject) user.sol, >> "velocity");CHKERRQ(ierr); >> void (*vX[1])(const PetscScalar *, const PetscScalar *, const PetscScalar >> *, const PetscScalar *, const PetscScalar *, const PetscScalar *, const >> PetscReal *, PetscScalar *) = {velx}; >> ierr = >> DMPlexProjectField(dm,user.u,vX,INSERT_ALL_VALUES,user.sol);CHKERRQ(ierr); >> ierr = VecViewFromOptions(user.sol,NULL,"-vec_view_x");CHKERRQ(ierr); >> >> void (*vY[1])(const PetscScalar *, const PetscScalar *, const PetscScalar >> *, const PetscScalar *, const PetscScalar *, const PetscScalar *, const >> PetscReal *, PetscScalar *) = {vely}; >> ierr = >> DMPlexProjectField(dm,user.u,vY,INSERT_ALL_VALUES,user.sol);CHKERRQ(ierr); >> ierr = VecViewFromOptions(user.sol,NULL,"-vec_view_y");CHKERRQ(ierr); >> >> if (spatialDim == 3) { >> void (*vZ[1])(const PetscScalar *, const PetscScalar *, const >> PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscScalar >> *, const PetscReal *, PetscScalar *) = {velz}; >> ierr = >> DMPlexProjectField(dm,user.u,vZ,INSERT_ALL_VALUES,user.sol);CHKERRQ(ierr); >> ierr = VecViewFromOptions(user.sol,NULL,"-vec_view_z");CHKERRQ(ierr); >> } >> ierr = VecDestroy(&user.sol);CHKERRQ(ierr); >> >> But when i look at the results, the dirichlet constrained values of u >> were projected into the respective locations in sol. I thought putting >> INSERT_ALL_VALUES would insert the pointwise functions into all locations >> including the constrained ones. How can I project the above pointwise >> functions into the constrained fields? Or what am I potentially missing >> here? >> > > There are no Dirichlet constrained values in the global vector user.sol. > When you call VecView() it gets projected to > an enlarged vector including the BC and output. What you want is to > project that vector into a space with no constraints, > so you really need another DM/Section combination. > > I think this will work: > > 1) Get a local vector from dm > > 2) Use DMPlexProjectFieldLocal(INSERT_ALL_VALUES) to fill it up > > 3) DMClone(dm) and give it a Section with no constraints > > 4) Use DMLocalToGlobal() into the new DM > > 5) VecView() that new global vector > > Thanks, > > Matt > > >> Thanks, >> >> >> -- >> Justin Chang >> PhD Candidate, Civil Engineering - Computational Sciences >> University of Houston, Department of Civil and Environmental Engineering >> Houston, TX 77004 >> (512) 963-3262 >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jiangwen84 at gmail.com Thu Apr 23 22:43:42 2015 From: jiangwen84 at gmail.com (Wen Jiang) Date: Thu, 23 Apr 2015 23:43:42 -0400 Subject: [petsc-users] call lapack subroutines in petsc Message-ID: Hi, Could any one tell me how to call a lapack subroutine in PETSc? I would like to use dggev to calculate a generalized eigenvalue problem without using SLEPc. Thanks. Regards, Wen -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Thu Apr 23 23:38:28 2015 From: hzhang at mcs.anl.gov (Hong) Date: Thu, 23 Apr 2015 23:38:28 -0500 Subject: [petsc-users] call lapack subroutines in petsc In-Reply-To: References: Message-ID: Wen: Petsc-Lapack interface functions are listed in petsc/include/petscblaslapck.h. I do not see dggev there. You may add such interface yourself, or call it via SLEPc. Hong > > > Could any one tell me how to call a lapack subroutine in PETSc? I would > like to use dggev to calculate a generalized eigenvalue problem without > using SLEPc. Thanks. > > Regards, > Wen > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hus003 at ucsd.edu Fri Apr 24 00:41:39 2015 From: hus003 at ucsd.edu (Sun, Hui) Date: Fri, 24 Apr 2015 05:41:39 +0000 Subject: [petsc-users] How do I scatter a Vec from one processor to many? In-Reply-To: <7586A95F-749A-45FF-ACC8-F06278691E1B@mcs.anl.gov> References: <7501CC2B7BBCC44A92ECEEC316170ECB010FE90D@XMAIL-MBX-BH1.AD.UCSD.EDU>, <7586A95F-749A-45FF-ACC8-F06278691E1B@mcs.anl.gov> Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB01102BEE@XMAIL-MBX-AH1.AD.UCSD.EDU> Thank you Barry. I think I have figured it out. ________________________________________ From: Barry Smith [bsmith at mcs.anl.gov] Sent: Thursday, April 23, 2015 5:57 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] How do I scatter a Vec from one processor to many? All the information about where the data comes from and where it goes is imbedded in the two IS that are passed to the VecScatterCreate(). Those are in terms of the global indices into the vectors so indicate from which particular process the data comes from and where it goes. All processes that share the vector (the MPI comm that it was created with) need to call the VecScatterBegin/End in case they have some values to contribute. > On Apr 23, 2015, at 6:48 PM, Sun, Hui wrote: > > I have a Sequential Vec v and a parallel Vec u, and I want to scatter v from processor rank 0 to Vec u. How shall I do it? > > If I directly call: > VecScatterBegin(ctx,v,u,INSERT_VALUES,SCATTER_FORWARD); > VecScatterEnd(ctx,v,u,INSERT_VALUES,SCATTER_FORWARD); > > Then how does the computer know that I want to scatter v from processor 0? > > If I call: > if (rank==0) { > VecScatterBegin(ctx,v,u,INSERT_VALUES,SCATTER_FORWARD); > VecScatterEnd(ctx,v,u,INSERT_VALUES,SCATTER_FORWARD); > > } > > Then how can other processors know that a scatter has been called? > > So what shall I do here? > > Best, > Hui From popov at uni-mainz.de Fri Apr 24 02:06:34 2015 From: popov at uni-mainz.de (anton) Date: Fri, 24 Apr 2015 09:06:34 +0200 Subject: [petsc-users] DMDAVecGetArray In-Reply-To: <76B38D4E-6245-43A6-BA4E-0B8E66E55587@mcs.anl.gov> References: <5538D770.2030807@uni-mainz.de> <5538F691.9080401@uni-mainz.de> <76B38D4E-6245-43A6-BA4E-0B8E66E55587@mcs.anl.gov> Message-ID: <5539EB7A.9080103@uni-mainz.de> On 04/23/2015 07:38 PM, Barry Smith wrote: >> On Apr 23, 2015, at 8:41 AM, anton wrote: >> >> >> On 04/23/2015 01:36 PM, Matthew Knepley wrote: >>> On Thu, Apr 23, 2015 at 6:28 AM, anton wrote: >>> Hi, >>> >>> I'm currently calling DMDAVecGet(Restore)Array on a subset of processors. It seems to work correctly. So why does documentation say they are collective on Vec? Are they really? Unlike VecNorm, for example, it's not really clear why DMDAVecGetArray should be collective. >>> >>> The documentation should be fixed. Its "logically collective" just like VecGetArray(), and unlike VecGetArrayRead(). >>> >>> Am I still on a safe side, or I'm seriously violating something that's not immediately obvious. >>> >>> You are violating something that is not obvious. GetArray() will increment the status marker on the Vec since you >>> could have changed something, but then you will get inconsistency across processes in this marker, so you could >>> make different decisions about whether to recompute norms, etc. Hard to debug errors. >>> >> OK, thanks Matt, I'll simply add dummy calls on those processes where I don't need the data. > If you are only "reading" information from the array you should switch to the DMDAVecGetArrayRead() versions available in the development version of PETSc, then you do not need to call on all processes. > > Barry Great, it's not collective. I'll check this out. Anton > >> Anton >> >>> Thanks, >>> >>> Matt >>> >>> Thanks, >>> Anton >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener From C.Klaij at marin.nl Fri Apr 24 06:46:34 2015 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Fri, 24 Apr 2015 11:46:34 +0000 Subject: [petsc-users] PETSC_NULL_OBJECT gets corrupt after call to MatNestGetISs in fortran Message-ID: <1429875994856.10134@marin.nl> Barry, Satish Any news on this issue? Chris > On Feb 12, 2015, at 07:13:08 CST, Smith, Barry wrote: > > Thanks for reporting this. Currently the Fortran stub for this function is generated automatically which means it does not have the logic for handling a PETSC_NULL_OBJECT argument. > > Satish, could you please see if you can add a custom fortran stub for this function in maint? > > Thanks > > Barry > > > On Feb 12, 2015, at 3:02 AM, Klaij, Christiaan wrote: > > > > Using petsc-3.5.3, I noticed that PETSC_NULL_OBJECT gets corrupt after calling MatNestGetISs in fortran. Here's a small example: > > > > $ cat fieldsplittry2.F90 > > program fieldsplittry2 > > > > use petscksp > > implicit none > > #include > > > > PetscErrorCode :: ierr > > PetscInt :: size,i,j,start,end,n=4,numsplit=1 > > PetscScalar :: zero=0.0,one=1.0 > > Vec :: diag3,x,b > > Mat :: A,subA(4),myS > > PC :: pc,subpc(2) > > KSP :: ksp,subksp(2) > > IS :: isg(2) > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr); CHKERRQ(ierr) > > call MPI_Comm_size(PETSC_COMM_WORLD,size,ierr); CHKERRQ(ierr); > > > > ! vectors > > call VecCreateMPI(MPI_COMM_WORLD,3*n,PETSC_DECIDE,diag3,ierr); CHKERRQ(ierr) > > call VecSet(diag3,one,ierr); CHKERRQ(ierr) > > > > call VecCreateMPI(MPI_COMM_WORLD,4*n,PETSC_DECIDE,x,ierr); CHKERRQ(ierr) > > call VecSet(x,zero,ierr); CHKERRQ(ierr) > > > > call VecDuplicate(x,b,ierr); CHKERRQ(ierr) > > call VecSet(b,one,ierr); CHKERRQ(ierr) > > > > ! matrix a00 > > call MatCreateAIJ(MPI_COMM_WORLD,3*n,3*n,PETSC_DECIDE,PETSC_DECIDE,1,PETSC_NULL_INTEGER,0,PETSC_NULL_INTEGER,subA(1),ierr);CHKERRQ(ierr) > > call MatDiagonalSet(subA(1),diag3,INSERT_VALUES,ierr);CHKERRQ(ierr) > > call MatAssemblyBegin(subA(1),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > call MatAssemblyEnd(subA(1),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > > ! matrix a01 > > call MatCreateAIJ(MPI_COMM_WORLD,3*n,n,PETSC_DECIDE,PETSC_DECIDE,1,PETSC_NULL_INTEGER,1,PETSC_NULL_INTEGER,subA(2),ierr);CHKERRQ(ierr) > > call MatGetOwnershipRange(subA(2),start,end,ierr);CHKERRQ(ierr); > > do i=start,end-1 > > j=mod(i,size*n) > > call MatSetValue(subA(2),i,j,one,INSERT_VALUES,ierr);CHKERRQ(ierr) > > end do > > call MatAssemblyBegin(subA(2),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > call MatAssemblyEnd(subA(2),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > > ! matrix a10 > > call MatTranspose(subA(2),MAT_INITIAL_MATRIX,subA(3),ierr);CHKERRQ(ierr) > > > > ! matrix a11 (empty) > > call MatCreateAIJ(MPI_COMM_WORLD,n,n,PETSC_DECIDE,PETSC_DECIDE,0,PETSC_NULL_INTEGER,0,PETSC_NULL_INTEGER,subA(4),ierr);CHKERRQ(ierr) > > call MatAssemblyBegin(subA(4),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > call MatAssemblyEnd(subA(4),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > > ! nested mat [a00,a01;a10,a11] > > call MatCreateNest(MPI_COMM_WORLD,2,PETSC_NULL_OBJECT,2,PETSC_NULL_OBJECT,subA,A,ierr);CHKERRQ(ierr) > > call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > print *, PETSC_NULL_OBJECT > > call MatNestGetISs(A,isg,PETSC_NULL_OBJECT,ierr);CHKERRQ(ierr); > > print *, PETSC_NULL_OBJECT > > > > call PetscFinalize(ierr) > > > > end program fieldsplittry2 > > $ ./fieldsplittry2 > > 0 > > 39367824 > > $ > > > > > > dr. ir. Christiaan Klaij > > CFD Researcher > > Research & Development > > E mailto:C.Klaij at marin.nl > > T +31 317 49 33 44 > > > > > > MARIN > > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > > dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From knepley at gmail.com Fri Apr 24 08:09:08 2015 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 24 Apr 2015 08:09:08 -0500 Subject: [petsc-users] DMPlexProjectField not including BC constraints? In-Reply-To: References: Message-ID: On Thu, Apr 23, 2015 at 10:20 PM, Justin Chang wrote: > Matt, > > I have attempted this but I am getting an error when trying to output the > vector into HDF5. This is my implementation: > > /* Initialize */ > PetscDS prob; > Vec vlocal, ulocal, vglobal; > void (*vX[1])(....) = {velx}; > > /* Create DM for velocity */ > ierr = DMClone(dm,&user.dmVel);CHKERRQ(ierr); > ierr = DMPlexCopyCoordinates(dm, user.dmVel);CHKERRQ(ierr); > ierr = PetscFECreateDefault(dm, spatialDim, 1, PETSC_TRUE, "solution_", > -1, &user.feVel);CHKERRQ(ierr); > ierr = DMGetDS(user.dmVel, &prob);CHKERRQ(ierr); > ierr = PetscDSSetDiscretization(prob, 0, (PetscObject) > user.feVel);CHKERRQ(ierr); > ierr = DMCreateGlobalVector(user.dmVel,&vglobal);CHKERRQ(ierr); > ierr = PetscObjectSetName((PetscObject) vglobal, "velocity");CHKERRQ(ierr); > > /* Obtain velocity */ > ierr = DMGetLocalVector(dm,&ulocal);CHKERRQ(ierr); > ierr = VecDuplicate(ulocal,&vlocal);CHKERRQ(ierr); > ierr = > DMGlobalToLocalBegin(dm,user.u,INSERT_ALL_VALUES,ulocal);CHKERRQ(ierr); > ierr = > DMGlobalToLocalEnd(dm,user.u,INSERT_ALL_VALUES,ulocal);CHKERRQ(ierr); > ierr = > DMPlexProjectFieldLocal(dm,ulocal,vX,INSERT_ALL_VALUES,vlocal);CHKERRQ(ierr); > ierr = > DMLocalToGlobalBegin(dm,vlocal,INSERT_ALL_VALUES,vglobal);CHKERRQ(ierr); > ierr = > DMLocalToGlobalEnd(dm,vlocal,INSERT_ALL_VALUES,vglobal);CHKERRQ(ierr); > 1) Shouldn't that be user.dmVel above, not dm? 2) Are you changing from vector to scalar here? I think so, and so you would need an extra step since that projection assumes that the layout is the same as the input. I do not currently have anything that maps between different layouts like you want. It will have to be written. THEN we can use the above code. Matt > ierr = VecViewFromOptions(vglobal,NULL,"-vec_view_x");CHKERRQ(ierr); > > And as soon as I get to the last line, it gives me this error: > > > [0]PETSC ERROR: PetscMallocValidate: error detected at > PetscViewerFileSetName_HDF5() line 251 in > /home/justin/petsc-dev/src/sys/classes/viewer/impls/hdf5/hdf5v.c > [0]PETSC ERROR: Memory at address 0x3077ac0 is corrupted > [0]PETSC ERROR: Probably write past beginning or end of array > [0]PETSC ERROR: Last intact block allocated in VecCreate_Seq() line 38 in > /home/justin/petsc-dev/src/vec/vec/impls/seq/bvec3.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Memory corruption: > http://www.mcs.anl.gov/petsc/documentation/installation.html#valgrind > [0]PETSC ERROR: > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2766-g0d12e5d GIT > Date: 2015-04-21 21:32:23 -0500 > [0]PETSC ERROR: ./main on a arch-linux2-c-debug named pacotaco by justin > Thu Apr 23 22:14:21 2015 > [0]PETSC ERROR: Configure options --download-chaco --download-exodusii > --download-fblaslapack --download-hdf5 --download-metis --download-mpich > --download-mumps --download-netcdf --download-parmetis --download-scalapack > --download-tetgen --download-triangle --with-cc=gcc --with-clanguage=cxx > --with-cmake=cmake --with-ctetgen --with-cxx=g++ --with-debugging=1 > --with-fc=gfortran --with-valgrind=1 PETSC_ARCH=arch-linux2-c-debug > [0]PETSC ERROR: #1 PetscMallocValidate() line 135 in > /home/justin/petsc-dev/src/sys/memory/mtr.c > [0]PETSC ERROR: #2 PetscViewerFileSetName_HDF5() line 251 in > /home/justin/petsc-dev/src/sys/classes/viewer/impls/hdf5/hdf5v.c > [0]PETSC ERROR: #3 PetscViewerFileSetName() line 623 in > /home/justin/petsc-dev/src/sys/classes/viewer/impls/ascii/filev.c > [0]PETSC ERROR: #4 PetscOptionsGetViewer() line 130 in > /home/justin/petsc-dev/src/sys/classes/viewer/interface/viewreg.c > [0]PETSC ERROR: #5 PetscObjectViewFromOptions() line 2622 in > /home/justin/petsc-dev/src/sys/objects/options.c > [0]PETSC ERROR: #6 main() line 1237 in > /home/justin/Dropbox/dmplex-nonneg/main.c > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -al 100 > [0]PETSC ERROR: -at 0.01 > [0]PETSC ERROR: -bcloc > 496000,496000,536000,542000,0,100,503000,503000,536000,542000,0,100,496000,503000,536000,536000,0,100,496000,503000,542000,542000,0,100 > [0]PETSC ERROR: -bcnum 4 > [0]PETSC ERROR: -bcval 1,1,1,1 > [0]PETSC ERROR: -dim 3 > [0]PETSC ERROR: -dm_refine 0 > [0]PETSC ERROR: -dt 0.01 > [0]PETSC ERROR: -edges 3,3,3 > [0]PETSC ERROR: -floc 499550,499600,538950,539000,0,100 > [0]PETSC ERROR: -flow datafiles/chrom_flow.dat > [0]PETSC ERROR: -fnum 1 > [0]PETSC ERROR: -ftime 0,99 > [0]PETSC ERROR: -fval -0.1 > [0]PETSC ERROR: -ksp_rtol 1.0e-9 > [0]PETSC ERROR: -ksp_type cg > [0]PETSC ERROR: -lower 0,0,0 > [0]PETSC ERROR: -mat_petscspace_order 0 > [0]PETSC ERROR: -mesh datafiles/chrom_mesh.dat > [0]PETSC ERROR: -mu 3.95e-5 > [0]PETSC ERROR: -nonneg 0 > [0]PETSC ERROR: -numsteps 0 > [0]PETSC ERROR: -options_left 0 > [0]PETSC ERROR: -pc_type jacobi > [0]PETSC ERROR: -progress 1 > [0]PETSC ERROR: -solution_petscspace_order 1 > [0]PETSC ERROR: -tao_type tron > [0]PETSC ERROR: -upper 1,1,1 > [0]PETSC ERROR: -vec_view_x hdf5:solx.h5 > [0]PETSC ERROR: -vec_view_y hdf5:soly.h5 > [0]PETSC ERROR: -vec_view_z hdf5:solz.h5 > [0]PETSC ERROR: -vtuname pressure > [0]PETSC ERROR: -vtuprintat 1,20,80,160,320,640,999 > [0]PETSC ERROR: -vtusteps 7 > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov---------- > > Any idea what's going on here? Also, if I replace the last line with > VecView(vglobal,PETSC_VIEWER_STDOUT_WORLD) I get a completely wacko error: > > Vec Object:@?? ??r???????b? ? j?r??. > > ~E?b??\2p}^r????"?.b?'xDs??q??|n??a?`K? Lwq??n??a??????p??g?u`?!? > > |? > > p?;? :2_???? ?5n??6/w $]???I!}?k?! c /?Z??s# li??=Y?? X?I% > ?[?f?^?u=??T??Sx??Yc??F ?u?Q?6 > e??_?;6??a3L? z_?R X??? > ?l?D?X?(?(?P?= ?/k?9?;z? I B? > > R??p$?? j?-? ???7??- ??H>?h?:?? ??6 5?{??2@?L??]P?I#B? e???? ?V? ?Y?g? ?3?2WY??X?j?)?&M?[?s?!Q&+m??*?l?V^????uo?? ?D`?z 1 MPI > processes > type: seq > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [0]PETSC ERROR: PetscMallocValidate: error detected at > PetscSignalHandlerDefault() line 149 in > /home/justin/petsc-dev/src/sys/error/signal.c > [0]PETSC ERROR: Memory at address 0x2575ac0 is corrupted > [0]PETSC ERROR: Probably write past beginning or end of array > [0]PETSC ERROR: Last intact block allocated in VecCreate_Seq() line 38 in > /home/justin/petsc-dev/src/vec/vec/impls/seq/bvec3.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Memory corruption: > http://www.mcs.anl.gov/petsc/documentation/installation.html#valgrind > [0]PETSC ERROR: > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2766-g0d12e5d GIT > Date: 2015-04-21 21:32:23 -0500 > [0]PETSC ERROR: ./main on a arch-linux2-c-debug named pacotaco by justin > Thu Apr 23 22:17:26 2015 > [0]PETSC ERROR: Configure options --download-chaco --download-exodusii > --download-fblaslapack --download-hdf5 --download-metis --download-mpich > --download-mumps --download-netcdf --download-parmetis --download-scalapack > --download-tetgen --download-triangle --with-cc=gcc --with-clanguage=cxx > --with-cmake=cmake --with-ctetgen --with-cxx=g++ --with-debugging=1 > --with-fc=gfortran --with-valgrind=1 PETSC_ARCH=arch-linux2-c-debug > [0]PETSC ERROR: #1 PetscMallocValidate() line 135 in > /home/justin/petsc-dev/src/sys/memory/mtr.c > [0]PETSC ERROR: #2 PetscSignalHandlerDefault() line 149 in > /home/justin/petsc-dev/src/sys/error/signal.c > > > However, if I run the program with debugging off (i.e., > --with-debugging=0), VecView gives me an answer that seems relatively > correct. > > Any idea what's going on here? > > Thanks, > > On Thu, Apr 23, 2015 at 4:00 AM, Matthew Knepley > wrote: > >> On Wed, Apr 22, 2015 at 9:57 PM, Justin Chang wrote: >> >>> Hello, >>> >>> I am using DMPlexProjectField(...) to postprocess my final solution u >>> into another vector sol. I have the following pointwise functions: >>> >>> /* x-component of gradient */ >>> void velx(const PetscScalar u[], const PetscScalar u_t[], const >>> PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const >>> PetscScalar a_x[], const PetscReal x[], PetscScalar v[]) >>> { >>> v[0] = -a[0]*mu*u_x[0]; >>> } >>> >>> /* y-component of gradient */ >>> void vely(const PetscScalar u[], const PetscScalar u_t[], const >>> PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const >>> PetscScalar a_x[], const PetscReal x[], PetscScalar v[]) >>> { >>> v[0] = -a[0]*mu*u_x[1]; >>> } >>> >>> /* z-component of gradient */ >>> void velz(const PetscScalar u[], const PetscScalar u_t[], const >>> PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const >>> PetscScalar a_x[], const PetscReal x[], PetscScalar v[]) >>> { >>> v[0] = -a[0]*mu*u_x[2]; >>> } >>> >>> Where a[0] is the cell-wise permeability and mu is the (inverse of) >>> viscosity. I am solving the diffusion equation (P1 elements). At the end of >>> my simulation I have these lines: >>> >>> ierr = VecDuplicate(user.u,&user.sol);CHKERRQ(ierr); >>> ierr = PetscObjectSetName((PetscObject) user.sol, >>> "velocity");CHKERRQ(ierr); >>> void (*vX[1])(const PetscScalar *, const PetscScalar *, const >>> PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscScalar >>> *, const PetscReal *, PetscScalar *) = {velx}; >>> ierr = >>> DMPlexProjectField(dm,user.u,vX,INSERT_ALL_VALUES,user.sol);CHKERRQ(ierr); >>> ierr = VecViewFromOptions(user.sol,NULL,"-vec_view_x");CHKERRQ(ierr); >>> >>> void (*vY[1])(const PetscScalar *, const PetscScalar *, const >>> PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscScalar >>> *, const PetscReal *, PetscScalar *) = {vely}; >>> ierr = >>> DMPlexProjectField(dm,user.u,vY,INSERT_ALL_VALUES,user.sol);CHKERRQ(ierr); >>> ierr = VecViewFromOptions(user.sol,NULL,"-vec_view_y");CHKERRQ(ierr); >>> >>> if (spatialDim == 3) { >>> void (*vZ[1])(const PetscScalar *, const PetscScalar *, const >>> PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscScalar >>> *, const PetscReal *, PetscScalar *) = {velz}; >>> ierr = >>> DMPlexProjectField(dm,user.u,vZ,INSERT_ALL_VALUES,user.sol);CHKERRQ(ierr); >>> ierr = VecViewFromOptions(user.sol,NULL,"-vec_view_z");CHKERRQ(ierr); >>> } >>> ierr = VecDestroy(&user.sol);CHKERRQ(ierr); >>> >>> But when i look at the results, the dirichlet constrained values of u >>> were projected into the respective locations in sol. I thought putting >>> INSERT_ALL_VALUES would insert the pointwise functions into all locations >>> including the constrained ones. How can I project the above pointwise >>> functions into the constrained fields? Or what am I potentially missing >>> here? >>> >> >> There are no Dirichlet constrained values in the global vector user.sol. >> When you call VecView() it gets projected to >> an enlarged vector including the BC and output. What you want is to >> project that vector into a space with no constraints, >> so you really need another DM/Section combination. >> >> I think this will work: >> >> 1) Get a local vector from dm >> >> 2) Use DMPlexProjectFieldLocal(INSERT_ALL_VALUES) to fill it up >> >> 3) DMClone(dm) and give it a Section with no constraints >> >> 4) Use DMLocalToGlobal() into the new DM >> >> 5) VecView() that new global vector >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> >>> >>> -- >>> Justin Chang >>> PhD Candidate, Civil Engineering - Computational Sciences >>> University of Houston, Department of Civil and Environmental Engineering >>> Houston, TX 77004 >>> (512) 963-3262 >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.knezevic at akselos.com Fri Apr 24 08:17:31 2015 From: david.knezevic at akselos.com (David Knezevic) Date: Fri, 24 Apr 2015 09:17:31 -0400 Subject: [petsc-users] TAO IPM In-Reply-To: References: <9ff483acd60147a7b2a437bc9a8bc1af@LUCKMAN.anl.gov> Message-ID: On Thu, Apr 23, 2015 at 2:57 PM, Jason Sarich wrote: > Hi David, > > The IPM code is a naive implementation and not very battle-tested. There > is nothing specifically broken, but the linear solve operator becomes very > ill-conditioned and for most cases will require a direct solver (e.g. > superlu). This is the major problem, and if you're ok with that then it may > work for you. > Unfortunately, we don't have anything else to recommend for generally > constrained problems. > A quick follow-up question: I notice that the "toy" example I referred to earlier in this thread uses VecCreateSeq and MatCreateSeq, so I just wanted to check if TAO IPM is restricted to sequential data? (My guess is that it's not restricted to sequential data, and that Seq is used only because the problem is so small, but I just wanted to check.) Thanks, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From mlohry at princeton.edu Fri Apr 24 10:13:22 2015 From: mlohry at princeton.edu (Mark Lohry) Date: Fri, 24 Apr 2015 11:13:22 -0400 Subject: [petsc-users] Any TSGL examples, or higher order BDFs? Message-ID: <553A5D92.2000600@princeton.edu> I'd like to use a 3rd+ order BDF with TS, which appears to only have 1st order (backward euler/TSBEULER), and then the bindings to SUNDIALS. I'd like to avoid SUNDIALS as it looks like it forces the use of it's own non-linear solvers. Am I correct in thinking the preferred way to construct a BDF with TS is to use TSGL? Are there any examples of TSGL in use? I'm not seeing any in the docs. -Mark Lohry From jed at jedbrown.org Fri Apr 24 10:20:08 2015 From: jed at jedbrown.org (Jed Brown) Date: Fri, 24 Apr 2015 09:20:08 -0600 Subject: [petsc-users] Any TSGL examples, or higher order BDFs? In-Reply-To: <553A5D92.2000600@princeton.edu> References: <553A5D92.2000600@princeton.edu> Message-ID: <874mo5a6zb.fsf@jedbrown.org> Mark Lohry writes: > I'd like to use a 3rd+ order BDF with TS, which appears to only have 1st > order (backward euler/TSBEULER), and then the bindings to SUNDIALS. What about TSARKIMEX, TSROSW, and TSEIMEX? These are really the methods we recommend. > I'd like to avoid SUNDIALS as it looks like it forces the use of it's > own non-linear solvers. Yeah. > Am I correct in thinking the preferred way to construct a BDF with TS is > to use TSGL? Are there any examples of TSGL in use? I'm not seeing any > in the docs. You can use -ts_type gl, but I haven't worked with those methods in a long time and they have not been robust in practice. I recommend using ROSW as a default. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From mlohry at princeton.edu Fri Apr 24 10:28:27 2015 From: mlohry at princeton.edu (Mark Lohry) Date: Fri, 24 Apr 2015 11:28:27 -0400 Subject: [petsc-users] Any TSGL examples, or higher order BDFs? In-Reply-To: <874mo5a6zb.fsf@jedbrown.org> References: <553A5D92.2000600@princeton.edu> <874mo5a6zb.fsf@jedbrown.org> Message-ID: <553A611B.5060200@princeton.edu> Thanks Jed. I'm not terribly familiar with those mixed implicit/explicit methods; are they appropriate for fully implicit solutions by just setting the explicit part to 0? (I realize that's more a math question than a petsc one) -Mark On 04/24/2015 11:20 AM, Jed Brown wrote: > Mark Lohry writes: > >> I'd like to use a 3rd+ order BDF with TS, which appears to only have 1st >> order (backward euler/TSBEULER), and then the bindings to SUNDIALS. > What about TSARKIMEX, TSROSW, and TSEIMEX? These are really the methods > we recommend. > >> I'd like to avoid SUNDIALS as it looks like it forces the use of it's >> own non-linear solvers. > Yeah. > >> Am I correct in thinking the preferred way to construct a BDF with TS is >> to use TSGL? Are there any examples of TSGL in use? I'm not seeing any >> in the docs. > You can use -ts_type gl, but I haven't worked with those methods in a > long time and they have not been robust in practice. I recommend using > ROSW as a default. From jed at jedbrown.org Fri Apr 24 10:34:09 2015 From: jed at jedbrown.org (Jed Brown) Date: Fri, 24 Apr 2015 09:34:09 -0600 Subject: [petsc-users] Any TSGL examples, or higher order BDFs? In-Reply-To: <553A611B.5060200@princeton.edu> References: <553A5D92.2000600@princeton.edu> <874mo5a6zb.fsf@jedbrown.org> <553A611B.5060200@princeton.edu> Message-ID: <871tj9a6by.fsf@jedbrown.org> Mark Lohry writes: > Thanks Jed. I'm not terribly familiar with those mixed implicit/explicit > methods; are they appropriate for fully implicit solutions by just > setting the explicit part to 0? Yes. They are implicit methods if you only set an IFunction (no RHSFunction). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From gideon.simpson at gmail.com Fri Apr 24 12:49:41 2015 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Fri, 24 Apr 2015 13:49:41 -0400 Subject: [petsc-users] preonly ksp with convergence testing? Message-ID: <6C6B8E1A-A3D4-4E9F-B13C-A958BE4835EE@gmail.com> Is it possible (for pedagogical purposes) to use a preonly ksp with, say, Jacobi or SOR, and have it terminate when a convergence criterion is met? -gideon -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Apr 24 13:02:13 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 24 Apr 2015 13:02:13 -0500 Subject: [petsc-users] preonly ksp with convergence testing? In-Reply-To: <6C6B8E1A-A3D4-4E9F-B13C-A958BE4835EE@gmail.com> References: <6C6B8E1A-A3D4-4E9F-B13C-A958BE4835EE@gmail.com> Message-ID: If you want multiple iterations (applications of the preconditioner) then use -ksp_type richardson see KSPRICHARDSON > On Apr 24, 2015, at 12:49 PM, Gideon Simpson wrote: > > Is it possible (for pedagogical purposes) to use a preonly ksp with, say, Jacobi or SOR, and have it terminate when a convergence criterion is met? > > -gideon > From bsmith at mcs.anl.gov Fri Apr 24 13:02:29 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 24 Apr 2015 13:02:29 -0500 Subject: [petsc-users] preonly ksp with convergence testing? In-Reply-To: <6C6B8E1A-A3D4-4E9F-B13C-A958BE4835EE@gmail.com> References: <6C6B8E1A-A3D4-4E9F-B13C-A958BE4835EE@gmail.com> Message-ID: <3EB71CED-7C57-436F-B057-166FCFD361D2@mcs.anl.gov> If you want multiple iterations (applications of the preconditioner) then use -ksp_type richardson see KSPRICHARDSON > On Apr 24, 2015, at 12:49 PM, Gideon Simpson wrote: > > Is it possible (for pedagogical purposes) to use a preonly ksp with, say, Jacobi or SOR, and have it terminate when a convergence criterion is met? > > -gideon > From danyang.su at gmail.com Fri Apr 24 13:05:19 2015 From: danyang.su at gmail.com (Danyang Su) Date: Fri, 24 Apr 2015 11:05:19 -0700 Subject: [petsc-users] Floating point exception Message-ID: <553A85DF.5000400@gmail.com> Hi All, One of my case crashes because of floating point exception when using 4 processors, as shown below. But if I run this case with 1 processor, it works fine. I have tested the codes with around 100 cases up to 768 processors, all other cases work fine. I just wonder if this kind of error is caused because of NaN in jacobi matrix, RHS or preconditioner? I can check all the entries of jacobi matrix to see if the value is valid, but this seems not a good idea as it takes a long time to reach this point. If I restart the simulation from a specified time (e.g., 7.685 in this case), then the error does not occur. Would you please give me any suggestion on debugging this case? Thanks and Regards, Danyang timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 Reduce time step for reactive transport timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 Reduce time step for reactive transport timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Floating point exception [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [2]PETSC ERROR: Floating point exception [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at end of function: Parameter number 3 [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at end of function: Parameter number 3 [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 15:38:52 2015 [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack [1]PETSC ERROR: #1 VecValidValues() line 34 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 15:38:52 2015 [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack [2]PETSC ERROR: #1 VecValidValues() line 34 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c [1]PETSC ERROR: #2 PCApply() line 442 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c [2]PETSC ERROR: #3 KSP_PCApply() line 230 in /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h #3 KSP_PCApply() line 230 in /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c [2]PETSC ERROR: #6 KSPSolve() line 459 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: #6 KSPSolve() line 459 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested [mpiexec at nwmop] Press Ctrl-C again to force abort From bsmith at mcs.anl.gov Fri Apr 24 13:12:23 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 24 Apr 2015 13:12:23 -0500 Subject: [petsc-users] Floating point exception In-Reply-To: <553A85DF.5000400@gmail.com> References: <553A85DF.5000400@gmail.com> Message-ID: <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> > On Apr 24, 2015, at 1:05 PM, Danyang Su wrote: > > Hi All, > > One of my case crashes because of floating point exception when using 4 processors, as shown below. But if I run this case with 1 processor, it works fine. I have tested the codes with around 100 cases up to 768 processors, all other cases work fine. I just wonder if this kind of error is caused because of NaN in jacobi matrix, RHS or preconditioner? Yes, almost for sure it is one of these places. First run the bad case with -fp_trap if all goes well you'll see the function where the FPE is generated. Then run also with -start_in_debugger and type cont in all four debugger windows. When the FPE happens the debugger should stop showing exactly where the FPE happens. Barry > I can check all the entries of jacobi matrix to see if the value is valid, but this seems not a good idea as it takes a long time to reach this point. If I restart the simulation from a specified time (e.g., 7.685 in this case), then the error does not occur. > > Would you please give me any suggestion on debugging this case? > > Thanks and Regards, > > Danyang > > > timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 > timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 > timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 > timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 > timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 > Reduce time step for reactive transport > timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 > Reduce time step for reactive transport > timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 > [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [1]PETSC ERROR: Floating point exception > [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [2]PETSC ERROR: Floating point exception > [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at end of function: Parameter number 3 > [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 > [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at end of function: Parameter number 3 > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 > [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 15:38:52 2015 > [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack > [1]PETSC ERROR: #1 VecValidValues() line 34 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c > ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 15:38:52 2015 > [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack > [2]PETSC ERROR: #1 VecValidValues() line 34 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c > [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c > [1]PETSC ERROR: #2 PCApply() line 442 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c > [2]PETSC ERROR: #3 KSP_PCApply() line 230 in /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h > #3 KSP_PCApply() line 230 in /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h > [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c > [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c > [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c > [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c > [2]PETSC ERROR: #6 KSPSolve() line 459 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: #6 KSPSolve() line 459 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c > ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested > [mpiexec at nwmop] Press Ctrl-C again to force abort From lb2653 at columbia.edu Fri Apr 24 13:16:20 2015 From: lb2653 at columbia.edu (Luc Berger-Vergiat) Date: Fri, 24 Apr 2015 14:16:20 -0400 Subject: [petsc-users] PCGASM in Fortran Message-ID: <553A8874.6030605@columbi.edu> Hi all, I am trying to use the restricted additive Schwarz preconditioner for my problem but most of the fortran stub are not available for the functions related to domain decomposition (e.g. PCGASMSetSubdomains is not available so I can't assign subdomains). Is there a reason for this? Is there an easy work-around? Or should I just re-implement additive Schwarz in Fortran (that's not to hard in serial but I'm not sure about the parallel version). Thanks for your help! -- Best, Luc From knepley at gmail.com Fri Apr 24 13:36:35 2015 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 24 Apr 2015 13:36:35 -0500 Subject: [petsc-users] PCGASM in Fortran In-Reply-To: <553A8874.6030605@columbi.edu> References: <553A8874.6030605@columbi.edu> Message-ID: On Fri, Apr 24, 2015 at 1:16 PM, Luc Berger-Vergiat wrote: > Hi all, > I am trying to use the restricted additive Schwarz preconditioner for my > problem but most of the fortran stub are not available for the functions > related to domain decomposition (e.g. PCGASMSetSubdomains is not available > so I can't assign subdomains). > Is there a reason for this? Is there an easy work-around? Or should I just > re-implement additive Schwarz in Fortran (that's not to hard in serial but > I'm not sure about the parallel version). > Can you just use PCASM, which has the all bindings? Thanks, Matt > Thanks for your help! > > -- > Best, > Luc > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From karpeev at mcs.anl.gov Fri Apr 24 13:42:36 2015 From: karpeev at mcs.anl.gov (Dmitry Karpeyev) Date: Fri, 24 Apr 2015 18:42:36 +0000 Subject: [petsc-users] PCGASM in Fortran In-Reply-To: References: <553A8874.6030605@columbi.edu> Message-ID: Unless you need subdomains that live on more than one processor, you should use PCASM. If you do need PCGASM, it is undergoing a major overhaul that will be ready in a couple of weeks. Dmitry. On Fri, Apr 24, 2015 at 1:36 PM Matthew Knepley wrote: > On Fri, Apr 24, 2015 at 1:16 PM, Luc Berger-Vergiat > wrote: > >> Hi all, >> I am trying to use the restricted additive Schwarz preconditioner for my >> problem but most of the fortran stub are not available for the functions >> related to domain decomposition (e.g. PCGASMSetSubdomains is not available >> so I can't assign subdomains). >> Is there a reason for this? Is there an easy work-around? Or should I >> just re-implement additive Schwarz in Fortran (that's not to hard in serial >> but I'm not sure about the parallel version). >> > > Can you just use PCASM, which has the all bindings? > > Thanks, > > Matt > > >> Thanks for your help! >> >> -- >> Best, >> Luc >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lb2653 at columbia.edu Fri Apr 24 13:48:14 2015 From: lb2653 at columbia.edu (Luc Berger-Vergiat) Date: Fri, 24 Apr 2015 14:48:14 -0400 Subject: [petsc-users] PCGASM in Fortran In-Reply-To: References: <553A8874.6030605@columbi.edu> Message-ID: <553A8FEE.7050100@columbi.edu> Ok, I'll check PCASM instead. Thanks Best, Luc On 04/24/2015 02:42 PM, Dmitry Karpeyev wrote: > Unless you need subdomains that live on more than one processor, you > should use PCASM. If you do need PCGASM, it is undergoing a major > overhaul that will be ready in a couple of weeks. > > Dmitry. > > On Fri, Apr 24, 2015 at 1:36 PM Matthew Knepley > wrote: > > On Fri, Apr 24, 2015 at 1:16 PM, Luc Berger-Vergiat > > wrote: > > Hi all, > I am trying to use the restricted additive Schwarz > preconditioner for my problem but most of the fortran stub are > not available for the functions related to domain > decomposition (e.g. PCGASMSetSubdomains is not available so I > can't assign subdomains). > Is there a reason for this? Is there an easy work-around? Or > should I just re-implement additive Schwarz in Fortran (that's > not to hard in serial but I'm not sure about the parallel > version). > > > Can you just use PCASM, which has the all bindings? > > Thanks, > > Matt > > Thanks for your help! > > -- > Best, > Luc > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to > which their experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Fri Apr 24 15:01:15 2015 From: danyang.su at gmail.com (Danyang Su) Date: Fri, 24 Apr 2015 13:01:15 -0700 Subject: [petsc-users] Floating point exception In-Reply-To: <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> Message-ID: <553AA10B.5090208@gmail.com> On 15-04-24 11:12 AM, Barry Smith wrote: >> On Apr 24, 2015, at 1:05 PM, Danyang Su wrote: >> >> Hi All, >> >> One of my case crashes because of floating point exception when using 4 processors, as shown below. But if I run this case with 1 processor, it works fine. I have tested the codes with around 100 cases up to 768 processors, all other cases work fine. I just wonder if this kind of error is caused because of NaN in jacobi matrix, RHS or preconditioner? > Yes, almost for sure it is one of these places. > > First run the bad case with -fp_trap if all goes well you'll see the function where the FPE is generated. Then run also with -start_in_debugger and > type cont in all four debugger windows. When the FPE happens the debugger should stop showing exactly where the FPE happens. > > Barry Hi Barry, If run with -fp_trap -start_in_debugger, I got the following error [0]PETSC ERROR: *** unknown floating point error occurred *** [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) [0]PETSC ERROR: where the result is a bitwise OR of the following flags: [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 [0]PETSC ERROR: Try option -start_in_debugger [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x7F4FEAB1C7D7 #1 0x7F4FEAB1CDDE #2 0x7F4FE9E1AD3F #3 0x7F4FE9E1ACC9 #4 0x7F4FE9E1E0D7 #5 0x7F4FEB0B6DCB #6 0x7F4FEB0B1825 #7 0x7F4FEB0B817F #8 0x7F4FE9E1AD3F #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) #10 0x4C6C87 in gcreact_ at gcreact.F90:678 #11 0x707E19 in initicrt_ at initicrt.F90:589 #12 0x4F42D0 in initprob_ at initprob.F90:430 #13 0x5AAF72 in driver_pc at driver_pc.F90:438 I checked the code at tprfrtlc.F90:2393, realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & (cx(ix),ix=1,nxout)/) All the values (time, c, cx) are reasonable, as shown below. The only possibility is that realbuffer_gb is in declared as real*4 if using sing precision output while time, c, cx are declared in real*8. I have a lot of similar data conversion from real*8 to real*4 output, other code does not return error. time 0.0000000000000000 c 1 9.9999999999999995E-008 c 2 3.1555251077549618E-003 c 3 7.1657814842179362E-008 c 4 1.0976214263087059E-067 c 5 5.2879822292305797E-004 c 6 9.9999999999999964E-005 c 7 6.4055731968811337E-005 c 8 3.4607572892578404E-020 cx 1 3.4376650636008101E-005 cx 2 7.3989678854017763E-012 cx 3 9.5317170613607207E-012 cx 4 2.2344525794718353E-015 cx 5 3.0624685689695889E-008 cx 6 1.0046157902783967E-007 cx 7 1.5320169154914984E-004 cx 8 8.6930292776346176E-014 cx 9 3.5944267559348721E-005 cx 10 3.0072645866951157E-018 cx 11 2.3592486321095017E-013 Thanks, Danyang > >> I can check all the entries of jacobi matrix to see if the value is valid, but this seems not a good idea as it takes a long time to reach this point. If I restart the simulation from a specified time (e.g., 7.685 in this case), then the error does not occur. >> >> Would you please give me any suggestion on debugging this case? >> >> Thanks and Regards, >> >> Danyang >> >> >> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >> Reduce time step for reactive transport >> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >> Reduce time step for reactive transport >> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >> [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [1]PETSC ERROR: Floating point exception >> [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [2]PETSC ERROR: Floating point exception >> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at end of function: Parameter number 3 >> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at end of function: Parameter number 3 >> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 15:38:52 2015 >> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack >> [1]PETSC ERROR: #1 VecValidValues() line 34 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 15:38:52 2015 >> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack >> [2]PETSC ERROR: #1 VecValidValues() line 34 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >> [1]PETSC ERROR: #2 PCApply() line 442 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >> #3 KSP_PCApply() line 230 in /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >> [2]PETSC ERROR: #6 KSPSolve() line 459 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >> [1]PETSC ERROR: #6 KSPSolve() line 459 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >> [mpiexec at nwmop] Press Ctrl-C again to force abort From bsmith at mcs.anl.gov Fri Apr 24 15:16:53 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 24 Apr 2015 15:16:53 -0500 Subject: [petsc-users] Floating point exception In-Reply-To: <553AA10B.5090208@gmail.com> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> Message-ID: It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate #elif defined PETSC_HAVE_XMMINTRIN_H _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); #else change it to #elif defined PETSC_HAVE_XMMINTRIN_H _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); #else Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. Barry > On Apr 24, 2015, at 3:01 PM, Danyang Su wrote: > > > > On 15-04-24 11:12 AM, Barry Smith wrote: >>> On Apr 24, 2015, at 1:05 PM, Danyang Su wrote: >>> >>> Hi All, >>> >>> One of my case crashes because of floating point exception when using 4 processors, as shown below. But if I run this case with 1 processor, it works fine. I have tested the codes with around 100 cases up to 768 processors, all other cases work fine. I just wonder if this kind of error is caused because of NaN in jacobi matrix, RHS or preconditioner? >> Yes, almost for sure it is one of these places. >> >> First run the bad case with -fp_trap if all goes well you'll see the function where the FPE is generated. Then run also with -start_in_debugger and >> type cont in all four debugger windows. When the FPE happens the debugger should stop showing exactly where the FPE happens. >> >> Barry > Hi Barry, > > If run with -fp_trap -start_in_debugger, I got the following error > > [0]PETSC ERROR: *** unknown floating point error occurred *** > [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the > [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) > [0]PETSC ERROR: where the result is a bitwise OR of the following flags: > [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 > [0]PETSC ERROR: Try option -start_in_debugger > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c > [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error > > Program received signal SIGABRT: Process abort signal. > > Backtrace for this error: > #0 0x7F4FEAB1C7D7 > #1 0x7F4FEAB1CDDE > #2 0x7F4FE9E1AD3F > #3 0x7F4FE9E1ACC9 > #4 0x7F4FE9E1E0D7 > #5 0x7F4FEB0B6DCB > #6 0x7F4FEB0B1825 > #7 0x7F4FEB0B817F > #8 0x7F4FE9E1AD3F > #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) > #10 0x4C6C87 in gcreact_ at gcreact.F90:678 > #11 0x707E19 in initicrt_ at initicrt.F90:589 > #12 0x4F42D0 in initprob_ at initprob.F90:430 > #13 0x5AAF72 in driver_pc at driver_pc.F90:438 > > I checked the code at tprfrtlc.F90:2393, > > realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & > (cx(ix),ix=1,nxout)/) > > All the values (time, c, cx) are reasonable, as shown below. The only possibility is that realbuffer_gb is in declared as real*4 if using sing precision output while time, c, cx are declared in real*8. I have a lot of similar data conversion from real*8 to real*4 output, other code does not return error. > > time 0.0000000000000000 > c 1 9.9999999999999995E-008 > c 2 3.1555251077549618E-003 > c 3 7.1657814842179362E-008 > c 4 1.0976214263087059E-067 > c 5 5.2879822292305797E-004 > c 6 9.9999999999999964E-005 > c 7 6.4055731968811337E-005 > c 8 3.4607572892578404E-020 > cx 1 3.4376650636008101E-005 > cx 2 7.3989678854017763E-012 > cx 3 9.5317170613607207E-012 > cx 4 2.2344525794718353E-015 > cx 5 3.0624685689695889E-008 > cx 6 1.0046157902783967E-007 > cx 7 1.5320169154914984E-004 > cx 8 8.6930292776346176E-014 > cx 9 3.5944267559348721E-005 > cx 10 3.0072645866951157E-018 > cx 11 2.3592486321095017E-013 > > Thanks, > > Danyang > >> >>> I can check all the entries of jacobi matrix to see if the value is valid, but this seems not a good idea as it takes a long time to reach this point. If I restart the simulation from a specified time (e.g., 7.685 in this case), then the error does not occur. >>> >>> Would you please give me any suggestion on debugging this case? >>> >>> Thanks and Regards, >>> >>> Danyang >>> >>> >>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> Reduce time step for reactive transport >>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> Reduce time step for reactive transport >>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [1]PETSC ERROR: Floating point exception >>> [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [2]PETSC ERROR: Floating point exception >>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at end of function: Parameter number 3 >>> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at end of function: Parameter number 3 >>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 15:38:52 2015 >>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack >>> [1]PETSC ERROR: #1 VecValidValues() line 34 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 15:38:52 2015 >>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack >>> [2]PETSC ERROR: #1 VecValidValues() line 34 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>> [1]PETSC ERROR: #2 PCApply() line 442 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>> #3 KSP_PCApply() line 230 in /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>> [2]PETSC ERROR: #6 KSPSolve() line 459 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>> [1]PETSC ERROR: #6 KSPSolve() line 459 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>> [mpiexec at nwmop] Press Ctrl-C again to force abort > From mlohry at princeton.edu Fri Apr 24 15:16:49 2015 From: mlohry at princeton.edu (Mark Lohry) Date: Fri, 24 Apr 2015 16:16:49 -0400 Subject: [petsc-users] Turning off TSADAPT still adapts time step In-Reply-To: <553AA10B.5090208@gmail.com> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> Message-ID: <553AA4B1.5010603@princeton.edu> Trying a simple stiff test case (y'=y^2*(1-y)) using TSBEULER, to get a feel for implicit stepping with JFNK. I'm giving it a pretty large timestep, and the default appears to start adapting the time step around the stiff part of the solution. So I tried turning it off: TSAdapt adapt; TSGetAdapt(its,&adapt); TSAdaptSetType(adapt,TSADAPTNONE); and even fixing the limits: TSAdaptSetStepLimits(adapt,40.0,40.0); And it still ignores it and adapts anyway. At t=800 (where the solution gets very stiff, so one would normally see adaptive stepping there if it was desired) it's dropping dt from 40 to 10. Printing (t,y) here from the ts monitor: 600.000000,0.002790; 640.000000,0.003198; 680.000000,0.003761; 720.000000,0.004606; 760.000000,0.006072; 800.000000,0.010155; 810.000000,0.011452; 820.000000,0.013161; 830.000000,0.015538; 840.000000,0.019126; 850.000000,0.025426; 860.000000,0.043639; 900.000000,0.975519; 940.000000,0.999402; 980.000000,0.999985; 1020.000000,1.000000; 1060.000000,1.000000; 1100.000000,1.000000; 1140.000000,1.000000; 1180.000000,1.000000; Any idea? Adaptive stepping is great, but I'd still like to be able to fix the time step. From balay at mcs.anl.gov Fri Apr 24 15:23:18 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 24 Apr 2015 15:23:18 -0500 Subject: [petsc-users] Floating point exception In-Reply-To: <553AA10B.5090208@gmail.com> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> Message-ID: c 4 1.0976214263087059E-067 I don't think this number can be stored in a real*4. Satish On Fri, 24 Apr 2015, Danyang Su wrote: > > > On 15-04-24 11:12 AM, Barry Smith wrote: > > > On Apr 24, 2015, at 1:05 PM, Danyang Su wrote: > > > > > > Hi All, > > > > > > One of my case crashes because of floating point exception when using 4 > > > processors, as shown below. But if I run this case with 1 processor, it > > > works fine. I have tested the codes with around 100 cases up to 768 > > > processors, all other cases work fine. I just wonder if this kind of error > > > is caused because of NaN in jacobi matrix, RHS or preconditioner? > > Yes, almost for sure it is one of these places. > > > > First run the bad case with -fp_trap if all goes well you'll see the > > function where the FPE is generated. Then run also with -start_in_debugger > > and > > type cont in all four debugger windows. When the FPE happens the debugger > > should stop showing exactly where the FPE happens. > > > > Barry > Hi Barry, > > If run with -fp_trap -start_in_debugger, I got the following error > > [0]PETSC ERROR: *** unknown floating point error occurred *** > [0]PETSC ERROR: The specific exception can be determined by running in a > debugger. When the > [0]PETSC ERROR: debugger traps the signal, the exception can be found with > fetestexcept(0x3d) > [0]PETSC ERROR: where the result is a bitwise OR of the following flags: > [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 > FE_UNDERFLOW=0x10 FE_INEXACT=0x20 > [0]PETSC ERROR: Try option -start_in_debugger > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 > /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c > [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped > floating point error > > Program received signal SIGABRT: Process abort signal. > > Backtrace for this error: > #0 0x7F4FEAB1C7D7 > #1 0x7F4FEAB1CDDE > #2 0x7F4FE9E1AD3F > #3 0x7F4FE9E1ACC9 > #4 0x7F4FE9E1E0D7 > #5 0x7F4FEB0B6DCB > #6 0x7F4FEB0B1825 > #7 0x7F4FEB0B817F > #8 0x7F4FE9E1AD3F > #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) > #10 0x4C6C87 in gcreact_ at gcreact.F90:678 > #11 0x707E19 in initicrt_ at initicrt.F90:589 > #12 0x4F42D0 in initprob_ at initprob.F90:430 > #13 0x5AAF72 in driver_pc at driver_pc.F90:438 > > I checked the code at tprfrtlc.F90:2393, > > realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & > (cx(ix),ix=1,nxout)/) > > All the values (time, c, cx) are reasonable, as shown below. The only > possibility is that realbuffer_gb is in declared as real*4 if using sing > precision output while time, c, cx are declared in real*8. I have a lot of > similar data conversion from real*8 to real*4 output, other code does not > return error. > > time 0.0000000000000000 > c 1 9.9999999999999995E-008 > c 2 3.1555251077549618E-003 > c 3 7.1657814842179362E-008 > c 4 1.0976214263087059E-067 > c 5 5.2879822292305797E-004 > c 6 9.9999999999999964E-005 > c 7 6.4055731968811337E-005 > c 8 3.4607572892578404E-020 > cx 1 3.4376650636008101E-005 > cx 2 7.3989678854017763E-012 > cx 3 9.5317170613607207E-012 > cx 4 2.2344525794718353E-015 > cx 5 3.0624685689695889E-008 > cx 6 1.0046157902783967E-007 > cx 7 1.5320169154914984E-004 > cx 8 8.6930292776346176E-014 > cx 9 3.5944267559348721E-005 > cx 10 3.0072645866951157E-018 > cx 11 2.3592486321095017E-013 > > Thanks, > > Danyang > > > > > > I can check all the entries of jacobi matrix to see if the value is valid, > > > but this seems not a good idea as it takes a long time to reach this > > > point. If I restart the simulation from a specified time (e.g., 7.685 in > > > this case), then the error does not occur. > > > > > > Would you please give me any suggestion on debugging this case? > > > > > > Thanks and Regards, > > > > > > Danyang > > > > > > > > > timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 > > > timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 > > > timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 > > > timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 > > > timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 > > > timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 > > > timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 > > > timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 > > > timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 > > > timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 > > > Reduce time step for reactive transport > > > timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 > > > timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 > > > Reduce time step for reactive transport > > > timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 > > > timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 > > > [1]PETSC ERROR: --------------------- Error Message > > > -------------------------------------------------------------- > > > [1]PETSC ERROR: Floating point exception > > > [2]PETSC ERROR: --------------------- Error Message > > > -------------------------------------------------------------- > > > [2]PETSC ERROR: Floating point exception > > > [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite > > > at end of function: Parameter number 3 > > > [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > > > for trouble shooting. > > > [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 > > > [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is > > > not-a-number or infinite at end of function: Parameter number 3 > > > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > > > for trouble shooting. > > > [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 > > > [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by > > > dsu Thu Apr 23 15:38:52 2015 > > > [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc > > > --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich > > > --download-mumps --download-hypre --download-superlu_dist --download-metis > > > --download-parmetis --download-scalapack > > > [1]PETSC ERROR: #1 VecValidValues() line 34 in > > > /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c > > > ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 > > > 15:38:52 2015 > > > [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc > > > --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich > > > --download-mumps --download-hypre --download-superlu_dist --download-metis > > > --download-parmetis --download-scalapack > > > [2]PETSC ERROR: #1 VecValidValues() line 34 in > > > /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c > > > [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in > > > /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c > > > [1]PETSC ERROR: #2 PCApply() line 442 in > > > /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c > > > [2]PETSC ERROR: #3 KSP_PCApply() line 230 in > > > /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h > > > #3 KSP_PCApply() line 230 in > > > /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h > > > [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in > > > /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c > > > [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in > > > /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c > > > [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in > > > /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c > > > [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in > > > /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c > > > [2]PETSC ERROR: #6 KSPSolve() line 459 in > > > /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c > > > [1]PETSC ERROR: #6 KSPSolve() line 459 in > > > /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c > > > ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested > > > [mpiexec at nwmop] Press Ctrl-C again to force abort > > > From bsmith at mcs.anl.gov Fri Apr 24 15:29:54 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 24 Apr 2015 15:29:54 -0500 Subject: [petsc-users] Turning off TSADAPT still adapts time step In-Reply-To: <553AA4B1.5010603@princeton.edu> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AA4B1.5010603@princeton.edu> Message-ID: Run in the debugger (with one process obviously) to see why it is still calling an adaptor. Barry > On Apr 24, 2015, at 3:16 PM, Mark Lohry wrote: > > Trying a simple stiff test case (y'=y^2*(1-y)) using TSBEULER, to get a feel for implicit stepping with JFNK. > > I'm giving it a pretty large timestep, and the default appears to start adapting the time step around the stiff part of the solution. So I tried turning it off: > > TSAdapt adapt; > TSGetAdapt(its,&adapt); > TSAdaptSetType(adapt,TSADAPTNONE); > > and even fixing the limits: > > TSAdaptSetStepLimits(adapt,40.0,40.0); > > And it still ignores it and adapts anyway. At t=800 (where the solution gets very stiff, so one would normally see adaptive stepping there if it was desired) it's dropping dt from 40 to 10. > > Printing (t,y) here from the ts monitor: > > 600.000000,0.002790; > 640.000000,0.003198; > 680.000000,0.003761; > 720.000000,0.004606; > 760.000000,0.006072; > 800.000000,0.010155; > 810.000000,0.011452; > 820.000000,0.013161; > 830.000000,0.015538; > 840.000000,0.019126; > 850.000000,0.025426; > 860.000000,0.043639; > 900.000000,0.975519; > 940.000000,0.999402; > 980.000000,0.999985; > 1020.000000,1.000000; > 1060.000000,1.000000; > 1100.000000,1.000000; > 1140.000000,1.000000; > 1180.000000,1.000000; > > Any idea? Adaptive stepping is great, but I'd still like to be able to fix the time step. > > > From danyang.su at gmail.com Fri Apr 24 15:54:18 2015 From: danyang.su at gmail.com (Danyang Su) Date: Fri, 24 Apr 2015 13:54:18 -0700 Subject: [petsc-users] Floating point exception In-Reply-To: References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> Message-ID: <553AAD7A.1030807@gmail.com> On 15-04-24 01:23 PM, Satish Balay wrote: > c 4 1.0976214263087059E-067 > > I don't think this number can be stored in a real*4. > > Satish Thanks, Satish. It is caused by this number. > > On Fri, 24 Apr 2015, Danyang Su wrote: > >> >> On 15-04-24 11:12 AM, Barry Smith wrote: >>>> On Apr 24, 2015, at 1:05 PM, Danyang Su wrote: >>>> >>>> Hi All, >>>> >>>> One of my case crashes because of floating point exception when using 4 >>>> processors, as shown below. But if I run this case with 1 processor, it >>>> works fine. I have tested the codes with around 100 cases up to 768 >>>> processors, all other cases work fine. I just wonder if this kind of error >>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>> Yes, almost for sure it is one of these places. >>> >>> First run the bad case with -fp_trap if all goes well you'll see the >>> function where the FPE is generated. Then run also with -start_in_debugger >>> and >>> type cont in all four debugger windows. When the FPE happens the debugger >>> should stop showing exactly where the FPE happens. >>> >>> Barry >> Hi Barry, >> >> If run with -fp_trap -start_in_debugger, I got the following error >> >> [0]PETSC ERROR: *** unknown floating point error occurred *** >> [0]PETSC ERROR: The specific exception can be determined by running in a >> debugger. When the >> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >> fetestexcept(0x3d) >> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >> [0]PETSC ERROR: Try option -start_in_debugger >> [0]PETSC ERROR: likely location of problem given in stack below >> [0]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> [0]PETSC ERROR: INSTEAD the line number of the start of the function >> [0]PETSC ERROR: is given. >> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >> floating point error >> >> Program received signal SIGABRT: Process abort signal. >> >> Backtrace for this error: >> #0 0x7F4FEAB1C7D7 >> #1 0x7F4FEAB1CDDE >> #2 0x7F4FE9E1AD3F >> #3 0x7F4FE9E1ACC9 >> #4 0x7F4FE9E1E0D7 >> #5 0x7F4FEB0B6DCB >> #6 0x7F4FEB0B1825 >> #7 0x7F4FEB0B817F >> #8 0x7F4FE9E1AD3F >> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >> #11 0x707E19 in initicrt_ at initicrt.F90:589 >> #12 0x4F42D0 in initprob_ at initprob.F90:430 >> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >> >> I checked the code at tprfrtlc.F90:2393, >> >> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >> (cx(ix),ix=1,nxout)/) >> >> All the values (time, c, cx) are reasonable, as shown below. The only >> possibility is that realbuffer_gb is in declared as real*4 if using sing >> precision output while time, c, cx are declared in real*8. I have a lot of >> similar data conversion from real*8 to real*4 output, other code does not >> return error. >> >> time 0.0000000000000000 >> c 1 9.9999999999999995E-008 >> c 2 3.1555251077549618E-003 >> c 3 7.1657814842179362E-008 >> c 4 1.0976214263087059E-067 >> c 5 5.2879822292305797E-004 >> c 6 9.9999999999999964E-005 >> c 7 6.4055731968811337E-005 >> c 8 3.4607572892578404E-020 >> cx 1 3.4376650636008101E-005 >> cx 2 7.3989678854017763E-012 >> cx 3 9.5317170613607207E-012 >> cx 4 2.2344525794718353E-015 >> cx 5 3.0624685689695889E-008 >> cx 6 1.0046157902783967E-007 >> cx 7 1.5320169154914984E-004 >> cx 8 8.6930292776346176E-014 >> cx 9 3.5944267559348721E-005 >> cx 10 3.0072645866951157E-018 >> cx 11 2.3592486321095017E-013 >> >> Thanks, >> >> Danyang >> >>>> I can check all the entries of jacobi matrix to see if the value is valid, >>>> but this seems not a good idea as it takes a long time to reach this >>>> point. If I restart the simulation from a specified time (e.g., 7.685 in >>>> this case), then the error does not occur. >>>> >>>> Would you please give me any suggestion on debugging this case? >>>> >>>> Thanks and Regards, >>>> >>>> Danyang >>>> >>>> >>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>> Reduce time step for reactive transport >>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>> Reduce time step for reactive transport >>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>> [1]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [1]PETSC ERROR: Floating point exception >>>> [2]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [2]PETSC ERROR: Floating point exception >>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >>>> at end of function: Parameter number 3 >>>> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>> not-a-number or infinite at end of function: Parameter number 3 >>>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>>> dsu Thu Apr 23 15:38:52 2015 >>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>> --download-parmetis --download-scalapack >>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >>>> 15:38:52 2015 >>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>> --download-parmetis --download-scalapack >>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>> #3 KSP_PCApply() line 230 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >> >> From balay at mcs.anl.gov Fri Apr 24 15:54:36 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 24 Apr 2015 15:54:36 -0500 Subject: [petsc-users] PETSC_NULL_OBJECT gets corrupt after call to MatNestGetISs in fortran In-Reply-To: <1429875994856.10134@marin.nl> References: <1429875994856.10134@marin.nl> Message-ID: Sorry for dropping the ball on this issue. I pushed the following fix to maint branch. https://bitbucket.org/petsc/petsc/commits/3a4d7b9a6c83003720b45dc0635fc32ea52a4309 To use this change with petsc-3.5.3 - you can drop in the attached replacement files at: src/mat/impls/nest/ftn-custom/zmatnestf.c src/mat/impls/nest/ftn-auto/matnestf.c Satish On Fri, 24 Apr 2015, Klaij, Christiaan wrote: > Barry, Satish > > Any news on this issue? > > Chris > > > On Feb 12, 2015, at 07:13:08 CST, Smith, Barry wrote: > > > > Thanks for reporting this. Currently the Fortran stub for this function is generated automatically which means it does not have the logic for handling a PETSC_NULL_OBJECT argument. > > > > Satish, could you please see if you can add a custom fortran stub for this function in maint? > > > > Thanks > > > > Barry > > > > > On Feb 12, 2015, at 3:02 AM, Klaij, Christiaan wrote: > > > > > > Using petsc-3.5.3, I noticed that PETSC_NULL_OBJECT gets corrupt after calling MatNestGetISs in fortran. Here's a small example: > > > > > > $ cat fieldsplittry2.F90 > > > program fieldsplittry2 > > > > > > use petscksp > > > implicit none > > > #include > > > > > > PetscErrorCode :: ierr > > > PetscInt :: size,i,j,start,end,n=4,numsplit=1 > > > PetscScalar :: zero=0.0,one=1.0 > > > Vec :: diag3,x,b > > > Mat :: A,subA(4),myS > > > PC :: pc,subpc(2) > > > KSP :: ksp,subksp(2) > > > IS :: isg(2) > > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr); CHKERRQ(ierr) > > > call MPI_Comm_size(PETSC_COMM_WORLD,size,ierr); CHKERRQ(ierr); > > > > > > ! vectors > > > call VecCreateMPI(MPI_COMM_WORLD,3*n,PETSC_DECIDE,diag3,ierr); CHKERRQ(ierr) > > > call VecSet(diag3,one,ierr); CHKERRQ(ierr) > > > > > > call VecCreateMPI(MPI_COMM_WORLD,4*n,PETSC_DECIDE,x,ierr); CHKERRQ(ierr) > > > call VecSet(x,zero,ierr); CHKERRQ(ierr) > > > > > > call VecDuplicate(x,b,ierr); CHKERRQ(ierr) > > > call VecSet(b,one,ierr); CHKERRQ(ierr) > > > > > > ! matrix a00 > > > call MatCreateAIJ(MPI_COMM_WORLD,3*n,3*n,PETSC_DECIDE,PETSC_DECIDE,1,PETSC_NULL_INTEGER,0,PETSC_NULL_INTEGER,subA(1),ierr);CHKERRQ(ierr) > > > call MatDiagonalSet(subA(1),diag3,INSERT_VALUES,ierr);CHKERRQ(ierr) > > > call MatAssemblyBegin(subA(1),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > call MatAssemblyEnd(subA(1),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > > > > ! matrix a01 > > > call MatCreateAIJ(MPI_COMM_WORLD,3*n,n,PETSC_DECIDE,PETSC_DECIDE,1,PETSC_NULL_INTEGER,1,PETSC_NULL_INTEGER,subA(2),ierr);CHKERRQ(ierr) > > > call MatGetOwnershipRange(subA(2),start,end,ierr);CHKERRQ(ierr); > > > do i=start,end-1 > > > j=mod(i,size*n) > > > call MatSetValue(subA(2),i,j,one,INSERT_VALUES,ierr);CHKERRQ(ierr) > > > end do > > > call MatAssemblyBegin(subA(2),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > call MatAssemblyEnd(subA(2),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > > > > ! matrix a10 > > > call MatTranspose(subA(2),MAT_INITIAL_MATRIX,subA(3),ierr);CHKERRQ(ierr) > > > > > > ! matrix a11 (empty) > > > call MatCreateAIJ(MPI_COMM_WORLD,n,n,PETSC_DECIDE,PETSC_DECIDE,0,PETSC_NULL_INTEGER,0,PETSC_NULL_INTEGER,subA(4),ierr);CHKERRQ(ierr) > > > call MatAssemblyBegin(subA(4),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > call MatAssemblyEnd(subA(4),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > > > > ! nested mat [a00,a01;a10,a11] > > > call MatCreateNest(MPI_COMM_WORLD,2,PETSC_NULL_OBJECT,2,PETSC_NULL_OBJECT,subA,A,ierr);CHKERRQ(ierr) > > > call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > print *, PETSC_NULL_OBJECT > > > call MatNestGetISs(A,isg,PETSC_NULL_OBJECT,ierr);CHKERRQ(ierr); > > > print *, PETSC_NULL_OBJECT > > > > > > call PetscFinalize(ierr) > > > > > > end program fieldsplittry2 > > > $ ./fieldsplittry2 > > > 0 > > > 39367824 > > > $ > > > > > > > > > dr. ir. Christiaan Klaij > > > CFD Researcher > > > Research & Development > > > E mailto:C.Klaij at marin.nl > > > T +31 317 49 33 44 > > > > > > > > > MARIN > > > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > > > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > > > > > > dr. ir. Christiaan Klaij > CFD Researcher > Research & Development > E mailto:C.Klaij at marin.nl > T +31 317 49 33 44 > > > MARIN > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: zmatnestf.c URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: matnestf.c URL: From mlohry at princeton.edu Fri Apr 24 16:08:22 2015 From: mlohry at princeton.edu (Mark Lohry) Date: Fri, 24 Apr 2015 17:08:22 -0400 Subject: [petsc-users] Turning off TSADAPT still adapts time step In-Reply-To: References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AA4B1.5010603@princeton.edu> Message-ID: <553AB0C6.5010503@princeton.edu> I'm new to PETSc and not sure how to go about debugging. Running with -start_in_debugger runs in gdb with the normal output (where would i put a breakpoint for this?) and running with -info dumps a lot of information but only related to SNES and KSP, nothing about time stepping or adapt. On 04/24/2015 04:29 PM, Barry Smith wrote: > Run in the debugger (with one process obviously) to see why it is still calling an adaptor. > > Barry > >> On Apr 24, 2015, at 3:16 PM, Mark Lohry wrote: >> >> Trying a simple stiff test case (y'=y^2*(1-y)) using TSBEULER, to get a feel for implicit stepping with JFNK. >> >> I'm giving it a pretty large timestep, and the default appears to start adapting the time step around the stiff part of the solution. So I tried turning it off: >> >> TSAdapt adapt; >> TSGetAdapt(its,&adapt); >> TSAdaptSetType(adapt,TSADAPTNONE); >> >> and even fixing the limits: >> >> TSAdaptSetStepLimits(adapt,40.0,40.0); >> >> And it still ignores it and adapts anyway. At t=800 (where the solution gets very stiff, so one would normally see adaptive stepping there if it was desired) it's dropping dt from 40 to 10. >> >> Printing (t,y) here from the ts monitor: >> >> 600.000000,0.002790; >> 640.000000,0.003198; >> 680.000000,0.003761; >> 720.000000,0.004606; >> 760.000000,0.006072; >> 800.000000,0.010155; >> 810.000000,0.011452; >> 820.000000,0.013161; >> 830.000000,0.015538; >> 840.000000,0.019126; >> 850.000000,0.025426; >> 860.000000,0.043639; >> 900.000000,0.975519; >> 940.000000,0.999402; >> 980.000000,0.999985; >> 1020.000000,1.000000; >> 1060.000000,1.000000; >> 1100.000000,1.000000; >> 1140.000000,1.000000; >> 1180.000000,1.000000; >> >> Any idea? Adaptive stepping is great, but I'd still like to be able to fix the time step. >> >> >> From jchang27 at uh.edu Fri Apr 24 17:38:31 2015 From: jchang27 at uh.edu (Justin Chang) Date: Fri, 24 Apr 2015 17:38:31 -0500 Subject: [petsc-users] DMPlexProjectField not including BC constraints? In-Reply-To: References: Message-ID: I actually made a mistake in PetscFECreateDefault(...), if that's what you're referring to, and put dm instead of user.dmVel. However the problem remains. Everything is a scalar (i.e., numComp is 1 for both DMs). dm and user.dmVel should have the same chart but user.dmVel does not invoke DMPlexAddBoundary(..). Each spatial index of u_x[] will be printed separately. Right now I am simply working with the x component. That said, is it possible to get this sort of mapping? Thanks, On Fri, Apr 24, 2015 at 8:09 AM, Matthew Knepley wrote: > On Thu, Apr 23, 2015 at 10:20 PM, Justin Chang wrote: > >> Matt, >> >> I have attempted this but I am getting an error when trying to output the >> vector into HDF5. This is my implementation: >> >> /* Initialize */ >> PetscDS prob; >> Vec vlocal, ulocal, vglobal; >> void (*vX[1])(....) = {velx}; >> >> /* Create DM for velocity */ >> ierr = DMClone(dm,&user.dmVel);CHKERRQ(ierr); >> ierr = DMPlexCopyCoordinates(dm, user.dmVel);CHKERRQ(ierr); >> ierr = PetscFECreateDefault(dm, spatialDim, 1, PETSC_TRUE, "solution_", >> -1, &user.feVel);CHKERRQ(ierr); >> ierr = DMGetDS(user.dmVel, &prob);CHKERRQ(ierr); >> ierr = PetscDSSetDiscretization(prob, 0, (PetscObject) >> user.feVel);CHKERRQ(ierr); >> ierr = DMCreateGlobalVector(user.dmVel,&vglobal);CHKERRQ(ierr); >> ierr = PetscObjectSetName((PetscObject) vglobal, >> "velocity");CHKERRQ(ierr); >> >> /* Obtain velocity */ >> ierr = DMGetLocalVector(dm,&ulocal);CHKERRQ(ierr); >> ierr = VecDuplicate(ulocal,&vlocal);CHKERRQ(ierr); >> ierr = >> DMGlobalToLocalBegin(dm,user.u,INSERT_ALL_VALUES,ulocal);CHKERRQ(ierr); >> ierr = >> DMGlobalToLocalEnd(dm,user.u,INSERT_ALL_VALUES,ulocal);CHKERRQ(ierr); >> ierr = >> DMPlexProjectFieldLocal(dm,ulocal,vX,INSERT_ALL_VALUES,vlocal);CHKERRQ(ierr); >> ierr = >> DMLocalToGlobalBegin(dm,vlocal,INSERT_ALL_VALUES,vglobal);CHKERRQ(ierr); >> ierr = >> DMLocalToGlobalEnd(dm,vlocal,INSERT_ALL_VALUES,vglobal);CHKERRQ(ierr); >> > > 1) Shouldn't that be user.dmVel above, not dm? > > 2) Are you changing from vector to scalar here? I think so, and so you > would need an extra step > since that projection assumes that the layout is the same as the > input. > > I do not currently have anything that maps between different layouts like > you want. It will have to > be written. THEN we can use the above code. > > Matt > > >> ierr = VecViewFromOptions(vglobal,NULL,"-vec_view_x");CHKERRQ(ierr); >> >> And as soon as I get to the last line, it gives me this error: >> >> >> [0]PETSC ERROR: PetscMallocValidate: error detected at >> PetscViewerFileSetName_HDF5() line 251 in >> /home/justin/petsc-dev/src/sys/classes/viewer/impls/hdf5/hdf5v.c >> [0]PETSC ERROR: Memory at address 0x3077ac0 is corrupted >> [0]PETSC ERROR: Probably write past beginning or end of array >> [0]PETSC ERROR: Last intact block allocated in VecCreate_Seq() line 38 in >> /home/justin/petsc-dev/src/vec/vec/impls/seq/bvec3.c >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Memory corruption: >> http://www.mcs.anl.gov/petsc/documentation/installation.html#valgrind >> [0]PETSC ERROR: >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2766-g0d12e5d GIT >> Date: 2015-04-21 21:32:23 -0500 >> [0]PETSC ERROR: ./main on a arch-linux2-c-debug named pacotaco by justin >> Thu Apr 23 22:14:21 2015 >> [0]PETSC ERROR: Configure options --download-chaco --download-exodusii >> --download-fblaslapack --download-hdf5 --download-metis --download-mpich >> --download-mumps --download-netcdf --download-parmetis --download-scalapack >> --download-tetgen --download-triangle --with-cc=gcc --with-clanguage=cxx >> --with-cmake=cmake --with-ctetgen --with-cxx=g++ --with-debugging=1 >> --with-fc=gfortran --with-valgrind=1 PETSC_ARCH=arch-linux2-c-debug >> [0]PETSC ERROR: #1 PetscMallocValidate() line 135 in >> /home/justin/petsc-dev/src/sys/memory/mtr.c >> [0]PETSC ERROR: #2 PetscViewerFileSetName_HDF5() line 251 in >> /home/justin/petsc-dev/src/sys/classes/viewer/impls/hdf5/hdf5v.c >> [0]PETSC ERROR: #3 PetscViewerFileSetName() line 623 in >> /home/justin/petsc-dev/src/sys/classes/viewer/impls/ascii/filev.c >> [0]PETSC ERROR: #4 PetscOptionsGetViewer() line 130 in >> /home/justin/petsc-dev/src/sys/classes/viewer/interface/viewreg.c >> [0]PETSC ERROR: #5 PetscObjectViewFromOptions() line 2622 in >> /home/justin/petsc-dev/src/sys/objects/options.c >> [0]PETSC ERROR: #6 main() line 1237 in >> /home/justin/Dropbox/dmplex-nonneg/main.c >> [0]PETSC ERROR: PETSc Option Table entries: >> [0]PETSC ERROR: -al 100 >> [0]PETSC ERROR: -at 0.01 >> [0]PETSC ERROR: -bcloc >> 496000,496000,536000,542000,0,100,503000,503000,536000,542000,0,100,496000,503000,536000,536000,0,100,496000,503000,542000,542000,0,100 >> [0]PETSC ERROR: -bcnum 4 >> [0]PETSC ERROR: -bcval 1,1,1,1 >> [0]PETSC ERROR: -dim 3 >> [0]PETSC ERROR: -dm_refine 0 >> [0]PETSC ERROR: -dt 0.01 >> [0]PETSC ERROR: -edges 3,3,3 >> [0]PETSC ERROR: -floc 499550,499600,538950,539000,0,100 >> [0]PETSC ERROR: -flow datafiles/chrom_flow.dat >> [0]PETSC ERROR: -fnum 1 >> [0]PETSC ERROR: -ftime 0,99 >> [0]PETSC ERROR: -fval -0.1 >> [0]PETSC ERROR: -ksp_rtol 1.0e-9 >> [0]PETSC ERROR: -ksp_type cg >> [0]PETSC ERROR: -lower 0,0,0 >> [0]PETSC ERROR: -mat_petscspace_order 0 >> [0]PETSC ERROR: -mesh datafiles/chrom_mesh.dat >> [0]PETSC ERROR: -mu 3.95e-5 >> [0]PETSC ERROR: -nonneg 0 >> [0]PETSC ERROR: -numsteps 0 >> [0]PETSC ERROR: -options_left 0 >> [0]PETSC ERROR: -pc_type jacobi >> [0]PETSC ERROR: -progress 1 >> [0]PETSC ERROR: -solution_petscspace_order 1 >> [0]PETSC ERROR: -tao_type tron >> [0]PETSC ERROR: -upper 1,1,1 >> [0]PETSC ERROR: -vec_view_x hdf5:solx.h5 >> [0]PETSC ERROR: -vec_view_y hdf5:soly.h5 >> [0]PETSC ERROR: -vec_view_z hdf5:solz.h5 >> [0]PETSC ERROR: -vtuname pressure >> [0]PETSC ERROR: -vtuprintat 1,20,80,160,320,640,999 >> [0]PETSC ERROR: -vtusteps 7 >> [0]PETSC ERROR: ----------------End of Error Message -------send entire >> error message to petsc-maint at mcs.anl.gov---------- >> >> Any idea what's going on here? Also, if I replace the last line with >> VecView(vglobal,PETSC_VIEWER_STDOUT_WORLD) I get a completely wacko error: >> >> Vec Object:@?? ??r???????b? ? j?r??. >> >> ~E?b??\2p}^r????"?.b?'xDs??q??|n??a?`K? Lwq??n??a??????p??g?u`?!? >> >> |? >> >> p?;? :2_???? ?5n??6/w $]???I!}?k?! c /?Z??s# li??=Y?? X?I% >> ?[?f?^?u=??T??Sx??Yc??F ?u?Q?6 >> e??_?;6??a3L? z_?R X??? >> ?l?D?X?(?(?P?= ?/k?9?;z? I B? >> >> R??p$?? j?-? ???7??- ??H>?h?:?? ??6 5?{??2@?L??]P?I#B?> e???? ?V? ?Y?g? ?3?2WY??X?j?)?&M?[?s?!Q&+m??*?l?V^????uo?? ?D`?z 1 MPI >> processes >> type: seq >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS >> X to find memory corruption errors >> [0]PETSC ERROR: PetscMallocValidate: error detected at >> PetscSignalHandlerDefault() line 149 in >> /home/justin/petsc-dev/src/sys/error/signal.c >> [0]PETSC ERROR: Memory at address 0x2575ac0 is corrupted >> [0]PETSC ERROR: Probably write past beginning or end of array >> [0]PETSC ERROR: Last intact block allocated in VecCreate_Seq() line 38 in >> /home/justin/petsc-dev/src/vec/vec/impls/seq/bvec3.c >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Memory corruption: >> http://www.mcs.anl.gov/petsc/documentation/installation.html#valgrind >> [0]PETSC ERROR: >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2766-g0d12e5d GIT >> Date: 2015-04-21 21:32:23 -0500 >> [0]PETSC ERROR: ./main on a arch-linux2-c-debug named pacotaco by justin >> Thu Apr 23 22:17:26 2015 >> [0]PETSC ERROR: Configure options --download-chaco --download-exodusii >> --download-fblaslapack --download-hdf5 --download-metis --download-mpich >> --download-mumps --download-netcdf --download-parmetis --download-scalapack >> --download-tetgen --download-triangle --with-cc=gcc --with-clanguage=cxx >> --with-cmake=cmake --with-ctetgen --with-cxx=g++ --with-debugging=1 >> --with-fc=gfortran --with-valgrind=1 PETSC_ARCH=arch-linux2-c-debug >> [0]PETSC ERROR: #1 PetscMallocValidate() line 135 in >> /home/justin/petsc-dev/src/sys/memory/mtr.c >> [0]PETSC ERROR: #2 PetscSignalHandlerDefault() line 149 in >> /home/justin/petsc-dev/src/sys/error/signal.c >> >> >> However, if I run the program with debugging off (i.e., >> --with-debugging=0), VecView gives me an answer that seems relatively >> correct. >> >> Any idea what's going on here? >> >> Thanks, >> >> On Thu, Apr 23, 2015 at 4:00 AM, Matthew Knepley >> wrote: >> >>> On Wed, Apr 22, 2015 at 9:57 PM, Justin Chang wrote: >>> >>>> Hello, >>>> >>>> I am using DMPlexProjectField(...) to postprocess my final solution u >>>> into another vector sol. I have the following pointwise functions: >>>> >>>> /* x-component of gradient */ >>>> void velx(const PetscScalar u[], const PetscScalar u_t[], const >>>> PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const >>>> PetscScalar a_x[], const PetscReal x[], PetscScalar v[]) >>>> { >>>> v[0] = -a[0]*mu*u_x[0]; >>>> } >>>> >>>> /* y-component of gradient */ >>>> void vely(const PetscScalar u[], const PetscScalar u_t[], const >>>> PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const >>>> PetscScalar a_x[], const PetscReal x[], PetscScalar v[]) >>>> { >>>> v[0] = -a[0]*mu*u_x[1]; >>>> } >>>> >>>> /* z-component of gradient */ >>>> void velz(const PetscScalar u[], const PetscScalar u_t[], const >>>> PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const >>>> PetscScalar a_x[], const PetscReal x[], PetscScalar v[]) >>>> { >>>> v[0] = -a[0]*mu*u_x[2]; >>>> } >>>> >>>> Where a[0] is the cell-wise permeability and mu is the (inverse of) >>>> viscosity. I am solving the diffusion equation (P1 elements). At the end of >>>> my simulation I have these lines: >>>> >>>> ierr = VecDuplicate(user.u,&user.sol);CHKERRQ(ierr); >>>> ierr = PetscObjectSetName((PetscObject) user.sol, >>>> "velocity");CHKERRQ(ierr); >>>> void (*vX[1])(const PetscScalar *, const PetscScalar *, const >>>> PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscScalar >>>> *, const PetscReal *, PetscScalar *) = {velx}; >>>> ierr = >>>> DMPlexProjectField(dm,user.u,vX,INSERT_ALL_VALUES,user.sol);CHKERRQ(ierr); >>>> ierr = VecViewFromOptions(user.sol,NULL,"-vec_view_x");CHKERRQ(ierr); >>>> >>>> void (*vY[1])(const PetscScalar *, const PetscScalar *, const >>>> PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscScalar >>>> *, const PetscReal *, PetscScalar *) = {vely}; >>>> ierr = >>>> DMPlexProjectField(dm,user.u,vY,INSERT_ALL_VALUES,user.sol);CHKERRQ(ierr); >>>> ierr = VecViewFromOptions(user.sol,NULL,"-vec_view_y");CHKERRQ(ierr); >>>> >>>> if (spatialDim == 3) { >>>> void (*vZ[1])(const PetscScalar *, const PetscScalar *, const >>>> PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscScalar >>>> *, const PetscReal *, PetscScalar *) = {velz}; >>>> ierr = >>>> DMPlexProjectField(dm,user.u,vZ,INSERT_ALL_VALUES,user.sol);CHKERRQ(ierr); >>>> ierr = VecViewFromOptions(user.sol,NULL,"-vec_view_z");CHKERRQ(ierr); >>>> } >>>> ierr = VecDestroy(&user.sol);CHKERRQ(ierr); >>>> >>>> But when i look at the results, the dirichlet constrained values of u >>>> were projected into the respective locations in sol. I thought putting >>>> INSERT_ALL_VALUES would insert the pointwise functions into all locations >>>> including the constrained ones. How can I project the above pointwise >>>> functions into the constrained fields? Or what am I potentially missing >>>> here? >>>> >>> >>> There are no Dirichlet constrained values in the global vector user.sol. >>> When you call VecView() it gets projected to >>> an enlarged vector including the BC and output. What you want is to >>> project that vector into a space with no constraints, >>> so you really need another DM/Section combination. >>> >>> I think this will work: >>> >>> 1) Get a local vector from dm >>> >>> 2) Use DMPlexProjectFieldLocal(INSERT_ALL_VALUES) to fill it up >>> >>> 3) DMClone(dm) and give it a Section with no constraints >>> >>> 4) Use DMLocalToGlobal() into the new DM >>> >>> 5) VecView() that new global vector >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks, >>>> >>>> >>>> -- >>>> Justin Chang >>>> PhD Candidate, Civil Engineering - Computational Sciences >>>> University of Houston, Department of Civil and Environmental Engineering >>>> Houston, TX 77004 >>>> (512) 963-3262 >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> Justin Chang >> PhD Candidate, Civil Engineering - Computational Sciences >> University of Houston, Department of Civil and Environmental Engineering >> Houston, TX 77004 >> (512) 963-3262 >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Apr 24 17:41:29 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 24 Apr 2015 17:41:29 -0500 Subject: [petsc-users] Turning off TSADAPT still adapts time step In-Reply-To: <553AB0C6.5010503@princeton.edu> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AA4B1.5010603@princeton.edu> <553AB0C6.5010503@princeton.edu> Message-ID: > On Apr 24, 2015, at 4:08 PM, Mark Lohry wrote: > > I'm new to PETSc and not sure how to go about debugging. Running with -start_in_debugger runs in gdb with the normal output (where would i put a breakpoint for this?) You could start by putting a break point in TSAdaptSetType() and see if your call is made and then if any call is made after that by other code changing it to something else. You could put a breakpoint in TSStep_Theta (which supports beuler) and "next" through that to see why it is using an adapter that actually does something. Barry > and running with -info dumps a lot of information but only related to SNES and KSP, nothing about time stepping or adapt. > > > > On 04/24/2015 04:29 PM, Barry Smith wrote: >> Run in the debugger (with one process obviously) to see why it is still calling an adaptor. >> >> Barry >> >>> On Apr 24, 2015, at 3:16 PM, Mark Lohry wrote: >>> >>> Trying a simple stiff test case (y'=y^2*(1-y)) using TSBEULER, to get a feel for implicit stepping with JFNK. >>> >>> I'm giving it a pretty large timestep, and the default appears to start adapting the time step around the stiff part of the solution. So I tried turning it off: >>> >>> TSAdapt adapt; >>> TSGetAdapt(its,&adapt); >>> TSAdaptSetType(adapt,TSADAPTNONE); >>> >>> and even fixing the limits: >>> >>> TSAdaptSetStepLimits(adapt,40.0,40.0); >>> >>> And it still ignores it and adapts anyway. At t=800 (where the solution gets very stiff, so one would normally see adaptive stepping there if it was desired) it's dropping dt from 40 to 10. >>> >>> Printing (t,y) here from the ts monitor: >>> >>> 600.000000,0.002790; >>> 640.000000,0.003198; >>> 680.000000,0.003761; >>> 720.000000,0.004606; >>> 760.000000,0.006072; >>> 800.000000,0.010155; >>> 810.000000,0.011452; >>> 820.000000,0.013161; >>> 830.000000,0.015538; >>> 840.000000,0.019126; >>> 850.000000,0.025426; >>> 860.000000,0.043639; >>> 900.000000,0.975519; >>> 940.000000,0.999402; >>> 980.000000,0.999985; >>> 1020.000000,1.000000; >>> 1060.000000,1.000000; >>> 1100.000000,1.000000; >>> 1140.000000,1.000000; >>> 1180.000000,1.000000; >>> >>> Any idea? Adaptive stepping is great, but I'd still like to be able to fix the time step. >>> >>> >>> > From knepley at gmail.com Fri Apr 24 17:41:48 2015 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 24 Apr 2015 17:41:48 -0500 Subject: [petsc-users] DMPlexProjectField not including BC constraints? In-Reply-To: References: Message-ID: On Fri, Apr 24, 2015 at 5:38 PM, Justin Chang wrote: > I actually made a mistake in PetscFECreateDefault(...), if that's what > you're referring to, and put dm instead of user.dmVel. However the problem > remains. > > Everything is a scalar (i.e., numComp is 1 for both DMs). dm and > user.dmVel should have the same chart but user.dmVel does not invoke > DMPlexAddBoundary(..). Each spatial index of u_x[] will be printed > separately. Right now I am simply working with the x component. > Okay, if everything is scalar, then it should work. Can you make a small example and I will debug it. Thanks, Matt > That said, is it possible to get this sort of mapping? > > Thanks, > > On Fri, Apr 24, 2015 at 8:09 AM, Matthew Knepley > wrote: > >> On Thu, Apr 23, 2015 at 10:20 PM, Justin Chang wrote: >> >>> Matt, >>> >>> I have attempted this but I am getting an error when trying to output >>> the vector into HDF5. This is my implementation: >>> >>> /* Initialize */ >>> PetscDS prob; >>> Vec vlocal, ulocal, vglobal; >>> void (*vX[1])(....) = {velx}; >>> >>> /* Create DM for velocity */ >>> ierr = DMClone(dm,&user.dmVel);CHKERRQ(ierr); >>> ierr = DMPlexCopyCoordinates(dm, user.dmVel);CHKERRQ(ierr); >>> ierr = PetscFECreateDefault(dm, spatialDim, 1, PETSC_TRUE, "solution_", >>> -1, &user.feVel);CHKERRQ(ierr); >>> ierr = DMGetDS(user.dmVel, &prob);CHKERRQ(ierr); >>> ierr = PetscDSSetDiscretization(prob, 0, (PetscObject) >>> user.feVel);CHKERRQ(ierr); >>> ierr = DMCreateGlobalVector(user.dmVel,&vglobal);CHKERRQ(ierr); >>> ierr = PetscObjectSetName((PetscObject) vglobal, >>> "velocity");CHKERRQ(ierr); >>> >>> /* Obtain velocity */ >>> ierr = DMGetLocalVector(dm,&ulocal);CHKERRQ(ierr); >>> ierr = VecDuplicate(ulocal,&vlocal);CHKERRQ(ierr); >>> ierr = >>> DMGlobalToLocalBegin(dm,user.u,INSERT_ALL_VALUES,ulocal);CHKERRQ(ierr); >>> ierr = >>> DMGlobalToLocalEnd(dm,user.u,INSERT_ALL_VALUES,ulocal);CHKERRQ(ierr); >>> ierr = >>> DMPlexProjectFieldLocal(dm,ulocal,vX,INSERT_ALL_VALUES,vlocal);CHKERRQ(ierr); >>> ierr = >>> DMLocalToGlobalBegin(dm,vlocal,INSERT_ALL_VALUES,vglobal);CHKERRQ(ierr); >>> ierr = >>> DMLocalToGlobalEnd(dm,vlocal,INSERT_ALL_VALUES,vglobal);CHKERRQ(ierr); >>> >> >> 1) Shouldn't that be user.dmVel above, not dm? >> >> 2) Are you changing from vector to scalar here? I think so, and so you >> would need an extra step >> since that projection assumes that the layout is the same as the >> input. >> >> I do not currently have anything that maps between different layouts like >> you want. It will have to >> be written. THEN we can use the above code. >> >> Matt >> >> >>> ierr = VecViewFromOptions(vglobal,NULL,"-vec_view_x");CHKERRQ(ierr); >>> >>> And as soon as I get to the last line, it gives me this error: >>> >>> >>> [0]PETSC ERROR: PetscMallocValidate: error detected at >>> PetscViewerFileSetName_HDF5() line 251 in >>> /home/justin/petsc-dev/src/sys/classes/viewer/impls/hdf5/hdf5v.c >>> [0]PETSC ERROR: Memory at address 0x3077ac0 is corrupted >>> [0]PETSC ERROR: Probably write past beginning or end of array >>> [0]PETSC ERROR: Last intact block allocated in VecCreate_Seq() line 38 >>> in /home/justin/petsc-dev/src/vec/vec/impls/seq/bvec3.c >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Memory corruption: >>> http://www.mcs.anl.gov/petsc/documentation/installation.html#valgrind >>> [0]PETSC ERROR: >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2766-g0d12e5d >>> GIT Date: 2015-04-21 21:32:23 -0500 >>> [0]PETSC ERROR: ./main on a arch-linux2-c-debug named pacotaco by justin >>> Thu Apr 23 22:14:21 2015 >>> [0]PETSC ERROR: Configure options --download-chaco --download-exodusii >>> --download-fblaslapack --download-hdf5 --download-metis --download-mpich >>> --download-mumps --download-netcdf --download-parmetis --download-scalapack >>> --download-tetgen --download-triangle --with-cc=gcc --with-clanguage=cxx >>> --with-cmake=cmake --with-ctetgen --with-cxx=g++ --with-debugging=1 >>> --with-fc=gfortran --with-valgrind=1 PETSC_ARCH=arch-linux2-c-debug >>> [0]PETSC ERROR: #1 PetscMallocValidate() line 135 in >>> /home/justin/petsc-dev/src/sys/memory/mtr.c >>> [0]PETSC ERROR: #2 PetscViewerFileSetName_HDF5() line 251 in >>> /home/justin/petsc-dev/src/sys/classes/viewer/impls/hdf5/hdf5v.c >>> [0]PETSC ERROR: #3 PetscViewerFileSetName() line 623 in >>> /home/justin/petsc-dev/src/sys/classes/viewer/impls/ascii/filev.c >>> [0]PETSC ERROR: #4 PetscOptionsGetViewer() line 130 in >>> /home/justin/petsc-dev/src/sys/classes/viewer/interface/viewreg.c >>> [0]PETSC ERROR: #5 PetscObjectViewFromOptions() line 2622 in >>> /home/justin/petsc-dev/src/sys/objects/options.c >>> [0]PETSC ERROR: #6 main() line 1237 in >>> /home/justin/Dropbox/dmplex-nonneg/main.c >>> [0]PETSC ERROR: PETSc Option Table entries: >>> [0]PETSC ERROR: -al 100 >>> [0]PETSC ERROR: -at 0.01 >>> [0]PETSC ERROR: -bcloc >>> 496000,496000,536000,542000,0,100,503000,503000,536000,542000,0,100,496000,503000,536000,536000,0,100,496000,503000,542000,542000,0,100 >>> [0]PETSC ERROR: -bcnum 4 >>> [0]PETSC ERROR: -bcval 1,1,1,1 >>> [0]PETSC ERROR: -dim 3 >>> [0]PETSC ERROR: -dm_refine 0 >>> [0]PETSC ERROR: -dt 0.01 >>> [0]PETSC ERROR: -edges 3,3,3 >>> [0]PETSC ERROR: -floc 499550,499600,538950,539000,0,100 >>> [0]PETSC ERROR: -flow datafiles/chrom_flow.dat >>> [0]PETSC ERROR: -fnum 1 >>> [0]PETSC ERROR: -ftime 0,99 >>> [0]PETSC ERROR: -fval -0.1 >>> [0]PETSC ERROR: -ksp_rtol 1.0e-9 >>> [0]PETSC ERROR: -ksp_type cg >>> [0]PETSC ERROR: -lower 0,0,0 >>> [0]PETSC ERROR: -mat_petscspace_order 0 >>> [0]PETSC ERROR: -mesh datafiles/chrom_mesh.dat >>> [0]PETSC ERROR: -mu 3.95e-5 >>> [0]PETSC ERROR: -nonneg 0 >>> [0]PETSC ERROR: -numsteps 0 >>> [0]PETSC ERROR: -options_left 0 >>> [0]PETSC ERROR: -pc_type jacobi >>> [0]PETSC ERROR: -progress 1 >>> [0]PETSC ERROR: -solution_petscspace_order 1 >>> [0]PETSC ERROR: -tao_type tron >>> [0]PETSC ERROR: -upper 1,1,1 >>> [0]PETSC ERROR: -vec_view_x hdf5:solx.h5 >>> [0]PETSC ERROR: -vec_view_y hdf5:soly.h5 >>> [0]PETSC ERROR: -vec_view_z hdf5:solz.h5 >>> [0]PETSC ERROR: -vtuname pressure >>> [0]PETSC ERROR: -vtuprintat 1,20,80,160,320,640,999 >>> [0]PETSC ERROR: -vtusteps 7 >>> [0]PETSC ERROR: ----------------End of Error Message -------send entire >>> error message to petsc-maint at mcs.anl.gov---------- >>> >>> Any idea what's going on here? Also, if I replace the last line with >>> VecView(vglobal,PETSC_VIEWER_STDOUT_WORLD) I get a completely wacko error: >>> >>> Vec Object:@?? ??r???????b? ? j?r??. >>> >>> ~E?b??\2p}^r????"?.b?'xDs??q??|n??a?`K? Lwq??n??a??????p??g?u`?!? >>> >>> |? >>> >>> p?;? :2_???? ?5n??6/w $]???I!}?k?! c /?Z??s# li??=Y?? X?I% >>> ?[?f?^?u=??T??Sx??Yc??F ?u?Q?6 >>> e??_?;6??a3L? z_?R X??? >>> ?l?D?X?(?(?P?= ?/k?9?;z? I B? >>> >>> R??p$?? j?-? ???7??- ??H>?h?:?? ??6 5?{??2@?L??]P?I#B?>> e???? ?V? ?Y?g? ?3?2WY??X?j?)?&M?[?s?!Q&+m??*?l?V^????uo?? ?D`?z 1 MPI >>> processes >>> type: seq >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >>> probably memory access out of range >>> [0]PETSC ERROR: Try option -start_in_debugger or >>> -on_error_attach_debugger >>> [0]PETSC ERROR: or see >>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >>> OS X to find memory corruption errors >>> [0]PETSC ERROR: PetscMallocValidate: error detected at >>> PetscSignalHandlerDefault() line 149 in >>> /home/justin/petsc-dev/src/sys/error/signal.c >>> [0]PETSC ERROR: Memory at address 0x2575ac0 is corrupted >>> [0]PETSC ERROR: Probably write past beginning or end of array >>> [0]PETSC ERROR: Last intact block allocated in VecCreate_Seq() line 38 >>> in /home/justin/petsc-dev/src/vec/vec/impls/seq/bvec3.c >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Memory corruption: >>> http://www.mcs.anl.gov/petsc/documentation/installation.html#valgrind >>> [0]PETSC ERROR: >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2766-g0d12e5d >>> GIT Date: 2015-04-21 21:32:23 -0500 >>> [0]PETSC ERROR: ./main on a arch-linux2-c-debug named pacotaco by justin >>> Thu Apr 23 22:17:26 2015 >>> [0]PETSC ERROR: Configure options --download-chaco --download-exodusii >>> --download-fblaslapack --download-hdf5 --download-metis --download-mpich >>> --download-mumps --download-netcdf --download-parmetis --download-scalapack >>> --download-tetgen --download-triangle --with-cc=gcc --with-clanguage=cxx >>> --with-cmake=cmake --with-ctetgen --with-cxx=g++ --with-debugging=1 >>> --with-fc=gfortran --with-valgrind=1 PETSC_ARCH=arch-linux2-c-debug >>> [0]PETSC ERROR: #1 PetscMallocValidate() line 135 in >>> /home/justin/petsc-dev/src/sys/memory/mtr.c >>> [0]PETSC ERROR: #2 PetscSignalHandlerDefault() line 149 in >>> /home/justin/petsc-dev/src/sys/error/signal.c >>> >>> >>> However, if I run the program with debugging off (i.e., >>> --with-debugging=0), VecView gives me an answer that seems relatively >>> correct. >>> >>> Any idea what's going on here? >>> >>> Thanks, >>> >>> On Thu, Apr 23, 2015 at 4:00 AM, Matthew Knepley >>> wrote: >>> >>>> On Wed, Apr 22, 2015 at 9:57 PM, Justin Chang wrote: >>>> >>>>> Hello, >>>>> >>>>> I am using DMPlexProjectField(...) to postprocess my final solution u >>>>> into another vector sol. I have the following pointwise functions: >>>>> >>>>> /* x-component of gradient */ >>>>> void velx(const PetscScalar u[], const PetscScalar u_t[], const >>>>> PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const >>>>> PetscScalar a_x[], const PetscReal x[], PetscScalar v[]) >>>>> { >>>>> v[0] = -a[0]*mu*u_x[0]; >>>>> } >>>>> >>>>> /* y-component of gradient */ >>>>> void vely(const PetscScalar u[], const PetscScalar u_t[], const >>>>> PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const >>>>> PetscScalar a_x[], const PetscReal x[], PetscScalar v[]) >>>>> { >>>>> v[0] = -a[0]*mu*u_x[1]; >>>>> } >>>>> >>>>> /* z-component of gradient */ >>>>> void velz(const PetscScalar u[], const PetscScalar u_t[], const >>>>> PetscScalar u_x[], const PetscScalar a[], const PetscScalar a_t[], const >>>>> PetscScalar a_x[], const PetscReal x[], PetscScalar v[]) >>>>> { >>>>> v[0] = -a[0]*mu*u_x[2]; >>>>> } >>>>> >>>>> Where a[0] is the cell-wise permeability and mu is the (inverse of) >>>>> viscosity. I am solving the diffusion equation (P1 elements). At the end of >>>>> my simulation I have these lines: >>>>> >>>>> ierr = VecDuplicate(user.u,&user.sol);CHKERRQ(ierr); >>>>> ierr = PetscObjectSetName((PetscObject) user.sol, >>>>> "velocity");CHKERRQ(ierr); >>>>> void (*vX[1])(const PetscScalar *, const PetscScalar *, const >>>>> PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscScalar >>>>> *, const PetscReal *, PetscScalar *) = {velx}; >>>>> ierr = >>>>> DMPlexProjectField(dm,user.u,vX,INSERT_ALL_VALUES,user.sol);CHKERRQ(ierr); >>>>> ierr = VecViewFromOptions(user.sol,NULL,"-vec_view_x");CHKERRQ(ierr); >>>>> >>>>> void (*vY[1])(const PetscScalar *, const PetscScalar *, const >>>>> PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscScalar >>>>> *, const PetscReal *, PetscScalar *) = {vely}; >>>>> ierr = >>>>> DMPlexProjectField(dm,user.u,vY,INSERT_ALL_VALUES,user.sol);CHKERRQ(ierr); >>>>> ierr = VecViewFromOptions(user.sol,NULL,"-vec_view_y");CHKERRQ(ierr); >>>>> >>>>> if (spatialDim == 3) { >>>>> void (*vZ[1])(const PetscScalar *, const PetscScalar *, const >>>>> PetscScalar *, const PetscScalar *, const PetscScalar *, const PetscScalar >>>>> *, const PetscReal *, PetscScalar *) = {velz}; >>>>> ierr = >>>>> DMPlexProjectField(dm,user.u,vZ,INSERT_ALL_VALUES,user.sol);CHKERRQ(ierr); >>>>> ierr = VecViewFromOptions(user.sol,NULL,"-vec_view_z");CHKERRQ(ierr); >>>>> } >>>>> ierr = VecDestroy(&user.sol);CHKERRQ(ierr); >>>>> >>>>> But when i look at the results, the dirichlet constrained values of u >>>>> were projected into the respective locations in sol. I thought putting >>>>> INSERT_ALL_VALUES would insert the pointwise functions into all locations >>>>> including the constrained ones. How can I project the above pointwise >>>>> functions into the constrained fields? Or what am I potentially missing >>>>> here? >>>>> >>>> >>>> There are no Dirichlet constrained values in the global vector >>>> user.sol. When you call VecView() it gets projected to >>>> an enlarged vector including the BC and output. What you want is to >>>> project that vector into a space with no constraints, >>>> so you really need another DM/Section combination. >>>> >>>> I think this will work: >>>> >>>> 1) Get a local vector from dm >>>> >>>> 2) Use DMPlexProjectFieldLocal(INSERT_ALL_VALUES) to fill it up >>>> >>>> 3) DMClone(dm) and give it a Section with no constraints >>>> >>>> 4) Use DMLocalToGlobal() into the new DM >>>> >>>> 5) VecView() that new global vector >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thanks, >>>>> >>>>> >>>>> -- >>>>> Justin Chang >>>>> PhD Candidate, Civil Engineering - Computational Sciences >>>>> University of Houston, Department of Civil and Environmental >>>>> Engineering >>>>> Houston, TX 77004 >>>>> (512) 963-3262 >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >>> >>> -- >>> Justin Chang >>> PhD Candidate, Civil Engineering - Computational Sciences >>> University of Houston, Department of Civil and Environmental Engineering >>> Houston, TX 77004 >>> (512) 963-3262 >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Sat Apr 25 01:05:13 2015 From: danyang.su at gmail.com (Danyang Su) Date: Fri, 24 Apr 2015 23:05:13 -0700 Subject: [petsc-users] Floating point exception In-Reply-To: <553AAD7A.1030807@gmail.com> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> Message-ID: <553B2E99.6010601@gmail.com> Hi Barry and Satish, How can I get rid of unknown floating point error when a very small value is multiplied. e.g., cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, cinfrt = cinfrt_dg(i1) * diff(ic,idim) I get the following error when run with "-fp_trap -start_in_debugger". Backtrace for this error: *** unknown floating point error occurred *** [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 where the result is a bitwise OR of the following flags: [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 [2]PETSC ERROR: Try option -start_in_debugger [2]PETSC ERROR: likely location of problem given in stack below Thanks, Danyang On 15-04-24 01:54 PM, Danyang Su wrote: > On 15-04-24 01:23 PM, Satish Balay wrote: >> c 4 1.0976214263087059E-067 >> >> I don't think this number can be stored in a real*4. >> >> Satish > Thanks, Satish. It is caused by this number. >> >> On Fri, 24 Apr 2015, Danyang Su wrote: >> >>> >>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su wrote: >>>>> >>>>> Hi All, >>>>> >>>>> One of my case crashes because of floating point exception when >>>>> using 4 >>>>> processors, as shown below. But if I run this case with 1 >>>>> processor, it >>>>> works fine. I have tested the codes with around 100 cases up to 768 >>>>> processors, all other cases work fine. I just wonder if this kind >>>>> of error >>>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>> Yes, almost for sure it is one of these places. >>>> >>>> First run the bad case with -fp_trap if all goes well you'll >>>> see the >>>> function where the FPE is generated. Then run also with >>>> -start_in_debugger >>>> and >>>> type cont in all four debugger windows. When the FPE happens the >>>> debugger >>>> should stop showing exactly where the FPE happens. >>>> >>>> Barry >>> Hi Barry, >>> >>> If run with -fp_trap -start_in_debugger, I got the following error >>> >>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>> [0]PETSC ERROR: The specific exception can be determined by running >>> in a >>> debugger. When the >>> [0]PETSC ERROR: debugger traps the signal, the exception can be >>> found with >>> fetestexcept(0x3d) >>> [0]PETSC ERROR: where the result is a bitwise OR of the following >>> flags: >>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>> [0]PETSC ERROR: Try option -start_in_debugger >>> [0]PETSC ERROR: likely location of problem given in stack below >>> [0]PETSC ERROR: --------------------- Stack Frames >>> ------------------------------------ >>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>> available, >>> [0]PETSC ERROR: INSTEAD the line number of the start of the >>> function >>> [0]PETSC ERROR: is given. >>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>> floating point error >>> >>> Program received signal SIGABRT: Process abort signal. >>> >>> Backtrace for this error: >>> #0 0x7F4FEAB1C7D7 >>> #1 0x7F4FEAB1CDDE >>> #2 0x7F4FE9E1AD3F >>> #3 0x7F4FE9E1ACC9 >>> #4 0x7F4FE9E1E0D7 >>> #5 0x7F4FEB0B6DCB >>> #6 0x7F4FEB0B1825 >>> #7 0x7F4FEB0B817F >>> #8 0x7F4FE9E1AD3F >>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>> >>> I checked the code at tprfrtlc.F90:2393, >>> >>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>> (cx(ix),ix=1,nxout)/) >>> >>> All the values (time, c, cx) are reasonable, as shown below. The only >>> possibility is that realbuffer_gb is in declared as real*4 if using >>> sing >>> precision output while time, c, cx are declared in real*8. I have a >>> lot of >>> similar data conversion from real*8 to real*4 output, other code >>> does not >>> return error. >>> >>> time 0.0000000000000000 >>> c 1 9.9999999999999995E-008 >>> c 2 3.1555251077549618E-003 >>> c 3 7.1657814842179362E-008 >>> c 4 1.0976214263087059E-067 >>> c 5 5.2879822292305797E-004 >>> c 6 9.9999999999999964E-005 >>> c 7 6.4055731968811337E-005 >>> c 8 3.4607572892578404E-020 >>> cx 1 3.4376650636008101E-005 >>> cx 2 7.3989678854017763E-012 >>> cx 3 9.5317170613607207E-012 >>> cx 4 2.2344525794718353E-015 >>> cx 5 3.0624685689695889E-008 >>> cx 6 1.0046157902783967E-007 >>> cx 7 1.5320169154914984E-004 >>> cx 8 8.6930292776346176E-014 >>> cx 9 3.5944267559348721E-005 >>> cx 10 3.0072645866951157E-018 >>> cx 11 2.3592486321095017E-013 >>> >>> Thanks, >>> >>> Danyang >>> >>>>> I can check all the entries of jacobi matrix to see if the value >>>>> is valid, >>>>> but this seems not a good idea as it takes a long time to reach this >>>>> point. If I restart the simulation from a specified time (e.g., >>>>> 7.685 in >>>>> this case), then the error does not occur. >>>>> >>>>> Would you please give me any suggestion on debugging this case? >>>>> >>>>> Thanks and Regards, >>>>> >>>>> Danyang >>>>> >>>>> >>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years >>>>> iter: 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years >>>>> iter: 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years >>>>> iter: 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years >>>>> iter: 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years >>>>> iter: 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> Reduce time step for reactive transport >>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years >>>>> iter: 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> Reduce time step for reactive transport >>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years >>>>> iter: 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> [1]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [1]PETSC ERROR: Floating point exception >>>>> [2]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [2]PETSC ERROR: Floating point exception >>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or >>>>> infinite >>>>> at end of function: Parameter number 3 >>>>> [2]PETSC ERROR: See >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> for trouble shooting. >>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>> [1]PETSC ERROR: See >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> for trouble shooting. >>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named >>>>> nwmop by >>>>> dsu Thu Apr 23 15:38:52 2015 >>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg >>>>> --with-cc=gcc >>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack >>>>> --download-mpich >>>>> --download-mumps --download-hypre --download-superlu_dist >>>>> --download-metis >>>>> --download-parmetis --download-scalapack >>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu >>>>> Apr 23 >>>>> 15:38:52 2015 >>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg >>>>> --with-cc=gcc >>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack >>>>> --download-mpich >>>>> --download-mumps --download-hypre --download-superlu_dist >>>>> --download-metis >>>>> --download-parmetis --download-scalapack >>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>> #3 KSP_PCApply() line 230 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>> >>> > From hus003 at ucsd.edu Sat Apr 25 01:30:10 2015 From: hus003 at ucsd.edu (Sun, Hui) Date: Sat, 25 Apr 2015 06:30:10 +0000 Subject: [petsc-users] Scattering for DM objects Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB011054F6@XMAIL-MBX-AH1.AD.UCSD.EDU> I have a DM MPI Vec u, I want to scatter it to a sequential Vec w in processor 0. There is a function VecScatterCreateToZero allowing me to do that. However, the global indices get easily messed up if I do this way. So I have to match the IS for u and w. And I think maybe I can use DMCompositeGetGlobalISs to get the IS, and then call VecScatterCreate(u,is[0],w,is[0],&ctx). But I'm not sure if I'm doing the right thing, and for now it gives me segmentation fault, memory errors on DMCompositeGetGlobalISs. So I need some help on this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Sat Apr 25 01:42:53 2015 From: dave.mayhem23 at gmail.com (Dave May) Date: Sat, 25 Apr 2015 08:42:53 +0200 Subject: [petsc-users] Scattering for DM objects In-Reply-To: <7501CC2B7BBCC44A92ECEEC316170ECB011054F6@XMAIL-MBX-AH1.AD.UCSD.EDU> References: <7501CC2B7BBCC44A92ECEEC316170ECB011054F6@XMAIL-MBX-AH1.AD.UCSD.EDU> Message-ID: If your DM is of type DMDA, you can use DMDACreateNaturalVector() DMDAGlobalToNaturalBegin() DMDAGlobalToNaturalEnd() to get the vector in the natural ordering, then you can use VecScatterCreateToZero() with your natural vector. This will give you the correctly ordered vector on rank 0. Cheers Dave On 25 April 2015 at 08:30, Sun, Hui wrote: > I have a DM MPI Vec u, I want to scatter it to a sequential Vec w in > processor 0. There is a function VecScatterCreateToZero allowing me to do > that. However, the global indices get easily messed up if I do this way. > > So I have to match the IS for u and w. And I think maybe I can use > DMCompositeGetGlobalISs to get the IS, and then call > VecScatterCreate(u,is[0],w,is[0],&ctx). > > But I'm not sure if I'm doing the right thing, and for now it gives me > segmentation fault, memory errors on DMCompositeGetGlobalISs. > > So I need some help on this. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hus003 at ucsd.edu Sat Apr 25 02:31:34 2015 From: hus003 at ucsd.edu (Sun, Hui) Date: Sat, 25 Apr 2015 07:31:34 +0000 Subject: [petsc-users] Scattering for DM objects In-Reply-To: References: <7501CC2B7BBCC44A92ECEEC316170ECB011054F6@XMAIL-MBX-AH1.AD.UCSD.EDU>, Message-ID: <7501CC2B7BBCC44A92ECEEC316170ECB01105508@XMAIL-MBX-AH1.AD.UCSD.EDU> Thank you Dave! It works out nicely. Hui ________________________________ From: Dave May [dave.mayhem23 at gmail.com] Sent: Friday, April 24, 2015 11:42 PM To: Sun, Hui Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Scattering for DM objects If your DM is of type DMDA, you can use DMDACreateNaturalVector() DMDAGlobalToNaturalBegin() DMDAGlobalToNaturalEnd() to get the vector in the natural ordering, then you can use VecScatterCreateToZero() with your natural vector. This will give you the correctly ordered vector on rank 0. Cheers Dave On 25 April 2015 at 08:30, Sun, Hui > wrote: I have a DM MPI Vec u, I want to scatter it to a sequential Vec w in processor 0. There is a function VecScatterCreateToZero allowing me to do that. However, the global indices get easily messed up if I do this way. So I have to match the IS for u and w. And I think maybe I can use DMCompositeGetGlobalISs to get the IS, and then call VecScatterCreate(u,is[0],w,is[0],&ctx). But I'm not sure if I'm doing the right thing, and for now it gives me segmentation fault, memory errors on DMCompositeGetGlobalISs. So I need some help on this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Sat Apr 25 13:13:05 2015 From: danyang.su at gmail.com (Danyang Su) Date: Sat, 25 Apr 2015 11:13:05 -0700 Subject: [petsc-users] Floating point exception In-Reply-To: <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> Message-ID: <553BD931.9050202@gmail.com> Hi Barry, With -fp_trap and -start_in_debugger options, the code crashed with the following error. The code at #21 0x41C49A in __solver_dd_MOD_solver_dd_snes_solve_react at solver_ddmethod.F90:2850 is "call KSPSolve(ksp_react,b_react,x_react,ierr)" I run this case with 4 processors and the preconditioner type is HYPRE. Does this mean something wrong in Matrix ksp_react or RHS b_react? Thanks, Danyang timestep: 1846 time: 3.392E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 Reduce time step for reactive transport timestep: 1846 time: 3.387E+00 years delt: 5.000E-03 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 Reduce time step for reactive transport timestep: 1846 time: 3.385E+00 years delt: 2.500E-03 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 [0]PETSC ERROR: *** unknown floating point error occurred *** [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) [0]PETSC ERROR: where the result is a bitwise OR of the following flags: [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 [0]PETSC ERROR: Try option -start_in_debugger [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [1]PETSC ERROR: *** unknown floating point error occurred *** [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) [1]PETSC ERROR: where the result is a bitwise OR of the following flags: [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 [1]PETSC ERROR: Try option -start_in_debugger [1]PETSC ERROR: likely location of problem given in stack below [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [1]PETSC ERROR: INSTEAD the line number of the start of the function [1]PETSC ERROR: is given. [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c [1]PETSC ERROR: [1] Hypre solve line 174 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/impls/hypre/hypre.c [1]PETSC ERROR: [1] PCApply_HYPRE line 161 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/impls/hypre/hypre.c [1]PETSC ERROR: [2]PETSC ERROR: *** unknown floating point error occurred *** [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) [2]PETSC ERROR: where the result is a bitwise OR of the following flags: [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 [2]PETSC ERROR: Try option -start_in_debugger [2]PETSC ERROR: likely location of problem given in stack below [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [2]PETSC ERROR: INSTEAD the line number of the start of the function [2]PETSC ERROR: is given. [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c [2]PETSC ERROR: [2] Hypre solve line 174 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/impls/hypre/hypre.c [2]PETSC ERROR: [2] PCApply_HYPRE line 161 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/impls/hypre/hypre.c [2]PETSC ERROR: [2] KSP_PCApply line 228 /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h [2]PETSC ERROR: [2] KSPInitialResidual line 44 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c [0]PETSC ERROR: [0] Hypre solve line 174 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/impls/hypre/hypre.c [0]PETSC ERROR: [0] PCApply_HYPRE line 161 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/impls/hypre/hypre.c [0]PETSC ERROR: [0] KSP_PCApply line 228 /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h [0]PETSC ERROR: [0] KSPInitialResidual line 44 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c [0]PETSC ERROR: [0] KSPSolve_GMRES line 224 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c [1] KSP_PCApply line 228 /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h [1]PETSC ERROR: [1] KSPInitialResidual line 44 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c [1]PETSC ERROR: [1] KSPSolve_GMRES line 224 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c [2]PETSC ERROR: [2] KSPSolve_GMRES line 224 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c [2]PETSC ERROR: [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error User provided function() line 0 in Unknown file trapped floating point error [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error Program received signal SIGABRT: Process abort signal. Backtrace for this error: Program received signal SIGABRT: Process abort signal. Backtrace for this error: Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x7FDC76F307D7 #0 0x7FA04C1207D7 #1 0x7FA04C120DDE #1 0x7FDC76F30DDE #2 0x7FA04B41ED3F #3 0x7FA04B41ECC9 #2 0x7FDC7622ED3F #4 0x7FA04B4220D7 #0 0x7F622A92F7D7 #3 0x7FDC7622ECC9 #5 0x7FA04C6BADCB #1 0x7F622A92FDDE #4 0x7FDC762320D7 #6 0x7FA04C6B5825 #2 0x7F6229C2DD3F #7 0x7FA04C6BC17F #5 0x7FDC774CADCB #8 0x7FA04B41ED3F #3 0x7F6229C2DCC9 #6 0x7FDC774C5825 #4 0x7F6229C310D7 #9 0x7FA04D9EF449 #7 0x7FDC774CC17F #10 0x7FA04D9EF055 #5 0x7F622AEC9DCB #8 0x7FDC7622ED3F #11 0x7FA04D99D2DD #6 0x7F622AEC4825 #9 0x7FDC787FF449 #12 0x7FA04D984ACD #7 0x7F622AECB17F #10 0x7FDC787FF055 #13 0x7FA04D973E63 #8 0x7F6229C2DD3F #11 0x7FDC787AD2DD #14 0x7FA04D27E8E3 #9 0x7F622C1FE449 #12 0x7FDC78794ACD #15 0x7FA04D2BEB04 #10 0x7F622C1FE055 #13 0x7FDC78783E63 #16 0x7FA04D3CABFA #11 0x7F622C1AC2DD #17 0x7FA04D3CB927 #14 0x7FDC7808E8E3 #12 0x7F622C193ACD #18 0x7FA04D361DE8 #15 0x7FDC780CEB04 #13 0x7F622C182E63 #16 0x7FDC781DABFA #19 0x7FA04D3A0E1D #20 0x7FA04D3DC121 #14 0x7F622BA8D8E3 #15 0x7F622BACDB04 #17 0x7FDC781DB927 #18 0x7FDC78171DE8 #16 0x7F622BBD9BFA #19 0x7FDC781B0E1D #17 0x7F622BBDA927 #20 0x7FDC781EC121 #18 0x7F622BB70DE8 #19 0x7F622BBAFE1D #20 0x7F622BBEB121 #21 0x41C49A in __solver_dd_MOD_solver_dd_snes_solve_react at solver_ddmethod.F90:2850 #21 0x41C49A in __solver_dd_MOD_solver_dd_snes_solve_react at solver_ddmethod.F90:2850 #21 0x41C49A in __solver_dd_MOD_solver_dd_snes_solve_react at solver_ddmethod.F90:2850 #22 0x6A25A5 in reactran_ at reactran.F90:954 #22 0x6A25A5 in reactran_ at reactran.F90:954 #22 0x6A25A5 in reactran_ at reactran.F90:954 #23 0x574836 in timeloop_ at timeloop.F90:1194 #23 0x574836 in timeloop_ at timeloop.F90:1194 #23 0x574836 in timeloop_ at timeloop.F90:1194 #24 0x5ABFD7 in driver_pc at driver_pc.F90:599 #24 0x5ABFD7 in driver_pc at driver_pc.F90:599 #24 0x5ABFD7 in driver_pc at driver_pc.F90:599 On 15-04-24 11:12 AM, Barry Smith wrote: >> On Apr 24, 2015, at 1:05 PM, Danyang Su wrote: >> >> Hi All, >> >> One of my case crashes because of floating point exception when using 4 processors, as shown below. But if I run this case with 1 processor, it works fine. I have tested the codes with around 100 cases up to 768 processors, all other cases work fine. I just wonder if this kind of error is caused because of NaN in jacobi matrix, RHS or preconditioner? > Yes, almost for sure it is one of these places. > > First run the bad case with -fp_trap if all goes well you'll see the function where the FPE is generated. Then run also with -start_in_debugger and > type cont in all four debugger windows. When the FPE happens the debugger should stop showing exactly where the FPE happens. > > Barry > >> I can check all the entries of jacobi matrix to see if the value is valid, but this seems not a good idea as it takes a long time to reach this point. If I restart the simulation from a specified time (e.g., 7.685 in this case), then the error does not occur. >> >> Would you please give me any suggestion on debugging this case? >> >> Thanks and Regards, >> >> Danyang >> >> >> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >> Reduce time step for reactive transport >> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >> Reduce time step for reactive transport >> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >> [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [1]PETSC ERROR: Floating point exception >> [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [2]PETSC ERROR: Floating point exception >> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at end of function: Parameter number 3 >> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at end of function: Parameter number 3 >> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 15:38:52 2015 >> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack >> [1]PETSC ERROR: #1 VecValidValues() line 34 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 15:38:52 2015 >> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack >> [2]PETSC ERROR: #1 VecValidValues() line 34 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >> [1]PETSC ERROR: #2 PCApply() line 442 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >> #3 KSP_PCApply() line 230 in /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >> [2]PETSC ERROR: #6 KSPSolve() line 459 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >> [1]PETSC ERROR: #6 KSPSolve() line 459 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >> [mpiexec at nwmop] Press Ctrl-C again to force abort From bsmith at mcs.anl.gov Sat Apr 25 13:32:07 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 25 Apr 2015 13:32:07 -0500 Subject: [petsc-users] Floating point exception In-Reply-To: <553B2E99.6010601@gmail.com> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> Message-ID: <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> I told you this yesterday. It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate #elif defined PETSC_HAVE_XMMINTRIN_H _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); #else change it to #elif defined PETSC_HAVE_XMMINTRIN_H _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); #else Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. Did you do this? Barry > On Apr 25, 2015, at 1:05 AM, Danyang Su wrote: > > Hi Barry and Satish, > > How can I get rid of unknown floating point error when a very small value is multiplied. > > e.g., > cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, > > cinfrt = cinfrt_dg(i1) * diff(ic,idim) > > I get the following error when run with "-fp_trap -start_in_debugger". > > Backtrace for this error: > *** unknown floating point error occurred *** > [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the > [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) > [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 > where the result is a bitwise OR of the following flags: > [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 > [2]PETSC ERROR: Try option -start_in_debugger > [2]PETSC ERROR: likely location of problem given in stack below > > Thanks, > > Danyang > > On 15-04-24 01:54 PM, Danyang Su wrote: >> On 15-04-24 01:23 PM, Satish Balay wrote: >>> c 4 1.0976214263087059E-067 >>> >>> I don't think this number can be stored in a real*4. >>> >>> Satish >> Thanks, Satish. It is caused by this number. >>> >>> On Fri, 24 Apr 2015, Danyang Su wrote: >>> >>>> >>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su wrote: >>>>>> >>>>>> Hi All, >>>>>> >>>>>> One of my case crashes because of floating point exception when using 4 >>>>>> processors, as shown below. But if I run this case with 1 processor, it >>>>>> works fine. I have tested the codes with around 100 cases up to 768 >>>>>> processors, all other cases work fine. I just wonder if this kind of error >>>>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>>> Yes, almost for sure it is one of these places. >>>>> >>>>> First run the bad case with -fp_trap if all goes well you'll see the >>>>> function where the FPE is generated. Then run also with -start_in_debugger >>>>> and >>>>> type cont in all four debugger windows. When the FPE happens the debugger >>>>> should stop showing exactly where the FPE happens. >>>>> >>>>> Barry >>>> Hi Barry, >>>> >>>> If run with -fp_trap -start_in_debugger, I got the following error >>>> >>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>> [0]PETSC ERROR: The specific exception can be determined by running in a >>>> debugger. When the >>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >>>> fetestexcept(0x3d) >>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>> [0]PETSC ERROR: Try option -start_in_debugger >>>> [0]PETSC ERROR: likely location of problem given in stack below >>>> [0]PETSC ERROR: --------------------- Stack Frames >>>> ------------------------------------ >>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>> [0]PETSC ERROR: is given. >>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>>> floating point error >>>> >>>> Program received signal SIGABRT: Process abort signal. >>>> >>>> Backtrace for this error: >>>> #0 0x7F4FEAB1C7D7 >>>> #1 0x7F4FEAB1CDDE >>>> #2 0x7F4FE9E1AD3F >>>> #3 0x7F4FE9E1ACC9 >>>> #4 0x7F4FE9E1E0D7 >>>> #5 0x7F4FEB0B6DCB >>>> #6 0x7F4FEB0B1825 >>>> #7 0x7F4FEB0B817F >>>> #8 0x7F4FE9E1AD3F >>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>> >>>> I checked the code at tprfrtlc.F90:2393, >>>> >>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>> (cx(ix),ix=1,nxout)/) >>>> >>>> All the values (time, c, cx) are reasonable, as shown below. The only >>>> possibility is that realbuffer_gb is in declared as real*4 if using sing >>>> precision output while time, c, cx are declared in real*8. I have a lot of >>>> similar data conversion from real*8 to real*4 output, other code does not >>>> return error. >>>> >>>> time 0.0000000000000000 >>>> c 1 9.9999999999999995E-008 >>>> c 2 3.1555251077549618E-003 >>>> c 3 7.1657814842179362E-008 >>>> c 4 1.0976214263087059E-067 >>>> c 5 5.2879822292305797E-004 >>>> c 6 9.9999999999999964E-005 >>>> c 7 6.4055731968811337E-005 >>>> c 8 3.4607572892578404E-020 >>>> cx 1 3.4376650636008101E-005 >>>> cx 2 7.3989678854017763E-012 >>>> cx 3 9.5317170613607207E-012 >>>> cx 4 2.2344525794718353E-015 >>>> cx 5 3.0624685689695889E-008 >>>> cx 6 1.0046157902783967E-007 >>>> cx 7 1.5320169154914984E-004 >>>> cx 8 8.6930292776346176E-014 >>>> cx 9 3.5944267559348721E-005 >>>> cx 10 3.0072645866951157E-018 >>>> cx 11 2.3592486321095017E-013 >>>> >>>> Thanks, >>>> >>>> Danyang >>>> >>>>>> I can check all the entries of jacobi matrix to see if the value is valid, >>>>>> but this seems not a good idea as it takes a long time to reach this >>>>>> point. If I restart the simulation from a specified time (e.g., 7.685 in >>>>>> this case), then the error does not occur. >>>>>> >>>>>> Would you please give me any suggestion on debugging this case? >>>>>> >>>>>> Thanks and Regards, >>>>>> >>>>>> Danyang >>>>>> >>>>>> >>>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>> Reduce time step for reactive transport >>>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>> Reduce time step for reactive transport >>>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [1]PETSC ERROR: Floating point exception >>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [2]PETSC ERROR: Floating point exception >>>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >>>>>> at end of function: Parameter number 3 >>>>>> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>> for trouble shooting. >>>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>> for trouble shooting. >>>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>>>>> dsu Thu Apr 23 15:38:52 2015 >>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>> --download-parmetis --download-scalapack >>>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >>>>>> 15:38:52 2015 >>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>> --download-parmetis --download-scalapack >>>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>> #3 KSP_PCApply() line 230 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort From bsmith at mcs.anl.gov Sat Apr 25 13:44:06 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 25 Apr 2015 13:44:06 -0500 Subject: [petsc-users] Floating point exception In-Reply-To: <553BD931.9050202@gmail.com> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553BD931.9050202@gmail.com> Message-ID: The FPE is occurring inside hypre code. You need to run with the additional command line option -start_in_debugger and enter cont in each xterm that comes up with the debugger then when it crashes type where to see where it crashed and print the variables to see if it is a divide by zero etc. Since it crashes in hypre it is not likely that the matrix values or right hand side had inf or Nan in them. Barry > On Apr 25, 2015, at 1:13 PM, Danyang Su wrote: > > Hi Barry, > > With -fp_trap and -start_in_debugger options, the code crashed with the following error. > > The code at #21 0x41C49A in __solver_dd_MOD_solver_dd_snes_solve_react at solver_ddmethod.F90:2850 is "call KSPSolve(ksp_react,b_react,x_react,ierr)" > > I run this case with 4 processors and the preconditioner type is HYPRE. Does this mean something wrong in Matrix ksp_react or RHS b_react? > > Thanks, > > Danyang > > > timestep: 1846 time: 3.392E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 > Reduce time step for reactive transport > timestep: 1846 time: 3.387E+00 years delt: 5.000E-03 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 > Reduce time step for reactive transport > timestep: 1846 time: 3.385E+00 years delt: 2.500E-03 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 > [0]PETSC ERROR: *** unknown floating point error occurred *** > [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the > [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) > [0]PETSC ERROR: where the result is a bitwise OR of the following flags: > [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 > [0]PETSC ERROR: Try option -start_in_debugger > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [1]PETSC ERROR: *** unknown floating point error occurred *** > [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the > [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) > [1]PETSC ERROR: where the result is a bitwise OR of the following flags: > [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 > [1]PETSC ERROR: Try option -start_in_debugger > [1]PETSC ERROR: likely location of problem given in stack below > [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [1]PETSC ERROR: INSTEAD the line number of the start of the function > [1]PETSC ERROR: is given. > [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c > [1]PETSC ERROR: [1] Hypre solve line 174 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/impls/hypre/hypre.c > [1]PETSC ERROR: [1] PCApply_HYPRE line 161 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/impls/hypre/hypre.c > [1]PETSC ERROR: [2]PETSC ERROR: *** unknown floating point error occurred *** > [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the > [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) > [2]PETSC ERROR: where the result is a bitwise OR of the following flags: > [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 > [2]PETSC ERROR: Try option -start_in_debugger > [2]PETSC ERROR: likely location of problem given in stack below > [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [2]PETSC ERROR: INSTEAD the line number of the start of the function > [2]PETSC ERROR: is given. > [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c > [2]PETSC ERROR: [2] Hypre solve line 174 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/impls/hypre/hypre.c > [2]PETSC ERROR: [2] PCApply_HYPRE line 161 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/impls/hypre/hypre.c > [2]PETSC ERROR: [2] KSP_PCApply line 228 /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h > [2]PETSC ERROR: [2] KSPInitialResidual line 44 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c > [0]PETSC ERROR: [0] Hypre solve line 174 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/impls/hypre/hypre.c > [0]PETSC ERROR: [0] PCApply_HYPRE line 161 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/impls/hypre/hypre.c > [0]PETSC ERROR: [0] KSP_PCApply line 228 /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h > [0]PETSC ERROR: [0] KSPInitialResidual line 44 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: [0] KSPSolve_GMRES line 224 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c > [1] KSP_PCApply line 228 /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h > [1]PETSC ERROR: [1] KSPInitialResidual line 44 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c > [1]PETSC ERROR: [1] KSPSolve_GMRES line 224 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c > [2]PETSC ERROR: [2] KSPSolve_GMRES line 224 /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c > [2]PETSC ERROR: [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error > User provided function() line 0 in Unknown file trapped floating point error > [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error > > Program received signal SIGABRT: Process abort signal. > > Backtrace for this error: > > Program received signal SIGABRT: Process abort signal. > > Backtrace for this error: > > Program received signal SIGABRT: Process abort signal. > > Backtrace for this error: > #0 0x7FDC76F307D7 > #0 0x7FA04C1207D7 > #1 0x7FA04C120DDE > #1 0x7FDC76F30DDE > #2 0x7FA04B41ED3F > #3 0x7FA04B41ECC9 > #2 0x7FDC7622ED3F > #4 0x7FA04B4220D7 > #0 0x7F622A92F7D7 > #3 0x7FDC7622ECC9 > #5 0x7FA04C6BADCB > #1 0x7F622A92FDDE > #4 0x7FDC762320D7 > #6 0x7FA04C6B5825 > #2 0x7F6229C2DD3F > #7 0x7FA04C6BC17F > #5 0x7FDC774CADCB > #8 0x7FA04B41ED3F > #3 0x7F6229C2DCC9 > #6 0x7FDC774C5825 > #4 0x7F6229C310D7 > #9 0x7FA04D9EF449 > #7 0x7FDC774CC17F > #10 0x7FA04D9EF055 > #5 0x7F622AEC9DCB > #8 0x7FDC7622ED3F > #11 0x7FA04D99D2DD > #6 0x7F622AEC4825 > #9 0x7FDC787FF449 > #12 0x7FA04D984ACD > #7 0x7F622AECB17F > #10 0x7FDC787FF055 > #13 0x7FA04D973E63 > #8 0x7F6229C2DD3F > #11 0x7FDC787AD2DD > #14 0x7FA04D27E8E3 > #9 0x7F622C1FE449 > #12 0x7FDC78794ACD > #15 0x7FA04D2BEB04 > #10 0x7F622C1FE055 > #13 0x7FDC78783E63 > #16 0x7FA04D3CABFA > #11 0x7F622C1AC2DD > #17 0x7FA04D3CB927 > #14 0x7FDC7808E8E3 > #12 0x7F622C193ACD > #18 0x7FA04D361DE8 > #15 0x7FDC780CEB04 > #13 0x7F622C182E63 > #16 0x7FDC781DABFA > #19 0x7FA04D3A0E1D > #20 0x7FA04D3DC121 > #14 0x7F622BA8D8E3 > #15 0x7F622BACDB04 > #17 0x7FDC781DB927 > #18 0x7FDC78171DE8 > #16 0x7F622BBD9BFA > #19 0x7FDC781B0E1D > #17 0x7F622BBDA927 > #20 0x7FDC781EC121 > #18 0x7F622BB70DE8 > #19 0x7F622BBAFE1D > #20 0x7F622BBEB121 > #21 0x41C49A in __solver_dd_MOD_solver_dd_snes_solve_react at solver_ddmethod.F90:2850 > #21 0x41C49A in __solver_dd_MOD_solver_dd_snes_solve_react at solver_ddmethod.F90:2850 > #21 0x41C49A in __solver_dd_MOD_solver_dd_snes_solve_react at solver_ddmethod.F90:2850 > #22 0x6A25A5 in reactran_ at reactran.F90:954 > #22 0x6A25A5 in reactran_ at reactran.F90:954 > #22 0x6A25A5 in reactran_ at reactran.F90:954 > #23 0x574836 in timeloop_ at timeloop.F90:1194 > #23 0x574836 in timeloop_ at timeloop.F90:1194 > #23 0x574836 in timeloop_ at timeloop.F90:1194 > #24 0x5ABFD7 in driver_pc at driver_pc.F90:599 > #24 0x5ABFD7 in driver_pc at driver_pc.F90:599 > #24 0x5ABFD7 in driver_pc at driver_pc.F90:599 > > On 15-04-24 11:12 AM, Barry Smith wrote: >>> On Apr 24, 2015, at 1:05 PM, Danyang Su wrote: >>> >>> Hi All, >>> >>> One of my case crashes because of floating point exception when using 4 processors, as shown below. But if I run this case with 1 processor, it works fine. I have tested the codes with around 100 cases up to 768 processors, all other cases work fine. I just wonder if this kind of error is caused because of NaN in jacobi matrix, RHS or preconditioner? >> Yes, almost for sure it is one of these places. >> >> First run the bad case with -fp_trap if all goes well you'll see the function where the FPE is generated. Then run also with -start_in_debugger and >> type cont in all four debugger windows. When the FPE happens the debugger should stop showing exactly where the FPE happens. >> >> Barry >> >>> I can check all the entries of jacobi matrix to see if the value is valid, but this seems not a good idea as it takes a long time to reach this point. If I restart the simulation from a specified time (e.g., 7.685 in this case), then the error does not occur. >>> >>> Would you please give me any suggestion on debugging this case? >>> >>> Thanks and Regards, >>> >>> Danyang >>> >>> >>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> Reduce time step for reactive transport >>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> Reduce time step for reactive transport >>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [1]PETSC ERROR: Floating point exception >>> [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [2]PETSC ERROR: Floating point exception >>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at end of function: Parameter number 3 >>> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at end of function: Parameter number 3 >>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 15:38:52 2015 >>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack >>> [1]PETSC ERROR: #1 VecValidValues() line 34 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 15:38:52 2015 >>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack >>> [2]PETSC ERROR: #1 VecValidValues() line 34 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>> [1]PETSC ERROR: #2 PCApply() line 442 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>> #3 KSP_PCApply() line 230 in /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>> [2]PETSC ERROR: #6 KSPSolve() line 459 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>> [1]PETSC ERROR: #6 KSPSolve() line 459 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>> [mpiexec at nwmop] Press Ctrl-C again to force abort > From bsmith at mcs.anl.gov Sat Apr 25 13:55:15 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 25 Apr 2015 13:55:15 -0500 Subject: [petsc-users] Floating point exception In-Reply-To: <553BE244.7060407@gmail.com> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> Message-ID: <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> > On Apr 25, 2015, at 1:51 PM, Danyang Su wrote: > > > > On 15-04-25 11:32 AM, Barry Smith wrote: >> I told you this yesterday. >> >> It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. >> >> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate >> >> #elif defined PETSC_HAVE_XMMINTRIN_H >> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >> #else >> >> change it to >> >> #elif defined PETSC_HAVE_XMMINTRIN_H >> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >> #else >> >> Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. >> >> Did you do this? >> >> Barry > Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc directory. I just did a double check and ran make gnumake again and got the following information this time. > > dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ make gnumake > Building PETSc using GNU Make with 10 build threads > ========================================== > make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' > make[1]: Nothing to be done for `all'. > make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' > ========================================= > > > Then I recompiled the codes, ran with -fp_trap and still got the following error > > Backtrace for this error: > Note: The EXACT line numbers in the stack are not available, > [2]PETSC ERROR: INSTEAD the line number of the start of the function > [2]PETSC ERROR: is given. > [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c > INSTEAD the line number of the start of the function > [3]PETSC ERROR: is given. > [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c > [2]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error > [3]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error > This is different then what you sent a few minutes ago where it crashed in hypre. Anyways you need to use the -start_in_debugger business I sent in the previous email to see the exact place the problem occurs. > Thanks, > > Danyang >>> On Apr 25, 2015, at 1:05 AM, Danyang Su wrote: >>> >>> Hi Barry and Satish, >>> >>> How can I get rid of unknown floating point error when a very small value is multiplied. >>> >>> e.g., >>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, >>> >>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>> >>> I get the following error when run with "-fp_trap -start_in_debugger". >>> >>> Backtrace for this error: >>> *** unknown floating point error occurred *** >>> [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>> [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 >>> where the result is a bitwise OR of the following flags: >>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>> [2]PETSC ERROR: Try option -start_in_debugger >>> [2]PETSC ERROR: likely location of problem given in stack below >>> >>> Thanks, >>> >>> Danyang >>> >>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>>> c 4 1.0976214263087059E-067 >>>>> >>>>> I don't think this number can be stored in a real*4. >>>>> >>>>> Satish >>>> Thanks, Satish. It is caused by this number. >>>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>>> >>>>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su wrote: >>>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> One of my case crashes because of floating point exception when using 4 >>>>>>>> processors, as shown below. But if I run this case with 1 processor, it >>>>>>>> works fine. I have tested the codes with around 100 cases up to 768 >>>>>>>> processors, all other cases work fine. I just wonder if this kind of error >>>>>>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>>>>> Yes, almost for sure it is one of these places. >>>>>>> >>>>>>> First run the bad case with -fp_trap if all goes well you'll see the >>>>>>> function where the FPE is generated. Then run also with -start_in_debugger >>>>>>> and >>>>>>> type cont in all four debugger windows. When the FPE happens the debugger >>>>>>> should stop showing exactly where the FPE happens. >>>>>>> >>>>>>> Barry >>>>>> Hi Barry, >>>>>> >>>>>> If run with -fp_trap -start_in_debugger, I got the following error >>>>>> >>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a >>>>>> debugger. When the >>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >>>>>> fetestexcept(0x3d) >>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>> ------------------------------------ >>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>> [0]PETSC ERROR: is given. >>>>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>>>>> floating point error >>>>>> >>>>>> Program received signal SIGABRT: Process abort signal. >>>>>> >>>>>> Backtrace for this error: >>>>>> #0 0x7F4FEAB1C7D7 >>>>>> #1 0x7F4FEAB1CDDE >>>>>> #2 0x7F4FE9E1AD3F >>>>>> #3 0x7F4FE9E1ACC9 >>>>>> #4 0x7F4FE9E1E0D7 >>>>>> #5 0x7F4FEB0B6DCB >>>>>> #6 0x7F4FEB0B1825 >>>>>> #7 0x7F4FEB0B817F >>>>>> #8 0x7F4FE9E1AD3F >>>>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>>>> >>>>>> I checked the code at tprfrtlc.F90:2393, >>>>>> >>>>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>>>> (cx(ix),ix=1,nxout)/) >>>>>> >>>>>> All the values (time, c, cx) are reasonable, as shown below. The only >>>>>> possibility is that realbuffer_gb is in declared as real*4 if using sing >>>>>> precision output while time, c, cx are declared in real*8. I have a lot of >>>>>> similar data conversion from real*8 to real*4 output, other code does not >>>>>> return error. >>>>>> >>>>>> time 0.0000000000000000 >>>>>> c 1 9.9999999999999995E-008 >>>>>> c 2 3.1555251077549618E-003 >>>>>> c 3 7.1657814842179362E-008 >>>>>> c 4 1.0976214263087059E-067 >>>>>> c 5 5.2879822292305797E-004 >>>>>> c 6 9.9999999999999964E-005 >>>>>> c 7 6.4055731968811337E-005 >>>>>> c 8 3.4607572892578404E-020 >>>>>> cx 1 3.4376650636008101E-005 >>>>>> cx 2 7.3989678854017763E-012 >>>>>> cx 3 9.5317170613607207E-012 >>>>>> cx 4 2.2344525794718353E-015 >>>>>> cx 5 3.0624685689695889E-008 >>>>>> cx 6 1.0046157902783967E-007 >>>>>> cx 7 1.5320169154914984E-004 >>>>>> cx 8 8.6930292776346176E-014 >>>>>> cx 9 3.5944267559348721E-005 >>>>>> cx 10 3.0072645866951157E-018 >>>>>> cx 11 2.3592486321095017E-013 >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Danyang >>>>>> >>>>>>>> I can check all the entries of jacobi matrix to see if the value is valid, >>>>>>>> but this seems not a good idea as it takes a long time to reach this >>>>>>>> point. If I restart the simulation from a specified time (e.g., 7.685 in >>>>>>>> this case), then the error does not occur. >>>>>>>> >>>>>>>> Would you please give me any suggestion on debugging this case? >>>>>>>> >>>>>>>> Thanks and Regards, >>>>>>>> >>>>>>>> Danyang >>>>>>>> >>>>>>>> >>>>>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>> Reduce time step for reactive transport >>>>>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>> Reduce time step for reactive transport >>>>>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>>>> -------------------------------------------------------------- >>>>>>>> [1]PETSC ERROR: Floating point exception >>>>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>>>> -------------------------------------------------------------- >>>>>>>> [2]PETSC ERROR: Floating point exception >>>>>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >>>>>>>> at end of function: Parameter number 3 >>>>>>>> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>> for trouble shooting. >>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>>>>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>> for trouble shooting. >>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>>>>>>> dsu Thu Apr 23 15:38:52 2015 >>>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>> --download-parmetis --download-scalapack >>>>>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >>>>>>>> 15:38:52 2015 >>>>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>> --download-parmetis --download-scalapack >>>>>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>> #3 KSP_PCApply() line 230 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort From danyang.su at gmail.com Sat Apr 25 14:24:04 2015 From: danyang.su at gmail.com (Danyang Su) Date: Sat, 25 Apr 2015 12:24:04 -0700 Subject: [petsc-users] Floating point exception In-Reply-To: <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> Message-ID: <553BE9D4.7050801@gmail.com> On 15-04-25 11:55 AM, Barry Smith wrote: >> On Apr 25, 2015, at 1:51 PM, Danyang Su wrote: >> >> >> >> On 15-04-25 11:32 AM, Barry Smith wrote: >>> I told you this yesterday. >>> >>> It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. >>> >>> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate >>> >>> #elif defined PETSC_HAVE_XMMINTRIN_H >>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>> #else >>> >>> change it to >>> >>> #elif defined PETSC_HAVE_XMMINTRIN_H >>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >>> #else >>> >>> Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. >>> >>> Did you do this? >>> >>> Barry >> Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc directory. I just did a double check and ran make gnumake again and got the following information this time. >> >> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ make gnumake >> Building PETSc using GNU Make with 10 build threads >> ========================================== >> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >> make[1]: Nothing to be done for `all'. >> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >> ========================================= >> >> >> Then I recompiled the codes, ran with -fp_trap and still got the following error >> >> Backtrace for this error: >> Note: The EXACT line numbers in the stack are not available, >> [2]PETSC ERROR: INSTEAD the line number of the start of the function >> [2]PETSC ERROR: is given. >> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >> INSTEAD the line number of the start of the function >> [3]PETSC ERROR: is given. >> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >> [2]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >> [3]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >> > This is different then what you sent a few minutes ago where it crashed in hypre. > > Anyways you need to use the -start_in_debugger business I sent in the previous email to see the exact place the problem occurs. Here is the information shown on gdb screen Program received signal SIGFPE, Arithmetic exception. 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, njamxc=34, cinfradx=..., radial_coordx=.FALSE., _suffix=3) at ../../gas_advection/velocity_g.F90:1344 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 (gdb) After type cont on gdb screen, I got error information as below [1]PETSC ERROR: *** unknown floating point error occurred *** [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) [1]PETSC ERROR: where the result is a bitwise OR of the following flags: [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 [1]PETSC ERROR: Try option -start_in_debugger [1]PETSC ERROR: likely location of problem given in stack below [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [1]PETSC ERROR: INSTEAD the line number of the start of the function [1]PETSC ERROR: is given. [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error [0]PETSC ERROR: *** unknown floating point error occurred *** [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) [0]PETSC ERROR: where the result is a bitwise OR of the following flags: [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 [0]PETSC ERROR: Try option -start_in_debugger [0]PETSC ERROR: likely location of problem given in stack below Thanks, Danyang > >> Thanks, >> >> Danyang >>>> On Apr 25, 2015, at 1:05 AM, Danyang Su wrote: >>>> >>>> Hi Barry and Satish, >>>> >>>> How can I get rid of unknown floating point error when a very small value is multiplied. >>>> >>>> e.g., >>>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, >>>> >>>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>>> >>>> I get the following error when run with "-fp_trap -start_in_debugger". >>>> >>>> Backtrace for this error: >>>> *** unknown floating point error occurred *** >>>> [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>> [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 >>>> where the result is a bitwise OR of the following flags: >>>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>> [2]PETSC ERROR: Try option -start_in_debugger >>>> [2]PETSC ERROR: likely location of problem given in stack below >>>> >>>> Thanks, >>>> >>>> Danyang >>>> >>>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>>>> c 4 1.0976214263087059E-067 >>>>>> >>>>>> I don't think this number can be stored in a real*4. >>>>>> >>>>>> Satish >>>>> Thanks, Satish. It is caused by this number. >>>>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>>>> >>>>>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>>>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su wrote: >>>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> One of my case crashes because of floating point exception when using 4 >>>>>>>>> processors, as shown below. But if I run this case with 1 processor, it >>>>>>>>> works fine. I have tested the codes with around 100 cases up to 768 >>>>>>>>> processors, all other cases work fine. I just wonder if this kind of error >>>>>>>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>>>>>> Yes, almost for sure it is one of these places. >>>>>>>> >>>>>>>> First run the bad case with -fp_trap if all goes well you'll see the >>>>>>>> function where the FPE is generated. Then run also with -start_in_debugger >>>>>>>> and >>>>>>>> type cont in all four debugger windows. When the FPE happens the debugger >>>>>>>> should stop showing exactly where the FPE happens. >>>>>>>> >>>>>>>> Barry >>>>>>> Hi Barry, >>>>>>> >>>>>>> If run with -fp_trap -start_in_debugger, I got the following error >>>>>>> >>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a >>>>>>> debugger. When the >>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >>>>>>> fetestexcept(0x3d) >>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>>> ------------------------------------ >>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>> [0]PETSC ERROR: is given. >>>>>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>>>>>> floating point error >>>>>>> >>>>>>> Program received signal SIGABRT: Process abort signal. >>>>>>> >>>>>>> Backtrace for this error: >>>>>>> #0 0x7F4FEAB1C7D7 >>>>>>> #1 0x7F4FEAB1CDDE >>>>>>> #2 0x7F4FE9E1AD3F >>>>>>> #3 0x7F4FE9E1ACC9 >>>>>>> #4 0x7F4FE9E1E0D7 >>>>>>> #5 0x7F4FEB0B6DCB >>>>>>> #6 0x7F4FEB0B1825 >>>>>>> #7 0x7F4FEB0B817F >>>>>>> #8 0x7F4FE9E1AD3F >>>>>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>>>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>>>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>>>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>>>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>>>>> >>>>>>> I checked the code at tprfrtlc.F90:2393, >>>>>>> >>>>>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>>>>> (cx(ix),ix=1,nxout)/) >>>>>>> >>>>>>> All the values (time, c, cx) are reasonable, as shown below. The only >>>>>>> possibility is that realbuffer_gb is in declared as real*4 if using sing >>>>>>> precision output while time, c, cx are declared in real*8. I have a lot of >>>>>>> similar data conversion from real*8 to real*4 output, other code does not >>>>>>> return error. >>>>>>> >>>>>>> time 0.0000000000000000 >>>>>>> c 1 9.9999999999999995E-008 >>>>>>> c 2 3.1555251077549618E-003 >>>>>>> c 3 7.1657814842179362E-008 >>>>>>> c 4 1.0976214263087059E-067 >>>>>>> c 5 5.2879822292305797E-004 >>>>>>> c 6 9.9999999999999964E-005 >>>>>>> c 7 6.4055731968811337E-005 >>>>>>> c 8 3.4607572892578404E-020 >>>>>>> cx 1 3.4376650636008101E-005 >>>>>>> cx 2 7.3989678854017763E-012 >>>>>>> cx 3 9.5317170613607207E-012 >>>>>>> cx 4 2.2344525794718353E-015 >>>>>>> cx 5 3.0624685689695889E-008 >>>>>>> cx 6 1.0046157902783967E-007 >>>>>>> cx 7 1.5320169154914984E-004 >>>>>>> cx 8 8.6930292776346176E-014 >>>>>>> cx 9 3.5944267559348721E-005 >>>>>>> cx 10 3.0072645866951157E-018 >>>>>>> cx 11 2.3592486321095017E-013 >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Danyang >>>>>>> >>>>>>>>> I can check all the entries of jacobi matrix to see if the value is valid, >>>>>>>>> but this seems not a good idea as it takes a long time to reach this >>>>>>>>> point. If I restart the simulation from a specified time (e.g., 7.685 in >>>>>>>>> this case), then the error does not occur. >>>>>>>>> >>>>>>>>> Would you please give me any suggestion on debugging this case? >>>>>>>>> >>>>>>>>> Thanks and Regards, >>>>>>>>> >>>>>>>>> Danyang >>>>>>>>> >>>>>>>>> >>>>>>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>> Reduce time step for reactive transport >>>>>>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>> Reduce time step for reactive transport >>>>>>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>>>>> -------------------------------------------------------------- >>>>>>>>> [1]PETSC ERROR: Floating point exception >>>>>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>>>>> -------------------------------------------------------------- >>>>>>>>> [2]PETSC ERROR: Floating point exception >>>>>>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >>>>>>>>> at end of function: Parameter number 3 >>>>>>>>> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>> for trouble shooting. >>>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>>>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>>>>>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>> for trouble shooting. >>>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>>>>>>>> dsu Thu Apr 23 15:38:52 2015 >>>>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >>>>>>>>> 15:38:52 2015 >>>>>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>> #3 KSP_PCApply() line 230 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>>>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Apr 25 20:03:25 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 25 Apr 2015 20:03:25 -0500 Subject: [petsc-users] Floating point exception In-Reply-To: <553BE9D4.7050801@gmail.com> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> Message-ID: <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> If this is what you got in your last run > at ../../gas_advection/velocity_g.F90:1344 > 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 then it is still catching floating point underflow, which we do not want. This means either the change I suggested you make in the fp.c code didn't work or it actually uses a different floating point trap than that one. BTW: absurd numbers like 1.0d-316 are often a symptom of uninitialized data; could that be a problem that diff is not filled correctly for all the ic, idim you are using? This going round and round is very frustrating and a waste of time. You need to be more proactive yourself and explore the code and poke around to figure out how to solve the problem. Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can see what FP trap is being used on your machine. Barry > On Apr 25, 2015, at 2:24 PM, Danyang Su wrote: > > > > On 15-04-25 11:55 AM, Barry Smith wrote: >>> On Apr 25, 2015, at 1:51 PM, Danyang Su >>> wrote: >>> >>> >>> >>> On 15-04-25 11:32 AM, Barry Smith wrote: >>> >>>> I told you this yesterday. >>>> >>>> It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. >>>> >>>> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate >>>> >>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>>> #else >>>> >>>> change it to >>>> >>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >>>> #else >>>> >>>> Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. >>>> >>>> Did you do this? >>>> >>>> Barry >>>> >>> Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc directory. I just did a double check and ran make gnumake again and got the following information this time. >>> >>> >>> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >>> make gnumake >>> Building PETSc using GNU Make with 10 build threads >>> ========================================== >>> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>> make[1]: Nothing to be done for `all'. >>> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>> ========================================= >>> >>> >>> Then I recompiled the codes, ran with -fp_trap and still got the following error >>> >>> Backtrace for this error: >>> Note: The EXACT line numbers in the stack are not available, >>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>> [2]PETSC ERROR: is given. >>> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>> INSTEAD the line number of the start of the function >>> [3]PETSC ERROR: is given. >>> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>> [2]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>> [3]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>> >>> >> This is different then what you sent a few minutes ago where it crashed in hypre. >> >> Anyways you need to use the -start_in_debugger business I sent in the previous email to see the exact place the problem occurs. >> > Here is the information shown on gdb screen > > Program received signal SIGFPE, Arithmetic exception. > 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, njamxc=34, > cinfradx=..., radial_coordx=.FALSE., _suffix=3) > at ../../gas_advection/velocity_g.F90:1344 > 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 > (gdb) > > After type cont on gdb screen, I got error information as below > > [1]PETSC ERROR: *** unknown floating point error occurred *** > [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the > [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) > [1]PETSC ERROR: where the result is a bitwise OR of the following flags: > [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 > [1]PETSC ERROR: Try option -start_in_debugger > [1]PETSC ERROR: likely location of problem given in stack below > [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [1]PETSC ERROR: INSTEAD the line number of the start of the function > [1]PETSC ERROR: is given. > [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c > [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error > [0]PETSC ERROR: *** unknown floating point error occurred *** > [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the > [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) > [0]PETSC ERROR: where the result is a bitwise OR of the following flags: > [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 > [0]PETSC ERROR: Try option -start_in_debugger > [0]PETSC ERROR: likely location of problem given in stack below > > Thanks, > > Danyang >> >>> Thanks, >>> >>> Danyang >>> >>>>> On Apr 25, 2015, at 1:05 AM, Danyang Su >>>>> wrote: >>>>> >>>>> Hi Barry and Satish, >>>>> >>>>> How can I get rid of unknown floating point error when a very small value is multiplied. >>>>> >>>>> e.g., >>>>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, >>>>> >>>>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>>>> >>>>> I get the following error when run with "-fp_trap -start_in_debugger". >>>>> >>>>> Backtrace for this error: >>>>> *** unknown floating point error occurred *** >>>>> [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>> [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 >>>>> where the result is a bitwise OR of the following flags: >>>>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>> [2]PETSC ERROR: Try option -start_in_debugger >>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>> >>>>> Thanks, >>>>> >>>>> Danyang >>>>> >>>>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>>> >>>>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>>>> >>>>>>> c 4 1.0976214263087059E-067 >>>>>>> >>>>>>> I don't think this number can be stored in a real*4. >>>>>>> >>>>>>> Satish >>>>>>> >>>>>> Thanks, Satish. It is caused by this number. >>>>>> >>>>>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>>>>> >>>>>>> >>>>>>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>>>>> >>>>>>>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> One of my case crashes because of floating point exception when using 4 >>>>>>>>>> processors, as shown below. But if I run this case with 1 processor, it >>>>>>>>>> works fine. I have tested the codes with around 100 cases up to 768 >>>>>>>>>> processors, all other cases work fine. I just wonder if this kind of error >>>>>>>>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>>>>>>>> >>>>>>>>> Yes, almost for sure it is one of these places. >>>>>>>>> >>>>>>>>> First run the bad case with -fp_trap if all goes well you'll see the >>>>>>>>> function where the FPE is generated. Then run also with -start_in_debugger >>>>>>>>> and >>>>>>>>> type cont in all four debugger windows. When the FPE happens the debugger >>>>>>>>> should stop showing exactly where the FPE happens. >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> >>>>>>>> Hi Barry, >>>>>>>> >>>>>>>> If run with -fp_trap -start_in_debugger, I got the following error >>>>>>>> >>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a >>>>>>>> debugger. When the >>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >>>>>>>> fetestexcept(0x3d) >>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>>>> ------------------------------------ >>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>> [0]PETSC ERROR: is given. >>>>>>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>>>>>>> floating point error >>>>>>>> >>>>>>>> Program received signal SIGABRT: Process abort signal. >>>>>>>> >>>>>>>> Backtrace for this error: >>>>>>>> #0 0x7F4FEAB1C7D7 >>>>>>>> #1 0x7F4FEAB1CDDE >>>>>>>> #2 0x7F4FE9E1AD3F >>>>>>>> #3 0x7F4FE9E1ACC9 >>>>>>>> #4 0x7F4FE9E1E0D7 >>>>>>>> #5 0x7F4FEB0B6DCB >>>>>>>> #6 0x7F4FEB0B1825 >>>>>>>> #7 0x7F4FEB0B817F >>>>>>>> #8 0x7F4FE9E1AD3F >>>>>>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>>>>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>>>>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>>>>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>>>>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>>>>>> >>>>>>>> I checked the code at tprfrtlc.F90:2393, >>>>>>>> >>>>>>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>>>>>> (cx(ix),ix=1,nxout)/) >>>>>>>> >>>>>>>> All the values (time, c, cx) are reasonable, as shown below. The only >>>>>>>> possibility is that realbuffer_gb is in declared as real*4 if using sing >>>>>>>> precision output while time, c, cx are declared in real*8. I have a lot of >>>>>>>> similar data conversion from real*8 to real*4 output, other code does not >>>>>>>> return error. >>>>>>>> >>>>>>>> time 0.0000000000000000 >>>>>>>> c 1 9.9999999999999995E-008 >>>>>>>> c 2 3.1555251077549618E-003 >>>>>>>> c 3 7.1657814842179362E-008 >>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>> c 5 5.2879822292305797E-004 >>>>>>>> c 6 9.9999999999999964E-005 >>>>>>>> c 7 6.4055731968811337E-005 >>>>>>>> c 8 3.4607572892578404E-020 >>>>>>>> cx 1 3.4376650636008101E-005 >>>>>>>> cx 2 7.3989678854017763E-012 >>>>>>>> cx 3 9.5317170613607207E-012 >>>>>>>> cx 4 2.2344525794718353E-015 >>>>>>>> cx 5 3.0624685689695889E-008 >>>>>>>> cx 6 1.0046157902783967E-007 >>>>>>>> cx 7 1.5320169154914984E-004 >>>>>>>> cx 8 8.6930292776346176E-014 >>>>>>>> cx 9 3.5944267559348721E-005 >>>>>>>> cx 10 3.0072645866951157E-018 >>>>>>>> cx 11 2.3592486321095017E-013 >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Danyang >>>>>>>> >>>>>>>> >>>>>>>>>> I can check all the entries of jacobi matrix to see if the value is valid, >>>>>>>>>> but this seems not a good idea as it takes a long time to reach this >>>>>>>>>> point. If I restart the simulation from a specified time (e.g., 7.685 in >>>>>>>>>> this case), then the error does not occur. >>>>>>>>>> >>>>>>>>>> Would you please give me any suggestion on debugging this case? >>>>>>>>>> >>>>>>>>>> Thanks and Regards, >>>>>>>>>> >>>>>>>>>> Danyang >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>> [1]PETSC ERROR: Floating point exception >>>>>>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>> [2]PETSC ERROR: Floating point exception >>>>>>>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >>>>>>>>>> at end of function: Parameter number 3 >>>>>>>>>> [2]PETSC ERROR: See >>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>> >>>>>>>>>> for trouble shooting. >>>>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>>>>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>>>>>>> [1]PETSC ERROR: See >>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>> >>>>>>>>>> for trouble shooting. >>>>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>>>>>>>>> dsu Thu Apr 23 15:38:52 2015 >>>>>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >>>>>>>>>> 15:38:52 2015 >>>>>>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>> #3 KSP_PCApply() line 230 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>>>>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>>>>>>>>> > From danyang.su at gmail.com Sat Apr 25 20:23:33 2015 From: danyang.su at gmail.com (Danyang Su) Date: Sat, 25 Apr 2015 18:23:33 -0700 Subject: [petsc-users] Floating point exception In-Reply-To: <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> Message-ID: <553C3E15.8030403@gmail.com> On 15-04-25 06:03 PM, Barry Smith wrote: > If this is what you got in your last run > >> at ../../gas_advection/velocity_g.F90:1344 >> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 > then it is still catching floating point underflow, which we do not want. This means either the change I suggested you make in the fp.c code didn't work or it actually uses a different floating point trap than that one. BTW: absurd numbers like 1.0d-316 are often a symptom of uninitialized data; could that be a problem that diff is not filled correctly for all the ic, idim you are using? > > This going round and round is very frustrating and a waste of time. You need to be more proactive yourself and explore the code and poke around to figure out how to solve the problem. > > Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can see what FP trap is being used on your machine. > > Barry Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise I cannot find this file. Do I need to reconfigure PETSc after changing the code you mentioned? Danyang > > > >> On Apr 25, 2015, at 2:24 PM, Danyang Su wrote: >> >> >> >> On 15-04-25 11:55 AM, Barry Smith wrote: >>>> On Apr 25, 2015, at 1:51 PM, Danyang Su >>>> wrote: >>>> >>>> >>>> >>>> On 15-04-25 11:32 AM, Barry Smith wrote: >>>> >>>>> I told you this yesterday. >>>>> >>>>> It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. >>>>> >>>>> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate >>>>> >>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>>>> #else >>>>> >>>>> change it to >>>>> >>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >>>>> #else >>>>> >>>>> Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. >>>>> >>>>> Did you do this? >>>>> >>>>> Barry >>>>> >>>> Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc directory. I just did a double check and ran make gnumake again and got the following information this time. >>>> >>>> >>>> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >>>> make gnumake >>>> Building PETSc using GNU Make with 10 build threads >>>> ========================================== >>>> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>> make[1]: Nothing to be done for `all'. >>>> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>> ========================================= >>>> >>>> >>>> Then I recompiled the codes, ran with -fp_trap and still got the following error >>>> >>>> Backtrace for this error: >>>> Note: The EXACT line numbers in the stack are not available, >>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>>> [2]PETSC ERROR: is given. >>>> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>> INSTEAD the line number of the start of the function >>>> [3]PETSC ERROR: is given. >>>> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>> [2]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>> [3]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>> >>>> >>> This is different then what you sent a few minutes ago where it crashed in hypre. >>> >>> Anyways you need to use the -start_in_debugger business I sent in the previous email to see the exact place the problem occurs. >>> >> Here is the information shown on gdb screen >> >> Program received signal SIGFPE, Arithmetic exception. >> 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, njamxc=34, >> cinfradx=..., radial_coordx=.FALSE., _suffix=3) >> at ../../gas_advection/velocity_g.F90:1344 >> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >> (gdb) >> >> After type cont on gdb screen, I got error information as below >> >> [1]PETSC ERROR: *** unknown floating point error occurred *** >> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >> [1]PETSC ERROR: Try option -start_in_debugger >> [1]PETSC ERROR: likely location of problem given in stack below >> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> [1]PETSC ERROR: INSTEAD the line number of the start of the function >> [1]PETSC ERROR: is given. >> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >> [0]PETSC ERROR: *** unknown floating point error occurred *** >> [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >> [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >> [0]PETSC ERROR: Try option -start_in_debugger >> [0]PETSC ERROR: likely location of problem given in stack below >> >> Thanks, >> >> Danyang >>>> Thanks, >>>> >>>> Danyang >>>> >>>>>> On Apr 25, 2015, at 1:05 AM, Danyang Su >>>>>> wrote: >>>>>> >>>>>> Hi Barry and Satish, >>>>>> >>>>>> How can I get rid of unknown floating point error when a very small value is multiplied. >>>>>> >>>>>> e.g., >>>>>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, >>>>>> >>>>>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>>>>> >>>>>> I get the following error when run with "-fp_trap -start_in_debugger". >>>>>> >>>>>> Backtrace for this error: >>>>>> *** unknown floating point error occurred *** >>>>>> [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>> [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 >>>>>> where the result is a bitwise OR of the following flags: >>>>>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>> [2]PETSC ERROR: Try option -start_in_debugger >>>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Danyang >>>>>> >>>>>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>>>> >>>>>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>>>>> >>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>> >>>>>>>> I don't think this number can be stored in a real*4. >>>>>>>> >>>>>>>> Satish >>>>>>>> >>>>>>> Thanks, Satish. It is caused by this number. >>>>>>> >>>>>>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>>>>>> >>>>>>>> >>>>>>>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>>>>>> >>>>>>>>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>> One of my case crashes because of floating point exception when using 4 >>>>>>>>>>> processors, as shown below. But if I run this case with 1 processor, it >>>>>>>>>>> works fine. I have tested the codes with around 100 cases up to 768 >>>>>>>>>>> processors, all other cases work fine. I just wonder if this kind of error >>>>>>>>>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>>>>>>>>> >>>>>>>>>> Yes, almost for sure it is one of these places. >>>>>>>>>> >>>>>>>>>> First run the bad case with -fp_trap if all goes well you'll see the >>>>>>>>>> function where the FPE is generated. Then run also with -start_in_debugger >>>>>>>>>> and >>>>>>>>>> type cont in all four debugger windows. When the FPE happens the debugger >>>>>>>>>> should stop showing exactly where the FPE happens. >>>>>>>>>> >>>>>>>>>> Barry >>>>>>>>>> >>>>>>>>> Hi Barry, >>>>>>>>> >>>>>>>>> If run with -fp_trap -start_in_debugger, I got the following error >>>>>>>>> >>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a >>>>>>>>> debugger. When the >>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >>>>>>>>> fetestexcept(0x3d) >>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>>>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>>>>> ------------------------------------ >>>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>> [0]PETSC ERROR: is given. >>>>>>>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>>>>>>>> floating point error >>>>>>>>> >>>>>>>>> Program received signal SIGABRT: Process abort signal. >>>>>>>>> >>>>>>>>> Backtrace for this error: >>>>>>>>> #0 0x7F4FEAB1C7D7 >>>>>>>>> #1 0x7F4FEAB1CDDE >>>>>>>>> #2 0x7F4FE9E1AD3F >>>>>>>>> #3 0x7F4FE9E1ACC9 >>>>>>>>> #4 0x7F4FE9E1E0D7 >>>>>>>>> #5 0x7F4FEB0B6DCB >>>>>>>>> #6 0x7F4FEB0B1825 >>>>>>>>> #7 0x7F4FEB0B817F >>>>>>>>> #8 0x7F4FE9E1AD3F >>>>>>>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>>>>>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>>>>>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>>>>>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>>>>>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>>>>>>> >>>>>>>>> I checked the code at tprfrtlc.F90:2393, >>>>>>>>> >>>>>>>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>>>>>>> (cx(ix),ix=1,nxout)/) >>>>>>>>> >>>>>>>>> All the values (time, c, cx) are reasonable, as shown below. The only >>>>>>>>> possibility is that realbuffer_gb is in declared as real*4 if using sing >>>>>>>>> precision output while time, c, cx are declared in real*8. I have a lot of >>>>>>>>> similar data conversion from real*8 to real*4 output, other code does not >>>>>>>>> return error. >>>>>>>>> >>>>>>>>> time 0.0000000000000000 >>>>>>>>> c 1 9.9999999999999995E-008 >>>>>>>>> c 2 3.1555251077549618E-003 >>>>>>>>> c 3 7.1657814842179362E-008 >>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>> c 5 5.2879822292305797E-004 >>>>>>>>> c 6 9.9999999999999964E-005 >>>>>>>>> c 7 6.4055731968811337E-005 >>>>>>>>> c 8 3.4607572892578404E-020 >>>>>>>>> cx 1 3.4376650636008101E-005 >>>>>>>>> cx 2 7.3989678854017763E-012 >>>>>>>>> cx 3 9.5317170613607207E-012 >>>>>>>>> cx 4 2.2344525794718353E-015 >>>>>>>>> cx 5 3.0624685689695889E-008 >>>>>>>>> cx 6 1.0046157902783967E-007 >>>>>>>>> cx 7 1.5320169154914984E-004 >>>>>>>>> cx 8 8.6930292776346176E-014 >>>>>>>>> cx 9 3.5944267559348721E-005 >>>>>>>>> cx 10 3.0072645866951157E-018 >>>>>>>>> cx 11 2.3592486321095017E-013 >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Danyang >>>>>>>>> >>>>>>>>> >>>>>>>>>>> I can check all the entries of jacobi matrix to see if the value is valid, >>>>>>>>>>> but this seems not a good idea as it takes a long time to reach this >>>>>>>>>>> point. If I restart the simulation from a specified time (e.g., 7.685 in >>>>>>>>>>> this case), then the error does not occur. >>>>>>>>>>> >>>>>>>>>>> Would you please give me any suggestion on debugging this case? >>>>>>>>>>> >>>>>>>>>>> Thanks and Regards, >>>>>>>>>>> >>>>>>>>>>> Danyang >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>> [1]PETSC ERROR: Floating point exception >>>>>>>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>> [2]PETSC ERROR: Floating point exception >>>>>>>>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >>>>>>>>>>> at end of function: Parameter number 3 >>>>>>>>>>> [2]PETSC ERROR: See >>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>>> >>>>>>>>>>> for trouble shooting. >>>>>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>>>>>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>>>>>>>> [1]PETSC ERROR: See >>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>>> >>>>>>>>>>> for trouble shooting. >>>>>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>>>>>>>>>> dsu Thu Apr 23 15:38:52 2015 >>>>>>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >>>>>>>>>>> 15:38:52 2015 >>>>>>>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>>> #3 KSP_PCApply() line 230 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>>>>>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>>>>>>>>>> -------------- next part -------------- MPICXX_SHOW = g++ -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O0 -fPIC -I/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/include -L/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -lmpichcxx -Wl,-rpath -Wl,/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -lmpich -lopa -lmpl -lrt -lpthread C_DEPFLAGS = -MMD -MP FC_DEFINE_FLAG = -D MPICC_SHOW = gcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 -I/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/include -L/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -Wl,-rpath -Wl,/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -lmpich -lopa -lmpl -lrt -lpthread AR_FLAGS = cr CXX_DEPFLAGS = -MMD -MP FC_DEPFLAGS = -MMD -MP MPIFC_SHOW = gfortran -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -g -O0 -I/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/include -I/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/include -L/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -lmpichf90 -Wl,-rpath -Wl,/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -lmpich -lopa -lmpl -lrt -lpthread FAST_AR_FLAGS = Scq FC_MODULE_OUTPUT_FLAG = -J PETSC_LANGUAGE = CONLY FC_LINKER_FLAGS = -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -g -O0 LIBNAME = ${INSTALL_LIB_DIR}/libpetsc.${AR_LIB_SUFFIX} SL_LINKER = /home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/bin/mpicc PETSC_BUILD_USING_CMAKE = 1 CC_FLAGS = -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 PETSC_PRECISION = double PETSC_LIB_BASIC = -lpetsc SUPERLU_DIST_LIB = -Wl,-rpath,/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -L/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -lsuperlu_dist_3.3 FC_FLAGS = -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -g -O0 BLASLAPACK_LIB = -Wl,-rpath,/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -L/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -lflapack -lfblas PETSC_MAT_LIB = ${C_SH_LIB_PATH} ${PETSC_WITH_EXTERNAL_LIB} PCC = /home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/bin/mpicc SL_LINKER_LIBS = ${PETSC_EXTERNAL_LIB_BASIC} VALGRIND_INCLUDE = MPI_LIB = PETSC_SNES_LIB = ${C_SH_LIB_PATH} ${PETSC_WITH_EXTERNAL_LIB} SSL_LIB = -lssl -lcrypto PETSC_EXTERNAL_LIB_BASIC = -Wl,-rpath,/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -L/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lHYPRE -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpichcxx -lstdc++ -lsuperlu_dist_3.3 -lflapack -lfblas -lparmetis -lmetis -lpthread -lssl -lcrypto -lm -lmpichf90 -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpichcxx -lstdc++ -Wl,-rpath,/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -L/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -Wl,-rpath,/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -ldl PETSC_TS_LIB = ${C_SH_LIB_PATH} ${PETSC_WITH_EXTERNAL_LIB} SL_LINKER_FLAGS = ${PCC_LINKER_FLAGS} PARMETIS_LIB = -Wl,-rpath,/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -L/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -lparmetis SUPERLU_DIST_INCLUDE = -I/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/include CC_SUFFIX = o PETSC_LIB = ${C_SH_LIB_PATH} ${PETSC_WITH_EXTERNAL_LIB} SCALAPACK_INCLUDE = SHLIBS = libpetsc METIS_INCLUDE = -I/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/include PETSC_CHARACTERISTIC_LIB = ${C_SH_LIB_PATH} ${PETSC_WITH_EXTERNAL_LIB} PTHREAD_LIB = -lpthread PETSC_SCALAR = real PETSC_FC_INCLUDES = -I/home/dsu/Soft/PETSc/petsc-3.5.2/include -I/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/include CPP_FLAGS = PETSC_TAO_LIB_BASIC = -lpetsc PETSC_KSP_LIB_BASIC = -lpetsc FPP_FLAGS = HYPRE_INCLUDE = -I/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/include FC_LINKER = /home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/bin/mpif90 HYPRE_LIB = -Wl,-rpath,/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -L/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -lHYPRE -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpichcxx -lstdc++ PETSC_KSP_LIB = ${C_SH_LIB_PATH} ${PETSC_WITH_EXTERNAL_LIB} CXX_FLAGS = -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O0 -fPIC PCC_LINKER_FLAGS = -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 SSL_INCLUDE = PETSC_CONTRIB = ${C_SH_LIB_PATH} ${PETSC_WITH_EXTERNAL_LIB} PETSC_CC_INCLUDES = -I/home/dsu/Soft/PETSc/petsc-3.5.2/include -I/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/include PCC_LINKER = /home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/bin/mpicc PETSC_SYS_LIB = ${C_SH_LIB_PATH} ${PETSC_WITH_EXTERNAL_LIB} PCC_FLAGS = -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 VALGRIND_LIB = PTHREAD_INCLUDE = METIS_LIB = -Wl,-rpath,/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -L/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -lmetis FBLASLAPACK_INCLUDE = BLASLAPACK_INCLUDE = PETSC_TS_LIB_BASIC = -lpetsc PETSC_VEC_LIB = ${C_SH_LIB_PATH} ${PETSC_WITH_EXTERNAL_LIB} CC_LINKER_SUFFIX = FBLASLAPACK_LIB = SL_LINKER_SUFFIX = so PARMETIS_INCLUDE = -I/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/include PETSC_DM_LIB = ${C_SH_LIB_PATH} ${PETSC_WITH_EXTERNAL_LIB} DESTDIR = /home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg FC_MODULE_FLAG = -I SCALAPACK_LIB = -Wl,-rpath,/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -L/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -lscalapack PETSC_WITH_EXTERNAL_LIB = -L/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -lpetsc -Wl,-rpath,/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lHYPRE -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpichcxx -lstdc++ -lsuperlu_dist_3.3 -lflapack -lfblas -lparmetis -lmetis -lpthread -lssl -lcrypto -lm -lmpichf90 -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpichcxx -lstdc++ -Wl,-rpath,/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -L/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -Wl,-rpath,/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -ldl CONFIGURE_OPTIONS = PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack MUMPS_INCLUDE = -I/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/include PETSC_TAO_LIB = ${C_SH_LIB_PATH} ${PETSC_WITH_EXTERNAL_LIB} MPI_INCLUDE = -I/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/include FC_SUFFIX = o MUMPS_LIB = -Wl,-rpath,/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -L/home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord SHELL = /bin/sh GREP = /bin/grep MV = /bin/mv PYTHON = /usr/bin/python MKDIR = /bin/mkdir -p SEDINPLACE = /bin/sed -i SED = /bin/sed DIFF = /usr/bin/diff -w GZIP = /bin/gzip RM = /bin/rm -f CP = /bin/cp CC_LINKER_SLFLAG = -Wl,-rpath, CC = /home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/bin/mpicc RANLIB = /usr/bin/ranlib DYNAMICLINKER = /home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/bin/mpicc CXX = /home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/bin/mpicxx FC = /home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/bin/mpif90 CXXCPP = /home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/bin/mpicxx -E FC_LINKER_SLFLAG = -Wl,-rpath, CPP = /home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/bin/mpicc -E AR_LIB_SUFFIX = a LD_SHARED = /home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/bin/mpicc AR = /usr/bin/ar DIR = /home/dsu/Soft/PETSc/petsc-3.5.2 OMAKE = /usr/bin/make --no-print-directory MAKE_IS_GNUMAKE = 1 MAKE = /usr/bin/make MAKE_NP = 10 OMAKE_PRINTDIR = /usr/bin/make --print-directory MPIEXEC = /home/dsu/Soft/PETSc/petsc-3.5.2/linux-gnu-dbg/bin/mpiexec GDB = /usr/bin/gdb DSYMUTIL = true SL_LINKER_FUNCTION = -shared -Wl,-soname,$(call SONAME_FUNCTION,$(notdir $(1)),$(2)) SONAME_FUNCTION = $(1).so.$(2) BUILDSHAREDLIB = yes TEST_RUNS = C Fortran SSL METIS PARMETIS SUPERLU_DIST HYPRE MUMPS Cxx F90 F90_NoComplex F2003 Fortran_NoComplex C_NoComplex CMAKE = /usr/bin/cmake From knepley at gmail.com Sat Apr 25 20:26:19 2015 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 25 Apr 2015 20:26:19 -0500 Subject: [petsc-users] Floating point exception In-Reply-To: <553C3E15.8030403@gmail.com> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> <553C3E15.8030403@gmail.com> Message-ID: On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su wrote: > > > On 15-04-25 06:03 PM, Barry Smith wrote: > >> If this is what you got in your last run >> >> at ../../gas_advection/velocity_g.F90:1344 >>> 1344 cinfrt = cinfrt_dg(i1) * >>> diff(ic,idim) !diff is a very small >>> value, e.g., 1.0d-316 >>> >> then it is still catching floating point underflow, which we do not >> want. This means either the change I suggested you make in the fp.c code >> didn't work or it actually uses a different floating point trap than that >> one. BTW: absurd numbers like 1.0d-316 are often a symptom of >> uninitialized data; could that be a problem that diff is not filled >> correctly for all the ic, idim you are using? >> >> This going round and round is very frustrating and a waste of time. >> You need to be more proactive yourself and explore the code and poke around >> to figure out how to solve the problem. >> >> Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can >> see what FP trap is being used on your machine. >> >> Barry >> > Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise I cannot > find this file. > Its include/petscconf.h > Do I need to reconfigure PETSc after changing the code you mentioned? > No, but you need to rebuild. Matt > Danyang > >> >> >> >> On Apr 25, 2015, at 2:24 PM, Danyang Su wrote: >>> >>> >>> >>> On 15-04-25 11:55 AM, Barry Smith wrote: >>> >>>> On Apr 25, 2015, at 1:51 PM, Danyang Su >>>>> wrote: >>>>> >>>>> >>>>> >>>>> On 15-04-25 11:32 AM, Barry Smith wrote: >>>>> >>>>> I told you this yesterday. >>>>>> >>>>>> It is probably stopping here on a harmless underflow. You need to >>>>>> edit the PETSc code to not worry about underflow. >>>>>> >>>>>> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and >>>>>> locate >>>>>> >>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>>>>> #else >>>>>> >>>>>> change it to >>>>>> >>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >>>>>> #else >>>>>> >>>>>> Then run make gnumake in the PETSc directory to compile the new >>>>>> version. Now link and run the program again with -fp_trap and see where it >>>>>> gets stuck this time. >>>>>> >>>>>> Did you do this? >>>>>> >>>>>> Barry >>>>>> >>>>>> Yes, I did change the code in fp.c and run 'make gnumake' in the >>>>> PETSc directory. I just did a double check and ran make gnumake again and >>>>> got the following information this time. >>>>> >>>>> >>>>> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >>>>> make gnumake >>>>> Building PETSc using GNU Make with 10 build threads >>>>> ========================================== >>>>> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>> make[1]: Nothing to be done for `all'. >>>>> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>> ========================================= >>>>> >>>>> >>>>> Then I recompiled the codes, ran with -fp_trap and still got the >>>>> following error >>>>> >>>>> Backtrace for this error: >>>>> Note: The EXACT line numbers in the stack are not available, >>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the >>>>> function >>>>> [2]PETSC ERROR: is given. >>>>> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>> INSTEAD the line number of the start of the function >>>>> [3]PETSC ERROR: is given. >>>>> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>> [2]PETSC ERROR: User provided function() line 0 in Unknown file >>>>> trapped floating point error >>>>> [3]PETSC ERROR: User provided function() line 0 in Unknown file >>>>> trapped floating point error >>>>> >>>>> >>>>> This is different then what you sent a few minutes ago where it >>>> crashed in hypre. >>>> >>>> Anyways you need to use the -start_in_debugger business I sent in >>>> the previous email to see the exact place the problem occurs. >>>> >>>> Here is the information shown on gdb screen >>> >>> Program received signal SIGFPE, Arithmetic exception. >>> 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, >>> njamxc=34, >>> cinfradx=..., radial_coordx=.FALSE., _suffix=3) >>> at ../../gas_advection/velocity_g.F90:1344 >>> 1344 cinfrt = cinfrt_dg(i1) * >>> diff(ic,idim) !diff is a very small >>> value, e.g., 1.0d-316 >>> (gdb) >>> >>> After type cont on gdb screen, I got error information as below >>> >>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>> [1]PETSC ERROR: The specific exception can be determined by running in a >>> debugger. When the >>> [1]PETSC ERROR: debugger traps the signal, the exception can be found >>> with fetestexcept(0x3d) >>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>> [1]PETSC ERROR: Try option -start_in_debugger >>> [1]PETSC ERROR: likely location of problem given in stack below >>> [1]PETSC ERROR: --------------------- Stack Frames >>> ------------------------------------ >>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>> available, >>> [1]PETSC ERROR: INSTEAD the line number of the start of the >>> function >>> [1]PETSC ERROR: is given. >>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 >>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>> floating point error >>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>> [0]PETSC ERROR: The specific exception can be determined by running in a >>> debugger. When the >>> [0]PETSC ERROR: debugger traps the signal, the exception can be found >>> with fetestexcept(0x3d) >>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>> [0]PETSC ERROR: Try option -start_in_debugger >>> [0]PETSC ERROR: likely location of problem given in stack below >>> >>> Thanks, >>> >>> Danyang >>> >>>> Thanks, >>>>> >>>>> Danyang >>>>> >>>>> On Apr 25, 2015, at 1:05 AM, Danyang Su >>>>>>> wrote: >>>>>>> >>>>>>> Hi Barry and Satish, >>>>>>> >>>>>>> How can I get rid of unknown floating point error when a very small >>>>>>> value is multiplied. >>>>>>> >>>>>>> e.g., >>>>>>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 >>>>>>> 8.6178408169776945E-317 respectively, >>>>>>> >>>>>>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>>>>>> >>>>>>> I get the following error when run with "-fp_trap >>>>>>> -start_in_debugger". >>>>>>> >>>>>>> Backtrace for this error: >>>>>>> *** unknown floating point error occurred *** >>>>>>> [2]PETSC ERROR: The specific exception can be determined by running >>>>>>> in a debugger. When the >>>>>>> [2]PETSC ERROR: debugger traps the signal, the exception can be >>>>>>> found with fetestexcept(0x3d) >>>>>>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) >>>>>>> 1.0250235986806329E-008 8.6178408169776945E-317 >>>>>>> where the result is a bitwise OR of the following flags: >>>>>>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>> [2]PETSC ERROR: Try option -start_in_debugger >>>>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Danyang >>>>>>> >>>>>>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>>>>> >>>>>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>>>>>> >>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>> >>>>>>>>> I don't think this number can be stored in a real*4. >>>>>>>>> >>>>>>>>> Satish >>>>>>>>> >>>>>>>>> Thanks, Satish. It is caused by this number. >>>>>>>> >>>>>>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>>>>>>> >>>>>>>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi All, >>>>>>>>>>>> >>>>>>>>>>>> One of my case crashes because of floating point exception when >>>>>>>>>>>> using 4 >>>>>>>>>>>> processors, as shown below. But if I run this case with 1 >>>>>>>>>>>> processor, it >>>>>>>>>>>> works fine. I have tested the codes with around 100 cases up to >>>>>>>>>>>> 768 >>>>>>>>>>>> processors, all other cases work fine. I just wonder if this >>>>>>>>>>>> kind of error >>>>>>>>>>>> is caused because of NaN in jacobi matrix, RHS or >>>>>>>>>>>> preconditioner? >>>>>>>>>>>> >>>>>>>>>>>> Yes, almost for sure it is one of these places. >>>>>>>>>>> >>>>>>>>>>> First run the bad case with -fp_trap if all goes well >>>>>>>>>>> you'll see the >>>>>>>>>>> function where the FPE is generated. Then run also with >>>>>>>>>>> -start_in_debugger >>>>>>>>>>> and >>>>>>>>>>> type cont in all four debugger windows. When the FPE happens the >>>>>>>>>>> debugger >>>>>>>>>>> should stop showing exactly where the FPE happens. >>>>>>>>>>> >>>>>>>>>>> Barry >>>>>>>>>>> >>>>>>>>>>> Hi Barry, >>>>>>>>>> >>>>>>>>>> If run with -fp_trap -start_in_debugger, I got the following error >>>>>>>>>> >>>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by >>>>>>>>>> running in a >>>>>>>>>> debugger. When the >>>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be >>>>>>>>>> found with >>>>>>>>>> fetestexcept(0x3d) >>>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following >>>>>>>>>> flags: >>>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>>>>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>>>>>> ------------------------------------ >>>>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>>>>>>>> available, >>>>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the >>>>>>>>>> function >>>>>>>>>> [0]PETSC ERROR: is given. >>>>>>>>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file >>>>>>>>>> trapped >>>>>>>>>> floating point error >>>>>>>>>> >>>>>>>>>> Program received signal SIGABRT: Process abort signal. >>>>>>>>>> >>>>>>>>>> Backtrace for this error: >>>>>>>>>> #0 0x7F4FEAB1C7D7 >>>>>>>>>> #1 0x7F4FEAB1CDDE >>>>>>>>>> #2 0x7F4FE9E1AD3F >>>>>>>>>> #3 0x7F4FE9E1ACC9 >>>>>>>>>> #4 0x7F4FE9E1E0D7 >>>>>>>>>> #5 0x7F4FEB0B6DCB >>>>>>>>>> #6 0x7F4FEB0B1825 >>>>>>>>>> #7 0x7F4FEB0B817F >>>>>>>>>> #8 0x7F4FE9E1AD3F >>>>>>>>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>>>>>>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>>>>>>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>>>>>>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>>>>>>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>>>>>>>> >>>>>>>>>> I checked the code at tprfrtlc.F90:2393, >>>>>>>>>> >>>>>>>>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>>>>>>>> (cx(ix),ix=1,nxout)/) >>>>>>>>>> >>>>>>>>>> All the values (time, c, cx) are reasonable, as shown below. The >>>>>>>>>> only >>>>>>>>>> possibility is that realbuffer_gb is in declared as real*4 if >>>>>>>>>> using sing >>>>>>>>>> precision output while time, c, cx are declared in real*8. I have >>>>>>>>>> a lot of >>>>>>>>>> similar data conversion from real*8 to real*4 output, other code >>>>>>>>>> does not >>>>>>>>>> return error. >>>>>>>>>> >>>>>>>>>> time 0.0000000000000000 >>>>>>>>>> c 1 9.9999999999999995E-008 >>>>>>>>>> c 2 3.1555251077549618E-003 >>>>>>>>>> c 3 7.1657814842179362E-008 >>>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>>> c 5 5.2879822292305797E-004 >>>>>>>>>> c 6 9.9999999999999964E-005 >>>>>>>>>> c 7 6.4055731968811337E-005 >>>>>>>>>> c 8 3.4607572892578404E-020 >>>>>>>>>> cx 1 3.4376650636008101E-005 >>>>>>>>>> cx 2 7.3989678854017763E-012 >>>>>>>>>> cx 3 9.5317170613607207E-012 >>>>>>>>>> cx 4 2.2344525794718353E-015 >>>>>>>>>> cx 5 3.0624685689695889E-008 >>>>>>>>>> cx 6 1.0046157902783967E-007 >>>>>>>>>> cx 7 1.5320169154914984E-004 >>>>>>>>>> cx 8 8.6930292776346176E-014 >>>>>>>>>> cx 9 3.5944267559348721E-005 >>>>>>>>>> cx 10 3.0072645866951157E-018 >>>>>>>>>> cx 11 2.3592486321095017E-013 >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Danyang >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I can check all the entries of jacobi matrix to see if the value >>>>>>>>>>>> is valid, >>>>>>>>>>>> but this seems not a good idea as it takes a long time to reach >>>>>>>>>>>> this >>>>>>>>>>>> point. If I restart the simulation from a specified time (e.g., >>>>>>>>>>>> 7.685 in >>>>>>>>>>>> this case), then the error does not occur. >>>>>>>>>>>> >>>>>>>>>>>> Would you please give me any suggestion on debugging this case? >>>>>>>>>>>> >>>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>> >>>>>>>>>>>> Danyang >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years >>>>>>>>>>>> iter: 1 >>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years >>>>>>>>>>>> iter: 1 >>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years >>>>>>>>>>>> iter: 1 >>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years >>>>>>>>>>>> iter: 1 >>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years >>>>>>>>>>>> iter: 1 >>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years >>>>>>>>>>>> iter: 1 >>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years >>>>>>>>>>>> iter: 1 >>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>> [1]PETSC ERROR: Floating point exception >>>>>>>>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>> [2]PETSC ERROR: Floating point exception >>>>>>>>>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number >>>>>>>>>>>> or infinite >>>>>>>>>>>> at end of function: Parameter number 3 >>>>>>>>>>>> [2]PETSC ERROR: See >>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>>>> >>>>>>>>>>>> for trouble shooting. >>>>>>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>>>>>>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>>>>>>>>> [1]PETSC ERROR: See >>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>>>> >>>>>>>>>>>> for trouble shooting. >>>>>>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg >>>>>>>>>>>> named nwmop by >>>>>>>>>>>> dsu Thu Apr 23 15:38:52 2015 >>>>>>>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg >>>>>>>>>>>> --with-cc=gcc >>>>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack >>>>>>>>>>>> --download-mpich >>>>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist >>>>>>>>>>>> --download-metis >>>>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu >>>>>>>>>>>> Thu Apr 23 >>>>>>>>>>>> 15:38:52 2015 >>>>>>>>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg >>>>>>>>>>>> --with-cc=gcc >>>>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack >>>>>>>>>>>> --download-mpich >>>>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist >>>>>>>>>>>> --download-metis >>>>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>>>> #3 KSP_PCApply() line 230 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>>>>>>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>>>>>>>>>>> >>>>>>>>>>>> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Sat Apr 25 20:31:18 2015 From: danyang.su at gmail.com (Danyang Su) Date: Sat, 25 Apr 2015 18:31:18 -0700 Subject: [petsc-users] Floating point exception In-Reply-To: References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> <553C3E15.8030403@gmail.com> Message-ID: <553C3FE6.5020203@gmail.com> On 15-04-25 06:26 PM, Matthew Knepley wrote: > On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su > wrote: > > > > On 15-04-25 06:03 PM, Barry Smith wrote: > > If this is what you got in your last run > > at ../../gas_advection/velocity_g.F90:1344 > 1344 cinfrt = > cinfrt_dg(i1) * diff(ic,idim) !diff is a very > small value, e.g., 1.0d-316 > > then it is still catching floating point underflow, which > we do not want. This means either the change I suggested you > make in the fp.c code didn't work or it actually uses a > different floating point trap than that one. BTW: absurd > numbers like 1.0d-316 are often a symptom of uninitialized > data; could that be a problem that diff is not filled > correctly for all the ic, idim you are using? > > This going round and round is very frustrating and a waste > of time. You need to be more proactive yourself and explore > the code and poke around to figure out how to solve the problem. > > Please email > $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can see > what FP trap is being used on your machine. > > Barry > > Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise > I cannot find this file. > > > Its include/petscconf.h > > Do I need to reconfigure PETSc after changing the code you mentioned? > > > No, but you need to rebuild. Yes, I have done 'gnumake'. > > Matt > > Danyang > > > > > On Apr 25, 2015, at 2:24 PM, Danyang Su > > wrote: > > > > On 15-04-25 11:55 AM, Barry Smith wrote: > > On Apr 25, 2015, at 1:51 PM, Danyang Su > > > wrote: > > > > On 15-04-25 11:32 AM, Barry Smith wrote: > > I told you this yesterday. > > It is probably stopping here on a harmless > underflow. You need to edit the PETSc code to > not worry about underflow. > > Edit the file > /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c > and locate > > #elif defined PETSC_HAVE_XMMINTRIN_H > _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); > #else > > change it to > > #elif defined PETSC_HAVE_XMMINTRIN_H > _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | > _MM_MASK_UNDERFLOW); > #else > > Then run make gnumake in the PETSc directory > to compile the new version. Now link and run > the program again with -fp_trap and see where > it gets stuck this time. > > Did you do this? > > Barry > > Yes, I did change the code in fp.c and run 'make > gnumake' in the PETSc directory. I just did a > double check and ran make gnumake again and got > the following information this time. > > > dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ > make gnumake > Building PETSc using GNU Make with 10 build threads > ========================================== > make[1]: Entering directory > `/home/dsu/Soft/PETSc/petsc-3.5.2' > make[1]: Nothing to be done for `all'. > make[1]: Leaving directory > `/home/dsu/Soft/PETSc/petsc-3.5.2' > ========================================= > > > Then I recompiled the codes, ran with -fp_trap and > still got the following error > > Backtrace for this error: > Note: The EXACT line numbers in the stack are not > available, > [2]PETSC ERROR: INSTEAD the line number of > the start of the function > [2]PETSC ERROR: is given. > [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 > /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c > INSTEAD the line number of the start of the > function > [3]PETSC ERROR: is given. > [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 > /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c > [2]PETSC ERROR: User provided function() line 0 in > Unknown file trapped floating point error > [3]PETSC ERROR: User provided function() line 0 in > Unknown file trapped floating point error > > > This is different then what you sent a few minutes > ago where it crashed in hypre. > > Anyways you need to use the -start_in_debugger > business I sent in the previous email to see the exact > place the problem occurs. > > Here is the information shown on gdb screen > > Program received signal SIGFPE, Arithmetic exception. > 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., > nmax=12, njamxc=34, > cinfradx=..., radial_coordx=.FALSE., _suffix=3) > at ../../gas_advection/velocity_g.F90:1344 > 1344 cinfrt = > cinfrt_dg(i1) * diff(ic,idim) !diff is a very > small value, e.g., 1.0d-316 > (gdb) > > After type cont on gdb screen, I got error information as > below > > [1]PETSC ERROR: *** unknown floating point error occurred *** > [1]PETSC ERROR: The specific exception can be determined > by running in a debugger. When the > [1]PETSC ERROR: debugger traps the signal, the exception > can be found with fetestexcept(0x3d) > [1]PETSC ERROR: where the result is a bitwise OR of the > following flags: > [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 > FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 > [1]PETSC ERROR: Try option -start_in_debugger > [1]PETSC ERROR: likely location of problem given in stack > below > [1]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [1]PETSC ERROR: Note: The EXACT line numbers in the stack > are not available, > [1]PETSC ERROR: INSTEAD the line number of the start > of the function > [1]PETSC ERROR: is given. > [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 > /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c > [1]PETSC ERROR: User provided function() line 0 in Unknown > file trapped floating point error > [0]PETSC ERROR: *** unknown floating point error occurred *** > [0]PETSC ERROR: The specific exception can be determined > by running in a debugger. When the > [0]PETSC ERROR: debugger traps the signal, the exception > can be found with fetestexcept(0x3d) > [0]PETSC ERROR: where the result is a bitwise OR of the > following flags: > [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 > FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 > [0]PETSC ERROR: Try option -start_in_debugger > [0]PETSC ERROR: likely location of problem given in stack > below > > Thanks, > > Danyang > > Thanks, > > Danyang > > On Apr 25, 2015, at 1:05 AM, Danyang Su > > > wrote: > > Hi Barry and Satish, > > How can I get rid of unknown floating > point error when a very small value is > multiplied. > > e.g., > cinfrt_dg(i1) and diff(ic,idim) are > 1.0250235986806329E-008 > 8.6178408169776945E-317 respectively, > > cinfrt = cinfrt_dg(i1) * diff(ic,idim) > > I get the following error when run with > "-fp_trap -start_in_debugger". > > Backtrace for this error: > *** unknown floating point error occurred *** > [2]PETSC ERROR: The specific exception can > be determined by running in a debugger. > When the > [2]PETSC ERROR: debugger traps the signal, > the exception can be found with > fetestexcept(0x3d) > [2]PETSC ERROR: > cinfrt_dg(i1),diff(ic,idim) > 1.0250235986806329E-008 > 8.6178408169776945E-317 > where the result is a bitwise OR of the > following flags: > [2]PETSC ERROR: FE_INVALID=0x1 > FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 > FE_UNDERFLOW=0x10 FE_INEXACT=0x20 > [2]PETSC ERROR: Try option -start_in_debugger > [2]PETSC ERROR: likely location of problem > given in stack below > > Thanks, > > Danyang > > On 15-04-24 01:54 PM, Danyang Su wrote: > > On 15-04-24 01:23 PM, Satish Balay wrote: > > c 4 > 1.0976214263087059E-067 > > I don't think this number can be > stored in a real*4. > > Satish > > Thanks, Satish. It is caused by this > number. > > On Fri, 24 Apr 2015, Danyang Su wrote: > > > On 15-04-24 11:12 AM, Barry > Smith wrote: > > On Apr 24, 2015, at > 1:05 PM, Danyang Su > > > wrote: > > Hi All, > > One of my case crashes > because of floating > point exception when > using 4 > processors, as shown > below. But if I run > this case with 1 > processor, it > works fine. I have > tested the codes with > around 100 cases up to 768 > processors, all other > cases work fine. I > just wonder if this > kind of error > is caused because of > NaN in jacobi matrix, > RHS or preconditioner? > > Yes, almost for sure > it is one of these places. > > First run the bad > case with -fp_trap if all > goes well you'll see the > function where the FPE is > generated. Then run also > with -start_in_debugger > and > type cont in all four > debugger windows. When the > FPE happens the debugger > should stop showing > exactly where the FPE happens. > > Barry > > Hi Barry, > > If run with -fp_trap > -start_in_debugger, I got the > following error > > [0]PETSC ERROR: *** unknown > floating point error occurred *** > [0]PETSC ERROR: The specific > exception can be determined by > running in a > debugger. When the > [0]PETSC ERROR: debugger traps > the signal, the exception can > be found with > fetestexcept(0x3d) > [0]PETSC ERROR: where the > result is a bitwise OR of the > following flags: > [0]PETSC ERROR: FE_INVALID=0x1 > FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 > FE_UNDERFLOW=0x10 FE_INEXACT=0x20 > [0]PETSC ERROR: Try option > -start_in_debugger > [0]PETSC ERROR: likely > location of problem given in > stack below > [0]PETSC ERROR: > --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The > EXACT line numbers in the > stack are not available, > [0]PETSC ERROR: INSTEAD > the line number of the start > of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] > PetscDefaultFPTrap line 379 > /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c > [0]PETSC ERROR: User provided > function() line 0 in Unknown > file trapped > floating point error > > Program received signal > SIGABRT: Process abort signal. > > Backtrace for this error: > #0 0x7F4FEAB1C7D7 > #1 0x7F4FEAB1CDDE > #2 0x7F4FE9E1AD3F > #3 0x7F4FE9E1ACC9 > #4 0x7F4FE9E1E0D7 > #5 0x7F4FEB0B6DCB > #6 0x7F4FEB0B1825 > #7 0x7F4FEB0B817F > #8 0x7F4FE9E1AD3F > #9 0x6972C8 in tprfrtlc_ at > tprfrtlc.F90:2393 > (discriminator 3) > #10 0x4C6C87 in gcreact_ at > gcreact.F90:678 > #11 0x707E19 in initicrt_ at > initicrt.F90:589 > #12 0x4F42D0 in initprob_ at > initprob.F90:430 > #13 0x5AAF72 in driver_pc at > driver_pc.F90:438 > > I checked the code at > tprfrtlc.F90:2393, > > > realbuffer_gb(1:nvars) = > (/time,(c(ic),ic=1,nc-1), & > (cx(ix),ix=1,nxout)/) > > All the values (time, c, cx) > are reasonable, as shown > below. The only > possibility is that > realbuffer_gb is in declared > as real*4 if using sing > precision output while time, > c, cx are declared in real*8. > I have a lot of > similar data conversion from > real*8 to real*4 output, other > code does not > return error. > > time 0.0000000000000000 > c 1 > 9.9999999999999995E-008 > c 2 > 3.1555251077549618E-003 > c 3 > 7.1657814842179362E-008 > c 4 > 1.0976214263087059E-067 > c 5 > 5.2879822292305797E-004 > c 6 > 9.9999999999999964E-005 > c 7 > 6.4055731968811337E-005 > c 8 > 3.4607572892578404E-020 > cx 1 > 3.4376650636008101E-005 > cx 2 > 7.3989678854017763E-012 > cx 3 > 9.5317170613607207E-012 > cx 4 > 2.2344525794718353E-015 > cx 5 > 3.0624685689695889E-008 > cx 6 > 1.0046157902783967E-007 > cx 7 > 1.5320169154914984E-004 > cx 8 > 8.6930292776346176E-014 > cx 9 > 3.5944267559348721E-005 > cx 10 > 3.0072645866951157E-018 > cx 11 > 2.3592486321095017E-013 > > Thanks, > > Danyang > > > I can check all the > entries of jacobi > matrix to see if the > value is valid, > but this seems not a > good idea as it takes > a long time to reach this > point. If I restart > the simulation from a > specified time (e.g., > 7.685 in > this case), then the > error does not occur. > > Would you please give > me any suggestion on > debugging this case? > > Thanks and Regards, > > Danyang > > > timestep: 2730 > time: 7.665E+00 years > delt: 1.000E-02 years > iter: 1 > timestep: max.sia: > 0.000E+00 tol.sia: > 0.000E+00 > timestep: 2731 > time: 7.675E+00 years > delt: 1.000E-02 years > iter: 1 > timestep: max.sia: > 0.000E+00 tol.sia: > 0.000E+00 > timestep: 2732 > time: 7.685E+00 years > delt: 1.000E-02 years > iter: 1 > timestep: max.sia: > 0.000E+00 tol.sia: > 0.000E+00 > timestep: 2733 > time: 7.695E+00 years > delt: 1.000E-02 years > iter: 1 > timestep: max.sia: > 0.000E+00 tol.sia: > 0.000E+00 > timestep: 2734 > time: 7.705E+00 years > delt: 1.000E-02 years > iter: 1 > timestep: max.sia: > 0.000E+00 tol.sia: > 0.000E+00 > Reduce time step for > reactive transport > timestep: 2734 > time: 7.700E+00 years > delt: 5.000E-03 years > iter: 1 > timestep: max.sia: > 0.000E+00 tol.sia: > 0.000E+00 > Reduce time step for > reactive transport > timestep: 2734 > time: 7.697E+00 years > delt: 2.500E-03 years > iter: 1 > timestep: max.sia: > 0.000E+00 tol.sia: > 0.000E+00 > [1]PETSC ERROR: > --------------------- > Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: > Floating point exception > [2]PETSC ERROR: > --------------------- > Error Message > -------------------------------------------------------------- > [2]PETSC ERROR: > Floating point exception > [2]PETSC ERROR: Vec > entry at local > location 0 is > not-a-number or infinite > at end of function: > Parameter number 3 > [2]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html > > for trouble shooting. > [2]PETSC ERROR: Petsc > Release Version 3.5.2, > Sep, 08, 2014 > [2]PETSC ERROR: > [1]PETSC ERROR: Vec > entry at local > location 0 is > not-a-number or > infinite at end of > function: Parameter > number 3 > [1]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html > > for trouble shooting. > [1]PETSC ERROR: Petsc > Release Version 3.5.2, > Sep, 08, 2014 > [1]PETSC ERROR: > ../min3p_thcm_petsc_dbg on > a linux-gnu-dbg named > nwmop by > dsu Thu Apr 23 > 15:38:52 2015 > [1]PETSC ERROR: > Configure options > PETSC_ARCH=linux-gnu-dbg > --with-cc=gcc > --with-cxx=g++ > --with-fc=gfortran > --download-fblaslapack > --download-mpich > --download-mumps > --download-hypre > --download-superlu_dist --download-metis > --download-parmetis > --download-scalapack > [1]PETSC ERROR: #1 > VecValidValues() line > 34 in > /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c > ../min3p_thcm_petsc_dbg on > a linux-gnu-dbg named > nwmop by dsu Thu Apr 23 > 15:38:52 2015 > [2]PETSC ERROR: > Configure options > PETSC_ARCH=linux-gnu-dbg > --with-cc=gcc > --with-cxx=g++ > --with-fc=gfortran > --download-fblaslapack > --download-mpich > --download-mumps > --download-hypre > --download-superlu_dist --download-metis > --download-parmetis > --download-scalapack > [2]PETSC ERROR: #1 > VecValidValues() line > 34 in > /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c > [2]PETSC ERROR: > [1]PETSC ERROR: #2 > PCApply() line 442 in > /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c > [1]PETSC ERROR: #2 > PCApply() line 442 in > /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c > [2]PETSC ERROR: #3 > KSP_PCApply() line 230 in > /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h > #3 KSP_PCApply() line > 230 in > /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h > [1]PETSC ERROR: #4 > KSPInitialResidual() > line 63 in > /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c > [2]PETSC ERROR: #4 > KSPInitialResidual() > line 63 in > /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c > [1]PETSC ERROR: #5 > KSPSolve_GMRES() line > 234 in > /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c > [2]PETSC ERROR: #5 > KSPSolve_GMRES() line > 234 in > /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c > [2]PETSC ERROR: #6 > KSPSolve() line 459 in > /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: #6 > KSPSolve() line 459 in > /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c > ^C[mpiexec at nwmop] > Sending Ctrl-C to > processes as requested > [mpiexec at nwmop] Press > Ctrl-C again to force > abort > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: petscconf.h Type: text/x-chdr Size: 16776 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sat Apr 25 21:02:18 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 25 Apr 2015 21:02:18 -0500 Subject: [petsc-users] Floating point exception In-Reply-To: <553C3FE6.5020203@gmail.com> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> <553C3E15.8030403@gmail.com> <553C3FE6.5020203@gmail.com> Message-ID: Ok, you do have #ifndef PETSC_HAVE_XMMINTRIN_H #define PETSC_HAVE_XMMINTRIN_H 1 #endif so the change you made should cause it to stop trapping underflow exceptions. Now in one email you reported a FPE within hypre, then I asked you to run with -start_in_debugger to determine where it happened exactly and then you reported the FPE happened in user code (what seemed to be an underflow issue). Why is this? Can you not run it where it generated the FPE in hypre using the -start_in_debugger option? Barry Perhaps you have multiple PETSC_ARCH or multiple PETSc installs to explain why you reported two different places where the exception occurred. > On Apr 25, 2015, at 8:31 PM, Danyang Su wrote: > > > > On 15-04-25 06:26 PM, Matthew Knepley wrote: >> On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su wrote: >> >> >> On 15-04-25 06:03 PM, Barry Smith wrote: >> If this is what you got in your last run >> >> at ../../gas_advection/velocity_g.F90:1344 >> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >> then it is still catching floating point underflow, which we do not want. This means either the change I suggested you make in the fp.c code didn't work or it actually uses a different floating point trap than that one. BTW: absurd numbers like 1.0d-316 are often a symptom of uninitialized data; could that be a problem that diff is not filled correctly for all the ic, idim you are using? >> >> This going round and round is very frustrating and a waste of time. You need to be more proactive yourself and explore the code and poke around to figure out how to solve the problem. >> >> Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can see what FP trap is being used on your machine. >> >> Barry >> Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise I cannot find this file. >> >> Its include/petscconf.h >> >> Do I need to reconfigure PETSc after changing the code you mentioned? >> >> No, but you need to rebuild. > Yes, I have done 'gnumake'. >> >> Matt >> >> Danyang >> >> >> >> On Apr 25, 2015, at 2:24 PM, Danyang Su wrote: >> >> >> >> On 15-04-25 11:55 AM, Barry Smith wrote: >> On Apr 25, 2015, at 1:51 PM, Danyang Su >> wrote: >> >> >> >> On 15-04-25 11:32 AM, Barry Smith wrote: >> >> I told you this yesterday. >> >> It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. >> >> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate >> >> #elif defined PETSC_HAVE_XMMINTRIN_H >> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >> #else >> >> change it to >> >> #elif defined PETSC_HAVE_XMMINTRIN_H >> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >> #else >> >> Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. >> >> Did you do this? >> >> Barry >> >> Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc directory. I just did a double check and ran make gnumake again and got the following information this time. >> >> >> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >> make gnumake >> Building PETSc using GNU Make with 10 build threads >> ========================================== >> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >> make[1]: Nothing to be done for `all'. >> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >> ========================================= >> >> >> Then I recompiled the codes, ran with -fp_trap and still got the following error >> >> Backtrace for this error: >> Note: The EXACT line numbers in the stack are not available, >> [2]PETSC ERROR: INSTEAD the line number of the start of the function >> [2]PETSC ERROR: is given. >> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >> INSTEAD the line number of the start of the function >> [3]PETSC ERROR: is given. >> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >> [2]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >> [3]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >> >> >> This is different then what you sent a few minutes ago where it crashed in hypre. >> >> Anyways you need to use the -start_in_debugger business I sent in the previous email to see the exact place the problem occurs. >> >> Here is the information shown on gdb screen >> >> Program received signal SIGFPE, Arithmetic exception. >> 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, njamxc=34, >> cinfradx=..., radial_coordx=.FALSE., _suffix=3) >> at ../../gas_advection/velocity_g.F90:1344 >> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >> (gdb) >> >> After type cont on gdb screen, I got error information as below >> >> [1]PETSC ERROR: *** unknown floating point error occurred *** >> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >> [1]PETSC ERROR: Try option -start_in_debugger >> [1]PETSC ERROR: likely location of problem given in stack below >> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> [1]PETSC ERROR: INSTEAD the line number of the start of the function >> [1]PETSC ERROR: is given. >> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >> [0]PETSC ERROR: *** unknown floating point error occurred *** >> [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >> [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >> [0]PETSC ERROR: Try option -start_in_debugger >> [0]PETSC ERROR: likely location of problem given in stack below >> >> Thanks, >> >> Danyang >> Thanks, >> >> Danyang >> >> On Apr 25, 2015, at 1:05 AM, Danyang Su >> wrote: >> >> Hi Barry and Satish, >> >> How can I get rid of unknown floating point error when a very small value is multiplied. >> >> e.g., >> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, >> >> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >> >> I get the following error when run with "-fp_trap -start_in_debugger". >> >> Backtrace for this error: >> *** unknown floating point error occurred *** >> [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >> [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 >> where the result is a bitwise OR of the following flags: >> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >> [2]PETSC ERROR: Try option -start_in_debugger >> [2]PETSC ERROR: likely location of problem given in stack below >> >> Thanks, >> >> Danyang >> >> On 15-04-24 01:54 PM, Danyang Su wrote: >> >> On 15-04-24 01:23 PM, Satish Balay wrote: >> >> c 4 1.0976214263087059E-067 >> >> I don't think this number can be stored in a real*4. >> >> Satish >> >> Thanks, Satish. It is caused by this number. >> >> On Fri, 24 Apr 2015, Danyang Su wrote: >> >> >> On 15-04-24 11:12 AM, Barry Smith wrote: >> >> On Apr 24, 2015, at 1:05 PM, Danyang Su >> wrote: >> >> Hi All, >> >> One of my case crashes because of floating point exception when using 4 >> processors, as shown below. But if I run this case with 1 processor, it >> works fine. I have tested the codes with around 100 cases up to 768 >> processors, all other cases work fine. I just wonder if this kind of error >> is caused because of NaN in jacobi matrix, RHS or preconditioner? >> >> Yes, almost for sure it is one of these places. >> >> First run the bad case with -fp_trap if all goes well you'll see the >> function where the FPE is generated. Then run also with -start_in_debugger >> and >> type cont in all four debugger windows. When the FPE happens the debugger >> should stop showing exactly where the FPE happens. >> >> Barry >> >> Hi Barry, >> >> If run with -fp_trap -start_in_debugger, I got the following error >> >> [0]PETSC ERROR: *** unknown floating point error occurred *** >> [0]PETSC ERROR: The specific exception can be determined by running in a >> debugger. When the >> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >> fetestexcept(0x3d) >> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >> [0]PETSC ERROR: Try option -start_in_debugger >> [0]PETSC ERROR: likely location of problem given in stack below >> [0]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> [0]PETSC ERROR: INSTEAD the line number of the start of the function >> [0]PETSC ERROR: is given. >> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >> floating point error >> >> Program received signal SIGABRT: Process abort signal. >> >> Backtrace for this error: >> #0 0x7F4FEAB1C7D7 >> #1 0x7F4FEAB1CDDE >> #2 0x7F4FE9E1AD3F >> #3 0x7F4FE9E1ACC9 >> #4 0x7F4FE9E1E0D7 >> #5 0x7F4FEB0B6DCB >> #6 0x7F4FEB0B1825 >> #7 0x7F4FEB0B817F >> #8 0x7F4FE9E1AD3F >> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >> #11 0x707E19 in initicrt_ at initicrt.F90:589 >> #12 0x4F42D0 in initprob_ at initprob.F90:430 >> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >> >> I checked the code at tprfrtlc.F90:2393, >> >> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >> (cx(ix),ix=1,nxout)/) >> >> All the values (time, c, cx) are reasonable, as shown below. The only >> possibility is that realbuffer_gb is in declared as real*4 if using sing >> precision output while time, c, cx are declared in real*8. I have a lot of >> similar data conversion from real*8 to real*4 output, other code does not >> return error. >> >> time 0.0000000000000000 >> c 1 9.9999999999999995E-008 >> c 2 3.1555251077549618E-003 >> c 3 7.1657814842179362E-008 >> c 4 1.0976214263087059E-067 >> c 5 5.2879822292305797E-004 >> c 6 9.9999999999999964E-005 >> c 7 6.4055731968811337E-005 >> c 8 3.4607572892578404E-020 >> cx 1 3.4376650636008101E-005 >> cx 2 7.3989678854017763E-012 >> cx 3 9.5317170613607207E-012 >> cx 4 2.2344525794718353E-015 >> cx 5 3.0624685689695889E-008 >> cx 6 1.0046157902783967E-007 >> cx 7 1.5320169154914984E-004 >> cx 8 8.6930292776346176E-014 >> cx 9 3.5944267559348721E-005 >> cx 10 3.0072645866951157E-018 >> cx 11 2.3592486321095017E-013 >> >> Thanks, >> >> Danyang >> >> >> I can check all the entries of jacobi matrix to see if the value is valid, >> but this seems not a good idea as it takes a long time to reach this >> point. If I restart the simulation from a specified time (e.g., 7.685 in >> this case), then the error does not occur. >> >> Would you please give me any suggestion on debugging this case? >> >> Thanks and Regards, >> >> Danyang >> >> >> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >> Reduce time step for reactive transport >> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >> Reduce time step for reactive transport >> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >> [1]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [1]PETSC ERROR: Floating point exception >> [2]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [2]PETSC ERROR: Floating point exception >> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >> at end of function: Parameter number 3 >> [2]PETSC ERROR: See >> http://www.mcs.anl.gov/petsc/documentation/faq.html >> >> for trouble shooting. >> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >> not-a-number or infinite at end of function: Parameter number 3 >> [1]PETSC ERROR: See >> http://www.mcs.anl.gov/petsc/documentation/faq.html >> >> for trouble shooting. >> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >> dsu Thu Apr 23 15:38:52 2015 >> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >> --download-mumps --download-hypre --download-superlu_dist --download-metis >> --download-parmetis --download-scalapack >> [1]PETSC ERROR: #1 VecValidValues() line 34 in >> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >> 15:38:52 2015 >> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >> --download-mumps --download-hypre --download-superlu_dist --download-metis >> --download-parmetis --download-scalapack >> [2]PETSC ERROR: #1 VecValidValues() line 34 in >> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >> [1]PETSC ERROR: #2 PCApply() line 442 in >> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >> #3 KSP_PCApply() line 230 in >> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >> [2]PETSC ERROR: #6 KSPSolve() line 459 in >> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >> [1]PETSC ERROR: #6 KSPSolve() line 459 in >> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >> [mpiexec at nwmop] Press Ctrl-C again to force abort >> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > > From danyang.su at gmail.com Sat Apr 25 21:34:53 2015 From: danyang.su at gmail.com (Danyang Su) Date: Sat, 25 Apr 2015 19:34:53 -0700 Subject: [petsc-users] Floating point exception In-Reply-To: References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> <553C3E15.8030403@gmail.com> <553C3FE6.5020203@gmail.com> Message-ID: <553C4ECD.6090905@gmail.com> Hi All, The "floating point underflow" is caused by a small value divided by a very large value. This result is forced to zero and then it does not report any underflow problem. I just rerun this bad case to see if it still get stuck later. This will take a while. Thanks for all your kindly reply, Danyang On 15-04-25 07:02 PM, Barry Smith wrote: > Ok, you do have > > #ifndef PETSC_HAVE_XMMINTRIN_H > #define PETSC_HAVE_XMMINTRIN_H 1 > #endif > > so the change you made should cause it to stop trapping underflow exceptions. > > Now in one email you reported a FPE within hypre, then I asked you to run with -start_in_debugger to determine where it happened exactly and then you reported the FPE happened in user code (what seemed to be an underflow issue). Why is this? Can you not run it where it generated the FPE in hypre using the -start_in_debugger option? > > Barry > > Perhaps you have multiple PETSC_ARCH or multiple PETSc installs to explain why you reported two different places where the exception occurred. > >> On Apr 25, 2015, at 8:31 PM, Danyang Su wrote: >> >> >> >> On 15-04-25 06:26 PM, Matthew Knepley wrote: >>> On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su wrote: >>> >>> >>> On 15-04-25 06:03 PM, Barry Smith wrote: >>> If this is what you got in your last run >>> >>> at ../../gas_advection/velocity_g.F90:1344 >>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>> then it is still catching floating point underflow, which we do not want. This means either the change I suggested you make in the fp.c code didn't work or it actually uses a different floating point trap than that one. BTW: absurd numbers like 1.0d-316 are often a symptom of uninitialized data; could that be a problem that diff is not filled correctly for all the ic, idim you are using? >>> >>> This going round and round is very frustrating and a waste of time. You need to be more proactive yourself and explore the code and poke around to figure out how to solve the problem. >>> >>> Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can see what FP trap is being used on your machine. >>> >>> Barry >>> Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise I cannot find this file. >>> >>> Its include/petscconf.h >>> >>> Do I need to reconfigure PETSc after changing the code you mentioned? >>> >>> No, but you need to rebuild. >> Yes, I have done 'gnumake'. >>> Matt >>> >>> Danyang >>> >>> >>> >>> On Apr 25, 2015, at 2:24 PM, Danyang Su wrote: >>> >>> >>> >>> On 15-04-25 11:55 AM, Barry Smith wrote: >>> On Apr 25, 2015, at 1:51 PM, Danyang Su >>> wrote: >>> >>> >>> >>> On 15-04-25 11:32 AM, Barry Smith wrote: >>> >>> I told you this yesterday. >>> >>> It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. >>> >>> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate >>> >>> #elif defined PETSC_HAVE_XMMINTRIN_H >>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>> #else >>> >>> change it to >>> >>> #elif defined PETSC_HAVE_XMMINTRIN_H >>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >>> #else >>> >>> Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. >>> >>> Did you do this? >>> >>> Barry >>> >>> Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc directory. I just did a double check and ran make gnumake again and got the following information this time. >>> >>> >>> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >>> make gnumake >>> Building PETSc using GNU Make with 10 build threads >>> ========================================== >>> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>> make[1]: Nothing to be done for `all'. >>> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>> ========================================= >>> >>> >>> Then I recompiled the codes, ran with -fp_trap and still got the following error >>> >>> Backtrace for this error: >>> Note: The EXACT line numbers in the stack are not available, >>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>> [2]PETSC ERROR: is given. >>> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>> INSTEAD the line number of the start of the function >>> [3]PETSC ERROR: is given. >>> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>> [2]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>> [3]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>> >>> >>> This is different then what you sent a few minutes ago where it crashed in hypre. >>> >>> Anyways you need to use the -start_in_debugger business I sent in the previous email to see the exact place the problem occurs. >>> >>> Here is the information shown on gdb screen >>> >>> Program received signal SIGFPE, Arithmetic exception. >>> 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, njamxc=34, >>> cinfradx=..., radial_coordx=.FALSE., _suffix=3) >>> at ../../gas_advection/velocity_g.F90:1344 >>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>> (gdb) >>> >>> After type cont on gdb screen, I got error information as below >>> >>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>> [1]PETSC ERROR: Try option -start_in_debugger >>> [1]PETSC ERROR: likely location of problem given in stack below >>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>> [1]PETSC ERROR: is given. >>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>> [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>> [0]PETSC ERROR: Try option -start_in_debugger >>> [0]PETSC ERROR: likely location of problem given in stack below >>> >>> Thanks, >>> >>> Danyang >>> Thanks, >>> >>> Danyang >>> >>> On Apr 25, 2015, at 1:05 AM, Danyang Su >>> wrote: >>> >>> Hi Barry and Satish, >>> >>> How can I get rid of unknown floating point error when a very small value is multiplied. >>> >>> e.g., >>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, >>> >>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>> >>> I get the following error when run with "-fp_trap -start_in_debugger". >>> >>> Backtrace for this error: >>> *** unknown floating point error occurred *** >>> [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>> [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 >>> where the result is a bitwise OR of the following flags: >>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>> [2]PETSC ERROR: Try option -start_in_debugger >>> [2]PETSC ERROR: likely location of problem given in stack below >>> >>> Thanks, >>> >>> Danyang >>> >>> On 15-04-24 01:54 PM, Danyang Su wrote: >>> >>> On 15-04-24 01:23 PM, Satish Balay wrote: >>> >>> c 4 1.0976214263087059E-067 >>> >>> I don't think this number can be stored in a real*4. >>> >>> Satish >>> >>> Thanks, Satish. It is caused by this number. >>> >>> On Fri, 24 Apr 2015, Danyang Su wrote: >>> >>> >>> On 15-04-24 11:12 AM, Barry Smith wrote: >>> >>> On Apr 24, 2015, at 1:05 PM, Danyang Su >>> wrote: >>> >>> Hi All, >>> >>> One of my case crashes because of floating point exception when using 4 >>> processors, as shown below. But if I run this case with 1 processor, it >>> works fine. I have tested the codes with around 100 cases up to 768 >>> processors, all other cases work fine. I just wonder if this kind of error >>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>> >>> Yes, almost for sure it is one of these places. >>> >>> First run the bad case with -fp_trap if all goes well you'll see the >>> function where the FPE is generated. Then run also with -start_in_debugger >>> and >>> type cont in all four debugger windows. When the FPE happens the debugger >>> should stop showing exactly where the FPE happens. >>> >>> Barry >>> >>> Hi Barry, >>> >>> If run with -fp_trap -start_in_debugger, I got the following error >>> >>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>> [0]PETSC ERROR: The specific exception can be determined by running in a >>> debugger. When the >>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >>> fetestexcept(0x3d) >>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>> [0]PETSC ERROR: Try option -start_in_debugger >>> [0]PETSC ERROR: likely location of problem given in stack below >>> [0]PETSC ERROR: --------------------- Stack Frames >>> ------------------------------------ >>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>> [0]PETSC ERROR: is given. >>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>> floating point error >>> >>> Program received signal SIGABRT: Process abort signal. >>> >>> Backtrace for this error: >>> #0 0x7F4FEAB1C7D7 >>> #1 0x7F4FEAB1CDDE >>> #2 0x7F4FE9E1AD3F >>> #3 0x7F4FE9E1ACC9 >>> #4 0x7F4FE9E1E0D7 >>> #5 0x7F4FEB0B6DCB >>> #6 0x7F4FEB0B1825 >>> #7 0x7F4FEB0B817F >>> #8 0x7F4FE9E1AD3F >>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>> >>> I checked the code at tprfrtlc.F90:2393, >>> >>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>> (cx(ix),ix=1,nxout)/) >>> >>> All the values (time, c, cx) are reasonable, as shown below. The only >>> possibility is that realbuffer_gb is in declared as real*4 if using sing >>> precision output while time, c, cx are declared in real*8. I have a lot of >>> similar data conversion from real*8 to real*4 output, other code does not >>> return error. >>> >>> time 0.0000000000000000 >>> c 1 9.9999999999999995E-008 >>> c 2 3.1555251077549618E-003 >>> c 3 7.1657814842179362E-008 >>> c 4 1.0976214263087059E-067 >>> c 5 5.2879822292305797E-004 >>> c 6 9.9999999999999964E-005 >>> c 7 6.4055731968811337E-005 >>> c 8 3.4607572892578404E-020 >>> cx 1 3.4376650636008101E-005 >>> cx 2 7.3989678854017763E-012 >>> cx 3 9.5317170613607207E-012 >>> cx 4 2.2344525794718353E-015 >>> cx 5 3.0624685689695889E-008 >>> cx 6 1.0046157902783967E-007 >>> cx 7 1.5320169154914984E-004 >>> cx 8 8.6930292776346176E-014 >>> cx 9 3.5944267559348721E-005 >>> cx 10 3.0072645866951157E-018 >>> cx 11 2.3592486321095017E-013 >>> >>> Thanks, >>> >>> Danyang >>> >>> >>> I can check all the entries of jacobi matrix to see if the value is valid, >>> but this seems not a good idea as it takes a long time to reach this >>> point. If I restart the simulation from a specified time (e.g., 7.685 in >>> this case), then the error does not occur. >>> >>> Would you please give me any suggestion on debugging this case? >>> >>> Thanks and Regards, >>> >>> Danyang >>> >>> >>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> Reduce time step for reactive transport >>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> Reduce time step for reactive transport >>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> [1]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [1]PETSC ERROR: Floating point exception >>> [2]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [2]PETSC ERROR: Floating point exception >>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >>> at end of function: Parameter number 3 >>> [2]PETSC ERROR: See >>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>> >>> for trouble shooting. >>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>> not-a-number or infinite at end of function: Parameter number 3 >>> [1]PETSC ERROR: See >>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>> >>> for trouble shooting. >>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>> dsu Thu Apr 23 15:38:52 2015 >>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>> --download-parmetis --download-scalapack >>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >>> 15:38:52 2015 >>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>> --download-parmetis --download-scalapack >>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>> [1]PETSC ERROR: #2 PCApply() line 442 in >>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>> #3 KSP_PCApply() line 230 in >>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>> >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >> From jychang48 at gmail.com Sun Apr 26 18:49:17 2015 From: jychang48 at gmail.com (Justin Chang) Date: Sun, 26 Apr 2015 18:49:17 -0500 Subject: [petsc-users] cached values Message-ID: Hello everyone, Pardon me if this might be a PETSc 101 question, but what exactly is meant when routines like MatSetValues() "caches the values"? Specifically, what happens say, after I iterate through each element within the ComputeJacobian() routine, when I call MatAssemblyBegin/End(). If there's documentation or a paper somewhere that explains this that would be very helpful, because I can't seem to find anything in the PETSc manual describing this process in detail. Thanks, Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Sun Apr 26 18:54:39 2015 From: jed at jedbrown.org (Jed Brown) Date: Sun, 26 Apr 2015 17:54:39 -0600 Subject: [petsc-users] cached values In-Reply-To: References: Message-ID: <87d22qv41s.fsf@jedbrown.org> Justin Chang writes: > Hello everyone, > > Pardon me if this might be a PETSc 101 question, but what exactly is meant > when routines like MatSetValues() "caches the values"? Specifically, what > happens say, after I iterate through each element within the > ComputeJacobian() routine, when I call MatAssemblyBegin/End(). If there's > documentation or a paper somewhere that explains this that would be very > helpful, because I can't seem to find anything in the PETSc manual > describing this process in detail. The exact protocol probably isn't interesting unless you're trying to develop something similar, in which case you can read the source code. The basic idea is that there is a rendezvous so that each process can determine from which other processes it will need to receive values, followed by actually sending/receiving those values. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jychang48 at gmail.com Sun Apr 26 19:00:54 2015 From: jychang48 at gmail.com (Justin Chang) Date: Sun, 26 Apr 2015 19:00:54 -0500 Subject: [petsc-users] cached values In-Reply-To: <87d22qv41s.fsf@jedbrown.org> References: <87d22qv41s.fsf@jedbrown.org> Message-ID: Jed, Thank you for the response. But where exactly are the values from MatSetValues() stored or located? That is, are they in main memory, or are they occupying space in the L1/L2/L3 cache? Thanks, Justin On Sun, Apr 26, 2015 at 6:54 PM, Jed Brown wrote: > Justin Chang writes: > > > Hello everyone, > > > > Pardon me if this might be a PETSc 101 question, but what exactly is > meant > > when routines like MatSetValues() "caches the values"? Specifically, what > > happens say, after I iterate through each element within the > > ComputeJacobian() routine, when I call MatAssemblyBegin/End(). If there's > > documentation or a paper somewhere that explains this that would be very > > helpful, because I can't seem to find anything in the PETSc manual > > describing this process in detail. > > The exact protocol probably isn't interesting unless you're trying to > develop something similar, in which case you can read the source code. > The basic idea is that there is a rendezvous so that each process can > determine from which other processes it will need to receive values, > followed by actually sending/receiving those values. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Sun Apr 26 19:13:04 2015 From: jed at jedbrown.org (Jed Brown) Date: Sun, 26 Apr 2015 18:13:04 -0600 Subject: [petsc-users] cached values In-Reply-To: References: <87d22qv41s.fsf@jedbrown.org> Message-ID: <87a8xuv373.fsf@jedbrown.org> Justin Chang writes: > Jed, > > Thank you for the response. But where exactly are the values from > MatSetValues() stored or located? That is, are they in main memory, or are > they occupying space in the L1/L2/L3 cache? Caches are not programmable in that way. PETSc incrementally allocates a data structure that holds the "stashed" values. It depends on the details of your application and the hardware cache semantics whether a particular access will trigger a cache miss or not. You can find out using hardware performance counters, but it's not likely to be worth your time unless you just want to learn those tools. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From patrick.sanan at gmail.com Sun Apr 26 19:29:23 2015 From: patrick.sanan at gmail.com (Patrick Sanan) Date: Sun, 26 Apr 2015 19:29:23 -0500 Subject: [petsc-users] cached values In-Reply-To: <87a8xuv373.fsf@jedbrown.org> References: <87d22qv41s.fsf@jedbrown.org> <87a8xuv373.fsf@jedbrown.org> Message-ID: This might be more basic than what you're trying to find out, but the reason that "assembly" is required in the first place (even for serial code) is that that during the process of defining the matrix, it's useful to be able to insert values anywhere in the matrix. I'm not sure what PETSc does precisely with digging into the source, but a common way to approach things is to maintain a data structure which looks like a C array of (row,col,val) entries (if you have ever used the Eigen library, they call these "Triples" and you can work with C++ STL vector of them). This is the COO ("COOrdinate list") format for storing a sparse matrix. Typically one uses a more highly-compressed format, like CSR ("Compressed Row Storage") , which does not allow of efficient insertion and deletion of entries. This tension motivates requiring the user to explicitly assemble the matrix. On Sun, Apr 26, 2015 at 7:13 PM, Jed Brown wrote: > Justin Chang writes: > > > Jed, > > > > Thank you for the response. But where exactly are the values from > > MatSetValues() stored or located? That is, are they in main memory, or > are > > they occupying space in the L1/L2/L3 cache? > > Caches are not programmable in that way. PETSc incrementally allocates > a data structure that holds the "stashed" values. It depends on the > details of your application and the hardware cache semantics whether a > particular access will trigger a cache miss or not. You can find out > using hardware performance counters, but it's not likely to be worth > your time unless you just want to learn those tools. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Apr 26 20:58:27 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 26 Apr 2015 20:58:27 -0500 Subject: [petsc-users] cached values In-Reply-To: References: Message-ID: <3573293D-F4CC-442E-91AF-64FF300F1C9B@mcs.anl.gov> The word cache here is not directly related to computer hardware caches. It comes from the meaning "store away in hiding or for future use." We also use the term stash "store (something) safely and secretly in a specified place". It is used to mean store temporarily (out of sight or knowledge) until we build the final matrix/vector data structure. For example if one sets a value into a matrix on one process that actually belongs on a different process we "stash it away" until the MatAssemblyBegin/End() when we move the data to the correct process and put it into its final resting place". As Patrick notes sometimes you may even want to stash values that that will eventually be stored locally. Barry The reason we use stash or cache with the "secret or hiding" status is that once the values are stashed the user doesn't have direct access to them at that time, it is only after the MatAssemblyBegin/End that you get that direct access again. > On Apr 26, 2015, at 6:49 PM, Justin Chang wrote: > > Hello everyone, > > Pardon me if this might be a PETSc 101 question, but what exactly is meant when routines like MatSetValues() "caches the values"? Specifically, what happens say, after I iterate through each element within the ComputeJacobian() routine, when I call MatAssemblyBegin/End(). If there's documentation or a paper somewhere that explains this that would be very helpful, because I can't seem to find anything in the PETSc manual describing this process in detail. > > Thanks, > Justin From jychang48 at gmail.com Sun Apr 26 21:02:47 2015 From: jychang48 at gmail.com (Justin Chang) Date: Sun, 26 Apr 2015 21:02:47 -0500 Subject: [petsc-users] cached values In-Reply-To: <3573293D-F4CC-442E-91AF-64FF300F1C9B@mcs.anl.gov> References: <3573293D-F4CC-442E-91AF-64FF300F1C9B@mcs.anl.gov> Message-ID: Okay that makes a lot of sense, I was confused because I thought the word cache referred to computer hardware caches. Thanks everyone for your input On Sun, Apr 26, 2015 at 8:58 PM, Barry Smith wrote: > > The word cache here is not directly related to computer hardware caches. > It comes from the meaning "store away in hiding or for future use." We also > use the term stash "store (something) safely and secretly in a specified > place". It is used to mean store temporarily (out of sight or knowledge) > until we build the final matrix/vector data structure. For example if one > sets a value into a matrix on one process that actually belongs on a > different process we "stash it away" until the MatAssemblyBegin/End() when > we move the data to the correct process and put it into its final resting > place". As Patrick notes sometimes you may even want to stash values that > that will eventually be stored locally. > > Barry > > The reason we use stash or cache with the "secret or hiding" status is > that once the values are stashed the user doesn't have direct access to > them at that time, it is only after the MatAssemblyBegin/End that you get > that direct access again. > > > > On Apr 26, 2015, at 6:49 PM, Justin Chang wrote: > > > > Hello everyone, > > > > Pardon me if this might be a PETSc 101 question, but what exactly is > meant when routines like MatSetValues() "caches the values"? Specifically, > what happens say, after I iterate through each element within the > ComputeJacobian() routine, when I call MatAssemblyBegin/End(). If there's > documentation or a paper somewhere that explains this that would be very > helpful, because I can't seem to find anything in the PETSc manual > describing this process in detail. > > > > Thanks, > > Justin > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fd.kong at siat.ac.cn Sun Apr 26 23:35:57 2015 From: fd.kong at siat.ac.cn (Fande Kong) Date: Sun, 26 Apr 2015 22:35:57 -0600 Subject: [petsc-users] matpartitioning metis Message-ID: Hi all, I was wondering if we have an implementation of matpartitioning using metis? There are any parameters to turn on sequential version of ParMETIS mat partitioning? Or just use petsc_comm_self as a comm? Thanks, Fande, -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailinglists at xgm.de Mon Apr 27 08:56:29 2015 From: mailinglists at xgm.de (Florian Lindner) Date: Mon, 27 Apr 2015 15:56:29 +0200 Subject: [petsc-users] SBAIJ: Distribution of rows Message-ID: <3568481.fPN5aupu15@asaru> Hello, just to make sure I don't get something wrong: An SBAIJ (especially MPISBAIJ) matrix just stores the uper triangle. When I invoke MatSetSizes with the global dimension set (and local to PETSC_DECIDE) petsc tries to evenly distribute rows amoung ranks. Same when I set local dimensions to the same on every processor. This leads to unbalanced behavior regarding the number of stored elements on each rank. The first row (on the first rank) is completely stored, the last row (on the last rank) contains just one element. Is that correct? I'm not trying to change / balance that, just to confirm it. ;-) Thanks, Florian From Carol.Brickley at awe.co.uk Mon Apr 27 09:28:27 2015 From: Carol.Brickley at awe.co.uk (Carol.Brickley at awe.co.uk) Date: Mon, 27 Apr 2015 14:28:27 +0000 Subject: [petsc-users] =?windows-1252?q?=93PETSC_ERROR=3A_Unable_to_find_r?= =?windows-1252?q?equested_PCtype_hypre=94?= Message-ID: <201504271428.t3RESWQZ011456@msw2.awe.co.uk> Hi, I am trying to run an executable built with petsc 3.5.3 and hypre 2.9.0b with flags ?-pc_type hypre ?ksp_type cg? but keep getting the error ?PETSC ERROR: Unable to find requested PCtype hypre? Any ideas why this might be happening? Thanks Carol Dr Carol Brickley BSc,PhD,ARCS,DIC,MBCS Senior Software Engineer Applied Computer Science DS+T, AWE Aldermaston Reading Berkshire RG7 4PR Direct: 0118 9855035 ___________________________________________________ ____________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR From hzhang at mcs.anl.gov Mon Apr 27 09:49:10 2015 From: hzhang at mcs.anl.gov (Hong) Date: Mon, 27 Apr 2015 09:49:10 -0500 Subject: [petsc-users] =?utf-8?q?=E2=80=9CPETSC_ERROR=3A_Unable_to_find_re?= =?utf-8?q?quested_PCtype_hypre=E2=80=9D?= In-Reply-To: <201504271428.t3RESWQZ011456@msw2.awe.co.uk> References: <201504271428.t3RESWQZ011456@msw2.awe.co.uk> Message-ID: Carol, Have you built your petsc with hypre? You can use ''--download-hypre' during petsc configuration. Hong On Mon, Apr 27, 2015 at 9:28 AM, wrote: > Hi, > > I am trying to run an executable built with petsc 3.5.3 and hypre 2.9.0b > with flags ?-pc_type hypre ?ksp_type cg? but keep getting the error > ?PETSC ERROR: Unable to find requested PCtype hypre? > > Any ideas why this might be happening? > > Thanks > > Carol > > > Dr Carol Brickley > BSc,PhD,ARCS,DIC,MBCS > > Senior Software Engineer > Applied Computer Science > DS+T, > AWE > Aldermaston > Reading > Berkshire > RG7 4PR > > Direct: 0118 9855035 > > > > ___________________________________________________ > ____________________________ > > The information in this email and in any attachment(s) is > commercial in confidence. If you are not the named addressee(s) > or > if you receive this email in error then any distribution, copying or > use of this communication or the information in it is strictly > prohibited. Please notify us immediately by email at > admin.internet(at)awe.co.uk, and then delete this message from > your computer. While attachments are virus checked, AWE plc > does not accept any liability in respect of any virus which is not > detected. > > AWE Plc > Registered in England and Wales > Registration No 02763902 > AWE, Aldermaston, Reading, RG7 4PR > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Mon Apr 27 09:54:18 2015 From: hzhang at mcs.anl.gov (Hong) Date: Mon, 27 Apr 2015 09:54:18 -0500 Subject: [petsc-users] SBAIJ: Distribution of rows In-Reply-To: <3568481.fPN5aupu15@asaru> References: <3568481.fPN5aupu15@asaru> Message-ID: Florian: > > > just to make sure I don't get something wrong: > > An SBAIJ (especially MPISBAIJ) matrix just stores the uper triangle. When > I invoke MatSetSizes with the global dimension set (and local to > PETSC_DECIDE) petsc tries to evenly distribute rows amoung ranks. Same when > I set local dimensions to the same on every processor. > Yes. > > This leads to unbalanced behavior regarding the number of stored elements > on each rank. The first row (on the first rank) is completely stored, the > last row (on the last rank) contains just one element. > Not necessarily, e.g., for a sparse banded matrix, this distribution is well-balanced. User can set their own distribution by input local rows and set global rows as PETSC_DECIDE. Hong -------------- next part -------------- An HTML attachment was scrubbed... URL: From Carol.Brickley at awe.co.uk Mon Apr 27 10:17:47 2015 From: Carol.Brickley at awe.co.uk (Carol.Brickley at awe.co.uk) Date: Mon, 27 Apr 2015 15:17:47 +0000 Subject: [petsc-users] =?windows-1252?q?EXTERNAL=3A_Re=3A__=93PETSC_ERROR?= =?windows-1252?q?=3A_Unable_to_find_requested_PCtype_hypre=94?= In-Reply-To: References: <201504271428.t3RESWQZ011456@msw2.awe.co.uk> Message-ID: <201504271517.t3RFHpJf032668@msw1.awe.co.uk> Hi, I thought I had but looking at the log summary it isn?t coming up that petsc was built properly with hypre. My script had a bug which caused it to ignore everything after the blas library entries. Carol Dr Carol Brickley BSc,PhD,ARCS,DIC,MBCS Senior Software Engineer Applied Computer Science DS+T, AWE Aldermaston Reading Berkshire RG7 4PR Direct: 0118 9855035 ________________________________ From: Hong [mailto:hzhang at mcs.anl.gov] Sent: 27 April 2015 15:49 To: Brickley Carol AWE Cc: PETSc users list Subject: EXTERNAL: Re: [petsc-users] ?PETSC ERROR: Unable to find requested PCtype hypre? Carol, Have you built your petsc with hypre? You can use ''--download-hypre' during petsc configuration. Hong On Mon, Apr 27, 2015 at 9:28 AM, > wrote: Hi, I am trying to run an executable built with petsc 3.5.3 and hypre 2.9.0b with flags ?-pc_type hypre ?ksp_type cg? but keep getting the error ?PETSC ERROR: Unable to find requested PCtype hypre? Any ideas why this might be happening? Thanks Carol Dr Carol Brickley BSc,PhD,ARCS,DIC,MBCS Senior Software Engineer Applied Computer Science DS+T, AWE Aldermaston Reading Berkshire RG7 4PR Direct: 0118 9855035 ___________________________________________________ ____________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR ___________________________________________________ ____________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR -------------- next part -------------- An HTML attachment was scrubbed... URL: From mlohry at princeton.edu Mon Apr 27 10:50:45 2015 From: mlohry at princeton.edu (Mark Lohry) Date: Mon, 27 Apr 2015 11:50:45 -0400 Subject: [petsc-users] Turning off TSADAPT still adapts time step In-Reply-To: References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AA4B1.5010603@princeton.edu> <553AB0C6.5010503@princeton.edu> Message-ID: <553E5AD5.1060208@princeton.edu> The TSAdaptSetType is being called. Looking at TSStep_Theta, is it possible that here: /* Register only the current method as a candidate because we're not supporting multiple candidates yet. */ ierr = TSGetAdapt(ts,&adapt);CHKERRQ(ierr); ierr = TSAdaptCandidatesClear(adapt);CHKERRQ(ierr); ierr = TSAdaptCandidateAdd(adapt,NULL,th->order,1,th->ccfl,1.0,PETSC_TRUE);CHKERRQ(ierr); ierr = TSAdaptChoose(adapt,ts,ts->time_step,&next_scheme,&next_time_step,&accept);CHKERRQ(ierr); TSAdaptCandidatesClear is unsetting whatever method I'm setting? On 04/24/2015 06:41 PM, Barry Smith wrote: >> On Apr 24, 2015, at 4:08 PM, Mark Lohry wrote: >> >> I'm new to PETSc and not sure how to go about debugging. Running with -start_in_debugger runs in gdb with the normal output (where would i put a breakpoint for this?) > You could start by putting a break point in TSAdaptSetType() and see if your call is made and then if any call is made after that by other code changing it to something else. You could put a breakpoint in TSStep_Theta (which supports beuler) and "next" through that to see why it is using an adapter that actually does something. > > Barry > > > >> and running with -info dumps a lot of information but only related to SNES and KSP, nothing about time stepping or adapt. >> >> >> >> On 04/24/2015 04:29 PM, Barry Smith wrote: >>> Run in the debugger (with one process obviously) to see why it is still calling an adaptor. >>> >>> Barry >>> >>>> On Apr 24, 2015, at 3:16 PM, Mark Lohry wrote: >>>> >>>> Trying a simple stiff test case (y'=y^2*(1-y)) using TSBEULER, to get a feel for implicit stepping with JFNK. >>>> >>>> I'm giving it a pretty large timestep, and the default appears to start adapting the time step around the stiff part of the solution. So I tried turning it off: >>>> >>>> TSAdapt adapt; >>>> TSGetAdapt(its,&adapt); >>>> TSAdaptSetType(adapt,TSADAPTNONE); >>>> >>>> and even fixing the limits: >>>> >>>> TSAdaptSetStepLimits(adapt,40.0,40.0); >>>> >>>> And it still ignores it and adapts anyway. At t=800 (where the solution gets very stiff, so one would normally see adaptive stepping there if it was desired) it's dropping dt from 40 to 10. >>>> >>>> Printing (t,y) here from the ts monitor: >>>> >>>> 600.000000,0.002790; >>>> 640.000000,0.003198; >>>> 680.000000,0.003761; >>>> 720.000000,0.004606; >>>> 760.000000,0.006072; >>>> 800.000000,0.010155; >>>> 810.000000,0.011452; >>>> 820.000000,0.013161; >>>> 830.000000,0.015538; >>>> 840.000000,0.019126; >>>> 850.000000,0.025426; >>>> 860.000000,0.043639; >>>> 900.000000,0.975519; >>>> 940.000000,0.999402; >>>> 980.000000,0.999985; >>>> 1020.000000,1.000000; >>>> 1060.000000,1.000000; >>>> 1100.000000,1.000000; >>>> 1140.000000,1.000000; >>>> 1180.000000,1.000000; >>>> >>>> Any idea? Adaptive stepping is great, but I'd still like to be able to fix the time step. >>>> >>>> >>>> From jed at jedbrown.org Mon Apr 27 10:58:35 2015 From: jed at jedbrown.org (Jed Brown) Date: Mon, 27 Apr 2015 09:58:35 -0600 Subject: [petsc-users] Turning off TSADAPT still adapts time step In-Reply-To: <553E5AD5.1060208@princeton.edu> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AA4B1.5010603@princeton.edu> <553AB0C6.5010503@princeton.edu> <553E5AD5.1060208@princeton.edu> Message-ID: <87383ltvf8.fsf@jedbrown.org> Mark Lohry writes: > The TSAdaptSetType is being called. Looking at TSStep_Theta, is it > possible that here: > > /* Register only the current method as a candidate because we're not > supporting multiple candidates yet. */ > ierr = TSGetAdapt(ts,&adapt);CHKERRQ(ierr); > ierr = TSAdaptCandidatesClear(adapt);CHKERRQ(ierr); > ierr = > TSAdaptCandidateAdd(adapt,NULL,th->order,1,th->ccfl,1.0,PETSC_TRUE);CHKERRQ(ierr); > ierr = > TSAdaptChoose(adapt,ts,ts->time_step,&next_scheme,&next_time_step,&accept);CHKERRQ(ierr); > > TSAdaptCandidatesClear is unsetting whatever method I'm setting? No. It does not affect the implementation, it just clears the candidate list that the implementation may choose from. PetscErrorCode TSAdaptCandidatesClear(TSAdapt adapt) { PetscErrorCode ierr; PetscFunctionBegin; PetscValidHeaderSpecific(adapt,TSADAPT_CLASSID,1); ierr = PetscMemzero(&adapt->candidates,sizeof(adapt->candidates));CHKERRQ(ierr); PetscFunctionReturn(0); } When you set -ts_adapt_type none, the function below should be called. Verify that in a debugger. Also, always run with -ts_adapt_monitor when trying to debug something like this. static PetscErrorCode TSAdaptChoose_None(TSAdapt adapt,TS ts,PetscReal h,PetscInt *next_sc,PetscReal *next_h,PetscBool *accept,PetscReal *wlte) { PetscFunctionBegin; *accept = PETSC_TRUE; *next_sc = 0; /* Reuse the same order scheme */ *next_h = h; /* Reuse the old step */ *wlte = -1; /* Weighted local truncation error was not evaluated */ PetscFunctionReturn(0); } -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From mlohry at princeton.edu Mon Apr 27 11:12:47 2015 From: mlohry at princeton.edu (Mark Lohry) Date: Mon, 27 Apr 2015 12:12:47 -0400 Subject: [petsc-users] Turning off TSADAPT still adapts time step In-Reply-To: <87383ltvf8.fsf@jedbrown.org> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AA4B1.5010603@princeton.edu> <553AB0C6.5010503@princeton.edu> <553E5AD5.1060208@princeton.edu> <87383ltvf8.fsf@jedbrown.org> Message-ID: <553E5FFF.6080905@princeton.edu> TSAdaptChoose_None is being called, yes. Output with -ts_adapt monitor looks like this: TSAdapt 'none': step 17 accepted t=680 + 4.000e+01 family='beuler' scheme=0:'(null)' dt=4.000e+01 TSAdapt 'none': step 18 accepted t=720 + 4.000e+01 family='beuler' scheme=0:'(null)' dt=4.000e+01 TSAdapt 'none': step 19 accepted t=760 + 4.000e+01 family='beuler' scheme=0:'(null)' dt=4.000e+01 TSAdapt 'none': step 20 stage rejected t=800 + 4.000e+01 retrying with dt=1.000e+01 TSAdapt 'none': step 20 accepted t=800 + 1.000e+01 family='beuler' scheme=0:'(null)' dt=1.000e+01 TSAdapt 'none': step 21 stage rejected t=810 + 4.000e+01 retrying with dt=1.000e+01 TSAdapt 'none': step 21 accepted t=810 + 1.000e+01 family='beuler' scheme=0:'(null)' dt=1.000e+01 TSAdapt 'none': step 22 stage rejected t=820 + 4.000e+01 retrying with dt=1.000e+01 TSAdapt 'none': step 22 accepted t=820 + 1.000e+01 family='beuler' scheme=0:'(null)' dt=1.000e+01 TSAdapt 'none': step 23 stage rejected t=830 + 4.000e+01 retrying with dt=1.000e+01 TSAdapt 'none': step 23 accepted t=830 + 1.000e+01 family='beuler' scheme=0:'(null)' dt=1.000e+01 I would have expected with "none" that no stages would be rejected, at the very least dt would be constant. Is there an error tolerance somewhere I need to set to force it to accept all stages? On 04/27/2015 11:58 AM, Jed Brown wrote: > Mark Lohry writes: > >> The TSAdaptSetType is being called. Looking at TSStep_Theta, is it >> possible that here: >> >> /* Register only the current method as a candidate because we're not >> supporting multiple candidates yet. */ >> ierr = TSGetAdapt(ts,&adapt);CHKERRQ(ierr); >> ierr = TSAdaptCandidatesClear(adapt);CHKERRQ(ierr); >> ierr = >> TSAdaptCandidateAdd(adapt,NULL,th->order,1,th->ccfl,1.0,PETSC_TRUE);CHKERRQ(ierr); >> ierr = >> TSAdaptChoose(adapt,ts,ts->time_step,&next_scheme,&next_time_step,&accept);CHKERRQ(ierr); >> >> TSAdaptCandidatesClear is unsetting whatever method I'm setting? > No. It does not affect the implementation, it just clears the candidate > list that the implementation may choose from. > > PetscErrorCode TSAdaptCandidatesClear(TSAdapt adapt) > { > PetscErrorCode ierr; > > PetscFunctionBegin; > PetscValidHeaderSpecific(adapt,TSADAPT_CLASSID,1); > ierr = PetscMemzero(&adapt->candidates,sizeof(adapt->candidates));CHKERRQ(ierr); > PetscFunctionReturn(0); > } > > > When you set -ts_adapt_type none, the function below should be called. > Verify that in a debugger. Also, always run with -ts_adapt_monitor when > trying to debug something like this. > > static PetscErrorCode TSAdaptChoose_None(TSAdapt adapt,TS ts,PetscReal h,PetscInt *next_sc,PetscReal *next_h,PetscBool *accept,PetscReal *wlte) > { > > PetscFunctionBegin; > *accept = PETSC_TRUE; > *next_sc = 0; /* Reuse the same order scheme */ > *next_h = h; /* Reuse the old step */ > *wlte = -1; /* Weighted local truncation error was not evaluated */ > PetscFunctionReturn(0); > } From danyang.su at gmail.com Mon Apr 27 11:44:32 2015 From: danyang.su at gmail.com (Danyang Su) Date: Mon, 27 Apr 2015 09:44:32 -0700 Subject: [petsc-users] Floating point exception In-Reply-To: <553C4ECD.6090905@gmail.com> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> <553C3E15.8030403@gmail.com> <553C3FE6.5020203@gmail.com> <553C4ECD.6090905@gmail.com> Message-ID: <553E6770.5010205@gmail.com> Hi Barry, I got the following arithemetic exception after the previous bug is fixed. Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 0x00007f3b23b98f20 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:81 81 ../sysdeps/unix/syscall-template.S: No such file or directory. (gdb) cont Continuing. Program received signal SIGFPE, Arithmetic exception. 0x00007f3b260df449 in gselim (A=0x1e4c580, x=0x1d30c40, n=9) at par_relax.c:3442 3442 x[k] /= A[k*n+k]; (gdb) I tried both PETSc 3.5.2 and 3.5.3 and they return the same error as shown above. For 3.5.3, i edited fp.c file and then configure and make. Thanks, Danyang On 15-04-25 07:34 PM, Danyang Su wrote: > Hi All, > > The "floating point underflow" is caused by a small value divided by a > very large value. This result is forced to zero and then it does not > report any underflow problem. I just rerun this bad case to see if it > still get stuck later. This will take a while. > > Thanks for all your kindly reply, > > Danyang > > On 15-04-25 07:02 PM, Barry Smith wrote: >> Ok, you do have >> >> #ifndef PETSC_HAVE_XMMINTRIN_H >> #define PETSC_HAVE_XMMINTRIN_H 1 >> #endif >> >> so the change you made should cause it to stop trapping underflow >> exceptions. >> >> Now in one email you reported a FPE within hypre, then I asked you >> to run with -start_in_debugger to determine where it happened exactly >> and then you reported the FPE happened in user code (what seemed to >> be an underflow issue). Why is this? Can you not run it where it >> generated the FPE in hypre using the -start_in_debugger option? >> >> Barry >> >> Perhaps you have multiple PETSC_ARCH or multiple PETSc installs to >> explain why you reported two different places where the exception >> occurred. >> >>> On Apr 25, 2015, at 8:31 PM, Danyang Su wrote: >>> >>> >>> >>> On 15-04-25 06:26 PM, Matthew Knepley wrote: >>>> On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su >>>> wrote: >>>> >>>> >>>> On 15-04-25 06:03 PM, Barry Smith wrote: >>>> If this is what you got in your last run >>>> >>>> at ../../gas_advection/velocity_g.F90:1344 >>>> 1344 cinfrt = cinfrt_dg(i1) >>>> * diff(ic,idim) !diff is a >>>> very small value, e.g., 1.0d-316 >>>> then it is still catching floating point underflow, which we do >>>> not want. This means either the change I suggested you make in the >>>> fp.c code didn't work or it actually uses a different floating >>>> point trap than that one. BTW: absurd numbers like 1.0d-316 are >>>> often a symptom of uninitialized data; could that be a problem that >>>> diff is not filled correctly for all the ic, idim you are using? >>>> >>>> This going round and round is very frustrating and a waste of >>>> time. You need to be more proactive yourself and explore the code >>>> and poke around to figure out how to solve the problem. >>>> >>>> Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so >>>> I can see what FP trap is being used on your machine. >>>> >>>> Barry >>>> Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise I >>>> cannot find this file. >>>> >>>> Its include/petscconf.h >>>> Do I need to reconfigure PETSc after changing the code you >>>> mentioned? >>>> >>>> No, but you need to rebuild. >>> Yes, I have done 'gnumake'. >>>> Matt >>>> Danyang >>>> >>>> >>>> >>>> On Apr 25, 2015, at 2:24 PM, Danyang Su wrote: >>>> >>>> >>>> >>>> On 15-04-25 11:55 AM, Barry Smith wrote: >>>> On Apr 25, 2015, at 1:51 PM, Danyang Su >>>> wrote: >>>> >>>> >>>> >>>> On 15-04-25 11:32 AM, Barry Smith wrote: >>>> >>>> I told you this yesterday. >>>> >>>> It is probably stopping here on a harmless underflow. You need >>>> to edit the PETSc code to not worry about underflow. >>>> >>>> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>> and locate >>>> >>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>>> #else >>>> >>>> change it to >>>> >>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >>>> #else >>>> >>>> Then run make gnumake in the PETSc directory to compile the new >>>> version. Now link and run the program again with -fp_trap and see >>>> where it gets stuck this time. >>>> >>>> Did you do this? >>>> >>>> Barry >>>> >>>> Yes, I did change the code in fp.c and run 'make gnumake' in the >>>> PETSc directory. I just did a double check and ran make gnumake >>>> again and got the following information this time. >>>> >>>> >>>> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >>>> make gnumake >>>> Building PETSc using GNU Make with 10 build threads >>>> ========================================== >>>> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>> make[1]: Nothing to be done for `all'. >>>> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>> ========================================= >>>> >>>> >>>> Then I recompiled the codes, ran with -fp_trap and still got the >>>> following error >>>> >>>> Backtrace for this error: >>>> Note: The EXACT line numbers in the stack are not available, >>>> [2]PETSC ERROR: INSTEAD the line number of the start of the >>>> function >>>> [2]PETSC ERROR: is given. >>>> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>> INSTEAD the line number of the start of the function >>>> [3]PETSC ERROR: is given. >>>> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>> [2]PETSC ERROR: User provided function() line 0 in Unknown file >>>> trapped floating point error >>>> [3]PETSC ERROR: User provided function() line 0 in Unknown file >>>> trapped floating point error >>>> >>>> >>>> This is different then what you sent a few minutes ago where >>>> it crashed in hypre. >>>> >>>> Anyways you need to use the -start_in_debugger business I sent >>>> in the previous email to see the exact place the problem occurs. >>>> >>>> Here is the information shown on gdb screen >>>> >>>> Program received signal SIGFPE, Arithmetic exception. >>>> 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, >>>> njamxc=34, >>>> cinfradx=..., radial_coordx=.FALSE., _suffix=3) >>>> at ../../gas_advection/velocity_g.F90:1344 >>>> 1344 cinfrt = cinfrt_dg(i1) >>>> * diff(ic,idim) !diff is a >>>> very small value, e.g., 1.0d-316 >>>> (gdb) >>>> >>>> After type cont on gdb screen, I got error information as below >>>> >>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>> [1]PETSC ERROR: The specific exception can be determined by running >>>> in a debugger. When the >>>> [1]PETSC ERROR: debugger traps the signal, the exception can be >>>> found with fetestexcept(0x3d) >>>> [1]PETSC ERROR: where the result is a bitwise OR of the following >>>> flags: >>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>> [1]PETSC ERROR: Try option -start_in_debugger >>>> [1]PETSC ERROR: likely location of problem given in stack below >>>> [1]PETSC ERROR: --------------------- Stack Frames >>>> ------------------------------------ >>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>> available, >>>> [1]PETSC ERROR: INSTEAD the line number of the start of the >>>> function >>>> [1]PETSC ERROR: is given. >>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file >>>> trapped floating point error >>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>> [0]PETSC ERROR: The specific exception can be determined by running >>>> in a debugger. When the >>>> [0]PETSC ERROR: debugger traps the signal, the exception can be >>>> found with fetestexcept(0x3d) >>>> [0]PETSC ERROR: where the result is a bitwise OR of the following >>>> flags: >>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>> [0]PETSC ERROR: Try option -start_in_debugger >>>> [0]PETSC ERROR: likely location of problem given in stack below >>>> >>>> Thanks, >>>> >>>> Danyang >>>> Thanks, >>>> >>>> Danyang >>>> >>>> On Apr 25, 2015, at 1:05 AM, Danyang Su >>>> wrote: >>>> >>>> Hi Barry and Satish, >>>> >>>> How can I get rid of unknown floating point error when a very small >>>> value is multiplied. >>>> >>>> e.g., >>>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 >>>> 8.6178408169776945E-317 respectively, >>>> >>>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>>> >>>> I get the following error when run with "-fp_trap -start_in_debugger". >>>> >>>> Backtrace for this error: >>>> *** unknown floating point error occurred *** >>>> [2]PETSC ERROR: The specific exception can be determined by running >>>> in a debugger. When the >>>> [2]PETSC ERROR: debugger traps the signal, the exception can be >>>> found with fetestexcept(0x3d) >>>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) >>>> 1.0250235986806329E-008 8.6178408169776945E-317 >>>> where the result is a bitwise OR of the following flags: >>>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>> [2]PETSC ERROR: Try option -start_in_debugger >>>> [2]PETSC ERROR: likely location of problem given in stack below >>>> >>>> Thanks, >>>> >>>> Danyang >>>> >>>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>> >>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>> >>>> c 4 1.0976214263087059E-067 >>>> >>>> I don't think this number can be stored in a real*4. >>>> >>>> Satish >>>> >>>> Thanks, Satish. It is caused by this number. >>>> >>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>> >>>> >>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>> >>>> On Apr 24, 2015, at 1:05 PM, Danyang Su >>>> wrote: >>>> >>>> Hi All, >>>> >>>> One of my case crashes because of floating point exception when >>>> using 4 >>>> processors, as shown below. But if I run this case with 1 >>>> processor, it >>>> works fine. I have tested the codes with around 100 cases up to 768 >>>> processors, all other cases work fine. I just wonder if this kind >>>> of error >>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>> >>>> Yes, almost for sure it is one of these places. >>>> >>>> First run the bad case with -fp_trap if all goes well you'll >>>> see the >>>> function where the FPE is generated. Then run also with >>>> -start_in_debugger >>>> and >>>> type cont in all four debugger windows. When the FPE happens the >>>> debugger >>>> should stop showing exactly where the FPE happens. >>>> >>>> Barry >>>> >>>> Hi Barry, >>>> >>>> If run with -fp_trap -start_in_debugger, I got the following error >>>> >>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>> [0]PETSC ERROR: The specific exception can be determined by running >>>> in a >>>> debugger. When the >>>> [0]PETSC ERROR: debugger traps the signal, the exception can be >>>> found with >>>> fetestexcept(0x3d) >>>> [0]PETSC ERROR: where the result is a bitwise OR of the following >>>> flags: >>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>> [0]PETSC ERROR: Try option -start_in_debugger >>>> [0]PETSC ERROR: likely location of problem given in stack below >>>> [0]PETSC ERROR: --------------------- Stack Frames >>>> ------------------------------------ >>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>> available, >>>> [0]PETSC ERROR: INSTEAD the line number of the start of the >>>> function >>>> [0]PETSC ERROR: is given. >>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file >>>> trapped >>>> floating point error >>>> >>>> Program received signal SIGABRT: Process abort signal. >>>> >>>> Backtrace for this error: >>>> #0 0x7F4FEAB1C7D7 >>>> #1 0x7F4FEAB1CDDE >>>> #2 0x7F4FE9E1AD3F >>>> #3 0x7F4FE9E1ACC9 >>>> #4 0x7F4FE9E1E0D7 >>>> #5 0x7F4FEB0B6DCB >>>> #6 0x7F4FEB0B1825 >>>> #7 0x7F4FEB0B817F >>>> #8 0x7F4FE9E1AD3F >>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>> >>>> I checked the code at tprfrtlc.F90:2393, >>>> >>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>> (cx(ix),ix=1,nxout)/) >>>> >>>> All the values (time, c, cx) are reasonable, as shown below. The only >>>> possibility is that realbuffer_gb is in declared as real*4 if using >>>> sing >>>> precision output while time, c, cx are declared in real*8. I have a >>>> lot of >>>> similar data conversion from real*8 to real*4 output, other code >>>> does not >>>> return error. >>>> >>>> time 0.0000000000000000 >>>> c 1 9.9999999999999995E-008 >>>> c 2 3.1555251077549618E-003 >>>> c 3 7.1657814842179362E-008 >>>> c 4 1.0976214263087059E-067 >>>> c 5 5.2879822292305797E-004 >>>> c 6 9.9999999999999964E-005 >>>> c 7 6.4055731968811337E-005 >>>> c 8 3.4607572892578404E-020 >>>> cx 1 3.4376650636008101E-005 >>>> cx 2 7.3989678854017763E-012 >>>> cx 3 9.5317170613607207E-012 >>>> cx 4 2.2344525794718353E-015 >>>> cx 5 3.0624685689695889E-008 >>>> cx 6 1.0046157902783967E-007 >>>> cx 7 1.5320169154914984E-004 >>>> cx 8 8.6930292776346176E-014 >>>> cx 9 3.5944267559348721E-005 >>>> cx 10 3.0072645866951157E-018 >>>> cx 11 2.3592486321095017E-013 >>>> >>>> Thanks, >>>> >>>> Danyang >>>> >>>> >>>> I can check all the entries of jacobi matrix to see if the value is >>>> valid, >>>> but this seems not a good idea as it takes a long time to reach this >>>> point. If I restart the simulation from a specified time (e.g., >>>> 7.685 in >>>> this case), then the error does not occur. >>>> >>>> Would you please give me any suggestion on debugging this case? >>>> >>>> Thanks and Regards, >>>> >>>> Danyang >>>> >>>> >>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years >>>> iter: 1 >>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years >>>> iter: 1 >>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years >>>> iter: 1 >>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years >>>> iter: 1 >>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years >>>> iter: 1 >>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>> Reduce time step for reactive transport >>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years >>>> iter: 1 >>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>> Reduce time step for reactive transport >>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years >>>> iter: 1 >>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>> [1]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [1]PETSC ERROR: Floating point exception >>>> [2]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [2]PETSC ERROR: Floating point exception >>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or >>>> infinite >>>> at end of function: Parameter number 3 >>>> [2]PETSC ERROR: See >>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> >>>> for trouble shooting. >>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>> not-a-number or infinite at end of function: Parameter number 3 >>>> [1]PETSC ERROR: See >>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> >>>> for trouble shooting. >>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named >>>> nwmop by >>>> dsu Thu Apr 23 15:38:52 2015 >>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg >>>> --with-cc=gcc >>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack >>>> --download-mpich >>>> --download-mumps --download-hypre --download-superlu_dist >>>> --download-metis >>>> --download-parmetis --download-scalapack >>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu >>>> Apr 23 >>>> 15:38:52 2015 >>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg >>>> --with-cc=gcc >>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack >>>> --download-mpich >>>> --download-mumps --download-hypre --download-superlu_dist >>>> --download-metis >>>> --download-parmetis --download-scalapack >>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>> #3 KSP_PCApply() line 230 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to >>>> which their experiments lead. >>>> -- Norbert Wiener >>> > From jed at jedbrown.org Mon Apr 27 11:55:07 2015 From: jed at jedbrown.org (Jed Brown) Date: Mon, 27 Apr 2015 10:55:07 -0600 Subject: [petsc-users] Turning off TSADAPT still adapts time step In-Reply-To: <553E5FFF.6080905@princeton.edu> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AA4B1.5010603@princeton.edu> <553AB0C6.5010503@princeton.edu> <553E5AD5.1060208@princeton.edu> <87383ltvf8.fsf@jedbrown.org> <553E5FFF.6080905@princeton.edu> Message-ID: <87zj5tse8k.fsf@jedbrown.org> Mark Lohry writes: > TSAdaptChoose_None is being called, yes. Output with -ts_adapt monitor > looks like this: > > TSAdapt 'none': step 17 accepted t=680 + 4.000e+01 > family='beuler' scheme=0:'(null)' dt=4.000e+01 > TSAdapt 'none': step 18 accepted t=720 + 4.000e+01 > family='beuler' scheme=0:'(null)' dt=4.000e+01 > TSAdapt 'none': step 19 accepted t=760 + 4.000e+01 > family='beuler' scheme=0:'(null)' dt=4.000e+01 > TSAdapt 'none': step 20 stage rejected t=800 + 4.000e+01 > retrying with dt=1.000e+01 Add -snes_converged_reason. What do you want to do when the solver fails? Pretend like it succeeded and get the wrong answer? Usually people shorten the time step and retry, which is what you see happening here. You can call TSAdaptSetCheckStage and have your function unconditionally set accept to PETSC_TRUE (and you'll likely get the wrong answer). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From mlohry at princeton.edu Mon Apr 27 12:16:29 2015 From: mlohry at princeton.edu (Mark Lohry) Date: Mon, 27 Apr 2015 13:16:29 -0400 Subject: [petsc-users] Turning off TSADAPT still adapts time step In-Reply-To: <87zj5tse8k.fsf@jedbrown.org> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AA4B1.5010603@princeton.edu> <553AB0C6.5010503@princeton.edu> <553E5AD5.1060208@princeton.edu> <87383ltvf8.fsf@jedbrown.org> <553E5FFF.6080905@princeton.edu> <87zj5tse8k.fsf@jedbrown.org> Message-ID: <553E6EED.7000604@princeton.edu> -snes_converged_reason: Nonlinear solve did not converge due to DIVERGED_LOCAL_MIN iterations 4 (which is expected) > Add -snes_converged_reason. What do you want to do when the solver > fails? Pretend like it succeeded and get the wrong answer? Usually > people shorten the time step and retry, which is what you see happening > here. Well yes, actually - I expected it to fail for this test case but it stubbornly gets the right answer. The explicit integrator has no problem keeping a fixed timestep and throwing the expected NaNs when it blows up. It's alarming to see the unexpected behavior of setting TSADAPTNONE and seeing it actually adapting (in a rudimentary fashion) the timestep, and keeping that reduced timestep even after the stiffness goes away. > You can call TSAdaptSetCheckStage and have your function unconditionally > set accept to PETSC_TRUE (and you'll likely get the wrong answer). Thanks for the help! From jed at jedbrown.org Mon Apr 27 12:24:34 2015 From: jed at jedbrown.org (Jed Brown) Date: Mon, 27 Apr 2015 11:24:34 -0600 Subject: [petsc-users] Turning off TSADAPT still adapts time step In-Reply-To: <553E6EED.7000604@princeton.edu> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AA4B1.5010603@princeton.edu> <553AB0C6.5010503@princeton.edu> <553E5AD5.1060208@princeton.edu> <87383ltvf8.fsf@jedbrown.org> <553E5FFF.6080905@princeton.edu> <87zj5tse8k.fsf@jedbrown.org> <553E6EED.7000604@princeton.edu> Message-ID: <87wq0xscvh.fsf@jedbrown.org> Mark Lohry writes: > -snes_converged_reason: > > Nonlinear solve did not converge due to DIVERGED_LOCAL_MIN iterations 4 > > (which is expected) If you set -snes_stol $something_large then SNES will exit SNES_CONVERGED_SNORM_RELATIVE instead of SNES_DIVERGED_LINE_SEARCH when it stagnates. >> Add -snes_converged_reason. What do you want to do when the solver >> fails? Pretend like it succeeded and get the wrong answer? Usually >> people shorten the time step and retry, which is what you see happening >> here. > Well yes, actually - I expected it to fail for this test case but it > stubbornly gets the right answer. Sorry about that. ;-) > The explicit integrator has no problem keeping a fixed timestep and > throwing the expected NaNs when it blows up. > > It's alarming to see the unexpected behavior of setting TSADAPTNONE and > seeing it actually adapting (in a rudimentary fashion) the timestep, and > keeping that reduced timestep even after the stiffness goes away. We figured it's not very useful to just crash or get the wrong answer, so the solver-based control is decoupled from error-based control. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From knepley at gmail.com Mon Apr 27 13:34:06 2015 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 28 Apr 2015 04:34:06 +1000 Subject: [petsc-users] Floating point exception In-Reply-To: <553E6770.5010205@gmail.com> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> <553C3E15.8030403@gmail.com> <553C3FE6.5020203@gmail.com> <553C4ECD.6090905@gmail.com> <553E6770.5010205@gmail.com> Message-ID: On Tue, Apr 28, 2015 at 2:44 AM, Danyang Su wrote: > Hi Barry, > > I got the following arithemetic exception after the previous bug is fixed. > > Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 > 0x00007f3b23b98f20 in __nanosleep_nocancel () > at ../sysdeps/unix/syscall-template.S:81 > 81 ../sysdeps/unix/syscall-template.S: No such file or directory. > (gdb) cont > Continuing. > > Program received signal SIGFPE, Arithmetic exception. > 0x00007f3b260df449 in gselim (A=0x1e4c580, x=0x1d30c40, n=9) > at par_relax.c:3442 > 3442 x[k] /= A[k*n+k]; > (gdb) > > I tried both PETSc 3.5.2 and 3.5.3 and they return the same error as shown > above. For 3.5.3, i edited fp.c file and then configure and make. > It looks like there is a 0 on the diagonal of some matrix A. We cannot know which one without the stack trace. Can you send the whole trace? Matt > Thanks, > > Danyang > > On 15-04-25 07:34 PM, Danyang Su wrote: > >> Hi All, >> >> The "floating point underflow" is caused by a small value divided by a >> very large value. This result is forced to zero and then it does not report >> any underflow problem. I just rerun this bad case to see if it still get >> stuck later. This will take a while. >> >> Thanks for all your kindly reply, >> >> Danyang >> >> On 15-04-25 07:02 PM, Barry Smith wrote: >> >>> Ok, you do have >>> >>> #ifndef PETSC_HAVE_XMMINTRIN_H >>> #define PETSC_HAVE_XMMINTRIN_H 1 >>> #endif >>> >>> so the change you made should cause it to stop trapping underflow >>> exceptions. >>> >>> Now in one email you reported a FPE within hypre, then I asked you to >>> run with -start_in_debugger to determine where it happened exactly and then >>> you reported the FPE happened in user code (what seemed to be an underflow >>> issue). Why is this? Can you not run it where it generated the FPE in hypre >>> using the -start_in_debugger option? >>> >>> Barry >>> >>> Perhaps you have multiple PETSC_ARCH or multiple PETSc installs to >>> explain why you reported two different places where the exception occurred. >>> >>> On Apr 25, 2015, at 8:31 PM, Danyang Su wrote: >>>> >>>> >>>> >>>> On 15-04-25 06:26 PM, Matthew Knepley wrote: >>>> >>>>> On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su >>>>> wrote: >>>>> >>>>> >>>>> On 15-04-25 06:03 PM, Barry Smith wrote: >>>>> If this is what you got in your last run >>>>> >>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>> 1344 cinfrt = cinfrt_dg(i1) * >>>>> diff(ic,idim) !diff is a very small >>>>> value, e.g., 1.0d-316 >>>>> then it is still catching floating point underflow, which we do >>>>> not want. This means either the change I suggested you make in the fp.c >>>>> code didn't work or it actually uses a different floating point trap than >>>>> that one. BTW: absurd numbers like 1.0d-316 are often a symptom of >>>>> uninitialized data; could that be a problem that diff is not filled >>>>> correctly for all the ic, idim you are using? >>>>> >>>>> This going round and round is very frustrating and a waste of >>>>> time. You need to be more proactive yourself and explore the code and poke >>>>> around to figure out how to solve the problem. >>>>> >>>>> Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I >>>>> can see what FP trap is being used on your machine. >>>>> >>>>> Barry >>>>> Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise I >>>>> cannot find this file. >>>>> >>>>> Its include/petscconf.h >>>>> Do I need to reconfigure PETSc after changing the code you mentioned? >>>>> >>>>> No, but you need to rebuild. >>>>> >>>> Yes, I have done 'gnumake'. >>>> >>>>> Matt >>>>> Danyang >>>>> >>>>> >>>>> >>>>> On Apr 25, 2015, at 2:24 PM, Danyang Su wrote: >>>>> >>>>> >>>>> >>>>> On 15-04-25 11:55 AM, Barry Smith wrote: >>>>> On Apr 25, 2015, at 1:51 PM, Danyang Su >>>>> wrote: >>>>> >>>>> >>>>> >>>>> On 15-04-25 11:32 AM, Barry Smith wrote: >>>>> >>>>> I told you this yesterday. >>>>> >>>>> It is probably stopping here on a harmless underflow. You need to >>>>> edit the PETSc code to not worry about underflow. >>>>> >>>>> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and >>>>> locate >>>>> >>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>>>> #else >>>>> >>>>> change it to >>>>> >>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >>>>> #else >>>>> >>>>> Then run make gnumake in the PETSc directory to compile the new >>>>> version. Now link and run the program again with -fp_trap and see where it >>>>> gets stuck this time. >>>>> >>>>> Did you do this? >>>>> >>>>> Barry >>>>> >>>>> Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc >>>>> directory. I just did a double check and ran make gnumake again and got the >>>>> following information this time. >>>>> >>>>> >>>>> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >>>>> make gnumake >>>>> Building PETSc using GNU Make with 10 build threads >>>>> ========================================== >>>>> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>> make[1]: Nothing to be done for `all'. >>>>> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>> ========================================= >>>>> >>>>> >>>>> Then I recompiled the codes, ran with -fp_trap and still got the >>>>> following error >>>>> >>>>> Backtrace for this error: >>>>> Note: The EXACT line numbers in the stack are not available, >>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the >>>>> function >>>>> [2]PETSC ERROR: is given. >>>>> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>> INSTEAD the line number of the start of the function >>>>> [3]PETSC ERROR: is given. >>>>> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>> [2]PETSC ERROR: User provided function() line 0 in Unknown file >>>>> trapped floating point error >>>>> [3]PETSC ERROR: User provided function() line 0 in Unknown file >>>>> trapped floating point error >>>>> >>>>> >>>>> This is different then what you sent a few minutes ago where it >>>>> crashed in hypre. >>>>> >>>>> Anyways you need to use the -start_in_debugger business I sent in >>>>> the previous email to see the exact place the problem occurs. >>>>> >>>>> Here is the information shown on gdb screen >>>>> >>>>> Program received signal SIGFPE, Arithmetic exception. >>>>> 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, >>>>> njamxc=34, >>>>> cinfradx=..., radial_coordx=.FALSE., _suffix=3) >>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>> 1344 cinfrt = cinfrt_dg(i1) * >>>>> diff(ic,idim) !diff is a very small >>>>> value, e.g., 1.0d-316 >>>>> (gdb) >>>>> >>>>> After type cont on gdb screen, I got error information as below >>>>> >>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>> [1]PETSC ERROR: The specific exception can be determined by running in >>>>> a debugger. When the >>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found >>>>> with fetestexcept(0x3d) >>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following >>>>> flags: >>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>> [1]PETSC ERROR: --------------------- Stack Frames >>>>> ------------------------------------ >>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>>> available, >>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the >>>>> function >>>>> [1]PETSC ERROR: is given. >>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file >>>>> trapped floating point error >>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>> [0]PETSC ERROR: The specific exception can be determined by running in >>>>> a debugger. When the >>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found >>>>> with fetestexcept(0x3d) >>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following >>>>> flags: >>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>> >>>>> Thanks, >>>>> >>>>> Danyang >>>>> Thanks, >>>>> >>>>> Danyang >>>>> >>>>> On Apr 25, 2015, at 1:05 AM, Danyang Su >>>>> wrote: >>>>> >>>>> Hi Barry and Satish, >>>>> >>>>> How can I get rid of unknown floating point error when a very small >>>>> value is multiplied. >>>>> >>>>> e.g., >>>>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 >>>>> 8.6178408169776945E-317 respectively, >>>>> >>>>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>>>> >>>>> I get the following error when run with "-fp_trap -start_in_debugger". >>>>> >>>>> Backtrace for this error: >>>>> *** unknown floating point error occurred *** >>>>> [2]PETSC ERROR: The specific exception can be determined by running in >>>>> a debugger. When the >>>>> [2]PETSC ERROR: debugger traps the signal, the exception can be found >>>>> with fetestexcept(0x3d) >>>>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 >>>>> 8.6178408169776945E-317 >>>>> where the result is a bitwise OR of the following flags: >>>>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>> [2]PETSC ERROR: Try option -start_in_debugger >>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>> >>>>> Thanks, >>>>> >>>>> Danyang >>>>> >>>>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>>> >>>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>>> >>>>> c 4 1.0976214263087059E-067 >>>>> >>>>> I don't think this number can be stored in a real*4. >>>>> >>>>> Satish >>>>> >>>>> Thanks, Satish. It is caused by this number. >>>>> >>>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>>> >>>>> >>>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>> >>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su >>>>> wrote: >>>>> >>>>> Hi All, >>>>> >>>>> One of my case crashes because of floating point exception when using 4 >>>>> processors, as shown below. But if I run this case with 1 processor, it >>>>> works fine. I have tested the codes with around 100 cases up to 768 >>>>> processors, all other cases work fine. I just wonder if this kind of >>>>> error >>>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>>> >>>>> Yes, almost for sure it is one of these places. >>>>> >>>>> First run the bad case with -fp_trap if all goes well you'll >>>>> see the >>>>> function where the FPE is generated. Then run also with >>>>> -start_in_debugger >>>>> and >>>>> type cont in all four debugger windows. When the FPE happens the >>>>> debugger >>>>> should stop showing exactly where the FPE happens. >>>>> >>>>> Barry >>>>> >>>>> Hi Barry, >>>>> >>>>> If run with -fp_trap -start_in_debugger, I got the following error >>>>> >>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>> [0]PETSC ERROR: The specific exception can be determined by running in >>>>> a >>>>> debugger. When the >>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found >>>>> with >>>>> fetestexcept(0x3d) >>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following >>>>> flags: >>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>> ------------------------------------ >>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>>> available, >>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the >>>>> function >>>>> [0]PETSC ERROR: is given. >>>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>>>> floating point error >>>>> >>>>> Program received signal SIGABRT: Process abort signal. >>>>> >>>>> Backtrace for this error: >>>>> #0 0x7F4FEAB1C7D7 >>>>> #1 0x7F4FEAB1CDDE >>>>> #2 0x7F4FE9E1AD3F >>>>> #3 0x7F4FE9E1ACC9 >>>>> #4 0x7F4FE9E1E0D7 >>>>> #5 0x7F4FEB0B6DCB >>>>> #6 0x7F4FEB0B1825 >>>>> #7 0x7F4FEB0B817F >>>>> #8 0x7F4FE9E1AD3F >>>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>>> >>>>> I checked the code at tprfrtlc.F90:2393, >>>>> >>>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>>> (cx(ix),ix=1,nxout)/) >>>>> >>>>> All the values (time, c, cx) are reasonable, as shown below. The only >>>>> possibility is that realbuffer_gb is in declared as real*4 if using >>>>> sing >>>>> precision output while time, c, cx are declared in real*8. I have a >>>>> lot of >>>>> similar data conversion from real*8 to real*4 output, other code does >>>>> not >>>>> return error. >>>>> >>>>> time 0.0000000000000000 >>>>> c 1 9.9999999999999995E-008 >>>>> c 2 3.1555251077549618E-003 >>>>> c 3 7.1657814842179362E-008 >>>>> c 4 1.0976214263087059E-067 >>>>> c 5 5.2879822292305797E-004 >>>>> c 6 9.9999999999999964E-005 >>>>> c 7 6.4055731968811337E-005 >>>>> c 8 3.4607572892578404E-020 >>>>> cx 1 3.4376650636008101E-005 >>>>> cx 2 7.3989678854017763E-012 >>>>> cx 3 9.5317170613607207E-012 >>>>> cx 4 2.2344525794718353E-015 >>>>> cx 5 3.0624685689695889E-008 >>>>> cx 6 1.0046157902783967E-007 >>>>> cx 7 1.5320169154914984E-004 >>>>> cx 8 8.6930292776346176E-014 >>>>> cx 9 3.5944267559348721E-005 >>>>> cx 10 3.0072645866951157E-018 >>>>> cx 11 2.3592486321095017E-013 >>>>> >>>>> Thanks, >>>>> >>>>> Danyang >>>>> >>>>> >>>>> I can check all the entries of jacobi matrix to see if the value is >>>>> valid, >>>>> but this seems not a good idea as it takes a long time to reach this >>>>> point. If I restart the simulation from a specified time (e.g., 7.685 >>>>> in >>>>> this case), then the error does not occur. >>>>> >>>>> Would you please give me any suggestion on debugging this case? >>>>> >>>>> Thanks and Regards, >>>>> >>>>> Danyang >>>>> >>>>> >>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: >>>>> 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: >>>>> 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: >>>>> 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: >>>>> 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: >>>>> 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> Reduce time step for reactive transport >>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: >>>>> 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> Reduce time step for reactive transport >>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: >>>>> 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> [1]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [1]PETSC ERROR: Floating point exception >>>>> [2]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [2]PETSC ERROR: Floating point exception >>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or >>>>> infinite >>>>> at end of function: Parameter number 3 >>>>> [2]PETSC ERROR: See >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> >>>>> for trouble shooting. >>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>> [1]PETSC ERROR: See >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> >>>>> for trouble shooting. >>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop >>>>> by >>>>> dsu Thu Apr 23 15:38:52 2015 >>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg >>>>> --with-cc=gcc >>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack >>>>> --download-mpich >>>>> --download-mumps --download-hypre --download-superlu_dist >>>>> --download-metis >>>>> --download-parmetis --download-scalapack >>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr >>>>> 23 >>>>> 15:38:52 2015 >>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg >>>>> --with-cc=gcc >>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack >>>>> --download-mpich >>>>> --download-mumps --download-hypre --download-superlu_dist >>>>> --download-metis >>>>> --download-parmetis --download-scalapack >>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>> #3 KSP_PCApply() line 230 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Apr 27 13:31:59 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 27 Apr 2015 13:31:59 -0500 Subject: [petsc-users] Turning off TSADAPT still adapts time step In-Reply-To: <87wq0xscvh.fsf@jedbrown.org> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AA4B1.5010603@princeton.edu> <553AB0C6.5010503@princeton.edu> <553E5AD5.1060208@princeton.edu> <87383ltvf8.fsf@jedbrown.org> <553E5FFF.6080905@princeton.edu> <87zj5tse8k.fsf@jedbrown.org> <553E6EED.7000604@princeton.edu> <87wq0xscvh.fsf@jedbrown.org> Message-ID: <66369816-2AB5-44FC-B15B-65DD6A691BC3@mcs.anl.gov> > On Apr 27, 2015, at 12:24 PM, Jed Brown wrote: > > Mark Lohry writes: > >> -snes_converged_reason: >> >> Nonlinear solve did not converge due to DIVERGED_LOCAL_MIN iterations 4 >> >> (which is expected) > > If you set -snes_stol $something_large then SNES will exit > SNES_CONVERGED_SNORM_RELATIVE instead of SNES_DIVERGED_LINE_SEARCH when > it stagnates. > >>> Add -snes_converged_reason. What do you want to do when the solver >>> fails? Pretend like it succeeded and get the wrong answer? Usually >>> people shorten the time step and retry, which is what you see happening >>> here. >> Well yes, actually - I expected it to fail for this test case but it >> stubbornly gets the right answer. > > Sorry about that. ;-) > >> The explicit integrator has no problem keeping a fixed timestep and >> throwing the expected NaNs when it blows up. >> >> It's alarming to see the unexpected behavior of setting TSADAPTNONE and >> seeing it actually adapting (in a rudimentary fashion) the timestep, and >> keeping that reduced timestep even after the stiffness goes away. > > We figured it's not very useful to just crash or get the wrong answer, > so the solver-based control is decoupled from error-based control. This should be made clear in the documentation and the API. Currently it is very murky especially with messages such as TSAdapt 'none': step 20 stage rejected t=800 + 4.000e+01 retrying with dt=1.000e+01 which start with the TSAdapt word but then talk about rejecting step which one would naturally think has something to do with TSAdapt. Perhaps if every rejected line indicated why such as TSAdapt 'none': step 20 stage rejected (nonlinear solve failed due to xxxx) t=800 + 4.000e+01 retrying with dt=1.000e+01 or TSAdapt 'something': step 20 stage rejected (error control to large due to xxxx) t=800 + 4.000e+01 retrying with dt=1.000e+01 Barry From bsmith at mcs.anl.gov Mon Apr 27 13:47:37 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 27 Apr 2015 13:47:37 -0500 Subject: [petsc-users] Floating point exception In-Reply-To: <553E6770.5010205@gmail.com> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> <553C3E15.8030403@gmail.com> <553C3FE6.5020203@gmail.com> <553C4ECD.6090905@gmail.com> <553E6770.5010205@gmail.com> Message-ID: Danyang, I'm glad you were finally able to get to the problematic point; sorry it took so long (I want to strangle whoever turned on catching of underflows in PETSc). The hypre folks have done a great deal of work on their relaxation code since the PETSc 3.5.3 release. There is a good chance they have fixed the divide by zero problem you are hitting here. You will need to upgrade to the development version of PETSc (that uses the latest version of hypre), here are the instructions on how to obtain it http://www.mcs.anl.gov/petsc/developers/index.html Please let us know if this resolves the problem with hypre failing. Barry > On Apr 27, 2015, at 11:44 AM, Danyang Su wrote: > > Hi Barry, > > I got the following arithemetic exception after the previous bug is fixed. > > Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 > 0x00007f3b23b98f20 in __nanosleep_nocancel () > at ../sysdeps/unix/syscall-template.S:81 > 81 ../sysdeps/unix/syscall-template.S: No such file or directory. > (gdb) cont > Continuing. > > Program received signal SIGFPE, Arithmetic exception. > 0x00007f3b260df449 in gselim (A=0x1e4c580, x=0x1d30c40, n=9) > at par_relax.c:3442 > 3442 x[k] /= A[k*n+k]; > (gdb) > > I tried both PETSc 3.5.2 and 3.5.3 and they return the same error as shown above. For 3.5.3, i edited fp.c file and then configure and make. > > Thanks, > > Danyang > > On 15-04-25 07:34 PM, Danyang Su wrote: >> Hi All, >> >> The "floating point underflow" is caused by a small value divided by a very large value. This result is forced to zero and then it does not report any underflow problem. I just rerun this bad case to see if it still get stuck later. This will take a while. >> >> Thanks for all your kindly reply, >> >> Danyang >> >> On 15-04-25 07:02 PM, Barry Smith wrote: >>> Ok, you do have >>> >>> #ifndef PETSC_HAVE_XMMINTRIN_H >>> #define PETSC_HAVE_XMMINTRIN_H 1 >>> #endif >>> >>> so the change you made should cause it to stop trapping underflow exceptions. >>> >>> Now in one email you reported a FPE within hypre, then I asked you to run with -start_in_debugger to determine where it happened exactly and then you reported the FPE happened in user code (what seemed to be an underflow issue). Why is this? Can you not run it where it generated the FPE in hypre using the -start_in_debugger option? >>> >>> Barry >>> >>> Perhaps you have multiple PETSC_ARCH or multiple PETSc installs to explain why you reported two different places where the exception occurred. >>> >>>> On Apr 25, 2015, at 8:31 PM, Danyang Su wrote: >>>> >>>> >>>> >>>> On 15-04-25 06:26 PM, Matthew Knepley wrote: >>>>> On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su wrote: >>>>> >>>>> >>>>> On 15-04-25 06:03 PM, Barry Smith wrote: >>>>> If this is what you got in your last run >>>>> >>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>> then it is still catching floating point underflow, which we do not want. This means either the change I suggested you make in the fp.c code didn't work or it actually uses a different floating point trap than that one. BTW: absurd numbers like 1.0d-316 are often a symptom of uninitialized data; could that be a problem that diff is not filled correctly for all the ic, idim you are using? >>>>> >>>>> This going round and round is very frustrating and a waste of time. You need to be more proactive yourself and explore the code and poke around to figure out how to solve the problem. >>>>> >>>>> Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can see what FP trap is being used on your machine. >>>>> >>>>> Barry >>>>> Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise I cannot find this file. >>>>> >>>>> Its include/petscconf.h >>>>> Do I need to reconfigure PETSc after changing the code you mentioned? >>>>> >>>>> No, but you need to rebuild. >>>> Yes, I have done 'gnumake'. >>>>> Matt >>>>> Danyang >>>>> >>>>> >>>>> >>>>> On Apr 25, 2015, at 2:24 PM, Danyang Su wrote: >>>>> >>>>> >>>>> >>>>> On 15-04-25 11:55 AM, Barry Smith wrote: >>>>> On Apr 25, 2015, at 1:51 PM, Danyang Su >>>>> wrote: >>>>> >>>>> >>>>> >>>>> On 15-04-25 11:32 AM, Barry Smith wrote: >>>>> >>>>> I told you this yesterday. >>>>> >>>>> It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. >>>>> >>>>> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate >>>>> >>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>>>> #else >>>>> >>>>> change it to >>>>> >>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >>>>> #else >>>>> >>>>> Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. >>>>> >>>>> Did you do this? >>>>> >>>>> Barry >>>>> >>>>> Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc directory. I just did a double check and ran make gnumake again and got the following information this time. >>>>> >>>>> >>>>> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >>>>> make gnumake >>>>> Building PETSc using GNU Make with 10 build threads >>>>> ========================================== >>>>> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>> make[1]: Nothing to be done for `all'. >>>>> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>> ========================================= >>>>> >>>>> >>>>> Then I recompiled the codes, ran with -fp_trap and still got the following error >>>>> >>>>> Backtrace for this error: >>>>> Note: The EXACT line numbers in the stack are not available, >>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [2]PETSC ERROR: is given. >>>>> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>> INSTEAD the line number of the start of the function >>>>> [3]PETSC ERROR: is given. >>>>> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>> [2]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>> [3]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>> >>>>> >>>>> This is different then what you sent a few minutes ago where it crashed in hypre. >>>>> >>>>> Anyways you need to use the -start_in_debugger business I sent in the previous email to see the exact place the problem occurs. >>>>> >>>>> Here is the information shown on gdb screen >>>>> >>>>> Program received signal SIGFPE, Arithmetic exception. >>>>> 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, njamxc=34, >>>>> cinfradx=..., radial_coordx=.FALSE., _suffix=3) >>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>> (gdb) >>>>> >>>>> After type cont on gdb screen, I got error information as below >>>>> >>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [1]PETSC ERROR: is given. >>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>> [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>> >>>>> Thanks, >>>>> >>>>> Danyang >>>>> Thanks, >>>>> >>>>> Danyang >>>>> >>>>> On Apr 25, 2015, at 1:05 AM, Danyang Su >>>>> wrote: >>>>> >>>>> Hi Barry and Satish, >>>>> >>>>> How can I get rid of unknown floating point error when a very small value is multiplied. >>>>> >>>>> e.g., >>>>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, >>>>> >>>>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>>>> >>>>> I get the following error when run with "-fp_trap -start_in_debugger". >>>>> >>>>> Backtrace for this error: >>>>> *** unknown floating point error occurred *** >>>>> [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>> [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 >>>>> where the result is a bitwise OR of the following flags: >>>>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>> [2]PETSC ERROR: Try option -start_in_debugger >>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>> >>>>> Thanks, >>>>> >>>>> Danyang >>>>> >>>>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>>> >>>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>>> >>>>> c 4 1.0976214263087059E-067 >>>>> >>>>> I don't think this number can be stored in a real*4. >>>>> >>>>> Satish >>>>> >>>>> Thanks, Satish. It is caused by this number. >>>>> >>>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>>> >>>>> >>>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>> >>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su >>>>> wrote: >>>>> >>>>> Hi All, >>>>> >>>>> One of my case crashes because of floating point exception when using 4 >>>>> processors, as shown below. But if I run this case with 1 processor, it >>>>> works fine. I have tested the codes with around 100 cases up to 768 >>>>> processors, all other cases work fine. I just wonder if this kind of error >>>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>>> >>>>> Yes, almost for sure it is one of these places. >>>>> >>>>> First run the bad case with -fp_trap if all goes well you'll see the >>>>> function where the FPE is generated. Then run also with -start_in_debugger >>>>> and >>>>> type cont in all four debugger windows. When the FPE happens the debugger >>>>> should stop showing exactly where the FPE happens. >>>>> >>>>> Barry >>>>> >>>>> Hi Barry, >>>>> >>>>> If run with -fp_trap -start_in_debugger, I got the following error >>>>> >>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>> [0]PETSC ERROR: The specific exception can be determined by running in a >>>>> debugger. When the >>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >>>>> fetestexcept(0x3d) >>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>> ------------------------------------ >>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [0]PETSC ERROR: is given. >>>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>>>> floating point error >>>>> >>>>> Program received signal SIGABRT: Process abort signal. >>>>> >>>>> Backtrace for this error: >>>>> #0 0x7F4FEAB1C7D7 >>>>> #1 0x7F4FEAB1CDDE >>>>> #2 0x7F4FE9E1AD3F >>>>> #3 0x7F4FE9E1ACC9 >>>>> #4 0x7F4FE9E1E0D7 >>>>> #5 0x7F4FEB0B6DCB >>>>> #6 0x7F4FEB0B1825 >>>>> #7 0x7F4FEB0B817F >>>>> #8 0x7F4FE9E1AD3F >>>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>>> >>>>> I checked the code at tprfrtlc.F90:2393, >>>>> >>>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>>> (cx(ix),ix=1,nxout)/) >>>>> >>>>> All the values (time, c, cx) are reasonable, as shown below. The only >>>>> possibility is that realbuffer_gb is in declared as real*4 if using sing >>>>> precision output while time, c, cx are declared in real*8. I have a lot of >>>>> similar data conversion from real*8 to real*4 output, other code does not >>>>> return error. >>>>> >>>>> time 0.0000000000000000 >>>>> c 1 9.9999999999999995E-008 >>>>> c 2 3.1555251077549618E-003 >>>>> c 3 7.1657814842179362E-008 >>>>> c 4 1.0976214263087059E-067 >>>>> c 5 5.2879822292305797E-004 >>>>> c 6 9.9999999999999964E-005 >>>>> c 7 6.4055731968811337E-005 >>>>> c 8 3.4607572892578404E-020 >>>>> cx 1 3.4376650636008101E-005 >>>>> cx 2 7.3989678854017763E-012 >>>>> cx 3 9.5317170613607207E-012 >>>>> cx 4 2.2344525794718353E-015 >>>>> cx 5 3.0624685689695889E-008 >>>>> cx 6 1.0046157902783967E-007 >>>>> cx 7 1.5320169154914984E-004 >>>>> cx 8 8.6930292776346176E-014 >>>>> cx 9 3.5944267559348721E-005 >>>>> cx 10 3.0072645866951157E-018 >>>>> cx 11 2.3592486321095017E-013 >>>>> >>>>> Thanks, >>>>> >>>>> Danyang >>>>> >>>>> >>>>> I can check all the entries of jacobi matrix to see if the value is valid, >>>>> but this seems not a good idea as it takes a long time to reach this >>>>> point. If I restart the simulation from a specified time (e.g., 7.685 in >>>>> this case), then the error does not occur. >>>>> >>>>> Would you please give me any suggestion on debugging this case? >>>>> >>>>> Thanks and Regards, >>>>> >>>>> Danyang >>>>> >>>>> >>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> Reduce time step for reactive transport >>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> Reduce time step for reactive transport >>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> [1]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [1]PETSC ERROR: Floating point exception >>>>> [2]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [2]PETSC ERROR: Floating point exception >>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >>>>> at end of function: Parameter number 3 >>>>> [2]PETSC ERROR: See >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> >>>>> for trouble shooting. >>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>> [1]PETSC ERROR: See >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> >>>>> for trouble shooting. >>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>>>> dsu Thu Apr 23 15:38:52 2015 >>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>> --download-parmetis --download-scalapack >>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >>>>> 15:38:52 2015 >>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>> --download-parmetis --download-scalapack >>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>> #3 KSP_PCApply() line 230 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>> -- Norbert Wiener >>>> >> > From jed at jedbrown.org Mon Apr 27 15:11:45 2015 From: jed at jedbrown.org (Jed Brown) Date: Mon, 27 Apr 2015 14:11:45 -0600 Subject: [petsc-users] Turning off TSADAPT still adapts time step In-Reply-To: <66369816-2AB5-44FC-B15B-65DD6A691BC3@mcs.anl.gov> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AA4B1.5010603@princeton.edu> <553AB0C6.5010503@princeton.edu> <553E5AD5.1060208@princeton.edu> <87383ltvf8.fsf@jedbrown.org> <553E5FFF.6080905@princeton.edu> <87zj5tse8k.fsf@jedbrown.org> <553E6EED.7000604@princeton.edu> <87wq0xscvh.fsf@jedbrown.org> <66369816-2AB5-44FC-B15B-65DD6A691BC3@mcs.anl.gov> Message-ID: <87iochs54u.fsf@jedbrown.org> Barry Smith writes: > This should be made clear in the documentation and the API. Currently it is very murky especially with messages such as > > TSAdapt 'none': step 20 stage rejected t=800 + 4.000e+01 retrying with dt=1.000e+01 > > which start with the TSAdapt word but then talk about rejecting step which one would naturally think has something to do with TSAdapt. It does have to do with TSAdapt, but not with the "type" of TSAdapt. The KSP convergence test does not depend on whether you use GMRES or CG (except perhaps in the default norm type). > Perhaps if every rejected line indicated why such as > > TSAdapt 'none': step 20 stage rejected (nonlinear solve failed due to xxxx) t=800 + 4.000e+01 retrying with dt=1.000e+01 > > or > > TSAdapt 'something': step 20 stage rejected (error control to large due to xxxx) t=800 + 4.000e+01 retrying with dt=1.000e+01 I'm concerned about these lines being too long and practically unreadable. Should we have -ts_adapt_monitor_verbose? It tends to be really clear what is going on when you use -ts_adapt_monitor -snes_converged_reason. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jchang27 at uh.edu Mon Apr 27 16:00:01 2015 From: jchang27 at uh.edu (Justin Chang) Date: Mon, 27 Apr 2015 16:00:01 -0500 Subject: [petsc-users] Optimization within Elemental package? Message-ID: Hello everyone, I see that PETSc can easily interface to the Elemental package for linear algebra, but are the convex optimization routines available through invoking certain PETSc/TAO functions, data structures, or command lines? All of my matrices and vectors are DM Created, so I am guessing some conversions need to happen in order to use optimization evaluation routines? Thanks, -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Apr 27 16:09:57 2015 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 28 Apr 2015 07:09:57 +1000 Subject: [petsc-users] Optimization within Elemental package? In-Reply-To: References: Message-ID: On Tue, Apr 28, 2015 at 7:00 AM, Justin Chang wrote: > Hello everyone, > > I see that PETSc can easily interface to the Elemental package for linear > algebra, but are the convex optimization routines available through > invoking certain PETSc/TAO functions, data structures, or command lines? > All of my matrices and vectors are DM Created, so I am guessing some > conversions need to happen in order to use optimization evaluation routines? > Not right now, however, you could call them manually. Thanks, Matt > Thanks, > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Apr 27 16:05:27 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 27 Apr 2015 16:05:27 -0500 Subject: [petsc-users] Turning off TSADAPT still adapts time step In-Reply-To: <87iochs54u.fsf@jedbrown.org> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AA4B1.5010603@princeton.edu> <553AB0C6.5010503@princeton.edu> <553E5AD5.1060208@princeton.edu> <87383ltvf8.fsf@jedbrown.org> <553E5FFF.6080905@princeton.edu> <87zj5tse8k.fsf@jedbrown.org> <553E6EED.7000604@princeton.edu> <87wq0xscvh.fsf@jedbrown.org> <66369816-2AB5-44FC-B15B-65DD6A691BC3@mcs.anl.gov> <87iochs54u.fsf@jedbrown.org> Message-ID: <873A5658-BFB7-4D72-9007-F8512B9B36FE@mcs.anl.gov> > On Apr 27, 2015, at 3:11 PM, Jed Brown wrote: > > Barry Smith writes: >> This should be made clear in the documentation and the API. Currently it is very murky especially with messages such as >> >> TSAdapt 'none': step 20 stage rejected t=800 + 4.000e+01 retrying with dt=1.000e+01 >> >> which start with the TSAdapt word but then talk about rejecting step which one would naturally think has something to do with TSAdapt. > > It does have to do with TSAdapt, but not with the "type" of TSAdapt. > The KSP convergence test does not depend on whether you use GMRES or CG > (except perhaps in the default norm type). > >> Perhaps if every rejected line indicated why such as >> >> TSAdapt 'none': step 20 stage rejected (nonlinear solve failed due to xxxx) t=800 + 4.000e+01 retrying with dt=1.000e+01 >> >> or >> >> TSAdapt 'something': step 20 stage rejected (error control to large due to xxxx) t=800 + 4.000e+01 retrying with dt=1.000e+01 > > I'm concerned about these lines being too long and practically > unreadable. It is silly to have something like it is now that is clearly confusing to users when adding 3 or 4 words to the output would remove that confusion. > Should we have -ts_adapt_monitor_verbose? It tends to be > really clear what is going on when you use -ts_adapt_monitor -snes_converged_reason. From bsmith at mcs.anl.gov Mon Apr 27 20:59:53 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 27 Apr 2015 20:59:53 -0500 Subject: [petsc-users] matpartitioning metis In-Reply-To: References: Message-ID: <4DDF7DEC-95E3-4844-8900-05ABD9A3033B@mcs.anl.gov> We don't have an interface directly to metis. You can certainly use the ParMETIS one on one process. Barry > On Apr 26, 2015, at 11:35 PM, Fande Kong wrote: > > Hi all, > > I was wondering if we have an implementation of matpartitioning using metis? There are any parameters to turn on sequential version of ParMETIS mat partitioning? Or just use petsc_comm_self as a comm? > > Thanks, > > Fande, From Sanjay.Kharche at manchester.ac.uk Tue Apr 28 07:40:59 2015 From: Sanjay.Kharche at manchester.ac.uk (Sanjay Kharche) Date: Tue, 28 Apr 2015 12:40:59 +0000 Subject: [petsc-users] VTS writer Message-ID: Hi I am using the following 3 lines to write vts files. It is a structured FD mesh. I now have difference space steps in the 3 directions: it is 1 and 1 in X and Y directions, and 2 in Z direction. Can anyone say how to write this to the vts header? cheers Sanjay From thronesf at gmail.com Tue Apr 28 09:49:31 2015 From: thronesf at gmail.com (Sharp Stone) Date: Tue, 28 Apr 2015 10:49:31 -0400 Subject: [petsc-users] Application Error Message-ID: Dear All, I'm using Petsc to do the parallel computation. But an error confuses me recently. When I use a small scale computation (the computational task is small), the code runs smoothly. However, when I use a much larger parallel computation domain/task, I always get the error (as in the attachment): [proxy:0:0 at node01] HYD_pmcd_pmip_control_cmd_cb (../../../../source/mpich-3.1.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:0 at node01] HYDT_dmxu_poll_wait_for_event (../../../../source/mpich-3.1.1/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:0 at node01] main (../../../../source/mpich-3.1.1/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event I don't know what's been wrong. Is this because there is a bug in MPI? Thank you in advance for any ideas and suggestions! -- Best regards, Feng -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: out.o367446 Type: application/octet-stream Size: 543 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: out.e367446 Type: application/octet-stream Size: 428 bytes Desc: not available URL: From haakon at hakostra.net Tue Apr 28 10:22:39 2015 From: haakon at hakostra.net (=?windows-1252?Q?H=E5kon_Strandenes?=) Date: Tue, 28 Apr 2015 17:22:39 +0200 Subject: [petsc-users] VTS writer In-Reply-To: References: Message-ID: <553FA5BF.4040508@hakostra.net> On 28. april 2015 14:40, Sanjay Kharche wrote: > > Hi > > I am using the following 3 lines to write vts files. It is a structured FD mesh. I now have difference space steps in the 3 directions: it is 1 and 1 in X and Y directions, and 2 in Z direction. Can anyone say how to write this to the vts header? > > cheers > Sanjay > Did you forget to attach/show some example code? H?kon From jed at jedbrown.org Tue Apr 28 10:10:46 2015 From: jed at jedbrown.org (Jed Brown) Date: Tue, 28 Apr 2015 09:10:46 -0600 Subject: [petsc-users] VTS writer In-Reply-To: References: Message-ID: <87vbggqoeh.fsf@jedbrown.org> Sanjay Kharche writes: > I am using the following 3 lines to write vts files. It is a > structured FD mesh. I now have difference space steps in the 3 > directions: it is 1 and 1 in X and Y directions, and 2 in Z > direction. Can anyone say how to write this to the vts header? Call DMDASetUniformCoordinates or DMSetCoordinates. You can write vts or vtr. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From danyang.su at gmail.com Tue Apr 28 12:55:25 2015 From: danyang.su at gmail.com (Danyang Su) Date: Tue, 28 Apr 2015 10:55:25 -0700 Subject: [petsc-users] Floating point exception In-Reply-To: References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> <553C3E15.8030403@gmail.com> <553C3FE6.5020203@gmail.com> <553C4ECD.6090905@gmail.com> <553E6770.5010205@gmail.com> Message-ID: <553FC98D.1000904@gmail.com> Hi Barry, The development version of PETSc does not help to solve my problem. It still crashed due to the same error information. As Matthew mentioned, I checked the matrix and non of the diagonal entry is zero. But the diagonal values for different rows range from -1.0d-18 to 1.0d33 and the matrix is not strict diagonal dominant. When using 1 processor, the matrix is also not strict diagonal dominant, but the diagonal values have a much smaller range and the codes can run without error. Maybe there is something wrong in the matrix for this case and I need to check the matrix first. Thanks, Danyang Program received signal SIGFPE, Arithmetic exception. 0x00007f668c1299d5 in gselim (A=0x2fe3330, x=0x2210680, n=9) at par_relax.c:4363 4363 x[k] /= A[k*n+k]; (gdb) [1]PETSC ERROR: *** unknown floating point error occurred *** [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) [1]PETSC ERROR: where the result is a bitwise OR of the following flags: [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 [1]PETSC ERROR: Try option -start_in_debugger [1]PETSC ERROR: likely location of problem given in stack below [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [1]PETSC ERROR: INSTEAD the line number of the start of the function [1]PETSC ERROR: is given. [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-dev/src/sys/error/fp.c [1]PETSC ERROR: [1] Hypre solve line 216 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c [1]PETSC ERROR: [1] PCApply_HYPRE line 203 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c [1]PETSC ERROR: [1] PCApply line 430 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/interface/precon.c [1]PETSC ERROR: [1] KSP_PCApply line 234 /home/dsu/Soft/PETSc/petsc-dev/include/petsc/private/kspimpl.h [1]PETSC ERROR: [1] KSPInitialResidual line 44 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itres.c [1]PETSC ERROR: [1] KSPSolve_GMRES line 224 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/impls/gmres/gmres.c [1]PETSC ERROR: [1] KSPSolve line 506 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error On 15-04-27 11:47 AM, Barry Smith wrote: > Danyang, > > I'm glad you were finally able to get to the problematic point; sorry it took so long (I want to strangle whoever turned on catching of underflows in PETSc). > > The hypre folks have done a great deal of work on their relaxation code since the PETSc 3.5.3 release. There is a good chance they have fixed the divide by zero problem you are hitting here. You will need to upgrade to the development version of PETSc (that uses the latest version of hypre), here are the instructions on how to obtain it http://www.mcs.anl.gov/petsc/developers/index.html > > Please let us know if this resolves the problem with hypre failing. > > Barry > > > >> On Apr 27, 2015, at 11:44 AM, Danyang Su wrote: >> >> Hi Barry, >> >> I got the following arithemetic exception after the previous bug is fixed. >> >> Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 >> 0x00007f3b23b98f20 in __nanosleep_nocancel () >> at ../sysdeps/unix/syscall-template.S:81 >> 81 ../sysdeps/unix/syscall-template.S: No such file or directory. >> (gdb) cont >> Continuing. >> >> Program received signal SIGFPE, Arithmetic exception. >> 0x00007f3b260df449 in gselim (A=0x1e4c580, x=0x1d30c40, n=9) >> at par_relax.c:3442 >> 3442 x[k] /= A[k*n+k]; >> (gdb) >> >> I tried both PETSc 3.5.2 and 3.5.3 and they return the same error as shown above. For 3.5.3, i edited fp.c file and then configure and make. >> >> Thanks, >> >> Danyang >> >> On 15-04-25 07:34 PM, Danyang Su wrote: >>> Hi All, >>> >>> The "floating point underflow" is caused by a small value divided by a very large value. This result is forced to zero and then it does not report any underflow problem. I just rerun this bad case to see if it still get stuck later. This will take a while. >>> >>> Thanks for all your kindly reply, >>> >>> Danyang >>> >>> On 15-04-25 07:02 PM, Barry Smith wrote: >>>> Ok, you do have >>>> >>>> #ifndef PETSC_HAVE_XMMINTRIN_H >>>> #define PETSC_HAVE_XMMINTRIN_H 1 >>>> #endif >>>> >>>> so the change you made should cause it to stop trapping underflow exceptions. >>>> >>>> Now in one email you reported a FPE within hypre, then I asked you to run with -start_in_debugger to determine where it happened exactly and then you reported the FPE happened in user code (what seemed to be an underflow issue). Why is this? Can you not run it where it generated the FPE in hypre using the -start_in_debugger option? >>>> >>>> Barry >>>> >>>> Perhaps you have multiple PETSC_ARCH or multiple PETSc installs to explain why you reported two different places where the exception occurred. >>>> >>>>> On Apr 25, 2015, at 8:31 PM, Danyang Su wrote: >>>>> >>>>> >>>>> >>>>> On 15-04-25 06:26 PM, Matthew Knepley wrote: >>>>>> On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su wrote: >>>>>> >>>>>> >>>>>> On 15-04-25 06:03 PM, Barry Smith wrote: >>>>>> If this is what you got in your last run >>>>>> >>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>> then it is still catching floating point underflow, which we do not want. This means either the change I suggested you make in the fp.c code didn't work or it actually uses a different floating point trap than that one. BTW: absurd numbers like 1.0d-316 are often a symptom of uninitialized data; could that be a problem that diff is not filled correctly for all the ic, idim you are using? >>>>>> >>>>>> This going round and round is very frustrating and a waste of time. You need to be more proactive yourself and explore the code and poke around to figure out how to solve the problem. >>>>>> >>>>>> Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can see what FP trap is being used on your machine. >>>>>> >>>>>> Barry >>>>>> Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise I cannot find this file. >>>>>> >>>>>> Its include/petscconf.h >>>>>> Do I need to reconfigure PETSc after changing the code you mentioned? >>>>>> >>>>>> No, but you need to rebuild. >>>>> Yes, I have done 'gnumake'. >>>>>> Matt >>>>>> Danyang >>>>>> >>>>>> >>>>>> >>>>>> On Apr 25, 2015, at 2:24 PM, Danyang Su wrote: >>>>>> >>>>>> >>>>>> >>>>>> On 15-04-25 11:55 AM, Barry Smith wrote: >>>>>> On Apr 25, 2015, at 1:51 PM, Danyang Su >>>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On 15-04-25 11:32 AM, Barry Smith wrote: >>>>>> >>>>>> I told you this yesterday. >>>>>> >>>>>> It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. >>>>>> >>>>>> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate >>>>>> >>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>>>>> #else >>>>>> >>>>>> change it to >>>>>> >>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >>>>>> #else >>>>>> >>>>>> Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. >>>>>> >>>>>> Did you do this? >>>>>> >>>>>> Barry >>>>>> >>>>>> Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc directory. I just did a double check and ran make gnumake again and got the following information this time. >>>>>> >>>>>> >>>>>> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >>>>>> make gnumake >>>>>> Building PETSc using GNU Make with 10 build threads >>>>>> ========================================== >>>>>> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>> make[1]: Nothing to be done for `all'. >>>>>> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>> ========================================= >>>>>> >>>>>> >>>>>> Then I recompiled the codes, ran with -fp_trap and still got the following error >>>>>> >>>>>> Backtrace for this error: >>>>>> Note: The EXACT line numbers in the stack are not available, >>>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>> [2]PETSC ERROR: is given. >>>>>> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>> INSTEAD the line number of the start of the function >>>>>> [3]PETSC ERROR: is given. >>>>>> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>> [2]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>> [3]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>> >>>>>> >>>>>> This is different then what you sent a few minutes ago where it crashed in hypre. >>>>>> >>>>>> Anyways you need to use the -start_in_debugger business I sent in the previous email to see the exact place the problem occurs. >>>>>> >>>>>> Here is the information shown on gdb screen >>>>>> >>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>> 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, njamxc=34, >>>>>> cinfradx=..., radial_coordx=.FALSE., _suffix=3) >>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>> (gdb) >>>>>> >>>>>> After type cont on gdb screen, I got error information as below >>>>>> >>>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>> [1]PETSC ERROR: is given. >>>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Danyang >>>>>> Thanks, >>>>>> >>>>>> Danyang >>>>>> >>>>>> On Apr 25, 2015, at 1:05 AM, Danyang Su >>>>>> wrote: >>>>>> >>>>>> Hi Barry and Satish, >>>>>> >>>>>> How can I get rid of unknown floating point error when a very small value is multiplied. >>>>>> >>>>>> e.g., >>>>>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, >>>>>> >>>>>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>>>>> >>>>>> I get the following error when run with "-fp_trap -start_in_debugger". >>>>>> >>>>>> Backtrace for this error: >>>>>> *** unknown floating point error occurred *** >>>>>> [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>> [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 >>>>>> where the result is a bitwise OR of the following flags: >>>>>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>> [2]PETSC ERROR: Try option -start_in_debugger >>>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Danyang >>>>>> >>>>>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>>>> >>>>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>>>> >>>>>> c 4 1.0976214263087059E-067 >>>>>> >>>>>> I don't think this number can be stored in a real*4. >>>>>> >>>>>> Satish >>>>>> >>>>>> Thanks, Satish. It is caused by this number. >>>>>> >>>>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>>>> >>>>>> >>>>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>>> >>>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su >>>>>> wrote: >>>>>> >>>>>> Hi All, >>>>>> >>>>>> One of my case crashes because of floating point exception when using 4 >>>>>> processors, as shown below. But if I run this case with 1 processor, it >>>>>> works fine. I have tested the codes with around 100 cases up to 768 >>>>>> processors, all other cases work fine. I just wonder if this kind of error >>>>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>>>> >>>>>> Yes, almost for sure it is one of these places. >>>>>> >>>>>> First run the bad case with -fp_trap if all goes well you'll see the >>>>>> function where the FPE is generated. Then run also with -start_in_debugger >>>>>> and >>>>>> type cont in all four debugger windows. When the FPE happens the debugger >>>>>> should stop showing exactly where the FPE happens. >>>>>> >>>>>> Barry >>>>>> >>>>>> Hi Barry, >>>>>> >>>>>> If run with -fp_trap -start_in_debugger, I got the following error >>>>>> >>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a >>>>>> debugger. When the >>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >>>>>> fetestexcept(0x3d) >>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>> ------------------------------------ >>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>> [0]PETSC ERROR: is given. >>>>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>>>>> floating point error >>>>>> >>>>>> Program received signal SIGABRT: Process abort signal. >>>>>> >>>>>> Backtrace for this error: >>>>>> #0 0x7F4FEAB1C7D7 >>>>>> #1 0x7F4FEAB1CDDE >>>>>> #2 0x7F4FE9E1AD3F >>>>>> #3 0x7F4FE9E1ACC9 >>>>>> #4 0x7F4FE9E1E0D7 >>>>>> #5 0x7F4FEB0B6DCB >>>>>> #6 0x7F4FEB0B1825 >>>>>> #7 0x7F4FEB0B817F >>>>>> #8 0x7F4FE9E1AD3F >>>>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>>>> >>>>>> I checked the code at tprfrtlc.F90:2393, >>>>>> >>>>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>>>> (cx(ix),ix=1,nxout)/) >>>>>> >>>>>> All the values (time, c, cx) are reasonable, as shown below. The only >>>>>> possibility is that realbuffer_gb is in declared as real*4 if using sing >>>>>> precision output while time, c, cx are declared in real*8. I have a lot of >>>>>> similar data conversion from real*8 to real*4 output, other code does not >>>>>> return error. >>>>>> >>>>>> time 0.0000000000000000 >>>>>> c 1 9.9999999999999995E-008 >>>>>> c 2 3.1555251077549618E-003 >>>>>> c 3 7.1657814842179362E-008 >>>>>> c 4 1.0976214263087059E-067 >>>>>> c 5 5.2879822292305797E-004 >>>>>> c 6 9.9999999999999964E-005 >>>>>> c 7 6.4055731968811337E-005 >>>>>> c 8 3.4607572892578404E-020 >>>>>> cx 1 3.4376650636008101E-005 >>>>>> cx 2 7.3989678854017763E-012 >>>>>> cx 3 9.5317170613607207E-012 >>>>>> cx 4 2.2344525794718353E-015 >>>>>> cx 5 3.0624685689695889E-008 >>>>>> cx 6 1.0046157902783967E-007 >>>>>> cx 7 1.5320169154914984E-004 >>>>>> cx 8 8.6930292776346176E-014 >>>>>> cx 9 3.5944267559348721E-005 >>>>>> cx 10 3.0072645866951157E-018 >>>>>> cx 11 2.3592486321095017E-013 >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Danyang >>>>>> >>>>>> >>>>>> I can check all the entries of jacobi matrix to see if the value is valid, >>>>>> but this seems not a good idea as it takes a long time to reach this >>>>>> point. If I restart the simulation from a specified time (e.g., 7.685 in >>>>>> this case), then the error does not occur. >>>>>> >>>>>> Would you please give me any suggestion on debugging this case? >>>>>> >>>>>> Thanks and Regards, >>>>>> >>>>>> Danyang >>>>>> >>>>>> >>>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>> Reduce time step for reactive transport >>>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>> Reduce time step for reactive transport >>>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [1]PETSC ERROR: Floating point exception >>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [2]PETSC ERROR: Floating point exception >>>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >>>>>> at end of function: Parameter number 3 >>>>>> [2]PETSC ERROR: See >>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>> >>>>>> for trouble shooting. >>>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>>> [1]PETSC ERROR: See >>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>> >>>>>> for trouble shooting. >>>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>>>>> dsu Thu Apr 23 15:38:52 2015 >>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>> --download-parmetis --download-scalapack >>>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >>>>>> 15:38:52 2015 >>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>> --download-parmetis --download-scalapack >>>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>> #3 KSP_PCApply() line 230 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>> -- Norbert Wiener >>>>> From jed at jedbrown.org Tue Apr 28 13:31:00 2015 From: jed at jedbrown.org (Jed Brown) Date: Tue, 28 Apr 2015 12:31:00 -0600 Subject: [petsc-users] Turning off TSADAPT still adapts time step In-Reply-To: <873A5658-BFB7-4D72-9007-F8512B9B36FE@mcs.anl.gov> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AA4B1.5010603@princeton.edu> <553AB0C6.5010503@princeton.edu> <553E5AD5.1060208@princeton.edu> <87383ltvf8.fsf@jedbrown.org> <553E5FFF.6080905@princeton.edu> <87zj5tse8k.fsf@jedbrown.org> <553E6EED.7000604@princeton.edu> <87wq0xscvh.fsf@jedbrown.org> <66369816-2AB5-44FC-B15B-65DD6A691BC3@mcs.anl.gov> <87iochs54u.fsf@jedbrown.org> <873A5658-BFB7-4D72-9007-F8512B9B36FE@mcs.anl.gov> Message-ID: <87k2wwqf4r.fsf@jedbrown.org> Barry Smith writes: >>> Perhaps if every rejected line indicated why such as >>> >>> TSAdapt 'none': step 20 stage rejected (nonlinear solve failed due to xxxx) t=800 + 4.000e+01 retrying with dt=1.000e+01 >>> >>> or >>> >>> TSAdapt 'something': step 20 stage rejected (error control to large due to xxxx) t=800 + 4.000e+01 retrying with dt=1.000e+01 *Stages* are not rejected due to local truncation error because the error estimate only occurs at step completion. Stages are rejected because the solve fails or the user's checkstage function rejects it. When a *step* is rejected due to an error estimate, the output currently reads: TSAdapt 'basic': step 25 rejected t=0.00175892 + 5.270e-05 wlte= 4.92 family='arkimex' scheme=0:'4' dt=3.185e-05 You can see that wlte (weighted local truncation error) is greater than 1. >> I'm concerned about these lines being too long and practically >> unreadable. > > It is silly to have something like it is now that is clearly > confusing to users when adding 3 or 4 words to the output would > remove that confusion. What exactly would you want it to say? You want it to dig into SNES and KSP to do a postmortem analysis of why the solver failed, and print all of that in a summary? We don't do that in SNES; we just say SNES_DIVERGED_LINEAR_SOLVE and the user can add -ksp_converged_reason to find out why. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From patrick.m.lacasse at gmail.com Tue Apr 28 13:31:58 2015 From: patrick.m.lacasse at gmail.com (Patrick Lacasse) Date: Tue, 28 Apr 2015 14:31:58 -0400 Subject: [petsc-users] inplace MatPtAP Message-ID: I have to compute a product Pt*A*P. My projection matrix P won't change the structure of A and I don't need to keep the original A. Is it possible to do it inplace (modify A instead of creating a new C)? Do you think it is feasible to expend MatPtAP to mpiaij and/or sbaij matrices? There is more implementation of MatMatMult. Is it possible to do Pt*A and A*P directly into A (without creating a copy)? Patrick Lacasse -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Apr 28 13:39:54 2015 From: jed at jedbrown.org (Jed Brown) Date: Tue, 28 Apr 2015 12:39:54 -0600 Subject: [petsc-users] inplace MatPtAP In-Reply-To: References: Message-ID: <87h9s0qepx.fsf@jedbrown.org> Patrick Lacasse writes: > I have to compute a product Pt*A*P. > > My projection matrix P won't change the structure of A and I don't need to > keep the original A. > Is it possible to do it inplace (modify A instead of creating a new C)? No. > Do you think it is feasible to expend MatPtAP to mpiaij and/or sbaij > matrices? You want A to be SBAIJ and P MPIAIJ? (Block formats for P usually don't make sense except for MAIJ because P often has non-square blocks.) > There is more implementation of MatMatMult. > Is it possible to do Pt*A and A*P directly into A (without creating a copy)? No. (Not from laziness, but because the semantics of matrix-matrix multiply don't work that way. We could allocate the memory internally and free the old copies, but that amount of seeming mutability is a terrible interface for users and it doesn't reduce peak memory usage.) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Tue Apr 28 13:40:40 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 28 Apr 2015 13:40:40 -0500 Subject: [petsc-users] Turning off TSADAPT still adapts time step In-Reply-To: <87k2wwqf4r.fsf@jedbrown.org> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AA4B1.5010603@princeton.edu> <553AB0C6.5010503@princeton.edu> <553E5AD5.1060208@princeton.edu> <87383ltvf8.fsf@jedbrown.org> <553E5FFF.6080905@princeton.edu> <87zj5tse8k.fsf@jedbrown.org> <553E6EED.7000604@princeton.edu> <87wq0xscvh.fsf@jedbrown.org> <66369816-2AB5-44FC-B15B-65DD6A691BC3@mcs.anl.gov> <87iochs54u.fsf@jedbrown.org> <873A5658-BFB7-4D72-9007-F8512B9B36FE@mcs.anl.gov> <87k2wwqf4r.fsf@jedbrown.org> Message-ID: <71FCA8D9-F3B2-4058-9254-7C222D1D8F65@mcs.anl.gov> > On Apr 28, 2015, at 1:31 PM, Jed Brown wrote: > > Barry Smith writes: >>>> Perhaps if every rejected line indicated why such as >>>> >>>> TSAdapt 'none': step 20 stage rejected (nonlinear solve failed due to xxxx) t=800 + 4.000e+01 retrying with dt=1.000e+01 >>>> >>>> or >>>> >>>> TSAdapt 'something': step 20 stage rejected (error control to large due to xxxx) t=800 + 4.000e+01 retrying with dt=1.000e+01 > > *Stages* are not rejected due to local truncation error because the > error estimate only occurs at step completion. Stages are rejected > because the solve fails or the user's checkstage function rejects it. And only YOU and Emil in the whole freaking world know this!!!!! Thus the message is just plain incomplete. At a minimum it should say stage rejected do to failed nonlinear solver or else say "rejected by user in check stage function". > When a *step* is rejected due to an error estimate, the output currently > reads: > > TSAdapt 'basic': step 25 rejected t=0.00175892 + 5.270e-05 wlte= 4.92 family='arkimex' scheme=0:'4' dt=3.185e-05 > > You can see that wlte (weighted local truncation error) is greater than > 1. > >>> I'm concerned about these lines being too long and practically >>> unreadable. >> >> It is silly to have something like it is now that is clearly >> confusing to users when adding 3 or 4 words to the output would >> remove that confusion. > > What exactly would you want it to say? You want it to dig into SNES and > KSP to do a postmortem analysis of why the solver failed, and print all > of that in a summary? We don't do that in SNES; we just say > SNES_DIVERGED_LINEAR_SOLVE and the user can add -ksp_converged_reason to > find out why. Well at a minimum it should say that the nonlinear solver failed and not assume that the user has Jed's knowledge and KNOWS that "stage rejected" means failed nonlinear solver. I do think that a short postmortem analysis would be useful here because requiring the user to also use -ksp_converged_reason means the user has to RESTART the APPLICATION and the converged reason is printed ALL the time cluttering the output when it has just converged fine. So yes it should at least say "why" the SNES failed. From bsmith at mcs.anl.gov Tue Apr 28 14:04:44 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 28 Apr 2015 14:04:44 -0500 Subject: [petsc-users] Application Error In-Reply-To: References: Message-ID: <97CF5EC6-8C49-4C61-BD3D-7A574760615E@mcs.anl.gov> Killed (signal 9) means that some process (generally external to the running process) has told the process to end. In HPC this often is because 1) the OS has started running low on memory so killed the process (that is taking much of the memory) 2) the batch system has killed the process because it has hit some limit that the has been set by the batch system (such as running too long). My guess is that it is an "out of memory" issue and you are simply using more memory than available. So to run the size problem you want to run you need to use more nodes on your system. It is not likely a "bug" in MPI or elsewhere. Barry > On Apr 28, 2015, at 9:49 AM, Sharp Stone wrote: > > Dear All, > > I'm using Petsc to do the parallel computation. But an error confuses me recently. When I use a small scale computation (the computational task is small), the code runs smoothly. However, when I use a much larger parallel computation domain/task, I always get the error (as in the attachment): > > [proxy:0:0 at node01] HYD_pmcd_pmip_control_cmd_cb (../../../../source/mpich-3.1.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed > [proxy:0:0 at node01] HYDT_dmxu_poll_wait_for_event (../../../../source/mpich-3.1.1/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status > [proxy:0:0 at node01] main (../../../../source/mpich-3.1.1/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event > > > I don't know what's been wrong. Is this because there is a bug in MPI? > Thank you in advance for any ideas and suggestions! > > -- > Best regards, > > Feng > From bsmith at mcs.anl.gov Tue Apr 28 13:15:03 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 28 Apr 2015 13:15:03 -0500 Subject: [petsc-users] Floating point exception in hypre BoomerAMG In-Reply-To: <553FC98D.1000904@gmail.com> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> <553C3E15.8030403@gmail.com> <553C3FE6.5020203@gmail.com> <553C4ECD.6090905@gmail.com> <553E6770.5010205@gmail.com> <553FC98D.1000904@gmail.com> Message-ID: <89DAC785-3218-4E2F-8760-2D0E99652014@mcs.anl.gov> I am forwarding this on to the hypre support email list; hopefully they can have some advice on how to proceed. hypre folks, this is with version hypre-2.10.0b (the previous version had the same behavior). We are getting a divide by zero in gselim() line 4363 (this happens in a time dependent problem after many successful solves with time dependent matrix). I looked at the code, below, and note that there are some checks for the diagonal of A not being zero but not for the line that causes the divide by zero; is this perhaps an oversight in the hypre code? Any advice appreciated. Thanks Barry HYPRE_Int gselim(A,x,n) HYPRE_Real *A; HYPRE_Real *x; HYPRE_Int n; { HYPRE_Int err_flag = 0; HYPRE_Int j,k,m; HYPRE_Real factor; if (n==1) /* A is 1x1 */ { if (A[0] != 0.0) { x[0] = x[0]/A[0]; return(err_flag); } else { err_flag = 1; return(err_flag); } } else /* A is nxn. Forward elimination */ { for (k = 0; k < n-1; k++) { if (A[k*n+k] != 0.0) { for (j = k+1; j < n; j++) { if (A[j*n+k] != 0.0) { factor = A[j*n+k]/A[k*n+k]; for (m = k+1; m < n; m++) { A[j*n+m] -= factor * A[k*n+m]; } /* Elimination step for rhs */ x[j] -= factor * x[k]; } } } } /* Back Substitution */ for (k = n-1; k > 0; --k) { x[k] /= A[k*n+k]; for (j = 0; j < k; j++) { if (A[j*n+k] != 0.0) { x[j] -= x[k] * A[j*n+k]; } } } x[0] /= A[0]; return(err_flag); } } > On Apr 28, 2015, at 12:55 PM, Danyang Su wrote: > > Hi Barry, > > The development version of PETSc does not help to solve my problem. It still crashed due to the same error information. > > As Matthew mentioned, I checked the matrix and non of the diagonal entry is zero. But the diagonal values for different rows range from -1.0d-18 to 1.0d33 and the matrix is not strict diagonal dominant. When using 1 processor, the matrix is also not strict diagonal dominant, but the diagonal values have a much smaller range and the codes can run without error. Maybe there is something wrong in the matrix for this case and I need to check the matrix first. > > Thanks, > > Danyang > > Program received signal SIGFPE, Arithmetic exception. > 0x00007f668c1299d5 in gselim (A=0x2fe3330, x=0x2210680, n=9) > at par_relax.c:4363 > 4363 x[k] /= A[k*n+k]; > (gdb) > > [1]PETSC ERROR: *** unknown floating point error occurred *** > [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the > [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) > [1]PETSC ERROR: where the result is a bitwise OR of the following flags: > [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 > [1]PETSC ERROR: Try option -start_in_debugger > [1]PETSC ERROR: likely location of problem given in stack below > [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [1]PETSC ERROR: INSTEAD the line number of the start of the function > [1]PETSC ERROR: is given. > [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-dev/src/sys/error/fp.c > [1]PETSC ERROR: [1] Hypre solve line 216 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c > [1]PETSC ERROR: [1] PCApply_HYPRE line 203 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c > [1]PETSC ERROR: [1] PCApply line 430 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/interface/precon.c > [1]PETSC ERROR: [1] KSP_PCApply line 234 /home/dsu/Soft/PETSc/petsc-dev/include/petsc/private/kspimpl.h > [1]PETSC ERROR: [1] KSPInitialResidual line 44 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itres.c > [1]PETSC ERROR: [1] KSPSolve_GMRES line 224 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/impls/gmres/gmres.c > [1]PETSC ERROR: [1] KSPSolve line 506 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error > > > > On 15-04-27 11:47 AM, Barry Smith wrote: >> Danyang, >> >> I'm glad you were finally able to get to the problematic point; sorry it took so long (I want to strangle whoever turned on catching of underflows in PETSc). >> >> The hypre folks have done a great deal of work on their relaxation code since the PETSc 3.5.3 release. There is a good chance they have fixed the divide by zero problem you are hitting here. You will need to upgrade to the development version of PETSc (that uses the latest version of hypre), here are the instructions on how to obtain it http://www.mcs.anl.gov/petsc/developers/index.html >> >> Please let us know if this resolves the problem with hypre failing. >> >> Barry >> >> >> >>> On Apr 27, 2015, at 11:44 AM, Danyang Su wrote: >>> >>> Hi Barry, >>> >>> I got the following arithemetic exception after the previous bug is fixed. >>> >>> Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 >>> 0x00007f3b23b98f20 in __nanosleep_nocancel () >>> at ../sysdeps/unix/syscall-template.S:81 >>> 81 ../sysdeps/unix/syscall-template.S: No such file or directory. >>> (gdb) cont >>> Continuing. >>> >>> Program received signal SIGFPE, Arithmetic exception. >>> 0x00007f3b260df449 in gselim (A=0x1e4c580, x=0x1d30c40, n=9) >>> at par_relax.c:3442 >>> 3442 x[k] /= A[k*n+k]; >>> (gdb) >>> >>> I tried both PETSc 3.5.2 and 3.5.3 and they return the same error as shown above. For 3.5.3, i edited fp.c file and then configure and make. >>> >>> Thanks, >>> >>> Danyang >>> >>> On 15-04-25 07:34 PM, Danyang Su wrote: >>>> Hi All, >>>> >>>> The "floating point underflow" is caused by a small value divided by a very large value. This result is forced to zero and then it does not report any underflow problem. I just rerun this bad case to see if it still get stuck later. This will take a while. >>>> >>>> Thanks for all your kindly reply, >>>> >>>> Danyang >>>> >>>> On 15-04-25 07:02 PM, Barry Smith wrote: >>>>> Ok, you do have >>>>> >>>>> #ifndef PETSC_HAVE_XMMINTRIN_H >>>>> #define PETSC_HAVE_XMMINTRIN_H 1 >>>>> #endif >>>>> >>>>> so the change you made should cause it to stop trapping underflow exceptions. >>>>> >>>>> Now in one email you reported a FPE within hypre, then I asked you to run with -start_in_debugger to determine where it happened exactly and then you reported the FPE happened in user code (what seemed to be an underflow issue). Why is this? Can you not run it where it generated the FPE in hypre using the -start_in_debugger option? >>>>> >>>>> Barry >>>>> >>>>> Perhaps you have multiple PETSC_ARCH or multiple PETSc installs to explain why you reported two different places where the exception occurred. >>>>> >>>>>> On Apr 25, 2015, at 8:31 PM, Danyang Su wrote: >>>>>> >>>>>> >>>>>> >>>>>> On 15-04-25 06:26 PM, Matthew Knepley wrote: >>>>>>> On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su wrote: >>>>>>> >>>>>>> >>>>>>> On 15-04-25 06:03 PM, Barry Smith wrote: >>>>>>> If this is what you got in your last run >>>>>>> >>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>> then it is still catching floating point underflow, which we do not want. This means either the change I suggested you make in the fp.c code didn't work or it actually uses a different floating point trap than that one. BTW: absurd numbers like 1.0d-316 are often a symptom of uninitialized data; could that be a problem that diff is not filled correctly for all the ic, idim you are using? >>>>>>> >>>>>>> This going round and round is very frustrating and a waste of time. You need to be more proactive yourself and explore the code and poke around to figure out how to solve the problem. >>>>>>> >>>>>>> Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can see what FP trap is being used on your machine. >>>>>>> >>>>>>> Barry >>>>>>> Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise I cannot find this file. >>>>>>> >>>>>>> Its include/petscconf.h >>>>>>> Do I need to reconfigure PETSc after changing the code you mentioned? >>>>>>> >>>>>>> No, but you need to rebuild. >>>>>> Yes, I have done 'gnumake'. >>>>>>> Matt >>>>>>> Danyang >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Apr 25, 2015, at 2:24 PM, Danyang Su wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 15-04-25 11:55 AM, Barry Smith wrote: >>>>>>> On Apr 25, 2015, at 1:51 PM, Danyang Su >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 15-04-25 11:32 AM, Barry Smith wrote: >>>>>>> >>>>>>> I told you this yesterday. >>>>>>> >>>>>>> It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. >>>>>>> >>>>>>> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate >>>>>>> >>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>>>>>> #else >>>>>>> >>>>>>> change it to >>>>>>> >>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >>>>>>> #else >>>>>>> >>>>>>> Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. >>>>>>> >>>>>>> Did you do this? >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc directory. I just did a double check and ran make gnumake again and got the following information this time. >>>>>>> >>>>>>> >>>>>>> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >>>>>>> make gnumake >>>>>>> Building PETSc using GNU Make with 10 build threads >>>>>>> ========================================== >>>>>>> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>> make[1]: Nothing to be done for `all'. >>>>>>> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>> ========================================= >>>>>>> >>>>>>> >>>>>>> Then I recompiled the codes, ran with -fp_trap and still got the following error >>>>>>> >>>>>>> Backtrace for this error: >>>>>>> Note: The EXACT line numbers in the stack are not available, >>>>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>> [2]PETSC ERROR: is given. >>>>>>> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>> INSTEAD the line number of the start of the function >>>>>>> [3]PETSC ERROR: is given. >>>>>>> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>> [2]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>> [3]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>> >>>>>>> >>>>>>> This is different then what you sent a few minutes ago where it crashed in hypre. >>>>>>> >>>>>>> Anyways you need to use the -start_in_debugger business I sent in the previous email to see the exact place the problem occurs. >>>>>>> >>>>>>> Here is the information shown on gdb screen >>>>>>> >>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>> 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, njamxc=34, >>>>>>> cinfradx=..., radial_coordx=.FALSE., _suffix=3) >>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>> (gdb) >>>>>>> >>>>>>> After type cont on gdb screen, I got error information as below >>>>>>> >>>>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>> [1]PETSC ERROR: is given. >>>>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Danyang >>>>>>> Thanks, >>>>>>> >>>>>>> Danyang >>>>>>> >>>>>>> On Apr 25, 2015, at 1:05 AM, Danyang Su >>>>>>> wrote: >>>>>>> >>>>>>> Hi Barry and Satish, >>>>>>> >>>>>>> How can I get rid of unknown floating point error when a very small value is multiplied. >>>>>>> >>>>>>> e.g., >>>>>>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, >>>>>>> >>>>>>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>>>>>> >>>>>>> I get the following error when run with "-fp_trap -start_in_debugger". >>>>>>> >>>>>>> Backtrace for this error: >>>>>>> *** unknown floating point error occurred *** >>>>>>> [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>> [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 >>>>>>> where the result is a bitwise OR of the following flags: >>>>>>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>> [2]PETSC ERROR: Try option -start_in_debugger >>>>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Danyang >>>>>>> >>>>>>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>>>>> >>>>>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>>>>> >>>>>>> c 4 1.0976214263087059E-067 >>>>>>> >>>>>>> I don't think this number can be stored in a real*4. >>>>>>> >>>>>>> Satish >>>>>>> >>>>>>> Thanks, Satish. It is caused by this number. >>>>>>> >>>>>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>>>>> >>>>>>> >>>>>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>>>> >>>>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su >>>>>>> wrote: >>>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> One of my case crashes because of floating point exception when using 4 >>>>>>> processors, as shown below. But if I run this case with 1 processor, it >>>>>>> works fine. I have tested the codes with around 100 cases up to 768 >>>>>>> processors, all other cases work fine. I just wonder if this kind of error >>>>>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>>>>> >>>>>>> Yes, almost for sure it is one of these places. >>>>>>> >>>>>>> First run the bad case with -fp_trap if all goes well you'll see the >>>>>>> function where the FPE is generated. Then run also with -start_in_debugger >>>>>>> and >>>>>>> type cont in all four debugger windows. When the FPE happens the debugger >>>>>>> should stop showing exactly where the FPE happens. >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> Hi Barry, >>>>>>> >>>>>>> If run with -fp_trap -start_in_debugger, I got the following error >>>>>>> >>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a >>>>>>> debugger. When the >>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >>>>>>> fetestexcept(0x3d) >>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>>> ------------------------------------ >>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>> [0]PETSC ERROR: is given. >>>>>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>>>>>> floating point error >>>>>>> >>>>>>> Program received signal SIGABRT: Process abort signal. >>>>>>> >>>>>>> Backtrace for this error: >>>>>>> #0 0x7F4FEAB1C7D7 >>>>>>> #1 0x7F4FEAB1CDDE >>>>>>> #2 0x7F4FE9E1AD3F >>>>>>> #3 0x7F4FE9E1ACC9 >>>>>>> #4 0x7F4FE9E1E0D7 >>>>>>> #5 0x7F4FEB0B6DCB >>>>>>> #6 0x7F4FEB0B1825 >>>>>>> #7 0x7F4FEB0B817F >>>>>>> #8 0x7F4FE9E1AD3F >>>>>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>>>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>>>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>>>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>>>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>>>>> >>>>>>> I checked the code at tprfrtlc.F90:2393, >>>>>>> >>>>>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>>>>> (cx(ix),ix=1,nxout)/) >>>>>>> >>>>>>> All the values (time, c, cx) are reasonable, as shown below. The only >>>>>>> possibility is that realbuffer_gb is in declared as real*4 if using sing >>>>>>> precision output while time, c, cx are declared in real*8. I have a lot of >>>>>>> similar data conversion from real*8 to real*4 output, other code does not >>>>>>> return error. >>>>>>> >>>>>>> time 0.0000000000000000 >>>>>>> c 1 9.9999999999999995E-008 >>>>>>> c 2 3.1555251077549618E-003 >>>>>>> c 3 7.1657814842179362E-008 >>>>>>> c 4 1.0976214263087059E-067 >>>>>>> c 5 5.2879822292305797E-004 >>>>>>> c 6 9.9999999999999964E-005 >>>>>>> c 7 6.4055731968811337E-005 >>>>>>> c 8 3.4607572892578404E-020 >>>>>>> cx 1 3.4376650636008101E-005 >>>>>>> cx 2 7.3989678854017763E-012 >>>>>>> cx 3 9.5317170613607207E-012 >>>>>>> cx 4 2.2344525794718353E-015 >>>>>>> cx 5 3.0624685689695889E-008 >>>>>>> cx 6 1.0046157902783967E-007 >>>>>>> cx 7 1.5320169154914984E-004 >>>>>>> cx 8 8.6930292776346176E-014 >>>>>>> cx 9 3.5944267559348721E-005 >>>>>>> cx 10 3.0072645866951157E-018 >>>>>>> cx 11 2.3592486321095017E-013 >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Danyang >>>>>>> >>>>>>> >>>>>>> I can check all the entries of jacobi matrix to see if the value is valid, >>>>>>> but this seems not a good idea as it takes a long time to reach this >>>>>>> point. If I restart the simulation from a specified time (e.g., 7.685 in >>>>>>> this case), then the error does not occur. >>>>>>> >>>>>>> Would you please give me any suggestion on debugging this case? >>>>>>> >>>>>>> Thanks and Regards, >>>>>>> >>>>>>> Danyang >>>>>>> >>>>>>> >>>>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>> Reduce time step for reactive transport >>>>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>> Reduce time step for reactive transport >>>>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>>> -------------------------------------------------------------- >>>>>>> [1]PETSC ERROR: Floating point exception >>>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>>> -------------------------------------------------------------- >>>>>>> [2]PETSC ERROR: Floating point exception >>>>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >>>>>>> at end of function: Parameter number 3 >>>>>>> [2]PETSC ERROR: See >>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>> >>>>>>> for trouble shooting. >>>>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>>>> [1]PETSC ERROR: See >>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>> >>>>>>> for trouble shooting. >>>>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>>>>>> dsu Thu Apr 23 15:38:52 2015 >>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>> --download-parmetis --download-scalapack >>>>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >>>>>>> 15:38:52 2015 >>>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>> --download-parmetis --download-scalapack >>>>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>> #3 KSP_PCApply() line 230 in >>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>> -- Norbert Wiener >>>>>> > From thronesf at gmail.com Tue Apr 28 16:12:26 2015 From: thronesf at gmail.com (Sharp Stone) Date: Tue, 28 Apr 2015 17:12:26 -0400 Subject: [petsc-users] Application Error In-Reply-To: <97CF5EC6-8C49-4C61-BD3D-7A574760615E@mcs.anl.gov> References: <97CF5EC6-8C49-4C61-BD3D-7A574760615E@mcs.anl.gov> Message-ID: Hi Barry, Thank you for your suggestive reply. I've also got a question that the routine PetscMemoryGetMaximumUsage would return the memory used for the program, but this is returned memory values used for the root processor or for all of the processors involved in the computation? Thanks! On Tue, Apr 28, 2015 at 3:04 PM, Barry Smith wrote: > > Killed (signal 9) > > means that some process (generally external to the running process) has > told the process to end. In HPC this often is because > > 1) the OS has started running low on memory so killed the process (that is > taking much of the memory) > > 2) the batch system has killed the process because it has hit some limit > that the has been set by the batch system (such as running too long). > > My guess is that it is an "out of memory" issue and you are simply > using more memory than available. So to run the size problem you want to > run you need to use more nodes on your system. It is not likely a "bug" in > MPI or elsewhere. > > Barry > > > On Apr 28, 2015, at 9:49 AM, Sharp Stone wrote: > > > > Dear All, > > > > I'm using Petsc to do the parallel computation. But an error confuses me > recently. When I use a small scale computation (the computational task is > small), the code runs smoothly. However, when I use a much larger parallel > computation domain/task, I always get the error (as in the attachment): > > > > [proxy:0:0 at node01] HYD_pmcd_pmip_control_cmd_cb > (../../../../source/mpich-3.1.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:885): > assert (!closed) failed > > [proxy:0:0 at node01] HYDT_dmxu_poll_wait_for_event > (../../../../source/mpich-3.1.1/src/pm/hydra/tools/demux/demux_poll.c:76): > callback returned error status > > [proxy:0:0 at node01] main > (../../../../source/mpich-3.1.1/src/pm/hydra/pm/pmiserv/pmip.c:206): demux > engine error waiting for event > > > > > > I don't know what's been wrong. Is this because there is a bug in MPI? > > Thank you in advance for any ideas and suggestions! > > > > -- > > Best regards, > > > > Feng > > > > -- Best regards, Feng -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Apr 28 15:11:59 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 28 Apr 2015 15:11:59 -0500 Subject: [petsc-users] divide by zero coming in hypre Message-ID: <828DB7D1-BE86-4928-ADF7-7F5BC92AB0B6@mcs.anl.gov> Trying again, hyper-support rejected my first email do to something in the subject line. I am forwarding this on to the hypre support email list; hopefully they = can have some advice on how to proceed.=20 hypre folks, this is with version hypre-2.10.0b (the previous version ha= d the same behavior). We are getting a divide by zero in gselim() line 4363= (this happens in a time dependent problem after many successful solves wit= h time dependent matrix). I looked at the code, below, and note that there= are some checks for the diagonal of A not being zero but not for the line = that causes the divide by zero; is this perhaps an oversight in the hypre c= ode? Any advice appreciated. Thanks Barry HYPRE_Int gselim(A,x,n) HYPRE_Real *A; HYPRE_Real *x; HYPRE_Int n; { HYPRE_Int err_flag =3D 0; HYPRE_Int j,k,m; HYPRE_Real factor; =20 if (n=3D=3D1) /* A is 1x1 */ =20 { if (A[0] !=3D 0.0) { x[0] =3D x[0]/A[0]; return(err_flag); } else { err_flag =3D 1; return(err_flag); } } else /* A is nxn. Forward elimination */= =20 { for (k =3D 0; k < n-1; k++) { if (A[k*n+k] !=3D 0.0) { =20 for (j =3D k+1; j < n; j++) { if (A[j*n+k] !=3D 0.0) { factor =3D A[j*n+k]/A[k*n+k]; for (m =3D k+1; m < n; m++) { A[j*n+m] -=3D factor * A[k*n+m]; } /* Elimination step for rhs */=20 x[j] -=3D factor * x[k]; =20 } } } } /* Back Substitution */ for (k =3D n-1; k > 0; --k) { x[k] /=3D A[k*n+k]; for (j =3D 0; j < k; j++) { if (A[j*n+k] !=3D 0.0) { x[j] -=3D x[k] * A[j*n+k]; } } } x[0] /=3D A[0]; return(err_flag); } } > On Apr 28, 2015, at 12:55 PM, Danyang Su wrote: >=20 > Hi Barry, >=20 > The development version of PETSc does not help to solve my problem. It st= ill crashed due to the same error information. >=20 > As Matthew mentioned, I checked the matrix and non of the diagonal entry = is zero. But the diagonal values for different rows range from -1.0d-18 to = 1.0d33 and the matrix is not strict diagonal dominant. When using 1 process= or, the matrix is also not strict diagonal dominant, but the diagonal value= s have a much smaller range and the codes can run without error. Maybe ther= e is something wrong in the matrix for this case and I need to check the ma= trix first. >=20 > Thanks, >=20 > Danyang >=20 > Program received signal SIGFPE, Arithmetic exception. > 0x00007f668c1299d5 in gselim (A=3D0x2fe3330, x=3D0x2210680, n=3D9) > at par_relax.c:4363 > 4363 x[k] /=3D A[k*n+k]; > (gdb) >=20 > [1]PETSC ERROR: *** unknown floating point error occurred *** > [1]PETSC ERROR: The specific exception can be determined by running in a = debugger. When the > [1]PETSC ERROR: debugger traps the signal, the exception can be found wit= h fetestexcept(0x3d) > [1]PETSC ERROR: where the result is a bitwise OR of the following flags: > [1]PETSC ERROR: FE_INVALID=3D0x1 FE_DIVBYZERO=3D0x4 FE_OVERFLOW=3D0x8 FE_= UNDERFLOW=3D0x10 FE_INEXACT=3D0x20 > [1]PETSC ERROR: Try option -start_in_debugger > [1]PETSC ERROR: likely location of problem given in stack below > [1]PETSC ERROR: --------------------- Stack Frames ---------------------= --------------- > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not availab= le, > [1]PETSC ERROR: INSTEAD the line number of the start of the functio= n > [1]PETSC ERROR: is given. > [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/pets= c-dev/src/sys/error/fp.c > [1]PETSC ERROR: [1] Hypre solve line 216 /home/dsu/Soft/PETSc/petsc-dev/s= rc/ksp/pc/impls/hypre/hypre.c > [1]PETSC ERROR: [1] PCApply_HYPRE line 203 /home/dsu/Soft/PETSc/petsc-dev= /src/ksp/pc/impls/hypre/hypre.c > [1]PETSC ERROR: [1] PCApply line 430 /home/dsu/Soft/PETSc/petsc-dev/src/k= sp/pc/interface/precon.c > [1]PETSC ERROR: [1] KSP_PCApply line 234 /home/dsu/Soft/PETSc/petsc-dev/i= nclude/petsc/private/kspimpl.h > [1]PETSC ERROR: [1] KSPInitialResidual line 44 /home/dsu/Soft/PETSc/petsc= -dev/src/ksp/ksp/interface/itres.c > [1]PETSC ERROR: [1] KSPSolve_GMRES line 224 /home/dsu/Soft/PETSc/petsc-de= v/src/ksp/ksp/impls/gmres/gmres.c > [1]PETSC ERROR: [1] KSPSolve line 506 /home/dsu/Soft/PETSc/petsc-dev/src/= ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped f= loating point error >=20 >=20 >=20 > On 15-04-27 11:47 AM, Barry Smith wrote: >> Danyang, >>=20 >> I'm glad you were finally able to get to the problematic point; sorr= y it took so long (I want to strangle whoever turned on catching of underfl= ows in PETSc). >>=20 >> The hypre folks have done a great deal of work on their relaxation c= ode since the PETSc 3.5.3 release. There is a good chance they have fixed t= he divide by zero problem you are hitting here. You will need to upgrade to= the development version of PETSc (that uses the latest version of hypre), = here are the instructions on how to obtain it http://www.mcs.anl.gov/petsc/= developers/index.html >>=20 >> Please let us know if this resolves the problem with hypre failing. >>=20 >> Barry >>=20 >>=20 >>=20 >>> On Apr 27, 2015, at 11:44 AM, Danyang Su wrote: >>>=20 >>> Hi Barry, >>>=20 >>> I got the following arithemetic exception after the previous bug is fix= ed. >>>=20 >>> Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 >>> 0x00007f3b23b98f20 in __nanosleep_nocancel () >>> at ../sysdeps/unix/syscall-template.S:81 >>> 81 ../sysdeps/unix/syscall-template.S: No such file or directory. >>> (gdb) cont >>> Continuing. >>>=20 >>> Program received signal SIGFPE, Arithmetic exception. >>> 0x00007f3b260df449 in gselim (A=3D0x1e4c580, x=3D0x1d30c40, n=3D9) >>> at par_relax.c:3442 >>> 3442 x[k] /=3D A[k*n+k]; >>> (gdb) >>>=20 >>> I tried both PETSc 3.5.2 and 3.5.3 and they return the same error as sh= own above. For 3.5.3, i edited fp.c file and then configure and make. >>>=20 >>> Thanks, >>>=20 From bsmith at mcs.anl.gov Tue Apr 28 16:16:30 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 28 Apr 2015 16:16:30 -0500 Subject: [petsc-users] Application Error In-Reply-To: References: <97CF5EC6-8C49-4C61-BD3D-7A574760615E@mcs.anl.gov> Message-ID: from the manual page PetscMemoryGetMaximumUsage - Returns the maximum resident set size (memory used) for the program. Not Collective this means each process gets its own memory usage. If you want across all processes you need to add them up. > On Apr 28, 2015, at 4:12 PM, Sharp Stone wrote: > > Hi Barry, > > Thank you for your suggestive reply. > > I've also got a question that the routine PetscMemoryGetMaximumUsage would return the memory used for the program, but this is returned memory values used for the root processor or for all of the processors involved in the computation? > > Thanks! > > On Tue, Apr 28, 2015 at 3:04 PM, Barry Smith wrote: > > Killed (signal 9) > > means that some process (generally external to the running process) has told the process to end. In HPC this often is because > > 1) the OS has started running low on memory so killed the process (that is taking much of the memory) > > 2) the batch system has killed the process because it has hit some limit that the has been set by the batch system (such as running too long). > > My guess is that it is an "out of memory" issue and you are simply using more memory than available. So to run the size problem you want to run you need to use more nodes on your system. It is not likely a "bug" in MPI or elsewhere. > > Barry > > > On Apr 28, 2015, at 9:49 AM, Sharp Stone wrote: > > > > Dear All, > > > > I'm using Petsc to do the parallel computation. But an error confuses me recently. When I use a small scale computation (the computational task is small), the code runs smoothly. However, when I use a much larger parallel computation domain/task, I always get the error (as in the attachment): > > > > [proxy:0:0 at node01] HYD_pmcd_pmip_control_cmd_cb (../../../../source/mpich-3.1.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed > > [proxy:0:0 at node01] HYDT_dmxu_poll_wait_for_event (../../../../source/mpich-3.1.1/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status > > [proxy:0:0 at node01] main (../../../../source/mpich-3.1.1/src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event > > > > > > I don't know what's been wrong. Is this because there is a bug in MPI? > > Thank you in advance for any ideas and suggestions! > > > > -- > > Best regards, > > > > Feng > > > > > > > -- > Best regards, > > Feng From hypre-support at llnl.gov Tue Apr 28 18:40:02 2015 From: hypre-support at llnl.gov (Ulrike Yang hypre Tracker) Date: Tue, 28 Apr 2015 23:40:02 -0000 Subject: [petsc-users] [issue1287] divide by zero coming in hypre In-Reply-To: <828DB7D1-BE86-4928-ADF7-7F5BC92AB0B6@mcs.anl.gov> Message-ID: <7D65D8EBE6ADFA42852A360449134C026DC46F45@PRDEXMBX-03.the-lab.llnl.gov> Ulrike Yang added the comment: Hi Barry, You are right. That's a bug which needs to be fixed. Thanks for pointing it out. Ulrike ---------- status: unread -> chatting _____________________________________________ hypre Issue Tracker _____________________________________________ From bsmith at mcs.anl.gov Tue Apr 28 19:27:41 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 28 Apr 2015 19:27:41 -0500 Subject: [petsc-users] Fwd: [issue1287] divide by zero coming in hypre References: <7D65D8EBE6ADFA42852A360449134C026DC46F45@PRDEXMBX-03.the-lab.llnl.gov> Message-ID: > Begin forwarded message: > > From: Ulrike Yang hypre Tracker > Subject: [petsc-users] [issue1287] divide by zero coming in hypre > Date: April 28, 2015 at 6:40:02 PM CDT > To: > Reply-To: hypre Issue Tracker > > > Ulrike Yang added the comment: > > Hi Barry, > You are right. That's a bug which needs to be fixed. Thanks for pointing it out. > Ulrike > > ---------- > status: unread -> chatting > > _____________________________________________ > hypre Issue Tracker > > _____________________________________________ From danyang.su at gmail.com Tue Apr 28 19:30:22 2015 From: danyang.su at gmail.com (Danyang Su) Date: Tue, 28 Apr 2015 17:30:22 -0700 Subject: [petsc-users] Floating point exception in hypre BoomerAMG In-Reply-To: <89DAC785-3218-4E2F-8760-2D0E99652014@mcs.anl.gov> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> <553C3E15.8030403@gmail.com> <553C3FE6.5020203@gmail.com> <553C4ECD.6090905@gmail.com> <553E6770.5010205@gmail.com> <553FC98D.1000904@gmail.com> <89DAC785-3218-4E2F-8760-2D0E99652014@mcs.anl.gov> Message-ID: <5540261E.2040502@gmail.com> Hi Barry, There seems another bug (not pretty sure) in PETSc-dev, as shown below. The case I used is similar with the one I mentioned recently. I have no problem running this case using PETSc 3.5.2. But it give out the following error using PETSc-dev, even with only one processor. Thanks, Danyang timestep: 2048 time: 4.615E+00 years delt: 1.460E-20 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Vec is locked read only, argument # 1 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2796-g23b6c49 GIT Date: 2015-04-27 11:32:44 -0500 [0]PETSC ERROR: ../min3p_thcm on a linux-gnu-dbg named nwmop by dsu Tue Apr 28 15:56:41 2015 [0]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack [0]PETSC ERROR: #349 VecGetArray() line 1646 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/interface/rvector.c [0]PETSC ERROR: #350 VecGetArrayPair() line 434 in /home/dsu/Soft/PETSc/petsc-dev/include/petscvec.h [0]PETSC ERROR: #351 VecScatterBegin_SSToSS() line 649 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c [0]PETSC ERROR: #352 VecScatterBegin() line 1694 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c [0]PETSC ERROR: #353 DMLocalToGlobalBegin_DA() line 56 in /home/dsu/Soft/PETSc/petsc-dev/src/dm/impls/da/dagtol.c [0]PETSC ERROR: #354 DMLocalToGlobalBegin() line 1944 in /home/dsu/Soft/PETSc/petsc-dev/src/dm/interface/dm.c Reduce time step for reactive transport no further time step reduction possible Optimal time increment computed by MIN3P 2.6644769214899893E-018 Minimum time increment specified by user 3.6499999999999998E-018 Please, reduce the MINIMUM TIME INCREMENT stop signal in time step reduction failed On 15-04-28 11:15 AM, Barry Smith wrote: > I am forwarding this on to the hypre support email list; hopefully they can have some advice on how to proceed. > > hypre folks, this is with version hypre-2.10.0b (the previous version had the same behavior). We are getting a divide by zero in gselim() line 4363 (this happens in a time dependent problem after many successful solves with time dependent matrix). I looked at the code, below, and note that there are some checks for the diagonal of A not being zero but not for the line that causes the divide by zero; is this perhaps an oversight in the hypre code? Any advice appreciated. > > Thanks > > Barry > > > > > HYPRE_Int gselim(A,x,n) > HYPRE_Real *A; > HYPRE_Real *x; > HYPRE_Int n; > { > HYPRE_Int err_flag = 0; > HYPRE_Int j,k,m; > HYPRE_Real factor; > > if (n==1) /* A is 1x1 */ > { > if (A[0] != 0.0) > { > x[0] = x[0]/A[0]; > return(err_flag); > } > else > { > err_flag = 1; > return(err_flag); > } > } > else /* A is nxn. Forward elimination */ > { > for (k = 0; k < n-1; k++) > { > if (A[k*n+k] != 0.0) > { > for (j = k+1; j < n; j++) > { > if (A[j*n+k] != 0.0) > { > factor = A[j*n+k]/A[k*n+k]; > for (m = k+1; m < n; m++) > { > A[j*n+m] -= factor * A[k*n+m]; > } > /* Elimination step for rhs */ > x[j] -= factor * x[k]; > } > } > } > } > /* Back Substitution */ > for (k = n-1; k > 0; --k) > { > x[k] /= A[k*n+k]; > for (j = 0; j < k; j++) > { > if (A[j*n+k] != 0.0) > { > x[j] -= x[k] * A[j*n+k]; > } > } > } > x[0] /= A[0]; > return(err_flag); > } > } > > >> On Apr 28, 2015, at 12:55 PM, Danyang Su wrote: >> >> Hi Barry, >> >> The development version of PETSc does not help to solve my problem. It still crashed due to the same error information. >> >> As Matthew mentioned, I checked the matrix and non of the diagonal entry is zero. But the diagonal values for different rows range from -1.0d-18 to 1.0d33 and the matrix is not strict diagonal dominant. When using 1 processor, the matrix is also not strict diagonal dominant, but the diagonal values have a much smaller range and the codes can run without error. Maybe there is something wrong in the matrix for this case and I need to check the matrix first. >> >> Thanks, >> >> Danyang >> >> Program received signal SIGFPE, Arithmetic exception. >> 0x00007f668c1299d5 in gselim (A=0x2fe3330, x=0x2210680, n=9) >> at par_relax.c:4363 >> 4363 x[k] /= A[k*n+k]; >> (gdb) >> >> [1]PETSC ERROR: *** unknown floating point error occurred *** >> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >> [1]PETSC ERROR: Try option -start_in_debugger >> [1]PETSC ERROR: likely location of problem given in stack below >> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> [1]PETSC ERROR: INSTEAD the line number of the start of the function >> [1]PETSC ERROR: is given. >> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-dev/src/sys/error/fp.c >> [1]PETSC ERROR: [1] Hypre solve line 216 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >> [1]PETSC ERROR: [1] PCApply_HYPRE line 203 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >> [1]PETSC ERROR: [1] PCApply line 430 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/interface/precon.c >> [1]PETSC ERROR: [1] KSP_PCApply line 234 /home/dsu/Soft/PETSc/petsc-dev/include/petsc/private/kspimpl.h >> [1]PETSC ERROR: [1] KSPInitialResidual line 44 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itres.c >> [1]PETSC ERROR: [1] KSPSolve_GMRES line 224 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/impls/gmres/gmres.c >> [1]PETSC ERROR: [1] KSPSolve line 506 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itfunc.c >> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >> >> >> >> On 15-04-27 11:47 AM, Barry Smith wrote: >>> Danyang, >>> >>> I'm glad you were finally able to get to the problematic point; sorry it took so long (I want to strangle whoever turned on catching of underflows in PETSc). >>> >>> The hypre folks have done a great deal of work on their relaxation code since the PETSc 3.5.3 release. There is a good chance they have fixed the divide by zero problem you are hitting here. You will need to upgrade to the development version of PETSc (that uses the latest version of hypre), here are the instructions on how to obtain it http://www.mcs.anl.gov/petsc/developers/index.html >>> >>> Please let us know if this resolves the problem with hypre failing. >>> >>> Barry >>> >>> >>> >>>> On Apr 27, 2015, at 11:44 AM, Danyang Su wrote: >>>> >>>> Hi Barry, >>>> >>>> I got the following arithemetic exception after the previous bug is fixed. >>>> >>>> Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 >>>> 0x00007f3b23b98f20 in __nanosleep_nocancel () >>>> at ../sysdeps/unix/syscall-template.S:81 >>>> 81 ../sysdeps/unix/syscall-template.S: No such file or directory. >>>> (gdb) cont >>>> Continuing. >>>> >>>> Program received signal SIGFPE, Arithmetic exception. >>>> 0x00007f3b260df449 in gselim (A=0x1e4c580, x=0x1d30c40, n=9) >>>> at par_relax.c:3442 >>>> 3442 x[k] /= A[k*n+k]; >>>> (gdb) >>>> >>>> I tried both PETSc 3.5.2 and 3.5.3 and they return the same error as shown above. For 3.5.3, i edited fp.c file and then configure and make. >>>> >>>> Thanks, >>>> >>>> Danyang >>>> >>>> On 15-04-25 07:34 PM, Danyang Su wrote: >>>>> Hi All, >>>>> >>>>> The "floating point underflow" is caused by a small value divided by a very large value. This result is forced to zero and then it does not report any underflow problem. I just rerun this bad case to see if it still get stuck later. This will take a while. >>>>> >>>>> Thanks for all your kindly reply, >>>>> >>>>> Danyang >>>>> >>>>> On 15-04-25 07:02 PM, Barry Smith wrote: >>>>>> Ok, you do have >>>>>> >>>>>> #ifndef PETSC_HAVE_XMMINTRIN_H >>>>>> #define PETSC_HAVE_XMMINTRIN_H 1 >>>>>> #endif >>>>>> >>>>>> so the change you made should cause it to stop trapping underflow exceptions. >>>>>> >>>>>> Now in one email you reported a FPE within hypre, then I asked you to run with -start_in_debugger to determine where it happened exactly and then you reported the FPE happened in user code (what seemed to be an underflow issue). Why is this? Can you not run it where it generated the FPE in hypre using the -start_in_debugger option? >>>>>> >>>>>> Barry >>>>>> >>>>>> Perhaps you have multiple PETSC_ARCH or multiple PETSc installs to explain why you reported two different places where the exception occurred. >>>>>> >>>>>>> On Apr 25, 2015, at 8:31 PM, Danyang Su wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 15-04-25 06:26 PM, Matthew Knepley wrote: >>>>>>>> On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 15-04-25 06:03 PM, Barry Smith wrote: >>>>>>>> If this is what you got in your last run >>>>>>>> >>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>>> then it is still catching floating point underflow, which we do not want. This means either the change I suggested you make in the fp.c code didn't work or it actually uses a different floating point trap than that one. BTW: absurd numbers like 1.0d-316 are often a symptom of uninitialized data; could that be a problem that diff is not filled correctly for all the ic, idim you are using? >>>>>>>> >>>>>>>> This going round and round is very frustrating and a waste of time. You need to be more proactive yourself and explore the code and poke around to figure out how to solve the problem. >>>>>>>> >>>>>>>> Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can see what FP trap is being used on your machine. >>>>>>>> >>>>>>>> Barry >>>>>>>> Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise I cannot find this file. >>>>>>>> >>>>>>>> Its include/petscconf.h >>>>>>>> Do I need to reconfigure PETSc after changing the code you mentioned? >>>>>>>> >>>>>>>> No, but you need to rebuild. >>>>>>> Yes, I have done 'gnumake'. >>>>>>>> Matt >>>>>>>> Danyang >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Apr 25, 2015, at 2:24 PM, Danyang Su wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 15-04-25 11:55 AM, Barry Smith wrote: >>>>>>>> On Apr 25, 2015, at 1:51 PM, Danyang Su >>>>>>>> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 15-04-25 11:32 AM, Barry Smith wrote: >>>>>>>> >>>>>>>> I told you this yesterday. >>>>>>>> >>>>>>>> It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. >>>>>>>> >>>>>>>> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate >>>>>>>> >>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>>>>>>> #else >>>>>>>> >>>>>>>> change it to >>>>>>>> >>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >>>>>>>> #else >>>>>>>> >>>>>>>> Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. >>>>>>>> >>>>>>>> Did you do this? >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc directory. I just did a double check and ran make gnumake again and got the following information this time. >>>>>>>> >>>>>>>> >>>>>>>> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >>>>>>>> make gnumake >>>>>>>> Building PETSc using GNU Make with 10 build threads >>>>>>>> ========================================== >>>>>>>> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>> make[1]: Nothing to be done for `all'. >>>>>>>> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>> ========================================= >>>>>>>> >>>>>>>> >>>>>>>> Then I recompiled the codes, ran with -fp_trap and still got the following error >>>>>>>> >>>>>>>> Backtrace for this error: >>>>>>>> Note: The EXACT line numbers in the stack are not available, >>>>>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>> [2]PETSC ERROR: is given. >>>>>>>> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>> INSTEAD the line number of the start of the function >>>>>>>> [3]PETSC ERROR: is given. >>>>>>>> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>> [2]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>> [3]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>> >>>>>>>> >>>>>>>> This is different then what you sent a few minutes ago where it crashed in hypre. >>>>>>>> >>>>>>>> Anyways you need to use the -start_in_debugger business I sent in the previous email to see the exact place the problem occurs. >>>>>>>> >>>>>>>> Here is the information shown on gdb screen >>>>>>>> >>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>> 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, njamxc=34, >>>>>>>> cinfradx=..., radial_coordx=.FALSE., _suffix=3) >>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>>> (gdb) >>>>>>>> >>>>>>>> After type cont on gdb screen, I got error information as below >>>>>>>> >>>>>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>> [1]PETSC ERROR: is given. >>>>>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Danyang >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Danyang >>>>>>>> >>>>>>>> On Apr 25, 2015, at 1:05 AM, Danyang Su >>>>>>>> wrote: >>>>>>>> >>>>>>>> Hi Barry and Satish, >>>>>>>> >>>>>>>> How can I get rid of unknown floating point error when a very small value is multiplied. >>>>>>>> >>>>>>>> e.g., >>>>>>>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, >>>>>>>> >>>>>>>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>>>>>>> >>>>>>>> I get the following error when run with "-fp_trap -start_in_debugger". >>>>>>>> >>>>>>>> Backtrace for this error: >>>>>>>> *** unknown floating point error occurred *** >>>>>>>> [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>> [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 >>>>>>>> where the result is a bitwise OR of the following flags: >>>>>>>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>> [2]PETSC ERROR: Try option -start_in_debugger >>>>>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Danyang >>>>>>>> >>>>>>>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>>>>>> >>>>>>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>>>>>> >>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>> >>>>>>>> I don't think this number can be stored in a real*4. >>>>>>>> >>>>>>>> Satish >>>>>>>> >>>>>>>> Thanks, Satish. It is caused by this number. >>>>>>>> >>>>>>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>>>>> >>>>>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su >>>>>>>> wrote: >>>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> One of my case crashes because of floating point exception when using 4 >>>>>>>> processors, as shown below. But if I run this case with 1 processor, it >>>>>>>> works fine. I have tested the codes with around 100 cases up to 768 >>>>>>>> processors, all other cases work fine. I just wonder if this kind of error >>>>>>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>>>>>> >>>>>>>> Yes, almost for sure it is one of these places. >>>>>>>> >>>>>>>> First run the bad case with -fp_trap if all goes well you'll see the >>>>>>>> function where the FPE is generated. Then run also with -start_in_debugger >>>>>>>> and >>>>>>>> type cont in all four debugger windows. When the FPE happens the debugger >>>>>>>> should stop showing exactly where the FPE happens. >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> Hi Barry, >>>>>>>> >>>>>>>> If run with -fp_trap -start_in_debugger, I got the following error >>>>>>>> >>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a >>>>>>>> debugger. When the >>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >>>>>>>> fetestexcept(0x3d) >>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>>>> ------------------------------------ >>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>> [0]PETSC ERROR: is given. >>>>>>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>>>>>>> floating point error >>>>>>>> >>>>>>>> Program received signal SIGABRT: Process abort signal. >>>>>>>> >>>>>>>> Backtrace for this error: >>>>>>>> #0 0x7F4FEAB1C7D7 >>>>>>>> #1 0x7F4FEAB1CDDE >>>>>>>> #2 0x7F4FE9E1AD3F >>>>>>>> #3 0x7F4FE9E1ACC9 >>>>>>>> #4 0x7F4FE9E1E0D7 >>>>>>>> #5 0x7F4FEB0B6DCB >>>>>>>> #6 0x7F4FEB0B1825 >>>>>>>> #7 0x7F4FEB0B817F >>>>>>>> #8 0x7F4FE9E1AD3F >>>>>>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>>>>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>>>>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>>>>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>>>>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>>>>>> >>>>>>>> I checked the code at tprfrtlc.F90:2393, >>>>>>>> >>>>>>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>>>>>> (cx(ix),ix=1,nxout)/) >>>>>>>> >>>>>>>> All the values (time, c, cx) are reasonable, as shown below. The only >>>>>>>> possibility is that realbuffer_gb is in declared as real*4 if using sing >>>>>>>> precision output while time, c, cx are declared in real*8. I have a lot of >>>>>>>> similar data conversion from real*8 to real*4 output, other code does not >>>>>>>> return error. >>>>>>>> >>>>>>>> time 0.0000000000000000 >>>>>>>> c 1 9.9999999999999995E-008 >>>>>>>> c 2 3.1555251077549618E-003 >>>>>>>> c 3 7.1657814842179362E-008 >>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>> c 5 5.2879822292305797E-004 >>>>>>>> c 6 9.9999999999999964E-005 >>>>>>>> c 7 6.4055731968811337E-005 >>>>>>>> c 8 3.4607572892578404E-020 >>>>>>>> cx 1 3.4376650636008101E-005 >>>>>>>> cx 2 7.3989678854017763E-012 >>>>>>>> cx 3 9.5317170613607207E-012 >>>>>>>> cx 4 2.2344525794718353E-015 >>>>>>>> cx 5 3.0624685689695889E-008 >>>>>>>> cx 6 1.0046157902783967E-007 >>>>>>>> cx 7 1.5320169154914984E-004 >>>>>>>> cx 8 8.6930292776346176E-014 >>>>>>>> cx 9 3.5944267559348721E-005 >>>>>>>> cx 10 3.0072645866951157E-018 >>>>>>>> cx 11 2.3592486321095017E-013 >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Danyang >>>>>>>> >>>>>>>> >>>>>>>> I can check all the entries of jacobi matrix to see if the value is valid, >>>>>>>> but this seems not a good idea as it takes a long time to reach this >>>>>>>> point. If I restart the simulation from a specified time (e.g., 7.685 in >>>>>>>> this case), then the error does not occur. >>>>>>>> >>>>>>>> Would you please give me any suggestion on debugging this case? >>>>>>>> >>>>>>>> Thanks and Regards, >>>>>>>> >>>>>>>> Danyang >>>>>>>> >>>>>>>> >>>>>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>> Reduce time step for reactive transport >>>>>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>> Reduce time step for reactive transport >>>>>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>>>> -------------------------------------------------------------- >>>>>>>> [1]PETSC ERROR: Floating point exception >>>>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>>>> -------------------------------------------------------------- >>>>>>>> [2]PETSC ERROR: Floating point exception >>>>>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >>>>>>>> at end of function: Parameter number 3 >>>>>>>> [2]PETSC ERROR: See >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>> >>>>>>>> for trouble shooting. >>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>>>>> [1]PETSC ERROR: See >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>> >>>>>>>> for trouble shooting. >>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>>>>>>> dsu Thu Apr 23 15:38:52 2015 >>>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>> --download-parmetis --download-scalapack >>>>>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >>>>>>>> 15:38:52 2015 >>>>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>> --download-parmetis --download-scalapack >>>>>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>> #3 KSP_PCApply() line 230 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>> From bsmith at mcs.anl.gov Tue Apr 28 20:50:15 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 28 Apr 2015 20:50:15 -0500 Subject: [petsc-users] Floating point exception in hypre BoomerAMG In-Reply-To: <5540261E.2040502@gmail.com> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> <553C3E15.8030403@gmail.com> <553C3FE6.5020203@gmail.com> <553C4ECD.6090905@gmail.com> <553E6770.5010205@gmail.com> <553FC98D.1000904@gmail.com> <89DAC785-3218-4E2F-8760-2D0E99652014@mcs.anl.gov> <5540261E.2040502@gmail.com> Message-ID: <6411F61F-CD41-4037-B727-B5D9CECE030F@mcs.anl.gov> We started enforcing more checks on writing to vectors that you should not write to. Where are you calling DMLocalToGlobalBegin() ? It looks like you are trying to copy values into a global array that you should not because it is a read only input to a function, for example it is the right hand side of a linear system or the input into a SNESFormFunction(). So check the output to that call. Barry > On Apr 28, 2015, at 7:30 PM, Danyang Su wrote: > > Hi Barry, > > There seems another bug (not pretty sure) in PETSc-dev, as shown below. The case I used is similar with the one I mentioned recently. I have no problem running this case using PETSc 3.5.2. But it give out the following error using PETSc-dev, even with only one processor. > > Thanks, > > Danyang > > timestep: 2048 time: 4.615E+00 years delt: 1.460E-20 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Vec is locked read only, argument # 1 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2796-g23b6c49 GIT Date: 2015-04-27 11:32:44 -0500 > [0]PETSC ERROR: ../min3p_thcm on a linux-gnu-dbg named nwmop by dsu Tue Apr 28 15:56:41 2015 > [0]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack > [0]PETSC ERROR: #349 VecGetArray() line 1646 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/interface/rvector.c > [0]PETSC ERROR: #350 VecGetArrayPair() line 434 in /home/dsu/Soft/PETSc/petsc-dev/include/petscvec.h > [0]PETSC ERROR: #351 VecScatterBegin_SSToSS() line 649 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c > [0]PETSC ERROR: #352 VecScatterBegin() line 1694 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c > [0]PETSC ERROR: #353 DMLocalToGlobalBegin_DA() line 56 in /home/dsu/Soft/PETSc/petsc-dev/src/dm/impls/da/dagtol.c > [0]PETSC ERROR: #354 DMLocalToGlobalBegin() line 1944 in /home/dsu/Soft/PETSc/petsc-dev/src/dm/interface/dm.c > Reduce time step for reactive transport > no further time step reduction possible > Optimal time increment computed by MIN3P 2.6644769214899893E-018 > Minimum time increment specified by user 3.6499999999999998E-018 > Please, reduce the MINIMUM TIME INCREMENT > stop signal in time step reduction failed > > On 15-04-28 11:15 AM, Barry Smith wrote: >> I am forwarding this on to the hypre support email list; hopefully they can have some advice on how to proceed. >> >> hypre folks, this is with version hypre-2.10.0b (the previous version had the same behavior). We are getting a divide by zero in gselim() line 4363 (this happens in a time dependent problem after many successful solves with time dependent matrix). I looked at the code, below, and note that there are some checks for the diagonal of A not being zero but not for the line that causes the divide by zero; is this perhaps an oversight in the hypre code? Any advice appreciated. >> >> Thanks >> >> Barry >> >> >> >> >> HYPRE_Int gselim(A,x,n) >> HYPRE_Real *A; >> HYPRE_Real *x; >> HYPRE_Int n; >> { >> HYPRE_Int err_flag = 0; >> HYPRE_Int j,k,m; >> HYPRE_Real factor; >> if (n==1) /* A is 1x1 */ >> { >> if (A[0] != 0.0) >> { >> x[0] = x[0]/A[0]; >> return(err_flag); >> } >> else >> { >> err_flag = 1; >> return(err_flag); >> } >> } >> else /* A is nxn. Forward elimination */ >> { >> for (k = 0; k < n-1; k++) >> { >> if (A[k*n+k] != 0.0) >> { >> for (j = k+1; j < n; j++) >> { >> if (A[j*n+k] != 0.0) >> { >> factor = A[j*n+k]/A[k*n+k]; >> for (m = k+1; m < n; m++) >> { >> A[j*n+m] -= factor * A[k*n+m]; >> } >> /* Elimination step for rhs */ >> x[j] -= factor * x[k]; >> } >> } >> } >> } >> /* Back Substitution */ >> for (k = n-1; k > 0; --k) >> { >> x[k] /= A[k*n+k]; >> for (j = 0; j < k; j++) >> { >> if (A[j*n+k] != 0.0) >> { >> x[j] -= x[k] * A[j*n+k]; >> } >> } >> } >> x[0] /= A[0]; >> return(err_flag); >> } >> } >> >> >>> On Apr 28, 2015, at 12:55 PM, Danyang Su wrote: >>> >>> Hi Barry, >>> >>> The development version of PETSc does not help to solve my problem. It still crashed due to the same error information. >>> >>> As Matthew mentioned, I checked the matrix and non of the diagonal entry is zero. But the diagonal values for different rows range from -1.0d-18 to 1.0d33 and the matrix is not strict diagonal dominant. When using 1 processor, the matrix is also not strict diagonal dominant, but the diagonal values have a much smaller range and the codes can run without error. Maybe there is something wrong in the matrix for this case and I need to check the matrix first. >>> >>> Thanks, >>> >>> Danyang >>> >>> Program received signal SIGFPE, Arithmetic exception. >>> 0x00007f668c1299d5 in gselim (A=0x2fe3330, x=0x2210680, n=9) >>> at par_relax.c:4363 >>> 4363 x[k] /= A[k*n+k]; >>> (gdb) >>> >>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>> [1]PETSC ERROR: Try option -start_in_debugger >>> [1]PETSC ERROR: likely location of problem given in stack below >>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>> [1]PETSC ERROR: is given. >>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-dev/src/sys/error/fp.c >>> [1]PETSC ERROR: [1] Hypre solve line 216 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >>> [1]PETSC ERROR: [1] PCApply_HYPRE line 203 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >>> [1]PETSC ERROR: [1] PCApply line 430 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/interface/precon.c >>> [1]PETSC ERROR: [1] KSP_PCApply line 234 /home/dsu/Soft/PETSc/petsc-dev/include/petsc/private/kspimpl.h >>> [1]PETSC ERROR: [1] KSPInitialResidual line 44 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itres.c >>> [1]PETSC ERROR: [1] KSPSolve_GMRES line 224 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/impls/gmres/gmres.c >>> [1]PETSC ERROR: [1] KSPSolve line 506 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itfunc.c >>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>> >>> >>> >>> On 15-04-27 11:47 AM, Barry Smith wrote: >>>> Danyang, >>>> >>>> I'm glad you were finally able to get to the problematic point; sorry it took so long (I want to strangle whoever turned on catching of underflows in PETSc). >>>> >>>> The hypre folks have done a great deal of work on their relaxation code since the PETSc 3.5.3 release. There is a good chance they have fixed the divide by zero problem you are hitting here. You will need to upgrade to the development version of PETSc (that uses the latest version of hypre), here are the instructions on how to obtain it http://www.mcs.anl.gov/petsc/developers/index.html >>>> >>>> Please let us know if this resolves the problem with hypre failing. >>>> >>>> Barry >>>> >>>> >>>> >>>>> On Apr 27, 2015, at 11:44 AM, Danyang Su wrote: >>>>> >>>>> Hi Barry, >>>>> >>>>> I got the following arithemetic exception after the previous bug is fixed. >>>>> >>>>> Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 >>>>> 0x00007f3b23b98f20 in __nanosleep_nocancel () >>>>> at ../sysdeps/unix/syscall-template.S:81 >>>>> 81 ../sysdeps/unix/syscall-template.S: No such file or directory. >>>>> (gdb) cont >>>>> Continuing. >>>>> >>>>> Program received signal SIGFPE, Arithmetic exception. >>>>> 0x00007f3b260df449 in gselim (A=0x1e4c580, x=0x1d30c40, n=9) >>>>> at par_relax.c:3442 >>>>> 3442 x[k] /= A[k*n+k]; >>>>> (gdb) >>>>> >>>>> I tried both PETSc 3.5.2 and 3.5.3 and they return the same error as shown above. For 3.5.3, i edited fp.c file and then configure and make. >>>>> >>>>> Thanks, >>>>> >>>>> Danyang >>>>> >>>>> On 15-04-25 07:34 PM, Danyang Su wrote: >>>>>> Hi All, >>>>>> >>>>>> The "floating point underflow" is caused by a small value divided by a very large value. This result is forced to zero and then it does not report any underflow problem. I just rerun this bad case to see if it still get stuck later. This will take a while. >>>>>> >>>>>> Thanks for all your kindly reply, >>>>>> >>>>>> Danyang >>>>>> >>>>>> On 15-04-25 07:02 PM, Barry Smith wrote: >>>>>>> Ok, you do have >>>>>>> >>>>>>> #ifndef PETSC_HAVE_XMMINTRIN_H >>>>>>> #define PETSC_HAVE_XMMINTRIN_H 1 >>>>>>> #endif >>>>>>> >>>>>>> so the change you made should cause it to stop trapping underflow exceptions. >>>>>>> >>>>>>> Now in one email you reported a FPE within hypre, then I asked you to run with -start_in_debugger to determine where it happened exactly and then you reported the FPE happened in user code (what seemed to be an underflow issue). Why is this? Can you not run it where it generated the FPE in hypre using the -start_in_debugger option? >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> Perhaps you have multiple PETSC_ARCH or multiple PETSc installs to explain why you reported two different places where the exception occurred. >>>>>>> >>>>>>>> On Apr 25, 2015, at 8:31 PM, Danyang Su wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 15-04-25 06:26 PM, Matthew Knepley wrote: >>>>>>>>> On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> On 15-04-25 06:03 PM, Barry Smith wrote: >>>>>>>>> If this is what you got in your last run >>>>>>>>> >>>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>>>> then it is still catching floating point underflow, which we do not want. This means either the change I suggested you make in the fp.c code didn't work or it actually uses a different floating point trap than that one. BTW: absurd numbers like 1.0d-316 are often a symptom of uninitialized data; could that be a problem that diff is not filled correctly for all the ic, idim you are using? >>>>>>>>> >>>>>>>>> This going round and round is very frustrating and a waste of time. You need to be more proactive yourself and explore the code and poke around to figure out how to solve the problem. >>>>>>>>> >>>>>>>>> Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can see what FP trap is being used on your machine. >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise I cannot find this file. >>>>>>>>> >>>>>>>>> Its include/petscconf.h >>>>>>>>> Do I need to reconfigure PETSc after changing the code you mentioned? >>>>>>>>> >>>>>>>>> No, but you need to rebuild. >>>>>>>> Yes, I have done 'gnumake'. >>>>>>>>> Matt >>>>>>>>> Danyang >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Apr 25, 2015, at 2:24 PM, Danyang Su wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 15-04-25 11:55 AM, Barry Smith wrote: >>>>>>>>> On Apr 25, 2015, at 1:51 PM, Danyang Su >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 15-04-25 11:32 AM, Barry Smith wrote: >>>>>>>>> >>>>>>>>> I told you this yesterday. >>>>>>>>> >>>>>>>>> It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. >>>>>>>>> >>>>>>>>> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate >>>>>>>>> >>>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>>>>>>>> #else >>>>>>>>> >>>>>>>>> change it to >>>>>>>>> >>>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >>>>>>>>> #else >>>>>>>>> >>>>>>>>> Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. >>>>>>>>> >>>>>>>>> Did you do this? >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> >>>>>>>>> Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc directory. I just did a double check and ran make gnumake again and got the following information this time. >>>>>>>>> >>>>>>>>> >>>>>>>>> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >>>>>>>>> make gnumake >>>>>>>>> Building PETSc using GNU Make with 10 build threads >>>>>>>>> ========================================== >>>>>>>>> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>>> make[1]: Nothing to be done for `all'. >>>>>>>>> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>>> ========================================= >>>>>>>>> >>>>>>>>> >>>>>>>>> Then I recompiled the codes, ran with -fp_trap and still got the following error >>>>>>>>> >>>>>>>>> Backtrace for this error: >>>>>>>>> Note: The EXACT line numbers in the stack are not available, >>>>>>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>> [2]PETSC ERROR: is given. >>>>>>>>> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>> INSTEAD the line number of the start of the function >>>>>>>>> [3]PETSC ERROR: is given. >>>>>>>>> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>> [2]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>> [3]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>> >>>>>>>>> >>>>>>>>> This is different then what you sent a few minutes ago where it crashed in hypre. >>>>>>>>> >>>>>>>>> Anyways you need to use the -start_in_debugger business I sent in the previous email to see the exact place the problem occurs. >>>>>>>>> >>>>>>>>> Here is the information shown on gdb screen >>>>>>>>> >>>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>>> 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, njamxc=34, >>>>>>>>> cinfradx=..., radial_coordx=.FALSE., _suffix=3) >>>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>>>> (gdb) >>>>>>>>> >>>>>>>>> After type cont on gdb screen, I got error information as below >>>>>>>>> >>>>>>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>> [1]PETSC ERROR: is given. >>>>>>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Danyang >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Danyang >>>>>>>>> >>>>>>>>> On Apr 25, 2015, at 1:05 AM, Danyang Su >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi Barry and Satish, >>>>>>>>> >>>>>>>>> How can I get rid of unknown floating point error when a very small value is multiplied. >>>>>>>>> >>>>>>>>> e.g., >>>>>>>>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, >>>>>>>>> >>>>>>>>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>>>>>>>> >>>>>>>>> I get the following error when run with "-fp_trap -start_in_debugger". >>>>>>>>> >>>>>>>>> Backtrace for this error: >>>>>>>>> *** unknown floating point error occurred *** >>>>>>>>> [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>> [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 >>>>>>>>> where the result is a bitwise OR of the following flags: >>>>>>>>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>> [2]PETSC ERROR: Try option -start_in_debugger >>>>>>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Danyang >>>>>>>>> >>>>>>>>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>>>>>>> >>>>>>>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>>>>>>> >>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>> >>>>>>>>> I don't think this number can be stored in a real*4. >>>>>>>>> >>>>>>>>> Satish >>>>>>>>> >>>>>>>>> Thanks, Satish. It is caused by this number. >>>>>>>>> >>>>>>>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>>>>>> >>>>>>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> One of my case crashes because of floating point exception when using 4 >>>>>>>>> processors, as shown below. But if I run this case with 1 processor, it >>>>>>>>> works fine. I have tested the codes with around 100 cases up to 768 >>>>>>>>> processors, all other cases work fine. I just wonder if this kind of error >>>>>>>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>>>>>>> >>>>>>>>> Yes, almost for sure it is one of these places. >>>>>>>>> >>>>>>>>> First run the bad case with -fp_trap if all goes well you'll see the >>>>>>>>> function where the FPE is generated. Then run also with -start_in_debugger >>>>>>>>> and >>>>>>>>> type cont in all four debugger windows. When the FPE happens the debugger >>>>>>>>> should stop showing exactly where the FPE happens. >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> >>>>>>>>> Hi Barry, >>>>>>>>> >>>>>>>>> If run with -fp_trap -start_in_debugger, I got the following error >>>>>>>>> >>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a >>>>>>>>> debugger. When the >>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >>>>>>>>> fetestexcept(0x3d) >>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>>>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>>>>> ------------------------------------ >>>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>> [0]PETSC ERROR: is given. >>>>>>>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>>>>>>>> floating point error >>>>>>>>> >>>>>>>>> Program received signal SIGABRT: Process abort signal. >>>>>>>>> >>>>>>>>> Backtrace for this error: >>>>>>>>> #0 0x7F4FEAB1C7D7 >>>>>>>>> #1 0x7F4FEAB1CDDE >>>>>>>>> #2 0x7F4FE9E1AD3F >>>>>>>>> #3 0x7F4FE9E1ACC9 >>>>>>>>> #4 0x7F4FE9E1E0D7 >>>>>>>>> #5 0x7F4FEB0B6DCB >>>>>>>>> #6 0x7F4FEB0B1825 >>>>>>>>> #7 0x7F4FEB0B817F >>>>>>>>> #8 0x7F4FE9E1AD3F >>>>>>>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>>>>>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>>>>>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>>>>>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>>>>>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>>>>>>> >>>>>>>>> I checked the code at tprfrtlc.F90:2393, >>>>>>>>> >>>>>>>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>>>>>>> (cx(ix),ix=1,nxout)/) >>>>>>>>> >>>>>>>>> All the values (time, c, cx) are reasonable, as shown below. The only >>>>>>>>> possibility is that realbuffer_gb is in declared as real*4 if using sing >>>>>>>>> precision output while time, c, cx are declared in real*8. I have a lot of >>>>>>>>> similar data conversion from real*8 to real*4 output, other code does not >>>>>>>>> return error. >>>>>>>>> >>>>>>>>> time 0.0000000000000000 >>>>>>>>> c 1 9.9999999999999995E-008 >>>>>>>>> c 2 3.1555251077549618E-003 >>>>>>>>> c 3 7.1657814842179362E-008 >>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>> c 5 5.2879822292305797E-004 >>>>>>>>> c 6 9.9999999999999964E-005 >>>>>>>>> c 7 6.4055731968811337E-005 >>>>>>>>> c 8 3.4607572892578404E-020 >>>>>>>>> cx 1 3.4376650636008101E-005 >>>>>>>>> cx 2 7.3989678854017763E-012 >>>>>>>>> cx 3 9.5317170613607207E-012 >>>>>>>>> cx 4 2.2344525794718353E-015 >>>>>>>>> cx 5 3.0624685689695889E-008 >>>>>>>>> cx 6 1.0046157902783967E-007 >>>>>>>>> cx 7 1.5320169154914984E-004 >>>>>>>>> cx 8 8.6930292776346176E-014 >>>>>>>>> cx 9 3.5944267559348721E-005 >>>>>>>>> cx 10 3.0072645866951157E-018 >>>>>>>>> cx 11 2.3592486321095017E-013 >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Danyang >>>>>>>>> >>>>>>>>> >>>>>>>>> I can check all the entries of jacobi matrix to see if the value is valid, >>>>>>>>> but this seems not a good idea as it takes a long time to reach this >>>>>>>>> point. If I restart the simulation from a specified time (e.g., 7.685 in >>>>>>>>> this case), then the error does not occur. >>>>>>>>> >>>>>>>>> Would you please give me any suggestion on debugging this case? >>>>>>>>> >>>>>>>>> Thanks and Regards, >>>>>>>>> >>>>>>>>> Danyang >>>>>>>>> >>>>>>>>> >>>>>>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>> Reduce time step for reactive transport >>>>>>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>> Reduce time step for reactive transport >>>>>>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>>>>> -------------------------------------------------------------- >>>>>>>>> [1]PETSC ERROR: Floating point exception >>>>>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>>>>> -------------------------------------------------------------- >>>>>>>>> [2]PETSC ERROR: Floating point exception >>>>>>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >>>>>>>>> at end of function: Parameter number 3 >>>>>>>>> [2]PETSC ERROR: See >>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>> >>>>>>>>> for trouble shooting. >>>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>>>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>>>>>> [1]PETSC ERROR: See >>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>> >>>>>>>>> for trouble shooting. >>>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>>>>>>>> dsu Thu Apr 23 15:38:52 2015 >>>>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >>>>>>>>> 15:38:52 2015 >>>>>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>> #3 KSP_PCApply() line 230 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>>>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>> > From jchang27 at uh.edu Tue Apr 28 22:19:28 2015 From: jchang27 at uh.edu (Justin Chang) Date: Tue, 28 Apr 2015 22:19:28 -0500 Subject: [petsc-users] Tao iterations In-Reply-To: References: <3112142f40d64968ae157f4a5e819353@LUCKMAN.anl.gov> Message-ID: Jason (or anyone), I am noticing that the iteration numbers reported by TaoGetSolutionStatus() for blmvm differ whenever I change the number of processes. The solution seems to remain the same though. Is there a reason why this could be happening? Thanks, On Tue, Apr 21, 2015 at 10:40 AM, Jason Sarich wrote: > Justin, > > 1) The big difference between TRON and BLMVM is that TRON requires hessian > information, BLMVM only uses gradient information. Thus TRON will usually > converge faster, but requires more information, memory, and a KSP solver. > GPCG (gradient projected conjugate gradient) is another gradient-only > option, but usually performs worse than BLMVM. > > 2) TaoGetLinearSolveIterations() will get the total number of KSP > iterations per solve > > Jason > > > On Tue, Apr 21, 2015 at 10:33 AM, Justin Chang > wrote: > >> Jason, >> >> Tightening the tolerances did the trick. Thanks. Though I do have a >> couple more related questions: >> >> 1) Is there a general guideline for choosing tron over blmvm or vice >> versa? Also is there another tao type that is also suitable given only >> bounded constraints? >> >> 2) Is it possible to obtain the total number of KSP and/or PG iterations >> from tron? >> >> Thanks, >> Justin >> >> On Tue, Apr 21, 2015 at 9:52 AM, Jason Sarich >> wrote: >> >>> Hi Justin, >>> >>> blmvm believes that it is already sufficiently close to a minimum, so it >>> doesn't do anything. You may need to tighten some of the tolerance to force >>> an iteration. >>> >>> Jason >>> >>> >>> On Tue, Apr 21, 2015 at 9:48 AM, Justin Chang >>> wrote: >>> >>>> Time step 1: >>>> >>>> Tao Object: 1 MPI processes >>>> type: blmvm >>>> Gradient steps: 0 >>>> TaoLineSearch Object: 1 MPI processes >>>> type: more-thuente >>>> Active Set subset type: subvec >>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>> Residual in Function/Gradient:=0.0663148 >>>> Objective value=-55.5945 >>>> total number of iterations=35, (max: 2000) >>>> total number of function/gradient evaluations=37, (max: 4000) >>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>> >>>> Time step 2: >>>> >>>> Tao Object: 1 MPI processes >>>> type: blmvm >>>> Gradient steps: 0 >>>> TaoLineSearch Object: 1 MPI processes >>>> type: more-thuente >>>> Active Set subset type: subvec >>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>> Residual in Function/Gradient:=0.0682307 >>>> Objective value=-66.9675 >>>> total number of iterations=23, (max: 2000) >>>> total number of function/gradient evaluations=25, (max: 4000) >>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>> >>>> Time step 3: >>>> >>>> Tao Object: 1 MPI processes >>>> type: blmvm >>>> Gradient steps: 0 >>>> TaoLineSearch Object: 1 MPI processes >>>> type: more-thuente >>>> Active Set subset type: subvec >>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>> Residual in Function/Gradient:=0.0680522 >>>> Objective value=-71.8211 >>>> total number of iterations=19, (max: 2000) >>>> total number of function/gradient evaluations=22, (max: 4000) >>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>> >>>> Time step 4: >>>> >>>> Tao Object: 1 MPI processes >>>> type: blmvm >>>> Gradient steps: 0 >>>> TaoLineSearch Object: 1 MPI processes >>>> type: more-thuente >>>> Active Set subset type: subvec >>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>> Residual in Function/Gradient:=0.0551556 >>>> Objective value=-75.1252 >>>> total number of iterations=18, (max: 2000) >>>> total number of function/gradient evaluations=20, (max: 4000) >>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>> >>>> Time step 5: >>>> >>>> Tao Object: 1 MPI processes >>>> type: blmvm >>>> Gradient steps: 0 >>>> TaoLineSearch Object: 1 MPI processes >>>> type: more-thuente >>>> Active Set subset type: subvec >>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>> Residual in Function/Gradient:=0.0675667 >>>> Objective value=-77.4414 >>>> total number of iterations=6, (max: 2000) >>>> total number of function/gradient evaluations=8, (max: 4000) >>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>> >>>> Time step 6: >>>> >>>> Tao Object: 1 MPI processes >>>> type: blmvm >>>> Gradient steps: 0 >>>> TaoLineSearch Object: 1 MPI processes >>>> type: more-thuente >>>> Active Set subset type: subvec >>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>> Residual in Function/Gradient:=0.059143 >>>> Objective value=-79.5007 >>>> total number of iterations=3, (max: 2000) >>>> total number of function/gradient evaluations=5, (max: 4000) >>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>> >>>> Time step 7: >>>> >>>> Tao Object: 1 MPI processes >>>> type: blmvm >>>> Gradient steps: 0 >>>> TaoLineSearch Object: 1 MPI processes >>>> type: more-thuente >>>> Active Set subset type: subvec >>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>> Residual in Function/Gradient:=0.0433683 >>>> Objective value=-81.3546 >>>> total number of iterations=5, (max: 2000) >>>> total number of function/gradient evaluations=8, (max: 4000) >>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>> >>>> Time step 8: >>>> >>>> Tao Object: 1 MPI processes >>>> type: blmvm >>>> Gradient steps: 0 >>>> TaoLineSearch Object: 1 MPI processes >>>> type: more-thuente >>>> Active Set subset type: subvec >>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>> Residual in Function/Gradient:=0.0840676 >>>> Objective value=-82.9382 >>>> total number of iterations=0, (max: 2000) >>>> total number of function/gradient evaluations=1, (max: 4000) >>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>> >>>> Time step 9: >>>> >>>> Tao Object: 1 MPI processes >>>> type: blmvm >>>> Gradient steps: 0 >>>> TaoLineSearch Object: 1 MPI processes >>>> type: more-thuente >>>> Active Set subset type: subvec >>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>> Residual in Function/Gradient:=0.0840676 >>>> Objective value=-82.9382 >>>> total number of iterations=0, (max: 2000) >>>> total number of function/gradient evaluations=1, (max: 4000) >>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>> >>>> Time step 10: >>>> >>>> Tao Object: 1 MPI processes >>>> type: blmvm >>>> Gradient steps: 0 >>>> TaoLineSearch Object: 1 MPI processes >>>> type: more-thuente >>>> Active Set subset type: subvec >>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>> Residual in Function/Gradient:=0.0840676 >>>> Objective value=-82.9382 >>>> total number of iterations=0, (max: 2000) >>>> total number of function/gradient evaluations=1, (max: 4000) >>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>> >>>> >>>> >>>> >>>> >>>> On Tue, Apr 21, 2015 at 9:28 AM, Jason Sarich >>>> wrote: >>>> >>>>> Hi Justin, >>>>> >>>>> what reason is blmvm giving for stopping the solve? (you can use >>>>> -tao_view or -tao_converged_reason to get this) >>>>> >>>>> Jason >>>>> >>>>> On Mon, Apr 20, 2015 at 6:32 PM, Justin Chang >>>>> wrote: >>>>> >>>>>> Jason, >>>>>> >>>>>> I am using TaoGetSolutionStatus(tao,&its, ...) and it gives me >>>>>> exactly what I want. However, I seem to be having an issue with blmvm >>>>>> >>>>>> I wrote my own backward euler code for a transient linear diffusion >>>>>> problem with lower bounds >= 0 and upper bounds <= 1. For the first several >>>>>> time steps I am getting its > 0, and it decreases over time due to the >>>>>> nature of the discrete maximum principles. However, at some point my its >>>>>> become 0 and the solution does not "update", which seems to me that >>>>>> TaoSolve is not doing anything after that. This doesn't happen if I were to >>>>>> use tron (my KSP and PC are cg and jacobi respectively). >>>>>> >>>>>> Do you know why this behavior may occur? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> On Tue, Apr 14, 2015 at 9:35 AM, Jason Sarich >>>>> > wrote: >>>>>> >>>>>>> Hi Justin, >>>>>>> >>>>>>> I have pushed these changes to the "next" branch, your code >>>>>>> snippet should work fine there. >>>>>>> >>>>>>> Note that there is also available (since version 3.5.0) the >>>>>>> routine TaoGetSolutionStatus(tao,&its,NULL,NULL,NULL,NULL,NULL) which will >>>>>>> provide the >>>>>>> same information >>>>>>> >>>>>>> Jason >>>>>>> >>>>>>> On Fri, Apr 10, 2015 at 6:28 PM, Justin Chang >>>>>>> wrote: >>>>>>> >>>>>>>> Whatever is convenient and/or follow the "PETSc" standards. >>>>>>>> Something similar to SNESGetIterationNumber() or KSPGetIterationNumber() >>>>>>>> would be nice. Ideally I want my code to look like this: >>>>>>>> >>>>>>>> ierr = TaoGetIterationNumber(tao,&its);CHKERRQ(ierr); >>>>>>>> ierr = PetscPrintf(PETSC_COMM_WORLD, "Number of Tao iterations = >>>>>>>> %D\n", its); >>>>>>>> >>>>>>>> Thanks :) >>>>>>>> >>>>>>>> On Fri, Apr 10, 2015 at 5:53 PM, Jason Sarich < >>>>>>>> jason.sarich at gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi Justin, I'll get this in. I assume that displaying the number >>>>>>>>> of iterations with tao_converged_reason is what you are asking for in >>>>>>>>> particular? Or did you have something else in mind? >>>>>>>>> >>>>>>>>> Jason >>>>>>>>> On Apr 10, 2015 16:42, "Smith, Barry F." >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Justin, >>>>>>>>>> >>>>>>>>>> Sorry TAO simply doesn't even collect this information >>>>>>>>>> currently. But yes we should definitely make it available! >>>>>>>>>> >>>>>>>>>> Jason, >>>>>>>>>> >>>>>>>>>> Could you please add this; almost all the TaoSolve_xxx() >>>>>>>>>> have a local variable iter; change that to tao->niter (I'm guess this is >>>>>>>>>> suppose to capture this information) and add a TaoGetIterationNumber() and >>>>>>>>>> the uses can access this. Also modify at the end of TaoSolve() >>>>>>>>>> -tao_converged_reason to also print the iteration count. At the same time >>>>>>>>>> since you add this you can add a tao->totalits which would accumulate all >>>>>>>>>> iterations over all the solves for that Tao object and the routine >>>>>>>>>> TaoGetTotalIterations() to access this. Note that TaoSolve() would >>>>>>>>>> initialize tao->niter = 0 at the top. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> Barry >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> > On Apr 10, 2015, at 4:16 PM, Justin Chang >>>>>>>>>> wrote: >>>>>>>>>> > >>>>>>>>>> > Hi all, >>>>>>>>>> > >>>>>>>>>> > Is there a way to generically obtain the number of Tao >>>>>>>>>> iterations? I am looking through the -help options for Tao and I don't see >>>>>>>>>> any metric where you can output this quantity in the manner that you could >>>>>>>>>> for SNES or KSP solves. I am currently using blmvm and tron, and the only >>>>>>>>>> way I can see getting this metric is by outputting -tao_view and/or >>>>>>>>>> -tao_monitor and manually finding this number. I find this cumbersome >>>>>>>>>> especially for transient problems where I would like to simply have this >>>>>>>>>> number printed for each step instead of ending up with unnecessary info. >>>>>>>>>> > >>>>>>>>>> > Thanks, >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > -- >>>>>>>>>> > Justin Chang >>>>>>>>>> > PhD Candidate, Civil Engineering - Computational Sciences >>>>>>>>>> > University of Houston, Department of Civil and Environmental >>>>>>>>>> Engineering >>>>>>>>>> > Houston, TX 77004 >>>>>>>>>> > (512) 963-3262 >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.R.Kharche at exeter.ac.uk Wed Apr 29 02:10:30 2015 From: S.R.Kharche at exeter.ac.uk (Kharche, Sanjay) Date: Wed, 29 Apr 2015 07:10:30 +0000 Subject: [petsc-users] VTS writer In-Reply-To: <553FA5BF.4040508@hakostra.net> References: , <553FA5BF.4040508@hakostra.net> Message-ID: Dear All I forgot to put the "3" or more lines of code I am using. Here they are. > I am using the following 3 lines to write vts files. It is a structured FD mesh. I now have difference space steps in the 3 directions: it is 1 and 1 in X and Y directions, and 2 in Z direction. Can anyone say how to write this to the vts header? // FD spacing in each direction #define DX 0.250 /* X internode spacing */ #define DY 0.250 /* Y internode spacing */ #define DZ 0.500 /* Z internode spacing: See suppl of paper. */ . . main(){ . . sprintf(str,"my_3d%d.vts",file_Counter++); PetscViewer viewer; PetscViewerVTKOpen(PETSC_COMM_WORLD,str,FILE_MODE_WRITE, &viewer); VecView(u, viewer); PetscViewerDestroy(&viewer); From jed at jedbrown.org Wed Apr 29 07:03:01 2015 From: jed at jedbrown.org (Jed Brown) Date: Wed, 29 Apr 2015 06:03:01 -0600 Subject: [petsc-users] VTS writer In-Reply-To: References: <553FA5BF.4040508@hakostra.net> Message-ID: <87iocfp2fe.fsf@jedbrown.org> "Kharche, Sanjay" writes: > Dear All > > I forgot to put the "3" or more lines of code I am using. Here they are. > >> I am using the following 3 lines to write vts files. It is a structured FD mesh. I now have difference space steps in the 3 directions: it is 1 and 1 in X and Y directions, and 2 in Z direction. Can anyone say how to write this to the vts header? > > > // FD spacing in each direction > #define DX 0.250 /* X internode spacing */ > #define DY 0.250 /* Y internode spacing */ > #define DZ 0.500 /* Z internode spacing: See suppl of paper. */ DMDASetUniformCoordinates > . > . > main(){ > . > . > sprintf(str,"my_3d%d.vts",file_Counter++); > PetscViewer viewer; > PetscViewerVTKOpen(PETSC_COMM_WORLD,str,FILE_MODE_WRITE, &viewer); > VecView(u, viewer); > PetscViewerDestroy(&viewer); -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jason.sarich at gmail.com Wed Apr 29 10:39:13 2015 From: jason.sarich at gmail.com (Jason Sarich) Date: Wed, 29 Apr 2015 10:39:13 -0500 Subject: [petsc-users] Tao iterations In-Reply-To: References: <3112142f40d64968ae157f4a5e819353@LUCKMAN.anl.gov> Message-ID: Hi Justin, This expected behavior due to the accumulation of numerical round-offs. If this is a problem or if you just want to confirm that this is the cause, you can try configuring PETSc for quad precision (--with-precision=__float128, works with GNU compilers) and the results should match better. Jason On Tue, Apr 28, 2015 at 10:19 PM, Justin Chang wrote: > Jason (or anyone), > > I am noticing that the iteration numbers reported by > TaoGetSolutionStatus() for blmvm differ whenever I change the number of > processes. The solution seems to remain the same though. Is there a reason > why this could be happening? > > Thanks, > > On Tue, Apr 21, 2015 at 10:40 AM, Jason Sarich > wrote: > >> Justin, >> >> 1) The big difference between TRON and BLMVM is that TRON requires >> hessian information, BLMVM only uses gradient information. Thus TRON will >> usually converge faster, but requires more information, memory, and a KSP >> solver. GPCG (gradient projected conjugate gradient) is another >> gradient-only option, but usually performs worse than BLMVM. >> >> 2) TaoGetLinearSolveIterations() will get the total number of KSP >> iterations per solve >> >> Jason >> >> >> On Tue, Apr 21, 2015 at 10:33 AM, Justin Chang >> wrote: >> >>> Jason, >>> >>> Tightening the tolerances did the trick. Thanks. Though I do have a >>> couple more related questions: >>> >>> 1) Is there a general guideline for choosing tron over blmvm or vice >>> versa? Also is there another tao type that is also suitable given only >>> bounded constraints? >>> >>> 2) Is it possible to obtain the total number of KSP and/or PG >>> iterations from tron? >>> >>> Thanks, >>> Justin >>> >>> On Tue, Apr 21, 2015 at 9:52 AM, Jason Sarich >>> wrote: >>> >>>> Hi Justin, >>>> >>>> blmvm believes that it is already sufficiently close to a minimum, so >>>> it doesn't do anything. You may need to tighten some of the tolerance to >>>> force an iteration. >>>> >>>> Jason >>>> >>>> >>>> On Tue, Apr 21, 2015 at 9:48 AM, Justin Chang >>>> wrote: >>>> >>>>> Time step 1: >>>>> >>>>> Tao Object: 1 MPI processes >>>>> type: blmvm >>>>> Gradient steps: 0 >>>>> TaoLineSearch Object: 1 MPI processes >>>>> type: more-thuente >>>>> Active Set subset type: subvec >>>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>>> Residual in Function/Gradient:=0.0663148 >>>>> Objective value=-55.5945 >>>>> total number of iterations=35, (max: 2000) >>>>> total number of function/gradient evaluations=37, (max: 4000) >>>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>>> >>>>> Time step 2: >>>>> >>>>> Tao Object: 1 MPI processes >>>>> type: blmvm >>>>> Gradient steps: 0 >>>>> TaoLineSearch Object: 1 MPI processes >>>>> type: more-thuente >>>>> Active Set subset type: subvec >>>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>>> Residual in Function/Gradient:=0.0682307 >>>>> Objective value=-66.9675 >>>>> total number of iterations=23, (max: 2000) >>>>> total number of function/gradient evaluations=25, (max: 4000) >>>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>>> >>>>> Time step 3: >>>>> >>>>> Tao Object: 1 MPI processes >>>>> type: blmvm >>>>> Gradient steps: 0 >>>>> TaoLineSearch Object: 1 MPI processes >>>>> type: more-thuente >>>>> Active Set subset type: subvec >>>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>>> Residual in Function/Gradient:=0.0680522 >>>>> Objective value=-71.8211 >>>>> total number of iterations=19, (max: 2000) >>>>> total number of function/gradient evaluations=22, (max: 4000) >>>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>>> >>>>> Time step 4: >>>>> >>>>> Tao Object: 1 MPI processes >>>>> type: blmvm >>>>> Gradient steps: 0 >>>>> TaoLineSearch Object: 1 MPI processes >>>>> type: more-thuente >>>>> Active Set subset type: subvec >>>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>>> Residual in Function/Gradient:=0.0551556 >>>>> Objective value=-75.1252 >>>>> total number of iterations=18, (max: 2000) >>>>> total number of function/gradient evaluations=20, (max: 4000) >>>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>>> >>>>> Time step 5: >>>>> >>>>> Tao Object: 1 MPI processes >>>>> type: blmvm >>>>> Gradient steps: 0 >>>>> TaoLineSearch Object: 1 MPI processes >>>>> type: more-thuente >>>>> Active Set subset type: subvec >>>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>>> Residual in Function/Gradient:=0.0675667 >>>>> Objective value=-77.4414 >>>>> total number of iterations=6, (max: 2000) >>>>> total number of function/gradient evaluations=8, (max: 4000) >>>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>>> >>>>> Time step 6: >>>>> >>>>> Tao Object: 1 MPI processes >>>>> type: blmvm >>>>> Gradient steps: 0 >>>>> TaoLineSearch Object: 1 MPI processes >>>>> type: more-thuente >>>>> Active Set subset type: subvec >>>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>>> Residual in Function/Gradient:=0.059143 >>>>> Objective value=-79.5007 >>>>> total number of iterations=3, (max: 2000) >>>>> total number of function/gradient evaluations=5, (max: 4000) >>>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>>> >>>>> Time step 7: >>>>> >>>>> Tao Object: 1 MPI processes >>>>> type: blmvm >>>>> Gradient steps: 0 >>>>> TaoLineSearch Object: 1 MPI processes >>>>> type: more-thuente >>>>> Active Set subset type: subvec >>>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>>> Residual in Function/Gradient:=0.0433683 >>>>> Objective value=-81.3546 >>>>> total number of iterations=5, (max: 2000) >>>>> total number of function/gradient evaluations=8, (max: 4000) >>>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>>> >>>>> Time step 8: >>>>> >>>>> Tao Object: 1 MPI processes >>>>> type: blmvm >>>>> Gradient steps: 0 >>>>> TaoLineSearch Object: 1 MPI processes >>>>> type: more-thuente >>>>> Active Set subset type: subvec >>>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>>> Residual in Function/Gradient:=0.0840676 >>>>> Objective value=-82.9382 >>>>> total number of iterations=0, (max: 2000) >>>>> total number of function/gradient evaluations=1, (max: 4000) >>>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>>> >>>>> Time step 9: >>>>> >>>>> Tao Object: 1 MPI processes >>>>> type: blmvm >>>>> Gradient steps: 0 >>>>> TaoLineSearch Object: 1 MPI processes >>>>> type: more-thuente >>>>> Active Set subset type: subvec >>>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>>> Residual in Function/Gradient:=0.0840676 >>>>> Objective value=-82.9382 >>>>> total number of iterations=0, (max: 2000) >>>>> total number of function/gradient evaluations=1, (max: 4000) >>>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>>> >>>>> Time step 10: >>>>> >>>>> Tao Object: 1 MPI processes >>>>> type: blmvm >>>>> Gradient steps: 0 >>>>> TaoLineSearch Object: 1 MPI processes >>>>> type: more-thuente >>>>> Active Set subset type: subvec >>>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>>> Residual in Function/Gradient:=0.0840676 >>>>> Objective value=-82.9382 >>>>> total number of iterations=0, (max: 2000) >>>>> total number of function/gradient evaluations=1, (max: 4000) >>>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Tue, Apr 21, 2015 at 9:28 AM, Jason Sarich >>>>> wrote: >>>>> >>>>>> Hi Justin, >>>>>> >>>>>> what reason is blmvm giving for stopping the solve? (you can use >>>>>> -tao_view or -tao_converged_reason to get this) >>>>>> >>>>>> Jason >>>>>> >>>>>> On Mon, Apr 20, 2015 at 6:32 PM, Justin Chang >>>>>> wrote: >>>>>> >>>>>>> Jason, >>>>>>> >>>>>>> I am using TaoGetSolutionStatus(tao,&its, ...) and it gives me >>>>>>> exactly what I want. However, I seem to be having an issue with blmvm >>>>>>> >>>>>>> I wrote my own backward euler code for a transient linear diffusion >>>>>>> problem with lower bounds >= 0 and upper bounds <= 1. For the first several >>>>>>> time steps I am getting its > 0, and it decreases over time due to the >>>>>>> nature of the discrete maximum principles. However, at some point my its >>>>>>> become 0 and the solution does not "update", which seems to me that >>>>>>> TaoSolve is not doing anything after that. This doesn't happen if I were to >>>>>>> use tron (my KSP and PC are cg and jacobi respectively). >>>>>>> >>>>>>> Do you know why this behavior may occur? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> On Tue, Apr 14, 2015 at 9:35 AM, Jason Sarich < >>>>>>> jason.sarich at gmail.com> wrote: >>>>>>> >>>>>>>> Hi Justin, >>>>>>>> >>>>>>>> I have pushed these changes to the "next" branch, your code >>>>>>>> snippet should work fine there. >>>>>>>> >>>>>>>> Note that there is also available (since version 3.5.0) the >>>>>>>> routine TaoGetSolutionStatus(tao,&its,NULL,NULL,NULL,NULL,NULL) which will >>>>>>>> provide the >>>>>>>> same information >>>>>>>> >>>>>>>> Jason >>>>>>>> >>>>>>>> On Fri, Apr 10, 2015 at 6:28 PM, Justin Chang >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Whatever is convenient and/or follow the "PETSc" standards. >>>>>>>>> Something similar to SNESGetIterationNumber() or KSPGetIterationNumber() >>>>>>>>> would be nice. Ideally I want my code to look like this: >>>>>>>>> >>>>>>>>> ierr = TaoGetIterationNumber(tao,&its);CHKERRQ(ierr); >>>>>>>>> ierr = PetscPrintf(PETSC_COMM_WORLD, "Number of Tao iterations = >>>>>>>>> %D\n", its); >>>>>>>>> >>>>>>>>> Thanks :) >>>>>>>>> >>>>>>>>> On Fri, Apr 10, 2015 at 5:53 PM, Jason Sarich < >>>>>>>>> jason.sarich at gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Justin, I'll get this in. I assume that displaying the number >>>>>>>>>> of iterations with tao_converged_reason is what you are asking for in >>>>>>>>>> particular? Or did you have something else in mind? >>>>>>>>>> >>>>>>>>>> Jason >>>>>>>>>> On Apr 10, 2015 16:42, "Smith, Barry F." >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Justin, >>>>>>>>>>> >>>>>>>>>>> Sorry TAO simply doesn't even collect this information >>>>>>>>>>> currently. But yes we should definitely make it available! >>>>>>>>>>> >>>>>>>>>>> Jason, >>>>>>>>>>> >>>>>>>>>>> Could you please add this; almost all the TaoSolve_xxx() >>>>>>>>>>> have a local variable iter; change that to tao->niter (I'm guess this is >>>>>>>>>>> suppose to capture this information) and add a TaoGetIterationNumber() and >>>>>>>>>>> the uses can access this. Also modify at the end of TaoSolve() >>>>>>>>>>> -tao_converged_reason to also print the iteration count. At the same time >>>>>>>>>>> since you add this you can add a tao->totalits which would accumulate all >>>>>>>>>>> iterations over all the solves for that Tao object and the routine >>>>>>>>>>> TaoGetTotalIterations() to access this. Note that TaoSolve() would >>>>>>>>>>> initialize tao->niter = 0 at the top. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>>> Barry >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> > On Apr 10, 2015, at 4:16 PM, Justin Chang >>>>>>>>>>> wrote: >>>>>>>>>>> > >>>>>>>>>>> > Hi all, >>>>>>>>>>> > >>>>>>>>>>> > Is there a way to generically obtain the number of Tao >>>>>>>>>>> iterations? I am looking through the -help options for Tao and I don't see >>>>>>>>>>> any metric where you can output this quantity in the manner that you could >>>>>>>>>>> for SNES or KSP solves. I am currently using blmvm and tron, and the only >>>>>>>>>>> way I can see getting this metric is by outputting -tao_view and/or >>>>>>>>>>> -tao_monitor and manually finding this number. I find this cumbersome >>>>>>>>>>> especially for transient problems where I would like to simply have this >>>>>>>>>>> number printed for each step instead of ending up with unnecessary info. >>>>>>>>>>> > >>>>>>>>>>> > Thanks, >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > -- >>>>>>>>>>> > Justin Chang >>>>>>>>>>> > PhD Candidate, Civil Engineering - Computational Sciences >>>>>>>>>>> > University of Houston, Department of Civil and Environmental >>>>>>>>>>> Engineering >>>>>>>>>>> > Houston, TX 77004 >>>>>>>>>>> > (512) 963-3262 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > > > -- > Justin Chang > PhD Candidate, Civil Engineering - Computational Sciences > University of Houston, Department of Civil and Environmental Engineering > Houston, TX 77004 > (512) 963-3262 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Wed Apr 29 12:15:25 2015 From: danyang.su at gmail.com (Danyang Su) Date: Wed, 29 Apr 2015 10:15:25 -0700 Subject: [petsc-users] Floating point exception in hypre BoomerAMG In-Reply-To: <6411F61F-CD41-4037-B727-B5D9CECE030F@mcs.anl.gov> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> <553C3E15.8030403@gmail.com> <553C3FE6.5020203@gmail.com> <553C4ECD.6090905@gmail.com> <553E6770.5010205@gmail.com> <553FC98D.1000904@gmail.com> <89DAC785-3218-4E2F-8760-2D0E99652014@mcs.anl.gov> <5540261E.2040502@gmail.com> <6411F61F-CD41-4037-B727-B5D9CECE030F@mcs.anl.gov> Message-ID: <554111AD.3060007@gmail.com> On 15-04-28 06:50 PM, Barry Smith wrote: > We started enforcing more checks on writing to vectors that you should not write to. Where are you calling DMLocalToGlobalBegin() ? It looks like you are trying to copy values into a global array that you should not because it is a read only input to a function, for example it is the right hand side of a linear system or the input into a SNESFormFunction(). So check the output to that call. > > Barry > This is used when local RHS vec is calculated and copied to global RHS. It is strange why this error does not occur at the first timestep. Thanks, Danyang >> On Apr 28, 2015, at 7:30 PM, Danyang Su wrote: >> >> Hi Barry, >> >> There seems another bug (not pretty sure) in PETSc-dev, as shown below. The case I used is similar with the one I mentioned recently. I have no problem running this case using PETSc 3.5.2. But it give out the following error using PETSc-dev, even with only one processor. >> >> Thanks, >> >> Danyang >> >> timestep: 2048 time: 4.615E+00 years delt: 1.460E-20 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Object is in wrong state >> [0]PETSC ERROR: Vec is locked read only, argument # 1 >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2796-g23b6c49 GIT Date: 2015-04-27 11:32:44 -0500 >> [0]PETSC ERROR: ../min3p_thcm on a linux-gnu-dbg named nwmop by dsu Tue Apr 28 15:56:41 2015 >> [0]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack >> [0]PETSC ERROR: #349 VecGetArray() line 1646 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/interface/rvector.c >> [0]PETSC ERROR: #350 VecGetArrayPair() line 434 in /home/dsu/Soft/PETSc/petsc-dev/include/petscvec.h >> [0]PETSC ERROR: #351 VecScatterBegin_SSToSS() line 649 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c >> [0]PETSC ERROR: #352 VecScatterBegin() line 1694 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c >> [0]PETSC ERROR: #353 DMLocalToGlobalBegin_DA() line 56 in /home/dsu/Soft/PETSc/petsc-dev/src/dm/impls/da/dagtol.c >> [0]PETSC ERROR: #354 DMLocalToGlobalBegin() line 1944 in /home/dsu/Soft/PETSc/petsc-dev/src/dm/interface/dm.c >> Reduce time step for reactive transport >> no further time step reduction possible >> Optimal time increment computed by MIN3P 2.6644769214899893E-018 >> Minimum time increment specified by user 3.6499999999999998E-018 >> Please, reduce the MINIMUM TIME INCREMENT >> stop signal in time step reduction failed >> >> On 15-04-28 11:15 AM, Barry Smith wrote: >>> I am forwarding this on to the hypre support email list; hopefully they can have some advice on how to proceed. >>> >>> hypre folks, this is with version hypre-2.10.0b (the previous version had the same behavior). We are getting a divide by zero in gselim() line 4363 (this happens in a time dependent problem after many successful solves with time dependent matrix). I looked at the code, below, and note that there are some checks for the diagonal of A not being zero but not for the line that causes the divide by zero; is this perhaps an oversight in the hypre code? Any advice appreciated. >>> >>> Thanks >>> >>> Barry >>> >>> >>> >>> >>> HYPRE_Int gselim(A,x,n) >>> HYPRE_Real *A; >>> HYPRE_Real *x; >>> HYPRE_Int n; >>> { >>> HYPRE_Int err_flag = 0; >>> HYPRE_Int j,k,m; >>> HYPRE_Real factor; >>> if (n==1) /* A is 1x1 */ >>> { >>> if (A[0] != 0.0) >>> { >>> x[0] = x[0]/A[0]; >>> return(err_flag); >>> } >>> else >>> { >>> err_flag = 1; >>> return(err_flag); >>> } >>> } >>> else /* A is nxn. Forward elimination */ >>> { >>> for (k = 0; k < n-1; k++) >>> { >>> if (A[k*n+k] != 0.0) >>> { >>> for (j = k+1; j < n; j++) >>> { >>> if (A[j*n+k] != 0.0) >>> { >>> factor = A[j*n+k]/A[k*n+k]; >>> for (m = k+1; m < n; m++) >>> { >>> A[j*n+m] -= factor * A[k*n+m]; >>> } >>> /* Elimination step for rhs */ >>> x[j] -= factor * x[k]; >>> } >>> } >>> } >>> } >>> /* Back Substitution */ >>> for (k = n-1; k > 0; --k) >>> { >>> x[k] /= A[k*n+k]; >>> for (j = 0; j < k; j++) >>> { >>> if (A[j*n+k] != 0.0) >>> { >>> x[j] -= x[k] * A[j*n+k]; >>> } >>> } >>> } >>> x[0] /= A[0]; >>> return(err_flag); >>> } >>> } >>> >>> >>>> On Apr 28, 2015, at 12:55 PM, Danyang Su wrote: >>>> >>>> Hi Barry, >>>> >>>> The development version of PETSc does not help to solve my problem. It still crashed due to the same error information. >>>> >>>> As Matthew mentioned, I checked the matrix and non of the diagonal entry is zero. But the diagonal values for different rows range from -1.0d-18 to 1.0d33 and the matrix is not strict diagonal dominant. When using 1 processor, the matrix is also not strict diagonal dominant, but the diagonal values have a much smaller range and the codes can run without error. Maybe there is something wrong in the matrix for this case and I need to check the matrix first. >>>> >>>> Thanks, >>>> >>>> Danyang >>>> >>>> Program received signal SIGFPE, Arithmetic exception. >>>> 0x00007f668c1299d5 in gselim (A=0x2fe3330, x=0x2210680, n=9) >>>> at par_relax.c:4363 >>>> 4363 x[k] /= A[k*n+k]; >>>> (gdb) >>>> >>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>> [1]PETSC ERROR: Try option -start_in_debugger >>>> [1]PETSC ERROR: likely location of problem given in stack below >>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>> [1]PETSC ERROR: is given. >>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-dev/src/sys/error/fp.c >>>> [1]PETSC ERROR: [1] Hypre solve line 216 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >>>> [1]PETSC ERROR: [1] PCApply_HYPRE line 203 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >>>> [1]PETSC ERROR: [1] PCApply line 430 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/interface/precon.c >>>> [1]PETSC ERROR: [1] KSP_PCApply line 234 /home/dsu/Soft/PETSc/petsc-dev/include/petsc/private/kspimpl.h >>>> [1]PETSC ERROR: [1] KSPInitialResidual line 44 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itres.c >>>> [1]PETSC ERROR: [1] KSPSolve_GMRES line 224 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/impls/gmres/gmres.c >>>> [1]PETSC ERROR: [1] KSPSolve line 506 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>> >>>> >>>> >>>> On 15-04-27 11:47 AM, Barry Smith wrote: >>>>> Danyang, >>>>> >>>>> I'm glad you were finally able to get to the problematic point; sorry it took so long (I want to strangle whoever turned on catching of underflows in PETSc). >>>>> >>>>> The hypre folks have done a great deal of work on their relaxation code since the PETSc 3.5.3 release. There is a good chance they have fixed the divide by zero problem you are hitting here. You will need to upgrade to the development version of PETSc (that uses the latest version of hypre), here are the instructions on how to obtain it http://www.mcs.anl.gov/petsc/developers/index.html >>>>> >>>>> Please let us know if this resolves the problem with hypre failing. >>>>> >>>>> Barry >>>>> >>>>> >>>>> >>>>>> On Apr 27, 2015, at 11:44 AM, Danyang Su wrote: >>>>>> >>>>>> Hi Barry, >>>>>> >>>>>> I got the following arithemetic exception after the previous bug is fixed. >>>>>> >>>>>> Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 >>>>>> 0x00007f3b23b98f20 in __nanosleep_nocancel () >>>>>> at ../sysdeps/unix/syscall-template.S:81 >>>>>> 81 ../sysdeps/unix/syscall-template.S: No such file or directory. >>>>>> (gdb) cont >>>>>> Continuing. >>>>>> >>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>> 0x00007f3b260df449 in gselim (A=0x1e4c580, x=0x1d30c40, n=9) >>>>>> at par_relax.c:3442 >>>>>> 3442 x[k] /= A[k*n+k]; >>>>>> (gdb) >>>>>> >>>>>> I tried both PETSc 3.5.2 and 3.5.3 and they return the same error as shown above. For 3.5.3, i edited fp.c file and then configure and make. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Danyang >>>>>> >>>>>> On 15-04-25 07:34 PM, Danyang Su wrote: >>>>>>> Hi All, >>>>>>> >>>>>>> The "floating point underflow" is caused by a small value divided by a very large value. This result is forced to zero and then it does not report any underflow problem. I just rerun this bad case to see if it still get stuck later. This will take a while. >>>>>>> >>>>>>> Thanks for all your kindly reply, >>>>>>> >>>>>>> Danyang >>>>>>> >>>>>>> On 15-04-25 07:02 PM, Barry Smith wrote: >>>>>>>> Ok, you do have >>>>>>>> >>>>>>>> #ifndef PETSC_HAVE_XMMINTRIN_H >>>>>>>> #define PETSC_HAVE_XMMINTRIN_H 1 >>>>>>>> #endif >>>>>>>> >>>>>>>> so the change you made should cause it to stop trapping underflow exceptions. >>>>>>>> >>>>>>>> Now in one email you reported a FPE within hypre, then I asked you to run with -start_in_debugger to determine where it happened exactly and then you reported the FPE happened in user code (what seemed to be an underflow issue). Why is this? Can you not run it where it generated the FPE in hypre using the -start_in_debugger option? >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> Perhaps you have multiple PETSC_ARCH or multiple PETSc installs to explain why you reported two different places where the exception occurred. >>>>>>>> >>>>>>>>> On Apr 25, 2015, at 8:31 PM, Danyang Su wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 15-04-25 06:26 PM, Matthew Knepley wrote: >>>>>>>>>> On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 15-04-25 06:03 PM, Barry Smith wrote: >>>>>>>>>> If this is what you got in your last run >>>>>>>>>> >>>>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>>>>> then it is still catching floating point underflow, which we do not want. This means either the change I suggested you make in the fp.c code didn't work or it actually uses a different floating point trap than that one. BTW: absurd numbers like 1.0d-316 are often a symptom of uninitialized data; could that be a problem that diff is not filled correctly for all the ic, idim you are using? >>>>>>>>>> >>>>>>>>>> This going round and round is very frustrating and a waste of time. You need to be more proactive yourself and explore the code and poke around to figure out how to solve the problem. >>>>>>>>>> >>>>>>>>>> Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can see what FP trap is being used on your machine. >>>>>>>>>> >>>>>>>>>> Barry >>>>>>>>>> Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise I cannot find this file. >>>>>>>>>> >>>>>>>>>> Its include/petscconf.h >>>>>>>>>> Do I need to reconfigure PETSc after changing the code you mentioned? >>>>>>>>>> >>>>>>>>>> No, but you need to rebuild. >>>>>>>>> Yes, I have done 'gnumake'. >>>>>>>>>> Matt >>>>>>>>>> Danyang >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Apr 25, 2015, at 2:24 PM, Danyang Su wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 15-04-25 11:55 AM, Barry Smith wrote: >>>>>>>>>> On Apr 25, 2015, at 1:51 PM, Danyang Su >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 15-04-25 11:32 AM, Barry Smith wrote: >>>>>>>>>> >>>>>>>>>> I told you this yesterday. >>>>>>>>>> >>>>>>>>>> It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. >>>>>>>>>> >>>>>>>>>> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate >>>>>>>>>> >>>>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>>>>>>>>> #else >>>>>>>>>> >>>>>>>>>> change it to >>>>>>>>>> >>>>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >>>>>>>>>> #else >>>>>>>>>> >>>>>>>>>> Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. >>>>>>>>>> >>>>>>>>>> Did you do this? >>>>>>>>>> >>>>>>>>>> Barry >>>>>>>>>> >>>>>>>>>> Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc directory. I just did a double check and ran make gnumake again and got the following information this time. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >>>>>>>>>> make gnumake >>>>>>>>>> Building PETSc using GNU Make with 10 build threads >>>>>>>>>> ========================================== >>>>>>>>>> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>>>> make[1]: Nothing to be done for `all'. >>>>>>>>>> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>>>> ========================================= >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Then I recompiled the codes, ran with -fp_trap and still got the following error >>>>>>>>>> >>>>>>>>>> Backtrace for this error: >>>>>>>>>> Note: The EXACT line numbers in the stack are not available, >>>>>>>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>> [2]PETSC ERROR: is given. >>>>>>>>>> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>> INSTEAD the line number of the start of the function >>>>>>>>>> [3]PETSC ERROR: is given. >>>>>>>>>> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>> [2]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>> [3]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> This is different then what you sent a few minutes ago where it crashed in hypre. >>>>>>>>>> >>>>>>>>>> Anyways you need to use the -start_in_debugger business I sent in the previous email to see the exact place the problem occurs. >>>>>>>>>> >>>>>>>>>> Here is the information shown on gdb screen >>>>>>>>>> >>>>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>>>> 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, njamxc=34, >>>>>>>>>> cinfradx=..., radial_coordx=.FALSE., _suffix=3) >>>>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>>>>> (gdb) >>>>>>>>>> >>>>>>>>>> After type cont on gdb screen, I got error information as below >>>>>>>>>> >>>>>>>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>> [1]PETSC ERROR: is given. >>>>>>>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Danyang >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Danyang >>>>>>>>>> >>>>>>>>>> On Apr 25, 2015, at 1:05 AM, Danyang Su >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi Barry and Satish, >>>>>>>>>> >>>>>>>>>> How can I get rid of unknown floating point error when a very small value is multiplied. >>>>>>>>>> >>>>>>>>>> e.g., >>>>>>>>>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, >>>>>>>>>> >>>>>>>>>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>>>>>>>>> >>>>>>>>>> I get the following error when run with "-fp_trap -start_in_debugger". >>>>>>>>>> >>>>>>>>>> Backtrace for this error: >>>>>>>>>> *** unknown floating point error occurred *** >>>>>>>>>> [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>> [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 >>>>>>>>>> where the result is a bitwise OR of the following flags: >>>>>>>>>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>> [2]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Danyang >>>>>>>>>> >>>>>>>>>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>>>>>>>> >>>>>>>>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>>>>>>>> >>>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>>> >>>>>>>>>> I don't think this number can be stored in a real*4. >>>>>>>>>> >>>>>>>>>> Satish >>>>>>>>>> >>>>>>>>>> Thanks, Satish. It is caused by this number. >>>>>>>>>> >>>>>>>>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>>>>>>> >>>>>>>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> One of my case crashes because of floating point exception when using 4 >>>>>>>>>> processors, as shown below. But if I run this case with 1 processor, it >>>>>>>>>> works fine. I have tested the codes with around 100 cases up to 768 >>>>>>>>>> processors, all other cases work fine. I just wonder if this kind of error >>>>>>>>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>>>>>>>> >>>>>>>>>> Yes, almost for sure it is one of these places. >>>>>>>>>> >>>>>>>>>> First run the bad case with -fp_trap if all goes well you'll see the >>>>>>>>>> function where the FPE is generated. Then run also with -start_in_debugger >>>>>>>>>> and >>>>>>>>>> type cont in all four debugger windows. When the FPE happens the debugger >>>>>>>>>> should stop showing exactly where the FPE happens. >>>>>>>>>> >>>>>>>>>> Barry >>>>>>>>>> >>>>>>>>>> Hi Barry, >>>>>>>>>> >>>>>>>>>> If run with -fp_trap -start_in_debugger, I got the following error >>>>>>>>>> >>>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a >>>>>>>>>> debugger. When the >>>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >>>>>>>>>> fetestexcept(0x3d) >>>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>>>>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>>>>>> ------------------------------------ >>>>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>> [0]PETSC ERROR: is given. >>>>>>>>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>>>>>>>>> floating point error >>>>>>>>>> >>>>>>>>>> Program received signal SIGABRT: Process abort signal. >>>>>>>>>> >>>>>>>>>> Backtrace for this error: >>>>>>>>>> #0 0x7F4FEAB1C7D7 >>>>>>>>>> #1 0x7F4FEAB1CDDE >>>>>>>>>> #2 0x7F4FE9E1AD3F >>>>>>>>>> #3 0x7F4FE9E1ACC9 >>>>>>>>>> #4 0x7F4FE9E1E0D7 >>>>>>>>>> #5 0x7F4FEB0B6DCB >>>>>>>>>> #6 0x7F4FEB0B1825 >>>>>>>>>> #7 0x7F4FEB0B817F >>>>>>>>>> #8 0x7F4FE9E1AD3F >>>>>>>>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>>>>>>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>>>>>>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>>>>>>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>>>>>>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>>>>>>>> >>>>>>>>>> I checked the code at tprfrtlc.F90:2393, >>>>>>>>>> >>>>>>>>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>>>>>>>> (cx(ix),ix=1,nxout)/) >>>>>>>>>> >>>>>>>>>> All the values (time, c, cx) are reasonable, as shown below. The only >>>>>>>>>> possibility is that realbuffer_gb is in declared as real*4 if using sing >>>>>>>>>> precision output while time, c, cx are declared in real*8. I have a lot of >>>>>>>>>> similar data conversion from real*8 to real*4 output, other code does not >>>>>>>>>> return error. >>>>>>>>>> >>>>>>>>>> time 0.0000000000000000 >>>>>>>>>> c 1 9.9999999999999995E-008 >>>>>>>>>> c 2 3.1555251077549618E-003 >>>>>>>>>> c 3 7.1657814842179362E-008 >>>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>>> c 5 5.2879822292305797E-004 >>>>>>>>>> c 6 9.9999999999999964E-005 >>>>>>>>>> c 7 6.4055731968811337E-005 >>>>>>>>>> c 8 3.4607572892578404E-020 >>>>>>>>>> cx 1 3.4376650636008101E-005 >>>>>>>>>> cx 2 7.3989678854017763E-012 >>>>>>>>>> cx 3 9.5317170613607207E-012 >>>>>>>>>> cx 4 2.2344525794718353E-015 >>>>>>>>>> cx 5 3.0624685689695889E-008 >>>>>>>>>> cx 6 1.0046157902783967E-007 >>>>>>>>>> cx 7 1.5320169154914984E-004 >>>>>>>>>> cx 8 8.6930292776346176E-014 >>>>>>>>>> cx 9 3.5944267559348721E-005 >>>>>>>>>> cx 10 3.0072645866951157E-018 >>>>>>>>>> cx 11 2.3592486321095017E-013 >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Danyang >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I can check all the entries of jacobi matrix to see if the value is valid, >>>>>>>>>> but this seems not a good idea as it takes a long time to reach this >>>>>>>>>> point. If I restart the simulation from a specified time (e.g., 7.685 in >>>>>>>>>> this case), then the error does not occur. >>>>>>>>>> >>>>>>>>>> Would you please give me any suggestion on debugging this case? >>>>>>>>>> >>>>>>>>>> Thanks and Regards, >>>>>>>>>> >>>>>>>>>> Danyang >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>> [1]PETSC ERROR: Floating point exception >>>>>>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>> [2]PETSC ERROR: Floating point exception >>>>>>>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >>>>>>>>>> at end of function: Parameter number 3 >>>>>>>>>> [2]PETSC ERROR: See >>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>> >>>>>>>>>> for trouble shooting. >>>>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>>>>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>>>>>>> [1]PETSC ERROR: See >>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>> >>>>>>>>>> for trouble shooting. >>>>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>>>>>>>>> dsu Thu Apr 23 15:38:52 2015 >>>>>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >>>>>>>>>> 15:38:52 2015 >>>>>>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>> #3 KSP_PCApply() line 230 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>>>>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>> From salazardetroya at gmail.com Wed Apr 29 12:57:51 2015 From: salazardetroya at gmail.com (Miguel Angel Salazar de Troya) Date: Wed, 29 Apr 2015 12:57:51 -0500 Subject: [petsc-users] Discrete adjoint and adaptive time stepping Message-ID: Hi I was wondering if your implementation to calculate sensitivities with discrete adjoints will account for the adaptive time stepping. The time step history depends on the parameters, so it should be included in the derivation of the discrete adjoint. Is this calculated in your implementation? If so, how is it done? Thanks in advance Miguel -- *Miguel Angel Salazar de Troya* Graduate Research Assistant Department of Mechanical Science and Engineering University of Illinois at Urbana-Champaign (217) 550-2360 salaza11 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Apr 29 13:30:58 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 29 Apr 2015 13:30:58 -0500 Subject: [petsc-users] Floating point exception in hypre BoomerAMG In-Reply-To: <554111AD.3060007@gmail.com> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> <553C3E15.8030403@gmail.com> <553C3FE6.5020203@gmail.com> <553C4ECD.6090905@gmail.com> <553E6770.5010205@gmail.com> <553FC98D.1000904@gmail.com> <89DAC785-3218-4E2F-8760-2D0E99652014@mcs.anl.gov> <5540261E.2040502@gmail.com> <6411F61F-CD41-4037-B727-B5D9CECE030F@mcs.anl.gov> <554111AD.3060007@gmail.com> Message-ID: <9C61B774-4A78-48E1-B34E-B23EC2D230B9@mcs.anl.gov> > On Apr 29, 2015, at 12:15 PM, Danyang Su wrote: > > On 15-04-28 06:50 PM, Barry Smith wrote: >> We started enforcing more checks on writing to vectors that you should not write to. Where are you calling DMLocalToGlobalBegin() ? It looks like you are trying to copy values into a global array that you should not because it is a read only input to a function, for example it is the right hand side of a linear system or the input into a SNESFormFunction(). So check the output to that call. >> >> Barry >> > This is used when local RHS vec is calculated and copied to global RHS. Huhh, why are you copying anything into the RHS? Is this before you call the linear system solve? Send the code that calls the DMLocalToGlobalBegin() Barry > It is strange why this error does not occur at the first timestep. > > Thanks, > > Danyang >>> On Apr 28, 2015, at 7:30 PM, Danyang Su wrote: >>> >>> Hi Barry, >>> >>> There seems another bug (not pretty sure) in PETSc-dev, as shown below. The case I used is similar with the one I mentioned recently. I have no problem running this case using PETSc 3.5.2. But it give out the following error using PETSc-dev, even with only one processor. >>> >>> Thanks, >>> >>> Danyang >>> >>> timestep: 2048 time: 4.615E+00 years delt: 1.460E-20 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [0]PETSC ERROR: Object is in wrong state >>> [0]PETSC ERROR: Vec is locked read only, argument # 1 >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2796-g23b6c49 GIT Date: 2015-04-27 11:32:44 -0500 >>> [0]PETSC ERROR: ../min3p_thcm on a linux-gnu-dbg named nwmop by dsu Tue Apr 28 15:56:41 2015 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack >>> [0]PETSC ERROR: #349 VecGetArray() line 1646 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/interface/rvector.c >>> [0]PETSC ERROR: #350 VecGetArrayPair() line 434 in /home/dsu/Soft/PETSc/petsc-dev/include/petscvec.h >>> [0]PETSC ERROR: #351 VecScatterBegin_SSToSS() line 649 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c >>> [0]PETSC ERROR: #352 VecScatterBegin() line 1694 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c >>> [0]PETSC ERROR: #353 DMLocalToGlobalBegin_DA() line 56 in /home/dsu/Soft/PETSc/petsc-dev/src/dm/impls/da/dagtol.c >>> [0]PETSC ERROR: #354 DMLocalToGlobalBegin() line 1944 in /home/dsu/Soft/PETSc/petsc-dev/src/dm/interface/dm.c >>> Reduce time step for reactive transport >>> no further time step reduction possible >>> Optimal time increment computed by MIN3P 2.6644769214899893E-018 >>> Minimum time increment specified by user 3.6499999999999998E-018 >>> Please, reduce the MINIMUM TIME INCREMENT >>> stop signal in time step reduction failed >>> >>> On 15-04-28 11:15 AM, Barry Smith wrote: >>>> I am forwarding this on to the hypre support email list; hopefully they can have some advice on how to proceed. >>>> >>>> hypre folks, this is with version hypre-2.10.0b (the previous version had the same behavior). We are getting a divide by zero in gselim() line 4363 (this happens in a time dependent problem after many successful solves with time dependent matrix). I looked at the code, below, and note that there are some checks for the diagonal of A not being zero but not for the line that causes the divide by zero; is this perhaps an oversight in the hypre code? Any advice appreciated. >>>> >>>> Thanks >>>> >>>> Barry >>>> >>>> >>>> >>>> >>>> HYPRE_Int gselim(A,x,n) >>>> HYPRE_Real *A; >>>> HYPRE_Real *x; >>>> HYPRE_Int n; >>>> { >>>> HYPRE_Int err_flag = 0; >>>> HYPRE_Int j,k,m; >>>> HYPRE_Real factor; >>>> if (n==1) /* A is 1x1 */ >>>> { >>>> if (A[0] != 0.0) >>>> { >>>> x[0] = x[0]/A[0]; >>>> return(err_flag); >>>> } >>>> else >>>> { >>>> err_flag = 1; >>>> return(err_flag); >>>> } >>>> } >>>> else /* A is nxn. Forward elimination */ >>>> { >>>> for (k = 0; k < n-1; k++) >>>> { >>>> if (A[k*n+k] != 0.0) >>>> { >>>> for (j = k+1; j < n; j++) >>>> { >>>> if (A[j*n+k] != 0.0) >>>> { >>>> factor = A[j*n+k]/A[k*n+k]; >>>> for (m = k+1; m < n; m++) >>>> { >>>> A[j*n+m] -= factor * A[k*n+m]; >>>> } >>>> /* Elimination step for rhs */ >>>> x[j] -= factor * x[k]; >>>> } >>>> } >>>> } >>>> } >>>> /* Back Substitution */ >>>> for (k = n-1; k > 0; --k) >>>> { >>>> x[k] /= A[k*n+k]; >>>> for (j = 0; j < k; j++) >>>> { >>>> if (A[j*n+k] != 0.0) >>>> { >>>> x[j] -= x[k] * A[j*n+k]; >>>> } >>>> } >>>> } >>>> x[0] /= A[0]; >>>> return(err_flag); >>>> } >>>> } >>>> >>>> >>>>> On Apr 28, 2015, at 12:55 PM, Danyang Su wrote: >>>>> >>>>> Hi Barry, >>>>> >>>>> The development version of PETSc does not help to solve my problem. It still crashed due to the same error information. >>>>> >>>>> As Matthew mentioned, I checked the matrix and non of the diagonal entry is zero. But the diagonal values for different rows range from -1.0d-18 to 1.0d33 and the matrix is not strict diagonal dominant. When using 1 processor, the matrix is also not strict diagonal dominant, but the diagonal values have a much smaller range and the codes can run without error. Maybe there is something wrong in the matrix for this case and I need to check the matrix first. >>>>> >>>>> Thanks, >>>>> >>>>> Danyang >>>>> >>>>> Program received signal SIGFPE, Arithmetic exception. >>>>> 0x00007f668c1299d5 in gselim (A=0x2fe3330, x=0x2210680, n=9) >>>>> at par_relax.c:4363 >>>>> 4363 x[k] /= A[k*n+k]; >>>>> (gdb) >>>>> >>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [1]PETSC ERROR: is given. >>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-dev/src/sys/error/fp.c >>>>> [1]PETSC ERROR: [1] Hypre solve line 216 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >>>>> [1]PETSC ERROR: [1] PCApply_HYPRE line 203 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >>>>> [1]PETSC ERROR: [1] PCApply line 430 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/interface/precon.c >>>>> [1]PETSC ERROR: [1] KSP_PCApply line 234 /home/dsu/Soft/PETSc/petsc-dev/include/petsc/private/kspimpl.h >>>>> [1]PETSC ERROR: [1] KSPInitialResidual line 44 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itres.c >>>>> [1]PETSC ERROR: [1] KSPSolve_GMRES line 224 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/impls/gmres/gmres.c >>>>> [1]PETSC ERROR: [1] KSPSolve line 506 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>> >>>>> >>>>> >>>>> On 15-04-27 11:47 AM, Barry Smith wrote: >>>>>> Danyang, >>>>>> >>>>>> I'm glad you were finally able to get to the problematic point; sorry it took so long (I want to strangle whoever turned on catching of underflows in PETSc). >>>>>> >>>>>> The hypre folks have done a great deal of work on their relaxation code since the PETSc 3.5.3 release. There is a good chance they have fixed the divide by zero problem you are hitting here. You will need to upgrade to the development version of PETSc (that uses the latest version of hypre), here are the instructions on how to obtain it http://www.mcs.anl.gov/petsc/developers/index.html >>>>>> >>>>>> Please let us know if this resolves the problem with hypre failing. >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>> >>>>>>> On Apr 27, 2015, at 11:44 AM, Danyang Su wrote: >>>>>>> >>>>>>> Hi Barry, >>>>>>> >>>>>>> I got the following arithemetic exception after the previous bug is fixed. >>>>>>> >>>>>>> Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 >>>>>>> 0x00007f3b23b98f20 in __nanosleep_nocancel () >>>>>>> at ../sysdeps/unix/syscall-template.S:81 >>>>>>> 81 ../sysdeps/unix/syscall-template.S: No such file or directory. >>>>>>> (gdb) cont >>>>>>> Continuing. >>>>>>> >>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>> 0x00007f3b260df449 in gselim (A=0x1e4c580, x=0x1d30c40, n=9) >>>>>>> at par_relax.c:3442 >>>>>>> 3442 x[k] /= A[k*n+k]; >>>>>>> (gdb) >>>>>>> >>>>>>> I tried both PETSc 3.5.2 and 3.5.3 and they return the same error as shown above. For 3.5.3, i edited fp.c file and then configure and make. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Danyang >>>>>>> >>>>>>> On 15-04-25 07:34 PM, Danyang Su wrote: >>>>>>>> Hi All, >>>>>>>> >>>>>>>> The "floating point underflow" is caused by a small value divided by a very large value. This result is forced to zero and then it does not report any underflow problem. I just rerun this bad case to see if it still get stuck later. This will take a while. >>>>>>>> >>>>>>>> Thanks for all your kindly reply, >>>>>>>> >>>>>>>> Danyang >>>>>>>> >>>>>>>> On 15-04-25 07:02 PM, Barry Smith wrote: >>>>>>>>> Ok, you do have >>>>>>>>> >>>>>>>>> #ifndef PETSC_HAVE_XMMINTRIN_H >>>>>>>>> #define PETSC_HAVE_XMMINTRIN_H 1 >>>>>>>>> #endif >>>>>>>>> >>>>>>>>> so the change you made should cause it to stop trapping underflow exceptions. >>>>>>>>> >>>>>>>>> Now in one email you reported a FPE within hypre, then I asked you to run with -start_in_debugger to determine where it happened exactly and then you reported the FPE happened in user code (what seemed to be an underflow issue). Why is this? Can you not run it where it generated the FPE in hypre using the -start_in_debugger option? >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> >>>>>>>>> Perhaps you have multiple PETSC_ARCH or multiple PETSc installs to explain why you reported two different places where the exception occurred. >>>>>>>>> >>>>>>>>>> On Apr 25, 2015, at 8:31 PM, Danyang Su wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 15-04-25 06:26 PM, Matthew Knepley wrote: >>>>>>>>>>> On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 15-04-25 06:03 PM, Barry Smith wrote: >>>>>>>>>>> If this is what you got in your last run >>>>>>>>>>> >>>>>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>>>>>> then it is still catching floating point underflow, which we do not want. This means either the change I suggested you make in the fp.c code didn't work or it actually uses a different floating point trap than that one. BTW: absurd numbers like 1.0d-316 are often a symptom of uninitialized data; could that be a problem that diff is not filled correctly for all the ic, idim you are using? >>>>>>>>>>> >>>>>>>>>>> This going round and round is very frustrating and a waste of time. You need to be more proactive yourself and explore the code and poke around to figure out how to solve the problem. >>>>>>>>>>> >>>>>>>>>>> Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can see what FP trap is being used on your machine. >>>>>>>>>>> >>>>>>>>>>> Barry >>>>>>>>>>> Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise I cannot find this file. >>>>>>>>>>> >>>>>>>>>>> Its include/petscconf.h >>>>>>>>>>> Do I need to reconfigure PETSc after changing the code you mentioned? >>>>>>>>>>> >>>>>>>>>>> No, but you need to rebuild. >>>>>>>>>> Yes, I have done 'gnumake'. >>>>>>>>>>> Matt >>>>>>>>>>> Danyang >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Apr 25, 2015, at 2:24 PM, Danyang Su wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 15-04-25 11:55 AM, Barry Smith wrote: >>>>>>>>>>> On Apr 25, 2015, at 1:51 PM, Danyang Su >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 15-04-25 11:32 AM, Barry Smith wrote: >>>>>>>>>>> >>>>>>>>>>> I told you this yesterday. >>>>>>>>>>> >>>>>>>>>>> It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. >>>>>>>>>>> >>>>>>>>>>> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate >>>>>>>>>>> >>>>>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>>>>>>>>>> #else >>>>>>>>>>> >>>>>>>>>>> change it to >>>>>>>>>>> >>>>>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >>>>>>>>>>> #else >>>>>>>>>>> >>>>>>>>>>> Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. >>>>>>>>>>> >>>>>>>>>>> Did you do this? >>>>>>>>>>> >>>>>>>>>>> Barry >>>>>>>>>>> >>>>>>>>>>> Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc directory. I just did a double check and ran make gnumake again and got the following information this time. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >>>>>>>>>>> make gnumake >>>>>>>>>>> Building PETSc using GNU Make with 10 build threads >>>>>>>>>>> ========================================== >>>>>>>>>>> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>>>>> make[1]: Nothing to be done for `all'. >>>>>>>>>>> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>>>>> ========================================= >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Then I recompiled the codes, ran with -fp_trap and still got the following error >>>>>>>>>>> >>>>>>>>>>> Backtrace for this error: >>>>>>>>>>> Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>> [2]PETSC ERROR: is given. >>>>>>>>>>> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>> INSTEAD the line number of the start of the function >>>>>>>>>>> [3]PETSC ERROR: is given. >>>>>>>>>>> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>> [2]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>> [3]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> This is different then what you sent a few minutes ago where it crashed in hypre. >>>>>>>>>>> >>>>>>>>>>> Anyways you need to use the -start_in_debugger business I sent in the previous email to see the exact place the problem occurs. >>>>>>>>>>> >>>>>>>>>>> Here is the information shown on gdb screen >>>>>>>>>>> >>>>>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>>>>> 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, njamxc=34, >>>>>>>>>>> cinfradx=..., radial_coordx=.FALSE., _suffix=3) >>>>>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>>>>>> (gdb) >>>>>>>>>>> >>>>>>>>>>> After type cont on gdb screen, I got error information as below >>>>>>>>>>> >>>>>>>>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>> [1]PETSC ERROR: is given. >>>>>>>>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Danyang >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Danyang >>>>>>>>>>> >>>>>>>>>>> On Apr 25, 2015, at 1:05 AM, Danyang Su >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Barry and Satish, >>>>>>>>>>> >>>>>>>>>>> How can I get rid of unknown floating point error when a very small value is multiplied. >>>>>>>>>>> >>>>>>>>>>> e.g., >>>>>>>>>>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, >>>>>>>>>>> >>>>>>>>>>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>>>>>>>>>> >>>>>>>>>>> I get the following error when run with "-fp_trap -start_in_debugger". >>>>>>>>>>> >>>>>>>>>>> Backtrace for this error: >>>>>>>>>>> *** unknown floating point error occurred *** >>>>>>>>>>> [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>> [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 >>>>>>>>>>> where the result is a bitwise OR of the following flags: >>>>>>>>>>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>> [2]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Danyang >>>>>>>>>>> >>>>>>>>>>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>>>>>>>>> >>>>>>>>>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>>>>>>>>> >>>>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>>>> >>>>>>>>>>> I don't think this number can be stored in a real*4. >>>>>>>>>>> >>>>>>>>>>> Satish >>>>>>>>>>> >>>>>>>>>>> Thanks, Satish. It is caused by this number. >>>>>>>>>>> >>>>>>>>>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>>>>>>>> >>>>>>>>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>> One of my case crashes because of floating point exception when using 4 >>>>>>>>>>> processors, as shown below. But if I run this case with 1 processor, it >>>>>>>>>>> works fine. I have tested the codes with around 100 cases up to 768 >>>>>>>>>>> processors, all other cases work fine. I just wonder if this kind of error >>>>>>>>>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>>>>>>>>> >>>>>>>>>>> Yes, almost for sure it is one of these places. >>>>>>>>>>> >>>>>>>>>>> First run the bad case with -fp_trap if all goes well you'll see the >>>>>>>>>>> function where the FPE is generated. Then run also with -start_in_debugger >>>>>>>>>>> and >>>>>>>>>>> type cont in all four debugger windows. When the FPE happens the debugger >>>>>>>>>>> should stop showing exactly where the FPE happens. >>>>>>>>>>> >>>>>>>>>>> Barry >>>>>>>>>>> >>>>>>>>>>> Hi Barry, >>>>>>>>>>> >>>>>>>>>>> If run with -fp_trap -start_in_debugger, I got the following error >>>>>>>>>>> >>>>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a >>>>>>>>>>> debugger. When the >>>>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >>>>>>>>>>> fetestexcept(0x3d) >>>>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>>>>>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>>>>>>> ------------------------------------ >>>>>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>> [0]PETSC ERROR: is given. >>>>>>>>>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>>>>>>>>>> floating point error >>>>>>>>>>> >>>>>>>>>>> Program received signal SIGABRT: Process abort signal. >>>>>>>>>>> >>>>>>>>>>> Backtrace for this error: >>>>>>>>>>> #0 0x7F4FEAB1C7D7 >>>>>>>>>>> #1 0x7F4FEAB1CDDE >>>>>>>>>>> #2 0x7F4FE9E1AD3F >>>>>>>>>>> #3 0x7F4FE9E1ACC9 >>>>>>>>>>> #4 0x7F4FE9E1E0D7 >>>>>>>>>>> #5 0x7F4FEB0B6DCB >>>>>>>>>>> #6 0x7F4FEB0B1825 >>>>>>>>>>> #7 0x7F4FEB0B817F >>>>>>>>>>> #8 0x7F4FE9E1AD3F >>>>>>>>>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>>>>>>>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>>>>>>>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>>>>>>>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>>>>>>>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>>>>>>>>> >>>>>>>>>>> I checked the code at tprfrtlc.F90:2393, >>>>>>>>>>> >>>>>>>>>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>>>>>>>>> (cx(ix),ix=1,nxout)/) >>>>>>>>>>> >>>>>>>>>>> All the values (time, c, cx) are reasonable, as shown below. The only >>>>>>>>>>> possibility is that realbuffer_gb is in declared as real*4 if using sing >>>>>>>>>>> precision output while time, c, cx are declared in real*8. I have a lot of >>>>>>>>>>> similar data conversion from real*8 to real*4 output, other code does not >>>>>>>>>>> return error. >>>>>>>>>>> >>>>>>>>>>> time 0.0000000000000000 >>>>>>>>>>> c 1 9.9999999999999995E-008 >>>>>>>>>>> c 2 3.1555251077549618E-003 >>>>>>>>>>> c 3 7.1657814842179362E-008 >>>>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>>>> c 5 5.2879822292305797E-004 >>>>>>>>>>> c 6 9.9999999999999964E-005 >>>>>>>>>>> c 7 6.4055731968811337E-005 >>>>>>>>>>> c 8 3.4607572892578404E-020 >>>>>>>>>>> cx 1 3.4376650636008101E-005 >>>>>>>>>>> cx 2 7.3989678854017763E-012 >>>>>>>>>>> cx 3 9.5317170613607207E-012 >>>>>>>>>>> cx 4 2.2344525794718353E-015 >>>>>>>>>>> cx 5 3.0624685689695889E-008 >>>>>>>>>>> cx 6 1.0046157902783967E-007 >>>>>>>>>>> cx 7 1.5320169154914984E-004 >>>>>>>>>>> cx 8 8.6930292776346176E-014 >>>>>>>>>>> cx 9 3.5944267559348721E-005 >>>>>>>>>>> cx 10 3.0072645866951157E-018 >>>>>>>>>>> cx 11 2.3592486321095017E-013 >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Danyang >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I can check all the entries of jacobi matrix to see if the value is valid, >>>>>>>>>>> but this seems not a good idea as it takes a long time to reach this >>>>>>>>>>> point. If I restart the simulation from a specified time (e.g., 7.685 in >>>>>>>>>>> this case), then the error does not occur. >>>>>>>>>>> >>>>>>>>>>> Would you please give me any suggestion on debugging this case? >>>>>>>>>>> >>>>>>>>>>> Thanks and Regards, >>>>>>>>>>> >>>>>>>>>>> Danyang >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>> [1]PETSC ERROR: Floating point exception >>>>>>>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>> [2]PETSC ERROR: Floating point exception >>>>>>>>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >>>>>>>>>>> at end of function: Parameter number 3 >>>>>>>>>>> [2]PETSC ERROR: See >>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>>> >>>>>>>>>>> for trouble shooting. >>>>>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>>>>>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>>>>>>>> [1]PETSC ERROR: See >>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>>> >>>>>>>>>>> for trouble shooting. >>>>>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>>>>>>>>>> dsu Thu Apr 23 15:38:52 2015 >>>>>>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >>>>>>>>>>> 15:38:52 2015 >>>>>>>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>>> #3 KSP_PCApply() line 230 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>>>>>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>>>> -- Norbert Wiener >>>>>>>>>> > From bsmith at mcs.anl.gov Wed Apr 29 13:44:42 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 29 Apr 2015 13:44:42 -0500 Subject: [petsc-users] Discrete adjoint and adaptive time stepping In-Reply-To: References: Message-ID: <8A885376-C369-4E9E-AAC0-140930EA3CED@mcs.anl.gov> > On Apr 29, 2015, at 12:57 PM, Miguel Angel Salazar de Troya wrote: > > Hi > > I was wondering if your implementation to calculate sensitivities with discrete adjoints will account for the adaptive time stepping. The time step history depends on the parameters, so it should be included in the derivation of the discrete adjoint. Is this calculated in your implementation? Sure, > If so, how is it done? We just keep the history of the time-step sizes and then use those time-step sizes when doing the backward integration. Seems simple to me, am I missing something? Barry > > Thanks in advance > Miguel > > -- > Miguel Angel Salazar de Troya > Graduate Research Assistant > Department of Mechanical Science and Engineering > University of Illinois at Urbana-Champaign > (217) 550-2360 > salaza11 at illinois.edu > From jed at jedbrown.org Wed Apr 29 13:51:49 2015 From: jed at jedbrown.org (Jed Brown) Date: Wed, 29 Apr 2015 12:51:49 -0600 Subject: [petsc-users] Discrete adjoint and adaptive time stepping In-Reply-To: <8A885376-C369-4E9E-AAC0-140930EA3CED@mcs.anl.gov> References: <8A885376-C369-4E9E-AAC0-140930EA3CED@mcs.anl.gov> Message-ID: <871tj2py2i.fsf@jedbrown.org> Barry Smith writes: >> If so, how is it done? > > We just keep the history of the time-step sizes and then use those time-step sizes when doing the backward integration. Seems simple to me, am I missing something? Barry, if you're using this for optimization, you might want the gradient to be exactly consistent with the objective functional. But for that, you would need to differentiate the controller, which is non-smooth in practice because the number of time steps can change and stages could be rejected (solver failure). One approach would be to save the timestep sequence and have the controller use that in subsequent *forward* runs. If the dynamical system behaves similarly for those steps, it would be okay to use the same timestep sequence. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From danyang.su at gmail.com Wed Apr 29 13:52:56 2015 From: danyang.su at gmail.com (Danyang Su) Date: Wed, 29 Apr 2015 11:52:56 -0700 Subject: [petsc-users] Floating point exception in hypre BoomerAMG In-Reply-To: <9C61B774-4A78-48E1-B34E-B23EC2D230B9@mcs.anl.gov> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> <553C3E15.8030403@gmail.com> <553C3FE6.5020203@gmail.com> <553C4ECD.6090905@gmail.com> <553E6770.5010205@gmail.com> <553FC98D.1000904@gmail.com> <89DAC785-3218-4E2F-8760-2D0E99652014@mcs.anl.gov> <5540261E.2040502@gmail.com> <6411F61F-CD41-4037-B727-B5D9CECE030F@mcs.anl.gov> <554111AD.3060007@gmail.com> <9C61B774-4A78-48E1-B34E-B23EC2D230B9@mcs.anl.gov> Message-ID: <55412888.90206@gmail.com> On 15-04-29 11:30 AM, Barry Smith wrote: >> On Apr 29, 2015, at 12:15 PM, Danyang Su wrote: >> >> On 15-04-28 06:50 PM, Barry Smith wrote: >>> We started enforcing more checks on writing to vectors that you should not write to. Where are you calling DMLocalToGlobalBegin() ? It looks like you are trying to copy values into a global array that you should not because it is a read only input to a function, for example it is the right hand side of a linear system or the input into a SNESFormFunction(). So check the output to that call. >>> >>> Barry >>> >> This is used when local RHS vec is calculated and copied to global RHS. > Huhh, why are you copying anything into the RHS? Is this before you call the linear system solve? Send the code that calls the DMLocalToGlobalBegin() Yes. This is called before the linear system solver. At present, we use PETSc KSP solver. The code is first developed in totally sequential version and then ported to the parallel version. So the subdomain has its own space for jacobi matrix and rhs. We do not use global RHS vector directly to set the RHS at this moment. All the RHS values of subdomain are copied to the global RHS. !> This is the function where RHS values of subdomain are copied to the global RHS values. subroutine compute_function(rank,da,x_array_loc,x_vec_loc, & x_vec_gbl,nngl,row_idx_l2pg,col_idx_l2pg, & b_non_interlaced) implicit none #include #include #include #include #include #include #include #include #include PetscInt :: rank DM :: da PetscReal, allocatable :: x_array_loc(:) Vec :: x_vec_loc Vec :: x_vec_gbl PetscInt :: nngl PetscInt, allocatable :: row_idx_l2pg(:) PetscInt, allocatable :: col_idx_l2pg(:) PetscBool :: b_non_interlaced PetscInt :: info_debug PetscInt :: i, j PetscErrorCode :: ierr PetscScalar, pointer :: vecpointer(:) !Zero entries call VecZeroEntries(x_vec_loc, ierr) !Get a pointer to vector data when you need access to the array call VecGetArrayF90(x_vec_loc, vecpointer, ierr) !Compute the function over the locally owned part of the grid if(b_non_interlaced) then j = nngl/2 do i = 1, j vecpointer(2*i-1) = x_array_loc(i) vecpointer(2*i) = x_array_loc(i+j) end do else do i = 1, nngl vecpointer(i) = x_array_loc(i) end do end if !Restore the vector when you no longer need access to the array call VecRestoreArrayF90(x_vec_loc,vecpointer,ierr) !Insert values into global vector call DMLocalToGlobalBegin(da,x_vec_loc,INSERT_VALUES, & x_vec_gbl,ierr) !By placing code between these two statements, computations can be !done while messages are in transition. call DMLocalToGlobalEnd(da,x_vec_loc,INSERT_VALUES, & x_vec_gbl,ierr) return end subroutine !> This is the function where reactive transport equations are solved. subroutine solver_dd_snes_solve_react(ilog,idetail,a_in,b_in, & x_inout,ia_in,ja_in,nngl_in,itsolv, & over_flow,rnorm,row_idx_l2pg,col_idx_l2pg, & b_non_interlaced) use gen, only : rank, node_idx_l2lg, ittot_rt, & b_output_matrix, b_enable_output use solver_snes_function, only : form_initial_guess, & compute_function, & compute_jacobian use petsc_mpi_common, only : petsc_mpi_finalize implicit none #include #include #include #include PetscInt :: ilog PetscInt :: idetail PetscInt :: nngl_in PetscReal, allocatable :: a_in(:) PetscReal, allocatable :: b_in(:) PetscReal, allocatable :: x_inout(:) PetscInt, allocatable :: ia_in(:) PetscInt, allocatable :: ja_in(:) PetscInt, allocatable :: row_idx_l2pg(:) PetscInt, allocatable :: col_idx_l2pg(:) PetscInt :: itsolv PetscBool :: over_flow PetscBool :: b_non_interlaced PetscReal :: rnorm PetscViewer :: viewer PetscScalar,pointer :: vecpointer(:) PetscErrorCode :: ierr PetscInt :: info_debug character(72) :: strinum info_debug = 0 if(b_output_matrix .and. b_enable_output) then write(strinum, *) ittot_rt strinum = "_"//trim(adjustl(strinum)) end if !Form initial guess, only assemble the local owned part, without ghost nodes call form_initial_guess(rank,dmda_react%da,x_inout,x_react_loc,& x_react,nngl_in, row_idx_l2pg,col_idx_l2pg, & b_non_interlaced) !Compute function, only assemble the local part, without ghost nodes call compute_function(rank,dmda_react%da,b_in,b_react_loc, & b_react,nngl_in,row_idx_l2pg,col_idx_l2pg, & b_non_interlaced) !Compute jacobian matrix, only assemble the local part, without ghost nodes call compute_jacobian(rank,dmda_react%da, & a_react,a_in,ia_in,ja_in,nngl_in, & row_idx_l2pg,col_idx_l2pg, & b_non_interlaced) ! Solve a x = b, where a is the Jacobian matrix. ! - First, set the KSP linear operators. Here the matrix that ! defines the linear system also serves as the preconditioning ! matrix. ! - Then solve the Newton system. #ifdef PETSC_V3_5_X call KSPSetOperators(ksp_react,a_react,a_react,ierr) #else call KSPSetOperators(ksp_react,a_react,a_react, & SAME_NONZERO_PATTERN,ierr) #endif call KSPSetDM(ksp_react,dmda_react%da,ierr) call KSPSetDMActive(ksp_react,PETSC_FALSE,ierr) call KSPSolve(ksp_react,b_react,x_react,ierr) !Get residual norm, by default, preconditioned residual norm is calculated. if (b_mykspconverged_react .and. & .not. b_use_petsc_default_react) then rnorm = rnorm_react else call KSPGetResidualNorm(ksp_react, rnorm, ierr) end if ! Scatter ghost points to local vector, using the 2-step process ! DMGlobalToLocalBegin(), DMGlobalToLocalEnd(). ! By placing code between these two statements, computations can be ! done while messages are in transition. call DMGlobalToLocalBegin(dmda_react%da,x_react,INSERT_VALUES, & x_react_loc,ierr) call DMGlobalToLocalEnd(dmda_react%da,x_react,INSERT_VALUES, & x_react_loc,ierr) call VecGetArrayF90(x_react_loc,vecpointer,ierr) x_inout = vecpointer call VecRestoreArrayF90(x_react_loc,vecpointer,ierr) return end subroutine > > Barry > >> It is strange why this error does not occur at the first timestep. >> >> Thanks, >> >> Danyang >>>> On Apr 28, 2015, at 7:30 PM, Danyang Su wrote: >>>> >>>> Hi Barry, >>>> >>>> There seems another bug (not pretty sure) in PETSc-dev, as shown below. The case I used is similar with the one I mentioned recently. I have no problem running this case using PETSc 3.5.2. But it give out the following error using PETSc-dev, even with only one processor. >>>> >>>> Thanks, >>>> >>>> Danyang >>>> >>>> timestep: 2048 time: 4.615E+00 years delt: 1.460E-20 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>> [0]PETSC ERROR: Object is in wrong state >>>> [0]PETSC ERROR: Vec is locked read only, argument # 1 >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2796-g23b6c49 GIT Date: 2015-04-27 11:32:44 -0500 >>>> [0]PETSC ERROR: ../min3p_thcm on a linux-gnu-dbg named nwmop by dsu Tue Apr 28 15:56:41 2015 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack >>>> [0]PETSC ERROR: #349 VecGetArray() line 1646 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/interface/rvector.c >>>> [0]PETSC ERROR: #350 VecGetArrayPair() line 434 in /home/dsu/Soft/PETSc/petsc-dev/include/petscvec.h >>>> [0]PETSC ERROR: #351 VecScatterBegin_SSToSS() line 649 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c >>>> [0]PETSC ERROR: #352 VecScatterBegin() line 1694 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c >>>> [0]PETSC ERROR: #353 DMLocalToGlobalBegin_DA() line 56 in /home/dsu/Soft/PETSc/petsc-dev/src/dm/impls/da/dagtol.c >>>> [0]PETSC ERROR: #354 DMLocalToGlobalBegin() line 1944 in /home/dsu/Soft/PETSc/petsc-dev/src/dm/interface/dm.c >>>> Reduce time step for reactive transport >>>> no further time step reduction possible >>>> Optimal time increment computed by MIN3P 2.6644769214899893E-018 >>>> Minimum time increment specified by user 3.6499999999999998E-018 >>>> Please, reduce the MINIMUM TIME INCREMENT >>>> stop signal in time step reduction failed >>>> >>>> On 15-04-28 11:15 AM, Barry Smith wrote: >>>>> I am forwarding this on to the hypre support email list; hopefully they can have some advice on how to proceed. >>>>> >>>>> hypre folks, this is with version hypre-2.10.0b (the previous version had the same behavior). We are getting a divide by zero in gselim() line 4363 (this happens in a time dependent problem after many successful solves with time dependent matrix). I looked at the code, below, and note that there are some checks for the diagonal of A not being zero but not for the line that causes the divide by zero; is this perhaps an oversight in the hypre code? Any advice appreciated. >>>>> >>>>> Thanks >>>>> >>>>> Barry >>>>> >>>>> >>>>> >>>>> >>>>> HYPRE_Int gselim(A,x,n) >>>>> HYPRE_Real *A; >>>>> HYPRE_Real *x; >>>>> HYPRE_Int n; >>>>> { >>>>> HYPRE_Int err_flag = 0; >>>>> HYPRE_Int j,k,m; >>>>> HYPRE_Real factor; >>>>> if (n==1) /* A is 1x1 */ >>>>> { >>>>> if (A[0] != 0.0) >>>>> { >>>>> x[0] = x[0]/A[0]; >>>>> return(err_flag); >>>>> } >>>>> else >>>>> { >>>>> err_flag = 1; >>>>> return(err_flag); >>>>> } >>>>> } >>>>> else /* A is nxn. Forward elimination */ >>>>> { >>>>> for (k = 0; k < n-1; k++) >>>>> { >>>>> if (A[k*n+k] != 0.0) >>>>> { >>>>> for (j = k+1; j < n; j++) >>>>> { >>>>> if (A[j*n+k] != 0.0) >>>>> { >>>>> factor = A[j*n+k]/A[k*n+k]; >>>>> for (m = k+1; m < n; m++) >>>>> { >>>>> A[j*n+m] -= factor * A[k*n+m]; >>>>> } >>>>> /* Elimination step for rhs */ >>>>> x[j] -= factor * x[k]; >>>>> } >>>>> } >>>>> } >>>>> } >>>>> /* Back Substitution */ >>>>> for (k = n-1; k > 0; --k) >>>>> { >>>>> x[k] /= A[k*n+k]; >>>>> for (j = 0; j < k; j++) >>>>> { >>>>> if (A[j*n+k] != 0.0) >>>>> { >>>>> x[j] -= x[k] * A[j*n+k]; >>>>> } >>>>> } >>>>> } >>>>> x[0] /= A[0]; >>>>> return(err_flag); >>>>> } >>>>> } >>>>> >>>>> >>>>>> On Apr 28, 2015, at 12:55 PM, Danyang Su wrote: >>>>>> >>>>>> Hi Barry, >>>>>> >>>>>> The development version of PETSc does not help to solve my problem. It still crashed due to the same error information. >>>>>> >>>>>> As Matthew mentioned, I checked the matrix and non of the diagonal entry is zero. But the diagonal values for different rows range from -1.0d-18 to 1.0d33 and the matrix is not strict diagonal dominant. When using 1 processor, the matrix is also not strict diagonal dominant, but the diagonal values have a much smaller range and the codes can run without error. Maybe there is something wrong in the matrix for this case and I need to check the matrix first. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Danyang >>>>>> >>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>> 0x00007f668c1299d5 in gselim (A=0x2fe3330, x=0x2210680, n=9) >>>>>> at par_relax.c:4363 >>>>>> 4363 x[k] /= A[k*n+k]; >>>>>> (gdb) >>>>>> >>>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>> [1]PETSC ERROR: is given. >>>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-dev/src/sys/error/fp.c >>>>>> [1]PETSC ERROR: [1] Hypre solve line 216 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >>>>>> [1]PETSC ERROR: [1] PCApply_HYPRE line 203 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >>>>>> [1]PETSC ERROR: [1] PCApply line 430 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/interface/precon.c >>>>>> [1]PETSC ERROR: [1] KSP_PCApply line 234 /home/dsu/Soft/PETSc/petsc-dev/include/petsc/private/kspimpl.h >>>>>> [1]PETSC ERROR: [1] KSPInitialResidual line 44 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itres.c >>>>>> [1]PETSC ERROR: [1] KSPSolve_GMRES line 224 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/impls/gmres/gmres.c >>>>>> [1]PETSC ERROR: [1] KSPSolve line 506 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>> >>>>>> >>>>>> >>>>>> On 15-04-27 11:47 AM, Barry Smith wrote: >>>>>>> Danyang, >>>>>>> >>>>>>> I'm glad you were finally able to get to the problematic point; sorry it took so long (I want to strangle whoever turned on catching of underflows in PETSc). >>>>>>> >>>>>>> The hypre folks have done a great deal of work on their relaxation code since the PETSc 3.5.3 release. There is a good chance they have fixed the divide by zero problem you are hitting here. You will need to upgrade to the development version of PETSc (that uses the latest version of hypre), here are the instructions on how to obtain it http://www.mcs.anl.gov/petsc/developers/index.html >>>>>>> >>>>>>> Please let us know if this resolves the problem with hypre failing. >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On Apr 27, 2015, at 11:44 AM, Danyang Su wrote: >>>>>>>> >>>>>>>> Hi Barry, >>>>>>>> >>>>>>>> I got the following arithemetic exception after the previous bug is fixed. >>>>>>>> >>>>>>>> Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 >>>>>>>> 0x00007f3b23b98f20 in __nanosleep_nocancel () >>>>>>>> at ../sysdeps/unix/syscall-template.S:81 >>>>>>>> 81 ../sysdeps/unix/syscall-template.S: No such file or directory. >>>>>>>> (gdb) cont >>>>>>>> Continuing. >>>>>>>> >>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>> 0x00007f3b260df449 in gselim (A=0x1e4c580, x=0x1d30c40, n=9) >>>>>>>> at par_relax.c:3442 >>>>>>>> 3442 x[k] /= A[k*n+k]; >>>>>>>> (gdb) >>>>>>>> >>>>>>>> I tried both PETSc 3.5.2 and 3.5.3 and they return the same error as shown above. For 3.5.3, i edited fp.c file and then configure and make. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Danyang >>>>>>>> >>>>>>>> On 15-04-25 07:34 PM, Danyang Su wrote: >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> The "floating point underflow" is caused by a small value divided by a very large value. This result is forced to zero and then it does not report any underflow problem. I just rerun this bad case to see if it still get stuck later. This will take a while. >>>>>>>>> >>>>>>>>> Thanks for all your kindly reply, >>>>>>>>> >>>>>>>>> Danyang >>>>>>>>> >>>>>>>>> On 15-04-25 07:02 PM, Barry Smith wrote: >>>>>>>>>> Ok, you do have >>>>>>>>>> >>>>>>>>>> #ifndef PETSC_HAVE_XMMINTRIN_H >>>>>>>>>> #define PETSC_HAVE_XMMINTRIN_H 1 >>>>>>>>>> #endif >>>>>>>>>> >>>>>>>>>> so the change you made should cause it to stop trapping underflow exceptions. >>>>>>>>>> >>>>>>>>>> Now in one email you reported a FPE within hypre, then I asked you to run with -start_in_debugger to determine where it happened exactly and then you reported the FPE happened in user code (what seemed to be an underflow issue). Why is this? Can you not run it where it generated the FPE in hypre using the -start_in_debugger option? >>>>>>>>>> >>>>>>>>>> Barry >>>>>>>>>> >>>>>>>>>> Perhaps you have multiple PETSC_ARCH or multiple PETSc installs to explain why you reported two different places where the exception occurred. >>>>>>>>>> >>>>>>>>>>> On Apr 25, 2015, at 8:31 PM, Danyang Su wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 15-04-25 06:26 PM, Matthew Knepley wrote: >>>>>>>>>>>> On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 15-04-25 06:03 PM, Barry Smith wrote: >>>>>>>>>>>> If this is what you got in your last run >>>>>>>>>>>> >>>>>>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>>>>>>> then it is still catching floating point underflow, which we do not want. This means either the change I suggested you make in the fp.c code didn't work or it actually uses a different floating point trap than that one. BTW: absurd numbers like 1.0d-316 are often a symptom of uninitialized data; could that be a problem that diff is not filled correctly for all the ic, idim you are using? >>>>>>>>>>>> >>>>>>>>>>>> This going round and round is very frustrating and a waste of time. You need to be more proactive yourself and explore the code and poke around to figure out how to solve the problem. >>>>>>>>>>>> >>>>>>>>>>>> Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can see what FP trap is being used on your machine. >>>>>>>>>>>> >>>>>>>>>>>> Barry >>>>>>>>>>>> Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise I cannot find this file. >>>>>>>>>>>> >>>>>>>>>>>> Its include/petscconf.h >>>>>>>>>>>> Do I need to reconfigure PETSc after changing the code you mentioned? >>>>>>>>>>>> >>>>>>>>>>>> No, but you need to rebuild. >>>>>>>>>>> Yes, I have done 'gnumake'. >>>>>>>>>>>> Matt >>>>>>>>>>>> Danyang >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Apr 25, 2015, at 2:24 PM, Danyang Su wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 15-04-25 11:55 AM, Barry Smith wrote: >>>>>>>>>>>> On Apr 25, 2015, at 1:51 PM, Danyang Su >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 15-04-25 11:32 AM, Barry Smith wrote: >>>>>>>>>>>> >>>>>>>>>>>> I told you this yesterday. >>>>>>>>>>>> >>>>>>>>>>>> It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. >>>>>>>>>>>> >>>>>>>>>>>> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate >>>>>>>>>>>> >>>>>>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>>>>>>>>>>> #else >>>>>>>>>>>> >>>>>>>>>>>> change it to >>>>>>>>>>>> >>>>>>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >>>>>>>>>>>> #else >>>>>>>>>>>> >>>>>>>>>>>> Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. >>>>>>>>>>>> >>>>>>>>>>>> Did you do this? >>>>>>>>>>>> >>>>>>>>>>>> Barry >>>>>>>>>>>> >>>>>>>>>>>> Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc directory. I just did a double check and ran make gnumake again and got the following information this time. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >>>>>>>>>>>> make gnumake >>>>>>>>>>>> Building PETSc using GNU Make with 10 build threads >>>>>>>>>>>> ========================================== >>>>>>>>>>>> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>>>>>> make[1]: Nothing to be done for `all'. >>>>>>>>>>>> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>>>>>> ========================================= >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Then I recompiled the codes, ran with -fp_trap and still got the following error >>>>>>>>>>>> >>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>> Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>>> [2]PETSC ERROR: is given. >>>>>>>>>>>> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>> INSTEAD the line number of the start of the function >>>>>>>>>>>> [3]PETSC ERROR: is given. >>>>>>>>>>>> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>> [2]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>>> [3]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> This is different then what you sent a few minutes ago where it crashed in hypre. >>>>>>>>>>>> >>>>>>>>>>>> Anyways you need to use the -start_in_debugger business I sent in the previous email to see the exact place the problem occurs. >>>>>>>>>>>> >>>>>>>>>>>> Here is the information shown on gdb screen >>>>>>>>>>>> >>>>>>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>>>>>> 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, njamxc=34, >>>>>>>>>>>> cinfradx=..., radial_coordx=.FALSE., _suffix=3) >>>>>>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>>>>>>> (gdb) >>>>>>>>>>>> >>>>>>>>>>>> After type cont on gdb screen, I got error information as below >>>>>>>>>>>> >>>>>>>>>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>>> [1]PETSC ERROR: is given. >>>>>>>>>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Danyang >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Danyang >>>>>>>>>>>> >>>>>>>>>>>> On Apr 25, 2015, at 1:05 AM, Danyang Su >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi Barry and Satish, >>>>>>>>>>>> >>>>>>>>>>>> How can I get rid of unknown floating point error when a very small value is multiplied. >>>>>>>>>>>> >>>>>>>>>>>> e.g., >>>>>>>>>>>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, >>>>>>>>>>>> >>>>>>>>>>>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>>>>>>>>>>> >>>>>>>>>>>> I get the following error when run with "-fp_trap -start_in_debugger". >>>>>>>>>>>> >>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>> *** unknown floating point error occurred *** >>>>>>>>>>>> [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>>> [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 >>>>>>>>>>>> where the result is a bitwise OR of the following flags: >>>>>>>>>>>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>> [2]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Danyang >>>>>>>>>>>> >>>>>>>>>>>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>>>>>>>>>> >>>>>>>>>>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>>>>>>>>>> >>>>>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>>>>> >>>>>>>>>>>> I don't think this number can be stored in a real*4. >>>>>>>>>>>> >>>>>>>>>>>> Satish >>>>>>>>>>>> >>>>>>>>>>>> Thanks, Satish. It is caused by this number. >>>>>>>>>>>> >>>>>>>>>>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>>>>>>>>> >>>>>>>>>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi All, >>>>>>>>>>>> >>>>>>>>>>>> One of my case crashes because of floating point exception when using 4 >>>>>>>>>>>> processors, as shown below. But if I run this case with 1 processor, it >>>>>>>>>>>> works fine. I have tested the codes with around 100 cases up to 768 >>>>>>>>>>>> processors, all other cases work fine. I just wonder if this kind of error >>>>>>>>>>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>>>>>>>>>> >>>>>>>>>>>> Yes, almost for sure it is one of these places. >>>>>>>>>>>> >>>>>>>>>>>> First run the bad case with -fp_trap if all goes well you'll see the >>>>>>>>>>>> function where the FPE is generated. Then run also with -start_in_debugger >>>>>>>>>>>> and >>>>>>>>>>>> type cont in all four debugger windows. When the FPE happens the debugger >>>>>>>>>>>> should stop showing exactly where the FPE happens. >>>>>>>>>>>> >>>>>>>>>>>> Barry >>>>>>>>>>>> >>>>>>>>>>>> Hi Barry, >>>>>>>>>>>> >>>>>>>>>>>> If run with -fp_trap -start_in_debugger, I got the following error >>>>>>>>>>>> >>>>>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a >>>>>>>>>>>> debugger. When the >>>>>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >>>>>>>>>>>> fetestexcept(0x3d) >>>>>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>>>>>>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>>>>>>>> ------------------------------------ >>>>>>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>>> [0]PETSC ERROR: is given. >>>>>>>>>>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>>>>>>>>>>> floating point error >>>>>>>>>>>> >>>>>>>>>>>> Program received signal SIGABRT: Process abort signal. >>>>>>>>>>>> >>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>> #0 0x7F4FEAB1C7D7 >>>>>>>>>>>> #1 0x7F4FEAB1CDDE >>>>>>>>>>>> #2 0x7F4FE9E1AD3F >>>>>>>>>>>> #3 0x7F4FE9E1ACC9 >>>>>>>>>>>> #4 0x7F4FE9E1E0D7 >>>>>>>>>>>> #5 0x7F4FEB0B6DCB >>>>>>>>>>>> #6 0x7F4FEB0B1825 >>>>>>>>>>>> #7 0x7F4FEB0B817F >>>>>>>>>>>> #8 0x7F4FE9E1AD3F >>>>>>>>>>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>>>>>>>>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>>>>>>>>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>>>>>>>>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>>>>>>>>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>>>>>>>>>> >>>>>>>>>>>> I checked the code at tprfrtlc.F90:2393, >>>>>>>>>>>> >>>>>>>>>>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>>>>>>>>>> (cx(ix),ix=1,nxout)/) >>>>>>>>>>>> >>>>>>>>>>>> All the values (time, c, cx) are reasonable, as shown below. The only >>>>>>>>>>>> possibility is that realbuffer_gb is in declared as real*4 if using sing >>>>>>>>>>>> precision output while time, c, cx are declared in real*8. I have a lot of >>>>>>>>>>>> similar data conversion from real*8 to real*4 output, other code does not >>>>>>>>>>>> return error. >>>>>>>>>>>> >>>>>>>>>>>> time 0.0000000000000000 >>>>>>>>>>>> c 1 9.9999999999999995E-008 >>>>>>>>>>>> c 2 3.1555251077549618E-003 >>>>>>>>>>>> c 3 7.1657814842179362E-008 >>>>>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>>>>> c 5 5.2879822292305797E-004 >>>>>>>>>>>> c 6 9.9999999999999964E-005 >>>>>>>>>>>> c 7 6.4055731968811337E-005 >>>>>>>>>>>> c 8 3.4607572892578404E-020 >>>>>>>>>>>> cx 1 3.4376650636008101E-005 >>>>>>>>>>>> cx 2 7.3989678854017763E-012 >>>>>>>>>>>> cx 3 9.5317170613607207E-012 >>>>>>>>>>>> cx 4 2.2344525794718353E-015 >>>>>>>>>>>> cx 5 3.0624685689695889E-008 >>>>>>>>>>>> cx 6 1.0046157902783967E-007 >>>>>>>>>>>> cx 7 1.5320169154914984E-004 >>>>>>>>>>>> cx 8 8.6930292776346176E-014 >>>>>>>>>>>> cx 9 3.5944267559348721E-005 >>>>>>>>>>>> cx 10 3.0072645866951157E-018 >>>>>>>>>>>> cx 11 2.3592486321095017E-013 >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Danyang >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I can check all the entries of jacobi matrix to see if the value is valid, >>>>>>>>>>>> but this seems not a good idea as it takes a long time to reach this >>>>>>>>>>>> point. If I restart the simulation from a specified time (e.g., 7.685 in >>>>>>>>>>>> this case), then the error does not occur. >>>>>>>>>>>> >>>>>>>>>>>> Would you please give me any suggestion on debugging this case? >>>>>>>>>>>> >>>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>> >>>>>>>>>>>> Danyang >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>> [1]PETSC ERROR: Floating point exception >>>>>>>>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>> [2]PETSC ERROR: Floating point exception >>>>>>>>>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >>>>>>>>>>>> at end of function: Parameter number 3 >>>>>>>>>>>> [2]PETSC ERROR: See >>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>>>> >>>>>>>>>>>> for trouble shooting. >>>>>>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>>>>>>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>>>>>>>>> [1]PETSC ERROR: See >>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>>>> >>>>>>>>>>>> for trouble shooting. >>>>>>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>>>>>>>>>>> dsu Thu Apr 23 15:38:52 2015 >>>>>>>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >>>>>>>>>>>> 15:38:52 2015 >>>>>>>>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>>>> #3 KSP_PCApply() line 230 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>>>>>>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Apr 29 14:12:43 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 29 Apr 2015 14:12:43 -0500 Subject: [petsc-users] Discrete adjoint and adaptive time stepping In-Reply-To: <871tj2py2i.fsf@jedbrown.org> References: <8A885376-C369-4E9E-AAC0-140930EA3CED@mcs.anl.gov> <871tj2py2i.fsf@jedbrown.org> Message-ID: <30260DDE-B439-4498-93C9-87ADBD9CE19F@mcs.anl.gov> > On Apr 29, 2015, at 1:51 PM, Jed Brown wrote: > > Barry Smith writes: >>> If so, how is it done? >> >> We just keep the history of the time-step sizes and then use those time-step sizes when doing the backward integration. Seems simple to me, am I missing something? > > Barry, if you're using this for optimization, you might want the > gradient to be exactly consistent with the objective functional. But > for that, you would need to differentiate the controller, which is > non-smooth in practice because the number of time steps can change and > stages could be rejected (solver failure). Ahh, yes for multiple forward runs yup. > > One approach would be to save the timestep sequence and have the > controller use that in subsequent *forward* runs. If the dynamical > system behaves similarly for those steps, it would be okay to use the > same timestep sequence. Presumably if that single set of dt (from the first run) is not sufficient for some later runs one could possibly use the union of the dt of several runs for all the runs. (that is run adaptively and inconsistently several runs to determine where dt needs to be controlled and then use the various smaller of the dt at the different time regions for a full set of consistent runs). Of course if the various smaller of the dt requires a tiny dt for all time steps then you are not getting an advantage of adaptive time-stepping, but ok. The idea of actually propagating the gradients through the time-step controller seems IMHO to be absurd; I won't even put it on our game plan until we have many more things done and much more practical experience. Barry From bsmith at mcs.anl.gov Wed Apr 29 14:19:31 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 29 Apr 2015 14:19:31 -0500 Subject: [petsc-users] Floating point exception in hypre BoomerAMG In-Reply-To: <55412888.90206@gmail.com> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> <553C3E15.8030403@gmail.com> <553C3FE6.5020203@gmail.com> <553C4ECD.6090905@gmail.com> <553E6770.5010205@gmail.com> <553FC98D.1000904@gmail.com> <89DAC785-3218-4E2F-8760-2D0E99652014@mcs.anl.gov> <5540261E.2040502@gmail.com> <6411F61F-CD41-4037-B727-B5D9CECE030F@mcs.anl.gov> <554111AD.3060007@gmail.com> <9C61B774-4A78-48E1-B34E-B23EC2D230B9@mcs.anl.gov> <55412888.90206@gmail.com> Message-ID: <0F8EF2B9-9A1C-4C94-B1CC-D520B6D7A0DE@mcs.anl.gov> Ok, your code seems fine in terms of logic. But the vector x_vec_gbl i.e. b_react should not be in a read only state. Are you somewhere calling a VecGetArray() and not calling the VecRestoreArray()? Barry > On Apr 29, 2015, at 1:52 PM, Danyang Su wrote: > > On 15-04-29 11:30 AM, Barry Smith wrote: >>> On Apr 29, 2015, at 12:15 PM, Danyang Su >>> wrote: >>> >>> On 15-04-28 06:50 PM, Barry Smith wrote: >>> >>>> We started enforcing more checks on writing to vectors that you should not write to. Where are you calling DMLocalToGlobalBegin() ? It looks like you are trying to copy values into a global array that you should not because it is a read only input to a function, for example it is the right hand side of a linear system or the input into a SNESFormFunction(). So check the output to that call. >>>> >>>> Barry >>>> >>>> >>> This is used when local RHS vec is calculated and copied to global RHS. >>> >> Huhh, why are you copying anything into the RHS? Is this before you call the linear system solve? Send the code that calls the DMLocalToGlobalBegin() > Yes. This is called before the linear system solver. At present, we use PETSc KSP solver. > The code is first developed in totally sequential version and then ported to the parallel version. So the subdomain has its own space for jacobi matrix and rhs. We do not use global RHS vector directly to set the RHS at this moment. All the RHS values of subdomain are copied to the global RHS. > > !> This is the function where RHS values of subdomain are copied to the global RHS values. > subroutine compute_function(rank,da,x_array_loc,x_vec_loc, & > x_vec_gbl,nngl,row_idx_l2pg,col_idx_l2pg, & > b_non_interlaced) > > implicit none > #include > #include > #include > #include > #include > #include > #include > #include > #include > > PetscInt :: rank > DM :: da > PetscReal, allocatable :: x_array_loc(:) > Vec :: x_vec_loc > Vec :: x_vec_gbl > PetscInt :: nngl > PetscInt, allocatable :: row_idx_l2pg(:) > PetscInt, allocatable :: col_idx_l2pg(:) > PetscBool :: b_non_interlaced > > > PetscInt :: info_debug > PetscInt :: i, j > PetscErrorCode :: ierr > PetscScalar, pointer :: vecpointer(:) > > > !Zero entries > call VecZeroEntries(x_vec_loc, ierr) > > !Get a pointer to vector data when you need access to the array > call VecGetArrayF90(x_vec_loc, vecpointer, ierr) > > !Compute the function over the locally owned part of the grid > if(b_non_interlaced) then > j = nngl/2 > do i = 1, j > vecpointer(2*i-1) = x_array_loc(i) > vecpointer(2*i) = x_array_loc(i+j) > end do > else > do i = 1, nngl > vecpointer(i) = x_array_loc(i) > end do > end if > > !Restore the vector when you no longer need access to the array > call VecRestoreArrayF90(x_vec_loc,vecpointer,ierr) > > !Insert values into global vector > call DMLocalToGlobalBegin(da,x_vec_loc,INSERT_VALUES, & > x_vec_gbl,ierr) > !By placing code between these two statements, computations can be > !done while messages are in transition. > call DMLocalToGlobalEnd(da,x_vec_loc,INSERT_VALUES, & > x_vec_gbl,ierr) > > return > > end subroutine > > > > !> This is the function where reactive transport equations are solved. > subroutine solver_dd_snes_solve_react(ilog,idetail,a_in,b_in, & > x_inout,ia_in,ja_in,nngl_in,itsolv, & > over_flow,rnorm,row_idx_l2pg,col_idx_l2pg, & > b_non_interlaced) > > use gen, only : rank, node_idx_l2lg, ittot_rt, & > b_output_matrix, b_enable_output > use solver_snes_function, only : form_initial_guess, & > compute_function, & > compute_jacobian > > use petsc_mpi_common, only : petsc_mpi_finalize > > implicit none > #include > #include > #include > #include > > PetscInt :: ilog > PetscInt :: idetail > PetscInt :: nngl_in > PetscReal, allocatable :: a_in(:) > PetscReal, allocatable :: b_in(:) > PetscReal, allocatable :: x_inout(:) > PetscInt, allocatable :: ia_in(:) > PetscInt, allocatable :: ja_in(:) > PetscInt, allocatable :: row_idx_l2pg(:) > PetscInt, allocatable :: col_idx_l2pg(:) > PetscInt :: itsolv > PetscBool :: over_flow > PetscBool :: b_non_interlaced > PetscReal :: rnorm > > PetscViewer :: viewer > PetscScalar,pointer :: vecpointer(:) > > PetscErrorCode :: ierr > PetscInt :: info_debug > character(72) :: strinum > > info_debug = 0 > > if(b_output_matrix .and. b_enable_output) then > write(strinum, *) ittot_rt > strinum = "_"//trim(adjustl(strinum)) > end if > > > !Form initial guess, only assemble the local owned part, without ghost nodes > call form_initial_guess(rank,dmda_react%da,x_inout,x_react_loc,& > x_react,nngl_in, row_idx_l2pg,col_idx_l2pg, & > b_non_interlaced) > > > !Compute function, only assemble the local part, without ghost nodes > call compute_function(rank,dmda_react%da,b_in,b_react_loc, & > b_react,nngl_in,row_idx_l2pg,col_idx_l2pg, & > b_non_interlaced) > > > !Compute jacobian matrix, only assemble the local part, without ghost nodes > call compute_jacobian(rank,dmda_react%da, & > a_react,a_in,ia_in,ja_in,nngl_in, & > row_idx_l2pg,col_idx_l2pg, & > b_non_interlaced) > > > ! Solve a x = b, where a is the Jacobian matrix. > ! - First, set the KSP linear operators. Here the matrix that > ! defines the linear system also serves as the preconditioning > ! matrix. > ! - Then solve the Newton system. > #ifdef PETSC_V3_5_X > call KSPSetOperators(ksp_react,a_react,a_react,ierr) > #else > call KSPSetOperators(ksp_react,a_react,a_react, & > SAME_NONZERO_PATTERN,ierr) > #endif > > call KSPSetDM(ksp_react,dmda_react%da,ierr) > call KSPSetDMActive(ksp_react,PETSC_FALSE,ierr) > > call KSPSolve(ksp_react,b_react,x_react,ierr) > > > !Get residual norm, by default, preconditioned residual norm is calculated. > if (b_mykspconverged_react .and. & > .not. b_use_petsc_default_react) then > rnorm = rnorm_react > else > call KSPGetResidualNorm(ksp_react, rnorm, ierr) > end if > > ! Scatter ghost points to local vector, using the 2-step process > ! DMGlobalToLocalBegin(), DMGlobalToLocalEnd(). > ! By placing code between these two statements, computations can be > ! done while messages are in transition. > call DMGlobalToLocalBegin(dmda_react%da,x_react,INSERT_VALUES, & > x_react_loc,ierr) > call DMGlobalToLocalEnd(dmda_react%da,x_react,INSERT_VALUES, & > x_react_loc,ierr) > > call VecGetArrayF90(x_react_loc,vecpointer,ierr) > > x_inout = vecpointer > call VecRestoreArrayF90(x_react_loc,vecpointer,ierr) > > return > > end subroutine > >> >> Barry >> >> >>> It is strange why this error does not occur at the first timestep. >>> >>> Thanks, >>> >>> Danyang >>> >>>>> On Apr 28, 2015, at 7:30 PM, Danyang Su >>>>> wrote: >>>>> >>>>> Hi Barry, >>>>> >>>>> There seems another bug (not pretty sure) in PETSc-dev, as shown below. The case I used is similar with the one I mentioned recently. I have no problem running this case using PETSc 3.5.2. But it give out the following error using PETSc-dev, even with only one processor. >>>>> >>>>> Thanks, >>>>> >>>>> Danyang >>>>> >>>>> timestep: 2048 time: 4.615E+00 years delt: 1.460E-20 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Object is in wrong state >>>>> [0]PETSC ERROR: Vec is locked read only, argument # 1 >>>>> [0]PETSC ERROR: See >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2796-g23b6c49 GIT Date: 2015-04-27 11:32:44 -0500 >>>>> [0]PETSC ERROR: ../min3p_thcm on a linux-gnu-dbg named nwmop by dsu Tue Apr 28 15:56:41 2015 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack >>>>> [0]PETSC ERROR: #349 VecGetArray() line 1646 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/interface/rvector.c >>>>> [0]PETSC ERROR: #350 VecGetArrayPair() line 434 in /home/dsu/Soft/PETSc/petsc-dev/include/petscvec.h >>>>> [0]PETSC ERROR: #351 VecScatterBegin_SSToSS() line 649 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c >>>>> [0]PETSC ERROR: #352 VecScatterBegin() line 1694 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c >>>>> [0]PETSC ERROR: #353 DMLocalToGlobalBegin_DA() line 56 in /home/dsu/Soft/PETSc/petsc-dev/src/dm/impls/da/dagtol.c >>>>> [0]PETSC ERROR: #354 DMLocalToGlobalBegin() line 1944 in /home/dsu/Soft/PETSc/petsc-dev/src/dm/interface/dm.c >>>>> Reduce time step for reactive transport >>>>> no further time step reduction possible >>>>> Optimal time increment computed by MIN3P 2.6644769214899893E-018 >>>>> Minimum time increment specified by user 3.6499999999999998E-018 >>>>> Please, reduce the MINIMUM TIME INCREMENT >>>>> stop signal in time step reduction failed >>>>> >>>>> On 15-04-28 11:15 AM, Barry Smith wrote: >>>>> >>>>>> I am forwarding this on to the hypre support email list; hopefully they can have some advice on how to proceed. >>>>>> >>>>>> hypre folks, this is with version hypre-2.10.0b (the previous version had the same behavior). We are getting a divide by zero in gselim() line 4363 (this happens in a time dependent problem after many successful solves with time dependent matrix). I looked at the code, below, and note that there are some checks for the diagonal of A not being zero but not for the line that causes the divide by zero; is this perhaps an oversight in the hypre code? Any advice appreciated. >>>>>> >>>>>> Thanks >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> HYPRE_Int gselim(A,x,n) >>>>>> HYPRE_Real *A; >>>>>> HYPRE_Real *x; >>>>>> HYPRE_Int n; >>>>>> { >>>>>> HYPRE_Int err_flag = 0; >>>>>> HYPRE_Int j,k,m; >>>>>> HYPRE_Real factor; >>>>>> if (n==1) /* A is 1x1 */ >>>>>> { >>>>>> if (A[0] != 0.0) >>>>>> { >>>>>> x[0] = x[0]/A[0]; >>>>>> return(err_flag); >>>>>> } >>>>>> else >>>>>> { >>>>>> err_flag = 1; >>>>>> return(err_flag); >>>>>> } >>>>>> } >>>>>> else /* A is nxn. Forward elimination */ >>>>>> { >>>>>> for (k = 0; k < n-1; k++) >>>>>> { >>>>>> if (A[k*n+k] != 0.0) >>>>>> { >>>>>> for (j = k+1; j < n; j++) >>>>>> { >>>>>> if (A[j*n+k] != 0.0) >>>>>> { >>>>>> factor = A[j*n+k]/A[k*n+k]; >>>>>> for (m = k+1; m < n; m++) >>>>>> { >>>>>> A[j*n+m] -= factor * A[k*n+m]; >>>>>> } >>>>>> /* Elimination step for rhs */ >>>>>> x[j] -= factor * x[k]; >>>>>> } >>>>>> } >>>>>> } >>>>>> } >>>>>> /* Back Substitution */ >>>>>> for (k = n-1; k > 0; --k) >>>>>> { >>>>>> x[k] /= A[k*n+k]; >>>>>> for (j = 0; j < k; j++) >>>>>> { >>>>>> if (A[j*n+k] != 0.0) >>>>>> { >>>>>> x[j] -= x[k] * A[j*n+k]; >>>>>> } >>>>>> } >>>>>> } >>>>>> x[0] /= A[0]; >>>>>> return(err_flag); >>>>>> } >>>>>> } >>>>>> >>>>>> >>>>>> >>>>>>> On Apr 28, 2015, at 12:55 PM, Danyang Su >>>>>>> wrote: >>>>>>> >>>>>>> Hi Barry, >>>>>>> >>>>>>> The development version of PETSc does not help to solve my problem. It still crashed due to the same error information. >>>>>>> >>>>>>> As Matthew mentioned, I checked the matrix and non of the diagonal entry is zero. But the diagonal values for different rows range from -1.0d-18 to 1.0d33 and the matrix is not strict diagonal dominant. When using 1 processor, the matrix is also not strict diagonal dominant, but the diagonal values have a much smaller range and the codes can run without error. Maybe there is something wrong in the matrix for this case and I need to check the matrix first. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Danyang >>>>>>> >>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>> 0x00007f668c1299d5 in gselim (A=0x2fe3330, x=0x2210680, n=9) >>>>>>> at par_relax.c:4363 >>>>>>> 4363 x[k] /= A[k*n+k]; >>>>>>> (gdb) >>>>>>> >>>>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>> [1]PETSC ERROR: is given. >>>>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-dev/src/sys/error/fp.c >>>>>>> [1]PETSC ERROR: [1] Hypre solve line 216 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >>>>>>> [1]PETSC ERROR: [1] PCApply_HYPRE line 203 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >>>>>>> [1]PETSC ERROR: [1] PCApply line 430 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/interface/precon.c >>>>>>> [1]PETSC ERROR: [1] KSP_PCApply line 234 /home/dsu/Soft/PETSc/petsc-dev/include/petsc/private/kspimpl.h >>>>>>> [1]PETSC ERROR: [1] KSPInitialResidual line 44 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itres.c >>>>>>> [1]PETSC ERROR: [1] KSPSolve_GMRES line 224 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/impls/gmres/gmres.c >>>>>>> [1]PETSC ERROR: [1] KSPSolve line 506 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 15-04-27 11:47 AM, Barry Smith wrote: >>>>>>> >>>>>>>> Danyang, >>>>>>>> >>>>>>>> I'm glad you were finally able to get to the problematic point; sorry it took so long (I want to strangle whoever turned on catching of underflows in PETSc). >>>>>>>> >>>>>>>> The hypre folks have done a great deal of work on their relaxation code since the PETSc 3.5.3 release. There is a good chance they have fixed the divide by zero problem you are hitting here. You will need to upgrade to the development version of PETSc (that uses the latest version of hypre), here are the instructions on how to obtain it >>>>>>>> http://www.mcs.anl.gov/petsc/developers/index.html >>>>>>>> >>>>>>>> >>>>>>>> Please let us know if this resolves the problem with hypre failing. >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Apr 27, 2015, at 11:44 AM, Danyang Su >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi Barry, >>>>>>>>> >>>>>>>>> I got the following arithemetic exception after the previous bug is fixed. >>>>>>>>> >>>>>>>>> Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 >>>>>>>>> 0x00007f3b23b98f20 in __nanosleep_nocancel () >>>>>>>>> at ../sysdeps/unix/syscall-template.S:81 >>>>>>>>> 81 ../sysdeps/unix/syscall-template.S: No such file or directory. >>>>>>>>> (gdb) cont >>>>>>>>> Continuing. >>>>>>>>> >>>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>>> 0x00007f3b260df449 in gselim (A=0x1e4c580, x=0x1d30c40, n=9) >>>>>>>>> at par_relax.c:3442 >>>>>>>>> 3442 x[k] /= A[k*n+k]; >>>>>>>>> (gdb) >>>>>>>>> >>>>>>>>> I tried both PETSc 3.5.2 and 3.5.3 and they return the same error as shown above. For 3.5.3, i edited fp.c file and then configure and make. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Danyang >>>>>>>>> >>>>>>>>> On 15-04-25 07:34 PM, Danyang Su wrote: >>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> The "floating point underflow" is caused by a small value divided by a very large value. This result is forced to zero and then it does not report any underflow problem. I just rerun this bad case to see if it still get stuck later. This will take a while. >>>>>>>>>> >>>>>>>>>> Thanks for all your kindly reply, >>>>>>>>>> >>>>>>>>>> Danyang >>>>>>>>>> >>>>>>>>>> On 15-04-25 07:02 PM, Barry Smith wrote: >>>>>>>>>> >>>>>>>>>>> Ok, you do have >>>>>>>>>>> >>>>>>>>>>> #ifndef PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>> #define PETSC_HAVE_XMMINTRIN_H 1 >>>>>>>>>>> #endif >>>>>>>>>>> >>>>>>>>>>> so the change you made should cause it to stop trapping underflow exceptions. >>>>>>>>>>> >>>>>>>>>>> Now in one email you reported a FPE within hypre, then I asked you to run with -start_in_debugger to determine where it happened exactly and then you reported the FPE happened in user code (what seemed to be an underflow issue). Why is this? Can you not run it where it generated the FPE in hypre using the -start_in_debugger option? >>>>>>>>>>> >>>>>>>>>>> Barry >>>>>>>>>>> >>>>>>>>>>> Perhaps you have multiple PETSC_ARCH or multiple PETSc installs to explain why you reported two different places where the exception occurred. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On Apr 25, 2015, at 8:31 PM, Danyang Su >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 15-04-25 06:26 PM, Matthew Knepley wrote: >>>>>>>>>>>> >>>>>>>>>>>>> On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 15-04-25 06:03 PM, Barry Smith wrote: >>>>>>>>>>>>> If this is what you got in your last run >>>>>>>>>>>>> >>>>>>>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>>>>>>>> then it is still catching floating point underflow, which we do not want. This means either the change I suggested you make in the fp.c code didn't work or it actually uses a different floating point trap than that one. BTW: absurd numbers like 1.0d-316 are often a symptom of uninitialized data; could that be a problem that diff is not filled correctly for all the ic, idim you are using? >>>>>>>>>>>>> >>>>>>>>>>>>> This going round and round is very frustrating and a waste of time. You need to be more proactive yourself and explore the code and poke around to figure out how to solve the problem. >>>>>>>>>>>>> >>>>>>>>>>>>> Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can see what FP trap is being used on your machine. >>>>>>>>>>>>> >>>>>>>>>>>>> Barry >>>>>>>>>>>>> Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise I cannot find this file. >>>>>>>>>>>>> >>>>>>>>>>>>> Its include/petscconf.h >>>>>>>>>>>>> Do I need to reconfigure PETSc after changing the code you mentioned? >>>>>>>>>>>>> >>>>>>>>>>>>> No, but you need to rebuild. >>>>>>>>>>>>> >>>>>>>>>>>> Yes, I have done 'gnumake'. >>>>>>>>>>>> >>>>>>>>>>>>> Matt >>>>>>>>>>>>> Danyang >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Apr 25, 2015, at 2:24 PM, Danyang Su >>>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 15-04-25 11:55 AM, Barry Smith wrote: >>>>>>>>>>>>> On Apr 25, 2015, at 1:51 PM, Danyang Su >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 15-04-25 11:32 AM, Barry Smith wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> I told you this yesterday. >>>>>>>>>>>>> >>>>>>>>>>>>> It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. >>>>>>>>>>>>> >>>>>>>>>>>>> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate >>>>>>>>>>>>> >>>>>>>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>>>>>>>>>>>> #else >>>>>>>>>>>>> >>>>>>>>>>>>> change it to >>>>>>>>>>>>> >>>>>>>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >>>>>>>>>>>>> #else >>>>>>>>>>>>> >>>>>>>>>>>>> Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. >>>>>>>>>>>>> >>>>>>>>>>>>> Did you do this? >>>>>>>>>>>>> >>>>>>>>>>>>> Barry >>>>>>>>>>>>> >>>>>>>>>>>>> Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc directory. I just did a double check and ran make gnumake again and got the following information this time. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >>>>>>>>>>>>> >>>>>>>>>>>>> make gnumake >>>>>>>>>>>>> Building PETSc using GNU Make with 10 build threads >>>>>>>>>>>>> ========================================== >>>>>>>>>>>>> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>>>>>>> make[1]: Nothing to be done for `all'. >>>>>>>>>>>>> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>>>>>>> ========================================= >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Then I recompiled the codes, ran with -fp_trap and still got the following error >>>>>>>>>>>>> >>>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>>> Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>>>> [2]PETSC ERROR: is given. >>>>>>>>>>>>> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>> INSTEAD the line number of the start of the function >>>>>>>>>>>>> [3]PETSC ERROR: is given. >>>>>>>>>>>>> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>> [2]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>>>> [3]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> This is different then what you sent a few minutes ago where it crashed in hypre. >>>>>>>>>>>>> >>>>>>>>>>>>> Anyways you need to use the -start_in_debugger business I sent in the previous email to see the exact place the problem occurs. >>>>>>>>>>>>> >>>>>>>>>>>>> Here is the information shown on gdb screen >>>>>>>>>>>>> >>>>>>>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>>>>>>> 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, njamxc=34, >>>>>>>>>>>>> cinfradx=..., radial_coordx=.FALSE., _suffix=3) >>>>>>>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>>>>>>>> (gdb) >>>>>>>>>>>>> >>>>>>>>>>>>> After type cont on gdb screen, I got error information as below >>>>>>>>>>>>> >>>>>>>>>>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>>>> [1]PETSC ERROR: is given. >>>>>>>>>>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> >>>>>>>>>>>>> Danyang >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> >>>>>>>>>>>>> Danyang >>>>>>>>>>>>> >>>>>>>>>>>>> On Apr 25, 2015, at 1:05 AM, Danyang Su >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Barry and Satish, >>>>>>>>>>>>> >>>>>>>>>>>>> How can I get rid of unknown floating point error when a very small value is multiplied. >>>>>>>>>>>>> >>>>>>>>>>>>> e.g., >>>>>>>>>>>>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, >>>>>>>>>>>>> >>>>>>>>>>>>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>>>>>>>>>>>> >>>>>>>>>>>>> I get the following error when run with "-fp_trap -start_in_debugger". >>>>>>>>>>>>> >>>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>>> *** unknown floating point error occurred *** >>>>>>>>>>>>> [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>>>> [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>>>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 >>>>>>>>>>>>> where the result is a bitwise OR of the following flags: >>>>>>>>>>>>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>> [2]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> >>>>>>>>>>>>> Danyang >>>>>>>>>>>>> >>>>>>>>>>>>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>>>>>> >>>>>>>>>>>>> I don't think this number can be stored in a real*4. >>>>>>>>>>>>> >>>>>>>>>>>>> Satish >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, Satish. It is caused by this number. >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi All, >>>>>>>>>>>>> >>>>>>>>>>>>> One of my case crashes because of floating point exception when using 4 >>>>>>>>>>>>> processors, as shown below. But if I run this case with 1 processor, it >>>>>>>>>>>>> works fine. I have tested the codes with around 100 cases up to 768 >>>>>>>>>>>>> processors, all other cases work fine. I just wonder if this kind of error >>>>>>>>>>>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>>>>>>>>>>> >>>>>>>>>>>>> Yes, almost for sure it is one of these places. >>>>>>>>>>>>> >>>>>>>>>>>>> First run the bad case with -fp_trap if all goes well you'll see the >>>>>>>>>>>>> function where the FPE is generated. Then run also with -start_in_debugger >>>>>>>>>>>>> and >>>>>>>>>>>>> type cont in all four debugger windows. When the FPE happens the debugger >>>>>>>>>>>>> should stop showing exactly where the FPE happens. >>>>>>>>>>>>> >>>>>>>>>>>>> Barry >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Barry, >>>>>>>>>>>>> >>>>>>>>>>>>> If run with -fp_trap -start_in_debugger, I got the following error >>>>>>>>>>>>> >>>>>>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a >>>>>>>>>>>>> debugger. When the >>>>>>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >>>>>>>>>>>>> fetestexcept(0x3d) >>>>>>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>>>>>>>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>>>>>>>>> ------------------------------------ >>>>>>>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>>>> [0]PETSC ERROR: is given. >>>>>>>>>>>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>>>>>>>>>>>> floating point error >>>>>>>>>>>>> >>>>>>>>>>>>> Program received signal SIGABRT: Process abort signal. >>>>>>>>>>>>> >>>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>>> #0 0x7F4FEAB1C7D7 >>>>>>>>>>>>> #1 0x7F4FEAB1CDDE >>>>>>>>>>>>> #2 0x7F4FE9E1AD3F >>>>>>>>>>>>> #3 0x7F4FE9E1ACC9 >>>>>>>>>>>>> #4 0x7F4FE9E1E0D7 >>>>>>>>>>>>> #5 0x7F4FEB0B6DCB >>>>>>>>>>>>> #6 0x7F4FEB0B1825 >>>>>>>>>>>>> #7 0x7F4FEB0B817F >>>>>>>>>>>>> #8 0x7F4FE9E1AD3F >>>>>>>>>>>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>>>>>>>>>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>>>>>>>>>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>>>>>>>>>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>>>>>>>>>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>>>>>>>>>>> >>>>>>>>>>>>> I checked the code at tprfrtlc.F90:2393, >>>>>>>>>>>>> >>>>>>>>>>>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>>>>>>>>>>> (cx(ix),ix=1,nxout)/) >>>>>>>>>>>>> >>>>>>>>>>>>> All the values (time, c, cx) are reasonable, as shown below. The only >>>>>>>>>>>>> possibility is that realbuffer_gb is in declared as real*4 if using sing >>>>>>>>>>>>> precision output while time, c, cx are declared in real*8. I have a lot of >>>>>>>>>>>>> similar data conversion from real*8 to real*4 output, other code does not >>>>>>>>>>>>> return error. >>>>>>>>>>>>> >>>>>>>>>>>>> time 0.0000000000000000 >>>>>>>>>>>>> c 1 9.9999999999999995E-008 >>>>>>>>>>>>> c 2 3.1555251077549618E-003 >>>>>>>>>>>>> c 3 7.1657814842179362E-008 >>>>>>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>>>>>> c 5 5.2879822292305797E-004 >>>>>>>>>>>>> c 6 9.9999999999999964E-005 >>>>>>>>>>>>> c 7 6.4055731968811337E-005 >>>>>>>>>>>>> c 8 3.4607572892578404E-020 >>>>>>>>>>>>> cx 1 3.4376650636008101E-005 >>>>>>>>>>>>> cx 2 7.3989678854017763E-012 >>>>>>>>>>>>> cx 3 9.5317170613607207E-012 >>>>>>>>>>>>> cx 4 2.2344525794718353E-015 >>>>>>>>>>>>> cx 5 3.0624685689695889E-008 >>>>>>>>>>>>> cx 6 1.0046157902783967E-007 >>>>>>>>>>>>> cx 7 1.5320169154914984E-004 >>>>>>>>>>>>> cx 8 8.6930292776346176E-014 >>>>>>>>>>>>> cx 9 3.5944267559348721E-005 >>>>>>>>>>>>> cx 10 3.0072645866951157E-018 >>>>>>>>>>>>> cx 11 2.3592486321095017E-013 >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> >>>>>>>>>>>>> Danyang >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I can check all the entries of jacobi matrix to see if the value is valid, >>>>>>>>>>>>> but this seems not a good idea as it takes a long time to reach this >>>>>>>>>>>>> point. If I restart the simulation from a specified time (e.g., 7.685 in >>>>>>>>>>>>> this case), then the error does not occur. >>>>>>>>>>>>> >>>>>>>>>>>>> Would you please give me any suggestion on debugging this case? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>>> >>>>>>>>>>>>> Danyang >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>>> [1]PETSC ERROR: Floating point exception >>>>>>>>>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>>> [2]PETSC ERROR: Floating point exception >>>>>>>>>>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >>>>>>>>>>>>> at end of function: Parameter number 3 >>>>>>>>>>>>> [2]PETSC ERROR: See >>>>>>>>>>>>> >>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> for trouble shooting. >>>>>>>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>>>>>>>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>>>>>>>>>> [1]PETSC ERROR: See >>>>>>>>>>>>> >>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> for trouble shooting. >>>>>>>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>>>>>>>>>>>> dsu Thu Apr 23 15:38:52 2015 >>>>>>>>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >>>>>>>>>>>>> 15:38:52 2015 >>>>>>>>>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>>>>> #3 KSP_PCApply() line 230 in >>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>>>>>>>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > From danyang.su at gmail.com Wed Apr 29 14:35:48 2015 From: danyang.su at gmail.com (Danyang Su) Date: Wed, 29 Apr 2015 12:35:48 -0700 Subject: [petsc-users] Floating point exception in hypre BoomerAMG In-Reply-To: <0F8EF2B9-9A1C-4C94-B1CC-D520B6D7A0DE@mcs.anl.gov> References: <553A85DF.5000400@gmail.com> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> <553C3E15.8030403@gmail.com> <553C3FE6.5020203@gmail.com> <553C4ECD.6090905@gmail.com> <553E6770.5010205@gmail.com> <553FC98D.1000904@gmail.com> <89DAC785-3218-4E2F-8760-2D0E99652014@mcs.anl.gov> <5540261E.2040502@gmail.com> <6411F61F-CD41-4037-B727-B5D9CECE030F@mcs.anl.gov> <554111AD.3060007@gmail.com> <9C61B774-4A78-48E1-B34E-B23EC2D230B9@mcs.anl.gov> <55412888.90206@gmail.com> <0F8EF2B9-9A1C-4C94-B1CC-D520B6D7A0DE@mcs.anl.gov> Message-ID: <55413294.1000906@gmail.com> On 15-04-29 12:19 PM, Barry Smith wrote: > Ok, your code seems fine in terms of logic. But the vector x_vec_gbl i.e. b_react should not be in a read only state. Are you somewhere calling a VecGetArray() and not calling the VecRestoreArray()? > > Barry I just made a triple check on VecGetArrayF90(). All the VecGetArrayF90() come with VecRestoreArrayF90(). No VecGetArray() is used in my codes. Danyang > > > >> On Apr 29, 2015, at 1:52 PM, Danyang Su wrote: >> >> On 15-04-29 11:30 AM, Barry Smith wrote: >>>> On Apr 29, 2015, at 12:15 PM, Danyang Su >>>> wrote: >>>> >>>> On 15-04-28 06:50 PM, Barry Smith wrote: >>>> >>>>> We started enforcing more checks on writing to vectors that you should not write to. Where are you calling DMLocalToGlobalBegin() ? It looks like you are trying to copy values into a global array that you should not because it is a read only input to a function, for example it is the right hand side of a linear system or the input into a SNESFormFunction(). So check the output to that call. >>>>> >>>>> Barry >>>>> >>>>> >>>> This is used when local RHS vec is calculated and copied to global RHS. >>>> >>> Huhh, why are you copying anything into the RHS? Is this before you call the linear system solve? Send the code that calls the DMLocalToGlobalBegin() >> Yes. This is called before the linear system solver. At present, we use PETSc KSP solver. >> The code is first developed in totally sequential version and then ported to the parallel version. So the subdomain has its own space for jacobi matrix and rhs. We do not use global RHS vector directly to set the RHS at this moment. All the RHS values of subdomain are copied to the global RHS. >> >> !> This is the function where RHS values of subdomain are copied to the global RHS values. >> subroutine compute_function(rank,da,x_array_loc,x_vec_loc, & >> x_vec_gbl,nngl,row_idx_l2pg,col_idx_l2pg, & >> b_non_interlaced) >> >> implicit none >> #include >> #include >> #include >> #include >> #include >> #include >> #include >> #include >> #include >> >> PetscInt :: rank >> DM :: da >> PetscReal, allocatable :: x_array_loc(:) >> Vec :: x_vec_loc >> Vec :: x_vec_gbl >> PetscInt :: nngl >> PetscInt, allocatable :: row_idx_l2pg(:) >> PetscInt, allocatable :: col_idx_l2pg(:) >> PetscBool :: b_non_interlaced >> >> >> PetscInt :: info_debug >> PetscInt :: i, j >> PetscErrorCode :: ierr >> PetscScalar, pointer :: vecpointer(:) >> >> >> !Zero entries >> call VecZeroEntries(x_vec_loc, ierr) >> >> !Get a pointer to vector data when you need access to the array >> call VecGetArrayF90(x_vec_loc, vecpointer, ierr) >> >> !Compute the function over the locally owned part of the grid >> if(b_non_interlaced) then >> j = nngl/2 >> do i = 1, j >> vecpointer(2*i-1) = x_array_loc(i) >> vecpointer(2*i) = x_array_loc(i+j) >> end do >> else >> do i = 1, nngl >> vecpointer(i) = x_array_loc(i) >> end do >> end if >> >> !Restore the vector when you no longer need access to the array >> call VecRestoreArrayF90(x_vec_loc,vecpointer,ierr) >> >> !Insert values into global vector >> call DMLocalToGlobalBegin(da,x_vec_loc,INSERT_VALUES, & >> x_vec_gbl,ierr) >> !By placing code between these two statements, computations can be >> !done while messages are in transition. >> call DMLocalToGlobalEnd(da,x_vec_loc,INSERT_VALUES, & >> x_vec_gbl,ierr) >> >> return >> >> end subroutine >> >> >> >> !> This is the function where reactive transport equations are solved. >> subroutine solver_dd_snes_solve_react(ilog,idetail,a_in,b_in, & >> x_inout,ia_in,ja_in,nngl_in,itsolv, & >> over_flow,rnorm,row_idx_l2pg,col_idx_l2pg, & >> b_non_interlaced) >> >> use gen, only : rank, node_idx_l2lg, ittot_rt, & >> b_output_matrix, b_enable_output >> use solver_snes_function, only : form_initial_guess, & >> compute_function, & >> compute_jacobian >> >> use petsc_mpi_common, only : petsc_mpi_finalize >> >> implicit none >> #include >> #include >> #include >> #include >> >> PetscInt :: ilog >> PetscInt :: idetail >> PetscInt :: nngl_in >> PetscReal, allocatable :: a_in(:) >> PetscReal, allocatable :: b_in(:) >> PetscReal, allocatable :: x_inout(:) >> PetscInt, allocatable :: ia_in(:) >> PetscInt, allocatable :: ja_in(:) >> PetscInt, allocatable :: row_idx_l2pg(:) >> PetscInt, allocatable :: col_idx_l2pg(:) >> PetscInt :: itsolv >> PetscBool :: over_flow >> PetscBool :: b_non_interlaced >> PetscReal :: rnorm >> >> PetscViewer :: viewer >> PetscScalar,pointer :: vecpointer(:) >> >> PetscErrorCode :: ierr >> PetscInt :: info_debug >> character(72) :: strinum >> >> info_debug = 0 >> >> if(b_output_matrix .and. b_enable_output) then >> write(strinum, *) ittot_rt >> strinum = "_"//trim(adjustl(strinum)) >> end if >> >> >> !Form initial guess, only assemble the local owned part, without ghost nodes >> call form_initial_guess(rank,dmda_react%da,x_inout,x_react_loc,& >> x_react,nngl_in, row_idx_l2pg,col_idx_l2pg, & >> b_non_interlaced) >> >> >> !Compute function, only assemble the local part, without ghost nodes >> call compute_function(rank,dmda_react%da,b_in,b_react_loc, & >> b_react,nngl_in,row_idx_l2pg,col_idx_l2pg, & >> b_non_interlaced) >> >> >> !Compute jacobian matrix, only assemble the local part, without ghost nodes >> call compute_jacobian(rank,dmda_react%da, & >> a_react,a_in,ia_in,ja_in,nngl_in, & >> row_idx_l2pg,col_idx_l2pg, & >> b_non_interlaced) >> >> >> ! Solve a x = b, where a is the Jacobian matrix. >> ! - First, set the KSP linear operators. Here the matrix that >> ! defines the linear system also serves as the preconditioning >> ! matrix. >> ! - Then solve the Newton system. >> #ifdef PETSC_V3_5_X >> call KSPSetOperators(ksp_react,a_react,a_react,ierr) >> #else >> call KSPSetOperators(ksp_react,a_react,a_react, & >> SAME_NONZERO_PATTERN,ierr) >> #endif >> >> call KSPSetDM(ksp_react,dmda_react%da,ierr) >> call KSPSetDMActive(ksp_react,PETSC_FALSE,ierr) >> >> call KSPSolve(ksp_react,b_react,x_react,ierr) >> >> >> !Get residual norm, by default, preconditioned residual norm is calculated. >> if (b_mykspconverged_react .and. & >> .not. b_use_petsc_default_react) then >> rnorm = rnorm_react >> else >> call KSPGetResidualNorm(ksp_react, rnorm, ierr) >> end if >> >> ! Scatter ghost points to local vector, using the 2-step process >> ! DMGlobalToLocalBegin(), DMGlobalToLocalEnd(). >> ! By placing code between these two statements, computations can be >> ! done while messages are in transition. >> call DMGlobalToLocalBegin(dmda_react%da,x_react,INSERT_VALUES, & >> x_react_loc,ierr) >> call DMGlobalToLocalEnd(dmda_react%da,x_react,INSERT_VALUES, & >> x_react_loc,ierr) >> >> call VecGetArrayF90(x_react_loc,vecpointer,ierr) >> >> x_inout = vecpointer >> call VecRestoreArrayF90(x_react_loc,vecpointer,ierr) >> >> return >> >> end subroutine >> >>> Barry >>> >>> >>>> It is strange why this error does not occur at the first timestep. >>>> >>>> Thanks, >>>> >>>> Danyang >>>> >>>>>> On Apr 28, 2015, at 7:30 PM, Danyang Su >>>>>> wrote: >>>>>> >>>>>> Hi Barry, >>>>>> >>>>>> There seems another bug (not pretty sure) in PETSc-dev, as shown below. The case I used is similar with the one I mentioned recently. I have no problem running this case using PETSc 3.5.2. But it give out the following error using PETSc-dev, even with only one processor. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Danyang >>>>>> >>>>>> timestep: 2048 time: 4.615E+00 years delt: 1.460E-20 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: Object is in wrong state >>>>>> [0]PETSC ERROR: Vec is locked read only, argument # 1 >>>>>> [0]PETSC ERROR: See >>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>> for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2796-g23b6c49 GIT Date: 2015-04-27 11:32:44 -0500 >>>>>> [0]PETSC ERROR: ../min3p_thcm on a linux-gnu-dbg named nwmop by dsu Tue Apr 28 15:56:41 2015 >>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack >>>>>> [0]PETSC ERROR: #349 VecGetArray() line 1646 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/interface/rvector.c >>>>>> [0]PETSC ERROR: #350 VecGetArrayPair() line 434 in /home/dsu/Soft/PETSc/petsc-dev/include/petscvec.h >>>>>> [0]PETSC ERROR: #351 VecScatterBegin_SSToSS() line 649 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c >>>>>> [0]PETSC ERROR: #352 VecScatterBegin() line 1694 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c >>>>>> [0]PETSC ERROR: #353 DMLocalToGlobalBegin_DA() line 56 in /home/dsu/Soft/PETSc/petsc-dev/src/dm/impls/da/dagtol.c >>>>>> [0]PETSC ERROR: #354 DMLocalToGlobalBegin() line 1944 in /home/dsu/Soft/PETSc/petsc-dev/src/dm/interface/dm.c >>>>>> Reduce time step for reactive transport >>>>>> no further time step reduction possible >>>>>> Optimal time increment computed by MIN3P 2.6644769214899893E-018 >>>>>> Minimum time increment specified by user 3.6499999999999998E-018 >>>>>> Please, reduce the MINIMUM TIME INCREMENT >>>>>> stop signal in time step reduction failed >>>>>> >>>>>> On 15-04-28 11:15 AM, Barry Smith wrote: >>>>>> >>>>>>> I am forwarding this on to the hypre support email list; hopefully they can have some advice on how to proceed. >>>>>>> >>>>>>> hypre folks, this is with version hypre-2.10.0b (the previous version had the same behavior). We are getting a divide by zero in gselim() line 4363 (this happens in a time dependent problem after many successful solves with time dependent matrix). I looked at the code, below, and note that there are some checks for the diagonal of A not being zero but not for the line that causes the divide by zero; is this perhaps an oversight in the hypre code? Any advice appreciated. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> HYPRE_Int gselim(A,x,n) >>>>>>> HYPRE_Real *A; >>>>>>> HYPRE_Real *x; >>>>>>> HYPRE_Int n; >>>>>>> { >>>>>>> HYPRE_Int err_flag = 0; >>>>>>> HYPRE_Int j,k,m; >>>>>>> HYPRE_Real factor; >>>>>>> if (n==1) /* A is 1x1 */ >>>>>>> { >>>>>>> if (A[0] != 0.0) >>>>>>> { >>>>>>> x[0] = x[0]/A[0]; >>>>>>> return(err_flag); >>>>>>> } >>>>>>> else >>>>>>> { >>>>>>> err_flag = 1; >>>>>>> return(err_flag); >>>>>>> } >>>>>>> } >>>>>>> else /* A is nxn. Forward elimination */ >>>>>>> { >>>>>>> for (k = 0; k < n-1; k++) >>>>>>> { >>>>>>> if (A[k*n+k] != 0.0) >>>>>>> { >>>>>>> for (j = k+1; j < n; j++) >>>>>>> { >>>>>>> if (A[j*n+k] != 0.0) >>>>>>> { >>>>>>> factor = A[j*n+k]/A[k*n+k]; >>>>>>> for (m = k+1; m < n; m++) >>>>>>> { >>>>>>> A[j*n+m] -= factor * A[k*n+m]; >>>>>>> } >>>>>>> /* Elimination step for rhs */ >>>>>>> x[j] -= factor * x[k]; >>>>>>> } >>>>>>> } >>>>>>> } >>>>>>> } >>>>>>> /* Back Substitution */ >>>>>>> for (k = n-1; k > 0; --k) >>>>>>> { >>>>>>> x[k] /= A[k*n+k]; >>>>>>> for (j = 0; j < k; j++) >>>>>>> { >>>>>>> if (A[j*n+k] != 0.0) >>>>>>> { >>>>>>> x[j] -= x[k] * A[j*n+k]; >>>>>>> } >>>>>>> } >>>>>>> } >>>>>>> x[0] /= A[0]; >>>>>>> return(err_flag); >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On Apr 28, 2015, at 12:55 PM, Danyang Su >>>>>>>> wrote: >>>>>>>> >>>>>>>> Hi Barry, >>>>>>>> >>>>>>>> The development version of PETSc does not help to solve my problem. It still crashed due to the same error information. >>>>>>>> >>>>>>>> As Matthew mentioned, I checked the matrix and non of the diagonal entry is zero. But the diagonal values for different rows range from -1.0d-18 to 1.0d33 and the matrix is not strict diagonal dominant. When using 1 processor, the matrix is also not strict diagonal dominant, but the diagonal values have a much smaller range and the codes can run without error. Maybe there is something wrong in the matrix for this case and I need to check the matrix first. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Danyang >>>>>>>> >>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>> 0x00007f668c1299d5 in gselim (A=0x2fe3330, x=0x2210680, n=9) >>>>>>>> at par_relax.c:4363 >>>>>>>> 4363 x[k] /= A[k*n+k]; >>>>>>>> (gdb) >>>>>>>> >>>>>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>> [1]PETSC ERROR: is given. >>>>>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-dev/src/sys/error/fp.c >>>>>>>> [1]PETSC ERROR: [1] Hypre solve line 216 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >>>>>>>> [1]PETSC ERROR: [1] PCApply_HYPRE line 203 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >>>>>>>> [1]PETSC ERROR: [1] PCApply line 430 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/interface/precon.c >>>>>>>> [1]PETSC ERROR: [1] KSP_PCApply line 234 /home/dsu/Soft/PETSc/petsc-dev/include/petsc/private/kspimpl.h >>>>>>>> [1]PETSC ERROR: [1] KSPInitialResidual line 44 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itres.c >>>>>>>> [1]PETSC ERROR: [1] KSPSolve_GMRES line 224 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>> [1]PETSC ERROR: [1] KSPSolve line 506 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 15-04-27 11:47 AM, Barry Smith wrote: >>>>>>>> >>>>>>>>> Danyang, >>>>>>>>> >>>>>>>>> I'm glad you were finally able to get to the problematic point; sorry it took so long (I want to strangle whoever turned on catching of underflows in PETSc). >>>>>>>>> >>>>>>>>> The hypre folks have done a great deal of work on their relaxation code since the PETSc 3.5.3 release. There is a good chance they have fixed the divide by zero problem you are hitting here. You will need to upgrade to the development version of PETSc (that uses the latest version of hypre), here are the instructions on how to obtain it >>>>>>>>> http://www.mcs.anl.gov/petsc/developers/index.html >>>>>>>>> >>>>>>>>> >>>>>>>>> Please let us know if this resolves the problem with hypre failing. >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Apr 27, 2015, at 11:44 AM, Danyang Su >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi Barry, >>>>>>>>>> >>>>>>>>>> I got the following arithemetic exception after the previous bug is fixed. >>>>>>>>>> >>>>>>>>>> Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 >>>>>>>>>> 0x00007f3b23b98f20 in __nanosleep_nocancel () >>>>>>>>>> at ../sysdeps/unix/syscall-template.S:81 >>>>>>>>>> 81 ../sysdeps/unix/syscall-template.S: No such file or directory. >>>>>>>>>> (gdb) cont >>>>>>>>>> Continuing. >>>>>>>>>> >>>>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>>>> 0x00007f3b260df449 in gselim (A=0x1e4c580, x=0x1d30c40, n=9) >>>>>>>>>> at par_relax.c:3442 >>>>>>>>>> 3442 x[k] /= A[k*n+k]; >>>>>>>>>> (gdb) >>>>>>>>>> >>>>>>>>>> I tried both PETSc 3.5.2 and 3.5.3 and they return the same error as shown above. For 3.5.3, i edited fp.c file and then configure and make. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Danyang >>>>>>>>>> >>>>>>>>>> On 15-04-25 07:34 PM, Danyang Su wrote: >>>>>>>>>> >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>> The "floating point underflow" is caused by a small value divided by a very large value. This result is forced to zero and then it does not report any underflow problem. I just rerun this bad case to see if it still get stuck later. This will take a while. >>>>>>>>>>> >>>>>>>>>>> Thanks for all your kindly reply, >>>>>>>>>>> >>>>>>>>>>> Danyang >>>>>>>>>>> >>>>>>>>>>> On 15-04-25 07:02 PM, Barry Smith wrote: >>>>>>>>>>> >>>>>>>>>>>> Ok, you do have >>>>>>>>>>>> >>>>>>>>>>>> #ifndef PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>>> #define PETSC_HAVE_XMMINTRIN_H 1 >>>>>>>>>>>> #endif >>>>>>>>>>>> >>>>>>>>>>>> so the change you made should cause it to stop trapping underflow exceptions. >>>>>>>>>>>> >>>>>>>>>>>> Now in one email you reported a FPE within hypre, then I asked you to run with -start_in_debugger to determine where it happened exactly and then you reported the FPE happened in user code (what seemed to be an underflow issue). Why is this? Can you not run it where it generated the FPE in hypre using the -start_in_debugger option? >>>>>>>>>>>> >>>>>>>>>>>> Barry >>>>>>>>>>>> >>>>>>>>>>>> Perhaps you have multiple PETSC_ARCH or multiple PETSc installs to explain why you reported two different places where the exception occurred. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On Apr 25, 2015, at 8:31 PM, Danyang Su >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 15-04-25 06:26 PM, Matthew Knepley wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 15-04-25 06:03 PM, Barry Smith wrote: >>>>>>>>>>>>>> If this is what you got in your last run >>>>>>>>>>>>>> >>>>>>>>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>>>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>>>>>>>>> then it is still catching floating point underflow, which we do not want. This means either the change I suggested you make in the fp.c code didn't work or it actually uses a different floating point trap than that one. BTW: absurd numbers like 1.0d-316 are often a symptom of uninitialized data; could that be a problem that diff is not filled correctly for all the ic, idim you are using? >>>>>>>>>>>>>> >>>>>>>>>>>>>> This going round and round is very frustrating and a waste of time. You need to be more proactive yourself and explore the code and poke around to figure out how to solve the problem. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can see what FP trap is being used on your machine. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Barry >>>>>>>>>>>>>> Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise I cannot find this file. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Its include/petscconf.h >>>>>>>>>>>>>> Do I need to reconfigure PETSc after changing the code you mentioned? >>>>>>>>>>>>>> >>>>>>>>>>>>>> No, but you need to rebuild. >>>>>>>>>>>>>> >>>>>>>>>>>>> Yes, I have done 'gnumake'. >>>>>>>>>>>>> >>>>>>>>>>>>>> Matt >>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Apr 25, 2015, at 2:24 PM, Danyang Su >>>>>>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 15-04-25 11:55 AM, Barry Smith wrote: >>>>>>>>>>>>>> On Apr 25, 2015, at 1:51 PM, Danyang Su >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 15-04-25 11:32 AM, Barry Smith wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> I told you this yesterday. >>>>>>>>>>>>>> >>>>>>>>>>>>>> It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate >>>>>>>>>>>>>> >>>>>>>>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>>>>>>>>>>>>> #else >>>>>>>>>>>>>> >>>>>>>>>>>>>> change it to >>>>>>>>>>>>>> >>>>>>>>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >>>>>>>>>>>>>> #else >>>>>>>>>>>>>> >>>>>>>>>>>>>> Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Did you do this? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Barry >>>>>>>>>>>>>> >>>>>>>>>>>>>> Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc directory. I just did a double check and ran make gnumake again and got the following information this time. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >>>>>>>>>>>>>> >>>>>>>>>>>>>> make gnumake >>>>>>>>>>>>>> Building PETSc using GNU Make with 10 build threads >>>>>>>>>>>>>> ========================================== >>>>>>>>>>>>>> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>>>>>>>> make[1]: Nothing to be done for `all'. >>>>>>>>>>>>>> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>>>>>>>> ========================================= >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Then I recompiled the codes, ran with -fp_trap and still got the following error >>>>>>>>>>>>>> >>>>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>>>> Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>>>>> [2]PETSC ERROR: is given. >>>>>>>>>>>>>> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>>> INSTEAD the line number of the start of the function >>>>>>>>>>>>>> [3]PETSC ERROR: is given. >>>>>>>>>>>>>> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>>> [2]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>>>>> [3]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is different then what you sent a few minutes ago where it crashed in hypre. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Anyways you need to use the -start_in_debugger business I sent in the previous email to see the exact place the problem occurs. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here is the information shown on gdb screen >>>>>>>>>>>>>> >>>>>>>>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>>>>>>>> 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, njamxc=34, >>>>>>>>>>>>>> cinfradx=..., radial_coordx=.FALSE., _suffix=3) >>>>>>>>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>>>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>>>>>>>>> (gdb) >>>>>>>>>>>>>> >>>>>>>>>>>>>> After type cont on gdb screen, I got error information as below >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>>>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>>>>> [1]PETSC ERROR: is given. >>>>>>>>>>>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Apr 25, 2015, at 1:05 AM, Danyang Su >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Barry and Satish, >>>>>>>>>>>>>> >>>>>>>>>>>>>> How can I get rid of unknown floating point error when a very small value is multiplied. >>>>>>>>>>>>>> >>>>>>>>>>>>>> e.g., >>>>>>>>>>>>>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, >>>>>>>>>>>>>> >>>>>>>>>>>>>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>>>>>>>>>>>>> >>>>>>>>>>>>>> I get the following error when run with "-fp_trap -start_in_debugger". >>>>>>>>>>>>>> >>>>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>>>> *** unknown floating point error occurred *** >>>>>>>>>>>>>> [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>>>>> [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>>>>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 >>>>>>>>>>>>>> where the result is a bitwise OR of the following flags: >>>>>>>>>>>>>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>>> [2]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>>>>>>> >>>>>>>>>>>>>> I don't think this number can be stored in a real*4. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Satish >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Satish. It is caused by this number. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>> >>>>>>>>>>>>>> One of my case crashes because of floating point exception when using 4 >>>>>>>>>>>>>> processors, as shown below. But if I run this case with 1 processor, it >>>>>>>>>>>>>> works fine. I have tested the codes with around 100 cases up to 768 >>>>>>>>>>>>>> processors, all other cases work fine. I just wonder if this kind of error >>>>>>>>>>>>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Yes, almost for sure it is one of these places. >>>>>>>>>>>>>> >>>>>>>>>>>>>> First run the bad case with -fp_trap if all goes well you'll see the >>>>>>>>>>>>>> function where the FPE is generated. Then run also with -start_in_debugger >>>>>>>>>>>>>> and >>>>>>>>>>>>>> type cont in all four debugger windows. When the FPE happens the debugger >>>>>>>>>>>>>> should stop showing exactly where the FPE happens. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Barry >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Barry, >>>>>>>>>>>>>> >>>>>>>>>>>>>> If run with -fp_trap -start_in_debugger, I got the following error >>>>>>>>>>>>>> >>>>>>>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a >>>>>>>>>>>>>> debugger. When the >>>>>>>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >>>>>>>>>>>>>> fetestexcept(0x3d) >>>>>>>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>>>>>>>>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>>>>>>>>>> ------------------------------------ >>>>>>>>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>>>>> [0]PETSC ERROR: is given. >>>>>>>>>>>>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>>>>>>>>>>>>> floating point error >>>>>>>>>>>>>> >>>>>>>>>>>>>> Program received signal SIGABRT: Process abort signal. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>>>> #0 0x7F4FEAB1C7D7 >>>>>>>>>>>>>> #1 0x7F4FEAB1CDDE >>>>>>>>>>>>>> #2 0x7F4FE9E1AD3F >>>>>>>>>>>>>> #3 0x7F4FE9E1ACC9 >>>>>>>>>>>>>> #4 0x7F4FE9E1E0D7 >>>>>>>>>>>>>> #5 0x7F4FEB0B6DCB >>>>>>>>>>>>>> #6 0x7F4FEB0B1825 >>>>>>>>>>>>>> #7 0x7F4FEB0B817F >>>>>>>>>>>>>> #8 0x7F4FE9E1AD3F >>>>>>>>>>>>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>>>>>>>>>>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>>>>>>>>>>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>>>>>>>>>>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>>>>>>>>>>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>>>>>>>>>>>> >>>>>>>>>>>>>> I checked the code at tprfrtlc.F90:2393, >>>>>>>>>>>>>> >>>>>>>>>>>>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>>>>>>>>>>>> (cx(ix),ix=1,nxout)/) >>>>>>>>>>>>>> >>>>>>>>>>>>>> All the values (time, c, cx) are reasonable, as shown below. The only >>>>>>>>>>>>>> possibility is that realbuffer_gb is in declared as real*4 if using sing >>>>>>>>>>>>>> precision output while time, c, cx are declared in real*8. I have a lot of >>>>>>>>>>>>>> similar data conversion from real*8 to real*4 output, other code does not >>>>>>>>>>>>>> return error. >>>>>>>>>>>>>> >>>>>>>>>>>>>> time 0.0000000000000000 >>>>>>>>>>>>>> c 1 9.9999999999999995E-008 >>>>>>>>>>>>>> c 2 3.1555251077549618E-003 >>>>>>>>>>>>>> c 3 7.1657814842179362E-008 >>>>>>>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>>>>>>> c 5 5.2879822292305797E-004 >>>>>>>>>>>>>> c 6 9.9999999999999964E-005 >>>>>>>>>>>>>> c 7 6.4055731968811337E-005 >>>>>>>>>>>>>> c 8 3.4607572892578404E-020 >>>>>>>>>>>>>> cx 1 3.4376650636008101E-005 >>>>>>>>>>>>>> cx 2 7.3989678854017763E-012 >>>>>>>>>>>>>> cx 3 9.5317170613607207E-012 >>>>>>>>>>>>>> cx 4 2.2344525794718353E-015 >>>>>>>>>>>>>> cx 5 3.0624685689695889E-008 >>>>>>>>>>>>>> cx 6 1.0046157902783967E-007 >>>>>>>>>>>>>> cx 7 1.5320169154914984E-004 >>>>>>>>>>>>>> cx 8 8.6930292776346176E-014 >>>>>>>>>>>>>> cx 9 3.5944267559348721E-005 >>>>>>>>>>>>>> cx 10 3.0072645866951157E-018 >>>>>>>>>>>>>> cx 11 2.3592486321095017E-013 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I can check all the entries of jacobi matrix to see if the value is valid, >>>>>>>>>>>>>> but this seems not a good idea as it takes a long time to reach this >>>>>>>>>>>>>> point. If I restart the simulation from a specified time (e.g., 7.685 in >>>>>>>>>>>>>> this case), then the error does not occur. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Would you please give me any suggestion on debugging this case? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>>>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>>>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>>>> [1]PETSC ERROR: Floating point exception >>>>>>>>>>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>>>> [2]PETSC ERROR: Floating point exception >>>>>>>>>>>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >>>>>>>>>>>>>> at end of function: Parameter number 3 >>>>>>>>>>>>>> [2]PETSC ERROR: See >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> for trouble shooting. >>>>>>>>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>>>>>>>>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>>>>>>>>>>> [1]PETSC ERROR: See >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> for trouble shooting. >>>>>>>>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>>>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>>>>>>>>>>>>> dsu Thu Apr 23 15:38:52 2015 >>>>>>>>>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>>>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>>>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >>>>>>>>>>>>>> 15:38:52 2015 >>>>>>>>>>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>>>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>>>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>>>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>>>>>> #3 KSP_PCApply() line 230 in >>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>>>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>>>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>>>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>>>>>>>>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> From knepley at gmail.com Wed Apr 29 14:39:19 2015 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 30 Apr 2015 05:39:19 +1000 Subject: [petsc-users] Floating point exception in hypre BoomerAMG In-Reply-To: <55413294.1000906@gmail.com> References: <553A85DF.5000400@gmail.com> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> <553C3E15.8030403@gmail.com> <553C3FE6.5020203@gmail.com> <553C4ECD.6090905@gmail.com> <553E6770.5010205@gmail.com> <553FC98D.1000904@gmail.com> <89DAC785-3218-4E2F-8760-2D0E99652014@mcs.anl.gov> <5540261E.2040502@gmail.com> <6411F61F-CD41-4037-B727-B5D9CECE030F@mcs.anl.gov> <554111AD.3060007@gmail.com> <9C61B774-4A78-48E1-B34E-B23EC2D230B9@mcs.anl.gov> <55412888.90206@gmail.com> <0F8EF2B9-9A1C-4C94-B1CC-D520B6D7A0DE@mcs.anl.gov> <55413294.1000906@gmail.com> Message-ID: On Thu, Apr 30, 2015 at 5:35 AM, Danyang Su wrote: > On 15-04-29 12:19 PM, Barry Smith wrote: > >> Ok, your code seems fine in terms of logic. But the vector x_vec_gbl >> i.e. b_react should not be in a read only state. Are you somewhere calling >> a VecGetArray() and not calling the VecRestoreArray()? >> >> Barry >> > I just made a triple check on VecGetArrayF90(). All the VecGetArrayF90() > come with VecRestoreArrayF90(). No VecGetArray() is used in my codes. > So you can check the lock status of the Vec: http://www.mcs.anl.gov/petsc/petsc-master/docs/manualpages/Vec/VecLockGet.html Put these in your code until you can see who is locking the Vec before the call. Thanks, Matt > Danyang > >> >> >> >> On Apr 29, 2015, at 1:52 PM, Danyang Su wrote: >>> >>> On 15-04-29 11:30 AM, Barry Smith wrote: >>> >>>> On Apr 29, 2015, at 12:15 PM, Danyang Su >>>>> wrote: >>>>> >>>>> On 15-04-28 06:50 PM, Barry Smith wrote: >>>>> >>>>> We started enforcing more checks on writing to vectors that you >>>>>> should not write to. Where are you calling DMLocalToGlobalBegin() ? It >>>>>> looks like you are trying to copy values into a global array that you >>>>>> should not because it is a read only input to a function, for example it is >>>>>> the right hand side of a linear system or the input into a >>>>>> SNESFormFunction(). So check the output to that call. >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>> This is used when local RHS vec is calculated and copied to global >>>>> RHS. >>>>> >>>>> Huhh, why are you copying anything into the RHS? Is this before >>>> you call the linear system solve? Send the code that calls the >>>> DMLocalToGlobalBegin() >>>> >>> Yes. This is called before the linear system solver. At present, we use >>> PETSc KSP solver. >>> The code is first developed in totally sequential version and then >>> ported to the parallel version. So the subdomain has its own space for >>> jacobi matrix and rhs. We do not use global RHS vector directly to set the >>> RHS at this moment. All the RHS values of subdomain are copied to the >>> global RHS. >>> >>> !> This is the function where RHS values of subdomain are copied >>> to the global RHS values. >>> subroutine compute_function(rank,da,x_array_loc,x_vec_loc, & >>> x_vec_gbl,nngl,row_idx_l2pg,col_idx_l2pg, & >>> b_non_interlaced) >>> implicit none >>> #include >>> #include >>> #include >>> #include >>> #include >>> #include >>> #include >>> #include >>> #include >>> >>> PetscInt :: rank >>> DM :: da >>> PetscReal, allocatable :: x_array_loc(:) >>> Vec :: x_vec_loc >>> Vec :: x_vec_gbl >>> PetscInt :: nngl >>> PetscInt, allocatable :: row_idx_l2pg(:) >>> PetscInt, allocatable :: col_idx_l2pg(:) >>> PetscBool :: b_non_interlaced >>> >>> PetscInt :: info_debug >>> PetscInt :: i, j >>> PetscErrorCode :: ierr >>> PetscScalar, pointer :: vecpointer(:) >>> !Zero entries >>> call VecZeroEntries(x_vec_loc, ierr) >>> !Get a pointer to vector data when you need access >>> to the array >>> call VecGetArrayF90(x_vec_loc, vecpointer, ierr) >>> !Compute the function over the locally owned part >>> of the grid >>> if(b_non_interlaced) then >>> j = nngl/2 >>> do i = 1, j >>> vecpointer(2*i-1) = x_array_loc(i) >>> vecpointer(2*i) = x_array_loc(i+j) >>> end do >>> else >>> do i = 1, nngl >>> vecpointer(i) = x_array_loc(i) >>> end do >>> end if >>> !Restore the vector when you no longer need access >>> to the array >>> call VecRestoreArrayF90(x_vec_loc,vecpointer,ierr) >>> !Insert values into global vector >>> call DMLocalToGlobalBegin(da,x_vec_loc,INSERT_VALUES, & >>> x_vec_gbl,ierr) >>> !By placing code between these two statements, computations >>> can be >>> !done while messages are in transition. >>> call DMLocalToGlobalEnd(da,x_vec_loc,INSERT_VALUES, & >>> x_vec_gbl,ierr) >>> return >>> end subroutine >>> >>> >>> >>> !> This is the function where reactive transport equations are >>> solved. >>> subroutine solver_dd_snes_solve_react(ilog,idetail,a_in,b_in, & >>> x_inout,ia_in,ja_in,nngl_in,itsolv, & >>> over_flow,rnorm,row_idx_l2pg,col_idx_l2pg, & >>> b_non_interlaced) >>> use gen, only : rank, node_idx_l2lg, ittot_rt, >>> & >>> b_output_matrix, b_enable_output >>> use solver_snes_function, only : form_initial_guess, & >>> compute_function, & >>> compute_jacobian >>> use petsc_mpi_common, only : petsc_mpi_finalize >>> implicit none >>> #include >>> #include >>> #include >>> #include >>> >>> PetscInt :: ilog >>> PetscInt :: idetail >>> PetscInt :: nngl_in >>> PetscReal, allocatable :: a_in(:) >>> PetscReal, allocatable :: b_in(:) >>> PetscReal, allocatable :: x_inout(:) >>> PetscInt, allocatable :: ia_in(:) >>> PetscInt, allocatable :: ja_in(:) >>> PetscInt, allocatable :: row_idx_l2pg(:) >>> PetscInt, allocatable :: col_idx_l2pg(:) >>> PetscInt :: itsolv >>> PetscBool :: over_flow >>> PetscBool :: b_non_interlaced >>> PetscReal :: rnorm >>> PetscViewer :: viewer >>> PetscScalar,pointer :: vecpointer(:) >>> >>> PetscErrorCode :: ierr >>> PetscInt :: info_debug >>> character(72) :: strinum >>> info_debug = 0 >>> if(b_output_matrix .and. b_enable_output) then >>> write(strinum, *) ittot_rt >>> strinum = "_"//trim(adjustl(strinum)) >>> end if >>> >>> !Form initial guess, only assemble the local >>> owned part, without ghost nodes >>> call form_initial_guess(rank,dmda_react%da,x_inout,x_react_loc,& >>> x_react,nngl_in, row_idx_l2pg,col_idx_l2pg, & >>> b_non_interlaced) >>> >>> !Compute function, only assemble the local part, >>> without ghost nodes >>> call compute_function(rank,dmda_react%da,b_in,b_react_loc, & >>> b_react,nngl_in,row_idx_l2pg,col_idx_l2pg, & >>> b_non_interlaced) >>> !Compute jacobian matrix, only assemble the >>> local part, without ghost nodes >>> call compute_jacobian(rank,dmda_react%da, & >>> a_react,a_in,ia_in,ja_in,nngl_in, & >>> row_idx_l2pg,col_idx_l2pg, & >>> b_non_interlaced) >>> >>> ! Solve a x = b, where a is the Jacobian matrix. >>> ! - First, set the KSP linear operators. Here the matrix >>> that >>> ! defines the linear system also serves as the >>> preconditioning >>> ! matrix. >>> ! - Then solve the Newton system. >>> #ifdef PETSC_V3_5_X >>> call KSPSetOperators(ksp_react,a_react,a_react,ierr) >>> #else >>> call KSPSetOperators(ksp_react,a_react,a_react, & >>> SAME_NONZERO_PATTERN,ierr) >>> #endif >>> call KSPSetDM(ksp_react,dmda_react%da,ierr) >>> call KSPSetDMActive(ksp_react,PETSC_FALSE,ierr) >>> >>> call KSPSolve(ksp_react,b_react,x_react,ierr) >>> !Get residual norm, by default, preconditioned >>> residual norm is calculated. >>> if (b_mykspconverged_react .and. & >>> .not. b_use_petsc_default_react) then >>> rnorm = rnorm_react >>> else >>> call KSPGetResidualNorm(ksp_react, rnorm, ierr) >>> end if >>> ! Scatter ghost points to local vector, using the >>> 2-step process >>> ! DMGlobalToLocalBegin(), DMGlobalToLocalEnd(). >>> ! By placing code between these two statements, computations >>> can be >>> ! done while messages are in transition. >>> call DMGlobalToLocalBegin(dmda_react%da,x_react,INSERT_VALUES, & >>> x_react_loc,ierr) >>> call DMGlobalToLocalEnd(dmda_react%da,x_react,INSERT_VALUES, & >>> x_react_loc,ierr) >>> call VecGetArrayF90(x_react_loc,vecpointer,ierr) >>> x_inout = vecpointer >>> call VecRestoreArrayF90(x_react_loc,vecpointer,ierr) >>> return >>> end subroutine >>> >>> Barry >>>> >>>> >>>> It is strange why this error does not occur at the first timestep. >>>>> >>>>> Thanks, >>>>> >>>>> Danyang >>>>> >>>>> On Apr 28, 2015, at 7:30 PM, Danyang Su >>>>>>> wrote: >>>>>>> >>>>>>> Hi Barry, >>>>>>> >>>>>>> There seems another bug (not pretty sure) in PETSc-dev, as shown >>>>>>> below. The case I used is similar with the one I mentioned recently. I have >>>>>>> no problem running this case using PETSc 3.5.2. But it give out the >>>>>>> following error using PETSc-dev, even with only one processor. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Danyang >>>>>>> >>>>>>> timestep: 2048 time: 4.615E+00 years delt: 1.460E-20 years >>>>>>> iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>> -------------------------------------------------------------- >>>>>>> [0]PETSC ERROR: Object is in wrong state >>>>>>> [0]PETSC ERROR: Vec is locked read only, argument # 1 >>>>>>> [0]PETSC ERROR: See >>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>> for trouble shooting. >>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>> v3.5.3-2796-g23b6c49 GIT Date: 2015-04-27 11:32:44 -0500 >>>>>>> [0]PETSC ERROR: ../min3p_thcm on a linux-gnu-dbg named nwmop by dsu >>>>>>> Tue Apr 28 15:56:41 2015 >>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg >>>>>>> --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack >>>>>>> --download-mpich --download-mumps --download-hypre --download-superlu_dist >>>>>>> --download-metis --download-parmetis --download-scalapack >>>>>>> [0]PETSC ERROR: #349 VecGetArray() line 1646 in >>>>>>> /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/interface/rvector.c >>>>>>> [0]PETSC ERROR: #350 VecGetArrayPair() line 434 in >>>>>>> /home/dsu/Soft/PETSc/petsc-dev/include/petscvec.h >>>>>>> [0]PETSC ERROR: #351 VecScatterBegin_SSToSS() line 649 in >>>>>>> /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c >>>>>>> [0]PETSC ERROR: #352 VecScatterBegin() line 1694 in >>>>>>> /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c >>>>>>> [0]PETSC ERROR: #353 DMLocalToGlobalBegin_DA() line 56 in >>>>>>> /home/dsu/Soft/PETSc/petsc-dev/src/dm/impls/da/dagtol.c >>>>>>> [0]PETSC ERROR: #354 DMLocalToGlobalBegin() line 1944 in >>>>>>> /home/dsu/Soft/PETSc/petsc-dev/src/dm/interface/dm.c >>>>>>> Reduce time step for reactive transport >>>>>>> no further time step reduction possible >>>>>>> Optimal time increment computed by MIN3P 2.6644769214899893E-018 >>>>>>> Minimum time increment specified by user 3.6499999999999998E-018 >>>>>>> Please, reduce the MINIMUM TIME INCREMENT >>>>>>> stop signal in time step reduction failed >>>>>>> >>>>>>> On 15-04-28 11:15 AM, Barry Smith wrote: >>>>>>> >>>>>>> I am forwarding this on to the hypre support email list; >>>>>>>> hopefully they can have some advice on how to proceed. >>>>>>>> >>>>>>>> hypre folks, this is with version hypre-2.10.0b (the previous >>>>>>>> version had the same behavior). We are getting a divide by zero in gselim() >>>>>>>> line 4363 (this happens in a time dependent problem after many successful >>>>>>>> solves with time dependent matrix). I looked at the code, below, and note >>>>>>>> that there are some checks for the diagonal of A not being zero but not for >>>>>>>> the line that causes the divide by zero; is this perhaps an oversight in >>>>>>>> the hypre code? Any advice appreciated. >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> HYPRE_Int gselim(A,x,n) >>>>>>>> HYPRE_Real *A; >>>>>>>> HYPRE_Real *x; >>>>>>>> HYPRE_Int n; >>>>>>>> { >>>>>>>> HYPRE_Int err_flag = 0; >>>>>>>> HYPRE_Int j,k,m; >>>>>>>> HYPRE_Real factor; >>>>>>>> if (n==1) /* A is 1x1 */ >>>>>>>> { >>>>>>>> if (A[0] != 0.0) >>>>>>>> { >>>>>>>> x[0] = x[0]/A[0]; >>>>>>>> return(err_flag); >>>>>>>> } >>>>>>>> else >>>>>>>> { >>>>>>>> err_flag = 1; >>>>>>>> return(err_flag); >>>>>>>> } >>>>>>>> } >>>>>>>> else /* A is nxn. Forward >>>>>>>> elimination */ >>>>>>>> { >>>>>>>> for (k = 0; k < n-1; k++) >>>>>>>> { >>>>>>>> if (A[k*n+k] != 0.0) >>>>>>>> { >>>>>>>> for (j = k+1; j < n; j++) >>>>>>>> { >>>>>>>> if (A[j*n+k] != 0.0) >>>>>>>> { >>>>>>>> factor = A[j*n+k]/A[k*n+k]; >>>>>>>> for (m = k+1; m < n; m++) >>>>>>>> { >>>>>>>> A[j*n+m] -= factor * A[k*n+m]; >>>>>>>> } >>>>>>>> /* Elimination step for rhs */ >>>>>>>> x[j] -= factor * x[k]; >>>>>>>> } >>>>>>>> } >>>>>>>> } >>>>>>>> } >>>>>>>> /* Back Substitution */ >>>>>>>> for (k = n-1; k > 0; --k) >>>>>>>> { >>>>>>>> x[k] /= A[k*n+k]; >>>>>>>> for (j = 0; j < k; j++) >>>>>>>> { >>>>>>>> if (A[j*n+k] != 0.0) >>>>>>>> { >>>>>>>> x[j] -= x[k] * A[j*n+k]; >>>>>>>> } >>>>>>>> } >>>>>>>> } >>>>>>>> x[0] /= A[0]; >>>>>>>> return(err_flag); >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Apr 28, 2015, at 12:55 PM, Danyang Su >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi Barry, >>>>>>>>> >>>>>>>>> The development version of PETSc does not help to solve my >>>>>>>>> problem. It still crashed due to the same error information. >>>>>>>>> >>>>>>>>> As Matthew mentioned, I checked the matrix and non of the diagonal >>>>>>>>> entry is zero. But the diagonal values for different rows range from >>>>>>>>> -1.0d-18 to 1.0d33 and the matrix is not strict diagonal dominant. When >>>>>>>>> using 1 processor, the matrix is also not strict diagonal dominant, but the >>>>>>>>> diagonal values have a much smaller range and the codes can run without >>>>>>>>> error. Maybe there is something wrong in the matrix for this case and I >>>>>>>>> need to check the matrix first. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Danyang >>>>>>>>> >>>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>>> 0x00007f668c1299d5 in gselim (A=0x2fe3330, x=0x2210680, n=9) >>>>>>>>> at par_relax.c:4363 >>>>>>>>> 4363 x[k] /= A[k*n+k]; >>>>>>>>> (gdb) >>>>>>>>> >>>>>>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>> [1]PETSC ERROR: The specific exception can be determined by >>>>>>>>> running in a debugger. When the >>>>>>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be >>>>>>>>> found with fetestexcept(0x3d) >>>>>>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following >>>>>>>>> flags: >>>>>>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>>>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>>>>> [1]PETSC ERROR: --------------------- Stack Frames >>>>>>>>> ------------------------------------ >>>>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>>>>>>> available, >>>>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the >>>>>>>>> function >>>>>>>>> [1]PETSC ERROR: is given. >>>>>>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 >>>>>>>>> /home/dsu/Soft/PETSc/petsc-dev/src/sys/error/fp.c >>>>>>>>> [1]PETSC ERROR: [1] Hypre solve line 216 >>>>>>>>> /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >>>>>>>>> [1]PETSC ERROR: [1] PCApply_HYPRE line 203 >>>>>>>>> /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >>>>>>>>> [1]PETSC ERROR: [1] PCApply line 430 >>>>>>>>> /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/interface/precon.c >>>>>>>>> [1]PETSC ERROR: [1] KSP_PCApply line 234 >>>>>>>>> /home/dsu/Soft/PETSc/petsc-dev/include/petsc/private/kspimpl.h >>>>>>>>> [1]PETSC ERROR: [1] KSPInitialResidual line 44 >>>>>>>>> /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itres.c >>>>>>>>> [1]PETSC ERROR: [1] KSPSolve_GMRES line 224 >>>>>>>>> /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>> [1]PETSC ERROR: [1] KSPSolve line 506 >>>>>>>>> /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>>>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file >>>>>>>>> trapped floating point error >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 15-04-27 11:47 AM, Barry Smith wrote: >>>>>>>>> >>>>>>>>> Danyang, >>>>>>>>>> >>>>>>>>>> I'm glad you were finally able to get to the problematic >>>>>>>>>> point; sorry it took so long (I want to strangle whoever turned on catching >>>>>>>>>> of underflows in PETSc). >>>>>>>>>> >>>>>>>>>> The hypre folks have done a great deal of work on their >>>>>>>>>> relaxation code since the PETSc 3.5.3 release. There is a good chance they >>>>>>>>>> have fixed the divide by zero problem you are hitting here. You will need >>>>>>>>>> to upgrade to the development version of PETSc (that uses the latest >>>>>>>>>> version of hypre), here are the instructions on how to obtain it >>>>>>>>>> http://www.mcs.anl.gov/petsc/developers/index.html >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Please let us know if this resolves the problem with hypre >>>>>>>>>> failing. >>>>>>>>>> >>>>>>>>>> Barry >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Apr 27, 2015, at 11:44 AM, Danyang Su >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Barry, >>>>>>>>>>> >>>>>>>>>>> I got the following arithemetic exception after the previous bug >>>>>>>>>>> is fixed. >>>>>>>>>>> >>>>>>>>>>> Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 >>>>>>>>>>> 0x00007f3b23b98f20 in __nanosleep_nocancel () >>>>>>>>>>> at ../sysdeps/unix/syscall-template.S:81 >>>>>>>>>>> 81 ../sysdeps/unix/syscall-template.S: No such file or >>>>>>>>>>> directory. >>>>>>>>>>> (gdb) cont >>>>>>>>>>> Continuing. >>>>>>>>>>> >>>>>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>>>>> 0x00007f3b260df449 in gselim (A=0x1e4c580, x=0x1d30c40, n=9) >>>>>>>>>>> at par_relax.c:3442 >>>>>>>>>>> 3442 x[k] /= A[k*n+k]; >>>>>>>>>>> (gdb) >>>>>>>>>>> >>>>>>>>>>> I tried both PETSc 3.5.2 and 3.5.3 and they return the same >>>>>>>>>>> error as shown above. For 3.5.3, i edited fp.c file and then configure and >>>>>>>>>>> make. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Danyang >>>>>>>>>>> >>>>>>>>>>> On 15-04-25 07:34 PM, Danyang Su wrote: >>>>>>>>>>> >>>>>>>>>>> Hi All, >>>>>>>>>>>> >>>>>>>>>>>> The "floating point underflow" is caused by a small value >>>>>>>>>>>> divided by a very large value. This result is forced to zero and then it >>>>>>>>>>>> does not report any underflow problem. I just rerun this bad case to see if >>>>>>>>>>>> it still get stuck later. This will take a while. >>>>>>>>>>>> >>>>>>>>>>>> Thanks for all your kindly reply, >>>>>>>>>>>> >>>>>>>>>>>> Danyang >>>>>>>>>>>> >>>>>>>>>>>> On 15-04-25 07:02 PM, Barry Smith wrote: >>>>>>>>>>>> >>>>>>>>>>>> Ok, you do have >>>>>>>>>>>>> >>>>>>>>>>>>> #ifndef PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>>>> #define PETSC_HAVE_XMMINTRIN_H 1 >>>>>>>>>>>>> #endif >>>>>>>>>>>>> >>>>>>>>>>>>> so the change you made should cause it to stop trapping >>>>>>>>>>>>> underflow exceptions. >>>>>>>>>>>>> >>>>>>>>>>>>> Now in one email you reported a FPE within hypre, then I >>>>>>>>>>>>> asked you to run with -start_in_debugger to determine where it happened >>>>>>>>>>>>> exactly and then you reported the FPE happened in user code (what seemed >>>>>>>>>>>>> to be an underflow issue). Why is this? Can you not run it where it >>>>>>>>>>>>> generated the FPE in hypre using the -start_in_debugger option? >>>>>>>>>>>>> >>>>>>>>>>>>> Barry >>>>>>>>>>>>> >>>>>>>>>>>>> Perhaps you have multiple PETSC_ARCH or multiple PETSc >>>>>>>>>>>>> installs to explain why you reported two different places where the >>>>>>>>>>>>> exception occurred. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Apr 25, 2015, at 8:31 PM, Danyang Su >>>>>>>>>>>>> > >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 15-04-25 06:26 PM, Matthew Knepley wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su < >>>>>>>>>>>>>>> danyang.su at gmail.com> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 15-04-25 06:03 PM, Barry Smith wrote: >>>>>>>>>>>>>>> If this is what you got in your last run >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>>>>>>>>> 1344 cinfrt = >>>>>>>>>>>>>>> cinfrt_dg(i1) * diff(ic,idim) !diff is >>>>>>>>>>>>>>> a very small value, e.g., 1.0d-316 >>>>>>>>>>>>>>> then it is still catching floating point underflow, >>>>>>>>>>>>>>> which we do not want. This means either the change I suggested you make in >>>>>>>>>>>>>>> the fp.c code didn't work or it actually uses a different floating point >>>>>>>>>>>>>>> trap than that one. BTW: absurd numbers like 1.0d-316 are often a symptom >>>>>>>>>>>>>>> of uninitialized data; could that be a problem that diff is not filled >>>>>>>>>>>>>>> correctly for all the ic, idim you are using? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This going round and round is very frustrating and a >>>>>>>>>>>>>>> waste of time. You need to be more proactive yourself and explore the code >>>>>>>>>>>>>>> and poke around to figure out how to solve the problem. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Please email >>>>>>>>>>>>>>> $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can see what FP trap >>>>>>>>>>>>>>> is being used on your machine. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Barry >>>>>>>>>>>>>>> Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? >>>>>>>>>>>>>>> Otherwise I cannot find this file. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Its include/petscconf.h >>>>>>>>>>>>>>> Do I need to reconfigure PETSc after changing the code you >>>>>>>>>>>>>>> mentioned? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> No, but you need to rebuild. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yes, I have done 'gnumake'. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Matt >>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Apr 25, 2015, at 2:24 PM, Danyang Su >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 15-04-25 11:55 AM, Barry Smith wrote: >>>>>>>>>>>>>>> On Apr 25, 2015, at 1:51 PM, Danyang Su >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 15-04-25 11:32 AM, Barry Smith wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I told you this yesterday. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It is probably stopping here on a harmless underflow. >>>>>>>>>>>>>>> You need to edit the PETSc code to not worry about underflow. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Edit the file >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>>>>>>>>>>>>>> #else >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> change it to >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | >>>>>>>>>>>>>>> _MM_MASK_UNDERFLOW); >>>>>>>>>>>>>>> #else >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Then run make gnumake in the PETSc directory to compile >>>>>>>>>>>>>>> the new version. Now link and run the program again with -fp_trap and see >>>>>>>>>>>>>>> where it gets stuck this time. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Did you do this? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Barry >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yes, I did change the code in fp.c and run 'make gnumake' in >>>>>>>>>>>>>>> the PETSc directory. I just did a double check and ran make gnumake again >>>>>>>>>>>>>>> and got the following information this time. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> make gnumake >>>>>>>>>>>>>>> Building PETSc using GNU Make with 10 build threads >>>>>>>>>>>>>>> ========================================== >>>>>>>>>>>>>>> make[1]: Entering directory >>>>>>>>>>>>>>> `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>>>>>>>>> make[1]: Nothing to be done for `all'. >>>>>>>>>>>>>>> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>>>>>>>>> ========================================= >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Then I recompiled the codes, ran with -fp_trap and still got >>>>>>>>>>>>>>> the following error >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>>>>> Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>>>>>> [2]PETSC ERROR: INSTEAD the line number of the start >>>>>>>>>>>>>>> of the function >>>>>>>>>>>>>>> [2]PETSC ERROR: is given. >>>>>>>>>>>>>>> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>>>> INSTEAD the line number of the start of the function >>>>>>>>>>>>>>> [3]PETSC ERROR: is given. >>>>>>>>>>>>>>> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>>>> [2]PETSC ERROR: User provided function() line 0 in Unknown >>>>>>>>>>>>>>> file trapped floating point error >>>>>>>>>>>>>>> [3]PETSC ERROR: User provided function() line 0 in Unknown >>>>>>>>>>>>>>> file trapped floating point error >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is different then what you sent a few minutes ago >>>>>>>>>>>>>>> where it crashed in hypre. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Anyways you need to use the -start_in_debugger business >>>>>>>>>>>>>>> I sent in the previous email to see the exact place the problem occurs. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Here is the information shown on gdb screen >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>>>>>>>>> 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., >>>>>>>>>>>>>>> nmax=12, njamxc=34, >>>>>>>>>>>>>>> cinfradx=..., radial_coordx=.FALSE., _suffix=3) >>>>>>>>>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>>>>>>>>> 1344 cinfrt = >>>>>>>>>>>>>>> cinfrt_dg(i1) * diff(ic,idim) !diff is >>>>>>>>>>>>>>> a very small value, e.g., 1.0d-316 >>>>>>>>>>>>>>> (gdb) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> After type cont on gdb screen, I got error information as >>>>>>>>>>>>>>> below >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>>>>> [1]PETSC ERROR: The specific exception can be determined by >>>>>>>>>>>>>>> running in a debugger. When the >>>>>>>>>>>>>>> [1]PETSC ERROR: debugger traps the signal, the exception can >>>>>>>>>>>>>>> be found with fetestexcept(0x3d) >>>>>>>>>>>>>>> [1]PETSC ERROR: where the result is a bitwise OR of the >>>>>>>>>>>>>>> following flags: >>>>>>>>>>>>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 >>>>>>>>>>>>>>> FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>>>> [1]PETSC ERROR: likely location of problem given in stack >>>>>>>>>>>>>>> below >>>>>>>>>>>>>>> [1]PETSC ERROR: --------------------- Stack Frames >>>>>>>>>>>>>>> ------------------------------------ >>>>>>>>>>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack >>>>>>>>>>>>>>> are not available, >>>>>>>>>>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start >>>>>>>>>>>>>>> of the function >>>>>>>>>>>>>>> [1]PETSC ERROR: is given. >>>>>>>>>>>>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown >>>>>>>>>>>>>>> file trapped floating point error >>>>>>>>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by >>>>>>>>>>>>>>> running in a debugger. When the >>>>>>>>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can >>>>>>>>>>>>>>> be found with fetestexcept(0x3d) >>>>>>>>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the >>>>>>>>>>>>>>> following flags: >>>>>>>>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 >>>>>>>>>>>>>>> FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack >>>>>>>>>>>>>>> below >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Apr 25, 2015, at 1:05 AM, Danyang Su >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Barry and Satish, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> How can I get rid of unknown floating point error when a >>>>>>>>>>>>>>> very small value is multiplied. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> e.g., >>>>>>>>>>>>>>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 >>>>>>>>>>>>>>> 8.6178408169776945E-317 respectively, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I get the following error when run with "-fp_trap >>>>>>>>>>>>>>> -start_in_debugger". >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>>>>> *** unknown floating point error occurred *** >>>>>>>>>>>>>>> [2]PETSC ERROR: The specific exception can be determined by >>>>>>>>>>>>>>> running in a debugger. When the >>>>>>>>>>>>>>> [2]PETSC ERROR: debugger traps the signal, the exception can >>>>>>>>>>>>>>> be found with fetestexcept(0x3d) >>>>>>>>>>>>>>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) >>>>>>>>>>>>>>> 1.0250235986806329E-008 8.6178408169776945E-317 >>>>>>>>>>>>>>> where the result is a bitwise OR of the following flags: >>>>>>>>>>>>>>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 >>>>>>>>>>>>>>> FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>>>> [2]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>>>> [2]PETSC ERROR: likely location of problem given in stack >>>>>>>>>>>>>>> below >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I don't think this number can be stored in a real*4. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Satish >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, Satish. It is caused by this number. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> One of my case crashes because of floating point exception >>>>>>>>>>>>>>> when using 4 >>>>>>>>>>>>>>> processors, as shown below. But if I run this case with 1 >>>>>>>>>>>>>>> processor, it >>>>>>>>>>>>>>> works fine. I have tested the codes with around 100 cases up >>>>>>>>>>>>>>> to 768 >>>>>>>>>>>>>>> processors, all other cases work fine. I just wonder if this >>>>>>>>>>>>>>> kind of error >>>>>>>>>>>>>>> is caused because of NaN in jacobi matrix, RHS or >>>>>>>>>>>>>>> preconditioner? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yes, almost for sure it is one of these places. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> First run the bad case with -fp_trap if all goes well >>>>>>>>>>>>>>> you'll see the >>>>>>>>>>>>>>> function where the FPE is generated. Then run also with >>>>>>>>>>>>>>> -start_in_debugger >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>> type cont in all four debugger windows. When the FPE happens >>>>>>>>>>>>>>> the debugger >>>>>>>>>>>>>>> should stop showing exactly where the FPE happens. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Barry >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Barry, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If run with -fp_trap -start_in_debugger, I got the following >>>>>>>>>>>>>>> error >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by >>>>>>>>>>>>>>> running in a >>>>>>>>>>>>>>> debugger. When the >>>>>>>>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can >>>>>>>>>>>>>>> be found with >>>>>>>>>>>>>>> fetestexcept(0x3d) >>>>>>>>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the >>>>>>>>>>>>>>> following flags: >>>>>>>>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 >>>>>>>>>>>>>>> FE_OVERFLOW=0x8 >>>>>>>>>>>>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack >>>>>>>>>>>>>>> below >>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>>>>>>>>>>> ------------------------------------ >>>>>>>>>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack >>>>>>>>>>>>>>> are not available, >>>>>>>>>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start >>>>>>>>>>>>>>> of the function >>>>>>>>>>>>>>> [0]PETSC ERROR: is given. >>>>>>>>>>>>>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>>>> [0]PETSC ERROR: User provided function() line 0 in Unknown >>>>>>>>>>>>>>> file trapped >>>>>>>>>>>>>>> floating point error >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Program received signal SIGABRT: Process abort signal. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>>>>> #0 0x7F4FEAB1C7D7 >>>>>>>>>>>>>>> #1 0x7F4FEAB1CDDE >>>>>>>>>>>>>>> #2 0x7F4FE9E1AD3F >>>>>>>>>>>>>>> #3 0x7F4FE9E1ACC9 >>>>>>>>>>>>>>> #4 0x7F4FE9E1E0D7 >>>>>>>>>>>>>>> #5 0x7F4FEB0B6DCB >>>>>>>>>>>>>>> #6 0x7F4FEB0B1825 >>>>>>>>>>>>>>> #7 0x7F4FEB0B817F >>>>>>>>>>>>>>> #8 0x7F4FE9E1AD3F >>>>>>>>>>>>>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 >>>>>>>>>>>>>>> (discriminator 3) >>>>>>>>>>>>>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>>>>>>>>>>>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>>>>>>>>>>>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>>>>>>>>>>>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I checked the code at tprfrtlc.F90:2393, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> realbuffer_gb(1:nvars) = >>>>>>>>>>>>>>> (/time,(c(ic),ic=1,nc-1), & >>>>>>>>>>>>>>> (cx(ix),ix=1,nxout)/) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> All the values (time, c, cx) are reasonable, as shown below. >>>>>>>>>>>>>>> The only >>>>>>>>>>>>>>> possibility is that realbuffer_gb is in declared as real*4 >>>>>>>>>>>>>>> if using sing >>>>>>>>>>>>>>> precision output while time, c, cx are declared in real*8. I >>>>>>>>>>>>>>> have a lot of >>>>>>>>>>>>>>> similar data conversion from real*8 to real*4 output, other >>>>>>>>>>>>>>> code does not >>>>>>>>>>>>>>> return error. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> time 0.0000000000000000 >>>>>>>>>>>>>>> c 1 9.9999999999999995E-008 >>>>>>>>>>>>>>> c 2 3.1555251077549618E-003 >>>>>>>>>>>>>>> c 3 7.1657814842179362E-008 >>>>>>>>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>>>>>>>> c 5 5.2879822292305797E-004 >>>>>>>>>>>>>>> c 6 9.9999999999999964E-005 >>>>>>>>>>>>>>> c 7 6.4055731968811337E-005 >>>>>>>>>>>>>>> c 8 3.4607572892578404E-020 >>>>>>>>>>>>>>> cx 1 3.4376650636008101E-005 >>>>>>>>>>>>>>> cx 2 7.3989678854017763E-012 >>>>>>>>>>>>>>> cx 3 9.5317170613607207E-012 >>>>>>>>>>>>>>> cx 4 2.2344525794718353E-015 >>>>>>>>>>>>>>> cx 5 3.0624685689695889E-008 >>>>>>>>>>>>>>> cx 6 1.0046157902783967E-007 >>>>>>>>>>>>>>> cx 7 1.5320169154914984E-004 >>>>>>>>>>>>>>> cx 8 8.6930292776346176E-014 >>>>>>>>>>>>>>> cx 9 3.5944267559348721E-005 >>>>>>>>>>>>>>> cx 10 3.0072645866951157E-018 >>>>>>>>>>>>>>> cx 11 2.3592486321095017E-013 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I can check all the entries of jacobi matrix to see if the >>>>>>>>>>>>>>> value is valid, >>>>>>>>>>>>>>> but this seems not a good idea as it takes a long time to >>>>>>>>>>>>>>> reach this >>>>>>>>>>>>>>> point. If I restart the simulation from a specified time >>>>>>>>>>>>>>> (e.g., 7.685 in >>>>>>>>>>>>>>> this case), then the error does not occur. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Would you please give me any suggestion on debugging this >>>>>>>>>>>>>>> case? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 >>>>>>>>>>>>>>> years iter: 1 >>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 >>>>>>>>>>>>>>> years iter: 1 >>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 >>>>>>>>>>>>>>> years iter: 1 >>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 >>>>>>>>>>>>>>> years iter: 1 >>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 >>>>>>>>>>>>>>> years iter: 1 >>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>>>>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 >>>>>>>>>>>>>>> years iter: 1 >>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>>>>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 >>>>>>>>>>>>>>> years iter: 1 >>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>>>>> [1]PETSC ERROR: Floating point exception >>>>>>>>>>>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>>>>> [2]PETSC ERROR: Floating point exception >>>>>>>>>>>>>>> [2]PETSC ERROR: Vec entry at local location 0 is >>>>>>>>>>>>>>> not-a-number or infinite >>>>>>>>>>>>>>> at end of function: Parameter number 3 >>>>>>>>>>>>>>> [2]PETSC ERROR: See >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> for trouble shooting. >>>>>>>>>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location >>>>>>>>>>>>>>> 0 is >>>>>>>>>>>>>>> not-a-number or infinite at end of function: Parameter >>>>>>>>>>>>>>> number 3 >>>>>>>>>>>>>>> [1]PETSC ERROR: See >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> for trouble shooting. >>>>>>>>>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>>>>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg >>>>>>>>>>>>>>> named nwmop by >>>>>>>>>>>>>>> dsu Thu Apr 23 15:38:52 2015 >>>>>>>>>>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg >>>>>>>>>>>>>>> --with-cc=gcc >>>>>>>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack >>>>>>>>>>>>>>> --download-mpich >>>>>>>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist >>>>>>>>>>>>>>> --download-metis >>>>>>>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>>>>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>>>>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>>>>>>>>>>>>>> dsu Thu Apr 23 >>>>>>>>>>>>>>> 15:38:52 2015 >>>>>>>>>>>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg >>>>>>>>>>>>>>> --with-cc=gcc >>>>>>>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack >>>>>>>>>>>>>>> --download-mpich >>>>>>>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist >>>>>>>>>>>>>>> --download-metis >>>>>>>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>>>>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>>>>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>>>>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>>>>>>> #3 KSP_PCApply() line 230 in >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>>>>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>>>>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>>>>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>>>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>>>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>>>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>>>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>>>>>>>>>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which >>>>>>>>>>>>>>> their experiments lead. >>>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Apr 29 14:44:00 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 29 Apr 2015 14:44:00 -0500 Subject: [petsc-users] Floating point exception in hypre BoomerAMG In-Reply-To: <55413294.1000906@gmail.com> References: <553A85DF.5000400@gmail.com> <553AA10B.5090208@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> <553C3E15.8030403@gmail.com> <553C3FE6.5020203@gmail.com> <553C4ECD.6090905@gmail.com> <553E6770.5010205@gmail.com> <553FC98D.1000904@gmail.com> <89DAC785-3218-4E2F-8760-2D0E99652014@mcs.anl.gov> <5540261E.2040502@gmail.com> <6411F61F-CD41-4037-B727-B5D9CECE030F@mcs.anl.gov> <554111AD.3060007@gmail.com> <9C61B774-4A78-48E1-B34E-B23EC2D230B9@mcs.anl.gov> <55412888.90206@gmail.com> <0F8EF2B9-9A1C-4C94-B1CC-D520B6D7A0DE@mcs.anl.gov> <55413294.1000906@gmail.com> Message-ID: Ok, run on one process in the debugger and put a break point in VecLockPush and VecLockPop then run and each time it breaks on VecLockPush or VecLockPop type bt then type cont send all the output, this will tell us where the vector got locked readonly and did not get unlocked. Barry > On Apr 29, 2015, at 2:35 PM, Danyang Su wrote: > > On 15-04-29 12:19 PM, Barry Smith wrote: >> Ok, your code seems fine in terms of logic. But the vector x_vec_gbl i.e. b_react should not be in a read only state. Are you somewhere calling a VecGetArray() and not calling the VecRestoreArray()? >> >> Barry > I just made a triple check on VecGetArrayF90(). All the VecGetArrayF90() come with VecRestoreArrayF90(). No VecGetArray() is used in my codes. > > Danyang >> >> >> >>> On Apr 29, 2015, at 1:52 PM, Danyang Su wrote: >>> >>> On 15-04-29 11:30 AM, Barry Smith wrote: >>>>> On Apr 29, 2015, at 12:15 PM, Danyang Su >>>>> wrote: >>>>> >>>>> On 15-04-28 06:50 PM, Barry Smith wrote: >>>>> >>>>>> We started enforcing more checks on writing to vectors that you should not write to. Where are you calling DMLocalToGlobalBegin() ? It looks like you are trying to copy values into a global array that you should not because it is a read only input to a function, for example it is the right hand side of a linear system or the input into a SNESFormFunction(). So check the output to that call. >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>> This is used when local RHS vec is calculated and copied to global RHS. >>>>> >>>> Huhh, why are you copying anything into the RHS? Is this before you call the linear system solve? Send the code that calls the DMLocalToGlobalBegin() >>> Yes. This is called before the linear system solver. At present, we use PETSc KSP solver. >>> The code is first developed in totally sequential version and then ported to the parallel version. So the subdomain has its own space for jacobi matrix and rhs. We do not use global RHS vector directly to set the RHS at this moment. All the RHS values of subdomain are copied to the global RHS. >>> >>> !> This is the function where RHS values of subdomain are copied to the global RHS values. >>> subroutine compute_function(rank,da,x_array_loc,x_vec_loc, & >>> x_vec_gbl,nngl,row_idx_l2pg,col_idx_l2pg, & >>> b_non_interlaced) >>> implicit none >>> #include >>> #include >>> #include >>> #include >>> #include >>> #include >>> #include >>> #include >>> #include >>> >>> PetscInt :: rank >>> DM :: da >>> PetscReal, allocatable :: x_array_loc(:) >>> Vec :: x_vec_loc >>> Vec :: x_vec_gbl >>> PetscInt :: nngl >>> PetscInt, allocatable :: row_idx_l2pg(:) >>> PetscInt, allocatable :: col_idx_l2pg(:) >>> PetscBool :: b_non_interlaced >>> >>> PetscInt :: info_debug >>> PetscInt :: i, j >>> PetscErrorCode :: ierr >>> PetscScalar, pointer :: vecpointer(:) >>> !Zero entries >>> call VecZeroEntries(x_vec_loc, ierr) >>> !Get a pointer to vector data when you need access to the array >>> call VecGetArrayF90(x_vec_loc, vecpointer, ierr) >>> !Compute the function over the locally owned part of the grid >>> if(b_non_interlaced) then >>> j = nngl/2 >>> do i = 1, j >>> vecpointer(2*i-1) = x_array_loc(i) >>> vecpointer(2*i) = x_array_loc(i+j) >>> end do >>> else >>> do i = 1, nngl >>> vecpointer(i) = x_array_loc(i) >>> end do >>> end if >>> !Restore the vector when you no longer need access to the array >>> call VecRestoreArrayF90(x_vec_loc,vecpointer,ierr) >>> !Insert values into global vector >>> call DMLocalToGlobalBegin(da,x_vec_loc,INSERT_VALUES, & >>> x_vec_gbl,ierr) >>> !By placing code between these two statements, computations can be >>> !done while messages are in transition. >>> call DMLocalToGlobalEnd(da,x_vec_loc,INSERT_VALUES, & >>> x_vec_gbl,ierr) >>> return >>> end subroutine >>> >>> >>> >>> !> This is the function where reactive transport equations are solved. >>> subroutine solver_dd_snes_solve_react(ilog,idetail,a_in,b_in, & >>> x_inout,ia_in,ja_in,nngl_in,itsolv, & >>> over_flow,rnorm,row_idx_l2pg,col_idx_l2pg, & >>> b_non_interlaced) >>> use gen, only : rank, node_idx_l2lg, ittot_rt, & >>> b_output_matrix, b_enable_output >>> use solver_snes_function, only : form_initial_guess, & >>> compute_function, & >>> compute_jacobian >>> use petsc_mpi_common, only : petsc_mpi_finalize >>> implicit none >>> #include >>> #include >>> #include >>> #include >>> >>> PetscInt :: ilog >>> PetscInt :: idetail >>> PetscInt :: nngl_in >>> PetscReal, allocatable :: a_in(:) >>> PetscReal, allocatable :: b_in(:) >>> PetscReal, allocatable :: x_inout(:) >>> PetscInt, allocatable :: ia_in(:) >>> PetscInt, allocatable :: ja_in(:) >>> PetscInt, allocatable :: row_idx_l2pg(:) >>> PetscInt, allocatable :: col_idx_l2pg(:) >>> PetscInt :: itsolv >>> PetscBool :: over_flow >>> PetscBool :: b_non_interlaced >>> PetscReal :: rnorm >>> PetscViewer :: viewer >>> PetscScalar,pointer :: vecpointer(:) >>> >>> PetscErrorCode :: ierr >>> PetscInt :: info_debug >>> character(72) :: strinum >>> info_debug = 0 >>> if(b_output_matrix .and. b_enable_output) then >>> write(strinum, *) ittot_rt >>> strinum = "_"//trim(adjustl(strinum)) >>> end if >>> >>> !Form initial guess, only assemble the local owned part, without ghost nodes >>> call form_initial_guess(rank,dmda_react%da,x_inout,x_react_loc,& >>> x_react,nngl_in, row_idx_l2pg,col_idx_l2pg, & >>> b_non_interlaced) >>> >>> !Compute function, only assemble the local part, without ghost nodes >>> call compute_function(rank,dmda_react%da,b_in,b_react_loc, & >>> b_react,nngl_in,row_idx_l2pg,col_idx_l2pg, & >>> b_non_interlaced) >>> !Compute jacobian matrix, only assemble the local part, without ghost nodes >>> call compute_jacobian(rank,dmda_react%da, & >>> a_react,a_in,ia_in,ja_in,nngl_in, & >>> row_idx_l2pg,col_idx_l2pg, & >>> b_non_interlaced) >>> >>> ! Solve a x = b, where a is the Jacobian matrix. >>> ! - First, set the KSP linear operators. Here the matrix that >>> ! defines the linear system also serves as the preconditioning >>> ! matrix. >>> ! - Then solve the Newton system. >>> #ifdef PETSC_V3_5_X >>> call KSPSetOperators(ksp_react,a_react,a_react,ierr) >>> #else >>> call KSPSetOperators(ksp_react,a_react,a_react, & >>> SAME_NONZERO_PATTERN,ierr) >>> #endif >>> call KSPSetDM(ksp_react,dmda_react%da,ierr) >>> call KSPSetDMActive(ksp_react,PETSC_FALSE,ierr) >>> >>> call KSPSolve(ksp_react,b_react,x_react,ierr) >>> !Get residual norm, by default, preconditioned residual norm is calculated. >>> if (b_mykspconverged_react .and. & >>> .not. b_use_petsc_default_react) then >>> rnorm = rnorm_react >>> else >>> call KSPGetResidualNorm(ksp_react, rnorm, ierr) >>> end if >>> ! Scatter ghost points to local vector, using the 2-step process >>> ! DMGlobalToLocalBegin(), DMGlobalToLocalEnd(). >>> ! By placing code between these two statements, computations can be >>> ! done while messages are in transition. >>> call DMGlobalToLocalBegin(dmda_react%da,x_react,INSERT_VALUES, & >>> x_react_loc,ierr) >>> call DMGlobalToLocalEnd(dmda_react%da,x_react,INSERT_VALUES, & >>> x_react_loc,ierr) >>> call VecGetArrayF90(x_react_loc,vecpointer,ierr) >>> x_inout = vecpointer >>> call VecRestoreArrayF90(x_react_loc,vecpointer,ierr) >>> return >>> end subroutine >>> >>>> Barry >>>> >>>> >>>>> It is strange why this error does not occur at the first timestep. >>>>> >>>>> Thanks, >>>>> >>>>> Danyang >>>>> >>>>>>> On Apr 28, 2015, at 7:30 PM, Danyang Su >>>>>>> wrote: >>>>>>> >>>>>>> Hi Barry, >>>>>>> >>>>>>> There seems another bug (not pretty sure) in PETSc-dev, as shown below. The case I used is similar with the one I mentioned recently. I have no problem running this case using PETSc 3.5.2. But it give out the following error using PETSc-dev, even with only one processor. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Danyang >>>>>>> >>>>>>> timestep: 2048 time: 4.615E+00 years delt: 1.460E-20 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>> [0]PETSC ERROR: Object is in wrong state >>>>>>> [0]PETSC ERROR: Vec is locked read only, argument # 1 >>>>>>> [0]PETSC ERROR: See >>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>> for trouble shooting. >>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2796-g23b6c49 GIT Date: 2015-04-27 11:32:44 -0500 >>>>>>> [0]PETSC ERROR: ../min3p_thcm on a linux-gnu-dbg named nwmop by dsu Tue Apr 28 15:56:41 2015 >>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack >>>>>>> [0]PETSC ERROR: #349 VecGetArray() line 1646 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/interface/rvector.c >>>>>>> [0]PETSC ERROR: #350 VecGetArrayPair() line 434 in /home/dsu/Soft/PETSc/petsc-dev/include/petscvec.h >>>>>>> [0]PETSC ERROR: #351 VecScatterBegin_SSToSS() line 649 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c >>>>>>> [0]PETSC ERROR: #352 VecScatterBegin() line 1694 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c >>>>>>> [0]PETSC ERROR: #353 DMLocalToGlobalBegin_DA() line 56 in /home/dsu/Soft/PETSc/petsc-dev/src/dm/impls/da/dagtol.c >>>>>>> [0]PETSC ERROR: #354 DMLocalToGlobalBegin() line 1944 in /home/dsu/Soft/PETSc/petsc-dev/src/dm/interface/dm.c >>>>>>> Reduce time step for reactive transport >>>>>>> no further time step reduction possible >>>>>>> Optimal time increment computed by MIN3P 2.6644769214899893E-018 >>>>>>> Minimum time increment specified by user 3.6499999999999998E-018 >>>>>>> Please, reduce the MINIMUM TIME INCREMENT >>>>>>> stop signal in time step reduction failed >>>>>>> >>>>>>> On 15-04-28 11:15 AM, Barry Smith wrote: >>>>>>> >>>>>>>> I am forwarding this on to the hypre support email list; hopefully they can have some advice on how to proceed. >>>>>>>> >>>>>>>> hypre folks, this is with version hypre-2.10.0b (the previous version had the same behavior). We are getting a divide by zero in gselim() line 4363 (this happens in a time dependent problem after many successful solves with time dependent matrix). I looked at the code, below, and note that there are some checks for the diagonal of A not being zero but not for the line that causes the divide by zero; is this perhaps an oversight in the hypre code? Any advice appreciated. >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> HYPRE_Int gselim(A,x,n) >>>>>>>> HYPRE_Real *A; >>>>>>>> HYPRE_Real *x; >>>>>>>> HYPRE_Int n; >>>>>>>> { >>>>>>>> HYPRE_Int err_flag = 0; >>>>>>>> HYPRE_Int j,k,m; >>>>>>>> HYPRE_Real factor; >>>>>>>> if (n==1) /* A is 1x1 */ >>>>>>>> { >>>>>>>> if (A[0] != 0.0) >>>>>>>> { >>>>>>>> x[0] = x[0]/A[0]; >>>>>>>> return(err_flag); >>>>>>>> } >>>>>>>> else >>>>>>>> { >>>>>>>> err_flag = 1; >>>>>>>> return(err_flag); >>>>>>>> } >>>>>>>> } >>>>>>>> else /* A is nxn. Forward elimination */ >>>>>>>> { >>>>>>>> for (k = 0; k < n-1; k++) >>>>>>>> { >>>>>>>> if (A[k*n+k] != 0.0) >>>>>>>> { >>>>>>>> for (j = k+1; j < n; j++) >>>>>>>> { >>>>>>>> if (A[j*n+k] != 0.0) >>>>>>>> { >>>>>>>> factor = A[j*n+k]/A[k*n+k]; >>>>>>>> for (m = k+1; m < n; m++) >>>>>>>> { >>>>>>>> A[j*n+m] -= factor * A[k*n+m]; >>>>>>>> } >>>>>>>> /* Elimination step for rhs */ >>>>>>>> x[j] -= factor * x[k]; >>>>>>>> } >>>>>>>> } >>>>>>>> } >>>>>>>> } >>>>>>>> /* Back Substitution */ >>>>>>>> for (k = n-1; k > 0; --k) >>>>>>>> { >>>>>>>> x[k] /= A[k*n+k]; >>>>>>>> for (j = 0; j < k; j++) >>>>>>>> { >>>>>>>> if (A[j*n+k] != 0.0) >>>>>>>> { >>>>>>>> x[j] -= x[k] * A[j*n+k]; >>>>>>>> } >>>>>>>> } >>>>>>>> } >>>>>>>> x[0] /= A[0]; >>>>>>>> return(err_flag); >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Apr 28, 2015, at 12:55 PM, Danyang Su >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi Barry, >>>>>>>>> >>>>>>>>> The development version of PETSc does not help to solve my problem. It still crashed due to the same error information. >>>>>>>>> >>>>>>>>> As Matthew mentioned, I checked the matrix and non of the diagonal entry is zero. But the diagonal values for different rows range from -1.0d-18 to 1.0d33 and the matrix is not strict diagonal dominant. When using 1 processor, the matrix is also not strict diagonal dominant, but the diagonal values have a much smaller range and the codes can run without error. Maybe there is something wrong in the matrix for this case and I need to check the matrix first. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Danyang >>>>>>>>> >>>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>>> 0x00007f668c1299d5 in gselim (A=0x2fe3330, x=0x2210680, n=9) >>>>>>>>> at par_relax.c:4363 >>>>>>>>> 4363 x[k] /= A[k*n+k]; >>>>>>>>> (gdb) >>>>>>>>> >>>>>>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>> [1]PETSC ERROR: is given. >>>>>>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-dev/src/sys/error/fp.c >>>>>>>>> [1]PETSC ERROR: [1] Hypre solve line 216 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >>>>>>>>> [1]PETSC ERROR: [1] PCApply_HYPRE line 203 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >>>>>>>>> [1]PETSC ERROR: [1] PCApply line 430 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/interface/precon.c >>>>>>>>> [1]PETSC ERROR: [1] KSP_PCApply line 234 /home/dsu/Soft/PETSc/petsc-dev/include/petsc/private/kspimpl.h >>>>>>>>> [1]PETSC ERROR: [1] KSPInitialResidual line 44 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itres.c >>>>>>>>> [1]PETSC ERROR: [1] KSPSolve_GMRES line 224 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>> [1]PETSC ERROR: [1] KSPSolve line 506 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>>>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 15-04-27 11:47 AM, Barry Smith wrote: >>>>>>>>> >>>>>>>>>> Danyang, >>>>>>>>>> >>>>>>>>>> I'm glad you were finally able to get to the problematic point; sorry it took so long (I want to strangle whoever turned on catching of underflows in PETSc). >>>>>>>>>> >>>>>>>>>> The hypre folks have done a great deal of work on their relaxation code since the PETSc 3.5.3 release. There is a good chance they have fixed the divide by zero problem you are hitting here. You will need to upgrade to the development version of PETSc (that uses the latest version of hypre), here are the instructions on how to obtain it >>>>>>>>>> http://www.mcs.anl.gov/petsc/developers/index.html >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Please let us know if this resolves the problem with hypre failing. >>>>>>>>>> >>>>>>>>>> Barry >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Apr 27, 2015, at 11:44 AM, Danyang Su >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Barry, >>>>>>>>>>> >>>>>>>>>>> I got the following arithemetic exception after the previous bug is fixed. >>>>>>>>>>> >>>>>>>>>>> Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 >>>>>>>>>>> 0x00007f3b23b98f20 in __nanosleep_nocancel () >>>>>>>>>>> at ../sysdeps/unix/syscall-template.S:81 >>>>>>>>>>> 81 ../sysdeps/unix/syscall-template.S: No such file or directory. >>>>>>>>>>> (gdb) cont >>>>>>>>>>> Continuing. >>>>>>>>>>> >>>>>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>>>>> 0x00007f3b260df449 in gselim (A=0x1e4c580, x=0x1d30c40, n=9) >>>>>>>>>>> at par_relax.c:3442 >>>>>>>>>>> 3442 x[k] /= A[k*n+k]; >>>>>>>>>>> (gdb) >>>>>>>>>>> >>>>>>>>>>> I tried both PETSc 3.5.2 and 3.5.3 and they return the same error as shown above. For 3.5.3, i edited fp.c file and then configure and make. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Danyang >>>>>>>>>>> >>>>>>>>>>> On 15-04-25 07:34 PM, Danyang Su wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi All, >>>>>>>>>>>> >>>>>>>>>>>> The "floating point underflow" is caused by a small value divided by a very large value. This result is forced to zero and then it does not report any underflow problem. I just rerun this bad case to see if it still get stuck later. This will take a while. >>>>>>>>>>>> >>>>>>>>>>>> Thanks for all your kindly reply, >>>>>>>>>>>> >>>>>>>>>>>> Danyang >>>>>>>>>>>> >>>>>>>>>>>> On 15-04-25 07:02 PM, Barry Smith wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Ok, you do have >>>>>>>>>>>>> >>>>>>>>>>>>> #ifndef PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>>>> #define PETSC_HAVE_XMMINTRIN_H 1 >>>>>>>>>>>>> #endif >>>>>>>>>>>>> >>>>>>>>>>>>> so the change you made should cause it to stop trapping underflow exceptions. >>>>>>>>>>>>> >>>>>>>>>>>>> Now in one email you reported a FPE within hypre, then I asked you to run with -start_in_debugger to determine where it happened exactly and then you reported the FPE happened in user code (what seemed to be an underflow issue). Why is this? Can you not run it where it generated the FPE in hypre using the -start_in_debugger option? >>>>>>>>>>>>> >>>>>>>>>>>>> Barry >>>>>>>>>>>>> >>>>>>>>>>>>> Perhaps you have multiple PETSC_ARCH or multiple PETSc installs to explain why you reported two different places where the exception occurred. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> On Apr 25, 2015, at 8:31 PM, Danyang Su >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 15-04-25 06:26 PM, Matthew Knepley wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 15-04-25 06:03 PM, Barry Smith wrote: >>>>>>>>>>>>>>> If this is what you got in your last run >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>>>>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>>>>>>>>>> then it is still catching floating point underflow, which we do not want. This means either the change I suggested you make in the fp.c code didn't work or it actually uses a different floating point trap than that one. BTW: absurd numbers like 1.0d-316 are often a symptom of uninitialized data; could that be a problem that diff is not filled correctly for all the ic, idim you are using? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This going round and round is very frustrating and a waste of time. You need to be more proactive yourself and explore the code and poke around to figure out how to solve the problem. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can see what FP trap is being used on your machine. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Barry >>>>>>>>>>>>>>> Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise I cannot find this file. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Its include/petscconf.h >>>>>>>>>>>>>>> Do I need to reconfigure PETSc after changing the code you mentioned? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> No, but you need to rebuild. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> Yes, I have done 'gnumake'. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Matt >>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Apr 25, 2015, at 2:24 PM, Danyang Su >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 15-04-25 11:55 AM, Barry Smith wrote: >>>>>>>>>>>>>>> On Apr 25, 2015, at 1:51 PM, Danyang Su >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 15-04-25 11:32 AM, Barry Smith wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I told you this yesterday. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>>>>>>>>>>>>>> #else >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> change it to >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >>>>>>>>>>>>>>> #else >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Did you do this? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Barry >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc directory. I just did a double check and ran make gnumake again and got the following information this time. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> make gnumake >>>>>>>>>>>>>>> Building PETSc using GNU Make with 10 build threads >>>>>>>>>>>>>>> ========================================== >>>>>>>>>>>>>>> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>>>>>>>>> make[1]: Nothing to be done for `all'. >>>>>>>>>>>>>>> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>>>>>>>>> ========================================= >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Then I recompiled the codes, ran with -fp_trap and still got the following error >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>>>>> Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>>>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>>>>>> [2]PETSC ERROR: is given. >>>>>>>>>>>>>>> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>>>> INSTEAD the line number of the start of the function >>>>>>>>>>>>>>> [3]PETSC ERROR: is given. >>>>>>>>>>>>>>> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>>>> [2]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>>>>>> [3]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is different then what you sent a few minutes ago where it crashed in hypre. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Anyways you need to use the -start_in_debugger business I sent in the previous email to see the exact place the problem occurs. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Here is the information shown on gdb screen >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>>>>>>>>> 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, njamxc=34, >>>>>>>>>>>>>>> cinfradx=..., radial_coordx=.FALSE., _suffix=3) >>>>>>>>>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>>>>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>>>>>>>>>> (gdb) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> After type cont on gdb screen, I got error information as below >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>>>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>>>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>>>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>>>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>>>>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>>>>>> [1]PETSC ERROR: is given. >>>>>>>>>>>>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Apr 25, 2015, at 1:05 AM, Danyang Su >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Barry and Satish, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> How can I get rid of unknown floating point error when a very small value is multiplied. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> e.g., >>>>>>>>>>>>>>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I get the following error when run with "-fp_trap -start_in_debugger". >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>>>>> *** unknown floating point error occurred *** >>>>>>>>>>>>>>> [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>>>>>> [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>>>>>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 >>>>>>>>>>>>>>> where the result is a bitwise OR of the following flags: >>>>>>>>>>>>>>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>>>> [2]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I don't think this number can be stored in a real*4. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Satish >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, Satish. It is caused by this number. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> One of my case crashes because of floating point exception when using 4 >>>>>>>>>>>>>>> processors, as shown below. But if I run this case with 1 processor, it >>>>>>>>>>>>>>> works fine. I have tested the codes with around 100 cases up to 768 >>>>>>>>>>>>>>> processors, all other cases work fine. I just wonder if this kind of error >>>>>>>>>>>>>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yes, almost for sure it is one of these places. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> First run the bad case with -fp_trap if all goes well you'll see the >>>>>>>>>>>>>>> function where the FPE is generated. Then run also with -start_in_debugger >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>> type cont in all four debugger windows. When the FPE happens the debugger >>>>>>>>>>>>>>> should stop showing exactly where the FPE happens. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Barry >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Barry, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If run with -fp_trap -start_in_debugger, I got the following error >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a >>>>>>>>>>>>>>> debugger. When the >>>>>>>>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >>>>>>>>>>>>>>> fetestexcept(0x3d) >>>>>>>>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>>>>>>>>>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>>>>>>>>>>> ------------------------------------ >>>>>>>>>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>>>>>> [0]PETSC ERROR: is given. >>>>>>>>>>>>>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>>>>>>>>>>>>>> floating point error >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Program received signal SIGABRT: Process abort signal. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>>>>> #0 0x7F4FEAB1C7D7 >>>>>>>>>>>>>>> #1 0x7F4FEAB1CDDE >>>>>>>>>>>>>>> #2 0x7F4FE9E1AD3F >>>>>>>>>>>>>>> #3 0x7F4FE9E1ACC9 >>>>>>>>>>>>>>> #4 0x7F4FE9E1E0D7 >>>>>>>>>>>>>>> #5 0x7F4FEB0B6DCB >>>>>>>>>>>>>>> #6 0x7F4FEB0B1825 >>>>>>>>>>>>>>> #7 0x7F4FEB0B817F >>>>>>>>>>>>>>> #8 0x7F4FE9E1AD3F >>>>>>>>>>>>>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>>>>>>>>>>>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>>>>>>>>>>>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>>>>>>>>>>>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>>>>>>>>>>>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I checked the code at tprfrtlc.F90:2393, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>>>>>>>>>>>>> (cx(ix),ix=1,nxout)/) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> All the values (time, c, cx) are reasonable, as shown below. The only >>>>>>>>>>>>>>> possibility is that realbuffer_gb is in declared as real*4 if using sing >>>>>>>>>>>>>>> precision output while time, c, cx are declared in real*8. I have a lot of >>>>>>>>>>>>>>> similar data conversion from real*8 to real*4 output, other code does not >>>>>>>>>>>>>>> return error. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> time 0.0000000000000000 >>>>>>>>>>>>>>> c 1 9.9999999999999995E-008 >>>>>>>>>>>>>>> c 2 3.1555251077549618E-003 >>>>>>>>>>>>>>> c 3 7.1657814842179362E-008 >>>>>>>>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>>>>>>>> c 5 5.2879822292305797E-004 >>>>>>>>>>>>>>> c 6 9.9999999999999964E-005 >>>>>>>>>>>>>>> c 7 6.4055731968811337E-005 >>>>>>>>>>>>>>> c 8 3.4607572892578404E-020 >>>>>>>>>>>>>>> cx 1 3.4376650636008101E-005 >>>>>>>>>>>>>>> cx 2 7.3989678854017763E-012 >>>>>>>>>>>>>>> cx 3 9.5317170613607207E-012 >>>>>>>>>>>>>>> cx 4 2.2344525794718353E-015 >>>>>>>>>>>>>>> cx 5 3.0624685689695889E-008 >>>>>>>>>>>>>>> cx 6 1.0046157902783967E-007 >>>>>>>>>>>>>>> cx 7 1.5320169154914984E-004 >>>>>>>>>>>>>>> cx 8 8.6930292776346176E-014 >>>>>>>>>>>>>>> cx 9 3.5944267559348721E-005 >>>>>>>>>>>>>>> cx 10 3.0072645866951157E-018 >>>>>>>>>>>>>>> cx 11 2.3592486321095017E-013 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I can check all the entries of jacobi matrix to see if the value is valid, >>>>>>>>>>>>>>> but this seems not a good idea as it takes a long time to reach this >>>>>>>>>>>>>>> point. If I restart the simulation from a specified time (e.g., 7.685 in >>>>>>>>>>>>>>> this case), then the error does not occur. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Would you please give me any suggestion on debugging this case? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>>>>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>>>>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>>>>> [1]PETSC ERROR: Floating point exception >>>>>>>>>>>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>>>>> [2]PETSC ERROR: Floating point exception >>>>>>>>>>>>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >>>>>>>>>>>>>>> at end of function: Parameter number 3 >>>>>>>>>>>>>>> [2]PETSC ERROR: See >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> for trouble shooting. >>>>>>>>>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>>>>>>>>>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>>>>>>>>>>>> [1]PETSC ERROR: See >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> for trouble shooting. >>>>>>>>>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>>>>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>>>>>>>>>>>>>> dsu Thu Apr 23 15:38:52 2015 >>>>>>>>>>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>>>>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>>>>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >>>>>>>>>>>>>>> 15:38:52 2015 >>>>>>>>>>>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>>>>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>>>>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>>>>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>>>>>>> #3 KSP_PCApply() line 230 in >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>>>>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>>>>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>>>>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>>>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>>>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>>>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>>>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>>>>>>>>>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> > From salazardetroya at gmail.com Wed Apr 29 14:48:27 2015 From: salazardetroya at gmail.com (Miguel Angel Salazar de Troya) Date: Wed, 29 Apr 2015 14:48:27 -0500 Subject: [petsc-users] Discrete adjoint and adaptive time stepping In-Reply-To: <30260DDE-B439-4498-93C9-87ADBD9CE19F@mcs.anl.gov> References: <8A885376-C369-4E9E-AAC0-140930EA3CED@mcs.anl.gov> <871tj2py2i.fsf@jedbrown.org> <30260DDE-B439-4498-93C9-87ADBD9CE19F@mcs.anl.gov> Message-ID: If we calculate the gradient using the discrete adjoint without differentiating the controller, and then calculate the same gradient using finite difference (allowing the time steps to freely change), how different these results are? Miguel On Wed, Apr 29, 2015 at 2:12 PM, Barry Smith wrote: > > > On Apr 29, 2015, at 1:51 PM, Jed Brown wrote: > > > > Barry Smith writes: > >>> If so, how is it done? > >> > >> We just keep the history of the time-step sizes and then use those > time-step sizes when doing the backward integration. Seems simple to me, am > I missing something? > > > > Barry, if you're using this for optimization, you might want the > > gradient to be exactly consistent with the objective functional. But > > for that, you would need to differentiate the controller, which is > > non-smooth in practice because the number of time steps can change and > > stages could be rejected (solver failure). > > Ahh, yes for multiple forward runs yup. > > > > > One approach would be to save the timestep sequence and have the > > controller use that in subsequent *forward* runs. If the dynamical > > system behaves similarly for those steps, it would be okay to use the > > same timestep sequence. > > Presumably if that single set of dt (from the first run) is not > sufficient for some later runs one could possibly use the union of the dt > of several runs for all the runs. (that is run adaptively and > inconsistently several runs to determine where dt needs to be controlled > and then use the various smaller of the dt at the different time regions > for a full set of consistent runs). Of course if the various smaller of the > dt requires a tiny dt for all time steps then you are not getting an > advantage of adaptive time-stepping, but ok. > > The idea of actually propagating the gradients through the time-step > controller seems IMHO to be absurd; I won't even put it on our game plan > until we have many more things done and much more practical experience. > > Barry > > > -- *Miguel Angel Salazar de Troya* Graduate Research Assistant Department of Mechanical Science and Engineering University of Illinois at Urbana-Champaign (217) 550-2360 salaza11 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From mlohry at princeton.edu Wed Apr 29 14:49:01 2015 From: mlohry at princeton.edu (Mark Lohry) Date: Wed, 29 Apr 2015 15:49:01 -0400 Subject: [petsc-users] Turning off TSADAPT still adapts time step In-Reply-To: <87zj5tse8k.fsf@jedbrown.org> References: <553A85DF.5000400@gmail.com> <4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov> <553AA10B.5090208@gmail.com> <553AA4B1.5010603@princeton.edu> <553AB0C6.5010503@princeton.edu> <553E5AD5.1060208@princeton.edu> <87383ltvf8.fsf@jedbrown.org> <553E5FFF.6080905@princeton.edu> <87zj5tse8k.fsf@jedbrown.org> Message-ID: <554135AD.50307@princeton.edu> To continue beating this horse, I wrote this: PetscErrorCode CheckStageTrue(TSAdapt adapt,TS ts,PetscBool *accept){ *accept = PETSC_TRUE; } TSAdaptSetCheckStage(adapt,CheckStageTrue); But TSAdapt then still rejects the stage, but quits altogether instead of continuing: Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 4 TSAdapt 'basic': step 0 stage rejected t=0 + 1.000e-01 retrying with dt=2.500e-02 Am I not using TSAdaptSetCheckStage correctly? On 04/27/2015 12:55 PM, Jed Brown wrote: > Mark Lohry writes: > >> TSAdaptChoose_None is being called, yes. Output with -ts_adapt monitor >> looks like this: >> >> TSAdapt 'none': step 17 accepted t=680 + 4.000e+01 >> family='beuler' scheme=0:'(null)' dt=4.000e+01 >> TSAdapt 'none': step 18 accepted t=720 + 4.000e+01 >> family='beuler' scheme=0:'(null)' dt=4.000e+01 >> TSAdapt 'none': step 19 accepted t=760 + 4.000e+01 >> family='beuler' scheme=0:'(null)' dt=4.000e+01 >> TSAdapt 'none': step 20 stage rejected t=800 + 4.000e+01 >> retrying with dt=1.000e+01 > Add -snes_converged_reason. What do you want to do when the solver > fails? Pretend like it succeeded and get the wrong answer? Usually > people shorten the time step and retry, which is what you see happening > here. > > You can call TSAdaptSetCheckStage and have your function unconditionally > set accept to PETSC_TRUE (and you'll likely get the wrong answer). From bsmith at mcs.anl.gov Wed Apr 29 15:00:45 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 29 Apr 2015 15:00:45 -0500 Subject: [petsc-users] Discrete adjoint and adaptive time stepping In-Reply-To: References: <8A885376-C369-4E9E-AAC0-140930EA3CED@mcs.anl.gov> <871tj2py2i.fsf@jedbrown.org> <30260DDE-B439-4498-93C9-87ADBD9CE19F@mcs.anl.gov> Message-ID: <18314E13-BFA7-4EFB-A002-3F980E7BF150@mcs.anl.gov> > On Apr 29, 2015, at 2:48 PM, Miguel Angel Salazar de Troya wrote: > > If we calculate the gradient using the discrete adjoint without differentiating the controller, and then calculate the same gradient using finite difference (allowing the time steps to freely change), how different these results are? Clearly I don't know but I image they could be very different. I would do as Jed suggests and use one run to get the adaptive dt and then use that same collect of dt for all the runs. Barry > > Miguel > > On Wed, Apr 29, 2015 at 2:12 PM, Barry Smith wrote: > > > On Apr 29, 2015, at 1:51 PM, Jed Brown wrote: > > > > Barry Smith writes: > >>> If so, how is it done? > >> > >> We just keep the history of the time-step sizes and then use those time-step sizes when doing the backward integration. Seems simple to me, am I missing something? > > > > Barry, if you're using this for optimization, you might want the > > gradient to be exactly consistent with the objective functional. But > > for that, you would need to differentiate the controller, which is > > non-smooth in practice because the number of time steps can change and > > stages could be rejected (solver failure). > > Ahh, yes for multiple forward runs yup. > > > > > One approach would be to save the timestep sequence and have the > > controller use that in subsequent *forward* runs. If the dynamical > > system behaves similarly for those steps, it would be okay to use the > > same timestep sequence. > > Presumably if that single set of dt (from the first run) is not sufficient for some later runs one could possibly use the union of the dt of several runs for all the runs. (that is run adaptively and inconsistently several runs to determine where dt needs to be controlled and then use the various smaller of the dt at the different time regions for a full set of consistent runs). Of course if the various smaller of the dt requires a tiny dt for all time steps then you are not getting an advantage of adaptive time-stepping, but ok. > > The idea of actually propagating the gradients through the time-step controller seems IMHO to be absurd; I won't even put it on our game plan until we have many more things done and much more practical experience. > > Barry > > > > > > -- > Miguel Angel Salazar de Troya > Graduate Research Assistant > Department of Mechanical Science and Engineering > University of Illinois at Urbana-Champaign > (217) 550-2360 > salaza11 at illinois.edu > From danyang.su at gmail.com Wed Apr 29 17:19:59 2015 From: danyang.su at gmail.com (Danyang Su) Date: Wed, 29 Apr 2015 15:19:59 -0700 Subject: [petsc-users] Floating point exception in hypre BoomerAMG In-Reply-To: References: <553A85DF.5000400@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> <553C3E15.8030403@gmail.com> <553C3FE6.5020203@gmail.com> <553C4ECD.6090905@gmail.com> <553E6770.5010205@gmail.com> <553FC98D.1000904@gmail.com> <89DAC785-3218-4E2F-8760-2D0E99652014@mcs.anl.gov> <5540261E.2040502@gmail.com> <6411F61F-CD41-4037-B727-B5D9CECE030F@mcs.anl.gov> <554111AD.3060007@gmail.com> <9C61B774-4A78-48E1-B34E-B23EC2D230B9@mcs.anl.gov> <55412888.90206@gmail.com> <0F8EF2B9-9A1C-4C94-B1CC-D520B6D7A0DE@mcs.anl.gov> <55413294.1000906@gmail.com> Message-ID: <5541590F.6070806@gmail.com> Hi Barry, I haven't tried your suggestion but it seems the bug from HYPRE. Here are my tests: Test A: If I run with start_in_debugger, I got PETSC ERROR: PCApply_HYPRE(), as shown below. If I run without start_in_debugger, then I got Vec lock problem as I mentioned before, exactly at the same timestep. timestep: 2048 time: 4.619E+00 years delt: 4.208E-03 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 Newton Iteration Convergence Summary: Newton maximum for maximum solver iteration update component residual iterations maxvol nexvol 1 3.0000D+00 o2(aq) 2.4957D-03 1 33 39 2 3.0000D+00 o2(aq) 6.4858D-03 1 33 39 3 3.0000D+00 o2(aq) 5.7149D-03 1 33 39 4 3.0000D+00 o2(aq) 4.9772D-03 1 33 39 5 -3.0000D+00 co3-2 5.9683D-06 180 33 39 6 3.0000D+00 h+1 2.5193D+04 1 1 39 7 3.0000D+00 co3-2 2.4783D+16 3 1 39 8 3.0000D+00 h+1 3.6929D+29 14 1 39 [0]PETSC ERROR: PCApply_HYPRE() line 174 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/impls/hypre/hypre.c Error in HYPRE solver, error code 1 Test B: I changed the preconditioner to bjacobi then all the errors are gone, whatever with or without start_in_debugger option. Usually we use HYPRE as preconditioner method as it is generally faster than other methods for our cases. Do I still need to test on this case to locate the error or just wait until the bug of HYPRE is fixed. Thanks, Danyang On 15-04-29 12:44 PM, Barry Smith wrote: > Ok, run on one process in the debugger and put a break point in VecLockPush and VecLockPop then run and each time it breaks on VecLockPush or VecLockPop type > > bt > > then type > > cont > > send all the output, this will tell us where the vector got locked readonly and did not get unlocked. > > Barry > > > >> On Apr 29, 2015, at 2:35 PM, Danyang Su wrote: >> >> On 15-04-29 12:19 PM, Barry Smith wrote: >>> Ok, your code seems fine in terms of logic. But the vector x_vec_gbl i.e. b_react should not be in a read only state. Are you somewhere calling a VecGetArray() and not calling the VecRestoreArray()? >>> >>> Barry >> I just made a triple check on VecGetArrayF90(). All the VecGetArrayF90() come with VecRestoreArrayF90(). No VecGetArray() is used in my codes. >> >> Danyang >>> >>> >>>> On Apr 29, 2015, at 1:52 PM, Danyang Su wrote: >>>> >>>> On 15-04-29 11:30 AM, Barry Smith wrote: >>>>>> On Apr 29, 2015, at 12:15 PM, Danyang Su >>>>>> wrote: >>>>>> >>>>>> On 15-04-28 06:50 PM, Barry Smith wrote: >>>>>> >>>>>>> We started enforcing more checks on writing to vectors that you should not write to. Where are you calling DMLocalToGlobalBegin() ? It looks like you are trying to copy values into a global array that you should not because it is a read only input to a function, for example it is the right hand side of a linear system or the input into a SNESFormFunction(). So check the output to that call. >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> >>>>>> This is used when local RHS vec is calculated and copied to global RHS. >>>>>> >>>>> Huhh, why are you copying anything into the RHS? Is this before you call the linear system solve? Send the code that calls the DMLocalToGlobalBegin() >>>> Yes. This is called before the linear system solver. At present, we use PETSc KSP solver. >>>> The code is first developed in totally sequential version and then ported to the parallel version. So the subdomain has its own space for jacobi matrix and rhs. We do not use global RHS vector directly to set the RHS at this moment. All the RHS values of subdomain are copied to the global RHS. >>>> >>>> !> This is the function where RHS values of subdomain are copied to the global RHS values. >>>> subroutine compute_function(rank,da,x_array_loc,x_vec_loc, & >>>> x_vec_gbl,nngl,row_idx_l2pg,col_idx_l2pg, & >>>> b_non_interlaced) >>>> implicit none >>>> #include >>>> #include >>>> #include >>>> #include >>>> #include >>>> #include >>>> #include >>>> #include >>>> #include >>>> >>>> PetscInt :: rank >>>> DM :: da >>>> PetscReal, allocatable :: x_array_loc(:) >>>> Vec :: x_vec_loc >>>> Vec :: x_vec_gbl >>>> PetscInt :: nngl >>>> PetscInt, allocatable :: row_idx_l2pg(:) >>>> PetscInt, allocatable :: col_idx_l2pg(:) >>>> PetscBool :: b_non_interlaced >>>> >>>> PetscInt :: info_debug >>>> PetscInt :: i, j >>>> PetscErrorCode :: ierr >>>> PetscScalar, pointer :: vecpointer(:) >>>> !Zero entries >>>> call VecZeroEntries(x_vec_loc, ierr) >>>> !Get a pointer to vector data when you need access to the array >>>> call VecGetArrayF90(x_vec_loc, vecpointer, ierr) >>>> !Compute the function over the locally owned part of the grid >>>> if(b_non_interlaced) then >>>> j = nngl/2 >>>> do i = 1, j >>>> vecpointer(2*i-1) = x_array_loc(i) >>>> vecpointer(2*i) = x_array_loc(i+j) >>>> end do >>>> else >>>> do i = 1, nngl >>>> vecpointer(i) = x_array_loc(i) >>>> end do >>>> end if >>>> !Restore the vector when you no longer need access to the array >>>> call VecRestoreArrayF90(x_vec_loc,vecpointer,ierr) >>>> !Insert values into global vector >>>> call DMLocalToGlobalBegin(da,x_vec_loc,INSERT_VALUES, & >>>> x_vec_gbl,ierr) >>>> !By placing code between these two statements, computations can be >>>> !done while messages are in transition. >>>> call DMLocalToGlobalEnd(da,x_vec_loc,INSERT_VALUES, & >>>> x_vec_gbl,ierr) >>>> return >>>> end subroutine >>>> >>>> >>>> >>>> !> This is the function where reactive transport equations are solved. >>>> subroutine solver_dd_snes_solve_react(ilog,idetail,a_in,b_in, & >>>> x_inout,ia_in,ja_in,nngl_in,itsolv, & >>>> over_flow,rnorm,row_idx_l2pg,col_idx_l2pg, & >>>> b_non_interlaced) >>>> use gen, only : rank, node_idx_l2lg, ittot_rt, & >>>> b_output_matrix, b_enable_output >>>> use solver_snes_function, only : form_initial_guess, & >>>> compute_function, & >>>> compute_jacobian >>>> use petsc_mpi_common, only : petsc_mpi_finalize >>>> implicit none >>>> #include >>>> #include >>>> #include >>>> #include >>>> >>>> PetscInt :: ilog >>>> PetscInt :: idetail >>>> PetscInt :: nngl_in >>>> PetscReal, allocatable :: a_in(:) >>>> PetscReal, allocatable :: b_in(:) >>>> PetscReal, allocatable :: x_inout(:) >>>> PetscInt, allocatable :: ia_in(:) >>>> PetscInt, allocatable :: ja_in(:) >>>> PetscInt, allocatable :: row_idx_l2pg(:) >>>> PetscInt, allocatable :: col_idx_l2pg(:) >>>> PetscInt :: itsolv >>>> PetscBool :: over_flow >>>> PetscBool :: b_non_interlaced >>>> PetscReal :: rnorm >>>> PetscViewer :: viewer >>>> PetscScalar,pointer :: vecpointer(:) >>>> >>>> PetscErrorCode :: ierr >>>> PetscInt :: info_debug >>>> character(72) :: strinum >>>> info_debug = 0 >>>> if(b_output_matrix .and. b_enable_output) then >>>> write(strinum, *) ittot_rt >>>> strinum = "_"//trim(adjustl(strinum)) >>>> end if >>>> >>>> !Form initial guess, only assemble the local owned part, without ghost nodes >>>> call form_initial_guess(rank,dmda_react%da,x_inout,x_react_loc,& >>>> x_react,nngl_in, row_idx_l2pg,col_idx_l2pg, & >>>> b_non_interlaced) >>>> >>>> !Compute function, only assemble the local part, without ghost nodes >>>> call compute_function(rank,dmda_react%da,b_in,b_react_loc, & >>>> b_react,nngl_in,row_idx_l2pg,col_idx_l2pg, & >>>> b_non_interlaced) >>>> !Compute jacobian matrix, only assemble the local part, without ghost nodes >>>> call compute_jacobian(rank,dmda_react%da, & >>>> a_react,a_in,ia_in,ja_in,nngl_in, & >>>> row_idx_l2pg,col_idx_l2pg, & >>>> b_non_interlaced) >>>> >>>> ! Solve a x = b, where a is the Jacobian matrix. >>>> ! - First, set the KSP linear operators. Here the matrix that >>>> ! defines the linear system also serves as the preconditioning >>>> ! matrix. >>>> ! - Then solve the Newton system. >>>> #ifdef PETSC_V3_5_X >>>> call KSPSetOperators(ksp_react,a_react,a_react,ierr) >>>> #else >>>> call KSPSetOperators(ksp_react,a_react,a_react, & >>>> SAME_NONZERO_PATTERN,ierr) >>>> #endif >>>> call KSPSetDM(ksp_react,dmda_react%da,ierr) >>>> call KSPSetDMActive(ksp_react,PETSC_FALSE,ierr) >>>> >>>> call KSPSolve(ksp_react,b_react,x_react,ierr) >>>> !Get residual norm, by default, preconditioned residual norm is calculated. >>>> if (b_mykspconverged_react .and. & >>>> .not. b_use_petsc_default_react) then >>>> rnorm = rnorm_react >>>> else >>>> call KSPGetResidualNorm(ksp_react, rnorm, ierr) >>>> end if >>>> ! Scatter ghost points to local vector, using the 2-step process >>>> ! DMGlobalToLocalBegin(), DMGlobalToLocalEnd(). >>>> ! By placing code between these two statements, computations can be >>>> ! done while messages are in transition. >>>> call DMGlobalToLocalBegin(dmda_react%da,x_react,INSERT_VALUES, & >>>> x_react_loc,ierr) >>>> call DMGlobalToLocalEnd(dmda_react%da,x_react,INSERT_VALUES, & >>>> x_react_loc,ierr) >>>> call VecGetArrayF90(x_react_loc,vecpointer,ierr) >>>> x_inout = vecpointer >>>> call VecRestoreArrayF90(x_react_loc,vecpointer,ierr) >>>> return >>>> end subroutine >>>> >>>>> Barry >>>>> >>>>> >>>>>> It is strange why this error does not occur at the first timestep. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Danyang >>>>>> >>>>>>>> On Apr 28, 2015, at 7:30 PM, Danyang Su >>>>>>>> wrote: >>>>>>>> >>>>>>>> Hi Barry, >>>>>>>> >>>>>>>> There seems another bug (not pretty sure) in PETSc-dev, as shown below. The case I used is similar with the one I mentioned recently. I have no problem running this case using PETSc 3.5.2. But it give out the following error using PETSc-dev, even with only one processor. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Danyang >>>>>>>> >>>>>>>> timestep: 2048 time: 4.615E+00 years delt: 1.460E-20 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>> [0]PETSC ERROR: Object is in wrong state >>>>>>>> [0]PETSC ERROR: Vec is locked read only, argument # 1 >>>>>>>> [0]PETSC ERROR: See >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>> for trouble shooting. >>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2796-g23b6c49 GIT Date: 2015-04-27 11:32:44 -0500 >>>>>>>> [0]PETSC ERROR: ../min3p_thcm on a linux-gnu-dbg named nwmop by dsu Tue Apr 28 15:56:41 2015 >>>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack >>>>>>>> [0]PETSC ERROR: #349 VecGetArray() line 1646 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/interface/rvector.c >>>>>>>> [0]PETSC ERROR: #350 VecGetArrayPair() line 434 in /home/dsu/Soft/PETSc/petsc-dev/include/petscvec.h >>>>>>>> [0]PETSC ERROR: #351 VecScatterBegin_SSToSS() line 649 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c >>>>>>>> [0]PETSC ERROR: #352 VecScatterBegin() line 1694 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c >>>>>>>> [0]PETSC ERROR: #353 DMLocalToGlobalBegin_DA() line 56 in /home/dsu/Soft/PETSc/petsc-dev/src/dm/impls/da/dagtol.c >>>>>>>> [0]PETSC ERROR: #354 DMLocalToGlobalBegin() line 1944 in /home/dsu/Soft/PETSc/petsc-dev/src/dm/interface/dm.c >>>>>>>> Reduce time step for reactive transport >>>>>>>> no further time step reduction possible >>>>>>>> Optimal time increment computed by MIN3P 2.6644769214899893E-018 >>>>>>>> Minimum time increment specified by user 3.6499999999999998E-018 >>>>>>>> Please, reduce the MINIMUM TIME INCREMENT >>>>>>>> stop signal in time step reduction failed >>>>>>>> >>>>>>>> On 15-04-28 11:15 AM, Barry Smith wrote: >>>>>>>> >>>>>>>>> I am forwarding this on to the hypre support email list; hopefully they can have some advice on how to proceed. >>>>>>>>> >>>>>>>>> hypre folks, this is with version hypre-2.10.0b (the previous version had the same behavior). We are getting a divide by zero in gselim() line 4363 (this happens in a time dependent problem after many successful solves with time dependent matrix). I looked at the code, below, and note that there are some checks for the diagonal of A not being zero but not for the line that causes the divide by zero; is this perhaps an oversight in the hypre code? Any advice appreciated. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> HYPRE_Int gselim(A,x,n) >>>>>>>>> HYPRE_Real *A; >>>>>>>>> HYPRE_Real *x; >>>>>>>>> HYPRE_Int n; >>>>>>>>> { >>>>>>>>> HYPRE_Int err_flag = 0; >>>>>>>>> HYPRE_Int j,k,m; >>>>>>>>> HYPRE_Real factor; >>>>>>>>> if (n==1) /* A is 1x1 */ >>>>>>>>> { >>>>>>>>> if (A[0] != 0.0) >>>>>>>>> { >>>>>>>>> x[0] = x[0]/A[0]; >>>>>>>>> return(err_flag); >>>>>>>>> } >>>>>>>>> else >>>>>>>>> { >>>>>>>>> err_flag = 1; >>>>>>>>> return(err_flag); >>>>>>>>> } >>>>>>>>> } >>>>>>>>> else /* A is nxn. Forward elimination */ >>>>>>>>> { >>>>>>>>> for (k = 0; k < n-1; k++) >>>>>>>>> { >>>>>>>>> if (A[k*n+k] != 0.0) >>>>>>>>> { >>>>>>>>> for (j = k+1; j < n; j++) >>>>>>>>> { >>>>>>>>> if (A[j*n+k] != 0.0) >>>>>>>>> { >>>>>>>>> factor = A[j*n+k]/A[k*n+k]; >>>>>>>>> for (m = k+1; m < n; m++) >>>>>>>>> { >>>>>>>>> A[j*n+m] -= factor * A[k*n+m]; >>>>>>>>> } >>>>>>>>> /* Elimination step for rhs */ >>>>>>>>> x[j] -= factor * x[k]; >>>>>>>>> } >>>>>>>>> } >>>>>>>>> } >>>>>>>>> } >>>>>>>>> /* Back Substitution */ >>>>>>>>> for (k = n-1; k > 0; --k) >>>>>>>>> { >>>>>>>>> x[k] /= A[k*n+k]; >>>>>>>>> for (j = 0; j < k; j++) >>>>>>>>> { >>>>>>>>> if (A[j*n+k] != 0.0) >>>>>>>>> { >>>>>>>>> x[j] -= x[k] * A[j*n+k]; >>>>>>>>> } >>>>>>>>> } >>>>>>>>> } >>>>>>>>> x[0] /= A[0]; >>>>>>>>> return(err_flag); >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Apr 28, 2015, at 12:55 PM, Danyang Su >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi Barry, >>>>>>>>>> >>>>>>>>>> The development version of PETSc does not help to solve my problem. It still crashed due to the same error information. >>>>>>>>>> >>>>>>>>>> As Matthew mentioned, I checked the matrix and non of the diagonal entry is zero. But the diagonal values for different rows range from -1.0d-18 to 1.0d33 and the matrix is not strict diagonal dominant. When using 1 processor, the matrix is also not strict diagonal dominant, but the diagonal values have a much smaller range and the codes can run without error. Maybe there is something wrong in the matrix for this case and I need to check the matrix first. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Danyang >>>>>>>>>> >>>>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>>>> 0x00007f668c1299d5 in gselim (A=0x2fe3330, x=0x2210680, n=9) >>>>>>>>>> at par_relax.c:4363 >>>>>>>>>> 4363 x[k] /= A[k*n+k]; >>>>>>>>>> (gdb) >>>>>>>>>> >>>>>>>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>> [1]PETSC ERROR: is given. >>>>>>>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-dev/src/sys/error/fp.c >>>>>>>>>> [1]PETSC ERROR: [1] Hypre solve line 216 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >>>>>>>>>> [1]PETSC ERROR: [1] PCApply_HYPRE line 203 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >>>>>>>>>> [1]PETSC ERROR: [1] PCApply line 430 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/interface/precon.c >>>>>>>>>> [1]PETSC ERROR: [1] KSP_PCApply line 234 /home/dsu/Soft/PETSc/petsc-dev/include/petsc/private/kspimpl.h >>>>>>>>>> [1]PETSC ERROR: [1] KSPInitialResidual line 44 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itres.c >>>>>>>>>> [1]PETSC ERROR: [1] KSPSolve_GMRES line 224 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>> [1]PETSC ERROR: [1] KSPSolve line 506 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>>>>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 15-04-27 11:47 AM, Barry Smith wrote: >>>>>>>>>> >>>>>>>>>>> Danyang, >>>>>>>>>>> >>>>>>>>>>> I'm glad you were finally able to get to the problematic point; sorry it took so long (I want to strangle whoever turned on catching of underflows in PETSc). >>>>>>>>>>> >>>>>>>>>>> The hypre folks have done a great deal of work on their relaxation code since the PETSc 3.5.3 release. There is a good chance they have fixed the divide by zero problem you are hitting here. You will need to upgrade to the development version of PETSc (that uses the latest version of hypre), here are the instructions on how to obtain it >>>>>>>>>>> http://www.mcs.anl.gov/petsc/developers/index.html >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Please let us know if this resolves the problem with hypre failing. >>>>>>>>>>> >>>>>>>>>>> Barry >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On Apr 27, 2015, at 11:44 AM, Danyang Su >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi Barry, >>>>>>>>>>>> >>>>>>>>>>>> I got the following arithemetic exception after the previous bug is fixed. >>>>>>>>>>>> >>>>>>>>>>>> Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 >>>>>>>>>>>> 0x00007f3b23b98f20 in __nanosleep_nocancel () >>>>>>>>>>>> at ../sysdeps/unix/syscall-template.S:81 >>>>>>>>>>>> 81 ../sysdeps/unix/syscall-template.S: No such file or directory. >>>>>>>>>>>> (gdb) cont >>>>>>>>>>>> Continuing. >>>>>>>>>>>> >>>>>>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>>>>>> 0x00007f3b260df449 in gselim (A=0x1e4c580, x=0x1d30c40, n=9) >>>>>>>>>>>> at par_relax.c:3442 >>>>>>>>>>>> 3442 x[k] /= A[k*n+k]; >>>>>>>>>>>> (gdb) >>>>>>>>>>>> >>>>>>>>>>>> I tried both PETSc 3.5.2 and 3.5.3 and they return the same error as shown above. For 3.5.3, i edited fp.c file and then configure and make. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Danyang >>>>>>>>>>>> >>>>>>>>>>>> On 15-04-25 07:34 PM, Danyang Su wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi All, >>>>>>>>>>>>> >>>>>>>>>>>>> The "floating point underflow" is caused by a small value divided by a very large value. This result is forced to zero and then it does not report any underflow problem. I just rerun this bad case to see if it still get stuck later. This will take a while. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for all your kindly reply, >>>>>>>>>>>>> >>>>>>>>>>>>> Danyang >>>>>>>>>>>>> >>>>>>>>>>>>> On 15-04-25 07:02 PM, Barry Smith wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Ok, you do have >>>>>>>>>>>>>> >>>>>>>>>>>>>> #ifndef PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>>>>> #define PETSC_HAVE_XMMINTRIN_H 1 >>>>>>>>>>>>>> #endif >>>>>>>>>>>>>> >>>>>>>>>>>>>> so the change you made should cause it to stop trapping underflow exceptions. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Now in one email you reported a FPE within hypre, then I asked you to run with -start_in_debugger to determine where it happened exactly and then you reported the FPE happened in user code (what seemed to be an underflow issue). Why is this? Can you not run it where it generated the FPE in hypre using the -start_in_debugger option? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Barry >>>>>>>>>>>>>> >>>>>>>>>>>>>> Perhaps you have multiple PETSC_ARCH or multiple PETSc installs to explain why you reported two different places where the exception occurred. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Apr 25, 2015, at 8:31 PM, Danyang Su >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 15-04-25 06:26 PM, Matthew Knepley wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 15-04-25 06:03 PM, Barry Smith wrote: >>>>>>>>>>>>>>>> If this is what you got in your last run >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>>>>>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>>>>>>>>>>> then it is still catching floating point underflow, which we do not want. This means either the change I suggested you make in the fp.c code didn't work or it actually uses a different floating point trap than that one. BTW: absurd numbers like 1.0d-316 are often a symptom of uninitialized data; could that be a problem that diff is not filled correctly for all the ic, idim you are using? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This going round and round is very frustrating and a waste of time. You need to be more proactive yourself and explore the code and poke around to figure out how to solve the problem. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can see what FP trap is being used on your machine. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Barry >>>>>>>>>>>>>>>> Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise I cannot find this file. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Its include/petscconf.h >>>>>>>>>>>>>>>> Do I need to reconfigure PETSc after changing the code you mentioned? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> No, but you need to rebuild. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yes, I have done 'gnumake'. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Matt >>>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Apr 25, 2015, at 2:24 PM, Danyang Su >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 15-04-25 11:55 AM, Barry Smith wrote: >>>>>>>>>>>>>>>> On Apr 25, 2015, at 1:51 PM, Danyang Su >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 15-04-25 11:32 AM, Barry Smith wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I told you this yesterday. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>>>>>>>>>>>>>>> #else >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> change it to >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >>>>>>>>>>>>>>>> #else >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Did you do this? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Barry >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc directory. I just did a double check and ran make gnumake again and got the following information this time. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> make gnumake >>>>>>>>>>>>>>>> Building PETSc using GNU Make with 10 build threads >>>>>>>>>>>>>>>> ========================================== >>>>>>>>>>>>>>>> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>>>>>>>>>> make[1]: Nothing to be done for `all'. >>>>>>>>>>>>>>>> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>>>>>>>>>> ========================================= >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Then I recompiled the codes, ran with -fp_trap and still got the following error >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>>>>>> Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>>>>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>>>>>>> [2]PETSC ERROR: is given. >>>>>>>>>>>>>>>> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>>>>> INSTEAD the line number of the start of the function >>>>>>>>>>>>>>>> [3]PETSC ERROR: is given. >>>>>>>>>>>>>>>> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>>>>> [2]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>>>>>>> [3]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This is different then what you sent a few minutes ago where it crashed in hypre. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Anyways you need to use the -start_in_debugger business I sent in the previous email to see the exact place the problem occurs. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Here is the information shown on gdb screen >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>>>>>>>>>> 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, njamxc=34, >>>>>>>>>>>>>>>> cinfradx=..., radial_coordx=.FALSE., _suffix=3) >>>>>>>>>>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>>>>>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>>>>>>>>>>> (gdb) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> After type cont on gdb screen, I got error information as below >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>>>>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>>>>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>>>>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>>>>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>>>>>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>>>>>>> [1]PETSC ERROR: is given. >>>>>>>>>>>>>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Apr 25, 2015, at 1:05 AM, Danyang Su >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Barry and Satish, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> How can I get rid of unknown floating point error when a very small value is multiplied. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> e.g., >>>>>>>>>>>>>>>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I get the following error when run with "-fp_trap -start_in_debugger". >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>>>>>> *** unknown floating point error occurred *** >>>>>>>>>>>>>>>> [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>>>>>>> [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>>>>>>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 >>>>>>>>>>>>>>>> where the result is a bitwise OR of the following flags: >>>>>>>>>>>>>>>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>>>>> [2]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I don't think this number can be stored in a real*4. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Satish >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, Satish. It is caused by this number. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> One of my case crashes because of floating point exception when using 4 >>>>>>>>>>>>>>>> processors, as shown below. But if I run this case with 1 processor, it >>>>>>>>>>>>>>>> works fine. I have tested the codes with around 100 cases up to 768 >>>>>>>>>>>>>>>> processors, all other cases work fine. I just wonder if this kind of error >>>>>>>>>>>>>>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Yes, almost for sure it is one of these places. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> First run the bad case with -fp_trap if all goes well you'll see the >>>>>>>>>>>>>>>> function where the FPE is generated. Then run also with -start_in_debugger >>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>> type cont in all four debugger windows. When the FPE happens the debugger >>>>>>>>>>>>>>>> should stop showing exactly where the FPE happens. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Barry >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Barry, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> If run with -fp_trap -start_in_debugger, I got the following error >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a >>>>>>>>>>>>>>>> debugger. When the >>>>>>>>>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >>>>>>>>>>>>>>>> fetestexcept(0x3d) >>>>>>>>>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>>>>>>>>>>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>>>>>>>>>>>> ------------------------------------ >>>>>>>>>>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>>>>>>> [0]PETSC ERROR: is given. >>>>>>>>>>>>>>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>>>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>>>>>>>>>>>>>>> floating point error >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Program received signal SIGABRT: Process abort signal. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>>>>>> #0 0x7F4FEAB1C7D7 >>>>>>>>>>>>>>>> #1 0x7F4FEAB1CDDE >>>>>>>>>>>>>>>> #2 0x7F4FE9E1AD3F >>>>>>>>>>>>>>>> #3 0x7F4FE9E1ACC9 >>>>>>>>>>>>>>>> #4 0x7F4FE9E1E0D7 >>>>>>>>>>>>>>>> #5 0x7F4FEB0B6DCB >>>>>>>>>>>>>>>> #6 0x7F4FEB0B1825 >>>>>>>>>>>>>>>> #7 0x7F4FEB0B817F >>>>>>>>>>>>>>>> #8 0x7F4FE9E1AD3F >>>>>>>>>>>>>>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>>>>>>>>>>>>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>>>>>>>>>>>>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>>>>>>>>>>>>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>>>>>>>>>>>>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I checked the code at tprfrtlc.F90:2393, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>>>>>>>>>>>>>> (cx(ix),ix=1,nxout)/) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> All the values (time, c, cx) are reasonable, as shown below. The only >>>>>>>>>>>>>>>> possibility is that realbuffer_gb is in declared as real*4 if using sing >>>>>>>>>>>>>>>> precision output while time, c, cx are declared in real*8. I have a lot of >>>>>>>>>>>>>>>> similar data conversion from real*8 to real*4 output, other code does not >>>>>>>>>>>>>>>> return error. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> time 0.0000000000000000 >>>>>>>>>>>>>>>> c 1 9.9999999999999995E-008 >>>>>>>>>>>>>>>> c 2 3.1555251077549618E-003 >>>>>>>>>>>>>>>> c 3 7.1657814842179362E-008 >>>>>>>>>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>>>>>>>>> c 5 5.2879822292305797E-004 >>>>>>>>>>>>>>>> c 6 9.9999999999999964E-005 >>>>>>>>>>>>>>>> c 7 6.4055731968811337E-005 >>>>>>>>>>>>>>>> c 8 3.4607572892578404E-020 >>>>>>>>>>>>>>>> cx 1 3.4376650636008101E-005 >>>>>>>>>>>>>>>> cx 2 7.3989678854017763E-012 >>>>>>>>>>>>>>>> cx 3 9.5317170613607207E-012 >>>>>>>>>>>>>>>> cx 4 2.2344525794718353E-015 >>>>>>>>>>>>>>>> cx 5 3.0624685689695889E-008 >>>>>>>>>>>>>>>> cx 6 1.0046157902783967E-007 >>>>>>>>>>>>>>>> cx 7 1.5320169154914984E-004 >>>>>>>>>>>>>>>> cx 8 8.6930292776346176E-014 >>>>>>>>>>>>>>>> cx 9 3.5944267559348721E-005 >>>>>>>>>>>>>>>> cx 10 3.0072645866951157E-018 >>>>>>>>>>>>>>>> cx 11 2.3592486321095017E-013 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I can check all the entries of jacobi matrix to see if the value is valid, >>>>>>>>>>>>>>>> but this seems not a good idea as it takes a long time to reach this >>>>>>>>>>>>>>>> point. If I restart the simulation from a specified time (e.g., 7.685 in >>>>>>>>>>>>>>>> this case), then the error does not occur. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Would you please give me any suggestion on debugging this case? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>>>>>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >>>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>>>>>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >>>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>>>>>> [1]PETSC ERROR: Floating point exception >>>>>>>>>>>>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>>>>>> [2]PETSC ERROR: Floating point exception >>>>>>>>>>>>>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >>>>>>>>>>>>>>>> at end of function: Parameter number 3 >>>>>>>>>>>>>>>> [2]PETSC ERROR: See >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> for trouble shooting. >>>>>>>>>>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>>>>>>>>>>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>>>>>>>>>>>>> [1]PETSC ERROR: See >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> for trouble shooting. >>>>>>>>>>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>>>>>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>>>>>>>>>>>>>>> dsu Thu Apr 23 15:38:52 2015 >>>>>>>>>>>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>>>>>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>>>>>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >>>>>>>>>>>>>>>> 15:38:52 2015 >>>>>>>>>>>>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>>>>>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>>>>>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>>>>>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>>>>>>>> #3 KSP_PCApply() line 230 in >>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>>>>>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>>>>>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>>>>>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>>>>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>>>>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>>>>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>>>>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>>>>>>>>>>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> From bsmith at mcs.anl.gov Wed Apr 29 17:30:32 2015 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 29 Apr 2015 17:30:32 -0500 Subject: [petsc-users] Floating point exception in hypre BoomerAMG In-Reply-To: <5541590F.6070806@gmail.com> References: <553A85DF.5000400@gmail.com> <553AAD7A.1030807@gmail.com> <553B2E99.6010601@gmail.com> <2BEB2291-7C98-4BF7-9481-1BC0F33880F5@mcs.anl.gov> <553BE244.7060407@gmail.com> <8E11BAE6-7DB2-43C6-AF40-21E6B2B49124@mcs.anl.gov> <553BE9D4.7050801@gmail.com> <0D581D35-64A0-4928-B365-7C3BEB61FA83@mcs.anl.gov> <553C3E15.8030403@gmail.com> <553C3FE6.5020203@gmail.com> <553C4ECD.6090905@gmail.com> <553E6770.5010205@gmail.com> <553FC98D.1000904@gmail.com> <89DAC785-3218-4E2F-8760-2D0E99652014@mcs.anl.gov> <5540261E.2040502@gmail.com> <6411F61F-CD41-4037-B727-B5D9CECE030F@mcs.anl.gov> <554111AD.3060007@gmail.com> <9C61B774-4A78-48E1-B34E-B23EC2D230B9@mcs.anl.gov> <55412888.90206@gmail.com> <0F8EF2B9-9A1C-4C94-B1CC-D520B6D7A0DE@mcs.anl.gov> <55413294.1000906@gmail.com> <5541590F.6070806@gmail.com> Message-ID: Ok, it sounds to me like you are not checking the error codes and so when hypre fails your code continues along as if nothing happened. Thus the lock set in KSPSolve() remains in place and hits your code later when you try to write to those locations. You really should be checking error codes and doing something different if an error occurred and not just keep on running. You Barry > On Apr 29, 2015, at 5:19 PM, Danyang Su wrote: > > Hi Barry, > > I haven't tried your suggestion but it seems the bug from HYPRE. Here are my tests: > > Test A: > If I run with start_in_debugger, I got PETSC ERROR: PCApply_HYPRE(), as shown below. If I run without start_in_debugger, then I got Vec lock problem as I mentioned before, exactly at the same timestep. > > timestep: 2048 time: 4.619E+00 years delt: 4.208E-03 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 > Newton Iteration Convergence Summary: > Newton maximum for maximum solver > iteration update component residual iterations maxvol nexvol > 1 3.0000D+00 o2(aq) 2.4957D-03 1 33 39 > 2 3.0000D+00 o2(aq) 6.4858D-03 1 33 39 > 3 3.0000D+00 o2(aq) 5.7149D-03 1 33 39 > 4 3.0000D+00 o2(aq) 4.9772D-03 1 33 39 > 5 -3.0000D+00 co3-2 5.9683D-06 180 33 39 > 6 3.0000D+00 h+1 2.5193D+04 1 1 39 > 7 3.0000D+00 co3-2 2.4783D+16 3 1 39 > 8 3.0000D+00 h+1 3.6929D+29 14 1 39 > [0]PETSC ERROR: PCApply_HYPRE() line 174 in /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/impls/hypre/hypre.c Error in HYPRE solver, error code 1 > > Test B: > I changed the preconditioner to bjacobi then all the errors are gone, whatever with or without start_in_debugger option. Usually we use HYPRE as preconditioner method as it is generally faster than other methods for our cases. > > Do I still need to test on this case to locate the error or just wait until the bug of HYPRE is fixed. > > Thanks, > > Danyang > > On 15-04-29 12:44 PM, Barry Smith wrote: >> Ok, run on one process in the debugger and put a break point in VecLockPush and VecLockPop then run and each time it breaks on VecLockPush or VecLockPop type >> >> bt >> >> then type >> >> cont >> >> send all the output, this will tell us where the vector got locked readonly and did not get unlocked. >> >> Barry >> >> >> >>> On Apr 29, 2015, at 2:35 PM, Danyang Su wrote: >>> >>> On 15-04-29 12:19 PM, Barry Smith wrote: >>>> Ok, your code seems fine in terms of logic. But the vector x_vec_gbl i.e. b_react should not be in a read only state. Are you somewhere calling a VecGetArray() and not calling the VecRestoreArray()? >>>> >>>> Barry >>> I just made a triple check on VecGetArrayF90(). All the VecGetArrayF90() come with VecRestoreArrayF90(). No VecGetArray() is used in my codes. >>> >>> Danyang >>>> >>>> >>>>> On Apr 29, 2015, at 1:52 PM, Danyang Su wrote: >>>>> >>>>> On 15-04-29 11:30 AM, Barry Smith wrote: >>>>>>> On Apr 29, 2015, at 12:15 PM, Danyang Su >>>>>>> wrote: >>>>>>> >>>>>>> On 15-04-28 06:50 PM, Barry Smith wrote: >>>>>>> >>>>>>>> We started enforcing more checks on writing to vectors that you should not write to. Where are you calling DMLocalToGlobalBegin() ? It looks like you are trying to copy values into a global array that you should not because it is a read only input to a function, for example it is the right hand side of a linear system or the input into a SNESFormFunction(). So check the output to that call. >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> >>>>>>> This is used when local RHS vec is calculated and copied to global RHS. >>>>>>> >>>>>> Huhh, why are you copying anything into the RHS? Is this before you call the linear system solve? Send the code that calls the DMLocalToGlobalBegin() >>>>> Yes. This is called before the linear system solver. At present, we use PETSc KSP solver. >>>>> The code is first developed in totally sequential version and then ported to the parallel version. So the subdomain has its own space for jacobi matrix and rhs. We do not use global RHS vector directly to set the RHS at this moment. All the RHS values of subdomain are copied to the global RHS. >>>>> >>>>> !> This is the function where RHS values of subdomain are copied to the global RHS values. >>>>> subroutine compute_function(rank,da,x_array_loc,x_vec_loc, & >>>>> x_vec_gbl,nngl,row_idx_l2pg,col_idx_l2pg, & >>>>> b_non_interlaced) >>>>> implicit none >>>>> #include >>>>> #include >>>>> #include >>>>> #include >>>>> #include >>>>> #include >>>>> #include >>>>> #include >>>>> #include >>>>> >>>>> PetscInt :: rank >>>>> DM :: da >>>>> PetscReal, allocatable :: x_array_loc(:) >>>>> Vec :: x_vec_loc >>>>> Vec :: x_vec_gbl >>>>> PetscInt :: nngl >>>>> PetscInt, allocatable :: row_idx_l2pg(:) >>>>> PetscInt, allocatable :: col_idx_l2pg(:) >>>>> PetscBool :: b_non_interlaced >>>>> >>>>> PetscInt :: info_debug >>>>> PetscInt :: i, j >>>>> PetscErrorCode :: ierr >>>>> PetscScalar, pointer :: vecpointer(:) >>>>> !Zero entries >>>>> call VecZeroEntries(x_vec_loc, ierr) >>>>> !Get a pointer to vector data when you need access to the array >>>>> call VecGetArrayF90(x_vec_loc, vecpointer, ierr) >>>>> !Compute the function over the locally owned part of the grid >>>>> if(b_non_interlaced) then >>>>> j = nngl/2 >>>>> do i = 1, j >>>>> vecpointer(2*i-1) = x_array_loc(i) >>>>> vecpointer(2*i) = x_array_loc(i+j) >>>>> end do >>>>> else >>>>> do i = 1, nngl >>>>> vecpointer(i) = x_array_loc(i) >>>>> end do >>>>> end if >>>>> !Restore the vector when you no longer need access to the array >>>>> call VecRestoreArrayF90(x_vec_loc,vecpointer,ierr) >>>>> !Insert values into global vector >>>>> call DMLocalToGlobalBegin(da,x_vec_loc,INSERT_VALUES, & >>>>> x_vec_gbl,ierr) >>>>> !By placing code between these two statements, computations can be >>>>> !done while messages are in transition. >>>>> call DMLocalToGlobalEnd(da,x_vec_loc,INSERT_VALUES, & >>>>> x_vec_gbl,ierr) >>>>> return >>>>> end subroutine >>>>> >>>>> >>>>> >>>>> !> This is the function where reactive transport equations are solved. >>>>> subroutine solver_dd_snes_solve_react(ilog,idetail,a_in,b_in, & >>>>> x_inout,ia_in,ja_in,nngl_in,itsolv, & >>>>> over_flow,rnorm,row_idx_l2pg,col_idx_l2pg, & >>>>> b_non_interlaced) >>>>> use gen, only : rank, node_idx_l2lg, ittot_rt, & >>>>> b_output_matrix, b_enable_output >>>>> use solver_snes_function, only : form_initial_guess, & >>>>> compute_function, & >>>>> compute_jacobian >>>>> use petsc_mpi_common, only : petsc_mpi_finalize >>>>> implicit none >>>>> #include >>>>> #include >>>>> #include >>>>> #include >>>>> >>>>> PetscInt :: ilog >>>>> PetscInt :: idetail >>>>> PetscInt :: nngl_in >>>>> PetscReal, allocatable :: a_in(:) >>>>> PetscReal, allocatable :: b_in(:) >>>>> PetscReal, allocatable :: x_inout(:) >>>>> PetscInt, allocatable :: ia_in(:) >>>>> PetscInt, allocatable :: ja_in(:) >>>>> PetscInt, allocatable :: row_idx_l2pg(:) >>>>> PetscInt, allocatable :: col_idx_l2pg(:) >>>>> PetscInt :: itsolv >>>>> PetscBool :: over_flow >>>>> PetscBool :: b_non_interlaced >>>>> PetscReal :: rnorm >>>>> PetscViewer :: viewer >>>>> PetscScalar,pointer :: vecpointer(:) >>>>> >>>>> PetscErrorCode :: ierr >>>>> PetscInt :: info_debug >>>>> character(72) :: strinum >>>>> info_debug = 0 >>>>> if(b_output_matrix .and. b_enable_output) then >>>>> write(strinum, *) ittot_rt >>>>> strinum = "_"//trim(adjustl(strinum)) >>>>> end if >>>>> >>>>> !Form initial guess, only assemble the local owned part, without ghost nodes >>>>> call form_initial_guess(rank,dmda_react%da,x_inout,x_react_loc,& >>>>> x_react,nngl_in, row_idx_l2pg,col_idx_l2pg, & >>>>> b_non_interlaced) >>>>> >>>>> !Compute function, only assemble the local part, without ghost nodes >>>>> call compute_function(rank,dmda_react%da,b_in,b_react_loc, & >>>>> b_react,nngl_in,row_idx_l2pg,col_idx_l2pg, & >>>>> b_non_interlaced) >>>>> !Compute jacobian matrix, only assemble the local part, without ghost nodes >>>>> call compute_jacobian(rank,dmda_react%da, & >>>>> a_react,a_in,ia_in,ja_in,nngl_in, & >>>>> row_idx_l2pg,col_idx_l2pg, & >>>>> b_non_interlaced) >>>>> >>>>> ! Solve a x = b, where a is the Jacobian matrix. >>>>> ! - First, set the KSP linear operators. Here the matrix that >>>>> ! defines the linear system also serves as the preconditioning >>>>> ! matrix. >>>>> ! - Then solve the Newton system. >>>>> #ifdef PETSC_V3_5_X >>>>> call KSPSetOperators(ksp_react,a_react,a_react,ierr) >>>>> #else >>>>> call KSPSetOperators(ksp_react,a_react,a_react, & >>>>> SAME_NONZERO_PATTERN,ierr) >>>>> #endif >>>>> call KSPSetDM(ksp_react,dmda_react%da,ierr) >>>>> call KSPSetDMActive(ksp_react,PETSC_FALSE,ierr) >>>>> >>>>> call KSPSolve(ksp_react,b_react,x_react,ierr) >>>>> !Get residual norm, by default, preconditioned residual norm is calculated. >>>>> if (b_mykspconverged_react .and. & >>>>> .not. b_use_petsc_default_react) then >>>>> rnorm = rnorm_react >>>>> else >>>>> call KSPGetResidualNorm(ksp_react, rnorm, ierr) >>>>> end if >>>>> ! Scatter ghost points to local vector, using the 2-step process >>>>> ! DMGlobalToLocalBegin(), DMGlobalToLocalEnd(). >>>>> ! By placing code between these two statements, computations can be >>>>> ! done while messages are in transition. >>>>> call DMGlobalToLocalBegin(dmda_react%da,x_react,INSERT_VALUES, & >>>>> x_react_loc,ierr) >>>>> call DMGlobalToLocalEnd(dmda_react%da,x_react,INSERT_VALUES, & >>>>> x_react_loc,ierr) >>>>> call VecGetArrayF90(x_react_loc,vecpointer,ierr) >>>>> x_inout = vecpointer >>>>> call VecRestoreArrayF90(x_react_loc,vecpointer,ierr) >>>>> return >>>>> end subroutine >>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>>> It is strange why this error does not occur at the first timestep. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Danyang >>>>>>> >>>>>>>>> On Apr 28, 2015, at 7:30 PM, Danyang Su >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi Barry, >>>>>>>>> >>>>>>>>> There seems another bug (not pretty sure) in PETSc-dev, as shown below. The case I used is similar with the one I mentioned recently. I have no problem running this case using PETSc 3.5.2. But it give out the following error using PETSc-dev, even with only one processor. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Danyang >>>>>>>>> >>>>>>>>> timestep: 2048 time: 4.615E+00 years delt: 1.460E-20 years iter: 1 max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>>> [0]PETSC ERROR: Object is in wrong state >>>>>>>>> [0]PETSC ERROR: Vec is locked read only, argument # 1 >>>>>>>>> [0]PETSC ERROR: See >>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>> for trouble shooting. >>>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2796-g23b6c49 GIT Date: 2015-04-27 11:32:44 -0500 >>>>>>>>> [0]PETSC ERROR: ../min3p_thcm on a linux-gnu-dbg named nwmop by dsu Tue Apr 28 15:56:41 2015 >>>>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mumps --download-hypre --download-superlu_dist --download-metis --download-parmetis --download-scalapack >>>>>>>>> [0]PETSC ERROR: #349 VecGetArray() line 1646 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/interface/rvector.c >>>>>>>>> [0]PETSC ERROR: #350 VecGetArrayPair() line 434 in /home/dsu/Soft/PETSc/petsc-dev/include/petscvec.h >>>>>>>>> [0]PETSC ERROR: #351 VecScatterBegin_SSToSS() line 649 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c >>>>>>>>> [0]PETSC ERROR: #352 VecScatterBegin() line 1694 in /home/dsu/Soft/PETSc/petsc-dev/src/vec/vec/utils/vscat.c >>>>>>>>> [0]PETSC ERROR: #353 DMLocalToGlobalBegin_DA() line 56 in /home/dsu/Soft/PETSc/petsc-dev/src/dm/impls/da/dagtol.c >>>>>>>>> [0]PETSC ERROR: #354 DMLocalToGlobalBegin() line 1944 in /home/dsu/Soft/PETSc/petsc-dev/src/dm/interface/dm.c >>>>>>>>> Reduce time step for reactive transport >>>>>>>>> no further time step reduction possible >>>>>>>>> Optimal time increment computed by MIN3P 2.6644769214899893E-018 >>>>>>>>> Minimum time increment specified by user 3.6499999999999998E-018 >>>>>>>>> Please, reduce the MINIMUM TIME INCREMENT >>>>>>>>> stop signal in time step reduction failed >>>>>>>>> >>>>>>>>> On 15-04-28 11:15 AM, Barry Smith wrote: >>>>>>>>> >>>>>>>>>> I am forwarding this on to the hypre support email list; hopefully they can have some advice on how to proceed. >>>>>>>>>> >>>>>>>>>> hypre folks, this is with version hypre-2.10.0b (the previous version had the same behavior). We are getting a divide by zero in gselim() line 4363 (this happens in a time dependent problem after many successful solves with time dependent matrix). I looked at the code, below, and note that there are some checks for the diagonal of A not being zero but not for the line that causes the divide by zero; is this perhaps an oversight in the hypre code? Any advice appreciated. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> Barry >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> HYPRE_Int gselim(A,x,n) >>>>>>>>>> HYPRE_Real *A; >>>>>>>>>> HYPRE_Real *x; >>>>>>>>>> HYPRE_Int n; >>>>>>>>>> { >>>>>>>>>> HYPRE_Int err_flag = 0; >>>>>>>>>> HYPRE_Int j,k,m; >>>>>>>>>> HYPRE_Real factor; >>>>>>>>>> if (n==1) /* A is 1x1 */ >>>>>>>>>> { >>>>>>>>>> if (A[0] != 0.0) >>>>>>>>>> { >>>>>>>>>> x[0] = x[0]/A[0]; >>>>>>>>>> return(err_flag); >>>>>>>>>> } >>>>>>>>>> else >>>>>>>>>> { >>>>>>>>>> err_flag = 1; >>>>>>>>>> return(err_flag); >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> else /* A is nxn. Forward elimination */ >>>>>>>>>> { >>>>>>>>>> for (k = 0; k < n-1; k++) >>>>>>>>>> { >>>>>>>>>> if (A[k*n+k] != 0.0) >>>>>>>>>> { >>>>>>>>>> for (j = k+1; j < n; j++) >>>>>>>>>> { >>>>>>>>>> if (A[j*n+k] != 0.0) >>>>>>>>>> { >>>>>>>>>> factor = A[j*n+k]/A[k*n+k]; >>>>>>>>>> for (m = k+1; m < n; m++) >>>>>>>>>> { >>>>>>>>>> A[j*n+m] -= factor * A[k*n+m]; >>>>>>>>>> } >>>>>>>>>> /* Elimination step for rhs */ >>>>>>>>>> x[j] -= factor * x[k]; >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> /* Back Substitution */ >>>>>>>>>> for (k = n-1; k > 0; --k) >>>>>>>>>> { >>>>>>>>>> x[k] /= A[k*n+k]; >>>>>>>>>> for (j = 0; j < k; j++) >>>>>>>>>> { >>>>>>>>>> if (A[j*n+k] != 0.0) >>>>>>>>>> { >>>>>>>>>> x[j] -= x[k] * A[j*n+k]; >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> x[0] /= A[0]; >>>>>>>>>> return(err_flag); >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Apr 28, 2015, at 12:55 PM, Danyang Su >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Barry, >>>>>>>>>>> >>>>>>>>>>> The development version of PETSc does not help to solve my problem. It still crashed due to the same error information. >>>>>>>>>>> >>>>>>>>>>> As Matthew mentioned, I checked the matrix and non of the diagonal entry is zero. But the diagonal values for different rows range from -1.0d-18 to 1.0d33 and the matrix is not strict diagonal dominant. When using 1 processor, the matrix is also not strict diagonal dominant, but the diagonal values have a much smaller range and the codes can run without error. Maybe there is something wrong in the matrix for this case and I need to check the matrix first. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Danyang >>>>>>>>>>> >>>>>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>>>>> 0x00007f668c1299d5 in gselim (A=0x2fe3330, x=0x2210680, n=9) >>>>>>>>>>> at par_relax.c:4363 >>>>>>>>>>> 4363 x[k] /= A[k*n+k]; >>>>>>>>>>> (gdb) >>>>>>>>>>> >>>>>>>>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>> [1]PETSC ERROR: is given. >>>>>>>>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-dev/src/sys/error/fp.c >>>>>>>>>>> [1]PETSC ERROR: [1] Hypre solve line 216 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >>>>>>>>>>> [1]PETSC ERROR: [1] PCApply_HYPRE line 203 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/impls/hypre/hypre.c >>>>>>>>>>> [1]PETSC ERROR: [1] PCApply line 430 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/pc/interface/precon.c >>>>>>>>>>> [1]PETSC ERROR: [1] KSP_PCApply line 234 /home/dsu/Soft/PETSc/petsc-dev/include/petsc/private/kspimpl.h >>>>>>>>>>> [1]PETSC ERROR: [1] KSPInitialResidual line 44 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itres.c >>>>>>>>>>> [1]PETSC ERROR: [1] KSPSolve_GMRES line 224 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>> [1]PETSC ERROR: [1] KSPSolve line 506 /home/dsu/Soft/PETSc/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 15-04-27 11:47 AM, Barry Smith wrote: >>>>>>>>>>> >>>>>>>>>>>> Danyang, >>>>>>>>>>>> >>>>>>>>>>>> I'm glad you were finally able to get to the problematic point; sorry it took so long (I want to strangle whoever turned on catching of underflows in PETSc). >>>>>>>>>>>> >>>>>>>>>>>> The hypre folks have done a great deal of work on their relaxation code since the PETSc 3.5.3 release. There is a good chance they have fixed the divide by zero problem you are hitting here. You will need to upgrade to the development version of PETSc (that uses the latest version of hypre), here are the instructions on how to obtain it >>>>>>>>>>>> http://www.mcs.anl.gov/petsc/developers/index.html >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Please let us know if this resolves the problem with hypre failing. >>>>>>>>>>>> >>>>>>>>>>>> Barry >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On Apr 27, 2015, at 11:44 AM, Danyang Su >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Barry, >>>>>>>>>>>>> >>>>>>>>>>>>> I got the following arithemetic exception after the previous bug is fixed. >>>>>>>>>>>>> >>>>>>>>>>>>> Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 >>>>>>>>>>>>> 0x00007f3b23b98f20 in __nanosleep_nocancel () >>>>>>>>>>>>> at ../sysdeps/unix/syscall-template.S:81 >>>>>>>>>>>>> 81 ../sysdeps/unix/syscall-template.S: No such file or directory. >>>>>>>>>>>>> (gdb) cont >>>>>>>>>>>>> Continuing. >>>>>>>>>>>>> >>>>>>>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>>>>>>> 0x00007f3b260df449 in gselim (A=0x1e4c580, x=0x1d30c40, n=9) >>>>>>>>>>>>> at par_relax.c:3442 >>>>>>>>>>>>> 3442 x[k] /= A[k*n+k]; >>>>>>>>>>>>> (gdb) >>>>>>>>>>>>> >>>>>>>>>>>>> I tried both PETSc 3.5.2 and 3.5.3 and they return the same error as shown above. For 3.5.3, i edited fp.c file and then configure and make. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> >>>>>>>>>>>>> Danyang >>>>>>>>>>>>> >>>>>>>>>>>>> On 15-04-25 07:34 PM, Danyang Su wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>> >>>>>>>>>>>>>> The "floating point underflow" is caused by a small value divided by a very large value. This result is forced to zero and then it does not report any underflow problem. I just rerun this bad case to see if it still get stuck later. This will take a while. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks for all your kindly reply, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 15-04-25 07:02 PM, Barry Smith wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Ok, you do have >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> #ifndef PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>>>>>> #define PETSC_HAVE_XMMINTRIN_H 1 >>>>>>>>>>>>>>> #endif >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> so the change you made should cause it to stop trapping underflow exceptions. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Now in one email you reported a FPE within hypre, then I asked you to run with -start_in_debugger to determine where it happened exactly and then you reported the FPE happened in user code (what seemed to be an underflow issue). Why is this? Can you not run it where it generated the FPE in hypre using the -start_in_debugger option? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Barry >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Perhaps you have multiple PETSC_ARCH or multiple PETSc installs to explain why you reported two different places where the exception occurred. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Apr 25, 2015, at 8:31 PM, Danyang Su >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 15-04-25 06:26 PM, Matthew Knepley wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 15-04-25 06:03 PM, Barry Smith wrote: >>>>>>>>>>>>>>>>> If this is what you got in your last run >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>>>>>>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>>>>>>>>>>>> then it is still catching floating point underflow, which we do not want. This means either the change I suggested you make in the fp.c code didn't work or it actually uses a different floating point trap than that one. BTW: absurd numbers like 1.0d-316 are often a symptom of uninitialized data; could that be a problem that diff is not filled correctly for all the ic, idim you are using? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> This going round and round is very frustrating and a waste of time. You need to be more proactive yourself and explore the code and poke around to figure out how to solve the problem. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can see what FP trap is being used on your machine. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Barry >>>>>>>>>>>>>>>>> Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise I cannot find this file. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Its include/petscconf.h >>>>>>>>>>>>>>>>> Do I need to reconfigure PETSc after changing the code you mentioned? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> No, but you need to rebuild. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Yes, I have done 'gnumake'. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Matt >>>>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Apr 25, 2015, at 2:24 PM, Danyang Su >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 15-04-25 11:55 AM, Barry Smith wrote: >>>>>>>>>>>>>>>>> On Apr 25, 2015, at 1:51 PM, Danyang Su >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 15-04-25 11:32 AM, Barry Smith wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I told you this yesterday. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT); >>>>>>>>>>>>>>>>> #else >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> change it to >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> #elif defined PETSC_HAVE_XMMINTRIN_H >>>>>>>>>>>>>>>>> _MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW); >>>>>>>>>>>>>>>>> #else >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Did you do this? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Barry >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc directory. I just did a double check and ran make gnumake again and got the following information this time. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> dsu at nwmop:~/Soft/PETSc/petsc-3.5.2$ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> make gnumake >>>>>>>>>>>>>>>>> Building PETSc using GNU Make with 10 build threads >>>>>>>>>>>>>>>>> ========================================== >>>>>>>>>>>>>>>>> make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>>>>>>>>>>> make[1]: Nothing to be done for `all'. >>>>>>>>>>>>>>>>> make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2' >>>>>>>>>>>>>>>>> ========================================= >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Then I recompiled the codes, ran with -fp_trap and still got the following error >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>>>>>>> Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>>>>>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>>>>>>>> [2]PETSC ERROR: is given. >>>>>>>>>>>>>>>>> [2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>>>>>> INSTEAD the line number of the start of the function >>>>>>>>>>>>>>>>> [3]PETSC ERROR: is given. >>>>>>>>>>>>>>>>> [3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>>>>>> [2]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>>>>>>>> [3]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> This is different then what you sent a few minutes ago where it crashed in hypre. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Anyways you need to use the -start_in_debugger business I sent in the previous email to see the exact place the problem occurs. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Here is the information shown on gdb screen >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Program received signal SIGFPE, Arithmetic exception. >>>>>>>>>>>>>>>>> 0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, njamxc=34, >>>>>>>>>>>>>>>>> cinfradx=..., radial_coordx=.FALSE., _suffix=3) >>>>>>>>>>>>>>>>> at ../../gas_advection/velocity_g.F90:1344 >>>>>>>>>>>>>>>>> 1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316 >>>>>>>>>>>>>>>>> (gdb) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> After type cont on gdb screen, I got error information as below >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [1]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>>>>>>> [1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>>>>>>>> [1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>>>>>>>> [1]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>>>>>>>> [1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>>>>>> [1]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>>>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>>>>>>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>>>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>>>>>>>> [1]PETSC ERROR: is given. >>>>>>>>>>>>>>>>> [1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>>>>>> [1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error >>>>>>>>>>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Apr 25, 2015, at 1:05 AM, Danyang Su >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Barry and Satish, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> How can I get rid of unknown floating point error when a very small value is multiplied. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> e.g., >>>>>>>>>>>>>>>>> cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> cinfrt = cinfrt_dg(i1) * diff(ic,idim) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I get the following error when run with "-fp_trap -start_in_debugger". >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>>>>>>> *** unknown floating point error occurred *** >>>>>>>>>>>>>>>>> [2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the >>>>>>>>>>>>>>>>> [2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d) >>>>>>>>>>>>>>>>> [2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317 >>>>>>>>>>>>>>>>> where the result is a bitwise OR of the following flags: >>>>>>>>>>>>>>>>> [2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>>>>>> [2]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>>>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 15-04-24 01:54 PM, Danyang Su wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 15-04-24 01:23 PM, Satish Balay wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I don't think this number can be stored in a real*4. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Satish >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, Satish. It is caused by this number. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Fri, 24 Apr 2015, Danyang Su wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 15-04-24 11:12 AM, Barry Smith wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Apr 24, 2015, at 1:05 PM, Danyang Su >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> One of my case crashes because of floating point exception when using 4 >>>>>>>>>>>>>>>>> processors, as shown below. But if I run this case with 1 processor, it >>>>>>>>>>>>>>>>> works fine. I have tested the codes with around 100 cases up to 768 >>>>>>>>>>>>>>>>> processors, all other cases work fine. I just wonder if this kind of error >>>>>>>>>>>>>>>>> is caused because of NaN in jacobi matrix, RHS or preconditioner? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Yes, almost for sure it is one of these places. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> First run the bad case with -fp_trap if all goes well you'll see the >>>>>>>>>>>>>>>>> function where the FPE is generated. Then run also with -start_in_debugger >>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>> type cont in all four debugger windows. When the FPE happens the debugger >>>>>>>>>>>>>>>>> should stop showing exactly where the FPE happens. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Barry >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Barry, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> If run with -fp_trap -start_in_debugger, I got the following error >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [0]PETSC ERROR: *** unknown floating point error occurred *** >>>>>>>>>>>>>>>>> [0]PETSC ERROR: The specific exception can be determined by running in a >>>>>>>>>>>>>>>>> debugger. When the >>>>>>>>>>>>>>>>> [0]PETSC ERROR: debugger traps the signal, the exception can be found with >>>>>>>>>>>>>>>>> fetestexcept(0x3d) >>>>>>>>>>>>>>>>> [0]PETSC ERROR: where the result is a bitwise OR of the following flags: >>>>>>>>>>>>>>>>> [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 >>>>>>>>>>>>>>>>> FE_UNDERFLOW=0x10 FE_INEXACT=0x20 >>>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger >>>>>>>>>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>>>>>>>>>>>>> ------------------------------------ >>>>>>>>>>>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>>>>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>>>>>>>>>> [0]PETSC ERROR: is given. >>>>>>>>>>>>>>>>> [0]PETSC ERROR: [0] PetscDefaultFPTrap line 379 >>>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c >>>>>>>>>>>>>>>>> [0]PETSC ERROR: User provided function() line 0 in Unknown file trapped >>>>>>>>>>>>>>>>> floating point error >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Program received signal SIGABRT: Process abort signal. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Backtrace for this error: >>>>>>>>>>>>>>>>> #0 0x7F4FEAB1C7D7 >>>>>>>>>>>>>>>>> #1 0x7F4FEAB1CDDE >>>>>>>>>>>>>>>>> #2 0x7F4FE9E1AD3F >>>>>>>>>>>>>>>>> #3 0x7F4FE9E1ACC9 >>>>>>>>>>>>>>>>> #4 0x7F4FE9E1E0D7 >>>>>>>>>>>>>>>>> #5 0x7F4FEB0B6DCB >>>>>>>>>>>>>>>>> #6 0x7F4FEB0B1825 >>>>>>>>>>>>>>>>> #7 0x7F4FEB0B817F >>>>>>>>>>>>>>>>> #8 0x7F4FE9E1AD3F >>>>>>>>>>>>>>>>> #9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3) >>>>>>>>>>>>>>>>> #10 0x4C6C87 in gcreact_ at gcreact.F90:678 >>>>>>>>>>>>>>>>> #11 0x707E19 in initicrt_ at initicrt.F90:589 >>>>>>>>>>>>>>>>> #12 0x4F42D0 in initprob_ at initprob.F90:430 >>>>>>>>>>>>>>>>> #13 0x5AAF72 in driver_pc at driver_pc.F90:438 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I checked the code at tprfrtlc.F90:2393, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), & >>>>>>>>>>>>>>>>> (cx(ix),ix=1,nxout)/) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> All the values (time, c, cx) are reasonable, as shown below. The only >>>>>>>>>>>>>>>>> possibility is that realbuffer_gb is in declared as real*4 if using sing >>>>>>>>>>>>>>>>> precision output while time, c, cx are declared in real*8. I have a lot of >>>>>>>>>>>>>>>>> similar data conversion from real*8 to real*4 output, other code does not >>>>>>>>>>>>>>>>> return error. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> time 0.0000000000000000 >>>>>>>>>>>>>>>>> c 1 9.9999999999999995E-008 >>>>>>>>>>>>>>>>> c 2 3.1555251077549618E-003 >>>>>>>>>>>>>>>>> c 3 7.1657814842179362E-008 >>>>>>>>>>>>>>>>> c 4 1.0976214263087059E-067 >>>>>>>>>>>>>>>>> c 5 5.2879822292305797E-004 >>>>>>>>>>>>>>>>> c 6 9.9999999999999964E-005 >>>>>>>>>>>>>>>>> c 7 6.4055731968811337E-005 >>>>>>>>>>>>>>>>> c 8 3.4607572892578404E-020 >>>>>>>>>>>>>>>>> cx 1 3.4376650636008101E-005 >>>>>>>>>>>>>>>>> cx 2 7.3989678854017763E-012 >>>>>>>>>>>>>>>>> cx 3 9.5317170613607207E-012 >>>>>>>>>>>>>>>>> cx 4 2.2344525794718353E-015 >>>>>>>>>>>>>>>>> cx 5 3.0624685689695889E-008 >>>>>>>>>>>>>>>>> cx 6 1.0046157902783967E-007 >>>>>>>>>>>>>>>>> cx 7 1.5320169154914984E-004 >>>>>>>>>>>>>>>>> cx 8 8.6930292776346176E-014 >>>>>>>>>>>>>>>>> cx 9 3.5944267559348721E-005 >>>>>>>>>>>>>>>>> cx 10 3.0072645866951157E-018 >>>>>>>>>>>>>>>>> cx 11 2.3592486321095017E-013 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I can check all the entries of jacobi matrix to see if the value is valid, >>>>>>>>>>>>>>>>> but this seems not a good idea as it takes a long time to reach this >>>>>>>>>>>>>>>>> point. If I restart the simulation from a specified time (e.g., 7.685 in >>>>>>>>>>>>>>>>> this case), then the error does not occur. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Would you please give me any suggestion on debugging this case? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Danyang >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>>>> timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>>>> timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>>>> timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>>>> timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1 >>>>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>>>>>>>>> timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1 >>>>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>>>> Reduce time step for reactive transport >>>>>>>>>>>>>>>>> timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1 >>>>>>>>>>>>>>>>> timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00 >>>>>>>>>>>>>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>>>>>>> [1]PETSC ERROR: Floating point exception >>>>>>>>>>>>>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>>>>>>> [2]PETSC ERROR: Floating point exception >>>>>>>>>>>>>>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite >>>>>>>>>>>>>>>>> at end of function: Parameter number 3 >>>>>>>>>>>>>>>>> [2]PETSC ERROR: See >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> for trouble shooting. >>>>>>>>>>>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is >>>>>>>>>>>>>>>>> not-a-number or infinite at end of function: Parameter number 3 >>>>>>>>>>>>>>>>> [1]PETSC ERROR: See >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> for trouble shooting. >>>>>>>>>>>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>>>>>>>>>>>>>> [1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by >>>>>>>>>>>>>>>>> dsu Thu Apr 23 15:38:52 2015 >>>>>>>>>>>>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>>>>>>>>> [1]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>>>>>>>>> ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23 >>>>>>>>>>>>>>>>> 15:38:52 2015 >>>>>>>>>>>>>>>>> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc >>>>>>>>>>>>>>>>> --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich >>>>>>>>>>>>>>>>> --download-mumps --download-hypre --download-superlu_dist --download-metis >>>>>>>>>>>>>>>>> --download-parmetis --download-scalapack >>>>>>>>>>>>>>>>> [2]PETSC ERROR: #1 VecValidValues() line 34 in >>>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c >>>>>>>>>>>>>>>>> [2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>>>>>>>>> [1]PETSC ERROR: #2 PCApply() line 442 in >>>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c >>>>>>>>>>>>>>>>> [2]PETSC ERROR: #3 KSP_PCApply() line 230 in >>>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>>>>>>>>> #3 KSP_PCApply() line 230 in >>>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h >>>>>>>>>>>>>>>>> [1]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>>>>>>>>> [2]PETSC ERROR: #4 KSPInitialResidual() line 63 in >>>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c >>>>>>>>>>>>>>>>> [1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>>>>>>>> [2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in >>>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>>>>>>>>> [2]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>>>>>>>> [1]PETSC ERROR: #6 KSPSolve() line 459 in >>>>>>>>>>>>>>>>> /home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>>>>>>>> ^C[mpiexec at nwmop] Sending Ctrl-C to processes as requested >>>>>>>>>>>>>>>>> [mpiexec at nwmop] Press Ctrl-C again to force abort >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> > From jchang27 at uh.edu Wed Apr 29 17:53:27 2015 From: jchang27 at uh.edu (Justin Chang) Date: Wed, 29 Apr 2015 17:53:27 -0500 Subject: [petsc-users] Tao iterations In-Reply-To: References: <3112142f40d64968ae157f4a5e819353@LUCKMAN.anl.gov> Message-ID: Okay that's what I figured, thanks you very much On Wed, Apr 29, 2015 at 10:39 AM, Jason Sarich wrote: > Hi Justin, > > This expected behavior due to the accumulation of numerical round-offs. If > this is a problem or if you just want to confirm that this is the cause, > you can try configuring PETSc for quad precision > (--with-precision=__float128, works with GNU compilers) and the results > should match better. > > Jason > > > On Tue, Apr 28, 2015 at 10:19 PM, Justin Chang wrote: > >> Jason (or anyone), >> >> I am noticing that the iteration numbers reported by >> TaoGetSolutionStatus() for blmvm differ whenever I change the number of >> processes. The solution seems to remain the same though. Is there a reason >> why this could be happening? >> >> Thanks, >> >> On Tue, Apr 21, 2015 at 10:40 AM, Jason Sarich >> wrote: >> >>> Justin, >>> >>> 1) The big difference between TRON and BLMVM is that TRON requires >>> hessian information, BLMVM only uses gradient information. Thus TRON will >>> usually converge faster, but requires more information, memory, and a KSP >>> solver. GPCG (gradient projected conjugate gradient) is another >>> gradient-only option, but usually performs worse than BLMVM. >>> >>> 2) TaoGetLinearSolveIterations() will get the total number of KSP >>> iterations per solve >>> >>> Jason >>> >>> >>> On Tue, Apr 21, 2015 at 10:33 AM, Justin Chang >>> wrote: >>> >>>> Jason, >>>> >>>> Tightening the tolerances did the trick. Thanks. Though I do have a >>>> couple more related questions: >>>> >>>> 1) Is there a general guideline for choosing tron over blmvm or vice >>>> versa? Also is there another tao type that is also suitable given only >>>> bounded constraints? >>>> >>>> 2) Is it possible to obtain the total number of KSP and/or PG >>>> iterations from tron? >>>> >>>> Thanks, >>>> Justin >>>> >>>> On Tue, Apr 21, 2015 at 9:52 AM, Jason Sarich >>>> wrote: >>>> >>>>> Hi Justin, >>>>> >>>>> blmvm believes that it is already sufficiently close to a minimum, so >>>>> it doesn't do anything. You may need to tighten some of the tolerance to >>>>> force an iteration. >>>>> >>>>> Jason >>>>> >>>>> >>>>> On Tue, Apr 21, 2015 at 9:48 AM, Justin Chang >>>>> wrote: >>>>> >>>>>> Time step 1: >>>>>> >>>>>> Tao Object: 1 MPI processes >>>>>> type: blmvm >>>>>> Gradient steps: 0 >>>>>> TaoLineSearch Object: 1 MPI processes >>>>>> type: more-thuente >>>>>> Active Set subset type: subvec >>>>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>>>> Residual in Function/Gradient:=0.0663148 >>>>>> Objective value=-55.5945 >>>>>> total number of iterations=35, (max: 2000) >>>>>> total number of function/gradient evaluations=37, (max: 4000) >>>>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>>>> >>>>>> Time step 2: >>>>>> >>>>>> Tao Object: 1 MPI processes >>>>>> type: blmvm >>>>>> Gradient steps: 0 >>>>>> TaoLineSearch Object: 1 MPI processes >>>>>> type: more-thuente >>>>>> Active Set subset type: subvec >>>>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>>>> Residual in Function/Gradient:=0.0682307 >>>>>> Objective value=-66.9675 >>>>>> total number of iterations=23, (max: 2000) >>>>>> total number of function/gradient evaluations=25, (max: 4000) >>>>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>>>> >>>>>> Time step 3: >>>>>> >>>>>> Tao Object: 1 MPI processes >>>>>> type: blmvm >>>>>> Gradient steps: 0 >>>>>> TaoLineSearch Object: 1 MPI processes >>>>>> type: more-thuente >>>>>> Active Set subset type: subvec >>>>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>>>> Residual in Function/Gradient:=0.0680522 >>>>>> Objective value=-71.8211 >>>>>> total number of iterations=19, (max: 2000) >>>>>> total number of function/gradient evaluations=22, (max: 4000) >>>>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>>>> >>>>>> Time step 4: >>>>>> >>>>>> Tao Object: 1 MPI processes >>>>>> type: blmvm >>>>>> Gradient steps: 0 >>>>>> TaoLineSearch Object: 1 MPI processes >>>>>> type: more-thuente >>>>>> Active Set subset type: subvec >>>>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>>>> Residual in Function/Gradient:=0.0551556 >>>>>> Objective value=-75.1252 >>>>>> total number of iterations=18, (max: 2000) >>>>>> total number of function/gradient evaluations=20, (max: 4000) >>>>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>>>> >>>>>> Time step 5: >>>>>> >>>>>> Tao Object: 1 MPI processes >>>>>> type: blmvm >>>>>> Gradient steps: 0 >>>>>> TaoLineSearch Object: 1 MPI processes >>>>>> type: more-thuente >>>>>> Active Set subset type: subvec >>>>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>>>> Residual in Function/Gradient:=0.0675667 >>>>>> Objective value=-77.4414 >>>>>> total number of iterations=6, (max: 2000) >>>>>> total number of function/gradient evaluations=8, (max: 4000) >>>>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>>>> >>>>>> Time step 6: >>>>>> >>>>>> Tao Object: 1 MPI processes >>>>>> type: blmvm >>>>>> Gradient steps: 0 >>>>>> TaoLineSearch Object: 1 MPI processes >>>>>> type: more-thuente >>>>>> Active Set subset type: subvec >>>>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>>>> Residual in Function/Gradient:=0.059143 >>>>>> Objective value=-79.5007 >>>>>> total number of iterations=3, (max: 2000) >>>>>> total number of function/gradient evaluations=5, (max: 4000) >>>>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>>>> >>>>>> Time step 7: >>>>>> >>>>>> Tao Object: 1 MPI processes >>>>>> type: blmvm >>>>>> Gradient steps: 0 >>>>>> TaoLineSearch Object: 1 MPI processes >>>>>> type: more-thuente >>>>>> Active Set subset type: subvec >>>>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>>>> Residual in Function/Gradient:=0.0433683 >>>>>> Objective value=-81.3546 >>>>>> total number of iterations=5, (max: 2000) >>>>>> total number of function/gradient evaluations=8, (max: 4000) >>>>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>>>> >>>>>> Time step 8: >>>>>> >>>>>> Tao Object: 1 MPI processes >>>>>> type: blmvm >>>>>> Gradient steps: 0 >>>>>> TaoLineSearch Object: 1 MPI processes >>>>>> type: more-thuente >>>>>> Active Set subset type: subvec >>>>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>>>> Residual in Function/Gradient:=0.0840676 >>>>>> Objective value=-82.9382 >>>>>> total number of iterations=0, (max: 2000) >>>>>> total number of function/gradient evaluations=1, (max: 4000) >>>>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>>>> >>>>>> Time step 9: >>>>>> >>>>>> Tao Object: 1 MPI processes >>>>>> type: blmvm >>>>>> Gradient steps: 0 >>>>>> TaoLineSearch Object: 1 MPI processes >>>>>> type: more-thuente >>>>>> Active Set subset type: subvec >>>>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>>>> Residual in Function/Gradient:=0.0840676 >>>>>> Objective value=-82.9382 >>>>>> total number of iterations=0, (max: 2000) >>>>>> total number of function/gradient evaluations=1, (max: 4000) >>>>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>>>> >>>>>> Time step 10: >>>>>> >>>>>> Tao Object: 1 MPI processes >>>>>> type: blmvm >>>>>> Gradient steps: 0 >>>>>> TaoLineSearch Object: 1 MPI processes >>>>>> type: more-thuente >>>>>> Active Set subset type: subvec >>>>>> convergence tolerances: fatol=0.0001, frtol=0.0001 >>>>>> convergence tolerances: gatol=0, steptol=0, gttol=0 >>>>>> Residual in Function/Gradient:=0.0840676 >>>>>> Objective value=-82.9382 >>>>>> total number of iterations=0, (max: 2000) >>>>>> total number of function/gradient evaluations=1, (max: 4000) >>>>>> Solution converged: estimated |f(x)-f(X*)|/|f(X*)| <= frtol >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Apr 21, 2015 at 9:28 AM, Jason Sarich >>>>> > wrote: >>>>>> >>>>>>> Hi Justin, >>>>>>> >>>>>>> what reason is blmvm giving for stopping the solve? (you can use >>>>>>> -tao_view or -tao_converged_reason to get this) >>>>>>> >>>>>>> Jason >>>>>>> >>>>>>> On Mon, Apr 20, 2015 at 6:32 PM, Justin Chang >>>>>>> wrote: >>>>>>> >>>>>>>> Jason, >>>>>>>> >>>>>>>> I am using TaoGetSolutionStatus(tao,&its, ...) and it gives me >>>>>>>> exactly what I want. However, I seem to be having an issue with blmvm >>>>>>>> >>>>>>>> I wrote my own backward euler code for a transient linear >>>>>>>> diffusion problem with lower bounds >= 0 and upper bounds <= 1. For the >>>>>>>> first several time steps I am getting its > 0, and it decreases over time >>>>>>>> due to the nature of the discrete maximum principles. However, at some >>>>>>>> point my its become 0 and the solution does not "update", which seems to me >>>>>>>> that TaoSolve is not doing anything after that. This doesn't happen if I >>>>>>>> were to use tron (my KSP and PC are cg and jacobi respectively). >>>>>>>> >>>>>>>> Do you know why this behavior may occur? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> On Tue, Apr 14, 2015 at 9:35 AM, Jason Sarich < >>>>>>>> jason.sarich at gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi Justin, >>>>>>>>> >>>>>>>>> I have pushed these changes to the "next" branch, your code >>>>>>>>> snippet should work fine there. >>>>>>>>> >>>>>>>>> Note that there is also available (since version 3.5.0) the >>>>>>>>> routine TaoGetSolutionStatus(tao,&its,NULL,NULL,NULL,NULL,NULL) which will >>>>>>>>> provide the >>>>>>>>> same information >>>>>>>>> >>>>>>>>> Jason >>>>>>>>> >>>>>>>>> On Fri, Apr 10, 2015 at 6:28 PM, Justin Chang >>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> Whatever is convenient and/or follow the "PETSc" standards. >>>>>>>>>> Something similar to SNESGetIterationNumber() or KSPGetIterationNumber() >>>>>>>>>> would be nice. Ideally I want my code to look like this: >>>>>>>>>> >>>>>>>>>> ierr = TaoGetIterationNumber(tao,&its);CHKERRQ(ierr); >>>>>>>>>> ierr = PetscPrintf(PETSC_COMM_WORLD, "Number of Tao iterations = >>>>>>>>>> %D\n", its); >>>>>>>>>> >>>>>>>>>> Thanks :) >>>>>>>>>> >>>>>>>>>> On Fri, Apr 10, 2015 at 5:53 PM, Jason Sarich < >>>>>>>>>> jason.sarich at gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Justin, I'll get this in. I assume that displaying the number >>>>>>>>>>> of iterations with tao_converged_reason is what you are asking for in >>>>>>>>>>> particular? Or did you have something else in mind? >>>>>>>>>>> >>>>>>>>>>> Jason >>>>>>>>>>> On Apr 10, 2015 16:42, "Smith, Barry F." >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Justin, >>>>>>>>>>>> >>>>>>>>>>>> Sorry TAO simply doesn't even collect this information >>>>>>>>>>>> currently. But yes we should definitely make it available! >>>>>>>>>>>> >>>>>>>>>>>> Jason, >>>>>>>>>>>> >>>>>>>>>>>> Could you please add this; almost all the TaoSolve_xxx() >>>>>>>>>>>> have a local variable iter; change that to tao->niter (I'm guess this is >>>>>>>>>>>> suppose to capture this information) and add a TaoGetIterationNumber() and >>>>>>>>>>>> the uses can access this. Also modify at the end of TaoSolve() >>>>>>>>>>>> -tao_converged_reason to also print the iteration count. At the same time >>>>>>>>>>>> since you add this you can add a tao->totalits which would accumulate all >>>>>>>>>>>> iterations over all the solves for that Tao object and the routine >>>>>>>>>>>> TaoGetTotalIterations() to access this. Note that TaoSolve() would >>>>>>>>>>>> initialize tao->niter = 0 at the top. >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> >>>>>>>>>>>> Barry >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > On Apr 10, 2015, at 4:16 PM, Justin Chang >>>>>>>>>>>> wrote: >>>>>>>>>>>> > >>>>>>>>>>>> > Hi all, >>>>>>>>>>>> > >>>>>>>>>>>> > Is there a way to generically obtain the number of Tao >>>>>>>>>>>> iterations? I am looking through the -help options for Tao and I don't see >>>>>>>>>>>> any metric where you can output this quantity in the manner that you could >>>>>>>>>>>> for SNES or KSP solves. I am currently using blmvm and tron, and the only >>>>>>>>>>>> way I can see getting this metric is by outputting -tao_view and/or >>>>>>>>>>>> -tao_monitor and manually finding this number. I find this cumbersome >>>>>>>>>>>> especially for transient problems where I would like to simply have this >>>>>>>>>>>> number printed for each step instead of ending up with unnecessary info. >>>>>>>>>>>> > >>>>>>>>>>>> > Thanks, >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > -- >>>>>>>>>>>> > Justin Chang >>>>>>>>>>>> > PhD Candidate, Civil Engineering - Computational Sciences >>>>>>>>>>>> > University of Houston, Department of Civil and Environmental >>>>>>>>>>>> Engineering >>>>>>>>>>>> > Houston, TX 77004 >>>>>>>>>>>> > (512) 963-3262 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >> >> -- >> Justin Chang >> PhD Candidate, Civil Engineering - Computational Sciences >> University of Houston, Department of Civil and Environmental Engineering >> Houston, TX 77004 >> (512) 963-3262 >> > > -- Justin Chang PhD Candidate, Civil Engineering - Computational Sciences University of Houston, Department of Civil and Environmental Engineering Houston, TX 77004 (512) 963-3262 -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Thu Apr 30 05:02:27 2015 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Thu, 30 Apr 2015 10:02:27 +0000 Subject: [petsc-users] PETSC_NULL_OBJECT gets corrupt after call to MatNestGetISs in fortran In-Reply-To: References: <1429875994856.10134@marin.nl>, Message-ID: <1430388147149.79103@marin.nl> Satish, Thanks for the files! I've copied them to the correct location in my petsc-3.5.3 dir and rebuild the whole thing from scratch but I still get the same memory corruption for the example below... Chris ________________________________________ From: Satish Balay Sent: Friday, April 24, 2015 10:54 PM To: Klaij, Christiaan Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] PETSC_NULL_OBJECT gets corrupt after call to MatNestGetISs in fortran Sorry for dropping the ball on this issue. I pushed the following fix to maint branch. https://bitbucket.org/petsc/petsc/commits/3a4d7b9a6c83003720b45dc0635fc32ea52a4309 To use this change with petsc-3.5.3 - you can drop in the attached replacement files at: src/mat/impls/nest/ftn-custom/zmatnestf.c src/mat/impls/nest/ftn-auto/matnestf.c Satish On Fri, 24 Apr 2015, Klaij, Christiaan wrote: > Barry, Satish > > Any news on this issue? > > Chris > > > On Feb 12, 2015, at 07:13:08 CST, Smith, Barry wrote: > > > > Thanks for reporting this. Currently the Fortran stub for this function is generated automatically which means it does not have the logic for handling a PETSC_NULL_OBJECT argument. > > > > Satish, could you please see if you can add a custom fortran stub for this function in maint? > > > > Thanks > > > > Barry > > > > > On Feb 12, 2015, at 3:02 AM, Klaij, Christiaan wrote: > > > > > > Using petsc-3.5.3, I noticed that PETSC_NULL_OBJECT gets corrupt after calling MatNestGetISs in fortran. Here's a small example: > > > > > > $ cat fieldsplittry2.F90 > > > program fieldsplittry2 > > > > > > use petscksp > > > implicit none > > > #include > > > > > > PetscErrorCode :: ierr > > > PetscInt :: size,i,j,start,end,n=4,numsplit=1 > > > PetscScalar :: zero=0.0,one=1.0 > > > Vec :: diag3,x,b > > > Mat :: A,subA(4),myS > > > PC :: pc,subpc(2) > > > KSP :: ksp,subksp(2) > > > IS :: isg(2) > > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr); CHKERRQ(ierr) > > > call MPI_Comm_size(PETSC_COMM_WORLD,size,ierr); CHKERRQ(ierr); > > > > > > ! vectors > > > call VecCreateMPI(MPI_COMM_WORLD,3*n,PETSC_DECIDE,diag3,ierr); CHKERRQ(ierr) > > > call VecSet(diag3,one,ierr); CHKERRQ(ierr) > > > > > > call VecCreateMPI(MPI_COMM_WORLD,4*n,PETSC_DECIDE,x,ierr); CHKERRQ(ierr) > > > call VecSet(x,zero,ierr); CHKERRQ(ierr) > > > > > > call VecDuplicate(x,b,ierr); CHKERRQ(ierr) > > > call VecSet(b,one,ierr); CHKERRQ(ierr) > > > > > > ! matrix a00 > > > call MatCreateAIJ(MPI_COMM_WORLD,3*n,3*n,PETSC_DECIDE,PETSC_DECIDE,1,PETSC_NULL_INTEGER,0,PETSC_NULL_INTEGER,subA(1),ierr);CHKERRQ(ierr) > > > call MatDiagonalSet(subA(1),diag3,INSERT_VALUES,ierr);CHKERRQ(ierr) > > > call MatAssemblyBegin(subA(1),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > call MatAssemblyEnd(subA(1),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > > > > ! matrix a01 > > > call MatCreateAIJ(MPI_COMM_WORLD,3*n,n,PETSC_DECIDE,PETSC_DECIDE,1,PETSC_NULL_INTEGER,1,PETSC_NULL_INTEGER,subA(2),ierr);CHKERRQ(ierr) > > > call MatGetOwnershipRange(subA(2),start,end,ierr);CHKERRQ(ierr); > > > do i=start,end-1 > > > j=mod(i,size*n) > > > call MatSetValue(subA(2),i,j,one,INSERT_VALUES,ierr);CHKERRQ(ierr) > > > end do > > > call MatAssemblyBegin(subA(2),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > call MatAssemblyEnd(subA(2),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > > > > ! matrix a10 > > > call MatTranspose(subA(2),MAT_INITIAL_MATRIX,subA(3),ierr);CHKERRQ(ierr) > > > > > > ! matrix a11 (empty) > > > call MatCreateAIJ(MPI_COMM_WORLD,n,n,PETSC_DECIDE,PETSC_DECIDE,0,PETSC_NULL_INTEGER,0,PETSC_NULL_INTEGER,subA(4),ierr);CHKERRQ(ierr) > > > call MatAssemblyBegin(subA(4),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > call MatAssemblyEnd(subA(4),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > > > > ! nested mat [a00,a01;a10,a11] > > > call MatCreateNest(MPI_COMM_WORLD,2,PETSC_NULL_OBJECT,2,PETSC_NULL_OBJECT,subA,A,ierr);CHKERRQ(ierr) > > > call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > print *, PETSC_NULL_OBJECT > > > call MatNestGetISs(A,isg,PETSC_NULL_OBJECT,ierr);CHKERRQ(ierr); > > > print *, PETSC_NULL_OBJECT > > > > > > call PetscFinalize(ierr) > > > > > > end program fieldsplittry2 > > > $ ./fieldsplittry2 > > > 0 > > > 39367824 > > > $ > > > > > > > > > dr. ir. Christiaan Klaij > > > CFD Researcher > > > Research & Development > > > E mailto:C.Klaij at marin.nl > > > T +31 317 49 33 44 > > > > > > > > > MARIN > > > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > > > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > > > > > > dr. ir. Christiaan Klaij > CFD Researcher > Research & Development > E mailto:C.Klaij at marin.nl > T +31 317 49 33 44 > > > MARIN > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > > From danyang.su at gmail.com Thu Apr 30 13:21:49 2015 From: danyang.su at gmail.com (Danyang Su) Date: Thu, 30 Apr 2015 11:21:49 -0700 Subject: [petsc-users] Questions on 1 billion unknowns and 64-bit-indices Message-ID: <554272BD.7030807@gmail.com> Dear All, I have run my codes successfully with up to 100 million total unknowns using 1000 processors on WestGrid Jasper Cluster, Canada. But when I scale the unknows up to 1 billion, the codes crashes with the following error. It's out of memory. Error message from valgrind output ==9344== Invalid read of size 16 ==9344== at 0xADB2906: __intel_sse2_strdup (in /lustre/jasper/software/intel/l_ics_2013.0.028/composer\ _xe_2013.1.117/compiler/lib/intel64/libintlc.so.5) ==9344== by 0xE6: ??? ==9344== by 0xE7: ??? ==9344== by 0x5: ??? ==9344== Address 0xb364410 is 16 bytes inside a block of size 17 alloc'd ==9344== at 0x4A0638D: malloc (vg_replace_malloc.c:291) ==9344== by 0x3DE7C6807C: vasprintf (in /lib64/libc-2.5.so) ==9344== by 0x3DE7C4CBE7: asprintf (in /lib64/libc-2.5.so) ==9344== by 0x9DC511E: opal_output_init (output.c:144) ==9344== by 0x9DC042D: opal_init_util (opal_init.c:207) ==9344== by 0x9CF4EBB: ompi_mpi_init (ompi_mpi_init.c:309) ==9344== by 0x9D0D802: PMPI_Init (pinit.c:84) ==9344== by 0x905E976: PMPI_INIT (pinit_f.c:75) ==9344== by 0x4D5280F: petscinitialize_ (in /lustre/jasper/software/petsc/petsc-3.5.1/lib/libpetsc.so.\ 3.5.1) ==9344== by 0x439D05: petsc_mpi_common_mp_petsc_mpi_initialize_ (in /lustre/home/danyangs/benchmark/ba\ sin/min3p_thcm) ==9344== by 0x5FDBB9: MAIN__ (in /lustre/home/danyangs/benchmark/basin/min3p_thcm) ==9344== by 0x4321FB: main (in /lustre/home/danyangs/benchmark/basin/min3p_thcm) ==9344== Error message from Jasper Cluster output --32725:0:aspacem <<< SHOW_SEGMENTS: out_of_memory (407 segments, 96 segnames) --32725:0:aspacem ( 0) /lustre/jasper/software/valgrind/valgrind-3.9.0/lib/valgrind/memcheck-amd64-linux --32725:0:aspacem ( 1) /lustre/home/danyangs/benchmark/basin/min3p_thcm --32725:0:aspacem ( 2) /lib64/ld-2.5.so --32725:0:aspacem ( 3) /data2/PBStmp/6456165.jasper-usradm.westgrid.ca/vgdb-pipe-shared-mem-vgdb-32725-b\ y-danyangs-on-cl2n050 --32725:0:aspacem ( 4) /lustre/jasper/software/valgrind/valgrind-3.9.0/lib/valgrind/vgpreload_core-amd64\ -linux.so --32725:0:aspacem ( 5) /lustre/jasper/software/valgrind/valgrind-3.9.0/lib/valgrind/vgpreload_memcheck-a\ md64-linux.so --32725:0:aspacem ( 6) /lustre/jasper/software/petsc/petsc-3.5.1/lib/libpetsc.so.3.5.1 --32725:0:aspacem ( 7) /lustre/jasper/software/openmpi/openmpi-1.6.5-intel/lib/libmpi_cxx.so.1.0.2 --32725:0:aspacem ( 8) /lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/mkl/lib/int\ el64/libmkl_scalapack_lp64.so --32725:0:aspacem ( 9) /lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/mkl/lib/int\ el64/libmkl_intel_lp64.so --32725:0:aspacem (10) /lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/mkl/lib/int\ el64/libmkl_sequential.so --32725:0:aspacem (11) /lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/mkl/lib/int\ el64/libmkl_core.so --32725:0:aspacem (12) /lustre/jasper/software/petsc/petsc-3.5.1/lib/libparmetis.so --32725:0:aspacem (13) /lustre/jasper/software/petsc/petsc-3.5.1/lib/libmetis.so --32725:0:aspacem (14) /lustre/jasper/software/openmpi/openmpi-1.6.5-intel/lib/openmpi/mca_paffinity_hwl\ oc.so --32725:0:aspacem (15) /usr/lib64/libX11.so.6.2.0 --32725:0:aspacem (16) /lib64/libpthread-2.5.so --32725:0:aspacem (17) /lib64/libssl.so.0.9.8e --32725:0:aspacem (18) /lib64/libcrypto.so.0.9.8e --32725:0:aspacem (19) /lustre/jasper/software/openmpi/openmpi-1.6.5-intel/lib/libmpi_f90.so.1.3.0 --32725:0:aspacem (20) /lustre/jasper/software/openmpi/openmpi-1.6.5-intel/lib/libmpi_f77.so.1.0.7 --32725:0:aspacem (21) /lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/compiler/li\ b/intel64/libimf.so The PETSc configuration is as below ================================================================================ Starting Configure Run at Tue Dec 16 10:42:20 2014 Configure Options: --configModules=PETSc.Configure --optionsModule=PETSc.compilerOptions --prefix=/global\ /software/petsc/petsc-3.5.1 --with-shared-libraries --with-mpirun=mpiexec --with-vendor-compiler=intel --\ with-blas-lapack-lib=-mkl=sequential --with-cc=mpicc --with-cxx=mpiCC --with-fc=mpif90 --COPTFLAGS=-O2 --\ CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-debugging=no --with-blacs=yes --with-blacs-include=/lustre/jasper/\ software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/mkl/include --with-blacs-lib=/lustre/jasper/softwa\ re/intel/l_ics_2013.0.028/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.a --with-scala\ pack=yes --with-scalapack-include=/lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/m\ kl/include --with-scalapack-lib="-L/lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/\ mkl/lib/intel64/ -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64" --download-metis=metis-5.0.2-p3.tar.gz --\ download-parmetis=yes --download-superlu_dist=yes --download-hypre=yes Working directory: /lustre/jasper/software-build/petsc/petsc-3.5.1 Machine platform: ('Linux', 'jasper.westgrid.ca', '2.6.18-274.el5', '#1 SMP Fri Jul 22 04:43:29 EDT 2011', 'x86_64', 'x86_6\ 4') Python version: 2.4.3 (#1, Sep 21 2011, 19:55:41) [GCC 4.1.2 20080704 (Red Hat 4.1.2-51)] ================================================================================ My case does not hold the conditions of 64-bit-indices as below. By default the type that PETSc uses to index into arrays and keep sizes of arrays is a PetscInt defined to be a 32 bit int. If your problem * involves more than 2^31 - 1 unknowns (around 2 billion) OR * your matrix might contain more than 2^31 - 1 nonzeros on a single process then you need to use this option. Otherwise you will get strange crashes. Do you guys have suggestions on this? Thanks and regards, Danyang -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Apr 30 13:29:13 2015 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 1 May 2015 04:29:13 +1000 Subject: [petsc-users] Questions on 1 billion unknowns and 64-bit-indices In-Reply-To: <554272BD.7030807@gmail.com> References: <554272BD.7030807@gmail.com> Message-ID: On Fri, May 1, 2015 at 4:21 AM, Danyang Su wrote: > Dear All, > > I have run my codes successfully with up to 100 million total unknowns > using 1000 processors on WestGrid Jasper Cluster, Canada. But when I scale > the unknows up to 1 billion, the codes crashes with the following error. > It's out of memory. > If you are running out of memory, you need to use more processors. Thanks, Matt > Error message from valgrind output > > ==9344== Invalid read of size 16 > ==9344== at 0xADB2906: __intel_sse2_strdup (in > /lustre/jasper/software/intel/l_ics_2013.0.028/composer\ > _xe_2013.1.117/compiler/lib/intel64/libintlc.so.5) > ==9344== by 0xE6: ??? > ==9344== by 0xE7: ??? > ==9344== by 0x5: ??? > ==9344== Address 0xb364410 is 16 bytes inside a block of size 17 alloc'd > ==9344== at 0x4A0638D: malloc (vg_replace_malloc.c:291) > ==9344== by 0x3DE7C6807C: vasprintf (in /lib64/libc-2.5.so) > ==9344== by 0x3DE7C4CBE7: asprintf (in /lib64/libc-2.5.so) > ==9344== by 0x9DC511E: opal_output_init (output.c:144) > ==9344== by 0x9DC042D: opal_init_util (opal_init.c:207) > ==9344== by 0x9CF4EBB: ompi_mpi_init (ompi_mpi_init.c:309) > ==9344== by 0x9D0D802: PMPI_Init (pinit.c:84) > ==9344== by 0x905E976: PMPI_INIT (pinit_f.c:75) > ==9344== by 0x4D5280F: petscinitialize_ (in > /lustre/jasper/software/petsc/petsc-3.5.1/lib/libpetsc.so.\ > 3.5.1) > ==9344== by 0x439D05: petsc_mpi_common_mp_petsc_mpi_initialize_ (in > /lustre/home/danyangs/benchmark/ba\ > sin/min3p_thcm) > ==9344== by 0x5FDBB9: MAIN__ (in > /lustre/home/danyangs/benchmark/basin/min3p_thcm) > ==9344== by 0x4321FB: main (in > /lustre/home/danyangs/benchmark/basin/min3p_thcm) > ==9344== > > Error message from Jasper Cluster output > --32725:0:aspacem <<< SHOW_SEGMENTS: out_of_memory (407 segments, 96 > segnames) > --32725:0:aspacem ( 0) > /lustre/jasper/software/valgrind/valgrind-3.9.0/lib/valgrind/memcheck-amd64-linux > --32725:0:aspacem ( 1) /lustre/home/danyangs/benchmark/basin/min3p_thcm > --32725:0:aspacem ( 2) /lib64/ld-2.5.so > --32725:0:aspacem ( 3) /data2/PBStmp/ > 6456165.jasper-usradm.westgrid.ca/vgdb-pipe-shared-mem-vgdb-32725-b\ > > y-danyangs-on-cl2n050 > --32725:0:aspacem ( 4) > /lustre/jasper/software/valgrind/valgrind-3.9.0/lib/valgrind/vgpreload_core-amd64\ > -linux.so > --32725:0:aspacem ( 5) > /lustre/jasper/software/valgrind/valgrind-3.9.0/lib/valgrind/vgpreload_memcheck-a\ > md64-linux.so > --32725:0:aspacem ( 6) > /lustre/jasper/software/petsc/petsc-3.5.1/lib/libpetsc.so.3.5.1 > --32725:0:aspacem ( 7) > /lustre/jasper/software/openmpi/openmpi-1.6.5-intel/lib/libmpi_cxx.so.1.0.2 > --32725:0:aspacem ( 8) > /lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/mkl/lib/int\ > el64/libmkl_scalapack_lp64.so > --32725:0:aspacem ( 9) > /lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/mkl/lib/int\ > el64/libmkl_intel_lp64.so > --32725:0:aspacem (10) > /lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/mkl/lib/int\ > el64/libmkl_sequential.so > --32725:0:aspacem (11) > /lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/mkl/lib/int\ > el64/libmkl_core.so > --32725:0:aspacem (12) > /lustre/jasper/software/petsc/petsc-3.5.1/lib/libparmetis.so > --32725:0:aspacem (13) > /lustre/jasper/software/petsc/petsc-3.5.1/lib/libmetis.so > --32725:0:aspacem (14) > /lustre/jasper/software/openmpi/openmpi-1.6.5-intel/lib/openmpi/mca_paffinity_hwl\ > oc.so > --32725:0:aspacem (15) /usr/lib64/libX11.so.6.2.0 > --32725:0:aspacem (16) /lib64/libpthread-2.5.so > --32725:0:aspacem (17) /lib64/libssl.so.0.9.8e > --32725:0:aspacem (18) /lib64/libcrypto.so.0.9.8e > --32725:0:aspacem (19) > /lustre/jasper/software/openmpi/openmpi-1.6.5-intel/lib/libmpi_f90.so.1.3.0 > --32725:0:aspacem (20) > /lustre/jasper/software/openmpi/openmpi-1.6.5-intel/lib/libmpi_f77.so.1.0.7 > --32725:0:aspacem (21) > /lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/compiler/li\ > b/intel64/libimf.so > > The PETSc configuration is as below > > > ================================================================================ > Starting Configure Run at Tue Dec 16 10:42:20 2014 > Configure Options: --configModules=PETSc.Configure > --optionsModule=PETSc.compilerOptions --prefix=/global\ > /software/petsc/petsc-3.5.1 --with-shared-libraries --with-mpirun=mpiexec > --with-vendor-compiler=intel --\ > with-blas-lapack-lib=-mkl=sequential --with-cc=mpicc --with-cxx=mpiCC > --with-fc=mpif90 --COPTFLAGS=-O2 --\ > CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-debugging=no --with-blacs=yes > --with-blacs-include=/lustre/jasper/\ > software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/mkl/include > --with-blacs-lib=/lustre/jasper/softwa\ > re/intel/l_ics_2013.0.028/composer_xe_2013.1.117/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.a > --with-scala\ > pack=yes > --with-scalapack-include=/lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/m\ > kl/include > --with-scalapack-lib="-L/lustre/jasper/software/intel/l_ics_2013.0.028/composer_xe_2013.1.117/\ > mkl/lib/intel64/ -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64" > --download-metis=metis-5.0.2-p3.tar.gz --\ > download-parmetis=yes --download-superlu_dist=yes --download-hypre=yes > Working directory: /lustre/jasper/software-build/petsc/petsc-3.5.1 > Machine platform: > ('Linux', 'jasper.westgrid.ca', '2.6.18-274.el5', '#1 SMP Fri Jul 22 > 04:43:29 EDT 2011', 'x86_64', 'x86_6\ > 4') > Python version: > 2.4.3 (#1, Sep 21 2011, 19:55:41) > [GCC 4.1.2 20080704 (Red Hat 4.1.2-51)] > > ================================================================================ > > My case does not hold the conditions of 64-bit-indices as below. > By default the type that PETSc uses to index into arrays and keep sizes of > arrays is a PetscInt defined to be a 32 bit int. If your problem > > - involves more than 2^31 - 1 unknowns (around 2 billion) OR > - your matrix might contain more than 2^31 - 1 nonzeros on a single > process > > then you need to use this option. Otherwise you will get strange crashes. > > Do you guys have suggestions on this? > > Thanks and regards, > > Danyang > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Apr 30 14:54:15 2015 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 30 Apr 2015 14:54:15 -0500 Subject: [petsc-users] PETSC_NULL_OBJECT gets corrupt after call to MatNestGetISs in fortran In-Reply-To: <1430388147149.79103@marin.nl> References: <1429875994856.10134@marin.nl>, <1430388147149.79103@marin.nl> Message-ID: Hm - it works for me with the test code you sent.. balay at asterix /home/balay/download-pine/x $ ./ex1f 0 40357424 balay at asterix /home/balay/download-pine/x $ ./ex1f 0 0 balay at asterix /home/balay/download-pine/x Satish $ On Thu, 30 Apr 2015, Klaij, Christiaan wrote: > Satish, > > Thanks for the files! I've copied them to the correct location in my petsc-3.5.3 dir and rebuild the whole thing from scratch but I still get the same memory corruption for the example below... > > Chris > ________________________________________ > From: Satish Balay > Sent: Friday, April 24, 2015 10:54 PM > To: Klaij, Christiaan > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] PETSC_NULL_OBJECT gets corrupt after call to MatNestGetISs in fortran > > Sorry for dropping the ball on this issue. I pushed the following fix to maint branch. > > https://bitbucket.org/petsc/petsc/commits/3a4d7b9a6c83003720b45dc0635fc32ea52a4309 > > To use this change with petsc-3.5.3 - you can drop in the attached replacement files at: > > src/mat/impls/nest/ftn-custom/zmatnestf.c > src/mat/impls/nest/ftn-auto/matnestf.c > > Satish > > On Fri, 24 Apr 2015, Klaij, Christiaan wrote: > > > Barry, Satish > > > > Any news on this issue? > > > > Chris > > > > > On Feb 12, 2015, at 07:13:08 CST, Smith, Barry wrote: > > > > > > Thanks for reporting this. Currently the Fortran stub for this function is generated automatically which means it does not have the logic for handling a PETSC_NULL_OBJECT argument. > > > > > > Satish, could you please see if you can add a custom fortran stub for this function in maint? > > > > > > Thanks > > > > > > Barry > > > > > > > On Feb 12, 2015, at 3:02 AM, Klaij, Christiaan wrote: > > > > > > > > Using petsc-3.5.3, I noticed that PETSC_NULL_OBJECT gets corrupt after calling MatNestGetISs in fortran. Here's a small example: > > > > > > > > $ cat fieldsplittry2.F90 > > > > program fieldsplittry2 > > > > > > > > use petscksp > > > > implicit none > > > > #include > > > > > > > > PetscErrorCode :: ierr > > > > PetscInt :: size,i,j,start,end,n=4,numsplit=1 > > > > PetscScalar :: zero=0.0,one=1.0 > > > > Vec :: diag3,x,b > > > > Mat :: A,subA(4),myS > > > > PC :: pc,subpc(2) > > > > KSP :: ksp,subksp(2) > > > > IS :: isg(2) > > > > > > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr); CHKERRQ(ierr) > > > > call MPI_Comm_size(PETSC_COMM_WORLD,size,ierr); CHKERRQ(ierr); > > > > > > > > ! vectors > > > > call VecCreateMPI(MPI_COMM_WORLD,3*n,PETSC_DECIDE,diag3,ierr); CHKERRQ(ierr) > > > > call VecSet(diag3,one,ierr); CHKERRQ(ierr) > > > > > > > > call VecCreateMPI(MPI_COMM_WORLD,4*n,PETSC_DECIDE,x,ierr); CHKERRQ(ierr) > > > > call VecSet(x,zero,ierr); CHKERRQ(ierr) > > > > > > > > call VecDuplicate(x,b,ierr); CHKERRQ(ierr) > > > > call VecSet(b,one,ierr); CHKERRQ(ierr) > > > > > > > > ! matrix a00 > > > > call MatCreateAIJ(MPI_COMM_WORLD,3*n,3*n,PETSC_DECIDE,PETSC_DECIDE,1,PETSC_NULL_INTEGER,0,PETSC_NULL_INTEGER,subA(1),ierr);CHKERRQ(ierr) > > > > call MatDiagonalSet(subA(1),diag3,INSERT_VALUES,ierr);CHKERRQ(ierr) > > > > call MatAssemblyBegin(subA(1),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > > call MatAssemblyEnd(subA(1),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > > > > > > ! matrix a01 > > > > call MatCreateAIJ(MPI_COMM_WORLD,3*n,n,PETSC_DECIDE,PETSC_DECIDE,1,PETSC_NULL_INTEGER,1,PETSC_NULL_INTEGER,subA(2),ierr);CHKERRQ(ierr) > > > > call MatGetOwnershipRange(subA(2),start,end,ierr);CHKERRQ(ierr); > > > > do i=start,end-1 > > > > j=mod(i,size*n) > > > > call MatSetValue(subA(2),i,j,one,INSERT_VALUES,ierr);CHKERRQ(ierr) > > > > end do > > > > call MatAssemblyBegin(subA(2),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > > call MatAssemblyEnd(subA(2),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > > > > > > ! matrix a10 > > > > call MatTranspose(subA(2),MAT_INITIAL_MATRIX,subA(3),ierr);CHKERRQ(ierr) > > > > > > > > ! matrix a11 (empty) > > > > call MatCreateAIJ(MPI_COMM_WORLD,n,n,PETSC_DECIDE,PETSC_DECIDE,0,PETSC_NULL_INTEGER,0,PETSC_NULL_INTEGER,subA(4),ierr);CHKERRQ(ierr) > > > > call MatAssemblyBegin(subA(4),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > > call MatAssemblyEnd(subA(4),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > > > > > > ! nested mat [a00,a01;a10,a11] > > > > call MatCreateNest(MPI_COMM_WORLD,2,PETSC_NULL_OBJECT,2,PETSC_NULL_OBJECT,subA,A,ierr);CHKERRQ(ierr) > > > > call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > > call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr) > > > > print *, PETSC_NULL_OBJECT > > > > call MatNestGetISs(A,isg,PETSC_NULL_OBJECT,ierr);CHKERRQ(ierr); > > > > print *, PETSC_NULL_OBJECT > > > > > > > > call PetscFinalize(ierr) > > > > > > > > end program fieldsplittry2 > > > > $ ./fieldsplittry2 > > > > 0 > > > > 39367824 > > > > $ > > > > > > > > > > > > dr. ir. Christiaan Klaij > > > > CFD Researcher > > > > Research & Development > > > > E mailto:C.Klaij at marin.nl > > > > T +31 317 49 33 44 > > > > > > > > > > > > MARIN > > > > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > > > > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > > > > > > > > > > dr. ir. Christiaan Klaij > > CFD Researcher > > Research & Development > > E mailto:C.Klaij at marin.nl > > T +31 317 49 33 44 > > > > > > MARIN > > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > > > > >