From jchludzinski at gmail.com Mon Aug 1 12:27:49 2011 From: jchludzinski at gmail.com (John Chludzinski) Date: Mon, 1 Aug 2011 13:27:49 -0400 Subject: [petsc-users] SLEPc generalized eigenvalue problem question? Message-ID: I create 2 matrices using: MatCreateSeqDense(PETSC_COMM_SELF, n, n, Ka, &A); MatCreateSeqDense(PETSC_COMM_SELF, n, n, Kb, &B); These matrices are 99% zeros ( 16,016,004 entries and 18660 non-zeros). They are symmetric and real. Their tri-diagonal elements are non-zero plus a few other entries. I tried to use ex7 for the generalized eigenvalue problem: ./ex7.exe -f1 k.dat -f2 m.dat -eps_gen_hermitian -eps_smallest_real > x.out 2>&1 without specifying an EPS and get: Generalized eigenproblem stored in file. Reading REAL matrices from binary files... Number of iterations of the method: 500 Number of linear iterations of the method: 4009 Solution method: krylovschur Number of requested eigenvalues: 1 Stopping condition: tol=1e-07, maxit=500 Number of converged approximate eigenpairs: 0 Is krylovschur inappropriate for this problem or have I set up the problem incorrectly by using MatCreateSeqDense(...) to create the matrix input files in PETSc binary form? ---John -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Aug 1 12:40:10 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 1 Aug 2011 17:40:10 +0000 Subject: [petsc-users] SLEPc generalized eigenvalue problem question? In-Reply-To: References: Message-ID: On Mon, Aug 1, 2011 at 5:27 PM, John Chludzinski wrote: > I create 2 matrices using: > > MatCreateSeqDense(PETSC_COMM_SELF, n, n, Ka, &A); > MatCreateSeqDense(PETSC_COMM_SELF, n, n, Kb, &B); > > These matrices are 99% zeros ( 16,016,004 entries and 18660 non-zeros). > They are symmetric and real. Their tri-diagonal elements are non-zero plus > a few other entries. > Please give some justification for doing this? On the surface, it just seems perverse. Matt > I tried to use ex7 for the generalized eigenvalue problem: > > ./ex7.exe -f1 k.dat -f2 m.dat -eps_gen_hermitian -eps_smallest_real > x.out > 2>&1 > > without specifying an EPS and get: > > Generalized eigenproblem stored in file. > > Reading REAL matrices from binary files... > Number of iterations of the method: 500 > Number of linear iterations of the method: 4009 > Solution method: krylovschur > > Number of requested eigenvalues: 1 > Stopping condition: tol=1e-07, maxit=500 > Number of converged approximate eigenpairs: 0 > > Is krylovschur inappropriate for this problem or have I set up the problem > incorrectly by using MatCreateSeqDense(...) to create the matrix input > files in PETSc binary form? > > ---John > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon Aug 1 12:46:56 2011 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 1 Aug 2011 19:46:56 +0200 Subject: [petsc-users] SLEPc generalized eigenvalue problem question? In-Reply-To: References: Message-ID: <7D43440C-51C4-4195-AA82-436BA951558B@dsic.upv.es> El 01/08/2011, a las 19:27, John Chludzinski escribi?: > I create 2 matrices using: > > MatCreateSeqDense(PETSC_COMM_SELF, n, n, Ka, &A); > MatCreateSeqDense(PETSC_COMM_SELF, n, n, Kb, &B); > > These matrices are 99% zeros ( 16,016,004 entries and 18660 non-zeros). They are symmetric and real. Their tri-diagonal elements are non-zero plus a few other entries. > > I tried to use ex7 for the generalized eigenvalue problem: > > ./ex7.exe -f1 k.dat -f2 m.dat -eps_gen_hermitian -eps_smallest_real > x.out 2>&1 > > without specifying an EPS and get: > > Generalized eigenproblem stored in file. > > Reading REAL matrices from binary files... > Number of iterations of the method: 500 > Number of linear iterations of the method: 4009 > Solution method: krylovschur > > Number of requested eigenvalues: 1 > Stopping condition: tol=1e-07, maxit=500 > Number of converged approximate eigenpairs: 0 > > Is krylovschur inappropriate for this problem or have I set up the problem incorrectly by using MatCreateSeqDense(...) to create the matrix input files in PETSc binary form? > > ---John The solver has reached the maximum number of iterations. Do you want to compute the leftmost part of the spectrum? Are those eigenvalues (relatively) large in magnitude? I guess not. If you need the smallest eigenvalues, instead of computing the smallest eigenvalues of (K,M) try computing the largest eigenvalues of (M,K) and then compute the reciprocals. Also, have a look at the chapter on spectral transformations. And of course do not use dense matrices, as pointed out by Matt. Jose From jchludzinski at gmail.com Mon Aug 1 13:12:45 2011 From: jchludzinski at gmail.com (John Chludzinski) Date: Mon, 1 Aug 2011 14:12:45 -0400 Subject: [petsc-users] SLEPc generalized eigenvalue problem question? In-Reply-To: References: Message-ID: I have 2 files (matrices) in simply binary form (IEEE-754, generated by some C code) and wished to get them into canonical "PETSc binary form". So I did: Mat A; PetscScalar *a; ierr = PetscMalloc(SIZE*SIZE*sizeof(PetscScalar),&a);CHKERRQ(ierr); // stored the file into the space malloc'ed for 'a'. MatCreateSeqDense(PETSC_COMM_SELF, n, n, a, &A); MatView(A,PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD)); This works when I use: -eps_type lapack. As long as I store the matrix in column major order. ---John On Mon, Aug 1, 2011 at 1:40 PM, Matthew Knepley wrote: On Mon, Aug 1, 2011 at 5:27 PM, John Chludzinski wrote: > >> I create 2 matrices using: >> >> MatCreateSeqDense(PETSC_COMM_SELF, n, n, Ka, &A); >> MatCreateSeqDense(PETSC_COMM_SELF, n, n, Kb, &B); >> >> These matrices are 99% zeros ( 16,016,004 entries and 18660 non-zeros). >> They are symmetric and real. Their tri-diagonal elements are non-zero plus >> a few other entries. >> > > Please give some justification for doing this? On the surface, it just > seems perverse. > > Matt > > >> I tried to use ex7 for the generalized eigenvalue problem: >> >> ./ex7.exe -f1 k.dat -f2 m.dat -eps_gen_hermitian -eps_smallest_real > >> x.out 2>&1 >> >> without specifying an EPS and get: >> >> Generalized eigenproblem stored in file. >> >> Reading REAL matrices from binary files... >> Number of iterations of the method: 500 >> Number of linear iterations of the method: 4009 >> Solution method: krylovschur >> >> Number of requested eigenvalues: 1 >> Stopping condition: tol=1e-07, maxit=500 >> Number of converged approximate eigenpairs: 0 >> >> Is krylovschur inappropriate for this problem or have I set up the problem >> incorrectly by using MatCreateSeqDense(...) to create the matrix input >> files in PETSc binary form? >> >> ---John >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jchludzinski at gmail.com Mon Aug 1 13:22:40 2011 From: jchludzinski at gmail.com (John Chludzinski) Date: Mon, 1 Aug 2011 14:22:40 -0400 Subject: [petsc-users] SLEPc generalized eigenvalue problem question? In-Reply-To: References: Message-ID: I'm a newbie with both PETSc & SLEPc and have had some trouble finding examples/tutorials for newbies. I've looked through the examples in the PETSc and SLEPc directories but still am having "issues" seeing how to set this up for the type of problem I have. SLEPc ex7.c is a good place to start but there's still how best to store the matrices and which EPS to use (besides LAPACK). Found a PDF, "MATRICES IN PETSc", (after much googling) but not sure which of the many forms will work and which is best. ---John On Mon, Aug 1, 2011 at 2:12 PM, John Chludzinski wrote: > I have 2 files (matrices) in simply binary form (IEEE-754, generated by > some C code) and wished to get them into canonical "PETSc binary form". So I > did: > > Mat A; > PetscScalar *a; > > ierr = PetscMalloc(SIZE*SIZE*sizeof(PetscScalar),&a);CHKERRQ(ierr); > // stored the file into the space malloc'ed for 'a'. > MatCreateSeqDense(PETSC_COMM_SELF, n, n, a, &A); > MatView(A,PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD)); > > This works when I use: -eps_type lapack. As long as I store the matrix in > column major order. > > ---John > > On Mon, Aug 1, 2011 at 1:40 PM, Matthew Knepley wrote: > > On Mon, Aug 1, 2011 at 5:27 PM, John Chludzinski wrote: >> >>> I create 2 matrices using: >>> >>> MatCreateSeqDense(PETSC_COMM_SELF, n, n, Ka, &A); >>> MatCreateSeqDense(PETSC_COMM_SELF, n, n, Kb, &B); >>> >>> These matrices are 99% zeros ( 16,016,004 entries and 18660 non-zeros). >>> They are symmetric and real. Their tri-diagonal elements are non-zero plus >>> a few other entries. >>> >> >> Please give some justification for doing this? On the surface, it just >> seems perverse. >> >> Matt >> >> >>> I tried to use ex7 for the generalized eigenvalue problem: >>> >>> ./ex7.exe -f1 k.dat -f2 m.dat -eps_gen_hermitian -eps_smallest_real > >>> x.out 2>&1 >>> >>> without specifying an EPS and get: >>> >>> Generalized eigenproblem stored in file. >>> >>> Reading REAL matrices from binary files... >>> Number of iterations of the method: 500 >>> Number of linear iterations of the method: 4009 >>> Solution method: krylovschur >>> >>> Number of requested eigenvalues: 1 >>> Stopping condition: tol=1e-07, maxit=500 >>> Number of converged approximate eigenpairs: 0 >>> >>> Is krylovschur inappropriate for this problem or have I set up the >>> problem incorrectly by using MatCreateSeqDense(...) to create the matrix >>> input files in PETSc binary form? >>> >>> ---John >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Aug 1 13:27:18 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 1 Aug 2011 18:27:18 +0000 Subject: [petsc-users] SLEPc generalized eigenvalue problem question? In-Reply-To: References: Message-ID: On Mon, Aug 1, 2011 at 6:22 PM, John Chludzinski wrote: > I'm a newbie with both PETSc & SLEPc and have had some trouble finding > examples/tutorials for newbies. I've looked through the examples in the > PETSc and SLEPc directories but still am having "issues" seeing how to set > this up for the type of problem I have. > > SLEPc ex7.c is a good place to start but there's still how best to store > the matrices and which EPS to use (besides LAPACK). > > Found a PDF, "MATRICES IN PETSc", (after much googling) but not sure which > of the many forms will work and which is best. > 1) PETSc and SLEPc are designed to be efficient for sparse matrices. If you want eigenvalues of dense matrices, use Elemental (as I pointed out in a previous message) 2) If you generate matrices with C code, why not just call MatSetValues() for each row in that code? Matt > ---John > > > On Mon, Aug 1, 2011 at 2:12 PM, John Chludzinski wrote: > >> I have 2 files (matrices) in simply binary form (IEEE-754, generated by >> some C code) and wished to get them into canonical "PETSc binary form". So I >> did: >> >> Mat A; >> PetscScalar *a; >> >> ierr = PetscMalloc(SIZE*SIZE*sizeof(PetscScalar),&a);CHKERRQ(ierr); >> // stored the file into the space malloc'ed for 'a'. >> MatCreateSeqDense(PETSC_COMM_SELF, n, n, a, &A); >> MatView(A,PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD)); >> >> This works when I use: -eps_type lapack. As long as I store the matrix in >> column major order. >> >> ---John >> >> On Mon, Aug 1, 2011 at 1:40 PM, Matthew Knepley wrote: >> >> On Mon, Aug 1, 2011 at 5:27 PM, John Chludzinski wrote: >>> >>>> I create 2 matrices using: >>>> >>>> MatCreateSeqDense(PETSC_COMM_SELF, n, n, Ka, &A); >>>> MatCreateSeqDense(PETSC_COMM_SELF, n, n, Kb, &B); >>>> >>>> These matrices are 99% zeros ( 16,016,004 entries and 18660 non-zeros). >>>> They are symmetric and real. Their tri-diagonal elements are non-zero plus >>>> a few other entries. >>>> >>> >>> Please give some justification for doing this? On the surface, it just >>> seems perverse. >>> >>> Matt >>> >>> >>>> I tried to use ex7 for the generalized eigenvalue problem: >>>> >>>> ./ex7.exe -f1 k.dat -f2 m.dat -eps_gen_hermitian -eps_smallest_real > >>>> x.out 2>&1 >>>> >>>> without specifying an EPS and get: >>>> >>>> Generalized eigenproblem stored in file. >>>> >>>> Reading REAL matrices from binary files... >>>> Number of iterations of the method: 500 >>>> Number of linear iterations of the method: 4009 >>>> Solution method: krylovschur >>>> >>>> Number of requested eigenvalues: 1 >>>> Stopping condition: tol=1e-07, maxit=500 >>>> Number of converged approximate eigenpairs: 0 >>>> >>>> Is krylovschur inappropriate for this problem or have I set up the >>>> problem incorrectly by using MatCreateSeqDense(...) to create the matrix >>>> input files in PETSc binary form? >>>> >>>> ---John >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jchludzinski at gmail.com Mon Aug 1 13:36:40 2011 From: jchludzinski at gmail.com (John Chludzinski) Date: Mon, 1 Aug 2011 14:36:40 -0400 Subject: [petsc-users] SLEPc generalized eigenvalue problem question? In-Reply-To: References: Message-ID: These are definitely not dense matrices (>99% zeros). The way I described storing them just happened to be a way I found that worked (for EPS = LAPACK). But I want to use methods intended for sparse matrices + MPI. Which is the best canonical PETSc form (for the matrices) and best EPS? ---John On Mon, Aug 1, 2011 at 2:27 PM, Matthew Knepley wrote: > On Mon, Aug 1, 2011 at 6:22 PM, John Chludzinski wrote: > >> I'm a newbie with both PETSc & SLEPc and have had some trouble finding >> examples/tutorials for newbies. I've looked through the examples in the >> PETSc and SLEPc directories but still am having "issues" seeing how to set >> this up for the type of problem I have. >> >> SLEPc ex7.c is a good place to start but there's still how best to store >> the matrices and which EPS to use (besides LAPACK). >> >> Found a PDF, "MATRICES IN PETSc", (after much googling) but not sure which >> of the many forms will work and which is best. >> > > 1) PETSc and SLEPc are designed to be efficient for sparse matrices. If you > want eigenvalues of dense matrices, use Elemental (as I pointed out in a > previous message) > > 2) If you generate matrices with C code, why not just call MatSetValues() > for each row in that code? > > Matt > > >> ---John >> >> >> On Mon, Aug 1, 2011 at 2:12 PM, John Chludzinski wrote: >> >>> I have 2 files (matrices) in simply binary form (IEEE-754, generated by >>> some C code) and wished to get them into canonical "PETSc binary form". So I >>> did: >>> >>> Mat A; >>> PetscScalar *a; >>> >>> ierr = PetscMalloc(SIZE*SIZE*sizeof(PetscScalar),&a);CHKERRQ(ierr); >>> // stored the file into the space malloc'ed for 'a'. >>> MatCreateSeqDense(PETSC_COMM_SELF, n, n, a, &A); >>> MatView(A,PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD)); >>> >>> This works when I use: -eps_type lapack. As long as I store the matrix >>> in column major order. >>> >>> ---John >>> >>> On Mon, Aug 1, 2011 at 1:40 PM, Matthew Knepley wrote: >>> >>> On Mon, Aug 1, 2011 at 5:27 PM, John Chludzinski >>> > wrote: >>>> >>>>> I create 2 matrices using: >>>>> >>>>> MatCreateSeqDense(PETSC_COMM_SELF, n, n, Ka, &A); >>>>> MatCreateSeqDense(PETSC_COMM_SELF, n, n, Kb, &B); >>>>> >>>>> These matrices are 99% zeros ( 16,016,004 entries and 18660 non-zeros). >>>>> They are symmetric and real. Their tri-diagonal elements are non-zero plus >>>>> a few other entries. >>>>> >>>> >>>> Please give some justification for doing this? On the surface, it just >>>> seems perverse. >>>> >>>> Matt >>>> >>>> >>>>> I tried to use ex7 for the generalized eigenvalue problem: >>>>> >>>>> ./ex7.exe -f1 k.dat -f2 m.dat -eps_gen_hermitian -eps_smallest_real > >>>>> x.out 2>&1 >>>>> >>>>> without specifying an EPS and get: >>>>> >>>>> Generalized eigenproblem stored in file. >>>>> >>>>> Reading REAL matrices from binary files... >>>>> Number of iterations of the method: 500 >>>>> Number of linear iterations of the method: 4009 >>>>> Solution method: krylovschur >>>>> >>>>> Number of requested eigenvalues: 1 >>>>> Stopping condition: tol=1e-07, maxit=500 >>>>> Number of converged approximate eigenpairs: 0 >>>>> >>>>> Is krylovschur inappropriate for this problem or have I set up the >>>>> problem incorrectly by using MatCreateSeqDense(...) to create the matrix >>>>> input files in PETSc binary form? >>>>> >>>>> ---John >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Aug 1 13:41:42 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 1 Aug 2011 18:41:42 +0000 Subject: [petsc-users] SLEPc generalized eigenvalue problem question? In-Reply-To: References: Message-ID: On Mon, Aug 1, 2011 at 6:36 PM, John Chludzinski wrote: > These are definitely not dense matrices (>99% zeros). > > The way I described storing them just happened to be a way I found that > worked (for EPS = LAPACK). But I want to use methods intended for sparse > matrices + MPI. > > Which is the best canonical PETSc form (for the matrices) and best EPS? > 1) MATAIJ 2) What is EPS? Matt > ---John > > > On Mon, Aug 1, 2011 at 2:27 PM, Matthew Knepley wrote: > >> On Mon, Aug 1, 2011 at 6:22 PM, John Chludzinski wrote: >> >>> I'm a newbie with both PETSc & SLEPc and have had some trouble finding >>> examples/tutorials for newbies. I've looked through the examples in the >>> PETSc and SLEPc directories but still am having "issues" seeing how to set >>> this up for the type of problem I have. >>> >>> SLEPc ex7.c is a good place to start but there's still how best to store >>> the matrices and which EPS to use (besides LAPACK). >>> >>> Found a PDF, "MATRICES IN PETSc", (after much googling) but not sure >>> which of the many forms will work and which is best. >>> >> >> 1) PETSc and SLEPc are designed to be efficient for sparse matrices. If >> you want eigenvalues of dense matrices, use Elemental (as I pointed out in a >> previous message) >> >> 2) If you generate matrices with C code, why not just call MatSetValues() >> for each row in that code? >> >> Matt >> >> >>> ---John >>> >>> >>> On Mon, Aug 1, 2011 at 2:12 PM, John Chludzinski >> > wrote: >>> >>>> I have 2 files (matrices) in simply binary form (IEEE-754, generated by >>>> some C code) and wished to get them into canonical "PETSc binary form". So I >>>> did: >>>> >>>> Mat A; >>>> PetscScalar *a; >>>> >>>> ierr = PetscMalloc(SIZE*SIZE*sizeof(PetscScalar),&a);CHKERRQ(ierr); >>>> // stored the file into the space malloc'ed for 'a'. >>>> MatCreateSeqDense(PETSC_COMM_SELF, n, n, a, &A); >>>> MatView(A,PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD)); >>>> >>>> This works when I use: -eps_type lapack. As long as I store the matrix >>>> in column major order. >>>> >>>> ---John >>>> >>>> On Mon, Aug 1, 2011 at 1:40 PM, Matthew Knepley wrote: >>>> >>>> On Mon, Aug 1, 2011 at 5:27 PM, John Chludzinski < >>>>> jchludzinski at gmail.com> wrote: >>>>> >>>>>> I create 2 matrices using: >>>>>> >>>>>> MatCreateSeqDense(PETSC_COMM_SELF, n, n, Ka, &A); >>>>>> MatCreateSeqDense(PETSC_COMM_SELF, n, n, Kb, &B); >>>>>> >>>>>> These matrices are 99% zeros ( 16,016,004 entries and 18660 >>>>>> non-zeros). They are symmetric and real. Their tri-diagonal elements are >>>>>> non-zero plus a few other entries. >>>>>> >>>>> >>>>> Please give some justification for doing this? On the surface, it just >>>>> seems perverse. >>>>> >>>>> Matt >>>>> >>>>> >>>>>> I tried to use ex7 for the generalized eigenvalue problem: >>>>>> >>>>>> ./ex7.exe -f1 k.dat -f2 m.dat -eps_gen_hermitian -eps_smallest_real > >>>>>> x.out 2>&1 >>>>>> >>>>>> without specifying an EPS and get: >>>>>> >>>>>> Generalized eigenproblem stored in file. >>>>>> >>>>>> Reading REAL matrices from binary files... >>>>>> Number of iterations of the method: 500 >>>>>> Number of linear iterations of the method: 4009 >>>>>> Solution method: krylovschur >>>>>> >>>>>> Number of requested eigenvalues: 1 >>>>>> Stopping condition: tol=1e-07, maxit=500 >>>>>> Number of converged approximate eigenpairs: 0 >>>>>> >>>>>> Is krylovschur inappropriate for this problem or have I set up the >>>>>> problem incorrectly by using MatCreateSeqDense(...) to create the matrix >>>>>> input files in PETSc binary form? >>>>>> >>>>>> ---John >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From vijay.m at gmail.com Mon Aug 1 13:53:58 2011 From: vijay.m at gmail.com (Vijay S. Mahadevan) Date: Mon, 1 Aug 2011 13:53:58 -0500 Subject: [petsc-users] SLEPc generalized eigenvalue problem question? In-Reply-To: References: Message-ID: > 2) What is EPS? Eigenvalue Problem Solver (SLEPc). The "best" eigensolver is problem dependent. You can use arpack if you have configured slepc with it or else use Krylov-schur to test the convergence. There are also Davidson family of solvers implemented in slepc which are quite effective for symmetric Hermitian systems. Have a look at ex13 in slepc for a generalized eigenproblem and how the sparse matrices are constructed using MatSetValue. Just like the petsc options, you should be able to change the type of solver dynamically from command line. On Mon, Aug 1, 2011 at 1:41 PM, Matthew Knepley wrote: > On Mon, Aug 1, 2011 at 6:36 PM, John Chludzinski > wrote: >> >> These are definitely not dense matrices (>99% zeros). >> The way I described storing them just happened to be a way I found that >> worked (for EPS = LAPACK). ?But I want to use methods intended for sparse >> matrices + MPI. >> >> Which is the best canonical PETSc form (for the matrices) and best EPS? > > 1) MATAIJ > 2) What is EPS? > ? ?Matt > >> >> ---John >> >> On Mon, Aug 1, 2011 at 2:27 PM, Matthew Knepley wrote: >>> >>> On Mon, Aug 1, 2011 at 6:22 PM, John Chludzinski >>> wrote: >>>> >>>> I'm a newbie with both PETSc & SLEPc and have had some trouble finding >>>> examples/tutorials for newbies. ?I've looked through the examples in the >>>> PETSc and SLEPc directories but still am having "issues" seeing how to set >>>> this up for the type of problem I have. >>>> SLEPc ex7.c is a good place to start but there's still how best to store >>>> the matrices and which EPS to use (besides LAPACK). >>>> Found a PDF, "MATRICES IN PETSc", (after much googling) but not sure >>>> which of the many forms will work and which is best. >>> >>> 1) PETSc and SLEPc are designed to be efficient for sparse matrices. If >>> you want eigenvalues of dense matrices, use Elemental (as I pointed out in a >>> previous message) >>> 2) If you generate matrices with C code, why not just call MatSetValues() >>> for each row in that code? >>> ? ?Matt >>> >>>> >>>> ---John >>>> >>>> On Mon, Aug 1, 2011 at 2:12 PM, John Chludzinski >>>> wrote: >>>>> >>>>> I have 2 files (matrices) in simply binary form (IEEE-754, generated by >>>>> some C code) and wished to get them into canonical "PETSc binary form". So I >>>>> did: >>>>> Mat A; >>>>> PetscScalar *a; >>>>> ierr = PetscMalloc(SIZE*SIZE*sizeof(PetscScalar),&a);CHKERRQ(ierr); >>>>> // stored the file into the space malloc'ed for 'a'. >>>>> MatCreateSeqDense(PETSC_COMM_SELF, n, n, a, &A); >>>>> MatView(A,PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD)); >>>>> This works when I use: -eps_type lapack. ?As long as I store the matrix >>>>> in column major order. >>>>> ---John >>>>> On Mon, Aug 1, 2011 at 1:40 PM, Matthew Knepley >>>>> wrote: >>>>>> >>>>>> On Mon, Aug 1, 2011 at 5:27 PM, John Chludzinski >>>>>> wrote: >>>>>>> >>>>>>> I create 2 matrices using: >>>>>>> MatCreateSeqDense(PETSC_COMM_SELF, n, n, Ka, &A); >>>>>>> MatCreateSeqDense(PETSC_COMM_SELF, n, n, Kb, &B); >>>>>>> These matrices are 99% zeros ( 16,016,004 entries and 18660 >>>>>>> non-zeros). ?They are symmetric and real. ?Their tri-diagonal elements are >>>>>>> non-zero plus a few other entries. >>>>>> >>>>>> Please give some justification for doing this? On the surface, it just >>>>>> seems perverse. >>>>>> ? ?Matt >>>>>> >>>>>>> >>>>>>> I tried to use ex7 for the generalized eigenvalue problem: >>>>>>> ./ex7.exe -f1 k.dat -f2 m.dat -eps_gen_hermitian -eps_smallest_real > >>>>>>> x.out 2>&1 >>>>>>> without specifying an EPS and get: >>>>>>> Generalized eigenproblem stored in file. >>>>>>> Reading REAL matrices from binary files... >>>>>>> Number of iterations of the method: 500 >>>>>>> Number of linear iterations of the method: 4009 >>>>>>> Solution method: krylovschur >>>>>>> Number of requested eigenvalues: 1 >>>>>>> Stopping condition: tol=1e-07, maxit=500 >>>>>>> Number of converged approximate eigenpairs: 0 >>>>>>> Is krylovschur inappropriate for this problem or have I set up the >>>>>>> problem incorrectly by using?? MatCreateSeqDense(...) to create the matrix >>>>>>> input files in PETSc binary form? >>>>>>> ---John >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>> >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >> > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > From xyuan at lbl.gov Mon Aug 1 15:35:03 2011 From: xyuan at lbl.gov (Xuefei (Rebecca) Yuan) Date: Mon, 1 Aug 2011 13:35:03 -0700 Subject: [petsc-users] Have you heard of UPC: unified Parallel C? Message-ID: <222CBEE7-B79E-40E6-84B9-537E40670C0B@lbl.gov> Hello all, Could I convert a C code using PETSc library to a UPC code using PETSc? Thanks, Xuefei (Rebecca) Yuan Postdoctoral Fellow Lawrence Berkeley National Laboratory Tel: 1-510-486-7031 From dominik at itis.ethz.ch Mon Aug 1 15:59:51 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Mon, 1 Aug 2011 22:59:51 +0200 Subject: [petsc-users] race conditions in VecSetValue(s) Message-ID: What will happen if VecSetValue(s) called from different processes want to update the value at the same position in the global MPI vector? Is there a some sort of queue/lock or will a race condition result? It's not discussed in the documentation. Thanks for any clarification, Dominik From adam1.byrd at gmail.com Mon Aug 1 16:00:58 2011 From: adam1.byrd at gmail.com (Adam Byrd) Date: Mon, 1 Aug 2011 17:00:58 -0400 Subject: [petsc-users] Optimizing MatMatSolve Message-ID: Hello, I'm looking for help reducing the time and communication of a parallel MatMatSolve using MUMPS. On a single processor I experience decent solve times (~9 seconds each), but when moving to multiple processors I see longer times with more cores. I've run with -log_summary and confirmed (practically) all the time is spent in MatMatSolve. I'm fairly certain it's all communication between nodes and I'm trying to figure out where I can make optimizations, or if it is even feasible for this type of problem. It is a parallel, dense, direct solve using MUMPS with an LU preconditioner. I know there are many smaller optimizations that can be done in other areas, but at the moment it is only the solve that concerns me. ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./cntor on a complex-c named hpc-1-0.local with 2 processors, by abyrd Mon Aug 1 16:25:51 2011 Using Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 Max Max/Min Avg Total Time (sec): 1.307e+02 1.00000 1.307e+02 Objects: 1.180e+02 1.00000 1.180e+02 Flops: 0.000e+00 0.00000 0.000e+00 0.000e+00 Flops/sec: 0.000e+00 0.00000 0.000e+00 0.000e+00 Memory: 2.091e+08 1.00001 4.181e+08 MPI Messages: 7.229e+03 1.00000 7.229e+03 1.446e+04 MPI Message Lengths: 4.141e+08 1.00000 5.729e+04 8.283e+08 MPI Reductions: 1.464e+04 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.3072e+02 100.0% 0.0000e+00 0.0% 1.446e+04 100.0% 5.729e+04 100.0% 1.730e+02 1.2% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ ########################################################## # # # WARNING!!! # # # # This code was compiled with a debugging option, # # To get timing results run config/configure.py # # using --with-debugging=no, the performance will # # be generally two or three times faster. # # # ########################################################## ########################################################## # # # WARNING!!! # # # # The code for various complex numbers numerical # # kernels uses C++, which generally is not well # # optimized. For performance that is about 4-5 times # # faster, specify --with-fortran-kernels=1 # # when running config/configure.py. # # # ########################################################## Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage MatSolve 14400 1.0 1.2364e+02 1.0 0.00e+00 0.0 1.4e+04 5.7e+04 2.0e+01 95 0100100 0 95 0100100 12 0 MatLUFactorSym 4 1.0 2.0027e-05 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 4 1.0 3.4223e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01 3 0 0 0 0 3 0 0 0 14 0 MatConvert 1 1.0 2.3644e-01 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.1e+01 0 0 0 0 0 0 0 0 0 6 0 MatAssemblyBegin 14 1.0 1.9959e-01 9.3 0.00e+00 0.0 3.0e+01 5.2e+04 1.2e+01 0 0 0 0 0 0 0 0 0 7 0 MatAssemblyEnd 14 1.0 1.9908e-01 1.1 0.00e+00 0.0 4.0e+00 2.8e+01 2.0e+01 0 0 0 0 0 0 0 0 0 12 0 MatGetRow 32 1.0 4.2677e-05 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetSubMatrice 4 1.0 7.6661e-03 1.0 0.00e+00 0.0 1.6e+01 1.2e+05 2.4e+01 0 0 0 0 0 0 0 0 0 14 0 MatMatSolve 4 1.0 1.2380e+02 1.0 0.00e+00 0.0 1.4e+04 5.7e+04 2.0e+01 95 0100100 0 95 0100100 12 0 VecSet 4 1.0 1.8590e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 28800 1.0 2.2810e+00 2.2 0.00e+00 0.0 1.4e+04 5.7e+04 0.0e+00 1 0100100 0 1 0100100 0 0 VecScatterEnd 14400 1.0 4.1534e+00 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 KSPSetup 4 1.0 1.1060e-0212.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 4 1.0 3.4280e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 5.6e+01 3 0 0 0 0 3 0 0 0 32 0 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 27 27 208196712 0 Vec 36 36 1027376 0 Vec Scatter 11 11 7220 0 Index Set 42 42 22644 0 Krylov Solver 1 1 34432 0 Preconditioner 1 1 752 0 ======================================================================================================================== Average time to get PetscTime(): 1.90735e-07 Average time for MPI_Barrier(): 3.8147e-06 Average time for zero size MPI_Send(): 7.51019e-06 #PETSc Option Table entries: -log_summary -pc_factor_mat_solver_package mumps -pc_type lu #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 16 Configure run at: Mon Jul 11 15:28:42 2011 Configure options: PETSC_ARCH=complex-cpp-mumps --with-cc=mpicc --with-fc=mpif90 --with-blas-lapack-dir=/usr/lib64 --with-shared --with-clanguage=c++ --with-scalar-type=complex --download-mumps=1 --download-blacs=1 --download-scalapack=1 --download-parmetis=1 --with-cxx=mpicxx ----------------------------------------- Libraries compiled on Mon Jul 11 15:39:58 EDT 2011 on sc.local Machine characteristics: Linux sc.local 2.6.18-194.11.1.el5 #1 SMP Tue Aug 10 19:05:06 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux Using PETSc directory: /panfs/storage.local/scs/home/abyrd/petsc-3.1-p8 Using PETSc arch: complex-cpp-mumps ----------------------------------------- Using C compiler: mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -g -fPIC Using Fortran compiler: mpif90 -fPIC -Wall -Wno-unused-variable -g ----------------------------------------- Using include paths: -I/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/include -I/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/include -I/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/include -I/usr/mpi/gnu/openmpi-1.4.2/include -I/usr/mpi/gnu/openmpi-1.4.2/lib64 ------------------------------------------ Using C linker: mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -g Using Fortran linker: mpif90 -fPIC -Wall -Wno-unused-variable -g Using libraries: -Wl,-rpath,/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib -L/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib -lpetsc -lX11 -Wl,-rpath,/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib -L/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -Wl,-rpath,/usr/lib64 -L/usr/lib64 -llapack -lblas -lnsl -lrt -Wl,-rpath,/usr/mpi/gnu/openmpi-1.4.2/lib64 -L/usr/mpi/gnu/openmpi-1.4.2/lib64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -lmpi_f90 -lmpi_f77 -lgfortran -lm -lm -lm -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl Respectfully, Adam Byrd -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PETScCntor.zip Type: application/zip Size: 78943 bytes Desc: not available URL: From aron.ahmadia at kaust.edu.sa Mon Aug 1 16:05:33 2011 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Tue, 2 Aug 2011 00:05:33 +0300 Subject: [petsc-users] Have you heard of UPC: unified Parallel C? In-Reply-To: <222CBEE7-B79E-40E6-84B9-537E40670C0B@lbl.gov> References: <222CBEE7-B79E-40E6-84B9-537E40670C0B@lbl.gov> Message-ID: Dear Rebecca, PETSc+UPC is certainly possible since UPC is an extension to C99, so underneath PETSc+UPC your parallelism model would be MPI+UPC. If you are planning on using PETSc for your distributed-memory parallelism and UPC for your 'accelerator' such as multicore or GPU, this would just require you to use UPC to parallelize your kernels. On the other hand, if you want UPC to handle your distributed memory parallelism as well in true PGAS fashion, you could extend a no-MPI PETSc, but at the effort in parallelizing that (introducing new Mat, Vec, KSP, SNES objects as needed with UPC parallelism) seems to not be worth the payoff. A On Mon, Aug 1, 2011 at 11:35 PM, Xuefei (Rebecca) Yuan wrote: > Hello all, > > Could I convert a C code using PETSc library to a UPC code using PETSc? > > Thanks, > > Xuefei (Rebecca) Yuan > Postdoctoral Fellow > Lawrence Berkeley National Laboratory > Tel: 1-510-486-7031 > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Aug 1 16:05:48 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 1 Aug 2011 15:05:48 -0600 Subject: [petsc-users] race conditions in VecSetValue(s) In-Reply-To: References: Message-ID: <91D59E06-5C69-4725-9F62-9329103CEF14@mcs.anl.gov> On Aug 1, 2011, at 2:59 PM, Dominik Szczerba wrote: > What will happen if VecSetValue(s) called from different processes > want to update the value at the same position in the global MPI > vector? > Is there a some sort of queue/lock No > or will a race condition result? No. > It's not discussed in the documentation. Yes it is :-) If ADD_VALUES is used then the location will always get the correct sum. If INSERT_VALUES is used then one of the values set will be used but it is not defined which one. Barry > > Thanks for any clarification, > Dominik From bsmith at mcs.anl.gov Mon Aug 1 16:09:24 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 1 Aug 2011 15:09:24 -0600 Subject: [petsc-users] Optimizing MatMatSolve In-Reply-To: References: Message-ID: <5CBFD41E-55B6-4AA4-9C17-4A2984EAB4D4@mcs.anl.gov> On Aug 1, 2011, at 3:00 PM, Adam Byrd wrote: > Hello, > > I'm looking for help reducing the time and communication of a parallel MatMatSolve using MUMPS. On a single processor I experience decent solve times (~9 seconds each), but when moving to multiple processors I see longer times with more cores. I've run with -log_summary and confirmed (practically) all the time is spent in MatMatSolve. I'm fairly certain it's all communication between nodes and I'm trying to figure out where I can make optimizations, or if it is even feasible for this type of problem. It is a parallel, dense, I hope you mean that the original matrix you use with MUMPS is sparse (you should not use MUMPS to solve dense linear systems). > direct solve using MUMPS with an LU preconditioner. I know there are many smaller optimizations that can be done in other areas, but at the moment it is only the solve that concerns me. MUMPS will run slower on 2 processors than 1, this is just a fact of life. You will only gain with parallel for MUMPS for large problems. Barry > > ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- > > ./cntor on a complex-c named hpc-1-0.local with 2 processors, by abyrd Mon Aug 1 16:25:51 2011 > Using Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 > > Max Max/Min Avg Total > Time (sec): 1.307e+02 1.00000 1.307e+02 > Objects: 1.180e+02 1.00000 1.180e+02 > Flops: 0.000e+00 0.00000 0.000e+00 0.000e+00 > Flops/sec: 0.000e+00 0.00000 0.000e+00 0.000e+00 > Memory: 2.091e+08 1.00001 4.181e+08 > MPI Messages: 7.229e+03 1.00000 7.229e+03 1.446e+04 > MPI Message Lengths: 4.141e+08 1.00000 5.729e+04 8.283e+08 > MPI Reductions: 1.464e+04 1.00000 > > Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) > e.g., VecAXPY() for real vectors of length N --> 2N flops > and VecAXPY() for complex vectors of length N --> 8N flops > > Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total counts %Total Avg %Total counts %Total > 0: Main Stage: 1.3072e+02 100.0% 0.0000e+00 0.0% 1.446e+04 100.0% 5.729e+04 100.0% 1.730e+02 1.2% > > ------------------------------------------------------------------------------------------------------------------------ > See the 'Profiling' chapter of the users' manual for details on interpreting output. > Phase summary info: > Count: number of times phase was executed > Time and Flops: Max - maximum over all processors > Ratio - ratio of maximum to minimum over all processors > Mess: number of messages sent > Avg. len: average message length > Reduct: number of global reductions > Global: entire computation > Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). > %T - percent time in this phase %F - percent flops in this phase > %M - percent messages in this phase %L - percent message lengths in this phase > %R - percent reductions in this phase > Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) > ------------------------------------------------------------------------------------------------------------------------ > > > ########################################################## > # # > # WARNING!!! # > # # > # This code was compiled with a debugging option, # > # To get timing results run config/configure.py # > # using --with-debugging=no, the performance will # > # be generally two or three times faster. # > # # > ########################################################## > > > > > ########################################################## > # # > # WARNING!!! # > # # > # The code for various complex numbers numerical # > # kernels uses C++, which generally is not well # > # optimized. For performance that is about 4-5 times # > # faster, specify --with-fortran-kernels=1 # > # when running config/configure.py. # > # # > ########################################################## > > > Event Count Time (sec) Flops --- Global --- --- Stage --- Total > Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > MatSolve 14400 1.0 1.2364e+02 1.0 0.00e+00 0.0 1.4e+04 5.7e+04 2.0e+01 95 0100100 0 95 0100100 12 0 > MatLUFactorSym 4 1.0 2.0027e-05 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatLUFactorNum 4 1.0 3.4223e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01 3 0 0 0 0 3 0 0 0 14 0 > MatConvert 1 1.0 2.3644e-01 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.1e+01 0 0 0 0 0 0 0 0 0 6 0 > MatAssemblyBegin 14 1.0 1.9959e-01 9.3 0.00e+00 0.0 3.0e+01 5.2e+04 1.2e+01 0 0 0 0 0 0 0 0 0 7 0 > MatAssemblyEnd 14 1.0 1.9908e-01 1.1 0.00e+00 0.0 4.0e+00 2.8e+01 2.0e+01 0 0 0 0 0 0 0 0 0 12 0 > MatGetRow 32 1.0 4.2677e-05 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatGetSubMatrice 4 1.0 7.6661e-03 1.0 0.00e+00 0.0 1.6e+01 1.2e+05 2.4e+01 0 0 0 0 0 0 0 0 0 14 0 > MatMatSolve 4 1.0 1.2380e+02 1.0 0.00e+00 0.0 1.4e+04 5.7e+04 2.0e+01 95 0100100 0 95 0100100 12 0 > VecSet 4 1.0 1.8590e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecScatterBegin 28800 1.0 2.2810e+00 2.2 0.00e+00 0.0 1.4e+04 5.7e+04 0.0e+00 1 0100100 0 1 0100100 0 0 > VecScatterEnd 14400 1.0 4.1534e+00 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 > KSPSetup 4 1.0 1.1060e-0212.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > PCSetUp 4 1.0 3.4280e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 5.6e+01 3 0 0 0 0 3 0 0 0 32 0 > ------------------------------------------------------------------------------------------------------------------------ > > Memory usage is given in bytes: > > Object Type Creations Destructions Memory Descendants' Mem. > Reports information only for process 0. > > --- Event Stage 0: Main Stage > > Matrix 27 27 208196712 0 > Vec 36 36 1027376 0 > Vec Scatter 11 11 7220 0 > Index Set 42 42 22644 0 > Krylov Solver 1 1 34432 0 > Preconditioner 1 1 752 0 > ======================================================================================================================== > Average time to get PetscTime(): 1.90735e-07 > Average time for MPI_Barrier(): 3.8147e-06 > Average time for zero size MPI_Send(): 7.51019e-06 > #PETSc Option Table entries: > -log_summary > -pc_factor_mat_solver_package mumps > -pc_type lu > #End of PETSc Option Table entries > Compiled without FORTRAN kernels > Compiled with full precision matrices (default) > sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 16 > Configure run at: Mon Jul 11 15:28:42 2011 > Configure options: PETSC_ARCH=complex-cpp-mumps --with-cc=mpicc --with-fc=mpif90 --with-blas-lapack-dir=/usr/lib64 --with-shared --with-clanguage=c++ --with-scalar-type=complex --download-mumps=1 --download-blacs=1 --download-scalapack=1 --download-parmetis=1 --with-cxx=mpicxx > ----------------------------------------- > Libraries compiled on Mon Jul 11 15:39:58 EDT 2011 on sc.local > Machine characteristics: Linux sc.local 2.6.18-194.11.1.el5 #1 SMP Tue Aug 10 19:05:06 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux > Using PETSc directory: /panfs/storage.local/scs/home/abyrd/petsc-3.1-p8 > Using PETSc arch: complex-cpp-mumps > ----------------------------------------- > Using C compiler: mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -g -fPIC > Using Fortran compiler: mpif90 -fPIC -Wall -Wno-unused-variable -g > ----------------------------------------- > Using include paths: -I/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/include -I/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/include -I/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/include -I/usr/mpi/gnu/openmpi-1.4.2/include -I/usr/mpi/gnu/openmpi-1.4.2/lib64 > ------------------------------------------ > Using C linker: mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -g > Using Fortran linker: mpif90 -fPIC -Wall -Wno-unused-variable -g > Using libraries: -Wl,-rpath,/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib -L/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib -lpetsc -lX11 -Wl,-rpath,/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib -L/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -Wl,-rpath,/usr/lib64 -L/usr/lib64 -llapack -lblas -lnsl -lrt -Wl,-rpath,/usr/mpi/gnu/openmpi-1.4.2/lib64 -L/usr/mpi/gnu/openmpi-1.4.2/lib64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -lmpi_f90 -lmpi_f77 -lgfortran -lm -lm -lm -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl > > Respectfully, > Adam Byrd > From szczerba.dominik at gmail.com Mon Aug 1 16:15:13 2011 From: szczerba.dominik at gmail.com (Dominik Szczerba) Date: Mon, 1 Aug 2011 23:15:13 +0200 Subject: [petsc-users] race conditions in VecSetValue(s) In-Reply-To: <91D59E06-5C69-4725-9F62-9329103CEF14@mcs.anl.gov> References: <91D59E06-5C69-4725-9F62-9329103CEF14@mcs.anl.gov> Message-ID: >> Is there a some sort of queue/lock > > ? No > >> or will a race condition result? > > ?No. > >> It's not discussed in the documentation. > > ? Yes it is :-) I did not notice it in the VecSetValue(s) man pages... > ? If ADD_VALUES is used then the location will always get the correct sum. > > ? If INSERT_VALUES is used then one of the values set will be used but it is not defined which one. Many thanks for the clarifications. Would the same be the case with MatSetValues and a distributed matrix? Regards, Dominik > > ? Barry > >> >> Thanks for any clarification, >> Dominik > > From knepley at gmail.com Mon Aug 1 16:21:48 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 1 Aug 2011 21:21:48 +0000 Subject: [petsc-users] race conditions in VecSetValue(s) In-Reply-To: References: <91D59E06-5C69-4725-9F62-9329103CEF14@mcs.anl.gov> Message-ID: On Mon, Aug 1, 2011 at 9:15 PM, Dominik Szczerba wrote: > >> Is there a some sort of queue/lock > > > > No > > > >> or will a race condition result? > > > > No. > > > >> It's not discussed in the documentation. > > > > Yes it is :-) > > I did not notice it in the VecSetValue(s) man pages... > > > If ADD_VALUES is used then the location will always get the correct > sum. > > > > If INSERT_VALUES is used then one of the values set will be used but it > is not defined which one. > > Many thanks for the clarifications. Would the same be the case with > MatSetValues and a distributed matrix? > Yes. Matt > Regards, > Dominik > > > > > Barry > > > >> > >> Thanks for any clarification, > >> Dominik > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Mon Aug 1 16:23:05 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Mon, 1 Aug 2011 23:23:05 +0200 Subject: [petsc-users] race conditions in VecSetValue(s) In-Reply-To: References: <91D59E06-5C69-4725-9F62-9329103CEF14@mcs.anl.gov> Message-ID: >> Is there a some sort of queue/lock > > No > >> or will a race condition result? > > No. > >> It's not discussed in the documentation. > > Yes it is :-) I did not notice it in the VecSetValue(s) man pages... > If ADD_VALUES is used then the location will always get the correct sum. > > If INSERT_VALUES is used then one of the values set will be used but it is not defined which one. Many thanks for the clarifications. Would the same be the case with MatSetValues and a distributed matrix? Regards, Dominik From adam1.byrd at gmail.com Mon Aug 1 16:31:06 2011 From: adam1.byrd at gmail.com (Adam Byrd) Date: Mon, 1 Aug 2011 17:31:06 -0400 Subject: [petsc-users] Optimizing MatMatSolve In-Reply-To: <5CBFD41E-55B6-4AA4-9C17-4A2984EAB4D4@mcs.anl.gov> References: <5CBFD41E-55B6-4AA4-9C17-4A2984EAB4D4@mcs.anl.gov> Message-ID: On Mon, Aug 1, 2011 at 5:09 PM, Barry Smith wrote: > > On Aug 1, 2011, at 3:00 PM, Adam Byrd wrote: > > > Hello, > > > > I'm looking for help reducing the time and communication of a parallel > MatMatSolve using MUMPS. On a single processor I experience decent solve > times (~9 seconds each), but when moving to multiple processors I see longer > times with more cores. I've run with -log_summary and confirmed > (practically) all the time is spent in MatMatSolve. I'm fairly certain it's > all communication between nodes and I'm trying to figure out where I can > make optimizations, or if it is even feasible for this type of problem. It > is a parallel, dense, > > I hope you mean that the original matrix you use with MUMPS is sparse > (you should not use MUMPS to solve dense linear systems). > Oops, yes. The original matrix is sparse. It requires the solution and identity matrix to be dense. I was typing faster than thinking. > > > direct solve using MUMPS with an LU preconditioner. I know there are many > smaller optimizations that can be done in other areas, but at the moment it > is only the solve that concerns me. > > MUMPS will run slower on 2 processors than 1, this is just a fact of > life. You will only gain with parallel for MUMPS for large problems. > I see. It looks like I took off in the wrong direction then. I'm trying to solve for the inverse of a sparse matrix in parallel. I'm starting at 3600x3600 and will be moving to 30,000x30,000+ in the future. Which solver suits this sort of problem? > > Barry > > > > > > > ---------------------------------------------- PETSc Performance Summary: > ---------------------------------------------- > > > > ./cntor on a complex-c named hpc-1-0.local with 2 processors, by abyrd > Mon Aug 1 16:25:51 2011 > > Using Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 > > > > Max Max/Min Avg Total > > Time (sec): 1.307e+02 1.00000 1.307e+02 > > Objects: 1.180e+02 1.00000 1.180e+02 > > Flops: 0.000e+00 0.00000 0.000e+00 0.000e+00 > > Flops/sec: 0.000e+00 0.00000 0.000e+00 0.000e+00 > > Memory: 2.091e+08 1.00001 4.181e+08 > > MPI Messages: 7.229e+03 1.00000 7.229e+03 1.446e+04 > > MPI Message Lengths: 4.141e+08 1.00000 5.729e+04 8.283e+08 > > MPI Reductions: 1.464e+04 1.00000 > > > > Flop counting convention: 1 flop = 1 real number operation of type > (multiply/divide/add/subtract) > > e.g., VecAXPY() for real vectors of length N > --> 2N flops > > and VecAXPY() for complex vectors of length N > --> 8N flops > > > > Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages > --- -- Message Lengths -- -- Reductions -- > > Avg %Total Avg %Total counts > %Total Avg %Total counts %Total > > 0: Main Stage: 1.3072e+02 100.0% 0.0000e+00 0.0% 1.446e+04 > 100.0% 5.729e+04 100.0% 1.730e+02 1.2% > > > > > ------------------------------------------------------------------------------------------------------------------------ > > See the 'Profiling' chapter of the users' manual for details on > interpreting output. > > Phase summary info: > > Count: number of times phase was executed > > Time and Flops: Max - maximum over all processors > > Ratio - ratio of maximum to minimum over all > processors > > Mess: number of messages sent > > Avg. len: average message length > > Reduct: number of global reductions > > Global: entire computation > > Stage: stages of a computation. Set stages with PetscLogStagePush() > and PetscLogStagePop(). > > %T - percent time in this phase %F - percent flops in this > phase > > %M - percent messages in this phase %L - percent message > lengths in this phase > > %R - percent reductions in this phase > > Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time > over all processors) > > > ------------------------------------------------------------------------------------------------------------------------ > > > > > > ########################################################## > > # # > > # WARNING!!! # > > # # > > # This code was compiled with a debugging option, # > > # To get timing results run config/configure.py # > > # using --with-debugging=no, the performance will # > > # be generally two or three times faster. # > > # # > > ########################################################## > > > > > > > > > > ########################################################## > > # # > > # WARNING!!! # > > # # > > # The code for various complex numbers numerical # > > # kernels uses C++, which generally is not well # > > # optimized. For performance that is about 4-5 times # > > # faster, specify --with-fortran-kernels=1 # > > # when running config/configure.py. # > > # # > > ########################################################## > > > > > > Event Count Time (sec) Flops > --- Global --- --- Stage --- Total > > Max Ratio Max Ratio Max Ratio Mess Avg len > Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > > > ------------------------------------------------------------------------------------------------------------------------ > > > > --- Event Stage 0: Main Stage > > > > MatSolve 14400 1.0 1.2364e+02 1.0 0.00e+00 0.0 1.4e+04 5.7e+04 > 2.0e+01 95 0100100 0 95 0100100 12 0 > > MatLUFactorSym 4 1.0 2.0027e-05 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > MatLUFactorNum 4 1.0 3.4223e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 2.4e+01 3 0 0 0 0 3 0 0 0 14 0 > > MatConvert 1 1.0 2.3644e-01 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 > 1.1e+01 0 0 0 0 0 0 0 0 0 6 0 > > MatAssemblyBegin 14 1.0 1.9959e-01 9.3 0.00e+00 0.0 3.0e+01 5.2e+04 > 1.2e+01 0 0 0 0 0 0 0 0 0 7 0 > > MatAssemblyEnd 14 1.0 1.9908e-01 1.1 0.00e+00 0.0 4.0e+00 2.8e+01 > 2.0e+01 0 0 0 0 0 0 0 0 0 12 0 > > MatGetRow 32 1.0 4.2677e-05 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > MatGetSubMatrice 4 1.0 7.6661e-03 1.0 0.00e+00 0.0 1.6e+01 1.2e+05 > 2.4e+01 0 0 0 0 0 0 0 0 0 14 0 > > MatMatSolve 4 1.0 1.2380e+02 1.0 0.00e+00 0.0 1.4e+04 5.7e+04 > 2.0e+01 95 0100100 0 95 0100100 12 0 > > VecSet 4 1.0 1.8590e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > VecScatterBegin 28800 1.0 2.2810e+00 2.2 0.00e+00 0.0 1.4e+04 5.7e+04 > 0.0e+00 1 0100100 0 1 0100100 0 0 > > VecScatterEnd 14400 1.0 4.1534e+00 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 > > KSPSetup 4 1.0 1.1060e-0212.6 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > PCSetUp 4 1.0 3.4280e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 5.6e+01 3 0 0 0 0 3 0 0 0 32 0 > > > ------------------------------------------------------------------------------------------------------------------------ > > > > Memory usage is given in bytes: > > > > Object Type Creations Destructions Memory Descendants' > Mem. > > Reports information only for process 0. > > > > --- Event Stage 0: Main Stage > > > > Matrix 27 27 208196712 0 > > Vec 36 36 1027376 0 > > Vec Scatter 11 11 7220 0 > > Index Set 42 42 22644 0 > > Krylov Solver 1 1 34432 0 > > Preconditioner 1 1 752 0 > > > ======================================================================================================================== > > Average time to get PetscTime(): 1.90735e-07 > > Average time for MPI_Barrier(): 3.8147e-06 > > Average time for zero size MPI_Send(): 7.51019e-06 > > #PETSc Option Table entries: > > -log_summary > > -pc_factor_mat_solver_package mumps > > -pc_type lu > > #End of PETSc Option Table entries > > Compiled without FORTRAN kernels > > Compiled with full precision matrices (default) > > sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 > sizeof(PetscScalar) 16 > > Configure run at: Mon Jul 11 15:28:42 2011 > > Configure options: PETSC_ARCH=complex-cpp-mumps --with-cc=mpicc > --with-fc=mpif90 --with-blas-lapack-dir=/usr/lib64 --with-shared > --with-clanguage=c++ --with-scalar-type=complex --download-mumps=1 > --download-blacs=1 --download-scalapack=1 --download-parmetis=1 > --with-cxx=mpicxx > > ----------------------------------------- > > Libraries compiled on Mon Jul 11 15:39:58 EDT 2011 on sc.local > > Machine characteristics: Linux sc.local 2.6.18-194.11.1.el5 #1 SMP Tue > Aug 10 19:05:06 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux > > Using PETSc directory: /panfs/storage.local/scs/home/abyrd/petsc-3.1-p8 > > Using PETSc arch: complex-cpp-mumps > > ----------------------------------------- > > Using C compiler: mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -g > -fPIC > > Using Fortran compiler: mpif90 -fPIC -Wall -Wno-unused-variable -g > > ----------------------------------------- > > Using include paths: > -I/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/include > -I/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/include > -I/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/include > -I/usr/mpi/gnu/openmpi-1.4.2/include -I/usr/mpi/gnu/openmpi-1.4.2/lib64 > > ------------------------------------------ > > Using C linker: mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -g > > Using Fortran linker: mpif90 -fPIC -Wall -Wno-unused-variable -g > > Using libraries: > -Wl,-rpath,/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib > -L/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib > -lpetsc -lX11 > -Wl,-rpath,/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib > -L/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib > -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis > -lscalapack -lblacs -Wl,-rpath,/usr/lib64 -L/usr/lib64 -llapack -lblas -lnsl > -lrt -Wl,-rpath,/usr/mpi/gnu/openmpi-1.4.2/lib64 > -L/usr/mpi/gnu/openmpi-1.4.2/lib64 > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.1.2 > -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl -lmpi -lopen-rte -lopen-pal > -lnsl -lutil -lgcc_s -lpthread -lmpi_f90 -lmpi_f77 -lgfortran -lm -lm -lm > -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lopen-rte -lopen-pal > -lnsl -lutil -lgcc_s -lpthread -ldl > > > > Respectfully, > > Adam Byrd > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Aug 1 16:34:30 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 1 Aug 2011 21:34:30 +0000 Subject: [petsc-users] Optimizing MatMatSolve In-Reply-To: References: <5CBFD41E-55B6-4AA4-9C17-4A2984EAB4D4@mcs.anl.gov> Message-ID: On Mon, Aug 1, 2011 at 9:31 PM, Adam Byrd wrote: > On Mon, Aug 1, 2011 at 5:09 PM, Barry Smith wrote: > >> >> On Aug 1, 2011, at 3:00 PM, Adam Byrd wrote: >> >> > Hello, >> > >> > I'm looking for help reducing the time and communication of a parallel >> MatMatSolve using MUMPS. On a single processor I experience decent solve >> times (~9 seconds each), but when moving to multiple processors I see longer >> times with more cores. I've run with -log_summary and confirmed >> (practically) all the time is spent in MatMatSolve. I'm fairly certain it's >> all communication between nodes and I'm trying to figure out where I can >> make optimizations, or if it is even feasible for this type of problem. It >> is a parallel, dense, >> >> I hope you mean that the original matrix you use with MUMPS is sparse >> (you should not use MUMPS to solve dense linear systems). >> > > Oops, yes. The original matrix is sparse. It requires the solution and > identity matrix to be dense. I was typing faster than thinking. > >> >> > direct solve using MUMPS with an LU preconditioner. I know there are >> many smaller optimizations that can be done in other areas, but at the >> moment it is only the solve that concerns me. >> >> MUMPS will run slower on 2 processors than 1, this is just a fact of >> life. You will only gain with parallel for MUMPS for large problems. >> > > I see. It looks like I took off in the wrong direction then. I'm trying to > solve for the inverse of a sparse matrix in parallel. I'm starting at > 3600x3600 and will be moving to 30,000x30,000+ in the future. Which solver > suits this sort of problem? > The key to parallel computing (and most other things) is choosing the right problem.This unfortunately, is not a problem that lends itself to parallelism. Matt > >> Barry >> >> >> >> > >> > ---------------------------------------------- PETSc Performance >> Summary: ---------------------------------------------- >> > >> > ./cntor on a complex-c named hpc-1-0.local with 2 processors, by abyrd >> Mon Aug 1 16:25:51 2011 >> > Using Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 >> > >> > Max Max/Min Avg Total >> > Time (sec): 1.307e+02 1.00000 1.307e+02 >> > Objects: 1.180e+02 1.00000 1.180e+02 >> > Flops: 0.000e+00 0.00000 0.000e+00 0.000e+00 >> > Flops/sec: 0.000e+00 0.00000 0.000e+00 0.000e+00 >> > Memory: 2.091e+08 1.00001 4.181e+08 >> > MPI Messages: 7.229e+03 1.00000 7.229e+03 1.446e+04 >> > MPI Message Lengths: 4.141e+08 1.00000 5.729e+04 8.283e+08 >> > MPI Reductions: 1.464e+04 1.00000 >> > >> > Flop counting convention: 1 flop = 1 real number operation of type >> (multiply/divide/add/subtract) >> > e.g., VecAXPY() for real vectors of length N >> --> 2N flops >> > and VecAXPY() for complex vectors of length >> N --> 8N flops >> > >> > Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages >> --- -- Message Lengths -- -- Reductions -- >> > Avg %Total Avg %Total counts >> %Total Avg %Total counts %Total >> > 0: Main Stage: 1.3072e+02 100.0% 0.0000e+00 0.0% 1.446e+04 >> 100.0% 5.729e+04 100.0% 1.730e+02 1.2% >> > >> > >> ------------------------------------------------------------------------------------------------------------------------ >> > See the 'Profiling' chapter of the users' manual for details on >> interpreting output. >> > Phase summary info: >> > Count: number of times phase was executed >> > Time and Flops: Max - maximum over all processors >> > Ratio - ratio of maximum to minimum over all >> processors >> > Mess: number of messages sent >> > Avg. len: average message length >> > Reduct: number of global reductions >> > Global: entire computation >> > Stage: stages of a computation. Set stages with PetscLogStagePush() >> and PetscLogStagePop(). >> > %T - percent time in this phase %F - percent flops in this >> phase >> > %M - percent messages in this phase %L - percent message >> lengths in this phase >> > %R - percent reductions in this phase >> > Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time >> over all processors) >> > >> ------------------------------------------------------------------------------------------------------------------------ >> > >> > >> > ########################################################## >> > # # >> > # WARNING!!! # >> > # # >> > # This code was compiled with a debugging option, # >> > # To get timing results run config/configure.py # >> > # using --with-debugging=no, the performance will # >> > # be generally two or three times faster. # >> > # # >> > ########################################################## >> > >> > >> > >> > >> > ########################################################## >> > # # >> > # WARNING!!! # >> > # # >> > # The code for various complex numbers numerical # >> > # kernels uses C++, which generally is not well # >> > # optimized. For performance that is about 4-5 times # >> > # faster, specify --with-fortran-kernels=1 # >> > # when running config/configure.py. # >> > # # >> > ########################################################## >> > >> > >> > Event Count Time (sec) Flops >> --- Global --- --- Stage --- Total >> > Max Ratio Max Ratio Max Ratio Mess Avg len >> Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >> > >> ------------------------------------------------------------------------------------------------------------------------ >> > >> > --- Event Stage 0: Main Stage >> > >> > MatSolve 14400 1.0 1.2364e+02 1.0 0.00e+00 0.0 1.4e+04 5.7e+04 >> 2.0e+01 95 0100100 0 95 0100100 12 0 >> > MatLUFactorSym 4 1.0 2.0027e-05 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> > MatLUFactorNum 4 1.0 3.4223e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 2.4e+01 3 0 0 0 0 3 0 0 0 14 0 >> > MatConvert 1 1.0 2.3644e-01 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 >> 1.1e+01 0 0 0 0 0 0 0 0 0 6 0 >> > MatAssemblyBegin 14 1.0 1.9959e-01 9.3 0.00e+00 0.0 3.0e+01 5.2e+04 >> 1.2e+01 0 0 0 0 0 0 0 0 0 7 0 >> > MatAssemblyEnd 14 1.0 1.9908e-01 1.1 0.00e+00 0.0 4.0e+00 2.8e+01 >> 2.0e+01 0 0 0 0 0 0 0 0 0 12 0 >> > MatGetRow 32 1.0 4.2677e-05 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> > MatGetSubMatrice 4 1.0 7.6661e-03 1.0 0.00e+00 0.0 1.6e+01 1.2e+05 >> 2.4e+01 0 0 0 0 0 0 0 0 0 14 0 >> > MatMatSolve 4 1.0 1.2380e+02 1.0 0.00e+00 0.0 1.4e+04 5.7e+04 >> 2.0e+01 95 0100100 0 95 0100100 12 0 >> > VecSet 4 1.0 1.8590e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> > VecScatterBegin 28800 1.0 2.2810e+00 2.2 0.00e+00 0.0 1.4e+04 5.7e+04 >> 0.0e+00 1 0100100 0 1 0100100 0 0 >> > VecScatterEnd 14400 1.0 4.1534e+00 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 >> > KSPSetup 4 1.0 1.1060e-0212.6 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> > PCSetUp 4 1.0 3.4280e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 5.6e+01 3 0 0 0 0 3 0 0 0 32 0 >> > >> ------------------------------------------------------------------------------------------------------------------------ >> > >> > Memory usage is given in bytes: >> > >> > Object Type Creations Destructions Memory Descendants' >> Mem. >> > Reports information only for process 0. >> > >> > --- Event Stage 0: Main Stage >> > >> > Matrix 27 27 208196712 0 >> > Vec 36 36 1027376 0 >> > Vec Scatter 11 11 7220 0 >> > Index Set 42 42 22644 0 >> > Krylov Solver 1 1 34432 0 >> > Preconditioner 1 1 752 0 >> > >> ======================================================================================================================== >> > Average time to get PetscTime(): 1.90735e-07 >> > Average time for MPI_Barrier(): 3.8147e-06 >> > Average time for zero size MPI_Send(): 7.51019e-06 >> > #PETSc Option Table entries: >> > -log_summary >> > -pc_factor_mat_solver_package mumps >> > -pc_type lu >> > #End of PETSc Option Table entries >> > Compiled without FORTRAN kernels >> > Compiled with full precision matrices (default) >> > sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 >> sizeof(PetscScalar) 16 >> > Configure run at: Mon Jul 11 15:28:42 2011 >> > Configure options: PETSC_ARCH=complex-cpp-mumps --with-cc=mpicc >> --with-fc=mpif90 --with-blas-lapack-dir=/usr/lib64 --with-shared >> --with-clanguage=c++ --with-scalar-type=complex --download-mumps=1 >> --download-blacs=1 --download-scalapack=1 --download-parmetis=1 >> --with-cxx=mpicxx >> > ----------------------------------------- >> > Libraries compiled on Mon Jul 11 15:39:58 EDT 2011 on sc.local >> > Machine characteristics: Linux sc.local 2.6.18-194.11.1.el5 #1 SMP Tue >> Aug 10 19:05:06 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux >> > Using PETSc directory: /panfs/storage.local/scs/home/abyrd/petsc-3.1-p8 >> > Using PETSc arch: complex-cpp-mumps >> > ----------------------------------------- >> > Using C compiler: mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -g >> -fPIC >> > Using Fortran compiler: mpif90 -fPIC -Wall -Wno-unused-variable -g >> > ----------------------------------------- >> > Using include paths: >> -I/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/include >> -I/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/include >> -I/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/include >> -I/usr/mpi/gnu/openmpi-1.4.2/include -I/usr/mpi/gnu/openmpi-1.4.2/lib64 >> > ------------------------------------------ >> > Using C linker: mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -g >> > Using Fortran linker: mpif90 -fPIC -Wall -Wno-unused-variable -g >> > Using libraries: >> -Wl,-rpath,/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib >> -L/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib >> -lpetsc -lX11 >> -Wl,-rpath,/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib >> -L/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib >> -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis >> -lscalapack -lblacs -Wl,-rpath,/usr/lib64 -L/usr/lib64 -llapack -lblas -lnsl >> -lrt -Wl,-rpath,/usr/mpi/gnu/openmpi-1.4.2/lib64 >> -L/usr/mpi/gnu/openmpi-1.4.2/lib64 >> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.1.2 >> -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl -lmpi -lopen-rte -lopen-pal >> -lnsl -lutil -lgcc_s -lpthread -lmpi_f90 -lmpi_f77 -lgfortran -lm -lm -lm >> -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lopen-rte -lopen-pal >> -lnsl -lutil -lgcc_s -lpthread -ldl >> > >> > Respectfully, >> > Adam Byrd >> > >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From aron.ahmadia at kaust.edu.sa Mon Aug 1 16:43:04 2011 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Tue, 2 Aug 2011 00:43:04 +0300 Subject: [petsc-users] Optimizing MatMatSolve In-Reply-To: References: <5CBFD41E-55B6-4AA4-9C17-4A2984EAB4D4@mcs.anl.gov> Message-ID: What Matt is getting at is that typically we measure the computational difficulty of a problem as a function of the 'unknowns'. If you are looking at turning a sparse matrix O(n) bytes into a dense inverse O(n^2) bytes, you've taken what was originally a potentially optimal problem and turned it into one of quadratic difficulty in terms of memory, and even more in terms of flops. You may have better luck using an explicitly dense method/algorithm for computing the inverse: it will scale well in the sense that adding more processors will work faster (this is the heart of the LINPACK computation after all), but you will need a lot of core-hours and memory to get there... A -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Aug 1 16:56:42 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 1 Aug 2011 15:56:42 -0600 Subject: [petsc-users] Optimizing MatMatSolve In-Reply-To: References: <5CBFD41E-55B6-4AA4-9C17-4A2984EAB4D4@mcs.anl.gov> Message-ID: On Aug 1, 2011, at 3:31 PM, Adam Byrd wrote: > On Mon, Aug 1, 2011 at 5:09 PM, Barry Smith wrote: > > On Aug 1, 2011, at 3:00 PM, Adam Byrd wrote: > > > Hello, > > > > I'm looking for help reducing the time and communication of a parallel MatMatSolve using MUMPS. On a single processor I experience decent solve times (~9 seconds each), but when moving to multiple processors I see longer times with more cores. I've run with -log_summary and confirmed (practically) all the time is spent in MatMatSolve. I'm fairly certain it's all communication between nodes and I'm trying to figure out where I can make optimizations, or if it is even feasible for this type of problem. It is a parallel, dense, > > I hope you mean that the original matrix you use with MUMPS is sparse (you should not use MUMPS to solve dense linear systems). > > Oops, yes. The original matrix is sparse. It requires the solution and identity matrix to be dense. I was typing faster than thinking. > > > direct solve using MUMPS with an LU preconditioner. I know there are many smaller optimizations that can be done in other areas, but at the moment it is only the solve that concerns me. > > MUMPS will run slower on 2 processors than 1, this is just a fact of life. You will only gain with parallel for MUMPS for large problems. > > I see. It looks like I took off in the wrong direction then. I'm trying to solve for the inverse of a sparse matrix in parallel. I'm starting at 3600x3600 and will be moving to 30,000x30,000+ in the future. Which solver suits this sort of problem? Actually this problem is so small that you will not want to parallelize inside the solve, instead you will want to have different processes do different right hand sides and each process will have called MUMPS on the entire matrix. There will be no communication in the solver and you will get very good speed up. Barry > > Barry > > > > > > > ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- > > > > ./cntor on a complex-c named hpc-1-0.local with 2 processors, by abyrd Mon Aug 1 16:25:51 2011 > > Using Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 > > > > Max Max/Min Avg Total > > Time (sec): 1.307e+02 1.00000 1.307e+02 > > Objects: 1.180e+02 1.00000 1.180e+02 > > Flops: 0.000e+00 0.00000 0.000e+00 0.000e+00 > > Flops/sec: 0.000e+00 0.00000 0.000e+00 0.000e+00 > > Memory: 2.091e+08 1.00001 4.181e+08 > > MPI Messages: 7.229e+03 1.00000 7.229e+03 1.446e+04 > > MPI Message Lengths: 4.141e+08 1.00000 5.729e+04 8.283e+08 > > MPI Reductions: 1.464e+04 1.00000 > > > > Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) > > e.g., VecAXPY() for real vectors of length N --> 2N flops > > and VecAXPY() for complex vectors of length N --> 8N flops > > > > Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- > > Avg %Total Avg %Total counts %Total Avg %Total counts %Total > > 0: Main Stage: 1.3072e+02 100.0% 0.0000e+00 0.0% 1.446e+04 100.0% 5.729e+04 100.0% 1.730e+02 1.2% > > > > ------------------------------------------------------------------------------------------------------------------------ > > See the 'Profiling' chapter of the users' manual for details on interpreting output. > > Phase summary info: > > Count: number of times phase was executed > > Time and Flops: Max - maximum over all processors > > Ratio - ratio of maximum to minimum over all processors > > Mess: number of messages sent > > Avg. len: average message length > > Reduct: number of global reductions > > Global: entire computation > > Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). > > %T - percent time in this phase %F - percent flops in this phase > > %M - percent messages in this phase %L - percent message lengths in this phase > > %R - percent reductions in this phase > > Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) > > ------------------------------------------------------------------------------------------------------------------------ > > > > > > ########################################################## > > # # > > # WARNING!!! # > > # # > > # This code was compiled with a debugging option, # > > # To get timing results run config/configure.py # > > # using --with-debugging=no, the performance will # > > # be generally two or three times faster. # > > # # > > ########################################################## > > > > > > > > > > ########################################################## > > # # > > # WARNING!!! # > > # # > > # The code for various complex numbers numerical # > > # kernels uses C++, which generally is not well # > > # optimized. For performance that is about 4-5 times # > > # faster, specify --with-fortran-kernels=1 # > > # when running config/configure.py. # > > # # > > ########################################################## > > > > > > Event Count Time (sec) Flops --- Global --- --- Stage --- Total > > Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > > ------------------------------------------------------------------------------------------------------------------------ > > > > --- Event Stage 0: Main Stage > > > > MatSolve 14400 1.0 1.2364e+02 1.0 0.00e+00 0.0 1.4e+04 5.7e+04 2.0e+01 95 0100100 0 95 0100100 12 0 > > MatLUFactorSym 4 1.0 2.0027e-05 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > MatLUFactorNum 4 1.0 3.4223e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01 3 0 0 0 0 3 0 0 0 14 0 > > MatConvert 1 1.0 2.3644e-01 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.1e+01 0 0 0 0 0 0 0 0 0 6 0 > > MatAssemblyBegin 14 1.0 1.9959e-01 9.3 0.00e+00 0.0 3.0e+01 5.2e+04 1.2e+01 0 0 0 0 0 0 0 0 0 7 0 > > MatAssemblyEnd 14 1.0 1.9908e-01 1.1 0.00e+00 0.0 4.0e+00 2.8e+01 2.0e+01 0 0 0 0 0 0 0 0 0 12 0 > > MatGetRow 32 1.0 4.2677e-05 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > MatGetSubMatrice 4 1.0 7.6661e-03 1.0 0.00e+00 0.0 1.6e+01 1.2e+05 2.4e+01 0 0 0 0 0 0 0 0 0 14 0 > > MatMatSolve 4 1.0 1.2380e+02 1.0 0.00e+00 0.0 1.4e+04 5.7e+04 2.0e+01 95 0100100 0 95 0100100 12 0 > > VecSet 4 1.0 1.8590e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > VecScatterBegin 28800 1.0 2.2810e+00 2.2 0.00e+00 0.0 1.4e+04 5.7e+04 0.0e+00 1 0100100 0 1 0100100 0 0 > > VecScatterEnd 14400 1.0 4.1534e+00 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 > > KSPSetup 4 1.0 1.1060e-0212.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > PCSetUp 4 1.0 3.4280e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 5.6e+01 3 0 0 0 0 3 0 0 0 32 0 > > ------------------------------------------------------------------------------------------------------------------------ > > > > Memory usage is given in bytes: > > > > Object Type Creations Destructions Memory Descendants' Mem. > > Reports information only for process 0. > > > > --- Event Stage 0: Main Stage > > > > Matrix 27 27 208196712 0 > > Vec 36 36 1027376 0 > > Vec Scatter 11 11 7220 0 > > Index Set 42 42 22644 0 > > Krylov Solver 1 1 34432 0 > > Preconditioner 1 1 752 0 > > ======================================================================================================================== > > Average time to get PetscTime(): 1.90735e-07 > > Average time for MPI_Barrier(): 3.8147e-06 > > Average time for zero size MPI_Send(): 7.51019e-06 > > #PETSc Option Table entries: > > -log_summary > > -pc_factor_mat_solver_package mumps > > -pc_type lu > > #End of PETSc Option Table entries > > Compiled without FORTRAN kernels > > Compiled with full precision matrices (default) > > sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 16 > > Configure run at: Mon Jul 11 15:28:42 2011 > > Configure options: PETSC_ARCH=complex-cpp-mumps --with-cc=mpicc --with-fc=mpif90 --with-blas-lapack-dir=/usr/lib64 --with-shared --with-clanguage=c++ --with-scalar-type=complex --download-mumps=1 --download-blacs=1 --download-scalapack=1 --download-parmetis=1 --with-cxx=mpicxx > > ----------------------------------------- > > Libraries compiled on Mon Jul 11 15:39:58 EDT 2011 on sc.local > > Machine characteristics: Linux sc.local 2.6.18-194.11.1.el5 #1 SMP Tue Aug 10 19:05:06 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux > > Using PETSc directory: /panfs/storage.local/scs/home/abyrd/petsc-3.1-p8 > > Using PETSc arch: complex-cpp-mumps > > ----------------------------------------- > > Using C compiler: mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -g -fPIC > > Using Fortran compiler: mpif90 -fPIC -Wall -Wno-unused-variable -g > > ----------------------------------------- > > Using include paths: -I/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/include -I/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/include -I/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/include -I/usr/mpi/gnu/openmpi-1.4.2/include -I/usr/mpi/gnu/openmpi-1.4.2/lib64 > > ------------------------------------------ > > Using C linker: mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -g > > Using Fortran linker: mpif90 -fPIC -Wall -Wno-unused-variable -g > > Using libraries: -Wl,-rpath,/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib -L/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib -lpetsc -lX11 -Wl,-rpath,/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib -L/panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -Wl,-rpath,/usr/lib64 -L/usr/lib64 -llapack -lblas -lnsl -lrt -Wl,-rpath,/usr/mpi/gnu/openmpi-1.4.2/lib64 -L/usr/mpi/gnu/openmpi-1.4.2/lib64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -lmpi_f90 -lmpi_f77 -lgfortran -lm -lm -lm -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl > > > > Respectfully, > > Adam Byrd > > > > From jack.poulson at gmail.com Mon Aug 1 16:57:08 2011 From: jack.poulson at gmail.com (Jack Poulson) Date: Mon, 1 Aug 2011 16:57:08 -0500 Subject: [petsc-users] Optimizing MatMatSolve In-Reply-To: References: <5CBFD41E-55B6-4AA4-9C17-4A2984EAB4D4@mcs.anl.gov> Message-ID: Agreed. According to your timings, it took ~130 seconds to invert a 3000 x 3000 matrix. With dense algorithms, the cost of LU with partial pivoting is 2/3 n^3, and performing triangle solves against n right hand sides requires n^3 flops. Thus the total work is 5/3 n^3. With n=3000, this is roughly 45 billion flops. Nearly any machine can easily hit at least 50% of theoretical peak on both of these operations, and thus, if your laptop has a theoretical peak of 5 GFlops, it should take 18 seconds with a dense algorithm. This is a pretty substantial improvement over what you currently are seeing, and dense algorithms parallelize nicely since there are O(n^3) operations to perform and only O(n^2) data. On the other hand, when you increase your problem size to n ~= 30,000, the asymptotic difference between sparse and dense algorithms will start to become obvious. In 2d, sparse factorization is O(n^3/2) and the cost of doing n triangle solves is O(n^2 log(n)), leading to a total complexity of O(n^2 log(n)) versus the dense O(n^3) approach. For 3d problems, the cost of the triangle solves again dominates at O(n^7/3) versus the direct O(n^3) cost. As a practical matter, according to the information you attached, you are both running in debug mode and not using optimized computational kernels. If you fix both of these issues, your performance should increase substantially. Jack On Mon, Aug 1, 2011 at 4:43 PM, Aron Ahmadia wrote: > What Matt is getting at is that typically we measure the computational > difficulty of a problem as a function of the 'unknowns'. If you are looking > at turning a sparse matrix O(n) bytes into a dense inverse O(n^2) bytes, > you've taken what was originally a potentially optimal problem and turned it > into one of quadratic difficulty in terms of memory, and even more in terms > of flops. You may have better luck using an explicitly dense > method/algorithm for computing the inverse: it will scale well in the sense > that adding more processors will work faster (this is the heart of the > LINPACK computation after all), but you will need a lot of core-hours and > memory to get there... > > A > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam1.byrd at gmail.com Mon Aug 1 17:24:47 2011 From: adam1.byrd at gmail.com (Adam Byrd) Date: Mon, 1 Aug 2011 18:24:47 -0400 Subject: [petsc-users] Optimizing MatMatSolve In-Reply-To: References: <5CBFD41E-55B6-4AA4-9C17-4A2984EAB4D4@mcs.anl.gov> Message-ID: Thanks for all the info. The impression I'm getting is I'm probably best off splitting up the problem in such a way that each node does it's own sequential solves. I believe this is feasible and relatively easy with my problem. I'm doing tens of thousands of solves that should be able to be done independently of one another. Thanks for the tip about the configuration options, I'm planning on using them once the code is more polished. On Mon, Aug 1, 2011 at 5:57 PM, Jack Poulson wrote: > Agreed. According to your timings, it took ~130 seconds to invert a 3000 x > 3000 matrix. With dense algorithms, the cost of LU with partial pivoting is > 2/3 n^3, and performing triangle solves against n right hand sides requires > n^3 flops. Thus the total work is 5/3 n^3. With n=3000, this is roughly 45 > billion flops. Nearly any machine can easily hit at least 50% of theoretical > peak on both of these operations, and thus, if your laptop has a theoretical > peak of 5 GFlops, it should take 18 seconds with a dense algorithm. This is > a pretty substantial improvement over what you currently are seeing, and > dense algorithms parallelize nicely since there are O(n^3) operations to > perform and only O(n^2) data. > > On the other hand, when you increase your problem size to n ~= 30,000, the > asymptotic difference between sparse and dense algorithms will start to > become obvious. In 2d, sparse factorization is O(n^3/2) and the cost of > doing n triangle solves is O(n^2 log(n)), leading to a total complexity of > O(n^2 log(n)) versus the dense O(n^3) approach. For 3d problems, the cost of > the triangle solves again dominates at O(n^7/3) versus the direct O(n^3) > cost. > > As a practical matter, according to the information you attached, you are > both running in debug mode and not using optimized computational kernels. If > you fix both of these issues, your performance should increase > substantially. > > Jack > > > On Mon, Aug 1, 2011 at 4:43 PM, Aron Ahmadia wrote: > >> What Matt is getting at is that typically we measure the computational >> difficulty of a problem as a function of the 'unknowns'. If you are looking >> at turning a sparse matrix O(n) bytes into a dense inverse O(n^2) bytes, >> you've taken what was originally a potentially optimal problem and turned it >> into one of quadratic difficulty in terms of memory, and even more in terms >> of flops. You may have better luck using an explicitly dense >> method/algorithm for computing the inverse: it will scale well in the sense >> that adding more processors will work faster (this is the heart of the >> LINPACK computation after all), but you will need a lot of core-hours and >> memory to get there... >> >> A >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Aug 1 21:17:54 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Aug 2011 02:17:54 +0000 Subject: [petsc-users] PETSc Mesh and Fortran In-Reply-To: References: Message-ID: On Wed, Apr 6, 2011 at 8:11 PM, Dharmendar Reddy wrote: > Hello, > Are there any examples of PETSc mesh usage in a Fortran code. The > Examples link on PETSc mesh man page ( > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mesh/index.html) > redirects to page not found. I would like to learn usage of PETSc mesh > utilities. I have an exodus file for the mesh. > 1) In petsc-dev, there is now SNES ex12 which uses Mesh to solve a simple equation 2) I will be in Austin giving a tutorial Sept. 5 3) There is a DMMeshCreateExodus() which is able to extract some information from ExodusII files Thanks, Matt > Thanks > Reddy > > -- > ----------------------------------------------------- > Dharmendar Reddy Palle > Graduate Student > Microelectronics Research center, > University of Texas at Austin, > 10100 Burnet Road, Bldg. 160 > MER 2.608F, TX 78758-4445 > e-mail: dharmareddy84 at gmail.com > Phone: +1-512-350-9082 > United States of America. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jchludzinski at gmail.com Tue Aug 2 02:24:09 2011 From: jchludzinski at gmail.com (John Chludzinski) Date: Tue, 2 Aug 2011 03:24:09 -0400 Subject: [petsc-users] SLEPc generalized eigenvalue problem question? In-Reply-To: <7D43440C-51C4-4195-AA82-436BA951558B@dsic.upv.es> References: <7D43440C-51C4-4195-AA82-436BA951558B@dsic.upv.es> Message-ID: Jose, I used: MatCreateSeqAIJ(PETSC_COMM_SELF, n, n, 0, cnt, &A); MatSetValue(A, i, j, *(a+i*SIZE+j), INSERT_VALUES); to create the input matrices. Their sizes collapsed dramatically: from 192208072 to 239944. But again, using: ./ex7.exe -f1 k.dat -f2 m.dat -eps_gen_hermitian -eps_smallest_real > x.out 2>&1, I get: Generalized eigenproblem stored in file. Reading REAL matrices from binary files... Number of iterations of the method: 500 Number of linear iterations of the method: 4009 Solution method: krylovschur Number of requested eigenvalues: 1 Stopping condition: tol=1e-07, maxit=500 Number of converged approximate eigenpairs: 0 I'm really only interested in the least 50 eigenvalues. Smallest 10 eigenvalues -0.000019 -0.000004 -0.000000 0.000005 0.000012 0.000016 2.795284 2.795307 21.235339 21.235340 81.582017 Largest 10 eigenvalues 176431487319.625000 176431532012.467468 176431562378.361359 176435480628.136292 176435488209.944031 176435555689.747253 176435563270.922424 663312473823.916260 663312473823.917969 663312666285.928589 663312666285.929810 On Mon, Aug 1, 2011 at 1:46 PM, Jose E. Roman wrote: El 01/08/2011, a las 19:27, John Chludzinski escribi?: > > > I create 2 matrices using: > > > > MatCreateSeqDense(PETSC_COMM_SELF, n, n, Ka, &A); > > MatCreateSeqDense(PETSC_COMM_SELF, n, n, Kb, &B); > > > > These matrices are 99% zeros ( 16,016,004 entries and 18660 non-zeros). > They are symmetric and real. Their tri-diagonal elements are non-zero plus > a few other entries. > > > > I tried to use ex7 for the generalized eigenvalue problem: > > > > ./ex7.exe -f1 k.dat -f2 m.dat -eps_gen_hermitian -eps_smallest_real > > x.out 2>&1 > > > > without specifying an EPS and get: > > > > Generalized eigenproblem stored in file. > > > > Reading REAL matrices from binary files... > > Number of iterations of the method: 500 > > Number of linear iterations of the method: 4009 > > Solution method: krylovschur > > > > Number of requested eigenvalues: 1 > > Stopping condition: tol=1e-07, maxit=500 > > Number of converged approximate eigenpairs: 0 > > > > Is krylovschur inappropriate for this problem or have I set up the > problem incorrectly by using MatCreateSeqDense(...) to create the matrix > input files in PETSc binary form? > > > > ---John > > > > The solver has reached the maximum number of iterations. Do you want to > compute the leftmost part of the spectrum? Are those eigenvalues > (relatively) large in magnitude? I guess not. If you need the smallest > eigenvalues, instead of computing the smallest eigenvalues of (K,M) try > computing the largest eigenvalues of (M,K) and then compute the reciprocals. > Also, have a look at the chapter on spectral transformations. > > And of course do not use dense matrices, as pointed out by Matt. > Jose > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Tue Aug 2 02:40:38 2011 From: jroman at dsic.upv.es (Jose E. Roman) Date: Tue, 2 Aug 2011 09:40:38 +0200 Subject: [petsc-users] SLEPc generalized eigenvalue problem question? In-Reply-To: References: <7D43440C-51C4-4195-AA82-436BA951558B@dsic.upv.es> Message-ID: El 02/08/2011, a las 09:24, John Chludzinski escribi?: > Jose, > > I used: > > MatCreateSeqAIJ(PETSC_COMM_SELF, n, n, 0, cnt, &A); > MatSetValue(A, i, j, *(a+i*SIZE+j), INSERT_VALUES); > > to create the input matrices. Their sizes collapsed dramatically: from 192208072 to 239944. But again, using: ./ex7.exe -f1 k.dat -f2 m.dat -eps_gen_hermitian -eps_smallest_real > x.out 2>&1, I get: > > Generalized eigenproblem stored in file. > > Reading REAL matrices from binary files... > Number of iterations of the method: 500 > Number of linear iterations of the method: 4009 > Solution method: krylovschur > > Number of requested eigenvalues: 1 > Stopping condition: tol=1e-07, maxit=500 > Number of converged approximate eigenpairs: 0 > > I'm really only interested in the least 50 eigenvalues. > > Smallest 10 eigenvalues > -0.000019 > -0.000004 > -0.000000 > 0.000005 > 0.000012 > 0.000016 > 2.795284 > 2.795307 > 21.235339 > 21.235340 > 81.582017 > > Largest 10 eigenvalues > 176431487319.625000 > 176431532012.467468 > 176431562378.361359 > 176435480628.136292 > 176435488209.944031 > 176435555689.747253 > 176435563270.922424 > 663312473823.916260 > 663312473823.917969 > 663312666285.928589 > 663312666285.929810 > You will never get convergence to the small eigenvalues in this way, since the largest ones are huge. Try something like: ./ex7.exe -f1 k.dat -f2 m.dat -eps_gen_hermitian -st_type sinvert -eps_target 1.0 Jose From jchludzinski at gmail.com Tue Aug 2 02:44:58 2011 From: jchludzinski at gmail.com (John Chludzinski) Date: Tue, 2 Aug 2011 03:44:58 -0400 Subject: [petsc-users] SLEPc generalized eigenvalue problem question? In-Reply-To: References: <7D43440C-51C4-4195-AA82-436BA951558B@dsic.upv.es> Message-ID: Looks good but it only gives 4 eigenvalues. How do I get the 50 smallest eigenvalues? ---John On Tue, Aug 2, 2011 at 3:40 AM, Jose E. Roman wrote: > > El 02/08/2011, a las 09:24, John Chludzinski escribi?: > > > Jose, > > > > I used: > > > > MatCreateSeqAIJ(PETSC_COMM_SELF, n, n, 0, cnt, &A); > > MatSetValue(A, i, j, *(a+i*SIZE+j), INSERT_VALUES); > > > > to create the input matrices. Their sizes collapsed dramatically: from > 192208072 to 239944. But again, using: ./ex7.exe -f1 k.dat -f2 m.dat > -eps_gen_hermitian -eps_smallest_real > x.out 2>&1, I get: > > > > Generalized eigenproblem stored in file. > > > > Reading REAL matrices from binary files... > > Number of iterations of the method: 500 > > Number of linear iterations of the method: 4009 > > Solution method: krylovschur > > > > Number of requested eigenvalues: 1 > > Stopping condition: tol=1e-07, maxit=500 > > Number of converged approximate eigenpairs: 0 > > > > I'm really only interested in the least 50 eigenvalues. > > > > Smallest 10 eigenvalues > > -0.000019 > > -0.000004 > > -0.000000 > > 0.000005 > > 0.000012 > > 0.000016 > > 2.795284 > > 2.795307 > > 21.235339 > > 21.235340 > > 81.582017 > > > > Largest 10 eigenvalues > > 176431487319.625000 > > 176431532012.467468 > > 176431562378.361359 > > 176435480628.136292 > > 176435488209.944031 > > 176435555689.747253 > > 176435563270.922424 > > 663312473823.916260 > > 663312473823.917969 > > 663312666285.928589 > > 663312666285.929810 > > > > You will never get convergence to the small eigenvalues in this way, since > the largest ones are huge. Try something like: > ./ex7.exe -f1 k.dat -f2 m.dat -eps_gen_hermitian -st_type sinvert > -eps_target 1.0 > > Jose > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Tue Aug 2 03:01:05 2011 From: jroman at dsic.upv.es (Jose E. Roman) Date: Tue, 2 Aug 2011 10:01:05 +0200 Subject: [petsc-users] SLEPc generalized eigenvalue problem question? In-Reply-To: References: <7D43440C-51C4-4195-AA82-436BA951558B@dsic.upv.es> Message-ID: <39E41716-710A-42FB-97E2-7165A31E44B4@dsic.upv.es> El 02/08/2011, a las 09:44, John Chludzinski escribi?: > Looks good but it only gives 4 eigenvalues. How do I get the 50 smallest eigenvalues? > > ---John > -eps_nev 50 Please don't ask questions that can be trivially answered by having a look at the documentation. I won't answer any other question of this kind. Jose From aron.ahmadia at kaust.edu.sa Tue Aug 2 04:05:04 2011 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Tue, 2 Aug 2011 12:05:04 +0300 Subject: [petsc-users] SLEPc generalized eigenvalue problem question? In-Reply-To: References: Message-ID: > > Found a PDF, "MATRICES IN PETSc", (after much googling) but not sure which > of the many forms will work and which is best. > > ---John > > As a general piece of advice, always consult the documentation provided by the developers of a software package before typing search terms in Google. This is particularly harmful with PETSc, since there are about 10 years worth of tutorials and references scattered around the web, many of them for out of date versions of the toolkit. You are trying to use SLEPc functionality, so you should start with their documentation: http://www.grycap.upv.es/slepc/documentation/manual.htm and its user manual: http://www.grycap.upv.es/slepc/documentation/slepc.pdf In the case of PETSc, there is extensive documentation available from here: http://www.mcs.anl.gov/petsc/petsc-as/documentation/index.html You should start with the user manual: http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manual.pdf A -------------- next part -------------- An HTML attachment was scrubbed... URL: From jchludzinski at gmail.com Tue Aug 2 04:26:22 2011 From: jchludzinski at gmail.com (John Chludzinski) Date: Tue, 2 Aug 2011 05:26:22 -0400 Subject: [petsc-users] SLEPc generalized eigenvalue problem question? In-Reply-To: <39E41716-710A-42FB-97E2-7165A31E44B4@dsic.upv.es> References: <7D43440C-51C4-4195-AA82-436BA951558B@dsic.upv.es> <39E41716-710A-42FB-97E2-7165A31E44B4@dsic.upv.es> Message-ID: SORRY! But with the volumes of documentation + examples + new algorithms for MPI + PETSc + SLEPc, it's a bit like taking a sip from a fire hydrant. On Tue, Aug 2, 2011 at 4:01 AM, Jose E. Roman wrote: > > El 02/08/2011, a las 09:44, John Chludzinski escribi?: > > > Looks good but it only gives 4 eigenvalues. How do I get the 50 smallest > eigenvalues? > > > > ---John > > > > -eps_nev 50 > > Please don't ask questions that can be trivially answered by having a look > at the documentation. I won't answer any other question of this kind. > > Jose > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From clemens.domanig at uibk.ac.at Tue Aug 2 04:28:24 2011 From: clemens.domanig at uibk.ac.at (Clemens Domanig) Date: Tue, 02 Aug 2011 11:28:24 +0200 Subject: [petsc-users] PCFactorSetMatSolverPackage / PCFactorGetMatSolverPackage Message-ID: <4E37C338.9010406@uibk.ac.at> Hi, usually I'm using MUMPS by puting '-pc_factor_mat_solver_package mumps' into the .petscrc. Now I want to change in runtime between MUMPS and non-MUMPS so I tried to turn it on with PCFactorSetMatSolverPackage( pre, MAT_SOLVER_MUMPS); but I doesn't have any effect. I also have a problem with PCFactorGetMatSolverPackage(). How can I get back an output as described in the documentation that has the type 'const'? Thx - Clemens From knepley at gmail.com Tue Aug 2 06:36:55 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Aug 2011 11:36:55 +0000 Subject: [petsc-users] PCFactorSetMatSolverPackage / PCFactorGetMatSolverPackage In-Reply-To: <4E37C338.9010406@uibk.ac.at> References: <4E37C338.9010406@uibk.ac.at> Message-ID: On Tue, Aug 2, 2011 at 9:28 AM, Clemens Domanig wrote: > Hi, > > usually I'm using MUMPS by puting '-pc_factor_mat_solver_package mumps' > into the .petscrc. > Now I want to change in runtime between MUMPS and non-MUMPS so I tried to > turn it on with > PCFactorSetMatSolverPackage( pre, MAT_SOLVER_MUMPS); > but I doesn't have any effect. > This is complicated because you have to set this at the right time during the solver setup process. Command line arguments are much easier. However, you should call this after the preconditioner type is set, but before it is setup. > I also have a problem with PCFactorGetMatSolverPackage(). How can I get > back an output as described in the documentation that has the type 'const'? > What is the problem? I do not understand? Matt > Thx - Clemens > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From clemens.domanig at uibk.ac.at Tue Aug 2 07:53:24 2011 From: clemens.domanig at uibk.ac.at (Clemens Domanig) Date: Tue, 02 Aug 2011 14:53:24 +0200 Subject: [petsc-users] PCFactorSetMatSolverPackage / PCFactorGetMatSolverPackage In-Reply-To: References: <4E37C338.9010406@uibk.ac.at> Message-ID: <4E37F344.40104@uibk.ac.at> Maybe I should describe the hole problem. I'm using MUMPS to make LDLt decomposition - all from command line. But sometimes I need the diagonal entries (respectively the number of negative entries) of the D matrix. But there is no way to get this information back from MUMPS via petsc ( as far as I found out). There was the suggestion to use MatGetIntertia but this is only for off-diagonal-information. So I used PCSetType( prec, PCCHOLESKY); PCFactorGetMatrix( prec, &M); MatGetDiagonal( M, z); It only works without MUMPS and gives me the diagonal entries I want in Matrix M - although the entries are 1/x. Vector z is full of zeros - only the last entry is the 'same' as in M (x instead of 1/x). So the idea was to somehow turn off MUMPS if I need the diagonal entries and then turn it on again. The problem with PCFactorGetMatSolverPackage is I didn't manage to use it. const MatSolverPackage pack; PCFactorGetMatSolverPackage( prec, &pack); PetscSynchronizedPrintf( PETSC_COMM_WORLD, "%s\n", pack); I expected 'mumps' as output but got 'H??????tdH3%0' Thx Matthew Knepley wrote: > On Tue, Aug 2, 2011 at 9:28 AM, Clemens Domanig > > wrote: > > Hi, > > usually I'm using MUMPS by puting '-pc_factor_mat_solver_package > mumps' into the .petscrc. > Now I want to change in runtime between MUMPS and non-MUMPS so I > tried to turn it on with > PCFactorSetMatSolverPackage( pre, MAT_SOLVER_MUMPS); > but I doesn't have any effect. > > > This is complicated because you have to set this at the right time > during the solver setup process. > Command line arguments are much easier. However, you should call this > after the preconditioner > type is set, but before it is setup. > > > I also have a problem with PCFactorGetMatSolverPackage(). How can I > get back an output as described in the documentation that has the > type 'const'? > > > What is the problem? I do not understand? > > Matt > > > Thx - Clemens > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener From jedbrown at mcs.anl.gov Tue Aug 2 08:10:28 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 2 Aug 2011 07:10:28 -0600 Subject: [petsc-users] PCFactorSetMatSolverPackage / PCFactorGetMatSolverPackage In-Reply-To: <4E37F344.40104@uibk.ac.at> References: <4E37C338.9010406@uibk.ac.at> <4E37F344.40104@uibk.ac.at> Message-ID: On Tue, Aug 2, 2011 at 06:53, Clemens Domanig wrote: > The problem with PCFactorGetMatSolverPackage is I didn't manage to use it. > > const MatSolverPackage pack; > Insert here: PCSetType(prec,PCCHOLESKY); > PCFactorGetMatSolverPackage( prec, &pack); > The type has to be set first, otherwise the PC doesn't have any concept of a solver package. I have updated this function to return NULL instead of an undefined value in this case, thanks. Barry, speak up if you would prefer an error. I think PetscUseMethod() is somewhat overused in query functions at present because there is a reasonable non-error condition that can be returned. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Aug 2 08:12:33 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Aug 2011 13:12:33 +0000 Subject: [petsc-users] PCFactorSetMatSolverPackage / PCFactorGetMatSolverPackage In-Reply-To: <4E37F344.40104@uibk.ac.at> References: <4E37C338.9010406@uibk.ac.at> <4E37F344.40104@uibk.ac.at> Message-ID: On Tue, Aug 2, 2011 at 12:53 PM, Clemens Domanig wrote: > Maybe I should describe the hole problem. > I'm using MUMPS to make LDLt decomposition - all from command line. > But sometimes I need the diagonal entries (respectively the number of > negative entries) of the D matrix. But there is no way to get this > information back from MUMPS via petsc ( as far as I found out). Yes, we have no access to MUMPS internal data. > There was the suggestion to use MatGetIntertia but this is only for > off-diagonal-information. I am not sure what you mean by that. > So I used > PCSetType( prec, PCCHOLESKY); > PCFactorGetMatrix( prec, &M); > MatGetDiagonal( M, z); > It only works without MUMPS and gives me the diagonal entries I want in > Matrix M - although the entries are 1/x. > Vector z is full of zeros - only the last entry is the 'same' as in M (x > instead of 1/x). > > So the idea was to somehow turn off MUMPS if I need the diagonal entries > and then turn it on again. > Here is the easiest way to do it. Keep two complete solvers a) KSP with MUMPS b) KSP with Cholesky, give this one KSPSetOptionsPrefix(ksp, "diag_"); That way -diag_pc_type cholesky will work. Then use the solver you want. > The problem with PCFactorGetMatSolverPackage is I didn't manage to use it. > > const MatSolverPackage pack; > PCFactorGetMatSolverPackage( prec, &pack); > PetscSynchronizedPrintf( PETSC_COMM_WORLD, "%s\n", pack); > I expected 'mumps' as output but got 'H??????tdH3%0' > It would seem its not yet setup. Matt > Thx > > > > Matthew Knepley wrote: > > On Tue, Aug 2, 2011 at 9:28 AM, Clemens Domanig < >> clemens.domanig at uibk.ac.at >> >> wrote: >> >> Hi, >> >> usually I'm using MUMPS by puting '-pc_factor_mat_solver_package >> mumps' into the .petscrc. >> Now I want to change in runtime between MUMPS and non-MUMPS so I >> tried to turn it on with >> PCFactorSetMatSolverPackage( pre, MAT_SOLVER_MUMPS); >> but I doesn't have any effect. >> >> >> This is complicated because you have to set this at the right time during >> the solver setup process. >> Command line arguments are much easier. However, you should call this >> after the preconditioner >> type is set, but before it is setup. >> >> I also have a problem with PCFactorGetMatSolverPackage(). How can I >> get back an output as described in the documentation that has the >> type 'const'? >> >> >> What is the problem? I do not understand? >> >> Matt >> >> Thx - Clemens >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Tue Aug 2 09:48:24 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 2 Aug 2011 09:48:24 -0500 Subject: [petsc-users] PCFactorSetMatSolverPackage / PCFactorGetMatSolverPackage In-Reply-To: References: <4E37C338.9010406@uibk.ac.at> <4E37F344.40104@uibk.ac.at> Message-ID: Clemens, Petsc's LDLt decomposition is likely different from mumps' LDLt. Are you sure you want Petsc's D, but use mumps LDLt? I suggest sending a request to mumps developer on how to accessing mumps' D. Hong On Tue, Aug 2, 2011 at 8:12 AM, Matthew Knepley wrote: > On Tue, Aug 2, 2011 at 12:53 PM, Clemens Domanig > wrote: >> >> Maybe I should describe the hole problem. >> I'm using MUMPS to make LDLt decomposition - all from command line. >> But sometimes I need the diagonal entries (respectively the number of >> negative entries) of the D matrix. But there is no way to get this >> information back from MUMPS via petsc ( as far as I found out). > > Yes, we have no access to MUMPS internal data. > >> >> There was the suggestion to use MatGetIntertia but this is only for >> off-diagonal-information. > > I am not sure what you mean by that. > >> >> So I used >> PCSetType( prec, PCCHOLESKY); >> PCFactorGetMatrix( prec, &M); >> MatGetDiagonal( M, z); >> It only works without MUMPS and gives me the diagonal entries I want in >> Matrix M - although the entries are 1/x. >> Vector z is full of zeros - only the last entry is the 'same' as in M (x >> instead of 1/x). >> >> So the idea was to somehow turn off MUMPS if I need the diagonal entries >> and then turn it on again. > > Here is the easiest way to do it. Keep two complete solvers > ? a) KSP with MUMPS > ? b) KSP with Cholesky, give this one KSPSetOptionsPrefix(ksp, "diag_"); > That way -diag_pc_type cholesky will work. Then use the solver you want. > >> >> The problem with PCFactorGetMatSolverPackage is I didn't manage to use it. >> >> const MatSolverPackage pack; >> PCFactorGetMatSolverPackage( prec, &pack); >> PetscSynchronizedPrintf( PETSC_COMM_WORLD, "%s\n", pack); >> I expected 'mumps' as output but got 'H??????tdH3%0' > > It would seem its not yet setup. > ? ?Matt > >> >> Thx >> >> >> >> Matthew Knepley wrote: >>> >>> On Tue, Aug 2, 2011 at 9:28 AM, Clemens Domanig >>> > wrote: >>> >>> ? ?Hi, >>> >>> ? ?usually I'm using MUMPS by puting '-pc_factor_mat_solver_package >>> ? ?mumps' into the .petscrc. >>> ? ?Now I want to change in runtime between MUMPS and non-MUMPS so I >>> ? ?tried to turn it on with >>> ? ?PCFactorSetMatSolverPackage( pre, MAT_SOLVER_MUMPS); >>> ? ?but I doesn't have any effect. >>> >>> >>> This is complicated because you have to set this at the right time during >>> the solver setup process. >>> Command line arguments are much easier. However, you should call this >>> after the preconditioner >>> type is set, but before it is setup. >>> >>> ? ?I also have a problem with PCFactorGetMatSolverPackage(). How can I >>> ? ?get back an output as described in the documentation that has the >>> ? ?type 'const'? >>> >>> >>> What is the problem? I do not understand? >>> >>> ? ?Matt >>> >>> ? ?Thx - Clemens >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >> > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > From bsmith at mcs.anl.gov Tue Aug 2 10:22:05 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 2 Aug 2011 09:22:05 -0600 Subject: [petsc-users] PCFactorSetMatSolverPackage / PCFactorGetMatSolverPackage In-Reply-To: References: <4E37C338.9010406@uibk.ac.at> <4E37F344.40104@uibk.ac.at> Message-ID: <6453BDF9-7A26-41E0-AFBC-C6B27FC87E0B@mcs.anl.gov> On Aug 2, 2011, at 7:10 AM, Jed Brown wrote: > On Tue, Aug 2, 2011 at 06:53, Clemens Domanig wrote: > The problem with PCFactorGetMatSolverPackage is I didn't manage to use it. > > const MatSolverPackage pack; > > Insert here: > > PCSetType(prec,PCCHOLESKY); > > PCFactorGetMatSolverPackage( prec, &pack); > > The type has to be set first, otherwise the PC doesn't have any concept of a solver package. I have updated this function to return NULL instead of an undefined value in this case, thanks. > > Barry, speak up if you would prefer an error. Hmm, I think both approaches: generating an error or returning null have some validity. When going with the non-error approach each function must document what it returns when not valid. Barry > I think PetscUseMethod() is somewhat overused in query functions at present because there is a reasonable non-error condition that can be returned. From vkuhlem at emory.edu Tue Aug 2 12:06:00 2011 From: vkuhlem at emory.edu (Kuhlemann, Verena) Date: Tue, 2 Aug 2011 17:06:00 +0000 Subject: [petsc-users] valgrind Message-ID: Hi, I am trying to find an error in my program with valgrind. The first message that I get is the following: Syscall param write(buf) points to uninitialised byte(s) ==22395== Syscall param write(buf) points to uninitialised byte(s) ==22393== at 0x73D92C0: __write_nocancel (in /lib64/libc-2.5.so) ==22393== by 0x7A76283: ibv_cmd_modify_qp (in /usr/lib64/libibverbs.so.1.0.0) ==22393== by 0x88B742A: ??? (in /usr/lib64/libmlx4-rdmav2.so) ==22393== by 0x7A7A363: ibv_modify_qp (in /usr/lib64/libibverbs.so.1.0.0) ==22393== by 0x67A3DDF: MPID_VIA_Init (viainit.c:1014) ==22393== by 0x67826BB: MPID_Init (mpid_init.c:66) ==22393== by 0x677438E: MPIR_Init (initutil.c:279) ==22393== by 0x432732: PetscInitialize (pinit.c:561) ==22395== at 0x73D92C0: __write_nocancel (in /lib64/libc-2.5.so) ==22395== by 0x7A76283: ibv_cmd_modify_qp (in /usr/lib64/libibverbs.so.1.0.0) ==22395== by 0x88B742A: ??? (in /usr/lib64/libmlx4-rdmav2.so) ==22395== by 0x7A7A363: ibv_modify_qp (in /usr/lib64/libibverbs.so.1.0.0) ==22395== by 0x67A3DDF: MPID_VIA_Init (viainit.c:1014) ==22395== by 0x67826BB: MPID_Init (mpid_init.c:66) ==22395== by 0x677438E: MPIR_Init (initutil.c:279) ==22395== by 0x432732: PetscInitialize (pinit.c:561) ==22395== by 0x4046FC: main (runtests.c:48) ==22395== Address 0x7feffef78 is on thread 1's stack I am not sure what this is suppose to tell me. Is the something wrong with my PetscInitialize? Thanks for the help, Verena ________________________________ This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited. If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments). -------------- next part -------------- An HTML attachment was scrubbed... URL: From vkuhlem at emory.edu Tue Aug 2 12:09:21 2011 From: vkuhlem at emory.edu (Kuhlemann, Verena) Date: Tue, 2 Aug 2011 17:09:21 +0000 Subject: [petsc-users] MatCreateMPIAIJWithSplitArrays Message-ID: HI, I am creating a matrix with MatCreateMPIAIJWithSplitArrays, but I would like the arrays that I am using to create the matrix to be freed once the matrix is destroyed. Is this the correct way to do this? MatCreateMPIAIJWithSplitArrays(PETSC_COMM_WORLD,ne,nf,PETSC_DETERMINE,PETSC_DETERMINE,ia_E_d,ja_E_d,v_E_d,ia_E_o,ja_E_o,v_E_o,E); //set flags so that arrays are freed once the matrix is destroyed mpimat= (Mat_MPIAIJ*)(*E)->data; a=(Mat_SeqAIJ*)(mpimat->A)->data; b=(Mat_SeqAIJ*)(mpimat->B)->data; a->free_a = PETSC_TRUE; a->free_ij = PETSC_TRUE; b->free_a = PETSC_TRUE; b->free_ij = PETSC_TRUE; Thanks, Verena ________________________________ This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited. If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments). -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Aug 2 12:14:39 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Aug 2011 17:14:39 +0000 Subject: [petsc-users] valgrind In-Reply-To: References: Message-ID: On Tue, Aug 2, 2011 at 5:06 PM, Kuhlemann, Verena wrote: > Hi, > > I am trying to find an error in my program with valgrind. > The first message that I get is the following: > No, MPICH is doing something that valgrind does not understand with memory. You can usually ignore anything that goes back into MPI. Matt > Syscall param write(buf) points to uninitialised byte(s) > ==22395== Syscall param write(buf) points to uninitialised byte(s) > ==22393== at 0x73D92C0: __write_nocancel (in /lib64/libc-2.5.so) > ==22393== by 0x7A76283: ibv_cmd_modify_qp (in > /usr/lib64/libibverbs.so.1.0.0) > ==22393== by 0x88B742A: ??? (in /usr/lib64/libmlx4-rdmav2.so) > ==22393== by 0x7A7A363: ibv_modify_qp (in > /usr/lib64/libibverbs.so.1.0.0) > ==22393== by 0x67A3DDF: MPID_VIA_Init (viainit.c:1014) > ==22393== by 0x67826BB: MPID_Init (mpid_init.c:66) > ==22393== by 0x677438E: MPIR_Init (initutil.c:279) > ==22393== by 0x432732: PetscInitialize (pinit.c:561) > ==22395== at 0x73D92C0: __write_nocancel (in /lib64/libc-2.5.so) > ==22395== by 0x7A76283: ibv_cmd_modify_qp (in > /usr/lib64/libibverbs.so.1.0.0) > ==22395== by 0x88B742A: ??? (in /usr/lib64/libmlx4-rdmav2.so) > ==22395== by 0x7A7A363: ibv_modify_qp (in > /usr/lib64/libibverbs.so.1.0.0) > ==22395== by 0x67A3DDF: MPID_VIA_Init (viainit.c:1014) > ==22395== by 0x67826BB: MPID_Init (mpid_init.c:66) > ==22395== by 0x677438E: MPIR_Init (initutil.c:279) > ==22395== by 0x432732: PetscInitialize (pinit.c:561) > ==22395== by 0x4046FC: main (runtests.c:48) > ==22395== Address 0x7feffef78 is on thread 1's stack > > I am not sure what this is suppose to tell me. Is the something wrong > with my PetscInitialize? > > Thanks for the help, > > Verena > > > ------------------------------ > > This e-mail message (including any attachments) is for the sole use of > the intended recipient(s) and may contain confidential and privileged > information. If the reader of this message is not the intended > recipient, you are hereby notified that any dissemination, distribution > or copying of this message (including any attachments) is strictly > prohibited. > > If you have received this message in error, please contact > the sender by reply e-mail message and destroy all copies of the > original message (including attachments). > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Aug 2 12:18:03 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Aug 2011 17:18:03 +0000 Subject: [petsc-users] MatCreateMPIAIJWithSplitArrays In-Reply-To: References: Message-ID: On Tue, Aug 2, 2011 at 5:09 PM, Kuhlemann, Verena wrote: > HI, > > I am creating a matrix with MatCreateMPIAIJWithSplitArrays, but I would > like the arrays that I am using > to create the matrix to be freed once the matrix is destroyed. Is this the > correct way to do this? > > > MatCreateMPIAIJWithSplitArrays(PETSC_COMM_WORLD,ne,nf,PETSC_DETERMINE,PETSC_DETERMINE,ia_E_d,ja_E_d,v_E_d,ia_E_o,ja_E_o,v_E_o,E); > //set flags so that arrays are freed once the matrix is destroyed > mpimat= (Mat_MPIAIJ*)(*E)->data; > a=(Mat_SeqAIJ*)(mpimat->A)->data; > b=(Mat_SeqAIJ*)(mpimat->B)->data; > a->free_a = PETSC_TRUE; > a->free_ij = PETSC_TRUE; > b->free_a = PETSC_TRUE; > b->free_ij = PETSC_TRUE; > Yes, this should work. Matt > Thanks, > > Verena > > ------------------------------ > > This e-mail message (including any attachments) is for the sole use of > the intended recipient(s) and may contain confidential and privileged > information. If the reader of this message is not the intended > recipient, you are hereby notified that any dissemination, distribution > or copying of this message (including any attachments) is strictly > prohibited. > > If you have received this message in error, please contact > the sender by reply e-mail message and destroy all copies of the > original message (including attachments). > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Aug 2 12:30:13 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 2 Aug 2011 11:30:13 -0600 Subject: [petsc-users] MatCreateMPIAIJWithSplitArrays In-Reply-To: References: Message-ID: <85B3F02F-597F-4715-9D8F-94E21F00345F@mcs.anl.gov> On Aug 2, 2011, at 11:09 AM, Kuhlemann, Verena wrote: > HI, > > I am creating a matrix with MatCreateMPIAIJWithSplitArrays, but I would like the arrays that I am using > to create the matrix to be freed once the matrix is destroyed. Is this the correct way to do this? > > MatCreateMPIAIJWithSplitArrays(PETSC_COMM_WORLD,ne,nf,PETSC_DETERMINE,PETSC_DETERMINE,ia_E_d,ja_E_d,v_E_d,ia_E_o,ja_E_o,v_E_o,E); > //set flags so that arrays are freed once the matrix is destroyed > mpimat= (Mat_MPIAIJ*)(*E)->data; > a=(Mat_SeqAIJ*)(mpimat->A)->data; > b=(Mat_SeqAIJ*)(mpimat->B)->data; > a->free_a = PETSC_TRUE; > a->free_ij = PETSC_TRUE; > b->free_a = PETSC_TRUE; > b->free_ij = PETSC_TRUE; > Try it. BUT the original arrays must be obtained with PetscMalloc() not plain malloc(). If you cannot use PetscMalloc() then you can use PetscContainerCreate(), PetscContainerSetUserDestroy() and then PetscObjectCompose() to attach the container to the original matrix (MPIAIJ) then when the matrix is destroyed the user provided container destroy function is automatically called and in it you can free the individual pointers anyway you like., Barry > > Thanks, > > Verena > > > This e-mail message (including any attachments) is for the sole use of > the intended recipient(s) and may contain confidential and privileged > information. If the reader of this message is not the intended > recipient, you are hereby notified that any dissemination, distribution > or copying of this message (including any attachments) is strictly > prohibited. > > If you have received this message in error, please contact > the sender by reply e-mail message and destroy all copies of the > original message (including attachments). From balay at mcs.anl.gov Tue Aug 2 12:35:40 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 2 Aug 2011 12:35:40 -0500 (CDT) Subject: [petsc-users] valgrind In-Reply-To: References: Message-ID: Also suggest using --download-mpich - for valgrind debugging. [it gives you a valgrind clean mpich] Satish On Tue, 2 Aug 2011, Matthew Knepley wrote: > On Tue, Aug 2, 2011 at 5:06 PM, Kuhlemann, Verena wrote: > > > Hi, > > > > I am trying to find an error in my program with valgrind. > > The first message that I get is the following: > > > > No, MPICH is doing something that valgrind does not understand with memory. > You can usually > ignore anything that goes back into MPI. > > Matt > > > > Syscall param write(buf) points to uninitialised byte(s) > > ==22395== Syscall param write(buf) points to uninitialised byte(s) > > ==22393== at 0x73D92C0: __write_nocancel (in /lib64/libc-2.5.so) > > ==22393== by 0x7A76283: ibv_cmd_modify_qp (in > > /usr/lib64/libibverbs.so.1.0.0) > > ==22393== by 0x88B742A: ??? (in /usr/lib64/libmlx4-rdmav2.so) > > ==22393== by 0x7A7A363: ibv_modify_qp (in > > /usr/lib64/libibverbs.so.1.0.0) > > ==22393== by 0x67A3DDF: MPID_VIA_Init (viainit.c:1014) > > ==22393== by 0x67826BB: MPID_Init (mpid_init.c:66) > > ==22393== by 0x677438E: MPIR_Init (initutil.c:279) > > ==22393== by 0x432732: PetscInitialize (pinit.c:561) > > ==22395== at 0x73D92C0: __write_nocancel (in /lib64/libc-2.5.so) > > ==22395== by 0x7A76283: ibv_cmd_modify_qp (in > > /usr/lib64/libibverbs.so.1.0.0) > > ==22395== by 0x88B742A: ??? (in /usr/lib64/libmlx4-rdmav2.so) > > ==22395== by 0x7A7A363: ibv_modify_qp (in > > /usr/lib64/libibverbs.so.1.0.0) > > ==22395== by 0x67A3DDF: MPID_VIA_Init (viainit.c:1014) > > ==22395== by 0x67826BB: MPID_Init (mpid_init.c:66) > > ==22395== by 0x677438E: MPIR_Init (initutil.c:279) > > ==22395== by 0x432732: PetscInitialize (pinit.c:561) > > ==22395== by 0x4046FC: main (runtests.c:48) > > ==22395== Address 0x7feffef78 is on thread 1's stack > > > > I am not sure what this is suppose to tell me. Is the something wrong > > with my PetscInitialize? > > > > Thanks for the help, > > > > Verena > > > > > > ------------------------------ > > > > This e-mail message (including any attachments) is for the sole use of > > the intended recipient(s) and may contain confidential and privileged > > information. If the reader of this message is not the intended > > recipient, you are hereby notified that any dissemination, distribution > > or copying of this message (including any attachments) is strictly > > prohibited. > > > > If you have received this message in error, please contact > > the sender by reply e-mail message and destroy all copies of the > > original message (including attachments). > > > > > > From vkuhlem at emory.edu Tue Aug 2 12:38:50 2011 From: vkuhlem at emory.edu (Kuhlemann, Verena) Date: Tue, 2 Aug 2011 17:38:50 +0000 Subject: [petsc-users] valgrind In-Reply-To: References: , Message-ID: So, the first real error that I get is Invalid write of size 4 ==24909== at 0x9B3B9F: Moc_ComputePartitionParams__ (kwayrefine.c:215) ==24909== by 0x9A1D28: Moc_Global_Partition__ (kmetis.c:231) ==24909== by 0x9A1D00: Moc_Global_Partition__ (kmetis.c:228) ==24909== by 0x9A1D00: Moc_Global_Partition__ (kmetis.c:228) ==24909== by 0x9A1D00: Moc_Global_Partition__ (kmetis.c:228) ==24909== by 0x9A1D00: Moc_Global_Partition__ (kmetis.c:228) ==24909== by 0x9A1619: ParMETIS_V3_PartKway (kmetis.c:137) ==24909== by 0x557AC8: MatPartitioningApply_Parmetis (pmetis.c:97) ==24909== by 0x554058: MatPartitioningApply (partition.c:236) ==24909== by 0x404DE2: main (runtests.c:99) Any ideas why? I will try to use --download-mpich for valgrind debugging./ Thanks, Verena ________________________________________ From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Satish Balay [balay at mcs.anl.gov] Sent: Tuesday, August 02, 2011 1:35 PM To: PETSc users list Subject: Re: [petsc-users] valgrind Also suggest using --download-mpich - for valgrind debugging. [it gives you a valgrind clean mpich] Satish On Tue, 2 Aug 2011, Matthew Knepley wrote: > On Tue, Aug 2, 2011 at 5:06 PM, Kuhlemann, Verena wrote: > > > Hi, > > > > I am trying to find an error in my program with valgrind. > > The first message that I get is the following: > > > > No, MPICH is doing something that valgrind does not understand with memory. > You can usually > ignore anything that goes back into MPI. > > Matt > > > > Syscall param write(buf) points to uninitialised byte(s) > > ==22395== Syscall param write(buf) points to uninitialised byte(s) > > ==22393== at 0x73D92C0: __write_nocancel (in /lib64/libc-2.5.so) > > ==22393== by 0x7A76283: ibv_cmd_modify_qp (in > > /usr/lib64/libibverbs.so.1.0.0) > > ==22393== by 0x88B742A: ??? (in /usr/lib64/libmlx4-rdmav2.so) > > ==22393== by 0x7A7A363: ibv_modify_qp (in > > /usr/lib64/libibverbs.so.1.0.0) > > ==22393== by 0x67A3DDF: MPID_VIA_Init (viainit.c:1014) > > ==22393== by 0x67826BB: MPID_Init (mpid_init.c:66) > > ==22393== by 0x677438E: MPIR_Init (initutil.c:279) > > ==22393== by 0x432732: PetscInitialize (pinit.c:561) > > ==22395== at 0x73D92C0: __write_nocancel (in /lib64/libc-2.5.so) > > ==22395== by 0x7A76283: ibv_cmd_modify_qp (in > > /usr/lib64/libibverbs.so.1.0.0) > > ==22395== by 0x88B742A: ??? (in /usr/lib64/libmlx4-rdmav2.so) > > ==22395== by 0x7A7A363: ibv_modify_qp (in > > /usr/lib64/libibverbs.so.1.0.0) > > ==22395== by 0x67A3DDF: MPID_VIA_Init (viainit.c:1014) > > ==22395== by 0x67826BB: MPID_Init (mpid_init.c:66) > > ==22395== by 0x677438E: MPIR_Init (initutil.c:279) > > ==22395== by 0x432732: PetscInitialize (pinit.c:561) > > ==22395== by 0x4046FC: main (runtests.c:48) > > ==22395== Address 0x7feffef78 is on thread 1's stack > > > > I am not sure what this is suppose to tell me. Is the something wrong > > with my PetscInitialize? > > > > Thanks for the help, > > > > Verena > > > > > > ------------------------------ > > > > This e-mail message (including any attachments) is for the sole use of > > the intended recipient(s) and may contain confidential and privileged > > information. If the reader of this message is not the intended > > recipient, you are hereby notified that any dissemination, distribution > > or copying of this message (including any attachments) is strictly > > prohibited. > > > > If you have received this message in error, please contact > > the sender by reply e-mail message and destroy all copies of the > > original message (including attachments). > > > > > > From vkuhlem at emory.edu Tue Aug 2 12:40:11 2011 From: vkuhlem at emory.edu (Kuhlemann, Verena) Date: Tue, 2 Aug 2011 17:40:11 +0000 Subject: [petsc-users] MatCreateMPIAIJWithSplitArrays In-Reply-To: <85B3F02F-597F-4715-9D8F-94E21F00345F@mcs.anl.gov> References: , <85B3F02F-597F-4715-9D8F-94E21F00345F@mcs.anl.gov> Message-ID: The arrays are obtained with PetscMalloc(), so I hope it will work. ________________________________________ From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Barry Smith [bsmith at mcs.anl.gov] Sent: Tuesday, August 02, 2011 1:30 PM To: PETSc users list Subject: Re: [petsc-users] MatCreateMPIAIJWithSplitArrays On Aug 2, 2011, at 11:09 AM, Kuhlemann, Verena wrote: > HI, > > I am creating a matrix with MatCreateMPIAIJWithSplitArrays, but I would like the arrays that I am using > to create the matrix to be freed once the matrix is destroyed. Is this the correct way to do this? > > MatCreateMPIAIJWithSplitArrays(PETSC_COMM_WORLD,ne,nf,PETSC_DETERMINE,PETSC_DETERMINE,ia_E_d,ja_E_d,v_E_d,ia_E_o,ja_E_o,v_E_o,E); > //set flags so that arrays are freed once the matrix is destroyed > mpimat= (Mat_MPIAIJ*)(*E)->data; > a=(Mat_SeqAIJ*)(mpimat->A)->data; > b=(Mat_SeqAIJ*)(mpimat->B)->data; > a->free_a = PETSC_TRUE; > a->free_ij = PETSC_TRUE; > b->free_a = PETSC_TRUE; > b->free_ij = PETSC_TRUE; > Try it. BUT the original arrays must be obtained with PetscMalloc() not plain malloc(). If you cannot use PetscMalloc() then you can use PetscContainerCreate(), PetscContainerSetUserDestroy() and then PetscObjectCompose() to attach the container to the original matrix (MPIAIJ) then when the matrix is destroyed the user provided container destroy function is automatically called and in it you can free the individual pointers anyway you like., Barry > > Thanks, > > Verena > > > This e-mail message (including any attachments) is for the sole use of > the intended recipient(s) and may contain confidential and privileged > information. If the reader of this message is not the intended > recipient, you are hereby notified that any dissemination, distribution > or copying of this message (including any attachments) is strictly > prohibited. > > If you have received this message in error, please contact > the sender by reply e-mail message and destroy all copies of the > original message (including attachments). From bsmith at mcs.anl.gov Tue Aug 2 12:49:31 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 2 Aug 2011 11:49:31 -0600 Subject: [petsc-users] valgrind In-Reply-To: References: , Message-ID: <7C6452BD-4ECC-490F-9F58-380455EBB27C@mcs.anl.gov> On Aug 2, 2011, at 11:38 AM, Kuhlemann, Verena wrote: > So, the first real error that I get is > > Invalid write of size 4 > ==24909== at 0x9B3B9F: Moc_ComputePartitionParams__ (kwayrefine.c:215) > ==24909== by 0x9A1D28: Moc_Global_Partition__ (kmetis.c:231) > ==24909== by 0x9A1D00: Moc_Global_Partition__ (kmetis.c:228) > ==24909== by 0x9A1D00: Moc_Global_Partition__ (kmetis.c:228) > ==24909== by 0x9A1D00: Moc_Global_Partition__ (kmetis.c:228) > ==24909== by 0x9A1D00: Moc_Global_Partition__ (kmetis.c:228) > ==24909== by 0x9A1619: ParMETIS_V3_PartKway (kmetis.c:137) > ==24909== by 0x557AC8: MatPartitioningApply_Parmetis (pmetis.c:97) > ==24909== by 0x554058: MatPartitioningApply (partition.c:236) > ==24909== by 0x404DE2: main (runtests.c:99) > > Any ideas why? bug in kwayrefine.c at line 215 Have fun debugging parmetis :-( Barry > > I will try to use --download-mpich for valgrind debugging./ > > Thanks, > > Verena > ________________________________________ > From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Satish Balay [balay at mcs.anl.gov] > Sent: Tuesday, August 02, 2011 1:35 PM > To: PETSc users list > Subject: Re: [petsc-users] valgrind > > Also suggest using --download-mpich - for valgrind debugging. [it gives > you a valgrind clean mpich] > > > Satish > > On Tue, 2 Aug 2011, Matthew Knepley wrote: > >> On Tue, Aug 2, 2011 at 5:06 PM, Kuhlemann, Verena wrote: >> >>> Hi, >>> >>> I am trying to find an error in my program with valgrind. >>> The first message that I get is the following: >>> >> >> No, MPICH is doing something that valgrind does not understand with memory. >> You can usually >> ignore anything that goes back into MPI. >> >> Matt >> >> >>> Syscall param write(buf) points to uninitialised byte(s) >>> ==22395== Syscall param write(buf) points to uninitialised byte(s) >>> ==22393== at 0x73D92C0: __write_nocancel (in /lib64/libc-2.5.so) >>> ==22393== by 0x7A76283: ibv_cmd_modify_qp (in >>> /usr/lib64/libibverbs.so.1.0.0) >>> ==22393== by 0x88B742A: ??? (in /usr/lib64/libmlx4-rdmav2.so) >>> ==22393== by 0x7A7A363: ibv_modify_qp (in >>> /usr/lib64/libibverbs.so.1.0.0) >>> ==22393== by 0x67A3DDF: MPID_VIA_Init (viainit.c:1014) >>> ==22393== by 0x67826BB: MPID_Init (mpid_init.c:66) >>> ==22393== by 0x677438E: MPIR_Init (initutil.c:279) >>> ==22393== by 0x432732: PetscInitialize (pinit.c:561) >>> ==22395== at 0x73D92C0: __write_nocancel (in /lib64/libc-2.5.so) >>> ==22395== by 0x7A76283: ibv_cmd_modify_qp (in >>> /usr/lib64/libibverbs.so.1.0.0) >>> ==22395== by 0x88B742A: ??? (in /usr/lib64/libmlx4-rdmav2.so) >>> ==22395== by 0x7A7A363: ibv_modify_qp (in >>> /usr/lib64/libibverbs.so.1.0.0) >>> ==22395== by 0x67A3DDF: MPID_VIA_Init (viainit.c:1014) >>> ==22395== by 0x67826BB: MPID_Init (mpid_init.c:66) >>> ==22395== by 0x677438E: MPIR_Init (initutil.c:279) >>> ==22395== by 0x432732: PetscInitialize (pinit.c:561) >>> ==22395== by 0x4046FC: main (runtests.c:48) >>> ==22395== Address 0x7feffef78 is on thread 1's stack >>> >>> I am not sure what this is suppose to tell me. Is the something wrong >>> with my PetscInitialize? >>> >>> Thanks for the help, >>> >>> Verena >>> >>> >>> ------------------------------ >>> >>> This e-mail message (including any attachments) is for the sole use of >>> the intended recipient(s) and may contain confidential and privileged >>> information. If the reader of this message is not the intended >>> recipient, you are hereby notified that any dissemination, distribution >>> or copying of this message (including any attachments) is strictly >>> prohibited. >>> >>> If you have received this message in error, please contact >>> the sender by reply e-mail message and destroy all copies of the >>> original message (including attachments). >>> >> >> >> >> > From knepley at gmail.com Tue Aug 2 12:56:37 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Aug 2011 17:56:37 +0000 Subject: [petsc-users] valgrind In-Reply-To: References: Message-ID: On Tue, Aug 2, 2011 at 5:38 PM, Kuhlemann, Verena wrote: > So, the first real error that I get is > > Invalid write of size 4 > ==24909== at 0x9B3B9F: Moc_ComputePartitionParams__ (kwayrefine.c:215) > ==24909== by 0x9A1D28: Moc_Global_Partition__ (kmetis.c:231) > ==24909== by 0x9A1D00: Moc_Global_Partition__ (kmetis.c:228) > ==24909== by 0x9A1D00: Moc_Global_Partition__ (kmetis.c:228) > ==24909== by 0x9A1D00: Moc_Global_Partition__ (kmetis.c:228) > ==24909== by 0x9A1D00: Moc_Global_Partition__ (kmetis.c:228) > ==24909== by 0x9A1619: ParMETIS_V3_PartKway (kmetis.c:137) > ==24909== by 0x557AC8: MatPartitioningApply_Parmetis (pmetis.c:97) > ==24909== by 0x554058: MatPartitioningApply (partition.c:236) > ==24909== by 0x404DE2: main (runtests.c:99) > > Any ideas why? > ParMetis is not robust to problems with the input (we are upgrading to the recent release which is supposed to do more checking). I believe it can fail if you have empty partitions. I would check the input. Matt > I will try to use --download-mpich for valgrind debugging./ > > Thanks, > > Verena > ________________________________________ > From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on > behalf of Satish Balay [balay at mcs.anl.gov] > Sent: Tuesday, August 02, 2011 1:35 PM > To: PETSc users list > Subject: Re: [petsc-users] valgrind > > Also suggest using --download-mpich - for valgrind debugging. [it gives > you a valgrind clean mpich] > > > Satish > > On Tue, 2 Aug 2011, Matthew Knepley wrote: > > > On Tue, Aug 2, 2011 at 5:06 PM, Kuhlemann, Verena > wrote: > > > > > Hi, > > > > > > I am trying to find an error in my program with valgrind. > > > The first message that I get is the following: > > > > > > > No, MPICH is doing something that valgrind does not understand with > memory. > > You can usually > > ignore anything that goes back into MPI. > > > > Matt > > > > > > > Syscall param write(buf) points to uninitialised byte(s) > > > ==22395== Syscall param write(buf) points to uninitialised byte(s) > > > ==22393== at 0x73D92C0: __write_nocancel (in /lib64/libc-2.5.so) > > > ==22393== by 0x7A76283: ibv_cmd_modify_qp (in > > > /usr/lib64/libibverbs.so.1.0.0) > > > ==22393== by 0x88B742A: ??? (in /usr/lib64/libmlx4-rdmav2.so) > > > ==22393== by 0x7A7A363: ibv_modify_qp (in > > > /usr/lib64/libibverbs.so.1.0.0) > > > ==22393== by 0x67A3DDF: MPID_VIA_Init (viainit.c:1014) > > > ==22393== by 0x67826BB: MPID_Init (mpid_init.c:66) > > > ==22393== by 0x677438E: MPIR_Init (initutil.c:279) > > > ==22393== by 0x432732: PetscInitialize (pinit.c:561) > > > ==22395== at 0x73D92C0: __write_nocancel (in /lib64/libc-2.5.so) > > > ==22395== by 0x7A76283: ibv_cmd_modify_qp (in > > > /usr/lib64/libibverbs.so.1.0.0) > > > ==22395== by 0x88B742A: ??? (in /usr/lib64/libmlx4-rdmav2.so) > > > ==22395== by 0x7A7A363: ibv_modify_qp (in > > > /usr/lib64/libibverbs.so.1.0.0) > > > ==22395== by 0x67A3DDF: MPID_VIA_Init (viainit.c:1014) > > > ==22395== by 0x67826BB: MPID_Init (mpid_init.c:66) > > > ==22395== by 0x677438E: MPIR_Init (initutil.c:279) > > > ==22395== by 0x432732: PetscInitialize (pinit.c:561) > > > ==22395== by 0x4046FC: main (runtests.c:48) > > > ==22395== Address 0x7feffef78 is on thread 1's stack > > > > > > I am not sure what this is suppose to tell me. Is the something wrong > > > with my PetscInitialize? > > > > > > Thanks for the help, > > > > > > Verena > > > > > > > > > ------------------------------ > > > > > > This e-mail message (including any attachments) is for the sole use of > > > the intended recipient(s) and may contain confidential and privileged > > > information. If the reader of this message is not the intended > > > recipient, you are hereby notified that any dissemination, distribution > > > or copying of this message (including any attachments) is strictly > > > prohibited. > > > > > > If you have received this message in error, please contact > > > the sender by reply e-mail message and destroy all copies of the > > > original message (including attachments). > > > > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From ping.rong at tuhh.de Tue Aug 2 13:23:47 2011 From: ping.rong at tuhh.de (Ping Rong) Date: Tue, 02 Aug 2011 20:23:47 +0200 Subject: [petsc-users] use Superlu as ilu preconditioner In-Reply-To: References: <4E2F2950.2030701@tuhh.de> Message-ID: <4E3840B3.4080809@tuhh.de> Thank you! I currently use 3.1-p8. Although I have also compiled petsc-dev, another problem is that I have programmed some solver by myself in petsc, and some of the interface, such as KSPDefaultFreeWork, seems not available any longer. I didn't have time to look into it yet, but I will try to compiled petsc-dev again some time later. Am 31.07.2011 18:41, schrieb Jed Brown: > On Fri, Jul 29, 2011 at 18:22, Jed Brown > wrote: > > Are you using the development version of libMesh? I built libMesh > with petsc-dev not long ago, and I send updates when I notice > changes in petsc-dev that affect libMesh. Is it not working now? > > > I have sent patches to libmesh-devel to support the latest petsc-dev. > You can apply them to your svn libmesh checkout or wait for upstream > to merge them. -- Ping Rong, M.Sc. Hamburg University of Technology Institut of modelling and computation Denickestra?e 17 (Room 3031) 21073 Hamburg Tel.: ++49 - (0)40 42878 2749 Fax: ++49 - (0)40 42878 43533 Email: ping.rong at tuhh.de -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Aug 2 13:32:32 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 2 Aug 2011 12:32:32 -0600 Subject: [petsc-users] use Superlu as ilu preconditioner In-Reply-To: <4E3840B3.4080809@tuhh.de> References: <4E2F2950.2030701@tuhh.de> <4E3840B3.4080809@tuhh.de> Message-ID: <7A64A0EF-0DF7-4122-9388-AEC3ABE2A407@mcs.anl.gov> On Aug 2, 2011, at 12:23 PM, Ping Rong wrote: > Thank you! I currently use 3.1-p8. Although I have also compiled petsc-dev, another problem is that I have programmed some solver by myself in petsc, and some of the interface, such as KSPDefaultFreeWork, seems not available any longer. Sorry that change is not well explained: You can call ierr = VecDestroyVecs(ksp->nwork,&ksp->work);CHKERRQ(ierr); to free the work vectors but KSPReset() which is called by KSPDestroy() automatically calls this so in fact you usually do not need to do anything to destroy the work vectors, they are now taken care of automatically. Barry > I didn't have time to look into it yet, but I will try to compiled petsc-dev again some time later. > > > Am 31.07.2011 18:41, schrieb Jed Brown: >> On Fri, Jul 29, 2011 at 18:22, Jed Brown wrote: >> Are you using the development version of libMesh? I built libMesh with petsc-dev not long ago, and I send updates when I notice changes in petsc-dev that affect libMesh. Is it not working now? >> >> I have sent patches to libmesh-devel to support the latest petsc-dev. You can apply them to your svn libmesh checkout or wait for upstream to merge them. > > > -- > Ping Rong, M.Sc. > Hamburg University of Technology > Institut of modelling and computation > Denickestra?e 17 (Room 3031) > 21073 Hamburg > > Tel.: ++49 - (0)40 42878 2749 > Fax: ++49 - (0)40 42878 43533 > Email: > ping.rong at tuhh.de From ping.rong at tuhh.de Tue Aug 2 13:50:51 2011 From: ping.rong at tuhh.de (Ping Rong) Date: Tue, 02 Aug 2011 20:50:51 +0200 Subject: [petsc-users] use Superlu as ilu preconditioner In-Reply-To: References: <4E2F2950.2030701@tuhh.de> <4E305497.1070405@tuhh.de> <4E327F53.2000500@tuhh.de> Message-ID: <4E38470B.7000403@tuhh.de> Hong: sorry for the delay. I have put the code pieces together, search for the keyword ATTENTION, I have put some comments there. the overwrite happens when the system matrix is passed to the solver by linear_solver->solve (*matrix, *matrix, *solution, *rhs, tol, maxits); In the file you should also find the a init() function, which is called when the linear_solver object is instantiated. By now I created an identical precond_matrix, and called the solve function by linear_solver->solve (*matrix, *precond_matrix, *solution, *rhs, tol, maxits); The overwritten only happens when using superlu as ilu preconditioner. petsc default ilu preconditioner doesn't have this issue. Please tell me, if you need any more information. Ping Am 29.07.2011 16:42, schrieb Hong Zhang: > Ping : > Please send me your code. I do not understand why the system matrix is > overwritten > by factored matrix when superlu is used. We do not support in-place > factorization > for external packages. Something is incorrect here. > > Hong >> I have found the problem. I don't know whether this is a bug. >> >> ierr = KSPSetOperators(_ksp, matrix->mat(), precond->mat(), >> this->same_preconditioner ? >> SAME_PRECONDITIONER : DIFFERENT_NONZERO_PATTERN); >> >> >> I used my system matrix as the precondition matrix, where matrix->mat() and >> precond->mat() are in deed the same matrix. I have left the >> KSPSetOperators(ksp) out before the KSPSolve(..) function, so >> KSPSetOperators() is called only once while the petsc_linear_solver object >> is initialized, which eliminated the petsc error ("object is in wrong >> state"). >> Also when I used petsc default ilu preconditioner, everything is fine, also >> the solutions are correct. However, as long as I apply the superlu as my >> solver package, it overwrites the system matrix. I compared the system >> matrix before and after solve(). With petsc ilu, it remains identical, but >> with superlu ilu, the system matrix is changed to the preconditioner matrix >> itself. Now I simply duplicate the system matrix, and use this copy as >> precondition matrix. Everything works perfectly. >> >> The code I used is pretty standard, just as in ex2. It took me quite a while >> to realize the problem, because when the system matrix is replaced by the >> preconditioner matrix, you still get a "correct" solution, which is however >> incorrectly scaled. >> >> Ping >> >> >> Am 28.07.2011 18:21, schrieb Hong Zhang: >>> Ping : >>> I tested similar calling procedure using >>> petsc-dev/src/ksp/ksp/examples/tutorials/ex5.c >>> successfully. >>> >>> Try following: >>> 1) switch to petsc-dev. See >>> http://www.mcs.anl.gov/petsc/petsc-as/developers/index.html on how to >>> get it. >>> 2) send us a short and stand-alone code that reproduce the error. We >>> can investigate it. >>> >>> Hong >>> >>>> Dear Hong, >>>> >>>> thank you very much for your time. Can you tell me when should >>>> KSPSetFromOptions(ksp) be called? >>>> well, libMesh has a class called "petsc_linear_solver", in its original >>>> code >>>> KSPSetFromOptions(ksp) is called while the object is created. I couldn't >>>> get >>>> any output, when I run the program with the option >>>> >>>> -pc_type ilu -pc_factor_mat_solver_package superlu >>>> -mat_superlu_ilu_droptol 0.1 -ksp_view >>>> >>>> The actual solve() function of the "petsc_linear_solver" contains the >>>> similar code as in the ex2.c >>>> ... >>>> ierr = KSPSetOperators(_ksp, matrix->mat(), precond->mat(), >>>> this->same_preconditioner ? >>>> SAME_PRECONDITIONER : DIFFERENT_NONZERO_PATTERN); >>>> CHKERRABORT(libMesh::COMM_WORLD,ierr); >>>> >>>> // Set the tolerances for the iterative solver. Use the user-supplied >>>> // tolerance for the relative residual& leave the others at default >>>> values. >>>> ierr = KSPSetTolerances (_ksp, tol, PETSC_DEFAULT, PETSC_DEFAULT, >>>> max_its); >>>> CHKERRABORT(libMesh::COMM_WORLD,ierr); >>>> >>>> ierr = KSPSetFromOptions (_ksp);<-------------- I added these two >>>> lines, >>>> CHKERRABORT(libMesh::COMM_WORLD,ierr);<-------------- then I got proper >>>> output. >>>> >>>> // Solve the linear system >>>> ierr = KSPSolve (_ksp, rhs->vec(), solution->vec()); >>>> CHKERRABORT(libMesh::COMM_WORLD,ierr); >>>> .... >>>> >>>> The problem is that I have to solve a multiple frequency system, and for >>>> each frequency I have several RHS. The same ksp context is used to solve >>>> the >>>> systems repeatedly. I always call KSPSetOperators() with the option >>>> DIFFERENT_NONZERO_PATTERN for the first RHS and SAME_PRECONDITIONER for >>>> the >>>> rest, since the system matrix remains the same. For the first frequency, >>>> everything is fine, both DIFFERENT_NONZERO_PATTERN and >>>> SAME_PRECONDITIONER >>>> options. I got following output from the -ksp_view. >>>> ... >>>> PC Object: >>>> type: ilu >>>> ILU: out-of-place factorization >>>> 0 levels of fill >>>> tolerance for zero pivot 1e-12 >>>> using diagonal shift to prevent zero pivot >>>> matrix ordering: natural >>>> factor fill ratio given 0, needed 0 >>>> Factored matrix follows: >>>> Matrix Object: >>>> type=seqaij, rows=2883, cols=2883 >>>> package used to perform factorization: superlu<---------------- >>>> superlu is properly set >>>> .... >>>> >>>> but when the second frequency comes up, KSPSetOperators() is called again >>>> with DIFFERENT_NONZERO_PATTERN option for the first RHS. the command line >>>> option doesn't seem to work anymore. >>>> ... >>>> PC Object: >>>> type: ilu >>>> ILU: out-of-place factorization >>>> 0 levels of fill >>>> tolerance for zero pivot 1e-12 >>>> using diagonal shift to prevent zero pivot >>>> matrix ordering: natural >>>> factor fill ratio given 1, needed 1 >>>> Factored matrix follows: >>>> Matrix Object: >>>> type=seqaij, rows=2883, cols=2883 >>>> package used to perform factorization: petsc<------------ not >>>> superlu anymore >>>> ... >>>> >>>> and then for the second RHS, when it calls KSPSetOperators(ksp) in the >>>> solve() routine, petsc throws the following error message. >>>> >>>> [0]PETSC ERROR: --------------------- Error Message >>>> ------------------------------------ >>>> [0]PETSC ERROR: Object is in wrong state! >>>> [0]PETSC ERROR: Cannot change solver matrix package after PC has been >>>> setup >>>> or used! >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 >>>> CDT 2011 >>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: ./PLTMainTest.opt on a linux-gnu named abe-vm by mubpr >>>> Wed >>>> Jul 27 19:41:22 2011 >>>> [0]PETSC ERROR: Libraries linked from >>>> /home/mubpr/libs/petsc/linux-gnu-acml-shared-opt/lib >>>> [0]PETSC ERROR: Configure run at Tue Jul 26 22:09:39 2011 >>>> [0]PETSC ERROR: Configure options >>>> >>>> --with-blas-lapack-lib=/home/mubpr/libs/acml/gfortran64_mp/lib/libacml_mp.so >>>> --with-blas-lapack-include=/home/mubpr/libs/acml/gfortran64_mp/include >>>> --with-scalar-type=complex --with-clanguage=c++ >>>> --with-mpi-dir=/home/mubpr/libs/mpich2 --download-superlu=yes >>>> --with-superlu=1 --download-parmetis=yes --with-parmetis=1 >>>> --download-superlu_dist=yes --with-superlu_dist=1 --download-mumps=yes >>>> --with-mumps=1 --download-blacs=yes --with-blacs=1 >>>> --download-scalapack=yes >>>> --with-scalapack=1 --download-spooles=yes --with-spooles=1 >>>> --download-umfpack=yes --with-umfpack=1 --with-debugging=no >>>> --with-shared=1 >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: PCFactorSetMatSolverPackage_Factor() line 183 in >>>> src/ksp/pc/impls/factor/factimpl.c >>>> [0]PETSC ERROR: PCFactorSetMatSolverPackage() line 276 in >>>> src/ksp/pc/impls/factor/factor.c >>>> [0]PETSC ERROR: PCSetFromOptions_Factor() line 275 in >>>> src/ksp/pc/impls/factor/factimpl.c >>>> [0]PETSC ERROR: PCSetFromOptions_ILU() line 112 in >>>> src/ksp/pc/impls/factor/ilu/ilu.c >>>> [0]PETSC ERROR: PCSetFromOptions() line 185 in >>>> src/ksp/pc/interface/pcset.c >>>> [0]PETSC ERROR: KSPSetFromOptions() line 323 in >>>> src/ksp/ksp/interface/itcl.c >>>> [0]PETSC ERROR: User provided function() line 696 in >>>> "unknowndirectory/"src/solvers/petsc_linear_solver.C >>>> >>>> Any idea what the problem may be? I would be very grateful, if you can >>>> give >>>> me any hint. thanks alot >>>> >>>> best regards >>>> Ping >>>> >>>> >>>> >>>> Am 27.07.2011 04:13, schrieb Hong Zhang: >>>>> Did you call KSPSetFromOptions(ksp)? >>>>> Run your code with '-options_table' to dump list of options inputted >>>>> or '-options_left' to dump list of unused options. >>>>> >>>>> I tested with petsc-3.1/src/ksp/ksp/examples/tutorials/ex2.c: >>>>> ./ex2 >>>>> ... >>>>> SuperLU run parameters: >>>>> ... >>>>> ILU_DropTol: 0.1 >>>>> ILU_FillTol: 0.01 >>>>> ILU_FillFactor: 10 >>>>> ILU_DropRule: 9 >>>>> ILU_Norm: 2 >>>>> ILU_MILU: 2 >>>>> >>>>> Hong >>>>> >>>>> On Tue, Jul 26, 2011 at 3:53 PM, Ping Rong wrote: >>>>>> Dear developers, >>>>>> >>>>>> I have compiled petsc-3.1-p8 for a while. Now I would like to use >>>>>> superlu >>>>>> as >>>>>> an ilu preconditioner, since it offers the drop tolerance option. I >>>>>> have >>>>>> read in a thread >>>>>> >>>>>> >>>>>> (https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2010-December/007439.html) >>>>>> that one can run the code with option >>>>>> >>>>>> -pc_type ilu -pc_factor_mat_solver_package superlu >>>>>> -mat_superlu_ilu_droptol >>>>>> <> >>>>>> >>>>>> to setup the ilu preconditioner. I also use the option "-help |grep >>>>>> superlu" >>>>>> to check the settings. However, no matter how I change the value of >>>>>> -mat_superlu_ilu_droptol, I always got the same result >>>>>> ... >>>>>> -mat_superlu_lwork<0>: size of work array in bytes used by >>>>>> factorization >>>>>> (None) >>>>>> -mat_superlu_ilu_droptol<0.0001>: ILU_DropTol (None) >>>>>> -mat_superlu_ilu_filltol<0.01>: ILU_FillTol (None) >>>>>> ... >>>>>> I dont know whether I understand it correctly. But it seems to me the >>>>>> value >>>>>> of the droptol has never been changed. In that maillist thread, it was >>>>>> also >>>>>> mentioned that the dev-version is recommended, because of some bugs in >>>>>> the >>>>>> superlu interface. I have then compiled the current dev-version. but >>>>>> the >>>>>> problem is my program is based on another library (libMesh) which uses >>>>>> petsc >>>>>> as a solver package. Since some of the interfaces have been changed in >>>>>> the >>>>>> dev-version, I would not be able to compile the libMesh with petsc >>>>>> anymore. >>>>>> Is there anyway I can correct the superlu interface in the 3.1-p8 >>>>>> version >>>>>> directly? Any help will be appreciated!! >>>>>> >>>>>> Best regards >>>>>> >>>>>> -- >>>>>> Ping Rong, M.Sc. >>>>>> Hamburg University of Technology >>>>>> Institut of modelling and computation >>>>>> Denickestra?e 17 (Room 3031) >>>>>> 21073 Hamburg >>>>>> >>>>>> Tel.: ++49 - (0)40 42878 2749 >>>>>> Fax: ++49 - (0)40 42878 43533 >>>>>> Email: ping.rong at tuhh.de >>>>>> >>>>>> >>>> -- >>>> Ping Rong, M.Sc. >>>> Hamburg University of Technology >>>> Institut of modelling and computation >>>> Denickestra?e 17 (Room 3031) >>>> 21073 Hamburg >>>> >>>> Tel.: ++49 - (0)40 42878 2749 >>>> Fax: ++49 - (0)40 42878 43533 >>>> Email: ping.rong at tuhh.de >>>> >>>> >> >> -- >> Ping Rong, M.Sc. >> Hamburg University of Technology >> Institut of modelling and computation >> Denickestra?e 17 (Room 3031) >> 21073 Hamburg >> >> Tel.: ++49 - (0)40 42878 2749 >> Fax: ++49 - (0)40 42878 43533 >> Email: ping.rong at tuhh.de >> >> -- Ping Rong, M.Sc. Hamburg University of Technology Institut of modelling and computation Denickestra?e 17 (Room 3031) 21073 Hamburg Tel.: ++49 - (0)40 42878 2749 Fax: ++49 - (0)40 42878 43533 Email: ping.rong at tuhh.de -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: libMesh-petsc_linear_solver.cpp URL: From hzhang at mcs.anl.gov Tue Aug 2 15:32:40 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 2 Aug 2011 15:32:40 -0500 Subject: [petsc-users] use Superlu as ilu preconditioner In-Reply-To: <4E38470B.7000403@tuhh.de> References: <4E2F2950.2030701@tuhh.de> <4E305497.1070405@tuhh.de> <4E327F53.2000500@tuhh.de> <4E38470B.7000403@tuhh.de> Message-ID: Ping, I'm sorry, the code is too difficult to read and is not executable by itself :-( I do not understand what you want to do. Do you want solve A*x=rhs with same A and multiple rhs vectors? >From your previous report, it seems that original matrix A is overwritten by matrix factorization when superlu's ilu is used. Please correct me if I'm wrong. Hong On Tue, Aug 2, 2011 at 1:50 PM, Ping Rong wrote: > Hong: > sorry for the delay. I have put the code pieces together, search for the > keyword ATTENTION, I have put some comments there. > the overwrite happens when the system matrix is passed to the solver by > > ? ? ? ?linear_solver->solve (*matrix, *matrix, *solution, *rhs, tol, > maxits); > > In the file you should also find the a init() function, which is called when > the linear_solver object is instantiated. > By now I created an identical precond_matrix, and called the solve function > by > > ? ? ? ?linear_solver->solve (*matrix, *precond_matrix, *solution, *rhs, tol, > maxits); > > The overwritten only happens when using superlu as ilu preconditioner. petsc > default ilu preconditioner doesn't have this issue. > Please tell me, if you need any more information. > > Ping > > Am 29.07.2011 16:42, schrieb Hong Zhang: >> >> Ping : >> Please send me your code. I do not understand why the system matrix is >> overwritten >> by ?factored matrix when superlu is used. We do not support in-place >> factorization >> for external packages. Something is incorrect here. >> >> Hong >>> >>> I have found the problem. I don't know whether this is a bug. >>> >>> ? ? ? ?ierr = KSPSetOperators(_ksp, matrix->mat(), precond->mat(), >>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? this->same_preconditioner ? >>> SAME_PRECONDITIONER : DIFFERENT_NONZERO_PATTERN); >>> >>> >>> I used my system matrix as the precondition matrix, where matrix->mat() >>> and >>> precond->mat() are in deed the same matrix. I have left the >>> KSPSetOperators(ksp) out before the KSPSolve(..) function, so >>> KSPSetOperators() is called only once while the petsc_linear_solver >>> object >>> is initialized, which eliminated the petsc error ("object is in wrong >>> state"). >>> Also when I used petsc default ilu preconditioner, everything is fine, >>> also >>> the solutions are correct. However, as long as I apply the superlu as my >>> solver package, it overwrites the system matrix. I compared the system >>> matrix before and after solve(). With petsc ilu, it remains identical, >>> but >>> with superlu ilu, the system matrix is changed to the preconditioner >>> matrix >>> itself. Now I simply duplicate the system matrix, and use this copy as >>> precondition matrix. Everything works perfectly. >>> >>> The code I used is pretty standard, just as in ex2. It took me quite a >>> while >>> to realize the problem, because when the system matrix is replaced by the >>> preconditioner matrix, you still get a "correct" solution, which is >>> however >>> incorrectly scaled. >>> >>> Ping >>> >>> >>> Am 28.07.2011 18:21, schrieb Hong Zhang: >>>> >>>> Ping : >>>> I tested similar calling procedure using >>>> petsc-dev/src/ksp/ksp/examples/tutorials/ex5.c >>>> successfully. >>>> >>>> Try following: >>>> 1) switch to petsc-dev. See >>>> http://www.mcs.anl.gov/petsc/petsc-as/developers/index.html on how to >>>> get it. >>>> 2) send us a short and stand-alone code that reproduce the error. We >>>> can investigate it. >>>> >>>> Hong >>>> >>>>> Dear Hong, >>>>> >>>>> thank you very much for your time. Can you tell me when should >>>>> ?KSPSetFromOptions(ksp) be called? >>>>> well, libMesh has a class called "petsc_linear_solver", in its original >>>>> code >>>>> KSPSetFromOptions(ksp) is called while the object is created. I >>>>> couldn't >>>>> get >>>>> any output, when I run the program with the option >>>>> >>>>> ? -pc_type ilu -pc_factor_mat_solver_package superlu >>>>> -mat_superlu_ilu_droptol 0.1 -ksp_view >>>>> >>>>> The actual solve() function of the "petsc_linear_solver" contains the >>>>> similar code as in the ex2.c >>>>> ... >>>>> ? ?ierr = KSPSetOperators(_ksp, matrix->mat(), precond->mat(), >>>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? this->same_preconditioner ? >>>>> SAME_PRECONDITIONER : DIFFERENT_NONZERO_PATTERN); >>>>> ? ?CHKERRABORT(libMesh::COMM_WORLD,ierr); >>>>> >>>>> ?// Set the tolerances for the iterative solver. ?Use the user-supplied >>>>> ?// tolerance for the relative residual& ? ?leave the others at default >>>>> values. >>>>> ?ierr = KSPSetTolerances (_ksp, tol, PETSC_DEFAULT, PETSC_DEFAULT, >>>>> max_its); >>>>> ?CHKERRABORT(libMesh::COMM_WORLD,ierr); >>>>> >>>>> ?ierr = KSPSetFromOptions (_ksp);<-------------- ?I added these two >>>>> lines, >>>>> ?CHKERRABORT(libMesh::COMM_WORLD,ierr);<-------------- ?then I got >>>>> proper >>>>> output. >>>>> >>>>> ?// Solve the linear system >>>>> ?ierr = KSPSolve (_ksp, rhs->vec(), solution->vec()); >>>>> ?CHKERRABORT(libMesh::COMM_WORLD,ierr); >>>>> .... >>>>> >>>>> The problem is that I have to solve a multiple frequency system, and >>>>> for >>>>> each frequency I have several RHS. The same ksp context is used to >>>>> solve >>>>> the >>>>> systems repeatedly. I always call KSPSetOperators() with the option >>>>> DIFFERENT_NONZERO_PATTERN for the first RHS and SAME_PRECONDITIONER for >>>>> the >>>>> rest, since the system matrix remains the same. ?For the first >>>>> frequency, >>>>> everything is fine, both DIFFERENT_NONZERO_PATTERN and >>>>> SAME_PRECONDITIONER >>>>> options. I got following output from the -ksp_view. >>>>> ... >>>>> PC Object: >>>>> ?type: ilu >>>>> ? ?ILU: out-of-place factorization >>>>> ? ?0 levels of fill >>>>> ? ?tolerance for zero pivot 1e-12 >>>>> ? ?using diagonal shift to prevent zero pivot >>>>> ? ?matrix ordering: natural >>>>> ? ?factor fill ratio given 0, needed 0 >>>>> ? ? ?Factored matrix follows: >>>>> ? ? ? ?Matrix Object: >>>>> ? ? ? ? ?type=seqaij, rows=2883, cols=2883 >>>>> ? ? ? ? ?package used to perform factorization: >>>>> superlu<---------------- >>>>> superlu is properly set >>>>> .... >>>>> >>>>> but when the second frequency comes up, KSPSetOperators() is called >>>>> again >>>>> with DIFFERENT_NONZERO_PATTERN option for the first RHS. the command >>>>> line >>>>> option doesn't seem to work anymore. >>>>> ... >>>>> PC Object: >>>>> ?type: ilu >>>>> ? ?ILU: out-of-place factorization >>>>> ? ?0 levels of fill >>>>> ? ?tolerance for zero pivot 1e-12 >>>>> ? ?using diagonal shift to prevent zero pivot >>>>> ? ?matrix ordering: natural >>>>> ? ?factor fill ratio given 1, needed 1 >>>>> ? ? ?Factored matrix follows: >>>>> ? ? ? ?Matrix Object: >>>>> ? ? ? ? ?type=seqaij, rows=2883, cols=2883 >>>>> ? ? ? ? ?package used to perform factorization: petsc<------------ not >>>>> superlu anymore >>>>> ... >>>>> >>>>> and then for the second RHS, when it calls KSPSetOperators(ksp) in the >>>>> solve() routine, petsc throws the following error message. >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> ------------------------------------ >>>>> [0]PETSC ERROR: Object is in wrong state! >>>>> [0]PETSC ERROR: Cannot change solver matrix package after PC has been >>>>> setup >>>>> or used! >>>>> [0]PETSC ERROR: >>>>> >>>>> ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >>>>> 13:37:48 >>>>> CDT 2011 >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>> [0]PETSC ERROR: >>>>> >>>>> ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: ./PLTMainTest.opt on a linux-gnu named abe-vm by mubpr >>>>> Wed >>>>> Jul 27 19:41:22 2011 >>>>> [0]PETSC ERROR: Libraries linked from >>>>> /home/mubpr/libs/petsc/linux-gnu-acml-shared-opt/lib >>>>> [0]PETSC ERROR: Configure run at Tue Jul 26 22:09:39 2011 >>>>> [0]PETSC ERROR: Configure options >>>>> >>>>> >>>>> --with-blas-lapack-lib=/home/mubpr/libs/acml/gfortran64_mp/lib/libacml_mp.so >>>>> --with-blas-lapack-include=/home/mubpr/libs/acml/gfortran64_mp/include >>>>> --with-scalar-type=complex --with-clanguage=c++ >>>>> --with-mpi-dir=/home/mubpr/libs/mpich2 --download-superlu=yes >>>>> --with-superlu=1 --download-parmetis=yes --with-parmetis=1 >>>>> --download-superlu_dist=yes --with-superlu_dist=1 --download-mumps=yes >>>>> --with-mumps=1 --download-blacs=yes --with-blacs=1 >>>>> --download-scalapack=yes >>>>> --with-scalapack=1 --download-spooles=yes --with-spooles=1 >>>>> --download-umfpack=yes --with-umfpack=1 --with-debugging=no >>>>> --with-shared=1 >>>>> [0]PETSC ERROR: >>>>> >>>>> ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: PCFactorSetMatSolverPackage_Factor() line 183 in >>>>> src/ksp/pc/impls/factor/factimpl.c >>>>> [0]PETSC ERROR: PCFactorSetMatSolverPackage() line 276 in >>>>> src/ksp/pc/impls/factor/factor.c >>>>> [0]PETSC ERROR: PCSetFromOptions_Factor() line 275 in >>>>> src/ksp/pc/impls/factor/factimpl.c >>>>> [0]PETSC ERROR: PCSetFromOptions_ILU() line 112 in >>>>> src/ksp/pc/impls/factor/ilu/ilu.c >>>>> [0]PETSC ERROR: PCSetFromOptions() line 185 in >>>>> src/ksp/pc/interface/pcset.c >>>>> [0]PETSC ERROR: KSPSetFromOptions() line 323 in >>>>> src/ksp/ksp/interface/itcl.c >>>>> [0]PETSC ERROR: User provided function() line 696 in >>>>> "unknowndirectory/"src/solvers/petsc_linear_solver.C >>>>> >>>>> Any idea what the problem may be? I would be very grateful, if you can >>>>> give >>>>> me any hint. thanks alot >>>>> >>>>> best regards >>>>> Ping >>>>> >>>>> >>>>> >>>>> Am 27.07.2011 04:13, schrieb Hong Zhang: >>>>>> >>>>>> Did you call KSPSetFromOptions(ksp)? >>>>>> Run your code with '-options_table' to dump list of options inputted >>>>>> or '-options_left' to dump list of unused options. >>>>>> >>>>>> I tested with petsc-3.1/src/ksp/ksp/examples/tutorials/ex2.c: >>>>>> ./ex2 >>>>>> ... >>>>>> ? ? ? ? ? ? SuperLU run parameters: >>>>>> ? ? ? ? ? ? ... >>>>>> ? ? ? ? ? ? ? ILU_DropTol: 0.1 >>>>>> ? ? ? ? ? ? ? ILU_FillTol: 0.01 >>>>>> ? ? ? ? ? ? ? ILU_FillFactor: 10 >>>>>> ? ? ? ? ? ? ? ILU_DropRule: 9 >>>>>> ? ? ? ? ? ? ? ILU_Norm: 2 >>>>>> ? ? ? ? ? ? ? ILU_MILU: 2 >>>>>> >>>>>> Hong >>>>>> >>>>>> On Tue, Jul 26, 2011 at 3:53 PM, Ping Rong >>>>>> ?wrote: >>>>>>> >>>>>>> Dear developers, >>>>>>> >>>>>>> I have compiled petsc-3.1-p8 for a while. Now I would like to use >>>>>>> superlu >>>>>>> as >>>>>>> an ilu preconditioner, since it offers the drop tolerance option. I >>>>>>> have >>>>>>> read in a thread >>>>>>> >>>>>>> >>>>>>> >>>>>>> (https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2010-December/007439.html) >>>>>>> that one can run the code with option >>>>>>> >>>>>>> -pc_type ilu -pc_factor_mat_solver_package superlu >>>>>>> -mat_superlu_ilu_droptol >>>>>>> <> >>>>>>> >>>>>>> to setup the ilu preconditioner. I also use the option "-help |grep >>>>>>> superlu" >>>>>>> to check the settings. However, no matter how I change the value of >>>>>>> -mat_superlu_ilu_droptol, I always got the same result >>>>>>> ... >>>>>>> ?-mat_superlu_lwork<0>: size of work array in bytes used by >>>>>>> factorization >>>>>>> (None) >>>>>>> ?-mat_superlu_ilu_droptol<0.0001>: ILU_DropTol (None) >>>>>>> ?-mat_superlu_ilu_filltol<0.01>: ILU_FillTol (None) >>>>>>> ... >>>>>>> I dont know whether I understand it correctly. But it seems to me the >>>>>>> value >>>>>>> of the droptol has never been changed. In that maillist thread, it >>>>>>> was >>>>>>> also >>>>>>> mentioned that the dev-version is recommended, because of some bugs >>>>>>> in >>>>>>> the >>>>>>> superlu interface. I have then compiled the current dev-version. but >>>>>>> the >>>>>>> problem is my program is based on another library (libMesh) which >>>>>>> uses >>>>>>> petsc >>>>>>> as a solver package. Since some of the interfaces have been changed >>>>>>> in >>>>>>> the >>>>>>> dev-version, I would not be able to compile the libMesh with petsc >>>>>>> anymore. >>>>>>> Is there anyway I can correct the superlu interface in the 3.1-p8 >>>>>>> version >>>>>>> directly? Any help will be appreciated!! >>>>>>> >>>>>>> Best regards >>>>>>> >>>>>>> -- >>>>>>> Ping Rong, M.Sc. >>>>>>> Hamburg University of Technology >>>>>>> Institut of modelling and computation >>>>>>> Denickestra?e 17 (Room 3031) >>>>>>> 21073 Hamburg >>>>>>> >>>>>>> Tel.: ++49 - (0)40 42878 2749 >>>>>>> Fax: ?++49 - (0)40 42878 43533 >>>>>>> Email: ping.rong at tuhh.de >>>>>>> >>>>>>> >>>>> -- >>>>> Ping Rong, M.Sc. >>>>> Hamburg University of Technology >>>>> Institut of modelling and computation >>>>> Denickestra?e 17 (Room 3031) >>>>> 21073 Hamburg >>>>> >>>>> Tel.: ++49 - (0)40 42878 2749 >>>>> Fax: ?++49 - (0)40 42878 43533 >>>>> Email: ping.rong at tuhh.de >>>>> >>>>> >>> >>> -- >>> Ping Rong, M.Sc. >>> Hamburg University of Technology >>> Institut of modelling and computation >>> Denickestra?e 17 (Room 3031) >>> 21073 Hamburg >>> >>> Tel.: ++49 - (0)40 42878 2749 >>> Fax: ?++49 - (0)40 42878 43533 >>> Email: ping.rong at tuhh.de >>> >>> > > > -- > Ping Rong, M.Sc. > Hamburg University of Technology > Institut of modelling and computation > Denickestra?e 17 (Room 3031) > 21073 Hamburg > > Tel.: ++49 - (0)40 42878 2749 > Fax: ?++49 - (0)40 42878 43533 > Email: ping.rong at tuhh.de > > From knepley at gmail.com Tue Aug 2 15:40:47 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Aug 2011 20:40:47 +0000 Subject: [petsc-users] Preallocation (Unstructured FE) In-Reply-To: <4DD2A80B.2000505@geology.wisc.edu> References: <4DBC4DDC.1010604@geology.wisc.edu> <4DBEAEC3.7020708@geology.wisc.edu> <4DD287F9.80209@geology.wisc.edu> <4DD2A80B.2000505@geology.wisc.edu> Message-ID: On Tue, May 17, 2011 at 4:53 PM, Tabrez Ali wrote: > ** > Matt > > Any linear 2/3d element would do. A linear tetrahedron would be good to > start with. > I have finally gotten to this. Sorry about the wait. You must have petsc-dev. I have added src/dm/impls/mesh/examples/tutorials/ex4f90.F I coded it using the Fortran datatypes option. If you do not use his, just change type(Mat) --> Mat and so on. It should work fine. It reads in an ExodusII mesh (I attach the one I used), and then defines a linear element. It prints out the properly allocated matrix. Let me know if this helps. Thanks, Matt > Thanks > Tabrez > > > On 05/17/2011 11:21 AM, Matthew Knepley wrote: > > On Tue, May 17, 2011 at 9:36 AM, Tabrez Ali wrote: > >> Matt >> >> An example which reads an unstructured mesh (ascii or exodus), partitions >> it and sets up Mat (with the right preallocation) would be great. If it is >> in Fortran then that would be perfect. Btw my problem also has Lagrange >> multiplier constraints. >> > > Yes, I should have been more specific. What element? > > Thanks, > > Matt > > >> Thanks >> Tabrez >> >> >> On 05/17/2011 08:25 AM, Matthew Knepley wrote: >> >> On Mon, May 2, 2011 at 8:16 AM, Tabrez Ali wrote: >> >>> Is there a way I can use this and other mesh routines from Fortran? The >>> manual doesn't say much on this. >>> >> >> Yes, but you are right that nothing is in the manual. DMMESH (in >> petsc-dev) now obeys the full DM interface, >> so that DMGetMatrix() will return you a properly allocated Mat. So what is >> the problem? Of course, it is that >> Petsc has no good way to specify what finite element you are dealing with. >> >> The way I was doing this is to encode it using some C++ classes. This >> turns out to be a bad way to do things. >> I am currently reworking it so that this information is stored in a >> simple C struct that you can produce. Should >> have this done soon. >> >> Can you mail me a description of an example you would like to run? >> >> Thanks, >> >> Matt >> >> >>> >>> Tabrez >>> >>> >>> On 05/01/2011 09:53 AM, Matthew Knepley wrote: >>> >>> On Sat, Apr 30, 2011 at 12:58 PM, Tabrez Ali wrote: >>> >>>> Petsc Developers/Users >>>> >>>> I having some performance issues with preallocation in a fully >>>> unstructured FE code. It would be very helpful if those using FE codes can >>>> comment. >>>> >>>> For a problem of size 100K nodes and 600K tet elements (on 1 cpu) >>>> >>>> 1. If I calculate the _exact_ number of non-zeros per row (using a >>>> running list in Fortran) by looping over nodes & elements, the code takes 17 >>>> mins (to calculate nnz's/per row, assemble and solve). >>>> 2. If I dont use a running list and simply get the average of the max >>>> number of nodes a node might be connected to (again by looping over nodes & >>>> elements but not using a running list) then it takes 8 mins >>>> 3. If I just magically guess the right value calculated in 2 and use >>>> that as average nnz per row then it only takes 25 secs. >>>> >>>> Basically in all cases Assembly and Solve are very fast (few seconds) >>>> but the nnz calculation itself (in 2 and 3) takes a long time. How can this >>>> be cut down? Is there a heuristic way to estimate the number (as done in 3) >>>> even if it slightly overestimates the nnz's per row or are efficient ways to >>>> do step 1 or 2. Right now I have do i=1,num_nodes; do j=1,num_elements ... >>>> which obviously is slow for large number of nodes/elements. >>>> >>> >>> If you want to see my code doing this, look at >>> >>> include/petscdmmesh.hh:preallocateOperatorNew() >>> >>> which handles the determination of nonzero structure for a FEM >>> operator. It should look mostly >>> like your own code. >>> >>> Matt >>> >>> >>>> Thanks in advance >>>> Tabrez >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Square.gen Type: application/octet-stream Size: 2852 bytes Desc: not available URL: From vkuhlem at emory.edu Tue Aug 2 18:21:33 2011 From: vkuhlem at emory.edu (Kuhlemann, Verena) Date: Tue, 2 Aug 2011 23:21:33 +0000 Subject: [petsc-users] valgrind In-Reply-To: References: , Message-ID: I am not quite sure what you mean with check the input. The input is a simple undirected graph. All edgeweights=1, there are no weights on the vertices. Each processor holds a non-empty part of the graph. There are no dangling nodes. Is there anything else I should check? ________________________________ From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Matthew Knepley [knepley at gmail.com] Sent: Tuesday, August 02, 2011 1:56 PM To: PETSc users list Subject: Re: [petsc-users] valgrind On Tue, Aug 2, 2011 at 5:38 PM, Kuhlemann, Verena > wrote: So, the first real error that I get is Invalid write of size 4 ==24909== at 0x9B3B9F: Moc_ComputePartitionParams__ (kwayrefine.c:215) ==24909== by 0x9A1D28: Moc_Global_Partition__ (kmetis.c:231) ==24909== by 0x9A1D00: Moc_Global_Partition__ (kmetis.c:228) ==24909== by 0x9A1D00: Moc_Global_Partition__ (kmetis.c:228) ==24909== by 0x9A1D00: Moc_Global_Partition__ (kmetis.c:228) ==24909== by 0x9A1D00: Moc_Global_Partition__ (kmetis.c:228) ==24909== by 0x9A1619: ParMETIS_V3_PartKway (kmetis.c:137) ==24909== by 0x557AC8: MatPartitioningApply_Parmetis (pmetis.c:97) ==24909== by 0x554058: MatPartitioningApply (partition.c:236) ==24909== by 0x404DE2: main (runtests.c:99) Any ideas why? ParMetis is not robust to problems with the input (we are upgrading to the recent release which is supposed to do more checking). I believe it can fail if you have empty partitions. I would check the input. Matt I will try to use --download-mpich for valgrind debugging./ Thanks, Verena ________________________________________ From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Satish Balay [balay at mcs.anl.gov] Sent: Tuesday, August 02, 2011 1:35 PM To: PETSc users list Subject: Re: [petsc-users] valgrind Also suggest using --download-mpich - for valgrind debugging. [it gives you a valgrind clean mpich] Satish On Tue, 2 Aug 2011, Matthew Knepley wrote: > On Tue, Aug 2, 2011 at 5:06 PM, Kuhlemann, Verena > wrote: > > > Hi, > > > > I am trying to find an error in my program with valgrind. > > The first message that I get is the following: > > > > No, MPICH is doing something that valgrind does not understand with memory. > You can usually > ignore anything that goes back into MPI. > > Matt > > > > Syscall param write(buf) points to uninitialised byte(s) > > ==22395== Syscall param write(buf) points to uninitialised byte(s) > > ==22393== at 0x73D92C0: __write_nocancel (in /lib64/libc-2.5.so) > > ==22393== by 0x7A76283: ibv_cmd_modify_qp (in > > /usr/lib64/libibverbs.so.1.0.0) > > ==22393== by 0x88B742A: ??? (in /usr/lib64/libmlx4-rdmav2.so) > > ==22393== by 0x7A7A363: ibv_modify_qp (in > > /usr/lib64/libibverbs.so.1.0.0) > > ==22393== by 0x67A3DDF: MPID_VIA_Init (viainit.c:1014) > > ==22393== by 0x67826BB: MPID_Init (mpid_init.c:66) > > ==22393== by 0x677438E: MPIR_Init (initutil.c:279) > > ==22393== by 0x432732: PetscInitialize (pinit.c:561) > > ==22395== at 0x73D92C0: __write_nocancel (in /lib64/libc-2.5.so) > > ==22395== by 0x7A76283: ibv_cmd_modify_qp (in > > /usr/lib64/libibverbs.so.1.0.0) > > ==22395== by 0x88B742A: ??? (in /usr/lib64/libmlx4-rdmav2.so) > > ==22395== by 0x7A7A363: ibv_modify_qp (in > > /usr/lib64/libibverbs.so.1.0.0) > > ==22395== by 0x67A3DDF: MPID_VIA_Init (viainit.c:1014) > > ==22395== by 0x67826BB: MPID_Init (mpid_init.c:66) > > ==22395== by 0x677438E: MPIR_Init (initutil.c:279) > > ==22395== by 0x432732: PetscInitialize (pinit.c:561) > > ==22395== by 0x4046FC: main (runtests.c:48) > > ==22395== Address 0x7feffef78 is on thread 1's stack > > > > I am not sure what this is suppose to tell me. Is the something wrong > > with my PetscInitialize? > > > > Thanks for the help, > > > > Verena > > > > > > ------------------------------ > > > > This e-mail message (including any attachments) is for the sole use of > > the intended recipient(s) and may contain confidential and privileged > > information. If the reader of this message is not the intended > > recipient, you are hereby notified that any dissemination, distribution > > or copying of this message (including any attachments) is strictly > > prohibited. > > > > If you have received this message in error, please contact > > the sender by reply e-mail message and destroy all copies of the > > original message (including attachments). > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From shitij.cse at gmail.com Wed Aug 3 00:23:21 2011 From: shitij.cse at gmail.com (Shitij Bhargava) Date: Wed, 3 Aug 2011 10:53:21 +0530 Subject: [petsc-users] Will using SBAIJ to solve for all eigenvalues and eigenvectors of a symmetric sparse matrix produced one row at a time be beneficial ? In-Reply-To: References: Message-ID: Oh, ok, i have verified that matrix assembly is not taking so long. The bottleneck was something else. But now I have encountered a different, more important problem regarding the solver itself, which I suppose I should ask in a different thread. Thank you. Shitij On 30 July 2011 21:56, Matthew Knepley wrote: > On Sat, Jul 30, 2011 at 11:11 AM, Shitij Bhargava wrote: > >> Hi !! >> >> I have just started using SLEPc/PETSc. >> >> The problem that I am trying to solve is that I have some routines >> calculating one row at a time of a matrix, which I want to insert in some >> sparse matrix representation, and then find the next row, insert it, and so >> on. (This is being done to avoid the need to have one big NxN matrix in >> memory).The matrix is also symmetric, so the row that the subroutines >> produce is such that any non-zero element can be present only in that part >> of the row corresponding to the upper triangle of the matrix. (all others >> are zero). After I have the matrix in sparse format, I want to find all the >> eigenvalues and eigenvectors. >> >> Till now I have been using the SeqAij matrix and MatSetValues method with >> preallocation set appropriately, but I am not satisfied with its performance >> (the way I am doing it). When one row is produced, I insert it into the AIJ >> matrix, so there are total of N calls to MatSetValues. Also even though the >> matrix is symmetric, i have to enter the full matrix, as otherwise the >> eigensolver doesnt seem to produce the right answer (Why ? I set the problem >> type to EPS_HEP, but still it requires me to store the full matrix ? It is >> written in documentation that problem type should always be specified so >> that the eigensolver can take advantage of it (symmetry in this case), but >> considering that I still have to store the full matrix, it meant 'advantage' >> only in time, not memory ?....when I set the problem type to EPS_HEP, it >> means that it will take only the upper or lower triangle right ? why should >> it require both ? Is this because the matrix type is not symmetric, unlike >> SBAIJ ?) >> > > You must set the type to SBAIJ in order to enter only the symmetric part. > > >> So, if I use SBAIJ and say, enter some bs rows at a time into the SBAIJ >> matrix, will it be faster ? Also, in that case would I still have to enter >> both upper and lower triangle or only one (I hope only one, otherwise the >> 'S' doesnt make sense to me, but in that case, which one ? The eigensolver >> doesnt give me an option to specify if the upper or lower triangle is stored >> in the matrix in case the problem type is EPS_HEP). Will the eigensolver >> produce the right answers in this case ? (say, storing only upper triangle >> in SBAIJ, instead of both) >> > > I do not think it will be noticeably faster. Value entry is usually not a > large part of the time. YOu can check this using -log_summary. > > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MATSEQSBAIJ.html#MATSEQSBAIJ > > show that only the upper triangle is stored. > > >> Otherwise, while using AIJ, if I insert multiple rows at a time instead of >> just one (thus reducing the number of calls to MatSetValues, at the cost of >> more memory), will this be much beneficial ? (Please note that the matrix is >> not very sparse) >> > > I cannot say anything until we see the timing from -log_summary. It is very > rare that matrix construction is a factor, > if it has been properly allocated, which you can check using -info. > > Matt > > >> Thank you very much in advance !!! >> >> >> Shitij >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shitij.cse at gmail.com Wed Aug 3 01:30:42 2011 From: shitij.cse at gmail.com (Shitij Bhargava) Date: Wed, 3 Aug 2011 12:00:42 +0530 Subject: [petsc-users] SLEPc eigensolver that uses minimal memory and finds ALL eigenvalues of a real symmetric sparse matrix in reasonable time Message-ID: Hi all !! The reason I have started using SLEPc/PETSc is that I want to reduce the memory requirements of a program that I already have in Intel MKL. The program's purpose is to calculate ALL eigenvalues and eigenvectors of a real symmetric sparse matrix. Intel MKL doesnt have a sparse solver for this (cant solve for eigenvalues of a matrix represented in sparse format), so I am using SLEPc, as it can do this. (NOTE: by "sparsity" I mean ZEROES*100/(ZEROES+NONZEROES). Also the tolerance I am using is 10^-6) Till now I have been using the LAPACK method (EPSLAPACK) for the slepc eigensolver, as I found out that it was the fastest (I configured PETSc to use Intel MKL BLAS/LAPACK routines). But I just found out that it is using a lot of memory irrespective of the sparsity of the matrix. For example, on a matrix of order 9600x9600 no matter how sparse/dense it is (I tested from 0.02% to 75% sparsity), it used 1.3gb of peak memory every time, whereas the part of the program in which the matrix is assembled in AIJ format (in case of 75% sparsity) takes only 550mb. The MKL program takes 3.1gb. Although memory requirement is less, the difference is constant irrespective of sparsity.So,* it seems that the LAPACK method will use memory dependent on the order of the matrix, irrespective of the sparsity.* So I guess its not very suitable for my purpose ? I also tried changing the NCV and MPD (Maximum projected Dimension) parameters hoping they would somehow reduce memory requirements even if at the cost of more time, but whatever I pass NCV and MPD (in EPSSetDimensions) as, they are not considered at all. They remain the same as the default values (as I verified with EPSGetDimensions). So I tried other solvers like the default, krylovschur, lanczos, but I think they are built to find only a few eigenvalues and not all, because they almost never converged to find ALL eigenvalues, even if I gave them a lot of time (big max number of iterations parameter). But their memory requirements seem to be pretty less. For example, I ran the program with EPSLANCZOS on the same 9600x9600 matrix (75% sparsity) with maximum number of iterations as 500,000. It ran it for 6.5 hrs (then I killed it), and at the end it was using 530mb only (although I dont know how near it was to completion). I am somewhat illiterate about the mathematics behind all this, and how the lapack,krylovschur, etc. work and I have no idea if what I am experiencing was expected/obvious, although I half-expected some things by reading the slepc documentation, where a table was given comparing different eigensolvers. Also, I have observed that as the sparsity of the matrix increases (more zeroes), more eigenvalues also become zeroes. Is there a way to exploit this using probably krylovschur/lanczos so that there is no need to converge to all eigenvalues (meaning that the time they will take will be reasonable), but to only the number of eigenvalues that are non-zero,so that I can assume the non-converged eigenvalues to be simply zeroes (but is there a way to know beforehand by sparsity approximately how many non-zeroe eigenvalues will be there, and the corresponding value of maximum iterations that are required ?) Is there a way to do what I want to do through SLEPc ??? * I want ALL eigenvalues and want that the memory requirements of the eigensolver should be less/in accordance with the sparsity of the matrix*, otherwise the very purpose of using a sparse matrix representation is somewhat defeated. I am doing all this in FORTRAN by the way. Thanks a lot in advance !!! Shitij -------------- next part -------------- An HTML attachment was scrubbed... URL: From clemens.domanig at uibk.ac.at Wed Aug 3 02:51:10 2011 From: clemens.domanig at uibk.ac.at (Clemens Domanig) Date: Wed, 03 Aug 2011 09:51:10 +0200 Subject: [petsc-users] PCFactorSetMatSolverPackage / PCFactorGetMatSolverPackage In-Reply-To: References: <4E37C338.9010406@uibk.ac.at> <4E37F344.40104@uibk.ac.at> Message-ID: <4E38FDEE.1040903@uibk.ac.at> Hong, I already asked at the mumps-user-list (by mistake I told them I want to do a Cholesky-decomp.) There they told me: >> Sorry this functionality is not available: the diagonal of the decomposition is hidden in internal, distributed, datastructures. Furthermore, the diagonal elemens depend on scalings, permutations and preprocessing options. Also MUMPS uses LDLt rather than Cholesky, and in some cases even LU on part of the matrix, even on symmetric matrices, when the last Schur complement is processed using scalapack. << But for me petsc's LDLt seem to be fine - I will see what will happen when I try the large-size problems. Thx Hong Zhang wrote: > Clemens, > Petsc's LDLt decomposition is likely different from mumps' LDLt. > Are you sure you want Petsc's D, but use mumps LDLt? > > I suggest sending a request to mumps developer on how to accessing mumps' D. > > Hong > > On Tue, Aug 2, 2011 at 8:12 AM, Matthew Knepley wrote: >> On Tue, Aug 2, 2011 at 12:53 PM, Clemens Domanig >> wrote: >>> Maybe I should describe the hole problem. >>> I'm using MUMPS to make LDLt decomposition - all from command line. >>> But sometimes I need the diagonal entries (respectively the number of >>> negative entries) of the D matrix. But there is no way to get this >>> information back from MUMPS via petsc ( as far as I found out). >> Yes, we have no access to MUMPS internal data. >> >>> There was the suggestion to use MatGetIntertia but this is only for >>> off-diagonal-information. >> I am not sure what you mean by that. >> >>> So I used >>> PCSetType( prec, PCCHOLESKY); >>> PCFactorGetMatrix( prec, &M); >>> MatGetDiagonal( M, z); >>> It only works without MUMPS and gives me the diagonal entries I want in >>> Matrix M - although the entries are 1/x. >>> Vector z is full of zeros - only the last entry is the 'same' as in M (x >>> instead of 1/x). >>> >>> So the idea was to somehow turn off MUMPS if I need the diagonal entries >>> and then turn it on again. >> Here is the easiest way to do it. Keep two complete solvers >> a) KSP with MUMPS >> b) KSP with Cholesky, give this one KSPSetOptionsPrefix(ksp, "diag_"); >> That way -diag_pc_type cholesky will work. Then use the solver you want. >> >>> The problem with PCFactorGetMatSolverPackage is I didn't manage to use it. >>> >>> const MatSolverPackage pack; >>> PCFactorGetMatSolverPackage( prec, &pack); >>> PetscSynchronizedPrintf( PETSC_COMM_WORLD, "%s\n", pack); >>> I expected 'mumps' as output but got 'H??????tdH3%0' >> It would seem its not yet setup. >> Matt >> >>> Thx >>> >>> >>> >>> Matthew Knepley wrote: >>>> On Tue, Aug 2, 2011 at 9:28 AM, Clemens Domanig >>>> > wrote: >>>> >>>> Hi, >>>> >>>> usually I'm using MUMPS by puting '-pc_factor_mat_solver_package >>>> mumps' into the .petscrc. >>>> Now I want to change in runtime between MUMPS and non-MUMPS so I >>>> tried to turn it on with >>>> PCFactorSetMatSolverPackage( pre, MAT_SOLVER_MUMPS); >>>> but I doesn't have any effect. >>>> >>>> >>>> This is complicated because you have to set this at the right time during >>>> the solver setup process. >>>> Command line arguments are much easier. However, you should call this >>>> after the preconditioner >>>> type is set, but before it is setup. >>>> >>>> I also have a problem with PCFactorGetMatSolverPackage(). How can I >>>> get back an output as described in the documentation that has the >>>> type 'const'? >>>> >>>> >>>> What is the problem? I do not understand? >>>> >>>> Matt >>>> >>>> Thx - Clemens >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >> >> >> -- >> What most experimenters take for granted before they begin their experiments >> is infinitely more interesting than any results to which their experiments >> lead. >> -- Norbert Wiener >> From jroman at dsic.upv.es Wed Aug 3 04:01:21 2011 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 3 Aug 2011 11:01:21 +0200 Subject: [petsc-users] SLEPc eigensolver that uses minimal memory and finds ALL eigenvalues of a real symmetric sparse matrix in reasonable time In-Reply-To: References: Message-ID: <359DE4F9-1B2E-459F-8EF1-4B8E78A16023@dsic.upv.es> El 03/08/2011, a las 08:30, Shitij Bhargava escribi?: > Hi all !! > > > > The reason I have started using SLEPc/PETSc is that I want to reduce the memory requirements of a program that I already have in Intel MKL. The program's purpose is to calculate ALL eigenvalues and eigenvectors of a real symmetric sparse matrix. Intel MKL doesnt have a sparse solver for this (cant solve for eigenvalues of a matrix represented in sparse format), so I am using SLEPc, as it can do this. > (NOTE: by "sparsity" I mean ZEROES*100/(ZEROES+NONZEROES). Also the tolerance I am using is 10^-6) If you really want to compute ALL eigenvalues, then SLEPc is not the way to go. SLEPc is intended for computing a few eigenpairs, and in some cases a few percentage, e.g. 20% at most. Computing more than that with SLEPc is probably going to be less efficient than other alternatives. > > Till now I have been using the LAPACK method (EPSLAPACK) for the slepc eigensolver, as I found out that it was the fastest (I configured PETSc to use Intel MKL BLAS/LAPACK routines). But I just found out that it is using a lot of memory irrespective of the sparsity of the matrix. For example, on a matrix of order 9600x9600 no matter how sparse/dense it is (I tested from 0.02% to 75% sparsity), it used 1.3gb of peak memory every time, whereas the part of the program in which the matrix is assembled in AIJ format (in case of 75% sparsity) takes only 550mb. The MKL program takes 3.1gb. Although memory requirement is less, the difference is constant irrespective of sparsity.So, it seems that the LAPACK method will use memory dependent on the order of the matrix, irrespective of the sparsity. So I guess its not very suitable for my purpose ? I also tried changing the NCV and MPD (Maximum projected Dimension) parameters hoping they would somehow reduce memory requirements even if at the cost of more time, but whatever I pass NCV and MPD (in EPSSetDimensions) as, they are not considered at all. They remain the same as the default values (as I verified with EPSGetDimensions). The EPSLAPACK solver converts the matrix to dense format. As explained in the documentation, this solver must be used only on small matrices for debugging purposes. NCV and MPD are not relevant in this setting. > > So I tried other solvers like the default, krylovschur, lanczos, but I think they are built to find only a few eigenvalues and not all, because they almost never converged to find ALL eigenvalues, even if I gave them a lot of time (big max number of iterations parameter). But their memory requirements seem to be pretty less. For example, I ran the program with EPSLANCZOS on the same 9600x9600 matrix (75% sparsity) with maximum number of iterations as 500,000. It ran it for 6.5 hrs (then I killed it), and at the end it was using 530mb only (although I dont know how near it was to completion). Krylov-Schur is almost always preferred over Lanczos, since the latter implements an explicit restart which is much worse than implicit restart in Krylov-Schur. Anyway, in both of them if you set NEV=9600 you are building a projected problem (dense matrix) of order equal to the original matrix so you have the same cost as with EPSLAPACK together with the cost of building the Krylov subspace. Again: do not use SLEPc for all eigenvalues! > > I am somewhat illiterate about the mathematics behind all this, and how the lapack,krylovschur, etc. work and I have no idea if what I am experiencing was expected/obvious, although I half-expected some things by reading the slepc documentation, where a table was given comparing different eigensolvers. A general recommendation is to try to understand a little bit how the algorithms work before attempting to use them. > > Also, I have observed that as the sparsity of the matrix increases (more zeroes), more eigenvalues also become zeroes. Is there a way to exploit this using probably krylovschur/lanczos so that there is no need to converge to all eigenvalues (meaning that the time they will take will be reasonable), but to only the number of eigenvalues that are non-zero,so that I can assume the non-converged eigenvalues to be simply zeroes (but is there a way to know beforehand by sparsity approximately how many non-zeroe eigenvalues will be there, and the corresponding value of maximum iterations that are required ?) There is no direct connection between the number of zero entries and the number of zero eigenvalues, unless you have rows/columns with all zeros, in which case the (permuted) matrix can be written as A=[A11 0; 0 0] so it is enough to compute eigenvalues of A11. > > Is there a way to do what I want to do through SLEPc ??? I want ALL eigenvalues and want that the memory requirements of the eigensolver should be less/in accordance with the sparsity of the matrix, otherwise the very purpose of using a sparse matrix representation is somewhat defeated. I am doing all this in FORTRAN by the way. Ask yourself this question: is it really necessary to compute all eigenvalues? If the answer is yes then maybe ScaLAPACK is what you need. Jose > > > Thanks a lot in advance !!! > > > Shitij From mirzadeh at gmail.com Wed Aug 3 04:08:45 2011 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Wed, 3 Aug 2011 02:08:45 -0700 Subject: [petsc-users] VecScatter Message-ID: Hi all, I have a question regarding the VecScatter routine. If I make a call like: VecScatterCreate(old_vec, PETSC_NULL, new_vec, is_scat, &vec_scat); VecScatterBegin(vec_scat, old_vec, new_vec, INSERT_VALUES, SCATTER_FORWARD); VecScatterEnd(vec_scat, old_vec, new_vec, INSERT_VALUES, SCATTER_FORWARD); what is it that will happen? Does this takes the values of old_vec vector and put them into new_vec and takes care of all communications? if so, is this done according to the global or the local numbering in the index set "is_scat"? Thanks, Mohammad -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Aug 3 08:47:06 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 3 Aug 2011 08:47:06 -0500 Subject: [petsc-users] VecScatter In-Reply-To: References: Message-ID: On Wed, Aug 3, 2011 at 04:08, Mohammad Mirzadeh wrote: > I have a question regarding the VecScatter routine. If I make a call like: > > VecScatterCreate(old_vec, PETSC_NULL, new_vec, is_scat, &vec_scat); > This creates map from the entire old_vec to the subset of new_vec indexed by the global indices is_scat. > VecScatterBegin(vec_scat, old_vec, new_vec, INSERT_VALUES, > SCATTER_FORWARD); > VecScatterEnd(vec_scat, old_vec, new_vec, INSERT_VALUES, SCATTER_FORWARD); > > what is it that will happen? Does this takes the values of old_vec vector > and put them into new_vec and takes care of all communications? > Yes, they go into new_vec according to the global indices is_scat. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Wed Aug 3 16:39:16 2011 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Wed, 3 Aug 2011 14:39:16 -0700 Subject: [petsc-users] VecScatter In-Reply-To: References: Message-ID: Thank you Jed. On Wed, Aug 3, 2011 at 6:47 AM, Jed Brown wrote: > On Wed, Aug 3, 2011 at 04:08, Mohammad Mirzadeh wrote: > >> I have a question regarding the VecScatter routine. If I make a call like: >> >> VecScatterCreate(old_vec, PETSC_NULL, new_vec, is_scat, &vec_scat); >> > > This creates map from the entire old_vec to the subset of new_vec indexed > by the global indices is_scat. > > >> VecScatterBegin(vec_scat, old_vec, new_vec, INSERT_VALUES, >> SCATTER_FORWARD); >> VecScatterEnd(vec_scat, old_vec, new_vec, INSERT_VALUES, SCATTER_FORWARD); >> >> what is it that will happen? Does this takes the values of old_vec vector >> and put them into new_vec and takes care of all communications? >> > > Yes, they go into new_vec according to the global indices is_scat. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shitij.cse at gmail.com Thu Aug 4 01:48:18 2011 From: shitij.cse at gmail.com (Shitij Bhargava) Date: Thu, 4 Aug 2011 12:18:18 +0530 Subject: [petsc-users] SLEPc eigensolver that uses minimal memory and finds ALL eigenvalues of a real symmetric sparse matrix in reasonable time In-Reply-To: <359DE4F9-1B2E-459F-8EF1-4B8E78A16023@dsic.upv.es> References: <359DE4F9-1B2E-459F-8EF1-4B8E78A16023@dsic.upv.es> Message-ID: Thank you very much for your reply. I am a summer trainee, and I talked to my guide about this, and now what I want might be a little different. I really do want all the eigenvalues. There cannot be any compromise on that. But please note that time is not much of a concern here, the much bigger concern is of memory. Even if I can distribute the memory somehow for the computation of all eigenvalues, that is fine. (doesnt matter if it takes a huge amount of time, he is willing to wait). Can any slepc Eigensolver distribute memory this way while solving for all eigenvalues ? I am sorry if what I am asking is obvious, I dont know much about distributed memory, etc. also. I am stupid at this moment, but I am learning. Also, when I ran the slepc program on a small cluster, and then ran the top command, I saw that while solving for eigenvalues, the program was already using upto 500% CPU. Does this mean that the eigensolver was "distributing memory" automatically among different nodes ? I understand that it must be distributing computational effort, but how do I know if it was distributing memory also ? It simply showed me the RES memory usage to be 1.3 gb, and the %MEM to be about 17%. Also, I did not use any MPIXXX data structures. I simply used SeqAIJ. Will using MPIAIJ "distribute memory" while solving for all eigenvalues ? (I understand that it will distribute memory to store the matrix, but the memory bottleneck is eigenvalue solver, so its more important for memory to be distributed then ) My guide said that one node will not have enough memory, but all nodes combined will, and hence the requirement of memory to be distributed. I read about ScaLAPACK, and I have already asked on their forums to confirm if it is suitable for me. I am waiting for an answer. Thank you once again. Shitij On 3 August 2011 14:31, Jose E. Roman wrote: > > El 03/08/2011, a las 08:30, Shitij Bhargava escribi?: > > > Hi all !! > > > > > > > > The reason I have started using SLEPc/PETSc is that I want to reduce the > memory requirements of a program that I already have in Intel MKL. The > program's purpose is to calculate ALL eigenvalues and eigenvectors of a real > symmetric sparse matrix. Intel MKL doesnt have a sparse solver for this > (cant solve for eigenvalues of a matrix represented in sparse format), so I > am using SLEPc, as it can do this. > > (NOTE: by "sparsity" I mean ZEROES*100/(ZEROES+NONZEROES). Also the > tolerance I am using is 10^-6) > > If you really want to compute ALL eigenvalues, then SLEPc is not the way to > go. SLEPc is intended for computing a few eigenpairs, and in some cases a > few percentage, e.g. 20% at most. Computing more than that with SLEPc is > probably going to be less efficient than other alternatives. > > > > > Till now I have been using the LAPACK method (EPSLAPACK) for the slepc > eigensolver, as I found out that it was the fastest (I configured PETSc to > use Intel MKL BLAS/LAPACK routines). But I just found out that it is using a > lot of memory irrespective of the sparsity of the matrix. For example, on a > matrix of order 9600x9600 no matter how sparse/dense it is (I tested from > 0.02% to 75% sparsity), it used 1.3gb of peak memory every time, whereas > the part of the program in which the matrix is assembled in AIJ format (in > case of 75% sparsity) takes only 550mb. The MKL program takes 3.1gb. > Although memory requirement is less, the difference is constant irrespective > of sparsity.So, it seems that the LAPACK method will use memory dependent on > the order of the matrix, irrespective of the sparsity. So I guess its not > very suitable for my purpose ? I also tried changing the NCV and MPD > (Maximum projected Dimension) parameters hoping they would somehow reduce > memory requirements even if at the cost of more time, but whatever I pass > NCV and MPD (in EPSSetDimensions) as, they are not considered at all. They > remain the same as the default values (as I verified with EPSGetDimensions). > > The EPSLAPACK solver converts the matrix to dense format. As explained in > the documentation, this solver must be used only on small matrices for > debugging purposes. NCV and MPD are not relevant in this setting. > > > > > So I tried other solvers like the default, krylovschur, lanczos, but I > think they are built to find only a few eigenvalues and not all, because > they almost never converged to find ALL eigenvalues, even if I gave them a > lot of time (big max number of iterations parameter). But their memory > requirements seem to be pretty less. For example, I ran the program with > EPSLANCZOS on the same 9600x9600 matrix (75% sparsity) with maximum number > of iterations as 500,000. It ran it for 6.5 hrs (then I killed it), and at > the end it was using 530mb only (although I dont know how near it was to > completion). > > Krylov-Schur is almost always preferred over Lanczos, since the latter > implements an explicit restart which is much worse than implicit restart in > Krylov-Schur. Anyway, in both of them if you set NEV=9600 you are building a > projected problem (dense matrix) of order equal to the original matrix so > you have the same cost as with EPSLAPACK together with the cost of building > the Krylov subspace. Again: do not use SLEPc for all eigenvalues! > > > > > I am somewhat illiterate about the mathematics behind all this, and how > the lapack,krylovschur, etc. work and I have no idea if what I am > experiencing was expected/obvious, although I half-expected some things by > reading the slepc documentation, where a table was given comparing different > eigensolvers. > > A general recommendation is to try to understand a little bit how the > algorithms work before attempting to use them. > > > > > Also, I have observed that as the sparsity of the matrix increases (more > zeroes), more eigenvalues also become zeroes. Is there a way to exploit this > using probably krylovschur/lanczos so that there is no need to converge to > all eigenvalues (meaning that the time they will take will be reasonable), > but to only the number of eigenvalues that are non-zero,so that I can assume > the non-converged eigenvalues to be simply zeroes (but is there a way to > know beforehand by sparsity approximately how many non-zeroe eigenvalues > will be there, and the corresponding value of maximum iterations that are > required ?) > > There is no direct connection between the number of zero entries and the > number of zero eigenvalues, unless you have rows/columns with all zeros, in > which case the (permuted) matrix can be written as A=[A11 0; 0 0] so it is > enough to compute eigenvalues of A11. > > > > > Is there a way to do what I want to do through SLEPc ??? I want ALL > eigenvalues and want that the memory requirements of the eigensolver should > be less/in accordance with the sparsity of the matrix, otherwise the very > purpose of using a sparse matrix representation is somewhat defeated. I am > doing all this in FORTRAN by the way. > > Ask yourself this question: is it really necessary to compute all > eigenvalues? If the answer is yes then maybe ScaLAPACK is what you need. > > Jose > > > > > > > Thanks a lot in advance !!! > > > > > > Shitij > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Thu Aug 4 06:27:46 2011 From: jroman at dsic.upv.es (Jose E. Roman) Date: Thu, 4 Aug 2011 13:27:46 +0200 Subject: [petsc-users] SLEPc eigensolver that uses minimal memory and finds ALL eigenvalues of a real symmetric sparse matrix in reasonable time In-Reply-To: References: <359DE4F9-1B2E-459F-8EF1-4B8E78A16023@dsic.upv.es> Message-ID: <52C4738E-914B-425C-9A2F-D64F8AB03564@dsic.upv.es> El 04/08/2011, a las 08:48, Shitij Bhargava escribi?: > Thank you very much for your reply. > > I am a summer trainee, and I talked to my guide about this, and now what I want might be a little different. > > I really do want all the eigenvalues. There cannot be any compromise on that. But please note that time is not much of a concern here, the much bigger concern is of memory. Even if I can distribute the memory somehow for the computation of all eigenvalues, that is fine. (doesnt matter if it takes a huge amount of time, he is willing to wait). Can any slepc Eigensolver distribute memory this way while solving for all eigenvalues ? > > I am sorry if what I am asking is obvious, I dont know much about distributed memory, etc. also. I am stupid at this moment, but I am learning. > > Also, when I ran the slepc program on a small cluster, and then ran the top command, I saw that while solving for eigenvalues, the program was already using upto 500% CPU. Does this mean that the eigensolver was "distributing memory" automatically among different nodes ? I understand that it must be distributing computational effort, but how do I know if it was distributing memory also ? It simply showed me the RES memory usage to be 1.3 gb, and the %MEM to be about 17%. Also, I did not use any MPIXXX data structures. I simply used SeqAIJ. Will using MPIAIJ "distribute memory" while solving for all eigenvalues ? (I understand that it will distribute memory to store the matrix, but the memory bottleneck is eigenvalue solver, so its more important for memory to be distributed then ) My guide said that one node will not have enough memory, but all nodes combined will, and hence the requirement of memory to be distributed. > > I read about ScaLAPACK, and I have already asked on their forums to confirm if it is suitable for me. I am waiting for an answer. > > Thank you once again. > > Shitij Try something like this: $ mpirun -np 8 ./program -eps_nev 4800 -eps_mpd 400 -eps_largest_real $ mpirun -np 8 ./program -eps_nev 4800 -eps_mpd 400 -eps_smallest_real But this may not work for the smallest ones. Jose From huyaoyu1986 at gmail.com Thu Aug 4 10:16:02 2011 From: huyaoyu1986 at gmail.com (huyaoyu) Date: Thu, 04 Aug 2011 23:16:02 +0800 Subject: [petsc-users] test failed on 2 processes and segmentation fault while initializing in deal.II Message-ID: <1312470962.23501.40.camel@ubuntu.ubuntu-domain> Hi everyone, I am new to PETSc. The reason I should use PETSc is because another package named deal.II will get help from PETSc. The computer has ubuntu 11.04 64bit installed. deal.II version is 7.0 So I downloaded the PETSc source code (3.1-p8) and do the configuration (after setting the environment variables correctly) like this: ./conf/configure.py --with-cc=gcc --with-fc=gfortran -download-f-blas-lapack=1 -download-mpich=1 --with-shared The configuration went well I supposed. Then I invoked the command: make all test But the test log (that appeared on the shell) showed that the test had failed when it was trying to perform a task in 2 processes. ========================================= Running test examples to verify correct installation C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI processes --------------Error detected during compile or link!----------------------- See http://www.mcs.anl.gov/petsc/petsc-2/documentation/troubleshooting.html /home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/bin/mpif90 -c -fPIC -Wall -Wno-unused-variable -g -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include -I/home/huyaoyu/Downloads/petsc-3.1-p8/include -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include -I/usr/lib/openmpi/include -I/usr/lib/openmpi/lib -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include -I/usr/lib/openmpi/include -I/usr/lib/openmpi/lib -o ex5f.o ex5f.F ex5f.F:92.72: call PetscOptionsGetReal(PETSC_NULL_CHARACTER,'-par',lambda, 1 Warning: Line truncated at (1) WITH SOME OTHER WARNINGS LIKE THE ABOVE ... THEN /home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/bin/mpif90 -fPIC -Wall -Wno-unused-variable -g -o ex5f ex5f.o -Wl,-rpath,/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/lib -L/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/lib -lpetsc -lX11 -Wl,-rpath,/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/lib -L/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/lib -lflapack -lfblas -lnsl -lrt -Wl,-rpath,/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/lib -L/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/lib -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.5.2 -L/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.5.2 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lmpich -lrt -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -lmpichf90 -lmpi_f90 -lmpi_f77 -lgfortran -lm -lm -lm -lm -ldl -lmpich -lrt -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl /bin/rm -f ex5f.o Fortran example src/snes/examples/tutorials/ex5f run successfully with 1 MPI process Completed test examples ================================================================== I just ignore it because I think if PETSc succeeds on 1 process and that is all I want for now. 1 process is enough for deal.II to work on my computer. (deal.II may work on cluster, but I only have on computer right now. And I just want to see some particular tutorial programs of deal.II that only work on cluster).Then after I re-build deal.II this PETSc support, I triggered the deal.II program that will use PETSc library. And I got the following segmentation fault: huyaoyu at ubuntu:~/Downloads/deal.II-non-threads/examples/step-17 $ ./step-17 [ubuntu:23497] *** Process received signal *** [ubuntu:23497] Signal: Segmentation fault (11) [ubuntu:23497] Signal code: Address not mapped (1) [ubuntu:23497] Failing at address: 0x44000098 [ubuntu:23497] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) [0x7f111b24ac60] [ubuntu:23497] [ 1] /usr/lib/libmpi.so.0(MPI_Comm_rank+0x5e) [0x7f111adbf3ce] [ubuntu:23497] [ 2] /home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/lib/libpetsc.so(PetscInitialize+0x5d1) [0x7f11190b97df] [ubuntu:23497] [ 3] ./step-17(main+0x3b) [0x42539f] [ubuntu:23497] [ 4] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main +0xff) [0x7f1119793eff] [ubuntu:23497] [ 5] ./step-17() [0x4252a9] [ubuntu:23497] *** End of error message *** Segmentation fault After debug the deal.II program I found that it failed at the following line pinit.c line 578: ierr = MPI_Comm_rank(MPI_COMM_WORLD,&PetscGlobalRank);CHKERRQ(ierr); gdb tells me that: MPI_COMM_WORLD = 1140850688 (0x44000000) PetscGlobalRank = -1 And MPI_Comm_rank failed at some where the disassembly code is "testb $0x90,0x98(%rbx)" The deal.II program triggers the same error every time it runs. Then I remove the whole PETSc directory and extract the source files again. Configure PETSc use the following line --with-cc=mpicc --with-fc=mpif90 --download-f-blas-lapack=1 --download-mpich=1 --with-shared Again, the PETSc test failed on 2 processes exactly the same with before. And the deal.II program gave the segmentation error at the identical place(pinit.c, ierr = MPI_Comm_rank(MPI_COMM_WORLD,&PetscGlobalRank);CHKERRQ(ierr);). I searched the mailing list and got this https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2008-July/003194.html The guys were talking about a problem just likes mine. They said that a fresh re-installation of PETSc will solve the problem. But the solution they got dose not work for me. How can I fix it? It seems that the installation of PETSc may be wrong. I have mpich2 installed on the system, it that matter? And I tried to configure PETSc without the argument "-download-mpich=1" then compile, but the same test error and segmentation error appeared. Does it have some thing to do with the 64bit issue? Or, do I have to post an email to petsc-maint at mcs.anl.gov along with the configure.log file attached? Any help, thanks! Best, HuYaoyu From balay at mcs.anl.gov Thu Aug 4 10:24:20 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 4 Aug 2011 10:24:20 -0500 (CDT) Subject: [petsc-users] test failed on 2 processes and segmentation fault while initializing in deal.II In-Reply-To: <1312470962.23501.40.camel@ubuntu.ubuntu-domain> References: <1312470962.23501.40.camel@ubuntu.ubuntu-domain> Message-ID: On Thu, 4 Aug 2011, huyaoyu wrote: > Hi everyone, > > I am new to PETSc. The reason I should use PETSc is because another > package named deal.II will get help from PETSc. The computer has ubuntu > 11.04 64bit installed. deal.II version is 7.0 > > So I downloaded the PETSc source code (3.1-p8) and do the configuration > (after setting the environment variables correctly) like this: > ./conf/configure.py --with-cc=gcc --with-fc=gfortran > -download-f-blas-lapack=1 -download-mpich=1 --with-shared > > The configuration went well I supposed. > > Then I invoked the command: > make all test > > But the test log (that appeared on the shell) showed that the test had > failed when it was trying to perform a task in 2 processes. > ========================================= > Running test examples to verify correct installation > C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 > MPI process > C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 > MPI processes > --------------Error detected during compile or > link!----------------------- > See > http://www.mcs.anl.gov/petsc/petsc-2/documentation/troubleshooting.html > /home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/bin/mpif90 -c > -fPIC -Wall -Wno-unused-variable -g > -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include > -I/home/huyaoyu/Downloads/petsc-3.1-p8/include > -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include > -I/usr/lib/openmpi/include -I/usr/lib/openmpi/lib > -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include > -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include > -I/usr/lib/openmpi/include -I/usr/lib/openmpi/lib -o ex5f.o ex5f.F > ex5f.F:92.72: > > call PetscOptionsGetReal(PETSC_NULL_CHARACTER,'-par',lambda, > > 1 > Warning: Line truncated at (1) > > WITH SOME OTHER WARNINGS LIKE THE ABOVE > ... > THEN > > /home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/bin/mpif90 -fPIC > -Wall -Wno-unused-variable -g -o ex5f ex5f.o > -Wl,-rpath,/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/lib > -L/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/lib -lpetsc > -lX11 > -Wl,-rpath,/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/lib > -L/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/lib -lflapack > -lfblas -lnsl -lrt > -Wl,-rpath,/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/lib > -L/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/lib > -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib > -Wl,-rpath,/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.5.2 > -L/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.5.2 > -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl > -lmpich -lrt -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread > -lmpichf90 -lmpi_f90 -lmpi_f77 -lgfortran -lm -lm -lm -lm -ldl -lmpich > -lrt -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl > /bin/rm -f ex5f.o > Fortran example src/snes/examples/tutorials/ex5f run successfully with 1 > MPI process > Completed test examples > ================================================================== > > I just ignore it because I think if PETSc succeeds on 1 process and that > is all I want for now. 1 process is enough for deal.II to work on my > computer. (deal.II may work on cluster, but I only have on computer > right now. And I just want to see some particular tutorial programs of > deal.II that only work on cluster).Then after I re-build deal.II this > PETSc support, I triggered the deal.II program that will use PETSc > library. And I got the following segmentation fault: > > huyaoyu at ubuntu:~/Downloads/deal.II-non-threads/examples/step-17 > $ ./step-17 > [ubuntu:23497] *** Process received signal *** > [ubuntu:23497] Signal: Segmentation fault (11) > [ubuntu:23497] Signal code: Address not mapped (1) > [ubuntu:23497] Failing at address: 0x44000098 > [ubuntu:23497] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) > [0x7f111b24ac60] > [ubuntu:23497] [ 1] /usr/lib/libmpi.so.0(MPI_Comm_rank+0x5e) ^^^^^^^^^^^^ Looks like this is system install MPI - not MPI from --download-mpich. > [0x7f111adbf3ce] > [ubuntu:23497] > [ 2] /home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/lib/libpetsc.so(PetscInitialize+0x5d1) [0x7f11190b97df] > [ubuntu:23497] [ 3] ./step-17(main+0x3b) [0x42539f] > [ubuntu:23497] [ 4] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main > +0xff) [0x7f1119793eff] > [ubuntu:23497] [ 5] ./step-17() [0x4252a9] > [ubuntu:23497] *** End of error message *** > Segmentation fault > > After debug the deal.II program I found that it failed at the following > line > > pinit.c line 578: > ierr = MPI_Comm_rank(MPI_COMM_WORLD,&PetscGlobalRank);CHKERRQ(ierr); > > gdb tells me that: > MPI_COMM_WORLD = 1140850688 (0x44000000) > PetscGlobalRank = -1 > And MPI_Comm_rank failed at some where the disassembly code is > "testb $0x90,0x98(%rbx)" > > The deal.II program triggers the same error every time it runs. > > Then I remove the whole PETSc directory and extract the source files > again. Configure PETSc use the following line > > --with-cc=mpicc --with-fc=mpif90 --download-f-blas-lapack=1 > --download-mpich=1 --with-shared > > Again, the PETSc test failed on 2 processes exactly the same with > before. And the deal.II program gave the segmentation error at the > identical place(pinit.c, ierr = > MPI_Comm_rank(MPI_COMM_WORLD,&PetscGlobalRank);CHKERRQ(ierr);). > > I searched the mailing list and got this > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2008-July/003194.html > The guys were talking about a problem just likes mine. They said that a > fresh re-installation of PETSc will solve the problem. But the solution > they got dose not work for me. > > How can I fix it? It seems that the installation of PETSc may be wrong. > I have mpich2 installed on the system, it that matter? And I tried to > configure PETSc without the argument "-download-mpich=1" then compile, > but the same test error and segmentation error appeared. > > Does it have some thing to do with the 64bit issue? Or, do I have to > post an email to petsc-maint at mcs.anl.gov along with the configure.log > file attached? > > Any help, thanks! Avoid mixing multiple MPI impls. use a single MPI for all packages [PETSc, deal.II etc..] So remove the currently install PETSc with mpich - and start from scratch with system installed MPI. i.e rm -rf linux-gnu-c-debug externalpackages ./configure --with-cc=mpicc --with-fc=mpif90 --download-f-blas-lapack=1 --with-shared Satish > > Best, > > HuYaoyu > > From jedbrown at mcs.anl.gov Thu Aug 4 10:26:42 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 4 Aug 2011 09:26:42 -0600 Subject: [petsc-users] test failed on 2 processes and segmentation fault while initializing in deal.II In-Reply-To: <1312470962.23501.40.camel@ubuntu.ubuntu-domain> References: <1312470962.23501.40.camel@ubuntu.ubuntu-domain> Message-ID: On Thu, Aug 4, 2011 at 09:16, huyaoyu wrote: > --------------Error detected during compile or > link!----------------------- > See > http://www.mcs.anl.gov/petsc/petsc-2/documentation/troubleshooting.html > /home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/bin/mpif90 -c > -fPIC -Wall -Wno-unused-variable -g > -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include > -I/home/huyaoyu/Downloads/petsc-3.1-p8/include > -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include > -I/usr/lib/openmpi/include -I/usr/lib/openmpi/lib > -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include > -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include > -I/usr/lib/openmpi/include -I/usr/lib/openmpi/lib -o ex5f.o ex5f.F > ex5f.F:92.72: > > call PetscOptionsGetReal(PETSC_NULL_CHARACTER,'-par',lambda, > > 1 > Warning: Line truncated at (1) > This is harmless, the Fortran compiler is creating a spurious warning. The problem has been fixed in gcc-4.6. > huyaoyu at ubuntu:~/Downloads/deal.II-non-threads/examples/step-17 > $ ./step-17 > [ubuntu:23497] *** Process received signal *** > [ubuntu:23497] Signal: Segmentation fault (11) > [ubuntu:23497] Signal code: Address not mapped (1) > [ubuntu:23497] Failing at address: 0x44000098 > [ubuntu:23497] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) > [0x7f111b24ac60] > [ubuntu:23497] [ 1] /usr/lib/libmpi.so.0(MPI_Comm_rank+0x5e) > [0x7f111adbf3ce] > [ubuntu:23497] > [ 2] > /home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/lib/libpetsc.so(PetscInitialize+0x5d1) > [0x7f11190b97df] > [ubuntu:23497] [ 3] ./step-17(main+0x3b) [0x42539f] > [ubuntu:23497] [ 4] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main > +0xff) [0x7f1119793eff] > [ubuntu:23497] [ 5] ./step-17() [0x4252a9] > [ubuntu:23497] *** End of error message *** > Segmentation fault > > After debug the deal.II program I found that it failed at the following > line > > pinit.c line 578: > ierr = MPI_Comm_rank(MPI_COMM_WORLD,&PetscGlobalRank);CHKERRQ(ierr); > Can you send the output of "ldd dealii-example"? This normally happens when a different MPI is linked from that which is compiled (or it's inconsistent between PETSc and deal.ii). -------------- next part -------------- An HTML attachment was scrubbed... URL: From ping.rong at tuhh.de Thu Aug 4 16:55:52 2011 From: ping.rong at tuhh.de (Ping Rong) Date: Thu, 04 Aug 2011 23:55:52 +0200 Subject: [petsc-users] compile own solver with petsc-dev Message-ID: <4E3B1568.1080600@tuhh.de> Hello all, I have developed a ksp solver, which is located in /src/ksp/ksp/impls/xxx. I used petsc-3.1.0-p8, when I changed the solver code, I updated petsc with cd /src/ksp/ksp/impls/xxx make ACTION=libfast cd $PETSC-ROOT make ranlib shared I have recently started using petsc-dev, which switches to cmake to compile the library. By the above method, the compilation still went through, however, when I try to link my program to libpetsc.so, it gives errors of unresolved symbols. Is there a new way to partially update the library without running the whole compilation over? any help would be appreciated! -- Ping Rong, M.Sc. Hamburg University of Technology Institut of modelling and computation Denickestra?e 17 (Room 3031) 21073 Hamburg Tel.: ++49 - (0)40 42878 2749 Fax: ++49 - (0)40 42878 43533 Email: ping.rong at tuhh.de From balay at mcs.anl.gov Thu Aug 4 17:00:02 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 4 Aug 2011 17:00:02 -0500 (CDT) Subject: [petsc-users] compile own solver with petsc-dev In-Reply-To: <4E3B1568.1080600@tuhh.de> References: <4E3B1568.1080600@tuhh.de> Message-ID: On Thu, 4 Aug 2011, Ping Rong wrote: > Hello all, > > I have developed a ksp solver, which is located in /src/ksp/ksp/impls/xxx. I > used petsc-3.1.0-p8, when I changed the solver code, I updated petsc with > cd /src/ksp/ksp/impls/xxx > make ACTION=libfast > cd $PETSC-ROOT > make ranlib shared > I have recently started using petsc-dev, which switches to cmake to compile > the library. If configure finds cmake - then 'make all' defaults to cmake build. Its still possible to get the old build with: make all-legacy > By the above method, the compilation still went through, however, > when I try to link my program to libpetsc.so, it gives errors of unresolved > symbols. Is there a new way to partially update the library without running > the whole compilation over? any help would be appreciated! With cmake build - you could do: [anywhere in petsc tree] make cmake Satish From stali at geology.wisc.edu Thu Aug 4 18:31:06 2011 From: stali at geology.wisc.edu (Tabrez Ali) Date: Thu, 04 Aug 2011 18:31:06 -0500 Subject: [petsc-users] Metis partitioning question Message-ID: <4E3B2BBA.5000408@geology.wisc.edu> Hello If a mesh is partitioned using Metis but the original order of nodes is used to create/partition the matrix then would this be significantly bad for performance and scalability. Also is there an example that uses Metis to partition an unstructured mesh? Thanks in advance. Tabrez From knepley at gmail.com Thu Aug 4 18:43:57 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 4 Aug 2011 23:43:57 +0000 Subject: [petsc-users] Metis partitioning question In-Reply-To: <4E3B2BBA.5000408@geology.wisc.edu> References: <4E3B2BBA.5000408@geology.wisc.edu> Message-ID: On Thu, Aug 4, 2011 at 11:31 PM, Tabrez Ali wrote: > Hello > > If a mesh is partitioned using Metis but the original order of nodes is > used to create/partition the matrix then would this be significantly bad for > performance and scalability. > That depends. Many mesh generators h ave fairly good ordering on output, but some do not. Definitely a bad order can make a huge difference in performance, both for cache traffic and conditioning reasons. > Also is there an example that uses Metis to partition an unstructured mesh? > This depends on what you mean. If you can encode your mesh as a Mat, you can use MatPartitoning. If not, you would need to use DMMesh things, such as SNES ex12. Matt > Thanks in advance. > > Tabrez -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Thu Aug 4 19:52:47 2011 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Thu, 4 Aug 2011 17:52:47 -0700 Subject: [petsc-users] Metis partitioning question In-Reply-To: References: <4E3B2BBA.5000408@geology.wisc.edu> Message-ID: if you want to use a general unstructured grid, you may want to have a look at $PETSC_DIR/src/dm/ao/examples/tutorials/ex2.c Mohammad On Thu, Aug 4, 2011 at 4:43 PM, Matthew Knepley wrote: > On Thu, Aug 4, 2011 at 11:31 PM, Tabrez Ali wrote: > >> Hello >> >> If a mesh is partitioned using Metis but the original order of nodes is >> used to create/partition the matrix then would this be significantly bad for >> performance and scalability. >> > > That depends. Many mesh generators h ave fairly good ordering on output, > but some do not. Definitely a bad order can make a huge > difference in performance, both for cache traffic and conditioning reasons. > > >> Also is there an example that uses Metis to partition an unstructured >> mesh? >> > > This depends on what you mean. If you can encode your mesh as a Mat, you > can use MatPartitoning. If not, > you would need to use DMMesh things, such as SNES ex12. > > Matt > > >> Thanks in advance. >> >> Tabrez > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Debao.Shao at brion.com Thu Aug 4 21:20:24 2011 From: Debao.Shao at brion.com (Debao Shao) Date: Thu, 4 Aug 2011 19:20:24 -0700 Subject: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate In-Reply-To: References: <4E3B2BBA.5000408@geology.wisc.edu> Message-ID: <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> DA: Sometimes I may encounter crashing when using PETSc, it doesn't happen always. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Unknown Name on a linux-gnu named r0leaf17 by tachyon Thu Aug 4 04:04:19 2011 [0]PETSC ERROR: Libraries linked from /home/dshao/dev_t2k/third_party/OOQP/OOQP-0.99.19/extras/petsc-3.1-p8/linux-gnu-c-debug/lib [0]PETSC ERROR: Configure run at Thu Apr 21 18:09:19 2011 [0]PETSC ERROR: Configure options --with-mpi=0 --with-debugging=0 -with-log=0 -with-info=0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file Do you happen to know the possible cause? I also have a question about the usage of "PetscInitialize" & "PetscFinalize". I can't totally understand them from the manual. Now, I wrote it as: Int main(int argc, char **argv) { PetscInitialize(0, 0, 0, 0 ); VerifyApp app(argc, argv); app.Initialize(); app.Run(); PetscFinalize(); return 0; } Then, built a binary, say "verify", and executed "verify" on a server first, then, started another "verify" on client by server. The program ran at client side can be multiple processes and multiple threads. Is my usage correct? Your answer is appreciated. Thanks, Debao ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From huyaoyu1986 at gmail.com Thu Aug 4 21:20:47 2011 From: huyaoyu1986 at gmail.com (huyaoyu) Date: Fri, 05 Aug 2011 10:20:47 +0800 Subject: [petsc-users] test failed on 2 processes and segmentation fault while initializing in deal.II In-Reply-To: References: Message-ID: <1312510847.25256.18.camel@ubuntu.ubuntu-domain> Thanks to Satish Balay and Jed Brown. Your advices are pretty good! I remove the petsc directory and deal.II directory. Re-build petsc and deal.II with the mpi installed system wide. The configuration lines are as follows: For PETSc: ./config/configure.py --with-cc=/usr/bin/mpicc --with-fc=/usr/bin/mpif90 --download-f-blas-lapack=1 --with-shared For deal.II: ./configure --enable-shared --disable-threads --with-petsc=$PETSC_DIR --with-petsc-arch=$(PETSC_ARCH) --with-p4est=PATH-TO-P4EST --with-mpi I will use p4est for grid distribution and I checked the configure output of deal.II. The output says deal.II will use /usr/bin/mpicc for CC variable and /usr/bin/mpiCC for CXX variable. And deal.II says: =================deal.II configure output========================= checking for PETSc library directory... /home/huyaoyu/Downloads/petsc-3.1-p8 checking for PETSc version... 3.1.0 checking for PETSc library architecture... linux-gnu-c-debug checking for PETSc libmpiuni library... not found checking for consistency of PETSc and deal.II MPI settings... yes checking for PETSc scalar complex... no ================= end of deal.II configure output================= After the compilation of PETSc and deal.II, I tried to compile the specific example program of deal.II which will use PETSc. And everything works well. No segmentation error any more! Great! However, the test of PETSc trigger the same error while trying to perform task in 2 process. Is it because I am using PETSc on a single machine but not a cluster of computers? Again, thanks to Satish Balay and Jed Brown, you are great! HuYaoyu. > > --------------Error detected during compile or > > link!----------------------- > > See > > http://www.mcs.anl.gov/petsc/petsc-2/documentation/troubleshooting.html > > /home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/bin/mpif90 -c > > -fPIC -Wall -Wno-unused-variable -g > > -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include > > -I/home/huyaoyu/Downloads/petsc-3.1-p8/include > > -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include > > -I/usr/lib/openmpi/include -I/usr/lib/openmpi/lib > > -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include > > -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include > > -I/usr/lib/openmpi/include -I/usr/lib/openmpi/lib -o ex5f.o ex5f.F > > ex5f.F:92.72: > > > > call PetscOptionsGetReal(PETSC_NULL_CHARACTER,'-par',lambda, > > > > 1 > > Warning: Line truncated at (1) > > > > This is harmless, the Fortran compiler is creating a spurious warning. The > problem has been fixed in gcc-4.6. > > > > huyaoyu at ubuntu:~/Downloads/deal.II-non-threads/examples/step-17 > > $ ./step-17 > > [ubuntu:23497] *** Process received signal *** > > [ubuntu:23497] Signal: Segmentation fault (11) > > [ubuntu:23497] Signal code: Address not mapped (1) > > [ubuntu:23497] Failing at address: 0x44000098 > > [ubuntu:23497] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) > > [0x7f111b24ac60] > > [ubuntu:23497] [ 1] /usr/lib/libmpi.so.0(MPI_Comm_rank+0x5e) > > [0x7f111adbf3ce] > > [ubuntu:23497] > > [ 2] > > /home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/lib/libpetsc.so(PetscInitialize+0x5d1) > > [0x7f11190b97df] > > [ubuntu:23497] [ 3] ./step-17(main+0x3b) [0x42539f] > > [ubuntu:23497] [ 4] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main > > +0xff) [0x7f1119793eff] > > [ubuntu:23497] [ 5] ./step-17() [0x4252a9] > > [ubuntu:23497] *** End of error message *** > > Segmentation fault > > > > After debug the deal.II program I found that it failed at the following > > line > > > > pinit.c line 578: > > ierr = MPI_Comm_rank(MPI_COMM_WORLD,&PetscGlobalRank);CHKERRQ(ierr); > > > > Can you send the output of "ldd dealii-example"? > > This normally happens when a different MPI is linked from that which is > compiled (or it's inconsistent between PETSc and deal.ii). From huyaoyu1986 at gmail.com Thu Aug 4 21:22:03 2011 From: huyaoyu1986 at gmail.com (huyaoyu) Date: Fri, 05 Aug 2011 10:22:03 +0800 Subject: [petsc-users] test failed on 2 processes and segmentation fault while initializing in deal.II In-Reply-To: References: Message-ID: <1312510923.25256.19.camel@ubuntu.ubuntu-domain> Thanks to Satish Balay and Jed Brown. Your advices are pretty good! I remove the petsc directory and deal.II directory. Re-build petsc and deal.II with the mpi installed system wide. The configuration lines are as follows: For PETSc: ./config/configure.py --with-cc=/usr/bin/mpicc --with-fc=/usr/bin/mpif90 --download-f-blas-lapack=1 --with-shared For deal.II: ./configure --enable-shared --disable-threads --with-petsc=$PETSC_DIR --with-petsc-arch=$(PETSC_ARCH) --with-p4est=PATH-TO-P4EST --with-mpi I will use p4est for grid distribution and I checked the configure output of deal.II. The output says deal.II will use /usr/bin/mpicc for CC variable and /usr/bin/mpiCC for CXX variable. And deal.II says: =================deal.II configure output========================= checking for PETSc library directory... /home/huyaoyu/Downloads/petsc-3.1-p8 checking for PETSc version... 3.1.0 checking for PETSc library architecture... linux-gnu-c-debug checking for PETSc libmpiuni library... not found checking for consistency of PETSc and deal.II MPI settings... yes checking for PETSc scalar complex... no ================= end of deal.II configure output================= After the compilation of PETSc and deal.II, I tried to compile the specific example program of deal.II which will use PETSc. And everything works well. No segmentation error any more! Great! However, the test of PETSc trigger the same error while trying to perform task in 2 process. Is it because I am using PETSc on a single machine but not a cluster of computers? Again, thanks to Satish Balay and Jed Brown, you are great! HuYaoyu. > > --------------Error detected during compile or > > link!----------------------- > > See > > http://www.mcs.anl.gov/petsc/petsc-2/documentation/troubleshooting.html > > /home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/bin/mpif90 -c > > -fPIC -Wall -Wno-unused-variable -g > > -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include > > -I/home/huyaoyu/Downloads/petsc-3.1-p8/include > > -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include > > -I/usr/lib/openmpi/include -I/usr/lib/openmpi/lib > > -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include > > -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include > > -I/usr/lib/openmpi/include -I/usr/lib/openmpi/lib -o ex5f.o ex5f.F > > ex5f.F:92.72: > > > > call PetscOptionsGetReal(PETSC_NULL_CHARACTER,'-par',lambda, > > > > 1 > > Warning: Line truncated at (1) > > > > This is harmless, the Fortran compiler is creating a spurious warning. The > problem has been fixed in gcc-4.6. > > > > huyaoyu at ubuntu:~/Downloads/deal.II-non-threads/examples/step-17 > > $ ./step-17 > > [ubuntu:23497] *** Process received signal *** > > [ubuntu:23497] Signal: Segmentation fault (11) > > [ubuntu:23497] Signal code: Address not mapped (1) > > [ubuntu:23497] Failing at address: 0x44000098 > > [ubuntu:23497] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) > > [0x7f111b24ac60] > > [ubuntu:23497] [ 1] /usr/lib/libmpi.so.0(MPI_Comm_rank+0x5e) > > [0x7f111adbf3ce] > > [ubuntu:23497] > > [ 2] > > /home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/lib/libpetsc.so(PetscInitialize+0x5d1) > > [0x7f11190b97df] > > [ubuntu:23497] [ 3] ./step-17(main+0x3b) [0x42539f] > > [ubuntu:23497] [ 4] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main > > +0xff) [0x7f1119793eff] > > [ubuntu:23497] [ 5] ./step-17() [0x4252a9] > > [ubuntu:23497] *** End of error message *** > > Segmentation fault > > > > After debug the deal.II program I found that it failed at the following > > line > > > > pinit.c line 578: > > ierr = MPI_Comm_rank(MPI_COMM_WORLD,&PetscGlobalRank);CHKERRQ(ierr); > > > > Can you send the output of "ldd dealii-example"? > > This normally happens when a different MPI is linked from that which is > compiled (or it's inconsistent between PETSc and deal.ii). From knepley at gmail.com Thu Aug 4 21:24:33 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 5 Aug 2011 02:24:33 +0000 Subject: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate In-Reply-To: <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> References: <4E3B2BBA.5000408@geology.wisc.edu> <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> Message-ID: On Fri, Aug 5, 2011 at 2:20 AM, Debao Shao wrote: > ** > > DA: **** > > ** ** > > Sometimes I may encounter crashing when using PETSc, it doesn?t happen > always. **** > > ** ** > > [0]PETSC ERROR: > ------------------------------------------------------------------------** > ** > > [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the > batch system) has told this process to end**** > > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > **** > > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSCERROR: or try > http://valgrind.org on GNU/linux**** > > and Apple Mac OS X to find memory corruption errors**** > > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run**** > > [0]PETSC ERROR: to get more information on the crash.**** > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------**** > > [0]PETSC ERROR: Signal received!**** > > [0]PETSC ERROR: > ------------------------------------------------------------------------** > ** > > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 > CDT 2011**** > > [0]PETSC ERROR: See docs/changes/index.html for recent updates.**** > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.**** > > [0]PETSC ERROR: See docs/index.html for manual pages.**** > > [0]PETSC ERROR: > ------------------------------------------------------------------------** > ** > > [0]PETSC ERROR: Unknown Name on a linux-gnu named r0leaf17 by tachyon Thu > Aug 4 04:04:19 2011**** > > [0]PETSC ERROR: Libraries linked from > /home/dshao/dev_t2k/third_party/OOQP/OOQP-0.99.19/extras/petsc-3.1-p8/linux-gnu-c-debug/lib > **** > > [0]PETSC ERROR: Configure run at Thu Apr 21 18:09:19 2011**** > > [0]PETSC ERROR: Configure options --with-mpi=0 --with-debugging=0 > -with-log=0 -with-info=0**** > > [0]PETSC ERROR: > ------------------------------------------------------------------------** > ** > > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file**** > > ** ** > > Do you happen to know the possible cause?**** > > ** ** > > I also have a question about the usage of ?PetscInitialize? & > ?PetscFinalize?. I can?t totally understand them from the manual. **** > > ** ** > > Now, I wrote it as: **** > > Int main(int argc, char **argv)**** > > {**** > > PetscInitialize(0, 0, 0, 0 ); > This is not a valid call. Try PetscInitialize(&argc, &argv, 0, 0); Signal 15 is a segmentation fault, which means an illegal memory read or write. Matt > **** > > ** ** > > VerifyApp app(argc, argv);**** > > app.Initialize();**** > > app.Run();**** > > ** ** > > PetscFinalize();**** > > return 0;**** > > }**** > > ** ** > > Then, built a binary, say ?verify?, and executed ?verify? on a server > first, then, started another ?verify? on client by server. **** > > The program ran at client side can be multiple processes and multiple > threads.**** > > ** ** > > Is my usage correct? **** > > Your answer is appreciated. **** > > ** ** > > Thanks,**** > > **De**bao**** > > ------------------------------ > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Aug 4 21:27:19 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 4 Aug 2011 21:27:19 -0500 (CDT) Subject: [petsc-users] test failed on 2 processes and segmentation fault while initializing in deal.II In-Reply-To: <1312510923.25256.19.camel@ubuntu.ubuntu-domain> References: <1312510923.25256.19.camel@ubuntu.ubuntu-domain> Message-ID: On Fri, 5 Aug 2011, huyaoyu wrote: > Thanks to Satish Balay and Jed Brown. Your advices are pretty good! > > I remove the petsc directory and deal.II directory. Re-build petsc and > deal.II with the mpi installed system wide. The configuration lines are > as follows: > > For PETSc: > ./config/configure.py --with-cc=/usr/bin/mpicc --with-fc=/usr/bin/mpif90 > --download-f-blas-lapack=1 --with-shared > > For deal.II: > ./configure --enable-shared --disable-threads --with-petsc=$PETSC_DIR > --with-petsc-arch=$(PETSC_ARCH) --with-p4est=PATH-TO-P4EST --with-mpi > > I will use p4est for grid distribution and I checked the configure > output of deal.II. The output says deal.II will use /usr/bin/mpicc for > CC variable and /usr/bin/mpiCC for CXX variable. And deal.II says: > =================deal.II configure output========================= > checking for PETSc library > directory... /home/huyaoyu/Downloads/petsc-3.1-p8 > checking for PETSc version... 3.1.0 > checking for PETSc library architecture... linux-gnu-c-debug > checking for PETSc libmpiuni library... not found > checking for consistency of PETSc and deal.II MPI settings... yes > checking for PETSc scalar complex... no > ================= end of deal.II configure output================= > > After the compilation of PETSc and deal.II, I tried to compile the > specific example program of deal.II which will use PETSc. And everything > works well. No segmentation error any more! Great! > > However, the test of PETSc trigger the same error while trying to > perform task in 2 process. Is it because I am using PETSc on a single > machine but not a cluster of computers? You mean the compiler warning for the fortan example in 'make test'? You can ignore that. Satish > > Again, thanks to Satish Balay and Jed Brown, you are great! > > HuYaoyu. > > > > --------------Error detected during compile or > > > link!----------------------- > > > See > > > http://www.mcs.anl.gov/petsc/petsc-2/documentation/troubleshooting.html > > > /home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/bin/mpif90 -c > > > -fPIC -Wall -Wno-unused-variable -g > > > -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include > > > -I/home/huyaoyu/Downloads/petsc-3.1-p8/include > > > -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include > > > -I/usr/lib/openmpi/include -I/usr/lib/openmpi/lib > > > -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include > > > -I/home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/include > > > -I/usr/lib/openmpi/include -I/usr/lib/openmpi/lib -o ex5f.o ex5f.F > > > ex5f.F:92.72: > > > > > > call PetscOptionsGetReal(PETSC_NULL_CHARACTER,'-par',lambda, > > > > > > 1 > > > Warning: Line truncated at (1) > > > > > > > This is harmless, the Fortran compiler is creating a spurious warning. The > > problem has been fixed in gcc-4.6. > > > > > > > huyaoyu at ubuntu:~/Downloads/deal.II-non-threads/examples/step-17 > > > $ ./step-17 > > > [ubuntu:23497] *** Process received signal *** > > > [ubuntu:23497] Signal: Segmentation fault (11) > > > [ubuntu:23497] Signal code: Address not mapped (1) > > > [ubuntu:23497] Failing at address: 0x44000098 > > > [ubuntu:23497] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) > > > [0x7f111b24ac60] > > > [ubuntu:23497] [ 1] /usr/lib/libmpi.so.0(MPI_Comm_rank+0x5e) > > > [0x7f111adbf3ce] > > > [ubuntu:23497] > > > [ 2] > > > /home/huyaoyu/Downloads/petsc-3.1-p8/linux-gnu-c-debug/lib/libpetsc.so(PetscInitialize+0x5d1) > > > [0x7f11190b97df] > > > [ubuntu:23497] [ 3] ./step-17(main+0x3b) [0x42539f] > > > [ubuntu:23497] [ 4] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main > > > +0xff) [0x7f1119793eff] > > > [ubuntu:23497] [ 5] ./step-17() [0x4252a9] > > > [ubuntu:23497] *** End of error message *** > > > Segmentation fault > > > > > > After debug the deal.II program I found that it failed at the following > > > line > > > > > > pinit.c line 578: > > > ierr = MPI_Comm_rank(MPI_COMM_WORLD,&PetscGlobalRank);CHKERRQ(ierr); > > > > > > > Can you send the output of "ldd dealii-example"? > > > > This normally happens when a different MPI is linked from that which is > > compiled (or it's inconsistent between PETSc and deal.ii). > > > > > > From Debao.Shao at brion.com Thu Aug 4 21:33:05 2011 From: Debao.Shao at brion.com (Debao Shao) Date: Thu, 4 Aug 2011 19:33:05 -0700 Subject: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate In-Reply-To: References: <4E3B2BBA.5000408@geology.wisc.edu> <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> Message-ID: <384FF55F15E3E447802DC8CCA85696980E143123A1@EX03> So the real problem is the wrong usage of "PetscInitialize(0, 0, 0, 0 )", but I do have many successive runs before with this setting. Thanks, Debao ________________________________ From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Friday, August 05, 2011 10:25 AM To: PETSc users list Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate On Fri, Aug 5, 2011 at 2:20 AM, Debao Shao > wrote: DA: Sometimes I may encounter crashing when using PETSc, it doesn't happen always. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Unknown Name on a linux-gnu named r0leaf17 by tachyon Thu Aug 4 04:04:19 2011 [0]PETSC ERROR: Libraries linked from /home/dshao/dev_t2k/third_party/OOQP/OOQP-0.99.19/extras/petsc-3.1-p8/linux-gnu-c-debug/lib [0]PETSC ERROR: Configure run at Thu Apr 21 18:09:19 2011 [0]PETSC ERROR: Configure options --with-mpi=0 --with-debugging=0 -with-log=0 -with-info=0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file Do you happen to know the possible cause? I also have a question about the usage of "PetscInitialize" & "PetscFinalize". I can't totally understand them from the manual. Now, I wrote it as: Int main(int argc, char **argv) { PetscInitialize(0, 0, 0, 0 ); This is not a valid call. Try PetscInitialize(&argc, &argv, 0, 0); Signal 15 is a segmentation fault, which means an illegal memory read or write. Matt VerifyApp app(argc, argv); app.Initialize(); app.Run(); PetscFinalize(); return 0; } Then, built a binary, say "verify", and executed "verify" on a server first, then, started another "verify" on client by server. The program ran at client side can be multiple processes and multiple threads. Is my usage correct? Your answer is appreciated. Thanks, Debao ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Aug 4 21:35:29 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 5 Aug 2011 02:35:29 +0000 Subject: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate In-Reply-To: <384FF55F15E3E447802DC8CCA85696980E143123A1@EX03> References: <4E3B2BBA.5000408@geology.wisc.edu> <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A1@EX03> Message-ID: On Fri, Aug 5, 2011 at 2:33 AM, Debao Shao wrote: > ** > > So the real problem is the wrong usage of ?PetscInitialize(0, 0, 0, 0 )?, > but I do have many successive runs before with this setting. > Any kind of memory overwrite gives an SEGV. So other code, making a different error will give that too. Matt > > > Thanks,**** > > **De**bao**** > ------------------------------ > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Matthew Knepley > *Sent:* Friday, August 05, 2011 10:25 AM > *To:* **PETSc users list** > *Subject:* Re: [petsc-users] PETSC ERROR: Caught signal number 15 > Terminate**** > > ** ** > > On Fri, Aug 5, 2011 at 2:20 AM, **De**bao Shao > wrote:**** > > DA: **** > > **** > > Sometimes I may encounter crashing when using PETSc, it doesn?t happen > always. **** > > **** > > [0]PETSC ERROR: > ------------------------------------------------------------------------** > ** > > [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the > batch system) has told this process to end**** > > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > **** > > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSCERROR: or try > http://valgrind.org on GNU/linux**** > > and Apple Mac OS X to find memory corruption errors**** > > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run**** > > [0]PETSC ERROR: to get more information on the crash.**** > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------**** > > [0]PETSC ERROR: Signal received!**** > > [0]PETSC ERROR: > ------------------------------------------------------------------------** > ** > > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 > CDT 2011**** > > [0]PETSC ERROR: See docs/changes/index.html for recent updates.**** > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.**** > > [0]PETSC ERROR: See docs/index.html for manual pages.**** > > [0]PETSC ERROR: > ------------------------------------------------------------------------** > ** > > [0]PETSC ERROR: Unknown Name on a linux-gnu named r0leaf17 by tachyon Thu > Aug 4 04:04:19 2011**** > > [0]PETSC ERROR: Libraries linked from > /home/dshao/dev_t2k/third_party/OOQP/OOQP-0.99.19/extras/petsc-3.1-p8/linux-gnu-c-debug/lib > **** > > [0]PETSC ERROR: Configure run at Thu Apr 21 18:09:19 2011**** > > [0]PETSC ERROR: Configure options --with-mpi=0 --with-debugging=0 > -with-log=0 -with-info=0**** > > [0]PETSC ERROR: > ------------------------------------------------------------------------** > ** > > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file**** > > **** > > Do you happen to know the possible cause?**** > > **** > > I also have a question about the usage of ?PetscInitialize? & > ?PetscFinalize?. I can?t totally understand them from the manual. **** > > **** > > Now, I wrote it as: **** > > Int main(int argc, char **argv)**** > > {**** > > PetscInitialize(0, 0, 0, 0 );**** > > ** ** > > This is not a valid call. Try**** > > ** ** > > PetscInitialize(&argc, &argv, 0, 0);**** > > ** ** > > Signal 15 is a segmentation fault, which means an illegal memory read or > write.**** > > ** ** > > Matt**** > > **** > > **** > > VerifyApp app(argc, argv);**** > > app.Initialize();**** > > app.Run();**** > > **** > > PetscFinalize();**** > > return 0;**** > > }**** > > **** > > Then, built a binary, say ?verify?, and executed ?verify? on a server > first, then, started another ?verify? on client by server. **** > > The program ran at client side can be multiple processes and multiple > threads.**** > > **** > > Is my usage correct? **** > > Your answer is appreciated. **** > > **** > > Thanks,**** > > **De**bao**** > > ** ** > ------------------------------ > > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt.**** > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener**** > > ------------------------------ > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Debao.Shao at brion.com Thu Aug 4 21:38:09 2011 From: Debao.Shao at brion.com (Debao Shao) Date: Thu, 4 Aug 2011 19:38:09 -0700 Subject: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate In-Reply-To: References: <4E3B2BBA.5000408@geology.wisc.edu> <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A1@EX03> Message-ID: <384FF55F15E3E447802DC8CCA85696980E143123A7@EX03> Doesn't SEGV fault is signal 11? ________________________________ From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Friday, August 05, 2011 10:35 AM To: PETSc users list Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate On Fri, Aug 5, 2011 at 2:33 AM, Debao Shao > wrote: So the real problem is the wrong usage of "PetscInitialize(0, 0, 0, 0 )", but I do have many successive runs before with this setting. Any kind of memory overwrite gives an SEGV. So other code, making a different error will give that too. Matt Thanks, Debao ________________________________ From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Friday, August 05, 2011 10:25 AM To: PETSc users list Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate On Fri, Aug 5, 2011 at 2:20 AM, Debao Shao > wrote: DA: Sometimes I may encounter crashing when using PETSc, it doesn't happen always. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Unknown Name on a linux-gnu named r0leaf17 by tachyon Thu Aug 4 04:04:19 2011 [0]PETSC ERROR: Libraries linked from /home/dshao/dev_t2k/third_party/OOQP/OOQP-0.99.19/extras/petsc-3.1-p8/linux-gnu-c-debug/lib [0]PETSC ERROR: Configure run at Thu Apr 21 18:09:19 2011 [0]PETSC ERROR: Configure options --with-mpi=0 --with-debugging=0 -with-log=0 -with-info=0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file Do you happen to know the possible cause? I also have a question about the usage of "PetscInitialize" & "PetscFinalize". I can't totally understand them from the manual. Now, I wrote it as: Int main(int argc, char **argv) { PetscInitialize(0, 0, 0, 0 ); This is not a valid call. Try PetscInitialize(&argc, &argv, 0, 0); Signal 15 is a segmentation fault, which means an illegal memory read or write. Matt VerifyApp app(argc, argv); app.Initialize(); app.Run(); PetscFinalize(); return 0; } Then, built a binary, say "verify", and executed "verify" on a server first, then, started another "verify" on client by server. The program ran at client side can be multiple processes and multiple threads. Is my usage correct? Your answer is appreciated. Thanks, Debao ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Aug 4 21:43:03 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 4 Aug 2011 20:43:03 -0600 Subject: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate In-Reply-To: <384FF55F15E3E447802DC8CCA85696980E143123A7@EX03> References: <4E3B2BBA.5000408@geology.wisc.edu> <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A1@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A7@EX03> Message-ID: <8287A816-74DC-4D96-9FAB-123E215D1AA7@mcs.anl.gov> On Aug 4, 2011, at 8:38 PM, Debao Shao wrote: > Doesn?t SEGV fault is signal 11? Yes., 15 is a software terminate single sent from some other process hence the error message: > Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end I've seen some batch systems send this single when they think you have run out of time. There may be other reasons as well. Do you get this right away or after a long time? Barry > > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley > Sent: Friday, August 05, 2011 10:35 AM > To: PETSc users list > Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate > > On Fri, Aug 5, 2011 at 2:33 AM, Debao Shao wrote: > So the real problem is the wrong usage of ?PetscInitialize(0, 0, 0, 0 )?, but I do have many successive runs before with this setting. > > Any kind of memory overwrite gives an SEGV. So other code, making a different error will give that too. > > Matt > > > Thanks, > Debao > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley > Sent: Friday, August 05, 2011 10:25 AM > To: PETSc users list > Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate > > On Fri, Aug 5, 2011 at 2:20 AM, Debao Shao wrote: > DA: > > Sometimes I may encounter crashing when using PETSc, it doesn?t happen always. > > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC ERROR: or try http://valgrind.org on GNU/linux > and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [0]PETSC ERROR: to get more information on the crash. > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Unknown Name on a linux-gnu named r0leaf17 by tachyon Thu Aug 4 04:04:19 2011 > [0]PETSC ERROR: Libraries linked from /home/dshao/dev_t2k/third_party/OOQP/OOQP-0.99.19/extras/petsc-3.1-p8/linux-gnu-c-debug/lib > [0]PETSC ERROR: Configure run at Thu Apr 21 18:09:19 2011 > [0]PETSC ERROR: Configure options --with-mpi=0 --with-debugging=0 -with-log=0 -with-info=0 > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > > Do you happen to know the possible cause? > > I also have a question about the usage of ?PetscInitialize? & ?PetscFinalize?. I can?t totally understand them from the manual. > > Now, I wrote it as: > Int main(int argc, char **argv) > { > PetscInitialize(0, 0, 0, 0 ); > > This is not a valid call. Try > > PetscInitialize(&argc, &argv, 0, 0); > > Signal 15 is a segmentation fault, which means an illegal memory read or write. > > Matt > > > VerifyApp app(argc, argv); > app.Initialize(); > app.Run(); > > PetscFinalize(); > return 0; > } > > Then, built a binary, say ?verify?, and executed ?verify? on a server first, then, started another ?verify? on client by server. > The program ran at client side can be multiple processes and multiple threads. > > Is my usage correct? > Your answer is appreciated. > > Thanks, > Debao > > -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. From knepley at gmail.com Thu Aug 4 21:43:35 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 5 Aug 2011 02:43:35 +0000 Subject: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate In-Reply-To: <384FF55F15E3E447802DC8CCA85696980E143123A7@EX03> References: <4E3B2BBA.5000408@geology.wisc.edu> <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A1@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A7@EX03> Message-ID: On Fri, Aug 5, 2011 at 2:38 AM, Debao Shao wrote: > ** > > Doesn?t SEGV fault is signal 11? > Yes, 15 is SIGTERM. It appears that either you have another signal handler installed which is terminating on an error, or something killed your program. I can guarantee you that the PetscInitialize(0,0,0,0) is a seg fault. Matt > > ------------------------------ > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Matthew Knepley > *Sent:* Friday, August 05, 2011 10:35 AM > > *To:* **PETSc users list** > *Subject:* Re: [petsc-users] PETSC ERROR: Caught signal number 15 > Terminate > **** > > ** ** > > On Fri, Aug 5, 2011 at 2:33 AM, **De**bao Shao > wrote:**** > > So the real problem is the wrong usage of ?PetscInitialize(0, 0, 0, 0 )?, > but I do have many successive runs before with this setting.**** > > ** ** > > Any kind of memory overwrite gives an SEGV. So other code, making a > different error will give that too.**** > > ** ** > > Matt**** > > **** > > **** > > Thanks,**** > > **De**bao**** > ------------------------------ > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Matthew Knepley > *Sent:* Friday, August 05, 2011 10:25 AM > *To:* **PETSc users list** > *Subject:* Re: [petsc-users] PETSC ERROR: Caught signal number 15 > Terminate**** > > **** > > On Fri, Aug 5, 2011 at 2:20 AM, **De**bao Shao > wrote:**** > > DA: **** > > **** > > Sometimes I may encounter crashing when using PETSc, it doesn?t happen > always. **** > > **** > > [0]PETSC ERROR: > ------------------------------------------------------------------------** > ** > > [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the > batch system) has told this process to end**** > > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > **** > > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSCERROR: or try > http://valgrind.org on GNU/linux**** > > and Apple Mac OS X to find memory corruption errors**** > > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run**** > > [0]PETSC ERROR: to get more information on the crash.**** > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------**** > > [0]PETSC ERROR: Signal received!**** > > [0]PETSC ERROR: > ------------------------------------------------------------------------** > ** > > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 > CDT 2011**** > > [0]PETSC ERROR: See docs/changes/index.html for recent updates.**** > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.**** > > [0]PETSC ERROR: See docs/index.html for manual pages.**** > > [0]PETSC ERROR: > ------------------------------------------------------------------------** > ** > > [0]PETSC ERROR: Unknown Name on a linux-gnu named r0leaf17 by tachyon Thu > Aug 4 04:04:19 2011**** > > [0]PETSC ERROR: Libraries linked from > /home/dshao/dev_t2k/third_party/OOQP/OOQP-0.99.19/extras/petsc-3.1-p8/linux-gnu-c-debug/lib > **** > > [0]PETSC ERROR: Configure run at Thu Apr 21 18:09:19 2011**** > > [0]PETSC ERROR: Configure options --with-mpi=0 --with-debugging=0 > -with-log=0 -with-info=0**** > > [0]PETSC ERROR: > ------------------------------------------------------------------------** > ** > > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file**** > > **** > > Do you happen to know the possible cause?**** > > **** > > I also have a question about the usage of ?PetscInitialize? & > ?PetscFinalize?. I can?t totally understand them from the manual. **** > > **** > > Now, I wrote it as: **** > > Int main(int argc, char **argv)**** > > {**** > > PetscInitialize(0, 0, 0, 0 );**** > > **** > > This is not a valid call. Try**** > > **** > > PetscInitialize(&argc, &argv, 0, 0);**** > > **** > > Signal 15 is a segmentation fault, which means an illegal memory read or > write.**** > > **** > > Matt**** > > **** > > **** > > VerifyApp app(argc, argv);**** > > app.Initialize();**** > > app.Run();**** > > **** > > PetscFinalize();**** > > return 0;**** > > }**** > > **** > > Then, built a binary, say ?verify?, and executed ?verify? on a server > first, then, started another ?verify? on client by server. **** > > The program ran at client side can be multiple processes and multiple > threads.**** > > **** > > Is my usage correct? **** > > Your answer is appreciated. **** > > **** > > Thanks,**** > > **De**bao**** > > **** > ------------------------------ > > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt.**** > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener**** > > ** ** > ------------------------------ > > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt.**** > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener**** > > ------------------------------ > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Debao.Shao at brion.com Thu Aug 4 21:50:42 2011 From: Debao.Shao at brion.com (Debao Shao) Date: Thu, 4 Aug 2011 19:50:42 -0700 Subject: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate In-Reply-To: References: <4E3B2BBA.5000408@geology.wisc.edu> <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A1@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A7@EX03> Message-ID: <384FF55F15E3E447802DC8CCA85696980E143123B7@EX03> Thanks, I'm trying your suggestion "PetscInitialize(&argc, &argv, 0, 0 )", hope it can fix the problem. Do you mind to explain a little "PetscInitialize" to me? 1, must it be call at the beginning of program? 2, what's the other purpose if I don't want to input any argc? 3, if I implement petsc as an optional solution, can I put "PetscInitialize" just before petsc is really executed, while not just put them at the beginning of program? Thanks, Debao ________________________________ From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Friday, August 05, 2011 10:44 AM To: PETSc users list Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate On Fri, Aug 5, 2011 at 2:38 AM, Debao Shao > wrote: Doesn't SEGV fault is signal 11? Yes, 15 is SIGTERM. It appears that either you have another signal handler installed which is terminating on an error, or something killed your program. I can guarantee you that the PetscInitialize(0,0,0,0) is a seg fault. Matt ________________________________ From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Friday, August 05, 2011 10:35 AM To: PETSc users list Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate On Fri, Aug 5, 2011 at 2:33 AM, Debao Shao > wrote: So the real problem is the wrong usage of "PetscInitialize(0, 0, 0, 0 )", but I do have many successive runs before with this setting. Any kind of memory overwrite gives an SEGV. So other code, making a different error will give that too. Matt Thanks, Debao ________________________________ From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Friday, August 05, 2011 10:25 AM To: PETSc users list Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate On Fri, Aug 5, 2011 at 2:20 AM, Debao Shao > wrote: DA: Sometimes I may encounter crashing when using PETSc, it doesn't happen always. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Unknown Name on a linux-gnu named r0leaf17 by tachyon Thu Aug 4 04:04:19 2011 [0]PETSC ERROR: Libraries linked from /home/dshao/dev_t2k/third_party/OOQP/OOQP-0.99.19/extras/petsc-3.1-p8/linux-gnu-c-debug/lib [0]PETSC ERROR: Configure run at Thu Apr 21 18:09:19 2011 [0]PETSC ERROR: Configure options --with-mpi=0 --with-debugging=0 -with-log=0 -with-info=0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file Do you happen to know the possible cause? I also have a question about the usage of "PetscInitialize" & "PetscFinalize". I can't totally understand them from the manual. Now, I wrote it as: Int main(int argc, char **argv) { PetscInitialize(0, 0, 0, 0 ); This is not a valid call. Try PetscInitialize(&argc, &argv, 0, 0); Signal 15 is a segmentation fault, which means an illegal memory read or write. Matt VerifyApp app(argc, argv); app.Initialize(); app.Run(); PetscFinalize(); return 0; } Then, built a binary, say "verify", and executed "verify" on a server first, then, started another "verify" on client by server. The program ran at client side can be multiple processes and multiple threads. Is my usage correct? Your answer is appreciated. Thanks, Debao ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Debao.Shao at brion.com Thu Aug 4 21:54:16 2011 From: Debao.Shao at brion.com (Debao Shao) Date: Thu, 4 Aug 2011 19:54:16 -0700 Subject: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate In-Reply-To: <8287A816-74DC-4D96-9FAB-123E215D1AA7@mcs.anl.gov> References: <4E3B2BBA.5000408@geology.wisc.edu> <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A1@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A7@EX03> <8287A816-74DC-4D96-9FAB-123E215D1AA7@mcs.anl.gov> Message-ID: <384FF55F15E3E447802DC8CCA85696980E143123BB@EX03> I get this error after a long process. For example, I have 4 jobs to do, the program may crash after done 1 job or 2, 3, 4 jobs. Also, sometimes, the program can run finished successfully. Thanks, Debao -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith Sent: Friday, August 05, 2011 10:43 AM To: PETSc users list Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate On Aug 4, 2011, at 8:38 PM, Debao Shao wrote: > Doesn't SEGV fault is signal 11? Yes., 15 is a software terminate single sent from some other process hence the error message: > Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end I've seen some batch systems send this single when they think you have run out of time. There may be other reasons as well. Do you get this right away or after a long time? Barry > > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley > Sent: Friday, August 05, 2011 10:35 AM > To: PETSc users list > Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate > > On Fri, Aug 5, 2011 at 2:33 AM, Debao Shao wrote: > So the real problem is the wrong usage of "PetscInitialize(0, 0, 0, 0 )", but I do have many successive runs before with this setting. > > Any kind of memory overwrite gives an SEGV. So other code, making a different error will give that too. > > Matt > > > Thanks, > Debao > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley > Sent: Friday, August 05, 2011 10:25 AM > To: PETSc users list > Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate > > On Fri, Aug 5, 2011 at 2:20 AM, Debao Shao wrote: > DA: > > Sometimes I may encounter crashing when using PETSc, it doesn't happen always. > > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC ERROR: or try http://valgrind.org on GNU/linux > and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [0]PETSC ERROR: to get more information on the crash. > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Unknown Name on a linux-gnu named r0leaf17 by tachyon Thu Aug 4 04:04:19 2011 > [0]PETSC ERROR: Libraries linked from /home/dshao/dev_t2k/third_party/OOQP/OOQP-0.99.19/extras/petsc-3.1-p8/linux-gnu-c-debug/lib > [0]PETSC ERROR: Configure run at Thu Apr 21 18:09:19 2011 > [0]PETSC ERROR: Configure options --with-mpi=0 --with-debugging=0 -with-log=0 -with-info=0 > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > > Do you happen to know the possible cause? > > I also have a question about the usage of "PetscInitialize" & "PetscFinalize". I can't totally understand them from the manual. > > Now, I wrote it as: > Int main(int argc, char **argv) > { > PetscInitialize(0, 0, 0, 0 ); > > This is not a valid call. Try > > PetscInitialize(&argc, &argv, 0, 0); > > Signal 15 is a segmentation fault, which means an illegal memory read or write. > > Matt > > > VerifyApp app(argc, argv); > app.Initialize(); > app.Run(); > > PetscFinalize(); > return 0; > } > > Then, built a binary, say "verify", and executed "verify" on a server first, then, started another "verify" on client by server. > The program ran at client side can be multiple processes and multiple threads. > > Is my usage correct? > Your answer is appreciated. > > Thanks, > Debao > > -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. From knepley at gmail.com Thu Aug 4 21:54:41 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 5 Aug 2011 02:54:41 +0000 Subject: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate In-Reply-To: <384FF55F15E3E447802DC8CCA85696980E143123B7@EX03> References: <4E3B2BBA.5000408@geology.wisc.edu> <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A1@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A7@EX03> <384FF55F15E3E447802DC8CCA85696980E143123B7@EX03> Message-ID: On Fri, Aug 5, 2011 at 2:50 AM, Debao Shao wrote: > ** > > Thanks, I?m trying your suggestion ?PetscInitialize(&argc, &argv, 0, 0 )?, > hope it can fix the problem. **** > > ** ** > > Do you mind to explain a little ?PetscInitialize? to me? **** > > 1, must it be call at the beginning of program? > It must be called before you call any PETSc function, or use MPI. > **** > > 2, what?s the other purpose if I don?t want to input any argc? > It initializes MPI. > **** > > 3, if I implement petsc as an optional solution, can I put > ?PetscInitialize? just before petsc is really executed, while not just put > them at the beginning of program? > Yes. Matt > Thanks,**** > > **De**bao**** > ------------------------------ > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Matthew Knepley > *Sent:* Friday, August 05, 2011 10:44 AM > *To:* **PETSc users list** > *Subject:* Re: [petsc-users] PETSC ERROR: Caught signal number 15 > Terminate**** > > ** ** > > On Fri, Aug 5, 2011 at 2:38 AM, **De**bao Shao > wrote:**** > > Doesn?t SEGV fault is signal 11?**** > > ** ** > > Yes, 15 is SIGTERM. It appears that either you have another signal handler > installed which**** > > is terminating on an error, or something killed your program. I can > guarantee you that the**** > > PetscInitialize(0,0,0,0) is a seg fault.**** > > ** ** > > Matt**** > > **** > > **** > ------------------------------ > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Matthew Knepley > *Sent:* Friday, August 05, 2011 10:35 AM**** > > > *To:* **PETSc users list** > *Subject:* Re: [petsc-users] PETSC ERROR: Caught signal number 15 > Terminate**** > > **** > > On Fri, Aug 5, 2011 at 2:33 AM, **De**bao Shao > wrote:**** > > So the real problem is the wrong usage of ?PetscInitialize(0, 0, 0, 0 )?, > but I do have many successive runs before with this setting.**** > > **** > > Any kind of memory overwrite gives an SEGV. So other code, making a > different error will give that too.**** > > **** > > Matt**** > > **** > > **** > > Thanks,**** > > **De**bao**** > ------------------------------ > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Matthew Knepley > *Sent:* Friday, August 05, 2011 10:25 AM > *To:* **PETSc users list** > *Subject:* Re: [petsc-users] PETSC ERROR: Caught signal number 15 > Terminate**** > > **** > > On Fri, Aug 5, 2011 at 2:20 AM, **De**bao Shao > wrote:**** > > DA: **** > > **** > > Sometimes I may encounter crashing when using PETSc, it doesn?t happen > always. **** > > **** > > [0]PETSC ERROR: > ------------------------------------------------------------------------** > ** > > [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the > batch system) has told this process to end**** > > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > **** > > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSCERROR: or try > http://valgrind.org on GNU/linux**** > > and Apple Mac OS X to find memory corruption errors**** > > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run**** > > [0]PETSC ERROR: to get more information on the crash.**** > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------**** > > [0]PETSC ERROR: Signal received!**** > > [0]PETSC ERROR: > ------------------------------------------------------------------------** > ** > > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 > CDT 2011**** > > [0]PETSC ERROR: See docs/changes/index.html for recent updates.**** > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.**** > > [0]PETSC ERROR: See docs/index.html for manual pages.**** > > [0]PETSC ERROR: > ------------------------------------------------------------------------** > ** > > [0]PETSC ERROR: Unknown Name on a linux-gnu named r0leaf17 by tachyon Thu > Aug 4 04:04:19 2011**** > > [0]PETSC ERROR: Libraries linked from > /home/dshao/dev_t2k/third_party/OOQP/OOQP-0.99.19/extras/petsc-3.1-p8/linux-gnu-c-debug/lib > **** > > [0]PETSC ERROR: Configure run at Thu Apr 21 18:09:19 2011**** > > [0]PETSC ERROR: Configure options --with-mpi=0 --with-debugging=0 > -with-log=0 -with-info=0**** > > [0]PETSC ERROR: > ------------------------------------------------------------------------** > ** > > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file**** > > **** > > Do you happen to know the possible cause?**** > > **** > > I also have a question about the usage of ?PetscInitialize? & > ?PetscFinalize?. I can?t totally understand them from the manual. **** > > **** > > Now, I wrote it as: **** > > Int main(int argc, char **argv)**** > > {**** > > PetscInitialize(0, 0, 0, 0 );**** > > **** > > This is not a valid call. Try**** > > **** > > PetscInitialize(&argc, &argv, 0, 0);**** > > **** > > Signal 15 is a segmentation fault, which means an illegal memory read or > write.**** > > **** > > Matt**** > > **** > > **** > > VerifyApp app(argc, argv);**** > > app.Initialize();**** > > app.Run();**** > > **** > > PetscFinalize();**** > > return 0;**** > > }**** > > **** > > Then, built a binary, say ?verify?, and executed ?verify? on a server > first, then, started another ?verify? on client by server. **** > > The program ran at client side can be multiple processes and multiple > threads.**** > > **** > > Is my usage correct? **** > > Your answer is appreciated. **** > > **** > > Thanks,**** > > **De**bao**** > > **** > ------------------------------ > > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt.**** > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener**** > > **** > ------------------------------ > > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt.**** > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener**** > > ** ** > ------------------------------ > > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt.**** > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener**** > > ------------------------------ > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Aug 4 21:55:31 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 5 Aug 2011 02:55:31 +0000 Subject: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate In-Reply-To: <384FF55F15E3E447802DC8CCA85696980E143123BB@EX03> References: <4E3B2BBA.5000408@geology.wisc.edu> <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A1@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A7@EX03> <8287A816-74DC-4D96-9FAB-123E215D1AA7@mcs.anl.gov> <384FF55F15E3E447802DC8CCA85696980E143123BB@EX03> Message-ID: On Fri, Aug 5, 2011 at 2:54 AM, Debao Shao wrote: > I get this error after a long process. > For example, I have 4 jobs to do, the program may crash after done 1 job or > 2, 3, 4 jobs. Also, sometimes, the program can run finished successfully. > This sounds like your system has a time limit (maybe ulimit) for jobs and kills your when they run too long. Matt > Thanks, > Debao > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith > Sent: Friday, August 05, 2011 10:43 AM > To: PETSc users list > Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate > > > On Aug 4, 2011, at 8:38 PM, Debao Shao wrote: > > > Doesn't SEGV fault is signal 11? > > Yes., 15 is a software terminate single sent from some other process > hence the error message: > > Caught signal number 15 Terminate: Somet process (or the batch system) > has told this process to end > > I've seen some batch systems send this single when they think you have > run out of time. There may be other reasons as well. Do you get this right > away or after a long time? > > Barry > > > > > > From: petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley > > Sent: Friday, August 05, 2011 10:35 AM > > To: PETSc users list > > Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate > > > > On Fri, Aug 5, 2011 at 2:33 AM, Debao Shao wrote: > > So the real problem is the wrong usage of "PetscInitialize(0, 0, 0, 0 )", > but I do have many successive runs before with this setting. > > > > Any kind of memory overwrite gives an SEGV. So other code, making a > different error will give that too. > > > > Matt > > > > > > Thanks, > > Debao > > From: petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley > > Sent: Friday, August 05, 2011 10:25 AM > > To: PETSc users list > > Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate > > > > On Fri, Aug 5, 2011 at 2:20 AM, Debao Shao wrote: > > DA: > > > > Sometimes I may encounter crashing when using PETSc, it doesn't happen > always. > > > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the > batch system) has told this process to end > > [0]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSCERROR: or try > http://valgrind.org on GNU/linux > > and Apple Mac OS X to find memory corruption errors > > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [0]PETSC ERROR: to get more information on the crash. > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > > [0]PETSC ERROR: Signal received! > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 > CDT 2011 > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [0]PETSC ERROR: See docs/index.html for manual pages. > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Unknown Name on a linux-gnu named r0leaf17 by tachyon Thu > Aug 4 04:04:19 2011 > > [0]PETSC ERROR: Libraries linked from > /home/dshao/dev_t2k/third_party/OOQP/OOQP-0.99.19/extras/petsc-3.1-p8/linux-gnu-c-debug/lib > > [0]PETSC ERROR: Configure run at Thu Apr 21 18:09:19 2011 > > [0]PETSC ERROR: Configure options --with-mpi=0 --with-debugging=0 > -with-log=0 -with-info=0 > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > > > > Do you happen to know the possible cause? > > > > I also have a question about the usage of "PetscInitialize" & > "PetscFinalize". I can't totally understand them from the manual. > > > > Now, I wrote it as: > > Int main(int argc, char **argv) > > { > > PetscInitialize(0, 0, 0, 0 ); > > > > This is not a valid call. Try > > > > PetscInitialize(&argc, &argv, 0, 0); > > > > Signal 15 is a segmentation fault, which means an illegal memory read or > write. > > > > Matt > > > > > > VerifyApp app(argc, argv); > > app.Initialize(); > > app.Run(); > > > > PetscFinalize(); > > return 0; > > } > > > > Then, built a binary, say "verify", and executed "verify" on a server > first, then, started another "verify" on client by server. > > The program ran at client side can be multiple processes and multiple > threads. > > > > Is my usage correct? > > Your answer is appreciated. > > > > Thanks, > > Debao > > > > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt. > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt. > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt. > > > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Aug 4 21:56:32 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 4 Aug 2011 20:56:32 -0600 Subject: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate In-Reply-To: <384FF55F15E3E447802DC8CCA85696980E143123BB@EX03> References: <4E3B2BBA.5000408@geology.wisc.edu> <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A1@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A7@EX03> <8287A816-74DC-4D96-9FAB-123E215D1AA7@mcs.anl.gov> <384FF55F15E3E447802DC8CCA85696980E143123BB@EX03> Message-ID: On Aug 4, 2011, at 8:54 PM, Debao Shao wrote: > I get this error after a long process. > For example, I have 4 jobs to do, the program may crash after done 1 job or 2, 3, 4 jobs. Also, sometimes, the program can run finished successfully. The program is not crashing. The batch system is kicking out the job, probably because it thinks it has run out of time. You'll need to set a longer time for the batch system. Barry > > Thanks, > Debao > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith > Sent: Friday, August 05, 2011 10:43 AM > To: PETSc users list > Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate > > > On Aug 4, 2011, at 8:38 PM, Debao Shao wrote: > >> Doesn't SEGV fault is signal 11? > > Yes., 15 is a software terminate single sent from some other process hence the error message: >> Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > > I've seen some batch systems send this single when they think you have run out of time. There may be other reasons as well. Do you get this right away or after a long time? > > Barry > > >> >> From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley >> Sent: Friday, August 05, 2011 10:35 AM >> To: PETSc users list >> Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate >> >> On Fri, Aug 5, 2011 at 2:33 AM, Debao Shao wrote: >> So the real problem is the wrong usage of "PetscInitialize(0, 0, 0, 0 )", but I do have many successive runs before with this setting. >> >> Any kind of memory overwrite gives an SEGV. So other code, making a different error will give that too. >> >> Matt >> >> >> Thanks, >> Debao >> From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley >> Sent: Friday, August 05, 2011 10:25 AM >> To: PETSc users list >> Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate >> >> On Fri, Aug 5, 2011 at 2:20 AM, Debao Shao wrote: >> DA: >> >> Sometimes I may encounter crashing when using PETSc, it doesn't happen always. >> >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC ERROR: or try http://valgrind.org on GNU/linux >> and Apple Mac OS X to find memory corruption errors >> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >> [0]PETSC ERROR: to get more information on the crash. >> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [0]PETSC ERROR: Signal received! >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Unknown Name on a linux-gnu named r0leaf17 by tachyon Thu Aug 4 04:04:19 2011 >> [0]PETSC ERROR: Libraries linked from /home/dshao/dev_t2k/third_party/OOQP/OOQP-0.99.19/extras/petsc-3.1-p8/linux-gnu-c-debug/lib >> [0]PETSC ERROR: Configure run at Thu Apr 21 18:09:19 2011 >> [0]PETSC ERROR: Configure options --with-mpi=0 --with-debugging=0 -with-log=0 -with-info=0 >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >> >> Do you happen to know the possible cause? >> >> I also have a question about the usage of "PetscInitialize" & "PetscFinalize". I can't totally understand them from the manual. >> >> Now, I wrote it as: >> Int main(int argc, char **argv) >> { >> PetscInitialize(0, 0, 0, 0 ); >> >> This is not a valid call. Try >> >> PetscInitialize(&argc, &argv, 0, 0); >> >> Signal 15 is a segmentation fault, which means an illegal memory read or write. >> >> Matt >> >> >> VerifyApp app(argc, argv); >> app.Initialize(); >> app.Run(); >> >> PetscFinalize(); >> return 0; >> } >> >> Then, built a binary, say "verify", and executed "verify" on a server first, then, started another "verify" on client by server. >> The program ran at client side can be multiple processes and multiple threads. >> >> Is my usage correct? >> Your answer is appreciated. >> >> Thanks, >> Debao >> >> -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. > > > -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. From huyaoyu1986 at gmail.com Thu Aug 4 21:56:51 2011 From: huyaoyu1986 at gmail.com (huyaoyu) Date: Fri, 05 Aug 2011 10:56:51 +0800 Subject: [petsc-users] test failed on 2 processes and segmentation fault while initializing in deal.II In-Reply-To: References: Message-ID: <1312513011.25256.28.camel@ubuntu.ubuntu-domain> Satish, Yes, PETSc outputs the fortran warings when I invoke "make test". As you said, I will just ignore them. But it also says that "Error detected during compile or link". =========PETSc output while testing======= huyaoyu at ubuntu:~/Downloads/petsc-3.1-p8$ make PETSC_DIR=/home/huyaoyu/Downloads/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug test Running test examples to verify correct installation C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI processes --------------Error detected during compile or link!----------------------- ... OTHER STUFF AND FORTRAN WARNINGS... Fortran example src/snes/examples/tutorials/ex5f run successfully with 1 MPI process Completed test examples =========End of PETSc output while testing== So, I just ignore this. Is that OK? Thanks! HuYaoyu > > Thanks to Satish Balay and Jed Brown. Your advices are pretty good! > > > > I remove the petsc directory and deal.II directory. Re-build petsc and > > deal.II with the mpi installed system wide. The configuration lines are > > as follows: > > > > For PETSc: > > ./config/configure.py --with-cc=/usr/bin/mpicc --with-fc=/usr/bin/mpif90 > > --download-f-blas-lapack=1 --with-shared > > > > For deal.II: > > ./configure --enable-shared --disable-threads --with-petsc=$PETSC_DIR > > --with-petsc-arch=$(PETSC_ARCH) --with-p4est=PATH-TO-P4EST --with-mpi > > > > I will use p4est for grid distribution and I checked the configure > > output of deal.II. The output says deal.II will use /usr/bin/mpicc for > > CC variable and /usr/bin/mpiCC for CXX variable. And deal.II says: > > =================deal.II configure output========================= > > checking for PETSc library > > directory... /home/huyaoyu/Downloads/petsc-3.1-p8 > > checking for PETSc version... 3.1.0 > > checking for PETSc library architecture... linux-gnu-c-debug > > checking for PETSc libmpiuni library... not found > > checking for consistency of PETSc and deal.II MPI settings... yes > > checking for PETSc scalar complex... no > > ================= end of deal.II configure output================= > > > > After the compilation of PETSc and deal.II, I tried to compile the > > specific example program of deal.II which will use PETSc. And everything > > works well. No segmentation error any more! Great! > > > > However, the test of PETSc trigger the same error while trying to > > perform task in 2 process. Is it because I am using PETSc on a single > > machine but not a cluster of computers? > > You mean the compiler warning for the fortan example in 'make test'? > > You can ignore that. > > Satish From bsmith at mcs.anl.gov Thu Aug 4 21:58:40 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 4 Aug 2011 20:58:40 -0600 Subject: [petsc-users] test failed on 2 processes and segmentation fault while initializing in deal.II In-Reply-To: <1312513011.25256.28.camel@ubuntu.ubuntu-domain> References: <1312513011.25256.28.camel@ubuntu.ubuntu-domain> Message-ID: It is not really an error. It comes from the fact that the Fortran compiler gave a warning message and it is not smart enough to ignore that error. Barry On Aug 4, 2011, at 8:56 PM, huyaoyu wrote: > Satish, > > Yes, PETSc outputs the fortran warings when I invoke "make test". As you > said, I will just ignore them. But it also says that "Error detected > during compile or link". > > =========PETSc output while testing======= > huyaoyu at ubuntu:~/Downloads/petsc-3.1-p8$ make > PETSC_DIR=/home/huyaoyu/Downloads/petsc-3.1-p8 > PETSC_ARCH=linux-gnu-c-debug test > Running test examples to verify correct installation > C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 > MPI process > C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 > MPI processes > --------------Error detected during compile or > link!----------------------- > ... OTHER STUFF AND FORTRAN WARNINGS... > Fortran example src/snes/examples/tutorials/ex5f run successfully with 1 > MPI process > Completed test examples > =========End of PETSc output while testing== > > So, I just ignore this. Is that OK? > > Thanks! > > HuYaoyu > >>> Thanks to Satish Balay and Jed Brown. Your advices are pretty good! >>> >>> I remove the petsc directory and deal.II directory. Re-build petsc and >>> deal.II with the mpi installed system wide. The configuration lines are >>> as follows: >>> >>> For PETSc: >>> ./config/configure.py --with-cc=/usr/bin/mpicc --with-fc=/usr/bin/mpif90 >>> --download-f-blas-lapack=1 --with-shared >>> >>> For deal.II: >>> ./configure --enable-shared --disable-threads --with-petsc=$PETSC_DIR >>> --with-petsc-arch=$(PETSC_ARCH) --with-p4est=PATH-TO-P4EST --with-mpi >>> >>> I will use p4est for grid distribution and I checked the configure >>> output of deal.II. The output says deal.II will use /usr/bin/mpicc for >>> CC variable and /usr/bin/mpiCC for CXX variable. And deal.II says: >>> =================deal.II configure output========================= >>> checking for PETSc library >>> directory... /home/huyaoyu/Downloads/petsc-3.1-p8 >>> checking for PETSc version... 3.1.0 >>> checking for PETSc library architecture... linux-gnu-c-debug >>> checking for PETSc libmpiuni library... not found >>> checking for consistency of PETSc and deal.II MPI settings... yes >>> checking for PETSc scalar complex... no >>> ================= end of deal.II configure output================= >>> >>> After the compilation of PETSc and deal.II, I tried to compile the >>> specific example program of deal.II which will use PETSc. And everything >>> works well. No segmentation error any more! Great! >>> >>> However, the test of PETSc trigger the same error while trying to >>> perform task in 2 process. Is it because I am using PETSc on a single >>> machine but not a cluster of computers? >> >> You mean the compiler warning for the fortan example in 'make test'? >> >> You can ignore that. >> >> Satish > > > From knepley at gmail.com Thu Aug 4 21:59:24 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 5 Aug 2011 02:59:24 +0000 Subject: [petsc-users] test failed on 2 processes and segmentation fault while initializing in deal.II In-Reply-To: <1312513011.25256.28.camel@ubuntu.ubuntu-domain> References: <1312513011.25256.28.camel@ubuntu.ubuntu-domain> Message-ID: On Fri, Aug 5, 2011 at 2:56 AM, huyaoyu wrote: > Satish, > > Yes, PETSc outputs the fortran warings when I invoke "make test". As you > said, I will just ignore them. But it also says that "Error detected > during compile or link". > > =========PETSc output while testing======= > huyaoyu at ubuntu:~/Downloads/petsc-3.1-p8$ make > PETSC_DIR=/home/huyaoyu/Downloads/petsc-3.1-p8 > PETSC_ARCH=linux-gnu-c-debug test > Running test examples to verify correct installation > C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 > MPI process > C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 > MPI processes > --------------Error detected during compile or > link!----------------------- > ... OTHER STUFF AND FORTRAN WARNINGS... > Fortran example src/snes/examples/tutorials/ex5f run successfully with 1 > MPI process > Completed test examples > =========End of PETSc output while testing== > > So, I just ignore this. Is that OK? > Yes, it is just indicating that warning. Matt > Thanks! > > HuYaoyu > > > > Thanks to Satish Balay and Jed Brown. Your advices are pretty good! > > > > > > I remove the petsc directory and deal.II directory. Re-build petsc and > > > deal.II with the mpi installed system wide. The configuration lines are > > > as follows: > > > > > > For PETSc: > > > ./config/configure.py --with-cc=/usr/bin/mpicc > --with-fc=/usr/bin/mpif90 > > > --download-f-blas-lapack=1 --with-shared > > > > > > For deal.II: > > > ./configure --enable-shared --disable-threads --with-petsc=$PETSC_DIR > > > --with-petsc-arch=$(PETSC_ARCH) --with-p4est=PATH-TO-P4EST --with-mpi > > > > > > I will use p4est for grid distribution and I checked the configure > > > output of deal.II. The output says deal.II will use /usr/bin/mpicc for > > > CC variable and /usr/bin/mpiCC for CXX variable. And deal.II says: > > > =================deal.II configure output========================= > > > checking for PETSc library > > > directory... /home/huyaoyu/Downloads/petsc-3.1-p8 > > > checking for PETSc version... 3.1.0 > > > checking for PETSc library architecture... linux-gnu-c-debug > > > checking for PETSc libmpiuni library... not found > > > checking for consistency of PETSc and deal.II MPI settings... yes > > > checking for PETSc scalar complex... no > > > ================= end of deal.II configure output================= > > > > > > After the compilation of PETSc and deal.II, I tried to compile the > > > specific example program of deal.II which will use PETSc. And > everything > > > works well. No segmentation error any more! Great! > > > > > > However, the test of PETSc trigger the same error while trying to > > > perform task in 2 process. Is it because I am using PETSc on a single > > > machine but not a cluster of computers? > > > > You mean the compiler warning for the fortan example in 'make test'? > > > > You can ignore that. > > > > Satish > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Debao.Shao at brion.com Fri Aug 5 05:00:07 2011 From: Debao.Shao at brion.com (Debao Shao) Date: Fri, 5 Aug 2011 03:00:07 -0700 Subject: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate In-Reply-To: References: <4E3B2BBA.5000408@geology.wisc.edu> <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A1@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A7@EX03> <8287A816-74DC-4D96-9FAB-123E215D1AA7@mcs.anl.gov> <384FF55F15E3E447802DC8CCA85696980E143123BB@EX03> Message-ID: <384FF55F15E3E447802DC8CCA85696980E14312523@EX03> Thanks a lot, Barry and Matthew. I found the root cause of my problem. 1, I put "PetscInitialize" at the beginning of main, and "PetscInitialize" registers a signal handler to capture "SIGTERM"; 2, the problem ran in server side will send SIGTERM to kill all of clients after one stage is done. 3, then, unfortunately, the signal "SIGTERM" is caught by the signal handler installed by "PetscInitialize", and caused to abort. With changing the position of "PetscInitialize", I fixed the issue. But I encounter a new problem, the situation is: 1, the matrix is big, and can be partitioned to several blocks; 2, started several threads to handle each block of matrix; 3, integrated all block matrices together. Again, the program crashed with reporting: "[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range" My libpetsc.a is built with option "--with-mpi=0 --with-debugging=0 -with-log=0 -with-info=0". Will it support this type of multiple threads? Please give your comments, thanks a lot. Regards, Debao -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith Sent: Friday, August 05, 2011 10:57 AM To: PETSc users list Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate On Aug 4, 2011, at 8:54 PM, Debao Shao wrote: > I get this error after a long process. > For example, I have 4 jobs to do, the program may crash after done 1 job or 2, 3, 4 jobs. Also, sometimes, the program can run finished successfully. The program is not crashing. The batch system is kicking out the job, probably because it thinks it has run out of time. You'll need to set a longer time for the batch system. Barry > > Thanks, > Debao > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith > Sent: Friday, August 05, 2011 10:43 AM > To: PETSc users list > Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate > > > On Aug 4, 2011, at 8:38 PM, Debao Shao wrote: > >> Doesn't SEGV fault is signal 11? > > Yes., 15 is a software terminate single sent from some other process hence the error message: >> Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > > I've seen some batch systems send this single when they think you have run out of time. There may be other reasons as well. Do you get this right away or after a long time? > > Barry > > >> >> From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley >> Sent: Friday, August 05, 2011 10:35 AM >> To: PETSc users list >> Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate >> >> On Fri, Aug 5, 2011 at 2:33 AM, Debao Shao wrote: >> So the real problem is the wrong usage of "PetscInitialize(0, 0, 0, 0 )", but I do have many successive runs before with this setting. >> >> Any kind of memory overwrite gives an SEGV. So other code, making a different error will give that too. >> >> Matt >> >> >> Thanks, >> Debao >> From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley >> Sent: Friday, August 05, 2011 10:25 AM >> To: PETSc users list >> Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate >> >> On Fri, Aug 5, 2011 at 2:20 AM, Debao Shao wrote: >> DA: >> >> Sometimes I may encounter crashing when using PETSc, it doesn't happen always. >> >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC ERROR: or try http://valgrind.org on GNU/linux >> and Apple Mac OS X to find memory corruption errors >> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >> [0]PETSC ERROR: to get more information on the crash. >> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [0]PETSC ERROR: Signal received! >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Unknown Name on a linux-gnu named r0leaf17 by tachyon Thu Aug 4 04:04:19 2011 >> [0]PETSC ERROR: Libraries linked from /home/dshao/dev_t2k/third_party/OOQP/OOQP-0.99.19/extras/petsc-3.1-p8/linux-gnu-c-debug/lib >> [0]PETSC ERROR: Configure run at Thu Apr 21 18:09:19 2011 >> [0]PETSC ERROR: Configure options --with-mpi=0 --with-debugging=0 -with-log=0 -with-info=0 >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >> >> Do you happen to know the possible cause? >> >> I also have a question about the usage of "PetscInitialize" & "PetscFinalize". I can't totally understand them from the manual. >> >> Now, I wrote it as: >> Int main(int argc, char **argv) >> { >> PetscInitialize(0, 0, 0, 0 ); >> >> This is not a valid call. Try >> >> PetscInitialize(&argc, &argv, 0, 0); >> >> Signal 15 is a segmentation fault, which means an illegal memory read or write. >> >> Matt >> >> >> VerifyApp app(argc, argv); >> app.Initialize(); >> app.Run(); >> >> PetscFinalize(); >> return 0; >> } >> >> Then, built a binary, say "verify", and executed "verify" on a server first, then, started another "verify" on client by server. >> The program ran at client side can be multiple processes and multiple threads. >> >> Is my usage correct? >> Your answer is appreciated. >> >> Thanks, >> Debao >> >> -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. > > > -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aron.ahmadia at kaust.edu.sa Fri Aug 5 05:04:23 2011 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Fri, 5 Aug 2011 13:04:23 +0300 Subject: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate In-Reply-To: <384FF55F15E3E447802DC8CCA85696980E14312523@EX03> References: <4E3B2BBA.5000408@geology.wisc.edu> <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A1@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A7@EX03> <8287A816-74DC-4D96-9FAB-123E215D1AA7@mcs.anl.gov> <384FF55F15E3E447802DC8CCA85696980E143123BB@EX03> <384FF55F15E3E447802DC8CCA85696980E14312523@EX03> Message-ID: "But I encounter a new problem, the situation is:**** 1, the matrix is big, and can be partitioned to several blocks;**** 2, started several threads to handle each block of matrix;**** 3, integrated all block matrices together."**** You should be using PETSc+MPI to handle this distribution for you. Threading will not get around any memory limits for a single process that your system may have. A -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 5 08:25:50 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 5 Aug 2011 07:25:50 -0600 Subject: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate In-Reply-To: <384FF55F15E3E447802DC8CCA85696980E14312523@EX03> References: <4E3B2BBA.5000408@geology.wisc.edu> <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A1@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A7@EX03> <8287A816-74DC-4D96-9FAB-123E215D1AA7@mcs.anl.gov> <384FF55F15E3E447802DC8CCA85696980E143123BB@EX03> <384FF55F15E3E447802DC8CCA85696980E14312523@EX03> Message-ID: On Fri, Aug 5, 2011 at 04:00, Debao Shao wrote: > 1, I put **"**PetscInitialize**"** at the beginning of main, and **"** > PetscInitialize**"** registers a signal handler to capture **"**SIGTERM**" > **;**** > > 2, the problem ran in server side will send SIGTERM to kill all of clients > after one stage is done.**** > > 3, then, unfortunately, the signal **"**SIGTERM**"** is caught by the > signal handler installed by **"**PetscInitialize**"**, and caused to > abort. > You can remove the order dependence by running with -no_signal_handler or setting your own signal handler with: http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Sys/PetscPushSignalHandler.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From amesga1 at tigers.lsu.edu Fri Aug 5 11:43:59 2011 From: amesga1 at tigers.lsu.edu (Ataollah Mesgarnejad) Date: Fri, 5 Aug 2011 11:43:59 -0500 Subject: [petsc-users] KSP convergence Message-ID: Dear all, I use KSPSetInitialGuessNonzero for my KSP solver and it works perfectly. But when I restart my computations (I read petsc binaries and set up my fields) the first KSP solve takes forever. I guess my question is: is there a way to set the initial guess for KSP back to where it was before termination (i.e. write the KSP out) or set it to be a predefined vector? Thanks, Ata From bsmith at mcs.anl.gov Fri Aug 5 11:47:55 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 5 Aug 2011 10:47:55 -0600 Subject: [petsc-users] KSP convergence In-Reply-To: References: Message-ID: <9301177B-FC18-49E9-A2B1-751B0333152C@mcs.anl.gov> On Aug 5, 2011, at 10:43 AM, Ataollah Mesgarnejad wrote: > Dear all, > > I use KSPSetInitialGuessNonzero for my KSP solver and it works perfectly. But when I restart my computations (I read petsc binaries and set up my fields) the first KSP solve takes forever. I guess my question is: is there a way to set the initial guess for KSP back to where it was before termination (i.e. write the KSP out) or set it to be a predefined vector? > > Thanks, > Ata There isn't a way automatically with the KSP. But why not save the last solution to the binary file and read it in at restart and put it into x before your first new KSPSolve? Barry From amesga1 at tigers.lsu.edu Fri Aug 5 11:57:42 2011 From: amesga1 at tigers.lsu.edu (Ataollah Mesgarnejad) Date: Fri, 5 Aug 2011 11:57:42 -0500 Subject: [petsc-users] KSP convergence In-Reply-To: <9301177B-FC18-49E9-A2B1-751B0333152C@mcs.anl.gov> References: <9301177B-FC18-49E9-A2B1-751B0333152C@mcs.anl.gov> Message-ID: <1C9ADF13-448F-4C89-AC44-CBA0645E3603@tigers.lsu.edu> On Aug 5, 2011, at 11:47 AM, Barry Smith wrote: > > On Aug 5, 2011, at 10:43 AM, Ataollah Mesgarnejad wrote: > >> Dear all, >> >> I use KSPSetInitialGuessNonzero for my KSP solver and it works perfectly. But when I restart my computations (I read petsc binaries and set up my fields) the first KSP solve takes forever. I guess my question is: is there a way to set the initial guess for KSP back to where it was before termination (i.e. write the KSP out) or set it to be a predefined vector? >> >> Thanks, >> Ata > > There isn't a way automatically with the KSP. But why not save the last solution to the binary file and read it in at restart and put it into x before your first new KSPSolve? > > Barry > Ok so if I get it correctly I need to pass ierr = KSPSolve(ksp,x ,u);CHKERRQ(ierr); and set "x" to be the last thing I had for "u"? Thanks, Ata -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Aug 5 12:05:24 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 5 Aug 2011 11:05:24 -0600 Subject: [petsc-users] KSP convergence In-Reply-To: <1C9ADF13-448F-4C89-AC44-CBA0645E3603@tigers.lsu.edu> References: <9301177B-FC18-49E9-A2B1-751B0333152C@mcs.anl.gov> <1C9ADF13-448F-4C89-AC44-CBA0645E3603@tigers.lsu.edu> Message-ID: KSPSolve(ksp,b,x) solves A x = b. So if you want to start with an initial guess that is not zero you must set into x the initial guess you wish to use (which you could get from a file). Barry On Aug 5, 2011, at 10:57 AM, Ataollah Mesgarnejad wrote: > On Aug 5, 2011, at 11:47 AM, Barry Smith wrote: > >> >> On Aug 5, 2011, at 10:43 AM, Ataollah Mesgarnejad wrote: >> >>> Dear all, >>> >>> I use KSPSetInitialGuessNonzero for my KSP solver and it works perfectly. But when I restart my computations (I read petsc binaries and set up my fields) the first KSP solve takes forever. I guess my question is: is there a way to set the initial guess for KSP back to where it was before termination (i.e. write the KSP out) or set it to be a predefined vector? >>> >>> Thanks, >>> Ata >> >> There isn't a way automatically with the KSP. But why not save the last solution to the binary file and read it in at restart and put it into x before your first new KSPSolve? >> >> Barry >> > > Ok so if I get it correctly I need to pass ierr = KSPSolve(ksp,x ,u);CHKERRQ(ierr); and set "x" to be the last thing I had for "u"? > > Thanks, > Ata From dominik at itis.ethz.ch Fri Aug 5 17:41:09 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Sat, 6 Aug 2011 00:41:09 +0200 Subject: [petsc-users] Fatal error in MPI_Allreduce: Error message texts are not available[cli_9]: aborting job: Message-ID: I have a 2x6core. My solver works fine only on up to 8 processes, above that it always crashes with the below cited error. I did not yet valgrind etc. because I am in a desperate need to fix it quickly. I am just wondering what can potentially be the culprit. PS. I am not using MPI_Allreduce anywhere in my code. Many thanks for any hints, Dominik Fatal error in MPI_Allreduce: Error message texts are not available[cli_9]: aborting job: Fatal error in MPI_Allreduce: Error message texts are not available Fatal error in MPI_Allreduce: Error message texts are not available[cli_1]: aborting job: Fatal error in MPI_Allreduce: Error message texts are not available Fatal error in MPI_Allreduce: Error message texts are not available[cli_7]: aborting job: Fatal error in MPI_Allreduce: Error message texts are not available INTERNAL ERROR: Invalid error class (66) encountered while returning from MPI_Allreduce. Please file a bug report. No error stack is available. Fatal error in MPI_Allreduce: Error message texts are not available[cli_11]: aborting job: Fatal error in MPI_Allreduce: Error message texts are not available [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 3 Quit: Some other process (or the batch system) has told this process to end [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] MatAssemblyBegin_MPIAIJ line 462 src/mat/impls/aij/mpi/mpiaij.c [0]PETSC ERROR: [0] MatAssemblyBegin line 4553 src/mat/interface/matrix.c [0]PETSC ERROR: [0] User provided functi[2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Caught signal number 3 Quit: Some other process (or the batch system) has told this process to end [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [2]PETSC ERROR: likely location of problem given in stack below [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [2]PETSC ERROR: INSTEAD the line number of the start of the function [2]PETSC ERROR: is given. [2]PETSC ERROR: [2] VecAssemblyBegin line 157 src/vec/vec/interface/vector.c [2]PETSC ERROR: [2] User provided function line 160 "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx [INTERNAL ERROR: Invalid error class (66) encountered while returning from MPI_Allreduce. Please file a bug report. No error stack is available. Fatal error in MPI_Allreduce: Error message texts are not available[cli_3]: aborting job: Fatal error in MPI_Allreduce: Error message texts are not available [4]PETSC ERROR: ------------------------------------------------------------------------ [4]PETSC ERROR: Caught signal number 3 Quit: Some other process (or the batch system) has told this process to end [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [4]PETSC ERROR: likely location of problem given in stack below [4]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [4]PETSC ERROR: INSTEAD the line number of the start of the function [4]PETSC ERROR: is given. [4]PETSC ERROR: [4] MatAssemblyBegin_MPIAIJ line 462 src/mat/impls/aij/mpi/mpiaij.c [4]PETSC ERROR: [4] MatAssemblyBegin line 4553 src/mat/interface/matrix.c [4]PETSC ERROR: [4] User provided functiINTERNAL ERROR: Invalid error class (66) encountered while returning from MPI_Allreduce. Please file a bug report. No error stack is available. Fatal error in MPI_Allreduce: Error message texts are not available[cli_5]: aborting job: Fatal error in MPI_Allreduce: Error message texts are not available [6]PETSC ERROR: ------------------------------------------------------------------------ [6]PETSC ERROR: Caught signal number 3 Quit: Some other process (or the batch system) has told this process to end [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [6]PETSC ERROR: likely location of problem given in stack below [6]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [6]PETSC ERROR: INSTEAD the line number of the start of the function [6]PETSC ERROR: is given. [6]PETSC ERROR: [6] MatAssemblyBegin_MPIAIJ line 462 src/mat/impls/aij/mpi/mpiaij.c [6]PETSC ERROR: [6] MatAssemblyBegin line 4553 src/mat/interface/matrix.c [6]PETSC ERROR: [6] User provided functiINTERNAL ERROR: Invalid error class (66) encountered while returning from MPI_Allreduce. Please file a bug report. No error stack is available. Fatal error in MPI_Allreduce: Error message texts are not available[cli_8]: aborting job: Fatal error in MPI_Allreduce: Error message texts are not available [10]PETSC ERROR: ------------------------------------------------------------------------ [10]PETSC ERROR: Caught signal number 3 Quit: Some other process (or the batch system) has told this process to end [10]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [10]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[10]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [10]PETSC ERROR: likely location of problem given in stack below [10]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [10]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [10]PETSC ERROR: INSTEAD the line number of the start of the function [10]PETSC ERROR: is given. [10]PETSC ERROR: [10] VecAssemblyBegin line 157 src/vec/vec/interface/vector.c [10]PETSC ERROR: [10] User provided function line 160 "unknowndirectory/"/home/dsz/src/framework/trunk/solve/on line 294 "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx [0]PETSC ERROR: [0] User provided function line 627 "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug 6 00:35:58 2011 [0]PETSC ERROR: Libraries linked from /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib [0]PETSC ERROR: Configure run at Sat Aug 6 00:02:58 2011 [0]PETSC ERROR: Config2]PETSC ERROR: [2] User provided function line 294 "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx [2]PETSC ERROR: [2] User provided function line 627 "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx [2]PETSC ERROR: --------------------- Error Message ------------------------------------ [2]PETSC ERROR: Signal received! [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [2]PETSC ERROR: See docs/changes/index.html for recent updates. [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [2]PETSC ERROR: See docs/index.html for manual pages. [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug 6 00:35:58 2011 [2]PETSC ERROR: Libraries linked from /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib [2]PETSC ERROR: Configure run at Sat Aug on line 294 "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx [4]PETSC ERROR: [4] User provided function line 627 "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx [4]PETSC ERROR: --------------------- Error Message ------------------------------------ [4]PETSC ERROR: Signal received! [4]PETSC ERROR: ------------------------------------------------------------------------ [4]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [4]PETSC ERROR: See docs/changes/index.html for recent updates. [4]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [4]PETSC ERROR: See docs/index.html for manual pages. [4]PETSC ERROR: ------------------------------------------------------------------------ [4]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug 6 00:35:58 2011 [4]PETSC ERROR: Libraries linked from /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib [4]PETSC ERROR: Configure run at Sat Aug 6 00:02:58 2011 [4]PETSC ERROR: Configon line 294 "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx [6]PETSC ERROR: [6] User provided function line 627 "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx [6]PETSC ERROR: --------------------- Error Message ------------------------------------ [6]PETSC ERROR: Signal received! [6]PETSC ERROR: ------------------------------------------------------------------------ [6]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [6]PETSC ERROR: See docs/changes/index.html for recent updates. [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [6]PETSC ERROR: See docs/index.html for manual pages. [6]PETSC ERROR: ------------------------------------------------------------------------ [6]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug 6 00:35:58 2011 [6]PETSC ERROR: Libraries linked from /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib [6]PETSC ERROR: Configure run at Sat Aug 6 00:02:58 2011 [6]PETSC ERROR: ConfigSM3T4mpi.cxx [10]PETSC ERROR: [10] User provided function line 294 "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx [10]PETSC ERROR: [10] User provided function line 627 "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx [10]PETSC ERROR: --------------------- Error Message ------------------------------------ [10]PETSC ERROR: Signal received! [10]PETSC ERROR: ------------------------------------------------------------------------ [10]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [10]PETSC ERROR: See docs/changes/index.html for recent updates. [10]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [10]PETSC ERROR: See docs/index.html for manual pages. [10]PETSC ERROR: ------------------------------------------------------------------------ [10]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug 6 00:35:58 2011 [10]PETSC ERROR: Libraries linked from /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib [10]PETSC ERRure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0[cli_0]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 6 00:02:58 2011 [2]PETSC ERROR: Configure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: User provided function() line 0 in unknown directory unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2[cli_2]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2 ure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 [4]PETSC ERROR: ------------------------------------------------------------------------ [4]PETSC ERROR: User provided function() line 0 in unknown directory unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4[cli_4]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4 ure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 [6]PETSC ERROR: ------------------------------------------------------------------------ [6]PETSC ERROR: User provided function() line 0 in unknown directory unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 6[cli_6]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 6 OR: Configure run at Sat Aug 6 00:02:58 2011 [10]PETSC ERROR: Configure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 [10]PETSC ERROR: ------------------------------------------------------------------------ [10]PETSC ERROR: User provided function() line 0 in unknown directory unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 10[cli_10]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 10 From knepley at gmail.com Fri Aug 5 17:51:49 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 5 Aug 2011 22:51:49 +0000 Subject: [petsc-users] Fatal error in MPI_Allreduce: Error message texts are not available[cli_9]: aborting job: In-Reply-To: References: Message-ID: On Fri, Aug 5, 2011 at 10:41 PM, Dominik Szczerba wrote: > I have a 2x6core. My solver works fine only on up to 8 processes, > above that it always crashes with the below cited error. I did not yet > valgrind etc. because I am in a desperate need to fix it quickly. I am > just wondering what can potentially be the culprit. > You are getting a SIGQUIT in a function you wrote (if it was a PETSc function it would show up in the stack). It looks like the system might be killing your job. Matt > PS. I am not using MPI_Allreduce anywhere in my code. > > Many thanks for any hints, > Dominik > > Fatal error in MPI_Allreduce: Error message texts are not > available[cli_9]: aborting job: > Fatal error in MPI_Allreduce: Error message texts are not available > Fatal error in MPI_Allreduce: Error message texts are not > available[cli_1]: aborting job: > Fatal error in MPI_Allreduce: Error message texts are not available > Fatal error in MPI_Allreduce: Error message texts are not > available[cli_7]: aborting job: > Fatal error in MPI_Allreduce: Error message texts are not available > INTERNAL ERROR: Invalid error class (66) encountered while returning from > MPI_Allreduce. Please file a bug report. No error stack is available. > Fatal error in MPI_Allreduce: Error message texts are not > available[cli_11]: aborting job: > Fatal error in MPI_Allreduce: Error message texts are not available > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 3 Quit: Some other process (or > the batch system) has told this process to end > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC > ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to > find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] MatAssemblyBegin_MPIAIJ line 462 > src/mat/impls/aij/mpi/mpiaij.c > [0]PETSC ERROR: [0] MatAssemblyBegin line 4553 src/mat/interface/matrix.c > [0]PETSC ERROR: [0] User provided functi[2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: Caught signal number 3 Quit: Some other process (or > the batch system) has told this process to end > [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [2]PETSC ERROR: or see > > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[2]PETSC > ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to > find memory corruption errors > [2]PETSC ERROR: likely location of problem given in stack below > [2]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [2]PETSC ERROR: INSTEAD the line number of the start of the function > [2]PETSC ERROR: is given. > [2]PETSC ERROR: [2] VecAssemblyBegin line 157 > src/vec/vec/interface/vector.c > [2]PETSC ERROR: [2] User provided function line 160 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [INTERNAL ERROR: Invalid error class (66) encountered while returning from > MPI_Allreduce. Please file a bug report. No error stack is available. > Fatal error in MPI_Allreduce: Error message texts are not > available[cli_3]: aborting job: > Fatal error in MPI_Allreduce: Error message texts are not available > [4]PETSC ERROR: > ------------------------------------------------------------------------ > [4]PETSC ERROR: Caught signal number 3 Quit: Some other process (or > the batch system) has told this process to end > [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [4]PETSC ERROR: or see > > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[4]PETSC > ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to > find memory corruption errors > [4]PETSC ERROR: likely location of problem given in stack below > [4]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [4]PETSC ERROR: INSTEAD the line number of the start of the function > [4]PETSC ERROR: is given. > [4]PETSC ERROR: [4] MatAssemblyBegin_MPIAIJ line 462 > src/mat/impls/aij/mpi/mpiaij.c > [4]PETSC ERROR: [4] MatAssemblyBegin line 4553 src/mat/interface/matrix.c > [4]PETSC ERROR: [4] User provided functiINTERNAL ERROR: Invalid error > class (66) encountered while returning from > MPI_Allreduce. Please file a bug report. No error stack is available. > Fatal error in MPI_Allreduce: Error message texts are not > available[cli_5]: aborting job: > Fatal error in MPI_Allreduce: Error message texts are not available > [6]PETSC ERROR: > ------------------------------------------------------------------------ > [6]PETSC ERROR: Caught signal number 3 Quit: Some other process (or > the batch system) has told this process to end > [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [6]PETSC ERROR: or see > > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[6]PETSC > ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to > find memory corruption errors > [6]PETSC ERROR: likely location of problem given in stack below > [6]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [6]PETSC ERROR: INSTEAD the line number of the start of the function > [6]PETSC ERROR: is given. > [6]PETSC ERROR: [6] MatAssemblyBegin_MPIAIJ line 462 > src/mat/impls/aij/mpi/mpiaij.c > [6]PETSC ERROR: [6] MatAssemblyBegin line 4553 src/mat/interface/matrix.c > [6]PETSC ERROR: [6] User provided functiINTERNAL ERROR: Invalid error > class (66) encountered while returning from > MPI_Allreduce. Please file a bug report. No error stack is available. > Fatal error in MPI_Allreduce: Error message texts are not > available[cli_8]: aborting job: > Fatal error in MPI_Allreduce: Error message texts are not available > [10]PETSC ERROR: > ------------------------------------------------------------------------ > [10]PETSC ERROR: Caught signal number 3 Quit: Some other process (or > the batch system) has told this process to end > [10]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [10]PETSC ERROR: or see > > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[10]PETSC > ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to > find memory corruption errors > [10]PETSC ERROR: likely location of problem given in stack below > [10]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [10]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [10]PETSC ERROR: INSTEAD the line number of the start of the function > [10]PETSC ERROR: is given. > [10]PETSC ERROR: [10] VecAssemblyBegin line 157 > src/vec/vec/interface/vector.c > [10]PETSC ERROR: [10] User provided function line 160 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/on line 294 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [0]PETSC ERROR: [0] User provided function line 627 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 > 13:37:48 CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug > 6 00:35:58 2011 > [0]PETSC ERROR: Libraries linked from > /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib > [0]PETSC ERROR: Configure run at Sat Aug 6 00:02:58 2011 > [0]PETSC ERROR: Config2]PETSC ERROR: [2] User provided function line > 294 "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [2]PETSC ERROR: [2] User provided function line 627 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [2]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [2]PETSC ERROR: Signal received! > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 > 13:37:48 CDT 2011 > [2]PETSC ERROR: See docs/changes/index.html for recent updates. > [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [2]PETSC ERROR: See docs/index.html for manual pages. > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug > 6 00:35:58 2011 > [2]PETSC ERROR: Libraries linked from > /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib > [2]PETSC ERROR: Configure run at Sat Aug on line 294 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [4]PETSC ERROR: [4] User provided function line 627 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [4]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [4]PETSC ERROR: Signal received! > [4]PETSC ERROR: > ------------------------------------------------------------------------ > [4]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 > 13:37:48 CDT 2011 > [4]PETSC ERROR: See docs/changes/index.html for recent updates. > [4]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [4]PETSC ERROR: See docs/index.html for manual pages. > [4]PETSC ERROR: > ------------------------------------------------------------------------ > [4]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug > 6 00:35:58 2011 > [4]PETSC ERROR: Libraries linked from > /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib > [4]PETSC ERROR: Configure run at Sat Aug 6 00:02:58 2011 > [4]PETSC ERROR: Configon line 294 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [6]PETSC ERROR: [6] User provided function line 627 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [6]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [6]PETSC ERROR: Signal received! > [6]PETSC ERROR: > ------------------------------------------------------------------------ > [6]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 > 13:37:48 CDT 2011 > [6]PETSC ERROR: See docs/changes/index.html for recent updates. > [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [6]PETSC ERROR: See docs/index.html for manual pages. > [6]PETSC ERROR: > ------------------------------------------------------------------------ > [6]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug > 6 00:35:58 2011 > [6]PETSC ERROR: Libraries linked from > /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib > [6]PETSC ERROR: Configure run at Sat Aug 6 00:02:58 2011 > [6]PETSC ERROR: ConfigSM3T4mpi.cxx > [10]PETSC ERROR: [10] User provided function line 294 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [10]PETSC ERROR: [10] User provided function line 627 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [10]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [10]PETSC ERROR: Signal received! > [10]PETSC ERROR: > ------------------------------------------------------------------------ > [10]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 > 13:37:48 CDT 2011 > [10]PETSC ERROR: See docs/changes/index.html for recent updates. > [10]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [10]PETSC ERROR: See docs/index.html for manual pages. > [10]PETSC ERROR: > ------------------------------------------------------------------------ > [10]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug > 6 00:35:58 2011 > [10]PETSC ERROR: Libraries linked from > /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib > [10]PETSC ERRure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 > PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=1 > --download-mpich=1 --download-hypre=1 --with-parmetis=1 > --download-parmetis=1 --with-x=0 --with-debugging=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0[cli_0]: > aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > 6 00:02:58 2011 > [2]PETSC ERROR: Configure options > PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug > --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 > --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2[cli_2]: > aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2 > ure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 > PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=1 > --download-mpich=1 --download-hypre=1 --with-parmetis=1 > --download-parmetis=1 --with-x=0 --with-debugging=1 > [4]PETSC ERROR: > ------------------------------------------------------------------------ > [4]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4[cli_4]: > aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4 > ure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 > PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=1 > --download-mpich=1 --download-hypre=1 --with-parmetis=1 > --download-parmetis=1 --with-x=0 --with-debugging=1 > [6]PETSC ERROR: > ------------------------------------------------------------------------ > [6]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 6[cli_6]: > aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 6 > OR: Configure run at Sat Aug 6 00:02:58 2011 > [10]PETSC ERROR: Configure options > PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug > --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 > --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 > [10]PETSC ERROR: > ------------------------------------------------------------------------ > [10]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 10[cli_10]: > aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 10 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Aug 5 17:57:33 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 5 Aug 2011 16:57:33 -0600 Subject: [petsc-users] Fatal error in MPI_Allreduce: Error message texts are not available[cli_9]: aborting job: In-Reply-To: References: Message-ID: On Fri, Aug 5, 2011 at 16:41, Dominik Szczerba wrote: > I did not yet > valgrind etc. because I am in a desperate need to fix it quickly. > It's usually harder to fix the problem by guessing than it is if you use tools designed to help you. In this case, I suspect that different processes are calling VecAssemblyBegin() and MatAssemblyBegin() with different arguments or in a different order. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Fri Aug 5 17:58:51 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Sat, 6 Aug 2011 00:58:51 +0200 Subject: [petsc-users] Fatal error in MPI_Allreduce: Error message texts are not available[cli_9]: aborting job: In-Reply-To: References: Message-ID: Why would it happen only for np>8? Dominik On Sat, Aug 6, 2011 at 12:57 AM, Jed Brown wrote: > On Fri, Aug 5, 2011 at 16:41, Dominik Szczerba wrote: >> >> I did not yet >> valgrind etc. because I am in a desperate need to fix it quickly. > > It's usually harder to fix the problem by guessing than it is if you use > tools designed to help you. In this case, I suspect that different processes > are calling VecAssemblyBegin() and MatAssemblyBegin() with different > arguments or in a different order. From jedbrown at mcs.anl.gov Fri Aug 5 18:02:13 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 5 Aug 2011 17:02:13 -0600 Subject: [petsc-users] Fatal error in MPI_Allreduce: Error message texts are not available[cli_9]: aborting job: In-Reply-To: References: Message-ID: On Fri, Aug 5, 2011 at 16:58, Dominik Szczerba wrote: > Why would it happen only for np>8? Depends on the logic in your code. I would start in a debugger with breakpoints on the lines calling MatAssemblyBegin() and VecAssemblyBegin(). You can get line numbers from the stack trace you sent, if you are calling those functions more than once. -------------- next part -------------- An HTML attachment was scrubbed... URL: From amesga1 at tigers.lsu.edu Fri Aug 5 18:11:19 2011 From: amesga1 at tigers.lsu.edu (Ataollah Mesgarnejad) Date: Fri, 5 Aug 2011 18:11:19 -0500 Subject: [petsc-users] Problem with VecLoadIntoVector Message-ID: <9B54B803-1B04-4DF7-B91D-60EA612D2F19@tigers.lsu.edu> Dear all, I'm having a very strange problem: I read my vectors from binary files I wrote with VecView, using VecLoadIntoVector; Everything goes smoothly but then when I try to get the local vector on multiple processors , my code stalls at DAGlobalToLocalEnd! Code does not generate any errors either with debug or optimal version and just stops working. And obviously everything works just fine on one processor. Any ideas on what can be going wrong? Thanks, Ata From knepley at gmail.com Fri Aug 5 18:13:30 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 5 Aug 2011 23:13:30 +0000 Subject: [petsc-users] Problem with VecLoadIntoVector In-Reply-To: <9B54B803-1B04-4DF7-B91D-60EA612D2F19@tigers.lsu.edu> References: <9B54B803-1B04-4DF7-B91D-60EA612D2F19@tigers.lsu.edu> Message-ID: On Fri, Aug 5, 2011 at 11:11 PM, Ataollah Mesgarnejad < amesga1 at tigers.lsu.edu> wrote: > Dear all, > > I'm having a very strange problem: I read my vectors from binary files I > wrote with VecView, using VecLoadIntoVector; Everything goes smoothly but > then when I try to get the local vector on multiple processors , my code > stalls at DAGlobalToLocalEnd! Code does not generate any errors either with > debug or optimal version and just stops working. And obviously everything > works just fine on one processor. Any ideas on what can be going wrong? > I am guessing you have mismatched calls somewhere. Get in the debugger and look at the stack traces on different processors. Matt > Thanks, > Ata -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Debao.Shao at brion.com Fri Aug 5 20:34:18 2011 From: Debao.Shao at brion.com (Debao Shao) Date: Fri, 5 Aug 2011 18:34:18 -0700 Subject: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate In-Reply-To: References: <4E3B2BBA.5000408@geology.wisc.edu> <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A1@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A7@EX03> <8287A816-74DC-4D96-9FAB-123E215D1AA7@mcs.anl.gov> <384FF55F15E3E447802DC8CCA85696980E143123BB@EX03> <384FF55F15E3E447802DC8CCA85696980E14312523@EX03> Message-ID: <384FF55F15E3E447802DC8CCA85696980E143125C6@EX03> Thanks a lot, Jed and Aron. Do you have ideas for this problem? But I encounter a new problem, the situation is: 1, the matrix is big, and can be partitioned to several blocks; 2, started several threads to handle each block of matrix; 3, integrated all block matrices together. Again, the program crashed with reporting: "[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range" My libpetsc.a is built with option "--with-mpi=0 --with-debugging=0 -with-log=0 -with-info=0". Will it support this type of multiple threads? Thanks, Debao ________________________________ From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Friday, August 05, 2011 9:26 PM To: PETSc users list Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate On Fri, Aug 5, 2011 at 04:00, Debao Shao > wrote: 1, I put "PetscInitialize" at the beginning of main, and "PetscInitialize" registers a signal handler to capture "SIGTERM"; 2, the problem ran in server side will send SIGTERM to kill all of clients after one stage is done. 3, then, unfortunately, the signal "SIGTERM" is caught by the signal handler installed by "PetscInitialize", and caused to abort. You can remove the order dependence by running with -no_signal_handler or setting your own signal handler with: http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Sys/PetscPushSignalHandler.html ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Aug 5 21:12:57 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 5 Aug 2011 20:12:57 -0600 Subject: [petsc-users] Fatal error in MPI_Allreduce: Error message texts are not available[cli_9]: aborting job: In-Reply-To: References: Message-ID: <6D16C18B-2744-441C-8706-B95D6AB95C17@mcs.anl.gov> Does the PETSc example src/vec/vec/examples/tutorials/ex1.c run correctly on 8+ processes? Are you sure the MPI shared libraries are the same on both systems? You can try the option -on_error_attach_debugger Barry On Aug 5, 2011, at 4:41 PM, Dominik Szczerba wrote: > I have a 2x6core. My solver works fine only on up to 8 processes, > above that it always crashes with the below cited error. I did not yet > valgrind etc. because I am in a desperate need to fix it quickly. I am > just wondering what can potentially be the culprit. > > PS. I am not using MPI_Allreduce anywhere in my code. > > Many thanks for any hints, > Dominik > > Fatal error in MPI_Allreduce: Error message texts are not > available[cli_9]: aborting job: > Fatal error in MPI_Allreduce: Error message texts are not available > Fatal error in MPI_Allreduce: Error message texts are not > available[cli_1]: aborting job: > Fatal error in MPI_Allreduce: Error message texts are not available > Fatal error in MPI_Allreduce: Error message texts are not > available[cli_7]: aborting job: > Fatal error in MPI_Allreduce: Error message texts are not available > INTERNAL ERROR: Invalid error class (66) encountered while returning from > MPI_Allreduce. Please file a bug report. No error stack is available. > Fatal error in MPI_Allreduce: Error message texts are not > available[cli_11]: aborting job: > Fatal error in MPI_Allreduce: Error message texts are not available > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 3 Quit: Some other process (or > the batch system) has told this process to end > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC > ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to > find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] MatAssemblyBegin_MPIAIJ line 462 > src/mat/impls/aij/mpi/mpiaij.c > [0]PETSC ERROR: [0] MatAssemblyBegin line 4553 src/mat/interface/matrix.c > [0]PETSC ERROR: [0] User provided functi[2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: Caught signal number 3 Quit: Some other process (or > the batch system) has told this process to end > [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [2]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[2]PETSC > ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to > find memory corruption errors > [2]PETSC ERROR: likely location of problem given in stack below > [2]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [2]PETSC ERROR: INSTEAD the line number of the start of the function > [2]PETSC ERROR: is given. > [2]PETSC ERROR: [2] VecAssemblyBegin line 157 src/vec/vec/interface/vector.c > [2]PETSC ERROR: [2] User provided function line 160 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [INTERNAL ERROR: Invalid error class (66) encountered while returning from > MPI_Allreduce. Please file a bug report. No error stack is available. > Fatal error in MPI_Allreduce: Error message texts are not > available[cli_3]: aborting job: > Fatal error in MPI_Allreduce: Error message texts are not available > [4]PETSC ERROR: > ------------------------------------------------------------------------ > [4]PETSC ERROR: Caught signal number 3 Quit: Some other process (or > the batch system) has told this process to end > [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [4]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[4]PETSC > ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to > find memory corruption errors > [4]PETSC ERROR: likely location of problem given in stack below > [4]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [4]PETSC ERROR: INSTEAD the line number of the start of the function > [4]PETSC ERROR: is given. > [4]PETSC ERROR: [4] MatAssemblyBegin_MPIAIJ line 462 > src/mat/impls/aij/mpi/mpiaij.c > [4]PETSC ERROR: [4] MatAssemblyBegin line 4553 src/mat/interface/matrix.c > [4]PETSC ERROR: [4] User provided functiINTERNAL ERROR: Invalid error > class (66) encountered while returning from > MPI_Allreduce. Please file a bug report. No error stack is available. > Fatal error in MPI_Allreduce: Error message texts are not > available[cli_5]: aborting job: > Fatal error in MPI_Allreduce: Error message texts are not available > [6]PETSC ERROR: > ------------------------------------------------------------------------ > [6]PETSC ERROR: Caught signal number 3 Quit: Some other process (or > the batch system) has told this process to end > [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [6]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[6]PETSC > ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to > find memory corruption errors > [6]PETSC ERROR: likely location of problem given in stack below > [6]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [6]PETSC ERROR: INSTEAD the line number of the start of the function > [6]PETSC ERROR: is given. > [6]PETSC ERROR: [6] MatAssemblyBegin_MPIAIJ line 462 > src/mat/impls/aij/mpi/mpiaij.c > [6]PETSC ERROR: [6] MatAssemblyBegin line 4553 src/mat/interface/matrix.c > [6]PETSC ERROR: [6] User provided functiINTERNAL ERROR: Invalid error > class (66) encountered while returning from > MPI_Allreduce. Please file a bug report. No error stack is available. > Fatal error in MPI_Allreduce: Error message texts are not > available[cli_8]: aborting job: > Fatal error in MPI_Allreduce: Error message texts are not available > [10]PETSC ERROR: > ------------------------------------------------------------------------ > [10]PETSC ERROR: Caught signal number 3 Quit: Some other process (or > the batch system) has told this process to end > [10]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [10]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[10]PETSC > ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to > find memory corruption errors > [10]PETSC ERROR: likely location of problem given in stack below > [10]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [10]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [10]PETSC ERROR: INSTEAD the line number of the start of the function > [10]PETSC ERROR: is given. > [10]PETSC ERROR: [10] VecAssemblyBegin line 157 src/vec/vec/interface/vector.c > [10]PETSC ERROR: [10] User provided function line 160 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/on line 294 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [0]PETSC ERROR: [0] User provided function line 627 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 > 13:37:48 CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug > 6 00:35:58 2011 > [0]PETSC ERROR: Libraries linked from > /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib > [0]PETSC ERROR: Configure run at Sat Aug 6 00:02:58 2011 > [0]PETSC ERROR: Config2]PETSC ERROR: [2] User provided function line > 294 "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [2]PETSC ERROR: [2] User provided function line 627 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [2]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [2]PETSC ERROR: Signal received! > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 > 13:37:48 CDT 2011 > [2]PETSC ERROR: See docs/changes/index.html for recent updates. > [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [2]PETSC ERROR: See docs/index.html for manual pages. > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug > 6 00:35:58 2011 > [2]PETSC ERROR: Libraries linked from > /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib > [2]PETSC ERROR: Configure run at Sat Aug on line 294 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [4]PETSC ERROR: [4] User provided function line 627 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [4]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [4]PETSC ERROR: Signal received! > [4]PETSC ERROR: > ------------------------------------------------------------------------ > [4]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 > 13:37:48 CDT 2011 > [4]PETSC ERROR: See docs/changes/index.html for recent updates. > [4]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [4]PETSC ERROR: See docs/index.html for manual pages. > [4]PETSC ERROR: > ------------------------------------------------------------------------ > [4]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug > 6 00:35:58 2011 > [4]PETSC ERROR: Libraries linked from > /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib > [4]PETSC ERROR: Configure run at Sat Aug 6 00:02:58 2011 > [4]PETSC ERROR: Configon line 294 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [6]PETSC ERROR: [6] User provided function line 627 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [6]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [6]PETSC ERROR: Signal received! > [6]PETSC ERROR: > ------------------------------------------------------------------------ > [6]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 > 13:37:48 CDT 2011 > [6]PETSC ERROR: See docs/changes/index.html for recent updates. > [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [6]PETSC ERROR: See docs/index.html for manual pages. > [6]PETSC ERROR: > ------------------------------------------------------------------------ > [6]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug > 6 00:35:58 2011 > [6]PETSC ERROR: Libraries linked from > /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib > [6]PETSC ERROR: Configure run at Sat Aug 6 00:02:58 2011 > [6]PETSC ERROR: ConfigSM3T4mpi.cxx > [10]PETSC ERROR: [10] User provided function line 294 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [10]PETSC ERROR: [10] User provided function line 627 > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx > [10]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [10]PETSC ERROR: Signal received! > [10]PETSC ERROR: > ------------------------------------------------------------------------ > [10]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 > 13:37:48 CDT 2011 > [10]PETSC ERROR: See docs/changes/index.html for recent updates. > [10]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [10]PETSC ERROR: See docs/index.html for manual pages. > [10]PETSC ERROR: > ------------------------------------------------------------------------ > [10]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug > 6 00:35:58 2011 > [10]PETSC ERROR: Libraries linked from > /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib > [10]PETSC ERRure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 > PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=1 > --download-mpich=1 --download-hypre=1 --with-parmetis=1 > --download-parmetis=1 --with-x=0 --with-debugging=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0[cli_0]: > aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > 6 00:02:58 2011 > [2]PETSC ERROR: Configure options > PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug > --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 > --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2[cli_2]: > aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2 > ure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 > PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=1 > --download-mpich=1 --download-hypre=1 --with-parmetis=1 > --download-parmetis=1 --with-x=0 --with-debugging=1 > [4]PETSC ERROR: > ------------------------------------------------------------------------ > [4]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4[cli_4]: > aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4 > ure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 > PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=1 > --download-mpich=1 --download-hypre=1 --with-parmetis=1 > --download-parmetis=1 --with-x=0 --with-debugging=1 > [6]PETSC ERROR: > ------------------------------------------------------------------------ > [6]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 6[cli_6]: > aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 6 > OR: Configure run at Sat Aug 6 00:02:58 2011 > [10]PETSC ERROR: Configure options > PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug > --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 > --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 > [10]PETSC ERROR: > ------------------------------------------------------------------------ > [10]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 10[cli_10]: > aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 10 From knepley at gmail.com Fri Aug 5 22:19:15 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 6 Aug 2011 03:19:15 +0000 Subject: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate In-Reply-To: <384FF55F15E3E447802DC8CCA85696980E143125C6@EX03> References: <4E3B2BBA.5000408@geology.wisc.edu> <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A1@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A7@EX03> <8287A816-74DC-4D96-9FAB-123E215D1AA7@mcs.anl.gov> <384FF55F15E3E447802DC8CCA85696980E143123BB@EX03> <384FF55F15E3E447802DC8CCA85696980E14312523@EX03> <384FF55F15E3E447802DC8CCA85696980E143125C6@EX03> Message-ID: On Sat, Aug 6, 2011 at 1:34 AM, Debao Shao wrote: > ** > > Thanks a lot, Jed and Aron. **** > > ** ** > > Do you have ideas for this problem?**** > > ** ** > > But I encounter a new problem, the situation is:**** > > 1, the matrix is big, and can be partitioned to several blocks;**** > > 2, started several threads to handle each block of matrix;**** > > 3, integrated all block matrices together.**** > > ** ** > > Again, the program crashed with reporting: **** > > **"**[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range**" > Yes, PETSc allows threads, but one of your threads is illegally accessing memory. I suggest using valgrind to track this down. Matt > My libpetsc.a is built with option **"**--with-mpi=0 --with-debugging=0 > -with-log=0 -with-info=0**"**. > > Will it support this type of multiple threads?**** > > ** ** > > Thanks,**** > > **De**bao**** > ------------------------------ > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Jed Brown > *Sent:* Friday, August 05, 2011 9:26 PM > *To:* **PETSc users list** > *Subject:* Re: [petsc-users] PETSC ERROR: Caught signal number 15 > Terminate**** > > ** ** > > On Fri, Aug 5, 2011 at 04:00, **De**bao Shao wrote: > **** > > 1, I put ** "**PetscInitialize**"** at the beginning of main, and **"** > PetscInitialize**"** registers a signal handler to capture **"**SIGTERM**" > **;**** > > 2, the problem ran in server side will send SIGTERM to kill all of clients > after one stage is done.**** > > 3, then, unfortunately, the signal **"**SIGTERM**"** is caught by the > signal handler installed by **"**PetscInitialize**"**, and caused to > abort.**** > > You can remove the order dependence by running with -no_signal_handler or > setting your own signal handler with:**** > > ** ** > > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Sys/PetscPushSignalHandler.html > **** > > ------------------------------ > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Sat Aug 6 00:05:29 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Sat, 6 Aug 2011 07:05:29 +0200 Subject: [petsc-users] Fatal error in MPI_Allreduce: Error message texts are not available[cli_9]: aborting job: In-Reply-To: <6D16C18B-2744-441C-8706-B95D6AB95C17@mcs.anl.gov> References: <6D16C18B-2744-441C-8706-B95D6AB95C17@mcs.anl.gov> Message-ID: On Sat, Aug 6, 2011 at 4:12 AM, Barry Smith wrote: > > ?Does the PETSc example src/vec/vec/examples/tutorials/ex1.c run correctly on 8+ processes? yes, as per: dsz at nexo:~/pack/petsc-3.1-p8/src/vec/vec/examples/tutorials$ ~/pack/petsc-3.1-p8/externalpackages/mpich2-1.0.8/bin/mpiexec -np 12 ./ex1 Vector length 20 Vector length 20 40 60 All other values should be near zero VecScale 0 VecCopy 0 VecAXPY 0 VecAYPX 0 VecSwap 0 VecSwap 0 VecWAXPY 0 VecPointwiseMult 0 VecPointwiseDivide 0 VecMAXPY 0 0 0 > ?Are you sure the MPI shared libraries are the same on both systems? I was not precise, I have only one system consisting of two 6core Intels. 12 cores in total. I have openmpi installed alongside, but was explicitly calling mpiexec from petsc external packages. > ? You can try the option -on_error_attach_debugger When run with np 12 It only opens 6 windows, saying: [9]PETSC ERROR: MPI error 14 [1]PETSC ERROR: MPI error 14 [7]PETSC ERROR: MPI error 14 [9]PETSC ERROR: PETSC: Attaching gdb to /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11798 on display localhost:11.0 on machine nexo [1]PETSC ERROR: PETSC: Attaching gdb to /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11790 on display localhost:11.0 on machine nexo [7]PETSC ERROR: PETSC: Attaching gdb to /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11796 on display localhost:11.0 on machine nexo [9]PETSC ERROR: PetscGatherNumberOfMessages() line 62 in src/sys/utils/mpimesg.c [1]PETSC ERROR: PetscGatherNumberOfMessages() line 62 in src/sys/utils/mpimesg.c [7]PETSC ERROR: PetscGatherNumberOfMessages() line 62 in src/sys/utils/mpimesg.c [1]PETSC ERROR: PETSC: Attaching gdb to /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11790 on display localhost:11.0 on machine nexo [9]PETSC ERROR: PETSC: Attaching gdb to /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11798 on display localhost:11.0 on machine nexo [7]PETSC ERROR: PETSC: Attaching gdb to /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11796 on display localhost:11.0 on machine nexo When now starting the program in the 6 windows with its expected args results in: [cli_9]: PMIU_parse_keyvals: unexpected key delimiter at character 54 in cmd [cli_9]: parse_kevals failed -1 I will not be able to do proper valgrinding/puryfying before next week. In the meantime I will still appreciate any hints. Regards, Dominik > > > ?Barry > > On Aug 5, 2011, at 4:41 PM, Dominik Szczerba wrote: > >> I have a 2x6core. My solver works fine only on up to 8 processes, >> above that it always crashes with the below cited error. I did not yet >> valgrind etc. because I am in a desperate need to fix it quickly. I am >> just wondering what can potentially be the culprit. >> >> PS. I am not using MPI_Allreduce anywhere in my code. >> >> Many thanks for any hints, >> Dominik >> >> Fatal error in MPI_Allreduce: Error message texts are not >> available[cli_9]: aborting job: >> Fatal error in MPI_Allreduce: Error message texts are not available >> Fatal error in MPI_Allreduce: Error message texts are not >> available[cli_1]: aborting job: >> Fatal error in MPI_Allreduce: Error message texts are not available >> Fatal error in MPI_Allreduce: Error message texts are not >> available[cli_7]: aborting job: >> Fatal error in MPI_Allreduce: Error message texts are not available >> INTERNAL ERROR: Invalid error class (66) encountered while returning from >> MPI_Allreduce. ?Please file a bug report. ?No error stack is available. >> Fatal error in MPI_Allreduce: Error message texts are not >> available[cli_11]: aborting job: >> Fatal error in MPI_Allreduce: Error message texts are not available >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Caught signal number 3 Quit: Some other process (or >> the batch system) has told this process to end >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC >> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to >> find memory corruption errors >> [0]PETSC ERROR: likely location of problem given in stack below >> [0]PETSC ERROR: --------------------- ?Stack Frames >> ------------------------------------ >> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> [0]PETSC ERROR: ? ? ? INSTEAD the line number of the start of the function >> [0]PETSC ERROR: ? ? ? is given. >> [0]PETSC ERROR: [0] MatAssemblyBegin_MPIAIJ line 462 >> src/mat/impls/aij/mpi/mpiaij.c >> [0]PETSC ERROR: [0] MatAssemblyBegin line 4553 src/mat/interface/matrix.c >> [0]PETSC ERROR: [0] User provided functi[2]PETSC ERROR: >> ------------------------------------------------------------------------ >> [2]PETSC ERROR: Caught signal number 3 Quit: Some other process (or >> the batch system) has told this process to end >> [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [2]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[2]PETSC >> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to >> find memory corruption errors >> [2]PETSC ERROR: likely location of problem given in stack below >> [2]PETSC ERROR: --------------------- ?Stack Frames >> ------------------------------------ >> [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> [2]PETSC ERROR: ? ? ? INSTEAD the line number of the start of the function >> [2]PETSC ERROR: ? ? ? is given. >> [2]PETSC ERROR: [2] VecAssemblyBegin line 157 src/vec/vec/interface/vector.c >> [2]PETSC ERROR: [2] User provided function line 160 >> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx >> [INTERNAL ERROR: Invalid error class (66) encountered while returning from >> MPI_Allreduce. ?Please file a bug report. ?No error stack is available. >> Fatal error in MPI_Allreduce: Error message texts are not >> available[cli_3]: aborting job: >> Fatal error in MPI_Allreduce: Error message texts are not available >> [4]PETSC ERROR: >> ------------------------------------------------------------------------ >> [4]PETSC ERROR: Caught signal number 3 Quit: Some other process (or >> the batch system) has told this process to end >> [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [4]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[4]PETSC >> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to >> find memory corruption errors >> [4]PETSC ERROR: likely location of problem given in stack below >> [4]PETSC ERROR: --------------------- ?Stack Frames >> ------------------------------------ >> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> [4]PETSC ERROR: ? ? ? INSTEAD the line number of the start of the function >> [4]PETSC ERROR: ? ? ? is given. >> [4]PETSC ERROR: [4] MatAssemblyBegin_MPIAIJ line 462 >> src/mat/impls/aij/mpi/mpiaij.c >> [4]PETSC ERROR: [4] MatAssemblyBegin line 4553 src/mat/interface/matrix.c >> [4]PETSC ERROR: [4] User provided functiINTERNAL ERROR: Invalid error >> class (66) encountered while returning from >> MPI_Allreduce. ?Please file a bug report. ?No error stack is available. >> Fatal error in MPI_Allreduce: Error message texts are not >> available[cli_5]: aborting job: >> Fatal error in MPI_Allreduce: Error message texts are not available >> [6]PETSC ERROR: >> ------------------------------------------------------------------------ >> [6]PETSC ERROR: Caught signal number 3 Quit: Some other process (or >> the batch system) has told this process to end >> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [6]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[6]PETSC >> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to >> find memory corruption errors >> [6]PETSC ERROR: likely location of problem given in stack below >> [6]PETSC ERROR: --------------------- ?Stack Frames >> ------------------------------------ >> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> [6]PETSC ERROR: ? ? ? INSTEAD the line number of the start of the function >> [6]PETSC ERROR: ? ? ? is given. >> [6]PETSC ERROR: [6] MatAssemblyBegin_MPIAIJ line 462 >> src/mat/impls/aij/mpi/mpiaij.c >> [6]PETSC ERROR: [6] MatAssemblyBegin line 4553 src/mat/interface/matrix.c >> [6]PETSC ERROR: [6] User provided functiINTERNAL ERROR: Invalid error >> class (66) encountered while returning from >> MPI_Allreduce. ?Please file a bug report. ?No error stack is available. >> Fatal error in MPI_Allreduce: Error message texts are not >> available[cli_8]: aborting job: >> Fatal error in MPI_Allreduce: Error message texts are not available >> [10]PETSC ERROR: >> ------------------------------------------------------------------------ >> [10]PETSC ERROR: Caught signal number 3 Quit: Some other process (or >> the batch system) has told this process to end >> [10]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [10]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[10]PETSC >> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to >> find memory corruption errors >> [10]PETSC ERROR: likely location of problem given in stack below >> [10]PETSC ERROR: --------------------- ?Stack Frames >> ------------------------------------ >> [10]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> [10]PETSC ERROR: ? ? ? INSTEAD the line number of the start of the function >> [10]PETSC ERROR: ? ? ? is given. >> [10]PETSC ERROR: [10] VecAssemblyBegin line 157 src/vec/vec/interface/vector.c >> [10]PETSC ERROR: [10] User provided function line 160 >> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/on line 294 >> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx >> [0]PETSC ERROR: [0] User provided function line 627 >> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Signal received! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >> 13:37:48 CDT 2011 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug >> 6 00:35:58 2011 >> [0]PETSC ERROR: Libraries linked from >> /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib >> [0]PETSC ERROR: Configure run at Sat Aug ?6 00:02:58 2011 >> [0]PETSC ERROR: Config2]PETSC ERROR: [2] User provided function line >> 294 "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx >> [2]PETSC ERROR: [2] User provided function line 627 >> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx >> [2]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [2]PETSC ERROR: Signal received! >> [2]PETSC ERROR: >> ------------------------------------------------------------------------ >> [2]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >> 13:37:48 CDT 2011 >> [2]PETSC ERROR: See docs/changes/index.html for recent updates. >> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [2]PETSC ERROR: See docs/index.html for manual pages. >> [2]PETSC ERROR: >> ------------------------------------------------------------------------ >> [2]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug >> 6 00:35:58 2011 >> [2]PETSC ERROR: Libraries linked from >> /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib >> [2]PETSC ERROR: Configure run at Sat Aug on line 294 >> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx >> [4]PETSC ERROR: [4] User provided function line 627 >> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx >> [4]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [4]PETSC ERROR: Signal received! >> [4]PETSC ERROR: >> ------------------------------------------------------------------------ >> [4]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >> 13:37:48 CDT 2011 >> [4]PETSC ERROR: See docs/changes/index.html for recent updates. >> [4]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [4]PETSC ERROR: See docs/index.html for manual pages. >> [4]PETSC ERROR: >> ------------------------------------------------------------------------ >> [4]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug >> 6 00:35:58 2011 >> [4]PETSC ERROR: Libraries linked from >> /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib >> [4]PETSC ERROR: Configure run at Sat Aug ?6 00:02:58 2011 >> [4]PETSC ERROR: Configon line 294 >> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx >> [6]PETSC ERROR: [6] User provided function line 627 >> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx >> [6]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [6]PETSC ERROR: Signal received! >> [6]PETSC ERROR: >> ------------------------------------------------------------------------ >> [6]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >> 13:37:48 CDT 2011 >> [6]PETSC ERROR: See docs/changes/index.html for recent updates. >> [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [6]PETSC ERROR: See docs/index.html for manual pages. >> [6]PETSC ERROR: >> ------------------------------------------------------------------------ >> [6]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug >> 6 00:35:58 2011 >> [6]PETSC ERROR: Libraries linked from >> /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib >> [6]PETSC ERROR: Configure run at Sat Aug ?6 00:02:58 2011 >> [6]PETSC ERROR: ConfigSM3T4mpi.cxx >> [10]PETSC ERROR: [10] User provided function line 294 >> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx >> [10]PETSC ERROR: [10] User provided function line 627 >> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx >> [10]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [10]PETSC ERROR: Signal received! >> [10]PETSC ERROR: >> ------------------------------------------------------------------------ >> [10]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >> 13:37:48 CDT 2011 >> [10]PETSC ERROR: See docs/changes/index.html for recent updates. >> [10]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [10]PETSC ERROR: See docs/index.html for manual pages. >> [10]PETSC ERROR: >> ------------------------------------------------------------------------ >> [10]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug >> 6 00:35:58 2011 >> [10]PETSC ERROR: Libraries linked from >> /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib >> [10]PETSC ERRure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 >> PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=1 >> --download-mpich=1 --download-hypre=1 --with-parmetis=1 >> --download-parmetis=1 --with-x=0 --with-debugging=1 >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: User provided function() line 0 in unknown directory >> unknown file >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0[cli_0]: >> aborting job: >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >> 6 00:02:58 2011 >> [2]PETSC ERROR: Configure options >> PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug >> --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 >> --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 >> [2]PETSC ERROR: >> ------------------------------------------------------------------------ >> [2]PETSC ERROR: User provided function() line 0 in unknown directory >> unknown file >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2[cli_2]: >> aborting job: >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2 >> ure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 >> PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=1 >> --download-mpich=1 --download-hypre=1 --with-parmetis=1 >> --download-parmetis=1 --with-x=0 --with-debugging=1 >> [4]PETSC ERROR: >> ------------------------------------------------------------------------ >> [4]PETSC ERROR: User provided function() line 0 in unknown directory >> unknown file >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4[cli_4]: >> aborting job: >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4 >> ure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 >> PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=1 >> --download-mpich=1 --download-hypre=1 --with-parmetis=1 >> --download-parmetis=1 --with-x=0 --with-debugging=1 >> [6]PETSC ERROR: >> ------------------------------------------------------------------------ >> [6]PETSC ERROR: User provided function() line 0 in unknown directory >> unknown file >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 6[cli_6]: >> aborting job: >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 6 >> OR: Configure run at Sat Aug ?6 00:02:58 2011 >> [10]PETSC ERROR: Configure options >> PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug >> --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 >> --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 >> [10]PETSC ERROR: >> ------------------------------------------------------------------------ >> [10]PETSC ERROR: User provided function() line 0 in unknown directory >> unknown file >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 10[cli_10]: >> aborting job: >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 10 > > From dominik at itis.ethz.ch Sat Aug 6 00:06:17 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Sat, 6 Aug 2011 07:06:17 +0200 Subject: [petsc-users] Fatal error in MPI_Allreduce: Error message texts are not available[cli_9]: aborting job: In-Reply-To: References: Message-ID: Debugger does not start (properly), see the other reply. Thanks Dominik On Sat, Aug 6, 2011 at 1:02 AM, Jed Brown wrote: > On Fri, Aug 5, 2011 at 16:58, Dominik Szczerba wrote: >> >> Why would it happen only for np>8? > > Depends on the logic in your code. I would start in a debugger with > breakpoints on the lines calling MatAssemblyBegin() and VecAssemblyBegin(). > You can get line numbers from the stack trace you sent, if you are calling > those functions more than once. From knepley at gmail.com Sat Aug 6 00:12:22 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 6 Aug 2011 05:12:22 +0000 Subject: [petsc-users] Fatal error in MPI_Allreduce: Error message texts are not available[cli_9]: aborting job: In-Reply-To: References: <6D16C18B-2744-441C-8706-B95D6AB95C17@mcs.anl.gov> Message-ID: On Sat, Aug 6, 2011 at 5:05 AM, Dominik Szczerba wrote: > On Sat, Aug 6, 2011 at 4:12 AM, Barry Smith wrote: > > > > Does the PETSc example src/vec/vec/examples/tutorials/ex1.c run > correctly on 8+ processes? > > yes, as per: > > dsz at nexo:~/pack/petsc-3.1-p8/src/vec/vec/examples/tutorials$ > ~/pack/petsc-3.1-p8/externalpackages/mpich2-1.0.8/bin/mpiexec -np 12 > ./ex1 > Vector length 20 > Vector length 20 40 60 > All other values should be near zero > VecScale 0 > VecCopy 0 > VecAXPY 0 > VecAYPX 0 > VecSwap 0 > VecSwap 0 > VecWAXPY 0 > VecPointwiseMult 0 > VecPointwiseDivide 0 > VecMAXPY 0 0 0 > Okay, this tells us you have a problem in your code. > > Are you sure the MPI shared libraries are the same on both systems? > > I was not precise, I have only one system consisting of two 6core > Intels. 12 cores in total. > I have openmpi installed alongside, but was explicitly calling mpiexec > from petsc external packages. > > > You can try the option -on_error_attach_debugger > > When run with np 12 It only opens 6 windows, saying: > > [9]PETSC ERROR: MPI error 14 > This is the error for message truncation on receive. That usually means you used the wrong type in an MPI call. Also, run with -start_in_debugger and get a stack trace when it fails. Matt > [1]PETSC ERROR: MPI error 14 > [7]PETSC ERROR: MPI error 14 > [9]PETSC ERROR: PETSC: Attaching gdb to > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11798 on > display localhost:11.0 on machine nexo > [1]PETSC ERROR: PETSC: Attaching gdb to > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11790 on > display localhost:11.0 on machine nexo > [7]PETSC ERROR: PETSC: Attaching gdb to > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11796 on > display localhost:11.0 on machine nexo > [9]PETSC ERROR: PetscGatherNumberOfMessages() line 62 in > src/sys/utils/mpimesg.c > [1]PETSC ERROR: PetscGatherNumberOfMessages() line 62 in > src/sys/utils/mpimesg.c > [7]PETSC ERROR: PetscGatherNumberOfMessages() line 62 in > src/sys/utils/mpimesg.c > [1]PETSC ERROR: PETSC: Attaching gdb to > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11790 on > display localhost:11.0 on machine nexo > [9]PETSC ERROR: PETSC: Attaching gdb to > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11798 on > display localhost:11.0 on machine nexo > [7]PETSC ERROR: PETSC: Attaching gdb to > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11796 on > display localhost:11.0 on machine nexo > > When now starting the program in the 6 windows with its expected args > results in: > > [cli_9]: PMIU_parse_keyvals: unexpected key delimiter at character 54 in > cmd > [cli_9]: parse_kevals failed -1 > > I will not be able to do proper valgrinding/puryfying before next > week. In the meantime I will still appreciate any hints. > > Regards, > Dominik > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Sat Aug 6 00:58:19 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Sat, 6 Aug 2011 07:58:19 +0200 Subject: [petsc-users] Fatal error in MPI_Allreduce: Error message texts are not available[cli_9]: aborting job: In-Reply-To: References: <6D16C18B-2744-441C-8706-B95D6AB95C17@mcs.anl.gov> Message-ID: >> [9]PETSC ERROR: MPI error 14 > > This is the error for message truncation on receive. That usually means you > used the > wrong type in an MPI call. Thanks, that's already something to start with. > Also, run with -start_in_debugger and get a stack trace when it fails. > ? ?Matt I get: [0]PETSC ERROR: PETSC: Attaching gdb to /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12238 on display localhost:11.0 on machine nexo [1]PETSC ERROR: PETSC: Attaching gdb to /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12239 on display localhost:11.0 on machine nexo [2]PETSC ERROR: PETSC: Attaching gdb to /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12240 on display localhost:11.0 on machine nexo [3]PETSC ERROR: PETSC: Attaching gdb to /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12241 on display localhost:11.0 on machine nexo [4]PETSC ERROR: [5]PETSC ERROR: PETSC: Attaching gdb to /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12243 on display localhost:11.0 on machine nexo PETSC: Attaching gdb to /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12242 on display localhost:11.0 on machine nexo [6]PETSC ERROR: PETSC: Attaching gdb to /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12244 on display localhost:11.0 on machine nexo [7]PETSC ERROR: [8]PETSC ERROR: PETSC: Attaching gdb to /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12246 on display localhost:11.0 on machine nexo PETSC: Attaching gdb to /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12245 on display localhost:11.0 on machine nexo [11]PETSC ERROR: [10]PETSC ERROR: PETSC: Attaching gdb to /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12249 on display localhost:11.0 on machine nexo PETSC: Attaching gdb to /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12248 on display localhost:11.0 on machine nexo [9]PETSC ERROR: PETSC: Attaching gdb to /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12247 on display localhost:11.0 on machine nexo I am indeed remotely logged in, but can open terminal windows etc. Also -on_error_attach_debugger opens the gdb consoles properly. Thanks, Dominik > >> >> [1]PETSC ERROR: MPI error 14 >> [7]PETSC ERROR: MPI error 14 >> [9]PETSC ERROR: PETSC: Attaching gdb to >> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11798 on >> display localhost:11.0 on machine nexo >> [1]PETSC ERROR: PETSC: Attaching gdb to >> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11790 on >> display localhost:11.0 on machine nexo >> [7]PETSC ERROR: PETSC: Attaching gdb to >> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11796 on >> display localhost:11.0 on machine nexo >> [9]PETSC ERROR: PetscGatherNumberOfMessages() line 62 in >> src/sys/utils/mpimesg.c >> [1]PETSC ERROR: PetscGatherNumberOfMessages() line 62 in >> src/sys/utils/mpimesg.c >> [7]PETSC ERROR: PetscGatherNumberOfMessages() line 62 in >> src/sys/utils/mpimesg.c >> [1]PETSC ERROR: PETSC: Attaching gdb to >> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11790 on >> display localhost:11.0 on machine nexo >> [9]PETSC ERROR: PETSC: Attaching gdb to >> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11798 on >> display localhost:11.0 on machine nexo >> [7]PETSC ERROR: PETSC: Attaching gdb to >> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11796 on >> display localhost:11.0 on machine nexo >> >> When now starting the program in the 6 windows with its expected args >> results in: >> >> [cli_9]: PMIU_parse_keyvals: unexpected key delimiter at character 54 in >> cmd >> [cli_9]: parse_kevals failed -1 >> >> I will not be able to do proper valgrinding/puryfying before next >> week. In the meantime I will still appreciate any hints. >> >> Regards, >> Dominik >> > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > From knepley at gmail.com Sat Aug 6 01:02:13 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 6 Aug 2011 06:02:13 +0000 Subject: [petsc-users] Fatal error in MPI_Allreduce: Error message texts are not available[cli_9]: aborting job: In-Reply-To: References: <6D16C18B-2744-441C-8706-B95D6AB95C17@mcs.anl.gov> Message-ID: On Sat, Aug 6, 2011 at 5:58 AM, Dominik Szczerba wrote: > >> [9]PETSC ERROR: MPI error 14 > > > > This is the error for message truncation on receive. That usually means > you > > used the > > wrong type in an MPI call. > > Thanks, that's already something to start with. > > > Also, run with -start_in_debugger and get a stack trace when it fails. > > Matt > > I get: > Your $DISPLAY is wrong. Matt > [0]PETSC ERROR: PETSC: Attaching gdb to > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12238 on > display localhost:11.0 on machine nexo > [1]PETSC ERROR: PETSC: Attaching gdb to > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12239 on > display localhost:11.0 on machine nexo > [2]PETSC ERROR: PETSC: Attaching gdb to > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12240 on > display localhost:11.0 on machine nexo > [3]PETSC ERROR: PETSC: Attaching gdb to > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12241 on > display localhost:11.0 on machine nexo > [4]PETSC ERROR: [5]PETSC ERROR: PETSC: Attaching gdb to > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12243 on > display localhost:11.0 on machine nexo > PETSC: Attaching gdb to > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12242 on > display localhost:11.0 on machine nexo > [6]PETSC ERROR: PETSC: Attaching gdb to > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12244 on > display localhost:11.0 on machine nexo > [7]PETSC ERROR: [8]PETSC ERROR: PETSC: Attaching gdb to > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12246 on > display localhost:11.0 on machine nexo > PETSC: Attaching gdb to > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12245 on > display localhost:11.0 on machine nexo > [11]PETSC ERROR: [10]PETSC ERROR: PETSC: Attaching gdb to > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12249 on > display localhost:11.0 on machine nexo > PETSC: Attaching gdb to > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12248 on > display localhost:11.0 on machine nexo > [9]PETSC ERROR: PETSC: Attaching gdb to > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12247 on > display localhost:11.0 on machine nexo > > I am indeed remotely logged in, but can open terminal windows etc. > Also -on_error_attach_debugger opens the gdb consoles properly. > > Thanks, > Dominik > > > > >> > >> [1]PETSC ERROR: MPI error 14 > >> [7]PETSC ERROR: MPI error 14 > >> [9]PETSC ERROR: PETSC: Attaching gdb to > >> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11798 on > >> display localhost:11.0 on machine nexo > >> [1]PETSC ERROR: PETSC: Attaching gdb to > >> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11790 on > >> display localhost:11.0 on machine nexo > >> [7]PETSC ERROR: PETSC: Attaching gdb to > >> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11796 on > >> display localhost:11.0 on machine nexo > >> [9]PETSC ERROR: PetscGatherNumberOfMessages() line 62 in > >> src/sys/utils/mpimesg.c > >> [1]PETSC ERROR: PetscGatherNumberOfMessages() line 62 in > >> src/sys/utils/mpimesg.c > >> [7]PETSC ERROR: PetscGatherNumberOfMessages() line 62 in > >> src/sys/utils/mpimesg.c > >> [1]PETSC ERROR: PETSC: Attaching gdb to > >> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11790 on > >> display localhost:11.0 on machine nexo > >> [9]PETSC ERROR: PETSC: Attaching gdb to > >> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11798 on > >> display localhost:11.0 on machine nexo > >> [7]PETSC ERROR: PETSC: Attaching gdb to > >> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11796 on > >> display localhost:11.0 on machine nexo > >> > >> When now starting the program in the 6 windows with its expected args > >> results in: > >> > >> [cli_9]: PMIU_parse_keyvals: unexpected key delimiter at character 54 in > >> cmd > >> [cli_9]: parse_kevals failed -1 > >> > >> I will not be able to do proper valgrinding/puryfying before next > >> week. In the meantime I will still appreciate any hints. > >> > >> Regards, > >> Dominik > >> > > -- > > What most experimenters take for granted before they begin their > experiments > > is infinitely more interesting than any results to which their > experiments > > lead. > > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Sat Aug 6 01:19:17 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Sat, 6 Aug 2011 08:19:17 +0200 Subject: [petsc-users] Fatal error in MPI_Allreduce: Error message texts are not available[cli_9]: aborting job: In-Reply-To: References: <6D16C18B-2744-441C-8706-B95D6AB95C17@mcs.anl.gov> Message-ID: >> Also, run with -start_in_debugger and get a stack trace when it fails. >> ? ?Matt > > I get: > > [0]PETSC ERROR: PETSC: Attaching gdb to > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12238 on > display localhost:11.0 on machine nexo OK after long waiting all 12 gdb windows showed up. Starting the app with its args leads to: Reading symbols from /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi...done. Attaching to program: /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi, process 12241 ptrace: No such process. /home/dsz/data/test-solve/SM/box/run3/12241: No such file or directory. (gdb) r run.xml Starting program: /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi run.xml [Thread debugging using libthread_db enabled] [cli_3]: PMIU_parse_keyvals: unexpected key delimiter at character 54 in cmd [cli_3]: parse_kevals failed -1 Any thoughts? Thanks Dominik From knepley at gmail.com Sat Aug 6 01:26:14 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 6 Aug 2011 06:26:14 +0000 Subject: [petsc-users] Fatal error in MPI_Allreduce: Error message texts are not available[cli_9]: aborting job: In-Reply-To: References: <6D16C18B-2744-441C-8706-B95D6AB95C17@mcs.anl.gov> Message-ID: On Sat, Aug 6, 2011 at 6:19 AM, Dominik Szczerba wrote: > >> Also, run with -start_in_debugger and get a stack trace when it fails. > >> Matt > > > > I get: > > > > [0]PETSC ERROR: PETSC: Attaching gdb to > > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12238 on > > display localhost:11.0 on machine nexo > > OK after long waiting all 12 gdb windows showed up. Starting the app > with its args leads to: > > Reading symbols from > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi...done. > Attaching to program: > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi, process 12241 > ptrace: No such process. > /home/dsz/data/test-solve/SM/box/run3/12241: No such file or directory. > (gdb) r run.xml > Starting program: /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi > run.xml > [Thread debugging using libthread_db enabled] > [cli_3]: PMIU_parse_keyvals: unexpected key delimiter at character 54 in > cmd > [cli_3]: parse_kevals failed -1 > > Any thoughts? > Google. That is an MPICH error, so somehow you are providing bad arugments to MPICH. Matt > Thanks > Dominik > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sat Aug 6 09:22:42 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Sat, 6 Aug 2011 09:22:42 -0500 (CDT) Subject: [petsc-users] Fatal error in MPI_Allreduce: Error message texts are not available[cli_9]: aborting job: In-Reply-To: References: <6D16C18B-2744-441C-8706-B95D6AB95C17@mcs.anl.gov> Message-ID: On Sat, 6 Aug 2011, Dominik Szczerba wrote: > >> Also, run with -start_in_debugger and get a stack trace when it fails. > >> ? ?Matt > > > > I get: > > > > [0]PETSC ERROR: PETSC: Attaching gdb to > > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12238 on > > display localhost:11.0 on machine nexo > > OK after long waiting all 12 gdb windows showed up. Starting the app > with its args leads to: > > Reading symbols from /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi...done. > Attaching to program: > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi, process 12241 > ptrace: No such process. > /home/dsz/data/test-solve/SM/box/run3/12241: No such file or directory. > (gdb) r run.xml > Starting program: /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi run.xml > [Thread debugging using libthread_db enabled] > [cli_3]: PMIU_parse_keyvals: unexpected key delimiter at character 54 in cmd > [cli_3]: parse_kevals failed -1 The application is already started with -start_in_debugger. So you don't do 'r run.xml' You place a break point at relavent locations - and then do 'c' for continue (gdb) b PetscError (gdb) c Satish From knepley at gmail.com Sat Aug 6 09:28:34 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 6 Aug 2011 14:28:34 +0000 Subject: [petsc-users] Fatal error in MPI_Allreduce: Error message texts are not available[cli_9]: aborting job: In-Reply-To: References: <6D16C18B-2744-441C-8706-B95D6AB95C17@mcs.anl.gov> Message-ID: On Sat, Aug 6, 2011 at 2:22 PM, Satish Balay wrote: > On Sat, 6 Aug 2011, Dominik Szczerba wrote: > > > >> Also, run with -start_in_debugger and get a stack trace when it fails. > > >> Matt > > > > > > I get: > > > > > > [0]PETSC ERROR: PETSC: Attaching gdb to > > > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12238 on > > > display localhost:11.0 on machine nexo > > > > OK after long waiting all 12 gdb windows showed up. Starting the app > > with its args leads to: > > > > Reading symbols from > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi...done. > > Attaching to program: > > /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi, process 12241 > > ptrace: No such process. > > /home/dsz/data/test-solve/SM/box/run3/12241: No such file or directory. > > (gdb) r run.xml > > Starting program: /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi > run.xml > > [Thread debugging using libthread_db enabled] > > [cli_3]: PMIU_parse_keyvals: unexpected key delimiter at character 54 in > cmd > > [cli_3]: parse_kevals failed -1 > > > The application is already started with -start_in_debugger. So you don't do > 'r run.xml' > > You place a break point at relavent locations - and then do 'c' for > continue > Satish is right. Also, at this point you can set the arguments using set args run.xml Matt > (gdb) b PetscError > (gdb) c > > Satish -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sat Aug 6 09:31:20 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Sat, 6 Aug 2011 09:31:20 -0500 (CDT) Subject: [petsc-users] Fatal error in MPI_Allreduce: Error message texts are not available[cli_9]: aborting job: In-Reply-To: References: <6D16C18B-2744-441C-8706-B95D6AB95C17@mcs.anl.gov> Message-ID: On Sat, 6 Aug 2011, Matthew Knepley wrote: > On Sat, Aug 6, 2011 at 2:22 PM, Satish Balay wrote: > > The application is already started with -start_in_debugger. So you don't do > > 'r run.xml' > > > > You place a break point at relavent locations - and then do 'c' for > > continue > > > > Satish is right. Also, at this point you can set the arguments using > > set args run.xml Actually the program is already started - so this can't be reset. Just set the arguments at start time. mpiexec -n 4 program run.xml -start_in_debugger [or whatever scheme is normally when the code is used without -start_in_debugger] Satish > > > > (gdb) b PetscError > > (gdb) c > > > > Satish > > > > > From gjost at tacc.utexas.edu Sun Aug 7 21:17:26 2011 From: gjost at tacc.utexas.edu (Gabriele Jost) Date: Mon, 8 Aug 2011 02:17:26 +0000 Subject: [petsc-users] Two questions about PC Object Message-ID: Hello! I am working with a user of XSEDE (forerly know as TeraGrid) on converting his code from Scalapack to using PETSc. His physical problem yields matrices which are sparse, but the structure is rather general. We have a version going that works very well for one type of problems. I can use MUMPS or SuperLU and I get results which are pretty close to what he gets when using Scalapack. However, I am running into problems with the matrices that correspond to a different problem type. I get results when using MUMPS, but they differ widely from the Scalapack results. With MUMPS I get an error like: Error reported by MUMPS in numerical factorization phase: INFO(1)=-1, INFO(2)=359 I noticed that for the matrix types where I get good results, the nonzeros reported for the PC Object type exactly the number of entries in my input data for the matrix (MPIAIJ format). For the type that does not work, petsc reports more nonzeros than what I enter. The message looks like this: linear system matrix = precond matrix: Matrix Object: type=mpiaij, rows=123120, cols=123120 total: nonzeros=59094408, allocated nonzeros=59173613 My questions are: 1) Does anybody know what the error INFO(1)=-1 means in MUMPS? 2) How does PETSc calculate the number of nonzeros for the PC matrix? Have banging my head against this problem for a while. Any advice is highly appreciated. Many thanks in advance! Gabriele Jost Texas Advanced Computing Center -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Aug 7 21:22:41 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 7 Aug 2011 21:22:41 -0500 Subject: [petsc-users] [petsc-dev] Two questions about PC Object In-Reply-To: References: Message-ID: <8ABF7D2A-9F93-4E8A-9ADB-2FA064DE5B18@mcs.anl.gov> On Aug 7, 2011, at 9:17 PM, Gabriele Jost wrote: > > > > Hello! > > I am working with a user of XSEDE (forerly know as TeraGrid) on converting his code from Scalapack to using PETSc. His physical problem yields matrices which are sparse, but the structure is rather general. > We have a version going that works very well for one type of problems. I can use MUMPS or SuperLU and I get results which are pretty close to what he gets when using Scalapack. > However, I am running into problems with the matrices that correspond to a different problem type. I get results when using MUMPS, but they differ widely from the Scalapack results. With MUMPS I get an error like: > > Error reported by MUMPS in numerical factorization phase: INFO(1)=-1, INFO(2)=359 > > I noticed that for the matrix types where I get good results, the nonzeros reported for the PC Object type exactly the number of entries in my input data for the matrix (MPIAIJ format). For the type that does not work, petsc reports more nonzeros than what I enter. The message looks like this: > > linear system matrix = precond matrix: > Matrix Object: > type=mpiaij, rows=123120, cols=123120 > total: nonzeros=59094408, allocated nonzeros=59173613 What is the number of entries you think you put in? Is it one of the two numbers above? Have you done matrix preallocation? In the example that works do the two numbers above match, i.e. is the preallocation perfect? My first guess is simply that the matrix entries are not going in the correct places? Anyway to test with a very small matrix and -mat_view? Barry > > My questions are: > > 1) Does anybody know what the error INFO(1)=-1 means in MUMPS? > 2) How does PETSc calculate the number of nonzeros for the PC matrix? > > Have banging my head against this problem for a while. Any advice is highly appreciated. > > Many thanks in advance! > > Gabriele Jost > Texas Advanced Computing Center > > > From knepley at gmail.com Sun Aug 7 21:26:54 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 8 Aug 2011 02:26:54 +0000 Subject: [petsc-users] [petsc-dev] Two questions about PC Object In-Reply-To: References: Message-ID: On Mon, Aug 8, 2011 at 2:17 AM, Gabriele Jost wrote: > Hello! > > I am working with a user of XSEDE (forerly know as TeraGrid) on > converting his code from Scalapack to using PETSc. His physical problem > yields matrices which are sparse, but the structure is rather general. > We have a version going that works very well for one type of problems. I > can use MUMPS or SuperLU and I get results which are pretty close to what he > gets when using Scalapack. > However, I am running into problems with the matrices that correspond to a > different problem type. I get results when using MUMPS, but they differ > widely from the Scalapack results. With MUMPS I get an error like: > > Error reported by MUMPS in numerical factorization phase: INFO(1)=-1, > INFO(2)=359 > > I noticed that for the matrix types where I get good results, the > nonzeros reported for the PC Object type exactly the number of entries in my > input data for the matrix (MPIAIJ format). For the type that does not work, > petsc reports more nonzeros than what I enter. The message looks like this: > > linear system matrix = precond matrix: > Matrix Object: > type=mpiaij, rows=123120, cols=123120 > total: nonzeros=59094408, allocated nonzeros=59173613 > > My questions are: > > 1) Does anybody know what the error INFO(1)=-1 means in MUMPS? > -1 means the error code is in INFO(2). I cannot find any online list of the error codes. I suggest mailing the MUMPS list. Thanks, Matt > 2) How does PETSc calculate the number of nonzeros for the PC matrix? > > Have banging my head against this problem for a while. Any advice is > highly appreciated. > > Many thanks in advance! > > Gabriele Jost > Texas Advanced Computing Center > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From shitij.cse at gmail.com Mon Aug 8 00:29:55 2011 From: shitij.cse at gmail.com (Shitij Bhargava) Date: Mon, 8 Aug 2011 10:59:55 +0530 Subject: [petsc-users] SLEPc eigensolver that uses minimal memory and finds ALL eigenvalues of a real symmetric sparse matrix in reasonable time In-Reply-To: <52C4738E-914B-425C-9A2F-D64F8AB03564@dsic.upv.es> References: <359DE4F9-1B2E-459F-8EF1-4B8E78A16023@dsic.upv.es> <52C4738E-914B-425C-9A2F-D64F8AB03564@dsic.upv.es> Message-ID: Thank you very much for your reply. Unfortunately, I do not have access to the cluster at this moment because there is some problem with it. But I made a seperate program that reads a matrix of size 2500x2500 from a file and calculates eigenvalues.The matrix itself is sparse, symmetric and has non-zero values randomly distributed. I read more about the krylovschur and other solvers, and parallel computing and mpi, and I probably understand what you mean. The eigensolvers are efficient for calculating eigenvalues at the peripheries, and especially quickly (less number of iterations) on the larger side than the smaller side, right ? Although I still dont understand what Maximum Projected Dimension exactly means. I played with mpd values nonetheless, and it doesnt seem to have much effect on memory usage. I am running it on my laptop. But the main problem seems to be that in the program, different threads dont seem to 'cooperate'. Every thread is calculating its own solution and storing its own matrix, as if running its own version of EPSSolve. This is because in the output to top command, each process (I used 2) uses nearly equal amount of memory, which is equal to the memory usage if I run the program with only 1 process. (so I suppose memory distribution is not happening ?) I am using MPIAIJ now. I am putting the code below: (I understand why the program prints things multiple times, but I know I can fix that by something like "if(processID=0) printSomething"....what I am not able to understand is how to distribute memory while solving for eigenvalues, which is the call to EPSSolve. There doesnt seem to be any parameter to tell it to solve while cooperating with other threads) *#include "slepceps.h" #include #include #include #define TOL 0.000001 #define MAXIT 1000 int main(int argc,char**argv) { FILE *fp; double entry; int N; int nzval=0,Istart,Iend; Mat A; PetscErrorCode ierr; double *value,kr,ki; int i,j,nconv,nev,maxit,its; int col_nz;/*number of non-zero columns in a row*/ Vec xr, xi; double tol,error,re,im; EPS eps; /*Eigen solver context*/ const EPSType type; int ncv,mpd; /*PetscInt rows;*/ int* columns; SlepcInitialize(&argc,&argv,(char*)0,PETSC_NULL); /*Read matrix from file*/ fp=fopen("sparseMatrix.txt","r"); fscanf(fp,"%d",&N); fscanf(fp,"%lf",&entry);/*not required, but only to move file pointer forward*/ /*Allocate memory to arrays*/ value=(double*)malloc(N*sizeof(double)); columns=(int*)malloc(N*sizeof(int)); ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N);CHKERRQ(ierr); ierr=MatSetType(A,MATMPIAIJ);CHKERRQ(ierr); //ierr=MatSeqAIJSetPreallocation(A,N,PETSC_NULL);CHKERRQ(ierr); ierr=MatMPIAIJSetPreallocation(A,N,PETSC_NULL,N,PETSC_NULL);CHKERRQ(ierr);//?????!!!!!!!!!!!!!!! ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr); for(i=0;i=Istart) //important, see where interval closed and open { for(j=0;j wrote: > > El 04/08/2011, a las 08:48, Shitij Bhargava escribi?: > > > Thank you very much for your reply. > > > > I am a summer trainee, and I talked to my guide about this, and now what > I want might be a little different. > > > > I really do want all the eigenvalues. There cannot be any compromise on > that. But please note that time is not much of a concern here, the much > bigger concern is of memory. Even if I can distribute the memory somehow > for the computation of all eigenvalues, that is fine. (doesnt matter if it > takes a huge amount of time, he is willing to wait). Can any slepc > Eigensolver distribute memory this way while solving for all eigenvalues ? > > > > I am sorry if what I am asking is obvious, I dont know much about > distributed memory, etc. also. I am stupid at this moment, but I am > learning. > > > > Also, when I ran the slepc program on a small cluster, and then ran the > top command, I saw that while solving for eigenvalues, the program was > already using upto 500% CPU. Does this mean that the eigensolver was > "distributing memory" automatically among different nodes ? I understand > that it must be distributing computational effort, but how do I know if it > was distributing memory also ? It simply showed me the RES memory usage to > be 1.3 gb, and the %MEM to be about 17%. Also, I did not use any MPIXXX data > structures. I simply used SeqAIJ. Will using MPIAIJ "distribute memory" > while solving for all eigenvalues ? (I understand that it will distribute > memory to store the matrix, but the memory bottleneck is eigenvalue solver, > so its more important for memory to be distributed then ) My guide said that > one node will not have enough memory, but all nodes combined will, and hence > the requirement of memory to be distributed. > > > > I read about ScaLAPACK, and I have already asked on their forums to > confirm if it is suitable for me. I am waiting for an answer. > > > > Thank you once again. > > > > Shitij > > Try something like this: > > $ mpirun -np 8 ./program -eps_nev 4800 -eps_mpd 400 -eps_largest_real > $ mpirun -np 8 ./program -eps_nev 4800 -eps_mpd 400 -eps_smallest_real > > But this may not work for the smallest ones. > > Jose > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Aug 8 00:35:51 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 8 Aug 2011 00:35:51 -0500 Subject: [petsc-users] SLEPc eigensolver that uses minimal memory and finds ALL eigenvalues of a real symmetric sparse matrix in reasonable time In-Reply-To: References: <359DE4F9-1B2E-459F-8EF1-4B8E78A16023@dsic.upv.es> <52C4738E-914B-425C-9A2F-D64F8AB03564@dsic.upv.es> Message-ID: On Mon, Aug 8, 2011 at 00:29, Shitij Bhargava wrote: > I* *ran it with: > > mpirun -np 2 ./slepcEigenMPI -eps_monitor > > I didnt do exactly what you said, because the matrix generation part in the > actual program is quite time consuming itself. But I assume what I am doing > is equivalent to what you meant to do? Also, I put MPD as PETSC_DECIDE, > because I didnt know what to put it for this matrix dimension. > > This is the output I get: (part of the output) > *MATRIX ASSMEBLY DONE !!!!!!!! > > MATRIX ASSMEBLY DONE !!!!!!!! > > 1 EPS nconv=98 first unconverged value (error) 1490.88 (1.73958730e-05) > 1 EPS nconv=98 first unconverged value (error) 1490.88 (1.73958730e-05) > 2 EPS nconv=282 first unconverged value (error) 3.04636e-27 > (2.49532175e-04) > 2 EPS nconv=282 first unconverged value (error) 3.04636e-27 > (2.49532175e-04)* The most likely case is that you have more than one MPI implementation installed and that you are running with a different implementation than you built with. Compare the outputs: $ ldd ./slepcEigenMPI $ which mpirun -------------- next part -------------- An HTML attachment was scrubbed... URL: From shitij.cse at gmail.com Mon Aug 8 02:14:39 2011 From: shitij.cse at gmail.com (Shitij Bhargava) Date: Mon, 8 Aug 2011 12:44:39 +0530 Subject: [petsc-users] SLEPc eigensolver that uses minimal memory and finds ALL eigenvalues of a real symmetric sparse matrix in reasonable time In-Reply-To: References: <359DE4F9-1B2E-459F-8EF1-4B8E78A16023@dsic.upv.es> <52C4738E-914B-425C-9A2F-D64F8AB03564@dsic.upv.es> Message-ID: Thank you Jed. That was indeed the problem. I installed a separate MPI for PETSc/SLEPc, but was running my program with a default, already installed one. Now, I have a different question. What I want to do is this: 1. Only 1 process, say root, calculates the matrix in SeqAIJ format 2. Then root creates the EPS context, eps and initializes,sets parameters, problem type,etc. properly 3. After this the root process broadcasts this eps object to other processes 4. I use EPSSolve to solve for eigenvalues (all process together in cooperation resulting in memory distribution) 5. I get the results from root is this possible ? I am not able to broadcast the EPS object, because it is not an MPI_DataType. Is there any PETSc/SLEPc function for this ? I am avoiding using MPIAIJ because that will mean making many changes in the existing code, including the numerous write(*,*) statements (i would have to convert them to PetscPrint in FORTRAN or something like that). So I want a single process to handle matrix generation and assembly, but want to solve the eigenproblem in parallel by different processes. Running the subroutine EPSSolve in parallel and hence distribute memory is the only reason why I want to use MPI. Thanks a lot !! Shitij On 8 August 2011 11:05, Jed Brown wrote: > On Mon, Aug 8, 2011 at 00:29, Shitij Bhargava wrote: > >> I* *ran it with: >> >> mpirun -np 2 ./slepcEigenMPI -eps_monitor >> >> I didnt do exactly what you said, because the matrix generation part in >> the actual program is quite time consuming itself. But I assume what I am >> doing is equivalent to what you meant to do? Also, I put MPD as >> PETSC_DECIDE, because I didnt know what to put it for this matrix dimension. >> >> This is the output I get: (part of the output) >> *MATRIX ASSMEBLY DONE !!!!!!!! >> >> MATRIX ASSMEBLY DONE !!!!!!!! >> >> 1 EPS nconv=98 first unconverged value (error) 1490.88 (1.73958730e-05) >> 1 EPS nconv=98 first unconverged value (error) 1490.88 (1.73958730e-05) >> 2 EPS nconv=282 first unconverged value (error) 3.04636e-27 >> (2.49532175e-04) >> 2 EPS nconv=282 first unconverged value (error) 3.04636e-27 >> (2.49532175e-04)* > > > The most likely case is that you have more than one MPI implementation > installed and that you are running with a different implementation than you > built with. Compare the outputs: > > $ ldd ./slepcEigenMPI > $ which mpirun > -------------- next part -------------- An HTML attachment was scrubbed... URL: From klaus.zimmermann at physik.uni-freiburg.de Mon Aug 8 02:36:21 2011 From: klaus.zimmermann at physik.uni-freiburg.de (Klaus Zimmermann) Date: Mon, 08 Aug 2011 09:36:21 +0200 Subject: [petsc-users] [petsc-dev] Two questions about PC Object In-Reply-To: References: Message-ID: <4E3F91F5.5050208@physik.uni-freiburg.de> Am 8/8/11 4:26 AM, schrieb Matthew Knepley: > On Mon, Aug 8, 2011 at 2:17 AM, Gabriele Jost > wrote: [...] > Error reported by MUMPS in numerical factorization phase: > INFO(1)=-1, INFO(2)=359 > [...] > My questions are: > > 1) Does anybody know what the error INFO(1)=-1 means in MUMPS? > > > -1 means the error code is in INFO(2). I cannot find any online list of > the error codes. I suggest mailing the MUMPS list. This information is available in the mumps userguide (http://graal.ens-lyon.fr/MUMPS/doc/userguide_4.10.0.pdf), Chapter 6. Excerpt: --- INFO(1) is 0 if the call to MUMPS was successful, negative if an error occurred (see Section 7), or positive if a warning is returned. INFO(2) holds additional information about the error or the warning. If INFO(1) = ?1, INFO(2) is the processor number (in communicator mumps par%COMM) on which the error was detected. --- So in your case there would seem to be an error on processor 359. Best, Klaus -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4872 bytes Desc: S/MIME Kryptografische Unterschrift URL: From jroman at dsic.upv.es Mon Aug 8 04:25:54 2011 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 8 Aug 2011 11:25:54 +0200 Subject: [petsc-users] SLEPc eigensolver that uses minimal memory and finds ALL eigenvalues of a real symmetric sparse matrix in reasonable time In-Reply-To: References: <359DE4F9-1B2E-459F-8EF1-4B8E78A16023@dsic.upv.es> <52C4738E-914B-425C-9A2F-D64F8AB03564@dsic.upv.es> Message-ID: <1B162FE1-AB06-448F-B972-D4340B5C1D53@dsic.upv.es> El 08/08/2011, a las 09:14, Shitij Bhargava escribi?: > Thank you Jed. That was indeed the problem. I installed a separate MPI for PETSc/SLEPc, but was running my program with a default, already installed one. > > Now, I have a different question. What I want to do is this: > > 1. Only 1 process, say root, calculates the matrix in SeqAIJ format > 2. Then root creates the EPS context, eps and initializes,sets parameters, problem type,etc. properly > 3. After this the root process broadcasts this eps object to other processes > 4. I use EPSSolve to solve for eigenvalues (all process together in cooperation resulting in memory distribution) > 5. I get the results from root > > is this possible ? I am not able to broadcast the EPS object, because it is not an MPI_DataType. Is there any PETSc/SLEPc function for this ? I am avoiding using MPIAIJ because that will mean making many changes in the existing code, including the numerous write(*,*) statements (i would have to convert them to PetscPrint in FORTRAN or something like that). > So I want a single process to handle matrix generation and assembly, but want to solve the eigenproblem in parallel by different processes. Running the subroutine EPSSolve in parallel and hence distribute memory is the only reason why I want to use MPI. > > Thanks a lot !! > > Shitij No, you cannot use a solver in parallel with a matrix in SeqAIJ format. The matrix must be MPIAIJ. If you want to generate the matrix only in process 0, you can do MPI_Comm_rank(PETSC_COMM_WORLD,&rank) and then enclose the matrix generation (setvalues only) in an if clause: if (!rank) { ierr = MatSetValues(A,...);CHKERRQ(ierr); } ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); However, it is better to do the matrix generation in parallel also. Jose From clemens.domanig at uibk.ac.at Mon Aug 8 08:40:30 2011 From: clemens.domanig at uibk.ac.at (Clemens Domanig) Date: Mon, 08 Aug 2011 15:40:30 +0200 Subject: [petsc-users] (Fast?) assembly of upper triangle and MatZeroRows Message-ID: <4E3FE74E.2080205@uibk.ac.at> Hi, I want my FE-program to be able to use LU- and LDLt-decomp (MUMPS). LU works fine and now I want to implement LDLt. My Questions: 1.) How can I speed up the assembly of the upper triangle matrix? * '-mat_ignore_lower_triangular' makes it terrible slow (1000 elements -> 1s explode to 18s * MatSetValues with Blocks of 6x6 and single values at the diagonal is even worse. 2.) SBAIJ doesn't like MatZeroRows. What are the alternatives to apply boundary conditions to my matrix? Thx for your help - Clemens From hzhang at mcs.anl.gov Mon Aug 8 09:06:55 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Mon, 8 Aug 2011 09:06:55 -0500 Subject: [petsc-users] [petsc-dev] Two questions about PC Object In-Reply-To: References: Message-ID: Gabriele: > However, I am running into problems with the matrices that correspond to a > different problem type. I get results when using MUMPS, but they differ > widely from the Scalapack results. With MUMPS I get an error like: > Error reported by MUMPS in numerical factorization phase: INFO(1)=-1, > INFO(2)=359 Checking mumps manual, "INFO(1)=-1, INFO(2)=359" means an error occurred on processor 359. How many processors do you use, more than 359? > I noticed that for the matrix types where I get good results, the nonzeros > reported for the PC Object type exactly the number of entries in my input > data for the matrix (MPIAIJ format). For the type that does not work, petsc > reports more nonzeros than what I enter. The message looks like this: > ??linear system matrix = precond matrix: > ??Matrix Object: > ?? ?type=mpiaij, rows=123120, cols=123120 > ?? ?total: nonzeros=59094408, allocated nonzeros=59173613 > My questions are: > 1) Does anybody know what the error INFO(1)=-1 means in MUMPS? Check mumps manual ~petsc/externalpackages/MUMPS_xxx/doc/userguide_xxx.pdf > 2) How does PETSc calculate the number of nonzeros for the PC matrix? When using mumps, mumps computes the nonzeros of the PC matrix. Factored matrix is far more denser than original matrix in general. Mumps sets default fill-in ratio as 20, which may still insufficient sometimes. You may experiment with different matrix orderings -mat_mumps_icntl_7 . Are you using petsc-dev with the latest mumps 4.10? If so, you may turn on parallel symbolic factorization '-mat_mumps_icntl_28 2'. Good luck, Hong > Have banging my head against this problem for a while. Any advice is highly > appreciated. > Many thanks in advance! > Gabriele Jost > Texas Advanced Computing Center > > From jedbrown at mcs.anl.gov Mon Aug 8 09:10:32 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 8 Aug 2011 09:10:32 -0500 Subject: [petsc-users] (Fast?) assembly of upper triangle and MatZeroRows In-Reply-To: <4E3FE74E.2080205@uibk.ac.at> References: <4E3FE74E.2080205@uibk.ac.at> Message-ID: On Mon, Aug 8, 2011 at 08:40, Clemens Domanig wrote: > Hi, > > I want my FE-program to be able to use LU- and LDLt-decomp (MUMPS). LU > works fine and now I want to implement LDLt. My Questions: > 1.) How can I speed up the assembly of the upper triangle matrix? > * '-mat_ignore_lower_triangular' makes it terrible slow > (1000 elements -> 1s explode to 18s > * MatSetValues with Blocks of 6x6 and single values at the diagonal is even > worse. > SBAIJ needs different preallocation. If you set it with Mat{Seq,MPI}SBAIJSetPreallocation(), assembly will be fast. > 2.) SBAIJ doesn't like MatZeroRows. What are the alternatives to apply > boundary conditions to my matrix? > That function would make the matrix nonsymmetric because the columns aren't also zeroed, so you couldn't use it with a symmetric solver anyway. I believe in imposing boundary conditions symmetrically at the local level (either by removing them or by imposing them while computing residuals for those nodes), but many people like to do it differently. To do boundary conditions by zeroing rows and columns after assembly, you would need to modify the right hand side in a compatible way. We could automate this by having the Mat "remember" the entries in the columns that it zeroed so that each right hand side could automatically be modified before starting the solve, but then solving A*x = b would either yield an x that doesn't satisfy this equation or modify b, both of which I think would be confusing. Suggestions for imposing boundary conditions globally without ruining symmetry or having confusing semantics would be welcome. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Mon Aug 8 09:24:59 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Mon, 8 Aug 2011 09:24:59 -0500 Subject: [petsc-users] (Fast?) assembly of upper triangle and MatZeroRows In-Reply-To: References: <4E3FE74E.2080205@uibk.ac.at> Message-ID: You can use petsc-dev AIJ matrix format for mumps' LDLt-decomp. See http://www.mcs.anl.gov/petsc/petsc-as/developers/index.html on how to get petsc-dev. Hong On Mon, Aug 8, 2011 at 9:10 AM, Jed Brown wrote: > On Mon, Aug 8, 2011 at 08:40, Clemens Domanig > wrote: >> >> Hi, >> >> I want my FE-program to be able to use LU- and LDLt-decomp (MUMPS). LU >> works fine and now I want to implement LDLt. My Questions: >> 1.) How can I speed up the assembly of the upper triangle matrix? >> * '-mat_ignore_lower_triangular' makes it terrible slow >> (1000 elements -> 1s explode to 18s >> * MatSetValues with Blocks of 6x6 and single values at the diagonal is >> even worse. > > SBAIJ needs different preallocation. If you set it with > Mat{Seq,MPI}SBAIJSetPreallocation(), assembly will be fast. > >> >> 2.) SBAIJ doesn't like MatZeroRows. What are the alternatives to apply >> boundary conditions to my matrix? > > That function would make the matrix nonsymmetric because the columns aren't > also zeroed, so you couldn't use it with a symmetric solver anyway. I > believe in imposing boundary conditions symmetrically at the local level > (either by removing them or by imposing them while computing residuals for > those nodes), but many people like to do it differently. > To do boundary conditions by zeroing rows and columns after assembly, you > would need to modify the right hand side in a compatible way. We could > automate this by having the Mat "remember" the entries in the columns that > it zeroed so that each right hand side could automatically be modified > before starting the solve, but then solving A*x = b would either yield an x > that doesn't satisfy this equation or modify b, both of which I think would > be confusing. > Suggestions for imposing boundary conditions globally without ruining > symmetry or having confusing semantics would be welcome. From gjost at tacc.utexas.edu Sun Aug 7 21:13:02 2011 From: gjost at tacc.utexas.edu (Gabriele Jost) Date: Mon, 8 Aug 2011 02:13:02 +0000 Subject: [petsc-users] Two questions about PC Object Message-ID: Hello! I am working with a user of XSEDE (forerly know as TeraGrid) on converting his code from Scalapack to using PETSc. His physical problem yields matrices which are sparse, but the structure is rather general. We have a version going that works very well for one type of problems. I can use MUMPS or SuperLU and I get results which are pretty close to what he gets when using Scalapack. However, I am running into problems with the matrices that correspond to a different problem type. I get results when using MUMPS, but they differ widely from the Scalapack results. With MUMPS I get an error like: Error reported by MUMPS in numerical factorization phase: INFO(1)=-1, INFO(2)=359 I noticed that for the matrix types where I get good results, the nonzeros reported for the PC Object type exactly the number of entries in my input data for the matrix (MPIAIJ format). For the type that does not work, petsc reports more nonzeros than what I enter. The message looks like this: linear system matrix = precond matrix: Matrix Object: type=mpiaij, rows=123120, cols=123120 total: nonzeros=59094408, allocated nonzeros=59173613 My questions are: 1) Does anybody know what the error INFO(1)=-1 means in MUMPS? 2) How does PETSc calculate the number of nonzeros for the PC matrix? Have banging my head against this problem for a while. Any advice is highly appreciated. Many thanks in advance! Gabriele Jost Texas Advanced Computing Center -------------- next part -------------- An HTML attachment was scrubbed... URL: From sistek at math.cas.cz Mon Aug 8 03:10:55 2011 From: sistek at math.cas.cz (Jakub Sistek) Date: Mon, 08 Aug 2011 10:10:55 +0200 Subject: [petsc-users] [petsc-dev] Two questions about PC Object In-Reply-To: References: Message-ID: <4E3F9A0F.8060409@math.cas.cz> Hello, let me add that MUMPS has a nice manual that describes well the error codes in Section 7. ( http://graal.ens-lyon.fr/MUMPS/doc/userguide_4.10.0.pdf ). > Error reported by MUMPS in numerical factorization phase: INFO(1)=-1, INFO(2)=359 > 1) Does anybody know what the error INFO(1)=-1 means in MUMPS? Regarding INFO(1) = -1 the manual says: "An error occurred on processor INFO(2)" I think that it works the way that all processes has the same INFO(1) and INFO(2) except proc 359, which in fact contain the 'real' error in its INFO(1). According to my experience with MUMPS, if you solve on large number of processors, you might get a bit underestimated sizes of work arrays to allocate. One then needs to relax the estimate by ICNTL(14) parameter which tells in percent how much memory to add to the estimate. This is achieved through PETSc e.g. by command line option -mat_mumps_icntl_14: |-ksp_type preonly -pc_type lu -pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 50| which tells MUMPS to add 50% to the estimate. This might be worth trying, it resolves most of similar problems to me, but I do not know if this is the source of your problems as well. I am not sure if one can get to INFO(1) value on process 359 through PETSc. Best regards, Jakub -- Jakub Sistek, Ph.D. postdoctoral researcher Institute of Mathematics of the AS CR http://www.math.cas.cz/~sistek/ tel: (+420) 222 090 710 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Aug 8 14:23:35 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 8 Aug 2011 14:23:35 -0500 Subject: [petsc-users] (Fast?) assembly of upper triangle and MatZeroRows In-Reply-To: References: <4E3FE74E.2080205@uibk.ac.at> Message-ID: PETSc developers, We have a MatZeroRowsColumns() for MPIAIJ and SeqSBAIJ, what about having one for MPISBAIJ? Anyone want to code it up? Barry On Aug 8, 2011, at 9:10 AM, Jed Brown wrote: > On Mon, Aug 8, 2011 at 08:40, Clemens Domanig wrote: > Hi, > > I want my FE-program to be able to use LU- and LDLt-decomp (MUMPS). LU works fine and now I want to implement LDLt. My Questions: > 1.) How can I speed up the assembly of the upper triangle matrix? > * '-mat_ignore_lower_triangular' makes it terrible slow > (1000 elements -> 1s explode to 18s > * MatSetValues with Blocks of 6x6 and single values at the diagonal is even worse. > > SBAIJ needs different preallocation. If you set it with Mat{Seq,MPI}SBAIJSetPreallocation(), assembly will be fast. > > 2.) SBAIJ doesn't like MatZeroRows. What are the alternatives to apply boundary conditions to my matrix? > > That function would make the matrix nonsymmetric because the columns aren't also zeroed, so you couldn't use it with a symmetric solver anyway. I believe in imposing boundary conditions symmetrically at the local level (either by removing them or by imposing them while computing residuals for those nodes), but many people like to do it differently. > > To do boundary conditions by zeroing rows and columns after assembly, you would need to modify the right hand side in a compatible way. We could automate this by having the Mat "remember" the entries in the columns that it zeroed so that each right hand side could automatically be modified before starting the solve, but then solving A*x = b would either yield an x that doesn't satisfy this equation or modify b, both of which I think would be confusing. > > Suggestions for imposing boundary conditions globally without ruining symmetry or having confusing semantics would be welcome. From bsmith at mcs.anl.gov Mon Aug 8 14:28:20 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 8 Aug 2011 14:28:20 -0500 Subject: [petsc-users] SLEPc eigensolver that uses minimal memory and finds ALL eigenvalues of a real symmetric sparse matrix in reasonable time In-Reply-To: References: <359DE4F9-1B2E-459F-8EF1-4B8E78A16023@dsic.upv.es> <52C4738E-914B-425C-9A2F-D64F8AB03564@dsic.upv.es> Message-ID: <06B0B044-A856-47DE-B317-476210E3A038@mcs.anl.gov> On Aug 8, 2011, at 2:14 AM, Shitij Bhargava wrote: > Thank you Jed. That was indeed the problem. I installed a separate MPI for PETSc/SLEPc, but was running my program with a default, already installed one. > > Now, I have a different question. What I want to do is this: > > 1. Only 1 process, say root, calculates the matrix in SeqAIJ format > 2. Then root creates the EPS context, eps and initializes,sets parameters, problem type,etc. properly > 3. After this the root process broadcasts this eps object to other processes > 4. I use EPSSolve to solve for eigenvalues (all process together in cooperation resulting in memory distribution) > 5. I get the results from root We do have an undocumented routine MatDistribute_MPIAIJ(MPI_Comm comm,Mat gmat,PetscInt m,MatReuse reuse,Mat *inmat) in src/mat/impls/aij/mpi/mpiaij.c that will take a SeqAIJ matrix and distribute it over a larger MPI communicator. Note that you cannot create the EPS context etc on a the root process and then broadcast the object but once the matrix is distributed you can simple create the EPS context etc on the parallel communicator where the matrix is and run with that. Barry > > is this possible ? I am not able to broadcast the EPS object, because it is not an MPI_DataType. Is there any PETSc/SLEPc function for this ? I am avoiding using MPIAIJ because that will mean making many changes in the existing code, including the numerous write(*,*) statements (i would have to convert them to PetscPrint in FORTRAN or something like that). > So I want a single process to handle matrix generation and assembly, but want to solve the eigenproblem in parallel by different processes. Running the subroutine EPSSolve in parallel and hence distribute memory is the only reason why I want to use MPI. > > Thanks a lot !! > > Shitij > > On 8 August 2011 11:05, Jed Brown wrote: > On Mon, Aug 8, 2011 at 00:29, Shitij Bhargava wrote: > I ran it with: > > mpirun -np 2 ./slepcEigenMPI -eps_monitor > > I didnt do exactly what you said, because the matrix generation part in the actual program is quite time consuming itself. But I assume what I am doing is equivalent to what you meant to do? Also, I put MPD as PETSC_DECIDE, because I didnt know what to put it for this matrix dimension. > > This is the output I get: (part of the output) > MATRIX ASSMEBLY DONE !!!!!!!! > > MATRIX ASSMEBLY DONE !!!!!!!! > > 1 EPS nconv=98 first unconverged value (error) 1490.88 (1.73958730e-05) > 1 EPS nconv=98 first unconverged value (error) 1490.88 (1.73958730e-05) > 2 EPS nconv=282 first unconverged value (error) 3.04636e-27 (2.49532175e-04) > 2 EPS nconv=282 first unconverged value (error) 3.04636e-27 (2.49532175e-04) > > The most likely case is that you have more than one MPI implementation installed and that you are running with a different implementation than you built with. Compare the outputs: > > $ ldd ./slepcEigenMPI > $ which mpirun > From jedbrown at mcs.anl.gov Mon Aug 8 14:37:56 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 8 Aug 2011 14:37:56 -0500 Subject: [petsc-users] (Fast?) assembly of upper triangle and MatZeroRows In-Reply-To: References: <4E3FE74E.2080205@uibk.ac.at> Message-ID: On Mon, Aug 8, 2011 at 14:23, Barry Smith wrote: > We have a MatZeroRowsColumns() for MPIAIJ and SeqSBAIJ, what about having > one for MPISBAIJ? Of course this makes sense to have, but I don't think it solves the problem. How do you solve more than one system? How would you use this interface with SNES? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Aug 8 14:44:09 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 8 Aug 2011 14:44:09 -0500 Subject: [petsc-users] (Fast?) assembly of upper triangle and MatZeroRows In-Reply-To: References: <4E3FE74E.2080205@uibk.ac.at> Message-ID: <79D9B108-1BCE-4A54-B5F1-181352A2BFB0@mcs.anl.gov> On Aug 8, 2011, at 2:37 PM, Jed Brown wrote: > On Mon, Aug 8, 2011 at 14:23, Barry Smith wrote: > We have a MatZeroRowsColumns() for MPIAIJ and SeqSBAIJ, what about having one for MPISBAIJ? > > Of course this makes sense to have, but I don't think it solves the problem. How do you solve more than one system? You don't > How would you use this interface with SNES? The user calls this as part the FormJacobian() (in the same way as if they explicitly remove those rows and columns (what Matt likes) in the matrix assembly. Barry From jedbrown at mcs.anl.gov Mon Aug 8 14:54:23 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 8 Aug 2011 14:54:23 -0500 Subject: [petsc-users] (Fast?) assembly of upper triangle and MatZeroRows In-Reply-To: <79D9B108-1BCE-4A54-B5F1-181352A2BFB0@mcs.anl.gov> References: <4E3FE74E.2080205@uibk.ac.at> <79D9B108-1BCE-4A54-B5F1-181352A2BFB0@mcs.anl.gov> Message-ID: On Mon, Aug 8, 2011 at 14:44, Barry Smith wrote: > > How would you use this interface with SNES? > > The user calls this as part the FormJacobian() (in the same way as if > they explicitly remove those rows and columns (what Matt likes) in the > matrix assembly. But they need access to the right hand side at the same time. It seems too fragile to reach into the SNES and modify the residual vector during Jacobian evaluation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Aug 8 17:16:24 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 8 Aug 2011 17:16:24 -0500 Subject: [petsc-users] (Fast?) assembly of upper triangle and MatZeroRows In-Reply-To: References: <4E3FE74E.2080205@uibk.ac.at> <79D9B108-1BCE-4A54-B5F1-181352A2BFB0@mcs.anl.gov> Message-ID: <4C866548-3115-4828-90AA-5B2F4D119011@mcs.anl.gov> On Aug 8, 2011, at 2:54 PM, Jed Brown wrote: > On Mon, Aug 8, 2011 at 14:44, Barry Smith wrote: > > How would you use this interface with SNES? > > The user calls this as part the FormJacobian() (in the same way as if they explicitly remove those rows and columns (what Matt likes) in the matrix assembly. > > But they need access to the right hand side at the same time. It seems too fragile to reach into the SNES and modify the residual vector during Jacobian evaluation. Add a KSP/SNES./TSSetDirichlet(ksp,snes,ts,IS listofnodes,Vec valuesfornodes)? Barry From jedbrown at mcs.anl.gov Mon Aug 8 17:31:57 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 8 Aug 2011 17:31:57 -0500 Subject: [petsc-users] (Fast?) assembly of upper triangle and MatZeroRows In-Reply-To: <4C866548-3115-4828-90AA-5B2F4D119011@mcs.anl.gov> References: <4E3FE74E.2080205@uibk.ac.at> <79D9B108-1BCE-4A54-B5F1-181352A2BFB0@mcs.anl.gov> <4C866548-3115-4828-90AA-5B2F4D119011@mcs.anl.gov> Message-ID: On Mon, Aug 8, 2011 at 17:16, Barry Smith wrote: > Add a KSP/SNES./TSSetDirichlet(ksp,snes,ts,IS listofnodes,Vec > valuesfornodes)? > This might be worthwhile. It needs a new Mat primitive which zeros the rows and columns while also "remembering" the column values that it zeroed (as another assembled matrix). I think it's best to apply the transformation immediately after residuals are calculated so that norms are correct (not in KSP at all). Matrix values are not available at this time so the Dirichlet equations may not be scaled correctly for GMG. We have to be careful that the residual evaluation passed to MatMFFD has the Dirichlet projection inside it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Aug 8 17:38:44 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 8 Aug 2011 22:38:44 +0000 Subject: [petsc-users] (Fast?) assembly of upper triangle and MatZeroRows In-Reply-To: References: <4E3FE74E.2080205@uibk.ac.at> <79D9B108-1BCE-4A54-B5F1-181352A2BFB0@mcs.anl.gov> <4C866548-3115-4828-90AA-5B2F4D119011@mcs.anl.gov> Message-ID: On Mon, Aug 8, 2011 at 10:31 PM, Jed Brown wrote: > On Mon, Aug 8, 2011 at 17:16, Barry Smith wrote: > >> Add a KSP/SNES./TSSetDirichlet(ksp,snes,ts,IS listofnodes,Vec >> valuesfornodes)? >> > > This might be worthwhile. It needs a new Mat primitive which zeros the rows > and columns while also "remembering" the column values that it zeroed (as > another assembled matrix). I think it's best to apply the transformation > immediately after residuals are calculated so that norms are correct (not in > KSP at all). Matrix values are not available at this time so the Dirichlet > equations may not be scaled correctly for GMG. We have to be careful that > the residual evaluation passed to MatMFFD has the Dirichlet projection > inside it. > This is exactly the reason that I abandoned this approach when I got to nonlinear equations. It is complex and fragile, and requires that too many things be coordinated out of sight of the user. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Aug 8 17:39:50 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 8 Aug 2011 17:39:50 -0500 Subject: [petsc-users] (Fast?) assembly of upper triangle and MatZeroRows In-Reply-To: References: <4E3FE74E.2080205@uibk.ac.at> <79D9B108-1BCE-4A54-B5F1-181352A2BFB0@mcs.anl.gov> <4C866548-3115-4828-90AA-5B2F4D119011@mcs.anl.gov> Message-ID: On Mon, Aug 8, 2011 at 17:31, Jed Brown wrote: > It needs a new Mat primitive which zeros the rows and columns while also > "remembering" the column values that it zeroed (as another assembled > matrix). I spoke too soon. If you apply the transformation while evaluating residuals and during assembly, there is no need to have this after-the-fact modification. Is there a reason this information should not be held by the DM? In Dohp, I use explicit projections into homogeneous and inhomogeneous spaces. Both are needed, so it's not clear to me that this can cleanly be placed in SNES or KSP without the weird semantics (modification of force vector or matrix, or having MatMult tell us that we didn't solve the equations. The concept of Dirichlet values (with homogeneous and inhomogeneous spaces) could be pushed all the way down into Mat and Vec (some of which I did for Dohp, in order to use VecGhost). -------------- next part -------------- An HTML attachment was scrubbed... URL: From maxwindiff at gmail.com Mon Aug 8 21:37:11 2011 From: maxwindiff at gmail.com (Max Ng) Date: Tue, 9 Aug 2011 10:37:11 +0800 Subject: [petsc-users] Building Windows DLL Message-ID: Hi, I would like to know is it possible to build PETSc 3.1 on Windows as a DLL (as opposed to static library)? I tried --with-shared=1 but it doesn't do anything. I see there was a Visual Studio 2003 project for building a PETSc 2.3.3 DLL, is there a similar one for PETSc 3.1? Thanks! Max -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Aug 8 21:47:11 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 8 Aug 2011 21:47:11 -0500 (CDT) Subject: [petsc-users] Building Windows DLL In-Reply-To: References: Message-ID: On Tue, 9 Aug 2011, Max Ng wrote: > Hi, > > I would like to know is it possible to build PETSc 3.1 on Windows as a DLL > (as opposed to static library)? I tried --with-shared=1 but it doesn't do > anything. can't build dlls on windows. > I see there was a Visual Studio 2003 project for building a PETSc > 2.3.3 DLL, is there a similar one for PETSc 3.1? well the old project files also didn't build dlls.. Satish From maxwindiff at gmail.com Mon Aug 8 21:59:38 2011 From: maxwindiff at gmail.com (Max Ng) Date: Tue, 9 Aug 2011 10:59:38 +0800 Subject: [petsc-users] Building Windows DLL In-Reply-To: References: Message-ID: Hi, I see, thanks! Max > On Tue, 9 Aug 2011, Max Ng wrote: > > >* Hi,*>* *>* I would like to know is it possible to build PETSc 3.1 on Windows as a DLL*>* (as opposed to static library)? I tried --with-shared=1 but it doesn't do*>* anything.* > can't build dlls on windows. > > > >* I see there was a Visual Studio 2003 project for building a PETSc*>* 2.3.3 DLL, is there a similar one for PETSc 3.1?* > well the old project files also didn't build dlls.. > > Satish > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shitij.cse at gmail.com Tue Aug 9 02:54:11 2011 From: shitij.cse at gmail.com (Shitij Bhargava) Date: Tue, 9 Aug 2011 13:24:11 +0530 Subject: [petsc-users] SLEPc eigensolver that uses minimal memory and finds ALL eigenvalues of a real symmetric sparse matrix in reasonable time In-Reply-To: <06B0B044-A856-47DE-B317-476210E3A038@mcs.anl.gov> References: <359DE4F9-1B2E-459F-8EF1-4B8E78A16023@dsic.upv.es> <52C4738E-914B-425C-9A2F-D64F8AB03564@dsic.upv.es> <06B0B044-A856-47DE-B317-476210E3A038@mcs.anl.gov> Message-ID: Thanks Jose, Barry. I tried what you said, but that gives me an error: *[0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Can only get local values, trying 9!* This is probably because here I am trying to insert all rows of the matrix through process 0, but process 0 doesnt own all the rows. In any case, this seems very "unnatural", so I am using MPIAIJ the right way as you said, where I assemble the MPIAIJ matrix in parallel instead of only on one process. I have done that actually, and am running the code on the cluster right now. Its going to take a long long time to finish, so I cant confirm some of my doubts, which I am asking below: 1. If I run the code with 1 process, and say it takes M memory (peak) while solving for eigenvalues, then when I run it with N processes, each will take nearly M/N memory (peak) (probably a little more) right ? And for doing this, I dont have to use any special MPI stuff....the fact that I am using MPIAIJ, and building the EPS object from it, and then calling EPSSolve() is enough ? I mean EPSSolve() is internally in some way distributing memory and computation effort automatically when I use MPIAIJ, and run the code with many processes, right ? This confusion is there because when I use top, while running the code with 8 processes, each of them showed me nearly 250 mb initially, but each has grown to use 270 mb in about 70 minutes. I understand that the method krylovschur is such that memory requirements increase slowly, but the peak on any process will be less (than if I ran only one process), right ? (Even though their memory requirements are growing, they will grow to some M/N only, right ?) Actually the fact that in this case, each of the process creates its own EPS context, initializes it itself, and then calls EPSSolve() itself without any "interaction" with other processes makes me wonder if they really are working together, or just individually (I would have verified this myself, but the program will take way too much time, and I know I would have to kill it sooner or later).....or the fact that they initialize their own EPS context with THEIR part of the MPI is enough to make them "cooperate and work together" ? (Although I think this is what Barry meant in that last post, but I am not too sure) I am not too comfortable with the MPI way of thinking right now, probably this is why I have this confusion. Anyways, I cant thank you guys enough. I would have been scrounging through documentation again and again to no avail if you guys had not helped me the way you did. The responses were always prompt, always to the point (even though my questions were sometimes not, probably because I didnt completely understand the problems I was facing.....but you always knew what I was asking) and very clear. At this moment, I dont know much about PETSc/SLEPc myself, but I will be sure to contribute back to this list when I do. I have nothing but sincere gratitude for you guys. Thank you very much ! Shitij On 9 August 2011 00:58, Barry Smith wrote: > > On Aug 8, 2011, at 2:14 AM, Shitij Bhargava wrote: > > > Thank you Jed. That was indeed the problem. I installed a separate MPI > for PETSc/SLEPc, but was running my program with a default, already > installed one. > > > > Now, I have a different question. What I want to do is this: > > > > 1. Only 1 process, say root, calculates the matrix in SeqAIJ format > > 2. Then root creates the EPS context, eps and initializes,sets > parameters, problem type,etc. properly > > 3. After this the root process broadcasts this eps object to other > processes > > 4. I use EPSSolve to solve for eigenvalues (all process together in > cooperation resulting in memory distribution) > > 5. I get the results from root > > We do have an undocumented routine MatDistribute_MPIAIJ(MPI_Comm > comm,Mat gmat,PetscInt m,MatReuse reuse,Mat *inmat) in > src/mat/impls/aij/mpi/mpiaij.c that will take a SeqAIJ matrix and distribute > it over a larger MPI communicator. > > Note that you cannot create the EPS context etc on a the root process and > then broadcast the object but once the matrix is distributed you can simple > create the EPS context etc on the parallel communicator where the matrix is > and run with that. > > Barry > > > > > is this possible ? I am not able to broadcast the EPS object, because it > is not an MPI_DataType. Is there any PETSc/SLEPc function for this ? I am > avoiding using MPIAIJ because that will mean making many changes in the > existing code, including the numerous write(*,*) statements (i would have to > convert them to PetscPrint in FORTRAN or something like that). > > So I want a single process to handle matrix generation and assembly, but > want to solve the eigenproblem in parallel by different processes. Running > the subroutine EPSSolve in parallel and hence distribute memory is the only > reason why I want to use MPI. > > > > Thanks a lot !! > > > > Shitij > > > > On 8 August 2011 11:05, Jed Brown wrote: > > On Mon, Aug 8, 2011 at 00:29, Shitij Bhargava > wrote: > > I ran it with: > > > > mpirun -np 2 ./slepcEigenMPI -eps_monitor > > > > I didnt do exactly what you said, because the matrix generation part in > the actual program is quite time consuming itself. But I assume what I am > doing is equivalent to what you meant to do? Also, I put MPD as > PETSC_DECIDE, because I didnt know what to put it for this matrix dimension. > > > > This is the output I get: (part of the output) > > MATRIX ASSMEBLY DONE !!!!!!!! > > > > MATRIX ASSMEBLY DONE !!!!!!!! > > > > 1 EPS nconv=98 first unconverged value (error) 1490.88 (1.73958730e-05) > > 1 EPS nconv=98 first unconverged value (error) 1490.88 (1.73958730e-05) > > 2 EPS nconv=282 first unconverged value (error) 3.04636e-27 > (2.49532175e-04) > > 2 EPS nconv=282 first unconverged value (error) 3.04636e-27 > (2.49532175e-04) > > > > The most likely case is that you have more than one MPI implementation > installed and that you are running with a different implementation than you > built with. Compare the outputs: > > > > $ ldd ./slepcEigenMPI > > $ which mpirun > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Debao.Shao at brion.com Tue Aug 9 03:12:18 2011 From: Debao.Shao at brion.com (Debao Shao) Date: Tue, 9 Aug 2011 01:12:18 -0700 Subject: [petsc-users] PETSC ERROR: PetscFinalize() line 968 in src/sys/objects/pinit.c In-Reply-To: References: Message-ID: <384FF55F15E3E447802DC8CCA85696980E1EA8102C@EX03> DA, Do you happen to know what may cause this error? "[0]PETSC ERROR: PetscFinalize() line 968 in src/sys/objects/pinit.c" After reporting this error, the job can run continued, but it will aborted when encountering next petsc related features even with calling PetscInitialize() again, and say: "Options have not been enabled. You might have forgotten to call PetscInitialize()." Here is line 968: ierr = MPI_Finalize();CHKERRQ(ierr); Thanks, Debao ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aron.ahmadia at kaust.edu.sa Tue Aug 9 03:38:50 2011 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Tue, 9 Aug 2011 11:38:50 +0300 Subject: [petsc-users] PETSC ERROR: PetscFinalize() line 968 in src/sys/objects/pinit.c In-Reply-To: <384FF55F15E3E447802DC8CCA85696980E1EA8102C@EX03> References: <384FF55F15E3E447802DC8CCA85696980E1EA8102C@EX03> Message-ID: Debao, What is the complete traceback for this problem? A On Tue, Aug 9, 2011 at 11:12 AM, Debao Shao wrote: > DA, **** > > ** ** > > Do you happen to know what may cause this error?**** > > ?[0]PETSC ERROR: PetscFinalize() line 968 in > src/sys/objects/pinit.c?**** > > ** ** > > After reporting this error, the job can run continued, but it will aborted > when encountering next petsc related features even with calling > PetscInitialize() again, and say:**** > > ?Options have not been enabled.**** > > You might have forgotten to call PetscInitialize().?**** > > ** ** > > Here is line 968:**** > > ierr = MPI_Finalize();CHKERRQ(ierr);**** > > ** ** > > Thanks,**** > > Debao**** > > ------------------------------ > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Debao.Shao at brion.com Tue Aug 9 03:58:47 2011 From: Debao.Shao at brion.com (Debao Shao) Date: Tue, 9 Aug 2011 01:58:47 -0700 Subject: [petsc-users] PETSC ERROR: PetscFinalize() line 968 in src/sys/objects/pinit.c In-Reply-To: References: <384FF55F15E3E447802DC8CCA85696980E1EA8102C@EX03> Message-ID: <384FF55F15E3E447802DC8CCA85696980E1EA81047@EX03> (gdb) bt #0 0x000000360462e21d in raise () from /lib64/tls/libc.so.6 #1 0x000000360462fa1e in abort () from /lib64/tls/libc.so.6 #2 0x0000000002256329 in Petsc_MPI_Abort () #3 0x00000000023fa16b in PetscOptionsInsert () #4 0x00000000023fbe45 in PetscOptionsFindPair_Private () #5 0x00000000023fd5af in PetscOptionsGetString () #6 0x0000000002339d26 in MatMFFDInitializePackage () #7 0x00000000022a2660 in MatInitializePackage () #8 0x0000000002343c4e in MatCreate () 1, does petsc support multiple threads, if I compiled libpetsc.a with -with-mpi=0? 2, is the following usage right? While(1) { ReceiveJobfromServer(); PetscInitialize(); //to avoid a SIGTERM error, I move the call of PetscInitialize into the loop. handleJob(); PetscFinalize(); } Thanks, Debao ________________________________ From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Aron Ahmadia Sent: Tuesday, August 09, 2011 4:39 PM To: PETSc users list Subject: Re: [petsc-users] PETSC ERROR: PetscFinalize() line 968 in src/sys/objects/pinit.c Debao, What is the complete traceback for this problem? A On Tue, Aug 9, 2011 at 11:12 AM, Debao Shao > wrote: DA, Do you happen to know what may cause this error? "[0]PETSC ERROR: PetscFinalize() line 968 in src/sys/objects/pinit.c" After reporting this error, the job can run continued, but it will aborted when encountering next petsc related features even with calling PetscInitialize() again, and say: "Options have not been enabled. You might have forgotten to call PetscInitialize()." Here is line 968: ierr = MPI_Finalize();CHKERRQ(ierr); Thanks, Debao ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aron.ahmadia at kaust.edu.sa Tue Aug 9 04:10:37 2011 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Tue, 9 Aug 2011 12:10:37 +0300 Subject: [petsc-users] PETSC ERROR: PetscFinalize() line 968 in src/sys/objects/pinit.c In-Reply-To: <384FF55F15E3E447802DC8CCA85696980E1EA81047@EX03> References: <384FF55F15E3E447802DC8CCA85696980E1EA8102C@EX03> <384FF55F15E3E447802DC8CCA85696980E1EA81047@EX03> Message-ID: Hi Debao, 1. see http://www.mcs.anl.gov/petsc/petsc-as/miscellaneous/petscthreads.html 2. There is no support for running PetscInitialize after PetscFinalize has been called. Generally these functions are only called in main just after the program has been started and right before it exits. You need to move PetscInitialize to before the loop and PetscFinalize to after the loop. A On Tue, Aug 9, 2011 at 11:58 AM, Debao Shao wrote: > (gdb) bt**** > > #0 0x000000360462e21d in raise () from /lib64/tls/libc.so.6**** > > #1 0x000000360462fa1e in abort () from /lib64/tls/libc.so.6**** > > #2 0x0000000002256329 in Petsc_MPI_Abort ()**** > > #3 0x00000000023fa16b in PetscOptionsInsert ()**** > > #4 0x00000000023fbe45 in PetscOptionsFindPair_Private ()**** > > #5 0x00000000023fd5af in PetscOptionsGetString ()**** > > #6 0x0000000002339d26 in MatMFFDInitializePackage ()**** > > #7 0x00000000022a2660 in MatInitializePackage ()**** > > #8 0x0000000002343c4e in MatCreate ()**** > > ** ** > > ** ** > > 1, does petsc support multiple threads, if I compiled libpetsc.a with > ?with-mpi=0?**** > > 2, is the following usage right?**** > > While(1)**** > > {**** > > ReceiveJobfromServer();**** > > PetscInitialize(); //to avoid a SIGTERM error, I move the call > of PetscInitialize into the loop.**** > > handleJob();**** > > PetscFinalize();**** > > }**** > > ** ** > > Thanks,**** > > Debao**** > ------------------------------ > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Aron Ahmadia > *Sent:* Tuesday, August 09, 2011 4:39 PM > *To:* PETSc users list > *Subject:* Re: [petsc-users] PETSC ERROR: PetscFinalize() line 968 in > src/sys/objects/pinit.c**** > > ** ** > > Debao,**** > > ** ** > > What is the complete traceback for this problem? **** > > ** ** > > A**** > > On Tue, Aug 9, 2011 at 11:12 AM, Debao Shao wrote:* > *** > > DA, **** > > **** > > Do you happen to know what may cause this error?**** > > ?[0]PETSC ERROR: PetscFinalize() line 968 in > src/sys/objects/pinit.c?**** > > **** > > After reporting this error, the job can run continued, but it will aborted > when encountering next petsc related features even with calling > PetscInitialize() again, and say:**** > > ?Options have not been enabled.**** > > You might have forgotten to call PetscInitialize().?**** > > **** > > Here is line 968:**** > > ierr = MPI_Finalize();CHKERRQ(ierr);**** > > **** > > Thanks,**** > > Debao**** > > ** ** > ------------------------------ > > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt.**** > > ** ** > > ------------------------------ > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Debao.Shao at brion.com Tue Aug 9 04:17:39 2011 From: Debao.Shao at brion.com (Debao Shao) Date: Tue, 9 Aug 2011 02:17:39 -0700 Subject: [petsc-users] PETSC ERROR: PetscFinalize() line 968 in src/sys/objects/pinit.c In-Reply-To: References: <384FF55F15E3E447802DC8CCA85696980E1EA8102C@EX03> <384FF55F15E3E447802DC8CCA85696980E1EA81047@EX03> Message-ID: <384FF55F15E3E447802DC8CCA85696980E1EA81054@EX03> I see. Aron, thanks a lot. ________________________________ From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Aron Ahmadia Sent: Tuesday, August 09, 2011 5:11 PM To: PETSc users list Subject: Re: [petsc-users] PETSC ERROR: PetscFinalize() line 968 in src/sys/objects/pinit.c Hi Debao, 1. see http://www.mcs.anl.gov/petsc/petsc-as/miscellaneous/petscthreads.html 2. There is no support for running PetscInitialize after PetscFinalize has been called. Generally these functions are only called in main just after the program has been started and right before it exits. You need to move PetscInitialize to before the loop and PetscFinalize to after the loop. A On Tue, Aug 9, 2011 at 11:58 AM, Debao Shao > wrote: (gdb) bt #0 0x000000360462e21d in raise () from /lib64/tls/libc.so.6 #1 0x000000360462fa1e in abort () from /lib64/tls/libc.so.6 #2 0x0000000002256329 in Petsc_MPI_Abort () #3 0x00000000023fa16b in PetscOptionsInsert () #4 0x00000000023fbe45 in PetscOptionsFindPair_Private () #5 0x00000000023fd5af in PetscOptionsGetString () #6 0x0000000002339d26 in MatMFFDInitializePackage () #7 0x00000000022a2660 in MatInitializePackage () #8 0x0000000002343c4e in MatCreate () 1, does petsc support multiple threads, if I compiled libpetsc.a with -with-mpi=0? 2, is the following usage right? While(1) { ReceiveJobfromServer(); PetscInitialize(); //to avoid a SIGTERM error, I move the call of PetscInitialize into the loop. handleJob(); PetscFinalize(); } Thanks, Debao ________________________________ From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Aron Ahmadia Sent: Tuesday, August 09, 2011 4:39 PM To: PETSc users list Subject: Re: [petsc-users] PETSC ERROR: PetscFinalize() line 968 in src/sys/objects/pinit.c Debao, What is the complete traceback for this problem? A On Tue, Aug 9, 2011 at 11:12 AM, Debao Shao > wrote: DA, Do you happen to know what may cause this error? "[0]PETSC ERROR: PetscFinalize() line 968 in src/sys/objects/pinit.c" After reporting this error, the job can run continued, but it will aborted when encountering next petsc related features even with calling PetscInitialize() again, and say: "Options have not been enabled. You might have forgotten to call PetscInitialize()." Here is line 968: ierr = MPI_Finalize();CHKERRQ(ierr); Thanks, Debao ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fpacull at fluorem.com Tue Aug 9 06:00:10 2011 From: fpacull at fluorem.com (francois pacull) Date: Tue, 09 Aug 2011 13:00:10 +0200 Subject: [petsc-users] parmetis weights In-Reply-To: <872EB3A7-16AD-49ED-8CFB-6F96A64E543C@mcs.anl.gov> References: <4CA22100.9040103@fluorem.com> <872EB3A7-16AD-49ED-8CFB-6F96A64E543C@mcs.anl.gov> Message-ID: <4E41133A.20908@fluorem.com> Dear All, I once submitted (last september) a question about weights in ParMETIS: after looking at mat/partition/impls/pmetis/pmetis.c, I had the feeling that the "wgtflag" variable was always set to zero and thus that the edge weights were not taken into account by ParMETIS... I eventually tried to use edge weights in order to improve a domain decomposition and indeed, I had to add "wgtflag = 1; " just before the call to ParMetis_V3_PartKway, otherwise the edge weights would always be equal to one (even if the adjacency matrix was weighted). The PETSc version I am using is 3.1-p8. Also, for people that would like to use weights, note that the edge weights need to be //PetscInt values (I had an overflow problem with very large weights, maybe because the "edgecut" variable summing all weights of the boundary is also an int4). Regards, francois. Barry Smith wrote: > It is possible that the code is not tested in that mode. If you determine that something is wrong when you get to this let us know. Perhaps you need to set that flag and see if it works and then tell us what change you made. > > Barry > > > > On Sep 28, 2010, at 12:08 PM, francois pacull wrote: > > >> Dear PETSc team, >> I just have a small and naive question about graph weights in Parmetis. When looking at mat/partition/impls/pmetis/pmetis.c, it seems that weight arrays (for vertices or edges) can be given to parmetis, however the wgtflag variable is always set to 0 from what I understand, which means that no weights are used in ParMETIS_V3_PartKway. So does this mean that the parmetis routine uses the weight arrays whenever they are not NULL, even if wgtflag is 0? I didn't try to use any weight yet, but will probably try it soon... >> Thank you for any help you can provide, >> francois pacull. >> >> >> > > > From clemens.domanig at uibk.ac.at Tue Aug 9 08:43:35 2011 From: clemens.domanig at uibk.ac.at (Clemens Domanig) Date: Tue, 09 Aug 2011 15:43:35 +0200 Subject: [petsc-users] MatGetDiagonal Message-ID: <4E413987.5090208@uibk.ac.at> Hi, I already talked to some of you about this in combination with some other problem, but somehow I don't get it. I want the diagonal entries of the matrix M. But my vector 'diag' is mainly full of zeros when using MatGetDiagonal(). I can print 'DiagM' (VecView) where I see all the diagonal entries but I cannot access them using MatGetValue (Object is in wrong state. Not for factored matrix!). How can I get access to the diagonal? Thx - Clemens KSPCreate( coml, &kspBA); KSPGetPC( kspBA, &precond); PCSetType( precond, PCCHOLESKY); PCFactorSetMatSolverPackage( precond, MAT_SOLVER_PETSC); KSPSetOptionsPrefix( kspBA, "diag_pc_type cholesky"); KSPSetFromOptions( kspBA); KSPSetOperators( kspBA, Kt, Kt, DIFFERENT_NONZERO_PATTERN); KSPSetUp( kspBA); PCFactorGetMatrix( precond, &DiagM); //DO NOT DESTROY DiagM !!! MatGetDiagonal( DiagM, diag); From hzhang at mcs.anl.gov Tue Aug 9 09:09:32 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 9 Aug 2011 09:09:32 -0500 Subject: [petsc-users] MatGetDiagonal In-Reply-To: <4E413987.5090208@uibk.ac.at> References: <4E413987.5090208@uibk.ac.at> Message-ID: Clemens : I can repeat your problem, could be a bug in petsc. I'm investigating it and will get back to you soon. Hong > Hi, > > I already talked to some of you about this in combination with some other > problem, but somehow I don't get it. > I want the diagonal entries of the matrix M. But my vector 'diag' is mainly > full of zeros when using MatGetDiagonal(). > I can print 'DiagM' (VecView) where I see all the diagonal entries but I > cannot access them using MatGetValue (Object is in wrong state. Not for > factored matrix!). > How can I get access to the diagonal? > Thx - Clemens > > > KSPCreate( coml, &kspBA); > KSPGetPC( kspBA, &precond); > PCSetType( precond, PCCHOLESKY); > PCFactorSetMatSolverPackage( precond, MAT_SOLVER_PETSC); > KSPSetOptionsPrefix( kspBA, "diag_pc_type cholesky"); > KSPSetFromOptions( kspBA); > KSPSetOperators( kspBA, Kt, Kt, DIFFERENT_NONZERO_PATTERN); > KSPSetUp( kspBA); > PCFactorGetMatrix( precond, &DiagM); ? ?//DO NOT DESTROY DiagM !!! > MatGetDiagonal( DiagM, diag); > From jroman at dsic.upv.es Tue Aug 9 10:27:30 2011 From: jroman at dsic.upv.es (Jose E. Roman) Date: Tue, 9 Aug 2011 17:27:30 +0200 Subject: [petsc-users] SLEPc eigensolver that uses minimal memory and finds ALL eigenvalues of a real symmetric sparse matrix in reasonable time In-Reply-To: References: <359DE4F9-1B2E-459F-8EF1-4B8E78A16023@dsic.upv.es> <52C4738E-914B-425C-9A2F-D64F8AB03564@dsic.upv.es> <06B0B044-A856-47DE-B317-476210E3A038@mcs.anl.gov> Message-ID: <845B370A-4DC5-4789-8BA2-BE7BBD4AB88A@dsic.upv.es> El 09/08/2011, a las 09:54, Shitij Bhargava escribi?: > Thanks Jose, Barry. > > I tried what you said, but that gives me an error: > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Argument out of range! > [0]PETSC ERROR: Can only get local values, trying 9! > > This is probably because here I am trying to insert all rows of the matrix through process 0, but process 0 doesnt own all the rows. > > In any case, this seems very "unnatural", so I am using MPIAIJ the right way as you said, where I assemble the MPIAIJ matrix in parallel instead of only on one process. I have done that actually, and am running the code on the cluster right now. Its going to take a long long time to finish, so I cant confirm some of my doubts, which I am asking below: > > 1. If I run the code with 1 process, and say it takes M memory (peak) while solving for eigenvalues, then when I run it with N processes, each will take nearly M/N memory (peak) (probably a little more) right ? And for doing this, I dont have to use any special MPI stuff....the fact that I am using MPIAIJ, and building the EPS object from it, and then calling EPSSolve() is enough ? I mean EPSSolve() is internally in some way distributing memory and computation effort automatically when I use MPIAIJ, and run the code with many processes, right ? > This confusion is there because when I use top, while running the code with 8 processes, each of them showed me nearly 250 mb initially, but each has grown to use 270 mb in about 70 minutes. I understand that the method krylovschur is such that memory requirements increase slowly, but the peak on any process will be less (than if I ran only one process), right ? (Even though their memory requirements are growing, they will grow to some M/N only, right ?) The solver allocates some dynamic memory when the actual computation starts, so it is normal that you see a growth in the memory footprint. No further increase should be observed afterwards. Jose > > Actually the fact that in this case, each of the process creates its own EPS context, initializes it itself, and then calls EPSSolve() itself without any "interaction" with other processes makes me wonder if they really are working together, or just individually (I would have verified this myself, but the program will take way too much time, and I know I would have to kill it sooner or later).....or the fact that they initialize their own EPS context with THEIR part of the MPI is enough to make them "cooperate and work together" ? (Although I think this is what Barry meant in that last post, but I am not too sure) > > I am not too comfortable with the MPI way of thinking right now, probably this is why I have this confusion. > > Anyways, I cant thank you guys enough. I would have been scrounging through documentation again and again to no avail if you guys had not helped me the way you did. The responses were always prompt, always to the point (even though my questions were sometimes not, probably because I didnt completely understand the problems I was facing.....but you always knew what I was asking) and very clear. At this moment, I dont know much about PETSc/SLEPc myself, but I will be sure to contribute back to this list when I do. I have nothing but sincere gratitude for you guys. > > > Thank you very much ! > > Shitij From hzhang at mcs.anl.gov Tue Aug 9 14:04:49 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 9 Aug 2011 14:04:49 -0500 Subject: [petsc-users] MatGetDiagonal In-Reply-To: <4E413987.5090208@uibk.ac.at> References: <4E413987.5090208@uibk.ac.at> Message-ID: Clemens : This is a bug in MatGetDiagonal_SeqSBAIJ() for cholesky factored matrix. I fixed it and pushed to petsc-dev. See http://www.mcs.anl.gov/petsc/petsc-as/developers/index.html on how to get petsc-dev. Thanks for reporting the problem. Contact us if you still get problem. Hong > Hi, > > I already talked to some of you about this in combination with some other > problem, but somehow I don't get it. > I want the diagonal entries of the matrix M. But my vector 'diag' is mainly > full of zeros when using MatGetDiagonal(). > I can print 'DiagM' (VecView) where I see all the diagonal entries but I > cannot access them using MatGetValue (Object is in wrong state. Not for > factored matrix!). > How can I get access to the diagonal? > Thx - Clemens > > > KSPCreate( coml, &kspBA); > KSPGetPC( kspBA, &precond); > PCSetType( precond, PCCHOLESKY); > PCFactorSetMatSolverPackage( precond, MAT_SOLVER_PETSC); > KSPSetOptionsPrefix( kspBA, "diag_pc_type cholesky"); > KSPSetFromOptions( kspBA); > KSPSetOperators( kspBA, Kt, Kt, DIFFERENT_NONZERO_PATTERN); > KSPSetUp( kspBA); > PCFactorGetMatrix( precond, &DiagM); ? ?//DO NOT DESTROY DiagM !!! > MatGetDiagonal( DiagM, diag); > From bsmith at mcs.anl.gov Tue Aug 9 15:32:33 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 9 Aug 2011 15:32:33 -0500 Subject: [petsc-users] SLEPc eigensolver that uses minimal memory and finds ALL eigenvalues of a real symmetric sparse matrix in reasonable time In-Reply-To: References: <359DE4F9-1B2E-459F-8EF1-4B8E78A16023@dsic.upv.es> <52C4738E-914B-425C-9A2F-D64F8AB03564@dsic.upv.es> <06B0B044-A856-47DE-B317-476210E3A038@mcs.anl.gov> Message-ID: <5F6973DE-0785-4122-B084-1BC78AF09F17@mcs.anl.gov> On Aug 9, 2011, at 2:54 AM, Shitij Bhargava wrote: > Thanks Jose, Barry. > > I tried what you said, but that gives me an error: > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Argument out of range! > [0]PETSC ERROR: Can only get local values, trying 9! > > This is probably because here I am trying to insert all rows of the matrix through process 0, but process 0 doesnt own all the rows. > > In any case, this seems very "unnatural", so I am using MPIAIJ the right way as you said, where I assemble the MPIAIJ matrix in parallel instead of only on one process. I have done that actually, and am running the code on the cluster right now. Its going to take a long long time to finish, It shouldn't take a long time to finish. Are you sure you are creating all the objects with the PETSC_COMM_WORLD and not PETSC_COMM_SELF? Have you done the correct matrix preallocation http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#efficient-assembly? Is each process generating just its part of the matrix? Barry > so I cant confirm some of my doubts, which I am asking below: > > 1. If I run the code with 1 process, and say it takes M memory (peak) while solving for eigenvalues, then when I run it with N processes, each will take nearly M/N memory (peak) (probably a little more) right ? And for doing this, I dont have to use any special MPI stuff....the fact that I am using MPIAIJ, and building the EPS object from it, and then calling EPSSolve() is enough ? I mean EPSSolve() is internally in some way distributing memory and computation effort automatically when I use MPIAIJ, and run the code with many processes, right ? > This confusion is there because when I use top, while running the code with 8 processes, each of them showed me nearly 250 mb initially, but each has grown to use 270 mb in about 70 minutes. I understand that the method krylovschur is such that memory requirements increase slowly, but the peak on any process will be less (than if I ran only one process), right ? (Even though their memory requirements are growing, they will grow to some M/N only, right ?) > > Actually the fact that in this case, each of the process creates its own EPS context, initializes it itself, and then calls EPSSolve() itself without any "interaction" with other processes makes me wonder if they really are working together, or just individually (I would have verified this myself, but the program will take way too much time, and I know I would have to kill it sooner or later).....or the fact that they initialize their own EPS context with THEIR part of the MPI is enough to make them "cooperate and work together" ? (Although I think this is what Barry meant in that last post, but I am not too sure) > > I am not too comfortable with the MPI way of thinking right now, probably this is why I have this confusion. > > Anyways, I cant thank you guys enough. I would have been scrounging through documentation again and again to no avail if you guys had not helped me the way you did. The responses were always prompt, always to the point (even though my questions were sometimes not, probably because I didnt completely understand the problems I was facing.....but you always knew what I was asking) and very clear. At this moment, I dont know much about PETSc/SLEPc myself, but I will be sure to contribute back to this list when I do. I have nothing but sincere gratitude for you guys. > > > Thank you very much ! > > Shitij > > > On 9 August 2011 00:58, Barry Smith wrote: > > On Aug 8, 2011, at 2:14 AM, Shitij Bhargava wrote: > > > Thank you Jed. That was indeed the problem. I installed a separate MPI for PETSc/SLEPc, but was running my program with a default, already installed one. > > > > Now, I have a different question. What I want to do is this: > > > > 1. Only 1 process, say root, calculates the matrix in SeqAIJ format > > 2. Then root creates the EPS context, eps and initializes,sets parameters, problem type,etc. properly > > 3. After this the root process broadcasts this eps object to other processes > > 4. I use EPSSolve to solve for eigenvalues (all process together in cooperation resulting in memory distribution) > > 5. I get the results from root > > We do have an undocumented routine MatDistribute_MPIAIJ(MPI_Comm comm,Mat gmat,PetscInt m,MatReuse reuse,Mat *inmat) in src/mat/impls/aij/mpi/mpiaij.c that will take a SeqAIJ matrix and distribute it over a larger MPI communicator. > > Note that you cannot create the EPS context etc on a the root process and then broadcast the object but once the matrix is distributed you can simple create the EPS context etc on the parallel communicator where the matrix is and run with that. > > Barry > > > > > is this possible ? I am not able to broadcast the EPS object, because it is not an MPI_DataType. Is there any PETSc/SLEPc function for this ? I am avoiding using MPIAIJ because that will mean making many changes in the existing code, including the numerous write(*,*) statements (i would have to convert them to PetscPrint in FORTRAN or something like that). > > So I want a single process to handle matrix generation and assembly, but want to solve the eigenproblem in parallel by different processes. Running the subroutine EPSSolve in parallel and hence distribute memory is the only reason why I want to use MPI. > > > > Thanks a lot !! > > > > Shitij > > > > On 8 August 2011 11:05, Jed Brown wrote: > > On Mon, Aug 8, 2011 at 00:29, Shitij Bhargava wrote: > > I ran it with: > > > > mpirun -np 2 ./slepcEigenMPI -eps_monitor > > > > I didnt do exactly what you said, because the matrix generation part in the actual program is quite time consuming itself. But I assume what I am doing is equivalent to what you meant to do? Also, I put MPD as PETSC_DECIDE, because I didnt know what to put it for this matrix dimension. > > > > This is the output I get: (part of the output) > > MATRIX ASSMEBLY DONE !!!!!!!! > > > > MATRIX ASSMEBLY DONE !!!!!!!! > > > > 1 EPS nconv=98 first unconverged value (error) 1490.88 (1.73958730e-05) > > 1 EPS nconv=98 first unconverged value (error) 1490.88 (1.73958730e-05) > > 2 EPS nconv=282 first unconverged value (error) 3.04636e-27 (2.49532175e-04) > > 2 EPS nconv=282 first unconverged value (error) 3.04636e-27 (2.49532175e-04) > > > > The most likely case is that you have more than one MPI implementation installed and that you are running with a different implementation than you built with. Compare the outputs: > > > > $ ldd ./slepcEigenMPI > > $ which mpirun > > > > From Debao.Shao at brion.com Tue Aug 9 21:36:58 2011 From: Debao.Shao at brion.com (Debao Shao) Date: Tue, 9 Aug 2011 19:36:58 -0700 Subject: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate In-Reply-To: References: <4E3B2BBA.5000408@geology.wisc.edu> <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A1@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A7@EX03> <8287A816-74DC-4D96-9FAB-123E215D1AA7@mcs.anl.gov> <384FF55F15E3E447802DC8CCA85696980E143123BB@EX03> <384FF55F15E3E447802DC8CCA85696980E14312523@EX03> <384FF55F15E3E447802DC8CCA85696980E143125C6@EX03> Message-ID: <384FF55F15E3E447802DC8CCA85696980E1EA81146@EX03> Hi, Matt: http://www.mcs.anl.gov/petsc/petsc-as/miscellaneous/petscthreads.html says "PETSc is not currently thread-safe". And, you said "PETSc allows threads", I was confused, can you explain a little for me? Thanks a lot. Debao ________________________________ From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Saturday, August 06, 2011 11:19 AM To: PETSc users list Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate On Sat, Aug 6, 2011 at 1:34 AM, Debao Shao > wrote: Thanks a lot, Jed and Aron. Do you have ideas for this problem? But I encounter a new problem, the situation is: 1, the matrix is big, and can be partitioned to several blocks; 2, started several threads to handle each block of matrix; 3, integrated all block matrices together. Again, the program crashed with reporting: "[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range" Yes, PETSc allows threads, but one of your threads is illegally accessing memory. I suggest using valgrind to track this down. Matt My libpetsc.a is built with option "--with-mpi=0 --with-debugging=0 -with-log=0 -with-info=0". Will it support this type of multiple threads? Thanks, Debao ________________________________ From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Friday, August 05, 2011 9:26 PM To: PETSc users list Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate On Fri, Aug 5, 2011 at 04:00, Debao Shao > wrote: 1, I put "PetscInitialize" at the beginning of main, and "PetscInitialize" registers a signal handler to capture "SIGTERM"; 2, the problem ran in server side will send SIGTERM to kill all of clients after one stage is done. 3, then, unfortunately, the signal "SIGTERM" is caught by the signal handler installed by "PetscInitialize", and caused to abort. You can remove the order dependence by running with -no_signal_handler or setting your own signal handler with: http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Sys/PetscPushSignalHandler.html ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Aug 9 22:42:34 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 9 Aug 2011 23:42:34 -0400 Subject: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate In-Reply-To: <384FF55F15E3E447802DC8CCA85696980E1EA81146@EX03> References: <4E3B2BBA.5000408@geology.wisc.edu> <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A1@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A7@EX03> <8287A816-74DC-4D96-9FAB-123E215D1AA7@mcs.anl.gov> <384FF55F15E3E447802DC8CCA85696980E143123BB@EX03> <384FF55F15E3E447802DC8CCA85696980E14312523@EX03> <384FF55F15E3E447802DC8CCA85696980E143125C6@EX03> <384FF55F15E3E447802DC8CCA85696980E1EA81146@EX03> Message-ID: On Tue, Aug 9, 2011 at 10:36 PM, Debao Shao wrote: > Hi, Matt: **** > > ** ** > > http://www.mcs.anl.gov/petsc/petsc-as/miscellaneous/petscthreads.html says > ?PETSc is not currently thread-safe?.**** > > And, you said ?PETSc allows threads?, I was confused, can you explain a > little for me? > You are not prevented from using threads, but PETSc data structures are not thread-safe (we do nothing with threads). By allow, I meant you can do whatever you want, but we do not supported a threaded model. Sorry it was not clear. Matt > > > Thanks a lot. **** > > ** ** > > Debao **** > ------------------------------ > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Matthew Knepley > *Sent:* Saturday, August 06, 2011 11:19 AM > > *To:* PETSc users list > *Subject:* Re: [petsc-users] PETSC ERROR: Caught signal number 15 > Terminate > **** > > ** ** > > On Sat, Aug 6, 2011 at 1:34 AM, Debao Shao wrote:** > ** > > Thanks a lot, Jed and Aron. **** > > **** > > Do you have ideas for this problem?**** > > **** > > But I encounter a new problem, the situation is:**** > > 1, the matrix is big, and can be partitioned to several blocks;**** > > 2, started several threads to handle each block of matrix;**** > > 3, integrated all block matrices together.**** > > **** > > Again, the program crashed with reporting: **** > > "[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range"**** > > ** ** > > Yes, PETSc allows threads, but one of your threads is illegally accessing > memory. I suggest using valgrind to track this down.**** > > ** ** > > Matt**** > > **** > > My libpetsc.a is built with option "--with-mpi=0 --with-debugging=0 > -with-log=0 -with-info=0".**** > > Will it support this type of multiple threads?**** > > **** > > Thanks,**** > > Debao**** > ------------------------------ > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Jed Brown > *Sent:* Friday, August 05, 2011 9:26 PM > *To:* PETSc users list > *Subject:* Re: [petsc-users] PETSC ERROR: Caught signal number 15 > Terminate**** > > **** > > On Fri, Aug 5, 2011 at 04:00, Debao Shao wrote:**** > > 1, I put "PetscInitialize" at the beginning of main, and "PetscInitialize" > registers a signal handler to capture "SIGTERM";**** > > 2, the problem ran in server side will send SIGTERM to kill all of clients > after one stage is done.**** > > 3, then, unfortunately, the signal "SIGTERM" is caught by the signal > handler installed by "PetscInitialize", and caused to abort.**** > > You can remove the order dependence by running with -no_signal_handler or > setting your own signal handler with:**** > > **** > > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Sys/PetscPushSignalHandler.html > **** > > ** ** > ------------------------------ > > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt.**** > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener**** > > ------------------------------ > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is > provided on an AS-IS basis without any express or implied warranties or > liabilities. To the extent you are relying on this information, you are > doing so at your own risk. If you are not the intended recipient, please > notify the sender immediately by replying to this message and destroy all > copies of this message and any attachments. ASML is neither liable for the > proper and complete transmission of the information contained in this > communication, nor for any delay in its receipt. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Debao.Shao at brion.com Tue Aug 9 22:50:00 2011 From: Debao.Shao at brion.com (Debao Shao) Date: Tue, 9 Aug 2011 20:50:00 -0700 Subject: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate In-Reply-To: References: <4E3B2BBA.5000408@geology.wisc.edu> <384FF55F15E3E447802DC8CCA85696980E1431238F@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A1@EX03> <384FF55F15E3E447802DC8CCA85696980E143123A7@EX03> <8287A816-74DC-4D96-9FAB-123E215D1AA7@mcs.anl.gov> <384FF55F15E3E447802DC8CCA85696980E143123BB@EX03> <384FF55F15E3E447802DC8CCA85696980E14312523@EX03> <384FF55F15E3E447802DC8CCA85696980E143125C6@EX03> <384FF55F15E3E447802DC8CCA85696980E1EA81146@EX03> Message-ID: <384FF55F15E3E447802DC8CCA85696980E1EA81191@EX03> Ok, I see. Thanks a lot. ________________________________ From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Wednesday, August 10, 2011 11:43 AM To: PETSc users list Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate On Tue, Aug 9, 2011 at 10:36 PM, Debao Shao > wrote: Hi, Matt: http://www.mcs.anl.gov/petsc/petsc-as/miscellaneous/petscthreads.html says "PETSc is not currently thread-safe". And, you said "PETSc allows threads", I was confused, can you explain a little for me? You are not prevented from using threads, but PETSc data structures are not thread-safe (we do nothing with threads). By allow, I meant you can do whatever you want, but we do not supported a threaded model. Sorry it was not clear. Matt Thanks a lot. Debao ________________________________ From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Saturday, August 06, 2011 11:19 AM To: PETSc users list Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate On Sat, Aug 6, 2011 at 1:34 AM, Debao Shao > wrote: Thanks a lot, Jed and Aron. Do you have ideas for this problem? But I encounter a new problem, the situation is: 1, the matrix is big, and can be partitioned to several blocks; 2, started several threads to handle each block of matrix; 3, integrated all block matrices together. Again, the program crashed with reporting: "[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range" Yes, PETSc allows threads, but one of your threads is illegally accessing memory. I suggest using valgrind to track this down. Matt My libpetsc.a is built with option "--with-mpi=0 --with-debugging=0 -with-log=0 -with-info=0". Will it support this type of multiple threads? Thanks, Debao ________________________________ From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Friday, August 05, 2011 9:26 PM To: PETSc users list Subject: Re: [petsc-users] PETSC ERROR: Caught signal number 15 Terminate On Fri, Aug 5, 2011 at 04:00, Debao Shao > wrote: 1, I put "PetscInitialize" at the beginning of main, and "PetscInitialize" registers a signal handler to capture "SIGTERM"; 2, the problem ran in server side will send SIGTERM to kill all of clients after one stage is done. 3, then, unfortunately, the signal "SIGTERM" is caught by the signal handler installed by "PetscInitialize", and caused to abort. You can remove the order dependence by running with -no_signal_handler or setting your own signal handler with: http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Sys/PetscPushSignalHandler.html ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener ________________________________ -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Peter.Graf at nrel.gov Wed Aug 10 07:39:57 2011 From: Peter.Graf at nrel.gov (Graf, Peter) Date: Wed, 10 Aug 2011 06:39:57 -0600 Subject: [petsc-users] SNES convergence failure (line search) for Neumann boundary condition Message-ID: Dear Petsc-Users, I am implementing a 1D solution of the (drift/diffusion) semiconductor equations. The case of Dirichlet boundaries works fine. The case of Neumann boundaries (i.e. current boundary conditions such as occur at semiconductor/metal contacts) does not. In sorting this out, I have modified one of your examples to recreate the problem: The (modified) example seeks to solve u`` + u^{2} = f on [0,1] with u'(0) = U0 u(1) = 1 With U0 = 0, the SNES solver converges. With U0=-2 (for example) it does not: 11 SNES Function norm 5.018284941187e+00 Nonlinear solve did not converge due to DIVERGED_LS_FAILURE (interestingly, it still comes up with roughly good solutions, e.g. at least visually, which suggests to me my discretization is not _total_ garbage) Along the way (with -info) I get messages of the form: SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 208.031 near zero implies inconsistent rhs At the end, I get: [0] SNESLSCheckLocalMin_Private(): || J^T F|| 80.3 near zero implies found a local minimum I have used -snes_type test to verify that my Jacobian is accurate. I have also dumped the Jacobians to a file and examined their condition numbers. The nonconverged cases have condition numbers around 10^7, whereas the converged cases have condition numbers around 10^3. This seems like a clue, but I'm not sure what to do about it. All this is completely analogous to the real case of interest (in that case, my condition numbers go to 10^13 or more, and there is similar visual evidence that the solver is "trying" to do the right thing). I would love to have help getting this to work. Specifically, what does the message about the "inconsistent rhs" tell me I'm doing wrong? What is the evidence of the condition number telling me? Do I have an error somewhere, or do I have a legitimately ill-conditioned Jacobian (in which case, what should I do about that?)? Thank you very much for any advice, Peter Graf NREL Scientific Computing Center Golden, CO From knepley at gmail.com Wed Aug 10 08:13:08 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Aug 2011 13:13:08 +0000 Subject: [petsc-users] SNES convergence failure (line search) for Neumann boundary condition In-Reply-To: References: Message-ID: On Wed, Aug 10, 2011 at 12:39 PM, Graf, Peter wrote: > Dear Petsc-Users, > I am implementing a 1D solution of the (drift/diffusion) semiconductor > equations. The case of Dirichlet boundaries works fine. The case of > Neumann boundaries (i.e. current boundary conditions such as occur at > semiconductor/metal contacts) does not. In sorting this out, I have > modified one of your examples to recreate the problem: > > The (modified) example seeks to solve > u`` + u^{2} = f > on [0,1] with > u'(0) = U0 > u(1) = 1 > > With U0 = 0, the SNES solver converges. With U0=-2 (for example) it does > not: > Just to be clear, both of these are Neumann conditions. > 11 SNES Function norm 5.018284941187e+00 > Nonlinear solve did not converge due to DIVERGED_LS_FAILURE > When asking about convergence, please send the entire output of -snes_view -snes_monitor -ksp_monitor -snes_converged_reason Since the Jacobian becomes more ill-conditioned, its possible the linear solver is not converging. When testing, its best to always use -ksp_type preonly -pc_type lu so that the solves are exact to start out. Thanks, Matt > (interestingly, it still comes up with roughly good solutions, e.g. at > least visually, which suggests to me my discretization is not _total_ > garbage) > > Along the way (with -info) I get messages of the form: > SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 208.031 near zero > implies inconsistent rhs > > At the end, I get: > [0] SNESLSCheckLocalMin_Private(): || J^T F|| 80.3 near zero implies found > a local minimum > > I have used -snes_type test to verify that my Jacobian is accurate. > I have also dumped the Jacobians to a file and examined their condition > numbers. The nonconverged cases have condition numbers around 10^7, whereas > the converged cases have condition numbers around 10^3. This seems like a > clue, but I'm not sure what to do about it. > > All this is completely analogous to the real case of interest (in that > case, my condition numbers go to 10^13 or more, and there is similar visual > evidence that the solver is "trying" to do the right thing). > I would love to have help getting this to work. Specifically, what does > the message about the "inconsistent rhs" tell me I'm doing wrong? What is > the evidence of the condition number telling me? Do I have an error > somewhere, or do I have a legitimately ill-conditioned Jacobian (in which > case, what should I do about that?)? > > Thank you very much for any advice, > > Peter Graf > NREL Scientific Computing Center > Golden, CO > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Peter.Graf at nrel.gov Wed Aug 10 09:06:09 2011 From: Peter.Graf at nrel.gov (Graf, Peter) Date: Wed, 10 Aug 2011 08:06:09 -0600 Subject: [petsc-users] SNES convergence failure (line search) for Neumann boundary condition In-Reply-To: Message-ID: Hi Matt, Thanks for your reply. Attached is the output as requested, specifically, of ./snes_test -snes_monitor -snes_converged_reason -ksp_converged_reason -snes_view -snes_monitor -ksp_monitor -ksp_type preonly -pc_type lu For good measure (?), I am also attaching the case with "-info". (R.e. terminology, I was contrasting the case here with the Dirichlet case of, e.g., u(0) =1, u(1) = 1. Actually I think I'm still wrong, in that my BCs are mixed type,... but thanks for being clear) Again, thank you very much for any help. -Peter On 8/10/11 7:13 AM, "Matthew Knepley" wrote: On Wed, Aug 10, 2011 at 12:39 PM, Graf, Peter wrote: Dear Petsc-Users, I am implementing a 1D solution of the (drift/diffusion) semiconductor equations. The case of Dirichlet boundaries works fine. The case of Neumann boundaries (i.e. current boundary conditions such as occur at semiconductor/metal contacts) does not. In sorting this out, I have modified one of your examples to recreate the problem: The (modified) example seeks to solve u`` + u^{2} = f on [0,1] with u'(0) = U0 u(1) = 1 With U0 = 0, the SNES solver converges. With U0=-2 (for example) it does not: Just to be clear, both of these are Neumann conditions. 11 SNES Function norm 5.018284941187e+00 Nonlinear solve did not converge due to DIVERGED_LS_FAILURE When asking about convergence, please send the entire output of -snes_view -snes_monitor -ksp_monitor -snes_converged_reason Since the Jacobian becomes more ill-conditioned, its possible the linear solver is not converging. When testing, its best to always use -ksp_type preonly -pc_type lu so that the solves are exact to start out. Thanks, Matt (interestingly, it still comes up with roughly good solutions, e.g. at least visually, which suggests to me my discretization is not _total_ garbage) Along the way (with -info) I get messages of the form: SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 208.031 near zero implies inconsistent rhs At the end, I get: [0] SNESLSCheckLocalMin_Private(): || J^T F|| 80.3 near zero implies found a local minimum I have used -snes_type test to verify that my Jacobian is accurate. I have also dumped the Jacobians to a file and examined their condition numbers. The nonconverged cases have condition numbers around 10^7, whereas the converged cases have condition numbers around 10^3. This seems like a clue, but I'm not sure what to do about it. All this is completely analogous to the real case of interest (in that case, my condition numbers go to 10^13 or more, and there is similar visual evidence that the solver is "trying" to do the right thing). I would love to have help getting this to work. Specifically, what does the message about the "inconsistent rhs" tell me I'm doing wrong? What is the evidence of the condition number telling me? Do I have an error somewhere, or do I have a legitimately ill-conditioned Jacobian (in which case, what should I do about that?)? Thank you very much for any advice, Peter Graf NREL Scientific Computing Center Golden, CO -------------- next part -------------- A non-text attachment was scrubbed... Name: verbose_output2 Type: application/octet-stream Size: 2325 bytes Desc: verbose_output2 URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: verbose_output_info Type: application/octet-stream Size: 25428 bytes Desc: verbose_output_info URL: From gdiso at ustc.edu Wed Aug 10 09:31:25 2011 From: gdiso at ustc.edu (Gong Ding) Date: Wed, 10 Aug 2011 22:31:25 +0800 Subject: [petsc-users] SNES convergence failure (line search) for Neumann boundary condition References: Message-ID: <26B40CD2E6434047A1D1C3734E2A0C0C@cogendaeda> Hmm, semiconductor simulation... I had spent years on this project. Just for tip, we had a open source version of semiconductor device simulator Genius at http://www.cogenda.com I guess this is the most well coded simulator you can find. Petsc is extensively used in genius code, you may find some useful information there. Gong Ding On Wed, Aug 10, 2011 at 12:39 PM, Graf, Peter wrote: Dear Petsc-Users, I am implementing a 1D solution of the (drift/diffusion) semiconductor equations. The case of Dirichlet boundaries works fine. The case of Neumann boundaries (i.e. current boundary conditions such as occur at semiconductor/metal contacts) does not. In sorting this out, I have modified one of your examples to recreate the problem: The (modified) example seeks to solve u`` + u^{2} = f on [0,1] with u'(0) = U0 u(1) = 1 With U0 = 0, the SNES solver converges. With U0=-2 (for example) it does not: Just to be clear, both of these are Neumann conditions. 11 SNES Function norm 5.018284941187e+00 Nonlinear solve did not converge due to DIVERGED_LS_FAILURE When asking about convergence, please send the entire output of -snes_view -snes_monitor -ksp_monitor -snes_converged_reason Since the Jacobian becomes more ill-conditioned, its possible the linear solver is not converging. When testing, its best to always use -ksp_type preonly -pc_type lu so that the solves are exact to start out. Thanks, Matt (interestingly, it still comes up with roughly good solutions, e.g. at least visually, which suggests to me my discretization is not _total_ garbage) Along the way (with -info) I get messages of the form: SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 208.031 near zero implies inconsistent rhs At the end, I get: [0] SNESLSCheckLocalMin_Private(): || J^T F|| 80.3 near zero implies found a local minimum I have used -snes_type test to verify that my Jacobian is accurate. I have also dumped the Jacobians to a file and examined their condition numbers. The nonconverged cases have condition numbers around 10^7, whereas the converged cases have condition numbers around 10^3. This seems like a clue, but I'm not sure what to do about it. All this is completely analogous to the real case of interest (in that case, my condition numbers go to 10^13 or more, and there is similar visual evidence that the solver is "trying" to do the right thing). I would love to have help getting this to work. Specifically, what does the message about the "inconsistent rhs" tell me I'm doing wrong? What is the evidence of the condition number telling me? Do I have an error somewhere, or do I have a legitimately ill-conditioned Jacobian (in which case, what should I do about that?)? Thank you very much for any advice, Peter Graf NREL Scientific Computing Center Golden, CO From bsmith at mcs.anl.gov Wed Aug 10 09:53:44 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 10 Aug 2011 09:53:44 -0500 Subject: [petsc-users] SNES convergence failure (line search) for Neumann boundary condition In-Reply-To: References: Message-ID: <6094BD6C-1631-40E0-A828-59CEC88BBB03@mcs.anl.gov> Peter, Matt gave some bad advice. Please rerun with -ksp_type fgmre -ksp_monitor -ksp_converged_reason as well as the -snes_monitor and send the output. (With the preonly it doesn't produce enough useful information about how the linear solver did). > Along the way (with -info) I get messages of the form: > SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 208.031 near zero implies inconsistent rhs > > At the end, I get: > [0] SNESLSCheckLocalMin_Private(): || J^T F|| 80.3 near zero implies found a local minimum These are fine, there is no local minimum. By "near zero" we mean like 10^-7. These are no where near zero. Barry On Aug 10, 2011, at 9:06 AM, Graf, Peter wrote: > Hi Matt, > Thanks for your reply. Attached is the output as requested, specifically, of > > ./snes_test -snes_monitor -snes_converged_reason -ksp_converged_reason -snes_view -snes_monitor -ksp_monitor -ksp_type preonly -pc_type lu > > For good measure (?), I am also attaching the case with "-info". > > (R.e. terminology, I was contrasting the case here with the Dirichlet case of, e.g., u(0) =1, u(1) = 1. Actually I think I'm still wrong, in that my BCs are mixed type,... but thanks for being clear) > > Again, thank you very much for any help. > -Peter > > On 8/10/11 7:13 AM, "Matthew Knepley" wrote: > > On Wed, Aug 10, 2011 at 12:39 PM, Graf, Peter wrote: > Dear Petsc-Users, > I am implementing a 1D solution of the (drift/diffusion) semiconductor equations. The case of Dirichlet boundaries works fine. The case of Neumann boundaries (i.e. current boundary conditions such as occur at semiconductor/metal contacts) does not. In sorting this out, I have modified one of your examples to recreate the problem: > > The (modified) example seeks to solve > u`` + u^{2} = f > on [0,1] with > u'(0) = U0 > u(1) = 1 > > With U0 = 0, the SNES solver converges. With U0=-2 (for example) it does not: > > Just to be clear, both of these are Neumann conditions. > > 11 SNES Function norm 5.018284941187e+00 > Nonlinear solve did not converge due to DIVERGED_LS_FAILURE > > When asking about convergence, please send the entire output of -snes_view -snes_monitor -ksp_monitor -snes_converged_reason > > Since the Jacobian becomes more ill-conditioned, its possible the linear solver is not converging. When testing, its best to always use > -ksp_type preonly -pc_type lu so that the solves are exact to start out. > > Thanks, > > Matt > > (interestingly, it still comes up with roughly good solutions, e.g. at least visually, which suggests to me my discretization is not _total_ garbage) > > Along the way (with -info) I get messages of the form: > SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 208.031 near zero implies inconsistent rhs > > At the end, I get: > [0] SNESLSCheckLocalMin_Private(): || J^T F|| 80.3 near zero implies found a local minimum > > I have used -snes_type test to verify that my Jacobian is accurate. > I have also dumped the Jacobians to a file and examined their condition numbers. The nonconverged cases have condition numbers around 10^7, whereas the converged cases have condition numbers around 10^3. This seems like a clue, but I'm not sure what to do about it. > > All this is completely analogous to the real case of interest (in that case, my condition numbers go to 10^13 or more, and there is similar visual evidence that the solver is "trying" to do the right thing). > I would love to have help getting this to work. Specifically, what does the message about the "inconsistent rhs" tell me I'm doing wrong? What is the evidence of the condition number telling me? Do I have an error somewhere, or do I have a legitimately ill-conditioned Jacobian (in which case, what should I do about that?)? > > Thank you very much for any advice, > > Peter Graf > NREL Scientific Computing Center > Golden, CO > > > From Peter.Graf at nrel.gov Wed Aug 10 11:10:48 2011 From: Peter.Graf at nrel.gov (Graf, Peter) Date: Wed, 10 Aug 2011 10:10:48 -0600 Subject: [petsc-users] SNES convergence failure (line search) for Neumann boundary condition In-Reply-To: <6094BD6C-1631-40E0-A828-59CEC88BBB03@mcs.anl.gov> Message-ID: Hi Barry, Thanks for your response. I did not see a "fgmre" option, but "fgmres" works, and here is the output (including -info). Thanks in advance for any help you can offer -Peter On 8/10/11 8:53 AM, "Barry Smith" wrote: Peter, Matt gave some bad advice. Please rerun with -ksp_type fgmre -ksp_monitor -ksp_converged_reason as well as the -snes_monitor and send the output. (With the preonly it doesn't produce enough useful information about how the linear solver did). > Along the way (with -info) I get messages of the form: > SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 208.031 near zero implies inconsistent rhs > > At the end, I get: > [0] SNESLSCheckLocalMin_Private(): || J^T F|| 80.3 near zero implies found a local minimum These are fine, there is no local minimum. By "near zero" we mean like 10^-7. These are no where near zero. Barry On Aug 10, 2011, at 9:06 AM, Graf, Peter wrote: > Hi Matt, > Thanks for your reply. Attached is the output as requested, specifically, of > > ./snes_test -snes_monitor -snes_converged_reason -ksp_converged_reason -snes_view -snes_monitor -ksp_monitor -ksp_type preonly -pc_type lu > > For good measure (?), I am also attaching the case with "-info". > > (R.e. terminology, I was contrasting the case here with the Dirichlet case of, e.g., u(0) =1, u(1) = 1. Actually I think I'm still wrong, in that my BCs are mixed type,... but thanks for being clear) > > Again, thank you very much for any help. > -Peter > > On 8/10/11 7:13 AM, "Matthew Knepley" wrote: > > On Wed, Aug 10, 2011 at 12:39 PM, Graf, Peter wrote: > Dear Petsc-Users, > I am implementing a 1D solution of the (drift/diffusion) semiconductor equations. The case of Dirichlet boundaries works fine. The case of Neumann boundaries (i.e. current boundary conditions such as occur at semiconductor/metal contacts) does not. In sorting this out, I have modified one of your examples to recreate the problem: > > The (modified) example seeks to solve > u`` + u^{2} = f > on [0,1] with > u'(0) = U0 > u(1) = 1 > > With U0 = 0, the SNES solver converges. With U0=-2 (for example) it does not: > > Just to be clear, both of these are Neumann conditions. > > 11 SNES Function norm 5.018284941187e+00 > Nonlinear solve did not converge due to DIVERGED_LS_FAILURE > > When asking about convergence, please send the entire output of -snes_view -snes_monitor -ksp_monitor -snes_converged_reason > > Since the Jacobian becomes more ill-conditioned, its possible the linear solver is not converging. When testing, its best to always use > -ksp_type preonly -pc_type lu so that the solves are exact to start out. > > Thanks, > > Matt > > (interestingly, it still comes up with roughly good solutions, e.g. at least visually, which suggests to me my discretization is not _total_ garbage) > > Along the way (with -info) I get messages of the form: > SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 208.031 near zero implies inconsistent rhs > > At the end, I get: > [0] SNESLSCheckLocalMin_Private(): || J^T F|| 80.3 near zero implies found a local minimum > > I have used -snes_type test to verify that my Jacobian is accurate. > I have also dumped the Jacobians to a file and examined their condition numbers. The nonconverged cases have condition numbers around 10^7, whereas the converged cases have condition numbers around 10^3. This seems like a clue, but I'm not sure what to do about it. > > All this is completely analogous to the real case of interest (in that case, my condition numbers go to 10^13 or more, and there is similar visual evidence that the solver is "trying" to do the right thing). > I would love to have help getting this to work. Specifically, what does the message about the "inconsistent rhs" tell me I'm doing wrong? What is the evidence of the condition number telling me? Do I have an error somewhere, or do I have a legitimately ill-conditioned Jacobian (in which case, what should I do about that?)? > > Thank you very much for any advice, > > Peter Graf > NREL Scientific Computing Center > Golden, CO > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: more_output Type: application/octet-stream Size: 31740 bytes Desc: more_output URL: From knepley at gmail.com Wed Aug 10 13:42:15 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Aug 2011 18:42:15 +0000 Subject: [petsc-users] SNES convergence failure (line search) for Neumann boundary condition In-Reply-To: References: <6094BD6C-1631-40E0-A828-59CEC88BBB03@mcs.anl.gov> Message-ID: On Wed, Aug 10, 2011 at 4:10 PM, Graf, Peter wrote: > Hi Barry, > Thanks for your response. I did not see a "fgmre" option, but "fgmres" > works, and here is the output (including -info). > Thanks in advance for any help you can offer > Okay, the linear system is being solved, but Newton convergence is stalling. It may be that Newton just cannot solve this system, but it is more likely that there is an error in the Jacobian. In order to test this, you can use -snes_fd which is very slow, but forms the entire Jacobian using finite differences. Thanks, Matt > -Peter > > On 8/10/11 8:53 AM, "Barry Smith" wrote: > > > > Peter, > > Matt gave some bad advice. Please rerun with -ksp_type fgmre > -ksp_monitor -ksp_converged_reason as well as the -snes_monitor and send the > output. (With the preonly it doesn't produce enough useful information > about how the linear solver did). > > > Along the way (with -info) I get messages of the form: > > SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 208.031 near zero > implies inconsistent rhs > > > > At the end, I get: > > [0] SNESLSCheckLocalMin_Private(): || J^T F|| 80.3 near zero implies > found a local minimum > > These are fine, there is no local minimum. By "near zero" we mean like > 10^-7. These are no where near zero. > > > Barry > > On Aug 10, 2011, at 9:06 AM, Graf, Peter wrote: > > > Hi Matt, > > Thanks for your reply. Attached is the output as requested, > specifically, of > > > > ./snes_test -snes_monitor -snes_converged_reason -ksp_converged_reason > -snes_view -snes_monitor -ksp_monitor -ksp_type preonly -pc_type lu > > > > For good measure (?), I am also attaching the case with "-info". > > > > (R.e. terminology, I was contrasting the case here with the Dirichlet > case of, e.g., u(0) =1, u(1) = 1. Actually I think I'm still wrong, in that > my BCs are mixed type,... but thanks for being clear) > > > > Again, thank you very much for any help. > > -Peter > > > > On 8/10/11 7:13 AM, "Matthew Knepley" wrote: > > > > On Wed, Aug 10, 2011 at 12:39 PM, Graf, Peter > wrote: > > Dear Petsc-Users, > > I am implementing a 1D solution of the (drift/diffusion) semiconductor > equations. The case of Dirichlet boundaries works fine. The case of > Neumann boundaries (i.e. current boundary conditions such as occur at > semiconductor/metal contacts) does not. In sorting this out, I have > modified one of your examples to recreate the problem: > > > > The (modified) example seeks to solve > > u`` + u^{2} = f > > on [0,1] with > > u'(0) = U0 > > u(1) = 1 > > > > With U0 = 0, the SNES solver converges. With U0=-2 (for example) it does > not: > > > > Just to be clear, both of these are Neumann conditions. > > > > 11 SNES Function norm 5.018284941187e+00 > > Nonlinear solve did not converge due to DIVERGED_LS_FAILURE > > > > When asking about convergence, please send the entire output of > -snes_view -snes_monitor -ksp_monitor -snes_converged_reason > > > > Since the Jacobian becomes more ill-conditioned, its possible the linear > solver is not converging. When testing, its best to always use > > -ksp_type preonly -pc_type lu so that the solves are exact to start out. > > > > Thanks, > > > > Matt > > > > (interestingly, it still comes up with roughly good solutions, e.g. at > least visually, which suggests to me my discretization is not _total_ > garbage) > > > > Along the way (with -info) I get messages of the form: > > SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 208.031 near zero > implies inconsistent rhs > > > > At the end, I get: > > [0] SNESLSCheckLocalMin_Private(): || J^T F|| 80.3 near zero implies > found a local minimum > > > > I have used -snes_type test to verify that my Jacobian is accurate. > > I have also dumped the Jacobians to a file and examined their condition > numbers. The nonconverged cases have condition numbers around 10^7, whereas > the converged cases have condition numbers around 10^3. This seems like a > clue, but I'm not sure what to do about it. > > > > All this is completely analogous to the real case of interest (in that > case, my condition numbers go to 10^13 or more, and there is similar visual > evidence that the solver is "trying" to do the right thing). > > I would love to have help getting this to work. Specifically, what does > the message about the "inconsistent rhs" tell me I'm doing wrong? What is > the evidence of the condition number telling me? Do I have an error > somewhere, or do I have a legitimately ill-conditioned Jacobian (in which > case, what should I do about that?)? > > > > Thank you very much for any advice, > > > > Peter Graf > > NREL Scientific Computing Center > > Golden, CO > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Peter.Graf at nrel.gov Wed Aug 10 14:19:31 2011 From: Peter.Graf at nrel.gov (Graf, Peter) Date: Wed, 10 Aug 2011 13:19:31 -0600 Subject: [petsc-users] SNES convergence failure (line search) for Neumann boundary condition In-Reply-To: Message-ID: Hi Matt, Thanks again for the attention. Here's what happens: bash-3.2$ ./snes_test -snes_monitor -snes_converged_reason -ksp_converged_reason -snes_view -snes_monitor -ksp_monitor -ksp_type fgmres -snes_fd > log same result: Nonlinear solve did not converge due to DIVERGED_LS_FAILURE log attached. Barry has my code to fiddle with. I'm pretty sure the model can be solved, and I'm just doing something wrong. Would be happy to have you guys point out what it is! Thanks again, -Peter On 8/10/11 12:42 PM, "Matthew Knepley" wrote: On Wed, Aug 10, 2011 at 4:10 PM, Graf, Peter wrote: Hi Barry, Thanks for your response. I did not see a "fgmre" option, but "fgmres" works, and here is the output (including -info). Thanks in advance for any help you can offer Okay, the linear system is being solved, but Newton convergence is stalling. It may be that Newton just cannot solve this system, but it is more likely that there is an error in the Jacobian. In order to test this, you can use -snes_fd which is very slow, but forms the entire Jacobian using finite differences. Thanks, Matt -Peter On 8/10/11 8:53 AM, "Barry Smith" wrote: Peter, Matt gave some bad advice. Please rerun with -ksp_type fgmre -ksp_monitor -ksp_converged_reason as well as the -snes_monitor and send the output. (With the preonly it doesn't produce enough useful information about how the linear solver did). > Along the way (with -info) I get messages of the form: > SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 208.031 near zero implies inconsistent rhs > > At the end, I get: > [0] SNESLSCheckLocalMin_Private(): || J^T F|| 80.3 near zero implies found a local minimum These are fine, there is no local minimum. By "near zero" we mean like 10^-7. These are no where near zero. Barry On Aug 10, 2011, at 9:06 AM, Graf, Peter wrote: > Hi Matt, > Thanks for your reply. Attached is the output as requested, specifically, of > > ./snes_test -snes_monitor -snes_converged_reason -ksp_converged_reason -snes_view -snes_monitor -ksp_monitor -ksp_type preonly -pc_type lu > > For good measure (?), I am also attaching the case with "-info". > > (R.e. terminology, I was contrasting the case here with the Dirichlet case of, e.g., u(0) =1, u(1) = 1. Actually I think I'm still wrong, in that my BCs are mixed type,... but thanks for being clear) > > Again, thank you very much for any help. > -Peter > > On 8/10/11 7:13 AM, "Matthew Knepley" wrote: > > On Wed, Aug 10, 2011 at 12:39 PM, Graf, Peter wrote: > Dear Petsc-Users, > I am implementing a 1D solution of the (drift/diffusion) semiconductor equations. The case of Dirichlet boundaries works fine. The case of Neumann boundaries (i.e. current boundary conditions such as occur at semiconductor/metal contacts) does not. In sorting this out, I have modified one of your examples to recreate the problem: > > The (modified) example seeks to solve > u`` + u^{2} = f > on [0,1] with > u'(0) = U0 > u(1) = 1 > > With U0 = 0, the SNES solver converges. With U0=-2 (for example) it does not: > > Just to be clear, both of these are Neumann conditions. > > 11 SNES Function norm 5.018284941187e+00 > Nonlinear solve did not converge due to DIVERGED_LS_FAILURE > > When asking about convergence, please send the entire output of -snes_view -snes_monitor -ksp_monitor -snes_converged_reason > > Since the Jacobian becomes more ill-conditioned, its possible the linear solver is not converging. When testing, its best to always use > -ksp_type preonly -pc_type lu so that the solves are exact to start out. > > Thanks, > > Matt > > (interestingly, it still comes up with roughly good solutions, e.g. at least visually, which suggests to me my discretization is not _total_ garbage) > > Along the way (with -info) I get messages of the form: > SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 208.031 near zero implies inconsistent rhs > > At the end, I get: > [0] SNESLSCheckLocalMin_Private(): || J^T F|| 80.3 near zero implies found a local minimum > > I have used -snes_type test to verify that my Jacobian is accurate. > I have also dumped the Jacobians to a file and examined their condition numbers. The nonconverged cases have condition numbers around 10^7, whereas the converged cases have condition numbers around 10^3. This seems like a clue, but I'm not sure what to do about it. > > All this is completely analogous to the real case of interest (in that case, my condition numbers go to 10^13 or more, and there is similar visual evidence that the solver is "trying" to do the right thing). > I would love to have help getting this to work. Specifically, what does the message about the "inconsistent rhs" tell me I'm doing wrong? What is the evidence of the condition number telling me? Do I have an error somewhere, or do I have a legitimately ill-conditioned Jacobian (in which case, what should I do about that?)? > > Thank you very much for any advice, > > Peter Graf > NREL Scientific Computing Center > Golden, CO > > > From Peter.Graf at nrel.gov Wed Aug 10 14:20:32 2011 From: Peter.Graf at nrel.gov (Graf, Peter) Date: Wed, 10 Aug 2011 13:20:32 -0600 Subject: [petsc-users] SNES convergence failure (line search) for Neumann boundary condition In-Reply-To: Message-ID: Matt, I forgot to attach the output file from the -snes_fd attempt. Here it is. Thanks, -Peter On 8/10/11 12:42 PM, "Matthew Knepley" wrote: On Wed, Aug 10, 2011 at 4:10 PM, Graf, Peter wrote: Hi Barry, Thanks for your response. I did not see a "fgmre" option, but "fgmres" works, and here is the output (including -info). Thanks in advance for any help you can offer Okay, the linear system is being solved, but Newton convergence is stalling. It may be that Newton just cannot solve this system, but it is more likely that there is an error in the Jacobian. In order to test this, you can use -snes_fd which is very slow, but forms the entire Jacobian using finite differences. Thanks, Matt -Peter On 8/10/11 8:53 AM, "Barry Smith" wrote: Peter, Matt gave some bad advice. Please rerun with -ksp_type fgmre -ksp_monitor -ksp_converged_reason as well as the -snes_monitor and send the output. (With the preonly it doesn't produce enough useful information about how the linear solver did). > Along the way (with -info) I get messages of the form: > SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 208.031 near zero implies inconsistent rhs > > At the end, I get: > [0] SNESLSCheckLocalMin_Private(): || J^T F|| 80.3 near zero implies found a local minimum These are fine, there is no local minimum. By "near zero" we mean like 10^-7. These are no where near zero. Barry On Aug 10, 2011, at 9:06 AM, Graf, Peter wrote: > Hi Matt, > Thanks for your reply. Attached is the output as requested, specifically, of > > ./snes_test -snes_monitor -snes_converged_reason -ksp_converged_reason -snes_view -snes_monitor -ksp_monitor -ksp_type preonly -pc_type lu > > For good measure (?), I am also attaching the case with "-info". > > (R.e. terminology, I was contrasting the case here with the Dirichlet case of, e.g., u(0) =1, u(1) = 1. Actually I think I'm still wrong, in that my BCs are mixed type,... but thanks for being clear) > > Again, thank you very much for any help. > -Peter > > On 8/10/11 7:13 AM, "Matthew Knepley" wrote: > > On Wed, Aug 10, 2011 at 12:39 PM, Graf, Peter wrote: > Dear Petsc-Users, > I am implementing a 1D solution of the (drift/diffusion) semiconductor equations. The case of Dirichlet boundaries works fine. The case of Neumann boundaries (i.e. current boundary conditions such as occur at semiconductor/metal contacts) does not. In sorting this out, I have modified one of your examples to recreate the problem: > > The (modified) example seeks to solve > u`` + u^{2} = f > on [0,1] with > u'(0) = U0 > u(1) = 1 > > With U0 = 0, the SNES solver converges. With U0=-2 (for example) it does not: > > Just to be clear, both of these are Neumann conditions. > > 11 SNES Function norm 5.018284941187e+00 > Nonlinear solve did not converge due to DIVERGED_LS_FAILURE > > When asking about convergence, please send the entire output of -snes_view -snes_monitor -ksp_monitor -snes_converged_reason > > Since the Jacobian becomes more ill-conditioned, its possible the linear solver is not converging. When testing, its best to always use > -ksp_type preonly -pc_type lu so that the solves are exact to start out. > > Thanks, > > Matt > > (interestingly, it still comes up with roughly good solutions, e.g. at least visually, which suggests to me my discretization is not _total_ garbage) > > Along the way (with -info) I get messages of the form: > SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 208.031 near zero implies inconsistent rhs > > At the end, I get: > [0] SNESLSCheckLocalMin_Private(): || J^T F|| 80.3 near zero implies found a local minimum > > I have used -snes_type test to verify that my Jacobian is accurate. > I have also dumped the Jacobians to a file and examined their condition numbers. The nonconverged cases have condition numbers around 10^7, whereas the converged cases have condition numbers around 10^3. This seems like a clue, but I'm not sure what to do about it. > > All this is completely analogous to the real case of interest (in that case, my condition numbers go to 10^13 or more, and there is similar visual evidence that the solver is "trying" to do the right thing). > I would love to have help getting this to work. Specifically, what does the message about the "inconsistent rhs" tell me I'm doing wrong? What is the evidence of the condition number telling me? Do I have an error somewhere, or do I have a legitimately ill-conditioned Jacobian (in which case, what should I do about that?)? > > Thank you very much for any advice, > > Peter Graf > NREL Scientific Computing Center > Golden, CO > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: log Type: application/octet-stream Size: 3509 bytes Desc: log URL: From shitij.cse at gmail.com Thu Aug 11 02:23:05 2011 From: shitij.cse at gmail.com (Shitij Bhargava) Date: Thu, 11 Aug 2011 12:53:05 +0530 Subject: [petsc-users] SLEPc eigensolver that uses minimal memory and finds ALL eigenvalues of a real symmetric sparse matrix in reasonable time In-Reply-To: <5F6973DE-0785-4122-B084-1BC78AF09F17@mcs.anl.gov> References: <359DE4F9-1B2E-459F-8EF1-4B8E78A16023@dsic.upv.es> <52C4738E-914B-425C-9A2F-D64F8AB03564@dsic.upv.es> <06B0B044-A856-47DE-B317-476210E3A038@mcs.anl.gov> <5F6973DE-0785-4122-B084-1BC78AF09F17@mcs.anl.gov> Message-ID: Thank you both for your replies. Actually, Barry, you are right. On eight CPUs, as it turns out, it will not take as long as I imagined. For a 9600x9600 matrix, to solve for half the largest eigenvalues, it took about 300 minutes. Although it would have taken much more time than this for solving half the smallest eigenvalues (I had to kill it at total time of 600 minutes). This is still much much longer than the LAPACK method, which takes (for calculating all the eigenvalues at once) about 90 minutes (at the cost of much memory, which cant even be distributed -- which is unacceptable. Also, I suppose you probably didnt know that I have to calculate ALL the eigenvalues). But still, I was expecting that it wouldnt complete even in "days", that is why I said it would take a very long time....I forgot to take into account the fact that it was now running nearly 8 times faster. (in one earlier instance I had to kill the process when I was running the same program on one process only (was not using MPI), as it had run over 13 hours and still hadnt completed !!) And yes, I am making all the objects with PETSC_COMM_WORLD only, also, the top command shows me eight processes running at nearly 100% CPU during call to EPSSolve(). (I suppose that verifies what I am saying ? Though I am not sure...) I have done preallocation carefully, and verified that that part doesnt take any extra time...so, actually when I referred to "time" here, I was talking about time taken by the eigensolver only (because I was already satisfied by the time taken for matrix generation and assembly) Each process is not "generating" its part of the matrix only at this moment. I am doing things in a very lousy way right now. Each process is generating all the rows of the matrix, but inserts only the rows which fall into its ownership range....but as I said, matrix generation and assembly is not the time bottleneck in the program....eigensolver is....I am satisfied with the time matrix generation and assembly is taking, although I understand that ideally matrix generation should also be done in parallel. My time for this summer training has ended, so I'll probably fix that later. The amount of work that I have been able to accomplish in this training, has been good because of your help. I might just have been stuck in some error otherwise. Thank you very much once again !! Shitij On 10 August 2011 02:02, Barry Smith wrote: > > On Aug 9, 2011, at 2:54 AM, Shitij Bhargava wrote: > > > Thanks Jose, Barry. > > > > I tried what you said, but that gives me an error: > > > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > > [0]PETSC ERROR: Argument out of range! > > [0]PETSC ERROR: Can only get local values, trying 9! > > > > This is probably because here I am trying to insert all rows of the > matrix through process 0, but process 0 doesnt own all the rows. > > > > In any case, this seems very "unnatural", so I am using MPIAIJ the right > way as you said, where I assemble the MPIAIJ matrix in parallel instead of > only on one process. I have done that actually, and am running the code on > the cluster right now. Its going to take a long long time to finish, > > > It shouldn't take a long time to finish. Are you sure you are creating > all the objects with the PETSC_COMM_WORLD and not PETSC_COMM_SELF? Have you > done the correct matrix preallocation > http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#efficient-assembly? > Is each process generating just its part of the matrix? > > Barry > > > > so I cant confirm some of my doubts, which I am asking below: > > > > 1. If I run the code with 1 process, and say it takes M memory (peak) > while solving for eigenvalues, then when I run it with N processes, each > will take nearly M/N memory (peak) (probably a little more) right ? And for > doing this, I dont have to use any special MPI stuff....the fact that I am > using MPIAIJ, and building the EPS object from it, and then calling > EPSSolve() is enough ? I mean EPSSolve() is internally in some way > distributing memory and computation effort automatically when I use MPIAIJ, > and run the code with many processes, right ? > > This confusion is there because when I use top, while running the code > with 8 processes, each of them showed me nearly 250 mb initially, but each > has grown to use 270 mb in about 70 minutes. I understand that the method > krylovschur is such that memory requirements increase slowly, but the peak > on any process will be less (than if I ran only one process), right ? (Even > though their memory requirements are growing, they will grow to some M/N > only, right ?) > > > > Actually the fact that in this case, each of the process creates its own > EPS context, initializes it itself, and then calls EPSSolve() itself without > any "interaction" with other processes makes me wonder if they really are > working together, or just individually (I would have verified this myself, > but the program will take way too much time, and I know I would have to kill > it sooner or later).....or the fact that they initialize their own EPS > context with THEIR part of the MPI is enough to make them "cooperate and > work together" ? (Although I think this is what Barry meant in that last > post, but I am not too sure) > > > > I am not too comfortable with the MPI way of thinking right now, probably > this is why I have this confusion. > > > > Anyways, I cant thank you guys enough. I would have been scrounging > through documentation again and again to no avail if you guys had not helped > me the way you did. The responses were always prompt, always to the point > (even though my questions were sometimes not, probably because I didnt > completely understand the problems I was facing.....but you always knew what > I was asking) and very clear. At this moment, I dont know much about > PETSc/SLEPc myself, but I will be sure to contribute back to this list when > I do. I have nothing but sincere gratitude for you guys. > > > > > > Thank you very much ! > > > > Shitij > > > > > > On 9 August 2011 00:58, Barry Smith wrote: > > > > On Aug 8, 2011, at 2:14 AM, Shitij Bhargava wrote: > > > > > Thank you Jed. That was indeed the problem. I installed a separate MPI > for PETSc/SLEPc, but was running my program with a default, already > installed one. > > > > > > Now, I have a different question. What I want to do is this: > > > > > > 1. Only 1 process, say root, calculates the matrix in SeqAIJ format > > > 2. Then root creates the EPS context, eps and initializes,sets > parameters, problem type,etc. properly > > > 3. After this the root process broadcasts this eps object to other > processes > > > 4. I use EPSSolve to solve for eigenvalues (all process together in > cooperation resulting in memory distribution) > > > 5. I get the results from root > > > > We do have an undocumented routine MatDistribute_MPIAIJ(MPI_Comm > comm,Mat gmat,PetscInt m,MatReuse reuse,Mat *inmat) in > src/mat/impls/aij/mpi/mpiaij.c that will take a SeqAIJ matrix and distribute > it over a larger MPI communicator. > > > > Note that you cannot create the EPS context etc on a the root process > and then broadcast the object but once the matrix is distributed you can > simple create the EPS context etc on the parallel communicator where the > matrix is and run with that. > > > > Barry > > > > > > > > is this possible ? I am not able to broadcast the EPS object, because > it is not an MPI_DataType. Is there any PETSc/SLEPc function for this ? I am > avoiding using MPIAIJ because that will mean making many changes in the > existing code, including the numerous write(*,*) statements (i would have to > convert them to PetscPrint in FORTRAN or something like that). > > > So I want a single process to handle matrix generation and assembly, > but want to solve the eigenproblem in parallel by different processes. > Running the subroutine EPSSolve in parallel and hence distribute memory is > the only reason why I want to use MPI. > > > > > > Thanks a lot !! > > > > > > Shitij > > > > > > On 8 August 2011 11:05, Jed Brown wrote: > > > On Mon, Aug 8, 2011 at 00:29, Shitij Bhargava > wrote: > > > I ran it with: > > > > > > mpirun -np 2 ./slepcEigenMPI -eps_monitor > > > > > > I didnt do exactly what you said, because the matrix generation part in > the actual program is quite time consuming itself. But I assume what I am > doing is equivalent to what you meant to do? Also, I put MPD as > PETSC_DECIDE, because I didnt know what to put it for this matrix dimension. > > > > > > This is the output I get: (part of the output) > > > MATRIX ASSMEBLY DONE !!!!!!!! > > > > > > MATRIX ASSMEBLY DONE !!!!!!!! > > > > > > 1 EPS nconv=98 first unconverged value (error) 1490.88 > (1.73958730e-05) > > > 1 EPS nconv=98 first unconverged value (error) 1490.88 > (1.73958730e-05) > > > 2 EPS nconv=282 first unconverged value (error) 3.04636e-27 > (2.49532175e-04) > > > 2 EPS nconv=282 first unconverged value (error) 3.04636e-27 > (2.49532175e-04) > > > > > > The most likely case is that you have more than one MPI implementation > installed and that you are running with a different implementation than you > built with. Compare the outputs: > > > > > > $ ldd ./slepcEigenMPI > > > $ which mpirun > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From clemens.domanig at uibk.ac.at Thu Aug 11 04:46:44 2011 From: clemens.domanig at uibk.ac.at (Clemens Domanig) Date: Thu, 11 Aug 2011 11:46:44 +0200 Subject: [petsc-users] MatGetDiagonal In-Reply-To: References: <4E413987.5090208@uibk.ac.at> Message-ID: <4E43A504.4050600@uibk.ac.at> I checked out the dev-version today but it doesn't fix my problem. Is it possible that your fix is not in th dev-version? Thx. [0]PETSC ERROR: --------------------- Error Message ---------------------------- -------- [0]PETSC ERROR: Object is in wrong state! [0]PETSC ERROR: Not for factored matrix! [0]PETSC ERROR: ---------------------------------------------------------------- -------- [0]PETSC ERROR: Petsc Development HG revision: 173a9704a10394abf9048b83aebd58c4c Hong Zhang wrote: > Clemens : > This is a bug in MatGetDiagonal_SeqSBAIJ() for cholesky factored matrix. > I fixed it and pushed to petsc-dev. > See http://www.mcs.anl.gov/petsc/petsc-as/developers/index.html > on how to get petsc-dev. > > Thanks for reporting the problem. Contact us if you still get problem. > > Hong > > >> Hi, >> >> I already talked to some of you about this in combination with some other >> problem, but somehow I don't get it. >> I want the diagonal entries of the matrix M. But my vector 'diag' is mainly >> full of zeros when using MatGetDiagonal(). >> I can print 'DiagM' (VecView) where I see all the diagonal entries but I >> cannot access them using MatGetValue (Object is in wrong state. Not for >> factored matrix!). >> How can I get access to the diagonal? >> Thx - Clemens >> >> >> KSPCreate( coml, &kspBA); >> KSPGetPC( kspBA, &precond); >> PCSetType( precond, PCCHOLESKY); >> PCFactorSetMatSolverPackage( precond, MAT_SOLVER_PETSC); >> KSPSetOptionsPrefix( kspBA, "diag_pc_type cholesky"); >> KSPSetFromOptions( kspBA); >> KSPSetOperators( kspBA, Kt, Kt, DIFFERENT_NONZERO_PATTERN); >> KSPSetUp( kspBA); >> PCFactorGetMatrix( precond, &DiagM); //DO NOT DESTROY DiagM !!! >> MatGetDiagonal( DiagM, diag); >> From knepley at gmail.com Thu Aug 11 07:37:41 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 11 Aug 2011 12:37:41 +0000 Subject: [petsc-users] SNES convergence failure (line search) for Neumann boundary condition In-Reply-To: References: Message-ID: On Wed, Aug 10, 2011 at 7:19 PM, Graf, Peter wrote: > Hi Matt, > Thanks again for the attention. Here's what happens: > > bash-3.2$ ./snes_test -snes_monitor -snes_converged_reason > -ksp_converged_reason -snes_view -snes_monitor -ksp_monitor -ksp_type fgmres > -snes_fd > log > > same result: > Nonlinear solve did not converge due to DIVERGED_LS_FAILURE > So the linear solve appears to work, FD and analytic Jacobians agree, so either this cannot be solved with Newton using the starting guess you provide, or there is an error in your residual function. I would try an exact solution to check the residual evaluation. Matt > log attached. > Barry has my code to fiddle with. I'm pretty sure the model can be solved, > and I'm just doing something wrong. Would be happy to have you guys point > out what it is! > Thanks again, > -Peter > > > On 8/10/11 12:42 PM, "Matthew Knepley" wrote: > > On Wed, Aug 10, 2011 at 4:10 PM, Graf, Peter wrote: > Hi Barry, > Thanks for your response. I did not see a "fgmre" option, but "fgmres" > works, and here is the output (including -info). > Thanks in advance for any help you can offer > > Okay, the linear system is being solved, but Newton convergence is > stalling. It may be that Newton just cannot solve > this system, but it is more likely that there is an error in the Jacobian. > In order to test this, you can use > > -snes_fd > > which is very slow, but forms the entire Jacobian using finite differences. > > Thanks, > > Matt > > -Peter > > On 8/10/11 8:53 AM, "Barry Smith" wrote: > > > > Peter, > > Matt gave some bad advice. Please rerun with -ksp_type fgmre > -ksp_monitor -ksp_converged_reason as well as the -snes_monitor and send the > output. (With the preonly it doesn't produce enough useful information > about how the linear solver did). > > > Along the way (with -info) I get messages of the form: > > SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 208.031 near zero > implies inconsistent rhs > > > > At the end, I get: > > [0] SNESLSCheckLocalMin_Private(): || J^T F|| 80.3 near zero implies > found a local minimum > > These are fine, there is no local minimum. By "near zero" we mean like > 10^-7. These are no where near zero. > > > Barry > > On Aug 10, 2011, at 9:06 AM, Graf, Peter wrote: > > > Hi Matt, > > Thanks for your reply. Attached is the output as requested, > specifically, of > > > > ./snes_test -snes_monitor -snes_converged_reason -ksp_converged_reason > -snes_view -snes_monitor -ksp_monitor -ksp_type preonly -pc_type lu > > > > For good measure (?), I am also attaching the case with "-info". > > > > (R.e. terminology, I was contrasting the case here with the Dirichlet > case of, e.g., u(0) =1, u(1) = 1. Actually I think I'm still wrong, in that > my BCs are mixed type,... but thanks for being clear) > > > > Again, thank you very much for any help. > > -Peter > > > > On 8/10/11 7:13 AM, "Matthew Knepley" wrote: > > > > On Wed, Aug 10, 2011 at 12:39 PM, Graf, Peter > wrote: > > Dear Petsc-Users, > > I am implementing a 1D solution of the (drift/diffusion) semiconductor > equations. The case of Dirichlet boundaries works fine. The case of > Neumann boundaries (i.e. current boundary conditions such as occur at > semiconductor/metal contacts) does not. In sorting this out, I have > modified one of your examples to recreate the problem: > > > > The (modified) example seeks to solve > > u`` + u^{2} = f > > on [0,1] with > > u'(0) = U0 > > u(1) = 1 > > > > With U0 = 0, the SNES solver converges. With U0=-2 (for example) it does > not: > > > > Just to be clear, both of these are Neumann conditions. > > > > 11 SNES Function norm 5.018284941187e+00 > > Nonlinear solve did not converge due to DIVERGED_LS_FAILURE > > > > When asking about convergence, please send the entire output of > -snes_view -snes_monitor -ksp_monitor -snes_converged_reason > > > > Since the Jacobian becomes more ill-conditioned, its possible the linear > solver is not converging. When testing, its best to always use > > -ksp_type preonly -pc_type lu so that the solves are exact to start out. > > > > Thanks, > > > > Matt > > > > (interestingly, it still comes up with roughly good solutions, e.g. at > least visually, which suggests to me my discretization is not _total_ > garbage) > > > > Along the way (with -info) I get messages of the form: > > SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 208.031 near zero > implies inconsistent rhs > > > > At the end, I get: > > [0] SNESLSCheckLocalMin_Private(): || J^T F|| 80.3 near zero implies > found a local minimum > > > > I have used -snes_type test to verify that my Jacobian is accurate. > > I have also dumped the Jacobians to a file and examined their condition > numbers. The nonconverged cases have condition numbers around 10^7, whereas > the converged cases have condition numbers around 10^3. This seems like a > clue, but I'm not sure what to do about it. > > > > All this is completely analogous to the real case of interest (in that > case, my condition numbers go to 10^13 or more, and there is similar visual > evidence that the solver is "trying" to do the right thing). > > I would love to have help getting this to work. Specifically, what does > the message about the "inconsistent rhs" tell me I'm doing wrong? What is > the evidence of the condition number telling me? Do I have an error > somewhere, or do I have a legitimately ill-conditioned Jacobian (in which > case, what should I do about that?)? > > > > Thank you very much for any advice, > > > > Peter Graf > > NREL Scientific Computing Center > > Golden, CO > > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Aug 11 09:07:46 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 11 Aug 2011 09:07:46 -0500 Subject: [petsc-users] SLEPc eigensolver that uses minimal memory and finds ALL eigenvalues of a real symmetric sparse matrix in reasonable time In-Reply-To: References: <359DE4F9-1B2E-459F-8EF1-4B8E78A16023@dsic.upv.es> <52C4738E-914B-425C-9A2F-D64F8AB03564@dsic.upv.es> <06B0B044-A856-47DE-B317-476210E3A038@mcs.anl.gov> <5F6973DE-0785-4122-B084-1BC78AF09F17@mcs.anl.gov> Message-ID: On Thu, Aug 11, 2011 at 02:23, Shitij Bhargava wrote: > On eight CPUs, as it turns out, it will not take as long as I imagined. For > a 9600x9600 matrix, to solve for half the largest eigenvalues, it took about > 300 minutes. Although it would have taken much more time than this for > solving half the smallest eigenvalues (I had to kill it at total time of 600 > minutes). This is still much much longer than the LAPACK method, which takes > (for calculating all the eigenvalues at once) about 90 minutes (at the cost > of much memory, which cant even be distributed -- which is unacceptable. You should try Elemental for distributed-memory dense eigensolvers. http://code.google.com/p/elemental/ > Also, I suppose you probably didnt know that I have to calculate ALL the > eigenvalues). Why do you need all of them? What underlying problem are you solving? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ked1 at rice.edu Thu Aug 11 11:00:01 2011 From: ked1 at rice.edu (ked1 at rice.edu) Date: Thu, 11 Aug 2011 11:00:01 -0500 Subject: [petsc-users] Problem With MatRestoreRow Message-ID: <20110811110001.31471yefz5ducsch@webmail.rice.edu> I am attempting to build a matrix and then get a row (or the number of nonzeros in the row) in Fortran 90. I can build the matrix and use it for calculations such as matrix multiplication, but if I call MatGetRow and then MatRestoreRow, I get an error about object type. However, if I reduce the size of the arrays I'm using in the MatSetValues command, everything works (except that I'm not building what I want). The section of my code giving trouble is below. ---------------------- CODE BEGINS ---------------------------- Mat M PetscInt dnnz(locneq), onnz(locneq), rows(64) PetscScalar Me(64*64) ... dnnz = 64 onnz = 0 call MatCreateMPIAIJ(petsc_comm_world,locneq,locneq,gnngu,gnngu,petsc_null,dnnz,petsc_null,onnz,M,ierr) Me = 1.0D0 if (procid==0) then do ii=1,ne ! for all elements do jj=1,64 ! for number of rows kk = uconn(ii,jj) ! get local node number rows(jj) = gnoden(kk)-1 ! get global node number end do call MatSetValues(M,64,rows,64,rows,Me,add_values,ierr) end do end if ! Assemble call MatAssemblyBegin(M,mat_final_assembly,ierr) call MatAssemblyEnd(M,mat_final_assembly,ierr) ! Get number of nonzero columns if (procid==0) then call MatGetRow(M,0,ii,petsc_null,petsc_null,ierr) print *, ii call MatRestoreRow(M,0,ii,petsc_null,petsc_null,ierr) end if ---------------------- CODE ENDS ---------------------------- The error I get is [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Invalid argument! [0]PETSC ERROR: Wrong type of object: Parameter # 1! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./semg3par on a linux-gnu named login1.sugar.rice.edu by ked1 Thu Aug 11 10:52:46 2011 [0]PETSC ERROR: Libraries linked from /projects/bch3862/petsc/linux-gnu-c-debug/lib [0]PETSC ERROR: Configure run at Sun Jul 17 15:34:01 2011 [0]PETSC ERROR: Configure options --with-cc=icc --with-fc=ifort --download-f-blas-lapack=1 --download-mpich=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatRestoreRow() line 388 in src/mat/interface/matrix.c Line 388 in matrix.c is PetscValidHeaderSpecific(mat,MAT_COOKIE,1); Again, if I lower the number of rows, it will not give an error. In this case, replacing all instances of 64 with 46 will give no error. Going to 47 gives a error. Thanks for any help. Kenny From hzhang at mcs.anl.gov Thu Aug 11 11:03:12 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Thu, 11 Aug 2011 11:03:12 -0500 Subject: [petsc-users] MatGetDiagonal In-Reply-To: <4E43A504.4050600@uibk.ac.at> References: <4E413987.5090208@uibk.ac.at> <4E43A504.4050600@uibk.ac.at> Message-ID: The fix is in petsc-dev: http://petsc.cs.iit.edu/petsc/petsc-dev/rev/a19846efa232 I just pushed a test of MatGetDiagonal(F,diag) for petsc Cholesky factorization in ~petsc-dev/src/ksp/ksp/examples/tutorials/ex52.c Do cd ~petsc-dev hg pull hg update or get the latest petsc-dev. Run ex52 with ./ex52 -use_petsc_ch ... Vector Object: 1 MPI processes type: seq 4 3.75 3.73333 3.73214 ... 3.24979 3.30916 Norm of error < 1.e-12 iterations 1 Let us know if you do not get the above result. Hong On Thu, Aug 11, 2011 at 4:46 AM, Clemens Domanig wrote: > I checked out the dev-version today but it doesn't fix my problem. > Is it possible that your fix is not in th dev-version? > Thx. > > [0]PETSC ERROR: --------------------- Error Message > ---------------------------- > -------- > [0]PETSC ERROR: Object is in wrong state! > [0]PETSC ERROR: Not for factored matrix! > [0]PETSC ERROR: > ---------------------------------------------------------------- > -------- > [0]PETSC ERROR: Petsc Development HG revision: > 173a9704a10394abf9048b83aebd58c4c > > > > > Hong Zhang wrote: >> >> Clemens : >> This is a bug in MatGetDiagonal_SeqSBAIJ() for cholesky factored matrix. >> I fixed it and pushed to petsc-dev. >> See http://www.mcs.anl.gov/petsc/petsc-as/developers/index.html >> on how to get petsc-dev. >> >> Thanks for reporting the problem. Contact us if you still get problem. >> >> Hong >> >> >>> Hi, >>> >>> I already talked to some of you about this in combination with some other >>> problem, but somehow I don't get it. >>> I want the diagonal entries of the matrix M. But my vector 'diag' is >>> mainly >>> full of zeros when using MatGetDiagonal(). >>> I can print 'DiagM' (VecView) where I see all the diagonal entries but I >>> cannot access them using MatGetValue (Object is in wrong state. Not for >>> factored matrix!). >>> How can I get access to the diagonal? >>> Thx - Clemens >>> >>> >>> KSPCreate( coml, &kspBA); >>> KSPGetPC( kspBA, &precond); >>> PCSetType( precond, PCCHOLESKY); >>> PCFactorSetMatSolverPackage( precond, MAT_SOLVER_PETSC); >>> KSPSetOptionsPrefix( kspBA, "diag_pc_type cholesky"); >>> KSPSetFromOptions( kspBA); >>> KSPSetOperators( kspBA, Kt, Kt, DIFFERENT_NONZERO_PATTERN); >>> KSPSetUp( kspBA); >>> PCFactorGetMatrix( precond, &DiagM); ? ?//DO NOT DESTROY DiagM !!! >>> MatGetDiagonal( DiagM, diag); >>> > > From bsmith at mcs.anl.gov Thu Aug 11 11:41:26 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 11 Aug 2011 11:41:26 -0500 Subject: [petsc-users] Problem With MatRestoreRow In-Reply-To: <20110811110001.31471yefz5ducsch@webmail.rice.edu> References: <20110811110001.31471yefz5ducsch@webmail.rice.edu> Message-ID: One problem is you cannot pass petsc_null in Fortran you must pass the specific type for example petsc_null_double or petsc_null_integer depending on the argument type. This may or may not completely resolve the problem, send email to petsc-maint at mcs.anl.gov if you still have difficulties Barry On Aug 11, 2011, at 11:00 AM, ked1 at rice.edu wrote: > I am attempting to build a matrix and then get a row (or the number of nonzeros in the row) in Fortran 90. I can build the matrix and use it for calculations such as matrix multiplication, but if I call MatGetRow and then MatRestoreRow, I get an error about object type. > > However, if I reduce the size of the arrays I'm using in the MatSetValues command, everything works (except that I'm not building what I want). > > > The section of my code giving trouble is below. > > ---------------------- CODE BEGINS ---------------------------- > Mat M > PetscInt dnnz(locneq), onnz(locneq), rows(64) > PetscScalar Me(64*64) > > ... > > dnnz = 64 > onnz = 0 > call MatCreateMPIAIJ(petsc_comm_world,locneq,locneq,gnngu,gnngu,petsc_null,dnnz,petsc_null,onnz,M,ierr) > > Me = 1.0D0 > if (procid==0) then > do ii=1,ne ! for all elements > do jj=1,64 ! for number of rows > kk = uconn(ii,jj) ! get local node number > rows(jj) = gnoden(kk)-1 ! get global node number > end do > call MatSetValues(M,64,rows,64,rows,Me,add_values,ierr) > end do > end if > > ! Assemble > call MatAssemblyBegin(M,mat_final_assembly,ierr) > call MatAssemblyEnd(M,mat_final_assembly,ierr) > > ! Get number of nonzero columns > if (procid==0) then > call MatGetRow(M,0,ii,petsc_null,petsc_null,ierr) > print *, ii > call MatRestoreRow(M,0,ii,petsc_null,petsc_null,ierr) > end if > ---------------------- CODE ENDS ---------------------------- > > The error I get is > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Invalid argument! > [0]PETSC ERROR: Wrong type of object: Parameter # 1! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./semg3par on a linux-gnu named login1.sugar.rice.edu by ked1 Thu Aug 11 10:52:46 2011 > [0]PETSC ERROR: Libraries linked from /projects/bch3862/petsc/linux-gnu-c-debug/lib > [0]PETSC ERROR: Configure run at Sun Jul 17 15:34:01 2011 > [0]PETSC ERROR: Configure options --with-cc=icc --with-fc=ifort --download-f-blas-lapack=1 --download-mpich=1 > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatRestoreRow() line 388 in src/mat/interface/matrix.c > > Line 388 in matrix.c is > > PetscValidHeaderSpecific(mat,MAT_COOKIE,1); > > Again, if I lower the number of rows, it will not give an error. In this case, replacing all instances of 64 with 46 will give no error. Going to 47 gives a error. > > Thanks for any help. > > Kenny > > > > > > From ked1 at rice.edu Thu Aug 11 12:08:20 2011 From: ked1 at rice.edu (ked1 at rice.edu) Date: Thu, 11 Aug 2011 12:08:20 -0500 Subject: [petsc-users] Problem With MatRestoreRow In-Reply-To: References: <20110811110001.31471yefz5ducsch@webmail.rice.edu> Message-ID: <20110811120820.610055jqyh7xxmdg@webmail.rice.edu> Barry, That worked. Thanks for the info. Kenny Quoting Barry Smith : > > One problem is you cannot pass petsc_null in Fortran you must pass > the specific type for example petsc_null_double or > petsc_null_integer depending on the argument type. > This may or may not completely resolve the problem, send email to > petsc-maint at mcs.anl.gov if you still have difficulties > > Barry > > On Aug 11, 2011, at 11:00 AM, ked1 at rice.edu wrote: > >> I am attempting to build a matrix and then get a row (or the number >> of nonzeros in the row) in Fortran 90. I can build the matrix and >> use it for calculations such as matrix multiplication, but if I >> call MatGetRow and then MatRestoreRow, I get an error about object >> type. >> >> However, if I reduce the size of the arrays I'm using in the >> MatSetValues command, everything works (except that I'm not >> building what I want). >> >> >> The section of my code giving trouble is below. >> >> ---------------------- CODE BEGINS ---------------------------- >> Mat M >> PetscInt dnnz(locneq), onnz(locneq), rows(64) >> PetscScalar Me(64*64) >> >> ... >> >> dnnz = 64 >> onnz = 0 >> call >> MatCreateMPIAIJ(petsc_comm_world,locneq,locneq,gnngu,gnngu,petsc_null,dnnz,petsc_null,onnz,M,ierr) >> >> Me = 1.0D0 >> if (procid==0) then >> do ii=1,ne ! for all elements >> do jj=1,64 ! for number of rows >> kk = uconn(ii,jj) ! get local node number >> rows(jj) = gnoden(kk)-1 ! get global node number >> end do >> call MatSetValues(M,64,rows,64,rows,Me,add_values,ierr) >> end do >> end if >> >> ! Assemble >> call MatAssemblyBegin(M,mat_final_assembly,ierr) >> call MatAssemblyEnd(M,mat_final_assembly,ierr) >> >> ! Get number of nonzero columns >> if (procid==0) then >> call MatGetRow(M,0,ii,petsc_null,petsc_null,ierr) >> print *, ii >> call MatRestoreRow(M,0,ii,petsc_null,petsc_null,ierr) >> end if >> ---------------------- CODE ENDS ---------------------------- >> >> The error I get is >> >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Invalid argument! >> [0]PETSC ERROR: Wrong type of object: Parameter # 1! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >> 13:37:48 CDT 2011 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./semg3par on a linux-gnu named >> login1.sugar.rice.edu by ked1 Thu Aug 11 10:52:46 2011 >> [0]PETSC ERROR: Libraries linked from >> /projects/bch3862/petsc/linux-gnu-c-debug/lib >> [0]PETSC ERROR: Configure run at Sun Jul 17 15:34:01 2011 >> [0]PETSC ERROR: Configure options --with-cc=icc --with-fc=ifort >> --download-f-blas-lapack=1 --download-mpich=1 >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: MatRestoreRow() line 388 in src/mat/interface/matrix.c >> >> Line 388 in matrix.c is >> >> PetscValidHeaderSpecific(mat,MAT_COOKIE,1); >> >> Again, if I lower the number of rows, it will not give an error. In >> this case, replacing all instances of 64 with 46 will give no >> error. Going to 47 gives a error. >> >> Thanks for any help. >> >> Kenny >> >> >> >> >> >> > > > From paul.anton.letnes at gmail.com Fri Aug 12 07:55:33 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Fri, 12 Aug 2011 13:55:33 +0100 Subject: [petsc-users] BiCGSTAB for general use Message-ID: <83F76379-F270-48A3-B498-C69934D83F82@gmail.com> Dear petsc users. I am attempting to solve a large, dense equation system. I would like to try using the BiCGSTAB algorithm, and specifically, the petsc library, as it is open source and seems to be well supported. I take it petsc is primarily aimed at people solving partial differential equations by 'brute force' type methods. In my case, I would like to either 1) supply my A and b in Ax = b, and have a library solve it (LU factorization from LAPACK works, but it's slow) or 2) supply matrix products A times a vector (this is what is used internally in BiCGSTAB), the right hand side If necessary, I have a decent idea for a solution guess for the first iteration. So my questions are: 1) Is it feasible, and reasonably convenient, to use petsc for this purpose? 2) If so, is there a good example/tutorial/pseudocode somewhere online, or on the mailinglist, for my use case? I have been reading the petsc examples, but they are somewhat tough to follow for a complete petsc beginner. Also, they seem to apply primarily to PDE solving. I am using the Fortran 90/95 language. Best regards Paul From paul.anton.letnes at gmail.com Fri Aug 12 07:53:29 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Fri, 12 Aug 2011 13:53:29 +0100 Subject: [petsc-users] BiCGSTAB for general use Message-ID: Dear petsc users. I am attempting to solve a large, dense equation system. I would like to try using the BiCGSTAB algorithm, and specifically, the petsc library, as it is open source and seems to be well supported. I take it petsc is primarily aimed at people solving partial differential equations by 'brute force' type methods. In my case, I would like to either 1) supply my A and b in Ax = b, and have a library solve it (LU factorization from LAPACK works, but it's slow) or 2) supply matrix products A times a vector (this is what is used internally in BiCGSTAB), the right hand side If necessary, I have a decent idea for a solution guess for the first iteration. So my questions are: 1) Is it feasible, and reasonably convenient, to use petsc for this purpose? 2) If so, is there a good example/tutorial/pseudocode somewhere online, or on the mailinglist, for my use case? I have been reading the petsc examples, but they are somewhat tough to follow for a complete petsc beginner. Also, they seem to apply primarily to PDE solving. I am using the Fortran 90/95 language. Best regards Paul From hzhang at mcs.anl.gov Fri Aug 12 09:22:12 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Fri, 12 Aug 2011 09:22:12 -0500 Subject: [petsc-users] BiCGSTAB for general use In-Reply-To: References: Message-ID: Paul : Is your matrix A dense or sparse? Petsc is a powerful tool for solving Ax=b with sparse matrix A. When A is dense, petsc supports wrapper to LAPACK (sequential) and PLAPACK (parallel). I would suggest read petsc user manual, and test ~petsc/src/ksp/ksp/examples/tutorials/ex1.c, ex2.c ... then replace matrix and rhs vector with yours. Hong > > I am attempting to solve a large, dense equation system. I would like to try using the BiCGSTAB algorithm, and specifically, the petsc library, as it is open source and seems to be well supported. I take it petsc is primarily aimed at people solving partial differential equations by 'brute force' type methods. In my case, I would like to either > 1) supply my A and b in Ax = b, and have a library solve it (LU factorization from LAPACK works, but it's slow) or > 2) supply matrix products A times a vector (this is what is used internally in BiCGSTAB), the right hand side > If necessary, I have a decent idea for a solution guess for the first iteration. > > So my questions are: > 1) Is it feasible, and reasonably convenient, to use petsc for this purpose? > 2) If so, is there a good example/tutorial/pseudocode somewhere online, or on the mailinglist, for my use case? I have been reading the petsc examples, but they are somewhat tough to follow for a complete petsc beginner. Also, they seem to apply primarily to PDE solving. > > I am using the Fortran 90/95 language. > > Best regards > Paul > > From jedbrown at mcs.anl.gov Fri Aug 12 09:51:12 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 12 Aug 2011 09:51:12 -0500 Subject: [petsc-users] BiCGSTAB for general use In-Reply-To: <83F76379-F270-48A3-B498-C69934D83F82@gmail.com> References: <83F76379-F270-48A3-B498-C69934D83F82@gmail.com> Message-ID: On Fri, Aug 12, 2011 at 07:55, Paul Anton Letnes < paul.anton.letnes at gmail.com> wrote: > I am attempting to solve a large, dense equation system. I would like to > try using the BiCGSTAB algorithm, and specifically, the petsc library, as it > is open source and seems to be well supported. I take it petsc is primarily > aimed at people solving partial differential equations by 'brute force' type > methods. If by "brute force", you mean volumetric discretizations of the differential equations, then this is indeed the largest user base. But there are many optimal methods in this category, such that I think "brute force" would be a misnomer. What sort of problem does your dense equation system come from? E.g. does it come from a boundary element method? Can you give a rough estimate of the condition number? Are the eigen/singular values well-clustered? For many dense problems, a Krylov method alone won't beat a direct solver like LAPACK, but if it has extra structure, and especially if it can be stored/applied in less than O(n^2) work, then iterative methods may be competitive. How large are these matrices likely to be and about how many processors would you like to run on? -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.anton.letnes at gmail.com Fri Aug 12 10:09:13 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Fri, 12 Aug 2011 16:09:13 +0100 Subject: [petsc-users] BiCGSTAB for general use In-Reply-To: References: <83F76379-F270-48A3-B498-C69934D83F82@gmail.com> Message-ID: On 12. aug. 2011, at 15.51, Jed Brown wrote: > If by "brute force", you mean volumetric discretizations of the differential equations, then this is indeed the largest user base. But there are many optimal methods in this category, such that I think "brute force" would be a misnomer. That's what I mean, yes. I'm unfamiliar with the terminology - I certainly do not mean that such methods are inefficient or in any way stupid. > What sort of problem does your dense equation system come from? E.g. does it come from a boundary element method? Can you give a rough estimate of the condition number? Are the eigen/singular values well-clustered? For many dense problems, a Krylov method alone won't beat a direct solver like LAPACK, but if it has extra structure, and especially if it can be stored/applied in less than O(n^2) work, then iterative methods may be competitive. The problem is a discretized integral equation. It does not quite fall into the boundary element category, but it's not too far off, in a sense. I did not do any sophisticated analysis of the singular values, but I do know that the condition number (largest over smallest singular value) is not too bad. The physics of the problem is light scattering from a (weakly, but not extremely weakly) rough surface. The matrix has a big diagonal, as to 0. order the specular (mirror-like) reflection is the same as for a flat interface. This also means I have a decent guess solution (I think). Another benefit of the iterative method is that I could possibly avoid building the matrix explicitly. Setting up the matrix currently takes orders of magnitude less time than solving the equation system, and eats enormous amounts of computer memory. My supervisor has used BiCGSTAB with success on a similar, but harder problem (larger roughness regime). > How large are these matrices likely to be and about how many processors would you like to run on? If I explicitly build the matrices, the ones I have done so far are 20 GB. More would give me better resolution, so that would be nice. I would like to run on at least a single node with 24 cores, as that's the machine architecture for the computer I've got CPU time on. It's possible that, if the method converges after very few iterations, that it could be faster to simply build the matrix on the fly as required. Cheers Paul From jedbrown at mcs.anl.gov Fri Aug 12 10:42:56 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 12 Aug 2011 10:42:56 -0500 Subject: [petsc-users] BiCGSTAB for general use In-Reply-To: References: <83F76379-F270-48A3-B498-C69934D83F82@gmail.com> Message-ID: On Fri, Aug 12, 2011 at 10:09, Paul Anton Letnes < paul.anton.letnes at gmail.com> wrote: > The problem is a discretized integral equation. It does not quite fall into > the boundary element category, but it's not too far off, in a sense. I did > not do any sophisticated analysis of the singular values, but I do know that > the condition number (largest over smallest singular value) is not too bad. > Thanks for the problem description. Is this a second kind integral operator? Such systems have "compact + identity" structure, which means that they can be approximated by low-rank perturbations of the identity. It also means that Krylov methods converge quickly once they pick up the few eigenvalues that are not tightly clustered near 1. (The "compact" part implies that the number of such outliers should be independent of the spatial resolution in your discretization.) If you have a fast way to apply the operator (and note that floating point units are currently 20x to 50x faster than memory bandwidth for matrix-vector products), even unpreconditioned Krylov methods may be able to solve your problem well. You can put your algorithm for applying the matrix inside a MatShell so that all the Krylov methods will work with it. http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Mat/MatCreateShell.html#MatCreateShell If you have a hierarchical discretization, or possibly better, a hierarchical way to apply your operator, then you may be able to use the hierarchy to put together a multigrid method. Don't worry about this part until you have gotten your solver working with a Krylov method, and only then if (a) the number of iterations is sitll large and (b) you want to direct a fair amount of research effort to this topic. If you meet both criteria, write back and we can discuss further. -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.anton.letnes at gmail.com Fri Aug 12 12:26:08 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Fri, 12 Aug 2011 18:26:08 +0100 Subject: [petsc-users] BiCGSTAB for general use In-Reply-To: References: <83F76379-F270-48A3-B498-C69934D83F82@gmail.com> Message-ID: On 12. aug. 2011, at 16.42, Jed Brown wrote: > On Fri, Aug 12, 2011 at 10:09, Paul Anton Letnes wrote: > The problem is a discretized integral equation. It does not quite fall into the boundary element category, but it's not too far off, in a sense. I did not do any sophisticated analysis of the singular values, but I do know that the condition number (largest over smallest singular value) is not too bad. > > Thanks for the problem description. Is this a second kind integral operator? Such systems have "compact + identity" structure, which means that they can be approximated by low-rank perturbations of the identity. It also means that Krylov methods converge quickly once they pick up the few eigenvalues that are not tightly clustered near 1. (The "compact" part implies that the number of such outliers should be independent of the spatial resolution in your discretization.) I'm not 100% sure what you mean by "second kind integral operator", but it is a Fredholm equation of the first kind, as far as I understand (my background is physics rather than mathematics). > > If you have a fast way to apply the operator (and note that floating point units are currently 20x to 50x faster than memory bandwidth for matrix-vector products), even unpreconditioned Krylov methods may be able to solve your problem well. You can put your algorithm for applying the matrix inside a MatShell so that all the Krylov methods will work with it. That's what I was thinking - so assuming I need to use only a few iterations of BiCGSTAB, it may pay off to skip storing the whole (huge!) matrix. I'll look at the MatShell thing, maybe it's what I need. > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Mat/MatCreateShell.html#MatCreateShell > > If you have a hierarchical discretization, or possibly better, a hierarchical way to apply your operator, then you may be able to use the hierarchy to put together a multigrid method. Don't worry about this part until you have gotten your solver working with a Krylov method, and only then if (a) the number of iterations is sitll large and (b) you want to direct a fair amount of research effort to this topic. If you meet both criteria, write back and we can discuss further. It looks like this is in the distant future. As I mentioned, similar (I think, more difficult) problems have been solved, in (if I recall correctly) about 5 iterations. Cheers, and thanks for the help... Paul From jedbrown at mcs.anl.gov Fri Aug 12 15:54:27 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 12 Aug 2011 15:54:27 -0500 Subject: [petsc-users] BiCGSTAB for general use In-Reply-To: References: <83F76379-F270-48A3-B498-C69934D83F82@gmail.com> Message-ID: On Fri, Aug 12, 2011 at 12:26, Paul Anton Letnes < paul.anton.letnes at gmail.com> wrote: > I'm not 100% sure what you mean by "second kind integral operator", but it > is a Fredholm equation of the first kind, as far as I understand (my > background is physics rather than mathematics). Is the thing you're trying to solve with actually of the first kind, not of the second kind? http://en.wikipedia.org/wiki/Fredholm_integral_equation The distinction is whether there is essentially in whether there is a local part to the equation or not. The issue, as I understand it, is that solving a first-kind integral equation is generally not a stable process because the eigenvalues of the integral operator decay to zero, implying that it is essentially low rank, thus not invertible. Maybe you use some regularization to get a system that is not essentially singular? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmhykes at ncsu.edu Sat Aug 13 09:57:45 2011 From: jmhykes at ncsu.edu (Josh Hykes) Date: Sat, 13 Aug 2011 10:57:45 -0400 Subject: [petsc-users] Ordering of preallocation and OwnershipRange Message-ID: Hello, I'm just starting to experiment with PETSc (v3.1), and I like the Python bindings provided by petsc4py (v1.1.2). So far things seem fairly straightforward, but I'm stumped on a small issue. While creating a parallel AIJ matrix, I'd like to preallocate it using arrays d_nnz and o_nnz. As I understand it, these arrays correspond to the processor's local rows. Currently I specify the global matrix size, and let PETSc decide on the decomposition of the rows. I'd like to ask PETSc what rows each processor has with the getOwnershipRange() function, and then do the preallocation. However, according to the error message > [1] MatAnyAIJSetPreallocation() line 393 in petsc4py-1.1.2/src/include/custom.h > [1] Operation done in wrong order > [1] matrix is already preallocated I'm not allowed to do it in this order. Thus, my question is: is it possible to let PETSc figure out the row decomposition while still using d_nnz and o_nnz for the preallocation? I figure that I could resolve the problem by doing my own decomposition, but it'd be nice if I could let those details up to PETSc. I'm including an example using petsc4py of what I'd like to do, run with 2 MPI processes. I apologize if this is a dumb question. Thank you for your help. -Josh # ----------------------------------------------- from petsc4py import PETSc as petsc M, N = 4, 6 global_d_nnz = [2, 1, 1, 2] global_o_nnz = [1, 3, 2, 1] A = petsc.Mat() A.create(petsc.COMM_WORLD) A.setSizes([M, N]) A.setType('aij') i_start, i_end = A.getOwnershipRange() A.setPreallocationNNZ([global_d_nnz[i_start:i_end], global_o_nnz[i_start:i_end]]) # error occurs here -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Aug 13 10:28:41 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 13 Aug 2011 15:28:41 +0000 Subject: [petsc-users] Ordering of preallocation and OwnershipRange In-Reply-To: References: Message-ID: On Sat, Aug 13, 2011 at 2:57 PM, Josh Hykes wrote: > Hello, > > I'm just starting to experiment with PETSc (v3.1), and I like the Python > bindings provided by petsc4py (v1.1.2). So far things seem fairly > straightforward, but I'm stumped on a small issue. > > While creating a parallel AIJ matrix, I'd like to preallocate it using > arrays d_nnz and o_nnz. As I understand it, these arrays correspond to the > processor's local rows. > > Currently I specify the global matrix size, and let PETSc decide on the > decomposition of the rows. I'd like to ask PETSc what rows each processor > has with the getOwnershipRange() function, and then do the preallocation. > However, according to the error message > > > [1] MatAnyAIJSetPreallocation() line 393 in > petsc4py-1.1.2/src/include/custom.h > > [1] Operation done in wrong order > > [1] matrix is already preallocated > > I'm not allowed to do it in this order. > > Thus, my question is: is it possible to let PETSc figure out the row > decomposition while still using d_nnz and o_nnz for the preallocation? I > figure that I could resolve the problem by doing my own decomposition, but > it'd be nice if I could let those details up to PETSc. > You are correct. We require that preallocation is done at the same time as decomposition. There are tricky dependencies in matrix creation. However, an easy workaround is to create a Vec at the same time with the same global size, since it is guaranteed to have the same layout. I will look into simplifying this if it is possible. Thanks, Matt > I'm including an example using petsc4py of what I'd like to do, run with 2 > MPI processes. > > I apologize if this is a dumb question. Thank you for your help. > > -Josh > > # ----------------------------------------------- > from petsc4py import PETSc as petsc > > M, N = 4, 6 > > global_d_nnz = [2, 1, 1, 2] > global_o_nnz = [1, 3, 2, 1] > > A = petsc.Mat() > A.create(petsc.COMM_WORLD) > A.setSizes([M, N]) > A.setType('aij') > > i_start, i_end = A.getOwnershipRange() > > A.setPreallocationNNZ([global_d_nnz[i_start:i_end], > global_o_nnz[i_start:i_end]]) # error occurs here > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Aug 13 17:23:52 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 13 Aug 2011 17:23:52 -0500 Subject: [petsc-users] Ordering of preallocation and OwnershipRange In-Reply-To: References: Message-ID: <427145D9-1733-4E7F-9E80-9EE859F81CB6@mcs.anl.gov> The PetscLayout object is used to manage default layout among processes so you can create a PetscLayout object and use that to determine how the Mat and Vec objects will be laid out. From the manual page PetscLayoutCreate - Allocates PetscLayout space and sets the map contents to the default. Collective on MPI_Comm Input Parameters: + comm - the MPI communicator - map - pointer to the map Level: developer Notes: Typical calling sequence PetscLayoutCreate(MPI_Comm,PetscLayout *); PetscLayoutSetBlockSize(PetscLayout,1); PetscLayoutSetSize(PetscLayout,n) or PetscLayoutSetLocalSize(PetscLayout,N); PetscLayoutSetUp(PetscLayout); PetscLayoutGetSize(PetscLayout,PetscInt *); or PetscLayoutGetLocalSize(PetscLayout,PetscInt *;) PetscLayoutDestroy(PetscLayout); The PetscLayout object and methods are intended to be used in the PETSc Vec and Mat implementions; it is recommended they not be used in user codes unless you really gain something in their use. Fortran Notes: Not available from Fortran On Aug 13, 2011, at 10:28 AM, Matthew Knepley wrote: > On Sat, Aug 13, 2011 at 2:57 PM, Josh Hykes wrote: > Hello, > > I'm just starting to experiment with PETSc (v3.1), and I like the Python bindings provided by petsc4py (v1.1.2). So far things seem fairly straightforward, but I'm stumped on a small issue. > > While creating a parallel AIJ matrix, I'd like to preallocate it using arrays d_nnz and o_nnz. As I understand it, these arrays correspond to the processor's local rows. > > Currently I specify the global matrix size, and let PETSc decide on the decomposition of the rows. I'd like to ask PETSc what rows each processor has with the getOwnershipRange() function, and then do the preallocation. However, according to the error message > > > [1] MatAnyAIJSetPreallocation() line 393 in petsc4py-1.1.2/src/include/custom.h > > [1] Operation done in wrong order > > [1] matrix is already preallocated > > I'm not allowed to do it in this order. > > Thus, my question is: is it possible to let PETSc figure out the row decomposition while still using d_nnz and o_nnz for the preallocation? I figure that I could resolve the problem by doing my own decomposition, but it'd be nice if I could let those details up to PETSc. > > You are correct. We require that preallocation is done at the same time as decomposition. There > are tricky dependencies in matrix creation. However, an easy workaround is to create a Vec at > the same time with the same global size, since it is guaranteed to have the same layout. I will look > into simplifying this if it is possible. > > Thanks, > > Matt > > I'm including an example using petsc4py of what I'd like to do, run with 2 MPI processes. > > I apologize if this is a dumb question. Thank you for your help. > > -Josh > > # ----------------------------------------------- > from petsc4py import PETSc as petsc > > M, N = 4, 6 > > global_d_nnz = [2, 1, 1, 2] > global_o_nnz = [1, 3, 2, 1] > > A = petsc.Mat() > A.create(petsc.COMM_WORLD) > A.setSizes([M, N]) > A.setType('aij') > > i_start, i_end = A.getOwnershipRange() > > A.setPreallocationNNZ([global_d_nnz[i_start:i_end], > global_o_nnz[i_start:i_end]]) # error occurs here > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From paul.anton.letnes at gmail.com Sun Aug 14 06:18:11 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Sun, 14 Aug 2011 12:18:11 +0100 Subject: [petsc-users] BiCGSTAB for general use In-Reply-To: References: <83F76379-F270-48A3-B498-C69934D83F82@gmail.com> Message-ID: <5236C63B-633D-4048-8EB6-5BDB9A5BE339@gmail.com> On 12. aug. 2011, at 21.54, Jed Brown wrote: > On Fri, Aug 12, 2011 at 12:26, Paul Anton Letnes wrote: > I'm not 100% sure what you mean by "second kind integral operator", but it is a Fredholm equation of the first kind, as far as I understand (my background is physics rather than mathematics). > > Is the thing you're trying to solve with actually of the first kind, not of the second kind? I believe of the first kind - there is. Our approach is to discretize the integral equation. The equations we are "really solving" are the Maxwell equations. > http://en.wikipedia.org/wiki/Fredholm_integral_equation > > The distinction is whether there is essentially in whether there is a local part to the equation or not. The issue, as I understand it, is that solving a first-kind integral equation is generally not a stable process because the eigenvalues of the integral operator decay to zero, implying that it is essentially low rank, thus not invertible. Maybe you use some regularization to get a system that is not essentially singular? I have downloaded and attempted to use a different BiCGSTAB code. It converges, but only after several hundred (about 400) for a very small (not physically interesting) problem. It would appear that if we are to get good performance, some form of preconditioning is necessary. So far, we have relied on direct LU factorization (LAPACK/MKL routines) for solving the equation system. In this case, no preconditioning is needed. I have also investigated the singular values of the matrix - the condition number seems to be decent, if not great (if I recall correctly, less than 1000). Paul. From jedbrown at mcs.anl.gov Sun Aug 14 10:42:19 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 14 Aug 2011 10:42:19 -0500 Subject: [petsc-users] BiCGSTAB for general use In-Reply-To: <5236C63B-633D-4048-8EB6-5BDB9A5BE339@gmail.com> References: <83F76379-F270-48A3-B498-C69934D83F82@gmail.com> <5236C63B-633D-4048-8EB6-5BDB9A5BE339@gmail.com> Message-ID: On Sun, Aug 14, 2011 at 06:18, Paul Anton Letnes < paul.anton.letnes at gmail.com> wrote: > I believe of the first kind - there is. Our approach is to discretize the > integral equation. The equations we are "really solving" are the Maxwell > equations. > Is this the sort of system you're working with? http://link.aps.org/doi/10.1103/PhysRevLett.104.223904 Note that the system has the form J_H(x_\parallel | \omega) = J_H(x_\parallel | \omega)_{inc} + \int (...) G(x | x') J_H(x_\parallel' | \omega) + \int (...) G(x | x') J_E(x_\parallel' | \omega) which is the form of a second order integral equation. I assume the incident field J_H(...)_{inc} is known in this equation. If you dropped the term on the left hand side in this equation, you would have a Fredholm integral equation of the first kind to "solve", which is problematic at a mathematical level due to ill-posedness. > > I have downloaded and attempted to use a different BiCGSTAB code. It > converges, but only after several hundred (about 400) for a very small (not > physically interesting) problem. It would appear that if we are to get good > performance, some form of preconditioning is necessary. > Do the eigenvalues decay quickly? Can you plot some eigenvalues? They should decay rapidly to a positive value like (with appropriate scaling) 1. -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.anton.letnes at gmail.com Sun Aug 14 11:33:36 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Sun, 14 Aug 2011 17:33:36 +0100 Subject: [petsc-users] BiCGSTAB for general use In-Reply-To: References: <83F76379-F270-48A3-B498-C69934D83F82@gmail.com> <5236C63B-633D-4048-8EB6-5BDB9A5BE339@gmail.com> Message-ID: <8110936E-C186-4DC4-879B-912CBC849E7F@gmail.com> On 14. aug. 2011, at 16.42, Jed Brown wrote: > On Sun, Aug 14, 2011 at 06:18, Paul Anton Letnes wrote: > I believe of the first kind - there is. Our approach is to discretize the integral equation. The equations we are "really solving" are the Maxwell equations. > > Is this the sort of system you're working with? > > http://link.aps.org/doi/10.1103/PhysRevLett.104.223904 Ingve is my supervisor - so it's a good guess! But, no. BiCGSTAB works well for his code, by the way. He gets a massive speedup compared to direct solvers. I am working on a different equation called the Reduced Rayleigh Equation (RRE) for 2 dimensions, which has not been solved directly before (AFAIK). Have a look here: http://www.tandfonline.com/doi/abs/10.1088/0959-7174/6/3/006 The form of the solution is given in Eq. 5, and the equation itself is given in Eq. 8 with some definitions in Eq. 9-11. As far as I can tell there is only 2 terms, so it's a first kind equation. But by all means, feel free to correct me on that one. The cited paper uses a completely different approach (perturbation theory where 0. order is the flat surface) to solving the RRE, by the way. Our approach is more "brute force" - discretize the integral on a 2D grid in \vec{q}_\parallel space and solve the linear equation system. The idea is that the RRE is less memory intensive than the rigorous approach in the paper you cited. However, if we can't avoid the LU factorization, the RRE approach will be much more CPU intensive, and of lesser interest than we would have hoped... We have submitted a first paper to ArXiv; it will appear on monday, I think. > Note that the system has the form > > J_H(x_\parallel | \omega) = J_H(x_\parallel | \omega)_{inc} + \int (...) G(x | x') J_H(x_\parallel' | \omega) + \int (...) G(x | x') J_E(x_\parallel' | \omega) > > which is the form of a second order integral equation. I assume the incident field J_H(...)_{inc} is known in this equation. If you dropped the term on the left hand side in this equation, you would have a Fredholm integral equation of the first kind to "solve", which is problematic at a mathematical level due to ill-posedness. But since it's of the second kind, it's better posed, right? > I have downloaded and attempted to use a different BiCGSTAB code. It converges, but only after several hundred (about 400) for a very small (not physically interesting) problem. It would appear that if we are to get good performance, some form of preconditioning is necessary. > > Do the eigenvalues decay quickly? Can you plot some eigenvalues? They should decay rapidly to a positive value like (with appropriate scaling) 1. I'll have a look at the eigenvalues. Is your suggestion to plot the magnitude of the complex eigenvectors as a function of their index, after sorting them? I guess that must be what you mean by "plotting". cheers Paul From gokhalen at gmail.com Sun Aug 14 14:21:23 2011 From: gokhalen at gmail.com (Nachiket Gokhale) Date: Sun, 14 Aug 2011 15:21:23 -0400 Subject: [petsc-users] targetting specific eigenvectors Message-ID: I am solving a coupled structural-acoustic calculation, and I want to find the most important "coupled modes", i.e. the modes which transfer the most energy between the structure and the fluid. One way to do this would be to solve the first N modes of the full generalized eigenproblem (Kx=Mx), and compute a measure of the coupling ( something like \pi = u'Sp ), where S is the coupling matrix, u is the displacement and p is there pressure. One could then sort the coupled modes according to \pi. However, important coupled modes which do not lie in the first "N" modes may not be found (unless there are matrix structuring results that I am not aware of). Are there any algorithms that guarantee that find first N_c important coupled modes as defined by an user defined criterion, and are there any code s that implement them? The only reference I could find was Alan R. Tackett, Massimiliano Di Ventra, Targeting specific eigenvectors and eigenvalues of a given Hamiltonian using arbitrary selection criteria, PHYSICAL REVIEW B 66, 245104 2002. Sorry for the slightly OT discussion. Thanks, Nachiket From jedbrown at mcs.anl.gov Sun Aug 14 17:22:18 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 14 Aug 2011 17:22:18 -0500 Subject: [petsc-users] BiCGSTAB for general use In-Reply-To: <8110936E-C186-4DC4-879B-912CBC849E7F@gmail.com> References: <83F76379-F270-48A3-B498-C69934D83F82@gmail.com> <5236C63B-633D-4048-8EB6-5BDB9A5BE339@gmail.com> <8110936E-C186-4DC4-879B-912CBC849E7F@gmail.com> Message-ID: On Sun, Aug 14, 2011 at 11:33, Paul Anton Letnes < paul.anton.letnes at gmail.com> wrote: > Ingve is my supervisor - so it's a good guess! But, no. BiCGSTAB works well > for his code, by the way. He gets a massive speedup compared to direct > solvers. > Heh, okay. So it works when the problem is of the kind I've seen before and the theory I'm familiar with says it should work. Comforting perhaps, but doesn't solve your problem. > > I am working on a different equation called the Reduced Rayleigh Equation > (RRE) for 2 dimensions, which has not been solved directly before (AFAIK). > Have a look here: > http://www.tandfonline.com/doi/abs/10.1088/0959-7174/6/3/006 > The form of the solution is given in Eq. 5, and the equation itself is > given in Eq. 8 with some definitions in Eq. 9-11. As far as I can tell there > is only 2 terms, so it's a first kind equation. But by all means, feel free > to correct me on that one. The cited paper uses a completely different > approach (perturbation theory where 0. order is the flat surface) to solving > the RRE, by the way. Our approach is more "brute force" - discretize the > integral on a 2D grid in \vec{q}_\parallel space and solve the linear > equation system. > Hmm, I would have to understand the spectral properties of this kernel better. Is involves products instead of differences (like the more conventional Green's function you showed earlier) and is clearly non-compact, so my earlier comment about trying to invert a compact operator does not apply. But you also likely won't get any nice decay in the spectrum with which to use unpreconditioned Krylov methods. > > The idea is that the RRE is less memory intensive than the rigorous > approach in the paper you cited. However, if we can't avoid the LU > factorization, the RRE approach will be much more CPU intensive, and of > lesser interest than we would have hoped... > > We have submitted a first paper to ArXiv; it will appear on monday, I > think. > > > Note that the system has the form > > > > J_H(x_\parallel | \omega) = J_H(x_\parallel | \omega)_{inc} + \int (...) > G(x | x') J_H(x_\parallel' | \omega) + \int (...) G(x | x') J_E(x_\parallel' > | \omega) > > > > which is the form of a second order integral equation. I assume the > incident field J_H(...)_{inc} is known in this equation. If you dropped the > term on the left hand side in this equation, you would have a Fredholm > integral equation of the first kind to "solve", which is problematic at a > mathematical level due to ill-posedness. > > But since it's of the second kind, it's better posed, right? > Right, because the eigenvalues decay rapidly to 1 instead of decaying rapidly to 0. > > > I have downloaded and attempted to use a different BiCGSTAB code. It > converges, but only after several hundred (about 400) for a very small (not > physically interesting) problem. It would appear that if we are to get good > performance, some form of preconditioning is necessary. > > > > Do the eigenvalues decay quickly? Can you plot some eigenvalues? They > should decay rapidly to a positive value like (with appropriate scaling) 1. > > I'll have a look at the eigenvalues. Is your suggestion to plot the > magnitude of the complex eigenvectors as a function of their index, after > sorting them? I guess that must be what you mean by "plotting". > Let's start with a scatter plot of the eigenvalues. Can you do a problem that is representative of the physics in less than, say, 1000 degrees of freedom? If so, I would just use Matlab (or Octave, etc). You want to be able to plot the eigenvector associated with a chosen eigenvalue in some way that is meaningful to you. We want to see if the wavelength (in terms of the variables you are discretizing over) of the modes has some useful correlation with the size of the associated eigenvalues. If so, we may be able to build some sort of multigrid preconditioner. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdiso at ustc.edu Mon Aug 15 03:13:39 2011 From: gdiso at ustc.edu (Gong Ding) Date: Mon, 15 Aug 2011 16:13:39 +0800 (CST) Subject: [petsc-users] Any beautiful way to calculate b=Ax-r Message-ID: <27347038.174821313396019621.JavaMail.coremail@mail.ustc.edu> Hi, I have a linear system f(x)=Ax-b=r For solving it, I must build rhs vector b=Ax-r. However, I can't find some direct API to do this. MatMultAdd only support b=Ax+r. At preset, I am using VecSet(L, -1.0); VecPointwiseMult(r, r, L); // r <- -r MatMultAdd(A, x, r, b); Any better method? Gong Ding From jroman at dsic.upv.es Mon Aug 15 05:46:26 2011 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 15 Aug 2011 12:46:26 +0200 Subject: [petsc-users] targetting specific eigenvectors In-Reply-To: References: Message-ID: <21F535A1-590F-4D35-B456-DBD6256FB40A@dsic.upv.es> El 14/08/2011, a las 21:21, Nachiket Gokhale escribi?: > I am solving a coupled structural-acoustic calculation, and I want to > find the most important "coupled modes", i.e. the modes which transfer > the most energy between the structure and the fluid. One way to do > this would be to solve the first N modes of the full generalized > eigenproblem (Kx=Mx), and compute a measure of the coupling ( > something like \pi = u'Sp ), where S is the coupling matrix, u is the > displacement and p is there pressure. One could then sort the coupled > modes according to \pi. However, important coupled modes which do not > lie in the first "N" modes may not be found (unless there are matrix > structuring results that I am not aware of). > > Are there any algorithms that guarantee that find first N_c important > coupled modes as defined by an user defined criterion, and are there > any code s that implement them? > > The only reference I could find was > > Alan R. Tackett, Massimiliano Di Ventra, Targeting specific > eigenvectors and eigenvalues of a given Hamiltonian using arbitrary > selection criteria, PHYSICAL REVIEW B 66, 245104 2002. > > Sorry for the slightly OT discussion. > > Thanks, > > Nachiket In SLEPc 3.1 there is the possibility of setting a user-defined selection criterion with EPSSetEigenvalueComparison. However, currently this only allows for criteria based on the eigenvalues and not on the eigenvectors. We will see if it is feasible to add this possibility for the next version. Jose From abhyshr at mcs.anl.gov Mon Aug 15 08:20:11 2011 From: abhyshr at mcs.anl.gov (Shrirang Abhyankar) Date: Mon, 15 Aug 2011 08:20:11 -0500 (CDT) Subject: [petsc-users] Any beautiful way to calculate b=Ax-r In-Reply-To: <27347038.174821313396019621.JavaMail.coremail@mail.ustc.edu> References: <27347038.174821313396019621.JavaMail.coremail@mail.ustc.edu> Message-ID: <33A77BD8-17D7-49AB-896F-FEF7CA977080@mcs.anl.gov> There is no API for Ax - r, you can do MatMult(A,x,b) VecAXPY(b,-1.0,r) On Aug 15, 2011, at 3:14 AM, "Gong Ding" wrote: > Hi, > I have a linear system f(x)=Ax-b=r > For solving it, I must build rhs vector b=Ax-r. > However, I can't find some direct API to do this. > MatMultAdd only support b=Ax+r. > > At preset, I am using > VecSet(L, -1.0); > VecPointwiseMult(r, r, L); // r <- -r > MatMultAdd(A, x, r, b); > > Any better method? > > Gong Ding > > > > > > > From dominik at itis.ethz.ch Wed Aug 17 11:55:00 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Wed, 17 Aug 2011 18:55:00 +0200 Subject: [petsc-users] valgrind error Message-ID: Hi, I am getting the following valgrind complaint: ==27366== Syscall param writev(vector[...]) points to uninitialised byte(s) ==27366== at 0x6B5E789: writev (writev.c:56) ==27366== by 0x5081F8: MPIDU_Sock_writev (sock_immed.i:610) ==27366== by 0x50A4E3: MPIDI_CH3_iSendv (ch3_isendv.c:84) ==27366== by 0x4EE2F6: MPIDI_CH3_EagerContigIsend (ch3u_eager.c:509) ==27366== by 0x4F0284: MPID_Isend (mpid_isend.c:118) ==27366== by 0x4D7BCA: MPIC_Isend (helper_fns.c:210) ==27366== by 0xEC3ECD: MPIR_Alltoall (alltoall.c:420) ==27366== by 0xEC4730: PMPI_Alltoall (alltoall.c:685) ==27366== by 0xE38B3D: SetUp__ (setup.c:122) ==27366== by 0xE3938C: PartitionSmallGraph__ (weird.c:39) ==27366== by 0xE36678: ParMETIS_V3_PartKway (kmetis.c:131) ==27366== by 0x6A0004: MatPartitioningApply_Parmetis (pmetis.c:97) ==27366== by 0x69C600: MatPartitioningApply (partition.c:236) ==27366== by 0x519266: PetscSolver::LoadMesh(std::string const&) (PetscSolver.cxx:415) ==27366== by 0x4C1801: CD3T10_BOX::CD3T10_BOX() (cd3t10mpi_main.cxx:64) ==27366== by 0x4C1318: main (cd3t10mpi_main.cxx:673) ==27366== Address 0xc635afc is 12 bytes inside a block of size 72 alloc'd ==27366== at 0x4C28FAC: malloc (vg_replace_malloc.c:236) ==27366== by 0xE4963A: GKmalloc__ (util.c:151) ==27366== by 0xE44AB1: PreAllocateMemory__ (memory.c:38) ==27366== by 0xE3655B: ParMETIS_V3_PartKway (kmetis.c:116) ==27366== by 0x6A0004: MatPartitioningApply_Parmetis (pmetis.c:97) ==27366== by 0x69C600: MatPartitioningApply (partition.c:236) ==27366== by 0x519266: PetscSolver::LoadMesh(std::string const&) (PetscSolver.cxx:415) ==27366== by 0x4C1801: CD3T10_BOX::CD3T10_BOX() (cd3t10mpi_main.cxx:64) ==27366== by 0x4C1318: main (cd3t10mpi_main.cxx:673) A few lines around PetscSolver.cxx:415: MatPartitioning part; ierr = MatPartitioningCreate(PETSC_COMM_WORLD, &part); CHKERRQ(ierr); ierr = MatPartitioningSetAdjacency(part, adj); CHKERRQ(ierr); ierr = MatPartitioningSetNParts(part, np); CHKERRQ(ierr); ierr = MatPartitioningSetFromOptions(part); CHKERRQ(ierr); IS isTetAssignment; 415 ierr = MatPartitioningApply(part, &isTetAssignment); CHKERRQ(ierr); ierr = MatPartitioningDestroy(part); CHKERRQ(ierr); ierr = PetscGetTime(&time1); CHKERRQ(ierr); I do not immediately see anything wrong. Is this a real error? Did I overlook anything? Many thanks for any opinions. Dominik From knepley at gmail.com Wed Aug 17 12:13:19 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 17 Aug 2011 19:13:19 +0200 Subject: [petsc-users] valgrind error In-Reply-To: References: Message-ID: On Wed, Aug 17, 2011 at 6:55 PM, Dominik Szczerba wrote: > Hi, > > I am getting the following valgrind complaint: > > ==27366== Syscall param writev(vector[...]) points to uninitialised byte(s) > ==27366== at 0x6B5E789: writev (writev.c:56) > ==27366== by 0x5081F8: MPIDU_Sock_writev (sock_immed.i:610) > ==27366== by 0x50A4E3: MPIDI_CH3_iSendv (ch3_isendv.c:84) > ==27366== by 0x4EE2F6: MPIDI_CH3_EagerContigIsend (ch3u_eager.c:509) > ==27366== by 0x4F0284: MPID_Isend (mpid_isend.c:118) > ==27366== by 0x4D7BCA: MPIC_Isend (helper_fns.c:210) > ==27366== by 0xEC3ECD: MPIR_Alltoall (alltoall.c:420) > ==27366== by 0xEC4730: PMPI_Alltoall (alltoall.c:685) > ==27366== by 0xE38B3D: SetUp__ (setup.c:122) > ==27366== by 0xE3938C: PartitionSmallGraph__ (weird.c:39) > ==27366== by 0xE36678: ParMETIS_V3_PartKway (kmetis.c:131) > ==27366== by 0x6A0004: MatPartitioningApply_Parmetis (pmetis.c:97) > ==27366== by 0x69C600: MatPartitioningApply (partition.c:236) > ==27366== by 0x519266: PetscSolver::LoadMesh(std::string const&) > (PetscSolver.cxx:415) > ==27366== by 0x4C1801: CD3T10_BOX::CD3T10_BOX() (cd3t10mpi_main.cxx:64) > ==27366== by 0x4C1318: main (cd3t10mpi_main.cxx:673) > ==27366== Address 0xc635afc is 12 bytes inside a block of size 72 alloc'd > ==27366== at 0x4C28FAC: malloc (vg_replace_malloc.c:236) > ==27366== by 0xE4963A: GKmalloc__ (util.c:151) > ==27366== by 0xE44AB1: PreAllocateMemory__ (memory.c:38) > ==27366== by 0xE3655B: ParMETIS_V3_PartKway (kmetis.c:116) > ==27366== by 0x6A0004: MatPartitioningApply_Parmetis (pmetis.c:97) > ==27366== by 0x69C600: MatPartitioningApply (partition.c:236) > ==27366== by 0x519266: PetscSolver::LoadMesh(std::string const&) > (PetscSolver.cxx:415) > ==27366== by 0x4C1801: CD3T10_BOX::CD3T10_BOX() (cd3t10mpi_main.cxx:64) > ==27366== by 0x4C1318: main (cd3t10mpi_main.cxx:673) > > > A few lines around PetscSolver.cxx:415: > > > MatPartitioning part; > ierr = MatPartitioningCreate(PETSC_COMM_WORLD, &part); > CHKERRQ(ierr); > ierr = MatPartitioningSetAdjacency(part, adj); CHKERRQ(ierr); > ierr = MatPartitioningSetNParts(part, np); CHKERRQ(ierr); > ierr = MatPartitioningSetFromOptions(part); CHKERRQ(ierr); > IS isTetAssignment; > 415 ierr = MatPartitioningApply(part, &isTetAssignment); CHKERRQ(ierr); > ierr = MatPartitioningDestroy(part); CHKERRQ(ierr); > ierr = PetscGetTime(&time1); CHKERRQ(ierr); > > > > I do not immediately see anything wrong. Is this a real error? Did I > overlook anything? > I think this is fine (its a common complaint of valgrind about MPICH). Matt > Many thanks for any opinions. > > Dominik -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.anton.letnes at gmail.com Wed Aug 17 12:48:50 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Wed, 17 Aug 2011 18:48:50 +0100 Subject: [petsc-users] BiCGSTAB for general use In-Reply-To: References: <83F76379-F270-48A3-B498-C69934D83F82@gmail.com> <5236C63B-633D-4048-8EB6-5BDB9A5BE339@gmail.com> <8110936E-C186-4DC4-879B-912CBC849E7F@gmail.com> Message-ID: > Let's start with a scatter plot of the eigenvalues. Can you do a problem that is representative of the physics in less than, say, 1000 degrees of freedom? If so, I would just use Matlab (or Octave, etc). You want to be able to plot the eigenvector associated with a chosen eigenvalue in some way that is meaningful to you. We want to see if the wavelength (in terms of the variables you are discretizing over) of the modes has some useful correlation with the size of the associated eigenvalues. If so, we may be able to build some sort of multigrid preconditioner. 1000 degrees of freedom is a bit little. I took a 4608 x 4608 matrix and plotted its eigenvalues as a scatterplot in the complex plane, as well as the magnitude of the eigenvalues. See attachments. I can give out better quality plots off-list. I'm really not sure as to how one can visualize eigenvectors with 4608 elements... I have not thought too much about the physical meaning of eigenvectors and eigenvalues. In fact, even this system is too small to be of physical interest, so I'm not sure what I'd get out of it, to be honest. I suppose some eigenvectors might be related to surface plasmon polaritons, and one eigenvector is probably related to the specular ("mirror-like") peak. Hope this is of any relevance. Paul -------------- next part -------------- A non-text attachment was scrubbed... Name: eigenvalues-scatter.png Type: image/png Size: 19875 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: eigenvalues-sorted.png Type: image/png Size: 5201 bytes Desc: not available URL: From dominik at itis.ethz.ch Wed Aug 17 13:08:28 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Wed, 17 Aug 2011 20:08:28 +0200 Subject: [petsc-users] problem with -malloc=0 Message-ID: I am using -malloc=0 (as per the pdf documentation) to see: WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-malloc=0 no value What am I doing wrong? Many thanks, Dominik From bsmith at mcs.anl.gov Wed Aug 17 13:19:51 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 17 Aug 2011 13:19:51 -0500 Subject: [petsc-users] problem with -malloc=0 In-Reply-To: References: Message-ID: -malloc no The command line options in PETSc never take = in them Barry On Aug 17, 2011, at 1:08 PM, Dominik Szczerba wrote: > I am using -malloc=0 (as per the pdf documentation) to see: > > WARNING! There are options you set that were not used! > WARNING! could be spelling mistake, etc! > Option left: name:-malloc=0 no value > > What am I doing wrong? > > Many thanks, > Dominik From dominik at itis.ethz.ch Wed Aug 17 15:06:25 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Wed, 17 Aug 2011 22:06:25 +0200 Subject: [petsc-users] problem with -malloc=0 In-Reply-To: References: Message-ID: I tried that before too, also with off, the warning is still there: WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-malloc value: no This is not command line, this is in a string passed to PetscOptionsInsertString (but it should not matter, should it?) Any ideas? Thanks, Dominik On Wed, Aug 17, 2011 at 8:19 PM, Barry Smith wrote: > > ?-malloc no > > ?The command line options in PETSc never take = in them > > ? Barry > > On Aug 17, 2011, at 1:08 PM, Dominik Szczerba wrote: > >> I am using -malloc=0 (as per the pdf documentation) to see: >> >> WARNING! There are options you set that were not used! >> WARNING! could be spelling mistake, etc! >> Option left: name:-malloc=0 no value >> >> What am I doing wrong? >> >> Many thanks, >> Dominik > > From bsmith at mcs.anl.gov Wed Aug 17 15:10:46 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 17 Aug 2011 15:10:46 -0500 Subject: [petsc-users] problem with -malloc=0 In-Reply-To: References: Message-ID: <51A7FBB0-5543-4471-B50A-F54192D72A34@mcs.anl.gov> On Aug 17, 2011, at 3:06 PM, Dominik Szczerba wrote: > I tried that before too, also with off, the warning is still there: > > WARNING! There are options you set that were not used! > WARNING! could be spelling mistake, etc! > Option left: name:-malloc value: no > > This is not command line, this is in a string passed to > PetscOptionsInsertString (but it should not matter, should it?) Won't work. The problem is startup options like -malloc are processed by PetscInitalize(). Since you set it after PetscInitialize() it won't have any affect. You need either on the command line, in the environmental variable PETSC_OPTIONS or in a file .petscrc either in your home directory or in the working directory. Barry > > Any ideas? > > Thanks, Dominik > > On Wed, Aug 17, 2011 at 8:19 PM, Barry Smith wrote: >> >> -malloc no >> >> The command line options in PETSc never take = in them >> >> Barry >> >> On Aug 17, 2011, at 1:08 PM, Dominik Szczerba wrote: >> >>> I am using -malloc=0 (as per the pdf documentation) to see: >>> >>> WARNING! There are options you set that were not used! >>> WARNING! could be spelling mistake, etc! >>> Option left: name:-malloc=0 no value >>> >>> What am I doing wrong? >>> >>> Many thanks, >>> Dominik >> >> From knepley at gmail.com Wed Aug 17 15:10:52 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 17 Aug 2011 22:10:52 +0200 Subject: [petsc-users] problem with -malloc=0 In-Reply-To: References: Message-ID: On Wed, Aug 17, 2011 at 10:06 PM, Dominik Szczerba wrote: > I tried that before too, also with off, the warning is still there: > > WARNING! There are options you set that were not used! > WARNING! could be spelling mistake, etc! > Option left: name:-malloc value: no > > This is not command line, this is in a string passed to > PetscOptionsInsertString (but it should not matter, should it?) > > Any ideas? > It matters when you are calling this. This is examined during initialization. If you provide it after this, it will come up as unexamined. Matt > Thanks, Dominik > > On Wed, Aug 17, 2011 at 8:19 PM, Barry Smith wrote: > > > > -malloc no > > > > The command line options in PETSc never take = in them > > > > Barry > > > > On Aug 17, 2011, at 1:08 PM, Dominik Szczerba wrote: > > > >> I am using -malloc=0 (as per the pdf documentation) to see: > >> > >> WARNING! There are options you set that were not used! > >> WARNING! could be spelling mistake, etc! > >> Option left: name:-malloc=0 no value > >> > >> What am I doing wrong? > >> > >> Many thanks, > >> Dominik > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From massing at simula.no Wed Aug 17 22:15:55 2011 From: massing at simula.no (Andre Massing) Date: Thu, 18 Aug 2011 05:15:55 +0200 Subject: [petsc-users] external libHYPRE is installed in lib64 and causes a configure error Message-ID: <4E4C83EB.3060401@simula.no> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dear all Today I wanted to install petsc-3.1-p8 on a openSUSE 11.4 (which I have done successfully in the past) but some reason I got a configure error =============================================================================== Configuring hypre; this may take several minutes =============================================================================== =============================================================================== Compiling hypre; this may take several minutes =============================================================================== TESTING: check from config.libraries(/home/andre/Work/software/FEniCS-dependencies/src/testdir/petsc-3.1-p8/config/BuildSystem/config/libraries.py:133) ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): - ------------------------------------------------------------------------------- Downloaded hypre could not be used. Please check install in /home/andre/Work/software/FEniCS-dependencies/src/testdir/petsc-3.1-p8/linux-gnu-cxx-opt when I executed export PARMETIS_DIR="/home/andre/Work/software/FEniCS-dependencies\/install" ./config/configure.py --with-debugging=0 --with-shared=1 \ --with-clanguage=cxx --with-parmetis=1 \ --with-parmetisdir=${PARMETIS_DIR} \ --download-umfpack=1 --download-hypre=1 The main reason seems to be that the libHYPRE.a for some reason is copied to linux-gnu-cxx-opt/lib64/ and not to linux-gnu-cxx-opt/lib According to the configure.log file Executing: mpicxx -o conftest -Wall -Wwrite-strings - -Wno-strict-aliasing -O conftest.o - -Wl,-rpath,/home/andre/Work/software/FEniCS-dependencies/src/testdir/petsc-3.1-p8/linux-gnu-cxx-opt/lib - -L/home/andre/Work/software/FEniCS-dependencies/src/testdir/petsc-3.1-p8/linux-gnu-cxx-opt/lib - -lHYPRE -llapack -lblas -Wl,-rpath,/usr/lib64/mpi/gcc/openmpi/lib64 - -L/usr/lib64/mpi/gcc/openmpi/lib64 - -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/4.5 - -L/usr/lib64/gcc/x86_64-suse-linux/4.5 - -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib - -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread - -lmpi_f90 -lmpi_f77 -lgfortran -lm -lm - -Wl,-rpath,/usr/lib64/mpi/gcc/openmpi/lib64 - -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/4.5 - -Wl,-rpath,/usr/x86_64-suse-linux/lib -lm -lm - -Wl,-rpath,/usr/lib64/mpi/gcc/openmpi/lib64 - -L/usr/lib64/mpi/gcc/openmpi/lib64 - -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/4.5 - -L/usr/lib64/gcc/x86_64-suse-linux/4.5 - -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib - -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl sh: Possible ERROR while running linker: /usr/lib64/gcc/x86_64-suse-linux/4.5/../../../../x86_64-suse-linux/bin/ld: cannot find -lHYPRE collect2: ld returned 1 exit status the lib64 location of libHYPRE.a causes a linking error since the above command does only include - -L/home/andre/Work/software/FEniCS-dependencies/src/testdir/petsc-3.1-p8/linux-gnu-cxx-opt/lib Do you have any idea what could cause that location mismatch? Is the libHYPRE.a location wrong or does the mpicxx command above omit a needed library path? This error is kind of strange since I have an old installation (same petsc version, same OS) where everything went through... Any pointers are very much appreciated!! Cheers, Andre -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJOTIPrAAoJEA79ggnbq9dmwDAIAJCM5SfUfh3I+qCr9fPG3pMW Spzc+hrJNQyZegIZIqR1RgHwCue0Fgkh2vmvglGmzYsfMiHl5CI88qBYxnxSF0wH imvZoXXeLqpy1pmPn2RtB4gRaMsarx1AMpnsVn0A6aIPEkn07rfJOEf19egSxO/r Yr+fnY2a4yXPrC4dfzgFjdsGHWSXCabzMwznQQldhQ+kF7bQ5V5InPPTYal6qSLm /0sz7ZwAsyuL+29vj8+8aD4uVN/yVEDIGJtNg5RburBK4FDqEcZOsrQMjZGNB/6B iPSJoQ8XK5b6yKqlzCmU9Qu9i/yxvKezUUTYMpGRK6Gd5mBEQSWOYz0jdz/CfYo= =1QvU -----END PGP SIGNATURE----- From bsmith at mcs.anl.gov Wed Aug 17 22:24:18 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 17 Aug 2011 22:24:18 -0500 Subject: [petsc-users] external libHYPRE is installed in lib64 and causes a configure error In-Reply-To: <4E4C83EB.3060401@simula.no> References: <4E4C83EB.3060401@simula.no> Message-ID: Send configure.log to petsc-maint at mcs.anl.gov On Aug 17, 2011, at 10:15 PM, Andre Massing wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Dear all > > Today I wanted to install petsc-3.1-p8 on a openSUSE 11.4 (which I > have done successfully in the past) but some reason I got a configure > error > > > =============================================================================== > Configuring hypre; this may take several minutes > > =============================================================================== > > > =============================================================================== > Compiling hypre; this may take several minutes > > =============================================================================== > > TESTING: check from > config.libraries(/home/andre/Work/software/FEniCS-dependencies/src/testdir/petsc-3.1-p8/config/BuildSystem/config/libraries.py:133) > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > for details): > - > ------------------------------------------------------------------------------- > Downloaded hypre could not be used. Please check install in > /home/andre/Work/software/FEniCS-dependencies/src/testdir/petsc-3.1-p8/linux-gnu-cxx-opt > > when I executed > export > PARMETIS_DIR="/home/andre/Work/software/FEniCS-dependencies\/install" > ./config/configure.py --with-debugging=0 --with-shared=1 \ > --with-clanguage=cxx --with-parmetis=1 \ > --with-parmetisdir=${PARMETIS_DIR} \ > --download-umfpack=1 --download-hypre=1 > > The main reason seems to be that the libHYPRE.a for some reason is > copied to > linux-gnu-cxx-opt/lib64/ > and not to > linux-gnu-cxx-opt/lib > > According to the configure.log file > > Executing: mpicxx -o conftest -Wall -Wwrite-strings > - -Wno-strict-aliasing -O conftest.o > - > -Wl,-rpath,/home/andre/Work/software/FEniCS-dependencies/src/testdir/petsc-3.1-p8/linux-gnu-cxx-opt/lib > - > -L/home/andre/Work/software/FEniCS-dependencies/src/testdir/petsc-3.1-p8/linux-gnu-cxx-opt/lib > - -lHYPRE -llapack -lblas -Wl,-rpath,/usr/lib64/mpi/gcc/openmpi/lib64 > - -L/usr/lib64/mpi/gcc/openmpi/lib64 > - -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/4.5 > - -L/usr/lib64/gcc/x86_64-suse-linux/4.5 > - -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib > - -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread > - -lmpi_f90 -lmpi_f77 -lgfortran -lm -lm > - -Wl,-rpath,/usr/lib64/mpi/gcc/openmpi/lib64 > - -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/4.5 > - -Wl,-rpath,/usr/x86_64-suse-linux/lib -lm -lm > - -Wl,-rpath,/usr/lib64/mpi/gcc/openmpi/lib64 > - -L/usr/lib64/mpi/gcc/openmpi/lib64 > - -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/4.5 > - -L/usr/lib64/gcc/x86_64-suse-linux/4.5 > - -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib > - -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl > sh: > Possible ERROR while running linker: > /usr/lib64/gcc/x86_64-suse-linux/4.5/../../../../x86_64-suse-linux/bin/ld: > cannot find -lHYPRE > collect2: ld returned 1 exit status > > > the lib64 location of libHYPRE.a causes a linking error since the > above command does only include > - > -L/home/andre/Work/software/FEniCS-dependencies/src/testdir/petsc-3.1-p8/linux-gnu-cxx-opt/lib > > Do you have any idea what could cause that location mismatch? Is the > libHYPRE.a location wrong or does the mpicxx command above omit a > needed library path? This error is kind of strange since I have an old > installation (same petsc version, same OS) where everything went > through... > > Any pointers are very much appreciated!! > > Cheers, > Andre > > > > > > > > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.0.18 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iQEcBAEBAgAGBQJOTIPrAAoJEA79ggnbq9dmwDAIAJCM5SfUfh3I+qCr9fPG3pMW > Spzc+hrJNQyZegIZIqR1RgHwCue0Fgkh2vmvglGmzYsfMiHl5CI88qBYxnxSF0wH > imvZoXXeLqpy1pmPn2RtB4gRaMsarx1AMpnsVn0A6aIPEkn07rfJOEf19egSxO/r > Yr+fnY2a4yXPrC4dfzgFjdsGHWSXCabzMwznQQldhQ+kF7bQ5V5InPPTYal6qSLm > /0sz7ZwAsyuL+29vj8+8aD4uVN/yVEDIGJtNg5RburBK4FDqEcZOsrQMjZGNB/6B > iPSJoQ8XK5b6yKqlzCmU9Qu9i/yxvKezUUTYMpGRK6Gd5mBEQSWOYz0jdz/CfYo= > =1QvU > -----END PGP SIGNATURE----- From pfeiffer at cita.utoronto.ca Thu Aug 18 02:49:36 2011 From: pfeiffer at cita.utoronto.ca (Harald Pfeiffer) Date: Thu, 18 Aug 2011 09:49:36 +0200 Subject: [petsc-users] roundoff differences In-Reply-To: <4E4A7477.1020408@wfu.edu> References: <4E4A7477.1020408@wfu.edu> Message-ID: <4E4CC410.4000407@cita.utoronto.ca> Hello, we use PETSc to solve the nonlinear system arising from pseudo-spectral discretization of certain elliptic PDEs in Einstein's equations. When running the same job multiple times on the same number of processors on the same workstation, we find roundoff differences. Is this expected, e.g. because MPI reduction calls may behave differently depending on the load of the machine? Or should we be concerned and investigate further? Thanks, Harald -------- Original Message -------- Subject: Re: Quick question about derivatives in SpEC Date: Tue, 16 Aug 2011 09:45:27 -0400 From: Gregory B. Cook To: Harald Pfeiffer CC: Larry Kidder , Mark Scheel Hi Harald, All of the tests I was doing were on the same 8 cores on my office workstation. It is running Ubuntu 11, and uses the default OpenMPI communication approach. To make sure it wasn't something I was doing, I ran two elliptic solves of the ExtendedConformalThinSandwich() volume terms. Here are the outputs of snes.dat for the different levels: Run 1 Run 2 Six0/snes.dat 0 7.3385297958166698 0 7.3385297958166698 1 5.1229060531500723 1 5.1229060531500723 2 0.32616852761238285 2 0.32616852761238285 3 0.012351417186533147 3 0.012351417186800266<***** 4 9.7478354935351385e-06 4 9.7478351511500114e-06 Six1/snes.dat 0 0.13405558402489681 0 0.13405558402540407 1 0.00068002100028642610 1 0.00068002089609322440 2 6.8764357250058596e-08 2 6.3738394418031232e-08 Six2/snes.dat 0 0.0063028244769771681 0 0.0063028058475922306 1 1.4538921141731714e-06 1 1.4545032695605256e-06 Six3/snes.dat 0 0.00061476105672438877 0 0.00061476093499534406 1 6.0267672358059814e-08 1 5.4897793428123648e-08 Six4/snes.dat 0 0.00053059501859595651 0 0.00053059591479892143 1 4.8003269489205705e-08 1 4.8079799390886591e-08 Six5/snes.dat 0 3.6402372419546429e-05 0 3.6402169997838670e-05 1 5.3117360561476420e-09 1 5.2732089856727503e-09 The differences are clearly at the level of roundoff, but it is "strange" that you cannot reproduce identical results. I've attached all of the .input files for this run in case you want to try to reproduce my findings. Greg On 08/16/2011 06:21 AM, Harald Pfeiffer wrote: > Hi Greg, > > some thoughts: > > Petsc is using standard MPI reduction calls, which may give results that > differ by roundoff. We have positively noticed this happening for > different number of processes, but perhaps this also happens depending > on where in a cluster your jobs run (different network topology > depending on whether all processors are on the same rack, vs. split > among racks; dynamic load-balancing of network communication). > > You might want to try reserving a few nodes interactively, and then > running the elliptic solver multiple times on this same set of nodes. > > The Mover does indeed load-balanced interpolation, but when doing so, > the MPI communication should not affect identical results. > > Once there are roundoff differences, they are typically amplified during > a petsc linear solve. The iterative algorithm takes different paths > toward the solution, and a difference of 1e-10 doesn't seem excessive. > > Harald > > ps. Preconditioning is done differently on-processor and off-processor > and depends therefore highly on the processor count. So if you were to > change number of processors, the iterative solve will proceed very > differently. > > On 8/15/11 10:27 PM, Gregory B. Cook wrote: >> Hi Larry, >> >> I ran a check using the default ExtendedConformalThinSandwich() volume >> terms and this also produced roundoff error differences between >> identical runs, so I feel better about that. I am using the same >> number of processors, but if there is any kind of dynamic load >> balancing for interpolation/communication/etc, then I can see that >> different runs might end up using different boundary communications. >> Maybe that's all there is to it? >> >> Greg >> >> On 08/15/2011 04:16 PM, Larry Kidder wrote: >>> Hi Greg, >>> >>> Harald is traveling, so I am not sure when he will answer. >>> My vague recollection is that there is something about how PETSc does >>> preconditioning in parallel that leads to not producing the same result; >>> but I don't recall if this happens in general, or only if you change the >>> distribution of processes. >>> >>> Larry >>> >>> Gregory B. Cook wrote: >>>> Hi Guys, >>>> >>>> I have a follow-up question that may be tangentially related to my >>>> original question about derivatives. This one is targeted at Harald. >>>> >>>> When I run a version of my code where the very small errors in the >>>> derivative of the metric are not present (I code them in differently), >>>> I find that running the exact same input files successively does not >>>> produce exactly the same results. This is a multi-level elliptic solve >>>> on a complex domain for binary black holes. On Level-0, the the >>>> results returned in snes.dat are identical. On Level-1, the initial >>>> and second snes norms are identical, but the third differs. After >>>> this, all snes norms differ. >>>> >>>> Is this to be expected? Does PETSc not produce identical results on >>>> consecutive solves with the same starting point? Is there something in >>>> the MPI communication that means that the results should differ? The >>>> differences start at the order of 10^-13, but grow by the 6th level to >>>> be of order 10^-10. >>>> >>>> Greg >>>> >>>> On 08/15/2011 01:02 PM, Larry Kidder wrote: >>>>> Hi Greg, >>>>> >>>>> Did you compute the norm of the metric itself? >>>>> What domain did you use? >>>>> >>>>> Larry >>>>> >>>>> Gregory B. Cook wrote: >>>>>> Hi Guys, >>>>>> >>>>>> I was doing a simple test as part of debugging some code I'm writing. >>>>>> I ended up placing the following relevant lines of code into the >>>>>> EllipticItems.input and EllipticObservers.input files: >>>>>> >>>>>> ---EllipticItems.input--- >>>>>> EvaluateMatrixFormula(Output=InvConformalMetric; Dim=3; Symm=11; >>>>>> M[0,0]=1; M[1,1]=1; M[2,2]=1), >>>>>> FirstDeriv(Input=InvConformalMetric; Output=dInvConformalMetric), >>>>>> SecondDeriv(Input=InvConformalMetric; Output=ddInvConformalMetric), >>>>>> >>>>>> FlattenDeriv(Input=dInvConformalMetric; >>>>>> Output=fdInvConformalMetric;DerivPosition=Last), >>>>>> FlattenDeriv(Input=ddInvConformalMetric; >>>>>> Output=fddInvConformalMetric;DerivPosition=Last), >>>>>> >>>>>> ---EllipticObservers.input--- >>>>>> NormOfTensor(Input=fdInvConformalMetric, fddInvConformalMetric; >>>>>> Filename=dInvCM_L2.dat;Op=L2; MetricForTensors=None), >>>>>> NormOfTensor(Input=fdInvConformalMetric, fddInvConformalMetric; >>>>>> Filename=dInvCM_Linf.dat;Op=Linf; MetricForTensors=None), >>>>>> >>>>>> >>>>>> The odd thing is that the norms that I get out are not exactly zero. >>>>>> They are very small, but I'm taking the first and second derivatives >>>>>> of the identity matrix, so I would expect them to evaluate to exactly >>>>>> zero. The fact that they don't leads me to think that there is >>>>>> something wrong either in my code or in how I have written the input >>>>>> files. >>>>>> >>>>>> Should these derivatives evaluate to exactly zero? >>>>>> >>>>>> Greg >>>>> >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Thu Aug 18 03:15:21 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Thu, 18 Aug 2011 10:15:21 +0200 Subject: [petsc-users] roundoff differences In-Reply-To: <4E4CC410.4000407@cita.utoronto.ca> References: <4E4A7477.1020408@wfu.edu> <4E4CC410.4000407@cita.utoronto.ca> Message-ID: Generally you do not expect digit-for-digit identical results between parallel runs, and yes, round off errors might result from different loading in-between. If a problem is ill-posed or poorly preconditioned this may even lead to divergence in some cases. Dominik On Thu, Aug 18, 2011 at 9:49 AM, Harald Pfeiffer wrote: > Hello, > > we use PETSc to solve the nonlinear system arising from pseudo-spectral > discretization of certain elliptic PDEs in Einstein's equations. When > running the same job multiple times on the same number of processors on the > same workstation, we find roundoff differences. Is this expected, e.g. > because MPI reduction calls may behave differently depending on the load of > the machine? Or should we be concerned and investigate further? > > Thanks, > Harald > > > > > > > > -------- Original Message -------- Subject: Re: Quick question about > derivatives in SpEC Date: Tue, 16 Aug 2011 09:45:27 -0400 From: Gregory > B. Cook To: Harald Pfeiffer > CC: Larry Kidder > , Mark Scheel > > > Hi Harald, > > All of the tests I was doing were on the same 8 cores on my office > workstation. It is running Ubuntu 11, and uses the default OpenMPI > communication approach. To make sure it wasn't something I was doing, I > ran two elliptic solves of the ExtendedConformalThinSandwich() volume > terms. Here are the outputs of snes.dat for the different levels: > > Run 1 Run 2 > Six0/snes.dat > 0 7.3385297958166698 0 7.3385297958166698 > 1 5.1229060531500723 1 5.1229060531500723 > 2 0.32616852761238285 2 0.32616852761238285 > 3 0.012351417186533147 3 0.012351417186800266 <***** > 4 9.7478354935351385e-06 4 9.7478351511500114e-06 > Six1/snes.dat > 0 0.13405558402489681 0 0.13405558402540407 > 1 0.00068002100028642610 1 0.00068002089609322440 > 2 6.8764357250058596e-08 2 6.3738394418031232e-08 > Six2/snes.dat > 0 0.0063028244769771681 0 0.0063028058475922306 > 1 1.4538921141731714e-06 1 1.4545032695605256e-06 > Six3/snes.dat > 0 0.00061476105672438877 0 0.00061476093499534406 > 1 6.0267672358059814e-08 1 5.4897793428123648e-08 > Six4/snes.dat > 0 0.00053059501859595651 0 0.00053059591479892143 > 1 4.8003269489205705e-08 1 4.8079799390886591e-08 > Six5/snes.dat > 0 3.6402372419546429e-05 0 3.6402169997838670e-05 > 1 5.3117360561476420e-09 1 5.2732089856727503e-09 > > The differences are clearly at the level of roundoff, but it is > "strange" that you cannot reproduce identical results. > > I've attached all of the .input files for this run in case you want to > try to reproduce my findings. > > Greg > > On 08/16/2011 06:21 AM, Harald Pfeiffer wrote: > > Hi Greg, > > > > some thoughts: > > > > Petsc is using standard MPI reduction calls, which may give results that > > differ by roundoff. We have positively noticed this happening for > > different number of processes, but perhaps this also happens depending > > on where in a cluster your jobs run (different network topology > > depending on whether all processors are on the same rack, vs. split > > among racks; dynamic load-balancing of network communication). > > > > You might want to try reserving a few nodes interactively, and then > > running the elliptic solver multiple times on this same set of nodes. > > > > The Mover does indeed load-balanced interpolation, but when doing so, > > the MPI communication should not affect identical results. > > > > Once there are roundoff differences, they are typically amplified during > > a petsc linear solve. The iterative algorithm takes different paths > > toward the solution, and a difference of 1e-10 doesn't seem excessive. > > > > Harald > > > > ps. Preconditioning is done differently on-processor and off-processor > > and depends therefore highly on the processor count. So if you were to > > change number of processors, the iterative solve will proceed very > > differently. > > > > On 8/15/11 10:27 PM, Gregory B. Cook wrote: > >> Hi Larry, > >> > >> I ran a check using the default ExtendedConformalThinSandwich() volume > >> terms and this also produced roundoff error differences between > >> identical runs, so I feel better about that. I am using the same > >> number of processors, but if there is any kind of dynamic load > >> balancing for interpolation/communication/etc, then I can see that > >> different runs might end up using different boundary communications. > >> Maybe that's all there is to it? > >> > >> Greg > >> > >> On 08/15/2011 04:16 PM, Larry Kidder wrote: > >>> Hi Greg, > >>> > >>> Harald is traveling, so I am not sure when he will answer. > >>> My vague recollection is that there is something about how PETSc does > >>> preconditioning in parallel that leads to not producing the same result; > >>> but I don't recall if this happens in general, or only if you change the > >>> distribution of processes. > >>> > >>> Larry > >>> > >>> Gregory B. Cook wrote: > >>>> Hi Guys, > >>>> > >>>> I have a follow-up question that may be tangentially related to my > >>>> original question about derivatives. This one is targeted at Harald. > >>>> > >>>> When I run a version of my code where the very small errors in the > >>>> derivative of the metric are not present (I code them in differently), > >>>> I find that running the exact same input files successively does not > >>>> produce exactly the same results. This is a multi-level elliptic solve > >>>> on a complex domain for binary black holes. On Level-0, the the > >>>> results returned in snes.dat are identical. On Level-1, the initial > >>>> and second snes norms are identical, but the third differs. After > >>>> this, all snes norms differ. > >>>> > >>>> Is this to be expected? Does PETSc not produce identical results on > >>>> consecutive solves with the same starting point? Is there something in > >>>> the MPI communication that means that the results should differ? The > >>>> differences start at the order of 10^-13, but grow by the 6th level to > >>>> be of order 10^-10. > >>>> > >>>> Greg > >>>> > >>>> On 08/15/2011 01:02 PM, Larry Kidder wrote: > >>>>> Hi Greg, > >>>>> > >>>>> Did you compute the norm of the metric itself? > >>>>> What domain did you use? > >>>>> > >>>>> Larry > >>>>> > >>>>> Gregory B. Cook wrote: > >>>>>> Hi Guys, > >>>>>> > >>>>>> I was doing a simple test as part of debugging some code I'm writing. > >>>>>> I ended up placing the following relevant lines of code into the > >>>>>> EllipticItems.input and EllipticObservers.input files: > >>>>>> > >>>>>> ---EllipticItems.input--- > >>>>>> EvaluateMatrixFormula(Output=InvConformalMetric; Dim=3; Symm=11; > >>>>>> M[0,0]=1; M[1,1]=1; M[2,2]=1), > >>>>>> FirstDeriv(Input=InvConformalMetric; Output=dInvConformalMetric), > >>>>>> SecondDeriv(Input=InvConformalMetric; Output=ddInvConformalMetric), > >>>>>> > >>>>>> FlattenDeriv(Input=dInvConformalMetric; > >>>>>> Output=fdInvConformalMetric;DerivPosition=Last), > >>>>>> FlattenDeriv(Input=ddInvConformalMetric; > >>>>>> Output=fddInvConformalMetric;DerivPosition=Last), > >>>>>> > >>>>>> ---EllipticObservers.input--- > >>>>>> NormOfTensor(Input=fdInvConformalMetric, fddInvConformalMetric; > >>>>>> Filename=dInvCM_L2.dat;Op=L2; MetricForTensors=None), > >>>>>> NormOfTensor(Input=fdInvConformalMetric, fddInvConformalMetric; > >>>>>> Filename=dInvCM_Linf.dat;Op=Linf; MetricForTensors=None), > >>>>>> > >>>>>> > >>>>>> The odd thing is that the norms that I get out are not exactly zero. > >>>>>> They are very small, but I'm taking the first and second derivatives > >>>>>> of the identity matrix, so I would expect them to evaluate to exactly > >>>>>> zero. The fact that they don't leads me to think that there is > >>>>>> something wrong either in my code or in how I have written the input > >>>>>> files. > >>>>>> > >>>>>> Should these derivatives evaluate to exactly zero? > >>>>>> > >>>>>> Greg > >>>>> > >>> > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From uerland at gmail.com Thu Aug 18 03:18:59 2011 From: uerland at gmail.com (Henning Sauerland) Date: Thu, 18 Aug 2011 10:18:59 +0200 Subject: [petsc-users] roundoff differences In-Reply-To: <4E4CC410.4000407@cita.utoronto.ca> References: <4E4A7477.1020408@wfu.edu> <4E4CC410.4000407@cita.utoronto.ca> Message-ID: <43CFCA4E-F404-4FF4-8EA7-FEBEC5D16070@gmail.com> Here is an interesting paper on the topic of reproducibility of parallel summation: Y. He and C. H. Q. Ding. Using accurate arithmetics to improve numerical reproducibility and stability in parallel applications. J. Supercomput., 18:259 ? 277, 2001. Henning On 18.08.2011, at 09:49, Harald Pfeiffer wrote: > Hello, > > we use PETSc to solve the nonlinear system arising from pseudo-spectral discretization of certain elliptic PDEs in Einstein's equations. When running the same job multiple times on the same number of processors on the same workstation, we find roundoff differences. Is this expected, e.g. because MPI reduction calls may behave differently depending on the load of the machine? Or should we be concerned and investigate further? > > Thanks, > Harald > > > > > > > > -------- Original Message -------- > Subject: Re: Quick question about derivatives in SpEC > Date: Tue, 16 Aug 2011 09:45:27 -0400 > From: Gregory B. Cook > To: Harald Pfeiffer > CC: Larry Kidder , Mark Scheel > > Hi Harald, > > All of the tests I was doing were on the same 8 cores on my office > workstation. It is running Ubuntu 11, and uses the default OpenMPI > communication approach. To make sure it wasn't something I was doing, I > ran two elliptic solves of the ExtendedConformalThinSandwich() volume > terms. Here are the outputs of snes.dat for the different levels: > > Run 1 Run 2 > Six0/snes.dat > 0 7.3385297958166698 0 7.3385297958166698 > 1 5.1229060531500723 1 5.1229060531500723 > 2 0.32616852761238285 2 0.32616852761238285 > 3 0.012351417186533147 3 0.012351417186800266 <***** > 4 9.7478354935351385e-06 4 9.7478351511500114e-06 > Six1/snes.dat > 0 0.13405558402489681 0 0.13405558402540407 > 1 0.00068002100028642610 1 0.00068002089609322440 > 2 6.8764357250058596e-08 2 6.3738394418031232e-08 > Six2/snes.dat > 0 0.0063028244769771681 0 0.0063028058475922306 > 1 1.4538921141731714e-06 1 1.4545032695605256e-06 > Six3/snes.dat > 0 0.00061476105672438877 0 0.00061476093499534406 > 1 6.0267672358059814e-08 1 5.4897793428123648e-08 > Six4/snes.dat > 0 0.00053059501859595651 0 0.00053059591479892143 > 1 4.8003269489205705e-08 1 4.8079799390886591e-08 > Six5/snes.dat > 0 3.6402372419546429e-05 0 3.6402169997838670e-05 > 1 5.3117360561476420e-09 1 5.2732089856727503e-09 > > The differences are clearly at the level of roundoff, but it is > "strange" that you cannot reproduce identical results. > > I've attached all of the .input files for this run in case you want to > try to reproduce my findings. > > Greg > > On 08/16/2011 06:21 AM, Harald Pfeiffer wrote: > > Hi Greg, > > > > some thoughts: > > > > Petsc is using standard MPI reduction calls, which may give results that > > differ by roundoff. We have positively noticed this happening for > > different number of processes, but perhaps this also happens depending > > on where in a cluster your jobs run (different network topology > > depending on whether all processors are on the same rack, vs. split > > among racks; dynamic load-balancing of network communication). > > > > You might want to try reserving a few nodes interactively, and then > > running the elliptic solver multiple times on this same set of nodes. > > > > The Mover does indeed load-balanced interpolation, but when doing so, > > the MPI communication should not affect identical results. > > > > Once there are roundoff differences, they are typically amplified during > > a petsc linear solve. The iterative algorithm takes different paths > > toward the solution, and a difference of 1e-10 doesn't seem excessive. > > > > Harald > > > > ps. Preconditioning is done differently on-processor and off-processor > > and depends therefore highly on the processor count. So if you were to > > change number of processors, the iterative solve will proceed very > > differently. > > > > On 8/15/11 10:27 PM, Gregory B. Cook wrote: > >> Hi Larry, > >> > >> I ran a check using the default ExtendedConformalThinSandwich() volume > >> terms and this also produced roundoff error differences between > >> identical runs, so I feel better about that. I am using the same > >> number of processors, but if there is any kind of dynamic load > >> balancing for interpolation/communication/etc, then I can see that > >> different runs might end up using different boundary communications. > >> Maybe that's all there is to it? > >> > >> Greg > >> > >> On 08/15/2011 04:16 PM, Larry Kidder wrote: > >>> Hi Greg, > >>> > >>> Harald is traveling, so I am not sure when he will answer. > >>> My vague recollection is that there is something about how PETSc does > >>> preconditioning in parallel that leads to not producing the same result; > >>> but I don't recall if this happens in general, or only if you change the > >>> distribution of processes. > >>> > >>> Larry > >>> > >>> Gregory B. Cook wrote: > >>>> Hi Guys, > >>>> > >>>> I have a follow-up question that may be tangentially related to my > >>>> original question about derivatives. This one is targeted at Harald. > >>>> > >>>> When I run a version of my code where the very small errors in the > >>>> derivative of the metric are not present (I code them in differently), > >>>> I find that running the exact same input files successively does not > >>>> produce exactly the same results. This is a multi-level elliptic solve > >>>> on a complex domain for binary black holes. On Level-0, the the > >>>> results returned in snes.dat are identical. On Level-1, the initial > >>>> and second snes norms are identical, but the third differs. After > >>>> this, all snes norms differ. > >>>> > >>>> Is this to be expected? Does PETSc not produce identical results on > >>>> consecutive solves with the same starting point? Is there something in > >>>> the MPI communication that means that the results should differ? The > >>>> differences start at the order of 10^-13, but grow by the 6th level to > >>>> be of order 10^-10. > >>>> > >>>> Greg > >>>> > >>>> On 08/15/2011 01:02 PM, Larry Kidder wrote: > >>>>> Hi Greg, > >>>>> > >>>>> Did you compute the norm of the metric itself? > >>>>> What domain did you use? > >>>>> > >>>>> Larry > >>>>> > >>>>> Gregory B. Cook wrote: > >>>>>> Hi Guys, > >>>>>> > >>>>>> I was doing a simple test as part of debugging some code I'm writing. > >>>>>> I ended up placing the following relevant lines of code into the > >>>>>> EllipticItems.input and EllipticObservers.input files: > >>>>>> > >>>>>> ---EllipticItems.input--- > >>>>>> EvaluateMatrixFormula(Output=InvConformalMetric; Dim=3; Symm=11; > >>>>>> M[0,0]=1; M[1,1]=1; M[2,2]=1), > >>>>>> FirstDeriv(Input=InvConformalMetric; Output=dInvConformalMetric), > >>>>>> SecondDeriv(Input=InvConformalMetric; Output=ddInvConformalMetric), > >>>>>> > >>>>>> FlattenDeriv(Input=dInvConformalMetric; > >>>>>> Output=fdInvConformalMetric;DerivPosition=Last), > >>>>>> FlattenDeriv(Input=ddInvConformalMetric; > >>>>>> Output=fddInvConformalMetric;DerivPosition=Last), > >>>>>> > >>>>>> ---EllipticObservers.input--- > >>>>>> NormOfTensor(Input=fdInvConformalMetric, fddInvConformalMetric; > >>>>>> Filename=dInvCM_L2.dat;Op=L2; MetricForTensors=None), > >>>>>> NormOfTensor(Input=fdInvConformalMetric, fddInvConformalMetric; > >>>>>> Filename=dInvCM_Linf.dat;Op=Linf; MetricForTensors=None), > >>>>>> > >>>>>> > >>>>>> The odd thing is that the norms that I get out are not exactly zero. > >>>>>> They are very small, but I'm taking the first and second derivatives > >>>>>> of the identity matrix, so I would expect them to evaluate to exactly > >>>>>> zero. The fact that they don't leads me to think that there is > >>>>>> something wrong either in my code or in how I have written the input > >>>>>> files. > >>>>>> > >>>>>> Should these derivatives evaluate to exactly zero? > >>>>>> > >>>>>> Greg > >>>>> > >>> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevin.richard.green at gmail.com Thu Aug 18 11:49:07 2011 From: kevin.richard.green at gmail.com (Kevin Green) Date: Thu, 18 Aug 2011 12:49:07 -0400 Subject: [petsc-users] roundoff differences In-Reply-To: <43CFCA4E-F404-4FF4-8EA7-FEBEC5D16070@gmail.com> References: <4E4A7477.1020408@wfu.edu> <4E4CC410.4000407@cita.utoronto.ca> <43CFCA4E-F404-4FF4-8EA7-FEBEC5D16070@gmail.com> Message-ID: On that note, do PETSc routines take into account ideas such as those in that paper for matrix-matrix matrix-vector etc. operations? I assume that that by default they don't (since they take more time), but are there flags/functions that do? Kevin On Thu, Aug 18, 2011 at 4:18 AM, Henning Sauerland wrote: > Here is an interesting paper on the topic of reproducibility of parallel > summation: > > Y. He and C. H. Q. Ding. Using accurate arithmetics to improve numerical > reproducibility and stability in parallel applications. J. Supercomput., > 18:259 ? 277, 2001. > > > Henning > > > On 18.08.2011, at 09:49, Harald Pfeiffer wrote: > > Hello, > > we use PETSc to solve the nonlinear system arising from pseudo-spectral > discretization of certain elliptic PDEs in Einstein's equations. When > running the same job multiple times on the same number of processors on the > same workstation, we find roundoff differences. Is this expected, e.g. > because MPI reduction calls may behave differently depending on the load of > the machine? Or should we be concerned and investigate further? > > Thanks, > Harald > > > > > > > > -------- Original Message -------- Subject: Re: Quick question about > derivatives in SpEC Date: Tue, 16 Aug 2011 09:45:27 -0400 From: Gregory > B. Cook To: Harald Pfeiffer > CC: Larry Kidder > , Mark Scheel > > > Hi Harald, > > All of the tests I was doing were on the same 8 cores on my office > workstation. It is running Ubuntu 11, and uses the default OpenMPI > communication approach. To make sure it wasn't something I was doing, I > ran two elliptic solves of the ExtendedConformalThinSandwich() volume > terms. Here are the outputs of snes.dat for the different levels: > > Run 1 Run 2 > Six0/snes.dat > 0 7.3385297958166698 0 7.3385297958166698 > 1 5.1229060531500723 1 5.1229060531500723 > 2 0.32616852761238285 2 0.32616852761238285 > 3 0.012351417186533147 3 0.012351417186800266 <***** > 4 9.7478354935351385e-06 4 9.7478351511500114e-06 > Six1/snes.dat > 0 0.13405558402489681 0 0.13405558402540407 > 1 0.00068002100028642610 1 0.00068002089609322440 > 2 6.8764357250058596e-08 2 6.3738394418031232e-08 > Six2/snes.dat > 0 0.0063028244769771681 0 0.0063028058475922306 > 1 1.4538921141731714e-06 1 1.4545032695605256e-06 > Six3/snes.dat > 0 0.00061476105672438877 0 0.00061476093499534406 > 1 6.0267672358059814e-08 1 5.4897793428123648e-08 > Six4/snes.dat > 0 0.00053059501859595651 0 0.00053059591479892143 > 1 4.8003269489205705e-08 1 4.8079799390886591e-08 > Six5/snes.dat > 0 3.6402372419546429e-05 0 3.6402169997838670e-05 > 1 5.3117360561476420e-09 1 5.2732089856727503e-09 > > The differences are clearly at the level of roundoff, but it is > "strange" that you cannot reproduce identical results. > > I've attached all of the .input files for this run in case you want to > try to reproduce my findings. > > Greg > > On 08/16/2011 06:21 AM, Harald Pfeiffer wrote: > > Hi Greg, > > > > some thoughts: > > > > Petsc is using standard MPI reduction calls, which may give results that > > differ by roundoff. We have positively noticed this happening for > > different number of processes, but perhaps this also happens depending > > on where in a cluster your jobs run (different network topology > > depending on whether all processors are on the same rack, vs. split > > among racks; dynamic load-balancing of network communication). > > > > You might want to try reserving a few nodes interactively, and then > > running the elliptic solver multiple times on this same set of nodes. > > > > The Mover does indeed load-balanced interpolation, but when doing so, > > the MPI communication should not affect identical results. > > > > Once there are roundoff differences, they are typically amplified during > > a petsc linear solve. The iterative algorithm takes different paths > > toward the solution, and a difference of 1e-10 doesn't seem excessive. > > > > Harald > > > > ps. Preconditioning is done differently on-processor and off-processor > > and depends therefore highly on the processor count. So if you were to > > change number of processors, the iterative solve will proceed very > > differently. > > > > On 8/15/11 10:27 PM, Gregory B. Cook wrote: > >> Hi Larry, > >> > >> I ran a check using the default ExtendedConformalThinSandwich() volume > >> terms and this also produced roundoff error differences between > >> identical runs, so I feel better about that. I am using the same > >> number of processors, but if there is any kind of dynamic load > >> balancing for interpolation/communication/etc, then I can see that > >> different runs might end up using different boundary communications. > >> Maybe that's all there is to it? > >> > >> Greg > >> > >> On 08/15/2011 04:16 PM, Larry Kidder wrote: > >>> Hi Greg, > >>> > >>> Harald is traveling, so I am not sure when he will answer. > >>> My vague recollection is that there is something about how PETSc does > >>> preconditioning in parallel that leads to not producing the same result; > >>> but I don't recall if this happens in general, or only if you change the > >>> distribution of processes. > >>> > >>> Larry > >>> > >>> Gregory B. Cook wrote: > >>>> Hi Guys, > >>>> > >>>> I have a follow-up question that may be tangentially related to my > >>>> original question about derivatives. This one is targeted at Harald. > >>>> > >>>> When I run a version of my code where the very small errors in the > >>>> derivative of the metric are not present (I code them in differently), > >>>> I find that running the exact same input files successively does not > >>>> produce exactly the same results. This is a multi-level elliptic solve > >>>> on a complex domain for binary black holes. On Level-0, the the > >>>> results returned in snes.dat are identical. On Level-1, the initial > >>>> and second snes norms are identical, but the third differs. After > >>>> this, all snes norms differ. > >>>> > >>>> Is this to be expected? Does PETSc not produce identical results on > >>>> consecutive solves with the same starting point? Is there something in > >>>> the MPI communication that means that the results should differ? The > >>>> differences start at the order of 10^-13, but grow by the 6th level to > >>>> be of order 10^-10. > >>>> > >>>> Greg > >>>> > >>>> On 08/15/2011 01:02 PM, Larry Kidder wrote: > >>>>> Hi Greg, > >>>>> > >>>>> Did you compute the norm of the metric itself? > >>>>> What domain did you use? > >>>>> > >>>>> Larry > >>>>> > >>>>> Gregory B. Cook wrote: > >>>>>> Hi Guys, > >>>>>> > >>>>>> I was doing a simple test as part of debugging some code I'm writing. > >>>>>> I ended up placing the following relevant lines of code into the > >>>>>> EllipticItems.input and EllipticObservers.input files: > >>>>>> > >>>>>> ---EllipticItems.input--- > >>>>>> EvaluateMatrixFormula(Output=InvConformalMetric; Dim=3; Symm=11; > >>>>>> M[0,0]=1; M[1,1]=1; M[2,2]=1), > >>>>>> FirstDeriv(Input=InvConformalMetric; Output=dInvConformalMetric), > >>>>>> SecondDeriv(Input=InvConformalMetric; Output=ddInvConformalMetric), > >>>>>> > >>>>>> FlattenDeriv(Input=dInvConformalMetric; > >>>>>> Output=fdInvConformalMetric;DerivPosition=Last), > >>>>>> FlattenDeriv(Input=ddInvConformalMetric; > >>>>>> Output=fddInvConformalMetric;DerivPosition=Last), > >>>>>> > >>>>>> ---EllipticObservers.input--- > >>>>>> NormOfTensor(Input=fdInvConformalMetric, fddInvConformalMetric; > >>>>>> Filename=dInvCM_L2.dat;Op=L2; MetricForTensors=None), > >>>>>> NormOfTensor(Input=fdInvConformalMetric, fddInvConformalMetric; > >>>>>> Filename=dInvCM_Linf.dat;Op=Linf; MetricForTensors=None), > >>>>>> > >>>>>> > >>>>>> The odd thing is that the norms that I get out are not exactly zero. > >>>>>> They are very small, but I'm taking the first and second derivatives > >>>>>> of the identity matrix, so I would expect them to evaluate to exactly > >>>>>> zero. The fact that they don't leads me to think that there is > >>>>>> something wrong either in my code or in how I have written the input > >>>>>> files. > >>>>>> > >>>>>> Should these derivatives evaluate to exactly zero? > >>>>>> > >>>>>> Greg > >>>>> > >>> > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Aug 18 12:40:10 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 18 Aug 2011 12:40:10 -0500 Subject: [petsc-users] roundoff differences In-Reply-To: References: <4E4A7477.1020408@wfu.edu> <4E4CC410.4000407@cita.utoronto.ca> <43CFCA4E-F404-4FF4-8EA7-FEBEC5D16070@gmail.com> Message-ID: <13462AC0-0068-4B66-A1AE-43DBD2D2A590@mcs.anl.gov> On Aug 18, 2011, at 11:49 AM, Kevin Green wrote: > On that note, do PETSc routines take into account ideas such as those in that paper for matrix-matrix matrix-vector etc. operations? I assume that that by default they don't (since they take more time), but are there flags/functions that do? No, we do not try to generate reproducible floating point results. Barry > > Kevin > > On Thu, Aug 18, 2011 at 4:18 AM, Henning Sauerland wrote: > Here is an interesting paper on the topic of reproducibility of parallel summation: > > Y. He and C. H. Q. Ding. Using accurate arithmetics to improve numerical reproducibility and stability in parallel applications. J. Supercomput., 18:259 ? 277, 2001. > > > Henning > > > On 18.08.2011, at 09:49, Harald Pfeiffer wrote: > >> Hello, >> >> we use PETSc to solve the nonlinear system arising from pseudo-spectral discretization of certain elliptic PDEs in Einstein's equations. When running the same job multiple times on the same number of processors on the same workstation, we find roundoff differences. Is this expected, e.g. because MPI reduction calls may behave differently depending on the load of the machine? Or should we be concerned and investigate further? >> >> Thanks, >> Harald >> >> >> >> >> >> >> >> -------- Original Message -------- >> Subject: Re: Quick question about derivatives in SpEC >> Date: Tue, 16 Aug 2011 09:45:27 -0400 >> From: Gregory B. Cook >> To: Harald Pfeiffer >> CC: Larry Kidder , Mark Scheel >> >> Hi Harald, >> >> All of the tests I was doing were on the same 8 cores on my office >> workstation. It is running Ubuntu 11, and uses the default OpenMPI >> communication approach. To make sure it wasn't something I was doing, I >> ran two elliptic solves of the ExtendedConformalThinSandwich() volume >> terms. Here are the outputs of snes.dat for the different levels: >> >> Run 1 Run 2 >> Six0/snes.dat >> 0 7.3385297958166698 0 7.3385297958166698 >> 1 5.1229060531500723 1 5.1229060531500723 >> 2 0.32616852761238285 2 0.32616852761238285 >> 3 0.012351417186533147 3 0.012351417186800266 <***** >> 4 9.7478354935351385e-06 4 9.7478351511500114e-06 >> Six1/snes.dat >> 0 0.13405558402489681 0 0.13405558402540407 >> 1 0.00068002100028642610 1 0.00068002089609322440 >> 2 6.8764357250058596e-08 2 6.3738394418031232e-08 >> Six2/snes.dat >> 0 0.0063028244769771681 0 0.0063028058475922306 >> 1 1.4538921141731714e-06 1 1.4545032695605256e-06 >> Six3/snes.dat >> 0 0.00061476105672438877 0 0.00061476093499534406 >> 1 6.0267672358059814e-08 1 5.4897793428123648e-08 >> Six4/snes.dat >> 0 0.00053059501859595651 0 0.00053059591479892143 >> 1 4.8003269489205705e-08 1 4.8079799390886591e-08 >> Six5/snes.dat >> 0 3.6402372419546429e-05 0 3.6402169997838670e-05 >> 1 5.3117360561476420e-09 1 5.2732089856727503e-09 >> >> The differences are clearly at the level of roundoff, but it is >> "strange" that you cannot reproduce identical results. >> >> I've attached all of the .input files for this run in case you want to >> try to reproduce my findings. >> >> Greg >> >> On 08/16/2011 06:21 AM, Harald Pfeiffer wrote: >> > Hi Greg, >> > >> > some thoughts: >> > >> > Petsc is using standard MPI reduction calls, which may give results that >> > differ by roundoff. We have positively noticed this happening for >> > different number of processes, but perhaps this also happens depending >> > on where in a cluster your jobs run (different network topology >> > depending on whether all processors are on the same rack, vs. split >> > among racks; dynamic load-balancing of network communication). >> > >> > You might want to try reserving a few nodes interactively, and then >> > running the elliptic solver multiple times on this same set of nodes. >> > >> > The Mover does indeed load-balanced interpolation, but when doing so, >> > the MPI communication should not affect identical results. >> > >> > Once there are roundoff differences, they are typically amplified during >> > a petsc linear solve. The iterative algorithm takes different paths >> > toward the solution, and a difference of 1e-10 doesn't seem excessive. >> > >> > Harald >> > >> > ps. Preconditioning is done differently on-processor and off-processor >> > and depends therefore highly on the processor count. So if you were to >> > change number of processors, the iterative solve will proceed very >> > differently. >> > >> > On 8/15/11 10:27 PM, Gregory B. Cook wrote: >> >> Hi Larry, >> >> >> >> I ran a check using the default ExtendedConformalThinSandwich() volume >> >> terms and this also produced roundoff error differences between >> >> identical runs, so I feel better about that. I am using the same >> >> number of processors, but if there is any kind of dynamic load >> >> balancing for interpolation/communication/etc, then I can see that >> >> different runs might end up using different boundary communications. >> >> Maybe that's all there is to it? >> >> >> >> Greg >> >> >> >> On 08/15/2011 04:16 PM, Larry Kidder wrote: >> >>> Hi Greg, >> >>> >> >>> Harald is traveling, so I am not sure when he will answer. >> >>> My vague recollection is that there is something about how PETSc does >> >>> preconditioning in parallel that leads to not producing the same result; >> >>> but I don't recall if this happens in general, or only if you change the >> >>> distribution of processes. >> >>> >> >>> Larry >> >>> >> >>> Gregory B. Cook wrote: >> >>>> Hi Guys, >> >>>> >> >>>> I have a follow-up question that may be tangentially related to my >> >>>> original question about derivatives. This one is targeted at Harald. >> >>>> >> >>>> When I run a version of my code where the very small errors in the >> >>>> derivative of the metric are not present (I code them in differently), >> >>>> I find that running the exact same input files successively does not >> >>>> produce exactly the same results. This is a multi-level elliptic solve >> >>>> on a complex domain for binary black holes. On Level-0, the the >> >>>> results returned in snes.dat are identical. On Level-1, the initial >> >>>> and second snes norms are identical, but the third differs. After >> >>>> this, all snes norms differ. >> >>>> >> >>>> Is this to be expected? Does PETSc not produce identical results on >> >>>> consecutive solves with the same starting point? Is there something in >> >>>> the MPI communication that means that the results should differ? The >> >>>> differences start at the order of 10^-13, but grow by the 6th level to >> >>>> be of order 10^-10. >> >>>> >> >>>> Greg >> >>>> >> >>>> On 08/15/2011 01:02 PM, Larry Kidder wrote: >> >>>>> Hi Greg, >> >>>>> >> >>>>> Did you compute the norm of the metric itself? >> >>>>> What domain did you use? >> >>>>> >> >>>>> Larry >> >>>>> >> >>>>> Gregory B. Cook wrote: >> >>>>>> Hi Guys, >> >>>>>> >> >>>>>> I was doing a simple test as part of debugging some code I'm writing. >> >>>>>> I ended up placing the following relevant lines of code into the >> >>>>>> EllipticItems.input and EllipticObservers.input files: >> >>>>>> >> >>>>>> ---EllipticItems.input--- >> >>>>>> EvaluateMatrixFormula(Output=InvConformalMetric; Dim=3; Symm=11; >> >>>>>> M[0,0]=1; M[1,1]=1; M[2,2]=1), >> >>>>>> FirstDeriv(Input=InvConformalMetric; Output=dInvConformalMetric), >> >>>>>> SecondDeriv(Input=InvConformalMetric; Output=ddInvConformalMetric), >> >>>>>> >> >>>>>> FlattenDeriv(Input=dInvConformalMetric; >> >>>>>> Output=fdInvConformalMetric;DerivPosition=Last), >> >>>>>> FlattenDeriv(Input=ddInvConformalMetric; >> >>>>>> Output=fddInvConformalMetric;DerivPosition=Last), >> >>>>>> >> >>>>>> ---EllipticObservers.input--- >> >>>>>> NormOfTensor(Input=fdInvConformalMetric, fddInvConformalMetric; >> >>>>>> Filename=dInvCM_L2.dat;Op=L2; MetricForTensors=None), >> >>>>>> NormOfTensor(Input=fdInvConformalMetric, fddInvConformalMetric; >> >>>>>> Filename=dInvCM_Linf.dat;Op=Linf; MetricForTensors=None), >> >>>>>> >> >>>>>> >> >>>>>> The odd thing is that the norms that I get out are not exactly zero. >> >>>>>> They are very small, but I'm taking the first and second derivatives >> >>>>>> of the identity matrix, so I would expect them to evaluate to exactly >> >>>>>> zero. The fact that they don't leads me to think that there is >> >>>>>> something wrong either in my code or in how I have written the input >> >>>>>> files. >> >>>>>> >> >>>>>> Should these derivatives evaluate to exactly zero? >> >>>>>> >> >>>>>> Greg >> >>>>> >> >>> >> > >> >> > > From zhenglun.wei at gmail.com Thu Aug 18 17:02:01 2011 From: zhenglun.wei at gmail.com (Alan Wei) Date: Thu, 18 Aug 2011 17:02:01 -0500 Subject: [petsc-users] DMDAVecGetArray vs DMDAVecRestoreArray Message-ID: Dear all, I hope you're having a nice day. I'm trying to get the x, y-value of a coordinate created by DMDACreate2d. Lines I wrote down are: DMDAGetCoordinateDA(da, &cda); DMDAGetGhostedCoordinates(da, &gc); DMDAVecGetArray(cda, gc, &coors); for (i = xs; i < xs+xm; i++) { for (j = ys; j < ys+ym; j++) { printf("%f %f\n", coors[j][i].x, coors[j][i].y); } } DMDAVecRestoreArray(cda, gc, &coors); I wonder why I need to use "DMDAVecRestoreArray(cda, gc, &coors);" I used it because the manual said I need. Actually, I tried to comment it out and there is no difference. Thanks, Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Aug 18 17:49:58 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 18 Aug 2011 17:49:58 -0500 Subject: [petsc-users] DMDAVecGetArray vs DMDAVecRestoreArray In-Reply-To: References: Message-ID: <0540AB69-BCF3-4A95-9099-04515A61EC3F@mcs.anl.gov> On Aug 18, 2011, at 5:02 PM, Alan Wei wrote: > Dear all, > I hope you're having a nice day. > I'm trying to get the x, y-value of a coordinate created by DMDACreate2d. Lines I wrote down are: > > DMDAGetCoordinateDA(da, &cda); > DMDAGetGhostedCoordinates(da, &gc); > DMDAVecGetArray(cda, gc, &coors); > > for (i = xs; i < xs+xm; i++) { > for (j = ys; j < ys+ym; j++) { > printf("%f %f\n", coors[j][i].x, coors[j][i].y); > } > } > > DMDAVecRestoreArray(cda, gc, &coors); > > I wonder why I need to use "DMDAVecRestoreArray(cda, gc, &coors);" I used it because the manual said I need. Actually, I tried to comment it out and there is no difference. > > Thanks, > Alan The practical reason is you might get memory bleeding if you don't. The philosphical reason is that a call DMDAVecGetArray() or VecGetArray() or others is a request to access the internal data structure of an object, the object then gives you permission to access that data. the RestoreArray() is a statement by you that after this call you will NOT access that data directly (without another GetArray call). Imagine if you did Barry From zhenglun.wei at gmail.com Thu Aug 18 20:07:55 2011 From: zhenglun.wei at gmail.com (Alan Wei) Date: Thu, 18 Aug 2011 20:07:55 -0500 Subject: [petsc-users] DMDAVecGetArray vs DMDAVecRestoreArray In-Reply-To: <0540AB69-BCF3-4A95-9099-04515A61EC3F@mcs.anl.gov> References: <0540AB69-BCF3-4A95-9099-04515A61EC3F@mcs.anl.gov> Message-ID: Dear Barry, Thanks for your reply. best, Alan On Thu, Aug 18, 2011 at 5:49 PM, Barry Smith wrote: > > On Aug 18, 2011, at 5:02 PM, Alan Wei wrote: > > > Dear all, > > I hope you're having a nice day. > > I'm trying to get the x, y-value of a coordinate created by > DMDACreate2d. Lines I wrote down are: > > > > DMDAGetCoordinateDA(da, &cda); > > DMDAGetGhostedCoordinates(da, &gc); > > DMDAVecGetArray(cda, gc, &coors); > > > > for (i = xs; i < xs+xm; i++) { > > for (j = ys; j < ys+ym; j++) { > > printf("%f %f\n", coors[j][i].x, coors[j][i].y); > > } > > } > > > > DMDAVecRestoreArray(cda, gc, &coors); > > > > I wonder why I need to use "DMDAVecRestoreArray(cda, gc, &coors);" I used > it because the manual said I need. Actually, I tried to comment it out and > there is no difference. > > > > Thanks, > > Alan > > The practical reason is you might get memory bleeding if you don't. The > philosphical reason is that a call DMDAVecGetArray() or VecGetArray() or > others is a request to access the internal data structure of an object, the > object then gives you permission to access that data. the RestoreArray() is > a statement by you that after this call you will NOT access that data > directly (without another GetArray call). Imagine if you did > > Barry > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From griffith at cims.nyu.edu Thu Aug 18 20:17:14 2011 From: griffith at cims.nyu.edu (Boyce Griffith) Date: Thu, 18 Aug 2011 21:17:14 -0400 Subject: [petsc-users] roundoff differences In-Reply-To: References: <4E4A7477.1020408@wfu.edu> <4E4CC410.4000407@cita.utoronto.ca> <43CFCA4E-F404-4FF4-8EA7-FEBEC5D16070@gmail.com> Message-ID: <4E4DB99A.7010000@cims.nyu.edu> On 8/18/11 12:49 PM, Kevin Green wrote: > On that note, do PETSc routines take into account ideas such as those in > that paper for matrix-matrix matrix-vector etc. operations? I assume > that that by default they don't (since they take more time), but are > there flags/functions that do? One source of this lack of reproducibility is in parallel accumulation, e.g., calling VecSetValues() with ADD_VALUES with input values coming from different processors. If efficiency is not an issue, it is not too hard to modify VecAssemblyBegin/End so that the final accumulation is done in extended precision arithmetic (e.g., with the QD library), or by using a more accurate summation algorithm. -- Boyce > > Kevin > > On Thu, Aug 18, 2011 at 4:18 AM, Henning Sauerland > wrote: > > Here is an interesting paper on the topic of reproducibility of > parallel summation: > > Y. He and C. H. Q. Ding. Using accurate arithmetics to improve > numerical reproducibility and stability in parallel applications. J. > Supercomput., 18:259 ? 277, 2001. > > > Henning > > > On 18.08.2011, at 09:49, Harald Pfeiffer wrote: > >> Hello, >> >> we use PETSc to solve the nonlinear system arising from >> pseudo-spectral discretization of certain elliptic PDEs in >> Einstein's equations. When running the same job multiple times on >> the same number of processors on the same workstation, we find >> roundoff differences. Is this expected, e.g. because MPI >> reduction calls may behave differently depending on the load of >> the machine? Or should we be concerned and investigate further? >> >> Thanks, >> Harald >> >> >> >> >> >> >> >> -------- Original Message -------- >> Subject: Re: Quick question about derivatives in SpEC >> Date: Tue, 16 Aug 2011 09:45:27 -0400 >> From: Gregory B. Cook >> To: Harald Pfeiffer >> >> CC: Larry Kidder >> , Mark Scheel >> >> >> >> >> Hi Harald, >> >> All of the tests I was doing were on the same 8 cores on my office >> workstation. It is running Ubuntu 11, and uses the default OpenMPI >> communication approach. To make sure it wasn't something I was doing, I >> ran two elliptic solves of the ExtendedConformalThinSandwich() volume >> terms. Here are the outputs of snes.dat for the different levels: >> >> Run 1 Run 2 >> Six0/snes.dat >> 0 7.3385297958166698 0 7.3385297958166698 >> 1 5.1229060531500723 1 5.1229060531500723 >> 2 0.32616852761238285 2 0.32616852761238285 >> 3 0.012351417186533147 3 0.012351417186800266<***** >> 4 9.7478354935351385e-06 4 9.7478351511500114e-06 >> Six1/snes.dat >> 0 0.13405558402489681 0 0.13405558402540407 >> 1 0.00068002100028642610 1 0.00068002089609322440 >> 2 6.8764357250058596e-08 2 6.3738394418031232e-08 >> Six2/snes.dat >> 0 0.0063028244769771681 0 0.0063028058475922306 >> 1 1.4538921141731714e-06 1 1.4545032695605256e-06 >> Six3/snes.dat >> 0 0.00061476105672438877 0 0.00061476093499534406 >> 1 6.0267672358059814e-08 1 5.4897793428123648e-08 >> Six4/snes.dat >> 0 0.00053059501859595651 0 0.00053059591479892143 >> 1 4.8003269489205705e-08 1 4.8079799390886591e-08 >> Six5/snes.dat >> 0 3.6402372419546429e-05 0 3.6402169997838670e-05 >> 1 5.3117360561476420e-09 1 5.2732089856727503e-09 >> >> The differences are clearly at the level of roundoff, but it is >> "strange" that you cannot reproduce identical results. >> >> I've attached all of the .input files for this run in case you want to >> try to reproduce my findings. >> >> Greg >> >> On 08/16/2011 06:21 AM, Harald Pfeiffer wrote: >> > Hi Greg, >> > >> > some thoughts: >> > >> > Petsc is using standard MPI reduction calls, which may give results that >> > differ by roundoff. We have positively noticed this happening for >> > different number of processes, but perhaps this also happens depending >> > on where in a cluster your jobs run (different network topology >> > depending on whether all processors are on the same rack, vs. split >> > among racks; dynamic load-balancing of network communication). >> > >> > You might want to try reserving a few nodes interactively, and then >> > running the elliptic solver multiple times on this same set of nodes. >> > >> > The Mover does indeed load-balanced interpolation, but when doing so, >> > the MPI communication should not affect identical results. >> > >> > Once there are roundoff differences, they are typically amplified during >> > a petsc linear solve. The iterative algorithm takes different paths >> > toward the solution, and a difference of 1e-10 doesn't seem excessive. >> > >> > Harald >> > >> > ps. Preconditioning is done differently on-processor and off-processor >> > and depends therefore highly on the processor count. So if you were to >> > change number of processors, the iterative solve will proceed very >> > differently. >> > >> > On 8/15/11 10:27 PM, Gregory B. Cook wrote: >> >> Hi Larry, >> >> >> >> I ran a check using the default ExtendedConformalThinSandwich() volume >> >> terms and this also produced roundoff error differences between >> >> identical runs, so I feel better about that. I am using the same >> >> number of processors, but if there is any kind of dynamic load >> >> balancing for interpolation/communication/etc, then I can see that >> >> different runs might end up using different boundary communications. >> >> Maybe that's all there is to it? >> >> >> >> Greg >> >> >> >> On 08/15/2011 04:16 PM, Larry Kidder wrote: >> >>> Hi Greg, >> >>> >> >>> Harald is traveling, so I am not sure when he will answer. >> >>> My vague recollection is that there is something about how PETSc does >> >>> preconditioning in parallel that leads to not producing the same result; >> >>> but I don't recall if this happens in general, or only if you change the >> >>> distribution of processes. >> >>> >> >>> Larry >> >>> >> >>> Gregory B. Cook wrote: >> >>>> Hi Guys, >> >>>> >> >>>> I have a follow-up question that may be tangentially related to my >> >>>> original question about derivatives. This one is targeted at Harald. >> >>>> >> >>>> When I run a version of my code where the very small errors in the >> >>>> derivative of the metric are not present (I code them in differently), >> >>>> I find that running the exact same input files successively does not >> >>>> produce exactly the same results. This is a multi-level elliptic solve >> >>>> on a complex domain for binary black holes. On Level-0, the the >> >>>> results returned in snes.dat are identical. On Level-1, the initial >> >>>> and second snes norms are identical, but the third differs. After >> >>>> this, all snes norms differ. >> >>>> >> >>>> Is this to be expected? Does PETSc not produce identical results on >> >>>> consecutive solves with the same starting point? Is there something in >> >>>> the MPI communication that means that the results should differ? The >> >>>> differences start at the order of 10^-13, but grow by the 6th level to >> >>>> be of order 10^-10. >> >>>> >> >>>> Greg >> >>>> >> >>>> On 08/15/2011 01:02 PM, Larry Kidder wrote: >> >>>>> Hi Greg, >> >>>>> >> >>>>> Did you compute the norm of the metric itself? >> >>>>> What domain did you use? >> >>>>> >> >>>>> Larry >> >>>>> >> >>>>> Gregory B. Cook wrote: >> >>>>>> Hi Guys, >> >>>>>> >> >>>>>> I was doing a simple test as part of debugging some code I'm writing. >> >>>>>> I ended up placing the following relevant lines of code into the >> >>>>>> EllipticItems.input and EllipticObservers.input files: >> >>>>>> >> >>>>>> ---EllipticItems.input--- >> >>>>>> EvaluateMatrixFormula(Output=InvConformalMetric; Dim=3; Symm=11; >> >>>>>> M[0,0]=1; M[1,1]=1; M[2,2]=1), >> >>>>>> FirstDeriv(Input=InvConformalMetric; Output=dInvConformalMetric), >> >>>>>> SecondDeriv(Input=InvConformalMetric; Output=ddInvConformalMetric), >> >>>>>> >> >>>>>> FlattenDeriv(Input=dInvConformalMetric; >> >>>>>> Output=fdInvConformalMetric;DerivPosition=Last), >> >>>>>> FlattenDeriv(Input=ddInvConformalMetric; >> >>>>>> Output=fddInvConformalMetric;DerivPosition=Last), >> >>>>>> >> >>>>>> ---EllipticObservers.input--- >> >>>>>> NormOfTensor(Input=fdInvConformalMetric, fddInvConformalMetric; >> >>>>>> Filename=dInvCM_L2.dat;Op=L2; MetricForTensors=None), >> >>>>>> NormOfTensor(Input=fdInvConformalMetric, fddInvConformalMetric; >> >>>>>> Filename=dInvCM_Linf.dat;Op=Linf; MetricForTensors=None), >> >>>>>> >> >>>>>> >> >>>>>> The odd thing is that the norms that I get out are not exactly zero. >> >>>>>> They are very small, but I'm taking the first and second derivatives >> >>>>>> of the identity matrix, so I would expect them to evaluate to exactly >> >>>>>> zero. The fact that they don't leads me to think that there is >> >>>>>> something wrong either in my code or in how I have written the input >> >>>>>> files. >> >>>>>> >> >>>>>> Should these derivatives evaluate to exactly zero? >> >>>>>> >> >>>>>> Greg >> >>>>> >> >>> >> > >> > > From zhenglun.wei at gmail.com Thu Aug 18 21:03:16 2011 From: zhenglun.wei at gmail.com (Alan Wei) Date: Thu, 18 Aug 2011 21:03:16 -0500 Subject: [petsc-users] Creating a 2d geometry by using vector Message-ID: Dear All, I'm trying to 'draw' a 2-D cylinder in PETSc. It simply restore the x and y coordinate values in two arrays. I wonder if I can use Vec in PETSc to implement it (i.e. defining a vec circle) so that I can call its x or y coordinate by circle.x or circle.y. Is this possible? also, is this a better way than just defining it as two arrays, i.e. x[ ] and y[ ]. thanks in advance, Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Aug 18 21:09:53 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 18 Aug 2011 21:09:53 -0500 Subject: [petsc-users] Creating a 2d geometry by using vector In-Reply-To: References: Message-ID: Alan, I'm not sure what you are getting at but we commonly use tricks like (see src/snes/examples/tutorials/ex19.c) typedef struct { PetscScalar u,v,omega,temp; } Field; Then in this case use a DMDAVecGetArray() to get f and x and then have nice notation like f[j][i].u = x[j][i].u; f[j][i].v = x[j][i].v; f[j][i].omega = x[j][i].omega - (x[j][i+1].v - x[j][i].v)*dhx; f[j][i].temp = x[j][i].temp; note that this means the values are stored interlaced in the vector. Barry On Aug 18, 2011, at 9:03 PM, Alan Wei wrote: > Dear All, > I'm trying to 'draw' a 2-D cylinder in PETSc. It simply restore the x and y coordinate values in two arrays. I wonder if I can use Vec in PETSc to implement it (i.e. defining a vec circle) so that I can call its x or y coordinate by circle.x or circle.y. Is this possible? also, is this a better way than just defining it as two arrays, i.e. x[ ] and y[ ]. > > thanks in advance, > Alan From zhenglun.wei at gmail.com Thu Aug 18 21:40:09 2011 From: zhenglun.wei at gmail.com (Alan Wei) Date: Thu, 18 Aug 2011 21:40:09 -0500 Subject: [petsc-users] Creating a 2d geometry by using vector In-Reply-To: References: Message-ID: Dear Barry, That is really a great idea, and it is something that I want. I've never noticed that the structure in C is so marvelous. However, I have a further question. In that example, a variable x is defined like Field **x, and DAVecGetArray(da,X,&x). I wonder if there is any relationship between da, X and x after using DAVecGetArray. Or it just a prerequisite for using the nice notation of 'x'. thanks, Alan On Thu, Aug 18, 2011 at 9:09 PM, Barry Smith wrote: > > Alan, > > I'm not sure what you are getting at but we commonly use tricks like (see > src/snes/examples/tutorials/ex19.c) > > typedef struct { > PetscScalar u,v,omega,temp; > } Field; > > Then in this case use a DMDAVecGetArray() to get f and x and then have > nice notation like > > f[j][i].u = x[j][i].u; > f[j][i].v = x[j][i].v; > f[j][i].omega = x[j][i].omega - (x[j][i+1].v - x[j][i].v)*dhx; > f[j][i].temp = x[j][i].temp; > > note that this means the values are stored interlaced in the vector. > > Barry > > On Aug 18, 2011, at 9:03 PM, Alan Wei wrote: > > > Dear All, > > I'm trying to 'draw' a 2-D cylinder in PETSc. It simply restore the x > and y coordinate values in two arrays. I wonder if I can use Vec in PETSc to > implement it (i.e. defining a vec circle) so that I can call its x or y > coordinate by circle.x or circle.y. Is this possible? also, is this a better > way than just defining it as two arrays, i.e. x[ ] and y[ ]. > > > > thanks in advance, > > Alan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Aug 18 22:12:04 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 18 Aug 2011 22:12:04 -0500 Subject: [petsc-users] Creating a 2d geometry by using vector In-Reply-To: References: Message-ID: <441A6F5C-CD4C-4C50-B344-A135F35126E6@mcs.anl.gov> On Aug 18, 2011, at 9:40 PM, Alan Wei wrote: > Dear Barry, > That is really a great idea, and it is something that I want. I've never noticed that the structure in C is so marvelous. However, I have a further question. In that example, a variable x is defined like Field **x, and DAVecGetArray(da,X,&x). I wonder if there is any relationship between da, X and x after using DAVecGetArray. Or it just a prerequisite for using the nice notation of 'x'. The only real relationship is that x looks into the X data structure to allow you to access the data as if it were in simple arrays. In other words the x and X share the same actual vector data. Barry > > thanks, > > Alan > > On Thu, Aug 18, 2011 at 9:09 PM, Barry Smith wrote: > > Alan, > > I'm not sure what you are getting at but we commonly use tricks like (see src/snes/examples/tutorials/ex19.c) > > typedef struct { > PetscScalar u,v,omega,temp; > } Field; > > Then in this case use a DMDAVecGetArray() to get f and x and then have nice notation like > > f[j][i].u = x[j][i].u; > f[j][i].v = x[j][i].v; > f[j][i].omega = x[j][i].omega - (x[j][i+1].v - x[j][i].v)*dhx; > f[j][i].temp = x[j][i].temp; > > note that this means the values are stored interlaced in the vector. > > Barry > > On Aug 18, 2011, at 9:03 PM, Alan Wei wrote: > > > Dear All, > > I'm trying to 'draw' a 2-D cylinder in PETSc. It simply restore the x and y coordinate values in two arrays. I wonder if I can use Vec in PETSc to implement it (i.e. defining a vec circle) so that I can call its x or y coordinate by circle.x or circle.y. Is this possible? also, is this a better way than just defining it as two arrays, i.e. x[ ] and y[ ]. > > > > thanks in advance, > > Alan > > From dominik at itis.ethz.ch Fri Aug 19 09:39:45 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 19 Aug 2011 16:39:45 +0200 Subject: [petsc-users] Can't open display: Message-ID: Starting in a debugger, I get the error: Can't open display: My DISPLAY is :0.0. Tried adding explicit hostname to no avail. Typing "xterm" in the console opens up a window as expected. Can not find any pointers in the FAQ or the docu. Please help. Thanks and regards, Dominik From dominik at itis.ethz.ch Fri Aug 19 10:12:30 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 19 Aug 2011 17:12:30 +0200 Subject: [petsc-users] Can't open display: In-Reply-To: References: Message-ID: Solved. Add -display :0.0 to the command line options. Dominik On Fri, Aug 19, 2011 at 4:39 PM, Dominik Szczerba wrote: > Starting in a debugger, I get the error: > > Can't open display: > > My DISPLAY is :0.0. Tried adding explicit hostname to no avail. Typing > "xterm" in the console opens up a window as expected. Can not find any > pointers in the FAQ or the docu. Please help. > > Thanks and regards, > Dominik > From dominik at itis.ethz.ch Fri Aug 19 10:27:05 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 19 Aug 2011 17:27:05 +0200 Subject: [petsc-users] debugger fails Message-ID: Hi, I am starting my app in the debugger as: mpiexec -np 2 sm3t4mpi run.xml -start_in_debugger -display :0.0 In the console I get: [1]PETSC ERROR: MPI error 14 in the two open terminals with gdb I get: 0x00007f2ecdd15590 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:82 82 ../sysdeps/unix/syscall-template.S: No such file or directory. in ../sysdeps/unix/syscall-template.S (gdb) I type 'c' nonetheless and see: (gdb) c Continuing. [New Thread 0x7f268e975700 (LWP 22388)] Program received signal SIGABRT, Aborted. 0x00007f268f421d05 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 64 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory. in ../nptl/sysdeps/unix/sysv/linux/raise.c How do I go on debugging? Many thanks for any hints, Dominik From balay at mcs.anl.gov Fri Aug 19 10:49:24 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 19 Aug 2011 10:49:24 -0500 (CDT) Subject: [petsc-users] debugger fails In-Reply-To: References: Message-ID: On Fri, 19 Aug 2011, Dominik Szczerba wrote: > Hi, > > I am starting my app in the debugger as: > > mpiexec -np 2 sm3t4mpi run.xml -start_in_debugger -display :0.0 > > In the console I get: > > [1]PETSC ERROR: MPI error 14 > > in the two open terminals with gdb I get: > > 0x00007f2ecdd15590 in __nanosleep_nocancel () at > ../sysdeps/unix/syscall-template.S:82 > 82 ../sysdeps/unix/syscall-template.S: No such file or directory. > in ../sysdeps/unix/syscall-template.S > (gdb) > > > I type 'c' nonetheless and see: > > (gdb) c > Continuing. > [New Thread 0x7f268e975700 (LWP 22388)] > > Program received signal SIGABRT, Aborted. > 0x00007f268f421d05 in raise (sig=6) at > ../nptl/sysdeps/unix/sysv/linux/raise.c:64 > 64 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory. > in ../nptl/sysdeps/unix/sysv/linux/raise.c > > > > How do I go on debugging? what do you get for: (gdb) where Satish > > Many thanks for any hints, > > Dominik > From dominik at itis.ethz.ch Fri Aug 19 10:57:41 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 19 Aug 2011 17:57:41 +0200 Subject: [petsc-users] debugger fails In-Reply-To: References: Message-ID: (gdb) where #0 0x00007fae5b941590 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:82 #1 0x00007fae5b94143c in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138 #2 0x000000000056cc48 in PetscSleep (s=10) at psleep.c:56 #3 0x0000000000838887 in PetscAttachDebugger () at adebug.c:410 #4 0x00000000005590a7 in PetscOptionsCheckInitial_Private () at init.c:392 #5 0x000000000055e40e in PetscInitialize (argc=0x7ffff403debc, args=0x7ffff403deb0, file=0x0, help=0x0) at pinit.c:639 #6 0x0000000000524a16 in PetscSolver::InitializePetsc (argc=0x7ffff403debc, argv=0x7ffff403deb0) at /home/dsz/src/framework/trunk/solve/PetscSolver.cxx:124 #7 0x00000000004c404f in main (argc=4, argv=0x7ffff403e4c8) at /home/dsz/src/framework/trunk/solve/cd3t10mpi_main.cxx:526 (gdb) PetscSolver.cxx:124: ierr = PetscInitialize(argc, argv, (char *)0, (char *)0); CHKERRQ(ierr); Hmmm, not very helpful..... The app runs on one cpu, but silently crashes on two. Any hints are very appreciated. Dominik On Fri, Aug 19, 2011 at 5:49 PM, Satish Balay wrote: > On Fri, 19 Aug 2011, Dominik Szczerba wrote: > >> Hi, >> >> I am starting my app in the debugger as: >> >> mpiexec -np 2 sm3t4mpi run.xml -start_in_debugger -display :0.0 >> >> In the console I get: >> >> [1]PETSC ERROR: MPI error 14 >> >> in the two open terminals with gdb I get: >> >> 0x00007f2ecdd15590 in __nanosleep_nocancel () at >> ../sysdeps/unix/syscall-template.S:82 >> 82 ? ? ?../sysdeps/unix/syscall-template.S: No such file or directory. >> ? ? ? ? in ../sysdeps/unix/syscall-template.S >> (gdb) >> >> >> I type 'c' nonetheless and see: >> >> (gdb) c >> Continuing. >> [New Thread 0x7f268e975700 (LWP 22388)] >> >> Program received signal SIGABRT, Aborted. >> 0x00007f268f421d05 in raise (sig=6) at >> ../nptl/sysdeps/unix/sysv/linux/raise.c:64 >> 64 ? ? ?../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory. >> ? ? ? ? in ../nptl/sysdeps/unix/sysv/linux/raise.c >> >> >> >> How do I go on debugging? > > what do you get for: > > (gdb) where > > Satish > > >> >> Many thanks for any hints, >> >> Dominik >> > > From aron.ahmadia at kaust.edu.sa Fri Aug 19 11:47:50 2011 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Fri, 19 Aug 2011 19:47:50 +0300 Subject: [petsc-users] debugger fails In-Reply-To: References: Message-ID: You want to do a 'where' on the second break, when your program is raising an abort signal... A On Fri, Aug 19, 2011 at 6:57 PM, Dominik Szczerba wrote: > (gdb) where > #0 0x00007fae5b941590 in __nanosleep_nocancel () at > ../sysdeps/unix/syscall-template.S:82 > #1 0x00007fae5b94143c in __sleep (seconds=0) at > ../sysdeps/unix/sysv/linux/sleep.c:138 > #2 0x000000000056cc48 in PetscSleep (s=10) at psleep.c:56 > #3 0x0000000000838887 in PetscAttachDebugger () at adebug.c:410 > #4 0x00000000005590a7 in PetscOptionsCheckInitial_Private () at init.c:392 > #5 0x000000000055e40e in PetscInitialize (argc=0x7ffff403debc, > args=0x7ffff403deb0, file=0x0, > help=0x0) at pinit.c:639 > #6 0x0000000000524a16 in PetscSolver::InitializePetsc > (argc=0x7ffff403debc, argv=0x7ffff403deb0) > at /home/dsz/src/framework/trunk/solve/PetscSolver.cxx:124 > #7 0x00000000004c404f in main (argc=4, argv=0x7ffff403e4c8) > at /home/dsz/src/framework/trunk/solve/cd3t10mpi_main.cxx:526 > (gdb) > > PetscSolver.cxx:124: > > ierr = PetscInitialize(argc, argv, (char *)0, (char *)0); > CHKERRQ(ierr); > > Hmmm, not very helpful..... > > The app runs on one cpu, but silently crashes on two. > > Any hints are very appreciated. > > Dominik > > > > On Fri, Aug 19, 2011 at 5:49 PM, Satish Balay wrote: > > On Fri, 19 Aug 2011, Dominik Szczerba wrote: > > > >> Hi, > >> > >> I am starting my app in the debugger as: > >> > >> mpiexec -np 2 sm3t4mpi run.xml -start_in_debugger -display :0.0 > >> > >> In the console I get: > >> > >> [1]PETSC ERROR: MPI error 14 > >> > >> in the two open terminals with gdb I get: > >> > >> 0x00007f2ecdd15590 in __nanosleep_nocancel () at > >> ../sysdeps/unix/syscall-template.S:82 > >> 82 ../sysdeps/unix/syscall-template.S: No such file or directory. > >> in ../sysdeps/unix/syscall-template.S > >> (gdb) > >> > >> > >> I type 'c' nonetheless and see: > >> > >> (gdb) c > >> Continuing. > >> [New Thread 0x7f268e975700 (LWP 22388)] > >> > >> Program received signal SIGABRT, Aborted. > >> 0x00007f268f421d05 in raise (sig=6) at > >> ../nptl/sysdeps/unix/sysv/linux/raise.c:64 > >> 64 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or > directory. > >> in ../nptl/sysdeps/unix/sysv/linux/raise.c > >> > >> > >> > >> How do I go on debugging? > > > > what do you get for: > > > > (gdb) where > > > > Satish > > > > > >> > >> Many thanks for any hints, > >> > >> Dominik > >> > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Fri Aug 19 12:22:51 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 19 Aug 2011 19:22:51 +0200 Subject: [petsc-users] debugger fails In-Reply-To: References: Message-ID: What do you mean by "the second break"? Dominik On Fri, Aug 19, 2011 at 6:47 PM, Aron Ahmadia wrote: > You want to do a 'where' on the second break, when your program is raising > an abort signal... > A > > On Fri, Aug 19, 2011 at 6:57 PM, Dominik Szczerba > wrote: >> >> (gdb) where >> #0 ?0x00007fae5b941590 in __nanosleep_nocancel () at >> ../sysdeps/unix/syscall-template.S:82 >> #1 ?0x00007fae5b94143c in __sleep (seconds=0) at >> ../sysdeps/unix/sysv/linux/sleep.c:138 >> #2 ?0x000000000056cc48 in PetscSleep (s=10) at psleep.c:56 >> #3 ?0x0000000000838887 in PetscAttachDebugger () at adebug.c:410 >> #4 ?0x00000000005590a7 in PetscOptionsCheckInitial_Private () at >> init.c:392 >> #5 ?0x000000000055e40e in PetscInitialize (argc=0x7ffff403debc, >> args=0x7ffff403deb0, file=0x0, >> ? ?help=0x0) at pinit.c:639 >> #6 ?0x0000000000524a16 in PetscSolver::InitializePetsc >> (argc=0x7ffff403debc, argv=0x7ffff403deb0) >> ? ?at /home/dsz/src/framework/trunk/solve/PetscSolver.cxx:124 >> #7 ?0x00000000004c404f in main (argc=4, argv=0x7ffff403e4c8) >> ? ?at /home/dsz/src/framework/trunk/solve/cd3t10mpi_main.cxx:526 >> (gdb) >> >> PetscSolver.cxx:124: >> >> ? ? ? ?ierr = PetscInitialize(argc, argv, (char *)0, (char *)0); >> CHKERRQ(ierr); >> >> Hmmm, not very helpful..... >> >> The app runs on one cpu, but silently crashes on two. >> >> Any hints are very appreciated. >> >> Dominik >> >> >> >> On Fri, Aug 19, 2011 at 5:49 PM, Satish Balay wrote: >> > On Fri, 19 Aug 2011, Dominik Szczerba wrote: >> > >> >> Hi, >> >> >> >> I am starting my app in the debugger as: >> >> >> >> mpiexec -np 2 sm3t4mpi run.xml -start_in_debugger -display :0.0 >> >> >> >> In the console I get: >> >> >> >> [1]PETSC ERROR: MPI error 14 >> >> >> >> in the two open terminals with gdb I get: >> >> >> >> 0x00007f2ecdd15590 in __nanosleep_nocancel () at >> >> ../sysdeps/unix/syscall-template.S:82 >> >> 82 ? ? ?../sysdeps/unix/syscall-template.S: No such file or directory. >> >> ? ? ? ? in ../sysdeps/unix/syscall-template.S >> >> (gdb) >> >> >> >> >> >> I type 'c' nonetheless and see: >> >> >> >> (gdb) c >> >> Continuing. >> >> [New Thread 0x7f268e975700 (LWP 22388)] >> >> >> >> Program received signal SIGABRT, Aborted. >> >> 0x00007f268f421d05 in raise (sig=6) at >> >> ../nptl/sysdeps/unix/sysv/linux/raise.c:64 >> >> 64 ? ? ?../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or >> >> directory. >> >> ? ? ? ? in ../nptl/sysdeps/unix/sysv/linux/raise.c >> >> >> >> >> >> >> >> How do I go on debugging? >> > >> > what do you get for: >> > >> > (gdb) where >> > >> > Satish >> > >> > >> >> >> >> Many thanks for any hints, >> >> >> >> Dominik >> >> >> > >> > > > From aron.ahmadia at kaust.edu.sa Fri Aug 19 12:28:27 2011 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Fri, 19 Aug 2011 20:28:27 +0300 Subject: [petsc-users] debugger fails In-Reply-To: References: Message-ID: The debugger stops when you start up, that's this code [1]. Then you want to hit 'continue' so your job runs normally to where it fails. You can also set a break point on PetscError since PETSc is catching the error from MPI. When you stop at the 'second breakpoint', you'll be at the part where your code has detected an error condition in MPI. Type a 'where' there to get the stack when the error was detected. [1] (gdb) where #0 0x00007fae5b941590 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:82 #1 0x00007fae5b94143c in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138 #2 0x000000000056cc48 in PetscSleep (s=10) at psleep.c:56 #3 0x0000000000838887 in PetscAttachDebugger () at adebug.c:410 #4 0x00000000005590a7 in PetscOptionsCheckInitial_Private () at init.c:392 #5 0x000000000055e40e in PetscInitialize (argc=0x7ffff403debc, args=0x7ffff403deb0, file=0x0, help=0x0) at pinit.c:639 #6 0x0000000000524a16 in PetscSolver::InitializePetsc (argc=0x7ffff403debc, argv=0x7ffff403deb0) at /home/dsz/src/framework/trunk/solve/PetscSolver.cxx:124 #7 0x00000000004c404f in main (argc=4, argv=0x7ffff403e4c8) at /home/dsz/src/framework/trunk/solve/cd3t10mpi_main.cxx:526 (gdb) On Fri, Aug 19, 2011 at 8:22 PM, Dominik Szczerba wrote: > What do you mean by "the second break"? > > Dominik > > On Fri, Aug 19, 2011 at 6:47 PM, Aron Ahmadia > wrote: > > You want to do a 'where' on the second break, when your program is > raising > > an abort signal... > > A > > > > On Fri, Aug 19, 2011 at 6:57 PM, Dominik Szczerba > > wrote: > >> > >> (gdb) where > >> #0 0x00007fae5b941590 in __nanosleep_nocancel () at > >> ../sysdeps/unix/syscall-template.S:82 > >> #1 0x00007fae5b94143c in __sleep (seconds=0) at > >> ../sysdeps/unix/sysv/linux/sleep.c:138 > >> #2 0x000000000056cc48 in PetscSleep (s=10) at psleep.c:56 > >> #3 0x0000000000838887 in PetscAttachDebugger () at adebug.c:410 > >> #4 0x00000000005590a7 in PetscOptionsCheckInitial_Private () at > >> init.c:392 > >> #5 0x000000000055e40e in PetscInitialize (argc=0x7ffff403debc, > >> args=0x7ffff403deb0, file=0x0, > >> help=0x0) at pinit.c:639 > >> #6 0x0000000000524a16 in PetscSolver::InitializePetsc > >> (argc=0x7ffff403debc, argv=0x7ffff403deb0) > >> at /home/dsz/src/framework/trunk/solve/PetscSolver.cxx:124 > >> #7 0x00000000004c404f in main (argc=4, argv=0x7ffff403e4c8) > >> at /home/dsz/src/framework/trunk/solve/cd3t10mpi_main.cxx:526 > >> (gdb) > >> > >> PetscSolver.cxx:124: > >> > >> ierr = PetscInitialize(argc, argv, (char *)0, (char *)0); > >> CHKERRQ(ierr); > >> > >> Hmmm, not very helpful..... > >> > >> The app runs on one cpu, but silently crashes on two. > >> > >> Any hints are very appreciated. > >> > >> Dominik > >> > >> > >> > >> On Fri, Aug 19, 2011 at 5:49 PM, Satish Balay > wrote: > >> > On Fri, 19 Aug 2011, Dominik Szczerba wrote: > >> > > >> >> Hi, > >> >> > >> >> I am starting my app in the debugger as: > >> >> > >> >> mpiexec -np 2 sm3t4mpi run.xml -start_in_debugger -display :0.0 > >> >> > >> >> In the console I get: > >> >> > >> >> [1]PETSC ERROR: MPI error 14 > >> >> > >> >> in the two open terminals with gdb I get: > >> >> > >> >> 0x00007f2ecdd15590 in __nanosleep_nocancel () at > >> >> ../sysdeps/unix/syscall-template.S:82 > >> >> 82 ../sysdeps/unix/syscall-template.S: No such file or > directory. > >> >> in ../sysdeps/unix/syscall-template.S > >> >> (gdb) > >> >> > >> >> > >> >> I type 'c' nonetheless and see: > >> >> > >> >> (gdb) c > >> >> Continuing. > >> >> [New Thread 0x7f268e975700 (LWP 22388)] > >> >> > >> >> Program received signal SIGABRT, Aborted. > >> >> 0x00007f268f421d05 in raise (sig=6) at > >> >> ../nptl/sysdeps/unix/sysv/linux/raise.c:64 > >> >> 64 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or > >> >> directory. > >> >> in ../nptl/sysdeps/unix/sysv/linux/raise.c > >> >> > >> >> > >> >> > >> >> How do I go on debugging? > >> > > >> > what do you get for: > >> > > >> > (gdb) where > >> > > >> > Satish > >> > > >> > > >> >> > >> >> Many thanks for any hints, > >> >> > >> >> Dominik > >> >> > >> > > >> > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Aug 19 12:53:42 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 19 Aug 2011 12:53:42 -0500 Subject: [petsc-users] industrial support for commercial packaging of PETSc Message-ID: <0C4200C1-8343-419D-A524-94DB4E0C8C03@mcs.anl.gov> PETSc users, Could any industrial/commerical users of PETSc who have any interest in the commercial packaging of PETSc please contact me if they would like to see that happening? PETSc will, of course, remain completely open source but we are investigating packaging models for PETSc industrial/commercial users with professional support and need to see some interest before proceeding. Thanks Barry From dominik at itis.ethz.ch Fri Aug 19 14:16:15 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 19 Aug 2011 21:16:15 +0200 Subject: [petsc-users] debugger fails In-Reply-To: References: Message-ID: Many thanks for the hint. Hitting "c" never returns the prompt, hitting "ctrl+c" and then "where" reveals a deadlock. Thanks again! Dominik On Fri, Aug 19, 2011 at 7:28 PM, Aron Ahmadia wrote: > The debugger stops when you start up, that's this code [1]. ?Then you want > to hit 'continue' so your job runs normally to where it fails. ?You can also > set a break point on PetscError since PETSc is catching the error from MPI. > ?When you stop at the 'second breakpoint', you'll be at the part where your > code has detected an error condition in MPI. ?Type a 'where' there to get > the stack when the error was detected. > [1] > (gdb) where > #0 ?0x00007fae5b941590 in __nanosleep_nocancel () at > ../sysdeps/unix/syscall-template.S:82 > #1 ?0x00007fae5b94143c in __sleep (seconds=0) at > ../sysdeps/unix/sysv/linux/sleep.c:138 > #2 ?0x000000000056cc48 in PetscSleep (s=10) at psleep.c:56 > #3 ?0x0000000000838887 in PetscAttachDebugger () at adebug.c:410 > #4 ?0x00000000005590a7 in PetscOptionsCheckInitial_Private () at init.c:392 > #5 ?0x000000000055e40e in PetscInitialize (argc=0x7ffff403debc, > args=0x7ffff403deb0, file=0x0, > ? ?help=0x0) at pinit.c:639 > #6 ?0x0000000000524a16 in PetscSolver::InitializePetsc > (argc=0x7ffff403debc, argv=0x7ffff403deb0) > ? ?at /home/dsz/src/framework/trunk/solve/PetscSolver.cxx:124 > #7 ?0x00000000004c404f in main (argc=4, argv=0x7ffff403e4c8) > ? ?at /home/dsz/src/framework/trunk/solve/cd3t10mpi_main.cxx:526 > (gdb) > > > > On Fri, Aug 19, 2011 at 8:22 PM, Dominik Szczerba > wrote: >> >> What do you mean by "the second break"? >> >> Dominik >> >> On Fri, Aug 19, 2011 at 6:47 PM, Aron Ahmadia >> wrote: >> > You want to do a 'where' on the second break, when your program is >> > raising >> > an abort signal... >> > A >> > >> > On Fri, Aug 19, 2011 at 6:57 PM, Dominik Szczerba >> > wrote: >> >> >> >> (gdb) where >> >> #0 ?0x00007fae5b941590 in __nanosleep_nocancel () at >> >> ../sysdeps/unix/syscall-template.S:82 >> >> #1 ?0x00007fae5b94143c in __sleep (seconds=0) at >> >> ../sysdeps/unix/sysv/linux/sleep.c:138 >> >> #2 ?0x000000000056cc48 in PetscSleep (s=10) at psleep.c:56 >> >> #3 ?0x0000000000838887 in PetscAttachDebugger () at adebug.c:410 >> >> #4 ?0x00000000005590a7 in PetscOptionsCheckInitial_Private () at >> >> init.c:392 >> >> #5 ?0x000000000055e40e in PetscInitialize (argc=0x7ffff403debc, >> >> args=0x7ffff403deb0, file=0x0, >> >> ? ?help=0x0) at pinit.c:639 >> >> #6 ?0x0000000000524a16 in PetscSolver::InitializePetsc >> >> (argc=0x7ffff403debc, argv=0x7ffff403deb0) >> >> ? ?at /home/dsz/src/framework/trunk/solve/PetscSolver.cxx:124 >> >> #7 ?0x00000000004c404f in main (argc=4, argv=0x7ffff403e4c8) >> >> ? ?at /home/dsz/src/framework/trunk/solve/cd3t10mpi_main.cxx:526 >> >> (gdb) >> >> >> >> PetscSolver.cxx:124: >> >> >> >> ? ? ? ?ierr = PetscInitialize(argc, argv, (char *)0, (char *)0); >> >> CHKERRQ(ierr); >> >> >> >> Hmmm, not very helpful..... >> >> >> >> The app runs on one cpu, but silently crashes on two. >> >> >> >> Any hints are very appreciated. >> >> >> >> Dominik >> >> >> >> >> >> >> >> On Fri, Aug 19, 2011 at 5:49 PM, Satish Balay >> >> wrote: >> >> > On Fri, 19 Aug 2011, Dominik Szczerba wrote: >> >> > >> >> >> Hi, >> >> >> >> >> >> I am starting my app in the debugger as: >> >> >> >> >> >> mpiexec -np 2 sm3t4mpi run.xml -start_in_debugger -display :0.0 >> >> >> >> >> >> In the console I get: >> >> >> >> >> >> [1]PETSC ERROR: MPI error 14 >> >> >> >> >> >> in the two open terminals with gdb I get: >> >> >> >> >> >> 0x00007f2ecdd15590 in __nanosleep_nocancel () at >> >> >> ../sysdeps/unix/syscall-template.S:82 >> >> >> 82 ? ? ?../sysdeps/unix/syscall-template.S: No such file or >> >> >> directory. >> >> >> ? ? ? ? in ../sysdeps/unix/syscall-template.S >> >> >> (gdb) >> >> >> >> >> >> >> >> >> I type 'c' nonetheless and see: >> >> >> >> >> >> (gdb) c >> >> >> Continuing. >> >> >> [New Thread 0x7f268e975700 (LWP 22388)] >> >> >> >> >> >> Program received signal SIGABRT, Aborted. >> >> >> 0x00007f268f421d05 in raise (sig=6) at >> >> >> ../nptl/sysdeps/unix/sysv/linux/raise.c:64 >> >> >> 64 ? ? ?../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or >> >> >> directory. >> >> >> ? ? ? ? in ../nptl/sysdeps/unix/sysv/linux/raise.c >> >> >> >> >> >> >> >> >> >> >> >> How do I go on debugging? >> >> > >> >> > what do you get for: >> >> > >> >> > (gdb) where >> >> > >> >> > Satish >> >> > >> >> > >> >> >> >> >> >> Many thanks for any hints, >> >> >> >> >> >> Dominik >> >> >> >> >> > >> >> > >> > >> > > > From zhenglun.wei at gmail.com Fri Aug 19 14:55:16 2011 From: zhenglun.wei at gmail.com (Alan Wei) Date: Fri, 19 Aug 2011 14:55:16 -0500 Subject: [petsc-users] Creating a 2d geometry by using vector In-Reply-To: <441A6F5C-CD4C-4C50-B344-A135F35126E6@mcs.anl.gov> References: <441A6F5C-CD4C-4C50-B344-A135F35126E6@mcs.anl.gov> Message-ID: Dear Barry, In ex19, the X is a kinda of global vec, while the x is a local vec. Since DAVecGetArray makes the x and X sharing the same actual vector data, the vec X is, actually, a global copy of the vec x, isnt it? best, Alan On Thu, Aug 18, 2011 at 10:12 PM, Barry Smith wrote: > > On Aug 18, 2011, at 9:40 PM, Alan Wei wrote: > > > Dear Barry, > > That is really a great idea, and it is something that I want. I've > never noticed that the structure in C is so marvelous. However, I have a > further question. In that example, a variable x is defined like Field **x, > and DAVecGetArray(da,X,&x). I wonder if there is any relationship between > da, X and x after using DAVecGetArray. Or it just a prerequisite for using > the nice notation of 'x'. > > The only real relationship is that x looks into the X data structure to > allow you to access the data as if it were in simple arrays. In other words > the x and X share the same actual vector data. > > Barry > > > > > > thanks, > > > > Alan > > > > On Thu, Aug 18, 2011 at 9:09 PM, Barry Smith wrote: > > > > Alan, > > > > I'm not sure what you are getting at but we commonly use tricks like > (see src/snes/examples/tutorials/ex19.c) > > > > typedef struct { > > PetscScalar u,v,omega,temp; > > } Field; > > > > Then in this case use a DMDAVecGetArray() to get f and x and then have > nice notation like > > > > f[j][i].u = x[j][i].u; > > f[j][i].v = x[j][i].v; > > f[j][i].omega = x[j][i].omega - (x[j][i+1].v - x[j][i].v)*dhx; > > f[j][i].temp = x[j][i].temp; > > > > note that this means the values are stored interlaced in the vector. > > > > Barry > > > > On Aug 18, 2011, at 9:03 PM, Alan Wei wrote: > > > > > Dear All, > > > I'm trying to 'draw' a 2-D cylinder in PETSc. It simply restore the > x and y coordinate values in two arrays. I wonder if I can use Vec in PETSc > to implement it (i.e. defining a vec circle) so that I can call its x or y > coordinate by circle.x or circle.y. Is this possible? also, is this a better > way than just defining it as two arrays, i.e. x[ ] and y[ ]. > > > > > > thanks in advance, > > > Alan > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Aug 19 15:51:37 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 19 Aug 2011 15:51:37 -0500 Subject: [petsc-users] Creating a 2d geometry by using vector In-Reply-To: References: <441A6F5C-CD4C-4C50-B344-A135F35126E6@mcs.anl.gov> Message-ID: On Aug 19, 2011, at 2:55 PM, Alan Wei wrote: > Dear Barry, > In ex19, the X is a kinda of global vec, while the x is a local vec. Since DAVecGetArray makes the x and X sharing the same actual vector data, the vec X is, actually, a global copy of the vec x, isnt it? Not really. If you pass a local (nonparallel) vector to the VecGetArray() the resulting array covers all the entries in the vector. If you pass a global (parallel) vector then the resulting array only gives you access to the values that are on that one process. Barry > > best, > Alan > > On Thu, Aug 18, 2011 at 10:12 PM, Barry Smith wrote: > > On Aug 18, 2011, at 9:40 PM, Alan Wei wrote: > > > Dear Barry, > > That is really a great idea, and it is something that I want. I've never noticed that the structure in C is so marvelous. However, I have a further question. In that example, a variable x is defined like Field **x, and DAVecGetArray(da,X,&x). I wonder if there is any relationship between da, X and x after using DAVecGetArray. Or it just a prerequisite for using the nice notation of 'x'. > > The only real relationship is that x looks into the X data structure to allow you to access the data as if it were in simple arrays. In other words the x and X share the same actual vector data. > > Barry > > > > > > thanks, > > > > Alan > > > > On Thu, Aug 18, 2011 at 9:09 PM, Barry Smith wrote: > > > > Alan, > > > > I'm not sure what you are getting at but we commonly use tricks like (see src/snes/examples/tutorials/ex19.c) > > > > typedef struct { > > PetscScalar u,v,omega,temp; > > } Field; > > > > Then in this case use a DMDAVecGetArray() to get f and x and then have nice notation like > > > > f[j][i].u = x[j][i].u; > > f[j][i].v = x[j][i].v; > > f[j][i].omega = x[j][i].omega - (x[j][i+1].v - x[j][i].v)*dhx; > > f[j][i].temp = x[j][i].temp; > > > > note that this means the values are stored interlaced in the vector. > > > > Barry > > > > On Aug 18, 2011, at 9:03 PM, Alan Wei wrote: > > > > > Dear All, > > > I'm trying to 'draw' a 2-D cylinder in PETSc. It simply restore the x and y coordinate values in two arrays. I wonder if I can use Vec in PETSc to implement it (i.e. defining a vec circle) so that I can call its x or y coordinate by circle.x or circle.y. Is this possible? also, is this a better way than just defining it as two arrays, i.e. x[ ] and y[ ]. > > > > > > thanks in advance, > > > Alan > > > > > > From zhenglun.wei at gmail.com Fri Aug 19 16:40:24 2011 From: zhenglun.wei at gmail.com (Alan Wei) Date: Fri, 19 Aug 2011 16:40:24 -0500 Subject: [petsc-users] Creating a 2d geometry by using vector In-Reply-To: References: <441A6F5C-CD4C-4C50-B344-A135F35126E6@mcs.anl.gov> Message-ID: Thanks Barry, ^_^ Alan On Fri, Aug 19, 2011 at 3:51 PM, Barry Smith wrote: > > On Aug 19, 2011, at 2:55 PM, Alan Wei wrote: > > > Dear Barry, > > In ex19, the X is a kinda of global vec, while the x is a local vec. > Since DAVecGetArray makes the x and X sharing the same actual vector data, > the vec X is, actually, a global copy of the vec x, isnt it? > > Not really. If you pass a local (nonparallel) vector to the VecGetArray() > the resulting array covers all the entries in the vector. If you pass a > global (parallel) vector then the resulting array only gives you access to > the values that are on that one process. > > Barry > > > > > best, > > Alan > > > > On Thu, Aug 18, 2011 at 10:12 PM, Barry Smith > wrote: > > > > On Aug 18, 2011, at 9:40 PM, Alan Wei wrote: > > > > > Dear Barry, > > > That is really a great idea, and it is something that I want. I've > never noticed that the structure in C is so marvelous. However, I have a > further question. In that example, a variable x is defined like Field **x, > and DAVecGetArray(da,X,&x). I wonder if there is any relationship between > da, X and x after using DAVecGetArray. Or it just a prerequisite for using > the nice notation of 'x'. > > > > The only real relationship is that x looks into the X data structure to > allow you to access the data as if it were in simple arrays. In other words > the x and X share the same actual vector data. > > > > Barry > > > > > > > > > > thanks, > > > > > > Alan > > > > > > On Thu, Aug 18, 2011 at 9:09 PM, Barry Smith > wrote: > > > > > > Alan, > > > > > > I'm not sure what you are getting at but we commonly use tricks like > (see src/snes/examples/tutorials/ex19.c) > > > > > > typedef struct { > > > PetscScalar u,v,omega,temp; > > > } Field; > > > > > > Then in this case use a DMDAVecGetArray() to get f and x and then have > nice notation like > > > > > > f[j][i].u = x[j][i].u; > > > f[j][i].v = x[j][i].v; > > > f[j][i].omega = x[j][i].omega - (x[j][i+1].v - x[j][i].v)*dhx; > > > f[j][i].temp = x[j][i].temp; > > > > > > note that this means the values are stored interlaced in the vector. > > > > > > Barry > > > > > > On Aug 18, 2011, at 9:03 PM, Alan Wei wrote: > > > > > > > Dear All, > > > > I'm trying to 'draw' a 2-D cylinder in PETSc. It simply restore > the x and y coordinate values in two arrays. I wonder if I can use Vec in > PETSc to implement it (i.e. defining a vec circle) so that I can call its x > or y coordinate by circle.x or circle.y. Is this possible? also, is this a > better way than just defining it as two arrays, i.e. x[ ] and y[ ]. > > > > > > > > thanks in advance, > > > > Alan > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenglun.wei at gmail.com Fri Aug 19 16:52:19 2011 From: zhenglun.wei at gmail.com (Alan Wei) Date: Fri, 19 Aug 2011 16:52:19 -0500 Subject: [petsc-users] Associate Values with DM variable Message-ID: Dear all, I have a question of the access of the DM in PETSc. 1) I used DMDACreate2d(..., &da), where 'da' is a DM parameter; 2) and a DMDASetUniformCoordinates(da, ...) was used to set up a coordinate system on 'da'. 3) Then I used DMDAGetCoordinateDA(da, &cda); DMDAGetGhostedCoordinates(da, &gc); DMDAVecGetArray(cda, gc, &coors); So that I can access the coordinate values of each points in 'da'. 4) My question is, 4.1) in spite of 'coors', is there any other values stored in 'da' for every points of it; 4.2) I want to store other parameters associating with 'da', what should I do for that? for example, since 'da' has a coordinate, I treat it as a uniform mesh. In addition to coordinate values, I hope I can store u and v (x/y-direction velocity magnitude) in each node of this coordinate. I thought about use DMDACreate2d to create other two distributed matrices. However, I'm not sure that these three DM (including 'da') has the same allocation in parallel computational nodes. Like, 'da' is assigned x = 0~4 to rank-0 and x = 5~10 to rank-1; while DM for 'u' is arranged x = 0~5 to rank-0 and x = 6~10 to rank-1. Thanks, Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Aug 19 16:56:16 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 19 Aug 2011 16:56:16 -0500 Subject: [petsc-users] Associate Values with DM variable In-Reply-To: References: Message-ID: <0A928386-D899-4BB4-8890-3C67E9A75BF1@mcs.anl.gov> On Aug 19, 2011, at 4:52 PM, Alan Wei wrote: > Dear all, > I have a question of the access of the DM in PETSc. > 1) I used > DMDACreate2d(..., &da), where 'da' is a DM parameter; > > 2) and a DMDASetUniformCoordinates(da, ...) was used to set up a coordinate system on 'da'. > > 3) Then I used > DMDAGetCoordinateDA(da, &cda); > DMDAGetGhostedCoordinates(da, &gc); > DMDAVecGetArray(cda, gc, &coors); > So that I can access the coordinate values of each points in 'da'. > > 4) My question is, > 4.1) in spite of 'coors', is there any other values stored in 'da' for every points of it; Nope. > 4.2) I want to store other parameters associating with 'da', what should I do for that? Those are never stored directly in the DA, each of those would be stored in a separate Vec obtained with DMGetGlobalVector(). If you want to carry around a bunch of things like that then you can make a struct where you keep them together for easy moving around. Barry > for example, since 'da' has a coordinate, I treat it as a uniform mesh. In addition to coordinate values, I hope I can store u and v (x/y-direction velocity magnitude) in each node of this coordinate. I thought about use DMDACreate2d to create other two distributed matrices. However, I'm not sure that these three DM (including 'da') has the same allocation in parallel computational nodes. Like, 'da' is assigned x = 0~4 to rank-0 and x = 5~10 to rank-1; while DM for 'u' is arranged x = 0~5 to rank-0 and x = 6~10 to rank-1. > > Thanks, > Alan From jedbrown at mcs.anl.gov Sat Aug 20 20:53:12 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 20 Aug 2011 18:53:12 -0700 Subject: [petsc-users] BiCGSTAB for general use In-Reply-To: References: <83F76379-F270-48A3-B498-C69934D83F82@gmail.com> <5236C63B-633D-4048-8EB6-5BDB9A5BE339@gmail.com> <8110936E-C186-4DC4-879B-912CBC849E7F@gmail.com> Message-ID: On Wed, Aug 17, 2011 at 10:48, Paul Anton Letnes < paul.anton.letnes at gmail.com> wrote: > > Let's start with a scatter plot of the eigenvalues. Can you do a problem > that is representative of the physics in less than, say, 1000 degrees of > freedom? If so, I would just use Matlab (or Octave, etc). You want to be > able to plot the eigenvector associated with a chosen eigenvalue in some way > that is meaningful to you. We want to see if the wavelength (in terms of the > variables you are discretizing over) of the modes has some useful > correlation with the size of the associated eigenvalues. If so, we may be > able to build some sort of multigrid preconditioner. > > 1000 degrees of freedom is a bit little. I took a 4608 x 4608 matrix and > plotted its eigenvalues as a scatterplot in the complex plane, as well as > the magnitude of the eigenvalues. See attachments. I can give out better > quality plots off-list. > This looks tricky. It certainly doesn't have the nice structure of a second-kind integral operator with compact kernel (or even first-kind, which would be the limit of vanishing regularization). If we're going to solve this efficiently, we probably need either: 1. A sparse system that approximates this one. This is probably unlikely, but you might have more insight. 2. A transformation that exposes sparsity. Usually these are hierarchical. Are there any fast transforms that can be used for your operator (e.g. FMM, H-matrices)? > I'm really not sure as to how one can visualize eigenvectors with 4608 > elements... > Your independent variables discretize a 2D space of angles, right? Then try a 2D color plot. > I have not thought too much about the physical meaning of eigenvectors and > eigenvalues. In fact, even this system is too small to be of physical > interest, so I'm not sure what I'd get out of it, to be honest. I suppose > some eigenvectors might be related to surface plasmon polaritons, and one > eigenvector is probably related to the specular ("mirror-like") peak. > For a problem like this, singular values might be more significant. You could plot the right and left singular vectors which would (if I understand the problem correctly correspond to incident and outgoing waveforms. It is worth trying -ksp_type cgne (conjugate gradients on the normal equations) in case the singular values are better behaved than the eigenvalues. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Sun Aug 21 15:24:25 2011 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Sun, 21 Aug 2011 16:24:25 -0400 Subject: [petsc-users] Freeing Preconditioner Matrix Space Message-ID: Hello I am attempting to implement a "hack" that was posted on the list a while back. I'm working with the adjoint linear system solver for a CFD solver. I'm using the ASM (or Block Jacobi) preconditioner with ILU(p) on each of the sub-domains. I use a different Preconditioner matrix (Pmat) than the actual jacobian. What I want to do is destroy the Pmat memory after the ILU factorization is performed. The hack that was posted is copied below: * PCASMFreeSpace(PC pc) ** { ** PC_ASM *osm = (PC_ASM*)pc->data; ** PetscErrorCode ierr; ** ** if (osm->pmat) { ** if (osm->n_local_true > 0) { ** ierr = MatDestroyMatrices(osm->n_local_true,&osm->pmat);CHKERRQ(ierr); ** } ** osm->pmat = 0; ** } ** if (pc->pmat) {ierr = MatDestroy(pc->pmat);CHKERRQ(ierr); pc->pmat = 0;} ** return 0; ** } * However, I've had no luck actually getting the function compiled into petsc. There are no erorrs reported with i type "make" in the asm directory, but when I try to use the function in my application it can't find the symbol while linking. Where does it go in the asm.c file? Does it use "static PetscErrorCode" or "PetscErrorCode PETSCKSP_DLLEXPORT"? Does it have to be added to the .h include files? What has to be done for it work with Fortran? Any suggestions would be greatly appreciated. This represents a significant chunk of my application's memory (10%-30%) and as such its too much to ignore. Also is there any chance something like this would make it into an actual PETSc release? Gaetan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Aug 21 17:22:28 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 21 Aug 2011 17:22:28 -0500 Subject: [petsc-users] Freeing Preconditioner Matrix Space In-Reply-To: References: Message-ID: <953121EF-B6AC-42EE-87BE-D4402C121652@mcs.anl.gov> You don't need to put that in the PETSc source. Just built it in the same directory you build your application and link it in like any of your application code. You will need to stick #include /*I "petscpc.h" I*/ typedef struct { PetscInt n, n_local, n_local_true; PetscInt overlap; /* overlap requested by user */ KSP *ksp; /* linear solvers for each block */ VecScatter *restriction; /* mapping from global to subregion */ VecScatter *localization; /* mapping from overlapping to non-overlapping subregion */ VecScatter *prolongation; /* mapping from subregion to global */ Vec *x,*y,*y_local; /* work vectors */ IS *is; /* index set that defines each overlapping subdomain */ IS *is_local; /* index set that defines each non-overlapping subdomain, may be NULL */ Mat *mat,*pmat; /* mat is not currently used */ PCASMType type; /* use reduced interpolation, restriction or both */ PetscBool type_set; /* if user set this value (so won't change it for symmetric problems) */ PetscBool same_local_solves; /* flag indicating whether all local solvers are same */ PetscBool sort_indices; /* flag to sort subdomain indices */ } PC_ASM; before the subroutine. Barry On Aug 21, 2011, at 3:24 PM, Gaetan Kenway wrote: > Hello > > I am attempting to implement a "hack" that was posted on the list a while back. I'm working with the adjoint linear system solver for a CFD solver. I'm using the ASM (or Block Jacobi) preconditioner with ILU(p) on each of the sub-domains. I use a different Preconditioner matrix (Pmat) than the actual jacobian. What I want to do is destroy the Pmat memory after the ILU factorization is performed. The hack that was posted is copied below: > PCASMFreeSpace(PC pc) > { > PC_ASM *osm = (PC_ASM*)pc->data; > PetscErrorCode ierr; > > if (osm->pmat) { > if (osm->n_local_true > 0) { > > ierr = MatDestroyMatrices(osm->n_local_true,&osm->pmat);CHKERRQ(ierr); > } > osm->pmat = 0; > } > if (pc->pmat) {ierr = MatDestroy(pc->pmat);CHKERRQ(ierr); pc->pmat = 0;} > > return 0; > } > > However, I've had no luck actually getting the function compiled into petsc. There are no erorrs reported with i type "make" in the asm directory, but when I try to use the function in my application it can't find the symbol while linking. Where does it go in the asm.c file? Does it use "static PetscErrorCode" or "PetscErrorCode PETSCKSP_DLLEXPORT"? Does it have to be added to the .h include files? What has to be done for it work with Fortran? > > Any suggestions would be greatly appreciated. This represents a significant chunk of my application's memory (10%-30%) and as such its too much to ignore. Also is there any chance something like this would make it into an actual PETSc release? > > Gaetan > From ram at ibrae.ac.ru Sun Aug 21 18:45:06 2011 From: ram at ibrae.ac.ru (=?UTF-8?B?0JDQu9C10LrRgdC10Lkg0KDRj9C30LDQvdC+0LI=?=) Date: Mon, 22 Aug 2011 03:45:06 +0400 Subject: [petsc-users] performance issue solving multiple linear systems of the same size with the different preconditioning methods Message-ID: Hello! Could you please help me to solve my performance problem. I have two programs. In 1st I solve one system with one method and one preconditioner and get some performance numbers. I run it 9 times with 9 different preconditioners. In 2nd I solve the same system with the same one method but with 9 different preconditioners consecutively one after another. I run it once and also get some performance info. In the 2nd case I have 2-5 times worse results, depending on used method. Each KSPSolve procedure placed in its own stage of course, so I can compare times, flops, messages an so.. I can see the difference but cant explain and eliminate it. For example for -ksp_type cgs -pc_type asm -sub_pc_type jacobi -sub_ksp_type preonly: Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total one stage frome 2nd: 5.5145e+00 14.9% 1.2336e+09 13.2% 2.494e+03 22.4% 3.230e+03 22.2% 2.250e+03 18.5% the once stage from 1st: 2.7541e+00 93.1% 1.2336e+09 99.8% 2.508e+03 99.3% 1.470e+04 97.4% 2.266e+03 88.0% My programs are pretty equivalent except the part with definition of preconditioners and the number of called KSPSolve procedures. I mean they use equal matrices, equally assemble them, use equal right hand sides, equal convergence monitors. Actually the 2nd one was made from the 1st. In 1st i use KSPSetFromOptions(KSP); and then just set the -ksp_type -pc_type -sub_pc_type -sub_ksp_type keys from command line In 2d i use for for nonblock PC: KSPGetPC(dKSP, &dPC); PCSetType(dPC, PCJACOBI); and for block PC: PCSetType(dPC, PCASM); KSPSetUp(dKSP); PCSetUp(dPC); PCASMGetSubKSP(dPC, &n_local, &first_local, &ASMSubKSP); for (i=0; i From jedbrown at mcs.anl.gov Sun Aug 21 18:54:12 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 21 Aug 2011 16:54:12 -0700 Subject: [petsc-users] performance issue solving multiple linear systems of the same size with the different preconditioning methods In-Reply-To: References: Message-ID: On Sun, Aug 21, 2011 at 16:45, ??????? ??????? wrote: > Hello! > > Could you please help me to solve my performance problem. > I have two programs. > > In 1st I solve one system with one method and one preconditioner and get > some performance numbers. > I run it 9 times with 9 different preconditioners. > > In 2nd I solve the same system with the same one method but with 9 > different preconditioners consecutively one after another. > I run it once and also get some performance info. > In the 2nd case I have 2-5 times worse results, depending on used method. > > Each KSPSolve procedure placed in its own stage of course, so I can compare > times, flops, messages an so.. > I can see the difference but cant explain and eliminate it. > > For example for -ksp_type cgs -pc_type asm -sub_pc_type jacobi > -sub_ksp_type preonly: > Summary of Stages: ----- Time ------ ----- Flops > ----- --- Messages --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg > %Total counts %Total Avg %Total counts %Total > one stage frome 2nd: 5.5145e+00 14.9% 1.2336e+09 13.2% 2.494e+03 > 22.4% 3.230e+03 22.2% 2.250e+03 18.5% > the once stage from 1st: 2.7541e+00 93.1% 1.2336e+09 99.8% 2.508e+03 > 99.3% 1.470e+04 97.4% 2.266e+03 88.0% > > > My programs are pretty equivalent except the part with definition of > preconditioners and the number of called KSPSolve procedures. > I mean they use equal matrices, equally assemble them, use equal right hand > sides, equal convergence monitors. > Actually the 2nd one was made from the 1st. > > In 1st i use KSPSetFromOptions(KSP); and then just set the -ksp_type > -pc_type -sub_pc_type -sub_ksp_type keys from command line > > In 2d i use for for nonblock PC: > > KSPGetPC(dKSP, &dPC); > > PCSetType(dPC, PCJACOBI); > > and for block PC: > > PCSetType(dPC, PCASM); > > KSPSetUp(dKSP); > > PCSetUp(dPC); > > PCASMGetSubKSP(dPC, &n_local, &first_local, &ASMSubKSP); > > for (i=0; i > { > > KSPGetPC(ASMSubKSP[i], &(SubPC[i])); > > PCSetType(SubPC[i], PCJACOBI); > > } > > > Im sure there is a mistake somewhere. Because 1st program compares Jacobi > and ASM-Jacobi preconditioners on my problem on the same KSP and tells me > that ASM-Jacobi is better and the 2nd shows otherwise results. > This could be a preload issue. You can use the PreLoadBegin()/PreLoadEnd() macros if you like, or otherwise solve a system first to make sure everything has been loaded. If the results are still confusing, run with -ksp_view -log_summary and send the output. There is no reason for ASM-Jacobi (with -sub_ksp_type preonly, which is default) to be better than Jacobi since it does the same algorithm with more communication. -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.anton.letnes at gmail.com Mon Aug 22 05:51:40 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Mon, 22 Aug 2011 11:51:40 +0100 Subject: [petsc-users] BiCGSTAB for general use In-Reply-To: References: <83F76379-F270-48A3-B498-C69934D83F82@gmail.com> <5236C63B-633D-4048-8EB6-5BDB9A5BE339@gmail.com> <8110936E-C186-4DC4-879B-912CBC849E7F@gmail.com> Message-ID: <7837C2E3-428C-4B7A-834A-B6832AFCFD27@gmail.com> > This looks tricky. It certainly doesn't have the nice structure of a second-kind integral operator with compact kernel (or even first-kind, which would be the limit of vanishing regularization). If we're going to solve this efficiently, we probably need either: > > 1. A sparse system that approximates this one. This is probably unlikely, but you might have more insight. One may exist, but how one can safely obtain one, I have no idea. > 2. A transformation that exposes sparsity. Usually these are hierarchical. Are there any fast transforms that can be used for your operator (e.g. FMM, H-matrices)? Not that I know of. The operator contains integrals over a stochastically rough surface. Due to its random nature, any sparsity structure will depend on each surface realization, so I don't think it's easy to construct a transform that will be helpful in general. > I'm really not sure as to how one can visualize eigenvectors with 4608 elements... > > Your independent variables discretize a 2D space of angles, right? Then try a 2D color plot. Sort of. That may be a good idea. Perhaps we could use something like that to learn more about the physics anyway. > For a problem like this, singular values might be more significant. You could plot the right and left singular vectors which would (if I understand the problem correctly correspond to incident and outgoing waveforms. It is worth trying -ksp_type cgne (conjugate gradients on the normal equations) in case the singular values are better behaved than the eigenvalues. I have so far not plotted or analyzed any singular vectors or eigenvectors. To simulate a physically meaningful system will probably take quite a few CPU hours... I attach a plot of singular values for a small (4608x4608) test matrix. It does not look too great at first glance. The largest singular value is 20107, the smallest 69.7, giving a ratio of 288 or so. Paul -------------- next part -------------- A non-text attachment was scrubbed... Name: svdvals.pdf Type: application/pdf Size: 70433 bytes Desc: not available URL: From amryazanov at gmail.com Mon Aug 22 06:12:28 2011 From: amryazanov at gmail.com (=?UTF-8?B?0JDQu9C10LrRgdC10Lkg0KDRj9C30LDQvdC+0LI=?=) Date: Mon, 22 Aug 2011 15:12:28 +0400 Subject: [petsc-users] performance issue solving multiple linear systems of the same size with the different preconditioning methods In-Reply-To: References: Message-ID: Thank you for your answer! I found too problematic to learn using PreLoad procedures. I tryed and failed. But of course there was no problem to change the PetscLogStagePush(StageNum1); KSPSolve(dKSP, dvec_origRHS, dvec_Solution); PetscLogStagePop(); part of my 2nd program to KSPSolve(dKSP, dvec_origRHS, dvec_Solution); PetscLogStagePush(StageNum1); KSPSolve(dKSP, dvec_origRHS, dvec_Solution); PetscLogStagePop(); to "make sure everything has been loaded". Of course out of the Stage-brackets. It was my epic fail. In this "preloaded mode" Stage statistics looks much worse. Here it is: 30x30x30 mesh 1st programm: NONE 248its : 2.4834e+00 92.6% 1.2134e+09 99.8% 1.490e+03 98.8% 1.423e+04 95.5% 1.246e+03 80.4% JACOBI 249its : 2.5627e+00 92.9% 1.2318e+09 99.8% 1.496e+03 98.8% 1.423e+04 95.5% 1.252e+03 80.4% SOR 153its : 2.7673e+00 93.2% 1.1769e+09 99.8% 9.200e+02 98.1% 1.412e+04 92.9% 7.710e+02 78.7% ASM-JACOBI 249its : 2.9196e+00 93.3% 1.2336e+09 99.8% 2.508e+03 99.3% 1.470e+04 97.4% 2.266e+03 88.0% ASM-SOR 135its : 2.7965e+00 93.2% 1.0915e+09 99.7% 1.368e+03 98.7% 1.494e+04 95.3% 1.238e+03 86.4% ASM-ILU 47its : 8.3341e-01 79.4% 3.8922e+08 99.2% 4.880e+02 96.4% 1.588e+04 88.8% 4.510e+02 80.5% 2nd programm: NONE 248its : 2.6047e+00 7.0% 1.2134e+09 13.0% 1.490e+03 13.4% 1.929e+03 13.3% 1.246e+03 10.2% JACOBI 249its : 3.0119e+00 8.1% 1.2318e+09 13.2% 1.496e+03 13.5% 1.937e+03 13.3% 1.251e+03 10.3% SOR 153its : 3.5001e+00 9.4% 1.1769e+09 12.6% 9.200e+02 8.3% 1.191e+03 8.2% 7.700e+02 6.3% ASM-JACOBI 249its : 5.4581e+00 14.6% 1.2336e+09 13.2% 2.494e+03 22.4% 3.230e+03 22.2% 2.250e+03 18.5% ASM-SOR 135its : 5.3384e+00 14.3% 1.0915e+09 11.7% 1.354e+03 12.2% 1.753e+03 12.0% 1.222e+03 10.0% ASM-ILU 47its : 1.9040e+00 5.1% 3.8922e+08 4.2% 4.740e+02 4.3% 6.138e+02 4.2% 4.350e+02 3.6% 2nd programm with "preloading" NONE 248its : 3.0820e+00 2.2% 1.2134e+09 6.5% 1.490e+03 6.7% 9.668e+02 6.7% 1.245e+03 5.1% JACOBI 249its : 4.5530e+00 3.3% 1.2318e+09 6.6% 1.496e+03 6.7% 9.707e+02 6.7% 1.250e+03 5.2% SOR 153its : 5.6799e+00 4.1% 1.1769e+09 6.3% 9.200e+02 4.1% 5.970e+02 4.1% 7.700e+02 3.2% ASM-JACOBI 249its : 1.5616e+01 11.2% 1.2336e+09 6.6% 2.494e+03 11.2% 1.618e+03 11.2% 2.248e+03 9.3% ASM-SOR 135its : 1.2508e+01 9.0% 1.0914e+09 5.9% 1.354e+03 6.1% 8.786e+02 6.1% 1.222e+03 5.0% ASM-ILU 47its : 4.0419e+00 2.9% 3.7906e+08 2.0% 4.740e+02 2.1% 3.076e+02 2.1% 4.300e+02 1.8% I also thought, that it all might be due the small number of mesh points per direction and used more complicated mesh, but got the same results: 45x45x45 mesh 1st programm NONE 727its : 3.1323e+01 97.9% 1.1996e+10 99.9% 4.364e+03 99.6% 3.227e+04 98.4% 3.641e+03 82.3% JACOBI 729its : 3.2667e+01 98.1% 1.2162e+10 99.9% 4.376e+03 99.6% 3.227e+04 98.4% 3.652e+03 82.3% SOR 495its : 3.9066e+01 98.5% 1.2901e+10 99.9% 2.972e+03 99.4% 3.220e+04 97.7% 2.481e+03 81.8% ASM-JACOBI 729its : 3.5595e+01 98.0% 1.2174e+10 99.9% 7.308e+03 99.8% 3.263e+04 99.1% 6.586e+03 89.3% ASM-SOR 468its : 4.1062e+01 98.4% 1.2608e+10 99.9% 4.698e+03 99.6% 3.276e+04 98.6% 4.235e+03 88.9% ASM-ILU 158its : 1.0881e+01 94.1% 4.2649e+09 99.8% 1.598e+03 98.9% 3.344e+04 96.0% 1.450e+03 86.8% 2nd programm NONE 727its : 3.2766e+01 8.9% 1.1996e+10 18.1% 4.364e+03 17.2% 5.585e+03 17.2% 3.641e+03 14.3% JACOBI 729its : 4.1030e+01 11.2% 1.2162e+10 18.4% 4.376e+03 17.3% 5.601e+03 17.2% 3.651e+03 14.4% SOR 495its : 5.6807e+01 15.5% 1.2901e+10 19.5% 2.972e+03 11.7% 3.804e+03 11.7% 2.480e+03 9.8% ASM-JACOBI 729its : 9.9106e+01 27.0% 1.2174e+10 18.4% 7.294e+03 28.8% 9.335e+03 28.7% 6.570e+03 25.9% ASM-SOR 468its : 9.8369e+01 28.3% 1.2608e+10 19.1% 4.684e+03 18.5% 5.995e+03 18.4% 4.219e+03 16.6% ASM-ILU 158its : 3.1968e+01 8.7% 4.2649e+09 6.4% 1.584e+03 6.3% 2.027e+03 6.2% 1.434e+03 5.6% By the way, for more accurate results, now i use the last one (45x45x45). I've also checked out the -ksp_view results. As I can see they are pretty much the same. I'm attaching -ksp_view -log_summary results from both programs. To the point: I always get this tiny petsc crush at the end of work, when i'm using -log_summary option. I think it can be caused by russian localisation of ubuntu or something like this. Actually it doesn't bug, but it happens - it often crushes at the end of work, but works properly. I use my own convergence monitor because I can't understand what's the point of estimating the preconditioned residual. So I build true residual. PetscErrorCode MyKSPConverged (KSP ksp, PetscInt it, PetscReal rnorm,KSPConvergedReason *reason, void* da) { PetscReal true_norm; PetscReal epsilon = 1.e-5; PetscInt maxits = 1500; Vec t,V; DAGetGlobalVector(da, &t); DAGetGlobalVector(da, &V); KSPBuildResidual(ksp, t, PETSC_NULL, &V); VecNorm(V, NORM_2, &true_norm); // PetscPrintf(PETSC_COMM_WORLD, "truenorm %d %20.18f\n", it, true_norm); DARestoreGlobalVector(da, &t); DARestoreGlobalVector(da, &V); *reason = 0; if (true_norm <= epsilon){ *reason = KSP_CONVERGED_ATOL; PetscPrintf(PETSC_COMM_WORLD, "RAMmonitor: KSP_Converged(): Linear solver has converged. Residual norm %e is less than absolute tolerance %e at Iteration %d\n", true_norm, epsilon, it); } if (it >= maxits){ *reason = KSP_CONVERGED_ITS; PetscPrintf(PETSC_COMM_WORLD, "RAMmonitor: Iteration %d > limit %d\n", it, maxits); } return 0; } AND THE MAIN PART: I have noticed, that when I comment the part of my 2nd programm the rest part of it begin to do such good timing results as 1st program do. In detail: Structure of my 1st prog: 0) INIT ALL 1) KSPSetFromOptions(dKSP); SOLVE: PetscLogStagePush(StageNum1); ierr = KSPSolve(dKSP, dvec_origRHS, dvec_Solution); PetscLogStagePop(); RUN IT WITH: -log_summary -ksp_type KSP -pc_type PC -sub_ksp_type subKSP -sub_pc_type subPC -ksp_view Structure of my 2nd prog: 0) INIT ALL 1) PCSetType(dPC, PCNONE); SOLVE 2) PCSetType(dPC, PCJACOBI); SOLVE 3) PCSetType(dPC, SOR); SOLVE 4) PCSetType(dPC, PCASM); KSPSetUp(dKSP); PCSetUp(dPC); PCASMGetSubKSP(dPC, &n_local, &first_local, &ASMSubKSP); for (i=0; i > On Sun, Aug 21, 2011 at 16:45, ??????? ??????? wrote: > >> Hello! >> >> Could you please help me to solve my performance problem. >> I have two programs. >> >> In 1st I solve one system with one method and one preconditioner and get >> some performance numbers. >> I run it 9 times with 9 different preconditioners. >> >> In 2nd I solve the same system with the same one method but with 9 >> different preconditioners consecutively one after another. >> I run it once and also get some performance info. >> In the 2nd case I have 2-5 times worse results, depending on used method. >> >> Each KSPSolve procedure placed in its own stage of course, so I can >> compare times, flops, messages an so.. >> I can see the difference but cant explain and eliminate it. >> >> For example for -ksp_type cgs -pc_type asm -sub_pc_type jacobi >> -sub_ksp_type preonly: >> Summary of Stages: ----- Time ------ ----- Flops >> ----- --- Messages --- -- Message Lengths -- -- Reductions -- >> Avg %Total Avg >> %Total counts %Total Avg %Total counts %Total >> one stage frome 2nd: 5.5145e+00 14.9% 1.2336e+09 13.2% 2.494e+03 >> 22.4% 3.230e+03 22.2% 2.250e+03 18.5% >> the once stage from 1st: 2.7541e+00 93.1% 1.2336e+09 99.8% 2.508e+03 >> 99.3% 1.470e+04 97.4% 2.266e+03 88.0% >> >> >> My programs are pretty equivalent except the part with definition of >> preconditioners and the number of called KSPSolve procedures. >> I mean they use equal matrices, equally assemble them, use equal right >> hand sides, equal convergence monitors. >> Actually the 2nd one was made from the 1st. >> >> In 1st i use KSPSetFromOptions(KSP); and then just set the -ksp_type >> -pc_type -sub_pc_type -sub_ksp_type keys from command line >> >> In 2d i use for for nonblock PC: >> >> KSPGetPC(dKSP, &dPC); >> >> PCSetType(dPC, PCJACOBI); >> >> and for block PC: >> >> PCSetType(dPC, PCASM); >> >> KSPSetUp(dKSP); >> >> PCSetUp(dPC); >> >> PCASMGetSubKSP(dPC, &n_local, &first_local, &ASMSubKSP); >> >> for (i=0; i> >> { >> >> KSPGetPC(ASMSubKSP[i], &(SubPC[i])); >> >> PCSetType(SubPC[i], PCJACOBI); >> >> } >> >> >> Im sure there is a mistake somewhere. Because 1st program compares Jacobi >> and ASM-Jacobi preconditioners on my problem on the same KSP and tells me >> that ASM-Jacobi is better and the 2nd shows otherwise results. >> > > This could be a preload issue. You can use the PreLoadBegin()/PreLoadEnd() > macros if you like, or otherwise solve a system first to make sure > everything has been loaded. If the results are still confusing, run with > -ksp_view -log_summary and send the output. > > There is no reason for ASM-Jacobi (with -sub_ksp_type preonly, which is > default) to be better than Jacobi since it does the same algorithm with more > communication. > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: KSPview 1st program.out Type: application/octet-stream Size: 11780 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: KSPview 2nd program 123456.out Type: application/octet-stream Size: 30805 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: KSPview 2nd program 456.out Type: application/octet-stream Size: 22794 bytes Desc: not available URL: From gjost at tacc.utexas.edu Mon Aug 22 13:35:10 2011 From: gjost at tacc.utexas.edu (Gabriele Jost) Date: Mon, 22 Aug 2011 18:35:10 +0000 Subject: [petsc-users] Number of nonzeros after assembly? Message-ID: Hello! I have a question about the number of nonzeros in a PETSc matrix (mpiaij format). The context is, that I am trying to debug a program which works fine for a certain type of problems, but gives incorrect results for a different type. Anyhow, the input matrix has 5222556 nonzeros, this is the smalles data set I can get from the user. However, if I look at the KSP info I get from PETSc, I see the message: KSP Object: type: preonly maximum iterations=5000, initial guess is zero tolerances: relative=1e-06, absolute=1e-50, divergence=10000 left preconditioning PC Object: type: lu LU: out-of-place factorization matrix ordering: natural LU: tolerance for zero pivot 1e-12 LU: factor fill ratio needed 0 Factored matrix follows Matrix Object: type=mpiaij, rows=13608, cols=13608 package used to perform factorization: superlu_dist total: nonzeros=0, allocated nonzeros=27216 SuperLU_DIST run parameters: Process grid nprow 4 x npcol 4 Equilibrate matrix TRUE Matrix input mode 1 Replace tiny pivots TRUE Use iterative refinement FALSE Processors in row 4 col partition 4 Row permutation LargeDiag Column permutation MMD_AT_PLUS_A Parallel symbolic factorization FALSE Repeated factorization SamePattern linear system matrix = precond matrix: Matrix Object: type=mpiaij, rows=13608, cols=13608 total: nonzeros=5227411, allocated nonzeros=5308814 So, there are more non-zeros than in my matrix. This discrepancy does not show up in the good cases. So, I was trying to count the number of non-zeros after the assembly stage. I do this by: ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); if (ierr != 0) SETERRQ(1,"assembly error"); { int p,row,col,start,end,icnt; icnt=0; ierr=MatGetOwnershipRangeColumn(A, &start, &end); for (row=Istarts[mytid]; row References: Message-ID: <4B53E639-1A8F-439E-8FCD-E7B0408CCE86@mcs.anl.gov> > for (col=start; col Hello! > I have a question about the number of nonzeros in a PETSc matrix (mpiaij format). > The context is, that I am trying to debug a program which works fine for a certain type of problems, but gives incorrect results for a different type. > Anyhow, the input matrix has 5222556 nonzeros, this is the smalles data set I can get from the user. However, if I look at the KSP info I get from PETSc, I see the message: > > KSP Object: > type: preonly > maximum iterations=5000, initial guess is zero > tolerances: relative=1e-06, absolute=1e-50, divergence=10000 > left preconditioning > PC Object: > type: lu > LU: out-of-place factorization > matrix ordering: natural > LU: tolerance for zero pivot 1e-12 > LU: factor fill ratio needed 0 > Factored matrix follows > Matrix Object: > type=mpiaij, rows=13608, cols=13608 > package used to perform factorization: superlu_dist > total: nonzeros=0, allocated nonzeros=27216 > SuperLU_DIST run parameters: > Process grid nprow 4 x npcol 4 > Equilibrate matrix TRUE > Matrix input mode 1 > Replace tiny pivots TRUE > Use iterative refinement FALSE > Processors in row 4 col partition 4 > Row permutation LargeDiag > Column permutation MMD_AT_PLUS_A > Parallel symbolic factorization FALSE > Repeated factorization SamePattern > linear system matrix = precond matrix: > Matrix Object: > type=mpiaij, rows=13608, cols=13608 > total: nonzeros=5227411, allocated nonzeros=5308814 > > So, there are more non-zeros than in my matrix. This discrepancy does not show up in the good cases. > So, I was trying to count the number of non-zeros after the assembly stage. I do this by: > > ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > if (ierr != 0) SETERRQ(1,"assembly error"); > { > int p,row,col,start,end,icnt; > icnt=0; > ierr=MatGetOwnershipRangeColumn(A, &start, &end); > for (row=Istarts[mytid]; row for (col=start; col PetscScalar v=0.; > ierr = MatGetValues(A,1,&row,1,&col,&v); CHKERRQ(ierr); > if (v!=0.) { > icnt++; > PetscSynchronizedPrintf(PETSC_COMM_WORLD,"%d %d %lf\n",row, col, v); > //PetscSynchronizedPrintf(PETSC_COMM_WORLD,"%d %d %lf\n",row, col,v); > } > } > } > PetscSynchronizedPrintf(PETSC_COMM_WORLD,"%s %d %d\n","Processor ",mytid,icnt); > PetscSynchronizedFlush(PETSC_COMM_WORLD); > } > > If I sum up the nonzeros for each processor, I get 591479... by far not enough. > I would greatly appreciate if somebody could advise me on what is going on here. Many thanks in advance for any hints! > > Kind regards, > Gabriele Jost From ram at ibrae.ac.ru Tue Aug 23 02:37:59 2011 From: ram at ibrae.ac.ru (=?UTF-8?B?0JDQu9C10LrRgdC10Lkg0KDRj9C30LDQvdC+0LI=?=) Date: Tue, 23 Aug 2011 11:37:59 +0400 Subject: [petsc-users] performance issue solving multiple linear systems of the same size with the different preconditioning methods In-Reply-To: References: Message-ID: Thank you for your answer! I tryed and failed to learn using PreLoad procedures. But of course there was no problem to change the PetscLogStagePush(StageNum1); KSPSolve(dKSP, dvec_origRHS, dvec_Solution); PetscLogStagePop(); part of my 2nd program to KSPSolve(dKSP, dvec_origRHS, dvec_Solution); PetscLogStagePush(StageNum1); KSPSolve(dKSP, dvec_origRHS, dvec_Solution); PetscLogStagePop(); to "make sure everything has been loaded". Of course out of the Stage-brackets. It was my epic fail. In this "preloaded mode" Stage statistics looks much worse. Here it is: 30x30x30 mesh 1st programm: NONE 248its : 2.4834e+00 92.6% 1.2134e+09 99.8% 1.490e+03 98.8% 1.423e+04 95.5% 1.246e+03 80.4% JACOBI 249its : 2.5627e+00 92.9% 1.2318e+09 99.8% 1.496e+03 98.8% 1.423e+04 95.5% 1.252e+03 80.4% SOR 153its : 2.7673e+00 93.2% 1.1769e+09 99.8% 9.200e+02 98.1% 1.412e+04 92.9% 7.710e+02 78.7% ASM-JACOBI 249its : 2.9196e+00 93.3% 1.2336e+09 99.8% 2.508e+03 99.3% 1.470e+04 97.4% 2.266e+03 88.0% ASM-SOR 135its : 2.7965e+00 93.2% 1.0915e+09 99.7% 1.368e+03 98.7% 1.494e+04 95.3% 1.238e+03 86.4% ASM-ILU 47its : 8.3341e-01 79.4% 3.8922e+08 99.2% 4.880e+02 96.4% 1.588e+04 88.8% 4.510e+02 80.5% 2nd programm: NONE 248its : 2.6047e+00 7.0% 1.2134e+09 13.0% 1.490e+03 13.4% 1.929e+03 13.3% 1.246e+03 10.2% JACOBI 249its : 3.0119e+00 8.1% 1.2318e+09 13.2% 1.496e+03 13.5% 1.937e+03 13.3% 1.251e+03 10.3% SOR 153its : 3.5001e+00 9.4% 1.1769e+09 12.6% 9.200e+02 8.3% 1.191e+03 8.2% 7.700e+02 6.3% ASM-JACOBI 249its : 5.4581e+00 14.6% 1.2336e+09 13.2% 2.494e+03 22.4% 3.230e+03 22.2% 2.250e+03 18.5% ASM-SOR 135its : 5.3384e+00 14.3% 1.0915e+09 11.7% 1.354e+03 12.2% 1.753e+03 12.0% 1.222e+03 10.0% ASM-ILU 47its : 1.9040e+00 5.1% 3.8922e+08 4.2% 4.740e+02 4.3% 6.138e+02 4.2% 4.350e+02 3.6% 2nd programm with "preloading" NONE 248its : 3.0820e+00 2.2% 1.2134e+09 6.5% 1.490e+03 6.7% 9.668e+02 6.7% 1.245e+03 5.1% JACOBI 249its : 4.5530e+00 3.3% 1.2318e+09 6.6% 1.496e+03 6.7% 9.707e+02 6.7% 1.250e+03 5.2% SOR 153its : 5.6799e+00 4.1% 1.1769e+09 6.3% 9.200e+02 4.1% 5.970e+02 4.1% 7.700e+02 3.2% ASM-JACOBI 249its : 1.5616e+01 11.2% 1.2336e+09 6.6% 2.494e+03 11.2% 1.618e+03 11.2% 2.248e+03 9.3% ASM-SOR 135its : 1.2508e+01 9.0% 1.0914e+09 5.9% 1.354e+03 6.1% 8.786e+02 6.1% 1.222e+03 5.0% ASM-ILU 47its : 4.0419e+00 2.9% 3.7906e+08 2.0% 4.740e+02 2.1% 3.076e+02 2.1% 4.300e+02 1.8% I also thought, that it all might be due the small number of mesh points per direction and used more complicated mesh, but got the same results: 45x45x45 mesh 1st programm NONE 727its : 3.1323e+01 97.9% 1.1996e+10 99.9% 4.364e+03 99.6% 3.227e+04 98.4% 3.641e+03 82.3% JACOBI 729its : 3.2667e+01 98.1% 1.2162e+10 99.9% 4.376e+03 99.6% 3.227e+04 98.4% 3.652e+03 82.3% SOR 495its : 3.9066e+01 98.5% 1.2901e+10 99.9% 2.972e+03 99.4% 3.220e+04 97.7% 2.481e+03 81.8% ASM-JACOBI 729its : 3.5595e+01 98.0% 1.2174e+10 99.9% 7.308e+03 99.8% 3.263e+04 99.1% 6.586e+03 89.3% ASM-SOR 468its : 4.1062e+01 98.4% 1.2608e+10 99.9% 4.698e+03 99.6% 3.276e+04 98.6% 4.235e+03 88.9% ASM-ILU 158its : 1.0881e+01 94.1% 4.2649e+09 99.8% 1.598e+03 98.9% 3.344e+04 96.0% 1.450e+03 86.8% 2nd programm NONE 727its : 3.2766e+01 8.9% 1.1996e+10 18.1% 4.364e+03 17.2% 5.585e+03 17.2% 3.641e+03 14.3% JACOBI 729its : 4.1030e+01 11.2% 1.2162e+10 18.4% 4.376e+03 17.3% 5.601e+03 17.2% 3.651e+03 14.4% SOR 495its : 5.6807e+01 15.5% 1.2901e+10 19.5% 2.972e+03 11.7% 3.804e+03 11.7% 2.480e+03 9.8% ASM-JACOBI 729its : 9.9106e+01 27.0% 1.2174e+10 18.4% 7.294e+03 28.8% 9.335e+03 28.7% 6.570e+03 25.9% ASM-SOR 468its : 9.8369e+01 28.3% 1.2608e+10 19.1% 4.684e+03 18.5% 5.995e+03 18.4% 4.219e+03 16.6% ASM-ILU 158its : 3.1968e+01 8.7% 4.2649e+09 6.4% 1.584e+03 6.3% 2.027e+03 6.2% 1.434e+03 5.6% By the way, for more accurate results, now i use the last one (45x45x45). I've also checked out the -ksp_view results. As I can see they are pretty much the same. I'm attaching -ksp_view -log_summary results from both programs. To the point: I always get this tiny petsc crush at the end of work, when i'm using -log_summary option. I think it can be caused by russian localisation of ubuntu or something like this. Actually it doesn't bug, but it happens - it often crushes at the end of work, but works properly. I use my own convergence monitor because I can't understand what's the point of estimating the preconditioned residual. So I build true residual. PetscErrorCode MyKSPConverged (KSP ksp, PetscInt it, PetscReal rnorm,KSPConvergedReason *reason, void* da) { PetscReal true_norm; PetscReal epsilon = 1.e-5; PetscInt maxits = 1500; Vec t,V; DAGetGlobalVector(da, &t); DAGetGlobalVector(da, &V); KSPBuildResidual(ksp, t, PETSC_NULL, &V); VecNorm(V, NORM_2, &true_norm); // PetscPrintf(PETSC_COMM_WORLD, "truenorm %d %20.18f\n", it, true_norm); DARestoreGlobalVector(da, &t); DARestoreGlobalVector(da, &V); *reason = 0; if (true_norm <= epsilon){ *reason = KSP_CONVERGED_ATOL; PetscPrintf(PETSC_COMM_WORLD, "RAMmonitor: KSP_Converged(): Linear solver has converged. Residual norm %e is less than absolute tolerance %e at Iteration %d\n", true_norm, epsilon, it); } if (it >= maxits){ *reason = KSP_CONVERGED_ITS; PetscPrintf(PETSC_COMM_WORLD, "RAMmonitor: Iteration %d > limit %d\n", it, maxits); } return 0; } AND THE MAIN PART: I have noticed, that when I comment the part of my 2nd programm the rest part of it begin to do such good timing results as 1st program do. In detail: Structure of my 1st prog: 0) INIT ALL 1) KSPSetFromOptions(dKSP); SOLVE: PetscLogStagePush(StageNum1); ierr = KSPSolve(dKSP, dvec_origRHS, dvec_Solution); PetscLogStagePop(); RUN IT WITH: -log_summary -ksp_type KSP -pc_type PC -sub_ksp_type subKSP -sub_pc_type subPC -ksp_view Structure of my 2nd prog: 0) INIT ALL 1) PCSetType(dPC, PCNONE); SOLVE 2) PCSetType(dPC, PCJACOBI); SOLVE 3) PCSetType(dPC, SOR); SOLVE 4) PCSetType(dPC, PCASM); KSPSetUp(dKSP); PCSetUp(dPC); PCASMGetSubKSP(dPC, &n_local, &first_local, &ASMSubKSP); for (i=0; i > On Sun, Aug 21, 2011 at 16:45, ??????? ??????? wrote: > >> Hello! >> >> Could you please help me to solve my performance problem. >> I have two programs. >> >> In 1st I solve one system with one method and one preconditioner and get >> some performance numbers. >> I run it 9 times with 9 different preconditioners. >> >> In 2nd I solve the same system with the same one method but with 9 >> different preconditioners consecutively one after another. >> I run it once and also get some performance info. >> In the 2nd case I have 2-5 times worse results, depending on used method. >> >> Each KSPSolve procedure placed in its own stage of course, so I can >> compare times, flops, messages an so.. >> I can see the difference but cant explain and eliminate it. >> >> For example for -ksp_type cgs -pc_type asm -sub_pc_type jacobi >> -sub_ksp_type preonly: >> Summary of Stages: ----- Time ------ ----- Flops >> ----- --- Messages --- -- Message Lengths -- -- Reductions -- >> Avg %Total Avg >> %Total counts %Total Avg %Total counts %Total >> one stage frome 2nd: 5.5145e+00 14.9% 1.2336e+09 13.2% 2.494e+03 >> 22.4% 3.230e+03 22.2% 2.250e+03 18.5% >> the once stage from 1st: 2.7541e+00 93.1% 1.2336e+09 99.8% 2.508e+03 >> 99.3% 1.470e+04 97.4% 2.266e+03 88.0% >> >> >> My programs are pretty equivalent except the part with definition of >> preconditioners and the number of called KSPSolve procedures. >> I mean they use equal matrices, equally assemble them, use equal right >> hand sides, equal convergence monitors. >> Actually the 2nd one was made from the 1st. >> >> In 1st i use KSPSetFromOptions(KSP); and then just set the -ksp_type >> -pc_type -sub_pc_type -sub_ksp_type keys from command line >> >> In 2d i use for for nonblock PC: >> >> KSPGetPC(dKSP, &dPC); >> >> PCSetType(dPC, PCJACOBI); >> >> and for block PC: >> >> PCSetType(dPC, PCASM); >> >> KSPSetUp(dKSP); >> >> PCSetUp(dPC); >> >> PCASMGetSubKSP(dPC, &n_local, &first_local, &ASMSubKSP); >> >> for (i=0; i> >> { >> >> KSPGetPC(ASMSubKSP[i], &(SubPC[i])); >> >> PCSetType(SubPC[i], PCJACOBI); >> >> } >> >> >> Im sure there is a mistake somewhere. Because 1st program compares Jacobi >> and ASM-Jacobi preconditioners on my problem on the same KSP and tells me >> that ASM-Jacobi is better and the 2nd shows otherwise results. >> > > This could be a preload issue. You can use the PreLoadBegin()/PreLoadEnd() > macros if you like, or otherwise solve a system first to make sure > everything has been loaded. If the results are still confusing, run with > -ksp_view -log_summary and send the output. > > There is no reason for ASM-Jacobi (with -sub_ksp_type preonly, which is > default) to be better than Jacobi since it does the same algorithm with more > communication. > -- Best regards, Alexey Ryazanov ______________________________________ Nuclear Safety Institute of Russian Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: KSPview 2nd program 456.out Type: application/octet-stream Size: 22794 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: KSPview 2nd program 123456.out Type: application/octet-stream Size: 30805 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: KSPview 1st program.out Type: application/octet-stream Size: 11780 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Tue Aug 23 18:00:06 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 23 Aug 2011 18:00:06 -0500 Subject: [petsc-users] BiCGSTAB for general use In-Reply-To: <7837C2E3-428C-4B7A-834A-B6832AFCFD27@gmail.com> References: <83F76379-F270-48A3-B498-C69934D83F82@gmail.com> <5236C63B-633D-4048-8EB6-5BDB9A5BE339@gmail.com> <8110936E-C186-4DC4-879B-912CBC849E7F@gmail.com> <7837C2E3-428C-4B7A-834A-B6832AFCFD27@gmail.com> Message-ID: On Mon, Aug 22, 2011 at 05:51, Paul Anton Letnes < paul.anton.letnes at gmail.com> wrote: > Not that I know of. The operator contains integrals over a stochastically > rough surface. Due to its random nature, any sparsity structure will depend > on each surface realization, > Are you integrating the probability density or just a single realization? Integrating over the stochastic dimensions may preserve more structure. > > Your independent variables discretize a 2D space of angles, right? Then > try a 2D color plot. > > Sort of. That may be a good idea. Perhaps we could use something like that > to learn more about the physics anyway. > At this point, I would hope that this gives some insight into structure that can be used to build a better solver. I have so far not plotted or analyzed any singular vectors or eigenvectors. > To simulate a physically meaningful system will probably take quite a few > CPU hours... I attach a plot of singular values for a small (4608x4608) test > matrix. It does not look too great at first glance. > > The largest singular value is 20107, the smallest 69.7, giving a ratio of > 288 or so. > I don't see any obvious useful structure here. -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.anton.letnes at gmail.com Wed Aug 24 03:37:35 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Wed, 24 Aug 2011 09:37:35 +0100 Subject: [petsc-users] BiCGSTAB for general use In-Reply-To: References: <83F76379-F270-48A3-B498-C69934D83F82@gmail.com> <5236C63B-633D-4048-8EB6-5BDB9A5BE339@gmail.com> <8110936E-C186-4DC4-879B-912CBC849E7F@gmail.com> <7837C2E3-428C-4B7A-834A-B6832AFCFD27@gmail.com> Message-ID: <1B37D4D1-28BD-4645-A36B-07E707425D1F@gmail.com> On 24. aug. 2011, at 00.00, Jed Brown wrote: > On Mon, Aug 22, 2011 at 05:51, Paul Anton Letnes wrote: > Not that I know of. The operator contains integrals over a stochastically rough surface. Due to its random nature, any sparsity structure will depend on each surface realization, > > Are you integrating the probability density or just a single realization? Integrating over the stochastic dimensions may preserve more structure. The integration is over a single realization. If one uses perturbation theory, one can integrate over the probability density, but that's last year's research at this point. Also, our approach should allow for rougher surfaces, since it includes all orders of perturbation theory. Thanks for all your insights! You've been very helpful. It's too bad, but it seems like we may be stuck with LU factorization for a while yet. Paul. From marek.schmitt at yahoo.com Wed Aug 24 08:25:01 2011 From: marek.schmitt at yahoo.com (Marek Schmitt) Date: Wed, 24 Aug 2011 06:25:01 -0700 (PDT) Subject: [petsc-users] PETSc on unstructured meshes / Sieve Message-ID: <1314192301.9656.YahooMailNeo@web125116.mail.ne1.yahoo.com> I would like to experiment with PETSc for learning FVM on unstructured grids. I get the impression that PETSc is primarily developed for structured grids with cartesian topology, is this true? Pylith and Fenics seem to use Sieve for unstructured grids. Is Sieve part of PETSc? Why is it so much hidden? The very silent sieve-dev mailing list exists since four years, but there is a recent post: "2) Unstructured meshes. This is not well-documented. There is a tutorial presentation and a repository of code for it. A few people have used this, but it is nowhere near the level of clarity and robustness that the rest of PETSc has." (from http://lists.mcs.anl.gov/pipermail/sieve-dev/2010-October/000098.html) Is the sieve-dev list about a different sieve than what is used by Pylith and Fenics? There is a PETSc FAQ "Do you have examples of doing unstructured grid finite element computations (FEM) with PETSc?". It mentions Sieve but no further links or documentation. Is the directory petsc-3.1-p8/include/sieve all that is needed to work with Sieve? Or are these only header files, and I have to link to the Sieve library from somewhere else (then where can I find Sieve)? Please shine some light into the mysterious Sieve. Marek From knepley at gmail.com Wed Aug 24 08:45:57 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 24 Aug 2011 13:45:57 +0000 Subject: [petsc-users] PETSc on unstructured meshes / Sieve In-Reply-To: <1314192301.9656.YahooMailNeo@web125116.mail.ne1.yahoo.com> References: <1314192301.9656.YahooMailNeo@web125116.mail.ne1.yahoo.com> Message-ID: On Wed, Aug 24, 2011 at 1:25 PM, Marek Schmitt wrote: > I would like to experiment with PETSc for learning FVM on unstructured > grids. I get the impression that PETSc is primarily developed for structured > grids with cartesian topology, is this true? > PETSc is not primarily developed for discretization and topology. It solves systems of nonlinear algebraic equations. It does have some extensions that handle structured (DMDA) and unstructured (DMMesh) under the recently developed DM interface. > Pylith and Fenics seem to use Sieve for unstructured grids. Is Sieve part > of PETSc? > Sieve is both the name of an interface (implemented by FEniCS) and an implementation of that interface (in PETSc). > Why is it so much hidden? The very silent sieve-dev mailing list exists > since four years, but there is a recent post: > I am not sure its hidden. I answer all the questions on the list. Its a big project for one person to support. I put most of my time into supporting PyLith (http://geodynamics.org/cig/software/pylith) which uses it for parallel FEM simulation. PETSc is 20 years old, so we have had time to make some components more robust. > "2) Unstructured meshes. This is not well-documented. There is a tutorial > presentation and a repository of code for it. A few people have used this, > but it is nowhere near the level of clarity and robustness that the rest of > PETSc has." (from > http://lists.mcs.anl.gov/pipermail/sieve-dev/2010-October/000098.html) > Is the sieve-dev list about a different sieve than what is used by Pylith > and Fenics? > They are two different implementations. > There is a PETSc FAQ "Do you have examples of doing unstructured grid > finite element computations (FEM) with PETSc?". It mentions Sieve but no > further links or documentation. > > Is the directory petsc-3.1-p8/include/sieve all that is needed to work with > Sieve? Or are these only header files, and I have to link to the Sieve > library from somewhere else (then where can I find Sieve)? > You must use petsc-dev in order for the examples to work, like SNES ex12. Thanks, Matt > Please shine some light into the mysterious Sieve. > Marek > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From w.drenth at gmail.com Wed Aug 24 09:33:21 2011 From: w.drenth at gmail.com (Wienand Drenth) Date: Wed, 24 Aug 2011 16:33:21 +0200 Subject: [petsc-users] MPI_Scan under Fortran Message-ID: dear all, For some parallellization project I am adapting example 10 of the SNES tutorials ($PETSC_DIR/src/snes/examples/tutorials/ex10d/). Since the project is writted in Fortran, and am trying to rewrite the example in Fortran. So far little issues, but the MPI_Scan routine gives me some more problems. In the said example, the call is like ierr = MPI_Scan(&user.Nvlocal,&rstart,1,MPIU_INT,MPI_SUM,PETSC_COMM_WORLD);CHKERRQ(ierr); For Fortran I rewrote it a little to look like call MPI_Scan(user%Nvlocal,rstart,1,MPIU_INT,MPI_SUM, PETSC_COMM_WORLD, ierr) The program compiles without troubles, but running it gives me an error. It is, I think, related to the MPI_SUM operation: [walrus:463] *** An error occurred in MPI_Scan: the reduction operation MPI_SUM is not defined on the MPI_BYTE datatype [walrus:463] *** on communicator MPI_COMM_WORLD [walrus:463] *** MPI_ERR_OP: invalid reduce operation [walrus:463] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) In addition, I needed to declare MPIU_INT explicitly as integer; this was not required for the C program. Am I missing a crucial header file with definitions? The solution will probably be very simple and straightforward, but I seem to overlook it. Any help will be greatly appreciated. regards, Wienand Drenth -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Aug 24 09:41:18 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 24 Aug 2011 09:41:18 -0500 Subject: [petsc-users] MPI_Scan under Fortran In-Reply-To: References: Message-ID: On Wed, Aug 24, 2011 at 09:33, Wienand Drenth wrote: > The program compiles without troubles, but running it gives me an error. It > is, I think, related to the MPI_SUM operation: > [walrus:463] *** An error occurred in MPI_Scan: the reduction operation > MPI_SUM is not defined on the MPI_BYTE datatype > [walrus:463] *** on communicator MPI_COMM_WORLD > [walrus:463] *** MPI_ERR_OP: invalid reduce operation > [walrus:463] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) > > In addition, I needed to declare MPIU_INT explicitly as integer; this was > not required for the C program. > I believe MPIU_INT (which is normally made to match PetscInt) is not defined in Fortran. You can use MPI_INTEGER4 or MPI_INTEGER8 as needed. Perhaps PETSc should define MPIU_INT, MPIU_SCALAR, etc in Fortran too? -------------- next part -------------- An HTML attachment was scrubbed... URL: From marek.schmitt at yahoo.com Wed Aug 24 09:55:31 2011 From: marek.schmitt at yahoo.com (Marek Schmitt) Date: Wed, 24 Aug 2011 07:55:31 -0700 (PDT) Subject: [petsc-users] PETSc on unstructured meshes / Sieve In-Reply-To: References: <1314192301.9656.YahooMailNeo@web125116.mail.ne1.yahoo.com> Message-ID: <1314197731.59568.YahooMailNeo@web125120.mail.ne1.yahoo.com> Thank you very much for the reply. However, I am even more confused now. Fenics is built on top of PETSc, isn't it? Then why do they use different Sieve implementations? Is one of the implementations much more advanced than the other, and if yes, could you show me some mailing list discussion or other information where the two are compared? (An acronym like "PETSc" is much easier to search for than "sieve") How is it meant that Sieve is "nowhere near the level of clarity and robustness that the rest of PETSc has"? Is it just that PETSc has very high standards that are not met by Sieve, or does Sieve have any major flaws or drawbacks? I just want to make sure that I don't work in impassable terrain. Marek ________________________________ From: Matthew Knepley To: Marek Schmitt ; PETSc users list Sent: Wednesday, August 24, 2011 3:45 PM Subject: Re: [petsc-users] PETSc on unstructured meshes / Sieve On Wed, Aug 24, 2011 at 1:25 PM, Marek Schmitt wrote: I would like to experiment with PETSc for learning FVM on unstructured grids. I get the impression that PETSc is primarily developed for structured grids with cartesian topology, is this true? > PETSc is not primarily developed for discretization and topology. It solves systems of nonlinear algebraic equations. It does have some extensions that handle structured (DMDA) and unstructured (DMMesh) under the recently developed DM interface. Pylith and Fenics seem to use Sieve for unstructured grids. Is Sieve part of PETSc? > Sieve is both the name of an interface (implemented by FEniCS) and an implementation of that interface (in PETSc). Why is it so much hidden? The very silent sieve-dev mailing list exists since four years, but there is a recent post: > I am not sure its hidden. I answer all the questions on the list. Its a big project for one person to support. I put most of my time into supporting PyLith (http://geodynamics.org/cig/software/pylith) which uses it for parallel FEM simulation. PETSc is 20 years old, so we have had time to make some components more robust. "2) Unstructured meshes. This is not well-documented. There is a tutorial presentation and a repository of code for it. A few people have used this, but it is nowhere near the level of clarity and robustness that the rest of PETSc has." (from http://lists.mcs.anl.gov/pipermail/sieve-dev/2010-October/000098.html) >Is the sieve-dev list about a different sieve than what is used by Pylith and Fenics? > They are two different implementations. There is a PETSc FAQ "Do you have examples of doing unstructured grid finite element computations (FEM) with PETSc?". It mentions Sieve but no further links or documentation. > >Is the directory petsc-3.1-p8/include/sieve all that is needed to work with Sieve? Or are these only header files, and I have to link to the Sieve library from somewhere else (then where can I find Sieve)? > You must use petsc-dev in order for the examples to work, like SNES ex12. ? Thanks, ? ? ? Matt Please shine some light into the mysterious Sieve. >Marek > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener????? From gdiso at ustc.edu Wed Aug 24 09:55:46 2011 From: gdiso at ustc.edu (Gong Ding) Date: Wed, 24 Aug 2011 22:55:46 +0800 Subject: [petsc-users] PETSc on unstructured meshes / Sieve References: <1314192301.9656.YahooMailNeo@web125116.mail.ne1.yahoo.com> Message-ID: <83F7C3DF281140A68D32A16CD2D73FAE@cogendaeda> ----- Original Message ----- From: "Marek Schmitt" To: Sent: Wednesday, August 24, 2011 9:25 PM Subject: [petsc-users] PETSc on unstructured meshes / Sieve >I would like to experiment with PETSc for learning FVM on unstructured grids. I get the impression that PETSc is primarily developed for structured grids with cartesian topology, is this true? > > Pylith and Fenics seem to use Sieve for unstructured grids. Is Sieve part of PETSc? > Why is it so much hidden? The very silent sieve-dev mailing list exists since four years, but there is a recent post: > "2) Unstructured meshes. This is not well-documented. There is a tutorial presentation and a repository of code for it. A few people have used this, but it is nowhere near the level of clarity and robustness that the rest of PETSc has." (from http://lists.mcs.anl.gov/pipermail/sieve-dev/2010-October/000098.html) > Is the sieve-dev list about a different sieve than what is used by Pylith and Fenics? > > There is a PETSc FAQ "Do you have examples of doing unstructured grid finite element computations (FEM) with PETSc?". It mentions Sieve but no further links or documentation. > > Is the directory petsc-3.1-p8/include/sieve all that is needed to work with Sieve? Or are these only header files, and I have to link to the Sieve library from somewhere else (then where can I find Sieve)? > > Please shine some light into the mysterious Sieve. > Marek Petsc is more or less a (linear) solver toolkit, not unstructured framework. Libmesh can be a good start point to you. It is really a beautiful design. And some other framework, i.e. deal.II, openfoam. From baagaard at usgs.gov Wed Aug 24 11:56:03 2011 From: baagaard at usgs.gov (Brad Aagaard) Date: Wed, 24 Aug 2011 09:56:03 -0700 Subject: [petsc-users] PETSc on unstructured meshes / Sieve In-Reply-To: <1314197731.59568.YahooMailNeo@web125120.mail.ne1.yahoo.com> References: <1314192301.9656.YahooMailNeo@web125116.mail.ne1.yahoo.com> <1314197731.59568.YahooMailNeo@web125120.mail.ne1.yahoo.com> Message-ID: <4E552D23.3020101@usgs.gov> Marek- I am one the primary uses of Sieve as one of the PyLith developers. We have been using Sieve since its infancy and have successfully made seven PyLith releases that depend on it over the last 5 years. Sieve is used by only a few projects and is maintained almost exclusively by Matt. As a result it is not tested as extensively as PETSc and little effort has gone into the documentation. Most of Matt's time goes into ongoing development in support of its use in projects rather than documentation. The downside is that it takes some time to figure out how to use it and one has to refer to the Sieve code directly rather than any documentation, but the upside is that one learns much more about Sieve and how it works in the process. Matt also adds features to support projects. Sieve is also not as mature so occasionally there are significant changes. However, it is quite powerful due to its generality. Brad On 08/24/2011 07:55 AM, Marek Schmitt wrote: > Thank you very much for the reply. However, I am even more confused now. > > Fenics is built on top of PETSc, > isn't it? Then why do they use different Sieve implementations? Is one > of the implementations much more advanced than the other, and if yes, > could you show me some mailing list discussion or other information > where the two are compared? (An acronym like "PETSc" is much easier to > search for than "sieve") > > > How is it meant that Sieve is > "nowhere near the level of clarity and robustness that the rest of PETSc has"? Is it just that PETSc has very high standards that are not met by Sieve, or does Sieve have any major flaws or drawbacks? I just want to > make sure that I don't work in impassable terrain. > > Marek > > ________________________________ > From: Matthew Knepley > To: Marek Schmitt; PETSc users list > Sent: Wednesday, August 24, 2011 3:45 PM > Subject: Re: [petsc-users] PETSc on unstructured meshes / Sieve > > > On Wed, Aug 24, 2011 at 1:25 PM, Marek Schmitt wrote: > > I would like to experiment with PETSc for learning FVM on unstructured grids. I get the impression that PETSc is primarily developed for structured grids with cartesian topology, is this true? >> > > PETSc is not primarily developed for discretization and topology. It solves systems of nonlinear algebraic equations. It does have > some extensions that handle structured (DMDA) and unstructured (DMMesh) under the recently developed DM interface. > > Pylith and Fenics seem to use Sieve for unstructured grids. Is Sieve part of PETSc? >> > > Sieve is both the name of an interface (implemented by FEniCS) and an implementation of that interface (in PETSc). > > Why is it so much hidden? The very silent sieve-dev mailing list exists since four years, but there is a recent post: >> > > I am not sure its hidden. I answer all the questions on the list. Its a big project for one person to support. I put most of > my time into supporting PyLith (http://geodynamics.org/cig/software/pylith) which uses it for parallel FEM simulation. > PETSc is 20 years old, so we have had time to make some components more robust. > > "2) Unstructured meshes. This is not well-documented. There is a tutorial presentation and a repository of code for it. A few people have used this, but it is nowhere near the level of clarity and robustness that the rest of PETSc has." (from http://lists.mcs.anl.gov/pipermail/sieve-dev/2010-October/000098.html) >> Is the sieve-dev list about a different sieve than what is used by Pylith and Fenics? >> > > They are two different implementations. > > There is a PETSc FAQ "Do you have examples of doing unstructured grid finite element computations (FEM) with PETSc?". It mentions Sieve but no further links or documentation. >> >> Is the directory petsc-3.1-p8/include/sieve all that is needed to work with Sieve? Or are these only header files, and I have to link to the Sieve library from somewhere else (then where can I find Sieve)? >> > > You must use petsc-dev in order for the examples to work, like SNES ex12. > > Thanks, > > Matt > > Please shine some light into the mysterious Sieve. >> Marek >> > > From likunt at andrew.cmu.edu Wed Aug 24 14:11:01 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Wed, 24 Aug 2011 15:11:01 -0400 Subject: [petsc-users] about solving linear equation Message-ID: Hello, I am solving a linear equation Ax=0 with Petsc. Previously i was following Numerical Recipe and using LU decomposition, for a non-singular matrix A, x should be zero. By applying LU in Recipe, i got x=0. When i solve it with ksp, x is some small value to the power of -9. The condition number of A is about 52, is there any bug in my code or i should find more stable linear solver? Thanks, Likun From jedbrown at mcs.anl.gov Wed Aug 24 14:17:54 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 24 Aug 2011 14:17:54 -0500 Subject: [petsc-users] about solving linear equation In-Reply-To: References: Message-ID: On Wed, Aug 24, 2011 at 14:11, Likun Tan wrote: > I am solving a linear equation Ax=0 with Petsc. > Previously i was following Numerical Recipe and using LU decomposition, > for a non-singular matrix A, x should be zero. By applying LU in Recipe, i > got x=0. > When i solve it with ksp, x is some small value to the power of -9. > This is normal. You can make it more accurate by reducing the tolerance, e.g. -ksp_rtol 1e-12. -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Aug 24 15:20:43 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 24 Aug 2011 15:20:43 -0500 (CDT) Subject: [petsc-users] MPI_Scan under Fortran In-Reply-To: References: Message-ID: On Wed, 24 Aug 2011, Jed Brown wrote: > On Wed, Aug 24, 2011 at 09:33, Wienand Drenth wrote: > > > The program compiles without troubles, but running it gives me an error. It > > is, I think, related to the MPI_SUM operation: > > [walrus:463] *** An error occurred in MPI_Scan: the reduction operation > > MPI_SUM is not defined on the MPI_BYTE datatype > > [walrus:463] *** on communicator MPI_COMM_WORLD > > [walrus:463] *** MPI_ERR_OP: invalid reduce operation > > [walrus:463] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) > > > > In addition, I needed to declare MPIU_INT explicitly as integer; this was > > not required for the C program. > > > > I believe MPIU_INT (which is normally made to match PetscInt) is not defined > in Fortran. You can use MPI_INTEGER4 or MPI_INTEGER8 as needed. > > Perhaps PETSc should define MPIU_INT, MPIU_SCALAR, etc in Fortran too? MPI does MPI_INT (c) vs MPI_INTEGER (fortran) - so we have MPIU_INTEGER on the fortran side. [in finclude/petscsys.h] And we do have MPIU_SCALAR on the fortran side. Satish > From zhenglun.wei at gmail.com Wed Aug 24 16:19:22 2011 From: zhenglun.wei at gmail.com (Alan Wei) Date: Wed, 24 Aug 2011 16:19:22 -0500 Subject: [petsc-users] Structure of DMMG Message-ID: Dear all, I hope you're having a nice day. I'm using PETSc as a beginner. I found it is very easy to find a definition of a function like 'DACreate2d' online and it explains very clearly. However, I think it is really confusing for those explanation of data structures like DMMG, DA, DM and so on, or even Vec. I wonder if there is more details that I can read, like what is inside DMMG. For example, as I see, DM is a part of DMMG. thanks, Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Aug 24 16:25:11 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 24 Aug 2011 16:25:11 -0500 Subject: [petsc-users] Structure of DMMG In-Reply-To: References: Message-ID: Alan, PETSc is designed as an object oriented package; as such the objects are defined by the operations they perform, not the data stored inside them. So, for example, the Vec object is defined by all the VecXXX() operations you can call which are listed in the manual pages list. You really don't want to see the details of the Vec data structure (for one thing it is different for different Vec implementations), they don't tell you what you can do with a Vec. (same of other objects). I suggest going through some of the examples and seeing the types of operations you can perform on different objects starting with the vec/vec/examples/tutorials examples. Barry On Aug 24, 2011, at 4:19 PM, Alan Wei wrote: > Dear all, > I hope you're having a nice day. > I'm using PETSc as a beginner. I found it is very easy to find a definition of a function like 'DACreate2d' online and it explains very clearly. However, I think it is really confusing for those explanation of data structures like DMMG, DA, DM and so on, or even Vec. I wonder if there is more details that I can read, like what is inside DMMG. For example, as I see, DM is a part of DMMG. > > thanks, > Alan From zhenglun.wei at gmail.com Wed Aug 24 16:43:21 2011 From: zhenglun.wei at gmail.com (Alan Wei) Date: Wed, 24 Aug 2011 16:43:21 -0500 Subject: [petsc-users] Structure of DMMG In-Reply-To: References: Message-ID: Dear Barry, You really answer what I want to know. Thank you so much and sorry for my knowledge lacking of programming. I will go through those tutorial examples. You are really nice. thanks, Alan On Wed, Aug 24, 2011 at 4:25 PM, Barry Smith wrote: > > Alan, > > PETSc is designed as an object oriented package; as such the objects are > defined by the operations they perform, not the data stored inside them. So, > for example, the Vec object is defined by all the VecXXX() operations you > can call which are listed in the manual pages list. You really don't want > to see the details of the Vec data structure (for one thing it is different > for different Vec implementations), they don't tell you what you can do with > a Vec. (same of other objects). I suggest going through some of the examples > and seeing the types of operations you can perform on different objects > starting with the vec/vec/examples/tutorials examples. > > Barry > > > > On Aug 24, 2011, at 4:19 PM, Alan Wei wrote: > > > Dear all, > > I hope you're having a nice day. > > I'm using PETSc as a beginner. I found it is very easy to find a > definition of a function like 'DACreate2d' online and it explains very > clearly. However, I think it is really confusing for those explanation of > data structures like DMMG, DA, DM and so on, or even Vec. I wonder if there > is more details that I can read, like what is inside DMMG. For example, as I > see, DM is a part of DMMG. > > > > thanks, > > Alan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Aug 24 17:53:02 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 24 Aug 2011 17:53:02 -0500 Subject: [petsc-users] performance issue solving multiple linear systems of the same size with the different preconditioning methods In-Reply-To: References: Message-ID: On Tue, Aug 23, 2011 at 02:37, ??????? ??????? wrote: > When i delete the 4-5-6 part of 2nd, 1-2-3 works great! with exact like 1st > results. > When i delete the 1-2-3 part of 2nd, 4-5-6 works great! with exact like 1st > results. > All program (1-2-3-4-5-6) works badly. > >From the -log_summary, you have a memory leak (many more vector creations than destructions). Try running with -malloc_dump to debug it. Perhaps you are creating a vector every time one of your functions is called? You should also build --with-debugging=0 when looking at timing results. (You can keep it in PETSC_ARCH=linux-gnu-opt.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From w.drenth at gmail.com Thu Aug 25 03:33:58 2011 From: w.drenth at gmail.com (Wienand Drenth) Date: Thu, 25 Aug 2011 10:33:58 +0200 Subject: [petsc-users] MPI_Scan under Fortran In-Reply-To: References: Message-ID: Satish, Jed, Thank you for the replies and suggestions for a solution. The MPI_INTEGER4 seems to work. However, the MPIU_INTEGER suggested by Satish gives the same error as did the MPIU_INT, though I include the petscsys.h. Could it relate to the Petsc version? For compatibility reasons I am using the 2.3.3 distribution. regards, Wienand On Wed, Aug 24, 2011 at 10:20 PM, Satish Balay wrote: > On Wed, 24 Aug 2011, Jed Brown wrote: > > > On Wed, Aug 24, 2011 at 09:33, Wienand Drenth > wrote: > > > > > The program compiles without troubles, but running it gives me an > error. It > > > is, I think, related to the MPI_SUM operation: > > > [walrus:463] *** An error occurred in MPI_Scan: the reduction operation > > > MPI_SUM is not defined on the MPI_BYTE datatype > > > [walrus:463] *** on communicator MPI_COMM_WORLD > > > [walrus:463] *** MPI_ERR_OP: invalid reduce operation > > > [walrus:463] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) > > > > > > In addition, I needed to declare MPIU_INT explicitly as integer; this > was > > > not required for the C program. > > > > > > > I believe MPIU_INT (which is normally made to match PetscInt) is not > defined > > in Fortran. You can use MPI_INTEGER4 or MPI_INTEGER8 as needed. > > > > Perhaps PETSc should define MPIU_INT, MPIU_SCALAR, etc in Fortran too? > > MPI does MPI_INT (c) vs MPI_INTEGER (fortran) - so we have > MPIU_INTEGER on the fortran side. [in finclude/petscsys.h] > > And we do have MPIU_SCALAR on the fortran side. > > Satish > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marek.schmitt at yahoo.com Thu Aug 25 06:11:28 2011 From: marek.schmitt at yahoo.com (Marek Schmitt) Date: Thu, 25 Aug 2011 04:11:28 -0700 (PDT) Subject: [petsc-users] PETSc on unstructured meshes / Sieve In-Reply-To: References: <1314192301.9656.YahooMailNeo@web125116.mail.ne1.yahoo.com> Message-ID: <1314270688.63732.YahooMailNeo@web125102.mail.ne1.yahoo.com> Thanks to all of you for your suggestions. I'll give Sieve a try. I do understand that PETSc is neither a FEM nor a FVM framework. What I expect from PETSc is that it helps with the parallel communication across mesh partition boundaries. I don't think there is a better toolkit for this? On the contrary, I would like to implement the matrix coefficients myself for learning purposes. For this reason projects like libmesh don't help (besides, it is for FEM, not FVM). By the way, why does libmesh reinvent their own mesh classes instead of using PETSc's? While the PETSc pdf manual has several pages on working with distributed arrays, I don't find anything on distributed meshes except for a description of index sets. Is the use of DMMesh documented somewhere? Marek ________________________________ From: Matthew Knepley To: Marek Schmitt ; PETSc users list Sent: Wednesday, August 24, 2011 3:45 PM Subject: Re: [petsc-users] PETSc on unstructured meshes / Sieve On Wed, Aug 24, 2011 at 1:25 PM, Marek Schmitt wrote: I would like to experiment with PETSc for learning FVM on unstructured grids. I get the impression that PETSc is primarily developed for structured grids with cartesian topology, is this true? > PETSc is not primarily developed for discretization and topology. It solves systems of nonlinear algebraic equations. It does have some extensions that handle structured (DMDA) and unstructured (DMMesh) under the recently developed DM interface. Pylith and Fenics seem to use Sieve for unstructured grids. Is Sieve part of PETSc? > Sieve is both the name of an interface (implemented by FEniCS) and an implementation of that interface (in PETSc). Why is it so much hidden? The very silent sieve-dev mailing list exists since four years, but there is a recent post: > I am not sure its hidden. I answer all the questions on the list. Its a big project for one person to support. I put most of my time into supporting PyLith (http://geodynamics.org/cig/software/pylith) which uses it for parallel FEM simulation. PETSc is 20 years old, so we have had time to make some components more robust. "2) Unstructured meshes. This is not well-documented. There is a tutorial presentation and a repository of code for it. A few people have used this, but it is nowhere near the level of clarity and robustness that the rest of PETSc has." (from http://lists.mcs.anl.gov/pipermail/sieve-dev/2010-October/000098.html) >Is the sieve-dev list about a different sieve than what is used by Pylith and Fenics? > They are two different implementations. There is a PETSc FAQ "Do you have examples of doing unstructured grid finite element computations (FEM) with PETSc?". It mentions Sieve but no further links or documentation. > >Is the directory petsc-3.1-p8/include/sieve all that is needed to work with Sieve? Or are these only header files, and I have to link to the Sieve library from somewhere else (then where can I find Sieve)? > You must use petsc-dev in order for the examples to work, like SNES ex12. ? Thanks, ? ? ? Matt Please shine some light into the mysterious Sieve. >Marek > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener????? From knepley at gmail.com Thu Aug 25 06:15:50 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 25 Aug 2011 11:15:50 +0000 Subject: [petsc-users] PETSc on unstructured meshes / Sieve In-Reply-To: <1314270688.63732.YahooMailNeo@web125102.mail.ne1.yahoo.com> References: <1314192301.9656.YahooMailNeo@web125116.mail.ne1.yahoo.com> <1314270688.63732.YahooMailNeo@web125102.mail.ne1.yahoo.com> Message-ID: On Thu, Aug 25, 2011 at 11:11 AM, Marek Schmitt wrote: > Thanks to all of you for your suggestions. I'll give Sieve a try. > > I do understand that PETSc is neither a FEM nor a FVM framework. What I > expect from PETSc is that it helps with the parallel communication across > mesh partition boundaries. I don't think there is a better toolkit for this? > On the contrary, I would like to > implement the matrix coefficients myself for learning purposes. For this > reason projects like libmesh don't help (besides, it is for FEM, not FVM). > By the way, why does libmesh reinvent their own mesh classes instead of > using PETSc's? > libmesh has been doing this for a long time, much longer than PETSc had anything about meshes. > While the PETSc pdf manual has several pages on working with distributed > arrays, I don't find anything on distributed meshes except for a description > of index sets. Is the use of DMMesh documented somewhere? No, not yet. I will write something soon. I did write something about dealing with values over meshes, but it only applies to the C++ interface, so I want to rewrite it for people who only want to use C. You can also read the first paper (Knepley Karpeev 2009). We have another paper which is nearly ready which I can send you when we finish. Thanks, Matt > Marek > > > ________________________________ > From: Matthew Knepley > To: Marek Schmitt ; PETSc users list < > petsc-users at mcs.anl.gov> > Sent: Wednesday, August 24, 2011 3:45 PM > Subject: Re: [petsc-users] PETSc on unstructured meshes / Sieve > > > On Wed, Aug 24, 2011 at 1:25 PM, Marek Schmitt > wrote: > > I would like to experiment with PETSc for learning FVM on unstructured > grids. I get the impression that PETSc is primarily developed for structured > grids with cartesian topology, is this true? > > > > PETSc is not primarily developed for discretization and topology. It solves > systems of nonlinear algebraic equations. It does have > some extensions that handle structured (DMDA) and unstructured (DMMesh) > under the recently developed DM interface. > > Pylith and Fenics seem to use Sieve for unstructured grids. Is Sieve part > of PETSc? > > > > Sieve is both the name of an interface (implemented by FEniCS) and an > implementation of that interface (in PETSc). > > Why is it so much hidden? The very silent sieve-dev mailing list exists > since four years, but there is a recent post: > > > > I am not sure its hidden. I answer all the questions on the list. Its a big > project for one person to support. I put most of > my time into supporting PyLith (http://geodynamics.org/cig/software/pylith) > which uses it for parallel FEM simulation. > PETSc is 20 years old, so we have had time to make some components more > robust. > > "2) Unstructured meshes. This is not well-documented. There is a tutorial > presentation and a repository of code for it. A few people have used this, > but it is nowhere near the level of clarity and robustness that the rest of > PETSc has." (from > http://lists.mcs.anl.gov/pipermail/sieve-dev/2010-October/000098.html) > >Is the sieve-dev list about a different sieve than what is used by Pylith > and Fenics? > > > > They are two different implementations. > > There is a PETSc FAQ "Do you have examples of doing unstructured grid > finite element computations (FEM) with PETSc?". It mentions Sieve but no > further links or documentation. > > > >Is the directory petsc-3.1-p8/include/sieve all that is needed to work > with Sieve? Or are these only header files, and I have to link to the Sieve > library from somewhere else (then where can I find Sieve)? > > > > You must use petsc-dev in order for the examples to work, like SNES ex12. > > Thanks, > > Matt > > Please shine some light into the mysterious Sieve. > >Marek > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From marek.schmitt at yahoo.com Thu Aug 25 07:17:55 2011 From: marek.schmitt at yahoo.com (Marek Schmitt) Date: Thu, 25 Aug 2011 05:17:55 -0700 (PDT) Subject: [petsc-users] PETSc on unstructured meshes / Sieve In-Reply-To: References: <1314192301.9656.YahooMailNeo@web125116.mail.ne1.yahoo.com> <1314270688.63732.YahooMailNeo@web125102.mail.ne1.yahoo.com> Message-ID: <1314274675.37353.YahooMailNeo@web125108.mail.ne1.yahoo.com> >>While the PETSc pdf manual has several pages on working with distributed arrays, I don't find anything on distributed meshes except for a description of index >>sets. Is the use of DMMesh documented somewhere? > >No, not yet. I will write something soon. I did write something about dealing with values over >meshes, but it only applies to the C++ interface, so I want to rewrite it for people who only >want to use C. You can also read the first paper (Knepley Karpeev 2009). We have another >paper which is nearly ready which I can send you when we finish. That would be very kind. If you have other (even unfinished) documentation on this topic that you are willing to share, it is very welcome. I am more interested in the C++ interface than C. Your 2009 paper is helpful to understand the Sieve concepts, but working with the implementation in PETSc is a different story. However, I think all the replies I got so far will allow me to get started now. Thanks Marek From jedbrown at mcs.anl.gov Thu Aug 25 08:18:09 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 25 Aug 2011 08:18:09 -0500 Subject: [petsc-users] MPI_Scan under Fortran In-Reply-To: References: Message-ID: On Thu, Aug 25, 2011 at 03:33, Wienand Drenth wrote: > The MPI_INTEGER4 seems to work. However, the MPIU_INTEGER suggested by > Satish gives the same error as did the MPIU_INT, though I include the > petscsys.h. Could it relate to the Petsc version? > Yes, checking the history, it was added shortly before the 3.1 release. > For compatibility reasons I am using the 2.3.3 distribution. > That is very old. We recommend that you upgrade. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Thu Aug 25 09:04:33 2011 From: zonexo at gmail.com (TAY wee-beng) Date: Thu, 25 Aug 2011 16:04:33 +0200 Subject: [petsc-users] No change in a.out compilation size after using PETSc shared library Message-ID: <4E565671.10907@gmail.com> Hi, I managed to compile PETSc shared library. However, when I compile my code, my a.out 's size is still around 12mb, which is similar to using PETSc static library. I thought it will be a much smaller exe. Shouldn't it be? Thank you! -- Yours sincerely, TAY wee-beng From bsmith at mcs.anl.gov Thu Aug 25 09:14:25 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 25 Aug 2011 09:14:25 -0500 Subject: [petsc-users] No change in a.out compilation size after using PETSc shared library In-Reply-To: <4E565671.10907@gmail.com> References: <4E565671.10907@gmail.com> Message-ID: Send configure.log make.log and ls ${PETSC_ARCH/lib to petsc-maint at mcs.anl.gov Barry On Aug 25, 2011, at 9:04 AM, TAY wee-beng wrote: > Hi, > > I managed to compile PETSc shared library. > > However, when I compile my code, my a.out 's size is still around 12mb, which is similar to using PETSc static library. > > I thought it will be a much smaller exe. Shouldn't it be? > > Thank you! > > -- > Yours sincerely, > > TAY wee-beng > From gdiso at ustc.edu Thu Aug 25 11:46:54 2011 From: gdiso at ustc.edu (Gong Ding) Date: Fri, 26 Aug 2011 00:46:54 +0800 Subject: [petsc-users] PETSc on unstructured meshes / Sieve References: <1314192301.9656.YahooMailNeo@web125116.mail.ne1.yahoo.com> <1314270688.63732.YahooMailNeo@web125102.mail.ne1.yahoo.com> Message-ID: <02D1A9FDD7FB4F80AD905FF9E7522409@cogendaeda> I had added vertex based FVM support to libmesh, for my semiconductor simulator, not very difficult. In the year of 2007, I had spent about one month to read though the source code of libmesh. And finally known how to write 3D, parallel numerical code based on unstructured grid. How to use OO to describe different mesh cells (tri, quad, tet, prism, hex...) with an uniform interface. How to organize the data structure of unstrctured mesh cells in memory. How to partition mesh between processsors. How to mapping variables to petsc vec and mat. How to synchronous values on ghost node/cell How to assemble petsc vec and mat. Many many issues. Thanks to all of you for your suggestions. I'll give Sieve a try. I do understand that PETSc is neither a FEM nor a FVM framework. What I expect from PETSc is that it helps with the parallel communication across mesh partition boundaries. I don't think there is a better toolkit for this? On the contrary, I would like to implement the matrix coefficients myself for learning purposes. For this reason projects like libmesh don't help (besides, it is for FEM, not FVM). By the way, why does libmesh reinvent their own mesh classes instead of using PETSc's? While the PETSc pdf manual has several pages on working with distributed arrays, I don't find anything on distributed meshes except for a description of index sets. Is the use of DMMesh documented somewhere? Marek ________________________________ From: Matthew Knepley To: Marek Schmitt ; PETSc users list Sent: Wednesday, August 24, 2011 3:45 PM Subject: Re: [petsc-users] PETSc on unstructured meshes / Sieve On Wed, Aug 24, 2011 at 1:25 PM, Marek Schmitt wrote: I would like to experiment with PETSc for learning FVM on unstructured grids. I get the impression that PETSc is primarily developed for structured grids with cartesian topology, is this true? > PETSc is not primarily developed for discretization and topology. It solves systems of nonlinear algebraic equations. It does have some extensions that handle structured (DMDA) and unstructured (DMMesh) under the recently developed DM interface. Pylith and Fenics seem to use Sieve for unstructured grids. Is Sieve part of PETSc? > Sieve is both the name of an interface (implemented by FEniCS) and an implementation of that interface (in PETSc). Why is it so much hidden? The very silent sieve-dev mailing list exists since four years, but there is a recent post: > I am not sure its hidden. I answer all the questions on the list. Its a big project for one person to support. I put most of my time into supporting PyLith (http://geodynamics.org/cig/software/pylith) which uses it for parallel FEM simulation. PETSc is 20 years old, so we have had time to make some components more robust. "2) Unstructured meshes. This is not well-documented. There is a tutorial presentation and a repository of code for it. A few people have used this, but it is nowhere near the level of clarity and robustness that the rest of PETSc has." (from http://lists.mcs.anl.gov/pipermail/sieve-dev/2010-October/000098.html) >Is the sieve-dev list about a different sieve than what is used by Pylith and Fenics? > They are two different implementations. There is a PETSc FAQ "Do you have examples of doing unstructured grid finite element computations (FEM) with PETSc?". It mentions Sieve but no further links or documentation. > >Is the directory petsc-3.1-p8/include/sieve all that is needed to work with Sieve? Or are these only header files, and I have to link to the Sieve library from somewhere else (then where can I find Sieve)? > You must use petsc-dev in order for the examples to work, like SNES ex12. Thanks, Matt Please shine some light into the mysterious Sieve. >Marek > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From likunt at andrew.cmu.edu Thu Aug 25 11:58:10 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Thu, 25 Aug 2011 12:58:10 -0400 Subject: [petsc-users] slower in petsc Message-ID: <9deec785e2b1dac81b4231a4ff17f8e3.squirrel@webmail.andrew.cmu.edu> Hello, I am computing Ax=b, while using petsc (currently uniprocessor) the computation is much slower. I realize generation of the matrix is very time-consuming, the elements are integrals of functions with complicated form, i.e.\\\int(f(r(x,y,z), s(x,y,z), t(x,y,z), phi(x,y,z,node)dxdydz, where the shape function has different forms on each node. In my C code, i calculated r, s, t and phi on the integration points first and saved the data in 3D array. In petsc, i use the same strategy by applying DA to set values to r, s, t and phi on each integration points. Since i need to solve Ax=b for million times, A is changing every time and i use DAVecGetArray repeatly, is that the reason for low efficiency and any suggestions for performance enhancement? btw, calculation of r,s,t..is the only part i use parallel computing, since I think the assemble of A and b is fast once the entries have been computed. Thanks, Likun From knepley at gmail.com Thu Aug 25 12:00:34 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 25 Aug 2011 17:00:34 +0000 Subject: [petsc-users] slower in petsc In-Reply-To: <9deec785e2b1dac81b4231a4ff17f8e3.squirrel@webmail.andrew.cmu.edu> References: <9deec785e2b1dac81b4231a4ff17f8e3.squirrel@webmail.andrew.cmu.edu> Message-ID: On Thu, Aug 25, 2011 at 4:58 PM, Likun Tan wrote: > Hello, > > I am computing Ax=b, while using petsc (currently uniprocessor) the > computation is much slower. > > I realize generation of the matrix is very time-consuming, the elements > are integrals of functions with complicated form, i.e.\\\int(f(r(x,y,z), > s(x,y,z), t(x,y,z), phi(x,y,z,node)dxdydz, where the shape function has > different forms on each node. In my C code, i calculated r, s, t and phi > on the integration points first and saved the data in 3D array. In petsc, > i use the same strategy by applying DA to set values to r, s, t and phi on > each integration points. > > Since i need to solve Ax=b for million times, A is changing every time and > i use DAVecGetArray repeatly, is that the reason for low efficiency and > any suggestions for performance enhancement? > > btw, calculation of r,s,t..is the only part i use parallel computing, > since I think the assemble of A and b is fast once the entries have been > computed. > There is no way to know what you are doing. As we say online and in the manual, configure with --with-debugging=0, look at the performance guidance chapter, and send the output of -log_summary to petsc-maint at mcs.anl.gov. Matt > Thanks, > Likun > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Aug 25 12:04:51 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 25 Aug 2011 12:04:51 -0500 Subject: [petsc-users] slower in petsc In-Reply-To: <9deec785e2b1dac81b4231a4ff17f8e3.squirrel@webmail.andrew.cmu.edu> References: <9deec785e2b1dac81b4231a4ff17f8e3.squirrel@webmail.andrew.cmu.edu> Message-ID: On Thu, Aug 25, 2011 at 11:58, Likun Tan wrote: > btw, calculation of r,s,t..is the only part i use parallel computing, > since I think the assemble of A and b is fast once the entries have been > computed. > As a bare minimum, configure --with-debugging=0 and send -log_summary output with a description of what you are comparing to or what you expect to see. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Thu Aug 25 13:29:09 2011 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Thu, 25 Aug 2011 14:29:09 -0400 Subject: [petsc-users] Freeing Preconditioner Matrix Space (Again) Message-ID: Hello I've managed to get the c-function for freeing preconditioner memory written. The contents of my new 'pcasmfreespace.c' is below: #include /*I "petscpc.h" I*/ typedef struct { PetscInt n, n_local, n_local_true; PetscInt overlap; /* overlap requested by user */ KSP *ksp; /* linear solvers for each block */ VecScatter *restriction; /* mapping from global to subregion */ VecScatter *localization; /* mapping from overlapping to non-overlapping subregion */ VecScatter *prolongation; /* mapping from subregion to global */ Vec *x,*y,*y_local; /* work vectors */ IS *is; /* index set that defines each overlapping subdomain */ IS *is_local; /* index set that defines each non-overlapping subdomain, may be NULL */ Mat *mat,*pmat; /* mat is not currently used */ PCASMType type; /* use reduced interpolation, restriction or both */ PetscInt type_set; /* if user set this value (so won't change it for symmetric problems) */ PetscInt same_local_solves; /* flag indicating whether all local solvers are same */ PetscInt sort_indices; /* flag to sort subdomain indices */ } PC_ASM; void pcasmfreespace_(PC pc) { PC_ASM *osm = (PC_ASM*)pc->data; PetscErrorCode ierr; if (osm->pmat) { if (osm->n_local_true > 0) { ierr = MatDestroyMatrices(osm->n_local_true,&osm->pmat);CHKERRQ(ierr); } osm->pmat = 0; } if (pc->pmat) {ierr = MatDestroy(pc->pmat);CHKERRQ(ierr); pc->pmat = 0;} } Note the underscore as I'm trying to call it from Fortran. When I call it from fortran, I use: call pcasmfreespace(global_pc) This calls, the function, ok, but (according to Valgrind) I have an invalid read on the line containing: if (osm->pmat){ I suspect this is something to do with passing the fortran "pointer" of the PC to c, or something along this lines. Is there anything else special you have to do to pass the "fortran" petsc objects to c? Thanks Gaetan Message: 2 Date: Sun, 21 Aug 2011 17:22:28 -0500 From: Barry Smith Subject: Re: [petsc-users] Freeing Preconditioner Matrix Space To: PETSc users list Message-ID: <953121EF-B6AC-42EE-87BE-D4402C121652 at mcs.anl.gov> Content-Type: text/plain; charset=us-ascii You don't need to put that in the PETSc source. Just built it in the same directory you build your application and link it in like any of your application code. You will need to stick #include /*I "petscpc.h" I*/ typedef struct { PetscInt n, n_local, n_local_true; PetscInt overlap; /* overlap requested by user */ KSP *ksp; /* linear solvers for each block */ VecScatter *restriction; /* mapping from global to subregion */ VecScatter *localization; /* mapping from overlapping to non-overlapping subregion */ VecScatter *prolongation; /* mapping from subregion to global */ Vec *x,*y,*y_local; /* work vectors */ IS *is; /* index set that defines each overlapping subdomain */ IS *is_local; /* index set that defines each non-overlapping subdomain, may be NULL */ Mat *mat,*pmat; /* mat is not currently used */ PCASMType type; /* use reduced interpolation, restriction or both */ PetscBool type_set; /* if user set this value (so won't change it for symmetric problems) */ PetscBool same_local_solves; /* flag indicating whether all local solvers are same */ PetscBool sort_indices; /* flag to sort subdomain indices */ } PC_ASM; before the subroutine. Barry On Aug 21, 2011, at 3:24 PM, Gaetan Kenway wrote: > Hello > > I am attempting to implement a "hack" that was posted on the list a while back. I'm working with the adjoint linear system solver for a CFD solver. I'm using the ASM (or Block Jacobi) preconditioner with ILU(p) on each of the sub-domains. I use a different Preconditioner matrix (Pmat) than the actual jacobian. What I want to do is destroy the Pmat memory after the ILU factorization is performed. The hack that was posted is copied below: > PCASMFreeSpace(PC pc) > { > PC_ASM *osm = (PC_ASM*)pc->data; > PetscErrorCode ierr; > > if (osm->pmat) { > if (osm->n_local_true > 0) { > > ierr = MatDestroyMatrices(osm->n_ local_true,&osm->pmat);CHKERRQ(ierr); > } > osm->pmat = 0; > } > if (pc->pmat) {ierr = MatDestroy(pc->pmat);CHKERRQ(ierr); pc->pmat = 0;} > > return 0; > } > > However, I've had no luck actually getting the function compiled into petsc. There are no erorrs reported with i type "make" in the asm directory, but when I try to use the function in my application it can't find the symbol while linking. Where does it go in the asm.c file? Does it use "static PetscErrorCode" or "PetscErrorCode PETSCKSP_DLLEXPORT"? Does it have to be added to the .h include files? What has to be done for it work with Fortran? > > Any suggestions would be greatly appreciated. This represents a significant chunk of my application's memory (10%-30%) and as such its too much to ignore. Also is there any chance something like this would make it into an actual PETSc release? > > Gaetan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenglun.wei at gmail.com Thu Aug 25 13:56:47 2011 From: zhenglun.wei at gmail.com (Alan Wei) Date: Thu, 25 Aug 2011 13:56:47 -0500 Subject: [petsc-users] Question on PETSc makefile Message-ID: Dear all, I hope you're having a nice day. I met a problem on makefile of PETSc. Original PETSc has the makefile in the same directory of the .c files. However, I want to put all .c files to another directory. I changed the makefile also; however it always has a error saying the PETSc header files can not be found: /home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/bin/mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -o ComputerRHS.o Programs/ComputeRHS.c -L/home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib -lpetsc -Wl,-rpath,/home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib -lflapack -lfblas -lm -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -lmpichf90 -lgfortran -lm -lm -ldl -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -ldl Programs/ComputeRHS.c:1:23: error: petscdmda.h: No such file or directory Programs/ComputeRHS.c:2:22: error: petscksp.h: No such file or directory Programs/ComputeRHS.c:3:23: error: petscpcmg.h: No such file or directory Programs/ComputeRHS.c:4:23: error: petscdmmg.h: No such file or directory In file included from Programs/ComputeRHS.c:5: Programs/def.h:6:22: error: petscsys.h: No such file or directory In file included from Programs/ComputeRHS.c:5: Programs/def.h:25: error: expected specifier-qualifier-list before PetscReal Programs/def.h:29: error: expected specifier-qualifier-list before PetscScalar Programs/def.h:35: error: expected specifier-qualifier-list before PetscInt Programs/def.h:41: error: expected specifier-qualifier-list before PetscScalar Programs/ComputeRHS.c:10: error: expected =, ,, ;, asm or __attribute__ before ComputeRHS make: [ComputeRHS.o] Error 1 (ignored) make: *** No rule to make target `Initialization.o', needed by `ex29'. Stop. My codes are attached. Could you please help me to figure this out? thanks, Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: MakeFileProblem.zip Type: application/zip Size: 7253 bytes Desc: not available URL: From bsmith at mcs.anl.gov Thu Aug 25 13:58:11 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 25 Aug 2011 13:58:11 -0500 Subject: [petsc-users] Freeing Preconditioner Matrix Space (Again) In-Reply-To: References: Message-ID: On the C side of anything called by Fortran you must declare it as PC *pc rather than pc and then get the PC inside for example void pcasmfreespace_(PC *inpc) { PC pc = *inpc; .... On Aug 25, 2011, at 1:29 PM, Gaetan Kenway wrote: > Hello > > I've managed to get the c-function for freeing preconditioner memory written. The contents of my new 'pcasmfreespace.c' is below: > > #include /*I "petscpc.h" I*/ > > typedef struct { > PetscInt n, n_local, n_local_true; > PetscInt overlap; /* overlap requested by user */ > KSP *ksp; /* linear solvers for each block */ > VecScatter *restriction; /* mapping from global to subregion */ > VecScatter *localization; /* mapping from overlapping to non-overlapping subregion */ > VecScatter *prolongation; /* mapping from subregion to global */ > Vec *x,*y,*y_local; /* work vectors */ > IS *is; /* index set that defines each overlapping subdomain */ > IS *is_local; /* index set that defines each non-overlapping subdomain, may be NULL */ > Mat *mat,*pmat; /* mat is not currently used */ > PCASMType type; /* use reduced interpolation, restriction or both */ > PetscInt type_set; /* if user set this value (so won't change it for symmetric problems) */ > PetscInt same_local_solves; /* flag indicating whether all local solvers are same */ > PetscInt sort_indices; /* flag to sort subdomain indices */ > } PC_ASM; > > void pcasmfreespace_(PC pc) > { > > PC_ASM *osm = (PC_ASM*)pc->data; > PetscErrorCode ierr; > > if (osm->pmat) { > if (osm->n_local_true > 0) { > ierr = MatDestroyMatrices(osm->n_local_true,&osm->pmat);CHKERRQ(ierr); > } > osm->pmat = 0; > } > > if (pc->pmat) {ierr = MatDestroy(pc->pmat);CHKERRQ(ierr); pc->pmat = 0;} > > } > > Note the underscore as I'm trying to call it from Fortran. When I call it from fortran, I use: > > call pcasmfreespace(global_pc) > > This calls, the function, ok, but (according to Valgrind) I have an invalid read on the line containing: > > if (osm->pmat){ > > I suspect this is something to do with passing the fortran "pointer" of the PC to c, or something along this lines. Is there anything else special you have to do to pass the "fortran" petsc objects to c? > > Thanks > > Gaetan > > > Message: 2 > Date: Sun, 21 Aug 2011 17:22:28 -0500 > From: Barry Smith > Subject: Re: [petsc-users] Freeing Preconditioner Matrix Space > To: PETSc users list > Message-ID: <953121EF-B6AC-42EE-87BE-D4402C121652 at mcs.anl.gov> > Content-Type: text/plain; charset=us-ascii > > > You don't need to put that in the PETSc source. Just built it in the same directory you build your application and link it in like any of your application code. You will need to stick > #include /*I "petscpc.h" I*/ > > typedef struct { > PetscInt n, n_local, n_local_true; > PetscInt overlap; /* overlap requested by user */ > KSP *ksp; /* linear solvers for each block */ > VecScatter *restriction; /* mapping from global to subregion */ > VecScatter *localization; /* mapping from overlapping to non-overlapping subregion */ > VecScatter *prolongation; /* mapping from subregion to global */ > Vec *x,*y,*y_local; /* work vectors */ > IS *is; /* index set that defines each overlapping subdomain */ > IS *is_local; /* index set that defines each non-overlapping subdomain, may be NULL */ > Mat *mat,*pmat; /* mat is not currently used */ > PCASMType type; /* use reduced interpolation, restriction or both */ > PetscBool type_set; /* if user set this value (so won't change it for symmetric problems) */ > PetscBool same_local_solves; /* flag indicating whether all local solvers are same */ > PetscBool sort_indices; /* flag to sort subdomain indices */ > } PC_ASM; > > before the subroutine. > > Barry > > On Aug 21, 2011, at 3:24 PM, Gaetan Kenway wrote: > > > Hello > > > > I am attempting to implement a "hack" that was posted on the list a while back. I'm working with the adjoint linear system solver for a CFD solver. I'm using the ASM (or Block Jacobi) preconditioner with ILU(p) on each of the sub-domains. I use a different Preconditioner matrix (Pmat) than the actual jacobian. What I want to do is destroy the Pmat memory after the ILU factorization is performed. The hack that was posted is copied below: > > PCASMFreeSpace(PC pc) > > { > > PC_ASM *osm = (PC_ASM*)pc->data; > > PetscErrorCode ierr; > > > > if (osm->pmat) { > > if (osm->n_local_true > 0) { > > > > ierr = MatDestroyMatrices(osm->n_ > local_true,&osm->pmat);CHKERRQ(ierr); > > } > > osm->pmat = 0; > > } > > if (pc->pmat) {ierr = MatDestroy(pc->pmat);CHKERRQ(ierr); pc->pmat = 0;} > > > > return 0; > > } > > > > However, I've had no luck actually getting the function compiled into petsc. There are no erorrs reported with i type "make" in the asm directory, but when I try to use the function in my application it can't find the symbol while linking. Where does it go in the asm.c file? Does it use "static PetscErrorCode" or "PetscErrorCode PETSCKSP_DLLEXPORT"? Does it have to be added to the .h include files? What has to be done for it work with Fortran? > > > > Any suggestions would be greatly appreciated. This represents a significant chunk of my application's memory (10%-30%) and as such its too much to ignore. Also is there any chance something like this would make it into an actual PETSc release? > > > > Gaetan > > > > From knepley at gmail.com Thu Aug 25 14:00:01 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 25 Aug 2011 19:00:01 +0000 Subject: [petsc-users] Question on PETSc makefile In-Reply-To: References: Message-ID: On Thu, Aug 25, 2011 at 6:56 PM, Alan Wei wrote: > Dear all, > I hope you're having a nice day. > I met a problem on makefile of PETSc. Original PETSc has the makefile > in the same directory of the .c files. However, I want to put all .c files > to another directory. I changed the makefile also; however it always has a > error saying the PETSc header files can not be found: > Do not write your own rules for *.o files Matt > /home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/bin/mpicc -Wall > -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -o > ComputerRHS.o Programs/ComputeRHS.c > -L/home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib -lpetsc > -Wl,-rpath,/home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib > -lflapack -lfblas -lm -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl -lmpich > -lopa -lmpl -lrt -lpthread -lgcc_s -lmpichf90 -lgfortran -lm -lm -ldl > -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -ldl > Programs/ComputeRHS.c:1:23: error: petscdmda.h: No such file or directory > Programs/ComputeRHS.c:2:22: error: petscksp.h: No such file or directory > Programs/ComputeRHS.c:3:23: error: petscpcmg.h: No such file or directory > Programs/ComputeRHS.c:4:23: error: petscdmmg.h: No such file or directory > In file included from Programs/ComputeRHS.c:5: > Programs/def.h:6:22: error: petscsys.h: No such file or directory > In file included from Programs/ComputeRHS.c:5: > Programs/def.h:25: error: expected specifier-qualifier-list before > PetscReal > Programs/def.h:29: error: expected specifier-qualifier-list before > PetscScalar > Programs/def.h:35: error: expected specifier-qualifier-list before PetscInt > Programs/def.h:41: error: expected specifier-qualifier-list before > PetscScalar > Programs/ComputeRHS.c:10: error: expected =, ,, ;, asm or __attribute__ > before ComputeRHS > make: [ComputeRHS.o] Error 1 (ignored) > make: *** No rule to make target `Initialization.o', needed by `ex29'. > Stop. > > My codes are attached. Could you please help me to figure this out? > > thanks, > Alan > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Aug 25 14:01:59 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 25 Aug 2011 14:01:59 -0500 Subject: [petsc-users] Question on PETSc makefile In-Reply-To: References: Message-ID: ComputeRHS.o: Programs/ComputeRHS.c Programs/ComputeRHS.h Programs/def.h chkopts -${CLINKER} -o ComputerRHS.o Programs/ComputeRHS.c ${PETSC_SNES_LIB} Don't use the CLINKER to compile code. That is only for linking the executable. So for example you should be able to us myprogram: Programs/ComputeRHS.o -${CLINKER} -o myprogram Programs/ComputeRHS.o ${PETSC_SNES_LIB} Also it generally is much easier to have make files that site in the same directory as the source code, less hassle and less likely for things to go wrong. Barry On Aug 25, 2011, at 1:56 PM, Alan Wei wrote: > Dear all, > I hope you're having a nice day. > I met a problem on makefile of PETSc. Original PETSc has the makefile in the same directory of the .c files. However, I want to put all .c files to another directory. I changed the makefile also; however it always has a error saying the PETSc header files can not be found: > > /home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/bin/mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -o ComputerRHS.o Programs/ComputeRHS.c -L/home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib -lpetsc -Wl,-rpath,/home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib -lflapack -lfblas -lm -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -lmpichf90 -lgfortran -lm -lm -ldl -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -ldl > Programs/ComputeRHS.c:1:23: error: petscdmda.h: No such file or directory > Programs/ComputeRHS.c:2:22: error: petscksp.h: No such file or directory > Programs/ComputeRHS.c:3:23: error: petscpcmg.h: No such file or directory > Programs/ComputeRHS.c:4:23: error: petscdmmg.h: No such file or directory > In file included from Programs/ComputeRHS.c:5: > Programs/def.h:6:22: error: petscsys.h: No such file or directory > In file included from Programs/ComputeRHS.c:5: > Programs/def.h:25: error: expected specifier-qualifier-list before PetscReal > Programs/def.h:29: error: expected specifier-qualifier-list before PetscScalar > Programs/def.h:35: error: expected specifier-qualifier-list before PetscInt > Programs/def.h:41: error: expected specifier-qualifier-list before PetscScalar > Programs/ComputeRHS.c:10: error: expected =, ,, ;, asm or __attribute__ before ComputeRHS > make: [ComputeRHS.o] Error 1 (ignored) > make: *** No rule to make target `Initialization.o', needed by `ex29'. Stop. > > My codes are attached. Could you please help me to figure this out? > > thanks, > Alan > From balay at mcs.anl.gov Thu Aug 25 14:11:01 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 25 Aug 2011 14:11:01 -0500 (CDT) Subject: [petsc-users] Question on PETSc makefile In-Reply-To: References: Message-ID: PETSc makefiles don't support multiple source dirs properly. As an alternative - you can try using gnumake's vpath feature [thats lets you treat multiple sourcedirs as a single source dir]. Attaching makefile with this change.. Satish On Thu, 25 Aug 2011, Alan Wei wrote: > Dear all, > I hope you're having a nice day. > I met a problem on makefile of PETSc. Original PETSc has the makefile in > the same directory of the .c files. However, I want to put all .c files to > another directory. I changed the makefile also; however it always has a > error saying the PETSc header files can not be found: > > /home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/bin/mpicc -Wall > -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -o > ComputerRHS.o Programs/ComputeRHS.c > -L/home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib -lpetsc > -Wl,-rpath,/home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib > -lflapack -lfblas -lm -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl -lmpich > -lopa -lmpl -lrt -lpthread -lgcc_s -lmpichf90 -lgfortran -lm -lm -ldl > -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -ldl > Programs/ComputeRHS.c:1:23: error: petscdmda.h: No such file or directory > Programs/ComputeRHS.c:2:22: error: petscksp.h: No such file or directory > Programs/ComputeRHS.c:3:23: error: petscpcmg.h: No such file or directory > Programs/ComputeRHS.c:4:23: error: petscdmmg.h: No such file or directory > In file included from Programs/ComputeRHS.c:5: > Programs/def.h:6:22: error: petscsys.h: No such file or directory > In file included from Programs/ComputeRHS.c:5: > Programs/def.h:25: error: expected specifier-qualifier-list before PetscReal > Programs/def.h:29: error: expected specifier-qualifier-list before > PetscScalar > Programs/def.h:35: error: expected specifier-qualifier-list before PetscInt > Programs/def.h:41: error: expected specifier-qualifier-list before > PetscScalar > Programs/ComputeRHS.c:10: error: expected =, ,, ;, asm or __attribute__ > before ComputeRHS > make: [ComputeRHS.o] Error 1 (ignored) > make: *** No rule to make target `Initialization.o', needed by `ex29'. > Stop. > > My codes are attached. Could you please help me to figure this out? > > thanks, > Alan > -------------- next part -------------- CFLAGS = FFLAGS = CPPFLAGS = FPPFLAGS = CLEANFILES = include ${PETSC_DIR}/conf/variables include ${PETSC_DIR}/conf/rules VPATH=Programs ex29.o: ex29.c ComputeRHS.o: ComputeRHS.c ComputeRHS.h def.h Initialization.o: Initialization.c Initialization.h def.h OBJS = ex29.o ComputeRHS.o Initialization.o ex29: ${OBJS} chkopts -${CLINKER} -o ex29 ${OBJS} ${PETSC_SNES_LIB} runex29: -@${MPIEXEC} -n 1 ./ex29 -ksp_monitor_short -dmmg_nlevels 9 > ex29_1.tmp 2>&1; \ if (${DIFF} output/ex29_1.out ex29_1.tmp) then true; \ else echo "Possible problem with ex29_1, diffs above"; fi; \ ${RM} -f ex29_1.tmp runex29_2: -@${MPIEXEC} -n 1 ./ex29 -ksp_monitor_short -bc_type neumann -dmmg_nlevels 9 > ex29_2.tmp 2>&1; \ if (${DIFF} output/ex29_2.out ex29_2.tmp) then true; \ else echo "Possible problem with ex29_2, diffs above"; fi; \ ${RM} -f ex29_2.tmp From zhenglun.wei at gmail.com Thu Aug 25 14:11:09 2011 From: zhenglun.wei at gmail.com (Alan Wei) Date: Thu, 25 Aug 2011 14:11:09 -0500 Subject: [petsc-users] Question on PETSc makefile In-Reply-To: References: Message-ID: Thanks a lot for the explanation. I will simply leave the makefile in the same directory of codes to avoid making mistakes. best, Alan On Thu, Aug 25, 2011 at 2:01 PM, Barry Smith wrote: > > ComputeRHS.o: Programs/ComputeRHS.c Programs/ComputeRHS.h Programs/def.h > chkopts > -${CLINKER} -o ComputerRHS.o Programs/ComputeRHS.c > ${PETSC_SNES_LIB} > > Don't use the CLINKER to compile code. That is only for linking the > executable. So for example you should be able to us > > myprogram: Programs/ComputeRHS.o > -${CLINKER} -o myprogram Programs/ComputeRHS.o ${PETSC_SNES_LIB} > > Also it generally is much easier to have make files that site in the same > directory as the source code, less hassle and less likely for things to go > wrong. > > Barry > > > On Aug 25, 2011, at 1:56 PM, Alan Wei wrote: > > > Dear all, > > I hope you're having a nice day. > > I met a problem on makefile of PETSc. Original PETSc has the makefile > in the same directory of the .c files. However, I want to put all .c files > to another directory. I changed the makefile also; however it always has a > error saying the PETSc header files can not be found: > > > > /home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/bin/mpicc -Wall > -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -o > ComputerRHS.o Programs/ComputeRHS.c > -L/home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib -lpetsc > -Wl,-rpath,/home/zlwei/soft/mercurial/petsc-dev/arch-linux2-c-debug/lib > -lflapack -lfblas -lm -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl -lmpich > -lopa -lmpl -lrt -lpthread -lgcc_s -lmpichf90 -lgfortran -lm -lm -ldl > -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -ldl > > Programs/ComputeRHS.c:1:23: error: petscdmda.h: No such file or directory > > Programs/ComputeRHS.c:2:22: error: petscksp.h: No such file or directory > > Programs/ComputeRHS.c:3:23: error: petscpcmg.h: No such file or directory > > Programs/ComputeRHS.c:4:23: error: petscdmmg.h: No such file or directory > > In file included from Programs/ComputeRHS.c:5: > > Programs/def.h:6:22: error: petscsys.h: No such file or directory > > In file included from Programs/ComputeRHS.c:5: > > Programs/def.h:25: error: expected specifier-qualifier-list before > PetscReal > > Programs/def.h:29: error: expected specifier-qualifier-list before > PetscScalar > > Programs/def.h:35: error: expected specifier-qualifier-list before > PetscInt > > Programs/def.h:41: error: expected specifier-qualifier-list before > PetscScalar > > Programs/ComputeRHS.c:10: error: expected =, ,, ;, asm or __attribute__ > before ComputeRHS > > make: [ComputeRHS.o] Error 1 (ignored) > > make: *** No rule to make target `Initialization.o', needed by `ex29'. > Stop. > > > > My codes are attached. Could you please help me to figure this out? > > > > thanks, > > Alan > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Thu Aug 25 15:49:26 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Thu, 25 Aug 2011 22:49:26 +0200 Subject: [petsc-users] Number of mallocs during Maximum nonzeros in any row Message-ID: I am profiling my application aiming to support a few grid types. I am getting the expected messages, like: MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 or [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 15 X 15; storage space: 0 unneeded,171 used but in some cases, and not other, I see: [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during Maximum nonzeros in any row is 12 What is it and does it indicate inaccurate calculation of d_nnz/o_nnz? Many thanks, Dominik From bsmith at mcs.anl.gov Thu Aug 25 15:54:37 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 25 Aug 2011 15:54:37 -0500 Subject: [petsc-users] Number of mallocs during Maximum nonzeros in any row In-Reply-To: References: Message-ID: <95437544-D982-4115-92B8-E5A80565A5B2@mcs.anl.gov> Unfortunately on some systems output from different processes get intermixed making it hard to read. It looks like the 12 is associated with the Max nonzeros per row here so you are ok. Barry On Aug 25, 2011, at 3:49 PM, Dominik Szczerba wrote: > I am profiling my application aiming to support a few grid types. I am > getting the expected messages, like: > > MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > or > > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 15 X 15; storage space: 0 > unneeded,171 used > > but in some cases, and not other, I see: > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during Maximum nonzeros > in any row is 12 > > What is it and does it indicate inaccurate calculation of d_nnz/o_nnz? > > Many thanks, > Dominik From stali at geology.wisc.edu Thu Aug 25 16:15:17 2011 From: stali at geology.wisc.edu (Tabrez Ali) Date: Thu, 25 Aug 2011 16:15:17 -0500 Subject: [petsc-users] Mesh partitioning and MPI calls Message-ID: <4E56BB65.3030205@geology.wisc.edu> Hello I have an unstructured FE mesh which I am partitioning using Metis. In the first case I only use the element partitioning info and discard the nodal partitioning info i.e., the original ordering is same as petsc's global ordering. In the second case I do use the nodal partitioning info and nodes are distributed accordingly. I would expect that in the 2nd scenario the total number of MPI messages (at the end of the solve) would be lower than the 1st. However I see that opposite is true. See the plot at http://stali.freeshell.org/mpi.png The number on the y axis is the last column of the "MPI messages:" field from the -log_summary output. Any ideas as to why this is happening. Does relying on total number of MPI messages as a performance measure even make sense. Please excuse my ignorance on the subject. Alternatively what is a good way to measure how good the Metis partitioning is? Thanks in advance Tabrez From dominik at itis.ethz.ch Thu Aug 25 16:14:25 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Thu, 25 Aug 2011 23:14:25 +0200 Subject: [petsc-users] Mesh partitioning and MPI calls In-Reply-To: <4E56BB65.3030205@geology.wisc.edu> References: <4E56BB65.3030205@geology.wisc.edu> Message-ID: Your expectation seems correct. How do you organize your points/dofs? It is very important for communication. Can you inspect which MPI messages are counted? Communication during matrix assembly may work better, but somewhere else in your code you may still assume cell ordering, thus contributing to the bigger total communication cost. Dominik On Thu, Aug 25, 2011 at 11:15 PM, Tabrez Ali wrote: > Hello > > I have an unstructured FE mesh which I am partitioning using Metis. > > In the first case I only use the element partitioning info and discard the > nodal partitioning info i.e., the original ordering is same as petsc's > global ordering. In the second case I do use the nodal partitioning info and > nodes are distributed accordingly. > > I would expect that in the 2nd scenario the total number of MPI messages (at > the end of the solve) would be lower than the 1st. However I see that > opposite is true. See the plot at http://stali.freeshell.org/mpi.png > > The number on the y axis is the last column of the "MPI messages:" field > from the -log_summary output. > > Any ideas as to why this is happening. Does relying on total number of MPI > messages as a performance measure even make sense. Please excuse my > ignorance on the subject. > > Alternatively what is a good way to measure how good the Metis partitioning > is? > > Thanks in advance > > Tabrez > > From stali at geology.wisc.edu Thu Aug 25 16:57:08 2011 From: stali at geology.wisc.edu (Tabrez Ali) Date: Thu, 25 Aug 2011 16:57:08 -0500 Subject: [petsc-users] Mesh partitioning and MPI calls In-Reply-To: References: <4E56BB65.3030205@geology.wisc.edu> Message-ID: <4E56C534.70401@geology.wisc.edu> I see the same behavior even if I stop the code right after my stiffness matrix is assembled. The only MPI comm before that is an epart/npart integer array BCast (from proc 0 after the partitioning routine is called). Tabrez On 08/25/2011 04:14 PM, Dominik Szczerba wrote: > Your expectation seems correct. > > How do you organize your points/dofs? It is very important for communication. > > Can you inspect which MPI messages are counted? Communication during > matrix assembly may work better, but somewhere else in your code you > may still assume cell ordering, thus contributing to the bigger total > communication cost. > > Dominik > > On Thu, Aug 25, 2011 at 11:15 PM, Tabrez Ali wrote: >> Hello >> >> I have an unstructured FE mesh which I am partitioning using Metis. >> >> In the first case I only use the element partitioning info and discard the >> nodal partitioning info i.e., the original ordering is same as petsc's >> global ordering. In the second case I do use the nodal partitioning info and >> nodes are distributed accordingly. >> >> I would expect that in the 2nd scenario the total number of MPI messages (at >> the end of the solve) would be lower than the 1st. However I see that >> opposite is true. See the plot at http://stali.freeshell.org/mpi.png >> >> The number on the y axis is the last column of the "MPI messages:" field >> from the -log_summary output. >> >> Any ideas as to why this is happening. Does relying on total number of MPI >> messages as a performance measure even make sense. Please excuse my >> ignorance on the subject. >> >> Alternatively what is a good way to measure how good the Metis partitioning >> is? >> >> Thanks in advance >> >> Tabrez >> >> From zhenglun.wei at gmail.com Thu Aug 25 18:24:09 2011 From: zhenglun.wei at gmail.com (Alan Wei) Date: Thu, 25 Aug 2011 18:24:09 -0500 Subject: [petsc-users] Computed Nodes, Ghosted Nodes and Boundary Nodes Message-ID: Dear all, I hope you had a nice day. I have a little bit confusion on src/ksp/ksp/example/tutorial/ex29.c, which is solving a Poisson equation. The key problem is that I confused what is the ghosted node, computed nodes and boundary nodes. 1) For example, if the mesh has 20 * 50 grid on x-, y- direction. For my understanding, nodes of i = 0, i = 19, j = 0, j = 49 are all boundary nodes and others are computation nodes. If a Dirichlet boundary condition is applied in node j = 49, then the value of node j = 49 should be a constant; while, if a Neumann boundary condition is applied in j = 49, then change rate from node j = 48 to j = 49 should be a constant. 2) Moreover, If this code is ran with 2 processes, which 2 on y-direction. This leads a 20 * 25 mesh for each nodes, then the ghost nodes is created around the local mesh in order to pass data back and forth with the other node. Is that correct? However, what is the ghost nodes coordinate value and index? Thanks in advance, Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Aug 25 23:49:58 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 26 Aug 2011 04:49:58 +0000 Subject: [petsc-users] Mesh partitioning and MPI calls In-Reply-To: <4E56BB65.3030205@geology.wisc.edu> References: <4E56BB65.3030205@geology.wisc.edu> Message-ID: On Thu, Aug 25, 2011 at 9:15 PM, Tabrez Ali wrote: > Hello > > I have an unstructured FE mesh which I am partitioning using Metis. > > In the first case I only use the element partitioning info and discard the > nodal partitioning info i.e., the original ordering is same as petsc's > global ordering. In the second case I do use the nodal partitioning info and > nodes are distributed accordingly. > > I would expect that in the 2nd scenario the total number of MPI messages > (at the end of the solve) would be lower than the 1st. However I see that > opposite is true. See the plot at http://stali.freeshell.org/**mpi.png > > The number on the y axis is the last column of the "MPI messages:" field > from the -log_summary output. > > Any ideas as to why this is happening. Does relying on total number of MPI > messages as a performance measure even make sense. Please excuse my > ignorance on the subject. > > Alternatively what is a good way to measure how good the Metis partitioning > is? > The thing to do here is take a case like 2 proc that can be completely understood, and get down to the details. I think there is a probably just a simple misunderstanding here. The first thing to check is that you are partitioning what you think. By default, Metis partitions the vertices of a graph, not elements, Thus you usually have to give Metis the "dual" of your finite element mesh. I would take a small (maybe 8 or 10 elements) mesh and look at the original and Metis partitions. If Metis does not look better, something is wrong. Matt > Thanks in advance > > Tabrez > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Aug 25 23:52:29 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 26 Aug 2011 04:52:29 +0000 Subject: [petsc-users] Computed Nodes, Ghosted Nodes and Boundary Nodes In-Reply-To: References: Message-ID: On Thu, Aug 25, 2011 at 11:24 PM, Alan Wei wrote: > Dear all, > I hope you had a nice day. > I have a little bit confusion on src/ksp/ksp/example/tutorial/ex29.c, > which is solving a Poisson equation. The key problem is that I confused > what is the ghosted node, computed nodes and boundary nodes. > 1) For example, if the mesh has 20 * 50 grid on x-, y- direction. For > my understanding, nodes of i = 0, i = 19, j = 0, j = 49 are all boundary > nodes and others are computation nodes. If a Dirichlet boundary condition > is applied in node j = 49, then the value of node j = 49 should be a > constant; while, if a Neumann boundary condition is applied in j = 49, > then change rate from node j = 48 to j = 49 should be a constant. > Fine. > 2) Moreover, If this code is ran with 2 processes, which 2 on > y-direction. This leads a 20 * 25 mesh for each nodes, then the ghost nodes > is created around the local mesh in order to pass data back and forth with > the other node. Is that correct? However, what is the ghost nodes coordinate > value and index? > For proc0, the ghost nodes would be j = 25, and for proc 1 j = 24. However, you should not need to worry about this. Matt > Thanks in advance, > Alan > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Fri Aug 26 01:59:32 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 26 Aug 2011 08:59:32 +0200 Subject: [petsc-users] Debugging MatAssemblyEnd Message-ID: I get the following crash: Fatal error in MPI_Allreduce: Error message texts are not available[cli_1]: aborting job: Fatal error in MPI_Allreduce: Error message texts are not available INTERNAL ERROR: Invalid error class (66) encountered while returning from MPI_Allreduce. Please file a bug report. No error stack is available. Fatal error in MPI_Allreduce: Error message texts are not available[cli_0]: aborting job: Fatal error in MPI_Allreduce: Error message texts are not available INTERNAL ERROR: Invalid error class (66) encountered while returning from MPI_Allreduce. Please file a bug report. No error stack is available. Fatal error in MPI_Allreduce: Error message texts are not available[cli_3]: aborting job: Fatal error in MPI_Allreduce: Error message texts are not available but only in one case, while a few others work as expected. Running in debugger points to the call in my code: ierr = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); and that in turn leads into the above listed MPI_Allreduce. I did all paranoid checks in my matrix assembly for illegal indices, all clean. It is my only call to MatAssemblyEnd, the problem is linear, but with several unknowns per point. A few other cases do not crash. Valgrind (on a small case) reports something probably irrelevant (attached at the bottom). How should I debug this? Thanks for any hints, Dominik ==9521== Syscall param writev(vector[...]) points to uninitialised byte(s) ==9521== at 0x6DD8789: writev (writev.c:56) ==9521== by 0x517348: MPIDU_Sock_writev (sock_immed.i:610) ==9521== by 0x519633: MPIDI_CH3_iSendv (ch3_isendv.c:84) ==9521== by 0x4FD446: MPIDI_CH3_EagerContigIsend (ch3u_eager.c:509) ==9521== by 0x4FF3D4: MPID_Isend (mpid_isend.c:118) ==9521== by 0x4E6D1A: MPIC_Isend (helper_fns.c:210) ==9521== by 0xEE90DD: MPIR_Alltoall (alltoall.c:420) ==9521== by 0xEE9940: PMPI_Alltoall (alltoall.c:685) ==9521== by 0xE5DD4D: SetUp__ (setup.c:122) ==9521== by 0xE5E59C: PartitionSmallGraph__ (weird.c:39) ==9521== by 0xE5B888: ParMETIS_V3_PartKway (kmetis.c:131) ==9521== by 0x72D82C: MatPartitioningApply_Parmetis (pmetis.c:97) ==9521== by 0x729E28: MatPartitioningApply (partition.c:236) ==9521== by 0x52A240: PetscSolver::LoadMesh(std::string const&) (PetscSolver.cxx:625) ==9521== by 0x4C61AC: SM3T4_USER::ProcessInputFile() (sm3t4mpi_main.cxx:227) ==9521== by 0x4C48F3: main (sm3t4mpi_main.cxx:665) ==9521== Address 0xc7ebe3c is 12 bytes inside a block of size 72 alloc'd ==9521== at 0x4C28FAC: malloc (vg_replace_malloc.c:236) ==9521== by 0xE6E84A: GKmalloc__ (util.c:151) ==9521== by 0xE69CC1: PreAllocateMemory__ (memory.c:38) ==9521== by 0xE5B76B: ParMETIS_V3_PartKway (kmetis.c:116) ==9521== by 0x72D82C: MatPartitioningApply_Parmetis (pmetis.c:97) ==9521== by 0x729E28: MatPartitioningApply (partition.c:236) ==9521== by 0x52A240: PetscSolver::LoadMesh(std::string const&) (PetscSolver.cxx:625) ==9521== by 0x4C61AC: SM3T4_USER::ProcessInputFile() (sm3t4mpi_main.cxx:227) ==9521== by 0x4C48F3: main (sm3t4mpi_main.cxx:665) From jedbrown at mcs.anl.gov Fri Aug 26 02:03:38 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 26 Aug 2011 02:03:38 -0500 Subject: [petsc-users] Debugging MatAssemblyEnd In-Reply-To: References: Message-ID: On Fri, Aug 26, 2011 at 01:59, Dominik Szczerba wrote: > ierr = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > When you run in the debugger and break after it has obviously hung, are all processes stopped at the same place? If you see an error condition, you can run CHKMEMQ; MPI_Barrier(((PetscObject)A)->comm); MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); If it hangs, check where every process is stuck. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Fri Aug 26 03:37:45 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 26 Aug 2011 10:37:45 +0200 Subject: [petsc-users] Debugging MatAssemblyEnd In-Reply-To: References: Message-ID: > When you run in the debugger and break after it has obviously hung, are all > processes stopped at the same place? Of course not, they are stuck at barriers elsewhere. Thanks for the valuable question. > If you see an error condition, you can > run > CHKMEMQ; > MPI_Barrier(((PetscObject)A)->comm); > MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); > If it hangs, check where every process is stuck. I obviously seem to be missing some barriers. But why would I need MPI_Barrier(((PetscObject)A)->comm) and not just MPI_Barrier(PETSC_COMM_WORLD)? Would that only force a barrier for A-related traffic? Dominik From knepley at gmail.com Fri Aug 26 04:01:17 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 26 Aug 2011 09:01:17 +0000 Subject: [petsc-users] Debugging MatAssemblyEnd In-Reply-To: References: Message-ID: On Fri, Aug 26, 2011 at 8:37 AM, Dominik Szczerba wrote: > > When you run in the debugger and break after it has obviously hung, are > all > > processes stopped at the same place? > > Of course not, they are stuck at barriers elsewhere. Thanks for the > valuable question. > > > If you see an error condition, you can > > run > > CHKMEMQ; > > MPI_Barrier(((PetscObject)A)->comm); > > MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); > > MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); > > If it hangs, check where every process is stuck. > > I obviously seem to be missing some barriers. But why would I need > MPI_Barrier(((PetscObject)A)->comm) and not just > MPI_Barrier(PETSC_COMM_WORLD)? Would that only force a barrier for > A-related traffic? The idea here is the following: 1) We would like to isolate the mismatch in synchronizations 2) We can place barriers in the code to delimit the sections which contain the offending code, and also eliminate bugs in MatAssembly as a possible source of problems. 3) Do you have any MPI code you wrote yourself in here? Matt > > Dominik > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Fri Aug 26 04:19:57 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 26 Aug 2011 11:19:57 +0200 Subject: [petsc-users] Debugging MatAssemblyEnd In-Reply-To: References: Message-ID: I seem to have had a classical deadlock, A was being assembled while some threads lurked around elsewhere. Adding some barriers seems to fix the problem, at least with the cases I currently have. What I still miss is what would be the advantage of MPI_Barrier(((PetscObject)A)->comm) over MPI_Barrier(PETSC_COMM_WORLD). Many thanks Dominik On Fri, Aug 26, 2011 at 11:01 AM, Matthew Knepley wrote: > On Fri, Aug 26, 2011 at 8:37 AM, Dominik Szczerba > wrote: >> >> > When you run in the debugger and break after it has obviously hung, are >> > all >> > processes stopped at the same place? >> >> Of course not, they are stuck at barriers elsewhere. Thanks for the >> valuable question. >> >> > If you see an error condition, you can >> > run >> > CHKMEMQ; >> > MPI_Barrier(((PetscObject)A)->comm); >> > MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); >> > MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); >> > If it hangs, check where every process is stuck. >> >> I obviously seem to be missing some barriers. But why would I need >> MPI_Barrier(((PetscObject)A)->comm) and not just >> MPI_Barrier(PETSC_COMM_WORLD)? Would that only force a barrier for >> A-related traffic? > > The idea here is the following: > ? 1) We would like to isolate the mismatch in synchronizations > ? 2) We can place barriers in the code to delimit the sections which contain > the offending code, > ? ? ? ?and also eliminate bugs in MatAssembly as a possible source of > problems. > ? 3) Do you have any MPI code you wrote yourself in here? > ? ? ?Matt > >> >> Dominik > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > From jedbrown at mcs.anl.gov Fri Aug 26 07:53:25 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 26 Aug 2011 07:53:25 -0500 Subject: [petsc-users] Debugging MatAssemblyEnd In-Reply-To: References: Message-ID: On Fri, Aug 26, 2011 at 04:19, Dominik Szczerba wrote: > I seem to have had a classical deadlock, A was being assembled while > some threads lurked around elsewhere. Adding some barriers seems to > fix the problem, at least with the cases I currently have. > Barriers should never affect the correctness of a pure MPI code that doesn't do weird things like communicate through the filesystem. We use the barriers for debugging, but they can generally be removed once the underlying issue is sorted out. Also, when you say "threads", are you referring to MPI processes, or are you using actual threads (e.g. pthreads or OpenMP)? > > What I still miss is what would be the advantage of > MPI_Barrier(((PetscObject)A)->comm) over > MPI_Barrier(PETSC_COMM_WORLD). > I don't know whether all processes on PETSC_COMM_WORLD are supposed to pass through this assembly. If A was on a subcommunicator, then only those processes should be calling assembly. Note that these communicators are the same for many users. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Fri Aug 26 08:12:32 2011 From: zonexo at gmail.com (TAY wee-beng) Date: Fri, 26 Aug 2011 15:12:32 +0200 Subject: [petsc-users] No change in a.out compilation size after using PETSc shared library In-Reply-To: References: <4E565671.10907@gmail.com> Message-ID: <4E579BC0.6030101@gmail.com> Hi Barry, Sorry, I am a mistake in the compilation directory and now it's about 1mb instead of 12mb. So it should be correct now. Yours sincerely, TAY wee-beng On 25/8/2011 4:14 PM, Barry Smith wrote: > Send configure.log make.log and ls ${PETSC_ARCH/lib to petsc-maint at mcs.anl.gov > > Barry > > On Aug 25, 2011, at 9:04 AM, TAY wee-beng wrote: > >> Hi, >> >> I managed to compile PETSc shared library. >> >> However, when I compile my code, my a.out 's size is still around 12mb, which is similar to using PETSc static library. >> >> I thought it will be a much smaller exe. Shouldn't it be? >> >> Thank you! >> >> -- >> Yours sincerely, >> >> TAY wee-beng >> From dominik at itis.ethz.ch Fri Aug 26 08:12:37 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 26 Aug 2011 15:12:37 +0200 Subject: [petsc-users] Debugging MatAssemblyEnd In-Reply-To: References: Message-ID: >> I seem to have had a classical deadlock, A was being assembled while >> some threads lurked around elsewhere. Adding some barriers seems to >> fix the problem, at least with the cases I currently have. > > Barriers should never affect the correctness of a pure MPI code that doesn't > do weird things like communicate through the filesystem. We use the barriers > for debugging, but they can generally be removed once the underlying issue > is sorted out. Do you claim I should not use MPI_Barrier at all when only programming Petsc? Is really all required synchronization managed automatically? I do some raw MPI programming before and after, but not inside the Petsc core solver. > Also, when you say "threads", are you referring to MPI processes, or are you > using actual threads (e.g. pthreads or OpenMP)? I meant the MPI processes, sorry for the confusion. >> What I still miss is what would be the advantage of >> MPI_Barrier(((PetscObject)A)->comm) over >> MPI_Barrier(PETSC_COMM_WORLD). > > I don't know whether all processes on PETSC_COMM_WORLD are supposed to pass > through this assembly. If A was on a subcommunicator, then only those > processes should be calling assembly. Note that these communicators are the > same for many users. They are not the same for me, but I think I see the point now. Thanks, Dominik From jedbrown at mcs.anl.gov Fri Aug 26 08:18:35 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 26 Aug 2011 08:18:35 -0500 Subject: [petsc-users] Debugging MatAssemblyEnd In-Reply-To: References: Message-ID: On Fri, Aug 26, 2011 at 08:12, Dominik Szczerba wrote: > Do you claim I should not use MPI_Barrier at all when only programming > Petsc? Is really all required synchronization managed automatically? > Yes, apart from unconventional/discouraged things like communicating through the file system, and even then, you should usually do without barriers. > > I do some raw MPI programming before and after, but not inside the > Petsc core solver. > Common problems include having one or more of your processes hung up in this earlier phase and possible tag collisions. If you are doing nontrivial things in your pure MPI code, you should either create the communicator yourself or MPI_Comm_dup() one that you get from PETSc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Fri Aug 26 10:17:35 2011 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Fri, 26 Aug 2011 11:17:35 -0400 Subject: [petsc-users] Freeing Preconditioner Memory Message-ID: Hello I've almost got the petsc preconditioner free memory function to work. It now run the actual free-ing part, but segfaults on when it uses GMRES to solve the linear system. The relevant section of code that setups the ksp/pc/localpc objects is below (in fortran) as well as the c-function for freeing the PC memory. !***************** Fortran Code ********************* ksp_solver_type = 'asm' ksp_subspace = 50 asm_overlap =1 local_pc_ilu_level = 1 local_pc_ordering = 'rcm' !Setup global_ksp type and get the global PC call KSPSetType(ksp,ksp_solver_type,ierr) call KSPGMRESSetRestart(ksp, ksp_subspace,ierr) call KSPSetPreconditionerSide(ksp,PC_RIGHT,ierr); call KSPGetPC(ksp,global_pc,ierr); ! Setup global_pc call PCSetType(global_pc,"asm",ierr) call PCASMSetOverlap(global_pc,asm_overlap,ierr) call PCSetup(global_pc,ierr) call PCASMGetSubKSP(global_pc, nlocal, first, local_ksp, ierr ) ! Setup local_pc call KSPGetPC(local_ksp, local_pc, ierr ) call PCSetType(local_pc, 'ilu', ierr) call PCFactorSetLevels(local_pc, local_pc_ilu_level, ierr) call PCFactorSetMatOrderingtype(local_pc, local_pc_ordering, ierr ) call KSPSetType(local_ksp, KSPPREONLY, ierr) call KSPSetup(global_ksp,ierr) call KSPSetUpOnBlocks(global_ksp,ierr) ! Free Memory call PCASMFreeSpace(global_pc) ! ************* C Code ************************** void pcasmfreespace_(PC *inpc) { PC pc = *inpc; PC_ASM *osm = (PC_ASM*)pc->data; PetscErrorCode ierr; if (osm->pmat) { if (osm->n_local_true > 0) { ierr = MatDestroyMatrices(osm->n_local_true,&osm->pmat); CHKERRQ(ierr); } osm->pmat = 0; } if (pc->pmat) { ierr = MatDestroy(pc->pmat); CHKERRQ(ierr); pc->pmat = 0; } } ! *********************************************8 When I run it, I get an error is PCApplyBAorAB [0]PETSC ERROR: [0] MatMult line 1877 src/mat/interface/matrix.c [0]PETSC ERROR: [0] PCApplyBAorAB line 540 src/ksp/pc/interface/precon.c [0]PETSC ERROR: [0] GMREScycle line 132 src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: [0] KSPSolve_GMRES line 227 src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: [0] KSPSolve line 308 src/ksp/ksp/interface/itfunc.c It appears then that you must keep the un-factored preconditioner matrix in memory. Is this actually the case? Or have I done something silly here? Thanks, Gaetan On Aug 25, 2011, at 1:29 PM, Gaetan Kenway wrote: > Hello > > I've managed to get the c-function for freeing preconditioner memory written. The contents of my new 'pcasmfreespace.c' is below: > > #include /*I "petscpc.h" I*/ > > typedef struct { > PetscInt n, n_local, n_local_true; > PetscInt overlap; /* overlap requested by user */ > KSP *ksp; /* linear solvers for each block */ > VecScatter *restriction; /* mapping from global to subregion */ > VecScatter *localization; /* mapping from overlapping to non-overlapping subregion */ > VecScatter *prolongation; /* mapping from subregion to global */ > Vec *x,*y,*y_local; /* work vectors */ > IS *is; /* index set that defines each overlapping subdomain */ > IS *is_local; /* index set that defines each non-overlapping subdomain, may be NULL */ > Mat *mat,*pmat; /* mat is not currently used */ > PCASMType type; /* use reduced interpolation, restriction or both */ > PetscInt type_set; /* if user set this value (so won't change it for symmetric problems) */ > PetscInt same_local_solves; /* flag indicating whether all local solvers are same */ > PetscInt sort_indices; /* flag to sort subdomain indices */ > } PC_ASM; > > void pcasmfreespace_(PC pc) > { > > PC_ASM *osm = (PC_ASM*)pc->data; > PetscErrorCode ierr; > > if (osm->pmat) { > if (osm->n_local_true > 0) { > ierr = MatDestroyMatrices(osm->n_ local_true,&osm->pmat);CHKERRQ(ierr); > } > osm->pmat = 0; > } > > if (pc->pmat) {ierr = MatDestroy(pc->pmat);CHKERRQ(ierr); pc->pmat = 0;} > > } > > Note the underscore as I'm trying to call it from Fortran. When I call it from fortran, I use: > > call pcasmfreespace(global_pc) > > This calls, the function, ok, but (according to Valgrind) I have an invalid read on the line containing: > > if (osm->pmat){ > > I suspect this is something to do with passing the fortran "pointer" of the PC to c, or something along this lines. Is there anything else special you have to do to pass the "fortran" petsc objects to c? > > Thanks > > Gaetan > > > Message: 2 > Date: Sun, 21 Aug 2011 17:22:28 -0500 > From: Barry Smith > Subject: Re: [petsc-users] Freeing Preconditioner Matrix Space > To: PETSc users list > Message-ID: <953121EF-B6AC-42EE-87BE-D4402C121652 at mcs.anl.gov> > Content-Type: text/plain; charset=us-ascii > > > You don't need to put that in the PETSc source. Just built it in the same directory you build your application and link it in like any of your application code. You will need to stick > #include /*I "petscpc.h" I*/ > > typedef struct { > PetscInt n, n_local, n_local_true; > PetscInt overlap; /* overlap requested by user */ > KSP *ksp; /* linear solvers for each block */ > VecScatter *restriction; /* mapping from global to subregion */ > VecScatter *localization; /* mapping from overlapping to non-overlapping subregion */ > VecScatter *prolongation; /* mapping from subregion to global */ > Vec *x,*y,*y_local; /* work vectors */ > IS *is; /* index set that defines each overlapping subdomain */ > IS *is_local; /* index set that defines each non-overlapping subdomain, may be NULL */ > Mat *mat,*pmat; /* mat is not currently used */ > PCASMType type; /* use reduced interpolation, restriction or both */ > PetscBool type_set; /* if user set this value (so won't change it for symmetric problems) */ > PetscBool same_local_solves; /* flag indicating whether all local solvers are same */ > PetscBool sort_indices; /* flag to sort subdomain indices */ > } PC_ASM; > > before the subroutine. > > Barry > > On Aug 21, 2011, at 3:24 PM, Gaetan Kenway wrote: > > > Hello > > > > I am attempting to implement a "hack" that was posted on the list a while back. I'm working with the adjoint linear system solver for a CFD solver. I'm using the ASM (or Block Jacobi) preconditioner with ILU(p) on each of the sub-domains. I use a different Preconditioner matrix (Pmat) than the actual jacobian. What I want to do is destroy the Pmat memory after the ILU factorization is performed. The hack that was posted is copied below: > > PCASMFreeSpace(PC pc) > > { > > PC_ASM *osm = (PC_ASM*)pc->data; > > PetscErrorCode ierr; > > > > if (osm->pmat) { > > if (osm->n_local_true > 0) { > > > > ierr = MatDestroyMatrices(osm->n_ > local_true,&osm->pmat);CHKERRQ(ierr); > > } > > osm->pmat = 0; > > } > > if (pc->pmat) {ierr = MatDestroy(pc->pmat);CHKERRQ(ierr); pc->pmat = 0;} > > > > return 0; > > } > > > > However, I've had no luck actually getting the function compiled into petsc. There are no erorrs reported with i type "make" in the asm directory, but when I try to use the function in my application it can't find the symbol while linking. Where does it go in the asm.c file? Does it use "static PetscErrorCode" or "PetscErrorCode PETSCKSP_DLLEXPORT"? Does it have to be added to the .h include files? What has to be done for it work with Fortran? > > > > Any suggestions would be greatly appreciated. This represents a significant chunk of my application's memory (10%-30%) and as such its too much to ignore. Also is there any chance something like this would make it into an actual PETSc release? > > > > Gaetan > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stali at geology.wisc.edu Fri Aug 26 12:32:28 2011 From: stali at geology.wisc.edu (Tabrez Ali) Date: Fri, 26 Aug 2011 12:32:28 -0500 Subject: [petsc-users] Mesh partitioning and MPI calls In-Reply-To: References: <4E56BB65.3030205@geology.wisc.edu> Message-ID: <4E57D8AC.1010005@geology.wisc.edu> Matt you were right I was stupidly passing the wrong array (column major order) to Metis. My code is in Fortran. I am surprised Metis didnt crash before. Thanks anyway. Tabrez On 08/25/2011 11:49 PM, Matthew Knepley wrote: > On Thu, Aug 25, 2011 at 9:15 PM, Tabrez Ali > wrote: > > Hello > > I have an unstructured FE mesh which I am partitioning using Metis. > > In the first case I only use the element partitioning info and > discard the nodal partitioning info i.e., the original ordering is > same as petsc's global ordering. In the second case I do use the > nodal partitioning info and nodes are distributed accordingly. > > I would expect that in the 2nd scenario the total number of MPI > messages (at the end of the solve) would be lower than the 1st. > However I see that opposite is true. See the plot at > http://stali.freeshell.org/mpi.png > > The number on the y axis is the last column of the "MPI messages:" > field from the -log_summary output. > > Any ideas as to why this is happening. Does relying on total > number of MPI messages as a performance measure even make sense. > Please excuse my ignorance on the subject. > > Alternatively what is a good way to measure how good the Metis > partitioning is? > > > The thing to do here is take a case like 2 proc that can be completely > understood, and get down to the details. I > think there is a probably just a simple misunderstanding here. > > The first thing to check is that you are partitioning what you think. > By default, Metis partitions the vertices of a graph, > not elements, Thus you usually have to give Metis the "dual" of your > finite element mesh. I would take a small (maybe > 8 or 10 elements) mesh and look at the original and Metis partitions. > If Metis does not look better, something is wrong. > > Matt > > Thanks in advance > > Tabrez > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at ibrae.ac.ru Fri Aug 26 15:57:01 2011 From: ram at ibrae.ac.ru (=?UTF-8?B?0JDQu9C10LrRgdC10Lkg0KDRj9C30LDQvdC+0LI=?=) Date: Sat, 27 Aug 2011 00:57:01 +0400 Subject: [petsc-users] performance issue solving multiple linear systems of the same size with the different preconditioning methods In-Reply-To: References: Message-ID: Thank you for your response! I have the memory leak in both my programs. But I don't create plenty of vectors. My code looks like: ***INIT_ALL*** PetscLogStageRegister("Iteration :", &StageNum1); PetscLogStagePush(StageNum1); KSPSolve(dKSP, dvec_origRHS, dvec_Solution); PetscLogStagePop(); ***DESTROY_ALL*** And when I comment (or delete) KSPSolve, the log_summary output is: ===================================================== Memory usage is given in bytes: Creations Destructions Memory Descendants' Mem Object Type Reports information only for process 0. --- Event Stage 0: Main Stage 1 1 729472 0 Application Order 1 1 225452 0 Distributed array 8 8 1533424 0 Vec 3 3 2604 0 Vec Scatter 8 8 613852 0 Index Set 1 1 221304 0 IS L to G Mapping 3 3 16603440 0 Matrix 1 1 832 0 Krylov Solver 1 1 688 0 Preconditioner 1 1 448 0 PetscRandom --- Event Stage 1: Iteration : ===================================================== When I run the code with KSPSolve instruction, it gives me: ===================================================== Memory usage is given in bytes: Creations Destructions Memory Descendants' Mem Object Type Reports information only for process 0. --- Event Stage 0: Main Stage 1 0 0 0 Application Order 1 0 0 0 Distributed array 8 17 4963592 0 Vec 3 2 1736 0 Vec Scatter 8 12 1425932 0 Index Set 1 0 0 0 IS L to G Mapping 3 5 50158132 0 Matrix 1 2 1664 0 Krylov Solver 1 2 1440 0 Preconditioner 1 1 448 0 PetscRandom 0 1 544 0 Viewer --- Event Stage 1: Iteration : 355 173 64692312 0 Vec 1 0 0 0 Vec Scatter 6 2 1024 0 Index Set 2 0 0 0 Matrix 1 0 0 0 Krylov Solver 1 0 0 0 Preconditioner 2 1 544 0 Viewer ===================================================== 2011/8/25 Jed Brown > On Tue, Aug 23, 2011 at 02:37, ??????? ??????? wrote: > >> When i delete the 4-5-6 part of 2nd, 1-2-3 works great! with exact like >> 1st results. >> When i delete the 1-2-3 part of 2nd, 4-5-6 works great! with exact like >> 1st results. >> All program (1-2-3-4-5-6) works badly. >> > > From the -log_summary, you have a memory leak (many more vector creations > than destructions). Try running with -malloc_dump to debug it. Perhaps you > are creating a vector every time one of your functions is called? You should > also build --with-debugging=0 when looking at timing results. (You can keep > it in PETSC_ARCH=linux-gnu-opt.) > -- Best regards, Alexey Ryazanov ______________________________________ Nuclear Safety Institute of Russian Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Fri Aug 26 16:05:16 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 26 Aug 2011 23:05:16 +0200 Subject: [petsc-users] Petsc has generated inconsistent data! Message-ID: When solving my linear system with -ksp_type bcgs I get: 0 KSP preconditioned resid norm 1.166190378969e+01 true resid norm 1.166190378969e+01 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.658835826231e-01 true resid norm 5.658835826231e-01 ||Ae||/||Ax|| 4.852411688762e-02 [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Petsc has generated inconsistent data! [1]PETSC ERROR: Divide by zero! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message ------------------------------------ [3]PETSC ERROR: Petsc has generated inconsistent data! [3]PETSC ERROR: Divide by zero! PETSc Option Table entries: -ksp_converged_reason -ksp_monitor -ksp_monitor_true_residual -ksp_norm_type unpreconditioned -ksp_right_pc -ksp_rtol 1e-3 -ksp_type bcgs -ksp_view -log_summary -pc_type jacobi When solving the same system with GMRES all works fine. This is a simple test diffusion problem. How can I find out what the problem is? Dominik From bsmith at mcs.anl.gov Fri Aug 26 16:17:56 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 26 Aug 2011 16:17:56 -0500 Subject: [petsc-users] Petsc has generated inconsistent data! In-Reply-To: References: Message-ID: <12468DD1-3BC8-4319-B8D0-EAAF0AD3D168@mcs.anl.gov> Are you sure that is the entire error message. It should print the routine and the line number where this happens. Likely it is at do { ierr = VecDot(R,RP,&rho);CHKERRQ(ierr); /* rho <- (r,rp) */ beta = (rho/rhoold) * (alpha/omegaold); ierr = VecAXPBYPCZ(P,1.0,-omegaold*beta,beta,R,V);CHKERRQ(ierr); /* p <- r - omega * beta* v + beta * p */ ierr = KSP_PCApplyBAorAB(ksp,P,V,T);CHKERRQ(ierr); /* v <- K p */ ierr = VecDot(V,RP,&d1);CHKERRQ(ierr); if (d1 == 0.0) SETERRQ(((PetscObject)ksp)->comm,PETSC_ERR_PLIB,"Divide by zero"); alpha = rho / d1; /* a <- rho / (v,rp) */ Which means bi-cg-stab has broken down. You'll need to consult references to Bi-CG-stab to see why this might happen (it can happen while GMRES is happy). It may be KSPBCGSL can proceed past this point with a problem values for Options Database Keys: + -ksp_bcgsl_ell Number of Krylov search directions - -ksp_bcgsl_cxpol Use a convex function of the MR and OR polynomials after the BiCG step - -ksp_bcgsl_xres Threshold used to decide when to refresh computed residuals but most likely the preconditioner or matrix is bogus in some way since I think Bi-CG-stab rarely breaks down in practice. Barry On Aug 26, 2011, at 4:05 PM, Dominik Szczerba wrote: > When solving my linear system with -ksp_type bcgs I get: > > 0 KSP preconditioned resid norm 1.166190378969e+01 true resid norm > 1.166190378969e+01 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.658835826231e-01 true resid norm > 5.658835826231e-01 ||Ae||/||Ax|| 4.852411688762e-02 > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Petsc has generated inconsistent data! > [1]PETSC ERROR: Divide by zero! > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 > 13:37:48 CDT 2011 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [3]PETSC ERROR: Petsc has generated inconsistent data! > [3]PETSC ERROR: Divide by zero! > > PETSc Option Table entries: > -ksp_converged_reason > -ksp_monitor > -ksp_monitor_true_residual > -ksp_norm_type unpreconditioned > -ksp_right_pc > -ksp_rtol 1e-3 > -ksp_type bcgs > -ksp_view > -log_summary > -pc_type jacobi > > When solving the same system with GMRES all works fine. This is a > simple test diffusion problem. How can I find out what the problem is? > > Dominik From dominik at itis.ethz.ch Fri Aug 26 16:27:49 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 26 Aug 2011 23:27:49 +0200 Subject: [petsc-users] Petsc has generated inconsistent data! In-Reply-To: <12468DD1-3BC8-4319-B8D0-EAAF0AD3D168@mcs.anl.gov> References: <12468DD1-3BC8-4319-B8D0-EAAF0AD3D168@mcs.anl.gov> Message-ID: Later in the message he only requested that I use "-ksp_norm_type unpreconditioned". So I did, and the error comes back, now fully documented below. As I wrote, it works fine with gmres, and the problem is very simple, diagonal dominant steady state diffusion. Any hints are highly appreciated. Dominik #PETSc Option Table entries: -ksp_converged_reason -ksp_converged_use_initial_residual_norm -ksp_monitor_true_residual -ksp_norm_type unpreconditioned -ksp_rtol 1e-3 -ksp_type bcgs -ksp_view -log_summary -pc_type jacobi #End of PETSc Option Table entries 0 KSP preconditioned resid norm 1.166190378969e+01 true resid norm 1.166190378969e+01 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.658835826231e-01 true resid norm 5.658835826231e-01 ||Ae||/||Ax|| 4.852411688762e-02 [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Petsc has generated inconsistent data! [0]PETSC ERROR: Divide by zero! [0]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: --------------------- Error Message ------------------------------------ [2]PETSC ERROR: Petsc has generated inconsistent data! [2]PETSC ERROR: Divide by zero! [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [2]PETSC ERROR: See docs/changes/index.html for recent updates. [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [2]PETSC ERROR: See docs/index.html for manual pages. [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: /home/dsz/build/framework-debug/trunk/bin/cd3t10mpi on a linux-gnu named tharsis by dsz Fri Aug 26 23:23:47 2011 [2]PETSC ERROR: Libraries linked from /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib [2]PETSC ERROR: Configure run at Mon Jul 25 14:20:10 2011 [2]PETSC ERROR: Configure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=CD3T10::SaveSolution() [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /home/dsz/build/framework-debug/trunk/bin/cd3t10mpi on a linux-gnu named tharsis by dsz Fri Aug 26 23:23:47 2011 [0]PETSC ERROR: Libraries linked from /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib [0]PETSC ERROR: Configure run at Mon Jul 25 14:20:10 2011 [0]PETSC ERROR: Configure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: KSPSolve_BCGS() line 75 in src/ksp/ksp/impls/bcgs/[1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Petsc has generated inconsistent data! [1]PETSC ERROR: Divide by zero! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: /home/dsz/build/framework-debug/trunk/bin/cd3t10mpi on a linux-gnu named tharsis by dsz Fri Aug 26 23:23:47 2011 [1]PETSC ERROR: Libraries linked from /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib [1]PETSC ERROR: Configure run at Mon Jul 25 14:20:10 2011 [1]PETSC ERROR: Configure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: KSPSolve_BCGS() line 75 in src/ksp/ksp/impls/bcgs/bcgs.c [2]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c [2]PETSC ERROR: User provided function() line 1215 in "unknowndirectory/"/home/dsz/src/framework/trunk/solve/PetscSolver.cxx [3]PETSC ERROR: --------------------- Error Message ------------------------------------ [3]PETSC ERROR: Petsc has generated inconsistent data! [3]PETSC ERROR: Divide by zero! [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [3]PETSC ERROR: See docs/changes/index.html for recent updates. [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [3]PETSC ERROR: See docs/index.html for manual pages. [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: /home/dsz/build/framework-debug/trunk/bin/cd3t10mpi on a linux-gnu named tharsis by dsz Fri Aug 26 23:23:47 2011 [3]PETSC ERROR: Libraries linked from /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib [3]PETSC ERROR: Configure run at Mon Jul 25 14:20:10 2011 [3]PETSC ERROR: Configure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=bcgs.c [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: User provided function() line 1215 in "unknowndirectory/"/home/dsz/src/framework/trunk/solve/PetscSolver.cxx 1 --download-mpich=1 --download-hypre=1 --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: KSPSolve_BCGS() line 75 in src/ksp/ksp/impls/bcgs/bcgs.c [1]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: User provided function() line 1215 in "unknowndirectory/"/home/dsz/src/framework/trunk/solve/PetscSolver.cxx 1 --download-mpich=1 --download-hypre=1 --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: KSPSolve_BCGS() line 75 in src/ksp/ksp/impls/bcgs/bcgs.c [3]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c [3]PETSC ERROR: User provided function() line 1215 in "unknowndirectory/"/home/dsz/src/framework/trunk/solve/PetscSolver.cxx PetscSolver::Finalize() PetscSolver::FinalizePetsc() On Fri, Aug 26, 2011 at 11:17 PM, Barry Smith wrote: > > ?Are you sure that is the entire error message. It should print the routine and the line number where this happens. > > ? Likely it is at > > ?do { > ? ?ierr = VecDot(R,RP,&rho);CHKERRQ(ierr); ? ? ? /* ? rho <- (r,rp) ? ? ?*/ > ? ?beta = (rho/rhoold) * (alpha/omegaold); > ? ?ierr = VecAXPBYPCZ(P,1.0,-omegaold*beta,beta,R,V);CHKERRQ(ierr); ?/* p <- r - omega * beta* v + beta * p */ > ? ?ierr = KSP_PCApplyBAorAB(ksp,P,V,T);CHKERRQ(ierr); ?/* ? v <- K p ? ? ? ? ? */ > ? ?ierr = VecDot(V,RP,&d1);CHKERRQ(ierr); > ? ?if (d1 == 0.0) SETERRQ(((PetscObject)ksp)->comm,PETSC_ERR_PLIB,"Divide by zero"); > ? ?alpha = rho / d1; ? ? ? ? ? ? ? ? /* ? a <- rho / (v,rp) ?*/ > > ?Which means bi-cg-stab has broken down. You'll need to consult references to Bi-CG-stab to see why this might happen (it can happen while GMRES is happy). It may be KSPBCGSL can proceed past this point with a problem values for > ? Options Database Keys: > + ?-ksp_bcgsl_ell Number of Krylov search directions > - ?-ksp_bcgsl_cxpol Use a convex function of the MR and OR polynomials after the BiCG step > - ?-ksp_bcgsl_xres Threshold used to decide when to refresh computed residuals > > but most likely the preconditioner or matrix is bogus in some way since I think Bi-CG-stab rarely breaks down in practice. > > > ?Barry > > > > On Aug 26, 2011, at 4:05 PM, Dominik Szczerba wrote: > >> When solving my linear system with -ksp_type bcgs I get: >> >> ?0 KSP preconditioned resid norm 1.166190378969e+01 true resid norm >> 1.166190378969e+01 ||Ae||/||Ax|| 1.000000000000e+00 >> ?1 KSP preconditioned resid norm 5.658835826231e-01 true resid norm >> 5.658835826231e-01 ||Ae||/||Ax|| 4.852411688762e-02 >> [1]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [1]PETSC ERROR: Petsc has generated inconsistent data! >> [1]PETSC ERROR: Divide by zero! >> [1]PETSC ERROR: >> ------------------------------------------------------------------------ >> [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >> 13:37:48 CDT 2011 >> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >> [1]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [3]PETSC ERROR: Petsc has generated inconsistent data! >> [3]PETSC ERROR: Divide by zero! >> >> PETSc Option Table entries: >> -ksp_converged_reason >> -ksp_monitor >> -ksp_monitor_true_residual >> -ksp_norm_type unpreconditioned >> -ksp_right_pc >> -ksp_rtol 1e-3 >> -ksp_type bcgs >> -ksp_view >> -log_summary >> -pc_type jacobi >> >> When solving the same system with GMRES all works fine. This is a >> simple test diffusion problem. How can I find out what the problem is? >> >> Dominik > > From ram at ibrae.ac.ru Fri Aug 26 16:29:59 2011 From: ram at ibrae.ac.ru (=?UTF-8?B?0JDQu9C10LrRgdC10Lkg0KDRj9C30LDQvdC+0LI=?=) Date: Sat, 27 Aug 2011 01:29:59 +0400 Subject: [petsc-users] performance issue solving multiple linear systems of the same size with the different preconditioning methods In-Reply-To: References: Message-ID: I have also checked KSPSolve behavior in my other PETSc programs and found the same memory lack 27 ??????? 2011 ?. 0:57 ???????????? ??????? ??????? ???????: > > Thank you for your response! > > I have the memory leak in both my programs. But I don't create plenty of > vectors. > > My code looks like: > > ***INIT_ALL*** > PetscLogStageRegister("Iteration :", &StageNum1); > PetscLogStagePush(StageNum1); > KSPSolve(dKSP, dvec_origRHS, dvec_Solution); > PetscLogStagePop(); > ***DESTROY_ALL*** > > > And when I comment (or delete) KSPSolve, the log_summary output is: > > ===================================================== > Memory usage is given in bytes: > > Creations Destructions Memory Descendants' Mem Object Type > Reports information only for process 0. > > --- Event Stage 0: Main Stage > 1 1 729472 0 Application Order > 1 1 225452 0 Distributed array > 8 8 1533424 0 Vec > 3 3 2604 0 Vec Scatter > 8 8 613852 0 Index Set > 1 1 221304 0 IS L to G Mapping > 3 3 16603440 0 Matrix > 1 1 832 0 Krylov Solver > 1 1 688 0 Preconditioner > 1 1 448 0 PetscRandom > > --- Event Stage 1: Iteration : > ===================================================== > > When I run the code with KSPSolve instruction, it gives me: > > ===================================================== > Memory usage is given in bytes: > Creations Destructions Memory Descendants' Mem Object Type > Reports information only for process 0. > > --- Event Stage 0: Main Stage > 1 0 0 0 Application Order > 1 0 0 0 Distributed array > 8 17 4963592 0 Vec > 3 2 1736 0 Vec Scatter > 8 12 1425932 0 Index Set > 1 0 0 0 IS L to G Mapping > 3 5 50158132 0 Matrix > 1 2 1664 0 Krylov Solver > 1 2 1440 0 Preconditioner > 1 1 448 0 PetscRandom > 0 1 544 0 Viewer > > --- Event Stage 1: Iteration : > 355 173 64692312 0 Vec > 1 0 0 0 Vec Scatter > 6 2 1024 0 Index Set > 2 0 0 0 Matrix > 1 0 0 0 Krylov Solver > 1 0 0 0 Preconditioner > 2 1 544 0 Viewer > ===================================================== > > > > > 2011/8/25 Jed Brown > >> On Tue, Aug 23, 2011 at 02:37, ??????? ??????? wrote: >> >>> When i delete the 4-5-6 part of 2nd, 1-2-3 works great! with exact like >>> 1st results. >>> When i delete the 1-2-3 part of 2nd, 4-5-6 works great! with exact like >>> 1st results. >>> All program (1-2-3-4-5-6) works badly. >>> >> >> From the -log_summary, you have a memory leak (many more vector creations >> than destructions). Try running with -malloc_dump to debug it. Perhaps you >> are creating a vector every time one of your functions is called? You should >> also build --with-debugging=0 when looking at timing results. (You can keep >> it in PETSC_ARCH=linux-gnu-opt.) >> > > > > -- > Best regards, > Alexey Ryazanov > ______________________________________ > Nuclear Safety Institute of Russian Academy of Sciences > > > -- Best regards, Alexey Ryazanov ______________________________________ Nuclear Safety Institute of Russian Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Aug 26 16:31:48 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 26 Aug 2011 16:31:48 -0500 Subject: [petsc-users] Petsc has generated inconsistent data! In-Reply-To: References: <12468DD1-3BC8-4319-B8D0-EAAF0AD3D168@mcs.anl.gov> Message-ID: First run with valgrind http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind to make sure it does not have some "code bug cause". Do you get the same message on one process. My previous message still holds, if it is not a "code bug" then bi-CG-stab is just breaking down on that matrix and preconditioner combination. Barry On Aug 26, 2011, at 4:27 PM, Dominik Szczerba wrote: > Later in the message he only requested that I use "-ksp_norm_type > unpreconditioned". So I did, and the error comes back, now fully > documented below. As I wrote, it works fine with gmres, and the > problem is very simple, diagonal dominant steady state diffusion. > > Any hints are highly appreciated. > > Dominik > > #PETSc Option Table entries: > -ksp_converged_reason > -ksp_converged_use_initial_residual_norm > -ksp_monitor_true_residual > -ksp_norm_type unpreconditioned > -ksp_rtol 1e-3 > -ksp_type bcgs > -ksp_view > -log_summary > -pc_type jacobi > #End of PETSc Option Table entries > 0 KSP preconditioned resid norm 1.166190378969e+01 true resid norm > 1.166190378969e+01 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.658835826231e-01 true resid norm > 5.658835826231e-01 ||Ae||/||Ax|| 4.852411688762e-02 > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Petsc has generated inconsistent data! > [0]PETSC ERROR: Divide by zero! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [2]PETSC ERROR: Petsc has generated inconsistent data! > [2]PETSC ERROR: Divide by zero! > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 > 13:37:48 CDT 2011 > [2]PETSC ERROR: See docs/changes/index.html for recent updates. > [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [2]PETSC ERROR: See docs/index.html for manual pages. > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: /home/dsz/build/framework-debug/trunk/bin/cd3t10mpi on > a linux-gnu named tharsis by dsz Fri Aug 26 23:23:47 2011 > [2]PETSC ERROR: Libraries linked from > /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib > [2]PETSC ERROR: Configure run at Mon Jul 25 14:20:10 2011 > [2]PETSC ERROR: Configure options > PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug > --download-f-blas-lapack=CD3T10::SaveSolution() > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 > 13:37:48 CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: /home/dsz/build/framework-debug/trunk/bin/cd3t10mpi on > a linux-gnu named tharsis by dsz Fri Aug 26 23:23:47 2011 > [0]PETSC ERROR: Libraries linked from > /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib > [0]PETSC ERROR: Configure run at Mon Jul 25 14:20:10 2011 > [0]PETSC ERROR: Configure options > PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug > --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 > --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: KSPSolve_BCGS() line 75 in > src/ksp/ksp/impls/bcgs/[1]PETSC ERROR: --------------------- Error > Message ------------------------------------ > [1]PETSC ERROR: Petsc has generated inconsistent data! > [1]PETSC ERROR: Divide by zero! > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 > 13:37:48 CDT 2011 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: /home/dsz/build/framework-debug/trunk/bin/cd3t10mpi on > a linux-gnu named tharsis by dsz Fri Aug 26 23:23:47 2011 > [1]PETSC ERROR: Libraries linked from > /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib > [1]PETSC ERROR: Configure run at Mon Jul 25 14:20:10 2011 > [1]PETSC ERROR: Configure options > PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug > --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 > --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: KSPSolve_BCGS() line 75 in src/ksp/ksp/impls/bcgs/bcgs.c > [2]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c > [2]PETSC ERROR: User provided function() line 1215 in > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/PetscSolver.cxx > [3]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [3]PETSC ERROR: Petsc has generated inconsistent data! > [3]PETSC ERROR: Divide by zero! > [3]PETSC ERROR: > ------------------------------------------------------------------------ > [3]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 > 13:37:48 CDT 2011 > [3]PETSC ERROR: See docs/changes/index.html for recent updates. > [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [3]PETSC ERROR: See docs/index.html for manual pages. > [3]PETSC ERROR: > ------------------------------------------------------------------------ > [3]PETSC ERROR: /home/dsz/build/framework-debug/trunk/bin/cd3t10mpi on > a linux-gnu named tharsis by dsz Fri Aug 26 23:23:47 2011 > [3]PETSC ERROR: Libraries linked from > /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib > [3]PETSC ERROR: Configure run at Mon Jul 25 14:20:10 2011 > [3]PETSC ERROR: Configure options > PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug > --download-f-blas-lapack=bcgs.c > [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: User provided function() line 1215 in > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/PetscSolver.cxx > 1 --download-mpich=1 --download-hypre=1 --with-parmetis=1 > --download-parmetis=1 --with-x=0 --with-debugging=1 > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: KSPSolve_BCGS() line 75 in src/ksp/ksp/impls/bcgs/bcgs.c > [1]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: User provided function() line 1215 in > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/PetscSolver.cxx > 1 --download-mpich=1 --download-hypre=1 --with-parmetis=1 > --download-parmetis=1 --with-x=0 --with-debugging=1 > [3]PETSC ERROR: > ------------------------------------------------------------------------ > [3]PETSC ERROR: KSPSolve_BCGS() line 75 in src/ksp/ksp/impls/bcgs/bcgs.c > [3]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c > [3]PETSC ERROR: User provided function() line 1215 in > "unknowndirectory/"/home/dsz/src/framework/trunk/solve/PetscSolver.cxx > PetscSolver::Finalize() > PetscSolver::FinalizePetsc() > > > On Fri, Aug 26, 2011 at 11:17 PM, Barry Smith wrote: >> >> Are you sure that is the entire error message. It should print the routine and the line number where this happens. >> >> Likely it is at >> >> do { >> ierr = VecDot(R,RP,&rho);CHKERRQ(ierr); /* rho <- (r,rp) */ >> beta = (rho/rhoold) * (alpha/omegaold); >> ierr = VecAXPBYPCZ(P,1.0,-omegaold*beta,beta,R,V);CHKERRQ(ierr); /* p <- r - omega * beta* v + beta * p */ >> ierr = KSP_PCApplyBAorAB(ksp,P,V,T);CHKERRQ(ierr); /* v <- K p */ >> ierr = VecDot(V,RP,&d1);CHKERRQ(ierr); >> if (d1 == 0.0) SETERRQ(((PetscObject)ksp)->comm,PETSC_ERR_PLIB,"Divide by zero"); >> alpha = rho / d1; /* a <- rho / (v,rp) */ >> >> Which means bi-cg-stab has broken down. You'll need to consult references to Bi-CG-stab to see why this might happen (it can happen while GMRES is happy). It may be KSPBCGSL can proceed past this point with a problem values for >> Options Database Keys: >> + -ksp_bcgsl_ell Number of Krylov search directions >> - -ksp_bcgsl_cxpol Use a convex function of the MR and OR polynomials after the BiCG step >> - -ksp_bcgsl_xres Threshold used to decide when to refresh computed residuals >> >> but most likely the preconditioner or matrix is bogus in some way since I think Bi-CG-stab rarely breaks down in practice. >> >> >> Barry >> >> >> >> On Aug 26, 2011, at 4:05 PM, Dominik Szczerba wrote: >> >>> When solving my linear system with -ksp_type bcgs I get: >>> >>> 0 KSP preconditioned resid norm 1.166190378969e+01 true resid norm >>> 1.166190378969e+01 ||Ae||/||Ax|| 1.000000000000e+00 >>> 1 KSP preconditioned resid norm 5.658835826231e-01 true resid norm >>> 5.658835826231e-01 ||Ae||/||Ax|| 4.852411688762e-02 >>> [1]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [1]PETSC ERROR: Petsc has generated inconsistent data! >>> [1]PETSC ERROR: Divide by zero! >>> [1]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >>> 13:37:48 CDT 2011 >>> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [1]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [3]PETSC ERROR: Petsc has generated inconsistent data! >>> [3]PETSC ERROR: Divide by zero! >>> >>> PETSc Option Table entries: >>> -ksp_converged_reason >>> -ksp_monitor >>> -ksp_monitor_true_residual >>> -ksp_norm_type unpreconditioned >>> -ksp_right_pc >>> -ksp_rtol 1e-3 >>> -ksp_type bcgs >>> -ksp_view >>> -log_summary >>> -pc_type jacobi >>> >>> When solving the same system with GMRES all works fine. This is a >>> simple test diffusion problem. How can I find out what the problem is? >>> >>> Dominik >> >> From likunt at andrew.cmu.edu Fri Aug 26 16:41:38 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Fri, 26 Aug 2011 17:41:38 -0400 Subject: [petsc-users] about MPI Message-ID: Hello, I try to run my code with MPI, but i got the message: 'mpiexec: command not found'. I installed mpi and tested it successfully on examples with 'make all test'. I also tried to specify the path for mpi with ./config/configure.py --with-mpi-dir=/home/likunt/petsc-3.1-p8/externalpackages/mpich2-1.0.8, but it shows 'unable to configure with given options'. Any suggestions on this? Thanks, Likun From dominik at itis.ethz.ch Fri Aug 26 16:51:38 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 26 Aug 2011 23:51:38 +0200 Subject: [petsc-users] Petsc has generated inconsistent data! In-Reply-To: References: <12468DD1-3BC8-4319-B8D0-EAAF0AD3D168@mcs.anl.gov> Message-ID: Valgrind excerpts below, none seems related, and both are there also with gmres. I just cant believe bicgstab would quit on such a trivial problem... Regards, Dominik ==10423== Syscall param writev(vector[...]) points to uninitialised byte(s) ==10423== at 0x6B5E789: writev (writev.c:56) ==10423== by 0x515658: MPIDU_Sock_writev (sock_immed.i:610) ==10423== by 0x517943: MPIDI_CH3_iSendv (ch3_isendv.c:84) ==10423== by 0x4FB756: MPIDI_CH3_EagerContigIsend (ch3u_eager.c:509) ==10423== by 0x4FD6E4: MPID_Isend (mpid_isend.c:118) ==10423== by 0x4E502A: MPIC_Isend (helper_fns.c:210) ==10423== by 0xED701D: MPIR_Alltoall (alltoall.c:420) ==10423== by 0xED7880: PMPI_Alltoall (alltoall.c:685) ==10423== by 0xE4BC95: SetUp__ (setup.c:122) ==10423== by 0xE4C4E4: PartitionSmallGraph__ (weird.c:39) ==10423== by 0xE497D0: ParMETIS_V3_PartKway (kmetis.c:131) ==10423== by 0x71B774: MatPartitioningApply_Parmetis (pmetis.c:97) ==10423== by 0x717D70: MatPartitioningApply (partition.c:236) ==10423== by 0x5287D7: PetscSolver::LoadMesh(std::string const&) (PetscSolver.cxx:676) ==10423== by 0x4C6157: CD3T10_USER::ProcessInputFile() (cd3t10mpi_main.cxx:321) ==10423== by 0x4C3B57: main (cd3t10mpi_main.cxx:568) ==10423== Address 0x71ce9b4 is 4 bytes inside a block of size 72 alloc'd ==10423== at 0x4C28FAC: malloc (vg_replace_malloc.c:236) ==10423== by 0xE5C792: GKmalloc__ (util.c:151) ==10423== by 0xE57C09: PreAllocateMemory__ (memory.c:38) ==10423== by 0xE496B3: ParMETIS_V3_PartKway (kmetis.c:116) ==10423== by 0x71B774: MatPartitioningApply_Parmetis (pmetis.c:97) ==10423== by 0x717D70: MatPartitioningApply (partition.c:236) ==10423== by 0x5287D7: PetscSolver::LoadMesh(std::string const&) (PetscSolver.cxx:676) ==10423== by 0x4C6157: CD3T10_USER::ProcessInputFile() (cd3t10mpi_main.cxx:321) ==10423== by 0x4C3B57: main (cd3t10mpi_main.cxx:568) ==10423== ==10423== Conditional jump or move depends on uninitialised value(s) ==10423== at 0x55B6510: inflateReset2 (in /lib/x86_64-linux-gnu/libz.so.1.2.3.4) ==10423== by 0x55B6605: inflateInit2_ (in /lib/x86_64-linux-gnu/libz.so.1.2.3.4) ==10423== by 0x5308C13: H5Z_filter_deflate (H5Zdeflate.c:110) ==10423== by 0x5308170: H5Z_pipeline (H5Z.c:1103) ==10423== by 0x518BB69: H5D_chunk_lock (H5Dchunk.c:2758) ==10423== by 0x518CB20: H5D_chunk_read (H5Dchunk.c:1728) ==10423== by 0x519BDDA: H5D_read (H5Dio.c:447) ==10423== by 0x519C248: H5Dread (H5Dio.c:173) ==10423== by 0x4CEB99: HDF5::HDF5Reader::readData(std::string const&) (HDF5Reader.cxx:634) ==10423== by 0x4CE305: HDF5::HDF5Reader::read(std::vector >&, std::string const&) (HDF5Reader.cxx:527) ==10423== by 0x4C71A3: CD3T10_USER::SetupConstraints() (cd3t10mpi_main.cxx:404) ==10423== by 0x4BA02A: CD3T10::Solve() (CD3T10mpi.cxx:658) ==10423== by 0x4C3BE1: main (cd3t10mpi_main.cxx:590) ==10423== On Fri, Aug 26, 2011 at 11:31 PM, Barry Smith wrote: > > ?First run with valgrind http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind to make sure it does not have some "code bug cause". > > ? Do you get the same message on one process. > > ? My previous message still holds, if it is not a "code bug" then bi-CG-stab is just breaking down on that matrix and preconditioner combination. > > ? Barry > > On Aug 26, 2011, at 4:27 PM, Dominik Szczerba wrote: > >> Later in the message he only requested that I use "-ksp_norm_type >> unpreconditioned". So I did, and the error comes back, now fully >> documented below. As I wrote, it works fine with gmres, and the >> problem is very simple, diagonal dominant steady state diffusion. >> >> Any hints are highly appreciated. >> >> Dominik >> >> #PETSc Option Table entries: >> -ksp_converged_reason >> -ksp_converged_use_initial_residual_norm >> -ksp_monitor_true_residual >> -ksp_norm_type unpreconditioned >> -ksp_rtol 1e-3 >> -ksp_type bcgs >> -ksp_view >> -log_summary >> -pc_type jacobi >> #End of PETSc Option Table entries >> ?0 KSP preconditioned resid norm 1.166190378969e+01 true resid norm >> 1.166190378969e+01 ||Ae||/||Ax|| 1.000000000000e+00 >> ?1 KSP preconditioned resid norm 5.658835826231e-01 true resid norm >> 5.658835826231e-01 ||Ae||/||Ax|| 4.852411688762e-02 >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Petsc has generated inconsistent data! >> [0]PETSC ERROR: Divide by zero! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [2]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [2]PETSC ERROR: Petsc has generated inconsistent data! >> [2]PETSC ERROR: Divide by zero! >> [2]PETSC ERROR: >> ------------------------------------------------------------------------ >> [2]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >> 13:37:48 CDT 2011 >> [2]PETSC ERROR: See docs/changes/index.html for recent updates. >> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [2]PETSC ERROR: See docs/index.html for manual pages. >> [2]PETSC ERROR: >> ------------------------------------------------------------------------ >> [2]PETSC ERROR: /home/dsz/build/framework-debug/trunk/bin/cd3t10mpi on >> a linux-gnu named tharsis by dsz Fri Aug 26 23:23:47 2011 >> [2]PETSC ERROR: Libraries linked from >> /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib >> [2]PETSC ERROR: Configure run at Mon Jul 25 14:20:10 2011 >> [2]PETSC ERROR: Configure options >> PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug >> --download-f-blas-lapack=CD3T10::SaveSolution() >> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >> 13:37:48 CDT 2011 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: /home/dsz/build/framework-debug/trunk/bin/cd3t10mpi on >> a linux-gnu named tharsis by dsz Fri Aug 26 23:23:47 2011 >> [0]PETSC ERROR: Libraries linked from >> /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib >> [0]PETSC ERROR: Configure run at Mon Jul 25 14:20:10 2011 >> [0]PETSC ERROR: Configure options >> PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug >> --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 >> --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: KSPSolve_BCGS() line 75 in >> src/ksp/ksp/impls/bcgs/[1]PETSC ERROR: --------------------- Error >> Message ------------------------------------ >> [1]PETSC ERROR: Petsc has generated inconsistent data! >> [1]PETSC ERROR: Divide by zero! >> [1]PETSC ERROR: >> ------------------------------------------------------------------------ >> [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >> 13:37:48 CDT 2011 >> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [1]PETSC ERROR: See docs/index.html for manual pages. >> [1]PETSC ERROR: >> ------------------------------------------------------------------------ >> [1]PETSC ERROR: /home/dsz/build/framework-debug/trunk/bin/cd3t10mpi on >> a linux-gnu named tharsis by dsz Fri Aug 26 23:23:47 2011 >> [1]PETSC ERROR: Libraries linked from >> /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib >> [1]PETSC ERROR: Configure run at Mon Jul 25 14:20:10 2011 >> [1]PETSC ERROR: Configure options >> PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug >> --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 >> --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 >> [2]PETSC ERROR: >> ------------------------------------------------------------------------ >> [2]PETSC ERROR: KSPSolve_BCGS() line 75 in src/ksp/ksp/impls/bcgs/bcgs.c >> [2]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c >> [2]PETSC ERROR: User provided function() line 1215 in >> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/PetscSolver.cxx >> [3]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [3]PETSC ERROR: Petsc has generated inconsistent data! >> [3]PETSC ERROR: Divide by zero! >> [3]PETSC ERROR: >> ------------------------------------------------------------------------ >> [3]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >> 13:37:48 CDT 2011 >> [3]PETSC ERROR: See docs/changes/index.html for recent updates. >> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [3]PETSC ERROR: See docs/index.html for manual pages. >> [3]PETSC ERROR: >> ------------------------------------------------------------------------ >> [3]PETSC ERROR: /home/dsz/build/framework-debug/trunk/bin/cd3t10mpi on >> a linux-gnu named tharsis by dsz Fri Aug 26 23:23:47 2011 >> [3]PETSC ERROR: Libraries linked from >> /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib >> [3]PETSC ERROR: Configure run at Mon Jul 25 14:20:10 2011 >> [3]PETSC ERROR: Configure options >> PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug >> --download-f-blas-lapack=bcgs.c >> [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: User provided function() line 1215 in >> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/PetscSolver.cxx >> 1 --download-mpich=1 --download-hypre=1 --with-parmetis=1 >> --download-parmetis=1 --with-x=0 --with-debugging=1 >> [1]PETSC ERROR: >> ------------------------------------------------------------------------ >> [1]PETSC ERROR: KSPSolve_BCGS() line 75 in src/ksp/ksp/impls/bcgs/bcgs.c >> [1]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c >> [1]PETSC ERROR: User provided function() line 1215 in >> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/PetscSolver.cxx >> 1 --download-mpich=1 --download-hypre=1 --with-parmetis=1 >> --download-parmetis=1 --with-x=0 --with-debugging=1 >> [3]PETSC ERROR: >> ------------------------------------------------------------------------ >> [3]PETSC ERROR: KSPSolve_BCGS() line 75 in src/ksp/ksp/impls/bcgs/bcgs.c >> [3]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c >> [3]PETSC ERROR: User provided function() line 1215 in >> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/PetscSolver.cxx >> PetscSolver::Finalize() >> PetscSolver::FinalizePetsc() >> >> >> On Fri, Aug 26, 2011 at 11:17 PM, Barry Smith wrote: >>> >>> ?Are you sure that is the entire error message. It should print the routine and the line number where this happens. >>> >>> ? Likely it is at >>> >>> ?do { >>> ? ?ierr = VecDot(R,RP,&rho);CHKERRQ(ierr); ? ? ? /* ? rho <- (r,rp) ? ? ?*/ >>> ? ?beta = (rho/rhoold) * (alpha/omegaold); >>> ? ?ierr = VecAXPBYPCZ(P,1.0,-omegaold*beta,beta,R,V);CHKERRQ(ierr); ?/* p <- r - omega * beta* v + beta * p */ >>> ? ?ierr = KSP_PCApplyBAorAB(ksp,P,V,T);CHKERRQ(ierr); ?/* ? v <- K p ? ? ? ? ? */ >>> ? ?ierr = VecDot(V,RP,&d1);CHKERRQ(ierr); >>> ? ?if (d1 == 0.0) SETERRQ(((PetscObject)ksp)->comm,PETSC_ERR_PLIB,"Divide by zero"); >>> ? ?alpha = rho / d1; ? ? ? ? ? ? ? ? /* ? a <- rho / (v,rp) ?*/ >>> >>> ?Which means bi-cg-stab has broken down. You'll need to consult references to Bi-CG-stab to see why this might happen (it can happen while GMRES is happy). It may be KSPBCGSL can proceed past this point with a problem values for >>> ? Options Database Keys: >>> + ?-ksp_bcgsl_ell Number of Krylov search directions >>> - ?-ksp_bcgsl_cxpol Use a convex function of the MR and OR polynomials after the BiCG step >>> - ?-ksp_bcgsl_xres Threshold used to decide when to refresh computed residuals >>> >>> but most likely the preconditioner or matrix is bogus in some way since I think Bi-CG-stab rarely breaks down in practice. >>> >>> >>> ?Barry >>> >>> >>> >>> On Aug 26, 2011, at 4:05 PM, Dominik Szczerba wrote: >>> >>>> When solving my linear system with -ksp_type bcgs I get: >>>> >>>> ?0 KSP preconditioned resid norm 1.166190378969e+01 true resid norm >>>> 1.166190378969e+01 ||Ae||/||Ax|| 1.000000000000e+00 >>>> ?1 KSP preconditioned resid norm 5.658835826231e-01 true resid norm >>>> 5.658835826231e-01 ||Ae||/||Ax|| 4.852411688762e-02 >>>> [1]PETSC ERROR: --------------------- Error Message >>>> ------------------------------------ >>>> [1]PETSC ERROR: Petsc has generated inconsistent data! >>>> [1]PETSC ERROR: Divide by zero! >>>> [1]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >>>> 13:37:48 CDT 2011 >>>> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [1]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message >>>> ------------------------------------ >>>> [3]PETSC ERROR: Petsc has generated inconsistent data! >>>> [3]PETSC ERROR: Divide by zero! >>>> >>>> PETSc Option Table entries: >>>> -ksp_converged_reason >>>> -ksp_monitor >>>> -ksp_monitor_true_residual >>>> -ksp_norm_type unpreconditioned >>>> -ksp_right_pc >>>> -ksp_rtol 1e-3 >>>> -ksp_type bcgs >>>> -ksp_view >>>> -log_summary >>>> -pc_type jacobi >>>> >>>> When solving the same system with GMRES all works fine. This is a >>>> simple test diffusion problem. How can I find out what the problem is? >>>> >>>> Dominik >>> >>> > > From bsmith at mcs.anl.gov Fri Aug 26 16:56:27 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 26 Aug 2011 16:56:27 -0500 Subject: [petsc-users] Petsc has generated inconsistent data! In-Reply-To: References: <12468DD1-3BC8-4319-B8D0-EAAF0AD3D168@mcs.anl.gov> Message-ID: <9D3BE4EA-8D70-4899-A245-9396633E5073@mcs.anl.gov> On Aug 26, 2011, at 4:51 PM, Dominik Szczerba wrote: > Valgrind excerpts below, none seems related, and both are there also with gmres. > > I just cant believe bicgstab would quit on such a trivial problem... The bicgstab is used extensively so it is unlikely it has a trivial bug in it or it would have come up before. Fortunately it stops during the first iteration so you can just run the program with the option -start_in_debugger then use cont in the debugger then when it errors out you can look at the numerical values and use VecView(vector,0) to look at the various vectors (run on one process, of course,) and look at the matrix to see why it is breaking down. Barry > > Regards, > Dominik > > ==10423== Syscall param writev(vector[...]) points to uninitialised byte(s) > ==10423== at 0x6B5E789: writev (writev.c:56) > ==10423== by 0x515658: MPIDU_Sock_writev (sock_immed.i:610) > ==10423== by 0x517943: MPIDI_CH3_iSendv (ch3_isendv.c:84) > ==10423== by 0x4FB756: MPIDI_CH3_EagerContigIsend (ch3u_eager.c:509) > ==10423== by 0x4FD6E4: MPID_Isend (mpid_isend.c:118) > ==10423== by 0x4E502A: MPIC_Isend (helper_fns.c:210) > ==10423== by 0xED701D: MPIR_Alltoall (alltoall.c:420) > ==10423== by 0xED7880: PMPI_Alltoall (alltoall.c:685) > ==10423== by 0xE4BC95: SetUp__ (setup.c:122) > ==10423== by 0xE4C4E4: PartitionSmallGraph__ (weird.c:39) > ==10423== by 0xE497D0: ParMETIS_V3_PartKway (kmetis.c:131) > ==10423== by 0x71B774: MatPartitioningApply_Parmetis (pmetis.c:97) > ==10423== by 0x717D70: MatPartitioningApply (partition.c:236) > ==10423== by 0x5287D7: PetscSolver::LoadMesh(std::string const&) > (PetscSolver.cxx:676) > ==10423== by 0x4C6157: CD3T10_USER::ProcessInputFile() > (cd3t10mpi_main.cxx:321) > ==10423== by 0x4C3B57: main (cd3t10mpi_main.cxx:568) > ==10423== Address 0x71ce9b4 is 4 bytes inside a block of size 72 alloc'd > ==10423== at 0x4C28FAC: malloc (vg_replace_malloc.c:236) > ==10423== by 0xE5C792: GKmalloc__ (util.c:151) > ==10423== by 0xE57C09: PreAllocateMemory__ (memory.c:38) > ==10423== by 0xE496B3: ParMETIS_V3_PartKway (kmetis.c:116) > ==10423== by 0x71B774: MatPartitioningApply_Parmetis (pmetis.c:97) > ==10423== by 0x717D70: MatPartitioningApply (partition.c:236) > ==10423== by 0x5287D7: PetscSolver::LoadMesh(std::string const&) > (PetscSolver.cxx:676) > ==10423== by 0x4C6157: CD3T10_USER::ProcessInputFile() > (cd3t10mpi_main.cxx:321) > ==10423== by 0x4C3B57: main (cd3t10mpi_main.cxx:568) > ==10423== > ==10423== Conditional jump or move depends on uninitialised value(s) > ==10423== at 0x55B6510: inflateReset2 (in > /lib/x86_64-linux-gnu/libz.so.1.2.3.4) > ==10423== by 0x55B6605: inflateInit2_ (in > /lib/x86_64-linux-gnu/libz.so.1.2.3.4) > ==10423== by 0x5308C13: H5Z_filter_deflate (H5Zdeflate.c:110) > ==10423== by 0x5308170: H5Z_pipeline (H5Z.c:1103) > ==10423== by 0x518BB69: H5D_chunk_lock (H5Dchunk.c:2758) > ==10423== by 0x518CB20: H5D_chunk_read (H5Dchunk.c:1728) > ==10423== by 0x519BDDA: H5D_read (H5Dio.c:447) > ==10423== by 0x519C248: H5Dread (H5Dio.c:173) > ==10423== by 0x4CEB99: HDF5::HDF5Reader::readData(std::string > const&) (HDF5Reader.cxx:634) > ==10423== by 0x4CE305: HDF5::HDF5Reader::read(std::vector std::allocator >&, std::string const&) (HDF5Reader.cxx:527) > ==10423== by 0x4C71A3: CD3T10_USER::SetupConstraints() > (cd3t10mpi_main.cxx:404) > ==10423== by 0x4BA02A: CD3T10::Solve() (CD3T10mpi.cxx:658) > ==10423== by 0x4C3BE1: main (cd3t10mpi_main.cxx:590) > ==10423== > > On Fri, Aug 26, 2011 at 11:31 PM, Barry Smith wrote: >> >> First run with valgrind http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind to make sure it does not have some "code bug cause". >> >> Do you get the same message on one process. >> >> My previous message still holds, if it is not a "code bug" then bi-CG-stab is just breaking down on that matrix and preconditioner combination. >> >> Barry >> >> On Aug 26, 2011, at 4:27 PM, Dominik Szczerba wrote: >> >>> Later in the message he only requested that I use "-ksp_norm_type >>> unpreconditioned". So I did, and the error comes back, now fully >>> documented below. As I wrote, it works fine with gmres, and the >>> problem is very simple, diagonal dominant steady state diffusion. >>> >>> Any hints are highly appreciated. >>> >>> Dominik >>> >>> #PETSc Option Table entries: >>> -ksp_converged_reason >>> -ksp_converged_use_initial_residual_norm >>> -ksp_monitor_true_residual >>> -ksp_norm_type unpreconditioned >>> -ksp_rtol 1e-3 >>> -ksp_type bcgs >>> -ksp_view >>> -log_summary >>> -pc_type jacobi >>> #End of PETSc Option Table entries >>> 0 KSP preconditioned resid norm 1.166190378969e+01 true resid norm >>> 1.166190378969e+01 ||Ae||/||Ax|| 1.000000000000e+00 >>> 1 KSP preconditioned resid norm 5.658835826231e-01 true resid norm >>> 5.658835826231e-01 ||Ae||/||Ax|| 4.852411688762e-02 >>> [0]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [0]PETSC ERROR: Petsc has generated inconsistent data! >>> [0]PETSC ERROR: Divide by zero! >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [2]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [2]PETSC ERROR: Petsc has generated inconsistent data! >>> [2]PETSC ERROR: Divide by zero! >>> [2]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [2]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >>> 13:37:48 CDT 2011 >>> [2]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [2]PETSC ERROR: See docs/index.html for manual pages. >>> [2]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [2]PETSC ERROR: /home/dsz/build/framework-debug/trunk/bin/cd3t10mpi on >>> a linux-gnu named tharsis by dsz Fri Aug 26 23:23:47 2011 >>> [2]PETSC ERROR: Libraries linked from >>> /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib >>> [2]PETSC ERROR: Configure run at Mon Jul 25 14:20:10 2011 >>> [2]PETSC ERROR: Configure options >>> PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug >>> --download-f-blas-lapack=CD3T10::SaveSolution() >>> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >>> 13:37:48 CDT 2011 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: /home/dsz/build/framework-debug/trunk/bin/cd3t10mpi on >>> a linux-gnu named tharsis by dsz Fri Aug 26 23:23:47 2011 >>> [0]PETSC ERROR: Libraries linked from >>> /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib >>> [0]PETSC ERROR: Configure run at Mon Jul 25 14:20:10 2011 >>> [0]PETSC ERROR: Configure options >>> PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug >>> --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 >>> --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: KSPSolve_BCGS() line 75 in >>> src/ksp/ksp/impls/bcgs/[1]PETSC ERROR: --------------------- Error >>> Message ------------------------------------ >>> [1]PETSC ERROR: Petsc has generated inconsistent data! >>> [1]PETSC ERROR: Divide by zero! >>> [1]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >>> 13:37:48 CDT 2011 >>> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [1]PETSC ERROR: See docs/index.html for manual pages. >>> [1]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [1]PETSC ERROR: /home/dsz/build/framework-debug/trunk/bin/cd3t10mpi on >>> a linux-gnu named tharsis by dsz Fri Aug 26 23:23:47 2011 >>> [1]PETSC ERROR: Libraries linked from >>> /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib >>> [1]PETSC ERROR: Configure run at Mon Jul 25 14:20:10 2011 >>> [1]PETSC ERROR: Configure options >>> PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug >>> --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 >>> --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 >>> [2]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [2]PETSC ERROR: KSPSolve_BCGS() line 75 in src/ksp/ksp/impls/bcgs/bcgs.c >>> [2]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c >>> [2]PETSC ERROR: User provided function() line 1215 in >>> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/PetscSolver.cxx >>> [3]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [3]PETSC ERROR: Petsc has generated inconsistent data! >>> [3]PETSC ERROR: Divide by zero! >>> [3]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [3]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >>> 13:37:48 CDT 2011 >>> [3]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [3]PETSC ERROR: See docs/index.html for manual pages. >>> [3]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [3]PETSC ERROR: /home/dsz/build/framework-debug/trunk/bin/cd3t10mpi on >>> a linux-gnu named tharsis by dsz Fri Aug 26 23:23:47 2011 >>> [3]PETSC ERROR: Libraries linked from >>> /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib >>> [3]PETSC ERROR: Configure run at Mon Jul 25 14:20:10 2011 >>> [3]PETSC ERROR: Configure options >>> PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug >>> --download-f-blas-lapack=bcgs.c >>> [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: User provided function() line 1215 in >>> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/PetscSolver.cxx >>> 1 --download-mpich=1 --download-hypre=1 --with-parmetis=1 >>> --download-parmetis=1 --with-x=0 --with-debugging=1 >>> [1]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [1]PETSC ERROR: KSPSolve_BCGS() line 75 in src/ksp/ksp/impls/bcgs/bcgs.c >>> [1]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c >>> [1]PETSC ERROR: User provided function() line 1215 in >>> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/PetscSolver.cxx >>> 1 --download-mpich=1 --download-hypre=1 --with-parmetis=1 >>> --download-parmetis=1 --with-x=0 --with-debugging=1 >>> [3]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [3]PETSC ERROR: KSPSolve_BCGS() line 75 in src/ksp/ksp/impls/bcgs/bcgs.c >>> [3]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c >>> [3]PETSC ERROR: User provided function() line 1215 in >>> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/PetscSolver.cxx >>> PetscSolver::Finalize() >>> PetscSolver::FinalizePetsc() >>> >>> >>> On Fri, Aug 26, 2011 at 11:17 PM, Barry Smith wrote: >>>> >>>> Are you sure that is the entire error message. It should print the routine and the line number where this happens. >>>> >>>> Likely it is at >>>> >>>> do { >>>> ierr = VecDot(R,RP,&rho);CHKERRQ(ierr); /* rho <- (r,rp) */ >>>> beta = (rho/rhoold) * (alpha/omegaold); >>>> ierr = VecAXPBYPCZ(P,1.0,-omegaold*beta,beta,R,V);CHKERRQ(ierr); /* p <- r - omega * beta* v + beta * p */ >>>> ierr = KSP_PCApplyBAorAB(ksp,P,V,T);CHKERRQ(ierr); /* v <- K p */ >>>> ierr = VecDot(V,RP,&d1);CHKERRQ(ierr); >>>> if (d1 == 0.0) SETERRQ(((PetscObject)ksp)->comm,PETSC_ERR_PLIB,"Divide by zero"); >>>> alpha = rho / d1; /* a <- rho / (v,rp) */ >>>> >>>> Which means bi-cg-stab has broken down. You'll need to consult references to Bi-CG-stab to see why this might happen (it can happen while GMRES is happy). It may be KSPBCGSL can proceed past this point with a problem values for >>>> Options Database Keys: >>>> + -ksp_bcgsl_ell Number of Krylov search directions >>>> - -ksp_bcgsl_cxpol Use a convex function of the MR and OR polynomials after the BiCG step >>>> - -ksp_bcgsl_xres Threshold used to decide when to refresh computed residuals >>>> >>>> but most likely the preconditioner or matrix is bogus in some way since I think Bi-CG-stab rarely breaks down in practice. >>>> >>>> >>>> Barry >>>> >>>> >>>> >>>> On Aug 26, 2011, at 4:05 PM, Dominik Szczerba wrote: >>>> >>>>> When solving my linear system with -ksp_type bcgs I get: >>>>> >>>>> 0 KSP preconditioned resid norm 1.166190378969e+01 true resid norm >>>>> 1.166190378969e+01 ||Ae||/||Ax|| 1.000000000000e+00 >>>>> 1 KSP preconditioned resid norm 5.658835826231e-01 true resid norm >>>>> 5.658835826231e-01 ||Ae||/||Ax|| 4.852411688762e-02 >>>>> [1]PETSC ERROR: --------------------- Error Message >>>>> ------------------------------------ >>>>> [1]PETSC ERROR: Petsc has generated inconsistent data! >>>>> [1]PETSC ERROR: Divide by zero! >>>>> [1]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >>>>> 13:37:48 CDT 2011 >>>>> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [1]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message >>>>> ------------------------------------ >>>>> [3]PETSC ERROR: Petsc has generated inconsistent data! >>>>> [3]PETSC ERROR: Divide by zero! >>>>> >>>>> PETSc Option Table entries: >>>>> -ksp_converged_reason >>>>> -ksp_monitor >>>>> -ksp_monitor_true_residual >>>>> -ksp_norm_type unpreconditioned >>>>> -ksp_right_pc >>>>> -ksp_rtol 1e-3 >>>>> -ksp_type bcgs >>>>> -ksp_view >>>>> -log_summary >>>>> -pc_type jacobi >>>>> >>>>> When solving the same system with GMRES all works fine. This is a >>>>> simple test diffusion problem. How can I find out what the problem is? >>>>> >>>>> Dominik >>>> >>>> >> >> From dominik at itis.ethz.ch Fri Aug 26 16:58:25 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Fri, 26 Aug 2011 23:58:25 +0200 Subject: [petsc-users] Petsc has generated inconsistent data! In-Reply-To: References: <12468DD1-3BC8-4319-B8D0-EAAF0AD3D168@mcs.anl.gov> Message-ID: OK I got it: initial guess. Was zero, why not, but perturbing it a bit off zero seems to work. I saw this already before with my own bicgstab implementation, and always thought this was some bug. So you either have a similar one, or its a bicgstab's feature :) Dominik On Fri, Aug 26, 2011 at 11:51 PM, Dominik Szczerba wrote: > Valgrind excerpts below, none seems related, and both are there also with gmres. > > I just cant believe bicgstab would quit on such a trivial problem... > > Regards, > Dominik > > ==10423== Syscall param writev(vector[...]) points to uninitialised byte(s) > ==10423== ? ?at 0x6B5E789: writev (writev.c:56) > ==10423== ? ?by 0x515658: MPIDU_Sock_writev (sock_immed.i:610) > ==10423== ? ?by 0x517943: MPIDI_CH3_iSendv (ch3_isendv.c:84) > ==10423== ? ?by 0x4FB756: MPIDI_CH3_EagerContigIsend (ch3u_eager.c:509) > ==10423== ? ?by 0x4FD6E4: MPID_Isend (mpid_isend.c:118) > ==10423== ? ?by 0x4E502A: MPIC_Isend (helper_fns.c:210) > ==10423== ? ?by 0xED701D: MPIR_Alltoall (alltoall.c:420) > ==10423== ? ?by 0xED7880: PMPI_Alltoall (alltoall.c:685) > ==10423== ? ?by 0xE4BC95: SetUp__ (setup.c:122) > ==10423== ? ?by 0xE4C4E4: PartitionSmallGraph__ (weird.c:39) > ==10423== ? ?by 0xE497D0: ParMETIS_V3_PartKway (kmetis.c:131) > ==10423== ? ?by 0x71B774: MatPartitioningApply_Parmetis (pmetis.c:97) > ==10423== ? ?by 0x717D70: MatPartitioningApply (partition.c:236) > ==10423== ? ?by 0x5287D7: PetscSolver::LoadMesh(std::string const&) > (PetscSolver.cxx:676) > ==10423== ? ?by 0x4C6157: CD3T10_USER::ProcessInputFile() > (cd3t10mpi_main.cxx:321) > ==10423== ? ?by 0x4C3B57: main (cd3t10mpi_main.cxx:568) > ==10423== ?Address 0x71ce9b4 is 4 bytes inside a block of size 72 alloc'd > ==10423== ? ?at 0x4C28FAC: malloc (vg_replace_malloc.c:236) > ==10423== ? ?by 0xE5C792: GKmalloc__ (util.c:151) > ==10423== ? ?by 0xE57C09: PreAllocateMemory__ (memory.c:38) > ==10423== ? ?by 0xE496B3: ParMETIS_V3_PartKway (kmetis.c:116) > ==10423== ? ?by 0x71B774: MatPartitioningApply_Parmetis (pmetis.c:97) > ==10423== ? ?by 0x717D70: MatPartitioningApply (partition.c:236) > ==10423== ? ?by 0x5287D7: PetscSolver::LoadMesh(std::string const&) > (PetscSolver.cxx:676) > ==10423== ? ?by 0x4C6157: CD3T10_USER::ProcessInputFile() > (cd3t10mpi_main.cxx:321) > ==10423== ? ?by 0x4C3B57: main (cd3t10mpi_main.cxx:568) > ==10423== > ==10423== Conditional jump or move depends on uninitialised value(s) > ==10423== ? ?at 0x55B6510: inflateReset2 (in > /lib/x86_64-linux-gnu/libz.so.1.2.3.4) > ==10423== ? ?by 0x55B6605: inflateInit2_ (in > /lib/x86_64-linux-gnu/libz.so.1.2.3.4) > ==10423== ? ?by 0x5308C13: H5Z_filter_deflate (H5Zdeflate.c:110) > ==10423== ? ?by 0x5308170: H5Z_pipeline (H5Z.c:1103) > ==10423== ? ?by 0x518BB69: H5D_chunk_lock (H5Dchunk.c:2758) > ==10423== ? ?by 0x518CB20: H5D_chunk_read (H5Dchunk.c:1728) > ==10423== ? ?by 0x519BDDA: H5D_read (H5Dio.c:447) > ==10423== ? ?by 0x519C248: H5Dread (H5Dio.c:173) > ==10423== ? ?by 0x4CEB99: HDF5::HDF5Reader::readData(std::string > const&) (HDF5Reader.cxx:634) > ==10423== ? ?by 0x4CE305: HDF5::HDF5Reader::read(std::vector std::allocator >&, std::string const&) (HDF5Reader.cxx:527) > ==10423== ? ?by 0x4C71A3: CD3T10_USER::SetupConstraints() > (cd3t10mpi_main.cxx:404) > ==10423== ? ?by 0x4BA02A: CD3T10::Solve() (CD3T10mpi.cxx:658) > ==10423== ? ?by 0x4C3BE1: main (cd3t10mpi_main.cxx:590) > ==10423== > > On Fri, Aug 26, 2011 at 11:31 PM, Barry Smith wrote: >> >> ?First run with valgrind http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind to make sure it does not have some "code bug cause". >> >> ? Do you get the same message on one process. >> >> ? My previous message still holds, if it is not a "code bug" then bi-CG-stab is just breaking down on that matrix and preconditioner combination. >> >> ? Barry >> >> On Aug 26, 2011, at 4:27 PM, Dominik Szczerba wrote: >> >>> Later in the message he only requested that I use "-ksp_norm_type >>> unpreconditioned". So I did, and the error comes back, now fully >>> documented below. As I wrote, it works fine with gmres, and the >>> problem is very simple, diagonal dominant steady state diffusion. >>> >>> Any hints are highly appreciated. >>> >>> Dominik >>> >>> #PETSc Option Table entries: >>> -ksp_converged_reason >>> -ksp_converged_use_initial_residual_norm >>> -ksp_monitor_true_residual >>> -ksp_norm_type unpreconditioned >>> -ksp_rtol 1e-3 >>> -ksp_type bcgs >>> -ksp_view >>> -log_summary >>> -pc_type jacobi >>> #End of PETSc Option Table entries >>> ?0 KSP preconditioned resid norm 1.166190378969e+01 true resid norm >>> 1.166190378969e+01 ||Ae||/||Ax|| 1.000000000000e+00 >>> ?1 KSP preconditioned resid norm 5.658835826231e-01 true resid norm >>> 5.658835826231e-01 ||Ae||/||Ax|| 4.852411688762e-02 >>> [0]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [0]PETSC ERROR: Petsc has generated inconsistent data! >>> [0]PETSC ERROR: Divide by zero! >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [2]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [2]PETSC ERROR: Petsc has generated inconsistent data! >>> [2]PETSC ERROR: Divide by zero! >>> [2]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [2]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >>> 13:37:48 CDT 2011 >>> [2]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [2]PETSC ERROR: See docs/index.html for manual pages. >>> [2]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [2]PETSC ERROR: /home/dsz/build/framework-debug/trunk/bin/cd3t10mpi on >>> a linux-gnu named tharsis by dsz Fri Aug 26 23:23:47 2011 >>> [2]PETSC ERROR: Libraries linked from >>> /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib >>> [2]PETSC ERROR: Configure run at Mon Jul 25 14:20:10 2011 >>> [2]PETSC ERROR: Configure options >>> PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug >>> --download-f-blas-lapack=CD3T10::SaveSolution() >>> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >>> 13:37:48 CDT 2011 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: /home/dsz/build/framework-debug/trunk/bin/cd3t10mpi on >>> a linux-gnu named tharsis by dsz Fri Aug 26 23:23:47 2011 >>> [0]PETSC ERROR: Libraries linked from >>> /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib >>> [0]PETSC ERROR: Configure run at Mon Jul 25 14:20:10 2011 >>> [0]PETSC ERROR: Configure options >>> PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug >>> --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 >>> --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: KSPSolve_BCGS() line 75 in >>> src/ksp/ksp/impls/bcgs/[1]PETSC ERROR: --------------------- Error >>> Message ------------------------------------ >>> [1]PETSC ERROR: Petsc has generated inconsistent data! >>> [1]PETSC ERROR: Divide by zero! >>> [1]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >>> 13:37:48 CDT 2011 >>> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [1]PETSC ERROR: See docs/index.html for manual pages. >>> [1]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [1]PETSC ERROR: /home/dsz/build/framework-debug/trunk/bin/cd3t10mpi on >>> a linux-gnu named tharsis by dsz Fri Aug 26 23:23:47 2011 >>> [1]PETSC ERROR: Libraries linked from >>> /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib >>> [1]PETSC ERROR: Configure run at Mon Jul 25 14:20:10 2011 >>> [1]PETSC ERROR: Configure options >>> PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug >>> --download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1 >>> --with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1 >>> [2]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [2]PETSC ERROR: KSPSolve_BCGS() line 75 in src/ksp/ksp/impls/bcgs/bcgs.c >>> [2]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c >>> [2]PETSC ERROR: User provided function() line 1215 in >>> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/PetscSolver.cxx >>> [3]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [3]PETSC ERROR: Petsc has generated inconsistent data! >>> [3]PETSC ERROR: Divide by zero! >>> [3]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [3]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >>> 13:37:48 CDT 2011 >>> [3]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [3]PETSC ERROR: See docs/index.html for manual pages. >>> [3]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [3]PETSC ERROR: /home/dsz/build/framework-debug/trunk/bin/cd3t10mpi on >>> a linux-gnu named tharsis by dsz Fri Aug 26 23:23:47 2011 >>> [3]PETSC ERROR: Libraries linked from >>> /home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib >>> [3]PETSC ERROR: Configure run at Mon Jul 25 14:20:10 2011 >>> [3]PETSC ERROR: Configure options >>> PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug >>> --download-f-blas-lapack=bcgs.c >>> [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: User provided function() line 1215 in >>> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/PetscSolver.cxx >>> 1 --download-mpich=1 --download-hypre=1 --with-parmetis=1 >>> --download-parmetis=1 --with-x=0 --with-debugging=1 >>> [1]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [1]PETSC ERROR: KSPSolve_BCGS() line 75 in src/ksp/ksp/impls/bcgs/bcgs.c >>> [1]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c >>> [1]PETSC ERROR: User provided function() line 1215 in >>> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/PetscSolver.cxx >>> 1 --download-mpich=1 --download-hypre=1 --with-parmetis=1 >>> --download-parmetis=1 --with-x=0 --with-debugging=1 >>> [3]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [3]PETSC ERROR: KSPSolve_BCGS() line 75 in src/ksp/ksp/impls/bcgs/bcgs.c >>> [3]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c >>> [3]PETSC ERROR: User provided function() line 1215 in >>> "unknowndirectory/"/home/dsz/src/framework/trunk/solve/PetscSolver.cxx >>> PetscSolver::Finalize() >>> PetscSolver::FinalizePetsc() >>> >>> >>> On Fri, Aug 26, 2011 at 11:17 PM, Barry Smith wrote: >>>> >>>> ?Are you sure that is the entire error message. It should print the routine and the line number where this happens. >>>> >>>> ? Likely it is at >>>> >>>> ?do { >>>> ? ?ierr = VecDot(R,RP,&rho);CHKERRQ(ierr); ? ? ? /* ? rho <- (r,rp) ? ? ?*/ >>>> ? ?beta = (rho/rhoold) * (alpha/omegaold); >>>> ? ?ierr = VecAXPBYPCZ(P,1.0,-omegaold*beta,beta,R,V);CHKERRQ(ierr); ?/* p <- r - omega * beta* v + beta * p */ >>>> ? ?ierr = KSP_PCApplyBAorAB(ksp,P,V,T);CHKERRQ(ierr); ?/* ? v <- K p ? ? ? ? ? */ >>>> ? ?ierr = VecDot(V,RP,&d1);CHKERRQ(ierr); >>>> ? ?if (d1 == 0.0) SETERRQ(((PetscObject)ksp)->comm,PETSC_ERR_PLIB,"Divide by zero"); >>>> ? ?alpha = rho / d1; ? ? ? ? ? ? ? ? /* ? a <- rho / (v,rp) ?*/ >>>> >>>> ?Which means bi-cg-stab has broken down. You'll need to consult references to Bi-CG-stab to see why this might happen (it can happen while GMRES is happy). It may be KSPBCGSL can proceed past this point with a problem values for >>>> ? Options Database Keys: >>>> + ?-ksp_bcgsl_ell Number of Krylov search directions >>>> - ?-ksp_bcgsl_cxpol Use a convex function of the MR and OR polynomials after the BiCG step >>>> - ?-ksp_bcgsl_xres Threshold used to decide when to refresh computed residuals >>>> >>>> but most likely the preconditioner or matrix is bogus in some way since I think Bi-CG-stab rarely breaks down in practice. >>>> >>>> >>>> ?Barry >>>> >>>> >>>> >>>> On Aug 26, 2011, at 4:05 PM, Dominik Szczerba wrote: >>>> >>>>> When solving my linear system with -ksp_type bcgs I get: >>>>> >>>>> ?0 KSP preconditioned resid norm 1.166190378969e+01 true resid norm >>>>> 1.166190378969e+01 ||Ae||/||Ax|| 1.000000000000e+00 >>>>> ?1 KSP preconditioned resid norm 5.658835826231e-01 true resid norm >>>>> 5.658835826231e-01 ||Ae||/||Ax|| 4.852411688762e-02 >>>>> [1]PETSC ERROR: --------------------- Error Message >>>>> ------------------------------------ >>>>> [1]PETSC ERROR: Petsc has generated inconsistent data! >>>>> [1]PETSC ERROR: Divide by zero! >>>>> [1]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >>>>> 13:37:48 CDT 2011 >>>>> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [1]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message >>>>> ------------------------------------ >>>>> [3]PETSC ERROR: Petsc has generated inconsistent data! >>>>> [3]PETSC ERROR: Divide by zero! >>>>> >>>>> PETSc Option Table entries: >>>>> -ksp_converged_reason >>>>> -ksp_monitor >>>>> -ksp_monitor_true_residual >>>>> -ksp_norm_type unpreconditioned >>>>> -ksp_right_pc >>>>> -ksp_rtol 1e-3 >>>>> -ksp_type bcgs >>>>> -ksp_view >>>>> -log_summary >>>>> -pc_type jacobi >>>>> >>>>> When solving the same system with GMRES all works fine. This is a >>>>> simple test diffusion problem. How can I find out what the problem is? >>>>> >>>>> Dominik >>>> >>>> >> >> > From ecoon at lanl.gov Fri Aug 26 17:00:21 2011 From: ecoon at lanl.gov (Ethan Coon) Date: Fri, 26 Aug 2011 16:00:21 -0600 Subject: [petsc-users] about MPI In-Reply-To: References: Message-ID: <1314396021.28940.13.camel@echo.lanl.gov> By default, PETSc doesn't put mpiexec in the path. Try: ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -n ... Assuming that works, you can put this directory, i.e. $PETSC_DIR/$PETSC_ARCH/bin in your $PATH environmental variable, and just using "mpiexec" should work fine then. Ethan On Fri, 2011-08-26 at 17:41 -0400, Likun Tan wrote: > Hello, > > I try to run my code with MPI, but i got the message: 'mpiexec: command > not found'. I installed mpi and tested it successfully on examples with > 'make all test'. > I also tried to specify the path for mpi with ./config/configure.py > --with-mpi-dir=/home/likunt/petsc-3.1-p8/externalpackages/mpich2-1.0.8, > but it shows 'unable to configure with given options'. > > Any suggestions on this? > > Thanks, > Likun > > > > > > -- ------------------------------------ Ethan Coon Post-Doctoral Researcher Applied Mathematics - T-5 Los Alamos National Laboratory 505-665-8289 http://www.ldeo.columbia.edu/~ecoon/ ------------------------------------ From likunt at andrew.cmu.edu Fri Aug 26 17:20:08 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Fri, 26 Aug 2011 18:20:08 -0400 Subject: [petsc-users] about MPI In-Reply-To: <1314396021.28940.13.camel@echo.lanl.gov> References: <1314396021.28940.13.camel@echo.lanl.gov> Message-ID: it works, thanks On Fri, August 26, 2011 6:00 pm, Ethan Coon wrote: > By default, PETSc doesn't put mpiexec in the path. > > > Try: > > > ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -n ... > > > Assuming that works, you can put this directory, i.e. > > > $PETSC_DIR/$PETSC_ARCH/bin > > > in your $PATH environmental variable, and just using "mpiexec" should work > fine then. > > Ethan > > > > On Fri, 2011-08-26 at 17:41 -0400, Likun Tan wrote: > >> Hello, >> >> >> I try to run my code with MPI, but i got the message: 'mpiexec: command >> not found'. I installed mpi and tested it successfully on examples >> with 'make all test'. >> > >> I also tried to specify the path for mpi with ./config/configure.py >> --with-mpi-dir=/home/likunt/petsc-3.1-p8/externalpackages/mpich2-1.0.8, >> but it shows 'unable to configure with given options'. >> >> Any suggestions on this? >> >> >> Thanks, >> Likun >> >> >> >> >> >> >> > > -- > ------------------------------------ > Ethan Coon > Post-Doctoral Researcher > Applied Mathematics - T-5 > Los Alamos National Laboratory > 505-665-8289 > > > http://www.ldeo.columbia.edu/~ecoon/ > ------------------------------------ > > > > From ram at ibrae.ac.ru Sat Aug 27 08:10:50 2011 From: ram at ibrae.ac.ru (=?UTF-8?B?0JDQu9C10LrRgdC10Lkg0KDRj9C30LDQvdC+0LI=?=) Date: Sat, 27 Aug 2011 17:10:50 +0400 Subject: [petsc-users] performance issue solving multiple linear systems of the same size with the different preconditioning methods In-Reply-To: References: Message-ID: I've figured out, that the amount of undestroyed objects depends on the type of the linear system matrix. Now I'm assembling 3d laplacian with Neuman boundary conditions, which is singular, and I have huge amount of undestroyed vectors, mappings and scatters. If I use zero matrix, the number of destroyed objects equals to the number of created. But if I even use unitary matrix, which is not singular,I already have some undestroyed objects. 27 ??????? 2011 ?. 1:29 ???????????? ??????? ??????? ???????: > I have also checked KSPSolve behavior in my other PETSc programs and found > the same memory lack > > 27 ??????? 2011 ?. 0:57 ???????????? ??????? ??????? ???????: > > >> Thank you for your response! >> >> I have the memory leak in both my programs. But I don't create plenty of >> vectors. >> >> My code looks like: >> >> ***INIT_ALL*** >> PetscLogStageRegister("Iteration :", &StageNum1); >> PetscLogStagePush(StageNum1); >> KSPSolve(dKSP, dvec_origRHS, dvec_Solution); >> PetscLogStagePop(); >> ***DESTROY_ALL*** >> >> >> And when I comment (or delete) KSPSolve, the log_summary output is: >> >> ===================================================== >> Memory usage is given in bytes: >> >> Creations Destructions Memory Descendants' Mem Object Type >> >> Reports information only for process 0. >> >> --- Event Stage 0: Main Stage >> 1 1 729472 0 Application Order >> 1 1 225452 0 Distributed array >> 8 8 1533424 0 Vec >> 3 3 2604 0 Vec Scatter >> 8 8 613852 0 Index Set >> 1 1 221304 0 IS L to G Mapping >> 3 3 16603440 0 Matrix >> 1 1 832 0 Krylov Solver >> 1 1 688 0 Preconditioner >> 1 1 448 0 PetscRandom >> >> --- Event Stage 1: Iteration : >> ===================================================== >> >> When I run the code with KSPSolve instruction, it gives me: >> >> ===================================================== >> Memory usage is given in bytes: >> Creations Destructions Memory Descendants' Mem Object Type >> >> Reports information only for process 0. >> >> --- Event Stage 0: Main Stage >> 1 0 0 0 Application Order >> 1 0 0 0 Distributed array >> 8 17 4963592 0 Vec >> 3 2 1736 0 Vec Scatter >> 8 12 1425932 0 Index Set >> 1 0 0 0 IS L to G Mapping >> 3 5 50158132 0 Matrix >> 1 2 1664 0 Krylov Solver >> 1 2 1440 0 Preconditioner >> 1 1 448 0 PetscRandom >> 0 1 544 0 Viewer >> >> --- Event Stage 1: Iteration : >> 355 173 64692312 0 Vec >> 1 0 0 0 Vec Scatter >> 6 2 1024 0 Index Set >> 2 0 0 0 Matrix >> 1 0 0 0 Krylov Solver >> 1 0 0 0 Preconditioner >> 2 1 544 0 Viewer >> ===================================================== >> >> >> >> >> 2011/8/25 Jed Brown >> >>> On Tue, Aug 23, 2011 at 02:37, ??????? ??????? wrote: >>> >>>> When i delete the 4-5-6 part of 2nd, 1-2-3 works great! with exact like >>>> 1st results. >>>> When i delete the 1-2-3 part of 2nd, 4-5-6 works great! with exact like >>>> 1st results. >>>> All program (1-2-3-4-5-6) works badly. >>>> >>> >>> From the -log_summary, you have a memory leak (many more vector creations >>> than destructions). Try running with -malloc_dump to debug it. Perhaps you >>> are creating a vector every time one of your functions is called? You should >>> also build --with-debugging=0 when looking at timing results. (You can keep >>> it in PETSC_ARCH=linux-gnu-opt.) >>> >> >> >> >> -- >> Best regards, >> Alexey Ryazanov >> ______________________________________ >> Nuclear Safety Institute of Russian Academy of Sciences >> >> >> > > > -- > Best regards, > Alexey Ryazanov > ______________________________________ > Nuclear Safety Institute of Russian Academy of Sciences > > > -- Best regards, Alexey Ryazanov ______________________________________ Nuclear Safety Institute of Russian Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at ibrae.ac.ru Sat Aug 27 08:32:39 2011 From: ram at ibrae.ac.ru (=?UTF-8?B?0JDQu9C10LrRgdC10Lkg0KDRj9C30LDQvdC+0LI=?=) Date: Sat, 27 Aug 2011 17:32:39 +0400 Subject: [petsc-users] performance issue solving multiple linear systems of the same size with the different preconditioning methods In-Reply-To: References: Message-ID: Oh BOY! It was all fault of my MyKSPConvergedTest. 27 ??????? 2011 ?. 1:29 ???????????? ??????? ??????? ???????: > I have also checked KSPSolve behavior in my other PETSc programs and found > the same memory lack > > 27 ??????? 2011 ?. 0:57 ???????????? ??????? ??????? ???????: > > >> Thank you for your response! >> >> I have the memory leak in both my programs. But I don't create plenty of >> vectors. >> >> My code looks like: >> >> ***INIT_ALL*** >> PetscLogStageRegister("Iteration :", &StageNum1); >> PetscLogStagePush(StageNum1); >> KSPSolve(dKSP, dvec_origRHS, dvec_Solution); >> PetscLogStagePop(); >> ***DESTROY_ALL*** >> >> >> And when I comment (or delete) KSPSolve, the log_summary output is: >> >> ===================================================== >> Memory usage is given in bytes: >> >> Creations Destructions Memory Descendants' Mem Object Type >> >> Reports information only for process 0. >> >> --- Event Stage 0: Main Stage >> 1 1 729472 0 Application Order >> 1 1 225452 0 Distributed array >> 8 8 1533424 0 Vec >> 3 3 2604 0 Vec Scatter >> 8 8 613852 0 Index Set >> 1 1 221304 0 IS L to G Mapping >> 3 3 16603440 0 Matrix >> 1 1 832 0 Krylov Solver >> 1 1 688 0 Preconditioner >> 1 1 448 0 PetscRandom >> >> --- Event Stage 1: Iteration : >> ===================================================== >> >> When I run the code with KSPSolve instruction, it gives me: >> >> ===================================================== >> Memory usage is given in bytes: >> Creations Destructions Memory Descendants' Mem Object Type >> >> Reports information only for process 0. >> >> --- Event Stage 0: Main Stage >> 1 0 0 0 Application Order >> 1 0 0 0 Distributed array >> 8 17 4963592 0 Vec >> 3 2 1736 0 Vec Scatter >> 8 12 1425932 0 Index Set >> 1 0 0 0 IS L to G Mapping >> 3 5 50158132 0 Matrix >> 1 2 1664 0 Krylov Solver >> 1 2 1440 0 Preconditioner >> 1 1 448 0 PetscRandom >> 0 1 544 0 Viewer >> >> --- Event Stage 1: Iteration : >> 355 173 64692312 0 Vec >> 1 0 0 0 Vec Scatter >> 6 2 1024 0 Index Set >> 2 0 0 0 Matrix >> 1 0 0 0 Krylov Solver >> 1 0 0 0 Preconditioner >> 2 1 544 0 Viewer >> ===================================================== >> >> >> >> >> 2011/8/25 Jed Brown >> >>> On Tue, Aug 23, 2011 at 02:37, ??????? ??????? wrote: >>> >>>> When i delete the 4-5-6 part of 2nd, 1-2-3 works great! with exact like >>>> 1st results. >>>> When i delete the 1-2-3 part of 2nd, 4-5-6 works great! with exact like >>>> 1st results. >>>> All program (1-2-3-4-5-6) works badly. >>>> >>> >>> From the -log_summary, you have a memory leak (many more vector creations >>> than destructions). Try running with -malloc_dump to debug it. Perhaps you >>> are creating a vector every time one of your functions is called? You should >>> also build --with-debugging=0 when looking at timing results. (You can keep >>> it in PETSC_ARCH=linux-gnu-opt.) >>> >> >> >> >> -- >> Best regards, >> Alexey Ryazanov >> ______________________________________ >> Nuclear Safety Institute of Russian Academy of Sciences >> >> >> > > > -- > Best regards, > Alexey Ryazanov > ______________________________________ > Nuclear Safety Institute of Russian Academy of Sciences > > > -- Best regards, Alexey Ryazanov ______________________________________ Nuclear Safety Institute of Russian Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From milan.v.mitrovic at gmail.com Sat Aug 27 11:17:26 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Sat, 27 Aug 2011 18:17:26 +0200 Subject: [petsc-users] Problem with SNES Message-ID: Hello everyone, I'm having trouble getting SNES to work. I get 0 for the iteration number. Here is the code I have: call SNESCreate(PETSC_COMM_WORLD, snes, info) call SNESSetType(snes, SNESLS, info) call SNESGetKSP(snes, ksp, info) call KSPGetPC(ksp, pc, info) call PCSetType(pc, PCNONE, info) call SNESSetFunction(snes, r, implicit_step, ctx, info) call SNESSetJacobian(snes, J, J, jacobian, PETSC_NULL, info) call SNESSetFromOptions(snes, info) call SNESSolve(snes, PETSC_NULL, x, info) call SNESGetIterationNumber(snes, i, info) the jacobian is created with: call MatCreateShell(PETSC_COMM_WORLD, Np, Np, Ng, Ng, 0, Jac, info) call MatShellSetOperation(Jac, MATOP_MULT, jacobian_multiplication, info) Any hints as to what I should do to find the problem are very welcome :) Thanks in advance, Milan From jedbrown at mcs.anl.gov Sat Aug 27 11:19:36 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 27 Aug 2011 11:19:36 -0500 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: On Sat, Aug 27, 2011 at 11:17, Milan Mitrovic wrote: > I get 0 for the iteration number. Run with -ksp_converged_reason -snes_converged_reason -snes_monitor -snes_view and send the output. -------------- next part -------------- An HTML attachment was scrubbed... URL: From milan.v.mitrovic at gmail.com Sat Aug 27 11:28:42 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Sat, 27 Aug 2011 18:28:42 +0200 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: Linear solve converged due to CONVERGED_RTOL iterations 0 Nonlinear solve did not converge due to DIVERGED_LS_FAILURE From jedbrown at mcs.anl.gov Sat Aug 27 11:33:53 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 27 Aug 2011 11:33:53 -0500 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: On Sat, Aug 27, 2011 at 11:28, Milan Mitrovic wrote: > Linear solve converged due to CONVERGED_RTOL iterations 0 > Nonlinear solve did not converge due to DIVERGED_LS_FAILURE > The Jacobian is likely incorrect, try running a small problem size with -snes_type test. -------------- next part -------------- An HTML attachment was scrubbed... URL: From milan.v.mitrovic at gmail.com Sat Aug 27 12:22:23 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Sat, 27 Aug 2011 19:22:23 +0200 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: Testing hand-coded Jacobian, if the ratio is O(1.e-8), the hand-coded Jacobian is probably correct. Run with -snes_test_display to show difference of hand-coded and finite difference Jacobian. [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type see http://www.mcs.anl.gov/petsc/petsc-as/documentation/installation.html#external! [1]PETSC ERROR: Unknown Mat type given: same! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ./ppm_pf on a darwin10. named milan-mitrovics-macbook-pro.local by milan Sat Aug 27 19:11:18 2011 [1]PETSC ERROR: Libraries linked from /Users/milan/Work/Library/petsc/lib [1]PETSC ERROR: Configure run at Fri May 6 16:32:02 2011 [1]PETSC ERROR: Configure options --with-mpi=true --with-mpi-dir=/Users/milan/Work/Library/openmpi.gcc/ --with-fortran=true --with-sundials=true --with-sundials-dir=/Users/milan/Work/Library/sundials/ --prefix=/Users/milan/Work/Library/petsc --------------------- Error Message ------------------------------------ [0]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: MatSetType() line 51 in src/mat/interface/matreg.c [1]PETSC ERROR: MatConvert() line 3521 in src/mat/interface/matrix.c [1]PETSC ERROR: SNESSolve_Test() line 47 in src/snes/impls/test/snestest.c [1]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c From jedbrown at mcs.anl.gov Sat Aug 27 12:42:29 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 27 Aug 2011 12:42:29 -0500 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: On Sat, Aug 27, 2011 at 12:22, Milan Mitrovic wrote: > [1]PETSC ERROR: Unknown Mat type given: same! What kind of Mat were you assembling? Are you passing any other flags? What does your code assemble for the Jacobian? I think the Jacobian you passed in does not implement MatDuplicate(). I just changed MatConvert() to be more robust in petsc-dev. With 3.1, you can try -pc_type lu -snes_monitor -ksp_monitor_singular_value. Also compare output with -snes_mf_operator. I think your Jacobian may be singular and this should help figure that out. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Aug 27 12:51:43 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 27 Aug 2011 12:51:43 -0500 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: <4E0BA676-4243-4DE2-B789-11216C837A9E@mcs.anl.gov> On Aug 27, 2011, at 12:42 PM, Jed Brown wrote: > On Sat, Aug 27, 2011 at 12:22, Milan Mitrovic wrote: > [1]PETSC ERROR: Unknown Mat type given: same! > > What kind of Mat were you assembling? Are you passing any other flags? What does your code assemble for the Jacobian? I think the Jacobian you passed in does not implement MatDuplicate(). I just changed MatConvert() to be more robust in petsc-dev. > > With 3.1, you can try -pc_type lu -snes_monitor -ksp_monitor_singular_value. Also compare output with -snes_mf_operator. I think your Jacobian may be singular and this should help figure that out. See also http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#newton for various tips on dealing with a non-converging Newton. Barry From milan.v.mitrovic at gmail.com Sat Aug 27 12:52:43 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Sat, 27 Aug 2011 19:52:43 +0200 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: for the jacobian i use: call MatCreateShell(PETSC_COMM_WORLD, Np, Np, Ng, Ng, 0, Jac, info) call MatShellSetOperation(Jac, MATOP_MULT, & jacobian_multiplication, info) and do the multiplication myself because it is much faster. I tried using parallel AIJ format but constructing the matrix took more than a minute for my problem with only ~150 nonzero entries per row... (maybe I was doing something very wrong) should I return to the AIJ version to see if the jacobian is correct? From bsmith at mcs.anl.gov Sat Aug 27 12:55:21 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 27 Aug 2011 12:55:21 -0500 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: On Aug 27, 2011, at 12:52 PM, Milan Mitrovic wrote: > for the jacobian i use: > > call MatCreateShell(PETSC_COMM_WORLD, Np, Np, Ng, Ng, 0, Jac, info) > call MatShellSetOperation(Jac, MATOP_MULT, & > jacobian_multiplication, info) > > and do the multiplication myself because it is much faster. I tried > using parallel AIJ format but constructing the matrix took more than a > minute for my problem with only ~150 nonzero entries per row... (maybe > I was doing something very wrong) > > should I return to the AIJ version to see if the jacobian is correct? If you have the code for both then add a simple switch in the code to allow changing back and forth very easily to debug things like Newton. For example with AIJ you can use a direct solver to eliminate most of the issues related to the linear solver from the problem of Newton's method. Barry From jedbrown at mcs.anl.gov Sat Aug 27 12:57:03 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 27 Aug 2011 12:57:03 -0500 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: On Sat, Aug 27, 2011 at 12:52, Milan Mitrovic wrote: > and do the multiplication myself because it is much faster. I tried > using parallel AIJ format but constructing the matrix took more than a > minute for my problem with only ~150 nonzero entries per row... (maybe > I was doing something very wrong) > Probably preallocation, see http://www.mcs.anl.gov/petsc/petsc-2/documentation/faq.html#efficient-assembly If you have a more efficient way to apply the action of the matrix (e.g. exploit a tensor product to reduce the memory usage), it makes sense to use the PETSc matrix formats. Also, most problems need preconditioning and it's convenient to have an assembled matrix available in a PETSc format so that you can try a variety of preconditioners. You might end up doing something clever in the end, for which fewer/smaller matrices are assembled, but having assembled matrices is very convenient for experimentation and to check code correctness. -------------- next part -------------- An HTML attachment was scrubbed... URL: From milan.v.mitrovic at gmail.com Sat Aug 27 13:04:55 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Sat, 27 Aug 2011 20:04:55 +0200 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: well... I added matrix elements by row (~150 nonzero elements) but I have 7.5k rows per processor... I know the exact nonzero pattern and I preallocated the storage so it is not that... and its all done locally... I couldn't insert blocks because there is no common structure to the nonzero pattern of different rows... so to me it seems that creating the matrix like this should be fast... computing the values took <3 seconds but the setvalues routine took more than a minute! From jedbrown at mcs.anl.gov Sat Aug 27 13:09:40 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 27 Aug 2011 13:09:40 -0500 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: On Sat, Aug 27, 2011 at 13:04, Milan Mitrovic wrote: > well... I added matrix elements by row (~150 nonzero elements) but I > have 7.5k rows per processor... I know the exact nonzero pattern and I > preallocated the storage so it is not that... and its all done > locally... I couldn't insert blocks because there is no common > structure to the nonzero pattern of different rows... > > so to me it seems that creating the matrix like this should be fast... > computing the values took <3 seconds but the setvalues routine took > more than a minute! > Preallocation must have been wrong. Perhaps it was destroyed/ignored by calling the function too early (the matrix type has to be set _before_ preallocation). Try MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE) and run in a debugger to see where new entries are being allocated. With petsc-dev, you can just run with -mat_new_nonzero_allocation_err. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Sat Aug 27 13:18:32 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Sat, 27 Aug 2011 20:18:32 +0200 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: What does -info say about mallocs? On Sat, Aug 27, 2011 at 8:04 PM, Milan Mitrovic wrote: > well... I added matrix elements by row (~150 nonzero elements) but I > have 7.5k rows per processor... I know the exact nonzero pattern and I > preallocated the storage so it is not that... and its all done > locally... I couldn't insert blocks because there is no common > structure to the nonzero pattern of different rows... > > so to me it seems that creating the matrix like this should be fast... > computing the values took <3 seconds but the setvalues routine took > more than a minute! > > From milan.v.mitrovic at gmail.com Sat Aug 27 13:31:26 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Sat, 27 Aug 2011 20:31:26 +0200 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: what do you mean by setting the matrix type before the preallocation? I did it like this: call MatCreateMPIAIJ(PETSC_COMM_WORLD, Np, Np, Ng, Ng, & nmax, PETSC_NULL, nmax, PETSC_NULL, Jac, info) call MatSetLocalToGlobalMapping(Jac, lgm, info) (i also had d_nnz and o_nnz there, but I deleted it when I switched to matrix shell) From jedbrown at mcs.anl.gov Sat Aug 27 13:35:34 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 27 Aug 2011 13:35:34 -0500 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: On Sat, Aug 27, 2011 at 13:31, Milan Mitrovic wrote: > what do you mean by setting the matrix type before the preallocation? > I did it like this: > > call MatCreateMPIAIJ(PETSC_COMM_WORLD, Np, Np, Ng, Ng, & > nmax, PETSC_NULL, nmax, PETSC_NULL, Jac, info) > call MatSetLocalToGlobalMapping(Jac, lgm, info) > This is fine. If you call MatSetType() or MatSetFromOptions(), then MatXXXSetPreallocation() needs to be called after, otherwise it can get lost. > > (i also had d_nnz and o_nnz there, but I deleted it when I switched to > matrix shell) > The preallocation was almost certainly incorrect. Otherwise there is no way it would take so long to insert the number of values you state. You can run with -info |grep -i malloc or set the option in my last message. -------------- next part -------------- An HTML attachment was scrubbed... URL: From milan.v.mitrovic at gmail.com Sat Aug 27 13:41:05 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Sat, 27 Aug 2011 20:41:05 +0200 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: hm... I cant get to that point now... I get: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: d_nnz cannot be less than 0: local row 64 value -2! even though I supplied PETSC_NULL... could this be one of the fortran compatibility problems? I don't remember having a problem with petsc_null before... From jedbrown at mcs.anl.gov Sat Aug 27 13:44:13 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 27 Aug 2011 13:44:13 -0500 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: On Sat, Aug 27, 2011 at 13:41, Milan Mitrovic wrote: > even though I supplied PETSC_NULL... > > could this be one of the fortran compatibility problems? I don't > remember having a problem with petsc_null before... > You need PETSC_NULL_INTEGER to pass a NULL integer pointer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From milan.v.mitrovic at gmail.com Sat Aug 27 13:55:42 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Sat, 27 Aug 2011 20:55:42 +0200 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: call MatCreateMPIAIJ(PETSC_COMM_WORLD, Np, Np, Ng, Ng, & nmax, PETSC_NULL_INTEGER, nmax, PETSC_NULL_INTEGER, Jac, info) well, now I get: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Null argument, when expecting valid pointer! [1]PETSC ERROR: Null Object: Parameter # 2! how can I create the matrix without d_nnz and o_nnz? From jedbrown at mcs.anl.gov Sat Aug 27 13:57:33 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 27 Aug 2011 13:57:33 -0500 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: On Sat, Aug 27, 2011 at 13:55, Milan Mitrovic wrote: > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Null argument, when expecting valid pointer! > [1]PETSC ERROR: Null Object: Parameter # 2! > > > how can I create the matrix without d_nnz and o_nnz? Please paste the full error message. -------------- next part -------------- An HTML attachment was scrubbed... URL: From milan.v.mitrovic at gmail.com Sat Aug 27 14:20:43 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Sat, 27 Aug 2011 21:20:43 +0200 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: sorry, it was the local to global mapping... forgot to turn it on again... it seems that there is a problem with preallocation as I get one of these: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: New nonzero at (93,351) caused a malloc! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ppm_pf on a darwin10. named milan-mitrovics-macbook-pro.local by milan Sat Aug 27 21:13:10 2011 [0]PETSC ERROR: Libraries linked from /Users/milan/Work/Library/petsc/lib [0]PETSC ERROR: Configure run at Fri May 6 16:32:02 2011 [0]PETSC ERROR: Configure options --with-mpi=true --with-mpi-dir=/Users/milan/Work/Library/openmpi.gcc/ --with-fortran=true --with-sundials=true --with-sundials-dir=/Users/milan/Work/Library/sundials/ --prefix=/Users/milan/Work/Library/petsc [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatSetValues_MPIAIJ() line 358 in src/mat/impls/aij/mpi/mpiaij.c even though in: call MatSetValuesLocal(Jac,1,(/i/),nzr(i),ind(i,:),val(i,:), & ADD_VALUES,info) maxval(nzr) < nmax (the one used for preallocate)... after that I get: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Object is in wrong state! [1]PETSC ERROR: Not for unassembled matrix! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ./ppm_pf on a darwin10. named milan-mitrovics-macbook-pro.local by milan Sat Aug 27 21:13:10 2011 [1]PETSC ERROR: Libraries linked from /Users/milan/Work/Library/petsc/lib [1]PETSC ERROR: Configure run at Fri May 6 16:32:02 2011 [1]PETSC ERROR: Configure options --with-mpi=true --with-mpi-dir=/Users/milan/Work/Library/openmpi.gcc/ --with-fortran=true --with-sundials=true --with-sundials-dir=/Users/milan/Work/Library/sundials/ --prefix=/Users/milan/Work/Library/petsc [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: MatMult() line 1883 in src/mat/interface/matrix.c [1]PETSC ERROR: SNESLSCheckResidual_Private() line 60 in src/snes/impls/ls/ls.c [1]PETSC ERROR: SNESSolve_LS() line 210 in src/snes/impls/ls/ls.c [1]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 0.0236436 is less than relative tolerance 800695 times initial right hand side norm 0.0236436 at iteration 0 [0] SNESSolve_LS(): iter=0, linear solve iterations=0 From jedbrown at mcs.anl.gov Sat Aug 27 14:29:37 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 27 Aug 2011 14:29:37 -0500 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: On Sat, Aug 27, 2011 at 14:20, Milan Mitrovic wrote: > sorry, it was the local to global mapping... forgot to turn it on > again... it seems that there is a problem with preallocation as I get > one of these: > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Argument out of range! > [0]PETSC ERROR: New nonzero at (93,351) caused a malloc! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 > 13:37:48 CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ppm_pf on a darwin10. named > milan-mitrovics-macbook-pro.local by milan Sat Aug 27 21:13:10 2011 > [0]PETSC ERROR: Libraries linked from /Users/milan/Work/Library/petsc/lib > [0]PETSC ERROR: Configure run at Fri May 6 16:32:02 2011 > [0]PETSC ERROR: Configure options --with-mpi=true > --with-mpi-dir=/Users/milan/Work/Library/openmpi.gcc/ > --with-fortran=true --with-sundials=true > --with-sundials-dir=/Users/milan/Work/Library/sundials/ > --prefix=/Users/milan/Work/Library/petsc > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatSetValues_MPIAIJ() line 358 in > src/mat/impls/aij/mpi/mpiaij.c > > even though in: > > call MatSetValuesLocal(Jac,1,(/i/),nzr(i),ind(i,:),val(i,:), & > ADD_VALUES,info) > > maxval(nzr) < nmax (the one used for preallocate)... > Do you ever try to insert the same row more than once? E.g. perhaps the LocalToGlobalMapping is not injective (so there are two local indices that map to the same global index). > > after that I get: > > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Object is in wrong state! > [1]PETSC ERROR: Not for unassembled matrix! Did you forget to call MatAssemblyBegin() and MatAssemblyEnd() ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From likunt at andrew.cmu.edu Sat Aug 27 15:10:59 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Sat, 27 Aug 2011 16:10:59 -0400 Subject: [petsc-users] question about DA Message-ID: <4f38351aefc7751a7b71757eb6000518.squirrel@webmail.andrew.cmu.edu> Dear all, I have a code which works fine with uniprocessor, but it gets errors with 2 np. The errors come from the following part: DAGetCorners(da3D, &xs, &ys, &zs, &m, &n, &p); DAVecGetArray(da3D, Vtemp, &temp); for(k=zs; k References: <4f38351aefc7751a7b71757eb6000518.squirrel@webmail.andrew.cmu.edu> Message-ID: On Sat, Aug 27, 2011 at 15:10, Likun Tan wrote: > DAGetCorners(da2D, &xs, &ys, 0, &m, &n, 0); > DAVecGetArray(da2D, VS, &S); > for(j=ys; j for(i=xs; i S[j][i]=g(temp); //function evalution > How does the function g know how to index temp[k][j][i]? I.e. how does it get access to those variables? Is it a macro instead of a function? Note that the value of k is stale here because it will be equal to zs+p. > } > } > > f(temp) and g(temp) are functions of temp[k][j][i], when i assigned values > to H, there is no error; while when setting values to S using temp, it > gives errors, if using f(i,j,k) to assign values to S, it works fine. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From likunt at andrew.cmu.edu Sat Aug 27 15:24:11 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Sat, 27 Aug 2011 16:24:11 -0400 Subject: [petsc-users] question about DA In-Reply-To: References: <4f38351aefc7751a7b71757eb6000518.squirrel@webmail.andrew.cmu.edu> Message-ID: <3d7d8197c229180652f16f3edc327ba5.squirrel@webmail.andrew.cmu.edu> g(temp) could be sum of temp[num][j][i] where num from 1 to N, so the index k is specified. On Sat, August 27, 2011 4:14 pm, Jed Brown wrote: > On Sat, Aug 27, 2011 at 15:10, Likun Tan wrote: > > >> DAGetCorners(da2D, &xs, &ys, 0, &m, &n, 0); >> DAVecGetArray(da2D, VS, &S); >> for(j=ys; j> //function evalution >> >> > > How does the function g know how to index temp[k][j][i]? I.e. how does it > get access to those variables? Is it a macro instead of a function? Note > that the value of k is stale here because it will be equal to zs+p. > > >> } >> } >> >> >> f(temp) and g(temp) are functions of temp[k][j][i], when i assigned >> values to H, there is no error; while when setting values to S using >> temp, it gives errors, if using f(i,j,k) to assign values to S, it works >> fine. >> > From jedbrown at mcs.anl.gov Sat Aug 27 15:28:02 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 27 Aug 2011 15:28:02 -0500 Subject: [petsc-users] question about DA In-Reply-To: <3d7d8197c229180652f16f3edc327ba5.squirrel@webmail.andrew.cmu.edu> References: <4f38351aefc7751a7b71757eb6000518.squirrel@webmail.andrew.cmu.edu> <3d7d8197c229180652f16f3edc327ba5.squirrel@webmail.andrew.cmu.edu> Message-ID: On Sat, Aug 27, 2011 at 15:24, Likun Tan wrote: > g(temp) could be sum of temp[num][j][i] where num from 1 to N, so the > index k is specified. > It could be, but you still didn't explain how the indices i and j were made available to g() or what that code looks like. That is almost certainly your problem. You might want to try running with valgrind. Also, this is totally the wrong memory layout for performance (you want to sum over the last index, unfortunately this requires changing your interpretation of the x,y,z directions). -------------- next part -------------- An HTML attachment was scrubbed... URL: From likunt at andrew.cmu.edu Sat Aug 27 15:39:01 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Sat, 27 Aug 2011 16:39:01 -0400 Subject: [petsc-users] question about DA In-Reply-To: References: <4f38351aefc7751a7b71757eb6000518.squirrel@webmail.andrew.cmu.edu> <3d7d8197c229180652f16f3edc327ba5.squirrel@webmail.andrew.cmu.edu> Message-ID: <596ebe9a9e53ec91aa45b563d12fd01a.squirrel@webmail.andrew.cmu.edu> I made s simple test, if i assign S[j][i]=temp[2][1][1] for all j, i; it is no problem. while, when assign S[j][i]=temp[2][3][3] for all j, i, it gives me error. Your last statement means that i should use temp[j][i][k] if i want to sum over k? On Sat, August 27, 2011 4:28 pm, Jed Brown wrote: > On Sat, Aug 27, 2011 at 15:24, Likun Tan wrote: > > >> g(temp) could be sum of temp[num][j][i] where num from 1 to N, so the >> index k is specified. >> > > It could be, but you still didn't explain how the indices i and j were > made available to g() or what that code looks like. That is almost > certainly your problem. You might want to try running with valgrind. > > Also, this is totally the wrong memory layout for performance (you want > to sum over the last index, unfortunately this requires changing your > interpretation of the x,y,z directions). > From jedbrown at mcs.anl.gov Sat Aug 27 15:45:49 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 27 Aug 2011 15:45:49 -0500 Subject: [petsc-users] question about DA In-Reply-To: <596ebe9a9e53ec91aa45b563d12fd01a.squirrel@webmail.andrew.cmu.edu> References: <4f38351aefc7751a7b71757eb6000518.squirrel@webmail.andrew.cmu.edu> <3d7d8197c229180652f16f3edc327ba5.squirrel@webmail.andrew.cmu.edu> <596ebe9a9e53ec91aa45b563d12fd01a.squirrel@webmail.andrew.cmu.edu> Message-ID: On Sat, Aug 27, 2011 at 15:39, Likun Tan wrote: > > I made s simple test, > > if i assign S[j][i]=temp[2][1][1] for all j, i; it is no problem. > > while, when assign S[j][i]=temp[2][3][3] for all j, i, it gives me error. > What are m,n,p the first time you call it and what are those values the second time? > > Your last statement means that i should use temp[j][i][k] if i want to sum > over k? > You can't just change the meaning of the indices when you index. Once you fix this problem, feel confident that you understand why, and want to make the memory access better for performance, you can look at the calls to DMDACreate3d() and the compatible DMDACreate2d() in src/snes/examples/tutorials/ex48.c. Also look at the calls to DMDAGetCorners() which changes the meaning of x,y,z relative to what PETSc uses internally. Unfortunately this can be somewhat error-prone, so you should understand why things work the way they do, otherwise it's too easy to get confused. -------------- next part -------------- An HTML attachment was scrubbed... URL: From likunt at andrew.cmu.edu Sat Aug 27 15:49:38 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Sat, 27 Aug 2011 16:49:38 -0400 Subject: [petsc-users] question about DA In-Reply-To: References: <4f38351aefc7751a7b71757eb6000518.squirrel@webmail.andrew.cmu.edu> <3d7d8197c229180652f16f3edc327ba5.squirrel@webmail.andrew.cmu.edu> <596ebe9a9e53ec91aa45b563d12fd01a.squirrel@webmail.andrew.cmu.edu> Message-ID: Thanks for your explanation. I attached my code again, p was used in defining temp and H, when defining S, we don't need p and zs: DAGetCorners(da3D, &xs, &ys, &zs, &m, &n, &p); DAVecGetArray(da3D, Vtemp, &temp); for(k=zs; k On Sat, Aug 27, 2011 at 15:39, Likun Tan wrote: > > >> >> I made s simple test, >> >> >> if i assign S[j][i]=temp[2][1][1] for all j, i; it is no problem. >> >> while, when assign S[j][i]=temp[2][3][3] for all j, i, it gives me >> error. >> > > What are m,n,p the first time you call it and what are those values the > second time? > > >> >> Your last statement means that i should use temp[j][i][k] if i want to >> sum over k? >> > > You can't just change the meaning of the indices when you index. Once you > fix this problem, feel confident that you understand why, and want to > make the memory access better for performance, you can look at the calls > to DMDACreate3d() and the compatible DMDACreate2d() in > src/snes/examples/tutorials/ex48.c. Also look at the calls to > DMDAGetCorners() which changes the meaning of x,y,z relative to what > PETSc > uses internally. > > Unfortunately this can be somewhat error-prone, so you should understand > why things work the way they do, otherwise it's too easy to get confused. > From jedbrown at mcs.anl.gov Sat Aug 27 15:53:58 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 27 Aug 2011 15:53:58 -0500 Subject: [petsc-users] question about DA In-Reply-To: References: <4f38351aefc7751a7b71757eb6000518.squirrel@webmail.andrew.cmu.edu> <3d7d8197c229180652f16f3edc327ba5.squirrel@webmail.andrew.cmu.edu> <596ebe9a9e53ec91aa45b563d12fd01a.squirrel@webmail.andrew.cmu.edu> Message-ID: On Sat, Aug 27, 2011 at 15:49, Likun Tan wrote: > I attached my code again, p was used in defining temp and H, when defining > S, we don't need p and zs: > You need p to know the limits of the integral. You need that the domain not be decomposed in the z direction if you are going to integrate without doing communication. You still haven't shown us how the DAs were constructed or what the returned values for (xs,ys,zs,m,n,p) were when got them from da3D and what (xs,ys,m,n) were when you got them from da2D. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at ibrae.ac.ru Sat Aug 27 15:58:22 2011 From: ram at ibrae.ac.ru (=?UTF-8?B?0JDQu9C10LrRgdC10Lkg0KDRj9C30LDQvdC+0LI=?=) Date: Sun, 28 Aug 2011 00:58:22 +0400 Subject: [petsc-users] performance issue solving multiple linear systems of the same size with the different preconditioning methods In-Reply-To: References: Message-ID: So. All memory and time issues appeared because of my KSPConvergedTest function, which calls KSPBuildResidual to monitor true residual. KSPBuildResidual incorrectly works with memory. As soon as I've commented the line with KSPBuildResidual, all troubles gone. So now I use default monitor and have to monitor preconditioned residual. This is inconvenient for me, but I cant figure out, how can I set up using true monitor for all methods without any memory or performance leaks. I'll try to get used t deal with preconditioned norm. Thank you very much for taking the time to reply to me. -- Best regards, Alexey Ryazanov ______________________________________ Nuclear Safety Institute of Russian Academy of Sciences 27 ??????? 2011 ?. 17:32 ???????????? ??????? ??????? ???????: > Oh BOY! It was all fault of my MyKSPConvergedTest. > > 27 ??????? 2011 ?. 1:29 ???????????? ??????? ??????? ???????: > > I have also checked KSPSolve behavior in my other PETSc programs and found >> the same memory lack >> >> 27 ??????? 2011 ?. 0:57 ???????????? ??????? ??????? ???????: >> >> >>> Thank you for your response! >>> >>> I have the memory leak in both my programs. But I don't create plenty of >>> vectors. >>> >>> My code looks like: >>> >>> ***INIT_ALL*** >>> PetscLogStageRegister("Iteration :", &StageNum1); >>> PetscLogStagePush(StageNum1); >>> KSPSolve(dKSP, dvec_origRHS, dvec_Solution); >>> PetscLogStagePop(); >>> ***DESTROY_ALL*** >>> >>> >>> And when I comment (or delete) KSPSolve, the log_summary output is: >>> >>> ===================================================== >>> Memory usage is given in bytes: >>> >>> Creations Destructions Memory Descendants' Mem Object Type >>> >>> Reports information only for process 0. >>> >>> --- Event Stage 0: Main Stage >>> 1 1 729472 0 Application Order >>> 1 1 225452 0 Distributed array >>> 8 8 1533424 0 Vec >>> 3 3 2604 0 Vec Scatter >>> 8 8 613852 0 Index Set >>> 1 1 221304 0 IS L to G Mapping >>> 3 3 16603440 0 Matrix >>> 1 1 832 0 Krylov Solver >>> 1 1 688 0 Preconditioner >>> 1 1 448 0 PetscRandom >>> >>> --- Event Stage 1: Iteration : >>> ===================================================== >>> >>> When I run the code with KSPSolve instruction, it gives me: >>> >>> ===================================================== >>> Memory usage is given in bytes: >>> Creations Destructions Memory Descendants' Mem Object Type >>> >>> Reports information only for process 0. >>> >>> --- Event Stage 0: Main Stage >>> 1 0 0 0 Application Order >>> 1 0 0 0 Distributed array >>> 8 17 4963592 0 Vec >>> 3 2 1736 0 Vec Scatter >>> 8 12 1425932 0 Index Set >>> 1 0 0 0 IS L to G Mapping >>> 3 5 50158132 0 Matrix >>> 1 2 1664 0 Krylov Solver >>> 1 2 1440 0 Preconditioner >>> 1 1 448 0 PetscRandom >>> 0 1 544 0 Viewer >>> >>> --- Event Stage 1: Iteration : >>> 355 173 64692312 0 Vec >>> 1 0 0 0 Vec Scatter >>> 6 2 1024 0 Index Set >>> 2 0 0 0 Matrix >>> 1 0 0 0 Krylov Solver >>> 1 0 0 0 Preconditioner >>> 2 1 544 0 Viewer >>> ===================================================== >>> >>> >>> >>> >>> 2011/8/25 Jed Brown >>> >>>> On Tue, Aug 23, 2011 at 02:37, ??????? ??????? wrote: >>>> >>>>> When i delete the 4-5-6 part of 2nd, 1-2-3 works great! with exact like >>>>> 1st results. >>>>> When i delete the 1-2-3 part of 2nd, 4-5-6 works great! with exact like >>>>> 1st results. >>>>> All program (1-2-3-4-5-6) works badly. >>>>> >>>> >>>> From the -log_summary, you have a memory leak (many more vector >>>> creations than destructions). Try running with -malloc_dump to debug it. >>>> Perhaps you are creating a vector every time one of your functions is >>>> called? You should also build --with-debugging=0 when looking at timing >>>> results. (You can keep it in PETSC_ARCH=linux-gnu-opt.) >>>> >>> >>> >>> >>> -- >>> Best regards, >>> Alexey Ryazanov >>> ______________________________________ >>> Nuclear Safety Institute of Russian Academy of Sciences >>> >>> >>> >> >> >> -- >> Best regards, >> Alexey Ryazanov >> ______________________________________ >> Nuclear Safety Institute of Russian Academy of Sciences >> >> >> > > > -- > Best regards, > Alexey Ryazanov > ______________________________________ > Nuclear Safety Institute of Russian Academy of Sciences > > > -- Best regards, Alexey Ryazanov ______________________________________ Nuclear Safety Institute of Russian Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Aug 27 16:02:46 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 27 Aug 2011 16:02:46 -0500 Subject: [petsc-users] performance issue solving multiple linear systems of the same size with the different preconditioning methods In-Reply-To: References: Message-ID: <977C5736-D992-4C44-88C3-29196D59CBE0@mcs.anl.gov> You can use -ksp_monitor_true_residual from the command line to always monitor the unpreconditioned residuals, no memory leaks. But this would be an unfair test for timing because 1) having ASCII IO screws up timing results 2) some methods like left preconditioned GMRES get the preconditioned residual "for free" in that it is needed in the algorithm but if you monitor the true residual this introduces a lot of additional operations (which are not needed for the linear solve) making the linear solver look slower than it really is. Barry On Aug 27, 2011, at 3:58 PM, ??????? ??????? wrote: > So. All memory and time issues appeared because of my KSPConvergedTest function, which calls KSPBuildResidual to monitor true residual. KSPBuildResidual incorrectly works with memory. As soon as I've commented the line with KSPBuildResidual, all troubles gone. So now I use default monitor and have to monitor preconditioned residual. This is inconvenient for me, but I cant figure out, how can I set up using true monitor for all methods without any memory or performance leaks. I'll try to get used t deal with preconditioned norm. > > Thank you very much for taking the time to reply to me. > > -- > Best regards, > Alexey Ryazanov > ______________________________________ > Nuclear Safety Institute of Russian Academy of Sciences > > > 27 ??????? 2011 ?. 17:32 ???????????? ??????? ??????? ???????: > Oh BOY! It was all fault of my MyKSPConvergedTest. > > 27 ??????? 2011 ?. 1:29 ???????????? ??????? ??????? ???????: > > I have also checked KSPSolve behavior in my other PETSc programs and found the same memory lack > > 27 ??????? 2011 ?. 0:57 ???????????? ??????? ??????? ???????: > > > Thank you for your response! > > I have the memory leak in both my programs. But I don't create plenty of vectors. > > My code looks like: > > ***INIT_ALL*** > PetscLogStageRegister("Iteration :", &StageNum1); > PetscLogStagePush(StageNum1); > KSPSolve(dKSP, dvec_origRHS, dvec_Solution); > PetscLogStagePop(); > ***DESTROY_ALL*** > > And when I comment (or delete) KSPSolve, the log_summary output is: > > ===================================================== > Memory usage is given in bytes: > > Creations Destructions Memory Descendants' Mem Object Type > Reports information only for process 0. > > --- Event Stage 0: Main Stage > 1 1 729472 0 Application Order > 1 1 225452 0 Distributed array > 8 8 1533424 0 Vec > 3 3 2604 0 Vec Scatter > 8 8 613852 0 Index Set > 1 1 221304 0 IS L to G Mapping > 3 3 16603440 0 Matrix > 1 1 832 0 Krylov Solver > 1 1 688 0 Preconditioner > 1 1 448 0 PetscRandom > > --- Event Stage 1: Iteration : > ===================================================== > > When I run the code with KSPSolve instruction, it gives me: > > ===================================================== > Memory usage is given in bytes: > Creations Destructions Memory Descendants' Mem Object Type > Reports information only for process 0. > > --- Event Stage 0: Main Stage > 1 0 0 0 Application Order > 1 0 0 0 Distributed array > 8 17 4963592 0 Vec > 3 2 1736 0 Vec Scatter > 8 12 1425932 0 Index Set > 1 0 0 0 IS L to G Mapping > 3 5 50158132 0 Matrix > 1 2 1664 0 Krylov Solver > 1 2 1440 0 Preconditioner > 1 1 448 0 PetscRandom > 0 1 544 0 Viewer > > --- Event Stage 1: Iteration : > 355 173 64692312 0 Vec > 1 0 0 0 Vec Scatter > 6 2 1024 0 Index Set > 2 0 0 0 Matrix > 1 0 0 0 Krylov Solver > 1 0 0 0 Preconditioner > 2 1 544 0 Viewer > ===================================================== > > > > 2011/8/25 Jed Brown > On Tue, Aug 23, 2011 at 02:37, ??????? ??????? wrote: > When i delete the 4-5-6 part of 2nd, 1-2-3 works great! with exact like 1st results. > When i delete the 1-2-3 part of 2nd, 4-5-6 works great! with exact like 1st results. > All program (1-2-3-4-5-6) works badly. > > From the -log_summary, you have a memory leak (many more vector creations than destructions). Try running with -malloc_dump to debug it. Perhaps you are creating a vector every time one of your functions is called? You should also build --with-debugging=0 when looking at timing results. (You can keep it in PETSC_ARCH=linux-gnu-opt.) > > > > -- > Best regards, > Alexey Ryazanov > ______________________________________ > Nuclear Safety Institute of Russian Academy of Sciences > > > > > > -- > Best regards, > Alexey Ryazanov > ______________________________________ > Nuclear Safety Institute of Russian Academy of Sciences > > > > > > -- > Best regards, > Alexey Ryazanov > ______________________________________ > Nuclear Safety Institute of Russian Academy of Sciences > > > > > > -- > Best regards, > Alexey Ryazanov > ______________________________________ > Nuclear Safety Institute of Russian Academy of Sciences > > From jedbrown at mcs.anl.gov Sat Aug 27 16:06:53 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 27 Aug 2011 16:06:53 -0500 Subject: [petsc-users] performance issue solving multiple linear systems of the same size with the different preconditioning methods In-Reply-To: References: Message-ID: 2011/8/27 ??????? ??????? > So. All memory and time issues appeared because of my KSPConvergedTest > function, which calls KSPBuildResidual to monitor true > residual. KSPBuildResidual incorrectly works with memory. As soon as I've > commented the line with KSPBuildResidual, all troubles gone. So now I use > default monitor and have to monitor preconditioned residual. This is > inconvenient for me, but I cant figure out, how can I set up using true > monitor for all methods without any memory or performance leaks. I'll try to > get used t deal with preconditioned norm. 1. You can actually run the Krylov method with unpreconditioned residuals by using -ksp_right_pc (some output is inconsistent with 3.1 this way, it's fixed in petsc-dev and you can just use -ksp_norm_type unpreconditioned). This is inexpensive and does the right thing. 2. You can run the Krylov method in the preconditioned norm, but monitor the true residual using -ksp_monitor_true_residual. The code is in src/ksp/ksp/interface/iterativ.c KSPMonitorTrueResidualNorm(). You can also use this code as a template to make a similar custom monitor without memory leaks. This calls KSPBuildResidual() which is very expensive for GMRES, but is cheap for some other methods (including GCR which is closely related). -------------- next part -------------- An HTML attachment was scrubbed... URL: From likunt at andrew.cmu.edu Sat Aug 27 16:09:54 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Sat, 27 Aug 2011 17:09:54 -0400 Subject: [petsc-users] question about DA Message-ID: the size of temp and H is 9*9*201, the size of S is 9*9. I used 2 processors, the returned values are temp: 0 0 0 9 9 101 temp: 0 0 101 9 9 100 H: 0 0 101 9 9 100 S: 0 5 101 9 4 100 I am not sure what you mean by how DAs were constructed, the command i used for construction is: DACreate2d(PETSC_COMM_WORLD, DA_NONPERIODIC, DA_STENCIL_STAR, 9, 9, PETSC_DECIDE, PETSC_DECIDE, 1, 0, PETSC_NULL, PETSC_NULL, &da2D); DACreate3d(PETSC_COMM_WORLD, DA_NONPERIODIC, DA_STENCIL_STAR, 9, 9, 201, PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, 1, 0, PETSC_NULL, PETSC_NULL, PETSC_NULL, &da3D); i got some sense of the problem after watching m, n, p, but i don't know how to avoid it. On Sat, August 27, 2011 4:53 pm, Jed Brown wrote: > On Sat, Aug 27, 2011 at 15:49, Likun Tan wrote: > > >> I attached my code again, p was used in defining temp and H, when >> defining S, we don't need p and zs: >> >> > > You need p to know the limits of the integral. You need that the domain > not be decomposed in the z direction if you are going to integrate without > doing communication. You still haven't shown us how the DAs were > constructed or what the returned values for (xs,ys,zs,m,n,p) were when got > them from da3D and what (xs,ys,m,n) were when you got them from da2D. > From bsmith at mcs.anl.gov Sat Aug 27 16:16:02 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 27 Aug 2011 16:16:02 -0500 Subject: [petsc-users] question about DA In-Reply-To: References: Message-ID: You cannot just split a 2d array among some processes and a 3d array across the same processes and expect somehow that the layouts of two of the indices will match between the two arrays. You can run, for example with -da_view to see how they two arrays are being laid out in parallel by default. You will need set certain values in the m,n, and p arguments to DMDACreate2d() and 3d to get layouts that match. Note that these are not the M,N, and P arguments. Barry On Aug 27, 2011, at 4:09 PM, Likun Tan wrote: > the size of temp and H is 9*9*201, the size of S is 9*9. > > I used 2 processors, the returned values are > temp: 0 0 0 9 9 101 > temp: 0 0 101 9 9 100 > H: 0 0 101 9 9 100 > S: 0 5 101 9 4 100 > > I am not sure what you mean by how DAs were constructed, the command i > used for construction is: > DACreate2d(PETSC_COMM_WORLD, DA_NONPERIODIC, DA_STENCIL_STAR, 9, 9, > PETSC_DECIDE, PETSC_DECIDE, 1, 0, PETSC_NULL, PETSC_NULL, &da2D); > DACreate3d(PETSC_COMM_WORLD, DA_NONPERIODIC, DA_STENCIL_STAR, 9, 9, 201, > PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, 1, 0, PETSC_NULL, PETSC_NULL, > PETSC_NULL, &da3D); > > i got some sense of the problem after watching m, n, p, but i don't know > how to avoid it. > > On Sat, August 27, 2011 4:53 pm, Jed Brown wrote: >> On Sat, Aug 27, 2011 at 15:49, Likun Tan wrote: >> >> >>> I attached my code again, p was used in defining temp and H, when >>> defining S, we don't need p and zs: >>> >>> >> >> You need p to know the limits of the integral. You need that the domain >> not be decomposed in the z direction if you are going to integrate without >> doing communication. You still haven't shown us how the DAs were >> constructed or what the returned values for (xs,ys,zs,m,n,p) were when got >> them from da3D and what (xs,ys,m,n) were when you got them from da2D. >> > > > > From jedbrown at mcs.anl.gov Sat Aug 27 16:16:32 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 27 Aug 2011 16:16:32 -0500 Subject: [petsc-users] question about DA In-Reply-To: References: Message-ID: On Sat, Aug 27, 2011 at 16:09, Likun Tan wrote: > the size of temp and H is 9*9*201, the size of S is 9*9. > > I used 2 processors, the returned values are > temp: 0 0 0 9 9 101 > temp: 0 0 101 9 9 100 > temp is partitioned in the z direction. Each processor holds the full 9x9 cross-section of the "rod". > H: 0 0 101 9 9 100 > S: 0 5 101 9 4 100 > The 9x9 cross-section is partitioned. The value of zs and p are just left over from the earlier call, so they don't mean anything here. > > I am not sure what you mean by how DAs were constructed, the command i > used for construction is: > DACreate2d(PETSC_COMM_WORLD, DA_NONPERIODIC, DA_STENCIL_STAR, 9, 9, > PETSC_DECIDE, PETSC_DECIDE, 1, 0, PETSC_NULL, PETSC_NULL, &da2D); > DACreate3d(PETSC_COMM_WORLD, DA_NONPERIODIC, DA_STENCIL_STAR, 9, 9, 201, > PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, 1, 0, PETSC_NULL, PETSC_NULL, > PETSC_NULL, &da3D); > You can prevent PETSc from partitioning in the z-direction by changing the "p" parameter from PETSC_DECIDE to 1: DACreate3d(PETSC_COMM_WORLD, DA_NONPERIODIC, DA_STENCIL_STAR, 9, 9, 201, PETSC_DECIDE, PETSC_DECIDE, 1, 1, 0, PETSC_NULL, PETSC_NULL, PETSC_NULL, &da3D); -------------- next part -------------- An HTML attachment was scrubbed... URL: From likunt at andrew.cmu.edu Sat Aug 27 16:35:53 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Sat, 27 Aug 2011 17:35:53 -0400 Subject: [petsc-users] question about DA In-Reply-To: References: Message-ID: Thanks. What is the difference between DACreate2d() and DMDACreate2d()? I found the definitions of these two are the same. On Sat, August 27, 2011 5:16 pm, Barry Smith wrote: > > You cannot just split a 2d array among some processes and a 3d array > across the same processes and expect somehow that the layouts of two of > the indices will match between the two arrays. > > You can run, for example with -da_view to see how they two arrays are > being laid out in parallel by default. You will need set certain values > in the m,n, and p arguments to DMDACreate2d() and 3d to get layouts that > match. Note that these are not the M,N, and P arguments. > > Barry > > > > On Aug 27, 2011, at 4:09 PM, Likun Tan wrote: > > >> the size of temp and H is 9*9*201, the size of S is 9*9. >> >> I used 2 processors, the returned values are >> temp: 0 0 0 9 9 101 >> temp: 0 0 101 9 9 100 >> H: 0 0 101 9 9 100 >> S: 0 5 101 9 4 100 >> >> >> I am not sure what you mean by how DAs were constructed, the command i >> used for construction is: DACreate2d(PETSC_COMM_WORLD, DA_NONPERIODIC, >> DA_STENCIL_STAR, 9, 9, >> PETSC_DECIDE, PETSC_DECIDE, 1, 0, PETSC_NULL, PETSC_NULL, &da2D); >> DACreate3d(PETSC_COMM_WORLD, DA_NONPERIODIC, DA_STENCIL_STAR, 9, 9, 201, >> PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, 1, 0, PETSC_NULL, >> PETSC_NULL, >> PETSC_NULL, &da3D); >> >> >> i got some sense of the problem after watching m, n, p, but i don't >> know how to avoid it. >> >> On Sat, August 27, 2011 4:53 pm, Jed Brown wrote: >> >>> On Sat, Aug 27, 2011 at 15:49, Likun Tan >>> wrote: >>> >>> >>> >>>> I attached my code again, p was used in defining temp and H, when >>>> defining S, we don't need p and zs: >>>> >>>> >>> >>> You need p to know the limits of the integral. You need that the >>> domain not be decomposed in the z direction if you are going to >>> integrate without doing communication. You still haven't shown us how >>> the DAs were constructed or what the returned values for >>> (xs,ys,zs,m,n,p) were when got >>> them from da3D and what (xs,ys,m,n) were when you got them from da2D. >>> >> >> >> >> > > > From jedbrown at mcs.anl.gov Sat Aug 27 16:37:42 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 27 Aug 2011 16:37:42 -0500 Subject: [petsc-users] question about DA In-Reply-To: References: Message-ID: On Sat, Aug 27, 2011 at 16:35, Likun Tan wrote: > What is the difference between DACreate2d() and DMDACreate2d()? I found > the definitions of these two are the same. > DMDACreate2d is the name in petsc-dev. It is functionally the same except that specification of boundaries is more flexible. -------------- next part -------------- An HTML attachment was scrubbed... URL: From likunt at andrew.cmu.edu Sat Aug 27 16:49:33 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Sat, 27 Aug 2011 17:49:33 -0400 Subject: [petsc-users] question about DA In-Reply-To: References: Message-ID: <6da02e97298f77cff58812768d1668ae.squirrel@webmail.andrew.cmu.edu> I c, thanks. So for my problem, if i want to set values to a 3D array and then to a 2D array, where the 2D array uses the values of 3D array, can i use DAGetCorners(da3D, &x, &y, &z, &m, &n, &p) to assign values to the 3D array, and then DAGetCorners(da2D, &x, &y, 0, &m, &n, 0)? I am sorry i still don't get what my problem is, i though when setting values to 2D, x, y, m and n will be reset, so the previous usage of the corner indices and width won't influence the later use. On Sat, August 27, 2011 5:37 pm, Jed Brown wrote: > On Sat, Aug 27, 2011 at 16:35, Likun Tan wrote: > > >> What is the difference between DACreate2d() and DMDACreate2d()? I found >> the definitions of these two are the same. >> > > DMDACreate2d is the name in petsc-dev. It is functionally the same except > that specification of boundaries is more flexible. > From jedbrown at mcs.anl.gov Sat Aug 27 16:54:44 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 27 Aug 2011 16:54:44 -0500 Subject: [petsc-users] question about DA In-Reply-To: <6da02e97298f77cff58812768d1668ae.squirrel@webmail.andrew.cmu.edu> References: <6da02e97298f77cff58812768d1668ae.squirrel@webmail.andrew.cmu.edu> Message-ID: On Sat, Aug 27, 2011 at 16:49, Likun Tan wrote: > I am sorry i still don't get what my problem is, i though when setting > values to 2D, x, y, m and n will be reset, so the previous usage of the > corner indices and width won't influence the later use. > The whole array is not duplicated onto every process, just the owned part and perhaps ghosts. You have to make sure that the decompositions are compatible and that you only index into the part of the array that your process has access to. This is really unlikely to happen "by accident", so you have to decide how you want the total volume decomposed and which parts you want local. Part of this (typically, if you are going to integrate) is to not decompose at all in the z direction. Once you ensure that everything is compatible, you can have one accessor that gives you the array bounds for the 3D part and use the 2D part in a compatible way. -------------- next part -------------- An HTML attachment was scrubbed... URL: From likunt at andrew.cmu.edu Sat Aug 27 17:09:06 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Sat, 27 Aug 2011 18:09:06 -0400 Subject: [petsc-users] question about DA In-Reply-To: References: <6da02e97298f77cff58812768d1668ae.squirrel@webmail.andrew.cmu.edu> Message-ID: <7e85753924c81a19810e743c0c6fefe4.squirrel@webmail.andrew.cmu.edu> Okay, i c, thanks. So if i want to sum over the z direction, it is better not to partition in z direction, i just tried and it works. Another question is, if the size is small in x and y directions, and large in z direction, i want more of processors, but the number is restricted to the size in x and y direction, since partition is not allowed in z direction, any way to improve efficiency? On Sat, August 27, 2011 5:54 pm, Jed Brown wrote: > On Sat, Aug 27, 2011 at 16:49, Likun Tan wrote: > > >> I am sorry i still don't get what my problem is, i though when setting >> values to 2D, x, y, m and n will be reset, so the previous usage of the >> corner indices and width won't influence the later use. >> > > The whole array is not duplicated onto every process, just the owned part > and perhaps ghosts. You have to make sure that the decompositions are > compatible and that you only index into the part of the array that your > process has access to. This is really unlikely to happen "by accident", > so you have to decide how you want the total volume decomposed and which > parts you want local. Part of this (typically, if you are going to > integrate) is to not decompose at all in the z direction. Once you ensure > that everything is compatible, you can have one accessor that gives you > the array bounds for the 3D part and use the 2D part in a compatible way. > From jedbrown at mcs.anl.gov Sat Aug 27 22:57:56 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 27 Aug 2011 22:57:56 -0500 Subject: [petsc-users] question about DA In-Reply-To: <7e85753924c81a19810e743c0c6fefe4.squirrel@webmail.andrew.cmu.edu> References: <6da02e97298f77cff58812768d1668ae.squirrel@webmail.andrew.cmu.edu> <7e85753924c81a19810e743c0c6fefe4.squirrel@webmail.andrew.cmu.edu> Message-ID: On Sat, Aug 27, 2011 at 17:09, Likun Tan wrote: > Another question is, if the size is small in x and y directions, and large > in z direction, i want more of processors, but the number is restricted to > the size in x and y direction, since partition is not allowed in z > direction, any way to improve efficiency? > This is a fundamental problem. If there is tight coupling in the z direction, then it is algorithmically good to keep it on a single process, but it is not good for parallel scalability because you are not allowed to decompose in that direction. You can write a new integral function with the z direction decomposed, but it's significantly more complicated. The practical solution is usually to not decompose in z and pay the price of somewhat reduced parallel scalability in exchange for simplicity, algorithmic performance, and efficiency. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at ibrae.ac.ru Sun Aug 28 09:31:50 2011 From: ram at ibrae.ac.ru (=?UTF-8?B?0JDQu9C10LrRgdC10Lkg0KDRj9C30LDQvdC+0LI=?=) Date: Sun, 28 Aug 2011 18:31:50 +0400 Subject: [petsc-users] performance issue solving multiple linear systems of the same size with the different preconditioning methods In-Reply-To: References: Message-ID: Thank you very much for your advice! -- Best regards, Alexey Ryazanov ______________________________________ Nuclear Safety Institute of Russian Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From marek.schmitt at yahoo.com Mon Aug 29 05:37:28 2011 From: marek.schmitt at yahoo.com (Marek Schmitt) Date: Mon, 29 Aug 2011 03:37:28 -0700 (PDT) Subject: [petsc-users] PETSc on unstructured meshes / Sieve In-Reply-To: <02D1A9FDD7FB4F80AD905FF9E7522409@cogendaeda> References: <1314192301.9656.YahooMailNeo@web125116.mail.ne1.yahoo.com> <1314270688.63732.YahooMailNeo@web125102.mail.ne1.yahoo.com> <02D1A9FDD7FB4F80AD905FF9E7522409@cogendaeda> Message-ID: <1314614248.35231.YahooMailNeo@web125105.mail.ne1.yahoo.com> Hello Gong Ding, thank you very much for sharing your work on FVM support for libmesh. I have not been able to find this part in libmesh. Is it available as a separate patch to the libmesh sources? Thanks Marek ----- Original Message ----- From: Gong Ding To: Marek Schmitt ; PETSc users list Cc: Sent: Thursday, August 25, 2011 6:46 PM Subject: Re: [petsc-users] PETSc on unstructured meshes / Sieve I had added vertex based FVM support to libmesh, for my semiconductor simulator, not very difficult. In the year of 2007, I had spent about one month to read though the source code of libmesh. And finally known how to write 3D, parallel numerical code based on unstructured grid. How to use OO to describe different mesh cells (tri, quad, tet, prism, hex...) with an uniform interface. How to organize the data structure of unstrctured mesh cells in memory. How to partition mesh between processsors. How to mapping variables to petsc vec and mat. How to synchronous values on ghost node/cell How to assemble petsc vec and mat. Many many issues. From milan.v.mitrovic at gmail.com Mon Aug 29 06:39:29 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Mon, 29 Aug 2011 13:39:29 +0200 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: Ok... I had a problem with indexing (-1 offset) in the local to global mapping and now I get the following: Testing hand-coded Jacobian, if the ratio is O(1.e-8), the hand-coded Jacobian is probably correct. Run with -snes_test_display to show difference of hand-coded and finite difference Jacobian. Norm of matrix ratio 3.24883e-07 difference 3.24883e-05 Norm of matrix ratio 3.25662e-07 difference 3.25662e-05 Norm of matrix ratio 3.26517e-07 difference 3.26517e-05 [0]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c so I guess this means that it is slightly wrong :) can you explain now what it is doing here (and why I get this error)... can I make it try to solve the equation with the approximated fd jacobian (is it doing that already?) From bsmith at mcs.anl.gov Mon Aug 29 07:36:24 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 29 Aug 2011 07:36:24 -0500 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: <68E17489-BE71-4D60-A225-FA8AF0FA22F0@mcs.anl.gov> Did you run with -snes_test_display it will display the matrices computed both ways so you can see which entries are different and maybe track down the problem. Barry On Aug 29, 2011, at 6:39 AM, Milan Mitrovic wrote: > Ok... I had a problem with indexing (-1 offset) in the local to global > mapping and now I get the following: > > Testing hand-coded Jacobian, if the ratio is > O(1.e-8), the hand-coded Jacobian is probably correct. > Run with -snes_test_display to show difference > of hand-coded and finite difference Jacobian. > Norm of matrix ratio 3.24883e-07 difference 3.24883e-05 > Norm of matrix ratio 3.25662e-07 difference 3.25662e-05 > Norm of matrix ratio 3.26517e-07 difference 3.26517e-05 > [0]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c > > so I guess this means that it is slightly wrong :) > > can you explain now what it is doing here (and why I get this > error)... can I make it try to solve the equation with the > approximated fd jacobian (is it doing that already?) From jedbrown at mcs.anl.gov Mon Aug 29 07:43:08 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 29 Aug 2011 07:43:08 -0500 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: On Mon, Aug 29, 2011 at 06:39, Milan Mitrovic wrote: > so I guess this means that it is slightly wrong :) > Or your system is somewhat ill-conditioned. If you add '-mat_fd_type ds' to the last command line, are the errors similar or improve a bit? A different way to test for a correct Jacobian is to run with -snes_mf_operator -pc_type lu and check that it converges in 1 or occasionally 2 iterations. This also helps if the Jacobian is incorrect, but not visibly so on the first iteration (e.g. because nonlinearities are silent for the initial guess). > can you explain now what it is doing here (and why I get this > error)... > The error to exit early because "test" is not intended to actually solve the equation, -snes_type test is supposed to do that. > can I make it try to solve the equation with the > approximated fd jacobian (is it doing that already?) > -snes_mf_operator : solve with matrix-free finite differenced Jacobian, preconditioned using the matrix you assembled -snes_fd : assemble the dense FD operator (same as created by -snes_type test) and solve using that -------------- next part -------------- An HTML attachment was scrubbed... URL: From likunt at andrew.cmu.edu Mon Aug 29 09:40:47 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Mon, 29 Aug 2011 10:40:47 -0400 Subject: [petsc-users] about ksp Message-ID: <97a57c54262a0f2deb483408a9e7e0f1.squirrel@webmail.andrew.cmu.edu> Dear all, I have linear systems Ax=b with different right-hand-side vectors. These vectors are independent. Could i generate b simultaneously and solve Ax=b simultaneously? best, Likun From knepley at gmail.com Mon Aug 29 09:47:14 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 29 Aug 2011 14:47:14 +0000 Subject: [petsc-users] about ksp In-Reply-To: <97a57c54262a0f2deb483408a9e7e0f1.squirrel@webmail.andrew.cmu.edu> References: <97a57c54262a0f2deb483408a9e7e0f1.squirrel@webmail.andrew.cmu.edu> Message-ID: On Mon, Aug 29, 2011 at 2:40 PM, Likun Tan wrote: > Dear all, > > I have linear systems Ax=b with different right-hand-side vectors. These > vectors are independent. Could i generate b simultaneously and solve Ax=b > simultaneously? > Unfortunately, no one has figured out how to exploit this for Krylov methods. Matt > best, > Likun > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Aug 29 09:52:44 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 29 Aug 2011 09:52:44 -0500 (CDT) Subject: [petsc-users] about ksp In-Reply-To: References: <97a57c54262a0f2deb483408a9e7e0f1.squirrel@webmail.andrew.cmu.edu> Message-ID: On Mon, 29 Aug 2011, Matthew Knepley wrote: > On Mon, Aug 29, 2011 at 2:40 PM, Likun Tan wrote: > > > Dear all, > > > > I have linear systems Ax=b with different right-hand-side vectors. These > > vectors are independent. Could i generate b simultaneously and solve Ax=b > > simultaneously? > > > > Unfortunately, no one has figured out how to exploit this for Krylov > methods. > Also - check ksp/ksp/examples/tutorials/ex16.c Satish From likunt at andrew.cmu.edu Mon Aug 29 09:59:41 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Mon, 29 Aug 2011 10:59:41 -0400 Subject: [petsc-users] about ksp In-Reply-To: References: <97a57c54262a0f2deb483408a9e7e0f1.squirrel@webmail.andrew.cmu.edu> Message-ID: Thanks, I read ex16.c, it seems that the two linear equations are solved sequentially? On Mon, August 29, 2011 10:52 am, Satish Balay wrote: > On Mon, 29 Aug 2011, Matthew Knepley wrote: > > >> On Mon, Aug 29, 2011 at 2:40 PM, Likun Tan >> wrote: >> >> >>> Dear all, >>> >>> >>> I have linear systems Ax=b with different right-hand-side vectors. >>> These >>> vectors are independent. Could i generate b simultaneously and solve >>> Ax=b >>> simultaneously? >>> >> >> Unfortunately, no one has figured out how to exploit this for Krylov >> methods. >> > > Also - check ksp/ksp/examples/tutorials/ex16.c > > > Satish > > > From knepley at gmail.com Mon Aug 29 10:08:39 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 29 Aug 2011 15:08:39 +0000 Subject: [petsc-users] about ksp In-Reply-To: References: <97a57c54262a0f2deb483408a9e7e0f1.squirrel@webmail.andrew.cmu.edu> Message-ID: On Mon, Aug 29, 2011 at 2:59 PM, Likun Tan wrote: > Thanks, > > I read ex16.c, it seems that the two linear equations are solved > sequentially? > Yes, this is a test for reuse of preconditioner information. As I said, no one knows how to do this for Krylov methods (and everyone has tried). Matt > On Mon, August 29, 2011 10:52 am, Satish Balay wrote: > > On Mon, 29 Aug 2011, Matthew Knepley wrote: > > > > > >> On Mon, Aug 29, 2011 at 2:40 PM, Likun Tan > >> wrote: > >> > >> > >>> Dear all, > >>> > >>> > >>> I have linear systems Ax=b with different right-hand-side vectors. > >>> These > >>> vectors are independent. Could i generate b simultaneously and solve > >>> Ax=b > >>> simultaneously? > >>> > >> > >> Unfortunately, no one has figured out how to exploit this for Krylov > >> methods. > >> > > > > Also - check ksp/ksp/examples/tutorials/ex16.c > > > > > > Satish > > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Aug 29 10:14:51 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 29 Aug 2011 10:14:51 -0500 Subject: [petsc-users] about ksp In-Reply-To: References: <97a57c54262a0f2deb483408a9e7e0f1.squirrel@webmail.andrew.cmu.edu> Message-ID: On Mon, Aug 29, 2011 at 10:08, Matthew Knepley wrote: > As I said, no one knows > how to do this for Krylov methods (and everyone has tried). > There are methods and even simply running multiple independent Krylov solves concurrently would be good for memory traffic (matrix entries get reused for multiple vectors). The problem with block Krylov methods (that try to share information between multiple simultaneous solves) is loss of orthogonality between the subspaces generated by each vector. And it's not so much that there are no ways to detect and account for this, but robustness is still a problem and making efficient software for it is more tricky and AFAIK, has not been done. -------------- next part -------------- An HTML attachment was scrubbed... URL: From milan.v.mitrovic at gmail.com Mon Aug 29 10:43:54 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Mon, 29 Aug 2011 17:43:54 +0200 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: > Or your system is somewhat ill-conditioned. If you add '-mat_fd_type ds' to > the last command line, are the errors similar or improve a bit? Norm of matrix ratio 3.24879e-07 difference 3.24879e-05 So I guess it is the same... > A different way to test for a correct Jacobian is to run with > -snes_mf_operator -pc_type lu > and check that it converges in 1 or occasionally 2 iterations. This also > helps if the Jacobian is incorrect, but not visibly so on the first > iteration (e.g. because nonlinearities are silent for the initial guess). > SNES Function norm 2.369969231195e-02 Linear solve converged due to CONVERGED_RTOL iterations 0 SNES Object: type: ls line search variant: SNESLineSearchCubic alpha=0.0001, maxstep=1e+08, minlambda=1e-12 maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=0 total number of function evaluations=1 KSP Object: type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=20, initial guess is zero tolerances: relative=800695, absolute=5.33098e-35, divergence=5.33098e-35 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: type: lu LU: out-of-place factorization tolerance for zero pivot 1e-12 matrix ordering: nd factor fill ratio given 5, needed 12.0255 Factored matrix follows: Matrix Object: type=seqaij, rows=10000, cols=10000 package used to perform factorization: petsc total: nonzeros=13464834, allocated nonzeros=13464834 using I-node routines: found 6447 nodes, limit used is 5 linear system matrix = precond matrix: Matrix Object: type=seqaij, rows=10000, cols=10000 total: nonzeros=1119686, allocated nonzeros=1280000 not using I-node routines Nonlinear solve did not converge due to DIVERGED_LS_FAILURE I tried this with both -pc_type lu and without and got the same results... > -snes_mf_operator : solve with matrix-free finite differenced Jacobian, > preconditioned using the matrix you assembled > -snes_fd : assemble the dense FD operator (same as created by -snes_type > test) and solve using that I also tried using -snes_fd and get: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Null argument, when expecting valid pointer! [0]PETSC ERROR: Null Object: Parameter # 1! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ppm_pf on a darwin10. named milan-mitrovics-macbook-pro.local by milan Mon Aug 29 17:41:15 2011 [0]PETSC ERROR: Libraries linked from /Users/milan/Work/Library/petsc/lib [0]PETSC ERROR: Configure run at Fri May 6 16:32:02 2011 [0]PETSC ERROR: Configure options --with-mpi=true --with-mpi-dir=/Users/milan/Work/Library/openmpi.gcc/ --with-fortran=true --with-sundials=true --with-sundials-dir=/Users/milan/Work/Library/sundials/ --prefix=/Users/milan/Work/Library/petsc [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatAssembled() line 4595 in src/mat/interface/matrix.c [0]PETSC ERROR: SNESDefaultComputeJacobian() line 64 in src/snes/interface/snesj.c [0]PETSC ERROR: SNESComputeJacobian() line 1198 in src/snes/interface/snes.c [0]PETSC ERROR: SNESSolve_LS() line 189 in src/snes/impls/ls/ls.c [0]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c so I still havent gotten it to converge a single time :( From jedbrown at mcs.anl.gov Mon Aug 29 11:09:41 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 29 Aug 2011 11:09:41 -0500 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: On Mon, Aug 29, 2011 at 10:43, Milan Mitrovic wrote: > SNES Function norm 2.369969231195e-02 > Linear solve converged due to CONVERGED_RTOL iterations 0 > SNES Object: > type: ls > line search variant: SNESLineSearchCubic > alpha=0.0001, maxstep=1e+08, minlambda=1e-12 > maximum iterations=50, maximum function evaluations=10000 > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > total number of linear solver iterations=0 > total number of function evaluations=1 > I don't know how CONVERGED_RTOL with 0 iterations should be able to happen. Do you have a custom convergence test? To help me out, run $ script milan-debug-session.log # If you don't have this utility, copy from the terminal $ ./your_app -start_in_debugger noxterm When the debugger comes up, enter at the (gdb) prompt (gdb) break SNESSolve (gdb) continue It will stop again. (gdb) print &snes->reason $1 = (SNESConvergedReason *) 0x96d8c8 (gdb) watch *$1 (gdb) continue You should see where snes->reason gets changed. When it does (gdb) backtrace full (gdb) continue when the program exists (gdb) quit $ exit And send us the log. -------------- next part -------------- An HTML attachment was scrubbed... URL: From milan.v.mitrovic at gmail.com Mon Aug 29 11:24:31 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Mon, 29 Aug 2011 18:24:31 +0200 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: I repeatedly call the solver (for implicit time stepping) so I quit after one iteration of the time loop... the log is attached... -------------- next part -------------- A non-text attachment was scrubbed... Name: milan-debug-session.log Type: application/octet-stream Size: 161211 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Mon Aug 29 11:35:47 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 29 Aug 2011 11:35:47 -0500 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: On Mon, Aug 29, 2011 at 11:24, Milan Mitrovic wrote: > the log is attached... Oops, I misdiagnosed. Can you skip the debugger part and run with -info? -------------- next part -------------- An HTML attachment was scrubbed... URL: From milan.v.mitrovic at gmail.com Mon Aug 29 11:41:29 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Mon, 29 Aug 2011 18:41:29 +0200 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: Here it is :) -------------- next part -------------- A non-text attachment was scrubbed... Name: milan-debug-session.log Type: application/octet-stream Size: 29689 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Mon Aug 29 11:50:59 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 29 Aug 2011 11:50:59 -0500 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: [0] KSPDefaultConverged(): Linear solver has converged. Residual norm 0.0236997 is less than relative tolerance 800695 times initial right hand side norm 0.0236997 at iteration 0 The relative tolerance can't possibly be what you want, sorry I didn't notice this earlier. Linear solve converged due to CONVERGED_RTOL iterations 0 [0] SNESSolve_LS(): iter=0, linear solve iterations=0 [0] SNESLSCheckResidual_Private(): ||J^T(F-Ax)||/||F-AX|| 1 near zero implies inconsistent rhs [0] SNESLineSearchCubic(): Search direction and size is 0 [0] SNESSolve_LS(): fnorm=2.3699692311947174e-02, gnorm=2.3699692311947174e-02, ynorm=0.0000000000000000e+00, lssucceed=0 [0] SNESLSCheckLocalMin_Private(): || J^T F|| 1 near zero implies found a local minimum SNES Object: type: ls line search variant: SNESLineSearchCubic alpha=0.0001, maxstep=1e+08, minlambda=1e-12 maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=0 total number of function evaluations=1 KSP Object: type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=20, initial guess is zero tolerances: relative=800695, absolute=5.33098e-35, divergence=5.33098e-35 How are you setting the linear solve tolerance? Perhaps you are passing arguments of the incorrect type to KSPSetTolerances()? (Fortran doesn't warn about this, sadly.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From milanm at student.ethz.ch Mon Aug 29 12:14:26 2011 From: milanm at student.ethz.ch (Milan Mitrovic) Date: Mon, 29 Aug 2011 19:14:26 +0200 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: THANK YOU! that was indeed the case! I switched to defaults and it now converges with one iteration, and it does so with my jacobian as well :) I'll go play now :) From likunt at andrew.cmu.edu Mon Aug 29 15:51:21 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Mon, 29 Aug 2011 16:51:21 -0400 Subject: [petsc-users] about ksp Message-ID: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> Instead of solving Ax=b with different right-hand-side sequentially, we can also form a sparse block diagonal matrix A and a vector b composed of all the elements. Then we can set values to each section of b concurrently and solve the enlarged system in parallel, is this an efficient way? And also, I found MatCreateMPIBDiag() is used to define a sparse block diagonal matrix, in my case, each block has the same format and elements, how could i set values efficiently? Thanks, Likun On Mon, August 29, 2011 11:14 am, Jed Brown wrote: > On Mon, Aug 29, 2011 at 10:08, Matthew Knepley wrote: > > >> As I said, no one knows >> how to do this for Krylov methods (and everyone has tried). >> > > There are methods and even simply running multiple independent Krylov > solves concurrently would be good for memory traffic (matrix entries get > reused for multiple vectors). The problem with block Krylov methods (that > try to share information between multiple simultaneous solves) is loss of > orthogonality between the subspaces generated by each vector. And it's not > so much that there are no ways to detect and account for this, but > robustness is still a problem and making efficient software for it is more > tricky and AFAIK, has not been done. > From knepley at gmail.com Mon Aug 29 16:05:28 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 29 Aug 2011 21:05:28 +0000 Subject: [petsc-users] about ksp In-Reply-To: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> Message-ID: On Mon, Aug 29, 2011 at 8:51 PM, Likun Tan wrote: > Instead of solving Ax=b with different right-hand-side sequentially, we > can also form a sparse block diagonal matrix A and a vector b composed of > all the elements. Then we can set values to each section of b concurrently > and solve the enlarged system in parallel, is this an efficient way? > > And also, I found MatCreateMPIBDiag() is used to define a sparse block > diagonal matrix, in my case, each block has the same format and elements, > how could i set values efficiently? > There is no reason to introduce synchronization across these problems. Just split the communicator, put a KSP in each subcommunicator, and solve. Matt > Thanks, > Likun > > > On Mon, August 29, 2011 11:14 am, Jed Brown wrote: > > On Mon, Aug 29, 2011 at 10:08, Matthew Knepley > wrote: > > > > > >> As I said, no one knows > >> how to do this for Krylov methods (and everyone has tried). > >> > > > > There are methods and even simply running multiple independent Krylov > > solves concurrently would be good for memory traffic (matrix entries get > > reused for multiple vectors). The problem with block Krylov methods (that > > try to share information between multiple simultaneous solves) is loss of > > orthogonality between the subspaces generated by each vector. And it's > not > > so much that there are no ways to detect and account for this, but > > robustness is still a problem and making efficient software for it is > more > > tricky and AFAIK, has not been done. > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Aug 29 16:16:57 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 29 Aug 2011 16:16:57 -0500 Subject: [petsc-users] about ksp In-Reply-To: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> Message-ID: On Mon, Aug 29, 2011 at 15:51, Likun Tan wrote: > Instead of solving Ax=b with different right-hand-side sequentially, we > can also form a sparse block diagonal matrix A and a vector b composed of > all the elements. Then we can set values to each section of b concurrently > and solve the enlarged system in parallel, is this an efficient way? > You can assemble a single AIJ matrix and then use MatCreateMAIJ() to make it apply to a multi-vector. You can then make a larger vector and run a normal Krylov method. The only downside is that you won't get the usual property that the Krylov method is spectrally adaptive to the right hand side (because there is only one inner product for all the components), but this can work alright anyway. Preconditioning will take more effort. I think this is likely premature optimization for you. I recommend solving the systems by calling KSPSolve() multiple times for now. Later, when everything is working, you might experiment with ways to solve the systems together. > > And also, I found MatCreateMPIBDiag() > This format was removed from PETSc a few years ago, but I don't think it's what you want anyway. -------------- next part -------------- An HTML attachment was scrubbed... URL: From likunt at andrew.cmu.edu Mon Aug 29 16:36:17 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Mon, 29 Aug 2011 17:36:17 -0400 Subject: [petsc-users] about ksp In-Reply-To: References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> Message-ID: <5e1327829ac527dbfe33c8743275fac8.squirrel@webmail.andrew.cmu.edu> Thank you very much. Since i have millions of such linear equations to solve, so efficiency is my big concern. If i keep using the uncoupled small system and use ksp to solve the linear equations sequentially, there is nowhere i can take advantage of parallel computing. I think of the coupled system with the hope that the efficiency can be improved. But it seems not a good approach. I have tried solving the small system sequentially and it works fine, now i need to think ways to optimize my code. Any hints on this aspect? btw, thanks for all these communications. On Mon, August 29, 2011 5:16 pm, Jed Brown wrote: > On Mon, Aug 29, 2011 at 15:51, Likun Tan wrote: > > >> Instead of solving Ax=b with different right-hand-side sequentially, we >> can also form a sparse block diagonal matrix A and a vector b composed >> of all the elements. Then we can set values to each section of b >> concurrently and solve the enlarged system in parallel, is this an >> efficient way? >> > > You can assemble a single AIJ matrix and then use MatCreateMAIJ() to make > it apply to a multi-vector. You can then make a larger vector and run a > normal Krylov method. > > > The only downside is that you won't get the usual property that the > Krylov > method is spectrally adaptive to the right hand side (because there is > only one inner product for all the components), but this can work alright > anyway. Preconditioning will take more effort. > > > I think this is likely premature optimization for you. I recommend > solving the systems by calling KSPSolve() multiple times for now. Later, > when everything is working, you might experiment with ways to solve the > systems together. > > >> >> And also, I found MatCreateMPIBDiag() >> >> > > This format was removed from PETSc a few years ago, but I don't think > it's what you want anyway. > From jedbrown at mcs.anl.gov Mon Aug 29 16:46:56 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 29 Aug 2011 16:46:56 -0500 Subject: [petsc-users] about ksp In-Reply-To: <5e1327829ac527dbfe33c8743275fac8.squirrel@webmail.andrew.cmu.edu> References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> <5e1327829ac527dbfe33c8743275fac8.squirrel@webmail.andrew.cmu.edu> Message-ID: On Mon, Aug 29, 2011 at 16:36, Likun Tan wrote: > If i keep using the uncoupled small system and use ksp to solve the linear > equations sequentially, there is nowhere i can take advantage of parallel > computing. > How small are your systems? Maybe you should just create the same matrix on every processor and do a direct solve with all the right hand sides. -------------- next part -------------- An HTML attachment was scrubbed... URL: From likunt at andrew.cmu.edu Mon Aug 29 16:49:09 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Mon, 29 Aug 2011 17:49:09 -0400 Subject: [petsc-users] about ksp In-Reply-To: References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> <5e1327829ac527dbfe33c8743275fac8.squirrel@webmail.andrew.cmu.edu> Message-ID: <244e0215f03d77322ee865b6ee7a5ea9.squirrel@webmail.andrew.cmu.edu> It is 27*27 with 343 non-zeros, i use LU for all the linear equations. On Mon, August 29, 2011 5:46 pm, Jed Brown wrote: > On Mon, Aug 29, 2011 at 16:36, Likun Tan wrote: > > >> If i keep using the uncoupled small system and use ksp to solve the >> linear equations sequentially, there is nowhere i can take advantage of >> parallel computing. >> > > How small are your systems? Maybe you should just create the same matrix > on every processor and do a direct solve with all the right hand sides. > From jedbrown at mcs.anl.gov Mon Aug 29 17:01:21 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 29 Aug 2011 17:01:21 -0500 Subject: [petsc-users] about ksp In-Reply-To: <244e0215f03d77322ee865b6ee7a5ea9.squirrel@webmail.andrew.cmu.edu> References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> <5e1327829ac527dbfe33c8743275fac8.squirrel@webmail.andrew.cmu.edu> <244e0215f03d77322ee865b6ee7a5ea9.squirrel@webmail.andrew.cmu.edu> Message-ID: On Mon, Aug 29, 2011 at 16:49, Likun Tan wrote: > It is 27*27 with 343 non-zeros, i use LU for all the linear equations. This is just too small and not sparse enough for sparse linear algebra to pay off. Create the same serial dense matrix redundantly on every process, factor it, put all your right hand sides for that process in a matrix B, and call MatMatSolve(). -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuentesdt at gmail.com Mon Aug 29 17:49:24 2011 From: fuentesdt at gmail.com (David Fuentes) Date: Mon, 29 Aug 2011 17:49:24 -0500 Subject: [petsc-users] petsc and goto blas Message-ID: Is there any reason why petsc compiled to link with goto blas shared libraries would not run multi-threaded by default ? I've set (OMP/GOTO)_NUM_THREADS=8 but when I call dgemm from PETSc I can't seem to get it to run on multiple cores (<= 100% cpu usage from top). I checked and the test routines installed with goto library build called w/o petsc runs multi-threaded (~600% cpu usage on top). I'm calling MatMatMult with dense matrices from petsc4py. from petsc4py import PETSc import numpy n = 10000 J1 = PETSc.Mat().createDense([n, n], array=numpy.random.rand(n,n), comm=PETSc.COMM_WORLD) J1.assemblyBegin();J1.assemblyEnd() J2 = PETSc.Mat().createDense([n, n], array=numpy.random.rand(n,n),comm=PETSc.COMM_WORLD) J2.assemblyBegin(); J2.assemblyEnd() X = J1.matMult(J2) From bsmith at mcs.anl.gov Mon Aug 29 19:40:58 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 29 Aug 2011 19:40:58 -0500 Subject: [petsc-users] petsc and goto blas In-Reply-To: References: Message-ID: On Aug 29, 2011, at 5:49 PM, David Fuentes wrote: > Is there any reason why petsc compiled to link with goto blas shared > libraries would not run multi-threaded by default ? We don't do anything to prevent it from using multi-threaded. Te first thing I would suggest is make sure it truly is linking and running against the threaded goblas and something else doesn't get in between. Barry > > I've set (OMP/GOTO)_NUM_THREADS=8 > > but when I call dgemm from PETSc I can't seem to get it to run on > multiple cores (<= 100% cpu usage from top). > I checked and the test routines installed with goto library build > called w/o petsc runs multi-threaded (~600% cpu usage on top). > > > I'm calling MatMatMult with dense matrices from petsc4py. > > from petsc4py import PETSc > import numpy > n = 10000 > J1 = PETSc.Mat().createDense([n, n], array=numpy.random.rand(n,n), > comm=PETSc.COMM_WORLD) > J1.assemblyBegin();J1.assemblyEnd() > J2 = PETSc.Mat().createDense([n, n], > array=numpy.random.rand(n,n),comm=PETSc.COMM_WORLD) > J2.assemblyBegin(); J2.assemblyEnd() > X = J1.matMult(J2) From stali at geology.wisc.edu Mon Aug 29 20:32:18 2011 From: stali at geology.wisc.edu (Tabrez Ali) Date: Mon, 29 Aug 2011 20:32:18 -0500 Subject: [petsc-users] Vecscatter question Message-ID: <4E5C3DA2.6000209@geology.wisc.edu> Hello If during Vecscatter (say from a global vector x to a local vector y) if 'is' and 'iy' are such that almost all values being scattered happen to be on the local proc then almost 0 or very little MPI calls would be made internally. Is this correct? Such a setup would not cause scalability problems except consuming some local proc memory due to 'y', 'is' and 'iy' being larger than required. Thanks in advance. Tabrez From bsmith at mcs.anl.gov Mon Aug 29 20:33:10 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 29 Aug 2011 20:33:10 -0500 Subject: [petsc-users] Vecscatter question In-Reply-To: <4E5C3DA2.6000209@geology.wisc.edu> References: <4E5C3DA2.6000209@geology.wisc.edu> Message-ID: <261CD6E0-6794-4BB6-9832-5A8A07C7B5FB@mcs.anl.gov> On Aug 29, 2011, at 8:32 PM, Tabrez Ali wrote: > Hello > > If during Vecscatter (say from a global vector x to a local vector y) if 'is' and 'iy' are such that almost all values being scattered happen to be on the local proc then almost 0 or very little MPI calls would be made internally. Is this correct? Yes, it does not use MPI to move the values locally. In fact, when possible it uses an efficient memcpy. > > Such a setup would not cause scalability problems except consuming some local proc memory due to 'y', 'is' and 'iy' being larger than required. > > Thanks in advance. > > Tabrez > > From jedbrown at mcs.anl.gov Mon Aug 29 20:36:41 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 29 Aug 2011 20:36:41 -0500 Subject: [petsc-users] Vecscatter question In-Reply-To: <4E5C3DA2.6000209@geology.wisc.edu> References: <4E5C3DA2.6000209@geology.wisc.edu> Message-ID: On Mon, Aug 29, 2011 at 20:32, Tabrez Ali wrote: > If during Vecscatter (say from a global vector x to a local vector y) if > 'is' and 'iy' are such that almost all values being scattered happen to be > on the local proc then almost 0 or very little MPI calls would be made > internally. Is this correct? > Very few values would be sent anywhere. The number of MPI calls depends on the number of processes that need to receive values and the scatter method (methods descriptions: http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Vec/VecScatterCreate.html ). -------------- next part -------------- An HTML attachment was scrubbed... URL: From khalid_eee at yahoo.com Mon Aug 29 22:43:43 2011 From: khalid_eee at yahoo.com (khalid ashraf) Date: Mon, 29 Aug 2011 20:43:43 -0700 (PDT) Subject: [petsc-users] Loading vector in parallel using PetscBinaryRead Message-ID: <1314675823.65364.YahooMailNeo@web112604.mail.gq1.yahoo.com> Hi,? I am trying to load a vector that I saved as PetscBinary file.? On a single processor, I have to put an initial PetscBinaryRead command & put the subsequent reads in the loop. Then I get the correct answer. I don't understand why an initial PetscBinaryRead is required. Anyway, with this, the code works fine on single processor. However, on multiple processors, the input file and the vector after reading are not in the same order. I think that somehow I have to pass the information to each processor as to which part of the binary file to read. Please tell me how to do this. Thanks. Khalid Here is the code: PetscViewerBinaryOpen(appctx->comm,"Pxbin_10.out",FILE_MODE_READ,&viewer); PetscViewerBinaryGetDescriptor(viewer,&fd); PetscBinaryRead(fd,header,1,PETSC_SCALAR); ?for (k=zs; k From knepley at gmail.com Tue Aug 30 01:06:47 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 30 Aug 2011 06:06:47 +0000 Subject: [petsc-users] Vecscatter question In-Reply-To: References: <4E5C3DA2.6000209@geology.wisc.edu> Message-ID: On Tue, Aug 30, 2011 at 1:36 AM, Jed Brown wrote: > On Mon, Aug 29, 2011 at 20:32, Tabrez Ali wrote: > >> If during Vecscatter (say from a global vector x to a local vector y) if >> 'is' and 'iy' are such that almost all values being scattered happen to be >> on the local proc then almost 0 or very little MPI calls would be made >> internally. Is this correct? >> > > Very few values would be sent anywhere. The number of MPI calls depends on > the number of processes that need to receive values and the scatter method > (methods descriptions: > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Vec/VecScatterCreate.html > ). > Also, you can make an event around these calls and get information on the number of messages and the average size. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Aug 30 01:08:59 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 30 Aug 2011 06:08:59 +0000 Subject: [petsc-users] Loading vector in parallel using PetscBinaryRead In-Reply-To: <1314675823.65364.YahooMailNeo@web112604.mail.gq1.yahoo.com> References: <1314675823.65364.YahooMailNeo@web112604.mail.gq1.yahoo.com> Message-ID: On Tue, Aug 30, 2011 at 3:43 AM, khalid ashraf wrote: > Hi, > I am trying to load a vector that I saved as PetscBinary file. > On a single processor, I have to put an initial PetscBinaryRead command & > put the subsequent reads in the loop. Then I get the correct answer. I don't > understand why an initial PetscBinaryRead is required. Anyway, with this, > the code works fine on single processor. > > However, on multiple processors, the input file and the vector after > reading are not in the same order. I think that somehow I have to pass the > information to each processor as to which part of the binary file to read. > Please tell me how to do this. > Is there a reason you are not using VecLoad, http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Vec/VecLoadIntoVector.html? Matt > Thanks. > > Khalid > > > Here is the code: > > PetscViewerBinaryOpen(appctx->comm,"Pxbin_10.out",FILE_MODE_READ,&viewer); > PetscViewerBinaryGetDescriptor(viewer,&fd); > PetscBinaryRead(fd,header,1,PETSC_SCALAR); > > for (k=zs; k for (j=ys; j for (i=xs; i PetscBinaryRead(fd,header,1,PETSC_SCALAR); > u_localptr[k][j][i] = header[0]; > } > } > } > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Aug 30 08:21:53 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 30 Aug 2011 08:21:53 -0500 Subject: [petsc-users] Problem with SNES In-Reply-To: References: Message-ID: On Mon, Aug 29, 2011 at 12:14, Milan Mitrovic wrote: > that was indeed the case! Gread, glad it's working now. I have added checks for valid parameter ranges in KSPSetTolerances() and SNESSetTolerances() to petsc-dev so this sort of type mismatch should be obvious sooner. -------------- next part -------------- An HTML attachment was scrubbed... URL: From likunt at andrew.cmu.edu Tue Aug 30 12:08:49 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Tue, 30 Aug 2011 13:08:49 -0400 Subject: [petsc-users] about ksp In-Reply-To: References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> <5e1327829ac527dbfe33c8743275fac8.squirrel@webmail.andrew.cmu.edu> <244e0215f03d77322ee865b6ee7a5ea9.squirrel@webmail.andrew.cmu.edu> Message-ID: Dear all, I am using MatCreateSeqDense() to create the matrix A, if i set np=2, i got the error 'Comm must be of size 1'. As you suggested, i should put the same dense matrix on every process, but how should i do it? And also, should the right hand side matrix B be partitioned by column? Thanks, Likun On Mon, August 29, 2011 6:01 pm, Jed Brown wrote: > On Mon, Aug 29, 2011 at 16:49, Likun Tan wrote: > > >> It is 27*27 with 343 non-zeros, i use LU for all the linear equations. >> > > > This is just too small and not sparse enough for sparse linear algebra to > pay off. Create the same serial dense matrix redundantly on every > process, factor it, put all your right hand sides for that process in a > matrix B, and call MatMatSolve(). > From balay at mcs.anl.gov Tue Aug 30 12:11:13 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 30 Aug 2011 12:11:13 -0500 (CDT) Subject: [petsc-users] about ksp In-Reply-To: References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> <5e1327829ac527dbfe33c8743275fac8.squirrel@webmail.andrew.cmu.edu> <244e0215f03d77322ee865b6ee7a5ea9.squirrel@webmail.andrew.cmu.edu> Message-ID: what communicator did you use with MatCreateSeqDense()? Try PETSC_COMM_SELF or MPI_COMM_SELF [and not PETSC_COMM_WORLD] Satish On Tue, 30 Aug 2011, Likun Tan wrote: > Dear all, > > I am using MatCreateSeqDense() to create the matrix A, if i set np=2, i > got the error 'Comm must be of size 1'. As you suggested, i should put the > same dense matrix on every process, but how should i do it? > > And also, should the right hand side matrix B be partitioned by column? > > Thanks, > Likun > > > On Mon, August 29, 2011 6:01 pm, Jed Brown wrote: > > On Mon, Aug 29, 2011 at 16:49, Likun Tan wrote: > > > > > >> It is 27*27 with 343 non-zeros, i use LU for all the linear equations. > >> > > > > > > This is just too small and not sparse enough for sparse linear algebra to > > pay off. Create the same serial dense matrix redundantly on every > > process, factor it, put all your right hand sides for that process in a > > matrix B, and call MatMatSolve(). > > > > > > > From bsmith at mcs.anl.gov Tue Aug 30 12:12:18 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 30 Aug 2011 12:12:18 -0500 Subject: [petsc-users] about ksp In-Reply-To: References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> <5e1327829ac527dbfe33c8743275fac8.squirrel@webmail.andrew.cmu.edu> <244e0215f03d77322ee865b6ee7a5ea9.squirrel@webmail.andrew.cmu.edu> Message-ID: <45C4BE1F-928F-4E34-8AAB-7D26AA34995B@mcs.anl.gov> On Aug 30, 2011, at 12:08 PM, Likun Tan wrote: > Dear all, > > I am using MatCreateSeqDense() to create the matrix A, if i set np=2, i > got the error 'Comm must be of size 1'. As you suggested, i should put the > same dense matrix on every process, but how should i do it? > > And also, should the right hand side matrix B be partitioned by column? The comm argument to PETSc create functions determines what processes share the object. A seqdense matrix cannot be shared, it must live on a single process hence using PETSC_COMM_SELF is appropriate; if this is called on several processes then each process will hold the entire matrix. Barry > > Thanks, > Likun > > > On Mon, August 29, 2011 6:01 pm, Jed Brown wrote: >> On Mon, Aug 29, 2011 at 16:49, Likun Tan wrote: >> >> >>> It is 27*27 with 343 non-zeros, i use LU for all the linear equations. >>> >> >> >> This is just too small and not sparse enough for sparse linear algebra to >> pay off. Create the same serial dense matrix redundantly on every >> process, factor it, put all your right hand sides for that process in a >> matrix B, and call MatMatSolve(). >> > > > > From likunt at andrew.cmu.edu Tue Aug 30 17:59:21 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Tue, 30 Aug 2011 18:59:21 -0400 Subject: [petsc-users] about MatTranspose In-Reply-To: <45C4BE1F-928F-4E34-8AAB-7D26AA34995B@mcs.anl.gov> References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> <5e1327829ac527dbfe33c8743275fac8.squirrel@webmail.andrew.cmu.edu> <244e0215f03d77322ee865b6ee7a5ea9.squirrel@webmail.andrew.cmu.edu> <45C4BE1F-928F-4E34-8AAB-7D26AA34995B@mcs.anl.gov> Message-ID: <92ee3133d6e2801fc1e3f727947cf3e6.squirrel@webmail.andrew.cmu.edu> Dear all, Can MatTranspose be used on non-squra matrix? I have a matrix with size 200*27, and i want to get the tranpose of it. Second question is, if the 200*27 is partitioned by row, how is the tranpose stored in each processor? Is it stored by column? Thanks, Likun From knepley at gmail.com Tue Aug 30 18:08:21 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 30 Aug 2011 23:08:21 +0000 Subject: [petsc-users] about MatTranspose In-Reply-To: <92ee3133d6e2801fc1e3f727947cf3e6.squirrel@webmail.andrew.cmu.edu> References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> <5e1327829ac527dbfe33c8743275fac8.squirrel@webmail.andrew.cmu.edu> <244e0215f03d77322ee865b6ee7a5ea9.squirrel@webmail.andrew.cmu.edu> <45C4BE1F-928F-4E34-8AAB-7D26AA34995B@mcs.anl.gov> <92ee3133d6e2801fc1e3f727947cf3e6.squirrel@webmail.andrew.cmu.edu> Message-ID: On Tue, Aug 30, 2011 at 10:59 PM, Likun Tan wrote: > > Dear all, > > Can MatTranspose be used on non-squra matrix? I have a matrix with size > 200*27, and i want to get the tranpose of it. > I assume you are talking about dense matrices. If so, yes. > Second question is, if the 200*27 is partitioned by row, how is the > tranpose stored in each processor? Is it stored by column? > For MPIDENSE, you cannot in-place transpose unless it is square. Otherwise, you provide the transpose matrix, so you determine the layout. Matt > Thanks, > Likun > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From likunt at andrew.cmu.edu Tue Aug 30 19:48:56 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Tue, 30 Aug 2011 20:48:56 -0400 Subject: [petsc-users] about MatTranspose In-Reply-To: References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> <5e1327829ac527dbfe33c8743275fac8.squirrel@webmail.andrew.cmu.edu> <244e0215f03d77322ee865b6ee7a5ea9.squirrel@webmail.andrew.cmu.edu> <45C4BE1F-928F-4E34-8AAB-7D26AA34995B@mcs.anl.gov> <92ee3133d6e2801fc1e3f727947cf3e6.squirrel@webmail.andrew.cmu.edu> Message-ID: <4bf0fcffeb9f2c0d9b6206327df6e8c1.squirrel@webmail.andrew.cmu.edu> I have MPIDense matrix, if i want to get the tranpose of the matrix, what should i do? Or i should use MPISparse instead? On Tue, August 30, 2011 7:08 pm, Matthew Knepley wrote: > On Tue, Aug 30, 2011 at 10:59 PM, Likun Tan > wrote: > > >> >> Dear all, >> >> >> Can MatTranspose be used on non-squra matrix? I have a matrix with size >> 200*27, and i want to get the tranpose of it. >> >> > > I assume you are talking about dense matrices. If so, yes. > > > >> Second question is, if the 200*27 is partitioned by row, how is the >> tranpose stored in each processor? Is it stored by column? >> > > For MPIDENSE, you cannot in-place transpose unless it is square. > Otherwise, > you provide the transpose matrix, so you determine the layout. > > Matt > > > >> Thanks, >> Likun >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. -- Norbert Wiener > > From knepley at gmail.com Tue Aug 30 21:07:31 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 31 Aug 2011 02:07:31 +0000 Subject: [petsc-users] about MatTranspose In-Reply-To: <4bf0fcffeb9f2c0d9b6206327df6e8c1.squirrel@webmail.andrew.cmu.edu> References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> <5e1327829ac527dbfe33c8743275fac8.squirrel@webmail.andrew.cmu.edu> <244e0215f03d77322ee865b6ee7a5ea9.squirrel@webmail.andrew.cmu.edu> <45C4BE1F-928F-4E34-8AAB-7D26AA34995B@mcs.anl.gov> <92ee3133d6e2801fc1e3f727947cf3e6.squirrel@webmail.andrew.cmu.edu> <4bf0fcffeb9f2c0d9b6206327df6e8c1.squirrel@webmail.andrew.cmu.edu> Message-ID: On Wed, Aug 31, 2011 at 12:48 AM, Likun Tan wrote: > > I have MPIDense matrix, if i want to get the tranpose of the matrix, what > should i do? Or i should use MPISparse instead? If it is rectangular, create the transpose matrix first, then call MatTranspose. Matt > On Tue, August 30, 2011 7:08 pm, Matthew Knepley wrote: > > On Tue, Aug 30, 2011 at 10:59 PM, Likun Tan > > wrote: > > > > > >> > >> Dear all, > >> > >> > >> Can MatTranspose be used on non-squra matrix? I have a matrix with size > >> 200*27, and i want to get the tranpose of it. > >> > >> > > > > I assume you are talking about dense matrices. If so, yes. > > > > > > > >> Second question is, if the 200*27 is partitioned by row, how is the > >> tranpose stored in each processor? Is it stored by column? > >> > > > > For MPIDENSE, you cannot in-place transpose unless it is square. > > Otherwise, > > you provide the transpose matrix, so you determine the layout. > > > > Matt > > > > > > > >> Thanks, > >> Likun > >> > >> > >> > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > their > > experiments lead. -- Norbert Wiener > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From likunt at andrew.cmu.edu Tue Aug 30 21:17:13 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Tue, 30 Aug 2011 22:17:13 -0400 Subject: [petsc-users] about MatTranspose In-Reply-To: References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> <5e1327829ac527dbfe33c8743275fac8.squirrel@webmail.andrew.cmu.edu> <244e0215f03d77322ee865b6ee7a5ea9.squirrel@webmail.andrew.cmu.edu> <45C4BE1F-928F-4E34-8AAB-7D26AA34995B@mcs.anl.gov> <92ee3133d6e2801fc1e3f727947cf3e6.squirrel@webmail.andrew.cmu.edu> <4bf0fcffeb9f2c0d9b6206327df6e8c1.squirrel@webmail.andrew.cmu.edu> Message-ID: Thank you very much. So i should create the transpose matrix as MPIDense and partition it corresponding? That is, if the matrix is partitioned into 3 parts by row, then the transpose matrix will be partitioned into 3 parts by column? I have two more questions: 1. I define a DA with PETSC_COMM_SELF, where is the array stored? Could i use this array in every process? 2. I vaguely remember that matrix can only be partitioned by row, is that true? best, Likun On Tue, August 30, 2011 10:07 pm, Matthew Knepley wrote: > On Wed, Aug 31, 2011 at 12:48 AM, Likun Tan > wrote: > > >> >> I have MPIDense matrix, if i want to get the tranpose of the matrix, >> what should i do? Or i should use MPISparse instead? > > > If it is rectangular, create the transpose matrix first, then call > MatTranspose. > > > Matt > > > >> On Tue, August 30, 2011 7:08 pm, Matthew Knepley wrote: >> >>> On Tue, Aug 30, 2011 at 10:59 PM, Likun Tan >>> wrote: >>> >>> >>> >>>> >>>> Dear all, >>>> >>>> >>>> >>>> Can MatTranspose be used on non-squra matrix? I have a matrix with >>>> size 200*27, and i want to get the tranpose of it. >>>> >>>> >>>> >>> >>> I assume you are talking about dense matrices. If so, yes. >>> >>> >>> >>> >>>> Second question is, if the 200*27 is partitioned by row, how is the >>>> tranpose stored in each processor? Is it stored by column? >>>> >>> >>> For MPIDENSE, you cannot in-place transpose unless it is square. >>> Otherwise, >>> you provide the transpose matrix, so you determine the layout. >>> >>> Matt >>> >>> >>> >>> >>>> Thanks, >>>> Likun >>>> >>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which >> their >>> experiments lead. -- Norbert Wiener >>> >>> >> >> >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. -- Norbert Wiener > > From fuentesdt at gmail.com Wed Aug 31 00:29:17 2011 From: fuentesdt at gmail.com (David Fuentes) Date: Wed, 31 Aug 2011 00:29:17 -0500 Subject: [petsc-users] petsc and goto blas In-Reply-To: References: Message-ID: thanks. ended up being something w/ the mpi compilers I was using. difficult to track down. works with the --download-mpi=yes petsc config option though! when using the "--download-mpi=yes" option on a small cluster, are the configure scripts able to detect an infiniband switch and configure/install mpich to communicate over it accordingly ? either way, thanks! df On Mon, Aug 29, 2011 at 7:40 PM, Barry Smith wrote: > > On Aug 29, 2011, at 5:49 PM, David Fuentes wrote: > >> Is there any reason why petsc compiled to link with goto blas shared >> libraries would not run multi-threaded by default ? > > ? We don't do anything to prevent it from using multi-threaded. Te first thing I would suggest is make sure it truly is linking and running against the threaded goblas and something else doesn't get in between. > > ? Barry > >> >> I've set (OMP/GOTO)_NUM_THREADS=8 >> >> but when I call dgemm from PETSc I can't seem to get it to run on >> multiple cores (<= 100% cpu usage from top). >> I checked and the test routines installed with goto library build >> called w/o petsc runs multi-threaded (~600% cpu usage on top). >> >> >> I'm calling MatMatMult with dense matrices from petsc4py. >> >> from petsc4py import PETSc >> import numpy >> n = 10000 >> J1 = PETSc.Mat().createDense([n, n], array=numpy.random.rand(n,n), >> comm=PETSc.COMM_WORLD) >> J1.assemblyBegin();J1.assemblyEnd() >> J2 = PETSc.Mat().createDense([n, n], >> array=numpy.random.rand(n,n),comm=PETSc.COMM_WORLD) >> J2.assemblyBegin(); J2.assemblyEnd() >> X = J1.matMult(J2) > > From jedbrown at mcs.anl.gov Wed Aug 31 00:50:20 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 31 Aug 2011 00:50:20 -0500 Subject: [petsc-users] about MatTranspose In-Reply-To: References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> <5e1327829ac527dbfe33c8743275fac8.squirrel@webmail.andrew.cmu.edu> <244e0215f03d77322ee865b6ee7a5ea9.squirrel@webmail.andrew.cmu.edu> <45C4BE1F-928F-4E34-8AAB-7D26AA34995B@mcs.anl.gov> <92ee3133d6e2801fc1e3f727947cf3e6.squirrel@webmail.andrew.cmu.edu> <4bf0fcffeb9f2c0d9b6206327df6e8c1.squirrel@webmail.andrew.cmu.edu> Message-ID: On Tue, Aug 30, 2011 at 21:17, Likun Tan wrote: > Thank you very much. So i should create the transpose matrix as MPIDense > and partition it corresponding? > What do you want to do with the transpose? Chances are that you can do that without changing what is stored in memory. There is MatCreateTranspose(), MatSolveTranspose(), KSPSolveTranspose(), etc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Aug 31 00:58:58 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 31 Aug 2011 00:58:58 -0500 Subject: [petsc-users] petsc and goto blas In-Reply-To: References: Message-ID: On Wed, Aug 31, 2011 at 00:29, David Fuentes wrote: > when using the "--download-mpi=yes" option on a small cluster, > are the configure scripts able to detect an infiniband switch and > configure/install mpich to communicate over it accordingly ? > MPICH2 does not use InfiniBand, you would use MVAPICH for that. Open MPI should use IB if it is available, but it may not be tuned by default (the system install might not have special tuning either). Run the programs you care about with -log_summary. If a lot of time is spent in communication, it indicates that the computation you are doing is network-limited and it would likely be worth your time to make the system MPI work for you, otherwise the generic one may be fine. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Aug 31 03:07:47 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 31 Aug 2011 08:07:47 +0000 Subject: [petsc-users] about MatTranspose In-Reply-To: References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> <5e1327829ac527dbfe33c8743275fac8.squirrel@webmail.andrew.cmu.edu> <244e0215f03d77322ee865b6ee7a5ea9.squirrel@webmail.andrew.cmu.edu> <45C4BE1F-928F-4E34-8AAB-7D26AA34995B@mcs.anl.gov> <92ee3133d6e2801fc1e3f727947cf3e6.squirrel@webmail.andrew.cmu.edu> <4bf0fcffeb9f2c0d9b6206327df6e8c1.squirrel@webmail.andrew.cmu.edu> Message-ID: On Wed, Aug 31, 2011 at 2:17 AM, Likun Tan wrote: > > Thank you very much. So i should create the transpose matrix as MPIDense > and partition it corresponding? That is, if the matrix is partitioned into > 3 parts by row, then the transpose matrix will be partitioned into 3 parts > by column? > 1 part per process. > I have two more questions: > 1. I define a DA with PETSC_COMM_SELF, where is the array stored? Could i > use this array in every process? > It does not store anything, but with PETSC_COMM_SELF, the array will not be partitioned. > 2. I vaguely remember that matrix can only be partitioned by row, is that > true? > Sparse matrices are generally partitioned by row. This is how the AIJ classes work, and also our dense classes to mirror it. Matt > best, > Likun > > > On Tue, August 30, 2011 10:07 pm, Matthew Knepley wrote: > > On Wed, Aug 31, 2011 at 12:48 AM, Likun Tan > > wrote: > > > > > >> > >> I have MPIDense matrix, if i want to get the tranpose of the matrix, > >> what should i do? Or i should use MPISparse instead? > > > > > > If it is rectangular, create the transpose matrix first, then call > > MatTranspose. > > > > > > Matt > > > > > > > >> On Tue, August 30, 2011 7:08 pm, Matthew Knepley wrote: > >> > >>> On Tue, Aug 30, 2011 at 10:59 PM, Likun Tan > >>> wrote: > >>> > >>> > >>> > >>>> > >>>> Dear all, > >>>> > >>>> > >>>> > >>>> Can MatTranspose be used on non-squra matrix? I have a matrix with > >>>> size 200*27, and i want to get the tranpose of it. > >>>> > >>>> > >>>> > >>> > >>> I assume you are talking about dense matrices. If so, yes. > >>> > >>> > >>> > >>> > >>>> Second question is, if the 200*27 is partitioned by row, how is the > >>>> tranpose stored in each processor? Is it stored by column? > >>>> > >>> > >>> For MPIDENSE, you cannot in-place transpose unless it is square. > >>> Otherwise, > >>> you provide the transpose matrix, so you determine the layout. > >>> > >>> Matt > >>> > >>> > >>> > >>> > >>>> Thanks, > >>>> Likun > >>>> > >>>> > >>>> > >>>> > >>> > >>> > >>> -- > >>> What most experimenters take for granted before they begin their > >>> experiments is infinitely more interesting than any results to which > >> their > >>> experiments lead. -- Norbert Wiener > >>> > >>> > >> > >> > >> > >> > >> > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > their > > experiments lead. -- Norbert Wiener > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Wed Aug 31 07:52:26 2011 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Wed, 31 Aug 2011 14:52:26 +0200 Subject: [petsc-users] Memory allocation failed Message-ID: I am getting the below cited error for a big linear problem. Would that mean int overflow and indicate the need for --with-64-bit-indices? Obviously it will elevate the memory requirements, but will it also reduce the performance? Many thanks, Dominik Error! ***Memory allocation failed for PreAllocateMemory: wspace->core. Requested size: -841742012 bytes From jedbrown at mcs.anl.gov Wed Aug 31 08:02:21 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 31 Aug 2011 08:02:21 -0500 Subject: [petsc-users] Memory allocation failed In-Reply-To: References: Message-ID: On Wed, Aug 31, 2011 at 07:52, Dominik Szczerba wrote: > I am getting the below cited error for a big linear problem. Would > that mean int overflow and indicate the need for > --with-64-bit-indices? > Yes, likely. > Obviously it will elevate the memory > requirements, but will it also reduce the performance? > Try it. Performance for sparse linear algebra is mostly limited by memory already. Using 64-bit indices increases the cost per nonzero in AIJ format, sizeof(PetscScalar)+sizeof(PetscInt), typically from 12 to 16 bytes. If you use BAIJ, the 64-bit indices will make much less difference. -------------- next part -------------- An HTML attachment was scrubbed... URL: From likunt at andrew.cmu.edu Wed Aug 31 09:41:35 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Wed, 31 Aug 2011 10:41:35 -0400 Subject: [petsc-users] about MatTranspose In-Reply-To: References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> <5e1327829ac527dbfe33c8743275fac8.squirrel@webmail.andrew.cmu.edu> <244e0215f03d77322ee865b6ee7a5ea9.squirrel@webmail.andrew.cmu.edu> <45C4BE1F-928F-4E34-8AAB-7D26AA34995B@mcs.anl.gov> <92ee3133d6e2801fc1e3f727947cf3e6.squirrel@webmail.andrew.cmu.edu> <4bf0fcffeb9f2c0d9b6206327df6e8c1.squirrel@webmail.andrew.cmu.edu> Message-ID: <62da0198e4d4ef47b647666e720c2dc9.squirrel@webmail.andrew.cmu.edu> Thanks, so what I want to do is using MatMatSolve() to solve Ax=B, where A is a 27*27 dense matrix, B is a 27*4000 dense matrix. 1. For efficiency, i want to generate the matrix B in parallel. Since it cannot be partitioned by row, i generated a 4000*27 matrix and try to get the transpose of it. Should i use MatCreateTranspose()? 2. Does MatMatSolve() solve Ax=B in parallel? If so, since A is not partitioned, is A visible in every process? or other steps i should do? 3. I have a 3D array (with PETSC_COMM_SELF) that will be used to generate all the elements of matrix B, how could I make it available in every process? Thanks, Likun On Wed, August 31, 2011 4:07 am, Matthew Knepley wrote: > On Wed, Aug 31, 2011 at 2:17 AM, Likun Tan wrote: > > >> >> Thank you very much. So i should create the transpose matrix as >> MPIDense >> and partition it corresponding? That is, if the matrix is partitioned >> into 3 parts by row, then the transpose matrix will be partitioned into >> 3 parts >> by column? >> > > 1 part per process. > > > >> I have two more questions: >> 1. I define a DA with PETSC_COMM_SELF, where is the array stored? Could >> i use this array in every process? >> > > It does not store anything, but with PETSC_COMM_SELF, the array will not > be partitioned. > > >> 2. I vaguely remember that matrix can only be partitioned by row, is >> that true? >> > > Sparse matrices are generally partitioned by row. This is how the AIJ > classes work, and also our dense classes to mirror it. > > Matt > > > >> best, Likun >> >> >> >> On Tue, August 30, 2011 10:07 pm, Matthew Knepley wrote: >> >>> On Wed, Aug 31, 2011 at 12:48 AM, Likun Tan >>> wrote: >>> >>> >>> >>>> >>>> I have MPIDense matrix, if i want to get the tranpose of the >>>> matrix, what should i do? Or i should use MPISparse instead? >>> >>> >>> If it is rectangular, create the transpose matrix first, then call >>> MatTranspose. >>> >>> >>> >>> Matt >>> >>> >>> >>> >>>> On Tue, August 30, 2011 7:08 pm, Matthew Knepley wrote: >>>> >>>> >>>>> On Tue, Aug 30, 2011 at 10:59 PM, Likun Tan >>>>> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> >>>>>> >>>>>> Dear all, >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Can MatTranspose be used on non-squra matrix? I have a matrix >>>>>> with size 200*27, and i want to get the tranpose of it. >>>>>> >>>>>> >>>>>> >>>>> >>>>> I assume you are talking about dense matrices. If so, yes. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Second question is, if the 200*27 is partitioned by row, how is >>>>>> the tranpose stored in each processor? Is it stored by column? >>>>>> >>>>> >>>>> For MPIDENSE, you cannot in-place transpose unless it is square. >>>>> Otherwise, >>>>> you provide the transpose matrix, so you determine the layout. >>>>> >>>>> Matt >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Thanks, >>>>>> Likun >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to >>>>> which >>>> their >>>>> experiments lead. -- Norbert Wiener >>>>> >>>>> >>>> >>>> >>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which >> their >>> experiments lead. -- Norbert Wiener >>> >>> >> >> >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. -- Norbert Wiener > > From jedbrown at mcs.anl.gov Wed Aug 31 09:50:37 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 31 Aug 2011 09:50:37 -0500 Subject: [petsc-users] about MatTranspose In-Reply-To: <62da0198e4d4ef47b647666e720c2dc9.squirrel@webmail.andrew.cmu.edu> References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> <5e1327829ac527dbfe33c8743275fac8.squirrel@webmail.andrew.cmu.edu> <244e0215f03d77322ee865b6ee7a5ea9.squirrel@webmail.andrew.cmu.edu> <45C4BE1F-928F-4E34-8AAB-7D26AA34995B@mcs.anl.gov> <92ee3133d6e2801fc1e3f727947cf3e6.squirrel@webmail.andrew.cmu.edu> <4bf0fcffeb9f2c0d9b6206327df6e8c1.squirrel@webmail.andrew.cmu.edu> <62da0198e4d4ef47b647666e720c2dc9.squirrel@webmail.andrew.cmu.edu> Message-ID: On Wed, Aug 31, 2011 at 09:41, Likun Tan wrote: > so what I want to do is using MatMatSolve() to solve Ax=B, where A is a > 27*27 dense matrix, B is a 27*4000 dense matrix. > > 1. For efficiency, i want to generate the matrix B in parallel. Since it > cannot be partitioned by row, i generated a 4000*27 matrix and try to get > the transpose of it. Should i use MatCreateTranspose()? > > 2. Does MatMatSolve() solve Ax=B in parallel? > Depends whether A is parallel, but you don't want it to anyway. > If so, since A is not > partitioned, is A visible in every process? or other steps i should do? > You should redundantly create A and redundantly create parts of B. That is, every process will have A : 27x27 dense matrix on COMM_SELF. B : 27x(4000/np) dense matrix on COMM_SELF. np is the number of processes in COMM_WORLD > 3. I have a 3D array (with PETSC_COMM_SELF) that will be used to generate > all the elements of matrix B, how could I make it available in every > process? > Where did it come from? (How did you get this array in the first place?) -------------- next part -------------- An HTML attachment was scrubbed... URL: From likunt at andrew.cmu.edu Wed Aug 31 10:02:36 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Wed, 31 Aug 2011 11:02:36 -0400 Subject: [petsc-users] about MatTranspose In-Reply-To: References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> <244e0215f03d77322ee865b6ee7a5ea9.squirrel@webmail.andrew.cmu.edu> <45C4BE1F-928F-4E34-8AAB-7D26AA34995B@mcs.anl.gov> <92ee3133d6e2801fc1e3f727947cf3e6.squirrel@webmail.andrew.cmu.edu> <4bf0fcffeb9f2c0d9b6206327df6e8c1.squirrel@webmail.andrew.cmu.edu> <62da0198e4d4ef47b647666e720c2dc9.squirrel@webmail.andrew.cmu.edu> Message-ID: <0c9c2be4b200e3bfddcb930ae8e02b58.squirrel@webmail.andrew.cmu.edu> Thanks. Where should i 'redundantly create A and B' ? Is this the correct procedure: MatCreateSeqDense(PETSC_COMM_SELF, M, M, PESTC_NULL, &A) MatCreateMPIDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, M, N, PETSC_NULL, &B) MatSetValues() //Set values to A and B MatCholeskyFactor()// Factorization MatMatSolve() The 3D array is generated before setting values to A and B. It comes from some known function evaluation, an i need the array in every process. On Wed, August 31, 2011 10:50 am, Jed Brown wrote: > On Wed, Aug 31, 2011 at 09:41, Likun Tan wrote: > > >> so what I want to do is using MatMatSolve() to solve Ax=B, where A is a >> 27*27 dense matrix, B is a 27*4000 dense matrix. >> >> >> 1. For efficiency, i want to generate the matrix B in parallel. Since >> it cannot be partitioned by row, i generated a 4000*27 matrix and try to >> get the transpose of it. Should i use MatCreateTranspose()? >> >> 2. Does MatMatSolve() solve Ax=B in parallel? >> >> > > Depends whether A is parallel, but you don't want it to anyway. > > > >> If so, since A is not >> partitioned, is A visible in every process? or other steps i should do? >> > > You should redundantly create A and redundantly create parts of B. That > is, every process will have > > A : 27x27 dense matrix on COMM_SELF. > B : 27x(4000/np) dense matrix on COMM_SELF. > > > np is the number of processes in COMM_WORLD > > >> 3. I have a 3D array (with PETSC_COMM_SELF) that will be used to >> generate all the elements of matrix B, how could I make it available in >> every process? >> > > Where did it come from? (How did you get this array in the first place?) > > From adam1.byrd at gmail.com Wed Aug 31 10:13:42 2011 From: adam1.byrd at gmail.com (Adam Byrd) Date: Wed, 31 Aug 2011 11:13:42 -0400 Subject: [petsc-users] Mumps Error Message-ID: Dear Users, I'm having trouble figuring out why the MUMPS solver is failing on a specific range of one of my parameters. When using the PETSc direct solver on a single processor I have no issues. There error is: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Error in external library! [0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFO(1)=-9, INFO(2)=13 ! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./cntor on a complex-c named hpc-1-14.local by abyrd Wed Aug 31 10:53:42 2011 [0]PETSC ERROR: Libraries linked from /panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib [0]PETSC ERROR: Configure run at Mon Jul 11 15:28:42 2011 [0]PETSC ERROR: Configure options PETSC_ARCH=complex-cpp-mumps --with-cc=mpicc --with-fc=mpif90 --with-blas-lapack-dir=/usr/lib64 --with-shared --with-clanguage=c++ --with-scalar-type=complex --download-mumps=1 --download-blacs=1 --download-scalapack=1 --download-parmetis=1 --with-cxx=mpicxx [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatFactorNumeric_MUMPS() line 517 in src/mat/impls/aij/mpi/mumps/mumps.c [0]PETSC ERROR: MatLUFactorNumeric() line 2587 in src/mat/interface/matrix.c [0]PETSC ERROR: PCSetUp_LU() line 158 in src/ksp/pc/impls/factor/lu/lu.c [0]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: InvertHamiltonian() line 102 in WDinvert.h I suspect it has something to do with the preconditioning or setup of the matrix I am trying to invert. The matrix becomes singular at energy = 0 eV, and is nearly singular for values close to that, but the code is failing on energies relatively far from that point. The affected energy interval is [-0.03095, 0.03095]. Is anyone able to point in the right direction to figure out what I'm not setting up properly? Respectfully, Adam Byrd -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PETScCntor.zip Type: application/zip Size: 80941 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Wed Aug 31 10:14:09 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 31 Aug 2011 10:14:09 -0500 Subject: [petsc-users] about MatTranspose In-Reply-To: <0c9c2be4b200e3bfddcb930ae8e02b58.squirrel@webmail.andrew.cmu.edu> References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> <244e0215f03d77322ee865b6ee7a5ea9.squirrel@webmail.andrew.cmu.edu> <45C4BE1F-928F-4E34-8AAB-7D26AA34995B@mcs.anl.gov> <92ee3133d6e2801fc1e3f727947cf3e6.squirrel@webmail.andrew.cmu.edu> <4bf0fcffeb9f2c0d9b6206327df6e8c1.squirrel@webmail.andrew.cmu.edu> <62da0198e4d4ef47b647666e720c2dc9.squirrel@webmail.andrew.cmu.edu> <0c9c2be4b200e3bfddcb930ae8e02b58.squirrel@webmail.andrew.cmu.edu> Message-ID: On Wed, Aug 31, 2011 at 10:02, Likun Tan wrote: > MatCreateSeqDense(PETSC_COMM_SELF, M, M, PESTC_NULL, &A) > yes > MatCreateMPIDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, M, N, > PETSC_NULL, &B) > No, use MatCreateSeqDense() here too. Do you also need the result array on every process? PetscInt n = PETSC_DECIDE; PetscSplitOwnership(PETSC_COMM_WORLD,&n,&N); MatCreateSeqDense(PETSC_COMM_SELF,M,n,PETSC_NULL,&B); MatSetValues() //Set values to A and B > MatCholeskyFactor()// Factorization > MatDuplicate(B,MAT_DO_NOT_COPY_VALUES,&X); > MatMatSolve() > > The 3D array is generated before setting values to A and B. It comes from > some known function evaluation, an i need the array in every process. > Okay, just do the same thing on every process, but only set the values into B for the part that will be done locally. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Aug 31 11:22:16 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 31 Aug 2011 11:22:16 -0500 Subject: [petsc-users] Mumps Error In-Reply-To: References: Message-ID: Adam, You need to use the MUMPS documentation to interpret what the error INFO(1)=-9, INFO(2)=13 means and then go from that. Barry On Aug 31, 2011, at 10:13 AM, Adam Byrd wrote: > Dear Users, > > I'm having trouble figuring out why the MUMPS solver is failing on a specific range of one of my parameters. When using the PETSc direct solver on a single processor I have no issues. There error is: > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Error in external library! > [0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFO(1)=-9, INFO(2)=13 > ! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./cntor on a complex-c named hpc-1-14.local by abyrd Wed Aug 31 10:53:42 2011 > [0]PETSC ERROR: Libraries linked from /panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib > [0]PETSC ERROR: Configure run at Mon Jul 11 15:28:42 2011 > [0]PETSC ERROR: Configure options PETSC_ARCH=complex-cpp-mumps --with-cc=mpicc --with-fc=mpif90 --with-blas-lapack-dir=/usr/lib64 --with-shared --with-clanguage=c++ --with-scalar-type=complex --download-mumps=1 --download-blacs=1 --download-scalapack=1 --download-parmetis=1 --with-cxx=mpicxx > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatFactorNumeric_MUMPS() line 517 in src/mat/impls/aij/mpi/mumps/mumps.c > [0]PETSC ERROR: MatLUFactorNumeric() line 2587 in src/mat/interface/matrix.c > [0]PETSC ERROR: PCSetUp_LU() line 158 in src/ksp/pc/impls/factor/lu/lu.c > [0]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: InvertHamiltonian() line 102 in WDinvert.h > > > I suspect it has something to do with the preconditioning or setup of the matrix I am trying to invert. The matrix becomes singular at energy = 0 eV, and is nearly singular for values close to that, but the code is failing on energies relatively far from that point. The affected energy interval is [-0.03095, 0.03095]. > > Is anyone able to point in the right direction to figure out what I'm not setting up properly? > > Respectfully, > Adam Byrd > From likunt at andrew.cmu.edu Wed Aug 31 12:31:27 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Wed, 31 Aug 2011 13:31:27 -0400 Subject: [petsc-users] about MatTranspose In-Reply-To: References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> <45C4BE1F-928F-4E34-8AAB-7D26AA34995B@mcs.anl.gov> <92ee3133d6e2801fc1e3f727947cf3e6.squirrel@webmail.andrew.cmu.edu> <4bf0fcffeb9f2c0d9b6206327df6e8c1.squirrel@webmail.andrew.cmu.edu> <62da0198e4d4ef47b647666e720c2dc9.squirrel@webmail.andrew.cmu.edu> <0c9c2be4b200e3bfddcb930ae8e02b58.squirrel@webmail.andrew.cmu.edu> Message-ID: <5e68241a6acd759ab3a3245e4c468503.squirrel@webmail.andrew.cmu.edu> Thank you very much. I implemented the following, but it seems not working: (I set np=2, basically i try to store the matrix B in 2 processors by column and the 3D array in 2 processors in z direction, since the columns are independent to one another) PetscInt petn=PETSC_DECIDE; PetscSplitOwnership(PETSC_COMM_WORLD, &petn, &N); MatCreateSeqDense(PETSC_COMM_SELF, M, petn, PETSC_NULL, &B); DACreate3d(PETSC_COMM_WORLD, DA_NONPERIODIC, DA_STENCIL_STAR, M, P, N, 1, 1, petn, 1, 0, PETSC_NULL, PETSC_NULL, &da3D); //set values to da3D with DAGetCorners(da3D, &xs, &ys, &zs, &m, &n, &p) DAVecGetArray()... //Set values to B with for(k=0, k On Wed, Aug 31, 2011 at 10:02, Likun Tan wrote: > > >> MatCreateSeqDense(PETSC_COMM_SELF, M, M, PESTC_NULL, &A) >> >> > > yes > > >> MatCreateMPIDense(PETSC_COMM_WORLD, PETSC_DECIDE, PETSC_DECIDE, M, N, >> PETSC_NULL, &B) >> >> > > No, use MatCreateSeqDense() here too. Do you also need the result array > on every process? > > PetscInt n = PETSC_DECIDE; > PetscSplitOwnership(PETSC_COMM_WORLD,&n,&N); > MatCreateSeqDense(PETSC_COMM_SELF,M,n,PETSC_NULL,&B); > > > MatSetValues() //Set values to A and B > >> MatCholeskyFactor()// Factorization >> >> > > MatDuplicate(B,MAT_DO_NOT_COPY_VALUES,&X); > > > >> MatMatSolve() >> >> > > >> The 3D array is generated before setting values to A and B. It comes >> from some known function evaluation, an i need the array in every >> process. >> > > Okay, just do the same thing on every process, but only set the values > into B for the part that will be done locally. > > From jedbrown at mcs.anl.gov Wed Aug 31 12:45:36 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 31 Aug 2011 12:45:36 -0500 Subject: [petsc-users] about MatTranspose In-Reply-To: <5e68241a6acd759ab3a3245e4c468503.squirrel@webmail.andrew.cmu.edu> References: <3033ca5d22302bbf5272f571c73a092f.squirrel@webmail.andrew.cmu.edu> <45C4BE1F-928F-4E34-8AAB-7D26AA34995B@mcs.anl.gov> <92ee3133d6e2801fc1e3f727947cf3e6.squirrel@webmail.andrew.cmu.edu> <4bf0fcffeb9f2c0d9b6206327df6e8c1.squirrel@webmail.andrew.cmu.edu> <62da0198e4d4ef47b647666e720c2dc9.squirrel@webmail.andrew.cmu.edu> <0c9c2be4b200e3bfddcb930ae8e02b58.squirrel@webmail.andrew.cmu.edu> <5e68241a6acd759ab3a3245e4c468503.squirrel@webmail.andrew.cmu.edu> Message-ID: On Wed, Aug 31, 2011 at 12:31, Likun Tan wrote: > Thank you very much. > > I implemented the following, but it seems not working: > "not working" isn't very helpful. What exactly did you try, what did you expect, and what happened (paste full error messages)? -------------- next part -------------- An HTML attachment was scrubbed... URL: From likunt at andrew.cmu.edu Wed Aug 31 13:02:52 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Wed, 31 Aug 2011 14:02:52 -0400 Subject: [petsc-users] about MatTranspose Message-ID: The error is Segmentation Violation, i think this is because when i generate matrix B, i use the part of the 3D array that is not in the same processor as B. Each column of B only depends on the values in the corresponding z direction of the 3D array. That is to say, the Kth column of B only requires the data Array[i][j][K] for all i and j. Theoretically if i split the matirx B in 2 processors by column and the 3D array in 2 processors in z direction, the evaluation of B is compatible. However, when i implemented the following code, it gives segmentation violation error: PetscInt petn=PETSC_DECIDE; PetscSplitOwnership(PETSC_COMM_WORLD, &petn, &N); MatCreateSeqDense(PETSC_COMM_SELF, M, petn, PETSC_NULL, &B); DACreate3d(PETSC_COMM_WORLD, DA_NONPERIODIC, DA_STENCIL_STAR, M, P, N, 1, 1, petn, 1, 0, PETSC_NULL, PETSC_NULL, &da3D); //set values to da3D with DAGetCorners(da3D, &xs, &ys, &zs, &m, &n, &p) DAVecGetArray()... //Set values to B with for(k=0, k On Wed, Aug 31, 2011 at 12:31, Likun Tan wrote: > > >> Thank you very much. >> >> >> I implemented the following, but it seems not working: >> >> > > "not working" isn't very helpful. What exactly did you try, what did you > expect, and what happened (paste full error messages)? > From jedbrown at mcs.anl.gov Wed Aug 31 13:05:52 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 31 Aug 2011 13:05:52 -0500 Subject: [petsc-users] about MatTranspose In-Reply-To: References: Message-ID: On Wed, Aug 31, 2011 at 13:02, Likun Tan wrote: > However, when i implemented the following code, it gives segmentation > violation error: > Run with Valgrind (valgrind.org). -------------- next part -------------- An HTML attachment was scrubbed... URL: From likunt at andrew.cmu.edu Wed Aug 31 15:23:18 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Wed, 31 Aug 2011 16:23:18 -0400 Subject: [petsc-users] about DA Message-ID: Hello, I have a quick question about DA. When use DAVecGetArray() to set values to the array and run the program with mpiexec, does DAVecGetArray() compute in parallel? Thanks in advance, Likun From ecoon at lanl.gov Wed Aug 31 16:11:30 2011 From: ecoon at lanl.gov (Ethan Coon) Date: Wed, 31 Aug 2011 15:11:30 -0600 Subject: [petsc-users] about DA In-Reply-To: References: Message-ID: <1314825090.11494.4.camel@echo.lanl.gov> On Wed, 2011-08-31 at 16:23 -0400, Likun Tan wrote: > Hello, > > I have a quick question about DA. When use DAVecGetArray() to set values > to the array and run the program with mpiexec, does DAVecGetArray() > compute in parallel? > DAVecGetArray() only gives you access to the local portion (plus any ghost nodes) of the global vec. The portion of the data you get is indexed by global numbering, so the indices into the pointer should range from xs --> xs+m where these values are given by DAGetGhostCorners() or DAGetCorners() http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/DA/DAGetGhostCorners.html#DAGetGhostCorners So yes, you should only do the local portion of the calculation to set set the values. Ethan > Thanks in advance, > Likun > > > > -- ------------------------------------ Ethan Coon Post-Doctoral Researcher Applied Mathematics - T-5 Los Alamos National Laboratory 505-665-8289 http://www.ldeo.columbia.edu/~ecoon/ ------------------------------------ From xyuan at lbl.gov Wed Aug 31 16:40:34 2011 From: xyuan at lbl.gov (Xuefei (Rebecca) Yuan) Date: Wed, 31 Aug 2011 14:40:34 -0700 Subject: [petsc-users] The default nonzero pattern for SuperLU_dist. Message-ID: <2EF00818-6652-4E41-86FC-475AEF952AEB@lbl.gov> Hello all, Is the default nonzero pattern for SuperLU_dist is SAME_NONZERO_PATTERN? Thanks! Best, Rebecca From bsmith at mcs.anl.gov Wed Aug 31 16:42:41 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 31 Aug 2011 16:42:41 -0500 Subject: [petsc-users] The default nonzero pattern for SuperLU_dist. In-Reply-To: <2EF00818-6652-4E41-86FC-475AEF952AEB@lbl.gov> References: <2EF00818-6652-4E41-86FC-475AEF952AEB@lbl.gov> Message-ID: <02A1F7C5-433E-448C-9D2F-3FE26B84B9DA@mcs.anl.gov> On Aug 31, 2011, at 4:40 PM, Xuefei (Rebecca) Yuan wrote: > Hello all, > > Is the default nonzero pattern for SuperLU_dist is SAME_NONZERO_PATTERN? There is no default nonzero pattern. You always must set it when you call KSPSetOperators() or in your SNES FormFunction. The PETSc SUPERLU_dist interface allows either pattern and should work correctly. Barry > > Thanks! > > Best, > > Rebecca > From xyuan at lbl.gov Wed Aug 31 16:48:34 2011 From: xyuan at lbl.gov (Xuefei (Rebecca) Yuan) Date: Wed, 31 Aug 2011 14:48:34 -0700 Subject: [petsc-users] The default nonzero pattern for SuperLU_dist. In-Reply-To: <02A1F7C5-433E-448C-9D2F-3FE26B84B9DA@mcs.anl.gov> References: <2EF00818-6652-4E41-86FC-475AEF952AEB@lbl.gov> <02A1F7C5-433E-448C-9D2F-3FE26B84B9DA@mcs.anl.gov> Message-ID: <36F5FE78-2395-4E99-95A7-25C5477EDDFC@lbl.gov> Dear Barry, Thanks for your kind reply. So if I set up the following: ierr = DAGetMatrix(DMMGGetDA(dmmg), MATAIJ, &jacobian);CHKERRQ(ierr); KSP ksp; ierr = SNESGetKSP(DMMGGetSNES(dmmg),&ksp);CHKERRQ(ierr); ierr = KSPSetOperators(ksp,jacobian,jacobian,SAME_NONZERO_PATTERN);CHKERRQ(ierr); ierr = DMMGSolve(dmmg);CHKERRQ(ierr); This will specify the nonzero pattern as the same? How could I check if I have set this up right? Thanks, Rebecca On Aug 31, 2011, at 2:42 PM, Barry Smith wrote: > > On Aug 31, 2011, at 4:40 PM, Xuefei (Rebecca) Yuan wrote: > >> Hello all, >> >> Is the default nonzero pattern for SuperLU_dist is SAME_NONZERO_PATTERN? > > There is no default nonzero pattern. You always must set it when you call KSPSetOperators() or in your SNES FormFunction. The PETSc SUPERLU_dist interface allows either pattern and should work correctly. > > > Barry > >> >> Thanks! >> >> Best, >> >> Rebecca >> > From likunt at andrew.cmu.edu Wed Aug 31 17:37:06 2011 From: likunt at andrew.cmu.edu (Likun Tan) Date: Wed, 31 Aug 2011 18:37:06 -0400 Subject: [petsc-users] about DA In-Reply-To: <1314825090.11494.4.camel@echo.lanl.gov> References: <1314825090.11494.4.camel@echo.lanl.gov> Message-ID: Thanks, Likun On Wed, August 31, 2011 5:11 pm, Ethan Coon wrote: > On Wed, 2011-08-31 at 16:23 -0400, Likun Tan wrote: > >> Hello, >> >> >> I have a quick question about DA. When use DAVecGetArray() to set >> values to the array and run the program with mpiexec, does >> DAVecGetArray() >> compute in parallel? >> > > DAVecGetArray() only gives you access to the local portion (plus any > ghost nodes) of the global vec. The portion of the data you get is indexed > by global numbering, so the indices into the pointer should range from > > xs --> xs+m > > where these values are given by > > DAGetGhostCorners() or DAGetCorners() > > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manual > pages/DA/DAGetGhostCorners.html#DAGetGhostCorners > > So yes, you should only do the local portion of the calculation to set > set the values. > > Ethan > > >> Thanks in advance, >> Likun >> >> >> >> >> > > -- > ------------------------------------ > Ethan Coon > Post-Doctoral Researcher > Applied Mathematics - T-5 > Los Alamos National Laboratory > 505-665-8289 > > > http://www.ldeo.columbia.edu/~ecoon/ > ------------------------------------ > > > > From rudolph at berkeley.edu Wed Aug 31 17:34:29 2011 From: rudolph at berkeley.edu (Max Rudolph) Date: Wed, 31 Aug 2011 15:34:29 -0700 Subject: [petsc-users] Mumps Error In-Reply-To: References: Message-ID: <8D782424-55CE-4C7A-9E6C-38A782D770B3@berkeley.edu> I'm not sure if you figured out a solution yet, but I think that you might want to run with -mat_mumps_icntl_14 100 Max >> Dear Users, >> >> I'm having trouble figuring out why the MUMPS solver is failing on a specific range of one of my parameters. When using the PETSc direct solver on a single processor I have no issues. There error is: >> >> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [0]PETSC ERROR: Error in external library! >> [0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFO(1)=-9, INFO(2)=13 >> ! >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./cntor on a complex-c named hpc-1-14.local by abyrd Wed Aug 31 10:53:42 2011 >> [0]PETSC ERROR: Libraries linked from /panfs/storage.local/scs/home/abyrd/petsc-3.1-p8/complex-cpp-mumps/lib >> [0]PETSC ERROR: Configure run at Mon Jul 11 15:28:42 2011 >> [0]PETSC ERROR: Configure options PETSC_ARCH=complex-cpp-mumps --with-cc=mpicc --with-fc=mpif90 --with-blas-lapack-dir=/usr/lib64 --with-shared --with-clanguage=c++ --with-scalar-type=complex --download-mumps=1 --download-blacs=1 --download-scalapack=1 --download-parmetis=1 --with-cxx=mpicxx >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: MatFactorNumeric_MUMPS() line 517 in src/mat/impls/aij/mpi/mumps/mumps.c >> [0]PETSC ERROR: MatLUFactorNumeric() line 2587 in src/mat/interface/matrix.c >> [0]PETSC ERROR: PCSetUp_LU() line 158 in src/ksp/pc/impls/factor/lu/lu.c >> [0]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: InvertHamiltonian() line 102 in WDinvert.h >> >> >> I suspect it has something to do with the preconditioning or setup of the matrix I am trying to invert. The matrix becomes singular at energy = 0 eV, and is nearly singular for values close to that, but the code is failing on energies relatively far from that point. The affected energy interval is [-0.03095, 0.03095]. >> >> Is anyone able to point in the right direction to figure out what I'm not setting up properly? >> >> Respectfully, >> Adam Byrd >> > > > > > _______________________________________________ > petsc-users mailing list > petsc-users at mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users From rongliang.chan at gmail.com Wed Aug 31 19:15:14 2011 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Wed, 31 Aug 2011 18:15:14 -0600 Subject: [petsc-users] Matrix is missing diagonal entry Message-ID: Hello, I got the error message "Matrix is missing diagonal entry 23548!" when I tried to run my code. I print out the 23548th row and find that the 23548th element of this row is one and the others are zero. Can anyone tell me what's maybe the problem and how to fix this? Thanks. Best Regards, Rongliang /bin/bash: SHELL: readonly variable /bin/bash: PATH: readonly variable [255]PETSC ERROR: --------------------- Error Message ------------------------------------ [255]PETSC ERROR: Object is in wrong state! [255]PETSC ERROR: Matrix is missing diagonal entry 23548! [255]PETSC ERROR: ------------------------------------------------------------------------ [255]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011 [255]PETSC ERROR: See docs/changes/index.html for recent updates. [255]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [255]PETSC ERROR: See docs/index.html for manual pages. [255]PETSC ERROR: ------------------------------------------------------------------------ [255]PETSC ERROR: /home/rchen/soft/fixedmesh/Cannula/./joab on a bgl-ibm-g named R02M1N0 by Unknown Wed Aug 31 17:36:32 2011 [255]PETSC ERROR: Libraries linked from /home/rchen/soft/petsc-3.1-p8-nodebug/bgl-ibm-goto-O3_440d/lib [255]PETSC ERROR: ------------------------------------------------------------------------ [255]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ_ilu0() line 1627 in src/mat/impls/aij/seq/aijfact.c [255]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ() line 1731 in src/mat/impls/aij/seq/aijfact.c [255]PETSC ERROR: MatILUFactorSymbolic() line 5464 in src/mat/interface/matrix.c [255]PETSC ERROR: PCSetUp_ILU() line 204 in src/ksp/pc/impls/factor/ilu/ilu.c [255]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c [255]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c [255]PETSC ERROR: PCSetUpOnBlocks_ASM() line 335 in src/ksp/pc/impls/asm/asm.c [255]PETSC ERROR: PCSetUpOnBlocks() line 828 in src/ksp/pc/interface/precon.c [255]PETSC ERROR: KSPSetUpOnBlocks() line 159 in src/ksp/ksp/interface/itfunc.c [255]PETSC ERROR: KSPSolve() line 354 in src/ksp/ksp/interface/itfunc.c [255]PETSC ERROR: SNES_KSPSolve() line 2944 in src/snes/interface/snes.c [255]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c [255]PETSC ERROR: TimeStep() line 1462 in /home/rlchen/rlchen/soft/fixedmesh/code_changed/joab.c [255]PETSC ERROR: SolveSteadyState() line 1362 in /home/rlchen/rlchen/soft/fixedmesh/code_changed/joab.c [255]PETSC ERROR: ComputefixedBoundary() line 505 in /home/rlchen/rlchen/soft/fixedmesh/code_changed/joab.c [255]PETSC ERROR: JoabCtxSolve() line 165 in /home/rlchen/rlchen/soft/fixedmesh/code_changed/joab.c [255]PETSC ERROR: main() line 107 in /home/rlchen/rlchen/soft/fixedmesh/code_changed/joab.c ..................... -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Aug 31 21:34:00 2011 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 31 Aug 2011 21:34:00 -0500 Subject: [petsc-users] Matrix is missing diagonal entry In-Reply-To: References: Message-ID: Maybe a mistake implementing boundary conditions? On Aug 31, 2011 7:15 PM, "Rongliang Chen" wrote: > Hello, > > I got the error message "Matrix is missing diagonal entry 23548!" when I > tried to run my code. I print out the 23548th row and find that the 23548th > element of this row is one and the others are zero. Can anyone tell me > what's maybe the problem and how to fix this? Thanks. > > Best Regards, > Rongliang > > /bin/bash: SHELL: readonly variable > /bin/bash: PATH: readonly variable > [255]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [255]PETSC ERROR: Object is in wrong state! > [255]PETSC ERROR: Matrix is missing diagonal entry 23548! > [255]PETSC ERROR: > ------------------------------------------------------------------------ > [255]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 > CDT 2011 > [255]PETSC ERROR: See docs/changes/index.html for recent updates. > [255]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [255]PETSC ERROR: See docs/index.html for manual pages. > [255]PETSC ERROR: > ------------------------------------------------------------------------ > [255]PETSC ERROR: /home/rchen/soft/fixedmesh/Cannula/./joab on a bgl-ibm-g > named R02M1N0 by Unknown Wed Aug 31 17:36:32 2011 > [255]PETSC ERROR: Libraries linked from > /home/rchen/soft/petsc-3.1-p8-nodebug/bgl-ibm-goto-O3_440d/lib > > [255]PETSC ERROR: > ------------------------------------------------------------------------ > [255]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ_ilu0() line 1627 in > src/mat/impls/aij/seq/aijfact.c > [255]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ() line 1731 in > src/mat/impls/aij/seq/aijfact.c > [255]PETSC ERROR: MatILUFactorSymbolic() line 5464 in > src/mat/interface/matrix.c > [255]PETSC ERROR: PCSetUp_ILU() line 204 in > src/ksp/pc/impls/factor/ilu/ilu.c > [255]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c > [255]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c > [255]PETSC ERROR: PCSetUpOnBlocks_ASM() line 335 in > src/ksp/pc/impls/asm/asm.c > [255]PETSC ERROR: PCSetUpOnBlocks() line 828 in > src/ksp/pc/interface/precon.c > [255]PETSC ERROR: KSPSetUpOnBlocks() line 159 in > src/ksp/ksp/interface/itfunc.c > [255]PETSC ERROR: KSPSolve() line 354 in src/ksp/ksp/interface/itfunc.c > [255]PETSC ERROR: SNES_KSPSolve() line 2944 in src/snes/interface/snes.c > [255]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c > [255]PETSC ERROR: TimeStep() line 1462 in > /home/rlchen/rlchen/soft/fixedmesh/code_changed/joab.c > [255]PETSC ERROR: SolveSteadyState() line 1362 in > /home/rlchen/rlchen/soft/fixedmesh/code_changed/joab.c > [255]PETSC ERROR: ComputefixedBoundary() line 505 in > /home/rlchen/rlchen/soft/fixedmesh/code_changed/joab.c > [255]PETSC ERROR: JoabCtxSolve() line 165 in > /home/rlchen/rlchen/soft/fixedmesh/code_changed/joab.c > [255]PETSC ERROR: main() line 107 in > /home/rlchen/rlchen/soft/fixedmesh/code_changed/joab.c > ..................... -------------- next part -------------- An HTML attachment was scrubbed... URL: